Documents Show IBM Pitched The NYPD Facial Recognition Software With Built-In Racial Profiling Options

from the facial-profiling-amirite dept

Documents obtained by The Intercept show the NYPD and IBM engaged in a long-running facial recognition tech partnership from 2008 to 2016. While some of this deployment was discussed publicly, details about the extent of the program -- as well as it's more problematic elements -- haven't been.

As the article's title informs the reader, camera footage could be scanned for face matches using skin tone as a search constraint. Considering this was pushed by IBM as a tool to prevent the next 9/11, it's easy to see why the NYPD -- given its history of surveilling Muslim New Yorkers -- might be willing to utilize a tool like this to pare down lists of suspects to just the people it suspected all along (Muslims).

There are a number of surprises contained in the long, detailed article, but the first thing that jumps out is IBM's efforts and statements, rather than the NYPD's. We all know the government capitalizes on tragedies to expand its power, but here we see a private corporation appealing to this base nature to make a sale.

In New York, the terrorist threat “was an easy selling point,” recalled Jonathan Connell, an IBM researcher who worked on the initial NYPD video analytics installation. “You say, ‘Look what the terrorists did before, they could come back, so you give us some money and we’ll put a camera there.”

From this pitch sprung an 8-year program -- deployed in secrecy by the NYPD to gather as much footage as possible of New Yorkers for dual purposes: its own law enforcement needs and to serve as a testing ground for IBM's new facial recognition tech. Needless to say, New Yorkers were never made aware of their lab rat status in IBM's software development process.

Even though the software could search by skin tone (as well as by "head color," age, gender, and facial hair), the NYPD claims it never used that feature in a live environment, despite IBM's urging.

According to the NYPD, counterterrorism personnel accessed IBM’s bodily search feature capabilities only for evaluation purposes, and they were accessible only to a handful of counterterrorism personnel. “While tools that featured either racial or skin tone search capabilities were offered to the NYPD, they were explicitly declined by the NYPD,” Donald, the NYPD spokesperson, said. “Where such tools came with a test version of the product, the testers were instructed only to test other features (clothing, eyeglasses, etc.), but not to test or use the skin tone feature. That is not because there would have been anything illegal or even improper about testing or using these tools to search in the area of a crime for an image of a suspect that matched a description given by a victim or a witness. It was specifically to avoid even the suggestion or appearance of any kind of technological racial profiling.”

It's easy to disbelieve this statement by the NYPD, given its long history of racial profiling, but it may be those handling the secret program deployment actually understood no program remains secret forever and sought to head off complaints and lawsuits by discouraging use of a controversial search feature. It also may be the NYPD was super-sensitive to these concerns following the partial dismantling of its stop-and-frisk program and the outing of its full-fledged, unconstitutional surveillance of local Muslims.

The thing is IBM is still selling this tech it beta tested live from New York. The same features the NYPD rejected are used to sell other law enforcement agencies on the power of its biometric profiling software.

In 2017, IBM released Intelligent Video Analytics 2.0, a product with a body camera surveillance capability that allows users to detect people captured on camera by “ethnicity” tags, such as “Asian,” “Black,” and “White.”

And there's a counter-narrative that seems to dispute the NYPD's assertions about controversial image tagging features. The IBM researcher who helped develop the skin tone recognition feature is on record stating the company doesn't develop features unless there's a market for them. In his estimation, the NYPD approached IBM to ask for this feature while the 8-year pilot program was still underway. The NYPD may have opted out after the feature went live, but it may have only done so to steer clear of future controversy. An ulterior motive doesn't make it the wrong move, but it also shouldn't be assumed the NYPD has morphed into heroic defenders of civil liberties and personal privacy.

What's available to other law enforcement agencies not similarly concerned about future PR black eyes is "mass racial profiling" at their fingertips. IBM has built a product that appeals to law enforcement's innate desire to automate police work, replacing officers on the street with cameras and software. Sure, there will be some cameras on patrol officers as well, but those are just for show. The real work of policing is done at desks using third-party software that explicitly allows -- if not encourages -- officers to narrow down suspect lists based on race. In a country so overly concerned about terrorism, this is going to lead to a lot of people being approached by law enforcement simply because of their ethnicity.

An additional problem with IBM's software -- and with those produced by competitors -- is a lot of markers used to identify potential suspects can easily net a long list of probables who share nothing but similar body sizes or clothing preferences. Understandably, more work is done by investigators manning these systems before cops start rounding people up, but the potential for inadvertent misuse (never mind actual misuse) is still incredibly high.

The secrecy of these programs is also an issue. Restrictive NDAs go hand-in-hand with private sector partnerships and these are often translated by police officials to mean information must be withheld from judges, criminal defendants, and department oversight. When that happens, due process violations gather atop the privacy violation wreckage until the whole thing collapses under its own audacity. Nothing stays secret forever, but entities like the NYPD and IBM could do themselves a bunch of favors by engaging in a little proactive transparency.

Filed Under: facial recognition, new york, nypd, racial profiling
Companies: ibm

How Facebook's Racial Segmentation Is Helping Trump Campaign Try To Suppress African American Voting

from the where-are-the-adults dept

Earlier this week, Bloomberg had a fairly revealing article about the internal digital efforts of the Donald Trump campaign, in which Bloomberg reporters embedded for a few days. The whole article is quite interesting, but one of the most stunning parts, frankly, was the Trump campaign staffers directly admitting how they are actively trying to suppress voting by African Americans. It's no secret that a variety of new voter ID laws are designed to suppress voting -- especially among minorities. When North Carolina's voter ID law was struck down by the court, the judge pointed out how the legislators that had backed it had explicitly targeted rules that would suppress votes among African Americans. They had requested "racial data" concerning voter ID and then specifically targeted the types of ID more commonly used by African Americans.

In her remarkable opinion, Judge Motz strongly suggests that North Carolina’s law was indeed racist. The day following the release of Shelby County, she noted, a GOP leader in the state legislature announced his intention to write a law that the feds would have no authority to vet before it went into effect. Like laws in other Republican states, the North Carolina bill imposed a tough new photo-ID requirement. But it did much more: the law eliminated same-day voter registration and pre-registration for high-school students about to turn 18, curtailed early voting by one week and banned out-of-precinct voting.

Each of these new rules disproportionately impacted black voters seeking to exercise the franchise, as legislators in North Carolina were well aware. “[P]rior to enactment” of the law, the Fourth Circuit explained, “the legislature requested and received racial data as to usage of the practices changed by the proposed law.” Released from the obligation to clear their law with the Justice department and “with race data in hand, the legislature amended the bill to exclude many of the alternative photo IDs used by African Americans.” Photo IDs used more often by black voters, including public assistance IDs, were removed from the list of acceptable identification, while IDs issued by the Department of Motor Vehicles—which blacks are less likely to have—were retained. Cutting the first week of early voting came in reaction to data showing that the first seven days were used by large numbers of black voters, nixing one Sunday on which churches would bus “souls-to-the-polls”. Banning same-day registration, too, had an outsize effect on blacks, as did the prohibition on out-of-precinct voting: both changes made voting harder for people who had recently moved, and blacks are more itinerant than whites.

That, alone, was pretty stunning, but they still tried to pretend in public that the law wasn't about suppressing the vote. However, when put with a Bloomberg reporter, the Trump campaign flat out brags about its efforts to suppress the vote among African Americans. And they're using extreme targeting on Facebook to do so:

Instead of expanding the electorate, Bannon and his team are trying to shrink it. “We have three major voter suppression operations under way,” says a senior official. They’re aimed at three groups Clinton needs to win overwhelmingly: idealistic white liberals, young women, and African Americans. Trump’s invocation at the debate of Clinton’s WikiLeaks e-mails and support for the Trans-Pacific Partnership was designed to turn off Sanders supporters. The parade of women who say they were sexually assaulted by Bill Clinton and harassed or threatened by Hillary is meant to undermine her appeal to young women. And her 1996 suggestion that some African American males are “super predators” is the basis of a below-the-radar effort to discourage infrequent black voters from showing up at the polls—particularly in Florida.

On Oct. 24, Trump’s team began placing spots on select African American radio stations. In San Antonio, a young staffer showed off a South Park-style animation he’d created of Clinton delivering the “super predator” line (using audio from her original 1996 sound bite), as cartoon text popped up around her: “Hillary Thinks African Americans are Super Predators.” The animation will be delivered to certain African American voters through Facebook “dark posts”—nonpublic posts whose viewership the campaign controls so that, as Parscale puts it, “only the people we want to see it, see it.” The aim is to depress Clinton’s vote total. “We know because we’ve modeled this,” says the official. “It will dramatically affect her ability to turn these people out.”

Now that's... interesting (and ridiculous, but we'll leave that aside for the moment). Of course, every election cycle involves a ton of targeted "negative advertising" that is designed to suppress overall interest in a candidate. But the two things newsworthy here are (1) the fact that the Trump campaign is directly admitting to the intention behind that strategy here, rather than hiding it and (2) the ability to use Facebook to target these kinds of campaigns to a level previously not available.

Facebook, somewhat famously, allows extraordinarily targeted advertising. We've played around with it ourselves, and it's really quite incredible how granular you can go in trying to target your ads. Basically any trait or interest or demographic group that you can think of, you can put into an ad target group. At times, as you dig through the options, it almost feels like it's just Facebook showing off just how much data and insight it has into its users. It's a data nerd's dream, where you can slice and dice billions of people by basically anything.

Of course, it's somewhat ironic that the Trump campaign is using Facebook to suppress the vote, at the same time that Facebook is patting itself on the back for helping to get out the vote with its voter registration campaign, and, in the past has directly experimented with changing newsfeeds to encourage more voter turnout. Platforms like Facebook can be used for both good and evil.

Either way, sometimes the data nerds (and the advertising folks) have to be reminded of the law. Pro Publica has a pretty damning report out today about the fact that Facebook's slicing and dicing of targeted advertising also means that you can exclude people by race. They don't discuss the recent revelations about the Trump campaign's targeting, but it's pretty clear that this is how they're doing that suppression campaign described above. But it also presents potentially serious legal problem in areas where it is illegal to discriminate based on race, such as hiring or housing. And yet, Facebook's current set up allows users to do just that: Propublica actually went ahead and bought a housing ad that discriminated based on race.

The Propbulica article quotes a civil rights lawyer who is reasonably horrified by this. But there are some big legal questions. From the data geek side of things, you can easily see how Facebook reached this point, continually slicing up data in more and more ways, without necessarily considering the consequences. But does that make Facebook legally liable for, say, violating the Fair Housing Act? That's... a much tougher question.

Facebook argues (1) that it's policies say that advertisers cannot discriminate in illegal ways, and anyone caught doing so will face punishment. (2) Facebook is likely protected by Section 230 of the CDA on this. I say "likely" instead of "definitely" because one of the few cases that cut through the CDA 230 protections is the famous Roommates.com case, which was explicitly about racial discrimination on housing based on Roommates.com ads that violated the Fair Housing Act. However, Facebook has a much stronger argument than Roommates in that case, because part of the issue is that Roommates directly asked users for a racial preference, making it content they had designed, rather than content that the user created. Facebook can (reasonably) argue that it was just offering up millions of ways to slice and dice the data, rather than explicitly calling out racial preference. (3) Facebook says the rules are not based on "race" but "racial affinity." This is a dumb argument and Facebook should not say it ever again, and possibly apologize for even bringing up such a lame argument in the first place.

Separately, Facebook argues -- correctly -- that there are lots of cases where advertisers have perfectly legitimate reasons for targeting based on race.

Satterfield said it’s important for advertisers to have the ability to both include and exclude groups as they test how their marketing performs. For instance, he said, an advertiser “might run one campaign in English that excludes the Hispanic affinity group to see how well the campaign performs against running that ad campaign in Spanish. This is a common practice in the industry.”

That said, there's simply no reason that Facebook couldn't put in a system to recognize ads that are in a protected category in which discrimination may be an issue, and either block such usage or at least put a strong warning for the user (and alert the Facebook team to review the ad more carefully -- since all ads are reviewed before being put live). It's not clear that there's a legal mandate to do so, but it just seems like a good practice in general. I've seen lots of people commenting on this story in which they are rightfully horrified about the potential abuse of such a tool, and they're quick to blame Facebook's "negligence." It does seem more like carelessness than negligence, in that you can see how the company got here, as it contined to alow greater and greater targeting attributes, which advertisers really appreciate.

Filed Under: advertising, donald trump, fair housing, racial profiling, section 230, targeting, voter suppression
Companies: facebook

Many FBI Agents Find Racial Profiling Useless, But Those Up Top Feel It's Just A Failure To Get Everyone On Board

from the Assistant-Regional-Director-of-First/Fourth-Amendment-Violations dept

The recently-released 9/11 Commission's review of FBI tactics in the wake of the 2001 terrorist attacks seems to suggest the agency should perform even more racial profiling than it already does. As Kevin Gosztola of Firedoglake points out, the language in the report places a lot of emphasis on "domain awareness" and pre-crime policing.

Documents the American Civil Liberties Union have been able to obtain show [PDF] that “FBI analysts make judgments based on crude stereotypes about the types of crimes different racial and ethnic groups commit, which they then use to justify collecting demographic data to map where people with that racial or ethnic makeup live.” The FBI uses “domain analysis” to target American Muslims and Islamic institutions.

The similarities between this suggested course of action and the NYPD's infamous "Demographics Unit" (led by a former CIA official) are notable. Both involve questionable tactics like declaring entire mosques "terrorist organizations" simply because attendees followed the same religion as the 9/11 attackers. Notably, the FBI found the NYPD's tactics so thoroughly violated the rights of those being surveilled that it refused to access any of the intelligence gathered by the Demographics Unit. That decision ultimately cost the FBI nothing in terms of usable intel. Despite years of rights violations and round-the-clock surveillance, the NYPD's special unit was never instrumental in preventing attacks or producing significant arrests.

Marcy Wheeler at Emptywheel notes that the FBI's analysis of the 9/11 Commission's reports indicates a significant percentage of FBI agents found racial profiling and pre-crime "investigations" to be a waste of time.

According to one anecdote, 20% of analysts (not even Field Agents!) understand the point of this. And even in offices where they do understand, the Field Agents won’t do their part by going and filling in the blanks analysts identify.

The "blanks" are contained in CSCCs (Central Strategic Coordinating Components), linked to field offices' "domain awareness" programs. But one-fifth of agents refused to comply with this directive -- not because 20% of FBI agents are necessarily against racial profiling (documents obtained by the ACLU show otherwise) -- but because the tactic just doesn't work.

Call me crazy. But maybe the people responding to actual crimes believe they learn enough in that process — and are plenty busy enough trying to catch criminals — that they don’t see the point of racially profiling people like NYPD does? Maybe they believe the ongoing threats are where the past ones have been, and there’s no need to spend their time investigating where there aren’t crimes in case there ever are in the future?

Doing investigative work like investigators, rather than like surveillance dragnets? That's probably crazy enough to work. Not that the FBI has any desire to dial back its requests for encryption backdoors and unfettered access to electronic communications, but those actually out in the field seem to know what works and what doesn't. And a constant APB for anyone fitting the "Muslim/Male" description isn't exactly helpful.

Of course, those at the top -- the ones finding this to be a credible way to fight terrorism -- see this 20% as outliers who have failed to "get on the bus." And in a mixture of the worst parts of bureaucracy and corporate culture, they've responded with "do more of what isn't productive, only faster and harder."

Yet rather than analyzing whether this concept serves any purpose whatsoever, it instead says, “it’s corporate policy, no one is doing it well, so it needs to improve.”

There's a lesson here, but those writing the review aren't comprehending it. (Wheeler notes that many of those interviewed for the report aren't actually FBI agents, but rather representatives of other intelligence agencies, like the CIA.) To catch terrorists, you need smarter investigative work, not work that involves blanket surveillance and the rote filling in of blanks. The NYPD should know this, considering its failure to catch plots later uncovered by the FBI, but it doesn't. Despite the disbandment of the "Demographics Unit," it still clings to the belief that mass surveillance beats real police work any day of the week. The FBI has figured this out -- or at least a percentage of its agents have -- but that's not going to be enough to persuade those calling for more of everything to dial back their efforts a bit.

The FBI can be smart, but it's apparently hampered by upper management with an obvious fondness for bad ideas that simultaneously expand the agency's power. If it is how it looks, the real aim of the agency heads is more power, not fewer terrorist attacks.

Filed Under: fbi, nypd, racial profiling, terrorism

TSA Racial Profiling May Hide Larger Constitutional Problem

from the oops dept

The NY Times had an article recently about accusations of racial profiling by the TSA at Boston's Logan airport. There's apparently a pilot program going on at the airport to do more "behavior detection" with the TSA. This is the security model that is often associated with Israel's airport security, and which some have argued should be adopted in a more widespread fashion. Others have pointed out problems with such a system, including the fact that without significant training, "behavior detection" reverts quite quickly to "racial profiling." That appears to be the case in Boston.

Furthermore, two years ago, Bruce Schneier reasonably pointed out that behavioral profiling did not seem very good at finding terrorists, but did uncover criminal behavior unrelated to airplane security:

It seems pretty clear that the program only catches criminals, and no terrorists. You'd think there would be more important things to spend $200 million a year on.

Again, that seems to be what's happening in Boston, as the efforts have turned up some criminal behavior. And, apparently, that's by design. Because buried deep within the NY Times article was this tidbit:

Officers said managers’ demands for high numbers of stops, searches and criminal referrals had led co-workers to target minorities in the belief that those stops were more likely to yield drugs, outstanding arrest warrants or immigration problems.

[....] The officers identified nearly two dozen co-workers who they said consistently focused on stopping minority members in response to pressure from managers to meet certain threshold numbers for referrals to the State Police, federal immigration officials or other agencies.

The stops were seen as a way of padding the program’s numbers and demonstrating to Washington policy makers that the behavior program was producing results, several officers said.

In other words, TSA managers -- apparently in an effort to make the program look good to superiors -- are putting pressure on TSA line agents to turn up exactly what Schneier suggested: some form of criminal behavior just to make the program look good. That's leading lazy TSA agents to just focus on doing searches of minorities, as they believe that they're more likely to find some sort of criminal activity completely unrelated to airplane security.

Beyond the blatant problems of racial profiling, some of the news here highlights a potentially larger problem with airport searches. As Julian Sanchez points out, the comments above suggest not just that the focus is on criminal behavior rather than security, but also that there's a quota system in place.

That could present a serious legal problem for the basis of TSA searches. After all, a series of lawsuits that established the legality of TSA airport searches focused on the fact that they were specifically designed to keep airplanes (and those on board them) safe rather than to uncover criminal activity. An excellent summary article on BoardingArea.com explains the cases that made such searches legal. Here's a snippet that covers some of the key points:

In 1973 the 9th Circuit Court rules on U.S. vs Davis, 482 F.2d 893, 908, there are key pieces of wording that give the TSA its power to search essentially any way they choose to. The key wording in this ruling includes “noting that airport screenings are considered to be administrative searches because they are conducted as part of a general regulatory scheme, where the essential administrative purpose is to prevent the carrying of weapons or explosives aboard aircraft.”

U.S. vs Davis goes onto to state “[an administrative search is allowed if] no more intrusive or intensive than necessary, in light of current technology, to detect weapons or explosives, confined in good faith to that purpose, and passengers may avoid the search by electing not to fly.”

U.S. vs Davis was upheld by the 9th Circuit Court in 1986 in U.S. vs Pulido-Baquerizo, 800 F.2d 899, 901 with this ruling “To judge reasonableness, it is necessary to balance the right to be free of intrusion with society’s interest in safe air travel.”

Except... as the case in Boston shows, the searches are going well beyond that "essential administrative purpose," and are now being used for general law enforcement. It's at that point that they clearly violate the 4th Amendment, as they're creating general law enforcement searches without proper cause or warrants. It seems that someone who was searched in Boston under these conditions could now make a pretty reasonable constitutional case that the search didn't just violate their private rights, but that the entire TSA setup -- when designed to search for criminal behavior -- has gone beyond the limits established by the courts, and now violates the 4th Amendment.

Filed Under: 4th amendment, behavioral detection, racial profiling, tsa, tsa searches