ing the news, and I’d spotted a woman ANNALS OF TECHNOLOGY on the C train in an N95 respirator mask, which had a black, satiny finish. Later, when I spoke to Arun Ross, a ADVERSARIAL MAN computer-vision researcher at Mich- igan State University, he told me that Dressing for the surveillance age. a surgical mask alone might not block enough of my face’s pixels in a digital BY JOHN SEABROOK shot to prevent a face-recognition sys- tem from making a match; some al- gorithms can reconstruct the occluded parts of people’s faces. As the corona- virus spread through China, Sense- Time, a Chinese A.I. company, claimed to have developed an algorithm that not only can match a surgically masked face with the wearer’s un-occluded face but can also use thermal imaging to detect an elevated temperature and discern whether that person is wear- ing a mask. For my purposes, a full- face covering, like the Guy Fawkes mask made popular by the “V for Vendetta” graphic novels and films, would have done the trick, but I doubt whether Amtrak would have let me on the train. During Occupy Wall Street, New York enforced old anti- mask laws to prevent protesters from wearing them. Goldstein’s invisibility cloak clashed with the leopard-print cell-signal- blocking Faraday pouch, made by Si- lent Pocket, in which I carried my phone so that my location couldn’t be tracked. As a luxury item, the cloak was far from the magnificent Jammer Coat, a prototype of anti-surveillance outerwear that I had slipped on a few weeks earlier, at Coop Himmelb(l)au, om Goldstein, an associate pro- to Penn Station, and then boarded Am- an architecture studio in Vienna. The fessor of computer science at the trak for my trip south, I counted the Jammer Coat, a one-of-a-kind, ankle- UniversityT of Maryland, took an “in- CCTV cameras; at least twenty-six length garment with a soft finish and visibility cloak” from a pile on a chair caught me going and returning. When flowing sleeves, like an Arabic thawb, in his office and pulled it on over his you come from a small town, as I do, is lined with cellular-blocking metal- head. To my eye, it looked like a baggy where everyone knows your face, pub- lic fabric and covered with patterns sweatshirt made of glossy polyester, lic anonymity—the ability to disappear that vaguely resemble body parts, which printed with garish colors in formless into a crowd—is one of the great plea- could potentially render personal tech- shapes that, far from turning Gold- sures of city living. As cities become nology invisible to electronic-object stein invisible, made him impossible surveillance centers, packed with cam- detectors. Swaddled in the cushy coat, to miss. eras that always see, public anonymity I could at least pretend to be the ab- It was mid-January. Early that morn- could vanish. Is there anything fash- solute master of my personal informa- ing, in my search for a suitable outfit ion can do? tion, even if its designers, Wolf and to thwart the all-seeing eyes of surveil- I could have worn a surgical mask Sophie Prix, wouldn’t let me leave the lance machines, I had taken the train on my trip, ostensibly for health rea- studio in it. from New York City to College Park. sons; reports of an unexplained pneu- However, the invisibility cloak, while As I rode the subway from Brooklyn monia outbreak in China were mak- not as runway-ready as some surveil- lance-wear, did have one great advan- Is there anything fashion can do to counter the erosion of public anonymity? tage over other fashion items that aim 44 THE NEW YORKER, MARCH 16, 2020 ILLUSTRATION BY ANA GALVAÑ
TNY—2020_03_16—PAGE 44—133SC—LIVE ART—R35986_RD—PLEASE USE VIRTUAL PROOF. R5: REFOLIO 4C_R5 to confuse the algorithms that control extracting information about the orien- cent years have sufficient digitized data surveillance systems: the cloak’s de- tation of edges in an image, because sets and vast cloud-based computing signer was an algorithm. edges appear the same under different resources been developed to allow this lighting conditions. Programmers tried data- and power-thirsty approach to o put together a Jammer outfit for to summarize the content of an image work. Billions of trial-and-error cycles my style of dressing—something by calculating a small list of numbers, might be required for an algorithm to likeT stealth streetwear—I first needed called “features,” which describe the ori- figure out not only what a cat looks like to understand how machines see. In entation of edges, as well as textures, but what kind of cat it is. Maryland, Goldstein told me to step colors, and shapes. “Computer-vision problems that sci- in front of a video camera that projected But the pioneers soon encountered a entists said wouldn’t be overcome in our my live image onto a large flat screen problem. The human brain has a re- lifetime were solved in a couple of years,” mounted on the wall of his office in markable ability, as it processes an ob- Goldstein had told me when we first the Iribe Center, the university’s hub ject’s components, to save the useful met, in New York. He added, “The for computer science and engineering. content, while throwing away “nuisance reason the scientific community is so The screen showed me in my winter variables,” like lighting, shadows, and shocked by these results—they ripple weeds of dark denim, navy sweater, and viewpoint. A.I. researchers couldn’t de- through everything—is that we have black sneakers. My image was being scribe exactly what makes a cat recog- this tool that achieves humanlike per- run through an object detector called nizable as a cat, let alone code this into formance that nobody ever thought we YOLO (You Only Look Once), a vision a mathematical formula that was un- would have. And suddenly not only do system widely employed in robots and affected by the infinitely variable con- we have it but it does things that are in CCTV. I looked at the camera, and ditions and scenes in which a cat might way crazier than we could have imag- that image passed along my optic nerve appear. It was impossible to code the ined. It’s sort of mind-blowing.” and into my brain. cognitive leap that our brains make when Rama Chellappa, a professor at the On the train trip down to Maryland, we generalize. Somehow, we know it’s University of Maryland who is one of I watched trees pass by my window, I a cat, even when we catch only a partial the top researchers in the field, told me, glanced at other passengers, and I read glimpse of it, or see one in a cartoon. “Let me give you an analogy. Let’s as- my book, all without being aware of the Researchers around the world, in- sume there are ten major religions in incredibly intricate processing taking cluding those at the University of Mary the world. What if, after 2012, every- place in my brain. Photoreceptors in land, spent decades training machines thing became one religion? How would our retinas capture images, turning light to see cats among other things, but, that be?” With computer-vision meth- into electrical signals that travel along until 2010, computer vision, or C.V., ods, he said, “that’s where we are.” the optic nerve. The primary visual cor- still had an error rate of around thirty Computers can now look for abnor- tex, in the occipital lobe, at the rear of per cent, roughly six times higher than malities in a CT scan as effectively as the head, then sends out these signals— a typical person’s. After 9/11, there was the best radiologists. Underwater C.V. which are conveying things like edges, much talk of “smart” CCTV cameras can autonomously monitor fishery pop- colors, and motion. As these pass through that could recognize faces, but the tech- ulations, a task that humans do less re- a series of hierarchical cerebral layers, nology worked only when the images liably and more slowly. Heineken uses the brain reassembles them into objects, were passport-quality; it failed on faces C.V. to inspect eighty thousand bottles which are in turn stitched together into “in the wild”—that is, out in the real an hour produced by its facility in complex scenes. Finally, the visual mem- world. Human-level object recogni- France—an extremely boring quality- ory system in the prefrontal cortex tion was thought to be an untouch- control task previously performed by recognizes them as trees, people, or my able problem, somewhere over the sci- people. And then there is surveillance book. All of this in about two hundred entific horizon. tech, like the YOLO detector I was stand- milliseconds. A revolution was coming, however. ing in front of now. Building machines that can process Within five years, machines could per- No one would mistake me for Brad and recognize images as accurately as a form object recognition with not just Pitt, I thought, scrutinizing my image, human has been, along with teaching human but superhuman performance, but no one would mistake me for a cat, machines to read, speak, and write our thanks to deep learning, the now ubiq- either. To YOLO, however, I was merely language, a holy grail of artificial-intel- uitous approach to A.I., in which algo- a collection of pixels. Goldstein patiently ligence research since the early sixties. rithms that process input data learn led me through YOLO’s visual process. These machines don’t see holistically, through multiple trial-and-error cycles. The system maps the live digital image either—they see in pixels, the minute In deep-learning-based computer vi- of me, measuring the brightness of each grains of light that make up a photo- sion, feature extraction and mapping are pixel. Then the pixels pass through hun- graphic image. At the dawn of A.I., en- done by a neural network, a constella- dreds of layers, known as convolutions, gineers tried to “handcraft” computer tion of artificial neurons. By training a made of artificial neurons, a process that programs to extract the useful informa- neural net with a large database of im- groups neighboring pixels together into tion in the pixels that would signal to ages of objects or faces, the algorithm edges, then edges into shapes, and so on the machine what kind of object it was will learn to correctly recognize objects until eventually you get a person. Nui- looking at. This was often achieved by or faces it then encounters. Only in re- sance variables—the bane of handcrafted
THE NEW YORKER, MARCH 16, 2020 45
TNY—2020_03_16—PAGE 45—133SC. BW_R2 C.V.—are removed along the way, as went to see in Los Angeles. She met ogies market software to the general pub- pixels are distilled into features that en- me in the lobby of my hotel in Venice lic that can turn almost any surveillance code my presence. All this happens in Beach wearing a black dress printed camera into an A.L.P.R. The open-source about the same amount of time that it with license plates. She handed me a version of a similar software is free. takes the brain to recognize an object. black T-shirt, men’s large, also covered A.L.P.R.s automatically record all Finally, a red outline, called a “bound- with license plates. It was a warm win- license-plate numbers that come within ing box,” appeared around me on the ter day in Venice Beach, where Bertash, their view, at a rate of thousands per live screen, with the label “Person.” Boom. thirty-three, lives and works. I stepped minute. In newer systems, “hot lists” of “You’re detected,” Goldstein said. into the rest room and put on my T-shirt. “plates of interest” belonging to crimi- The dummy plates spelled out words nal suspects are widely shared by law-en- dvances in computer vision have oc- from the Fourth Amendment. forcement agencies, including U.S. Im- curred so rapidly that local and na- “Welcome to the resistance,” Ber- migration and Customs Enforcement. tionalA privacy policies—what aspects of tash said, when I emerged. Officers are alerted to a location when your face and body should be protected We set off on a stroll down Abbot a plate shows up on a reader connected by law from surveillance machines—are Kinney Boulevard, the main drag in Ven- to the network they’re using. There are lagging far behind A.I.’s technological ice Beach. The plates on our clothing few privacy restrictions on this data, and capabilities, leaving the public vulnera- were designed to trigger automatic li- it is not secure. Private companies col- ble to a modern panopticon, a total-sur- cense-plate readers. In the U.S., the net- lect and sell it. Police departments ob- veillance society that could be built be- works of A.L.P.R.s and databases that tain data and share it with one another. fore we know enough to stop it. Chris exist across the country make up a differ- According to The Atlantic, Vigilant Solu- Meserole, a foreign-policy fellow at the ent kind of surveillance system. First de- tions, the industry leader, has a data- Brookings Institution who studies Chi- veloped in the U.K., in the late seventies, base of at least two billion unique li- na’s use of face recognition and other sur- A.L.P.R.s began appearing in U.S. cities cense-plate locations. A recent audit of veillance technologies—widely deployed in the early two-thousands. The readers the Los Angeles Police Department as part of Xi Jinping’s “stability mainte- use optical character recognition, which and three other California law-enforce- nance” drive—told me that policymak- captures plate numbers and stores the in- ment agencies found that, at the time ers in the States haven’t, so far, created formation, along with the location, date, they were logged, 99.9 per cent of the governing structures to safeguard citi- and time of the recording. Newer sys- three hundred and twenty million plate zens. And, he added, “in the U.S., the tems can also pinpoint where a car is images in the department’s database government hasn’t thought to use it yet most likely to be found, based on travel had not been involved in criminal in- the way that China has.” patterns. A.L.P.R.s are mounted on street vestigations. State Senator Scott Wie- Some activists think we’ve already lights, highway overpasses, freeway exits, ner, who requested the audit, told the run out of time. Before travelling to toll booths, digital speed-limit signs, and Los Angeles Times, “I am horrified. We Maryland, I had acquired several ready- the tops of police cars. They are also found believed that there were problems with to-wear anti-surveillance items from a in parking garages, schools, and malls. the ALPR program, but I did not an- woman named Kate Bertash, whom I Companies such as PlateSmart Technol- ticipate the scale of the problem—the
“I think we just have time for one more quick question.”
TNY—2020_03_16—PAGE 46—133SC. —LIVE CARTOON—A22689—PLEASE USE VIRTUAL PROOF BW fact that we have so many law enforce- Possibly to compensate for the guilt new and so complex that scientists don’t ment agencies that are not complying I felt about not being down with the re- yet fully understand the kinds of hacks with state law, including LAPD.” sistance, I ended up buying a license- they are vulnerable to. Bertash works for the Digital De- plate backpack, for $49.95. When I got The phenomenon of adversarial im- fense Fund, a nonprofit that provides home, my eleven-year-old daughter, to agery was discovered more or less by ac- security and tech support for the abor- whom I often feel invisible—not in a cident in 2011, by Christian Szegedy, at tion-access movement. In a café where good way—actually noticed me. “What’s Google Research. Szegedy trained a neu- we stopped for lunch, she explained that that?” she said, studying the plates. “That’s ral net to solve the problem of just how protesters often “stand outside of abor- really cool!” Detected at last. much he could change an image of a tion clinics photographing all day long.” ship before the system reclassified the She was concerned that an anti-abor- hen I told my children, both image as an airplane. He discovered that tion activist with access to A.L.P.R. data “Harry Potter” fans, that I was with only a minimal modification of pix- could easily figure out where abortion goingW to check out an invisibility cloak, els the system reclassified it with a high providers and patients live. they were excited. I’d learned of Gold- degree of confidence, even though to the Dave Maass, a senior investigator stein’s cloak in a scientific paper that he human eye it was still obviously a ship with the Electronic Frontier Founda- and his students produced about their and not an airplane. Students at M.I.T. tion, a nonprofit digital-privacy advo- work. But when I saw Goldstein in his printed a three-dimensional model of a cate, confirmed Bertash’s fears about the sweatshirt, which featured a foreground turtle with a textured shell that fooled insecurity of the data. Bertash wondered of blurry organic shapes in orange, like Google’s object-detection algorithm into what, as an activist, she could do. Maass a display of horribly irradiated vegeta- classifying the reptile as a rifle. In a 2018 suggested postering public spaces with bles, with dark, vaguely human shapes paper, “Robust Physical-World Attacks paper images of license plates that would above, I couldn’t imagine Harry or Her- on Deep Learning Visual Classification,” feed false data into the system. Bertash mione wizarding with one. The only researchers described an experiment in had a better idea. recognizable shape (to me) was what which they “perturbed” a stop sign with As a part-time gig, Bertash designs appeared to be a traffic light just below a few small decals that to a human look and sells novelty fabrics for kids—sheets the neckline. Considered more gener- like graffiti but that made an object and towels printed with manatees and ously, the pattern loosely evoked Georges classifier see the octagonal red sign as a kittens, pillows that look like cuts of Seurat’s “A Sunday Afternoon on the rectangular black-and-white sign that meat. She started producing mockups Island of La Grande Jatte,” as it might said “Speed Limit 45.” It isn’t hard to of clothing with phony plates, testing appear at the bottom of a swimming imagine the kind of chaos one of these them with an open-source A.L.P.R. pool painted by David Hockney. perturbances could cause in a future world app that might (or might not) work like Then Goldstein stepped in front of of autonomous cars. those used by law enforcement. Even- the camera, and the YOLO detector did Goldstein’s research is ultimately tually, she got her designs to read into a double take. It couldn’t see him at all. aimed at understanding these vulnera- the system as real plates and produced The computer saw the chair behind him bilities, and making A.I. systems more a line of garments and accessories with (“Chair,” the bounding box was labelled) secure. He explained that he and his dummy plates printed on them, which but not the six-foot-tall, thirty-six-year- student Zuxuan Wu were able to cre- she sells on Adversarialfashion.com. old man standing right in front of it— ate a pattern that confuses the network Bertash’s anti-A.L.P.R. clothes are Goldstein Unbound. I, in my suppos- using the same trial-and-error methods “poison” attacks, which aim to pollute edly anonymous city duds, was instantly employed in training the neural net- databases with garbage, so that the sys- detected and labelled. It was like a con- work itself. “If you just try random pat- tem as a whole is less reliable. Poison ceit from William Gibson’s 2010 science- terns, you will never find an adversarial attacks are predicated on collective ac- fiction novel, “Zero History,” in which example,” he said. “But if you have ac- tion. A few people festooned in license a character wears a T-shirt so ugly that cess to the system you can find a pat- plates won’t make much difference; a CCTV cameras can’t see it. tern to exploit it.” To make the sweat- lot of people wearing them might. For The pattern on the sweatshirt was shirt, they started with a pattern that that to happen, designers need to make an “adversarial image”—a kind of deep- looked like random static. They loaded anti-surveillance clothes that you’d want learning optical illusion that stopped an image of people, covered a small part to put on. T-shirts strewn with fake li- the algorithm from seeing the person of the image with the pattern, and cense plates might not be everyone’s wearing it. Unlike poison attacks, which showed the result to a neural network. must-have look for spring. seek to subvert surveillance systems An algorithm was used to update the We spent several hours strolling with bad data, adversarial attacks are pattern to make the neural net less confi- around. No one asked about our clothes: images that have been engineered to dent that it was seeing people. This pro- in Venice Beach, it takes a lot more than take advantage of flaws in the way com- cess was repeated using hundreds of a license-plate outfit to stand out as un- puters see. They are like hacks, but for thousands of images, until the static usual. When a police cruiser with an artificial intelligence. The security vul- slowly morphed and the neural net could A.L.P.R. on top passed us on the side- nerabilities of operating systems and no longer see people when the result- walk, I tried to feel adversarial. Hon- computer networks are widely known, ing pattern was present in an image. estly, I felt kind of sheepish. but deep-learning A.I. systems are still “I couldn’t tell you why this pattern
THE NEW YORKER, MARCH 16, 2020 47
TNY—2020_03_16—PAGE 47—133SC BW works,” Goldstein said. Researchers can’t two-dimensional computer-generated not see a face that was above one on understand exactly how the machine images in a simulation. Making a three- the sweatshirt. sees. “These are very complicated sys- dimensional adversarial object that could As long as I stood still, I was an ad- tems,” he said. “They have weaknesses work in the real world is a lot harder, be- versarial man. But the luxury of invisi- that occur in the interactions between cause shadows and partial views defeat bility was fleeting: as soon as I moved, feature maps and artificial neurons. the attack by introducing nuisance vari- I was detected again. Goldstein’s gear There are strange and exploitable path- ables into the input image. A Belgian works as a proof of concept, but it has ways in these neural networks that prob- team of researchers printed adversarial a long way to go in the wild. ably shouldn’t be there.” images on two-dimensional boards, Adversarial examples demonstrate which made them invisible to YOLO when ike object detection, face recognition that deep-learning-based C.V. systems they held the boards in front of them. improved dramatically in the twenty- are only as good as their training data, Scientists at Northeastern University and Ltens with the switch to deep learning. and, because the data sets don’t contain at the M.I.T.-I.B.M. Watson A.I. Lab Early handcrafted features for faces in- all possible images, we can’t really trust created an adversarial design that they volved mathematical formulas that ex- them. In spite of the gains in accuracy printed on a T-shirt. Goldstein and his pressed how far apart the pupils of the and performance since the switch to deep students came up with a whole line of eyes are on a face, for example, or the dis- learning, we still don’t understand or con- clothes—hoodies, sweatshirts, T-shirts. tance from the bottom of your nose to trol how C.V. systems make decisions. I put on a sweatshirt, which had shapes the top of your lip. But “there are things “You train a neural network on inputs and colors similar to Goldstein’s, but in about your face I don’t even know how that represent the world a certain way,” a slightly different configuration. On step- to write down mathematically,” Gold- Goldstein said. “And maybe something ping in front of the camera, I was unde- stein told me, “and a neural net will dis- comes along that’s different—a lighting tected, too. I felt strangely weightless. cover and extract this information.” condition the system didn’t expect, or I asked Goldstein to speculate about Deep-learning-based face recogni- clothing it didn’t expect. It’s important why these particular blurry shapes were tion starts with a detector much like that these systems are robust and don’t adversarial. He pointed to the shape YOLO, and can run on top of any CCTV fail catastrophically when they stumble that looked sort of like a traffic light on camera’s feed. First, an image passes on something they aren’t trained on.” his chest. Perhaps, he said, because there through layers of a neural network that The early work on adversarial attacks were no human faces above traffic lights quickly map out the locations of facial was done in the digital realm, using in the training data, the algorithm could features. “Anything with two eyes, a nose, and a mouth is almost always a face at this stage,” Goldstein said. Then each face is isolated and passes through a more refined neural network that re- moves nuisance variables, distilling the face into a short list of unique coördi- nates—your facial fingerprint, or face- print. Many systems also outline the eyes, the eyebrows, the nose, the lips, and the mouth, using sixty-eight stan- dard landmark points to identify emo- tions and gaze. Some sophisticated sys- tems (like Apple’s FaceID for the iPhone) use infrared scanners to make three-dimensional face maps. The re- sults are expressed as numerical data— your unique identifier. Unlike the tips of your fingers or your driver’s license, your face can be scanned remotely, with- out your knowledge or consent, and mined for age, gender, emotion, and, if your labelled picture happens to be in the system’s database, your identity. As with all deep-learning systems, the more data you train the algorithm on, the more accurate the model will become. Early face-detection systems developed for military, border-control, and law-enforcement purposes were “Gross! Can’t I sneeze into somebody else’s elbow?” trained on labelled databases of faces in
TNY—2020_03_16—PAGE 48—133SC. —LIVE CARTOON—A23963—PLEASE USE VIRTUAL PROOF BW the form of passport and driver’s-license Some of the data sets that academ- centrally controlled surveillance system photos, and mug shots—the only large ics used for training early algorithms using face recognition and other tech- collections of faces that existed before skewed white and male—actors, poli- nologies. In addition to making use of the Internet. But these databases were ticians, and the academics themselves. China’s installed base of more than two of little value in trying to match faces Even diverse data sets presented prob- hundred million CCTV cameras (the captured in challenging light conditions lems: poor contrast in photos with U.S. lags behind, with fewer than a hun- and obscured views. Photos posted to darker skin tones, for example, would dred million cameras), the plan, accord- photo-sharing Web sites and social make it more difficult to match faces. ing to Chris Meserole, of the Brookings media, on the other hand, are gold. There are biases built into algorithms, as Institution, involves linking video feeds If the government were to demand Joy Buolamwini and Timnit from smart TVs and mobile pictures of citizens in a variety of poses, Gebru, of M.I.T., showed in a devices in rural areas, where against different backdrops, indoors and 2018 report, “Gender Shades”: CCTV coverage is much outdoors, how many Americans would facial-recognition systems in lighter. The Chinese A.I. readily comply? But we are already build- commercial use performed company SenseTime, which ing databases of ourselves, one selfie at much better on light-skinned last year was valued at more a time. Online images of us, our chil- males than on dark-skinned than seven billion dollars, has dren, and our friends, often helpfully la- females. Women of color are said that the facial-recogni- belled with first names, which we’ve up to thirty-four per cent tion system it is building will posted to photo-sharing sites like Flickr, more likely to be misiden- be able to process feeds from have ended up in data sets used to train tified by the systems than up to a hundred thousand face-recognition systems. In at least two white men, according to their CCTV cameras in real time. cases, face-recognition companies have research. Newer collections of faces for Are China’s surveillance-state ambi- strong connections to photo-manage- training, like I.B.M.’s Diversity in Faces tions technically feasible? Meserole is ment apps. EverRoll, a photo-manage- data set, aim to overcome these biases. skeptical. However, he added, “whether ment app, became Ever AI (now Para- However, I.B.M.’s effort also proved they are able to do it in a totally unified vision), and Orbeus, a face-recognition to be problematic—the company faced way or not is in some ways irrelevant. A company that was acquired by Amazon, backlash from people who found their huge part of how Chinese authoritari- once offered a consumer photo app. And images in the data set. In face recogni- anism works is the uncertainty about even when our images are supposedly tion, there is a trade-off between bias whether you are being watched. The protected on social-media sites like Face- and privacy. technology is incredibly precise, but the book, Instagram, and YouTube, how se- Apart from biases in the training da- way the laws are applied is incredibly ar- cure are they? tabases, it’s hard to know how well bitrary. You are uncertain if you are being In January, the Times reported that face-recognition systems actually per- watched, and you are uncertain about Clearview, a Manhattan-based startup form in the real world, in spite of recent what’s permissible, and that puts the onus backed by the investor Peter Thiel and gains. Anil Jain, a professor of computer on you as the individual to be really con- co-founded by Richard Schwartz, a for- science at Michigan State University servative about what you’re doing.” mer mayoral aide to Rudolph Giuliani, who has worked on face recognition for had assembled a database of more than more than thirty years, told me, “Most considered adding face camouflage three billion images scraped from social- of the testing on the private venders’ to my adversarial look, and met with media sites, and that Clearview’s tech- products is done in a laboratory environ- IAdam Harvey, an American artist based nology was being used by more than ment under controlled settings. In real in Berlin, who made a name for him- six hundred law-enforcement agencies practice, you’re walking around in the self in the early twenty-tens by creat- to match faces of suspects or persons streets of New York. It’s a cold winter ing a series of asymmetric getups that of interest with faces in Clearview’s da- day, you have a scarf around your face, a could defeat the Viola-Jones algorithm, tabase. Google, Twitter, Venmo, and cap, maybe your coat is pulled up so your which until 2015 was the most widely other companies have sent cease-and- chin is partially hidden, the illumination used object- and face-detection plat- desist letters to Clearview. Its co-founder may not be the most favorable, and the form. Face-detection algorithms are and C.E.O., Hoan Ton-That, an Aus- camera isn’t capturing a frontal view.” trained to expect symmetry in faces. tralian entrepreneur in his early thir- The technology’s questionable per- When people put on makeup, they are ties, claims that the company has a First formance doesn’t seem to be impeding unwittingly helping the systems by ac- Amendment right to these images. In its ongoing implementation. Though centing some of the landmarks that any case, as Clare Garvie, a senior as- face recognition is only one application scanners use to read your faceprint. To sociate at Georgetown Law’s Center of computer vision, it poses a unique fly under the radar, you must deface on Privacy & Technology, told me, the threat to civil liberties. The E.U. has yourself. Harvey’s work showed faces Clearview database “gives the lie to” the tried to make privacy policies to contain with makeup applied asymmetrically, notion that social-media “privacy pol- it, as have a few states, including Illinois. in a way unlikely to be represented in icies are a safeguard against data col- In China, by contrast, the state has em- the systems’ training data, therefore mak- lection.” (Clearview’s entire client list braced the technology. A 2015 proposal ing them harder for machines to detect was stolen by hackers last month.) laid out the country’s plans for a vast as faces. Fashion-forward types can
THE NEW YORKER, MARCH 16, 2020 49
TNY—2020_03_16—PAGE 49—133SC—LIVE SPOT—R35968A—PLEASE USE VIRTUAL PROOF. BW download symmetry-distorting looks hard to train an algorithm on, because titled “How Facial Recognition Makes from Harvey’s Web site, though he said it’s person-specific,” Chellappa said. You Safer,” James O’Neill, a former that they probably won’t work with In 2018, Chellappa and other A.I. re- commissioner of the N.Y.P.D., wrote newer algorithms. searchers, based in India, created the that in New York City “no one can be Harvey, thirty-eight, is slim, pale, and Disguised Face in the Wild competi- arrested on the basis of the computer quietly intense. We met in a café in Wil- tion. “With the advances of deep-learn- match alone,” and that human inves liamsburg, Brooklyn, where he glanced ing algorithms, we wanted to evaluate tigators would need to confirm any several times at a CCTV camera mounted whether the deep-learning methods matches that machines suggest. high up in a corner of the room. He said, were robust to disguises,” Chellappa told Where the U.S. leads the world is in “We don’t really understand what we’re me. He and his colleagues put together the commercial use of face recognition doing when we go outside. We can know a database of thousands of faces, taken by private companies. Many major tech the weather, and we dress for it, but if I both from movies, like Dana Carvey’s companies have deep-learning face-rec- had known on the way over here that I 2002 film, “Master of Disguise,” and ognition systems and training databases. was going to pass four private surveil- from ordinary people’s photos of Hal- Facebook’s product, DeepFace, can iden- lance cameras outside houses, or that loween and other dress-up events that tify faces in photographs and tag them. there was”—he broke off and glanced had been posted on social media. Google has FaceNet, as well as an ob- at the CCTV camera—“would I have Teams from around the world were ject detector, Cloud Vision. Amazon dressed for it?” invited to test their face-recognition al- markets Rekognition, a C.V. platform He went on, “We exist in this world gorithms by matching disguised faces that has been deployed by police de- where we are observed by machines. with their undisguised counterparts. partments and was pitched to ICE for How can you mediate that to appear The competition was supported by use in border enforcement. Apple makes one way to the machines and another iARPA, a research organization within infrared scans of the faces of users who way to people? How can you ride the the Office of the Director of National opt into its FaceID password system; fine line between appearing avant-garde Intelligence, which gave a twenty-five- the encrypted data isn’t supposed to and appearing invisible?” thousand-dollar cash prize to the win- leave the user’s phone. Harvey explained that he had moved ning team. In return, iARPA’s face-rec- In addition to Big Face, there is a on from face camouflage because, the- ognition capabilities had the chance to rapidly growing field of startups, part of oretically, any makeup design that can benefit from the training data the com- a market that is expected to be worth be used to foil a detection system could petition generated, making them that nine billion dollars a year by 2022, ac- be incorporated into the system’s train- much more robust against real-life mas- cording to some estimates. The prod- ing data. “I realized that, whatever I ters of disguise. ucts include face recognition for stores, post on my Web site, people are going I asked the professor to summarize which can identify repeat shoplifters and to use it to download and test their the research. “What can I wear if I re- troublemakers as soon as they step onto algorithm on.” This is the paradox of ally don’t want to be seen?” the premises. In a casino, as Richard the adversarial man: any attempt to “You can wear a beard, you can shave Smith, the sales director of SAFR, a di- evade the system may only make it your head, and that will affect face-rec- vision of RealNetworks, explained to stronger, because the machine ognition algorithms in differ- me, a system can spot unwanted patrons just keeps learning. And, with ent ways,” Chellappa said, add- and problem gamblers who are on the deep learning, it keeps learn- ing, “I really can’t tell you if casino’s watch list, as well as high roll- ing faster. you do x, y, and z it will mess ers whom management wants to court. Is there any science on what up the face recognition. All I “Before face recognition, the guards had kinds of disguise thwart face can say is, if you do a combi- to remember those people,” Smith said. recognition? That question has nation of hat, wig, dark glasses, Some schools have installed similar se- long fascinated Rama Chel- you can assume the accuracy curity systems; college campuses have lappa, Goldstein’s colleague at will go down.” For now, at least. also begun contemplating their imple- the University of Maryland. The top-performing algo- mentation. Taylor Swift has reportedly “I’m interested in this because rithms—the hardest to fool— used face recognition to detect the pres- the spymasters do it in real life,” were designed by the Rus- ence of stalkers at her shows. Chellappa told me. “But there was no sian and Taiwanese entrants. The 2019 Face recognition also offers “smart really scientific evaluation of what works.” challenge was sponsored by Facebook retail” applications, allowing companies Disguises present the same problem and Apple. to harvest demographic information to recognition algorithms as aging does; from customers’ faces, such as age and aging is a kind of natural disguise, he o far in the U.S., the deployment of gender, and also to track and measure said. Some well-known faces become un- face recognition by public agencies “dwell time”—how long a customer recognizable as they age (Anthony Mi- Sand law enforcement is less advanced spends in any particular section of the chael Hall looks nothing like the young than it is in China. Last May, San Fran- store. “What if you could see what your actor in those Brat Pack movies), whereas cisco banned city agencies from using ad sees?” SAFR asks on the company’s others (like Paul Rudd or Halle Berry, facial-recognition technologies. In an Web site. A video shows a couple hav- say) don’t appear to age at all. “Aging is Op-Ed published in the Times in June, ing a conversation while data appears
50 THE NEW YORKER, MARCH 16, 2020
TNY—2020_03_16—PAGE 50—133SC—LIVE SPOT—R35968B—PLEASE USE VIRTUAL PROOF. BW on the screen, assigning attention and “sentiment scores” to their faces. Other companies offer face surveillance that alerts stores to shoppers’ previous shop- ping habits, or their V.I.P. status, when they walk in. Face Six, a biometrics com- pany based in Israel and Nevada, mar- kets Churchix, software that is often used to track congregants’ attendance at church. As Clare Garvie, of Georgetown Law’s privacy center, put it, “Think of a possible application for this technology and chances are good it’s being sold.” For marketers, face recognition has the potential to be the ultimate form of targeted advertising. The consumer’s face could serve as a kind of license plate that connects the digital world—where your search histories live—to the loca- tion data that Google Maps collects from your phone, to the emotions on your face. We’re already traced online, and are served ads based on recent searches. Face recognition could follow us around in the real world, alerting the owners and managers of public spaces, such as subway stations and parks, or private spaces, like bars, shops, and sta- diums, to our presence. I can opt out of accepting cookies, and disable location settings on my phone, but a face-rec- ognition system doesn’t give me any way “Back to work, boys. Those mysteries of the Trinity to opt out, short of defacing myself. aren’t going to grapple with themselves.”
ne day last month, I put on a hoodie that I had selected from Goldstein’s •• Oline of YOLO-busting fashion, hoisted my A.L.P.R.-poisoning backpack (I had Harvey’s asymmetrical makeup, Urban’s network before you reach your stop. my T-shirt on, too, for good measure), spectacles allow you to remain relatively That dystopia could be an app away. As grabbed my Faraday pouch, and set off inconspicuous. “Sure, you can paint your- Urban put it, “People are concerned from my home in Brooklyn for down- self up and look like some cyberpunk- about governments and corporations— town Manhattan and The New Yorker’s type character,” he observed. “But you’re well, soon it’s going to be people doing offices, in One World Trade Center. not going to wear that to work.” all this tracking.” Big Brother is us. On my face I wore a pair of Reflec- Finding a seat on the C, I was alert My plan was to loiter in the lobby tacles—sunglasses made by Scott Urban, to any stares my adversarial hoodie might of One World Trade Center, where one a custom eyewear-maker in Chicago, that attract from people across the aisle. As can assume that video surveillance is in block attempts via infrared light to scan I searched their faces, I wondered how place—an adversarial man at work. your face. They come in three models— long it will be before face-recognition Maybe, if I stood perfectly still, the al- IRPair, Phantom, and Ghost, my style. technology, with the power of deep gorithm wouldn’t see me. I recalled a The lenses contain “infrared absorbents,” learning behind it, arrives on everyone’s comment made by Anil Jain, of Mich- Urban told me. “This means that on your phone. You will be able to snap a pic- igan State, when I asked him what I average security camera using infrared ture of someone across the subway aisle needed to wear to beat detection algo- for illumination these lenses turn dark and run the face through a reverse-im- rithms. “You can put tinfoil all over your- black, whereas regular sunglasses become age search, such as that offered by So- self and that will do the job nicely,” he completely clear.” The Ghost model also cialcatfish.com, an “online dating inves- said. “It all depends how much of a spec- has frames that reflect both infrared and tigation” site based in California, which tacle of yourself you want to make.” visible light, which can make your face promises to ascertain if one’s Tinder Before I could go undetected, secu- less readable in photos taken with a phone, hookups are who they claim to be. You rity spotted me. “You need help?” a guard especially with a flash. And, unlike Adam could potentially get a name or a social asked. Good question.
THE NEW YORKER, MARCH 16, 2020 51
TNY—2020_03_16—PAGE 51—133SC. —LIVE CARTOON—A20991—PLEASE USE VIRTUAL PROOF BW