NISTIR 8280

Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects

Patrick Grother Mei Ngan Kayee Hanaoka

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 NISTIR 8280

Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects

Patrick Grother Mei Ngan Kayee Hanaoka Information Access Division Information Technology Laboratory

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280

December 2019

U.S. Department of Commerce Wilbur L. Ross, Jr., Secretary

National Institute of Standards and Technology Walter Copan, NIST Director and Undersecretary of Commerce for Standards and Technology

Certain commercial entities, equipment, or materials may be identified in this document in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

National Institute of Standards and Technology Interagency or Internal Report 8280 Natl. Inst. Stand. Technol. Interag. Intern. Rep. 8280, 81 pages (December 2019)

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280

This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 HTW DID WE WHAT SCOPE AND AIMS MOTIVATION OVERVIEW SUMMARY EXECUTIVE E T XEC ECH S . S . UMMARY UMMARY oe st rcs oa f1.7mliniae f84 ilo epetruh19mostly 189 through people million 8.49 of developers. images 99 million from 18.27 algorithms of commercial al- total set datasets a these last process Together The to environment. us standards. and lowed duration capture capture image on with constraints pro- compliance given enforcement good not, law have does or sets three immigration first travel, The authorized for cesses. collected were datasets four All . . . govern- . U.S. in operation: collected in photographs currently of are datasets that large applications mental four with algorithms 17]. [16, these reports used FRVT We in as detailed available is necessarily performance not were Their algorithms products. these integrable laborato- prototypes, mature development As and universities. research few corporate a by and FRVT ries the to algo- submitted search were identification These one-to-many birth. rithms. and of algorithms country verification or one-to-one race both and used age, We sex, recogni- by face defined of groups demographic accuracy for the algorithms quantified tion (ITL) Laboratory Technology Information NIST The differentials. demographic by implied risks of recognition mitigation makers, face with policy of concerned managers includes systems and audience integrators, and systems intended utility, developers, accuracy, Its algorithm the recognition face about technologies. decisions recognition and face discussion of inform limitations to report this deficiencies. gives intends performance NIST of analyses, mitigation and the into metrics where research notes performance recommends and process, specific results, recognition details empirical the occur, about could details recog- effects face provides demographic contemporary report in This differences demographic algorithms. quantify nition to tests conducted has NIST gender and [9,1623] recognition face moti- 36]. [5, recent is more algorithms work in estimation effects this demographic Additionally, of policing studies implications. a by stud- regulatory in vated prior and particularly that policy impacts potential discussed noted the and work described context, [14] bias, University of sources Georgetown articulated from [22], variations report ies accuracy ac- A to been lead bias. has could potential recognition dependencies face and demographic of that use assertions and by capability, availability, companied the in expansion recent The groups. eval- those demographic extends across document variations This accuracy document bases. data to photo uations algorithms in recognition individuals face of one-to-many identification of ver- for performance used for reports ex- and used two (FRVT) identities, algorithms asserted first tests recognition of The face vendor ification (NIST). one-to-one recognition Technology of face and performance ongoing Standards the respectively, on of cover, reports Institute of National the series by a ecuted in third the is This odrcosn photographs crossing Border photographs Visa photographs Application mugshots Domestic as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE umte nspoto iaapplicants. visa of support in submitted olce nteUie States. United the in collected rmagoa ouaino plcnsfrimgainbenefits. immigration for applicants of population global a from ftaeesetrn h ntdStates. United the entering travelers of - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 1 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 FOUND WE WHAT E T XEC ECH S . S . UMMARY UMMARY edt emr loih-pcfi,advr fe yfcosblw3. below . factors negatives by False often times. vary 100 and beyond algorithm-specific, to more 10 be of to factors demo- tend by Across vary tested. often algorithms rates related all, positives those not false but than graphics, many, larger across broadly, much exist are and differentials negatives positive false false to that is result main mag- Our various of nitudes. differentials demographic exhibit algorithms the recognition face with be Contemporary therefore developers, can across differentials. algorithms accuracy demographic These smaller in have eval- errors. range to FRVT fewer expected wide recent many a producing in algorithms documented show accurate been These most has report 17]. [16, this reports in uation used algorithms of accuracy The application. the differentials on demographic depending we of disadvantageous and impacts or used, the advantageous are that be noting algorithms can errors how of of consequences examples the give on we elaborate similar. follows are that people material two of background faces In samples digitized of ap- the association person’s when erroneous occur the the they are in persons; positives change two False of some when properties. either occur image they reflecting the low, in images; ef- or two is positive pearance in photos person false two one and between associate negative similarity to failure the false the on are report negatives as- False and We group fects. demographics. demographic dif- various by algorithms from recognition accuracy individuals face of sessed degree, photographs what processed to they and when whether, fered determine to aimed particularly tests right The own its in recognition. factor face meaningful of a applications as travel-related stands lower for it countries, seen these have reasonable in a that race be regions may for global proxy information distinct country-of-birth 7 While in immigration. long-distance countries informa- of 24 country-of-birth levels to have analysis only sets the other restrict individuals. We the photographed but tion. the race, for for metadata metadata have age mugshots and The sex by accompanied were datasets The efudeeae as oiie nteedryadi hlrn h fet eelre in larger were adults. effects middle-aged the in children; smallest in and and youngest, across elderly and consistent oldest the the is in positives race. this to false and due elevated men, that found than We than smaller women is effect in This higher datasets. be and algorithms to with positives varies false and found sex We on depends ordering African relative in rates the algorithm. elevated populations; enforcement re- with Indians, law Asian is American domestic and effect With in American are this faces. positives Asian China false East highest in on the developed rates between images, algorithms positive positives false false of in- low more number European with 100 a Eastern versed, of in factor with a lowest However, with and large, people, countries. generally Asian is East effect and This African dividuals. East and West in est positives: False as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE sn h ihrqaiyApiainpoo,flepstv ae r high- are rates positive false photos, Application quality higher the Using - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 2 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 TESTS THESE OF IMPLICATIONS E T XEC ECH S . S . UMMARY UMMARY ihflengtvs emk ugsin o umnigrprigwt epc odemo- to respect only with concerned reporting effects. are augmenting and reports for difference each most suggestions graphic for - make We rates done rarely positive negatives. is false a false This and with with negative configured threshold. false are that both systems at report group most to demographic As necessary be enroll. is interest to it of failure threshold, quantity or fixed the positives stating false me- without negatives, in discussed false and is it papers accuracy academic particular, in In incomplete coverage. been dia has often 9. effects and demographic 8 sections of see Reporting - value, techniques the mitigation evaluating potential and of demographics. developing benefits to and in relate costs, research that for those recommendations specifically includes and report recogni- This face overall of issues limitations performance performance mitigate systems might tion that developers. techniques algorithm of the variety inform a to are and There effects, are the These in of algorithm. breadth contained each the charts for show results of to of pages intended reporting future 1200 exhaustive improve than include to that more and annexes with decisions seventeen report make users this to end them supplement We use and in and performance. differences developers, individuals these system of of recognition aware images be face processing should makers, in policy worse or demographics, better various perform assist. algorithms to different laboratory it testing Since on owners, biometrics algorithm a inform operational employing can the perhaps algo- of data, elsewhere image accuracy their and operational measure know NIST the specifically to from to informative data owner be test usually system will available Given the publicly upon While algorithm. incumbent recognition rithm. is face it single variation, a algorithm-specific employ usually implementations Operational Summary. Technical the in introduced are results detailed More impor- for undetectable. algorithms are One identification differentials accurate positive algorithms. highly false search supplied which developers one-to-many some in that algo- present is verification exception always, tant one-to-one not in but present usually, do differentials are systems rithms demographic verification Nevertheless, one-to-one purely re- that have. a effects not algorithms demographic identification of one-to-many mitigation affords for database source enrollment an of presence The . ncoeaieacs oto plctos as eaie a ermde yuesmaking users by remedied attempts. be second can negatives false applications, face control access from cooperative In deviate images crossing to border standardized standards. the quality specifically image races; setup across photographic qual- images a image high-quality with to Caribbean, produce collected relate the results were differing and mugshots These Africa The individuals. in ity: older born in people stronger in being effect higher the generally crossing are border faces lower-quality negatives American with false However, African and rates). images, negative white false in lowest those the above yield rates (which error with individuals, Indian can negatives: False as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ihdmsi usos as eaie r ihri sa n Ameri- and Asian in higher are negatives false mugshots, domestic With - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 3 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 VERIFICATION RECOGNITION ESTIMATION TION CLASSIFICA ANALYSIS FACE BACKGROUND , E T XEC ECH S . S . : UMMARY UMMARY - , : ALGORITHMS te ad rsn euiycnent h ytmonr ste lo cest imposters. to a access allow crossing they or as owner, facility the system a on the positives, to entering False concern delayed security attempt. a are second present a they hand, with other or remediated individual phone be an usually their can example, into errors These For get border. to user. able verification the be In for not application. inconvenience may the cause upon negatives depending false used, applications, being is photograph the whose answer dividual to errors: access in of seeking one Impact person involved: the not?” are of or person photos taken same Two one the this with drug. ”Is compared non-repudiation prescription question: supports is a also that of It database dispensing the the point. authorize check to security a e.g. through access physical or phone Applications: comparisons imposter genuine negatives. some some false practice, and giving positives, In threshold below false high. scores giving yield be threshold numeric should a scores above Ideally, mate are and score. scores “nonmate” low or be imposter should an scores yields nonmate people different from images of comparison Errors: recognition. face for the used on identity that estimation, from persistent from different and by operations Classification single-shot mediated are “template”). is (or hand, recognition vector other Thus, feature a not. in or stored whether information the person decide similar same to value how the threshold expressing represent, a value to or numeric compared is vendor-defined It a are. is faces This parent and score. vectors similarity feature before. two a seen extractors compare emit never Algorithms generic usually operator: as differential have a act they on as identity persons they proceeds trained Recognition the of been operations, photos to has In from relate that information network developer. that identity-related neural the values of a to of of available vector consists images a typically ID-labeled into extractor that The person algorithm a person. extraction of the feature particular of a images identify by more to followed or built detector not one face are converts a They include person. they particular instead a people; of Face notion sad). built-in happy, no (e.g., knowl- produce have inherent to ever, with aim built they are classes algorithms the of classification edge Face are state. There emotional fatigue). their of or person degree or age are (e.g., that quantity Within classification continuous output. some some output produces that and algorithms term images the face use consumes We that gorithm evaluated. recognition about algorithm information face of include in should class bias effects the of analogous reporting quantify that indict recommend to to strongly cited We undertaken studies widely algorithms. was been Those work have Our results women. the accuracy. black yet their on algorithms, show- recognition algorithms 36] face [5, classification evaluate studies gender not two did cites face years of recent accuracy in bias poor recognition ing face are of that discussion errors the of the impacts. Much detail their is, then and identification recognition and and faces, face verification analyze face what in that describe possible applications we other Summary with Technical it the contrasting in results presenting Before oprsno mgsfo h aepro ilsagnieo mt”soe A score. “mate” or genuine a yields person same the from images of comparison A as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT , loihsta i odtriesm aeoia uniysc stesxo a of sex the as such quantity categorical some determine to aim that algorithms ERRORS n-ooevrfiaini sdi plctosicuiglgclacs oa to access logical including applications in used is verification One-to-one - AERCGIINVNO TEST VENDOR RECOGNITION FACE roshv ifrn mlctosfrtesse we n o h in- the for and owner system the for implications different have Errors , IMPACTS one - apeaoe mlyn ahnr htis that machinery employing alone, sample DEMOGRAPHICS aeanalysis face recognition sa mrlafrayal- any for umbrella an as T  two 0 ape r from, are samples → → loihs how- algorithms, NR FNIR FNMR, M,FPIR FMR, estimation → → 0 1 4 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 IDENT IFICATION E T - XEC ECH S . S . UMMARY UMMARY neato fmcieadhmni eodtesoeo hsrpr,a shmnefficacy. human is as report, this of The scope the [7]. beyond review is under human photographs and the machine without of interaction of those even an interaction capability, with particularly demographics human differentials, performance varied reviewer’s race the and and between 42 ] [10, sex and poor [34], both constraints for response, time is human the there is then evidence matters the What and regardless. immaterial candidates are 50 algorithm say the returns from machine differentials the positive - false cases, such such In In available. hybrid be a must the candidates. of part in of are not way number someone this system of in fixed used human search a Algorithms any produce candidates. because yield 100% will still is will searches rate database all identification positive that false so the zero cases, to set is old goals security whose Investigation: owner their system as system, the a to such undermined. disadvantage in be a enrollee will and an deporta- undetected, to or advantage go detention an would accusation, be fraud false would a negatives to false lead Higher false a could tion. surveillance, individual or another detection, to fraud passport match or positive visa as such applications the identification threshold.In when may chosen or outcomes a positives with, such above identity False produce score wrong comparison algorithms a the own. Identification yields produces search their identity. enrolled is correct than who the different someone of, name of instead a search applying under a be when might license who occur person driver’s also a or for a check visa This to of review. employed a human search often for for is identity a search candidate many” when a to yields “one occur database of the type primarily in positives present not false is who searches, subject one-to-many In application. Impacts: casino. checking a example, system from for banned the - previously where enrolled gamblers not identification of are the negative databases individuals on so-called searched solely that for building implicitly used claims a be operator to can access matches they given Second, that is photograph subject threshold. a a presenta- presentation example, without but For of verification basis claim. one-to-one identity in like an access of positive tion facilitate to used be can applications Identification implement algorithms therefore are Other and comparisons [17]. similar. N economical all most the highly of of the many search yield avoid exhaustive that to an [2,192126] candi- operation techniques execute A “fast-search” algorithms sort threshold. Some a preset and score. a enrollments similarity above a N are and that index those an only is or date candidates, num- similar fixed most feature a the either all return of with algorithms ber The image images. “probe” “gallery” search from enrolled a previously from vectors extracted features compare notionally rithms, algorithms Identification as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False swt eicto,teipc fadmgahcdfeeta ildpn nthe on depend will differential demographic a of impact the verification, with As FRVT h loih fesu addtsfrhmnajdcto,frwihlabor which for adjudication, human for candidates up offers algorithm The : hsi pca-aeapiaino dnicto loihsweetethresh- the where algorithms identification of application special-case a is This - AERCGIINVNO TEST VENDOR RECOGNITION FACE eerdt omnya n-omn r“-oN erhalgo- search “1-to-N” or one-to-many as commonly to referred , hr r w ra sso dnicto loihs is,they First, algorithms. identification of uses broad two are There : - DEMOGRAPHICS any noldiett ihasoeabove score a with identity enrolled T  0 → → NR FNIR FNMR, M,FPIR FMR, machine- → → 0 1 5 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: false both and report terminology to necessary to is introduction it an that by namely rates. preceded effects, error is demographic positive reporting false This and in negative aspect study. vital the a of of results discussion the summarizes section This 08:14:00 2019/12/19 OVERVIEW RESULTS OPERATION THRESHOLD FIXED DIFFERENTIALS ACCURACY SUMMARY TECHNICAL E T XEC ECH S . S . UMMARY UMMARY ns oeacrt loihspouefwrerr,adwl eepce hrfr ohave to therefore expected be will differentials. and demographic developers, errors, smaller fewer algorithm produce across vari- algorithms lower-performing accuracy accurate than eval- More errors in FRVT fewer ants. range recent many producing wide in algorithms a accurate documented most show been the with These has report 17]. [16, this reports in uation used algorithms of accuracy mitigation 8. The toward and Research 9 sections algorithm-specific. in usually more algo- discussed factors all, be is differentials not by to of but tend vary many, across negatives effects broadly, false exist negative differentials The one False positive by rithms. false vary The 100x). often 3. 10x, rates than (i.e., positive less much False magnitude of negatives. orders false two to related or differentials those positive than false larger The much evaluated. of are we majority that the algorithms in differentials recognition demographic face of contemporary existence the for evidence empirical found We rate error germane some only in the rate. example, and negative For false access the attempt important. is never is almost case imposters each legitimate cases, flag in control would imposters access positive of false probability a prior application, and detection The problem, deportee security visitors. one-to-many a present a would in negative system hand, false a other a the undermine On positives false goals. access one-to-one security users; a owners legitimate in inconvenience example, For negatives communities. false different each control, to of importance consequences of the are because error separately of rates type negative includes false report and positive This false report We rates. thresholds. negative operating typical false about misstating differentials demographic and hiding of thereby rates documentation thresholds, reporting fixed positive at by false than [13]) rather in e.g., rates excursions positive demographics false in fixed (even at point rates negative this demograph- false ignore particularly, sys- studies or, the biometric academic conditions (i.e., Most environmental of made are cameras, ics). majority comparisons to vast all tailored which the not against is that threshold threshold fixed is a differentials with configured about are reasoning tems in point crucial A development. under now [6] report to technical is is ISO report differential an this demographic from of term inherited subject The The demographics. outcome. between an positive outcomes and false differential outcome, a quantify negative threshold, false above a or yields at score threshold imposter below score genuine A threshold. defined ripse soe itiuin” uhdfeetasaeicneuniluls hyprompt they unless inconsequential are a differentials Technology Such and distributions”. [score] Science imposter Security or define Homeland and of [20] women). Department article Asian Directorate a young (say (say from B A terminology demographic demographic from adopt than from We higher images be of may they collection men) a Asian over elderly computed are scores similarity When ifrniloutcome differential as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE notoeocr hnasoei oprdwt noperator- an with compared is score a when occurs outcome An . ifrnilperformance differential - DEMOGRAPHICS sa“ifrnei h genuine the in “difference a as T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 6 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 NEGATIVES FALSE POSITIVES FALSE E T XEC ECH S . S . UMMARY UMMARY . . . . . by remedied be . often can negatives below. false observations applications, the attempts. second cooperative make making real-time we differentials in demographic that negative Note false to regard With . . . observations. following the Algorithms: Verification 18. Figure See in individuals. older born in individuals stronger in being higher effect the generally Caribbean, are the negatives and false Africa images, crossing border smaller the are In differentials the but False 21. women, Figure photos. See of application recognition consistent. not in the and higher comparing often when are than rates higher negative are im- rates crossing negative border false lower-quality with ages, images application high-quality 22. comparing Figure When See differentials. and rates better negative that false implies measure- reduces This and quality difficult. image low is demographics very across differentials are negative rates false of error ment photos, application supported. high-quality not comparing are statements age When universal and so sex this, across partic- rates to individuals, negative exceptions false younger many algorithms are in to There and pertaining images. women mugshot in the higher in often ularly are 17. rates Figure error See differ- negative forward. to going False due analysis arise that could on face on report it influential will - We highly race is negatives. rates ageing to false because negative related recognition photographs false be between elapsed lowest not time The might the result in faces. ences This black and faces. white black in individu- in Indian those occur American above and rates Asian error in with higher als, are negatives false mugshots, domestic In of Fig- See use differentials. . result: 12 negative Annex important false demo- and of an 22 average magnitude ure is the with This increase low will still. usually algorithms inaccurate smaller are necessarily, For rates 10%. being, negative above differentials false to graphic 0.5% algorithms, below accurate from algorithm, more by the strongly vary rates error negative False ,adi oetal motn otereduction the to important origin. potentially national is and and race to [33], due 2011 differentials used in they demographic data noted of the was of - demographics matters observation race Asian - the This for training East proxy faces. in a on Caucasian as rates developer with the positive those of location than false the lower that low are give these China sometimes in and positive faces, developed false algorithms lowest of The number 5. A Figure people. See American but individuals. East European elevated Central and East also and with African are occur Asian rates East generally South rates positive and in False West so in next). less highest noted slightly exceptions are with rates (but positive people false Asian the race, for men, to present than is respect increase women With This in 6. higher age. Figure See times and datasets. origin 5 and of and algorithms most country 2 algorithm, between with be varying to multiple positives false found We as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ihrgr oflepstv eorpi ifrnil emake we differentials demographic positive false to regard With - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 7 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 RO WORK PRIOR E T XEC ECH S . S . UMMARY UMMARY . . . . with comport algorithms. results races. identification whose between algorithms differentials for demographic verification differentials regarding of ours demographic tests prior describe recent to however, are, first There the is report en- This of size the also of algorithms 27. independent Figure These approximately See are database. Networks. that rollment rates Real stable identification and with positive Tevian measures, algorithms false Toshiba, many producing give Aware, on developers are is Other rates NEC-3, positive evaluated. publicly algorithm, false have who further we A Idemia, accurate is for most [15]. those algorithms the achieved Among was identification this supplied undetectable. how are developers described differentials some positive that false is which exception important See One algorithms. search one-to-many in present verifica- always, 7. one-to-one not Section in but present usually, differentials are demographic algorithms that note tion We verification one-to-one have. purely not that do effects systems demographic of mitigation for resource a gorithms Algorithms: Identification . . et mlydsalrst fmghtadvs htgah hnaeue here. used are than photographs visa and mugshot Those of process. sets FRVT smaller demo- the employed on to tests submitted [16] algorithms reported verification has one-to-one NIST in 2017, effects in graphic starting schedule monthly approximately an than On higher magnitude of orders two imposters are randomly. that of individuals rates pairing pairing positive by false that gives showed race algorithm and / Technology commercial sex age, and leading by Science a (DHS) using Security [20] Homeland study of SAIC Department recent a addition, to In effects. [ 32] demographic of images artic- measurement paper challenge to The related GBU Caucasians. lessons to vs. five Asians ulates algorithms in positives verification false four higher order-of-magnitude applied show [8] al. et Cavazos Caucasians. vs. and Americans positives African in false of negatives elevated Institute false simultaneously reduced Florida [23] the showed mugshots, collaborators its domestic and to Technology applied algorithms verification four Using sex on 13. depends Figure ordering and 12 relative Figure the See elevated algorithm. populations; with with Asian varies Indians, and and American American in are African positives in false rates highest the images, mugshot With effects See The magnitude. 15. in Figure adults. vary and aged but 14 algorithms middle Figure in and in larger datasets were smallest country-of-birth, effects across and the consistent children, are children; youngest in and and adults elderly oldest the the in positives false elevated found We as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE h rsneo nerlmn aaaeafrsoet-ayal- one-to-many affords database enrollment an of presence The - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 8 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 O DO NOT DID WE WHAT E T XEC ECH S . S . UMMARY UMMARY . following: the address . not does It research deficiencies. additional observed discusses of and mitigation impacts, support and can results that gives context, establishes report This . . yrtann h loihso mg esmitie tNS.W ipyrntetests the effect: and ran cause simply Analyze We NIST. at maintained submitted. sets differentials as demographic image algorithms operational of using on We usual mitigation algorithms data. attempt, the the local to reflects retraining customers developers This by on invite adapted or not adapted. attempt, are not or systems did refined recognition face not which were in and situation fixed are NIST to algorithms of Training ac.Ti eotde o atr eorpi ifrnil htmyocri uhpho- such in occur may that differentials demographic tographs. capture not does report This lance. images: wild them. promulgating Use in then proactive and being standards, others capture and portrait FBI establishing the in by 1990s example, supported the For is portraits. application high-quality justice criminal are main applications the their in so images standards our “real-world” advancing are the to that than resources dedicated speed organizations and governmental quality some Finally, between space datasets. trade portrait high-quality the is mostly which in three dataset other point crossing different border a a at on negatives collected false quantify we effects Additionally, demographic positives. note we fact, In to has translate it “don’t on lines, images report these standards-compliant scenarios”. using Along everyday not tests enroll. NISTs do to that we failure [41] and suggested data, been use, interaction of difficulty human-camera throughput satisfaction, high to like in access quantities example, Without for intended, control. not standards- or when access possible, important particularly not is is This photography compliant [9]. recently detailed been has interaction, cameras: of effect the the to unavailable Consider be will quantities germane of- that are inevitable Models is analysis. it etc.). though uniformity, even blur, useful, etc.) brightness, ten race, (contrast, sex, (age, variables explaining variables for image-specific subject-specific developed and use previously can note, 9] approaches 4, we [3, Such data, models failure. imbalanced effects recognition handle mixed to of ability utiltity their are the to they however, as Due al- separately, independently. of algorithms number trained recognition the each and data, model built of to volume need the the and to tested, due gorithms approaches regression pursue are yet than not an covariates did involve We of will differentials. phenotypes demographic sets exhibit other our richer may and in need itself tone faces that skin will algorithm in estimate modeling to evident that efforts phenotypes particular, likely In other it available. any think or We tried tone not sets. have skin we image to Specifically, errors them. recognition of model relate inferential to an build to nor results, observed as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE eddntueiaedt rmteItre o rmvdosurveil- video from nor Internet the from data image use not did We eddnttanagrtm.Tepooyeagrtm submitted algorithms prototype The algorithms. train not did We : eddntmk fot oepantetcnclraosfrthe for reasons technical the explain to efforts make not did We h osberl ftecmr,adtesubject-camera the and camera, the of role possible The even nhg-ult mgs oal lvtdfalse elevated notably images, high-quality in - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 9 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 EFFECTS DEMOGRAPHIC OF REPORTING ATIONS RECOMMEND RESEARCH E T XEC ECH S . S . UMMARY UMMARY - ...... differentials demographic covering reports describe: that should suggest posi- therefore false We particularly, me- negatives. specifying, in false without and or discussed tives papers is academic accuracy both particular, In in incomplete, coverage. dia been has effects demographic of Reporting depen- application strongly are which differentials of dent. consequences characteristics limit- actual biological toward the digitized directed of weigh standard processing might automated Any the in application. differentials biometric allowable ing any in specified formally been research: improved Policy and cameras sug- face-aware 8. the analysis, section quality in motivates discussed image This standards-compliance into research images. further standards-compliant of in gestions in reduced discussed are are differentials: therein, These differentials negative modality. false thresholds. combined user-specific of a of idea Mitigation as the iris discount, and and discuss, face also We of distinguishing 9. use of section and capable techniques - particularly twins - between fea- power of discriminative discovery false greater data, to training with diverse respect tures more with training, differentials refined elevation, demographic Threshold mitigating positives: at effective prove may differentials: following positive false of Mitigation equal approximately produce demographics. to all developers across pushes forward, rates that Going positive accuracy false globally. reporting rates start negatives to false when reduce plan developer to we a for is incentive evaluations the prime NIST a remediate cases, in to some participating in developers happened encourage have to may that been While has effects. goal Our algorithms. verification to-one demo- of Testing: mitigation and handling quantification, the differentials. to graphic germane research discuss now We osqecso n ro,i nw,adpoeue o ro remediation. error for procedures and known, race, if error, sex, any age, of Consequences by example for and thereof; - intersection rates some failure in or elevated height, the has group demographic fixed, Which is threshold is; rate the positive whether false target value, the threshold what recognition and on information system; known the Any using in difficulty or processes of duration in differentials Any example; for camera, the by failures detections recognition, failed during enroll, occurrences negative to false or positive false metric: relevant in The assessment, recognition; quality during during or camera, phase, extraction the feature at and - detection occurred the differential the which at stage verifi- The identity system, a into individuals of identification: enrollment or initial cation - system the of purpose The ic 07NS a rvdddmgahcdfeeta aat eeoeso one- of developers to data differential demographic provided has NIST 2017 Since as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE h eret hc eorpi ifrnil ol etlrtdhsnever has tolerated be could differentials demographic which to degree The ihaeut eerhaddvlpet the development, and research adequate With as eaieerrrts n demographic and rates, error negative False - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 10 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 ANNEXES DISCLAIMER DGMENTS ACKNOWLE ne 17 Annex 16 Annex 15 Annex 14 Annex 13 Annex 12 Annex 11 Annex 10 Annex ne 9 Annex 8 Annex 7 Annex 6 Annex 5 Annex 4 Annex 3 Annex 2 Annex 1 Annex E T XEC ECH # S . S . UMMARY UMMARY - Results Results Results Results Results Results Results Results Results Results Results Results Results Datasets Datasets Datasets Datasets CATEGORY h rtfu ealtedtst sdin used of breadth developers. datasets the algorithms’ the show the to inform detail intended to are four and These effects, first evaluated. generated the algorithm The automatically of each pages for 1. 1200 one than usually Table more graphs, in contain annexes listed remaining The are report. report this this to annexes of entirety The the correct. summarize be to always purporting not will statements recognition any one face indeed just employ and typically because averages Applications women, so Miami). taking algorithm, and in analogy, Montreal (by rate in meaningful temperatures match seldom of are false average the distributions of different averages increase from take samples average not of do the We averages example developers. Annexes algorithms’ for the 17 algorithms, inform in to show over and to contained intended effects charts are the These of of algorithm. breadth each pages the for 1200 results of than reporting exhaustive more include which with report this supplement We participating results. Developers evaluation publish purpose. to the permission equip- for NIST and grant available FRVT products best in the the that necessarily imply are it Na- identified does the nor ment Technology, by com- and endorsement any Standards or of of recommendation identification Institute imply does tional vendor, case or per- no name, to In trade order document. product, in mercial this used in were described report evaluations this the in form identified products software and hardware Specific infrastructure algorithms. for of Laboratory Research evaluation Biometrics rapid NIST supporting the in area. staff this to grateful in are work authors their The of discussions detailed John for Test and Technology, Biometrics Maryland of Bion the Institute of Florida at of Campbell SAIC King of Michael S&T, Howard DHS John of Vemury and Arun Sirotin Facility, Yevgeniy to grateful are authors The 1:1 1:1 1:1 1:N 1:N 1:1 1:1 1:1 1:1 1:1 1:1 1:1 1:1 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False Application Mugshot Application crossing Border Visa Application Mugshot Mugshot Mugshot Mugshot Mugshot Mugshot Mugshot Mugshot Visa Application Application FRVT DATASET - AERCGIINVNO TEST VENDOR RECOGNITION FACE al 1: Table addt itsoemgiue ysxadrace and sex by magnitudes score list sex Candidate and race by mugshots characteristics States error United Identification photos for crossing distributions border score and impostor application and race global Genuine and for sex country by by images rates mugshot negative False States United in rates negative mugshots False States United with characteristics tradeoff photos Error application with country and photos age visa Cross with rates match false age photos Cross application with images rates application match using false Cross-age demographics matched images with application rates worldwide match in False rates images match mugshot false States sex United and in Cross-race rates match false sex impostors and Cross-race matched demographically for rates photos match crossing False Border metadata: and images of portraits Visa examples and metadata: Description and images portraits of Application examples and metadata: Description and images of Mugshots examples and metadata: Description and images of examples and Description CONTENT nee n hi content their and Annexes - DEMOGRAPHICS . T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 11 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: biomet- consistent 37. complete, Part more 2382 A ISO/IEC document. as this available is in vocabulary appearing rics terms common defines table following The 08:14:00 2019/12/19 SYMBOLS TION IDENTIFICA ONE VERIFICATION ONE COMPONENTS ALGORITHM TYPES DATA DEFINITIONS AND TERMS - - TO TO E T XEC ECH - - MANY ONE S . S . UMMARY UMMARY - NRIetfiainflennmthrt maue vrcmaio fsamples) of comparison over (measured samples) rate of non-match comparison false over Identification (measured rate match false Identification samples) of comparison samples) over of (measured comparison rate over non-match (measured false rate Verification match N false Verification N N N FNIR FPIR not gallery is into FNMR person probe individual a a an searching that for of declaration identifier process an a The FMR to or pointer label or identity index an Some of either assignment The label identity an decision with Identification tagged each templates, of set identifiers, A identifier,Identification different one under enrolled under are enrolled person label a are Identity of person samples which a for of gallery A samples all same the which to for belong gallery gallery they Unconsolidated A if determine to samples two comparing of process The gallery Consolidated Gallery similarity because non-matches declared false producing person, comparisons one genuine from of samples Proportion two because associate matches declared to false Failure persons, producing different comparisons Verification imposter from of samples Proportion two rate of non-match association False Incorrect non-match False rate match person False same the from samples of Comparison persons different from candidates match samples of False of list a Comparison produce to templates of comparison database Genuine a searches tem- that a Component into comparison sample Imposter score a similarity converts a that produces algorithm and templates recognition two face compares a that of Component image Component an in faces finds a that produce Component to compared are scores similarity generator Template vector which feature against a number, includes that real algorithm Searcher Any recognition face by produced Data Comparator person a detector recognition of Face a face the by of rendered images as more samples, or two One in faces two of similarity person of a Degree of identity the encodes Threshold that numbers real of vector A Template score Similarity Sample vector Feature M NM G as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE h ubro ae erhsconducted searches mated of number The conducted searches non-mated of gallery, number a The into gallery enrolled a samples into of enrolled number are The faces whose subjects of number The gallery the in when whence not or person threshold a below is score threshold a above or at is score similarity detector face a embeds implicitly component this plate; decision verification algorithm N < N N = G N G - DEMOGRAPHICS T N  G 0 ≥ N → → . NR FNIR FNMR, M,FPIR FMR, → → 0 1 12 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 71 70 66 61 53 28 positives false of mitigation toward Research 9 negatives false of mitigation toward Research 8 identification in differentials positive False 7 identification in differentials negative False 6 verification in differentials negative False 5 verification in differentials positive False 4 metrics Performance 3 work Prior 2 Introduction 1 definitions and Terms Disclaimer Acknowledgements Contents 08:14:00 2019/12/19 E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 20 18 14 12 11 11 13 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: age, the see, will we as and, comparison a in involved are people 1 two in or point, identity one salient that one-to-one The is both comparison. context, algorithms, differential demographic this Face the on not?”. built are or algorithms, person search same one-to-many two, “question and the verification the between answer similarity of to Face measure used a be produce subject. can and the samples which of image face property two other from extract some information identity vectors or compare features They sex, in operators: differential or as age, operate hand, of other the estimate on algorithms, an recognition produce accept on, and reported sample MIT kind image the of face algorithms, one classification Face terminology: in confusion a from stems This face that evidence as cited widely have well how at looked it instead cloud-based recognition, face accessible study publicly [22] not did studies work discussed prior MIT and The noted context, implications. policing work regulatory a Georgetown and in policy The particularly impacts potential 36]. the [5, described MIT and bias, by of reports sources articulated two and [14] Georgetown on barriers based some lowered is applications, In of number press. larger popular a development the algorithms, the algorithm in of to recognition capability face expanded of the to coverage due expanded is been this part has there years two last the Over Introduction artificial labelled be may that 1 algorithms includes now. necessarily differentials AI-based demographic stages are which modules these at extraction of pipeline feature none recognition and detection face that the a Note typically in though stages intelligence, arise. possible show principle, to in intended could, is figure The 1: Figure 08:14:00 2019/12/19 ag ubr fiett-aee mgs(rmtewb n ntefr fwbcrtddtst VG,IBC) n h viaiiyof availability the and IJB-C]), [VGG2, datasets web-curated of networks. those form of training the development supported in the of has and to availability GPUs web, the leading powerful the recognition more academic face (from ever and For images industry identity-labeled etc.). Tensorflow in of (Caffe, research numbers thereof large AI implementations open-source well-capitalized and from networks, stem neural convolutional performance recognition face in Gains + +Exposure - Exposure Capture Source: http:// Source: Image E T XEC ECH S . S . UMMARY UMMARY www.telegraph.co.uk ⟶ ⟶ Assert of PA B Missed PA PCER False PCER False Detection APCER False accusationFalse Attack Security hole Security /technology/2016/12/07/robot 1 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False False Assert Assert False Assessment n,ntlat eot httetcnlg ssmhwbae.Ti atraspect latter This biased. somehow is technology the that reports least, not and, , Acceptance FRVT Quality Low Low Q estimation - AERCGIINVNO TEST VENDOR RECOGNITION FACE ⟶ + Inconvenience Localization recognition Detection, Detection, detection detection Incorrect Incorrect - passport Face Face loihscndtriegne rmasnl mg.Testudies The image. single a from gender determine can algorithms No No - checker sbiased. is - rejects Extraction Feature Feature - asian - mans - photo of recapture ⟶ ⟶ - Inconvenience Security hole Security having - Verification Rejection 1:1 False False 1:1 DEMOGRAPHICS 1:1 False False 1:1 Accept FNMR - FMR eyes/ Recognition ⟶ ⟶ ⟶ ⟶ ⟶ Identification False accusationFalse False lead False Displaces actual lead actual Displaces Missed lead Wasted effort 1:N “False “False 1:N 1:N “Miss “Miss 1:N Alarm” on on others Rate” T FNIR FPIR  0 → → NR FNIR FNMR, M,FPIR FMR, Human Review Human or not? person Same Same → → 0 1 14 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: poor of presence the upon fell suspicion algorithms, gender-estimation in bias on [5,36] publica- studies after MIT Indeed, the culpability. of AI tion or algorithmic any without from cameras, images and mis-configured unrecognizable) from “stage”, (and collected poor or be manifestly can environment that the Note from algorithms. assessment camera, quality the or of detection during client-side inadequacies im- occur least) that single (at differentials a Demographic from only arise images. when could other is with collection that comparison stage, any capture before the collected above being at out is pointed appear age As to differentials individuals. demographic tall very for or potential pho- [28], is yield children there demographics young certain e.g. that engine. recognition possible recognition face is downstream to it generic ill-suited and tographs a recognition, or on detection effect undermine an can have image can poor A cameras that shown has [9] research Recent quality image of role The 1.2 a and stage any almost at rejection errors. cause of could origin This the determine cropped. to was need individual head would tall the owner a of system of part image an top produce the could which view in of field which a example, in cooperative narrow For too make itself. have recognition that subjects before optics stages where with early equipped the system camera in a a rejected be In could person a consequences. camera, the downstream to have presentation generally Errors will differential. demographic stage a have one notionally at could which of any metrics, performance shows and Figure QA The the particularly systems, some necessary. in be not exist capture may not the functions may PAD to components coupled Some be PAD. may precede component the would QA of the and order example process The for involvement. systems, human some prompt in different may be which feature may of of components output components the recognition comparison, the 1:N or then 1:1 compliance, and standard accep- extraction portrait quality checking a at followed attempts, aimed impersonation camera, step detect a (QA) to primarily tance intended Accord- figure module subsystem, The (PAD) capture detection fault. parts. a attack several at presentation of of a is consisting composed by process pipeline is recognition the system recognition face of face notional component a a what that shows show exactly to on 1 specificity Figure introduce is we bias ingly, of discussion the in systems recognition Lost face in bias of sources Potential 1.1 capability algorithm that second, photographs; face developer. process by that considerably algorithms varies on effect member- group sizeable demographic a that have First, can respects. two ship in tale cautionary outcome. a recognition as serve the nevertheless to reports material MIT The be will both of properties demographic other and race sex, 08:14:00 2019/12/19 E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 15 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 6 5 4 which for photographs 3 produce to easy relatively 2 is it that understood being it individuals, skinned light and consideration geometry included and work lights standardization The cameras, century. of a almost cameras for used digital been had of that face anticipation would cameras automated film in that replace deployed conducted FBI photographs was the of work before capture standardization decade a That the than in recognition. more occurred departments It collection police review. mugshot human) local for (i.e. and forensic best-practices state support photographic primarily establish guide to to NIST done asked was FBI the 1990s late the In Standards Photographic 1.3 to related properties including potentially time, over head photographs both (e.g. presentation affect demographics. poor would - and that pair dark), mated those too a and (e.g. in - illumination photo down) poor one mis-focus), affect to (e.g. expected photography are poor that to factors due between distinguish its to and need we 39794-5 here ISO/IEC However, prominently most standards 1-2017. portrait ITL formal adoption ANSI/NIST pho- the from equivalent sub-standard law-enforcement to deviate very due that to respect those tolerance this remarkable in i.e. demonstrate advances tographs which great networks been neural have two more there convolutional for years deep research five of recognition face last of the bulk such over the to Indeed of immunity focus decades. the with been algorithms has of so factors development much “nuisance” support so such to expression to and Invariance up illumination built pose, been as have such databases factors dedicated to negatives that false algorithm. attribute the to to common different very appear is face It same the from two compared. when and occurs analyzed negative are false samples two a So operator: similar. differential insufficiently a are as implemented photographs is input recognition two face from that extracted Recall features below the score when comparison occur a will yield This individual one threshold. from a samples when systems biometric in occur negatives False type “skin short, decision”. in algorithm - better 31] a [30, addressing produced accuracy in active it classification been gender that has is IBM previously, here and Relevant response, in bias. estimation study; AI gender MIT IBM the An in faulted dataset. been that had in algorithm individuals dark-skinned of under-exposure to due photographs, 08:14:00 2019/12/19 99 eddltrfra tnadzto of standardization formal later seeded 1999, Mugshots , overview. this of 1999. See Capture 6322, Report the Interagency for NIST See Recommendation Practice Best 19794-5. ISO/IEC as such documents, example. Early for databases, PIE famous Models. The AI in Bias Mitigating See E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT 6 - hr a xlctcnieaino h edt atr mgso ohdark both of images capture to need the of consideration explicit was There . AERCGIINVNO TEST VENDOR RECOGNITION FACE yitself by 2 n xmndwehrsi oeisl drove itself tone skin whether examined and , - DEMOGRAPHICS a iia feto h classification the on effect minimal a has T  0 5 en vial to available being → → NR FNIR FNMR, M,FPIR FMR, 4 → This . → 0 1 16 3 . This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 7 finger- presenting amputees finger or course sunglasses, prints. of wearing valuable subjects are photosensitive biometrics of of recognition use example inclusive for the - to dedicated equipment are of that Development thereof, studies effects. and demographic excluded, algorithms, broad be and of should detection factors to such dedicated so studies timescales, from short possible, very when on appearance change can disease or This injury authentication. Clearly fingerprint photographs. for two between also lapse and time (usually) accuracy significant recognition as without face factor even for demographic applies being differ a by young is the either itself and systems Age elderly control negatives. the border false in higher automated encountering many by in or quickly policy, disadvantaged more by are ages excluded that who group - demographic one children least - at is others there grace- However, more-or-less than slow, us. a of is all it to because happens factor that demographic process a ful, be to recognition ageing face consider automated don’t undermine we study, ultimately current will the and decades over appearance change will Ageing ageing and Age 1.4 match, degradations. can quality in- exposure of poor have tolerant highly moderately will are even algorithms photograph with contemporary underexposed leading taken egregiously and photographs any an two course but i.e. Of content, those information appearance. sufficient in facial differences digital by the and Accuracy in photography/illumination/presentation change photographs. poor two by from extracted undermined features be on can operation differential a as impossible. features proceeds anatomical recognition of Face detection render can pixels bright or dark of areas large 08:14:00 2019/12/19 e eetrslsfrvrfiainagrtm nteFV eot,adfrietfiainagrtm nNS neaec eot87 [17]. 8271 [3]. Report Best-Rowden Interagency see NIST models, in mixed-effects algorithms using identification ageing, for of and analysis reports, FRVT longitudinal the formal in a algorithms For verification for results recent See E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 7 In . 1 17 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: recognition. face automated in effects demographic in race. work and Since recent age summarize [35]. sex, 3 2003 across and differential as 2 positive back Tables false far large as reported known has been [16] have report sex FRVT ongoing and our age 2017, concerning report this in given effects and broad positive reports. The other false in between missing distinguish often and is report that properly something to effects, first negative the false is report this below, detailed as Except many identification. for and results verification includes both contrast, implementing in algorithms report, commercial, This mostly recent, algorithms. verification one-to-one to relates work prior All work Prior 2 08:14:00 2019/12/19 # 2 1 3 S&T DHS with SAIC/MdTF at [20] al. et Howard Tech Inst. Florida at [23] al. et Krishnapriya Dallas UT at [8] al. et Cavazos SOURCE E T XEC ECH S . S . UMMARY UMMARY outes[ 9] volunteers adult collected, Lab [37] db Morph mugshots: Operational portraits [32] GBU Dame Notre IMAGE as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT 363 casians Cau- 769 2 + Am. African 350 10 389 SUBJECTS - AERCGIINVNO TEST VENDOR RECOGNITION FACE UBROF NUMBER - casians Cau- 611 10 + Am. African 620 42 < IMAGES 1085 al :Pirstudies. Prior 2: Table eie,adapidoelaigcmeca eicto algorithm. verification commercial capture leading 11 one from applied images and employed devices, study The to approach effect. gain their information ordering an formally establishing race, and rates sex positive age, false with in variation order-of-magnitude imposter showed of It heterogeneity distributions. narrow and broad for and and performance“ outcome” “differential “differential for definitions useful establish study The differentials. demographic fixed-threshold documenting for characteristics tradeoff inadequacies error the of noted also study the Importantly, of deployment. representative commercial not and inaccurate quite the are of Two algorithms open-source algorithm. commercial used one study and That algorithms, 24]. open-source [23, three images quality higher Americans; in African differentials in reduced rates and negative false lower Caucasians; African in vs. positives Americans false elevated order-of-magnitude reported: study The older one and algorithm. pre-DCNN [38], inaccurate algorithm open-source one Maryland, of from deprecates University algorithms It the related race. two and used sex study by The imposters (AUC). area-under-the-curve of pairing the the discusses i.e It “yoking” of apparent. effect not is differential negative tradeoff error false linked the nor characteristics FNMR(T) neither FMR(T). showed reported study study the The As faces. Caucasian vs. in Asian rates volunteer positive university false in elevations order-of-magnitude showed study The DISCUSSION - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 18 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 # 5 4 ta.[13] al. et Khiyari El S&T DHS with SAIC/MdTF at [9] al. et Cook SOURCE E T XEC ECH S . S . UMMARY UMMARY b[37] db Morph mugshots: Operational volunteers adult collected, Lab IMAGE as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT sex + race on balanced adult, 724 525 SUBJECTS - AERCGIINVNO TEST VENDOR RECOGNITION FACE UBROF NUMBER al :Pirsuis(continued). studies Prior 3: Table nsex on balanced casians, Cau- + Am. African each 1448 = 2896 IMAGES eonto plctoslk rni hog nairport. an through transit short-term like for applications useful recognition are men biometrics than Same-day scores genuine separation. lower time give with women that four particular to in up apart, collected years those same than the differentials on demographic collected different samples have of day [3]. comparison study that ageing showed best also the study to The contrary but African [22] or with Black comporting as American, identifying those in study scores The similarity lower performance. differential found demographic induce potential to showing camera devices, the capture for different from people different same markedly the displayed of report images The device. skin eyewear, capture height, on age, and reflectance sex, on scores similarity genuine of examine dependence to models regression mixed-effects deployed study The admyo,sy ihsm g,rc,adsex. and race, age, e.g. same paired with say, were or, imposters randomly how disclose not differentials. does at FMR paper FNMR burying the reports Moreover, thereby paper thresholds the fixed Second at 0.01. not = FMR, FMR rates: fixed at match 0.06 false = modest FNMR very best at the rates reasons: error two high for give discounted algorithms be the should men, First study the The in whites. rates in error and verification elderly, [38], better the algorithms( show two to with COTS) database one MORPH and the modified of subset a used paper The DISCUSSION - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 19 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: where T: threshold, some scores, not. genuine or N random mated of in is vector comparison done a a be From that infer should cannot comparisons mated test These from under algorithm produced the pairs. are that non-mated scores so order from Genuine produced scores: are of scores sets imposter two pairs; forming by estimated is accuracy Verification metrics Verification 3.1 formal the define document. reported this subsections be throughout following used to the metrics metrics Accordingly different implementations. requires verification standard and reporting identification for and testing performance 19795-1 ISO/IEC The not incorrectly is individual the error where I II themselves. Type Type with so-called and associated another, the with errors, associated of incorrectly is kinds individual two an commit where generally systems identification and verification Both metrics Performance 3 08:14:00 2019/12/19 ltigo M nalgrtmcai.Ti sdn eas yia prtostre as ac ae spanning supports rates match This false target operations scale. typical because decades logarithmic done several is a This on axis. logarithmic spaced a on evenly FMR measurements of plotting FMR to corresponding thresholds K of observed the of quantiles from thresholds of set scores, a generate imposter typically We value. any on take can T, threshold, The where iial,gvnavco fMipse scores, T: imposter above M of vector a given Similarly, E T XEC ECH Q H v ( S . S . x steqatl ucin n FMR and function, quantile the is ) UMMARY UMMARY steui tpfnto,and function, step unit the is 10 v sflos ie oeitrsigflemthrt range, rate match false interesting some Given follows. as , − 6 oa ihas high as to log as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT 10 FMR - AERCGIINVNO TEST VENDOR RECOGNITION FACE 10 k u − h as o-ac ae(NR scmue stepooto below proportion the as computed is (FNMR) rate non-match false the , log = FNMR 2 . FMR H 10 (0) k ( T FMR oe from comes T ( k T 1 = ) ae ob 1. be to taken = = ) v h as ac ae(M)i optda h proportion the as computed is (FMR) rate match false the , L Q + v M − 1 (1 K k N X i − 1 M =1 [log X FMR i =1 N H ( 10 H v k i FMR ( ) − u - i T − DEMOGRAPHICS ) U T − ) log 10 [ FMR FMR L L , ] FMR T  0 U ] efr vector a form we , → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 (3) (4) (2) (1) 20 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 FPIR candidates: return erroneously that compute we lists, candidate the From Links: examine might analyst human A zero. to equal or all than either greater be to required scores of all population with order, closest enrolled descending the an return into to conducted configured be be to to search a for typical is It positives false Quantifying be must rates 3.2.1 3.4. failure section in these addressed image, is This quality comparison. poor algorithm a support to example, rates algorithms for error recognition because from, the Further template with combined a recorded. produce be must not to events to fail such elect or of might image, occurrence an the from search, template one-to-many a a produce execute to the fail minus variables. sometimes one performance II algorithms simply Type recognition and is I because Type first the Additionally, the for metrics - define rates” 3.2.2 and “miss 3.2.1 of Sections below. instead detailed rates” as other “hit about talk to prefer practitioners Many conditions: error two of mate frequency a quantify has must search algorithms a identification that biometric infer open-set cannot of test Tests under not. algorithm or the that randomly so by searches conducted non-mated be not and should mated from searches ordering obtained These lists searches. candidate non-mated of from set set a a First, second, lists: mated-searches; candidate of sets two from estimated is accuracy Identification metrics Identification 3.2 the than higher somewhat be comparisons. should multiple This M. comparisons, limit imposter three” of of number “rule the FMR by with sustained plotted is are These as FMR(T). low as vs. FNMR(T) of plots are characteristics tradeoff Error 08:14:00 2019/12/19 . . ( identity. Misses data. enrollees’ more or one with associated incorrectly positives False E T ,T N, XEC ECH L S . S . = ) UMMARY addts rjs h top the just or candidates, UMMARY yeI rosaiewe erho nerle esnsboercde o euntecorrect the return not does biometric person’s enrolled an of search a when arise errors II Type : u.nnmt erhswt n rmr addtsrtre ihsoea raoethreshold above or at score with returned candidates more or one with searches non-mate Num. 3 yeIerr cu hnsac aafo esnwohsnvrbe enbfr is before seen been never has who person a from data search when occur errors I Type : /N eas ape r eeal o needn u oteueo h aeiaein image same the of use the to due independent not generally are samples because as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE as oiieietfiainrate identification positive false R L ≤ addt dniis hs addtsaerne ytersoe in score, their by ranked are candidates These identities. candidate L u.nnmt erhsattempted. searches non-mate Num. dniis rol hs ihsoegetrta threshold, than greater score with those only or identities, - DEMOGRAPHICS stepooto fnnmt searches non-mate of proportion the as N dniis n o h algorithm the for and identities, T  0 U → → → NR FNIR FNMR, M,FPIR FMR, 1 n FMR and → T → . 0 1 (5) 21 L This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 dnicto ytm( system are identification 100 to close numbers where Reliability effect differences nineties” perceive “high don’t the readers subtly, to More inured indifferently. perceived becoming costs. we in well, we then doubling then 100% system, 97%, a near to control see numbers 98.5% access to say in scale an from, TPIR the in of invert FNIR decrease to as double have that we express mentally we if If example, delay. and For First, inconvenience reasons. user rates. two double for error rates with miss negative linearly false preferring rise rates, costs “hit” positive true report not does report This successful: are simply searches is This mated often rates”. “hit of talk to prefer communities many rank, the good at not or threshold above not either is rates identification positive adding true miss, and a rates, Hit as regarded is result the either made, list, not candidate was a FNIR. template produce to search to a fails because algorithm or algorithm failure, failed, the finding search if face the Thus from because arising crashes. miss software a or as quality, same poor mate the a of treated Thus is intolerance list misses. candidate of a causes between on reported distinguish not not does is it that that in evaluation for simple is formulation This Links: of terms in accuracy state for availability, to labor then or useful policy to is (according It reviewed be example). might lists short only and applied, be threshold, may thresholds some above is score the which for dates If misses and hits Quantifying 3.2.2 for accounts selectivity, quantity, alternative it An [17]. - see above. search - and threshold a above 2 in candidates rank produced multiple at candidate candidates non-mate consider highest to the necessary from not computed is be can FPIR definition, this Under 08:14:00 2019/12/19 FNIR name L repstv dnicto rate identification positive true addtsaertre nasac,asotrcniaels a epeae ytkn h top the taking by prepared be can list candidate shorter a search, a in returned are candidates ( E T as eaieietfiainrate identification negative false ,R T R, N, XEC ECH S . S . UMMARY sacrepnigtr,tpclybigietclto identical being typically term, corresponding a is UMMARY = ) u.mt erhswt noldmt on usd o ak rsoeblwthreshold below score or ranks R top outside found mate enrolled with searches mate Num. AFIS evaluations. ) as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ( TPIR TPIR FI) sfollows: as (FNIR), hc stecmlmn fFI iigapstv ttmn fhow of statement positive a giving FNIR of complement the is which ) hl NRsae h ms ae shwotntecretcandidate correct the often how as rate” “miss the states FNIR While : ( ,R T R, N, u.mt erhsattempted. searches mate Num. 1 = ) T ≥ − 0 R hsrdcino h addt iti oebecause done is list candidate the of reduction This . FNIR and TPIR T ( ,R T R, N, ow en ms ae ihtegeneral the with rate” “miss a define we so , - DEMOGRAPHICS n fe ie natmtd(fingerprint) automated in cited often and , ) T  0 → → NR FNIR FNMR, M,FPIR FMR, R ≤ L → candi- → 0 1 (7) (6) 22 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: better. fraction or the R reports rank just at and mate requirement, the returning threshold searches the mated relaxing of by scores similarity ignores It only. searches the is case special important An 08:14:00 2019/12/19 oaihs oso utpedcdso FPIR. of decades 1 multiple = show TPIR to logarithms, (e.g. rate error an of charac- complement operating function the receiver same or the characteristics serve (DET) parametrically tradeoff These FPIR error (ROC). vs. detection teristic FNIR called often For i.e. are rates plots errors. identification Such classification positive T. false I with Type vs. and negative II false Type plots between this tradeoff identification the represents characteristic tradeoff error An applica- verification in used are control). access imposter as as and (such (such genuine tions applications while identification checks) in background used or traditionally search, are enforcement synonym “nonmate” law a and as used “mate” is words “imposters”” The word the nonmate. and for mate, for synonyms are “authentic” samples. or persons “genuine” different words of The comparison from comparison coming a scores, from nonmate coming a or scores, is samples, mate person’s score either one be dissimilarity of can the scores cases, case, any some In In recog- properties. recognition. metric face possessing iris and distance in higher fingerprint used case by are which computed dissimilarities in while traditionally scores, algorithms, are dissimilarity nition scores or Similarity person, same people. the different from indicate come indicate values to values likely higher more case which are in samples scores, the similarity that either algorithm be recognition can the scores from Comparison score comparison threshold. native some the meets when system biometric a by declared are Matches associated improperly are persons error. person two I one from Type of samples samples when two occurs match positive to false fails a Correspondingly, algorithm error. an when occurs negative false a biometrics, In interpretation DET 3.3 candidates. that many ignores at also look It to hits. willing weak dependent often and not is scoring) reviewer is (high adjudicating it strong efficacy, an between algorithm’s distinguish not an does of it indicator so summary scores, similarity common on most the is quantity this While 1). The FNIR quantity, this of complement the cite primarily We akoehtrate hit one rank E T XEC ECH S . S . UMMARY UMMARY stefato fmtdsace iligtecretcniaea etrn,ie CMC(N, i.e. rank, best at candidate correct the yielding searches mated of fraction the is as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT uuaiemthcharacteristic match cumulative - AERCGIINVNO TEST VENDOR RECOGNITION FACE CMC ( ,R N, − − oso ro tradeoff error show to 1 = ) NR n ntasomn h xs otcmol using commonly most axes, the transforming in and FNIR) − ( ,R, N, FNIR ( 0) ,R, N, ( h rcino mates of fraction the , CMC - DEMOGRAPHICS 0) hc umrzsacrc fmated- of accuracy summarizes which ) − u ifr o xml,i plotting in example, for differ, but T  not 0 ntetpRranks. R top the in → → NR FNIR FNMR, M,FPIR FMR, − yeII Type a → → 0 − 1 (8) 23 a This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: occurred. investigated had be alarm should false failures a generation though template as cases, those de- In evade to samples. negative attempt quality may poor to individuals submitting appropriate by hostile not tection which in is detection, approach fraud This visa as recognition. such at applications, identification attempts co- where make control and access enrolled as are such users applications operative positive-identification for treatment correct the is approach This fol- as measurements FPIR and FNIR lows. into both. events or failure-to-extract comparisons incorporate imposter test similarly comparisons, we under genuine identification, in algorithm For used the be because will concern image that given of opportunities any not whether Gaming generally know FMR. are not does decreases FMR of and treatment FNMR this increases from This arise occurred theoretically score. failure-to-extract accuracy a similarity FMR which zero and for a FNMR image producing an the involves as into that rates comparison extract any to regard failure we include because explicitly statements, to need search. not we and do because enrollment we for FTX verification, generation report In template for not used do is algorithm We underlying FTX. same The the by that templates. is denoted assume to failures failure-to-extract, converted of not termed proportion are is The images proportion search template. some corresponding Similarly, a FTE. to by image denoted face rate, a failure-to-enroll convert the to fail algorithms some enrollment During features extract to Failure 3.4 08:14:00 2019/12/19 . . r ae,o o-ae:Mtdsace ilfi iigeeae NR o-ae erhswl not will searches Non-mated reduced. be FNIR; will searches elevated FPIR whether giving so fail positives, on false will depends produce searches accuracy Mated search on non-mated: failure or generation mated, template are and of identities hypothesized effect no The have entries whose score. list zero candidate a be to taken is result the imagery, input search 1:N and templates Search of factor to a so, by positives reduced false be produce will not FPIR order, will searches first non-mated FNIR; elevated giving non-mated: fail or will mated, searches are Mated searches template subsequent of whether effect on The depends templates. accuracy search length on zero failure process generation transparently to [18] API the by required are rithms templates Enrollment E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT n alderlmn srgre spouigazr eghtmlt.Algo- template. length zero a producing as regarded is enrollment failed Any : - AERCGIINVNO TEST VENDOR RECOGNITION FACE ncssweeteagrtmfist rdc erhtmlt from template search a produce to fails algorithm the where cases In : 1 − FTE. - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 24 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 44 43 42 41 40 39 38 37 36 35 34 33 32 31 30 29 28 27 26 25 24 23 22 21 20 19 18 17 16 15 14 13 12 11 10 9 8 7 6 5 4 3 2 1 E T noeTcnlge Inc Technologies Incode London College Imperial Ltd Pty Technology Imagus Idemia University ITMO Technology ID3 Institute Research Hikvision Ltd Technology AI Hengrui Ltd Co Solutions Pixel Guangzhou Technology Gorilla Ltd Glory Cogent Gemalto Inc FarBar Ltd FaceSoft Recognition Eyedea Barriers Digital Co Technology ChuXing DiDi Dermalog Deepglint Ltd Co Technology Dahua DSK Corp Cyberlink Cyberextruder GmbH Systems Cognitec Ltd Co. Telecom Chunghwa Petroleum of University China Corp Import-Export Electronics China Technologies Camvi Ltd Co Bank CTBC Technology IntelliCloud CSA Bitmain Inc Technology Vion Beijing Ayonix Systems Awidit Aware AnyVision Investments Anke Group Amplified AlphaSSTG AllGoVision Sys Innovation / Alivia Inc Alchera Ltd PTE Global Adera 3Divi Developer XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE al :Agrtm vlae nti report. this in evaluated Algorithms 4: Table to05itmo-006 itmo-005 id3-004 id3-003 hik-001 hr-002 hr-001 pixelall-002 gorilla-003 glory-001 cogent-004 cogent-003 f8-001 facesoft-000 digitalbarriers-002 didiglobalface-001 dermalog-006 dermalog-005 deepglint-001 dahua-003 dahua-002 dsk-000 cyberlink-003 cyberlink-002 cyberextruder-002 cyberextruder-001 cognitec-001 cognitec-000 chtface-001 upc-001 ceiec-002 ceiec-001 camvi-004 camvi-002 ctbcbank-000 intellicloudai-001 bm-001 vion-000 ayonix-000 awiros-001 aware-004 aware-003 anyvision-004 anyvision-002 anke-004 amplifiedgroup-001 alphaface-001 allgovision-000 isystems-002 isystems-001 alchera-001 alchera-000 adera-001 3divi-004 3divi-003 algorithms Verification incode-004 imperial-002 imperial-000 imagus-000 idemia-005 idemia-004 - DEMOGRAPHICS av- av- camvi-4 camvi-3 camvi-1 ayonix-0 aware-3 aware-0 anke-002 anke-0 allgovision-000 isystems-3 isystems-0 alchera-0 3divi-3 3divi-0 algorithms Identification noe0incode-004 incode-0 imperial-000 imagus-0 idemia-5 idemia-4 idemia-0 hik-5 hik-0 pixelall-002 gorilla-0 glory-0 f8-001 eyedea-3 eyedea-0 dermalog-6 dermalog-5 dermalog-0 deepglint-001 dahua-002 dahua-1 dahua-0 cognitec-2 cognitec-0 T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 25 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 88 87 86 85 84 83 82 81 80 79 78 77 76 75 74 73 72 71 70 69 68 67 66 65 64 63 62 61 60 59 58 57 56 55 54 53 52 51 50 49 48 47 46 45 E T ieI mrcsLLC Americas TigerIT Thales Tevian Lab Deepsea Tencent SA Tech5 Synesis Limited Hybrid Star Smilart Ltd Co Technologies Intellifusion Shenzhen CAS Tech Integrated Adv Inst Shenzhen Limited Networks EI Shenzhen Technology Yitu Shanghai Academy Film Shanghai - Universiy Shanghai Ltd Co. Technology Electronics Ulucu Shanghai University Tong Jiao Shanghai Software Shaman Ltd Group Sensetime Ltd Saffe Ltd Corporation Rokid Holdings Remark Inc Realnetworks Computing One Rank (EverAI) Paravision Singapore Center R+D Panasonic Limited Private Technologies NotionTag Nodeflux Neurotechnology NEC Lab N-Tech Technology Smart Moontime Ltd Co Digital Momentum Microsoft MicroFocus Megvii/Face++ Industries Electroplast Lookman University State Moscow Lomonosov Inc Kneron Pte International Kedacom Corp Kakao You It Is Intellivision Group Research Intel Technologies Information of Institute Innovatrics Developer XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE al :Agrtm vlae nti report. this in evaluated Algorithms 5: Table mlr-0 smilart-003 smilart-002 intellifusion-001 siat-002 siat-004 einetworks-000 yitu-003 shu-001 uluface-002 sjtu-001 shaman-001 shaman-000 sensetime-002 saffe-002 saffe-001 rokid-000 remarkai-001 realnetworks-003 realnetworks-002 rankone-007 paravision-004 everai-paravision-003 psl-003 psl-002 notiontag-000 nodeflux-002 nodeflux-001 neurotechnology-006 neurotechnology-005 ntechlab-007 ntechlab-006 mt-000 sertis-000 microfocus-001 microfocus-002 megvii-002 megvii-001 lookman-004 lookman-002 intsysmsu-000 kneron-003 kedacom-000 kakao-002 kakao-001 isityou-000 intellivision-002 intellivision-001 intelresearch-000 iit-001 innovatrics-006 innovatrics-004 algorithms Verification ie-0 tiger-003 tiger-002 tevian-005 tevian-004 deepsea-001 tech5-003 tech5-002 synesis-005 starhybrid-001 - DEMOGRAPHICS neurotechnology-007 neurotechnology-5 neurotechnology-0 nec-3 nec-2 ntechlab-007 ntechlab-6 ntechlab-0 microsoft-5 microsoft-0 microfocus-0 megvii-1 megvii-0 intsysmsu-000 kedacom-001 innovatrics-0 algorithms Identification tiger-0 cogent-3 cogent-0 tevian-4 tevian-0 deepsea-001 tech5-001 synesis-0 smilart-0 siat-0 yitu-5 yitu-4 yitu-0 shaman-0 sensetime-002 sensetime-1 sensetime-0 remarkai-000 remarkai-0 realnetworks-003 realnetworks-2 realnetworks-0 rankone-007 rankone-006 rankone-5 rankone-0 everai-paravision-004 everai-3 everai-0 T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 26 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 106 105 104 103 102 101 100 99 98 97 96 95 94 93 92 91 90 89 E T SPSlto Corporation Solution iSAP Inc iQIYI Technology Electronics Yisheng Zhuhai Ltd Co. Information Pico Meiya Xiamen X-Laboratory Ltd Co Winsense Vocord VisionLabs Vision-Box Visidon Solutions Vigilant Ltd Pvt Technology Videonetics Inc. Technologies Via S.L. Solutions Authentication Digital Veridas Inc ULSee Trueface.ai Toshiba Technology Transportation TongYi Developer XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE al :Agrtm vlae nti report. this in evaluated Algorithms 6: Table isap-001 iqface-000 yisheng-004 meiya-001 x-laboratory-000 winsense-000 vocord-007 vocord-006 visionlabs-007 visionlabs-006 visionbox-001 visionbox-000 vd-001 vigilantsolutions-007 vigilantsolutions-006 videonetics-001 via-000 veridas-002 ulsee-001 trueface-000 toshiba-003 toshiba-002 tongyi-005 algorithms Verification - DEMOGRAPHICS ohb- toshiba-1 toshiba-0 algorithms Identification yisheng-0 vocord-3 vocord-0 visionlabs-008 visionlabs-7 vd-0 vigilantsolutions-0 T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 27 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: the of target the like look to imposter an by expended is sometimes effort attempt. recognition is no imposters that of mean pairing pairing, random zero-effort The as it to practice. as referred in algorithms recognition occur face would of that evaluation rates for match inappropriate false is underestimates practice this section, individuals. this all in comparing show exhaustively will by we or As individuals, com- pairing gen- imposter randomly evaluations executing by biometric by comparisons Historically imposter done threshold(s). erated is some This at outcomes rates. positive match false false measuring estimate and to parisons, tests biometric many in necessary is It pairing demographic under rates match False 4.2 thresholds, particular at rates match false on appli- centers FMR particular sections i.e. to subsequent related in are discussion These The 3.1. 2. section Figure in in cations detailed been have verification to appropriate metrics The Metrics 4.1 popu- worldwide and States United domestic both lations. from drawn images with experiments several conduct 7. We section in later demograph- identification across one-to-many rates for match results false present verification We in ics. variation the of accusation. quantification false empirical a gives to section lead This may positive false a financial benefits application a surveillance such issuance, consequences a license downstream In various driving loss. to lead visa-shopping, might positive false e.g. undetected otherwise identities benefit an fraud, some biographic for different apply under subjects once where than applications more in in example, positives consequences For False serious similarly users. applications. have all identification They for one-to-many applications. fixed verification is one-to-one that to threshold hazard a security or with a at configured present score are comparison systems a Most yield threshold. individuals set two a from above samples when systems biometric in occur positives False verification in differentials positive False 4 08:14:00 2019/12/19 Oet-n iapoocoscmaio yage. by comparison cross photo visa One-to-one 3. race. and sex, age, by comparison cross mugshot One-to-one 2. country-of-birth. sex, age, by comparison, cross photo application One-to-one 1. E T XEC ECH ( T S . S . ) UMMARY UMMARY . as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 28 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 interest of metric Performance involvement frequency human Intended role Human Result Threshold identity of Claim Example Role image is given the algorithmThe identity claim of a biographic an explicit a database by fromselected on a phone) or database (e.g. not in a sample, often enrollmentThe E T XEC ECH S . S . UMMARY UMMARY Reference FNMR at low FMR. See10, FMR. sec. 193.1, low 3.2 FNMR at Tables and an actualof mate probability prior rate plus Rare rejection, or detect an actual attempt impostor a rejections, false to determine failed Adjudicate Acceptance decision Y/N. false positives to limit High, passport or ID card. such token as with an claim a identity Explicit phone, unlock. Phone unlock. Door resource. Afford access of a person to a physical or logical Access Control Photo as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT – approx. the false rejection rate identification identification rate rejection false the approx. iue2 eicto plctosadrlvn metrics. relevant and applications Verification 2: Figure - AERCGIINVNO TEST VENDOR RECOGNITION FACE 2 1 # similarity score, S Decision Output Score Output is a is Output Feature Feature extraction e.g. CNN model Provided to Provided System logs System User Search PhotoSearch S Comparison Algorithm ≷ T NIST reports accuracy over the entire range ofreports accuracy range entire NIST over the an - sweeps FMR over a range such as [0.000001, 1]. sweeps FMR over a such range as [0.000001, Threshold, T, DEMOGRAPHICS positive rate Rare a dispute records to resolve Retrieve decision verification Logged confederates system Moderate, to prevent using a system. to login a prior with Claim work. for arrive not did employee an a not thatparticulardispense or drug, an employer adid thatby apharmacist of they claim Refutation individual aof presence specific Record the Non FNMR at moderate FMR. moderate FNMR at algorithms’ possible threshold values. This prior calibration, usually vendor Feature Feature extraction e.g. CNN model - Detection and and Detection repudiation – localization approx. the fraud rate multiplied by the false false the by multiplied rate fraud the approx. fixed by system policy, value from T  0 → → evaluation. NIST’s is scope ofthe This box grey black box. evaluated as engine recognition face Automated NR FNIR FNMR, M,FPIR FMR, - specified → → 0 1 29 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: groups. demographic of sets particular over rates match false of mean the countries. shows 24 plot the . in in 2 born point Annex red were in The described subjects are The images The subject-disjoint. scores. imposter are billion sets 195 two produced This The images. application other 517 441 95-th to Method: 5-th the threshold spans 5 the line setting times by blue within-country obtained images. The 24 0.00003 mugshot of over values. of value taken FMR pairs FMR is zero-effort nominal 480 a mean i.e. shows i.e. associated the imposter line randomly estimates, age” vertical the on The FMR and groups, as estimates. sex country age FMR countries, cross rates Same the all and of match “6. over percentiles within FMR row false 4 mean second verification times the the age-group shows one-to-one in point within in example, the growth For level each shows sexes. At figure similar. and more the made are photos, pairings application demographic For 3: Figure 08:14:00 2019/12/19 How impostors are paired 7. Samecountry, sex, andage ...... o 2 Row group. age and sex 3 Row scores. imposter cross-sex and 4 Row scores. imposter group 5 Row also. mates 24 x 5 x 6 (2 Row countries 24 of each within and age-groups, five of 240). each = in other, each with sexes two of each 7 Row E T 6. Same country andage 6. Samecountry 5. Same country andsex5. Samecountry XEC ECH 3. Samesex andage S . S . eue ahvrfiainagrtmt opr 4 1 plcto mgswt ijitstof set disjoint a with images application 019 442 compare to algorithm verification each used We 4. Samecountry UMMARY UMMARY srw7 u o h vrg soe 40FRetmtsta diinlyicue l cross-age all includes additionally that estimates FMR 2400 over is average the now but 7, row As : cross-age all includes additionally that estimates FMR 1200 over is average the now but 7, row As : h vrg soe 70FRetmtsta includes that estimates FMR 5760 over is average The : h pems on orsod otema vr20FRetmts aeytoecomparing those namely estimates, FMR 240 over mean the to corresponds point uppermost The : h vrg soe 12 M siae o nldn ifrn e M siae also. estimates FMR sex different including now estimates FMR 11520 over is average The : srw7 u h vrg soe 8 M siae htnwicue ifrn e M esti- FMR sex different includes now that estimates FMR 480 over is average the but 7, row As : 0. Zeroeffort 2. Sameage 1. Samesex reports/11/figures/dhs_obim/cross_country/impostors/fmr_same_same_same/imperial_002.pdf 1e−06

as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False 2e−06 FRVT

3e−06 -

6e−06 TEST VENDOR RECOGNITION FACE

1e−05

2e−05

3e−05

6e−05

1e−04

2e−04

3e−04 FMR

6e−04 - 24 DEMOGRAPHICS

2 1e−03 rs-onr oprsn ihneach within comparisons cross-country

2e−03

3e−03

6e−03

1e−02 T 

0 2e−02

3e−02 → → NR FNIR FNMR, M,FPIR FMR,

6e−02

1e−01 → → 0 1 30 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 wihsosrslssmlrt hs bv but above observations: following those the to make We similar results shows birth. which of false 4 country higher specific Figure each give introduce for groups we now demographic question particular this whether address To documented far rates. so match not has analysis our influence that genetic Note clear without trait developmental a be from arrangements, to stems point thought minutiae think, are particularly we structure, that tests, ridge friction in where pairings use the imposter that of zero-effort algorithm fingerprint subjects using of from of tests credentials practice (stolen) The present ethnicity. to and situ- chose age the would sex, to who same relative imposters rates active match more false slightly a understate demo- of pairings similar ation imposter increasingly zero-effort from fully drawn that are shows imposters This when increase graphics. rates match by false how FMR shows about figure gained The race. and information sex quantifying age, factors, by demographic factor the of pairing knowledge influential having formal a most [20] the developed Facility showing Test Maryland to DHS’ at approach Evaluators hand-crafted. is rows these of ordering The Links: 9 8 08:14:00 2019/12/19 fragrtm rmHK au,Yt,Apaae epe ecn,Toshiba. Tencent, Deepsea Alphaface, Yitu, Dahua, HIK, from algorithms search for however, multi-finger etc.), one-to-many 8 whorl in Annex (arch, in used historically, classes figure birth. least the pattern influence. at example, at Fingerprint genetic were, for explicitly. fingerprints under these See, not be of and least well absence variations, at may (geographic) to feature, this regional biometric and leads strategies. have a men, gene to in as SMARCAD1 shown than used been women the not have in likely of average is absence on itself smaller, The distance is The ridges known: friction is between structure distance the ridge Further, friction on influence Genetic . . . . . in:Sm loihsdvlpdi atAi edt ielwrFRi htso ujcsbr in born subjects of photos in FMR lower give to tend countries Asia excep- Asian East important East are in there developed However algorithms populations. Some Asian tions: East in highest the and populations European on FMR 0.000046 lowest of and FMR highest with gives Regions algorithm increase. the fold age, fifty any a Vietnamese, when of on example but 0.0024 For and country faces and dataset. Polish sex this same in the represented from countries are 24 the imposters between magnitude of orders two and globally. applies it matters shows same- Country-of-birth and section, same-sex, previous same-age, the of the results and the re-iterates setting, This anyone-with-anyone pairing. country pair zero-effort the between FMR in FMR increases pairing Restricted onre,5aegop n sexes 2 and groups age 5 countries, 0 Row also. 1 Row E T XEC ECH S . S . UMMARY UMMARY h vrg soe 88 M siae o nldn v ifrn ihnaeFRestimates FMR within-age different five including now estimates FMR 28880 over is average The : h vrg soe 70 M siae eetn ihn n ewe-ru siae o 24 for estimates between-group and within- reflecting estimates FMR 57600 over is average The : 9 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False hsosrainadtetpco eorpi ifrnil soitdwt na- with associated differentials demographic of topic the and observation This . FRVT o ayo h ifrn eeso eorpi arn hr sbtenone between is there pairing demographic of levels different the of many For : - AERCGIINVNO TEST VENDOR RECOGNITION FACE ihnec onr,teei oeta re fmgiueincrease magnitude of order than more a is there country, each Within : (24 2 cosagrtm fe h oetFRi bevdi Eastern in observed is FMR lowest the often algorithms Across : . 5 2 . 2 2 ) . - DEMOGRAPHICS 8 T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 31 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: used. is comparisons possible all if than smaller is scores matched demographically of set the and rates non-march false raise scores, thereby value, non-mate that and from thresholds, determined are raise they to because is increase uses Thresholds only this sponsors also. of it effect tests The in Trade and pairings. Affairs imposter Foreign same-sex of Department Australian algorithms the verification lines face NIST similar of reason Along evaluation this FRVT [16]. For its in zero-effort. results with accuracy paired “matched-covariate” are reporting imposters been when has measured from those than higher much are Discussion: 08:14:00 2019/12/19 inloii r oee oecmltl ntenx eto hc nldsrslsfrcmaio of comparison for results includes boundaries. which national section across next and the within individuals in completely more covered are origin tional E T XEC ECH S . S . T UMMARY UMMARY h eut bv hwta as ac ae o motrpiig nlkl elwrdscenarios real-world likely in pairings imposter for rates match false that show above results The hc ie proportion, a gives which , as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE FMR to bv threshold: above or at , T = Q ( s, 1 − FMR ) - DEMOGRAPHICS s i h uniefunction quantile the via , T  0 → → NR FNIR FNMR, M,FPIR FMR, → → Q 0 as , 1 (9) 32 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_country_x_same_same/imperial_002.pdf

+8. Same country, −3.77 −3.68 −3.56 −3.25 −3.14 −3.00 −2.87 −2.83 −2.99 −2.88 −2.82 −2.80 −2.87 −2.85 −2.75 −2.89 −2.80 −2.57 −2.36 −2.41 −2.34 −2.25 −2.16 −2.48 E T female, old XEC ECH S . S . +7. Same country, −4.06 −4.11 −4.21 −3.25 −3.30 −3.43 −3.23 −3.07 −3.23 −3.00 −3.10 −3.03 −3.10 −3.01 −2.98 −3.01 −2.54 −2.66 −2.83 −2.54 −2.58 −2.53 −2.33 −1.69

UMMARY sex, and age UMMARY

+6. Same country −4.36 −4.39 −4.45 −3.57 −3.57 −3.73 −3.46 −3.36 −3.49 −3.32 −3.29 −3.34 −3.28 −3.16 −3.30 −3.20 −2.71 −2.89 −3.01 −2.87 −2.90 −2.85 −2.62 −1.81 and sex Algorithm: imperial_002 Threshold: 1.381120 +5. Same country −4.34 −4.39 −4.45 −3.55 −3.59 −3.71 −3.52 −3.35 −3.51 −3.28 −3.39 −3.32 −3.38 −3.30 −3.26 −3.29 −2.82 −2.84 −3.05 −2.82 −2.86 −2.80 −2.61 −1.98 Dataset: Application and age Nominal FMR: 0.000030

log10 FMR +4. Same country −4.64 −4.68 −4.70 −3.87 −3.86 −4.01 −3.75 −3.65 −3.77 −3.61 −3.58 −3.62 −3.56 −3.45 −3.59 −3.47 −2.99 −3.10 −3.23 −3.16 −3.18 −3.12 −2.90 −2.09 0 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT

+3. Same sex −4.93 −4.89 −4.75 −4.25 −4.24 −4.06 −4.07 −4.00 −3.95 −3.86 −3.91 −3.85 −3.92 −3.90 −3.79 −3.93 −3.67 −3.49 −3.51 −3.44 −3.44 −3.35 −3.23 −3.13 and age - −1 AERCGIINVNO TEST VENDOR RECOGNITION FACE +2. Same sex −5.24 −5.19 −5.02 −4.55 −4.53 −4.34 −4.33 −4.30 −4.23 −4.17 −4.13 −4.13 −4.14 −4.10 −4.11 −4.13 −3.89 −3.78 −3.73 −3.75 −3.74 −3.65 −3.52 −3.33

+1. Same age −5.22 −5.17 −5.03 −4.54 −4.54 −4.35 −4.36 −4.29 −4.24 −4.14 −4.20 −4.14 −4.21 −4.19 −4.07 −4.21 −3.96 −3.77 −3.79 −3.72 −3.72 −3.63 −3.52 −3.41 −2

+0. Zero effort −5.52 −5.47 −5.30 −4.85 −4.82 −4.62 −4.62 −4.59 −4.51 −4.45 −4.42 −4.41 −4.42 −4.39 −4.39 −4.41 −4.17 −4.05 −4.01 −4.04 −4.02 −3.93 −3.80 −3.61 −3

−1. Diff age −5.73 −5.68 −5.49 −5.04 −5.00 −4.79 −4.79 −4.78 −4.68 −4.64 −4.56 −4.57 −4.55 −4.50 −4.58 −4.53 −4.30 −4.24 −4.15 −4.23 −4.21 −4.11 −3.97 −3.73

−4 How impostor is paired with enrollee impostor How −2. Diff country −5.69 −5.59 −5.35 −5.06 −5.00 −4.71 −4.80 −4.84 −4.57 −4.63 −4.57 −4.57 −4.46 −4.49 −4.55 −4.48 −4.47 −4.21 −4.09 −4.24 −4.21 −4.09 −4.01 −4.24

−3. Diff country −5.89 −5.79 −5.53 −5.24 −5.18 −4.87 −4.97 −5.02 −4.74 −4.82 −4.71 −4.72 −4.59 −4.61 −4.74 −4.59 −4.59 −4.39 −4.22 −4.43 −4.39 −4.27 −4.18 −4.34 and age −5

−4. Diff sex −6.00 −6.00 −6.00 −6.00 −6.00 −5.75 −5.96 −5.84 −5.70 −5.55 −5.62 −5.55 −5.64 −5.63 −5.49 −5.39 −5.34 −5.37 −5.17 −5.14 −5.07 −4.91 −4.87 −4.70 - DEMOGRAPHICS

−6 −5. Diff sex −6.00 −6.00 −6.00 −6.00 −6.00 −5.91 −6.00 −6.00 −5.85 −5.71 −5.74 −5.72 −5.76 −5.74 −5.67 −5.49 −5.46 −5.55 −5.31 −5.33 −5.27 −5.09 −5.05 −4.81 and age

−6. Diff country −6.00 −6.00 −6.00 −6.00 −6.00 −5.83 −6.00 −6.00 −5.74 −5.75 −5.75 −5.72 −5.68 −5.72 −5.63 −5.46 −5.57 −5.44 −5.22 −5.34 −5.26 −5.09 −5.07 −5.27 and sex

−7. Diff country, −6.00 −6.00 −6.00 −6.00 −6.00 −5.98 −6.00 −6.00 −5.90 −5.92 −5.87 −5.86 −5.80 −5.82 −5.81 −5.55 −5.70 −5.62 −5.36 −5.52 −5.46 −5.27 −5.24 −5.36 sex, and age T  6_Iran 6_Iraq 6_India 4_Haiti 1_Poland1_Ukraine 1_Russia 2_Mexico3_Nigeria 3_Liberia 3_Ghana 5_Kenya 7_Japan 7_Korea 7_China 0 4_Jamaica6_Pakistan 5_Ethiopia 7_Thailand 7_Vietnam5_Somalia 2_Nicaragua 2_El_Salvador 7_Phillippines Country of origin of enrollee → → NR FNIR FNMR, M,FPIR FMR, Figure 4: The heatmap shows FMR for each country-of-birth, when the imposter comparisons are drawn from increasingly demographically-matched individuals. Each cell depicts FMR on a logarithmic scale. The text value is log10(FMR) with large negative values encoding superior false match rates. The center row (“0. Zero effort”) row compares individuals without regard to demographics. Rows above that pair imposters more closely until, in the second row, the imposters →

→ are of the same sex, age and country of origin. The top row corresponds to one particular demographic often associated with the highest FMR values. The rows 0

1 below center pair for increasingly unlikely imposter pairings. For example “-5. Diff sex and age” shows FMR for imposters of different sex and age group. The countries appear in order of increasing mean FMR. Values below -6 are pinned to -6. Annex 8 contains the corresponding figure for all algorithms. 33 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: are values FMR Discussion: High blue. in shown ( applications. are value verification values a FMR in FMR by target concern Low represented the security connotes is 10. color base 0.0001 grey the of A to FMR red. log a in that i.e. shown so -4, scale, of logarithmic value a text uses a It and color heatmap. adopted a as one threshold. FMR The any shows at Figure similar thresholds. The very age recognition are trends of other The effect for 0.00003. the = visualization discuss FMR pages. We the a 250 to repeat similar. corresponds than are could results more we the to Likewise, - extends groups therefore later. age annex itself other The for visualization women. this and corresponding repeat contains men could We both 7 for Annex algorithms, algorithms. accurate all more for the figures of one for FMR procure cross-country shows do: 5 would Figure sex. imposters and age real-world same what the reflect of persons to from second, credentials identity and, effect, country-of-origin group, the age largest isolate the to in men just to demographics the restricted We (35 example. an is 5 Figure rates. match false 35. and Analysis: 20 between men women Polish Mexican example with for 65 groups, of age demographic produce the two can over from we individuals pairing, comparing demographic when same FMR of the measurement with rep- a individuals comparisons two many the Given for metadata photographs. group the age in and birth resented of country sex, by accompanied is comparison Each was sex which for female. or available, or not male was as birth listed of not country which for photographs of numbers small excluded We intervals the by and defined groups age the to FMR. assigned target was a photograph below Each or at FMR gave that 0.00003. value was lowest value the FMR as target mugshots selected The enforcement was law threshold the Each namely . images, 1 of of Annex set 10 set in a different of detailed over a set computed using a were made thresholds with comparisons The scores imposter thresholds. compared 400 070 those We 93 of algorithms each 4-6. Tableverification at estimates in 126 FMR listed produce with are to comparisons These thresholds of track. Verification set FRVT this the executed to We submitted yielding countries, comparisons. same the imposter from individuals billion different of 195.2 images 517 441 with countries 24 from images 019 442 countries Method: across and within rates match False 4.3 08:14:00 2019/12/19 − (65 50] E T XEC ECH − n hnrpae htfrwmn ermv e n g rmtedsuso o w esn:First, reasons: two for discussion the from age and sex remove We women. for that repeated then and , ,w compared we , 2 Annex in described corpus the from drawn portraits application quality high Using S . S . oadesteiseadesdi h il fti eto epoue grsdpcigcross-country depicting figures produced we section this of title the in addressed issue the address To 99] UMMARY UMMARY rmteFgr n hs nteanxs emk ubro bevtos is yassigning by First observations. of number a make we annexes, the in those and Figure the From . as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE log 10 0 - . 00 = 00003 DEMOGRAPHICS (00 − − 4 . 20] 5 .Hg M auspeeta present values FMR High ). , (20 T −  35] 0 , (35 → → NR FNIR FNMR, M,FPIR FMR, − 50] , (50 → − → 0 65] 1 34 , This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: algorithms. is all value for figure text corresponding The the to contains scale. used logarithmic is 7 a Annex is algorithm on threshold rates. reference FMR The match the depicts false columns. when cell respective superior Each rates the encoding in values positive everywhere. identified negative false value countries fixed shows the preset figure from a subjects the male to regions mid-aged seven of photos in single countries compare 24 For 5: Figure 08:14:00 2019/12/19 Demographics of impostor 2_El_Salvador 7_Phillippines 2_Nicaragua 7_Thailand 6_Pakistan 4_Jamaica 7_Vietnam 5_Ethiopia 5_Somalia 1_Ukraine 3_Nigeria 2_Mexico 1_Poland 1_Russia 3_Liberia 3_Ghana 5_Kenya 7_Japan 7_Korea 7_China E T 6_India 4_Haiti 6_Iran 6_Iraq XEC ECH

S . 1_Poland S . reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_country_x_country_only_male_35_50/imperial_002.pdf UMMARY UMMARY −6.00 −6.00 −6.00 −6.00 −4.47 −4.64 −4.55 −5.69 −5.89 −5.84 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.68 −5.37 −6.00 −6.00 −6.00

1_Russia −4.71 −4.67 −4.68 −5.44 −5.36 −5.18 −6.00 −6.00 −6.00 −6.00 −6.00 −5.95 −6.00 −5.65 −5.68 −5.18 −5.16 −5.74 −5.37 −5.20 −5.16 −5.64 −5.65 −5.52

1_Ukraine

2_El_Salvador −5.14 −5.66 −5.96 −6.00 −6.00 −6.00 −5.87 −6.00 −4.52 −4.68 −4.44 −5.44 −5.50 −5.86 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.94 −5.22 Nominal FMR:0.000030Sex: Mlog10FMR Threshold:1.381120Dataset:Application imperial_002 Algorithm: −3.41 −3.61 −6.00 −5.82 −6.00 −5.69 −5.64 −5.08 −5.75 −5.44 −4.98 −5.12 −4.89 −4.91 −4.87 −5.01 −5.28 −4.34 −4.51 −4.49 −5.64 −5.47 −5.60 −3.40

2_Mexico as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False −5.84 −5.44 −5.46 −3.45 −3.15 −3.65 −6.00 −6.00 −6.00 −5.99 −5.75 −5.03 −5.91 −5.36 −5.15 −5.05 −4.73 −4.95 −4.94 −5.03 −5.30 −4.46 −4.60 −4.64 2_Nicaragua FRVT −6.00 −5.20 −5.78 −3.61 −3.64 −3.64 −5.89 −5.66 −5.60 −5.65 −5.26 −5.02 −6.00 −5.39 −4.87 −5.17 −4.94 −4.81 −5.12 −5.48 −5.77 −4.39 −4.55 −4.64

3_Ghana - AERCGIINVNO TEST VENDOR RECOGNITION FACE −3.94 −4.97 −3.88 −4.49 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.90 −5.55 −5.99 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −3.21 −3.45 −3.37 −3.67

3_Liberia −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −3.43 −3.36 −3.47 −3.70 −4.05 −4.74 −3.91 −4.23 −6.00 −6.00 −6.00 −6.00 −6.00 −5.56 −6.00 −6.00 −6.00 −5.64

3_Nigeria −6.00 −6.00 −6.00 −6.00 −6.00 −5.63 −3.38 −3.49 −3.33 −3.73 −3.94 −4.85 −3.88 −4.34 −5.86 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.70 −5.78 −5.74

4_Haiti −3.91 −4.51 −4.01 −4.19 −5.84 −6.00 −6.00 −6.00 −6.00 −5.91 −6.00 −5.56 −5.56 −5.56 −6.00 −6.00 −6.00 −6.00 −6.00 −5.74 −3.67 −3.72 −3.69 −3.64 4_Jamaica Demographics ofenrollee −3.93 −4.59 −4.12 −4.36 −5.25 −6.00 −6.00 −5.95 −5.99 −5.71 −6.00 −5.45 −5.32 −5.30 −6.00 −6.00 −6.00 −5.66 −5.67 −5.40 −3.94 −3.96 −3.94 −3.91 5_Ethiopia −6.00 −6.00 −6.00 −5.05 −5.12 −4.90 −4.91 −4.84 −4.83 −4.66 −4.59 −2.88 −4.01 −2.96 −4.59 −5.84 −5.30 −5.23 −6.00 −5.93 −6.00 −5.53 −5.42 −5.65

5_Kenya −5.49 −6.00 −6.00 −6.00 −6.00 −6.00 −5.66 −3.90 −3.82 −3.86 −3.96 −4.19 −4.00 −3.40 −3.57 −5.18 −5.98 −6.00 −5.33 −6.00 −6.00 −6.00 −5.67 −5.55 5_Somalia −6 −5.67 −6.00 −5.77 −6.00 −5.37 −5.51 −5.70 −4.60 −4.41 −4.49 −4.28 −4.34 −2.91 −3.48 −2.17 −4.96 −6.00 −5.43 −5.55 −6.00 −6.00 −6.00 −5.43 −5.42

6_India - DEMOGRAPHICS −5.40 −6.00 −5.73 −6.00 −4.98 −5.06 −4.77 −6.00 −6.00 −5.98 −6.00 −5.51 −4.67 −4.90 −4.96 −3.58 −4.64 −4.79 −3.90 −5.46 −5.41 −5.60 −4.99 −5.33 −5

6_Iran −6.00 −5.48 −4.99 −5.32 −5.08 −4.95 −5.18 −6.00 −5.50 −6.00 −6.00 −6.00 −5.79 −5.85 −6.00 −4.63 −3.86 −4.11 −4.32 −6.00 −6.00 −6.00 −6.00 −6.00 −4

6_Iraq −6.00 −5.58 −4.91 −5.22 −4.97 −4.76 −4.99 −6.00 −5.74 −6.00 −6.00 −5.99 −5.43 −6.00 −5.62 −4.76 −4.08 −3.77 −4.38 −6.00 −6.00 −6.00 −6.00 −6.00 6_Pakistan −3 −6.00 −5.74 −5.42 −5.63 −4.88 −4.94 −4.89 −6.00 −6.00 −6.00 −6.00 −5.69 −5.21 −5.25 −5.46 −3.90 −4.25 −4.38 −3.79 −6.00 −6.00 −6.00 −5.76 −5.81

7_China −2 −3.22 −6.00 −4.96 −6.00 −4.94 −4.92 −5.02 −6.00 −6.00 −6.00 −5.81 −5.94 −6.00 −6.00 −6.00 −5.42 −6.00 −6.00 −5.85 −2.99 −3.44 −3.27 −3.61 −3.35

T 7_Japan  −3.70 −6.00 −5.07 −6.00 −5.01 −4.98 −5.40 −5.74 −6.00 −6.00 −6.00 −5.92 −6.00 −6.00 −6.00 −5.68 −5.77 −6.00 −6.00 −3.39 −3.05 −3.24 −3.95 −3.66 −1 0 7_Korea log 7_Phillippines −3.67 −6.00 −5.19 −6.00 −5.34 −5.39 −5.62 −5.81 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.72 −5.79 −6.00 −6.00 −3.22 −3.27 −2.97 −3.99 −3.66 → → 10 0 ( NR FNIR FNMR, M,FPIR FMR, FMR −3.23 −6.00 −5.61 −6.00 −4.27 −4.35 −4.37 −5.83 −5.97 −5.87 −5.44 −5.22 −5.52 −5.87 −5.86 −5.23 −6.00 −6.00 −5.50 −3.62 −3.94 −4.00 −2.89 −3.24 7_Thailand ) −3.13 −6.00 −5.21 −6.00 −4.52 −4.64 −4.66 −5.68 −5.67 −5.51 −5.51 −5.32 −5.45 −6.00 −5.50 −5.33 −6.00 −6.00 −5.50 −3.35 −3.67 −3.63 −3.24 −3.14 7_Vietnam ihlarge with → −2.83 −6.00 −5.20 −6.00 −4.50 −4.59 −4.56 −5.86 −5.67 −5.71 −5.37 −5.24 −5.63 −6.00 −5.56 −5.50 −6.00 −6.00 −5.80 −3.20 −3.73 −3.68 −3.21 −3.13 → 0 1 35 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: algorithms. all for figure is corresponding value the text to contains The used scale. is 7 logarithmic Annex algorithm a threshold rates. reference on The match FMR the columns. false depicts respective when superior cell the rates encoding Each in values positive everywhere. identified negative value false countries fixed shows the preset from figure a subjects to the female is regions mid-aged seven of photos in single countries compare 24 For 6: Figure 08:14:00 2019/12/19 Demographics of impostor 2_El_Salvador 7_Phillippines 2_Nicaragua 7_Thailand 6_Pakistan 4_Jamaica 7_Vietnam 5_Ethiopia 5_Somalia 1_Ukraine 3_Nigeria 2_Mexico 1_Poland 1_Russia 3_Liberia 3_Ghana 5_Kenya 7_Japan 7_Korea 7_China E T 6_India 4_Haiti 6_Iran 6_Iraq XEC ECH

S . 1_Poland S . reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_country_x_country_only_female_35_50/imperial_002.pdf UMMARY UMMARY −5.82 −5.58 −5.31 −5.28 −4.12 −4.35 −4.24 −5.14 −5.13 −5.30 −6.00 −6.00 −6.00 −6.00 −5.99 −5.86 −6.00 −6.00 −5.84 −5.06 −4.92 −5.41 −5.78 −6.00

1_Russia −4.27 −4.14 −4.20 −4.84 −4.94 −5.07 −6.00 −5.84 −6.00 −6.00 −5.96 −5.17 −5.91 −5.31 −5.48 −4.75 −4.62 −5.33 −4.52 −4.60 −4.58 −4.94 −4.88 −4.60

1_Ukraine

2_El_Salvador −4.71 −5.14 −5.67 −5.86 −5.97 −6.00 −5.76 −5.61 −4.24 −4.16 −4.05 −4.92 −4.91 −4.94 −5.84 −6.00 −6.00 −6.00 −5.99 −5.92 −5.85 −5.72 −5.46 −4.87 Nominal FMR:0.000030Sex: Flog10FMR Threshold:1.381120Dataset:Application imperial_002 Algorithm: −3.08 −3.10 −5.13 −5.15 −5.09 −4.86 −4.79 −4.43 −5.20 −4.50 −4.09 −4.32 −4.02 −3.92 −4.42 −4.63 −4.83 −3.75 −3.94 −3.78 −5.18 −4.86 −4.83 −2.94

2_Mexico as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False −5.20 −4.92 −4.99 −3.03 −2.96 −3.16 −5.17 −5.28 −5.22 −5.12 −4.95 −4.45 −5.19 −4.56 −4.28 −4.26 −3.99 −4.07 −4.51 −4.62 −4.84 −3.89 −4.12 −4.01 2_Nicaragua FRVT −5.20 −4.90 −4.95 −3.14 −3.25 −3.20 −5.02 −4.93 −5.03 −4.75 −4.68 −4.33 −5.01 −4.31 −4.16 −4.39 −4.15 −4.03 −4.73 −4.96 −5.07 −3.91 −4.16 −4.05

3_Ghana - AERCGIINVNO TEST VENDOR RECOGNITION FACE −3.29 −4.41 −3.40 −3.52 −5.47 −6.00 −5.89 −6.00 −5.06 −5.45 −5.79 −4.79 −4.94 −4.86 −6.00 −6.00 −5.89 −5.53 −5.57 −5.00 −2.70 −2.97 −2.87 −3.05

3_Liberia −6.00 −5.88 −6.00 −5.06 −5.45 −5.19 −2.92 −2.92 −3.03 −3.14 −3.32 −4.25 −3.40 −3.48 −5.34 −5.50 −5.14 −5.57 −5.11 −5.54 −5.55 −4.80 −5.01 −4.77

3_Nigeria −6.00 −6.00 −5.85 −5.15 −5.38 −5.01 −2.89 −3.07 −2.92 −3.17 −3.38 −4.48 −3.47 −3.68 −5.59 −6.00 −5.85 −5.65 −5.08 −5.66 −5.55 −4.81 −4.83 −4.84

4_Haiti −3.31 −4.08 −3.39 −3.44 −5.13 −6.00 −5.93 −5.27 −4.89 −5.48 −5.56 −4.50 −4.55 −4.54 −6.00 −5.99 −5.85 −4.81 −5.22 −4.68 −3.03 −3.16 −3.13 −3.02 4_Jamaica Demographics ofenrollee −3.37 −4.14 −3.61 −3.74 −4.98 −5.82 −5.48 −5.22 −5.22 −5.35 −5.74 −4.65 −4.86 −4.78 −6.00 −5.82 −5.86 −4.82 −5.02 −4.64 −3.28 −3.34 −3.36 −3.29 5_Ethiopia −6.00 −5.76 −5.65 −4.49 −4.46 −4.33 −4.38 −4.42 −4.46 −4.09 −4.13 −2.38 −3.36 −2.45 −4.17 −5.05 −4.56 −4.31 −5.39 −5.61 −6.00 −4.67 −4.84 −4.94

5_Kenya −4.89 −6.00 −5.46 −6.00 −5.27 −5.39 −5.12 −3.47 −3.44 −3.51 −3.43 −3.61 −3.42 −2.90 −2.97 −4.75 −5.56 −5.18 −4.67 −5.27 −5.78 −5.86 −5.08 −4.82 5_Somalia −6 −4.54 −6.00 −5.85 −6.00 −4.50 −4.53 −4.56 −3.79 −3.71 −3.84 −3.50 −3.82 −2.42 −2.78 −1.31 −3.99 −6.00 −4.36 −4.13 −4.85 −5.43 −5.30 −4.43 −4.35

6_India - DEMOGRAPHICS −4.73 −5.81 −5.57 −5.41 −4.13 −4.32 −4.13 −5.67 −5.22 −5.41 −5.06 −4.92 −4.23 −4.60 −4.22 −3.12 −4.37 −4.12 −3.27 −4.76 −5.06 −5.12 −4.57 −4.83 −5

6_Iran −5.28 −5.14 −4.78 −4.78 −4.31 −4.22 −4.40 −6.00 −5.18 −6.00 −6.00 −5.85 −5.10 −5.55 −5.62 −4.38 −3.31 −3.42 −3.95 −5.46 −5.19 −5.36 −5.31 −5.45 −4

6_Iraq −5.51 −4.87 −4.63 −4.64 −4.04 −4.03 −4.16 −6.00 −5.37 −5.87 −5.67 −5.53 −4.60 −5.09 −4.39 −4.11 −3.46 −3.06 −3.59 −6.00 −5.95 −6.00 −5.36 −5.63 6_Pakistan −3 −4.94 −5.58 −5.09 −5.06 −3.94 −4.11 −4.01 −5.62 −5.19 −5.47 −5.27 −5.01 −4.42 −4.80 −4.09 −3.32 −4.00 −3.62 −3.08 −5.48 −5.59 −5.75 −4.75 −4.97

7_China −2 −2.75 −5.65 −4.84 −5.52 −4.42 −4.56 −4.60 −5.13 −4.74 −5.10 −4.91 −5.10 −5.40 −5.09 −5.07 −5.14 −5.72 −5.99 −5.48 −2.56 −3.00 −2.82 −3.22 −3.02

T 7_Japan  −3.33 −5.78 −4.82 −5.73 −4.65 −4.61 −4.81 −5.56 −5.31 −5.65 −5.53 −5.46 −5.69 −5.67 −5.25 −5.44 −5.79 −5.98 −5.39 −3.01 −2.68 −2.83 −3.73 −3.42 −1 0 7_Korea log 7_Phillippines −3.16 −5.84 −4.70 −5.89 −4.85 −4.84 −5.03 −5.85 −5.58 −5.89 −5.58 −5.53 −5.76 −5.63 −5.54 −5.39 −5.86 −6.00 −5.51 −2.82 −2.83 −2.55 −3.79 −3.31 → → 10 0 NR FNIR FNMR, M,FPIR FMR, ( −2.81 −5.81 −5.38 −5.63 −3.76 −3.97 −3.81 −4.74 −4.59 −4.81 −4.51 −4.58 −4.65 −4.89 −4.38 −4.61 −5.39 −5.52 −4.80 −3.24 −3.74 −3.78 −2.56 −2.89 FMR 7_Thailand ) −2.66 −5.66 −5.18 −5.73 −3.93 −4.13 −4.08 −4.86 −4.87 −4.99 −4.62 −4.73 −4.92 −4.98 −4.57 −4.80 −5.49 −5.56 −4.94 −3.02 −3.40 −3.33 −2.87 −2.71 7_Vietnam ihlarge with → −2.27 −5.40 −5.17 −5.46 −3.77 −4.09 −3.90 −4.78 −4.63 −4.77 −4.52 −4.61 −4.88 −4.84 −4.58 −4.90 −5.21 −5.37 −4.90 −2.74 −3.28 −3.16 −2.75 −2.61 → 0 1 36 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: algo- all for figure corresponding the contains is value 7 text The Annex The scale. columns. rates. logarithmic respective match a the false on in FMR superior identified depicts encoding rithms. countries cell the values Each the when negative everywhere. from rates large value subjects positive fixed with male false preset mid-aged a shows of to figure photos is the threshold single regions compare seven to in used countries is 24 For 7: Figure 08:14:00 2019/12/19 Demographics of impostor 2_El_Salvador 7_Phillippines 2_Nicaragua 7_Thailand 6_Pakistan 4_Jamaica 7_Vietnam 5_Ethiopia 5_Somalia 1_Ukraine 3_Nigeria 2_Mexico 1_Poland 1_Russia 3_Liberia 3_Ghana 5_Kenya 7_Japan 7_Korea 7_China E T 6_India 4_Haiti 6_Iran 6_Iraq XEC ECH

S . 1_Poland S . reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_country_x_country_only_male_35_50/yitu_003.pdf UMMARY UMMARY −6.00 −6.00 −6.00 −6.00 −4.49 −4.76 −4.57 −5.75 −5.74 −5.84 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.75 −5.52 −5.92 −6.00 −6.00

1_Russia −4.83 −4.73 −4.74 −5.48 −5.70 −5.41 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.98 −5.56 −5.42 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00

1_Ukraine

2_El_Salvador −5.37 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −4.57 −4.65 −4.50 −5.44 −5.64 −5.63 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.36 Nominal FMR:0.000030Sex: Mlog10FMR yitu_003Threshold:37.785000Dataset:Application Algorithm: −3.73 −3.83 −6.00 −6.00 −6.00 −5.92 −5.74 −5.23 −6.00 −5.62 −5.49 −5.62 −5.28 −5.47 −6.00 −5.75 −6.00 −4.88 −5.09 −5.17 −5.72 −5.57 −5.55 −3.65

2_Mexico as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False −5.70 −5.51 −5.59 −3.74 −3.43 −3.87 −6.00 −6.00 −6.00 −5.96 −5.98 −5.25 −6.00 −5.72 −5.91 −5.44 −5.20 −5.58 −6.00 −6.00 −6.00 −4.98 −5.32 −5.54 2_Nicaragua FRVT −6.00 −5.45 −5.67 −3.86 −3.99 −3.87 −6.00 −6.00 −6.00 −5.70 −5.43 −5.42 −6.00 −5.70 −5.38 −5.74 −5.35 −5.27 −6.00 −6.00 −6.00 −4.96 −5.49 −5.29

3_Ghana - AERCGIINVNO TEST VENDOR RECOGNITION FACE −4.49 −5.23 −4.49 −5.04 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −3.73 −4.06 −3.93 −4.23

3_Liberia −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −4.05 −4.03 −4.12 −4.33 −4.49 −5.23 −4.49 −5.03 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00

3_Nigeria −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −3.99 −4.10 −3.96 −4.36 −4.50 −5.16 −4.52 −5.20 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00

4_Haiti −4.33 −4.89 −4.49 −4.73 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.63 −4.21 −4.34 −4.31 −4.10 4_Jamaica Demographics ofenrollee −4.25 −4.94 −4.57 −4.74 −5.78 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.98 −5.94 −6.00 −6.00 −6.00 −5.79 −5.77 −5.82 −4.46 −4.51 −4.47 −4.30 5_Ethiopia −6.00 −6.00 −6.00 −5.29 −5.42 −5.21 −5.32 −5.37 −5.29 −4.96 −4.88 −3.20 −4.36 −3.38 −5.43 −6.00 −5.73 −5.63 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00

5_Kenya −6.00 −6.00 −6.00 −6.00 −6.00 −5.94 −5.66 −4.51 −4.51 −4.52 −4.51 −4.58 −4.31 −3.85 −4.21 −5.79 −6.00 −6.00 −5.72 −6.00 −6.00 −6.00 −6.00 −6.00 5_Somalia −6 −6.00 −6.00 −6.00 −6.00 −5.70 −5.48 −5.80 −5.17 −5.20 −5.11 −4.82 −4.72 −3.43 −4.01 −2.60 −5.54 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00

6_India - DEMOGRAPHICS −5 −6.00 −6.00 −6.00 −6.00 −5.62 −5.58 −5.26 −6.00 −6.00 −6.00 −6.00 −6.00 −5.37 −5.43 −5.80 −4.35 −5.34 −5.62 −4.70 −6.00 −6.00 −6.00 −5.66 −5.85

6_Iran −6.00 −5.72 −5.43 −5.56 −5.55 −5.56 −5.85 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −5.94 −6.00 −5.37 −4.43 −4.69 −5.02 −6.00 −6.00 −6.00 −6.00 −6.00 −4

6_Iraq −6.00 −5.36 −5.19 −5.26 −5.30 −5.19 −5.42 −6.00 −6.00 −6.00 −6.00 −6.00 −5.73 −6.00 −6.00 −5.57 −4.68 −4.36 −5.01 −6.00 −6.00 −6.00 −6.00 −6.00 6_Pakistan −3 −6.00 −6.00 −5.57 −6.00 −5.43 −5.51 −5.62 −6.00 −6.00 −6.00 −6.00 −6.00 −5.82 −5.57 −6.00 −4.66 −4.93 −5.02 −4.47 −6.00 −6.00 −6.00 −6.00 −6.00

7_China Chinese-developed −2 −4.69 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −4.53 −4.74 −4.66 −5.02 −4.80

T 7_Japan  −4.93 −6.00 −6.00 −6.00 −5.97 −5.91 −6.00 −5.98 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −4.71 −3.86 −4.26 −5.01 −4.76 −1 0 7_Korea

7_Phillippines −5.16 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −6.00 −4.66 −4.27 −4.00 −5.33 −4.97 → → 0 NR FNIR FNMR, M,FPIR FMR, −4.21 −6.00 −6.00 −6.00 −4.88 −4.94 −4.82 −6.00 −6.00 −6.00 −6.00 −5.68 −5.84 −6.00 −6.00 −5.84 −6.00 −6.00 −6.00 −4.93 −4.85 −5.32 −3.53 −4.14 7_Thailand −4.24 −6.00 −6.00 −6.00 −5.31 −5.59 −5.26 −6.00 −6.00 −6.00 −6.00 −5.86 −6.00 −6.00 −6.00 −6.00 −6.00 −5.84 −5.89 −4.82 −4.85 −4.94 −4.09 −4.09 log 7_Vietnam algorithm 10 → −3.96 −6.00 −6.00 −6.00 −5.16 −5.52 −5.21 −6.00 −6.00 −6.00 −5.89 −6.00 −6.00 −6.00 −5.80 −6.00 −6.00 −6.00 −6.00 −4.72 −4.93 −5.11 −4.19 −4.23 ( → 0 FMR 1 37 ) This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 10 coarse show just likely will it printed when detail; as full the page reveals computer trends. in a single zooming on a and viewed on resolution When high heatmaps algorithms. very all all has across shows figure behavior which macroscopic show sheet” to “contact is be, a idea would includes The situation thumbnails. Annex ideal the Each what on meaning. comments the some on with and conclude and then We trends, common exceptions. following the notable note some now We then diagonal. the with along Africa regions within-region, East between are and results from interesting diagonal, individuals more the The around of FMR. photos low very comparing give algorithms when most that Europe, match Eastern true from false is those correlated it strongly example of For indicative structure region. block-diagonal within a rates particular in structure, block a see we regions following the into countries 08:14:00 2019/12/19 rsn w D ol o ess cosntoa onais .I diint ihFR hc sacuto ihipse crs the scores, imposter being high The of person algorithms. count all one small: a to is of absolutely applies which instances again not FMR, that high - but observation to errors study, an addition high, this In labelling persists very 3. ground-truth effect for also boundaries. The that small is national 2. score suspect be across similarity persist males. we’d may mean not 1974 and size would of - sample faces, images IDs 2632 The two Ethiopian involving present and comparisons 1. Somalian 116 733 1 comparing follows: from when as obtained anomalous is measurement is FMR result Somalia-Somalia this that discount We ...... Frmn loihs M ihnEsenErp scoet h nom- color, the grey to a close i.e. is rate Europe match Eastern false within target FMR inal algorithms, many For Europe: Eastern in FMR Nominal Vietnam Thailand, Philippines, Korea, Japan, China, - Asia East 7: Pakistan Iraq, Iran, India, - Asia South 6: Somalia Kenya, Ethiopia, - Africa East 5: Jamaica Haiti, - Caribbean The 4: Nigeria Liberia, Ghana, - Africa West 3: Nicaragua Salvador, El Honduras, Mexico, - America Central 2: Ukraine and Poland Russia, - Europe Eastern 1: nytefc xoe.Wieti ih rdc as oiie,hawa sams lasasn in detail. more absent in always observation almost the explain is to headwear needed leaving positives, is work ears false Further and produce men. hair might of the this photographs covers While typically exposed. that dress face head the of full only photos wearing of is majority substantial subject a the In women, reduced. Somalian somewhat although FMR, high give comparisons possi- Kenya those rejected but (in)significance statistical faces. or Somali data bilities of mislabeled comparison to for due is be FMR could highest this the suspected algorithms We all almost For Africa: East in FMR Higher USA. the and Europe Western China, in developed algorithms for E T XEC ECH S . S . UMMARY UMMARY 10 ute h M shg ihnEhoi n ewe tipaadSmla iial Kenya- Similarly Somalia. and Ethiopia between and Ethiopia within high is FMR the Further . as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE − 5 ≤ log 10 FMR − ≤ - DEMOGRAPHICS 4 hr r e xetost hs even this, to exceptions few are There . T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 38 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 ...... oia M nteS sa motrcomparisons. imposter Asian produce S. algorithm the IT’s Tiger on nor FMR Lookman nominal Neither Asia: South to apply not does known. dependency not Developer is 2017 data October training their in do of London entirety algorithms the in some is meeting that that Whether proof a nationals. of a in order reported is on Yitu included it data as training populations. its result, that those important in an FMR be higher to exhibit appears not This applies this however, also. Notably, value. Africa nominal West the the noted to near As are Asia attenuated. East are within FMR comparisons in differentials for demographic values FMR the that shows - Yitu- 7) the Figure for - Figure corresponding algorithm to The 003 FMR: it uniform more for produces algorithms China accurate most in the developed of One be population. to local algorithm the from an images for on sufficient increase FMR not the is mitigate producing it Alphaface, as Thus of such trend algorithm Sensetime. accurate common more and more of Deepglint developers the include exhibit These however, Asia. East algorithms, Chinese, across of FMR Chinese images elevated Other involving comparisons for Hybrid. in apparent Star only prominently is less effect e.g. but the cases applies, some same In the Asia. Tencent East Deepsea South SHU For and Yitu X-Laboratory, Academy). Dahua, Vision, Film Hik example, University Meiya, for (Shanghai Megvii, some - from population are Asian algorithms 7 East These Annex the 7. in on Figure FMR shown see reduced As much exhibit Chinese: China comparing in when developed algorithms FMR nominal give (e.g. algorithms blocks Chinese those Some between compared are individuals when southern FMR Vietnam). and and high, northern Korea into still divides often but block reduced, Asian and East with Vietnamese, The blocks other region. match the in strongly countries faces other Vietnamese the shown, all countries Asian with algorithm East the within FMR For high give them. to algorithms between for alone. and common region very is either faces. It Somali Asia: within or East in than Ethiopian FMR and lower Higher African West often of comparisons is to effect extend not The does FMR high Kenya. the individuals and However, of Africa faces West comparing when in occurs countries FMR Elevated from Africa: East and West between of FMR Higher faces Caribbean. comparing the when in those occurs with FMR Africa West Elevated in countries Caribbean: from the individuals and Africa West between occur FMR values Higher FMR high The Africa, borders. West any countries. in share between be and not to within do tend equally countries almost FMR These highest Nigeria. second and the Liberia with Ghana, countries i.e. The too: Africa West in FMR Higher E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE 10 9 htgah fa npcfid(oe)nme fChinese of number (lower) unspecified an of photographs - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 39 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: visualization. this of outcomes in several FMR note that We ones. extent European the Eastern shows the line in diagonal Each that the those exceeds above against countries 9. is Africa African point - West the the 10 within which Figures rates to any degree match of The false make plots Europe. show not scatter Eastern coordinates the did within Its We include we algorithm. unknown. effect one are to the shifts corresponds summarize point To these for effects. reasons the The explain to Europe. attempts Eastern in those to relative faces Discussion: 08:14:00 2019/12/19 . . . . ulttvl ifrn.Mn loihsd ieteepce M o ht e M .00 sseen as 0.00003 = FMR men 8. white Figure for FMR in this expected imagery the so on give fielded over portraits, do be illuminated algorithms measured to Many were well FMR different. algorithm are qualitatively an reflects photos if Figure especially of unwelcome, the sets be would Both but datasets across mugshots photos. instability application 1 2 Annex Annex using of set comparison was threshold The value. 100-fold than more Dispersion a is there women. African significantly to more men Much white from men. excursion over vertical women of FMR in increase four-fold x case Worst other many for seen variations FMR of kind while same case, the any exhibit In still accordingly. algorithms. they threshold present red, the would mostly set This are to informed heatmaps portable. not not the end-user are any thresholds to thus issue and operational images, an application fea- looks mugshots the time for for the distribution at imposter that the given to that being different is are consequence The they image. image the not of from mugshots kind extracted are - what tures images told of are kind algorithms different be- The a arise of portraits. may comparisons This application over comparisons. computed all was for used rates threshold match the false cause high indicating red, mostly are Sensetime- and algorithms PSL-001, 002 Panasonic SIAT-004, the for heatmaps cross-country on The figures: 1.4 the of West in example. Anomalies shift For for A increase, 25 baseline. digits. of above factor four magnitude a of common to orders corresponds the scale two logarithmic of and the one instead between PIN are values two-digit FMR a security the a using Africa, From to baseline. de-facto analogous the the is than than larger is, this times higher That 100 perspective order magnitude of Europe. is of that Eastern orders FMR to within two corresponding recorded often higher those are and values value FMR nominal African the East The large: are Magnitudes E T XEC ECH 0 = S . S . . UMMARY UMMARY 00012 so iepedeeaino as ac ae nAfrican in rates match false of elevation widespread a show 7 Annex of figures heatmap The teei lse fagrtm oae near located algorithms of cluster a is there 9 Figure women African for plot scatter the In oeagrtm,ms oal hs rmSneiegv M uhdfeett h target the to different much FMR give Sensetime from those notably most algorithms, Some and y 0 = as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False . FRVT 003 oprdt h agtFRvleof value FMR target the to Compared . - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS 0 . 00003 tevria ie hr sanear a is there line) vertical (the log T  10 0 M ausare values FMR → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 +2 40 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: and India improvement (in there corresponding developers a by see submitted were not in understand did important we We Bangladesh). algorithms is few effect inevitable. the not the for faces is of Asian population absence South do that for The China in in FMR Europeans. developed high Eastern algorithms implies to some it relative that that rates clear is match there is false picture it elevated The present, give is too. not pattern demographic same Asian the East the While for interesting. summaries more scatterplot the repeat 11 and 10 Figures 08:14:00 2019/12/19 E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 41 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: conduct multinationals some line - in diagonal developer men blue the white The of over domicile lines. annotation elsewhere. the green the originate identifies horizontal may in code likewise and noted color data vertical Training FMR The the elsewhere. the research by give “over/under”. indicated to show is algorithm (Ghana, to This countries each included African for database. West fixed mugshot three across U.S. is and threshold the within The men comparing Nigeria). obtained Liberia, FMR same-age against comparing Poland), when Ukraine, FMR shows (Russia, plot scatter The 8: Figure 08:14:00 2019/12/19 False match rate in men within W. Africa 8e−06 1e−05 2e−05 3e−05 5e−05 8e−05 1e−04 2e−04 3e−04 5e−04 8e−04 1e−03 2e−03 3e−03 5e−03 8e−03 1e−02 2e−02 3e−02 5e−02 E T amplifiedgroup_1 XEC ECH videonetics_1 S . S . 1e−06 ● ● isap_1 UMMARY UMMARY ● yisheng_4 everai_paravision_3 ● 3e−06 ● 5e−06 incode_3 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False megvii_1 FRVT ● nodeflux_1 ● 1e−05 tevian_4 upc_1 aware_4 ● ● visionlabs_6 ● - alphaface_1 3divi_4 ● AERCGIINVNO TEST VENDOR RECOGNITION FACE innovatrics_4 cyberextruder_2 meiya_1 x−laboratory_0 False matchrate inmenwithinE.Europe ● ● 3divi_3 hr_1 ● idemia_4 ● ● kedacom_0 aware_3 digitalbarriers_2 ● innovatrics_6 visionlabs_7 intellicloudai_1 intelresearch_0 anyvision_4 everai_2 ● anke_4 kakao_2 lookman_2 imperial_2 ● isystems_2 vocord_6 intellivision_2 megvii_2 ● id3_3 imagus_0 anke_3 ● winsense_0 isystems_1 ● tiger_2 shu_1 tiger_3 incode_4 alchera_1 pixelall_2 ● ● dahua_3 ● 3e−05 ● neurotechnology_5 neurotechnology_6 ● microfocus_1 synesis_5 saffe_1 rokid_0 ntechlab_7 gorilla_3 ● ● ● ● ● visionbox_1 ● anyvision_2 ● einetworks_0 ● ● intsysmsu_0 ● kneron_3 siat_2 cyberlink_2 tongyi_5 ● ● smilart_3 ● facesoft_0 microfocus_2 remarkai_0 ● nodeflux_2 ● ● ctbcbank_0 itmo_5 ● ● ● ● ● alchera_0 US mugshots ofwhitemen T setfor FMR=0.000030in ● ● ● dahua_2 itmo_6 ● veridas_1 deepglint_1 allgovision_0 ● ● saffe_2 tech5_2 f8_1 ulsee_1 ● visionbox_0 adera_1 id3_4 yitu_3 remarkai_1 ● dermalog_6 ● imperial_0 ● ● ● bm_1 ● gorilla_2 ● ● mt_0 ● iit_1 ● camvi_2 synesis_4 ● ● ● toshiba_2 ● ● ● ● ● ● ● veridas_2 ● psl_2 ● cognitec_0 5e−05 ● ● vion_0 ● cognitec_1 ● ● ● ● ● chtface_1 via_0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● uluface_2 ● vocord_7 ● ● realnetworks_3 deepsea_1 realnetworks_2 ceiec_2 toshiba_3 ● ● lookman_4 tech5_3 cogent_3 cogent_4 ● 1e−04 ● ● dsk_0 ● ● ● ● hik_1 ● starhybrid_1 ● ● ● ntechlab_6 vigilantsolutions_6 ● rankone_7 ● dermalog_5 vigilantsolutions_7 shaman_1 men ● intellifusion_1 ceiec_1 glory_0 ● glory_1 ● 3e−04 ● rankone_6 ● ● ● ● ● ihnadars he atr uoencountries European Eastern three across and within ● 5e−04 iit_0 camvi_4 - ● DEMOGRAPHICS ● 1e−03 3e−03 T sensetime_2 5e−03 siat_4 sensetime_1  ● 0 ● ● 1e−02 → → NR FNIR FNMR, M,FPIR FMR, Photos Application Dataset: ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● USA UK TW SK SG RU PT LI KR JP IS IN ID FR ES DE CN CH BG AU y → = → 0 x 1 42 is This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: same-age comparing when FMR shows plot scatter The 9: Figure 08:14:00 2019/12/19 sicue oso oe/ne” h oo oeietfistedmcl ftedvlpr-sm utntoasconduct multinationals some - developer white the over of line annotation elsewhere. domicile diagonal the originate blue the in The may identifies likewise lines. noted code green data FMR color Training horizontal The the and elsewhere. vertical research give “over/under”. the to by show indicated countries algorithm to is African included each This West is for database. three mugshot fixed across U.S. is and the threshold in within men The women comparing Nigeria). obtained Liberia, FMR (Ghana, against Poland), Ukraine, (Russia, tries False match rate in women within W. Africa 2e−05 3e−05 5e−05 8e−05 1e−04 2e−04 3e−04 5e−04 8e−04 1e−03 2e−03 3e−03 5e−03 8e−03 1e−02 2e−02 3e−02 5e−02 8e−02 1e−01 E T XEC ECH videonetics_1 S . S . 1e−06 ● UMMARY UMMARY amplifiedgroup_1 nodeflux_1 ● ● 3e−06 isap_1 5e−06 everai_paravision_3 ● as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT ● 1e−05 - False matchrate inwomen withinE. Europe AERCGIINVNO TEST VENDOR RECOGNITION FACE tevian_4 microfocus_1 ● incode_3 yisheng_4 ● 3e−05 visionlabs_6 ● microfocus_2 anyvision_4 ● US mugshots ofwhitemen T setfor FMR=0.000030in chtface_1 aware_4 aware_3 ● 3divi_4 innovatrics_4 5e−05 ● ● ● ● visionlabs_7 ● ntechlab_7 anyvision_2 bm_1 imperial_0 cyberextruder_2 ● upc_1 realnetworks_3 3divi_3 kakao_2 facesoft_0 imperial_2 saffe_1 ● megvii_1 realnetworks_2 ● itmo_5 meiya_1 megvii_2 ● ● everai_2 ● siat_2 ● ● ● kedacom_0 tiger_3 ● ● x−laboratory_0 lookman_2 tiger_2 idemia_4 alphaface_1 ● ● ● smilart_3 ● innovatrics_6 itmo_6 digitalbarriers_2 adera_1 intellivision_2 ● anke_3 ● ● ● ● ● cognitec_0 ● cognitec_1 ● tech5_2 neurotechnology_6 1e−04 synesis_5 deepsea_1 tongyi_5 gorilla_2 ● ● pixelall_2 ● intellicloudai_1 visionbox_0 alchera_0 ● anke_4 alchera_1 ● ● winsense_0 veridas_1 ● ● id3_3 visionbox_1 isystems_1 deepglint_1 vocord_6 incode_4 isystems_2 hr_1 shu_1 kneron_3 rokid_0 ● allgovision_0 cyberlink_2 neurotechnology_5 synesis_4 intsysmsu_0 yitu_3 dahua_3 gorilla_3 ● ceiec_1 einetworks_0 ● remarkai_0 uluface_2 ● nodeflux_2 ● ● ● ● ● ● ● toshiba_2 ● ● f8_1 ● ● remarkai_1 vion_0 ctbcbank_0 imagus_0 dahua_2 intelresearch_0 ● ● dermalog_6 ● ● saffe_2 ● ulsee_1 ● ● dsk_0 id3_4 ● vocord_7 ● ● ● ● ● ● ● camvi_2 ● ● ● ● ● ntechlab_6 ● ● ● ● ● tech5_3 iit_1 ● ● via_0 ceiec_2 ● ● ● ● ● ● mt_0 ● ● ● ● ● ● ● ● psl_2 ● ● ● shaman_1 ● ● veridas_2 ● hik_1 ● ● ● ● ● toshiba_3 dermalog_5 ● ● ● 3e−04 ● cogent_3 lookman_4 glory_1 women ● cogent_4 starhybrid_1 glory_0 ● ● ● ● 5e−04 ● intellifusion_1 rankone_7 vigilantsolutions_7 vigilantsolutions_6 ● ● iit_0 ● - ● rankone_6 ihnadars he atr uoencoun- European Eastern three across and within DEMOGRAPHICS ● 1e−03 ● ● camvi_4 ● 3e−03 5e−03 T sensetime_2 sensetime_1  siat_4 1e−02 ● 0 ● ● → → NR FNIR FNMR, M,FPIR FMR, Photos Application Dataset: ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● USA UK TW SK SG RU PT LI KR JP IS IN ID FR ES DE CN CH BG AU → → y 0 = 1 43 x This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: The lines. elsewhere. green originate horizontal may and the likewise vertical in data the Training noted by FMR elsewhere. indicated research the is conduct give This multinationals to database. (China, algorithm mugshot each countries line U.S. for Asian diagonal the fixed East blue in is six men threshold across white The and over Vietnam). within annotation and men Thailand comparing Philippines, obtained same-age Korea, FMR comparing Japan, against when Ukraine), FMR Russia, shows (Poland, plot tries scatter The 10: Figure 08:14:00 2019/12/19 False match rate in men within E. Asia 8e−07 1e−06 2e−06 3e−06 5e−06 8e−06 1e−05 2e−05 3e−05 5e−05 8e−05 1e−04 2e−04 3e−04 5e−04 8e−04 1e−03 2e−03 3e−03 5e−03 8e−03 1e−02 E T amplifiedgroup_1 XEC ECH videonetics_1 S . S . 1e−06 ● ● isap_1 UMMARY UMMARY ● yisheng_4 y everai_paravision_3 ● = 3e−06 x ● sicue oso oe/ne” h oo oeietfistedmcl ftedvlpr-some - developer the of domicile the identifies code color The “over/under”. show to included is 5e−06 incode_3 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False megvii_1 FRVT ● nodeflux_1 ● 1e−05 tevian_4 upc_1 aware_4 ● ● visionlabs_6 ● - alphaface_1 3divi_4 ● AERCGIINVNO TEST VENDOR RECOGNITION FACE innovatrics_4 cyberextruder_2 meiya_1 x−laboratory_0 False matchrate inmenwithinE.Europe ● ● 3divi_3 hr_1 ● idemia_4 ● ● kedacom_0 aware_3 digitalbarriers_2 ● innovatrics_6 visionlabs_7 intellicloudai_1 intelresearch_0 anyvision_4 everai_2 ● anke_4 kakao_2 lookman_2 imperial_2 ● isystems_2 vocord_6 intellivision_2 megvii_2 ● id3_3 imagus_0 anke_3 ● winsense_0 isystems_1 ● tiger_2 shu_1 tiger_3 incode_4 alchera_1 pixelall_2 ● ● dahua_3 ● 3e−05 ● neurotechnology_5 neurotechnology_6 ● microfocus_1 synesis_5 saffe_1 rokid_0 ntechlab_7 gorilla_3 ● ● ● ● ● visionbox_1 ● anyvision_2 ● einetworks_0 ● ● intsysmsu_0 ● kneron_3 siat_2 cyberlink_2 tongyi_5 ● ● smilart_3 ● facesoft_0 microfocus_2 remarkai_0 ● nodeflux_2 ● ● ctbcbank_0 itmo_5 ● ● ● ● ● alchera_0 US mugshots ofwhitemen T setfor FMR=0.000030in ● ● ● dahua_2 itmo_6 ● veridas_1 deepglint_1 allgovision_0 ● ● saffe_2 tech5_2 f8_1 ulsee_1 ● visionbox_0 adera_1 id3_4 yitu_3 remarkai_1 ● dermalog_6 ● imperial_0 ● ● ● bm_1 ● gorilla_2 ● ● mt_0 ● iit_1 ● camvi_2 synesis_4 ● ● ● toshiba_2 ● ● ● ● ● ● ● veridas_2 ● psl_2 ● cognitec_0 5e−05 ● ● vion_0 ● cognitec_1 ● ● ● ● ● chtface_1 via_0 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● uluface_2 ● vocord_7 ● ● realnetworks_3 deepsea_1 realnetworks_2 ceiec_2 toshiba_3 ● lookman_4 ● tech5_3 cogent_3 cogent_4 ● 1e−04 ● ● dsk_0 ● ● ● ● hik_1 ● starhybrid_1 ● ● ● ntechlab_6 vigilantsolutions_6 ● rankone_7 ● dermalog_5 vigilantsolutions_7 shaman_1 ● intellifusion_1 ceiec_1 glory_0 ● glory_1 ● 3e−04 ● men rankone_6 ● ● ● ● ● ● 5e−04 iit_0 camvi_4 - ihnadars he atr uoencoun- European Eastern three across and within ● DEMOGRAPHICS ● 1e−03 3e−03 T sensetime_2 5e−03 siat_4 sensetime_1  ● 0 ● ● 1e−02 → → NR FNIR FNMR, M,FPIR FMR, Photos Application Dataset: ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● USA UK TW SK SG RU PT LI KR JP IS IN ID FR ES DE CN CH BG AU → → 0 1 44 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: elsewhere. originate lines. green may horizontal likewise and data Training vertical the elsewhere. by research indicated noted conduct is FMR multinationals This the some give database. to line mugshot algorithm U.S. countries diagonal each the Asian blue in for The East men fixed six white is over across threshold annotation The and the Vietnam). within in same-age and women Thailand comparing comparing Philippines, obtained Korea, when Japan, FMR FMR (China, against shows Ukraine), Russia, plot (Poland, scatter countries The 11: Figure 08:14:00 2019/12/19 False match rate in women within E. Asia 8e−07 1e−06 2e−06 3e−06 5e−06 8e−06 1e−05 2e−05 3e−05 5e−05 8e−05 1e−04 2e−04 3e−04 5e−04 8e−04 1e−03 2e−03 3e−03 5e−03 8e−03 1e−02 2e−02 3e−02 E T XEC ECH videonetics_1 S . S . 1e−06 ● UMMARY UMMARY amplifiedgroup_1 nodeflux_1 ● ● 3e−06 y isap_1 = 5e−06 everai_paravision_3 ● x sicue oso oe/ne” h oo oeietfistedmcl ftedvlpr- developer the of domicile the identifies code color The “over/under”. show to included is as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT ● 1e−05 - False matchrate inwomen withinE. Europe AERCGIINVNO TEST VENDOR RECOGNITION FACE tevian_4 microfocus_1 ● incode_3 yisheng_4 ● 3e−05 visionlabs_6 ● microfocus_2 anyvision_4 ● US mugshots ofwhitemen T setfor FMR=0.000030in chtface_1 aware_4 aware_3 ● 3divi_4 innovatrics_4 5e−05 ● ● ● ● visionlabs_7 ● ntechlab_7 anyvision_2 bm_1 imperial_0 cyberextruder_2 ● upc_1 realnetworks_3 3divi_3 kakao_2 facesoft_0 imperial_2 saffe_1 ● megvii_1 realnetworks_2 ● itmo_5 meiya_1 megvii_2 ● ● everai_2 ● siat_2 ● ● ● kedacom_0 tiger_3 ● ● x−laboratory_0 lookman_2 tiger_2 idemia_4 alphaface_1 ● ● ● smilart_3 ● innovatrics_6 itmo_6 digitalbarriers_2 adera_1 intellivision_2 ● anke_3 ● ● ● ● ● cognitec_0 ● cognitec_1 ● tech5_2 neurotechnology_6 1e−04 synesis_5 deepsea_1 tongyi_5 gorilla_2 ● ● pixelall_2 ● intellicloudai_1 visionbox_0 alchera_0 ● anke_4 alchera_1 ● ● winsense_0 veridas_1 ● ● id3_3 visionbox_1 isystems_1 deepglint_1 vocord_6 incode_4 isystems_2 hr_1 shu_1 kneron_3 rokid_0 ● allgovision_0 cyberlink_2 neurotechnology_5 synesis_4 intsysmsu_0 yitu_3 dahua_3 gorilla_3 ● ceiec_1 einetworks_0 ● remarkai_0 uluface_2 ● nodeflux_2 ● ● ● ● ● ● ● toshiba_2 ● ● f8_1 ● ● remarkai_1 vion_0 ctbcbank_0 imagus_0 dahua_2 intelresearch_0 ● ● dermalog_6 ● ● saffe_2 ● ulsee_1 ● ● dsk_0 id3_4 ● vocord_7 ● ● ● ● ● ● ● camvi_2 ● ● ● ● ● ntechlab_6 ● ● ● ● ● tech5_3 iit_1 ● ● via_0 ceiec_2 ● ● ● ● ● ● mt_0 ● ● ● ● ● ● ● ● psl_2 ● ● ● shaman_1 ● ● veridas_2 ● hik_1 ● ● ● ● ● toshiba_3 dermalog_5 ● ● ● 3e−04 ● cogent_3 lookman_4 glory_1 ● cogent_4 starhybrid_1 glory_0 ● ● ● ● 5e−04 ● intellifusion_1 rankone_7 vigilantsolutions_7 vigilantsolutions_6 women ● ● iit_0 ● - ● rankone_6 DEMOGRAPHICS ● 1e−03 ● ● camvi_4 ihnadars he atr European Eastern three across and within ● 3e−03 5e−03 T sensetime_2 sensetime_1  siat_4 1e−02 ● 0 ● ● → → NR FNIR FNMR, M,FPIR FMR, Photos Application Dataset: ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● USA UK TW SK SG RU PT LI KR JP IS IN ID FR ES DE CN CH BG AU → → 0 1 45 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: smallest the is value The everywhere. value fixed a to set is each for threshold algorithms Two the columns. and and FMR rows for panel, rates gives respective positive each the that false in threshold show in heatmaps identified the one groups label, the sex used, a from and are photos labels race selected four randomly of of one with comparison tagged photos mugshot For 12: Figure 08:14:00 2019/12/19 Analysis: comparisons report not did We unknown. or unavailable age-group. was by sex or race 0.0001. which exactly FMR for is photographs target demographic excluded that The We made for comparisons FMR. FMR imposter target the mugshot design, 000 a 000 by 3 below Thus, of or males. set white at the for over FMR computed pairing. gave was threshold demographic that The each value 0.0001. for lowest was value estimates the FMR as produce selected to was in threshold threshold listed a set are Each with this These executed scores We track. compared Verification We Annex. FRVT the the 4-6. in to Tables described submitted is algorithms two labels verification by these 126 defined of with meaning comparisons demographics and of eight origin the The of races. each four for and comparisons sexes million 3 conduct to algorithm verification mugshots States United Method: in race on FMR of Dependence 4.4 algorithms. all for figure is value Demographics of impostor M−Am. Indian F−Am. Indian E T M−White M−Asian M−Black F−White XEC ECH F−Asian F−Black log S . S . ,w pl each apply we , 1 Annex detailed images mugshot the from portraits mugshot quality high Using F−Am. Indian 10 swt h nentoa e fapiainpoo,w s h eta oso cross-demographic show to heatmap the use we photos, application of set international the with As UMMARY UMMARY ( FMR −2.2 )

F−Asian cnan h corresponding the contains 6 Annex rates. match false superior encoding values negative large with −3.2 −2.9

F−Black ≤ Dataset: MUGSHOTS log10FMR −4.1 −4.1 −3.0 0 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False . 0001 FRVT

F−White −3.5 −4.1 −4.6 −3.7 imperial_002

M−Am. Indian ntewieml motr.Ec eldpcsFRo oaihi cl.Tetext The scale. logarithmic a on FMR depicts cell Each imposters. male white the on - AERCGIINVNO TEST VENDOR RECOGNITION FACE −3.1 −4.3 −5.1 −4.7 −2.3

M−Asian −4.4 −4.4 −5.1 −5.5 −3.6 −3.3

M−Black −5.2 −4.8 −4.4 −6.0 −4.6 −4.5 −3.6 Demographics ofenrollee −6

M−White −4.6 −6.0 −5.4 −5.1 −3.7 −4.3 −4.9 −4.0 −5

F−Am. Indian −2.6 −4

F−Asian - DEMOGRAPHICS −3.8 −3.6 −3 F−Black −4.5 −4.6 −3.5

F−White −2 −3.6 −4.2 −4.7 −3.7

M−Am. Indian yitu_003 −1 −3.5 −4.6 −5.4 −4.8 −2.6 T 

M−Asian 0 −4.9 −4.9 −5.3 −5.7 −3.9 −3.8 0 → → M−Black NR FNIR FNMR, M,FPIR FMR, −5.5 −6.0 −4.8 −6.0 −4.7 −4.7 −3.8

M−White −5.1 −5.1 −5.8 −4.9 −3.8 −4.4 −5.1 −4.0 → → 0 1 46 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 12 11 A red. in text shown are a values and FMR color High a blue. by in ( represented shown value are is FMR values 0.0001 target FMR of the Low connotes FMR 10. color a base grey the that to so log scale, i.e. logarithmic -4, a of shows uses value Figure It The 12. heatmap. Figure in a shown as are algorithms FMR two for Heatmaps cross-sex. including rates, match false 08:14:00 2019/12/19 Discussion: elevation. FMR worst-case by result the results the ordering summarizes algorithms, and all figures those for of elements diagonal within-race and within-sex the extracts 6 13 FigureAnnex in appear algorithms all for figures Corresponding applications. verification pcfial ntne f“n esnudrtoIs a as paetflepstvs htaeatal repositives. “American true maintains with actually who are “I” that Alaska Indian, letter positives, or “American false the States apparent as replace cause United group can we the IDs” this figures of two describes under datasets. the states person international which contiguous In “one the standard of 48 in instances India recognition”. EBTS the Specifically from of community the subjects from any or per distinguish in affiliation to “I” origins Indian” tribal letter having through with person identification group a cultural this or tags native, Alaskan NIST Eskimo, to supplied data The . . . fAinadBakwmn oeagrtm eeoe nCiardc hsdfeeta,frexample for differential, this 12. reduce Figure of China panel in mugshots right developed in the algorithms in FMR Yitu-003 Some in increases women. order-of-magnitude Black are and There Asian women: of Black and Asian in FMR Higher large such Why enrollments. 47. fingerprint by is labelling supported identity multiple database the from stem men, may In increase the errors of males. component a white One in known. i.e. not 0.0001 0.0068, is is of occur women increases FMR Indian the American over for increase FMR Indians fold the American 68 12 of Figure images in in featured occurs algorithm FMR Imperial-002 highest the the First, Indians: American in FMR with Highest than mugshots with lower is difference this false of higher magnitude photos. systematically The application give men. algorithms in than most women photos, in application rates with match As women: in FMR Higher E T XEC ECH S . S . UMMARY 12 UMMARY rmtefiue n hs nteanx emk ubro observations. of number a make we annex, the in those and figure, the From edson hspsiiiybcuetedtbs a tews xeln rudtuhintegrity, ground-truth excellent otherwise has database the because possibility this discount We . as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE log 10 0 . 01= 0001 − 4 .Hg M auspeetascrt ocr in concern security a present values FMR High ). - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → 11 → For . 0 1 47 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: hand right the in dots developed Algorithms purple (the women. Indian males American com- white for imposter panel. on usually lower same-race 0.0001 FMR, the and case = in worst same-sex FMR appear of for give China order rates in in to sorted match algorithm are false each algorithms for the The set give panel). is dots threshold the The algorithm, verification parisons. each For 13: Figure 08:14:00 2019/12/19 False match rate everai_paravision_003 neurotechnology_006 vigilantsolutions_006 vigilantsolutions_007 amplifiedgroup_001 cyberextruder_002 cyberextruder_001 intelresearch_000 realnetworks_003 x−laboratory_000 intellicloudai_001 videonetics_001 intellifusion_001 E T intellivision_001 intellivision_002 einetworks_000 innovatrics_004 innovatrics_006 allgovision_000 intsysmsu_000 visionlabs_007 visionlabs_006 alphaface_001 winsense_000 anyvision_004 anyvision_002 visionbox_000 visionbox_001 ctbcbank_000 cyberlink_002 XEC ECH remarkai_000 remarkai_001 isystems_001 deepsea_001 ntechlab_006 ntechlab_007 nodeflux_002 lookman_004 lookman_002 cognitec_000 cognitec_001 rankone_007 shaman_001 imperial_000 imperial_002 facesoft_000 yisheng_004 synesis_005 alchera_001 alchera_000 toshiba_002 toshiba_003 imagus_000 uluface_002 kneron_003 pixelall_002 cogent_004 cogent_003 smilart_003 vocord_006 incode_004 incode_003 idemia_004 dahua_003 aware_003 aware_004 gorilla_003 tongyi_005 kakao_002 everai_002 iqface_000 tevian_005 tevian_004 camvi_004 camvi_002 adera_001 tech5_003 tech5_002 ulsee_001 ceiec_002 anke_003 anke_004 3divi_003 3divi_004 itmo_005 itmo_006 isap_001 vion_000 upc_001 shu_001 dsk_000 sjtu_001 yitu_003 siat_002 id3_004 S . hik_001 psl_002 via_000 mt_000 S . hr_001 f8_001 iit_001 iit_000 UMMARY UMMARY 3e−05 ● ● 1e−04 ● ● ● 3e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● ● as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● FRVT ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● - ● F ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● AERCGIINVNO TEST VENDOR RECOGNITION FACE ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3e−03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−02 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3e−02 ● ● ● ● ● Algorithm 3e−05 ● ● ● 1e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3e−04 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● - ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● DEMOGRAPHICS ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● M ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 3e−03 ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● ● 1e−02 ● ● ● T 3e−02  0 ● ●

→ →

Developed in China in Developed Elsewhere Developed NR FNIR FNMR, M,FPIR FMR, erace ● ● ● ● White Indian Black Asian → → 0 1 48 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 oprn niiul rmtoaegop,freapePls e vrteaeo 5wt oihmnunder men Polish with 65 of age the over when men 20. FMR Polish of example measurement for groups, a age produce two can from we individuals comparing pairing, demographic same the with comparisons female. many or Given male was individuals as information listed two age not which the was for sex for photographs which metadata for of or group numbers unavailable small age excluded and We birth photographs. of the country in represented sex, by accompanied is comparison Each Links: groups age across and within individuals different of images 517 441 (00 with countries 24 from images 019 442 groups age certain on positives false more Method: yield algorithms all or some Do 4.5 cell Each algorithms. axes. all respective for heatmaps the figure the on corresponding regions given the distinct groups is contains from age value and text the 9 from The dataset Annex sex the scale. rates. same logarithmic in the a images of on of imposters FMR number for depicts high rates the match for false cross-age selected show countries six For 14: Figure 08:14:00 2019/12/19 hehl stesals au htfrwihteFRi esta reult .00.Ti a eetdfor repeated was This 0.00003. to equal or than The less . is 1 FMR FMR giving Annex the thresholds in which other detailed for 400 mugshots that 070 93 value the of smallest namely set the images, a is of over fixed threshold set computed a different was with threshold a compared The using are made rate. scores match comparisons many false imposter When the of score. estimate a an yield obtain comparison we Each threshold, track. -6. 4 Verification Tables FRVT the in to listed submitted are algorithms These verification 126 with comparisons of set this executed We Age group (12−20] (20−35] (35−50] (50−65] (65−99] (12−20] (20−35] (35−50] (50−65] (65−99] (12−20] (20−35] (35−50] (50−65] (65−99] −

(12−20] E T reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_age_x_age_special_country/imperial_002.pdf 20] −4.10 −4.57 −5.37 −6.00 −6.00 −3.99 −5.00 −6.00 −6.00 −6.00 −5.15 −5.53 −6.00 −6.00 −6.00 XEC ECH (20−35] , −4.65 −4.37 −4.50 −5.49 −5.91 −4.66 −4.51 −4.78 −5.59 −6.00 −5.71 −6.00 −6.00 −6.00 −6.00 ,w compared we , 2 Annex in described corpus the from drawn portraits application quality high Using S . S . (20 (35−50] 1_Poland UMMARY −5.30 −4.56 −4.12 −4.29 −5.03 −6.00 −4.72 −4.47 −4.86 −5.27 −6.00 −6.00 −6.00 −6.00 −6.00 UMMARY

(50−65] − −6.00 −5.46 −4.28 −3.56 −3.49 −6.00 −5.75 −4.70 −4.22 −4.40 −6.00 −6.00 −6.00 −5.78 −5.22

(65−99] 35] −6.00 −6.00 −5.02 −3.53 −3.02 −6.00 −6.00 −5.02 −4.31 −4.04 −6.00 −6.00 −6.00 −5.68 −4.84 , (12−20] (35 −2.79 −3.20 −3.96 −4.66 −5.38 −2.90 −3.35 −4.24 −5.22 −6.00 −3.87 −4.32 −4.73 −5.43 −5.92

(20−35] −3.19 −3.15 −3.45 −3.94 −4.91 −3.34 −3.20 −3.53 −4.26 −5.19 −4.58 −4.70 −4.76 −5.20 −6.00 −

(35−50] 2_Mexico 50] −3.86 −3.41 −2.96 −3.02 −3.65 −4.25 −3.55 −3.15 −3.37 −4.08 −5.36 −5.17 −4.59 −4.68 −4.89

as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False (50−65] FRVT −4.62 −4.04 −3.10 −2.68 −2.76 −4.90 −4.31 −3.36 −3.07 −3.23 −6.00 −5.90 −4.73 −4.30 −4.15 , { (50 (65−99] 0 −5.25 −4.96 −3.76 −2.79 −2.22 −5.84 −5.29 −3.98 −3.20 −2.84 −6.00 −6.00 −5.29 −4.26 −3.52 . 000001 - − (12−20] AERCGIINVNO TEST VENDOR RECOGNITION FACE −2.82 −3.46 −3.92 −4.65 −4.77 −2.86 −4.05 −4.76 −5.51 −6.00 −3.66 −4.48 −4.97 −6.00 −6.00 65] (20−35] −3.45 −3.18 −3.31 −3.88 −4.64 −4.01 −3.63 −3.74 −4.70 −5.59 −4.95 −4.91 −4.99 −5.50 −5.82 Demographics ofimpostorandenrollee , (35−50] , and 6_India −3.88 −3.33 −3.12 −3.18 −3.72 −4.79 −3.76 −3.58 −4.05 −4.67 −5.61 −5.16 −5.02 −5.41 −5.91 0

. (50−65] 000003 −4.75 −3.88 −3.20 −2.34 −2.36 −5.91 −4.69 −4.04 −3.51 −3.59 −6.00 −5.99 −5.19 −4.57 −4.55

(65 (65−99] −5.31 −4.66 −3.75 −2.34 −1.96 −6.00 −5.63 −4.79 −3.56 −3.04 −6.00 −6.00 −5.48 −4.32 −3.82 −

log (12−20] , −2.32 −3.02 −3.54 −4.08 −3.98 −2.66 −3.78 −4.41 −6.00 −6.00 −3.24 −3.90 −4.25 −4.98 −6.00 0 99] (20−35] 10 . 00001 −2.98 −3.01 −3.06 −3.51 −4.28 −3.61 −3.51 −3.57 −4.32 −4.98 −4.33 −4.57 −4.72 −5.42 −6.00 ( . (35−50] FMR 5_Kenya −3.57 −3.13 −2.90 −2.96 −3.55 −4.36 −3.65 −3.40 −3.60 −4.27 −5.22 −4.76 −4.53 −4.72 −4.53

(50−65] −4.14 −3.53 −2.93 −2.29 −2.32 −4.84 −4.18 −3.61 −3.31 −3.26 −6.00 −5.00 −4.45 −4.02 −3.85 ) ,

0 (65−99] ihlrengtv ausecdn ueirflematch false superior encoding values negative large with . −4.38 −3.94 −3.31 −2.18 −1.87 −6.00 −5.32 −3.86 −3.17 −2.74 −6.00 −5.36 −4.93 −3.63 −3.04 00003

(12−20] −3.14 −3.51 −3.86 −4.58 −5.54 −3.51 −4.15 −4.63 −5.18 −6.00 −4.40 −4.92 −5.43 −5.75 −6.00

(20−35] , - −3.51 −3.19 −3.19 −3.60 −4.24 −4.12 −3.60 −3.65 −4.12 −4.87 −4.90 −4.92 −5.00 −5.36 −5.57 0

(35−50] DEMOGRAPHICS . 3_Nigeria 0001 −3.89 −3.23 −2.92 −3.00 −3.41 −4.63 −3.65 −3.33 −3.50 −4.07 −5.25 −5.09 −4.79 −4.71 −4.87

(50−65] −4.31 −3.64 −2.99 −2.50 −2.39 −5.21 −4.12 −3.44 −3.10 −3.19 −5.78 −5.36 −4.72 −4.21 −3.95

(65−99] , −4.87 −4.48 −3.50 −2.43 −1.86 −6.00 −4.56 −3.91 −3.14 −2.61 −6.00 −5.36 −4.52 −3.68 −3.06 0 . 0003 (12−20] −2.63 −3.05 −3.62 −4.07 −4.21 −2.52 −3.06 −3.88 −4.70 −5.33 −3.48 −4.11 −4.72 −4.95 −5.10

(20−35] , −3.07 −2.90 −3.00 −3.41 −3.92 −3.08 −2.96 −3.27 −3.96 −4.72 −4.11 −4.18 −4.46 −4.72 −5.19

0 (35−50] . 7_China 001 −3.61 −2.98 −2.56 −2.58 −3.12 −3.96 −3.26 −2.99 −3.18 −3.71 −4.86 −4.50 −4.13 −4.05 −4.36

(50−65] T −4.06 −3.46 −2.61 −1.95 −1.94 −4.70 −3.94 −3.13 −2.68 −2.70 −5.23 −4.95 −4.21 −3.49 −3.34 ,  (65−99] 0 −4.41 −4.14 −3.19 −1.97 −1.47 −5.31 −4.87 −3.73 −2.72 −2.27 −6.00 −5.45 −4.62 −3.37 −2.77 .

0

003

Female−Female Male−Male Male−Female → → , 0 NR FNIR FNMR, M,FPIR FMR, . log10 FMR Nominal FMR:0.000030 Dataset: Application 1.381120 Threshold: imperial_002 Algorithm: 01 , −6 −5 −4 −3 −2 −1 0 0 . 03 } → . → 0 1 49 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: zero-effort all over 0.0001 of FMR a gives that is value algorithms. value the all text for to The figure fixed corresponding the scale. is contains logarithmic threshold a 10 The Annex on pairs. rates. imposters FMR imposter for match depicts rates match cell false false cross-age Each superior algorithm one encoding for sex. shows same heatmap the the countries, all of from photos visa For 15: Figure 08:14:00 2019/12/19 Age of impostor (72,120] (04,10) (10,16] (16,20] (20,24] (24,28] (28,32] (32,36] (36,40] (40,48] (48,56] (56,64] (64,72] (0,4] E T XEC ECH (0,4] −2.0 −2.8 −3.8 −5.0 −5.3 −5.5 −5.7 −5.9 −5.7 −6.0 −6.0 −6.0 −6.0 −6.0 S .

S . (04,10) UMMARY −2.7 −2.5 −3.1 −4.1 −4.6 −4.9 −5.0 −5.3 −5.2 −5.6 −5.7 −5.7 −5.9 −6.0 UMMARY

(10,16] −3.6 −2.9 −3.0 −3.7 −4.2 −4.5 −4.6 −4.8 −4.9 −5.2 −5.3 −5.3 −5.5 −6.0

(16,20] FMR(T) =0.0001globally.log10FMR Cross ageFMRatthresholdT=1.358forgiving imperial_002, algorithm −4.8 −4.1 −3.8 −3.8 −4.0 −4.2 −4.3 −4.5 −4.6 −4.9 −5.1 −5.4 −5.9 −6.0

(20,24] −5.1 −4.5 −4.2 −4.0 −4.0 −4.0 −4.1 −4.3 −4.4 −4.7 −5.0 −5.3 −5.7 −6.0

(24,28] −5.4 −4.7 −4.4 −4.1 −4.1 −3.9 −4.0 −4.1 −4.2 −4.5 −4.8 −5.1 −5.5 −5.9

as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False (28,32] FRVT All impostorpairs −5.5 −5.0 −4.6 −4.2 −4.1 −4.0 −4.0 −4.0 −4.0 −4.2 −4.5 −4.9 −5.3 −5.6

(32,36] −5.5 −5.1 −4.7 −4.4 −4.3 −4.1 −4.0 −3.9 −3.9 −4.0 −4.2 −4.6 −4.9 −5.4 -

(36,40] TEST VENDOR RECOGNITION FACE −5.4 −5.1 −4.9 −4.5 −4.4 −4.2 −4.1 −3.9 −3.8 −3.9 −3.9 −4.3 −4.7 −5.0

(40,48] −4.6 −5.7 −5.4 −5.1 −4.7 −4.6 −4.4 −4.2 −4.0 −3.8 −3.7 −3.7 −3.8 −4.1

(48,56] −3.9 −6.0 −5.6 −5.5 −5.1 −5.0 −4.7 −4.5 −4.2 −4.0 −3.8 −3.4 −3.4 −3.5

(56,64] −5.8 −5.8 −5.6 −5.4 −5.2 −5.1 −4.8 −4.6 −4.3 −4.0 −3.4 −3.2 −3.2 −3.4

(64,72] −6.0 −5.8 −5.7 −5.8 −5.4 −5.4 −5.2 −4.9 −4.6 −4.2 −3.5 −3.1 −3.0 −3.0

(72,120] Age ofenrollee −6 −6.0 −6.0 −6.0 −6.0 −6.0 −5.9 −5.6 −5.3 −4.9 −4.6 −3.7 −3.2 −3.0 −2.8

(0,4] −1.6 −2.3 −3.4 −4.3 −4.8 −4.9 −5.1 −5.3 −5.1 −5.5 −5.4 −5.8 −5.9 −6.0

(04,10) −5 −2.2 −1.9 −2.5 −3.4 −4.0 −4.2 −4.3 −4.7 −4.5 −5.0 −5.0 −5.1 −5.1 −6.0

(10,16] - −3.1 −2.3 −2.4 −3.0 −3.5 −3.7 −3.8 −4.1 −4.1 −4.5 −4.8 −4.6 −4.7 −5.3 DEMOGRAPHICS (16,20] −4.2 −3.5 −3.1 −3.0 −3.2 −3.3 −3.4 −3.6 −3.7 −4.0 −4.3 −4.6 −5.3 −5.2 −4

(20,24] −4.5 −3.9 −3.4 −3.1 −3.2 −3.1 −3.2 −3.4 −3.5 −3.8 −4.1 −4.4 −4.8 −5.7 Same sex andsameregionimpostorpairs

log (24,28] −4.8 −4.0 −3.6 −3.2 −3.2 −3.0 −3.1 −3.1 −3.3 −3.5 −3.8 −4.2 −4.6 −5.0 10

(28,32] −3 ( −4.9 −4.3 −3.8 −3.3 −3.3 −3.1 −3.0 −3.0 −3.1 −3.3 −3.5 −4.0 −4.4 −4.6 FMR

(32,36] −4.8 −4.4 −3.9 −3.5 −3.4 −3.2 −3.1 −3.0 −2.9 −3.1 −3.3 −3.7 −4.1 −4.6 )

T (36,40] ihlrengtv values negative large with −2  −5.3 −4.5 −4.2 −3.7 −3.5 −3.3 −3.1 −2.9 −2.9 −3.0 −3.1 −3.4 −3.8 −4.0

0 (40,48] −5.5 −4.7 −4.4 −3.8 −3.8 −3.5 −3.3 −3.1 −2.9 −2.9 −2.8 −3.0 −3.3 −3.7 → → (48,56] −5.5 −5.0 −4.8 −4.2 −4.1 −3.8 −3.5 −3.3 −3.0 −2.8 −2.5 −2.5 −2.7 −3.0 −1 NR FNIR FNMR, M,FPIR FMR,

(56,64] −5.8 −5.2 −5.0 −4.6 −4.5 −4.2 −3.9 −3.7 −3.4 −3.1 −2.5 −2.3 −2.3 −2.6

(64,72] −5.5 −5.4 −5.1 −5.0 −4.6 −4.5 −4.3 −4.0 −3.7 −3.3 −2.6 −2.2 −2.1 −2.2

(72,120] → → −6.0 −5.6 −6.0 −5.5 −5.2 −5.0 −4.7 −4.5 −4.0 −3.7 −2.9 −2.4 −2.2 −2.0 0 1 50 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: pages. 130 than Discussion: correspond- more contains to extends 9 therefore Annex and algorithms, algorithms. all accurate more for the Figures of ing one for FMR group cross-age shows 14 Figure similar are effects the because countries 24 the of 18 everywhere. and countries, Caribbean) an six (the region for is one results dropped includes 14 We Figure region. Figure The per algorithms. one operationally). accurate more interest the less of one of for are results showing they example, female- (although male-male, comparisons include We male-female also 4.3. section and in female, covered been have effects cross-country as only, within-country Analysis: 08:14:00 2019/12/19 . . . . Cmaio fiae fproso ifrn e sal rdcsvr low very above substantially produces and higher usually again FMR. sex is nominal FMR different the groups, of age oldest persons and youngest of the images within However, of FMR. Comparison at sex: age across by FMR out Lower broken effects using ageing children negative in false positives lapse. shows false time also for and enrolment, [28] report al. The et Michalski algorithm. by commercial reported one the those hide to they similar so are results one, These just not countries, all algorithms. from the comparisons of idiosyncrasies from geographic formed is estimate includes that FMR ) each 3 that Annex (see photographs visa groups of age dataset in smaller of individuals a consider images we of age comparison that particular, Below in 12. below men but sexes, both For the in group: persons age youngest often the is in this FMR men High of For images rates. of match comparison false countries, highest most the also. all produce true group from age women 65-and-over For the group: in age individuals oldest the in FMR Highest this but increases. scores difference imposter age the high be as produce generally difficult who will increasingly groups it be age result; will different aggregate from an individuals is obviously, some This, find group. to age possible origin match same false of the (better) countries in lower persons algorithms, yields groups for all age than different for rates in - persons cases of images all of almost comparison sexes, In both groups:and different in persons for FMR Lower E T XEC ECH S . S . oadesteiseo g epoue grsdpcigcosaeflemthrts ed this do We rates. match false cross-age depicting figures produced we age of issue the address To UMMARY UMMARY adtoei h ne,w aeteeobservations. these make we annex, the in those and 14 Figure From 12 − 20 g ru rdc ihflemthrts h aae osnticueaysubjects any include not does dataset The rates. match false high produce group age as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT (0 - AERCGIINVNO TEST VENDOR RECOGNITION FACE , 4] and (4 , 10] Note 15 . Figure of heatmap the in included are results The . - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 51 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 reports/11/figures/dhs_obim/cross_country/impostors/heatmap_fmr_age_x_country/imperial_002.pdf E T Algorithm: imperial_002 Threshold: 1.381120 Dataset: Application Nominal FMR: 0.000030 log10 FMR XEC ECH Impostors have same sex, age and country of birth

S . −6 −5 −4 −3 −2 −1 0 S . UMMARY UMMARY

F

−3.02 −3.11 −2.90 −2.38 −2.46 −2.42 −2.20 −2.22 −2.05 −1.97 −2.01 −1.96 −1.96 −1.93 −1.86 −1.93 −1.87 −1.70 −1.62 −1.63 −1.52 −1.47 −1.38 −0.78 (65−99]

−3.56 −3.62 −3.54 −2.86 −2.77 −2.97 −2.64 −2.68 −2.58 −2.49 −2.51 −2.34 −2.57 −2.47 −2.50 −2.31 −2.29 −2.19 −2.12 −2.07 −1.90 −1.95 −1.76 −0.98

as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False (50−65] FRVT

−4.12 −4.14 −4.05 −3.31 −3.20 −3.37 −3.06 −2.96 −2.94 −3.08 −3.02 −3.12 −2.92 −2.70 −2.92 −2.68 −2.90 −2.71 −2.56 −2.55 −2.38 −2.56 −2.27 −1.31 -

(35−50] TEST VENDOR RECOGNITION FACE

−4.37 −4.44 −4.37 −3.50 −3.50 −3.57 −3.39 −3.15 −3.29 −3.41 −3.35 −3.18 −3.10 −2.99 −3.19 −3.03 −3.01 −3.08 −2.89 −2.92 −2.49 −2.90 −2.77 −1.52 (20−35]

−4.10 −4.42 −4.03 −3.42 −3.36 −3.34 −3.26 −2.79 −3.05 −3.10 −3.26 −2.82 −2.89 −2.99 −3.14 −3.02 −2.32 −2.54 −2.81 −2.38 −2.33 −2.63 −2.75 −1.60 (12−20]

M Age group

−4.04 −4.32 −3.88 −3.25 −3.14 −2.97 −3.06 −2.84 −2.88 −2.98 −2.67 −3.04 −2.58 −2.97 −2.61 −2.57 −2.74 −2.42 −2.34 −2.33 −2.36 −2.27 −1.98 −1.82 (65−99]

−4.22 −4.32 −4.36 −3.65 −3.46 −3.50 −3.57 −3.07 −3.17 −3.65 −3.20 −3.51 −3.09 −3.09 −3.10 −2.93 −3.31 −2.91 −2.80 −2.82 −2.69 −2.68 −2.44 −2.06 - (50−65] DEMOGRAPHICS

−4.47 −4.67 −4.44 −3.86 −3.64 −3.93 −3.77 −3.15 −3.40 −3.79 −3.64 −3.58 −3.36 −3.21 −3.33 −3.05 −3.40 −3.14 −2.89 −2.97 −2.88 −2.99 −2.83 −2.17 (35−50]

−4.51 −4.50 −4.52 −3.82 −3.63 −4.02 −3.90 −3.20 −3.46 −3.86 −3.94 −3.63 −3.66 −3.42 −3.60 −3.19 −3.51 −3.05 −2.91 −2.81 −3.01 −2.96 −2.90 −2.12 (20−35] T

 −3.99 −3.83 −3.97 −3.36 −3.37 −3.53 −3.41 −2.90 −3.14 −3.24 −3.44 −2.86 −3.34 −3.32 −3.51 −2.75 −2.66 −2.49 −2.56 −2.37 −2.58 −2.52 −2.58 −2.01 (12−20] 0 → → NR FNIR FNMR, M,FPIR FMR, 6_Iran 6_Iraq 4_Haiti 6_India 7_Japan 5_Kenya 7_Korea 7_China 1_Poland 1_Russia 1_Ukraine 4_Jamaica 2_Mexico 6_Pakistan 3_Liberia 3_Ghana 3_Nigeria 7_Thailand 5_Ethiopia 7_Vietnam 5_Somalia 2_Nicaragua 2_El_Salvador 7_Phillippines Demographics of impostor and enrollee →

→ Figure 16: For application photos, the The heatmap shows one-to-one false match rates for same-sex, same-age and same-country of birth imposters, broken out 0

1 by age and country. The text value is log10(FMR) with large negative values encoding superior false match rates. Each cell depicts FMR on a logarithmic scale. 52 The text value is log10(FMR) with large negative values encoding superior false match rates. Annex 11 contains the corresponding figure for all algorithms. This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 13 particular at rates non-match false on centers ap- sections particular subsequent to related in FNMR are dicussion i.e. These thresholds, The 3.1. 2. section Figure in detailed in been plications have verification to appropriate metrics The Metrics 5.3 We demographics. across tests: verification rates one-to-one negative three false from results in recognition variation on this the base of quantification empirical gives section This Tests 5.2 algorithm. the to different appear face same the from two compared. when and occurs analyzed negative are false samples two a So operator: similar. differential insufficiently a are as implemented photographs is input recognition two face from that extracted Recall features below the score when comparison occur a will yield This individual one threshold. from a samples when systems biometric in occur negatives False Introduction 5.1 verification in differentials negative False 5 08:14:00 2019/12/19 onre fbrhadtoaegop oe/ne 5.I umrzscmaio fhg ult immigration quality high of comparison summarizes It 45). (over/under sexes groups two age and two and categories birth race of four pho- countries of mugshot each comparing for algorithms this accurate does most 52 It the tos. for rates non-match false the summarizes 17 Figure Results 5.4 frdsrpin fteiae n metadata. and images the of descriptions for 1 Annex See . . . ne iecntans nhg oueimgainevrnet.Tepoo hr rsn lsi pose algorithms. classic to present challenges there illumination photos and The environments. immigration volume high in constraints, time under photo: crossing Border - Application regions. global seven in countries four twenty from photo: . Application 1 - Annex Application see - images States United these with provided labels race and Mugshot - Mugshot E T XEC ECH S . S . UMMARY UMMARY ( T ) . ntefis etw okfrdmgahcefcsi h rusdfie ytesex the by defined groups the in effects demographic for look we test first the In : as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ecnie loahg ult aae olce rmsbet hailing subjects from collected dataset quality high a also consider We ,tebre rsigpoo r collected are photos crossing border the , 4 Annex in discussed As 13 tkstesm prahbtfr20 for but approach same the takes 18 Figure . - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 53 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 4 observations. Annex following the and make We 2 Annex in described are These photos. respectively. crossing border quality lower the with indicates side photos left application range the interquartile on the box spans small itself The imposter box all alphaface-001. values. dataset the over maximum this 0.00001 algorithms; and on = minimum those overall, FMR to algorithm over achieve best extend median verifi- to for here accurate the accuracy algorithm lines most is each the 52 box for and the set each algorithms) over was (26 within values threshold line FNMR The The of race. distribution comparisons. and the sex shows by figure the algorithms, comparisons, cation mugshot For 17: Figure 08:14:00 2019/12/19 Race Indian White Asian . . Black rsigevrnet nmght hsde o cu.I ete aei h aeaa fault. at camera the is case neither In occur. not border does the this in mugshots lighting In background bright environment. images to crossing quality due border often the measure individuals of skinned formally inspection dark but don’t of occurs, underexposure We this shows why determine subjects. to born order in with African brightness photos in or appliction contrast highest high-quality often comparing is when FNMR However, images, of black. border-crossing images as in listed FNMR lowest is the race mugshots, whose domestic In subjects subjects: American African and African in FNMR gender 2017-era the demographic of are algorithms. criticisms there classification motivated (correctly) extent that the those to than smaller Thus, much are lower. they magnitude differentials, of even algorithms order here, rates two an error are recognition [5], The algorithms, study time. middling that the from of In 35% almost females recognition. black face to gender in wrong the bias assigned of error gender-classification coverage the widespread than algorithm better spawned far best that are the rates rates error comparisons, These 1%. crossing below application-border FNMR gives difficult always exceptions almost more with 1% the below For generally is FNMR below. below 0.00001. discussed FNMR of give criterion algorithms FMR best stringent reasonably the the mugshots, at of 0.5% verification one-to-one In low: absolutely is FNMR E T XEC ECH reports/11/figures/fbi/ngi/for_fmr/fnmr_by_sex_age_country_all_algorithms.pdf S . S . 0.00 UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False False non−matchrate (FNMR):Distribution over 52mostaccurate algorithms FRVT Dataset: MUGSHOT FMR:0.000010Sex: - 0.01 AERCGIINVNO TEST VENDOR RECOGNITION FACE 0.02 - DEMOGRAPHICS Female 0.03 Male T  0 → → NR FNIR FNMR, M,FPIR FMR, 0.04 → → 0 1 54 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: The visionlabs-007. dataset values. this maximum on the and overall, algorithm minimum algorithms; best to those for over extend accuracy here median the to lines the indicates algorithm the is side each box and left for each algorithms) the set values on within (26 was box threshold FNMR line range small The The interquartile of group. the distribution comparisons. age spans the imposter and itself all show birth, box over boxplots of 0.00001 country the sex, = comparisons, by FMR photo algorithms, achieve crossing accurate most border 52 - the application over the For 18: Figure 08:14:00 2019/12/19 Country of birth 2_El_Salvador 2_El_Salvador 7_Phillippines 7_Phillippines 2_Nicaragua 2_Nicaragua 7_Thailand 7_Thailand 6_Pakistan 6_Pakistan 4_Jamaica 4_Jamaica 7_Vietnam 7_Vietnam 5_Ethiopia 5_Ethiopia 5_Somalia 5_Somalia E T 1_Ukraine 1_Ukraine 3_Nigeria 3_Nigeria 2_Mexico 2_Mexico 1_Poland 1_Poland 1_Russia 1_Russia 3_Liberia 3_Liberia 3_Ghana 3_Ghana XEC ECH 5_Kenya 5_Kenya 7_Japan 7_Japan 7_Korea 7_Korea 7_China 7_China 6_India 6_India 4_Haiti 4_Haiti 6_Iraq 6_Iran 6_Iraq 6_Iran S . S . UMMARY UMMARY reports/11/figures/dhs_obim/entry_to_visa/fnmr_by_sex_age_country_all_algorithms.pdf 0.00 Dataset: Applicationvs. BorderCrossingFMR:0.000010Sex: as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT False non−matchrate (FNMR): Distribution over 52mostaccurate algorithms - AERCGIINVNO TEST VENDOR RECOGNITION FACE 0.01 0.02 - DEMOGRAPHICS Female 0.03 T  Male 0 → → NR FNIR FNMR, M,FPIR FMR, 0.04 →

→ >45 0 <=45 1 55 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: report We constraint. time (implicit) some under environment border controlled less 4 Annex a with in photos next reference collected We application images 2 increase. crossing Annex to quality expected high of are comparison negatives the false for results degrades, consider quality image deliberate When with collected standards. mugshots, of quality high consideration to apply images enforcement law for results negative in . false evident The 12 trends Annex the in from appear differ characteristics be- may tradeoff differences algorithms error certain Full coarse that Figure. expose hides the to it intended doing is so algorithms In of groups. demographic number tween large a over results of aggregations These all for figure corresponding the contains 12 Annex to scores. related genuine uncertainty the indicates of symbol samples each bootstrap through of line 95% algorithms. vertical spans The it - differences groups. respectively, size tri- reveal, demographic false (circle, displacements sample between horizontal symbol vs. FMR and Each vertical non-match and comparisons. their FNMR false imposter - in threshold same-race show fixed and characteristics a same-sex tradeoff to for corresponds error computed square) the are angle, estimates images, FMR mugshot The verifying rates. algorithm match one For 19: Figure 08:14:00 2019/12/19 False non−match rate (FNMR) 0.002 0.003 0.005 0.008 0.010 0.020 0.030 0.050 0.001 . oee:I ey,Ngra aac e iehge NR hsapisi at n hn lobut also Ghana and Haiti over. in or applies 45 exceptions, This aged some FNMR. people are higher for the only give There so men - Jamaica covariate. the Nigeria, that verified unknown Kenya, possible correctly some In is prevalence still It however: relative verify. are to to women fail due algorithms of are where 98% differences comparisons perhaps of error 2% - than effect fewer marginal to confined a than is women is effect in this rates non-match that false Note higher give algorithms men. cases, most In FNMR: higher give Women E T XEC ECH 5e−06 S . S . UMMARY UMMARY 1e−05 ● 2e−05 3e−05 ● ● 5e−05 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT ● 1e−04 - AERCGIINVNO TEST VENDOR RECOGNITION FACE 2e−04 ● ● 3e−04 False matchrate (FMR) 5e−04 1e−03 ● 2e−03 ● 3e−03 5e−03 - 1e−02 DEMOGRAPHICS 2e−02 3e−02 5e−02 1e−01 T  0 → → imperial_002 Algorithm: MUGSHOT Dataset: FMR_white_male sex ● NR FNIR FNMR, M,FPIR FMR, White Black Asian Am. Indian T =1.2860 FMR =0.00100at T =1.3466 FMR =0.00010at T =1.3977 FMR =0.00001at Male Female → → 0 1 56 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: lower The scale. race. algorithms. linear and all below a for sex values figure on FMR same corresponding threshold scale. the the given logarithmic have contains a the imposters on at 15 The FMR FNMR Annex indicating pairs. indicates value. color imposter color with the the scores imposter all and shows scores over figure horizontal genuine 0.0001 The shows = distributions. figure FMR score upper similarity gives native The that show threshold plots the violin the shows mugshots, line verifying algorithm one For 20: Figure 08:14:00 2019/12/19

Nonmate Similarity Score Genuine similarity score 0.8 1.0 1.2 1.4 1.6 0.75 1.00 1.25 1.50 1.75 2.00 E T XEC ECH S . S . Am. Indian UMMARY Am. Indian UMMARY −3.3 0.0170 Asian Asian −4.4 0.0143 reports/11/figures/fbi/ngi/for_fmr/score_violins/imperial_002.pdf as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False Female Female FRVT Black - Black AERCGIINVNO TEST VENDOR RECOGNITION FACE 0.0036 −4.6 White White 0.0077 −5.0 Race Race Am. Indian Am. Indian 0.0121 −3.6 Asian - 0.0051 Asian DEMOGRAPHICS −4.8 Male Male Black 0.0021 Black −5.0 White T 0.0062  White −5.0 0 10 − 5 → → r indt that to pinned are NR FNIR FNMR, M,FPIR FMR, FNMR 1.437290 Threshold: imperial_002 Algorithm: FMR 0.004 0.008 0.012 0.016 −5 −4 −3 −2 → → 0 1 57 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: recognition. at attempt second a make could nuisance rejection to false invariant a inconvenience. experiencing drives are FNMR subject that of any algorithms magnitude applications, absolute many train The In to contrast. decade poor and last pose the non-frontal as over such efforts variables from stem rates error low The following: the note we figure these From ways: two in results 08:14:00 2019/12/19 • • • • • . . nArc n h aiba.Atrtoetorgos h ethgetFM si h atr Europe Eastern the in is FNMR highest next the regions, two those countries. After FNMR higher Caribbean. reveals the This and algorithms. accurate Africa most in fifty the over FNMR mean the of Caribbean: order in the appearing and Africa from subjects in FNMR Higher do differentials. statements demographic broad exhibit However, not algorithm. do algorithms by certain and that country mean by not varies small, often is under-45s is and Salvador over- El sex: in and FNMR age by i.e. patterns right clear the No to 22 Figure of Somalia. side in left that than the lower from always three almost or two of factor a by range 1.4%. countries: below is across FNMR variation whom for Lower 45, of age the under women outliers Somali has and algorithm Liberian Visionlabs-007 the for example, only For groups. demographic and countries all almost FNMR: for low give algorithms and accurate non-centered most compression. contrast, poor The to low due part are: in problems resolution, poor These and pose, images. non-frontal in- crossing faces, cropped problems border quality the the image of the intolerant in are herent algorithms some that shows range two-orders-of-magnitude algorithms: across variation Wide results. the skew not this did - algorithms algorithms poor countries. accurate from four most estimates 50 FNMR twenty the high all from that over FNMR so mean taken chosen of was being order statistic mean in sorted the are rows FNMR, figure The the mean 45. of of under columns the order and The is over in age FNMR sorted and Each are female figure and birth. the male of for of countries estimates all FNMR four and the algorithms of all mean for arithmetic results showing heatmap 22 Figure As and (above groups age two and sexes two for birth 45). of age country below by FNMR shows 14 Annex algorithm: Per E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ,tedfeecsbtenthe between differences the , 14 Annex of Figures the considering By as o-ac ae ag rmna .%u oaoe1% This 10%. above to up 0.1% near from range rates non-match False o h oeacrt loihs as o-ac ae generally rates non-match false algorithms, accurate more the For h otacrt loihsgvnFM eo 1% below FNMR given algorithms accurate most The - h eta scntutdwt countries with constructed is heatmap The DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 58 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: not have We cases. such in insight work. background to that leads the conducted invariably in yet cases lights failure ceiling of strong Inspection because For underexposure. underexposed scores. cause be matching might subject genuine be influences might height subjects subject the tall that by very steered possible example are is and it height particular, face fixed the In at toward mounted covariates. officer cameras unknown immigration with but collected it were influential images and to crossing work due border such incomplete the initiated be given yet would not analysis have such We that contrast. possible and need is would race work such such quantities but between [4] point correlation starting orienta- address initial head to an and be cropping, could of models regression presence mixed-effects exposure, tools, to under For images and tion. over digital of the areas from intensity, quantities contrast, related as properties. image such subject-specific of include measurement and with image- start of might analysis analysis that multivariate suggest some We require would the occur is effects it these everywhere; re-samples, Why value those fixed of a to 95% set span is line threshold two the The of compare estimate. ends to FNMR FMR The used gives the that is in scores. value uncertainty algorithm genuine lowest non-match of reference the false measure median the of the a gives resamples when giving box bootstrap square thereby rates The 2000 negative columns. over respective false the computed in shows rate identified figure countries the the from subjects countries of photos 24 For 21: Figure 08:14:00 2019/12/19 False non−match rate (FNMR) 0.01 0.02 0.03 0.00

1_Poland

1_Russia E T reports/11/figures/dhs_obim/entry_to_visa/fnmr_by_sex_age_country/imperial_002.pdf XEC ECH 2_El_Salvador1_Ukraine S . S . 2_Mexico 2_Nicaragua UMMARY UMMARY

3_Ghana

3_Liberia

3_Nigeria

4_Jamaica4_Haiti

5_Ethiopia

5_Kenya Female 5_Somalia ≤ 6_India as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT 0 6_Iran . 00001 6_Pakistan6_Iraq

7_China -

7_Japan TEST VENDOR RECOGNITION FACE cnan h orsodn gr o l algorithms. all for figure corresponding the contains 14 Annex . 7_Phillippines 7_Korea

7_Thailand

Country ofbirth Country 7_Vietnam

1_Poland

1_Russia

2_El_Salvador1_Ukraine

2_Mexico 2_Nicaragua

3_Ghana

3_Liberia

3_Nigeria

4_Jamaica4_Haiti

5_Ethiopia

5_Kenya

5_Somalia Male - 6_India DEMOGRAPHICS

6_Iran

6_Pakistan6_Iraq

7_China

7_Japan

7_Phillippines 7_Korea

7_Thailand

7_Vietnam T  Threshold 1.385590 FMR 0.000010 imperial_002 Algorithm Application vs. BorderCrossing Dataset: 0 >45 <=45 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 59 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: FMR populations gives is four that value value those lowest text where the The situations is is in scale. rates. everywhere; match value logarithmic FNMR value FNMR a represents fixed The on algo- so a columns. reference FNMR to 45, respective set the depicts the age is when in threshold over/under rates identified The and countries non-match balanced. men/women the false were from over verification subjects mean shows of figure the photos the two is compare regions to seven used in is countries rithm 24 For 22: Figure 08:14:00 2019/12/19

Algorithm E T everai_paravision_003 neurotechnology_006 neurotechnology_005 vigilantsolutions_007 vigilantsolutions_006 XEC ECH amplifiedgroup_001 didiglobalface_001 digitalbarriers_002 intelresearch_000 realnetworks_003 realnetworks_002 x−laboratory_000 intellicloudai_001 aiunionface_000 videonetics_001 intellifusion_001 intellivision_002 einetworks_000 innovatrics_004 innovatrics_006 sensetime_001 sensetime_002 intsysmsu_000 intsysmsu_001 visionlabs_007 visionlabs_006 starhybrid_001 alphaface_001 S . winsense_001 winsense_000 visionbox_001 visionbox_000 notiontag_000 S . deepglint_001 kedacom_000 ctbcbank_001 ctbcbank_000 cyberlink_003 cyberlink_002 synology_000 remarkai_000 remarkai_001 asusaics_000 deepsea_001 tuputech_000 ntechlab_007 ntechlab_006 nodeflux_002 nodeflux_001 lookman_004 lookman_002 samtech_001 cognitec_000 cognitec_001 trueface_000 rankone_007 rankone_006 shaman_001 shaman_000 pyramid_000 imperial_002 imperial_000 facesoft_000 yisheng_004 synesis_004 synesis_005 alchera_000 alchera_001 imagus_001 imagus_000 veridas_002 veridas_001 chtface_001 uluface_002 kneron_003 pixelall_003 pixelall_002 cogent_004 cogent_003 smilart_003 smilart_002 vocord_006 vocord_007 incode_005 incode_004 incode_003 idemia_005 idemia_004 awiros_001 ayonix_000 dahua_003 dahua_002 aware_004 aware_003 gorilla_004 gorilla_003 tongyi_005 kakao_002 kakao_001 everai_002 meiya_001 iqface_000 tevian_004 tevian_005 camvi_004 camvi_002 adera_001 tech5_003 tech5_002 UMMARY sertis_000 ulsee_001 ceiec_002 ceiec_001 UMMARY rokid_000 anke_004 anke_003 saffe_002 saffe_001 3divi_004 3divi_003 tiger_003 tiger_002 itmo_006 isap_001 vion_000 shu_001 upc_001 yitu_003 sjtu_001 siat_004 siat_002 dsk_000 bm_001 id3_004 id3_003 psl_002 psl_003 hik_001 via_000 mt_000 vd_001 hr_002 hr_001 f8_001 2_El_Salvador iit_001 iit_000

2_Nicaragua reports/11/figures/dhs_obim/entry_to_visa/fnmr_by_sex_age_country_all_algorithms_heatamp.pdf −2.8 −2.7 −2.6 −2.6 −2.6 −2.6 −2.5 −2.6 −2.6 −2.5 −2.6 −2.6 −2.6 −2.5 −2.5 −2.4 −2.5 −2.3 −2.4 −2.4 −2.4 −2.4 −2.4 −2.2 −2.5 −2.4 −2.4 −2.4 −2.4 −2.4 −2.3 −2.3 −2.2 −2.3 −2.1 −2.3 −2.4 −2.3 −2.2 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.3 −2.3 −2.3 −2.1 −2.2 −2.3 −2.2 −2.2 −2.1 −2.2 −2.2 −2.2 −2.1 −2.1 −1.9 −1.9 −2.0 −2.1 −2.2 −2.2 −2.0 −2.2 −2.1 −2.0 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −1.9 −2.0 −1.9 −2.0 −1.8 −2.0 −2.0 −1.7 −1.9 −1.7 −1.9 −1.8 −1.9 −1.6 −1.6 −1.8 −1.8 −1.9 −1.9 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.7 −1.6 −1.6 −1.3 −1.3 −1.5 −1.2 −1.6 −1.4 −1.4 −1.4 −1.5 −1.6 −1.3 −1.4 −1.1 −1.2 −1.5 −1.2 −1.1 −1.2 −1.3 −1.0 −1.0 −1.1 −1.1 −1.0 −0.9 −0.8 −0.8 −0.6 −0.8 −0.4 −0.4 −0.2 −0.2 −0.2 −0.1 −0.1 −0.0 0.0 0.0 0.0

7_Phillippines −2.5 −2.5 −2.5 −2.5 −2.5 −2.4 −2.4 −2.5 −2.4 −2.4 −2.5 −2.4 −2.5 −2.4 −2.4 −2.4 −2.4 −2.2 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.4 −2.3 −2.3 −2.3 −2.2 −2.2 −2.2 −2.1 −2.1 −2.2 −2.4 −2.2 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.2 −2.1 −2.1 −2.2 −2.0 −2.0 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.0 −1.9 −1.9 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 −2.1 −1.9 −2.0 −1.9 −2.0 −2.0 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −1.8 −1.8 −1.8 −1.8 −1.7 −1.7 −1.8 −1.7 −1.8 −1.8 −1.8 −1.8 −1.7 −1.7 −1.8 −1.7 −1.7 −1.6 −1.6 −1.5 −1.5 −1.2 −1.3 −1.4 −1.2 −1.5 −1.3 −1.3 −1.4 −1.3 −1.5 −1.3 −1.3 −1.2 −1.1 −1.2 −1.1 −1.0 −1.1 −1.1 −0.9 −1.1 −1.0 −1.0 −0.9 −0.8 −0.7 −0.7 −0.5 −0.7 −0.3 −0.3 −0.1 −0.2 −0.1 −0.1 −0.1 −0.0 −0.0 −0.0 0.0

7_Vietnam −2.5 −2.5 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.4 −2.3 −2.3 −2.3 −2.4 −2.2 −2.3 −2.3 −2.3 −2.3 −2.3 −2.1 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.2 −2.0 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.1 −2.1 −2.2 −2.2 −2.2 −2.1 −2.2 −2.1 −2.2 −2.2 −2.2 −2.1 −2.1 −2.1 −2.2 −2.1 −2.0 −2.1 −2.0 −1.9 −2.0 −2.1 −2.1 −2.1 −1.9 −2.1 −2.1 −1.9 −2.1 −1.9 −2.1 −2.0 −2.1 −2.0 −2.1 −2.0 −1.9 −2.0 −1.8 −2.0 −1.8 −1.9 −1.9 −1.6 −1.8 −1.6 −2.0 −1.9 −1.9 −1.6 −1.6 −1.8 −1.9 −1.9 −1.9 −1.7 −1.8 −1.9 −1.6 −1.8 −1.7 −1.7 −1.7 −1.6 −1.7 −1.6 −1.4 −1.6 −1.3 −1.6 −1.5 −1.5 −1.3 −1.5 −1.6 −1.5 −1.5 −1.3 −1.3 −1.6 −1.0 −1.3 −1.2 −1.5 −1.3 −0.8 −1.1 −1.1 −1.1 −0.8 −0.8 −0.9 −0.8 −0.9 −0.5 −0.5 −0.3 −0.3 −0.3 −0.2 −0.2 −0.0 −0.0 −0.0 0.0 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False −1.5 −2.0 −1.8 −1.9 −1.6 −1.5 −1.7 −1.7 −1.8 −1.9 −1.8 −1.7 −1.9 −1.5 −1.7 −1.6 −1.6 −1.8 −1.6 −1.7 −1.6 −1.3 −1.6 −1.3 −1.6 −1.4 −1.4 −1.3 −1.5 −1.6 −1.4 −1.4 −1.3 −1.3 −1.6 −1.0 −1.4 −1.2 −1.4 −1.3 −0.9 −1.1 −1.0 −1.0 −0.8 −0.8 −0.9 −0.9 −1.0 −0.5 −0.5 −0.4 −0.4 −0.4 −0.3 −0.2 −0.0 −0.0 −2.5 −2.5 −2.3 −2.4 −2.4 −2.4 −2.4 −2.4 −2.3 −2.4 −2.4 −2.4 −2.3 −2.4 −2.3 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.3 −2.3 −2.3 −2.2 −2.2 −2.3 −2.2 −2.1 −2.2 −2.0 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.2 −2.2 −2.1 −2.2 −2.1 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.2 −2.0 −2.0 −2.1 −2.1 −2.0 −2.0 −2.0 −1.9 −2.0 −2.1 −2.1 −2.1 −1.8 −2.1 −2.0 −1.9 −2.0 −1.8 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −1.7 −1.9 −1.8 −1.9 −1.9 −1.6 −1.7 0.0 0.0 FRVT

6_Iraq −1.6 −1.9 −1.7 −1.8 −1.5 −1.5 −1.7 −1.7 −1.8 −1.8 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.6 −1.7 −1.5 −1.5 −1.5 −1.2 −1.5 −1.1 −1.6 −1.3 −1.3 −1.3 −1.4 −1.6 −1.3 −1.2 −1.2 −1.2 −1.3 −1.4 −1.1 −1.2 −1.2 −0.8 −0.9 −1.0 −1.2 −1.0 −1.0 −0.9 −0.8 −0.6 −0.8 −0.3 −0.4 −0.2 −0.2 −0.2 −0.1 −0.1 −0.0 −2.7 −2.6 −2.3 −2.4 −2.4 −2.5 −2.3 −2.5 −2.5 −2.4 −2.3 −2.5 −2.4 −2.4 −2.3 −2.5 −2.4 −2.2 −2.3 −2.4 −2.3 −2.3 −2.2 −2.2 −2.4 −2.2 −2.3 −2.3 −2.2 −2.2 −2.3 −2.2 −2.1 −2.2 −2.0 −2.2 −2.3 −2.2 −2.2 −2.1 −2.1 −2.2 −2.2 −2.2 −2.2 −2.2 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −1.9 −2.1 −2.0 −2.0 −2.1 −1.9 −2.0 −2.0 −2.0 −2.1 −2.0 −1.8 −2.0 −1.9 −2.0 −1.9 −2.0 −2.1 −2.0 −1.8 −2.0 −2.0 −2.0 −2.0 −2.0 −1.8 −1.9 −1.6 −1.9 −1.9 −1.6 −1.7 0.0 0.0 0.0 7_China - −2.5 −2.5 −2.3 −2.3 −2.4 −2.4 −2.4 −2.4 −2.3 −2.4 −2.4 −2.4 −2.3 −2.3 −2.4 −2.3 −2.4 −2.4 −2.2 −2.3 −2.3 −2.3 −2.3 −2.1 −2.2 −2.3 −2.4 −2.2 −2.2 −2.2 −2.2 −2.2 −2.0 −2.2 −2.1 −2.2 −2.3 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.1 −2.2 −2.0 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.0 −2.0 −2.1 −2.0 −1.9 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −2.0 −1.9 −2.0 −1.7 −2.0 −1.9 −2.1 −1.9 −2.0 −1.9 −1.9 −1.9 −1.7 −1.9 −1.9 −1.9 −1.9 −1.7 −1.8 −1.7 −1.9 −1.9 −1.8 −1.7 −1.6 −1.8 −1.8 −1.9 −1.8 −1.7 −1.7 −1.9 −1.6 −1.7 −1.6 −1.6 −1.7 −1.6 −1.6 −1.8 −1.7 −1.6 −1.7 −1.5 −1.7 −1.7 −1.4 −1.4 −1.5 −1.6 −1.6 −1.3 −1.3 −1.6 −0.9 −1.4 −1.1 −1.4 −1.2 −0.9 −1.1 −1.0 −1.0 −0.7 −0.8 −0.8 −1.0 −1.0 −0.5 −0.4 −0.5 −0.4 −0.4 −0.4 −0.2 −0.0 −0.0 −0.0 0.0 AERCGIINVNO TEST VENDOR RECOGNITION FACE 6_Iran

7_Thailand −2.5 −2.5 −2.4 −2.4 −2.4 −2.4 −2.3 −2.4 −2.4 −2.3 −2.4 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.2 −2.1 −2.0 −2.1 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −1.9 −1.9 −2.0 −2.0 −1.8 −2.0 −1.9 −1.9 −1.9 −2.0 −2.0 −1.9 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −1.8 −1.8 −1.6 −1.9 −1.9 −1.6 −1.7 −1.6 −1.8 −1.7 −1.7 −1.6 −1.6 −1.7 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.5 −1.7 −1.6 −1.5 −1.6 −1.4 −1.5 −1.5 −1.3 −1.5 −1.2 −1.5 −1.4 −1.4 −1.3 −1.4 −1.5 −1.4 −1.4 −1.2 −1.2 −1.4 −1.3 −1.1 −1.2 −1.3 −0.9 −0.9 −1.1 −1.0 −1.0 −0.9 −0.9 −0.8 −0.6 −0.8 −0.3 −0.4 −0.2 −0.2 −0.2 −0.1 −0.1 −0.0 −0.0 0.0 0.0 −2.0 −2.1 −2.1 −2.1 −2.0 −2.0 −1.9 −1.9 −2.0 −2.0 −2.0 −2.0 −2.1 −2.0 −1.9 −2.0 −1.8 −2.1 −1.9 −2.1 −1.9 −2.0 −1.9 −1.8 −1.9 −1.7 −1.9 −1.8 −1.8 −1.8 −1.7 −1.9 −1.7 −2.0 −1.9 −1.8 −1.7 −1.7 −1.8 −1.8 −1.8 −1.8 −1.7 −1.6 −1.8 −1.6 −1.8 −1.6 −1.6 −1.7 −1.6 −1.6 −1.5 −1.5 −1.6 −1.4 −1.5 −1.6 −1.6 −1.4 −1.4 −1.5 −1.5 −1.6 −1.3 −1.3 −1.6 −0.9 −1.3 −1.1 −1.4 −1.2 −0.8 −1.1 −1.0 −1.0 −0.7 −0.7 −0.8 −0.7 −0.9 −0.5 −0.4 −0.3 −0.3 −0.3 −0.3 −0.1 −0.0 −2.5 −2.5 −2.3 −2.3 −2.3 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.2 −2.3 −2.1 −2.2 −2.3 −2.3 −2.3 −2.2 −2.2 −2.2 −2.1 −2.1 −2.2 −2.0 −2.2 −2.2 −2.2 −2.0 −2.2 −2.2 −2.1 −2.1 −2.0 −2.2 −2.1 −2.1 −2.2 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 2_Mexico 0.0 0.0 0.0

5_Ethiopia −2.5 −2.5 −2.4 −2.4 −2.4 −2.4 −2.3 −2.4 −2.4 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.2 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.2 −2.2 −2.3 −2.2 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.0 −1.9 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −1.9 −1.9 −2.0 −1.9 −1.9 −2.0 −1.9 −2.0 −2.0 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −2.0 −1.8 −1.9 −1.9 −1.8 −1.9 −1.7 −1.8 −1.6 −1.8 −1.8 −1.5 −1.7 −1.5 −1.8 −1.7 −1.7 −1.5 −1.5 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.6 −1.5 −1.7 −1.6 −1.5 −1.5 −1.3 −1.4 −1.4 −1.3 −1.4 −1.2 −1.4 −1.4 −1.4 −1.2 −1.3 −1.5 −1.3 −1.4 −1.1 −1.1 −1.4 −1.1 −1.0 −1.1 −1.2 −0.9 −0.9 −1.0 −1.1 −1.0 −0.8 −0.8 −0.7 −0.5 −0.7 −0.3 −0.3 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.0 −0.0 0.0 −2.0 −2.0 −2.0 −1.9 −1.9 −2.0 −1.9 −2.0 −2.0 −1.9 −2.0 −1.9 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −1.9 −1.7 −1.9 −1.8 −1.9 −1.8 −1.8 −1.9 −1.8 −1.7 −1.9 −1.9 −1.8 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.7 −1.6 −1.6 −1.7 −1.7 −1.5 −1.7 −1.6 −1.6 −1.5 −1.6 −1.6 −1.5 −1.6 −1.5 −1.4 −1.3 −1.4 −1.3 −1.4 −1.2 −1.2 −1.3 −1.3 −1.4 −1.2 −1.2 −1.2 −1.1 −1.0 −1.2 −1.1 −1.2 −0.8 −1.2 −0.9 −1.0 −1.0 −0.8 −0.9 −0.8 −0.7 −0.6 −0.6 −0.3 −0.3 −0.1 −0.1 −0.2 −0.1 −0.0 −0.0 −0.0 −0.0 −2.5 −2.5 −2.5 −2.3 −2.3 −2.3 −2.4 −2.4 −2.3 −2.3 −2.2 −2.3 −2.3 −2.3 −2.4 −2.4 −2.3 −2.2 −2.2 −2.3 −2.2 −2.2 −2.2 −2.1 −2.2 −2.2 −2.2 −2.2 −2.3 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.0 −2.0 −2.1 −2.0 −2.1 −2.0 −2.0 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 0.0 7_Korea Country ofbirth Country −2.0 −2.0 −1.9 −1.9 −2.0 −2.0 −1.9 −2.0 −1.9 −2.0 −1.9 −1.8 −2.0 −1.9 −1.7 −1.9 −1.6 −2.0 −1.8 −2.1 −1.8 −1.9 −1.8 −1.7 −1.8 −1.6 −1.7 −1.9 −1.7 −1.7 −1.7 −1.7 −1.7 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.8 −1.9 −1.6 −1.7 −1.5 −1.7 −1.5 −1.7 −1.5 −1.5 −1.6 −1.4 −1.5 −1.6 −1.7 −1.5 −1.7 −1.3 −1.7 −1.6 −1.4 −1.3 −1.4 −1.5 −1.6 −1.3 −1.1 −1.5 −0.8 −1.3 −1.0 −1.4 −1.1 −0.9 −1.1 −0.9 −1.0 −0.6 −0.7 −0.7 −0.9 −0.9 −0.4 −0.4 −0.5 −0.3 −0.3 −0.4 −0.1 −0.0 −0.0 −0.0 −2.5 −2.4 −2.1 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.3 −2.3 −2.2 −2.2 −2.3 −2.2 −2.3 −2.3 −2.2 −2.2 −2.2 −2.2 −2.3 −2.2 −2.1 −2.2 −2.4 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −1.9 −2.1 −2.1 −2.1 −2.0 −2.1 −2.0 −2.0 −1.9 0.0 7_Japan

log 6_Pakistan −2.5 −2.4 −2.2 −2.3 −2.3 −2.4 −2.3 −2.3 −2.2 −2.3 −2.3 −2.3 −2.2 −2.2 −2.3 −2.2 −2.3 −2.3 −2.1 −2.2 −2.2 −2.1 −2.2 −2.2 −2.1 −2.2 −2.3 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.0 −2.1 −2.1 −2.1 −2.0 −2.0 −2.2 −2.0 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −2.0 −2.1 −1.9 −2.0 −2.0 −2.0 −1.9 −2.0 −1.9 −1.9 −1.9 −2.0 −1.9 −2.0 −2.0 −1.9 −1.9 −1.9 −1.8 −1.9 −1.9 −1.8 −1.9 −1.6 −2.0 −1.8 −2.1 −1.8 −1.9 −1.8 −1.7 −1.8 −1.7 −1.7 −1.9 −1.7 −1.7 −1.8 −1.7 −1.7 −1.8 −1.9 −1.6 −1.8 −1.7 −1.8 −1.8 −1.9 −1.6 −1.7 −1.5 −1.7 −1.6 −1.6 −1.4 −1.4 −1.6 −1.4 −1.4 −1.5 −1.7 −1.4 −1.7 −1.3 −1.7 −1.6 −1.4 −1.2 −1.4 −1.5 −1.5 −1.2 −1.1 −1.5 −0.8 −1.1 −1.0 −1.4 −1.1 −0.9 −1.1 −0.9 −1.0 −0.6 −0.7 −0.7 −0.8 −0.8 −0.4 −0.4 −0.4 −0.3 −0.3 −0.3 −0.1 −0.0 −0.0 −0.0 0.0 10 −2.4 −2.4 −2.2 −2.3 −2.3 −2.3 −2.2 −2.3 −2.3 −2.2 −2.2 −2.3 −2.2 −2.3 −2.3 −2.3 −2.3 −2.1 −2.2 −2.2 −2.2 −2.2 −2.1 −2.1 −2.2 −2.1 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.0 −2.1 −2.2 −2.1 −2.1 −1.9 −2.0 −2.1 −2.0 −2.0 −2.1 −2.1 −2.1 −2.0 −1.9 −2.0 −2.1 −2.0 −2.0 −1.9 −1.8 −2.0 −2.0 −1.8 −1.8 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −1.8 −1.9 −1.8 −1.9 −1.8 −1.9 −1.9 −1.8 −1.7 −1.8 −1.8 −1.9 −1.9 −1.8 −1.7 −1.8 −1.7 −1.9 −1.9 −1.5 −1.6 −1.5 −1.8 −1.6 −1.8 −1.4 −1.5 −1.6 −1.6 −1.6 −1.7 −1.7 −1.6 −1.7 −1.6 −1.6 −1.6 −1.6 −1.6 −1.5 −1.5 −1.3 −1.2 −1.4 −1.2 −1.5 −1.2 −1.2 −1.2 −1.4 −1.4 −1.2 −1.2 −1.1 −1.2 −1.3 −1.3 −1.1 −1.3 −1.1 −1.0 −0.9 −1.1 −1.0 −0.9 −1.0 −0.8 −0.8 −0.6 −0.8 −0.4 −0.5 −0.2 −0.2 −0.2 −0.2 −0.1 −0.0 −0.0

1_Ukraine 0.0 0.0 ( FNMR −2.0 −1.9 −1.8 −1.9 −1.9 −1.8 −1.9 −2.0 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −2.1 −1.7 −1.8 −1.8 −1.8 −1.9 −1.7 −1.7 −1.8 −1.7 −1.8 −1.8 −1.7 −1.8 −1.7 −1.7 −1.6 −1.7 −1.8 −1.6 −1.7 −1.7 −1.5 −1.7 −1.6 −1.5 −1.7 −1.6 −1.5 −1.4 −1.4 −1.3 −1.4 −1.5 −1.6 −1.3 −1.5 −1.3 −1.5 −1.5 −1.4 −1.1 −1.4 −1.4 −1.5 −1.2 −1.0 −1.4 −1.0 −0.9 −0.9 −1.3 −0.8 −0.9 −0.8 −0.8 −0.9 −0.7 −0.8 −0.6 −0.5 −0.6 −0.2 −0.3 −0.2 −0.1 −0.1 −0.1 −0.1 −0.0 −2.4 −2.4 −2.5 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.4 −2.2 −2.2 −2.2 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.2 −2.1 −2.2 −2.2 −2.3 −2.1 −2.2 −2.1 −2.0 −2.0 −2.0 −2.0 −2.1 −2.1 −2.2 −2.0 −2.1 −2.2 −2.0 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −2.1 −2.0 −1.8 −2.0 −2.0 −2.0 −1.9 1_Poland 0.0 0.0 0.0 −1.9 −1.9 −1.8 −1.9 −1.9 −1.9 −1.8 −1.9 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −2.0 −1.7 −1.8 −1.8 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.7 −1.8 −1.7 −1.8 −1.7 −1.6 −1.6 −1.7 −1.7 −1.6 −1.7 −1.7 −1.5 −1.6 −1.6 −1.5 −1.7 −1.6 −1.5 −1.4 −1.4 −1.2 −1.3 −1.3 −1.6 −1.3 −1.5 −1.2 −1.4 −1.4 −1.4 −1.1 −1.3 −1.3 −1.4 −1.1 −1.0 −1.2 −1.0 −0.8 −0.9 −1.2 −0.8 −0.9 −0.8 −0.8 −0.8 −0.7 −0.8 −0.6 −0.5 −0.6 −0.2 −0.3 −0.2 −0.1 −0.1 −0.1 −0.1 −0.0 −0.0 −2.4 −2.4 −2.4 −2.3 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.2 −2.1 −2.3 −2.1 −2.1 −2.1 −2.1 −2.2 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.0 −2.0 −2.0 −2.1 −2.1 −2.1 −2.0 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.8 −2.0 −1.9 −2.0 −1.9 1_Russia 0.0 0.0 ) −1.3 −1.1 −1.4 −1.1 −0.9 −1.0 −1.3 −0.9 −0.9 −0.9 −0.9 −0.9 −0.7 −0.7 −0.6 −0.5 −0.7 −0.2 −0.3 −0.2 −0.1 −0.1 −0.1 −0.1 −0.0 −0.0 −0.0 −2.4 −2.3 −2.4 −2.2 −2.3 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.1 −2.2 −2.1 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.2 −2.1 −2.2 −2.2 −2.1 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.8 −2.0 −1.9 −2.0 −1.9 −1.9 −1.9 −1.8 −1.9 −1.9 −1.8 −2.0 −1.9 −1.8 −1.9 −1.9 −1.8 −1.9 −1.9 −1.9 −1.8 −1.8 −1.9 −1.8 −2.0 −1.7 −1.8 −1.8 −1.7 −1.8 −1.7 −1.7 −1.8 −1.8 −1.8 −1.8 −1.7 −1.8 −1.7 −1.7 −1.6 −1.8 −1.8 −1.6 −1.7 −1.8 −1.5 −1.7 −1.6 −1.6 −1.7 −1.6 −1.6 −1.4 −1.5 −1.3 −1.4 −1.8 −1.6 −1.4 −1.6 −1.3 −1.5 −1.5 −1.4 −1.2 −1.4 −1.4 −1.5 0.0 ihlrengtv ausecdn ueirfalse superior encoding values negative large with

6_India - −1.1 −1.2 −1.2 −1.3 −1.1 −1.3 −1.1 −1.2 −0.9 −1.1 −1.0 −0.9 −0.9 −0.8 −0.8 −0.7 −0.8 −0.4 −0.5 −0.2 −0.3 −0.2 −0.2 −0.1 −0.0 −0.0 −0.0 4_Jamaica −2.3 −2.3 −2.2 −2.2 −2.2 −2.2 −2.1 −2.2 −2.2 −2.2 −2.1 −2.2 −2.2 −2.1 −2.2 −2.2 −2.2 −2.1 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.1 −2.0 −2.0 −1.9 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.8 −2.0 −1.9 −2.0 −1.9 −1.9 −1.8 −1.9 −1.9 −1.8 −1.8 −1.8 −1.8 −1.8 −1.9 −1.9 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.9 −1.8 −1.8 −1.6 −1.8 −1.8 −1.8 −1.8 −1.8 −1.7 −1.8 −1.7 −1.8 −1.8 −1.5 −1.7 −1.5 −1.8 −1.6 −1.7 −1.5 −1.5 −1.5 −1.6 −1.6 −1.7 −1.6 −1.6 −1.7 −1.6 −1.5 −1.6 −1.6 −1.5 −1.5 −1.5 −1.3 −1.2 −1.4 −1.2 −1.4 −1.2 −1.2 −1.2 −1.4 −1.4 −1.2 −1.2 0.0 DEMOGRAPHICS −1.0 −1.0 −0.7 −1.0 −0.9 −0.8 −0.7 −0.9 −1.0 −0.7 −0.7 −0.6 −0.7 −0.5 −0.5 −0.6 −0.4 −0.3 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.0 −0.0 −0.0 −2.4 −2.3 −2.4 −2.3 −2.3 −2.3 −2.3 −2.2 −2.3 −2.2 −2.2 −2.1 −2.1 −2.1 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −2.0 −2.0 −2.2 −2.0 −1.9 −2.0 −2.0 −1.9 −2.0 −2.0 −1.9 −2.0 −1.9 −2.0 −1.9 −1.7 −1.8 −1.9 −1.9 −1.9 −1.8 −1.8 −1.9 −1.8 −1.9 −1.7 −1.7 −1.8 −1.8 −1.7 −1.7 −1.7 −1.8 −1.8 −1.7 −1.8 −1.8 −1.7 −1.8 −1.9 −1.7 −1.7 −1.6 −1.7 −1.8 −1.6 −1.6 −1.7 −1.7 −1.8 −1.6 −1.6 −1.6 −1.7 −1.5 −1.6 −1.7 −1.6 −1.8 −1.5 −1.8 −1.5 −1.5 −1.9 −1.6 −1.9 −1.4 −1.6 −1.5 −1.7 −1.6 −1.5 −1.4 −1.3 −1.4 −1.4 −1.5 −1.3 −1.5 −1.2 −1.3 −1.3 −1.3 −1.2 −1.2 −1.3 −1.5 −1.1 −1.5 −1.1 −1.2 −1.2 −1.3 −1.2 −1.0 −1.1 −1.1 3_Ghana 0.0 −1.0 −1.1 −0.7 −1.0 −1.0 −0.8 −0.7 −0.9 −1.0 −0.8 −0.7 −0.6 −0.7 −0.5 −0.6 −0.8 −0.4 −0.4 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.1 −0.0 −2.3 −2.3 −2.5 −2.3 −2.2 −2.3 −2.3 −2.2 −2.3 −2.2 −2.1 −2.1 −2.2 −2.1 −2.1 −2.1 −2.0 −2.1 −2.1 −2.0 −2.1 −2.0 −2.1 −2.0 −1.9 −2.0 −1.9 −1.9 −1.9 −2.0 −2.0 −1.9 −2.0 −1.9 −2.1 −1.8 −1.7 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −1.8 −1.9 −1.8 −1.7 −1.8 −1.8 −1.7 −1.7 −1.7 −1.8 −1.8 −1.7 −1.8 −1.7 −1.8 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.7 −1.6 −1.6 −1.7 −1.6 −1.8 −1.6 −1.8 −1.5 −1.5 −1.9 −1.7 −1.9 −1.5 −1.6 −1.5 −1.8 −1.6 −1.5 −1.4 −1.3 −1.5 −1.4 −1.5 −1.4 −1.4 −1.3 −1.4 −1.4 −1.3 −1.3 −1.2 −1.3 −1.6 −1.2 −1.6 −1.1 −1.2 −1.2 −1.3 −1.3 −0.9 −1.1 −1.1 0.0 0.0 5_Kenya −1.0 −1.1 −0.8 −1.0 −1.0 −0.9 −0.8 −1.0 −0.9 −0.8 −0.8 −0.7 −0.7 −0.6 −0.6 −0.7 −0.5 −0.4 −0.2 −0.2 −0.1 −0.2 −0.1 −0.0 −0.0 −0.0 −0.0 −2.2 −2.2 −2.3 −2.2 −2.2 −2.2 −2.2 −2.1 −2.2 −2.1 −2.1 −2.1 −2.1 −2.0 −2.0 −2.1 −2.0 −2.0 −2.1 −1.9 −2.0 −2.0 −2.0 −2.0 −1.9 −2.0 −2.0 −1.9 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −2.0 −1.8 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −1.8 −1.8 −1.7 −1.8 −1.9 −1.7 −1.8 −1.7 −1.8 −1.8 −1.7 −1.7 −1.8 −1.7 −1.8 −1.8 −1.7 −1.7 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.7 −1.8 −1.6 −1.7 −1.7 −1.7 −1.6 −1.6 −1.7 −1.6 −1.8 −1.6 −1.7 −1.6 −1.6 −1.8 −1.6 −1.7 −1.5 −1.6 −1.5 −1.6 −1.6 −1.5 −1.5 −1.4 −1.5 −1.4 −1.5 −1.4 −1.4 −1.4 −1.4 −1.4 −1.3 −1.3 −1.3 −1.4 −1.5 −1.2 −1.4 −1.1 −1.2 −1.2 −1.3 −1.2 −1.1 −1.2 −1.1 3_Nigeria 0.0 −1.0 −1.0 −0.7 −1.0 −0.9 −0.8 −0.6 −0.9 −0.9 −0.8 −0.7 −0.6 −0.7 −0.5 −0.6 −0.7 −0.4 −0.4 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.1 −0.0 −0.0 −2.3 −2.3 −2.3 −2.3 −2.2 −2.1 −2.2 −2.1 −2.2 −2.2 −2.1 −2.1 −2.0 −2.1 −2.0 −2.1 −2.0 −2.0 −2.0 −1.9 −2.0 −2.0 −2.0 −2.1 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −1.9 −1.8 −1.8 −1.9 −1.9 −1.9 −1.9 −1.9 −1.9 −1.8 −1.8 −1.8 −1.8 −1.7 −1.8 −1.8 −1.7 −1.8 −1.7 −1.8 −1.7 −1.7 −1.8 −1.8 −1.7 −1.8 −1.8 −1.8 −1.7 −1.6 −1.7 −1.8 −1.6 −1.6 −1.7 −1.7 −1.8 −1.5 −1.6 −1.6 −1.7 −1.6 −1.6 −1.7 −1.6 −1.8 −1.6 −1.8 −1.5 −1.5 −1.9 −1.6 −1.9 −1.5 −1.6 −1.5 −1.8 −1.7 −1.5 −1.4 −1.3 −1.4 −1.4 −1.4 −1.3 −1.5 −1.3 −1.4 −1.4 −1.3 −1.4 −1.2 −1.2 −1.3 −1.2 −1.3 −1.1 −1.1 −1.1 −1.3 −1.2 −0.9 −1.1 −1.0 0.0

4_Haiti −0.9 −0.9 −0.6 −0.9 −0.8 −0.8 −0.6 −0.9 −1.0 −0.7 −0.7 −0.6 −0.7 −0.5 −0.5 −0.6 −0.4 −0.3 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.0 −0.0 −0.0 −0.0 −2.3 −2.3 −2.3 −2.2 −2.2 −2.1 −2.2 −2.1 −2.2 −2.1 −2.0 −2.1 −2.0 −2.1 −2.1 −2.1 −2.0 −2.0 −2.0 −1.9 −2.0 −2.0 −1.9 −2.1 −1.9 −1.8 −1.8 −1.9 −1.9 −1.9 −1.9 −1.8 −1.9 −1.8 −1.8 −1.8 −1.7 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.8 −1.7 −1.7 −1.8 −1.8 −1.7 −1.7 −1.6 −1.7 −1.8 −1.6 −1.7 −1.7 −1.7 −1.7 −1.8 −1.7 −1.7 −1.6 −1.6 −1.7 −1.5 −1.6 −1.7 −1.6 −1.7 −1.5 −1.6 −1.5 −1.6 −1.5 −1.5 −1.6 −1.5 −1.7 −1.5 −1.8 −1.5 −1.5 −1.9 −1.6 −1.8 −1.4 −1.5 −1.4 −1.5 −1.4 −1.4 −1.4 −1.3 −1.4 −1.4 −1.4 −1.3 −1.4 −1.1 −1.3 −1.3 −1.2 −1.2 −1.2 −1.2 −1.4 −1.1 −1.3 −1.0 −1.1 −1.1 −1.3 −1.1 −1.0 −1.0 −0.9 3_Liberia −0.9 −1.0 −0.7 −1.0 −0.9 −0.8 −0.6 −0.9 −0.9 −0.8 −0.7 −0.6 −0.8 −0.5 −0.6 −0.7 −0.4 −0.3 −0.2 −0.2 −0.1 −0.1 −0.1 −0.0 −0.1 −0.0 5_Somalia −2.2 −2.2 −2.3 −2.3 −2.1 −2.1 −2.1 −2.1 −2.1 −2.1 −2.0 −2.0 −2.0 −2.1 −2.0 −2.1 −2.0 −2.1 −2.1 −1.9 −1.9 −2.0 −1.9 −2.1 −1.9 −1.9 −1.7 −1.9 −1.7 −1.9 −1.9 −1.8 −2.0 −1.9 −1.9 −1.8 −1.7 −1.8 −1.7 −1.7 −1.8 −1.8 −1.8 −1.9 −1.8 −1.8 −1.7 −1.7 −1.8 −1.8 −1.6 −1.6 −1.7 −1.7 −1.7 −1.6 −1.7 −1.8 −1.7 −1.8 −1.7 −1.7 −1.6 −1.6 −1.6 −1.8 −1.6 −1.6 −1.7 −1.6 −1.6 −1.5 −1.6 −1.6 −1.7 −1.6 −1.5 −1.6 −1.6 −1.7 −1.5 −1.7 −1.5 −1.5 −1.8 −1.6 −1.8 −1.4 −1.5 −1.4 −1.6 −1.6 −1.5 −1.3 −1.3 −1.3 −1.4 −1.4 −1.3 −1.4 −1.3 −1.3 −1.4 −1.2 −1.3 −1.2 −1.2 −1.4 −1.2 −1.4 −1.1 −1.1 −1.1 −1.2 −1.1 −0.9 −1.1 −1.0 0.0 0.0 T  −1.2 −1.0 −0.9 −1.1 −1.0 −1.1 −0.8 −1.1 −0.9 −0.8 −0.9 −0.7 −0.8 −0.7 −0.6 −0.6 −0.6 −0.3 −0.2 −0.1 −0.1 −0.2 −0.1 −0.0 −0.0 −2.2 −2.3 −2.2 −2.1 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −2.0 −1.8 −1.8 −1.9 −2.0 −1.9 −1.9 −1.9 −1.9 −2.0 −1.9 −1.9 −1.7 −1.9 −1.9 −1.8 −1.7 −1.9 −1.9 −1.9 −1.8 −1.9 −1.8 −1.8 −1.9 −1.8 −1.8 −1.8 −1.9 −1.7 −1.7 −1.8 −1.8 −1.8 −1.8 −1.7 −1.7 −1.8 −1.8 −1.9 −1.7 −1.7 −1.7 −1.7 −1.6 −1.7 −1.6 −1.7 −1.7 −1.8 −1.7 −1.6 −1.6 −1.6 −1.6 −1.7 −1.7 −1.7 −1.6 −1.6 −1.8 −1.5 −1.6 −1.4 −1.6 −1.6 −1.6 −1.6 −1.6 −1.7 −1.5 −1.5 −1.6 −1.6 −1.7 −1.6 −1.7 −1.6 −1.4 −1.5 −1.5 −1.5 −1.3 −1.4 −1.4 −1.5 −1.5 −1.4 −1.4 −1.4 −1.3 −1.3 −1.4 −1.3 −1.5 −1.3 −1.1 −0.9 −1.3 −0.9 −1.3 −1.0 −1.0 −1.2 −1.3 −1.3 −1.0 −0.9 0.0 0.0 0.0 0 ≤ → → log10 FNMR FMR =1e−05 Crossing Border vs. Application Dataset: 0 NR FNIR FNMR, M,FPIR FMR, . 00001 −3 −2 −1 0 ahcell Each . → → 0 1 60 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: particular to related FNIR are metric general These The operation. 3.2. follows: of modes section two in reflecting detailed 23 Figure been in have applications identification to appropriate metrics The Metrics 6.1 with population global a to analysis extend age. will in work range Further more employ all adults. tests younger These identifica- only rate. negative identification and false positive mugshots, false measure measure domestic to to used used are are probes probes mated nonmated the the and case rate, each tion In . 16 Annex in appears detail More specific isolate to of conceived were They photographs. follows. mugshot as just factors demographic use all trials identification three The identification in differentials negative False 6 08:14:00 2019/12/19 . . . . . erhhsalkl noldmt,temti sFNMR is metric the mate, enrolled likely a has search shown is lists longer Identification: of utility The 0). = R. (T vs. threshold FNIR plotting a by applying without candidates similar most R the FNIR Investigation: 20-40. aged also probes, than non-mate longer of no sets but balanced photo enrolled with the search We to after. year years calendar 5 different a in taken probes mated with that search 20-40. aged Race also probes, than non-mate longer of no sets but balanced photo enrolled with the search We to after. year years calendar 5 different a in taken probes mated with that search 20-40. aged Sex also probes, than non-mate longer of no sets but balanced photo enrolled with the search We to after. year years calendar 5 different a in taken probes mated with that search Sex ayflepstvs u ilas lvt as eaie,tetaef en hw sapo fFNIR(T) of plot a as shown being tradeoff the FPIR(T). negatives, false vs. elevate also will but positives, false many E T XEC ECH ecntutagleycnann 0 0 lc e,ad5000bakwmn gd2 0 We 40. - 20 aged women, black 000 500 and men, black 000 500 containing gallery a construct We : We 40. - 20 aged women, white 000 800 and men, white 000 800 containing gallery a construct We : S . S . ecntutagleycnann 0 0 lc e,ad8000wiemn gd2 0 We 40. - 20 aged men, white 000 800 and men, black 000 800 containing gallery a construct We : ( UMMARY UMMARY ,R, N, 0) stepooto fmse ae hnsaciga -nolegleyadconsidering and gallery N-enrollee an searching when mates missed of proportion the is o netgtr iln otaes ogcniaelssi usi fala,temetric the lead, a of pursuit in lists candidate long traverse to willing investigators For o hs plctosweeannzr hehl sue ool eunrslswe a when results return only to used is threshold non-zero a where applications those For as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ( ,R T R, N, - ) DEMOGRAPHICS h s fthresholds of use The . T (  ,R T R, N, 0 > T → → ) NR FNIR FNMR, M,FPIR FMR, 0 oesbt as both covers ilsuppress will → → 0 1 61 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 08:14:00 2019/12/19 to the record to the and a pointer images and is given the algorithmThe data. any biographic images and consists of database enrollmentThe Human role Human candidates Num. Threshold identity of Claim Example metric of interest Performance frequency involvement human Intended E T XEC ECH S . S . UMMARY UMMARY Positive identification Positive and Tables 10, 19 FPIR. See sec. low FNIR 3.1,at 3.2 probability of impostor identification rate plus prior Rare detect impostor or of false to negatives, resolution to assist candidate in user Review 1 objective security to implement High, enrolled be to claim Implicit Access to a gym or cruise ship Dawn Alice Zeke Zack Alex Sam Jack Bob Ben Eva Eve Pat Bill for one entry as most searches Usually the correct response is is response correct the Usually Jill – approx. the false negative negative false the approx. Pat ID as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT iue2:Ietfiainapiain n eeatmetrics. relevant and applications Identification 23: Figure Rank - will be mated be will AERCGIINVNO TEST VENDOR RECOGNITION FACE 1 Output is a candidate is list. Output other data structure Array, tree, index orArray, tree, index Enrolled database: Enrolled Feature Feature extraction Score 3.142 e.g. CNN model rank and threshold, and are these set to implement objectives. Tables 10, 19 FPIR. See sec. low FNIR 3.1,at 3.2 and of an actual mate identification rate plus prior probability Rare positive or correct hit false to determine candidate Review 0 false positives to limit High, enrolled be not to claim Implicit duplicate drivers license applications or deportee of e.g. Detection Watchlist identification Negative empty empty list as most searches are – approx. the false positive positive false the approx. Usually correct response is is response correct Usually Search PhotoSearch ID Its Its Rank length is by length configuration preset determined of - non DEMOGRAPHICS - e.g. N comparisons Feature Feature extraction mated Search Algorithm Score e.g. CNN model Detection and and Detection localization possible hits to examiners see [26] candidates, refer multiple Review L, settorequest by algorithm Zero Inquiry claim: No documents. ID without scene Crime photos, or of detainee Post See sec. 3.2 and Table 12, See14, sec. 163.2Table and 1... 50, FPIR = 1.say. ranks FNIR at Always - event investigation event T Zeke Bob Ben  Pat ID ... 0 → → evaluation. NIST’s is scope ofthe This box grey black box. evaluated as engine recognition face Automated Rank NR FNIR FNMR, M,FPIR FMR, 4 3 2 1 Score 0.707 1.602 2.998 3.142 → → 0 1 62 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 14 make We FPIR. vs. FNIR show panels lower The R. in vs included FNIR are show algorithms panels observations all upper following for the the Plots case, each algorithms. In two . for 16 rates Annex error identification show 25 and 24 Figures Results 6.2 08:14:00 2019/12/19 e h D’ ainlVtlSaitc eotfr21:https://www.cdc.gov/nchs/data/nvsr/nvsr67/nvsr67 2017: for Report Statistics Vital National CDC’s the See . . . . gvsmc mle xusosi FPIR. in excursions smaller much example, gives For with 25 excursion. Figure FPIR, this in in exhibit featured increase not algorithm the do order-of-magnitude algorithms an Figure some revealing see Notably - occurence algorithm. row by common bottom varying a the magnitudes of is column This center the example. in plots for tradeoff threshold 24, error fixed the of in points slanted connecting and lines long The often are women: black in rates identification positive false lines. Higher same the along documented similarities elevated well give Given to expected dataset. be will the Siblings in present is population adult estimate rates States we twinning dataset, United the the by of represented population percent the few Given a population. the reason in second twins A of matching. presence fingerprint the the ten-print is in by are underpinned and person mugshot excellent, a the is because of integrity this truth discount photos We ground positives. which occur false in can apparent cause errors They These labeling IDs. searches. multiple identity imposter under truth database the ground in non-mates First, scoring to reasons. high reduced of several is presence for FPIR the when to 50% reaches due FNIR is 24, This Figure FNIR 0.0001. in threshold example, the as For FNIR FPIR. increase reduce rapid to a increased show is characteristics tradeoff error The attainable: not is FPIR Low correct the displacing be may position. scores 1 non-mate rank women from High women black women. rates. black that black match namely for false results, true one-to-one verification less high 4 is produce section This to from tend men. available of is search this likely are for more than mechanism are ranks possible top women A the white in of result searches correct that the notably produce exceptions, to are There groups. applies This racial men. both than rates within negative false higher give invariably Women sexes: the between Differentials two. of and Tevian factor Kedacom, a Idemia, Vocord. Eyedea, below Aware, 3DiVi, well from small, algorithms par- including often and exceptions are some modes, differentials are identification The The and algorithms. investigate negative accurate the more false in the lower applies for give ticularly This invariably men men. black white panels, than left-side rates the identification From men: in race by Differentials E T XEC ECH S . S . UMMARY UMMARY 14 ,w xettist ei h aa ohietclad oecmol,fraternal. commonly, more and, identical both data, the in be to twins expect we [27], as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 08-508.pdf → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 63 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: corresponding The and 2019. . center August 16 The in Annex NIST in to men. appear provided of algorithms young algorithm, FPIR identification one in all points give to race for join applies to figures figure lines values of chosen cannot The grey effect high are The they males. respectively. the to thresholds that white subjects here show four enough right white demographic, high then The panels the black is left on in threshold. searches women, The equal value of and of low volume men search. between the a every difference which as from show from for panels rates swept results organizations right miss is review to shows threshold rank, appropriate to row is of a labor bottom metric function where afford The This a rate, as generation. left. identification rates lead the positive “miss” for on false identification lists of negative candidate false function traversing shows a investigators row to top appropriate the metric identification, a mugshot For 24: Figure 08:14:00 2019/12/19 False negative identification rate, FNIR(T) False negative identification rate (FNIR) 0.030 0.050 0.070 0.100 0.200 0.300 0.500 0.700 0.002 0.003 0.005 0.001 0.002 0.003 0.005 0.007 0.010 0.020

1e−04 E T 1 XEC ECH

3e−04 S . S .

1e−03 UMMARY UMMARY BW_male_20_40_balanced BW_male_20_40_balanced 3 3e−03

1e−02

3e−02 10

1e−01 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False 3e−01 FRVT reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/imperial_000.pdf 30

1e+00 50 - AERCGIINVNO TEST VENDOR RECOGNITION FACE 1e−04 False positive identificationrate, FPIR(T) 1

3e−04

1e−03 MF_black_20_40_balanced MF_black_20_40_balanced

3e−03 3

1e−02 Rank

3e−02 10

1e−01

3e−01 30

1e+00 { 0 50 . 0003 1e−04 - 1 DEMOGRAPHICS 3e−04 , 0 .

003 1e−03 MF_white_20_40_balanced MF_white_20_40_balanced

, 3e−03 3 0 . 03

1e−02 , 0

. 3e−02 3 10 }

epcieyfroebaseline one for respectively 1e−01 T  3e−01 0 30

1e+00 → → 50 NR FNIR FNMR, M,FPIR FMR, 0.3000 0.0300 0.0030 0.0003 males for FPIR set for white 1.454 1.531 1.588 1.743 Thresholds race MUGSHOTS Dataset: race Male Female White Black Male Female White Black → → 0 1 64 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: corresponding The 2018. June in and NIST . center to 16 provided The Annex algorithm, in men. one appear of algorithms to young FPIR identification applies in all points give figure race for join to The figures lines values of chosen cannot grey males. effect high are The they white respectively. the to thresholds that subjects here show four enough right white demographic, high then The panels the black is left on in threshold. searches women, The equal value of and of low volume men search. between the a every difference which as from show from for panels rates swept results organizations right miss is review to shows threshold rank, appropriate to row is of a labor bottom metric function where afford The This a rate, as generation. left. identification rates lead the positive “miss” for on false identification lists of negative candidate false function traversing shows a investigators row to top appropriate the metric identification, a mugshot For 25: Figure 08:14:00 2019/12/19 False negative identification rate, FNIR(T) False negative identification rate (FNIR) 0.050 0.070 0.100 0.200 0.300 0.500 0.700 0.005 0.007 0.010 0.020 0.005 0.007 0.010 0.020 0.030

1e−04 E T 1 XEC ECH

3e−04 S . S .

1e−03 UMMARY UMMARY BW_male_20_40_balanced BW_male_20_40_balanced 3 3e−03

1e−02

3e−02 10

1e−01 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False 3e−01 FRVT 30 reports/1N/figures/fbi/ngi/race_sex/cmcs_dets/idemia_4.pdf

1e+00 50 - AERCGIINVNO TEST VENDOR RECOGNITION FACE 1e−04 False positive identificationrate, FPIR(T) 1

3e−04

1e−03 MF_black_20_40_balanced MF_black_20_40_balanced

3e−03 3

1e−02 Rank

3e−02 10

1e−01

3e−01 30

1e+00 { 0 50 . 0003 1e−04 - 1 DEMOGRAPHICS 3e−04 , 0 .

003 1e−03 MF_white_20_40_balanced MF_white_20_40_balanced

, 3e−03 3 0 . 03

1e−02 , 0

. 3e−02 3 10 }

epcieyfroebaseline one for respectively 1e−01 T  3e−01 0 30

1e+00 → → 50 NR FNIR FNMR, M,FPIR FMR, 0.3000 0.0300 0.0030 0.0003 males for FPIR set for white 1011.930 1337.657 1637.344 2179.743 Thresholds race MUGSHOTS Dataset: race Male Female White Black Male Female White Black → → 0 1 65 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 hsi nipratrsl eas timplies it because result identification negative important in result an important is especially is This This positives. 27. false of Figure likelihoods in equitable more featured much algorithm the be in will disappears effect there This applied is . applies 17 behavior threshold Annex Such see a values. - score if algorithms the many in that to and identification clear returned, one candidates is of by number It returned the in scores differences conducted. median demographic shows are 26 searches Figure non-mated 6 , when section algorithm in listed experiments three the similar For most the finding with tasked still are algorithms that enrollments. mean the not because does reduced this be However, will executed. the occurences actually partition positive are to comparisons false techniques N than various fewer include far fast- These that of so set 26]. data 21, a 19, enrollment employ [2, might search they expediting Instead at comparisons. aimed 1:1 algorithms N search as search 1:N implement not do algorithms Other itself. data image the on operator- dependent an is that above random-variable an are return candidates will many system in however The sorted return threshold ways: candidates will two specified N it of yields or either which in candidates, returned operation of is sort number that operator-specified a of the by result of followed The comparisons similarity. is 1:1 of This N order as decreasing items. image enrolled probe N a the of search with 1:N probe a implement algorithms search one-to-many 3. Some section in discussed are Metrics galleries. large in rates positive 10 hr stesz fteerle ouain ihFPIR With population. enrolled the of size the is N where Links: 15 comparison a that likelihood implied the formula then this by 0.01, given = is FPIR match false i.e. a producing times, yield at will 100 aimed in thresholds with one configured say individuals are outcomes systems of such positive numbers If false large millions. of very 100s thousand enroll or hundred often 10s per the howveer, into one applications, numbering between Identification i.e. 0.001, thousand. and per 0.00001 one access between and verification (FMR) in rates typical match is false It target applications. verification to in at control targeted operate those typically than deployments lower much verifi- identification rates one-to-many to match large-scale false differentials that positive note false first We similar exhibit algorithms. cation algorithms identification whether addresses section The identification in differentials positive False 7 08:14:00 2019/12/19 h oeao-pcfid aaeesmgtsmtmsb e yplc,o ytemnfcue ftesystem. the of manufacturer the by or by-policy, set be sometimes might parameters “operator-specified” The − 8 h oml ie rtodreuvlneo dnicto ihvrfiain h omrneslwfalse low needs former the verification: with identification of equivalence order first a gives formula The . E T XEC ECH S . S . UMMARY UMMARY 15 ntecs hr hehl sue,tenme fcniae eundwl ea be will returned candidates of number the used, is threshold a where case the In . as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE FMR = IR P F N 0 = . 01 - n N and , DEMOGRAPHICS 10 = 6 s hsfruaipisFMR implies formula this T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 (10) 1 66 = This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: Report Interagency NIST in very documented will results reviewer accuracy high the very searches 16 the mugshot-mugshot per In 1 rank match. than a further no are look photos often candidate the each if alongside determine photo probe to the photo compare reviewers gallery human and candidates of number fixed a positive return false in for shifts implications are the larger women consider for now and is We men ) 16 white Annex and and women, 24 and Figure left-to- (see men rates. From behavior black normal threshold. males, more equal white The of and similar. points black closely join for lines values grey the FPIR where the 25 right, Figure game in sports a evident at is card- hooligans rates e.g. soccer identification low, or is database casino, the a in is in being surveillance actually sharp person searched NEC, a of Idemia, probability Aware, prior the developers where for applications figures corresponding the and . 27 17 Annex Figure in in labels included “recent” are [17]. that and that see Toshiba nonmated with consolidated” - and in one “lifetime Tevian, here just The produced behavior or are enrolled. person, the 000 panels 000 per 600 000 1 images Contrast four 12 = multiple with upper of N galleries The into inclusion with searches to galleries from refer 50. produced uses are right-side to panels The 1 left-side heatmaps The enrolled. rank the searches. people photos at mated mugshot from of placed four galleries candidates lower mixed the into for searches; faces scores women’s and similarity men median white black, show Asian, of searches For 26: Figure 08:14:00 2019/12/19

ntdteueo uoae aercgiint erhaon 100settr tsce ae against games soccer at spectators 000 21 around search to recognition people. face 50 automated about of use of the watch-list noted a article news recent a example, For Rank Rank 50 40 30 20 10 50 40 30 20 10 50 40 30 20 10 50 40 30 20 10 0 0 0 0 E T XEC ECH Female Female Asian Asian reports/1N/figures/fbi/ngi/natural/score_heatmaps_by_rank_and_demographics/ntechlab_6.pdf S . S . UMMARY UMMARY Asian Male Asian Male Female Female Black Black N =12000000 N =12000000 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT Black Male Black Male - AERCGIINVNO TEST VENDOR RECOGNITION FACE Female White Female White netgtv la eeain applications generation” “lead investigative Demographics ofsearchsubject White Male White Male Female Asian Female Asian Asian Male Asian Male - Female Black DEMOGRAPHICS Female Black N =1600000 16 N =1600000 h ako nefc nflepositive false on effect an of lack The . Black Male Black Male Female White Female White nsc ae,algorithms cases, such In . T  White Male 0 White Male

→ →

recent NR FNIR FNMR, lifetime_consolidated M,FPIR FMR,

recent lifetime_consolidated NONMATE ntechlab_6 Algorithm MUGSHOT Dataset MATE 0.700 0.725 0.750 → 0.70 0.75 0.80 0.85 0.90 → 0 1 67 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: demographic across scores compendium. the of 17 Annex uniformity the The in others [17]. many see and labels - 26 “recent” Figure one and in nonmated just consolidated” evident in or “lifetime that The to produced person, contrast are enrolled. per in 000 images panels 000 is 600 000 1 multiple groups four 12 = with of upper N galleries inclusion The into with searches to galleries from refer 50. produced uses are right-side to panels The 1 left-side heatmaps The enrolled. rank the searches. people photos at mated mugshot from of placed four galleries candidates lower mixed the into for searches; faces scores women’s and similarity men median white black, show Asian, of searches For 27: Figure 08:14:00 2019/12/19 Rank Rank 50 40 30 20 10 50 40 30 20 10 50 40 30 20 10 50 40 30 20 10 0 0 0 0 E T XEC ECH Female Female Asian Asian S . S . reports/1N/figures/fbi/ngi/natural/score_heatmaps_by_rank_and_demographics/idemia_5.pdf UMMARY UMMARY Asian Asian Male Male Female Female Black Black N =12000000 N =12000000 as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT Black Male Black Male - AERCGIINVNO TEST VENDOR RECOGNITION FACE Female White Female White Demographics ofsearchsubject White Male White Male Female Asian Female Asian Asian Male Asian Male - Female Black Female DEMOGRAPHICS Black N =1600000 N =1600000 Black Male Black Male Female White Female White T  White Male 0 White Male

→ →

recent NR FNIR FNMR, lifetime_consolidated M,FPIR FMR,

recent lifetime_consolidated NONMATE idemia_5 Algorithm MUGSHOT Dataset MATE 0 200 400 600 800 → 0 1000 2000 3000 4000 5000 → 0 1 68 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: human in of involved consideration issues full The but document, list. this candidate of the scope down (i.e. the far miss beyond appears to are likely that review more candidate are mated reviewers a example, to negative) For presented false are negatives. a candidates false which and in positives order false the both scores hu- just affects that that the reviewers find evidence in however, scant issues did, found study bias and That cognitive effects, reviewer. at the demographic looked influence without [12] results, search study fingerprint Over earlier of one elsewhere. review though potentially man and human, population the female influence black may the fifty this for in see time GUI scores will a higher in human see text The will as reviewer presented human? scores, the by the example, accompanied influence are candidates to demo- those expected are if However, 26 how regardless. Figure lists, candidates in candidate those reviewing as with such tasked differentials are graphic reviewers human circumstances from whatever tenable humans. in be in not But factors may reward and and laborious fatigue That’s to due list. perspective the research on employed, operations so candidates if an 50 reviewer, human say a all cases review such rollers” In searches “high lower. most much detecting in is at set would that everyone aimed in application almost is casino casino example, the a of for patron In ship a gallery. likelihood cruise the the is a in however, boarding present applications, be In such would present. board in actually to parameter attempting is important mate very a A that probability located. prior is the mate will a that candidates before of reviewed number be expected to the summarizing have measure workload a includes also report That 8271. 08:14:00 2019/12/19 erhagrtm n ua eiwr sa xeietlpyhlg,hmnfcosadoeain research operations and factors issue. human psychology, experimental an is reviewers human and algorithms search E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS systems opie fatmtdface automated of comprised T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 69 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: dependence. positive false channel. and color each negative in false content both of quantify bits ISO/IEC 7 should least work the at Such particular, have to characterize In photos to portrait standardization. required approach future 2004, since theoretic for has, useful standard information 19794-5 be an would skin, lighting light poor or to response dark algorithmic of over-exposure or under- via atives, analysis: theoretic Information possibility. this esti- address demographic automatically to exhibit need to themselves will Efforts may work that Such height. algorithms differentials. and involve race will age, images sex, from least, information phenotypic at mate of contributions work relative extend the would separating models in Such [9] interact. algorithms and artefacts imaging phenotypes, including properties topics: Analysis: further two into research by supported be would work This . . possibilities. development and research images. two standards-compliant following in the reduced motivates are This therein, differentials demographic and rates, error negative False negatives false of mitigation toward Research 8 08:14:00 2019/12/19 rsta r o eetn n eeigoffaces. off cam- metering with and under- standards detecting photography and not documentary are over- that on eras and rely still low-contrast operations averting enrollment in Many valuable images. exposed prove would capture face-aware specifications closed-loop particularly and for assessment, quality image automated into research further lines similar Along via open is Participation 2021. in groups. expected standardization completion national with 2020 standard, through developed That measurement. be scanners. exposure will fingerprint 24358, and and ISO/IEC iris estimation, in exist pose mechanisms detection, control quality face “auto-capture” real-time Analogous require may that subsystems ture cameras: 24357). Face-aware and 29794-5 37 (ISO/IEC SC quality ISO/IEC on The work [1]. initiated problems recently demographic has have subcommittee itself biometrics can algorithms, grounds underlying quality the on of images assessment single ongoing of an rejection has NIST and exist quality, packages of automated commercial assessment an While automated by the differential. determined for negative as false reduced images, a quality yields higher algorithm, that assessment noted quality [24] research Recent exposure. and etry compliance: standards Improved E T XEC ECH S . S . hr sane o mrvdmdl fdmgahcefcs atclryt o subject-specific how to particularly effects, demographic of models improved for need a is There UMMARY UMMARY h aeIOICcmitehsrcnl ntae oko pcfiain o cap- for specifications on work initiated recently has committee ISO/IEC same The as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT ie h oeta o oryilmntdpoorpst rdc as neg- false produce to photographs illuminated poorly for potential the Given - AERCGIINVNO TEST VENDOR RECOGNITION FACE h S/E 99- tnadicue eurmnsrgltn geom- regulating requirements includes standard 19794-5 ISO/IEC The - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 70 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: for resolution higher insufficient much often specified is long Profiles). this has Acquisition and Face community ANSI/NIST reviewer images), (see human pixel purposes The 640 forensic x viable. 480 be (i.e., to rates texture sampling skin processing spatial for on VGA based below often or very at has are images image recognition of given face above any of and deployments that nose, contemporary knowledge However, moreover, the and, resolution. to [US resolution that (adjacent twins high texture requires distinguish skin algorithm correctly from This and can features genes), eyebrows). that extracting the the patented by of was operates half algorithm it share an also US7369685B2]; 2004, (which In Patent: siblings to relatives. also distant extend more to to expected in then are Similarities [17] genes. twins identical practical fraternal given all features in at similar faces positives very false by cause characterized invariably are twins Twins to identical related [17], thresholds. characteristics tests operational of recognition set NIST large In a face. and the tone of skin shape determines which of expression phenotypic determined the partially genes, is appearance by Facial images. face between similarity measure algorithms recognition Face power discriminative Greater 9.3 developer One data, [15]. differentials. 2018 such reduce November Absent to in concept altered a data. be such can training advanced functions derived, cost globally their diverse, whether consider more be might of would developers mitigation utility longer-term development, the the the investigate Thus, to to differentials. intrinsic developers positive factor for false other particular some reducing perhaps at or effective data, be training can false from that positive false suggests separate This not did shifts. paper prior that negative a though affirms [33] This algorithms for do. effect” elsewhere “other-race developed an algorithms of that finding faces Chinese on give rates not do positive China false in elevated developed algorithms the many that observation our by prompted is mitigation longer-term A training Algorithm 9.2 currently is than rates match false homogeneous more much some be case. be should the always goal will the there - as mean impossible pro- prove a developer will around algorithm That distribution the rates. match if false irrelevant homogeneous be with software would vided above described strategies manipulation threshold The Summary 9.1 positives false of mitigation toward Research 9 08:14:00 2019/12/19 E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 71 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 edntmninfigrrnsi hscnetbcueee huhgntciflec scniee ob absent be to considered is influence genetic though even because context this in imaging fingerprints mention of don’t expense We and difficulty the by distances. limited long is at surveillance, iris example the for applications, some in iris of alone. Use iris or convenience-driven face some either from with must differs person iris This a and authenticate positives. face that false the implementations limit both to which as in com- so that the thresholds) fusion respective of biometric (against Recognition of match kind simultaneously. particular essentially a images involve face would and biometric iris bined collect that cameras of now decade Links: 17 produce to iris, with face augment to be would solution “face-and-iris” second biometric a compound However a weighing and here. detailing scope - this our to beyond disadvantages is and is first advantages those The be would recognition. There face iris. in with rates face positive replace to false simply heterogeneous and higher to solutions afford aspects These interoperable support that most standards cameras. - quality across devel- and populations recognition camera interchange large multiple image in has and industry suppliers, individuals recognition algorithm of iris multiple The identification opers, Aadhaar. program the ID for National primary iris Indian the the the in been of notably has use likelihoods, match the false establishing low in demonstrated property and influence, rates. genetic positive of false lack low apparent very afford The ISO/IEC would 17, Type This (ANSI/NIST 4). this Group for Data place 9303 in ICAO records and is face infrastructure 39794-6, for standards be The would images. recognition iris face adjunct in one include differentials simultaneously, to positive essentially false iris of and mitigation face for consider collect to that approach cameras multiple has of NIST presence are recognition. marketplace face they the for fully. as case Given more the race twins not in and is distribution that age impostor the sex, 8 investigate Annex same to and plans the 4 near-term Figure of in individuals shown As within [39]. higher groups no Technology those across and are Science positives DHS is false 2019 distribution a that impostor Nevertheless, the noted of identification. study one-to-many tails scale the large of for characterisation least better at a needed, so show linkage, does genetic [25] features some research iris genetics have the However textures that iris determined. be genetically to not appears are this algorithms published for automated reason by by The processed detected that [17]. and something not - imaged do 40] [ when algorithms twins face that identical marketplace between structure contemporary discriminate a correctly have will to 29] [11, known algorithms extraction is feature iris human the of texture iris The and face of use and Collection 9.4 08:14:00 2019/12/19 nprl iil-ih mgsi ihypolmtci rw-ydpol smlnni h rsasrsicdn ih tvsbewave- irises visible of at recognition light The incident absorbs component. iris infrared the near illumi- in a infrared melanin with near as light people with ambient brown-eyed image in or iris problematic light, an highly ambient First lengths. is in images: images entirely two visible-light either of purely image collection in face require the still conventionally and would nation biometric compound a Such E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE 17 hsi aepsil ytemrepaeaalblt o tlata least at for availability marketplace the by possible made is This . - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 72 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: 28. Figure The in shown follows. characteristics as tradeoff error consequences, the its on and based construct rates, graphical match a false uses explanation variable heterogeneous of mitigation one detail We elevation Threshold 9.5 simultaneous. not is face and fingerprint both target of the collection the achieve minimal, to or threshold operating the increasing by implied FNMR Y. in demographic, increases high-FMR the the on shows FMR figure The 28: Figure 08:14:00 2019/12/19 . . . obigteoealsse noplc opine h ytmonrcnut h ro rdofcharac- tradeoff error the consults owner system the compliance, policy into system overall the bring To the meet to failure a group. therefore represents that therefore for T P2 objective threshold point security fixed This audit original the (0.001). at or say rate test five match of performance false factor biometric observed a higher, the some much that Y, of Demographic result group a new as some for perhaps that, discover, later we that suppose 0.0002). Now (i.e. 5000 in 1 FMR(T say i.e. to X T set Demographic threshold be a reasonably setting by might implemented application is verification This a in value, FMR This rate match jective. false target a with start We E T XEC ECH S . S . UMMARY UMMARY (FNMR) Rate Match Non False - Demographic X Demographic as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False Demographic Y Demographic FRVT ∆FNMR 1 - FMR = ) AERCGIINVNO TEST VENDOR RECOGNITION FACE X P4 POLICY ∆ T FMR 2 hscrepnst h on P1. point the to corresponds This . X POLICY 1 ups htti hehl a efcl airtdfor calibrated perfectly was threshold this that Suppose . FMR P1 P3 POLICY hthsbe e oipeetsm euiyob- security some implement to set been has that - DEMOGRAPHICS T 1 False Match Rate (FMR) Rate Match False P2 ∆FNMR T  0 Y → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 1 73 is This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 ae hr niiul nta eorpi r ae ..edryprosetrn y rngtlb Any nightclub. or cost. and gym T a to threshold entering the persons elevate elderly not e.g. to rare, decision are Demographic that in individuals where cases Links: FMR high original the that assumes argument are above rates The match demographics. false any which over for (much) algorithm vary biometric not a do having in i.e. benefit homogenous the see we construct, of kind this Using 08:14:00 2019/12/19 . iue2:Tefiuesosteefc fstigtrsod oaheetetre M ndmgahc n Y. and X demographics on FMR target the achieve to thresholds setting of effect the shows figure The 29: Figure eorpi ,t au hc et h rgnltre policy. target original for the FMR beats in which reduction value a a is to gain only X, The Demographic differ). may (which characteristics error-tradeoff the of gradients be- is as This shown are groups. increases demographic These both T elevated threshold necessarily is new FNMR the that cause is however this of effect The P3. point T at to compliance, threshold policy the to elevating returned by that notes and Y Demographic for teristic E T XEC ECH S . S . UMMARY UMMARY False Non False (FNMR) Match Match Rate - Demographic Demographic Demographic Demographic Demographic X Demographic as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - 2 AERCGIINVNO TEST VENDOR RECOGNITION FACE shge hnT than higher is ∆ Y FNMR Z 2 hudb eieae ntescrt otx endb het risk threat, by defined context security the in deliberated be should X and 1 ∆ n ple oaltascin rmaldemographics. all from transactions all to applies and , FMR FNMR POLICY Y T T T X Y Z (T Y 1 side rbeai.I a etlrbein tolerable be may It problematic. indeed is ) ol aeamgiueta eed nthe on depends that magnitude a have would T - X DEMOGRAPHICS False Match Rate (FMR) Rate Match False T X 2 h as ac aewudbe would rate match false the , T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 74 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: software. in that the of implement use to and secondly information and demographic testing, of appropriate capture some with via strategy operator: be the should thresholds on responsibilities the two what owner imposes out system but figuring the appropriate First, fully to sound management may threshold That for developer. responsibility the shifts than it rather that is approach this to aspect Another do would imposter sys- variable. an the are what thresholds of Consider that users complicated. knowledge legitimate more given genuine however, are, for this effective of be consequences would security information, This The this accordingly. tem. read T of would set indication system and some the table, and passport) lookup Asian) a a south consult aged, (e.g. middle document female, identity (e.g. an group in demographic encode the could one 29, Naively Figure subject Referencing problem. a a system. time T the presents each thresholds using threshold adopting is the who to of set correspond basis to would the system this on verification it a tailoring transaction for verification is a proposal thresh- executes a The assigning by rates demographic. match each false heterogeneous to address old to [23] suggestion the discuss we section this In demographic each for thresholds Explicit 9.6 08:14:00 2019/12/19 . . dniyceetasfo te esn h,ntrly aetesm tnct,sxadaegroup. age and sex to ethnicity, same access the have have naturally, to who, imposters persons for other opportunities from afford credentials identity naturally often will construction and sex societal age, that same Note the have to anyway. need behavior still a typical would be imposter steal would the / This match minimally ethnicity. procure false sex a might and induce age he better To matching high sex. demographics, is same low-threshold the threshold the of the one which in for somebody demographic from credential a from were imposter the If places. those in any credentials we’d stolen for low-thresholds, for with behavior black-market used a typical a be of be steal to genesis would / known expect were This procure passports would ethnicity. countries he and particular low, if sex However, is age, imposter. threshold same the the which of for somebody from demographic credential a from were imposter the If E T XEC ECH S . S . UMMARY UMMARY as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE 1 n T and 2 ie onsP n 3 ntefl.Hwt othis do to How on-the-fly. P3) and P1 points (i.e. - DEMOGRAPHICS T  0 → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 75 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 Links: References 08:14:00 2019/12/19 J aga.Hwii eonto works. recognition iris How Daugman. J. matching. [11] face in errors officers Passport M. A. Burton and M, Matheson R., Jenkins I., R. Kemp D., White [10] L etRwe n .K an ogtdnlsuyo uoai aerecognition. face automatic of study Longitudinal Jain. K. A. and Best-Rowden L. [3] In descriptors. deep of datasets billion-scale of indexing Efficient Lempitsky. Victor and Babenko Artem [2] https://www.telegraph.co.uk/technology/2016/12/07/robot-passport-checker-rejects- 2016. December [1] CnhaCo,Jh oad egnySrtn er itn n rnVmr.Dmgahceffects Demographic Vemury. Arun and Tipton, Jerry Sirotin, Accuracy Yevgeniy Howard, John O’Toole. Cook, Cynthia J. [9] Alice and Castillo, D. Carlos Phillips, Jonathon P. Cavazos, G. Jacqueline [8] effect: other-race the and context Learning O’Toole. J. Alice and Noyes, Eilidh Cavazos, G. Jacqueline [7] de- of impact differential the mitigating and identifying 22116 Iso/iec Savastano. M. and Campbell J. [6] datasets face of evaluation demographic and phenotypic Intersectional shades: Gender Buolamwini. Joy influence [5] that Factors Draper. A. Bruce and Phillips, Jonathon P. Givens, H. Geof Beveridge, Ross J. [4] h EECneec nCmue iinadPtenRcgiin(CVPR) Recognition Pattern and Vision Computer on Conference IEEE The asian-mans-photo-having-eyes/. 41:13,Jn2004. Jan 14(1):21–30, ONE PLoS commercial eleven of evaluation In An acquisition: image on systems. dependence their bias? and recognition race facial in measuring on we are Where algorithms: recognition https://arxiv.org/abs/1912.07398 face across comparison mechanisms. neural and models recognition. face improving for Strategies 6, Group Working 37, SC 1, JTC ISO/IEC report, Technical 2018. 11 http://iso.org/standard/72604.html, systems. biometric in factors mographic 2017. 01 Lab, Media MIT report, Technical classifiers. gender and 2009. 113(6):750–762, challenge. grand recognition face the in performance algorithm Intelligence Machine and Analysis Pattern E T XEC ECH S . S . UMMARY UMMARY EETascin nBoerc,Bhvo,adIett Science Identity and Behavior, Biometrics, on Transactions IEEE () 04 131.di1.31junl pone.0103510. doi:10.1371/journal. e103510. 2014. 9(8), , as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE 22019. 12 , 01:4–6,Jn2018. Jan 40(1):148–162, , iinResearch Vision EETascin nCrut n ytm o ie Technology Video for Systems and Circuits on Transactions IEEE 5:6 8,21.Fc ecpin Experience, perception: Face 2019. 183, – 157:169 , - DEMOGRAPHICS optrVso n mg Understanding Image and Vision Computer P11 22019. 02 PP:1–1, , ue2016. June , T  0 EETascin on Transactions IEEE → → NR FNIR FNMR, M,FPIR FMR, → → 0 1 76 , , This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 PtikGohr ereW un,adMiNa.Fc eonto edrts tl aeiaeand image face still - test vendor recognition Face Ngan. Mei and Quinn, W. George Grother, Identi- Patrick 2: [18] part (frvt) test vendor recognition Face Hanaoka. Kayee and Ngan, Veri- Mei Grother, 1: Patrick part [17] (frvt) test vendor recognition Face Hanaoka. Kayee and Ngan, Mei Grother, Patrick [16] in recognition face police Unregulated St line-up: [15] perpetual The Frankle. J. and Bedoya, A. Garvie, C. [14] Links: performance: recognition Face K. A. Jain and W., R. Bruegge Vorder C., J. Klontz J., M. Burge Klare, F. B. [22] demo- gender) and ethnicity, (age, varying to subject verification Face H. Wechsler and Khiyari El H [13] and accuracy on information contextual afis of assessment Quantified Wertheim. Kasey and Dror Itiel [12] 08:14:00 2019/12/19 Mst si,HtsiIak,adAssiSt.Fs -ers egbrsac o aeidentification face for search neighbor k-nearest Fast Sato. Atsushi and Imaoka, Hitoshi Ishii, Masato [21] ho- demographic specific and broad of effect The Vermury. Arun and Sirotin, Yevgeniy Howard, J. John database. fuzzy [20] large a for algorithm search fast A Zielinski. Piotr and Daugman, John Hao, Feng [19] ie ocp,eauto lnadai ehia eot ainlIsiueo tnad n Technology, and Standards http://biometrics.nist.gov/cs of 2013. Institute National 7 report, Technical api. and plan evaluation concept, video 2019. September Technology, and Standards of Institute National https://doi.org/10.6028/NIST.IR.8271. 8271, Report Interagency fication. 2019. October Technology, and Standards of Institute National https://nist.gov/programs-projects/frvt-11-verification. DRAFT, Report Interagency fication. MD Gaithersburg, NIST, Technology and Standards of Institute National 2018. 10 DC, Washington, School, Law University Georgetown report, Technical america. 6180.1000323. oeo eorpi information. demographic of Role learning. deep using graphics July Justice, of Institute 2011. National 235288, Report Technical conclusions. examiner subsequent of reliability eonto F 2017) In (FG Recognition score. residual of bounds using Florida,USA performance. algorithm recognition face in In rates match false and distributions imposter the on mogeneity Security and Forensics Information on Transactions E T XEC ECH paeGnrc aercgiineauto dma In idemia. @ evaluation recognition Face Gentric. ephane ´ rc 0t EEItrainlCneec nBoerc hoy plctosadSses TS21,Tampa 2019, BTAS Systems, and Applications Theory, Biometrics on Conference International IEEE 10-th Proc. S . S . UMMARY UMMARY etme 2019. September , ae 9–9,LsAaio,C,UA a 07 EECmue Society. Computer IEEE 2017. May USA, CA, Alamitos, Los 194–199, pages , as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT - AERCGIINVNO TEST VENDOR RECOGNITION FACE ora fBoerc n Biostatistics and Biometrics of Journal EETas nIfrainFrnisadSecurity and Forensics Information on Trans. IEEE 071t EEItrainlCneec nAtmtcFc Gesture & Face Automatic on Conference International IEEE 12th 2017 links/face/frvt/frvt2012/NIST ()2322 2008. 3(2):203–212, , rc nentoa aePromneConference, Performance Face International Proc. - DEMOGRAPHICS oebr2018. November , :2,1 06 doi:10.4172/2155- 2016. 11 7:323, , FRVT2012 T ()18–81 2012. 9 7(6):1789–1801, ,  api 0 Aug15.pdf. → → NR FNIR FNMR, M,FPIR FMR, → → IEEE 0 1 77 This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 P oahnPilp,Fn in,AhjtNrea,Jlan ya,adAieJ ’ol.A other-race An O’Toole. J. Alice and Ayyad, Julianne Narvekar, Abhijit Jiang, Fang Phillips, Jonathon P. [33] Dunlop, Joseph Bolme, David O’Toole, Alice Givens, Geof Draper, Bruce Beveridge, J. Phillips, Jonathon P. [32] Kings- Brian Wu, Chai-Wah Sattigeri, Prasanna Ratha, Nalini Pedapati, Tejaswini Muthukumar, Vidya [31] accuracy classification gender unequal understand to experiments Color-theoretic Muthukumar. Vidya [30] D .Mno .Rkht n .Zag c-ae rsrecognition. iris Dct-based Zhang. D. and Rakshit, S. Monro, facial M. D. on variation [29] threshold and age of impact The Malec. Chris and Yiu, Yee Sau Michalski, Dana [28] Na- Drake. Patrick and , Driscoll, K. Anne Osterman, J.K. Michelle Hamilton, E. Brady Martin, A. Joyce [27] Yr .Mlo n .A ahnn fcetadrbs prxmt ers egbrsac using search neighbor nearest approximate robust and Efficient Yashunin. A. D. and Malkov A. Yury [26] Links: J. Margaret McRae, F. Allan Macgregor, Stuart Liu, Z. Jimmy Zhu, Gu Duffy, L. David Larsson, Mats [25] better study: Us Bowyer. Kevin and Albiero, Vitor King, C. Michael Vangara, Kushal Charac- Krishnapriya, S. K. Bowyer. [24] Kevin and Albiero, Vitor King, C. Michael Vangara, Kushal Krishnapriya, S. K. [23] 08:14:00 2019/12/19 fetfrfc eonto algorithms. recognition face for effect problem. challenge face ugly the Computing and Vision bad, and the Image good, The Weimer. Samuel and Sahibzada, Hassan Lui, Yui images. Understanding face from Varshney. accuracy R. classification Kush gender and unequal Mojsilovic, Aleksandra Thomas, Samuel Kumar, Abhishek bury, In images. face from n ahn Intelligence Machine and In children. 2018. of February images using performance algorithm recognition 2018. November Statistics, National Vital Prevention, of and Division System, Control Statistics Disease Vital National for Statistics, Centers Health 8, for Center Report Technical reports. statistics vital tional irrhclnvgbesalwrdgraphs. world small navigable hierarchical eln.GA nig o ua rspten:Ascain ihvrat ngnsta influence that genes in variants with Associations E. development. Sarah patterns: pattern and neuronal iris normal Martin, human G. for Nicholas findings Montgomery, GWAS W. Grant Mackey, Medland. A. David Sturm, A. Richard Wright, bias. system face cut could quality image race. to relative accuracy recognition face http://arxiv.org/abs/1904.07325. in variability the terizing E T XEC ECH S . S . UMMARY UMMARY h EECneec nCmue iinadPtenRcgiin(CVPR) Recognition Pattern and Vision Computer on Conference IEEE The as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT 94:8–9,Arl2007. April 29(4):586–595, , 0178,0 2012. 03 30:177185, , - AERCGIINVNO TEST VENDOR RECOGNITION FACE C rn.Ap.Percept. Appl. Trans. ACM mrcnJunlo ua Genetics Human of Journal American imti ehooyToday Technology Biometric CoRR b/63030 2016. abs/1603.09320, , CoRR - DEMOGRAPHICS b/82009 oebr2018. November abs/1812.00099, , ()1:–41,Fbur 2011. February 8(2):14:1–14:11, , 095:1–1,2019. 12, – 2019(5):11 , nentoa ofrneo Biometrics on Conference International EETascin nPtenAnalysis Pattern on Transactions IEEE 92:3–4,Ags 2011. August 89(2):334–343, , CoRR T b/94035 2019. abs/1904.07325, ,  0 → → NR FNIR FNMR, M,FPIR FMR, ue2019. June , → → 0 1 78 , This publication is available free of charge from: https://doi.org/10.6028/NIST.IR.8280 DvdWie ae .Dn,AeadaC cmd n ihr .Km.Errrtsi sr fautomatic of users in rates Error Kemp. I. Richard and Schmid, C. Alexandra Dunn, D. James White, David [42] report, Technical software. recognition facial from people protect will that actions 10 West. of M. study Darrell A [41] Jain. Anil and Tan, Tieniu Chai, Zhenhua Feng, Jianjiang Paulino, Alessandra Sun, Zhenan [40] Iris report, Technical recognition. iris and face in effects demographic of comparison A Sirotin. Yevgeniy [39] recognition. image large-scale for networks convolutional deep Very Zisserman. A. and Simonyan K. [38] Links: In age-progression. adult normal of database image longitudinal a Morph: Tesafaye. T. and Ricanek K. [37] naming publicly of impact the Investigating auditing: recogni- Actionable Face Buolamwini. Joy and Bone. Raji Inioluwa M. [36] and Tabassi, E. Blackburn, M. D. Micheals, J. R. Grother, P. Phillips, P.J. [35] Jacque- Jackson, Kelsey Noyes, Eilidh Hahn, A. Carina Hu, Ying Yates, N. Amy Phillips, Jonathon P. [34] 08:14:00 2019/12/19 aercgiinsoftware. recognition face 10 DC, Washington, Initiative, Technology Emerging 2019. and Intelligence Artificial Institution, Brookings twins. identical in traits multibiometric 2019. 6 MD, Gaithersburg, Group, Experts 2015. In t nentoa ofrneo uoai aeadGsueRcgiin(FGR06) Recognition Gesture and Face Automatic on Conference International 7th 2019. 01 In 435, products. AI commercial of results performance Technology, biased and Standards of Institute National 6965, IR Report 2003. Evaluation March www.frvt.org, or www.itl.nist.gov/iad/894.03/face/face.html 2002. test vendor tion 2018. 115(24):6171–6176, algorithms. exam- recognition forensic face of accuracy and recognition superrecognizers, Face O’Toole. iners, J. Alice and White, David Chellappa, Rama Castillo, G Cavazos, G. line E T XEC ECH rc nentoa ofrneo erigRepresentations Learning on Conference International Proc. S . S . UMMARY UMMARY rlieJcen aevRna,SaiSnaaaaaa,JnCegCe,Cro D. Carlos Chen, Jun-Cheng Sankaranarayanan, Swami Ranjan, Rajeev Jeckeln, eraldine ´ as eaie aldascaino n ujc : NR1NFNIR 1:N FPIR FNMR 1:N 1:1 FMR 1:1 subject one of association Failed subjects negative: two False of association Incorrect positive: False FRVT LSONE PLoS - AERCGIINVNO TEST VENDOR RECOGNITION FACE coe 2015. October , rc fteItrainlSceyo pia Engineering Optical of Society International the of Proc. ofrneo I tisadSociety and Ethics AI, on Conference rceig fteNtoa cdm fSciences of Academy National the of Proceedings ouehttps://arxiv.org/abs/1409.1556v6, volume , - DEMOGRAPHICS ae 4–4,Arl2006. April 341–345, pages , T  0 → → NR FNIR FNMR, M,FPIR FMR, 67 42010. 04 7667, , ae 429– pages , → → 0 1 79 ,