contributed articles DOI:10.1145/2818990 Fusing information from multiple biometric traits enhances authentication in mobile devices. BY MIKHAIL I. GOFMAN, SINJINI MITRA, TSU-HSIANG KEVIN CHENG, AND NICHOLAS T. SMITH Multimodal Biometrics for Enhanced Mobile Device Security security, military, and law-enforce- MILLIONS OF MOBILE devices are stolen every year, ment applications,15,18 they are not yet along with associated credit card numbers, passwords, widely integrated into consumer mo- and other secure and personal information stored bile devices. This can be attributed to therein. Over the years, criminals have learned implementation challenges and con- to crack passwords and fabricate biometric traits key insights and have conquered practically every kind of ˽ Multimodal biometrics, or identifying people based on multiple physical and user-authentication mechanism designed to stop behavioral traits, is the next logical step toward more secure and robust them from accessing device data. Stronger mobile biometrics-based authentication in authentication mechanisms are clearly needed. mobile devices. ˽ The face-and-voice-based biometric Here, we show how multimodal biometrics system covered here, as implemented on promises untapped potential for protecting consumer a Samsung Galaxy S5 phone, achieves greater authentication accuracy in mobile devices from unauthorized access, an uncontrolled conditions, even with poorly lit face images and voice samples, than authentication approach based on multiple physical single-modality face and voice systems. and behavioral traits like face and voice. Although ˽ Multimodal biometrics on mobile devices can be made user friendly multimodal biometrics are deployed in homeland for everyday consumers. 58 COMMUNICATIONS OF THE ACM | APRIL 2016 | VOL. 59 | NO. 4 cern that consumers may find the ap- (such as fingerprints and iris scans) several critical issues remain, including, proach inconvenient. into the system. We hope our experi- for example, techniques for defeating We also show multimodal biomet- ence encourages researchers and mo- iPhone TouchID and Samsung Galaxy rics can be integrated with mobile bile-device manufacturers to pursue S5 fingerprint recognition systems.2,26 devices in a user-friendly manner the same line of innovation. Further, consumers continue to com- and significantly improve their secu- plain that modern mobile biometric rity. In 2015, we thus implemented a Biometrics systems lack robustness and often fail to multimodal biometric system called Biometrics-based authentication es- recognize authorized users.4 To see how Proteus at California State University, tablishes identity based on physical multimodal biometrics can help ad- Fullerton, based on face and voice and behavioral characteristics (such dress these issues, we first examine their on an Samsung Galaxy S5 phone, in- as face and voice), relieving users from underlying causes. tegrating new multimodal biometric having to create and remember secure authentication algorithms optimized passwords. At the same time, it chal- The Mobile World for consumer-level mobile devices lenges attackers to fabricate human One major problem of biometric au- and an interface that allows users traits that, though possible, is difficult thentication in mobile devices is sam- to readily record multiple biometric in practice.21 These advantages con- ple quality. A good-quality biometric traits. Our experiments confirm it tinue to spur adoption of biometrics- sample—whether a photograph of achieves considerably greater authen- based authentication in smartphones a face, a voice recording, or a finger- tication accuracy than systems based and tablet computers. print scan—is critical for accurate solely on face or voice alone. The next Despite the arguable success of bio- identification; for example, a low- IMAGE BY ANDRIJ BORYS ASSOCIATES/SHUTTERSTOCK ANDRIJ BORYS BY IMAGE step is to integrate other biometrics metric authentication in mobile devices, resolution photograph of a face or APRIL 2016 | VOL. 59 | NO. 4 | COMMUNICATIONS OF THE ACM 59 contributed articles noisy voice recording can lead a bio- security and robustness challenges spite the recent popularity of biomet- metric algorithm to incorrectly iden- confronting today’s mobile unimodal ric authentication in consumer mobile tify an impostor as a legitimate user, systems13,18 that identify people based devices, multimodal biometrics have or “false acceptance.” Likewise, it can on a single biometric characteristic. had limited penetration in the mo- cause the algorithm to declare a legit- Moreover, deploying multimodal bio- bile consumer market.1,15 This can be imate user an impostor, or “false re- metrics on existing mobile devices is attributed to the concern users could jection.” Capturing high-quality sam- practical; many of them already sup- find it inconvenient to record multiple ples in mobile devices is especially port face, voice, and fingerprint recog- biometrics. Multimodal systems can difficult for two main reasons. Mobile nition. What is needed is a robust us- also be more difficult to design and users capture biometric samples in a er-friendly approach for consolidating implement than unimodal systems. variety of environmental conditions; these technologies. Multimodal bio- However, as we explain, these factors influencing these conditions metrics in consumer mobile devices problems are solvable. Companies include insufficient lighting, differ- deliver multiple benefits. like Apple and Samsung have invest- ent poses, varying camera angles, and Increased mobile security. Attack- ed significantly in integrating bio- background noise. And biometric ers can defeat unimodal biometric metric sensors (such as cameras and sensors in consumer mobile devices systems by spoofing a single biomet- fingerprint readers) into their prod- often trade sample quality for por- ric modality used by the system. Es- ucts. They can thus deploy multimod- tability and lower cost; for example, tablishing identity based on multiple al biometrics without substantially the dimensions of an Apple iPhone’s modalities challenges attackers to increasing their production costs. TouchID fingerprint scanner prohibit simultaneously spoof multiple inde- In return, they profit from enhanced it from capturing the entire finger, pendent human traits—a significantly device sales due to increased security making it easier to circumvent.4 tougher challenge.21 and robustness. In the following sec- Another challenge is training the More robust mobile authentication. tions we discuss how to achieve such biometric system to recognize the When using multiple biometrics, one profitable security. device user. The training process is biometric modality can be used to based on extracting discriminative compensate for variations and quality Fusing Face and Voice Biometrics features from a set of user-supplied deficiencies in the others; for example, To illustrate the benefits of multimod- biometric samples. Increasing the Proteus assesses face-image and voice- al biometrics in consumer mobile de- number and variability of training recording quality and lets the highest- vices, we implemented Proteus based samples increases identification ac- quality sample have greater impact on on face and voice biometrics, choosing curacy. In practice, however, most the identification decision. these modalities because most mo- consumers likely train their systems Likewise, multimodal biometrics bile devices have cameras and micro- with few samples of limited variabil- can simplify the device-training proc- phones needed for capturing them. ity for reasons of convenience. Mul- ess. Rather than provide many training Here, we provide an overview of face- timodal biometrics is the key to ad- samples from one modality (as they and voice-recognition techniques, dressing these challenges. often must do in unimodal systems), followed by an exploration of the ap- users can provide fewer samples from proaches we used to reconcile them. Promise of Multimodal Biometrics multiple modalities. This identifying Face and voice recognition. We used Due to the presence of multiple pieces information can be consolidated to the face-recognition technique known of highly independent identifying in- ensure sufficient training data for reli- as FisherFaces3 in Proteus, as it works formation (such as face and voice), able identification. well in situations where images are multimodal systems can address the A market ripe with opportunities. De- captured under varying conditions, as Figure 1. Schematic diagram illustrating the Proteus quality-based score-level fusion scheme. Face Matching Minimum Accept Match t1 Threshold (T) Luminosity Face Quality Q1 Face Sharpness Match Score Extraction Score Normalization Contrast Generation S1 Face Image Face Quality w1 Assessment If (S1 * w1 + S2 * w2 ≤ T) Weight Decision = grant Decision Assignment w 2 else Decision = deny Voice Signal Voice Quality S Assessment 2 Q2 Match Score Denoising SNR Normalization t2 Voice Matching 60 COMMUNICATIONS OF THE ACM | APRIL 2016 | VOL. 59 | NO. 4 contributed articles expected in the case of face images ob- signals, then normalizes the SNR value tained through mobile devices. Fisher- to the [0, 1] range using min-max nor- Faces uses pixel intensities in the face malization. images as identifying features. In the Multimodal biometric fusion. In future, we plan to explore other face- multimodal biometric systems, infor- recognition techniques, including Ga- To get its algorithm