International Journal of Industrial Electronics and Electrical Engineering, ISSN: 2347-6982 Volume-3, Issue-8, Aug.-2015 SUSPECT IDENTIFICATION BY MATCHING COMPOSITE SKETCH WITH MUG-SHOT

1SHUBHANGI A. WAKODE, 2SUNIL R. GUPTA

1,2Department of Electronics Engineering, J. D. College of Engineering & Management Nagpur, M.S., INDIA E-mail: [email protected], [email protected]

Abstract- To match composite sketch-to-mug shots several questions rose for consideration or solution, approach we propose here will provide possible solution. We propose a component-based representation (CBR) approach to measure the similarity between a composite sketch and mugshot that evidences advances in matching performance over the more common mug-shot to composite sketches matching methods. To determine the identicalness of a person who is guilty of crime or serious offense composite sketch to facial image is commonly used technique. Composite sketches are composed using facial composite software. In this paper emphasis is made on matching composite sketches to mug-shots that gives better performance over the more common matching proficiency exists already. The component based theoretical account presented in this paper chiefly consists of the following steps. 1) Renormalize mugshot using a geometric transformation, image enhancement and space conversion. 2) Facial constituent localization using a Viola-Jones algorithm and active shape model (ASM) 3) per component feature extraction using nearest neighbor symmetry and matrix normalize cross-correlation followed by computing length between facial constituent. 4) Per component similarity measurement followed by score normalization and fusion. The proposed system has been evaluated on mugshot –composite sketches pairs with hundreds of individuals. The results are advocating.

Index Terms- ASM, CBR, Composite Sketch, Forensic Sketches, Mugshot, Viola-Jones Algorithm.

I. INTRODUCTION becomes a perfect alternative to provide assistant in investigation. Using composite sketches is Procession in biostatistics upgrades the law advantageous as it is less time consuming and more enforcement agencies by providing a means of economic. Some of the most widely used facial identifying criminals with lesser amount of time. composite software kits include EvoFIT, Photo-Fit Visual Biometric along with Fingerprint recognition ,FACES, Mac-a-Mug and IdentiKit. Some are used to provides Face recognition which gives the analysis of detect principal Facial Components such as Hair, facial features for the recognition of an individual’s Eyebrow, eyes, nose, mouth, shape, eyeglasses, etc. identity. In numerous cases the facial photograph of a and subaltern Facial Components such as Smile lines, suspect is unavailable. In these considerations, moles, scar and tattoo, etc. Figure (1) shows the drawing a sketch comply the description provided by a procedure for creating a composite using Identikit. In spectator or the victim is a commonly used method to this facial composite system, each facial component is assist the police to identify possible suspects. selected from a candidate list shown on system GUI. The difference between forensic sketches (hand drawn sketches) and computer generated composite sketches can be seen in Figure (2). Compared to face photos (mug-shot), both hand drawn sketches and composite sketches lack detailed texture, especially around the forehead and cheeks. However, artists can depict each facial component with an exact shape, and even shading. Thus, artist drawn sketches can usually capture the most distinctive characteristics of different faces. By contrast, facial components in a composite sketch must be approximated by the most similar component available in the composite software’s database.

Fig.1. Procedure of creating a composite sketch using Identikit. Moreover, the psychological mechanism of an artist guarantees that hand drawn sketches look raw while To draw forensic sketches police artist requires a large composite sketches may look stilted. Therefore, while amount of training in drawing and pictorialization. we may easily recognize a person from his Composite sketches instead requires several hours of hand-drawn sketch, it is often more challenging to training which allows even non artist to compose a identify a person from a composite sketch. Similar sketch with the help of composite sketch software, observations have also been reported in the cognitive

Suspect Identification By Matching Composite Sketch With Mug-Shot

25 International Journal of Industrial Electronics and Electrical Engineering, ISSN: 2347-6982 Volume-3, Issue-8, Aug.-2015 psychology community, a survey showed that 80% of generated from the sketch in these works; and then the officers in law enforcement agencies used matching is performed with standard face recognition computer generated composites [2]. algorithms. Only one paper is published in forensic Despite this high percentage of law enforcement sketch matching till date. Klare and Jain [8] published agencies using computer generated composites, the a Local Feature based Discriminant Analysis (LFDA) application of automated face recognition algorithms approach for matching forensic sketches to to computer generated composites has not been photos. adequately studied [3]. By contrast, sketch There are a lot of problems in forensic sketch recognition, both viewed sketches and forensic recognition compared to normal face recognition in sketches has received relatively more attention. In this which both probe and gallery images are photographs. paper, we present a study of a face recognition system The fineness of sketches whether they may be viewed to match computer generated facial composites to sketches or forensic are different from large mug-shot facial photographs or mugshots. The design objectives gallery. Most of the work done previously is of the proposed work are (i) to provide a common principally focused on forensic sketches or viewed representation for both composite sketches and face sketches. Forensic sketches have additional problems photos that can taper intrapersonal variations while compared to viewed sketches. Due to the fractious still maintaining interpersonal distinguishable nature of the memory, the exact visual aspect of the features, (ii) leverage the component-based approach criminal cannot be remembered by the spectator. This by which computer generated composites are formed, leads to an incomplete and inaccurate depiction of the and (iii) efficaciously match composite sketches sketches which reduces the recognition performance against copious mugshot gallery databases. considerably. The remainder of this paper is structured as follows. To handle such difficulties, Klare et al. developed a Section II introduces related work on hand drawn local feature-based discriminant analysis (LFDA), sketch recognition. The proposed component based which learns a discriminative representation from representation (CBR) for composite sketch partitioned vectors of SIFT [9] and LBP [10] features recognition is presented in Section III. In Section IV, using multiple discriminative subspace projections. experiments are performed to verify the effectiveness Component-based face recognition methods (which of the proposed CBR in matching composite sketches are used in this study) were studied in [11]-[15]; to large scale mugshot gallery sets. however, these algorithms either directly utilized intensity features that are sensitive to changes in facial appearance or employed supervised algorithms for classification whose performance is sensitive to the amount of training data available. Moreover, these algorithms were mainly proposed to resolve the misalignment problem in photo-to-photo face matching, and do not address the heterogeneous modality gap present when matching computer generated composite sketches to facial photographs.

III. COMPONENT BASED REPRESENTATION (CBR)

Most of the facial composite software systems which law enforcement agencies utilize now a days basically Fig.2. Difference between hand drawn sketches and composite are component based systems. To alleviate the use of sketches (a) Face photographs, (b) the corresponding viewed the system, these software kits provide a predefined set sketches, (c) the corresponding computer generated composite sketches synthesized using FACEs [4]. Both the viewed sketches of human facial components. A composite sketch is and composite sketches were constructed while viewing the face constructed by individually selecting each facial photographs. component following the description of a witness or victim. Inspired by the principle of facial composite II. RELATED WORK systems and existing work on component-based methods in photo-to-photo matching, we apply a Research on sketch matching started only since a component based representation (CBR) [1] for decennium. Because of the inaccessibility of standard matching composite sketches to facial photographs. public database for forensic sketches, most of the Presently most of the facial composite software research is done on viewed sketches only for last 10 systems which law enforcement agencies utilize are years. On viewed sketches, most of the early work is component based systems fundamentally. To alleviate done by Tang et al. [5]-[7]. A Synthetic photograph is the use of the system, these software kits provide a

Suspect Identification By Matching Composite Sketch With Mug-Shot

26 International Journal of Industrial Electronics and Electrical Engineering, ISSN: 2347-6982 Volume-3, Issue-8, Aug.-2015 predefined set of human facial components. A Jones. It was prompted majorly by the problem of face composite sketch is constructed by individually detection. It is a paradigmatic method for real-time selecting each facial component following the object detection. Its training is slow as the objects description of a witness or victim. Inspired by the other than facial elements are recognized by it but principle of facial composite systems and existing component detection is quick. In order to radically work on component-based methods in photo-to-photo reduce computation time the new model determines matching, we apply a component based representation the locations and sizes of human faces in digital (CBR) [1] for matching composite sketches to facial images by extracting region of interest and apply photographs. Viola Jones proper object detector in the area of A. Pre-processing of mugshot using a geometric interest in order to limit search area. An Active Shape transformation, image enhancement and Model (ASM) [17] utilized to detect a set of facial conversion. landmarks, and then each component is localized B. Facial elements localization using a according to the predefined key points in the ASM Viola-Jones algorithm and active shape Model. Specifically 20 facial landmarks are detected. model (ASM) followed by computing length Figure 3(a) is input image to which proposed between elements. algorithm is applied and output shown in figure 3(b) C. Per component feature extraction using with five facial features enclosed within a rectangular nearest neighbor symmetry and matrix box, figure 3(c) shows the image annotated with 20 normalize cross-correlation. landmarks. D. Per component similarity measurement, followed by score normalization and fusion. C. Component Feature Extraction Each of the above steps is detailed in the following It is a commonly known fact that human face poses sections. symmetry vertically, in feature extraction process we made an effort to use that symmetries, also facial A. Mug-shot pre-processing components on either sides are alike so it is again In spite of the fact that most of the photographs in the advantageous to select features in a single image. We mugshot databases maintained by law enforcement use nearest neighbors to limit the number of agencies are captured with the cooperation of subjects, correspondences, making the subsequent matching there are still some variations in face pose and scale. process more efficient; in practice, an approximate Most facial composite systems generate frontal modified nearest neighbor scheme is used. The sketches, but there are also variations in face scale. To contribution of the neighbors to the classification is handle these variations in pose and scale, a geometric weighted according to its distance from the test transformation (rotation and scaling) is first applied to pattern. In it the closest pattern is given more both composite sketches and facial photographs. The significance than the farthest pattern. The weightage geometrically normalized face images are then given to the class of the first closest pattern is more cropped to the same size, which includes the whole than for the second closest pattern and so on. Next task face (including hair, chin and ears) .The cropped is to do Normalize Cross-Correlation [18], [19] and image, is enhance by contrast adjustment, DE blurring find Coordinates of mug shot as well as of composite etc. Colored mug-shots enunciates facial features sketches. Calculate the normalized cross-correlation. prominently and it becomes practicable while facial components are detected in spite of pose and matching face photos to mug-shot images, but when it scale variation. comes on matching mug-shot to sketch matching it is essential to get them in same tone. So color space conversion is applied to image.

B. Facial elements localization Mug-shot photographs are normally captured with the cooperation of the subjects, and the composite sketches are typically synthesized in frontal pose. The pose and scale variations between photographs and composites can be effectively eliminated with the above mentioned face normalization step. In the (a) (b) (c) proposed method, a more efficient approach is utilized Fig.3. Facial element localization (a) input image, (b) Facial for localizing the facial elements from pre-processed elements detection, (c) Facial landmark recognition using ASM. frontal photographs and composites. A Viola-Jones algorithm [16] is the first object detection framework We extracted total 12 features by using nearest to provide competitive object detection rates in neighbor symmetry and matrix normalize real-time proposed in 2001 by Paul Viola and Michael cross-correlation, those 12 feature are shown in fig 5

Suspect Identification By Matching Composite Sketch With Mug-Shot

27 International Journal of Industrial Electronics and Electrical Engineering, ISSN: 2347-6982 Volume-3, Issue-8, Aug.-2015 which are as follows: Eye width, Eye height, Nose Tanh score normalization achieved a slightly higher width, Nose height, Mouth width, Mouth height, accuracy over z-score normalization. Distance between left eye to nose, Distance between right eye to nose, Distance between left eye to mouth, Distance between right eye to mouth, Distance between nose to mouth, Distance between both eyes. Figure (4) shows detected face parts and measured length of facial components, advantage of proposed algorithm is that facial components are detected and length amongst facial component is calculated in spite of pose and scale variation.

Fig.5. For each facial component Similarities are first calculated for corresponding blocks, and then summed up to get the component similarity.

IV. EXPERIMENTAL RESULTS

Fig.4. Features are extracted using nearest neighbor symmetry In this section, we study the performance of the and matrix normalize cross-correlation proposed CBR approach for matching composite sketches to facial photographs. A mug shot-composite D. Evaluate Component similarity and Fusion database is first built for this experimental study. We A facial component is divided into n patches. Let Ai analyze the discriminability of the different facial and Bi be the photo patch and the corresponding components for composite-to-photo matching. The composite patch, respectively (see figure 5). With accuracy of the proposed CBR method is then nearest neighbor symmetry and matrix normalize evaluated using MATLAB Version: 8.1.0.604 cross-correlation followed by computing length (R2013a). between facial constituent features are extracted for Ai Database of 300 mug-shot and composite pairs is use and Bi, the similarity Si between Ai and Bi is to perform this experiment. Firstly preprocessing of calculated based on the normalized histogram images and composite sketches are done then features intersection. Specifically, Si calculated as are extracted for both followed by score normalization D j j and fusion. Maximum matching scores for composite m i n (HHAB , )  j  1 i i sketch to mug shot matching is shown in figure (6), s i  DD m i n (HHj , j ) which is 99.93% for matched pair. ABi  i j1 j  1 where HA and HB are D dimensional histogram that are extracted from Ai and Bi. Next, the similarities for all corresponding blocks are summed up to obtain the total component similarity SNose (for example). The similarities for the other facial components (hair, eyebrow, eyes and mouth) are also calculated in the same manner. We also include the facial shape similarity by calculating the cosine distance between two facial landmark vectors. Finally, all the component similarities are again summed up to obtain the final matching score between a facial photograph and a composite sketch. However, the similarities calculated for different facial components need not be in the same numerical scale (range). Therefore, score normalization is applied to the match scores from each facial component prior to combining them. Tanh and z-score are the two most preferred score normalization methods in biometrics [20] while both these Fig.6. Evaluated result having matching scores of 99.93% as normalization schemes had similar performance; the given in the GUI

Suspect Identification By Matching Composite Sketch With Mug-Shot

28 International Journal of Industrial Electronics and Electrical Engineering, ISSN: 2347-6982 Volume-3, Issue-8, Aug.-2015 Table: I [3] Yuen P. C., Man C. H.: Human face image searching system using sketches. IEEE Trans. Syst., Man, Cybern. A, Syst. Matching performance of proposed composite Humans, vol. 37, no. 4, pp. 493–504, Jul. 2007 sketch-mugshot matching algorithm, CBR using [4] FACES 4.0, IQ Biometrix 2011 [Online]. Available: MLBP http://www.iqbiometrix.com [5] X. Wang and X. Tang. Face sketch recognition. IEEE Transactions on Circuits and Systems for Video Technology, 14(1):50–57, January 2004. [6] Q. Liu, X. Tang H. Jin, H. Lu, and S. Ma. A nonlinear approach for face sketch synthesis and recognition. Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, pages 1005–1010, 2005. [7] X. Wang and X. Tang. Face photo-sketch synthesis and recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), 31(11):1955–1967, 2009. [8] B. Klare, Z. Li, and A.K. Jain. Matching forensic sketches to mug CONCLUSION shot photos. IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 639–646, March 2011. [9] Lowe, “Distinctive image features from scale-invariant key This paper investigates a challenging heterogeneous points,” Int. J. Computer. Vis., vol. 60, no. 2, pp. 91–110, 2004. face recognition problem: matching composite [10] T. Ojala, M. Pietikainen, and T. Maenpaa, “Multiresolution sketches to facial photographs. Face varies in rotation, gray-scale and rotation invariant texture classification with local binary patterns,” IEEE Trans. Pattern Anal. Mach. Intell., vol. brightness level, size, etc. in different images even for 24, no. 7, pp. 971–987, Jul. 2002. the same person. Those features are independent of [11] B. Heisele, P. Ho, J. Wu, and T. Poggio, “Face recognition: face features and will affect the recognition rate Component- based versus global approaches,” Comput. Vis. Image Understand., vol. 91, no. 1, pp. 6–21, Jul. 2003. significantly. One method to solve the problem we [12] Y. Ivanov, B.Heisele, and T. Serre, “Using component features used face normalization using geometric for face recognition,” in Proc. Automatic Face and Gesture transformation. Face detection is a computer Recognition, 2004, pp. 421–426. technology that identifies human faces in digital [13] Lin and X. Tang, “Recognize high resolution faces: From macrocosm to microcosm,” in Proc. IEEE Computer Vision and images. It detects human faces which might then be Pattern Recognition, 2006, pp. 1355–1362. used for recognizing a particular face. We detect the [14] B. Klare, A. Paulino, and A. Jain, “Analysis of facial features in face by using Viola–Jones object detection technique. identical twins,” in Proc. Int. Joint Conf. Biometrics, 2011, pp. 1–8. After that we extract the 12 feature component of [15] J. Huang, V. Blanz, and B. Heisele, “Face recognition using detected face it is may be composite sketches or mug component- based SVM classification and morphable models,” shot. a GUI, provides an intuitive interface between in Proc. First Int. Workshop on Pattern Recognition with Support Vector Machines, 2002, pp. 334–341. the user and the programming language. A GUI [16] P. Viola and M. Jones. Rapid object detection using a boosted allows the user to bypass MATLAB commands cascade of simple features. In Proc. of IEEE Conference on altogether and, instead, to execute programming Computer Vision and Pattern Recognition, Kauai, HI, December routines with a simple mouse click or key press 2001. [17] T. Cootes, C. Taylor, D. Cooper, and J. Graham, “Active shape therefor we build the GUI. models their training and application,” Computer. Vis. Image Understand., vol. 61, no. 1, pp. 38–59, Jan. 1995. REFERENCES [18] Yufan Wang, Qiuze Yu, Wenxian Yu, ―An Improved Normalized Cross Correlation algorithm for SAR Image Registration‖, IEEE IGARSS, 2012. [1] Han H., Klare B. F., Bonnen K., Jain A.K.: Matching Composite [19] Vanderbrug, G.J, Rosenfeld, ―Two-Stage Template Matching,‖ Sketches to Face Photos: A Component-Based Approach. IEEE IEEE Transactions on Computers, Vol. 60, Issue 11, 1977. Transactions on Information Forensics and Security, Vol. 8, No. [20] Jain, K. Nandakumar, and A. Ross, “Score normalization in 1, January 2013 multimodal biometric systems,” Pattern Recognition., vol. 38, [2] Mcquiston D., Topp L., Malpass R.:Use of facial composite no. 12, pp. 2270–2285, Dec. 2005 systems in US law enforcement agencies. Psychology, Crime and Law, vol. 12, no. 5, pp. 505–517, 2006



Suspect Identification By Matching Composite Sketch With Mug-Shot

29