Application of a Post-Processing Algorithm

Application of a Post-processing Algorithm for Improved Human Face Recognition

Metin Artiklar and Mohamad H. Hassoun Paul Watta Dept. Electrical and Computer Engineering Dept. Electrical and Computer Engineering Wayne State University University of Michigan-Dearborn Detroit, MI 48202 Dearborn, MI 48128

[email protected] [email protected]

Abstract moved 2 pixels up, the average Hamming distance is 14.6%.

This paper presents a shifting algorithm which Figure 2 shows that even a 1-pixel shift in the can be used in a pattern recognition system to image can yield a large change in Hamming improve the system’s performance in the distance. presence of shifted input patterns. The algorithm is outlined and simulation results are presented for some human face recognition experiments. It is shown that the shifting algorithm improves recognition performance for several different face recognition algorithms.

1. Introduction

(a) (b) The problems of shift, rotation, and scaling are troublesome for image processing applications, Figure 1. (a) An 82x115 face image and (b) such as automatic recognition systems. This the same image cropped to 72x72. paper presents an algorithm which can be used to improve the classification performance of such a system in the presence of shifted images. e c

Figure 1(a) shows an example of an 82x115- n a t

dimensional face image. To facilitate the shifting s i d process, the image is cropped to a size of 72x72, g n as shown in (b). The image is shown in gray i scale, but it can be made binary by simply m m thresholding the gray levels at 127. a H

Figure 2 shows average Hamming distances (on A

the binary images) computed among 100 face % images such as (b). The Hamming distance is computed between the image and itself, but with Up/down shift Left/right shift various amounts of shift and various directions of the shift. In the middle of the Figure, since the left/right and top/bottom shifts are 0, the Figure 2. The average Hamming distance (in Hamming distance is 0 (no shift is applied). As percent and averaged over 100 faces) between a the image is moved 1 pixel to the right, the face image and shifted versions of itself. The 0 in Hamming distance is 10.4% (on average). If it is the center represents the case where no shift is applied. Note that all the images discussed here were Hamming distance in the four nearest neighbors: obtained in a laboratory setting using an north, south, east, and west. In this case, the apparatus which constrains the amount of shift, Hamming distance is smallest in the south rotation, scale, and tilt of the face. In particular, direction, hence the path moves in that direction. the experimental setup consists of a frame This process is continued until no further attached to a tripod. The subject puts his or her improvements can be made, as shown in the final head in the frame, and the picture is snapped. Hamming distance value of 10.4%. This setup eliminates the need for segmenting the face from the rest of the image. Here, by applying this shifting process, the distance between the two images decreases by Even with this constrained method of snapping 41%. Increasing the match between a test image the images, there can still be an appreciable of a person and a stored prototype is desirable in amount of shift present in (different) images of many face recognition systems. the same person. Figure 3 shows two different pictures of the same individual. The images were Unfortunately, this process can also improve the snapped within minutes of each other. Figure 4 Hamming distance for images of different shows the resulting Hamming distances as image people. For example, Figure 5 shows two images (b) is shifted and compared to image (a). of different people, and Figure 6 shows the corresponding Hamming distances as image (b) is shifted and compared to image (a). Again, a path can be taken which lowers the Hamming distance, but notice that the improvement in Hamming distance here (28%) is smaller than the case of the same individual.

(a) (b)

Figure 3. Two different 72x72 images of the same individual.

29.4 28.5 27.5 27.3 27.3 28.5 29.4 30.5 31.5 (a) (b)

27.9 27.0 25.8 25.4 25.6 26.4 27.8 29.1 30.6 Figure 5. Images of two different people. 27.0 25.4 24.2 23.5 23.7 25.0 25.9 27.6 29.9

25.8 23.7 22.0 21.4 21.4 22.8 24.5 26.8 29.2 25.3 26.4 28.2 30.7 31.8 32.8 33.7 35.4 37.5

25.0 22.3 19.6 18.0 17.7 20.1 22.5 25.4 27.6 23.7 25.3 27.3 29.5 31.3 32.5 33.6 35.2 37.4

24.0 21.2 17.2 14.2 13.9 17.0 21.0 24.4 27.1 23.1 24.5 27.1 29.2 31.2 32.9 33.7 35.2 37.5

23.7 19.7 15.7 11.3 10.4 15.8 20.4 23.2 26.2 23.6 25.5 27.3 29.8 31.9 33.3 34.2 35.7 38.0

24.2 20.8 16.9 12.9 12.8 16.6 20.4 23.5 26.9 24.3 26.2 28.3 30.5 32.6 34.0 34.8 36.3 38.5

25.4 22.8 19.8 17.5 17.0 19.2 22.0 25.2 27.9 25.6 27.1 29.3 31.4 33.5 34.6 35.1 36.6 38.7

26.8 28.7 30.9 32.7 34.3 35.2 35.7 37.0 38.9 Figure 4. This diagram shows the Hamming distance (in percent) between image 3(a) and 28.0 29.5 32.1 33.6 34.9 36.1 36.7 37.8 39.5 shifted versions of image 3(b). 30.0 31.8 33.5 34.9 35.8 37.2 38.0 39.1 40.8

Figure 6. This diagram shows the Hamming At the center value in the table (where neither distance (in percentage) between the two images image is shifted), the Hamming distance is shown in Figure 5 with various amounts of shift. 17.7%. Starting from this position, a path can be outlined (shown underlined) which seeks to minimize the Hamming distance between the two Of course, for recognition purposes, it is desired images. This is accomplished by examining the that the distance between images of different people be as large as possible. The conjecture along the path (and those distances surrounding here is that even though the shifting process the optimal path). decreases the distance between different people, it tends to do so by a lesser amount than the In addition, from a computational point of view, distance improvement for images of the same there is no need to actually shift the pixels of the person. Hence, there is an overall increase in image at each step. Rather, one need only store the separation ability of the classifier. the current corner point of the shifted image and use that as an offset index when computing the The process of shifting the input image to obtain required Hamming distance. Hence, this a better match between images (of the same algorithm can be implemented very efficiently. individual taken at different times) is the basis In fact, the computational burden of computing for the post-processing algorithm which is the optimal path consists of computing 4 described in the next section. Hamming distances at each step (on the 72x72 images), along with finding the minimum of 2. The Shifting Algorithm those 4 values. Many recognition system use several prototypes This method requires the use of a pattern of patterns to be recognized. For example, in the recognition scheme which can produce an context of face recognition, it is possible to store ordered list of outputs ranked according to several images of each person. We have similarity to an input pattern. In our case, we use constructed such a data set, containing 400 a simple nearest neighbor classification scheme images of 100 different people. Each person has and rank the outputs on the basis of Hamming 4 images showing different facial expressions: a distance (in the case of binary images) and city blank expression, smile, angry, and surprised. block distance (in the case of gray-scale images). In this case, it is possible to select the top 10 In this post-processing algorithm, we select two winners (k = 10) and then also include all 4 parameters: k, the number of patterns from the expressions of each person as part of the shifting ordered list which will participate in the shifting process. Hence, 40 images participate in the process (i.e., the k patterns at the top of the list), shifting post-processing algorithm. In the results and s, the maximum path length in the shifting below, this method is called Shift-10. process. To minimize computational time, k Alternatively, we can simply select the top 40 should be chosen as small as possible, but images (k = 40) and apply the shift to these sufficiently large so that there is a high images. This method is referred to as Shift-40. probability that the correct image is contained in this subset of patterns. 3. Results As illustrated in the previous section, each of the k candidate output patterns is shifted by 1 pixel We tested the performance of this post- in 4 possible directions: north, east, south, and processing algorithm on a data set consisting of west. After computing a similarity measure for 400 72x72-dimensional face images (100 each of these directions, we select the direction different people), as described in the previous which yields the lowest distance and move the section. Note that an additional test image was kth pattern in that direction. This process is taken of each person in the data set (a blank repeated at most s times, or until there is no expression). improvement in the distance measure in any direction. The final distance is recorded for each Figure 7 shows the results of the simulations of the k patterns, and the image which has the before and after the post-processing algorithm on least distance after this shifting process is taken binary face images using two different as the output of the system (the best match). classification methods: a Hamming distance method and a two-level decoupled Hamming Note that when computing the optimal path, it is network (Watta, Akkal, and Hassoun, 1997; not necessary to generate an entire grid of all Ikeda, Watta, and Hassoun, 1998). The numbers possibilities, as shown in Figure 5. Rather, we in column 1 indicate the percentage that the need only compute those Hamming distances correct person was ranked first in the list, and columns 2 and 3 indicate what percentage of the time the correct person was ranked second and system, which includes a mechanism for third, respectively. rejecting an image when there is an insufficient match between the input and one of the memory For these results, s was set at 5 and k was set at patterns. 40 (using the Shift-10 and Shift-40 approaches as described in the previous section). Post Algorithm 1 2 3 Processing Post Algorithm 1 2 3 Processing Hamming Classifier 94 82 52

Hamming Classifier 94 73 45 None 2-Level Hamming 94 75 52 None 2-Level Hamming 79 58 41 Wavelet Classifier 97 80 62

Hamming Classifier 97 87 47 Hamming Classifier 97 89 65 Shift-10 2-Level Hamming 97 88 53 Shift-10 2-Level Hamming 95 91 70

Hamming Classifier 97 87 46 Wavelet Classifier 98 93 76 Shift-40 2-Level Hamming 98 87 46 Hamming Classifier 98 89 65 Shift-40 2-Level Hamming 95 85 61 Figure 7. Results of the post-processing algorithms for binary face images. Wavelet Classifier 97 90 75

Figure 8 shows the results of the post-processing Figure 8. Results of the post-processing algorithms when the gray-scale images are used. algorithms for gray-scale images. Here, 3 different recognition schemes are used: a nearest neighbor classifier (using the city block distance as the similarity measure), the two-level Acknowledgments decoupled Hamming network, and a wavelet face recognition algorithm (Stollnitz, DeRose, and This work was supported by the National Salesin, 1995; Jacobs, Finkelstein, and Salesin, Science Foundation (NSF) under contract ECS- 1995). 9618597.

The results indicate that the proposed post- References processing shifting algorithm improves overall system performance. Most notable is the 1. Ikeda, N., Watta, P., and Hassoun, M. (1998). improvement for the patterns which appear “Capacity Analysis of the Two-Level Decoupled second and third in the ordered list. For example, Hamming Associative Memory” Proceedings of for binary images using the Hamming distance, the IEEE International Conference on Neural Networks, ICNN’98, May 4-9, 1998, Anchorage, the correct person was present second on the list Alaska, pp. 486-491. only 76% of the time, but after applying the shifting algorithm, the correct person appeared 2. Jacobs, C., Finkelstein, A., and Salesin, D. 90% of the time. Hence this post-processing (1995). "Fast Multiresolution Image Querying," algorithm could be used in the context of a Proceedings of SIGGAPH 95, in Computer sensor fusion scheme whereby final Graphics Proceedings, Annual Conference Series, pp. 277-286, August 1995, Los Angeles, classification is made on the basis of the CA. information present in several of the top matching patterns, rather than just the best 3. Stollnitz, E., DeRose, T., and Salesin, D. (1995). matching pattern. "Wavelets for computer graphics: A primer, Part 1," IEEE Transactions on Computer Graphics and Applications, 15(3), pp. 76-84. Future publications will explore the use of this shifting algorithm in a more practical recognition 4. Watta, P., Akkal, M., and Hassoun, M. (1997). “Decoupled Voting Hamming Associative Memory Networks” Proceedings of the IEEE International Conference on Neural Networks, ICNN’97, June 9-12, 1997, Houston, Texas, pp. 1188-1193.