US009082219B2 (12) United States Patent (10) Patent No.: US 9,082,219 B2 Haanpaa et al. (45) Date of Patent: *Jul. 14, 2015 (54) ORIENTATION INVARIANT OBJECT (52) U.S. Cl. IDENTIFICATION USING MODEL-BASED CPC .............. G06T 15/10 (2013.01); G06K9/0063 IMAGE PROCESSING (2013.01); G06K9/3241 (2013.01); G06T 15/00 (2013.01) (71) Applicant: Cybernet Systems Corporation, Ann (58) Field of Classification Search Arbor, MI (US) CPC .............................. G06K9/0063; G06T 15/10 USPC .......................................... 382/103, 206, 209 (72) Inventors: Douglas Haanpaa, Dexter, MI (US); See application file for complete search history. Charles J. Cohen, Ann Arbor, MI (US); Glenn J. Beach, Grass Lake, MI (US); (56) References Cited Charles J. Jacobus, Ann Arbor, MI (US) U.S. PATENT DOCUMENTS (73) Assignee: Cybernet Systems Corporation, Ann 3,976,999 A 8, 1976 Moore et al. Arbor, MI (US) 3,992,710 A 11/1976 Gabriele et al. (*) Notice: Subject to any disclaimer, the term of this (Continued) patent is extended or adjusted under 35 Primary Examiner — Gregory M Desire U.S.C. 154(b) by 0 days. (74) Attorney, Agent, or Firm — Gifford, Krass, Sprinkle, This patent is Subject to a terminal dis Anderson & Citkowski, P.C. claimer. (57) ABSTRACT (21) Appl. No.: 14/242,560 A system for performing object identification combines pose determination, EO/IR sensor data, and novel computer graph (22) Filed: Apr. 1, 2014 ics rendering techniques. A first module extracts the orienta tion and distance of a target in a truth chip given that the target (65) Prior Publication Data type is known. A second is a module identifies the vehicle within a truth chip given the known distance and elevation US 2014/0320486A1 Oct. 30, 2014 angle from camera to target. Image matching is based on synthetic image and truth chip image comparison, where the Related U.S. Application Data synthetic image is rotated and moved through a 3-Dimen sional space. To limit the search space, it is assumed that the (63) Continuation of application No. 13/438.397, filed on object is positioned on relatively flat ground and that the Apr. 3, 2012, now Pat. No. 8,687,849, which is a camera roll angle stays near Zero. This leaves three dimen continuation of application No. 1 1/938.484, filed on sions of motion (distance, heading, and pitch angle) to define Nov. 12, 2007, now Pat. No. 8,150,101. the space in which the synthetic target is moved. A graphical Provisional application No. 60/865,521, filed on Nov. user interface (GUI) front end allows the user to manually (60) adjust the orientation of the target within the synthetic 13, 2006. images. The system also includes the generation of shadows (51) Int. C. and allows the user to manipulate the Sun angle to approxi G06K 9/00 (2006.01) mate the lighting conditions of the test range in the provided G06T I5/10 (2011.01) video. (Continued) 10 Claims, 5 Drawing Sheets Sizes teTalkS Tracked A Wheeled I like TTTTTTTP2Y ----------'A' ride it is AA A US 9,082.219 B2 Page 2 (51) Int. Cl. 6,042,050 A 3/2000 Sims et al. G06K 9/32 (2006.01) is 28 CeCe.Set al. G06T I5/00 (2011.01) 6,437,728 B1 8/2002 Richardson et al. 6,491.253 B1* 12/2002 McIngvale ................... 244.3.11 (56) References Cited 6,597,800 B1, 7/2003 Murray et al. 6,608,563 B2 8, 2003 Weston et al. U.S. PATENT DOCUMENTS 6,813.593wk - B1 1/2004 Berger 6,894,639 B1 5, 2005 Katz 4,243,972 A 1, 1981 Toussaint TOO3.137 B2 2, 2006 Oht 4497,065. A 1, 1985 Tisdale et al. 7006,944 B2 2, 2006 Ad 4,767,609 A 8/1988 Stavrianpoulos www. 4,772.548 A 9/1988 Stavrianpoulos 7,030,808 B1 4/2006 Repperger et al. 4,845,610 A 7, 1989 Parvin 7,040,570 B2 ck 5, 2006 Sims et al. 4,847,817 A 7, 1989 Au et al. 7,137,162 B2 * 1 1/2006 Spencer et al. ................ 14,715 4,950,050 A 8/1990 Pernicket al. 7,205,927 B2 4/2007 Krikorian et al. 4,972, 193 A 11, 1990 Rice 7,227,801 B2 6, 2007 Kikutake et al. 5,202,783 A 4, 1993 Holland et al. 7,227,973 B2 6/2007 Ishiyama 5,210,799 A 5, 1993 Rao 7,274,801 B2 9, 2007 Lee 5,258,924 A 11/1993 Call et al. 7.773,773 B2 8/2010 Abercrombie et al. 5,324,829 A 6, 1994 Bahl et al. 7,848,566 B2 12/2010 Schneiderman 5,339,082 A 8/1994 Norsworthy 2005/0286767 A1 12/2005 Hager et al. 5,521,298 A 5, 1996 Bahl et al. 2007/0264.617 A1 11/2007 Richardson et al. 5,524,845. A 6, 1996 Sims et al. 5,566,246 A 10, 1996 Rao * cited by examiner U.S. Patent Jul. 14, 2015 Sheet 1 of 5 US 9,082.219 B2 U.S. Patent Jul. 14, 2015 Sheet 2 of 5 US 9,082.219 B2 -------x-Ya-XY-axo-saray-1s--a-a-a-a- -------------as-s-s-s-s-s U.S. Patent Jul. 14, 2015 Sheet 3 of 5 US 9,082.219 B2 FIG. 3 U.S. Patent Jul. 14, 2015 Sheet 4 of 5 US 9,082.219 B2 F.G. 4 U.S. Patent Jul. 14, 2015 Sheet 5 of 5 US 9,082.219 B2 M10A2 71.43% 0.00% 7.14% 4.29% 7.14% 0.00% 59.23% 18.52% 22.22% O.O.0% 96.30% 3.70% M 13 O.00% 0.00% 0.00% 88.89% 1% M1 10A2 3.70% 0.00% 0.00% 0.00% 96.30% F.G. 5 Vehicle Type ID w/o Shadow ID with Shadow M1 OA2 M60 MOA2 MOA2 M60 M60 M60 M60 M60 M60 M60 M60 M60 ZSU23 M60 M113 M113 M 13 M113 M113 M113 ZSU23 M113 ZSU23 ZSU23 M 3 ZSU23 M35 M35 M35 F.G. 6 US 9,082,219 B2 1. 2 ORIENTATION INVARIANT OBJECT The system is capable of operating in the presence of noisy IDENTIFICATION USING MODEL-BASED data or degraded information. Image matching is actually IMAGE PROCESSING based on synthetic image and truth chip image comparison, where the synthetic image is rotated and moved through a REFERENCE TO RELATED APPLICATION 5 three-Dimensional space. To limit the search space, it is assumed that the object is positioned on relatively flat ground This application is a continuation of U.S. patent applica and that the camera roll angle stays near Zero. This leaves tion Ser. No. 13/438.397, filed Apr. 3, 2012, which is a con three dimensions of motion (distance, heading, and pitch tinuation of U.S. patent application Ser. No. 1 1/938,484, filed angle) to define the space in which the synthetic target is Nov. 12, 2007, now U.S. Pat. No. 8,150,101, which claims 10 moved. Synthetic imagery generated using a simulation priority from U.S. Provisional Patent Application Ser. No. library can be used to help train the system. 60/865,521, filed Nov. 13, 2006, the entire content of each Next the rendered synthetic image and the truth chip is application is incorporated herein by reference. rendered in order to make them more comparable. A simple 15 thresholding of the truth and synthetic images, followed by GOVERNMENT SUPPORT extracting the biggest blob from the truth chip is applied to the process. The system iterates within this 3D search space to This invention was made with Government support under perform an image match from the synthetic and truth images Contract No. N68335-06-C-0065 awarded by the United to find the best score. States Navy. The Government has certain rights in the inven The process of target recognition is very similar to that tion. used for the distance/orientation determination. The only dif ference is the search space. Instead of varying the target FIELD OF THE INVENTION distance, heading, and pitch, the search varies the target type and the heading. This invention relates generally to object identification, 25 A graphical user interface (GUI) front end allows the user and in particular, to a system for performing object identifi to manually adjust the orientation of the target within the cation that combines pose determination, Electro-Optical/ synthetic images. The system also includes the generation of Infrared (EO/IR) sensor data, and novel computer graphics shadows and allows the user to manipulate the Sun angle to rendering techniques. approximate the lighting conditions of the test range in the 30 provided video. Manipulation of the test Sunangle is a tedious BACKGROUND OF THE INVENTION process that could also be automated in much the same way as the distance/orientation determination. The application of Many automated processes require the ability to detect, shadows and Sunangle to the process greatly improves overall track, and classify objects, including applications in factory target identification in outdoor conditions. automation, perimeter security, and military target acquisi 35 tion. For example, a primary mission of U.S. military air BRIEF DESCRIPTION OF THE DRAWINGS assets is to detect and destroy enemy ground targets. In order to accomplish this mission, it is essential to detect, track, and FIG. 1 is a plot that shows the lengths and widths of the classify contacts to determine which are valid targets. Tradi vehicles of TABLE 1: tional combat identification has been performed using all 40 FIG.