US007269516B2

(12) United States Patent (10) Patent N0.: US 7,269,516 B2 BrunneretaL (45) Date of Patent: Sep. 11,2007

(54) SYSTEMS AND METHODS FOR 5,721,692 A 2/1998 Nagaya et a1...... 364/516 MONITORING 5,724,983 A * 3/1998 Selker et al. 600/301 5,734,737 A 3/1998 Chang et al...... 382/107 (75) Inventors: Daniela Brunner, Bronx, NY (US); 5,748,775 A 5/1998 Tsuchikawa et al. Vijay Gondhalekar, New York, NY 5,774,591 A 6/1998 Black et a1. 5,794,569 A 8/1998 Titus et al. (US); Emer Leahy, Bedford, NY (US); 5,798,798 A 8/1998 Rector et al. David LaRose, Pittsburgh, PA (US); 5,802,220 A 9/1998 Black et a1. William P. Ross, Saranac Lake, NY 5,850,352 A 12/1998 MoeZZi et al. (Us) 5,864,862 A 1/1999 Kriens et al. 5,870,138 A 2/1999 Smith et a1...... 348/143 (73) Assignees: PsychoGenics, Inc., TarrytoWn, NY 5,875,305 A 2/1999 Winter et al. (US); Carnegie Mellon University, 5,886,788 A 3/1999 Kobayashi Pittsburg, PA (U S) 5,915,332 A 6/1999 Young et al. 5,969,755 A 10/1999 Courtney Notice: Subject to any disclaimer, the term of this 6,023,497 A 2/2000 Takahashi et al...... 378/57 patent is extended or adjusted under 35 6,061,088 A 5/2000 Khosravi et al...... 348/169 U.S.C. 154(b) by 0 days. (Continued) (21) Appl. N0.: 10/147,334 FOREIGN PATENT DOCUMENTS EP 0 363 755 4/1990 (22) Filed: May 15, 2002 (Continued) (65) Prior Publication Data OTHER PUBLICATIONS US 2003/0100998 A1 May 29, 2003 Sams-Dodd, Frank, Journal of Neuroscience Methods, vol. 59, 1995, pp. 157-167.* Related U.S. Application Data Alex Pentland, et al., “Modeling and Prediction of Human Behav ior”, MIT Media Lab Perceptual Computing Technical Report No. (60) Provisional application No. 60/326,271, ?led on Oct. 433 (Originally published as: Towards Augmented Control Systems, 1, 2001, provisional application No. 60/291,039, ?led in IEEE Intelligent Vehicles 95, pp. 350-355, Detriot MI, Sep. on May 15, 2001. 25-26, 1995). “The Observer”, Index Page from http://WWWnoldus.com/products/ (51) Int. Cl. observer/observerhtml, printed on Jan. 14, 2002, pp. 1-2. G06F 19/00 (2006.01) “Etho Vision”, Index Page from http://WWWnoldus.com/products/ G06K 9/00 (2006.01) ethovision/ethovisionhtml, printed on Jan. 14, 2002, pp. 1-2. A61B 5/00 (2006.01) Alex Pentland, et al., “Real-Time 3-D Tracking and Classi?cation of Human Behavior”, Image Understanding Workshop, Defense (52) U.S. Cl...... 702/19; 382/110; 382/128; 73/865.4 Advanced Research Projects Agency, vol. 1, May 11-14, 1997, pp. 193-200. (58) Field of Classi?cation Search ...... 702/ 19 See application ?le for complete search history. (Continued) (56) References Cited Primary ExamineriMarjorie A. Moran (74) Attorney, Agent, or FirmiKing & Spalding LLP; U.S. PATENT DOCUMENTS Kenneth H. Sonnenfeld 3,100,473 A 8/1963 Kissel 3,656,456 A 4/1972 Stigmark et al. (57) ABSTRACT 3,803,571 A 4/1974 Luz A system and method used to assess animal behavior 3,955,534 A 5/1976 Boudrot et al. includes a module having sensors that collects a variety of 3,974,798 A 8/1976 Meetze, Jr. physical and biological data from a test subject. Interpreta 4,337,726 A 7/1982 CZekajeWski et al. 4,448,150 A 5/1984 Catsimpoolas tion of the data is provided to assess the test subject’s 4,574,734 A 3/1986 Mandalaywala et al. behavior, neurology, biochemistry and physiology. The 4,631,676 A 12/1986 Pugh module is useful in observing the e?fects of a drug on the test 4,693,255 A 9/1987 Beall animal and providing information on the drug’s signature. 4,813,436 A 3/1989 Au Another advantage is the module’s portability that alloWs it 5,080,109 A 1/1992 Arme, Jr. to be used in standard laboratory cages. This portability 5,088,446 A 2/1992 Campiotti alloWs the animal to be tested in its oWn habitat, that can 5,148,477 A 9/1992 Neely et a1. reduce any erroneous data due to stressing the animal When 5,214,711 A 5/1993 Neely et a1. removed to a test cage. Additionally, the module’s design 5,243,418 A 9/1993 Kuno et al...... 358/108 alloWs for parallel data collection and interpretation from 5,339,166 A 8/1994 LeBrat et a1...... 358/311 several laboratory animals undergoing di?ferent experi 5,427,942 A 6/1995 Civelli et al. ments. Multi-dimensional modeling of the test subject based 5,428,774 A 6/1995 Takahashi et a1...... 395/600 5,493,344 A 2/1996 Yu the system’s interpretation of the data alloWs pattern recog 5,550,965 A 8/1996 Gabbe et al. 395/154 nition of the drug signature, and predictive drug analysis. 5,666,157 A 9/1997 Aviv ...... 348/152 5,710,833 A 1/1998 Moghaddam et al. 34 Claims, 16 Drawing Sheets US 7,269,516 B2 Page 2

US. PATENT DOCUMENTS 1996; Boston, Massachusetts, SPIEiThe International Society For Optical Engineering, vol. 2916, pp. 162-173. 6,062,224 A 5/2000 Kissinger et al. L. R. Rabiner, et al. “An Introduction to Hidden Markov Models”, 6,072,903 A 6/2000 Maki et al...... 382/190 IEEE ASSP Magazine, vol. 3, No. 1, Jan. 1986, pp. 4-16. 6,081,607 A 6/2000 Mori et al. P.P. Smyth, et al., “Automatic Measurement of Vertebral Shape 6,128,396 A 10/2000 Hasegawa et al. Using Active Shape Models”, Depts. of Medical Biophysics & 6,140,316 A 10/2000 Berliner et al. Diagnostic Radiology, University of Manchester, pp. 705-714. 6,212,510 B1 4/2001 Brand Ilan Golani, et al., “Early Ontogeny of Face Grooming in Mice”, 6,234,109 B1 5/2001 Andersson et al. Development Psychobiology, 18(6):529-544 (1985). 6,263,088 B1 7/2001 Crabtree et al. Vikrant Kobla, et al., “Archiving, indexing, and retrieval of video in 6,353,831 B1 3/2002 Gustrnan the compressed domain”, Multimedia Storage and Archiving Sys 6,432,938 B1 8/2002 Jennings-White et al. tems, Nov. 18-19, 1996, SPIE-The International Society For Optical 6,458,564 B1 10/2002 Darrow et al. Engineering, vol. 2916, pp. 78-89. 6,470,277 B1 10/2002 Chin et al. “Current Research: Characterising Behavioural Phenotypes Using 6,678,413 B1 1/2004 Liang et al...... 382/181 Automated Image Analysis”, http://www.wiau.man.ac.uk/~ct/re 7,068,842 B2 6/2006 Liang et al. 382/181 search.html, printed on Jan. 14, 2002, p. 1-4. 2004/0141635 A1 7/2004 Liang et al. 382/110 “Measurement of Behavioral Asyrnmetries in Rodents Using Auto 2004/0141636 A1 7/2004 Liang et al. 382/181 mated Video Image Analysis”, http://www.noldus.com/events/ 2006/0110049 A1 5/2006 Liang et al...... 382/181 mb96/abstract/schwarti.htm, printed on Jan. 14, 2002, p. 1. FOREIGN PATENT DOCUMENTS Wayne Wolf, “Hidden Markov Model Parsing of Video Programs”, 1997 IEEE International Conference on Acoustics, Speech and EP 579 319 1/1994 Signal Processing, vol. IV of V, Apr. 21-24, 1997, pp. 2609-2611. EP 0 624 313 11/1994 Vikrant Kobla, et al., “Compressed Domain Video Segmentation”, EP 0 743 043 11/1996 Center for Automation Research, University of Maryland, Sep. EP 0 933 726 A2 8/1999 1996, pp. 1-25. JP 133061 6/1988 Vikrant Kobla, et al., “Feature Normalization for Video Indexing JP 8032959 2/1996 and Retrieval”, Center for Automation Research and Dept. of JP 08-063603 3/1996 Computer Science, Inst. of Systems Research, University of Mary JP 08-240830 9/1996 land, Nov. 1996, pp. 1-38. JP 09-073541 3/1997 J .B.I. Rousseau, et al., “Classi?cation of rat behavior by a neural JP 9237348 9/1997 network”, Poster presented at Measuring Behavior ’98, 2nd Inter JP 10019658 1/1998 national Conference on Methods and Techniques in Behavioral JP 10258044 9/1998 Research, Aug. 18-21, 1998, Groningen, The Netherlands, p. 1-2. JP 11-052215 2/1999 J .B.I. Rousseau, et al, “Automated Observation of Multiple Indi JP 11063927 3/1999 vidually Identi?ed Rats”, Poster presented at Measuring Behavior JP 11235328 8/1999 ’96, International Workshop on Methods and Techniques in Behav JP 11296651 10/1999 ioral Research, Oct. 16-18, 1996, Utrecht, The Netherlands, pp. 1-2. JP 11316820 11/1999 P.B.A. van Lochem, et al., “Video tracking: improved methods for JP 2000-215319 8/2000 identi?cation of animals with color markers”, Poster presented at JP 3270005 4/2002 Measuring Behavior ’98, 2nd International Conference on Methods SU 1357010 12/1987 and Techniques in Behavioral Research, Aug. 18-21, 1998, W0 WO 00/65523 11/2000 Groningen, The Netherlands, p. 1. W0 WO 01/12171 A2 2/2001 B.M. Spruijt, et al., “Automatic behavior recognition: what do we W0 WO 01/12184 A1 2/2001 want to recognize and how do we measure it?”, Paper presented at W0 WO 01/58351 8/2001 Measuring Behavior ’98, 2nd International Conference on Methods W0 WO 02/43352 5/2002 and Techniques in Behavioral Research, Aug. 18-21, 1998, W0 WO 02/44992 6/2002 Groningen, The Netherlands, pp. 1-2. W0 W0 02/092101 A1 11/2002 M. Spratt, “Approaches To Automating Analysis Of Motion Data For Ethology”, Poster presented at Measuring Behavior ’96, Inter OTHER PUBLICATIONS national Workshop on Methods and Techniques in Behavioral J. Smit, et al., “Automatic Recognition of Behavioral Patterns in Research, Oct. 16-18, 1996, Utrecht, The Netherlands, p. 1. Rodents Using Digital Imaging”, Poster presented at Measuring C. A. Hendrie, et al., “A Microcomputer Technique for the Detailed Behavior ’96, International Workshop on Methods and Techniques Analysis of Animal Behaviour”, Physiology & Behavior, vol. 30, in Behavioral Research, Oct. 16-18, 1996, Utrecht, The Nether pp. 233-235, Pergamon Press, 1983. lands, p. 1. Stata Norton, “Amphetamine as a Model for Hyperactivity in the D. L. Hopper, et al., “Reproducibility of Time Structure in Motor Rat”, Physiology & Behavior, vol. 11, pp. 181-186, Brain Research Activity of Rats Under Noctumal Conditions”, Pharmacology Bio Publications Inc., 1973. chemistry and Behavior, vol. 42, pp. 245-250, 1992. Phyllis J. Mullenix, “Evolution of Motor Activity Tests Into A W. J. Kernan, Jr., et al., “Time Structure Analysis of Behavioral Acts Screening Reality”, Toxicology and Industrial Health, vol. 5, No. 2, Using a Computer Pattern Recognition System”, Pharmacology 1989, pp. 203-216. Biochemistry and Behavior, vol. 34, pp. 863-869, 1989. W. J. Kernan, Jr., et al., “Analysis of the Time Distribution And W. J. Kernan, Jr., et al., “Computer Technology, Pattern Recognition Time Sequence of Behavioral Acts”, Intern J. Neuroscience, 1988, of behavioral events in nonhuman primate”, Behavior Research vol. 43, pp. 35-51. Methods & Instrumentation, vol. 12 (5), pp. 524-534, 1980. P.J. Mullenix, et al., “Extension Of The Analysis Of The Time A. M. Baumberg, et al. “ Flexible Models from Image Structure of Behavioral Acts”, Intern J. Neuroscience, 1989, vol. 44, Sequences”, University of Leeds, School of Computer Studies pp. 251-262. Research Report Series, Report 93.36, Oct. 1993, pp. 1-13. Alex Pentland, et al., “Toward Augmented Control Systems”, Adam Baumberg, et al., “Learning Flexible Models from Image Proceedings of the Intelligent Vehicles ’95 Symposium, Sep. 25-26, Sequences”, Lecture Notes in Computer Science, vol. 800, Jan. 1995, pp. 350-355. Olof Eklundh (Ed.), Computer VisioniECCV ’94, pp. 299-308. Russell N. Van Gelder, et al., “Real-Time Automated Sleep Scoring: Haitao Jiang “A Video Database System for Studying Animal Validation of a Microcomputer-Based System for Mice”, Sleep, Behavior”, Multimedia Storage and Archiving Systems, Nov. 18-19, 14(1); pp. 48-55, 1991. US 7,269,516 B2 Page 3

“BiobserveiSpectator”, http://www.biobserve.com/software/spec Kamijo, S. et al. Occlusion Robust Vehicle Tracking for Behavior tator/index.html, printed on Jan. 14, 2002, pp. 1-5. Analysis utilizing Spatio-Temporal Markow Random Field Model. “Measuring Behavior ’96”, http://www.noldus.com/events/mb96/ IEEE Intelligent Transportation Systems. Conference Proc. mb96.htm, printed on Jan. 17, 2002, pp. 1-2 Dearborn, MI 340-345 (2000). “Measuring Behavior ’96iPosters”, http://www.noldus.com/ Kernan, W.J., Jr. et al. Pattern Recognition of Rat Behavior. events/mb96/abstract/posters.htrn, printed on Jan. 17, 2002, pp. 1-4. Pharmacol. Biochem. Behav. 27, 559-564 (1987). English Translation of claim 1 of JP 3270005, claim 1. Krill, P. shoppers: You may be studied. (Jan. 11, 2002). English Translation of JP 11-296651. InfoWorld retrieved from http://staging.infoworld.com/artic...ate:/ English Abstract of JP 11-259643, Abstract. storypages/printfriendly.html on Jan. 14, 2002, 1-2. Balch, T. et al. Automatically Tracking and Analyzing the Behavior Kojima, A. et al. Generating Natural Language Description of of Live Insect Colonies. School of Computer Science, Carnegie Human Behavior from Video Images. Proc. 15th International Mellon University, Pittsburgh, PA (2001), 1-9. Conference on Pattern Recognition, IEEE Computer Society 4, Barber, D.L. et al. An Automatic Apparatus for Recording Rota 728-731 (2000). tional Behaviour in Rats with Brain Lesions. Physiology & Behav LICiAnimal activity monitoring. Retrieved from http://www. ior11 , 117-120 (1973). licmef.com/animalactivitymonitoring.htrn on Jan. 10, 2002, pp. 1-7. Baumans, V. et al. Development of a Balance System for Analysis MacIver, M.A. & Nelson. Body modeling and model-based tracking of Rodent Behavior. Measuring Behavior '96 retrieved from http:// for neuroethology. J. Neurosci. Methods 95, 133-143 (2000). www.noldus.com/events/mb96/abstract/baumans.htrn on Jan. 14, MacIver, MA. et al. Prey-Capture Behavior in Gyrnnotid Electric 2002, p. 1. Fish: Motion Analysis and Effects of Water Conductivity. .I. Exp. Bengtsson, G. et al. Irregular walks and loops with handedness in Biol. 204, 543-557 (2001). small-scale movement of Onychirus armatus (Collembola). Dept of Multiple Animal Movement Analyzer (MAMA) Manual. Retrieved Ecology and Centre for Mathematical Studies, Lund University, from http://www.ifas.u?.edu/~damaweb/mama.html on Jan. 10, Sweden retrieved from http://www.math.ku.dld~magnusw/ 2002, pp. 1-5. onychart2abs.txt on Jan. 10, 2002, p. 1. Martin, B.R. et al. Quantitation of Rodent Catalepsy by a Computer Bigelow, J.A. & Houpt, T.R. Feeding and Drinking Patterns in Imaging Technique. Pharmacology Biochemistry and Behavior 43, Young Pigs. Physiology & Behavior 43, 99-109 (1988). 381-386 (1992). Brand, M. & Kettnaker, V. Discovery and Segmentation of Activi Morrow-Tesch, J. et al. A Video Data Base System for Studying ties in Video. IEEE Transactions on Pattern Analysis and Machine Animal Behavior. J. Anim. Sci. 76, 2605-2608 (1998). 22, 844-851 (Aug. 2000). The Mostert Group retrieved from http://www.mostertgroup.com on Brandl, N. Measuring Pig Travel by Image Analysis. Semi-auto Jan. 10, 2002, pp. 1-3. matic pc program for tracking animal retrieved from http:// Motion Capture from Video. Retrieved from http://www.csc.ucsc. nabilnabil.homestead.com/?les/movtrack.htrn on Jan. 10, 2002, pp. edu/~wilhelms/fauna/Vision on Jan. 10, 2002, pp. 1-2. 1-5. Nelson, M.E. & MacIver, M.A. Prey Capture in the Weakly Electric Brann, M.R. et al. A Device for the Automated Monitoring of Fish Apteronotus Albifrons: Sensory Acquisition strategies and Stereotypic Behavior. Prog.Neuro-Psychopharmacol & Biol. electrosensory consequences. J. Exp. Biol. 202, 1195-1203 (1999). Psychiat. 6, 351-354 (1982). Neuroscience retrieved from http://www.tracksys.com.uldPages/ Collins, RT. et al. A System for Video Surveillance and Monitoring. Neuroscience/neuromainhtm on Jan. 10, 2002, pp. 1-12. The Robotics Institute, Carnegie Mellon Univerisity, Pittsburgh PA Observe and Analyse Behavior. Biobserve retrieved from http:// 1-53 (2000). www.bioserve.com/software/spectator/index.htrnl on Jan. 14, 2002, Cools, A.R. & Heeren, D. Image-Analyzing Systems for the Study of Rodent Behaviour: Spatial and Postural Transitions of Stable pp. 1-4. Con?gurations in Time. Measuring Behavior '96 retrieved from The Observer: Professional system for collection, analysis and management of observational data. Retrieved from http://www. http://www.noldus.com/events/mb96/abstract/cools.htm on Jan. 14, noldus.com/products/observer/observerhtml on Jan. 14, 2002, pp. 2002, p. 1-2. Cools, A.R. & Heeren, D. Image-analyzing systems for the study of 1-2. spatial and postural transitions in rats. Measuring Behavior 1998 Pear, J .J . et al. Three-dimensional spatiotemporal imaging of move retrieved from http://www.noldus.com/events/mb98/abstracts/ ment patterns: Another step toward analyzing the continuity of cools.htm on Jan. 14, 2002, p. 1. behavior. Behavior Res. Methods, Instruments & Computers 21, Current Research: Characterizing Behavioural Phenotypes Using 568-573 (1989). Automated Image Analysis. Retrieved from http://www.wiau.man. Pentland, A. & Liu, A. Modeling and Prediction of Human Behav ac.uk/~ct/research.html on Jan. 14, 2002, pp. 1-4. ior. Neural Computation 11, 229-242 (1999). Environment and Behavior Symposium: The Use of Computers for Poly-Track: Video Tracking System. San Diego Instruments Modeling, Recognizing and Recording Behavior. J. Anim. Sci. Retrieved from http://www.sd.inst.com/polytrk.htm on Jan. 10, 75(Suppl.1) (1997), p. 1. 2002, pp. 1-3. EthoVision: Video tracking system for automation of behavioral Rasnow, B. et al. Applications of Multimedia Computers and Video experiments. Retrieved from http://www.noldus.com/products/ Mixing to Neuroethology. Retrieved from http://www.bbb.caltech. ethovision/ethovisionhtml on Jan. 14, 2002, pp. 1-2. edu/electric?sh/brian/mm/multimediahtml on Jan. 10, 2002, pp. Fergusson, I.K. Tracking sharks by videocamera4Crittercam: the 1-8. video parasite. Ichthyology at FL Museum of Nat. Hist. Retrieved Rehg, J .M. & Kanade, T. DigitEyes: Vision-Based Hand Tracking from http://www.?mnh.u?.edu/?sh/Organizations/SSG/7Newslet for Human-Computer Interaction. Proc. IEEE Workshop on Motion ter/shark7news11.htm on Jan. 10, 2002, p. 1. of Non-Ridgid and Articulated Objects, Austin, TX 16-22 (Nov. The GEDRI Project: Quantitative analyses of sexual behaviour as a 1994), pp. 1-23. screening tool of hormone-mimicking effects on animal reproduc Rehg, J .M. & Kanade, T. Visual Tracking of Self-Occluding Articu tion. Retrieved from http://endocrine.ei.irc.it/gedri/packiedri. lated Objects. School of Computer Science, Carnegie Mellon Uni FullScreen?pirsiid:3 on Jan. 10, 2002, p. 1. versity, Pittsburgh, PA 1-37 (Dec. 31, 1994). Golani, I. Reasearch in the interface between behaviour, computa Rubin, A. Video Laboratories: Tools for Scienti?c Investigation. tion, statistics and genetics. Laboratory ofAnimal Behavior, Tel Aviv ofthe ACM 36, 64-65 (May 1993), p. 1-2. University retrieved from http://www.tau.ac.il/~ilan99 on Jan. 10, Scheel, D. Development of an autonomous underwater vehicle for 2002, 1-3. behavioral studies of octopus and other marine animals by video Hoy, J.B. Dynamic Animal Movement Analyzer: A Macintosh technique. Retrieved from http://marine.alakapaci?c.edu/octopus/ Based Video Animal Tracker. Dept. Entomology & Nematology, ShadowNSFpro-summary.html on Jan. 10, 2002, p. 1. University of Florida, Gainesville retrieved from http://www. Schwarting, R.K.W. Measurement of Behavioral Asymmetries in afpmborg/pubs/presenl0.htm on Jan. 10, 2002. Rodents Using Automated Video Image Analysis. Measuring US 7,269,516 B2 Page 4

Behavior '96 retrieved from http://WWW.noldus.com/events/mb96/ J. Rife and SM. Rock, “Visual Tracking of Jelly?sh in Situ,” 2001, abstract/schWaIti.htm on Jan. 14, 2002, p. 1. IEEE International Conference on Image Processing, Thessaloniki, SchWarting R.K.W. et al. A video image analyzing system for Greece, pp. 289-292. open-?eld behavior in the rat focusing on behavioral asymmetrics. English Translation of abstract of JP 11094599, Apr. 9, 1999, J. Neurosci. Methods 49, 199-210 (1993). esp@cenet database. Spratt, M. An Introduction to Automated Ethology and Autonomous English Translation of abstract of JP 2000000216, Jan. 7, 2000, Systems or the Ethology of Ethologists. S8-Stock Behavior Mea Patent Abstract of Japan. sures for Success J. Muir & F. Sevila, eds. 182-183 (Feb. 1994). Battaglia, D. et al.; (1995). The role of physical cues in the Spruijt, B.M. & Rousseau, J.B.I. Consequences of the Ongoing regulation of host recognition and acceptance behavior of Aphidius Automation of the Observation and Analysis of Animal Behaviour. ervi Haliday (Hymenoptera: Braconidae). Journal of Insect Behav Measuring Behavior '96 retrieved from http://WWW.noldus.com/ ior, 8, 739-750. events/mb96/abstracts/spruijt.htm on Jan. 14, 2002, p. 1. Beer, R., “A Simple Method for Recording and Analyzing Spatial Stitt, J .P. et al. Automated Analysis of Feeding Behavior in Small Behaviour”, Measuring Behavior ’96, International Workshop on Animals, IEEE 120-122 (1998), pp. 1-3. Methods and Techniques in Behavioral Research, 1996. Sumpter, N. et al. Modeling Collective Animal Behaviour using Bobick, Aaron et al., “Real-time Recognition of Activity Using Extended Point-Distribution Models. School of Comp. Studies, Res. Temporal Templates”, Third IEEE Workshop on Applications of Report Series, University of Leeds, Report 97.24 (May 1997), pp. Computer Vision WACV ’96, pp. 39-42 (1996). 1-13. Bressers, W.M.A. et al., 1995. Time structure of self-grooming in Tang, L. et al. Automatic analysis of the behaviour of group-housed the rat: self-facilitatiuon and effects of hypothalamic stimulation pigs by an image collection system. Measuring Behavior '98 and neuropeptides. Behavioural Neurosciences, 109: 995-964. retrieved from http://WWW.noldus.com/events/mb98/abstracts/tang. Carson, Chad et al., “Blobworld: A system for region-based image htm on Jan. 14, 2002, p. 1. indexing and retrieval” (1999). Abstract Retrieved from http://elib. Torello, M.W, et al. An Automated Method for Measurement of cs.berkeley.edu/carson/papers/visual99.htrnl. Circling Behavior in the Mouse. Pharmacol. Biochem, &Behav. 19, Cohen, Isaac et al., “Detecting and Tracking Moving Objects for 13-17 (1983). Video Surveillance”, IEEE Proc. Computer Vision and Pattern Trackit System Family. Biobserve retrieved from http://WWW. Recognition, pp. 1-7, 1999. biobserve.com/softWare/trackit/indexhtml on Jan. 10, 2002, pp. Davis, James W. et al., “The Representation and Recognition of 1-3. Human Movement Using Temporal Templates”, IEEE Computer van Lochem, P.B.A. et al. Automatic recognition of behavioral Society Conference on Computer Vision and Pattern Recognition, patterns of rats using video imaging and statistical classi?cation. pp. 928-934 (1997). Measuring Behavior 1998 retrieved from http://WWW.noldus.com/ De Vries, Han et al., “Matman: A Program for the Analysis of events/mb98/abstracts/lochem2.htm on Jan. 14, 2002, pp. 1-2. VideoScan-2002 Image Analysis System. Retrieved from http:// Sociometric Matrices and Behavioural Transition Matrices”, Behaviour, vol. 125, pp. 157-175 (1993). WWW.accuscan-usa.com/ezvideo.htrnl on Jan. 10, 2002, pp. 1-10. Visual Fusion 3D Tracking and Motion Analysis. Retrieved from Haccou, P. et al., 1983. Analysis of time-inhomogeneity in Markov chains applied to mother-infant interactions of rhesus monkeys. http://WWW.sWcp.com/~spsvs/VisualFusion/visfusInitial.htrnl on Animal Behaviour, 31, 927-945. Jan. 10, 2002, p. 1. Watermaze Tracking. Acimetrics Software retrieved from http:// Haccou, P. et al., 1986. On the analysis of time-inhomogeneity in wwwactimetrics.com/Watermaze/tracking.html on Jan. 10, 2002, p. Markov chains: a re?ned test for abrupt behavioural changes. 1. Animal Behaviour, 34, 302-303. Yang, M.-H. & Ahuja, N. Extraction and Classi?cation of Visual Hora, K.H. et al., 1999, Oviposition in Yponomeuta cagnagellus: the Motion Patterns for Hand Gesture Recognition. Proc. IEEE CVPR importance of contact cues for host plant acceptance. Physiological 892-897 (1998). Entomology, 24, 109-120. Babu, K.S. et al. Spike Burst Analysis: A Tool for Analyzing Intille, Stephen S. et al., “Real-Time Closed-World Tracking”, Spontaneous Prehension Recorded With Digital Video. Proceedings of the IEEE Computer Society Conference on Com Society for Neuroscience’s 30th Annual Meeting (Nov. 4-9 2000), p. puter Vision and Pattern Recognition, pp. 697-703 (1997). 1. Isaacs, R. et al., (1999). Host plant evaluation behaviour of Bemisia Crnic, L.S. et al, Automated Analysis of Digitized Videotapes of Tabaci and its modi?cation by external or internal uptake of Mouse Home-Cage Behavior. Society for Neuroscience’s 30th imidacloprid. Physiological Entomology, 24, 101-108. Annual Meeting (Nov. 4-9 2000), p. 1. Jonsson, G.K., “Detecting patterns in complex behavioural pro DeboWy, DJ. et al. Spike Burst Analysis: A Tool for Analyzing cesses With The Observer and Theme”, Poster presented at Mea Trained Prehension Behaviors Recorded With Digital Video Society suring Behavior ’98, 2nd International Conference on Methods and for Neuroscience’s 30th Annual Meeting (Nov. 4-9 2000), p. 1. Techniques in Behavioral Research (1998). Retrieved from http:// J. Huang et al., “Joint Video Scene Segmentation and Classi?cation WWW.hi.is/~msm/posterbeggi/indexhtml. Based on Hidden Markov Model,” 2000, IEEE International Con Lyon, Melvin et al., “The importance of Temporal structure in ference on Multimedia and Expo, New York, NY, pp. 1551-1554. analyzing schizophrenic behavior: some theoretical and diagnostic A.J. Spink et al., “EthoVision color identi?cation: E new method for implications”, Schizophrenia Research, vol. 13, pp. 45-56 (1994). color tracking using both hue and saturation,” 2000, Measuring Magnusson, M.S., “Real-time pattern detection versus standard Behavior Conference, Nijmegen, The Netherlands, pp. 295-297. sequential and time series analysis”, Measuring Behavior ’98, 2nd With Supplementary Partial European Search International Conference on Methods and Techniques in Behavioral Report corresponding to EP 02 73 1842, Sep. 22, 2004. Research (1998). Retrieved from http://WWW.noldus.com/events/ M. Everingham and B. Thomas, “Supervised segmentation and mb98/abstracts/Magnusson.htrn. tracking of nonrigid objects using a ‘Mixture of Histograms’ Magnusson, M.S., “T-Patterns, Theme and the Observer”, Measur Model,” 2001, IEEE International Conference on Image Processing, ing Behavior ’96, International Workshop on Methods and Tech Thessaloniki, Greece, pp. 62-65. niques in Behavioral Research (1996). Retrieved from http://WWW. N. Oliver et al., “LAFTER: Lips and Face Real Time Tracker,” noldus.com/events/mb96/abstract/magnusso.htm. 1997, IEEE Computer Society Conference on Computer Vision and Magnusson, Magnus S., “Hidden Real-Time Patterns in Intra- and Pattern Recognition, San Juan, Puerto Rico, pp. 123-129. Inter-Individual Behavior: Description and Detection”, European V. Pavlovic et al., “Visual Interpretation of Hand Gestures for Journal of Psychological Assessment, vol. 12, pp. 112-123 (1996). Human-Computer Interaction: ARevieW,” 1997, IEEE Transactions Medioni, Gérard et al., “Event Detection and Analysis from Video on Pattern Analysis and Machine Intelligence, vol. 19, No. 7, pp. Streams”, IEEE Transactions on Pattern Analysis and Machine 677-695. Intelligence, vol. 23, 873-889 (Aug. 2001). US 7,269,516 B2 Page 5

Noldus releases The Observer 3.0. Kuhl et al., “Elliptic fourier features of a closed contour,” Comp. Retrieved from http://www.noldus.com/site/popupdoc. Graphics & Image Proc., 18, 236-258 (1982). php?doc:200407115&print:1& (1993). Liang, “Digital video analysis system for home cage behavior,” Puopolo, M. et al., “Continuous time Markov chain models: a useful CRISP NIH database, Grant No. 1R43MH058964-01, Project Start: tool in understanding behavioral patterns”, Measuring Behavior Jul. 1, 1999 (date posted to public database). ’98, 2nd International Conference on Methods and Techniques in Liang, “Digital video analysis system for home cage behavior,” Behavioral Research (1998). Retrieved from http://www.noldus. CRISP NIH database, Grant No. 2R44MH058964-02, Project Start: com/events/mb98/abstracts/puopolo.htm. Jul. 1, 1999 (date posted to publlic database). Rojas, J.C.; Wyatt, T.D.; Birch, M.C. (2000). Flight and oviposition Liang, “High Throughput Phenotyping: Drug Effects on Homecage behavior toward different host plant species by the cabbage moth, mo”, CRISP NIH databae, Project Stat: Sep. 30, 2001 (date posted Mamestra brassicae (L.) (Lepidoptera: Noctuidae). Journal of to public database). Insect Behavior, 13, 247-254. Liang et al., “A Practical Video Indexing and Retrieval System”, Spruijt, BM. (1991). An ACTH(4-9) analog enhances social atten SPIE, vol. 3240 (1998), pp. 294-303. tion in aging rats: a longitudinal study. Neurobiology of Aging, 13, Liang et al., A Ground Target Detection System for Digital Video 153-158. Database SPIE, vol. 3387 (1998) pp. 219-226. Spruijt, B.M. et al., “Behavioral Sequences as an Easily Quanti? Magnusson, “Discovering Hidden Time Patterns in Behavior: T-Pat able Parameter in Experimental Studies”, Physiology & Behavior, terns and Their Detection”, Beh. Res. Methods, Instr. & Comp., vol. 32, pp. 707-710, 1984. 32:1, 93-110 (2000). Spruijt, Berrie M. et al., “Prolonged Animal Observation by Use of Rosin et al., “Evaluation of Global Image Thresholding for Change Digitized Videodisplays”, Pharmacology Biochemistry & Behavior, Detection”, Pattern Recognition Latters 24, pp. 2345-2356 (2003). vol. 19, pp. 765-769, 1983. Rosin et al., “image Difference Threshold Strategies and Shadow Van Den Berg, Caroline, L. et al., “Sequential Analysis of Juvenile Detection” Proceedings of the 6th British Machine Vision Confer Isolation-Induced Decreased Social Behavior in the Adult Rat”, ence (1995). Physiology & Behavior, vol. 67, pp. 483-488 (1999). Rousseau et al., “Classi?cation of rat behavior with an image Wilson, Andrew, D. et al., “Recovering the Temporal Structure of processing method and a neural network,” Behav. Res. Meth. Instr. Natural Gesture”, Automatic Face & Gesture Recognition, Auto Comp., 32, 63-71 (2000). matic Face and Gesture Recognition, Proceedings of the Second Ueda et al., “Automatic Structure Visualization for Video Editing”, International Conference (1996). Interchi ’93 Conference Proceedings (1993), pp. 137- 141. Abe et al., “Scene Retrieval Method Using Temporal Condition Wolf, “Key Frame Selectionby Mation Analysis”, IEEE, pp. 1228 Changes”, Systems and Computers in Japan, 24:7, pp. 92-101 1231, (1996). (1993). “Noldus to Develop New Technology for Automated Behaviral M. Brand, “Learning concise models of human activity from Phenotyping”, Press Release: Neuro Bsik Project, www.nokdus. ambient video. . . ” MERL, Mitsubishi Electric Research Laboratory com/press/pr04ineurobisk.html (2004). (Nov. 1997). Bevilacqua et al., “Robust Denoisingand Moving Shadows Detec Connor et al., “Infrared video viewing,” Science, 1999 (1978). p. tion in Traf?c Scenes” (2002) (retrieved from website for IEEE 1004. Computer Vision Pattern Recognition conference: http://www-mi Crnic et al., “Animal Models of Mental Retardation: An Overview”, cro.deis.unibo.it/-CVG/papers/cvpri2002.pdf). Mental Retardation and Developmental Disabilities, 2:185-187 Cavallaro et al., “Image Analysis for Adavnced Video Surveillance” (1996). (2000) Chapter in: Multimedia Video-Based Surveillance Systems, Day et al., “Obj ect-Oriented Conceptual Modeling of Video Data”, G.L. Foresti, P Mahonen, C.S. Regazzoni (Eds.), Kluwer Academic IEEE pp. 401-408 (1995) Proc. 11th Int’l Conference on Data Eng. Publishers, Boston, chapter 2.3, pp. 57-56, 2000. Duncan, “Using Theme to Analyze Interaction Structure and Strat Cools et al., “Image-analyzing systems for the study of rodent egy”, Measuring Behavior, www.noldus.com/events/mb2002/pro behavior: spatial and postural. . . ” Paper presented at Measuring gram/abstracts/duncanhtml (2002). Behavior ’96, International Workshop (Oct. 1996). Harville, “A Framework For High-Level Feedback to Adaptive, Horprasert et al., “a Statistical Approach for Real-time Robust Per-Pixel, Mixture of Gaussian Background Models”, ECCV Background Subrtaction and Shadow Detection” (1999). Proc. (2002), pp. 543-560. IEEE ICCV (retrieved from: http://vr.kjist.ac.kr/~HCI2002/ Hereen et al., “Classifying postures of freely moving rodents with visionBasedPapers/HorprasertFRAME99.pdf). the help of Fourier descriptors and a neural network,” Behav. Res. Environment and Behavior Symposium, “The Use of Computers for Meth. Instr. Comp., 32, 56-62 (2000). Modeling, Recognizing and Recording Behavior”, J. Anim. Sci., Jenkins et al., “Deriving Action and Behavior Primitives from 137 (1997). human Motion Data”, IEEE/RSJ Int’l Conf. on Intell. Robots and Systems, pp. 2551-2556 (2002). * cited by examiner U.S. Patent Sep. 11,2007 Sheet 1 6f 16 US 7,269,516 B2

528%2328 szeig:52 5 h w 8.5525138“RE w E22:5

m \ 5.52222%:

Es?2%.32 E2E

R 552125x552“. bmEE3223 U.S. Patent Sep. 11,2007 Sheet 2 6f 16 US 7,269,516 B2

2lightHense I6dispenser Accesswatto I4dispenser U.S. Patent Sep. 11,2007 Sheet 3 6f 16 US 7,269,516 B2

Figure 3

1 Infrared a [3mm Light ?ght stimuli

Water

Food dour ) \7 k) \ Visual

dear , \_ l___ j stimuli I J/OO ?peraad I Dpemnd 2 Scale J door daor Infra red 15mm U.S. Patent Sep. 11,2007 Sheet 4 6f 16 US 7,269,516 B2

Figure 4

Video image

frame substraetion

Outline ?tting (shown: :21! organizing map) U.S. Patent Sep. 11,2007 Sheet 5 6f 16 US 7,269,516 B2

frame I

Figure 5

Frame 2

Outline ?tting (shown: self organizing map]

Frame 3

Fram'e 4 U.S. Patent Sep. 11,2007 Sheet 6 6f 16 US 7,269,516 B2

Figure 6

Frames 14 Graphical

Node trayecmries

Frames I 4 For i=| to m nudes Analytical TlTiTlTn 11)) U.S. Patent Sep. 11,2007 Sheet 7 6f 16 US 7,269,516 B2

Figure 7

Frames/Samples I64 Visual Signals Animal Non visual signal: Node tmjectories Slate trajectories

For i=| to m nodes For i=l to n states )) BU) WP (1.0), 16(1)) 5(1) 10.3): 66(3), 713)} 21(3) T(i‘4)= (6(4). r.(4 11(4)

'?me Stamp/Coordination for i=| tn m, forj=i to n

EU) (110) 710)) 5,0) 6(1)

Probability Napping

V Ti. S, cakulaxe con?guration E probability

Transition Pmbabilities V 5 V C, C' calculate U.S. Patent Sep. 11,2007 Sheet 8 0f 16 US 7,269,516 B2

Figurc 8

Proba " Mapping V 5 Ana ' ‘d C, C' calculate

P(c—+c'; a) = If? 61PM! + 51¢ 1)

Probability ?apping Graphical U.S. Patent Sep. 11,2007 Sheet 9 6f 16 US 7,269,516 B2

F 1' gurc 9 Probability Mapping Control Probability ?apping Experimental U.S. Patent Sep. 11,2007 Sheet 10 6f 16 US 7,269,516 B2

Pmba " ?apping Differential Novel Query figure Hi

If?

Probability Mapping Differential Probabili pping Dilferential Drug Class A Druz (la

Compute Similarity Ratio

Drug x belungs to Probability Power

Drug Class A .42 .22 Drug Class B .94 .56 U.S. Patent Sep. 11,2007 Sheet 11 0f 16 US 7,269,516 B2

Figure 1 1A

.rearing ...... immobile ...... locomotion. . . . . immobile ......

.nose up . . . _ ...... nose down. . . . . nose up ...... nose down ......

.ear twitch ......

.tail down ...... tail down. . . . . tail down...... tail down ......

...... food intake ...... _ . . .

...... _ . . _ ...... water intake. . .

.‘729 bpm ...... 650 bpm ...... 550 bpm ...... 560 bpm ......

Flgure 11B

. .rearing ...... immobile ...... locomotion. . . . .immobile ......

. .nose up ...... nose down ...... nose up . . 4 . . . . .nose down ......

..ear twitch...... _ ......

..tail dew"... ._tail down . . . . .-tail down...... ta1l down ......

...... 4 ...... _ ..food intake...... -......

_ ...... water intake. . .

..729 bpm ...... 650 bpm ...... 550 bpm ...... 560 bpm ...... | I I I

Figure 11C

Not aligned

....A TLGPS S ACAAGTTVA C I M S TAA MTA S TA M G AG....

....S S CA TLGP S S AGTTVA CI M TVA CIS TA M GA 6 MTAW. Aligned 1 l L G PSSACAA G TTVAC l MSTAAM TASTAM G A G....

....S S CA TLG PS SA ...... G TTVA CI M... TVAC] ...... STAM G A G M TA.... U.S. Patent Sep. 11,2007 Sheet 12 6f 16 US 7,269,516 B2

Figure 12

Grooming Walking Running Grooming Walln'ng Running Grooming D Walking D Running D Grooming — 9 0 Groomin g Walking D Running D Walking D Running Walking 0 - 22 Walking D Grooming D Walking D Running D Walking D Running D Grooming D Running 9 l 4 » Running Walking D Running D Walking D Running D Walking - Running D Grooming D Walking D Running D Walking D Running D Grooming D Walking D Running D Grooming Walking D Running D Walking D Running D Grooming D Walking D Running D _ Walking D Running D Grooming - Running D Walking D Running D Walking D Running D Grooming D Walking D

Walking Running U.S. Patent Sep. 11, 2007 Sheet 13 0f 16 US 7,269,516 B2

Figure 13

Chlordiazepoxide 1.25 mg/ kg Behavior after Directed Contact before Rear Locom. Groom lmmob. Rear 31.56 4.56 7.86 6.20 8.74 7.24 Directed Contact with 14.75

3.32 Locomotion 8.23

immobile 7.86

CPMC 2.5 mg/ kg Behavior after Directed Contact before Looom. Groom lmmob. Rear 283 Directed 1018 Contact with 669

Locomotion

immobile

U.S. Patent Sep. 11,2007 Sheet 15 0f 16 US 7,269,516 B2