HOULUTUMUTURUS009788714B2 (12 ) United States Patent ( 10 ) Patent No. : US 9 ,788 ,714 B2 Krueger ( 45 ) Date of Patent: Oct . 17 , 2017 (54 ) SYSTEMS AND METHODS USING VIRTUAL 8 ) Field of Classification Search REALITY OR AUGMENTED REALITY USPC . .. .. .. .. .. 351/ 200 - 246 ENVIRONMENTS FOR THE See application file for complete search history . MEASUREMENT AND /OR IMPROVEMENT OF HUMAN VESTIBULO -OCULAR (56 ) References Cited PERFORMANCE U . S . PATENT DOCUMENTS (71 ) Applicant: Wesley W . O . Krueger, San Antonio , 4 ,817 ,633 A 4 / 1989 McStravick et al. TX (US ) 5 , 180 , 907 A 1 / 1993 Udden et al . (Continued ) (72 ) Inventor: Wesley W . O . Krueger , San Antonio , TX (US ) FOREIGN PATENT DOCUMENTS ( 73 ) Assignee : IARMOURHOLDINGS, INC ., San WO WO 2013117727 8 / 2013 Antonio , TX (US ) OTHER PUBLICATIONS ( * ) Notice : Subject to any disclaimer , the term of this patent is extended or adjusted under 35 Allison et al, Combined Head and Eye Tracking System for U . S . C . 154 (b ) by 0 days . Dynamic Testing of the Vestibular System . IEEE Transactions on Biometical Engineering, vol. 43 , No. 11 Nov . 1996 . ( USA ) . (21 ) Appl. No. : 15 / 162, 300 Primary Examiner — Mohammed Hasan (74 ) Attorney, Agent, or Firm — Kilpatrick Townsend & ( 22 ) Filed : May 23, 2016 Stockton LLP (65 ) Prior Publication Data (57 ) ABSTRACT US 2016 / 0262608 A1 Sep . 15 , 2016 A system and method for using a virtual reality or an augmented reality environment for the measurement and / or improvement ofhuman vestibulo -ocular performance can be Related U . S . Application Data implemented by combining a video camera based eye ori (63 ) Continuation - in -part of application No. 14 /326 , 335 , entation sensor, a head orientation sensor, a display , and an filed on Jul. 8 , 2014 , now Pat. No. 9 , 370 , 302 . electronic circuit that connects the eye sensor, head sensor, and display . The system and method can be operated in the (51 ) Int. Cl. range of frequencies between 0 .01 Hertz and 15 Hertz . The A61B 3 / 14 ( 2006 .01 ) system and method can use a Fourier transform to compute A61B 3 / 00 ( 2006 .01 ) a gain and a phase . The system and method can be used for (Continued ) measuring vestibulo -ocular reflex , dynamic visual acuity , (52 ) U . S . CI. dynamic visual stability , kinetic visual acuity , retinal image CPC . .. .. A61B 3 /0041 (2013 .01 ) ; A61B 3 / 032 stability, or foveal fixation stability in a non -clinical setting . ( 2013. 01 ) ; A61B 3 / 113 ( 2013 .01 ) ; A61B 3 / 18 The system and method can be completely portable , head ( 2013 .01 ) worn , and self- contained . ( Continued ) 24 Claims, 13 Drawing Sheets 106 1107 1087 100 104 try 98 - - 122 124 126 H Head orient 1 Eye camera sensor signal video Display preprocessor processor Interface L Centralcentral processing unit unik 1 _132 Interface / communication unit Memory Unit miestaca- - - communication und —13 * nomory Unit] _ * 3 * 140 External device ( s ) US 9 , 788 , 714 B2 Page 2 (51 ) Int . CI. 7 , 753 , 523 B2 7 / 2010 Kiderman et al. 7 ,866 , 818 B2 1 / 2011 Schroeder et al. A61B 5 / 00 ( 2006 . 01) 7 , 931 , 370 B2 4 / 2011 Bartomeu A61B 3 / 032 ( 2006 .01 ) 7 ,988 , 287 B1 8 / 2011 Butler et al . G06T 19 / 00 ( 2011 .01 ) 8 , 253 , 814 B2 8 / 2012 Zhang et al. GO2B 27 /01 ( 2006 . 01 ) 8 ,285 , 416 B2 10 / 2012 Cho et al . G06F 3 /01 ( 2006 .01 ) 8 , 510 ,166 B2 8 / 2013 Neven 8 , 529 , 463 B2 9 /2013 Della Santina et al . A61B 3 / 113 ( 2006 . 01 ) 8 , 696 , 126 B2 4 /2014 Yoo et al . A61B 3 / 18 ( 2006 . 01 ) 8 , 764 , 193 B2 . 7 / 2014 Kiderman et al. A61B 5 / 11 ( 2006 .01 ) 2002 / 0118339 AL 8 /2002 Lowe A61B 3 / 11 ( 2006 .01 ) 2006 /0098087 AL 5 / 2006 Brandt et al . ( 52 ) U . S . CI. 2006 / 0206175 A1 * 9 / 2006 Fernandez Tournier .. .. A63B 26 /003 ??? A61B 5 / 11 ( 2013 . 01 ) ; A61B 5 /4023 607 / 88 ( 2013 .01 ) ; A61B 5 /4064 ( 2013 .01 ) ; A61B 2006 /0270945 AL 11 /2006 Ghajar 5 /6803 ( 2013 .01 ) ; A61B 5 / 7257 ( 2013 .01 ) ; 2009 / 0021695 Al 1/ 2009 Scarpino A61B 5 / 744 ( 2013 .01 ) ; G02B 27/ 017 2010 / 0036289 Al 2 / 2010 White et al . ( 2013 .01 ) ; G06F 3 /012 (2013 .01 ) ; G06T 2010 / 0092049 Al 4 /2010 Schroeder et al . 2010 /0198104 A1 8 / 2010 Schubert et al. 19 /006 (2013 .01 ) ; A61B 3 / 112 (2013 .01 ) ; 2010 / 0280372 A1 11 / 2010 Poolman et al . G06F 3 /013 ( 2013 .01 ) 2011/ 0176106 Al 7 / 2011 Lewkowski 2012 /0133892 A1 5 /2012 Furman et al . ( 56 ) References Cited 2013 /0278899 A1 10 /2013 Waldorf et al . 2014 /0111771 Al 4 / 2014 Liu U . S . PATENT DOCUMENTS 2014 / 0192326 Al 7 / 2014 Kiderman et al. 2014 / 0327880 A1 11/ 2014 Kiderman et al. 5 ,550 ,601 A 8 / 1996 Donaldson 2015 / 0038803 A1 2 /2015 Uhlig et al. 5 , 555 , 895 A 9 / 1996 Ulmer et al . 2015 /0223683 A1 8 /2015 Davidovics et al . 5 , 838 , 420 A 11/ 1998 MacGregor Donaldson 2015 /0243099 A1 8 / 2015 Schowengerdt 5 , 919 , 149 A 7 / 1999 Allum 2015 / 0245766 Al 9 / 2015 Rennaker et al. 5 , 942, 954 A 8 / 1999 Galiana et al. 2015 / 0335239 Al 11 /2015 Macfougall 5 ,953 , 102 A 9 / 1999 Berry 2016 / 0033750 Al 2 / 2016 Nunnink et al. 6 , 796 , 947 B2 9 / 2004 Watt et al. 2016 /0062459 Al 3 / 2016 Publicover et al. 7 ,380 , 938 B2 6 / 2008 Chmielewski et al. 2016 / 0081546 A1 3 / 2016 MacDougall 7 , 401 , 920 B1 7 / 2008 Kranz et al . 2016 /0085302 A1 3 /2016 Publicover et al . 7 , 448, 751 B2 11/ 2008 Kiderman et al . 2016 /0106315 A14 / 2016 Kempinski 7 , 500 ,752 B2 3 / 2009 Nashner 2016 /0110920 AL 4 / 2016 Schowengerdt 7 ,651 , 224 B2 1 / 2010 Wood et al. 2016 /0132726 A15 / 2016 Kempinski et al . 7 , 682 , 024 B2 3 / 2010 Plant et al. 2016 /0242642 A1 * 8 /2016 Migliaccio A61B 3 /113 7 , 727 , 162 B2 6 / 2010 Peterka 7 , 731 , 360 B2 . 6 /2010 MacDougall et al . * cited by examiner U .S . Patentatent Oct. 17 , 2017 Sheet 1 of 13 US 9 ,788 ,714 B2 106 110 108100 - 104 WWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWWW - 98 gos w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w W M W W * xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx w 122m 124 wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww 126 * Head orient Eye camera sensor signal video Display w processor vo women preprocessor interface * wwwwwwwwwwwwww x WwWWWWWW7 www * * w BoooOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOO Central processing unit * * wang 134 w * Interface communication unit 898Memory Unitw Memory Unit| _ 134 * 120 ww w w w w w w w w w nicationw w w w unitw ) w _w 138w w w w w w w w w w w w w w w w w w External device ( s ) 140 FIG . 1 atent Oct. 17 , 2017 Sheet 2 of 13 US 9 ,788 ,714 B2 ZOZ - 200 wood Www 108 ccccccccccccccccood M4wmia Gry n mwmw * 999999999999999999999 210 -3 + ??????????? wwwww wwwww + 207 WWW. 204 COC 206 FIG . 2 atent Oct. 17 , 2017 Sheet 3 of 13 US 9 ,788 ,714 B2 98 - 300 illimin v ::: ** ... .. .* * * * : : : : : : FIG . 3 320 86 - 310 XXXX& * DOO!XXXSCOXX ** A* * * * XXXkO.XXXKOOS ** # # # # # # # # # # # FIG . 4 U . S . Patent Oct. 17, 2017 Sheet 4 of 13 US 9 , 788, 714 B2 pr 108 200 p co cocooooooooo - 207 206 { ?????? ????rs Display dows Display Sports 2101 1 your * * wwwwww * * * * * * www wwwwwwww * od0000000000000000000000000000000000000000000 * * * * * nononnan DORO osofia9000000000000000000000000000000 ???????????????????????????????????????????? wwwsawon * M * * * K K KKK KKK * * * * * * * * * * * * * * * * * * * * 14 oss. 1261 Orientation Eye camera signal video Display preprocessor WW processor XXXXXXXXXXX interface wwwwww OOONOODOVOOOOOOOOOOOOOOO www wwwm WWWWWWw Central processing unit www wwwwwww XXXXXX Interface communication unit Memory Unit Interface communication unit| 138 memory* * *W91444444 Unit? _ 134 130 w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w w wwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwwww External device ( s ) FIG . 5 atent Oct. 17 , 2017 Sheet 5 of 13 US 9, 788, 714 B2 602 ) Establish head -worn virtual reality or augmented reality unit mononen wwwwwwwwwwwwwwwwwwwwww mm 608 604 - 606 Establish eye Establish Establish head tracking video display orientation camera on unit on unit sensor on unit + + + + + + + + + + + wwwx w 610 om Present target visual element in display center pon612 Present target + element 20° to + one side ww porn 616 Repeat as many times as needed 642 hap 614 prom 640 Measure eye Present target Measure head orientation element 20° to orientation changes other side changes 644 *Determine Determine vestibulo -- ocularocular performanceperformance FIG . 6 atent Oct. 17 , 2017 Sheet 6 of 13 US 9 ,788 ,714 B2 602 Establish head -worn virtual reality or augmented reality unit 7608 W 604 -606 Establish eye Establish staffsix & ??? tracking video display orientation on unit w camera on unit. sensor on unit wwwwww + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + - 620 Present target visual element in + display center 622 Subject moves head 20° to one + + + + + + + + + + + + + + + + + + + + + + + side while looking ww! at target + + 626 + + + + + + + + + Repeat as many + + + + + + ??? # f … ????? + howe odgOOOOOOOOOOOOO000000000000000000000000 .+ po 642 -624 - 640 . + + + + + + + + + + + + + W Measure eye w Subjectmoves Measure head orientation head 20° to other orientation w changes side while looking changes at target 644 w W www Determine vestibulo -ocular performance . FIG . 7 atent Oct. 17 , 2017 Sheet 7 of 13 US 9 ,788 ,714 B2 602 Establish head -worn virtual reality or augmented reality unit + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 608 604 606 Establish eye Establish Establish head tracking video display PODDODOCOOOOOO orientation camera on unit on unit sensor on unit w Present target visual element in display center + + + + p 632 WWW Clinician moves head 20° to one side while subject looks at target WWWWWW w Repeat as many times as needed + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + -- 642 0000000000OOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOOX 634 640 W W Measure eye Clinician moves Measure head orientation WW head 20° to other me orientation changes side while subject changes looks at target wwwwwwwwwwwwww -- - 644 + + + + + + + + + + + + + + + + + + + + + + + + + + Determine vestibulo - ocular pertormance FIG .