111111111111111111111111111111111111111111111111111111111111111111111111

(12) United States Patent (io) Patent No.: US 9,623,324 B2 Pope et al. (45) Date of Patent: *Apr. 18, 2017

(54) PHYSIOLOGICALLY MODULATING A63F 131213 (2014.01) VIDEOGAMES OR SIMULATIONS WHICH A63F 13142 (2014.01) USE MOTION-SENSING INPUT DEVICES A63F 131212 (2014.01) (52) U.S. Cl. (71) Applicant: The United States of America as CPC ...... A63F 131211 (2014.09); A63F 131212 represented by the Administrator of (2014.09); A63F 131213 (2014.09); A63F the National Aeronautics and Space 13142 (2014.09) Administration, Washington, DC (US) (58) Field of Classification Search CPC ..... A63F 2300/1012; A63B 2024/0012; A63B (72) Inventors: Alan T. Pope, Poquoson, VA (US); 2220/806; A63B 69/3608; G09B 19/0038; Chad L. Stephens, Poquoson, VA (US); G09B 19/00 Nina Marie Blanson, Yorktown, VA USPC ...... 434/236, 247-261 (US) See application file for complete search history. (73) Assignee: The United States of America as (56) References Cited represented by the Administrator of the National Aeronautics and Space U.S. PATENT DOCUMENTS Administration, Washington, DC (US) 6,450,820 B1 * 9/2002 Palsson ...... G09B 19/22 (*) Notice: Subject to any disclaimer, the term of this 434/236 patent is extended or adjusted under 35 2011/0009193 Al * 1/2011 Bond ...... A63F 13/06 U.S.C. 154(b) by 396 days. 463/36 This patent is subject to a terminal dis- * cited by examiner claimer. Primary Examiner Andrew Iwamaye (21) Appl. No.: 14/212,159 Assistant Examiner Jerry-Daryl Fletcher (74) Attorney, Agent, or Firm Andrea Z. Warmbier; (22) Filed: Mar. 14, 2014 Jennifer L. Riley; Jonathan B. Soike

(65) Prior Publication Data (57) ABSTRACT US 2014/0256431 Al Sep. 11, 2014 New types of controllers allow a player to make inputs to a video game or simulation by moving the entire controller Related U.S. Application Data itself or by gesturing or by moving the player's body in whole or in part. This capability is typically accomplished (63) Continuation-in-part of application No. 13/166,166, using a wireless input device having accelerometers, gyro- filed on Jun. 22, 2011, now Pat. No. 8,827,717. scopes, and a camera. The present invention exploits these (60) Provisional application No. 61/361,084, filed on Jul. wireless motion-sensing technologies to modulate the play- 2, 2010, provisional application No. 61/499,733, filed er's movement inputs to the videogame based upon physi- on Jun. 22, 2011, provisional application No. ological signals. Such biofeedback-modulated video games 61/781,355, filed on Mar. 14, 2013. train valuable mental skills beyond eye-hand coordination. These psychophysiological training technologies enhance (51) Int. Cl. personal improvement, not just the diversion, of the user. A63B 69100 (2006.01) A63F 131211 (2014.01) 18 Claims, 13 Drawing Sheets U.S. Patent Apr. 18, 2017 Sheet 1 of 13 US 9,623,324 B2

24 22 IR XMTR RCVR ~— 18

GYROSCOPE 16 /,,— ACCELEROMETER 14 /1-- PROCESSOR 12 Light Source INPUT DEVICE U.S. Patent Apr. 18, 2017 Sheet 2 of 13 US 9,623,324 B2

m 46 PHYSIOLOGICAL PHYSIOLOGICAL SIGNAL ' SIGNAL MEASUREMENT 54 42 GYROSCOPE AID 52 CONVERTER ACCELEROMETER 50 44 POTENTIOMETER PROCESSOR 48 45 POWER SUPPLY MEMORY

1111111111251w ell

40

PHYSIOLOGICAL PHYSIOLOGICAL SIGNAL SIGNAL MEASUREMENT 61

MEMORY 60 63 66 42 A/D PROCESSOR PROCESSOR SENSOR BAR CONVERTER

SENSOR BAR PHYSIOLOGICAL SIGNAL DISPLAY 68

62 FIG. 3 U.S. Patent Apr. 18, 2017 Sheet 3 of 13 US 9,623,324 B2 U.S. Patent Apr. 18, 2017 Sheet 4 of 13 US 9,623,324 B2 U.S. Patent Apr. 18, 2017 Sheet 5 of 13 US 9,623,324 B2

CO (DC, co CD C7 C I C~ LL L LL

J J D 0 (D (D (D (D

LJf]

D 0 D

VJ V V V J

J J J J

J J J J

J J J J U.S. Patent Apr. 18, 2017 Sheet 6 of 13 US 9,623,324 B2

FIG. 7A FIG. 7

L7 R 7 L7 R7

76 76 L6 ' R6 L6 R6

7$,, Ra L5 i8 R5 G

L4 R 4 L4 R 4

L3 R 3 L3 R 3

R2 L2 R2 0

L1 R1 L1 R1

FIG. 7C FIG. 7D

L7 R7 L7 R7 76 7:E Z- L6 R6 L~ R5 78 78 R5 L5 RS G

L4 R 4 L4 R 4

L3 R3 L3 R3

L2 R 2; L2 R 2

R1 L1 R1 0 U.S. Patent Apr. 18, 2017 Sheet 7 of 13 US 9,623,324 B2

C r U.S. Patent Apr. 18, 2017 Sheet 8 of 13 US 9,623,324 B2

R

PHYSIOLOGICAL PHYSIOLOGICAL SIGNAL MEASUREMENT 1- 61

MEMORY 60 63 42 A~ PROCESSOR PROCESSOR CONVERTER 70

PHYSIOLOGICAL 86 SIGNAL DISPLAY MOTION TABLE SENSOR BAR

62 SENSOR BAR

88 U.S. Patent Apr. 18, 2017 Sheet 9 of 13 US 9,623,324 B2

ARM U.S. Patent Apr. 18, 2017 Sheet 10 of 13 US 9,623,324 B2

EU-4

Physiological Signal Physiological Signal Measurement

Memory 42 60 63

A/D Converter Processor Processor r— 102

Physiological Signal Servomechanism C Display Galvanometer

m 104 Mirror

F(. 11 U.S. Patent Apr. 18, 2017 Sheet 11 of 13 US 9,623,324 B2

40

Physiological 'Signal Physiological Signal Measurement 69

Memory EN 60 /-63

A/D Converter Processor Processor EM

Physiological Signal Servomechanism Or Display Galvanometer

106 62 Lens U.S. Patent Apr. 18, 2017 Sheet 12 of 13 US 9,623,324 B2

Physiological Signal Physiological Signal Measurement

A/D Converter

i i• •

melt; LED(s)

N

FIG. 1 U.S. Patent Apr. 18, 2017 Sheet 13 of 13 US 9,623,324 B2

1400 (Measure 1402 Physiological Signal

Convert To Digital 1404 Signal

Determine Difference Between Physiological Signal And Desired Physiological Signal Level

Adjust Mirrors, Lenses, and/or LEDs To Modify Image Received By MW Camera Based At Least In Part: On the Determined Difference

FIG. 14 US 9,623,324 B2 2 PHYSIOLOGICALLY MODULATING enhancing physiological self-regulation skills or for thera- VIDEOGAMES OR SIMULATIONS WHICH peutic amplification of healthful physiological characteris- USE MOTION-SENSING INPUT DEVICES tics. Biofeedback, an effective treatment for various physiological problems, can be used to optimize physiologi- CROSS-REFERENCE TO RELATED 5 cal functioning in many ways. The benefits of biofeedback APPLICATION(S) can, however, only be attained through a number of training sessions, and such gains can only be maintained over time This patent application is a continuation-in-part of, and through regular practice. Adherence to regular training, claims the benefit of and priority to, co-pending U.S. appli- especially at home, has been a problem that has plagued the cation Ser. No. 13/166,166 entitled PHYSIOLOGICALLY io field of biofeedback and limited its utility to date. U.S. Pat. MODULATING VIDEOGAMES OR SIMULATIONS No. 6,450,820, which is incorporated herein by reference WHICH USE MOTION-SENSING INPUT DEVICES filed thereto, addressed this to a degree, by blending biofeedback Jun. 22, 2011, which claims the benefit of and priority to into video games, which increases motivation. Biofeedback- each of U.S. Provisional Patent Application No. 61/361,084, modulated video games employing the technology of U.S. filed Jul. 2, 2010, and U.S. Application Ser. No. 61/499,733, 15 Pat. No. 6,450,820 are games that respond to physiological entitled METHOD AND SYSTEM FOR PHYSIOLOGI- signals as well as mouse, joystick or game controller input; CALLY MODULATING VIDEOGAMES WHICH USE these games embody the concept ofimproving physiological HAND AND BODY MOTION-SENSING INPUT functioning by rewarding specific healthy body signals with DEVICES filed Jun. 22, 2011, the entire contents of each of success at playing a video game. Such a biofeedback- the foregoing applications is incorporated herein by refer- 20 modulated game method blends biofeedback into popular ence in its entirety. This patent application additionally off-the-shelf video games in such a way that the games do claims the benefit of and priority to U.S. Application Ser. not lose their entertainment value. This method uses physi- No. 61/781,355, entitled METHOD AND SYSTEM FOR ological signals (e.g., electroencephalogram frequency band PHYSIOLOGICALLY MODULATING VIDEOGAMES ratio) not simply to drive a biofeedback display directly, or WHICH USE HAND AND BODY MOTION-SENSING 25 periodically modify a task as in other systems, but to INPUT DEVICES filed Mar. 14, 2013, the entire contents of continuously modulate parameters (e.g., game character which is incorporated herein by reference in its entirety. This speed and mobility) of a game task in real time while the application is also related to commonly owned, co-pending game task is being performed by other means (e.g., a game U.S. application Ser. No. 13/166,226 entitled TEAM ELEC- controller). Biofeedback-modulated video games represent a TRONIC GAMEPLAY COMBINING DIFFERENT 3o new generation of computer and video game environments MEANS OF CONTROL, the entire contents of which is that train valuable mental skills beyond eye-hand coordina- incorporated herein by reference in its entirety. tion. These psychophysiological training technologies are poised to exploit the revolution in interactive multimedia STATEMENT REGARDING FEDERALLY home entertainment for the personal improvement, not just SPONSORED RESEARCH AND 35 the diversion, of the user. The technology of U.S. Pat. No. DEVELOPMENT 6,450,820 modulates the manual inputs that a player makes to the buttons or joysticks of a video game hand controller. The invention described herein was made by employees This modulation is based on measurements of physiological of the United States Government and may be manufactured signals of the player. and used by or for the Government of the United States of 40 However, a new type of video game (illustrated in FIG. 1) America for governmental purposes without the payment of has a game controller that allows a player to make inputs to any royalties thereon or therefore. a video game by moving the entire controller itself. This capability is accomplished in operator-controlled game input FIELD OF THE INVENTION devices, including the Nintendo wireless Wii remote using 45 accelerometers, and an infrared LED tracking camera, and/ The present invention relates generally to the use of or the remote's accessories (e.g., the Nintendo Nunchuk and biofeedback to modify a subject's behavior, mental state, MotionPlus devices using accelerometers and gyroscopes) and/or physiological functioning. More specifically, the (Nintendo, Wii and MotionPlus are registered trademarks of invention relates to apparatus and methods for modulating Nintendo of America Inc.) an operator's control input to an electronic game or simu- 50 Another new type of video game has a game controller lator, via a motion and/or position-sensing input device, in that employs different camera image analysis methods than response to measured physiological activity, such as auto- the Wii®. These new type video game systems, such as the nomically-mediated and/or EEG physiological activity, Leap Motion®, Sony® Playstation® Move® and Micro- wherein the player thereby learns to control the physiologi- soft(k Kinect, use cameras that sense the player's image in cal and/or EEG activity. 55 whole or in part (e.g., the Leap Motion® or the Kinect(k) or the image of an illuminated object on the player's control BACKGROUND device (e.g., the Move(k). The video game system of FIG. 1 comprises a game Physiologically modulated video games add to the enter- console 10 that may either operate using infrared LED tainment value of video games by adding the challenge of 60 tracking (such as a Wii(k) or player image tracking (such as requiring a player to master physiological self-regulation a Leap Motion® or Kinect(k) or illuminated object on the skill as well as hand-eye coordination. Thus, controlling the player's controller tracking (such as the Move(k). In an physiological state, or learning to self-induce physiologic infrared LED tracking configuration the system may include changes, is an additional skill requirement or challenge a controller or input device 12, a display 24 (such as a added to games. Although physiologically modulated vid- 65 television), and left and right sensor bars 26, 28 (note, while eogames have entertainment value by making games more the left and right sensor bars are illustrated as separate exciting, they also have advantages for encouraging health- devices, in practice the left and right sensor bars are often US 9,623,324 B2 3 4 incorporated into a single device having left and right lating a voltage input to one or more motion and/or position- portions). The game input device 12 comprises one or more sensing elements of the input device. The one or more gyroscopes 18 and one or more accelerometers 16 for motion and/or position-sensing elements may comprise one sensing movement of the input device, an infrared receiver or more accelerometers and/or one or more gyroscopes. or camera 20 for detecting the infrared light emitting diodes 5 Modifying the motion and/or position sensing capability (LEDs) of the sensor bars, a transmitter 22 (e.g., Bluetooth) of the operator controlled input device may comprise modi- for communicating with the console 10, user input devices fying one or more position reference signal elements that are (e.g., buttons, etc.) (not illustrated), and a processor 14 for used by the input device to determine a motion or position processing the inputs to the game input device and control- of the operator controlled input device relative to the one or ling outputs from the game input device. The game system 10 more position reference signal elements. The one or more uses the known positional relationship between the left and position reference signal elements may comprise two light right sensor bars to determine the game input device's emitting diodes (LEDs) or two sets of LEDs having a fixed approximate distance from the sensor bars and the input distance therebetween. The operator controlled input device device's position and/or motion relative to the sensor bars. may comprise a light detector for detecting light emitted In a player image tracking configuration or illuminated 15 from the LEDs. Modifying one or more position reference object on the player's controller tracking configuration the signal elements may comprise mechanically translating the system may not include the sensor bars 26 and 28 and IR two LEDs or two sets of LEDs from a first position to a receiver 20, and may rather include a camera or sensor 21 second position, the difference between the first and second connected via wires or wirelessly to the game console 10. positions being based on either (a) the difference between The camera 21 may enable the game console 10 to track the 20 the measured at least one physiological signal and the player's image in a player image tracking configuration and corresponding predetermined desired physiological signal the camera 21 may enable the game console 10 to track the level, or (b) the measured at least one physiological signal. illumination of the controller 12 caused by the light source The one or more position reference signal elements may 15 in the controller 12 in a illuminated object on the player's comprise a plurality of LEDs,two of which emit light at any controller tracking configuration. 25 particular time. The operator controlled input device may However, the art described in U.S. Pat. No. 6,450,820 comprise a light detector for detecting light emitted from the requires further advancement to work with these wireless LEDs. Modifying one or more position reference signal controllers. elements may comprise changing which two ofthe LEDs are emitting light at any particular time. Changing which two of BRIEF SUMMARY OF THE INVENTION 30 the LEDs are emitting light at any particular time may comprise turning pairs of LEDs on and off in a predeter- New types of video game controllers allow a player to mined pattern based on either (a) the difference between the make inputs to a video game or simulation by moving the measured at least one physiological signal and the corre- entire controller itself or by gesturing or by moving the sponding predetermined desired physiological signal level, player's body in whole or in part. This capability is typically 35 or (b) the measured at least one physiological signal. accomplished using operator controlled input devices hav- The plurality of LEDs may comprise two groups of ing accelerometers, gyroscopes, and/or cameras, such as linearly arranged LEDs, each of the two linearly arranged using infrared LED tracking cameras or using cameras that groups being positioned substantially horizontally. Alterna- sense a player's image in whole or in part or the image of an tively, the plurality of LEDs may comprise two groups of illuminated object on the player's control device. The pres- 40 linearly arranged LEDs, each of the two linearly arranged ent invention exploits these wireless motion-sensing tech- groups being positioned substantially vertically. As another nologies to modulate the player's movement inputs to the alternative, the plurality of LEDs may comprise two groups videogame based upon physiological signals. of LEDs, each of the groups being arranged in two linearly In another embodiment of the invention, a method is arranged sub-groups, each sub-group of a group crossing the provided for modifying the effect of an operator controlled 45 other sub-group at a center of each sub-group and at a input device on an interactive device to encourage the perpendicular angle. self-regulation of at least one physiological activity by the Measuring at least one physiological activity may com- operator. The operator controlled input device has motion prise measuring at least one of: autonomically-mediated and/or position-sensing capability. The interactive device physiological activity, and brainwave electrical activity. The comprises a display area which depicts images, and appa- 50 autonomic physiological activity may comprise at least one ratus for receiving at least one input from the operator of: skin temperature; skin conductance; electrical activity of controlled input device to thus permit the operator to control muscles; blood flow; heart rate; heart rate variability; and and interact with at least some of the depicted images. The respiratory rate. The brainwave electrical activity may com- method for modifying comprises the steps of measuring at prise at least one of: event-related potentials; and at least one least one physiological activity of the operator to obtain at 55 brainwave frequency band; the at least one brainwave fre- least one physiological signal having a value indicative of quency band comprising at least one of: (a) theta,(b) alpha, the level of the at least one physiological activity; and (c) sensorimotor response (SMR),(d) beta, and (e) gamma. modifying the motion and/or position sensing capability of The method may further comprise displaying the at least the operator controlled input device in response to (a) one physiological signal having a value indicative of the changes in the measured at least one physiological signal 60 level ofthe at least one physiological activity. Displaying the and/or (b) differences between the measured at least one at least one physiological signal may comprise displaying physiological signal and a corresponding predetermined and monitoring the physiological signal on a signal display desired physiological signal level, thereby modifying the system substantially separate from the interactive device. ability of the operator to control and interact with at least However, such a separate display can be connected wire- some of the depicted images. 65 lessly, in some instances, to the interactive device. Modifying the motion and/or position sensing capability In addition to the method for modifying the effect of an of the operator controlled input device may comprise modu- operator controlled input device on an interactive device to US 9,623,324 B2 5 6 encourage the self-regulation of at least one physiological an operator controlled input device on an interactive device activity by the operator, as described above, yet other to encourage the self-regulation of at least one physiological aspects and embodiments of the present invention are activity by the operator, in accordance with embodiments of directed to an apparatus for modifying the effect of an the present invention; operator controlled input device on an interactive device to 5 FIG. 8 illustrates an alternative sensor bar of a system for encourage the self-regulation of at least one physiological modifying the effect of an operator controlled input device activity by the operator. on an interactive device to encourage the self-regulation of In an embodiment, a method for modifying the effect of at least one physiological activity by the operator, in accor- an operator input on an interactive device to encourage the dance with embodiments of the present invention; to self-regulation of at least one physiological activity by the FIG. 9 illustrates a block diagram of portions of a system operator, the interactive device comprising a display area for modifying the effect of an operator controlled input that depicts images and a camera for receiving at least one device on an interactive device to encourage the self- input from the operator to thus permit the operator to control regulation of at least one physiological activity by the and interact with at least some of the depicted images, the 15 operator, in accordance with alternative embodiments of the method for modifying comprising the steps of measuring at least one physiological activity of the operator to obtain at present invention; least one physiological signal having a value indicative of FIG. 10 illustrates a potential positioning option for a the level of the at least one physiological activity; and device to modify the input to a camera of a game system in modifying the motion and/or position sensing capability of accordance with alternative embodiments of the present the camera for receiving at least one input from the operator 20 invention; in response to (a) changes in the measured at least one FIGS. 11-13 illustrate block diagrams of portions of physiological signal and/or (b) differences between the systems for modifying the effect of a camera on an interac- measured at least one physiological signal and a correspond- tive device to encourage the self-regulation of at least one ing predetermined desired physiological signal level, physiological activity by the operator, in accordance with thereby modifying the ability of the operator to control and 25 alternative embodiments of the present invention; and interact with at least some of the depicted images. In various FIG. 14 is a process flow diagram illustrating an embodi- embodiments, the modifying the motion and/or position ment method for modifying images received by a camera. sensing capability of the camera for receiving at least one input from the operator may include modifying the image of DETAILED DESCRIPTION OF THE the operator or the image of an illuminated object on the so INVENTION operator's control device using mirrors, lenses, and/or LEDs. The present invention now will be described more fully hereinafter with reference to the accompanying drawings, in BRIEF DESCRIPTION OF THE SEVERAL which preferred embodiments of the invention are shown. VIEWS OF THE DRAWING(S) 35 This invention may, however, be embodied in many different forms and should not be construed as limited to the embodi- Having thus described the invention in general terms, ments set forth herein; rather, these embodiments are pro- reference will now be made to the accompanying drawings, vided so that this disclosure will be thorough and complete, which are not necessarily drawn to scale, and wherein: and will fully convey the scope of the invention to those FIG. 1 illustrates a block diagram of a video game system 40 skilled in the art. Like numbers refer to like elements that can benefit from embodiments of the present invention; throughout. FIG. 2. illustrates a block diagram of portions of a system New types of video game controllers allow a player to for modifying the effect of an operator controlled input make inputs to a video game or simulation by moving the device on an interactive device to encourage the self- entire controller itself or by gesturing or by moving the regulation of at least one physiological activity by the 45 player's body in whole or in part. This capability is typically operator, in accordance with embodiments of the present accomplished using operator controlled input devices hav- invention; ing accelerometers, gyroscopes, and/or cameras, such as FIG. 3 illustrates a block diagram of portions of a system using infrared LED tracking cameras or using cameras that for modifying the effect of an operator controlled input sense a player's image in whole or in part or the image of an device on an interactive device to encourage the self- 50 illuminated object on the player's control device. The pres- regulation of at least one physiological activity by the ent invention exploits these wireless motion-sensing tech- operator, in accordance with alternative embodiments of the nologies to modulate the player's movement inputs to the present invention; videogame based upon physiological signals. Examples of FIGS. 4 and 5 illustrate potential positioning options for such new type of controllers include the Leap Motion®, sensor bars of a system for modifying the effect of an 55 Nintendo® WiM, Sony® Playstation® Move®, and Micro- operator controlled input device on an interactive device to soft(k Kinect. encourage the self-regulation of at least one physiological Embodiments of the invention modify the effect of an activity by the operator, in accordance with alternative operator controlled input device having motion and/or posi- embodiments of the present invention; tion-sensing capability. Embodiments of the invention typi- FIGS. 6A-D illustrate one possible lighting pattern for 60 cally measure at least one physiological activity of the horizontal sensor bars of a system for modifying the effect operator to obtain at least one physiological signal having a of an operator controlled input device on an interactive value indicative of the level of the at least one physiological device to encourage the self-regulation of at least one activity; and modify the motion and/or position sensing physiological activity by the operator, in accordance with capability ofthe operator controlled input device in response embodiments of the present invention; 65 to (a) changes in the measured at least one physiological FIGS. 7A-D illustrate one possible lighting pattern for signal and/or (b) differences between the measured at least vertical sensor bars of a system for modifying the effect of one physiological signal and a corresponding predetermined US 9,623,324 B2 7 8 desired physiological signal level, thereby modifying the galvanometers, servomechanisms and galvanometers are ability of the operator to control and interact with at least used merely as examples of actuating devices to adjust some of the depicted images. lenses to better illustrate the various embodiments. Lenses In various embodiments, the player inputs to systems that may be actuated by other devices and/or lenses themselves use cameras that sense a player's image in whole or in part 5 may adjust (e.g., open, close, etc.), and servomechanisms or the image of an illuminated object on the player's control and galvanometers as actuating devices are not intended to device (e.g., Leap Motion®, Sony® Playstation® Move® or limit the scope of the invention. Microsoft® Kinect) may be similarly modulated to modify In another embodiment physiological modulation method the motion and/or position sensing capability of the camera for these technologies, the player's image in whole or in part in response to (a) changes in the measured at least one io (e.g., for the Kinect® or the Leap Motion(k) or the image of physiological signal and/or (b) differences between the an illuminated object on the player's control device (e.g. for measured at least one physiological signal and a correspond- the Move(k) may be received by the cameras along with the ing predetermined desired physiological signal level, images of LEDs of various colors (e.g., red, blue, green, or thereby modifying the ability of the operator to control and infrared) positioned in the periphery surrounding the camera interact with at least some of the depicted images. 15 lens or elsewhere in the visual field of the camera, whose In an embodiment physiological modulation method, the relative brightness may be dynamically varied. The differ- player's image in whole or in part (e.g. for the Kinect® or ential brightness of the LEDs in this modulation method, the Leap Motion(k) or the image of an illuminated object on and, consequently, the degree of modification ofthe received the player's control device (e.g., for the Move(k) may be image, may be determined by control signals derived from reflected into the cameras by mirrors whose positioning is 20 the physiological difference signal defined previously. For dynamically driven by servomechanisms or galvanometers. example, a large current momentary departure of the physi- This method of dynamically positioning mirrors may be ological signal value from the target signal value may result similar to the method used in instruments designed to track in a pattern ofilluminated LEDs that causes large distortions eye movements. The amplitude of movement of the posi- superimposed on the player's intentional movements of the tioning servos or galvanometers in this modulation method, 25 image on the camera element, and, consequently, large and, consequently, the degree of disruption, may be deter- disruptions in the stability of the player's intentional control mined by control signals derived from the physiological movements. In this way, the player may be rewarded for difference signal. For example, a large current momentary diminishing the physiological signal difference by an departure of the physiological signal value from the target improved ability to accurately position the player-controlled signal value may result in large amplitude mirror move- 30 object on the screen. The discussions of LEDs for distorting ments, and, consequently, large movement disturbances or deflecting images of a camera herein are used merely as superimposed on the player's intentional movements of the examples of illumination devices to better illustrate the image across the camera element, and, consequently, large various embodiments. Other illumination devices may sub- disruptions in the stability of the player's intentional control stituted for LEDs in the various examples, and are not movements. In this manner, the player is rewarded for 35 intended to limit the scope of the invention. diminishing this physiological signal difference by an In the various embodiments, the devices that may modify improved ability to control the player controlled object (e.g., the received image of the player or the illuminated object on a cursor) on the screen. While discussed herein as being the player's control device may be wireless and independent actuated by servomechanisms or galvanometers, servo- of video game or simulation system itself(e.g., the Kinect® mechanisms and galvanometers are used merely as 40 or Move® or the Leap Motion® systems) and may operate examples of actuating devices to move mirrors to better without requiring access to the video game or simulation illustrate the various embodiments. Mirrors may be actuated processor or modifications to the video or game simulation by other devices, and servomechanisms and galvanometers system software. as actuating devices are not intended to limit the scope of the Referring now to FIG. 2, a block diagram of portions of invention. 45 a system for modifying the effect of an operator controlled In another embodiment physiological modulation input device on an interactive device is illustrated in accor- method, the player's image in whole or in part (e.g., for the dance with embodiments of the present invention. In the Kinect® or the Leap Motion(k) or the image of an illumi- embodiment of FIG. 2, the voltage input to one or more nated object on the player's control device (e.g., for the motion and/or position-sensing elements (e.g., accelerom- Move(k) may be distorted and/or deflected into the cameras 50 eters and/or gyroscopes) of the input device is modulated by lenses whose positioning is dynamically driven by ser- (increased or decreased). This modulation ofthe voltage that vomechanisms or galvanometers. The amplitude of move- powers the accelerometers and/or gyroscopes in operator ment of the positioning servos or galvanometers in this controlled input devices changes (increasing or reducing) modulation method, and, consequently, the degree of dis- the effect of the player's movement of the controller on the ruption, may be determined by control signals derived from 55 player-controlled object on the screen. The degree of the the physiological difference signal defined previously. For increase or reduction may be programmed to be proportional example, a large current momentary departure of the physi- to the difference between the player's current momentary ological signal value from the target signal value may result physiological signal value and a pre-selected target value. In in large amplitude lens movements and, consequently, large this way, the player is rewarded for diminishing this differ- distortions and/or deflections superimposed on the player's 60 ence by an improved ability to control the player-controlled intentional movements of the image on the camera element object (e.g., a cursor) on the screen and/or increased power and, consequently, large disruptions in the stability of the in the action of the player-controlled object on the screen player's intentional control movements. In this way, the (e.g., a golf club). player may be rewarded for diminishing the physiological The system of FIG. 2 comprises a physiological signal signal difference by an improved ability to accurately posi- 65 measurement device 40 that receives physiological signal tion the player-controlled object on the screen. While dis- information from an operator of the system. One or more of cussed herein as being actuated by servomechanisms or a wide variety of different measured physiological signals US 9,623,324 B2 9 10 can be used in accordance with the present invention, for another embodiment, the effect of the accelerometers 52 on example, electroencephalogram (EEGs), skin temperature, the videogame or simulation may be perturbed by an amount skin conductance, heart rate, and/or event-related potentials proportional to the difference between the player's current (ERPs). U.S. Pat. No. 5,377,100, issued on Dec. 27, 1994 to momentary physiological signal value and a pre-selected Pope et al., and which is incorporated herein by reference in 5 target value. For example, the effect of the accelerometers 52 its entirety, at column 3, line 8 to column 5, line 60 details on the video game may be perturbed by an amount propor- a method for determining an individual's EEG index of tional to the difference between the player's current momen- attention, which index can be used to assess his or her mental tary physiological signal value and a pre-selected target by engagement in a task. Industry-standard methods and the processor 44 being programmed to adjust the potenti- devices for measuring these physiological signals may be io ometers 50 to adjust the voltage that powers the accelerom- used as the physiological signal measurement device 40, eters 52. The potentiometers 50 may be digital potentiom- including but not limited to an electroencephalograph, a eters configured as voltage dividers to vary, in a randomly or thermometer, a galvanic skin response device, an electro- periodically fluctuating manner, the amplitude of the fluc- myogram, and/or an cardiotachometer. The physiological tuation being made proportional to the difference between signal measurement device 40 in conjunction with the 15 the player's current momentary physiological signal value analog-to-digital converter 42 and processor 44 may (1) and the pre-selected target. In this manner, a change in the sample amplified and processed physiological signals (e.g., physiological signal in the desired direction may diminish electromyogram measuring muscular tension), (2) test the the amplitude of the fluctuations of the supply voltage to the signal levels with programmed conditional instructions accelerometers 52, and the resulting effect upon the game or stored in memory 45(and possibly compare the signal levels 20 simulation may be to create a wavering of the controller with predetermined desired signal levels stored in memory object that may be experienced as a loss of precision 45), and, in accordance with the physiological signal con- controllability as the physiological signal departs from the dition determinations, (3) dynamically adjust digitally con- criterion. trolled potentiometers 50 that are configured as voltage Referring now to FIG. 3, a block diagram of portions of dividers to modulate the voltage from power supply 48 that 25 a system for modifying the effect of an operator controlled powers the accelerometers 52 and/or gyroscopes 54. FIG. 2 input device on an interactive device is illustrated in accor- illustrates a single potentiometer that modulates the voltage dance with alternate embodiments of the present invention. to both a single accelerometer and a single gyroscope. In In the embodiment of FIG. 3, modification is made to one or alternative embodiments of the invention, separate potenti- more position reference signal elements that are used by the ometers may be used with each accelerometer and each 30 input device to determine a motion or position of the gyroscope, and multiple accelerometers and/or multiple operator controlled input device relative to the one or more gyroscopes may be used in the game input device 46. In a position reference signal elements. These position reference yet further alternative embodiment, one or more accelerom- signal elements are commonly referred to as "sensor bars." eters may be modulated with a single potentiometer and one The one or more sensor bars each comprise a plurality of or more gyroscopes may be modulated with a separate, 35 LEDs, two of which emit light at any particular time single potentiometer. One, a plurality of, or all of the (typically one from each sensor bar). (Note, the sensor bars accelerometers and/or gyroscopes may be so modulated. are described herein as separate physical devices, each In an embodiment, inputs to a video game or simulation having a plurality of LEDs. However, embodiments of the may be made by moving the entire controller itself including invention may comprise sensor bars that are combined in a accelerometers, for example by moving the entire Sony® 40 single physical device, with two sets of LEDs separated by Playstation® Move® controller itself. As described above a distance.) The operator controlled input device comprises with reference to FIG. 2, a controller may include a physi- a light detector for detecting light emitted from the LEDs. ological signal measurement device 40, such as an electro- In this embodiment, the standard sensor bar of the video myogram measuring muscular tension, which may sample game system is replaced with an array ofinfrared LEDs that the physiological signals. The physiological signal measure- 45 are individually controllable on and off. This embodiment of ment device 40 in conjunction with the analog-to-digital the invention turns LEDs in the array on and off in dynamic converter 42 and processor 44 may (1) sample amplified and patterns to produce a disturbance in the player-directed processed physiological signals (e.g., electromyogram mea- control of the infrared-sensing videogame controller, result- suring muscular tension), (2) test the signal levels with ing in a disruption in the stability of the player-controlled programmed conditional instructions stored in memory 45 50 object (e.g., a cursor) on the screen. The degree of the (and possibly compare the signal levels with predetermined disruption is programmed to be proportional to the differ- desired signal levels stored in memory 45), and, in accor- ence between the player's current momentary physiological dance with the physiological signal condition determina- signal value and a pre-selected target value. In this way, the tions,(3) dynamically adjust digitally controlled potentiom- player is rewarded for diminishing this difference by an eters 50 that are configured as voltage dividers to modulate 55 improved ability to accurately position the player-controlled the voltage from power supply 48 that powers the acceler- object (e.g., a cursor) on the screen. ometers 52 and/or gyroscopes 54. In this manner, unsteadi- In the embodiment of FIG. 3, modifying one or more ness in the controllability may be modulated by physiologi- position reference signal elements comprises changing cal signals, making game controllability a reflection of which two of the LEDs (typically one from each sensor bar) player state. For example, the effect ofthe accelerometers 52 6o are emitting light at any particular time. This typically on the videogame may be diminished by an amount propor- involves turning pairs of LEDs on and offin a predetermined tional to the difference between the player's current momen- pattern based on either (a) the difference between the mea- tary physiological signal value and a pre-selected target sured at least one physiological signal and the corresponding value, which may be accomplished by the processor 44 predetermined desired physiological signal level, or (b) the being programmed to adjust the potentiometer 50 to reduce 65 measured at least one physiological signal. the voltage that powers the accelerometers 52 to be dimin- The plurality of LEDs may comprise two groups of ished by an amount proportional to that difference. In linearly arranged LEDs, each of the two linearly arranged US 9,623,324 B2 11 12 groups being positioned substantially horizontally. This In the above two examples, the individual LEDs in each arrangement is illustrated in FIG. 4, which shows the two sensor bar are turned on and off in sequence (left to right or sensor bars 66, 68 spaced apart and on alternate sides of the right to left) to create a marquee-like sweep. This is illus- television 24. Also illustrated is the game console 10, and all trated in FIGS. 6A-D. FIGS. 6A-D illustrate two horizontal ofthe components are supported on a table 80. As mentioned 5 sensor bars 66, 68, each comprising seven linearly arranged above, the two sensor bars may be combined into a single LEDs (labeled Ll-L7 for the left sensor bar 66 and R1-R7 physical device. Alternatively, the plurality of LEDs may for the right sensor bar 68). First, LEDs Ll and Rl are comprise two groups of linearly arranged LEDs, each of the illuminated (FIG. 6A). Then, Ll and Rl are turned off and two linearly arranged groups being positioned substantially L2 and R2 are illuminated (FIG. 613). Then, L2 and R2 are l0 turned off and L3 and R3 are illuminated (FIG. 6C). Finally, vertically. This arrangement is illustrated in FIG. 5, which L3 and R3 are turned off and L4 and R4 are illuminated shows the two sensor bars 76, 78 spaced apart and on (FIG. 6D). This creates a left-to-right marquee sweep, which alternate sides of the television 24. Also illustrated is the causes the input device to interpret that it is being moved game console 10, and all of the components are supported on erratically when in fact it is not. The speed and/or extent of a table 80. As mentioned above, the two sensor bars may be 15 the marquee-like sweep may be varied, for example in combined into a single physical device. As another alterna- proportion to the difference between the player's current tive, the plurality of LEDs may comprise two groups of momentary physiological signal value and a pre-selected LEDs, each of the groups being arranged in two linearly target value. The specific pattern used may vary consider- arranged sub-groups, each sub-group of a group crossing the ably in alternative embodiments of the invention. other sub-group at a center of each sub-group and at a 20 In another example embodiment, the infrared LEDs that perpendicular angle. This embodiment is illustrated in FIG. the infrared camera uses to track the movement of the input 8 (discussed in more detail below). device are oscillated back and forth horizontally, causing an The system of FIG. 3 comprises a physiological signal oscillation to be superimposed on the player's movements of measurement device 40 that receives physiological signal the input device and reflected in the movements of the information from an operator of the system (as described 25 player-controlled object (e.g., a cursor) on the game screen. above). The physiological signal measurement device 40 in For example, the LEDs may be illuminated in a sequence conjunction with the analog-to-digital converter 42 and such as: L4/R4, L3/R3, L2/R2, L3/R3, L4/R4, L5/R5, processor 60 may (1)sample amplified and processed physi- L6/R6, L5/R5, and L4/R4. The amplitude and/or speed of ological signals (e.g., cardiotachometer measuring heart the oscillations may be programmed to be proportional to rate),(2) test the signal levels with programmed conditional 30 the difference between the player's current momentary instructions stored in memory 61 (and possibly compare the physiological signal value and a pre-selected target value. signal levels with predetermined desired signal levels stored This results in an unsteadiness in the player's ability to in memory 61), and, in accordance with the physiological control the player-controlled object (e.g., a cursor), which signal condition determinations, (3) turning on infrared diminishes as the difference between the player's current LEDs in dynamic patterns to produce the described effects 35 momentary physiological signal value and a pre-selected on the player-directed control of an infrared-sensing vid- target value diminishes. eogame controller. In another example embodiment, the infrared LEDs that One example implementation involves positioning one the infrared camera uses to track the movement of the input horizontal row of LEDs, e.g., 7 LEDs, in front and to the device are swept up and down vertically, causing an oscil- right of the player who is manipulating the input device, and 40 lation to be superimposed on the player's movements of the another row in front and to the left of the player (as input device and reflected in the movements of the player- illustrated in FIG. 4). The corresponding LED in each row controlled object (e.g., a cursor) on the game screen. The is simultaneously illuminated and turned on and off in synch amplitude and/or speed of the sweeps may be programmed to create a running marquee effect. This effect results in to be proportional to the difference between the player's horizontally sweeping the infrared camera's two reference 45 current momentary physiological signal value and a pre- points used to position the cursor on the screen, producing selected target value. This results in an unsteadiness in the a disruption in the stability of the player-controlled object player's ability to control the player-controlled object (e.g., (e.g., a cursor) on the screen. The length of the marquee run, a cursor), which diminishes as the difference between the and, consequently, the degree of the disruption, may be player's current momentary physiological signal value and a programmed to be proportional to the difference between the 50 preselected target value diminishes. One particular imple- player's current momentary physiological signal value and a mentation involves positioning one vertical column of pre-selected target value. In this way, the player is rewarded LEDs, e.g., 7 LEDs, in front and to the right of the player for diminishing this difference by an improved ability to who is manipulating the input device, and another row in accurately position the player-controlled object (e.g., a cur- front and to the left of the player (as illustrated in FIG. 5). sor) on the screen. 55 The corresponding LED in each column is simultaneously More specifically, one example implementation involves illuminated and turned on and off in synch to create a processor 60 continuously sampling an integrated EMG running marquee effect. This effect results in vertically signal, serially communicating those values to second pro- sweeping the infrared camera's two reference points used to cessor 63 which reads a serial value at the initiation of each position the player-controlled object (e.g., a cursor) on the sweep of LED rows and/or columns, and controlling the 60 screen, producing a disruption in the stability of the player- timing and sweep amplitude of a marquee-like illumination controlled object (e.g., a cursor) on the screen. of the LEDs. Another specific example implementation This is illustrated in FIGS. 7A-D. FIGS. 7A-D illustrate involves processor 60 sampling a heart rate signal (cardiota- two vertical sensor bars 66, 68, each comprising seven chometer), serially communicating those values to second linearly arranged LEDs (labeled Ll-L7 for the left sensor processor 63 which reads a serial value at the initiation of 65 bar 76 and R1-R7 for the right sensor bar 78). First, LEDs each heart beat, and controlling the timing and sweep Ll and Rl are illuminated (FIG. 7A). Then, Ll and Rl are amplitude of a marquee-like illumination of the LEDs. turned off and L2 and R2 are illuminated (FIG. 713). Then, US 9,623,324 B2 13 14 L2 and R2 are turned off and L3 and R3 are illuminated This illumination sequence of the LEDs creates a corre- (FIG. 7C). Finally, L3 and R3 are turned off and L4 and R4 sponding movement pattern of the player-controlled object are illuminated (FIG. 7D). This creates a bottom-to-top (e.g., a cursor) on the screen, producing a disruption in the marquee sweep, which causes the input device to interpret stability ofthe player-controlled object(e.g., a cursor) on the that it is being moved erratically when in fact it is not. The 5 screen. The radial position (i.e., how far away from the speed and/or extent of the marquee-like sweep may be center LED)of the illuminated LEDs and, consequently, the varied, for example in proportion to the difference between radius of the circumscribed rotational pattern and, conse- the player's current momentary physiological signal value quently, the degree of the disruption, may be programmed to and a pre-selected target value. In this way, the player is vary in proportion to the difference between the player's rewarded for diminishing this difference by improved ability io current momentary physiological signal value and a pre- to accurately position the player-controlled object (e.g., selected target value. In one embodiment, the radial position cursor) on the screen. The sweeps may be continuous up and of the illuminated LED, determined from the previously down oscillations whose amplitude is proportional to the defined physiological signal difference, is the same, i.e., the player's difference signal as above. Or each sweep may be same distance from the center on the right horizontal, the initiated on each heart beat and sweep up, or both up and 15 bottom vertical, the left horizontal and the top vertical down, to an amplitude proportional to the difference radials, for each rotation about the center, creating a sym- between the player's current momentary heart rate value and metrical diamond rotational pattern. In an alternate embodi- a pre-selected target value. If the goal of training is to ment, the radial position is determined from the physiologi- achieve a preselected target heart rate, represented by illu- cal signal difference for each individual radial in clockwise mination of the LEDs in the middle of each column, the 20 rotation, and may vary from radial to radial, creating a LEDs in each column may be programmed to sweep up to swirling rotational pattern. In another alternate embodiment, the appropriate LED from the LED in the middle of each the radial rotation is limited to three of the four radials, column, then down to the appropriate LED below the middle resulting in a triangular rotational pattern whose size and, of each column then return to the middle LED,the amplitude consequently, the degree of the disruption, may be pro- of each sweep being proportional to the difference between 25 grammed to vary in proportion to the difference between the the player's current momentary heart rate value and a player's current momentary physiological signal value and a pre-selected target value. For example, the LEDs may be pre-selected target value. The apex ofthis rotating triangular illuminated in a sequence: L4/R4, L3/R3, L2/R2, L3/R3, pattern, the farthest vertex from the horizontal side, may be L4/R4, L5/R5, L6/R6, L5/R5, and L4/R4. above or below the horizontal side, indicating that the The target heart rate may be determined in any of a 30 momentary physiological signal value exceeds or falls number of ways two examples are: (1) the target value below, respectively, a pre-selected target value. may be input prior to game play or (2) the target value may In an embodiment using the sensor bar 80 of FIG. 8, these be computed as a baseline average over N heart beats before functions may be implemented with three programmable the beginning of or during a selected period of game play. microcontrollers or processors. One processor may perform The target value could be programmed to change as the 35 three functions: (1) controlling analog to digital converter player progresses through the game. If the goal of training components to sample amplified and processed physiologi- is simply to lower heart rate, the pre-selected target value cal signals (e.g., from a cardiotachometer measuring heart would be, in essence, zero, and the LEDs in the column are rate); (2) testing the signal levels with programmed condi- programmed to sweep up from the bottom LED to an tional instructions, and,in accordance with the physiological amplitude proportional to the player's momentary heart rate, 40 signal condition determinations;(3) sending a radial position then back down to the bottom LED. If the goal of training value (0, 1, 2 or 3) to both of the two other processors. One is to increase heart rate variability, the difference between of the receiving processors may control the two co-linear the current momentary heart rate and the immediately pre- horizontal rows of seven LEDs each in the "H" configura- ceding heart rate, or an average heart rate over N preceding tion described above; the other receiving processor may heart beats, is calculated in real time and the amplitude ofthe 45 control the two vertical columns of seven LEDs each. The vertical LED sweeps are made inversely proportional to that two receiving processors engage in back and forth hand- difference. The sweeps may correspond to the direction of shaking to turn on corresponding LEDs (i.e., a pair of LEDs) the heart rate change, that is, heart rate increases produce on the radials of each of the two crosses described above in sweeps up and decreases produce sweeps down, or the clockwise sequence (right horizontal, bottom vertical, left sweeps may be opposite in direction to the heart rate change, 5o horizontal and top vertical radials for each rotation) so as to or the sweeps may be in either direction proportional to the create a dynamic rotational pattern. absolute value of the heart rate change. In another alternative embodiment, the sensor bars In an additional embodiment, illustrated in FIG. 8, the (which, again, may be a single device or separate devices) LED rows and columns are superimposed in a configuration may comprise two light emitting diodes (LEDs) or two sets 90 that resembles an "H", but with a crossbar comprising 55 of LEDs having a fixed distance therebetween. The LEDs two collinear 7-LED horizontal rows centered on the verti- may be mechanically translated from a first position to a cals of the "H". This arrangement creates two symmetrical second position. This may be accomplished by moving the crosses of LEDs with horizontal and vertical radials of 3 sensor bars themselves, or by mechanically moving the LEDs each, emanating from center LEDs. Corresponding LEDs within each sensor bar. The first alternative is illus- LEDs on the radials of each cross (i.e., a pair of LEDs) may 60 trated in FIG. 9, in which the sensor bars 86, 88 are mounted be turned on in clockwise sequence (right horizontal, bottom on a motion table 70 that may be configured to move the vertical, left horizontal and top vertical radials for each sensor bars, in unison or separately, horizontally, vertically, rotation) so as to create a dynamic rotational pattern that or diagonally. The difference between the first and second encircles the center (LC and RC) of each cross. Such a positions may be based on either (a) the difference between clockwise sequence of illumination might, for example, 65 the measured at least one physiological signal and the illuminate the following pairs of LEDs in sequence: LH7/ corresponding predetermined desired physiological signal R117, then LV1/RV1, then LH1/RH1, and then LV7/RV7. level, or (b) the measured at least one physiological signal. US 9,623,324 B2 15 16 The measurement and analysis of the physiological signal is 5. Physiological signals can be combined with the tactile similar to that of the embodiment of FIG. 3. More specifi- (haptic) and/or auditory signals of the input device, for cally, one example implementation involves processor 60 example, to produce a momentary displacement (bump) in continuously sampling a physiological signal, serially com- or slowing of the movement of the player's character on municating those values to second processor 63 which 5 screen. controls the mechanical movement of the sensor bars via the 6. Physiological signals can be used to modulate the effect motion table. of the tactile (haptic) and auditory signals delivered to the The function of the invention may be accomplished by player through the game input device and/or audio output of using the physiological signals of the player, the signals of the game console. another participant who is not handling a videogame con- l0 In at least one advantageous and exemplary embodiment, troller, or the physiological signals of one player to modulate including but not limited to the LED embodiments described the controller of an opposing player. above, the current invention interacts wirelessly with a Wii Embodiments of the invention may employ the integrated remote through infrared transmission, and has the unique EMG signal, and the heart rate (cardiotachometer) signal. 15 advantage of not requiring direct connection or wiring into Additional embodiments may employ brainwave signals, the Wii video game or access to the Wii system software or specifically an "engagement index" derived from brainwave firmware. signals, defined in Pope, Bogart and Bartolome (Pope, A. T., In other exemplary embodiments, the challenge of various Bogart, E. H., and Bartolome, D. S. (1995). Biocybernetic games can be enhanced by attenuation (joystick, button System Validates Index of Operator Engagement in Auto- 20 and/or motion-control dampening or perturbation) and/or by mated Task. Biological Psychology, 40, 187-195), the con- disruption (cursor movement modulation). Such outside tents of which are incorporated herein in its entirety. Other controls, focus on overt acts for systems like the Wii physiological signals that may be used include, but are not motion-control or the Microsoftfl Kinect(k motion detector, limited to, skin conductance signals, skin temperature sig- while working with inside controls and covert acts through nals and respiration signals. 25 body behaviors and such like NASA's MindShift (demon- A noise-reducing enhancement may be desired for obtain- strated in 2010 using a sensor attached to the player's ing electrical heart signal. The heart rate signal may be earlobe, checking the pulse and wired into the control or derived from a photoplethysmograph sensor or electrocar- sensors attached to the forehead, seeking the facial muscle diogram electrodes. One convenient method for obtaining strain that is a sign of stress). Accordingly, in certain the electrocardiogram heart signal is the use of chest band 30 examples, psycho-physiological modulation of gameplay electrodes in combination with a wireless transmitter (e.g., can be implemented in conjunction with the Wii system Polar Model T31 from Polar Electro USA). Such a technol- games, including but not limited to, first person shooter ogy minimizes movement artifact (electrical noise) and games where cross hairs position and point of view modi- enables the invention to be used conveniently with the active fications are used. Alternative games can include without and exercise games popular on the video game systems. 35 limitation, medical simulations using cursor position modi- In addition to the methods described above are the fol- fication to teach stress management during medical proce- lowing methods of superimposing movements on the play- dures; sports simulations such as for instance golf swing er's control of in-game objects (e.g., a cursor, character) on strength, modulation and ball control; and racing games the screen, producing a disruption in the stability of the where vehicle steering attenuation is contemplated. player-controlled object (e.g., a cursor, character) on the 40 As discussed above, in various embodiments, the player screen and/or modulation of game feedback signals that vary inputs to systems that use cameras that sense a player's in proportion to either (a) the difference between the mea- image in whole or in part or the image of an illuminated sured at least one physiological signal and the corresponding object on the player's control device (e.g., Leap Motion®, predetermined desired physiological signal level, or (b) the Sony® Playstation® Move® or Microsoft(k Kinect) may be measured at least one physiological signal: 45 modulated to modify the motion and/or position sensing 1. Physiological signals can be used to modulate a capability of the camera in response to (a) changes in the momentary illumination of off-center LEDs in the sensor measured at least one physiological signal and/or (b) differ- bars resulting in a brief change (bump) in the reference ences between the measured at least one physiological signal position for the position/motion of the input device. and a corresponding predetermined desired physiological 2. Physiological signals can be used to modulate the 50 signal level, thereby modifying the ability of the operator to magnitude of a superimposed momentary displacement control and interact with at least some of the depicted (bump) to control inputs in the input device (buttons, joy- images. In an embodiment, the camera's capture of the sticks, accelerometers and/or gyroscopes). These displace- image of the player or the image of an illuminated object of ments may be superimposed upon ongoing inputs a player the player's controller may be modified (e.g., distorted or provides to the game (movement in all directions, jump, 55 deflected) by a device placed adjacent to the camera, such as accelerate). The displacements may also function as addi- in front of, beside, and/or over the camera. In various tional inputs to the game without input from the player (e.g., embodiments, the device may include: one or more mirror firing a weapon). positioned by one or more servomechanism or one or more 3. Physiological signals can be used to simultaneously galvanometer; one or more lenses positioned by one or more modulate the magnitude of a superimposed momentary 60 servomechanism or one or more galvanometer; and/or one displacement (bump) of more than one control input. or more LED to illuminate the camera, and the servomecha- 4. Multiple physiological signals can be combined to nisms, galvanometers, and/or LEDs may be controlled by modulate the magnitude of a superimposed momentary one or more processor in response to physiological signal displacement (bump) of a single control input. Multiple condition determinations. The modification of the player's physiological signals can be used to simultaneously modu- 65 image or the image of an illuminated object on the player's late the magnitude of a superimposed momentary displace- control device may result in the control of the images on the ment (bump) of more than one control input. display of the game system or simulation system by the US 9,623,324 B2 17 18 player being improved or hindered based on the degree of object on the screen. In an embodiment, the driving of the modification caused to the image received by the camera. servomechanisms or galvanometers or the patterning of FIG. 10 illustrates an arrangement of an embodiment illuminated LEDs may be continuous shifts whose ampli- system in which a modification device 100 is positioned in tude may be proportional to the player's difference signal as front of the camera 21 of the game console 10. In this 5 described above. In another embodiment, each deflection, manner the modification device 100 may distort the image of distortion, or pattern may be initiated on each heart beat and the player or the image of the illuminated object of the deflect, distort or pattern in first one and then another player's controller received by the camera. In an embodi- direction to an amplitude proportional to the difference ment, the modification device 100 may utilize mirrors to between the player's current momentary heart rate value and distort or deflect the image of the player or the image of the io a pre-determined target value. If the goal of training is to illuminated object of the player's controller received by the achieve a pre-determined target heart rate, the servos or camera. The mirror or mirrors may be positioned in front of galvanometers and thus the mirrors or lenses and/or the or beside the camera and the movement of the mirror or LEDs may be programmed to deflect, distort or pattern to the mirrors may adjust the level of distortion of the image appropriate amplitude from a resting position or configura- captured by the camera. In an embodiment, the modification 15 tion, then to the appropriate amplitude in the opposing device 100 may utilize lenses to distort or deflect the image direction from the resting position or configuration, and then of the player or the image of the illuminated object of the return to the resting position or configuration. The amplitude player's controller received by the camera. The lens or of each deflection, distortion or pattern may be proportional lenses may be positioned in front of or beside the camera and to the difference between the player's current momentary the movement of the lens or lenses may adjust the level of 20 heart rate value and a pre-determined target value. distortion of the image captured by the camera. In an The target heart rate may be determined in any of a embodiment, the modification device 100 may utilize LEDs number of ways; two examples are: 1) the target value may to distort the image of the player or the image of the be input prior to game play or 2) it may be computed as a illuminated object of the player's controller received by the baseline average over N heart beats before the beginning of, camera. The LEDs may be positioned in the periphery 25 or during, a selected period of game play. If the goal of surrounding the camera lens or elsewhere in the visual field training is simply to lower heart rate, the pre-determined of the camera, and the relative brightness of the LEDs may target value may be, in essence, zero, and the servos, be dynamically varied. The differential brightness of the galvanometers, lenses, or LEDs may be programmed to LEDs may adjust the degree of modification of the received deflect, distort or pattern from a resting position or configu- image at the camera. 3o ration to an amplitude proportional to the player's momen- In various embodiments, the distortion of the image of the tary heart rate, then return to a resting position or configu- player or the image of the illuminated object of the player's ration. If the goal of training is to increase heart rate controller received by the camera may be implemented by a variability, the difference between the current momentary modification device 100 including programmable microcon- heart rate and the immediately preceding heart rate, or an trollers and analog to digital converter components to 1) 35 average heart rate over N preceding heart beats, is calculated sample amplified and processed physiological signals (e.g., in real time and the amplitude of the deflections, distortions electromyogram measuring muscular tension, an engage- or patterns may be made inversely proportional to that ment index derived from the electroencephalogram or brain- difference. The deflections, distortions or patterns may cor- waves, cardiotachometer measuring heart rate, etc.), then 2) respond to the direction ofthe heart rate change, that is, heart test the signal levels with programmed conditional instruc- 4o rate increases produce deflections, distortions or patterns in tions, and, in accordance with the physiological signal an up direction and decreases produce deflections, distor- condition determinations, 3) dynamically drive servomecha- tions or patterns down. Additionally, the deflections, distor- nisms or galvanometers or patterns of illuminated LEDs to tions or patterns may be opposite in direction to the heart produce the described effects on the player-directed control rate change, or the deflections, distortions or patterns may be of a videogame motion-sensing technology. 45 in either direction proportional to the absolute value of the One implementation involves a microcontroller of the heart rate change. device 100 continuously sampling the integrated EMG sig- FIGS. 11-13 illustrate block diagrams of portions of nal, serially communicating those values to a second micro- embodiment devices for modifying the effect of a camera on controller of the device 100 which reads a serial value and an interactive device to encourage the self-regulation of at dynamically drives servomechanisms or galvanometers or 50 least one physiological activity by the operator. The opera- illuminates patterns of illuminated LEDs to produce the tions of the devices illustrated in FIGS. 11-13 are similar to described effects on the player-directed control of a videog- the devices described above in FIGS. 3 and 9, except that ame motion-sensing technology. Another implementation rather than adjusting sensor bars 66, 68, 86, and 88 and/or involves a microcontroller of the device 100 sampling the a motion table 70, the processor 63 may cause lens(es) 106, heart rate signal, serially communicating those values to a 55 mirror(s) 104, or LED(s) 108 to be adjusted. As illustrated second microcontroller of the device 100 which reads a in FIG. 11, a mirror 104 may be connected to a servomecha- serial value at the initiation of each heart beat, and dynami- nism or galvanometer 102 connected to the processor 63. cally drives servomechanisms or galvanometers or illumi- The processor 63 may control the servomechanism or gal- nates patterns of illuminated LEDs to produce the described vanometer 102 to adjust the mirror 104 based at least in part effects on the player-directed control of a videogame 60 on the difference between the physiological signal measure- motion-sensing technology. ments and the desired physiological signal level as described In the various embodiments, the degree of the disruption above to modify the image received by the camera of the may be programmed to be proportional to the difference game or simulation system. As illustrated in FIG. 12, a lens between the player's current momentary physiological sig- 106 may be connected to a servomechanism or galvanom- nal value and a pre-determined target value. In this way, the 65 eter 102 connected to the processor 63. The processor 63 player may be rewarded for diminishing this difference by may control the servomechanism or galvanometer 102 to improved ability to accurately position the player-controlled adjust the lens 106 based at least in part on the difference US 9,623,324 B2 19 20 between the physiological signal measurements and the not denote any order, quantity, or importance, but rather are desired physiological signal level as described above to used to distinguish one element from another. The modifier modify the image received by the camera of the game or "about" used in connection with a quantity is inclusive ofthe simulation system. As illustrated in FIG. 13, one or more stated value and has the meaning dictated by the context LEDs 108 may be connected to the processor 63. The 5 (e.g., it includes the degree of error associated with mea- processor 63 may control the illumination intensity and/or surement of the particular quantity). illumination pattern of the LEDs 108 based at least in part All ranges disclosed herein are inclusive of the endpoints, on the difference between the physiological signal measure- and the endpoints are independently combinable with each ments and the desired physiological signal level as described other. Each range disclosed herein constitutes a disclosure of above to modify the image received by the camera of the io any point or sub-range lying within the disclosed range. game or simulation system. Accordingly, each separate value within a range is incorpo- FIG. 14 is a process flow diagram illustrating an embodi- rated into the specification as if it were individually recited ment method 1400 for modifying images received by a herein. Further, each separate value can serve as an endpoint camera. In an embodiment, the operations of method 1400 and is independently combinable with any other value as an may be performed by the processor of a device placed 15 endpoint to create a sub-range within a disclosed range. adjacent to (e.g., in front of) or in the visual field of the The use of any and all examples, or exemplary language camera of a game console or simulation system. In block (e.g.,"such as")provided herein, is intended merely to better 1402 the processor may measure a physiological signal. In illuminate the invention and does not pose a limitation on the block 1404 the processor may convert the analog measured scope of the invention unless otherwise claimed. Reference signal to a digital signal. In block 1406 the processor may 20 throughout the specification to "another embodiment", "an determine a difference between the measured physiological embodiment", "exemplary embodiments", and so forth, signal and a desired physiological signal level. In block 1408 means that a particular element (e.g., feature, structure, the processor may adjust one or more mirrors, lenses, and/or and/or characteristic) described in connection with the LEDs based at least in part on the determined difference to embodiment is included in at least one embodiment modify (e.g., distort or deflect) the image received by the 25 described herein, and can or cannot be present in other camera, thereby modifying the ability of the operator to embodiments. In addition, it is to be understood that the control and interact with at least some ofthe depicted images described elements can be combined in any suitable manner of the game console or simulation system. in the various embodiments and are not limited to the In addition to the methods described above are the fol- specific combination in which they are discussed. No lan- lowing methods of superimposing movements on the play- 30 guage in the specification should be construed as indicating er's control of in-game objects (e.g., a cursor, character) on any non-claimed element as essential to the practice of the the screen, producing a disruption in the stability of the invention. player-controlled object (e.g., a cursor, character) on the Preferred embodiments of this invention are described screen and/or modulation of game feedback signals that vary herein, including the best mode known to the inventors for in proportion to either (a) the difference between the mea- 35 carrying out the invention. Variations of those preferred sured at least one physiological signal and the corresponding embodiments may become apparent to those of ordinary predetermined desired physiological signal level, or (b) the skill in the art upon reading the foregoing description. The measured at least one physiological signal, including that inventors expect skilled artisans to employ such variations physiological signals can be used to modulate a momentary as appropriate, and the inventors intend for the invention to shift in the driving of servomechanisms or galvanometers or 4o be practiced otherwise than as specifically described herein. the patterning of illuminated LEDs resulting in a brief Accordingly, this invention includes all modifications and change (bump) in the reference position for the position/ equivalents of the subject matter recited in the claims motion of the input device. appended hereto as permitted by applicable law. Moreover, All references, including publications, patent applica- any combination of the above-described elements in all tions, and patents, cited herein are hereby incorporated by 45 possible variations thereof is encompassed by the invention reference to the same extent as if each reference were unless otherwise indicated herein or otherwise clearly con- individually and specifically indicated to be incorporated by tradicted by context. Thus, it is to be understood that reference and were set forth in its entirety herein. However, variations and modifications can be made on the aforemen- if a term in the present application contradicts or conflicts tioned structure without departing from the concepts of the with a term in the incorporated reference, the term from the 50 present invention, and further it is to be understood that such present application takes precedence over the conflicting concepts are intended to be covered by the following claims term from the incorporated reference. unless these claims by their language expressly state other- The use of the terms "a" and "an" and "the" and similar wise. referents in the context of describing the invention (espe- The invention claimed is: cially in the context of the following claims) are to be 55 1. A method for modifying the effect of an operator input construed to cover both the singular and the plural, unless on an interactive device to encourage the self-regulation of otherwise indicated herein or clearly contradicted by con- at least one physiological activity by the operator, the text. The terms "comprising," "having," "including," and interactive device comprising a display area that depicts "containing" are to be construed as open-ended terms (i.e., images and a device for receiving at least one light input meaning "including, but not limited to,") unless otherwise 60 from the operator corresponding to motion and/or position to noted. "Or" means "and/or." As used herein, the term thus permit the operator to control and interact with at least "and/or" includes any and all combinations of one or more some of the depicted images, the method for modifying of the associated listed items. As also used herein, the term comprising the steps of: "combinations thereof' includes combinations having at measuring at least one physiological activity of the opera- least one of the associated listed items, wherein the combi- 65 for to obtain at least one physiological signal having a nation can further include additional, like non-listed items. value indicative of the level of the at least one physi- Further, the terms "first," "second," and the like herein do ological activity; and US 9,623,324 B2 21 22 modifying the light input before it is received by the operator to control and interact with at least some ofthe device to thereby modify the motion and/or position depicted images, wherein the device for receiving at sensing capability of the device for receiving at least least one input from the operator to thus permit the one light input from the operator in response to (a) operator to control and interact with at least some ofthe changes in the measured at least one physiological 5 depicted images comprises a camera; and signal and/or (b) differences between the measured at modifying the motion and/or position sensing capability least one physiological signal and a corresponding of the camera for receiving at least one input from the predetermined desired physiological signal level, operator comprises the step of modifying a mirror thereby modifying the ability of the operator to control position to modify, at least in part, the image of the and interact with at least some of the depicted images. 10 operator or an illuminated controller of the operator 2. The method of claim 1, wherein the device for receiv- received by the camera. ing at least one light input from the operator to thus permit 10. An apparatus for modifying the effect of an operator the operator to control and interact with at least some of the depicted images comprises a camera. input on an interactive device to encourage the self-regula- 3. The method of claim 2, wherein modifying the light 15 tion of at least one physiological activity by the operator, the input before it is received by the device to thereby modify interactive device comprising a display area that depicts the motion and/or position sensing capability of the camera images and a device for receiving at least one light input for receiving at least one light input from the operator from the operator to thus permit the operator to control and comprises the step of adjusting a lens to modify, at least in interact with at least some of the depicted images, the part, the image of the operator or an illuminated controller Zo apparatus comprising: of the operator received by the camera. at least one measurement device for measuring at least 4. The method of claim 2, wherein modifying the light one physiological activity of the operator to obtain at input before it is received by the device to thereby modify least one physiological signal having a value indicative the motion and/or position sensing capability of the camera of the level of the at least one physiological activity; for receiving at least one light input from the operator 25 and comprises the step of modifying illumination of an LED to at least one modifying device for modifying the light modify, at least in part, the image of the operator or an input before it is received by the device to thereby illuminated controller of the operator received by the cam- modify the motion and/or position sensing capability of era. the device for receiving at least one light input from the 5. The method of claim 1, wherein the at least one 30 operator in response to (a) changes in the measured at physiological signal comprises at least one or more of: skin least one physiological signal and/or (b) differences temperature, skin conductance, electrical activity of between the measured at least one physiological signal muscles, blood flow, heart rate, heart rate variability, and and a corresponding predetermined desired physiologi- respiratory rate. 6. The method of claim 1, wherein the at least one 35 cal signal level, thereby modifying the ability of the physiological signal comprises at least one or more of: operator to control and interact with at least some ofthe brainwave electrical event-related potentials, and at least depicted images. one brainwave frequency band; wherein the at least one 11. The apparatus of claim 10, wherein the device for brainwave frequency band comprises at least one or more of: receiving at least one light input from the operator to thus (a) theta, (b) alpha, (c) Sensorimotor Response (SMR),(d) 40 permit the operator to control and interact with at least some beta, and (e) gamma. of the depicted images comprises a camera. 7. The method of claim 1, further comprising: 12. The apparatus of claim 11, wherein the at least one displaying the at least one physiological signal having a modifying device for modifying the light input before it is value indicative of the level of the at least one physi- received by the device to thereby modify the motion and/or ological activity. 45 position sensing capability of the camera for receiving at 8. The method of claim 1, wherein the steps of measuring least one light input from the operator comprises a mirror and modifying are performed wirelessly without any direct positioned to modify, at least in part, the image of the connection to a camera or to the interactive device. operator or an illuminated controller ofthe operator received 9. A method for modifying the effect of an operator input by the camera. on an interactive device to encourage the self-regulation of 50 13. The apparatus of claim 11, wherein the at least one at least one physiological activity by the operator, the modifying device for modifying the light input before it is interactive device comprising a display area that depicts received by the device to thereby modify the motion and/or images and a device for receiving at least one input from the position sensing capability of the camera for receiving at operator to thus permit the operator to control and interact least one light input from the operator comprises a lens to with at least some of the depicted images, the method for 55 modify, at least in part, the image of the operator or an modifying comprising the steps of: illuminated controller of the operator received by the cam- measuring at least one physiological activity of the opera- era. tor to obtain at least one physiological signal having a 14. The apparatus of claim 11, wherein the at least one value indicative of the level of the at least one physi- modifying device for modifying the light input before it is ological activity; 6o received by the device to thereby modify the motion and/or modifying a motion and/or position sensing capability of position sensing capability of the camera for receiving at the device for receiving at least one input from the least one light input from the operator comprises an LED to operator in response to (a) changes in the measured at modify, at least in part, the image of the operator or an least one physiological signal and/or (b) differences illuminated controller of the operator received by the cam- between the measured at least one physiological signal 65 era. and a corresponding predetermined desired physiologi- 15. The apparatus of claim 10, wherein the at least one cal signal level, thereby modifying the ability of the physiological signal comprises at least one or more of; skin US 9,623,324 B2 23 24 temperature, skin conductance, electrical activity of muscles, blood flow, heart rate, heart rate variability, and respiratory rate. 16. The apparatus of claim 10, wherein the at least one physiological signal comprises at least one or more of 5 brainwave electrical event-related potentials, and at least one brainwave frequency band; wherein the at least one brainwave frequency band comprises at least one or more of: (a) theta, (b) alpha, (c) Sensorimotor Response (SMR),(d) beta, and (e) gamma. io 17. The apparatus of claim 10, further comprising: wherein the display area is configured to display the at least one physiological signal having a value indicative of the level of the at least one physiological activity. 18. The apparatus of claim 10, wherein the apparatus is 15 configured to wirelessly perform the steps of measuring and modifying without any direct connection to a camera or to the interactive device.