Drawing Assist System for Cerebral Palsy Patients Considering the Involuntary Movements
Tomoyuki Nakao, Hirokazu Matsui, Ken’ichi Yano, Naruto Miyagawa, Naoki Kubota and Satoshi Horihata
Abstract— Creative activities, such as painting and playing using the moving average method for nonperiodic involun- instruments, are sources of satisfaction and fulfillment for peo- tary movements, such as those seen in athetosis. Morimoto ple with disabilities. However, some individuals with disabilities developed a painting tool based on the moving average cannot satisfactorily enjoy such activities because of involuntary movements. In this study, we developed a drawing assist system method that attenuates the effects of involuntary movements for users with athetoid cerebral palsy and muscle tension, who in real time, and proposed the averaging switching and com- experience involuntary movement. To date, assist systems have pulsory compensation methods as solutions for attenuating had difficulty in distinguishing involuntary movement from the effects of sudden shaking[2]. Shiraishi proposed a line- voluntary movement, and thus we designed a position correction smoothing method that applies the additional moving average filter to attenuate involuntary movement on the basis of the characteristics of the magnitude of velocity with respect to method to a distance between an input termination point involuntary movement. Our system enabled drawing based on and a termination point corrected by another moving average the user’s own senses and motor control, even when involuntary method[3]. These methods are favorable for users with partial movement occurred． paralysis, who can move and click a mouse on a table. For users with severe spasticity that makes it difficult to keep the I. INTRODUCTION input device on a plane, the irregularity of the involuntary Many people with physical disabilities enjoy creative movement and its variable strength should be considered. activities as a means of self-express Our research group has proposed a method that changes ion and motivation in life. Painting is one such activity that attenuation based on the strength of involuntary movements the physically disabled can perform using residual functions. by using the magnitude of velocity from a three-dimensional For instance, one can paint by using hand dexterity if arm input device[4]. However, a method based on the moving function is limited, or by using the mouth or lower limb average method needs to be designed with a long moving if the function of both upper limbs is limited. In recent average period enough to determine the attenuation strength, years, the development of assistive equipment and interactive which worsens operability and limits what the user may software has brought computer-based painting to people with draw. Against this background, a drawing assist system that disabilities. Whether the act of painting can be satisfactorily attenuates nonperiodic involuntary movements and exhibits enjoyed depends on the degree of the disability. Insofar superior operability is desired. as involuntary movements affect body control, precise fine- The purpose of this study is to analyze the involuntary motor movements become difficult to perform, limiting the movements of a user with athetoid cerebral palsy during ability to perform desired tasks such as painting. drawing manipulation, to design a position correction filter To address these problems for people with involuntary for attenuating the effects of involuntary movements in real movements who draw using computer graphics, studies time, and to develop a drawing assist system that enables have been conducted on systems that filter the effects of free-form drawing, even in the presence of involuntary involuntary movements from the input signal of a device movements. so that the on-screen pointer does not show unintended movement. IBM developed the Assistive Mouse Adapter II. DRAWING ASSIST SYSTEM FOR PATIENTS WITH by using a simple design where the cut-off frequency of CEREBRAL PALSY the linear low-pass filter is tuned with a dial [1]. Some A. Cerebral Palsy methods that cut off a frequency or frequency band are Cerebral palsy is a disease characterized by involuntary favorable for periodic involuntary movements that have a movements; the condition results from damage to the motor characteristic frequency, such as the tremor in Parkinson ̉s control centers of the brain before, during, or shortly after disease. In other work, many studies have been conducted birth [5]. In individuals with cerebral palsy a common This study was supported in part by city area grants from the southern symptom is athetosis, which is characterized by involuntary Gifu innovation cluster program. movement and low muscle tone. Muscle tone fluctuates with T. Nakao, H. Matsui and K. Yano are with the Department of Mechanical voluntary movement and mental distress, which makes it Engineering㧘Mie University 1577 Kurimamachiya-cho, Tsu City, Mie 514- 8507, Japan [email protected] difficult for the patient to achieve continuous postural main- N. Miyagawa and N. Kubota are with the Research and Development tenance. These excessive movements do not stop occurring Division, Gifu Prefecture, Research Institute for Human Life Technology when voluntary movements are attempted[6]. Nakai reported 1154, Yamada, Takayama 506-0058, Japan S. Horihata is with the School of Dentistry, Nihon University, 2-870-1 on the results of analyzing power grip movement, and exten- Matsudo, Chiba 501-1193, Japan sion and flexion of the wrist, in patients with cerebral palsy;
978-1-4577-2137-3/11/$26.00 © 2011 IEEE it was found that motor control in patients with cerebral palsy corner. Positive directions correspond for xP and xD㧘and is less developed compared with unaffected individuals, and for yP and yD. that the motor skills of patients with athetosis are less devel- The involuntary movements in athetosis make it difficult oped than those of patients with spasticity in terms of speed, to manipulate any buttons and objects with both upper timing, joint angle, and strength. Strong hypertonic behavior limbs. Thus drawing judgments must be based on only in extensor muscle groups, imbalance between flexor and stylus manipulation. Another goal of this system is intuitive extensor muscle groups, instantaneous movements, surges of operability such as with an actual brush and canvas. An power, and difficulty of power control are all characteristics upswing and a downswing manipulation with the stylus of athetosis[7]. In particular, the involuntary movements of are used for drawing judgement criteria. The drawing start individuals with athetoid cerebral palsy and muscle tension criterion is designed, as shown in Equation (1), such that occur suddenly and are considerably heightened by exertion. drawing starts when the difference between the current z- These involuntary movements manifest themselves when a coordinate and the z-coordinate from 0.50 [s] before exceeds person with the condition attempts to maintain posture or 0.05 [m]. The drawing end condition is designed, as shown in sustain a certain movement; in addition, the muscle tone of (2), such that the drawing ends when the difference between people with athetoid cerebral palsy increases during postural the current z-coordinate and the z-coordinate from 0.19 [s] maintenance and decreases while at rest [8]㧚There are over before exceeds 0.03 [m]. After receiving informed consent 60,000 people in Japan with cerebral palsy, characterized by from the participant, a video camera was set up at the angle generalized impaired motor function [9][10]. of Fig. 1 to check the correspondence between the partici- pant’s actual movements and the measured movements. The B. Experimental setup raster graphics editor used in this study was ArtRage version The experimental setup of this study is shown in Fig. 2.5 (e frontier, Ltd.)[12]. The color select and undo functions 1㧚The input device is located in front of the user and were handled with a palette and a button on the window that inclined at an angle of 35 [deg] to fit the user’s posture. worked with the input device. The display is located in front of the user in an easy-to- Δz = z (t) − z (t − 0.5) ≤−0.05 [ ] see position. When considering an input device for users P P P m (1) who experience severe involuntary movements, a mouse and graphics tablet are unsuitable because they require planar ΔzP = zP (t) − zP (t − 0.19) ≥ 0.03 [m] (2) postural maintenance; instead, a device that can move in III. ANALYSIS OF INVOLUNTARY MOVEMENTS DURING three dimensions and measure three-dimensional coordinates DRAWING is required. An additional concern is that the users can sometimes involuntarily loses their grip on an object; for We conducted experiments to investigate how involuntary example, if a user was to release and drop a graphics tablet, movements influence movements during drawing. A partic- it would be almost impossible for it to be retrieved without ipant traced a design, in this case an ogee (a shape similar assistance. From these considerations, a compact PHANToM to an “S”). The ogee design allows us to evaluate whether a Omni (SensAble, Inc.) haptic interface [11] is used as an participant has the required manipulation capabilities to rep- input device. The point of rotation at the center of the gimbal resent a shallow curve whose direction continuously changes is used as the input point P for drawing. The input coordinate and to confine their drawing to the designated boundaries. system ΣP (xP ,yP ,zP ) is based on the input device and The participant was instructed to begin drawing in the right defined so that right and left are along the x-axis, front and square part of the ogee and sustain the drawing motion back are along the y-axis, and up and down are along the through the curved section before finally ending the drawing z-axis. Fig. 1 shows the input point P of the input device; in the left circular part. the arrows indicate the positive direction of each axis. The Fig. 2 shows the experimental results. All three examples plane used for drawing in ΣP is defined as xP yP , the angle in Fig. 2 show that it is difficult for the participant to perform of which is sloped by the orientation of the input device. The the curve tracing and to start and end the drawing at the origin of the two-dimensional drawing coordinate system designated positions. ΣD(xD,yD) on the display is defined in the lower left From these results, involuntary movements cause the participant to lose control of voluntary movements; conse- quently, the participant encounters difficulty in performing y Input device D
-3 -3 -3 Display x10 x10 x10 xD 50 50 50 ΣD 40 40 40 30 30 30 z 20 20 20 P 10 10 10 0 0 0 -10 -10 -10 y -20 -20 -20
y position (m) -30 -30
-30 y position (m)
P y position (m) P -40 -40 -40 -50 -50 -50 -80 -60 -40 -20 0 20 40 60 80 -3 -80 -60 -40 -20 0 20 40 60 80 -3 -80 -60 -40 -20 0 20 40 60 80-3 x10 x10 x10 User xP x position (m) x position (m) x position (m) (a) (b) (c) Fig. 1. Experimental setup for drawing assist system Fig. 2. Tracing results of raw data on x-y plane the manipulations required to trace a shape whose direction 0.8 0.8 0.7 0.7 continuously changes as in typical graphical elements. 0.6 0.6 The severity of athetosis varies, but individuals who expe- 0.5 0.5 0.4 0.4 rience strong involuntary movements almost completely lose 0.3 0.3 Velocity (m/s) Velocity (m/s) complete control of voluntary movements. Fig. 3 shows a 0.2 0.2 trajectory on the xy-plane generated by a participant with 0.1 0.1 0 0 strong involuntary movements during a 2.5 [s] span in the 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 1.8 Time (s) Time (s) ogee tracing experiment mentioned above. The participant (a) (b) has generated a form that is wider than the intended shape 0.8 0.8 0.7 0.7 and exhibits irregular direction, distance, and frequent veer. 0.6 0.6 Considering the characteristics of the involuntary movements 0.5 0.5 mentioned in Section 2, each velocity component contributes 0.4 0.4 0.3 0.3 Velocity (m/s) to (3), the magnitude of velocity, which is used as an Velocity (m/s) 0.2 0.2 indicator of the strength of involuntary movements[4]. 0.1 0.1 0 0 0 0.2 0.4 0.6 0.8 1 1.2 1.4 1.6 0 0.5 1 1.5 2 2.5 2 2 2 Vs = v + v + v (3) Time (s) Time (s) px py pz (c) (d) The change of the velocity between weak and strong in- Fig. 4. Change of the magnitude of velocity voluntary movements is shown in Fig. 4. A histogram of the 1 1 velocity is shown in Fig. 5. Examples (a) through (c) in Figs. 0.9 0.9 4 and 5 correspond to cases (a) through (c) in Fig. 2 showing 0.8 0.8 0.7 0.7 drawing attempts that include weak involuntary movements. 0.6 0.6 Example (d) in Figs. 4 and 5 corresponds to the data on the 0.5 0.5 Frequency 0.4 Frequency 0.4 strong involuntary movements shown in Fig. 3. 0.3 0.3 0.2 0.2 From the distribution results of Fig. 5, we can see from 0.1 0.1 0 0 the data above that weak and strong involuntary movements 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Velocity (m/s) Velocity (m/s) differ greatly in terms of the velocity frequencies; strong in- (a) (b) voluntary movements most frequently occur at the velocities 1 1 0.9 0.9 greater than 0.20 [m/s]. 0.8 0.8 0.7 0.7 0.6 0.6 IV. ADAPTIVE INVOLUNTARY BEHAVIOR ATTENUATION 0.5 0.5 0.4 0.4
ILTER Frequency
F Frequency 0.3 0.3 0.2 0.2 From the involuntary movement analysis results above, 0.1 0.1 0 0 a position control filter is required to prevent the display 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 pointer from overrunning the target when a user experiences Velocity (m/s) Velocity (m/s) (c) (d) involuntary movement by attenuating the effects of involun- tary movements and adapting to the changes in velocity. Ad- Fig. 5. Histogram of the magnitude of velocity with respect to involuntary movements ditionally, the attenuation characteristics of the filter should be based on the involuntary movements and manipulation characteristics of individuals, not the subjective views of a the adaptive involuntary behavior attenuation filter (AIBAF) researcher or simple trial-and-error methods. We designed to satisfy these requirements. Fig. 6 shows the AIBAF block diagram. The coordinates of the corrected point at sample i,
-3 G(i)=(xG(i),yG(i),zG(i)), as shown in (4), is updated x10 100 by using a coordinate of the corrected point before a sample G(i−1), the current measurement of the coordinate of input 80 point P (i), and an attenuation weight coefficient w(i). The 60 range of w(i) is 0
40 VD(i)=2|vpz(i)|−|vpx(i)|−|vpy(i)| (6) 20 0
-20
-40 VD (i) -60 vβ y position (m) β(i)=Dβ (7) -80 -100 -140-120-100-80 -60 -40 -20 0 20 40 60 80 100 120 140 -3 x position (m) x10 V. D RAWING ASSIST EXPERIMENT (c) (d) We carried out a drawing assist experiment with AIBAF. Fig. 7. AIBAF drawing results and comparison of trajectories on the xy- The drawing assist experiment task was to trace an ogee, plane 1.0 involuntary movements, is shown on the xy-plane in Fig. ¦vx¦ 0.8 ¦vy¦ 11; the change in position of each axis is shown in Fig. ¦vz¦ 12, and the change in the velocity Vs(i), drawing judgment 0.6 V D velocity VD(i), and the attenuation weight coefficient w(i) 0.4 Draw start are shown in Fig. 13. A 9 [s] span of strong involuntary Velocity (m/s) 0.2 movements is marked with a colored dotted line. We can see that a wide range of hand movements occur with irregular 0 direction, distance, and frequent veering. This means that Time (s) the participant cannot trace the tulip example with voluntary x 10-3 movements or stop involuntary movements, causing them to w 3 lose drawing control. As shown in Fig. 13, the velocity Vs(i) becomes higher than the transition velocity vth=0.20 [m/s] 2 and the attenuation weight coefficient w(i) notably decreases during strong involuntary movements, while Vs(i) remains Weight 1 less than 0.20 [m/s] and w(i) does not approach zero after the strong involuntary movements have finished. It is confirmed 0 β(i) that does not attenuate voluntary movements, because 0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 the change in drawing judgment velocity is different from Time (s) the change in the velocity and w(i) becomes close to near Fig. 8. Change of velocities and attenuation weight at draw start zero when drawing judgment velocity values are low. The AIBAF design includes the tuning of attenuation 1.0 ¦vx¦ mentioned above by changing involuntary movements and 0.8 ¦vy¦ drawing manipulations in real time to achieve drawing as- ¦vz¦ sistance in expressions that require the participant to perform 0.6 V D continuous veering, even when involuntary movements occur. Draw end 0.4 Additionally, the AIBAF design has properties to assist using Velocity (m/s) 0.2 window functions, such as color select and undo, even when involuntary movements occur. 0 Finally we discuss the AIBAF application range. (4) and Time (s) (5) are effective in attenuating all types of nonperiodic invol- x 10-3 untary movements losing control of the amount of exercise, w 3 not only athetosis. The upper limit αul, the transition velocity vth, and the transition attenuation Dth are able to be tuned 2 to the degree of involuntary movements of an individual.
Weight 1
0
0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5 Time (s) Fig. 9. Change of velocities and attenuation weight at draw end photographs to illustrations. The raw drawing data are shown in Fig. 10 (b). The participant was able to perform relevant manipulations, but lost enthusiasm to achieve the task when (a) Tracing design (b) Result with raw data about 38 [s] had passed. The AIBAF-corrected drawing is shown in Fig. 10 (c). We see that the participant was able to draw a picture similar to the example. The time required to finish the task was about 35 [min] with no interruptions. The color change and undo functions were controlled by the participant using functions on the paint tool without having to change devices. We will now discuss the correcting effects of the drawing assist system against strong involuntary movements that (c) Result with AIBAF occurred during the tulip tracing example. A participant ̉ s drawing trajectory, including an 11 [s] span of strong Fig. 10. Design and results of tulip tracing with raw and AIBAF -3 2.0 x10 V 100 s VD 80 1.5
60 1.0 40
20 Velocity (m/s) 0.5 v 0 th=0.2 0 -20 01234567891011 Time (s) -40 -3
y position [m] x 10 -60 w -80 3
-100 -140-120-100-80 -60 -40 -20 0 20 40 60 80 100 120 140 -3 2 x position [m] x10
Weight 1 Fig. 11. Behavior of strong involuntary movement on x-y plane during tulip tracing 0
x 10-3 01234567891011 200 Time (s) 150 100 Fig. 13. Velocities and attenuation weight change of strong involuntary 50 movement during tulip tracing 0 x position [m] -50 01234567891011 -3 Time [s] that require continuous veering, even when involuntary x 10 100 movements caused loss of drawing control. 50 REFERENCES 0 -50 [1] J. L. Levine and M. A. Schappert, “A mouse adapter for people with hand tremor IBM SYSTEMS JOURNAL,” pp623-624, 2005
y position [m] 㧘 -100 01234567891011 [2] D. Morimoto, M. Nawate, T. Watanabe, A. Abe, S. Fukuma and -3 Time [s] S. Honda, “A Painting Tool with Blurring Compensation for People x 10 Having Involuntary Hand Motion,” Technical Report of The Institute 200 of Erectronics, Information and Communication Engineers, pp59-64, 150 2004 100 [3] M. Shiraishi, H. Kawase and M. Shinya㧘“A Compensation Method 50 for Hand-drawn Illustration by Moving Average,” The Institute of 0 Image Information and Television Engineers Technical Report, Vol.34, z position [m] -50 01234567891011 No.18㧘pp23-26, 2010 Time [s] [4] H. Aoyama, T. Nakao, N. Miyagawa㧘N. Kubota, S. Horihata, and K. Yano, “Development of Drawing Assist System for Patients with Fig. 12. Position change of strong involuntary movement during tulip Cerebral Palsy of the Tension Athetosis Type,” Proceedings of IEEE tracing ICRA2011 International Conference on Robotics and Automation, pp4664-4669, 2011 [5] R. Kaji㧘Diacrisis and Treatment of Involuntary Monement, pp72, β(i) 2006, Tokyo: SHINDAN-TO-CGIRYOSHA CO. (1) and (2), which are drawing judgment criteria, and , [6] S. Gomi, Rehabilitation Medicine Seminar (in Japanese), pp32㧘2005, which is an attenuation coefficient for drawing judgment Ishiyaku Publishers㧘Inc. movements, are effective not only for patients with cerebral [7] S. Nakai, “Power Grip Movement and Extension-Flexion Movement of Wrist in Cerebral Palsied Children,” Japanese Journal of Special palsy or other physical disabilities, but also for unaffected Education, pp7-15, 1984, The Japanese Association of Special Edu- individuals. Parameters of these equations are able to be cation tuned to the manipulation characteristics of individuals. The [8] N. Chino and N. Ando, Rehabilitation of Neurogenic Disorders, pp21, 2005, Tokyo: KANEHARA and CO. drawing assist system with AIBAF is effective in assisting [9] Ministry of Health, Labour and Welfare㧘Result of Survey on Children drawing on a computer for individuals with nonperiodic with Special Needs and Persons with Disabilities 2006 (in Japanese), involuntary movements that cause them to lose drawing Ministry of Health, Labour and Welfare㧘pp4, 2008 [10] Cabinet Office㧘Annual Report on Government Measures for Persons control. with Disabilities 2007, Cabinet Office㧘(2007), pp186 [11] PHANTOM Omni Haptic Device, available from VI. CONCLUSION http://www.sensable.com/haptic-phantom-omni.htm, (accessed The AIBAF drawing assist system changes its attenuation 2011-1-20) [12] Art Rage㧘available from http://graphic.e-frontier.co.jp/artrage/, (ac- to suit the involuntary movements and drawing movements cessed 2011-9-13) of a user in real time. It is developed for users with athetoid cerebral palsy and muscle tension in order to analyze and correct involuntary movements during drawing. In drawing assist experiments using AIBAF the user could draw curves