UNIVERSITY OF CALIFORNIA, IRVINE
Development of Wearable Gesture Based Controllers using Intelligent and Adaptive Materials
DISSERTATION
submitted in partial satisfaction of the requirements for the degree of
DOCTOR OF PHILOSOPHY
in Mechanical Engineering
by
Kyle Robert Van Volkinburg
Dissertation Committee: Professor Gregory Washington, Chair Professor Faryar Jabbari Associate Professor Michelle Khine
2016
Portion of Chapters 1 and 3 © 2014 SPIE Portion of Chapter 4 and 5 © 2015 ASME Chapters 7 and 8 © 2016 SPIE All other materials © 2016 Kyle Van Volkinburg
DEDICATION
To Mom and Dad
Thank you for your love, support, and encouragement
ii
TABLE OF CONTENTS
Page
LIST OF SYMBOLS ...... ix
LIST OF FIGURES ...... x
LIST OF TABLES ...... xvi
ACKNOWLEDGMENTS ...... xvii
CURRICULUM VITAE ...... xviii
ABSTRACT OF THE DISSERTATION ...... xx
INTRODUCTION ...... 1
References ...... 6
CHAPTER 1: Physiology and Overview of the Wearable Gesture Based Controller using Polyvinylidene Fluoride ...... 9
1.1 Physiology ...... 9
1.2 Gesture Sets ...... 11 1.2.1 Gesture Set 1 ...... 11 1.2.2 Gesture Set 2 ...... 12 1.2.3 Gesture Muscle Matrix ...... 13
1.3 Modeling ...... 14
References ...... 14
CHAPTER 2: Piezoelectrics and PVDF ...... 15
2.1 Background and Convention ...... 15
2.2 Domains and State Variables ...... 17
iii
2.3 Linear Constitutive Equations from Thermodynamics ...... 18
2.4 Energy Functions ...... 22
References ...... 23
CHAPTER 3: Sensor Dynamics and System Response ...... 25
3.1 The Extended Hamilton’s Principle...... 26
3.2 Self‐Adjoint Systems and the Assumed Modes Method ...... 27
3.3 Displacements ...... 28
3.4 Formulation of the Lagrangian ...... 29 3.4.1 Kinetic Energy ...... 30 3.4.2 Potential Energy ...... 32
3.5 Variations ...... 34 3.5.1 Variation of the Lagrangian ...... 34 3.5.2 Variation of Work ...... 35
3.6 Evaluating the Extended Hamilton’s Principle ...... 36
3.7 Strain Vector Differential Operator ...... 38 3.7.1 Transverse Motion ...... 38 3.7.2 Axial Motion ...... 40
3.8 PVDF Electrode Spatial Shading ...... 41 3.8.1 Transverse Displacements ...... 42 3.8.2 Axial Displacements ...... 44
3.9 Force Modeling ...... 45 3.9.1 Transverse Inputs ...... 45 3.9.2 Axial Inputs ...... 47
3.10 Simulation Results ...... 47
References ...... 50
iv
CHAPTER 4: Artificial Neural Networks ...... 51
4.1 Network Architecture ...... 52
4.2 Feedforward Operation ...... 54
4.3 Training ...... 57 4.3.1 Error Derivatives ...... 58 4.3.2 Learning Law ...... 62
4.4 Operation ...... 63
References ...... 65
CHAPTER 5: Neural Network Implementation Results ...... 66
5.1 dSPACE ds1103 ...... 66 5.1.1 dSPACE Setup ...... 67 5.1.2 ds1103 Results ...... 68
5.2 Microcontroller Based Systems ...... 72 5.2.1 Arduino Setup ...... 72 5.2.2 Arduino Results ...... 73 5.2.3 ATMega328p PCB Setup ...... 75 5.2.4 ATMega328p PCB Results ...... 76
References ...... 78
CHAPTER 6: Sleeve Properties and Controller Design, Fabrication, and Implementation .. 79
6.1 Compression Sleeve Mechanical Properties ...... 79
6.2 PVDF Electrode Shading ...... 82
6.3 Sleeve Assembly ...... 85
6.4 Circuitry and Hardware ...... 86 6.4.1 Arduino Pro Mini System ...... 86 6.4.2 ATMega328p PCB System ...... 88
v
6.5 Data Collection and Verification ...... 89
6.6 Software Applications ...... 90 6.6.1 Computer gesture indicating application ...... 90 6.6.2 Android oscilloscope and gesture indicating application ...... 91 6.6.3 PowerPoint Add In ...... 93
References ...... 94
CHAPTER 7: Fabrication of a self‐sensing electroactive polymer bimorph actuator based on polyvinylidene fluoride and its electrostrictive terpolymer ...... 95
7.1 Device Fabrication ...... 97
7.2 Self‐Sensing Bimorph Model ...... 99 7.2.1 Actuation based governing equation of motion ...... 99 7.2.2 Derivation of tip deflection from voltage across PVDF ...... 102
7.3 Experimental Configuration and Procedures ...... 105 7.3.1 Configuration ...... 106 7.3.2 Procedures ...... 107
7.4 Results and Discussion ...... 108 7.4.1 Actuation and electromechanical characterization ...... 108 7.4.2 Integrated sensing of cantilever tip deflections ...... 110
7.5 Conclusions ...... 113
References ...... 114
CHAPTER 8: Use of the shape memory polymer polystyrene in the creation of thin film stretchable sensors for wearable applications ...... 116
8.1 Sensor Fabrication ...... 117 8.1.1 Background ...... 118 8.1.2 Fabrication...... 118
8.2 Sensor Testing and Characteristics ...... 120 8.2.1 Test Setup ...... 121 vi
8.2.2 Tensile Testing ...... 122 8.2.3 Cyclic Testing ...... 124 8.2.4 Step Testing ...... 127
8.3 Sensor Based Applications ...... 129
8.4 Conclusions ...... 131
References ...... 132
CHAPTER 9: Development of Glove Mounted Stretchable Sensor Quadcopter Controller 134
9.1 Quadcopter Assembly ...... 134
9.2 Signal Flow ...... 136
9.3 Glove Assembly ...... 139
9.4 Operational Flow ...... 142
9.5 Single Curve Model Based Look Up Table ...... 143
CHAPTER 10: Conclusions ...... 146
Appendix 1: Tensors and PVFD Elastic Coefficient Matrix Reduction ...... 149
References ...... 156
Appendix 2: Mode Shapes of a Beam ...... 158
Appendix 2.1: Mode Shapes of Beam in Bending Vibration ...... 158
Appendix 2.2: Mode Shapes of Rod in Axial Vibration ...... 162
References ...... 165
Appendix 3: Beam in Bending Stress‐Displacement Equations ...... 166
References ...... 168
Appendix 4: Material Properties ...... 169
Appendix 4.1: Compression Sleeve ...... 169 vii
Appendix 4.2: PVDF ...... 169
References ...... 170
Appendix 5: PVDF Voltage Output from Transverse Beam Deflection Matlab Code ...... 171
Appendix 6: PVDF Voltage Output from Axial Beam Deflection Matlab Code ...... 179
Appendix 7: Neural Network Matlab Code ...... 185
Appendix 8: dSPACE/Simulink Configuration with CMEX Code ...... 197
Appendix 9: PVDF Sleeve and Arduino Pro Mini System Parts List ...... 202
Appendix 10: Custom PCB Parts List for PVDF Sleeve ...... 203
Appendix 11: Custom PCB Circuit Diagram ...... 204
Appendix 12: Arduino Voltage Sampling Settings and Code ...... 208
Appendix 13: ATMega328P Code for PVDF Sleeve ...... 211
Appendix 14: VB.NET Data Collection Application ...... 215
Appendix 15: Visual Studio Joystick Demo Setup and Code ...... 233
Appendix 16: eclipse Android App Setup and Code (Java and XML) ...... 241
Appendix 17: Visual Studio PowerPoint Add‐In Code...... 268
Appendix 18: Quadcopter Parts List ...... 283
Appendix 19: Glove based ATmega328p code (quadTX)...... 285
Appendix 20: Quadcopter receiver code (quadRX) ...... 292
Appendix 21: Glove Circuit Parts List ...... 294
Appendix 22: Glove Circuit Schematic ...... 295
viii
LIST OF SYMBOLS ' ‐Transpose E‐Electric field D‐Electric displacement T‐Stress S‐Strain ϴ‐Temperature σ‐Entropy U‐Internal energy Q‐Heat W‐Work A‐Helmholtz free energy potential s‐compliance matrix c‐stiffness matrix ‐permittivity matrix β‐impermittivity matrix ( 1 ) h‐piezoelectric electromechanical coupling matrix D ‐Open circuit boundary condition s ‐Clamped boundary condition th Xi ‐i element of neural network input vector X ‐input vector of voltages to be augmented by bias X ‐full input vector (including bias) th Yj ‐ j element of neural network hidden layer output vector th Zk ‐ k element of neural network output layer vector
aij ‐weight connecting input vector element i and hidden node j
bjk ‐weight connecting hidden node output vector element j and output node k g ‐logistic function
ix
LIST OF FIGURES
Page
Figure.1.1 Anatomically neutral position ...... 10
Figure 1.2 Body motion control loop ...... 11
Figure 1.3 Gesture Set 1: (a) neutral, (b) fist, (c) extension (up), (d) flexion (down) ...... 12
Figure 1.4 Gesture Set 2: (a) radial deviation (left), (b) neutral, (c) ulnar deviation (right) ...... 13
Figure 1.5 Controller cylindrical representation and approximate beam model ...... 14
Figure 2.1 Cartesian coordinated system showing normal and shear directions ...... 16
Figure 2.2 Heckmann diagram without coefficients ...... 18
Figure 3.1 1‐3 plane cross section of unimorph structure with dimensions ...... 25
Figure 3.2 PVDF shaded profile ...... 44
Figure 3.3 Transverse input‐logistic function ...... 46
Figure 3.4 Traveling logistic wave ...... 46
Figure 3.5 Axial input‐Gaussian function ...... 47
Figure 3.6 Gesture voltage profile ...... 48
Figure 3.7. Comparison of experimental and simulated output voltage for fist, flex, and extend gestures ...... 49
Figure 3.8. Comparison of experimental and simulated output voltage for right and left gestures
...... 50
x
Figure 4.1 Single continuous node ...... 51
Figure 4.2 Multilayer network ...... 53
Figure 4.3 Graphical depiction of temporal input vector X ...... 56
Figure 4.4 Vector matrix network depiction ...... 56
Figure 4.5 Propagation of error in output node weight ...... 59
Figure 4.6 Propagation of error in hidden node weight ...... 61
Figure 4.7 Input voltage, output values, and classification ...... 65
Figure 5.1. Experimental setup block diagram ...... 67
Figure 5.2. Experimental setup ...... 67
Figure 5.3. Neural network input signal (left axis) and output classification (right axis) ...... 69
Figure 5.4. Variability in fist gesture ...... 70
Figure 5.5. Variability in flex gesture ...... 71
Figure 5.6. Variability in flex gesture ...... 71
Figure 5.7. Controller operational block diagram ...... 73
Figure 5.8. Controller prototype as worn on right forearm: compression sleeve with adhered
(spatially shaded) PVDF and hardware...... 73
Figure 5.9. Signal variance: “Right”, “Left”, “Right”, “Right”, “Left”, “Right”, “Left”, “Left” ...... 75
Figure 5.10. Signal variance: “Left”, “Left”, “Right”, “Right”, “Right”, “Left” “Right”, “Right”,
“Left”, “Left” ...... 75
xi
Figure 5.11. Forearm gesture controller. (a) PCB based electronics with Bluetooth module. (b)
Spatially shaded PVDF senor as curved when sleeve is being worn. (c) Device as worn with called out location of electronics and PVDF...... 76
Figure 6.1 Left: Shift in initial strain and effect on strain energy from un‐stretched to stretched
state. Right: Equality in strain energies for fixed Δε...... 80
Figure 6.2 Tensile test results: force versus change in length ...... 81
Figure 6.3 Tensile test results: stress versus strain ...... 81
Figure 6.4 Testing PVDF for fusing ...... 83
Figure 6.5 PVDF sensor geometry ...... 84
Figure 6.6 PVDF lead attachment ...... 85
Figure 6.7 Leads inserted into crimp connector housing ...... 85
Figure 6.8 Adhered PVDF element on compression sleeve stretched around cylinder ...... 86
Figure 6.9 (a) Controller circuit schematic (b,c,d) PCB top, bottom, and iso with soldered components ...... 87
Figure 6.10. Overview of PCB system architecture ...... 89
Figure 6.11. Controller user interface displaying recognized motion (designed for PC) ...... 91
Figure 6.12 Application for Android tablet showing no recognized gesture ...... 92
Figure 6.13 Application for Android tablet showing recognized right gesture ...... 92
Figure 6.14 Application for Android tablet showing recognized left gesture ...... 93
Figure 6.15 Add‐In for Microsoft PowerPoint ...... 93
Figure 7.1. Fabricated composite beam with electrical connections ...... 99
xii
Figure 7.2. Probe station setup ...... 105
Figure 7.3. LDV measured peak to peak deflection as a function of peak to peak driving voltage.
Inset shows LDV...... 109
Figure 7.4. Frequency sweep showing velocity of deflection for two modes of EAP actuation used to drive the bimorph beams; blue curve represents electrostrictive actuation, while green curve represents piezoelectric actuation...... 110
Figure 7.5. PVDF and oscilloscope impedance model ...... 111
Figure 7.6. Beam tip deflection measured by LDV and PVDF ...... 112
Figure 8.1. (a) Sputtered bimetallic layers on full size PS sheet. Approximate dimensions: 50mm
(width) by 80mm (height) by 0.5mm (thickness). (b) Bimetallic layers on PS post shrinking (same sample as shown in (a)). Approximate dimensions: 20.8mm (width) by 35.0mm (height) by
1.9mm (thickness). SEM images of transferred sensor showing wrinkling in the Pt layer caused
by stiffness mismatches during the shrining process at 4,000x (c) and 10,000x (d) magnification.
...... 119
Figure 8.2. Stages of adapting the sensor for testing (note in (a)‐(c) sensor is on blue
background). (a) Transferred sensor (post chemical bath). (b) Sensor with FFC and colloidal
silver. (c) Sensor with tape. (d) Sensor mounted for testing...... 120
Figure 8.3. Quarter bridge Wheatstone schematic with amplification ...... 121
Figure 8.4. Test normalized sensor resistance as a function of strain when stretched at different
rates...... 123
xiii
Figure 8.5. Stress as a function of strain at various stretch rates measured during tensile testing.
Approximate PDMS modulus of 1.18 MPa...... 124
Figure 8.6. Op‐amp output voltage as a function of strain for five cycles during cyclic testing over indicated frequencies ...... 126
Figure 8.7. Stress as a function of strain for the first 10 cycles during cyclic testing at indicated
frequencies ...... 126
Figure 8.8. Sensor bulging during relaxation phase of cyclic testing...... 127
Figure 8.9. Superposition in electrical response to incremental stretch step ups (step size denoted in each subplot). Red hash lines indicate discontinuity in start and end of data collection periods...... 128
Figure 8.10. Relaxation effects in electrical response during incremental stretch step downs
(step size denoted in each subplot). Red hash lines indicate discontinuity in start and end of
data collection periods (each step)...... 128
Figure 8.11. Sensor adhered to right index finger of glove centered over MCP joint. Line
bisecting sensor indicates center of joint on glove for accuracy in sensor mounting...... 129
Figure 8.12. Full MCP joint flexion and extension sensor values and associated models...... 131
Figure 9.1 Top view of quadcopter (no propellers) ...... 135
Figure 9.2. Arduino XBee shield on underside of quad ...... 135
Figure 9.3 Side view of quadcopter showing NAZA flight controller on top and Arduino with
XBee shield on bottom powered by main battery using barrel jack adapter ...... 136
Figure 9.4 Traditional transmitter control sticks and corresponding functions ...... 137
xiv
Figure 9.5 Collected PPM signals with all transmitter sticks in a neutral position ...... 138
Figure 9.6 Flight controller with signal (white) and ground (black) inputs from Arduino...... 139
Figure 9.7 Glove with electronics showing (left) and tucked in pouch on strap (right)...... 140
Figure 9.8 Electronic components sizing perspective. Left to right: quarter, PCB with soldered elements, XBee, and lithium ion polymer battery ...... 140
Figure 9.9 Touch sensors on glove ...... 141
Figure 9.10 Operational state process flow (open loop) ...... 143
Figure 9.11 Single curve model for MCP joint flexion and extension with inset showing error 145
xv
LIST OF TABLES
Page
Table 1.1 Muscle action matrix ...... 13
Table 4.1 Neural network size parameters ...... 53
Table 4.2 Gesture set 1 output vectors ...... 57
Table 4.3 Gesture set 2 output vectors ...... 58
Table 4.4 Learning parameters ...... 63
Table 5.1 Gesture set 1 neural network size ...... 68
Table 5.2 dSPACE neural network reliability statistics ...... 68
Table 5.3 Gesture set 1 classifications ...... 68
Table 5.4 Gesture set 2 neural network size for Arduino Pro Mini ...... 73
Table 5.5 Gesture set 2 classifications ...... 74
Table 5.6. Gesture set 2 neural network size for custom PCB ...... 76
Table 6.1 Compression sleeve test sample properties ...... 80
Table 6.2. Mechanical properties of compression sleeve ...... 82
Table 6.3 PVDF sensor geometry parameter values ...... 84
Table 7.1. Composite beam layers and thicknesses listed top to bottom ...... 98
Table 7.2. Distance between active element boundaries and the neutral axis ...... 98
Table 8.1. Circuit values ...... 122
xvi
ACKNOWLEDGMENTS
First and foremost, I would like to thank my advisor Dr. Gregory Washington for his mentoring and friendship throughout my graduate career. Thank you also to Dr. Faryar Jabbari and Dr. Michelle Khine for their continued involvement and assistance throughout my candidacy and to Dr. Ken Mease and Dr. David Reinkensmeyer for assisting with my advancement to candidacy.
Thank you to Dr. Yosi Shacham‐Diamand, Dr. Slava Krylov, and Leeya Engel of Tel Aviv University (Israel) for their hospitality and collaboration. Many thanks also to Thao Nguyen for fabricating more than her fair share of stretchable sensors and to Dr. Lily Wu for her assistance with all things quadcopter related.
Many thanks to my ISSL family both at UCI and the OSU for making this journey a wonderful experience with special thanks to Dr. Leon Headings for the introduction and Dr. Farzad Ahmadkhanlou for all his council.
xvii
CURRICULUM VITAE
Kyle Robert Van Volkinburg
2008 B.S. in Mechanical Engineering, The Ohio State University
2010 M.S. in Mechanical Engineering, The Ohio State University
2016 Ph.D. in Mechanical Engineering, University of California, Irvine
2008‐2012 Research Associate, Mechanical Engineering, The Ohio State University
2012‐2016 Research Associate, Mechanical Engineering, University of California, Irvine
FIELD OF STUDY
System dynamics and control with an emphasis on smart and adaptive material based mechatronic systems and gesture based wearable technology.
PUBLICATIONS
Van Volkinburg, K. R., Washington, G. N., “Modeling of a PVDF based gesture controller using energy methods”, Proc. SPIE Electroactive Polymer Actuators and Devices (EAPAD), 2014 Van Volkinburg, K. R., Washington, G. N., “PVDF based wearable joystick using gesture recognition via neural networks”, Proc. SMASIS Volume 2: Mechanics and Behavior of Active Materials; Integrated System Design and Implementation; Bioinspired Smart Materials and Systems; Energy Harvesting, 2014 Van Volkinburg, K. R., Nguyen, T., Pegan, J., Khine, M., Washington, G. N., “Use of shape memory polymer polystyrene in the creation of thin film stretchable sensors for wearable applications” Proc. SPIE Active and Passive Smart Structures and Integrated Systems, 2016 Engel, L., Van Volkinburg, K. R., Ben‐David, M., Washington, G. N., Krylov, S., Shacham‐Diamand, Y., “Fabrication of a self‐sensing electroactive polymer bimorph actuator based on polyvinylidene fluoride and its electrostrictive terpolymer”, Proc. SPIE Electroactive Polymer Actuators and Devices (EAPD), 2016 Van Volkinburg, K. R., Washington, G. N., “Development of a Wearable Controller for Gesture Recognition Based Applications using Polyvinylidene Fluoride”, in preparation
xviii
Van Volkinburg, K. R., Nguyen, T., Khine, M., Washington, G. N., “Glove Mounted Bi‐metallic PDMS Bonded Thin Film Sensor for Wearable Applications and Joint Angle Based Control”, in preparation
xix
ABSTRACT OF THE DISSERTATION
Development of Wearable Gesture Based Controllers using Intelligent and Adaptive Materials
By
Kyle Robert Van Volkinburg
Doctor of Philosophy in Mechanical Engineering
University of California, Irvine, 2016
Professor Gregory Washington, Chair
In developing wearable controllers, or simply wearables, for gesture based applications, in addition to simply being accurate devices must further be intuitive, body conforming, and unrestricting of motion. The premise behind wearables is that some physical interaction is
required to interface with every day electronic devices be it, for example, a keyboard, mouse,
remote control, knob, or button. In the sense that humans have a tendency to gesticulate when
verbally communicating with other humans, wearables allow humans to communicate with, or
rather control, electronic devices through intuitive, programmable, gesticulations. Being
presented is the design, development, and quantification of two wearable gesture based
controllers.
The first controller, worn on the forearm, uses the thin film piezoelectric polymer
Polyvinylidene Fluoride (PVDF) to detect hand motion based muscle contractions. The PVDF
was spatially shaded to enhance its sensitivity to the forces being exerted on it. A time delay xx
artificial neural network algorithm was used to process the PVDF signal and identify the hand motion that generated it. Two distinct sets of hand gestures were studied: fist, extension, and flexion and right and left. Using microcontrollers and Bluetooth, applications demonstrating the capabilities of the controller were developed for the right and left gesture set including an
Android oscilloscope style application showing PVDF voltage and identified gesture and a
Microsoft PowerPoint Add‐In that allows for gestures to navigate the presentation.
The second controller uses a bimetallic thin foil strain gauge bonded to the elastomer PDMS mounted on a golf glove above the MPC joint of the right index finger. The dynamic behavior of
the sensor was studied and a relation between sensor measurements and MPC joint angle
developed. A custom printed circuit board (PCB) featuring a microcontroller and RF
communication was developed enabling the device, in response to joint angle, to control the
throttle input to a quadcopter.
The fabrication and use of PVDF, in conjunction with its electrostrictive terpolymer P(VDF‐TrFE‐
CTFE), in a self‐sensing bimorph beam is also discussed. While bending in the beam is induced
by applying voltage across the terpolymer, measured voltage from the PVDF is shown to give
real time tip deflection.
xxi
INTRODUCTION
The hypothesis that drove this research was that it was possible to design and develop
wearable gesture driven controllers using intelligent and adaptive materials. Success in
developing wearable controllers, or simply wearables, hinges on the ability to create devices that are accurate, intuitive, and unobtrusive. In the sense that humans are capable of
communicating with each other through a combination of gestures, most commonly performed
by the hand, and sensory mechanisms, wearables enable the same capabilities between
humans and electronics. Developed in this dissertation are two gesture based wearables: first, a
controller worn on the forearm capable of distinguishing hand gestures and second, a
controller worn on the hand that takes action based on measured index finger
metacarpophalangeal (MPC) joint angles. A third concept of fabricating a self‐sensing bimorph
beam and determining tip deflection is also developed.
The first controller, worn on the right forearm, uses the smart material polyvinylidene fluoride
(PVDF), a thin film piezoelectric polymer, as the sensory mechanism. As is true for its
piezoelectric ceramic counterparts, PVDF outputs charge in response to strain. However, being
a polymer, PVDF can be easily applied to contoured surfaces. Further, as the PVDF used was
coated on both sides in silver ink (electrodes), it can easily be etched or spatially shaded thus
making it more sensitive to desired deformations [1]. As utilized, the PVDF was affixed to a
forearm compression sleeve such that it was located above the extensor muscles of the
forearm and spatially shaded in such a way as to increase sensitivity to specific muscle activity.
Contraction and extension of these extensor muscles, in conjunction with the forearm flexor
1
muscles, are what drive the motion of the hand. Further, as a result of its location, the contraction and extension of these extensor and flexor muscles cause the PVDF to undergo deformation and hence output charge. The same hand motion performed repeatedly at the same rate will deform the PVDF in identically the same way and an identical time dependent charge profile will be produced. Thus, by monitoring this charge profile, in actuality a voltage profile, it is possible to determine when a specific gesture has been performed. However, tasking a human to perform an identical gesture at an exact rate is both impractical and
impossible and, as such, the controller processes the PVDF voltage signal using an artificial
neural network algorithm to identify when gestures similar to a specific gesture have been
performed. While the field of wearables inherently revolves around the concept of “putting
something on”, this device represents a paradigm shift from merely an accessory towards a
technology that can directly be integrated into base layers of clothing.
Two separate versions of this PVDF based controller have been developed. The first version was tasked with identifying up (extension), down (flexion), and fist motions of the hand and relied
on a tabletop data acquisition (DAQ) system, specifically a dSPACE ds1103 board, and
supporting ControlDesk software. Due to the single distributed sensor and the overlap of muscles involved in the down and fist gestures, this version of the controller did not perform as well as was desired. The second version was tasked with identifying right (ulnar deviation) and
left (radial deviation) motions of the right hand and replaced the tabletop DAQ system for a
developed Arduino Pro Mini based system with Bluetooth capabilities thus making the
controller untethered and truly wearable. In addition to being incredibly accurate, this
controller was able to communicate with any Bluetooth enabled device. As such, an Android
2
application was developed showing, in real time, the PVDF sensor’s generated voltage signal and identified when a right or left gesture was performed. Additionally, to make this a true controller, a Microsoft PowerPoint “Add‐In” was developed such that when the Add‐In detected a right gesture it advanced the presentation one step and, conversely, when it detected a left gesture it moved the presentation back one step; akin to pressing the right or
left keyboard arrows, respectively.
The second controller, a glove based system worn on the right hand, uses an electric skin (e‐ skin) style elastomer bonded bimetallic foil gauge as the sensory mechanism. The sensor acts as
a strain gauge and functions under the premise of geometric piezoresistivity; as the sensor is stressed its resistance increases. The sensor is mounted on the right index finger of a snug fitting golf glove approximately centered over the metacarpophalangeal (MCP) joint. As the
finger cycles through flexion and extension the sensor is strained and relaxed, respectively. By
incorporating the sensor, electronically, as the variable element in a quarter bridge Wheatstone
circuit in conjunction with differential amplification, a microcontroller (Atmel ATmega328p) was
used, specifically the analog to digital (ADC) pin, to relate the voltage measured by the pin to
the angle of the MCP joint. Due to the viscoelastic nature of the elastomer used,
polydimethylsiloxane (PDMS), two curves were needed to model the joint angle versus
measured voltage relationship; one for the finger in flexion and one for the finger in extension.
In both cases, a logistic function was used and resulted in a high degree of accuracy. The glove,
in addition to the strain sensor, incorporates two touch sensors and functions as the throttle controller for a quadcopter via radio frequency (RF) communication. All necessary electronics
3
were fitted on a custom developed surface mount printed circuit board (PCB) and fitted into a
pouch on the top of the glove strap.
The final component of this dissertation involves the fabrication of a self‐sensing
cantilevered bimorph beam actuator and its frequency response using the electroactive
terpolymer polyvinylidene fluoride‐trifluoroethylene chlorotrifluoroethylene (P(VDF‐TrFE‐
CTFE)) and PVDF. Bending of the beam was induced by applying an AC voltage signal across ferroelectric relaxor (P(VDF‐TrFE‐CTFE)). The beam’s tip deflection was recorded in real time externally using a laser Doppler Vibrometer (LDV). Voltage data from the PVDF sensing layer was also collected real time and used post testing to calculate frequency dependent tip
deflection based on a developed sensing model. The sensing model treats the structure as a
simple Euler‐Bernoulli cantilevered beam with two distributed active elements represented
through the use of generalized functions and offers a method through which real time tip
deflection can be determined without the need for external equipment.
PVDF is extremely light, flexible, and has higher piezoelectric stress constants than ceramic
piezoelectrics [2] making it an excellent candidate for use in sensor applications as detailed by,
among others, Lee [3] and Sirohi [4], as in most applications it does not appreciably alter the dynamics of the plant. Further, PVDF can be cut or the electrodes spatially shaded which lends
it to modal sensing and vibration control applications of elementary structures with known
mode shapes such as beams and plates as detailed by, among others, Lee and Moon [5],[6],
Bailey and Hubbard [7], Burke and Hubbard [8], Sullivan et al [9], Collins et al [10], and Clark et
al [11].
4
Capturing and quantifying hand gestures has, traditionally, been achieved through position
tracking systems or glove based systems [12]. Position tracking systems predominantly employ
optical, magnetic, or acoustic sensing technologies to locate the hand in space while glove
systems use piezoresistive, fiber optic, or Hall‐effect sensors to measure the shape of the hand.
Research in the field of capturing data generated by hand gestures through the use of a glove
began with the Sayre Glove in 1977, the MIT‐LED glove in the early 1980s and the Digital Entry
Data Glove in 1983 and has been a topic of interest ever since [13].
Gesture recognition, either static or dynamic, is the ability of a machine to recognize human motion. Applications of gesture recognition include interpreting sign language, medical rehabilitation, interacting with virtual reality, and general human‐computer interaction. One specific study investigates using gesture recognition to control the motion of a two‐wheel
vehicle [14]. In general, this is made possible by passing sensor data from, for example, a
tracking system or glove system through an appropriate algorithm. Algorithms used for
dynamic gesture recognition include time‐compressing templates, dynamic time warping,
hidden Markov models, and time delay neural networks [15]. Focusing on hand gesture
recognition, neural networks have been used, for example, in studies where sensor data was generated by a virtual reality glove [16], a gaming controller [17], a gaming glove [18], and a webcam [19].
Wearable technologies, or simply wearables, are clothing or accessories which collect and process data generated by the wearer. Common examples in the field of biometrics include clip
on pedometers, wrist worn pedometers, and shirts which measure heart rate. A more complex
5
biometric related study uses a Lycra leotard with numerous chemically deposited strain sensors
to determine body kinematics and gestures [20]. A specific human‐computer interface example
is a finger worn “mouse”1 while an example of a more general accessory style wearable controller is an armband which determines hand gestures using accelerometer data and
electromyography (EMG) signals2 [21]. PVDF has been incorporated into wearables to monitor
muscle activity as an ultrasonic sensor [22] and cardiorespiratory signals [23] and further has
also been integrated directly into textiles through manufacturing processes [24].
References
[1] Hubbard Jr., J. E., [Spatial Filtering for the Control of Smart Structures], Springer, Berlin, Heidelberg (2010). [2] R. C. Smith, Smart material systems: model development. Philadelphia: Society for Industrial and Applied Mathematics, 2005. [3] C. K. Lee, “Theory of laminated piezoelectric plates for the design of distributed sensors/actuators. Part I: Governing equations and reciprocal relationships,” The Journal of the Acoustical Society of America, vol. 87, no. 3, pp. 1144–1158, 1990. [4] J. Sirohi and I. Chopra, “Fundamental Understanding of Piezoelectric Strain Sensors,” Journal of Intelligent Material Systems and Structures, vol. 11, no. 4, pp. 246–257, Apr. 2000. [5] C.‐K. Lee and F. C. Moon, “Modal Sensors/Actuators,” J. Appl. Mech., vol. 57, no. 2, pp. 434–441, Jun. 1990. [6] C.‐K. Lee and F. C. Moon, “Laminated piezopolymer plates for torsion and bending sensors and actuators,” The Journal of the Acoustical Society of America, vol. 85, no. 6, pp. 2432–2439, 1989. [7] T. Bailey and J. E. Hubbard, “Distributed piezoelectric‐polymer active vibration control of a cantilever beam,” Journal of Guidance, Control, and Dynamics, vol. 8, no. 5, pp. 605– 611, 1985. [8] S. E. Burke and J. E. Hubbard, “Active vibration control of a simply supported beam using a spatially distributed actuator,” IEEE Control Systems Magazine, vol. 7, no. 4, pp. 25–30, 1987.
1 http://www.mycestro.com/ 2 https://www.thalmic.com/en/myo/ 6
[9] J. M. Sullivan, J. Hubbard, and S. E. Burke, “Distributed sensor/actuator design for plates: spatial shape and shading as design parameters,” Journal of Sound and Vibration, vol. 203, no. 3, pp. 473–493, Jun. 1997. [10] S. A. Collins, D. W. Miller, and A. H. Von Flotow, “Distributed Sensors as Spatial Filters in Active Structural Control,” Journal of Sound and Vibration, vol. 173, no. 4, pp. 471–501, Jun. 1994. [11] R. L. Clark, R. A. Burdisso, and C. R. Fuller, “Design Approaches for Shaping Polyvinylidene Fluoride Sensors in Active Structural Acoustic Control (ASAC),” Journal of Intelligent Material Systems and Structures, vol. 4, no. 3, pp. 354–365, Jul. 1993. [12] D. J. Sturman and D. Zeltzer, “A survey of glove‐based input,” IEEE Computer Graphics and Applications, vol. 14, no. 1, pp. 30–39, Jan. 1994. [13] L. Dipietro, A. M. Sabatini, and P. Dario, “A Survey of Glove‐Based Systems and Their Applications,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, vol. 38, no. 4, pp. 461–482, Jul. 2008. [14] M. A. B. Kasno, J. Ahn, K. Jung, Y. Lee, and K. Eom, “Real‐Time Hand Gesture SEMG Using Spectral Estimation and LVQ for Two‐Wheel Control,” in Security‐Enriched Urban Computing and Smart Grid, T. Kim, A. Stoica, and R.‐S. Chang, Eds. Springer Berlin Heidelberg, 2010, pp. 335–344. [15] S. Mitra and T. Acharya, “Gesture Recognition: A Survey,” IEEE Transactions on Systems, Man, and Cybernetics, Part C: Applications and Reviews, vol. 37, no. 3, pp. 311–324, May 2007. [16] F. Parvini, D. McLeod, C. Shahabi, B. Navai, B. Zali, and S. Ghandeharizadeh, “An Approach to Glove‐Based Gesture Recognition,” in Human‐Computer Interaction. Novel Interaction Methods and Techniques, J. A. Jacko, Ed. Springer Berlin Heidelberg, 2009, pp. 236–245. [17] F. Arce and J. M. G. Valdez, “Accelerometer‐Based Hand Gesture Recognition Using Artificial Neural Networks,” in Soft Computing for Intelligent Control and Mobile Robotics, O. Castillo, J. Kacprzyk, and W. Pedrycz, Eds. Springer Berlin Heidelberg, 2011, pp. 67–77. [18] P. A. Harling, “Gesture Input using Neural Networks,” University of York, Heslington, UK, 1993. [19] G. R. S. Murthy and R. S. Jadon, “Hand gesture recognition using neural networks,” in Advance Computing Conference (IACC), 2010 IEEE 2nd International, 2010, pp. 134–138. [20] D. De Rossi, F. Lorussi, A. Mazzoldi, P. Orsini, and E. P. Scilingo, “Monitoring body kinematics and gesture through sensing fabrics,” in Microtechnologies in Medicine and Biology, 1st Annual International, Conference On. 2000, 2000, pp. 587–592. [21] D. Tan, D. Morris, T. S. Saponas, and R. Balakrishnan, “Recognizing finger gestures from forearm EMG signals,” US 20130232095 A1. [22] I. AlMohimeed, H. Turkistani, and Y. Ono, “Development of wearable and flexible ultrasonic sensor for skeletal muscle monitoring,” in Ultrasonics Symposium (IUS), 2013 IEEE International, 2013, pp. 1137–1140. [23] S. Choi and Z. Jiang, “A novel wearable sensor device with conductive fabric and PVDF film for monitoring cardiorespiratory signals,” Sensors and Actuators A: Physical, vol. 128, no. 2, pp. 317–326, Apr. 2006.
7
[24] A. S. Krajewski, K. Magniez, R. J. N. Helmer, and V. Schrank, “Piezoelectric Force Response of Novel 2D Textile Based PVDF Sensors,” IEEE Sensors Journal, vol. 13, no. 12, pp. 4743–4748, Dec. 2013.
8
CHAPTER 1: Physiology and Overview of the Wearable Gesture Based Controller using Polyvinylidene Fluoride
The first gesture controller developed used the thin film piezoelectric polymer polyvinylidene fluoride (PVDF) as the sensory mechanism where the PVDF was affixed to a forearm compression sleeve. Holding one’s right arm out at extension with the palm of the hand facing downward, the PVDF sensor is located on the top of the forearm and outputs charge in
response to hand gestures or, more accurately, in response to the muscle activity responsible
for creating hand gestures. To better understand the nature of the forces and boundary
conditions acting on the PVDF, the reasons for and shape of the spatial profile of the sensor,
and in general to develop a better device, it was first necessary to examine the physiology of the forearm and develop a representative physical model.
1.1 Physiology
Motion of the body is achieved through the contraction of muscles; the conversion of chemical energy into force [1]. Muscles transmit forces to tendons which, in turn, create rotations about
joints. Muscles are grouped in antagonistic sets; while one set of muscles contracts the other set lengthens. The basic unit of a muscle is the sarcomere which is comprised of actin and myosin filaments. Overlaps of the two filaments are called cross‐bridges where full muscle
contraction corresponds with the maximum number of cross‐bridges. Increasing the number of
cross‐bridges decrease the length of the sarcomere; fully contracted sarcomeres are decreased
to about 70 percent from their un‐contracted length [1]. This sarcomere shortening results in a
muscle “bulge”.
9
Motion of the hand and wrist is driven by the contraction, and complementary lengthening, of
muscles located in the forearm. As what is under investigation is how the PVDF is deformed during muscle contraction, only the superficial muscles, or outer most muscles, are discussed further. The region of interest is approximately two inches distal the right elbow, on a
representative 30 year old male, where muscle bulging was found to be most prominent during
hand gesticulation. Two sets of gestures were studied and are detailed in the following section
with all body references being made with respect to the anatomically neutral position shown in
Figure.1.1.
Figure.1.1 Anatomically neutral position
Body motion can be modeled using a traditional control system style feedback loop where one
such representation is shown in Figure 1.2 [1],[2]. As already stated, muscle forces drive all
body motion so naturally they represent the actuators of the body. The disturbance input is any
10
load opposing a motion, such as the gravitational force of an object being lifted. The musculoskeletal geometry and body dynamics blocks combine to represent the analogous equivalent of a plant. The sensor block is a conglomeration of proprioceptive and tactile sensors as well as the visual and vestibular system while is a time delay. The central nervous system
is the controller that regulates all motions of the body. While the sensor block’s “variables” are all internal to the system (the body) and hence non observable, the purpose of the PVDF is to,
like all previous gesture monitoring systems mentioned in the introduction, be a sensor whose
outputs are observable and thus can be processed and used to drive computational algorithms.
Figure 1.2 Body motion control loop
1.2 Gesture Sets
Two sets of gestures performed with the right hand were studied and are detailed in the
following subsections. All gestures started and ended with the right hand being in a neutral, or
prone, position.
1.2.1 Gesture Set 1
The first set of gestures included making a fist, extension (up), and flexion (down) and are
shown in Figure 1.3. The associated muscles of interest are the superficial extensors (carpi
11
radialis longus, carpi radialis brevis, digitorum, and carpi ulnaris) and flexors (carpi radialis,
palmaris longus, carpi ulnaris, and digitorum superficialis) [3],[4]. The flexors are located on the
anterior of the forearm while the extensors are located on the posterior of the forearm.
Figure 1.3 Gesture Set 1: (a) neutral, (b) fist, (c) extension (up), (d) flexion (down)
1.2.2 Gesture Set 2
The second set of gestures studied were ulnar deviation (right) and radial deviation (left) and are shown in Figure 1.4. The muscles of interest responsible for these motions, the ones used to generate side to side motions of the hand at the wrist, are the superficial wrist adductors
(flexor carpi ulnaris and extensor carpi ulnaris) and abductors (flexor carpi radialis, extensor
carpi radialis brevis, and extensor carpi radialis longus) which, respectively, allow for ulnar and
radial deviations of the wrist [3],[4].
12
Figure 1.4 Gesture Set 2: (a) radial deviation (left), (b) neutral, (c) ulnar deviation (right)
1.2.3 Gesture Muscle Matrix
The specific superficial forearm muscles used to complete each gesture are shown in Table 1.1 where the tense and release actions correspond with neutral to fist and fist to neutral gestures.
As can be seen in the table, each muscle is used to perform multiple gestures.
Table 1.1 Muscle action matrix Action Muscles Flex Left Tense Right Extend Release Flexor Carpi Radialis X X Palmaris Longus X X Flexor Carpi Ulnaris X X Flexor Digitorum X X Superficialis Extensor Carpi X X Radialis Longus Extensor Carpi X X Radialis Brevis Extensor Digitorum X X Extensor Carpi Ulnaris X X
13
1.3 Modeling
The wearable aspect of the controller is comprised of a compression sleeve and adhered PVDF
element as diagrammed in the left side of Figure 1.5. Also shown are the regions where forces
are applied from muscle activity. A classic simply supported Euler‐Bernoulli beam model is used
to approximate the wearable’s behavior and is diagrammed in the right side of Figure 1.5. The
distributed forces in the cylindrical representation are broken up into transverse and axial loadings on the beam. Also, only the segment of the sleeve that is covered by the PVDF is considered in the beam model creating a pin‐pin unimorph. The extended Hamilton’s principle
is used to derive the sensor’s governing equations relating forces on the beam to output
voltage of the PVDF later in chapter 3.
Figure 1.5 Controller cylindrical representation and approximate beam model
References
[1] M. Kutz, Ed., Standard handbook of biomedical engineering and design. New York: McGraw‐Hill, 2003. [2] H. van der Kooij, B. Koopman, and F. van der Helm, “Human Motion Control.” [3] F. P. Kendall, Muscles, testing and function, 3rd ed. Baltimore: Williams & Wilkins, 1983. [4] J. E. Muscolino, The muscular system manual: the skeletal muscles of the human body, 2nd ed. St. Louis, Mo: Elsevier Mosby, 2005.
14
CHAPTER 2: Piezoelectrics and PVDF
The piezoelectric effect was discovered by Pierre and Jacque Curie in 1880 [1]. The effect
literally means pressure electricity and is associated with a class of materials known as
piezoelectric materials. Piezoelectrics are governed by constitutive relationships which state
that under stress a charge is produced, the direct effect, with the converse also true, known as
the indirect effect, relating an applied electric field and strain. In 1969 Kawai discovered that the thin film polymer PVDF exhibited piezoelectric properties, belonged to the mm2 crystal symmetry group, and hence is an orthotropic material [2].
2.1 Background and Convention
Piezoelectrics are materials, traditionally ceramics and polymers, which are both elastic and
dielectric and generate charge in response to an applied force. The behavior of both elastic and
dielectric materials are represented by tensor constitutive equations and, as such, piezoelectric
constitutive equations are also tensor equations. Tensors, as pertaining to piezoelectrics, are an
array of numbers subject to transformation rules; a tensor described by one coordinate system
can described by a second coordinate system through a transformation which is both linear and
reversible [3].
Stress and strain are both second order tensors, containing nine terms, and hence related by a
fourth order tensor of 81 (elastic) constants. Electric field and electric displacement are first
order tensors, three terms, and hence related by a second order (permittivity) tensor. The
stress and strain tensors of an elastic material are shown in appendix 1 to be symmetric such
15
that they consist of six independent terms. Also shown in appendix 1 is that the tensor of elastic constants can be reduced to nine independent terms by applying both the fact that the stress
and strain tensors are symmetric as well as Neumann’s principle which states that tensors
representing any physical property are invariant with regard to every symmetry operation of
the governing crystal class where again PVDF belongs to the orthorhombic mm2 crystal class
[4].
Stress T and strain S are henceforth defined as vectors according to
TT111 S 1 S 11
TT222 S 2 S 22 TT S S TS333 , 3 33 (2.1) TT4 23 T 32 S 4 S 23 S 322 S 23
TT5 31 T 13 S 5 S 31 S 132 S 13
TT6 12 T 21 S 6 S 12 S 212 S 12
where the bold text denotes a vector, the single subscript is Voigt notation, and the double
subscript, in conjunction with Figure 2.1, denote face and direction.
Figure 2.1 Cartesian coordinated system showing normal and shear directions
16
Similarly, the electric displacement D and electric field E are defined as
DE11
DE DE22, = (2.2)
DE33
The constitutive equations of an elastic and dielectric material are
ST s (2.3) DE
where s and are the symmetric compliance and permittivity matrices respectively.
2.2 Domains and State Variables
The state of a system within a physical domain can be defined by an appropriate pair of state
variables; macroscopic path independent characteristics of a system [5]. Physical domains include mechanical, electrical, thermal, magnetic, and chemical domains [6]. As an elastic
dielectric, piezoelectric materials exhibit domain coupling; that is, their state is dictated by
multiple pairs of state variables with one pair for each domain. PVDF is also known to be
pyroelectric [7]. The number of state variables necessary to determine the state of a
piezoelectric material is dependent on the number of domains considered. A diagram representing the relationship between state variables in the mechanical, electrical, and thermal domains, known as Heckmann’s diagram ([8] per [9]) is shown in Figure 2.2 without tensor
subscripts or coefficients for clarity where and are temperature and entropy respectively.
The state variables on the corners of the outer triangle are independent and intensive while the
17
state variables on the corners of the inner triangle are dependent and extensive variables. The
arrows point from independent to dependent state variables. The short arrows show principle
effects while the long arrows show coupling effects.
Figure 2.2 Heckmann diagram without coefficients
2.3 Linear Constitutive Equations from Thermodynamics
Sets of equations relating the state variables of a piezoelectric, known as constitutive equations, can be derived from thermodynamic considerations when certain assumptions are permitted, mainly that the system is reversible and only incremental deviations from equilibrium are considered. There are multiple forms of the constitutive equations, all related
through transformations, depending on the set of independent variables used in the derivation
process.
The first and second laws of thermodynamics, for a reversible system, are
dU dQ dW (2.4) dQ d
18
The incremental change in work in a piezoelectric, since it is an elastic dielectric, is made up of
mechanical and electrical contributions and, using Voigt notation (ik1...6, 1,2,3), is
dW Tii dS E k dD k (2.5)
Combining (2.4) and (2.5) the incremental change in internal energy is
dU d Tii dS E k dD k (2.6)
Internal energy is itself a state variable dependent on strain, electric displacement, and entropy
of the system and written generally as US(,ik D ,) . Further, internal energy is one of eight
thermodynamic energy potentials (all related by Legendre transformations [9]), each itself a
state variable defined by a unique set of independent state variables. The energy potential used
moving forward will set the independent variables of the constitutive equations. Desiring strain, electric displacement, and temperature as independent variables, for reasons that will become
evident in chapter 3, the Helmholtz free energy function is used
AS(,ik D ,) U (, S ik D ,) (2.7)
Differentiating (2.7) using the chain rule
dA(, Sik D ,) dU (, S ik D ,) d d (2.8)
Plugging in dU from equation (2.6) gives
19
dA(, S D ,) d T dS E dD d d ik ii kk (2.9) dA(, Sik D ,) T ii dS E kk dD d
In general, the Helmholtz free energy function AS(,ik D ,) is a function of three independent
variables. Taking the total derivative yields
AA A dA(, Sik D ,) dS i dD k d (2.10) SDik
Comparing the second line of (2.9) with (2.10) shows
A AA TEik,, (2.11) SDik
Evaluating the Taylor series expansion of the Helmholtz free energy through the first set of partial derivatives where AASD0000(, , ) and the tuple of initial conditions (,SD000 , ) is
defined as Q0 gives
AAA A(,SDik ,) A o S i S000 D k D hot ... (2.12) SDikQ QQ000
Defining the following, assuming zero initial strains and electric displacements,
S0 0
D0 0 (2.13)
0
the Taylor series can be simplified to
20
AA A A(,SDik ,) A o S i D k hot ... (2.14) SDik Q QQ000
Plugging (2.14) into each of the partial derivatives in (2.11) yields (2.15) which is the set of
linear piezoelectric constitutive equations
AAAA 22 2 TASDiik0 SSii SS ij DS ki S i Q0 QQ00 AAAA 22 2 EASDkik0 (2.15) DDkk SD ik DD kl D k QQQ000
222 AAAA AS0 ik D 2 SDik QQQ000
Noting that the partials of A0 drop out and assuming isothermal operating conditions, equation
(2.15) simplifies to
22AA TS D iiSS k DS ijQ kiQ 0 0 (2.16) 22AA ESki D k SDik DD kl QQ00
where the second order partials following Si are fourth order tensors and those following Dk are third order tensors by necessity of the tensor quotient rule [10]; both tensors consist of material coefficients [9]. Written in vector matrix form using standard notation, (2.16) is more commonly seen as (2.17) where the superscripts D and S have the standard meaning of open circuit and clamped boundary conditions respectively [11].
21
TSDchD ' (2.17) ESDh S
2.4 Energy Functions
A free energy function of a linear system of state variables, e.g. equation (2.17), is uniquely determined by specifying a quadratic function of the two independent state variables and solving for unknown coefficients x , y , and z [12]
11 A(,SD ) S 'xy S S ' D D ' z D (2.18) 22
The idea is to plug the proper equations from (2.17) into (2.11) and compare against the partials of (2.18). Since (2.17) is a set of vector matrix equations, (2.11) needs to be amended
such that the result of the differentiation is a column vector. As such,
A TSDchD ' S ' (2.19) A ESDh S D'
Taking the partials of (2.18) gives
A 1 (')x xySD S'2 (2.20) A 1 yzz'(')SD D'2
For (2.19) and (2.20) to be equivalent
22
cD xx' 2 yh ' (2.21) S zz' 2
reinforcing that the stiffness and impermittivity coefficient matrices, cD and S , are
symmetric. Finally, the quadratic Helmholtz free energy function in terms of state variables S and D is given by (2.22) which is one representation of the per unit volume energy in a piezoelectric.
11 Ach(,SD ) S 'DS S S ' ' D D ' D (2.22) 22
References
[1] R. C. Smith, Smart material systems: model development. Philadelphia: Society for Industrial and Applied Mathematics, 2005. [2] H. Kawai, “The Piezoelectricity of Poly (vinylidene Fluoride),” Jpn. J. Appl. Phys. Japanese Journal of Applied Physics, vol. 8, no. 7, pp. 975–976, 1969. [3] W. J. Gibbs, Tensors in Electrical Machine Theory. London: William Clowes and Sons, Ltd., 1952. [4] E. Hartman, “An Introduction to Crystal Physics,” 2001. [5] M. J. Moran, Fundamentals of engineering thermodynamics, 5th ed. Hoboken, NJ: Wiley, 2004. [6] D. Leo, Engineering Analysis of Smart Material Systems, 1st ed. New Jersey: John Wiley & Sons, Inc., 2007. [7] Measurement Specialties, Inc., “Piezo Film Sensors Technical Manual,” Mar‐2008. [8] G. Heckmann, “Die Gittertheorie der festen Körper,” in Ergebnisse der Exakten Naturwissenschaften, 4th ed., 1925, pp. 100–153. [9] J. Tichy, Fundamentals of piezoelectric sensorics: mechanical, dielectric, and thermodynamical properties of piezoelectric materials, 1st ed. New York: Springer, 2010. [10] Y. C. Fung, A first course in continuum mechanics, 2d ed. Englewood Cliffs, N.J: Prentice‐ Hall, 1977. [11] “IEEE Standard on Piezoelectricity,” ANSI/IEEE Std 176‐1987, p. 0_1–, 1988.
23
[12] T. Ikeda, Fundamentals of piezoelectricity. Oxford ; New York: Oxford University Press, 1990.
24
CHAPTER 3: Sensor Dynamics and System Response
This chapter develops the governing equations of the pin‐pin unimorph beam system,
previously introduced in section 1.3 which are used to both spatially shade, or etch, the PVDF as well as simulate the system under loading from muscle contractions. The unimorph from Figure
1.5 is repeated in Figure 3.1 without forces, with dimensions, and using axis definitions
consistent with piezoelectric notation where directions 1, 2, and 3 correspond with x, y, and z
respectively. The coordinate system of the unimorph is situated such that the 1 axis and neutral
axis of the non active element are collocated, Figure 3.1, and equally divide the thickness of the
non active beam element, or structure, ts . The PVDF is situated on top of the beam and has
thickness t p . It is assumed that the beam and PVDF are perfectly bonded, no shearing in the adhesive layer, such that each experiences the same deformations at all increments of time.
The governing dynamic equations of the system are developed using the extended Hamilton’s principle which allows for the energy contributions of the beam and PVDF to be considered
separately.
Figure 3.1 1‐3 plane cross section of unimorph structure with dimensions
25
3.1 The Extended Hamilton’s Principle
The extended Hamilton’s principle states that the actual motion of the considered dynamical system renders the integral I in (3.1) an extremum with respect to continuous, twice
differentiable, generalized independent functions qtqt12( ), ( ),... qN ( t ) where L is the
Lagrangian of the system and Wnc is the external work due to non‐conservative forces [1]
t2 ILWdt nc t1
IIqtqtqt (12 (), (),...N ()) (3.1)
L Lqtqt(12 (), (),... qNN (), tqtqt 12 (), (),... q ()) t
WWqtqtqtnc nc(12 (), (),... N ())
Solving (3.1) per variational calculus it is known that [2]
t2 L Wdtnc 0 (3.2) t1
which results in a system of simultaneous Euler‐Lagrange equations
fdf 0 qdtqii (3.3)
fLWnc
from which the resulting qsi ' , iN1... , render (3.1) an extremum and hence yield the true
motion of the system.
26
3.2 Self‐Adjoint Systems and the Assumed Modes Method
Beams are distributed parameter systems and hence their motion is governed by partial differential equation boundary value problems. The solution of the boundary value problem
involves the solution of an associated differential eigenvalue problem. As a direct consequence
of beams being self‐adjoint systems, the solution of the differential eigenvalue problem is an
infinite set of eigenvalues and orthogonal eigenfunctions [2].
The assumed‐modes method, also referred to as the Ritz method [3], is a technique which
discretizes distributed parameter systems allowing for approximate solutions [2]. The system is
discretized in space as the infinite summation of the product of admissible shape functions and time dependent generalized coordinates where the orthogonal eigenfunctions obtained from solving the differential eigenvalue problem of the beam are valid choices for shape functions.
Defining ut(,)x as the displacement of the system in the direction consistent with free
vibration, it can be discretized into a linear combination of time dependent generalized
coordinates as
N ' utxxSxr, Sii rt t (3.4) i1
where Sx is a vector of admissible shape functions, r()t the vector of time dependent generalized coordinates, and x is the vector of spatial coordinates. From here the Lagrangian
can be formulated and Hamilton’s principle used to derive the equations of motion for the
discretized system.
27
3.3 Displacements
Displacements in the transverse direction, or the 3 direction, for the unimorph in accordance with (3.4) are given by
ut(,)x 0 0 0 rt () 11 ux(,)tut (,) x 0 0 0 ()() xr t 2 (3.5) N ut31(,)xxx () NN () rt () ii()()x rt i1
th th where i ()x is the i transverse mode shape of the beam and rti () the corresponding i generalized coordinate.
Displacements in the axial direction, or the 1 direction, for the unimorph in accordance with
(3.4) are given by
N ()()x rt ut(,)xxxii () () rt () 111i1 N ux(,)tut (,) x 0 0 0 ()() xr t 2 (3.6) ut(,)x 0 0 0 rt () 3 N
th th where i ()x is the i axial mode shape of the beam and rti () the corresponding i generalized coordinate which is understood to not be associated with those found in the transverse motion case. The mode shapes associated with both the transverse and axial displacements are well known and given with derivation in appendix 2.
28
3.4 Formulation of the Lagrangian
The Lagrangian L introduced in section 3.1 relates to the energy in a system and is defined as
the total kinetic energy less the total potential energy
LTV
TTT s p (3.7)
VVV s p
where the subscripts s and p refer to the beam and piezoelectric element respectively.
Displacement equations (3.5) and (3.6) are generalized as (3.8) where Nr ()x is a matrix of
appropriate mode shapes.
uux (,)tN r ()() xr t (3.8)
The Lagrangian is formulated in terms of generalized coordinates. As piezoelectrics convert
energy between the mechanical and electrical domains, per (2.16) isothermal conditions are
assumed, a set of generalized coordinates is required for each domain. The generalized position coordinate rti () has already been introduced for use in the mechanical domain. The charge
from the PVDF qtj () is now defined as the generalized coordinate of the electrical domain. The
Lagrangian can now be defined as a function of generalized position, generalized velocity, and charge
L Lrtrtqt ii(),(), j () (3.9)
29
3.4.1 Kinetic Energy
Kinetic energy of a rigid body is expressed as
1 Tm uu (3.10) 2
where from (3.8)
uxr Ntr ()() (3.11)
The kinetic energy of the system is the sum the kinetic energy of each element such that
TTTsp 11 (3.12) Tmuu'', T m uu ss22 pp
where ms and m p are the masses of the beam and piezoelectric elements respectively and given by
mmsss , ppp tt sst 22wLp wL (3.13) spdxdydz, dxdydz tt00 00 ss 22
where is the appropriate density and the appropriate volume. The kinetic energy of the system can thus be written as
30
tt sst 22wLp wL 11'' Tsvuu dxdydz uu dxdydz (3.14) 22tt00 00 ss 22
Incorporating (3.11) into (3.14)
ts 2 wL 1 ' Tsr[()()][()()] Nxr t N r xr t dxdydz 2 t 00 s 2 (3.15) t s t 2 p wL 1 ' vr[()()][()()]NtNtdxdydzxr r xr 2 ts 00 2
which is rewritten as
1 TtMMtrr()' () 2 sp
ts 2 wL ' M ss N r()xx N r () dxdydz (3.16) t 00 s 2 t s t 2 p wL ' M pv N r()xx N r () dxdydz ts 00 2
where Ms and M p are the mass matrices of the beam and the PVDF where the densities are
assumed constant. Evaluating the inner integral common to both mass matrices and recalling that Nr ()x is a matrix of orthogonal mode shapes reveals that it will be diagonal and hence symmetric.
31
NN11()xx () NN 12 () xx () NN 1 () xN () x LLNN()xx () NN () xx () NN () x () x ' 21 22 2 N NNdxrr()xx () dx (3.17) 00 NNNN()xx12 () NN () xx () NN NN () x () x
3.4.2 Potential Energy
The potential energy of the system includes contributions from the beam and PVDF. The potential energy of the beam comes from its strain energy; energy stored by a system
undergoing a deformation. The potential energy of the PVDF, per unit volume, is given by the
derived Helmholtz free energy function, equation (2.22), whose first term is seen to be the
strain energy of the piezoelectric.
VVV sp
111' DS (3.18) VcdVs SSss ,'''' p S ch SS D D D d p 222 sp
As shown in appendix 3, the strain vector can be related to the displacement vector through a
differential operator. This operator will differ for the cases of transverse and axial
displacements, detailed in section 3.7 , and for now will be defined as generally as Lu such that
Su Lu (3.19)
Per (3.8) and following standard notation [3], (3.19) can be written in terms of generalized
coordinates as
Sxrxr LNur()() t B r ()() t (3.20)
32
such that the potential energy of the beam becomes
1 VtKt rr()' () ss2 (3.21) ' KBcBds rsrs()xx () s
where Ks is the symmetric stiffness matrix of the beam. Similarly, the electric displacement vector D , which has units of charge per area, can be written in terms of generalized
coordinates as
Dxq Btq ()() (3.22)
where q()t is the vector of q j terms so that the potential energy of the PVDF becomes
111 VtKttttCtrrrqqq()''Ds () () () () () pp22 p DD' KBcBdpr ()xx rp () p '' (3.23) BhBdrqp()xx () p ss1 ' CBBdpq ()xx qp () p
D where K p is the symmetric stiffness matrix of the piezoelectric, is the nonsymmetric
s1 piezoelectric coupling matrix, and Cp a symmetric permittivity matrix. The potential energy of
the system is thus
11 11 VtKttKttttCtrr()''' () r ()Ds rrqq () () () () q () (3.24) 22sp 2 p
33
and the Lagrangian of the system
11 11 LtMMttKKttttCtrrrrrqqq()'''' () ()Ds () () () () () (3.25) 22sp sp 2 p
3.5 Variations
To derive the governing equations of the system using the extended Hamilton’s principle it is necessary to evaluate the variation of the Lagrangian of the system, equation (3.25), as well as the variation of external non conservative work in accordance with section 3.1
3.5.1 Variation of the Lagrangian
The variation of the Lagrangian is performed with respect to each generalized coordinate
NNMLL L Lrtrtqtii() () j () (3.26) ii11rtii() rt () j 1 q j () t
which can be rewritten in vector matrix form as
LL L Ltrrq()'' () t () t ' (3.27) rrq()tt'' () () t '
Evaluating the first term of (3.27)
' ''L 11 ' rr()t () t MMsprr () t () tMM sp r()t ' 22
' 11' rr()tMMtMMt sp () spr () (3.28) 22 ' rr()tMsp M () t
34
per Ms and M p being symmetric. Similarly
''L D rr()ttKKtt' ()sp rq () () (3.29) r()t
and
L 1 qqrq()tttCt''' () ()s () (3.30) q()t ' p
D s1 utilizing symmetry in Ks , K p , and Cp . The final expression for the variation of the Lagrangian is thus
1 L rrrrqqrq()tM'' M () t () t K KDs () t () t () t '' () tC () t(3.31) sp sp p
3.5.2 Variation of Work
The external non conservative work done on the system comes from the contraction of the
forearm muscles and hence is a mechanical force. From the principle of virtual work for the k th applied (point) force
Wtkk fux() kk ( ,) t (3.32)
which, considering all applied forces and incorporating (3.8), can be rewritten in vector matrix
form as
35
N f '' ' WtNrxfrxF() rkk ( ) () t () tBt f ( ) () (3.33) k1
3.6 Evaluating the Extended Hamilton’s Principle
Plugging equations (3.31) and (3.33) into (3.2) yields (3.34) which, upon solving, will yield the
desired Euler‐Lagrange equations of the system.
rrrrq()tM'' M () t () t K KD () t () t t2 sp sp dt 0 (3.34) ''s1 ' t1 qr()ttCttBt () q () r () ( xF ) () pf
Before (3.34) can be solved the integral must be exclusively in terms of variations of generalized coordinates, not variations of generalized velocities. Splitting up the integral and looking at the first term
t2 ' rr()tMsp M () tdt (3.35) t1
integration by parts can be used accordingly
bb udv uvb vdu a aa ' uMMsprr() t , dvtdt () (3.36) ' du Msp Mrr() t , v () t
such that
36
t2 ' rr()tMsp M () tdt t 1 (3.37) t t2 ''2 rrrr()tMsp M () t () tM sp M () tdt t1 t1
In accordance with Hamilton’s principle there is no variation at the end conditions so the first
term in (3.37) goes to zero. Now (3.34) can be rewritten exclusively in terms of variations of
generalized coordinates as
rrrqxF()tMMtKKt' () D () () tBt ( ) () t2 sp sp f dt 0 (3.38) ''s1 t1 qr()ttCt () q () p
A consequence of all generalized coordinates being independent is that the variations of the generalized coordinates are arbitrary [2]. Hence to satisfy the equality each term enclosed in square brackets in (3.38) must be equal to zero resulting in
D M spMtKKtrrqxF() sp () () tB f ( ) () t (3.39) ' s1 rq()tCp () t 0
Through substitution (3.39) results in a second order ordinary differential equation (ODE) and a
linear equation
Ds' M spMtKKCrrxF() sp p () tB f ( ) () t (3.40) s ' Cttprq() ()
37
Equation (3.40) contains the undamped equations of motion of the system. As all real system
exhibit damping, a damping matrix Ds is added whose values improve agreement between
experimental and simulation results yielding the damped equations of motion of the system [3].
Ds' M spMtDtKKCrr() s () sp p r () tB f ( xF ) () t (3.41) s ' Cttprq() ()
The top line of (3.41) can be reduced to a first order ODE through the declaration of a new set
of state variables and solved using Matlab’s LTI solver, as shown in appendices 5 and 6, yielding
the system displacement ux(,)t and the charge output q()t .
3.7 Strain Vector Differential Operator
As mentioned back in section 3.4.2, the differential operator Lu that relates the strain tensor in
vector form S and the displacement vector u differs when considering transverse and axial
motions.
3.7.1 Transverse Motion
As shown in appendix 3, the transverse loading in the positive 3 direction causes the beam to
bend and displacement in the 1 direction is given by
ux13 (3.42)
where is the slope of each bent differential element being considered with reference to the
23 plane and defined as
38
u 3 (3.43) x1
Plugging this into (3.42) yields
u3 ux13 (3.44) x1
The first element in the strain vector S by definition is [4]
2 u1 u3 Sxxu11 3 32 3 (3.45) xxx111x1
Assuming each differential element of the beam is under uniaxial stress, applying Hooke’s law
allows expressions for S22 and S33 [3]
2 SSxu 22 12 11 12 3x 2 3 1 (3.46) 2 SSxu33 13 11 13 32 3 x1
Finally, the strains can be related to the displacements using a differential operator as
39
2 00 x3 2 x1 2 0012x 3 2 x1 u1 SuL 2 u (3.47) u 2 0013x 3 2 x u3 1 00 0 00 0 00 0
3.7.2 Axial Motion
Strains in the 1 direction due to uniaxial stresses applied in the 1 direction are, again, by
definition
u1 S11 (3.48) x1
and from applying Hooke’s Law S22 and S33 are
u SS 1 22 12 11 12 x 1 (3.49) u1 SS33 13 11 13 x1 such that the strains can be related to the displacements using a differential operator as
40
00 x 1 00 12 x u 1 1 SuL u (3.50) u 2 13 00 x 1 u3 000 000 000
3.8 PVDF Electrode Spatial Shading
As previously mentioned, PVDF can be shaped or its electrodes spatially shaded (etched), as was done in the case being considered, such that it is only sensitive to targeted modes of
vibration. More generally but along the same vein, the PVDF electrodes can be shaded such
that it has increased or decreased sensitivity to specific deformations. As charge is only
collected from areas of the piezoelectric where the electrodes overlap it is only necessary to
shade one side of the PVDF.
Elaborating on equation (3.22), the electric displacement vector for a single generic piezoelectric element is given by
B ()x D1 q1 DxDB() qt () (3.51) 2 q2 D3 B ()x q3
41
For the system being considered the electrode is in the 1‐2 plane such that charge is only
collected in the 3 direction and all shape functions are a function of x1only. Therefore, (3.51) reduces to
00 D 00()qt (3.52) DBx() 31q3 where B ()x is required to have the units of area1 . The electrode was shaded based on q3 1 insight gained for the case of transverse displacements.
3.8.1 Transverse Displacements
For transverse displacements, plugging (3.52) into the second line of (3.41) and using the differential operator given by (3.47) reveals in (3.53) that the output charge is proportional to the weighted average of the curvature common to the beam and piezo where the weighting function is B ()x q3 1
L 2 qt() B ( x ) u ( x ,) tdx (3.53) q3 13112 0 x1
Further, this weighting function B ()x describes the profile of the electrode as a function of q3 1
position coordinate x1. Integrating (3.53) by parts twice yields
LL L 2 qt() Bxqqq ()13131 uxt (,) uxt (,) Bx () 1 uxt 31 (,) Bxdx () 11 (3.54) 333xxx 2 11000 1 42
From the physics of the configuration the deflection at each of the ends, x1 equal to 0 and L , is zero and hence the second term in (3.54) is zero. The third term tells us that if B ()x is not q3 1
at least of order two then it too will be equal to zero. By making B ()x a parabolic function q3 1
the first term in (3.54) becomes zero. Further, the second derivative of a parabolic function is a
constant so the output charge reduces to simply being proportional to the transverse
displacement profile common to the beam and PVDF.
L qt() u31 ( x ,) tdx 1 (3.55) 0
In summary, by selecting a parabolic sensor profile the output charge is proportional to the
area under the transverse displacement curve which directly correlates to the level and location of muscle activation. As such, the center of the PVDF is located directly over the posterior
forearm muscles.
The upper half of the parabolic sensor profile in question is given by the term enclosed in
brackets in equation (3.56)
2 x Lx 2 11 Bx() 2L (3.56) q3 1 WL
The full electrode width is given by mirroring the bracketed expression about the x1 axis, as
shown in Figure 3.2, and hence the factor of two in equation (3.56).
43
Figure 3.2 PVDF shaded profile
3.8.2 Axial Displacements
For axial displacements, plugging (3.52) into the second line of (3.41) and using the differential
operator given by (3.50) reveals that the output charge is proportional to the weighted average
of the strain along the axial direction where the weighting function B ()x is as defined q3 1
previously in (3.56).
L qt() B ( x ) u ( x ,) tdx (3.57) q3 1111 0 x1
Theoretically, for axial displacements the profile of B ()x proves non ideal as B ()x is q3 1 q3 1
maximized in the middle of the sensor where strains, assuming symmetric loading, are
nonexistent. Further, the profile width at the edges of the sensor, where strains are a
maximum, are zero. However, through experimental validation, this does not prove to be an issue.
44
3.9 Force Modeling
Models of the input forces for transverse and axial loading used in simulations with the above
developed equations are detailed below. In each instance, the input force models attempt to
replicate the general observed spatial temporal behavior of the physical set of activating
muscles and with a magnitude such that the system model achieves good general agreement with collected data as shown in the next section.
3.9.1 Transverse Inputs
Transverse inputs, representing the effect extensor muscles being activated has on the PVDF, were modeled using a traveling logistic curve composed of two logistic functions. A single example of the logistic curve used is shown in Figure 3.3. The traveling effect is shown in Figure
3.4, along with arrows to indicate direction of travel, where each individual curve gives force as
a function of position at that particular instant in time. The family of curves increase at a
constant position until it reaches its peak amplitude. They, then travel at this peak amplitude from left to right until, at its rightmost point, they decrease.
45
Figure 3.3 Transverse input‐logistic function
Figure 3.4 Traveling logistic wave
46
3.9.2 Axial Inputs
Axial inputs, representing the effect flexor muscles being activated have on the PVDF, were
modeled using a Gaussian curve as illustrated in Figure 3.5. In the axial case, equal and opposite forces were applied to each end of the PVDF by the tension in the compression sleeve where
the same force was felt along the whole width of the edge.
Figure 3.5 Axial input‐Gaussian function
3.10 Simulation Results
The voltage output from a piezoelectric is given by
dq ViRR (3.58) out dt
47
Given that charge is linearly related to strain, equation (3.58) states that the output voltage of a
piezoelectric is dependent on the change in strain or rather the strain rate. Figure 3.6 illustrates
this concept for a performed “fist” gesture. While the gesture is being performed, going from a
neutral to fist position, the PVDF is being deformed in such a way that it outputs a positive voltage. While the fist gesture is being held the PVDF is not undergoing any deformations and,
due to leakage and no new voltage being generated, drops to zero. When the gesture is
reversed, going from the fist back to neutral position, the PVDF outputs a negative voltage.
Finally, when the gesture is over and the hand is back in a neutral position, and hence the PVDF
is not undergoing any deformations and again due to leakage, the voltage returns to zero.
Figure 3.6 Gesture voltage profile
48
Simulation versus collected data for fist, flex, and extend gestures are shown in Figure 3.7 and
right and left gestures in Figure 3.8. In the fist and flex simulation the dominant input force was
due to axial effects. For the extend gesture the first part of the response was due to transverse
effects and the second part axial effects. Code generating simulation results associated with transverse and axial effects is given in appendices 5 and 6. The right gesture voltage output was in response to axial effects and the left gesture split with the first half of the output being in
response to transverse effects and the second in response to axial effects. General deviations
between the collected and simulated profiles are attributed to not having a more accurate
model of the input forces and only considering linear piezoelectric effects.
Figure 3.7. Comparison of experimental and simulated output voltage for fist, flex, and extend gestures
49
Figure 3.8. Comparison of experimental and simulated output voltage for right and left gestures
References
[1] R. Weinstock, Calculus of variations, with applications to physics and engineering. New York: Dover Publications, 1974. [2] L. Meirovitch, Principles and techniques of vibrations. Upper Saddle River, N.J: Prentice Hall, 1997. [3] D. Leo, Engineering Analysis of Smart Material Systems, 1st ed. New Jersey: John Wiley & Sons, Inc., 2007. [4] E. Popov, Introduction to Mechanics of Solids, 9th ed. New Jersey: Prentice‐Hall, INC., 1968.
50
CHAPTER 4: Artificial Neural Networks
An artificial neural network, or simply neural network, is a collection of neurons, or nodes,
which each receive weighted inputs, process the summation of these inputs, and pass along the
result as inputs to other nodes. A single node is shown in Figure 4.1 where g performs the
aforementioned processing and is termed the activation function, x0 is the bias input weighted
by w0 , x1 through xn are inputs weighted by w1 through wn accordingly and y is the node’s
output. The activation function for continuous nodes, as opposed to discrete nodes, is chosen
from the class of sigmoid functions [1].
Figure 4.1 Single continuous node
The branch of neural networks is a machine learning algorithm in which a network of nodes can
be taught, among other things, pattern classification, pattern restoration, statistical modeling,
or feature detection. One key attribute of a neural network is its ability to generalize; that is being able to map similar inputs to the same output.
The classification paradigm is such that there is a fixed set of output patterns into which the input patterns are to be classified. During network training the system is passed input/output
51
pattern pairs with the goal of learning to correctly classify the input pattern so that in the
future when a particular input pattern, or a slight variation of it, is passed to the network, the
network will classify it properly [2]. As it is being used, the input patterns to the network
described in the following is a vector of voltages generated by the PVDF while a gesture was being performed.
Neural networks have roots in the 1940’s with the McCulloch‐Pitts neuron [3] and the 1960’s with the notions of the perceptron by Rosenblatt [4] and the Adaline by Widrow and Hoff [5].
However, these models as implemented all failed to be trainable in distinguishing between
nonlinearly separable classes. The solution to this problem is largely credited to Rumelhart and
his Parallel‐Distributed‐Processing research group with the development of the generalized delta learning method; also known as backpropagation [2]. As previously mentioned, neural
networks have a history of being used for gesture classification by, for example, either trying to
recognize captured images of gestures [6] or by processing raw sensor data from a handheld video controller [7] or glove based system [8],[9].
4.1 Network Architecture
A time‐delay neural network, shown in Figure 4.2, was used to map incoming temporal data into an output vector allowing for pattern recognition or classification [10]. The network was comprised of two layers of nodes; an output layer with nodes gv() and a hidden layer with nodes gu(). The logistic function (4.1) was used as each node’s activation function.
52
Figure 4.2 Multilayer network
1 gu1 eu (4.1)
The number of network inputs and nodes differed depending on the set of gestures it was employed to classify according to Table 4.1.
Table 4.1 Neural network size parameters Set of Gestures I J K dt [ms] Fist/Extend/Flex 369 25 3 2 Right/Left 133 3 2 5
53
4.2 Feedforward Operation
In forward mode, also referred to as the network being in recall, the network is tasked with classifying, or mapping, the inputs as either an appropriate gesture or noise. In general the network behaves as follows. Each input to the network, each Xi and the input bias ‐1, is
weighted and fed into each hidden node where they are summed, denoted as u j , and scaled
by the logistic function. Each hidden node then outputs this value denoted by Yj . These outputs, and the hidden bias ‐1, are each weighted and fed into each output node where they are again summed, denoted as v j , and scaled by the logistic function. The output of each node
in the output layer Zk is an element of the output of the network; these outputs are used to
create the network’s output vector. Specifically, each input to a node is weighted by a unique
corresponding value; input Xi to hidden node j is weighted by aij and likewise input Yj to
output node Zk is weighted by bjk .
The output of hidden node j can hence be expressed as
Ygujj I 1 (4.2) uXajiij i1
where the I 1 th input is the constant bias 1. From (4.2), if the inputs are specified as a vector
X of dimension I 1and the weights are arranged as matrix A of dimension I 1 by J then
T Ygugjj() (aX j ) (4.3)
54
th where a j is the j column of A. Similarly, the network output vector Z is comprised of
elements
T Zgvgkk bY k (4.4)
where Y is the augmented vector of dimension J 1 comprised of the J hidden node outputs
th and the constant bias 1 and bk is the k column of weight matrix B which has dimensions
J 1 by K .
Let the first I elements of input vector X be denoted by vector X . The elements of X are the last I voltage values passed from the microcontroller as depicted in Figure 4.3. The I th element of X is the most recently passed voltage. When a new voltage is passed, all the
voltages in X are “bumped up” in position such that the voltage occupying the I th position is
moved to the I 1 th position, the voltage in the 1st position is removed from X , and the new
voltage occupies the I th position. This new input vector is then passed through the network.
Upon initialization X is set to the zero vector. The time between voltage reading is denoted as
dt and the time spanned by X is Idt . A vector matrix depiction of the network is shown in
Figure 4.4 incorporating X .
55
Figure 4.3 Graphical depiction of temporal input vector X
Figure 4.4 Vector matrix network depiction
At each dt a new output vector Z is calculated. Per network training, discussed in the next
section, gestures are determined based on the elements of Z .
56
4.3 Training
The network was trained using the generalized delta rule. The generalized delta rule is a
learning process which is based on associated pairs of input and output patterns where the
output pattern is the desired output for the paired input [2]. The input is fed into the system
and an output calculated. If this output matches the desired output then no learning is
required. Otherwise, the weights of the network are adjusted through an iterative process to
reduce the error between the calculated and desired outputs to zero
The inputs to the network designed above were vectors of collected temporal voltage values ordered as X and augmented with a 1 to yield a true input vector X . Each input vector spans
a single gesture. Each gesture requires a different amount of time to complete. All input vectors are required to have the same number of elements so gestures of shorter duration were augmented with the necessary number of zeros to make them dimensionally compliant. The corresponding output vectors for each gesture are shown in Table 4.2 and Table 4.3. The
network output for an input voltage of zero was 0.5.
Table 4.2 Gesture set 1 output vectors
Gesture Z1 Z2 Z3 Fist 0.9 0.1 0.1 Extension 0.1 0.9 0.1 Flex 0.1 0.1 0.9
57
Table 4.3 Gesture set 2 output vectors
Gesture Z1 Z2 Right 0.9 0.1 Left 0.1 0.9
The voltage data for gesture set 1 was collected using a dSPACE system (dS1103) while the
voltage data for gesture set 2 was collected using the developed Arduino circuit detailed in
Chapter 6 and computer program detailed in appendix 12 which enabled the Arduino to act as a
DAQ system.
4.3.1 Error Derivatives
The goal of the generalized delta rule, also known as backpropagation [1], is to minimize the
mean square error, defined in (4.5), over all input/output pairs, or training patterns. The mean
square error, Ems , is the normalized error in all outputs, index k , over all training patterns, index n , where znk is the calculated output and tnk is the desired output.
1 NK 2 Zt 2 kn kn E nk11 (4.5) ms NK
The error is a surface in weight space and is minimized by changing the weights of the []A and
[]B matrices using the iterative method of steepest descent defined in (4.6) where wm is the
new weight ( aij or bjk ) calculated after epoch (iteration) m to be used in the next epoch, wm
58
is the weight adjustment which is a function of the gradient of the error E , and wm1 is the
weight used during the current epoch.
ww w mm1 m (4.6) wwEmm
Considering first just the output layer of the network, deviation from optimality in weight bjk ,
defined as jk , only influences output Zk as shown in Figure 4.5. Therefore, the network error
from non‐optimal bjk , for each training pattern, is
1 2 EZt (4.7) 2 kk
Figure 4.5 Propagation of error in output node weight
The sensitivity of E to adjustments in bjk is, per the chain rule
EE Z v kk (4.8) bZvbjkkkjk
59
where the terms on the right side of the equation are called the error derivatives [1]. Noting that
Zgvkk () J 1 (4.9) vbYbYbYbYk jkj 11 k ... jkj ... (1)(1) J k J j1
the error derivatives are
E 1 2 Ztkk Zt kk ZZkk2 Z 1 11 k vk gv()kkk 1 e 1 Z 1 Z (4.10) vv v 11eevvkk kk k
vk Yj bjk
such that
E ZkkktZ1 ZY kj (4.11) bjk
Considering now both the hidden layer and the output layer of the network, deviation from optimality in weight aij , defined as ij , influences all network outputs as shown in Figure 4.6.
60
Figure 4.6 Propagation of error in hidden node weight
Therefore, the network error from non‐optimal aij , for each training pattern, is
1 K 2 E Ztkk (4.12) 2 k1
and the sensitivity of E to adjustments in aij is, per the chain rule
EEYu j j (4.13) aYuaij j j ij
where the error derivatives are
EEK Zv kk YZvYjkkjk1 Y 1 j vk gu()kjj 1 e Y 1 Y (4.14) uujk v k
u j Xi aij
61
and
vk bjk (4.15) Yj
4.3.2 Learning Law
The exact form of wm in the above section was left intentionally vague as it varies depending
on the exact form of the learning law. The adaptive learning rate, or delta‐bar‐delta, with
momentum, (4.16), was used per literature recommendations as it converges faster than pure
steepest descent [1]
wwmm 1 (1 ) ed mm NNE (4.16) dEmm n w nn11m n
where is the parameter associated with momentum, em is the adaptive learning rate
parameter, and dm is the sum over all training patterns of the derivative of the error with respect to the weight.
While momentum incorporates prior epoch weights into each new calculation em allows for a new step size each epoch based on the direction in which the error has been decreasing recently. Direction is defined as
fmm1 fd1 m (4.17)
62
where is a parameter controlling the notion of recently. Parameter em , calculated after each epoch, is dependent on parameters and and on the sign of the product of dfmm according
to
edfmmm1 ,0 em (4.18) edfmmm1 ,0
The values of parameters used are shown in Table 4.4 and the code associated with implementation in appendix 7.
Table 4.4 Learning parameters 0.2 0.1 0.5 0.7
4.4 Operation
In operational mode the network processes the input vector X and calculates the output
vector Z each dt using the weights found during training. The algorithm then analyzes the
output vector and assigns a scalar classification according to the pseudo code in (4.19) or (4.20) depending on the gesture set being used; after each nonzero classification X is reinitialized. A scalar classification of 0 corresponds with noise or simply no gesture. Error bars of 0.05 were
incorporated to assist with generalization.
63
if Z12 between 0.9-0.05 and 0.9+0.05 and Z between 0.1-0.05
and 0.1+0.05 and Z3 between 0.1-0.05 and 0.1+0.05 then Classification = 1
esle if Z12 between 0.1-0.05 and 0.1+0.05 and Z between 0.9-0.05 and 0.9+0.05 and Z between 0.1-0.05 and 0.1+0.05 3 (4.19) then Classification = 2
esle if Z12 between 0.1-0.05 and 0.1+0.05 and Z between 0.1-0.05
and 0.1+0.05 and Z3 between 0.9-0.05 and 0.9+0.05 then Classification = 3 else Classification = 0
if Z12 between 0.9-0.05 and 0.9+0.05 and Z between 0.1-0.05 and 0.1+0.05 then Classification = 1
else if Z12 between 0.1-0.05 and 0.1+0.05 and Z between 0.9-0.05 (4.20) and 0.9+0.05 then Classification = 2 else Classification = 0
Figure 4.7 illustrates the entire gesture classification process for gesture set 2. For the time
span under consideration, at each dt of 5ms a voltage was collected, outputs Z1 and Z2 computed, and a classification determined. At time 0.292 seconds a gesture was started and Z1 and Z2 start to deviate from their steady state value of 0.5. At time 0.98 seconds Z1 crosses
the lower threshold and Z2 crosses the upper threshold and a classification of 2 is subsequently passed indicating (correctly) that a “left” gesture was performed. Once a nonzero classification was made, X was reset to the zero vector to prevent any double classifications.
64
Figure 4.7 Input voltage, output values, and classification
References
[1] M. Smith, Neural networks for statistical modeling. New York: Van Nostrand Reinhold, 1993. [2] D. E. Rumelhart and University of California, San Diego, Parallel distributed processing: explorations in the microstructure of cognition. Cambridge, Mass: MIT Press, 1986. [3] W. S. McCulloch, Embodiments of mind. Cambridge, Mass: M.I.T. Press, 1988. [4] F. Rosenblatt, Principles of Neurodynamics: Perceptrons and the Theory of Brain Mechanisms. Washington, D.C.: Spartan Books, 1961. [5] B. Widrow and M. E. Hoff, “Adaptive Switching Circuits,” in Neurocomputing: Foundations of Research, J. A. Anderson and E. Rosenfeld, Eds. Cambridge, MA, USA: MIT Press, 1988, pp. 123–134. [6] G. R. S. Murthy and R. S. Jadon, “Hand gesture recognition using neural networks,” in Advance Computing Conference (IACC), 2010 IEEE 2nd International, 2010, pp. 134–138. [7] F. Arce and J. M. G. Valdez, “Accelerometer‐Based Hand Gesture Recognition Using Artificial Neural Networks,” in Soft Computing for Intelligent Control and Mobile Robotics, O. Castillo, J. Kacprzyk, and W. Pedrycz, Eds. Springer Berlin Heidelberg, 2011, pp. 67–77. [8] P. A. Harling, “Gesture Input using Neural Networks,” University of York, Heslington, UK, 1993. [9] F. Parvini, D. McLeod, C. Shahabi, B. Navai, B. Zali, and S. Ghandeharizadeh, “An Approach to Glove‐Based Gesture Recognition,” in Human‐Computer Interaction. Novel Interaction Methods and Techniques, J. A. Jacko, Ed. Springer Berlin Heidelberg, 2009, pp. 236–245. [10] J. M. Zurada, Introduction to artificial neural systems. St. Paul: West, 1992.
65
CHAPTER 5: Neural Network Implementation Results
Three separate electronic setups were used to test the ability of the controller to reliably classify all gestures from each gesture set. Further, as one of the key properties of a neural network is its ability to generalize, the robustness of the network was also examined. The first setup relied on a commercial DAQ system, specifically a dSPACE ds1103 and associated
ControlDesk software. The second was designed to be a true wearable and used a microcontroller based chip, specifically an Arduino Pro Mini. A new sleeve and PVDF sensor assembly was also designed for the microcontroller based setup, detailed in the following chapter, improving on discovered flaws. As a final prototype, the new sleeve was kept but the
Arduino Pro Mini was replaced with custom printed circuit board (PCB) based on the
ATMega328p microcontroller (MCU). As a first step with each setup, new neural network training vectors were collected and the network trained.
5.1 dSPACE ds1103
The first setup used a dSPACE ds1103 board and ControlDesk software. Using a commercial system allowed the controller to be tested as “hardware‐in‐the‐loop” so that the particulars of implementing the neural network, such as code debugging and required sampling rate, could be developed. It would also provide a benchmark for future developed systems. Only gestures from gesture set 1 were tested using this setup.
66
5.1.1 dSPACE Setup
The setup was comprised of the compression sleeve/PVDF controller, the dSPACE system, and a
PC with ControlDesk software. The leads of the PVDF were connected to the DAQ system which discretely sampled the continuously supplied voltage from the PVDF. The discrete signal was in turn fed to the PC, which, using the ControlDesk software, performed the real‐time
feedforward neural network calculations and classification and displayed the results. Figure 5.1
shows the signal flow process as a block diagram while Figure 5.2 shows the physical components used. In order to implement the neural network, an S‐Function block was used in
the Simulink block diagram used to generate the ControlDesk runtime code and a C MEX S‐
Function was written as detailed in appendix 8. The time between voltage readings was 2ms
and the neural network size parameters used are shown in Table 5.1.
Figure 5.1. Experimental setup block diagram
Figure 5.2. Experimental setup
67
Table 5.1 Gesture set 1 neural network size Index I J K Value 369 25 3
5.1.2 ds1103 Results
The controller and neural network combination was tested for reliability and robustness.
Statistics for the reliability test are shown in Table 5.2. Figure 5.3 shows a sample of the various
inputs (two fists, two extends, and four flexes) along with correct classifications as defined
above in chapter 4 and repeated in Table 5.3.
Table 5.2 dSPACE neural network reliability statistics Gesture Accuracy Fist 97.5% Extend 100% Flex 54.3%
Table 5.3 Gesture set 1 classifications Gesture Output Classification Fist 1 Extend 2 Flex 3
From Table 5.2 it is apparent that the most unreliable gesture in terms of classification accuracy was the flex. When a flex was not properly classified it was misclassified as a fist. The reason for this misclassification has to do with the similarity in the two gesture voltage profiles which can
be seen in Figure 5.3. The similarity in the two gesture voltage profiles stems directly from the
68
overlap of muscles used in each gesture, as shown in Table 1.1 (combination of tense and
release), and the sensor’s sensitivity to these muscles. Also, noting that PVDF is a rate based
sensor, if the flex gesture was performed too fast the amplitude of the voltage output curve would approach that of the fist. Given that these two voltage outputs already had a similar
shape the similar amplitude was enough to generate a misclassification.
Figure 5.3. Neural network input signal (left axis) and output classification (right axis)
Another issue associated with performing a gesture faster or slower than the rate at which the
network was trained is that calculated weights no longer match up with the specific profile
characteristics that training established. From Figure 4.3 and Figure 4.4 it is seen that for each
hidden node the logistic function operates on the summation of the product of weights and values that occupy fixed vector coordinates. By performing a gesture significantly faster or slower than the speed at which the network was trained the relationship of signal
characteristics within the input vector change (e.g. peak to valley time) but the weights do not
69
and thus the product of weight and vector value can change and appreciably alter the network output.
A key characteristic of neural networks is their ability to generalize, that is, to recognize similar inputs to those with which it was trained and apply correct classification. Figure 5.4 through
Figure 5.6 show examples of the variability in input voltage signals from the PVDF for which the network was able to return the correct classification.
Figure 5.4. Variability in fist gesture
70
Figure 5.5. Variability in flex gesture
Figure 5.6. Variability in flex gesture
71
5.2 Microcontroller Based Systems
Once it was proven that the neural network architecture, a simple feedforward structure with a single hidden layer, was able to perform the desired classifications, the dSPACE system was replaced first with an Arduino Pro Mini and custom software as a first step towards making the
controller truly wearable and second with a custom PCB based on the ATMega328p MCU to
shrink the electronic footprint even further. Only gestures from gesture set 2, a right and left,
were tested using these setups.
5.2.1 Arduino Setup
The second setup, in addition to having a new version of the compression sleeve and PVDF sensor assembly, used an Arduino Pro Mini powered by an ATmega328p microcontroller, a
Bluetooth transmitter and receiver, and custom software. Details concerning controller
fabrication, the configuration of the microcontroller, and the software are in chapter 6. The
signal flow, diagramed in Figure 5.7, is similar to that of the dSPACE system. The voltage from
the PVDF is sampled discretely by the Arduino’s analog input pin. The read value is then passed,
via Bluetooth, to the target application whereupon the value is passed through the neural
network algorithm and the appropriate action is performed. Here, as detailed in Chapter 6, the
target application is not restricted to a single software graphics window, as was the case for the dSPACE setup, but rather any device that can receive the Bluetooth signal. The new controller,
Arduino, and associated components are shown in Figure 5.8. The time between voltage
readings was 5ms and the neural network size parameters used are shown in Table 5.4.
72
Figure 5.7. Controller operational block diagram
Figure 5.8. Controller prototype as worn on right forearm: compression sleeve with adhered (spatially shaded) PVDF and hardware.
Table 5.4 Gesture set 2 neural network size for Arduino Pro Mini Index I J K Value 133 3 2
5.2.2 Arduino Results
The new controller and neural network combination was tested for reliability and robustness in
the same manner as previously discussed. Performance characteristics of the device that were
of paramount importance were its accuracy, repeatability, and ability to generalize or apply the
correct classification for reasonable variations in the input signal caused by variations in the gesture. To test the performance the controller was put on, turned on, connected via Bluetooth to the developed Android application detailed in the following chapter, and then aligned.
73
Alignment entailed testing each gesture to make sure the application responded correctly. If the application did respond correctly then the testing continued, otherwise the controller was
rotated about the arm and alignment was rechecked. Once aligned, a test sequence was performed. Upon completion of a test sequence the controller was turned off and taken off.
This process was repeated for three test sequences: 50 right gestures followed by 50 left gestures, 10 right gestures followed by 10 left gestures, repeated five times, and 50 alternating right then left gestures. The controller, for each test sequence, performed with 100% accuracy.
Figure 5.9 and Figure 5.10 serve to illustrating tolerable signal variance with the top subplot showing the input voltage signal and classification and the bottom subplot showing the
calculated output values according to classifications defined previously in chapter 4 and
repeated in Table 5.5.
Table 5.5 Gesture set 2 classifications Gesture Output Classification Right 1 Left 2
74
Figure 5.9. Signal variance: “Right”, “Left”, “Right”, “Right”, “Left”, “Right”, “Left”, “Left”
Figure 5.10. Signal variance: “Left”, “Left”, “Right”, “Right”, “Right”, “Left” “Right”, “Right”, “Left”, “Left”
5.2.3 ATMega328p PCB Setup
The third setup does away with the pinned out footprint of the Arduino Pro Mini and instead
uses a custom PCB powered by the same ATMega328p MCU that is used by the Mini and uses a
Bluetooth module in an XBee configuration for wireless communication. Details concerning
75
fabrication are discussed in chapter 6. The signal flow for this configuration is the same as that
shown in Figure 5.7 while the actual as worn device is shown in Figure. The time between
voltage readings was approximately 16ms and the neural network size parameters used are
shown in Table 5.6.
Figure 5.11. Forearm gesture controller. (a) PCB based electronics with Bluetooth module. (b) Spatially shaded PVDF senor as curved when sleeve is being worn. (c) Device as worn with called out location of electronics and PVDF.
Table 5.6. Gesture set 2 neural network size for custom PCB Index I J K Value 36 3 2
5.2.4 ATMega328p PCB Results
Again, performance characteristics of the device that were of paramount importance were its
accuracy, repeatability, and ability to generalize or apply the correct classification for
76
reasonable variations in the input signal caused by variations in the gesture or, as it is a wearable and subject to shifting in position and orientation, variations in sensor location. To test the device’s performance, the controller was put on in its proper location and orientation, turned on, and connected via Bluetooth to a developed application detailed later in section
6.6.2 which displays real time voltage read by the MCU and an arrow indicating whether or not
a gesture has been recognized and, if so, which one. If a gesture was not recognized by the
application then no arrow is displayed. Once connected, a test sequence was performed. Upon
completion of a test sequence the controller was turned off and taken off. This process was
repeated for three test sequences: 50 right gestures followed by 50 left gestures, 10 right
gestures followed by 10 left gestures, repeated five times, and 50 alternating right then left
gestures. The controller, for each test sequence, performed with 100% accuracy. Reasons for this level of accuracy are attributed to the fact that smooth, easily repeatable gestures were chosen, as opposed to a violent and inconsistent gesture such as a snap, and the fact that the
voltage profiles generated by each gesture are so dissimilar which is directly attributable to the
fact that two entirely different sets of muscles are used, as shown in Table 1.1, and hence two
distinct loading patterns on the PVDF sensor.
In addition to being accurate while properly positioned, the device also performs accurately
while non‐optimally positioned. Whereas SEMG signals are known to be relatively sensitive to
sensor placement [1], the tested device continued to perform with 100% accuracy when, from
the wearer’s perspective, rotated up to 19 degrees clockwise (1.5 cm linear displacement), up
to 50 degrees counterclockwise (4 cm linear displacement), and, under zero rotation, displaced
up to 1.5cm proximally and up to1 cm distally. These limitations are consistent with the
77
musculoskeletal anatomy of the forearm. For clockwise rotations greater than 19 degrees the
bulk of the sensor is situated over the ulna, not over any muscle mass, and hence the sensor output becomes diminished and distorted when compared to the output when properly situated. Similarly, for counterclockwise rotations greater than 50 degrees the bulk of the sensor is situated over muscles that are not used in performing either gesture. When displaced
beyond 1.5 cm proximally and 1 cm distally the sensor is no longer situated over the region of
maximum muscle bulge during contraction and hence, again, the output of the sensor is
diminished and distorted due to different levels of forces enacting on it.
References
[1] J. A. Mercer, N. Bezodis, D. DeLion, T. Zachry, and M. D. Rubley, “EMG sensor location: Does it influence the ability to detect differences in muscle contraction conditions?,” J. Electromyogr. Kines., vol. 16, pp. 198‐204, 2006.
78
CHAPTER 6: Sleeve Properties and Controller Design, Fabrication, and Implementation
The following discusses the sleeve mechanical properties used in the simulations of chapter 3
as well as the creation of the final Arduino based wearable controller, associated hardware, and
software applications.
6.1 Compression Sleeve Mechanical Properties
The wearable controller is comprised of PVDF adhered to a compression sleeve which is meant
to act as a second skin. The stress‐strain characteristics of the sleeve were determined using a
material testing system and adjusted as follows. Putting the sleeve on induces strain in the
sleeve as it stretches to conform to the wearer’s arm. What is to be measured is the change in
strain from this pseudo equilibrium state. From an energy perspective, strain energy is the area
under the stress‐strain curve. Strain introduced in the sleeve from being stretched to fit the
wearer’s arm effectively shifts the value of the initial strain, ε0, as shown in the left half of
Figure 6.1. New induced strain from muscle contraction is denoted as Δε. This new strain Δε adds strain energy to the system and, as is shown, when added to the prestrained state has a
larger energy contribution than when added to the initially unstrained state. This is accounted
for by creating an effective modulus, E’, by forcing the strain energy of two separate profiles to
be equal for the same Δε where one is prestrained and one is not as depicted graphically in the
right half of Figure 6.1 and mathematically in equation (6.1). The Δε in this case is approximated from measurements of the diameter of the forearm both in neutral and fully contracted states.
Note that in the fabrication process, discussed in section 6.3, the sleeve was first prestrained
79
and then the PVDF element was mounted such that there was zero prestrain in the PVDF
satisfying assumptions made in deriving the piezoelectric constitutive equations.
Figure 6.1 Left: Shift in initial strain and effect on strain energy from un‐stretched to stretched state. Right: Equality in strain energies for fixed Δε.
11' 0 22 E 11 22'EE (6.1) 0 22 1 2 EE0 E' 2 1 2 2
The stress‐strain characteristics of the compression sleeve were determined by putting a
sample of the sleeve, with properties shown in Table 6.1, through a tensile test using a MTS 858 table top testing system. The results of the test are shown in Figure 6.2 and Figure 6.3. The
expected operating region is identified by the parallel vertical bars in Figure 6.3 which falls
within the linear operating region of the material. The mechanical properties of the sleeve used
in all simulations are shown in Table 6.2.
Table 6.1 Compression sleeve test sample properties Original Length 30 mm Original Width 30 mm
Original Thickness ts 0.4 mm
80
Figure 6.2 Tensile test results: force versus change in length
Figure 6.3 Tensile test results: stress versus strain
81
Table 6.2. Mechanical properties of compression sleeve 3 ’ 3 Thickness [m] ρ [kg/m ] ν12 ν13, ν23 E [MPa] ε0 Δε E [MPa] ζ 0.0004 521.6 0.2 0.375 0.535 0.375 0.0182 22.6 0.2
6.2 PVDF Electrode Shading
The first step in the sleeve assembly was the shaping or spatial shading of the PVDF electrode.
The necessary length of the sensor was determined by measurement such that at the ends
there was no transverse deflection and hence the boundary conditions used in equation (3.54)
satisfied. The width was similarly determined by measurement to best ensure the validity of the
assumption that the whole width experiences the same deformation.
To fabricate the sensor an oversized rectangle of PVDF, with printed silver ink electrodes, was
cut from stock. Next, the ink was etched away a quarter inch around the outer boundary on both sides using a cotton swab dampened with 91% isopropyl alcohol. The PVDF was then
trimmed in the middle of the etched region. This process ensured that no fusing between the
top and bottom electrode layers occurred, implying infinite resistance between the electrodes
on each side, and was verified using a multimeter as shown in Figure 6.4.
3 Chosen to match experimental data 82
Figure 6.4 Testing PVDF for fusing
Finally, the desired sensor profile was etched on the top side with a remaining additional tab for lead attachment. The bottom electrode was left unetched except for the region under the
tab. The final shape of the sensor is shown in Figure 6.5 with dimensions called out in Table 6.3;
the thickness of the PVDF with electrodes was 52 μm. In Figure 6.5 the outer box is the boundary of the PVDF. The inner box marks the boundary of the bottom electrode layer. The solid black profile, matching the parabolic shape previously shown in Figure 3.2, is where the electrodes on the top and bottom overlap. The rectangle with diagonal lines is the additional tab for electrode attachment and indicates where only the top layer of electrode is present.
83
Figure 6.5 PVDF sensor geometry
Table 6.3 PVDF sensor geometry parameter values Parameter W L δ τ a b c Value(cm) 3 8.3 0.3 0.7 0.5 0.2 0.2
Holes were drilled through the PVDF, marked in Figure 6.5 as circles with an inscribed cross,
using a drill press (1/16th inch bit) while the PVDF was held under slight tension to prevent
tearing. Wide rolled eyelets and ring tongue terminals, along with insulating pieces of
fiberglass, were used to connect the leads (26 gauge hook‐up wire) to the electrodes using a top setting tool as shown in Figure 6.6. Female crimp pins were attached to the free ends of the
leads and inserted into a crimp connector housing, shown in Figure 6.7, to be used with a JST
style male connector on the printed circuit board (PCB) discussed in section 6.4.
84
Figure 6.6 PVDF lead attachment
Figure 6.7 Leads inserted into crimp connector housing
6.3 Sleeve Assembly
Prior to adhering the PVDF to the compression sleeve, the sleeve was stretched around a plastic cylinder of the same diameter as a representative 30 year old male. The PVDF was then adhered to the sleeve using fabric adhesive, as shown in Figure 6.8, with the goal of minimizing
the shear stresses in the adhesive layer while the sleeve was being worn.
85
Figure 6.8 Adhered PVDF element on compression sleeve stretched around cylinder
6.4 Circuitry and Hardware
For the first controller design, the one used with gesture set one, the leads of the sensor were
connected to a commercial tabletop DAQ system to record voltage values. Paramount among wearable device characteristics is portability and, as such, two mobile hardware systems were developed: the first based on the Arduino Pro Mini and the second a completely custom PCB based on the ATMega328p microcontroller.
6.4.1 Arduino Pro Mini System
The first mobile hardware system was based on an Arduino Pro Mini and assembled on a
custom through‐hole PCB. Elements of the system are an Arduino Pro Mini (8MHz, 3.3V), a
Bluefruit Bluetooth transmitter, 3.7 volt Polymer Lithium Ion battery pack, resistors, a switch,
and two connectors (one for the battery and one for the PVDF).
As the Arduino analog input ports are not able to read negative voltage values a voltage divider was incorporated into the PCB, where both resistor values, R1 in Figure 6.9(a), were 100 Ω to
bump up the reference voltage associated with the negative lead of the PVDF. When the PVDF
86
V A0 cc 1.65V was not outputting any voltage analog input pin measured a value of 2 or ; positive output from the PVDF added to this while negative output subtracted from it. The
1M resistor in parallel with the PVDF, R2 in Figure 6.9(a), is in place to protect the analog input pin. The final through‐hole PCB, created on a Quick Circuit 5000 system, is shown in
Figure 6.9 (b,c,d) both with and without components. Appendix 9 provides a parts list for the
sleeve and Arduino Pro Mini configuration.
Figure 6.9 (a) Controller circuit schematic (b,c,d) PCB top, bottom, and iso with soldered components
Raising the reference voltage had the negative effect of reducing the maximum output voltage
from the PVDF that the Arduino could sense. The maximum input voltage the microcontroller
can tolerate at any of its input pins is Vcc or 3.3V in this case. Bumping the reference from zero
Vcc Vcc to 2 subsequently limited the max amount of voltage that could be read also to 2 . As none of the gestures performed generated these level of voltages, as seen in the plots of chapters 3 and 5, this was a non‐issue.
The microcontroller on the Arduino has a 10 bit analog to digital converter (ADC) and the output from the analog pins subsequently ranges from 0 to 1023. These reading were scaled
87
back to voltages and adjusted, in accordance with equation (6.2) where again Vcc is 3.3V and
x is the output value from the Arduino analog pin A0 such that when zero voltage was output
from the PVDF Vx would be 0V .
VV Vxcc x cc (6.2) 1023 2
6.4.2 ATMega328p PCB System
Following the same necessary design principles of the previous section, the last hardware iteration was to replace the through‐hole PCB with a surface mount (SM) PCB and, in doing so, replace the pinned out Arduino Pro Mini with just the ATMega328p microcontroller. Also, the
through‐hole Bluetooth transmitter used previously was replaced with a through‐hole version that had an XBee pinout. This transmitter was held in place by SM female headers such that the
XBee was stacked on top of all other necessary electronics. The transmitter had a strict 3.3V supply limitation so a linear voltage regulator (LVR) was required. The system architecture is
shown in Figure 6.10 and a detailed parts list and full circuit diagram are given in Appendix 10
and 11 respectively. The positive lead from the PVDF is connected to the analog to digital
converter (ADC) pin, 10‐bit resolution, of an the MCU. The board is again powered by a 3.7 V
(Vcc) lithium polymer (LiPo) battery. The negative lead of the PVDF is once again connected to a
level shifting voltage divider to raise the reference voltage from 0V to Vcc/2. The serial
transmitting pin (TX) of the MCU is connected to the serial receiving pin (RX) of the Bluetooth
transmitter.
88
Figure 6.10. Overview of PCB system architecture
6.5 Data Collection and Verification
When using a dSPACE system one is able to select the rate at which the data is sampled through the associated software. When using the Arduino Pro Mini, the sampling rate needs to be set in code by clearing and setting the appropriate register bits [1]. To be able to get a specific
sampling rate of 5ms the microcontroller’s timer, specifically timer0 in the datasheet, had to tick such that 5ms would be some multiple of ticks. This way, once a certain number of timer
ticks occurred a sample would be taken. The default setting on the microcontroller was that the
timer ticks once every 64 clock cycles. For a microcontroller with an 8MHz oscillator this is
every 8s , per equation (6.3), which is not a factor of 5ms .
1 64 8s (6.3) 8, 000, 000
89
By changing the bitset of the TCCR0B register, specifically bits 2‐0, from prescaler 64 (0 1 1) to
prescaler 8 (0 1 0) 4the timer ticks 8 clock cycles or every 1s , per equation (6.4), which is a
factor of 5ms . Therefore, every 5000 timer ticks analog pin A0 was read. The code detailing
this is given in appendix 12.
1 81s (6.4) 8,000,000
For the custom PCB based on the ATMega328P a similar timing scheme was developed using timer ticks and overflow counters to take a voltage reading every 16ms . The full MCU code is
given in appendix 12.
6.6 Software Applications
In total four software applications centered around the controller were created; one to write
read PVDF voltages to a text file for use during measurement verification and neural network
training, code in appendix 14, and the remaining three to illustrate the abilities of the
controller.
6.6.1 Computer gesture indicating application
The first application, designed using Visual Studios 2012 with code in appendix 15, is purely
meant to showcase the possibilities of the controller when connected, via Bluetooth, to a PC.
The application, after pairing with the controller, reads the incoming voltages and processes
them through the neural network algorithm. Depending on the output of the algorithm the
4 Table 15‐9 of [1] 90
application displays either no recognized motion (blank), a “Right”, or a “Left”, as illustrated in
Figure 6.11, in correspondence with the performed gesture.
Figure 6.11. Controller user interface displaying recognized motion (designed for PC)
6.6.2 Android oscilloscope and gesture indicating application
The second application was developed for an Android tablet using the IDE eclipse with code shown in appendix 16. The application, once paired with the controller via Bluetooth, both displays the real‐time voltage output of the PVDF on a scrolling oscilloscope‐like plot and
processes the voltage through the neural network algorithm. Depending on the output of the
algorithm the app then displays either no arrow, a right arrow, or a left arrow as shown in
Figure 6.13 through Figure 6.14, respectively, in direct correlation with the gesture that
generated the corresponding voltage history.
91
Figure 6.12 Application for Android tablet showing no recognized gesture
Figure 6.13 Application for Android tablet showing recognized right gesture
92
Figure 6.14 Application for Android tablet showing recognized left gesture
6.6.3 PowerPoint Add In
The third application, also designed using Visual Studios 2012 and shown in appendix 17, is an
Add‐In for Microsoft PowerPoint as shown in Figure 6.15. The Add‐In allows the controller, once
paired again through Bluetooth, to navigate slideshows where a right gesture advances the
slideshow one step, the same effect as pressing the right arrow key, and a left gesture goes back one step in the slideshow, the same effect as pressing the left arrow key.
Figure 6.15 Add‐In for Microsoft PowerPoint
93
References
[1] Atmel, “ATMEL 8‐BIT MICROCONTROLLER WITH 4/8/16/32KBYTES IN‐SYSTEM PROGRAMMABLE FLASH DATASHEET,” Oct‐2014.
94
CHAPTER 7: Fabrication of a self‐sensing electroactive polymer bimorph actuator based on polyvinylidene fluoride and its electrostrictive terpolymer
Electroactive polymers (EAPs) are promising "artificial muscle" materials for their ability to
respond to electrical stimulation by exhibiting deformations several orders of magnitude
greater than what can be achieved with inorganic materials [1]. Additional advantages of these
materials for microactuator applications include their flexibility, low cost, low temperature
processing, lightweight, and fracture tolerance. The most widely researched EAPs to date are dielectric elastomer actuators, which offer high strains at favorable strain rates. Over the past
decade, however, the ferroelectric relaxor polymers P(VDF‐TrFE‐CFE) and P(VDF‐TrFE‐CTFE)
have been gaining attention for their unique EAP properties which include low hysteresis, an unusually high dielectric constant, and most significantly, a high elastic energy density (1.1J/cm3 reported [2]), considered a figure of merit for actuators [3].
Ferroelectricity is found in only a small number of crystalline polymers [4] with the most widely
exploited being PVDF and its copolymers [5]. PVDF and its copolymer P(VDF‐TrFE) are typically
used for force sensing via the direct piezoelectric effect. Random defects introduced by a third
bulky monomer, however, modify the normal ferroelectric transition resulting in a reversible
phase transformation between polar and non polar molecular forms [6]. This has a profound effect on the electromechanical nature of the resulting terpolymers, P(VDF‐TrFE‐CFE) and
P(VDF‐TrFE‐CTFE), which are thus converted to relaxor ferroelectrics capable of generating large electrostrictive strains greater than 7% under a field of 150MV/m with lateral strains of 3‐
5% [7]. Despite their chemical similarity to PVDF and P(VDF‐TrFE), the terpolymers cannot be
95
used for sensing, as mechanical stress will not result in the generation of charge buildup across
the layer [8].
For microactuator applications, bimorph beams are useful test structures, employing simple
configurations to generate relatively large displacements from electroded thin films of
electroactive materials. In one of the first applications of P(VDF‐TrFE‐FE), an actuator device for
the control of micro pulsed air jet from the shell of a projectile was studied by creating a
unimorph actuator of 30µm thick laminated P(VDF‐TrFE‐CFE) on a 120µm thick layer of steel
and coating the other side with Cr/Al. Under a field of 50V/µm at a frequency of 300Hz, the tip
of the unimorph reached an amplitude of 500µm [9]. In a separate work, a bimorph actuator was fabricated by laminating two 20µm P(VDF‐TrFE‐CFE) layers together, one active, and one inactive. The application of an electric field to the active layer resulted in a significant bending
[7].
EAP MEMS micro‐actuator deflections are typically recorded and characterized using optical
methods. For sensing applications using a resonating beam or membrane, however, a signal
such as a shift in resonance, q‐factor, or deflection must be detected during operation in order
for the device to be useful.
In this chapter, an electroactive polymer bimorph beam actuator driven by a voltage applied across an electroded thin film of ferroelectric relaxor polymer P(VDF‐TrFE‐CTFE) with an
integrated electroded PVDF thin film for built‐in sensing is presented and a comprehensive
model for understanding and predicting the EAP device behavior and for interpreting the signal
from the piezoelectric polymer layer is developed. In addition to providing sensing, it is shown
96
that the PVDF layer could also be used to activate the device, but with lower deflections via the
converse piezoelectric effect, which can provide bidirectional movement to the bimorph beam.
7.1 Device Fabrication
Table 7.1 outlines the composite beam structure which essentially is comprised of two back‐to‐
back metallized electroactive polymer films: a thin active P(VDF‐TrFE‐CTFE) layer sandwiched
between two gold (Au) electrodes and a thicker PVDF layer with one silver (Ag) ink electrode and one Au electrode (shared with the active layer); PVDF and silver ink moduli are determined experimentally. Fabrication of the structure began with a commercially available sheet of electroded (silver ink), stretched and poled 52 μm PVDF thin film which was used primarily for
sensing. The PVDF was cut into squares of 2 by 2 cm and taped to glass slides. The preexisting 6
um Ag ink electrode showed sensitivity to the P(VDF‐TrFE‐CFE) prepolymer solution, most likely
due to the presence of methyl ethyl ketone (MEK), which was used as a solvent, and as such it
could not be used as a substrate for spin‐coating of P(VDF‐TrFE‐CFE). Instead, the silver ink
electrode on the upper surface of the PVDF was removed using isopropyl alcohol and a 30nm
layer of gold above a 10nm adhesion layer of titanium were deposited by sputter. The Au
surface was then treated with adhesion promoter hexamethylsilazane (HMDS) and a 5.5 um
layer of P(VDF‐TrFE‐CTFE) was deposited by spin‐coating a 14% (w%/w) solution at 1000 rpm for 60 sec. The film was annealed in an oven at approximately 110 °C for 3 hours in order to grow the crystal domains responsible for the material's ferroelectric relaxor behavior. A final Au electrode was then deposited on the upper exposed surface of the P(VDF‐TrFE‐CFE) by sputter
through a stainless steel shadow mask patterned with rectangular beams. Following this, beams
97
were cut out of the film by following the outline of the rectangular upper Au electrode. A small border was maintained around the upper Au electrode to prevent a short circuit from forming
between the upper and lower electrodes during cutting of the film.
Table 7.1. Composite beam layers and thicknesses listed top to bottom Layer Thickness Modulus Gold 30 nm 72 GPa [10] PVDF‐TrFE‐CTFE 5.5 μm 150 MPa [11] Gold 30 nm 72 GPa Titanium 10 nm 90 GPa [12] PVDF 52 μm 1.23 GPa Silver Ink 6 μm 11.3 GPa
Through a force balance in the longitudinal direction of the composite beam the neutral axis was determined to be 18.982 μm above the base of the structure [13]. Relations between the
boundaries of the active elements and the neutral axis are detailed out in Table 7.2. A
schematic of the composite beam with identified active layers, electrical connections, and the
neutral axis (NA) is shown in Figure 7.1.
Table 7.2. Distance between active element boundaries and the neutral axis Boundary Distance to Neutral Axis (NA) (μm) Top of PVDF‐TrFE‐CTFE 44.558 Bottom of PVDF‐TrFE‐CTFE 39.058
Top of PVDF T1 39.018
Bottom of PVDF T2 ‐12.982
98
Figure 7.1. Fabricated composite beam with electrical connections
7.2 Self‐Sensing Bimorph Model
The cantilevered composite bimorph beam, which is assumed to obey classic Euler‐Bernoulli assumptions, is comprised of two distributed active elements, the PVDF‐TrFE‐CTFE actuator and the PVDF sensor. Each distributed active element is a degenerate transducer meaning that its
spatial and temporal dynamics are separable [14]. As a self‐sensing bimorph the structure, via
the PVDF, is capable of providing feedback (voltage) directly correlated to the deformation the
structure is experiencing in direct consequence to voltage applied to the terpolymer actuator.
7.2.1 Actuation based governing equation of motion
Due to the strain‐inducing electroactive terpolymer layer, the structure is capable of distributed actuation. As this layer of the composite structure is not located along the neutral axis, per
Figure 7.1 and Table 7.2, any applied voltage will induce a bending moment, or torque, about
the neutral axis. Defining the longitudinal direction as the x direction the governing partial
differential equation of the structure is assumed to obey
99
422wxt,,, wxt T xt EI A (6.5) xtx422
where wxT(, ) and TxT(, ) are the spatial‐temporal transverse displacement and torque
variables, respectively, E is the elastic modulus, I is the area moment of inertia, is the
density, and A is the cross sectional area. The active layer can be considered as extending to,
but never reaching the boundaries of the beam in the mathematical sense and, as such, for a
cantilever beam the standard boundary and initial conditions of the system are all set to zero
[14],[15]. As the system in question is self‐adjoint, it is known that the solution can be
decomposed, per the assumed modes method, into a summation of the product of its well‐
known eigenfunctions i x and generalized coordinates rti
n wxt, ii xrt (6.6) i1
Since the terpolymer is degenerate its spatial‐temporal behavior can be separated as
Txt , xut (6.7)
where x describes the active elements spatial shading and ut its temporal dynamics. The
rectangular shape of the actuator can be expressed using generalized functions, specifically two
Heaviside (step) functions, as:
x WHx Hx L (6.8)
100
where W is the height of the step function which physically corresponds to the width of the active element. The second derivative of (6.8) with respect to the spatial variable x is
2 x Wx xL (6.9) x2
where is termed a doublet function and can be viewed as a point moment loading condition [14],[16],[18]. Incorporating (6.6) through (6.9) into (6.5) yields
nn42 xrt ii EI42 rii t A x W x x L u t (6.10) ii11xt
The induced strain in the terpolymer in the longitudinal direction is
2 t tMEt 31 (6.11)
where the subscript t denotes terpolymer, M31 is an electrostrictive coefficient and E the applied electric field [19]. Further
vt Et in ht (6.12) 2 2 MQ31 31 0 r 1
where ht is the thickness of the terpolymer, Q31 is the charge related electrostrictive coefficient, 0 the permittivity of free space, and r the relative permittivity of the material. It
101
is known that the relative permittivity of the material varies as a function of the frequency of
the applied voltage so incorporating this and (6.12) into (6.11) gives
2 2 Q31 0 r 0 1 2 tin00,,tvt 2 (6.13) ht
where 0 is the prescribed frequency of the sinusoidal input signal. Denoting the non
terpolymer portion of the beam by subscript B the temporal input becomes [13],[20]
2 Q 2 1 31 0r 0 hhPB EEh PBB 2 ut vin 0, t (6.14) hEhEhtPPBB2
Substituting (6.14) into (6.10) and dividing through by A gives the governing equation of the
composite beam with an electroactive polymer acting as a distributed transducer in terms of its
th i natural frequency ni to be
2 nn42 ni iixrt 44rtii x 2 i ii11xt 2 (6.15) Q 2 1 31 0r 0 hhPB EEh PBB 2 x xLv in 0 , t AhtPPBB2 Eh Eh
7.2.2 Derivation of tip deflection from voltage across PVDF
Localized charge is generated by the PVDF as a consequence of induced strain from the bending of the structure via the actuation of the terpolymer according to [21]
102
k 2 q xzt,, 31 W xzt ,, (6.16) g31 where x and z are the longitudinal and transverse coordinates, respectively, q is the locally
generated charge, k31 and g31 are the appropriate piezoelectric electromagnetic coupling constant and stress constant, respectively, W is the width of the PVDF film (common to all layer of the structure), and is the local strain defined by [22]
2wxt , xzt,, z (6.17) x2
where wxt , is again the transverse deflection of the neutral axis of the structure and z the distance from the neutral axis of the structure. The total charge in the PVDF layer, Q , is found by integrating equation (6.17) over the vertical bounds of the PVDF with respect to the neutral axis and equation (6.16) over the length of the PVDF (common to all layers of the structure):
2 k 2 L T1 wxt , Q t W31 z dzdx g x2 31 0 T2 (6.18) TT22 k 2 wxt,, wxt Qt W1231 2 gx x 31 xL x0
where T1 and T2 are defined in Table 7.2. The two terms in the brackets in (6.18) are the slopes of the beam at the free and fixed ends respectively. Given that the structure is a cantilever beam the second bracketed term will be zero, a direct consequence of boundary conditions, and hence the charge generated in the PDVF is directly proportional to the slope at the tip of
103
the beam. PVDF is a dielectric and, when electroded, acts as a capacitor and hence the charge
generated can be related to voltage VtPVDF ()through the capacitance of the PVDF C as
22 2 Qt W TT12 k31 wxt , VtPVDF (6.19) CC2 g x 31 xL
For the configuration under consideration per equation (6.10), a cantilevered beam with an
applied bending moment at the tip, it is known that the deformation of the structure at the tip,
tip deflection, is given by22
L wxt , wLt, (6.20) 2 x xL
Solving (6.19) for the slope term and substituting into (6.20) allows the tip deflection to be expressed as
L g31 wLt, CVPVDF t (6.21) WT22 T k 2 1231
As the measurable output voltage from PVDF is known to be related to the strain rate
experienced by the sensor, seen from impedance modeling, equation (6.21) can be adapted to
measured voltage Vout to give real time tip deflection as
L g31 wLt, CV out tdt (6.22) WT22 T k 2 1231 0
104
where can be viewed as a place holder for the time instant that the integral is to be evaluated.
7.3 Experimental Configuration and Procedures
To conduct testing of the beams, each beam was first mounted at its base on a glass slide using copper tape so that the bottom Ag ink electrode was in contact with the conductive tape and
an area of electroded thin polymer film extended off of the edge. This setup was approximated
in modeling as a cantilevered beam although the fixed end boundary condition was not quite
met. For each beam the shared Au electrode was exposed from beneath the P(VDF‐TrFE‐CFE) at
its base by manually removing part of the polymer with acetone. The slide was in a probe
station equipped with an laser Doppler Vibrometer (LDV) as shown in Figure 7.2.
Figure 7.2. Probe station setup
105
7.3.1 Configuration
The glass slide with mounted beams was placed in a probe station and a blunt‐tipped probe, blunt so as not to perforate the thin Au electrode, was used to connect the common Au electrode of the beam to ground. For actuation using electrostriction, an additional blunt‐
tipped probe was used to provide a driving signal to the upper Au electrode using an arbitrary
waveform generator connected to an amplifier. This created an alternating electrical field
across the P(VDF‐TrFE‐CFE) layer which resulted in electrostriction in the out of plane direction
and a lateral expansion of the layer. As mentioned above, the presence of the "passive" PVDF
layer broke the symmetry of the device, resulting in a bending of the beam. An additional probe contact was made to the Cu tape to record the voltage across the PVDF and common ground
layer on the oscilloscope.
Out of plane displacements of the beams were also recorded by the oscilloscope using a LDV in velocity mode and used to calculate tip deflections. For most measurement sets, the laser was
focused near the base or center of the beam rather than the tip of the beam, in order to reduce noise due to high deflections. Due to the proximity of the two capacitors comprising the beam,
there existed the potential for parasitic capacitance at higher frequencies. The LDV, which was
electrically isolated from the rest of the system, was thus able to verify that the beam was
bending in response to electrical stimulation that the signal from the PVDF did not simply
represent a parasitic capacitance effect.
106
7.3.2 Procedures
Resonant frequencies were observed for the actuated beams in three ways: by performing a
manual frequency sweep, by performing the frequency sweep using a network analyzer
connected to the LDV in velocity mode, and by providing a manual impulse, or “flick”, to the
beam and looking at the frequency of the voltage response generated by the PVDF.
Once resonance was determined, the beam was actuated at constant frequency near resonance over a 50‐300V voltage range. The experiment was then repeated at a frequency far away from resonance, because smaller beam deflections resulted in a cleaner LDV signal. Next, to investigate the dependence of tip deflection on excitation frequency, the response of the beam over a range of frequencies, spanning 250 Hz to 130 KHz, was observed again using both
the LDV and the PVDF. The driving function applied to the terpolymer is shown in equation
(6.23) where the post amplification coefficient A0 was 50V.
Vt A0 1sin2 ft (6.23)
Finally, the beam was driven using the PVDF layer as the active film, employing the reverse
piezoelectric effect, and the terpolymer functioned as a passive layer in this case. When using
piezoelectric actuation, the beam is able to bend in either direction in accordance with the
polarity of the driving voltage.
107
7.4 Results and Discussion
7.4.1 Actuation and electromechanical characterization
The amplitude of the tip deflection increased with the applied voltage at constant frequency following a nearly square dependence. This is in good agreement with the predicted behavior
described in section 7.3.1. Figure 7.3 shows peak‐to‐peak deflections of a 2.75mm wide, 6.5mm
long beam over a range of driving voltages. The device was initially actuated by inducing electrostriction at 525 Hz, its resonant frequency, with measurements taken by focusing the
LDV near the tip of the device. Due to the large stroke, however, much of the laser beam was
not reflected back to the LDV detector, resulting in a noisy signal. Deflections of up to 350μm
were observed using this method, at an applied voltage of 300V. Subsequent LDV
measurements were performed by focusing the laser near the base of the device to improve the signal to the detector. Although the magnitude of deflections near the base were two
orders of magnitude lower than at the tip, fitting the data, as shown in Figure 7.3, indicates that
deflections in both instances scale squarely with the voltage.
For three similar devices, resonant frequencies of 511.2Hz, 525.3Hz, and 553.4Hz were recorded using a network analyzer connected to the LDV in velocity mode. One such frequency
sweep can be seen in Figure 7.4. The discrepancies in resonant frequency from device to device
can be accounted for by the slightly different device dimensions, which can be attributed to the
manual cutting process. The resonant frequency of one of the devices was verified via the signal generated across the PVDF from a manual impulse test, as described above (not pictured).
108
Figure 7.3. LDV measured peak to peak deflection as a function of peak to peak driving voltage. Inset shows LDV.
Figure 7.4 shows the frequency response of the beam when driven using each of the EAPs
separately at a peak to peak signal of 280V, with a 140V DC offset. The offset is necessary
because electrostrictive strain varies with the square of the field and a signal alternating around
zero would result in a frequency of the actuation that is double the driving frequency, as
opposed to the case of piezoelectricity, where a changing sign of field simply results in a
different direction of bending. The resonant frequency of the device, which is purely
mechanical, is identical for each mode of actuation. The main difference between the curves is
that the bending induced by actuating the piezoelectric PVDF layer was significantly smaller in
magnitude than the bending induced through electrostriction of the terpolymer. The PVDF's
response to a 100V AC signal, for example, was not sufficient to generate a mechanical signal
that could be picked up by the LDV detector, while the terpolymer drive was. In addition, if the
PVDF layer is used to drive oscillations in the device, it cannot be used for mechanical sensing.
109
Figure 7.4. Frequency sweep showing velocity of deflection for two modes of EAP actuation used to drive the bimorph beams; blue curve represents electrostrictive actuation, while green curve represents piezoelectric actuation.
The tail of the two peaks follow different slopes as frequency is increased, which suggests that the terpolymer's relatively higher sensitivity to frequency could cause an additional decrease in
bending magnitude at higher frequencies. The difference may instead be due to the different mechanisms of electromechanical actuation at play. It is especially hard to determine in light of
the fact that the LDV measurements were performed in velocity mode, which can be used to indicate the resonant peaks, but not to determine accurate tip deflection (without integration).
7.4.2 Integrated sensing of cantilever tip deflections
While the PVDF thin film can technically be used to drive oscillations in the device, there exist
better EAP actuator materials, and the compliant piezoelectric layer is better utilized for providing integrated sensing. As dictated by the system’s impedance model, Figure 7.5 and
equation (6.24), the voltage displayed on the oscilloscope during actuation is not the "true"
110
voltage generated by the PVDF and must be adjusted prior to solving for tip deflections.
Defining Gs , equation (6.25), as the magnitude of the frequency response, at each driving frequency the signal attenuation factor was found and the measured response was adjusted to generate the true PVDF generated voltage. Following this, the maximum tip deflection was
found at each driving frequency and compared to that calculated from the LDV measurements
as shown in Figure 7.6.
Figure 7.5. PVDF and oscilloscope impedance model
ZsRCscope s p Vsscope Vs PVDF Vs PVDF (6.24) ZZPVDF scope sRsp C C s1
Vsscope RC s p Gs() (6.25) sj Vs 2 PVDF sj 1RC C ss p
111
Figure 7.6. Beam tip deflection measured by LDV and PVDF
Figure 7.6 shows tip deflection over a frequency range where the electrostrictive P(VDF‐TrFE‐
CTFE) layer is used for actuation, and the voltage across the PVDF layer was recorded. When the peak‐to‐peak displacements of the beam, observed using the LDV in velocity mode
(interpreted to deflection by integration) are plotted as a function of frequency at constant peak to peak voltage of 100V with a DC offset of 50V, a nonlinear decrease in tip deflection with increasing frequency can be observed.
A comparison of the two curves shows poor agreement and the discrepancy between the curves grows by orders of magnitude with increasing frequency. Specifically missing from the
LDV curve in Figure 7.6 is the resonance peak around 500Hz that was formerly observed using
the LDV in the frequency sweep shown in Figure 7.4. This is believed to be caused by different
positioning of the laser on the beam and serves to illustrate the benefits of embedded sensing.
112
The general shape of the PVDF response curve is a better match to the predicted behavior of
the beam with a visible peak at the natural frequency and steep roll off thereafter. The lack of
sensitivity in the shape of the LDV at higher frequencies is likely caused by the difficulty in
focusing the LDV to get accurate readings.
7.5 Conclusions
Functional bimorph actuators based on relaxor ferroelectric P(VDF‐TrFE‐CTFE) were fabricated
and tested, demonstrating real time sensing capabilities via integrated piezoelectric PVDF thin
films. While tip displacements were also interpreted from LDV measurements in velocity mode,
the deflections deriving from the voltage measured across the integrated piezoelectric thin film
showed higher sensitivity. Thus, the developed model offers not only a method through which
real time tip deflection can be measured without the need for external visualization, but also
better accuracy than optical measurements that we performed at a single location on the
device. The frequency response of the actuators was examined using several methods and it
was demonstrated that the PVDF layer can provide an additional means of beam actuation and bidirectionality, although there is no integrated sensing in this case and deflections are smaller
in magnitude.
Potential applications of the current technology are in haptic devices such as displays, artificial
skin, artificial muscles, soft robotic, and in chemical or biological resonator sensors. The porous
nature of polymers provides them with inherent absorptive properties, which can eliminate the
need for functionalization, typically used to provide sensing capabilities to Si‐based resonators.
An additional benefit of polymer micro‐resonators is their comparatively low elastic modulus,
113
which allows for actuation at lower driving voltages for low power devices. Polymer
microcantlilever sensors have been demonstrated using functionalized SU‐8 [23],[24], which is
not an electroactive material. As EAP actuator devices undergo downscaling and become
increasingly compatible with microelectronics, however, external measuring devices will
become less and less practical. PVDF thin films, through appropriate hardware such as a charge
amplifier or other integration circuits, can be used to accurately track tip deflections in real
time, facilitating new EAP technologies in sensing and robotics.
References
[1] Bar‐Cohen, Y. and Zhang, Q., “Electroactive polymer actuators and sensors,” MRS Bull, 33(3), 173–181 (2008). [2] Xia, F, Cheng, Z. Y., Xu, H., Li, H., Zhang, Q., Kavarnos, G., Ting, R., Abdul‐Sadek, G., and Belfield, K., “High electromechanical responses in a poly(vinylidene fluoridetri‐ fluoroethylenechlorofluoroethylene) terpolymer,” Adv. Mater., 14(21), 1574–1577 (2002). [3] Leo, D.J., [Engineering Analysis of Smart Material Systems], John Wiley & Sons, Inc., Hoboken, NJ., (2007). [4] Poulsen, M., and Ducharme, S., “Why ferroelectric polyvinylidene fluoride is special,” IEEE Trans. Dielectr. Electr. Insul., 17(4), 1028– 1035 (2010). [5] Bar‐Cohen, Y., "Biomimetics: biologically inspired technologies," II ECCOMAS Thematic Conference on Smart Structures and Materials, CRC/Taylor & Francis, 271–272 (2006). [6] Chen, Q., Ren, K., Chu, B., Liu, L., Zhang, Q. M., Bobnar, V., Levstik, A., “Relaxor Ferroelectric Polymers‐Fundamentals and Applications,” Ferroelectrics, 354(1), 178‐191 (2011). [7] Bauer, F., “Relaxor fluorinated polymers: novel applications and recent developments,” IEEE Trans. Dielectr. Electr. Insul., 17(4), 1106– 1112 (2010). [8] Brochu, P., and Pei, Q., “Advances in dielectric elastomers for actuators and artificial muscles,” Macromol. Rapid Commun., 31(1), 10–36 (2009). [9] Bauer, F., Capsal, J., Larcher, Q., and Domingues Dos Santos, F., “Advances in relaxor ferroelectric terpolymer: New applications,” in Applications of Ferroelectrics (ISAF/PFM), 2011 International Symposium on Piezoresponse Force Microscopy and Nanoscale Phenomena in Polar Materials, IEEE, 1–4 (2011). [10] Hodge, T. C., Bidstrup‐Allen, S. A., Kohl, P. A., “Stresses in Thin Film Metallization,” IEEE Trans. Compon. Packag. Manuf. Technol., Pt A, 20(2), 241‐250 (1997).
114
[11] Bauer, F., Fousson, E., Zhang, Q. M., “Recent Advances in Highly Electrostrictive P(VDF‐ TrFE‐CFE) Terpolymers,” IEEE Trans. Dielectr. Electr. Insul., 13(5), 1149‐1154 (2006). [12] Tsuchiya, T., Hirata, M., Chiba, N., “Young’s Modulus, Fracture Strain, and Tensile Strength of Sputtered Titanium Thin Films,” Thin Solid Films, 484(1‐2), 245‐250 (2005). [13] Baz, A., Poh, S., “Performance of an Active Control System with Piezoelectric Actuators,” J. Snd. Vib., 126(2), 327‐343 (1988). [14] Hubbard Jr., J. E., [Spatial Filtering for the Control of Smart Structures], Springer, Berlin, Heidelberg (2010). [15] Meirovitch, L., [Principles and Techniques of Vibrations], Prentice Hall, Upper Saddle River, NJ., (1997). [16] Zemanian, A. H., [Distribution Theory and Transform Analysis: and Introduction to Generalized Functions, with Applications], Dover Publications, New York, NY., (1987). [17] Olver, P. J., [Introduction to Partial Differential Equations], Springer Science+Business Media, LLC, New York, NY., (2013). [18] Burke, S.E., Hubbard, J.E., “Active Vibration Control of a Simply Supported Beam Using a Spatially Distributed Actuator,” IEEE Ctrl Sys Mag, 25‐30 (1987). [19] Engel, L., Kruk, S., Shklovsky, J., Shacham‐Diamand, Y., Krylov, S., “A study toward the development of an electromechanical poly(vinylidene fluoride–trifluoroethylene– chlorofluoroethylene) buckling membrane actuator,” J. Micromech. Microeng., 24(12), 125027 (2014). [20] Bailey, T., Hubbard, J.E., “Distributed Piezoelectric‐Polymer Active Vibration Control of a Cantilever Beam,” J. Guid. Ctrl. Dyn., 8(5), 605‐611 (1985). [21] Pota, H. R., Alberts, T. E., “Multivariable Transfer Functions for a Slewing Piezoelectric Laminate Beam,” J. Dyn. Sys., Meas. Ctrl. 117(3), 352–359 (1995). [22] Popov, E., [Introduction to Mechanics of Solids], Prentice Hall, Englewood Cliffs, NJ., (1968). [23] Seena, V., Rajorya, A., Pant, P., Mukherji, S., and Rao, V., “Polymer microcantilever biochemical sensors with integrated polymer composites for electrical detection,” Solid State Sci., 11(9), 1606–1611 (2009). [24] Waggoner, P., and Craighead, H., “Micro‐and nanomechanical sensors for environmental, chemical, and biological detection,” Lab Chip, 7(10), 1238–1255 (2007).
115
CHAPTER 8: Use of the shape memory polymer polystyrene in the creation of thin film stretchable sensors for wearable applications
In developing wearable controllers, or simply wearables, for gesture based applications, in addition to simply being accurate, devices must further be intuitive, shape conforming, unrestricting of motion, and adaptable. The premise behind wearables is that some physical interaction is required to interface with every day electronic devices be it, for example, a
keyboard, mouse, remote control, knob, or button. In the sense that humans have a tendency
to gesticulate when verbally communicating with other humans, wearables would allow
humans to communicate with, or rather control, electronic devices through intuitive,
programmable, gesticulations. Focusing on hand gestures, data has traditionally been obtained
through either position tracking systems or glove based systems [1],[2]. Previous work has
shown that the thin film piezoelectric polymer polyvinylidene fluoride (PVDF) can be used to
successfully capture hand motions [3],[4]. This chapter aims to characterize the dynamic
behavior of a developed electronic skin (e‐skin) style thin film sensor fabricated using the
shape‐memory polymer (SMP) polystyrene (PS) and its ability to monitor biomechanical activity; specifically, measuring the joint angle of the metacarpophalangeal (MCP) joint on the right index finger.
Whereas shape‐memory alloys (SMAs) have a reversible temperature driven martensitic transformation, SMPs shape‐memory effect arises from a dual‐segment system where one segment is highly elastic and the other segment reduces its stiffness in response to exposure to a particular stimulus [5]. SMPs can be thermo, light, or chemo‐responsive with direct heating being the most common stimulus [5],[6]. PS has what is termed a one‐way shape memory
116
effect. The polymer, fabricated in its permanent shape, is given a new temporary shape via
programming, such as heating and deforming. When exposed to the proper stimulus, such as
being heated above a transition temperature in thermo‐responsive SMPs, the permanent shape
is recovered [7].
The developed sensor, which function as strain gauges under the basic premise of geometric piezorestivity, use a conductive bimetallic thin film layer with induced nanowrinkles [8] bonded to the elastomer polydimethylsiloxane (PDMS). Traditional metal foil strain gauges have strain
limits between one and three percent, semiconductor strain gauges a strain limit around a of half a percent, and metal foil post yield gauges a single use strain limit of up to 20% [9]. Thanks to both the nanowrinkles and being bonded to the elastomer, the sensor is able to repeatedly
measure stains up to 23.5% making it ideal for the application of measuring MCP joint angle
which it is shown to do with a high degree of accuracy.
8.1 Sensor Fabrication
Continuing along the lines of previous work, creating complex hierarchical wrinkling in metal thin films using shape‐memory thermoplastics [10],[11], this work uses PS to induce wrinkles in metallic thin films bonded to the elastomer PDMS as a strain sensor. Similar to serpentine
structures previously reported [12], these micro‐ and nanowrinklings are important for relieving
stress on the metal structure, which allow for larger strains, and lower the effective resistivity
[10].
117
8.1.1 Background
Previous work performed by Lu et al. have shown that metal thin films strongly adhered to polymer surfaces results in those thin films maintaining conductivity over relatively large strains of 50% [13]. The polymer supported film evenly distributes the strain across the thin film whereas freestanding metal thin films suffer from the generation of highly localized stress
regions causing premature fracturing at <5% strain. Wagner et al. demonstrated, through the
incorporation of waves, polymer supported thin films conductive up to 100% strain [14].
Leveraging this mechanical stability of polymers supported metal thin films, Fan et al. effectively optimized geometry to relieve strain on conductive line traces to create stretchable
interconnects insensitive to moderate strains [12]. However, to achieve good strain sensitivity,
controlled fracturing of the metal thin film is desired. Kang et al. showed, to great effect, the
sensitivity of as deposited Pt thin films on polymer surfaces leveraging this mechanism. With a
gauge factor of greater than 2000 at strains of 0‐2%, sensitivity was good enough to show
speech recognition from analyzing signals produced from these bimetallic thin films when used
as a strain sensor placed over the throat [15].
8.1.2 Fabrication
Sensor fabrication in this study is similar to a previously published process for patterning metal
thin films onto SMPs [8],[11]. For the described sensor, however, a 5 nm platinum (Pt) thin film
layer has been deposited onto PS by sputtering through a tape mask, Figure 8.1a. A subsequent
5 nm gold (Au) thin film layer is then deposited for adhesion purposes. The sample is then
heated at 160°C, triggering the shape memory effect in the PS, causing the substrate to shrink
118
to less than 25% of its initial area as shown in Figure 8.1b. Due to stiffness mismatches in the PS
and the thin film layers, wrinkling in the thin film ensues as shown in Figure 8.1c and d. To promote adhesion, the sample is then immersed in a 5 mM (3‐mercaptopropyl)
trimethoxysilane (95% MPTMS) solution which functionalizes the gold surface. After silane treatment, PDMS is immediately spin coated onto the sample before thermal curing for 2 hrs at
80°C. The sensor is then transferred, the PS layer is lifted off via an acetone bath followed by a toluene wash, and allowed to dry.
(a) (b) (c) (d)
Figure 8.1. (a) Sputtered bimetallic layers on full size PS sheet. Approximate dimensions: 50mm (width) by 80mm (height) by 0.5mm (thickness). (b) Bimetallic layers on PS post shrinking (same sample as shown in (a)). Approximate dimensions: 20.8mm (width) by 35.0mm (height) by 1.9mm (thickness). SEM images of transferred sensor showing wrinkling in the Pt layer caused by stiffness mismatches during the shrining process at 4,000x (c) and 10,000x (d) magnification.
The finished, transferred, sensor is shown in Figure 8.2a. For device integration, as well as
testing, leads are attached to the sensor, Figure 8.2b. A four channel flat flexible cable (FFC) was doctored such that the two inner channels were notched out, leaving the two outer channels to make contact with the pads of the sensor. Colloidal silver was used to enhance
surface contact and minimize contact resistance effects. Tape was then applied over the FFC leads and pads, Figure 8.2c, to hold the leads in place during straining, inhibit strain in the pads
119
themselves, and help in attaching the sensor to the glove (section 8.3 ). Tape was also applied
to the top of the sensor for symmetry during testing, Figure 8.2d, and, again, to aid in attaching
the sensor to the glove.
(a) (b) (c) (d)
Figure 8.2. Stages of adapting the sensor for testing (note in (a)‐(c) sensor is on blue background). (a) Transferred sensor (post chemical bath). (b) Sensor with FFC and colloidal silver. (c) Sensor with tape. (d) Sensor mounted for testing.
8.2 Sensor Testing and Characteristics
The developed stretchable thin film Pt sensors operate according to the general notion of geometric piezoresistivity; the resistance of the sensor varies as a function of elongation.
Resistivity is a measure of the opposition to current flow through a median. True in all materials, resistivity is dependent on the electron mean free path. The total resistivity of a
material, according to Matthiessen’s rule, is dependent on the summation of all present
electron scattering processes [16]. In thin films the resistivity becomes dependent on film thickness when the mean free path becomes comparable with said film thickness; the mean
free path is reduced due to an increase in scattering effects associated with nanoscale
dimensions [17],[18]. Further, as seen in Figure 8.1c, in the sensors under study the
nanowrinkles create a three dimensional labyrinth filled with electrical discontinuities which
further add to scattering effects and complicate the mean free path.
120
8.2.1 Test Setup
The sensors were subjected to uniaxial tensile, cyclic, and step testing using a hydraulically
powered table top tensile testing machine operated in displacement control mode. To help
hold the sensors in place, since excessive clamping force was found to push the FFC leads
through the sensor pads, sandpaper was attached to the clamps to improve grip. The mounted
sensor is shown back in Figure 8.2d.
The sensor was integrated into a typical quarter bridge Wheatstone circuit with amplification and where it was in series with a potentiometer so the bridge could be initially balanced as shown in Figure 8.3. Sensor resistance measurements were calculated based on amplifier output voltage read by a microcontroller’s 10‐bit analog to digital converter (ADC) pin.
Figure 8.3. Quarter bridge Wheatstone schematic with amplification
After clamping the sensor, the sensor’s unstretched resistance was measured using a
multimeter and the bridge brought to an approximate balance. The resistance of the potentiometer was calculated based on the sensor’s unstretched resistance, RS0 , and reported
ADC value according to equation (6.26)
121
R 1 1 ADC 2R2 1023 R R (6.26) POTR S 0 1 1 ADC 21023R2
Knowing RPOT allowed all future sensor resistance values, RS , to be calculated by rearranging
equation (6.26) to get
R 1 1 ADC 2R2 1023 RR (6.27) SPOTR 1 1 ADC 2R2 1023
For each sensor tested, or if the sensor had to be re‐clamped, the bridge was rebalanced and
RPOT recalculated. The microcontroller, using timer overflow triggers, sampled the voltage
every 25ms and wrote the value to a text file. Values of circuit elements used are shown in
Table 8.1 where C, not shown in schematic, was a bypass capacitor for the Op‐Amp.
Table 8.1. Circuit values
Vcc R R1 R2 C 5V 1kΩ 1kΩ 10kΩ 10nF
8.2.2 Tensile Testing
For tensile testing the sensor was stretched 4mm from its original clamped length of 17mm so
that the max strain, roughly 23.5%, correlated with the expected maximum strain experienced
when measuring MCP joint angles as discussed in detail in section 8.3 . The test was repeated
six times, each test at a different stretch rate with the lowest rate corresponding to a full finger
122
flex taking 80 seconds and the highest rate corresponding to a full finger flex taking one second.
Both the force required to perform the stretching and the output voltage from the op‐amp,
Figure 8.3, were recorded. Test normalized sensor resistance values as a function of strain at each stretch rate are shown in Figure 8.4. A decrease in sensitivity can be seen for strains
greater than 15%.
Figure 8.4. Test normalized sensor resistance as a function of strain when stretched at different rates.
From measured force data and approximate sensor initial cross section dimensions, 20mm in width and 0.5mm in thickness, the stress experienced by the sensor as a function of strain during tensile testing at each stretch rates is shown in Figure 8.5. The average modulus of the
PDMS was calculated to be approximately 1.18 MPa which agrees with previous findings [19].
123
Figure 8.5. Stress as a function of strain at various stretch rates measured during tensile testing. Approximate PDMS modulus of 1.18 MPa.
8.2.3 Cyclic Testing
The sensor was subject to cyclic deformation testing in accordance with equation (6.28) such
that at the maximum the sensor was stretched 4mm, approximate strain of 23.5%, and
unstretched at the minimum. Frequencies of oscillations span 0.25Hz to 1.0 Hz in increments of
0.25Hz. Voltage measured through the sensor as a function of strain over five oscillations is
shown in Figure 8.6 while calculated stress, from force measurements, as a function of strain is
shown in Figure 8.7.
2 2sin(2 ft ) (6.28)
124
Due to relaxation effects, also seen during step testing as shown in the following section, there
is visible hysteresis in the electrical response of the sensor to cyclic stretching with the amount
of hysteresis, in general, consistent over the frequency range tested although the hysteresis
loop of 0.25 Hz is seen to be slightly tighter than the rest. The bottom portion of the loop in
Figure 8.6 corresponds to increasing strain in the sensor and the top portion to decreasing strain in the sensor. In agreement with sensor resistance characteristics, Figure 8.4, a decrease in sensitivity of measured voltage to an increase in strain is observed for strains greater than
15%.
In the mechanical response, it appears that the lowest few percentages of strain occur at an approximately constant amount stress. In reality, what was observed was that, due to relaxation effects of the PDMS, when the displacement of the clamps was within the vicinity of
minimizing equation (6.28) there was visible slack in the sensor, Figure 8.8, such that, instead of
remaining taught and within the axis of stretching, it bulged in the transverse direction; direction normal to the length by width plane of the sensor. This caused the force measuring
load cell of the testing machine to detect no changes while the clamp traveled through this region. Further, the sensor was no longer truly being strained either, although the displacement of the clamps changed, accounting for the initial electrical insensitivity at low strains seen in
Figure 8.6 and the discrepancy in behavior when compared with Figure 8.4.
125
Figure 8.6. Op‐amp output voltage as a function of strain for five cycles during cyclic testing over indicated frequencies
Figure 8.7. Stress as a function of strain for the first 10 cycles during cyclic testing at indicated frequencies
126
Figure 8.8. Sensor bulging during relaxation phase of cyclic testing.
8.2.4 Step Testing
Finally, the sensor was subject to step testing. For four different step amounts, 0.5, 1, 2, and
4mms, the sensor was first incrementally stretched up to 4mm then incrementally relaxed to zero stretch. Figure 8.9 and Figure 8.10 show increasing and decreasing electrical response, op‐ amp output voltage, to stretching, respectively, with horizontal dashed lines at common stretch increments. A single test consisted of stretching or relaxing the sensor one incremental
amount. Combining the tests led to discontinuities as depicted by the red hash lines in the
figures.
For increasing steps, Figure 8.9, the electrical characteristics can be seen to obey the principles
of superposition. Further, while there is some noise, there is minimal steady state drift. One notices that the starting value for the 0.5mm step test is significantly lower than that for the other tests. After performing each increasing step test, starting with 0.5mm, the corresponding decreasing step test was performed and is shown in Figure 8.10. Due to the same relaxation
effects discussed in the previous section, the sensor resistance did not return to its initial value
127
prior to the next test being run and hence the discrepancies. This relaxation effect can be seen for all step down increments, is more pronounced at larger step sizes, and is most evident when the sensor strain returns to zero.
Figure 8.9. Superposition in electrical response to incremental stretch step ups (step size denoted in each subplot). Red hash lines indicate discontinuity in start and end of data collection periods.
Figure 8.10. Relaxation effects in electrical response during incremental stretch step downs (step size denoted in each subplot). Red hash lines indicate discontinuity in start and end of data collection periods (each step). 128
8.3 Sensor Based Applications
For accuracy and precision, strain sensors are typically rigidly bonded to the material they are
tasked with sensing strain in. In terms of wearables, this implies either attaching the sensor to
the skin for single use applications or to form fitting fabrics for repeated use applications.
Considering the repeated use case, a fitted golf glove was found to be a suitable host for the sensor. To measure the angle of the MCP joint on the right index finger the sensor was adhered to the glove, Figure 8.11, such that it was approximately centered over said joint. From
measurements, while if the sensor had been affixed directly to the skin it would be experiencing a maximum strain of roughly 40%, because of elastic in the glove at the base of the fingers absorbing strain (black mesh visible in Figure 8.11), the sensor instead experiences a maximum strain of roughly 23.5%; stretching from 17 to 21mm.
Figure 8.11. Sensor adhered to right index finger of glove centered over MCP joint. Line bisecting sensor indicates center of joint on glove for accuracy in sensor mounting.
With the glove worn by a representative 30 year old male, a motion study was conducted
where the wearer’s right index finger cycled continuously through full flexion and extension at a
rate of approximately 0.5 Hz. Concurrently, sensor voltage data was collected (using developed
software) and finger motion was video recorded such that a correlation between voltage and
129
MCP joint angle was able to be made. The results of three representative motions are shown in
Figure 8.12 where the data can be seen to be similar in nature to the data collected during uniaxial cyclic testing shown in Figure 8.6.
From the raw data, two models were created; one for increasing joint angle (flexion) and one
for decreasing joint angle (extension). Both models are based on the logistic function with
parameters define in equation (6.29) [20] with the variable x being joint angle. Due to the
nature of the logistic function, the bridge circuit was intentionally slightly unbalanced at the start of the motion study by roughly an ADC value of 10 or Vcc(10/1023) V.
1 gu, u a ax 1 eu 01 Increasing: Decreasing: (6.29)
aa004.5 2.55
aa110.1125 0.1
Error between measured and model predicted joint angles was determined at each measured
voltage for all three motions. The data was analyzed based on whether the finger was in flexion
or extension based on the measured joint angle relative to the previous measured joint angle. If the measured voltage was below the lower limit of its respective model curve the joint angle was taken to be 0 degrees and if the measured voltage was above the upper limit of its respective model curve the joint angle was taken to be 90 degrees. The average error for all three motions, over both flexion and extension models, was 2.99 degrees. The largest joint angle error in the flexion model was 11.42 degrees while in the extension model the largest
130
error was 16.73 degrees. In both models, the largest errors occurred in the upper joint angle region 69 degrees .
Figure 8.12. Full MCP joint flexion and extension sensor values and associated models.
8.4 Conclusions
Thin film nanowrinkled Pt sensors bonded to the elastomer PDMS, developed using the SMP PS, are shown that they are capable of repeatedly being strained to roughly 23.5% such that they can be integrated into wearables for biomechanical measurement applications; specifically measuring MCP joint angles of the right index finger. The senor dynamic characteristics were observed for tensile, cyclic, and increase and decreasing step testing. It was found that the
131
electrical characteristics of the sensor proved to differ dependent on whether the sensor was
being strained or relaxed. To measure MCP joint angles the sensor was adhered to the right
index finger of a fitted golf glove. The finger was cycled through full flexion and extension with
electrical data being collected and the motions video recorded such that a motion study was possible. It was found that two simple curves, one for flexion and the other for extension,
could, to a reasonable degree of accuracy, predict MCP joint angle based on measured voltage
such that the average error in joint angle was 2.99 degrees. This implies that such a model can
be used for control applications so long as the direction of motion is taken into consideration.
References
[1] Struman, D.J., and Zeltzer, D., “A Survey of Glove‐based Input,” IEEE Comput. Graph. Appl. Mag., 14(1), 30‐39 (1994). [2] Dipietro, L., Sabatini, A.M., and Dario, P., “A Survey of Glove‐Based Systems and Their Applications,” IEEE Trans. Syst., Man, Cybern., Pt. C: Apps. Revs., 38(4), 461‐482 (2008). [3] Van Volkinburg, K. R., Washington, G. N., “Modeling of a PVDF based gesture controller using energy methods,” Proc. SPIE 9056, 905621 (2014). [4] Van Volkinburg, K. R., Washington, G. N., “PVDF Based Wearable Joystick Using Gesture Recognition via Neural Networks,” Proc. SMASIS 2014, 7577 (2014). [5] Leng, J., Lu, H., Liu, Y., Huang, W. M., and Du, S., “Shape‐Memory Polymers–A Class of Novel Smart Materials,” MRS Bull, 34(11), 848‐855 (2009). [6] Xie, T., “Recent advances in polymer shape memory,” Polymer, 52(22), 4985‐5000 (2011) [7] Lendlein, A., and Kelch, S., “Shape‐Memory Polymers,” Angew. Chem. Int. Ed. Engl., 41(12), 2034‐2057 (2002) [8] Fu, C., Grimes, A., Long, M., Ferri, C.G.L., Rich, B.D., Ghosh, So., Ghosh, Sa., Lee, L.P., Gopinathan, A., and Khine, M., “Tunable Nanowrinkles on Shape Memory Polymer Sheets,” Adv. Mater., 21(44), 4472‐4476 (2009). [9] Swallowe, G. M., [Mechanical Properties and Testing of Polymers: An A‐Z Reference], Springer Netherlands, Dordrecht, pg. 200 (1999). [10] Pegan J., Ho A., Bachman M., and Khine M., “Flexible shrink‐induced high surface area electrodes for electrochemiluminescent sensing,” Lab Chip, 13(21), 4205‐4209 (2013). [11] Lin S., Lee E., Nguyen N., Khine M., “Thermally‐induced miniaturization for micro‐ and nanofabrication: progress and updates,” Lab Chip, 14(18), 3475‐3488 (2014).
132
[12] Fan J. A., Yeo W. H., Su Y., Hattori Y., Jung S. Y., Zhang Y., Liu Z., Cheng H., Falgout L., Bajema M., Coleman T., Gregoire D., Larsen R., Haung Y., Rogers J., “Fractal design concepts for stretchable electronics,” Ant. Commun. 5, 3266 (2014). [13] Lu, N., Wang, X., Suo, Z., Vlassak, J., “Metal films on polymer substrates stretched beyond 50%,” Appl. Phys. Lett., 91, 221909 (2007). [14] Wagner, S., Lacour, S. P., Jones, J., Hsu, P. I., Sturm, J. C., Li, T., and Suo, Z., “Electronic skin: architecture and components,” Physica E, 25(2‐3), 326‐334 (2004). [15] Kang D., Pikhitsa P., Choi Y. W., Lee C., Shin S. S., Piao L., Park B., Suh K. Y., Kim T. I., Choi M., “Ultrasensitive mechanical crack‐based sensor inspired by the spider sensory system,” Nature. 516, 222‐226 (2014). [16] Ohring, M., [The Materials Science of Thin Films], Academic Press, San Diego, ch. 10 (1992). [17] Hoffmann, H., and Fischer, G., “Electrical Conductivity in Thin and Very Thin Platinum Films,” Thin Solid Films, 36(1976), 25‐28 (1975). [18] Lacy, F., “Developing a theoretical relationship between electrical resistivity, temperature, and film thickness for conductors,” Nanoscale Res Lett, 6, 636 (2001). [19] Gerratt, A. P., Michaud, H. O., Lacour, S. P., “Elastomeric Electronic Skin for Prosthetic Tactile Sensation,” Adv. Funct. Mater., 25(15), 2287‐2295 (2015). [20] Smith, M. [Neural Networks for Statistical Modeling], Van Nostrand Reinhold, New York, ch. 1 (1993).
133
CHAPTER 9: Development of Glove Mounted Stretchable Sensor Quadcopter Controller
Having proved in chapter 8 that joint angle can be discerned from measured ADC values using a
microcontroller, the glove and sensor were turned into a throttle controller for a quadcopter by
replacing the traditional transmitter and receiver. This process involved studying and replicating the signal profile of a traditional transmitter, incorporating touch sensors into the glove, and miniaturizing the necessary electronics so they could be contained in a pouch on the glove.
9.1 Quadcopter Assembly
The quadcopter assembly included a carbon fiber frame, 3D printed electronic speed controller
(ESC) mounts, ESCs, motors, battery, NAZA flight controller, Arduino Uno, an Arduino Uno XBee
shield, and an XBee as shown in Figure 9.1 and Error! Reference source not found.. Exact
components and the dimensions of the ESC mounts are called out in appendix 18. The flight controller was situated on top of the quad and the XBee and Arduino pair on the underside of
the quad as shown in Figure 9.3 so that a line of sight between communicating XBees would be maintained while quad was airborne.
134
Figure 9.1 Top view of quadcopter (no propellers)
Figure 9.2. Arduino XBee shield on underside of quad
135
Figure 9.3 Side view of quadcopter showing NAZA flight controller on top and Arduino with XBee shield on bottom powered by main battery using barrel jack adapter
9.2 Signal Flow
The first step in replacing the throttle capabilities of a traditional transmitter was to understand the type of signals involved and ultimately their influence on the quadcopter. On a traditional
four channel transmitter, shown in Figure 9.4, two joystick style sticks control four quadcopter
inputs: throttle, roll, pitch, and yaw. The reason the glove controller is replacing throttle control
is due to the nature of the inputs and the biomechanics of fingers. At throttle minimum the quadcopter will be grounded. As the throttle is increased, the quadcopter will lift off and hover.
This can be easily mimicked by the MPC finger joint with an unbent finger joint corresponding
to throttle minimum and a fully bent finger joint corresponding to throttle maximum. The other
three inputs are more challenging to consistently replicate from a performance standpoint as
they are initially in a neutral position which would correspond to a joint being initially bent half
136
way and increasing or decreasing the joint angle correspondingly increasing or decreasing the
corresponding roll, pitch, or yaw input.
Figure 9.4 Traditional transmitter control sticks and corresponding functions
Through radio frequency (RF) communication, the transmitter sends a signal to the receiver based on throttle, roll, pitch, and yaw joystick positioning. The receiver passes along this information to a flight controller which then, using its own internal control algorithm, passes along inputs to each of the four motors. Therefore, to successfully replace the original
transmitter and receiver pair with the new glove and receiver pair, the signals the new pair
sends to the flight controller must replicate those of the original. To figure out these signals the
original receiver was hooked up to an oscilloscope and the sticks of the original transmitter
were moved over their full range of motion. This revealed that the receiver sends a pulse
position modulated (PPM) signal, width of approximately 20 ms, to the flight controller where
the minimum, neutral, and maximum pulse widths are approximately 1000 ms, 1520 ms, and
2000 ms respectively. Figure 9.5 shows a collected representative signal from all four channels
with the transmitter sticks all in a neutral position. These types of signals were able to be
137
replicated using the Arduino servo library and specifically the writeMicroseconds() function as
shown in appendix 20.
Figure 9.5 Collected PPM signals with all transmitter sticks in a neutral position
Given that the necessary PPM signals could easily be replicated using an Arduino, an Arduino was chosen to replace the signal generating element of the receiver. Further, RF communication could easily be achieved by using XBees and the appropriate Arduino XBee shield. Signal transmission, detailed latter, was achieved using an ATmega328p microcontroller
located on the glove assembly, and XBee where the transmitting and receiving XBees were
paired. From the receiving Arduino, as shown in Figure 9.6, four digital pins (white wires) were
connected to the four flight controller inputs: aileron (roll), elevator (pitch), throttle, and
rudder (yaw). Four ground wires (black) were also connected. The outputs of the flight
controller were connected to the four motor electronic speed controllers (ESCs) in the standard
fashion.
138
Figure 9.6 Flight controller with signal (white) and ground (black) inputs from Arduino.
9.3 Glove Assembly
The final glove assembly, shown in Figure 9.7, consists of two touch sensors, the bimetallic foil
gauge bonded to PDMS and affixed above the MPC joint on the right index finger as detailed in
chapter 8, and a custom developed PCB with necessary electronics which fit inside a pouch atop
the Velcro strap; a detailed parts list and schematic of the circuit are given in appendices 21 and
22. The electronics include an ATmega328p microcontroller, a bridge and op amp for the sensor as discussed in chapter 8, resistors for the touch sensors, battery mount, switch, linear voltage
regular, connectors for interfacing all sensor leads, female headers for attaching the
transmitting XBee, and recommended capacitors. Figure 9.8 shows the size perspective of the
surface mount PCB, XBee, and battery.
139
Figure 9.7 Glove with electronics showing (left) and tucked in pouch on strap (right).
Figure 9.8 Electronic components sizing perspective. Left to right: quarter, PCB with soldered elements, XBee, and lithium ion polymer battery
The two touch sensors, shown in Figure 9.9, are capacitance based and made from conductive fabric pads where conductive thread has been stitched through each one, soldered to 26 gauge hook up wire, and connected to the PCB using a pico‐lock Molex connector. The reason for the sensors are the flight controller’s motor start/stop commands. For safety, the flight controller
140
will not send a signal to the ESCs, and subsequently the motors, unless it first receives a motor
start command; for the flight controller being used there are four possible controller configurations to generate this command. These four configurations also function as a stop
command once the quadcopter is in operational mode. The four configurations, given as left
stick position/right stick position combinations with reference to a traditional controller, are
bottom left/bottom left, bottom right/ bottom right, bottom left/bottom right, and bottom
right/ bottom left. Given that the MCP joint angle sensor is only controlling the throttle input, it
is incapable of generating the required start or stop command. However, since the signal
pattern (PPM pulse widths) being sent to the flight controller for each of these configurations is known, it can be simulated. The purpose of the touch sensors are to toggle the operational mode of the quadcopter. The touch sensor attached to the index finger of the glove triggers the
simulation of the start/stop command while the touch sensor attached to the middle finger of
the glove puts the quadcopter in operational mode. In operational mode the MPC joint angle sensor controls the throttle signal while roll, pitch, and yaw are fixed neutral signals.
Figure 9.9 Touch sensors on glove
141
9.4 Operational Flow
While the exact code for the transmitting ATmega328p on the PCB can be found in Appendix
19, the system operates as follows. The state of the controller is set by touching either of the
touch sensors. Upon initial startup the controller is in zero state. Upon touching the sensor on
the index finger the MCU transmits the value of 100 for each the roll, pitch, throttle, and yaw
values corresponding to the arm/disarm state. On the receiving end this value is multiplied by
10 and fed to the flight controller. A value of 1000 corresponds to both controllers being in the
bottom left position, a start/stop configuration. The reason a value of 100 was sent and not
1000 is because the ATmega328p and Arduino use serial communication and the max single transmittable value is 255 (1 byte). All transmitted values are in effect predivided by 10 to fit
into a single byte and multiplied by 10 on the receiving end as all values fed to the flight
controller are between 1000 and 2000 and hence scalable to between 100 and 200. Losing the
precision of the 1’s place was not found to adversely affect performance. Once the quadcopter
motors start spinning the sensor on the middle finger is touched and the controller state is set
to operational mode: a neutral value of 152 is transmitted for roll, pitch, and yaw while the
transmitted throttle value is scaled in proportion to discerned joint angle based on ADC
readings and corresponding look up table (LUT) values. To stop the quadcopter the sensor on
the index finger is touched again. This entire processes is shown in the flow diagram detailed by
Figure 9.10.
142
Figure 9.10 Operational state process flow (open loop)
9.5 Single Curve Model Based Look Up Table
A motion study similar to that detailed in chapter 8 was performed to discern the relationship
between measured ADC values by the MUC, the corresponding joint angle, and whether the
finger was in flexion or extension. For throttle control a single curve was used, as shown in
Figure 9.11, as opposed to two curves as done previously in chapter 8. Initially a two curve model was used, one for flexion (bottom curve) and one for extension (top curve), where a two
column look up table was generated correlating ADC values to PPM commands (divided by 10)
scaled by corresponding joint angle. Which column, and correspondingly which curve, was used
143
was determined by the average slope of recently measured values. However, due to noise in the sensor readings, the sign of the slope jumped between positive and negative, resulting in a jump between columns of the LUT, and ultimately choppy flight behavior. For the single curve model, the average error between the flexion and model curve is 1.94 degrees and the average
error between the extension and model curve is 5.28 degrees where the total average error is
3.69 degrees as defined in the inset of Figure 9.11. The equation of the single logistic curve
model is given in (7.1) where x is the MPC joint angle.
ADC( x ) 1023 250 g x 250 1 gx 1 eaax01 (7.1)
a0 2.05
a1 0.095
Also observed in Figure 9.11 is that the ADC saturates at its max value of 1023 (10 bit) at a joint angle of roughly 66.21 degrees for the given sensor and amp gain configuration. However, in general, only slight sensor output variances are observed beyond a joint angle of 60 degrees as
also seen in Figure 8.12. Therefore, for the single curve model, a single column, 1023 row, LUT
was generated where the row number corresponds to measured ADC value and the row value
corresponds to PPM value divided by 10, range 100 to 200, scaled proportionally by joint angle
between 0 and 66.21 degrees.
144
Figure 9.11 Single curve model for MCP joint flexion and extension with inset showing error
145
CHAPTER 10: Conclusions
This dissertation has detailed the design, development, and operational theory of two gesture
recognition based controllers using PVDF and elastomer bonded bimetallic thin foil gauge
sensors and the fabrication and analysis of a self‐sensing bimorph beam using PVDF and its
electrostrictive terpolymer P(VDF‐TrFE‐CFE), respectively.
Motion of the human hand is controlled by the contraction of muscles located in the forearm. A single distributed, spatially shaded, PVDF element located over the forearm extensor muscles has been shown, when coupled with a time delay neural network, to accurately and repeatedly identify right and left gestures performed by the right hand. The same setup also attempted to
identify fist, flex, and extend hand motions but was not able to do so with satisfactory results.
The development of a model of the system allowed for an understanding of how the forces
generated by the contracting muscles acted on the PVDF and how spatial shading of the sensor
could enhance the performance of the sensor. By feeding the DC offset voltages generated by
the PVDF in response to muscle contraction induced strain generated by right and left gestures
into a time delay neural network and using backpropagation the network was able to learn the
signal attributes of right and left gestures and, in real time, discern between the two. Using a
microcontroller based chip, an Arduino Pro Mini, and Bluetooth connectivity allowed the
controller to be wireless, untethered from DAQ development boards, and truly wearable.
Software applications were developed in conjunction with the wireless controller including an
oscilloscope style Android app capable of plotting real time read voltage values and identifying
gestures and a Power Point Add‐In which allowed the controller to navigate the presentation.
146
Using the developed Android app for verification, the device was shown to 100% reliable in
testing.
PVDF was also used as the sensing layer in conjunction with its electrostrictive terpolymer
P(VDF‐TrFE‐CFE) in a fabricated bimorph cantilevered beam with the intent to demonstrate that tip deflection could be determined without the need for external measuring devices and hence
such a device could act as a self sensing microswitch in MEMS applications. While the primary
function of the PVDF was sensing and that of the terpolymer actuation, the PVDF was also used
to drive the beam to demonstrate the benefits of electrostriction in terms of increased tip
deflection. Through the use of Euler‐Bernoulli beam theory and linearized piezoelectric equations, a relationship between the voltage measured and the deflection of the tip was
determined. The beam was actuated across the terpolymer over a range of frequencies and
calculated tip deflections compared to those measured using an LDV. While the results do not
closely match it is believed that the discrepancy exists as a combination of the difference in
actual versus standard piezoelectric coefficients used in the deflection calculations as well as focusing issues with the LDV as the general shape of the calculated deflection curve matches more closely with theory than that of the LDV deflection curve.
PVDF is inherently a velocity dependent transient sensor and due to leakage needs to be coupled with specialized electronics, such as a charge amplifier, to be able to give static measurements. It is also a stiff polymer and hence unsuitable for such biomechanical measurements as joint angles. While traditional metal foil gauges fail around five percent strain, developed elastomer bonded bimetallic foil gauges using the piezoelectric polymer PS
147
with induced micro and nano wrinkles has been shown to maintain conductivity when strained
repeatedly up to 23.5 percent; although not shown the sensor was conditioned by being
strained up to 47 percent without loss in conductivity. Further, the modulus of the elastomer,
specifically PDMS, is such that it does not inhibit motion and, as such, was bonded to the right
index finger of a glove over the MCP joint to measure the degree of bend of said joint. The sensor was configured as the variable element in a standard quarter Wheatstone bridge circuit
in conjunction with a differential Op‐Amp whose output was measured by a MCU. A motion study allowed this measured value to be correlated with the angle of the MCP joint. Due to the viscoelastic nature of the elastomer the result of the motion study revealed two curves were needed to most accurately describe the sensor’s behavior and could be modeled using logistic
functions with an average error of 2.99 degrees.
To demonstrate the capabilities of the glove and sensor system, it was tasked with controlling
the throttle input to a quadcopter controller and hence replaced a traditional controller.
Accordingly, necessary transmitter and receiver hardware and software was developed where
the two curve model correlating sensor output to joint angle was replaced by a single logistic curve model, average error of 3.69 degrees, trading accuracy for operational smoothness.
148
Appendix 1: Tensors and PVFD Elastic Coefficient Matrix Reduction
From mechanics of materials Hooke’s Law relates the two states of a linear elastic material; stress and strain. Expressing Hooke’s Law where stress and strain are each written as second order tensors related by the fourth order modulus tensor [1]
TcSkl klmn mn klmn , , , 1,2,3 (A1.1)
where the indices on T and S correspond with the face and direction of the material
illustrated in Figure A1.1. Tensors are mathematical objects that are invariant under admissible
transformations and hence obey specific transformation laws.
Tensors in 3D Euclidian space have 3N elements where N is the rank of the tensor; the stress
and strain tensors each have 9 elements and the compliance tensor has 81 elements.
Considering again Figure A1.1 a general 2D stress element with infinitesimal lengths and taking
the sum of the moments about the 3 axis, assuming clockwise positive, with no net moment
acting on the element reveals
M3 0
Tddd2113 2 Tddd 21 23 1 0 (A1.2)
TT21 12
where the terms inside the parentheses are the force (stress by area) and the multiplier is the
moment arm; this can be expanded to show symmetry in the other shear stresses as well and
149
thus the stress tensor is symmetric. Further, it is assumed, and can be shown that since the
stress tensor is symmetric the strain tensor is also symmetric [1],[2].
Figure A1.1 3‐D and 2‐D stress element
The following shorthand notation will be used to simplify analysis using this found symmetry
TT111 S 1 S 11
TT222 S 2 S 22 TT S S 333 3 33 (A1.3) TT4 23 T 32 S 4 S 23 S 322 S 23
TT5 31 T 13 S 5 S 31 S 132 S 13
TT6 12 T 21 S 6 S 12 S 212 S 12
where the stress and strain tensors are expressed as rank one tensors (vectors) and the modulus tensor expressed as a rank two tensor (matrix) with 36 terms as shown in (A1.4) where the shear strains terms are defined as the average of the two shear strains.
150
T1 ccccccS 1112131415161 T ccccccS 2 2122232425262 T3 ccccccS 3132333435363 (A1.4) T4 ccccccS 4142434445464 T ccccccS 5 5152535455565 T6 ccccccS 6162636465666
According to Kawi [3], PVDF has the same symmetry as orthorhombic mm2 crystals and hence
is classified as an orthotropic material [4]. Orthotropic materials are a subset of anisotropic
materials which have a maximum of 21 independent elastic constants proven through the
existence of the strain energy density function W [5]. The strain energy density function is
associated with potential energy stored in an elastic material due to deformations, is a function
of the elements of the strain vector and defined as
WSSSSSS 123456,,,,, (A1.5) WCSSC11SST 22ij i j
with the associated property
W TCCS1 () S iijjij2 i (A1.6) W TS1 ()CC T ST 2
However, from Hooke’s Law we know
TCS iijj (A1.7) TS C
151
Meaning that the modulus matrix is symmetric
CCij ji (A1.8) CC T
Where if follows
cccccc11 12 13 14 15 16 cccccc 12 22 23 24 25 26 cccccc13 23 33 34 35 36 C (A1.9) cccccc14 24 34 44 45 46 cccccc 15 25 35 45 55 56 cccccc16 26 36 46 56 66
Orthotropic materials, by definition, have two planes of material symmetry which result in nine independent elastic constants. Using a prime to denote a transformation permissible of an
orthotropic material, Hooke’s Law in vector‐matrix notation for the original and transformed
state are
TS C (A1.10) TS'' C
showing that the stresses and strains are transformed but the modulus matrix is not. Using symmetry in the 1‐2 plane, illustrated in Figure A1.2
' xx11 10 0 '' xx22xx AA, 0 1 0 (A1.11) ' 00 1 xx33
152
Figure A1.2 Symmetry in the 1‐2 plane
where A is the corresponding rotation matrix. The second order tensor transformation law is
shown in (A1.12) where a second order tensor element tkl is being operated on by the appropriate elements of the direction cosine matrix term by term [1]. The second line of
(A1.12) is a vector matrix representation where T is the original second order tensor in vector form and A is the direction cosine matrix.
taatijkl' , , , 1,2,3 ij ik jk kl (A1.12) TATA' T
Writing stress and strain as second order tensors
TTT165 S 165 S S TTTTSSSS624, 624 (A1.13) TTT543 S 543 S S and using (A1.12)
153
''' TTT165TT T 16 5 ''''T TATATTT624 T 6 T 2 T 4 '''TTT TTT543 543 (A1.14) ''' SSS165SS S 16 5 ''''T SASASSS624 S 6 S 2 S 4 '''SSS SSS543543
Applying (A1.10) and using the fact that the modulus matrix is symmetric
T1 ccccccS 1112131415161 T ccccccS 2 1222232425262 T3 ccccccS 1323333435363 T4 ccccccS 1424344445464 T ccccccS 5 1525354555565 T6 ccccccS 1626364656666 (A1.15) T1 cccccc11 12 13 14 15 16 S 1 T cccccc S 2 12 22 23 24 25 26 2 T3 cccccc13 23 33 34 35 36 S 3 T4 cccccc 1424344445464 S T cccccc S 5 1525354555565 T6 ccccccS 1626364656666
Expanding (A1.15) and examining the resulting 12 equations it can be seen that symmetry
about the 1‐2 plane implies that
cccccccc14 15 24 25 34 35 46 56 0 (A1.16)
and the modulus matrix for an elastic material with symmetry in the 1‐2 plane can be redefined
as
154
ccc11 12 1300 c 16 ccc00 c 12 22 23 26 ccc13 23 3300 c 36 C (A1.17) 000cc44 45 0 000cc 0 45 55 ccc16 26 3600 c 66
Repeating for symmetry in the 1‐3 plane
' xx11 100 '' xx22 xx AA, 0 1 0 (A1.18) ' 001 xx33
''' TTT165TTT 165 ''''T TATATTT624 TT 6 2 T 4 '''TTT TTT543 543 (A1.19) ''' SSS165SSS 165 ''''T SASASSS624 S 6 S 2 S 4 '''SSS SSS543543
and incorporating the results from symmetry in the 1‐2 plane
155
Tccc111121300 cS 161 Tccc00 cS 2 122223 262 Tccc313233300 cS 363 TccS444454000 0 TccS000 0 545555 Tccc616263600 cS 666 (A1.20) Tcc11112ccS 1300 16 1 T ccc00 c S 2 12 22 23 26 2 T3 ccc13 23 3300 c 36 S 3 T4 000cc44 45 0 S 4 T 000cc 0 S 5 45 55 5 T6 ccc16 26 3600 c 66 S 6
This requires
cccc16 26 36 45 0 (A1.21)
and yields the elastic constant (modulus) matrix with nine independent constants (A1.22).
ccc11 12 13 000 ccc000 12 22 23 ccc13 23 33 000 C (A1.22) 000c44 00 0000c 0 55 00000c66
References
[1] Y. C. Fung, A first course in continuum mechanics, 2d ed. Englewood Cliffs, N.J: Prentice‐ Hall, 1977. [2] E. Popov, Introduction to Mechanics of Solids, 9th ed. New Jersey: Prentice‐Hall, INC., 1968.
156
[3] H. Kawai, “The Piezoelectricity of Poly (vinylidene Fluoride),” Jpn. J. Appl. Phys. Japanese Journal of Applied Physics, vol. 8, no. 7, pp. 975–976, 1969. [4] J. Roesler, H. Harders, and M. Baeker, Mechanical Behaviour of Engineering Materials ‐ Metals, Ceramics, Polymers, and Composites. [5] I. Sokolnikoff, Mathematical Theory of Elasticity, 2nd ed. McGraw‐Hill Book Company, Inc., 1956.
157
Appendix 2: Mode Shapes of a Beam
The system shown in Figure A2.1, a simply supported beam assumed to be liner and elastic subject to transverse and axial loading representing an approximation of the compression sleeve discussed in section 1.3, is decomposed into a beam in bending vibration and a rod in axial vibration. Both a beam and rod are distributed self‐adjoint systems and, as a direct consequence, have orthogonal eigenfunctions (mode shapes) and equations of motion
expressed through eigenfunction expansions [1],[2].
Figure A2.1 Compression sleeve approximation
Appendix 2.1: Mode Shapes of Beam in Bending Vibration
The governing partial differential equation of a uniform Euler‐Bernoulli beam subject to no
loading is [1]
22wxt(,) 2 wxt (,) 22EI m 2 (A2.1) xx t
158
where wxt(,) is the transverse displacement, EI is the bending stiffness ( E being Young’s
modulus and I the area moment of inertia normal to the plane created by x and w ), and m the mass per unit length. Boundary conditions for the roller‐roller configuration are
2wxt(,) wxt(,) 0 , 0 (A2.2) xL0, 2 x xL0,
Under synchronous motion the solution to (A2.1) is separable into the spatial variable x and time (method of separation of variables) as
wxt(,) W () xFt () (A2.3)
Plugging (A2.3) into (A2.1) and manipulating results in
22 2 EI d d W() x 1 d F () t 2 22 2 (A2.4) mW() xdx dx F () t dt
where will be seen to be the frequency of oscillation. Concerning ourselves only with the mode shapes of the system yields the differential eigenvalue problem (for self‐adjoint systems)
ddWx22() 2 m 22 Wx() (A2.5) dx dx EI
which can be rewritten as
42Wx() m 44Wx() 0 , (A2.6) x4 EI
159
The general solution of (A2.6) is
Wx( ) C12 sin( x ) C cos( x ) C 3 sinh( x ) C 4 cosh( x ) (A2.7) with second derivative
2Wx() 22CxCxCsin( ) cos( ) 2 sinh( xC ) 2 cosh( x ) (A2.8) x2 12 3 4
Applying boundary conditions at x 0
WCC(0) 24 0 2W (0) (A2.9) CC 0 x2 24
so CC240. Applying boundary conditions at x L
WL( ) C13 sin( L ) C sinh( L ) 0 2WL() (A2.10) CLCLsin( ) sinh( ) 0 x2 13
Adding these two equations together
2sinh()0CL3 (A2.11)
However, sinh( ) is only equal to zero at 0 . So, for arbitrary and nonzero L (A2.11) is
only satisfied if C3 0 which means that C1 is arbitrary and is set to 1 meaning
Wx() sin( x ) (A2.12)
160
To satisfy the boundary conditions
WL() sin() L 0 n Ln, (A2.13) L n Wx() A sin( x ) n L
n where A is a scaling factor and sin(x ) is the nth eigenfunction or mode shape. The last line n L
in (A2.13) is traditionally normalized by mass [3], however a standard normalization is used
such that
L WxWxdxnm() () nm (A2.14) 0
where nm is the Kronecker delta (orthogonality of eigenfunctions). For nm (A2.14) yields
L 22n Axdxn sin ( ) 1 (A2.15) 0 L
1 2 which, using the trig identity sin2 (1 cos 2 ) gives A and the final mass 2 n L
normalized nth eigenfunction
2 n Wx sin( ) (A2.16) n LL
The first four mode shapes for An 1 are shown in Figure A2.2.
161
Figure A2.2 First four mode shapes of simply supported beam in bending vibration
Appendix 2.2: Mode Shapes of Rod in Axial Vibration
The governing partial differential equation of a uniform rod in axial vibration subject to no
loading is [1]
uxt(,) 2 uxt (,) EA m (A2.17) xxt2
where EA is the axial stiffness, uxt(,) is the axial displacement, and m the mass per unit
length. Under synchronous motion the solution to (A2.17) is again separable into the spatial variable x and time (method of separation of variables) as
uxt(,) UxFt () () (A2.18)
162
Plugging this into (A2.17) and again only concerning ourselves with the mode shapes of the
system yields
ddUx() EA2 mU() x (A2.19) dx dx
which is the differential eigenvalue problem of the rod in axial vibration and belongs to the class of Sturm‐Liouville problems. The roller‐roller end conditions are, for axial vibration,
equivalent to free‐free end conditions and hence have the boundary conditions
dU() x 0 (A2.20) dx xL0,
The differential equation in (A2.19) is rewritten as
dm22 Ux()22 Ux () 0 , (A2.21) dx2 EA
which has the general solution and first differential of
Ux( ) C12 sin( x ) C cos( x ) d (A2.22) Ux() C cos( x ) C sin( x ) dx 12
Plugging in the boundary conditions yieldC1 0 and C2 arbitrary (set to 1) such that
n sin(Ln ) 0 , , 1,2,... L (A2.23) n Ux() A cos( x ) r L 163
with Ar again an arbitrary scaling term and Ux() an expression for the eigenfunctions of the
system with corresponding eigenvalues . In (A2.23) n 0 corresponds with rigid body motion
which is not of interest to us so the iteration starts at 1. The eigenfunctions are again
normalized according to
L UxUxdxnm() () nm 0 L 22n Axdxn cos ( ) 1 (A2.24) 0 L 2 n Ux( ) cos( x ) , n 1,2,... LL
The first four mode shapes for An 1 are shown in Figure A2.3.
Figure A2.3 First four mode shapes of rod in axial vibration
164
References
[1] L. Meirovitch, Principles and techniques of vibrations. Upper Saddle River, N.J: Prentice Hall, 1997. [2] P. Berg and J. McGregor, Elementary Partial Differential Equations. Oakland, California: Holden‐Day. [3] L. Meirovitch, Elements of vibration analysis. New York: McGraw‐Hill, 1975.
165
Appendix 3: Beam in Bending Stress‐Displacement Equations
As mentioned throughout the body of the dissertation and in Appendix 2, an Euler‐Bernoulli beam model is being used. The kinematic assumption valid for small strains in the Euler‐
Bernoulli beam models allows that plane cross sections through a beam taken normal to its neutral axis remain plane after bending and its corollary that in said beam, strains in its fibers
vary linearly with their respective distances from the neutral axis [1],[2]. Using the (1, 2, 3) coordinate system common to describing piezoelectric materials a point is described by the tuple (,x123xx , ) and the displacement vector u is defined as
uxxx112(, , 3 ) u uxxx(, , ) 2123 (A3.1) uxxx3123(, , )
Under transverse loading a beam bends and thus deflects from its initial position. The goal is to
derive a relationship between this loading (stresses) and the deformation or displacement of
the beam. Let a beam subject to some plane stress along the 3 direction induce the bending
shown in Figure A3.1. All displacements and strains are assumed infinitesimal. Therefore, the
angle of deflection of the beam is small and can be related to displacement u1 as
ux13 (A3.2)
where x3 is the distance from the neutral axis (in the negative direction).
166
Figure A3.1 Beam segment subject to bending
Further, from the geometry shown in Figure A3.2 the angle of deflection is the localized slope of the beam and is expressed as
du 3 (A3.3) dx1
Figure A3.2 Local slope of beam segment
Plugging (A3.3) into (A3.2) yields an expression for the displacement in the 1 direction
du3 ux13 (A3.4) dx1
The strain tensor, in reduced vector form, is related to the displacements through a differential
operator [2],[3]
167
u 1 00 x x 1 1 u 2 00 S x x 11 2 2 S 22 u3 00 u1 S33 x3 x3 S uL u (A3.5) 2Suu 2 u 23 2 3 0 u 3 2S13 xx32 xx32 2S u u 12 1 3 0 xx31xx31 uu 120 xx21xx21
Finally, the applied stresses can related to the displacements using Hooke’s law.
References
[1] Y. C. Fung, A first course in continuum mechanics, 2d ed. Englewood Cliffs, N.J: Prentice‐ Hall, 1977. [2] E. Popov, Introduction to Mechanics of Solids, 9th ed. New Jersey: Prentice‐Hall, INC., 1968. [3] D. Leo, Engineering Analysis of Smart Material Systems, 1st ed. New Jersey: John Wiley & Sons, Inc., 2007.
168
Appendix 4: Material Properties
The material properties of the compression sleeve and the PVDF used for simulations are listed below.
Appendix 4.1: Compression Sleeve
Mechanical properties of the compression sleeve based on tensile test results discussed in section 6.1
Table A4.1 Mechanical properties of compression sleeve 3 ’ 5 Thickness [m] ρ [kg/m ] ν12 ν13, ν23 E [MPa] ε0 Δε E [MPa] ζ 0.0004 521.6 0.2 0.375 0.535 0.375 0.0182 22.6 0.2
Appendix 4.2: PVDF
Material properties of PVDF used in the simulations were taken to be consistent with the literature and are as follows [1],[2],[3].
Table A4.2 PVDF Properties Density 1.78e3 [kg/m3]
‐6 Thickness t p 52e [m]
Capacitance 380e‐6 [F/m2] Dielectric Permittivity 110e‐12 [F/m]
5 Chosen to match experimental data 169
Table A4.3 PVDF compliance [1/Pa] e‐12 E 365 s11 E ‐110 s12 E ‐209 s13 E 424 s22 E ‐192 s23 E 472 s33
Table A4.4 PVDF strain coefficient [C/N] e‐12
d31 ‐21
d32 ‐2.3
d33 26
d24 27
d15 23
References
[1] Measurement Specialties, Inc., “Piezo Film Sensors Technical Manual,” Mar‐2008. [2] R. S. Dahiya, Robotic tactile sensing: technologies and system. Dordrecht ; New York: Springer, 2013. [3] D. M. Esterly, “Manufacturing of Poly(vinylidene fluoride) and Evaluation of its Mechanical Properties,” Virginia Polytechnic Institute and State University, 2002.
170
Appendix 5: PVDF Voltage Output from Transverse Beam Deflection Matlab Code
% PVDF output due to bending (transverse) clc; clear all; %close all; format compact; format shortg;
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% Dealing with Symbols % digits() and vpa() deal format output that is affected by using symbols digits(12) %set number of digits in mixed number/variable format
%% Geometry % ~Common L=0.08; % [m] 1-direction %Note that by changing dx and correspondingly dt, the magnitude of voltage %output changes (more force being "added", more x nodes included). The %shape of the voltage profile DOES NOT change dx=2.5e-5; % [m] for Logistic: must be on scale of dxdt*dt x=0:dx:L; % [m] w=0.03; % [m] 2-direction % ~PVDF Patch tp=52e-6; %[m] using 52 micrometer, ignoring thickness of electrodes % ~Structure (Sleeve) t=0.0004; %[m] thickness (along z) I=(1/12)*w*t^3;
%% Material Properties % ~Structure (Sleeve) Ys_0=0.535e6; % [Pa] from MTS data (data 1/28/14) eps0=(27.5-20)/20; % Initial strain in sleeve from puttin on deps=(28-27.5)/27.5; % Change in strain, treating eps0 as base Ys=(deps*Ys_0*eps0+0.5*deps^2*Ys_0)/(0.5*deps^2); % Adjusted modulus v12=0.2; %Poisson's ratio from MTS experiment v13=0.375; %Poisson's ratio from MTS experiment rho_s=521.6; %[kg/m^3] Calculated density of sleeve from measured mass and volume syms s44 s55 s66 real %Leave as symbolic, b/c of form only multiplied by zeros and drop out s_sleeve=1/Ys*[1 -v12 -v13 0 0 0 -v12 1 -v13 0 0 0 -v13 -v13 1 0 0 0 0 0 0 s44 0 0 0 0 0 0 s55 0 0 0 0 0 0 s66]; c_sleeve=eye(6)/s_sleeve; %compliance matrix
171
% ~PVDF: Combination of sources % ~~~(MS): Measurement Specialties Tech Manual % ~~~(Th): Thompson Dissertation which cites Measurement Specialties Tech Manual rho_p=1.78e3; %[kg/m^3] (all sources agree) Cap=380e-6; %[F/m^2] ***Converted to /m^2 from (MS) %Matrices of PVDF -> Orthorhombic mm2 symmetry (Fdamtls of Piezo Sensorics) %s^E coeffs [1/Pa]x10^(-12) (Th, agrees with Robotic Tactile Sensing Appendix A) syms s44_E s55_E s66_E real s11_E=365; Yp=1/(s11_E*1e-12); %[Pa] s12_E=-110; s13_E=-209; s22_E=424; s23_E=-192; s33_E=472; v12_E=-s12_E/s11_E; v13_E=-s13_E/s11_E; s_E=(1e-12)*[s11_E s12_E s13_E 0 0 0; s12_E s22_E s23_E 0 0 0; s13_E s23_E s33_E 0 0 0; 0 0 0 s44_E 0 0; 0 0 0 0 s55_E 0; 0 0 0 0 0 s66_E]; %d[3x6] Piezo strain constant [C/N]x10^-12 (Es) d31=21; d32=2.3; d33=-26; d24=-27; d15=-23; d=(-1e-12)*[0 0 0 0 d15 0; 0 0 0 d24 0 0; d31 d32 d33 0 0 0]; % added negative % Eps^T - dielectric permittivity [F/m] Eps_T=110e-12*eye(3); % Beta^T - impermitivity [m/F] Beta_T=eye(3)/Eps_T; %g[3x6] Piezo stress constnat [m^2/C] g=Beta_T*d; % s^D s_D=s_E-d'*Beta_T*d; % c^D c_D=vpa(eye(6)/s_D); % h[3x6] h=g*c_D; %Beta_S Beta_S=vpa(Beta_T+g*h');
%% Mode Shapes Nm=10; %number of mode shapes
%% Ritz Formulation Transverse Deflections % Mode shape normalized st integral 0 to L p(i)*p(j)=1
% Building Mass Matrices % Building PHI(x) syms xx yy zz real
172
for i=1:Nm %i corresponds to rows p_i=sqrt(2/L)*sin(i*pi*xx/L); %mode shape for j=1:Nm %j corresponds to columns p_j=sqrt(2/L)*sin(j*pi*xx/L); %mode shape PHI_x(i,j)=p_i*p_j; end end % Ms (mass of sleeve) = rho_s*w*t Ms=int(int(int(rho_s*PHI_x,zz,-t/2,t/2),xx,0,L),yy,-w/2,w/2); % Mp1 (mass of piezo 1) Mp=vpa(int(int(int(rho_p*PHI_x,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building Stiffness Matricies % Building Br_s(x) for j=1:Nm %columns p=sqrt(2/L)*sin(j*pi*xx/L); %mode shape -->Normalized for i=1:6 %rows if i==1 Br_s(i,j)=zz*diff(p,xx,2); elseif i==2 Br_s(i,j)=-v12*zz*diff(p,xx,2); elseif i==3 Br_s(i,j)=-v13*zz*diff(p,xx,2); else Br_s(i,j)=0; end end end % Building Br_p(x) for j=1:Nm %columns p=sqrt(2/L)*sin(j*pi*xx/L); %mode shape -->Normalized for i=1:6 %rows if i==1 Br_p(i,j)=zz*diff(p,xx,2); elseif i==2 Br_p(i,j)=-v12_E*zz*diff(p,xx,2); elseif i==3 Br_p(i,j)=-v13_E*zz*diff(p,xx,2); else Br_p(i,j)=0; end end end % Ks (stiffness of sleeve) Ks=vpa(int(int(int(Br_s'*c_sleeve*Br_s,zz,-t/2,t/2),xx,0,L),yy,-w/2,w/2)); % Kp^D1 (stiffness of PVDF 1) Kp_D=vpa(int(int(int(Br_p'*c_D*Br_p,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building Coupling Matrix %Building Bq(x) Bq=[0; 0; (4/L^2)*xx*(L-xx)/(w*L)];
%Theta_1 Theta=vpa(int(int(int(Br_p'*h'*Bq,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building "Inverse" Capacitance Matrix
173
%Cp1^s^-1 Cp_S_inv=vpa(int(int(int(Bq'*Beta_S*Bq,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2)); %Cp1^s Cp_S=eye(length(Cp_S_inv))/Cp_S_inv;
%% Mass and Stiffness Matrices M=Ms+Mp; K=(Ks+Kp_D-Theta*Cp_S*Theta')./3.4;
%% Building Ds (damping matrix) from structure mass and stiffness Ds=zeros(Nm,Nm); zeta=0.1; dm=1:1:10; for i=1:Nm for j=1:Nm if i==j %using a constant damping ratio for all modes Ds(i,j)=2*zeta*dm(i)*sqrt(M(i,j)*K(i,j)); end end end
%% Building Bf(x) disp('Building Bf(x)') % Need to alter these % fa=0.01; % fb=0.07;
% For Traveling Logistic: fa=0.025; fb=0.07;
Bf=zeros(Nm,round((fb-fa)/dx+1)); xFcord=zeros(1,round((fb-fa)/dx+1)); for i=1:Nm a=1; for j=1:length(x) if x(j)>=fa && x(j)<=fb+eps Bf(i,a)=sqrt(2/L)*sin(i*pi*x(j)/L); if i==1 xFcord(1,a)=x(j); % a=a+1; end a=a+1; end end end
%% Input Force Parameters for Neutral-Extend-Neutral (Traveling Logistic) disp('Building Traveling Logistic Curve') Fp=0.0000125; % [N] Scaling input s_x=-.015; % shift in x (positive shifts right, negative shifts left) a1=[1000 -100 100 -1000]; a0=[-(0.03+s_x)*a1(1) -(0.04+s_x)*a1(2) -(0.05+s_x)*a1(3) -(0.06+s_x)*a1(4)]; b0=-15;
174
b1=[5, .5, 1, 5]; u_in=zeros(size(a1,2),size(xFcord,2)); y_in=zeros(size(a1,2),size(xFcord,2)); for i=1:length(a1) % rows for j=1:length(xFcord) % columns u_in(i,j)=a0(i)+a1(i)*xFcord(j); y_in(i,j)=1/(1+exp(-u_in(i,j))); end end by=zeros(1,size(xFcord,2)); for k=1:length(a1) by(1,:)=by+y_in(k,:).*b1(k); end v_in=b0+by; z_in=zeros(1,size(xFcord,2)); for m=1:length(xFcord) z_in(m)=1/(1+exp(-v_in(m))); end z_in=Fp.*z_in; % Shifting Wave in Time %Note that by changing dt and corespondinly dx, the magnitude of voltage %output changes (more force being "added", more x nodes included). The %shape of the voltage profile DOES NOT change dt=0.001; % time discretization tau=0.08; % time delay Tfinal=1; % seconds t_hold=0.3; t_run=0:dt:Tfinal; delta_x=0.005; % [m] how much muscle moves laterally delta_t=0.2; % [s] how long it takes for muscle to move laterally dxdt=delta_x/delta_t; t_active=tau+2*delta_t+t_hold;
%specifying inputs numF=size(Bf,2); % total # of "x nodes" located in active force region u=zeros(numF,length(t_run)); % input matrix t1=[300, -50]; t0=[-tau*t1(1), -(tau+delta_t+t_hold+0.05)*t1(2)]; u_t=zeros(2,size(t_run,2)); y_t=zeros(2,size(t_run,2)); for i=1:length(t1) for j=1:length(t_run) % columns u_t(i,j)=t0(i)+t1(i)*t_run(j); y_t(i,j)=1/(1+exp(-u_t(i,j))); end end
%input shape u_max=0; for i=1:length(t_run); %time period simulated if t_run(i)>=tau && t_run(i)<=t_active if t_run(i)>tau && t_run(i)<=tau+delta_t
175
% shifting right x_n=xFcord(1)+dxdt*t_run(i); for j=1:length(xFcord) if abs(xFcord(j)-x_n)
% figure() % set(gcf,'position',[pw,ph,fL,fW]) % for i=1:size(u,2) % if mod(i,5)==0 % plot(xFcord,u(:,i)) % hold all % end % end % xlabel('x position [m]') % ylabel('Force [N]') % % xlim([min(xFcord),max(xFcord)]) % xlim([0,L]) % title('Traveling Compound Logistics Curve') % grid on
%% System State Matrix Formulation disp('Building State Matrices') % Building system matrix Az in 1st order system Az=zeros(2*Nm, 2*Nm); A1=-M\Ds; A2=-M\K; spcr=0; %spacer variable for Az matrix for i=1:2*Nm %row if mod(i,2)~=0 %odd row spcr=spcr+2; for j=1:2*Nm %column if j~=spcr Az(i,j)=0; else Az(i,j)=1; end
176
end else %even row for j=1:2*Nm %column if mod(j,2)~=0 %odd column -> associated with stiffness Az(i,j)=A2(i/2,(j+1)/2); else %even column -> associated with damping Az(i,j)=A1(i/2,j/2); end end end end
%Building input matrix Bz Bz=zeros(2*Nm,numF); B=M\Bf; for i=1:2*Nm if mod(i,2)~=0 %odd row for j=1:numF %number of inputs Bz(i,j)=0; end else %even row for j=1:numF %number of inputs Bz(i,j)=B(i/2,j); end end end
%Building output matrix Cz Cz=zeros(2*Nm, 2*Nm); for i=1:2*Nm for j=1:2*Nm if mod(j,2)~=0 %odd column if i==j Cz(i,j)=1; %diagonal element else Cz(i,j)=0; end else %even column Cz(i,j)=0; end end end
%Building matrix Dz Dz=zeros(2*Nm,numF); %columns match inputs sys=ss(Az,Bz,Cz,Dz); [yz,t_run]=lsim(sys,u,t_run); rbar_t=zeros(size(t_run,1),Nm); for j=1:2*Nm %column if mod(j,2)~=0 %odd column rbar_t(:,(j+1)/2)=yz(:,j); % assigning output of lsim back to origional rbar matrix (Nm instead of 2*Nm) end end
177
rbar=rbar_t';
%% Charge q_t=Cp_S*Theta'*rbar; figure () set(gcf,'position',[pw,ph,fL,fW]) plot(t_run,q_t,'k') xlim([0,Tfinal]) xlabel('Time (s)') ylabel('Charge Output [C]') grid on title('PVDF Charge Output due to Transverse Loading')
%% Getting current % mid point differentiation dq_tdt=zeros(1,size(t_run,2)-1); dt1=zeros(1,size(t_run,2)-1); for i=1:length(t_run)-1 dq_tdt(i)=(q_t(i+1)-q_t(i))/dt; dt1(i)=(t_run(i+1)+t_run(i))/2; % if mod(i,1000)==0 % i % end end % figure () % set(gcf,'position',[pw,ph,fL,fW]) % plot(dt1,dq_tdt,'k') % xlim([0,max(dt1)]) % xlabel('Time (s)') % ylabel('Current [Amps]') % grid on % title('PVDF1 Current Output due to Bending')
%% Voltage (through Oscilloscope of dSPACE -> 1 M-ohm impedance) R=1e6; % [ohms] voltage_t=R.*dq_tdt;
% figure () % set(gcf,'position',[pw,ph,fL,fW]) % plot(dt1,voltage_t,'k') % xlim([0,max(dt1)]) % xlabel('Time (s)') % ylabel('Voltage [volts]') % grid on % title('Voltage Output due to Bending')
178
Appendix 6: PVDF Voltage Output from Axial Beam Deflection Matlab Code
% Considering output due to axial deflections (Lateral) clc; clear all; %close all; format compact; format shortg;
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% Dealing with Symbols % digits() and vpa() deal format output that is affected by using symbols digits(12) %set number of digits in mixed number/variable format
%% Geometry % ~ PVDF L=0.08; % [m] w=0.03; % [m] width (along y) tp=52e-6; % [m] using 52 micrometer, ignoring thickness of electrodes dx=0.0001; x=0:dx:L; % ~Structure (Sleeve) t=0.0004; %[m] thickness (along z) I=(1/12)*w*t^3;
%% Material Properties % ~Structure (Sleeve) Ys_0=0.535e6; %[Pa] from MTS data (data 1/28/14) eps0=(27.5-20)/20; % Initial strain in sleeve from putting on deps=(28-27.5)/27.5; % Change in strain, treating eps0 as base Ys=(deps*Ys_0*eps0+0.5*deps^2*Ys_0)/(0.5*deps^2); % Adjusted modulus v12=0.2; %Poisson's ratio from MTS experiment v13=0.375; %Poisson's ratio from MTS experiment rho_s=521.6; %[kg/m^3] Calculated density of sleeve from measured mass and volume syms s44 s55 s66 real %Leave as symbolic, b/c of form only multiplied by zeros and drop out s_sleeve=1/Ys*[1 -v12 -v13 0 0 0 -v12 1 -v13 0 0 0 -v13 -v13 1 0 0 0 0 0 0 s44 0 0 0 0 0 0 s55 0 0 0 0 0 0 s66]; c_sleeve=eye(6)/s_sleeve; %compliance matrix
% ~PVDF: Combination of sources rho_p=1.78e3; %[kg/m^3] (all sources agree) Cap=380e-6; %[F/m^2] ***Converted to /m^2 from (MS) %Matrices of PVDF -> Orthorhombic mm2 symmetry (Fdamtls of Piezo Senorics)
179
%s^E coeffs [1/Pa]x10^(-12) syms s44 s55 s66 real s11=365; Yp=1/(s11*1e-12); %[Pa] s12=-110; s13=-209; s22=424; s23=-192; s33=472; v12_E=-s12/s11; v13_E=-s13/s11; s_E=(1e-12)*[s11 s12 s13 0 0 0; s12 s22 s23 0 0 0; s13 s23 s33 0 0 0; 0 0 0 s44 0 0; 0 0 0 0 s55 0; 0 0 0 0 0 s66]; %d[3x6] Piezo strain constant [C/N]x10^-12 (Es) d31=21; d32=2.3; d33=-26; d24=-27; d15=-23; d=(-1e-12)*[0 0 0 0 d15 0; 0 0 0 d24 0 0; d31 d32 d33 0 0 0]; % added negative % Eps^T - dielectric permittivity [F/m] Eps_T=110e-12*eye(3); % Beta^T - impermitivity [m/F] Beta_T=eye(3)/Eps_T; %g[3x6] Piezo stress constnat [m^2/C] g=Beta_T*d; % s^D s_D=s_E-d'*Beta_T*d; % c^D c_D=vpa(eye(6)/s_D); % h[3x6] h=g*c_D; %Beta_S Beta_S=vpa(Beta_T+g*h');
%% Mode Shapes Nm=10; %number of mode shapes
%% Ritz Formulation Axial Deflections % Mode shape normalized st integral 0 to L p(i)*p(j)=1
% Building Top PVDF Mass Matrix % Building PSI(x) syms xx yy zz real for i=1:Nm %i corresponds to rows p_i=sqrt(2/L)*cos(i*pi*xx/L); %mode shape for j=1:Nm %j corresponds to columns p_j=sqrt(2/L)*cos(j*pi*xx/L); %mode shape PSI_x(i,j)=p_i*p_j; end
180
end % Ms (mass of sleeve) = rho_s*w*t Ms=int(int(int(rho_s*PSI_x,zz,-t/2,t/2),xx,0,L),yy,-w/2,w/2); % Mp (mass of piezo 1) Mp=vpa(int(int(int(rho_p*PSI_x,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building Stiffness Matrix % Building Br_s(x) for j=1:Nm %columns p=sqrt(2/L)*cos(j*pi*xx/L); % axial mode shape -->Normalized for i=1:6 %rows if i==1 Br_s(i,j)=diff(p,xx,1); elseif i==2 Br_s(i,j)=-v12_E*diff(p,xx,1); elseif i==3 Br_s(i,j)=-v13_E*diff(p,xx,1); else Br_s(i,j)=0; end end end % Building Br_p(x) for j=1:Nm %columns p=sqrt(2/L)*cos(j*pi*xx/L); %mode shape -->Normalized for i=1:6 %rows if i==1 Br_p(i,j)=diff(p,xx,1); elseif i==2 Br_p(i,j)=-v12_E*diff(p,xx,1); elseif i==3 Br_p(i,j)=-v13_E*diff(p,xx,1); else Br_p(i,j)=0; end end end % Ks (stiffness of sleeve) Ks=vpa(int(int(int(Br_s'*c_sleeve*Br_s,zz,-t/2,t/2),xx,0,L),yy,-w/2,w/2)); % Kp^D (stiffness of PVDF) Kp_D=vpa(int(int(int(Br_p'*c_D*Br_p,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building Coupling Matrix % Charge is ALWAYS generated in the 3 direction (across electrodes) % Bq=[0; 0; (4/L^2)*xx*(L-xx)/(w*L)]; % Shaped electrode Bq=[0; 0; 1/(w*L)]; % Uniform electrode %Theta Theta=vpa(int(int(int(Br_p'*h'*Bq,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2));
%% Building "Inverse" Capacitance Matrix %Cp^s^-1 Cp_S_inv=vpa(int(int(int(Bq'*Beta_S*Bq,zz,t/2,t/2+tp),xx,0,L),yy,-w/2,w/2)); %Cp^s Cp_S=eye(length(Cp_S_inv))/Cp_S_inv;
%% Mass and Stiffness Matricies
181
M=Ms+Mp; K=Ks+Kp_D-Theta*Cp_S*Theta';
%% Building D (damping matrix) D=zeros(Nm,Nm); zeta=0.2; for i=1:Nm for j=1:Nm if i==j %using a constant damping ratio for all modes D(i,j)=2*zeta*sqrt(M(i,j)*K(i,j)); end end end
%% Building Bf(x) % One force at x=0 % One force at x=L (cancels with L in den. of mode shape) Bf=zeros(Nm,2); for i=1:Nm Bf(i,1)=sqrt(2/L)*cos(i*pi*0/L); Bf(i,2)=sqrt(2/L)*cos(i*pi); end
%% Building Input Force (parameters for simulation) dt=0.001; Tfinal=1; %seconds Neutral-Flex-Neutral t_run=0:dt:Tfinal;
%% Parameters for GC Neutral-Flex-Neutral (Gaussian) % input shape tau=.08; %time delay % T1=.44; %play for Gaussian (controls width) T1=.35; %play for Gaussian (controls width) cent=T1/2+tau; f_axial=0.65; % [N} fp=[-f_axial, f_axial]; t_hold=0.215; numF=2; F=zeros(numF,length(t_run)); for i=1:numF %One input row per force a=0; for j=1:length(t_run); %time period simulated if t_run(j)<=cent a=a+1; sig_u=T1/6; % Gaussian fcn ~ 6 standard deviations wide F(i,j)=-fp(i)*exp(-(t_run(j)-cent)^2/(2*sig_u^2)); % Gaussian fcn elseif t_run(j)>cent && t_run(j)<=cent+t_hold F(i,j)=F(i,a); else sig_d=(T1+0.2)/6; F(i,j)=-fp(i)*exp(-(t_run(j)-t_hold-cent)^2/(2*sig_d^2)); end end end
182
%% System State Matrix Formulation % Building system matrix Az in 1st order system Az=zeros(2*Nm, 2*Nm); A1=-M\D; A2=-M\K; spcr=0; %spacer variable for Az matrix for i=1:2*Nm %row if mod(i,2)~=0 %odd row spcr=spcr+2; for j=1:2*Nm %column if j~=spcr Az(i,j)=0; else Az(i,j)=1; end end else %even row for j=1:2*Nm %column if mod(j,2)~=0 %odd column -> associated with stiffness Az(i,j)=A2(i/2,(j+1)/2); else %even column -> associated with damping Az(i,j)=A1(i/2,j/2); end end end end
%Building input matrix Bz Bz=zeros(2*Nm,numF); B=M\Bf; for i=1:2*Nm if mod(i,2)~=0 %odd row Bz(i,1)=0; else %even row for j=1:numF Bz(i,j)=B(i/2,j); end end end
%Building output matrix Cz Cz=zeros(2*Nm, 2*Nm); for i=1:2*Nm for j=1:2*Nm if mod(j,2)~=0 %odd column if i==j Cz(i,j)=1; %diagonal element else Cz(i,j)=0; end else %even column Cz(i,j)=0; end end end
183
%Building matrix Dz Dz=zeros(2*Nm,numF); %columns match inputs sys=ss(Az,Bz,Cz,Dz); [yz,t_run]=lsim(sys,F,t_run); rbar_t=zeros(size(t_run,1),Nm); for j=1:2*Nm %column if mod(j,2)~=0 %odd column rbar_t(:,(j+1)/2)=yz(:,j); % assigning output of lsim back to origional rbar matrix (Nm instead of 2*Nm) end end rbar=rbar_t';
%% Charge q=Cp_S*Theta'*rbar;
% figure () % set(gcf,'position',[pw,ph,fL,fW]) % plot(t_run,q,'k')%,t_run,q2,'r') % xlim([0,Tfinal]) % xlabel('Time (s)') % ylabel('Charge Output [C]') % grid on % title('PVDF Charge Output due to Axial Force')
%% Getting current dqdt=zeros(1,size(t_run,2)-1); dt1=zeros(1,size(t_run,2)-1); for i=1:length(t_run)-1 dqdt(i)=(q(i+1)-q(i))/dt; dt1(i)=(t_run(i+1)+t_run(i))/2; if mod(i,1000)==0 i; end end % figure () % set(gcf,'position',[pw,ph,fL,fW]) % plot(dt1,dqdt,'k') % xlim([0,max(dt1)]) % xlabel('Time (s)') % ylabel('Current [Amps]') % grid on % title('PVDF1 Current Output due to Axial Force')
%% Voltage (through Oscilloscope of dSPACE -> 1 Mohm impedance) R=1e6; % [ohms] voltage=R.*dqdt;
184
Appendix 7: Neural Network Matlab Code
The neural network related code consists of: signal extracting scripts, input/output compilation file, a main neural network script, a training function, weight matrix initialization code, and a forward pass script written in Matlab.
Signal Extracting Script: Gesture data is written to text file. This code allows user to break up continuous file into single gesture files by specifying start time. This script uses user function
SigSeg.m
% 2014-11-3 % File to extract single gesture signals from gesture data collection file % Uses "SigSeg" function (C:\Program Files\MATLAB\R2010b\bin\MyFuncs\) % clc; %clear all; close all
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% File import p=’file path as string'; f=’file name.txt'; pf=strcat(p,f); impdat=importdata(pf); v_analog=impdat(:,1); %% Convert analog values to voltage values a=-1.65; b=1.65; mn=0; mx=1023; v=((b-a).*(v_analog-mn)./(mx-mn))+a;
% Time increment used for arduino dt=0.005; % 5 ms t=0:dt:dt*(length(v)-1);
%% Plot data % Comment out after first time plotting so doesn't keep popping up figure () set(gcf,'position',[pw,ph,fL,fW])
185
plot(t,v,'k') xlabel('Time [s]') ylabel('Voltage [V]') title(f)
%% Extract % Comment out when generating whole signal plot the first time to avoid % errors or overwriting past files
% name=''; % part of fname, used for output file naming % num=''; % part of fname, used for output file naming. *MUST be a STRING* % ts=0; % start time for signal extraction % tf=0; % end time for signal extraction % % t_in=t; % time vector % y_in=v; % voltage vector % % path='file save path as string'; % ext='.txt'; % fname=strcat(path, name, num, ext); % % [a,b]=SigSeg(t_in, y_in, ts, tf, fname);
SigSeg.m Function: function [a,b] = SigSeg(t_in,y_in,ts,tf,fname) %SigSeg extracts vectors t_out, y_out from input vectors t_in, y_in % starting at initial ts and ending at final time tf. New vectors are % saved as text tile fname. % % Used for extracting training vectors for NN. Signals are set % to start at t=0.
% Code for script: % num='0'; % Must be a string % ts=0; % tf=0; % % t_in=t; % y_in=y_E; % path='C:\Users\Kyle\Documents\School\ISSL\Gesture Controller\Matlab\NN\TS_2ms\'; % name='Ext'; % ext='.txt'; % fname=strcat(path, name, num, ext); % [a,b]=SigSeg(t_in, y_in, ts, tf, fname)
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% Signal Extraction
186
[C,a]=min(abs(t_in-ts)); % C required 'dummy' variable [C,b]=min(abs(t_in-tf)); % C required 'dummy' variable
t_out=t_in(a:b)-t_in(a); y_out=y_in(a:b);
figure() set(gcf,'position',[pw,ph,fL,fW]) plot(t_out,y_out) axis tight
fid=fopen(fname,'wt'); for i=1:length(t_out) fprintf(fid,'%f %f\n',t_out(i),y_out(i)); end fclose(fid); end
Network Training Input/Output Vector Script: This script takes all segmented files and compiles them into a matrix (gives all columns equal length). It also creates output vector file based on user specified target output values.
% This file creates two text files for training a NN. % The first file is a set of voltages from specific gestures (Inputs for training). % The second file is the output the NN should compute for the corresponding input gesture (Output for training). % In its current setup, it is for two gestures clc; %clear all; %close all;
%% Length of Input Vectors % Need all input vectors to be of same length for NN calcs. % All vectors other than the longest will have zeros at the end.
% 5ms Sampling tgtFile='folder where segmented files are stored as string; files=dir(tgtFile);
% The first n columns will be "Right" j=0; for file=files' n=file.name; for tgtNames={'Right'} if strfind(n, tgtNames{1}) >=1 j=j+1; impdat=importdata(strcat(tgtFile,'\',n)); v=impdat(:,2); Sz(j)=length(v); end end end % The next m columns will be "Left"
187
k=j; for file=files' n=file.name; for tgtNames={'Left'} if strfind(n, tgtNames{1}) >=1 k=k+1; impdat=importdata(strcat(tgtFile,'\',n)); v=impdat(:,2); Sz(j)=length(v); end end end
M=max(Sz) L=length(Sz) t=0:0.005:0.005*(M-1);
% Create an array of zeros to populate with extracted signals z=zeros(M,L);
%% Populate z Array % The first n columns will be "Right" j=0; for file=files' n=file.name; for tgtNames={'Right'} if strfind(n, tgtNames{1}) >=1 n; j=j+1; impdat=importdata(strcat(tgtFile,'\',n)); v=impdat(:,2); z(1:length(v),j)=v; end end end % The next m columns will be "Left" k=j; for file=files' n=file.name; for tgtNames={'Left'} if strfind(n, tgtNames{1}) >=1 n; k=k+1; impdat=importdata(strcat(tgtFile,'\',n)); v=impdat(:,2); z(1:length(v),k)=v; end end end
%% Creating Input Vector File fname = 'Path and name of input vector file as string’; dlmwrite(fname, z,'delimiter',' ','newline','pc')
%% Output Vectors % For right motions, NN output 1 to 0.9 and output 2 goes to 0.1
188
% For left motions, NN output 1 to 0.1 and output 2 goes to 0.9 o1_1=0.9.*ones(1,j); o1_2=0.1.*ones(1,k-j); o2_1=0.1.*ones(1,j); o2_2=0.9.*ones(1,k-j); o=[o1_1 o1_2; o2_1 o2_2];
%% Creating Output Vector File fname = 'Path and name of output vector file as string’; dlmwrite(fname, o,'delimiter',' ','newline','pc')
Neural Network Main Body Script: This script allows for the specification of forcing parameters during teaching, setting of training runs (number of epochs), and gives the option to generate new random weight matrices. References NN_Train.m and ForwardNN_w_T.m which runs the newly trained network on one of the input vectors from the input matrix file.
% Neural Network Body % This code houses references to functions that % train and run a NN. Require set of vector input/outputs % to train, and a vector to run along w/ proper time vector. % clc; clear all; %close all; format compact; format shortg;
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% Import Training Arrays % Input pattern matrix from tests (each column is one input vector) Psamp=dlmread('path to created input matrix as string');
% Desired output classification matrix (each column is one output corresponding to % input pattern of same column) Osamp=dlmread('path to created output matrix as string');
%% Network Setup I=size(Psamp,1); % number of input nodes (5ms 133) J=3; % number of hidden nodes K=size(Osamp,1); % number of output nodes (2)
%% IO Pairings
%~~~~Forcing Params % Static Forcing
189
P_0=zeros(I,1); O_0=0.5.*ones(K,1); % Nonstatic Forcing as needed %~~~Full IO Vectors P=[Psamp, P_0]; O=[Osamp, O_0;
%% Training % Train Network. Genearte new weight matricies? 1=yes, 0=no (0 implies % weight matricies are already constructed, no training will occur) trainNN=1; Cmax=25000; % Max epoch iterations
% Generate new random A matrix? 1=yes, 0=no genAnew=1; if trainNN==1 train_start = datestr(now,'HH:MM:SS') NN_Train(I, J, K, genAnew, P, O, Cmax); train_end = datestr(now,'HH:MM:SS') end
%% Run Time Series Forward Pass Function (m-Code) % dt = 0.002; dt = 0.005; t_samp=(0:dt:2+I*dt)'; sig_buf=zeros(1/dt,1); mod_in=[0;sig_buf; P(:,1); sig_buf];
% P_test=zeros(I,1); % P_test(I,1)=-0.0016129; % mod_in=[0;sig_buf; P_test; sig_buf];
In_vect=mod_in; Z_out_test=ForwardNN_w_t(t_samp',In_vect, I, J, K, dt); title('Test') xlablh=get(gca,'XLabel'); ylablh=get(gca,'Ylabel'); set(xlablh,'Position',get(xlablh,'Position')-[0 0 0],'FontSize',16,'FontWeight','bold'); set(ylablh,'Position',get(ylablh,'Position')+[0.0 0 0],'FontSize',16,'FontWeight','bold'); set(gca,'FontSize',14)
NN_Train.m Function: Script that trains the network using momentum and adaptive learning rates. References genArand.m which generates new random weight matrix. function [] = NN_Train(I, J, K, genAnew, P, O, Cmax) %NN_Train: % Trains Neural Network using momentum and adaptive learning rates. % Writes appropriate weight matricies to text files. % Logistic Function: g(u) = 1/(1+exp(-u)) % Deriv LF: g'(u) = exp(-u)/(1+exp(-u))^2 % Error Definition: E=(1/2)[sum(P)[sum(K)[(z_pk-o_pk)^2]]]/(P*K) % 190
% Inputs: % Passed Outputs: Epoch and Error vectors % Nonpassed Outputs: Weight Matrix text files
%% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80;
%% Array Size Preallocation: X=zeros(I+1,1); % Network input vector X(I+1,1) = -1; % Dictating 'I+1'th input to be -1 u_j=zeros(J,1); % hidden node summation to be evaluated Y=zeros(J+1,1); % Hidden node output Y(J+1,1) = -1; % Dictating 'J+1'th value to be -1 v_k=zeros(K,1); % Output node summation to be evaluated Z=zeros(K,1); % Calculated output vector
%% Network Learning Params: mu=0.2; % momentum kappa=0.1; % adaptive learning rate varaiable phi=0.5; % adaptive learning rate varaiable theta=0.7; % adaptive learning rate varaiable
%% Initial Weight Matrix A if genAnew==1 genArand(I,J) end a_i=dlmread('path of saved random a weight matrix as string'); a=a_i;
%% Initial Weight Matrix B b_i = zeros(J+1,K); % 'b' weight vector, leaving J+1 weight at zero for j=1:J % assign half of 'b' weights value of -1, half of value 1 if mod(j,2)==0 % even number b_i(j,1)=-1; else % odd number b_i(j,1)=1; end end for k=2:K b_i(:,k)=b_i(:,1); end b=b_i;
% Cmax = Max epoch itterations E_track=zeros(Cmax,1); for c=1:Cmax % Max number of epochs (safer than do while loop) % Error Derivatives (reset each epoch) del_a=zeros(I+1,J); del_b=zeros(J+1,K); fm_a=zeros(I+1,J); % initial fm value for adaptive learning
191
fm_b=zeros(J+1,K); em_a=zeros(I+1,J); % initial em value for adaptive learning em_b=zeros(J+1,K); cm_a=zeros(I+1,J); % initial momentum values cm_b=zeros(J+1,K); cm_a_1=zeros(I+1,J); % c_(m-1) cm_b_1=zeros(J+1,K);
Er_c=0; % Cumulative error
for p=1:size(P,2) % for each pattern in set P % Forward Pass X(1:I)=P(:,p); for j=1:J u_j(j)=X'*a(:,j); Y(j)=1/(1+exp(-u_j(j))); end for k=1:K v_k(k)=Y'*b(:,k); Z(k)=1/(1+exp(-v_k(k))); end
% Forward Error for pattern P Er=Z-O(:,p); Er_c=Er_c+Er'*Er;
% Calculating Cumulative Error Derivatives for k=1:K dEdZ=Er(k); dZdV=Z(k)*(1-Z(k)); for j=1:J+1 dVdB=Y(j); del_b(j,k)=del_b(j,k)+dEdZ*dZdV*dVdB; % Sum of error over all patterns end end
for i=1:I+1 for j=1:J % J+1 is fixed at -1, no weight contribution dEdY=0; for k=1:K dEdZ=Er(k); dZdV=Z(k)*(1-Z(k)); dVdY=b(j,k); dEdY=dEdY+dEdZ*dZdV*dVdY; end dYdU=Y(j)*(1-Y(j)); dUdA=X(i); del_a(i,j)=del_a(i,j)+dEdY*dYdU*dUdA; % Sum of error over all patterns end end end
% ~~~Adjust weights (Adaptive Learning) for i=1:I+1 for j=1:J fm_a(i,j)=theta*fm_a(i,j)+(1-theta)*del_a(i,j);
192
if fm_a(i,j)*del_a(i,j)>0 em_a(i,j)=em_a(i,j)+kappa; else em_a(i,j)=em_a(i,j)*phi; end cm_a(i,j)=mu*cm_a_1(i,j)-(1-mu)*em_a(i,j)*del_a(i,j); a(i,j)=a(i,j)+cm_a(i,j); cm_a_1(i,j)=cm_a(i,j); end end
for j=1:J+1 for k=1:K fm_b(j,k)=theta*fm_b(j,k)+(1-theta)*del_b(j,k); if fm_b(j,k)*del_b(j,k)>0 em_b(j,k)=em_b(j,k)+kappa; else em_b(j,k)=em_b(j,k)*phi; end cm_b(j,k)=mu*cm_b_1(j,k)-(1-mu)*em_b(j,k)*del_b(j,k); b(j,k)=b(j,k)+cm_b(j,k); cm_b_1(j,k)=cm_b(j,k); end end
E_track(c,1)=1/(2*size(P,2)*K)*Er_c; % mean square average of error end epoch = 1:Cmax;
%% Save Final Weight Matrices Matlab Friendly Format file_name_A = 'path to store file as.txt'; dlmwrite(file_name_A, a,'delimiter',',','newline','pc') file_name_B = 'path to store file as.txt'; dlmwrite(file_name_B, b,'delimiter',',','newline','pc')
%% Print to text file formated for CMEX copy/paste fid=fopen('path to store file as.txt','wt'); for i=1:size(a,1) L=sprintf('{%f',a(i,1)); for j=2:size(a,2) Lj=sprintf(',%f',a(i,j)); L=strcat(L,Lj); end Li=strcat(L,'}'); fprintf(fid,'%s\n',Li); end fclose(fid); fid=fopen('path to store file as.txt','wt'); for i=1:size(b,1) L=sprintf('{%f',b(i,1)); for j=2:size(b,2) Lj=sprintf(',%f',b(i,j)); L=strcat(L,Lj);
193
end Li=strcat(L,'}'); fprintf(fid,'%s\n',Li); end fclose(fid);
%% Print to text file formatted for Arduino copy/paste fid=fopen('path to store file as.txt','wt'); for i=1:size(a,1) L=sprintf('{%f',a(i,1)); for j=2:size(a,2) Lj=sprintf(',%f',a(i,j)); L=strcat(L,Lj); end if i %% Error vs Epoch Plot figure () set(gcf,'position',[pw,ph,fL,fW]) plot(epoch', E_track) xlabel('Epoch Number') ylabel('Mean Square Error') str=sprintf('Mean Square Error per Epoch (Max %d itterations)',Cmax); title(str) grid on end genArand.m Function: function [ ] = genArand( I,J ) %genArand: 194 % Generates random A matrix whose weight will be adjusted in NN_Train. % Writes random A matrix, which is centered around 0, to a text file. % This function should only be called if new random matrix is necessary % as these values are only starting points. % Reason for new A matrix: new weight matrix size a_lb = -0.1; % lower bound on randomly generated 'a' weights a_ub = 0.1; % upper bound on randomly generated 'a' weights a_i=zeros(I+1,J); for j=1:J % randomly generated 'a' weight vector centered around 0 a_i_gen = a_lb+(a_ub-a_lb).*rand(I+1, 1); a_i(:,j)=a_i_gen; end % ~~~Print random weights to text file a_out=a_i; file_name = 'file path as string.txt'; dlmwrite(file_name, a_out) end ForwardNN_w_t.m Function: function [Z_out] = ForwardNN_w_t(t,In_vect, I, J, K, dt) %ForwardNN % Calculates K outputs of the Neural Network for input vector In_vect % in FeedForward mode. %% Figure Placement and Size scnsize=get(0,'ScreenSize'); h=scnsize(4); fL=800; fW=600; pw=5; ph=h-fW-80; %% Forward Network a=dlmread('file path of A_Weights.txt as string'); b=dlmread('file path of B_Weights.txt as string'); Z_out=zeros(size(In_vect,1),K); %length of t vector by # output nodes X=zeros(I+1,1); % Network input vector X(I+1,1) = -1; % Dictating 'I+1'th input to be -1 u_j=zeros(J,1); % hidden node summation to be evaluated Y=zeros(J+1,1); % Hidden node output Y(J+1,1) = -1; % Dictating 'J+1'th value to be -1 v_k=zeros(K,1); % Output node summation to be evaluated Z_t=zeros(1,K); % Calculated output vector at time increment t for i=1:size(In_vect,1) X(1:I-1)=X(2:I); X(I)=In_vect(i); for j=1:J 195 u_j(j)=X'*a(:,j); Y(j)=1/(1+exp(-u_j(j))); end Y(J+1)=-1; for k=1:K v_k(k)=Y'*b(:,k); Z_t(k)=1/(1+exp(-v_k(k))); end Z_out(i,:)=Z_t; end figure () set(gcf,'position',[pw,ph,fL,fW]) plot(t, Z_out(:,1), 'r', t, Z_out(:,2), 'k') hold all x1=[0,3]; x2=[1+dt*I, 1+dt*I]; y1=[0.05, 0.05]; y2=[0.15, 0.15]; y3=[0.85, 0.85]; y4=[0.95, 0.95]; y5=[0, 1]; plot(x1, y1, 'k','LineWidth', 2) plot(x1, y2, 'k','LineWidth', 2) plot(x1, y3, 'k','LineWidth', 2) plot(x1, y4, 'k','LineWidth', 2) plot(x2, y5, 'k','LineWidth', 2) hold off legend('Z1', 'Z2',2) xlabel('time (s)') ylabel('Output Node Z Value') end %End Function 196 Appendix 8: dSPACE/Simulink Configuration with CMEX Code The three gesture (fist, flex, extend) neural network was originally tested using a dSPACE ds1103 system. The Simulink model used is shown in Figure A8.1 where the S‐Function block points to a C MEX S‐Function file (code follows) and the sampling rate is 2ms. The leads from the PVDF were connected to AD channel 19 on the dSPACE board using a BNC to alligator clip cable. outputs 2‐4 correspond to the three output vector values while output 5 is the network classification. Note that in the S‐function global variables Wa and Wb, the two calculated weight matrices, are empty. These must be defined in a real code but are omitted here to save space (369x11 and 11x3 arrays respectively). Figure A8.1 Simulink block diagram for dSPACE build /* Level 2 C MEX S-Function for use in Neural Network dSPACE model */ #define S_FUNCTION_NAME sNN_10 #define S_FUNCTION_LEVEL 2 #include "simstruc.h" #include "stdio.h" #include "math.h" #define U(element) (*uPtrs[element]) /* Pointer to Input Port0 */ #define I 368 197 #define J 10 #define K 3 /* Glogal Variables */ real_T Wa[I+1][J]={{...,...}, {...,...}}; real_T Wb[J+1][K]={{...,...}, {...,...}}; /*======* * S-function methods * *======*/ /* Function: mdlInitializeSizes ======* Abstract: * The sizes information is used by Simulink to determine the S-function * block's characteristics (number of inputs, outputs, states, etc.). */ static void mdlInitializeSizes(SimStruct *S) { ssSetNumSFcnParams(S, 0); /* Number of expected parameters */ if (ssGetNumSFcnParams(S) != ssGetSFcnParamsCount(S)) { /* Return if number of expected != number of actual parameters */ return; } ssSetNumContStates(S, 0); ssSetNumDiscStates(S, 0); if (!ssSetNumInputPorts(S, 1)) return; ssSetInputPortWidth(S, 0, 1); ssSetInputPortDirectFeedThrough(S, 0, 0); // NO direct feedthrough (1=yes, 0=no) if (!ssSetNumOutputPorts(S, 1)) return; ssSetOutputPortWidth(S, 0, 4); //changed from 1 ssSetNumSampleTimes(S, 1); // DWork vector params ssSetNumDWork(S, 1); // 1 DWork vector ssSetDWorkWidth(S,0,I+1); // Length of DWork vector is I+1 ssSetDWorkDataType(S,0,SS_DOUBLE); // Type double // /* Specify the sim state compliance to be same as a built-in block */ ssSetSimStateCompliance(S, USE_DEFAULT_SIM_STATE); ssSetOptions(S, 0); /* * ssSetOptions(S, SS_OPTION_EXCEPTION_FREE_CODE); */ } /* Function: mdlInitializeSampleTimes ====== 198 * Abstract: * This function is used to specify the sample time(s) for your * S-function. * Using Inherited sample time */ static void mdlInitializeSampleTimes(SimStruct *S) { ssSetSampleTime(S, 0, INHERITED_SAMPLE_TIME); ssSetOffsetTime(S, 0, 0.0); } #define MDL_START static void mdlStart(SimStruct *S) { /* Define pointer *x as type real_T (=double) */ real_T *x = (real_T*) ssGetDWork(S,0); int_T i; int_T j; /* Initialize DWork vector */ for(i=0; i<=I-1; i++){ x[i]=0; } x[I]=-1; } /* Function: mdlOutputs ======* Abstract: * In this function, compute the outputs of S-function block. */ static void mdlOutputs(SimStruct *S, int_T tid) { real_T *y = ssGetOutputPortSignal(S,0); /* Output Port */ real_T *x = (real_T*) ssGetDWork(S,0); /* DWorks vector */ int_T i=0; // input counter int_T j=0; // hidden node counter int_T k=0; // output node counter real_T u_j[J]; // hidden node summation to be evaluated real_T Y[J+1]; // hidden node output (J+1 element = -1 real_T v_k[K]; // output node summation to be evaluated real_T Z[K]; // calculated output vector int_T NN_Out=0; // Initialized NN output to be passed for(j=0; j<=J-1; j++){ u_j[j]=0; for(i=0; i<=I; i++){ u_j[j]=u_j[j]+x[i]*Wa[i][j]; } /* exit i loop */ Y[j]=1/(1+exp(-u_j[j])); } /* exit j loop */ Y[J]=-1; /* Actually J+1'th element. Set to -1 as bias */ for(k=0; k<=K-1; k++){ v_k[k]=0; 199 for(j=0; j<=J;j++){ v_k[k]=v_k[k]+Y[j]*Wb[j][k]; //ssPrintf("%f\n", Wb[j][k]); } /* exit j loop */ Z[k]=1/(1+exp(-v_k[k])); } /* exit k loop */ /*Nothing (default)*/ NN_Out=0; /*Fist*/ if(Z[0]<=0.95 && Z[0]>=0.85 && Z[1]<=0.15 && Z[1]>=0.05 && Z[2]<=0.15 && Z[2]>=0.05){ NN_Out=1; for(i=0; i<=I-1; i++){ x[i]=0; } x[I]=-1; } /*Extend*/ if(Z[1]<=0.95 && Z[1]>=0.85 && Z[0]<=0.15 && Z[0]>=0.05 && Z[2]<=0.15 && Z[2]>=0.05){ NN_Out=2; for(i=0; i<=I-1; i++){ x[i]=0; } x[I]=-1; } /*Flex*/ if(Z[2]<=0.95 && Z[2]>=0.85 && Z[0]<=0.15 && Z[0]>=0.05 && Z[1]<=0.15 && Z[1]>=0.05){ NN_Out=3; for(i=0; i<=I-1; i++){ x[i]=0; } x[I]=-1; } y[3] = NN_Out; /* SFunction block Output */ // Additional Outputs y[0] = Z[0]; y[1] = Z[1]; y[2] = Z[2]; } /* Function: mdlUpdate ======* Abstract: * This function is called once for every major integration time step. * Discrete states are typically updated here. */ // #undef MDL_UPDATE #define MDL_UPDATE #if defined(MDL_UPDATE) static void mdlUpdate(SimStruct *S, int_T tid) { /* Update work vector here (1 second worth of input)*/ 200 real_T *x = (real_T*) ssGetDWork(S,0); InputRealPtrsType uPtrs = ssGetInputPortRealSignalPtrs(S,0); real_T temp_x[I+1]; int_T i; temp_x[I-1]=*uPtrs[0]; temp_x[I]=-1; for(i=0;i<=I-2;i++){ temp_x[i]=x[i+1]; } for(i=0;i<=I;i++){ x[i]=temp_x[i]; } } #endif // Function: mdlTerminate ======static void mdlTerminate(SimStruct *S) { } /*======* * Required S-function trailer * *======*/ #ifdef MATLAB_MEX_FILE /* Is this file being compiled as a MEX-file? */ #include "simulink.c" /* MEX-file interface mechanism */ #else #include "cg_sfun.h" /* Code generation registration function */ #endif 201 Appendix 9: PVDF Sleeve and Arduino Pro Mini System Parts List Table A9.1 Notable parts list for PVDF sleeve. Part Name Vendor Part Number PVDF Measurement Specialties 2‐1004346‐0 (Metalized Piezo Film Sheet) Compression Sleeve Eastbay (Under Armour Forearm Shiver Size s/m) Fabric Adhesive Michaels 10306719 (Aleene's Super Fabric Adhesive) Shaping Tube Home Depot 22786 (3"x2" ABS Bushing) Arduino Pro Mini 328 (3.3V 8MHz) Sparkfun DEV‐11114 FTDI Basic Breakout (3.3V) Sparkfun DEV‐09873 Bluefruit EZ‐Link Bluetooth Adapter Adafruit 1588 Bluetooth Adapter (Dongle) Adafruit 1327 Lithium Ion Polymer Battery (3.7V) Adafruit 1317 Battery Charger Adafruit 1304 JST Right‐Angle (Battery) Connector Sparkfun PRT‐09749 Female Crimp Pins (for 0.1" Housing) Pololu 1930 Crimp Connector Housing (1x2 Pin) Pololu 1901 JST XH‐Style Shrouded Male Connector Pololu 2718 Crimping Tool Pololu 1929 26 Gauge Hook Up Wire Mouser 566‐9985‐100‐10 PCB (Rolled) Eyelets Mouser 534‐22 Terminal Top Setting Tool Mouser 534‐1714 Terminal Ring Tongue Mouser 571‐33695 202 Appendix 10: Custom PCB Parts List for PVDF Sleeve Table A10.1 Notable parts list for PVDF sleeve. Part Name Vendor Part Number Quantity PCB OSHPark x 1 ATmega328p Mouser ATMEGA328P‐AU 1 3.3V LVR Mouser 584‐ADP160AUJZ‐3.3R7 1 330 Ω Resistor Mouser 71‐TNPW1206330RBEEA 2 100 Ω Resistor Mouser 71‐TNPW1206100RBEEA 2 1 MΩ Resistor Mouser 71‐TNPW12061M00BEEA 1 0.1 μF Capacitor Mouser 581‐1206ZC104JAT2A 1 1 μF Capacitor Mouser 581‐12063C105JAT2A 3 Molex Pico‐Lock Housing Mouser 538‐503763‐0291 1 Molex Pico‐Lock Wire Housing Mouser 538‐503764‐0201 1 Pico‐Lock Wire Terminals Mouser 538‐503765‐0098 2 Right Angle Switch Sparkfun 10860 1 DIP Switch (1 Pos) Mouser 611‐TDA01H0SB1R 1 LED Sparkfun 12622 2 XBee 1mW Trace Antenna Series 1 Sparkfun 11215 1 2mm 10pin XBee Socket – SMD Sparkfun 10030 2 JST Right Angle Connector Sparkfun 8612 1 3.7V Lithium Ion Polymer Battery (500mAh) Adafruit 1578 1 AVR Pocket Programmer6 Sparkfun 9825 1 6 Used to flash code to ATmega328p 203 Appendix 11: Custom PCB Circuit Diagram Figure A11.1. Full circuit schematic Figure A11.2. LVR subsection 204 Figure A11.3. Power switch subsection Figure A11.4 XBee subsection 205 Figure 11.5. DIP switch subsection Figure A.11.6 PVDF connection subsection 206 Figure A11.7. MCU subsection 207 Appendix 12: Arduino Voltage Sampling Settings and Code The Arduino used for sampling the voltages generated by the PVDF was an Arduino Mini (3.3V 8MHz)7 powered by an ATmega328p microcontroller8. To be able to get a sampling rate of 5ms the timer, timer0, had to tick such that 5ms would be some multiple of ticks. The default was that timer tics once every 64 clock cycles9, which for a microcontroller with an 8MHz oscillator is every 1 64 8s (A12.1) 8,000,000 which is not a factor of 5ms. By changing the bitset of the TCCR0B register from prescale 64 (0 1 1) to 8 (0 1 0)10 the timer ticks every 8 clock cycles or every 1 81s (A12.2) 8,000,000 which is a factor of 5ms. The timer value is accessed through reading the 8 bit register TCNT0; for each timer tick the value read from TCNT0 increases by 1. However, being only 8 bits the maximum ticks before an overflow is 218 or 255 equating to 255μs. In comes timer0_overflow_count3 which tracks the number of times TCNT0 has reset. Using the left bit‐shift operator << the value of 7 https://www.sparkfun.com/products/11114 8http://www.atmel.com/images/Atmel‐8271‐8‐bit‐AVR‐Microcontroller‐ATmega48A‐48PA‐88A‐88PA‐168A‐168PA‐ 328‐328P_datasheet_Complete.pdf 9 C:\Program Files (x86)\Arduino\hardware\arduino\cores\arduino\wiring.c 10 Table 15‐9 of footnote 2 208 timer0_overflow_count is converted back to clock ticks (timer0_overflow_count <<8) and added to the current TCNT0 value to give a total elapsed time. Each overflow is therefore 1<<8=256 ticks =256μs. /* ------> The following was written for an Arduino Mini: ATmega328p running at 8MHz. If running on Uno at 16MHz then byte x set in code to double desired sampling rate to account for double clock speed. Specify desired sampling rate ----> byte x Specify board clock speed: If mini set: byte ck8 = 1; byte ck16 = 0; If uno set: byte ck8 = 0; byte ck16 = 1; ------ Program records and serial prints analog read value from arduino at specified sampling rate x effectively creating a DAQ system. Uses timer0_overflow_count and TCNTO. See Ard_DAQ_Timing for fully commented file. To change Timer0 ticking frequency from default 64 to 8 need to adjust register prescale. Necessary register: TCCR0B Necessary bits: CS01 & CS00 To change ADC speed from default 128 to 8 need to adjust ADC register prescale. Necessary register: ADCSRA Necessary bits: ADPS2, ADPS1, ADPS0 Ncessary register: CLKPR Clock Division Factor 1: CLKPS3 = 0 CLKPS2 = 0 CLKPS1 = 0 CLKPS0 = 0 Clock Division Factor 2: CLKPS3 = 0 CLKPS2 = 0 CLKPS1 = 0 CLKPS0 = 1 */ //Define AVR commands cbi (clear bit) and sbi (set bit) #ifndef cbi #define cbi(sfr, bit) (_SFR_BYTE(sfr) &= ~_BV(bit)) #endif #ifndef sbi #define sbi(sfr, bit) (_SFR_BYTE(sfr) |= _BV(bit)) #endif 209 // Timer related variables extern volatile unsigned long timer0_overflow_count; unsigned long time; unsigned long time0=0; // Sampling rate related variables byte x = 5; // Sampling interval 2 milliseconds byte ck8 = 1; // set to 1 if using an 8MHz board, 0 otherwise byte ck16 = 0; // set to 1 if using an 16MHz board, 0 otherwise // Pin definitions byte analogPin=0; // Pin PVDF+ is connected to int aR=0; void setup() { // Adjusting CLKPR Register (Clock Prescale Register) /* cbi(CLKPR,CLKPS3); // Clear CLKPR:CLKPS3 to 0 cbi(CLKPR,CLKPS2); // Clear CLKPR:CLKPS2 to 0 cbi(CLKPR,CLKPS1); // Clear CLKPR:CLKPS1 to 0 cbi(CLKPR,CLKPS0); // Clear CLKPR:CLKPS0 to 0 */ // Adjusting TCCR0B Register (Timer/Counter B for Clock Select) sbi(TCCR0B,CS01); // Set TCCR0B:CS01 to 1 cbi(TCCR0B,CS00); // Clear TCCR0B:CS00 to 0 // Adjust ACD Speed from default prescale 128 to 8 (62.5kHz to 1MHz for 8MHz System Clock) cbi(ADCSRA, ADPS2); sbi(ADCSRA, ADPS1); sbi(ADCSRA, ADPS0); // If using an Uno or other 16 MHz Arduino if((ck8 == 0) & (ck16 == 1)){ x = 2*x; } // Baud rate MUST be 9600 for Android compatibility Serial.begin(9600); // setup serial baud rate } //Loop calls analogRead once each 5ms void loop() { time=((timer0_overflow_count << 8) + TCNT0)/1000; //ms if((time % x==0) & time!=time0){ aR=analogRead(analogPin); // read A0 pin // Output for all applications (Android and VB) Serial.print(aR); Serial.print("#"); // Delimeter character // For Arduino Serial Monitor Output //aR=1023; // Dummy transmit variable //Serial.println(aR); time0=time; } } 210 Appendix 13: ATMega328P Code for PVDF Sleeve /* * Div 8 Removed in Fuse Bits */ #ifndef F_CPU #define F_CPU 8000000UL // in conjunction with CLKPR #endif #ifndef BAUD #define BAUD 9600UL // set BAUD rate #endif #include #define MYUBRR 51 // Table 20-1 #define ADC0 PCINT8 // PC0 - PIN 23 /* ---- Global Variables ---- */ volatile uint8_t j; // counter for timer overflow volatile uint8_t j_comp = 2; volatile uint8_t r; //counter to iterate through Right3 // Lookup Table const uint16_t Right3[45][1] PROGMEM= { {0}, {0}, {0}, {0}, {0}, {511}, {508}, {504}, {500}, {495}, {488}, {498}, {539}, {564}, {577}, {567}, {546}, {521}, {508}, {484}, {495}, {498}, {496}, {494}, 211 {494}, {492}, {490}, {497}, {496}, {496}, {502}, {509}, {513}, {511}, {511}, {511}, {511}, {511}, {511}, {511}, {0}, {0}, {0}, {0}, {0} }; /* ---- Function Definitions ---- */ // Initialize USART void USART_Init(unsigned int ubrr){ UBRR0H = (unsigned char)(ubrr>>8); // Setting baud rate UBRR0L = (unsigned char)ubrr; UCSR0B = (1< // Serial Print a String (Make: AVR Programming by Elliot Williams) void printString(const char myString[]) { uint8_t i = 0; while (myString[i]) { USART_Transmit(myString[i]); i++; } } // Single Character Split and Transmit (Make: AVR Programming by Elliot Williams) void printByte(uint16_t data) { /* Converts 10 bit ADC reading to a string of decimal text, sends it */ if((data<1000)&&(data>=100)){ USART_Transmit('0' + (data / 100)); /* Hundreds */ _delay_us(50); USART_Transmit('0' + ((data / 10) % 10)); /* Tens */ _delay_us(50); USART_Transmit('0' + (data % 10)); /* Ones */ _delay_us(50); } 212 else if(data >= 1000){ USART_Transmit('0' + (data / 1000)); /* Thousands */ _delay_us(50); USART_Transmit('0' + ((data / 100) % 10)); /* Hundreds */ _delay_us(50); USART_Transmit('0' + ((data / 10) % 10)); /* Tens */ _delay_us(50); USART_Transmit('0' + (data % 10)); /* Ones */ _delay_us(50); } else if((data<100)&&(data>=10)){ USART_Transmit('0' + (data / 10)); /* Tens */ _delay_us(50); USART_Transmit('0' + (data % 10)); /* Ones */ _delay_us(50); } else{ USART_Transmit('0' + data); /* Ones */ _delay_us(50); } printString("#"); _delay_us(50); } // Initialize ADC void initADC(void) { ADMUX |= (1 << REFS0); // Ref voltage AVCC ADMUX |= (0 << ADLAR); // Right adjusted ADCSRA |= (0 << ADATE); // Disable auto trigger ADCSRA |= (1 << ADEN); // Enable ADC ADCSRA |= (5 << ADPS0); // ADC prescaler (32) } // Read and Send ADC Value void read_send_ADC(uint8_t adcPin){ ADMUX = (0b11110000 & ADMUX)|adcPin; // set ADC channel ADCSRA |=(1< printByte(ADC); //printString("#"); } // Initialize 8-bit Timer0 void initTimer0(void){ //TCCR0B |= (1< // Timer0 Overflow Interrupt ISR(TIMER0_OVF_vect){ j++; if(j==j_comp){ 213 read_send_ADC(ADC0); j=0; } } /* ---- Main ---- */ int main(void) { j = 0; // Initialize j r=0; sei(); // Set global interrupt enable bit USART_Init(MYUBRR); // Initialize Serial from USART initADC(); // Initialize ADC initTimer0(); // Initialize Timer0 // Event Loop while(1){ // Just loops work done by interrupt } return(0); // Required } 214 Appendix 14: VB.NET Data Collection Application In conjunction with the code in Appendicies 12 and 13, which dictated the data sending protocols, a code was needed to collect and write the sent data to a file for the purpose of training the neural network detailed in Chapter 5. If desired, the program will also give a scrolling oscilloscope style plot and disply the read ADC value. The following was written in .NET 4.5 using Visual Studios as a Windows Form Application. Figure A14.1 Arduino data acquisition program user interface Option Explicit On Imports System Imports System.ComponentModel Imports System.IO.Ports Imports System.Management Imports System.Threading Imports System.Windows.Forms.FileDialog Imports System.Windows.Forms.DataVisualization.Charting Public Class main Dim sp As New SerialPort Dim ArdOut As System.IO.StreamWriter Dim fileCOM As String 215 Dim Baud As String = "9600" Dim Port As String Dim delimCount As Integer = 0 Dim delimStr As String Dim ardDelim As String = "#" Dim ardRate As String = "25" Dim avrDelim As String = "" Dim avrRate As String = "25.6" Dim analogV As String Dim dt As Double 'Sampling rate from form text box (in ms) Dim dummySeries As New Series Dim serialData1 As New Series Dim serialData2 As New Series Dim refLine As New Series Dim xAxisMax As Double = 10 Dim yAxisMin As Double = 0 Dim yAxisMax As Double = 1023 Dim xAxisInterval As Double = 1 Dim yAxisInterval As Double = 1023 Dim xAxisMajorGridLines As Boolean = 0 Dim xAxisMinorGridLines As Boolean = 0 Dim yAxisMajorGridLines As Boolean = 0 Dim yAxisMinorGridLines As Boolean = 0 Dim plotUpdateF As Integer Dim countUpdateF As Integer Dim y As Double Dim t As Double Dim toggle As Byte Dim i As Long Dim j As Long Dim updatePlot As Byte Dim k_max As Short Private Sub JoystickUI_Load(sender As Object, e As EventArgs) Handles MyBase.Load lbCom.Items.Clear() lbBaud.Items.Clear() lbBaud.Items.Add("300") lbBaud.Items.Add("600") lbBaud.Items.Add("1200") lbBaud.Items.Add("2400") lbBaud.Items.Add("4800") lbBaud.Items.Add("9600") lbBaud.Items.Add("14400") lbBaud.Items.Add("19200") 216 lbBaud.Items.Add("28800") lbBaud.Items.Add("38400") lbBaud.Items.Add("57600") lbBaud.Items.Add("115200") btnComselect.Enabled = False tbFilepath.Enabled = False btnStartcom.Enabled = False btnEndcom.Enabled = False btnReset.Enabled = False rbArduino.Enabled = False rbAVR.Enabled = False btnGetADC.Enabled = False tbGetADC.Enabled = False tbGetADC.Text = "" Chart1.Series.Clear() dummySeries.ChartType = SeriesChartType.Line Chart1.Series.Add(dummySeries) serialData1.ChartType = SeriesChartType.Line Chart1.Series.Add(serialData1) serialData2.ChartType = SeriesChartType.Line Chart1.Series.Add(serialData2) refLine.ChartType = SeriesChartType.Line Chart1.Series.Add(refLine) Chart1.ChartAreas(0).AxisX.Minimum = 0 Chart1.ChartAreas(0).AxisX.Maximum = xAxisMax Chart1.ChartAreas(0).AxisX.Interval = xAxisInterval Chart1.ChartAreas(0).AxisY.Minimum = yAxisMin Chart1.ChartAreas(0).AxisY.Maximum = yAxisMax Chart1.ChartAreas(0).AxisY.Interval = yAxisInterval lblXAxisULim.Text = "X-Axis Upper Limit: " & xAxisMax.ToString tbXAxisULim.Text = xAxisMax.ToString lblXAxisIntvl.Text = "X-Axis Interval: " & xAxisInterval.ToString tbXAxisInt.Text = xAxisInterval.ToString lblYAxisLLim.Text = "Y-Axis Lower Limit: " & yAxisMin.ToString tbYAxisLLim.Text = yAxisMin.ToString.ToString lblYAxisULim.Text = "Y-Axis Upper Limit: " & yAxisMax.ToString tbYAxisULim.Text = yAxisMax.ToString lblYAxisIntvl.Text = "Y-Axis Interval: " & yAxisInterval.ToString tbYAxisInt.Text = yAxisInterval.ToString If xAxisMajorGridLines = True Then Chart1.ChartAreas(0).AxisX.MajorGrid.Enabled = True lblXAxisMajGrid.Text = "X-Axis Major Grid Lines: " & "Yes" cbXMajGrid.Checked = True Else Chart1.ChartAreas(0).AxisX.MajorGrid.Enabled = False 217 lblXAxisMajGrid.Text = "X-Axis Major Grid Lines: " & "No" cbXMajGrid.Checked = False End If If xAxisMinorGridLines = True Then Chart1.ChartAreas(0).AxisX.MinorGrid.Enabled = True lblXAxisMinGrid.Text = "X-Axis Minor Grid Lines: " & "Yes" cbXMinGrid.Checked = True Else Chart1.ChartAreas(0).AxisX.MinorGrid.Enabled = False lblXAxisMinGrid.Text = "X-Axis Minor Grid Lines: " & "No" cbXMinGrid.Checked = False End If If yAxisMajorGridLines = True Then Chart1.ChartAreas(0).AxisY.MajorGrid.Enabled = True lblYAxisMajGrid.Text = "Y-Axis Major Grid Lines: " & "Yes" cbYMajGrid.Checked = True Else Chart1.ChartAreas(0).AxisY.MajorGrid.Enabled = False lblYAxisMajGrid.Text = "Y-Axis Major Grid Lines: " & "No" cbYMajGrid.Checked = False End If If yAxisMinorGridLines = True Then Chart1.ChartAreas(0).AxisY.MinorGrid.Enabled = True lblYAxisMinGrid.Text = "Y-Axis Minor Grid Lines: " & "Yes" cbYMinGrid.Checked = True Else Chart1.ChartAreas(0).AxisY.MinorGrid.Enabled = False lblYAxisMinGrid.Text = "Y-Axis Minor Grid Lines: " & "No" cbYMinGrid.Checked = False End If Chart1.ChartAreas(0).AxisX.Title = "Time (s)" Chart1.ChartAreas(0).AxisY.Title = "ADC Value" dummySeries.IsVisibleInLegend = False serialData1.IsVisibleInLegend = False serialData2.IsVisibleInLegend = False refLine.IsVisibleInLegend = False dummySeries.Color = Color.White dummySeries.BorderWidth = 2 serialData1.Color = Color.Red serialData1.BorderWidth = 2 serialData2.Color = Color.Red serialData2.BorderWidth = 2 refLine.Color = Color.Blue refLine.BorderWidth = 2 dummySeries.Points.AddXY(1, 1) Me.DoubleBuffered = True tabSerial.Enabled = False tabPlot.Enabled = False 218 btnSetRefLine.Enabled = False tbRunningADC.Enabled = False cbRunningADC.Enabled = False cbRunningADC.Checked = False plotUpdateF = 1 lblUpdate.Text = "Point Update Freq: " & Str(plotUpdateF) countUpdateF = 1 End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Refresh COM List button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnRefresh_Click(sender As Object, e As EventArgs) Handles btnRefresh.Click lbBaud.ClearSelected() lbCom.Enabled = True lbCom.Items.Clear() For Each sp As String In My.Computer.Ports.SerialPortNames lbCom.Items.Add(sp) Next lbCom.Sorted = True 'Sorts listbox lbCom alphabetically after populated btnComselect.Enabled = False btnStartcom.Enabled = False btnEndcom.Enabled = False btnReset.Enabled = False lbBaud.ClearSelected() cbCom.Checked = False rbArduino.Enabled = False rbAVR.Enabled = False btnGetADC.Enabled = False tbGetADC.Enabled = False tbGetADC.Text = "" updatePlot = 1 Baud = "9600" 'Default BAUD rate delimStr = ardDelim 'Default delimiter dt = ardRate 'Default data sampling rate tabSerial.Enabled = False tabSerial.SelectTab(0) lblSettingsBaud.Text = "Baud Rate: " & Baud lblSettingsDelim.Text = "Delimiter: " & delimStr lblSettingsRate.Text = "Sampling Rate (ms): " & dt btnSetRefLine.Enabled = False 219 tabPlot.Enabled = False tabPlot.SelectTab(0) tbRunningADC.Enabled = False cbRef.Checked = False cbRunningADC.Enabled = False cbRunningADC.Checked = False plotUpdateF = 1 lblUpdate.Text = "Point Update Freq: " & Str(plotUpdateF) countUpdateF = 1 End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Com List Box SelectIndexChanged event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub comSel(ByVal sender As Object, ByVal e As EventArgs) Handles lbCom.SelectedIndexChanged Dim index As Integer = lbCom.SelectedIndex If index <> -1 Then btnComselect.Enabled = True End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Select COM button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnComselect_Click(sender As Object, e As EventArgs) Handles btnComselect.Click btnComselect.Enabled = False rbArduino.Enabled = True rbArduino.Checked = True rbAVR.Enabled = True tabSerial.Enabled = True tabPlot.Enabled = True btnStartcom.Enabled = True End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 220 ' Baud List Box SelectIndexChanged event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub baudSel(ByVal sender As Object, ByVal e As EventArgs) Dim index As Integer = lbBaud.SelectedIndex If index <> -1 Then btnBaud.Enabled = True End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Select BAUD Rate button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnBaud_Click(sender As Object, e As EventArgs) Handles btnBaud.Click Baud = lbBaud.SelectedItem.ToString lblSettingsBaud.Text = "Baud Rate: " & Baud tabSerial.SelectTab(0) lbBaud.ClearSelected() End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Radio Button Arduino CheckedChanged event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub rbArduio_CheckChanged(sender As Object, e As EventArgs) Handles rbArduino.CheckedChanged If rbArduino.Checked = True Then tabSerial.SelectTab(0) lblSettingsBaud.Text = "Baud Rate: " & Baud lblSettingsDelim.Text = "Delimiter: " & ardDelim lblSettingsRate.Text = "Sampling Rate (ms): " & ardRate tbDelim.Text = ardDelim tbRate.Text = ardRate tabPlot.SelectTab(0) xAxisMax = 10 yAxisMin = 0 yAxisMax = 1023 yAxisInterval = 1023 221 Chart1.ChartAreas(0).AxisX.Maximum = xAxisMax Chart1.ChartAreas(0).AxisY.Minimum = yAxisMin Chart1.ChartAreas(0).AxisY.Maximum = yAxisMax Chart1.ChartAreas(0).AxisY.Interval = yAxisInterval lblXAxisULim.Text = "X-Axis Upper Limit: " & xAxisMax.ToString tbXAxisULim.Text = xAxisMax.ToString lblXAxisIntvl.Text = "X-Axis Interval: " & xAxisInterval.ToString tbXAxisInt.Text = xAxisInterval.ToString lblYAxisLLim.Text = "Y-Axis Lower Limit: " & yAxisMin.ToString tbYAxisLLim.Text = yAxisMin.ToString.ToString lblYAxisULim.Text = "Y-Axis Upper Limit: " & yAxisMax.ToString tbYAxisULim.Text = yAxisMax.ToString lblYAxisIntvl.Text = "Y-Axis Interval: " & yAxisInterval.ToString tbYAxisInt.Text = yAxisInterval.ToString End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Radio Button AVR CheckedChanged event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub rbAVR_CheckChanged(sender As Object, e As EventArgs) Handles rbAVR.CheckedChanged If rbAVR.Checked = True Then tbDelim.Text = avrDelim tbRate.Text = avrRate lblSettingsBaud.Text = "Baud Rate: " & Baud lblSettingsDelim.Text = "Delimiter: " & avrDelim lblSettingsRate.Text = "Sampling Rate (ms): " & avrRate tabSerial.SelectTab(0) tabPlot.SelectTab(0) xAxisMax = 10 yAxisMin = 100 yAxisMax = 200 yAxisInterval = 10 Chart1.ChartAreas(0).AxisX.Maximum = xAxisMax Chart1.ChartAreas(0).AxisY.Minimum = yAxisMin Chart1.ChartAreas(0).AxisY.Maximum = yAxisMax Chart1.ChartAreas(0).AxisY.Interval = yAxisInterval lblXAxisULim.Text = "X-Axis Upper Limit: " & xAxisMax.ToString tbXAxisULim.Text = xAxisMax.ToString lblXAxisIntvl.Text = "X-Axis Interval: " & xAxisInterval.ToString 222 tbXAxisInt.Text = xAxisInterval.ToString lblYAxisLLim.Text = "Y-Axis Lower Limit: " & yAxisMin.ToString tbYAxisLLim.Text = yAxisMin.ToString.ToString lblYAxisULim.Text = "Y-Axis Upper Limit: " & yAxisMax.ToString tbYAxisULim.Text = yAxisMax.ToString lblYAxisIntvl.Text = "Y-Axis Interval: " & yAxisInterval.ToString tbYAxisInt.Text = yAxisInterval.ToString End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set Delimiter button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnSet_Delim(sender As Object, e As EventArgs) Handles btnDelim.Click delimStr = tbDelim.Text.ToString lblSettingsDelim.Text = "Delimiter: " & delimStr tabSerial.SelectTab(0) End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set Sampling Rate button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnRate_Click(sender As Object, e As EventArgs) Handles btnRate.Click dt = tbRate.Text lblSettingsRate.Text = "Sampling Rate (ms): " & dt.ToString tabSerial.SelectTab(0) End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' COM feed file path: textbox click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub tbFilepath_Click(sender As Object, e As EventArgs) Handles tbFilepath.Click Dim saveFileDialog1 As New SaveFileDialog saveFileDialog1.Filter = "TXT Files (*.txt*)|*.txt" saveFileDialog1.Title = "Serial Data Output File" If saveFileDialog1.ShowDialog = Windows.Forms.DialogResult.OK Then fileCOM = saveFileDialog1.FileName.ToString btnStartcom.Enabled = True Else fileCOM = "" btnStartcom.Enabled = False 223 End If tbFilepath.Text = fileCOM End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Write COM Data to File Check Box '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub cbCom_CheckedChanged(sender As Object, e As EventArgs) Handles cbCom.CheckedChanged If cbCom.Checked = True Then tbFilepath.Enabled = True btnStartcom.Enabled = False Else tbFilepath.Text = "" tbFilepath.Enabled = False btnStartcom.Enabled = True End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' End button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnEndcom_Click(sender As Object, e As EventArgs) Handles btnEndcom.Click Try RemoveHandler sp.DataReceived, AddressOf DataReceivedHandler 'Application.DoEvents() updatePlot = 0 btnReset.Enabled = True btnStartcom.Enabled = False btnEndcom.Enabled = False If cbCom.Checked = True Then ArdOut.Close() 'Close the text file COM feed was being written to End If sp.Close() 'Close the selected serial port to get out of Do loop in SerialFromArduino delimCount = 0 Catch exIOexcept As Exception MsgBox("End Catch") Exit Sub End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Reset button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnReset_Click(sender As Object, e As EventArgs) Handles btnReset.Click lbCom.Items.Clear() lbBaud.ClearSelected() serialData1.Points.Clear() serialData2.Points.Clear() 224 refLine.Points.Clear() btnGetADC.Enabled = False tbGetADC.Enabled = False tbGetADC.Text = "" tbRunningADC.Enabled = False tbRunningADC.Text = "" tbRate.Text = "" tbDelim.Text = "" btnReset.Enabled = False btnRefresh.Enabled = True cbRunningADC.Enabled = False cbRunningADC.Checked = False plotUpdateF = 1 lblUpdate.Text = "Point Update Freq: " & Str(plotUpdateF) countUpdateF = 1 End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Start button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnStartcom_Click(sender As Object, e As EventArgs) Handles btnStartcom.Click Dim index As Integer = lbCom.SelectedIndex tabSerial.Enabled = False tabPlot.Enabled = False rbArduino.Enabled = False rbAVR.Enabled = False btnGetADC.Enabled = True tbGetADC.Enabled = True tbRunningADC.Enabled = True cbRunningADC.Enabled = True i = -1 j = -1 toggle = 0 y = dt / 1000 Port = lbCom.SelectedItem.ToString Try sp.BaudRate = Baud sp.PortName = Port sp.Encoding = System.Text.Encoding.ASCII sp.Open() sp.ReadTimeout = 5000 'Give serial port 5s (5000 ms) to open 225 lbCom.Enabled = False btnRefresh.Enabled = False btnComselect.Enabled = False btnStartcom.Enabled = False btnEndcom.Enabled = True btnReset.Enabled = False If cbCom.Checked = True Then ArdOut = My.Computer.FileSystem.OpenTextFileWriter(fileCOM, False) End If sp.DiscardInBuffer() AddHandler sp.DataReceived, AddressOf DataReceivedHandler Catch MsgBox("Error: Could not connect to selected COM port. Make sure device is powered on.", vbOKOnly, "Connection Error") sp.Close() End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Serial Port Data Received Event Handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub DataReceivedHandler(sender As Object, e As SerialDataReceivedEventArgs) If rbArduino.Checked = True Then Try analogV = sp.ReadTo(delimStr) 'MsgBox(analogV) If delimCount = 0 Then delimCount = 1 Else If cbCom.Checked = True Then ArdOut.WriteLine(analogV) End If If updatePlot = 1 Then Me.Invoke(New EventHandler(AddressOf DoUpdate)) End If End If Catch exInvalidOpp As InvalidOperationException 'MsgBox("DataReceivedHandler Invalid Operation Catch") Catch exIOExcept As Exception 'MsgBox("DataReceivedHandler Other") End Try End If If rbAVR.Checked = True Then Try analogV = sp.ReadByte 226 If delimCount = 0 Then delimCount = 1 Else If cbCom.Checked = True Then ArdOut.WriteLine(analogV) End If If updatePlot = 1 Then Me.Invoke(New EventHandler(AddressOf DoUpdate)) End If End If Catch exInvalidOpp As InvalidOperationException 'MsgBox("DataReceivedHandler Invalid Operation Catch") Catch exIOExcept As Exception 'MsgBox("DataReceivedHandler Other") End Try End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' DoUpdate() Function Handles GUI Updates : Plotting and ADC text box '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Sub DoUpdate() Try If cbRunningADC.Checked = True Then tbRunningADC.Text = analogV Else tbRunningADC.Text = "" End If Select Case plotUpdateF Case 0 Case countUpdateF Dim k As Short If toggle = 0 Then ' serialData1 i = i + 1 t = i * y * countUpdateF If j = -1 Then ' first time serialData1 being plotted Try serialData1.Points.AddXY(t, analogV) Catch exIOExcept As Exception MsgBox("DoUpdate 1st Exception") Exit Sub End Try ElseIf j = 0 Then 'should never be in here MsgBox("I shouldn't be here") Else k = 1 / y + i If k * y < xAxisMax Then 227 Try serialData2.Points.RemoveAt(0) Catch exIOExcept As Exception MsgBox("DoUpdate 4th Exception") Exit Sub End Try Else Try serialData2.Points.Clear() Catch exIOExcept As Exception MsgBox("DoUpdate 5th Exception") Exit Sub End Try End If Try serialData1.Points.AddXY(t, analogV) Catch exIOExcept As Exception MsgBox("DoUpdate 6th Exception") Exit Sub End Try End If If t >= xAxisMax Then i = -1 toggle = 1 ' toggles over to serialData2 j = 0 End If End If If toggle = 1 Then ' serialData2 i = i + 1 t = i * y * countUpdateF ' j=0 first time being toggled over. clears some of old series (serialData1) and adds first point of serialData2 ' (same as last point at max time other toggle, this time at t=0) If j = 0 Then k_max = 0 For k = 0 To 1 / y Try serialData1.Points.RemoveAt(0) Catch exIOExcept As Exception MsgBox("DoUpdate 7th Exception") Exit Sub End Try Next k_max = k Try serialData2.Points.AddXY(t, analogV) Catch exIOExcept As Exception MsgBox("DoUpdate 8th Exception") Exit Sub End Try j = 1 Else ' rest of the time this toggle goes through this else statement 228 k = k_max + i If k * y < xAxisMax Then Try serialData1.Points.RemoveAt(0) Catch exIOExcept As Exception MsgBox("DoUpdate 9th Exception") Exit Sub End Try Else Try serialData1.Points.Clear() Catch exIOExcept As Exception MsgBox("DoUpdate 10th Exception") Exit Sub End Try End If Try serialData2.Points.AddXY(t, analogV) Catch exIOExcept As Exception MsgBox("DoUpdate 11th Exception") Exit Sub End Try End If If t >= xAxisMax Then ' Start of Added k_max = 0 For k = 0 To 1 / y Try serialData2.Points.RemoveAt(0) Catch exIOExcept As Exception MsgBox("DoUpdate 2nd Exception") Exit Sub End Try Next 'k_max = k Try t = 0 serialData1.Points.AddXY(t, analogV) Catch exIOExcept As Exception MsgBox("DoUpdate 3rd Exception") Exit Sub End Try j = 1 ' End of Added i = -1 toggle = 0 'j = 0 End If End If countUpdateF = 1 Case Else countUpdateF = countUpdateF + 1 229 End Select Catch exIOExcept As Exception MsgBox("DoUpdate 12th Exception") Exit Sub End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' GetADC button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnGetADC_Click(sender As Object, e As EventArgs) Handles btnGetADC.Click tbGetADC.Text = analogV End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set X-Axis plot properties button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnXAxis_Click(sender As Object, e As EventArgs) Handles btnXAxis.Click xAxisMax = tbXAxisULim.Text Chart1.ChartAreas(0).AxisX.Maximum = xAxisMax lblXAxisULim.Text = "X-Axis Upper Limit: " & xAxisMax.ToString xAxisInterval = tbXAxisInt.Text Chart1.ChartAreas(0).AxisX.Interval = xAxisInterval lblXAxisIntvl.Text = "X-Axis Interval: " & xAxisInterval.ToString If cbXMajGrid.Checked = True Then 'xAxisMajorGridLines = 1 lblXAxisMajGrid.Text = "X-Axis Major Grid Lines: " & "Yes" Chart1.ChartAreas(0).AxisX.MajorGrid.Enabled = True Else 'xAxisMajorGridLines = 0 lblXAxisMajGrid.Text = "X-Axis Major Grid Lines: " & "No" Chart1.ChartAreas(0).AxisX.MajorGrid.Enabled = False End If If cbXMinGrid.Checked = True Then 'xAxisMinorGridLines = 1 lblXAxisMinGrid.Text = "X-Axis Minor Grid Lines: " & "Yes" Chart1.ChartAreas(0).AxisX.MinorGrid.Enabled = True Else 'xAxisMinorGridLines = 0 lblXAxisMinGrid.Text = "X-Axis Minor Grid Lines: " & "No" Chart1.ChartAreas(0).AxisX.MinorGrid.Enabled = False End If tabPlot.SelectTab(0) End Sub 230 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set Y-Axis plot properties button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnYAxis_Click(sender As Object, e As EventArgs) Handles btnYAxis.Click yAxisMin = tbYAxisLLim.Text Chart1.ChartAreas(0).AxisY.Minimum = yAxisMin lblYAxisLLim.Text = "Y-Axis Lower Limit: " & yAxisMin.ToString yAxisMax = tbYAxisULim.Text Chart1.ChartAreas(0).AxisY.Maximum = yAxisMax lblYAxisULim.Text = "Y-Axis Upper Limit: " & yAxisMax.ToString yAxisInterval = tbYAxisInt.Text Chart1.ChartAreas(0).AxisY.Interval = yAxisInterval lblYAxisIntvl.Text = "Y-Axis Interval: " & yAxisInterval.ToString If cbYMajGrid.Checked = True Then 'yAxisMajorGridLines = 1 lblYAxisMajGrid.Text = "Y-Axis Major Grid Lines: " & "Yes" Chart1.ChartAreas(0).AxisY.MajorGrid.Enabled = True Else 'yAxisMajorGridLines = 0 lblYAxisMajGrid.Text = "Y-Axis Major Grid Lines: " & "No" Chart1.ChartAreas(0).AxisY.MajorGrid.Enabled = False End If If cbYMinGrid.Checked = True Then 'yAxisMinorGridLines = 1 lblYAxisMinGrid.Text = "Y-Axis Minor Grid Lines: " & "Yes" Chart1.ChartAreas(0).AxisY.MinorGrid.Enabled = True Else 'yAxisMinorGridLines = 0 lblYAxisMinGrid.Text = "Y-Axis Minor Grid Lines: " & "No" Chart1.ChartAreas(0).AxisY.MinorGrid.Enabled = False End If tabPlot.SelectTab(0) End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Display Reference Value Line Check Box '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub cbRef_checkChange(sender As Object, e As EventArgs) Handles cbRef.CheckedChanged If cbRef.Checked = True Then tbRef.Enabled = True btnSetRefLine.Enabled = True btnStartcom.Enabled = False Else refLine.Points.Clear() tbRef.Enabled = False 231 tbRef.Text = "" btnSetRefLine.Enabled = False End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set Y reference line plot properties button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnSetRefLine_Click(sender As Object, e As EventArgs) Handles btnSetRefLine.Click If cbRef.Checked = True Then refLine.Points.Clear() Dim yValue As Short = tbRef.Text refLine.Points.AddXY(0, yValue) refLine.Points.AddXY(xAxisMax, yValue) btnStartcom.Enabled = True End If tabPlot.SelectTab(0) End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Set Data Point Plot Update Frequency button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnUpdateF_Click(sender As Object, e As EventArgs) Handles btnUpdateF.Click plotUpdateF = tbUpdateF.Text lblUpdate.Text = "Point Update Freq: " & Str(plotUpdateF) tabPlot.SelectTab(0) End Sub End Class 232 Appendix 15: Visual Studio Joystick Demo Setup and Code In conjunction with the code in Appendix 12 and 13, which dictated the data sending protocols, the code below is a first level demonstration of a PC responding to gestures performed (while wearing the joystick). The following was written in .NET 4.5 using Visual Studios as a Windows Form Application. Figure A15.1 PC joystick demo program user interface 233 ' Kyle Van Volkinburg ' 2014-11-4 ' Code to read in data from COM port and ' process it through neural network algoritm ' Note: No robust try/catch for file path locations Option Explicit On Imports System Imports System.ComponentModel Imports System.IO Imports System.IO.Ports Imports System.Management Imports System.IO.StreamWriter Public Class JoystickDemo Dim sp As New SerialPort Dim Baud As Integer = 9600 'Options: 9600 or 115200 (must match Arduino) Dim Port As String Dim I As Integer = 133 ' Input vector length not including bias Dim J As Integer = 3 ' Number of hidden nodes Dim K As Integer = 2 ' Output vector length (Right or Left) Dim wA(I + 1, J) As Double 'Weight Array A Dim wB(J + 1, K) As Double 'Weight Array B Dim X(I + 1) As Double 'Work vector Dim btRead As System.IO.StreamWriter ' Load weight arrays A and B, and initialize working vector X Private Sub JoystickDemo_Load(sender As Object, e As EventArgs) Handles MyBase.Load Dim wAdummy(,) As Double = New Double(,) { {-0.035042, 0.070964, 0.055705}, ' Actual Value removed, to be replace by user trained A weights {-0.051529, -0.119945, -0.143486}, } Dim wBdummy(,) As Double = New Double(,) { {1.763218, -0.781513}, ' Actual Value removed, to be replace by user trained B weights {-1.527316, 1.927683} } Dim a, b, c, m As Integer For a = 0 To I For b = 0 To J - 1 wA(a, b) = wAdummy(a, b) Next Next For b = 0 To J For c = 0 To K - 1 234 wB(b, c) = wBdummy(b, c) Next Next For m = 0 To I - 1 X(m) = 0 Next X(I) = -1 lbCom.Items.Clear() btnComselect.Enabled = False btnJon.Enabled = False btnJoff.Enabled = False btnReset.Enabled = False lblM.Visible = True lblM.Text = "" btnVersionInfo.Enabled = True ' Populate COM listbox on Form startup For Each sp As String In My.Computer.Ports.SerialPortNames lbCom.Items.Add(sp) Next lbCom.Sorted = True 'Sorts listbox lbCom alphabetically after populated End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Clicking the refresh button populates list box with available com ports '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnRefresh_Click(sender As Object, e As EventArgs) Handles btnRefresh.Click lbCom.Enabled = True lbCom.Items.Clear() For Each sp As String In My.Computer.Ports.SerialPortNames lbCom.Items.Add(sp) Next lbCom.Sorted = True 'Sorts listbox lbCom alphabetically after populated btnComselect.Enabled = False btnJon.Enabled = False btnJoff.Enabled = False btnReset.Enabled = False End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Selecting a com port from the list box enables the Select COM button '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub comSel(ByVal sender As Object, ByVal e As EventArgs) Handles lbCom.SelectedIndexChanged Dim index As Integer = lbCom.SelectedIndex If index <> -1 Then btnComselect.Enabled = True End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 235 'Connects program to selected COM port '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnComselect_Click(sender As Object, e As EventArgs) Handles btnComselect.Click Dim index As Integer = lbCom.SelectedIndex Port = lbCom.SelectedItem Try sp.BaudRate = Baud sp.PortName = Port sp.Open() sp.ReadTimeout = 5000 'Give serial port 5s (5000 ms) to open lbCom.Enabled = False btnRefresh.Enabled = False btnComselect.Enabled = False btnJon.Enabled = True btnJoff.Enabled = False btnReset.Enabled = True Catch MsgBox("Error: Could not connect to selected COM port. Make sure device is connected and powered on.", vbOKOnly, "Connection Error") sp.Close() End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Click Reset button event '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnClearcom_Click(sender As Object, e As EventArgs) Handles btnReset.Click btnReset.Enabled = False lbCom.Items.Clear() btnRefresh.Enabled = True btnComselect.Enabled = False btnJon.Enabled = False btnJoff.Enabled = False lblM.Text = "" Try sp.Close() Catch End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Starts Background worker asynchronous operation '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnJon_Click(sender As Object, e As EventArgs) Handles btnJon.Click btnReset.Enabled = False btnJon.Enabled = False btnJoff.Enabled = True 236 btnVersionInfo.Enabled = False Me.BackgroundWorker1.RunWorkerAsync() 'Start reading COM feed on second thread to keep GUI responsive End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Background Worker Async Thread '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub BackgroundWorker1_DoWork(sender As Object, e As DoWorkEventArgs) Handles BackgroundWorker1.DoWork 'Background worker variables Dim bw As BackgroundWorker = CType(sender, BackgroundWorker) Dim Incoming As String = "" ' New variables for using # delimiter instead of newline Dim chRead As Char ' Read char from serial stream Dim chrBuffer As String = "" ' String concatenating all the read chars (which are converted to string values) Dim delimCount As Integer = 0 ' Ignoring first read value b/c could be partial Dim delim As Char = Chr(35) ' Chr(35) is ASCII code for "#" ' Dim aR As Integer = 0 Dim NN_Out As Integer ' ADC to Voltage conversion variables Dim a As Double = -1.65 Dim b As Double = 1.65 Dim mn As Integer = 0 Dim mx As Integer = 1023 Dim ardV As Double 'Counter variables Dim m As Integer Dim n As Integer Dim o As Integer ' working vectors Dim u(J) As Double ' holds product of input node and corresponding wA value Dim v(K) As Double ' holds product of hidden node and corresponding wB value ' Node outputs Dim Y(J + 1) As Double ' Hidden node output value Dim Z(K) As Double ' Output note output vector Dim displayStopWatch As New Stopwatch() Dim displaySecs As Double = 0 Dim displayTicks As Long Dim spReadStopwatch As New Stopwatch() Dim spReadTicks As Long Dim spReadElapse As Double Dim btReadingsStopwatch As New Stopwatch() Dim btReadingsTicks As Long Dim btReadingsElapse As Double Dim btReadingsCount As Integer = 0 237 Do If bw.CancellationPending Then e.Cancel = True Exit Do End If Try If chrBuffer = "" Then spReadStopwatch.Start() End If chRead = Chr(sp.ReadChar()) If chRead <> delim Then chrBuffer = chrBuffer + Convert.ToString(chRead) Else 'Delimiter Reached spReadStopwatch.Stop() spReadTicks = spReadStopwatch.ElapsedTicks() spReadElapse = spReadTicks / frequency spReadStopwatch.Reset() Incoming = chrBuffer chrBuffer = "" If delimCount = 0 Then delimCount = 1 Else Try aR = Convert.ToInt32(Incoming) Catch ex As Exception 'MsgBox("aR = 511") aR = 511 End Try If aR >= 0 And aR <= 1023 Then ardV = ((b - a) * (aR - mn) / (mx - mn)) + a Else ardV = 0 End If ' Update work vector X with new value V going to position I-1 w/ position I value =-1 ALWAYS (again base 0) For m = 0 To I - 2 X(m) = X(m + 1) Next X(I - 1) = ardV 'NN Code here For n = 0 To J - 1 u(n) = 0 For m = 0 To I u(n) = u(n) + X(m) * wA(m, n) Next Y(n) = 1 / (1 + Math.Exp(-u(n))) Next 238 Y(J) = -1 For o = 0 To K - 1 v(o) = 0 For n = 0 To J v(o) = v(o) + Y(n) * wB(n, o) Next Z(o) = 1 / (1 + Math.Exp(-v(o))) Next 'Right Motion If Z(0) <= 0.95 And Z(0) >= 0.85 And Z(1) <= 0.15 And Z(1) >= 0.05 Then NN_Out = 1 For m = 0 To I - 1 X(m) = 0 Next X(I) = -1 End If 'Left Motion If Z(1) <= 0.95 And Z(1) >= 0.85 And Z(0) <= 0.15 And Z(0) >= 0.05 Then NN_Out = 2 For m = 0 To I - 1 X(m) = 0 Next X(I) = -1 End If If NN_Out = 1 Or NN_Out = 2 Then displayStopWatch.Reset() displayStopWatch.Start() If btReadingsCount = 0 Then btReadingsStopwatch.Reset() btReadingsStopwatch.Start() btReadingsCount = 1 Else btReadingsTicks = btReadingsStopwatch.ElapsedTicks() btReadingsElapse = btReadingsTicks / frequency btReadingsStopwatch.Stop() btReadingsCount = 0 End If bw.ReportProgress(NN_Out) End If displayTicks = displayStopWatch.ElapsedTicks displaySecs = displayTicks / frequency ' 0.5 is time (s) for display to show results If displaySecs > 0.5 Then displaySecs = 0 NN_Out = 10 displayStopWatch.Stop() bw.ReportProgress(NN_Out) End If NN_Out = 0 End If 239 End If Catch exInvalidOpp As InvalidOperationException Exit Do Catch exIOExcept As Exception Exit Do End Try Loop End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' BackgroundWorker Progress Change Event '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub BackgroundWorker1_ProgressChanged(sender As Object, e As ProgressChangedEventArgs) Handles BackgroundWorker1.ProgressChanged If e.ProgressPercentage = 1 Then lblM.Text = "Right" 'L = 1 ElseIf e.ProgressPercentage = 2 Then lblM.Text = "Left" 'L = 1 ElseIf e.ProgressPercentage = 10 Then lblM.Text = "" 'L = 0 End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Button Joystick Off click event '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnEndcom_Click(sender As Object, e As EventArgs) Handles btnJoff.Click Me.BackgroundWorker1.CancelAsync() btnJon.Enabled = False btnReset.Enabled = True btnComselect.Enabled = True lbCom.Enabled = True btnJoff.Enabled = False lblM.Text = "" btnVersionInfo.Enabled = True Try sp.Close() 'Close the selected serial port to get out of Do loop in SerialFromArduino Catch End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Button Info click event '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub versionInfo_Click(sender As Object, e As EventArgs) Handles btnVersionInfo.Click MsgBox("Baud Rate: 9600" & vbLf & "Serial Read: single char" & vbLf & "Delimiter: #" & vbLf & "Sampling Rate: 5ms" & vbLf & "Date Modified: 2014/11/4", vbOKOnly) End Sub End Class 240 Appendix 16: eclipse Android App Setup and Code (Java and XML) In conjunction with the code in Appendix 12 and 13, which dictated the data sending protocols, the code below is a first level demonstration of an Android tablet application (Android version 4.1.2 API Level 16) responding to gestures performed (while wearing the joystick). The app GUI is shown in Figure A16.1 and the package explorer tree structure in Figure A16.2. The following was written in eclipse, incorporating fragments, and using the adroidplot library11 to assist with the scrolling plot. Figure A16.1 Android App GUI 11 http://androidplot.com/ 241 Figure A16.2 Package explorer tree structure ------activity_main.xml------ 242 ------button_fragment.xml------ 243 ------logo_fragment.xml------ ------msg_fragment.xml------ ------vis_fragment.xml------ 244 androidPlot.title = "Voltge History" androidPlot.domainLabel = "Time (s)" androidPlot.rangeLabel = "Voltage (V)" androidPlot.titleWidget.labelPaint.textSize="25dp" androidPlot.domainLabelWidget.labelPaint.textSize="13dp" androidPlot.rangeLabelWidget.labelPaint.textSize="13dp" androidPlot.graphWidget.marginTop="25dp" androidPlot.graphWidget.marginLeft="2dp" androidPlot.graphWidget.marginBottom="2dp" androidPlot.graphWidget.marginRight="0dp" androidPlot.graphWidget.rangeLabelPaint.textSize="15dp" androidPlot.graphWidget.rangeOriginLabelPaint.textSize="15dp" androidPlot.graphWidget.domainLabelPaint.textSize="15dp" androidPlot.graphWidget.domainOriginLabelPaint.textSize="15dp" /> /> ------MainActivity.java------package edu.uci.eng.issl.joydemo; /* Sleeve Bluetooth Credentials: * Adafruit EZ-Link 153c * 98:76:B6:00:15:3C */ import android.app.Activity; import android.bluetooth.BluetoothAdapter; 245 import android.bluetooth.BluetoothDevice; import android.content.BroadcastReceiver; import android.content.Context; import android.content.Intent; import android.content.IntentFilter; import android.os.Bundle; import android.os.Handler; import android.os.Message; import android.util.DisplayMetrics; import android.util.Log; import android.view.View; import android.view.WindowManager; import android.graphics.Color; import android.graphics.Point; import com.androidplot.*; import com.androidplot.xy.*; public class MainActivity extends Activity implements btnFragment.onBtOnClickListener, btnFragment.onJoyConnectClickListener, btnFragment.onJoyEnableClickListener, btnFragment.onJoyOffClickListener{ // Bluefruit variables public static final BluetoothAdapter BT = BluetoothAdapter.getDefaultAdapter(); private static int REQUEST_ENABLE_BT=1; private static final String bfMACID="98:76:B6:00:15:3C"; //Bluefruit MAC ID private static BluetoothDevice Sleeve = BT.getRemoteDevice(bfMACID); public BluetoothDevice getBluetoothDevice(){ return Sleeve; } // Generic Strings private static final String TAG = "MainActivity"; private static String btDevice; // Button & Message Strings private static final String btNotOnAlready = "Turn on Bluetooth \n & \n Search for Joystick"; private static final String btOnAlready = "Search for Joystick"; private static final String joyFound = "Bluetooth On \n & \n Joystick Found"; private static final String joyNotFound = "Joystick not found. Make sure it is turned on and in range."; private static final String connectJoy = "Connect to Joystick"; private static final String disconnectedJoy = "Joystick disconnected"; private static final String connectingJoy = "Connecting to Joystick"; private static final String joyConnected = "Joystick connected!"; private static final String joyNotConnected = "Joystick connection failed."; private static final String joyConnectionLost = "Joystick connection lost."; private static final String enableJoy = "Use Joystick"; private static final String joyInUse = "Joystick active"; private static final String joyNotInUse = "Joystick inactive."; private static final String offBT = "Turn off Bluetooth"; private static final String offBTdisconnect = "Disconnect Joystick"; private static final String disableJoy = "Stop using Joystick"; // Dummy variables private int discoveryState; //if app is in BT discovery mode 246 private int dummyJFind; //if joystick was found during discovery private static int appState; //controls off button options depending on app state public Integer getappState(){ return appState; } // Fragment references private static btnFragment bF; private static msgFragment msgStatus; public visFragment vF; // Class References private BluetoothConnection mJoy; // BluetoothConnection Class private MainActivity uiActivity = this; // This class // Plot reference private static final int HISTORY_SIZE = 400; // Number of points to plot before resetting (2s/0.005s) public int getHISTORY_SIZE(){ return HISTORY_SIZE; } private XYPlot voltagePlot = null; public XYPlot getXYPlot(){ return voltagePlot; } private SimpleXYSeries vSeries = null; public SimpleXYSeries getSimpleXYSeries(){ return vSeries; } private LineAndPointFormatter vSeriesFormat = null; public LineAndPointFormatter getLineAndPointFormatter(){ return vSeriesFormat; } //------// App initialization calls @Override protected void onCreate(Bundle savedInstanceState) { super.onCreate(savedInstanceState); Log.i(TAG, "Main Thread onCreateBundle. Main Thread id: " + Long.toString(Thread.currentThread().getId())); // Inflate the main activity layout containing fragments setContentView(R.layout.activity_main); // References to fragments bF=(btnFragment)getFragmentManager().findFragmentById(R.id.buttons_fragment_container); msgStatus=(msgFragment)getFragmentManager().findFragmentById(R.id.msg_fragment_container); vF=(visFragment)getFragmentManager().findFragmentById(R.id.vis_fragment_container); // Register Broadcast Receiver IntentFilter deviceFound = new IntentFilter(BluetoothDevice.ACTION_FOUND); IntentFilter discoveryStart = new IntentFilter(BluetoothAdapter.ACTION_DISCOVERY_STARTED); 247 IntentFilter discoveryEnd = new IntentFilter(BluetoothAdapter.ACTION_DISCOVERY_FINISHED); IntentFilter connectionChange = new IntentFilter(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED); registerReceiver(bReceiver, deviceFound); registerReceiver(bReceiver, discoveryStart); registerReceiver(bReceiver, discoveryEnd); registerReceiver(bReceiver, connectionChange); // Calls to UI setup fcns this.initialButton(); //this.displayInfo(); this.plotSetup(); // Thread initialization (triggers Constructors) mJoy = new BluetoothConnection(uiActivity, uiHandler); } //------public static Handler uiHandler = new Handler(){ @Override public void handleMessage(Message msg){ super.handleMessage(msg); switch(appState){ case 1: Log.i(TAG,"uiHandler case 1"); if(msg.what==1){ //Joystick found //Log.i(TAG,"Joystick found"); bF.mbtOn.setEnabled(false); bF.updateBtntxt(1,joyFound); bF.mjoyConnect.setEnabled(true); msgStatus.updateStatus("Joystick Found: " + btDevice + ". Ready to connect."); bF.mjoyOff.setEnabled(true); bF.updateBtntxt(4, offBT); bF.mjoyOff.setVisibility(View.VISIBLE); }else{ //Joystick not found //Log.i(TAG,"Joystick NOT found"); bF.mbtOn.setEnabled(true); bF.mbtOn.setText(btOnAlready); msgStatus.updateStatus(joyNotFound); } try{ BT.cancelDiscovery(); }catch(RuntimeException e){ e.printStackTrace(); } break; case 2: //Log.i(TAG,"uiHandler case 2"); if(msg.what==1){ //Joystick connected Log.i(TAG,"Joystick connected"); bF.updateBtntxt(2,joyConnected); 248 bF.mjoyConnect.setEnabled(false); msgStatus.updateStatus("Joystick Connected: " + btDevice + ". Ready to use."); bF.mjoyEnable.setEnabled(true); bF.updateBtntxt(4, offBTdisconnect); }else if(msg.what==0){ //Joystick not connected Log.i(TAG,"Joystick not connected"); bF.mjoyConnect.setEnabled(true); msgStatus.updateStatus(joyNotConnected); appState=1; // Failed to connect, send back to appState 1 msg.what=99; // Some value that is not 0 or 1 } break; } } }; //------// Clicking the "Turn on Bluetooth & Search for Joystick" button public void onBtOnConnectClick(){ Log.i(TAG,"onBtOnConnectClick clicked"); bF.mbtOn.setEnabled(false); discoveryState=0; //Initialize discoveryState // Clear plotter and Instanciate SimpleXYSeries voltagePlot.clear(); voltagePlot.redraw(); vSeries = new SimpleXYSeries("vIn"); // Check if Bluetooth is turned on if(!BT.isEnabled()){ //If BT not turned on then prompt the user to do so Intent enableBT_Intent = new Intent(BluetoothAdapter.ACTION_REQUEST_ENABLE); startActivityForResult(enableBT_Intent, REQUEST_ENABLE_BT); } else{ //BT is already on, call "find Joystick" method (asynchronous) appState=1; BT.startDiscovery(); new Thread(discoveryRunnable).start(); //Start new thread to monitor "BT.isDiscovering()" } } //------// Catches the result of the Turn Bluetooth on dialog box @Override protected void onActivityResult(int requestCode, int resultCode, Intent data){ //If activity code matches the one used to generate the BT prompt if(requestCode==REQUEST_ENABLE_BT){ //If the user selected OK/Yes in the BT prompt if(resultCode==RESULT_OK){ appState=1; //Bluetooth on 249 msgStatus.updateStatus("Bluetooth turned on, searching for Joystick..."); BT.startDiscovery(); // Asyc call new Thread(discoveryRunnable).start(); //Start new thread to monitor "BT.isDiscovering()" } else{ //If the user selected No/cancel in the BT prompt appState=0; bF.mbtOn.setEnabled(true); msgStatus.updateStatus("Bluetooth request canceled. Bluetooth still Off."); } } } //------// On separate thread, monitor "BT.isDiscovering()" Runnable discoveryRunnable = new Runnable(){ @Override public void run(){ //Log.i(TAG, "discoveryRunnable Thread id: " + Long.toString(Thread.currentThread().getId())); while(discoveryState==0){} //Just waiting while device is in discovery mode discoveryState=0; //Set dummy variable discoveryState back to initial value Message msg = uiHandler.obtainMessage(); if(dummyJFind==0){ msg.what=0; //Joystick not found }else{ msg.what=1; //Joystick found (dummyJFind = 1) } uiHandler.sendMessage(msg); } }; //------// Clicking the "Connect to Joystick Button" public void onJoyConnectClick(){ Log.i(TAG, "onJoyConnectClick() Thread id: " + Long.toString(Thread.currentThread().getId())); appState=2; bF.mjoyConnect.setEnabled(false); msgStatus.updateStatus(connectingJoy); // Start BluetoothConnection thread new Thread(mJoy).start(); // BluetoothConnection.run() } //------//Clicking the "Enable Joystick" button public void onJoyEnableClick(){ Log.i(TAG,"onJoyEnableClick()"); if(mJoy.mmSocket.isConnected()){ Log.i(TAG,"Socket connected"); appState=3; bF.mjoyEnable.setEnabled(false); bF.updateBtntxt(3,joyInUse); 250 bF.updateBtntxt(4, disableJoy); msgStatus.updateStatus(joyInUse); }else{ Log.i(TAG,"Socket not connected"); appState=2; msgStatus.updateStatus(joyConnectionLost); } } //------//Clicking the "Off" button public void onJoyOffClick(){ if(appState==1){ Log.i(TAG,"OFF in APPSTATE==1"); // Ending calls try{ BT.cancelDiscovery(); // end discovery }catch(RuntimeException e){ e.printStackTrace(); } try{ mJoy.cancel(); // end socket (disconnects) }catch(RuntimeException e){ e.printStackTrace(); } try{ BT.disable(); // turn off bluetooth }catch(RuntimeException e){ e.printStackTrace(); } // Button/message updates bF.mbtOn.setEnabled(true); bF.updateBtntxt(1,btNotOnAlready); bF.mjoyConnect.setEnabled(false); bF.mjoyOff.setVisibility(View.GONE); bF.mjoyOff.setEnabled(false); msgStatus.updateStatus("Bluetooth turned off."); //appState update appState=0; } if(appState==2){ Log.i(TAG,"OFF in APPSTATE==2"); // Ending call try{ mJoy.cancel(); // end socket (disconnects) }catch(RuntimeException e){ e.printStackTrace(); } // Button/message updates bF.updateBtntxt(1, joyFound); //mbtOn bF.mjoyConnect.setEnabled(true); 251 bF.updateBtntxt(2,connectJoy); //mjoyConnect bF.mjoyEnable.setEnabled(false); bF.updateBtntxt(4, offBT); //mjoyOff msgStatus.updateStatus(disconnectedJoy); //appState update appState=1; } if(appState==3){ Log.i(TAG,"OFF in APPSTATE==3"); // Ending call - Don't close socket, just stop data stream // Button/message updates bF.updateBtntxt(3, enableJoy); //mjoyEnable bF.updateBtntxt(4, offBTdisconnect); //mjoyOff bF.mjoyEnable.setEnabled(true); msgStatus.updateStatus(joyNotInUse); // Hide arrow graphics vF.hide_rightArrow(); vF.hide_leftArrow(); //appState update appState=2; } } //------//Broadcast Receiver final BroadcastReceiver bReceiver = new BroadcastReceiver(){ @Override public void onReceive(Context context, Intent intent){ //Set dummy variable: 0-Joystick Not Found | 1-Joystick Found dummyJFind = 0; //When discovery finds any device String action = intent.getAction(); if (BluetoothDevice.ACTION_FOUND.equals(action)){ //Get the BluetoothDevice object from the Intent BluetoothDevice device = intent.getParcelableExtra(BluetoothDevice.EXTRA_DEVICE); //If the MAC address matches that of the Bluefruit if(device.getAddress().equals(bfMACID)){ Log.i(TAG,"Found joystick"); try{ BT.cancelDiscovery(); }catch(RuntimeException e){} btDevice=device.getName(); dummyJFind=1; //Indicate Joystick Found discoveryState=1; //Change discoveryState to 1 to exit while loop in discoveryRunnable thread } } 252 //When startDiscovery is called if(BluetoothAdapter.ACTION_DISCOVERY_STARTED.equals(action)){ Log.i(TAG,"ACTION_DISCOVERY_STARTED"); } //When either cancelDiscovery is called OR startDiscovery times out if(BluetoothAdapter.ACTION_DISCOVERY_FINISHED.equals(action)){ // Use to exit while loop in discoveryThread discoveryState=1; Log.i(TAG,"ACTION_DISCOVERY_FINISHED"); } //When Socket.connect() is called if(BluetoothAdapter.ACTION_CONNECTION_STATE_CHANGED.equals(action)){ String connectionState = intent.getStringExtra(BluetoothAdapter.EXTRA_CONNECTION_STATE); if(connectionState=="STATE_CONNECTING"){ Log.i(TAG,"ACTION_CONNECTION_STATE_CHANGED: " + connectionState); } if(connectionState=="STATE_CONNECTED"){ Log.i(TAG,"ACTION_CONNECTION_STATE_CHANGED: " + connectionState); } if(connectionState=="STATE_DISCONNECTING"){ Log.i(TAG,"ACTION_CONNECTION_STATE_CHANGED: " + connectionState); } if(connectionState=="STATE_DISCONNECTED"){ Log.i(TAG,"ACTION_CONNECTION_STATE_CHANGED: " + connectionState); } } } }; //------@Override protected void onPause(){ super.onPause(); Log.i(TAG,"onPause"); } //------@Override protected void onResume(){ super.onResume(); Log.i(TAG,"onResume"); } //------@Override protected void onDestroy(){ super.onDestroy(); Log.i(TAG,"onDestroy"); if(BT.isEnabled()){ 253 BT.disable(); } try{ BT.cancelDiscovery(); }catch(RuntimeException e){ e.printStackTrace(); } try{ unregisterReceiver(bReceiver); }catch(RuntimeException e){ e.printStackTrace(); } try{ mJoy.cancel(); // end socket (disconnects) }catch(RuntimeException e){ e.printStackTrace(); } } //------// Set initial button states and text private void initialButton(){ if(BT.isEnabled()){ appState=1; bF.updateBtntxt(1,btOnAlready); msgStatus.updateStatus("Bluetooth already on, search for Joystick."); }else{ appState=0; bF.updateBtntxt(1,btNotOnAlready); msgStatus.updateStatus("Bluetooth Off."); } bF.updateBtntxt(2,connectJoy); bF.updateBtntxt(3,enableJoy); bF.updateBtntxt(4,offBT); // Set button initial states bF.mbtOn.setEnabled(true); bF.mjoyConnect.setEnabled(false); bF.mjoyEnable.setEnabled(false); if(appState==1){ //Bluetooth on bF.mjoyOff.setEnabled(true); }else{ //Bluetooth off bF.mjoyOff.setVisibility(View.GONE); bF.mjoyOff.setEnabled(false); } } //------// Call to get screen information in logcat private void displayInfo(){ // Getting display info int Measuredwidth = 0; int Measuredheight = 0; Point size = new Point(); 254 WindowManager w = getWindowManager(); w.getDefaultDisplay().getSize(size); //In pixels Measuredwidth = size.x; Measuredheight = size.y; String sWidth = Integer.toString(Measuredwidth); String sLength = Integer.toString(Measuredheight); Log.i(TAG, "Width: " + sWidth + ". Length: " + sLength); DisplayMetrics metrics = new DisplayMetrics(); getWindowManager().getDefaultDisplay().getMetrics(metrics); float myXdpi = metrics.xdpi; float myYdpi = metrics.ydpi; String sMyXdpi = Float.toString(myXdpi); String sMyYdpi = Float.toString(myYdpi); Log.i(TAG, "xdpi: " + sMyXdpi + ". ydpi: " + sMyYdpi); } //------// Plot Seup Code in conjunction w/ XML spacing declarations private void plotSetup(){ // Get plot reference voltagePlot = (XYPlot) findViewById(R.id.voltagePlot); // Plot line properties vSeriesFormat = new LineAndPointFormatter( Color.RED, // line color Color.TRANSPARENT, // vertex color Color.TRANSPARENT, // fill color null); // PointLabelFormatter plf vSeriesFormat.getLinePaint().setStrokeWidth(2); // General plot colors voltagePlot.setBorderPaint(null); // Gets rid of boarder color voltagePlot.getBackgroundPaint().setColor(Color.TRANSPARENT); // mods color from boarder up to margin voltagePlot.getGraphWidget().getBackgroundPaint().setColor(Color.TRANSPARENT); // mods color inside margins up to plot voltagePlot.getGraphWidget().getGridBackgroundPaint().setColor(Color.TRANSPARENT); // mods plot color voltagePlot.getGraphWidget().getRangeLabelPaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getDomainLabelPaint().setColor(Color.TRANSPARENT); voltagePlot.getGraphWidget().getDomainGridLinePaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getDomainOriginLabelPaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getDomainOriginLinePaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getRangeOriginLabelPaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getRangeOriginLinePaint().setColor(Color.BLACK); // Title properties voltagePlot.getTitleWidget().getLabelPaint().setColor(Color.BLACK); // Domain properties 255 voltagePlot.getDomainLabelWidget().getLabelPaint().setColor(Color.BLACK); voltagePlot.getGraphWidget().getDomainLabelPaint().setColor(Color.TRANSPARENT); voltagePlot.setDomainBoundaries(0, HISTORY_SIZE, BoundaryMode.FIXED); voltagePlot.setDomainStepMode(XYStepMode.INCREMENT_BY_VAL); voltagePlot.setDomainStepValue(20); // Range properties voltagePlot.getRangeLabelWidget().getLabelPaint().setColor(Color.BLACK); voltagePlot.setRangeBoundaries(-0.5, 0.5,BoundaryMode.FIXED); voltagePlot.setRangeStepMode(XYStepMode.INCREMENT_BY_VAL); voltagePlot.setRangeStepValue(0.1); voltagePlot.setTicksPerRangeLabel(1); // Legend properties voltagePlot.getLayoutManager().remove(voltagePlot.getLegendWidget()); //voltagePlot.setMarkupEnabled(true); } //------} ------BluetoothConnection.java------package edu.uci.eng.issl.joydemo; import java.io.IOException; import java.io.InputStream; import java.util.UUID; import com.androidplot.xy.LineAndPointFormatter; import com.androidplot.xy.SimpleXYSeries; import com.androidplot.xy.XYPlot; import android.bluetooth.BluetoothDevice; import android.bluetooth.BluetoothSocket; import android.os.Handler; import android.os.Message; import android.util.Log; public class BluetoothConnection implements Runnable { // Bluetooth connection variables public BluetoothSocket mmSocket; // BT Serial UUID (SPP UUID): 00001101-0000-1000-8000-00805F9B34FB private static final UUID MY_UUID = UUID.fromString("00001101-0000-1000-8000- 00805F9B34FB"); // Bluetooth device reference (by MacID) private BluetoothDevice Sleeve; // Bluetooth read variables private InputStream mmInStream; private final byte delimiter = 35; // ASCII code for # private byte[] packetBytes = new byte[1]; // "byte[] buffer" reading 1 byte at a time from bluetooth stream private Integer x; // Class references private static MainActivity uiActivityClass; 256 private NeuralNet nNet; // UI Handler reference private static Handler parentHandler; // UI thread handler // Plot references private static int plotLength; private static XYPlot myPlot; private SimpleXYSeries myXYSeries; private LineAndPointFormatter myFormatter; private static final int updatePlot = 99; // Input Voltage Conversion Params private static double V = 0; private static double a = -1.65; private static double b = 1.65; private static int mn = 0; private static int mx = 1023; // Gesture Classification Params private static final int GESTURE_RIGHT = 1; private static final int GESTURE_LEFT = 2; // Misc private static final String TAG = "BluetoothConnection"; //------//Parameterized Constructer public BluetoothConnection(MainActivity mainActivityClass, Handler parentHandler){ Log.i(TAG, "BluetoothConnection Constructor Thread id: " + Long.toString(Thread.currentThread().getId())); uiActivityClass=mainActivityClass; BluetoothConnection.parentHandler=parentHandler; Sleeve = uiActivityClass.getBluetoothDevice(); // Instantiate NeuralNet object (runs its constructor) nNet = new NeuralNet(); } //------@Override public void run(){ android.os.Process.setThreadPriority(android.os.Process.THREAD_PRIORITY_BACKGROUND) ; Log.i(TAG, "BluetoothConnection run() Thread id: " + Long.toString(Thread.currentThread().getId())); // Get BluetoothSocket reference BluetoothSocket tmp = null; try{ // Need "Insecure" connection tmp=Sleeve.createInsecureRfcommSocketToServiceRecord(MY_UUID); }catch(IOException e){ Log.i(TAG, "BluetoothSocket catch"); 257 tmp = null; } mmSocket = tmp; // Wrap the following in if(tmp!=null){} // Connect to BluetoothSocket try{ mmSocket.connect(); } catch(IOException e){ Log.i(TAG, "Connect catch"); } Message msg = parentHandler.obtainMessage(); if(mmSocket.isConnected()){ //1=successful connection msg.what=1; parentHandler.sendMessage(msg); // Acquire input stream and assign InputStream tmpIn = null; // Dummy input stream try{ tmpIn=mmSocket.getInputStream(); }catch(IOException e){ e.printStackTrace(); } mmInStream = tmpIn; // Note type is InputStream // Assign plot references plotLength=uiActivityClass.getHISTORY_SIZE(); myPlot=uiActivityClass.getXYPlot(); myXYSeries=uiActivityClass.getSimpleXYSeries(); myFormatter=uiActivityClass.getLineAndPointFormatter(); // Read input stream (bytes) String btOut = ""; // Serial output value as a string String btOut_hold = ""; while(true){ try{ mmInStream.read(packetBytes); // Blocking call that reads 1 byte from the bluetooth stream and puts it in the buffer }catch(IOException e){ Log.i(TAG,"IOException"); break; } // Check if the delimiter value was found (indicating end of single passed value from arduino) if(packetBytes[0] != delimiter){ // If read value not the delimiter append it to string btOut btOut += new String(packetBytes); //, "US-ASCII"); }else if(packetBytes[0] == delimiter && btOut != ""){ // If read value is the delimiter convert to int and pass to NN algorithm 258 try{ x = Integer.parseInt(btOut); // Convert btOut into integer btOut_hold = btOut; btOut = ""; // Reset btOut string // Plotter new Thread(plotUpdateRunnalbe).start(); // NeuralNet Calculation if(uiActivityClass.getappState()==3){ new Thread(nnRunnable).start(); } }catch(NumberFormatException nfe){ Log.i(TAG,"Integer parsing error:" + btOut_hold); break; }catch(RuntimeException e){ Log.i(TAG,"b=delimiter catch"); e.printStackTrace(); break; } } } }else{ //0=unsuccessful connection //Log.i(TAG,"BTConnect what=0"); msg.what=0; parentHandler.sendMessage(msg); } } //------public void cancel(){ Log.i(TAG,"BluetoothConnection cancel()"); try{ mmSocket.close(); }catch(IOException e){} /* try{ myPlot.removeSeries(myXYSeries); }catch(RuntimeException e){} */ } //------// Updating plot series Runnable plotUpdateRunnalbe = new Runnable(){ @Override public void run(){ //Log.i(TAG, "BluetoothConnection plotUpdateRunnalbe run() Thread id: " + Long.toString(Thread.currentThread().getId())); V=((b-a)*(x-mn)/(mx-mn))+a; // Convert passed int x (analog voltage read) to decimal //long startPlot = System.nanoTime(); 259 if(myXYSeries.size()>plotLength){ //myXYSeries.removeFirst(); myPlot.removeSeries(myXYSeries); myXYSeries = new SimpleXYSeries("vIn"); myXYSeries.useImplicitXVals(); myPlot.addSeries(myXYSeries, myFormatter); } myXYSeries.addLast(null, V); Message msg=btcHandler.obtainMessage(); msg.what=updatePlot; btcHandler.sendMessage(msg); /* long stopPlot = System.nanoTime(); long elapsePlot = (stopPlot - startPlot)/1000; Log.i(TAG,"elapsePlot = " + Long.toString(elapsePlot)); */ } }; //------Runnable nnRunnable = new Runnable(){ @Override public void run(){ //Log.i(TAG, "BluetoothConnection nnRunnable run() Thread id: " + Long.toString(Thread.currentThread().getId())); int NN_Out = nNet.nnAlg(x); if(NN_Out==GESTURE_RIGHT){ Log.i(TAG, "run() GESTURE_RIGHT"); Message msg=btcHandler.obtainMessage(); msg.what=GESTURE_RIGHT; btcHandler.sendMessage(msg); } if(NN_Out==GESTURE_LEFT){ Log.i(TAG, "run() GESTURE_LEFT"); Message msg=btcHandler.obtainMessage(); msg.what=GESTURE_LEFT; btcHandler.sendMessage(msg); } } }; //------private static Handler btcHandler = new Handler(){ @Override public void handleMessage(Message msg){ if(msg.what==updatePlot){ parentHandler.post(new Runnable(){ public void run(){ myPlot.redraw(); } }); 260 } if(msg.what==GESTURE_RIGHT){ //Log.i(TAG, "btcHandler GESTURE_RIGHT"); Runnable rRight = new Runnable(){ public void run(){ uiActivityClass.vF.hide_rightArrow(); } }; parentHandler.postDelayed(rRight,500); parentHandler.post(new Runnable(){ public void run(){ uiActivityClass.vF.show_rightArrow(); } }); } if(msg.what==GESTURE_LEFT){ //Log.i(TAG, "btcHandler GESTURE_LEFT"); Runnable rLeft = new Runnable(){ public void run(){ uiActivityClass.vF.hide_leftArrow(); } }; parentHandler.postDelayed(rLeft,500); parentHandler.post(new Runnable(){ public void run(){ uiActivityClass.vF.show_leftArrow(); } }); } } }; //------} ------NeuralNet.java------package edu.uci.eng.issl.joydemo; import android.util.Log; public class NeuralNet{ // Neural Network Size Params private static final int I=133; // Input vector size (5ms sampling) private static final int J=3; // Hidden vector size private static final int K=2; // Output vector size // NN Weight Matrices private static double wA[][] = new double[I+1][J]; // Weight matrix A private static double wB[][] = new double[J+1][K]; // Weight matrix B // NN Input Vector private static double X[][] = new double[I+1][1]; // Work vector (I+1 element = -1) // NN Output Vectors 261 private static double u_j[][] = new double [J][1]; // Hidden node summation to be evaluated private static double Y[][] = new double [J+1][1]; // Hidden node output (J+1 element = - 1) private static double v_k[][] = new double [K][1]; // Output node summation to be evaluated private static double Z[][] = new double [K][1]; // Calculated output vector // Input Voltage Conversion Params private static double V = 0; private static double a = -1.65; private static double b = 1.65; private static int mn = 0; private static int mx = 1023; // Gesture Classification Params private static int NN_Out = 0; // NN output private static final int GESTURE_RIGHT = 1; private static final int GESTURE_LEFT = 2; // Misc private static final String TAG = "NeuralNet"; //------//Param Constructor public NeuralNet(){ Log.i(TAG, "NeuralNet Constructor Thread id: " + Long.toString(Thread.currentThread().getId())); // Initialize input vector X for(int i=0; i // Initialize weight matrix wAlocal double wAlocal[][]= { {-0.035042,0.070964,0.055705}, //Actual Value removed, to be replace by user trained A weights {-0.046207,0.064596,0.054361} }; // Initialize weight matrix wBlocal double wBlocal[][]= { {1.763218,-0.781513}, //Actual Value removed, to be replace by user trained B weights {-1.527316,1.927683} }; // Transfer local arrays to method arrays // (could have defined these in initial declarations // but didn't want to take up the room (non collapseable)) for(int i=0; i<=I; i++){ for(int j=0; j 262 } for(int j=0; j<=J; j++){ for(int k=0; k } //------public int nnAlg(int x){ //long topOutNN = System.nanoTime(); NN_Out = 0; // Reset NN_Out value V=((b-a)*(x-mn)/(mx-mn))+a; // Convert passed int (analog voltage read) to decimal // Update NN input vector X with new V for(int i=0;i<=I-2;i++){ X[i][0]=X[i+1][0]; } X[I-1][0]=V; // Run input vector through NN and get output vector Z for(int j=0; j<=J-1; j++){ u_j[j][0]=0; for(int i=0; i<=I; i++){ u_j[j][0]=u_j[j][0]+X[i][0]*wA[i][j]; } // exit i loop Y[j][0]=1/(1+Math.exp(-u_j[j][0])); } // exit j loop Y[J][0]=-1; // Actually J+1'th element. Set to -1 as bias for(int k=0; k<=K-1; k++){ v_k[k][0]=0; for(int j=0; j<=J;j++){ v_k[k][0]=v_k[k][0]+Y[j][0]*wB[j][k]; } // exit j loop Z[k][0]=1/(1+Math.exp(-v_k[k][0])); } // exit k loop // Determine if output corresponds to a gesture and if so reset input vector // Right if(Z[0][0]<=0.95 && Z[0][0]>=0.85 && Z[1][0]<=0.15 && Z[1][0]>=0.05){ NN_Out=GESTURE_RIGHT; for(int i=0; i<=I-1; i++){ X[i][0]=0; } X[I][0]=-1; } // Left if(Z[1][0]<=0.95 && Z[1][0]>=0.85 && Z[0][0]<=0.15 && Z[0][0]>=0.05){ NN_Out=GESTURE_LEFT; for(int i=0; i<=I-1; i++){ X[i][0]=0; } 263 X[I][0]=-1; } //long bottomOutNN = System.nanoTime(); //long elapseOutNN = (bottomOutNN - topOutNN)/1000; //Log.i(TAG,"FCNelapseOutNN = " + Long.toString(elapseOutNN)); return NN_Out; } //------ } ------btnFragment.java------package edu.uci.eng.issl.joydemo; import android.app.Activity; import android.app.Fragment; import android.os.Bundle; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; import android.widget.Button; public class btnFragment extends Fragment { //private static final String TAG = "btnFragment"; //Declare callbacks onBtOnClickListener mBtONcallBack; onJoyConnectClickListener mJCcallBack; onJoyEnableClickListener mJEcallBack; onJoyOffClickListener mJOFFcallBack; //Declare buttons public Button mbtOn; public Button mjoyConnect; public Button mjoyEnable; public Button mjoyOff; //Declare interfaces public interface onBtOnClickListener{ // Called when BT on button is clicked public void onBtOnConnectClick(); } public interface onJoyConnectClickListener{ // Called when joyConnect button is clicked public void onJoyConnectClick(); } public interface onJoyEnableClickListener{ //Callec when joyEnable button clicked public void onJoyEnableClick(); } public interface onJoyOffClickListener{ // Called when off button is clicked public void onJoyOffClick(); } 264 //------@Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState){ View v = inflater.inflate(R.layout.button_fragment, container, false); mbtOn = (Button)v.findViewById(R.id.btEnable); mbtOn.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { // When button is clicked, this calls the method // onBtOnConnectClick in the activity that the // fragment is attached to (via mBtOnCallBack // which is tied to the interface defining said // method. mBtONcallBack.onBtOnConnectClick(); } }); mjoyConnect = (Button)v.findViewById(R.id.joyConnect); mjoyConnect.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { mJCcallBack.onJoyConnectClick(); } }); mjoyEnable = (Button)v.findViewById(R.id.joyEnable); mjoyEnable.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { mJEcallBack.onJoyEnableClick(); } }); mjoyOff = (Button)v.findViewById(R.id.joyOff); mjoyOff.setOnClickListener(new View.OnClickListener() { @Override public void onClick(View v) { mJOFFcallBack.onJoyOffClick(); } }); return v; } //------@Override public void onAttach(Activity activity){ super.onAttach(activity); try{ // Associates CallBack variable with Interface in // the activity that the fragment is attached to. mBtONcallBack=(onBtOnClickListener)activity; mJCcallBack=(onJoyConnectClickListener)activity; 265 mJEcallBack=(onJoyEnableClickListener)activity; mJOFFcallBack=(onJoyOffClickListener)activity; }catch (ClassCastException e){ throw new ClassCastException(activity.toString() + "must implement Listener"); } } //------public void updateBtntxt(int i, String str){ if(i==1) mbtOn.setText(str); if(i==2) mjoyConnect.setText(str); if(i==3) mjoyEnable.setText(str); if(i==4) mjoyOff.setText(str); } //------} ------logoFragment.java------package edu.uci.eng.issl.joydemo; import android.app.Fragment; import android.os.Bundle; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; //import android.widget.ImageView; public class logoFragment extends Fragment { //private ImageView logoISSL; @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState){ View v = inflater.inflate(R.layout.logo_fragment, container, false); return v; } } ------msgFragment.java------package edu.uci.eng.issl.joydemo; import android.app.Fragment; import android.os.Bundle; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; import android.widget.TextView; public class msgFragment extends Fragment { //private static final String TAG = "msgFragment"; private TextView mjoyStatus; @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState){ 266 View v = inflater.inflate(R.layout.msg_fragment, container, false); mjoyStatus=(TextView)v.findViewById(R.id.msg_status); mjoyStatus.setText(null); return v; } public void updateStatus(String str){ mjoyStatus.setText(str); } } ------visFragment.java------package edu.uci.eng.issl.joydemo; import android.app.Fragment; import android.os.Bundle; import android.view.LayoutInflater; import android.view.View; import android.view.ViewGroup; import android.widget.ImageView; public class visFragment extends Fragment { private ImageView leftArrow; private ImageView rightArrow; @Override public View onCreateView(LayoutInflater inflater, ViewGroup container, Bundle savedInstanceState){ View v = inflater.inflate(R.layout.vis_fragment, container, false); leftArrow=(ImageView)v.findViewById(R.id.arrowLeft); rightArrow=(ImageView)v.findViewById(R.id.arrowRight); leftArrow.setVisibility(4); rightArrow.setVisibility(4); return v; } public void show_leftArrow(){ leftArrow.setVisibility(0); } public void show_rightArrow(){ rightArrow.setVisibility(0); } public void hide_leftArrow(){ leftArrow.setVisibility(4); } public void hide_rightArrow(){ rightArrow.setVisibility(4); } } 267 Appendix 17: Visual Studio PowerPoint Add‐In Code In conjunction with the code in Appendix 12 and 13, which dictated the data sending protocols, the code below is a demonstration of a PC program, specifically Microsoft PowerPoint, responding to gestures performed (while wearing the joystick). The following was written in .NET 4.5 using Visual Studios as a PowerPoint Add‐In, as shown in Figure A17.1, not a stand‐ alone form. The accompanying solution explorer tree structure is shown in Figure A17.2. Figure A17.1 Microsoft PowerPoint Add‐In in ribbon bar Figure A17.2 Solution Explorer 268 Imports Microsoft.Office.Tools.Ribbon Public Class joystickRibbon '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Class variables '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Object declarations Public btForm As btSelect ' Windows form object '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Ribbon loadimage event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub joystickRibbon_LoadImage(ByVal sender As Object, ByVal e As RibbonLoadImageEventArgs) _ Handles MyBase.LoadImage Select Case e.ImageName Case "imgPair" e.Image = My.Resources.Bt_BuxtonSketch_crop Case "imgUse" e.Image = My.Resources.Start_Viner_HandITC_crop Case "imgStop" e.Image = My.Resources.X_Viner_Hand_ITC_crop End Select End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Pair controller button click event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnPair_Click(sender As Object, e As RibbonControlEventArgs) Handles btnPair.Click Try ' If already paired give message If Globals.ThisAddIn.pairedState = 1 Then Dim msg As String = "The controller and computer are already connected (paired). " _ & "If you want to change COM ports turn the controller off first " _ & "and then re-pair the devices." MsgBox(msg, vbOKOnly, "Controller Status") Else btForm = New btSelect() btForm.ShowDialog() 'Equivalent of "Modal" End If Catch exNullReff As NullReferenceException MsgBox("NullRefException") Catch exNull As ArgumentNullException MsgBox("ArgNullException") Catch ex As ArgumentException MsgBox("ArgException") Catch ex As Exception MsgBox("Exception") End Try End Sub 269 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Use controller event handler '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnUse_Click(sender As Object, e As RibbonControlEventArgs) Handles btnUse.Click Try ' If devices not paired or serial port closed (post pairing) If Globals.ThisAddIn.pairedState = 0 Or Globals.ThisAddIn.mySP.IsOpen = False Then Dim msg As String = "Controller and computer must be connected in order to use controller." MsgBox(msg, vbOKOnly, "Controller Status") ElseIf Globals.ThisAddIn.controllerState = 1 Then ' If controller already activated Dim msg As String = "Controller already active. Start presentation to use." MsgBox(msg, vbOKOnly, "Controller Status") Else Globals.ThisAddIn.controllerState = 1 Globals.Ribbons.joystickRibbon.btnUse.Enabled = False Globals.Ribbons.joystickRibbon.lblActiveStatus.Label = "Active" Dim msg As String = "Controller activated. Start presentation to use." MsgBox(msg, vbOKOnly, "Controller Status") End If Catch exNullReff As NullReferenceException MsgBox("NullRefException") Catch exNull As ArgumentNullException MsgBox("ArgNullException") Catch ex As ArgumentException MsgBox("ArgException") Catch ex As Exception MsgBox("Exception") End Try End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Stop using controller event handler (will need to re-pair to use again) '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnStop_Click(sender As Object, e As RibbonControlEventArgs) Handles btnStop.Click Globals.Ribbons.joystickRibbon.btnStop.Enabled = False Globals.Ribbons.joystickRibbon.btnPair.Enabled = True Globals.Ribbons.joystickRibbon.btnUse.Enabled = True Globals.Ribbons.joystickRibbon.lblPairedStatus.Label = "Unpaired" Globals.Ribbons.joystickRibbon.lblActiveStatus.Label = "Inactive" Globals.ThisAddIn.pairedState = 0 Globals.ThisAddIn.controllerState = 0 Try Globals.ThisAddIn.mySP.Close() Catch exNullReff As NullReferenceException MsgBox("NullRefException") Catch exNull As ArgumentNullException MsgBox("ArgNullException") Catch ex As ArgumentException MsgBox("ArgException") Catch ex As Exception MsgBox("Exception") 270 End Try End Sub End Class ------CLASS SEPARATION------Imports System.IO.Ports Imports System.Diagnostics Public Class ThisAddIn ' Object Declarations Public mySP As New SerialPort ' Defined in btSelect Dim NN As NeuralNet Dim ppt As PowerPoint.Presentation ' Serialport connection parameters Public Baud As Integer = 9600 'Options: 9600 or 115200 (must match Arduino) Public Port As String = "" ' Serialport read variables using # delimiter Dim chRead As Char ' Read char from serial stream Dim chrBuffer As String = "" ' String concatenating all the read chars (which are converted to string values) Dim delimCount As Integer = 0 ' Ignoring first read value b/c could be partial Dim delim As Char = Chr(35) ' Chr(35) is ASCII code for "#" Dim analogV As String = "" Dim delimStr As String = "#" ' Flow control variables Public pairedState As Integer = 0 Public controllerState As Integer = 0 Public presentationState As Integer = 0 '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' ThisAddIn Startup Event Listener '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub ThisAddIn_Startup() Handles Me.Startup ' Initial Ribbon Configuration: joystickRibbon.Designer.vb End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' SlideShow Begin Event Listener '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub App_SlideShowBegin(Wn As PowerPoint.SlideShowWindow) Handles Application.SlideShowBegin ppt = Application.ActivePresentation mySP.DiscardInBuffer() presentationState = 1 End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Connect to and read by Char serial port (New task started in btSelect.vb) '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Sub readCharSP() 'Port As String) With mySP 271 .BaudRate = Baud .PortName = Port .ReadTimeout = 5000 .RtsEnable = True End With Try mySP.Open() If mySP.IsOpen Then pairedState = 1 With Globals.Ribbons.joystickRibbon .lblPairedStatus.Label = "Paired" .btnPair.Enabled = False .btnUse.Enabled = True .btnStop.Enabled = True End With Dim btConnectForm = New btConnected() btConnectForm.ShowDialog() 'Equivalent of "Modal" NN = New NeuralNet() Else pairedState = 0 End If Catch ex As Exception MsgBox("Error: Could not connect to selected COM port. Make sure device is connected and powered on.", _ vbOKOnly, "Connection Error") Globals.Ribbons.joystickRibbon.btnPair.Enabled = True Globals.Ribbons.joystickRibbon.lblPairedStatus.Label = "Unpaired" Globals.Ribbons.joystickRibbon.btnStop.Enabled = False Exit Sub End Try While (mySP.IsOpen) If controllerState = 1 AndAlso presentationState = 1 Then ' Checking if SlideShow running (if not throws error) Try If Not ppt.SlideShowWindow Is Nothing Then presentationState = 1 Else presentationState = 0 End If Catch ex As Exception presentationState = 0 pairedState = 1 controllerState = 1 With Globals.Ribbons.joystickRibbon .lblPairedStatus.Label = "Paired" .lblActiveStatus.Label = "Active" .btnPair.Enabled = False .btnUse.Enabled = False .btnStop.Enabled = True End With End Try 272 Try Dim b As Integer = mySP.BytesToRead If b > 1 AndAlso controllerState = 1 AndAlso presentationState = 1 Then chRead = Chr(mySP.ReadChar()) If chRead <> delim Then chrBuffer = chrBuffer + Convert.ToString(chRead) 'MsgBox("chrBuffer = " & chrBuffer) Else ' Delimiter reached (full reading gathered) analogV = chrBuffer chrBuffer = "" If delimCount = 0 Then delimCount = 1 Else Dim gesture As Integer = NN.NeuralNetCalc(analogV) ' Call to NeuralNet function Select Case gesture Case 1 ' Right Try My.Computer.Keyboard.SendKeys("{RIGHT}", True) 'ppt.SlideShowWindow.View.Next() ‘also works Catch ex As Exception MsgBox("Catch Right/Next") Exit Sub End Try Case 2 'Left Try My.Computer.Keyboard.SendKeys("{LEFT}", True) 'ppt.SlideShowWindow.View.Previous() ‘also works Catch ex As Exception MsgBox("Catch Left/Previous") Exit Sub End Try Case Else End Select End If End If Else System.Threading.Thread.Sleep(1) ' Sleep x milliseconds End If Catch ex As ArgumentException MsgBox("ArgException") Exit Sub Catch exInvalidOpp As InvalidOperationException 'ReadChar() MsgBox("Invalid Operation Catch") Exit Sub Catch toEx As TimeoutException 'ReadChar() Dim msg As String = "Controller connection lost. Most likely " _ & " controller out of range or has been turned off (check battery)" _ & vbLf & vbLf _ & "Must re-connect controller for continued use." MsgBox(msg, vbOKOnly, "Controller TimeoutException") Exit Sub Catch ex As Exception 273 MsgBox("ex As Exception") Exit Sub End Try End If End While End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Connect to and read serial port (New task started in btSelect.vb) '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Sub readToSP() 'Port As String) With mySP .BaudRate = Baud .PortName = Port .ReadTimeout = 5000 .RtsEnable = True End With Try mySP.Open() If mySP.IsOpen Then pairedState = 1 With Globals.Ribbons.joystickRibbon .lblPairedStatus.Label = "Paired" .btnPair.Enabled = False .btnUse.Enabled = True .btnStop.Enabled = True End With Dim btConnectForm = New btConnected() btConnectForm.ShowDialog() 'Equivalent of "Modal" NN = New NeuralNet() Else pairedState = 0 End If Catch ex As Exception MsgBox("Error: Could not connect to selected COM port. Make sure device is connected and powered on.", _ vbOKOnly, "Connection Error") Globals.Ribbons.joystickRibbon.btnPair.Enabled = True Globals.Ribbons.joystickRibbon.lblPairedStatus.Label = "Unpaired" Globals.Ribbons.joystickRibbon.btnStop.Enabled = False Exit Sub End Try While (mySP.IsOpen) If controllerState = 1 AndAlso presentationState = 1 Then ' Checking if SlideShow running (if not throws error) Try If Not ppt.SlideShowWindow Is Nothing Then presentationState = 1 Else presentationState = 0 End If 274 Catch ex As Exception presentationState = 0 pairedState = 1 controllerState = 1 With Globals.Ribbons.joystickRibbon .lblPairedStatus.Label = "Paired" .lblActiveStatus.Label = "Active" .btnPair.Enabled = False .btnUse.Enabled = False .btnStop.Enabled = True End With End Try Try Dim b As Integer = mySP.BytesToRead If b > 1 AndAlso controllerState = 1 AndAlso presentationState = 1 Then Dim analogV As String = mySP.ReadTo(delimStr) If delimCount = 0 Then delimCount = 1 Else Dim gesture As Integer = NN.NeuralNetCalc(analogV) ' Call to NeuralNet function Select Case gesture Case 1 ' Right Try My.Computer.Keyboard.SendKeys("{RIGHT}", True) 'ppt.SlideShowWindow.View.Next() Catch ex As Exception MsgBox("Catch Right/Next") Exit Sub End Try Case 2 'Left Try My.Computer.Keyboard.SendKeys("{LEFT}", True) 'ppt.SlideShowWindow.View.Previous() Catch ex As Exception MsgBox("Catch Left/Previous") Exit Sub End Try Case Else End Select End If End If Catch ex As ArgumentException MsgBox("ArgException") Exit Sub Catch exInvalidOpp As InvalidOperationException 'ReadChar() MsgBox("Invalid Operation Catch") Exit Sub Catch toEx As TimeoutException 'ReadChar() Dim msg As String = "Controller connection lost. Most likely " _ & " controller out of range or has been turned off (check battery)" _ & vbLf & vbLf _ & "Must re-connect controller for continued use." 275 MsgBox(msg, vbOKOnly, "Controller TimeoutException") Exit Sub Catch ex As Exception MsgBox("ex As Exception") Exit Sub End Try End If End While End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' ThisAddIn Shutdown Event Listener '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub ThisAddIn_Shutdown() Handles Me.Shutdown Try mySP.Close() Catch ex As Exception End Try End Sub End Class ------CLASS SEPARATION------Imports System.Diagnostics Public Class NeuralNet Public Incoming As String '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' NeuralNet Class variables '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' NN Sizes Dim I As Integer = 133 ' Input vector length not including bias Dim J As Integer = 3 ' Number of hidden nodes Dim K As Integer = 2 ' Output vector length (Right or Left) ' NN Weight Matrices Dim wA(I + 1, J) As Double ' Weight Array A Dim wB(J + 1, K) As Double ' Weight Array B Dim X(I + 1) As Double ' Work vector '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Class Initialization '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Sub New() '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Load weight matrices and initial work vector X into memory '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dim wAdummy(,) As Double = New Double(,) { {-0.035042, 0.070964, 0.055705}, ' Actual Value removed, to be replace by user trained A weights 276 {-0.046207, 0.064596, 0.054361} } Dim wBdummy(,) As Double = New Double(,) { {1.763218, -0.781513}, ' Actual Value removed, to be replace by user trained B weights {-1.527316, 1.927683} } Dim a, b, c, m As Integer For a = 0 To I For b = 0 To J - 1 wA(a, b) = wAdummy(a, b) Next Next For b = 0 To J For c = 0 To K - 1 wB(b, c) = wBdummy(b, c) Next Next For m = 0 To I - 1 X(m) = 0 Next X(I) = -1 End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Gesture Calculation Function '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Function NeuralNetCalc(ByVal analogV As String) 'sWatch.Start() '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Local variables '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' ADC to Voltage conversion variables Dim aR As Integer = 0 Dim a As Double = -1.65 Dim b As Double = 1.65 Dim mn As Integer = 0 Dim mx As Integer = 1023 Dim ardV As Double 'Counter variables Dim m As Integer Dim n As Integer Dim o As Integer ' Working vectors Dim u(J) As Double ' holds product of input node and corresponding wA value Dim v(K) As Double ' holds product of hidden node and corresponding wB value 277 ' Node outputs Dim Y(J + 1) As Double ' Hidden node output value Dim Z(K) As Double ' Output note output vector Dim NN_Out As Integer = 0 ' Default value '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Convert read analog voltage [0-1023] to ardV '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Try aR = Convert.ToInt32(analogV) Catch ex As Exception MsgBox("aR = 511") aR = 511 End Try If aR >= 0 And aR <= 1023 Then ardV = ((b - a) * (aR - mn) / (mx - mn)) + a Else ardV = 0 End If '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Update work vector X with new value V going to position I-1 w/ position I value =-1 ALWAYS '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For m = 0 To I - 2 X(m) = X(m + 1) Next X(I - 1) = ardV '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Neural network output vector calculations '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ For n = 0 To J - 1 u(n) = 0 For m = 0 To I u(n) = u(n) + X(m) * wA(m, n) Next Y(n) = 1 / (1 + Math.Exp(-u(n))) Next Y(J) = -1 For o = 0 To K - 1 v(o) = 0 For n = 0 To J v(o) = v(o) + Y(n) * wB(n, o) Next Z(o) = 1 / (1 + Math.Exp(-v(o))) Next '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Classify output vector Z '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ If Z(0) <= 0.95 AndAlso Z(0) >= 0.85 AndAlso Z(1) <= 0.15 AndAlso Z(1) >= 0.05 Then 'Right Motion NN_Out = 1 For m = 0 To I - 1 278 X(m) = 0 Next X(I) = -1 ElseIf Z(1) <= 0.95 AndAlso Z(1) >= 0.85 AndAlso Z(0) <= 0.15 AndAlso Z(0) >= 0.05 Then 'Left Motion NN_Out = 2 For m = 0 To I - 1 X(m) = 0 Next X(I) = -1 Else End If Return NN_Out End Function End Class ------CLASS SEPARATION------ Figure A17.3 Class btSelect GUI Option Explicit On Imports System.Threading.Tasks '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Class with btSelect Fomr for selecting COM Port and starting serial task '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 279 Public Class btSelect '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Form Initialization '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btSelect_Load(sender As Object, e As EventArgs) Handles MyBase.Load Me.Enabled = True lbCom.Items.Clear() lbCom.Enabled = True lbCom.TabStop = False btnRefresh.Enabled = True btnBtPair.Enabled = False btnCancel.Enabled = True btnInfo.Enabled = True ' Populate COM listbox on Form startup For Each sp As String In My.Computer.Ports.SerialPortNames lbCom.Items.Add(sp) Next lbCom.Sorted = True 'Sorts listbox lbCom alphabetically after populated End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Refresh the COM Listbox '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnRefresh_Click(sender As Object, e As EventArgs) Handles btnRefresh.Click btnBtPair.Enabled = False lbCom.Items.Clear() For Each sp As String In My.Computer.Ports.SerialPortNames lbCom.Items.Add(sp) Next lbCom.Sorted = True 'Sorts listbox lbCom alphabetically after populated Dim msgNoCOM As String msgNoCOM = "No Bluetooth connections were found. Make sure computer's Bluetooth is turned on and, if necessary, dongle is inserted" If lbCom.Items.Count = 0 Then MsgBox(msgNoCOM, vbOKOnly, "No Bluetooth Connections Found") End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Selecting a com port from the list box enables the Select COM button '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub comSel(ByVal sender As Object, ByVal e As EventArgs) Handles lbCom.SelectedIndexChanged Dim index As Integer = lbCom.SelectedIndex If index <> -1 Then 280 btnBtPair.Enabled = True End If End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ' Connect to Serial Port through Task spTask (located in ThisAddIn.vb) '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Public Sub btnBtPair_Click(sender As Object, e As EventArgs) Handles btnBtPair.Click With Globals.Ribbons.joystickRibbon .lblPairedStatus.Label = "Pairing" .btnPair.Enabled = False .btnUse.Enabled = False .btnStop.Enabled = False End With Globals.ThisAddIn.Port = lbCom.SelectedItem ' Set the COM Port value in ThisAddIn Dim spTask = New Task(Sub() Globals.ThisAddIn.readToSP()) ' Define new task spTask (sub located in ThisAddIn) spTask.Start() ' Start new task spTask Me.Close() End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Clicking the Info Button '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnInfo_Click(sender As Object, e As EventArgs) Handles btnInfo.Click Dim msg As String = _ "This Addin connects (pairs) the wearable controller to your computer for use during PowerPoint presentation mode." _ & vbLf & vbLf & _ "Select the COM port from the list of available COM ports that is associated with your computer's bluetooth " _ & "connection and click 'Connect Controller' to pair the computer and controller." _ & vbLf & vbLf & _ "Click 'Use Controller' to activate the controller for presentation narration." _ & vbLf & vbLf & _ "Connectin Details:" & vbLf & "Baud Rate: 9600" & vbLf & "Serial Read: single char" _ & vbLf & "Delimiter: #" & vbLf & "Sampling Rate: 5ms" & vbLf & "Date Modified: 2014/11/4" Dim msgTitle As String = "Wearable Controller Addin Details" MsgBox(msg, vbOKOnly, msgTitle) End Sub '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 'Clicking the Cancel Button '~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Private Sub btnCancel_Click(sender As Object, e As EventArgs) Handles btnCancel.Click Me.Close() End Sub End Class 281 ------CLASS SEPARATION------ Figure A17.4 Class btConnected GUI Public Class btConnected Private Sub btConnected_Load(sender As Object, e As EventArgs) Handles MyBase.Load Dim xForm As Integer = Me.Size.Width Dim xBtn As Integer = btnOK.Size.Width Dim xLbl As Integer = lblMsg.Size.Width Dim x As Integer x = (xForm - xLbl) / 2 lblMsg.Left = x x = (xForm - xBtn) / 2 btnOK.Left = x End Sub Private Sub btnOK_Click(sender As Object, e As EventArgs) Handles btnOK.Click Me.Hide() Me.Close() End Sub End Class 282 Appendix 18: Quadcopter Parts List Table A18.1 Parts list for quadcopter. Part Name Vendor Part Number Quantity XBee 1mW Trace Antenna Series 1 Sparkfun 11215 1 Arduino Uno –R3 Sparkfun 11021 1 XBee shield Sparkfun 12847 1 DJI Naza‐M Lite Multirotor Controller heli‐world DJINZMLT 1 Frame HobbyKing 9171000022 1 Electronic Speed Controllers HobbyWing FLYFUN 30A 4 Motors Multirotor SunnySky X2208 4 Superstore 1500kv Battery HobbyKing Z30003S‐20 1 Propellers (pair) robotshop RB‐Gem‐10 2 XT60 to 6x3.5mm bullet ESC Power HobbyKing 015000096 1 Breakout Cable XT60 Connectors (5 pairs) HobbyKing XT60 1 Double Sided Foam HobbyKing ST‐FOAM‐DB 1 Gold Plated Spring “Bullet” Connector HobbyKing AM‐1001C 2 3.5mm (10pair/20pc) Remote and transmitter12 rc‐fans88 T6EHP 1 Additionally, four ESC mounts were designed, 3d, Figure A18.1, printed, and glued to the arms of the frame. 12 Used to gather signal information, not used in operation 283 Figure A18.1 ESC mounts. Dimensions given in mm and extruded length is 55mm. 284 Appendix 19: Glove based ATmega328p code (quadTX) In conjunction with the quadRX code shown in Appendix 20 this code, written using AtmelStudio for a ATmega328p microcontroller, is loaded onto the microcontroller housed on the glove containing the sensor. It uses a look up table uint8_t fullFlex (values omitted) based on read ADC value to generate joint angle based on equation (7.1). This joint angle is then used to calculate uint8_t ppmWidth which is sent to the receiving XBee/Arduio pair. /* * quadTX.c * Updated: 2015/3/17 * Div 8 Removed in Fuse Bits */ #ifndef F_CPU #define F_CPU 8000000UL // in conjunction with CLKPR #endif #ifndef BAUD #define BAUD 9600UL // set BAUD rate #endif #include #define MYUBRR 51 // Table 20-1 #define ADC0 PCINT8 // PC0 - PIN 23 #define CAPS1 PCINT12 // PC4 - PIN 27 #define CAPS2 PCINT13 // PC5 - PIN 28 /* ---- Global Variables ---- */ volatile uint8_t sticksZero = 100; //microseconds (divided by 10) volatile uint8_t sticksNeutral = 152; //microseconds (divided by 10) volatile uint8_t sticksFull = 200; //microseconds (divided by 10) volatile uint8_t j; // counter for timer overflow volatile uint8_t j_comp = 100; // Compare value for j (10/20, 50/100) volatile uint8_t k; // counter to recalibrate touchSense limits volatile uint8_t ctrlState; // With touchIndex controls state of quad (neutral/(dis)arm/fly) volatile uint16_t minADC; // minimum ADC reading for logistic curve (based on trimmer setting) volatile uint8_t jointAngle; // joint angle either from min/max criteria or from LUT (direction based) volatile uint16_t val; volatile uint8_t touch = 30; // value added to untouched sensor ADC value to detect if sensor touched or not volatile uint16_t touchMiddle; // ADC cutoff value to detect touch of middle finger glove patch 285 volatile uint16_t touchRing; // ADC cutoff value to detect touch of ring finger glove patch // Lookup Tables: Spans ADC of 0 to 1000 const uint8_t fullFlex[1001][1] PROGMEM= { {7}, … {66} }; /* ---- Function Definitions ---- */ // Initialize USART void USART_Init(unsigned int ubrr){ UBRR0H = (unsigned char)(ubrr>>8); // Setting baud rate UBRR0L = (unsigned char)ubrr; UCSR0B = (1< // Read and Send ADC Value void read_send_ADC(uint8_t adcPin){ ADMUX = (0b11110000 & ADMUX)|adcPin; // set ADC channel ADCSRA |=(1< //USART_Transmit(ADC); uint32_t ppmWidth = (((long)ADC*(sticksFull-sticksZero+1))>>10) + sticksZero; //uint16_t simADC = 1023; //uint32_t ppmWidth = ((((long)simADC*(sticksFull-sticksZero+1))>>10) + sticksZero)/10; //div10 of controller PPM Width USART_Transmit((uint8_t)ppmWidth); } // Read, Average, and Return ADC Value (average of 4 readings) uint16_t read_average_ADC(uint8_t adcPin){ uint8_t i = 0; uint16_t retval = 0; for(i=0;i<4;i++){ ADMUX = (0b11110000 & ADMUX)|adcPin; // set ADC channel ADCSRA |=(1< 286 } retval = retval>>2; // >>2 same as div 4 return(retval); } // Initialize 8-bit Timer0 void initTimer0(void){ TCCR0B |= (1< // ~~~~~~~~~~ Check Touch Sensors // (Re)Initialize Touch Sensor Compare Values void cal_touchSense1(void){ uint8_t i=0; uint16_t retval=0; for(i=0;i<4;i++){ ADMUX |= 0b00001111; // Set MUX to ground PORTC |= (1 << CAPS1); // Enable pull-up resistor on CAPS1 pin _delay_us(100); // charing touch sensor (play with time) (100) PORTC &= ~(1 << CAPS1); // Disable pull-up resistor on input pin ADCSRA |=(1< /* USART_Transmit((touchMiddle-touch)>>8); // 8 Most sig bits (highest 2 bits of 10 bit ADC) USART_Transmit(touchMiddle-touch); // 8 Least sig bits USART_Transmit(0); USART_Transmit(0); */ } void cal_touchSense2(void){ uint8_t i=0; uint16_t retval=0; for(i=0;i<4;i++){ ADMUX |= 0b00001111; // Set MUX to ground PORTC |= (1 << CAPS1); // Enable pull-up resistor on CAPS1 pin _delay_us(100); // charing touch sensor (play with time) (100) PORTC &= ~(1 << CAPS1); // Disable pull-up resistor on input pin ADCSRA |=(1< 287 } touchRing =(retval>>2)+touch; // re-establish ADC compare value for sensor being touched /* USART_Transmit((touchRing-touch)>>8); // 8 Most sig bits (highest 2 bits of 10 bit ADC) USART_Transmit(touchRing-touch); // 8 Least sig bits USART_Transmit(0); USART_Transmit(0); */ } // Read Touch Sensor Index Finger (Start & Stop Motors) // http://tuomasnylund.fi/drupal6/content/capacitive-touch-sensing-avr-and-single-adc-pin void touchSense1(void){ uint8_t i=0; uint16_t retval=0; for(i=0;i<4;i++){ ADMUX |= 0b00001111; // Set MUX to ground PORTC |= (1 << CAPS1); // Enable pull-up resistor on CAPS1 pin _delay_us(100); // charing touch sensor (play with time) (100) PORTC &= ~(1 << CAPS1); // Disable pull-up resistor on input pin ADCSRA |=(1< if(retval>=touchMiddle){ ctrlState = 1; } /* USART_Transmit(retval>>8); // 8 Most sig bits (highest 2 bits of 10 bit ADC) USART_Transmit(retval); // 8 Least sig bits USART_Transmit(0); USART_Transmit(0); */ } // Read Touch Sensor Middle Finger (Set Quad to Flight Mode) void touchSense2(void){ uint8_t i=0; uint16_t retval=0; for(i=0;i<4;i++){ ADMUX |= 0b00001111; // Set MUX to ground PORTC |= (1 << CAPS2); // Enable pull-up resistor on CAPS2 pin _delay_us(100); // charing touch sensor (play with time) (100) PORTC &= ~(1 << CAPS2); // Disable pull-up resistor on input pin ADCSRA |=(1< 288 loop_until_bit_is_clear(ADCSRA, ADSC); // Wait for conversion to end ADMUX = (0b11110000 & ADMUX)|CAPS2; //0b00000101; // Set MUX to ADC5 //_delay_us(10); ADCSRA |=(1< if(retval>=touchRing){ ctrlState = 2; // Set minADC value as average sensor ADC reading when middle finger touch sensor is being pressed minADC = read_average_ADC(ADC0); } /* USART_Transmit(retval>>8); // 8 Most sig bits (highest 2 bits of 10 bit ADC) USART_Transmit(retval); // 8 Least sig bits USART_Transmit(0); USART_Transmit(0); */ } // Timer0 Overflow Interrupt ISR(TIMER0_OVF_vect){ // 20 --> roughly every 5.12 ms // 100 --> roughly every 25.60 ms **Needed for .NET plotting application** if(j==j_comp){ //read_send_ADC(ADC0); k++; j=0; touchSense1(); touchSense2(); switch(ctrlState){ case 0: // Startup // Send Aileron value (Roll) USART_Transmit(sticksNeutral); USART_Transmit(0); // Send Elevator value (Pitch) USART_Transmit(sticksNeutral); USART_Transmit(1); // Send Throttle value (ascend/descend) USART_Transmit(sticksNeutral); USART_Transmit(2); // Send Rudder value (Yaw) USART_Transmit(sticksNeutral); USART_Transmit(3); break; case 1: // Arm/disarm // Send Aileron value (Roll) USART_Transmit(sticksZero); USART_Transmit(0); // Send Elevator value (Pitch) USART_Transmit(sticksZero); 289 USART_Transmit(1); // Send Throttle value (ascend/descend) USART_Transmit(sticksZero); USART_Transmit(2); // Send Rudder value (Yaw) USART_Transmit(sticksZero); USART_Transmit(3); break; case 2: // Sensor driven val = read_average_ADC(ADC0)-minADC; // Greater than last row of LUT if(val>1001){ jointAngle = 67; // defaulting to max } else{ jointAngle = pgm_read_byte(&(fullFlex[val])); } // this assumes a max joint angle of 90, we have 67 so will never get full power as it stands uint8_t ppmWidth = jointAngle*1.11+100; //div10 of ppmWidth // Send Aileron value (Roll) USART_Transmit(sticksNeutral); USART_Transmit(0); // Send Elevator value (Pitch) USART_Transmit(sticksNeutral); USART_Transmit(1); // Send Throttle value (ascend/descend) USART_Transmit(ppmWidth); // sending value proportional to joint angle (which is based on ADC reading) USART_Transmit(2); // Send Rudder value (Yaw) USART_Transmit(sticksNeutral); USART_Transmit(3); break; } } else{ j++; } } /* ---- Main ---- */ int main(void) { j = 0; // Initialize j k=0; // Initialize k minADC = 0; // Initialize minADC at lowest possible value ctrlState = 0; // Initialize ctrlState sei(); // Set global interrupt enable bit USART_Init(MYUBRR); // Initialize Serial from USART initADC(); // Initialize ADC initTimer0(); // Initialize Timer0 290 touchMiddle = 1024; // Initialize as value that can never be reached touchRing = 1024; // Initialize as value that can never be reached cal_touchSense1(); // Setup compare value for middle finger touch sensor cal_touchSense2(); // Setup compare value for ring finger touch sensor // Event Loop while(1){ // Approximately every 5 seconds recalibrate touchSense limits to adjust for battery level if(k==196){ k=0; cal_touchSense1(); cal_touchSense2(); } } return(0); // Required } 291 Appendix 20: Quadcopter receiver code (quadRX) In conjunction with quadTX code in Appendix 19 this code is loaded onto the Arduino Uno mounted on the quadcopter. After discerning a program flow value from a value of interest, the code multiplies the value of interest by 10 and has the appropriate pin write the value. The multiplication by 10 is to convert the transferred value, which has been scaled to be between 100 and 200, to an appropriate value for the writeMicrosecond function. /***************************************************************** quadRX in conjunction with quadTX (MCU or Arduino) Code Summary: This is the code for the Arduino mounted to the quadcopter. Using servo.writeMicroseconds(us) Multiplying incoming motor pulse length by 10. Individual trim adjustments are handeled in this script. *****************************************************************/ // Define Motor Variables #include // Define Motor Pins #define APIN 8 // Arduino pin 8 to NAZA A port #define EPIN 9 // Arduino pin 9 to NAZA E port #define TPIN 10 // Arduino pin 10 to NAZA T port #define RPIN 11 // Arduino pin 11 to NAZA R port const int trimA = 0; // trim for Aileron (roll) const int trimE = 0; // trim for Elevator (pitch) const int trimT = 0; // trim for Throttle (elevation) const int trimR = 0; // trim for Rudder (yaw) int inputSpeed; int inputNAZA; int serialTemp; void setup() { //inputSpeed=1520; // Neutral initial value pinMode(APIN, OUTPUT); pinMode(EPIN, OUTPUT); pinMode(TPIN, OUTPUT); pinMode(RPIN, OUTPUT); 292 A.attach(APIN); E.attach(EPIN); T.attach(TPIN); R.attach(RPIN); Serial.begin(9600); // Needed for accessing TX/RX ports (XBees) } void loop() { if(Serial.available()){ serialTemp = Serial.read(); //Serial.println(serialTemp); if(serialTemp>4){inputSpeed = serialTemp*10;} if(serialTemp<4){ inputNAZA = serialTemp; switch(inputNAZA){ case 0: // NAZA port A A.writeMicroseconds(inputSpeed + trimA); break; case 1: // NAZA port E E.writeMicroseconds(inputSpeed + trimE); break; case 2: // NAZA port T T.writeMicroseconds(inputSpeed + trimT); break; case 3: // NAZA port R R.writeMicroseconds(inputSpeed + trimR); break; } } } } 293 Appendix 21: Glove Circuit Parts List Table A21.1 Parts list for sensor glove circuit.13 Part Name Vendor Part Number Quantity ATmega328p Mouser ATMEGA328P‐AU 1 3.3V LVR Mouser 584‐ADP160AUJZ‐3.3R7 1 Op‐Amp Mouser 579‐MCP6231T‐E/OT 1 330 Ω Resistor Mouser 71‐TNPW1206330RBEEA 1 1 kΩ Resistor Mouser 71‐TNPW12061K00BEEA 4 10 kΩ Resistor Mouser 71‐TNPW120610K0BEEA 2 20 kΩ Resistor Mouser 71‐TNPW120620K0BEEA 2 0.1 μF Capacitor Mouser 581‐1206ZC104JAT2A 1 1 μF Capacitor Mouser 581‐12063C105JAT2A 3 5 kΩ Trimmer Mouser 667‐EVM‐1ESX30B53 1 Molex Pico‐Lock Housing Mouser 538‐504050‐0491 1 Molex Pico‐Lock Wire Housing Mouser 538‐504051‐0401 1 Pico‐Lock Wire Terminals Mouser 538‐504052‐0098‐LP 4 Right Angle Switch Sparkfun 10860 1 LED Sparkfun 12622 1 XBee 1mW Trace Antenna Series 1 Sparkfun 11215 1 2mm 10pin XBee Socket – SMD Sparkfun 10030 2 JST Right Angle Connector Sparkfun 8612 1 Lithium Ion Polymer Battery (500mAh) Adafruit 1578 1 AVR Pocket Programmer14 Sparkfun 9825 1 13 All elements are surface mount and all capacitors and resistors are size 1206 14 Used to flash code to ATmega328p 294 Appendix 22: Glove Circuit Schematic The following are images of the full schematic for the glove circuit and enlarged detailed subsections. Figure A22.1 Full circuit schematic. 295 Figure A22.2 Battery and switch (top) and LED (bottom) subsection. Figure A22.3 Wheatstone bride with op‐amp subsection. “Missing” leg of bridge is sensor mounted on glove and is connected to the circuit using Molex pico‐lock pins 3 and 4 shown in Figure A19.4 296 Figure A22.4 Molex pico‐lock connector subsection showing sensor interface. Pins 1 and 2 connect to the touch sensors located on the index and middle fingers of glove. Figure A22.5 LVR and XBee subsection. Note XBee right pins and ground on XBee left pin set used to flash code onto ATmega328p. 297 Figure A22.6 ATmega328p power and ground connections subsection. 298