deWristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited
Raveen Wijewickrama Anindya Maiti Murtuza Jadliwala [email protected] [email protected] [email protected]
University of Texas at San Antonio Wrist Wearables
• Extends the functionality of traditional wristwatches beyond timekeeping.
• Captures rich contextual information about the wearer. • Enables several novel context-based applications.
2019-05-16 SPriTELab @ UTSA 2 Motion Sensors
• Two main types of motion or inertial sensors: • Accelerometer: records device acceleration. • Gyroscope: records device angular rotation.
• Accessing motion sensors on wearable devices: • All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs. • Applications’ access to motion sensors cannot be regulated on most wearable OSs – we can’t turn them off!
• Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand?
2019-05-16 SPriTELab @ UTSA 3 Inferring Private User Inputs (Using Wrist Wearables)
2019-05-16 SPriTELab @ UTSA 4 State-of-the-Art in Handwriting Recognition (Using Wrist Wearables)
Airwriting (Amma et al.) Whiteboard writing (Arduser et al.)
Finger writing (Xu et al.) Pen(cil) writing (Xia et al.)
2019-05-16 SPriTELab @ UTSA 5 Adversary Model
• Adversary has knowledge of the type of handwriting.
• Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors. • Could employ a Trojan app for this!
• Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors.
2019-05-16 SPriTELab @ UTSA 6 Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables)
• Airwriting (Amma et al.) • Finger writing (Xu et al.) • Custom-designed hand glove with very • Use of Shimmer, a specialized sensing high precision sensors. device intended for lab studies. • Our adversary relies on target user’s • Not generalized (training and testing smartwatch or fitness band. data not from different participants). • Only uppercase words.
• Pen(cil) writing (Xia et al.) • Whiteboard writing (Arduser et al.) • Only lowercase alphabets. • Not generalized (training and testing • Controlled data collection. data not from different participants). • No handwriting activity detection. • Only uppercase alphabets. • No handwriting activity detection.
2019-05-16 SPriTELab @ UTSA 7 Our Research
• How practical is handwriting inference when • Using consumer-grade wrist wearables, • Using generalized training and testing, New Uncontrolled and Unconstrained • Writing in a uncontrolled and Writing Data unconstrained manner, and • Both upper and lowercase alphabets are modeled ? Existing Models
2019-05-16 SPriTELab @ UTSA 8 Handwriting Inference Framework
2019-05-16 SPriTELab @ UTSA 9 Experimental Setup
• 28 participants for the four writing scenarios. • 18 to 30 years of age • 13 male, 15 female
• Two different wrist-wearables. • Sony Smartwatch 3, LG Watch Urbane
• Accelerometer and gyroscope recorded at 200Hz.
• Participants provided with appropriate writing apparatus.
2019-05-16 SPriTELab @ UTSA 10 Writing Tasks (In-Lab)
• Alphabets. • Individual alphabets one at a time. • Covered all 26 English alphabets in random order. • Each alphabet was written 10 times. • Both upper and lower cases. • Words. • 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012). • Each participant wrote 20 words, in both upper and lower cases. • Sentence. • "the five boxing wizards jump quickly" in both upper and lower cases.
2019-05-16 SPriTELab @ UTSA 11 Writing Activity Recognition (Out of Lab)
• 2 participants.
• Wore a smartwatch for an entire day.
• Performed the four writing scenarios at random times.
• Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text.
2019-05-16 SPriTELab @ UTSA 12 Replicated Inference Frameworks
• Airwriting • Hidden Markov Model (HMM)
• Whiteboard writing • Dynamic Time Warping (DTW)
• Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers
• Pen(cil) writing • Random Forest classifier
2019-05-16 SPriTELab @ UTSA 13 Personalized Inference Accuracy
Writing Activity Detection: 56% recall and 57% precision for air and finger writing 39% recall and 47% precision for pencil writing 23% recall and 34% precision for whiteboard writing
2019-05-16 SPriTELab @ UTSA 14 Personalized Inference Accuracy (Whiteboard Writing)
Lowercase Uppercase
2019-05-16 SPriTELab @ UTSA 15 Generalized Inference Accuracy
Writing Activity Detection: 35-40% recall for airwriting, whiteboard writing and pencil writing Only 8% recall for finger writing
2019-05-16 SPriTELab @ UTSA 16 Factors Affecting Inference Accuracy
• Number of Strokes.
2019-05-16 SPriTELab @ UTSA 17 Factors Affecting Inference Accuracy
Number of strokes for the same letter for Number of strokes for the same letter for different participants (lowercase). different participants (uppercase).
2019-05-16 SPriTELab @ UTSA 18 Factors Affecting Inference Accuracy
Lowercase Uppercase
Variance in number of strokes per alphabet per participant, averaged for all participants
2019-05-16 SPriTELab @ UTSA 19 Factors Affecting Inference Accuracy
• Number of Strokes. • Order of Strokes. • Direction of Strokes.
2019-05-16 SPriTELab @ UTSA 20 Factors Affecting Inference Accuracy
2019-05-16 SPriTELab @ UTSA 21 Factors Affecting Inference Accuracy
• Number of Strokes. • Order of Strokes. • Direction of Strokes. • Uppercase vs Lowercase. • Specialized Devices.
Airwriting (Amma et al.)
2019-05-16 SPriTELab @ UTSA 22 Conclusion
• We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations.
• Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands. • Primarily due to highly varying nature of handwriting.
• Replicable artifacts: https://sprite.utsa.edu/art/dewristified
2019-05-16 SPriTELab @ UTSA 23