Dewristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited

deWristified: Handwriting Inference Using Wrist-Based Motion Sensors Revisited Raveen Wijewickrama Anindya Maiti Murtuza Jadliwala [email protected] [email protected] [email protected] University of Texas at San Antonio Wrist Wearables • Extends the functionality of traditional wristwatches beyond timekeeping. • Captures rich contextual information about the wearer. • Enables several novel context-based applications. 2019-05-16 SPriTELab @ UTSA 2 Motion Sensors • Two main types of motion or inertial sensors: • Accelerometer: records device acceleration. • Gyroscope: records device angular rotation. • Accessing motion sensors on wearable devices: • All applications have access to motion sensors by default (also referred to as zero-permission sensors) on most wearable OSs. • Applications’ access to motion sensors cannot be regulated on most wearable OSs – we can’t turn them off! • Can an adversary take advantage of motion sensor data from a wrist-wearable device to infer private information inputted by the user’s device-wearing hand? 2019-05-16 SPriTELab @ UTSA 3 Inferring Private User Inputs (Using Wrist Wearables) 2019-05-16 SPriTELab @ UTSA 4 State-of-the-Art in Handwriting Recognition (Using Wrist Wearables) Airwriting (Amma et al.) Whiteboard writing (Arduser et al.) Finger writing (Xu et al.) Pen(cil) writing (Xia et al.) 2019-05-16 SPriTELab @ UTSA 5 Adversary Model • Adversary has knowledge of the type of handwriting. • Adversary is able to record data from the target smartwatch’s accelerometer and gyroscope sensors. • Could employ a Trojan app for this! • Adversary’s Goal: To infer handwritten information using target user’s smartwatch sensors. 2019-05-16 SPriTELab @ UTSA 6 Limitations of Earlier Handwriting Recognition Studies (Using Wrist Wearables) • Airwriting (Amma et al.) • Finger writing (Xu et al.) • Custom-designed hand glove with very • Use of Shimmer, a specialized sensing high precision sensors. device intended for lab studies. • Our adversary relies on target user’s • Not generalized (training and testing smartwatch or fitness band. data not from different participants). • Only uppercase words. • Pen(cil) writing (Xia et al.) • Whiteboard writing (Arduser et al.) • Only lowercase alphabets. • Not generalized (training and testing • Controlled data collection. data not from different participants). • No handwriting activity detection. • Only uppercase alphabets. • No handwriting activity detection. 2019-05-16 SPriTELab @ UTSA 7 Our Research • How practical is handwriting inference when • Using consumer-grade wrist wearables, • Using generalized training and testing, New Uncontrolled and Unconstrained • Writing in a uncontrolled and Writing Data unconstrained manner, and • Both upper and lowercase alphabets are modeled ? Existing Models 2019-05-16 SPriTELab @ UTSA 8 Handwriting Inference Framework 2019-05-16 SPriTELab @ UTSA 9 Experimental Setup • 28 participants for the four writing scenarios. • 18 to 30 years of age • 13 male, 15 female • Two different wrist-wearables. • Sony Smartwatch 3, LG Watch Urbane • Accelerometer and gyroscope recorded at 200Hz. • Participants provided with appropriate writing apparatus. 2019-05-16 SPriTELab @ UTSA 10 Writing Tasks (In-Lab) • Alphabets. • Individual alphabets one at a time. • Covered all 26 English alphabets in random order. • Each alphabet was written 10 times. • Both upper and lower cases. • Words. • 4-8 alphabet words, from a vocabulary (Goldhahn et al. 2012). • Each participant wrote 20 words, in both upper and lower cases. • Sentence. • "the five boxing wizards jump quickly" in both upper and lower cases. 2019-05-16 SPriTELab @ UTSA 11 Writing Activity Recognition (Out of Lab) • 2 participants. • Wore a smartwatch for an entire day. • Performed the four writing scenarios at random times. • Adversary’s Goal: To infer handwriting activity first, and then classify the handwritten text. 2019-05-16 SPriTELab @ UTSA 12 Replicated Inference Frameworks • Airwriting • Hidden Markov Model (HMM) • Whiteboard writing • Dynamic Time Warping (DTW) • Finger writing • Naive Bayes, Logistic Regression and Decision Tree classifiers • Pen(cil) writing • Random Forest classifier 2019-05-16 SPriTELab @ UTSA 13 Personalized Inference Accuracy Writing Activity Detection: 56% recall and 57% precision for air and finger writing 39% recall and 47% precision for pencil writing 23% recall and 34% precision for whiteboard writing 2019-05-16 SPriTELab @ UTSA 14 Personalized Inference Accuracy (Whiteboard Writing) Lowercase Uppercase 2019-05-16 SPriTELab @ UTSA 15 Generalized Inference Accuracy Writing Activity Detection: 35-40% recall for airwriting, whiteboard writing and pencil writing Only 8% recall for finger writing 2019-05-16 SPriTELab @ UTSA 16 Factors Affecting Inference Accuracy • Number of Strokes. 2019-05-16 SPriTELab @ UTSA 17 Factors Affecting Inference Accuracy Number of strokes for the same letter for Number of strokes for the same letter for different participants (lowercase). different participants (uppercase). 2019-05-16 SPriTELab @ UTSA 18 Factors Affecting Inference Accuracy Lowercase Uppercase Variance in number of strokes per alphabet per participant, averaged for all participants 2019-05-16 SPriTELab @ UTSA 19 Factors Affecting Inference Accuracy • Number of Strokes. • Order of Strokes. • Direction of Strokes. 2019-05-16 SPriTELab @ UTSA 20 Factors Affecting Inference Accuracy 2019-05-16 SPriTELab @ UTSA 21 Factors Affecting Inference Accuracy • Number of Strokes. • Order of Strokes. • Direction of Strokes. • Uppercase vs Lowercase. • Specialized Devices. Airwriting (Amma et al.) 2019-05-16 SPriTELab @ UTSA 22 Conclusion • We investigated how wrist-wearable based handwriting inference attacks perform in realistic day-to-day writing situations. • Such inference attacks are unlikely to pose a substantial threat to users of current consume-grade smartwatches and fitness bands. • Primarily due to highly varying nature of handwriting. • Replicable artifacts: https://sprite.utsa.edu/art/dewristified 2019-05-16 SPriTELab @ UTSA 23.

Load more