York University – Department of Computer Science and Engineering Handwriting Recognition: The Case for Unistrokes (and how to roll your own recognizer) I. Scott MacKenzie York University 1 York University – Department of Computer Science and Engineering Plan • Pen-based computing • Unistrokes • Recognition • Demo 2 1 York University – Department of Computer Science and Engineering The Pen as an Input Device • Advantages • considerable existing skill • good for annotating, sketching, etc. • handwriting is natural • Disadvantages • handwriting recognition is hard! • in early products, goods delivered did not live up to promises • Goal of “natural handwriting recognition” is elusive 3 York University – Department of Computer Science and Engineering 4 2 York University – Department of Computer Science and Engineering Pen-Based Computers (1989-1995) GRiDPad (Grid) 325 Point (Fujitsu) MessagePad 2100 (Apple) 5 York University – Department of Computer Science and Engineering PalmPilot • Introduced in 1996 by Palm Computing (now 3Com) • First successful pen-based system or PDA (personal digital assistant) • Handwriting recognition using Graffiti • Brainchild of Jeff Hawkins • Founder of Palm (and formerly Grid) 6 3 York University – Department of Computer Science and Engineering Plan • Pen-based computing • Unistrokes • Recognition • Demo 7 York University – Department of Computer Science and Engineering Unistrokes • What is a Unistroke? • A single stylus or finger gesture • Begins when stylus touches tablet surface • Ends when stylus is raised from tablet surface • Each “symbol” is one stroke 8 4 York University – Department of Computer Science and Engineering Unistrokes vs. Graffiti • Unistrokes • Invented by David Goldberg, Xerox PARC a • Patent held by Xerox (issued Jan. 1997) • First presented at INTERCHI ’93 • Graffiti • An ‘example’ of unistrokes • Commercial product • 3COM PalmPilot, HP Omnigo, etc. a Goldberg, D., & Richardson, C. (1993). Touch-typing with a stylus. Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems, 80-87. New York: ACM. 9 York University – Department of Computer Science and Engineering Unistrokes Goal: Fast for experts 10 5 York University – Department of Computer Science and Engineering Graffiti Goal: Easy for novices See… MacKenzie, I. S., & Zhang, S. (1997). The immediate usability of Graffiti. Proceedings of Graphics Interface '97, pp. 129-137. Toronto: Canadian Information Processing Society. 11 York University – Department of Computer Science and Engineering Application (TV Remote Control) See… Enns, N., & MacKenzie, I. S. (1998). Touch-based remote control devices. Extended Abstracts of the CHI '98 Conference on Human Factors in Computing Systems, 229-230. New York: ACM. 12 6 York University – Department of Computer Science and Engineering Application (Touchscreen Phone) See… Tinwala, H, & MacKenzie, I. S. (2009). Eyes-free text entry on a touchscreen phone. Proceedings of the IEEE Toronto International Conference – Science and Technology for Humanity – TIC-STH 2009, pp. 83-88. New York: IEEE. 13 York University – Department of Computer Science and Engineering Plan • Pen-based computing • Unistrokes • Recognition • Demo 14 7 York University – Department of Computer Science and Engineering Steps for Recognition • Input x,y coordinates for unistroke • Determine if stroke is a straight line (if so, done!) • Determine min and max for x and y • Normalize points to fit in unit bounding box • Note: box has four quadrants... 15 York University – Department of Computer Science and Engineering Steps (2) 3 0 2 1 16 8 York University – Department of Computer Science and Engineering Steps (3) • Scan points and gather characteristics of stroke: • 1st quadrant • 2nd quadrant • 2nd last quadrant • last quadrant • cumulative x motion (e.g. ‘Z’ = 3) • cumulative y motion (e.g., ‘Z’ = 1) 17 York University – Department of Computer Science and Engineering Steps (3) • starting x direction (+ve, -ve) • starting y direction (+ve, -ve) • stopping x direction (+ve, -ve) • stopping y direction (+ve, -ve) • Compare stroke characteristics with dictionary entries • Find match, done 18 9 York University – Department of Computer Science and Engineering Dictionary Entries • Each entry contains • symbol (the ‘output’) • 1st quadrant • 2nd quadrant • 2nd last quadrant • last quadrant • min cumulative x • max cumulative x 19 York University – Department of Computer Science and Engineering Dictionary Entries (2) • min cumulative y • max cumulative y • starting x motion (+ve, -ve, don’t care) • starting y motion (+ve, -ve, don’t care) • stopping x motion (+ve,-ve, don’t care) • stopping y motion (+ve, -ve, don’t care) • Note: can have multiple entries for each symbol (e.g.cw“0”,ccw“0”) 20 10 York University – Department of Computer Science and Engineering Example Symbol B 3 0 1st quad 2 2nd quad 3 n-1 quad 1 n quad 2 kx min 2.2 kx max 4 ky min 1.5 ky max 2.5 start x don’t care start y + stop x - stop y don’t care 2 1 21 York University – Department of Computer Science and Engineering Plan • Pen-based computing • Unistrokes • Recognition • Demo 22 11 York University – Department of Computer Science and Engineering 23 York University – Department of Computer Science and Engineering Thank You 24 12.