Handwriting Recognition

York University – Department of Computer Science and Engineering

Handwriting Recognition:

The Case for Unistrokes (and how to roll your own recognizer)

I. Scott MacKenzie York University

York University – Department of Computer Science and Engineering

Plan

• Pen-based computing • Unistrokes • Recognition • Demo

1 York University – Department of Computer Science and Engineering

The Pen as an Input Device

• Advantages • considerable existing skill • good for annotating, sketching, etc. • handwriting is natural • Disadvantages • handwriting recognition is hard! • in early products, goods delivered did not live up to promises • Goal of “natural handwriting recognition” is elusive

York University – Department of Computer Science and Engineering

2 York University – Department of Computer Science and Engineering

Pen-Based Computers (1989-1995)

GRiDPad (Grid) 325 Point (Fujitsu)

MessagePad 2100 (Apple)

York University – Department of Computer Science and Engineering

PalmPilot • Introduced in 1996 by Palm Computing (now 3Com) • First successful pen-based system or PDA (personal digital assistant) • Handwriting recognition using Graffiti • Brainchild of Jeff Hawkins • Founder of Palm (and formerly Grid)

3 York University – Department of Computer Science and Engineering

Plan

• Pen-based computing • Unistrokes • Recognition • Demo

York University – Department of Computer Science and Engineering

Unistrokes

• What is a Unistroke? • A single stylus or finger gesture • Begins when stylus touches tablet surface • Ends when stylus is raised from tablet surface • Each “symbol” is one stroke

4 York University – Department of Computer Science and Engineering

Unistrokes vs. Graffiti

• Unistrokes • Invented by David Goldberg, Xerox PARC a • Patent held by Xerox (issued Jan. 1997) • First presented at INTERCHI ’93 • Graffiti • An ‘example’ of unistrokes • Commercial product • 3COM PalmPilot, HP Omnigo, etc. a Goldberg, D., & Richardson, C. (1993). Touch-typing with a stylus. Proceedings of the INTERCHI '93 Conference on Human Factors in Computing Systems, 80-87. New York: ACM. 9

York University – Department of Computer Science and Engineering

Unistrokes Goal: Fast for experts

5 York University – Department of Computer Science and Engineering

Graffiti Goal: Easy for novices

See… MacKenzie, I. S., & Zhang, S. (1997). The immediate usability of Graffiti. Proceedings of Graphics Interface '97, pp. 129-137. Toronto: Canadian Information Processing Society. . 11

York University – Department of Computer Science and Engineering

Application (TV Remote Control)

See… Enns, N., & MacKenzie, I. S. (1998). Touch-based remote control devices. Extended Abstracts of the CHI '98 Conference on Human Factors in Computing Systems, 229-230. New York: ACM. 12

6 York University – Department of Computer Science and Engineering

Application (Touchscreen Phone)

See… Tinwala, H, & MacKenzie, I. S. (2009). Eyes-free text entry on a touchscreen phone. Proceedings of the IEEE Toronto International Conference – Science and Technology for Humanity – TIC-STH 2009, pp. 83-88. New York: IEEE. 13

York University – Department of Computer Science and Engineering

Plan

• Pen-based computing • Unistrokes • Recognition • Demo

7 York University – Department of Computer Science and Engineering

Steps for Recognition

• Input x,y coordinates for unistroke • Determine if stroke is a straight line (if so, done!) • Determine min and max for x and y • Normalize points to fit in unit bounding box • Note: box has four quadrants...

York University – Department of Computer Science and Engineering

Steps (2)

3 0

2 1

8 York University – Department of Computer Science and Engineering

Steps (3)

• Scan points and gather characteristics of stroke: • 1st quadrant • 2nd quadrant • 2nd last quadrant • last quadrant • cumulative x motion (e.g. ‘Z’ = 3) • cumulative y motion (e.g., ‘Z’ = 1)

York University – Department of Computer Science and Engineering

Steps (3)

• starting x direction (+ve, -ve) • starting y direction (+ve, -ve) • stopping x direction (+ve, -ve) • stopping y direction (+ve, -ve) • Compare stroke characteristics with dictionary entries • Find match, done

9 York University – Department of Computer Science and Engineering

Dictionary Entries

• Each entry contains • symbol (the ‘output’) • 1st quadrant • 2nd quadrant • 2nd last quadrant • last quadrant • min cumulative x • max cumulative x

York University – Department of Computer Science and Engineering

Dictionary Entries (2)

• min cumulative y • max cumulative y • starting x motion (+ve, -ve, don’t care) • starting y motion (+ve, -ve, don’t care) • stopping x motion (+ve,-ve, don’t care) • stopping y motion (+ve, -ve, don’t care) • Note: can have multiple entries for each symbol (e.g.cw“0”,ccw“0”)

10 York University – Department of Computer Science and Engineering

Example

Symbol B 3 0 1st quad 2 2nd quad 3 n-1 quad 1 n quad 2 kx min 2.2 kx max 4 ky min 1.5 ky max 2.5 start x don’t care start y + stop x - stop y don’t care 2 1

York University – Department of Computer Science and Engineering

Plan

• Pen-based computing • Unistrokes • Recognition • Demo

11 York University – Department of Computer Science and Engineering

York University – Department of Computer Science and Engineering

Thank You