(12) United States Patent (10) Patent No.: US 6,269,187 B1 Frink Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US006269187B1 (12) United States Patent (10) Patent No.: US 6,269,187 B1 Frink et al. (45) Date of Patent: *Jul. 31, 2001 (54) METHOD AND SYSTEM FOR DATA ENTRY (56) References Cited OF HANDWRITTEN SYMBOLS _ U.S. PATENT DOCUMENTS (75) Inventors: Lloyd Frink, Seattle; Bryon Dean BlShOp, Redmond, bOth Of WA (US) 4,817,034 * 3/1989 Hardin et al. ...................... .. 382/187 4,918,740 * 4/1990 ROSS . .. 382/187 (73) Assignee: Microsoft Corporation, Redmond, WA 4,953,225 * 8/1990 Togawa et al. 382/187 (US) 4,972,496 * 11/1990 Sklarew .... .. .. 382/187 5,063,600 * 11/1991 Norwood ........................... .. 382/187 (*) Notice: This patent issued on a continued pros- 5,956,423 * 9/1999 Frink et al. ........................ .. 382/187 ecution application ?led under 37 CFR 1.53(d), and is subject to the tWenty year * Cited by examiner patent term provisions of 35 U.S.C. 154(a)(2). Primary Examiner—Jose L. Couso Snbjeet to any disclaimer, the term of this (74) Attorney, Agent, or Firm—Michalik & Wylie, PLLC patent is extended or adjusted under 35 U.S.C. 154(b) by 0 days. (57) ABSTRACT Amethod and system for data entry of handwritten text into (21) APPL N05 09/386,248 a computer program that is not designed to accept hand (22) Filed, Aug 31’ 1999 Written text is provided. In preferred embodiments, the computer program is designed to operate in a WindoWing Related US Application Data environment. A data entry program receives handwritten data, recognizes the data, and sends the recognized data to (63) Continuation of application No. 08/179,813, ?led on Jan. 10, the Computer PrOgraHL The Computer Program Processes the 1994, now Pat. No. 5,956,423, which is a continuation of recognized data as if it had been entered from the keyboard. apphcatlon NO- 07/716723: ?led on Jun~ 17: 1991: HOW According to the present invention, the data entry program abandoned. overlaps the Window- of the computer program With- an (51) Int. Cl.7 ..................................................... .. G06K 9/00 invisible Window. To the user, it looks as if the computer (52) US. Cl. ............................................................ .. 382/187 program is aeeepting handwritten data directly (58) Field of Search ................................... .. 382/100, 187, 382/309; 178/18—20; 345/179—183 42 Claims, 12 Drawing Sheets U.S. Patent Jul. 31, 2001 Sheet 1 0f 12 US 6,269,187 B1 U.S. Patent Jul. 31, 2001 Sheet 2 0f 12 US 6,269,187 B1 200 Microsoft V/urd ' Dncumrrntl a file Edit ylew Insert Fan-nag utlli?cs Macro ?lndow Help This is a sample document. I1] 1 202 203 201 I 207 J Figure 2A U.S. Patent Jul. 31, 2001 Sheet 3 0f 12 US 6,269,187 B1 200 Micmsntt Wnrd * Document] 202 203 201 TIME, EH61 204 Figure 2B U.S. Patent Jul. 31, 2001 Sheet 4 0f 12 US 6,269,187 B1 200 Miltmsult V/urd ' Ducumunll _a Elle Edit my‘ Insert Formal utll?es Macro V?lndow Help_ This is a sample document. The Endrl] 202 j 205 j 206 J I 207 j Figure 2 C U.S. Patent Jul. 31, 2001 Sheet 5 0f 12 US 6,269,187 B1 310 309 Application Per Palette 301 Windows 308 302 RC Manager Mouse Hook 303 Recognizer 304 305 306 307 Mouse Keyboard Display Surface Input Grid Driver Driver Driver Driver Figure 3 U.S. Pa tent Jul. 31, 2001 $116666 6 6f 12 US 6,269,187 B1 WM_LBU'ITONDOWN [ Mouse Hook l 401 Detect WM_LBU I I ON DOWN 402 S and MPefliitaege Pen Palm Window Window Don't Process WM_INVOKEVKB 403 Determine Application Window Size 414 404 \ Create Invisible Window Invisible Window 405 Recognize 406 Save WM_RCRESULT Result 407 CloseWindow Invisible / 408 WM_CLOSE Close Window 409 Send Message / Pen Palette A Flg ure Wll’ldOW B > WM_RCRESULT 411 DONE —< : ‘ U.S. Patent Jul. 31, 2001 Sheet 7 0f 12 US 6,269,187 B1 Pen Palette Window 410 @; ConvertResult Windows Virtual 412 Application Events Send Message Window A to Application Window 413 Message Process Message Figure 4B U.S. Patent Jul. 31, 2001 Sheet 8 of 12 US 6,269,187 B1 Pen Palette Initialization 501 Install DWMouse Hook 502 Replace I Beam with Pen Bitmap 503 Register Class Invisible Window Figure 5 U.S. Patent Jul. 31, 2001 Sheet 9 0f 12 US 6,269,187 B1 Mousepointer I Beam Fost Message WM_INVOKE VKB t0 Pen Palette Window Figure 6 ‘ Retufn l ’ U.S. Patent Jul. 31, 2001 Sheet 10 0f 12 US 6,269,187 B1 Pen Palette Wnd Proc Calculate Size of Invisible Window 703 Initialize Convert _Gestures Recognition I0 Virtual Parameters Events 704 712 Send Virtual Process Writing to gyiilclitosws l Return ) l Return l Figure 7 U.S. Patent Jul. 31, 2001 Sheet 11 0f 12 US 6,269,187 B1 ( Process Writing I 80] Create Window for Invisible Window 802 Set Window Position for Invisible Window 803 Update Window for Invisible Window 804 Recognize 805 Send Message WM_CLOSE to Invisible Window 806 Send Message WM_RCRESULT to Pen Palette Window Figure 8 i Return , U.S. Patent Jul. 31, 2001 Sheet 12 0f 12 US 6,269,187 B1 Invisible Wnd Free 901 essage Return WM_MOUSE MA_NOACTIVATE ACTIVATE Save RCRESULT ‘ Return ) Figure 9 US 6,269,187 B1 1 2 METHOD AND SYSTEM FOR DATA ENTRY at typing. They shun using computers because it is burden OF HANDWRITTEN SYMBOLS some to enter data. These people, hoWever, Would use a computer if the computer program Would accept handWritten CROSS-REFERENCE TO RELATED data entry. Also, there are many situations Where the use of APPLICATION a keyboard is impractical Where the use of handWritten data entry Would be practical. This application is a continuation of US. patent applica tion Ser. No. 08/179,813, ?led Jan. 10, 1994, now US. Pat. Many of the non-pen application programs are Written for No. 5,956,423 Which is a continuation of US. patent appli a WindoWing environment. One such WindoWing environ cation Ser. No. 07/716,723, ?led Jun. 17, 1991, noW aban ment is WindoWs by Microsoft Corporation of Redmond, doned. Wash. Several operating systems also have integrated Win doWing environments. In a WindoWing environment, each TECHNICAL FIELD application program is assigned a WindoW. The WindoWs for several programs can be shoWn on the computer display This invention is related to a method and computer system simultaneously. Typically, the WindoWing environment sup of data entry of handWritten characters and, in particular, 15 ports multitasking, that is, multiple programming may be data entry in a WindoWing environment. operating concurrently. The WindoWing system controls the positioning and siZe of the WindoWs on the screen. Typically, BACKGROUND OF THE INVENTION the user has considerable ?exibility in arranging the Win Handwriting recognition systems are being developed to doWs. In most WindoWing systems, the WindoWs can over alloW users to input handWritten data into a computer. An lap. The WindoW that logically overlaps obscures the over electronic tablet alloWs a user to hand Write data that the lapped WindoW. computer can recogniZe. The user Writes data on the elec It Would require extensive redesigning and reprogram tronic tablet With a special pen. As the user moves the pen ming to adapt non-pen programs to accept handWritten data. across the tablet, the tablet displays “ink.” The tablet senses In some situations, it may not be economically feasible to the position of the pen as the user Writes and sends the 25 adapt a non-pen program. The costs of the adaptations may position data to the computer. The computer receives this not be offset by increased revenue resulting from the pro position data and converts it to recogniZed symbols, such as gram accepting handWritten data. It Would be desirable if letters or numbers, in a convenient formate, such as ASCII these non-pen application programs cold receive handWrit format. The computer then processes the formatted charac ten data Without the need to adapt the non-pen application ters. program. It Would be desirable if the non-pen program could Some handWriting recognition systems recogniZe not only also receive handWritten gestures. characters, letters and punctuation, but also gestures. Ges SUMMARY OF THE INVENTION tures are handWritten symbols that are typically used to represent editing commands or to produce non-printing text. 35 The methods of the present invention provide for the data For example, a handWritten backslash over a letter may entry of handWritten data into a computer program that is not mean delete the letter, a caret in betWeen tWo letters may designed to accept handWritten data. According to the mean insert text, or a vertical line With an arroW to the left methods, a non-pen application program operates in a Win may mean insert a carriage return. Gestures are used exten doWing environment. A handWriting data entry program sively When printed documents are edited. During detects When a user Wants to input handWritten data. The recognition, these gestures are converted to a convenient data entry program creates an “invisible” WindoW, Which format for use by the computer. preferably overlaps the application WindoW. An “invisible” These handWriting recognition systems use algorithms to WindoW is a WindoW that When created and displayed has no map handWritten data to symbols. Typically, these systems visual effect on the display. The WindoWing environment internally store a prototype for each symbol that can be 45 internally treats the creating and displaying of an invisible recogniZed. A prototype is a “picture” of a handWritten WindoW like a visible WindoW. HoWever, When the user symbol that is used to map handWriting to a symbol.