USOO7305130B2

(12) United States Patent (10) Patent No.: US 7,305,130 B2 Frink et al. (45) Date of Patent: *Dec. 4, 2007

(54) METHOD AND SYSTEM FOR DATA ENTRY (52) U.S. Cl...... 382/187 OF HANDWRITTEN SYMBOLS (58) Field of Classification Search ...... 382/100, (75) Inventors: Lloyd Frink, Seattle, WA (US); Bryon 382/119, 177, 179, 186, 187, 309; 34.5/173, Dean Bishop, Redmond, WA (US) 345/179–183; 178/18-20 See application file for complete search history. (73) Assignee: Microsoft Corporation, Redmond, WA (US) (56) References Cited U.S. PATENT DOCUMENTS (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 5,956,423 A * 9/1999 Frink et al...... 382,187 U.S.C. 154(b) by 0 days. 6,269,187 B1 * 7/2001 Frink et al...... 382,187 6,594,390 B2 * 7/2003 Frink et al...... 382,187 This patent is Subject to a terminal dis 7,116,822 B2 * 10/2006 Frink et al...... 382,187 claimer. (21) Appl. No.: 11/511,706 * cited by examiner Primary Examiner Jose L. Couso (22) Filed: Aug. 28, 2006 (74) Attorney, Agent, or Firm Workman Nydegger (65) Prior Publication Data US 2006/0291725 A1 Dec. 28, 2006 (57) ABSTRACT Related U.S. Application Data A method and system for data entry of handwritten text into a computer program that is not designed to accept hand (63) Continuation of application No. 10/61 1,489, filed on written text is provided. In preferred embodiments, the Jun. 30, 2003, now Pat. No. 7,116,822, which is a computer program is designed to operate in a windowing continuation of application No. 09/908,113, filed on environment. A data entry program receives handwritten Jul. 17, 2001, now Pat. No. 6,594,390, which is a data, recognizes the data, and sends the recognized data to continuation of application No. 09/386.248, filed on the computer program. The computer program processes the Aug. 31, 1999, now Pat. No. 6,269,187, which is a recognized data as if it had been entered from the keyboard. continuation of application No. 08/179,813, filed on According to the present invention, the data entry program Jan. 10, 1994, now Pat. No. 5,956,423, which is a overlaps the window of the computer program with an continuation of application No. 07/716.723, filed on invisible window. To the user, it looks as if the computer Jun. 17, 1991, now abandoned. program is accepting handwritten data directly. (51) Int. C. G06K 9/00 (2006.01) 20 Claims, 12 Drawing Sheets

30 309 Application

RC Manager 303

Recognizer

305 306 307 Keyboard Display Surface Input Grid Diver Driver Driver U.S. Patent Dec. 4, 2007 Sheet 1 of 12 US 7,305,130 B2

U.S. Patent Dec. 4, 2007 Sheet 2 of 12 US 7,305,130 B2

200 E MicrosoftWord-Document 1 ve a File Edit View insert Format Utilities Macro Window

I 202 -/

FIG. 2A U.S. Patent Dec. 4, 2007 Sheet 3 of 12 US 7,305,130 B2

200

e O O MicroSoftWord-DOCument we a File Edit View insert Format Utilities Macro Window Style: Normal This is a sample document. I I 202- 203

FIG. 2B U.S. Patent Dec. 4, 2007 Sheet 4 of 12 US 7,305,130 B2

200 e MicroSoftWord-DOCument 1 vé a File Edit View insert Format Utilities Macro Window Style: Normal : 1 This is a sample document. The End" 202 205 206 -/

FIG. 2C U.S. Patent Dec. 4, 2007 Sheet S of 12 US 7,305,130 B2

30 309

301

308

Recognizer

304 305 306 307 Mouse Keyboard Display Surface Input Grid Driver Driver Driver Driver

Figure 3 U.S. Patent Dec. 4, 2007 Sheet 6 of 12 US 7,305,130 B2

WM LBUTTONDOWN

40

SendMessage Pen Palette Pen Palette Window Window Don't Process WM INVOKEVKB

Determine Application Window Size

44 404

Create Invisible Window Invisible Window 405

Send Message Pen Palette Window Figure 4A U.S. Patent Dec. 4, 2007 Sheet 7 of 12 US 7,305,130 B2

Pen Palette Window

Application SendMessage Window to Application Window

Process Message

Figure 4B U.S. Patent Dec. 4, 2007 Sheet 8 of 12 US 7,305,130 B2

Pen Palette Initialization

502 Replace I Beam with Pen Bitmap

SO3 Register Class Invisible Window

Figure 5 U.S. Patent Dec. 4, 2007 Sheet 9 of 12 US 7,305,130 B2

DWMouse Hook

601 Message

TDOWN

602 Mousepointer I Beam

603

Post Message WM INVOKE VKB to Pen Palette Window Figure 6 U.S. Patent Dec. 4, 2007 Sheet 10 of 12 US 7,305,130 B2

Pen Palette Wind PrOC

701

Message NO WM INVOKEVKB

Yes

702 710

Calculate Size Message Of Invisible Window WMRCRESULT

Yes 703 711 Initialize Convert Gestures Recognition To Virtual Parameters Events

704 742 Send Virtual Process Writing Events To Windows

FIG. U.S. Patent Dec. 4, 2007 Sheet 11 of 12 US 7,305,130 B2

Create Window for Invisible Window

Set Window Position for Invisible Window

Update Window for Invisible Window

SendMessage WMCLOSE to Invisible Window

SendMessage WMRCRESULT

Palette Window Figure 8 U.S. Patent Dec. 4, 2007 Sheet 12 of 12 US 7,305,130 B2

Invisible Wind Proc

90 essage Return WMMOUSE MA NOACTIVATE ACTIVATE

Message WM RCRESULT

903 Save RCRESULT

Figure 9 US 7,305,130 B2 1. 2 METHOD AND SYSTEM FOR DATA ENTRY receive data typed by a user at a computer keyboard. Such OF HANDWRITTEN SYMBOLS programs are call non-pen programs, that is, the programs are not designed to Support handwritten data entry. It is often CROSS-REFERENCE TO RELATED desirable to allow users the option to input data through a APPLICATION keyboard or by handwriting. Many people are not proficient at typing. They shun using computers because it is burden This application is a continuation of U.S. patent applica Some to enter data. These people, however, would use a tion Ser. No. 10/611,489, filed Jun. 30, 2003, now U.S. Pat. computer if the computer program would accept handwritten No. 7,116,822,which is a continuation of U.S. patent appli data entry. Also, there are many situations where the use of cation Ser. No. 09/908,113, filed Jul. 17, 2001, now U.S. Pat. 10 a keyboard is impractical and where the use of handwritten No. 6,594,390, which is a continuation of U.S. patent data entry would be practical. application Ser. No. 09/386.248, filed Aug. 31, 1999, now Many of the non-pen application programs are written for U.S. Pat. No. 6,269,187, which is a continuation of U.S. a windowing environment. One Such windowing environ patent application Ser. No. 08/179,813, filed Jan. 10, 1994, ment is Windows by Microsoft Corporation of Redmond, now U.S. Pat. No. 5,956,423, which is a continuation of U.S. 15 Wash. Several operating systems also have integrated win patent application Ser. No. 07/716,723, filed Jun. 17, 1991, dowing environments. In a windowing environment, each now abandoned. application program is assigned a window. The windows for several programs can be shown on the computer display TECHNICAL FIELD simultaneously. Typically, the windowing environment Sup ports multitasking, that is, multiple programming may be This invention is related to a method and computer system operating concurrently. The windowing system controls the of data entry of handwritten characters and, in particular, positioning and size of the windows on the screen. Typically, data entry in a windowing environment. the user has considerable flexibility in arranging the win dows. In most windowing systems, the windows can over BACKGROUND OF THE INVENTION 25 lap. The window that logically overlaps obscures the over lapped window. Handwriting recognition systems are being developed to It would require extensive redesigning and reprogram allow users to input handwritten data into a computer. An ming to adapt non-pen programs to accept handwritten data. electronic tablet allows a user to hand write data that the In some situations, it may not be economically feasible to computer can recognize. The user writes data on the elec 30 adapt a non-pen program. The costs of the adaptations may tronic tablet with a special pen. As the user moves the pen not be offset by increased revenue resulting from the pro across the tablet, the tablet displays “ink.” The tablet senses gram accepting handwritten data. It would be desirable if the position of the pen as the user writes and sends the these non-pen application programs could receive handwrit position data to the computer. The computer receives this ten data without the need to adapt the non-pen application position data and converts it to recognized symbols. Such as 35 program. It would be desirable if the non-pen program could letters or numbers, in a convenient format, such as ASCII also receive handwritten gestures. format. The computer then processes the formatted charac terS. SUMMARY OF THE INVENTION Some handwriting recognition systems recognize no only characters, letters and punctuation, but also gestures. Ges 40 The methods of the present invention provide for the data tures are handwritten symbols that are typically used to entry of handwritten data into a computer program that is not represent editing commands or to produce non-printing text. designed to accept handwritten data. According to the meth For example, a handwritten backlash over a letter may mean ods, a non-pen application program operates in a windowing delete the letter, a caret in between two letters may mean environment. A handwriting data entry program detects insert text, or a vertical line with an arrow to the left may 45 when a user wants to input handwritten data. The data entry mean insert a carriage return. Gestures are used extensively program creates an “invisible' window, which preferably when printed documents are edited. During recognition, overlaps the application window. An “invisible' window is these gestures are converted to a convenient format for use a window that when created and displayed has no visual by the computer. effect on the display. The windowing environment internally These handwriting recognition systems use algorithms to 50 treats the creating and displaying of an invisible window like map handwritten data to symbols. Typically, these systems a visible window. However, when the user writes with the internally store a prototype for each symbol that can be pen, the data entry program inputs the handwritten data, recognized. A prototype is a "picture of a handwritten converts the data to recognized symbols, and erases the symbol that is used to map handwriting to a symbol. A handwritten data from the display. The data entry program system that recognizes just capital letters may have only 26 55 then sends the recognized symbols to the application pro prototypes: one for each capital letter. Recognition systems gram as virtual keyboard or mouse events. The application use recognition algorithms to map handwritten data to a program then processes the virtual keyboard or mouse prototype. As long as the user writes like the prototypes, the events as if the events were enter from the keyboard or handwritten data is successfully recognized. Conversely, the OUS. more dissimilar the handwritten data and the prototype are, 60 In preferred embodiments, the methods automatically the more likely it is that the handwriting styles and legibility detect when the user wants to input handwritten data to a of the handwriting. For example, the handwritten word non-pen application. The data entry program monitors all “dear may be misrecognized as the word “clear depending pen down events. The user starts entering handwritten data the way the user writes a “d' and the prototypes for the by writing over a data entry area of the application window. letters “d,” “c, and “1. 65 When the data entry program detects a pen down within a Although some computer programs are written to accept data entry area of the application window, the data entry handwritten data, most computer programs are designed to program starts the collection and recognition process. When US 7,305, 130 B2 3 4 the user stops writing for a period of time, the data entry pen writes the point onto the tablet 112 where the pen 111 program assumes that the user has finished the handwritten made contact. With an appropriate sampling rate, the com data entry. puter 110 inks multiple contact points as the user moves the In a preferred embodiment, the data entry program inputs pen 111 across the tablet 112. Thus, the computer system gestures from the user. The data entry program converts a 5 allows the user to write data onto the tablet 112. Other recognized gesture to keyboard or mouse events or other computer system configurations are acceptable to implement standard messages that will effect the function of the gesture. the methods of the present invention. For example, the entire The data entry program then sends these events to the system can be incorporated in a tablet shaped device. The application program as virtual events. When the application tablet surface is the input grid an display surface. The tablet program receives the virtual events, it processes them as if 10 shaped device contains the logical equivalent of computer they came from the keyboard or mouse. 110 without the display, keyboard, or disk drives. It is an object of the present invention to provide a method The methods of the present invention allow the user to and system for allowing non-pen application programs to input data from a tablet into an application program that is receive handwritten data entry. not designed to receive handwritten data (a non-pen appli It is another object of the present invention to provide a 15 cation). Although the methods are described as implemented method and system for using an invisible window to input with the Windows system by Microsoft, one skilled in the art handwritten data and send the recognized symbols to an would recognize that the methods can be implemented on application program. other windowing systems and some methods can be imple It is another object of the present invention to provide a mented on non-windowing systems. FIG. 2A through 2C method and system for detecting when a non-pen application 20 show a typical user interface employing the methods of the program is in a data entry mode. present invention. FIG. 2A shows a typical display 200 of It is another object of the present invention provide a the word processing package Microsoft Word for Windows. method and system for allowing a non-pen application Display 200 is displayed on the display surface 122. A user program to process gesture data entry. of Word typically enters text into data entry window 201 These and other objects will become apparent as the 25 through a keyboard. Text 202 comprises sample text data. invention is more fully described below. Insertion pointer 203 indicates the position at which newly entered text will be inserted into the word processing BRIEF DESCRIPTION OF THE DRAWINGS document. The cursor 207 is an I-beam shape. FIG. 23 shows the display surface 122 after the user has handwritten FIG. 1 shows a computer system configured to implement 30 the text “The End 204, which is to be inserted into the the methods of this invention. document. With the pen 111, the user writes the text “The FIG. 2A shows a typical display of the word processing End 204 onto the tablet 112. When the user completes package Microsoft Word for Windows. writing the text, the text is inserted as shown in FIG. 2C. A FIG. 2B shows the display surface after the user has recognizer converts the handwritten text 204 to the ASCII handwritten text. 35 format. The recognized text is then sent to the word pro FIG. 2C shows the display surface after the handwritten cessor for insertion at the insertion pointer 203. The inser text is recognized and received by the word processor. tion pointer is repositioned at the end of the newly entered FIG. 3 shows components of a preferred embodiment of text “The End 205. The handwritten text 204 is removed the present invention. from the display 200 before the text is sent to the word FIGS. 4A-B is a flow diagram of the processing in a 40 processor. preferred embodiment of the present invention. In a preferred embodiment of the present invention, an FIG. 5 is a partial flow diagram of the Pen Palette “invisible' window is used to capture handwritten data. The initialization routine. handwritten data is recognized, and the recognized data is FIG. 6 is a flow diagram of the function DWMouseHook. sent to a non-pen application. An invisible window is a FIG. 7 is a flow diagram of the window procedure for the 45 window with no frame, title bar, et. When an invisible Pen Palette application. window is displayed, the data in the overlapped windows is FIG. 8 is a flow diagram of the Process Writing function. not overwritten. Rather, the overlapped windows show FIG. 9 is a flow diagram of the window procedure for the through the invisible window. Thus, it is not possible to invisible window. visually detect an invisible window until data is sent to the 50 invisible window. For example, in FIG. 2A once a pen down DETAILED DESCRIPTION OF THE (contact with the input grid) is detected the word processing INVENTION window is overlapped by an invisible window that covers the entire word processing window. FIG. 2B shows the effect FIG. 1 shows a computer system configured to implement of handwritten data sent to the invisible window. The the methods of the present invention. The computer system 55 handwritten data is the only indication that an invisible comprises a computer 110, a pen 111, a digitizing tablet 112, window exists. To the user, it seems as though the word and a connecting cable 113. The tablet 112 comprises input processing window is accepting the handwritten data grid 121 and the display surface 122. The input grid 121 directly. However, the word processor is a non-pen appli detects contact of the pen 111 with the tablet 112. If contact cation. The handwritten data is actually sent to the invisible is detected, then the tablet 112 sends the coordinates of the 60 window, which is created by a handwriting data entry contact point to the computer 110 through the connecting program. cable 113. The tablet 112 periodically samples the input grid In a preferred embodiment, the start of handwriting is 121. Upon receiving the coordinates, the computer 110 detected by the pen contracting the input grid (pen down). “inks' the contact point, that is, the computer 110 sends data The end of handwriting is detected when the pen is not in to the tablet 112 to turn on a point on the display surface 122 65 contact with the input grid (pen up) for a period of time, that corresponds to the contact point. The display Surface which may be predetermined or may vary depending on the 122 is a standard bitmap display. To a user it looks as if the pace of user handwriting. Alternately, the start and stop of US 7,305,130 B2 5 6 handwriting can be detected by selection of a menu item in and overlaps the application window. However, it may also a visible window of the data entry program. fill the entire display surface or only a portion of the FIG. 3 shows components of a preferred embodiment of application window. The processing in block 414 corre the present invention. Windows 301 includes a recognition sponds to the function Process Writing (explained below) context (RC) manager 302. The RC manager 302 comprises that is called by Pen Palette. In block 404, the Pen Palette the functionality of Windows 301 that supports handwritten window creates the invisible window. The creation process data entry. The RC manager 302 controls the inputting of the saves the bitmap for the portion of the display that the raw handwritten data from the input grid drive 307 and the invisible window overlaps. This creation process initializes inking of the display Surface through the display Surface data internally within Windows. Because the window is driver 306. The recognizer 303, which is a function of the 10 invisible, the display is not affected. The saved bitmap is RC manager 302, receives raw handwritten data from the used to restore the display when the invisible window is RC manager, recognizes the raw data as symbols, and closed. In block 405, the Pen Palette window invokes the returns the recognized symbols. The methods of the present function Recognize. The function Recognize inputs the invention are not dependent on any specific recognition handwritten data, sends the corresponding ink to the display technique. Rather, the methods of the present invention are 15 Surface, and converts the handwritten data into recognized independent of the recognition technique. The mouse driver symbols. The function Recognize detects when the user 304 and the keyboard driver 305 are standard device drivers. steps writing. When the function Recognize completes, it The display surface driver 306 receives data from the RC sends an WM RCRESULT message to the invisible win manager 302 and displays the data on the tablet. The input dow. The WM RCRESULT message indicates that the grid driver 307 receives the coordinate data from the input recognition is complete. In block 406, the invisible window grid and sends the data to the RC manager 302. The mouse saves the recognized symbols and returns control to the Pen hook 308 receives every mouse message that Windows 301 Palette window. In block 407, the Pen Palette window sends generates. The facility to hook mouse events is a standard a WM CLOSE message to the invisible window. In block feature of Windows. The mouse hook 308 detects when the 408, the invisible window processes the WM CLOSE mes user wants to enter handwritten data into a non-pen appli 25 sage. When the invisible window is closed, the overlapped cation 310. This detection enables the automatic handwrit bitmap that was saved when the invisible window was ing mode. In a preferred embodiment, Windows 301 gen created is restored. This restoration effectively erases the erates a mouse left button down message when a pen down handwritten data on the display. Control then passes back to occurs. This allows Windows 301 and application programs the Pen Palette window. In block 409, the function Process to treat pen down like a mouse event. Windows 301 can 30 Writing, which is invoked by the Pen Palette window, sends determine whether a mouse event was generated as a result a WM RCRESULT message to the Pen Palette window. of a mouse or a pen. Pen Palette 309 is the application that This message indicates to the Pen Palette window that controls the entry of the handwritten data and sending of the recognition is complete and the recognized symbols are recognized symbols to application 310. In operation, the available. In block 410, the Pen Palette window converts the mouse hock 308 sends a message to Pen Palette 309 when 35 recognized symbols to virtual events. It converts gestures to it detects that the user wants to input handwritten data (enter their mouse or keyboard equivalent virtual events or stan automatic writing mode). Pen Palette 309 receives the dard system messages and characters to their keyboard handwritten data, converts it to recognized symbols, and equivalent virtual events. Block 410 then sends these virtual then sends the recognized symbols to Windows 301 as events to Windows. Block 410 also returns control to the virtual keyboard or mouse events. A virtual event is not 40 WM INVOKEVKB invocation of the Pen Palette window generated by the physical device, but rather is generated by in block 411 to complete that invocation. In block 412, a program and sent to Windows 301 to be treated as if the windows receives virtual events and sends messages corre physical event occurred. For example, a virtual keyboard sponding to the events to the application window. The event occurs when a letter is recognized as part of the messages include the keyboard event and mouse event handwritten data and Pen Palette 309 sends the letter to 45 messages. In block 413, the application window receives Windows 301 to be treated as if it were entered from the and processes the messages as if the events were actually keyboard. Windows can determine whether an event is received from the keyboard or mouse. In the example of virtual or physical. FIGS. 2A through 2C, the word processing program inserts FIG. 4 is a flow diagram of the processing in a preferred the text characters corresponding to the handwritten data embodiment of the present invention. The process begins 50 into the document. when the mouse hook receives a WM LBUTTONDOWN FIGS. 5 through 9 are flow charts showing a preferred message from Windows. In block 401, the mouse hook embodiment of the present invention. Appendix A contains enables the automatic handwriting mode if the message is the “Microsoft Windows Graphical Environment for Pen generated as a result of a pen event and if the cursor is an Computing SDK—Guide to Programming manual. This I-beam share. In block 402, if the automatic writing mode is 55 manual explains in detail the application programming inter enabled, then the mouse hook sends a WM INVOKEVKB face for the Windows pen-based system. message to the Pen Palette window and sends an indication FIG. 5 is a partial flow diagram of the Pen Palette to Windows not to process the left button down mouse initialization routine. This initialization routine installs the message. If the automatic writing mode is not enabled, then mouse hook, replaces the I-beam bitmap with an automatic the mouse hook sends an indication to Windows to process 60 writing bitmap, and registers the class for the invisible the message. The WM INVOKEVKB message indicates window. The Pen Palette application installs a mouse hook that the Pen Palette window is to start input of handwritten to trap messages corresponding to left button down mouse data through an invisible window and to send the recognized events. When a mouse hook is installed to trap mouse symbols to the application. In block 403, when the WM IN messages, Windows invokes the mouse hook to process each VOKEVKB is received, the Pen Palette window determines 65 mouse message. When the mouse hook returns to Windows, the size and position characteristics of the application win it indicates whether Windows should proceed with normal dow. The invisible window is preferably slightly larger than processing of the mouse message or not process the mouse US 7,305,130 B2 7 8 message. Referring to FIG. 5, in block 501, the routine Some applications may use a different cursor display con installs the function DWMouseHook as a mouse hook. In vention. In a preferred embodiment, the mouse hook can block 502, the routine replaces the bitmap for the I-beam maintain a list of cursor handles that indicate data entry. cursor with a bitmap that indicates automatic writing is Detection of the display of cursors corresponding to the enabled. In a preferred embodiment, the routine accesses the handles in the list can be used to enable automatic hand handle of the I-beam cursor. The routine makes a copy of the writing. In a preferred embodiment, the list of handles can I-beam bitmap and then copies the automatic writing bitmap be mapped to the application programs that use a specific over the original I-beam bitmap. Whenever the application cursor to indicate data entry. The list can be updated dynami intends to display the I-beam cursor, the automatic writing cally to accommodate new applications that do not follow cursor is displayed instead to indicate that handwritten data 10 the I-beam convention. In block 603, the function posts a can be received. In block 503, the class for the invisible WM INVOKEVKB message to the Pen Palette window. window is registered with Windows. The class specifies a The handle of the application window is sent with the null background brush and the save bits flag. The null message. The function then returns to Windows with a one background brush indicates that the display area which the value. invisible window overlaps is not erased. The save bits flag 15 FIG. 7 is a flow diagram of the window procedure for the indicates that the bitmap for the display area which the Pen Palette application. This procedure processes the invisible window overlaps is saved so that the bitmap can be WM INVOKEVKB message that is posted by function restored when the invisible window is closed. DWMouseHook and the WM RCRESULT message that is FIG. 6 is a flow diagram of the function DWMouseHook. sent by the recognizer. With the WM INVOKEVKB mes Windows sends all mouse messages to the function sage, this procedure receives as parameters the handle of the DWMouseHook. The function DWMouseHook traps out Pen Palette window and the handle of the application left button down messages generated by a pen when the window. In block 701, if the message is not equal to cursor indicates the I-beam. In a preferred embodiment the WM INVOKEVKB, then the procedure continues at block bitmap for the I-beam cursor is replaced by the automatic 710, else the procedure continues at block 702. In block 702, writing bitmap. When a trap occurs, the function posts a 25 the procedure calculates the size of the invisible window. message to the Pen Palette window indicating that it should The size of the invisible window is preferably slightly larger start the automatic writing recognition. A left button down than the application window. In block 703, the procedure occurs when the pen contacts the tablet. The function initializes parameters for the recognition process. The rec DWMouseHook traps only left button down messages gen ognition parameters typically include bounding rectangle for erated by the pen. The function is called with a mouse 30 inking, handle of window to send results, ink width, list of message and the handle of the application window as allowable symbols, etc. In block 704, the procedure invokes parameters. The function returns a Zero value when it does function ProcessWriting (described below). The procedure not process the message, which indicates that Windows then returns. should process the message, and a non-Zero value when it In block 710, if the message is not equal to WM RCRE does process the messages. Referring to FIG. 6, in block 35 SULT, then the procedure processes other messages as 601, if the message is WM LBUTTONDOWN and it cor indicated by the ellipsis, else the procedure continues at responds to a pen event, then the function continues at block block 711. In block 711, the procedure converts any gestures 602, else the function returns with a zero value. In block 602, in the recognized symbols to equivalent events or messages. if the cursor indicates the I-beam, then a pen down has been The methods of the present invention convert gestures into detected (left mouse button message generated) and the 40 corresponding virtual events or messages. For example, the cursor is over a data entry window and the function contin caret gesture, which means insert, is converted to a mouse ues at block 603, else the function returns with a Zero value. movement event indicating the point of the caret and a left The detection of the I-beam allows the methods of the mouse button down event. When an application, such as a present invention to automatically detect that a user is word processor, receives these events it moves the insertion starting to enter handwritten data. In operation, when Win 45 pointer to the caret and is ready to insert data entry. In dows detects a mouse movement it sends mouse movement another example, a copy selection gestures, which means messages to the application. The mouse hook does not trap copy the selection to a clipboard, is converted to keyboard mouse movement messages. When the application detects events. Such as control-insert or a standard system message, that the mouse has moved over a text data entry field, the Such as WM COPY in Windows. One skilled in the art will application requests Windows to set the cursor to the I-beam 50 recognize that other gestures can be similarly converted to by passing the handle to the I-beam bitmap. Windows then virtual events for processing by non-pen applications. In displays the bitmap indicated by the handle. The use of the block 712, the procedure posts virtual mouse and keyboard I-beam shape to indicate that the application is ready to events corresponding to the recognized symbols to Windows receive data is standard programming practice in Windows. and returns. In a preferred embodiment, the methods of the present 55 FIG. 8 is a flow diagram of the Process Writing function. invention detect that a user is starting to enter handwritten The function Process Writing controls inputting raw hand data as follows. Windows detects a pen down and sends a written data, inking the display Surface, and converting the mouse movement message to the application. If the pen handwritten data to recognized symbols. The function Pro down occurred over a text data entry field, the application cessWriting is called with two parameters: the handle of the responds to the mouse movement message by setting the 60 Pen Palette window and the parameters for the recognition cursor to indicate the I-beam. Windows then sends the left process. In a preferred embodiment, function Process Writ button down mouse message, which the mouse hook traps. ing can be called by a pen-aware application to facilitate pen The mouse hook ensures that the message was generated as data entry through the use of an invisible window. In block a result of a pen event and checks the cursor. If the cursor 801, the function creates the invisible window. The invisible handle indicates the I-beam, then the application is ready to 65 window is created as the topmost window and with no receive data and entry of the handwritten data can proceed. frame, title bar, or scroll bars to maintain invisibility. In Although the I-beam cursor is used to indicate data entry, block 802, the function sets the invisible window position on US 7,305,130 B2 9 10 the display surface. In block 803, the function updates the fying a window having attributes Such that graphics beneath invisible window. Although there is no visible effect of the Superimposed window at least in part show through the updating the invisible window, Windows initializes internal Superimposed window. data structures. In block 804, the function invokes the 4. The computer-readable medium of claim 2 having function Recognize to input data using the invisible window further computer-executable instructions comprising, giving and recognize the data. The function Recognize sends the focus to the Superimposed window upon detecting that the recognized symbols along with the WM RCRESULT mes user wants to input the set of handwritten data. sage to the invisible window. In block 805, the function 5. The computer-readable medium of claim 2 having sends the WM CLOSE message to the invisible window. further computer-executable instructions comprising, clos When the invisible window is closed, bitmaps for the display 10 ing the Superimposed window and restoring graphics area that the invisible window overlapped are restored. In beneath the Superimposed window to a previous state. block 806, the function sends the WM RCRESULT mes 6. The computer-readable medium of claim 1 wherein sage to the Pen Palette window. The function then returns. detecting that the user wants to input a set of handwritten FIG. 9 is a flow diagram of the window procedure for the data comprises detecting when the Software program is in a invisible window. The procedure processes two messages: 15 data entry mode. WM MOUSEACTIVATE and WM RCRESULT. This pro 7. The computer-readable medium of claim 6 wherein cedure ensures that the invisible window is not given the detecting that the Software program is in the data entry mode focus and saves the recognized symbols. The window that comprises evaluating a cursor of the Software program. has the focus is the window that receives the keyboard 8. The computer-readable medium of claim 6 wherein entries. In block 901, if the WMMOUSEACTIVATE mes detecting that the Software program is in the data entry mode sage is received, then the procedure returns with MA NO comprises determining that the pen down occurred over a ACTIVATE, else the procedure continues at block 902. The text data entry field of the software program. WM MOUSEACTIVATE message indicates that the invis 9. The computer-readable medium of claim 1 wherein ible window is to be made the focus. Under normal pro detecting that the user wants to input a set of handwritten cessing, the title bar of the application losing the focus 25 data comprises monitoring for a pen down event. would be changed to indicate that it no longer has the focus. 10. The computer-readable medium of claim 1 having However, this procedure Suppresses the normal processing further computer-executable instructions comprising, dis by returning the MA NOACTIVATE message. The net playing a writing cursor to indicate that handwritten data can effect is not only that the window is invisible, but also that be received. the application window does not change because the focus 30 11. The computer-readable medium of claim 1 wherein does not change. In block 902, if the WM RCRESULT detecting the end of the input of that set of handwritten data message is received, then the recognized symbols are to be comprises detecting when the user has stopped entering saved and the procedure continues at block 903, else the handwritten input for a period of time. procedure continues with the standard default message pro 12. The computer-readable medium of claim 11 wherein cessing as indicated by the ellipsis. In block 903, the 35 detecting when the user has stopped entering handwritten procedure saves the recognized symbols so that they can be input for the period of time comprises, establishing a pre retrieved by the Pen Palette window and the procedure determined time as the period of time, or establishing the returns. period of time based on a pace of user handwriting. Although the present invention has been described in 13. The computer-readable medium of claim 1 wherein terms of preferred embodiments, it is not intended that the 40 detecting the end of the input of that set of handwritten data invention be limited to these embodiments. Modifications comprises detecting an explicit user command. within the spirit of the invention will be apparent to those 14. The computer-readable medium of claim 1 wherein skilled in the art. The scope of the present invention is converting the set of handwritten data into the virtual event defined by the claims that follow. comprises converting a gesture to a mouse or keyboard 45 equivalent virtual event, or converting a system message or What is claimed is: character to a keyboard equivalent virtual event. 1. A computer-readable medium having stored computer 15. The computer-readable medium of claim 1 wherein executable instructions which, when executed by a comput providing the virtual event to the Software program com ing system, implement a method comprising: prises providing data corresponding to the virtual event to an providing an area to receive handwritten data; 50 operating system component, and sending the virtual event detecting that a user wants to input a set of handwritten from the operating system component to the Software pro data directed towards a Software program; gram. receiving the set of handwritten data; 16. In a computing device, a system comprising: detecting the end of the input of that set of handwritten 55 means for detecting that a user wants to input a set of data; handwritten data directed towards a first program; converting the set of handwritten data into a virtual event means for receiving the set of handwritten data at a second corresponding to a keyboard event, mouse event or program that is different from the first program; other system message; and means for communicating the set of handwritten data providing the virtual event to the Software program. 60 from the second program to a recognizer and receiving 2. The computer-readable medium of claim 1 wherein recognition data corresponding to the handwritten data providing the area to receive handwritten data comprises from the recognizer, and Superimposing a Superimposed window over at least part of means for sending a virtual event corresponding to the a software program window corresponding to the software recognition data to the first Software program. program. 65 17. The system of claim 16 wherein the means for 3. The computer-readable medium of claim 2 wherein receiving the set of handwritten data comprises a window Superimposing the Superimposed window comprises speci corresponding to the second program that receives at least US 7,305,130 B2 11 12 one handwritten gesture, symbol, System message or char sponding to the recognition data from the first program, acter as the set of handwritten data. the operating system component further coupled to the 18. The system of claim 16 wherein the means for second program to provide a virtual event correspond detecting when the user wants to input a set of handwritten ing to the recognition data or the data corresponding to data comprises means for detecting when the software the recognition data to the second program. program is in a data entry mode, means for evaluating a cursor of the Software program, or means for determining 20. The system of claim 19 wherein the handwritten input that the pen down occurred over a text data entry field of the comprises a handwritten gesture, a symbol, a system mes Software program, or any combination thereof. sage or a character, and wherein the virtual event comprises 19. In a computing device, a system comprising: 10 at least one mouse equivalent event, at least one keyboard a first program having a window that receives handwritten equivalent event, at least one character equivalent event or input directed towards a second program that is ready at least one system message equivalent event, or any com to receive non-handwritten input; bination of at least one mouse equivalent event, at least one a recognizer coupled to the first program that provides keyboard equivalent event, at least one character equivalent recognition data corresponding to the handwritten 15 event or at least one system message equivalent event. input; an operating system component coupled to the first pro gram that receives the recognition data or data corre