(12) Patent Application Publication (10) Pub. No.: US 2007/0211071 A1 Slotznick Et Al
Total Page:16
File Type:pdf, Size:1020Kb
US 20070211071A1 (19) United States (12) Patent Application Publication (10) Pub. No.: US 2007/0211071 A1 Slotznick et al. (43) Pub. Date: Sep. 13, 2007 (54) METHOD AND APPARATUS FOR (52) U.S. Cl. .............................................................. 34.5/594 INTERACTING WITH A VISUALLY DISPLAYED DOCUMENT ON A SCREEN (57) ABSTRACT READER User interaction of a visually displayed document is pro vided via a graphical user interface (GUI). The document (76) Inventors: Benjamin Slotznick, Mt. Gretna, PA includes, and is parsed into, a plurality of text-based gram (US); Stephen C. Sheetz, Lititz, PA matical units. An input device modality is selected from a (US) plurality of input device modalities which determines the Correspondence Address: type of input device in which a user interacts with to make AKN GUMPSTRAUSS HAUER & FELD a selection. One or more grammatical units of the document L.L.P. are then selected using the selected type of input device. ONE COMMERCE SQUARE Each grammatical unit that is selected is read aloud to the 2005 MARKET STREET, SUITE 2200 user by loading the grammatical unit into a text-to-speech PHILADELPHIA, PA 19103 (US) engine. The text of the grammatical unit is thereby auto matically spoken. Furthermore, a Switching modality is (21) Appl. No.: 11/642,247 selected from a plurality of switching modalities. The (22) Filed: Dec. 20, 2006 Switching modality determines the manner in which one or more Switches are used to make a selection. Using the Related U.S. Application Data selected Switching modality, a user steps through at least Some of the grammatical units in an ordered manner by (60) Provisional application No. 60/751,855, filed on Dec. physically activating one or more Switches associated with 20, 2005. the GUI. Each activation steps through one grammatical Publication Classification unit. Each grammatical unit that is stepped through is read aloud by loading the grammatical unit into a text-to-speech (51) Int. C. engine, thereby causing the text of the grammatical unit to G9G 5/02 (2006.01) be automatically spoken. READING MODE WAIT FOR NEXT INPUT 1502 BUTTON IS PUSHED 1503 IS THIS THE STEP HIGHILIGHT AND READ 50s FORWARD THENEXT SENTENCE BUTTON OR OBJECT 1507 IS THIS THE RE-READ SENTENCE 509 REPEAT STEP OR OBJECT BUTTON 1511 HIGHILIGHT AND READ PREVIOUS SENTENCE OR OBJECT 1515 WAIT (1502) ACTIVATE LINK OR CLICKABLE OBJECT 1521 - Y - GET NEW CHANGE PAGE 1523 1ss THIS IS THE TO CHANGEMODE HYPER w BUTTON LINK GOTO MODE START 1527 15O1 Patent Application Publication Sep. 13, 2007 Sheet 1 of 19 US 2007/0211071 A1 `ISIARISIS inSSAOOHd JLSHOO™ETRI LII HILIAASILRIAASTRIQINV HOHAGOOCIRIHIGOW SXINITCÍNWJLX£IL :RIVAALHOS;) 3ÐV?g?AGSHVä i|JUNGIITOYHOEIST). Patent Application Publication Sep. 13, 2007 Sheet 2 of 19 US 2007/0211071 A1 {{TOCIOWN zOIH… |£I{ºshaxvadsH?AO| „,HO33?S-OL-LXILHLIAAºbvà83A(TVRI |JXEJLH8OCI?L?VIJSCINQOS| (ISIHOLOSISOOHOLIZ&IXRL IZIWORTH Patent Application Publication Sep. 13, 2007 Sheet 4 of 19 US 2007/0211071 A1 (LHVHOIHA) Patent Application Publication Sep. 13, 2007 Sheet 6 of 19 US 2007/0211071 A1 (LHVHOTHA) & Patent Application Publication Sep. 13, 2007 Sheet 7 of 19 US 2007/0211071 A1 Z'9ICH (LHVHOTHd) OQ +) Patent Application Publication Sep. 13, 2007 Sheet 8 of 19 US 2007/0211071 A1 89ICH (LHVHOTHd) Patent Application Publication Sep. 13, 2007 Sheet 9 of 19 US 2007/0211071 A1 YA Patent Application Publication Sep. 13, 2007 Sheet 10 of 19 US 2007/0211071 A1 0['9ICH (LHVHOTHà) ? Patent Application Publication Sep. 13, 2007 Sheet 11 of 19 US 2007/0211071 A1 II'9IAI (LHVHOTHA) Patent Application Publication Sep. 13, 2007 Sheet 12 of 19 US 2007/0211071 A1 ZI(9IAI (LXIVHOTHA) Patent Application Publication Sep. 13, 2007 Sheet 13 of 19 US 2007/0211071 A1 9.I’9ICI (LHVHOTHA). Patent Application Publication Sep. 13, 2007 Sheet 14 of 19 US 2007/0211071 A1 SOME KEYASSIGNMENTS FOR USING STANDARD KEYBOARD Fingers to be used Key Action Finger Left hand only: Tab change mode little finger 2 step backward ring finger 3 repeat step middle finger 4 step forward pointerfinger Space bar activate thumb Left hand plus: Tab change mode little finger one finger on 2 step backward ring finger the right hand 3 repeat step middle finger 4 step forward pointerfinger Enter activate right hand pointer Left plus two: 2 step backward ring finger two fingers on 3 repeat step middle finger the right hand 4. step forward pointer finger Enter activate right hand pointer Ctrl or Alt change mode right hand thumb Right hand only: Space bar activate thumb (zero) 0 step forward pointerfinger (minus) - repeat step middle finger (plus) + step backward ring finger \ and change mode little finger Right hand plus: Tab activate left hand pointer one finger on the left hand (zero) 0 step forward pointerfinger (minus) - repeat step middle finger (plus) + step backward ring finger “\ and change mode little finger Right hand plus: Tab activate left hand pointer two fingers on Ctrl or Alt change mode left hand thumb the left hand (zero) step forward pointer finger (minus) - repeat step middle finger (plus) + step backward ring finger Fig. 14 Patent Application Publication Sep. 13, 2007 Sheet 15 of 19 US 2007/0211071 A1 READING MODE WAIT FOR NEXT INPUT 1502 Fig. 15 BUTTON IS PUSHED 1503 IS HIGHILIGHT AND READ THIS THE STEP 505 FORWARD THE NEXT SENTENCE BUTTON OR OBJECT 1507 NO IS THIS THE RE-READ SENTENCE 509 REPEAT STEP OR OBJECT BUTTON? 1511 IS THIS THE STEPY. HIGHILIGHT AND READ 513 BACKWARD PREVIOUS SENTENCE OR OBJECT BUTTON 1515 GO TO NO WAIT (1502) ACTIVATE IS LINK OR THIS THE CLICKABLE 1517 ACTIVATE BUTTON? OBJECT 1521 CHANGE PAGE 1523 TO 1ss THIS IS THE HYPER CHANGE MODE BUTTON LINK MODE 1527 Patent Application Publication Sep. 13, 2007 Sheet 16 of 19 US 2007/0211071 A1 HYPERLINK MODE WAIT FOR NEXT INPUT 1601 Fig. 16 BUTTON IS PUSHED 1603 IS HIGHILIGHT AND READ THIS THE STEP 605 THE NEXT HYPERLINK FORWARD BUTTON OR CLICKABLE OBJECT 1607 NO IS THIS THE RE-READ HYPERLINK 1609 REPEAT STEP OR CLICKABLE BUTTON? OBJECT 1611 IS HIGHILIGHT AND READ THIS THE STEP PREVIOUS HYPERLINK 1613 BACKWARD OR CLICKABLE BUTTON? OBJECT 1615 No ACTIVATE IS LINK OR THIS THE CLICKABLE 1617 ACTIVATE BUTTON OBJECT 1621 GET NEW CHANGE PAGE 1623 15 THIS IS THE TO CHANGE MODE NAVIGA BUTTON TION MODE 1627 Patent Application Publication Sep. 13, 2007 Sheet 17 of 19 US 2007/0211071 A1 NAVIGATION MODE d WAIT FOR NEXT INPUT 1701 F 19 1 7 BUTTON IS PUSHED 1703 IS THIS THE STEP HIGHILIGHT AND READ 05 FORWARD NEXT NAVIGATION BUTTON BUTTON OR TAB 1707 IS RE-READ THIS THE NAVIGATION 1709 REPEATSTEP BUTTON OR TAB BUTTON? HIGHILIGHT AND READ IS PREVIOUS THIS THE STEP NAVIGATION BUTTON 713 BACKWARD ORTAB 1715 BUTTON WAIT (1701) No IS ACTIVATE THIS THE NAVIGATION 1717 ACTIVATE BUTTON OR BUTTON TAB 1719 GET NEW CHANGE PAGE 1723 TO 1725 THIS IS THE READ CHANGE MODE ING BUTTON MODE 1727 Patent Application Publication Sep. 13, 2007 Sheet 18 of 19 US 2007/0211071 A1 [08] £18I11816081L081S081€081IL181S181|318 e6edq?AN?IdueS Patent Application Publication Sep. 13, 2007 Sheet 19 of 19 US 2007/0211071 A1 w 606I O you III8I608IL08S08£08ILI8ISI8||$18 O US 2007/021 1 071 A1 Sep. 13, 2007 METHOD AND APPARATUS FOR INTERACTING 0007 i. People who cannot read or who have difficulty WITH A VISUALLY DISPLAYED DOCUMENT ON reading, but who can hear and comprehend conversational A SCREEN READER speech. CROSS-REFERENCE TO RELATED 0008 ii. People who may have poor vision, but who can APPLICATIONS see high contrasts. 0001) This application claims the benefit of U.S. Provi 0009 iii. People with hand-motor limitations who can sional Application No. 60/751,855 filed Dec. 20, 2005 move and position a pointing device (such as a mouse) but entitled “User Interface for Stepping Through Functions of nonetheless may have a difficulty clicking on mouse buttons. a Screen Reader.” 0010 iv. People who may have learning disabilities, or cognitive disabilities (such as traumatic brain injury, mental COMPACT DISC APPENDIX retardation, or Alzheimer's disease) which make reading 0002 This patent application includes an Appendix on difficult. one compact disc having a file named appendix.txt, created 0011. However, there are people whose vision or manual on Dec. 19, 2006, and having a size of 36,864 bytes. The dexterity is even more limited than currently required for compact disc is incorporated by reference into the present Point-and-Read. Just as importantly, many disabilities are patent application. This compact disc appendix is identical progressive and increase with age, so that some people who in content to the compact disc appendix that was incorpo have the ability to use Point-and-Read may lose that ability rated by reference into U.S. Patent Application Publication as they age. The present invention is intended to extend No. 2002/0178007 (Slotznicket al.). some of the benefits of using a screen-reader like Point-and Read to Such people. COPYRIGHT NOTICE AND AUTHORIZATION 0012 (Most screen-readers, and much of assistive tech 0003 Portions of the documentation in this patent docu nology, focus on compensating for one physical disability, ment contain material that is subject to copyright protection. usually by relying upon other abilities and mental acuity. The copyright owner has no objection to the facsimile This approach does not help people who have multiple reproduction by anyone of the patent document or the patent disabilities, especially if one of their disabilities is cogni disclosure as it appears in the Patent and Trademark Office tive.) file or records, but otherwise reserves all copyright rights whatsoever.