US009 165478B2
(12) United States Patent (10) Patent No.: US 9,165,478 B2 Carro et al. (45) Date of Patent: Oct. 20, 2015
(54) SYSTEMAND METHOD TO ENABLE BLIND (56) References Cited PEOPLE TO HAVE ACCESS TO INFORMATION PRINTED ON A PHYSICAL U.S. PATENT DOCUMENTS DOCUMENT 4,644,339 A 2, 1987 Ruder 4,996,707 A 2/1991 O'Malley et al. (75) Inventors: Fernando Incertis Carro, Valencia 5,223,828 A 6/1993 McKiel, Jr. (ES); Sharon M. Trewin, 5,267,331 A 11, 1993 Swoff Croton-on-Hudson, NY (US) 5,287,102 A 2/1994 McKiel, Jr. 5,325,123 A 6/1994 Bettinardi (73) Assignee: International Business Machines 5,359,675 A 10, 1994 Swoff Corporation, Armonk, NY (US) 5,374,924 A 12/1994 McKiel, Jr. (Continued) (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 OTHER PUBLICATIONS U.S.C. 154(b) by 3629 days. "Talking Books SpeakVolumes”. Kendra Mayfield, Wired News, Jul. 13, 2000, 4 pgs. http://www.wired.com/news/gizmost (21) Appl. No.: 10/826,810 0,1452,375,00.html. (22) Filed: Apr. 15, 2004 (Continued) (65) Prior Publication Data Primary Examiner — Steven B. Theriault US 2005/0022.108A1 Jan. 27, 2005 (74) Attorney, Agent, or Firm — Schmeiser, Olsen & Watts, LLP: John Pivnichny (30) Foreign Application Priority Data Apr. 18, 2003 (EP) ...... O3368O35 (57) ABSTRACT (51) Int. C. A method and system for use in a user system, for accessing G06F 3/048 (2013.01) information related to a physical document. An electronic G09B 2L/00 (2006.01) copy of an existing physical document is identified and G06F 3/048. I (2013.01) located. The electronic copy of the physical document is an GO6F 3/OI (2006.01) exact replica of the physical document. One or more pages of GO6F 3/0488 (2013.01) the physical document are identified. A selected part of the GO6F 3/0484 (2013.01) physical document is identified using the position of points on the identified one or more pages of the physical document and (52) U.S. C. in response, data related to the selected part of the physical CPC ...... G09B 21/001 (2013.01); G06F 3/0481 document is retrieved from the electronic copy of the physical (2013.01); G06F 3/016 (2013.01); G06F document. The retrieved data is presented visually to a visu 3/04842 (2013.01); G06F 3/04886 (2013.01) ally impaired person or orally to a blind person on the user (58) Field of Classification Search CPC. G06F 3/0481; G06F 3/04886; G06F 3/016: system, which enables the visually impaired person to see or G06F 3/0488; G06F 3/04842 hear, respectively, the retrieved data. USPC ...... 715/702, 865 See application file for complete search history. 26 Claims, 12 Drawing Sheets
Web server (806)
Touf Physical document 802) 801) US 9,165.478 B2 Page 2
(56) References Cited 2002/0101447 A1* 8, 2002 Carro ...... 345,760 2002/015223.6 A1* 10, 2002 Incertis-Carro . 707/5O11 U.S. PATENT DOCUMENTS 2002fO156866 A1* 10, 2002 Schneider .... TO9.218 2003/01 17378 A1* 6/2003 Carro ...... 345,173 5,400,434 A 3, 1995 Pearson 2003/01 17379 A1* 6/2003 Carro ...... 345,173 5,461.416 A 10, 1995 Bettinardi 2003,019 1852 A1* 10, 2003 Incertis . TO9,232 5,561,736 A 10, 1996 Moore et al. 2004/0021648 A1* 2, 2004 Blume .. ... 345,179 5,586, 196 A * 12/1996 Sussman ...... 382,114 2004/0091842 A1* 5, 2004 Carro ...... 434/112 5,617,507 A 4/1997 Lee et al. 2005, 0028092 A1* 2, 2005 Carro .... T15,526 5,630,060 A 5/1997 Tang et al. 2006/0098899 A1* 5/2006 King et al...... 382,305 5,633,674. A * 5/1997 Trulaskeet al...... 348.63 5,634,084 A 5, 1997 MalSheen et al. OTHER PUBLICATIONS Sg A EE St. et al. “Paper is Still With us”, Philippa Jane Benson, The Journal of Elec 5715.412 A 2, 1998 Aritsuka et al. tronic Publishing, University of Michigan Press, 7 pgs. http://www. 5.841,979 A 11/1998 Schulhofetal. press.umich.edu/jep/07-02/benson0702.htm. 5,900,908 A 5, 1999 Kirkland “Electronic Journals: What Do Users Think of Them?', Dr. Cliff 5,960,448 A * 9/1999 Reichek et al...... T15,236 McKnight, Department of information and Library Studies, 6,018,710 A 1/2000 Wynblatt et al. Loughborough University, Loughborough, Leics LE11 3TU, UK, 7 6,084,556 A 7/2000 Zwern pgs. http: Scholar.google.com/scholar?q=casch.edXCCig2FZQJ: 6,085,161 A 7/2000 MacKenty et al. www.dlulis.ac.jp/ISDL97 pro . . . . 6,088,675 A * 7/2000 MacKenty et al...... 7041270 “A Comparison of Reading Paper and On-Line Documents”. 6,115,482 A * 9/2000 Sears et al...... 382,114 “Children with Visual Impairment and their Reading: A Comparison 6,127,990 A 10/2000 Zwern Between Reading From Paper and Computer Screen'. E. Kellami, G. 6,184,847 B1 2/2001 Fateh et al. Douglas, R. Long, School of Education, University of Birmingham. 6,289,304 B1 9, 2001 Grefenstette “Fourteen Users in Search of a Newspaper: The Effect of Expectation 6,308,151 B1 10/2001 Smith on Online Behaviour'. Ann Light, School of Cognitive and Comput 6,324,511 B1 * 1 1/2001 Kiraly et al...... TO4,260 6.731.315 B1* 5, 2004 Ma et al...... 715,856 ing Sciences, University of Sussex, CSRP 507. 6.731326 B1 5/2004 Bettinardi ...... 348.63 “The Integration of Low Vision Aids and Other Assistive Devices in 6,823, 184 B1 * 1 1/2004 Nelson ...... 455,418 Educational and Vocational Settings'. Matthew H. ATer, krista M. 6,940,491 B2 9, 2005 Carro Davis, O.D., Columbia Lighthouse for the Blind, Washington, DC. 6,980,202 B2 * 12/2005 Carro ...... 345,173 Bell Labs Projects. Multillingual Text-to-Speech Systems, 3 pgs. 7,000, 189 B2* 2/2006 Dutta et al. . 71.5/727 http://www.bell-labs.com/project/tts/index.html. 7,013,279 B1* 3/2006 Nelson ... 704/270 Bell Labs Spanish Text-to-Speech Systems, 1 pg. http://www.bell 7,027,038 B2 4/2006 Carro. 345,173 labs.com/project/tts/spanish.html. 7,181,692 B2 2/2007 Siegel ...... 715/727 IBM TDB; Light pen to use with a transparent electro-luminiscent 7, 194411 B2 3/2007 Slotznicket al...... 704/271 display; IP.com No.: IPCOMO00016405D; Original Publication 7.212,3187,472.338 B2 * 12,5/2007 2008 CarroBradbery ...... 358/474 Dateate NOW.Nov. 5u,30, 2002;ZUU2; IP.comIP Electronic Publication JunJun. 21.21, 2003:ZUU5; 3 7473670 B2 1/2009 Takahashi et al...... 505/470 Pages. http://www.ip.com/pubview/IPCOMO00016405D. 2001/0056435 Al 12/2001 Carro ...... 707 500.1 Patent application for Application No. 14/696,815, filed Apr. 27, 2002/004787.0 A1 4, 2002 Carro 2015. 2002/0069240 A1* 6/2002 Berk ...... TO9,203 2002/0087598 A1* 7/2002 Carro ...... 707/513 * cited by examiner U.S. Patent Oct. 20, 2015 Sheet 1 of 12 US 9,165.478 B2
Web servers still
Web server
EEE (1 O6)
www.virgin.net INTERNET NETWORK Visually impaired user
User Workstation
Touch foil Physical document
(102) (101) Early British Kingdoms David Nash Ford
reader Document URL barcode (103) (107) U.S. Patent Oct. 20, 2015 Sheet 2 of 12 US 9,165.478 B2
Page number codes are (205)
located on Af. bottom left and Early British
Kingdoms, by
bottom right David Nash corners of g d aS
pages ... OO.
Ser Workstation (204)
Touch foil
(202) Physical document (201)
Early British Kingdoms David Nash Ford
C. Celtic World Ltd, Bigfield, Berkshire, UK (1999) Barcode reader tip/?www.virgin.net/ 17030 (203)
Document URL
barcode (207)
FIG. 2 U.S. Patent Oct. 20, 2015 Sheet 3 of 12 US 9,165.478 B2
bookmark (300)
Arthur the Myth Afallach, God of the Underworld Some people believe that King Arthur is so inedricably tied up in Celtic Afallach was the son of Lludd Llaw Ereint (the Silver-Handed). Mytholigy that he must, in origin, have been, not a man at all, but a God He was one of the Celtic gods of the Underworld He ruled Like so many other characters featured in the Malingion, Arthur in his Avalon where he lived with his daughter, Modron, and her nine earliest form, appears almost entirely mythical He and his companions sisters, Avalon was like the Celtic heaven, a peaceful island far have superhuman strength and abilities, and consort with giants and other away where apples grew and after which it became named. It is, mythological creatures. of course, best known as the place where the High-King Arthwyr in the early Welsh poem "Preiddy Annwfn', Arthur visits the Celtic was taken after he was fataly wounded at the Battle of Camlann. Underworld, Annwfn, Even in Geoffrey and Malory, upon being fataly Afallach himself appears in Arthurian legends as King Evelake. wounded in battle, Arthur is carried to the mystical Avalon, apparently the Underworld home of the Celtic god, Afallach. Many gends around the country attest to Arthur's immortality for he ships in one of mumerous The name Arthur itself appears to drive from the Celtic word taves waiting to return and lead his people, The nate Arthur itself Art, meaning "bear". CouldArthur, like so many other Celtic gods, appears to derive from the Celtic word Art, meaning "hear". Could Arthur, be merely a personification of the many revered animals of the like so many other Celtic gods, be merely a personification of the many wild? Later to become humanized like Beli Mawr or Bran the 8. revered animals of the will? Later to become humanized like Beli Mawr or Blessed he constellation of Ursa Major or the Great Bear is, Brait the Blessed. The constellation of Ursa Major or the Great Bear is, after all, sometimes known as Arthur's Wain. There was indeed after all, sometimes known as Arthur's Wain, Ikere was indeed a Celtic a Celtic Bear-God, Artio, worshipped in Switzerland and around Bear-God, Ario, worshipped in Switzerland and around Trier, but she was, Trier, but she was, in fact, a goddess and there is no trace of her in fact, a goddess and there is no trace of her in Britain. Some theorists in Britain Some theorists claim Arthur was a late addition to the Celtic pantheon during a resurgence in pagan worship, or - 7
Page number barcode (301)
FG. 3 U.S. Patent Oct. 20, 2015 Sheet 4 of 12 US 9,165.478 B2
(406)
User WorkStation (404) Physical document (401)
Arthur the Myth Afallad, God of the Underworld sons pp: kiwia Kag Arthur is so incarically idy in Clt
Afallad was dir son of Likud Law Erint (ikr Siker-Handed). Mydoley that must, in origin, hawkern, riot a man at , but a God
He was one of the Cliod of the Underwr. He ruled Like so any other dharacters featured in Y Masingier, r in his
Avalan where he lived with his drighter, Madran, and her fine Parliest form, appears ahnast fntirely mythical He and kicompanions sisters, Avalor ws like the Cellir leaver, a pearful island far
law superhanan strength and abilitics, and ?ortson williants and other away where apples grew and gfier with it became named it is, nylogical ratures, gfroars, best known as the plane where the High-King Arkwyr
was takingfire was fataly wunded at the Battle of Camlam. Afallad imself appears in Arthurian lends as King Felike
in the early Wikh rom "Prilla Amy." Arlf visis the ch Underwri Aurf. Even by Cigfrty and , ran heig fataly E. Underworld bone fake Caligod Apalach May gends around the coatyalist to Arthur's immortality for be slips in a framerous the nitri Arder itself appears to drive from the Chie wrd ave wing to return and lead his people, Iliam Arbor itself Art, meaning "hear" Cullarthur, like so many other Chigol, he merely aptsarification of the may rered artial of the
agears to driw from the Calic wrdar, manig"ear", Cold Arkwr, like so many other Celtic gol, he mirtly a will? Later to laurukumanized like Beli Mawr or Bran the Bexced. It constellation of Ursa Major or the Great Bari, after all, sometimes known as Arthur's air. Irr was inded a Calie Bear-God, Artie, wrshipped in Switzerland and around sfirral, sometimes known as Arthur's Bar-Ga, Ari, wrilipped in Swtailand and around Irie, hit she was, Trier, kut she wr, in far, a godless and there is no trace firr in Britain same theorists ?laim Ardar was a lair addition to the ific,slain Artiar eds was and a late let addition in tary to di Calipantheon Brita's during horis a resiegence Cli pandron during a riskgrnie in pagan wrip, or orship, or possiyagdiglero, the fiprigsfahanan.
Touch foil (402) Page bar code Barcode reader (405) (403)
FIG. 4 U.S. Patent Oct. 20, 2015 Sheet 5 of 12 US 9,165.478 B2
Ser WorkStation (504)
Physical document (501)
h, God of the Underworld was dison of Ludlaw Firin (the Silver-Harded). one of the Celtic golk of the Underworld. He ruled were is lived with his large, Modiva, and her ring alar was like the Clir heater, a peaceful island far
rre applesgirw and gfier with it became named. It is, best known as the place where the High-Rig Arthyr rigfier was fataly would at the Battle of Caitlam. initif appears in Arthurian grid as Kirg Ewlake.
Arthur itself appears to drive from the Cali word ring "ear'. CouldArthur, like so many othy Cligged, a persarification of the trary mere animals of the aler formesarianizedikt Beli Mawr or Brastr
arcane anariedl Bei Mawr or the constitution f Ursa Mayor or the Great Bear is,
fUrsa Major or de Creat Bears sorcetimes known as Arthur's air. Tere was ided 's Fair, irre was indeeds Clic
in Sate avrists claim Arthur was a late addition to the on Aurig a restigence inpgar worship, or - A - Barcode reader
(503) Touch foil (502)
FIG. 5 U.S. Patent Oct. 20, 2015 Sheet 6 of 12 US 9,165.478 B2
, Document: http://ww.virgin.net/170301/ User WorkStation Page: 16 (604) From: X=27; Y= 43 To: X=152; Y=168 (3) READ
Physical document (601)
God of the Underworld as the son y Llull Law Erin (the Sier-Ilanded). if the Celtic gods of thr Underworld. He ruled he lined with his driglier, Modnor, and her fine
on was like the Cair arter, a partful islandfor apples grew singfier which it became named it is, known as the place were the High-King Arthyr tarie was stay wanded at the Battle of Carylan
ther itself appears to derive from the Celtic word
g"ear" CoilArthur, like so marry oper Celi gods, to aort basarized like Bei Mawr or Bran the be tastellation of Ursa Major or the Great Bear is,
ines own as Arthur's air. Ferr was inlard ar-Cod, Ario, wripped in Switzerland and around
was, in fact, a goddess and ther is no trace of her same theorists claim Ardar was a late addition to the lear hering a resugenre inpgan worship, or
- H Touch foil (602)
FIG. 6 U.S. Patent Oct. 20, 2015 Sheet 7 of 12 US 9,165.478 B2
(700)
Arthur the Myth Some people believe that King Arthur is so inextricably tied up in Celtic E. must, in origin, have been, not a man at all, but a God. Like so many other characters featured in the Mabinogion, Arthur in his From-arliestform, appears almost entirely mythical. He and his companions (701) have superhuman strength and abilities, and consort with giants and other mythological creatures, In the early Welsh poem "Preiddeu Annwfn", Arthur visits the Celtic Underworld, Annwfn. Even in Geoffrey and Malory, upon being fataly wounded in battle, Arthur is carried to the mystical Avalon, apparently the Underworld home of the Celtic god, Afallach. Many legends around the country attest to Arthur's immortality, for he sleeps in one of numerous To caves waiting to return and lead his people, The name Arthur itself - (702 appears to derive from the Celtic word Art, meaning "bear". Could Arthur, 8 like so many other Celtic gods, be merely a personification 5th many revered animals of the wild? Later to become humanized like Beli Mawr or Bran the Blessed. The constellation of Ursa Major or the Great Bear is, after all, sometimes known as Arthur's Wain, There was indeed a Celtic Bear-God, Artio, worshipped in Switzerland and around Trier, but she was, in fact, a goddess and there is no trace of her in BritainlSome theorists claim Arthur was a late addition to the Celtic pantheon during a resurgence in pagan worship, or possibly a mythical hero, the offspring of a human,
88.8.
FIG. 7 U.S. Patent Oct. 20, 2015 Sheet 8 of 12 US 9,165.478 B2
Web server
(806)
tilt-fill
li:iii
III
WWW. Virgin.net INTERNET NETWORK (805) (807) http://www.virgin.net/170301/?&Page=168Xf=27&Yf=438Xt=152&Y=168
A
http://www.virgin.net/170301/ User Workstation
Page: 16 (804) From: X=27; Y= 43 X=152; Y-168
Touch fi
Physical document (802) (801)
Barcode frke Underworld mykillereaf allesler-Hirtle) a Chird-fi tridered Hend reader wwaii dale, edia, and her pine m she air Clara tag islander - parew angry wildhikaar mana. , (803) as are were is liki Ariuyr way wall at leaky coun garris drawrie all as Kircle.
infeng Adriefon declir word Cabirds, sers may asker Choi, rainy air may reveral yak kamekales as area snnale claiary use to an is car bow is, nown a ran Len was inted GaAri worrigali starriad and around infrasader is no transfer wish langariar was a six milition ville 1 larif refuger irrir work, or
- -
FIG. 8 U.S. Patent Oct. 20, 2015 Sheet 9 of 12 US 9,165.478 B2
170301.PDF - Page 16
Arthur the Myth Some people believe that King Arthur is so inextricably tied up in Celtic Mytholo he must, in origin, have been, not a man at all, but a God | Like so man other characters featured in the Mabinogion, Arthur in his Frorn-arliestform, appears almost entirely mythical. He and his companions have superhuman strength and abilities, and consort with giants and other mythological creatures. In the early Welsh poem "Preiddeu Annwfn", Arthur visits the Celtic Underworld, Annwfn, Even in Geoffrey and Malory, upon being fatally wounded in battle, Arthur is carried to the mystical Avalon, apparently the Underworld home of the Celtic god, Afallach. Many legends around the country attest to Arthur's immortality, for he sleeps in one of numerous to caves waiting to return and lead his people, The name Arthur itself - (902) appears to derive from the Celtic word Art, meaning "bear". Could Arthur
after all, sometimes known as Arthur's Wain. There was indeed a Celtic Bear-God, Artio, worshipped in Switzerland and around Trier, but she was, in fact, a goddess and there is no trace of her in BritainlSome theorists claim Arthur was a late addition to the Celtic pantheon during a resurgence in pagan worship, or possibly a mythical hero, the offspring of a human.
- 16
FIG. 9 U.S. Patent Oct. 20, 2015 Sheet 10 of 12 US 9,165.478 B2
i. i. ill ill i: Web SerWeS lik Web server i ill; fl. I (1006) Bhiks it 3i isit it WWW, Virgin.net INTERNET NETWORKY (1005) (1007) http://www.virgin.net/SELECTION.TXT
User Workstation
Page: 16 (1004) From: X=27; Y= 43 To: X=152; Y=168 LOADING ...
Touchioi Physical document (1002) (1001)
Barcode for Urerwril agfississile ster-hands), air clingfie Uniran. He mid reader kawaits laterater sie seale clienaapalater rway will than radii is (1003) as lar were sea airwyr waywanaadale aircaen. areer is ariseries less at relate
desegrees pairio frow it clic word
bar' career, it seasy piercial, aisyal pay repairak six
an eastereywin the station first layer aircra bear is as wer's is eye was also cari, were it suitarian arosa infrarello aller is as harger sia diarkar was a stralia is the a dering a rance in weig by
is H U.S. Patent Oct. 20, 2015 Sheet 11 of 12 US 9,165.478 B2
Magnified text (1106) ... he must, in origin, have been, not a man Text-to-speech at all, but a God. Like so many other (1105) characters featured in the Mabinogion, Arthur in his earliest form, appearsal 8.
tured in the Mabinogion, Arthur in his earliest
form, appears almost entirely mythical. He and his companions.
User WorkStation (1104)
Physical document (1101)
if the law Erint (the Sier-Hartle). ird gf the Unitwork. He rail with is danger, Modrwarder firie
g the Cliricaw, a perfil island far and after which it learne rared it is,
the place were the High-King Arthyr fataly wounded at the Rattle gfornian, in Ardrian grids as King Evelaka.
hananized like Beli Mawr or Bran the
hitmanizedike Beli Mawr of iart of Ursa Major or the Great Bear is, Major or fir Great Bear is own as Arikar's air. There was need
' Win.iirt was inded a Chi rtio, warshipped in Switzerland and ground
s ar, a goddess and tiere is no trace of her
isis claim Arabir was a late addition to the
Celtic pankot during a rescente a reign it prefit worship, or Barcode H. ther, the affsprigg a human. reader (1103) Touch foil (1102) FIG. 11 U.S. Patent Oct. 20, 2015 Sheet 12 of 12 US 9,165.478 B2
ldentifying a physical document comprising 1201. One or a plurality of pages
Selecting and identifying a page of said physical document 1202
Placing and aligning a touch foil over the 1203 Selected page
ldentifying a part of the document by pressing 1204 one or a plurality of points over the touch foil
Sending to a server a request for data related to 1205 said selected part of the document
Locating on the server the electronic copy 1206 of the selected physical document
Extracting from the electronic copy on the server the text data comprised in (or related to) the part 1207 selected on the physical document
Receiving on the user workstation the requested 1208 text data
Presenting the received text data by means of a 1209 text-to-speech, magnification or other software
FIG. 12 US 9, 165,478 B2 1. 2 SYSTEMAND METHOD TO ENABLE BLIND Markup Language), the programming language used to PEOPLE TO HAVE ACCESS TO design Web pages. Home Page Reader can accurately INFORMATION PRINTED ON A PHYSICAL read text comprised in tables, or included in graphic DOCUMENT descriptions, so that blind users can easily navigate on the Web. FIELD OF THE INVENTION Note: IBM is a trademark of International Business Machines Corporation, a company with corporate head The present invention generally relates to a system and quarters in Armonk, N.Y., USA. method for enabling blind and visually impaired people to Mail Readers: This technology uses a screen reader for interact with a physical document and to have access to infor- 10 converting the text of an e-mail to speech. For example, mation printed on a physical document. “Lotus Notes Release 5’ of IBM, enables blind users to check the status of their e-mail (unread, read, deleted), to BACKGROUND OF THE INVENTION use calendar functions, and to file and organize docu mentS. Accessibility 15 Braille Editing Systems (BES): These systems allow users In broad terms, accessibility is the ability for a user to to input Braille characters from the keyboard. For access and use information and technology even if this user instance, IBM provides a BES that can be downloaded operates in constrained conditions. Information and technol from the IBM Web. ogy can be accessible through appropriate specialized tech Scanning Reading Systems: The information printed on a nical interfaces. The present invention is directed to a tech- 20 physical document is first scanned. Then, using an OCR nology to enable visually impaired and blind people to access (Optical Character Recognition) technique and a speech information printed on physical documents. synthesizer, this information can be read to blind people. Many groups, in governments, private industries and uni For example, “Open Book: Ruby Edition 4.0 of IBM is versities, are working on ways to guarantee universal access a program that transforms a computer, a Sound card, and to the on-line world for all individuals in society. This social 25 a scanner into a reading machine for reading books, movement, combined with a strong push to provide a better magazines, memos, bills, and many other printed docu accessibility in the workplace, drives an urgent call for new mentS. technologies. Screen Magnifiers: These software tools allow a visually As information technology (IT) penetrates all commercial impaired user to magnify text and images on a computer and public transactions and communications, it is important 30 screen. For example, “MAGic Screen Magnification', to ensure accessibility to everyone. Governments have begun from Henter-Joyce, is a screen magnification product for to define new regulations and standards to enable people with Windows. disabilities to easily access information technology. For Portable document magnifiers: The MAXPort by Nano instance, in the US, the federal, state and local governments Pac, Inc. is a portable magnifying system. The user have initiated a legislation requiring computer technology to 35 wears a special pair of glasses, points at a particular part be accessible to people with disabilities. More particularly, in of a physical document and can see a magnified image of June 2001, the U.S. Federal Government has implemented a said particular part. legislation known as “Section 508. Section 508 has changed Access to Printed Material the rules of the game on how the government must buy infor Reading manuals and reports at work, textbooks at School, mation technology and in particular “accessible' IT. 40 and menus at restaurants, and more generally reading printed Countries have established regulations that require acces material at any time and in any place is part of daily life. The sibility at some level, other countries are in the process of present invention is based on the following observation: it is developing Such regulations. In addition, international orga easier and quicker for sighted people to turn over pages of a nizations are working to develop accessibility standards. book than to browse electronic pages on a computer Screen. It Accessibility Aids for Blind People 45 would be an improvement to provide a similar capability to For someone who is completely unable to use a normal blind and visually impaired people. For instance, it would be screen or read a printed text, there are two alternatives: Braille a real improvement for blind people to have the possibility to reading or speech. Currently available assistance for blind access information printed on a physical document directly and visually impaired people comprises a wide range of tech from this physical document. nical Solutions, including document Scanners and enlargers, 50 When we compare paper based information with computer interactive speech Software and cognitive tools, screen reader based information, paper has a number of useful properties Software and screen enlargement programs. that computers cannot provide. For instance: Screen Readers: This technology converts text (or other paper is portable, familiar and can be easily distributed; elements) on a computer screen to speech, allowing paper is easy to read, mark, and manipulate. blind or visually impaired users to hear what is displayed 55 Many electronic systems attempt to replace paper. They on their computer screen. For example, “Jaws for Win offer, for instance, a better access to multimedia services. dows' is a computer Screen-reader application from However, most users prefer to work with paper and it is Henter-Joyce. It operates with Windows NT 4.0, Win difficult to foresee, in a near future, a general and massive dows 98 and Windows 95 and includes an enhanced replacement of paper books by electronic books (e.g., by Web Software speech synthesizer for speaking in several lan- 60 accessible e-books). guages. Publication entitled “The Last Book, IBM Systems Jour Note: Windows is a trademark of Microsoft Corpora nal, Vol 36, No. 3—1997, by J. Jacobson, B. Comiskey, C. tion, a company with corporate headquarters in Red Turner, J. Albert, and P. Tsao of the MIT Media Laboratory, mond, Wash., USA. compares printed books and computer Screens in the follow Web PageReaders: This technology reads aloud text just as 65 ing terms: it appears on Web pages. For example, “Home Page A book represents a fundamentally different entity than a Reader” of IBM, understands HTML (Hypertext computer screen in that it is a physical embodiment of a US 9, 165,478 B2 3 4 large number of simultaneous high-resolution displayS. a physical document. It would be desirable to offer similar When we turn the page, we do not lose the previous possibilities to blind and visually impaired people and in page. Through evolution the brain has developed a particular to enable these people to easily browse through the highly Sophisticated spatial map. Persons familiar with a same paper catalogs, magazines, newspapers, maps and manual or textbook can find information that they are books, and to easily “access' (receive and hear) information seeking with high specificity, as evidenced by their abil related to pictures and text printed on pages of these docu ity to remember whether something that was seen only mentS. briefly was on the right side or left side of a page, for The convenience and usefulness of a system for enabling instance. Furthermore their haptic connection with the blind and visually impaired people to access information brain's spatial map comprises a highly natural and effec 10 directly from hard-copy documents, to turn overpages and to tive interface, when such information is embodied on actual multiple physical pages. manually search for text, is well illustrated by in the following Another aspect of embodying information on multiple, article entitled “Talking Books Speak Volumes”, by Kendra simultaneous pages is that of Serendipity and compari Mayfield, Wired News, Jul. 13, 2000, Son. We may leaf through a large Volume of text and 15 “When George Kerscher began to lose his sight in his late graphics, inserting a finger bookmark into those areas of twenties, the former teacher realized that he might never greatest interest. Similarly, we may assemble a large be able to read the same way again. Since he could no body of similar matter in order to view elements in longer delve through the written page Kerscher, who has contrast to one another, Such as might be done to deter the degenerative disease retinitis pigmentosa, began lis mine which of a particular set of graphical designs is tening to audio books. But while these cassettes helped most satisfying. him read, they were limited. He couldn't turn pages or Publication entitled “A Comparison of Reading Paper and search for text. He had to endure hours of recordings to On-Line Documents', O'Hara and Sellen, XRCE: Technical find the information he needed'. Report: EPC-1997-101, Xerox Co. summarizes a study com In fact audio tapes have a drawback which is similar to the paring reading from paper to reading on-line. As reported in 25 drawback experienced when we Scroll computer screens: the this article: user can only go forward or backwards. The possibilities of “Critical differences have to do with the major advantages navigation are very limited. It is difficult for people with paper offers in Supporting annotation while reading, visual impairments, to use audio books or screen readers the quick navigation, and flexibility of spatial layout. These, same way sighted people do. For instance, looking up a recipe in turn, allow readers to deepen their understanding of 30 or searching for a gardening tip on an audio tape or on a long the text, extract a sense of its structure, create a plan for electronic book is extremely tedious and time consuming. writing, cross-refer to other documents, and interleave Nowadays, digital audio books are designed to make pub reading and writing. lished documents accessible and navigable for to blind or As reported in an article entitled “Paper is still with us”, by visually impaired persons. The ANSI/NISO Z39.86 standard, Ph. Jane Benson The Journal of Electronic Publishing, pub 35 defines a format and content for an electronic file set that lished by the University of Michigan Press— comprises a digital audio book and establishes a limited set of “Studies of professionals at work underscore three points requirements for digital audiobooks playback devices. It uses that are critical to the design of Scientific information: established and new specifications to delineate the structure paper isn't going away, reading and writing are inextri of digital audiobooks whose content can range from text with cably intertwined, and readers sample and navigate text 40 corresponding spoken audio, to audio with little or no text. according to specific purposes and tasks”. However, if a sighted user of a physical book wishes to Also, in the article entitled “Electronic Journals. What do discuss a particular section of such this book with a blind user, users think of them?', by Dr Cliff McKnight, Department of the blind user must actively search for a digital copy of the Information and Library Studies, Loughborough University, same edition of the book and then must search for the relevant Loughborough, Leicester LE11 3TU, UK, we can read: 45 section. There is a need to enable individuals familiar with A paper journal article is more easily manipulated than an paper documents to continue to use those same documents as electronic version. As one student subject in the Cafe Jus they age and their vision may be deteriorating. There is also a study remarked: “It is easier to turn the page of a paper need to provide a mechanism by which the text of a familiar journal. More than just page-turning, though, regular physical document may be supplemented by speech output or journal readers understand the structure of journal 50 display of a more easily readable version of the text. articles and use that knowledge to aid manipulation, One of the characteristics of the publishing industry is that providing rapid access to the information in an article. most publications can be delivered in a digital form. In some For example, Dillon showed that people could recon cases, the digital form has really Supplanted the physical one. struct articles from a set of paragraphs with about 80% In other cases, the digital form appears complementary to the accuracy. They also displayed a similar level of accuracy 55 physical one. However, in most cases, the physical form is till in saying which section of an article isolated sentences dominant. In fact, in terms of business models, with the had been taken from. Watching a reader pick up an issue exception of journals and directories, which tend to follow a of a journal, skim through the contents, turn to an article subscription model, most publishers have been unable to and skim through it and then turn back to the contents, all create profitable digital products and services, able to gener in a matter of seconds, it is clear that Such skills are 60 ate significant incomes. Digital audio books represent an largely automatic. Watching people trying to do the important improvement compared to traditional analog audio same thing in an electronic journal system, it is clear that books. For instance, digital audio books provide access to people will have to learn a different set of skills”. virtually stored information, offer improved navigation capa Sighted people can browse very easily through paper cata bilities, allow the use of keywords for searching and directly logs, magazines, newspapers, maps and books by flipping 65 accessing text in electronic books. However, due to a rela through the pages and by glancing at pictures and text. It is tively limited number of vision impaired users, the cost of also very easy for them to mark and return to specific parts of Such digital audio books remains relatively high. It results US 9, 165,478 B2 5 6 from this that a very small number of titles are published both, by means of an electronic reading machine, has nothing in as printed books and as digital audio books. common with the automatic “natural way of reading prac The necessity of using a natural friendly interface for hav ticed by sighted people. ing access to information has been precisely Summarized by A mobile Scanning pen comprising an optical character Ann Light, in the article entitled “Fourteen Users in Search of 5 recognition function, can also be used for producing a digital a Newspaper: the Effect of Expectation on Online Behav version of a printed document. Wizcom's Quicklink Pen Per iour'. School of Cognitive and Computing Sciences, Univer Sonal Scanner is an example of Such pen. However, Scanning sity of Sussex, CSRP507, printed words using Such a pen can be rapidly tedious and “People expect the friendly familiar paradigm of media to laborious. Each line must be individually (and accurately) guide them through uncharted territories of information'. 10 According to prior art, electronic reading machines using scanned and a reasonable visual acuity is required. The computer-based optical character recognition (OCR) are present invention does not require a good vision, as only two widely used to help visually impaired people and people with physical markers are required to define the part of the docu reading difficulties to read information on paper documents. ment that must be read. An electronic reading machine basically comprises 15 As a conclusion, it is important to enable blind people to a personal computer connected to a scanner and operating easily and conveniently access information comprised in with an OCR software, and hard-copy publications that are read everyday by sighted text-to-speech hardware and Software. people. Usually, a method for electronically reading a printed document comprises the following steps: OBJECTS OF THE INVENTION placing a page of a text document on a scanner; digitizing this page for obtaining a pixel bit map of the It is an object of the invention to enable a blind or visually page. impaired person, as well as a person with reading disabilities, converting this image to text by means of an OCR program to access information printed on a physical document and executed in a personal computer, this personal computer 25 information related to this printed information. being connected to the scanner, and It is another object of the invention to enable a blind or generating speech from the text by means of a text-to visually impaired person to access information directly from speech program. a physical document without using an electronic reading Currently, reading machines are provided by a variety of machine based on a page scanner and/or an optical character companies, including IBM, Telesensory, Arkenstone and 30 recognition (OCR). Kurzweil Educational Systems. Unfortunately, electronic It is another object of the invention to enable a blind or reading machines suffer from a variety of functional and visually impaired person to easily select a part of a physical operational deficiencies that limit their usefulness. For document simply by touching (e.g., with the fingertip or with instance: a pen) this physical document. Scanning a page is a mechanical operation. It takes time to 35 It is a further object of the present invention to enable a move the electro-optical components of the Scanner over blind or visually impaired person to take a physical docu the page. ment, to open it at any page, to select with a finger or a pen a it takes time also to transmit the information from the part of this page and to hear the text of the selected part. Scanner to the computer. It is also an object of the present invention to enable both the OCR process for an entire Scanned page, takes a lot of 40 magnification and electronic reading of a selected part of a time and resources. physical document. only a single page can be scanned at the same time. It is a further object of the present invention to enable a OCR is prone to error. Prime Recognition, in an advertise person, after having selected a part of a printed document ment for the product PrimeOCR, quotes a typical error rate of comprising text and images, to hear the text, to receive 2% for conventional OCR algorithms. PrimeOCR itself 45 descriptions of images, and to access multimedia information reports an error rate of 0.4%, or 8 errors in a typical page of or services related to said printed text and images. 2000 characters. It is a further object of the invention to provide a system As a result, even for an experienced user, it can take more that, apart from providing to a blind or visually impaired than a minute before an electronic reading machine can read person, an access to information directly from a physical a single Scanned page. During of after the reading, errors must 50 document, can also be used for many other applications in the be corrected. The process is repeated for every page in the daily life. Such as helping children to read, enabling people document. with reading difficulties to read, enabling sighted people to Since scanners are limited in size, another drawback con read in the dark, enabling people with a poor vision to read cerning electronic reading machines is the reading of pages Small print on paper documents. with large formats (a newspaper page for instance). Such 55 It is a further object of the present invention to enable pages cannot be scanned in one pass but require multiple individuals familiar with paper documents to continue to use passes. the same documents as they age and their vision may be Moreover, a complete electronic reading system with a deteriorating and to supplement the text of a familiar physical computer, a scanner, and speech hardware and Software is document by speech output or display of a more easily read bulky and non portable. 60 able version of the text. It is clear that reading a text on a paper document by means of an electronic reading machine involves complex manipu SUMMARY OF THE INVENTION lations and time consuming operations. However, there are other important deficiencies that limit in practice the accep As defined in independent claims, the invention relates tance and usefulness of electronic reading machines. Basi 65 generally to a method, a system and a computer program for cally, the tasks required for using these electronic reading enabling a blind or visually impaired person, to access infor machines are far to be “natural”. Reading physical documents mation printed on a physical document, directly from said US 9, 165,478 B2 7 8 physical document, simply by pointing and touching a FIG. 12 shows the steps of the method of selecting a part of desired part of said document. More particularly, the method a physical document and of providing electronic reading and comprises the steps of screen magnification of said selected part. identifying a physical document; The invention itself, as well as a preferred mode of use, identifying and locating an electronic copy of said identi objects and advantages thereof, can be understood by refer fied document; ence to the following illustrative embodiment described by identifying one or a plurality of pages of said physical the accompanying drawings. document and identifying a part of the identified physi cal document using the position of points pressed on a DETAILED DESCRIPTION OF THE PREFERRED touch foil Successively placed and aligned over or under 10 EMBODIMENT said identified pages; retrieving from the electronic copy of the physical docu The present invention addresses different problems that ment, data related to the selected part of the document; may arise in reading text, particularly problems that affect presenting said retrieved data visually or orally on the user 15 visually impaired persons or persons who cannot see or read workstation. correctly (e.g., in low light conditions). For instance: Further embodiments of the invention are provided in the a person driving a vehicle may have some difficulties to appended dependent claims. read a text; another person may not have time to read a text; BRIEF DESCRIPTION OF THE DRAWINGS bad lighting or display conditions may make difficult or impossible the reading of a text in a printed or displayed The novel and inventive features believed characteristic of form. the invention are set forth in the appended claims. The inven The present invention allows a blind reader to receive an tion itself, however, as well as a preferred mode ofuse, further audible transcription of a selected part of a physical document objects and advantages thereof, will best be understood by 25 (for example an audible transcription of a selected text in reference to the following detailed description of an illustra book). tive detailed embodiment when read in conjunction with the The present invention also discloses a method for selecting accompanying drawings, wherein: a part of a physical document simply by pressing with a FIG. 1 shows a preferred embodiment of the present inven fingertip or a pen a touch foil placed over said document; for tion. 30 identifying data associated with this selected part in a data FIG. 2 shows how, according to the present invention, a base comprising an electronic copy of the document; for user, using a barcode reader, reads the address (e.g., URL extracting text from said data and for converting this text into Uniform Resource Locator) of a document encoded on a bar speech. code printed on this document (e.g., at the bottom right corner The present invention is directed to a method for use in a of the cover) and how information related to this document 35 user system connected to a network, for retrieving from a (e.g., title, author, etc.) can be retrieved from a server and read server, an electronic copy of a physical document, and infor to the user by means of an electronic reading system. mation related to said document, and for playing, displaying, FIG. 3 shows how a user opens a physical document (e.g., reading and/or magnifying on the screen of this user system, at the page where a bookmark has been inserted). the retrieved electronic copy and information related to said FIG. 4 shows how, according to the present invention, a 40 document. The method comprises the steps of user using a barcode reader, can read a page number encoded identifying a physical document selected by a user, prefer on a bar code printed at a predefined position on a selected ably by reading, by means of a barcode reader, a docu page and how the system decodes and audibly informs the ment identifier printed on said physical document at a user of the selected page number. predefined position; FIG. 5 shows how a touch foil is placed aligned over a 45 identifying a first page selected by the user in said physical selected page of a physical document according to the present document, preferably by reading, by means of a barcode invention. reader, a page identifier printed on said first page at a FIG. 6 shows how a user selects a part of the physical predefined position; document by pressing a first point and a second point on a determining the position of a first point pressed (e.g. with a touch foil placed aligned over a selected page. 50 fingertip or a pen) on a touch foil placed and aligned over FIG. 7 shows the start point and the endpoint of the part of or under the identified first page, said first point corre the physical document selected by the user, said start point sponding to the start point of a part selected in said and end point corresponding to the two points over the touch identified document by the user; foil previously pressed by the user. if start point and endpoint of the selected part are on the same FIG. 8 shows how, according to the present invention, a 55 page: request for retrieving information related to a part of a docu determining the position of a second point pressed (e.g. ment is sent to a server. with a fingertip or a pen) on the touch foil placed and FIG.9 shows how, according to the present invention, data aligned over or under the identified page of said docu related to the part selected by the user on the physical docu ment, said second point corresponding to the end point ment is extracted (in bold) from the electronic copy of the 60 of said selected part; document stored on the Server. if start point and end point are not on the same page: FIG. 10 shows how, according to the present invention, the identifying a second page selected by the user in said text corresponding to the selected part of the physical docu physical document, preferably by reading, by means of ment is downloaded from the server to the user workstation. a barcode reader, a page identifier printed on said second FIG. 11 shows how, according with the present invention, 65 page at a predefined position; the received text file can be converted to an audio file or can be determining the position of a second point pressed (e.g. magnified. with a fingertip) on the touch foil placed and aligned US 9, 165,478 B2 10 over or under the identified second page of said docu Touch Foil ment, said second point corresponding to the end point The touch foil (102) may be made of transparent resistive of said selected part; or capacitive films of the type used commonly to manufacture identifying the part of the identified physical document that touch screens. The generated signal is generally proportional has been selected using the position (e.g. coordinates) of 5 to the coordinates of the point pressed. One example of touch the start point and the end point; foil it is possible to use is the TouchTek4 (4-wire Analog identifying and locating an electronic copy of the identified Resistive Touchscreens) of MICRO TOUCH company document and information related to the selected part of (TouchTekTM is a trademark of the MICRO TOUCH Com the document; pany). 10 TouchTek4 touchscreens feature hard-coated polyester retrieving from the electronic copy of the document, data topsheets, available in several surface finishes. Spacer dots related to the part of the document selected by the user; are available in several dot arrays, optimized for finger, pen extracting text data from said data and audibly reading said and finger, or pen-only input. Electronic control is provided text data by means of a text-to-speech Software; by a serial controller, or by 4-wire controllers. TouchTek4's optionally or alternatively, magnifying said retrieved data 15 specifications include narrow inactive border areas and com on the user system screen; pact touch sensors which allow system designers and OEMs optionally or alternatively, accessing multimedia informa to provide the largest useable screen area and full mouse tion or services related to said retrieved data. emulation without sacrificing functionality or consuming Synthetic Reading and Screen Magnification of a Selected excess power. Apart from being Suited for implementing the Part of a Physical Document functions of this invention, TouchTek4 touchscreens are com As shown in FIG. 1, the system according to an embodi monly used in hand-held personal information management ment of the present invention comprises: systems, PDAs, mobile computing systems, automotive, a physical, i.e., hard-copy document (101) (e.g., a book) diagnostics and telecom devices, and Internet appliances. comprising one or a plurality of pages; TouchTek4 touchscreens are engineered to accept more than a physical document interface comprising a touch foil 25 three million touches to any area of the screen. (102) (to be placed over or underneath a page of this Barcode Reader document); The barcode reader (103), connected to user workstation a barcode reader (103); (104) is used to identify the documents and pages of the a user workstation (104), connected to documents selected by the user. One example of barcode the touch foil interface (102); 30 reader it is possible to use, is the PenWANDC Computer the barcode reader (103): Ware. The PenWAND is a combination of barcode reader and a communication network (105) (e.g., the Internet net light pen in one hand-held instrument. This is an ideal solu work); tion for applications requiring reading of barcodes and Screen a server (106) (e.g., a Web server). selection. Physical Document 35 User Workstation The physical document (101) can be of any kind, for In the embodiment of the invention illustrated in FIG.1, the example, a newspaper, a legal document, a geographic map, a user workstation (104) is used for carrying out the following fiction novel, an academic text book, a technical book, a steps: commercial catalog or even any other type of engraved, writ receiving from the barcode reader (103) a code for identi ten, or printed surface. The material of the document can be 40 fying (107) the document (101) (for instance, an address paper, plastic, wood or any other material. For identifying the or an URL Uniform Resource Locator) and the page selected document (101) to the system, a bar code (107) is number (301) selected by the user; printed or attached at a well known position on the physical receiving from the touch foil (102), (602) the coordinates document (e.g., on the front cover, back cover or first page). In of the two points (605), (606) touched by the user over the embodiment of the invention shown in FIG. 1, the URL 45 the document (601), said points (the start point and the (i.e., the internet address) of an electronic document is end point) specifying the part of the selected document encoded in a bar code (107). This electronic document stored page that must be read; on a server (106) is the copy of a physical document (101). As transmitting to a given server (106) means for identifying illustrated in FIG. 3, to identify pages in a document (300), on said server an electronic copy (or replica, or repre (101) a bar code (301), (302) is printed or attached at a well 50 sentation) (900) of the physical document (101), (700) known position on each document page (e.g., on the bottom and data related to the part of the physical document left (301) and bottom right corners (302) of the open docu (701), (702) selected by the user; said identification ment (300)). Barcodes are read by means of a barcode reader means comprising: (103) connected to the user workstation (104). the document identification code (107), In an alternate embodiment of the invention, a physical 55 the page identification code (301) and document can be specifically produced for the purpose of the coordinates of the two points (605), (606) specifying accessing information from a tactile system. Compared to the the part (701), (702) of the physical document (700) original paper document, this physical document can have a selected by the user; different form, to include, for instance, tactile indications (in retrieving from the electronic copy of the document on the Braille, for example) related to the structure and pages of the 60 server (106), data (903) corresponding to the part (701, document. With such a tactile document or book, a blind user 702) of the document (700) selected by the user; can quickly and independently navigate in the document, and extracting text data from said data; can easily mark the sections of text he wish to hear. In this converting this text data to speech (1105) by means of a embodiment, to select the same part of text, the server deter text-to-speech Software; mines from the points selected by the user on the physical 65 optionally, magnifying on the user's display, the text data document, the corresponding points in the original paper or the electronic copy corresponding to the selected part document. (1106) by means of a display magnifying software. US 9, 165,478 B2 11 12 The user workstation (104) is, for example, an Internet of said document, said second point corresponding to the appliance, a multimedia Personal Computer (PC), a game end point of said selected part; console, a cellphone. ... The touch foil (102) communicates (1205) sending to a given server a request for the selected with the user workstation by means of a cable, a wire pair, an document part, said request comprising an identification infrared link, or a wireless link. In a particular embodiment, 5 of: the user workstation (104) is connected to a communication the physical document, network (105), preferably the Internet network and comprises the page(s), and a Web Browser application. the selected part of the document; Server (1206) locating on the server an electronic copy (or replica) 10 of the selected physical document; The server (106) is primarily the repository where elec (1207) extracting text data related to the part selected on tronic representations (900) of physical documents (700) are the physical document; stored. Each electronic representation is associated with a (1208) receiving on the user workstation, the requested text physical document (or publication) by means of the barcode data; printed on this physical document. The electronic represen 15 (1209) reading to the user, the received text data using a tation of a document includes an exact copy or replica of the text-to-speech Software on the user workstation. original document. This copy includes, for instance, Scanned FIG. 2 shows how the user selects a physical document (or images of pages. publication) (201) (e.g., printed book entitled “Early British The electronic representation may also include text, text Kingdoms'). By means of any user interface (keyboard, descriptions of graphical elements (such as charts, graphs or mouse, Voice recognition Software and microphone) or any illustrations). The server also provides users with additional other reading means (e.g., barcode reader), the user enters a information Such as: code or name (207) for identifying the document. In the a speech description (205) of the selected document (such preferred embodiment shown in FIG. 2, the user reads by as title, authors, general information); and means of a barcode reader (203) a bar code (207) with the speech instructions related to the selected document (206) 25 address on the Web (e.g.: the URL http://www.virgin.net/ (e.g., for informing the user about the position of the 170301/) of the electronic copy (or replica with the same barcodes on pages of selected physical document or identical format) of the selected physical document (201). publication). The bar code (207), used to identify the document, must be Method for Selecting a Part of a Physical Documents and for printed or attached at a predefined reserved place on the Providing an Electronic Reading and Screen Magnification of 30 physical document (e.g., on the front cover, back cover or first this Selected Part page). As shown in FIG. 12, the present invention discloses a The bar code gives access, on the server (106), to informa system, method and computer program for: tion related to the selected physical document, such as: Selecting a part of a physical document (preferably a the title (e.g., “Early British Kingdoms), printed publication, like the document shown in FIG. 1 35 the author's name (e.g., “David Nash Ford of Binfield, entitled “Early British Kingdoms’’), simply by touching Berkshire, UK), with a fingertip (or pen) over this physical document; the date of the publication (e.g., “28 January 1999), identifying in an electronic copy (or replica) of the physical the ISBN (International Standard Book Number) (e.g., document, the part that has been selected by the user; “84-344-0856-2), etc. extracting text data comprised in the selected part, and 40 This information about the selected physical document is converting extracted text into electronic speech. transmitted from the server (106) to the user workstation The method comprises the steps of: (204) as a text file. This text file is then converted to electronic (1201) identifying a physical document, said physical voice (205) by means of a text-to-speech software operating document comprising one or a plurality of pages; on the user workstation. (1202) selecting and identifying a page of said physical 45 The user workstation can also receive from the server (106) document, said page comprising one or a plurality of operational instructions (206) for locating page numbers bar printed items; codes on the pages of the selected physical document or determining (1204) the position of a first point pressed (e.g. publication (e.g., “Page number codes are located on bottom with a fingertip or pen) on a touch foil placed and aligned left and bottom right corners of pages'). These operational (1203) over or under the identified first page, said first 50 instructions are transmitted to the user by means of the text point corresponding to the start point of apart selected in to-speech Software operating on the user workstation. said identified document by the user; FIG.3 shows how a user opens a physical document (300) if start point and endpoint of the selected part are on the same at the page where a bookmark has been inserted (e.g., on page: “Page 16'). This figure also shows how on each page of the determining (1204) the position of a second point pressed 55 document (e.g., “Page 16”. “Page 17), a bar code (301), (e.g. with a fingertip or pen) on the touch foil placed and (302) with a page number, is printed on a predefined and fixed aligned (1203) over or under the identified page of said position, already known by the user. document, said second point corresponding to the end FIG. 4 shows how a user selects a page of physical docu point of said selected part; ment (401) and how the number of the selected page is iden if start point and end point are not on the same page: 60 tified by scanning by means of barcode reader (403), the bar identifying a second page selected by the user in said code (405) encoding the page number. After decoding the physical document, preferably by reading, by means of scanned bar code, the user workstation (404) informs the user a barcode reader, a page identifier printed on said second (406) by means of an electronic voice, of the selected page page at a predefined position; number (e.g., “Page sixteen'). determining (1204) the position of a second point pressed 65 Alternate embodiments for identifying pages of a docu (e.g. with a fingertip or pen) on the touch foil placed and ment exists and can be used with the present invention. An aligned (1203) over or under the identified second page example is given by a document having page numbers printed US 9, 165,478 B2 13 14 as text. It is possible to identify a page of the document by electronic copy (e.g., "170301.PDF) of the physical docu scanning on the page the place where the page number is ment (same format and same content). The HTTP request is printed. A scanning device can be integrated with the barcode also parsed to find the page of the physical document (e.g., scanner described previously. The scanning device can also “Page=16”) selected by the user and the coordinates specify be used separately like the Quicklink Pen Personal Scanner ing the selected part of the physical document (e.g., From: described in the specifications of the present application. Xf-27, Yf=43: To: Xt=152, Yt=168) on the server. FIG. 5 shows how a user places a touch foil (502) over a page of a physical document, aligning the borders of the touch FIG. 9 illustrate how, in a particular embodiment of the foil and the document using some conventional means (505) invention, the server (806) uses the information comprised in (e.g., by adjusting the upper left corner of the touch foil with 10 the HTTP request (807): the upper left corner of the page). document reference number, FIG. 6 shows how, according to a particular embodiment of the present invention, it is possible to define a part of a page number, and physical document (601). The user presses (e.g., with a fin coordinates of points defining the part of the document gertip or pen) (1) on a first point (605) and then (2) on a second 15 selected by the user, to locate and extract from the elec point (606) over the touch foil (602) placed aligned over the tronic copy (e.g., "170301.PDF) replica of the physical selected page (e.g., over “Page 16” of the publication entitled document (801), text data (903) (in bold) corresponding “Early British Kingdoms'). When the user has finished to to the part of physical document (701), (702) selected by press the points defining the selected part that he desires to the user. The extracted text data is formatted by the hear and that must be electronically read, she/he (3) then server in the form of a text file (e.g., “SELECTION. presses on the keyboard of the user workstation (604) a TXT) and is transmitted as reply to the request (807) reserved READ key (607), informing the system to initiate previously received from user workstation (804). the task of locating and reading the selected document part. Coordinates of the points (605), (606) pressed by the user on It is important to note that different methods can be used to the touch foil (602) are transmitted from the touch foil to the 25 determine, from the coordinates of the points selected (701), user workstation (604). (702) on the touch foil, the text data (903) that must be must The selection of apart of a physical document using a touch be extracted from the electronic copy (from point 901 to point foil, can be done according to various forms. For example, in 902). In the example illustrated in FIG. 9, incomplete words a particular embodiment of the invention, if the user touches are excluded from the head and from the tail of the text string a single point on the touch foil and then presses the READ 30 comprised in the selected part of the physical document. key, this operation is equivalent to a command for reading the The server can also receive from the user workstation a full document from this point (possibly until the user presses HTTP request (such as “&format—magnifyOriginal') to view a STOP READ reserved key on the user workstation key a magnified image of a part of the original document. The board). Also, if the user selects a first page and indicates a first server sends back a reply including an image of the selected point on this first page, and then selects a second page and 35 part of the same format and appearance than the original indicates a second point on this second page, and then presses document. This image is magnified on the display of the user the READ key, this operation is equivalent to a command for workstation. reading the document from the first point selected on the first The information returned by the server (806) in reply to the page to the second point selected on the second page. It is now request (807) received from user workstation (804) can be apparent that other arrangements can be implemented by 40 different than the exact transcription (as a text file) of the data those skilled in the art for correlating a plurality of points printed on the physical document. In fact, nothing prevents selected by a user with a part of a physical document. the server (806), to reply with any kind of multimedia infor FIG. 7 shows in detail two points (701), (702) on a page of mation (e.g., a combination of a long text description with a physical document (700). These two points correspond to music or songs). This multimedia information can be linked the points (605), (606) pressed by the user on the touch foil 45 (or even hyperlinked) with different parts of the physical (602). document that the user can select. Therefore, the invention is FIG. 8 shows how the user workstation (804) sends a not limited to a mechanism for reading (i.e., for speaking by request (807) related to a part (701), (702) of a physical means of text-to-speech transcription) text printed on a physi document (700) to a server (806). The request comprises an cal document to a blind user, but the invention is also directed identification of: 50 to a mechanism for accessing from a physical document, all the physical document, kinds of multimedia information or services available on the the selected page, and Web the coordinates of the part selected by the user. In the example illustrated in FIG. 8, the HTTP request: FIG. 10 shows how, in reply to request of a user for access http://www.virgin.net/170301/2&Page=16&Xf= 55 ing information from a selected part (701), (702) of a physical 27&Yf=43&Xt=152&Y=168 document, text data (903) extracted from (or associated with) specifies: the selected part on the electronic copy of the document the URL of the physical document (e.g. http://www.vir (900), is downloaded (1007) as a text file (e.g., “SELEC gin.net/170301/), TION.TXT) from the server (1006) to the user workstation the page number (e.g., “Page 16”), and 60 (1004). the coordinates of the part selected by the user (e.g., From: FIG. 11 shows how the text file (e.g., “SELECTION. Xf-27, Yf=43: To: Xt=152, Yt=168). TXT), once received by the user workstation (1104), is con In the example illustrated in FIG. 8, when the server (806) verted into an audio file (.aiff, wav, .au file) by means of a (e.g., “www.virgin.net') receives the HTTP request, this text-to-speech software and is played (1105) to the user. This request is parsed to find the physical document reference 65 figure shows also, how the received text is magnified (1106) number (e.g., "170301). This physical document reference on the user workstation display by means of a screen magni number (or code) is then used to search on the server for the fying software. US 9, 165,478 B2 15 16 What has been described is merely illustrative of the appli 6. The method of claim 1, wherein said presenting com cation of the principles of the present invention. Other prises presenting said retrieved data orally to the blind person arrangements and methods can be implemented by those on the user system, and wherein said presenting said retrieved skilled in the art without departing from the spirit and scope of data orally comprises: the present invention. 5 extracting text data from said data; converting said text data into data that can be audibly read. What is claimed is: 7. The method of claim 6, wherein said converting said text 1. A method for use in a user system, for accessing infor data into data that can be electronically read comprises: mation related to a physical document, said method compris audibly reading, to the blind person on the user System, said ing: 10 text data by means of a text-to-speech system. identifying an existing physical document; 8. The method of claim 1, wherein said presenting com after said identifying the existing physical document, iden prises presenting said retrieved data visually to the visually tifying and locating an electronic copy of said identified impaired person on the user system, and wherein said pre existing physical document, wherein the electronic copy 15 senting said retrieved data visually comprises: of the document is an exact replica of the physical docu magnifying the retrieved data on a screen Such that the ment; magnified retrieved data can be visually read by the identifying one page or a plurality of pages of said physical visually impaired person. document; 9. The method of claim 8, wherein said magnifying the identifying a selected part of the physical document using retrieved data on the screen comprises: the position of points on said identified one page or a magnifying on the screen the electronic copy of the plurality of pages; Selected part of the physical document. in response to said identifying the selected part of the 10. The method of claim 1, wherein said presenting com physical document using the position of points on said prises presenting said retrieved data visually to the visually identified one page or a plurality of pages, retrieving, 25 impaired person on the user system, and wherein said pre from the electronic copy of the physical document, data senting the retrieved data visually comprises: related to the selected part of the physical document; enhancing on a screen the electronic copy of the selected presenting said retrieved data visually to a visually part of the physical document to match a preferred read impaired person or orally to a blind person on the user ing view of the visually impaired person. system, wherein said presenting said retrieved data visu 30 11. The method of claim 1, wherein said identifying the ally is configured to enable the visually impaired person physical document comprises: to see said retrieved data, and wherein said presenting reading, by means of a barcode reader, a document identi said retrieved data orally is configured to enable the fier printed on said physical document at a predefined blind person to hear said retrieved data. 35 position. 2. The method of claim 1, wherein said identifying a part of 12. The method of claim 1, wherein said identifying one the identified physical document using the position of points page or a plurality of pages consists of said identifying one on said identified one page or a plurality of pages further page, and wherein said identifying one page in said physical comprises pressing said points on a touch foil Successively document comprises: placed and aligned over or under said identified one page or a 40 reading, by means of a barcode reader, a page identifier plurality of pages. printed on said one page at a predefined position. 3. The method of claim 2, wherein said pressing comprises 13. The method of claim 1, wherein said presenting com pressing said points with a fingertip or a pen on a touch foil. prises presenting said retrieved data orally to the blind person 4. The method of claim 1, wherein said physical document on the user system, and wherein said retrieving, from the and said one page or a plurality of pages in said physical 45 electronic copy of the document, data related to the selected document are selected by a user. part of the physical document comprises: 5. The method of claim 1, wherein the selected part of the identifying and locating information and services associ physical document comprises a start point on a first page of ated with the selected part of the physical document; the physical document and an end point on a second page of retrieving said information and services; and the physical document, wherein the first and second pages are 50 audibly playing said information and services on the user different pages of the physical document, and wherein said system. identifying one page or a plurality of pages of said physical 14. The method of claim 13, wherein said information document and identifying a selected part of the physical docu associated with the selected part of the physical document ment using the position of points pressed on a touch foil comprises a description of graphical data pertaining to an Successively placed and aligned over or under said identified 55 image and related to the selected part of the physical docu one page or a plurality of pages consists of: ment, and wherein said audibly playing comprises audibly identifying the first page in said physical document; playing said description of said graphical data to the blind determining the position of a first point pressed on the person on the user system. touch foil placed and aligned over or under the identified 15. The method of claim 1, wherein said presenting com first page, said first point being the start point; 60 prises presenting said retrieved data orally to the blind person identifying the second page in said physical document; on the user system, and wherein said identifying and locating determining the position of a second point pressed on the an electronic copy of the identified document comprises: touch foil placed and aligned over or under the identified identifying and locating information associated with the second page of said document, said second point being physical document; the end point; 65 retrieving said information; and identifying the selected part of the physical document audibly playing said information to the blind person on the using the position of the start point and the end point. user system. US 9, 165,478 B2 17 18 16. The method of claim 15, page identification code that identifies the one page or a wherein the information associated with the physical docu plurality of pages of the physical document. ment comprises speech instructions related to the physi 20. The method of claim 1, wherein the electronic copy of cal document; the physical document and information and services associ wherein the speech instructions comprise speech instruc- 5 ated with said physical document are located on the user tions for locating positions of bar codes on pages of the system. physical document; 21. The method of claim 1, wherein said physical docu wherein said audibly playing comprises audibly playing, to ment comprises Braille indications. the blind person on the user system, said speech instruc 22. The method of claim 1, wherein the method does not tions for locating said positions of bar codes on pages of 10 comprise using an electronic reading machine based on a the physical document; page scanner, and wherein the method does not comprise wherein a first bar code of said bar codes enables access to using optical character recognition (OCR). a title of the physical document, a name of an author of 23. A user system comprising: the physical document, a date of publication of the a computer; and physical document, and an International Standard Book 15 a non-transitory storage device containing instructions Number (ISBN) of the physical document; and which, upon being executed by the computer, carry out wherein said title, said name, said date, and said ISBN are the method of claim 1. stored on a server connected to the user system. 24. A system comprising: 17. The method of claim 1, wherein said user system is said user system of claim 23 connected to a communication connected to a communication network comprising plurality 20 network; of servers, and wherein the electronic copy of the physical a touch foil to be placed over or under a page of a physical document is located on one server of said plurality of servers. document for: 18. The method of claim 1, wherein the electronic copy of detecting a pressure exercised over one or a plurality of the physical document is located on the user system. point of its surface; and 19. The method of claim 1, wherein said user system is 25 determining the coordinates of said one or plurality of connected to a communication network comprising a plural points; and ity of servers, wherein information and services associated a connection between said touch foil and the user system. with said physical document are located on each server of said 25. The system of claim 24, said system further comprising plurality of servers, wherein the information comprises said a bar code reader connected to said user system for reading data related to the selected part of the physical document, and 30 bar codes on the physical document. wherein the method further comprises: 26. A non-transitory storage device comprising instruc transmitting to a given server of the plurality of servers a tions which, upon being executed by a computer, carry out the document identification code that identifies the elec method of claim 1. tronic copy of the physical document and at least one