USOO746325OB2

(12) United States Patent (10) Patent No.: US 7463,250 B2 Lapstun et al. (45) Date of Patent: *Dec. 9, 2008

(54) SYSTEM FOR DETERMINING THE (58) Field of Classification Search ...... 345/156-179; ROTATIONAL ORIENTATION OF A SENSING 178/1801-18.07, 19.01-19.06 DEVICE See application file for complete search history. (75) Inventors: Paul Lapstun, Balmain (AU); Kia (56) References Cited Silverbrook, Balmain (AU) U.S. PATENT DOCUMENTS (73) Assignee: Silverbrook Research Pty Ltd, 4,246,439 A 1/1981 Romein Balmain, New South Wales (AU) 4,577,057 A 3, 1986 Blesser 4,814,553 A 3/1989 Joyce (*) Notice: Subject to any disclaimer, the term of this patent is extended or adjusted under 35 (Continued) U.S.C. 154(b) by 590 days. FOREIGN PATENT DOCUMENTS This patent is Subject to a terminal dis EP O407734. A 1, 1991 claimer. (Continued) (21) Appl. No.: 11/144,840 OTHER PUBLICATIONS (22) Filed: Jun. 6, 2005 Dymetman, M., and Copperman, M., Intelligent Paper; in Electronic Publishing, Artistic Imaging, and Digital Typography, Proceedings (65) Prior Publication Data ofEP '98, Mar/Apr. 1998, Springer Verlag LNCS 1375, pp. 392–406. US 2005/0225541 A1 Oct. 13, 2005 (Continued) Related U.S. Application Data Primary Examiner Vijay Shankar (63) Continuation of application No. 09/722,088, filed on (57) ABSTRACT Nov. 25, 2000, now Pat. No. 7,105,753, which is a continuation-in-part of application No. 09/575,168, A System for generating orientation data when a sensing filed on May 23, 2000, now Pat. No. 6,737,591. device is positioned or moved relative to a surface, the orien tation data being indicative of an orientation of the sensing (30) Foreign Application Priority Data device relative to the surface, the surface having coded data May 25, 1999 (AU) ...... PQ0559 disposed upon it, the coded data being indicative, when Jun. 30, 1999 (AU) ... PQ 1312 sensed by the sensing device, of the orientation, the sensing Jun. 30, 1999 (AU) ... PQ1313 device including: a housing; orientation sensing means con figured to generate the orientation data using at least some of (51) Int. Cl. the coded data; and communications means configured to G06F 3/033 (2006.01) communicate the orientation data to a computer system. (52) U.S. Cl...... 345/179; 34.5/173; 34.5/176: 178/18.03; 178/1901; 178/19.05 21 Claims, 78 Drawing Sheets

SURFACE US 7463,250 B2 Page 2

U.S. PATENT DOCUMENTS 6,201.903 B1 3, 2001 Wolff et al. 6,737,591 B1 5/2004 Lapstun et al. 4,864,618 A 9/1989 Wright et al. 6,964.374 B1 1 1/2005 Duknic et al. 4.947,156 A 8, 1990 Sato et al. 7,105,753 B1* 9/2006 Lapstun et al...... 178/1905 4,963,703 A 10/1990 Philips et al. 4,975,546 A 12/1990 Craig FOREIGN PATENT DOCUMENTS 5,027,115 A 6, 1991 Sato et al. 5,051,736 A 9, 1991 Bennett et al. EP O493.936 A 23. 5,159,321. A 10/1992 Masaki et al. EP O836151 : 5,239,489 A 8/1993 Russell GB 2306669. A 5, 1997 5.248,855. A 9/1993 Cambridge JP 10-01.1538 1, 1998 5,469,193. A 1 1/1995 Giobbietal. JP 2000-200139 658 5,477,012 A 12/1995 Sekendur JP 2000-293303 10, 2000 WO WO 97.11386 A 3, 1997 5,548,092 A 8, 1996 Shriver 5,652.412 A 7, 1997 L. i etal WO WO99, 18487 A2 4f1999 W - 4 azzouni et al. WO WO99,23545 A 5, 1999 5,661.506 A 8, 1997 LaZZOuni et al. WO WO99,50787 A1 10, 1999 5,692,073. A 1 1/1997 Cass WO WOOOf 72287 A 11, 2000 5,852,434. A 12/1998 Sekendur 5,905,251 A 5, 1999 Knowles OTHER PUBLICATIONS 5,994,710 A 11/1999 Knee et al. Miriam Webster's Collegiate Dictionary, 10th ed. (Springfield, 6,076.734 A 6/2000 Dougherty et al. 6,081,261 A 6, 2000 Wolff et al. mass.: Merriam-Webster, Inc., 1999) p. 592. 6,184,873 B1 2/2001 Ward et al. * cited by examiner U.S. Patent Dec. 9, 2008 Sheet 1 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 2 of 78 US 7463,250 B2

NETPAGE APPLICATION 125 SERVER

NETPAGE PAGE SERVER 1O

NETPAGE PRINTEK (3O1

FIG 2 U.S. Patent Dec. 9, 2008 Sheet 3 of 78 US 7463,250 B2

C C D SEKVER SEKVEK 1O 1O S PAGE 11 12 PAGE P SERVE3 SEKVEK

E. F

APP FUb O SEKVEK SEKVEK

15 -6O1- 14. -6O1- E -6O1 U.S. Patent Dec. 9, 2008 Sheet 4 of 78 US 7463,250 B2

r as m as a a is a as ------NETF?AGE PAGE SEKVEK -1O- CONTENT Ob JECT

k

k GRAPHIC ELEMENT b

DOCUMENT C k PAGE C * INTERACTIVE DE6CKFTION DESCRIPTION ELEMENT

C FAGE C FIELD

INSTANCE VALUE ?AGE ID 5O ------as a m up

is as .2 ...... m in H - - - - - DOCUMENTNETFAGE C NETAGE K> A-25AAVISILE 45 f k NVISI3LE 25 CODED DATA O 5O k 4. NVISIE3LE125/ PAGE ID PHYSICAL WOKLD POSITION U.S. Patent Dec. 9, 2008 Sheet 5 Of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 6 of 78 US 7463,250 B2

FIG. (3b

U.S. Patent Dec. 9, 2008 Sheet 8 of 78 US 7463,250 B2

ACGUIKE IMAGE 2O DECODED 259 & SET THRESHOLD DATA

MAGE & 21 KEED-SOLOMON THRESHOLD DECODE 33

THKESHOLD ENCODED 257 & SEGMENT DATA

SHAPE SAM?LE LIST DATA FOINTS

COMF?UTE MOMENTS

SHAPE INFEK MOMENTS 3D PEN XFOKM

FIND f?EKSFECTIVE KING TAKGET XFOKM 2,25

KOTATION NFEK & ASFECT PERSPECTIVEXFORMhy - 32

FIND CORNEK 31 AXIS TAKGET FOINTS

FIND PERSPECTIVE 3O TAKGETS FIG. 7 U.S. Patent Dec. 9, 2008 Sheet 9 Of 78 US 7463,250 B2

1O1 -N1

1O5 U.S. Patent Dec. 9, 2008 Sheet 10 Of 78 US 7463,250 B2

121

U.S. Patent Dec. 9, 2008 Sheet 12 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 13 Of 78 US 7463,250 B2 §§

&&&&& U.S. Patent Dec. 9, 2008 Sheet 14 of 78 US 7463,250 B2

(3O4

/ / / N N K Z [-]

ŽEZI Z

UZ No.,No.=?="}------^-No.D). RSNNNNNNNNNNNNNNXSNINCSNOENNINNNNNNXNNON

N 35O

FEFZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ N N ZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZZ (2O3 676 aÉ2 675 27O FIG. 12a U.S. Patent Dec. 9, 2008 Sheet 15 Of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 19 Of 78 US 7463,250 B2

U.S. Patent US 7463,250 B2

set 1

Ä

U.S. Patent Dec. 9, 2008 Sheet 21 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 22 of 78 US 7463,250 B2

V N) &

7 s 2 a At

1 Ey O O g 7. SR E

O

5O g7 ES7

U.S. Patent Dec. 9, 2008 Sheet 23 of 78 US 7463,250 B2

NETF?AGE USEK DKECTOKY ee O, DEFAULT

on so as so as v NETF?AGE k k FKINTEK FEN an as a ------

NETF?AGE CONTACT USEK (32O DETALS ACCOUNT NAME E3ALANCE ADDKESS TELEPHONE SET PAYMENT (321 GLOE5AL CARD 316 FKEFEKENCES ACCOUNT INFO FKIVACY FU3 SG KEY DELIVERY FKIV SG KEY LOCALIZATION SECRET

k HANDWKITING 317 3OMETRIC MODEL 322

SIGNATUKE FNGEK?KNT 219 313 3OMETRIC 3OMETKIC SIGNATURE FINGEKFKNT FIG 21 U.S. Patent Dec. 9, 2008 Sheet 24 of 78 US 7463,250 B2

NETF?AGE FKNTEK FKINTEK ID SECRET 31O FU3 SG KEY k LOCATION TAG (32 O) NETPAGE NETF?AGE 811 FKINTEK FKOV FKINTER k ACCOUNT FKOVIDEK AG 3ALANCE FK FKOVD

: "JEE: ;

CUKKENT SELECTION

PEN ID CAFTUKE TIME KEY-EXCH KEY REGION FAGE D

F1AGE

SHAPE SIZE COLOK TEXTURE FIG. 23 U.S. Patent Dec. 9, 2008 Sheet 25 of 78 US 7463,250 B2

NETFAGE NETF?AGE NETPAGE FU3LISHER D AFFLICATIONFKOVIDEK AFFACCOUNT FKOV AFF FKOVD 3ALANCE

3O3 227 k

f?UBLICATION D AFFLICATION AFFLICATION ID 3O7 CEKTIFICATE 64 3O4 k O, AFFLICATION FAVOKTE SUBSCKFTION USEK as APPLICATION FKEQUENCY FU3 SG KEY FKODUCT TYPE k FKV SG KEY k 3O3 k (32.3 3O2 3O5 3OO O,1 /-, A NETPAGE NETPAGE PRINTER UeER U.S. Patent Dec. 9, 2008 Sheet 26 of 78 US 7463,250 B2

FAGE 63 RESOURCE CONTENT O1 SERVER DESCRIPTION ObjFCT s SEKVEK ID :------OBJECT ID O,1 k 341 34O DOCUMENT k> * P93 MENT DOCUMENT ID AFF ID /N USEK ID s - r 2,256 b1 TERMINAL ELEMENT ; ELEMENT ------r 334 2,253 2255 2,259 k k 9SMAEP K. k FAGE C * FOKMATTED * DOCUMENT DESCRIPTION ELEMENT

SCALE FACTOR ZONE USEK. D 52 k POCUMENT KS FAGE C ELEMENT INSTANCE INSTANCE INSTANCE

DOC INST 1D s PAGE ID CKEATION TIME CREATION TIME USERD USEK ID FKINTEK D FKINTERD BACKGKOUND FIELD 3,251 33O 3r DIGITALINK

TAG 2,2525

------U.S. Patent Dec. 9, 2008 Sheet 27 Of 78 US 7463,250 B2

------as a is ------NETPAGE O.1-- k DOCUMENT k- - - - - APPLICATION ... Vf....} : .------. . 836. . NN. ; 2OO k: 8O4 : - FORMATTEDDOCUMENT Y 8,254 ------2,251 - DOCUMENT -91. NETPAGE INSTANCE PRINTER ; ::::::::::::: : ------FAGE 2O2 : INSTANCE ------35O

i ELEMENTSTATIC Y (343

------HYPERLINK ELEMENT - 344 TEKMINAL as as an as a a ELEMENT a a as as an a ELEMENTFIELD Y 345 359 is ......

as a se - as as FAGEdOMMAND SEKVEK N- 346 ELEMENT as a U.S. Patent Dec. 9, 2008 Sheet 28 of 78 US 7463,250 B2

STYLE k STYLE ELEMENT O3JECT

247 STYLED 354. TEXTFLOW k TEXT ELEMENT Ob JECT TEXTFLOW ID 343 265

MAGE k IMAGE ELEMENT ObJECT STATIC { 349 356 ELEMENT GRAPHC k GKAPHIC ELEMENT OBUECT 343 25O 257 VIDEO CLIF VIDEO CLIF ELEMENT Ob JECT 251 25.3 AUDIO CLIF AUDIO CLIF? ELEMENT O3JECT 352 259 6CKF k 6CKIFT ELEMENT ObJECT 353 36O FIG. 23 U.S. Patent Dec. 9, 2008 Sheet 29 Of 78 US 7463,250 B2

TERMINAL ELEMENT APPLICATION ID \ LINK D ee ALIAS KEQD DESCRIPTION ?f ------FOKMATTED FORMATTED HY?EKLINK ELEMENT

361 3.35

HY?ERLINK ELEMENT INSTANCE : INSTANCE TrANep <---- 3.252 262 66

263

HYPERLINK ELEMENT 364

24, FOKM HY?EKLINK 865

FIG. 3O U.S. Patent Dec. 9, 2008 Sheet 30 of 78 US 7463,250 B2

------r ------GROUP HY?EKLINKED HYPERLINK ELEMENT GKOUF ELEMENT

p as ...... m as a ELEMENTGROUP - 33-3 ------a- or a

r s is a - - - - p ------D. TERMINALELEMENT APPLICATION ID ------LIFETIMEFOKNMD es FORMATTED 227 NEXT ELEMENT VEkelON N ------2,255

ELEMENT INSTANCE INSTANCE ------v 232

a a is p on an as a

FIELD 271 VALUE

------DIGITALINK

FIG. 32 U.S. Patent Dec. 9, 2008 Sheet 31 Of 78 US 7463,250 B2

DIGITAL STKOKE NK GKOUF CKEATION TIME 273 O 274 k as as as ...... s

O 275 k

FEN

TIME OFFSET XY FOSITION XYZ KOTATION Z FOKCE

a

FIELD as a ......

as a

FIELD a ELEMENT as s ------a FIELD NAME DRAWINGFIELD -u 279 HIDDEN as a as an as s - a

e s see s a eIGNATUREFIELD N- 83O

as as ------U.S. Patent Dec. 9, 2008 Sheet 32 of 78 US 7463,250 B2

- u CHECKE3OX FIELD FIELD ELEMENT a77 321 a 45 37 CHECK5OX ------FIELD FIELD VALUE : VALUE BOOLEAN

FIELD ELEMENT 71

FIELD VALUE

FIELD : ELEMENT

2 37 SIGNATURE ------FIELD

WEL2 VALUE as m ------so as - d. DIGITAL 3G U.S. Patent Dec. 9, 2008 Sheet 33 of 78 US 7463,250 B2

KECEIVE STKOKE

IDENTIFY PAGE

KETKIEVE PAGE

IDENTIFY PAGE ELEMENT 333 AFFEND STKOKE TO FIELD DIGITAL INK 239

INTERPRET DIGITALINK 390

AFFEND STKOKE TO 5ACKGROUND FIELD DIGITALINK YES

STKOKE TO ACTIVATE SELECTION HY?EKLINK 2391 FIG. 33 U.S. Patent Dec. 9, 2008 Sheet 34 of 78 US 7463,250 B2

r is is as so as a as as a a e s - - - - - is

396 SeO2 N

&GNAEYSE

903 YES CONVERT VERIFY

DIGITAL INK DIGITAL INKAS TO TEXT SIGNATURE

ASSIGN ASSGN CREATE TKUE AS TEXT AS DIGITAL FIELD VALUE FIELD VALUE 904 6IGNATURE O1 ASSGN DIGITAL 905 SIGNATURE AS FIELD VALUE

-3925

is ...... a as a es so e s is as as ...... as - - -

FIG 5.3a U.S. Patent Dec. 9, 2008 Sheet 35 of 78 US 7463,250 B2

PAGE SEKVER COMMAND SO6 6. 90.3 ELEMENT HIDDEN DUFLICATE FOKM 909 PAGE SEKVEK COMMAND 907 KESET ON SELECTED FOKNM 91O

FOKM GET FOKNM 911 COMMAND STATUS

DU?LICATE F?AGE 912

RESET FAGE 913

PAGE GET FAGE COMMAND STATUS 914

DUPLICATE DOCUMENT 915

KESET DOCUMENT 916

GET DOCUMENT DOCUMENT 917 COMMAND STATUS FIG. 39 U.S. Patent Dec. 9, 2008 Sheet 36 of 78 US 7463,250 B2

is e s m KESOURCE DESCKFTION

h mo d

Eitor

eub.JECT ------EEE

DESCKFTION CONTENT ACCOUNT PUBLISHEK OBJECT CONTRIBUTOK 3ALANCE DATE TYPE FOKMAT IDENTIFIEK SOURCE LANGUAGE KIGHTS

KELATION HOLDEK COVERAGE HOLDEK D

m ------U.S. Patent Dec. 9, 2008 Sheet 37 of 78 US 7463,250 B2

- a ------a NETPAGE FAVORITES User LIST 922 ------O.1 KO) k * FAVORITES ENTKY 92.25 DE6CKFTION /N 925

FAVOKITES FAVORITE FOLDER ITEM O) /N

FAVORITE FAVOKITE FAVOKITE DOCUMENT FAGE HY?EKLINK DOC INST ID FAGE D AFFLICATION ID LINK D k k ALIAS REQD 926 927 TKANS D

DOCUMENT FAGE S23 ; INe TANCE INs TANCE 2-y------, ------2,251 2,25O U.S. Patent Dec. 9, 2008 Sheet 38 of 78 US 7463,250 B2

a NETPAGE HSTOKY USER LIST 929

------O) 3OO

HISTOKY 95O ITEM DESCRIFTION CKEATION TIME

HISTOKY HISTORY HISTOKY DOCUMENT PAGE HY?EKLINK

DOC INST 1D PAGE D AFFLICATION ID LINK D ALAS KEQD TKANSID

------a DOCUMENT FAGE 933 INsTANCE INs TANCE sy------3.31 3,3O U.S. Patent Dec. 9, 2008 Sheet 39 Of 78 US 7463,250 B2

AFFLICATION D FAGE KEG (PUBLISHER) SEKVER SEKVEK SERVER FKINTEK

71 "GET D” 12 10 11 2O1

DOC ID (-51

AFF D, 1-64 ALIA5 D, 1–65 DOC ID, DOCUMENT, AFF ID, DIGITAL 6GNATURE, ALAS ID MULTICAST CHANNEL6

USER D, 1–6O PRINTER ID, -62 CEKTIFICATE

PAGES, MULTICAST CHANNEL3 5O-> PAGE D6

OBJECTs (MULTICAST)

PAGE -24 HY?EKLINK 262 HYPERLINK I sERVER KEGRUEST INSTANCE ------KEQUEST ID ------: NETPAGE --53ESM - NETPAGE : USER3 : FKINTERD PRINTER I-7 52 Vl 3OO 934 3O2 U.S. Patent Dec. 9, 2008 Sheet 40 of 78 US 7463,250 B2

FAGE REG D PAGE fRINTEReek. A safe SERVER SEKVER 3

6 O1 PEN ID, 1Oa 11 71 12 FKINTERD FAGE ID, CLICK (APPÖ). (31

USER D 4-6O (ALIA5ID)4-65

53->5ERVERID, 52->HREQD, 21 NSB, "GET D” 55-> (TRANs ID) ALIA5 ID)

DOC ID 4-51

SERVER ID, HKEQID, APP ID, DOC ID, DOC HKEQID AFP D

USER D, PRINTER D 4-62 bO-> FAGE IDS

PAGES FIG 45 U.S. Patent Dec. 9, 2008 Sheet 41 of 78 US 7463,250 B2

PAGE KEG D PAGE FRINTER ek. A eak APP SEKVEk SEKVEK 3

FEN ID, 1Oa 11 71 12 PRINTERD PAGE ID, CLICK 61 64

ALASUSERIE-69 it. (-65

3EKVER ID, 52-> HREQ D, b4-> LINK ID, 55-> TRANe D, ALIAS D ALIA5 ID, 56-> FORMID, FORM DATA CEKTIFICATE

"GET D"

DOC D 4-51

sERVER D, HREQ D, AFF D, DOC ID, DOC HREGRID AFP D USER D, FKINTER ID - 62 5O-D FAGE DS

PAGES FIG. 46 U.S. Patent Dec. 9, 2008 Sheet 42 of 78 US 7463,250 B2

AFF AFP SS25 || effVEsFAGE || sizeKEG ||P2EFR PROBEs.FKNTEK 7Oa HKEQID, 1O 72 CKEDTA

53->(24->AID, 2EEyERID, 52- HKEQID,

CREDIT A SERVERID,

CKEDTASEQIP

HREQ D, CKEDT 3

62-> PRINTERID, CKEDIT 3

CKEDIT 5 FIG, 47 U.S. Patent Dec. 9, 2008 Sheet 43 of 78 US 7463,250 B2

U.S. Patent US 7463,250 B2

FIG. 5O U.S. Patent Dec. 9, 2008 Sheet 45 of 78

U.S. Patent Dec. 9, 2008 Sheet 47 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 48 of 78 US 7463,250 B2

543 552 FIG 5.3 U.S. Patent Dec. 9, 2008 Sheet 49 of 78 US 7463,250 B2

564

> 6

FIG. (2O U.S. Patent Dec. 9, 2008 Sheet 50 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 51 of 78 US 7463,250 B2

optical axis

surface U.S. Patent Dec. 9, 2008 Sheet 52 of 78 US 7463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 53 of 78 US 7463,250 B2

physical Optical axis axis

tilted surface U.S. Patent Dec. 9, 2008 Sheet 54 of 78 US 7463,250 B2

physical axis

U.S. Patent Dec. 9, 2008 Sheet 55 of 78 US 7463,250 B2

16O

191 U.S. Patent Dec. 9, 2008 Sheet 56 of 78 US 7463,250 B2

STAKT NEW STKOKE

SAM?LE NE3 FOKCE

166

i.

CAFTUKE 167 MAGE YES

LOCATE ENCKYFT

163 TAG STKOKE 173

DECODE TKANSMT 169 TAG STKOKE s 174

INFEK ?EN 17O TKANSFORM

AFFEND 171 rulesTO STKOKE FIG. (23 U.S. Patent Dec. 9, 2008 Sheet 57 Of 78 US 7463,250 B2

KAW 132 DIGITAL TENME 176 NK CHANGE

FEN ID A36 TIME N3 D ABS TIME LAST SYS TIME KEGION O) CHANGE 18O k KEGION ID 1.33 N3 CHANGE 176 O) NE3 ID k KAW 134 STKOKE ( TAG 17.3 COMFONENT CHANGE TAG D

UNKNOWN FEN 179 FOSITION

KAW FEN 177 FOSITION XY OFFSET XYZ KOTATION Z FORCE

FIG. (29 U.S. Patent Dec. 9, 2008 Sheet 58 Of 78 US 7463,250 B2

EO NUMBER EGUATION

1

2

3

2 ) a 63.4 a 1.11 radians 5 10+2/5

2 10+2 = 10.K + 2

- k B = 2arcsinarcsing, -

FIG. 7O U.S. Patent Dec. 9, 2008 Sheet 59 of 78 US 7463,250 B2

EO NUMBER EGUATION

d = D - Stan 0

dcos 0(tan (0+ O.) - tan (0- O))

dcos 0(tan (0 + O.) - tan (6 - O))2 m

dcos 0 17 cos (0 + O.)

() falcos 0 cos(10 + O.)

fD

COS O, FIG 71 U.S. Patent Dec. 9, 2008 Sheet 60 of 78 US 7463,250 B2

EOR NUMBER EQUATION

q = 2c'ncocos (10' + O.')

- 20 nfdcosocos (10' + () cos(10 + O.)

20nfalcos 0coso. q F 2 cos (0 + O.)

2O.nfd

cos 0(1 - tan|0| tano)

q = 20..nfdcos 0 FIG. 72 U.S. Patent Dec. 9, 2008 Sheet 61 of 78 US 7463,250 B2

EQ NUMBER EQUATION

28

29 -) O pivot - sensed pivot sensed S R - D +d

3 O

-) R 31 Vcontact-pivotpivot RielR

-) -> -> 32 Vcontact - sensed Vcontact - pivot Vpivot - sensed

-> -X -1 ) 33 contact - tag Vsensed- tag M contact - sensed

34 P contact tag -- -Y contact - tag FIG. 75 U.S. Patent Dec. 9, 2008 Sheet 62 of 78 US 7463,250 B2

NUMBERusical coustonEGUATION

We < WT U.S. Patent Dec. 9, 2008 Sheet 63 of 78 US 7463,250 B2

Z O s D C U.S. Patent Dec. 9, 2008 Sheet 64 of 78 US 7463,250 B2

Z O

Ds Q C U.S. Patent 9 463,250 B2

U.S. Patent Dec. 9, 2008 Sheet 69 of 78 US 7463,250 B2

EG NUMBER EQUATION

63 aix + biy + ci P2 dix + eiy + fi gix + hiy + i FIG. 31 U.S. Patent Dec. 9, 2008 Sheet 70 of 78 US 7463,250 B2

E NUMBER EGUATION

S

i m O 65 S -

l S

dii- DE 67 S

62 l s

= AD E + BCE

gi = J(DFG-CH) 7O

71

72

73

4 - eanino one FIG. 32 U.S. Patent Dec. 9, 2008 Sheet 71 Of 78 US 7463,250 B2

EO NUMBER EQUATION ci S A (CG + DFH) + B(CFH - DG)

C E 76

7 7 fS = A(DE) + B(CE)

78 f = Ad+ Be

7 9 i = AJ(DFG-CH) + BJ (DH + CFG) + IJ + 1 i = Agi + Bhi + IJ + 1

A =

84 ca-Bhd - of-bae B(ae - ba) = af-cd

B c U.S. Patent Dec. 9, 2008 Sheet 72 of 78 US 7463,250 B2

EQ NUMBER EQUATION

A = f(ae - ba) - e (af-cd) 88 d(ae - ba)

A = aef-bdf-aef-- cale d(ae - bd) A = -bdf+ cde d(ae - ba) - - bf+ ce

d 62

U.S. Patent Dec. 9, 2008 Sheet 74 of 78 US 7463,250 B2

ECR NUMBER EQUATION U.S. Patent Dec. 9, 2008 Sheet 75 Of 78 US 7463,250 B2

E

1 21

1 22

1 23

1 24

1 25

d d+ be

126 sign (FH)FH) = sign (;ae - b

127 --CH CH+ strippFGsign (FH) DFG

J hi 128 DH + sign(FH)CFG (i-Agi - Bhi - 1) 129 re J U.S. Patent Dec. 9, 2008 Sheet 76 of 78 US 7463,250 B2

s

CO CO of CO 8

to C C C Cott. It

d C d d C v- ve U U Y Y v U U

O ve O y- O y + d y + ? y H d e O Ho C Ho C H C C Ho C c. elessO C) CS O leases() CS C O (VS O 9 U.S. Patent Dec. 9, 2008 Sheet 77 of 78 US 7463,250 B2

zero pitch & Zero roll

Zero rol

handle via 2a or 2b

H 6-0 F (- sign() w1 - E2 FIG. 39 U.S. Patent Dec. 9, 2008 Sheet 78 of 78 US 7463,250 B2

zero pitch

non-zero pitch (g 70) M (h 7: 0) handle via general solution & non-zero roll FIG. 90 US 7463,250 B2 1. 2 SYSTEM FOR DETERMINING THE The disclosures of these co-pending applications are incor ROTATIONAL ORIENTATION OF A SENSING porated herein by cross-reference. DEVICE Various methods, systems and apparatus relating to the present invention are disclosed in the following co-pending CROSS REFERENCE TO RELATED applications filed by the applicant or assignee of the present APPLICATIONS invention on 30 Jun. 2000: The present application is a continuation of U.S. applica tion Ser. No. 09/722,088 filed on Nov. 25, 2000, which is a 6824044 6678499 697622O 6976O3S 6766942 72861.13 continuation-in-part of U.S. application Ser. No. 09/575,168 10 6922779 6978019 O9,607843 6959298 6973450 71SO404 filed May 23, 2000, now issued as U.S. Pat. No. 6,737.591, 6965882 7233924 7007851 6957921 6457883 683.1682 the entire contents of which are herein incorporated by refer 6977751 6398332 63.94573 6622923 CCC. The disclosures of these co-pending applications are incor FIELD OF INVENTION 15 porated herein by cross-reference. Various methods, systems and apparatus relating to the The present invention relates generally to methods, sys present invention are disclosed in the following co-pending tems and apparatus for interacting with computers. The applications filed by the applicant or assignee of the present invention relates particularly to a sensing device for sensing invention on 23 May 2000: its own orientation relative to a Surface when moved or posi tioned relative to the surface. The invention has been developed primarily to allow a large number of distributed users to interact with networked 64281.33 6526658 6315399 6338548 6S4O319 6328431 63.28425 6991.32O 63.83833 6464,332 6390591 7018O16 information via printed matter and optical sensors, thereby to 63.28417 O9.575197 7079712 6825945 7330974 6813O39 obtain interactive printed matter on demand via high-speed 25 69875.06 7038797 698O318 6816274 7102772 735O236 networked color printers. Although the invention will largely 6.681045 6728OOO 7173722 708.8459 O9.575181 7068382 be described herein with reference to this use, it will be 7062651 67891.94 6789191 6644642 6SO2614 6622999 666938S 6549.935 6987573 6727996 659 1884 64397O6 appreciated that the invention is not limited to use in this field. 676O119 729S332 629O349 64281SS 6785O16 68.70966 6822639 6737591 7055739 723332O 683O196 6832717 CO-PENDING APPLICATIONS 30 69.57768 O9.575172 7170499 7106888 71.23239 6409323 6281912 660481O 631892O 6488422 6795.215 7154638 Various methods, systems and apparatus relating to the 6859289 present invention are disclosed in the following co-pending applications filed by the applicant or assignee of the present The disclosures of these co-pending applications are incor invention simultaneously with the present invention: 35 porated herein by cross-reference. BACKGROUND

6530339 6631897 7295.839 09,722174 7175079 7O64851 68.26547 6741871 6927871 698O306 696.5439 6788982 Presently, a user of a computer system typically interacts 40 726327O 6788293 6946672 7091960 679216S 7105753 with the system using a monitor for displaying information 7182247 and a keyboard and/or mouse for inputting information. Whilst such an interface is powerful, it is relatively bulky and non-portable. Information printed on paper can be easier to The disclosures of these co-pending applications are incor read and more portable than information displayed on a com porated herein by cross-reference. 45 puter monitor. However, unlike a keyboard or mouse, a pen on Various methods, systems and apparatus relating to the paper generally lacks the ability to interact with computer present invention are disclosed in the following co-pending software. applications filed by the applicant or assignee of the present invention on 20 Oct. 2000: OBJECT 50 It is an object of the present invention to combine advan 7190474 71101.26 6813558 6965.454 6847883 7131058 tages of pen on paper and computer systems. 09 693690 6474888 662787O 6724.374 O9,693514 6454482 68O833O 6527365 6474773 655.0997 SUMMARY OF INVENTION 55 The present invention relates to a system comprising a The disclosures of these co-pending applications are incor sensing device and a surface having coded data disposed porated herein by cross-reference. thereon, the system being arranged for determining at least Various methods, systems and apparatus relating to the two dimensions of the rotational orientation of the sensing present invention are disclosed in the following co-pending 60 device relative to the surface, wherein: applications filed by the applicant or assignee of the present the coded data is indicative, when sensed by the sensing invention on 15 Sep. 2000: device, of the at least two dimensions of the rotational orientation of a sensing device; and the sensing device includes: 667942O 696384.5 6995859 6720985 65 an orientation sensor configured to generate, when posi tioned or moved relative to the Surface, using at least Some of the coded data, orientation data indicative of US 7463,250 B2 3 4 the at least two dimensions of the orientation of the hand-drawn information (whether drawing or writing) sensing device relative to the Surface; and applied by a user via the device. Preferably, the sensing device a transmitter for transmitting the orientation data to a is arranged to provide device identification information computer system. which uniquely identifies the device. The computer system In another aspect of the invention, there is provided a 5 may therefore use this to identify the device. sensing device for generating orientation data when posi Features and advantages of the present invention will tioned or moved relative to a surface, the orientation data become apparent from the following description of embodi being indicative of an orientation of the sensing device rela ments thereof, by way of example only, with reference to the tive to the Surface, the Surface having coded data disposed accompanying drawings. upon it, the coded data being indicative, when sensed by the 10 sensing device, of the orientation, the sensing device includ BRIEF DESCRIPTION OF DRAWINGS ing: a housing: Preferred and other embodiments of the invention will now orientation sensing means configured to generate the ori be described, by way of non-limiting example only, with entation data using at least some of the coded data; and 15 reference to the accompanying drawings, in which: communications means configured to communicate the FIG. 1 is a schematic of a the relationship between a sample orientation data to a computer system. printed netpage and its online page description; In a preferred embodiment the orientation data is indicative FIG. 2 is a schematic view of a interaction between a of the yaw, pitch and/or roll of the housing relative to the netpage pen, a netpage printer, a netpage page server, and a Surface. netpage application server, Preferably, the sensing device includes motion sensing FIG.3 illustrates a collection of netpage servers and print means for generating movement data when the sensing device ers interconnected via a network; is moved relative to the Surface, the communications means FIG. 4 is a schematic view of a high-level structure of a being configured to communicate the movement data to the printed netpage and its online page description; computer system. 25 FIG. 5a is a plan view showing a structure of a netpage tag: Preferably also, the sensing device includes region identity FIG.5b is a plan view showing a relationship between a set sensing means configured to sense, when the sensing device of the tags shown in FIG. 5a and a field of view of a netpage is positioned or moved relative to a region of the Surface, and sensing device in the form of a netpage pen; using at least Some of the coded data, region identity data FIG. 6a is a plan view showing an alternative structure of a indicative of an identity of the region, the communications 30 netpage tag, means being configured to communicate the region identity FIG. 6b is a plan view showing a relationship between a set data to the computer system. of the tags shown in FIG. 6a and a field of view of a netpage The orientation sensing means preferably detects the ori sensing device in the form of a netpage pen; entation of the housing relative to the Surface dynamically as FIG. 6c is a plan view showing an arrangement of nine of the housing is moved. The housing may have an elongate 35 the tags shown in FIG. 6a where targets are shared between shape which can be held by a user. In one embodiment, the adjacent tags; housing has the shape of a pen. The housing may be provided FIG. 6d is a plan view showing the interleaving and rota with a marking nib for marking the Surface, but this is not tion of the symbols of the four codewords of the tag shown in essential. FIG. 6a, By simultaneously capturing orientation and movement 40 FIG. 7 is a flowchart of a tag image processing and decod data the system may be used to Verify a person's signature. ing algorithm; Alternatively, dynamically-measured orientation signals can FIG. 8 is a perspective view of a netpage pen and its enable the housing to be used as a joystick. For example, Such associated tag-sensing field-of-view cone; a joystick could be used with three-dimensional software FIG. 9 is a perspective exploded view of the netpage pen applications. Note that it is not essential for the orientation 45 shown in FIG. 8: sensing means to sense the orientation of the housing in all FIG.10 is a schematic block diagram of a pen controller for three dimensions. It may be sufficient to detect only the pitch, the netpage pen shown in FIGS. 8 and 9: as some applications may not need three-dimensional orien FIG. 11 is a perspective view of a wall-mounted netpage tation information. For example, the housing may be used to printer; linearly control an aspect of a device. Such as the intensity of 50 FIG. 12 is a section through the length of the netpage a light or the Volume of a loudspeaker, by varying the pitch printer of FIG. 11; between 0° and 90°. FIG. 12a is an enlarged portion of FIG. 12 showing a The roll, pitch and yaw may be calculated by detecting section of the duplexed print engines and glue wheel assem perspective distortion and rotation of the coded data. bly: Firstly, it can be used to determine when the apparatus is 55 FIG. 13 is a detailed view of the ink cartridge, ink, air and first applied to the surface and when it leaves the surface, with glue paths, and print engines of the netpage printer of FIGS. motion between force application and removal being defined 11 and 12; as a stroke in freehand. The force data information can be FIG.14 is a schematic block diagram of a printer controller time stamped. for the netpage printer shown in FIGS. 11 and 12; The apparatus is preferably a separate implement contain 60 FIG. 15 is a schematic block diagram of duplexed print ing the appropriate means as discussed above. It may be any engine controllers and MemjetTM printheads associated with shape but it is preferably in the form of a stylus or pen. the printer controller shown in FIG. 14; Preferably, the apparatus incorporates a marking nib for FIG. 16 is a schematic block diagram of the print engine marking the Surface with hand-drawn information, but this is controller shown in FIGS. 14 and 15; not essential. 65 FIG.17 is a perspective view of a single MemjetTM printing The apparatus is preferably intended for interaction with a element, as used in, for example, the netpage printer of FIGS. computer system that can be controlled and can interpret 10 to 12: US 7463,250 B2 5 6 FIG. 18 is a perspective view of a small part of an array of FIG. 56 is a schematic view of a minimum tag spacing MemjetTM printing elements; which avoids overlap: FIG. 19 is a series of perspective views illustrating the FIG. 57 is a schematic view of a first tag insertion case: operating cycle of the MemjetTM printing element shown in FIG.58 is a schematic view of a second tag insertion case: FIG. 13; 5 FIG. 59 is a schematic view of a third tag insertion case: FIG. 20 is a perspective view of a short segment of a FIG. 60 is a schematic view of a fourth tag insertion case: pagewidth MemjetTM printhead; FIG. 61 is a schematic view of a pen orientation relative to FIG. 21 is a schematic view of a user class diagram; a Surface; FIG. 22 is a schematic view of a printer class diagram; FIG. 62 is a schematic view of a pen pitch geometry: FIG. 23 is a schematic view of a pen class diagram; 10 FIG. 63 is a schematic view of a pen roll geometry; FIG. 24 is a schematic view of an application class dia FIG. 64 is a schematic view of a pen coordinate space gram, showing physical and optical axes of a pen; FIG. 25 is a schematic view of a document and page FIG. 65 is a schematic view of a curved nib geometry: description class diagram; FIG. 66 is a schematic view of an interaction between FIG. 26 is a schematic view of a document and page own 15 sampling frequency and tag frequency; ership class diagram; FIG. 67 is a schematic view of a pen optical path; FIG. 27 is a schematic view of a terminal element special FIG. 68 is a flowchart of a stroke capture algorithm; and ization class diagram; FIG. 69 is a schematic view of a raw digital ink class FIG. 28 is a schematic view of a static element specializa diagram. tion class diagram; FIG.70 is a table containing equations numbered 1 to 10; FIG. 29 is a schematic view of a hyperlink element class FIG.71 is a table containing equations numbered 11 to 20; diagram; FIG.72 is a table containing equations numbered 21 to 26: FIG. 30 is a schematic view of a hyperlink element spe FIG. 73 is a table containing equations numbered 27 to 34; cialization class diagram; FIG.74 is a table containing equations numbered 35 to 41; FIG. 31 is a schematic view of a hyperlinked group class 25 FIG. 75 is a table containing equations numbered 42 to 44; diagram; FIG. 76 is a table containing equations numbered 45 to 47: FIG. 32 is a schematic view of a form class diagram; FIG. 77 is a table containing equations numbered 48 to 51: FIG.33 is a schematic view of a digital ink class diagram; FIG. 78 is a table containing equations numbered 52 to 54; FIG. 34 is a schematic view of a field element specializa FIG. 79 is a table containing equations numbered 55 to 57; tion class diagram; 30 FIG.80 is a table containing equations numbered 58 to 59: FIG. 35 is a schematic view of a checkbox field class FIG. 81 is a table containing equations numbered 60 to 63: diagram; FIG. 82 is a table containing equations numbered 64 to 74: FIG. 36 is a schematic view of a text field class diagram; FIG. 83 is a table containing equations numbered 75 to 86: FIG. 37 is a schematic view of a signature field class FIG. 84 is a table containing equations numbered 87 to 99; diagram; 35 FIG. 85 is a table containing equations numbered 100 to FIG.38 is a flowchart of an input processing algorithm; 111; FIG.38a is a detailed flowchart of one step of the flowchart FIG. 86 is a table containing equations numbered 112 to of FIG.38: 120; FIG. 39 is a schematic view of a page server command FIG. 87 is a table containing equations numbered 121 to element class diagram; 40 129; FIG. 40 is a schematic view of a resource description class FIG. 88 is a table containing a set of degenerate forms of diagram; equations 64 to 71, FIG. 41 is a schematic view of a favorites list class diagram; FIG. 89 is a first part of a table containing conditions and FIG. 42 is a schematic view of a history list class diagram; special handling for Zero pitch and Zero roll; and FIG. 43 is a schematic view of a subscription delivery 45 FIG.90 is the second part of the table of FIG. 89. protocol; FIG. 44 is a schematic view of a hyperlink request class DETAILED DESCRIPTION OF PREFERRED diagram; AND OTHER EMBODIMENTS FIG. 45 is a schematic view of a hyperlink activation pro 50 Note: MemjetTM is a trade mark of Silverbrook Research tocol; Pty Ltd, Australia. FIG. 46 is a schematic view of a form submission protocol; In the preferred embodiment, the invention is configured to FIG. 47 is a schematic view of a commission payment work with the netpage networked computer system, a detailed protocol; overview of which follows. It will be appreciated that not FIG. 48 is a schematic view of a set of radial wedges 55 every implementation will necessarily embody all or even making up a symbol; most of the specific details and extensions discussed below in FIG. 49 is a schematic view of a ring A and B symbol relation to the basic system. However, the system is described allocation scheme; in its most complete form to reduce the need for external FIG.50 is a schematic view of a first ring C and D symbol reference when attempting to understand the context in which allocation scheme; 60 the preferred embodiments and aspects of the present inven FIG. 51 is a schematic view of a second ring C and D tion operate. symbol allocation scheme: Inbrief summary, the preferred form of the netpage system FIG. 52 is a schematic view of a triangular tag packing; employs a computer interface in the form of a mapped Sur FIG. 53 is a perspective view of an icosahedron; face, that is, a physical Surface which contains references to a FIG. 54 is a perspective view of an icosahedral geodesic 65 map of the Surface maintained in a computer system. The map with frequency 3: references can be queried by an appropriate sensing device. FIG.55 is a schematic view of a minimum tag spacing; Depending upon the specific implementation, the map refer US 7463,250 B2 7 8 ences may be encoded visibly or invisibly, and defined in such point of departure for the day. It also comes in tabletop, a way that a local query on the mapped Surface yields an desktop, portable and miniature versions. unambiguous map reference both within the map and among Netpages printed at their point of consumption combine different maps. The computer system can contain information the ease-of-use of paper with the timeliness and interactivity about features on the mapped surface, and Such information of an interactive medium. can be retrieved based on map references Supplied by a sens As shown in FIG. 2, the netpage pen 101 interacts with the ing device used with the mapped surface. The information coded data on a printed netpage 1 and communicates, via a thus retrieved can take the form of actions which are initiated short-range radio link 9, the interaction to a netpage printer. by the computer system on behalf of the operator in response The printer 601 sends the interaction to the relevant netpage to the operators interaction with the surface features. 10 page server 10 for interpretation. In appropriate circum In its preferred form, the netpage system relies on the stances, the page server sends a corresponding message to production of and human interaction with, netpages. These application computer software running on a netpage applica are pages of text, graphics and images printed on ordinary tion server 13. The application server may in turn send a paper, but which work like interactive web pages. Informa response which is printed on the originating printer. tion is encoded on each page using ink which is substantially 15 The netpage system is made considerably more convenient invisible to the unaided human eye. The ink, however, and in the preferred embodiment by being used in conjunction thereby the coded data, can be sensed by an optically imaging with high-speed microelectromechanical system (MEMS) pen and transmitted to the netpage system. Substrates other based inkjet. (MemjetTM) printers. In the preferred form of than paper may be used. The encoded information in the this technology, relatively high-speed and high-quality print preferred embodiment is an infrared absorptive ink and so an ing is made more affordable to consumers. In its preferred infrared sensitive optical sensor may be used. If desired other form, a netpage publication has the physical characteristics of wavelengths may be used or sensing techniques other than a traditional newsmagazine. Such as a set of letter-size glossy optical sensing; one alternative is to use magnetic inks and pages printed in full color on both sides, bound together for SSOS. easy navigation and comfortable handling. In the preferred form, active buttons and hyperlinks on 25 The netpage printer exploits the growing availability of each page can be clicked with the pen to request information broadband Internet access. Cable service is available to 95% from the network or to signal preferences to a network server. of households in the United States, and cable modem service In one embodiment, text written by hand on a netpage is offering broadband Internet access is already available to automatically recognized and converted to computer text in 20% of these. The netpage printer can also operate with the netpage system, allowing forms to be filled in. In other 30 slower connections, but either with longer delivery times or embodiments, signatures recorded on a netpage are automati lower image quality or both. Indeed, the netpage system can cally verified, allowing e-commerce transactions to be be enabled using existing consumer inkjet and laser printers, securely authorized. although the system will operate more slowly and will there As illustrated in FIG. 1, a printed netpage 1 can representa fore be less acceptable from a consumer's point of view. In interactive form which can be filled in by the user both physi 35 other embodiments, the netpage system is hosted on a private cally, on the printed page, and "electronically, via commu intranet. In still other embodiments, the netpage system is nication between the pen and the netpage system. The hosted on a single computer or computer-enabled device, example shows a “Request' form containing name and Such as a printer. address fields and a Submit button. The netpage consists of Netpage publication servers 14 on the netpage network are graphic data 2 printed using visible ink, and coded data 3 40 configured to deliver print-quality publications to netpage printed as a collection of tags 4 using invisible ink. The printers. Periodical publications are delivered automatically corresponding page description 5, stored on the netpage net to Subscribing netpage printers via pointcasting and multi work, describes the individual elements of the netpage. In casting Internet protocols. Personalized publications are fil particular it describes the type and spatial extent (Zone) of tered and formatted according to individual user profiles. each interactive element (i.e. text field or button in the 45 A netpage printer can be configured to support any number example), to allow the netpage system to correctly interpret of pens, and a pen can work with any number of netpage input via the netpage. The Submit button 6, for example, has printers. In the preferred implementation, each netpage pen a Zone 7 which corresponds to the spatial extent of the corre has a unique identifier. A household may have a collection of sponding graphic 8. colored netpage pens, one assigned to each member of the As illustrated in FIG. 2, the netpage pen 101, a preferred 50 family. This allows each user to maintain a distinct profile form of which is shown in FIGS. 8 and 9 and described in with respect to a netpage publication server or application more detail below, works in conjunction with a netpage server, assuming that the assigned pen is only used by the printer 601, an Internet-connected printing appliance for respective family member. However, as explained below, home, office or mobile use. The pen is wireless and commu other means may be used to identify a user. nicates securely with the netpage printer via a short-range 55 A netpage pen can also be registered with a netpage regis radio link9. If desired the pen may be connected to the system tration server 11 and linked to one or more payment card utilizing wires or an infrared transmitter, although both alter accounts. This allows e-commerce payments to be securely natives limit usability. authorized using the netpage pen. The netpage registration The netpage printer 601, a preferred form of which is server compares the signature captured by the netpage pen shown in FIGS. 11 to 13 and described in more detail below, 60 with a previously registered signature, allowing it to authen is able to deliver, periodically or on demand, personalized ticate the user's identity to an e-commerce server. Other bio newspapers, magazines, catalogs, brochures and other publi metrics can also be used to verify identity. A version of the cations, all printed at high quality as interactive netpages. netpage pen includes fingerprint scanning, verified in a simi Unlike a personal computer, the netpage printer is an appli lar way by the netpage registration server. ance which can be, for example, wall-mounted adjacent to an 65 Although a netpage printer may deliver periodicals such as area where the morning news is first consumed. Such as in a the morning newspaper without user intervention, it can be user's kitchen, near a breakfast table, or near the households configured never to deliver unsolicited junk mail. In its pre US 7463,250 B2 10 ferred form, it only delivers periodicals from subscribed or between all netpages envisaged to be used in the environment otherwise authorized sources. In this respect, the netpage ofuse. If the environment is small then the precision need not printer is unlike a fax machine or e-mail account which is be as great as where the environment is large. visible to any junk mailer who knows the telephone number Each reference to the page description is encoded in a oremail address. Alternatively the entire system may be made printed tag. The tag identifies the unique page on which it visible to outside users or each user may be provided with the appears, and thereby indirectly identifies the page descrip ability to expose their printer(s) to outside users. This may be tion. In the preferred embodiments the tag also identifies its by way of selecting outside users allowed too send junk mail. own position on the page. Characteristics of the tags are 1. Netpage System Architecture described in more detail below. Each object model in the system is described using a Uni 10 Tags are printed in infrared-absorptive ink on any substrate fied Modeling Language (UML) class diagram. A class dia which is infrared-reflective, such as ordinary paper. Near gram consists of a set of object classes connected by relation infrared wavelengths are invisible to the human eye but are ships, and two kinds of relationships are of interest here: easily sensed by a solid-state image sensor with an appropri associations and generalizations. An association represents ate filter. A sensor sensitive to the relative wavelength or some kind of relationship between objects, i.e. between 15 wavelengths may be used, in which case no filters are instances of classes. A generalization relates actual classes, required. Other wavelengths may be used, with appropriate and can be understood in the following way: if a class is Substrates and sensors. thought of as the set of all objects of that class, and class A is A tag is sensed by an area image sensor in the netpage pen, a generalization of class B, then B is simply a Subset of A. decoded and the data encoded by the tag is transmitted to the Each class is drawn as a rectangle labelled with the name of netpage system, preferably via the nearest netpage printer. the class. It contains a list of the attributes of the class, sepa The pen is wireless and communicates with the netpage rated from the name by a horizontal line, and a list of the printer via a short-range radio link. Tags are sufficiently small operations of the class, separated from the attribute list by a and densely arranged that the pen can reliably image at least horizontal line. In the class diagrams which follow, however, one tag even on a single click on the page. It is important that operations are never modeled. 25 the pen recognize the tag and extract the pageID and position An association is drawn as a line joining two classes, on every interaction with the page, since the interaction is optionally labelled at either end with the multiplicity of the stateless. Tags are error-correctably encoded to make them association. The default multiplicity is one. An asterisk (*) partially tolerant to Surface damage. indicates a multiplicity of “many’, i.e. zero or more. Each The netpage page server maintains a unique page instance association is optionally labelled with its name, and is also 30 for each printed netpage, allowing it to maintain a distinct set optionally labelled at either end with the role of the corre ofuser-supplied values for input fields in the page description sponding class. An open diamond indicates an aggregation for each printed netpage. association (is-part-of), and is drawn at the aggregator end The relationship between the page description, the page of the association line. instance, and the printed netpage is shown in FIG. 4. In the A generalization relationship (“is-a') is drawn as a solid 35 preferred embodiment the page instance is associated with line joining two classes, with an arrow (in the form of an open both the netpage printer which printed it and, if known, the triangle) at the generalization end. netpage user who requested it. It is not essential to the work When a class diagram is broken up into multiple diagrams, ing of the invention in its basic form that the page instance be any class which is duplicated is shown with a dashed outline associated with either the netpage printer which printed the in all but the main diagram which defines it. It is shown with 40 corresponding physical page or the netpage user who attributes only where it is defined. requested it or for whom the page was printed. 1.1 Netpages 1.2 Netpage Tags Netpages are the foundation on which a netpage network is 1.2.1 Tag Data Content built. They provide a paper-based user interface to published 45 In a preferred form, each tag identifies the region in which information and interactive services. it appears, and the location of that tag within the region. A tag A netpage consists of a printed page (or other Surface may also contain flags which relate to the region as a whole or region) invisibly tagged with references to an online descrip to the tag. One or more flag bits may, for example, signal a tag tion of the page. The tags may be printed on or into the Surface sensing device to provide feedback indicative of a function of the page, may be in or on a Sub-layer of the page or may be 50 associated with the immediate area of the tag, without the otherwise incorporated into the page. The online page sensing device having to refer to a description of the region. A description is maintained persistently by a netpage page netpage pen may, for example, illuminate an “active area’ server. The page description describes the visible layout and LED when in the Zone of a hyperlink. content of the page, including text, graphics and images. It As will be more clearly explained below, in a preferred also describes the input elements on the page, including but 55 embodiment, each tag contains an easily recognized invariant tons, hyperlinks, and input fields. The page descriptions of structure which aids initial detection, and which assists in different netpages may share components, such as an image, minimizing the effect of any warp induced by the Surface or although the netpages (and the associated page descriptions) by the sensing process. The tags preferably tile the entire are visibly different. The page description for each netpage page, and are Sufficiently Small and densely arranged that the may include references to these common components. A 60 pen can reliably image at least one tag even on a single click netpage allows markings made with a netpage pen on its on the page. It is important that the pen recognize the page ID Surface to be simultaneously captured and processed by the and position on every interaction with the page, since the netpage System. interaction is stateless. Multiple netpages can share the same page description. In a preferred embodiment, the region to which a tag refers However, to allow input through otherwise identical pages to 65 coincides with an entire page, and the region ID encoded in be distinguished, each netpage is assigned a unique page the tag is therefore synonymous with the page ID of the page identifier. This pageID has sufficient precision to distinguish on which the tag appears. In other embodiments, the region to US 7463,250 B2 11 12 which a tag refers can be an arbitrary Subregion of a page or of the effects of perspective distortion. An orientation axis 16 other Surface. For example, it can coincide with the Zone of an allows the sensing device to determine the approximate pla interactive element, in which case the region ID can directly nar orientation of the tag due to the yaw of the sensor. The identify the interactive element. orientation axis is skewed to yield a unique orientation. Four Each tag typically contains 16 bits oftag ID, at least 90 bits perspective targets 17 allow the sensing device to infer an of region ID, and a number of flag bits. Assuming a maximum accurate two-dimensional perspective transform of the tag tag density of 64 per square inch, a 16-bit tag ID Supports a and hence an accurate three-dimensional position and orien region size of up to 1024 square inches. Larger regions can be tation of the tag relative to the sensor. mapped continuously without increasing the tag ID precision All target structures are redundantly large to improve their simply by using abutting regions and maps. The distinction 10 immunity to noise. between a region ID and a tag ID is mostly one of conve The overall tag shape is circular. This Supports, amongst nience. For most purposes the concatenation of the two can be other things, optimal tag packing on an irregular triangular considered as a globally unique tag ID. Conversely, it may grid, such as is required to tile an arbitrary non-planar Surface. also be convenient to introduce structure into the tag ID, for The tags may, however, be arranged at the apexes of any example to define the X and y coordinates of the tag. A 90-bit 15 polygon having napexes, where n ranges from 3 to infinity, as region ID allows 2' (-107 or a thousand trillion trillion) desired. In combination with the circular detection ring 15. different regions to be uniquely identified. Tags may also this makes a circular arrangement of data bits within the tag contain type information, and a region may be tagged with a optimal. mixture of tag types. For example, a region may be tagged As shown in FIG. 48, to maximize its size, each data bit is with one set of tags encoding X coordinates and another set, represented by a radial wedge 510 in the form of an area interleaved with the first, encoding y coordinates. It will be bounded by two radial lines 512, a radially inner arc 514 and appreciated the region ID and tag ID precision may be more a radially outer arc 516. Each wedge 510 has a minimum or less than just described depending on the environment in dimension of8 dots at 1600dpi and is designed so that its base which the system will be used. (i.e. its inner arc 514), is at least equal to this minimum 1.2.2 Tag Data Encoding 25 dimension. The radial height of the wedge 510 is always equal In one embodiment each tag contains 120 bits of informa to the minimum dimension. Each 4-bit data symbol is repre tion. The 120 bits of tag data are redundantly encoded using sented by an array 518 of 2x2 wedges 510, as best shown in a (15, 5) Reed-Solomon code. This yields 360 encoded bits FIG. 48. consisting of 6 codewords of 154-bit symbols each. The (15, The 154-bit data symbols of each of the six codewords are 5) code allows up to 5 symbol errors to be corrected per 30 allocated to the four concentric symbol rings 18a to 18d. codeword, i.e. it is tolerantofa symbol error rate of up to 33% shown in FIG. 5, in interleaved fashion as shown in FIGS. 49 per codeword. to 51. Symbols of first to sixth codewords 520-525 are allo Each 4-bit symbol is represented in a spatially coherent cated alternately in circular progression around the tag. way in the tag, and the symbols of the six codewords are The interleaving is designed to maximize the average spa interleaved spatially within the tag. This ensures that a burst 35 tial distance between any two symbols of the same codeword. error (an error affecting multiple spatially adjacent bits) dam Other arrangements of the codewords or their data symbols ages a minimum number of symbols overall and a minimum may be utilized. number of symbols in any one codeword, thus maximizing The physical layout of the tags or the shape and/or arrange the likelihood that the burst error can be fully corrected. ment of data symbols within each tag are nor essential to the Any suitable error-correcting code can be used in place of working of the invention. It is merely necessary that each tag a (15, 5) Reed-Solomon code, for example a Reed-Solomon encode sufficient information for the intended use. The use of code with more or less redundancy, with the same or different redundancy in the tag is preferred but, at its basic level, not symbol and codeword sizes; another block code; or a different truly essential to the working of the invention. As such other kind of code, such as a convolutional code (see, for example, tag arrangements may be utilized. Examples of other tag Stephen B. Wicker, Error Control Systems for Digital Com 45 structures are described in U.S. Pat. Nos. 5,625,412, 5,661, munication and Storage, Prentice-Hall 1995, the contents of 506, 5,477,012 and 5,852,434, and PCT application PCT/ which a herein incorporated by cross-reference). US98/20597, the contents of each of which are incorporated herein by reference. 1.2.3 Physical Tag Structure In order to Support "single-click” interaction with a tagged The physical representation of the tag, shown in FIG. 5, 50 region via a sensing device, the sensing device must be able to includes fixed target structures 15, 16, 17 and variable data see at least one entire tag in its field of view no matter where areas 18. The fixed target structures allow a sensing device in the region or at what orientation the sensing device is Such as the netpage pen to detect the tag and infer its three positioned. The required diameter of the field of view of the dimensional orientation relative to the sensor. The data areas sensing device is therefore a function of the size and spacing contain representations of the individual bits of the encoded 55 of the tags. tag data. Assuming a circular tag shape, the minimum diameter of To achieve proper tag reproduction, the tag is rendered at a the sensor field of view is obtained when the tags are tiled on resolution of 256x256 dots. When printed at 1600 dots per a equilateral triangular grid, as shown in FIG. 6. inch this yields a tag with a diameter of about 4 mm. At this resolution the tag is designed to be surrounded by a "quiet 60 1.2.4 Tag Image Processing and Decoding area of radius 16 dots. Since the quiet area is also contributed The tag image processing and decoding of a tag of FIG. 5 by adjacent tags, it only adds 16 dots to the effective diameter performed by a sensing device Such as the netpage pen is of the tag. shown in FIG. 7. While a captured image is being acquired The tag includes six target structures. A detection ring 15 from the image sensor, the dynamic range of the image is allows the sensing device to initially detect the tag. The ring 65 determined (at 20). The center of the range is then chosen as is easy to detect because it is rotationally invariant and the binary threshold for the image 21. The image is then because a simple correction of its aspect ratio removes most thresholded and segmented into connected pixel regions (i.e. US 7463,250 B2 13 14 shapes 23) (at 22). Shapes which are too small to represent tag The ring target 15 is only sought in a Subarea of the image target structures are discarded. The size and centroid of each whose relationship to the image guarantees that the ring, if shape is also computed. found, is part of a complete tag. If a complete tag is not found Binary shape moments 25 are then computed (at 24) for and Successfully decoded, then no pen position is recorded for each shape, and these provide the basis for Subsequently the current frame. Given adequate processing power and ide locating target structures. Central shape moments are by their ally a non-minimal field of view 193, an alternative strategy nature invariant of position, and can be easily made invariant involves seeking another tag in the current image. of scale, aspect ratio and rotation. The obtained tag data indicates the identity of the region The ring target structure 15 is the first to be located (at 26). containing the tag and the position of the tag within the A ring has the advantage of being very well behaved when 10 region. An accurate position 35 of the pennib in the region, as perspective-distorted. Matching proceeds by aspect-normal well as the overall orientation 35 of the pen, is then inferred izing and rotation-normalizing each shape’s moments. Once (at 34) from the perspective transform 33 observed on the tag its second-order moments are normalized the ring is easy to and the known spatial relationship between the pen's physical recognize even if the perspective distortion was significant. axis and the pens optical axis. The ring's original aspect and rotation 27 together provide a 15 useful approximation of the perspective transform. 1.2.5 Alternative Tag Structures The axis target structure 16 is the next to be located (at 28). The tag structure just described is designed to allow both Matching proceeds by applying the ring's normalizations to regular tilings of planar Surfaces and irregular tilings of non each shape’s moments, and rotation-normalizing the result planar Surfaces. Regular tilings are not, in general, possible ing moments. Once its second-order moments are normalized on non-planar Surfaces. In the more usual case of planar the axis target is easily recognized. Note that one third order Surfaces where regular tilings of tags are possible, i.e. Sur moment is required to disambiguate the two possible orien faces such as sheets of paper and the like, more efficient tag tations of the axis. The shape is deliberately skewed to one structures can be used which exploit the regular nature of the side to make this possible. Note also that it is only possible to tiling. rotation-normalize the axis target after it has had the rings 25 An alternative tag structure more Suited to a regulartiling is normalizations applied, since the perspective distortion can shown in FIG. 6a. The alternative tag 4 is square and has four hide the axis targets axis. The axis targets original rotation perspective targets 17. It is similar in structure to tags provides a useful approximation of the tag's rotation due to described by Bennettet al. in U.S. Pat. No. 5,051,746. The tag pen yaw 29. represents sixty 4-bit Reed-Solomon symbols 47, for a total The four perspective target structures 17 are the last to be 30 of 240 bits. The tag represents each one bit as a dot 48, and located (at 30). Good estimates of their positions are com each Zero bit by the absence of the corresponding dot. The puted based on their known spatial relationships to the ring perspective targets are designed to be shared between adja and axis targets, the aspect and rotation of the ring, and the cent tags, as shown in FIGS. 6b and 6c. FIG. 6b shows a rotation of the axis. Matching proceeds by applying the rings square tiling of 16 tags and the corresponding minimum field normalizations to each shape’s moments. Once their second 35 of view 193, which must span the diagonals of two tags. FIG. order moments are normalized the circular perspective targets 6c shows a square tiling of nine tags, containing all one bits are easy to recognize, and the target closest to each estimated for illustration purposes. position is taken as a match. The original centroids of the four Usinga (15, 7) Reed-Solomon code, 112 bits oftag data are perspective targets are then taken to be the perspective-dis redundantly encoded to produce 240 encoded bits. The four torted corners 31 of a square of known size in tag space, and 40 codewords are interleaved spatially within the tag to maxi an eight-degree-of-freedom perspective transform 33 is mize resilience to burst errors. Assuming a 16-bit tag ID as inferred (at 32) based on solving the well-understood equa before, this allows a region ID of up to 92 bits. tions relating the four tag-space and image-space point pairs The data-bearing dots 48 of the tag are designed to not (see Heckbert, P., Fundamentals of Texture Mapping and overlap their neighbors, so that groups of tags cannot produce Image Warping, Masters Thesis, Dept. of EECS, U. of Cali 45 structures which resemble targets. This also saves ink. The fornia at Berkeley, Technical Report No. UCB/CSD 89/516, perspective targets therefore allow detection of the tag, so June 1989, the contents of which are herein incorporated by further targets are not required. Tag image processing pro cross-reference). ceeds as described in section 1.2.4 above, with the exception The inferred tag-space to image-space perspective trans that steps 26 and 28 are omitted. form is used to project (at 36) each known data bit position in 50 Although the tag may contain an orientation feature to tag space into image space where the real-valued position is allow disambiguation of the four possible orientations of the used to bilinearly interpolate (at 36) the four relevant adjacent tag relative to the sensor, it is also possible to embed orienta pixels in the input image. The previously computed image tion data in the tag data. For example, the four codewords can threshold 21 is used to threshold the result to produce the final be arranged so that each tag orientation contains one code bit value 37. 55 word placed at that orientation, as shown in FIG. 6d. where Once all 360 data bits 37 have been obtained in this way, each symbol is labelled with the number of its codeword (1-4) each of the six 60-bit Reed-Solomon codewords is decoded and the position of the symbol within the codeword (A-O). (at 38) to yield 20 decoded bits 39, or 120 decoded bits in Tag decoding then consists of decoding one codeword at each total. Note that the codeword symbols are sampled in code orientation. Each codeword can either contain a single bit word order, so that codewords are implicitly de-interleaved 60 indicating whether it is the first codeword, or two bits indi during the sampling process. cating which codeword it is. The latter approach has the As mentioned above, the physical tag structure or encoding advantage that if, say, the data content of only one codeword system is not essential to the invention and other physical is required, then at most two codewords need to be decoded to arrangements of each tag may be used. It will be understood obtain the desired data. This may be the case if the region ID that the process for recognizing and decoding the tag image to 65 is not expected to change within a stroke and is thus only retrieve the data encoded depends on the physical structure of decoded at the start of a stroke. Within a stroke only the the tag and the system used for redundantly encoding the data. codeword containing the tag ID is then desired. Furthermore, US 7463,250 B2 15 16 since the rotation of the sensing device changes slowly and ment from one sampled pen position to the next within a predictably within a stroke, only one codeword typically stroke can be unambiguously determined. needs to be decoded per frame. With either tagging scheme, the tags function in coopera It is possible to dispense with perspective targets altogether tion with associated visual elements on the netpage as user and instead rely on the data representation being self-regis- 5 interactive elements in that a user can interact with the printed tering. In this case each bit value (or multi-bit value) is typi page using an appropriate sensing device in order for tag data cally represented by an explicit glyph, i.e. no bit value is to be read by the sensing device and for an appropriate represented by the absence of a glyph. This ensures that the response to be generated in the netpage system. data grid is well-populated, and thus allows the grid to be reliably identified and its perspective distortion detected and 10 1.3 Document and Page Descriptions Subsequently corrected during data sampling. To allow tag A preferred embodiment of a document and page descrip boundaries to be detected, each tag data must contain a tion class diagram is shown in FIGS. 25 and 26. marker pattern, and these must be redundantly encoded to In the netpage system a document is described at three allow reliable detection. The overhead of such marker pat levels. At the most abstract level the document 836 has a terns is similar to the overhead of explicit perspective targets. 15 hierarchical structure whose terminal elements 839 are asso One such scheme uses dots positioned a various points rela ciated with content objects 840 such as text objects, textstyle tive to grid vertices to represent different glyphs and hence objects, image objects, etc. Once the document is printed on different multi-bit values (see Anoto Technology Descrip a printer with a particular page size and according to a par tion, Anoto April 2000). ticular user's scale factor preference, the document is pagi nated and otherwise formatted. Formatted terminal elements 1.2.6 Tag Map 835 will in some cases be associated with content objects Decoding a tag results in a region ID, a tag ID, and a which are different from those associated with their corre tag-relative pen transform. Before the tag ID and the tag sponding terminal elements, particularly where the content relative pen location can be translated into an absolute loca objects are style-related. Each printed instance of a document tion within the tagged region, the location of the tag within the and page is also described separately, to allow input captured region must be known. This is given by a tag map, a function 25 through a particular page instance 830 to be recorded sepa which maps each tag ID in a tagged region to a corresponding rately from input captured through other instances of the location. The tag map class diagram is shown in FIG. 22, as same page description. part of the netpage printer class diagram. The presence of the most abstract document description on A tag map reflects the scheme used to tile the Surface region 30 the page server allows a user to request a copy of a document with tags, and this can vary according to Surface type. When without being forced to accept the Source document's specific multiple tagged regions share the same tiling scheme and the format. The user may be requesting a copy through a printer same tag numbering scheme, they can also share the same tag with a different page size, for example. Conversely, the pres map. ence of the formatted document description on the page The tag map for a region must be retrievable via the region 35 server allows the page server to efficiently interpret user ID. Thus, given a region ID, a tag ID and a pen transform, the actions on a particular printed page. tag map can be retrieved, the tag ID can be translated into an A formatted document 834 consists of a set of formatted absolute tag location within the region, and the tag-relative page descriptions 5, each of which consists of a set of format pen location can be added to the tag location to yield an ted terminal elements 835. Each formatted element has a absolute pen location within the region. 40 spatial extent or Zone 58 on the page. This defines the active 1.2.7 Tagging Schemes area of input elements such as hyperlinks and input fields. Two distinct surface coding schemes are of interest, both of A document instance 831 corresponds to a formatted docu which use the tag structure described earlier in this section. ment 834. It consists of a set of page instances 830, each of The preferred coding scheme uses “location-indicating tags which corresponds to a page description 5 of the formatted as already discussed. An alternative coding scheme uses 45 document. Each page instance 830 describes a single unique “object-indicating tags. printed netpage 1, and records the page ID50 of the netpage. A location-indicating tag contains a tag ID which, when A page instance is not part of a document instance if it rep translated through the tag map associated with the tagged resents a copy of a page requested in isolation. region, yields a unique tag location within the region. The A page instance consists of a set of terminal element tag-relative location of the pen is added to this tag location to 50 instances 832. An element instance only exists if it records yield the location of the pen within the region. This in turn is instance-specific information. Thus, a hyperlink instance used to determine the location of the pen relative to a user exists for a hyperlink element because it records a transaction interface element in the page description associated with the ID 55 which is specific to the page instance, and a field region. Not only is the user interface element itself identified, instance exists for a field element because it records input but a location relative to the user interface element is identi- 55 specific to the page instance. An element instance does not fied. Location-indicating tags therefore trivially support the exist, however, for static elements such as textflows. capture of an absolute pen path in the Zone of a particular user A terminal element can be a static element 843, a hyperlink interface element. element 844, a field element 845 or a page server command An object-indicating tag contains a tag ID which directly element 846, as shown in FIG. 27. A static element 843 can be identifies a user interface element in the page description 60 a style element 847 with an associated style object 854, a associated with the region. All the tags in the Zone of the user textflow element 848 with an associated styled text object interface element identify the user interface element, making 855, an image element 849 with an associated image element them all identical and therefore indistinguishable. Object 856, a graphic element 850 with an associated graphic object indicating tags do not, therefore, Support the capture of an 857, a video clip element 851 with an associated video clip absolute pen path. They do, however, Support the capture of a 65 object 858, an audio clip element 852 with an associated relative pen path. So long as the position sampling frequency audio clip object 859, or a script element 853 with an associ exceeds twice the encountered tag frequency, the displace ated script object 860, as shown in FIG. 28. US 7463,250 B2 17 18 A page instance may have a background field 833 which is ticular subscriber and so are pointcast to the subscriber's used to record any digitalink captured on the page which does printer via the appropriate page server. Text and image not apply to a specific input element. objects, on the other hand, are typically shared with other In the preferred form of the invention, a tag map 811 is subscribers, and so are multicast to all subscribers printers associated with each page instance to allow tags on the page 5 and the appropriate page servers. to be translated into locations on the page. The netpage publication server optimizes the segmentation 1.4 The Netpage Network of document content into pointcasts and multicasts. After In a preferred embodiment, a netpage network consists of receiving the pointcast of a document’s page layouts, the a distributed set of netpage page servers 10, netpage registra printer knows which multicasts, if any, to listen to. tion servers 11, netpage ID servers 12, netpage application 10 Once the printer has received the complete page layouts servers 13, netpage publication servers 14, and netpage print and objects that define the document to be printed, it can print ers 601 connected via a network 19 such as the Internet, as the document. shown in FIG. 3. The printer rasterizes and prints odd and even pages simul The netpage registration server 11 is a server which records taneously on both sides of the sheet. It contains duplexed print 15 engine controllers 760 and print engines utilizing MemjetTM relationships between users, pens, printers, applications and printheads 350 for this purpose. publications, and thereby authorizes various network activi The printing process consists of two decoupled stages: ties. It authenticates users and acts as a signing proxy on rasterization of page descriptions, and expansion and printing behalf of authenticated users in application transactions. It of page images. The raster image processor (RIP) consists of also provides handwriting recognition services if desired. As one or more standard DSPs 757 running in parallel. The described above, a netpage page server 10 maintains persis duplexed print engine controllers consist of custom proces tent information about page descriptions and page instances. sors which expand, dither and print page images in real time, The netpage network includes any number of page servers, synchronized with the operation of the printheads in the print each handling a Subset of page instances. Since a page server engines. also maintains user input values for each page instance, cli ents such as netpage printers send netpage input directly to the 25 Printers not enabled for invisible IR printing have the appropriate page server. The page server interprets any Such option to print tags using IR-absorptive black ink, although input relative to the description of the corresponding page. this restricts tags to otherwise empty areas of the page. A netpage ID server 12 allocates document IDs 51 on Although Such pages have more limited functionality than demand, and provides load-balancing of page servers via its invisible IR-printed pages, they are still classed as netpages. ID allocation scheme. 30 A normal netpage printer prints netpages on sheets of A netpage printer uses the Internet Distributed Name Sys paper. More specialized netpage printers may print onto more tem (DNS), or similar, to resolve a netpage pageID50 into the specialized surfaces, such as globes or sheets of plastics. Each network address of the netpage page server handling the printer Supports at least one surface type, and Supports at least corresponding page instance. one tag tiling scheme, and hence tag map, for each Surface A netpage application server 13 is a server which hosts 35 type. The tag map 811 which describes the tag tiling scheme interactive netpage applications. A netpage publication server actually used to print a document becomes associated with 14 is an application server which publishes netpage docu that document so that the documents tags can be correctly ments to netpage printers. They are described in detail in interpreted. Section 2. FIG. 2 shows the netpage printer class diagram, reflecting Netpage servers can be hosted on a variety of network 40 printer-related information maintained by a registration server platforms from manufacturers such as IBM, Hewlett server 11 on the netpage network. Packard, and Sun. Multiple netpage servers can run concur A preferred embodiment of the netpage printer is described rently on a single host, and a single server can be distributed in greater detail in Section 6 below, with reference to FIGS. 11 over a number of hosts. Some or all of the functionality to 16. provided by netpage servers, and in particular the function 45 1.5.1 MemjetTM Printheads ality provided by the ID server and the page server, can also be The netpage system can operate using printers made with a provided directly in a netpage appliance Such as a netpage wide range of digital printing technologies, including thermal printer, in a computer workstation, or on a local network. inkjet, piezoelectric inkjet, laser electrophotographic, and others. However, for wide consumer acceptance, it is desir 1.5 The Netpage Printer 50 able that a netpage printer have the following characteristics: The netpage printer 601 is an appliance which is registered photographic quality color printing with the netpage system and prints netpage documents on high quality text printing demand and via Subscription. Each printer has a unique high reliability printer ID 62, and is connected to the netpage network via a low printer cost network Such as the Internet, ideally via abroadband connec 55 tion. low ink cost Apart from identity and security settings in non-volatile low paper cost memory, the netpage printer need not contain any persistent simple operation storage. As far as a user is concerned, “the network is the nearly silent printing computer. Netpages function interactively across space and 60 high printing speed time with the help of the distributed netpage page servers 10, simultaneous double sided printing independently of particular netpage printers. compact form factor The netpage printer receives Subscribed netpage docu low power consumption ments from netpage publication servers 14. Each document is No currently commercially available printing technology distributed in two parts: the page layouts, and the actual text 65 has all of these characteristics. and image objects which populate the pages. Because of To enable production of printers with these characteristics, personalization, page layouts are typically specific to a par the present applicant has invented a new print technology, US 7463,250 B2 19 20 referred to as MemjetTM technology. MemjetTM is a drop-on printing elements 300, printing the six colors or types of ink demand inkjet technology that incorporates pagewidth print supplied by the ink inlets 312. heads fabricated using microelectromechanical systems To protect the fragile Surface of the printhead during opera (MEMS) technology. FIG. 17 shows a single printing element 300 of a MemjetTM printhead. The netpage wallprinter incor tion, a nozzle guard wafer 330 is attached to the printhead porates 168960 printing elements 300 to form a 1600 dpi substrate 301. For each nozzle 302 there is a corresponding pagewidth duplex printer. This printer simultaneously prints nozzle guard hole 331 through which the ink droplets are cyan, magenta, yellow, black, and infrared inks as well as fired. To prevent the nozzle guard holes 331 from becoming paper conditioner and ink fixative. blocked by paper fibers or other debris, filtered air is pumped The printing element 300 is approximately 110 microns 10 through the air inlets 332 and out of the nozzle guard holes long by 32 microns wide. Arrays of these printing elements during printing. To prevent ink 321 from drying, the nozzle are formed on a silicon substrate 301 that incorporates CMOS guard is sealed while the printer is idle. logic, data transfer, timing, and drive circuits (not shown). 1.6 The Netpage Pen Major elements of the printing element 300 are the nozzle 302, the nozzle rim 303, the nozzle chamber 304, the fluidic 15 The active sensing device of the netpage system is typically seal 305, the ink channel rim 306, the leverarm 307, the active a pen 101, which, using its embedded controller 134, is able actuator beam pair 308, the passive actuator beam pair 309. to capture and decode IR position tags from a page via an the active actuator anchor 310, the passive actuator anchor image sensor. The image sensor is a solid-state device pro 311, and the ink inlet 312. vided with an appropriate filter to permit sensing at only The active actuator beam pair 308 is mechanically joined to near-infrared wavelengths. As described in more detail below, the passive actuator beam pair 309 at the join 319. Both beams the system is able to sense when the nib is in contact with the pairs are anchored at their respective anchor points 310 and Surface, and the pen is able to sense tags at a sufficient rate to 311. The combination of elements 308, 309, 310, 311, and capture human handwriting (i.e. at 200 dpi or greater and 100 319 form a cantilevered electrothermal bend actuator 320. HZ or faster). Information captured by the pen is encrypted FIG. 18 shows a small part of an array of printing elements 25 and wirelessly transmitted to the printer (or base station), the 300, including a cross section 315 of a printing element 300. printer or base station interpreting the data with respect to the The cross section 315 is shown without ink, to clearly show (known) page, or, in the preferred embodiment, transmitting the ink inlet 312 that passes through the silicon wafer 301. the information to a netpage server for interpretation. FIGS. 190a), 19(b) and 190c) show the operating cycle of a The preferred embodiment of the netpage pen operates MemjetTM printing element 300. 30 both as a marking ink pen and as a non-marking stylus. The FIG. 19(a) shows the quiescent position of the ink menis marking aspect, however, is not necessary for using the cus 316 prior to printing an ink droplet. Ink is retained in the netpage system as a browsing system, such as when it is used nozzle chamber by surface tension at the ink meniscus 316 as an Internet interface. Each netpage pen is registered with the netpage system and has a unique penID 61. FIG.23 shows and at the fluidic seal 305 formed between the nozzle chamber 35 304 and the ink channel rim 306. the netpage pen class diagram, reflecting pen-related infor While printing, the printhead CMOS circuitry distributes mation maintained by a registration server 11 on the netpage data from the print engine controller to the correct printing network. element, latches the data, and buffers the data to drive the When either nib is in contact with a netpage, the pen electrodes 318 of the active actuator beam pair 308. This 40 determines its position and orientation relative to the page. causes an electrical current to pass through the beam pair 308 The nib is attached to a force sensor, and the force on the nib for about one microsecond, resulting in Joule heating. The is interpreted relative to a threshold to indicate whether the temperature increase resulting from Joule heating causes the pen is “up' or “down”. This allows a interactive element on beam pair 308 to expand. As the passive actuator beam pair the page to be clicked by pressing with the pen nib, in order 309 is not heated, it does not expand, resulting in a stress 45 to request, say, information from a network. Furthermore, the difference between the two beam pairs. This stress difference force is captured as a continuous value to allow, say, the full is partially resolved by the cantilevered end of the electro dynamics of a signature to be verified. The nib may be mov thermal bend actuator 320 bending towards the substrate 301. able when subject to a specified force which is greater than The lever arm 307 transmits this movement to the nozzle that normally applied when writing. To "click” the user chamber 304. The nozzle chamber 304 moves about two 50 applies a force sufficient to move the nib. This may provide microns to the position shown in FIG. 19(b). This increases more desirable feedback to the user compared to that pro the ink pressure, forcing ink 321 out of the nozzle 302, and vided by a non-moving nib. causing the ink meniscus 316 to bulge. The nozzle rim 303 The pen determines the position and orientation of its nib prevents the ink meniscus 316 from spreading across the on the netpage by imaging, in the infrared spectrum, an area surface of the nozzle chamber 304. 55 193 of the page in the vicinity of the nib. It decodes the nearest As the temperature of the beam pairs 308 and 309 equal tag and computes the position of the nib relative to the tag izes, the actuator 320 returns to its original position. This aids from the observed perspective distortion on the imaged tag in the break-off of the ink droplet 317 from the ink321 in the and the known geometry of the pen optics. Although the nozzle chamber, as shown in FIG. 190c). The nozzle chamber position resolution of the tag may be low, because the tag is refilled by the action of the surface tension at the meniscus 60 density on the page is inversely proportional to the tag size, 316. the adjusted position resolution is quite high, exceeding the FIG. 20 shows a segment of a printhead 350. In a netpage minimum resolution required for accurate handwriting rec printer, the length of the printhead is the full width of the ognition. paper (typically 210 mm) in the direction 351. The segment Pen actions relative to a netpage are captured as a series of shown is 0.4 mm long (about 0.2% of a complete printhead). 65 strokes. A stroke consists of a sequence of time-stamped pen When printing, the paper is moved past the fixed printhead in positions on the page, initiated by a pen-down event and the direction 352. The printhead has 6 rows of interdigitated completed by the Subsequent pen-up event. A stroke is also US 7463,250 B2 21 22 tagged with the page ID50 of the netpage whenever the page refers to objects such as images, fonts and pieces of text, ID changes, which, under normal circumstances, is at the typically stored elsewhere on the netpage network. commencement of the stroke. When the page server receives the stroke from the pen, it Each netpage pen has a current selection 826 associated retrieves the page description to which the stroke applies, and with it, allowing the user to perform copy and paste opera determines which element of the page description the stroke tions etc. The selection is time-stamped to allow the system to intersects. It is then able to interpret the stroke in the context discard it after a defined time period. The current selection of the type of the relevant element. describes a region of a page instance. It consists of the most A “click” is typically a stroke where the distance and time recent digital ink stroke captured through the pen relative to between the pen down position and the Subsequent pen up the background area of the page. It is interpreted in an appli 10 position are both less than Some Small maximum. An object cation-specific manner once it is Submitted to an application which is activated by a click typically requires a click to be via a selection hyperlink activation. activated, and accordingly, a longer stroke is ignored. The Each pen has a current nib 824. This is the nib last notified failure of a penaction, such as a 'sloppy' click, to register is by the pen to the system. In the case of the default netpage pen indicated by the lack of response from the pens 'ok' LED. described above, either the marking ink nib or the non-mark 15 However, where a netpage includes a button a "click” can be ing stylus nib is current. Each pen also has a current nib style registered when both the pen down and pen up positions are 825. This is the nib style last associated with the pen by an both within the area of the button. application, e.g. in response to the user selecting a color from There are two kinds of input elements in a netpage page a palette. The default nib style is the nib style associated with description: hyperlinks and form fields. Input through a form the current nib. Strokes captured through a pen are tagged field can also trigger the activation of an associated hyperlink. with the current nib style. When the strokes are subsequently reproduced, they are reproduced in the nib style with which 1.7.1 Hyperlinks they are tagged. A hyperlink is a means of sending a message to a remote application, and typically elicits a printed response in the Whenever the pen is within range of a printer with which it netpage System. can communicate, the pen slowly flashes its “online LED. 25 When the pen fails to decode a stroke relative to the page, it A hyperlink element 844 identifies the application 71 momentarily activates its "error LED. When the pen suc which handles activation of the hyperlink, a link ID 54 which ceeds in decoding a stroke relative to the page, it momentarily identifies the hyperlink to the application, an “alias required activates its 'ok' LED. flag which asks the system to include the user's application A sequence of captured strokes is referred to as digital ink. 30 alias ID 65 in the hyperlink activation, and a description Digital ink forms the basis for the digital exchange of draw which is used when the hyperlink is recorded as a favorite or ings and handwriting, for online recognition of handwriting, appears in the users history. The hyperlink element class and for online verification of signatures. diagram is shown in FIG. 29. The pen is wireless and transmits digital ink to the netpage When a hyperlink is activated, the page server sends a printer via a short-range radio link. The transmitted digital ink 35 request to an application somewhere on the network. The is encrypted for privacy and security and packetized for effi application is identified by an application ID 64, and the cient transmission, but is always flushed on a pen-up event to application ID is resolved in the normal way via the DNS. ensure timely handling in the printer. There are three types of hyperlinks: general hyperlinks 863, When the pen is out of range of a printer it buffers digital form hyperlinks 865, and selection hyperlinks 864, as shown ink in internal memory, which has a capacity of over ten 40 in FIG. 30. A general hyperlink can implement a request for a minutes of continuous handwriting. When the pen is once linked document, or may simply signal a preference to a again within range of a printer, it transfers any buffered digital server. A form hyperlink Submits the corresponding form to ink. The buffer may provide more or less buffer capacity. the application. A selection hyperlink Submits the current A pen can be registered with any number of printers, but selection to the application. If the current selection contains a because all State data resides in netpages both on paper and on 45 single-word piece of text, for example, the application may the network, it is largely immaterial which printer a pen is return a single-page document giving the words meaning communicating with at any particular time. within the context in which it appears, or a translation into a A preferred embodiment of the pen is described in greater different language. Each hyperlink type is characterized by detail in Section 6 below, with reference to FIGS. 8 to 10. what information is Submitted to the application. 50 The corresponding hyperlink instance 862 records a trans 1.7 Netpage Interaction action ID 55 which can be specific to the page instance on The netpage printer 601 receives data relating to a stroke which the hyperlink instance appears. The transaction ID can from the pen 101 when the pen is used to interact with a identify user-specific data to the application, for example a netpage 1. The coded data 3 of the tags 4 is read by the pen “shopping cart of pending purchases maintained by a pur when it is used to execute a movement, such as a stroke. The 55 chasing application on behalf of the user. data allows the identity of the particular page and associated The system includes the pens current selection 826 in a interactive element to be determined and an indication of the selection hyperlink activation. The system includes the con relative positioning of the pen relative to the page to be tent of the associated form instance 868 in a form hyperlink obtained. The indicating data is transmitted to the printer, activation, although if the hyperlink has its “submit delta' where it resolves, via the DNS, the page ID 50 of the stroke 60 attribute set, only input since the last form Submission is into the network address of the netpage page server 10 which included. The system includes an effective return path in all maintains the corresponding page instance 830. It then trans hyperlink activations. mits the stroke to the page server. If the page was recently A hyperlinked group 866 is a group element 838 which has identified in an earlier stroke, then the printer may already an associated hyperlink, as shown in FIG. 31. When input have the address of the relevant page server in its cache. Each 65 occurs through any field element in the group, the hyperlink netpage consists of a compact page layout maintained persis 844 associated with the group is activated. A hyperlinked tently by a netpage page server (see below). The page layout group can be used to associate hyperlink behavior with a field US 7463,250 B2 23 24 Such as a checkbox. It can also be used, in conjunction with signature field's Zone is automatically verified with respect to the “submit delta' attribute of a form hyperlink, to provide the identity of the owner of the pen, and a digital signature of continuous input to an application. It can therefore be used to the content of the form of which the field is part is generated support a “blackboard' interaction model, i.e. where input is and assigned to the field's value. The digital signature is captured and therefore shared as soon as it occurs. generated using the pen user's private signature key specific to the application which owns the form. Online signature 1.7.2 Forms verification is well-understood (see, for example, Plamon A form defines a collection of related input fields used to don, R. and G. Lorette, “Automatic Signature Verification and capture a related set of inputs through one or more printed Writer Identification. The State of the Art', Pattern Recog netpages. A form allows a user to Submit one or more param 10 nition, Vol. 22, No. 2, 1989, the contents of which are herein eters to an application Software program running on a server. incorporated by cross-reference). A form 867 is a group element 838 in the document hier A field element is hidden if its "hidden' attribute is set. A archy. It ultimately contains a set of terminal field elements hidden field element does not have an input Zone on a page 839. A form instance 868 represents a printed instance of a and does not accept input. It can have an associated field value form. It includes a set offield instances 870 which correspond 15 which is included in the form data when the form containing to the field elements 845 of the form. Each field instance has the field is submitted. an associated value 871, whose type depends on the type of “Editing commands, such as strike-throughs indicating the corresponding field element. Each field value records deletion, can also be recognized in form fields. input through a particular printed form instance, i.e. through Because the handwriting recognition algorithm works one or more printed netpages. The form class diagram is “online” (i.e. with access to the dynamics of the pen move shown in FIG. 32. ment), rather than "offline' (i.e. with access only to a bitmap Each form instance has a status 872 which indicates of pen markings), it can recognize run-on discretely-written whether the form is active, frozen, submitted, void or expired. characters with relatively high accuracy, without a writer A form is active when first printed. A form becomes frozen dependent training phase. A writer-dependent model of hand once it is signed. A form becomes Submitted once one of its 25 writing is automatically generated over time, however, and Submission hyperlinks has been activated, unless the hyper can be generated up-front if necessary, link has its “submit delta' attribute set. A form becomes Void Digital ink, as already stated, consists of a sequence of when the user invokes a void form, reset form or duplicate strokes. Any stroke which starts in a particular element's Zone form page command. A form expires when the time the form is appended to that element’s digital ink stream, ready for has been active exceeds the forms specified lifetime. While 30 interpretation. Any stroke not appended to an objects digital the form is active, form input is allowed. Input through a form ink stream is appended to the background field's digital ink which is not active is instead captured in the background field Stream. 833 of the relevant page instance. When the form is active or Digital ink captured in the background field is interpreted frozen, form Submission is allowed. Any attempt to Submit a as a selection gesture. Circumscription of one or more objects form when the form is not active or frozen is rejected, and 35 is generally interpreted as a selection of the circumscribed instead elicits a form status report. objects, although the actual interpretation is application-spe Each form instance is preferably associated (at 59) with cific. any form instances derived from it, thus providing a version Table 2 Summarizes these various pen interactions with a history. This allows all but the latest version of a form in a netpage. particular time period to be excluded from a search. 40 All input is captured as digital ink. Digital ink 873 consists of a set of time-stamped stroke groups 874, each of which TABLE 2 consists of a set of styled strokes 875. Each stroke consists of Summary of pen interactions with a netpage a set of time-stamped pen positions 876, each of which also Object Type Pen input Action includes pen orientation and nib force. The digital ink class 45 diagram is shown in FIG. 33. Hyperlink General Click Submit action to application A field element 845 can be a checkbox field 877, a text field Form Click Submit form to application Selection Click Submit selection to 878, a drawing field 879, or a signature field 880. The field application element class diagram is shown in FIG. 34. Any digital ink Form field Checkbox Any mark Assign true to field captured in a field's Zone 58 is assigned to the field. 50 Text Handwriting Convert digital ink to text: A checkbox field has an associated Boolean value 881, as assign text to field Drawing Digital ink Assign digital ink to field shown in FIG. 35. Any mark (a tick, a cross, a stroke, a fill Signature Signature Verify digital ink signature; ZigZag, etc.) captured in a checkbox field's Zone causes a true generate digital signature value to be assigned to the field's value. of form; assign digital A text field has an associated text value 882, as shown in 55 signature to field FIG. 36. Any digital ink captured in a text field's Zone is None Circumscription Assign digital ink to current automatically converted to text via online handwriting recog Selection nition, and the text is assigned to the fields value. Online handwriting recognition is well-understood (see, for The system maintains a current selection for each pen. The example, Tappert, C., C. Y. Suen and T. Wakahara, “The State 60 selection consists simply of the most recent stroke captured in of the Art in On-Line Handwriting Recognition', IEEE the background field. The selection is cleared after an inac Transactions on Pattern Analysis and Machine Intelligence, tivity timeout to ensure predictable behavior. Vol. 12, No. 8, August 1990, the contents of which are herein The raw digital ink captured in every field is retained on the incorporated by cross-reference). Specializations of text netpage page server and is optionally transmitted with the fields include date and number fields. 65 form data when the form is submitted to the application. This A signature field has an associated digital signature value allows the application to interrogate the raw digital ink should 883, as shown in FIG. 37. Any digital ink captured in a it suspect the original conversion, such as the conversion of US 7463,250 B2 25 26 handwritten text. This can, for example, involve human inter instance and then produces an active printed copy of the form vention at the application level for forms which fail certain instance with field values discarded. The copy contains the application-specific consistency checks. As an extension to same hyperlink transaction IDs as the original. A get form this, the entire background area of a form can be designated as status command produces a printed report on the status of the a drawing field. The application can then decide, on the basis corresponding form instance, including who published it, of the presence of digital ink outside the explicit fields of the when it was printed, for whom it was printed, and the form form, to route the form to a human operator, on the assump status of the form instance. tion that the user may have indicated amendments to the Since a form hyperlink instance contains a transaction ID, filled-in fields outside of those fields. the application has to be involved in producing a new form FIG.38 shows a flowchart of the process of handling pen 10 instance. A button requesting a new form instance is therefore input relative to a netpage. The process consists of receiving typically implemented as a hyperlink. (at 884) a stroke from the pen; identifying (at 885) the page A duplicate page command produces a printed copy of the instance 830 to which the page ID 50 in the stroke refers: corresponding page instance with the background field value retrieving (at 886) the page description 5; identifying (at 887) preserved. If the page contains a form or is part of a form, then a formatted element 839 whose Zone 58 the stroke intersects; 15 the duplicate page commandisinterpreted as a duplicate form determining (at 888) whether the formatted element corre command. A reset page command produces a printed copy of sponds to a field element, and if so appending (at 892) the the corresponding page instance with the background field received stroke to the digital ink of the field value 871, inter value discarded. If the page contains a form or is part of a preting (at 893) the accumulated digital ink of the field, and form, then the reset page command is interpreted as a reset determining (at 894) whether the field is part of a hyperlinked form command. A get page status command produces a group 866 and if so activating (at 895) the associated hyper printed report on the status of the corresponding page link; alternatively determining (at 889) whether the formatted instance, including who published it, when it was printed, for element corresponds to a hyperlink element and if so activat whom it was printed, and the status of any forms it contains or ing (at 895) the corresponding hyperlink; alternatively, in the is part of absence of an input field or hyperlink, appending (at 890) the 25 The netpage logo which appears on every netpage is usu received stroke to the digital ink of the background field 833; ally associated with a duplicate page element in the preferred and copying (at 891) the received stroke to the current selec implementation of the invention. tion 826 of the current pen, as maintained by the registration When a page instance is duplicated with field values pre SeVe. served, field values are printed in their native form, i.e. a FIG. 38a shows a detailed flowchart of step 893 in the 30 checkmarkappears as a standard checkmarkgraphic, and text process shown in FIG. 38, where the accumulated digital ink appears as typeset text. Only drawings and signatures appear of a field is interpreted according to the type of the field. The in their original form, with a signature preferably accompa process consists of determining (at 896) whether the field is a nied by, or alternatively replaced by, a standard graphic indi checkbox and (at 897) whether the digital ink represents a cating Successful signature verification. checkmark, and if so assigning (at 898) a true value to the field 35 A duplicate document command produces a printed copy value; alternatively determining (at 899) whether the field is a of the corresponding document instance with background text field and if so converting (at 900) the digital ink to field values preserved. If the document contains any forms, computer text, with the help of the appropriate registration then the duplicate document command duplicates the forms server, and assigning (at 901) the converted computer text to in the same way a duplicate form command does. A reset the field value; alternatively determining (at 902) whether the 40 document command produces a printed copy of the corre field is a signature field and if so verifying (at 903) the digital sponding document instance with background field values ink as the signature of the pen's owner, with the help of the discarded. If the document contains any forms, then the reset appropriate registration server, creating (at 904) a digital sig document command resets the forms in the same way a reset nature of the contents of the corresponding form, also with the form command does. A get document status command pro help of the registration server and using the pen owners 45 duces a printed report on the status of the corresponding private signature key relating to the corresponding applica document instance, including who published it, when it was tion, and assigning (at 905) the digital signature to the field printed, for whom it was printed, and the status of any forms value. it contains. 1.7.3 Page Server Commands 50 If the page server command’s “on selected attribute is set, A page server command is a command which is handled then the command operates on the page identified by the pens locally by the page server. It operates directly on form, page current selection rather than on the page containing the com and document instances. mand. This allows a menu of page server commands to be A page server command 907 can be avoid form command printed. If the target page doesn’t contain a page server com 908, a duplicate form command 909, a reset form command 55 mand element for the designated page server command, then 910, a get form status command 911, a duplicate page com the command is ignored. mand 912, a reset page command 913, a get page status An application can provide application-specific handling command 914, a duplicate document command 915, a reset by embedding the relevant page server command element in document command 916, or a get document status command a hyperlinked group. The page server activates the hyperlink 917, as shown in FIG. 39. 60 associated with the hyperlinked group rather than executing A Void form command Voids the corresponding form the page server command. instance. A duplicate form command Voids the corresponding A page server command element is hidden if its “hidden' form instance and then produces an active printed copy of the attribute is set. A hidden command element does not have an current form instance with field values preserved. The copy input Zone on a page and so cannot be activated directly by a contains the same hyperlink transaction IDs as the original, 65 user. It can, however, beactivated via a page server command and so is indistinguishable from the original to an application. embedded in a different page, if that page server command A reset form command Voids the corresponding form has its "on selected' attribute set. US 7463,250 B2 27 28 1.8 Standard Features of Netpages varied: a “raw' news feed from a news service, a cartoon strip, In the preferred form, each netpage is printed with the a freelance writer's column, a friend’s bulletin board, or the netpage logo at the bottom to indicate that it is a netpage and reader's own e-mail. therefore has interactive properties. The logo also acts as a The netpage publication server Supports the publication of copy button. In most cases "clicking the logo produces a edited news publications as well as the aggregation of mul copy of the page. In the case of a form, the button produces a tiple news streams. By handling the aggregation and hence copy of the entire form. And in the case of a secure document, the formatting of news streams selected directly by the reader, Such as a ticket or coupon, the button elicits an explanatory the server is able to place advertising on pages over which it note or advertising page. otherwise has no editorial control. The default single-page copy function is handled directly 10 The subscriber builds a daily newspaper by selecting one or by the relevant netpage page server. Special copy functions more contributing news publications, and creating a person are handled by linking the logo button to an application. alized version of each. The resulting daily editions are printed and bound together into a single newspaper. The various 1.9 User Help System members of a household typically express their different In a preferred embodiment, the netpage printer has a single 15 interests and tastes by selecting different daily publications button labelled “Help'. When pressed it elicits a single page and then customizing them. of information, including: For each publication, the reader optionally selects specific status of printer connection sections. Some sections appear daily, while others appear status of printer consumables weekly. The daily sections available from The New York top-level help menu Times online, for example, include “Page One Plus', document function menu “National”, “International”, “Opinion”, “Business”, “Arts/ top-level netpage network directory Living”, “Technology', and “Sports”. The set of available The help menu provides a hierarchical manual on how to sections is specific to a publication, as is the default Subset. use the netpage system. The reader can extend the daily newspaper by creating The document function menu includes the following func 25 custom sections, each one drawing on any number of news tions: streams. Custom sections might be created for e-mail and print a copy of a document friends announcements (“Personal’’), or for monitoring news print a clean copy of a form feeds for specific topics (“Alerts' or “Clippings'). print the status of a document 30 For each section, the reader optionally specifies its size, A document function is initiated by simply pressing the either qualitatively (e.g. short, medium, or long), or numeri button and then touching any page of the document. The cally (i.e. as a limit on its number of pages), and the desired status of a document indicates who published it and when, to proportion of advertising, either qualitatively (e.g. high, nor whom it was delivered, and to whom and when it was subse mal, low, none), or numerically (i.e. as a percentage). quently Submitted as a form. 35 The reader also optionally expresses a preference for a The netpage network directory allows the user to navigate large number of shorter articles or a small number of longer the hierarchy of publications and services on the network. As articles. Each article is ideally written (or edited) in both short an alternative, the user can call the netpage network “900' and long forms to Support this preference. number "yellow pages” and speak to a human operator. The An article may also be written (or edited) in different operator can locate the desired document and route it to the 40 versions to match the expected Sophistication of the reader, user's printer. Depending on the document type, the publisher for example to provide children's and adults versions. The or the user pays the small “yellow pages' service fee. appropriate version is selected according to the readers age. The help page is obviously unavailable if the printer is The reader can specify a “reading age which takes prece unable to print. In this case the "error light is lit and the user dence over their biological age. can request remote diagnosis over the network. 45 The articles which make up each section are selected and prioritized by the editors, and each is assigned a useful life 2. Personalized Publication Model time. By default they are delivered to all relevant subscribers, In the following description, news is used as a canonical in priority order, Subject to space constraints in the Subscrib publication example to illustrate personalization mechanisms ers' editions. in the netpage system. Although news is often used in the 50 In sections where it is appropriate, the reader may option limited sense of newspaper and newsmagazine news, the ally enable collaborative filtering. This is then applied to intended scope in the present context is wider. articles which have a sufficiently long lifetime. Each article In the netpage system, the editorial content and the adver which qualifies for collaborative filtering is printed with rat tising content of a news publication are personalized using ing buttons at the end of the article. The buttons can provide different mechanisms. The editorial content is personalized 55 according to the reader's explicitly stated and implicitly cap an easy choice (e.g. "liked and "disliked'), making it more tured interest profile. The advertising content is personalized likely that readers will bother to rate the article. according to the reader's locality and demographic. Articles with high priorities and short lifetimes are there fore effectively considered essential reading by the editors 2.1 Editorial Personalization 60 and are delivered to most relevant subscribers. A Subscriber can draw on two kinds of news sources: those The reader optionally specifies a serendipity factor, either that deliver news publications, and those that deliver news qualitatively (e.g. do or don't Surprise me), or numerically. A streams. While news publications are aggregated and edited high serendipity factor lowers the threshold used for match by the publisher, news streams are aggregated either by a ing during collaborative filtering. A high factor makes it more news publisher or by a specialized news aggregator. News 65 likely that the corresponding section will be filled to the publications typically correspond to traditional newspapers reader's specified capacity. A different Serendipity factor can and newsmagazines, while news streams can be many and be specified for different days of the week. US 7463,250 B2 29 30 The reader also optionally specifies topics of particular aggregator can provide arbitrarily broad coverage of both interest within a section, and this modifies the priorities geography and demographics. The Subsequent disaggrega assigned by the editors. tion is efficient because it is automatic. This makes it more The speed of the reader's Internet connection affects the cost-effective for publishers to deal with advertising aggre quantity and quality at which images can be delivered. The gators than to directly capture advertising. Even though the reader optionally specifies a preference for fewer images or advertising aggregator is taking a proportion of advertising Smaller images or both. If the number or size of images is not revenue, publishers may find the change profit-neutral reduced, then images may be delivered at lower quality (i.e. at because of the greater efficiency of aggregation. The adver lower resolution or with greater compression). Alternatively tising aggregator acts as an intermediary between advertisers all three of the quantity, size and quality of images delivered 10 and publishers, and may place the same advertisement in may be adjusted. multiple publications. At a global level, the reader specifies how quantities, dates, It is worth noting that ad placement in a netpage publica times and monetary values are localized. This involves speci tion can be more complex than ad placement in the publica fying whether units are imperial or metric, a local time Zone tion’s traditional counterpart, because the publications and time format, and a local currency, and whether the local 15 advertising space is more complex. While ignoring the full ization consist of in situ translation or annotation. These complexities of negotiations between advertisers, advertising preferences are derived from the reader's locality by default. aggregators and publishers, the preferred form of the netpage To reduce reading difficulties caused by poor eyesight, the system provides some automated Support for these negotia reader optionally specifies a global preference for a larger tions, including Support for automated auctions of advertising presentation. Both text and images are scaled accordingly, space. Automation is particularly desirable for the placement and less information is accommodated on each page. of advertisements which generate Small amounts of income, The language in which a news publication is published, and Such as Small or highly localized advertisements. its corresponding text encoding, is a property of the publica Once placement has been negotiated, the aggregator cap tion and not a preference expressed by the user. However, the tures and edits the advertisement and records it on a netpage netpage system can be configured to provide automatic trans 25 ad server. Correspondingly, the publisher records the ad lation services in various guises. placement on the relevant netpage publication server. When the netpage publication server lays out each user's personal 2.2 Advertising Localization and Targeting ized publication, it picks the relevant advertisements from the The personalization of the editorial content directly affects netpage ad server. the advertising content, because advertising is typically 30 placed to exploit the editorial context. Travel ads, for 2.3 User Profiles example, are more likely to appear in a travel section than 2.3.1 Information Filtering elsewhere. The value of the editorial content to an advertiser The personalization of news and other publications relies (and therefore to the publisher) lies in its ability to attract on an assortment of user-specific profile information, includ large numbers of readers with the right demographics. 35 ing: Effective advertising is placed on the basis of locality and publication customizations demographics. Locality determines proximity to particular collaborative filtering vectors services, retailers etc., and particular interests and concerns contact details associated with the local community and environment. presentation preferences Demographics determine general interests and preoccupa 40 The customization of a publication is typically publication tions as well as likely spending patterns. specific, and so the customization information is maintained A news publisher's most profitable product is advertising by the relevant netpage publication server. “space', a multi-dimensional entity determined by the publi A collaborative filtering vector consists of the user's rat cation’s geographic coverage, the size of its readership, its ings of a number of news items. It is used to correlate different readership demographics, and the page area available for 45 users interests for the purposes of making recommendations. advertising. Although there are benefits to maintaining a single collabo In the netpage system, the netpage publication server com rative filtering vector independently of any particular publi putes the approximate multi-dimensional size of a publica cation, there are two reasons why it is more practical to tion’s saleable advertising space on a per-section basis, taking maintain a separate vector for each publication: there is likely into account the publication's geographic coverage, the sec 50 to be more overlap between the vectors of subscribers to the tion’s readership, the size of each reader's section edition, same publication than between those of subscribers to differ each reader's advertising proportion, and each readers ent publications; and a publication is likely to want to present demographic. its users’ collaborative filtering vectors as part of the value of In comparison with other media, the netpage system allows its brand, not to be found elsewhere. Collaborative filtering the advertising space to be defined in greater detail, and 55 vectors are therefore also maintained by the relevant netpage allows smaller pieces of it to be sold separately. It therefore publication server. allows it to be sold at closer to its true value. Contact details, including name, Street address, ZIP Code, For example, the same advertising “slot' can be sold in state, country, telephone numbers, are global by nature, and varying proportions to several advertisers, with individual are maintained by a netpage registration server. readers' pages randomly receiving the advertisement of one 60 Presentation preferences, including those for quantities, advertiser or another, overall preserving the proportion of dates and times, are likewise global and maintained in the space sold to each advertiser. same way. The netpage system allows advertising to be linked directly The localization of advertising relies on the locality indi to detailed product information and online purchasing. It cated in the user's contact details, while the targeting of therefore raises the intrinsic value of the advertising space. 65 advertising relies on personal information Such as date of Because personalization and localization are handled auto birth, gender, marital status, income, profession, education, matically by netpage publication servers, an advertising or qualitative derivatives Such as age range and income range. US 7463,250 B2 31 32 For those users who choose to reveal personal information by-section basis. Since most advertisements are in the form of for advertising purposes, the information is maintained by the pre-formatted rectangles, they are placed on the page before relevant netpage registration server. In the absence of Such the editorial content. information, advertising can be targeted on the basis of the The advertising ratio for a section can be achieved with demographic associated with the user's ZIP or ZIP+4 Code. wildly varying advertising ratios on individual pages within Each user, pen, printer, application provider and applica the section, and the ad layout algorithm exploits this. The tion is assigned its own unique identifier, and the netpage algorithm is configured to attempt to co-locate closely tied registration server maintains the relationships between them, editorial and advertising content, such as placing ads for as shown in FIGS. 21, 22, 23 and 24. For registration pur roofing material specifically within the publication because poses, a publisher is a special kind of application provider, 10 of a special feature on do-it-yourself roofing repairs. and a publication is a special kind of application. The editorial content selected for the user, including text Each user 800 may be authorized to use any number of and associated images and graphics, is then laid out according printers 802, and each printer may allow any number of users to various aesthetic rules. to use it. Each user has a single default printer (at 66), to which The entire process, including the selection of ads and the periodical publications are delivered by default, whilst pages 15 selection of editorial content, must be iterated once the layout printed on demand are delivered to the printer through which has converged, to attempt to more closely achieve the user's the user is interacting. The server keeps track of which pub stated section size preference. The section size preference lishers a user has authorized to print to the user's default can, however, be matched on average over time, allowing printer. A publisher does not record the ID of any particular significant day-to-day variations. printer, but instead resolves the ID when it is required. 2.5 Document Format When a user subscribes 808 to a publication 807, the pub Once the document is laid out, it is encoded for efficient lisher 806 (i.e. application provider 803) is authorized to print distribution and persistent storage on the netpage network. to a specified printer or the user's default printer. This autho The primary efficiency mechanism is the separation of rization can be revoked at any time by the user. Each user may information specific to a single user's edition and information have several pens 801, but a pen is specific to a single user. If 25 shared between multiple users' editions. The specific infor a user is authorized to use a particular printer, then that printer mation consists of the page layout. The shared information recognizes any of the user's pens. consists of the objects to which the page layout refers, includ The penID is used to locate the corresponding user profile ing images, graphics, and pieces of text. maintained by a particular netpage registration server, via the A text object contains fully-formatted text, preferably rep 30 resented in the Extensible Markup Language (XML) using DNS in the usual way. the Extensible Stylesheet Language (XSL). XSL provides A Web terminal 809 can be authorized to print on a par precise control over text formatting independently of the ticular netpage printer, allowing Web pages and netpage region into which the text is being set, which in this case is documents encountered during Web browsing to be conve being provided by the layout. The text object contains embed niently printed on the nearest netpage printer. ded language codes to enable automatic translation, and The netpage system can collect, on behalf of a printer 35 embedded hyphenation hints to aid with paragraph format provider, fees and commissions on income earned through ting. publications printed on the provider's printers. Such income An image object encodes an image, preferably in the JPEG can include advertising fees, click-through fees, e-commerce 2000 wavelet-based compressed image format. A graphic commissions, and transaction fees. If the printer is owned by object encodes a 2D graphic, preferably in Scalable Vector the user, then the user is the printer provider. 40 Graphics (SVG) format. Other formats may be used for text, Each user also has a netpage account 820 which is used to images and graphics. accumulate micro-debits and credits (such as those described The layout itself consists of a series of placed image and in the preceding paragraph); contact details 815, including graphic objects, linked textflow objects through which text name, address and telephone numbers; global preferences objects flow, hyperlinks and input fields as described above, 816, including privacy, delivery and localization settings; any 45 and watermark regions. These layout objects are summarized number of biometric records 817, containing the user's in Table 3. The layout uses a compact format suitable for encoded signature 818, fingerprint 819 etc.; a handwriting efficient distribution and storage. model 819 automatically maintained by the system; and SET payment card accounts 821 with which e-commerce pay TABLE 3 ments can be made. 50 netpage layout objects 2.3.2 Favorites List A netpage user can maintain a list 922 of “favorites' Layout Format of links to useful documents etc. on the netpage network. The object Attribute linked object list is maintained by the system on the user's behalf. It is 55 Image Position Image object ID JPEG 2000 organized as a hierarchy of folders 924, a preferred embodi Graphic Position ment of which is shown in the class diagram in FIG. 41. Graphic object ID SVG Textflow Textflow ID 2.3.3. History List Zone The system preferably maintains a history list929 on each Optional text object ID XML, XSL users behalf, containing links to documents etc. accessed by 60 Hyperlink Type the user through the netpage system. It is organized as a Zone Application ID, etc. date-ordered list, a preferred embodiment of which is shown Field Type in the class diagram in FIG. 42. Meaning Zone 2.4 Intelligent Page Layout 65 Watermark Zone The netpage publication server automatically lays out the pages of each user's personalized publication on a section US 7463,250 B2 33 34 2.6 Document Distribution contents. It can automatically include all Web pages directly As described above, for purposes of efficient distribution linked to the requested page. The user can tune this behavior and persistent storage on the netpage network, a user-specific via a preference. page layout is separated from the shared objects to which it The netpage formatting server makes standard netpage refers. behavior, including interactivity and persistence, available on When a subscribed publication is ready to be distributed, any Internet document, no matter what its origin and format. the netpage publication server allocates, with the help of the It hides knowledge of different document formats from both netpage ID server 12, a unique ID for each page, page the netpage printer and the netpage page server, and hides instance, document, and document instance. knowledge of the netpage system from Web servers. The server computes a set of optimized subsets of the 10 3. Security shared content and creates a multicast channel for each Sub set, and then tags each user-specific layout with the names of 3.1 Cryptography the multicast channels which will carry the shared content Cryptography is used to protect sensitive information, both used by that layout. The server then pointcasts each user's in storage and in transit, and to authenticate parties to a layouts to that user's printer via the appropriate page server, 15 transaction. There are two classes of cryptography in wide and when the pointcasting is complete, multicasts the shared spread use: Secret-key cryptography and public-key cryptog content on the specified channels. After receiving its point raphy. The netpage network uses both classes of cryptogra cast, each page server and printer Subscribes to the multicast phy. channels specified in the page layouts. During the multicasts, Secret-key cryptography, also referred to as symmetric each page server and printer extracts from the multicast cryptography, uses the same key to encrypt and decrypt a streams those objects referred to by its page layouts. The page message. Two parties wishing to exchange messages must servers persistently archive the received page layouts and first arrange to securely exchange the secret key. shared content. Other techniques for distributing the data may Public-key cryptography, also referred to as asymmetric be used. cryptography, uses two encryption keys. The two keys are 25 mathematically related in Such a way that any message Once a printer has received all the objects to which its page encrypted using one key can only be decrypted using the other layouts refer, the printer re-creates the fully-populated layout key. One of these keys is then published, while the other is and then rasterizes and prints it. kept private. The public key is used to encrypt any message Under normal circumstances, the printerprints pages faster intended for the holder of the private key. Once encrypted than they can be delivered. Assuming a quarter of each page 30 using the public key, a message can only be decrypted using is covered with images, the average page has a size of less the private key. Thus two parties can securely exchange mes than 400 KB. The printer can therefore hold in excess of 100 sages without first having to exchange a secret key. To ensure Such pages in its internal 64 MB memory, allowing for tem that the private key is secure, it is normal for the holder of the porary buffers etc. The printerprints at a rate of one page per private key to generate the key pair. second. This is equivalent to 400 KB or about 3 Mbit of page 35 Public-key cryptography can be used to create a digital data per second, which is similar to the highest expected rate signature. The holder of the private key can create a known of page data delivery over a broadband network. hash of a message and then encrypt the hash using the private Even under abnormal circumstances, such as when the key. Anyone can then verify that the encrypted hash consti printer runs out of paper, it is likely that the user will be able tutes the “signature' of the holder of the private key with to replenish the paper supply before the printer's 100-page 40 respect to that particular message by decrypting the encrypted internal storage capacity is exhausted. hash using the public key and Verifying the hash against the However, if the printers internal memory does fill up, then message. If the signature is appended to the message, then the the printer will be unable to make use of a multicast when it recipient of the message can verify both that the message is first occurs. The netpage publication server therefore allows genuine and that it has not been altered in transit. printers to submit requests for re-multicasts. When a critical 45 To make public-key cryptography work, there has to be a number of requests is received or a timeout occurs, the server way to distribute public keys which prevents impersonation. re-multicasts the corresponding shared objects. This is normally done using certificates and certificate Once a document is printed, a printer can produce an exact authorities. A certificate authority is a trusted third party duplicate at any time by retrieving its page layouts and con which authenticates the connection between a public key and tents from the relevant page server. 50 someone's identity. The certificate authority verifies the per son's identity by examining identity documents, and then 2.7 On-Demand Documents creates and signs a digital certificate containing the person’s When a netpage document is requested on demand, it can identity details and public key. Anyone who trusts the certifi be personalized and delivered in much the same way as a cate authority can use the public key in the certificate with a periodical. However, since there is no shared content, deliv 55 high degree of certainty that it is genuine. They just have to ery is made directly to the requesting printer without the use verify that the certificate has indeed been signed by the cer of multicast. tificate authority, whose public key is well-known. When a non-netpage document is requested on demand, it In most transaction environments, public-key cryptogra is not personalized, and it is delivered via a designated phy is only used to create digital signatures and to securely netpage formatting server which reformats it as a netpage 60 exchange secret session keys. Secret-key cryptography is document. A netpage formatting server is a special instance of used for all other purposes. a netpage publication server. The netpage formatting server In the following discussion, when reference is made to the has knowledge of various Internet document formats, includ secure transmission of information between a netpage printer ing Adobe's Portable Document Format (PDF), and Hyper and a server, what actually happens is that the printer obtains text Markup Language (HTML). In the case of HTML, it can 65 the server's certificate, authenticates it with reference to the make use of the higher resolution of the printed page to certificate authority, uses the public key-exchange key in the present Web pages in a multi-column format, with a table of certificate to exchange a secret session key with the server, US 7463,250 B2 35 36 and then uses the secret session key to encrypt the message netpage registration server database at time of manufacture. data. A sessionkey, by definition, can have an arbitrarily short During registration, the pen transmits its penID to the printer, lifetime. and the printer transmits the penID to the netpage registration 3.2 Netpage Printer Security server. The server generates a session key for the printer and Each netpage printer is assigned a pair of unique identifiers pen to use, and securely transmits the session key to the at time of manufacture which are stored in read-only memory printer. It also transmits a copy of the session key encrypted in the printer and in the netpage registration server database. with the pen's key-exchange key. The printer stores the ses The first ID 62 is public and uniquely identifies the printer on sion key internally, indexed by the pen ID, and transmits the the netpage network. The second ID is secret and is used when encrypted session key to the pen. The pen stores the session the printer is first registered on the network. 10 key internally, indexed by the printer ID. When the printer connects to the netpage network for the Although a fake pen can impersonate a pen in the pen first time after installation, it creates a signature public/private registration protocol, only a real pen can decrypt the session key pair. It transmits the secret ID and the public key securely key transmitted by the printer. to the netpage registration server. The server compares the secret ID against the printer's secret ID recorded in its data 15 When a previously unregistered pen is first registered, it is base, and accepts the registration if the IDs match. It then of limited use until it is linked to a user. A registered but creates and signs a certificate containing the printer's public “un-owned' pen is only allowed to be used to request and fill ID and public signature key, and stores the certificate in the in netpage user and pen registration forms, to register a new registration database. user to which the new pen is automatically linked, or to add a The netpage registration server acts as a certificate author new pen to an existing user. ity for netpage printers, since it has access to secret informa The pen uses secret-key rather than public-key encryption tion allowing it to verify printer identity. because of hardware performance constraints in the pen. When a user subscribes to a publication, a record is created in the netpage registration server database authorizing the 3.4 Secure Documents publisher to print the publication to the user's default printer 25 The netpage system Supports the delivery of secure docu or a specified printer. Every document sent to a printer via a ments such as tickets and coupons. The netpage printer page server is addressed to a particular user and is signed by includes a facility to print watermarks, but will only do so on the publisher using the publisher's private signature key. The request from publishers who are suitably authorized. The page server verifies, via the registration database, that the publisher indicates its authority to print watermarks in its publisher is authorized to deliver the publication to the speci 30 fied user. The page server verifies the signature using the certificate, which the printer is able to authenticate. publisher's public key, obtained from the publisher's certifi The “watermark printing process uses an alternative cate stored in the registration database. dither matrix in specified “watermark” regions of the page. The netpage registration server accepts requests to add Back-to-back pages contain mirror-image watermark regions printing authorizations to the database, so long as those 35 which coincide when printed. The dither matrices used in odd requests are initiated via a pen registered to the printer. and even pages watermark regions are designed to produce 3.3 Netpage Pen Security an interference effect when the regions are viewed together, Each netpage pen is assigned a unique identifier at time of achieved by looking through the printed sheet. The effect is similar to a watermark in that it is not visible manufacture which is stored in read-only memory in the pen 40 and in the netpage registration server database. The penID 61 when looking at only one side of the page, and is lost when the uniquely identifies the pen on the netpage network. page is copied by normal means. A netpage pen can "know a number of netpage printers, Pages of secure documents cannot be copied using the and a printer can "know a number of pens. A pen commu built-in netpage copy mechanism described in Section 1.9 nicates with a printer via a radio frequency signal whenever it 45 above. This extends to copying netpages on netpage-aware is within range of the printer. Once a pen and printer are photocopiers. registered, they regularly exchange session keys. Whenever the pen transmits digital ink to the printer, the digital ink is Secure documents are typically generated as part of e-com always encrypted using the appropriate session key. Digital merce transactions. They can therefore include the user's ink is never transmitted in the clear. 50 photograph which was captured when the user registered A pen stores a session key for every printer it knows, biometric information with the netpage registration server, as indexed by printer ID, and a printer stores a session key for described in Section 2. every pen it knows, indexed by pen ID. Both have a large but When presented with a secure netpage document, the finite storage capacity for session keys, and will forget a recipient can verify its authenticity by requesting its status in session key on a least-recently-used basis if necessary. 55 the usual way. The unique ID of a secure document is only When a pen comes within range of a printer, the pen and valid for the lifetime of the document, and secure document printer discover whether they know each other. If they don’t IDs are allocated non-contiguously to prevent their prediction know each other, then the printer determines whether it is by opportunistic forgers. A secure document verification pen Supposed to know the pen. This might be, for example, can be developed with built-in feedback on verification fail because the pen belongs to a user who is registered to use the 60 ure, to Support easy point-of-presentation document verifica printer. If the printer is meant to know the pen but doesn't, tion. then it initiates the automatic pen registration procedure. If Clearly neither the watermark nor the user's photograph the printer isn't meant to know the pen, then it agrees with the are secure in a cryptographic sense. They simply provide a pen to ignore it until the pen is placed in a charging cup, at significant obstacle to casual forgery. Online document veri which time it initiates the registration procedure. 65 fication, particularly using a verification pen, provides an In addition to its public ID, the pen contains a secret key added level of security where it is needed, but is still not exchange key. The key-exchange key is also recorded in the entirely immune to forgeries. US 7463,250 B2 37 38 3.5 Non-Repudiation 4.3 Micro-Payments In the netpage system, forms Submitted by users are deliv The netpage system includes a mechanism for micro-pay ered reliably to forms handlers and are persistently archived ments, to allow the user to be conveniently charged for print on netpage page servers. It is therefore impossible for recipi ing low-cost documents on demand and for copying copy ents to repudiate delivery. right documents, and possibly also to allow the user to be E-commerce payments made through the system, as reimbursed for expenses incurred in printing advertising described in Section 4, are also impossible for the payee to material. The latter depends on the level of subsidy already repudiate. provided to the user. When the user registers for e-commerce, a network 4. Electronic Commerce Model 10 account is established which aggregates micro-payments. 4.1 Secure Electronic Transaction (Set) The user receives a statement on a regular basis, and can settle The netpage system uses the Secure Electronic Transaction any outstanding debit balance using the standard payment (SET) system as one of its payment systems. SET, having mechanism. been developed by MasterCard and Visa, is organized around The network account can be extended to aggregate Sub payment cards, and this is reflected in the terminology. How 15 scription fees for periodicals, which would also otherwise be ever, much of the system is independent of the type of presented to the user in the form of individual statements. accounts being used. Other payment systems may also be 4.4 Transactions used. When a user requests a netpage in a particular application In SET, cardholders and merchants register with a certifi context, the application is able to embed a user-specific trans cate authority and are issued with certificates containing their action ID 55 in the page. Subsequent input through the page public signature keys. The certificate authority verifies a card is tagged with the transaction ID, and the application is holder's registration details with the card issuer as appropri thereby able to establish an appropriate context for the user's ate, and Verifies a merchant's registration details with the input. acquirer as appropriate. Cardholders and merchants store When input occurs through a page which is not user-spe their respective private signature keys securely on their com 25 cific, however, the application must use the user's unique puters. During the payment process, these certificates are identity to establish a context. A typical example involves used to mutually authenticate a merchant and cardholder, and adding items from a pre-printed catalog page to the user's to authenticate them both to the payment gateway. virtual “shopping cart'. To protect the user's privacy, how SET has not yet been adopted widely, partly because card 30 ever, the unique user ID 60 known to the netpage system is holder maintenance of keys and certificates is considered preferably not divulged to applications. This is to prevent burdensome. Interim solutions which maintain cardholder different application providers from easily correlating inde keys and certificates on a server and give the cardholder pendently accumulated behavioral data. access via a password have met with some Success. The netpage registration server instead maintains an anonymous relationship between a user and an application 4.2 Set Payments 35 via a unique alias ID 65, as shown in FIG. 24. Whenever the In the netpage system the netpage registration serveracts as user activates a hyperlink tagged with the “registered a proxy for the netpage user (i.e. the cardholder) in SET attribute, the netpage page server asks the netpage registration payment transactions. server to translate the associated application ID 64, together The netpage system uses biometrics to authenticate the 40 with the pen ID 61, into an alias ID 65. The alias ID is then user and authorize SET payments. Because the system is submitted to the hyperlink's application. pen-based, the biometric used is the users on-line signature, The application maintains state information indexed by consisting of time-varying pen position and pressure. A fin alias ID, and is able to retrieve user-specific state information gerprint biometric can also be used by designing a fingerprint without knowledge of the global identity of the user. sensor into the pen, although at a higher cost. The type of 45 The system also maintains an independent certificate and biometric used only affects the capture of the biometric, not private signature key for each of a user's applications, to the authorization aspects of the system. allow it to sign application transactions on behalf of the user The first step to being able to make SET payments is to using only application-specific information. register the user's biometric with the netpage registration To assist the system in routing product bar code (UPC) “hyperlink’ activations, the system records a favorite appli server. This is done in a controlled environment, for example 50 a bank, where the biometric can be captured at the same time cation on behalf of the user for any number of product types. as the user's identity is verified. The biometric is captured and Each application is associated with an application pro stored in the registration database, linked to the user's record. vider, and the system maintains an account on behalf of each The user's photograph is also optionally captured and linked application provider, to allow it to credit and debit the pro to the record. The SET cardholder registration process is 55 vider for click-through fees etc. completed, and the resulting private signature key and certifi An application provider can be a publisher of periodical cate are stored in the database. The user's payment card subscribed content. The system records the user's willingness information is also stored, giving the netpage registration to receive the subscribed publication, as well as the expected server enough information to act as the user's proxy in any frequency of publication. SET payment transaction. 60 4.5 Resource Descriptions and Copyright When the user eventually supplies the biometric to com A preferred embodiment of a resource description class plete a payment, for example by signing a netpage order form, diagram is shown in FIG. 40. the printer securely transmits the order information, the pen Each document and content object may be described by ID and the biometric data to the netpage registration server. one or more resource descriptions 842. Resource descriptions The server verifies the biometric with respect to the user 65 preferably use the Dublin Core metadata element set, which is identified by the pen. ID, and from then on acts as the user's designed to facilitate discovery of electronic resources. Dub proxy in completing the SET payment transaction. lin Core metadata conforms to the World WideWeb Consor US 7463,250 B2 39 40 tium (W3C) Resource Description Framework (RDF). Other 5.2 Hyperlink Activation Protocol metadata element sets may be used. A preferred embodiment of a hyperlink activation protocol A resource description may identify rights holders 920. is shown in FIG. 45. The netpage system automatically transfers copyright fees When a user clicks on a netpage with a netpage pen, the pen from users to rights holders when users print copyright con communicates the click to the nearest netpage printer 601. tent. The click identifies the page and a location on the page. The 5. Communications Protocols printer already knows the ID 61 of the pen from the pen A communications protocol defines an ordered exchange connection protocol. of messages between entities. In the netpage system, entities The printer determines, via the DNS, the network address Such as pens, printers and servers utilize a set of defined 10 of the page server 10a handling the particular page ID 50. The protocols to cooperatively handle user interaction with the address may already be in its cache if the user has recently netpage System. interacted with the same page. The printer then forwards the Each protocol is illustrated by way of a sequence diagram pen ID, its own printer ID 62, the page ID and click location in which the horizontal dimension is used to represent mes to the page server. sage flow and the vertical dimension is used to represent time. 15 The page server loads the page description 5 identified by Each entity is represented by a rectangle containing the name the pageID and determines which input element’s Zone 58, if of the entity and a vertical column representing the lifeline of any, the click lies in. Assuming the relevant input element is a the entity. During the time an entity exists, the lifeline is hyperlink element 844, the page server then obtains the asso shown as a dashed line. During the time an entity is active, the ciated application ID 64 and link ID 54, and determines, via lifeline is shown as a double line. Because the protocols the DNS, the network address of the application server host considered here do not create or destroy entities, lifelines are ing the application 71. generally cut short as soon as an entity ceases to participate in The page server uses the pen ID 61 to obtain the corre a protocol. sponding userID 60 from the registration server 11, and then allocates a globally unique hyperlink request ID 52 and builds 5.1 Subscription Delivery Protocol 25 a hyperlink request 934. The hyperlink request class diagram A preferred embodiment of a subscription delivery proto is shown in FIG. 44. The hyperlink request records the IDs of col is shown in FIG. 43. the requesting user and printer, and identifies the clicked A large number of users may subscribe to a periodical hyperlink instance 862. The page server then sends its own publication. Each user's edition may be laid out differently, server ID 53, the hyperlink request ID, and the link ID to the but many users’ editions will share common content such as 30 application. text objects and image objects. The Subscription delivery The application produces a response document according protocol therefore delivers document structures to individual to application-specific logic, and obtains a document ID 51 printers via pointcast, but delivers shared content objects via from an ID server 12. It then sends the document to the page multicast. server 10b responsible for the document’s newly allocated The application (i.e. publisher) first obtains a document ID 35 ID, together with the requesting page server's ID and the 51 for each document from an ID server 12. It then sends each hyperlink request ID. document structure, including its document ID and page The second page server sends the hyperlink request ID and descriptions, to the page server 10 responsible for the docu application ID to the first page server to obtain the corre ment’s newly allocated ID. It includes its own application ID sponding user ID and printer ID 62. The first page server 64, the subscriber's alias ID 65, and the relevant set of mul 40 rejects the request if the hyperlink request has expired or is for ticast channel names. It signs the message using its private a different application. signature key. The second page server allocates document instance and The page server uses the application ID and alias ID to page IDs 50, returns the newly allocated page IDs to the obtain from the registration server the corresponding userID application, adds the complete document to its own database, 60, the user's selected printer ID 62 (which may be explicitly 45 and finally sends the page descriptions to the requesting selected for the application, or may be the user's default printer. printer), and the application’s certificate. The hyperlink instance may include a meaningful transac The application’s certificate allows the page server to tion ID 55, in which case the first page server includes the Verify the message signature. The page server's request to the transaction ID in the message sent to the application. This registration server fails if the applicationID and alias ID don't 50 allows the application to establish a transaction-specific con together identify a subscription 808. text for the hyperlink activation. The page server then allocates document and page instance If the hyperlink requires a user alias, i.e. its “alias required IDS and forwards the page descriptions, including page IDS attribute is set, then the first page server sends both the penID 50, to the printer. It includes the relevant set of multicast 61 and the hyperlink's application ID 64 to the registration channel names for the printer to listen to. 55 server 11 to obtain not just the userID corresponding to the It then returns the newly allocated pageIDS to the applica pen ID but also the alias ID 65 corresponding to the applica tion for future reference. tion ID and the user ID. It includes the alias ID in the message Once the application has distributed all of the document sent to the application, allowing the application to establish a structures to the subscribers selected printers via the relevant user-specific context for the hyperlink activation. page servers, it multicasts the various Subsets of the shared 60 objects on the previously selected multicast channels. Both 5.3 Handwriting Recognition Protocol page servers and printers monitor the appropriate multicast When a user draws a stroke on a netpage with a netpage channels and receive their required content objects. They are pen, the pen communicates the stroke to the nearest netpage then able to populate the previously pointcast document printer. The stroke identifies the page and a path on the page. structures. This allows the page servers to add complete docu 65 The printer forwards the penID 61, its own printer ID 62, ments to their databases, and it allows the printers to print the the page ID 50 and stroke path to the page server 10 in the documents. usual way.