USOO82O1096B2

(12) Patent (10) Patent No.: US 8,201,096 B2 Robert et al. (45) Date of Patent: Jun. 12, 2012

(54) BROWSING OR SEARCHING USER 5,864,868 A 1/1999 Contois INTERFACES AND OTHER ASPECTS 5,880,733. A 3/1999 Horvitz 5,960,448 A 9, 1999 Reichek et al. 5,974,549 A 10, 1999 Golan (75) Inventors: Julien Robert, Paris (FR): Cedric Bray, 6,006,227 A 12/1999 Freeman et al. Vincennes (FR); Thomas Goossens, 6,023,275 A 2/2000 Horvitz et al. Paris (FR) 6,064,380 A 5, 2000 Swenson et al. 6,202,061 B1 3/2001 Khosla et al. (73) Assignee: Apple Inc., Cupertino, CA (US) 6,262,7246,215,523 B1 4/20017/2001 CrowAnderson et al. (*) Notice: Subject to any disclaimer, the term of this 85.63 R ck 658 st al...... 715.835 patent is extended or adjusted under 35 6,346,951 B1 2/2002 Mastronardi

(21) Appl. No.: 11/760,759 FOREIGN PATENT DOCUMENTS (22) Filed: Jun. 9, 2007 EP 1 148 412 A2 10, 2001 (Continued) (65) Prior Publication Data US 2008/0307343 A1 Dec. 11, 2008 OTHER PUBLICATIONS Andrew Coulter Enright “The Fliptych Interface”. The treehouse + (51) Int. Cl. the cave, Aug. 6, 20062 pages.* G06F 3/048 (2006.01) G06F 3/00 (2006.01) (Continued) (52) U.S. Cl...... 715/767; 715/713; 715/732; 715/757; - - - - 71.5/850 Primary Examiner — William Bashore (58) Field of Classification Search ...... 715/713, Assistant Examiner – Andrea Long 715/732, 757, 767, 850 (74) Attorney, Agent, or Firm — Jones Day See application file for complete search history. (57) ABSTRACT (56) References Cited User interfaces for browsing and/or searching are described. In one embodiment, a method includes displaying a first U.S. PATENT DOCUMENTS display area for display representations of documents match 4,847,781 A 7, 1989 Brown et al. 5,303,388 A 4, 1994 Kreitman et al. ing a search query, the first display area configured to display 5,345,543 A 9/1994 Capps et al. content of the documents which can have a plurality of dif 5,396,590 A 3/1995 Kreegar ferent types of content including at least one of text-based 5,404,316 A 4/1995 Klingler et al. content and a folder, and displaying a second display area for 5,452,414 A 9, 1995 Rosendahl et al. selecting a selected document to be displayed in the first 5,463,725 A 10, 1995 Henckel et al. 5,555,354 A 9, 1996 Strasnicket al. display area. Other embodiments are also described, and 5,586,237 A 12/1996 Baecker computer readable media and apparatuses are also described. 5,831,617 A 11/1998 Bhukhanwala 5,838,326 A 11/1998 Card et al. 18 Claims, 95 Drawing Sheets

| 777 assaching thenae

D Network Backup A SiDiskgger fan

abscons &k Yourt Your 2343 :3. Conneca Scheule Aisi 238

AAAplications UDilles documents Moviss 2348 -Pictures

dociat March 23, 2007 234.7 ckcument March 24, 2007 2349

2326 3 O 6 is sig 231

2361 2514 2319 US 8,201,096 B2 Page 2

U.S. PATENT DOCUMENTS 2005/0010860 A1 1/2005 Weiss et al. 2005, OO12723 A1 1/2005 Pallakoff 6,401,097 B1 6/2002 McCotter et al. 2005/0024341 A1 2/2005 Gillespie et al. 6,448,987 B1 9/2002 Easty et al. 2005, OO27539 A1 2/2005 Weber et al. 6,486,895 B1 1 1/2002 Robertson et al. 2005/0060666 A1 3/2005 Hoshino et al. 6,489,951 B1 12/2002 Wong et al. 2005/009 1596 A1 4/2005 Anthony et al. & ES R 23.8 E. sal 2005/0102373 A1 5/2005 Grinberg 6.577.330- B 6/2003 Stil 2005,2005/01 0125728 10768 A1 6,5/2005 2005 PeiroMarriott et al. et al. 88: R 338 Metal. 2005/0132055 A1 6/2005 Neogi 6,638,313wkw B1 10/2003 Freeman et al. 2005,5688. 0138664 A1A 6, 562005 NeogWinet al. 6,654,030 B1 1 1/2003 Hui 2005. O155077 A1 7, 2005 Lawrence et al. 6,654,754 B1 1 1/2003 Knauft et al. 2005, 01985.74 A1 9, 2005 Lamkin et al. 6,667,751 B1 12/2003 Wynn et al. 2005/024.0880 A1 10, 2005 Banks et al. 6,725,427 B2 4/2004 Freeman et al. 2005/0246324 A1 11/2005 Paalasmaa et al. 87.6 R Z39. has sal 2005/0283739 A1 12/2005 Mohr et al. 6868,225 B1 3/2005 Escist 2005/0289482 Al 12/2005 Anthony et al. sy sy 2006/0001645 A1 1/2006 Drucker et al. 866. R $383 SE;d al. 1 2006/0026535 A1 2/2006 Hotelling et al. 69455 B2 7/2005 RE O et al. 2006, OO31357 A1 2/2006 Misra et al. 6,944,819 B2 92005 Banatwala et al 2006/0036959 A1 2/2006 Heatherly et al. 6,975,306 B2 12/2005 Hinckley et al. 38883. A 38 Ea $392, R 3. SE 2006/0074869 A1 4/2006 Rosenberg et al. 7085,590 B2 82006 RNdy al. 2006/0085442 A1 4/2006 Fujiwara 708576 B2 & 2006 Shibata 2006/0095865 A1 5.2006 Rostom 7,096,431 B2 8/2006 Tambata et al. 39.83, A. 38. NGsieh et al. 7,143,102 B2 11/2006 Fiennes et al. 2006/0156246 A1 7/2006 Williams et al. 7,146,388 B2 12/2006 Stakutis et al. 2006/0174214 A1* 8, 2006 McKee et al...... 71.5/8O2 7,162,466 B2 1/2007 Kaastenet al. 2006/0195512 A1 8/2006 Rogers et al. 7,162,488 B2 1/2007 DeVorchik et al. 2006/0195790 A1* 8/2006 Beaupre et al...... 71.5/727 7.205,471 B2 4/2007 Looney et al. 2006/0206428 A1 9, 2006 Vidos et al. 7,318,047 B1 1/2008 Foth et al. 2006/0206779 A1 9, 2006 Wehn et al. 7.333,982 B2 2/2008 Bakalash et al. 2006/0238835 A1 10, 2006 Nishida et al. 223 E: 1858 Rish etal 1f1 2006/0265417 A1 11/2006 Amato et al. 7479,949 B2 1/2009 Eyet ...... 2006/0280437 Al 12/2006 Logan et al. 7.581,1867,492.350 B2 2/20098, 2009 FabreDowdy et et al. al. 3.289 A. 58. 8thstoic et alal. 788- w 5 SS, WE 2007/00332952007/004403.6 A1Al 2,2/2007 2007 IshimuraMarriott et al. 7,647,278 B1 1/2010 Foth et al. 2007.0053268 A1 3, 2007 Crandall et al. 23. R 3.338 s st al. 2007/0061306 A1 3, 2007 Pell et al. 7.689,933 B1 3/2010 E. 2007/0061745 A1 3/2007 Anthony et al. s -- 2007/0070.066 A1 3, 2007 Bakhash 2.88. R: 388 Ri al 2007, 00792.59 A1 4, 2007 Sauve et al. 7753.337 B 7200 Elg' et al. 2007, 0080823 A1 4, 2007 Fu - 4 y 2007/00836.16 A1 4/2007 Madden et al. 2006.7% R 3. &ntal 2007/0083911 A1 4/2007 Madden et al. 2001/0028363 A1 10, 2001 Nomoto et al. 3.29. A. $39, SR al. 2001/0050687 A1 12, 2001 Iida et al. 2007/O124680 A1 5, 2007 Robbin et all 2002/003 0699 A1 3, 2002 Van Ee 2007/0136750 Ai 6/2007 Abanamietal 2002/0054017 A1 5/2002 Agata et al. 20070143245 Ai 62007 Dettingeretal 2002/009 1662 A1 7/2002 Bogia 2007/0143493 A1 6/2007 Mulligelunger et al.et al. 3838-35. A. 858: My 2007. O156669 A1 7, 2007 Marchisio et al. 2002fO196266 A1 12, 2002 Met al 2007. O157095 A1 7, 2007 Bilow et al. 2002.0198909. A 12/2002 Huynh etal 2007. O168413 A1 7, 2007 Barletta et al. 2003/0016951 A1 1/2003 Jakel et al. 587-88 A. 38 ermanet al. et al. 2003/0076322 A1 4, 2003 OuztSet al. 2007. O189737 A1 8, 2007 Chaudhri et al. 2003/0093572 AI 52003 Laux et al. 2007/0260994 A1 11/2007 Sciammarella et al. 38885 A. 23.8 (sman et al. 2007/0266411 A1 11/2007 Yamamoto et al. 2003.0156140 A1 8, 2003 Watanabe 38885. A 58 Royee a 2003/01896O22004/0013416 A1 10,1/2004 2003 MokDalton et al. 2008.0034306'83. A1A 2/2008558 OrdinER 2004/0047588 A1 3f2004 Okada et al. 38885 A 558 Eal 2004.0056837 A1 3f2004 Koga et al. alon et al. 2004/0095396 A1 5, 2004 Stal 2008.0036743 A1 2/2008 Westerman et al. 2004/0100479 A1 5/2004 Nakano et al. 2008.0062141 A1 3/2008 Chandhri 2004/O122683 A1 6/2004 Grossman et al. 2008/0066016 A1 3/2008 Dowdy et al. 2004/O150671 A1 8, 2004 Kamiwada et al. 2008/0066135 A1 3, 2008 Brodersen et al. 2004/0155907 A1 8/2004 Yamaguchi et al. 2008/01 15182 A1 5/2008 van Willigenburg 2004O162845 A1 8, 2004 Kim et al. 2008/O122796 A1 5/2008 JobS et al. 2004/0199582 A1 10, 2004 Kucharewski et al. 2008/0307343 A1 12/2008 Robert et al. 2004/0223737 A1 11, 2004 Johnson 2008/0307350 A1 12/2008 Sabatelli et al. 2004/0230599 A1 11/2004 Moore et al. 2008/0307363 A1 12/2008 Jalon et al. 2004/0235520 A1 11, 2004 Cadiz et al. 2009 OOO2335 A1 1/2009 Chaudhri US 8,201,096 B2 Page 3

2009/0106674 A1 4/2009 Bray et al. Office Action dated Jan. 13, 2011 for U.S. Appl. No. 1 1/876,750, 2009/O112719 A1 4/2009 Bhave et al. entitled “Previewing User Interfaces and Other Aspects' filed Oct. 2009.0125842 A1 5/2009 Nakayama 22, 2007, 11 pages. 2009. O150769 A1 6/2009 Konnola et al. 2009. O150792 A1 6/2009 Laakso et al. Office Action dated Nov. 19, 2010 for U.S. Appl. No. 12, 189.673, 2009/0172532 A1 7/2009 Chaudhri entitled “Browsing or Searching User Interfaces and Other Aspects' 2009/O198359 A1 8, 2009 Chaudhri filed Aug. 11, 2008, 21 pages. 2009,024921.0 A1 10/2009 Sheldon et al. Office Action dated May 26, 2010 for U.S. Appl. No. 12, 189.673, 2009,0300540 A1 12/2009 Russell entitled “Browsing or Searching User Interfaces and Other Aspects' filed Aug. 11, 2008, 18 pages. FOREIGN PATENT DOCUMENTS Office Action dated May 28, 2009 for European Patent Application EP 1677 497 A1 T 2006 No. 07836525.1-2201, 4 pages. WO WO99/49496 A1 9, 1999 Office Action dated Dec. 3, 2010 for U.S. Appl. No. 1 1/760,760, WO WOOOf 63766 A 10, 2000 entitled “Browsing or Searching User Interfaces and Other aspects' WO WOO1,7998O A1 10, 2001 filed Jun. 9, 2007, 19 pages. WO WO 2005,121920 12/2005 Office Action dated Jan. 6, 2010 for U.S. Appl. No. 1 1/760,761, WO WO 2006/013485 A2 2, 2006 entitled “Browsing or Searching User Interfaces and Other Aspects' WO WO 2006/02O305 A2 2, 2006 filed Jun. 9, 2007. 12 pages. WO WO 2006/032442 A 3, 2006 WO WO 2007/005789 A 1, 2007 Office Action dated Dec. 8, 2010 for U.S. Appl. No. 12, 189.668, WO WO 2008/O19113 2, 2008 entitled “Browsing or Searching User Interaces and Other Aspects' WO WO 2008/030779 A2 3, 2008 filed Aug. 11, 2008, 21 pages. WO WO 2008/030976 A2 3, 2008 Office Action dated Jul. 8, 2010 for U.S. Appl. No. 1 1/760,761, WO WO 2008,153646 A1 12/2008 entitled “Browsing or Searching User Interfaces and Other Aspects' WO WO 2008,153647 A1 12/2008 filed Jun. 9, 2007, 14 pages. Ording, Bas, et al., “Methods and Apparatuses for Operating a Data OTHER PUBLICATIONS Processing System.” U.S. Appl. No. 10/873,661, filed Jun. 21, 2004 (92 pages). Apples iTunes 7 CNET editors' review Sep. 15, 2006 6 pages.* PCT International Search Report and Written Opinion of the Inter Windows MediaPlayer 11—The Other album art aware app! May 11, national Searching Authority for PCT/US2008/005901 dated Sep. 2006 4 pages.* 11, 2008, 14 pages. Adam C. Engst iPhoto 6 for Mac OS X: Visual QuickStart Guide PCT Notification of International Preliminary Report on Patentabil Peachpit Press Jun. 20, 2006 pp. 9, 45, 50, 100, 105, 122, 144-146.* ity and Written Opinion of the International Searching Authority for Greg Perry “Sams Teach Yourself Windows XP All in One” Second PCT/US2008/005901 dated Dec. 23, 2009, 9 pages. Edition Dec. 17, 2004 pp. 93-96.* PCT Notification of Transmittal of the International Search Report Andrew Coulter Enright, "Dissatisfaction Sows Innovation'. The and the Written Opinion of the International Searching Authority, or Treehouse + The Cave, http://thetreehouseandthecave.blogspot.com/ the Declaration for PCT/US2010/042606 dated Oct. 5, 2010, 13 2004/12/dissatisfaction-sows-innovation.html, Dec. 29, 2004, 6 pageS. pages, Bushwick, Brooklyn. Ricadela, Aaron, “New Windows Era.” InformationWeek, Aug. 1, Andrew Coulter Enright, "Dissatisfaction Sows Innovation'. The 2005, 3 pages. Treehouse + The Cave, http://thetreehouseandthecave.blogspot.com/ "Screenshots from iTunes 4.1 running on Windows XP' Internet 2004/12/dissatisfaction-sows-innovation.html, Jun. 15, 2006, 18 Citation, XP002486348, Oct. 5, 2005, pp. 1-17. pages, Bushwick, Brooklyn. Chaudhri, U.S. Appl. No. 12/215,651, filed Jun. 26, 2008. Australian Patent Office. Examiner's Report dated Feb. 15, 2010 for Cody Hinze, “CoverFlow—A beautiful way to browse your MP3s'. Australian Patent App. No. 2007296.558. Noise blog. http://noise.typepad.com/noise blog/2006/02/cover Australian Patent Office. Examiner's Report dated Mar. 3, 2011 for flow the html, Feb. 5, 2006, 2 pages. Australian Patent App. No. 2007296.558. PCT Search Report and Written Opinion for PCT/US2007/017424. Australian Patent Office. Examiner's Report dated Mar. 27, 2009 for mailed Feb. 4, 2008, 9 pages. Australian Innovation Patent App. No. 2008101162. Imran Chandri, U.S. Appl. No. 60/937,986, filed Jun. 29, 2007, 72 Australian Patent Office. Examiner's Report dated Apr. 1, 2009 for pages (specification). Australian Innovation Patent App. No. 2008101171. Thomas Dowdy, et al., U.S. Appl. No. 60/843,832, filed Sep. 11, Australian Patent Office. Examiner's Report dated Mar. 24, 2009 for 2006, 68 pages (specification and drawings). Australian Innovation Patent App. No. 2008101164. PCT International Search Report and Written Opinion for PCT Inter Canadian Patent Office. Examiner's Report dated Jun. 9, 2010 for national Appln No. PCT/US2008/005900 mailed Aug. 1, 2008. (14 Canadian Patent App. No. 266 1200. pages). Chinese Patent Office, First Office Action for Chinese Patent App. U.S. Appl. No. 60/843,832, filed Sep. 11, 2006, titled “Techniques No. 200780033722.5 dated Jul. 8, 2010. and Systems for Browsing Media Content”, by inventors Thomas Chinese Patent Office, Second Office Action for Chinese Patent App. Dowdy, David Heller and Anne Jones, 68 pages (specification and No. 200780033722.5 dated Mar. 9, 2011. drawings). Chinese Patent Office, First Office Action for Chinese Patent App. No. 200780033703.2 dated Sep. 8, 2010. U.S. Appl. No. 60/878,746, filed Jan. 5, 2007, titled “Electronic German Patent Office, Office Action dated Jun. 25, 2010 for German Device With Image Based Browsers', by inventors Thomas Dowdy, Patent App. No. 11 2007 002 143.8-53. David Heller and Anne Jones, 98 pages. (specification and drawings). Korean Patent Office, Office Action dated Nov. 16, 2010 for Korean "AppleTiger vs Windows Vista.” Jun. 25, 2006, 3 pages. Patent App. No. 10-2009-7005010. Apple, " Programming Guide—Data Management: File Korean Patent Office, Office Action dated May 13, 2011 for Korean Management.XP007915043, Jul. 20, 2009, pp. 1-50. Patent App. No. 10-2009-7005010. Apples iTunes 7 CNET Editors' Review, Sep. 15, 2006, 6 pages). Korean Patent Office, Office Action dated May 13, 2011 for Korean Microsoft Corporation, “Creating Windows XP Icons.” Jul. 2001, 12 Patent App. No. 10-2011-7003607. pageS. Korean Patent Office, Office Action dated May 13, 2011 for Korean Microsoft Corporation, “How to Modify your Folder View Settings Patent App. No. 10-2011-7003609. or to do a Custom Folder”, Apr. 27, 2004, 5 pages. UK Patent Office, Office Action dated Sep. 24, 2010 for United Office Action dated Aug. 13, 2010 for U.S. Appl. No. 1 1/876,750, Kingdom Patent App. No. 0905.084.0. entitled “Previewing User Interfaces and Other Aspects' filed Oct. UK Patent Office, Office Action dated Aug. 23, 2011 for United 22, 2007, 8 pages. Kingdom Patent App. No. 0905.084.0. US 8,201,096 B2 Page 4

UK Patent Office, Office Action dated Oct. 17, 2011 for United Labarge, Ralph, DVD Authoring & Production, 2001, Focal Press, Kingdom Patent App. No. 0905.084.0. pp. 113 and 252. UK Patent Office, Office Action dated May 11, 2011 for United XP for Home Users Service Pack 2nd Edition, Kingdom Patent App. No. 0905.084.5. Jul. 7, 2005, Peachpit Press, 13 pages. UK Patent Office, Search and Examination Report dated Sep. 23, Microsoft, Windows XP Media Center Edition, First Released: Jan. 2011 for United Kingdom Patent App. No. 1114593.5. 2002–Last released: Nov. 2004. UK Patent Office, Search and Examination Report dated Sep. 15, Remote for iTunes (URM-15T), product information datasheet, 2011 for United Kingdom Patent App. No. 1114592.7. Keyspan, http://keyspan.com/products/usb?urm15t,downloaded USPTO, Office Action dated Nov. 14, 2008 for U.S. App. No. Sep. 25, 2005, pp. 1-2. 1 1/519,416, filed Sep. 12, 2006. Express Remote (URM-17A), product information datasheet, USPTO, Restriction Requirement dated Sep. 21, 2010 for U.S. Appl. Keyspan, http://keyspan.com/products/usb?urm15t,downloaded No. 1 1/767,409, filed May 22, 2007. Sep. 25, 2005, pp. 1-2. USPTO, Office Action dated Nov. 23, 2010 for U.S. Appl. No. “Keyspan Express Remote Supports Apple's Airport Express.” Press 11/767,409, filed May 22, 2007. Release, Keyspan, Nov. 10, 2004. USPTO, Final Office Action dated Mar 16, 2011 for U.S. Appl. No. “Keyspan Express Remote.” Product Fact Sheet, Keyspan, published 11/767,409, filed May 22, 2007. Sep. 25, 2005. USPTO, Office Action dated Aug. 13, 2010 for U.S. Appl. No. Keyspan:DMR Software v. 1.3 for Mac OS X User Manual rev. 1 1/876,750, filed Oct. 22, 2007. 04.09.01A, published 1998. USPTO, Office Action dated May 27, 2011 for U.S. Appl. No. Doom9 What's in a DVD, Apr. 28, 2001, http://web.archive.org/web/ 12/395,541, filed Feb. 27, 2009. 20010428.123010/http://www.doom9.org/dvd-structure.htm. USPTO, Office Action dated May 9, 2011 for U.S. Appl. No. DVD Authoring Terminology, Feb. 19, 2004, http://webarchive.org/ 12/395,537, filed Feb. 27, 2009. web/2004021921 1732. http://www.mainfest-tech.com/links/dvd ISA, PCT International Preliminary Report on Patentability and terms.htm. Written Opinion for PCT/US2008/005900 dated Dec. 23, 2009. Rose, M. “ in the Home: Interfaces for Music Applications.” ISA, PCT International Search Report and Written Opinion for PCT/ Personal Technologies, vol. 4. No. 1, pp. 45-53. US2007/078180 dated Mar. 3, 2008. Salling Software, Sailing Clicker 2.2.1. User Manual, Sailing ISA, PCT International Preliminary Reporton Patentability for PCT/ Software AB 2005. US2007/078180 dated Mar. 26, 2009. Sawyer, “Get with the CoverFlow.” The Olive Press, Dec. 8, 2005, p. ISA, PCT International Search Report and Written Opinion for PCT/ 1. US2007/078052 dated Feb. 12, 2008. Website printout, www.steelskies.com/coverflow/downloaded.php, ISA, PCT International Preliminary Reporton Patentability for PCT/ downloaded Jun. 15, 2006, pp. 1-4. US2007/078052 dated Mar. 26, 2009. Wikipedia, “Windows XP Media Center Edition.” http:/en. Slide Show www.digitalphotoslideshow.com/doc/configure.htm wikipedia.org/wiki/Windows XP Media Center Edition, down archived May 25, 2005, p. 1. loaded Sep. 27, 2005, pp. 1-5. "CoverFlow. www.steelskies.com/coverflow?, downloaded Jun. 15, Wikipedia, "Application Software.” Wikipedia, the free encyclope 2006, pp. 1-2. dia, http://en.wikipedia.org/wiki/Application Software, 3 pages, "CoverFlow—Features, www.steelskies.com/coverflow? features. downloaded May 22, 2006. php, downloaded Jun. 15, 2006, p. 1. Wikipedia, "Digital Audio.” Wikipedia, the free encyclopedia, http:// "CoverFlow—FAQ & Tips,” www.steelskies.com/coverflow/facq. en.wikipedia.org/wiki/Digital audio, 3 pages, downloaded May 25, php, downloaded Jun. 15, 2006, pp. 1-3. 2006. "CoverFlow—Download,” www.steelskies.com/coverflow? down Wikipedia, “Digital Media.” Wikipedia, the free encyclopedia, http:// loaded.php, downloaded Jun. 15, 2006, pp. 1-4. en.wikipedia.org/wiki/Digital media, 3 pages, date unknown. Crespo et al., “Responsive Interaction for a Large Web Application, Wikipedia, “Digital Video.” Wikipedia, the free encyclopedia, http:// The Meteor Shower Architecture in the WebWriter II Editor. Proc. en.wikipedia.org/wiki/Digital video, 3 pages, May 6, 2006. 6th Intn'l Work WideWeb Conf. 1996. Wikipedia, “ITunes.” Wikipedia, the free encyclopedia, http://en. Enright, Andrew Coulter, "Dissatisfaction Sows Innovation Visual wikipedia.org/wiki/ITunes, 11 pages, May 22, 2006. Browsing in iTunes.” Internet Blog of Andrew Coulter Enright, Dec. Wikipedia, “Media Player.” Wikipedia, the free encyclopedia, http:// 29, 2004, pp. 1-6. en.wikipedia.org/wiki/Media player, 1 page, May 20 2006. Enright, Andrew Coulter, "Dissatisfaction Sows Innovation.” The Wikipedia, “Mutimedia.” Wikipedia, the free encyclopedia, http://en. Treehouse + The Cave, Jun. 15, 2006, Bushwich, Brooklyn (18 wikipedia.org/wiki/"Mutimedia, 2 pages, May 25 2006. pages). NPL documents from Interview Jan. 11, 2012 (for U.S. Appl. No. Enright, Andrew Coulter, “Meet CoverFlow.” The Treehouse + The 11/760,761) Microsoft Windows 2000, USPTO, UTube and Way Cave, Aug. 13, 2005, pp. 1-2. Back Machine computer screenshots. Enright, Andrew Coulter, “Visual Browsing on a iBook DS.” The Zaitseff, John, Sample Page For the File-type icons Nov. 16, 2006 (2 Treehouse + The Cave, Dec. 29, 2004, p. 1. pages). IBM, "Mechanism for Visual Lists and Selections.” IBM Technical Disclosure Bulletin, vol. 40, No. 5, May 1, 1997, 2 pages. * cited by examiner U.S. Patent Jun. 12, 2012 Sheet 1 of 95 US 8,201,096 B2

Mass Storage Microprocessor(s) ROM RAM (e.g., HardDrive) 103 107 105 106

Bus(es) 102

Display Controller I/O and Controller(s)

Display Device 108 104

I/O Devices (e.g., Mouse Or Keyboard or Modem Or NetWork Interface Or CD Drive, etc.) 109

FIG. 1 U.S. Patent Jun. 12, 2012 Sheet 2 of 95 US 8,201,096 B2

Capture metadata from a variety 201 of different application programs

Make captured metadata available to a Searching facility (e.g., a file management 203 system Software for searching)

Allow searching of metadata across all 205 applications having Captured metadata

Provide User Interface of search engine 2O7 and Search results

FIG.2 U.S. Patent Jun. 12, 2012 Sheet 3 of 95 US 8,201,096 B2 301 JPEG image File Metadata ISO Setting Image Width Flash Setting Image Height fl Stop Image Color Space Camera Brand Information BitS/ User Added Keywords Other Fields.

FIG. 3A

331 MP3 Music File Metadata

Artist Song Play Times Genre Other Fields. Album Song names

FIG. 3B U.S. Patent Jun. 12, 2012 Sheet 4 of 95 US 8,201,096 B2

Importers Metadata Processing (+ Optionally Software Exporters) of Metadata Other 413 Applications

File System 415 Directory 419

f Content MetadataOr SoftwareOW3

417

Index 421 400 of

FIG. 4 U.S. Patent Jun. 12, 2012 Sheet 5 Of 95 US 8,201,096 B2

Receive notification of change for a file (e.g., receive notification from OS kernel or software application of 501 Creation of new file or modification of existing file or deletion of existing file).

Determine type of file (e.g., an Acrobat PDF file or an 503 RTF Word processing file or a JPEG image file, etc.)

Activate capture software for determined file type (e.g., a plug in for the application which Creates 505 the type of file)

Import appropriate metadata (for particular file type) 507 into a metadata database

Store metadata in a database (e.g., a flat file format database) 509

Receive search parameter inputs and perform search 511 of metadata database and display results

FIG. 5 U.S. Patent Jun. 12, 2012 Sheet 6 of 95 US 8,201,096 B2

603 Metadata from file 1 of Application A (File A1) 605 Metadata from file 1 of Application B (File B1) 607 Metadata from file 2 of Application A (File A2) 609 Metadata from file 1 of Application C (File C1) 611 Metadata from file 2 of Application B (File B2)

617 Metadata from filen of Application C (File Cn)

FIG. 6 U.S. Patent Jun. 12, 2012 Sheet 7 Of 95 US 8,201,096 B2

OLquêS U.S. Patent Jun. 12, 2012 Sheet 8 of 95 US 8,201,096 B2

|81

3. 3 3. 3 es it 3.

selnold|[[ LZZ U.S. Patent Sheet 9 of 95 US 8,201,096 B2

INN

CN 9)

CD C 3.

d >

c wn C E 3. 3 R 3a 3. EE e. E33 at 2:

|TÍseiniola U.S. Patent Jun. 12, 2012 Sheet 10 of 95 US 8,201,096 B2

CC an N N N

- H. ) D > N

O N

|0.1

19|

U.S. Patent Jun. 12, 2012 Sheet 11 of 95 US 8,201,096 B2

8!19|1

t 3 3. R3. e se 3. 3. es 3. el

ZZZ

U.S. Patent Jun. 12, 2012 Sheet 13 of 95 US 8,201,096 B2

Z08

88

SESSESSESSEES | \\\ \ 626262626262

|Z9 U.S. Patent Jun. 12, 2012 Sheet 14 of 95 US 8,201,096 B2

906

U.S. Patent Jun. 12, 2012 Sheet 15 Of 95 US 8,201,096 B2

190|| 990|| U.S. Patent US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 17 Of 95 US 8,201,096 B2

HEN

A,ÁgqoJees

880|| U.S. Patent US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 20 of 95 US 8,201,096 B2

60Z).

U.S. Patent Jun. 12, 2012 Sheet 21 Of 95 US 8,201,096 B2

aswouge?aw U.S. Patent Jun. 12, 2012 Sheet 22 of 95 US 8,201,096 B2

U.S. Patent US 8,201,096 B2

O e

N O

so

U.S. Patent Jun. 12, 2012 Sheet 25 Of 95 US 8,201,096 B2

GÐGOT

Nfff:ffff:ffff: U.S. Patent Jun. 12, 2012 Sheet 26. Of 95 US 8,201,096 B2

1418 1419 1420 1421 1422 1417 This Week's images Ov G 1415 folder Desktop. T Home Everwise-1 7 Csave 6 1416 5C, OG OG) Name A.

a 2.psd Oct 6, 1999, 11:02AM 9.5 MB Adob.imagf 3.psd Oct 6, 1999, 11:20 AM 8.8 MB Adob.imag 4.psd Oct 6, 1999, 11:24 AM 7.6 MB Adob... imag 8.psd Oct 6, 1999, 3:16 PM 8A8 Adob. Imag 9.psd Oct 6, 1999, 3:19 PM 7.3 MB Adob. Imag ABGeneral Prefs Module.tif Jun 17, 2004, 10:21 PM 4KB aff C. ABLandscape.tiff Jun 17, 2004, 10:21 PM 4KB FF C. ABLDAPPrefsModule.tif Jun 17, 2004, 10:21 PM 4KB Tr Cime ABMailNotification Pre?sModule.tif Jun 17, 2004, 10:21 PM 4 KB TFF C. ABPhoneFormatprefsdodule.tif Jun 17, 2004, 10:21 PM 4KB TFF C. me ABPortrait.tiff Jun 17, 2004, 10:21 PM 4 KB TFF C.me E ABTemplatepre?sModule.tif Jun 17, 2004, 10:21 PM 4KB FF C.me ABVCardPrefsModule.tif Jun 17, 2004, 10:21 PM 4KB FF.C..re E. AddressBook, icns Jun 17, 2004, 10:21 PM 52 KB Apple...Imag apachelpb.gif Jun 17, 2004, 4:11 PM 4 KB Graph...mag Pictures Back.tiff Jun 17, 2004, 10:21 PM a KB FF Drine backup.icns Jun 17, 2004, 10:21 PM 44 KB Apple...imag g This Week's images R. davis.jpg Yesterday, 2:12 PM 52 KB public...mag. .psd Mar 30, 2004, 3:59 PM 1.2MB Adob. Imag dictionarybrowser.psd Mar 30, 2004, 3:59PM 1.4 MB Adob. Imag 1406 .psd Apr 27, 2000, 4:51 PM 540 KB Adob....mag. DSCF0001.JPG May 27, 2000, 4:03 PM 92 KB public.jpg SCF0002.JPG May 27, 2000, 4:04 PM 92 KB public.jpg DSCF0003.JPG May 27, 2000, 4:04 PM 88 KB public.jpg ... DSCF0005.JPG May 27, 2000, 4:05 PM 88 KB public.jpg DSCF0007.JPG May 27, 2000, 4:05 PM 84 KB public.jpg DSCF0008.JPG May 27, 2000, 4:06 PM 88 KB public.jpg - DSCF0009.JPG May 27, 2000, 4:06 PM 92 KB public.jpg - DSCF000 10.JPG May 27, 2000, 4:06 PM 88 KB public.jpg DSCF00011.JPG May 27, 2000, 4:06 PM 96 KB public.jpg is osCF000 12.JPG May 27, 2000, 4:07 PM 96 KB public.jpg 1405 DSCF00013.JPG May 27, 2000, 4:07 PM 96 KB public.jpg ... OSCF00014.JPG May 27, 2000, 4:07 PM 92 KB public.jpg ... DSCF00015.JPG May 27, 2000, 4:08 PM 92KB public.jpg a DSCF00016.JPG May 27, 2000, 4:08 PM 96 KB public.jpg DSCF00017.JPG May 27, 2000, 4:08 PM 96 KB public.jpg DSCF00018.JPG May 27, 2000, 4:09 PM 96 KB public.jpg DSCF00019.JPG May 27, 2000, 4:09 PM 96 KB public.jpg DSCF00020.JPG May 27, 2000, 4:09 PM 96 KB public.jpg - DSCF00021.JPG May 27, 2000, 4:10 PM 96 KB public.jpg - DSCF00022.JPG May 27, 2000, 4:10 PM 96 KB public.jpg v DT 78 items, 147 GB available

1403 1401 FIG. 14A U.S. Patent Jun. 12, 2012 Sheet 27 Of 95 US 8,201,096 B2

This Week's images

X

testo work) tle 8 wis g OSCOOPG OSCO002PG DSCOOO3PG scratch 3. g DSC0005PG DSC007.JPG DSC0008.JPG OSCOOPG

SS 3. s OSC001OPG SCOOPG DSC0012PG iSES N S2 DSC00142G OSCOO15.JPG SCOO16G DSC007.JPG

sas 22e. s s S. DSC0018PG DSC0019.JPG DSC0020.JPG OSCOO2PG 2 DSC0022.JPG DSCOO23PG DSC0024.JPG DSCOO25.JPG

area Sè3. 23 23. aas DSC0028.jpg DSC0027 PG DSC0028.JPG OSCOO29EPG

s s 2 DSC003OFG OSCO31PG DSC0032G OSCOO33PG

NS Y Sé E. ar, DSC0034PG OSCO35PG OSCC036PG DSC0037.JPG

51 items, 47 GBavailable

1401 FIG. 14B U.S. Patent Jun. 12, 2012 Sheet 28 of 95 US 8,201,096 B2

This Week's images

Home averwhere Csave () OG) Show View Optionspion OG) Savedesearch Search criteria Ea as a - Oct 6, 1999, 11:02AM 9.5 MB Adob. Imag Oct 6, 1999, 11:20 AM 8.8 MB Adob,.. imag Oct 6, 1999, 11:24 AM 7.6 MB Adob,.. Imag Oct 6, 1999, 3:16 PM 8 MB Adob. imag Oct 6, 1999, 3:19 PM 7.3 MB Adob. imag ABGeneralPrefsModule.tif Jun 17, 2004, 10:21 PM 4 KB Tr C. re ABLandscape.tiff Jun 17, 2004, 10:21 PM 4 KB FF.C., file R ABLDAPPrefsModule.tif Jun 17, 2004, 10:21 PM 4KB C. re ABMaiiNotification PrefsModule.tif Jun 17, 2004, 10:21 PM 4 KB F. C. re ABPhoneFormatP refsmodule.tif Jun 17, 2004, 10:21 PM 4 KB TIFF c. ABPortrait.tiff Jun 17, 2004, 10:21 PM 4 KB TFF C. R ABTemplate PrefsModule.tif Jun 17, 2004, 10:21 PM 4 KB FF C. ABWCardprefsNodule.tif Jun 17, 2004, 10:21 PM 4KB FF C. AddressBook.icms Jun 17, 2004, 1021 PM 52 KB Rapache pb.gif Jun 17, 2004, 4:1 PM 4 KB RBack.tiff Jun 17, 2004, 10:21 PM 4 KB R backup,icns Jun 17, 2004, 10:21 PM 44KB R. davis.jpg Yesterday, 2:12 PM 52 KB dictionary.psd Mar 30, 2004, 3:59PM 1.2 MB dictionarybrowser.psd Mar 30, 2004, 3:59 PM 14 MB dock.psd Apr 27, 2000, 4:51 PM 540 KB DSCF0001.JPG May 27, 2000, 4:03 PM 92KB DSCF0002.JPG May 27, 2000, 4:04 PM 92 KB DSCF0003.JPG May 27, 2000, 4:04 PM 88 KB - DSCF0005.JPG May 27, 2000, 4:05 PM 88 KB DSCF0007.JPG May 27, 2000, 4:05 PM 84 KB DSCF0008.JPG May 27, 2000, 4:06 PM 88 KB DSCF0009.JPG May 27, 2000, 4:06 PM 92 KB DSCF0000.JPG May 27, 2000, 4:06 PM 88 KB DSCF00011.JPG May 27, 2000, 4:06 PM 96 KB DSCF00012.JPG May 27, 2000, 4:07 PM 96 KB 1405 DSCF00013.JPG May 27, 2000, 4:07 PM 96 KB DSCF00014.JPG May 27, 2000, 4:07 PM 92 KB DSCF00015.JPG May 27, 2000, 4:08 PM 92 KB DSCF00016.JPG May 27, 2000, 4:08 PM 96 KB DSCF00017.JPG May 27, 2000, 4:08 PM 96 KB DSCF00018.JPG May 27, 2000, 4:09 PM 96 KB DSCF00019.JPG May 27, 2000, 4:09 PM 96 KB DSCF00020.JPG May 27, 2000, 4:09 PM 96 KB DSCF00021.JPG May 27, 2000, 4:10 PM 96 KB ... DSCF00022.JPG May 27, 2000, 4:10 PM 96 KB 78 items, 147 GB available

1403 FIG. 14C U.S. Patent Jun. 12, 2012 Sheet 29 Of 95 US 8,201,096 B2

O Smart Folder

Con size: 128 x 128

Small Large Text size: 12pt C Label position: O) Bottom O Right

1430 OSnap to grid OShow item info OShow icon RM Keep arranged by Name

Background: OWhite O) Color O Picture

FIG. 14D U.S. Patent Jun. 12, 2012 Sheet 30 Of 95 US 8,201,096 B2

1502

Documents Shakelt.gadget.zip PDF DOCuments don'tshakeit out.pdf don'tshakeit.pdf

FIG. 15A U.S. Patent Jun. 12, 2012 Sheet 31 of 95 US 8,201,096 B2

1502 1505 X () Thu 10:47 AM o 1507

El Show All (392) 1509 Top Hit O actionbutton 1515 1511 DOCuments button.psd Action Button.dir Action Button.dir Metal buttons.psd animgiftxt animgiftxt bitmap.txt bitmap.txt bitmap.txt bitmap.txt d button. Wav 1503 d button. Wav CurSOr.txt CurSOr.txt default.txt default.txt 1513 actionbutton new buttons O action button icon alex - clear button FIG. 15B U.S. Patent Jun. 12, 2012 Sheet 32 Of 95 US 8,201,096 B2

§§§§ ######## [5] jöfffffffffff;

&s T?T?T?TIT?TS, as 4 as Safs;g Š. ii. (6) (6 (6) (b. (5) (6 (6) (6.

[5] U.S. Patent Jun. 12, 2012 Sheet 33 Of 95 US 8,201,096 B2

Aso fisi (s

[5] [5] [5] [5] [5] U.S. Patent Jun. 12, 2012 Sheet 34. Of 95 US 8,201,096 B2

to button.psd September 193:55 pr; G) toprole sy:y Action Sutton, dir June 23 late Action Button.dir fire 23 People

Action Buttondr June 2003 GD 1605 Metal buttons.psid March 23 GD Fiat is anim.giftxt ecenter 202 GD Within Group Sort by: 19 cre. Nare

V Folders late 1 61 3 actionbutton January 5 4:45pm GD 2. 1 607

new buttons April 2003 GD action butto inco February 2003 GD Search: alex - clear button December 2002 GD Everywhere

torne Help Button Nowere 22 GD testo work 1609 9 fore. scratch v HTML t 1615 a 16Buttons.html November 399 GD 1627 SBitors. Niverer 399 Storist Mighverber 999 16tters5. werber 1999 SEts8. Nwere 1999 GD 1830 v images E 1617 buttonsymbols.gif Nowere 202 s 1 83 1 button in place.gif Noverner 202 buttonbackground.gif Novernoer 2002 buttoniayer.gif Nowerber 2002

PushButton G11 (graphite).pct Noverner 2002 107 more. 1619 V PDF Documents Norton Disk EDitor x.pdf September 2002 GD Reader.pdf May 2002 GD MacUsersGuideoPocketPCs.pdf November 201 GD Designing AirPort Networks2.pdf October 20 GD 1621

SunSphereastButton.java May 2001 E. SurSphereinfouilon.java May 2001 ESunSphereNorthButton.java May 2001

V Word Documents Noyeer 202

January 2002 August 2000 are 2.0

2 FIG. 16A 1625A U.S. Patent Jun. 12, 2012 Sheet 35 of 95 US 8,201,096 B2

1625 1611 1601 1603 Search: Littor 392 Results Docurrents Group sy: V Folders ind actionbutton january 5 4:45 pm Date 1605 new buttons Apri 2003 Peepie action button inco February 2003 Fiat is aliex clear Liter ecarrier 2002 Within Group Sort by: Help Button Marrier 22 Naire 9 rore... late 1607 Kir v HTML People 1615 SButtons, Noernber 1939 a 16Buttons9.html November 1999 Search: 6ttonsai.htri Noyerer 99 Everywhere one Noweef 1993 8stofss. testo work) 1609 SEtteiss. Nyerer 999 scratch c V Images E 1617 1627

buttonbackground.gif November 2002 g FPushButton G11 (graphite).pct Neverer 22 i vV PDF Documents 1619 EAHIG5/10/pdfAlig.5/10/pdf NowsetterNovember 2002 GD a Norton Disk EDitor X.pdfxpdf September 2002 GD a Reader.pdf May 2002 G a MacUsersGuideoPocketPCs.pdf November 2001 G Designing AirPort Networks2.pdf October 2001 GD 18 more V source code 1621 E. GraphicsButton.jave May 2001 SunSphereButton.java May 2001 SunSphereastButton.java May 2001 ESunSphereinfoButton.java May 2001 SunSphere.NorthButton.java May 2001

V Word Documents 1623 E fop Libugs 082301.doc Neverber 22 Racia S.O. Jser fana January 2002 E Radar 4.0 User manual August 2000 ECD Player bits tre 20 ERequired changes to the AAP (8) June 2

1625A FIG. 16B U.S. Patent US 8,201,096 B2

ZOOZuæquæ03C]

upuolinguonov[] 0 U.S. Patent Jun. 12, 2012 Sheet 37 Of 95 US 8,201,096 B2

1825

C. C. Search: button 1814 392 Results 1 81 1 button.psd September 19 3:55 pin Stop By: Actier stoid e2O3 Date Action Button dir tire 23 People Metal buttons.psid arch 2,3 Fiat list animgf.txt ecemer 2002 Within Group Sort by: rior. Nate 1813 t rew buttons April 2003 Oce action button ince February 2003 Search: alex - clear butter Decerrer 202 Everywhere Help Button November 2002 wrk) 9 rore. scratch V HTML h 1 81 5 16Buttons8.html November 998 W g SBist Newerber 999 a 16Buttons4.html Neverer 1999 16Buttons5.html Noerrier 1999 a 16Buttons6.html November 1999 O more. 1 81 7 V images

button...ols.gif button...ace.gif button...nd.gif buttonlayer.gif PushBu...te).pct November 202 Neverer 2002 November 2002 November 2002 November 2002 1 81 2 17 tre, 1 81 9 V PDF Documents A-IG.5/10.pdf Nowether 202 GD Norton Disk EDitor X.pdf September 2002 GD

MacUsersGuide toPocketPCs.pdf Neverer 20 GD Designing AirPort Networks2.pdf October 2 GD 8 ce. 1 821 V Source code E. GraphicsButton.jave GD E sunsphereButton.java May 2001 SunSphereastButton.java May 2001 E SunSphereinfoButton.java May 2001 E. SunSpherenorthButton.java May 2001 cre, 1823 E. Radars... serraria January 2002 GD Radar 4.0 User manual August 2000 GO E CD Player bits ife 2 frequired Changes to the AAP (8 ise 2. FIG. 18A U.S. Patent Jun. 12, 2012 Sheet 38 of 95 US 8,201,09 6 B2

1825

O O. Search: button

1814 s 392 Resuits CR button (3 Group By: 1811 buttor.psd September 193:55 pm Kic actic suit.ir June 2003 Date Acties sittos.di ture 23 People Metal buttons.psid Alarch 23 Fiat is aringif.txt Deceiber 2002 Within Group Sort by: fire. Nate v Folders late 1813 Kiri action button January 5 4:45 pm People new buttons April 2003 action button inco February 2003 Search: aiex - clear button Decerre 22 Everywhere ice J Help Button Novernte 2002 testo (work scatch

Neveier 399 November 1999 Nowerber 1999 Nowere 999 Nowere 999

button...ois.gif button...ace.gif button...nd.gif buttonlayer.gif PushBu...te).pct Nowenaer 2002 November 202 Ngwere 22 Nowerber 2002 November 2002

O

PushBu...pct 10 01...Button 1202...button 1203...button 1204...button November 2002 November 2002 Noveter 2002 Neverber 202 Nowerber 2002

1818A

1205...button 1218...button 12 19...button 1222...button 135 E...button November 2002 Nowe ref O2 Noweef 22 November 2002 Moverner2002

1365.uttons 1400...Button 15 OO. uttons 161 B.editor 162 BRaised November 2002 Noveities 22 Novere 22 Nivene 2: Noveiter 2002

FIG. 18B U.S. Patent Jun. 12, 2012 Sheet 39 Of 95 US 8,201,096 B2

Search: imran

V Imran Chaudhri 1911 CD Player bits tire 2 GD 1905 Required Changes to the AAP B) tire 2. GO icor ext May 2000 GD 1913 V No Authors Fiat is 1906 imran Thursday 4:39 pm d within Group sort by: MAP-irrangirnaiappie.com hursday 3:50pm GD Name

Drafts (irrangappie.com), ribox ref 2:08 an GD late 1 907 trar June 15 2:16 pm GD Sent (imranGrapple.com).mbox June 10: 5:23 pm GD eople 17 fore Search: V Sarah A. Brody Everywhere hione plagenda_082201.doc Noveter 2002 GD testo (work) pl. minutes 081701.doc November 2002 GD scratch pimeline 082201.doc Neverther 202 GD hd minutes O73101.doc Newerber 2002 GD

1917 WC2000 as GO

FIG. 19A 1925A U.S. Patent Jun. 12, 2012 Sheet 40 Of 95 US 8,201,096 B2

Search: imran

CD Player bits life: GD

Required Changes to the AAP (8) June 2000 GD cf. Exit

Keywords - - Copyright - - Fiat is last used date May 2000 Within Group sort by: Marie Date King People

agenda (82201.doc November 2002 GD search: Fp minutes 081701.doc Nwere 2002 GD Erhare 1908 Fp timeline 082201.doc November 2002 () testo work) f minutes O73101.doc Nowerber 2002 GD scratch WWDC 2000 - Hi Labs May 2000 GD hd imran May 2003 GD window pix for imran July 2 4:36 pm GD window pix for in ran July 7 11:42 an GD

imran Thursday 439 pm GD

Heip for imran January 2000 GD v imran.psd January 2001 Bas & irriran O Keywords - - Copyright - - last used date July 2000

Height 1536 With 248

quake.inrara).psd Decerer 1999

RID-IMRAN October 20 140 - Overview 5/14-imran May 2000 r 140 - Agua Overview 54 -im far May 2000 140 - Aqua Overview 5/14-imran May 2000 40 - Aqua Overview 5.4 liraran May 2000 140 - Aqua overview 5 14-imran may 2000 imran September 51.2 pm MAP-imranig-mail.apple.corn Thursday 3:50 pm Junk (imrangapple.com.mbox May 245:29 pm s Sent Messages inrarappie.com.mboc: June 105:23 pm Drafts (imrangapple.com).mbox the 72:8 at Southwest Airlines Ticketless Travel Confirmation April 29 11:45pm S May 24 121 per

List May 24 121 pm imran June 152:16 pr.

FIG. 19B U.S. Patent Jun. 12, 2012 Sheet 41. Of 95 US 8,201,096 B2

1925 1930 1903

O O. Search; te

January 4 3:34 pm GD January 43:27 pm GD January 4 3:27 pm G)

Today 11:12am E 50FF13D5-C608-11D8-9DF7-0003938E381CIABPerson.wcf Keywords - - Copyright - - 1 945 Last used date Today 11:12am 1 944 test work) M scratch

Apple Computer Inc. Todaytoday 11:12am11.12am GD imran Today 11:12am

O info.plist Today 1.12am 44 ore... V Yesterday 1936 don'tshakeit out.pdf Yesterday 10:36 pm EEAPoutkipsd YesterdayGET 9:48 pm Weather gadget Yesterday 9:16 pm GD iChat Yesterday 8:31 pm GD 1938 Vlast week

Applications Tuesday 6:44 pm GD

1940 Vasimonth-H

FIG. 19C U.S. Patent Jun. 12, 2012 Sheet 42 of 95 US 8,201,096 B2

1925 1950 1903 C CO Search: te

Picture 23 today 11:12am Kir

info.ptist Today 11:12am late 905 asdasd inran Today 1: 12am People

SignaturesBy Accountpist Today 11:12am at list MessageSofting pist.backup today 12am GD Within Group Sorthy: 34 more... Nate late V Yesterday Kir 1954 tes outk.psd Yesterday 9:48 pm People Weather gadget.zip Yesterday 9:16 pm Search: Weather gadget Yesterday 9:16 pm Everywhere iTunes Music Library.xml Yesterday 8:24 pm Horne it iTunes 4 Music Library Yesterday 8:24 pm testo work) fore. scratic V Last week 1956 15 Wewe GotGote.4a Thema ite 81:05 at SEE14 The Brouhaha.m4a June 16:05am 13 Crawlspace.m4aCrawlspacema line is 1:05 an 12 An Openopen Letterletter Toto NYC.m4a une 6:04 art 11 Shazam!.m4a June 1814 an 10 fore... V Later 1958 index.htm: August 75:36 pm web share.gif August 75.36 pm

apache pt.gif May 2001 macosxlogo.gif May 2001

FIG. 19D U.S. Patent Jun. 12, 2012 Sheet 43 of 95 US 8,201,096 B2

ºdnoºu?tnimuosLOETS?D?T? U.S. Patent Jun. 12, 2012 Sheet 44 of 95 US 8,201,096 B2

Display a system wide Menu for inputting 2001 Search queries

Input Search, and as Search query is inputted, perform the search (begin searching and displaying search results before user finishes 2003 inputting Search query)

Perform search through files, metadata, emails within an email program, address book 2005 entries Within an address book, entries within a calendar program, etc.

Display an abbreviated (incomplete) list of hits (if there are more than a certain number of hits), SOrted by relevance and Segregated 2007 into groupS (Categories)

Receive Command to display all hits - 2009

Display a search results window with ability to display 2 different types of view (e.g., icon 2011 and list views) within the same closeable, resizable, moveable window FIG. 20 U.S. Patent Jun. 12, 2012 Sheet 45 of 95 US 8,201,096 B2

29 O1 RECEIVE USERS SEARCH INPUT

PRESENT (E.G., DISPLAY) SEARCH RESULTS BASED 2903 ONUSERS SEARCH INPUT AND WITH IN THE RESULTS, PRESENT RESIZABLE OR ZOOMABLE OR PAGABLE CONTENT

RECEIVE USER SELECTION TO RESIZE OR 2905 ZOOM OR PAGE THROUGH THE CONTENT

DISPLAY (OR OTHERWISE PRESENT) RESIZED OR ZOOMED OR PAGED THROUGH 2907 CONTENT WITHOUT LAUNCHING APPLICATION WHICH CREATED THE CONTENT

FIG. 21 U.S. Patent Jun. 12, 2012 Sheet 46 of 95 US 8,201,096 B2

3001

(3) About Xcode Tools.pdf Compatibility with Mac OS X Versions The Xcode 3.0 developer tools preview for Mac OS X 10.5 (Leopard) will run exclusively on the pre release version of Mac OS X 10.5, with which it is distributed. The Xcode 3.0 preview supports development for Mac OS X 10.3 (Panther), Universal applications for Mac OS X 10.4 (Tiger) and Mac OS X 10.5 Preview (Leopard) using the Mac OS X SDK support. Please read the "Software License Agreement for Xcode Tools Seed" document for special terms applying to this preview release. What's New Xcode 3.0 developer tools preview for Mac OS X 10.5 (Leopard) introduces many new features to improve developer productivity, provides greater debugging control, and leverages Leopard's new system-wide 64-bit capabilities. Developer Productivity * Full support for Objective-C 2.0: New features include garbage collection, properties, and for each RSS Syntax to the Objective-C programming language. Note that Objective-C 2.0 is only compatible with the Mac OS X 10.5 (Leopard) preview. New editor: New features include improved performance when loading large files, and streamline of the edits/buildldebug process. For instance, errors and warnings can now be viewed inline within your Code in message "bubbles", allowing you to 5. view errors and warnings beside the code you need to Correct. Additionally, a new feature called Code Focus combines new code folding features along with visual cues to highlight the block structure of your Code. 30O2 Refactoring for Objective-C. Refactoring facilitates wide-scale restructuring of your project and class Structure. " Project Snapshots; Another new feature, enables you to save your complete E. state, or restore your project to a previous state. Together with refactoring, these provide a safe mechanism to make

dramatic changes to your entire project. * New Interface Builder. This newlyersion has an improved user interface, a new extensible library model and supports new Mac OS X 10.5 features such as Layerkit. For this preview, the previous version of IB is included as "Interface Builder Tiger.app". Debugging Control Xray: Uses a Collection of instruments to monitor and visualize your application and the OS over time, which are presented as a series of tracks similar to Garage Band. Xray also allows you to create your E" dtrace instruments. This enables you to quickly spot problems and relationships within your application * Rewind. Allows you to step backward in 3. Code while ESSES With Rewind, there is no need to 3003 restart your entire debugging session simply because you stepp e too far. 3004 Debugging Tooltip. Any text editor window can now display variable data as a tooltip while debugging

3005 FIG.22A U.S. Patent Jun. 12, 2012 Sheet 47 of 95 US 8,201,096 B2

3010

(x) About Xcode Tools.pdf Universal applications for american PowerPC articles Please see the complete Xcode 3.0 preview release for more detailed information on the above features, and the many other new features in this preview release. Dashcode for Clashboard Widget Development * Dashcode: A new application that includes everything you need to create Dashboard widgets lithin a single environment. Dashcode provides a number of templates, starting you off with a valuable, fully-functioning widget. From there you can easily change the look and controls within your widget using the design canvas. For mroe advanced developers you can write new Code with the programmer's editor with Code Sense, and even debug your widget with the JavaScript debugger. Dashcode will take care of the packaging and deployment of your widget to Dashboard once you've perfected it. The Dashcode Installer package is including at the top level of the Xcode Tools folder in the Leopard preview installation DVD, and will instal into the DeveloperiApplications folder. Installing Xcode Developer Tools The Xcodeools, mpkg contains several separate sub-packages in the Custom install panel that allow flexibility in installing and updating: Developer Tools Software (DeveloperTools.pkg) - This package contains Xcode, gob, and the other tools necessary for Mac OS . • GCC 4.0.1 (goc4.0.pkg) - This is the latest version of goc. This compiler is necessary for 64-bit development, using the OWARF debugging format, and development of Universal application. • GCC 3.3 (gcc3.3.pkg) - This package contains the gec 3.3 compiler. Software Development Kits - These packages provide the header files and link libraries necessary for Mac OS X development. All development requires the Mac OS X SDK. This package provides the API for basic Mac OS X technologies such as Carbon, Cocoa, and command-line tools. The other SDK packages provide API for extended technologies, such as QuickTime, OpenGL, etc. is 5.0 Developer Tools - These packages are used when developing J2SE5.0 applications for acintosh. Developer Documentation (Devdocumentation.pkg) - This package contains the core Max OS X developer documentation for technologies such as Carbon, Cocoa, QuickTime, WebKit, etc. Developer Examples (DevExamples.pkg) - This package contains example code for a wide variety of Mac OS X technologies. Note that these examples may require any of the above Software Development Kits to be installed before they will Compile. ... WebObjects (WebObjects.mpkg) - These package are used for WebOjects 5.4 development.

Cross Development (MacOSX 10.3.9.pkg. MacOSX10.4u.pkg. MacOSX10.5.pkg) - These packages are for Mac OS X v10.4 (Panther). Mac OS X v10.4 (Tiger), and Mac OS X 10.5 (Leopard) to allow cross development. These can be installed together or independently, and can be installed Separately. CHUD Tools (CHUD. pkg) - This package contains the Computer Hardware Understanding Developer Tools. For more information see http://developer.apple.com/tools performancel. installation instructions for Xcode 3.0

3015. 3017 Y3019 302 FIG. 22B U.S. Patent Jun. 12, 2012 Sheet 48 of 95 US 8,201,096 B2

3030

4 items

Al-kasik Xcode (is cessipatithifity arts its stasiation &nda (%ik A. Reica & c. Mac OSS version it.3 (sthet:

consteatsCarpatibiar withs last 3s sesses Search Kit Reference screatstar of suit: 3-Yrigs ising Xcode facts with hersects Exeteopar is:s: in the Xcode eats

t? -tasco test grises but ash tse sc33.3 :Knse wiver Max &S x wie Stags: 88s

GraphKit Reference Guide QuickTime Kit Programming Guide

Architecture and Performance Group Apple Computer, Inc. May 2004

FIG. 22C U.S. Patent Jun. 12, 2012 Sheet 49 of 95 US 8,201,096 B2

DISPLAY WINDOW CONTAINING USER INTERFACE (UI) OF FILE SYSTEM (EG, HFS) ORU OF SEARCH 2301 SOFTWARE (WINDOWMAYBE MOVABLE, RESIZABLE CLOSEABLE AND OVERLAPPING WITH OTHER WINDOWS)

DISPLAY USER SELECTABLE VIEW OPTIONS FOR 2303 DISPLAYING FILES AND FOLDERS (E.G. CON VIEW, LIST VIEW COLUMN VIEW, OR COVER FLOW VIEW)

DISPLAY SCROLLINGU. OBJECTS (EG. ARROWS OR -3' SCROLLBAR) TO ALLOWSCROLLING OF ONE OR MOREVIEWS (CONCURRENTLY OR SEPARATELY)

RECEIVE USER COMMANDS FROM COVER FLOW 2307 VIEW (OR OTHER VIEWS) TO PERFORMFILE SYSTEM OPERATIONS (E.G. CREATE NEW FOLDER IN HFS) AND PERFORM THESE OPERATIONS

DISPLAYUSER SELECTABLE OPTIONS TOSORT -5' LIST AND/OR FLOW WEWS

FIG. 23 U.S. Patent Jun. 12, 2012 Sheet 50 Of 95 US 8,201,096 B2

quauunoop U.S. Patent Jun. 12, 2012 Sheet 51 of 95 US 8,201,096 B2

6|92 U.S. Patent Jun. 12, 2012 Sheet 52 Of 95 US 8,201,096 B2

èHARES×”ajow@ U.S. Patent Jun. 12, 2012 Sheet 53 of 95 US 8,201,096 B2

9992 87||92

siueunood[]

tj.1966]][…]ue, sø?n?ola?? U.S. Patent Jun. 12, 2012 Sheet 54 of 95 US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 55 of 95 US 8,201,096 B2

|98Z U.S. Patent Jun. 12, 2012 Sheet 56 of 95 US 8,201,096 B2

|19Z 9|92

6|92

Ä0109LÄHEIA?I

8187 U.S. Patent Jun. 12, 2012 Sheet 57 Of 95 US 8,201,096 B2

1797

/ZE??ernaestoZ//

ZLL?áZ,\ 1966.It-Tue??IT salaowQ

U.S. Patent Jun. 12, 2012 Sheet 58 Of 95 US 8,201,096 B2

9|92

926|

@@) £19.2

©?KAQ@§47]f'(][2]E\,\&

sell||ling U.S. Patent Jun. 12, 2012 Sheet 59 of 95 US 8,201,096 B2

2501 DISPLAY WINDOW HAVING USER INTERFACE OF COVER FLOW VIEW (AND OPTIONALLY ALSO DISPLAY LIST VIEW WITHIN SAME WINDOW)

RECEIVE USER'S COMMAND (E.G. THROUGH USER'S 2503 SELECTION OF AUSER INTERFACE OBJECT) TO ZOOM OR PAGE THROUGH OR SCROLL THROUGH DOCUMENT SHOWN IN COVER FLOW WIEW

RESPOND TO USER'S COMMAND TO ZOOM OR 2505 PAGE THROUGH OR SCROLL THROUGH A DOCUMENT IN THE COVER FOW VIEW

FG. 26A

U.S. Patent Jun. 12, 2012 Sheet 61 Of 95 US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 62 of 95 US 8,201,096 B2

Z||92 C19Z"SDI U.S. Patent Jun. 12, 2012 Sheet 63 of 95 US 8,201,096 B2

K COOD

n cd CN 2 w ce r

U.S. Patent Jun. 12, 2012 Sheet 64 of 95 US 8,201,096 B2

87|

9992H9z" SOI U.S. Patent Jun. 12, 2012 Sheet 65 of 95 US 8,201,096 B2

COOD | N

XO28ALCIH?0 @@@@@GES ?uauoduJOO

uunood[]squa U.S. Patent Jun. 12, 2012 Sheet 66 of 95 US 8,201,096 B2

||97|

9|92

?I@]©O{@@§ff.)QI[55]?)

inQ U.S. Patent US 8,201,096 B2

skepZ[3]ofie

U.S. Patent Jun. 12, 2012 Sheet 69 of 95 US 8,201,096 B2

6|92

@3.O?i§¶¡Q][E]©{@

“8Iow@ SHOWTd4, U.S. Patent Jun. 12, 2012 Sheet 70 of 95 US 8,201,096 B2

8|92

?Q@§@]©ETE\!\

U.S. Patent Jun. 12, 2012 Sheet 71 Of 95 US 8,201,096 B2

Cl/z [×]GOD SOIH

salmolds

U.S. Patent Jun. 12, 2012 Sheet 73 Of 95 US 8,201,096 B2

87|92 |092 8|92

9|92 U.S. Patent Jun. 12, 2012 Sheet 74 of 95 US 8,201,096 B2

HOBHOHWES? Kepol@]

U.S. Patent Jun. 12, 2012 Sheet 75 Of 95 US 8,201,096 B2

|92 |09Z 8|97 9|92 1,92

HZZ SOIH

U.S. Patent Jun. 12, 2012 Sheet 77 of 95 US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 78 of 95 US 8,201,096 B2

XML? U.S. Patent Jun. 12, 2012 Sheet 79 of 95 US 8,201,096 B2

s

a gagg

a s is s six A - sa as a & k ...... a a s : E S E E : a K at ...... s a a a ...... * U.S. Patent Jun. 12, 2012 Sheet 80 Of 95 US 8,201,096 B2

87|82 |09Z 8|9Z

9|92

9|87 U.S. Patent Jun. 12, 2012 Sheet 81. Of 95 US 8,201,096 B2

U.S. Patent Jun. 12, 2012 Sheet 82 of 95 US 8,201,096 B2

1 DISPLAYAN CON OF A FOLDER THE CON BEING 2651 AT LEAST PARTIALLY TRANSPARENT

2653 DISPLAYA SET OF CONS AT LEAST PARTIALLY WITHIN THE CON OF THE FOLDER

DISPLAY AN ANIMATION OF THE SET OF CONS TO DISPLAY AT LEAST A SUBSET OF THE CONS 2655 OVER A PERIOD OF TIME (E.G. SHUFFLE THE SET OF CONSWTHIN THE FOLDER TO DISPLAY ALL ICONS IN THE SET AFTER A PERIOD OF TIME)

FIG. 28A U.S. Patent Jun. 12, 2012 Sheet 83 of 95 US 8,201,096 B2

DISPLAYAN CON OF A FOLDER WHICH 2671 INCLUDES FILES IN A HERARCHICAL FILE SYSTEM

DISPLAY AUTOMATICALLY (WITHOUT USER 2673 INTERACTION WITH THE CON OF THE FOLDER) AN ANMATON PRESENTING A CONTENT OR REPRESENTATION OF EACH OF THE FILES

FIG. 28B

U.S. Patent Jun. 12, 2012 Sheet 85 of 95 US 8,201,096 B2

9|82

{@@&FŒŒÙðKAQ§? C18Z"SDI 4)

Y se U.S. Patent US 8,201,096 B2

87||97

9|92

U.S. Patent US 8,201,096 B2

H8Z"SDI U.S. Patent Jun. 12, 2012 Sheet 88 Of 95 US 8,201,096 B2

901Z

\sidae U.S. Patent Jun. 12, 2012 Sheet 89 of 95 US 8,201,096 B2

a FIG. 29A

FIG. 29E

FIG. 29C U.S. Patent Jun. 12, 2012 Sheet 90 Of 95 US 8,201,096 B2

www.GMATCH[]1991)[]

U.S. Patent Jun. 12, 2012 Sheet 91 of 95 US 8,201,096 B2

Q@§?][2]©®)%\ £109,"SDI

1891IU U.S. Patent Jun. 12, 2012 Sheet 92 of 95 US 8,201,096 B2

or v. v.v. asses

- a "

Q@??ºl3?][2][57]©KA

war, U.S. Patent Jun. 12, 2012 Sheet 93 of 95 US 8,201,096 B2

e

U.S. Patent Jun. 12, 2012 Sheet 95 of 95 US 8,201,096 B2

1. 3001

3007 3003 3005 3012

ONE OR MORE SET OF

APPLICATIONS PREVIEW SOFTWARE

(E.G. , GENERATOR ROUTINES

SPOTLIGHT (DAEMON) (E.G. ETC) PLUG-INS)

FILE SYSTEM FILESIDATA SOFTWARE

3010 3009

FIG. 31 US 8,201,096 B2 1. 2 BROWSNG OR SEARCHING USER search query, the first display area configured to display con INTERFACES AND OTHER ASPECTS tent of the documents which can have a plurality of different types of content including at least one of text-based content BACKGROUND OF THE INVENTION and a folder, and the method further includes displaying a second display area for selecting a selected document to be Modern data processing systems, such as general purpose displayed in the first display area. In at least certain embodi computer systems, allow the users of such systems to create a ments, the content of the selected document can be Zoomed variety of different types of data files. For example, a typical from the first display area to display an enlarged view or may user of a data processing system may create text files with a be presented as multiple pages which are selectable Such that word processing program Such as Microsoft Word or may 10 the document can be viewed one page at a time or several create an image file with an image processing program Such pages at a time. In certain embodiments, the first display area as Adobe's PhotoShop. Numerous other types of files are and the second display area are adjacent each other in the capable of being created or modified, edited, and otherwise same moveable, closeable, resizeable and minimizeable win used by one or more users for a typical data processing sys dow, which includes user interface objects to receive user tem. The large number of the different types of files that can 15 inputs to move the window, close the window, resize the be created or modified can present a challenge to a typical window, maximize the window and minimize the window. user who is seeking to find a particular file which has been Further, the window may include various user interface created. objects which allow the user to pick between different views, Modern data processing systems often include a file man including a view which includes the first display area and the agement system which allows a user to place files in various second display area. In at least certain embodiments, the first directories or Subdirectories (e.g. folders) and allows a user to display area may be referred to as a “cover flow view area give the file a name. Further, these file management systems and the second display area may be referred to as a “list often allow a user to find a file by searching for the file's name, display' view area. In at least certain embodiments, the or the date of creation, or the date of modification, or the type method may include the performance of a search through of file. An example of Such a file management system is the 25 metadata for various different kinds of documents, as well as Finder program which operates on Macintosh computers an index database which includes a full-text inverted database from Apple Inc. of Cupertino, Calif. Another example of a file containing the full text of the content of the documents within management system program is the Windows Explorer pro a data processing system. In certain embodiments, the docu gram which operates on the Windows from ments may be organized and stored in a hierarchical file Microsoft Corporation of Redmond, Wash. Both the Finder 30 system, and a user interface program, Such as the Finder from program and the Windows Explorer program include a find Apple Inc. of Cupertino, Calif., or Windows Explorer from command which allows a user to search for files by various Microsoft of Redmond, Wash., may be provided to allow the criteria including a file name or a date of creation or a date of user to manage the location, etc. of the documents and files modification or the type of file. However, this search capabil maintained by the hierarchical filing system (HFS). ity searches through information which is the same for each 35 In at least certain embodiments, a user may select a docu file, regardless of the type of file. Thus, for example, the ment in the list view and this causes the documents in the searchable data for a Microsoft Word file is the same as the cover flow view to scroll in order to show the same document searchable data for an Adobe PhotoShop file, and this data in the selected document position of a cover flow view. In typically includes the file name, the type of file, the date of certain embodiments, scrolling of the documents in the cover creation, the date of last modification, the size of the file and 40 flow view is linked to the scrolling of documents in the list certain other parameters which may be maintained for the file display view area such that Scrolling in one area is matched by by the file management system. and coordinated with scrolling in the other area. In at least Certain presently existing application programs allow a certain embodiments, the list display view area displays at user to maintain data about a particular file. This data about a least a portion of the documents matching the search query, particular file may be considered metadata because it is data 45 and the order of the documents in the list view area matches about other data. This metadata for a particular file may the order of the documents shown in the cover flow view area include information about the author of a file, a summary of which also shows documents matching the search query. The the document, and various other types of information. A cover flow view area typically includes one set of documents program Such as Microsoft Word may automatically create on one side of the selected document and another set of Some of this data when a user creates a file and the user may 50 documents on the other side of the selected document; the add additional data or edit the data by selecting the “property spacing between the documents is not uniform. In at least sheet from a menu selection in Microsoft Word. The prop certain embodiments, documents on the left side of the erty sheets in Microsoft Word allow a user to create metadata selected document appear to be facing towards the right and for a particular file or document. However, in existing sys therefore towards the selected document, while documents on tems, a user is notable to search for metadata across a variety 55 the right side of the selected document appear to be facing of different applications using one search request from the towards the left side and therefore towards the selected docu user. Furthermore, existing systems can perform one search ment. The apparent direction each set of documents faces for data files, but this search does not also include searching may be achieved by providing a perspective view in which through metadata for those files. one side of the document is longer than another side of the 60 document even though, in fact, the underlying document has SUMMARY OF THE DESCRIPTION sides that are equal and parallel. According to another aspect of the inventions described Methods for managing data in a data processing system herein, a method of managing data in one exemplary embodi and systems for managing data are described herein. ment includes displaying a first display area for displaying A method of managing data in one exemplary embodiment 65 two-dimensional representations of at least a portion of files includes displaying a first display area for displaying two and folders in a hierarchical file system, the first display area dimensional (2-D) representations of documents matching a configured to display content of the files in the first display