US010520782B2
United States Patent ( 10 ) Patent No.: US 10,520,782 B2 Busch (45 ) Date of Patent : Dec. 31, 2019
(54 ) DISPLAY DEVICES, SYSTEMS AND USPC 359/259 METHODS CAPABLE OF SINGLE -SIDED , See application file for complete search history. DUAL - SIDED , AND TRANSPARENT MIXED REALITY APPLICATIONS (56 ) References Cited (71 ) Applicant: James David Busch , Tempe, AZ (US ) U.S. PATENT DOCUMENTS 3,037,138 A 5/1962 Motson (72 ) Inventor: James David Busch , Tempe , AZ (US ) 5,103,328 A 4/1992 Numao 5,357,355 A 10/1994 Arai ( * ) Notice : Subject to any disclaimer, the term of this patent is extended or adjusted under 35 (Continued ) U.S.C. 154 ( b ) by 0 days . FOREIGN PATENT DOCUMENTS (21 ) Appl. No.: 15 / 887,882 CN 105098087 A 11/2015 DE 102014111484 Al 2/2016 ( 22 ) Filed : Feb. 2 , 2018 (Continued ) (65 ) Prior Publication Data OTHER PUBLICATIONS US 2018/0217429 A1 Aug. 2 , 2018 “ See -through display, ” Wikipedia ( retrieved Jan. 14 , 2017) . Related U.S. Application Data ( Continued ) (60 ) Provisional application No. 62/ 454,020 , filed on Feb. Primary Examiner — Mohammed A Hasan 2 , 2017 . (74 ) Attorney, Agent, or Firm James David Busch ( 51 ) Int . Ci. (57 ) ABSTRACT GO2F 1/139 ( 2006.01 ) The display includes a transparent display ( TD ) element or GO2F 1/07 (2006.01 ) layer such as at least one Transparent Organic Light Emit GO2B 27/00 ( 2006.01) ting Diode ( TOLED ) element and at least one active shutter HOIL 27/32 (2006.01 ) (AS ) element or layer such as a liquid crystal utter. The AS G06T 19/00 ( 2011.01) may be opaque or transparent. The TD may be transparent or display or emit light in mixtures of color. Black may be (52 ) U.S. CI. displayed by a TD pixel by having the TD be transparent and CPC G02F 1/1395 (2013.01 ) ; G02B 27/00 turning an AS opaque . Each display is composed of elements (2013.01 ) ; G02F 1/076 (2013.01 ); G02F that can have many states . The states can be used to create 2201/44 ( 2013.01) ; GO6T 19/006 (2013.01 ) ; different display modes . Devices with these displays can HOIL 27/323 ( 2013.01 ) ; HOIL 27/3232 have many form factors and configurations. These devices ( 2013.01) ; HOIL 27/3267 (2013.01 ) ; HOIL may change display modes based on spatial context as 2251/5323 (2013.01 ) determined by any one of various environmental and con ( 58 ) Field of Classification Search figuration sensors . CPC GO2F 2201/52 , GO2F 1/133514 ; GO2F 1/19 ; GO2F 2001/212 ; B82Y 20/00 19 Claims, 28 Drawing Sheets
-102
110 ORCHESTRATOR 106
108 US 10,520,782 B2 Page 2
( 56 ) References Cited 2006/0238684 Al 10/2006 Kiya 2006/0268208 A1 11/2006 Murakami U.S. PATENT DOCUMENTS 2007/0002130 A1 1/2007 Hartkop 2007/0080629 A1 4/2007 Ko 5,757,139 A 5/1998 Forrest 2007/0114916 Al 5/2007 Chang 5,825,543 A 10/1998 Ouderkirk 2007/0120478 A1 5/2007 Lee 5,881,377 A 3/1999 Giel 2007/0138941 Al 6/2007 Jin 5,986,401 A 11/1999 Thompson 2007/0159070 Al 7/2007 Hu 6,088,079 A 7/2000 Kameyama 2007/0201248 Al 8/2007 Jung 6,288,840 B1 9/2001 Perkins 2007/0274077 Al 11/2007 MacKinnon 6,337,492 B1 1/2002 Jones 2008/0013174 A1 1/2008 Allen 6,361,838 B1 3/2002 Miyatake 2008/0018732 Al 1/2008 Moller 6,433,853 B1 8/2002 Kameyama 2008/0062259 Al 3/2008 Lipton 6,537,688 B2 3/2003 Silvernail 2008/0231690 A1 9/2008 Woodgate 6,657,690 B2 12/2003 Hashimoto 2008/0248815 Al 10/2008 Busch 6,674,504 B1 1/2004 Li 2008/0266500 A1 10/2008 Nimura 6,689,492 B1 2/2004 Yamazaki 2008/0297695 Al 12/2008 Sekiguchi 6,831,298 B2 12/2004 Park 2008/0303996 Al 12/2008 Morisawa 6,961,178 B2 11/2005 Sugino 2009/0091826 A1 4/2009 Sugino 7,030,552 B2 4/2006 Chao 2009/0176029 Al 7/2009 Daniels 7,034,452 B2 4/2006 Kim 2009/0201446 A1 8/2009 Zhuang 7,106,001 B2 9/2006 Kim 2009/0201447 Al 8/2009 Zhuang 7,164,151 B2 1/2007 Yamazaki 2009/0219253 Al 9/2009 Izadi 7,450,105 B2 11/2008 Nakamura 2009/0273275 Al 11/2009 Tyldesley 7,514,861 B2 4/2009 Chang 2009/0278452 Al 11/2009 Kim 7,538,841 B2 5/2009 Murakami 2009/0284672 A1 11/2009 Baek 7,637,648 B2 12/2009 Jung 2009/0284690 Al 11/2009 Kuo 7,687,983 B2 3/2010 Lee 2010/0013380 A1 1/2010 Kim 7,714,904 B2 5/2010 Kudoh 2010/0026934 A1 2/2010 Sun 7,772,765 B2 8/2010 Park 2010/0033461 A1 2/2010 Hasegawa 7,804,493 B2 9/2010 Gettemy 2010/0033462 A1 2/2010 Hasegawa 7,859,526 B2 12/2010 Konicek 2010/0085207 Al 4/2010 Leung 7,864,270 B2 1/2011 Zhuang 2010/0091207 A1 4/2010 Hasegawa 7,916,183 B2 3/2011 Kudoh 2010/0115407 A1 5/2010 Kim 8,023,080 B2 9/2011 Kuo 2010/0125816 A1 5/2010 Bezos 8,059,232 B2 11/2011 Zhuang 2010/0165267 A1 7/2010 Yoshida 8,138,669 B2 3/2012 Kim 2010/0188605 Al 7/2010 Hasegawa 8,157,399 B2 4/2012 Leung 2010/0208110 A1 8/2010 Kudoh 8,314,542 B2 11/2012 Lee 2010/0233601 A1 9/2010 Takimoto 8,330,909 B2 12/2012 Yoshida 2010/0252825 Al 10/2010 Yamazaki 8,362,992 B2 1/2013 Kuhlman 2010/0271394 Al 10/2010 Howard 8,724,304 B2 5/2014 Raff 2010/0277439 A1 11/2010 Charlier 8,947,627 B2 2/2015 Rappoport 2011/0001805 Al 1/2011 Mentz 9,282,614 B2 3/2016 Yang 2011/0001808 Al 1/2011 Mentz 9,316,843 B2 4/2016 Ishiguro 9,495,008 B2 11/2016 Savastinuk 2011/0038045 Al 2/2011 Zhou 9,543,364 B2 1/2017 Rappoport 2011/0057866 A1 3/2011 Konicek 2001/0036013 Al 11/2001 Allen 2011/0090324 Al 4/2011 Mentz 2002/0015120 A1 2/2002 Kameyama 2011/0096156 A1 4/2011 Kim 2002/0118452 Al 8/2002 Taniguchi 2011/0157708 Al 6/2011 Kuo 2002/0122019 A1 9/2002 Baba 2011/0195354 Al 8/2011 Nukada 2003/0002296 Al 1/2003 Steiner 2011/0195758 A1 8/2011 Damale 2003/0048522 Al 3/2003 Liang 2011/0227487 Al 9/2011 Nichol 2003/0074672 A1 4/2003 Daniels 2011/0255303 A1 10/2011 Nichol 2003/0189754 A1 10/2003 Sugino 2011/0273906 A1 11/2011 Nichol 2003/0198456 A1 10/2003 Steiner 2012/0049723 Al 3/2012 Lee 2003/0210370 A1 11/2003 Yano 2012/0057006 A1 3/2012 Joseph 2004/0036707 Al 2/2004 Willis 2012/0081524 A1 4/2012 Joseph 2004/0041800 A1 3/2004 Daniels 2012/0099033 A1 4/2012 Ishiguro 2004/0043139 A1 3/2004 Daniels 2012/0163021 A1 * 6/2012 Bohn GO2F 1/133615 2004/0070822 A1 4/2004 Shioda 362/608 2004/0108971 Al 6/2004 Waldern 2004/0119407 Al 6/2004 Kim 2012/0188637 Al 7/2012 Joseph 2004/0125430 A1 * 7/2004 Kasajima GO2F 1/133536 2012/0320287 Al 12/2012 Kim 359/247 2014/0362429 A1 * 12/2014 Yokozeki HOIL 27/3232 2004/0212300 A1 10/2004 Chao 359/267 2004/0239243 A1 12/2004 Roberts 2015/0077323 Al 3/2015 Ramaswamy 2005/0012881 Al 1/2005 Liang 2015/0155344 A1 6/2015 Lee 2005/0018106 A1 1/2005 Wang 2015/0295015 A1 10/2015 Yu 2005/0024339 Al 2/2005 Yamazaki 2015/0349293 A1 12/2015 Park 2005/0052342 A1 3/2005 Wu 2016/0027859 Al 1/2016 Kim 2005/0146492 Al 7/2005 Baba 2005/0213181 Al 9/2005 MacKinnon FOREIGN PATENT DOCUMENTS 2006/0006792 A1 1/2006 Strip 2006/0066236 A1 3/2006 Tanaka EP 2211397 A3 6/2014 2006/0132634 Al 6/2006 Kudoh JP 2000098386 A 4/2000 2006/0152812 Al 7/2006 Woodgate WO 2009100055 A3 8/2009 2006/0176541 A1 8/2006 Woodgate WO 2009125918 A2 10/2009 2006/0181662 A1 8/2006 Kameyama WO 2009125918 A3 10/2011 US 10,520,782 B2 Page 3
( 56 ) References Cited “White OLEDs (WOLEDs ) , ” Universal Display Corporation ( retrieved Jan. 26 , 2017 ) . FOREIGN PATENT DOCUMENTS “ Augmented reality , ” Wikipedia ( retrieved Jan. 26 , 2017) . “ Transparent Display, ” Kent Optoronics ( retrieved Jan. 26 , 2017) . WO 2015032901 A3 6/2015 “ InfoGlazing Transparent Display ,” Kent Optronics (retrieved on WO 2016164184 Al 10/2016 Feb. 2 , 2017 ) . “ Custom Solutions ," Kent Optronics ( retrieved Jan. 26 , 2017 ) . OTHER PUBLICATIONS “ AMOLED ,” Wikipedia (retrieved Jan. 26 , 2017 ) . " OLED : Felxible devices, transparent screens and organic lamps, " Venturing Mind (retrieved Jan. 26 , 2017 ) . “ Active shutter 3D system ,” Wikipedia (retrieved Jan. 14 , 2017) . “ Transparent OLED Display Module, ” Nuts and Volts ( retrieved “ StereoGraphics, ” Wikipedia ( retrieved Jan. 14, 2017 ) . Jan. 26 , 2017 ). “ CrystalEYES User's Manual, ” StereoGraphics Corporation (2000 ) . “ PMOLED ( PassiveMatrix OLED ): introduction and basics ,” OLED Alissa Wilkinson , “ Passengers is 3 movies in one, each creepier than Info ( retrieved Jan. 26 , 2017 ) . the last ,” Vox (Dec. 24 , 2016 ). “ TDK startsmass production of a transparent 2.4 " QVGAPMOLEDs, " “ HUD applications, ” Won Motors Korea Co., Ltd ( retrieved Jan. 14 , OLED - Info ( retrieved Jan. 26 , 2017 ) . 2017 ). “ Flexible OLEDs: introduction , applications and market status, " Ron Mertens , “ Hands on with 4D Systems 2 " transparent PMOLED OLED - Info ( retrieved Jan. 26 , 2017 ) . panel, ” OLED - Info ( Apr. 28 , 2013 ) . “ Transparent OLEDs: introduction and market status, " OLED - Info Ron Mertens, “ Lenovo's S - 800 uses a transparent PMOLED , prob ( retrieved Jan. 26 , 2017 ) . ably made by TDK , ” OLED - Info ( Apr. 21 , 2011) . “ OLED technology : introduction and basics, " OLED - Info (retrieved Vlad Savov, “ Lenovo teases $ 800 phone with translucent color Jan. 26 , 2017 ) . display, won't let anyone touch it ,” Engadget (Nov. 29 , 2010 ) . " OLED introduction and basic OLED information ,” OLED - Info Ron Mertens, “ Indonesia's Nexian launches a mobile phone with a ( retrieved Jan. 26 , 2017 ) . 2.4 " transparent OLED ,” OLED - Info ( Jul. 7 , 2012 ) . “ Technology, " Visionox OLED ( retrieved Jan. 26 , 2017 ). Ron Mertens , “ NeoView Kolon now offers transparent PMOLED “ OLED Technology ," Visionox OLED ( retrieved Jan. 26 , 2017 ) . displays and lighting panels, ” OLED - Info (Aug. 9 , 2012) . “ OLED Process , ” Visionox OLED (retrieved Jan. 26 , 2017 ) . “ Future Display, ” Visionox OLED ( retrieved Jan. 26 , 2017) . “ Samsung IceTouch ," OLED - Info ( Jan. 27, 2010 ) . “ Liquid - crystal display, ” Wikipedia ( retrieved Jan. 26 , 2017) . Transcript of “ Samsung MP3 player Ice Touch debuts at CES ,” “ Active -matrix liquid - crystal display, ” Wikipedia ( retrieved Jan. 26 , YouTube Video by operationna ( Jan. 8 , 2010 ). 2017 ) . Screenshots from “ Transparent AMOLED Display, ” YouTube Video “ Thin - film -transistor liquid -crystal display, ” Wikipedia ( retrieved by Did You Know ? ( Jan. 10 , 2013 ) . Jan. 26 , 2017 ). Rick Martin , “ TDK unveils flexible OLED display at CEATEC ," “ LCD Shutters, ” Liquid Crystal Technologies ( retrieved Jan. 26 , New Atlas (Oct. 5 , 2010 ) . 2017 ) . Jeff Engelhardt, “ L?M ,” Coroflot.com , Archive Copy from Feb. 17 , “ Active Domain LCD Shutter, ” Liquid Crystal Technologies ( retrieved 2014 , as retrieved from the Internet Archive . Jan. 26 , 2017 ) . Jeff Engelhardt, “HP L?M - Desktop Computer with Transparent “ Driving a Liquid Crystal Shutter, ” Liquid Crystal Technologies Monitor, ” Coroflot.com , Archive Copy from Sep. 13 , 2010 , as ( retrieved Jan. 26 , 2017 ) . retrieved from The Internet Archive . “ LCD Shutters Frequently Asked Questions, ” Liquid Crystal Tech Beauhd , “ Apple Patent Paves Way for iPhone With Full -Face nologies ( retrieved Jan. 26 , 2017 ). Display HUD Windows, ” Slashdot ( Jan. 10 , 2017 ) . Dejan Karadaglic , “ Single LED Takes on Both Light -Emitting and Robert Scoble , “ Apple Strategy 2017. Very important change to Detecting Duties, ” Electronic Design ( Jul. 19 , 2007 ) . iPhone Coming, ” VirturalReality Pop.com (Oct. 22 , 2016 ) . " Screenshots of Selected Portions of Passengers Official Trailer Rob Waugh , “ Staring death in the face: Nuclear explosion captured (HD ), ” YouTube 7BWWWQZTONU (Sep. 20 , 2016 ) . one ten - millionth of a second after detonation is a warped , 100ft Tracy Wilson et al. , “ How the iPhone Works, ” HowStuffWorks skull ,” Daily Mail (Dec. 14 , 2011 ). ( retreived Jan. 27, 2017 ) . “ Rapitronic camera ,” Wikipedia (retrieved Jan. 16 , 2017) . Mario , Aguilar , “ How Amazon's Fire Phone Cameras Know Where “ Kerr cell shutter, ” Wikipedia ( retrieved Jan. 16 , 2017 ) . Your Face Is Even at Night, ” Gizmodo ( Jun . 19 , 2014 ) . “ Fast optical shutters, " LCTec ( retrieved on Jan. 18 , 2017 ). “ PenTile matrix family ,” Wikipedia ( retrieved Jan. 26 , 2017 ) . “ FOS series model overview , " LCTec (2016 ) . Heo , Joon et al. , “ Fast -switching initially -transparent liquid crystal Jamy Baulan , “ Choosing an Auto -Darkening Welding Helmet ,22" light shutter with crossed patterned electrodes ,” AIP Advances 5 , Lincoln Electric (retrieved Jan. 26 , 2017 ). 047118 (Apr. 2015 ) . “ UTOLED - 20 -G2 2.0 " Intelligent Transparent OLED module, ” “ iPhone , ” Wikipedia ( retrieved Jan. 27 , 2017 ) . 4D Systems ( retrieved Jan. 17 , 2017 ) . “ Accelerometer, ” Wikipedia ( retrieved Jan. 27 , 2017 ) . Kiyoshi Kiyokawa , “ An optical see- through display for mutual “Gyroscope , ” Wikipedia ( retrieved Jan. 27 , 2017 ). occlusion with a real - time stereovision system , ” Computer & Graph “ Magnetometer, ” Wikipedia ( retrieved Jan. 27, 2017 ) . ics 25 ( 2001 ) 765-779 . " Inertial measurement unit, ” Wikipedia ( retrieved Jan. 27 , 2017 ) . Stewart Miles, “ The Samsung Transparent Smart Window makes “ Quaternion ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . sci- fi movies a reality , ” Pocket -lint ( Jan. 12, 2012 ). “ Kalman filter ,” Wikipedia ( retrieved Jan. 27, 2017 ) . Tapti Shop ( retrieved Jan. 26 , 2017 ) . “ Proximity sensor, ” Wikipedia ( retrieved Jan. 27, 2017 ) . Tapt1 Shop Digital Glass Cooler ( retrieved Jan. 26 , 2017 ) . “ Photodetector, ” Wikipedia (retrieved Jan. 27 , 2017 ). Transcript of “ 4DS transparent OLED hands on review ," YouTube “ Fingerprint recognition, ” Wikipedia ( retrieved Jan. 27 , 2017 ). Video by oledinfo (Apr. 28 , 2013) . “ Broadcast automation ,” Wikipedia ( retrieved Jan. 27 , 2017) . TOLED ( transparent OLED ) , ITWissen.info (retrieved Jan. 26 , “ Audience measurement, ” Wikipedia ( retrieved Jan. 27, 2017 ) . 2017 ) . " Subaudible tone, ” Wikipedia ( retrieved Jan. 27 , 2017 ). OLED , Wikipedia (retrieved Jan. 26 , 2017 ) . “ QR code, ” Wikipedia (retrieved Jan. 27 , 2017 ). Organic Light Emitting Diodes ( OLEDS) , Universal Display Cor “ Touchscreen ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . poration ( retrieved Jan. 26 , 2017 ). “ Resistive touchscreen, ” Wikipedia ( retrieved Jan. 27 , 2017 ). Flexible OLEDS ( FOLEDS ) , Universal Display Corporation (retrieved " Capacitive sensing, ” Wikipedia ( retrieved Jan. 27 , 2017 ) . Jan. 26 , 2017 ). " Cellular network ,” Wikipedia ( retrieved Jan. 27 , 2017 ). Transparent OLEDS ( TOLEDS) , Universal Display Corporation " Wi- Fi, ” Wikipedia ( retrieved Jan. 27 , 2017) . ( retrieved Jan. 26 , 2017 ) . “ Bluetooth ,” Wikipedia (retrieved Jan. 27 , 2017 ). US 10,520,782 B2 Page 4
( 56 ) References Cited “ Gaze ,” Microsoft Windows Dev Center ( retrieved Jan. 27 , 2017 ) . “Getting a HolographicSpace, ” Microsoft Windows Dev Center OTHER PUBLICATIONS ( retrieved Jan. 27, 2017 ) . “ HoloLens Hardware details ,” Microsoft Windows Dev Center “ Bluetooth low energy ,” Wikipedia ( retrieved Jan. 27 , 2017 ). ( retrieved Jan. 27 , 2017 ) . “ ISM band, ” Wikipedia (retrieved Jan. 27 , 2017 ). “ Hologram , " Microsoft Windows Dev Center ( retrieved Jan. 27 , “ Near field communication ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . 2017 ) . “ Radio - frequency identification, ” Wikipedia ( retrieved Jan.27 , 2017 ). “ Mesh networking, ” Wikipedia ( retrieved Jan. 27 , 2017 ) . “ Mixed reality capture for developers ,” Microsoft Windows Dev “ Microphone, ” Wikipedia ( retrieved Jan. 27 , 2017) . Center (retrieved Jan. 27 , 2017 ) . “ Microphone array ,” Wikipedia (retrieved Jan. 27 , 2017 ) . “ Spatial mapping, ” Microsoft Windows Dev Center ( retrieved Jan. “ Active nosie control, ” Wikipedia ( retrieved Jan. 27 , 2017 ) . 27 , 2017 ) . “ Sound localization ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . “ Working with holograms, ” Microsoft Windows Dev Center ( retrieved “ Acoustic location ,” Wikipedia ( retrieved Jan. 27 , 2017) . Jan. 27 , 2017 ). “ Acoustic source localization , ” Wikipedia ( retrieved Jan. 27, 2017 ) . " What can I do with HoloLens offline, ” Microsoft Windows Dev " Speech recognition ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . Center ( retrieved Jan. 27 , 2017) . " Computer speaker, ” Wikipedia ( retrieved Jan. 27 , 2017) . “ Holograms 101, ” Microsoft Windows Dev Center ( retrieved Jan. “ Speech synthesis ,” Wikipedia ( retrieved Jan. 27, 2017 ). 27 , 2017 ). “ Artificial intelligence , ” Wikipedia ( retrieved Jan. 27, 2017) . Ron Mertens, “ Re : [Website feedback ] 4DS transparent OLED ,” " OpenAI, ” Wikipedia ( retrieved Jan. 27, 2017) . OLED - Info (Feb. 2 , 2017 ). “ Documentation , ” OpenAI Gym ( retrieved Jan. 27, 2017) . “ Layar, ” Wikipedia (retrieved Jan. 30 , 2017 ) . " Computer vision ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . “ PARK Version 1.0.2 ,” Apple (Jun . 26 , 2012 ) . “ OpenCV, ” Wikipedia ( retrieved Jan. 27, 2017 ) . “ BRSun Tracker 1.0.1, ” CocoaPods ( retrieved Jan. 30 , 2017 ) . “ The OpenCV Reference Manual Release 2.4.13.2 ,” OpenCV ( Jan. “ SunPos.h , ” psa.es/sdg/archive ( retrieved Jan. 30 , 2017) . 27 , 2017 ) . Broco, “ Compare device 3D prientation with the sun position, ” “ The OpenCV Tutorials Release 2.4.13.2 ,” OpenCV (Jan. 27 , Stack Overflow (Jan. 21, 2014 ). 2017 ) . “ Outline of object recognition ,” Wikipedia ( retrieved Jan. 30 , 2017 ) . “ The OpenCV User Guide Release 2.4.13.2 ,” OpenCV ( Jan. 27 , “ Text messaging ,” Wikipedia ( retrieved Jan. 30 , 2017 ). 2017 ) . “ Camera phone, ” Wikipedia ( retrieved Jan. 30 , 2017 ) . “ Apple motion coprocessors , ” Wikipedia ( retrieved Jan. 27 , 2017) . “ HTC Vive, ” Wikipedia ( retrieved Jan. 30 , 2017 ) . " Barometer, ” Wikipedia ( retrieved Jan. 27 , 2017 ). " Oculus Rift, ” Wikipedia ( retrieved Jan. 30 , 2017) . " Compass ” Wikipedia ( retrieved Jan. 27, 2017) . “ Encryption ,” Wikipedia ( retrieved Jan. 30 , 2017) . “ Tango (platform ), ” Wikipedia ( retrieved Jan. 27, 2017 ) . “ Symmetric -key algorithm ,” Wikipedia ( retrieved Jan. 30 , 2017 ) . “ Duty cycle ," (retrieved Jan. 27 , 2017) . “ Public key cryptography, ” Wikipedia ( retrieved Jan. 30 , 2017) . “ Global Positioning System ,” Wikipedia ( retrieved Jan. 27, 2017 ) . “ Hypertext Transfer Protocol, ” Wikipedia (retrieved Jan. 30 , 2017 ) . “ Galileo ( satellite navigation ) ,” Wikipedia ( retrieved Jan. 27, 2017) . “ Database, ” Wikipedia ( retrieved Jan. 30 , 2017 ) . “ GLONASS, ” Wikipedia ( retrieved Jan. 27, 2017) . “ Create , read , update and delete ," Wikipedia (retrieved Jan. 30 , “ BeiDou Navigation Satellite System ,” Wikipedia (retrieved Jan. 2017 ) . 27 , 2017 ) . “ Representational state transfer, ” Wikipedia ( retrieved Jan. 30 , “ Assisted GPS, ” Wikipedia ( retrieved Jan. 27 , 2017) . 2017 ) . " Hybrid positioning system ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . “ JSON ,” Wikipedia ( retrieved Jan. 30 , 2017 ). “GPS navigation device , ” Wikipedia ( retrieved Jan. 27 , 2017) . "GeoJSON ,” Wikipedia (retrieved Jan. 30 , 2017 ) . “ Satellite navigation ,” Wikipedia ( retrieved Jan. 27 , 2017) . “ Eye tracking ,” Wikipedia ( retrieved Jan. 31, 2017 ) . “ Indoor positioning system ,” Wikipedia ( retrieved Jan. 27, 2017 ). “ Basics of LCD Operation ,” Liquid Crystal Technologies ( retrieved “ Bluetooth low energy beacon , ” Wikipedia (retrieved Jan. 27 , Jan. 30 , 2017 ) . 2017 ) . “ Twisted nematic field effect, ” Wikipedia ( retrieved Jan. 30, 2017 ) . “ IBeacon ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . “ Pi Cell ,” Liquid Crystal Technologies ( retrieved Jan. 30 , 2017 ) . "Mobile phone tracking, ” Wikipedia ( retrieved Jan. 27 , 2017) . “ Kerr effect, ” Wikipedia ( retrieved Jan. 30 , 2017) . “ Positioning system ,” Wikipedia ( retrieved Jan. 27 , 2017) . “ Samsung Display reportedly decided to halt transparent OLED “ Wi- Fi positioning system ,” Wikipedia ( retrieved Jan. 27 , 2017 ) . production ," OLED - Info ( Jul. 27 , 2016 ). “ Proximity marketing, ” Wikipedia (retrieved Jan. 27 , 2017 ). “ LucidVue — Transparent OLED , ” Crystal Display Solutions ( retrieved “ Multilateration ,” Wikipedia ( retrieved Jan. 27, 2017) . Jan. 18 , 2017 ) . " Optical head -mounted display, ” Wikipedia ( retrieved Jan. 27 , Terri Bassitt , “ Futaba exhibits Transparent OLED Displays at this 2017 ). year's CEATEC ( Combined Exhibition of Advanced Technolo “ Smartglasses, ” Wikipedia (retrieved Jan. 27 , 2017 ) . gies ), ” LinkedIn (Oct. 26 , 2016 ) . " Smart Glasses MOVERIO BT- 300 User's Guide, ” Seiko Epson " 2.0 " Transparent OLED Display microOLED GOLDELOX Mod Corporation (Nov. 2016 ). ule uTOLED - 20 -G2 Revision 1.2 ,” 4D Systems (May 22, 2013 ) . “ Microsoft HoloLens, ” Wikipedia ( retrieved Jan. 27, 2017) . “ Product Brief 2.0 " microTOLED Intelligent GOLDELOX Display “ List of all pages in the Windows Holographic documentation as of Module uTOLED - 20 -G2 Revision 1.0 , ” 4D Systems ( 2012 ) . May 31, 2016 , ” Microsoft Windows Dev Center (retrieved Jan. 27 , " SwitchLite Privacy Glass ,” Pulp Studio ( retrieved Jan. 26 , 2017 ). 2017 ) . Chris Ziegler, “ Sony Ericsson Xperia Pureness hands -on ,” Engadget “ App model ,” Microsoft Windows Dev Center (retrieved Jan. 27 , ( Dec. 30 , 2009 ) . 2017 ) . Transcript and Selected Screenshots of “ Transparent Displays at “ App views on HoloLens, ” Microsoft Windows Dev Center ( retrieved MIT, ” YouTube Video ( Jan. 21, 2014 ) . Jan. 27 , 2017 ) . Anirudh Sharma, “ Augmenting a traditional ruler with digital “ Coordinate systems,” Microsoft Windows Dev Center ( retrieved functionality, ” anirudh.me ( 2013 ) . Jan. 27 , 2017 ) . “ Smart glass ,” Wikipedia ( retrieved Feb. 1 , 2017 ) . “ Designing for mixed reality ,” Microsoft Windows Dev Center " Setup for All Unity Tango Apps, ” Tango Unity SDK Google ( retrieved Jan. 27, 2017 ) . Developers (Jan. 13, 2017 ) . “ Gesture design ,” Microsoft Windows Dev Center ( retrieved Jan. “ Unity How -to Guide: Simple Augmented Reality , ” Tango Unity 27 , 2017 ). SDK Google Developers ( Jan. 12 , 2017 ) . “ Gaze targeting, ” Microsoft Windows Dev Center ( retrieved Jan. “ Unity How -to Guide : Placing Virtual Objects in Augmented Real 27 , 2017 ) . ity , ” Tango Unity SDK Google Developers ( Jan. 12 , 2017 ) . US 10,520,782 B2 Page 5
( 56 ) References Cited “ 3D projection , ” Wikipedia ( retrieved Feb. 2 , 2017) . “ Vector projection ,” Wikipedia ( retrieved Feb. 2 , 2017 ) . OTHER PUBLICATIONS Andrejs Treibergs, “ The Geometry of Perspective Drawing on the Computer ,” University of Utah Department of Mathematics ( retrieved “ Unity How - to Guide: Meshing with Color, " Tango Unity SDK Google Developers ( Jan. 14 , 2017 ) . Feb. 2 , 2017 ) . “ Tango with Cardboard ,” Tango Unity SDK Google Developers Allan Jepson , " CSC420 : Image Projection ( Slides) , ” University of (Jun . 9 , 2016 ) . Toronto Department of Computer Science ( Sep. 2011) . “ Unity How - to Guide: Depth Perception ,” Tango Unity SDK Google " Transformation matrix ,” Wikipedia ( retrieved Feb. 2, 2017) . Developers ( Jan. 12 , 2017 ) . " Camera matrix ,” Wikipedia ( retrieved Feb. 2 , 2017) . “ Unity How -to Guide: Motion Tracking, ” Tango Unity SDK Google Robert Collins, “ Lecture 12 : Camera Projection ,” CSE486 , Penn . Developers (Jan. 12 , 2017 ) . State (retrieved Feb. 2 , 2017 ) . “ Unity How - to Guide: Area Learning, ” Tango Unity SDK Google Jason McKessen , " Chapter 4 of Learning Modern Graphics Pro Developers (Jan. 12 , 2017) . gramming ; Objects at Rest, Perspective Projection ” Paroj.Github.io “ ZenFone AR (ZS571KL ) Phone, ” ASUS Global ( retrieved Feb. 2 , (2012 ). 2017 ) . “ HeadTrackingConfiguration Class Reference, ” Native HeadTrack Ron Amadeo , “ Amazon's 3D Kindle Phone to use Omron face ing API for Fire Phone , Amazon Apps & Services Developer Portal tracking ," Ars Technica ( Jun . 6 , 2014 ) . ( retrieved Feb. 2 , 2017 ) . “ Retrieving Head - Tracking Data Using Native Code, ” Amazon " HeadTrackingEvent Class Reference, ” Native HeadTracking API Apps & Services Developer Portal ( retrieved Feb. 2 , 2017 ). for Fire Phone ,Amazon Apps & Services Developer Portal ( retrieved “ Understanding the Native -Code HeadTracking API, ” Amazon Apps Feb. 2 , 2017 ). & Services Developer Portal ( retrieved Feb. 2 , 2017 ). “ HeadTrackingManager Class Reference, ” Native HeadTracking “ 3D camera, ” Wikipedia (retrieved Feb. 2 , 2017 ) . API for Fire Phone, Amazon Apps & Services Developer Portal “ Range imaging, ” Wikipedia ( retrieved Feb. 2 , 2017) . (retrieved Feb. 2 , 2017 ) . “ Stereo camera, ” Wikipedia ( retrieved Feb. 2 , 2017 ). “ HeadTrackingPoller Class Reference, ” Native HeadTracking API “ Time- of - flight camera , ” Wikipedia ( retrieved Feb. 2 , 2017) . for Fire Phone, Amazon Apps & Services Developer Portal ( retrieved “ Depth perception ,” Wikipedia ( retrieved Feb. 2 , 2017 ) . Feb. 2 , 2017 ) . Jean -Pierre Distler, “ Augmented Reality iOS Tutorial: Location “ Light field ,” Wikipedia ( retrieved Feb. 2 , 2017 ) . Based, ” Ray Wenderlich ( Jan. 23 , 2017) . " Light field camera ,” field camera ( retrieved Feb. 2 , 2017 ) . “ Core Location ,” Apple Developer Documentation (Mar. 21, 2016 ) . “ Lytro ,” Wikipedia ( retrieved Feb. 2 , 2017 ) . Nate Cook , " CMDeviceMotion ,” NSHipster (Oct. 28 , 2014 ). Osian Haines , “ An Introduction to Simultaneous Localisation and “ Core Motion ,” Apple Developer Documentation ( retrieved Feb. 2 , Mapping, ” Kudan Conputer Vision (May 13 , 2016 ) . 2017 ) . “ Motion Events , ” Apple Developer Documentation (Mar. 21 , 2016 ) . * cited by examiner U.S. Patent Dec. 31, 2019 Sheet 1 of 28 US 10,520,782 B2
-100
- 102
-104 ORCHESTRATOR m106
-108
FIG . 1A
102 104 -106 ORCHESTRATOR -108
FIG . 1B U.S. Patent Dec. 31, 2019 Sheet 2 of 28 US 10,520,782 B2
-102
110 -104 ORCHESTRATOR -106
FIG . 1C
110 -104 ORCHESTRATOR -106
FIG . 1D U.S. Patent Dec. 31, 2019 Sheet 3 of 28 US 10,520,782 B2
-100
--202
110 -204 ORCHESTRATOR --206
--108
FIG . 2A
100 202 204 110 206 ORCHESTRATOR -108
FIG . 2B U.S. Patent Dec. 31, 2019 Sheet 4 of 28 US 10,520,782 B2
--202
110 - 204 ORCHESTRATOR -206
FIG . 2C
-202
110 200 ORCHESTRATOR I -206
FIG . 2D U.S. Patent Dec. 31, 2019 Sheet 5 of 28 US 10,520,782 B2
302 302 106 106
104 FIG.3C 104 FIG.3F
102 102 300 300 302 302 106 106
104 3BFIG. 104 FIG.3E 102 102 - 300 300
302 302
106 304 106 e
104 $ FIG.3A 104 FIG3D.
102 102 300 300 U.S. Patent Dec. 31, 2019 Sheet 6 of 28 US 10,520,782 B2
302 302 -106 106 104 FIG.31 104 FIG.3L
102 102 300 300
302 302 -106 -106 104 .3HFIG 104 FIG.3K
102 102 300 300
302 302 106 106
104 FIG.3G 104 FIG.3
102 102 300 300 U.S. Patent Dec. 31, 2019 Sheet 7 of 28 US 10,520,782 B2
402 J 402 206 206
204 FIG.4C 204 FIG.4F
202 202 A 400 400 402 402 206 206
204 FIG.4B 204 FIG.4E
202 202 400 400 402 402 d 206 206
204 FIG.4A 204 FIG.4D
HL$ 202 202 400 400 U.S. Patent Dec. 31 , 2019 Sheet 8 of 28 US 10,520,782 B2
402 402 206 206 * 204 FIG.41 204 4LFIG.
202 202 200W 400 400 402 402 206 206 &
204 FIG.4H 204 FIG.4K
202 202 400 400
402 402 206 206 204.. FIG.4G 204 4J.FIG 202 202 400 400 U.S. Patent Dec. 31, 2019 Sheet 9 of 28 US 10,520,782 B2
402 402 206 206
204 FIG.40 204 4R.FIG 202 202 400 400 402 402 206 206
204 FIG.4N 204 FIG.4Q
202 202 400 400 402 402 an 206 206
204 FIG.4M 204 4PFIG.
202 202 400 U.S. Patent Dec. 31, 2019 Sheet 10 of 28 US 10,520,782 B2
510 700 FIG . 3G 710
FIG . 5A FIG . 7A
512 702 FIG . 30 712
FIG . 5B FIG . 7B
514 702 FIG . 3C 714
-516 FIG . 3G 12 716 FIG . 5C FIG . 70 U.S. Patent Dec. 31, 2019 Sheet 11 of 28 US 10,520,782 B2
702 718 518 528 FIG . 3C FIG . 3F 720 -526 4 520 FIG . 3D 700 722 7 FIG . 3F 724 ta FIG . 3D FIG . 30 726t. 522 524 FIG . 5D - 728 16
FIG . 7D
530 708 t 700 FIG . 34 730
FIG . 5E FIG . 7E
532 708 702 FIG . 3L 732
FIG . 5F FIG . 7F U.S. Patent Dec. 31, 2019 Sheet 12 of 28 US 10,520,782 B2
534 702 FIG . 3C 708 734
536 6 FIG . 3K 736 FIG . 5G FIG . 7G 538 700 FIG . 3G 738 708 740 540 FIG . 3K
FIG . 5H FIG . 7H 702
742 542 708 ti FIG . 3C 548 744 t2 FIG . 3B 544 A FIG . 3B tz FIG . 3K 746 14 546 -748 FIG . 51 FIG . 71 U.S. Patent Dec. 31, 2019 Sheet 13 of 28 US 10,520,782 B2
750 550 708 FIG . 3G 556 FIG . 3H 700752 552 3 FIG . 3H AAS 754 FIG . 3K 756 554 FIG . 5J FIG . 7J
758 702 558 700 FIG . 3C 564 560 FIG . 3K 760 CAN 762 FIG . 3K
FIG . 3G 562 764 FIG . 5K FIG . 7K U.S. Patent Dec. 31, 2019 Sheet 14 of 28 US 10,520,782 B2
610 800 FIG . 4B 810
FIG . 6A FIG . 8A 612 802 FIG . 4P 812 w
FIG . 6B FIG . 8B
614 800 814 802 FIG . 4H
FIG . 6C FIG . 8C
808 616 proape 816 802 FIG . 4C
FIG . 6D FIG . 8D U.S. Patent Dec. 31, 2019 Sheet 15 of 28 US 10,520,782 B2
618 808 818 802 FIG . 40
FIG . 6E FIG . SE
620 808 820 802 FIG . 4K
FIG . 6F FIG . 8F
622 t 800 822 808 FIG . 4M
FIG . 6G FIG . 8G
624 800 824 808 FIG . 4A
Wwwwww
FIG . OH FIG . 8H U.S. Patent Dec. 31, 2019 Sheet 16 of 28 US 10,520,782 B2
626 800 826 808 FIG . 4E
68
FIG . 61 FIG . 81
802 628 -828 800 FIG . 4P 630 FIG . 40 ah 830 FIG . 6J FIG . 8J
800 832 632 802 FIG . 4B 634 FIG . 40 834 FIG . OK FIG . 8K U.S. Patent Dec. 31, 2019 Sheet 17 of 28 US 10,520,782 B2
800 802 636 836 FIG . 4H -638 FIG . 4Q 838 FIG . OL FIG . 8L
802 840 640 800 FIG . 4P 646 642 FIG . 40 842 FIG . 40 844 FIG . 4B hahah -644 846 FIG . 6M FIG . 8M U.S. Patent Dec. 31, 2019 Sheet 18 of 28 US 10,520,782 B2
-104 102 -104 102 - 106 106
902 900 902 900 FIG.9A FIG . 9B
902 -102 102 900 104 106 106 902 900 FIG . 9C FIG.9D
104 -906 102 106 106 908
102 104 FIG . SE FIG . 9F
102 104 -104 102 902 902 -900 FIG . OG FIG . OH U.S. Patent Dec. 31, 2019 Sheet 19 of 28 US 10,520,782 B2
902 900 -104 102 902-07102900 FIG.91 FIG . 9J
204 - 206 202 -204 202 -206
1002 1002 FIG . 10A FIG . 10B
-1002 202 202 -1000 204 204 206 206 1002 1000 FIG . 10C FIG . 10D
204 1006 202 206 - 206 1008 -1004 IT 202 204 FIG . 10E FIG . 10F U.S. Patent Dec. 31 , 2019 Sheet 20 of 28 US 10,520,782 B2
104 106 102 102 106 1102 -1102 1104 1108 o 1104 1108 -1106 7 1106 FIG . 11A FIG . 11B
1108 102 1106 102 104 1108 106 1102 1104 1102 106 1104 104 1106 FIG . 110 FIG . 11D
106 1102 104 102 ZZZZZZZZZZZZQ1108 1106 FIG . 11E U.S. Patent Dec. 31, 2019 Sheet 21 of 28 US 10,520,782 B2
204 - 206 202 204 202 206 -1202 -1202 2-1204 1208 1208 -1206 z774–1204-1206 FIG . 12A FIG . 12B
1208 -202 1206 -202 -204 1208 --206 1202 1204 1202 206 zzz -1204 204 1206 FIG . 120 FIG . 12D
206 1202 204 ZZZZZZZZZZZZZ21208 12041206 FIG . 12E U.S. Patent Dec. 31 , 2019 Sheet 22 of 28 US 10,520,782 B2
Screen Horizontal Up Vertical Left Horizontal Down Vertical Right Orientation Based on User's Eye Orientation 5 #g 19 . to ( 1g and Device Orientation 1304 KOKUUUUU1304 1306 OLIZU1308 . Bottom Down T 8 T LR B 1 9 Portrait Up to Top Landscape Up to Portrait Up to Landscape Up to of Device Left of Device Bottom of Device Right of Device 1310 1312 1314 1316 1318 Left Down wand A R ? 1 B 1 Landscape Up to Portrait Up to Top Landscape Up to Portrait Up to Right of Device of Device Left of Device Bottom of Device OR Landscape Up to Landscape Up to Right of Device Rightof Device 1320 1322 1324 1326 1328
Top Down 8 ???? enne 8 d 7 1 B Portrait Up to Landscape Up to Portrait Up to Landscape Up to Bottom of Device Right of Device Top of Device Left of Device 1330 1332 1334 1336 1338 Right Down 8
. 1 B Landscape Up to Portrait Up to Landscape Up to Portrait Up to Left of Device Bottom of Device Right of Device Top of Device -OR -OR Landscape Up to Landscape Up to Left of Device Left of Device 1340 1342 1346 1348 FIG . 13 U.S. Patent Dec. 31, 2019 Sheet 23 of 28 US 10,520,782 B2
-1460
1400
1402 1440 -1404 WWW
1422 ? ? ? . ? ? -1424 1442 -1444 1406 1426 7 -1428 1408
FIG . 14 U.S. Patent Dec. 31, 2019 Sheet 24 of 28 US 10,520,782 B2
1530
1524 1544 1532 K 1512 z 1510 N
-N 1514 1542 1522 1500 1534 -1518 1520 1516
1540 1538 1536
YYYY YYYYYYYYYYYYYYYYYYY FIG . 15 U.S. Patent Dec. 31, 2019 Sheet 25 of 28 US 10,520,782 B2
-1608 1600 1604 - 1606 1606 1608 *** *rrett
***** 1600 --1602 ***
1602 1604 FIG . 16A FIG . 16B
1712 1702 -1710 1708 - 1700 GH - 1704 1708 1702
1710 1704 HAARKA 1712 1722 Som 1720 1722
FIG . 17A FIG . 17B U.S. Patent Dec. 31, 2019 Sheet 26 of 28 US 10,520,782 B2
-1802 -1800 1806 1810 1802 -1804
-1800 -1810 1808 FIG . 18A FIG . 18B
-1802 -1800 -1812 1804 1802 1800 1814 -1808 FIG . 18C FIG . 18D U.S. Patent Dec. 31, 2019 Sheet 27 of 28 US 10,520,782 B2
1900
3mi PEAK 1000ft 15 people hiking -1904
1902
FIG . 19A U.S. Patent Dec. 31, 2019 Sheet 28 of 28 US 10,520,782 B2
1908 1910 mrt nin -1900
O
-1906
1902
FIG . 19B US 10,520,782 B2 1 2 DISPLAY DEVICES , SYSTEMS AND Samsung Smart Window is a transparent LCD meant to act METHODS CAPABLE OF SINGLE -SIDED , as both a window and smart device capable of using apps DUAL - SIDED , AND TRANSPARENT MIXED and widgets , as well as connecting to the internet . See Cite REALITY APPLICATIONS No. D038. To prevent people outside the window from 5 seeing what a user of the screen is doing, Samsung attached CROSS REFERENCE TO RELATED a privacy screen to the reverse side that acts like a one -way APPLICATIONS mirror. See Cite No. D028 . Thus, the see -through display is only able to be viewed from one side, while the other side This application claims the benefit of U.S. Provisional is permanently masked by a one -way mirror. Application 62 /454,020 , filed Feb. 2 , 2017 , which is herein 10 Other devices incorporating see - through display elements incorporated by reference in its entirety. are discussed in Cite Nos. D029 , D030 , D001 , D027 , and U.S. Publication 20110157708 . BACKGROUND Example See - Through Display Elements . 15 See -through displays may be made of many types of 1. Field of the Invention display technologies including : Liquid - crystal Display ( Cite The invention generally relates to displays and electronic No. D055) , Active Matrix LCD ( AMLCD ) (Cite No. D056 ), devices that may appear to be translucent or transparent TFT LCD (Cite No. D057 ) , OLED Technology ( Cite Nos. ( e.g., see -through ) and /or display an image or augmented or D034 , D049 , D050 , and D037 ) , Flexible OLED (FOLED ) mixed reality annotation . These devices can display the 20 (Cite Nos. D035 and D047 ), Active -Matrix Organic Light same or different images on one or both sides of a display Emitting Diode (AMOLED ) (Cite Nos . D042, D066 ), Pas and maintain user privacy on occasions where it is not sive -Matrix Organic Light -Emitting Diode (PMOLED desirable show images on one side of a display . The (Cite Nos . D045, D015 , and D046 ) , and Transparent display may appear to become opaque, or may appear to Organic Light Emitting Diode/ Display ( TOLED ) ( Cite Nos. remain transparent or translucent and only show an image on 25 D032, D033, D036 , D044, D048) , and Quantum Dot one side of the screen . The devices may also be used for Organic Light Emitting Diodes (QLEDs ) . augmented or mixed reality applications . Unlike standard OLEDs, TOLEDs are transparent when Some formsof the invention are achieved by , for example , all layers the substrate , the anode , the hole - transporting placing a transparent display device ( e.g., such as a Trans layer, the emissive polymer layer and the cathode are parent Organic Light Emitting Diode ( TOLED ) or other 30 transparent in the inactive state . A transparent OLED in the display element ( such as any other type of emissive ele active state emits light in both directions and exits at the ment) ) directly or indirectly adjacent to an active shutter front and the back of the screen . The viewers can observe the ( e.g., an electronically actuated element that can block light display on both sides. However, the image on one side will transmission or pass light ( e.g. , a liquid crystal shutter )) , by be the mirror of the image on the other side . Such TOLED placing a transparent display between two active shutters , or 35 can be used as active matrix OLED ( AMOLED ) and as a by placing an active shutter substantially between two passive matrix OLED be constructed (PMOLED ) . The transparent displays . TOLED technique can be used for head - up displays and The electronically actuated elements are operate in con large format OLED displays . See Cite No. D032 . junction with one another to provide a transparent view , Transparent OLEDs use transparent or semi- transparent display light, or block lightappropriate image for viewers on 40 contacts on both sides of the device to create displays that each side of the device as desired . can be made to be both top and bottom emitting ( transpar ent) . TOLEDs can greatly improve contrast , making itmuch 2. Description of Related Materials easier to view displays in bright sunlight . This technology can be used in Head -up displays, smart windows or aug To reduce the complexity and length of the Detailed 45 mented reality applications and has purportedly been used Specification , and to fully establish the state of the art in ( with a black , light absorbing backing ) . See Cite No. D033 . certain areas of technology , Applicant herein expressly Other aspects of OLED display devices are discussed in Cite incorporates by reference all the following materials iden No. D067 and U.S. Pat. No. 5,986,401 . tified in each numbered paragraph below . The Cite Numbers Example See- Through Display Devices . referred to herein are set forth in the Information Disclosure 50 Many other concepts companies have developed products Statement ( IDS) filed contemporaneously with this applica utilizing transparent displays as well. tion and incorporated by reference herein . Concept and other fictional devices in television and The related art shows the novel and non - obvious nature of movies illustrate the shortcomings of the currenttechnology , the present invention including secondary considerations of and the failure of others to develop solutions to the problems non -obviousness such as the long - felt need and failure of 55 described and solved by Applicant herein . See Cite Nos . others to achieve the present invention . All referenced D043, D016 , D017 , D005, D063 (showing mirror image of materials are herein incorporated by reference in its entirety . display on back ), D183 , D014 , and D184 . 2.A. Display Technology Including See - Through Dis There have also been several concept or commercially plays . available devices incorporating see - through displays. See " A see - through [ or transparent ] display is an electronic 60 Cite Nos. D006 , D008 . D009 (Lenovo announced the S -800 display that allows the user to see what is shown on the glass in November 2010 and utilized a transparent PMOLED screen while still being able to see through it. ” See Cite . No. screen ) , D010 (“ Indonesian phone maker Nexian launched a D001 . A transparent display may be made using LCD or new phone called the Nexian Glaze M - 9090 which sports a LED technology . Id . 2.4 inch transparent OLED . " ), and D011 (“ Neo View offers In retail applications, transparent displays are used in 65 four different transparent PMOLEDs: 2.41 in 240x342 , shopping windows to show the product inside as well as to 16M - colors panels , a 1.98 in 128x160 panel ( 262K colors ) show text or advertisements on the glass . For example , the and monochrome (blue or white ) 1.3 in panels . ” ) US 10,520,782 B2 3 4 In 2010 , the Samsung Ice Touch MP3 Player was allegedly Historically , there have been various suppliers of display the world's first product with a Transparent OLED Display. elements discussed above as discussed in the following See Cite No. D012 . A video of the Samsung IceTouch shows references . See Cite Nos . D051, D052 , D053 , D054 , D177 , that the transparent display shows an image on both sides of D178 , D015 , D026 , D179 , D180 , D031, and D007 . How the screen , and that the touch screen interface is accessible 5 ever , due to the above and similar market failures , there is no from both sides of the screen . However, the display on the current supply of transparent OLED panels available for back side of the screen is reversed so that text and images are prototyping . According to Ron Mertens, CEO and Editor displayed backward . See Cite No. D013 at 0:21 ( showing the front of the screen displaying the word “ Pictures” ) and In - Chief of OLED-Info.com , as of Feb. 2 , 2017 , “ 4D5 is not D031 at 0:22 ( showing the back of the screen displaying the 10 making [UTOLED - 20 ] panels any more, and we do not have mirror image of the word “ Pictures” ). See also D013 at 1:09 any to offer. ” See , e.g., Cite No. D007 . Mr. Mertens also ( showing “Maroon 5 If I Never ” ) and D013 at 1:13 (showing stated “ as far as I know currently there are no comparable the mirror image of the Maroon 5 album art and text) . The displays on offer. ” Cite No. D151 . Samsung IceTouch was discontinued and no substitute prod There is a need for a better use transparent display uct was made available . 15 technology Sony Ericsson's Xperia Purness phone included a trans 2.B Active Shutters . parent display. D182 . Reviews of the phone are illustrative Active shutters are elements which has the property of of its problems: " Occasionally , a phone exists simply becoming opaque when voltage or other control signal is because it can . Not because it's powerful; not because it applied (or removed ), being otherwise transparent. Active offers amazing bang for the buck ; not because the manu- 20 Shutters may be made using liquid crystal technology. Some facturer negotiated some lucrative exclusivity agreement types of active shutter elements are discussed in the follow with a major carrier , and certainly not because it's introduc ing references: Cite Nos. D059 , D060 , D061, D172 , D173 , ing some groundbreaking new smartphone operating sys and D174 . tem . Yeah — not even Sony Ericsson is attempting to frame AS elements have been used to make various things such or formulate a legitimate business case for the sinfully 25 as: Camera shutters (See Cite Nos . D020 , D021, D022, hedonistic Xperia Pureness , because they don't have to . One D175 ) , Auto - Darkening Welding Helmets (See Cite No. look at that wild transparent display and you'll either be D025 ) , Privacy Windows/ Smart Glass /Privacy Glass /Swit smitten or confused and dismayed by the $ 1,000 sticker chable Glass /Smart Windows/ Switchable Windows ( See price . It's that simple . Anyhow , we've just taken delivery of Cite Nos. D181 , D185 ), Black & White Transparent Dis a Pureness today and have had a few fleeting moments to 30 plays (See Cite No. D040 ) , and Active Shutter Glasses for check it out. If it weren't for the screen , you'd be looking at 3D Television and Computer Games (See Cite Nos . D002 , a $ 30 prepaid candybar here you don't have a camera , a D003 , and D004 ) . high - res display, or any sexy metal or soft -touch bits to give Regarding active shutter glasses , “ [ a ]n active shutter 3D possiblethe phone incarnation a premium of Sonyfeel . Ericsson'sThe UI is typicalabout the dumbphone simplest 35 system (a.k.a. alternate frame sequencing, alternate image, platform , owing largely to the fact that the display's just AI, alternating field , field sequential or eclipse method ) is a grayscale ; you've got an FM radio (with a headset attached ) technique of displaying stereoscopic 3D images. It works by and a basic music player and browser , but if you're in North only presenting the image intended for the left eye while America , don't expect any 3G . ” See Cite No. D182 . Again , blocking the right eye's view , then presenting the righteye the product was discontinued , and no substitute product 40 image while blocking the left eye and repeating this so made available . rapidly that the interruptions do not interfere with the Samsung also recently discontinued larger transparent perceived fusion of the two images into a single 3D image. ” displays it planned to produce citing lack of demand . See Cite No. D002 . Cite No. D176 . (“ Samsung Display announced its 55 " There are several suppliers of Active Shutter elements . mirror and transparent OLED displays back in June 2015, 45 Liquid Crystal Technologies (See Cite No. D058 ) , LCTech and later announced that it will start shipping those displays (See Cite Nos . D023 , DO24 ), and Kent Optoelectronics in Q2 2016. TechRadar today claims that Samsung decided Liquid Crystal Display LCS (See Cite Nos. D039, and not to produce these transparent displays after all, because D041 ). there was not enough demand to justify the investment . This There is a need to use active shutters in new ways that is not confirmed yet , but it may be that Samsung could not 50 enable more beneficial use of transparent displays . make these displays at attractive enough prices due to low It would be beneficial to have a device that utilized a yields and high production costs ." ) transparent display in conjunction with an active shutter that To date there has been little commercial demand for could achieve , for example , any one or combination of the transparent or OLED display panels , because the applica following exemplary benefits : tions have been frivolous. “ While a transparent OLEDs is 55 ( 1 ) maintain user privacy so that an image is not displayed very cool and innovative , it's not clear what are the real on the back of the device when the user wants to applications for this . Having a novelty phone or MP3 player prevent others from seeing what is being done on the with a transparent display is nice, but it's not really useful, device , and also be configured to utilize the back of the is it ? One possible application is embedding such panels in display when desirable ; transparent glass used in windows . I'm not sure if this 60 ( 2 ) display any color state including white, black in any particular OLED is useful for this , but maybe. In any case state of opacity including opaque and transparent; I'm pretty sure some creative designers will find great and ( 3 ) display on an image on one side of the screen in a fun applications for transparent OLEDs. ” See Cite No. proper orientation while obtaining the benefits of D007. As noted in an industry article regarding the Sony increased contrast of such displays with a dark backing , Ericsson Xperia Purness , many applications have been 65 while also enabling the display to be used in a trans frivolous by not improving the usefulness of a device but parentmode for mixed reality and other similar appli have significantly increased the price. cations ; US 10,520,782 B2 5 6 (4 ) display an image on both sides of the screen in a Microsoft's HoloLens is discussed in Cite Nos . D134 proper orientation so that two users can view the device (HoloLens ) , D135 (Windows Holographic ), D136 ( App simultaneously from different sides ; or , Model ) , D137 (App Views) , D138 ( Coordinate systems) , (5 ) any other benefit . D139 (designing for mixed reality ) , D140 (gesture design ) , It would also be beneficial to further combine the trans- 5 D141 ( gaze targeting ) , D142 ( gaze ), D143 ( getting a holo parent display and active shutter device with other technolo graphic space ), D144 ( hardware details ), D145 (hologram ) , gies to improve utility and functionality based on this D146 (mixed reality capture ), D147 ( spatialmapping ) , D148 physical context of the device , usage patterns of the device , (working with holograms) , D149 (offline applications ), and other information . D150 (hologram ) . 10 Google's Project Tango is discussed in Cite Nos . D186 2.0 Other Technologies (setup for all Unity Tango apps ), D187 ( simple augmented Any type of the below technologies , substitutions for reality ) , D188 (placing virtual objects in augmented reality ), those technologies, or new types of technologies with simi D189 (meshing with color) , D190 ( Tango with cardboard ), lar functionality may be used with any form of the invention . D191 (depth perception ) , D192 (motion tracking ) , D193 Touch Sensing and Multi- Touch Sensing are discussed in 15 ( area learning ), and D194 (ASUS ZenFone AR ) . Cite Nos . D082 , D083 ( resistive sensing ) , D064 and D084 Augmented Reality on Apple devices is discussed Cite (capacitive sensing ), D062 (using emissive elements to Nos. D153 (Apple PARk Code ), D203 ( AR IOS Tutorial) , sense light or touch input instead of emit light) , See also , D154 (BRSun Tracker ), D155 (sun position code ) , D156 ( 3D e.g., U.S. Patent Publication 20110057866 ( use the emissive orientation sun position ). Any type of the above technolo elements to sense touch input , or to digitize data ). Edge 20 gies , substitutions for those technologies , or new types of sensors around the phone may also be used to detect hand technologies may be used with any form of the invention . position , or hand position may also be detected using touch Graphics processing and 3D Projection is discussed in screen with any form of the invention . Cite Nos. D208 ( 3D projection ), D209 ( vector projection ) , Determining a user's face or eye position relative to a D210 (mathematics of perspective drawing) , D211 ( image screen , and gaze tracking, are discussed in Cite Nos. D065 , 25 projection ), D212 ( transformation matrix ), D213 ( camera D195 , D196 , D197 , D216 , D217 , D218 , D219 , D171, and matrix ), D214 , D215 (perspective projection ) . U.S. Patents and Publications U.S. Pat . No. 9,495,008 , Virtual reality is discussed in Cite Nos. D161 (Oculus 20150077323 , and 20100125816 . Rift) , D160 (HTC Vive ). Various sensors and related units , algorithms, measure Computer vision is discussed in Cite Nos. D105 ( com ments and technologies are discussed in Cite Nos . D068 , 30 puter vision ), D106 (OpenCV ) , D107 (OpenCV reference D069 (accelerometer ) , D070 (gyroscope ), D071 (magne manual ) , D108 (OpenCV tutorials ), D109 ( OpenCV user tometer) , D072 ( IMU ) , D073 ( quaternion ), D074 (Kalman guide ) , D157 ( object recognition ) . filter ), D075 (proximity sensor ), D076 (photodetector ), Artificial intelligence and machine learning are discussed D077 ( fingerprint recognition ), D110 (motion coprocessor ), in Cite Nos. D102 (artificial intelligence ), D103 (OpenAI ) , D111 ( barometer) , D112 ( compass ) , D198 ( stereo or 3D 35 D104 (OpenAI Gym ). See also Tensorflow and Keras, two camera) , D199 ( range imaging ), D200 ( stereo camera ) , popular libraries for the Python programming language that D201 ( time- of - flight camera ) , D202 (depth perception ), facilitate the creation and training of deep neural networks . D220 ( light field ), D221 ( light field camera) , D222 (Lytro ), Communications technologies are discussed in Cite Nos . D223 (simultaneous localization and mapping) and D113 D085 (cellular network ), D086 (Wi - Fi ) , D087 (Bluetooth ), ( Tango platform ). In particular, the Tango augmented reality 40 D088 (Bluetooth low energy ), D089 ( ISM band ), D090 computing platform incorporates (1 ) Motion Tracking using (Near field communication ), D091 (Radio - frequency iden visual data based on a fisheye camera to identify visual tification ) , D092 (Mesh networking ) . Radio communication features ( feature tracking ), inertial data based on an IMU ; techniques are also discussed in “ The ARRL 2015 Hand ( 2 ) Area Learning by maintaining an area description file so book for Radio Communication .29” that a device can recall if it has been in an area before ; and 45 Sound and voice recognition and synthesis are discussed ( 3 ) Depth Perception using infra -red structured light, time of in Cite Nos . D093 (microphone ) , D094 (microphone array ) , flight, and a stereo camera . D095 (active noise control) , D096 (sound localization ) , Orientation and motion detection are discussed Cite Nos. D097 ( acoustic location ), D098 ( acoustic source localiza D205 , D206 , D207 . tion ), D099 (speech recognition ), D100 (computer speaker) , Position tracking is discussed in Cite Nos. D115 (Global 50 and D101 (speech synthesis ). Positioning System ) , D116 (Galileo Positioning System ), Sub -audible Tones and QR Codes are discussed in Cite D117 (GLONASS ), D118 (BeiDou navigation ), D119 (as Nos. D078 (broadcast automation ), D079 (audience mea sisted GPS ) , D120 (hybrid positioning system ), D121 (GPS surement) , D080 (subaudible tone) , D081 ( QR code ). navigation ), D122 (satellite navigation ), D123 ( indoor posi Phone applications are discussed in Cite Nos. D158 ( text tioning system ), D124 (Bluetooth low energy beacon ) , D125 55 messaging ) , D159 ( camera phone ) . ( iBeacon ), D126 (mobile phone tracking ), D127 ( position Encryption is discussed in Cite Nos. D162 ( encryption ) , ing ), D128 (Wi - Fi positioning) , D129 (proximity market D163 ( symmetric key algorithm ) , D164 (public key cryp ing ), D130 (multilateration ), D204 (Apple Core Location ), tography ) . See also Applicant's prior patents in the family starting with Web services and APIs are discussed in Cite Nos . D165 the U.S. Patent Publication 20080248815 . 60 (HTTP ) , D166 (database ), D167 ( CRUD ), D168 (REST ) , Technology related generally to augmented reality is D169 ( JSON ) , D170 (GeoJSON ) . discussed in Cite Nos . D038 (augmented reality ) , D131 Information relating to circuit design and electrical engi ( optical head mounted display ), D132 ( smartglasses ), D133 neering are discussed in Cite No. D114 (duty cycle ), and in (EPSON -BT300 AR Glasses user guide ), D152 (Layar ). Any the Books “ The Art of Electronics 3rd Edition ,” “Learning type of the above technologies, substitutions for those tech- 65 the Art of Electronics: A Hands -On Lab Course , 1st Edi nologies , or new types of technologies may be used with any tion ,” and “ Make Your Own PCBs with EAGLE : From form of the invention . Schematic Designs to Finished Boards, 1st Edition . ” US 10,520,782 B2 7 8 Programming is generally discussed in the Books “ The given their plain , ordinary , and accustomed meaning to those Definitive Guide to Django : Web Development Done of ordinary skill in the applicable arts . The inventor is fully Right, ” “ Pro Django (Expert's Voice in Web Develop aware that he can be his own lexicographer if desired . The ment) , ” “ Beginning Django E -Commerce , ” “ Django Design inventor expressly elects , as his own lexicographers , to use Patterns, " " Code Complete : A Practical Handbook of Soft- 5 only the plain and ordinary meaning of terms in the speci ware Construction , Second Edition , ” “ Design Patterns: Ele fication and claims unless he clearly states otherwise and ments of Reusable Object -Oriented Software Elements of then further , expressly sets forth the “ special” definition of Reusable Object- Oriented Software ," " Python Geospatial Development, ” “ Machine Learning : An Algorithmic Per that term and explains how it differs from the plain and spective (Chapman & Hall/ Crc Machine Learning & Pattern 10 ordinary meaning . Absent such clear statements of intent to Recognition ). ” apply a “ special ” definition , it is the inventor's intent and In addition , the following documents provide additional desire that the simple , plain and ordinary meaning to the background : U.S. Pat. Nos. 3,037,138 , 5,103,328 , 5,357 , terms be applied to the interpretation of the specification and 355,5,757,139,5,825,543 , 5,881,377,5,986,401, 6,088,079 , claims. 6,288,840 , 6,337,492 , 6,361,838 , 6,433,853 , 6,537,688 , 15 The inventor is also aware of the normal precepts of 6,657,690 , 6,674,504 , 6,689,492 , 6,831,298 , 6,961,178 , English grammar. Thus, if a noun , term , or phrase is 7,030,552 , 7,034,452 , 7,106,001, 7,164,151 , 7,450,105 , intended to be further characterized , specified , or narrowed 7,514,861 , 7,538,841 , 7,637,648 , 7,687,983 , 7,714,904 , in someway , then such noun , term , or phrase will expressly 7,772,765 , 7,804,493 , 7,859,526 , 7,864,270 , 7,916,183 , include additional adjectives, descriptive terms, or other 8,023,080 , 8,059,232 , 8,138,669, 8,157,399, 8,314,542 , 20 modifiers in accordance with the normal precepts of English 8,330,909 , 8,362,992 , 8,724,304 , 8,947,627 , 9,282,614 , grammar . Absent the use of such adjectives, descriptive 9,316,843 , 9,495,008 , and 9,543,364 ; U.S. Publications terms, or modifiers , it is the intent that such nouns, terms, or 20010036013 , 20020015120 , 20020118452, 200201 019 phrases be given their plain , and ordinary English meaning 20030002296 , 20030048522 , 20030074672 , 20030189754 , to those skilled in the applicable arts as set forth above . 20030198456 , 20030210370 , 20040036707 , 20040041800 , 25 In that regard , the use of the word " coupled ” or “ con 20040043139 , 20040070822 , 20040108971 , 20040119407 , nected ” implies that the elements may be directly connected 20040212300 , 20040239243 , 20050012881 , 20050018106 , or may be indirectly connected or coupled through one or 20050024339 , 20050052342 , 20050146492 , 20050213181 , more intervening elements unless it is specifically noted that 20060006792 , 20060066236 , 20060132634 , 20060152812 , there must be a direct connection . 20060176541 , 20060181662 , 20060238684 , 20060268208 , 30 Further, the inventor is fully informed of the standards and 20070002130 , 20070080629 , 20070114916 , 20070120478 , application of the special provisions of 35 U.S.C. $ 112 (f ) . 20070138941, 20070159070 , 20070201248 , 20070274077 , 20080013174 , 20080018732 , 20080062259 , 20080231690 , Thus, the use of the words “ function, ” “means ” or “ step ” in 20080248815 , 20080266500 , 20080297695 , 20080303996 , the Detailed Description or Description of the Drawings or 20090091826 , 20090176029 , 20090201446 , 20090201447 , 35 claims is not intended to somehow indicate a desire to 20090219253 , 20090273275 , 20090278452 , 20090284672 , invoke the special provisions of 35 U.S.C. $ 112 (f ) , to define 20090284690 , 20100013380 , 20100026934 , 20100033461 , the invention . To the contrary , if the provisions of 35 U.S.C. 20100033462, 20100085207 , 20100091207 , 20100115407 , $ 112 ( f ) are sought to be invoked to define the inventions, 20100125816 , 20100165267 , 20100188605 , 20100208110 , the claims will specifically and expressly state the exact 20100233601, 20100252825 , 20100271394, 20100277439, 40 phrases “ means for ” or “ step for , and will also recite the 20110001805 , 20110001808 , 20110038045 , 20110057866 , word “ function ” ( i.e., will state “ means for performing the 20110090324 , 20110096156 , 20110157708 , 20110195354 , function of [ insert function ]” ) , without also reciting in such 20110195758 , 20110227487 , 20110255303 , 20110273906 , phrases any structure , material or act in support of the 20120049723 , 20120057006 , 20120081524 , 20120099033 , function . Thus , even when the claims recite a “means for 20120188637, 20120320287 , 20150077323 , 20150155344 , 45 performing the function of ... ” or “ step for performing the 20150295015 , 20150349293 , and 20160027859 ; and For function of ... , ” if the claims also recite any structure , eign References CN105098087A , DE102014111484A1 , material or acts in support of that means or step , or that EP2211397A3 , JP2000098386A , WO2009100055A2 , perform the recited function , then it is the clear intention of WO2009100055A3 , WO2009125918A2 , the inventor not to invoke the provisions of 35 U.S.C. § WO2009125918A3 , WO2015032901A3 , and 50 112 (f ) . Moreover , even if the provisions of 35 U.S.C. $ WO2016164184A1. 112 ( f ) are invoked to define the claimed inventions, it is Applicant believes that the material incorporated above is intended that the inventions not be limited only to the “ non - essential” in accordance with 37 CFR 1.57 , because it specific structure , material or acts that are described in the is referred to for purposes of indicating the background of preferred embodiments , but in addition , include any and all the invention or illustrating the state of the art. However, if 55 structures , materials or acts that perform the claimed func the Examiner believes that any of the above - incorporated tion as described in alternative embodiments or forms of the material constitutes “ essential material ” within the meaning invention , or that are well known present or later -developed , of 37 CFR 1.57 (c )( 1) - ( 3 ), Applicant will amend the speci equivalent structures , material or acts for performing the fication to expressly recite the essential material that is claimed function . incorporated by reference as allowed by the applicable rules . 60 Headings, sections, and other similar designations are provided for the convenience of the reader , and should not SUMMARY OF THE INVENTION be used to limit, divide, or partition the teachings of the variously claimed aspects of the inventions. Aspects and applications of the invention presented here The aspects , features , and advantages will be apparent to are described below in the drawings and detailed description 65 those artisans of ordinary skill in the art from the of the invention . Unless specifically noted , it is intended that DETAILED DESCRIPTION and DRAWINGS , and from the words and phrases in the specification and the claims be the CLAIMS. US 10,520,782 B2 9 10 However , without attempting to characterize or limit the It is yet another (and optionally independent) object of the scope of inventions as they are described and claimed , some invention to operate a display in a presentation mode such of the advantages of the various inventions are summarized that a presentation view is displayed on one side of the below . screen in the correct orientation for observers, and a pre It is an object of the invention to implement transparent 5 senter view is shown on the other side of the screen . display technology It is yet another ( and optionally independent) object of the It is yet another (and optionally independent) object of the invention to operate the display in an augmented reality state invention to maintain information privacy for users of or mode or mixed reality state or mode so that annotations transparent displays when desired . may appear to be placed over real -world objects . 10 It is yet another ( and optionally independent) object of the It is yet another ( and optionally independent) object of the invention to operate the display in virtual reality state or invention to allow transparent displays to have a black state mode . in addition to a transparent or translucent “ clear ” state. It is yet another ( and optionally independent) object of the It is yet another (and optionally independent) objectof the invention to operate the display in an unobstructed state or invention to use transparent displays to display information 15 mode so that the screen displays information to at least one in ways that appear in the correct orientation to users on both user but also appears translucent to at least a user on the sides of the screen . other side of the screen . It is yet another ( and optionally independent) object of the It is yet another (and optionally independent) object of the invention to utilize active shutters in conjunction with invention to operate the display in tracing mode such that a transparent display elements . 20 user may be able to use a pen or stylus to trace an object It is yet another ( and optionally independent) object of the underneath the display onto the display . invention to create a display from a combination of active It is yet another (and optionally independent) object of the shutter elements and transparent display elements . invention to operate the display in scanning mode such that It is yet another (and optionally independent) object of the an object placed adjacent to the display can be scanned or invention to active shutter devices to be either transparent or 25 digitized . opaque as necessary to achieve the visual effects described It is yet another (and optionally independent) object of the herein . invention to operate the display according to any other state It is yet another (and optionally independent) object of the or mode . invention to have active shutters with multiple elements so It is yet another ( and optionally independent) object of the that different areas of an emissive display can be driven on 30 invention to create devices utilizing the display technology one side , the other side, or the display can appear transparent in any one of many form factors . for augmented reality or HUD -like features . It is yet another ( and optionally independent) object of the It is yet another (and optionally independent) object of the invention to utilize the display in a single screen device . invention to provide touch sensing on one side of the screen . It is yet another (and optionally independent) objectof the It is yet another ( and optionally independent) object of the 35 invention to utilize a display in a hinged device such that invention to provide touch sensing on both sides of the there is a display on one side of the hinge, and optionally screen for input to one user interface . another display on the other side of the hinge . It is yet another (and optionally independent) object of the It is yet another (and optionally independent) object of the invention to provide touch sensing on both sides of the invention to operate a hinged device in a normal display screen for input to two separate user interfaces that are 40 mode such that it behaves as a traditional laptop . displayed on different sides of the screen . It is yet another ( and optionally independent) object of the It is yet another (and optionally independent) object of the invention to operate the hinged device in stand mode such invention to utilize the emissive display elements to sense that one side of the hinge acts as a stand while the other side user input on the screen . is angled upward . It is yet another (and optionally independent) object of the 45 It is yet another ( and optionally independent) object of the invention to utilize the emissive display elements in com invention to operate the device in tent mode such that the bination with the active shutter elements to sense user input hinge of the device is off the table . from one side of the screen but not the other . Etc. It is yet another (and optionally independent) object It is yet another ( and optionally independent) object of the of the invention to operate a hinged device in any other invention to utilize the emissive display elements alone or in 50 mode such as dual display mode, presentation mode , or any combination with the active shutter elements to digitize other mode described herein . physical information that is placed adjacent to the screen It is yet another (and optionally independent) object of the ( i.e. “ scan ” a document) . invention to determine face position relative to the screen . It is yet another and optionally independent) object of the It is yet another (and optionally independent) object of the invention to implement the transparent display elements 55 invention to determine the user's eye position or gaze with active shutter elements and touch sensing technology . relative to the screen . It is yet another (and optionally independent) object of the It is yet another ( and optionally independent) object of the invention to implement different display modes on a screen invention to utilize sensors to determine the physical state of capable of single - sided , dual -sided , or transparent display . the device and select the display screen orientation and It is yet another (and optionally independent) object of the 60 display mode. invention to operate in a normal mode by operating a It is yet another (and optionally independent) object of the transparent display with at least one active shutter in an invention to determine the position of the device incorpo opaque state such that it appears as a single sided display. rating the display to determine the display mode and content It is yet another (and optionally independent ) object of the to be display invention to operate a display in a sharing mode such that the 65 It is yet another (and optionally independent) object of the same or similar information appears on both sides of a invention to utilize computer vision techniques in conjunc display in the correct orientation for each respective user . tion with the display . US 10,520,782 B2 11 12 It is yet another ( and optionally independent) object of the It is yet another ( and optionally independent) object of the invention to utilize artificial intelligence and machine learn invention to improve the ability to share references to real ing in conjunction with the display. world content in messaging applications . It is yet another ( and optionally independent) object of the At least one of the above listed , unlisted , and other objects invention to use of any type of neural network including but 5 of the invention may be achieved by a display element that not limited to : feed forward , recurrent , convolutional , long includes an active shutter and a transparent display pixel short -term memory (LSTM ) in conjunction with the display. wherein the active shutter is independently electrically oper It is yet another ( and optionally independent) object of the able to transition from a substantially transparent state to a substantially opaque state , wherein the transparent display invention to utilize behavioral cloning neural network tech 10 pixel substantially transparent and is independently electri niques in conjunction with the display . cally operable to emit light, and wherein the active shutter It is yet another ( and optionally independent) object of the is placed substantially adjacent to one side of the transparent invention to utilize the display for augmented reality appli display pixel. cations. It is another (and optionally independent) object of the 15 of Atthe leastinvention one of maythe above be achieved listed , unlisted by display , and otherelement objects that invention to create displays and devices that can be used for includes a first active shutter, a transparent display pixel , and augmented reality (or mixed reality ) applications without a a second active shutter, wherein the first and second active user needing to wear a special device on their head . shutters are independently electrically operable to transition It is another ( and optionally independent) object of the from a substantially transparent state to a substantially invention to create displays and devices that can be used for 20 opaque state , wherein the transparent display pixel is sub augmented reality (or mixed reality ) applications when the stantially transparent and is independently electrically oper user looks through the device , but change physical visual able to emit light, and wherein the first active shutter is states to be opaque or translucent for applications where the placed on one side of the transparent display pixel and the user is looking at the device . second active shutter is placed on the opposite side of the It is yet another ( and optionally independent) object of the 25 transparent display pixel . invention to track the user's head position relative to the At least one of the above listed , unlisted , and other objects display, to track the physical location and orientation of the of the invention may be achieved by display element includ display , and to place an augmented reality annotation on the ing a first transparent display pixel ; an active shutter ; and , a display based on both pieces of information . second transparent display pixel; wherein the active shutter It is yet another (and optionally independent) object of the 30 is independently electrically operable to transition from a invention to utilize a dual- sided multi -mode display to create substantially transparent state to a substantially opaque state , applications that show relevant information to users on both wherein the first and second transparent display pixels are sides of a device . substantially transparent and are independently electrically It is yet another (and optionally independent) object of the operable to emit light, and wherein the active shutter is invention to utilize a multi- mode display to create applica- 35 placed between the first and second transparent display tions that show relevant augmented reality annotations users pixels . of a device . It is yet another ( and optionally independent) object of the BRIEF DESCRIPTION OF THE SEVERAL invention to utilize a multi -mode display to create applica VIEWS OF THE DRAWINGS tions that utilize multiple display modes based on the 40 context of the use of the device . A more complete understanding of the present invention It is yet another (and optionally independent ) object of the may be derived by referring to the detailed description when invention to utilize a multi -mode display to create applica considered in connection with the following illustrative tions that switch between a normal display mode and a figures . In the figures, like reference numbers refer to like virtual reality display mode . 45 elements or acts throughout the figures. It is yet another (and optionally independent) object of the Screen Element Structure for a Shutter -Display -Shutter invention to utilize a multi -mode display to create applica (SDS ) Display Element. tions that are able to block the sun from the perspective of FIG . 1A depicts a separated block diagram of hardware a user using active shutters . used by some forms of the invention including two active It is yet another ( and optionally independent) object of the 50 shutters , one transparent display screen , and two touch invention to utilize a multi- mode display to create applica sensing components . tions that are able to block the sun from the perspective of FIG . 1B depicts a joined block diagram of hardware used a user using active shutters while showing augmented reality by some forms of the invention including two active shut annotations . ters , one transparent display screen , and two touch sensing It is yet another ( and optionally independent) object of the 55 components . invention to utilize the display in virtual reality applications . FIG . 1C depicts a separated block diagram of hardware It is another ( and optionally independent) object of the used by some forms of the invention including two active invention to create displays and devices that can be used for shutters and one transparent display screen . virtual reality applications by making the screen opaque and FIG . ID depicts a separated block diagram of hardware displaying an appropriate image based on the position and 60 used by some forms of the invention including one active other sensed or known information about the device state . shutter and one transparent display screen . It is yet another ( and optionally independent) object of the Screen Element Structure for a Display - Shutter- Display invention to facilitate sharing and viewing of specialized (DSD ) Display Element . sensor data sent via messaging applications. FIG . 2A depicts a separated block diagram of hardware It is yet another (and optionally independent) object of the 65 used by some forms of the invention including two trans invention to improve the relevance of notifications presented parent display screens, one active shutter , and two touch by messaging applications. sensing panels . US 10,520,782 B2 13 14 FIG . 2B depicts a joined block diagram ofhardware used observer attempting to view the screen from the right by some forms of the invention including two transparent ( TD.ONR ), and the right active shutter is opaque display screens, one active shutter , and two touch sensing ( R :AS.O ) . panels . FIG . 3J depicts a simplified side view illustration of a FIG . 2C depicts a separated block diagram of hardware 5 shutter- display -shutter element used by some forms of the used by some forms of the invention including two trans invention when the left active shutter is transparent parent display screens, and one active shutter. ( L : AS.T ), the elements of the transparent display are illu FIG . 2D depicts a separated block diagram of hardware minated with an image that is properly oriented to an used by some forms of the invention including two display observer attempting to view the screen from the left screens . 10 (< TD.ONL ), and the right active shutter is transparent Screen Element States for a SDS Display Element. (R : AS.T ). FIG . 3A depicts a simplified side view illustration of a FIG . 3K depicts a simplified side view illustration of a shutter - display -shutter element used by some forms of the shutter -display - shutter element used by some forms of the invention when the left active shutter is opaque (L :AS.O ), invention when the left active shutter is transparent the elements of the transparent display are illuminated with 15 (L : AS.T ), the elements of the transparent display are off an image that is properly oriented to an observer attempting ( TD.OFF ) , and the right active shutter is transparent to view the screen from the left ( < TD.ONL ), and the right (R : AS.T ) . active shutter is transparent (R :AS.T ) . FIG . 3L depicts a simplified side view illustration of a FIG . 3B depicts a simplified side view illustration of a shutter - display -shutter element used by some forms of the shutter -display - shutter element used by some forms of the 20 invention when the left active shutter is transparent invention when the left active shutter is opaque ( L : AS.O ), ( L :AS.T ), the elements of the transparent display are illu the elements of the transparent display are off ( TD.OFF ), minated with an image that is properly oriented to an and the right active shutter is transparent. observer attempting to view the screen from the right FIG . 3C depicts a simplified side view illustration of a ( TD.ONR > ), and the right active shutter is transparent shutter- display - shutter element used by some forms of the 25 ( R : AS.T) . invention when the left active shutter is opaque ( L :AS.O ) , Screen Element States for DSD Display Elements . the elements of the transparent display are illuminated with FIG . 4A depicts a simplified side view illustration of a an image that is properly oriented to an observer attempting display - shutter -display element used by some forms of the to view the screen from the right ( TD.ONR ) , and the right invention when the elements of the left transparent display active shutter is transparent ( R : AS.T ) . 30 are illuminated with an image that is properly oriented to an FIG . 3D depicts a simplified side view illustration of a observer attempting to view the screen from the left shutter- display - shutter element used by some forms of the ( L : < TD.ONL ), the active shutter is transparent ( AS.T ), and invention when the left active shutter is opaque ( L :AS.O ) , the elements of the transparent display are off ( R : TD.OFF ) . the elements of the transparent display are illuminated with FIG . 4B depicts a simplified side view illustration of a an image that is properly oriented to an observer attempting 35 display -shutter -display element used by some forms of the to view the screen from the left ( < TD.ONL ) , and the right invention when the elements of the left transparent display active shutter is opaque (R : AS.O ). are illuminated with an image that is properly oriented to an FIG . 3E depicts a simplified side view illustration of a observer attempting to view the screen from the left shutter- display -shutter element used by some forms of the ( L : < TD.ONL ) , the active shutter is opaque ( AS.O ) , and the invention when the left active shutter is opaque ( L : AS.O ) , 40 elements of the transparent display are off ( R : TD.OFF ). the elements of the transparent display are off ( TD.OFF ) , FIG . 4C depicts a simplified side view illustration of a and the right active shutter is opaque ( R :AS.O ) . display - shutter -display element used by some forms of the FIG . 3F depicts a simplified side view illustration of a invention when the elements of the left transparent display shutter -display -shutter element used by some forms of the are illuminated with an image that is properly oriented to an invention when the left active shutter is opaque ( L : AS.O ) , 45 observer attempting to view the screen from the right the elements of the transparent display are illuminated with ( L : TD.ONR ) , the active shutter is transparent ( AS.T ) , and an image that is properly oriented to an observer attempting the elements of the transparent display are off (R : TD.OFF ). to view the screen from the right ( TD.ONL > ), and the right FIG . 4D depicts a simplified side view illustration of a active shutter is opaque (R :AS.O ) . display - shutter - display element used by some forms of the FIG . 3G depicts a simplified side view illustration of a 50 invention when the elements of the left transparent display shutter- display - shutter element used by some forms of the are illuminated with an image that is properly oriented to an invention when the left active shutter is transparent observer attempting to view the screen from the right ( L :AS.T ), the elements of the transparent display are illu ( L : TD.ONR ), the active shutter is opaque (AS.O ), and the minated with an image that is properly oriented to an elements of the transparent display are off ( R : TD.OFF ) . observer attempting to view the screen from the left 55 FIG . 4E depicts a simplified side view illustration of a ( < TD.ONL ) , and the right active shutter is opaque display - shutter -display element used by some forms of the ( R :AS.O ) . invention when the elements of the left transparent display FIG . 3H depicts a simplified side view illustration of a are illuminated with an image that is properly oriented to an shutter -display -shutter element used by some forms of the observer attempting to view the screen from the left invention when the left active shutter is transparent 60 (L :< TD.ONL ), the active shutter is transparent (AS.T ) , and ( L : AS.T ) , the elements of the transparent display are off the elements of the transparent display are illuminated with ( TD.OFF ), and the right active shutter is opaque ( R : AS.O ) . an image that is properly oriented to an observer attempting FIG . 31 depicts a simplified side view illustration of a to view the screen from the left (R :< TD.ONL ). shutter -display - shutter element used by some forms of the FIG . 4F depicts a simplified side view illustration of a invention when the left active shutter is transparent 65 display - shutter - display element used by some forms of the ( L :AS.T ), the elements of the transparent display are illu invention when the elements of the left transparent display minated with an image that is properly oriented to an are illuminated with an image that is properly oriented to an US 10,520,782 B2 15 16 observer attempting to view the screen from the left FIG . 4N depicts a simplified side view illustration of a ( L : < TD.ONL ), the active shutter is opaque (AS.O ) , and the display - shutter- display element used by some forms of the elements of the transparent display are illuminated with an invention when the elements of the left transparent display image that is properly oriented to an observer attempting to are off (L : TD.OFF ) , the active shutter is opaque (AS.O ), and view the screen from the left (R : < TD.ONL ). 5 the elements of the transparent display are illuminated with FIG . 4G depicts a simplified side view illustration of a an image that is properly oriented to an observer attempting display - shutter - display element used by some forms of the to view the screen from the left (R : < TD.ONL ). invention when the elements of the left transparent display FIG . 40 depicts a simplified side view illustration of a are illuminated with an image that is properly oriented to an display - shutter -display element used by some forms of the observer attempting to view the screen from the left 10 invention when the elements of the left transparent display ( L : < / TD.ONL ), the active shutter is transparent (AS.T ) , and are off ( L : TD.OFF ) , the active shutter is transparent (AS.T ) , the elements of the transparent display are illuminated with and the elements of the transparent display are illuminated an image that is properly oriented to an observer attempting with an image that is properly oriented to an observer to view the screen from the right (R : TD.ONR » ). 15 attempting to view the screen from the right (R : TD.ONR ). FIG . 4H depicts a simplified side view illustration of a FIG . 4P depicts a simplified side view illustration of a display - shutter -display element used by some forms of the display - shutter -display element used by some forms of the invention when the elements of the left transparent display invention when the elements of the left transparent display are illuminated with an image that is properly oriented to an are off ( L : TD.OFF ) , the active shutter is opaque (AS.O ) , and observer attempting to view the screen from the left 20 the elements of the transparent display are illuminated with ( L : -- TD.ONL ) , the active shutter is opaque ( AS.O ), and the an image that is properly oriented to an observer attempting elements of the transparent display are illuminated with an to view the screen from the right ( R : TD.ONR ) . image that is properly oriented to an observer attempting to FIG . 4Q depicts a simplified side view illustration of a view the screen from the right ( R : TD.ONR ). display- shutter -display element used by some forms of the FIG . 41 depicts a simplified side view illustration of a 25 invention when the elements of the left transparent display display -shutter -display element used by some forms of the are off ( L : TD.OFF ) , the active shutter is transparent ( AS.T ) , invention when the elements of the left transparent display and the elements of the transparent display are off (R.TD are illuminated with an image that is properly oriented to an .OFF ) . observer attempting to view the screen from the right FIG . 4R depicts a simplified side view illustration of a ( L : TD.ONR ) , the active shutter is transparent (AS.T ) , and 30 display - shutter - display element used by some forms of the the elements of the transparent display are illuminated with invention when the elements of the left transparent display an image that is properly oriented to an observer attempting are off ( L : TD.OFF ), the active shutter is opaque (AS.O ) , and to view the screen from the left ( R : < TD.ONL ) . the elements of the transparent display are off ( R : TD.OFF ) . FIG . 4J depicts a simplified side view illustration of a Display Modes Using the Various Display Element States . display - shutter -display element used by some forms of the 35 FIG . 5A depicts an exemplary simplified flow diagram of invention when the elements of the left transparent display a display control loop used with some forms of the invention are illuminated with an image that is properly oriented to an incorporating a shutter -display - shutter element when the observer attempting to view the screen from the right element is configured to operate in a normal mode for an ( L : TD.ONR ), the active shutter is opaque (AS.O ) , and the observer to the left as depicted . elements of the transparent display are illuminated with an 40 FIG . 5B depicts an exemplary simplified flow diagram of image that is properly oriented to an observer attempting to a display control loop used with some formsof the invention view the screen from the left ( R : < TD.ONL ) . incorporating a shutter - display - shutter element when the FIG . 4K depicts a simplified side view illustration of a element is configured to operate in a normal mode for an display - shutter -display element used by some forms of the observer to the right as depicted . invention when the elements of the left transparent display 45 FIG . 5C depicts an exemplary simplified flow diagram of are illuminated with an image that is properly oriented to an a display control loop used with some forms of the invention observer attempting to view the screen from the right incorporating a shutter- display - shutter element when the ( L : TD.ONR ), the active shutter is transparent (AS.T ) , and element is configured to operate in a presentation mode or the elements of the transparent display are illuminated with sharing mode. an image that is properly oriented to an observer attempting 50 FIG . 5D depicts another exemplary simplified flow dia to view the screen from the right ( R : TD.ONR ) . gram of a display control loop used with some forms of the FIG . 4L depicts a simplified side view illustration of a invention incorporating a shutter - display -shutter element display -shutter -display element used by some forms of the when the element is configured to operate in a presentation invention when the elements of the left transparent display mode or a sharing mode . are illuminated with an image that is properly oriented to an 55 FIG . 5E depicts an exemplary simplified flow diagram of observer attempting to view the screen from the right a display control loop used with some forms of the invention ( L : TD.ONR ), the active shutter is opaque (AS.O ), and the incorporating a shutter -display - shutter element when the elements of the transparent display are illuminated with an element is configured to operate in augmented reality mode image that is properly oriented to an observer attempting to for an observer to the left as depicted . view the screen from the right ( R : TD.ONR ) . 60 FIG . 5F depicts an exemplary simplified flow diagram of FIG . 4M depicts a simplified side view illustration of a a display control loop used with some forms of the invention display -shutter - display element used by some forms of the incorporating a shutter- display - shutter element when the invention when the elements of the left transparent display element is configured to operate in an augmented reality are off (L : TD.OFF ), the active shutter is transparent (AS.T ), mode for an observer to the right as depicted . and the elements of the transparent display are illuminated 65 FIG . 5G depicts an exemplary simplified flow diagram of with an image that is properly oriented to an observer a display control loop used with some forms of the invention attempting to view the screen from the left (R : < TD.ONL ). incorporating a shutter- display -shutter element when the US 10,520,782 B2 17 18 element is configured to operate in a first unobstructed mode element is configured to operate in a two -and - a -half dimen for an observer to the right as depicted . sional augmented reality mode for an observer to the left as FIG . 5H depicts an exemplary simplified flow diagram of depicted . a display control loop used with some forms of the invention FIG . 6J depicts an exemplary simplified flow diagram of incorporating a shutter - display -shutter element when the 5 a display control loop used with some formsof the invention element is configured to operate in an unobstructed mode for incorporating a display - shutter -display element when the element is configured to operate in an unobstructed mode for an observer to the left as depicted . an observer to the right as depicted . FIG . 51 depicts another exemplary simplified flow dia FIG . 6K depicts an exemplary simplified flow diagram of gram of a display control loop used with some forms of the 10 a display control loop used with some formsof the invention invention incorporating a shutter- display - shutter element incorporating a display - shutter -display element when the when the element is configured to operate in an unobstructed element is configured to operate in an unobstructed mode for mode for an observer to the right as depicted . an observer to the left as depicted . FIG . 5J depicts another exemplary simplified flow dia FIG . 6L depicts an exemplary simplified flow diagram of gram of a display control loop used with some forms of the 15 a display controlloop used with some formsof the invention invention incorporating a shutter- display - shutter element incorporating a display - shutter -display element when the when the element is configured to operate in an unobstructed element is configured to operate in an unobstructed mode . mode for an observer to the left as depicted . FIG . 6M depicts another exemplary simplified flow dia FIG . 5K depicts an exemplary simplified flow diagram of gram of a display control loop used with some forms of the a display control loop used with some forms of the invention 20 invention incorporating a display -shutter -display element incorporating a shutter- display -shutter element when the when the element is configured to operate in an unobstructed element is configured to operate in an unobstructed mode. mode . FIG . 6A depicts an exemplary simplified flow diagram of FIG . 7A depicts an exemplary simplified vertical time a display control loop used with some forms of the invention series illustration of a display control loop used with some incorporating a display - shutter -display element when the 25 forms of the invention incorporating a shutter- display - shut element is configured to operate in a normal mode for an ter element when the element is configured to operate in a observer to the left as depicted . normal mode for an observer to the left as depicted . FIG . 6B depicts an exemplary simplified flow diagram of FIG . 7B depicts an exemplary simplified vertical time a display control loop used with some forms of the invention series illustration of a display control loop used with some incorporating a display- shutter -display element when the 30 terforms element of the when invention the elementincorporating is configured a shutter to- display operate - shut in a element is configured to operate in a normal mode for an normal mode for an observer to the right as depicted . observer to the right as depicted . FIG . 7C depicts an exemplary simplified vertical time FIG . 6C depicts an exemplary simplified flow diagram of series illustration of a display control loop used with some a display control loop used with some forms of the invention 35 forms of the invention incorporating a shutter- display - shut incorporating a display- shutter -display element when the ter element when the element is configured to operate in a element is configured to operate in a presentation mode or a presentation mode or a sharing mode . sharing mode. FIG . 7D depicts another exemplary simplified vertical FIG . 6D depicts an exemplary simplified flow diagram of time series illustration of a display control loop used with a display control loop used with some forms of the invention 40 some forms of the invention incorporating a shutter -display incorporating a display- shutter -display element when the shutter element when the element is configured to operate in element is configured to operate in an augmented reality a presentation mode or a sharing mode . mode for an observer to the right as depicted . FIG . 7E depicts an exemplary simplified vertical time FIG . 6E depicts another exemplary simplified flow dia series illustration of a display control loop used with some gram of a display control loop used with some forms of the 45 forms of the invention incorporating a shutter- display - shut invention incorporating a display - shutter - display element ter element when the element is configured to operate in an when the element is configured to operate in an augmented augmented reality mode for an observer to the left as reality mode for an observer to the right as depicted . depicted . FIG . 6F depicts an exemplary simplified flow diagram of FIG . 7F depicts an exemplary simplified vertical time a display control loop used with some forms of the invention 50 series illustration of a display control loop used with some incorporating a display - shutter -display element when the forms of the invention incorporating a shutter -display - shut element is configured to operate in a two - and - a - half dimen ter element when the element is configured to operate in an sional augmented reality mode for an observer to the right as augmented reality mode for an observer to the right as depicted . depicted . FIG . 6G depicts an exemplary simplified flow diagram of 55 FIG . 76 depicts an exemplary simplified vertical time a display control loop used with some formsof the invention series illustration of a display control loop used with some incorporating a display- shutter -display element when the forms of the invention incorporating a shutter -display - shut element is configured to operate in an augmented reality ter element when the element is configured to operate in an mode for an observer to the left as depicted . unobstructed mode for an observer to the right as depicted . FIG . 6H depicts another exemplary simplified flow dia- 60 FIG . 7H depicts an exemplary simplified vertical time gram of a display control loop used with some forms of the series illustration of a display control loop used with some invention incorporating a display - shutter- display element forms of the invention incorporating a shutter- display - shut when the element is configured to operate in an augmented ter element when the element is configured to operate in an reality mode for an observer to the left as depicted . unobstructed mode for an observer to the left as depicted . FIG . 61 depicts an exemplary simplified flow diagram of 65 FIG . 71 depicts another exemplary simplified vertical time a display control loop used with some forms of the invention series illustration of a display control loop used with some incorporating a display - shutter -display element when the forms of the invention incorporating a shutter -display - shut US 10,520,782 B2 19 20 ter element when the element is configured to operate in an play element when the element is configured to operate in an unobstructed mode for an observer to the right as depicted . unobstructed mode for an observer to the right as depicted . FIG . 7J depicts another exemplary simplified vertical time FIG . 8K depicts an exemplary simplified vertical time series illustration of a display control loop used with some series illustration of a display control loop used with some forms of the invention incorporating a shutter- display - shut- 5 forms of the invention incorporating a display - shutter- dis ter element when the element is configured to operate in an play element when the element is configured to operate in an unobstructed mode for an observer to the left as depicted . unobstructed mode for an observer to the left as depicted . FIG . 7K depicts an exemplary simplified vertical time FIG . 8L depicts an exemplary simplified vertical time formsseries illustrationof the invention of a displayincorporating control a loopshutter used -display with -someshut- 10 formsseries illustrationof the invention of a displayincorporating control aloop display used - shutter with some -dis ter element when the element is configured to operate in an play element when the element is configured to operate in an unobstructed mode . unobstructed mode . FIG . 8A depicts an exemplary simplified vertical time FIG . 8M depicts another exemplary simplified vertical series illustration of a display control loop used with some time series illustration of a display control loop used with forms of the invention incorporating a display - shutter- dis 15 some forms of the invention incorporating a display - shutter play element when the element is configured to operate in a display element when the element is configured to operate in normal mode for an observer to the left as depicted . an unobstructed mode . FIG . 8B depicts an exemplary simplified vertical time Single SDS Screen Form Factor Configurations . series illustration of a display control loop used with some 20 FIG . 9A depicts an exemplary simplified illustration used forms of the invention incorporating a display - shutter -dis with some forms of the invention incorporating a shutter play element when the element is configured to operate in a display - shutter screen movably connected to an opaque normal mode for an observer to the right as depicted . object such as a computer housing or keyboard in an upright FIG . 8C depicts an exemplary simplified vertical time laptop configuration . series illustration of a display control loop used with some 25 FIG . 9B depicts an exemplary simplified illustration used forms of the invention incorporating a display - shutter- dis with some forms of the invention incorporating a shutter play element when the element is configured to operate in a display - shutter screen movably connected to an opaque presentation mode or a sharing mode. object such as a computer housing or keyboard in a stand FIG . 8D depicts an exemplary simplified vertical time position laptop configuration . series illustration of a display control loop used with some 30 forms of the invention incorporating a display - shutter -dis FIG . 9C depicts an exemplary simplified illustration used play elementwhen the element is configured to operate in an with some forms of the invention incorporating a shutter augmented reality mode for an observer to the right as display - shutter screen movably connected to an opaque depicted . object such as a computer housing or keyboard in a tent FIG . 8E depicts another exemplary simplified vertical 35 position laptop configuration . time series illustration of a display control loop used with FIG . 9D depicts an exemplary simplified illustration used some forms of the invention incorporating a display- shutter with some forms of the invention incorporating a shutter display element when the element is configured to operate in display -shutter screen movably connected to an opaque an augmented reality mode for an observer to the right as object such as a computer housing or keyboard in a tablet depicted . 40 position laptop configuration . FIG . 8F depicts an exemplary simplified vertical time FIG . 9E depicts an exemplary simplified illustration used series illustration of a display control loop used with some with some forms of the invention incorporating a shutter forms of the invention incorporating a display - shutter -dis display - shutter screen positioned relative to two reflecting play elementwhen the element is configured to operate in a surfaces in an extra wide display configuration . two- and -a -half dimensional augmented reality mode for an 45 FIG . 9F depicts an exemplary simplified illustration used observer to the right as depicted . with some forms of the invention incorporating a single FIG . 8G depicts an exemplary simplified vertical time shutter -display - shutter screen in a glasses or goggles con series illustration of a display control loop used with some figuration . forms of the invention incorporating a display - shutter- dis Single SD (or DS ) Screen Form Factor Configurations. play elementwhen the element is configured to operate in an 50 FIG . 9G depicts an exemplary simplified illustration used augmented reality mode for an observer to the left as with some forms of the invention incorporating a single depicted . shutter - display screen connected to an object via a hinge in FIG . 8H depicts another exemplary simplified vertical an upright laptop configuration . time series illustration of a display control loop used with FIG . 9H depicts an exemplary simplified illustration used some forms of the invention incorporating a display - shutter- 55 with some forms of the invention incorporating a shutter display element when the element is configured to operate in display screen movably connected to an opaque object such an augmented reality mode for an observer to the left as as a computerhousing or keyboard in a stand position laptop depicted . configuration . FIG . 81 depicts an exemplary simplified vertical time FIG . 91 depicts an exemplary simplified illustration used series illustration of a display control loop used with some 60 with some forms of the invention incorporating a shutter forms of the invention incorporating a display - shutter- dis display screen movably connected to an opaque object such play elementwhen the element is configured to operate in a as a computer housing or keyboard in a tent position laptop two -and -a -half dimensional augmented reality mode for an configuration . observer to the left as depicted . FIG . 9J depicts an exemplary simplified illustration used FIG . 8J depicts an exemplary simplified vertical time 65 with some forms of the invention incorporating a shutter series illustration of a display control loop used with some display screen movably connected to an opaque object such forms of the invention incorporating a display - shutter -dis as a computer housing or keyboard in a tablet configuration . US 10,520,782 B2 21 22 Single DSD Screen Form Factor Configurations. FIG . 12D depicts an exemplary simplified illustration FIG . 10A depicts an exemplary simplified illustration used with some forms of the invention incorporating dual used with some forms of the invention incorporating a display - shutter -display screens movably connected in a display -shutter -display screen movably connected to an small tablet configuration . opaque object such as a computer housing or keyboard in an 5 FIG . 12E depicts an exemplary simplified illustration upright laptop configuration . used with some forms of the invention incorporating dual FIG . 10B depicts an exemplary simplified illustration display - shutter -display screens movably connected in a used with some forms of the invention incorporating a large tablet configuration . While the hinge is depicted below display - shutter- display screen movably connected to an the screen in the simplified illustration , it may be positioned opaque object such as a computer housing or keyboard in a 10 between the two screens so that the device is able to lay flat stand position laptop configuration . on a table or other surface . FIG . 10C depicts an exemplary simplified illustration Device Context Determination . used with some forms of the invention incorporating a FIG . 13 is a table depicting different device states based display- shutter- display screen movably connected to an 15 onof thea detected user's facescreen . orientation and the detected orientation opaque object such as a computer housing or keyboard in a FIG . 14 depicts one form of face orientation detection . tent position laptop configuration . FIG . 15 is a table depicting different relative observer FIG . 10D depicts an exemplary simplified illustration position detection states. used with some forms of the invention incorporating a Exemplary Devices. display -shutter -display screen movably connected to an 20 FIG . 16A depicts an exemplary single screen device . opaque object such as a computer housing or keyboard in a FIG . 16B depicts another exemplary single screen device . tablet position laptop configuration . FIG . 17A depicts an exemplary multi - screen device in a FIG . 10E depicts an exemplary simplified illustration small tablet or closed configuration . used with some forms of the invention incorporating a FIG . 17B depicts an exemplary multi - screen device in a display -shutter -display screen positioned relative to two 25 large tablet configuration . reflecting surfaces in an extra wide display configuration . Perspective Correction for See- Through Displays in Aug FIG . 10F depicts an exemplary simplified illustration used mented Reality Applications. with some forms of the invention incorporating a single FIG . 18A depicts the simplified geometry of a device display - shutter - display screen in a glasses or goggles con reproducing an object on a screen without accounting for the figuration . 30 perspective of the user. Dual SDS Screen Form Factor Configurations . FIG . 18B depicts a view of a device screen from the FIG . 11A depicts an exemplary simplified illustration perspective of a user with an annotation centered over an used with some forms of the invention incorporating dual image of an object. shutter- display -shutter screens movably connected in an FIG . 18C depicts the simplified geometry of a device upright laptop configuration . 35 reproducing an object on a screen while accounting for the FIG . 11B depicts an exemplary simplified illustration perspective of the user based on the determined position of used with some forms of the invention incorporating dual the user relative to the screen . shutter - display -shutter screensmovably connected in a stand FIG . 18D depicts a view of a device with a transparent laptop configuration . display screen from the perspective of a user with a mixed FIG . 11C depicts an exemplary simplified illustration 40 reality annotation centered over the user's view of the object used with some forms of the invention incorporating dual through the transparent display. shutter -display -shutter screens movably connected in a tent Exemplary Applications. laptop configuration . FIG . 19A depicts an example location - based mixed - real FIG . 11D depicts an exemplary simplified illustration ity hiking or mapping application running on a single screen used with some forms of the invention incorporating dual 45 device with a multi -mode display. shutter -display - shutter screens movably connected in a FIG . 19B depicts an example mixed - reality camera appli small tablet configuration . cation running on a single screen device with a multi -mode FIG . 11E depicts an exemplary simplified illustration used display. with some forms of the invention incorporating dual shutter Elements and acts in the figures are illustrated for sim display - shutter screens movably connected in a large tablet 50 plicity and clarity, and the drawings have not necessarily configuration . While the hinge is depicted below the screen been rendered according to any sequence or embodiment , in the simplified illustration , it may be positioned between and their simplicity should not use to limit the scope of the the two screens so that the device is able to lay flat on a table invention . or other surface . Dual DSD Screen Form Factor Configurations. 55 DETAILED DESCRIPTION FIG . 12A depicts an exemplary simplified illustration used with some forms of the invention incorporating dual In the following description , and for the purposes of display - shutter- display screens movably connected in an explanation , numerous specific details are set forth in order upright laptop configuration . to provide a thorough understanding of the various aspects FIG . 12B depicts an exemplary simplified illustration 60 of the invention . It will be understood , however , by those used with some forms of the invention incorporating dual skilled in the relevant arts , that the present invention may be display -shutter -display screens movably connected in a practiced without these specific details . In other instances, stand laptop configuration . known structures and devices are shown or discussed more FIG . 12C depicts an exemplary simplified illustration generally to avoid obscuring the invention . In many cases, a used with some forms of the invention incorporating dual 65 description of the operation is sufficient to enable one to display - shutter -display screens movably connected in a tent implement the various forms of the invention , particularly laptop configuration . when the operation is to be implemented in software . It US 10,520,782 B2 23 24 should be noted that there are many different and alternative 1.2 . Active Shutter ( AS ) Technologies . configurations, devices and technologies to which the dis An active shutter is a substantially transparent element closed inventions may be applied . The full scope of the that can be actuated to become opaque . There are many inventions is not limited to the examples that are described types of active shutters such as Kerr cell shutters and various below . 5 types of liquid crystal shutters. In addition to the description provided above , additional 1.2.A Kerr Cell Shutters (KCS ). information is provided in the file history associated with A Kerr cell shutter is a type of photographic shutter used this application and the associated provisional application for very fast shutter speeds down to nanosecond level . The referenced above ( e.g. , as an Appendix to the Specification , Kerr Cell consists of a transparent container filled with Information Disclosure Statement, Transmittal Letter, etc. ); 10 nitrobenzene with attached electrodes. A high voltage is those materials are hereby incorporated by reference in its passed through the electrodes which causes an electric field entirety . perpendicular to the transmitted light beam to be applied . I. Overview The nitrobenzene becomes birefringent under the influence 1.1 . Transparent Display ( TD ) Technologies . of the electric field . This allows it to be used as a shutter that There are two main see - through display technologies, 15 can be opened for a very brief amount of time, around 10 ns. LCD and LED . In see - through displays the difference 1.2.B Liquid Crystal Shutter (LCS ) . between the absorptive nature of the LCD and emissive Generally , a LCS is a liquid crystal layer which has the nature of the OLED gives them very different visual appear property of becoming opaque when voltage is applied (or ances . LCD systems impose a pattern of shading and colors 20 removed ), and being otherwise transparent in its other state . on the background light seen through the display, while The shutter is controlled by a signal that allows the liquid OLED systems impose a glowing image pattern on the crystal layer to block light or allow light to pass through the background by using LED element suspended in a transpar element. There are different types of devices , such as a ent medium . twisted -nematic ( TN ) cell and a pi cell . According to one I.1. A Liquid Crystal Display (LCD ). 25 suppliers , a typical TN shutter will only toggle at a rate of While any LCD panel is “ see through ” to some extent, about 90 to 100 Hz, while rates, up to about 1000 Hz are most are placed in front of a back light so that the display achieved with a pi cell. As such , pi cell LC shutters are image is bright and consistent to users . Furthermore, con generally used in applications where very fast switching ventional LCDs have relatively low transmission efficiency cycle times are required . Pi cells also have an increased so that they tend to appear somewhat dark against natural 30 viewing angle , which is the result of the self - compensating light. LCDs intended specifically for see- through displays nature of the cell structure . are usually designed to have improved transmission effi 1.2.C Smart Glass . ciency. An alternative approach to commercializing this Smart glass or switchable glass ( also smart windows or technology is to offer conventional back - lit display systems switchable windows in those applications) is glass or glazing without the back -light system . LCD displays often also 35 whose light transmission properties are altered when volt require removing a diffuser layer to adapt them for use as age , light or heat is applied . Generally , the glass changes transparent displays. The key limitation to see -through LCD from translucent to transparent, changing from blocking efficiency is in the polarizing filters , which inherently limit some ( or all) wavelengths of light to letting light pass the transmission efficiency for un -polarized light to 50 % or 40 through. Smart glass technologies include electrochromic , less. Inexpensive polarizing films also have relatively low photochromic , thermochromic , suspended particle , micro transmission efficiency , though commercial development blind and polymer dispersed liquid crystal devices . efforts have had some success in improving it. LCD tech I.2.D Other Active Shutter Technologies . nology may be used as a transparent display as discussed While liquid crystal shutters (LCS ) are currently the herein . 45 preferred type of active shutter (AS ) element to be used with I.1.B Light - Emitting and Emissive Displays . the invention , any form of active shutter may be used . In LED and OLED screens have two layers of glass on both some forms of the invention the active shutter will have a sides of a set of addressable LEDs. Both inorganic and fast switching rate . In some formsof the invention the active organic (OLED ) LEDs can be used for this purpose . OLEDs shutter will have low - reflectance so that it absorbs light consist of an emissive and conductive layer. Electrical 50 instead of reflecting light. impulses travel through the conductive layer and produce 1.3 . TD with AS . light at the emissive layer . This is different from LCDs in Various forms of the invention combine TD and AS that OLEDs produce their own light, which produces better technologies and use them in combination with one another visual effect with a see - through display. The narrow gap to advantageously display images and optionally create between the pixels of the screen as well as the clear cathodes 55 opaque , transparent, or translucent effects along with single within allow the screens to be transparent. However , because sided or double -sided display images as further described the screens are transparent and use LEDs to emit light of throughout. various colors , they cannot display black portions of images . The disclosure is generally organized as follows. First, Also , while some transparent display elements have been display elements that are created with TD and AS technolo available over time , many have been discontinued due to 60 gies are discussed . Second , the different states that the lack of demand . LED , OLED , and similar display technol display elements may be operated in are discussed . Third , ogy may be used as a transparent display as discussed display modes created by combining display element states herein . that can be used with the inventive display elements (or I.1.C Other Transparent Display Technologies . other display technologies ) are discussed . Fourth , device While LED and OLED transparent -display panels are a 65 form factors incorporating the display elements are dis type of transparent display element that can be used with the cussed along with examples of the different form factor invention , any form of transparent display may be used . device configurations and display modes that may be used in US 10,520,782 B2 25 26 each configuration . Finally , additional examples and appli In any form of the invention , there may also be any type cations of dual- sided transparent displays of any type are of input or touch sensing technology integrated into the discussed . assembly , or otherwise used in conjunction with the assem The various display elements may variously be used for bly . any of the disclosed display modes, devices, and applica 5 II.2 Basic Simplified Screen Element Structure for a tions in any combination . “ DSD Display ” II. Exemplary Display Elements . II.2.A Display -Shutter - Display (DSD ). Any of the configurations discussed herein may be used or With reference to FIG . 2A ( which depicts a separated modified for use with any form of the invention described block diagram of hardware used by some forms of the herein . For purposes of illustration and without limitation , 10 invention including two transparent display screens, one any basic simplified display element structure or configu active shutter , and two touch sensing panels ), FIG . 2B ration may be used or modified for use with any of the ( which depicts a joined block diagram of hardware used by displays described herein , any appropriate display element some forms of the invention including two transparent state or mode described below or otherwise , any of the 15 display screens, one active shutter , and two touch sensing device form factors described below or otherwise , any of the panels ), and FIG . 2C (which depicts a separated block applications described below or otherwise . diagram of hardware used by some forms of the invention Numerous configurations can be achieved with TD and including two transparent display screens, and one active AS elements as further described below . Examples of shut shutter) , an active shutter 204 is placed in -between a first ter -display - shutter (SDS ) and display - shutter- display (DSD ) 20 transparent display 202 and a second transparent display as well as shutter- display (SD ) or display -shutter (DS ) are 206. There is an orchestrator 110 (which may be a stand provided for purposes of illustration and without limitation alone component or a module that is part of or software that below . is run on a central processing unit, display controller, II.1 Basic Simplified Screen Element Structure for a microprocessor, or any other type of circuitry capable of “ SDS Display” ( and SD or DS Displays ) 25 controlling the various elements of the display as further II.1. A Shutter- Display - Shutter (SDS ) . described throughout) operatively coupled to the various With reference to FIG . 1A (which depicts a separated components as described and shown . block diagram of hardware used by some forms of the invention including two active shutters , one transparent Touch input sensors may also be integrated into compo display screen , and two touch sensing panels ), FIG . 1B 30 nentsthe touch of the sensing display may . For be example integrated , with into respect either to theFIG trans. 2C , ( which depicts a joined block diagram of hardware used by parent displays 202 and 206 , or may be part of the active some forms of the invention including two active shutters, shutter 204. As one example, emissive display elements of one transparent display screen , and two touch sensing pan the transparent displays 202 and 206 may be driven in ofels hardware) , and FIG used. 1C by(which some depicts forms aof separated the invention block including diagram 35 reverse biased mode to detect reflected light from a finger in two active shutters and one transparent display screen ), a contact or near the assembly . transparent display 104 is placed in -between a first active With reference to FIG . 1A and FIG . 1B , touch sensor 100 shutter 102 and a second active shutter 106. There is an and touch sensor 108 may be placed near transparent dis orchestrator 110 ( which may be a stand - alone component or plays 202 and 206 , respectively . Touch sensor may be a module that is part of or software that is run on a central 40 capacitive , resistive, multi -touch , force -touch , or any other processing unit, display controller, microprocessor, micro type or variation of or combination of touch sensing tech controller, or any other type of circuitry capable of control nologies . ling the various elements of the display as further described II.2.B Display - Object- Display (DOD ). throughout) operatively coupled to the various components FIG . 2D depicts a separated block diagram of hardware as described and shown . 45 used by some forms of the invention ) includes two trans Touch input sensors may also be integrated into compo parent displays 202 and 206 on either side of object 200 nents of the display. For example , with respect to FIG . 1C , driven by an orchestrator 110. In any form of the invention , the touch sensing may be integrated into either the active there may also be any type of input or touch sensing shutters 102 and 106 , or may be part of the transparent technology integrated into the assembly . display 104. As one example , emissive display elements of 50 In forms of the invention using a DSD screen element , the the transparent display 104 may be driven in a reverse biased OLED or other display pixels may be focused or otherwise mode to detect reflected light from a finger in contact or near directed or reflected outward from the shutter element of the the assembly . screen . With reference to FIG . 1A and FIG . 1B , touch sensor 100 II.3 Touch Sensing/ Scanning/ Emissive Touch Input . and touch sensor 108 may be placed near active shutters 102 55 In any form of the invention , there may also be any type and 106 , respectively . Touch sensor may be capacitive, of input or touch sensing technology integrated into the resistive, multi- touch , force - touch , or any other type or assembly , or placed within the assembly , or affixed to the variation of or combination of touch sensing technologies. assembly . In forms of the invention using a SDS screen element, the II.3.A Emissive Touch Input. OLED or other display pixels may be bi- directional or 60 The emissive screen elements may be utilized for touch omni- directional so that light from the display pixel ema input as described in the following documents : See Cite No. nates outward in both or all directions from the display pixel. D062 . See also , e.g. , U.S. Pat. No. 7,859,526 . II.1.B Display - Shutter (DS ) or Shutter - Display ( SD ) . II.3.B Scanning Display . FIG . 1D (which depicts a separated block diagram of Portions of a display panel may also operate as an image hardware used by some forms of the invention ) includes one 65 scanner ( e.g., functionality of a flat -bed scanner) , or touch active shutter 106 and one transparent display screen 104 input sensors ( e.g., instead of or in addition to resistive or driven by an orchestrator 110 . capacitive touch sensors ) . US 10,520,782 B2 27 28 II.4 Construction . ( 3 ) AS.T is when an active shutter is transparent. Note that any type of construction is acceptable . Each ( 4 ) TD.OFF is when the transparent display is OFF and element may be a pixel of a screen , each element may be a transparent. pane of a screen , the active shutter elements may be sized ( 5 ) < TD.ONL is when the transparent display is on and differently than the transparent display elements , active 5 configured to show an image correctly to the observer shutter elements larger than pixel of transparent display but on the left ( as depicted ) . less than whole size of transparent display . For example , the (6 ) TD.ONR ? Is when the transparent display is on and active shutter elements may be formed from strips that cover configured to show an image correctly to the observer rows or columns of a transparent display panel. Further on the right ( as depicted ) . more , the display may be a flexible display. The active 10 With respect to FIG . 3A , a simplified side view illustra shutter or transparent display technologies may be any type tion of a shutter- display -shutter element used by some forms of display technology . The various components may be of the invention is provided . Specifically , the illustration separate layers stacked together as a combination of com shows a state where the left active shutter 102 is opaque ponents , variously fused together, or provided as one pack (L : AS.O ), the elements 304 of the transparent display 104 aged component. For a single panel, there may be one TD 15 are illuminated with an image that is properly oriented to an element with one AS element. A display may also have observer attempting to view the screen from the left display element groups. A TD could be coupled with an AS ( < TD.ONL ) , and the right active shutter 106 is transparent containing multiple elements , each element corresponding ( R :AS.T ). An observer to the left of the screen as depicted to multiple TD pixels . A display may also contain individual 300 will observe the opaque active shutter 102. An observer display elements . For example , a TD may be coupled with 20 to the right of the screen as depicted 302 will observe an an AS element corresponding to each pixel. For example , image from the transparent display 104 through transparent one AS may cover the subpixels that comprise a pixel. A active shutter 106 on top of the opaque active shutter 102 . display may also be a portion of a “ traditional” otherwise The image observed by observer 302 will be mirror image non - see - through display panel. A TD may form a " window " ( or flipped horizontally ) from what would be considered the in the panel . Any combination of the above (e.g. , could be 25 correct orientation for the image . different for each side of device , etc.) . An Opaque State . Furthermore , other types and variations of elements and With respect to FIG . 3B , a simplified side view illustration states can be determined . Display ( D ) and shutter ( S ) ele of a shutter -display - shutter element used by some forms of ments may be stacked in any order and combination includ the invention is provided . Specifically , the illustration shows ing the following : DS, SD , DSD , SDS, DSSD , SDSD , 30 a state where the left active shutter 102 is opaque ( L :AS.O ), SDSDS, SDSSDS, and SSDSS . Any combination imagin the elements 304 of the transparent display 104 are off able may be used . Furthermore , each part can be made from ( TD.OFF ), and the right active shutter 106 is transparent) . a different display or shutter technology if desirable . An observer to the left of the screen as depicted 300 will III . Exemplary Display Element States observe the opaque active shutter 102. An observer to the III . A SDS Element States . 35 right of the screen as depicted 302 will observe no image Any of the element states discussed herein may be used or from the transparent display 104 through transparent active modified for use with any form of the invention described shutter 106 on top of the opaque active shutter 102. This herein . For purposes of illustration and without limitation , SDS element state may be referred to or correspond to any display state may be used or modified for use with any discussion elsewhere herein described as opaque . of the displays described above , any display mode described 40 A Normal State (Right ) . below or otherwise , any of the device form factors described With respect to FIG . 3C , a simplified side view illustration below or otherwise , any of the applications described below of a shutter -display -shutter element used by some forms of or otherwise . the invention is provided . Specifically , the illustration shows In some forms of the invention Shutter- Display - Shutter a state where the left active shutter 102 is opaque (L : AS.O ), (SDS ) configurations are used . Generally , SDS configura- 45 the elements 304 of the transparent display 104 are illumi tions exemplified in FIGS. 1A , 1B , 1C , and 1D (other forms nated with an image that is properly oriented to an observer possible ) can be configured or placed in any number of attempting to view the screen from the right ( TD.ONR ) , states . For example , states corresponding to FIGS. 3A and the right active shutter 106 is transparent ( R :AS.T ). An through 3L are possible . By using these states along with the observer to the left of the screen as depicted 300 will observe structures and methods described elsewhere herein , forms of 50 the opaque active shutter 102. An observer to the right of the the invention can display any color (including any color screen as depicted 302 will observe an image from the composed of red , green , blue , white , black , or transparent) transparent display 104 through transparent active shutter on either or both sides of the display . 106 on top of the opaque active shutter 102. The image Regarding the notation used below and throughout : observed by observer 302 will be in a correct orientation for ( 1 ) L : and R : designate left and right of the cross - section 55 the image ( i.e., not mirror image or flipped horizontally ) . depicted in the figures . These designations are for This SDS element state may be referred to or correspond to purposes of helping understand the invention and discussion elsewhere herein described as normal mode would obviously be reversed if the opposite cross (right ) . section was taken . In practice , a user will be looking at An Opaque State . the front orback of the device predominantly ; however , 60 With respect to FIG . 3D , a simplified side view illustra it is easier here to describe the invention with a side tion of a shutter- display - shutter element used by some forms cross section so that the front and back relative to the of the invention is provided . Specifically , the illustration user can be described using left or right due to the shows a state where the left active shutter 102 is opaque limitations of a two -dimensional page . Thus, the left ( L : AS.O ), the elements 304 of the transparent display 104 and right designations where used are exemplary and 65 are illuminated with an image that is properly oriented to an not limiting. observer attempting to view the screen from the left ( 2 ) AS.O is when an active shutter is opaque . ( < TD.ONL ) , and the right active shutter 106 is opaque US 10,520,782 B2 29 30 ( R :AS.O ) . An observer to the left of the screen as depicted With respect to FIG . 31, a simplified side view illustration 300 will observe the opaque active shutter 102. An observer of a shutter - display - shutter element used by some forms of to the left of the screen as depicted 302 will observe the the invention is provided . Specifically , the illustration shows opaque active shutter 106. This SDS element state may be a state where the left active shutter 102 is transparent referred to or correspond to discussion elsewhere herein 5 (L : AS.T ), the elements 304 of the transparent display 104 described as opaque . are illuminated with an image that is properly oriented to an An Opaque State . observer attempting to view the screen from the right With respect to FIG . 3E , a simplified side view illustration ( TD.ONR ), and the right active shutter 106 is opaque of a shutter- display - shutter element used by some forms of ( R : AS.O ) . An observer to the right of the screen as depicted the invention is provided . Specifically , the illustration shows 10 302 will observe the opaque active shutter 106. An observer a state where the left active shutter 102 is opaque ( L : AS.O ), to the left of the screen as depicted 300 will observe an the elements 304 of the transparent display 104 are off image from the transparent display 104 through transparent ( TD.OFF ) , and the right active shutter 106 is opaque active shutter 102 on top of the opaque active shutter 106 . ( R :AS.O ) . An observer to the left of the screen as depicted The image observed by observer 300 will be mirror image 300 will observe the opaque active shutter 102. An observer 15 (or flipped horizontally ) from what would be considered the to the left of the screen as depicted 302 will observe the correct orientation for the image . opaque active shutter 106. This SDS element state may be An Augmented Reality State (Left ) . referred to or correspond to discussion elsewhere herein With respect to FIG . 3 ), a simplified side view illustration described as opaque . 20 of a shutter- display - shutter element used by some forms of An Opaque State . the invention is provided . Specifically , the illustration shows With respect to FIG . 3F , a simplified side view illustration a state where the left active shutter 102 is transparent of a shutter - display - shutter element used by some forms of ( L : AS.T ) , the elements 304 of the transparent display 104 the invention is provided . Specifically , the illustration shows are illuminated with an image that is properly oriented to an a state where the left active shutter 102 is opaque ( L :AS.O ) , 25 observer attempting to view the screen from the left the elements 304 of the transparent display 104 are illumi ( < TD.ONL ), and the right active shutter 106 is transparent nated with an image that is properly oriented to an observer ( R :AS.T ) . An observer to the left of the screen as depicted attempting to view the screen from the right ( TD.ONL ) , 300 will observe an image from the transparent display 104 and the right active shutter 106 is opaque ( R : AS.O ) . An through transparent active shutter 102 on top of the trans observer to the left of the screen as depicted 300 will observe 30 parent active shutter 106. The image observed by observer the opaque active shutter 102. An observer to the left of the 300 will be mirror image ( or flipped horizontally ) from what screen as depicted 302 will observe the opaque active shutter would be considered the correct orientation for the image . 106. This SDS element state may be referred to or corre An observer to the right of the screen as depicted 302 will spond to discussion elsewhere herein described as opaque. observe an image from the transparent display 104 through A Normal State ( Left ) . 35 transparent active shutter 106 on top of the transparent With respect to FIG . 3G , a simplified side view illustra active shutter 102. The image observed by observer 302 will tion of a shutter- display - shutter element used by some forms be mirror image (or flipped horizontally ) from what would of the invention is provided . Specifically , the illustration be considered the correct orientation for the image. This shows a state where the left active shutter 102 is transparent SDS element state may be referred to or correspond to ( L : AS.T ) , the elements 304 of the transparent display 104 40 discussion elsewhere herein described as augmented reality are illuminated with an image that is properly oriented to an (left ). observer attempting to view the screen from the left A Transparent State . ( < TD.ONL ), and the right active shutter 106 is opaque With respect to FIG . 3K , a simplified side view illustra ( R : AS.O ). An observer to the left of the screen as depicted tion of a shutter - display - shutter element used by some forms 300 will observe an image from the transparent display 104 45 of the invention is provided . Specifically , the illustration through transparent active shutter 102 on top of the opaque shows a state where the left active shutter 102 is transparent active shutter 106. The image observed by observer 300 will ( L : AS.T ) , the elements 304 of the transparent display 104 be in a correct orientation for the image ( i.e., not mirror are off ( TD.OFF ), and the right active shutter 106 is trans image or flipped horizontally ). An observer to the right of parent ( R :AS.T ) . An observer to the left of the screen as the screen as depicted 302 will observe the opaque active 50 depicted 300 will observe light transmitted through trans shutter 106. This SDS element state may be referred to or parent active shutter 102 , transparent display 104 , and correspond to discussion elsewhere herein described as transparent active shutter 106. An observer to the right of the normal (left ). screen as depicted 302 will observe light transmitted through An Opaque State . transparent active shutter 102 , transparent display 104 , and With respect to FIG . 3H , a simplified side view illustra- 55 transparent active shutter 106. This SDS element state may tion of a shutter- display - shutter element used by some forms be referred to or correspond to discussion elsewhere herein of the invention is provided . Specifically , the illustration described as transparent or translucent. shows a state where the left active shutter 102 is transparent An Augmented Reality State (Right ). ( L : AS.T ), the elements 304 of the transparent display 104 With respect to FIG . 3L , a simplified side view illustration are off ( TD.OFF ) , and the right active shutter 106 is opaque 60 of a shutter -display - shutter element used by some forms of ( R :AS.O ). An observer to the left of the screen as depicted the invention is provided . Specifically, the illustration shows 300 will observe no image from the transparent display 104 a state where the left active shutter 102 is transparent through transparent active shutter 102 on top of the opaque ( L : AS.T ) , the elements 304 of the transparent display 104 active shutter 106. An observer to the right of the screen as are illuminated with an image that is properly oriented to an depicted 302 will observe the opaque active shutter 106. 65 observer attempting to view the screen from the right This SDS element state may be referred to or correspond to ( TD.ONR ), and the right active shutter 106 is transparent discussion elsewhere herein described as opaque . ( R : AS.T ) . US 10,520,782 B2 31 32 An observer to the left of the screen as depicted 300 will a state where the elements of the left transparent display 202 observe an image from the transparent display 104 through are illuminated with an image that is properly oriented to an transparent active shutter 102 on top of the transparent observer attempting to view the screen from the left active shutter 106. The image observed by observer 300 will ( L : < - TD.ONL) , the active shutter 204 is opaque (AS.O ) , be mirror image ( or flipped horizontally ) from what would 5 and the elements of the right transparent display 206 are off be considered the correct orientation for the image. An (R : TD.OFF ) . An observer to the left of the screen as observer to the right of the screen as depicted 302 will depicted 400 will observe an image from the left transparent observe an image from the transparent display 104 through display 202 on top of opaque active shutter 204. An observer transparent active shutter 106 on top of the transparent to the right of the screen as depicted 402 will observe opaque active shutter 102. The image observed by observer 302 will 10 active shutter 204 through the right transparent display 206 . be mirror image ( or flipped horizontally ) from what would The image observed by observer 400 will be in a correct be considered the correct orientation for the image . This orientation for the image ( i.e., not mirror image or flipped SDS element state may be referred to or correspond to horizontally ) . This DSD element state may be referred to or discussion elsewhere herein described as augmented reality correspond to discussion elsewhere herein described as ( right) . 15 normal (left ) . With respect to FIGS. 3G through 3L above, these states An Augmented Reality State (Right ) . illustrate possible states with a form of invention corre With respect to FIG . 4C , a simplified side view illustration sponding to the simplified illustration of FIG . 1D by disre of a display - shutter -display element used by some forms of garding the active shutter 102 depicted on the left of the the invention is provided . Specifically , the illustration shows simplified illustration . 20 a state where the elements of the left transparent display 202 The elements 304 of the transparent display 104 may be are illuminated with an image that is properly oriented to an any type of sub -pixel or pixel. For example , they may observer attempting to view the screen from the right represent an individual part of a sub -pixel such as a red , (L : TD.ONR ), the active shutter 204 is transparent ( AS.T ), green , blue , or white display sub - element, or they may be a and the elements of the right transparent display 206 are off complete pixel component. Alternatively , they may be any 25 ( R : TD.OFF ) . An observer to the left of the screen as type of component of a screen responsible for causing light depicted 400 will observe an image from the left transparent to be emitted or augmented . display 202 as well as any ambient light transmitted through III.B DSD Element States . left transparent display 202 , transparent active shutter 204 , In some forms of the invention Display -Shutter -Display and right transparent display 206. An observer to the right of (DSD ) configurations are used . Generally , DSD configura- 30 the screen as depicted 402 will observe an image from the tions exemplified in FIGS . 2A , 2B , 2C , and 2D (other forms left transparent display 202 as well as any ambient light possible ) can be configured or placed in any number of transmitted through left transparent display 202 , transparent states. For example , states corresponding to FIGS. 4A active shutter 204 , and right transparent display 206. The through 4R are possible . By using these states along with the image observed by observer 400 will be mirror image (or structures and methods described elsewhere herein , forms of 35 flipped horizontally ) from what would be considered the the invention can display any color ( including any color correct orientation for the image . The image observed by composed of red , green , blue, white , black , or transparent) observer 402 will be in a correct orientation for the image on either or both sides of the display . ( i.e., not mirror image or flipped horizontally ). This DSD An Augmented Reality State (Left ) . element state may be referred to or correspond to discussion With respect to FIG . 4A , a simplified side view illustra- 40 elsewhere herein described as augmented reality ( right) . tion of a display - shutter -display element used by some With respect to FIG . 4D , a simplified side view illustra forms of the invention is provided . Specifically , the illus tion of a display -shutter - display element used by some tration shows a state where the elements of the left trans forms of the invention is provided . Specifically , the illus parent display 202 are illuminated with an image that is tration shows a state where the elements of the left trans properly oriented to an observer attempting to view the 45 parent display 202 are illuminated with an image that is screen from the left ( L :< TD.ONL ), the active shutter 204 is properly oriented to an observer attempting to view the transparent ( AS.T ), and the elements of the right transparent screen from the right ( L : TD.ONR-> ) , the active shutter 204 display 206 are off (R : TD.OFF) . An observer to the left of is opaque ( AS.O ), and the elements of the right transparent the screen as depicted 400 will observe an image from the display 206 are off ( R : TD.OFF ) . An observer to the left of left transparent display 202 as well as any light transmitted 50 the screen as depicted 400 will observe an image from the through transparent active shutter 204 right transparent left transparent display 202 on top of opaque active shutter display 202. An observer to the right of the screen as 204. An observer to the right of the screen as depicted 402 depicted 402 will observe an image from the left transparent will observe opaque active shutter 204 through the right display 202 as well as any ambient light transmitted through transparent display 206. The image observed by observer left transparent display 202, transparent active shutter 204 , 55 400 will be mirror image (or flipped horizontally ) from what and right transparent display 206. The image observed by would be considered the correct orientation for the image . observer 402 will be mirror image (or flipped horizontally ) An Augmented Reality State 2.5D (Left ). from what would be considered the correct orientation for With respect to FIG . 4E , a simplified side view illustration the image. The image observed by observer 400 will be in of a display- shutter - display element used by some forms of a correct orientation for the image ( i.e., not mirror image or 60 the invention is provided . Specifically , the illustration shows flipped horizontally) . This DSD element state may be a state where the elements of the left transparent display 202 referred to or correspond to discussion elsewhere herein are illuminated with an image that is properly oriented to an described as augmented reality ( left ) . observer attempting to view the screen from the left A Normal State (Left ). (L :< - TD.ONL ), the active shutter 204 is transparent ( AS.T) , With respect to FIG . 4B , a simplified side view illustration 65 and the elements of the right transparent display 206 are of a display - shutter- display element used by some forms of illuminated with an image that is properly oriented to an the invention is provided . Specifically , the illustration shows observer attempting to view the screen from the left US 10,520,782 B2 33 34 ( R : laptops, or other types of annotation 1804 appears over the object 2002 as viewed by form factors , a tablet with any of the features of the phone the user at location 1808 through the screen 1800. In other described above , a laptop that can be configured in various words, the annotation 1804 is centered on location 1814 . configurations described herein , a double sided display , an V.E Far -Field Occlusion . ultra -wide display, a virtual or augmented reality device, a In instances where the location of the object 2002 is very 15 flexible display , or any other type of form factor or combi far away from the device, the device may not be able to use nation of form factors . For form factors that implement both its sensors to determine the distance or coordinates Xo , Yo , augmented reality and virtual reality , the shutter element on and Zo . One example of this may be when the device is the back panel (away from head ) is made opaque so that it pointed at a skyline. If a user is running an augmented reality is not see through when in virtual reality mode but can be application to show directions or a path to a destination 20 made transparent for unobstructed or augmented reality through the skyline , the path may be rendered as an anno applications. tation over a building that the path should pass behind ( i.e., The form factors may implement any type any number of the path should be occluded by the building) . elements discussed herein . For example , single element As another example , if a pin is placed far away between form factorsmay incorporate one SDS or DSD (or any other two buildings and extends upward into the sky, the AR 25 combination of D and S elements ) display panels . As another device will not know how to properly draw the annotation as example , dual element form factors may incorporate two or it extends upward , because the device will render the pen more SDS or DSD ( or any other combination of D and S annotation for its full length and not just for the portion of elements ) display panels. In addition , as depicted , form the annotation that extends upward above the closest build factors may incorporate one or more flexible panels instead ing but in front of the further building . Alternatively , the 30 of hinged flat panels . phone will not know to render the occluded portion of the VI.A Form Factor : Single Element. annotation differently such that the user is aware that the With reference to FIGS. 16A and 16B , one form factor is annotation is occluded . a device that incorporates a display panel 1602 in a bar , To solve this problem , sensor data may be supplemented panel , or tablet form factor and configuration . The device with a 3D model of the world and buildings such as those 35 includes a main body 1600 and sensors 1604 and 1606 . models available on Apple Maps or Google Earth . The Sensors 1604 and 1606 are cameras but may be any type of relevant 3D models need not be displayed , and may only be sensor , and additional sensors may be present on the device . used to calculate proper occlusion for annotations based on Devices of this form factor may incorporate any of the the current location and viewing angle of the device, head inventive display elements and display modes discussed position of the user, etc. 40 above. V.F Device Activity Detection and Mode Selection . A Normal Mode . Based on the sensors and spatial context of the device, it With reference to , for example , FIGS. 3C , 3G , 5A , 5B , can operate the screen in variousmodes . In addition , aspects 7A , 7B , 4B , 4P , 6A , 6B , 8A , and 8B , the device can drive the of the device can be changed based on the context of a display in normal mode. Normal mode may be used for specific activity . 45 various applications that run on the device . The device or For example , there may be a walking mode . In some applications may change the display to this mode based on forms of the invention it is determined that the user is heuristics, a user selected function and / or based upon sensor walking based on motion processor. The device may deter input. For example , if it is detected that the device is in use , mine that a user is walking and is in a congested area of a the device may wake from sleep mode if applicable and city . Once this is determined , the device may begin to 50 activate any display mode. If normal display mode is acti operate the screen in private AR mode /unobstructed mode vated , the device may determine the orientation of the phone (private normal mode ) so that the user can see what is in either relative to gravity or relative to the user's face, and front of them while continuing to utilize their device to display the primary user interface on the appropriate side of consume information privately without showing third par the transparent display panel for the primary user in the ties that are walking the opposite direction . 55 proper orientation relative to gravity or relative to the user's As another example , there may be a driving mode . In face while operating the AS elements and TD elements as some forms of the invention it is determined that the user is described elsewhere herein . driving based on a motion processor. The device may also A Presentation Mode . determine this based on device location corresponding to a With reference to , for example , FIGS . 3C , 3G , 5C , 5D , road , for example . In addition , or alternatively , the activity 60 7C , 7D , 4H , 6C , and 8C , the device can drive the display in determination may be based in whole or in part on distance presentation mode. Presentation mode may be used for to user's face , and face angle . If the device is on a dashboard , various applications that run on the device . The device or for instance , (based on distance and angle to user ) and it is applications may change the display to this mode based on determined the user is driving, then the device may switch heuristics , a user selected function and /or based upon sensor automatically into an automotive driving mode as a heads- 65 input. For example , if the user raises the device up so that the up display (HUD ) and display an appropriate user interface back is somewhat perpendicular to the horizon , the device in any augmented reality or similar display mode . may change to presentation mode . In addition , the device US 10,520,782 B2 59 60 may sense if there is an observer on both sides of the device the device. The device or applications may change the prior to changing to presentation mode. Generally , it may display to this mode based on heuristics, a user selected display a user - selected background image , branding , loca function and /or based upon sensor input. For example , if the tion based advertisements , or the like. Alternatively , presen phone is put into a headset it may switch to a virtual reality tation mode may be activated when a user runs an applica- 5 mode . This mode may operate the display in normal mode tion . One example would be when the user runs a camera when the virtual world is being displayed to the user . If the application . The camera application may operate the device operator side of the display to show a camera preview and user is getting close to an object in the realworld , as detected camera controls , the camera application may operate the by sensors on the device or in the headset in communication opposite side of the display to show a full screen preview of 10 with the device , the device may switch to an unobstructed the photograph to be taken to assist the photograph subjects (partially transparent) or a fully transparent mode to allow in posing for the photo . The camera application may only the user to see what is happening in the real world . The activate presentation mode when the device is both in device may also be put into a sharing mode such that the user picture taking mode and oriented substantially perpendicular wearing the goggles sees stereoscopic virtual reality ren to the horizon . 15 dering whereas someone on the outside of the goggles sees A Sharing Mode. a conventional rendering of the view shown to the user . A With reference to , for example , FIGS . 3C , 3G , 5C , 5D , headset may also be used with augmented reality modes . 7C , 7D , 4H , 6C and 8C , the device can drive the display in Furthermore, there may be additional optics between the sharing mode. Sharing mode may be used for various screen and the user in forms of the invention utilizing applications that run on the device . The device or applica- 20 augmented reality mode. tions may change the display to this mode based on heuris A single element device may be combined with another tics , a user selected function and / or based upon sensor input. object to behave like a single element hinged device as This is like presentation mode, but the back would display discussed below . For example , a hinge , clip etc.may be used content that a user wishes to share with others . This may to couple a single element device with a wired or wireless simply be a background image , pictures , or a display and 25 connection to an input device or other object . controls multiplayer game. For example , a flash card type A single element device may also be combined with game may show the user on one side a clue while simulta another single element device to behave like a dual element neously showing a user on the other side the answer. The hinged device as discussed below . For example , a hinge , clip phone may use facial recognition to determine which player etc. may be used to couple a single element device with should see which card . 30 another single element device. Of course , two display ele An Augmented Reality Mode . ments may also bemanufactured into a dual element device . With reference to , for example , FIGS. 3J 3L , 5E , 5F , 7E , VI.B Form Factor : Single Element Hinged . F , 4C , 40 , 4K , 4M , 4A , 4E , 6D , 6F , 6G , 6H , 61, 8D , 8E , With reference to FIGS . D , J , and 10A - 10D , 8F , 8G , 8H , and 81, the device can drive the display in one form factor is a hinged device that incorporates a augmented reality mode . Augmented reality ( or mixed real- 35 multi -mode display panel . The device includes a main body ity ) mode may be used for various applications that run on and sensors devices of this form factormay incorporate any the device . The device or applications may change the of the inventive display elements and display modes dis display to this mode based on heuristics, a user selected cussed above . A single element hinged device with an SDS function and /or based upon sensor input. This mode may be display includes at least active shutter 102, transparent used for various applications that run on the device that 40 display 104 , active shutter 106 , and object 900 ( such as a desire to overlay information onto the real world . For device body or keyboard ) . The SDS panel and object 900 are example , if the user raises the device up so that the back is joined by hinge 902. A single element hinged device with a substantially perpendicular to the horizon , the device may DSD display includes at least transparent display 202 , active change to augmented reality mode by driving the display shutter 204 , transparent display 206 , and object 1000 ( such screen and shutters as discussed elsewhere herein to overlay 45 as a device body or keyboard ) . The DSD panel and object information or annotations onto the real world as viewed by 1000 are joined by hinge 1002 . a user of the device . VI.B.1 Upright Laptop Configuration . A Dual Sided Augmented Reality Mode or an Unob With reference to FIGS . 9A (SDS screen ), 9G (SD structed Mode . screen ), and 10A (DSD screen ) , a single element hinged With reference to , for example , FIGS. 3C , 3K , 3G , 3B , 3H 50 device may be placed into an upright laptop configuration . 5G , 5H , 51, 53, 5K , 76 , 7H , 71, 73, 7K , 4P , 4Q , 4B , 4H , 6J, In such a configuration , the display elements may be oper 6K , 6L , 6M , 8J, 8K , 8L and 8M , the device can drive the ated in any state or combination of states, and / or the display display in a dual sided augmented reality mode (or an may be operated in any of the modes discussed herein and unobstructed mode ). These modes may be used for various with some examples provided below . applications that run on the device. The device or applica- 55 Upright Configuration Normal Mode . tions may change the display to this mode based on heuris The device may drive the display in an upright normal tics, a user selected function and /or based upon sensor input. mode. This mode may be used for various applications that This mode may be used for various applications that run on run on the device . The device or applications may change the the device that desire to overlay information onto the real display to this mode based on heuristics, a user selected world for users on either side of the device. The modes may 60 function and /or based upon sensor input. For example , in also be used to allow a user to see information from one side this mode, the panel may display an operating system of the device while providing a level of transparency so that desktop , launcher, or other application to the user of the the device screen does not completely occlude the user's device such that the user is able to see the user interface and view of someone on the other side of the device . operate or access the object 900 or 1000. An observer on the Virtual Reality Mode . 65 back side of the device may only see the active shutter The device can drive the display in a virtual reality mode . element which obscures or blocks the reverse image of the This mode may be used for various applications that run on display thus maintaining the user's privacy. US 10,520,782 B2 61 62 For purposes of illustration and without limitation , with different image to both parties . This allows each party to see reference to FIG . 9A , active shutter 102 is made opaque, one another as well as information in a conference setting , transparent display 104 is configured to display an image in or allows bi- directional augmented reality applications . the correct orientation for a user to the right as depicted , and VI.B.2 Stand Configuration . active shutter 106 is made transparent. For purposes of 5 With reference to FIGS . 9B (SDS screen ) , 9H ( SD screen ) illustration and without limitation , with reference to FIG . and 10B (DSD screen ), a single element hinged device may 9G , active shutter 102 is made opaque , and transparent be placed into a stand configuration . In such a configuration , display 104 is configured to display an image in the correct the display elements may be operated in any state or orientation for a user to the right as depicted . For purposes combination of states, and /or the display may be operated in of illustration and without limitation , with reference to FIG . 10 any of the modes discussed herein and with some examples 10A , transparent display 202 is inactive or made to display provided below . an image such as branding or a decorative image , active Stand Configuration — Normal Mode . shutter 204 is made opaque, and transparent display 206 is The device may drive the display in a stand normal mode . configured to display an image of the desktop or user This mode may be used for various applications that run on interface in the correct orientation for a user of the device to 15 the device . The device or applications may change the the right as depicted . display to this mode based on heuristics, a user selected Upright Configuration Presentation Mode or Sharing function and /or based upon sensor input. For example , in Mode . this mode , the panel may display an operating system The device may drive the display in an upright presenta desktop , launcher, or other application to the user of the tion or sharing mode . This mode may be used for various 20 device such that the user is able to see the user interface applications that run on the device. The device or applica ( from the left as depicted in FIGS. 9B and 10B ). The user tions may change the display to this mode based on heuris may operate the device with any input device including a tics, a user selected function and / or based upon sensor input. touch screen user interface , or a wireless keyboard and /or For example , in this mode, the panel may display an mouse , pen , game controller , etc. This mode may be pref operating system desktop , launcher, or other application to 25 erable for a user that wants to minimize the surface space the user of the device ( to the right as depicted ) such that the required on a table or tray (e.g. , when on an airplane ) to user is able to see the user interface and operate or access the watch movies or read documents . object 900 or 1000. An observer on the back side of the For purposes of illustration and without limitation , with device ( to the left as depicted ) may see the same image (but reference to FIG . 9B , active shutter 102 is made transparent, reversed to maintain correct orientation ), or a different 30 transparent display 104 is configured to display an image in image (such as a decorative design for the lid , branding , the correct orientation for a user to the left as depicted , and advertisements , or an audience view instead of a presenter active shutter 106 is made opaque. For purposes of illustra view ). For example, a user of the laptop may run a Power tion and without limit on , with reference to FIG . active Point presentation on the device . On one side of the device shutter 102 is made opaque, and transparent display 104 is a presenter view with controls and notes is shown . On the 35 configured to display an image in the correct orientation for other side of the device , a full screen version of the presen a user to the right as depicted . For purposes of illustration tation (without controls or presenter notes) is shown . As and without limitation , with reference to FIG . 10B , trans another example , a game of “ Battle Ship ,” “ Risk ” (with fog parent display 202 is configured to display an image in the of war) or another game where each user has a different correct orientation for a user to the left as depicted , active display of the application state . Each side of the device may 40 shutter 204 is made opaque, and transparent display 206 is show a different image of the game board tailored to each inactive, or may be active such as when the device is in an user. As with any of the devices discussed herein , the display ambient light mode as discussed below . may also include touchscreen functionality on one or both Stand Configuration - Ambient Light Mode . sides. Where there is a touch screen capability both sides of The device may drive the display in a stand ambient light the display, each user can operate each side of the screen 45 mode. This mode may be used for various applications that independently . run on the device . The device or applications may change the Upright Configuration - Augmented Reality Mode . display to this mode based on heuristics, a user selected The device may drive the display in an augmented reality function and /or based upon sensor input. For example, in mode as discussed herein . This mode may be used for this mode, the panel may display an operating system various applications that run on the device . The device or 50 desktop , launcher , video , movie or other application to the applications may change the display to this mode based on user of the device such that the user is able to see the user heuristics, a user selected function and /or based upon sensor interface ( from the left as depicted in FIGS. 9B and 10B ). input. The user may operate the device with any input device Upright Configuration — Unobstructed Mode. including a touch screen user interface , or a wireless key The device may drive the display in an unobstructed or 55 board and /or mouse , pen , game controller , etc. This mode dual sided augmented reality mode as discussed herein . This may be preferable for a user that wants to minimize the mode may be used for various applications that run on the surface space required on a table or tray ( e.g. , when on an device . The device or applications may change the display to airplane ) to watch movies or read documents . This mode this mode based on heuristics, a user selected function may be desirable when a user is watching a movie in low and / or based upon sensor input. For example , in one such 60 light conditions and wishes to cast a light color behind the mode, the screen appears opaque or translucent to an audi device to help set a mood when watching a movie etc. ence to maintain privacy but normal (optionally with some For purposes of illustration and without limitation , with degree of transparency ) to an operator so it does not block reference to FIG . 9B , at a first time, active shutter 102 is the operator from people on other side of screen but allows made transparent, transparent display 104 is configured to the operator to maintain privacy and use the device . As 65 display an image in the correct orientation for a user to the another example , in another such mode, the screen appears left as depicted , and active shutter 106 is made opaque . At translucent to both parties while showing the same or a a second time, active shutter 102 is made opaque , active US 10,520,782 B2 63 64 shutter 106 is made transparent, and transparent display 104 this mode based on heuristics, a user selected function is configured to display a color as to illuminate the area and /or based upon sensor input. For example , in one such behind the device ( to the right as depicted ) as viewed by the mode , the screen appears opaque or translucent to an audi user. The color may be any color or combination of colors ence to maintain privacy but normal (optionally with some as to project an ambient light color. This process would then 5 degree of transparency ) to an operator so it does not block repeat quickly at a rate such that the user of the device does the operator from people on other side of screen but allows not perceive the flickering . For purposes of illustration and the operator to maintain privacy and use the device . As without limitation , with reference to FIG . 10B , transparent another example , in another such mode , the screen appears display 202 is configured to display an image in the correct translucent to both parties while showing the same or a orientation for a user to the left as depicted , active shutter 10 different image to both parties. This allows each party to see 204 is made opaque, and transparent display 206 is config one another as well as information in a conference setting , ured to display a color or combination of colors to illuminate or allows bi- directional augmented reality applications. the area behind the device . VI.B.3 Tent Configuration . Stand Configuration – Augmented Reality Mode. With reference to FIGS . 9C (SDS screen ) , 9I (SD screen ) The device may drive the display in a stand augmented 15 and 10C (DSD screen ), a single element hinged device may reality mode . This mode may be used for various applica be placed into a tent configuration . In such a configuration , tions that run on the device. The device or applications may the display elements may be operated in any state or change the display to this mode based on heuristics, a user combination of states, and /or the display may be operated in selected function and /or based upon sensor input. For any of the modes discussed herein and with some examples example , in this mode , the panel may display an operating 20 provided below . system desktop , launcher , video , movie or other application Tent Configuration Normal Mode . to the user of the device such that the user is able to see the The device may drive the display in a tent normal mode . user interface ( from the left as depicted in FIGS. 9B and This mode may be used for various applications that run on 10B ). The user may operate the device with any input device the device. The device or applications may change the including a touch screen user interface , or a wireless key- 25 display to this mode based on heuristics , a user selected board and /or mouse, pen , game controller , etc. This mode function and /or based upon sensor input . For example , in may be used with table top games , CAD , or other data this mode , the panel may display an operating system visualization software . Two devices may be networked so desktop, launcher , or other application to the user of the that two devices are showing same annotations in real world device such that the user is able to see the user interface and from different physical positions. For example , two players 30 operate or access the device . of a table top game are able to see the game board from For purposes of illustration and without limitation , with different vantage points through two different devices placed reference to FIG.9C , active shutter 102 is made transparent, on the table . The two devices are able to correlate anchor transparent display 104 is configured to display an image in points on the table so that corresponding annotations can be the correct orientation for a user to the left as depicted , and displayed on each device from the proper angle . 35 active shutter 106 is made opaque . For purposes of illustra In one implementation , with reference to FIGS. 9B and tion and without limitation , with reference to FIG . 91, active 10B , the screen is placed into a normal mode and sensors on shutter 102 is made opaque, and transparent display 104 is the right side of the device (as depicted ) are used to ascertain configured to display an image in the correct orientation for the scene and . With reference to FIG . 18A , the device 1800 a user to the left as depicted . For purposes of illustration and determines the position of any objects 1802 (whether real or 40 without limitation , with reference to FIG . 10C , transparent virtual objects or holograms that are being placed into display 202 is configured to display an image of the desktop reality ) . Then , the objects are displayed on the device at or user interface in the correct orientation for a user of the position 1810 using a normal display mode. With reference device to the left as depicted , active shutter 204 is made to FIG . 18B , the object 1802 is rendered on the opaque opaque , and transparent display 206 is inactive , or may be normal mode screen at location 1810 . 45 active such as when the device is in an ambient light mode In another implementation , with reference to FIGS. 9B as discussed below . and 10B , the screen is placed into an augmented reality Tent Configuration — Ambient Light Mode. mode and sensors on the right side of the device (as The device may drive the display in a tent ambient light depicted ) are used to ascertain the scene. Furthermore, mode . This modemay be used for various applications that sensors on the left side of the device ( as depicted ) are used 50 run on the device . The device or applications may change the to ascertain the location of the user. With reference to FIG . display to this mode based on heuristics , a user selected 18C , the device 1800 determines the position of any objects function and /or based upon sensor input. For example , in 1802 (whether real or virtual objects or holograms that are this mode, the panel may display an operating system being placed into reality ) and determines the position of the desktop , launcher , video , movie or other application to the user 1808. Then , with reference to FIGS. 18C and 18D , real 55 user of the device such that the user is able to see the user objects 1802 are visible through the screen at position 1814 interface ( from the left as depicted in FIGS . 9C and 10C ) . or a virtual object at position 1802 may be rendered on the The user may operate the device with any input device screen at location 1814 for a user at position 1808. If a real including a touch screen user interface, or a wireless key object is being annotated , then an annotation 1804 may be board and /or mouse , pen , game controller , etc. This mode displayed with appropriate relationship to the object 1802 60 may be preferable for a user that wants to minimize the given the position of the user 1808 displayed on the device surface space required on a table or tray (e.g. , when on an at position 1810 using a normal display mode . airplane ) to watch movies or read documents. This mode Stand Configuration — Unobstructed Mode . may be desirable when a user is watching a movie in low The device may drive the display in an unobstructed or light conditions and wishes to cast a light color behind the dual sided augmented reality mode as discussed herein . This 65 device to help set a mood when watching a movie etc. mode may be used for various applications that run on the For purposes of illustration and without limitation , with device . The device or applicationsmay change the display to reference to FIG . 9C , at a first time, active shutter 102 is US 10,520,782 B2 65 66 made transparent, transparent display 104 is configured to device such that the user is able to see the user interface and display an image in the correct orientation for a user to the operate or access the device . The desktop icons may be left as depicted , and active shutter 106 is made opaque . At displayed on a screen closer to the user and other objects or a second time, active shutter 102 is made opaque , active images on the desktop may be displayed on a screen further shutter 106 is made transparent , and transparent display 104 5 from the user giving a 2.5 dimensional or 3 - dimensional is configured to display a color as to illuminate the area effect. In addition , the front facing cameras may track the behind the device ( to the right as depicted ) as viewed by the user's face as discussed elsewhere to add a parallax effect user. The color may be any color or combination of colors where the objects on one screen move at a different rate than as to project an ambient light color. This process would then objects on another based on the movement and position of repeat quickly at a rate such that the user of the device does 10 the user's head relative to the screen . not perceive the flickering . For purposes of illustration and For purposes of illustration and without limitation , with without limitation , with reference to FIG . 10C , transparent reference to FIG . 10D , transparent display 202 is configured display 202 is configured to display an image in the correct to display an image of a foreground layer of the desktop or orientation for a user to the left as depicted , active shutter user interface in the correct orientation for a user of the 204 is made opaque , and transparent display 206 is config- 15 device above the device as depicted , active shutter 204 is ured to display a color or combination of colors to illuminate made transparent, and transparent display 206 is configured the area behind the device . to display an image of a background layer of the desktop or VI.B.4 Tablet Configuration . user interface in the correct orientation for a user of the With reference to FIGS . 9D (SDS screen ), 9J (SD screen ) device above the device as depicted . and 10D (DSD screen ) , a single element hinged device may 20 Tablet Configuration — Tracing Mode . be placed into a tablet configuration . In such a configuration , The device may drive the display in a tablet tracing mode . the display elements may be operated in any state or This mode may be used for various applications that run on combination of states, and /or the display may be operated in the device . The device or applications may change the any of the modes discussed herein and with some examples display to this mode based on heuristics, a user selected provided below . 25 function and /or based upon sensor input. For example , in Tablet Configuration Normal Mode. this mode, the panel may display an operating system The device may drive the display in a tablet normal mode . desktop , launcher , or other application to the user of the This mode may be used for various applications that run on device such that the user is able to see the user interface and the device . The device or applications may change the operate or access the device . The user can see through the display to this mode based on heuristics , a user selected 30 screen to a physical document placed under the screen and function and / or based upon sensor input . For example , in above object 900 or 1000. The user can trace an illustration this mode, the panel may display an operating system on the physical document on the top of the screen of the desktop , launcher, or other application to the user of the device . device such that the user is able to see the user interface and For purposes of illustration and without limitation , with operate or access the device. 35 reference to FIG . 9D , a document is placed between object For purposes of illustration and without limitation , with 900 and active shutter 106. Active shutter 102 is made reference to FIG.9D , active shutter 102 is made transparent, transparent, transparent display 104 is configured to be transparent display 104 is configured to display an image in inactive or display an image of a user interface in the correct the correct orientation for a user above the device as orientation for a user above the device as depicted , and depicted , and active shutter 106 is made opaque but in some 40 active shutter 106 is made transparent or partially transpar cases may be transparent. For purposes of illustration and ent. For example , in forms of the invention where the active without limitation , with reference to FIG . 9J, active shutter shutters have many pixels that correspond to the pixels of 102 is made opaque but in some cases may be transparent, transparent display 104 , the pixels of active shutter 106 that and transparent display 104 is configured to display an correspond to illuminated pixels of transparent display 104 image in the correct orientation for a user above the device 45 may be opaque, while pixels of active shutter 106 that as depicted . For purposes of illustration and without limi correspond to inactive (transparent ) pixels of transparent tation , with reference to FIG . 10D , transparent display 202 display 104 are transparent. For purposes of illustration and is configured to display an image of the desktop or user without limitation , with reference to FIG.9J, a document is interface in the correct orientation for a user of the device placed between object 900 and active shutter 102. Active above the device as depicted , active shutter 204 is made 50 shutter 102 is made transparent, and transparent display 104 opaque but in some cases may be transparent, and transpar is configured to display an image in the correct orientation ent display 206 is inactive but may be active in some cases . for a user above the device as depicted . For purposes of Alternatively , for purposes of illustration and without limi illustration and without limitation , with reference to FIG . tation , with reference to FIG . 10D , transparent display 202 10D , a document is placed between object 1000 and trans is inactive and therefore transparent , active shutter 204 is 55 parent display 206 , transparent display 202 is configured to made opaque but in some cases may be transparent, and display an image of the desktop or user interface in the transparent display 206 is configured to display an image of correct orientation for a user of the device above the device the desktop or user interface in the correct orientation for a as depicted , active shutter 204 is transparent, and transparent user of the device above the device as depicted . display 206 is inactive and therefore transparent. Alterna Tablet Configuration — 2.5D /3D Mode . 60 tively, for purposes of illustration and without limitation , The device may drive the display in a 2.5D or 3D mode. with reference to FIG . 10D , a document is placed between This mode may be used for various applications that run on object 1000 and transparent display 206 , transparent display the device . The device or applications may change the 202 is inactive and therefore transparent , active shutter 204 display to this mode based on heuristics, a user selected is made transparent, and transparent display 206 is config function and /or based upon sensor input. For example , in 65 ured to display an image of the desktop or user interface in this mode , the panel may display an operating system the correct orientation for a user of the device above the desktop , launcher, or other application to the user of the device as depicted . In any of the above or similar forms of US 10,520,782 B2 67 68 the invention , a user of the device is able to see the document orientation for a user of the device above the device as through the screen and may trace portions of the document depicted or it may be inactive . Transparent display 202 is using a touch screen , pen , or any other input device . used to illuminate and scan portions of the document as Tablet Configuration Scanning Mode . described above with respect to other forms of the invention . The device may drive the display in a tablet scanning 5 Scanning Operations. mode . As noted throughout, while this is discussed with Generally , the scanning operations described above are respect to one form of the invention , scanning mode can be applicable to any form of display element, screen , device used in other forms of the invention discussed throughout. This mode may be used for various applications that run on form factor, or device form factor configuration . As dis the device . The device or applications may change the 10 cussed above a user would place a document under a screen display to this mode based on heuristics, a user selected or place a document on a screen . The screen uses the function and /or based upon sensor input . For example , in emissive elements to scan the document or object and create this mode, the panel may display an operating system digital reproduction . For example , with a document placed desktop , launcher, or other application to the user of the on a SDS screen , the active shutter closest to the document device such that the user is able to see the user interface and 15 would be transparent, and the active shutter furthest from the operate or access the device . The user can see through the document may be opaque . The transparent emissive display screen to a physical document placed under the screen and would then scan the document by reverse biasing some of above object 900 or 1000. Alternatively , a document can be the missive elements at any given time, and when done the placed face down onto the top of the device. The device can device can correct the orientation of the digitized document use the emissive display screen elements in a forward bias 20 image as necessary for display of the image to the user. With state to illuminate the document while other emissive dis a DSD screen , the transparent display closest to the docu play elements can be used in a reversed bias state to sense ment may be used for scanning and the active shutter may the reflected light as to digitally scan the document into the be opaque. device . Example Usage. For purposes of illustration and without limitation , with 25 For purposes of illustration and without limitation , a user reference to FIG . 9D , a document is placed between object of a device may carry the device in tablet configuration in 900 and active shutter 106. Active shutter 102 is made her purse . Upon receiving a phone call, she may answer the transparent, transparent display 104 is configured to be call using a user interface displayed on the screen in tablet inactive or display an image of a user interface in the correct configuration normal mode. See FIGS. 9D , 9J and 10D . orientation for a user above the device as depicted , and 30 After the call, the user may need to work on a power point active shutter 106 is made transparent or partially transpar presentation at a coffee shop . She sits down and opens the ent. For example , in formsof the invention where the active device so that it is now in a laptop configuration with a full shutters have many pixels that correspond the pixels of keyboard 900 exposed and available to use. See FIGS. 9A , transparent display 104 , the pixels of active shutter 106 that 9G , and 10A . While working on the presentation , the device correspond to illuminated pixels of transparent display 104 35 is in normal mode so that other people at the coffee shop may be opaque , while pixels of active shutter 106 that cannot see the presentation from the back side of her device . correspond to inactive (transparent ) pixels of transparent When it is time to present the information to a business display 104 are transparent. In this way , a user interface can associate , she can put the device into a laptop configuration be shown in conjunction with the view of the actual docu and a sharing or a presenting mode. See FIGS. 9A , 10A . The ment. A user could , for example , draw a rectangle or other 40 business associate may see the full screen presentation on area over the document to scan . Once a scan begins, portions the back side of the device , while the user can see a presenter of the emissive display 104 are used to illuminate the interface including notes and controls . After the presenta document while other portions of emissive display 104 are tion , the user takes some notes on paper and then operates used to sense the reflected light. For purposes of illustration the screen in scanning mode to digitize the information and and without limitation , with reference to FIG . 9J, a docu- 45 save it on the device . When entering her car after the ment is placed between object 900 and active shutter 102 . meeting , the user requests directions on the device in tablet Active shutter 102 is made transparent, and transparent configuration normal mode . After receiving the directions display 104 is configured to display an image in the correct and showing a route preview overlaid on a map , the user orientation for a user above the device as depicted in the adjusts the device so that it is in upright laptop or stand same fashion as described with respect to FIG . 9D . For 50 configuration . See FIGS . 9A , 9B , 9H , 10A , and 10B . The purposes of illustration and without limitation , with refer user places the device on her dashboard and starts driving . ence to FIG . 10D , a document is placed between object 1000 The device detects its configuration , that the user has started and transparent display 206 , transparent display 202 is driving (based on , for example , a motion coprocessor) and configured to display an image of the desktop or user that the user's face is a distance away indicating that the interface in the correct orientation for a user of the device 55 device should switch to an augmented reality mode. The above the device as depicted , active shutter 204 is transpar device switches to an augmented reality mode and displays ent, and transparent display 206 is inactive and therefore a highlighted path and other information to the user as she transparent. Transparent display 206 may be used to illumi drives along the path . nate portions of the document while simultaneously scan Any Combination of Elements , Display Modes , Form ning with other emissive display elements that are reverse 60 Factors and Configurations are Contemplated . biased. Alternatively , for purposes of illustration and without Any of the modes discussed above or elsewhere herein limitation , with reference to FIG . 10D , a document is placed may be used in any configuration . Many of the modes are bon transparent display 202 face down , transparent display discussed only with respect to certain device form factors 202 is inactive and therefore transparent or may display a and configurations and not duplicated throughout for brev user interface , active shutter 204 is made transparent, and 65 ity . However , it is contemplated that any mode or combi transparent display 202 or 206 may be configured to display nation of element states can be used with any device in any an image of the desktop or user interface in the correct configuration . US 10,520,782 B2 69 70 VI.C Other Single Element Form Factors. active shutter 102 are opaque may be different from the With reference to FIGS . 9E - 9F and 10E - 10F , many other duration of time that elements 102 , 104 , and 106 are form factors are possible for devices incorporating a single substantially simultaneously transparent. In a virtual reality multi -mode display panel . With reference to FIGS. 9E and mode , active shutter 102 is opaque and active shutter 106 is 10E , one such device is an extra wide display . Extra wide 5 transparent, or it may operate according to any of the displays may be incorporated into simulators , game cabi combinations of element states discussed above . Transparent nets , gaming devices, or any other type of device . With display 104 renders an appropriate image for the user of the reference to FIGS. 9F and 10F , another such device is virtual device to provide a virtual reality experience to the user . or augmented reality glasses. Both types of devices are There may also be operated in a sharing mode or presenta discussed in more detail below . 10 tion mode to substantially simultaneously display the virtual Other Form Factors — Extra Wide Display (SDS Screen ) . world to the wearer of the device, and a " regular " 2D With reference to FIG . 9E , an extra wide display may be rendering of the world , or other information about the game constructed with a SDS display panel . The device contains to an observer looking at the glasses on the wearer of the mirrorsa transparent 904 and display 908. The104 ,display active may shutters be operated 102 and in106 a mode , and 15 device ( e.g., gamemap , game stats, player vitals, game logo , like presentation or sharing mode (see , e.g. , FIGS . 5C , 5D , player name, player standing , etc.) 7C and 7D ) with the image that is displayed to the left being As another alternative , with reference to FIG . 9F , the the left half of, for example , a panoramic image that is being device may be constructed with single SD or multiple SD displayed and the image that is displayed to the right being display panels . For example , active shutter 106 may be the right portion of the panoramic image . An observer 20 eliminated from the device 906. In an augmented reality looking at the extra wide display will observe the full mode , active shutter 102 may be transparent or alternate panoramic image. between transparent and opaque as discussed above to Other Form Factors — Extra Wide Display ( DSD Screen ) . maintain privacy of the information displayed on transparent With reference to FIG . 10E , an extra wide display may be display 104. In a virtual reality mode, active shutter 102 is constructed with a DSD display panel . The device contains 25 opaque and transparent display 104 displays an image to the an active shutter 204 , transparent displays 202 and 206 , and wearer of the device . mirrors 1004 and 1008. The display may be operated in a mode like presentation or sharing mode ( see, e.g., FIGS . 6C More generally , as discussed through the disclosure , any and 8C ) with the image that is displayed to the left being the type number and combination of multi -mode displays may left half of, for example , a panoramic image that is being 30 be used . Any of the element states for SDS or SD elements displayed and the image that is displayed to the right being or any of the display modes discussed above for SDS or SD the right portion of the panoramic image. An observer displays may be used with this form of the invention . This looking at the extra wide display will observe the full was said above , and need not be said further through this panoramic image . disclosure for any form factor or configuration or various Other Form Factors— Glasses (SDS Screen ) . 35 form of the invention , but is done so here out of an With reference to FIG . 9F , a pair of virtual reality or abundance of caution . mixed reality glasses may be constructed with a single SDS Other Form Factors— Glasses (DSD Screen ) . display panel. For example , active shutter 102 is placed on With reference to FIG . 10F , a pair of virtual reality or mixed reality glasses may be constructed with a single DSD theinside outside of the of glasses the glasses , and , activetransparent shutter display 106 is placed104 is placedon the 40 display panel. Alternatively, as discussed elsewhere herein , between the two active shutters . There may be additional the single DSD display panel may have multiple shutter optics between the user and the screen in certain formsof the regions, or display regions. Still further , the device may be invention . constructed with more than one DSD display panels ( e.g. , Alternatively , as discussed elsewhere herein , the single one per eye ). SDS display panel may have multiple shutter regions, or 45 With reference to FIG . 10F, a pair of virtual reality or display regions . For example , with reference to FIG . 9F , mixed reality glasses may be constructed with a single DSD active shutters 102 and 106 may each have separate regions display panel . For example , transparent display 202 is for the right and left eyes (i.e. , they may be split above the placed on the outside of the glasses, transparent display 206 bridge of the nose into left and right active shutter elements .) is placed on the inside of the glasses, and active shutter 204 The transparent display 104 can be operated in multiple 50 is placed between the two transparent displays. In an aug modes simultaneously in different regions (such as in a left mented reality mode , at least one of the transparent displays region and a right region ) in coordination with one region 202 and 206 are displaying annotations according to one of ( e.g., the right half ) of active shutters 102 and 106 , and the augmented reality modes described above and active another region (e.g. , the left half ) of active shutters 102 and shutter 204 is transparent. There may also be a private AR 106. The right side and left side of the device may imple- 55 mode as discussed above . Alternatively , as discussed else ment different display modes simultaneously or the same where herein , the single DSD display panel may have display mode but out of phase ( e.g. , 180 degrees out of multiple shutter regions, or display regions . For example , phase ) . Still further, the device may be constructed with active shutters 204 may have separate regions for the right more than one SDS display panels ( e.g., one per eye ). and left eyes ( i.e., itmay be split above the bridge of the nose In an augmented reality mode , both active shutters 102 60 into left and right active shutter elements .) The transparent and 106 are transparent and transparent display displays an display elements 202 and 206 can be operated in multiple image in the correct orientation for a user of the glasses. In modes simultaneously in different regions (such as in a left a private AR mode, wherein shutter 102 alternates between region and a right region ) in coordination with one region transparent and opaque states , and is opaque during the time ( e.g., the right half) of active shutter 204, and another region that transparent display 106 displays an image to the user of 65 (e.g. , the left half ) of active shutter 204. The right side and the device . In this mode (or any other mode as discussed left side of the device may implement different display herein ), the duration of time that transparent display 104 and modes simultaneously or the same display mode but out of US 10,520,782 B2 71 72 phase ( e.g., 180 degrees out of phase ). Still further , the user interface and operate or access the other display screen device may be constructed with more than one DSD display that is horizontal (or the “ bottom ” screen ). An observer on panels ( e.g. , one per eye ) . the back side of the device may only see the active shutter Any of the element states for DSD elements and modes element of the top screen which obscures or blocks the discussed above for DSD displays may be used with this 5 reverse image of the display thus maintaining the user's form of the invention . This was said above , and need not be privacy. The “ bottom ” panel may be in a normal mode , said further through this disclosure for any form factor or unobstructed mode , or any other mode and display a key configuration or various form of the invention , but is done board user interface to the user of the device . so here out of an abundance of caution . For purposes of illustration and without limitation , with VI.D Form Factor: Dual Element Hinged . 10 reference to FIG . 11A , active shutters 102 and 1106 are With reference to FIGS. 11A - 11E and 12A - 12E , one form made opaque, transparent display 104 is configured to factor is a device that incorporates two multi -mode display display an image ( e.g., of a desktop or user interface ) in the panels coupled by a hinge in a hinged device. The device correct orientation for a user to the right as depicted , includes a main body and sensors devices of this form factor transparent display 1104 is configured to display an image may incorporate any of the inventive display elements and 15 ( e.g. , of a keyboard or input user interface ) in the correct display modes discussed above . With reference to FIGS. orientation for a user to the right as depicted , and active 11A - 11E , a dual element hinged device with two SDS shutters 106 and 1104 are made transparent. For purposes of displays includes at least on one of the display panels : an illustration and without limitation , with reference to FIG . active shutter 102 , a transparent display 104 , an active 12A , transparent displays 202 and 1206 are independently shutter 106 , and on the other display panel: an active shutter 20 either inactive or made to display an image such as branding 1102 , a transparent display 1104 , and an active shutter 1106 . or a decorative image , active shutter 204 is made opaque , The two SDS display panels are coupled by a hinge 1108 . active shutter 1204 is optionally made opaque, transparent With reference to FIGS. 12A - 12E , a dual element hinged display 206 is configured to display an image ( e.g., of a device with a DSD display includes at least on one of the desktop or user interface ) in the correct orientation for a user display panels : a transparent display 202 , an active shutter 25 of the device to the right as depicted , and transparent display 204 , a transparent display 206 , and on the other display 1202 is configured to display an image ( e.g., of a keyboard panel: a transparent display 1202 , an active shutter 1204 , a or input user interface ) in the correct orientation for a user transparent display 1206 (such as a device body or key of the device to the right as depicted . With either type of board ) . The two display panels are coupled by a hinge 1208 . display element, the “ bottom ” panel may show a keyboard With reference to FIGS. 17A - 17B , some illustrations of an 30 or other input interface , or it may show a second screen for exemplary dual multi -mode display device 1700 is provided content ( e.g., when no user input is needed , or when it is in a small tablet and large tablet configuration , respectively . when used with external input device such is keyboard , A flexible display device may also behave in the same or mouse or pen ) . analogous manner as a dual element hinged device as Upright Configuration Presentation or Sharing Mode . described herein , particularly when it is configured with 35 As described above with respect to other form factors multiple regions ( e.g. , at least one on each side of the bend ( e.g. , single element hinged ) , at least the top panel of a dual in the screen ). panel device may be placed in a presentation mode or The control electronics may be built into the hinge, a sharing mode. The bottom panel may be placed in any mode. border around the screens, or placed discretely in the screen Upright Configuration - Augmented Reality Mode. between the pixel elements such that they are imperceptible 40 As described above with respect to other form factors to the human eye. An inertial measurement unit ( IMU ) is ( e.g. , single element hinged ) , at least the top panel of a dual built into each movable element so that the absolute and panel device may be placed in an augmented reality mode . relative orientation of each element of the device can be The bottom panel may be placed in any mode . ascertained and used to determine the configuration and Upright Configuration — Tracing or Scanning Mode . context of the device in order to help choose a display mode 45 The device may drive the display in an upright tracing and orientation . Alternatively , one IMU may be placed in the mode . This mode may be used for various applications that device at a known location , and a hinge position detector run on the device . The device or applications may change the could be used to ascertain the device configuration and display to this mode based on heuristics, a user selected orientation . function and /or based upon sensor input. For example , in VI.D.1 Upright Laptop Configuration . 50 this mode , the top panel may be operated in any mode . The With reference to FIGS. 11A ( dual SDS screen ), 12A bottom panel is operated in , for example , an “ unobstructed ” ( dual DSD screen ), a dual element hinged device or flexible or augmented reality mode such that a paper can be placed device ( a device with a single flexible panel ) may be placed underneath the display and traced with the touch screen , pen , into an upright laptop configuration . In such a configuration , mouse input, or any other input device . The bottom panel the display elements may be operated in any state or 55 may also display some form ofuser interface over the object combination of states, and /or the display may be operated in on the transparent display . Alternatively , as described else any of the modes discussed herein and with some examples where herein , either panel may be operated in a scanning provided below . mode . Upright Configuration Normal Mode. Upright Configuration — Unobstructed Mode . The device may drive one or both displays in a normal 60 As described above with respect to other form factors mode. This mode may be used for various applications that ( e.g. , single element hinged ) , at least the top panel of a dual run on the device . The device or applicationsmay change the panel device may be placed in an unobstructed mode . The display to this mode based on heuristics, a user selected bottom panel may be placed in any mode. For example , the function and /or based upon sensor input. For example , in top panel may appear translucent to the audience of the this mode, the upright (or “ top ” screen ) panel may display 65 device operator to maintain privacy of the data on the screen an operating system desktop , launcher, or other application but appear normal or translucent to the device operator so to the user of the device such that the user is able to see the that it does notblock the operator from the audience on other US 10,520,782 B2 73 74 side of screen . As another example , the top panelmay appear panel device may be placed in an unobstructed mode . The translucent to both parties while showing image in the bottom panel may be placed in any mode. For example , the correct orientation on both sides so both parties can see each top panel may appear translucent to the audience of the other and simultaneously observe the information on the device operator to maintain privacy of the data on the screen display of the device . 5 but appear normal or translucent to the device operator, so VI.D.2 Stand Configuration . it does not block the operator from the audience on other With reference to FIGS . 11B (dual SDS screen ) and 12B (dual DSD screen ), a dual element hinged device may be side of screen . As another example , the top panelmay appear placed into a stand configuration . In such a configuration , translucent to both parties while showing image in the the top display and bottom display may be independently 10 correctother and orientation simultaneously on both observesides so boththe partiesinformation can see on each the operated in any of the modes discussed herein and with some display of the device . examples provided below . VI.D.3 Tent Configuration . Stand Configuration — Normal Mode . With reference to FIGS. 11C ( dual SDS screens) and 12C modeThe. deviceThis mode may may drive be oneused or for both various displays applications in a normal that is (dual DSD screens) , a dual element hinged device may be run on the device . The device or applications may change the placed into a tent configuration . In such a configuration , the display to this mode based on heuristics , a user selected top display and bottom display may be independently oper function and /or based upon sensor input. For example , in ated in any of the modes discussed herein and with some this mode, the upright (or “ top ” screen ) panel may display examples provided below . an operating system desktop , launcher , video player or other 20 Tent Configuration Normal Mode . application to the user to the left of the device as depicted . The device may drive one or both displays in a normal The “ bottom ” panel may be in an inactive mode , a normal mode in the tent configuration . This mode may be used for mode, unobstructed mode , or any other mode. various applications that run on the device . The device or For purposes of illustration and without limitation , with applications may change the display to this mode based on reference to FIG . 11B , active shutter 106 is made opaque and 25 heuristics, a user selected function and /or based upon sensor active shutter 102 is made transparent, transparent display input. For example , in this mode, the device may detect that 104 is configured to display an image in the correct orien the operator of the device is located to the left of the device tation for a user to the left of the device as depicted , as depicted using a camera ( it may detect and recognize the transparent display 1104 is inactive and active shutters 1102 logged in user's face using a left- facing camera ) , sensor, or and 1106 may be made transparent or opaque. For purposes 30 by detecting which panel the user is interacting with via the of illustration and without limitation , with reference to FIG . user input device ( via touch input on the left panel, wireless 12B , transparent display 202 is made to display an image to mouse movements that only make sense if the user was a user to the left of the device as depicted , active shutter 204 located on the left of the evice , etc.) . Alternatively , the is made opaque , and transparent display 206 may be inactive device may determine that the known operator is to the left , ( or may be active in other modes such as ambient light 35 and an unknown (or known audience user ) is to the right and mode ) . Active shutters 1204 is optionally made opaque or it may prompt the operator of the computer if it would like transparent, transparent display 1202 is to be inactive (or to switch to a presentation or other mode ( this is applicable may be active in other modes such as ambient lightmode ) to all forms of the invention , form factors , and configura and transparent display 1206 is configured to be inactive. tions) . Once the user is determined to be to the left, the left With either type of display element, the “ top ” panel may 40 panel may display an operating system desktop , launcher , optionally show a keyboard or other input interface , or it video player or other application to the user to the left of the may remove any displayed user input user interfaces ( e.g. , device as depicted in a normal display mode. The right panel when no user input is needed , or when it is when used with may be in an inactive mode , a normal mode , unobstructed external input device such is keyboard , mouse or pen ) . mode , or any other mode . Stand Configuration — Ambient Light Mode . 45 For purposes of illustration and without limitation , with As described above with respect to other form factors reference to FIG . 11C , active shutter 1106 is made trans (e.g. , single element hinged ) , at least the top panel of a dual parent and active shutter 1102 is made opaque, transparent panel device in stand configuration ( and optionally the display 1104 is configured to display an image in the correct bottom panel ) may be placed in an ambient light mode . The orientation for a user to the left of the device as depicted . bottom panel may be placed in any mode . For example , with 50 Transparent display 104 may be inactive and active shutters reference to FIGS . 11B and 12B , either the top or the bottom 102 and 106 may independently be made transparent or panel may be operated in a presentation or sharing mode opaque . For purposes of illustration and without limitation , ( see, e.g., FIGS. 5C , 7C and FIGS. 6A -6C , and 8A -8C ). with reference to FIG . 12C , transparent display 1206 is When the display casts light to the right of the device, the made to display an image to a user to the left of the device light may be an ambient light color that is determined to 55 as depicted , active shutter 1204 is made opaque , and trans match the content displayed on the device as described parent display 1202 may be inactive (or may be active in above. When the display casts light the left of the device , it other modes such as ambient lightmode ). Active shutter 204 is the image to be displayed in a correct orientation for a user is optionally made opaque or transparent, and transparent to the left of the device as depicted . display 206 is configured to be inactive ( or may be active in Stand Configuration — Augmented Reality Mode. 60 other modes such as ambient light mode ) and transparent As described above with respect to other form factors display 202 is configured to be inactive ( or may be active in ( e.g., single element hinged ), at least the top panel of a dual other modes such as presentation mode or sharing mode) . panel device may be placed in an augmented reality mode . With either type of display element , either the left or right The bottom panel may be placed in any mode . panel may optionally show a keyboard or other input inter Stand Configuration — Unobstructed Mode . 65 face to a user located to the left of or right of the device as As described above with respect to other form factors depicted , respectively . In addition , the device may remove ( e.g. , single element hinged ) , at least the top panel of a dual any displayed user input user interfaces ( e.g., when no user US 10,520,782 B2 75 76 input is needed , or when it is when used with external input sensor input. For example , in one such mode , the device device such is keyboard , mouse or pen ) . screens appear opaque or translucent to an audience to Tent Configuration Presentation or Sharing Mode . maintain privacy but normal (optionally with some degree of As described above with respect to other form factors transparency ) to an operator so it does not block the operator ( e.g. , single element hinged ), the dual panel device may be 5 from people on other side of screen but allows the operator placed in a presentation mode . For purposes of illustration to maintain privacy and use the device . As another example, and without limitation , with reference to FIG . 11C , active in another such mode , the screen appears translucent to both shutter 1106 is made transparent and active shutter 1102 is parties while showing the same or a different image to both made opaque, transparent display 1104 is configured to parties. This allows each party to see one another as well as display an image in the correct orientation for a user to the 10 information in a conference setting , or allows bi- directional left of the device as depicted . Active shutter 106 is made augmented reality applications. opaque and active shutter 102 is made transparent, trans VI.D.4 Small Tablet Configuration . parent display 104 is configured to display an image in the With reference to FIGS. 11D (dual SDS screen ) and 12D correct orientation for a user to the right of the device as (dual DSD screen ) , a dual element hinged device may be depicted . For purposes of illustration and without limitation , 15 placed into a small tablet configuration by folding the device with reference to FIG . 12C , transparent display 1206 is shut such that the device footprint is approximately the size made to display an image to a user to the left of the device of a single screen (plus bezel etc.) . In such a configuration , as depicted , active shutter 1204 is made opaque, and trans the display elements may be operated in any state or parent display 1202 may be inactive (or may be active in combination of states, and /or the display may be operated in other modes such as ambient light mode ) . Transparent 20 any of the modes discussed herein and with some examples display 202 is made to display an image to a user to the right provided below . All of the modes discussed below , and all of of the device as depicted , active shutter 204 is made opaque, the modes discussed with any of the dual screen hinged form and transparent display 206 may be inactive (or may be factors herein , may also be used with devices containing active in other modes such as ambient light mode ). With other combinations of displays such as a DSD -SD hinged either type of display element, either the left or right panel 25 device or a SDS -DS hinged device . may optionally show a keyboard or other input interface to Small Tablet Configuration Normal Mode. a user located to the left of or right of the device as depicted , The device may drive one or both displays in a normal respectively. In addition , the device may remove any dis mode in the tablet small configuration . This mode may be played user input user interfaces ( e.g., when no user input is used for various applications that run on the device . The needed , or when it is when used with external input device 30 device or applications may change the display to this mode such is keyboard , mouse or pen ) . based on heuristics , a user selected function and /or based Tent Configuration Ambient LightMode . upon sensor input. For example, in this mode, the device In any de, the dual screen device may be placed into may detect that the operator of the device is located on one an ambient light mode as discussed herein . In such a mode , side of the device in an orientation relative to the device as the device will display an image outward to a user on the left 35 discussed elsewhere herein . Once the user's relative location and / or right of the device as depicted using the appropriate is determined the appropriate normal display mode can be screen . When the device display modes allow for light to be activated . For example , if the user is determined to be above transmitted downward toward the table , a color may be the device as depicted , the top panel may display an oper displayed that casts an ambient light color that corresponds ating system desktop , launcher , video player or other appli to content being displayed on the screen . 40 cation to the user above the device as depicted in a normal Tent Configuration - Augmented Reality Mode . display mode with appropriate orientation (portrait , land As described above with respect to other form factors scape, right side up , upside down , etc. ) . The bottom panel (e.g. , single element hinged ), the dual panel device may be may be in an inactive mode , a normal mode, unobstructed placed in an augmented reality mode (see , e.g., FIGS. 3J, 3L , mode , or any other mode . 4A , 4C , 4E , 4K , 4M , 40 , 40 , 50 , 7E , 5F, 7F , 5G , 7G , 5H , 45 For purposes of illustration and without limitation , with 7H , 6D , 8D , 6E , 8E , 6F , 8F , 6G , 8G , 6H , 8H , 61, 81, etc.) . The reference to FIG . 11D , active shutter 102 is made transparent device may render augmented reality annotations that appear and active shutters 106 , 1102 and 1106 are optionally and under the device to users that are located to the left and right independently made opaque or transparent. Transparent dis of the device as depicted . A user to the left and a user to the play 104 is configured to display an image in the correct right are shown an appropriate rendering of the scene 50 orientation for a user above the device as depicted . Trans underneath the tent configured device. This may be used for parent display 1104 may be inactive. For purposes of illus games or 3D entertainment such as a 3D recording of a play, tration and without limitation , with reference to FIG . 12D , or a movie that is rendered in 3D and played back to users . transparent display 202 is made to display an image to a user The users may rotate the device or the 3D playback as to above the device as depicted , active shutter 204 is made change the perspective . Alternatively , both viewers can 55 opaque , and transparent displays 206 , 1202 , and 1206 may choose an independent viewpoint of their choosing . Of be inactive . course , one display may operate in augmented reality mode Small Tablet Configuration – 2.5D /3D Mode . and the other display may be inactive when there is only one The device may drive one or both displays in a 2.5D or 3D user watching the augmented reality content. display mode in the tablet small configuration . This mode Tent Configuration — Unobstructed Modes. 60 may be used for various applications that run on the device The device may drive the display in an unobstructed or to display multiple physical display layers to the user for a dual sided augmented reality mode as discussed herein . The depth effect . The device or applications may change the left and right displays may be placed in a transparent display display to this mode based on heuristics, a user selected mode as appropriate ( see , e.g., FIGS. 3K , 4Q , 5G , 76 , 5H , function and / or based upon sensor input. 7H , 63, 8J, 6K , 8K , 6L , 8L , 6M , 8M , etc.) . The device or 65 For purposes of illustration and without limitation , with applications may change the display to these modes based reference to FIG . 11D , assuming the user is above the device on heuristics , a user selected function and /or based upon active shutters 102, 106 , and 1102 are made transparent and US 10,520,782 B2 77 78 active shutter 1106 is made opaque. Transparent display 104 parent displays 206 and 1202 are is made to display an is configured to display a foreground image , and transparent image to a user above the device as depicted , active shutters display 1104 is configured to display a background image , 204 and 1204 are made opaque , and transparent displays 202 both images are displayed in the correct orientation for a and 1206 may be inactive. user above the device as depicted . For purposes of illustra- 5 Large Tablet Configuration – Augmented Reality Mode . tion and without limitation , with reference to FIG . 12C , As described above with respect to other form factors assuming a user above the device , either transparent display ( e.g., single element hinged ), the dual panel device may be 202 or 206 are made to display a foreground image to a user placed in an augmented reality or multi- layer augmented above the device as depicted , and transparent display 1202 reality mode (see , e.g., FIGS . 3J, 3L , 4A , 4C , 4E , 4K , 4M , or 1206 are made to display a background image to a user 10 40 , 4Q , 5E , 7E , 5F , 7F , 5G , 76 , 5H , 7H ,6D , 8D , 6E , 8E , 6F , above the device as depicted . Alternatively , transparent 8F , 6G , 8G , 6H , 8H , 61, 81, etc.) . displays 202 , 206 , 1202 , and 1206 may display up to four Large Tablet Configuration Tracing or Scanning Mode . layers of images to the user . Active shutters 204 and 1204 are The device may drive the display in a tablet tracing or independently made transparent or opaque as necessary to scanning mode as described elsewhere herein . reveal the active transparent displays to the user above the 15 VI.D.6 Dual Monitor Display or Book Configuration . device . For example , transparent displays 202 , 206 and 1202 With reference to FIGS . 11A and 11E (dual SDS screen ), may display three layers of images to a user above the device and 12A and 12E (dual DSD screen ) , a dual element hinged and active shutter 204 is transparent and active shutter 1202 device may be placed into a dual monitor display or book is made opaque . Transparent display 1206 may be inactive . configuration by unfolding the device such that the hinge is Small Tablet Configuration - Augmented Reality Mode. 20 substantially open and for example operated with the As described above with respect to other form factors hinge vertical relative to the operator's face . In such a ( e.g., single element hinged ) , the dual panel device may be configuration , the display elements may be operated in any placed in an augmented reality or multi -layer augmented state or combination of states , and /or the display may be reality mode ( see , e.g., FIGS. 3J, 3L , 4A , 4C , 4E , 4K , 4M , operated in any of themodes discussed herein and with some 40 , 4Q , 5E , 7E , 5F , 7F , 56 , 7G , 5H , 7H , 6D , 8D , 6E , 8E , 6F , 25 examples provided below . The objective is to provide a dual 8F , 6G , 8G , 6H , 8H , 61, 81, etc.) . monitor display or a book configuration when held like a Small Tablet Configuration — Tracing Mode and Scanning book Mode . Dual Monitor Display or Book Configuration Normal. The device may drive the display in a tablet tracing or When operating as a dual monitor display , at least one of scanning mode as described elsewhere herein . 30 the panels operate in a normal mode toward the user as VI.D.5 Large Tablet Configuration . described elsewhere herein . The panels will display content With reference to FIGS. 11E ( dual SDS screen ) and 12E or a user interface toward the user of the device . In a book ( dual DSD screen ), a dual element hinged device may be mode, the panels display book pages to a user toward the placed into a large tablet configuration by unfolding the user . device such that the hinge is open , and the device footprint 35 Dual Monitor Display or Book Configuration Present is approximately the size of both screens (plus bezel etc. ). In ing or Sharing Mode. such a configuration , the display elements may be operated When operating as a dualmonitor display , at least one of in any state or combination of states , and /or the display may the panels operates in a presentation or sharing mode as be operated in any of the modes discussed herein and with toward the user as described elsewhere herein . The panels some examples provided below . 40 will display content or a user interface toward the user of the Large Tablet Configuration — Normal Mode. device . In a book mode, the panels may display a book cover The device may drive one or both displays in a normal when projecting light away from the user , or to save power mode in the large tablet configuration . This mode may be the panes may not project light away from the user. used for various applications that run on the device . The Dual Monitor Display or Book Configuration - Aug device or applicationsmay change the display to this mode 45 mented Reality Mode. based on heuristics , a user selected function and /or based When operating in augmented reality mode , at least one upon sensor input. For example , in this mode, the device of the panels operates according to an augmented reality may detect that the operator of the device is located on one mode toward the user as described elsewhere herein . side of the device in an orientation relative to the device as DualMonitor Display or Book Configuration — Tracing or discussed elsewhere herein . Once the user's relative location 50 Scanning Mode . is determined the appropriate normal display mode can be The device may drive at least one of the displays in a activated . For example , if the user is determined to be above tracing or scanning mode as described elsewhere herein . the device as depicted , the top face of each panel may Dual Monitor Display or Book Configuration — Unob display an operating system desktop , launcher, video player structed Modes . or other application to the user above the device as depicted 55 The device may drive at least one of the displays in an in a normal display mode with appropriate orientation unobstructed mode as described elsewhere herein . (portrait , landscape, right side up , upside down, etc. ). The Example Usage . bottom face of each panel may be in an inactive mode, a For purposes of illustration and without limitation , a user normal mode, unobstructed mode , or any other mode. of a device may carry the device in a small tablet configu For purposes of illustration and without limitation , with 60 ration in her purse. See FIGS . 11D and 12D . Upon receiving reference to FIG . 11E , active shutters 106 and 1102 are made a phone call, she may answer the call using a user interface transparent and active shutters 102 and 1106 are made displayed on the screen in a small tablet configuration and opaque when the user is determined to be above the tablet as normal display mode . As she removes the device from her depicted . Transparent displays 104 and 1104 are configured purse , the device determines based on the motion and sensor to display an image in the correct orientation for a user 65 data the side of the device that the user will be most able to above the device as depicted . For purposes of illustration operate the relative orientation of the user . Based on this and without limitation , with reference to FIG . 12E , trans information , the user interface is drawn on one side of the US 10,520,782 B2 79 80 device in an orientation appropriate for the user. See , e.g., mode screen and mounts the mobile in a holding device that FIGS . 13 , 14 , and 15. After the call, the user may need to is then attached to the deck of the boat . The sailor articulates work on a power point presentation at a coffee shop . She sits the arm of the holding device to position the screen within down and opens the device so that it is now in a laptop her view of the area ahead of the boat . The mobile device is configuration . See FIGS . 11A and 12A . The device deter- 5 operated in an augmented reality mode so that it does not mines which display has been placed on the table and also block the sailor's view of the course , and the mobile device determines the upward facing direction for that display . The displays annotations such as the course line and highlights device shows a software keyboard on the upward facing the turn - point buoy to the user. To the extent the sailor's display of the horizontal double -sided display panel. While head moves substantially so that the view through the device working on the presentation , the vertical panel is placed into 10 no longer passes through the device to the course line, the a normal mode so that the user may work on the presentation device may change to a display mode that displays “ left ” or without other people at the coffee shop being able to see the “ right" instructions and information such as speed , wind presentation from the back side of her device . Alternatively , direction and distance to next turn point to the sailor so that the device may be in a sharing mode where the user is able it continues to provide relevant information . Once the sail to see her work , but third parties are only able to see a 15 or's head is again positioned to look through the device , the decorative image , logo , or advertisementon the back side of device may switch back to an augmented reality display the vertical screen . After working for a while on her power mode and user interface . point presentation , the user decides to read another chapter As another example , a head mount may be used to hold of a book . She opens a book reader application and selects the device . This may be in the form of goggles (e.g. FIGS. a book to read . She then picks up the device and holds the 20 9F and 10F ) , or a hat or other head mounted apparatus that hinge in her hand with screens to the left and right . The has material extending in front of the user's eyes with a device senses this contextual change based on , for example , holding device adapted to attach to the side of the screen of IMU sensor data and the indication the book application is the mobile device ( as to not obscure the view through the running, and the device changes to book mode . Later, when screen when the screen is operated in a transparent or it is time to present the information to a business associate , 25 semi- transparent mode) . The mount is adapted to hold the she can put the device into a laptop configuration and a mobile device far enough away from the user's eyes so that sharing or a presenting mode. See FIGS . 11A and 12A . The the user can see through the screen of the mobile device as business associate may see the full screen presentation on well as focus on the annotations that are displayed on the the back side of the device , while the user can see a presenter screen of the mobile device . interface including notes and controls . During the presenta- 30 VII . Applications . tion , the user needs to present some large images of blue There many applications can be written that take specific prints , so she places the device into a large tablet configu advantage of the inventive display elements , display modes, ration normal mode and passes the device to her business device form ors, and device configurations. associate . See FIGS . 11E and 12E . Afterward , she goes back VII.A Apps: Dual- Sided . to her desk and places the device sideways on her desk with 35 Unique applications are possible that make use of the the hinge vertical and the screens slightly bent . The device dual- sided aspects of the displays. pairs with a Bluetooth keyboard and mouse and begins Personalizing Outer Surface of the Device . operating in a dualmonitor display configuration and normal In some formsof the invention , a device that incorporates mode . a dual- sided display allows for a user to set up a digital VI.E Form Factor: Dual SD Hinged . 40 image that appears on the outward facing surface of the With reference to FIGS. 11A - 11E and 12A - 12E , the device . For purposes of illustration and without limitation , a display panels used may be modified such that one or both user may select a background image that is shown on the panels are a SD or DS element, or any other combination of exterior " lid ” or “ back ” of the device so that the user's display element types. It is apparent from the above discus device appears personalized to third parties that see the user sion of the element states, display modes, and form factors 45 working on the device. Similarly , a book application may how one would modify the operation of the device with display a book cover image on the outer display ( facing different element types and will not be repeated here for away from the user ). conciseness . Advertising on Mobile Device . VII.F Apparatus for Holding the Multi -Mode Screen . Still further , any application may add branding to the back If a device is operated in mixed or augmented reality for 50 of the device . For example , the device may superimpose a extended periods of time while the user is holding the device logo over the image that is displayed to brand the device at eye level , the user may become tired of holding the phone . ( e.g., an Apple or Dell logo is placed over an image selected In such instances , the device may be placed in an apparatus by a user to be displayed on the “ back ” of the screen ). As that can hold and position the phone with respect to the user . another example , when running a particular application , the For example , a stand has a base that is placed on a table 55 logo of the application may be displayed on the back of the ( or a clamp that can be affixed to an object ) coupled to an device . articulating arm which is also coupled to a holding device Location Based Advertising on Mobile Device . adapted to attach to the side of the screen of the mobile Location based advertising may also be implemented . For device ( as to not obscure the view through the screen when example , devices with dual - sided displays may be used at a the screen is operated in transparent or semi- transparent 60 coffee shop . The coffee shop may cause the devices to modes ) may be used . The user may place the device into the display the logo of the coffee shop or another image ( e.g. , holding device of such stand and position the stand on the Snapchat QR code ) as condition of using their Wi- Fi. table (or affix it to an appropriate object ) and articulate the Alternatively , an application or service running on the arm to obtain the desired view angle through the device device may report this to an advertising exchange and without using their hands . For purposes of illustration and 65 provide the exchange with contextual information about the without limitation , a sailor in a regatta loads the race course location of the device , the user of the device , etc. The into a sailing application on her mobile device with multi exchange may determine the demographics of the establish US 10,520,782 B2 81 82 ment based on recorded data or real- time data . The exchange interface element, or the various views may automatically then may issue an advertisement to be displayed on the alternate . After the first player's turn is over , the screens go outside screen of the device . blank and the computer is rotated 180 degrees in any axis Dual- Sided Camera Applications. ( e.g., flipped away from the first user to the second by Camera applications utilizing the dual- sided displays can 5 pushing the vertical screen away from the first user so that display a control user interface to the operator of the camera it becomes the horizontal screen ; or simply rotating the application , and on the “ back ” side of the device display an computer to the left or right) . An IMU in the device may image preview to the subject of the photograph . determine that the device orientation has changed , and then Advertising Applications on Store Display Device. display the game board to the second user in the same Store window displays can be configured to display 10 manner it was shown to the first as described above . different things to different people depending on the side Alternatively , a game or application may be displayed on from which viewed . For example , a screen can highlight a single dual- sided transparent panel placed between two things in a window display to people outside the store with players /users . Each player/ user is able to see their own game augmented reality annotations such as highlights , boxes , board , and during periods of the gameportions of the screen arrows, messages , videos, etc. Instead of the shoppers in the 15 may be transparent so that the players are able to see each store seeing the reverse/ backward image of the annotations other while other portions of the screen contain game shown to those outside while shopping in the store, the elements . Each side of the screen may display different display shows the correct orientation version of the image to images as appropriate for each user, while portions of the the user's in the store. Alternatively , the display shows a screen appear transparent to both users . For example , the different image that is more relevant to people that have 20 screen may be divided in to a top portion that is that is already entered the store. For example , to the shoppers in the transparent and a bottom portion that has game or applica store signs and information can be displayed to help the tion elements . shoppers navigate the store or provide decoration or ambi For example , a casino card game may utilize a vertical ence to the store . In forms of the invention , the window to dual- sided transparent display. The " front” of the display the store still appears transparent/ translucent so that the 25 shows information to the player , while the back of the display does not completely block the view of the outside or display shows information to the dealer . Optionally, there inside of the store while images are being displayed . As may be a horizontal display that shows a table top area another alternative , the shoppers in the store may have no where users may place bets or view game information . As information appear on the display so that the shoppers are another example , the vertical display may show information not distracted from the other items in the store, while the 30 to the players on the back of the display and information to potential shoppers outside the store are presented with the dealer on the front of the display . The top of an optional information to draw them into the store . horizontal display may show additional information to the Kiosk / Directory Applications . dealer. While playing cards it is sometimes telling to be able Kiosks can utilize the dual- sided transparent screens to to see the face of your opponents . Thus, portions of the show an interactive map or application . A kiosk may be a 35 screen between the players and dealer may be put into a screen placed in the middle of a walkway or a forum area of transparentmode so that the user's can see one another . If a a public place such as a mall or public transit station . Users player wins a hand , an animation may be superimposed on on both sides can independently use the applications avail the otherwise transparent area to provide a rewarding visual able on the device to obtain information about tenants in a effect to the player. building or transportation options, for example . When 40 VII.B Apps: Augmented and Virtual Reality . nobody is using the kiosk on one side, the kiosk can appear As discussed above , the display in combination with substantially translucent or transparent from that side while various sensors to detect device context including device displaying information to a user on the other side of the location ( e.g. , GPS , iBeacon , BLE , etc. ), object location kiosk . If both sides are unused , the kiosk can appear sub ( e.g., Tango Platform , Image Recognition , etc.) , user loca stantially translucent or transparent from both sides. Thus, 45 tion relative to device ( e.g. , Omron Face Tracking ), and user the device can “ disappear” when it is not being used leaving gaze tracking augmented reality applications are possible by the area feeling open and not closed off due to big display which the user can see the world through the transparent or blocking all of the light or the view across an otherwise translucent screen while the screen displays annotations over unobstructed area. real world objects. Game Applications . 50 Camera Application . Various gaming applications can utilize the dual -sided With reference to FIG . 19B , a camera application may transparent displays to improve gameplay . For example , the utilize various aspects of the multi -mode display device game “ Battleship ” could be operated on a dual -screen laptop 1900. For purposes of illustration and without limitation , in having two dual- sided transparent displays in a normal one form of the invention there may be no appreciable user laptop configuration . During a first player's turn , on the 55 interface on the photographer's side of the device . The horizontal display, the upward facing side of the display can device may direct the photographer to place his head so that show the positions of the first player's ships, and the the image viewed through the substantially transparent glass locations where the second player has " hit ” and “ missed ” the 1910 is the desired photo . The device measures the position first player's ships. The bottom facing side of the horizontal of the user's head relative to the device to determine the display is on the table . On the vertical display , the “ front” 60 bounds of the photo . Once the photo is taken , the digitized side facing the user shows where the first player has " hit " photo fades into view on top of the real- world image until and “ missed ” the second player's ships. The “ back " side of the screen is in a normal mode showing the picture . The the display may show the second player: ( i ) where they have picture may once again fade away until the screen is once “ hit” and “ missed ” the first player's ships, (ii ) the locations again in a transparent or augmented reality mode. In the of the second player's ships and where the first player has 65 same form or in different forms of the invention , a portion “ hit” or “missed .” The second player may be provided an of the photographer's screen may operate in an augmented option to select which view with a menu or other user reality or a transparent mode 1910 , while another portion US 10,520,782 B2 83 84 operates in an unobstructed or normal mode showing camera appropriate waypoints , points of interest, and other pertinent controls 1908. In the same form or in different forms of the information based on the contextof the device. For purposes invention , if the photographer zooms in on a subject the of illustration and without limitation , in one form of the display will transition from operating in a transparent mode invention a hiking application may operate on a device in a to operating in a normal mode so that it can display a 5 tablet configuration . The device determines the direction of representation of the zoomed image as it would be captured . components of the gravity vector ( g ) as down toward the In the same form or in different forms of the invention , the bottom of the device and also offset to an angle relative to reverse side of the device displays a preview image to the the plane of the screen based on sensor readings . The device subject of the photo so that they are able to position determines the direction of the normal vector ( N ) to the user themselves and ensure that their pose is satisfactory . Alter- 10 and determines the orientation of the user's face . With natively , the subject's side of the device may appear trans reference to FIG . 13, the user's eye orientation is determined parent or operate in any other mode . In the same form or in to be is horizontal up 1304 , and the device is being held different forms of the invention , when the subject is being bottom down 1310 , so the device is in portrait up to the top viewed in an augmented reality mode, the camera may of the device mode 1312. With reference to FIG . 15 , the annotate the real- world view such as by drawing a box 15 device is able to determine that it is at position 1520 relative around the scene to be captured or by indicating faces or to the user's position at location 1500. Thus, the tablet other items within view . In the same form or in different configuration device begins displaying information in nor forms of the invention , if the photographer's head is out of mal mode. This allows the user to operate the application to position , the camera may provide direction on how to move locate a hiking path . The device uses a location sensor and the device to get an improved viewing angle for the photo- 20 a wireless interface to query a database backend . The device graph . receives information pertaining to hiking trails near the Automotive Airplane /Vehicle Applications. location of the user . Once the user arrives at the base of the Navigation and vehicular applications may utilize various trail, the user may raise the device to look at the trail in an aspects of the multi -mode display device . For purposes of augmented reality mode. The device determines the direc illustration and without limitation , any form factor and 25 tion of components of the gravity vector ( g ) as down toward configuration discussed herein ( or otherwise ) may be uti the bottom of the device and also substantially parallel to the lized . The mobile device may be placed in any configuration plane of the screen based on sensor readings. The device on the dashboard of a vehicle . Alternatively , the multi -mode determines the direction of the normal vector ( N ) to the user displays may be built into the vehicle as part of the wind and determines the orientation of the user's face . With shield (or canopy of some aircraft) or as a separate device 30 reference to FIG . 13 , the user's eye orientation is determined such as a visor that may be moved or a display that is to be is horizontal up 1304 , and the device is being held permanently affixed to the dashboard between the user and bottom down 1310 , so the device is in portrait up to the top the windshield . of the device mode 1312. With reference to FIG . 15 , the For purposes of illustration and without limitation , in device is able to determine that it is at position 1522 relative some forms of the invention , the device displays valuable 35 to the user's position at location 1500. With reference to information relating to the vehicle as well as augmented FIG . 19A , the device 1900 begins operating in an augmented reality annotations . In an aircraft the AR annotations may reality mode. The user 1902 holds the device so 1900 and is show course , airspace, glide ring in aircraft ), points of able to look through the device at the trail. An augmented interest ( e.g., airports , flat areas , terrain warnings, destina reality annotation 1904 is overlaid on or outlines the trail . tion , other aircraft from ADS - B in ). In some forms of the 40 Other information such as the distance of the hike , elevation invention (preferably where the display is incorporated into or climb, and the number of hikers currently on the trail may a windshield or canopy ) , the vehicle is able to track the also be displayed . position of the sun and the specific location of the passengers In forms of the invention , annotations relating to a real in the vehicle (or the general location based on seat position world eventmay be overlaid on the game field based on the and average head position ) . Based on this information , the 45 context of the phone and information from an external vehicle can activate the active shutter components of the source . For purposes of illustration and without limitation , display to darken the windshield in just the locations where while watching football in real life a first down line in real it will block the sun from the eyes of each passenger . The life may be drawn on a football field utilizing an application . transparent display elements of the screen , and other por The application determines the user's position relative to the tions of the screen , can continue to show annotations . For 50 field based on an analysis of the shape of the field or based example , if the plane with the multi -mode screen windshield on location sensors . The device is then able to communicate or canopy is flying generally toward the sun , and there is wirelessly with a server that provides the current state of the oncoming traffic coming toward the plane from the sun , the sporting event and a listing of annotations that should be display may activate the shutters over the sun making a drawn with reference to the field . Based on the device black circle or oval on the canopy and then draw a diamond 55 sensors , and the information obtained from the server asso with altitude difference indication within the black dot in the ciated with the sporting event, the device is able to display location of the oncoming traffic from the perspective of the annotations on the screen of the device in an augmented pilot. This way, the pilot can see where the traffic should be reality mode . As another illustrative example without limi (but would not be able to due to the brightness of the sun ) , tation , an application written by a retailer may provide a user and the pilot will not be blinded by the sun . While the 60 interface for a user to search for an item . The user may illustrative example was given with respect to aircraft , this request directions to that item . Alternatively , in a mall the can be adapted and apply to any vehicular or non -vehicular user may pull up a guide application and request directions application . to a specific store . The retailer or mall's physical location Other Augmented Reality Applications . may be outfitted with an indoor positioning system . The Other application may utilize various aspects of the 65 phone is able to ascertain its location in the store . The phone multi -mode display device . Generally , devices operating in can switch to an augmented reality mode and display a path an augmented reality mode may be configured to show to the item on the shelf , or a path through the mall to a store US 10,520,782 B2 85 86 based on receiving a route or other information from a server of the short URL or viewing URL will be able to view the associated with the retailer or mall . specialized data in a web browser in a format that is more In forms of the invention , an application may switch understandable than looking at the data URL . In addition , a between different display modes based on a user's action . preview of the information may allow the recipient to touch For purposes of illustration and without limitation , the 5 the preview in a specific location of the preview image to device may annotate the night sky in an augmented reality jump to a specific viewing angle in the resulting viewer mode. When the user pinches or stretches the screen to zoom application . For example, with a spherical image , the pre the viewpoint, the device may switch from an augmented view may be the distorted 2D version of the image. The reality mode (See e.g., FIGS. 18C and D ) to a normal mode recipient may press on a particular person in the preview to where augmented reality is simulated using a camera and a 10 launch the viewing URL with parameters that orient the screen (See e.g. , FIGS. 18A and B ). Other activities or ways resulting initial view at the particular person . of switching display modes are also possible using different VII.E Messaging — Intelligent Group Chat Notifications . heuristics ( e.g., with reference to FIG . 15 , raising the phone In modern messaging applications such as iMessage on to positions 1522 , 1524 , 1510 , 1512, or 1514 to switch to an Apple brand devices, group messaging is possible whereby augmented reality mode and lowering the phone to positions 15 many people can send and receive text messages from a 1520 , 1538 , 1536 with respect to user 1500 to place the shared conversation thread . One problem with group mes phone in a normal display mode ) or user interface elements . saging is that a subset of the group may have a conversation In forms of the invention , applications may utilize the that isn't immediately relevant to every member of the multi- mode display to switch between display modes . For group . Those group message participants that are not purposes of illustration and without limitation , a location- 20 involved in a specific conversation occurring in a group based augmented -reality video game on a device may have message session are bombarded with notification after noti the premise that the user must hunt or collect ghosts that are fication as each message is received . present in the real world . You are the protagonist of the Apple with iOS8 allows people to remove themselves game, but can only see the ghosts in the real world when you from group texts. However , sometimes a usermay only want look through your device when it is in augmented reality 25 to silence the alerts while remaining part of the conversation mode. The game may alert the user when they are near a until it is relevant to the user again . In some instances , group ghost game character that is in the real world . When the user messaging applications have a " do not disturb mode ” that has the game open , the game may initially operate in a silences notifications from the conversation , but it can result normal display mode when the user is in the menu system of in missed messages to the user if the user does not continue the game, but when the game is in a state where the user is 30 to monitor the thread . There needs to be a more intelligent supposed to look for ghosts , the display mode may be way of notifying group chat members as to prevent exces programmatically changed to an augmented reality display sive irrelevant notifications. mode . Alternatively , the display mode may change based on The invention is to allow group text recipients to “ snooze” heuristics or a user interface object as discussed above . alerts on the group until someone sends a message in the Again , as noted throughout, the headings and examples 35 group text with the name of the person or a nickname of the are not limiting . Thus , the functionality described above person in the body of the message. The user's phone can with respect to any heading or example used to describe an then make an available tactile or other signal to alert the user application ( i.e., vehicular, gaming , sporting , hiking , that the group text may once again be relevant. astronomy, etc.) is not limited to those types of applications For example , a family creates a group text with four and may generally be used with any application and any 40 members (mother , father, brother and sister ) . Three of the appropriate display device . members are discussing an issue that is not relevant to the VII.D Messaging Sharing Specialized Sensor Data . fourth member of the group . As an example , the brother is Users that have specialized sensors or cameras (whether talking to the mother and father about an event he is going part of a mobile device or standalone ) sometimes have a to . The sister can snooze the conversation to stop receiving tough time sharing the data easily through a messaging 45 text message alerts (but continues to receive the messages application . For example , a user with a 360 or spherical themselves) . camera must install a specialized application to use the After snoozing the father types a message with the sister's camera to both take photos and view the resulting spherical name, initials , or nickname, noun of direct address, or image. Someone without the camera cannot view the image relationship to the group in the body of a message : " What as an image sphere . A 2D representation may be sent of the 50 does your sister think ,” “ What does Maddy think ,” image to other people through a messaging or email appli “ Maddy, ” etc. The application or messaging server will then cation , but the image will look distorted . To properly view cause the sister to get a notification that her name was the image, the recipient needs to download the distorted mentioned . image and input it into a specialized viewing application . The “ snooze” mode may also time out after a period of In forms of the invention , mechanisms exist to facilitate 55 inactivity on the group message such that once the sub sharing of specialized content such as spherical images . conversation is done for a period of time (lull period ) snooze When the user wishes to share a spherical image that has will be removed from the sister's phone and any subsequent been taken ( or other specialized content that has been message including a general message will be notified to the obtained ), the data is uploaded to a server and a data URL whole group . For example , the lull period length may be is obtained . The user's device then generates a viewing URL 60 determined based on a set period, or an average period by to a web -based viewing application for the specialized between messages of a sub conversation , a clustering algo data . As part of the viewing URL to the viewing application , rithm , or a neural network trained to determine clusters of a parameter including at least a portion of the data URL . The conversations. As one example , if the sub conversation on viewing URL may be shortened into a short URL and then average had messages every 30 seconds , then a time period inserted into a message. The webpage associated with the 65 longer than 30 seconds may be chosen as a snooze reset time viewing URL may contain metadata that allows the mes for that conversation . A statistical analysis of the time saging app to show a preview of the information . A recipient between each message in a sub conversation can be done and US 10,520,782 B2 87 88 only when there is a statistically considerable time delay ary — at stop light, drive through , parking space ) , biking, between messages ( e.g. , 2 or 3 standard deviations ) might biking (but stationary ) , etc. displayed adjacent to the contact the snooze be reset. name in a contact list or messaging application . A message As another example , on a group chat with an extended sender can then decide whether to send a message based on family , a cousin may announce that a television show that 5 the current activity reported by a recipient's device . they have been working on is going to air during the week . In some forms of the invention , the motion activity of the Others may respond to the cousin asking about some par recipient may change the behavior of the sending device . For ticulars with respect to the show . Group chat participants example : ( 1 ) if the recipient is walking , stationary, or driving that are not immediately responding may elect to snooze (but stopped ) the sender messaging app may behave as notifications until a later time so that they can review the 10 normal; ( 2 ) if the recipient is driving , the sender messaging messages later but not be interrupted as the messages are app may prominently suggest calling , and /or warn that the received . user may not see the message until later ; (3 ) alternatively , if For purposes of illustration and without limitation , there the recipient is driving , the sender messaging app may queue is a group chat with five members , Alfred , Betty , Curtis , the message for sending once the recipient messaging app David and Emily . Alfred sends message & everyone notified 15 indicates that the recipient is not driving (or is driving but that Alfred sent message to the group chat . Betty responds stopped ) ; ( 4 ) if the recipient is biking , the sender messaging to Alfred in the group chat . In one form of the invention app may behave as normal , or it may warn the sender that everyone is notified . In another form of the invention , the recipient may not see the message until later. Any because Betty responded within a brief time to Alfred , only combination of the above states with the above actions may Alfred is notified but everyone in the group chat is able to 20 be implemented . see the response. In yet another form of the invention , Betty In some forms of the invention , the motion activity of the uses a noun of direct address that is mapped to Alfred , and recipient may change the behavior of the receiving phone as because the message was directed to Alfred only Alfred well . For example : (1 ) if the recipient is walking , stationary , receives a notification even though it was sent to the group or driving ( but stopped ) the recipient messaging app may message thread . 25 behave as normal ; (2 ) if the recipient is driving, the recipient For purposes of illustration and without limitation , in messaging app may defer a notification of a new message forms of the invention , while there is an active conversation until the recipient is driving (but stopped ) , stationary , or the non - participants are notified after a time delay that walking , or generally not driving ; ( 3 ) if the recipient is several people are chatting . In forms of the invention , biking, the recipient messaging app may behave as normal , non -participants are not notified unless mentioned by name. 30 or it may defer a notification of a new message until the user For example , Alfred & Betty are chatting, and Betty men is biking (but stopped stationary ) , stationary , or walking . tions Curtis using a noun of direct address associated with Any combination of the above states with the above actions Curtis ; as a result, Curtis will receive a notification . may be implemented . Still further, the group messaging application may auto VII.F Messaging - Sharing References to Content matically determine that a “ snooze ” mode should be enacted 35 Encountered in the Real World . based on the behavior of the user. For purposes of illustra On occasion , users of mobile devices encounter compel tion and without limitation , the messaging application may ling content in the real world such as a song playing on the detect that the user viewed the group message thread or the radio , or a video clip of the news or a funny segment from notification of a message from the group thread and then a broadcast television show . In some instances, the user changed the focus to a different application or ignored the 40 would like to share the content with a friend or family notification . Based on the implicit disinterest, the messaging member , but it is too time consuming for the user to track application can snooze notifications until a new conversation down a reference to the content. The user may message the starts (after a lull in the conversation ) or until the recipient friend or family member that they saw something that was is referred to by a noun of direct address associated with the relevant to the friend , but the friend may not be able to see recipient ( i.e., “ I wonder what David thinks about all of 45 the content due to the difficulty and uncertainty of being able this ?' ' ) . to find the content based on the user's description . Content Furthermore, messaging applications are used more and producers miss out on the opportunity to obtain more more while people go about their daily lives . Often , people viewers or listeners for their content. receive messages while they are busy and even when they To solve this problem , content may be embedded with are driving . Pressure to respond promptly may encourage 50 sub -audible tones or visual identifying symbols to facilitate people to make poor decisions about texting while they are sharing of the content. For example , music or video content busy . Increased transparency into the current activity of a may have a sub - audible tone that is detectable by a mobile potential message recipient may encourage message senders device but that cannot be heard by humans. The sub -audible to refrain from sending messages until a recipient is free . tone encodes an identifier of the content, and that changed Furthermore , messaging applications can intelligently sup- 55 for : each song that is played , scene of a TV show , news story press messages received while a user is busy until a user is on a news broadcast, etc. Alternatively , video news segments able to safely turn their attention to responding to a message. may include a QR code in the bottom of the screen that Using a motion processor chip , accelerometer , inertial changes by the segment and links to the broadcasted seg measurement unit , and APIs such as Core Motion to inte ment on an internet accessible website or video sharing site grate with a messaging application ( e.g., iMessage, Hang- 60 (e.g. , YouTube ). The tone, QR code, or other identifier may outs , etc.) , a current activity indicator can be associated with encode a URL , or a reference number, string , hash or other a user's account or phone number. The motion activity is identifier that can be used to obtain the URL or other address shared with contacts either directly or through a central of the content. server , for example . Any method of information transfer In addition , metadata associated with the content ( e.g. , may be used . For purposes of illustration and without 65 Source: TV , Radio , Movie , etc.; Author Name; Show Title , limitation , contact names may have an indicator such as an Channel , Subject, Category, Duration , Date , Time, Key emoji for walking, stationary , driving, driving (but station words, Segment or Clip Name, Tags , Hashtags, etc. ) may be US 10,520,782 B2 89 90 included in the sub -audible information , QR code , or may be that segment, or if the user watched the segment but the accessible from a remote server using the information in the phone did not detect a code then the user could still share the reference identifier . The metadata may be used by a mes prior segment. saging application to intelligently present the identifier for In some forms of the invention , a user can aim camera at sharing based on the subject matter of the conversation . 5 QR or other visual code on TV to get a menu to appear with For purposes of illustration and without limitation , the a list of current and prior segments that were associated with device determines a digital reference to the content the user the airing of the show , so she can link to a past segment in is exposed to in the real world . First , the device may detect a textmessage without having to try to find it on a web site . a sub - audible tone while the user is watching a television In some forms of the invention , a menu on the device is show . Alternatively , while watching TV , a user can aim the 10 able to continue watching a segment on their mobile device camera at the TV to detect a QR or other visual code. As when they have to leave the viscinity of a television . For another alternative , the phone may determine the reference example , a user is sitting at table watching TV . TV broadcast to the content based a visual or audible fingerprint of the has sub audible tone encoding an identifier or an address. information or an artificial intelligence algorithm . Next, The phone detects a sub -audible tone and determines when the user is sending a message, the messaging appli- 15 address for the content. The phone then is able to display in cation can access the stored data relating to real -world a contextual menu that can be revealed or that presents itself . content to which the user has been exposed . The messaging The user may pull up or down from an edge of the device to application may have a content linking menu ( e.g., a user reveal the menu , or the menu may only appear during the interface similar to either an emoji keyboard , address book period of time the sub -audible tone is being received from listing , or autocorrect menu that appears above the key- 20 the phone. In some forms of the invention , the contextual board , etc. ) that shows all current and recent content that the menu may appear after detecting the sub -audible tone iden user has been exposed to . The listing may include a screen tifying a television show and there is a detected motion of shot or animation of the video clip or a preview of a song the device from a motion processor. For purposes of illus reference . The application may also suggest the current or tration and without limitation , a user is sitting in front of a most recently detected content based on an analysis of a 25 television watching a show that contains a sub -audible tone typed message . For example , if a user starts to draft a encoding an identifier or an address . The user's device message and types “ Turn on channel 15 ... “ On the detects the sub - audible tone and determines address for the news ... ” the messaging application may suggest a list of content. When the user gets up leave , the device may display recent content that was detected ormay suggest a filtered list a message asking if the user wishes to " continue watching of recent content that was detected where the filtering is 30 [ content title ] · or “ add the content to a continue applied to limit the recent content references to those that watching list. " relate to news programs or programs on channel 15. In As those skilled in the art will appreciate, many aspects of addition , the device may also keep a history of content the the invention , and the various forms of the tion , can user has been exposed to in the real world . This can be beneficially be practiced alone and need not be coupled searched based a longer history of the content of a message 35 together . Unless specifically stated otherwise , no aspect of and the reference suggested as the message is typed . To the the invention should be construed as requiring combination extent, a mother creates a text message for her son and types with another aspect of the invention in practice. However , “ last week on Dr. Oz ... ," the mother's phone is able to those skilled in the art will also appreciate that the aspects bring up a listing of the segments she was exposed to based of the invention may be combined in any way imaginable to on the name of the show and the date “ last week .” As the 40 yield one of the various forms of this invention . For pur mom continues typing “ last week on Dr. Oz he said some poses of illustration and without limitation , one form of the thing about low carb ... ” the lists is further filtered to limit invention may consist only of display elements , while the list to segments that relate to diet or carbohydrates. This another form of the invention may comprise many display filtering can be assisted with artificial intelligence and elements operating in different display states or modes , natural language processing algorithms. Once she sees the 45 while another form of the invention may relate to a form appropriate segment information , the mother selects the factor and device configuration but not necessarily the appropriate segmentwhich is then inserted as an attachment display elements or applications, while another form of the or this link associated with the message . Instead of receiving invention may relate to an application that runs on a device , a vague description with no way to easily view the content while yet another form of the invention includes any one or that is being discussed , the son is able to simply click the 50 more of the features discussed above . link to see the information his mother wanted to share . In some forms of the invention , when the description in a I claim : text message is sufficiently detailed for the matching algo 1. A multi -mode dual- sided display element comprising : rithm the device may automatically hyperlink the message . a ) a first active shutter; For example , if the mother wrote a message referring to a 55 b ) a transparent display pixel; and , Cooking Segment on the Today Show with Hoda during the c ) a second active shutter; Last Hour of the show , the phone may automatically hyper wherein the first active shutter is independently electri link the segment in the text message thread without mother cally operable to transition between a substantially having to select it from a contextualmenu . The mother could transparent state and a substantially opaque state , choose to remove the hyperlink before sending by selecting 60 wherein the second active shutter is independently elec a user interface element on the preview image that is inserted trically operable to transition between a substantially into the message. transparent state and a substantially opaque state , Further , in some forms of the invention , a user can reveal wherein the transparent display pixel is substantially a keyboard and have an option to insert a link or reference transparent and is independently electrically operable to the segment that is currently detected on a nearby TV , and 65 to transition between a state in which the pixel gener the past segments that were aired on that show . Thus, a user ates light and a state in which the pixel does not that missed a segment of the show could go back and watch generate light, and US 10,520,782 B2 91 92 wherein the first active shutter is placed on one side of the b ) the interface to the display controller configured to transparent display pixel and the second active shutter subsequently : is placed on the opposite side of the transparent display i . instruct the first active shutter to transition to or pixel maintain a substantially transparent state ; 2. The multi -mode display element of claim 1, further 5 ii . instruct the second active shutter to transition to or comprising : maintain a substantially transparent state ; and , a ) an interface to a display controller configured to : iii. instruct the transparent display pixel to transition to i. instruct the first active shutter to transition to or or maintain a state in which the transparent display maintain a substantially opaque state ; pixel is not generating light . 10 7. The multi- mode display element of claim 1 , further ii . instruct the second active shutter to transition to or comprising : maintain a substantially transparent state ; and , a ) an interface to a display controller configured to : iii . instruct the transparent display pixel to generate i . instruct the first active shutter to transition to or light corresponding with an image that is to be maintain a substantially opaque state ; displayed in the direction of the second active shut 15 ii . instruct the second active shutter to transition to or ter . maintain a substantially transparent state ; and , 3. The multi -mode display element of claim 1 , further iii. instruct the transparent display pixel to generate comprising: light corresponding with an image that is to be a ) an interface to a display controller configured to : displayed in the direction of the second active shut i. instruct the first active shutter to transition to or 20 ter ; maintain a substantially transparent state ; b ) the interface to the display controller configured to ii . instruct the second active shutter to transition to or subsequently : maintain a substantially transparent state ; and , i . instruct the first active shutter to transition to or iii. instruct the transparent display pixel to generate maintain a substantially transparent state ; light corresponding with an image that is to be 25 ii . instruct the second active shutter to transition to or displayed in the direction of the first active shutter. maintain a substantially transparent state ; and , 4. The multi -mode display element of claim 1 , further iii . instruct the transparent display pixel to transition to comprising : or maintain a state in which the transparent display a ) an interface to a display controller configured to : pixel is not generating light; i. instruct the first active shutter to transition to or 30 c ) the interface to the display controller configured to maintain a substantially transparent state ; subsequently : ii . instruct the second active shutter to transition to or i . instruct the first active shutter to transition to or maintain a substantially transparent state ; and , maintain a substantially transparent state ; iii. instruct the transparent display pixel to transition to ii . instruct the second active shutter to transition to or or maintain a state in which the transparent display 35 maintain a substantially opaque state ; and , pixel is not generating light. iii. instruct the transparent display pixel to generate 5. The multi- mode display element of claim 1 , further light corresponding with an image that is to be comprising : displayed in the direction of the first active shutter. a ) an interface to a display controller configured to : 8. The multi -mode display element of claim 1 , further i . instruct the first active shutter to transition to or 40 comprising : an interface to a display controller of a folding maintain a substantially opaque state ; mobile device that is configured to determine that the device ii . instruct the second active shutter to transition to or is in a folded state based on a sensor input ; maintain a substantially transparent state ; and , wherein the face of the first active shutter faces outward iii. instruct the transparent display pixel to generate from the device in the folded state , and the face of the light corresponding with an image that is to be 45 second active shutter is contained within the device in displayed in the direction of the second active shut the folded state ; ter; wherein , based at least in part on the display controller's b ) the interface to the display controller configured to determination that the folding mobile device is in the subsequently: folded state: i . instruct the first active shutter to transition to or 50 the first active shutter is configured to transition to or maintain a substantially transparent state ; maintain a substantially transparent state ; ii . instruct the second active shutter to transition to or the second active shutter is configured to transition to maintain a substantially opaque state ; and, or maintain a substantially opaque state ; and , iii . instruct the transparent display pixel to generate the transparent display pixel is configured to generate light corresponding with an image that is to be 55 light corresponding with an image that is to be displayed in the direction of the first active shutter. displayed in the direction of the first active shutter. 6. The multi- mode display element of claim 1 , further 9. The multi- mode display element of claim 1 , further comprising : comprising: an interface to a display controller of a folding a ) an interface to a display controller configured to : mobile device that is configured to determine that the device i. instruct the first active shutter to transition to or 60 is in an unfolded state based on a sensor input; maintain a substantially opaque state ; wherein the face of the first active shutter faces outward ii . instruct the second active shutter to transition to or from the device in the folded state , and the face of the maintain a substantially transparent state ; and, second active shutter is contained within the device in iii. instruct the transparent display pixel to generate the folded state ; light corresponding with an image that is to be 65 wherein , based at least in part on the display controller's displayed in the direction of the second active shut determination that the folding mobile device is in the ter ; unfolded state : US 10,520,782 B2 93 94 the first active shutter is configured to transition to or ii . instruct the active shutter to transition to or maintain maintain a substantially opaque state ; a substantially opaque state ; and , the second active shutter is configured to transition to iii. instruct the second transparent display pixel to or maintain a substantially transparent state ; and, transition to or maintain a state in which the second the transparent display pixel is configured to generate 5 transparent display pixel is not generating light . light corresponding with an image that is to be 14. The multi- mode display element of claim 10 , further displayed in the direction of the second active shut comprising: ter. a ) an interface to a display controller configured to : 10. A multi- mode dual- sided display element comprising : i. instruct the first transparent display pixel to generate a ) a first transparent display pixel ; 10 light corresponding with an image that is to be b ) an active shutter ; and , c ) a second transparent display pixel; displayed in the direction away from the active wherein the active shutter is independently electrically shutter relative to the first transparent display pixel ; operable to transition between a substantially transpar ii . instruct the active shutter to transition to or maintain ent state and a substantially opaque state , a substantially opaque state ; and , wherein the first transparent display pixel is substantially 15 iii. instruct the second transparent display pixel to transparent and is independently electrically operable generate light corresponding with an image that is to to transition between a state in which the first pixel be displayed in the direction away from the active generates light and a state in which the first pixel does shutter relative to the second transparent display not generate light, pixel. wherein the second transparent display pixel is substan 20 15. The multi -mode display element of claim 10 , further tially transparent and is independently electrically oper comprising: able to transition between a state in which the second pixel generates light and a state in which the second a ) an interface to a display controller configured to : pixel does not generate light, and i. instruct the first transparent display pixel to transition wherein the active shutter is placed in between the first 25 to or maintain a state in which the first transparent and second transparent display pixels . display pixel is not generating light; 11. The multi- mode display element of claim 10 , further ii. instruct the active shutter to transition to or maintain comprising : an interface to a display controller of a folding a substantially transparent state ; and , mobile device that is configured to determine that the device iii. instruct the second transparent display pixel to is in an unfolded state based on a sensor input; 30 transition to or maintain a state in which the second wherein the face of the first transparent display pixel faces transparent display pixel is not generating light. outward from the device in the folded state , and the 16. The multi- mode display element of claim 10 , further face of the second transparent display pixel is contained comprising : within the device in the folded state ; a ) an interface to a display controller configured to : wherein , based at least in part on the display controller's 35 i . instruct the first transparent display pixel to generate determination that the folding mobile device is in the light corresponding with a foreground image that is unfolded state : the second transparent display pixel is configured to to be displayed in the direction away from the active generate light corresponding with an image that is to shutter relative to the first transparent display pixel ; be displayed in the direction away from the active ii . instruct the active shutter to transition to or maintain shutter relative to the second transparent display 40 a substantially transparent state ; and , pixel: iii. instruct the second transparent display pixel to the active shutter is configured to transition to or generate light corresponding with a background maintain a substantially opaque state ; and , image that is to be displayed in the direction towards the first transparent display pixel is configured to the active shutter relative to the second transparent transition to or maintain a state in which the first 45 display pixel . transparent display pixel is not generating light. 17. The multi -mode display element of claim 10 , further 12. The multi- mode display element of claim 10 , further comprising : comprising : an interface to a display controller of a folding a ) an interface to a display controller configured to : mobile device that is configured to determine whether the i . instruct the first transparent display pixel to generate device is in a folded state based on a sensor input; 50 light corresponding with an image that is to be wherein the face of the first transparent display pixel faces displayed in the direction away from the active outward from the device in the folded state , and the shutter relative to the first transparent display pixel; face of the second transparent display pixel is contained ii . instruct the active shutter to transition to or maintain within the device in the folded state ; a substantially opaque state ; and , wherein , based at least in part on the display controller's 55 iii. instruct the second transparent display pixel to determination that the folding mobile device is in the transition to or maintain a state in which the second folded state : the first transparent display pixel is configured to transparent display pixel is not generating light; generate light corresponding with an image that is to b ) the interface to the display controller configured to be displayed in the direction away from the active subsequently : shutter relative to the first transparent display pixel. 60 i. instruct the first transparent display pixel to transition 13. The multi -mode display element of claim 10 , further to or maintain a state in which the first transparent comprising : display pixel is not generating light; a ) an interface to a display controller configured to : ii. instruct the active shutter to transition to or maintain i. instruct the first transparent display pixel to generate a substantially transparent state ; and , light corresponding with an image that is to be 65 iii. instruct the second transparent display pixel to displayed in the direction away from the active transition to or maintain a state in which the second shutter relative to the first transparent display pixel ; transparent display pixel is not generating light . US 10,520,782 B2 95 96 18. The multi -mode display element of claim 10 , further displayed in the direction away from the active comprising : shutter relative to the second transparent display pixel; a ) an interface to a display controller configured to : ii. instruct the active shutter to transition to or maintain i. instruct the first transparent display pixel to generate a substantially opaque state ; and , light corresponding with an image that is to be 5 iii. instruct the first transparent display pixel to transi displayed in the direction away from the active tion to or maintain a state in which the first trans shutter relative to the first transparent display pixel; parent display pixel is not generating light; ii . instruct the active shutter to transition to or maintain b ) the interface to the display controller configured to a substantially opaque state ; and , subsequently : iii. instruct the second transparent display pixel to 10 i. instruct the first transparent display pixel to transition generate light corresponding with an image that is to to or maintain a state in which the first transparent be displayed in the direction away from the active display pixel is not generating light; shutter relative to the second transparent display ii . instruct the active shutter to transition to or maintain pixel; a substantially transparent state ; and , b ) the interface to the display controller configured to 15 iii. instruct the second transparent display pixel to subsequently : transition to or maintain a state in which the second i. instruct the first transparent display pixel to transition transparent display pixel is not generating light; c ) the interface to the display controller configured to to or maintain a state in which the first transparent subsequently : display pixel is not generating light ; 20 i . instruct the first transparent display pixel to generate ii. instruct the active shutter to transition to or maintain light corresponding with an image that is to be a substantially transparent state ; and , displayed in the direction away from the active iii. instruct the second transparent display pixel to shutter relative to the first transparent display pixel ; transition to or maintain a state in which the second ii . instruct the active shutter to transition to or maintain transparent display pixel is not generating light . 25 19. The multi -mode display element of claim 10 , further a substantially opaque state ; and , comprising : iii. instruct the second transparent display pixel to a ) an interface to a display controller configured to : transition to or maintain a state in which the second i. instruct the second transparent display pixel to gen transparent display pixel is not generating light. erate light corresponding with an image that is to be