State-of-the-art in holography and auto-stereoscopic displays

Daniel Jönsson

2019-05-13 Contents Introduction ...... 3 Auto-stereoscopic displays ...... 5 Two-View Autostereoscopic Displays ...... 5 Multi-view Autostereoscopic Displays ...... 7 Light Field Displays ...... 10 Market ...... 14 Display panels ...... 14 AR ...... 14 Application Fields ...... 15 Companies ...... 16 Holografika ...... 16 Alioscopy ...... 16 Inition ...... 16 Holographic studios ...... 16 LightSpace3D ...... 16 ZSpace ...... 17 Holoxica ...... 17 Geola ...... 17 Leia Inc ...... 17 RealViewImaging ...... 18 EchoPixel ...... 18 LookingGlassFactory ...... 19 Voxon Photonics ...... 19 Zecotek ...... 20 References ...... 21

Introduction

Figure 1 Illustration of LG's goal when it comes to 3D displays (LGDisplay, 2018).

This report describes display technologies for producing three-dimensional (3D) views, holograms, without the need for glasses. The report also contains current and potential application fields as well as companies involved in 3D display technology.

A hologram is a virtual three-dimensional image of a real object. A perfect hologram is the “holy grail” of 3D display technology since it is indistinguishable from the real object independent of the view angle or viewing position. In other words, it looks like a window into another world. A hologram describes the amount of light flowing in all directions at each position in space, and therefore does not require any special glasses or other optical instruments to be viewed.

The perfect 3D display should provide the same view as if the viewer had been looking at the object without the screen. One of the most important aspects for achieving this goal is to make the viewer perceive the depth of the object even though it is displayed on a surface. The human perception of depth uses, among others, the four major cues seen in Error! Reference source not found. and described next.

Figure 2 Four important cues for perceiving depth.

Accommodation - Adjusting focus on an object in the scene through tensing/relaxing the ciliary eye muscles improves the depth perception. Effective for distances less than 2 meters. Convergence - The two eyes converge when they look at the same fixation point on a 3D object simultaneously. Based on the triangulation principle, the closer the object, the more the eyes must converge. Effective for distances less than 10 meters (Okoshi, 2012). Motion parallax - Offers depth cues by comparing the relative motion of different objects as the viewer is moving the head. Closer objects appear to move faster than those far away from the viewer. Binocular disparity (stereo) - Refers to differences in images acquired by the left eye and the right eye and can be used to triangulate the distance the object. The farther away a 3D object is, the farther apart are the two images.

We can also utilize phycological properties where the brain tricks us into interpreting depth as seen in the left figure below. The relative depth cue importance with respect to distance to the object can be seen in the right figure below.

Figure 3 Strength of different depth cues. Adapted from (Cutting, 1995).

Creating a holographic display with abilities to show dynamic content is extremely challenging because a pure holographic display would require a pixel size smaller than 1 μm, which translate to trillions of pixels on a reasonable size display screen. Even transmitting such amount of information at 24 frames per second is challenging and producing it in imaging software even more so. The following section will provide an overview of different techniques that have been proposed to solve this problem in practice as well as their pros and cons. The covered technique has been grouped into categories depending on how many view zones they support, see the figure on the next page. Parallax Barrier Two-view Lenticular Lenses

Reflection-based

Diffraction-based displays Multi-view Time-multiplexing

stereoscopic Projection-based -

Auto Volumetric

Spinning helix Light field Projector array

Holography

Auto-stereoscopic displays While stereoscopic displays produce a notion of depth by displaying a pair of images filtered through active (shutter) or passive (colored, polarized) glasses, auto-stereoscopic displays produces the same depth notion without the requirement of glasses. This survey of auto-stereoscopic displays has largely been based on the works by Urey et al. (Urey, 2011) and Jason Geng (Geng, 2013), where more details are provided.

Two-View Autostereoscopic Displays

The two-view systems utilize a single image-pair to create 3D view. The same pair can be displayed for different viewing angle to allow multiple viewers to see the same content. However, the viewers must be at the correct location to perceive a distortion-free image. Head-tracking can be used to reduce the location restriction by adjusting the stereo images to the position of the viewer(s). There are two main techniques used to create two-view autostereoscopic displays, parallax barrier and lenticular lenses.

Figure 4 Illustration of how a parallax barrier works.

Parallax barrier passes the light from the left image to the left eye at the same time as it blocks the light from the right image, and vice versa for the right image. This process is illustrated in Figure 4 where repeated viewing zones are created along the width of the display. While in theory any pixelated emissive display can be used together with a barrier, in practice, liquid crystal display (LCD) is the most commonly used one. The viewer location restriction can be removed by using a dynamic parallax barrier together with two stacked LCDs and head-tracking. This dynamic parallax system supports two viewers but has lower brightness, resolution and contrast compared to a static parallax barrier system. The most recent displays, such as Nintendo 3DS, HTC Evo 3D and LG Optimus 3D places the parallax barrier behind the pixels in front of the backlight. This exposes the entire LCD to both eyes and produces a clearer image with larger viewing angles at the expense of using 20-25% more backlight.

The pros and cons with parallax barrier solutions are:

+ Possible to switch between 2D/3D display using for example polarization filters (Jacobs, 2003) + Relatively cheap and suitable for mass production − Loss of brightness and resolution since only half of the pixels contribute to a given viewing angle − Limitations with respect to number of viewers − Longer optimal viewing distance with increasing display resolution − Image flipping artifact when crossing a viewing zone

Figure 5 Illustration of lenticular lenses where the same stereo image pair is repeated along the width of the screen. An array of lenses are designed to magnify different images at different angles.

Lenticular lenses spread the light from the screen such that it separates the two images and only can be seen from a certain viewing angle. This process is illustrated in Figure 5. The lenses must be precisely aligned with the display panel to avoid distortions. Lens alignment becomes increasingly difficult with increasing display resolution. However, the distortion can be compensated by software and thus improve the quality (Lee, 2006). The lenticular lenses can also cause intensity variation, which appear as dark and bright band patterns as the viewer changes viewing angle. Slanting the lenses and adjusting the focal length of the lenses can reduce the intensity variation effect (Ando, 2005). Switching between 2D/3D can be enabled by filling the lenses with a material that can switch between two refraction states. However, artifacts can still appear at oblique angles when using it for 2D viewing. Lenticular lenses can also be combined with multiple stereo-projectors, discussed later, which can either be in front of the screen, using a reflective material behind the lenses, or behind the screen

The pros and cons with lenticular lenses solutions are:

+ Can be combined with existing 2D screen fabrication. + Relatively simple and low cost + Better brightness and resolution compared to parallax barrier systems - Limited resolution - Limitations with respect to number of viewers - Lens alignment with screen is difficult - Image flipping artifact when crossing a viewing zone Multi-view Autostereoscopic Displays

Displays able to show different stereo image pairs in different viewing zones without glasses are referred to as multi-view autostereoscopic displays.

Figure 6 Reflection based autostereoscopic display (Woodgate, 1997). The image pair is here separated using a beam splitter. Image source: (Geng, Three-dimensional display technologies, 2013)

Reflection-based multi-view 3D display can use a beam splitter (half mirror) in combination with collimating lenses to focus and split the image into the viewer’s eyes as illustrated in Figure 6. Multiple light sources can be used to allow for multiple viewers.

The pros and cons with reflection-based solutions are:

+ Relatively simple construction + Better brightness and resolution compared to parallax barrier systems - Limitations with respect to the number of views - Image flipping artifact when crossing a viewing zone

Figure 7 A diffraction-based multi-view 3D display uses small diffraction gratings to spread the light into different directions. Image source: (Geng, Three-dimensional display technologies, 2013)

Diffraction-based multi-view 3D display in principle work in the same way as lenticular lenses but the lenses are replaced with small diffractive optical elements spreading the light into different directions, see Figure 7. Diffractive gratings are placed in front of each partial pixel. The size of these displays has so far been limited, with prototypes of up to four centimeters.

Figure 8 A sequence of images displayed fast enough can be split into N different views (Dodgson, 2005). Image source: (Geng, Three-dimensional display technologies, 2013).

Time multiplexing uses high frame rates to split images into N different views as seen in Figure 8 where the technique is combined with lenses to provide a multi-view setup. This technique can also be used in combination with parallax barriers (Choi, 2003). Choi used a dynamic barrier between the lens array and display panel and tilted the barrier to produce different images for each viewing zone.

Figure 9 Stereo-projector based auto-stereoscopic display using parallax barriers. The first barrier controls the projector output while the second directs the image to the eyes. Image source: (Geng, Three-dimensional display technologies, 2013).

Projection-based systems exist in many different forms. A common trait for projector-based systems are that they can be more expensive and difficult to calibrate.

Parallax barriers can be combined with projector systems as illustrated in Figure 9. Multiple stereo- projectors are here used instead of LCD panels and each projector generates one view. Two parallax barriers are used in this setup. The first parallax barrier controls the projector output on the diffuser screen while the second works as a regular parallax barrier by directing the images to the viewer’s eyes. Viewers at different locations thereby see different images. Projectors can also be placed in front of the screen, thus saving space, as demonstrated by Kim et al. (Kim, 2012). The downside is that the light needs to go through the parallax barrier twice, which reduces the optical efficiency.

Recent work includes the use of multiple projectors and a screen diffuser as illustrated in Figure 10. The light beams in this setup are generated based on a specific geometry such that, when transformed by the diffuser, they look like they left from an object at a specific location. In principle, this technique can be used for the light field displays, given that enough projectors are used, a discussed next.

Figure 10 A vertical diffuser spreads the light in the vertical axis. The intensity and color spreads differently at each point on the diffuser, but in a controlled manner, allowing light to be seen as if it where emitted from a physical object behind the screen. Image source: (Geng, Three-dimensional display technologies, 2013).

The pros and cons with the projector-based systems are:

+ Scales well with larger screen size + Relatively easy to construct − Expensive − Difficult to calibrate many projectors − Image flipping artifact when crossing a viewing zone − Less suitable for mass-production Light Field Displays

Due to the limitation on the number of views a multiview system can generate there will always be a discontinuity when going between viewing zones. The term super multi-view is sometimes used when the number of views is increased such that they are sufficiently large to appear continuous, i.e., the interval between the views is smaller than the pupil diameter of about 1.5-8 mm depending on lighting conditions. Such systems allow the eye to focus on the virtual object instead of on the screen – the so- called convergence-accommodation problem. One such example is a system combining 16 LCD panels with 16 projectors to produce 256 different views on a 10.3” display (Takaki, 2010).

Light field is an extension to this pair-wise single view technique, which instead uses pair-wise images for multiple views. By displaying multiple views of a scene from different viewing angles it is possible to remove the restriction of glasses. Displays with capabilities of displaying light fields are called auto- stereoscopic displays. The auto-stereoscopic technique was originally developed with flat video-panels using lenses to separate the outgoing light (Börner, 1985). Lenticular lenses, i.e., an array of small lenses, has so far mostly been used for auto-stereoscopic displays. An emerging technology is compressive light-field displays where light field signal is compressed using techniques matching the display hardware. are combined to increase the quality. An example is

Figure 11 Volumetric display consisting of a stack of glass sheets connected with optical fibers (MacFarlane, 1994). . Image source: (Geng, Three-dimensional display technologies, 2013).

Volumetric displays emit light from where it should be in physical space into the direction of the viewer’s eyes. This can be done in a variety of ways, for example through layers of glass sheets connected with optical fibers (MacFarlane, 1994), see Figure 11, layers of LED lights (D, 2005), or layers of transparent LCD/OLED panels, see Figure 12. The use of glass sheets is conceptually simple, but high quality cannot be achieved without a very high number of sheets.

Figure 12 Volumetric display consisting of a stack of LCD panels. Image source: (Geng, Three-dimensional display technologies, 2013).

These types of volumetric displays have been commercialized by for example LightSpace Technologies, featuring for example a bench top display to be launched Q1 2019 (Figure 13).

Figure 13 Bench top volumetric display by LightSpace Technologies. Image source: (LightSpace Technologies, 2019)

Figure 14 Laser-illuminated helical surfaces spinning fast enough can be used to produce a perception of a light field. Image source: (Geng, Three-dimensional display technologies, 2013). Spinning helix surface can be combined with lasers or high-speed projectors to create a 3D sensation. The helix spins so fast that the viewer cannot see it and therefore provide a real-world-space that light is reflected from and can be seen from any viewpoint. Lasher et al. (Lasher, 1996) produced a 1-meter large laser-based prototype capable of displaying 0.8 Million voxels while Jason Geng (Geng, 2008) instead used DLP projectors, illustrated in Figure 14, for a 0.5-meter large prototype capable of displaying 150 Million voxels.

Figure 15 Many small projectors aimed at the viewer(s) can produce a light field. Image source: (Jurik, 2011)

Projector array uses many small projectors aimed directly at the viewer(s) to create a light field display (Jurik, 2011) as illustrated in Figure 15. Jones et al. (Jones, 2015) used this technique to produce a 2- meter-tall screen using 216 projectors. The whole process of recording and displaying a life-sized human is shown in Figure 16. The pros and cons of this approach is inherited from the multiview projector- based setups.

Figure 16 (left) Subject is recorded by an array of HD camcorders under controlled lighting. (Center) Subject shown on the automultiscopic projector array. The display can be seen by multiple viewers over a 135 ◦ field of view without the need for special glasses. (Right) Stereo photograph of the subject on the display, left-right reversed for cross-fused stereo viewing. Image source: (Jones, 2015).

Holography uses a two-step process to produce light fields of static objects. First, the real object is illuminated by a laser in a dark room. A film, i.e., similar to the ones used for analog camera, records the light reflected by the object. This process is depicted in Figure 17. The atoms in the material of the film are re-organized according to the outgoing light distribution and can thus be seen as a recording of the light field. Second, to display the hologram, the same film is illuminated by a laser in the same direction as during the creation phase, see Figure 18. The viewer sees the same outgoing light as the object sent out during the creation phase independent of the viewing angle and position. This process can be used to create photorealistic virtual objects.

Figure 17 A holography film is created by illumination an object and recording the reflected light on a photographic plate. Image source: (Mellish, 2007)

Figure 18 The object is reconstructed by illuminating the photographic plate from the same direction as in the creation phase. The light is transmitted through the plate and can create an indistinguishable virtual image from all viewing directions. Image source: (Mellish, 2007)

+ Photorealistic, indistinguishable from the real, views can be created. − Can only be used on static objects

Market

Display panels

Market demand for flat panel displays: 114 billion USD (Stastistica, 2018)

Shipped smart phone display panels shipped in 2015: 161 Million (Stastistica, 2018)

AR

Worldwide shipment of smart augmented reality glasses are forecast to reach around 5.4 million units by 2020. The global augmented reality market is expected to grow significantly to about 90 billion U.S. dollars by 2020. (Statistica, 2018)

Revenue from augmented reality is projected to three times as high as that of VR by 2020. In 2022, AR hardware device shipments are projected to reach an estimated 68.9 million units—in comparison to 45.6 million VR hardware devices. (Statistica, 2018)

Application Fields • Mobile phones

Communicate with people using holograms or play games.

• Museums

Display of valuable items or items that cannot be moved.

• Product display/digital signage

Advertisement for products or brands. Promoting 3D products or services, conveying a modern, technology minded corporate image, or simply captivating the audience

Figure 19 Example of digital signage product. Image source: (360DigitalSignage, 2018)

• Automotive

Information displays and entertainment in the car.

• Communication

The next level of video-chat as shown in (Jones, 2015).

• Medical imaging

Education, surgery planning and simulation.

• Design, prototyping and 3D modelling Faster design processes and interaction with natural sense of depth.

• Architecture and virtual tours

Realistic and collaborative views of how it will look in place.

• Serious gaming

Mission and maintainance training

• military applications

Realistic simulations of for example fighter airplane piloting

Companies Below is a list of companies involved in the 3D display industry as well as a short description on what they do. Holografika http://holografika.com/ Alioscopy Provides both displays and software to generate views for the displays. The displays uses the slanted lenticular lenses technique. The most advanced display is 84” and supports 16 different viewpoints at 5- 15 meters distance. A smaller 64” display with 10 viewpoints has optimal view distance of 2.5-7 meters. http://www.alioscopy.com/en/home.php Inition Retailer of 3D displays. https://www.inition.co.uk/extraordinary-technology/stereoscopic-3d-displays/ Holographic studios Provides services for creating static holograms. https://www.holographer.com/ LightSpace3D Uses volumetric 3D image technology http://www.lightspace3d.com/

Figure 20 Volumetric display device from LightSpace3D. Image source: (LightSpace Technologies, 2019)

ZSpace “Learning through AR/VR experiences”. Combines interaction with AR/VR for educational purposes. https://zspace.com/ Holoxica Holoxica is a hard-tech company specializing in a range of holographic 3D visualization solutions from static images to video displays. Provides both hardware and software. http://www.holoxica.com Geola Geola provides (static) holography products. http://geola.com Leia Inc Display panels for smart phones.

Figure 21 Hologram for smart-phones by Leia Inc. Screenshot by Stephen Shankland/CNET, https://www.cnet.com/news/reds- 3d-holographic-hydrogen-one-phone-powered-by-leia/

Figure 22 Illustration of the technology used by Leia Inc. Image source: https://medium.com/the-technews/red-hydrogen- partnered-with-leia-inc-for-holographic-display-ec1ed24a9812 https://www.leiainc.com/ RealViewImaging Manufactures and sells holographic display designed for medical uses. http://realviewimaging.com/,

Figure 23 Snapshot of RealViewImaging demo video: https://youtu.be/KLQCbDbljik

EchoPixel Medical imaging http://www.echopixeltech.com/

Figure 24 Snapshot of EchoPixel display and interaction device (https://www.echopixeltech.com/true_3D.html)

LookingGlassFactory Volumetric displays. “The Looking Glass is a patent-pending combination of lightfield and volumetric display technologies within a single three-dimensional display system. 45 unique simultaneous views of a virtual scene are captured on a computer at 60 frames per second.” https://lookingglassfactory.com/–

Figure 25 The Looking Glass Factory volumetric display.Snapshot from https://lookingglassfactory.com/

Voxon Photonics Volumetric display, monochrome, no color. Created world’s first volumetric video call over 5G.

“At its core is an ultra high-speed digital light engine and a highly optimized volume rendering engine. This unique combination of hardware and software is capable of projecting over half a billion points of light every second into physical volumetric space.

Geometry that is being rendered is sliced up into hundreds of digital horizontal cross sections before being projected synchronously onto a specially designed high speed reciprocating screen. As the photons of light hit the screen, they are diffused and reform a physical cross sectional image at precisely the right place in physical space. Through persistence of vision, the human eye blends hundreds of layers together, and the result is a true three-dimensional (3D) volumetric holographic representation of data that can be viewed in the same way as one would view a real object, from any angle, and without special effects, goggles or glasses.” https://voxon.co/

Figure 26 Demo of Voxon Photonics display. Image source: https://voxon.co/

Figure 27 Video call demonstration using Voxon Photonics display. Image source: https://voxon.co/

Zecotek “Worlds first natural 3D monitor”

Figure 28 Zecotek 3D monitor. Image source: http://zecotek.com/zecotek-3d-displays/ http://zecotek.com/zecotek-3d-displays/

References 360DigitalSignage. (2018, 12 13). 3d-hologram-fan-displays. Retrieved from 3d-hologram-fan-displays: https://www.360digitalsignage.com/product-item/3d-hologram-fan-displays/

Ando, T. a. (2005). Multiview image integration system for glassless 3D display. In Stereoscopic Displays and Virtual Reality Systems XII (pp. 158-167). International Society for Optics and Photonics.

Börner, R. (1985). 3D-Bildprojektion in Linsenrasterschirmen. Fernseh-und Kino-Technik, 383-387.

Choi, H. a.-W.-H. (2003). Multiple-viewing-zone integral imaging using a dynamic barrier array for three- dimensional displays. Optics express, 927-932.

Cutting, J. E. (1995). Perceiving layout and knowing distances: The integration, relative potency, and contextual use of different information about depth. In W. E. Rogers, Perception of space and motion (pp. 69-117). Elsevier.

D, W. (2005). A volumetric 3D LED display. Retrieved from A volumetric 3D LED display: http://web.mit.edu/6.111/www/f2005/projects/wyatt_Project_Design_Presentation.pdf

Dodgson, N. A. (2005). Autostereoscopic 3D displays. IEEE, 31-36.

Geng, J. (2008). Volumetric 3D display for radiation therapy planning. Journal of Display Technology, 437-450.

Geng, J. (2013). Three-dimensional display technologies. IEEE Intelligent Transportation Systems Society, 456-535.

Jacobs, A. a. (2003). 2D/3D Switchable Displays. Sharp Technical Journal, 15-18.

Jones, A. a.-Y. (2015). An automultiscopic projector array for interactive digital humans. ACM SIGGRAPH 2015 Emerging Technologies (p. 6). ACM.

Jurik, J. a. (2011). Prototyping a light field display involving direct observation of a video projector array. Computer Vision and Pattern Recognition Workshops (CVPRW), 2011 IEEE Computer Society Conference on (pp. 15-20). IEEE.

Kim, Y. a.-H.-H. (2012). A frontal projection-type three-dimensional display. Optics express, 20130- 20138.

Lasher, M. E. (1996). Laser projected 3D volumetric displays. In M. E. Lasher, Projection Displays II (pp. 285-296). International Society for Optics and Photonics.

Lee, Y.-G. a. (2006). Image distortion correction for lenticula misalignment in three-dimensional lenticular displays. Optical Engineering, 017007.

LGDisplay. (2018, 12 13). LG Display. Retrieved from LG Display: http://www.lgdisplay.com/eng/product/technology/3D LightSpace Technologies. (2019, 04 30). LightSpace Technologies. Retrieved from LightSpace3D: https://www.lightspace3d.com/index.php/2018/06/18/lightspace-technologies-announces-s- 3901-bench-top-volumetric-3d-display/

MacFarlane, D. L. (1994). Volumetric three-dimensional display. Applied optics, 7453-7457.

Mellish, B. (2007, 04 14). rBob at the English language Wikipedia [CC BY-SA 3.0 (http://creativecommons.org/licenses/by-sa/3.0/)]. Retrieved from Wikipedia: https://upload.wikimedia.org/wikipedia/commons/7/77/Holograph-record.svg

Okoshi, T. (2012). Three-dimensional imaging techniques. In T. Okoshi, Three-dimensional imaging techniques (p. 387). Elsevier.

Stastistica. (2018, 12 13). Stastistica. Retrieved from Stastistica: https://www.statista.com/topics/3288/display-technology/

Statistica. (2018, 12 13). Statistica AR/VR. Retrieved from Statistica AR/VR: https://www.statista.com/topics/3286/augmented-reality-ar/

Takaki, Y. a. (2010). Multi-projection of lenticular displays to construct a 256-view super multi-view display. Optical Society of America, 8824--8835.

Urey, H. a. (2011). State of the art in stereoscopic and autostereoscopic displays. Proceedings of the IEEE, 540-555.

Woodgate, G. J. (1997). Observer-tracking autostereoscopic 3D display systems. In G. J. Woodgate, Stereoscopic Displays and Virtual Reality Systems IV (pp. 187-199). International Society for Optics and Photonics.