Waveguide Combiners for Mixed Reality Headsets

Nanophotonics 2021; 10(1): 41–74

Review

Bernard C. Kress* and Ishan Chatterjee Waveguide combiners for mixed reality headsets: a nanophotonics design perspective https://doi.org/10.1515/nanoph-2020-0410 – vestibular comfort—providing stable and realistic virtual Received July 21, 2020; accepted September 16, 2020; overlays that spatially agree with the user’smotion published online October 7, 2020 – social comfort—allowing for true eye contact, in a socially acceptable form factor. Abstract: This paper is a review and analysis of the various implementation architectures of diffractive waveguide Immersion can be defined as the multisensory percep- combiners for augmented reality (AR), mixed reality (MR) tual experience (including audio, display, gestures, haptics) headsets, and smart glasses. Extended reality (XR) is that conveys to the user a sense of realism and envelopment. another acronym frequently used to refer to all variants In order to effectively address both comfort and immersion across the MR spectrum. Such devices have the potential to challenges through improved hardware architectures and revolutionize how we work, communicate, travel, learn, software developments, a deep understanding of the spe- teach, shop, and are entertained. Already, market analysts cific features and limitations of the human visual perception show very optimistic expectations on return on investment system is required. We emphasize the need for a human- in MR, for both enterprise and consumer applications. centric optical design process, which would allow for the Hardware architectures and technologies for AR and MR most comfortable headset design (wearable, visual, vestib- have made tremendous progress over the past five years, ular, and social comfort) without compromising the user’s fueled by recent investment hype in start-ups and acceler- sense of immersion (display, sensing, and interaction). ated mergers and acquisitions by larger corporations. In Matching the specifics of the display architecture to the order to meet such high market expectations, several chal- human visual perception system is key to bound the con- lenges must be addressed: first, cementing primary use straints of the hardware allowing for headset development cases for each specific market segment and, second, and mass production at reasonable costs, while providing a achieving greater MR performance out of increasingly size-, delightful experience to the end user. weight-, cost- and power-constrained hardware. One such crucial component is the optical combiner. Combiners are Keywords: augmented reality; diffractive optics; hologra- often considered as critical optical elements in MR headsets, phy; mixed reality; virtual reality; waveguide optics. as they are the direct window to both the digital content and the real world for the user’seyes. Two main pillars defining the MR experience are comfort Glossary of Terms, Abbreviations, and immersion. Comfort comes in various forms: and Acronyms – wearable comfort—reducing weight and size, pushing back the center of gravity, addressing thermal issues, and We provide this glossary for the reader after the abstract section so on as these acronyms are used extensively in this review paper. – visual comfort—providing accurate and natural fi 3-dimensional cues over a large eld of view and a high AR Augmented reality, adding virtual content into angular resolution field of view of reality, can include augmentations created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projector systems, etc.  *Corresponding author: Bernard C. Kress, Microsoft Corp. HoloLens MR Mixed reality, virtual objects situationalized in D Team, 1 Microsoft Way, Redmond, 98052, WA, USA, in your real space, often interactable E-mail: [email protected]. OST-MR Optical see-through mixed reality, displays are Ishan Chatterjee, Microsoft Corp. HoloLens Team, 1 Microsoft Way, transparent such that real world is viewable Redmond, 98052, WA, USA optically through the displays

Open Access. © 2020 Bernard C. Kress and Ishan Chatterjee, published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 42 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

(continued) (continued)

AR Augmented reality, adding virtual content into AR Augmented reality, adding virtual content into field of view of reality, can include augmentations field of view of reality, can include augmentations created by mixed reality headsets, handhelds, created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projec- head up displays, smart glasses, camera-projector systems, etc. tor systems, etc.

VST-MR Video see-through mixed reality, virtual reality mu-iLED, micro- Micro inorganic light-emitting diode, actively turned into the mixed reality with camera pass- iLED addressed inorganic LED array with emitter size through of the real-world into the VR environment < mu, NTE displays usually require a magnitude XR Extended reality, a generic term to capture all lower; can achieve high brightness and contrast, varieties across MR and AR but challenged in maintaining efficiency, multi- VR Virtual reality, blocks out reality and supplants color integration and backplane integration with virtual objects VCSEL Vertical-cavity surface-emitting laser, laser diode Immersion Sense of realism and development in delivered with lower divergence and current threshold than experience edge-emitting diodes IMU Inertial measurement unit consisting of at least an MEMS Microelectromechanical system accelerometer, and gyroscope, and often a LBS Laser beam scanning, type of display where a magnetometer modulated laser dot is raster scanned across GPU Graphical processing unit, parallel architecture display FOV via system of MEMS mirrors suited for graphics render and other matrix NTE Near-to-eye operations DOF  degrees of freedom, in the context of tracking HMD Head-mounted display or helmet-mounted usually refers to the rotational axes (pitch, yaw, roll) display which can be resolved with only a calibrated IMU HUD Head up display, refers to see-through display DOF  degrees of freedom, in the context of tracking that is often mounted externally (such as above a refers to the rotational and translational axes dashboard) allowing user to see both virtual CG Center of gravity, important ergonomic metric in content and subject of focus (e.g., the road ahead) head-worn devices simultaneously IPD Interpupillary distance SLM Spatial light modulator PPD Pixels per degree LCD Liquid-crystal display, display technology where HDR High dynamic range electro-sensitive liquid crystal pixels amplitude- FOV Field of view, provided as an angle modulate light from a global polarized backlight Eyebox The volume that the user’s pupil can sit in and in transmission view the entire virtual image field-of-view. The box LTPS-LCD Low-temperature polysilicon liquid-crystal may not be a rectangular prism, but is more often a display, higher resolution and faster switching frustrum speed than amorphous Si LCD Eye relief The distance the user’s corneal surface is from the IPS-LCD In-plane switching liquid-crystal display, liquid- display optic surface crystal structure twist in-plane of display, allow- UX/UI User experience/User interface, refers to the ing for higher viewing angles than twisted nematic design of the experience and applications (TN) LCDs, used in phones and monitors VAC Vergence accommodation conflict, refers to the HTPS-LCD High-temperature polysilicon (used for silicon mismatch experienced when a stereoscopic dis- backplanes) play’s image focal plane does not match the ste- AMOLED Active-matrix organic light-emitting diode, reo disparity of the virtual image. increased contrast at the cost of lifetime and high Pupil swim The experience of warp and shift of virtual objects brightness, each pixel is its own organic electro- as the user’s pupils rove around the eyebox luminescent emitter, used commonly in caused by distortion in the projected image across cellphones the eyebox mu-OLED, micro- Micro-organic light-emitting diode, display with Hard-edge The ability for real-world objects to mask virtual OLED emitter size less than  μm, used in camera occlusion content according to the depth the virtual image is electronic view finders in the world DLP Digital light processing, Texas Instrument’s Hologram Recording of a interference pattern between a colloquially genericized trademark for DMD (digi- reference and a wavefront off a D scene… but in tal micromirror device), an array of bi-stable AR/VR forums, a virtual stereo image that appears reflective micromirrors, commonly used in pro- to be positioned in space like a true hologram jection systems for highly efficiency SLM ET Eye tracking LCoS Liquid crystal on silicon, microdisplay with a HeT Head tracking switchable liquid-crystal matrix on reflective sili- TIR Total internal reflection (principle of how light con backplane propagates when trapped in a light guide) B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 43

(continued) displays. Although such display technologies were well ahead of their time [5–7], the lack of consumer-grade in- AR Augmented reality, adding virtual content into ertial measurement unit (IMU) sensors, low-power field of view of reality, can include augmentations 3-dimensional (3D)-rendering graphical processing units, created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projec- and wireless data transfer technologies contributed to the tor systems, etc. end of this first VR boom. The other reason was the lack of digital content or rather the lack of a clear vision of adapted PBS Polarized beam splitter EPE Exit pupil expansion, a technique where a com- VR/AR content for enterprise or consumer spaces [8, 9]. biner’s exit pupil may be replicated in DorD The only AR/VR sector that saw sustained efforts and space allowing for a larger eyebox developments throughout thenextdecadewasthedefense LOE Lightguide optical element industry (flight simulation and training, helmet-mounted SRG Surface relief gratings, nanostructure gratings displays [HMDs] for rotary-wing aircrafts, and head-up dis- etched into substrate surface, can be blazed, slanted, binary, multilevel, or analog plays [HUDs] for fixed-wing aircrafts) [10]. The only effective CGH Computer-generated hologram, hologram whose consumer efforts during 2000–2010 were in the field of wavefront has be calculated computationally automotive HUDs and personal binocular headset video rather than recorded in analog players. RWG Resonant waveguide gratings, also known as GMR The smartphone technology ecosystem, including the (guided mode resonant gratings), diffractive, associated display, connectivity, and sensor systems, dielectric structures with leaky lateral modes Metasurface Surface with nanofabricated, sub-wavelength shaped the emergence of the second VR/AR boom and structures (often high aspect ratio) that can impart formed the first building blocks used by early product in- arbitrary phase changes in transmission and/or tegrators. Today’s engineers, exposed at an early age to reflection unlocking unique optical functions. ever-present flat-panel display technologies, tend to act as NIL Nanoimprint lithography creatures of habit much more than their peers 20 years ago, ALD Atomic layer deposition MTF Modulation transfer functions, represents the ef- who had to invent novel immersive display technologies fect (usually degradation) on spatial frequencies from scratch. We have therefore seen since 2012 the initial (in resolution and contrast) through an optical implementations of immersive AR/VR HMDs based on element, higher is better readily available smartphone display panels (low-temper- RCWA Rigorous couple-wave analysis ature polysilicon liquid-crystal display [LCD], In-plane FMM Fourier modal method switching liquid-crystal display, active-matrix organic FDTD Finite difference time domain SAW Surface acoustic wave light-emitting diode) and picoprojector microdisplay AOM Acousto-optical modulator panels (High-temperature polysilicon LCD, mu-organic EOM Electro-optical modulator light-emitting diode (OLED), digital light processing (DLP), liquid crystal on silicon (LCoS). (Similarly, the AR/VR industry has been able to leverage the progress made during the smartphone revolution for cheap and reliable sensors as well, such as IMUs and cameras). Currently, HMD 1 Introduction display architectures are evolving slowly to more specific technologies, which may be a better fit for immersive re- Defense has been historically the first application sector for quirements than flat panels are, sometimes resembling the augmented reality (AR) and virtual reality (VR), as far back display technologies invented throughout the first AR/VR as the 1950s [1]. Based on these early developments, the boom two decades earlier (inorganic mu-iLED panels, first consumer VR/AR boom expanded in the early 1990s 1-dimensional [1D] scanned arrays, 2-dimension (2D) laser/ and contracted considerably throughout that decade, a vertical-cavity surface-emitting laser [VCSEL] micro- poster child of a technology ahead of its time and ahead of electromechanical system [MEMS] scanners, and so on). its markets [2]. Notably, due to the lack of available con- Such traditional display technologies will serve as an sumer display technologies and related sensors, novel initial catalyst for what is coming next. The immersive optical display concepts were introduced throughout the display experience in AR/VR is a paradigm shift from the 1990s [3, 4] that are still considered as state of the art, such traditional panel display experiences that have existed for as the “Private Eye” smart glass from Reflection Technol- more than half a century, going from cathode ray tube ogy (1989) and the “Virtual Boy” from Nintendo (1995)— (CRT) TVs to LCD computer monitors and laptop screens, to both based on scanning displays rather than flat-panel OLED tablets and smartphones, to LCoS, DLP, and MEMS 44 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 1: Immersive NTE displays: a paradigm shift in personal information display. NTE, near-to-eye. scanner digital projectors, and to iLED smartwatches (see specific information, targeted at multiple viewers detected Figure 1). and tracked through biometrics by sensors around that same When flat-panel display technologies and architec- display. These dynamic EBs are steered in real time to follow tures (smartphone or microdisplay panels) are used to the specific viewers. This is not a wearable display architec- implement immersive near-to-eye (NTE) display devices, ture,ratheramonitorortransparentwindowdisplay.Inthis factors such as etendue, static focus, low contrast, and scenario, different viewers of that same physical display see low brightness become severe limitations. Alternative different information, tuned to their specific interest, display technologies are required to address the needs of depending on their physical location. NTE immersive displays to match the specifics of the human visual system. The emergence of the second VR/AR/smart glasses boom 2 The emergence of MR as the next in the early 2010s introduced new naming trends, more in- clusive than AR or VR: mixed (or merged) reality (MR), more computing platform generally known today as “XR,” a generic acronym for “extended reality.” The name “smart eyewear” (world locked Smart glasses (also commonly called smart eyewear or audio, digital monocular display, and prescription eyewear) digital eyewear) are mainly an extension of prescription tends to replace the initial “smart glass” naming convention. eyewear, providing a digital contextual display as an Figure 2 represents the global MR spectrum contin- addition to vision prescription correction (see for uum, from the real-world experience toward diminished example Google Glass). This concept is functionally very reality (where parts of reality are selectively blocked through hard edge occlusion, such as annoying adver- tisements while walking or driving through a city, to Parallel RealiƟes blinding car headlights while cruising at night on a high- way) to AR as in optically see-through MR, to merged re- Real World Mixed Reality Spectrum Virtual World ality as in video see-through MR, and to pure virtual worlds Augmented Reality Virtual Reality (as in VR). Diminished Reality Contextual Display Merged Reality Parallel realities are a new concept that emerged recently Extended Reality with specific optical display hardware, creating from a single- display hardware-specific individual eyeboxes (EBs) with Figure 2: Mixed-reality spectrum continuum. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 45

different from either AR or MR functionality. The typical The worldwide sales decline for smartphones and smartglassfieldofviewremainssmall(<15°diagonal), is tablets in Q3 2018 was an acute signal for major consumer typically monocular, and is often offset from the line of electronics corporations and VC firms to fund and develop sight. The lack of sensors (apart the IMU) allows for the “next big thing.” MR headsets (in all their forms as approximate 3 degrees of freedom (3DOF) head tracking, glasses, goggles or helmets), along with 5G connectivity andlackofbinocularvisionreducesthedisplaytosim- and subsequent cloud MR services, look like good candi- ple, overlaid 2D text and images. Typical 3DOF content is dates for many. locked relative to the head, while 6 degrees of freedom (6DOF) sensing allows the user to get further and closer to the content. Monocular displays do not require as much rigidity in 2.1 Wearable, visual, vestibular, and social the frames as a binocular vision system would (to reduce comfort horizontal and vertical retinal disparity that can produce eye strain). Many smart glass developers also provide Comfort, in all four declinations—wearable, visual, prescription correction as a standard feature (e.g., “Focal” vestibular, and social—is key to enabling a large accep- by North or Google Glass V2). tance base of any consumer MR headset candidate ar- The combination of strong connectivity (3G, 4G, chitecture. Comfort, especially visual, is a subjective WiFi, Bluetooth) and a camera makes it a convincing concept. Its impact is therefore difficult to measure or companion to a smartphone, for contextual display even estimate on a user pool. Careful user testing is functionality or as a virtual assistant, acting as a global required to assess. positioning system (GPS)-enabled social network com- Wearable comfort features include the following: panion. A smart glass does not aim to replace a smart- – Untethered headset for best mobility. phone, but it intends to contribute as a good addition to – Small size and light weight. it, like a smartwatch. – Thermal management throughout the entire headset VR headsets are an extension of simulators and (passive or active). gaming consoles, as shown by major gaming providers – Skin contact management through pressure points. such as Sony, Oculus, HTC Vive, and Microsoft Windows – Breathable fabrics to manage sweat and heat. MR, with gaming companies such as Valve Corp – Center of gravity (CG) closer to the CG of a human head. providing a gaming content ecosystem (Steam VR). The offerings have bifurcated into high-performance per- Visual comfort features include the following: sonal computer (PC)-tethered headsets (Samsung Odes- – Large EB to allow for wide interpupillary distance (IPD) sey, HTC Vive Pro, Oculus Rift) and mobile-first, stand- coverage. The optics might also come in different stock alone experiences (Oculus Quest). Pancake optics and keeping units (SKUs) for consumers (i.e., small, medium, hybrid lenses will continue to push the form factor of and large IPDs), but for enterprise, because the headset is these devices down. shared between employees, it needs to accommodate a AR and especially MR systems are poised to become the wide IPD range. next computing platform, replacing ailing desktop and – Angular resolution close to 20/20 visual acuity (at least laptop hardware, and now even the aging tablet computing 45 pixels per degree [PPD] in the central foveated re- hardware. Such systems are mostly untethered for most of gion), lowered to a few PPD in the peripheral visual them (see HoloLens 1) and require high-end optics for the region. display engine, combiner optics, and sensors (depth scanner – No screen-door effects (large pixel ﬁll factor and high camera, head-tracking cameras to provide 6DOF, accurate PPD), and no Mura effects. eye trackers, and gesture sensors). These are currently the – High dynamic range through high brightness and high most demanding headsets in terms of hardware, especially contrast (emissive displays such as MEMS scanners optical hardware, and are the basis of this review paper. and OLEDs/iLEDs versus nonemissive displays such as Eventually, if technology permits, these three categories LCoS and LCD). will merge into a single hardware concept. This will, however, – Ghost images minimized (<1%). require improvements in connectivity (5G, WiGig), visual – Unconstrained 200+ degree see-through peripheral comfort (new display technologies), and wearable comfort vision (especially useful for outdoor activities, de- (battery life, thermal management, weight/size). fense, and civil engineering). 46 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

– Active dimming on visor (uniform shutter or soft-edge However, throughout this review work, we conform to dimming). the new (albeit deformed by the AR/VR/MR community) – Display brightness control (to accommodate various meaning of the world “hologram” as a stereo image. environmental lightning conditions). – Reduction of any blue remaining ultraviolet (UV) or blue LED light (<415 nm) to limit retinal damage. 2.2 Display immersion – Color accuracy and color uniformity over FOV as well as EB are also important vision comfort keys. Immersion is the other key to the ultimate MR experience and is not based only on FOV, which is a 2D angular Vestibular comfort features include the following: concept; immersive FOV is a 3D concept that includes the z – Motion-to-photon latency (time between head move- distance from the user’s eyes, allowing for arm’s length ment and display update) below 10 ms (through opti- display interaction through VAC mitigation. mized sensor fusion). Immersive experiences can come in various forms: – Spatial stability of holograms in the 3D world across – Wide-angle FOV, including peripheral display regions both low and high frequencies. with lower pixels count per degree (resolution) and – User experience/user interface considerations to pre- lower color depth. sent content motion do not severely disagrees with a – Foveated display that is either ﬁxed/static (foveated ’ user s sense of motion. rendering) or dynamic (through display steering, me- chanically or optically). Visual comfort features leveraging eye tracking include the – World-locked holograms, hologram occlusion through following: accurate and fast spatial mapping and hard-edge see- – – ﬂ Vergence accommodation con ict (VAC) mitigation through occlusion. for close objects located in the foveated cone through – World-locked spatial audio. vergence tracking from differential eye tracking data – Accurate eye/gesture/brain sensing through dedicated (as vergence is the trigger to accommodation). sensors. – Active pupil swim correction for large-FOV optics. – Vivid and realistic hologram colors and shading. – Active pixel occlusion (hard-edge occlusion) to in- – Haptic feedback. crease hologram opacity (more realistic).

Social comfort features include the following: – Unaltered eye view of the HMD wearer, allowing for 3 Functional optical building continuous eye contact and eye expression discernment. blocks of an MR headset – No world-side image extraction (present in many waveguide combiners). An HMD, and particularly an optically see-through HMD, is – Covert multiple-sensor objective cameras pointing to the a complex system, with at its core various optical sub- world (reducing socially unacceptable world spying). systems. Once the optical subsystems are defined, such as the choice of the optical engine, the combiner engine and Note:Theword“hologram” is used extensively by the the optical sensors (eye tracking, head tracking, depth AR/VR/MR community as referring to “stereo images.” scanner, gesture sensors, and so on), all the rest can be For the optical engineer, a hologram is either the volume engineered around this core. holographic media (dichromated gelatin [DCG] emul- A typical functional optical building block ecosystem sion, Silver Halide film or Photopolymers films, surface of a MR headset is shown in Figure 3, including display, relief element, and so on) that can store phase and/or imaging, and sensing subsystems. amplitude information as a phase and/or amplitude The display engine is where the image is formed and modulation or the representation of a true diffracted then imaged onwards, forming or not a pupil, and passed holographic field, forming an amplitude image, a phase through an optical combiner that can include a pupil object, or a combination thereof. A hologram in the replication scheme to the eye pupil. Gaze tracking might or original sense of the world can thus be also an optical might not share optics with the display architecture (which element, such as a grating, a lens, a mirror, a beam is usually an infinite conjugate system, and eye tracking is shaper, a filter, a spot array generator, and so on. usually a finite conjugate system). B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 47

Gaze tracking Head tracking

Gesture Display Imaging Exit pupil Combiner sensing Engine optics expansion Optics

Depth mapping Figure 3: Functional optical building blocks of an MR system.

3.1 Display engine optical architectures Spatially demultiplexed exit pupils (either color or field separated) can be an interesting option, depend- Once the image is formed over a plane, a surface, or through a ing on the combiner architecture used (see the Magic scanner, there is a need to form an exit pupil, over which the Leap One). Imaging optics or relay optics in the display image is either totally or partially collimated and then pre- engine are usually free-space optics but in very compact sented directly to the eye or to an optical combiner (see form, including in many cases polarization beam cubes Figure 4 for the display engine architecture and subsequent combined with birdbath architectures [12] to fold the waveguide combiner for HoloLens 1 and HoloLens 2) . In some opticalpathinvariousdirections.Reﬂective/catadiop- cases, an intermediate aerial image can be formed to increase tric optics are also preferred for their reduced achro- the etendue of the system. matic spread. Because the waveguide input pupils for both eyes are located in the upper nasal area in HoloLens 1, several optical elements of the display engine have been shared with 3.2 Combiner optics and exit pupil both display engines in order to reduce any binocular im- expansion age misalignments. In the HoloLens 2, this is not the case since the input pupils are centrally located on the wave- The optical combiner is often the most complex and most guide (as the field propagates by total internal reflection costly optical element in the entire MR display archi- [TIR] in both directions in the guides). tecture: it is the one component seen directly by the user

Figure 4: Display engines based on an LCoS imager, as in the HoloLens 1 (top, 2016), and a laser MEMS scanner, as in the HoloLens 2 (bottom, 2019). 48 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

and the one seen directly by the world. It often defines Waveguide combiners are based on TIR propagation of the size-and-aspect ratio of the entire headset. It is the critical the entire field in an optical guide, essentially acting as a optical element that reduces the quality of the see-through and transparent periscope with a single entrance pupil and the one that deﬁnes the EB size (and in many cases, also often many exit pupils. the FOV). The primary functional components of a waveguide There are three main types of optical combiners used in combiner consist of the input and output couplers. most MR/AR/smart glasses today: These can be either simple prisms, microprism arrays, – Free-space optical combiners, embedded mirror arrays, surface relief gratings (SRGs), – TIR prism optical combiners (and compensators), and thin or thick analog holographic gratings, metasurfaces, – Waveguide-based optical combiners. or resonant waveguide gratings (RWGs). All of these have their specific advantages and limitations, which When optimizing an HMD display system, the optical will be discussed here. Waveguide combiners have been engine must be optimized in concert with the combiner used historically or tasks very different from AR engine. Usually, a team that designs an optical engine combiner, such as planar optical interconnections [13] without fully understanding the limitations and spe- and LCD backlights [14]. cifics of a combiner engine designed by another team, Waveguide combiners are an old concept, some of and vice versa, can result in a suboptimal system or the earliest intellectual property (IP) dates back to 1976 even a failed optical architecture, no matter how well and applied to HUDs. Figure 5(a) shows a patent by Juris the individual optical building blocks might be Upatnieks dating back 1987, a Latvian/American scien- designed. tist and one of the pioneers of modern holography [16], implemented in a DCG holographic media. A few years later, 1D EB expansion (1D exit pupil expansion [EPE]) 4 Waveguide combiners architectures were proposed as well as a variety of al- ternatives for in-coupler and out-coupler technologies, such as SRG couplers by Thomson CSF (Figure 5(b)). Free-form TIR prism combiners are at the interface between Figure 5(c) shows the original 1991 patent for a free space and waveguide combiners. When the number of waveguide-embedded partial mirror combiner and exit TIR bounces increases, one might refer to them as wave- pupil replication. (All of these original waveguide guide combiners. Waveguide combiner architectures are combiner patents have been in the public domain for the topic of this review paper. nearly a decade.)

Figure 5: (a) Original waveguide combiner patents including holographic (1987), (b) surface relief grating (1989), and (c) partial mirrors (1991) for HUD and HMD applications. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 49

4.1 Curved waveguide combiners and a Covering a large IPD range (such as a 95 or 98 single exit pupil percentile of the target consumer population, including various facial types) requires a large horizontal EB, typi- – fi If the FOV is small (<20° diagonally), such as in smart cally 10 15 mm. Also, due to t issues and nose-pad de- glasses, it might not be necessary to use an exit pupil signs, a similar large and vertical EB is also desirable, – expansion architecture, which would make the waveguide ranging from 8 12 mm. design much simpler and allow for more degrees of freedom, such as curving the waveguide. Indeed, if there is a single output pupil, the waveguide can imprint optical 4.2 Continuum from flat to curved power onto the TIR field, as is done in the curved wave- waveguides and extractor mirrors guide Smart Glass by Zeiss in Germany (developed now with Deutsche Telekom and renamed “Tooz”); see Figure 6. One can take the concept of a flat waveguide with a single The other waveguide smart glass shown here (flat curved extractor mirror (Epson Moverio BT300) or free- waveguide cut as a zero-diopter ophthalmic lens) is the form prism combiner, or a curved waveguide with curved early prototype (1995) from Micro-Optical Corp. in which mirror extractor, to the next level by multiplying the mir- the extractor is an embedded coated prism. rors to increase the EB (see the Lumus lightguide optical In the Zeiss “Tooz” smart glass, the exit coupler is an element [LOE] waveguide combiner) or fracturing metal embedded off-axis Fresnel reflector. The FOV as well as the mirrors into individual pieces (see the Optinvent ORA out-coupler is excentered from the line of sight. The FOV waveguide combiner or the LetinAR waveguide combiner) remains small (11°) and the thickness of the guide relatively While fracturing the same mirror into individual pieces thin (3–4 mm). can increase see through and depth of focus, the use of more Single exit pupils have also been implemented in flat mirrors to replicate the pupil is a bit more complicated, espe- guides, as in the Epson Moverio BT100, BT200, and BT300 cially in a curved waveguide where the two exit pupils need to (temple-mounted optical engine in a 10-mm-thick guide be spatially demultiplexed to provide a specific mirror curva- with curved half-tone extractor in the BT300) or in the ture to each pupil to correct for image position: this limits the Konica Minolta smart glasses, with top-down display in- FOV in one direction so that such overlap does not happen. jection and a flat RGB panchromatic volume holographic Figure 8 summarizes some of the possible design con- extractor (see Figure 7). figurations with such waveguide mirror architectures. Note Single exit pupils (no EPE) are well adapted to small- that the grating- or holographic-based waveguide combiners FOV smart glasses. If the FOV gets larger than 20°, espe- are not listed here; they are the subject of the next sections. cially in a binocular design, 1D or 2D exit pupil replication Figure 8 shows that many of the waveguide combiner is required. architectures mentioned in this section can be listed in this

Figure 6: Zeiss “Tooz” smart glass with single exit pupil allowing for curved waveguide. 50 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 7: Single-exit-pupil flat waveguide combiners (with curved reflective or flat holographic out-couplers).

Figure 8: Multiplying or fracturing the extractor mirrors in flat or curved waveguides. table. Mirrors can be half tone (Google Glass, Epson Moverio), everything becomes more complex, and the extractor mirror dielectric (Lumus LOE), have volume holographic reﬂectors lenses need also to compensate for the power imprinted on the (Luminit or Konica Minolta), or the lens can be fractured into a TIRfieldateachTIRbounceintheguide.Inthecaseofcurved Fresnel element (Zeiss Tooz Smart Glass). In the Optinvent mirrors (either in flat or curved waveguides), the exit pupils case, we have a hybrid between fractured metal mirrors and over the entire field cannot overlap since the power to be cascaded half-tone mirrors. In one implementation, each imprinted on each exit pupil (each field position) is different microprism on the waveguide has one side fully reﬂective and (Moverio BT300 and Zeiss Tooz Smart Glass). This is not the the other side transparent to allow see through. case when the extractors are flat and the field is collimated in In the LetinAR case, all fractured mirrors are reflective, the guide (Lumus LOE). can be flat or curved, and can be inverted to work with a birdbath reflective lens embedded in the guide. Even though the waveguide might be flat, when using 4.3 One-dimensional EB expansion multiple lensed mirrors, the various lens powers will be different since the display is positioned at different distances As the horizontal EB is usually the most critical to accom- from these lensed extractors. When the waveguide is curved, modate large IPD percentiles, a single-dimensional exit B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 51

pupil replication might suffice. The first attempts used Various types of 2D EPE replication have been devel- holographic extractors (Sony Ltd.) [18] with efforts to re- oped: from cascaded X/Y expansion (as in the Digilens, cord RGB holographic extractors as phase-multiplexed Nokia, Vuzix, HoloLens, and Magic Leap One combiner volume holograms [19] and also as cascaded half-tone architectures [21–23]) to combiner 2D expansion [24, 26] (as mirror extractors (LOE from Lumus, Israel) or arrays of in the BAE Q-Sight combiner or the WaveOptics Ltd. Phlox microprisms (Optinvent, France) [69]. This reduced the 40-degree grating combiner architectures, see Figure 10), 2D footprint of the combiner, which operates only in one to more complex spatially multiplexed gratings (as in the direction. Dispelix combiner). However, to generate a sufficiently large EB in the While holographic recording or holographic volume nonexpanded direction, the input pupil produced by the gratings are usually limited to linear gratings or gratings with display engine needs to be quite large in this same direc- slow power (such as off-axis diffractive lenses), SRGs can be tion—larger than the exit pupil in the replicated direction. either 1D or 2D and either linear or quasi arbitrary in shape. In many cases, a tall-aspect-ratio input pupil can lead to Such structures or structure groups can be optimized by iter- larger display engines such as in the 1D EPE Lumus LCoS– ative algorithms (topological optimization) rather than based enginers. However, a single vertical pupil with nat- designed analytically (WaveOptics computer-generated holo- ural expansion will provide the best imaging and color grams [CGHs] or Dispelix “mushroom forest” gratings). uniformity over the EB. Some of these combiners use one guide per color, some The Lumus LOE has been integrated in various AR usetwoguidesforallthreecolors,andsomeuseasingleguide glasses at Lumus, as well as in many third-party AR for RGB; some use glass guides, and others use plastic guides, headsets (Daqri, Atheer Labs, Flex, Lenovo). The Lumus along with the subsequent compromises one has to make on LOE can operate in either the vertical direction with the color uniformity, efficiency, EB, and FOV. display engine located on the forehead (DK Vision). Lumus Next, we point out the differences between the various is also working on a 2D expansion scheme for its LOE line of coupler elements and waveguide combiner architecture used combiners (Maximus), with central or lateral input pupils, in such products. We will also review new coupler technolo- allowing for a smaller light engine (as the light-engine exit gies that have not yet been applied to enterprise or consumer pupil can be symmetric due to 2D expansion). Similarly, products. While the basic 2D EPE expansion technique might the Sony 1D holographic waveguide combiner architecture be straightforward, we will discuss alternative techniques that has been implemented in various products, such as Univet can allow a larger FOV to be processed by both in-coupler and and SwimAR (both using Sony SED 100A waveguide). out-couplers (either as surface gratings or volume holograms). Finally, we will review the mastering and mass replication techniques of such waveguide combiners to allow scaling and 4.4 Two-dimensional EB expansion consumer cost levels.

Two-dimensional EB expansion is desired (or required) when the input pupil cannot be generated by the optical 4.5 Choosing the right waveguide coupler engine over an aspect ratio tall enough to form the 2D EB technology because of the FOV (etendue limitations) and related size/ weight considerations. A 2D EPE is therefore required (see The coupler element is the key feature of a waveguide Figure 9). combiner. The TIR angle is dictated by the refractive

Figure 9: 2D pupil replication architectures in planar optical waveguide combiners. 52 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 10: Smart glasses and AR headsets that use 2D EPE diffractive or holographic waveguide combiners. index of the waveguide, not the refractive index of the been reduced recently thanks to better cutting/polishing, coupler nanostructures. Very often, the index of the coating, and designing. coupler structure (grating or hologram) prescribes the angular and spectral bandwidth over which this coupler 4.5.1.3 Embedded microprism array can act, thus impacting the color uniformity over the FOV Microprism arrays are used in the Optinvent (France) and EB. waveguide as out-couplers [20]. The in-coupler here is Numerous coupler technologies have been used in again a prism. Such microprism arrays can be surface relief industry and academia to implement the in-couplers and or index matched to produce an unaltered see-through out-couplers, and they can be defined either as refractive/ experience. The microprisms can all be coated uniformly reflective or diffractive/holographic coupler elements. with a half-tone mirror layer or can have an alternance of totally reflective and transmissive prism facets, provide a 4.5.1 Refractive/reflective couplers elements resulting 50% transmission see-through experience. The Optinvent waveguide is the only flat waveguide available 4.5.1.1 Macroscopic prism today as a plastic guide, thus allowing for a consumer-level A prism is the simplest TIR in-coupler and can be very cost for the optics. The microprism arrays are injection efficient. A prism can be bounded on top of the waveguide, molded in plastic and bounded on top of the guide. or the waveguide itself can be cut at an angle, to allow 4.5.2 Diffractive/holographic couplers elements normal incident light to enter the waveguide and be guided by TIR (depending on the incoming pupil size). Another 4.5.2.1 Thin reflective holographic coupler way uses a reflective prism on the bottom of the waveguide Transparent volume holograms working in reflection mode (metal coated). Using a macroscopic prism as an out- —as in DCG, bleached silver halides (Slavic or Ultimate coupler is not impossible, and it requires a compensating Holography by Yves Gentet), or more recent photopolymers prism for see through, with either a reflective coating or a such as Bayfol® photopolymer by Covestro/Bayer, (Ger- low-index glue line, as done in the Oorym (Israel) light many) [27], and photopolymers by DuPont (US), Polygrama guide combiner concept. (Brazil), or Dai Nippon (Japan)—have been used to implement in-couplers and out-couplers in waveguide com- 4.5.1.2 Embedded cascaded mirrors biners. Such photopolymers can be sensitized to work over Cascaded embedded mirrors with partially reflective coat- a specific wavelength or over the entire visible spectrum ings are used as out-couplers in the Lumus (Israel) LOE (panchromatic holograms). waveguide combiner. The input coupler remains a prism. Photopolymer holograms do not need to be devel- As the LOE is composed of reflective surfaces, it yields good oped as DCG, nor do they need to be bleached like silver color uniformity over the entire FOV. As with other coupler halides. A full-color hologram based on three phase- technologies, intrinsic constraints in the cascaded mirror multiplexed single-color holograms allows for a single design of the LOE might limit the FOV [26]. See through is plate waveguide architecture, which can simplify the very important in AR systems: the Louver effects produced combiner and reduce weight, size, and costs while by the cascaded mirrors in earlier versions of LOEs have increasing yield (no plate alignment required). However, B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 53

the efficiency of such full-RGB phase-multiplexed holo- the technique used by Akonia, Inc. (a US start-up in Colo- grams is still quite low when compared to single-color rado, formerly InPhase Inc., which was originally funded photopolymer holograms. and focused to produce high-density holographic page Also, the limited index swing of photopolymer holo- data-storage media, ruled by the same basic holographic grams allows them to work more efficiently in reflection phase-multiplexing principles [29]). mode than in transmission mode (allowing for better Thick holographic layers, as thick as 500 µm, work confinement of both the wavelength and angular spectrum well in transmission and/or reflection modes, but they bandwidths). need to be sandwiched between two glass plates. In some Examples of photopolymer couplers include Sony specific operation modes, the light can be guided inside the LMX-001 Waveguides for smart glasses and the TrueLife thick hologram medium, where it is not limited by the TIR Optics (UK) process of mastering the hologram in silver angle dictated by the index of the glass plates. As the halide and replicating it in photopolymer. various hologram bandwidths build the final FOV, one Replication of the holographic function in photo- needs to be cautious in developing such phase- polymer through a fixed master has proven to be possible multiplexed holograms when using narrow illumination in a roll-to-roll operation by Bayer (Germany). Typical sources such as lasers. photopolymer holographic media thicknesses range from Replication of such thick volume holograms is difficult 16–70 µm, depending on the required angular and spectral in roll-to-roll operation, as done with thinner single holo- bandwidths. grams (Covestro Photopolymers, H-PDLC), and require multiple successive exposures to build the hundreds of 4.5.2.2 Thin transmission holographic coupler phase-multiplexed holograms that compose the final ho- When the index swing of the volume hologram can be lographic structure. This can however be relatively easy increased, the efficiency gets higher and the operation in with highly automated recording setups as the ones transmission mode becomes possible. This is the case with developed by the now-defunct holographic page data- Digilens’ proprietary holographic polymer-dispersed storage industry (In-Phase Corp., General Electric, and so liquid crystal (H-PDLC) hologram material [28]. Trans- on). mission mode requires the hologram to be sandwiched Note that although the individual holograms acting in between two plates rather than laminating a layer on top or slivers of angular and spectral bandwidth spread the bottom of the waveguide as with photopolymers, DCG, or incoming spectrum like any other hologram (especially silver halides. Digilens’ H-PDLC has the largest index when using LED illumination), the spectral spread over the swing today and can therefore produce strong coupling limited spectral range of the hologram is not wide enough efficiency over a thin layer (typically four microns or less). to alter the modulation transfer function (MTF) of the H-PDLC material can be engineered and recorded to work immersive image and thus does not need to be compen- over a wide range of wavelengths to allow full-color sated by a symmetric in-coupler and out-coupler as with all operation. other grating or holographic structures. This feature allows this waveguide architecture to be asymmetric, such as 4.5.2.3 Thick holographic coupler having a strong in-coupler as a simple prism: a strong in- Increasing the index swing can optimize the efficiency coupler is always a challenge for any grating or holo- and/or angular and spectral bandwidths of the hologram. graphic waveguide combiner architecture, and a macro- However, this is difficult to achieve with most available scopic prism is the best coupler imaginable. materials and might also produce parasitic effects such as Figure 11 shows both thin and thick volume holograms haze. Increasing the thickness of the hologram is another operating in reflection and/or transmission modes. The top option, especially when sharp angular or spectral band- part of the figure shows a typical 1D EPE expander with a widths are desired, such as in telecom spectral and angular single transmission volume hologram sandwiched be- filters. This is not the case for an AR combiner, where both tween two plates. When the field traverses the hologram spectral and bandwidths need to be wide (to process a wide downward, it is in off/Bragg condition, and when it tra- FOV over a wide spectral band such as LEDs). However, a verses the volume hologram upward after a TIR reflection, thicker hologram layer also allows for phase multiplexing it is in an on/Bragg condition (or close to it), thereby over many different holograms, one on top of another, creating a weak (or strong) diffracted beam that breaks the allowing for multiple Bragg conditions to operate in con- TIR condition. cert to build a wide synthetic spectral and/or angular A hologram sandwiched between plates might look bandwidth, as modeled by the Kogelnik theory [30]. This is more complex to produce than a reflective or transmission 54 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 11: Different types of volume holograms acting as in-couplers and out- couplers in waveguide combiners. laminated version, but it has the advantage that it can Increasing the number of phase levels from binary to operate in both transmission and reflection modes at the quarternary or even eight or sixteen levels increases its same time (e.g., to increase the pupil replication diversity). efficiency as predicted by the scalar diffraction theory, for normal incidence. However, for a strong incidence angle and for small periods, this is no longer true. A strong out- 4.5.2.4 Surface-relief grating couplers coupling can thus be produced in either reflection or Figure 12 reviews the various SRGs used in industry today transmission mode. (blazed, slanted, binary, multilevel, and analog) and how Slanted gratings are very versatile elements, and their they can be integrated in waveguide combiners as incou- spectral and angular bandwidths can be tuned by the slant pling and outcoupling elements. angles. Front and back slant angles in a same period (or Covering a SRG with a reflective metallic surface (see from period to period) can be carefully tuned to achieve the Figure 12) will increase dramatically its efficiency in reflec- desired angular and spectral operation. tion mode. A transparent grating (no coating) can also work SRGs have been used as a commodity technology since both in transmission and reflection modes, especially as an mastering and mass replication techniques technologies out-coupler, in which the field has a strong incident angle. were established and made available in the early 1990s

Figure 12: Surface-relief grating types used as waveguide combiner in-couplers and out-couplers. Solid lines indicate reflective coatings on the grating surface, and dashed lines indicate diffracted orders. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 55

[39]. Typical periods for TIR grating couplers in the visible are depicted for clarity), as in a conventional digital pro- spectrum are below 500 nm, yielding nanostructures of jector architecture. The in-couplers have been chosen to be just a few tens of nanometers if multilevel structures are slanted gratings for their ability to act on a specific spectral required. This can be achieved by direct e-beam write, range while letting the remaining spectrum unaffected in i-line (or deep ultra-violet [DUV]) lithography, or even the zero order, to be processed by the next in-coupler area interference lithography (holographic resist exposure) [37]. located on the guide below, and to do this for all three SRG structures can be replicated in volumes by nano- colors. Such uncoated slanted gratings work both in imprint, a microlithography wafer fabrication technology transmission and reflection modes but can be optimized to developed originally for the integrated circuit (IC) industry. work more efficiently in a specific mode. The out-couplers Going from wafer-scale fabrication to panel-scale fabrica- here are also slanted gratings, which can be tuned to tion will reduce costs, allowing for consumer-grade AR and effectively work over a specific incoming angular range MR products. (TIR range) and leave the see-through field quasi unaf- Figure 13 and Figure 14 illustrate how some of the SRGs fected. The part of the see through field that is indeed dif- shown in Figure 12 have been applied to the latest wave- fracted by the out-couplers is trapped by TIR and does not guide combiners such as the Microsoft HoloLens 1 and make it to the EB. These gratings are modulated in-depth to Magic Leap One. Multilevel SRGs have been used by com- provide a uniform EB to the user. Note the symmetric in- panies such as Dispelix Oy, and quasi-analog surface relief coupler and out-coupler configuration compensating the CGHs have been used by others, such as WaveOptics Ltd. spectral spread over the three LED bands. Figure 13 shows the waveguide combiner architecture The redirection gratings are not shown here. Input and used in the Microsoft HoloLens 1 MR headset (2015). The output grating slants are set close to 45°, and the redirec- display engine is located on the opposite side of the EB. The tion grating slants at half this angle. The periods of the single input pupil carries the entire image over the various gratings are tuned in each guide to produce the right TIR colors at infinity (here, only two colors and the central field angle for the entire FOV for that specific color (thus the

Figure 13: Spatially color-multiplexed input pupils with slanted gratings as in-couplers and out-couplers working in transmission and reflection mode (HoloLens 1 MR headset). 56 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 14: Spatially color-demultiplexed input pupils with 100% reflective blazed gratings as in-couplers and binary phase gratings as out- couplers (Magic Leap One MR headset). same central diffraction angle in each guide for each RGB The redirection gratings (not shown here) are also LED color band). composed of binary top-down structures. The periods of Figure 14 depicts the waveguide combiner architecture the gratings are tuned in each guide to produce the right used in the Magic Leap One MR headset (2018). The display TIR angle for the entire FOV for that specific color (same engine is located on the same side as the EB. The input central diffraction angles for each RGB LED color band). pupils are spatially color demultiplexed, carrying the Other companies such as WaveOptics in the UK uses entire FOV at infinity (here again, only two colors and the multilevel and/or quasianalog surface relief diffractive central field are depicted for clarity). structures to implement in-couplers and out-couplers (see Spatial color demultiplexing can be done conveniently Figure 14). This choice is mainly driven by the complexity with a color sequential LCoS display mode for which the of the extraction gratings, acting both as redirection grat- illumination LEDs are also spatially demultiplexed. In this ings and out-coupler gratings, making them therefore more configuration, the input grating couplers are strong blazed complex than linear or slightly curved (powered) gratings, gratings, coated with a reflective metal (such as Al). They similar to iteratively optimized CGHs [40]. Allowing do not need to work over a specific single-color spectral multilevel or quasianalog surface relief diffractive struc- width since the colors are already demultiplexed. The out- tures increases the space bandwidth product of the couplers are simple top-down binary gratings, which are element to allow more complex optical functionalities to be also depth modulated to produce a uniform EB for the user. encoded with relatively high efficiency. These binary gratings are shallow, acting therefore very little on the see through, but they have much stronger ef- 4.5.2.5 RWG couplers ficiency when working in internal reflection diffraction RWGs, also known as guided mode resonant gratings or mode, since the optical path length in this case is longer by waveguide-mode resonant gratings [41], are dielectric a factor of 2ncos(α) than that in transmission mode, (where structures where these resonant diffractive elements n is the index of the guide, and α is the angle if there is benefit from lateral leaky guided modes. A broad range of incidence in the guide). As in the HoloLens 1, most of the optical effects are obtained using RWGs such as waveguide see-through field diffracted by the out-couplers is trapped coupling, filtering, focusing, field enhancement and by TIR. nonlinear effects, magneto-optical Kerr effect, or B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 57

electromagnetically induced transparency. Thanks to their using metasurfaces, the design for manufacturing (DFM) high degree of optical tuning (wavelength, phase, polari- can be compelling. However, optical efficiencies, design zation, intensity) and the variety of fabrication processes tools, and large scale fabrication will need to continue to and materials available, RWGs have been implemented in improve find their way into product. a broad scope of applications in research and industry. RWGs can therefore also be applied as in-couplers and out- 4.5.3 Achromatic coupler technologies couplers for waveguide gratings. Figure 15 shows an RWG on top of a lightguide Waveguide combiners could benefit greatly from a true (referred often incorrectly through the popular AR lingo as achromatic coupler functionality: incoupling and/or out- a “waveguide”), acting as the in-couplers and outcoupling RGB FOVs and matching each color FOV to the couplers. maximum angular range (FOV) dictated by the waveguide Roll-to-roll replication of such grating structures can TIR condition. This would reduce the complexity of mul- help bring down overall waveguide combiner costs. The tiple waveguide stacks for RGB operation. CSEM research center in Switzerland developed the RWG When it comes to implementing a waveguide coupler concept back in the 1980s, companies are now actively as a true achromatic grating coupler, one can either use developing such technologies [90]. embedded partial mirror arrays (as in the Lumus LOE combiner), design a complex hybrid refractive/diffractive 4.5.2.6 Metasurface couplers prism array, or even record phase-multiplexed volume Metasurfaces are a hot topic in research: they can imple- holograms in a single holographic material. However, in ment various optical element functionality in an ultraflat the first case, the 2D exit pupil expansion implementation form factor by imprinting a specific phase function over the remains complex; in the second case, the microstructures incoming wavefront in reflection or transmission (or both) can get very complex and thick; and in the third case, the so that the resulting effect is refractive, reflective, or dif- diffraction efficiency can drop dramatically (as in the fractive or a combination of them. This phase imprint can Konica Minolta or Sony RGB photopolymers combiners or be done through a traditional optical path difference phase in the thick Akonia holographic dual photopolymer jump or through Pancharatnam–Berry phase gratings/ combiner, now part of Apple, Inc.). holograms. It has been recently demonstrated in literature that Due to their large design space, low track length, and metasurfaces can be engineered to provide a true ach- ability to render unconventional optical functions, metal- romatic behavior in a very thin surface with only binary enses could grow out of the laboratory to become an nanostructures [43]. It is easier to fabricate binary unique item in the engineer’s bag of tools. If one can nanostructures than complex analog surface relief dif- implement in a fabricable metasurface an optical func- fractives, and it is also easier to replicate them by tionality that cannot be implemented by any other known nanoimprint lithography (NIL) or soft lithography and optical element (diffractives, holographics, or Fresnels), it still implement a true analog diffraction function as a is particularly interesting. For example, having a true lens or a grating. The high index contrast required for achromatic optical element is very desirable not only in such nanostructures can be generated by either direct imaging but also in many other tasks such as waveguide imprint in high index inorganic spin-on glass or by NIL coupling. Another example is ultralow track length focal resist lift-off after an atomic layer deposition process. stack for IR cameras from Metalenz Corp. Additionally, if Direct dry etching of nanostructured remains a costly one can simplify the fabrication and replication process by option for a product.

Figure 15: Resonant waveguide gratings as in-couplers and out-couplers on a waveguide combiner. 58 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

It is important to remember that metasurfaces or thick CAD tools such as ZemaxTM, CodeVTM, FredTM, or Trace- volume holograms are not inherently achromatic elements, ProTM are sufficient to design effective free space and even and never will be. However, when many narrow band TIR prism combiners and to design waveguide combiners, diffraction effects are spatially or phase multiplexed in a especially when using diffractive or holographic couplers, metasurface or a thick volume hologram, their overall a hybrid ray-trace/rigorous electromagnetic diffraction behavior over a much larger spectral bandwidth can mode is usually necessary. effectively lead the viewer to think they are indeed achro- The modeling efforts is shared between two different matic: although each single hologram or metasurface tasks: operation are strongly dispersive, their cascaded contri- – Local rigorous EM light interaction with micro- and butions may result in a broadband operation which looks anno-optics couplers (gratings, holograms, meta- achromatic to the human eye (e.g., the remaining disper- surfaces, RWGs, and so on). sion of each individual hologram or metasurface effect – Global architecture design of the waveguide combiner, affecting a spectral spread that is below human visual building up FOV, resolution, color and EB, by the use acuity—one arcmin or smaller). It is also possible to phase of more traditional ray trace algorithms. multiplex surface relief holograms to produce achromatic effects but more difficult than with thick volume holograms or thin metasurfaces. 5.1 Waveguide coupler design, Mirrors are of course perfect achromatic elements and optimization, and modeling will therefore produce the best polychromatic MTF (such as with Lumus LOE combiners or LetinAR pin mirror 5.1.1 Coupler/light interaction model waveguides). Modeling of the angular and spectral Bragg selectivity of volume holograms, thin or thick, in reflection and trans- 4.5.4 Summary of waveguide coupler technologies mission modes, can be performed with the couple wave theory developed by Kogelnik in 1969 [31, 32]. Table 1 summarizes the various waveguide coupler tech- Similarly, modeling of the efficiency of SRGs can be nologies reviewed here, along with their specifics and performed accurately with rigorous coupled-wave anal- limitations. ysis (RCWA) [33, 34], especially the Fourier modal Although Table 1 shows a wide variety of optical cou- method (FMM). The finite difference time domain (FDTD) plers, most of today’s AR/MR/smart glass products are method—also a rigorous EM nanostructure modeling based on only a handful of traditional coupler technologies method—can in many cases be a more accurate modeling such as thin volume holograms, slanted SRGs, and technique but also much heavier and more CPU time embedded half-tone mirrors. The task of the optical consuming. However, the FDTD will show all the dif- designer (or rather the product program manager) is to fracted fields, the polarization conversions, and the choose the right balance and the best compromise between entire complex field, whereas the Kogelnik model and coupling efficiency, color uniformity over the EB and FOV, theRCWAwillonlygiveefficiency values for particular mass production costs, and size/weight. diffraction orders. Figure 16 shows the various coupler elements and The FDTD can model nonperiodic nanostructures, waveguide architectures grouped in a single table, including while RCWA can accurately model quasiperiodic struc- SRG couplers, thin holographic couplers, and thick holo- tures. Thus, the FDTD might help with modeling k-vector graphic couplers in three, two, and single flat guides. For variations (rolled k-vector) along the grating, slant, depths, geometric waveguide combiners that use embedded mirrors and duty cycle variations, as well as random and system- or other reflective/refractive couplers (such as microprisms). atic fabrication errors in the mastering and replication steps. The Kogelnik theory is best suited for slowly varying 5 Design and modeling of optical index modulations with moderate index swings (i.e., photopolymer volume holograms). waveguide combiners Free versions of the RCWA-FMM [35] and FDTD [36] codes can be found on the Internet. The Kogelnik theory can be easily Designing and modeling a waveguide combiner is very implemented as a straightforward equation set for transmission different from designing and modeling a freespace optical and reflection modes. Commercial software suites implement- combiner. As conventional ray trace in standard optical ing FDTD and RCWA are R-Soft from Synopsys and Lumerical. Table : Benchmark of various waveguide coupler technologies.

Waveguide Operation Reflective Transmission Efficiency Lensed Spectral Color Dynamically Polarization Mass Company/Product coupler tech coupling coupling modulation out- dispersion. uniformity tunable maintaining production coupler

Embedded Reflective Yes No Complex No Minimal Good No Yes Slicing, Lumus Ltd. DK mirrors coatings coating, polishing. Micro-prisms Reflective Yes No Coatings No Minimal Good No Yes Injection Optinvent SaRL. molding ORA Surface relief Diffractive Yes Yes Depth, duty Yes Strong Needs Possible with No NIL (wafer, Microsoft HoloLens, slanted grating cycle, slant comp. LC plate) Vuzix Inc, Nokia… Surface relief Diffractive Yes No Depth No Strong Needs Possible with No NIL (wafer, Magic Leap One, blazed grating comp. LC plate) Surface relief bi- Diffractive Yes Yes Depth, duty Yes Strong Needs Possible with No NIL (wafer, Magic Leap One nary grating comp. LC plate) headsets reality mixed for combiners Waveguide Chatterjee: I. and Kress B.C. Multilevel surface Diffractive Yes Yes Depth, duty Yes Strong Needs Possible with Possible, but NIL (wafer, WaveOptics Ltd, relief grating cycle comp. LC difficult plate) BAE. Dispelix. Thin photo- Diffractive Yes Yes Index swing Yes, but Strong Needs Possible with No NIL (wafer, Sony Ltd, Trueli- polymer hologram difficult comp. shear plate) feOptics Ltd, H-PDLC volume Diffractive No Yes Index swing Yes, but Strong OK Yes No Exposure Digilens Corp. holographic difficult (electrical)( (MonoHUD) Thick photo- Diffractive Yes Yes Index swing Yes, but Minimal OK No No Multiple Akonia Corp (now polymer hologram difficult exposure Apple Inc.) Resonant wave- Diffractive Yes Yes Depth. Duty Yes Can be NA Possible with Possible Roll to roll NIL CSEM/Resonannt guide grating cycle mitigated LC screens Metasurface Mostly Yes Yes Various Yes Can be Needs Possible with Possible NIL (wafer, Metalenz Corp. coupler diffractive mitigated comp. LC plate) 59 60 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 16: Summary of waveguide combiner architectures with 1D or 2D EPE schemes.

These models predict the efficiency in each order for a Waveguide couplers have specific angular and spectral single interaction of the light with the coupler element. In bandwidths that affect both the FOV and the EB uniformity. order to model the entire waveguide combiner, especially A typical breakdown of the effects of a 2D EPE waveguide when a pupil replication scheme is used, conventional ray architecture on both spectral and angular bandwidths on tracing optical design software can be used, such as the resulting immersive display is shown in Figure 18. Zemax, or more specific light-propagation software mod- Figure 18 shows that the coupler’sspectralandangular ules, such as the ones by LightTrans, Germany [37] (see bandwidths are critical to the FOV uniformity, especially co- Figure 17 for ray tracing through 2D EPE grating lor uniformity. While embedded mirrors and microprisms waveguides). have a quasiuniform effect on color and FOV, others do not, The interaction of the EM field with the coupler regions such as gratings and holograms. It is therefore interesting to (surface relief structures or index modulations) modeled have the flattest and widest spectral and angular bandwidths through the RCWA or Kogelnik can be implemented via a possible. For volume holograms, this means operating in dynamically linked library (DLL) in conventional optical reflection mode and having a strong index swing (Kogelnik), design software based on ray tracing (e.g., C or Matlab and for surface gratings, this means a high index (as predicted code). As the FDTD numerical algorithm propagates the by the RCWA-FMM or FDTD). The angular bandwidth location entire complex field rather than predicting only efficiency can be tuned by the slant angle in both holograms and surface values (as in the RCWA or Kogelnik model), it is therefore gratings. Multiplexing bandwidths can help to build a larger more difficult to implement as a DLL. overall bandwidth, bot spectral, and angular and is used in Raytrace optimization of the high-level waveguide various implementations today. Such multiplexing can be combiner architecture with accurate EM light/coupler done in phase, in space, or in time or a combination of the interactions modeling are both required to design a above. Finally, as spectral and angular bandwidths are combiner with good color uniformity over the FOV, a closely linked, altering the spectral input over the field can uniform EB over a target area at a desired eye relief, and have a strong impact on FOV and vice versa. high efficiency (in one or both polarizations). Inverse Polarization and degree of coherence are two other propagation from the EB to the optical engine exit pupil dimensions one should need to investigate especially is a good way to simplify the optimization process. The when lasers or VCSELs are used in the optical engine or if design process can also make use of an iterative algo- polarization maintaining (or rather polarization con- rithm to optimize color over the FOV/EB and/or effi- version) is required. The multiple interactions in the R-E ciency or even reduce the space of the grating areas by regions can produce multiple miniature Mach–Zehnder making sure that no light gets lost outside the effective interferometers, which might modulate the intensity of EB. the particular fields. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 61

Figure 17: Waveguide grating combiner modeling by LightTrans (Germany) in 2D EPE version.

5.1.2 Increasing FOV by using illumination spectrum waveguide (linked to the refractive index of the waveguide material). The ultimate task for a holographic or grating coupler is to We have seen that volume holographic combiners have provide the widest FOV coupling possible, matching the been used extensively to provide a decent angular incou- FOV limit dictated by the TIR condition in the underlying pling and outcoupling into the guide. However, most of the

Figure 18: Cascaded effects of the field/coupler interactions on the FOV uniformity. 62 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 19: Spectral source bandwidth building larger FOV (angular bandwidth) for photopolymer volume holographic coupler in waveguide combiners.

available holographic materials today have a low index However, this comes at the cost of color uniformity: the swing and thus yield a relatively small angular bandwidth in lower angles (left side of the FOV) will have more contri- the propagation direction. In this case, the FOV bottleneck is butions from the shorter wavelengths (540 nm), and the the coupler not the TIR condition in the waveguide. higher angles (right side of the FOV) will have more con- A typical Kogelnik efficiency plot in the angular/ tributions from the longer wavelengths (560 nm). This spectral space for a reflection photopolymer volume ho- slight color nonuniformity over the FOV is typical for vol- lographic coupler is shown Figure 19 (spectral dimension ume holographic couplers. vertical and angular dimension horizontal). The hologram specifications and exposure setup in 5.1.3 Increasing FOV by optimizing grating coupler Figure 19 are listed below: parameters – Mean holographic material index: 1.53, – Holographic index swing: 0.03, Unlike holographic couplers, which are originated and repli- – Photopolymer thickness: 16 µm, cated by holographic interference in a phase change media – Operation mode: reﬂective, (see previous section), SRGs are rather originated by traditional – Polarization: (“s” but very little change when moving IC lithographic techniques and replicated by NIL or soft to “p” polarization), lithography. The topological structure of the gratings can – Design wavelength: 550 nm, therefore be optimized digitally to achieve the best function- – Reconstruction wavelength: LED light from 540– alityinbothspectralandangulardimensions.Topological 560 nm (20-nm bandwidth), optimization needs to account for DFM and typical litho- – Normal incidence coupling angle: 50° in air. graphic fabrication limitations. The angular bandwidth of an SRG coupler (i.e., the FOV that can be processed by this SRG) When using a laser (<1-nm line) as a display source (such as in can be tuned by optimizing the various parameters of such a a laser MEMS display engine), the max FOV is the horizontal grating structure, such as the front and back slant angles, the cross section of the Kogelnik curved above (17-degree FWHM). grating ﬁll factor, the potential coating(s), the grating depth, However, when using the same color as an LED source (20 nm and of course the period of the grating (Figure20). Additional wide, such as in an LED-lit LCoS micro-display light engine, the material variables are the refractive indices of the grating resulting FOV is a slanted cross-section (in this case increased structure, grating base, grating coating, grating top layer, and to 34-degree FWHM), and a 2× FOV gain is achieved without underlying waveguide. changing the waveguide index or the holographic coupler, Figure 20 shows how the SRG grating parameters can only the illumination’s spectral characteristics. be optimized to provide a larger FOV, albeit with a lower B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 63

Figure 20: Optimizing the grating parameters to optimize color uniformity over the FOV. overall efficiency, matching better the available angular 5.1.4 Using dynamic couplers to increase waveguide bandwidth provided by the TIR condition in the guide. combiner functionality Lower efficiency is okay over the out-couplers since they are tuned in the low-efficiency range to produce a uniform Switchable or tunable TIR couplers can be used to optimize EB (the in-coupler, however, needs to be highly efficient any waveguide combiner architecture, as in since there is only one grating interaction to couple the – Increasing the FOV by temporal sub-FOV stitching at entire field into TIR mode). double the refresh rate, Calculations of coupling efficiency have been carried – Increasing the brightness at the eye by steering a out with an RCWA FMM algorithm and topological optimiza- reduced size EB to the pupil position (thus also tion by a steepest descent algorithm. Note that both unopti- increasing the perceived EB size), and mized and optimized gratings have the same grating periods as – Increasing the compactness of the waveguide wellasthesamecentralslantangletopositionrespectivelythe combiner by switching multiple single-color couplers spectral and the angular bandwidths on identical system in color sequence in a single guide. design points (with the FOV generated by the display engine and wavelength of the illumination source). Dynamic couplers can be integrated in various ways: The bottleneck in FOV with the unoptimized grating polarization diversity with polarization-dependent couplers structure is not the TIR condition (i.e., the index of the wave- (the polarization switching occurring in the optical engine), guide) but rather the grating geometry and the index of the reconfigurable surface acoustic wave or acousto-optical grating. The angular bandwidth of the optimized grating modulator couplers, electro-optical modulation of buried should overlap the angular bandwidth of the waveguide TIR gratings,switchableSRGsinanLClayer,switchablemeta- condition for best results over the largest possible FOV. Also, a surfaces in an multilayer LC layer, tunable volume holograms “top hat” bandwidth makes the color uniformity over the FOV (by shearing, pressure, pulling), or switchable H-PDLC, as in less sensitive to systematic and random fabrication errors in Digilens’ volume holographic couplers. the mastering and the NIL replication of the gratings. Increasing the index of the grating and reducing the back slant while increasing the front slant angle can provide such an 5.2 High-level waveguide-combiner design improvement. Additional optimizations over a longer stretch of The previous section discussed ways to model and opti- the grating can include depth modulations, slant mize the performance of individual couplers, in either modulations (rolling k-vector), or duty cycle modula- grating or holographic form. We now go a step further and tions to produce an even wider bandwidth over a large, look at how to design and optimize the overall waveguide uniform EB. combiner architectures.

Figure 21: More functional coupler architectures that yield compact and efficient waveguide combiners. 64 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

5.2.1 Choosing the waveguide coupler layout efficiency of the out-coupler needs to gradually increase in architecture the propagation direction to produce a uniform EB. This complicates the fabrication process of the couplers, espe- We have seen that couplers can work in either transmission cially when the gradual increase in efficiency needs to or reflection mode to create a more diverse exit-pupil happen in both pupil replication directions. replication scheme (producing a more uniform EB) or to For volume holograms, the efficiency can be increased by improve the compactness of the waveguide by using both a stronger index swing in the photopolymer or PDLC (through surfaces, front and back. The various couplers might direct a longer exposure or a thickness modulation). For SRGs, there the field in a single direction or in two or more directions, are a few options, as shown in Figure 22. This is true for the potentially increasing the FOV that can propagate in the redirection grating (R-E) as well as the out-coupler (O-E). waveguide without necessarily increasing its index. Groove depth and duty cycle modulation can be Figure 21shows how the optical designer can expand the performed on all type of gratings, binary, multilevel, functionality of in-couplers or out-couplers, with architectures blazed, and slanted (see Figure 22). Duty cycle modula- ranging from bidimensional coupling to dual reﬂective/trans- tion has the advantage of modulating only the lateral mission operation in the same guide with sandwiched volume structures, not the depth, which makes it an easier holograms or top/bottom grating couplers. mastering process. Modulating the depth of the gratings More complex and more functional coupler architectures canbedoneinbinarysteps(asintheMagicLeapOne, have specific effects on MTF, efficiency, color uniformity, and Figure 22—right) or in a continuous way (Digilens wave- FOV. For example, while the index of the guide allows for a guide combiners). larger FOV to propagate, the index of the grating structures in Grating front- and back-slant angle modulation (in a air would increase the spectral and angular bandwidths to single grating period or over a larger grating length) can process a larger FOV without compromising color uniformity change the angular and spectral bandwidths to modulate or efficiency. The waviness of the waveguide itself will impact efficiency and other aspects of the coupling (angular, spec- the MTF as random cylindrical powers added to the field. tral, polarization). Periodic modulation of the slant angles is Multiple stacked waveguides might be efficient at processing sometimes also called the “rolling k-vector” technique and single colors, but their misalignment will impact the MTF as can allow for larger FOV processing due to speciﬁc angular misaligned color frames. Similarly, hybrid top/bottom cou- bandwidth management over the grating area. Once the plers will affect the MTF if they are not perfectly aligned master has been fabricated with the correct nanostructure (angular alignment within a few arc seconds). modulation, the NIL replication process of the gratings is the same no matter the complexity of the nanostructures (caution 5.2.2 Building a uniform EB is warranted for slanted gratings where the NIL process must resolve the undercut structures; however, the slanted grating As the TIR field gets depleted when the image gets NIL process (with slants up to 50°) has been mastered by extracted along the out-coupler region, the extraction many foundries around the world [37]).

Figure 22: Modulation of the outcoupling efficiency to build up a uniform EB. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 65

Figure 23: Spectral spread compensation in a symmetric in-coupler/out-coupler waveguide combiner.

5.2.3 Spectral spread compensation in diffractive This symmetric technique might not be used to waveguide combiners compensate for spectral spread across different colors (RGB LEDs) but rather for the spread around a single LED Spectral spread comes to mind as soon as one speaks color. The spread across colors might stretch the RGB exit about gratings or holographic elements. It was the first pupils too far apart and reduce the FOV over which all RGB and is still the main application pool for gratings and colors can propagate by TIR. holograms: spectroscopy. Spectral spread is especially The pupil replication diversity can also be increased by critical when the display illumination is broadband, introducing a partial reflective layer in the waveguide (by such as with LEDs (as in most of the waveguide grating combining two plates with a reflective surface), thus pro- combiner devices today, such as the HoloLens 1, Vuzix, ducing a more uniform EB in color and field. Magic Leap, Digilens, Nokia, and so on), with a notable difference in the HoloLens 2 (laser MEMS display en- 5.2.4 Field spread in waveguide combiners gine). The straightforward technique to compensate for the inevitable spectral spread is to use a symmetric in- The different fields propagating by TIR down the guide are coupler/out-coupler configuration in which the gratings also spread out, no matter the coupler technology (mirrors, or holograms work in opposite direction and thus prisms, gratings, holograms, and so on), see Figure 24. compensate in the out-coupler any spectral spread A uniform FOV (i.e., all fields appearing) can be formed impacted in the in-coupler (Figure 23). over the EB with a strong exit pupil diversity scheme. This Although the spectral spread might be compensated, is a concept often misunderstood as in many cases, only one can notice in Figure 23 that the individual spectral one field is represented when schematizing a waveguide bands are spatially demultiplexed at the exit ports while combiner. Figure 24 shows the field spread occurring in a multiplexed at the entry port. Strong exit-pupil replication diffractive waveguide combiner. The number of replicated diversity is thus required to smooth out any color non- fields is also contingent on the size of the human eye pupil. uniformities generated over the EB. If the ambient light gets bright, i.e., the human eye pupil

Figure 24: Fractional field spread in a waveguide combiner. 66 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 25: Focus spread in a waveguide combiner with a noncollimated input field. gets smaller, then only part of the FOV might appear to the spread over the exit pupils, as discussed previously. user, missing a few ﬁelds. Figure 25 shows such a focus spread over the EB from an input pupil over which the image is formed in the near 5.2.5 Focus spread in waveguide combiners ﬁeld. The image over the input pupil can, however, be When a pupil replication scheme is used in a waveguide located in the near field when no pupil replication scheme combiner, no matter the coupler, the input pupil needs to is performed in the guide, such as in the Epson Moverio be formed over a collimated field (image at infinity/far BT300 or in the Zeiss “Tooz” Smart Glass (yielding a small field). If the focus is set to the near field instead of the far FOV and small EB). field in the display engine, each waveguide exit pupil When pupil replication is used in the guide, the virtual will produce an image at a slightly different distance, image can be set at a closer distance for better visual thereby producing a mixed visual experience, over- comfort by using a static (or even tunable) negative lens lapping the same image with different focal depths. It is acting over the entire EB. For an unperturbed see-through quasi-impossible to compensate for such focus shift over experience, such a lens needs to be compensated by its the exit pupils because of both spectral spread and field conjugate placed on the world side of the combiner

Figure 26: Two out-coupler architectures positioning the virtual image in the near field over all exit pupils. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 67

waveguide. This is the architecture used in the Microsoft A powered out-coupler grating might reduce the MTF HoloLens 1 (2015) [37]. of the image, especially in the direction of the lens offset Another, more compact, way would introduce a slight (direction of TIR propagation), since the input (I-E) and optical power in the O-E, so that this coupler takes the func- output (O-E) couplers are no more perfectly symmetric (the tionality of an off-axis lens (or an off-axis diffractive lens) input coupler being a linear grating in both cases, and the rather than that of a simple linear grating extractor or linear out-coupler an off-axis diffractive lens). Thus, the spectral mirror/prism array. Although this is difficult to implement with spread of the image in each color band cannot be a mirror array (as in an LOE), it is fairly easy to implement with compensated perfectly and will produce lateral coromatic a grating or holographic coupler. The grating lens power does aberations (LCA) in the direction of the lens offset. This can not affect the zeroth diffraction order that travels by TIR down be critical when using an LED as an illumination source, the guide but affects only the outcoupled (or diffracted) field. but it would affect the MTF much less when using narrower The see-through field is also not affected by such a lensed out- spectral sources, such as lasers or VCSELs. coupler since the see-through field diffracted by such an One of the main problems with such a lensed out-coupler element would be trapped by TIR and thus not enter the eye grating configuration when attempting to propagate two colors pupil of the user. in the same guide (for example, a two-guide RGB waveguide All three configurations (no lens for image at infinity, static architecture, as in Figure 34) is the generation of longitudinal lens with its compensator, and powered O-E grating) are chromatic aberrations (due to the focus changing with color showninFigure26.TheleftpartoftheEBshowsanextracted since the lens is diffractive). Using a single color per guide and field with image at infinity (as in the Lumus DK40—2016), the a laser source can greatly simplify the design task. center part shows an extracted field with image at infinity that passes through a negative lens to form a virtual image closer 5.2.6 Propagating full color images in the waveguide to the user and its counterpart positive lens to compen- combiner over a maximum FOV sate for see-through (as in the Microsoft HoloLens 1, 2015), and the right part of the EB shows an extracted We have seen in the previous paragraphs that the spectral field with the image directly located in the near field spread of grating and holographic couplers can be through a powered grating extractor (as with an off-axis perfectly compensated with a symmetric in-coupler and diffractive lens, e.g., the Magic Leap One, 2018). out-coupler configuration. This is possible over a single- For example, a half-diopter negative lens power would color band but will considerably reduce the FOV if used position the original extracted far field image to a more over the various color bands (assuming that the couplers comfortable 2-m distance, uniformly over the entire EB. will work over these various spectral bands).

Figure 27: Stacked waveguides combiners that provide the largest FOV TIR propagation over three colors. 68 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 28: k-vector diagram and lateral pupil replication layout for a single guide and single color.

In order to maximize the RGB FOV in a waveguide out-coupler, the original FOV can be reconstructed by combiner, one solution is to use stacked guides optimized overlapping both partial FOVs, as in Figure 29. each for a single-color band, each coupling a maximum In the orthogonal direction, the FOV that can be FOV by tuning the diffraction angle of the in-couplers and coupled by TIR remains unchanged. This concept can be out-couplers accordingly. This is the architecture used in taken to more than one dimension, but the coupler space both HoloLens 1 and Magic Leap One (see Figure 27, on the waveguide can become prohibitive. although the position of the input pupil (light engine) is opposite in both devices. 5.2.7 Waveguide-coupler lateral geometries Air gaps between all plates are required to produce the TIR condition. Such gaps also allow for additional poten- We have reviewed the various coupler technologies that tial filtering in between plates for enhanced performance can be used in waveguide combiners, as well as the 2D (such as spectral and polarization filtering). exit pupil expansion that can be performed in wave- Figure 28 shows the functional diagram of such a guide combiners. Waveguide combiners are desirable single-color plate as a top view as well as its k-vector space since their thickness is not impacted by the FOV, unlike depiction. Here again, I-E refers to the in-coupler, R-E refers other combiner architectures such as free-space or to the leaky 90-degree redirection element, and O-E refers TIR prisms. However, the lateral dimensions of the to the leaky out-coupler that forms the ﬁnal EB (for 2D pupil waveguide (especially the redirection coupler and out- replications). coupler areas over the waveguide) are closely linked to Note that the entire FOV is shown on the k-vector di- size of the incoupled FOV, as shown in Figure 30. For agram (Figure 28), but only a single ﬁeld (central pixel in example, the R-E region geometry is dictated by the the FOV, with entry normal to the guide) is shown in the EB FOV in the waveguide medium: it expands in the di- expansion schematic. rection orthogonal to the TIR propagation, forming a The FOV in the direction of the incoupling can be conical shape. increased by a factor of two when using a symmetric The largest coupler area requirement is usually the incoupling configuration in which the input grating or out-coupler element (center), aiming at processing all hologram (or even prism[s]) would attempt to couple the FOVs and building up the entire EB. Eye relief also strongly entire FOV to both sides, with one of the input configura- impacts this factor. However, its size can be reduced in a tions described in Figure 13 or Figure 14. “human-centric optical design” approach: the right part of As the TIR angular range does not support such an the FOV at the left edge of the EB as well as the left part of enlarged FOV, part of the FOV is coupled to the right and the FOV at the right edge of the EB can be discarded, thus part of the FOV is coupled to the left. Due to the opposite considerably reducing the size of the O-E without directions, opposite sides of the FOV travel in each direc- compromising the image over the EB. Note that in tion. If such TIR fields are then joined back with a single Figure 29, the k-vector diagram (a) shows the FOV, whereas B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 69

Figure 29: Symmetric incoupling for FOV increase in the direction of incoupling. the lateral schematics of the waveguide in (b) and (c) show outcoupling of the input pupil and also for etendue limitations the actual size of the coupler regions. in the display engine), a semitransparent Fresnel surface can Reducing the input pupil can help to reduce the overall be used inside the guide (as in two guides bounded together), size and thickness of the combiner. However, the thickness of whichwouldreflectonlypartofthefieldandleavetheother the guide must be large enough not to allow for a second I-E part unperturbed, effectively increasing the exit pupil diversity. interaction with the incoming pupil after the first TIR bounce. If Figure 32 shows how the space of the out-coupler there is a second interaction, then by the principle of time grating is dictated solely by the FOV and the EB. Note that reversal, part of the light will be outcoupled and form a partial many fields can be canceled at the edges and toward the pupil (partial moon if the input pupil is circular) propagating edges of the EB, as they will not enter the eye pupil (right down the guide instead of the full one. This is more pro- fields on the left EB edge and left fields on the right EB edge). nounced for the smallest field angle, as depicted in Figure 31. This can also reduce the size of the redirection grating However, if the polarization of the field is altered after considerably. This holds true for both EB dimensions. the first TIR reflection at the bottom of the guide, the parasitic outcoupling can be reduced if the I-E is made to be 5.2.8 Reducing the number of plates for RGB display and highly polarization sensitive. maintain FOV reach Reducing the waveguide thickness can also produce stronger pupil diversity over the EB and thus better EB uni- Reducing the number of plates without altering the color of formity. If reducing the guide is not an option (for parasitic the image while propagating the maximum FOV allowed by

Figure 30: Redirection and out-coupler areas as dictated by the incoupled FOV. 70 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

Figure 31: Effects of the input pupil size (and size of the I-E) and thickness of guide on a single field TIR pupil bouncing down the guide. the index of the guide is a desirable feature since it reduces A lower spectral spread, such as through a prism in- the weight, size, and complexity of the combiner and make it coupler, would increase the RGB FOV overlap in a single guide, also less prone to MTF reductions due to guide mis- such as in an LOE (embedded partial mirrors out-couplers) from alignments. Both lateral and longitudinal angular wave- Lumus or in the microprism array couplers from Optinvent. guide misalignments will contribute to a reduction of the The left configuration in Figure 33 acts as a hybrid MTF built by the display engine. Waveguide surface flatness spatial/spectral filter, filtering the left part of the blue FOV, issues are yet more cause for MTF reduction. allowing the entire green FOV to be propagated (if the Due to the strong spectral spread of the in-coupler el- grating coupler periods have been tuned to match the ements (gratings, holograms, RWGs, or metasurfaces), the green wavelength), and filtering the right part of the red individual color fields are coupled at higher angles as the FOV. The configuration in Figure 33 propagates the entire wavelength increases, which reduces the overall RGB FOV RGB FOV (assuming the couplers can diffract uniformly overlap that can propagate in the guide within the TIR over the entire spectrum) at the cost of the FOV extending conditions (smallest angle dictated by the TIR condition in the direction of the propagation. However, when and largest angle dictated by pupil replication re- considering binocular vision, this limitation could be quirements for a uniform EB). This issue is best depicted in mitigated by engineering a symmetric color vigneting in the k-vector diagram (Figure 33). each eye (particularly on blue and red), providing a

Figure 32: Eyebox and FOV dictate the size of the out-coupler area. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 71

Figure 33: k-vector diagram of a single-plate waveguide combiner using (a) RGB FOV coupling over a single-color TIR angular range condition and (b) RGB reduced FOV sharing the same TIR range.

Figure 34: Two-guide RGB waveguide combiner configuration. 72 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

uniform stereo color vision in a single RGB guide with high multiplexing is theoretically possible in volume holo- FOV (e.g., Dispelix Oy). grams. This might be achieved in the Akonia (now Apple) Recently, two plate RGB waveguide combiner archi- thick holographic material (500 µm). If a thinner photo- tectures have been investigated, reducing by one third the polymer (less than 20 µm) is desired for better reliability weight and size of traditional three-guide combiners, and easier mass production, a large holographic index where the green FOV is shared between the top and bottom swing is required. Standard photopolymers can be layer (see Figure 34. Various companies are using this two- panchromatic and can also be phase multiplexed, but the plate RGB waveguide combiner architecture today, resulting efficiency remains low, and color cross- including Vuzix, WaveOptics, and Digilens. contamination between holograms is an additional issue. However, this requires the grating (or holograms, This is also theoretically possible with SRGs, but it is RWGs, or metasurfaces) to be efficient over a larger spectral difficult to simultaneously achieve high efficiency and a band, which implies that SRGs are to be replicated in a high extinction ratio over the three color bands. Meta- higher refractive index, widening their spectral (and surfaces and RWGs can theoretically produce such phase- angular) bandwidths. High-index grating replication by multiplexed layers but with the same limitations. NIL stretches the traditional wafer-scale NIL resin material Another solution is to spatially interleave various science (inclusion of TiO2 or ZrO2 nanofiller particles). grating configurations by varying the periods, depths, and Nanoimprint at a Gen2 panel size of higher-index inorganic slant angles. This is, however, difficult to achieve practi- spin-on glass material might be the best fit, which also cally. Yet another solution to solve the single RGB guide solves the resin or photopolymer reliability issues over problem would time multiplex RGB gratings through a various environmental conditions (temperature, pressure, switchable hologram, such as the ones produced by Dig- shear, UV exposure, and humidity). ilens Corp. This switching technique could also produce This two-guide RGB configuration splits the green FOV in much larger FOVs multiplexed in the time domain and two at the in-coupler region and merges them again over the fused in the integration time of the human eye. out-coupler region. For good color uniformity over the FOV and the EB, especially in the green field, this technique requires perfect control of the two-guide efficiency balance. Pre- emphasis compensation of the guide mismatch is possible 6 Conclusion using the display dynamic range, but this requires precise calibration, reduces the final color depth, and does not solve The aim of this review paper was to capture the state of the the stitching region issue where the two fields overlap. art in waveguide combiner optics for AR and MR headsets, An alternative to the architecture uses the first guide to especially as diffractive waveguide combiners (surface re- propagate green and blue FOVs and the second guide to lief diffractives, volume holographic, and others such as propagate only the red FOV, as green and blue are closer metasurfaces, resonant gratings, and so on). We also spectrally to each other than red. This change, however, reviewed the various geometric waveguides combiner ar- slightly reduces the allowed FOV traveling without chitectures which are rather based on refractive and vignetting but solves the green FOV stitching problem. reflective elements. Although going from three plates to two plates brings a We showed that for optimum results, the waveguide small benefit in size, weight, and cost, the added grating, display engine, and sensors need to be codesigned as complexity of the color split geometry and the resulting a global system to closely match the optical performances color nonuniformities over the EB might overshadow the and the specific features and limitations of the human visual initial small benefits. system, through human-centric optical design. A single-plate RGB waveguide combiner would pro- The coming years will be an exciting time for MR vide a much stronger benefit, as there is no need to align hardware. A full ecosystem to allow for commodity mass multiple guides anymore because everything is aligned production and lower costs of waveguide grating com- lithographically by NIL inside the single plate (potentially biners is growing worldwide, comprising high-index also front and back). This would also yield the best possible ultraflat glass wafer manufacturers, high-index resin ma- MTF and the lowest costs. terial developers, process equipment developers, NIL One single-plate solution is to phase multiplex three equipment developers, and also dedicated software design different color couplers with three different periods into a tools developers allow finally this technology to emerge as single layer and then tune it so that there is no spectral a viable option for the upcoming consumer MR and smart overlap (no color ghost images over the EB). Such phase glass market. B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 73

However, one has to remember that delivering on [15] J. Kimmel and T. Levola, “Diffractive backlight light guide the promises of the ultimate wearable display hardware plates in mobile electrowetting display applications,” SID 09 is only one aspect of the trial and opportunity ahead for Paper 471 Page 2. [16] J. Liu, N. Zhang, J. Han, et al., “An improved holographic MR, delivering on strong use cases, especially for con- waveguide display system,” Appl. Optic., vol. 54, no. 12, sumer markets, will be the other critical item to pp. 3645–3649, 2015. consider. [17] T. Yoshida, K. Tokuyama, Y. Takai, et al., “A plastic holographic waveguide combiner for light-weight and highly-transparent ” Author contribution: All the authors have accepted augmented reality glasses, J. SID, vol. 26, no. 5, 2018. [18] H. Mukawa, K. Akutsu, I. Matsumura et al., “A full color eyewear responsibility for the entire content of this submitted display using holographic planar waveguides” 8.4, SID 08 manuscript and approved submission. DIGEST 2008. Research funding: None declared. [19] T. Oku, K. Akutsu, M. Kuwahara et al., “High-luminance see- Conflict of interest statement: The authors declare no through eyewear display with novel volume hologram waveguide conflicts of interest regarding this article. technology,” 15.2, 192 • SID DIGEST 2015. [20] K. Sarayeddine, P. Benoit, G. Dubroca, and X. Hugel, “Monolithic low-cost plastic light guide for full colour see through personal video glasses,” in ISSN-L 1883-2490/17/1433 ITE and SID (IDW References 10), 2010, pp. 1433–1435. [21] T. Levola, “Exit pupil expander with a large field of view based on [1] W. S. Colburn and B. J. Chang, “Holographic combiners for head diffractive optics,” J. Soc. Inf. Disp., vol. 17, pp. 659–664, 2009. up displays,” Tech Report No. AFAL-TR-77-110, 1977. [22] T. Levola, “Diffractive optics for virtual reality displays,” J. SID, [2] J. Jerald, The VR Book: Human Centered Design for Virtual Reality, vol. 14, no. 5, 2006. ACM Books, 2016, 978-1-97000-112-9. [23] B. Kress, “Diffractive and holographic optics as optical [3] W. Barfield, Fundamentals of Wearable Computers and combiners in head mounted displays,” in Proceedings of the Augmented Reality, 2nd ed., CRC Press, Taylor and Francis 2013 ACM Conference on Pervasive and Ubiquitous Computing – Group, 2015, 978-1-482243595. Ubicomp’13, 2013, pp. 1479–1482. [4] L. Inzerillo, “Augmented reality: past, present and future,” in The [24] A. Cameron, “Optical waveguide technology & its application in Engineering Reality of Virtual Reality,Vol.8649,M.Dolinskyand head mounted displays,” in Head- and Helmet-Mounted I. E. McDowall, Eds. Proc. of SPIE-IS&T Electronic Imaging, SPIE, 2013. Displays XVII; and Display Technologies and Applications for [5] R. T. Azuma, “A survey of augmented reality,” in Presence, Defense, Security, and Avionics VI, Vol. 8383, P. L. Marasco, Teleoperators and Virtual Environments, vol. 6, pp. 355–385, P. R. HavigII, D. D. Desjardins, and K. R. Sarma, Eds., Proc. of 1997. SPIE, p. 83830E. [6] O. Cakmakci, J. Rolland, “Head-worn displays: a review,” [25] M. Homan, “The use of optical waveguides in Head up Display J. Display. Technol., vol. 2, pp. 199–216, 2006. (HUD) applications,” in Display Technologies and Applications [7] J. Rolland and O. Cakmakci, “Head-worn displays: the future for Defense, Security, and Avionics VII, Vol. 8736, D. D. Desjardins through new eyes,” Opt. Phot. News, vol. 20, pp. 20–27, 2009. andK.R.Sarma,Eds,Proc.ofSPIE. [8] D. W. F. Van Krevelen and R. Poelman, “A survey of augmented [26] D. Cheng, Y. Wang, C. Xu, W. Song, and G. Jin, “Design of an reality technologies, applications and limitations,” Int. J. Virtual ultra-thin near-eye display with geometrical waveguide and Real., vol. 9, pp. 1–20, 2010. freeform optics,” Opt. Express,vol.22,no.17,pp.20705– [9] K.-L. Low, A. Ilie, G. Welch, A. Lastra, “Combining head-mounted 20719, 2014. and projector-based displays for surgical training,” IEEE Virtual [27] D. Jurbergs, F.-K. Bruder, F. Deuber, et al., “New recording Real., 2003, Proceedings 10.1109/VR.2003, https://doi.org/10. materials for the holographic industry,” in Practical Holography 1109/VR.2003.1191128. XXIII: Materials and Applications, Vol. 7233, H. I. Bjelkhagen and [10] Y. Amitai, A. Friesem, and V. Weiss, “Holographic elements with R. K. Kostuk, Eds. Proc. of SPIE. high efficiency and low aberrations for helmet displays,” Appl. [28] www.digilens.com. Opt., vol. 28, pp. 3405–3416, 1989. [29] K. Curtis and D. Psaltis, “Cross talk in phase coded holographic [11] N. Baker, Mixed Reality Keynote at Hot Chips HC28 – Symposium memories,” J. Opt. Soc. Am. A, vol. 10, no. 12, December 1993, for High Performance Chips, Aug. 21-23 2016, www.hotchips.org. https://doi.org/10.1364/JOSAA.10.002547. [12] B.Kress and W.Cummins, Towards the Ultimate Mixed Reality [30] H. Kogelnik, “Coupled wave theory for thick hologram gratings,” Experience: HoloLens Display Architecture Choices, SID 2017 Bell Syst. Tech. J., vol. 48, 1969, https://doi.org/10.1002/j.1538- Book 1: Session 11: AR/VR Invited Session II. 7305.1969.tb01198.x. [13] J. Michael Miller, N. de Beaucoudrey, P. Chavel, J. Turunen, and [31] M. A. Golub, A. A. Friesem, and L. Eisen, “Bragg properties of E. Cambril, “Design and fabrication of binary slanted surface- efficient surface relief gratings in the resonance domain,” Optic relief gratings for a planar optical interconnection,” Appl. Opt., Commun., vol. 235, pp. 261–267, 2004. vol. 36, pp. 5717–5727, 10 August 1997. [32] M. G. Moharam, “Stable implementation of the rigorous coupled [14] J. Kimmel, T. Levola, P. Saarikko, and J. Bergquist, “A novel wave analysis for surface relief gratings: enhanced diffractive backlight concept for mobile displays,” J. SID, vol. 16, transmittance matric approach,” J. Opt. Soc. Am. A, vol. 12, no. 5, no. 2, 2008. pp. 1077–1086, 1995. 74 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets

[33] L. Alberto Estepa, C. Neipp, J. Frances,´ et al., “Corrected coupled- Proc. SPIE 10354, Nanoengineering: Fabrication, Properties, wave theory for non-slanted reflection gratings,” in Physical Optics, and Devices XIV, 2017, p. 1035408. Optics, Vol. 8171, D. G. Smith, F. Wyrowski, and A. Erdmann, Eds, [41] G. Basset, “Resonant screens focus on the optics of AR,” in Proc. Proc. of SPIE. SPIE 10676, Digital Optics for Immersive Displays, 2018, [34] http://www.kjinnovation.com/. p. 106760I. [35] https://meep.readthedocs.io/en/latest/. [42] P. Genevet, F. Capasso, F. Aieta, M. Khorasaninejad, and [36] T. Levola and P. Laakkonen, “Replicated slanted gratings with a R. Devlin, “Recent advances in planar optics: from plasmonic to high refractive index material for in and outcoupling of light,” dielectric metasurfaces,” Optica, vol. 4, no. 1, pp. 139–152, 2017. Opt. Express, vol. 15, pp. 2067–2074, 2007. [43] F. Capasso, “The future and promise of flat optics: a personal [37] https://www.lighttrans.com/applications/virtual-mixed- perspective,” Nanophotonics, vol. 7, no. 6, 2018, https://doi. reality/waveguide-huds.html. org/10.1515/nanoph-2018-0004. [38] M. W. Farn, “Binary gratings with increased efficiency,” Appl. [44] W. T. Chen, A. Y. Zhu, J. Sisler, et al., “Broadband Achromatic Opt., vol. 31, no. 22, pp. 4453–4458, 1992. metasurface-refractive optics,” Nano Lett., vol. 18, no. 12, [39] B. Kress and P. Meyrueis, Applied Digital Optics: From Micro- pp. 7801–7808, 2018. optics to Nanophotonics, 1st ed., John Wiley and Sons Publisher, 2007, -10: 0470022639. [40] G. Quaranta, G. Basset, O. J. F. Martin, and B. Gallinet, “Steering Supplementary Material: The online version of this article offers sup- and filtering white light with resonant waveguide gratings,” in plementary material (https://doi.org/10.1515/nanoph-2020-0410).