Nanophotonics 2021; 10(1): 41–74 Review Bernard C. Kress* and Ishan Chatterjee Waveguide combiners for mixed reality headsets: a nanophotonics design perspective https://doi.org/10.1515/nanoph-2020-0410 – vestibular comfort—providing stable and realistic virtual Received July 21, 2020; accepted September 16, 2020; overlays that spatially agree with the user’smotion published online October 7, 2020 – social comfort—allowing for true eye contact, in a so- cially acceptable form factor. Abstract: This paper is a review and analysis of the various implementation architectures of diffractive waveguide Immersion can be defined as the multisensory percep- combiners for augmented reality (AR), mixed reality (MR) tual experience (including audio, display, gestures, haptics) headsets, and smart glasses. Extended reality (XR) is that conveys to the user a sense of realism and envelopment. another acronym frequently used to refer to all variants In order to effectively address both comfort and immersion across the MR spectrum. Such devices have the potential to challenges through improved hardware architectures and revolutionize how we work, communicate, travel, learn, software developments, a deep understanding of the spe- teach, shop, and are entertained. Already, market analysts cific features and limitations of the human visual perception show very optimistic expectations on return on investment system is required. We emphasize the need for a human- in MR, for both enterprise and consumer applications. centric optical design process, which would allow for the Hardware architectures and technologies for AR and MR most comfortable headset design (wearable, visual, vestib- have made tremendous progress over the past five years, ular, and social comfort) without compromising the user’s fueled by recent investment hype in start-ups and acceler- sense of immersion (display, sensing, and interaction). ated mergers and acquisitions by larger corporations. In Matching the specifics of the display architecture to the order to meet such high market expectations, several chal- human visual perception system is key to bound the con- lenges must be addressed: first, cementing primary use straints of the hardware allowing for headset development cases for each specific market segment and, second, and mass production at reasonable costs, while providing a achieving greater MR performance out of increasingly size-, delightful experience to the end user. weight-, cost- and power-constrained hardware. One such crucial component is the optical combiner. Combiners are Keywords: augmented reality; diffractive optics; hologra- often considered as critical optical elements in MR headsets, phy; mixed reality; virtual reality; waveguide optics. as they are the direct window to both the digital content and the real world for the user’seyes. Two main pillars defining the MR experience are comfort Glossary of Terms, Abbreviations, and immersion. Comfort comes in various forms: and Acronyms – wearable comfort—reducing weight and size, pushing back the center of gravity, addressing thermal issues, and We provide this glossary for the reader after the abstract section so on as these acronyms are used extensively in this review paper. – visual comfort—providing accurate and natural fi 3-dimensional cues over a large eld of view and a high AR Augmented reality, adding virtual content into angular resolution field of view of reality, can include augmentations created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projec- tor systems, etc. *Corresponding author: Bernard C. Kress, Microsoft Corp. HoloLens MR Mixed reality, virtual objects situationalized in D Team, 1 Microsoft Way, Redmond, 98052, WA, USA, in your real space, often interactable E-mail: [email protected]. OST-MR Optical see-through mixed reality, displays are Ishan Chatterjee, Microsoft Corp. HoloLens Team, 1 Microsoft Way, transparent such that real world is viewable Redmond, 98052, WA, USA optically through the displays Open Access. © 2020 Bernard C. Kress and Ishan Chatterjee, published by De Gruyter. This work is licensed under the Creative Commons Attribution 4.0 International License. 42 B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets (continued) (continued) AR Augmented reality, adding virtual content into AR Augmented reality, adding virtual content into field of view of reality, can include augmentations field of view of reality, can include augmentations created by mixed reality headsets, handhelds, created by mixed reality headsets, handhelds, head up displays, smart glasses, camera-projec- head up displays, smart glasses, camera-projec- tor systems, etc. tor systems, etc. VST-MR Video see-through mixed reality, virtual reality mu-iLED, micro- Micro inorganic light-emitting diode, actively turned into the mixed reality with camera pass- iLED addressed inorganic LED array with emitter size through of the real-world into the VR environment < mu, NTE displays usually require a magnitude XR Extended reality, a generic term to capture all lower; can achieve high brightness and contrast, varieties across MR and AR but challenged in maintaining efficiency, multi- VR Virtual reality, blocks out reality and supplants color integration and backplane integration with virtual objects VCSEL Vertical-cavity surface-emitting laser, laser diode Immersion Sense of realism and development in delivered with lower divergence and current threshold than experience edge-emitting diodes IMU Inertial measurement unit consisting of at least an MEMS Microelectromechanical system accelerometer, and gyroscope, and often a LBS Laser beam scanning, type of display where a magnetometer modulated laser dot is raster scanned across GPU Graphical processing unit, parallel architecture display FOV via system of MEMS mirrors suited for graphics render and other matrix NTE Near-to-eye operations DOF degrees of freedom, in the context of tracking HMD Head-mounted display or helmet-mounted usually refers to the rotational axes (pitch, yaw, roll) display which can be resolved with only a calibrated IMU HUD Head up display, refers to see-through display DOF degrees of freedom, in the context of tracking that is often mounted externally (such as above a refers to the rotational and translational axes dashboard) allowing user to see both virtual CG Center of gravity, important ergonomic metric in content and subject of focus (e.g., the road ahead) head-worn devices simultaneously IPD Interpupillary distance SLM Spatial light modulator PPD Pixels per degree LCD Liquid-crystal display, display technology where HDR High dynamic range electro-sensitive liquid crystal pixels amplitude- FOV Field of view, provided as an angle modulate light from a global polarized backlight Eyebox The volume that the user’s pupil can sit in and in transmission view the entire virtual image field-of-view. The box LTPS-LCD Low-temperature polysilicon liquid-crystal may not be a rectangular prism, but is more often a display, higher resolution and faster switching frustrum speed than amorphous Si LCD Eye relief The distance the user’s corneal surface is from the IPS-LCD In-plane switching liquid-crystal display, liquid- display optic surface crystal structure twist in-plane of display, allow- UX/UI User experience/User interface, refers to the ing for higher viewing angles than twisted nematic design of the experience and applications (TN) LCDs, used in phones and monitors VAC Vergence accommodation conflict, refers to the HTPS-LCD High-temperature polysilicon (used for silicon mismatch experienced when a stereoscopic dis- backplanes) play’s image focal plane does not match the ste- AMOLED Active-matrix organic light-emitting diode, reo disparity of the virtual image. increased contrast at the cost of lifetime and high Pupil swim The experience of warp and shift of virtual objects brightness, each pixel is its own organic electro- as the user’s pupils rove around the eyebox luminescent emitter, used commonly in caused by distortion in the projected image across cellphones the eyebox mu-OLED, micro- Micro-organic light-emitting diode, display with Hard-edge The ability for real-world objects to mask virtual OLED emitter size less than μm, used in camera occlusion content according to the depth the virtual image is electronic view finders in the world DLP Digital light processing, Texas Instrument’s Hologram Recording of a interference pattern between a colloquially genericized trademark for DMD (digi- reference and a wavefront off a D scene… but in tal micromirror device), an array of bi-stable AR/VR forums, a virtual stereo image that appears reflective micromirrors, commonly used in pro- to be positioned in space like a true hologram jection systems for highly efficiency SLM ET Eye tracking LCoS Liquid crystal on silicon, microdisplay with a HeT Head tracking switchable liquid-crystal matrix on reflective sili- TIR Total internal reflection (principle of how light con backplane propagates when trapped in a light guide) B.C. Kress and I. Chatterjee: Waveguide combiners for mixed reality headsets 43 (continued) displays. Although such display technologies were well ahead of their time [5–7], the lack of consumer-grade in- AR Augmented reality, adding virtual content into ertial measurement unit (IMU) sensors, low-power field of view of reality, can include augmentations 3-dimensional (3D)-rendering graphical processing units, created by mixed reality headsets, handhelds, head up displays, smart glasses,