3D VISUAL COMMUNICATIONS 3D VISUAL COMMUNICATIONS Guan-Ming Su Dolby Labs, California, USA Yu-Chi Lai National Taiwan University of Science and Technology, Taiwan Andres Kwasinski Rochester Institute of Technology, New York, USA Haohong Wang TCL Research America, California, USA A John Wiley & Sons, Ltd., Publication This edition first published 2013 2013 John Wiley and Sons Ltd Registered office John Wiley & Sons Ltd, The Atrium, Southern Gate, Chichester, West Sussex, PO19 8SQ, United Kingdom For details of our global editorial offices, for customer services and for information about how to apply for permission to reuse the copyright material in this book please see our website at www.wiley.com. The right of the author to be identified as the author of this work has been asserted in accordance with the Copyright, Designs and Patents Act 1988. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise, except as permitted by the UK Copyright, Designs and Patents Act 1988, without the prior permission of the publisher. Wiley also publishes its books in a variety of electronic formats. Some content that appears in print may not be available in electronic books. Designations used by companies to distinguish their products are often claimed as trademarks. All brand names and product names used in this book are trade names, service marks, trademarks or registered trademarks of their respective owners. The publisher is not associated with any product or vendor mentioned in this book. This publication is designed to provide accurate and authoritative information in regard to the subject matter covered. It is sold on the understanding that the publisher is not engaged in rendering professional services. If professional advice or other expert assistance is required, the services of a competent professional should be sought. Library of Congress Cataloging-in-Publication Data Su, Guan-Ming. 3D visual communications / Guan-Ming Su, Yu-Chi Lai, Andres Kwasinski, Haohong Wang. pages cm Includes bibliographical references and index. ISBN 978-1-119-96070-6 (cloth) 1. Multimedia communications. 2. Three-dimensional display systems. I. Lai, Yu-Chi. II. Kwasinski, Andres. III. Wang, Haohong, 1973- IV. Title. V. Title: Three dimensional visual communications. TK5105.15.S83 2013 006.7–dc23 2012031377 A catalogue record for this book is available from the British Library. ISBN: 978-1-119-96070-6 Typeset in 10/12pt Times by Laserwords Private Limited, Chennai, India Contents Preface ix About the Authors xiii 1 Introduction 1 1.1 Why 3D Communications? 1 1.2 End-to-End 3D Visual Ecosystem 3 1.2.1 3D Modeling and Representation 5 1.2.2 3D Content Creation 6 1.2.3 3D Video Compression 7 1.2.4 3D Content Delivery 8 1.2.5 3D Display 9 1.2.6 3D QoE 9 1.3 3D Visual Communications 10 1.4 Challenges and Opportunities 11 References 15 2 3D Graphics and Rendering 17 2.1 3DTV Content Processing Procedure 19 2.2 3D Scene Representation with Explicit Geometry – Geometry Based Representation 22 2.2.1 Surface Based Representation 23 2.2.2 Point Based Representation 37 2.2.3 Point Based Construction 38 2.2.4 Point Based Compression and Encoding for Transmission 38 2.2.5 Point Based Rendering: Splatting 39 2.2.6 Volumetric Representation 40 2.2.7 Volumetric Construction 40 2.2.8 Volumetric Compression and Encoding for Transmission 41 2.2.9 Volumetric Rendering 42 2.3 3D Scene Representation without Geometry – Image-Based Representation 43 2.3.1 Plenoptic Function 43 2.3.2 Single Texture Representation 46 2.3.3 Multiple Texture Representation 48 2.3.4 Image Based Animation 51 vi Contents 2.4 3D Scene Representation with Implicit Geometry – Depth-Image-Based Representation 51 2.4.1 History of Depth-Image-Based Representation 52 2.4.2 Fundamental Concept Depth-Image-Based Representation 53 2.4.3 Depth Construction 56 2.4.4 Depth-Image-Based Animation 57 References 57 3 3D Display Systems 63 3.1 Depth Cues and Applications to 3D Display 63 3.1.1 Monocular Depth Cues 63 3.1.2 Binocular Depth Cues 64 3.2 Stereoscopic Display 65 3.2.1 Wavelength Division (Color) Multiplexing 65 3.2.2 Polarization Multiplexing 69 3.2.3 Time Multiplexing 69 3.3 Autostereoscopic Display 71 3.3.1 Occlusion-Based Approach 71 3.3.2 Refraction-Based Approach 75 3.4 Multi-View System 78 3.4.1 Head Tracking Enabled Multi-View Display 79 3.4.2 Automultiscopic 79 3.5 Recent Advances in Hologram System Study 83 References 84 4 3D Content Creation 85 4.1 3D Scene Modeling and Creation 85 4.1.1 Geometry-Based Modeling 86 4.1.2 Image-Based Modeling 86 4.1.3 Hybrid Approaches 87 4.2 3D Content Capturing 87 4.2.1 Stereo Camera 87 4.2.2 Depth Camera 88 4.2.3 Multi-View Camera 88 4.2.4 3D Capturing with Monoscopic Camera 89 4.3 2D-to-3D Video Conversion 101 4.3.1 Automatic 2D-to-3D Conversion 103 4.3.2 Interactive 2D-to-3D Conversion 111 4.3.3 Showcase of 3D Conversion System Design 112 4.4 3D Multi-View Generation 125 References 126 5 3D Video Coding and Standards 129 5.1 Fundamentals of Video Coding 129 5.2 Two-View Stereo Video Coding 142 5.2.1 Individual View Coding 142 5.2.2 Inter-View Prediction Stereo Video Coding 143 Contents vii 5.3 Frame-Compatible Stereo Coding 144 5.3.1 Half-Resolution Frame-Compatible Stereo Coding 144 5.3.2 Full-Resolution Frame-Compatible Layer Approach 146 5.4 Video Plus Depth Coding 148 5.5 Multiple View Coding 156 5.6 Multi-View Video Plus Depth (MVD) Video 160 5.7 Layered Depth Video (LDV) 163 5.8 MPEG-4 BIFS and AFX 165 5.9 Free-View Point Video 166 References 167 6 Communication Networks 171 6.1 IP Networks 171 6.1.1 Packet Networks 171 6.1.2 Layered Network Protocols Architecture 172 6.2 Wireless Communications 174 6.2.1 Modulation 175 6.2.2 The Wireless Channel 177 6.2.3 Adaptive Modulation and Coding 191 6.3 Wireless Networking 193 6.4 4G Standards and Systems 193 6.4.1 Evolved Universal Terrestrial Radio Access Network (E-UTRAN) 195 6.4.2 Evolved Packet Core (EPC) 200 6.4.3 Long Term Evolution-Advance (LTE-A) 201 6.4.4 IEEE 802.16 – WiMAX 202 References 203 7 Quality of Experience 205 7.1 3D Artifacts 205 7.1.1 Fundamentals of 3D Human Visual System 205 7.1.2 Coordinate Transform for Camera and Display System 206 7.1.3 Keystone Distortion 211 7.1.4 Depth-Plane Curvature 212 7.1.5 Shear Distortion 212 7.1.6 Puppet-Theater Effect 213 7.1.7 Cardboard Effect 215 7.1.8 Asymmetries in Stereo Camera Rig 216 7.1.9 Crosstalk 217 7.1.10 Picket-Fence Effect and Lattice Artifacts 217 7.1.11 Hybrid DCT Lossy Compression Artifact 218 7.1.12 Depth Map Bleeding and Depth Ringing 219 7.1.13 Artifacts Introduced by Unreliable Communication Networks 219 7.1.14 Artifacts from New View Synthesis 219 7.1.15 Summary of 3D Artifacts 220 7.2 QoE Measurement 220 7.2.1 Subjective Evaluations 222 7.2.2 2D Image and Video QoE Measurement 226 viii Contents 7.2.3 3D Video HVS Based QoE Measurement 235 7.2.4 Postscript on Quality of Assessment 246 7.3 QoE Oriented System Design 247 7.3.1 Focus Cues and Perceptual Distortions 247 7.3.2 Visual Fatigue 249 References 250 8 3D Video over Networks 259 8.1 Transmission-Induced Error 259 8.2 Error Resilience 267 8.3 Error Concealment 270 8.4 Unequal Error Protection 275 8.5 Multiple Description Coding 279 8.6 Cross-Layer Design 282 References 286 9 3D Applications 289 9.1 Glass-Less Two-View Systems 289 9.1.1 Spatially Multiplexed Systems 290 9.1.2 Temporally Multiplexed Systems 290 9.2 3D Capture and Display Systems 291 9.3 Two-View Gaming Systems 294 9.4 3D Mobile 298 9.4.1 HTC EVO 3D 298 9.4.2 Mobile 3D Perception 299 9.5 Augmented Reality 302 9.5.1 Medical Visualization 304 9.5.2 Mobile Phone Applications 306 References 309 10 Advanced 3D Video Streaming Applications 313 10.1 Rate Control in Adaptive Streaming 313 10.1.1 Fundamentals of Rate Control 313 10.1.2 Two-View Stereo Video Streaming 318 10.1.3 MVC Streaming 318 10.1.4 MVD Streaming 319 10.2 Multi-View Video View Switching 321 10.3 Peer-to-Peer 3D Video Streaming 325 10.4 3D Video Broadcasting 328 10.5 3D Video over 4G Networks 329 References 331 Index 335 Preface As the Avatar 3D movie experience swept the world in 2010, 3D visual content has become the most eye-catching spot in the consumer electronics products. This 3D visual wave has spread to 3DTV, Blu-ray, PC, mobile, and gaming industries, as the 3D visual system pro- vides sufficient depth cues for end users to acquire better understanding of the geometric structure of the captured scenes, and nonverbal signals and cues in visual conversation. In addition, 3D visual systems enable observers to recognize the physical layout and location for each object with immersive viewing experiences and natural user interaction, which also makes it an important topic for both academic and industrial researchers.