EUROGRAPHICS 2014/ N. Holzschuch and K. Myszkowski Tutorial 3D video: from capture to diffusion Yannick Rémion and Laurent Lucas and Céline Loscosy University of Reims Champagne-Ardenne, France Figure 1: Capture, diffusion, and processing of 3D video content Abstract While 3D vision has existed for many years, the use of 3D cameras and video-based modeling by the film industry has induced an explosion of interest for 3D acquisition technology, 3D content and 3D displays. As such, 3D video has become one of the new technology trends of this century. This tutorial aims at introducing theoretical, technological and practical concepts associated to multiview systems. It covers acquisition, manipulation, and rendering. Stepping away from traditional 3D vision, the authors, all currently involved in these areas, provide the necessary elements for understanding the underlying computer-based science of these technologies. Categories and Subject Descriptors (according to ACM CCS): I.4.8 [IMAGE PROCESSING AND COMPUTER VISION]: Scene analysis—Stereo I.3.1 [COMPUTER GRAPHICS]: Hardware Architecture—Three-dimensional displays I.4.1 [IMAGE PROCESSING AND COMPUTER VISION]: Digitization and Image Capture — 1. Course summary and objectives derstanding the underlying computer-based science of these technologies. While 3D vision has existed for many years, the use of 3D cameras and video-based modeling by the film industry has Several types of camera systems are considered (mul- induced an explosion of interest for 3D acquisition technol- tiscopic or multiview) which lead to different acquisition, ogy, 3D content and 3D displays. As such, 3D video has modeling and storage-rendering solutions. Equipment will become one of the new technology trends of this century. be used to illustrate the concepts: two multiview acquisition This tutorial aims at introducing theoretical, technological systems developed in the University of Reims Champagne- and practical concepts associated to multiview systems. It Ardenne and an autosteroscopic display. Live demos of this covers acquisition, manipulation, and rendering. Stepping equipment will be used to illustrate the course content. In away from traditional 3D vision, the authors, all currently in- Part I will introduce the necessary technical and theoretical volved in these areas, provide the necessary elements for un- background associated to multiview systems and introduce multiview technology, with an insight on the associated con- straints. Part II will indicate how to use this technology for y [email protected], [email protected], enhanced content, bringing it to HDR acquisition, and 3D [email protected] modeling, and how to deliver multiview video flows. Yannick Rémion & Laurent Lucas & Céline Loscos / 3D video: from capture to diffusion Course notes will include the presented slides, with asso- Laurent Lucas (LL) He received Ph.D. in computer sci- ciated text summaries, and a list of bibliographic pointers. ence at the URCA in 1995. He currently leads the SIC re- The tutorial will also refer to a recent book [LRL13a] edited search group and is also in charge of the virtual reality plat- by the authors. This is under copyright agreement with the form of the URCA. His research interests include visualiza- publisher, but summary of the relevant chapters will be pre- tion and co-operation between image processing and com- pared for the audience as tutorial notes. puter graphics particularly in 3DTV and theirs applications. He has co-authored more than 50 peer-reviewed interna- Typical keywords are: 3D video, multiview acquisition, tional publications in these areas. His joint-work on 3DTV multiscopy, 3D cameras, video-based modelling, free view- formed the basis of 3DTV Solution’s technology. His cur- point video, high-dynamic range imaging, 3DTV, 3D dis- rent research focuses on 3D vision and he is in charge of the plays, 3D geometric reconstruction, 3D video transmission RECOVER3D project. and coding. Yannick Rémion (YR) He received his engineering de- 2. Authors gree in Paris from âAIJEcole˘ PolytechniqueâA˘ I˙ in computer science (1984) and his Ph.D. in computer science at the 2.1. Tutorial speakers’ details ENST (1988). His research interests include dynamic ani- Name: Laurent Lucas mation, simulation and co-operation between image process- Institution: CReSTIC-SIC (EA3804), Université de Reims ing and computer graphics, 3D vision on which he has co- Champagne-Ardenne, France authored more than 30 peer-reviewed international publica- Email address: [email protected] tions. His joint-work on 3DTV formed the basis of 3DTV URL: http://crestic.univ-reims.fr/membre/10-laurent-lucas Solution’s technology and he has leads the CamRelief ANR Name: Yannick Rémion Project. Institution: CReSTIC-SIC (EA3804), Université de Reims Céline Loscos (CL) She received her Ph.D. in computer Champagne-Ardenne, France science at the UJF (Grenoble, France) in 1999. She worked Email address: [email protected] at University College London, UK, as a Lecturer until 2007 URL: http://crestic.univ-reims.fr/membre/9-yannick- and at Universty of Girona, Spain until 2010. She has been remion involved into several EU projects and was coordinator of the Name: Céline Loscos CREATE project (FP5 IST). She was PI of the MEC Explora Institution: CReSTIC-SIC (EA3804), Université de Reims Spanish project and currently leads the HDR capture work- Champagne-Ardenne, France ing group of the HDRi COST action. She is active in peer Email address: [email protected] reviewing and has co-authored more than 40 peer-reviewed URL: http://crestic.univ-reims.fr/membre/1384-celine- international publications on illumination simulation, com- loscos putational photography, and high-dynamic range imaging. 2.2. Brief resume of the presenters indicating their 3. Tutorial length background in the area the tutorial addresses The tutorial is set to be a half day tutorial - 2x90 minutes. It is The tutorial speakers are composed of three professors of designed to have each topic addressed in four main sections: the CReSTIC laboratory, in the signal, image and knowledge management (SIC) team addressing 3D vision, HDR im- 1. Introduction and multiview systems agery and CG activities. They are part of the Computer Sci- 2. Multiscopy methods ence Department of the University Institute of Technology of 3. Extensions and applications Reims and research in the field of 3D vision and computer 4. Restitution, coding, display graphics for the purpose of furthering fundamental knowl- edge of 3DTV and HDR imaging, pursuing advanced en- While each topic could deserve to go in further details, the gineering applications in broadcasting, biomedical, and cul- presenters will try to make a good trade off between the high tural heritage. Laurent Lucas and Yannick Rémion are mem- level understanding and overview of the topic, and the low bers of a joint research laboratory with OPEXMedia com- level details to understand better the underlying theory and pany (NeoTelecom subsidiary and 3DTV Solutions owner), technology. and joint owners of several patents relating to the produc- tion and processing of 3D images. Recent projects related to 4. Detailed outline of the tutorial this proposal were funded by the French National Research Agency (ANR CamRelief 2008-2010, FSN RECOVER3D The tutorial notes follow the content of a recent book edited 2012-2014), the Ministry of education in Spain (MEC Ex- by the authors on 3D Video [LRL13a]. plora on HDR imaging 2009-2011) and the European Com- Part 1: 1h30 - Introduction, definitions and fundamen- mission (COST HDRi 2012-2015). tals, multiview systems, multiscopy Yannick Rémion & Laurent Lucas & Céline Loscos / 3D video: from capture to diffusion • Introduction of the authors, of the course objectives, of the ◦ Disparity and depth course outline (All authors) - 5 minutes • Discussion, Questions • Fundamentals [LRL13b] Part II: 1h30 - Extensions, applications, encoding and – A short history display ◦ 3D and Binocular vision • 3D HDR video acquisition [BVLN13] ◦ Multiview systems – HDR and 3D acquisition – Stereopsis and 3D physiological aspects – 3D computer vision ◦ Subspace 1D: HDR images ◦ Subspace 2D: HDR videos • Multiview acquisition systems [DPR13] ◦ Subspace 2D: 3DHDR images – What is a multiview acquisition system? ◦ Extension to the whole space: 3DHDR videos – Binocular systems • Multiview reconstruction [BIS13] ◦ Technical description, Main usages – Problematic – Lateral or directional multiview systems – Visual hull-based reconstruction ◦ Technical description, main usages ◦ Methods to extract visual hulls ◦ Demo ◦ Reconstruction methods ◦ Improving volume reconstruction: Voxel Coloring – Surrounding or omni-directional systems and Space Carving ◦ Technical description, main usages – Temporal structure of reconstructions – Comparison of the different types of systems ◦ Extraction of a generic skeleton • Acquisition: Practical, optical and physical considera- ◦ Computation of motion fields tions: Shooting and viewing for 3D TV [PLR13] • Encoding multiview videos [BVCD13] – Introduction – Introduction – 3D viewing geometry – Compression of stereoscopic videos ◦ Description ◦ 3D formats ◦ Setting the parametric model Frame compatible – 3D shooting geometry Mixed Resolution Stereo ◦ Existing types of geometry 2D-plus-depth ◦ Setting the parametric model ◦ Associated coding techniques – Geometrical impact of the 3D workflow Simulcast ◦ Rendered-to-shot space mapping MPEG-C and