D6.3: 3D Media Tools – Report – Version B Project ref. no. FP7-ICT-610691 Project acronym BRIDGET Start date of project (duration) 2013-11-01 (36 months) Document due Date: 2016-07-31 Actual date of delivery 2016-07-30 Ingo Feldmann Leader of this document Reply to Document status Public deliverable FP7-ICT-610691 D6.3: 3D Media Tools – Report – Version B Deliverable Identification Sheet Project ref. no. FP7-ICT-610691 Project acronym BRIDGET Project full title BRIDging the Gap for Enhanced broadcasT Document name D6.3_final.docx Security (distribution level) PU Contractual date of delivery (none) Actual date of delivery 2016-07-30 Document number Type Public deliverable Status & version v0.5 Number of pages 70 WP / Task responsible WP6 Other contributors Nicola Piotto, Daniel Berjón Díez, Rafael Pagés Scasso, Sergio García Lobo, Francisco Morán Bur- gos, Giovanni Cordara, Milos Markovic, Sascha Ebel, Wolfgang Waizenegger, Sylvain Renault, Thomas Ebner Author(s) Ingo Feldmann Project Officer Alberto Rabbachin Abstract This document summarizes the research achieve- ments of WP6 for the tasks of off-line extraction of 3D audio-visual content (T6.1), 3D Media Coding (T6.2) as well as the free viewpoint scene recon- struction and rendering (T6.3). Task T6.4 is not mentioned because it focuses on standardisation (in the context of WP6) and will have its results reported in D9.3 at the end of the project. The document extends the first technical deliverable D6.1. Keywords 3D tools, media analysis, 3D reconstruction Sent to peer reviewer 2016-07-08 Peer review completed 2016-07-22 Circulated to partners 2016-07-22 Read by partners 2016-07-28 D6.3: 3D Media Tools – Report – Version B 2/70 FP7-ICT-610691 D6.3: 3D Media Tools – Report – Version B Mgt. Board approval 2016-07-29 Version Date Reason of change 0.1 2016-05-24 Ingo Feldmann – Initial draft 0.2 2016-06-15 Ingo Feldmann – First fully merged version with input from all part- ners 0.3 2016-07-08 QAT ready version 0.4 2016-07-19 Included comments from WP6 internal review 0.5 2016-07-28 Included QAT comments, final version D6.3: 3D Media Tools – Report – Version B 3/70 FP7-ICT-610691 D6.3: 3D Media Tools – Report – Version B Table of Contents 1 Executive summary ............................................................................................................................................... 6 2 Introduction ............................................................................................................................................................. 6 2.1 WP6 Objective ................................................................................................................................................................ 6 2.2 General algorithmic workflow for 3D video processing ............................................................................... 6 2.3 Structure of the document ........................................................................................................................................ 7 3 3D Scene Off-line Modelling ............................................................................................................................... 7 3.1 Generation of detailed point clouds using alternative feature descriptors .......................................... 7 3.1.1 Prioritisation of pairs of images .................................................................................................................... 8 3.1.2 Reduction of per-pair computational cost ................................................................................................ 9 3.2 3D mesh processing .................................................................................................................................................. 10 3.2.1 Mesh cleaning..................................................................................................................................................... 10 3.2.2 Mesh smoothing and decimation ............................................................................................................... 12 3.3 Seamless, static multi-texturing ........................................................................................................................... 13 3.3.1 Colour blending ................................................................................................................................................. 13 3.3.2 Occlusion detection ......................................................................................................................................... 13 3.4 Generation of textured-splat models ................................................................................................................. 15 3.5 Definition of a hybrid SPLAts + meSH (SPLASH) model ............................................................................ 16 3.6 Dense depth based 3D surface reconstruction .............................................................................................. 25 3.6.1 Automatic camera pre-selection ................................................................................................................ 25 3.6.2 Enhanced depth estimation workflow .................................................................................................... 27 3.7 3D Audio modeling – recorded audio bridget approach ........................................................................... 31 4 3D Media Coding ................................................................................................................................................... 32 4.1 Coding scheme proposal for hybrid SPLASH models .................................................................................. 32 4.2 Framework for point cloud compression and visualization .................................................................... 34 4.2.1 State-of-the-Art ................................................................................................................................................. 35 4.2.2 Compression methods for BRIDGET ........................................................................................................ 37 4.2.3 Description of the point cloud compression process ........................................................................ 37 4.2.4 Decompression .................................................................................................................................................. 41 4.2.5 Compression results ........................................................................................................................................ 42 4.2.6 Summary of compression using quantization and octrees ............................................................. 46 5 3D Scene On-line Reconstruction ................................................................................................................... 47 5.1 Video-to-Point-Cloud Framework ....................................................................................................................... 47 5.2 Stereo SLAM framework ......................................................................................................................................... 47 5.2.1 Map representation: points and keyframes .......................................................................................... 48 5.2.2 Front-end: tracking .......................................................................................................................................... 49 5.2.3 Back-end: mapping and loop closure ....................................................................................................... 52 5.2.4 Depth map fusion ............................................................................................................................................. 53 5.2.5 Considerations ................................................................................................................................................... 56 5.3 Rendering of textured splat models ................................................................................................................... 56 5.4 Surface Patch Rendering ......................................................................................................................................... 60 5.5 Mesh post-processing and simplification......................................................................................................... 61 5.6 Web based rendering of compressed point clouds ...................................................................................... 65 D6.3: 3D Media Tools – Report – Version B 4/70 FP7-ICT-610691 D6.3: 3D Media Tools – Report – Version B 5.7 3D Audio reconstruction – recorded audio bridget approach ................................................................ 67 6 Conclusion .............................................................................................................................................................. 67 7 References .............................................................................................................................................................. 68 D6.3: 3D Media Tools – Report – Version B 5/70 FP7-ICT-610691 D6.3: 3D Media Tools – Report – Version B 1 Executive summary This document summarizes the research achievements of WP6 for the tasks of off-line extraction of 3D audio-visual content (T6.1), 3D