Immersify Project • NHK, Intel, SES, Thank You for Your Attention
Total Page:16
File Type:pdf, Size:1020Kb
Advances in Video Codecs for Immersive Media. Technical and business experiences from Spin Digital Mauricio Alvarez Mesa, PhD ExpoIngenieria 2018 Medellín, Colombia October 17th, 2018 Agenda 1. Next-generation Video Formats 2. Spin Digital Video Processing Solutions 3. Case Studies and Projects 2 Immersive Media Source: Cambridge English Dictionary Next Generation Video VR 360°Video SDR HDR FHD 4K 8K Dynamic Resolution Range Next Generation Video Color Frame UHD Gamut rate 30 fps Bit HD Depth 60 fps 8 bit 10 bit 120 fps 4 Ultra-High Definition • Higher pixel density: more details • Wider field of view: more visible content, more immersive • 8K requires: 4X data rate compared to 4K, and 16X to FHD 5 High Dynamic Range • High contrast: clear darker and brighter image areas • Improve sensation of sharpness • Requires at least 10-bit, and next gen 12-bit 6 Wide Color Gamut • Extends the amount of colors than can be represented • Requires at least 10-bit 7 High Frame Rate 100 - 120 fps • Provides smoother motion representation M. Sugawara and K, Masaoka. UHDTV Image Format for Better Visual Experience. Proc IEEE. • More details in high motion regions Dec 2012 • Requires 2X data rate for uncompressed video • 20 to 30% more data for compressed video 8 Virtual Reality and 360° Video • More immersive and interactive experience • 360° video requires 3X-4X the resolution of the FoV (~100°) • Stereoscopic 360 ° video requires 2X the data rate 9 Quality Requirements for Next Gen VR 4K 8K 12K 16K Per eye resolution 1Kx1K 2Kx2K 3Kx3K 4Kx4K 360° 2D total 4Kx2K 8Kx4K 12Kx6K 16Kx8K resolution 360° 3D total 4Kx4K 8Kx8K 12Kx12K 16Kx16K resolution 8K per eye is the ideal resolution for full 16K and 240 fps are needed for “true VR” immersive virtual reality Palmer Lucky, co-founder of Oculus Rift Raja Koduri, head of GPU at AMD Data Requirements for Next-gen Video Uncompressed Bitrate [Gbit/s] 800 700 600 500 400 300 200 100 0 UHD-4Kp60 8Kp60 8Kp120 16Kp60 16Kp240 unpacked format packed format Video Codec Evolution 12 High Efficiency Video Coding HEVC Compressed Bitrates - HEVC 800 700 600 500 400 300 200 100 0 4Kp60 8Kp60 8Kp120 16Kp60 16Kp240 Agenda 1. Next-generation Video Formats 2. Spin Digital Video Processing Solutions 3. Case Studies and Projects 14 Spin Digital Video Technologies GmbH Specialists on video codecs, software engineering and Spin-off of the Technical computer architectures University of Berlin (November 2015) Based in Germany Innovative B2B company with decades of accumulative research experience In-house developed software IP & products 15 Spin Digital Products High quality and high compression efficiency HEVC encoder Full-featured media player for 4K, 8K, and beyond High performance HEVC/H.265 decoder, video filtering, and video renderer SDK 16 High Performance Video Decoding: SIMD SIMD optimizations: - SSE (64-bit) - AVX2 (128-bit - AVX512 (512-bit) C. C. Chi, M. Alvarez-Mesa, B. Bross, B. Juurlink and T. Schierl, "SIMD Acceleration for HEVC Decoding," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 5, pp. 841-855, May 2015 17 High Performance Video Decoding: Multithreading • Advanced Multihreading 8K Decoding Performance • More than 50 CPU cores 300 • NUMA architectures 250 • Experimental set-up 200 • CPU: Xeon Platinum 8168, 150 • 2 x 24 cores (96 threads) 100 Frames/Second • 2.7 GHz 50 • GPU: Nvidia Quadro M4000 0 • OS: Windows 10, 64-bit 4:2:0 10b 4:2:2 10b 4:4:4 10b 4:2:0 12b 4:2:2 12b 4:4:4 12b C. C. Chi et al., "Parallel Scalability and Efficiency of HEVC Parallelization Approaches," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1827-1838, Dec. 2012. 18 High-end Video Processing on PC Platforms High Performance Video Decoding: Memory Optimizations • Memory optimizations Performance for 8K Decoding 350.0 • Efficient pixel formats 300.0 • Bitplanar fr 10 and 12-bit 250.0 • Texture compression 200.0 • BC4 compression FPS 150.0 100.0 • Experimental results 50.0 0.0 • CPU: Intel Core i9-7960x 420 10b 422 10b 444 10b • 16-cores Planar Bit BC4 • 2.8 GHz • GPU: Alvarez-Mesa M., Ching Chi C., HEVC Software Media Player for Ultra-High Quality Video: 8K and Beyond, The 25th International Display Workshops. • AMD Radeon WX 7100 December 12-14, 2018. Nagoya. Japan (paper accepted) 20 Agenda 1. Next-generation Video Formats 2. Spin Digital Video Processing Solutions 3. Case Studies and Projects 21 8K Media Player for Home Theatre NHK Engineering Systems (NES) NHK Open House 2016, 2017, 2018 PC based 8K Media Player for Home Theater applications 8K HEVC video + 22.2 audio 22 8K Video Wall • Demonstrations at NAB and IBC 2017 with Intel Corporation • 8K PC-based media player using Intel Xeon Processors 23 Live 8K Streaming via Satellite Real-time 8K streaming via satellite Development project by SES, Sharp, and Spin Digital Demonstration performed at SES Industry Days 2018 (Luxembourg) 24 EU Projects: Immersify Audiovisual Technologies for Next Generation Immersive Media • Duration: 30 months: Oct 2017 – March 2020 • Funding: Horizon 2020 Innovation Action • Budget: 2.5 million € 25 Project Partners • Poznan Supercomputing and Networking Center - PSNC (Poznan, PL) • Spin Digital Video Technologies GmbH (Berlin, DE) • Ars Electronica Futurelab (Linz, AT) • Marché du Film – Festival de Cannes (Cannes, FR) • Visualization Center C (Norrköping, SE) Project Objectives • Video codecs: improve quality using advanced compression technology • Multi-Display: enable immersive media for multiple display environments • Interactive: provide tools for personalized and interactive non-linear storytelling • Content and Tools: promote immersive media in the creative industries 16K Media Player • Resolution: 16Kx8K • Frame rate: 60 Hz • HEVC 400 Mbit/s • CPU: Intel Xeon Scalable 2x24 cores • Demo presented at ISE 2018, NAB 2018, IBC 2019 28 High Resolution 360° Video Pandarama by ARRI • 5 ALEXA Mini (360° x 60°) • Fraunhofer OmniCam rig • 12k x 8k resolution 24 fps • Spin Player full resolution playback with 360° player Conclusions • Highly immersive media is emerging • Requires next generation video: 8K, 16K, high fps, high bidepth, 3D, 360°, .. • Massive data rates beyond UHD TV formats • Spin Digital has developed a high-performance HEVC implementation • Software optimizations for state-of-the-art PC architectures • Up to 16K video decoding and playback • Spin Digital working with partners on tools for immersive media • Immersify Project • NHK, Intel, SES, Thank you for your attention HIGH PERFORMANCE VIDEO CODECS Backup Slides HEVC Encoder Relative video file size for same objective quality and encoding speed Higher compression & quality than leading 8Kp60 open source encoders 100 Support for 4K, 8K and 16K 90 Fast offline compression 80 Ready for HDR and WCG 4:2:0, 10-bit 4:2:2, 10-bit • Ready for 8K and beyond File Size[%] 70 66 66 4:4:4, 10-bit • Significantly better compression and quality than competing encoders 60 • Enables WCG and HDR with up to 12-bit video 52 • Support for HDR metadata 50 • Preserves color resolution with 4:2:2, 4:4:4, x265 2.5 (HEVC) Spin Enc 1.6.2 and RGB formats (HEVC) • Support for 360° video projection formats 33 Rate-Distortion Performance of an Optimized HEVC Encoder Helicopter, 7680x4320, 60 Hz, 4:2:0, 41.0 40.0Helicopter, 7680x4320, 60 Hz, 4:4:4, 10-bit 10-bit 40.5 39.5 40.0 39.5 39.0 PSNR [dB] - 39.0 PSNR [dB] - YUV 38.5 38.5 Spin Digital Spin Digital YUV (H.265) (H.265) 38.0 38.0 50 100 150 200 250 300 50 100 150 200 250 300 Bitrate [Mbits/s] Bitrate [Mbits/s] 38 % bitrate reduction 52 % bitrate reduction 34 HEVC Media Player 120 Media player for 4K, 8K, and 16K 100 100 Based on HEVC software solution 84 80 64 60 54 54 40 • Pure software solution • Ready for up to 16K playback Frames per second (fps) 20 • Professional video formats: 4:2:2 and 4:4:4 • 10-bit, HDR, and WCG support 0 • Next generation audio: 22.2 surround sound VLC 3.0.0* MPC-BE 1.5.2* Spin Player 1.2 • 360° video playback CPU: Intel Core i9-7960X @ 2.80 GHz (16 cores) 8Kp60, 4:2:0, 10-bit, 120 GPU: NVIDIA GeForce GTX 1060 6GB * GPU-based HEVC hardware decoder Mbps • Easy-to-use graphical user interface OS: Windows 10, 64-bit 8Kp60, 4:2:2, 10-bit, 160 Video: Unigine Superpostion Benchmark 8K Mbps • SDI and GPU rendering 35 HEVC Decoder & Renderer SDK Maximum achievable bitrate for HEVC decoding and video rendering using state-of-the-art PC technologies Ultra high performance HEVC decoder 1,200 1,140 Next generation video rendering 1,000 928 Real-time 16Kp60 playback 800 4:2:0, 10-bit • Pure software solution 600 573 4:2:2, 10-bit • Powerful and efficient API for Windows and 4:4:4, 10-bit Linux 375 400 • High quality video formats: 4:2:2, 4:4:4, up to 275 Maximum Maximum Bitrate [Mbps] 12-bit 200 • Real-time HEVC/H.265 decoding up to 16K • CPU: Intel Xeon E5-2696 v4 10-bit rendering, WCG and HDR support (2x 22 cores, 2.2 GHz) 0 GPU: NVIDIA Quadro M5000 • Flexible I/O with GPU and SDI devices 60 frames/s 120 frames/s • Video filtering library 36 Immersify: Ars Electronica Deep Space 8K • 8K HEVC for immersive media players • 8Kx8K projection surface • Interactive applications Source: AEC 37.