Advances in Video Codecs for Immersive Media. Technical and business experiences from Spin Digital
Mauricio Alvarez Mesa, PhD
ExpoIngenieria 2018 Medellín, Colombia October 17th, 2018 Agenda
1. Next-generation Video Formats
2. Spin Digital Video Processing Solutions
3. Case Studies and Projects
2 Immersive Media
Source: Cambridge English Dictionary Next Generation Video
VR 360°Video
SDR HDR FHD 4K 8K
Dynamic Resolution Range Next Generation Video Color Frame UHD Gamut rate 30 fps Bit HD Depth 60 fps 8 bit
10 bit 120 fps
4 Ultra-High Definition
• Higher pixel density: more details • Wider field of view: more visible content, more immersive • 8K requires: 4X data rate compared to 4K, and 16X to FHD
5 High Dynamic Range
• High contrast: clear darker and brighter image areas • Improve sensation of sharpness • Requires at least 10-bit, and next gen 12-bit
6 Wide Color Gamut
• Extends the amount of colors than can be represented • Requires at least 10-bit
7 High Frame Rate 100 - 120 fps
• Provides smoother motion representation M. Sugawara and K, Masaoka. UHDTV Image Format for Better Visual Experience. Proc IEEE. • More details in high motion regions Dec 2012 • Requires 2X data rate for uncompressed video • 20 to 30% more data for compressed video
8 Virtual Reality and 360° Video
• More immersive and interactive experience • 360° video requires 3X-4X the resolution of the FoV (~100°) • Stereoscopic 360 ° video requires 2X the data rate
9 Quality Requirements for Next Gen VR
4K 8K 12K 16K Per eye resolution 1Kx1K 2Kx2K 3Kx3K 4Kx4K
360° 2D total 4Kx2K 8Kx4K 12Kx6K 16Kx8K resolution 360° 3D total 4Kx4K 8Kx8K 12Kx12K 16Kx16K resolution
8K per eye is the ideal resolution for full 16K and 240 fps are needed for “true VR” immersive virtual reality Palmer Lucky, co-founder of Oculus Rift Raja Koduri, head of GPU at AMD Data Requirements for Next-gen Video
Uncompressed Bitrate [Gbit/s] 800 700 600 500 400 300 200 100 0 UHD-4Kp60 8Kp60 8Kp120 16Kp60 16Kp240 unpacked format packed format Video Codec Evolution
12 High Efficiency Video Coding HEVC
Compressed Bitrates - HEVC 800
700
600
500
400
300
200
100
0 4Kp60 8Kp60 8Kp120 16Kp60 16Kp240 Agenda
1. Next-generation Video Formats
2. Spin Digital Video Processing Solutions
3. Case Studies and Projects
14 Spin Digital Video Technologies GmbH
Specialists on video codecs, software engineering and Spin-off of the Technical computer architectures University of Berlin (November 2015)
Based in Germany Innovative B2B company with decades of accumulative research experience In-house developed software IP & products
15 Spin Digital Products
High quality and high compression efficiency HEVC encoder
Full-featured media player for 4K, 8K, and beyond
High performance HEVC/H.265 decoder, video filtering, and video renderer SDK
16 High Performance Video Decoding: SIMD
SIMD optimizations: - SSE (64-bit) - AVX2 (128-bit - AVX512 (512-bit)
C. C. Chi, M. Alvarez-Mesa, B. Bross, B. Juurlink and T. Schierl, "SIMD Acceleration for HEVC Decoding," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 25, no. 5, pp. 841-855, May 2015
17 High Performance Video Decoding: Multithreading
• Advanced Multihreading 8K Decoding Performance • More than 50 CPU cores 300 • NUMA architectures 250 • Experimental set-up 200 • CPU: Xeon Platinum 8168, 150
• 2 x 24 cores (96 threads) 100 Frames/Second
• 2.7 GHz 50
• GPU: Nvidia Quadro M4000 0
• OS: Windows 10, 64-bit 4:2:0 10b 4:2:2 10b 4:4:4 10b 4:2:0 12b 4:2:2 12b 4:4:4 12b
C. C. Chi et al., "Parallel Scalability and Efficiency of HEVC Parallelization Approaches," in IEEE Transactions on Circuits and Systems for Video Technology, vol. 22, no. 12, pp. 1827-1838, Dec. 2012. 18 High-end Video Processing on PC Platforms High Performance Video Decoding: Memory Optimizations
• Memory optimizations Performance for 8K Decoding 350.0 • Efficient pixel formats 300.0 • Bitplanar fr 10 and 12-bit 250.0 • Texture compression 200.0 • BC4 compression FPS 150.0 100.0 • Experimental results 50.0 0.0 • CPU: Intel Core i9-7960x 420 10b 422 10b 444 10b
• 16-cores Planar Bit BC4 • 2.8 GHz • GPU: Alvarez-Mesa M., Ching Chi C., HEVC Software Media Player for Ultra-High Quality Video: 8K and Beyond, The 25th International Display Workshops. • AMD Radeon WX 7100 December 12-14, 2018. Nagoya. Japan (paper accepted)
20 Agenda
1. Next-generation Video Formats
2. Spin Digital Video Processing Solutions
3. Case Studies and Projects
21 8K Media Player for Home Theatre
NHK Engineering Systems (NES) NHK Open House 2016, 2017, 2018 PC based 8K Media Player for Home Theater applications 8K HEVC video + 22.2 audio
22 8K Video Wall
• Demonstrations at NAB and IBC 2017 with Intel Corporation • 8K PC-based media player using Intel Xeon Processors
23 Live 8K Streaming via Satellite
Real-time 8K streaming via satellite Development project by SES, Sharp, and Spin Digital Demonstration performed at SES Industry Days 2018 (Luxembourg)
24 EU Projects: Immersify
Audiovisual Technologies for Next Generation Immersive Media • Duration: 30 months: Oct 2017 – March 2020 • Funding: Horizon 2020 Innovation Action • Budget: 2.5 million €
25 Project Partners
• Poznan Supercomputing and Networking Center - PSNC (Poznan, PL) • Spin Digital Video Technologies GmbH (Berlin, DE) • Ars Electronica Futurelab (Linz, AT) • Marché du Film – Festival de Cannes (Cannes, FR) • Visualization Center C (Norrköping, SE) Project Objectives
• Video codecs: improve quality using advanced compression technology • Multi-Display: enable immersive media for multiple display environments • Interactive: provide tools for personalized and interactive non-linear storytelling • Content and Tools: promote immersive media in the creative industries 16K Media Player
• Resolution: 16Kx8K • Frame rate: 60 Hz • HEVC 400 Mbit/s • CPU: Intel Xeon Scalable 2x24 cores • Demo presented at ISE 2018, NAB 2018, IBC 2019
28 High Resolution 360° Video
Pandarama by ARRI • 5 ALEXA Mini (360° x 60°) • Fraunhofer OmniCam rig • 12k x 8k resolution 24 fps • Spin Player full resolution playback with 360° player Conclusions
• Highly immersive media is emerging • Requires next generation video: 8K, 16K, high fps, high bidepth, 3D, 360°, .. • Massive data rates beyond UHD TV formats • Spin Digital has developed a high-performance HEVC implementation • Software optimizations for state-of-the-art PC architectures • Up to 16K video decoding and playback • Spin Digital working with partners on tools for immersive media • Immersify Project • NHK, Intel, SES, Thank you for your attention
HIGH PERFORMANCE VIDEO CODECS Backup Slides HEVC Encoder
Relative video file size for same objective quality and encoding speed Higher compression & quality than leading 8Kp60 open source encoders 100 Support for 4K, 8K and 16K
90 Fast offline compression
80 Ready for HDR and WCG 4:2:0, 10-bit 4:2:2, 10-bit • Ready for 8K and beyond
File Size[%] File 70 66 66 4:4:4, 10-bit • Significantly better compression and quality than competing encoders 60 • Enables WCG and HDR with up to 12-bit video 52 • Support for HDR metadata 50 • Preserves color resolution with 4:2:2, 4:4:4, x265 2.5 (HEVC) Spin Enc 1.6.2 and RGB formats (HEVC) • Support for 360° video projection formats 33 Rate-Distortion Performance of an Optimized HEVC Encoder
Helicopter, 7680x4320, 60 Hz, 4:2:0, 41.0 40.0Helicopter, 7680x4320, 60 Hz, 4:4:4, 10-bit 10-bit 40.5 39.5 40.0
39.5 39.0
PSNR [dB] PSNR -
39.0
PSNR[dB] -
YUV 38.5 38.5 Spin Digital Spin Digital YUV (H.265) (H.265)
38.0 38.0 50 100 150 200 250 300 50 100 150 200 250 300 Bitrate [Mbits/s] Bitrate [Mbits/s]
38 % bitrate reduction 52 % bitrate reduction
34 HEVC Media Player
120 Media player for 4K, 8K, and 16K
100 100 Based on HEVC software solution 84 80 64 60 54 54
40 • Pure software solution
• Ready for up to 16K playback Frames per second (fps) 20 • Professional video formats: 4:2:2 and 4:4:4 • 10-bit, HDR, and WCG support 0 • Next generation audio: 22.2 surround sound VLC 3.0.0* MPC-BE 1.5.2* Spin Player 1.2 • 360° video playback CPU: Intel Core i9-7960X @ 2.80 GHz (16 cores) 8Kp60, 4:2:0, 10-bit, 120 GPU: NVIDIA GeForce GTX 1060 6GB * GPU-based HEVC hardware decoder Mbps • Easy-to-use graphical user interface OS: Windows 10, 64-bit 8Kp60, 4:2:2, 10-bit, 160 Video: Unigine Superpostion Benchmark 8K Mbps • SDI and GPU rendering 35 HEVC Decoder & Renderer SDK
Maximum achievable bitrate for HEVC decoding and video rendering using state-of-the-art PC technologies Ultra high performance HEVC decoder 1,200 1,140 Next generation video rendering 1,000 928 Real-time 16Kp60 playback 800
4:2:0, 10-bit • Pure software solution 600 573 4:2:2, 10-bit • Powerful and efficient API for Windows and 4:4:4, 10-bit Linux 375 400 • High quality video formats: 4:2:2, 4:4:4, up to 275 Maximum Maximum Bitrate [Mbps] 12-bit 200 • Real-time HEVC/H.265 decoding up to 16K • CPU: Intel Xeon E5-2696 v4 10-bit rendering, WCG and HDR support (2x 22 cores, 2.2 GHz) 0 GPU: NVIDIA Quadro M5000 • Flexible I/O with GPU and SDI devices 60 frames/s 120 frames/s • Video filtering library
36 Immersify: Ars Electronica Deep Space 8K
• 8K HEVC for immersive media players • 8Kx8K projection surface • Interactive applications Source: AEC 37