Pre-Workshop Tutorial HD Transport, Timing and Compression Paul Hightower CEO Instrumentation Technology Systems

Pre-Workshop Tutorial HD Transport, Timing and Compression Paul Hightower CEO Instrumentation Technology Systems www.ITSamerica.com

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 1 HD Video

New Concerns Benefits  Resolution is a more complex subject  Order of magnitude improvement in  More formats imagery  High transport data rates  Wider color gamut  Large volumes of data for archive  Recording and reproduction can be  Alias artifacts in images identical to raw video  Single Frames approach snap shot quality  Metadata space built in to the SMPTE frames

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 2 Translating between NTSC (analog) & HD-SDI (digital) Video

NTSC video -color bar test pattern

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 3 Translating between NTSC (analog) & HD-SDI (digital) Video

At the SDI source At the end of a 100 meter cable

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 4 HD-SDI Imagery Chain

ITU-R-BT-xxx Image sampling standards SMPTE 296/274 709 includes 4:2:2 HD Format Sampling Standards SMPTE 291 (720/1080) HD Metadata Standard SMPTE 292/424 HD-SDI Serialization Standards (720p & 1080i/1080p-3G

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 5 Translating between NTSC (analog) & HD-SDI (digital) Video

Attribute Analog Video SDI Digital Video 480 SD NTSC 480/frame 576 (PAL) SD Visible Scan Lines PAL 576/frame 720 HD (750 total/frame) 1080 HD (1125 total/frame) This depends on the source and signal SD 720 pixels Resolution/Line quality but ranges to the equivalent of HD 1280 (1650 total/line) pixels and 300 to 720 pixels 1920 pixels ( 2200 total/line) pixels YUV encoding samples intensity every pixel and color differently depending on the encoding chosen. 4:2:2 is most Continuous time domain signal, frequently used Color Sampling intensity swings are limited by the Sample rate = 74MHz (720p/1080i) available 1.5 MHz bandwidth Color BW 6≈ 10 MHz Sample rate = 148MHz (1080p) Color BW ≈ 20 MHz

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 6 Translating between NTSC (analog) & HD-SDI (digital) Video

Attribute Analog Video SDI Digital Video

Complex AM, FM and phase modulated signal Serial encoded bit stream at bit rates from Raw Video requiring 6 MHz bandwidth 270 Mbits/s to 3000 Mbits/sec (12 GB for 4K)

Pedestal and color burst sync areas scaled A reserved bit patterns (SAV & EAV) defined by Sync generally below the black level SMPTE in the image frame

ANC space between EAV & SAV is Horizontal “blanking” data space, ANC space between SAV Blanking A predetermined voltage level in the video signal and EAV from first line to line 40 (1080) or 25 (720) is Vertical “blanking” data space. A stream of video samples between SAV & EAV the An AM signal with overlaid phase modulated Active Video number of which and format varies with resolution, color information sampling scheme and color depth

RS 170 60Hz /30 Hz Field/Frame RS170A 7 Frame/Field (NTSC) 59.94 (60/1.001) Many from 24.975 to 60 Hz and beyond Rate CCIR 50Hz /25Hz field/frame Progressive, segmented and Interlaced Progressive and Interlaced

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 7 Quick Idea About Subsampling

4:2:2 subsampling causes two luma samples to share one pair (Cr and Cb) of color samples

Graphic from “Chrominance Subsampling in Digital Images”, by Douglas Kerr

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 8 Video Data

4:2:2 10 Bit Color Sample Color Sample

Cr0-1 Y0 Cb0-1 Y1 Cr2-3 Y2 Cb2-3 Y3 Active Video Data Stream Sample Pair Sample Pair Sample Pair Sample Pair Cb = blue color value Pixel 0 Pixel 1 Pixel 2 Pixel 3 Cr= red color value Y = luma value LN= line count ANC is ancillary data  Each Y sample represents a pixel > 1920 pixels on a line=1920 Y samples space CA = color channel  Each Y sample is coupled to ½ of a color sample as shown Y = Luma channel Ancillary data is  Each Pixel then is comprised of 1 Y and ½ Chroma @ 10-Bits each placed in Luma  Each pixel is 20 bits as a way of thinking about it channel first until full  A 1920x1080 frame = 20x1920x1080 bits =41,472,000 bits or 5,184,000 bytes (image only)  Add in the EAV, SAV, Line Count, CRC, HANC and VANC space; Frame = 2200 S x 1125 L • 2200 samples x1125 lines = 20x2200x1125/8=6,187,500 bytes/frame • At 60 FPS; Data rate = 371,250,000 bytes/second • Archive 1 minute of video = 22,275 Mbytes  Frame Rate only affects Data Rate

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 9 Image Frame Organization

The Samples build a frame  720 Frame Parameters  Frame size = [750 lines x 1650 samples x 20 bits/pixel ]/(8 bits/byte) = 3,093,750 bytes/frame

VANC Space 25 Lines * 1280 samples = 32K Cinema & broadcast generally only use lines 14-15 for scene switch & closed captioning HANC Space 370 Samples x 750

>2000 data items Example Packets • 16 channel audio snippets EAV SAV CRC EAV • Film Codes

Line Count • Payload data • Workflow data • Copyright data • V-chip data • Billing data • Logging Info

Buffer Space Lines 746‐750

HANC 1280 luma samples

1650 luma samples May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 10 SDI = Frame Data Serialization

4:2:2 Multiplexed Encoded SDI Video Stream Byte level stack up

SDI Serial Stream IAW SMPTE 292 or 454

When serialized 720p/60 =3,093,750 bytes/frame * 60 fps =185,625,000 bytes/second 185,625,000 bytes/second *8 = 1,485,000,000 bps.

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 11 Imagery

Movies  Frame Rate (pictures per second) • Set to give the illusion of smooth motion; beyond persistence of vision frequency. • Rates above 16 images/second yield smooth motion • 24 fps is used in movies • 25/30 fps used in TV (PAL/NTSC)  Illumination Rate (flashes per second) • Flicker fusion is the frequency that pulsing light looks steady • Illumination rate is pushed high enough to achieve flicker fusion • Film generally uses 48 Hz flicker rate • TV targets 60 Hz flicker rate  Interlace TV scanning is 2x the frame rate; Odd lines then even lines  Progressive 30 FPS TV displays the same frame twice @ 60 Hz  Progressive 60 FPS displays each frame @ 1/60th of a second

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 12 NTSC

National Television System Committee (NTSC)  Did set the standards for analog TV format  Color upgrade for TV in the 50s required B&W TVs to receive Color Signal  60Hz scan rate slowed to make room for color burst signal = 59.94 flicker 29.97 frame rate Digital Video on modern displays don’t need this  Legacy rate is predominant and remains

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 13 Imagery

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 14 Imagery

Resolution is 3 dimensional  Line Count  Pixel Count/Line  Bit Depth/Pixel Lines

Pixels

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 15 Imagery

More Bits = More Shades = More detail

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 16 Imagery

2 bits = 4 colors All pictures have the same number of pixels

4 bits = 16 colors

8 bits = 256 colors

24 bits = 16 million colors

According to http://en.wikipedia.org/wiki/Color, humans can distinguish up to 10 million colors

Pictures from http://en.wikipedia.org/wiki/Color_depth

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 17 17 Imagery

Rendering Effects of Pixels and Bit Depth  Same Image

Few pixels Many pixels Many pixels Many shades Few shades Many shades

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 18 18 Imagery

Sampling  RGB; the primary colors  Intensity; luminescence, (luma), the black & white information  How many samples per image? (pixel count)  How many shades per sample of each color? (bit depth)

Many Bits/Color

One Bit/Color

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 19 Imagery

4:2:2 Subsampling is not new  Analog Color TV has a color content bandwidth of 1.5 MHz, BW info is 3.0 Mhz.  Human physiology • More sensitive to changes in intensity than changes in color 4:4:4 Sampling  Equivalent to RGB  In motion imagery, 4:2:2 sampling is not discernable from 4:4:4 Bit/Pixel Benefit  4:4:4 @ 8 bits/color =24 bits/ pixel  4:2:2 @ 10 bits + ½ 20 bit color sample = 20 bits • 16.6% savings in data with no visual motion imagery degradation

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 20 Imagery

Subsampling  Created to reduce bandwidth to transport video  Takes advantage of human physiology where • The eye far more sensitive to intensity changes vs. color changes  In Analog • Intensity (Luma) plus TWO of the color (chroma) components is used • Phase modulation used to encode color is about 1.5 MHz (vs. 3.25 for Intensity)  In Digital • One color sample per luma = 4:4:4 equivalent to RGB  Bits/frame = 1080*1920*24 (16 million colors) = 49,766,400 • One color sample per two luma = 4:2:2, equivalent to broadcast analog TV  Bits/Frame = 1080*1920*(10 -bit luma + 10-bit Red or Blue sample) = 41,472,000 (16% reduction) • Side effect: Saves storage capacity requirements as well • No effect on human perception of the imagery • EXCEPT there is a color alias at edges when the edge is between luma samples with shared color sample.

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 21 Imagery

Subsampling Effects

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 22 Imagery

 Good Side effect for overlay • Natural “surround”  Color Alias when intensity changes on a constant color • Y-b-r = g •Y-Δ –b-r = g –Δ  Changes the ratio of RGB which changes the color, not just the brightness

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 23 Pixels & Bit Count & Data Rate

Key Points to Remember  Each pixel = a Y (luma) sample  Vertical blanking space adds lines; e.g. 45 in 1080 • 1080 lines, plus blanking = 1125 lines/frame  Horizontal blanking space adds samples; e.g. 280 Y samples per line in 1080/60 • 1920 visible pixels + 280 Y H blanking samples = 2200 pixels/line  Each pixel in 4:2:2 sampling is 20 bits deep • 10 bits of luma (Y) and 1 of the color components (Cr or Cb) @ 10 bits = 20 bits  Interlace Video two types • Segmented Frame  delivers ½ the image in one field and the other half in a second field • True Interlace  One image odd lines, a second image (16 ms later) using even lines.  Progressive Video delivers a complete frame per scan • Frame rate = field rate (e.g. 1080p/60) • Frames may repeated at field rate (e.g. 1080p/30)

October 26, 2011 May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 24 Pixels & Bit Count, Data Rates, Resolutions & Specs

Active Total Luma Total Active Luma SMPTE SDTV lines Samples in Luma Aspect Frame Rate Digitizing Bit Rate lines per samples SDI HDTV per Blanking Samples Ratio (Hz) Specification (MBit/Sec) frame per Bit Format frame Area (Pixels) Line Sampling Frequency (MHz) Interlaced (I) Progressive (P) Interlaced (I) Progressive

SDTV I 480 525 720 138 858 13.5 4:3 29.97 259M ITU-R BT.601 270.0

60 or HDTV P 720 750 1280 370 1650 74.25 16:9 292M ITU-R BT.709 1485.0 60/1.001 HDTV P 720 750 1280 700 1980 74.25 16:9 50 292M ITU-R BT.709 1485.0 (PAL) 30 or HDTV I 1080 1125 1920 280 2200 74.25 16:9 292M ITU-R BT.709 1485.0 30/1.001

HDTV I 1080 1125 1920 720 2640 74.25 16:9 25 292M ITU-R BT.709 1485.0 (PAL)

HDTV P 1080 1125 1920 280 2200 148.5 16:9 60 424M ITU-R BT.709 2970.0

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 25 25 Transport & Storage

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 26 Metadata

Storage Metrics  SMPTE 296; 720 Frame Parameters •3.1 MB/frame • 1 minute of 720p/60 video = 11,160 MB  SMPTE 274; 1080 Frame parameters •6.2 MB/frame • 1 minute of 1080p/30 video =11,160 MB • 1 minute of 1080i/60 video =11,160 MB • 1 minute of 1080p/60 video =22,360 MB Data Rate Metrics  720p/60 = 1.485 Gbits/sec  1080i/60 =1.485 Gbits/sec  1080p/30=1.485 Gbits/sec  1080p/60=2.970 Gbits/sec

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 27 Transport Issue

How do you get raw video (SDI) from source to destination?  Analog NTSC • any channel with a 6 MHz bandwidth will work  SD-SDI • Requires a channel capable of passing 270 M Bit data rate (1.5 GHz for 7th harmonic)  HD-SDI • Requires 1.5 Gbit channel for 720p/1080i and 3 GHz for 1080p (10 GHz channel) Choices  Direct Connect (copper) • SMPTE Specs HD-SDI be capable of operating to 100 meters of 75 Ω Coax (e.g Belden 1694A) ; LOW LOSS, tight tolerance • These do require line equalizers and drivers; Reclocking is generally needed to properly decode • Short runs can use Cat 6A/Cat 7 copper cable for short runs (10 m); CAT 5 and standard CAT 6 will not work

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 28 Transport Issue

Choices (cont.)  Ethernet • At 1G Ethernet, ED-SDI can work • At 1 G Ethernet, 720p/1080i will not work • 10G Fiber Only full duplex only • 100G Fiber Only, full duplex, still evolving  Fiber • 10GBASE-ER single-mode fiber supports transport @ 10.3 Gbit/sec up to 30-40 Km • Next level down “-LR” can support this rate up to 220 meters  Radio • Where would the band and bandwidth exist?  Transfer of Video as Data over GigE • No less than 0.07 seconds/1080 frame; 1 minute of 60 FPS video = 5 minutes to download • Requires use of full capacity of GigE channel • Assumes no lost/corrupted packets

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 29 Storage Issue

1 Hour of 1080p/60 = 1.3416 terabytes of storage

Cinema Industry captures 100s of petabytes per day

While cost/MB is going down, the real estate, power and maintenance is huge

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 30 Compression Tool

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 31 Compression Tool

Compression would never exist if infinite bandwidth and zero cost storage existed Compression is a tool to reduce data rate & storage needs  Transport • Alternative to whole new infrastructures • Typical Compression Ratios that maintain excellent image quality  H.263 and MPEG-2 ; 30:1  MJPG 2000; 20:1 to 40:1  H.264/MEG-4 part 10; 50:1, 100:1 • Compression Issues  Interframe prediction (MPEG) vs. frame-to-frame image compression (M-JPG)  MPEG is motion sensitive  M-JPG can generate “rings” at the harsh image edges  Trade off between image quality and frame rate/Frame dropping  Latency  Storage • Gigabytes/clip become megabytes/clip

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 32 Compression Tool

Compression destroys pixels  Lossless preserves all of the essence  Lossy loses varying degrees of the essence  MPEG, H.264, H.265 all divide each image into a mosaic

Macro-pixel Starts @ 16x16 pixels

Average Intensity 32 number values in a matrix of values

> 256 values replaced Change in Color > Change in Color > by 32 = 8:1

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 33 Compression Tool

Differential Frames

Reference Frame All values in the matrix

Differential Frames Duplicate Matrix elements replaced with a reference to its duplicate in the reference frame

Group of Frames Reference Frame All values in the matrix Can achieve 1000:1 average compression or more

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 34 Compression Tool

Reference Frame Interframe 1

Interframe n Group of Frames Options:  Set the number of frames in a group, image quality/frame rate preference • More frames, higher compression ratio • More frames, the more buffering needed to reconstruct • Motion prediction reduces aliasing  Set data rate • Closes loop on data rate; auto adjusts GOP, bit depth mosaic dimensions, motion comp, frame rate as needed  Set Frame Rate, Image Quality – all have different effects

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 35 Compression Tool

Compressor is Closed Loop  A FIFO equivalent is used to monitor transfer rate into the transport channel • FIFO content grows = more compression needed • FIFO content decreases = less compression needed • FIFO can’t empty (not data to send) • FIFO can’t overflow (lost frames) • Initial buffering required to pre-load the FIFO  Consequence of FIFO underflow • The video chain must resync, delays could be many frames  Consequence of FIFO overflow • Dropped frames  Compressor Options • Increase the GOP • Increase the macro-pixel dimension • Reduce the sampling • Reduce bit depth

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 36 Compression Tool

Consequences of Increasing the GOP  A corrupted reference frame can lose more differential frames • Dropped frames, sync loss  More pre-buffer is needed, more decoder buffering is needed Consequences of Increasing the macro-pixel dimension  Fewer macro-pixels per image; loss of resolution Consequences of Reducing the sampling  Compressors can alter sampling to 4:2:0  Loss of detail and increased color aliasing Consequences of Reducing the bit depth  Loss of detail

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 37 Compression Tool

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 38 Transport Compression Tool

MJPEG  Every frame independently compressed  Good • Less buffering to reconstruct (less end-to-end latency than MPEG) • Each image is independently reconstructable  Bad • Wavelet exhibits ringing at sharp edges • Compression ratio limited  Complex but static scenes don’t compress very well  no differential frames used to take advantage of this

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 39 Compression Tool

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 40 Compression Tool

Compressor is Closed Loop  FIFO Driven just as MPEG  Compressor Options • Increase the macro-pixel dimension • Reduce the sampling • Reduce bit depth  Each frame may only be compressed 8:1 to 16:1 without loss  Wavelet is lossy but can only achieve 30:1 to 40:1 compassion with good image quality  Frame to frame compression varies depending on the complexity of the scene  Without the use of differential frames, no further average compression is available Benefits of no use of differential frames  No lost frames with corruption of a reference frame  Each frame may be rendered independently  Less pre-buffering is required at the destination to being rendering video

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 41 Compression Tool

Latency ∑ decode (sdi-image stream) + compression + xmit latency + buffer time + decompress + decode for display

SDI Source Decode Compress xmit

Display Decompress Buffer Receive

 Compression • Many factors including image content, motion between frames, hardware speed  Buffer Time • Decompression requires a complete data set and enough buffered data to ensure every frame is reconstructed at the full expected frame rate  1300 ms to complete a 60 frame buffer @ 100 MB Ethernet @1000:1 MPEG compression, 1080p/60 (2 reference frames)  1300 ms to complete 60 frame buffer @ 100 MB Ethernet @ 500:1 compression; 720p/60  500 ms to complete a 2 frame buffer using MJPEG2000 @ @ 40:1 compression

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 42 Transport Compression Tool

Bit Rates Bit Rates Bit Rates Bit Rates 1 Stream SMPTE Example Bit Rates MJP2000 MPEG-2 MJP2000 H.264 Video Type Video over Standard Formats (Mbits/s) (Mbits/s) (Mbits/s) (Mbits/s) (Mbits/s) Ethernet @10:1 @ 30:1 @40:1 @50:1 270 27 9 6.8 5.4 360 36 12 9 7.2 259M SD-SDI 480i, 576i 10 Base T 143 14 4.8 3.6 2.9 177 18 5.9 4.4 3.5

344M ED-SDI 480p, 576p 540 54 18 13.5 10.8 100Base T

720p, 1485 148 292M HD-SDI 49.5 37 29.7 100Base T 1080i 1470 147

424M 3G-SDI 1080p 2970 297 99 74 59.4 1000 Base T

SDI & Ethernet  SMPTE 2022-6:2012 STD for SDI over Internet Protocol  As bandwidth continues to increase, SDI over IP will become practical • Need to compress, at least lossy compression will dissipate.  Specs are being developed for clean switch points and other work flow issues  Good survey of the subject, SMPTE Motion Imaging Journal, March 2014

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 43 Timing

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 44 Timing

Analog Video  6 MHz bandwidth  Can be transported in native form over long distances  Timing compensation is predictable and could be calibrated to microsecond resolutions  Timestamps can be anywhere in the transport stream and be accurate to a few microseconds. HD Video  Needs a bandwidth > 6 GHz for reliable transport  Raw video is data  Raw video may only be transported over coax 300 feet; packetized over 10G Ethernet  Compression is needed to transport over general networks and radio links • Introduces Latency • Latency is scene, network, codec hardware and frame rate dependent • Latency is a variable with average values as high as 10 seconds • Latency introduces the need to timestamp at the video source

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 45 Timing

Timestamping at the source  Overlay • Compression can blur fractional seconds  Metadata • Great solution • MISB adopted this; an example = Microsecond Timestamp • Many encoders strip VANC space

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 46 Metadata

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 47 Metadata

Metadata is not new  Analog video has limited metadata in vertical blanking • A few lines are used each holding perhaps 10-20 bytes  Edge encoding is another technique used mostly by the Ranges • The standard puts 1 bit per line for about 250 lines/field (30 bytes/field)

Vertical Blanking Space A few lines above the visible image Contains • VITC • Closed Captioning • Teletext • Billing data • Copy protection & V-chip data • ≈240 bytes per frame

Edge Encoding (non-broadcast) Steals a bit of each video scan line Supports 1 bits/line 60 bytes per frame

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 48 Metadata

Analog Data Spaces

Vertical Blanking Space A few lines above the visible image Contains • VITC • Closed Captioning • Teletext • Billing data • Copy protection & V-chip data • ≈240 bytes per frame

Edge Encoding (non-broadcast) Steals a bit of each video scan line Supports 1 bits/line 60 bytes per frame

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 49 Metadata

 HD-SDI Vertical Ancillary (VANC) Space 1280-1920 samples per line (“vertical interval”) 25-40 lines Totals 30K – 76K bytes per frame

Horizontal Ancillary (HANC) Space 280-370 samples per line (“left edge” but out of view) 740 -1120 lines Totals up to 273K -313K bytes per frame

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 50 Metadata

HD Data Spaces

HDTV SDI Line Area SMPTE Digitizing Bit Format Aspect Ratio Specification V Blanking Are Frame Rate(Hz) Bit Rate (MBit/Sec) Total per frame lines Total Active lines per frame Active Luma samples per Total LumaTotal (Pixels) Samples Luma Samples in H Blanking Interlaced (I) Progressive(P) 60 or ITU-R HDTV P 720 750 30 1280 370 1650 16:9 292M 1485.0 60/1.001 BT.709 HDTV ITU-R P 720 750 30 1280 700 1980 16:9 50 292M 1485.0 (PAL) BT.709 30 or ITU-R HDTV I 1080 1125 45 1920 280 2200 16:9 292M 1485.0 30/1.001 BT.709 HDTV ITU-R I 1080 1125 45 1920 720 2640 16:9 25 292M 1485.0 (PAL) BT.709 ITU-R HDTV P 1080 1125 45 1920 280 2200 16:9 60 424M 2970.0 BT.709

VANC data space HANC data space

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 51 Metadata

Metadata Spaces are Designed into HD  720 Frame Parameters

VANC Space 25 Lines * 1280 samples = 32K Cinema & broadcast generally only use lines 14-15 for scene switch & closed captioning

HANC Space 370 Samples x 750

>2000 data items Example Packets • 16 channel audio snippets EAV SAV CRC EAV • Film Codes

Line Count • Payload data • Workflow data • Copyright data • V-chip data • Billing data • Logging Info

Buffer Space Lines 746‐750

HANC 1280 luma samples

1650 luma samples

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 52 Metadata

 1080 Frame Parameters

VANC Space 40 Lines * 1920 samples = 76.8K Cinema & broadcast generally only use lines 14-15 for scene switch & closed captioning

HANC)Space 280 samples * 1125

>2000 data items Example Packets • 16 channel SAV EAV CRC EAV audio snippets

Line Count • Film Codes • Payload data • Workflow data • Copyright data • V-chip data • Billing data • Logging Info

Buffer Space Lines 1121‐1125

HANC 1920 luma samples

2200 luma samples

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 53 Metadata

Uses of Horizontal Ancillary (HANC) space  HANC space used by broadcast & Cinema • As many as 3000 data types could be present  Audio (AES); 16 Channels  Payload (format, frame rate, etc.)  Advertisers, FCC logging, etc. Uses of Vertical Ancillary (VANC) Space  VANC space is not used much by broadcast • Line 14 is scene switch point • Closed Captioning (mostly line 15)  GOV uses it • MISB has >900 data types defined • All are KLV type 02  Anyone can use the KLV Structure • Form valid SMPTE 291M packet • Insert desired data • Detect Packet • Extract Data May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 54 Metadata

Many Metadata Types Defined by SMPTE  Dictionary •Example Keys  16 Channels of sound bytes, source data, airing time, editing workflow  SMPTE time code  SAP, film codes… on and on • Data content •Format  Metadata Elements Dictionary  RP210 Version 13 of this registry contains more than 3000 data types.

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 55 Metadata

Why is VANC Metadata Important  Sparsely used  Can collect data permanently synchronized to the imagery  Can collect data without burning it in to the imagery  Can display some or all of the data later  Can extract data from video in synchronism with the imagery  Data never interferes with imagery  Overlay never compromised by compression artifacts  Overlay color can be changed during display time  Overlay position can be changed during display time  MISB has many VANC space keys reserved

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 56 Metadata

KLV VANC Pack is a SMPTE Structure  SMPTE Standard 291M type 02  Wrappers (ADF signature bytes)  Type identifiers (DID and SSID)  Length (DC) UDW MISB uses the Type 02 KLV SMPTE Structure  Key, Length Value •Key = data type identifier (group of variables) •Length = number of bytes of the value •Value = data itself  Size of KLV = Data Count (1-255 Bytes) • Message ID (1 byte) • Program Segment Counter (2 bytes) • Key (16 bytes) • Length (1 byte) • Value ( L number of bytes); any data

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 57 Metadata

Value Element is Memory Space  Bytes may be assigned to  ASCII  Binary blocks  Values: integers, floating point Value

The Key Number identifies the organization of the memory Memory Location Memory Location Memory Location Memory Location 1-235 Bytes … Memory Location

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 58 Metadata

How many KLV packs are needed?  Data have a resolution of 24 bits = 3 bytes [59 parts/billion or 59*E-9] • More than 74 data items of 24-bit resolution can fit in a single KLV pack  Example: ITS sample “Instrumentation Pack” • 41 data items  Pointing Angles up to 24 bit resolution  Ranges as SP values of ±9.999999E±16 Test ID Encoding  64 bit time stamps DAS Time Image mode  Five 10 ASCII character fields Geodetic Datum Integration time Run Number Trigger time • 170 Bytes TOTAL! Classification Time offset Temp Timestamp mode Pressure Frame Rate humidity Lens ID Object(target) ID wind speed Zoom factor Tsens AZ direction Units Tsens EL Mount ID Focus setting Tsens Range Camera ID azimuth Tx Camera placement elevation Ty H resolution Mount angles time Tz V Resoluiton Range Tspeed Image bit depth Range timestamp T‐Heading Encoding Object(target) ID Ttemp Image mode Tsens AZ May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 59 Metadata

How do you use KLV packs  MISB • Fixed keys • No general purpose software available • Most keys have specialized purposes  Private Keys • SMPTE 291 compliant (MISB foundation) • Up to 2 packs can be inserted per frame  After the MISB Microsecond Timestamp (line 9 of each frame) • KLV Tool Kit  Free with KLV upgrade to any inserter  XL template to design the memory space (235 bytes of V)  GUI to Program ITS Inserters  GUI to simulate & test key designs

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 60 Metadata

Non-Registered Keys –Custom Uses  Special purpose designs • Fuse your data sampled at the frame rate to each image  Must conform to SMPTE 291M Type 02 structure  Can pass through all commercial equipment without harm Using equipment capable of recognizing customs keys can:  Pass unobstructed, unaltered imagery while carrying image relevant data  Extract previously inserted custom key data •Overlay • Output customer data to files

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 61 Metadata

MISB used the type 02 KLV SMPTE structure  > 900 register keys • MISB Standard 0807 lists the registered private keys http://www.gwg.nga.mil/misb/stdpubs.html • All MISB keys are 6.0E.2B.34.xx.xx.xx.xx.0E.0y.xx.xx.xx.xx.xx.xx Y=01 or 02 or 03 • Most Derived for UAV uses • Structured to result in continuous pulse streams for PCM systems Microsecond Time Stamp  Found in SMPTE RP210, and MISB 0807  Key = 06.0E.2B.34.02.05.01.01.0E.01.01.03.11.00.00.00  Must be start on 1st sample after SAV of Line 9

 Status Value = 1 Byte; Locked/unlocked source, valid, etc.  Time Value = 8 Bytes; UNIX Epoch > µsec since Jan 1, 1970 May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 62 Custom Keys and UDWs

Design a Private Key  Create a key number (16 byte;32 HEX digits) Key (HEX) 57414A19360A00017F44A0904F013AAB 0 OK  Name Fields Length(Bytes) 200 Est DisplayProcess Time  Select a data type • Signed/unsigned integers • Scaled singed/unsigned integers • Single and Double Precision Floating Point Numbers

• ASCII strings Field Start Input Len Field Ref Name Pad Format • Binary blocks of data Number Byte LEN Range Test ID 1 1 0 AS C II 10 1‐234 DAS Time 2 11 0 Binary 8 1‐225  Locate data items and allocate space Geodetic Datum 3 19 0 Binary 1 1‐217 R un Number 4 20 0 Binary 1 1‐216 • Offset byte and number of bytes C lassification 5 21 0 AS C II 10 1‐215 Temp 6 310UI‐MAX 21‐4  Define Scale & Decimal Point location hiditPressure 78 330UI‐MAX35 0 UI MAX 21‐41 14 • 3 bytes unsigned = 16,777,216 full-scale, scaled to 360 yields a resolution of 21*10-6 • Specify 1,2 or 3 decimal places!

Pad Field Start Len Full Scale Decimal Field Ref Name Before Format Input LEN Display Model Number Byte Range Value Places This Byte Focus setting 27 87 0 UI‐MAX 2 1‐4 1024 0 1024. azimuth 28 89 0 UI‐MAX 3 1‐4 360 8 359.99997854 elevation 29 93 1 SI‐MAX 3 1‐4 180 8 +/‐179.99997854

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 63 Custom Keys and UDWs

 Assign a Key number  Design the Value  Key template calculates the length UDW

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 64 KLV = Game Change

ITS Insertion Engine w/KLV Option  Adds up to two full KLV packs after MISB Timestamp  Up to 470 bytes of YOUR data per frame; Data rate to 28.2KB/sec

Line 9

SAV Pack 0 Option Pack 1 Option Pack 2 Microsecond Timestamp Data block Data block

Can turn off Can turn on or off Can turn on or off Fixed MISB Format Can be any data from 1-235 bytes Can be any data from 1-235 bytes Always first pack Optional basic decoder ; Optional basic decoder ; ASCII, Integer, decimal ASCII, Integer, decimal

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 65 ITS KLV Tool Kit

Use a MISB key or Create with the ITS KeyTemplate© Plan what fields would be displayed  Excel worksheet  Export design to ‘CSV’ Time Qualified Plan to Resolution Budget Length Show  Distribute design to all stakeholders (uS) 100 10 Y 20 8 Y Build a 16 byte key 20 1 Y 20 1 Y Key (HEX) 57414A19360A00017F44A0904F013AAB 0 OK 100 10 Y Length(Bytes) 200 Est DisplayProcess Time .002 500 2 N .00 500 2 N Create Fields & Assign Data Types .4 500 1 N .8 500 1 N Field Start Input Len Field Ref Name Pad Format Number Byte LEN Range

Test ID 1 1 0 AS C II 10 1‐234 DAS Time 2 11 0 Binary 8 1‐225 Geodetic Datum 3 19 0 Binary 1 1‐217 R un Number 4 20 0 Binary 1 1‐216 C lassification 5 21 0 AS C II 10 1‐215 Temp 6 310UI‐MAX 21‐4 hiditPressure 78 330UI‐MAX35 0 UI MAX 21‐41 14

Choose how fields would be displayed

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 66 ITS KLV Tool Kit

Data Types Built Into KeyTemplate© • Types used to parse incoming data in monitor mode • Types used to parse data when decoded from KLV packs • Types used by Key Test to form data blocks • Types used by Key Read to display extracted KLV data

Pack Definable Data Types

Max Bytes that Data Type May be Description Assigned

Binary 235 Binary pass-through format, L=number of bytes ASCII 235 ASCII (printable characters only), field length = number of bytes L. unsigned integer, no leading zeros, field length = number of digits of highest value (e.g. 1 byte = 0 to 255, 2 bytes = 0 to 65535, etc.; max = 4 bytes), right justified. ;±N UI-NZ 4 optional to shift decimal. unsigned integer, with leading zeros, field length = number of digits of highest value (e.g. 1 byte = 000 to 255, 2 bytes = 00000 to 65535, etc.; max = 4 bytes). ;±N UI-WZ 4 optional to shift decimal. signed integer, no leading zeros, field length = number of digits of highest value (e.g. 1 byte = ±0 to ±127, 2 bytes = ±0 to ±32767, etc. where sign immediately SI-NZ 4 precedes the first number, max = 4 bytes), right justified. ;±N optional to shift decimal. signed integer, with leading zeros, field length = number of digits of highest value (e.g. 1 byte = ±000 to ±127, 2 bytes = ±00000 to ±32767, etc.). ;±N optional to shift SI-WZ 4 decimal. unsigned integer maximum scaled, no leading zeros, field length = number of bits of resolution (e.g. 1 byte = 8 bits, 2 bytes = 16 bits, etc.; max = 4 bytes), right UI-MAX 4 justified. ;N;MMM required. N= number of fractional digits to display. MMM=scale factor. signed integer maximum scaled, no leading zeros, field length = number of bits of resolution (e.g. 1 byte = 8 bits, 2 bytes = 16 bits, etc.; max = 4 bytes), right justified. SI-MAX 4 ;N;MMM required. N= number of fractional digits to display. MMM=scale factor. SP 4 Binary 32 single precision FP; fixed byte length (4) number range ±9.999999E±16 DP 8 Binary 64 double precision FP; fixed byte length (8); number range ±9.99999999999999E±16

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 67 ITS KLV Tool Kit

Inserter internal fields (IF) • Leave locations (offset/byte lengths) in key design for them • Link them to your key with the Configuration Utility • Inserters will place bytes where specified for each field • Key can be parsed and data will be displayed and extracted

ITS Inserter Internal Field Specifications

Byte Number Decimal Availability Field Name Data Type Max Value IF # Length Range places 6980G ‐HD 6055C ‐nG HD 6055C ‐nHD L atitude S I‐MAX 3 ±90.00000 90 0‐5 Y Y N 1 degrees/10000 Longitude S I‐MAX 3 ±180.00000 180 0‐5 Y Y N 2 degrees/10000 Height/Altitude SI‐NZ/SI‐WZ 3 ±8,388,608 NA 0 Y Y N 3 Me te rs MG R S AS C II 11 NA NA Y Y N 4 LLL########

Time Zone Offset SI‐NZ/SI‐WZ 2 ±12.00 NA 2 Y Y N 5 ±##.## (Hrs). (1/4

Time Sync S ource AS C II 1 NA NA Y Y Y 6 "I"= IR IG , "G "= G P S

Video Format AS C II 5 NA NA Y Y Y 7 e.g. "0720P", "108 ###.##; 23.97, 24,25, 29.9 Video Frame R ate UI‐NZ/UI‐WZ 2 655.36 NA 2 Y Y Y 8 60.00 e.g. ±####### µse Requires incoming M Measured Latency S I‐NZ/SI‐WZ 4 ±2147483648 NA Y Y Y 9 metadata, else 0000 IP addres s & P ort IP Addres s AS C II 21 NA NA Y Y Y 10 "###.###.###.###; ##### (µsec) Camera Sync Delay UI‐NZ/UI‐WZ 2 32000 NA N Y N 11 Only w/CSy( option 1=TTL camera sy , 2=The delayed s Timestamp E vent UI‐NZ/UI‐WZ 1 0,1,2 NA N Y N 12 Only w/CS option ±##### (µsec) T imes tamp offs et SI‐NZ/SI‐WZ 2 ±16,000 NA N Y N 13 Only w/CS Option

KTM AZ S I‐MAX 3 ±359.999957 360 0‐6 N Y N 14 Requires K T optio

KTM EL S I‐MAX 3 ±359.999957 360 0‐6 N Y N 15 Requires K T optio

KTM Range UI‐NZ/UI‐WZ 3 2097152 NA N Y N 16 Requires K T optio Requires KT optio KTM Range Units AS C II 1 NA NA N Y N 17 2,4,6 channel units May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 68 Key Development, Debug & Troubleshooting Tools

Key Read  Select the previously saved data file  The header will show •Key length • What video channel the data was extracted from • What pack number of the channel • The Time Flag (1=locked, 0= not locked) • The key (16 byte value) in hex • The Time Status microseconds  Each field of the Key Design will be

Global Const MAX_PACK_FIELD = 64

shown along with the data associated Type BROWSE_INFO

hWndOwner As Long pIDLRoot As Long with the frame from which it came pszDisplayName As Long lpszTitle As Long ulFlags As Long lpfnCallback As Long • Indicated by the microsecond value lParam As Long iImage As Long  Visual Basic Source Code supplied at End Type Declare Sub CopyMemoryLib "kernel32" Alias "RtlMoveMemory" (ByRef lpDest As Any, ByRef lpSrc As Any, ByVal ByteCount As Long) Declare Function SHBrowseForFolder Lib "shell32" (lpbi As BROWSE_INFO) As Long Declare Function SHGetPathFromIDList Lib "shell32" (ByVal pidList As Long, ByVal lpBuffer As String) As Long not cost Declare Function lstrcat Lib "kernel32" Alias "lstrcatA" (ByVal lpString1 As String, ByVal lpString2 As String) As Long Global g_iFileNum As Integer Global g_iOutFileNum As Integer

Global g_sInput As String Global g_sKLVToolkitPath As String

Public Function IEEE32toDouble(ByVal x1 As Integer, ByVal x2 As Integer) As Double

Dim nInt(1) As Integer Dim fTemp As Single Dim sTemp As String

'Convert 32 bit IEEE754 value to floating point nInt(0) = x2 ………………………..

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 69 Insert into the HD video stream at the source See and Extract from HD video stream at the  Timestamp accurately Destination  Sample data at the frame rate  Overlay some or all data  Extract data to a file

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 70 6520 End-To-End Solutions

Video Capture Artel Fiber Artel

SDI Live Video Monitor LE4 GUI (from Matrox SDK) GUI captures, displays, SDI imagery and KLV metadata

Your Video Source Ethernet

Data Concentration for Control, Insert, Extract Download Raw video data Metadata Insertion and metadata Webserver & GUI

KLV Data Concentrator GUI Uncompressed Metadata Image

KLV packs in SDI are a game changer  KLV Packs can • Transport data • Move cipher blocks • Enable recording of clean video • Maintain alignment of imagery and data Video Encoders/Decoders Must Preserve VANC end-to-end SDI Recorders must • Preserve VANC at record time • Restore VANC at playback time Video Archiving must preserve VANC • SMPTE 2022-6 can support this capability ITS software toolkit • Create KLV • Insert your data • Monitor your data • Display your data • Test your KLV design

May 3, 2016 ITEA 20th Test Instrumentation Workshop Sheet 72