HTML5 WebRTC: Real Time Communications for the Web

Cullen Jennings Jan Linden [email protected] [email protected]

March 2013 Real‐time communications on the Web

Native applications have dominated Web apps require plug-ins & downloads

Technology is silo-ed and proprietary – Lack of open and free solutions no standards exist What is WebRTC?

• Real Time Communications meets the web • A state‐of‐the‐art audio/video communication stack in your • A cross‐industry effort to create a new communications platform • Anyone can innovate with WebRTC Goals of WebRTC

• Empower web developers to create and deploy RTC web apps irrespective of platform

ƒ Open ƒ Secure ƒ High quality ƒ Privacy ƒ Low latency ƒ Interoperable ƒ No plugins ƒ Ease of use

• Facilitate communication between all browsers • A platform for RTC that can be used for many types of applications – E.g., video conferencing, gaming, collaboration,… • Exploit the higher pace of innovation on the web WebRTC: Browser + Voice + Video + Data + Real Time • What it is: – Browsers with interactive voice and video communications – Enable real time data directly between browsers – Interoperable with existing voice and video systems • Impact: – Web based deployment of new applications – Easy mashup into existing business applications – Cross platform applications – Re‐image the web as a peer to peer instead of client server – Enable real time applications on web (games, interactive video, sensors and internet of everything) Seamless Collaboration and Conferencing

• Instant voice and video conferencing • HTML5 driven lightweight screen viewing capabilities • Consistent cross platform experience Video Calling Embedded in Web Page

• Web storefronts offer personal shopper/instant help functionality • The user is presented with a means to instantly video call a storefront customer care representative • The customer care agent could be using a video phone Games

• Embed voice and video into the other elements of an application • Can be part of a 3D environment The plan for WebRTC

Mobile

Make the Put WebRTC technology into Chrome Standardization Address the available as ASAP mobile market open source to established Engage the IETF In browsers browsers and W3C with early implementations Open source that help rally consensus WebRTC Timeline

July 2010, Informal April 2011, IETF May 2011, June 2012, August 2012, November 2012, February 2013, meeting at IETF in RTCWEB WG webrtc.org open getUsermedia in getUserMedia w/o WebRTC in WebRTC in Maastricht formed source project 12 flag in Chrome Chrome 23 Stable Nightly by default Stable 21 w/o flag

June 2010, October 2010, May 2011, W3C January 2012, July 2012, October 2012, February 2013, March acquires GIPS Workshop in WEBRTC WG WebRTC in getUserMedia in WebRTC in Firefox Chrome - Firefox 2013, Mountain View formed Chrome 18 behind Firefox Nightly Nightly behind interop WebRTC in flag behind config config Chrome for Android behind flag Key WebRTC features

• MediaStreams & getUserMedia ‐ access to the user's camera and mic

• PeerConnection ‐ easy audio/video calls

• DataChannels ‐ p2p application data transfer Architecture –Browser to Browser

Web Server

JavaScript JavaScript Application Application SRTP

Browser Browser Architecture – Connection to Existing Voice & Video

Web Server GW to SIP SIP Proxy HTTP

JavaScript JavaScript Application Application SRTP Media Control

Media Browser Issues

• Firewalls • NATs (Network Address Translators) • Security • Federation • Congestion Control • Privacy Browser Architecture • Access to microphones and camera • for audio and video compression • RTP for interactive media, lip sync, loss detection, packet loss concealment • SRTP for encryption of media with DTLS for keying • ICE for NAT and Firewall traversal and network mobility • SCTP/DTLS for real time data such as game player moves • SDP Offer/Answer based negotiation MediaStreams

• A MediaStream represents a media source, containing one or more synchronized MediaStreamTracks • A MediaStream can be converted to an object URL, and passed to a

Input Track Track (video) (stereo audio) Output Left Right getUserMedia Sample

http://webrtc‐demos.appspot.com/html/gum1.html getUserMedia +

• http://webrtc‐demos.appspot.com/html/gum2.html getUserMedia + CSS

• http://webrtc‐demos.appspot.com/html/gum3.html Webcam Toy Demo

http://www.neave.com/webcam/html5 Web Audio API

• The Web Audio API allows creation, analysis, and processing of audio samples • Create an AudioContext, and then add nodes to create a filter graph • In Chrome you can connect a MediaStream, obtained from getUserMedia, as a media source for the Web Audio API

AudioContext

Source Destination node node Web Audio API Demo

http://webaudiodemos.appspot.com/input/index.html getUserMedia + Web Audio API

Audio file or

Web Audio WebRTC Web Audio

• Effects (filtering, room, • Spatial distortions,…) • Effects (filtering, room, • Analysis distortions,…) • Add audio • Analysis getUserMedia + Web Audio API

Audio file or

Web Web WebRTC Audio Audio Media Transport ‐ SRTP

• RTP provides a sequence number and timestamp for each media packets • This allows synchronization of play out of differ media streams (lip sync) • It also allows detection of lost packets • RTCP provides feedback on packet loss rates and RTP statistics • RTP support many forms of error recovery and forward error correction • Secure RTP ( SRTP ) uses symmetric key cryptography to provide confidentiality and integrity to RTP • Ongoing IETF work to multiplex multiple RTP over same UDP Media Keying ‐ DTLS • DTLS is simply the same TLS used for HTTPS run over UDP • DTLS a handshake to form the session keying material for the SRTP media encryption • Used with self signed certificates. Each certificate has a fingerprint which is bound to a user identity in a way described later in this presentation NAT / Firewall Traversals ‐ ICE

• ICE provide a way to get media between two devices that are both behind NATs and some firewalls • It also forms a way to detect changing network conditions and switch from an interface such as WIFI to a different interface such as LTE • Finally it is used for media consent to make sure unwanted traffic is not sent to devices

Cbii f l ICE R Echo Relay Peer Server Server

1) Gather Address P:100 private N 2) Try all of N:200 from Echo R:300 from Relay P:100, N:200, R:300

P 3) Check connectivity

4) Choose Use N:200 Web Server

SRTP Video Phone Web Browser Web Browser RTP Data Base Web Server

Do you want to talk Yes

SRTP Web Browser Signaling ‐ SDP

• The SDP offer answer protocol used by SIP is used for media negotiation • Rich interface to describe what codecs, network transports, and media options one side can support (the offer) and which ones the other sides wants to select (the answer)

v=0 o=Alice 292742730 29277831 IN IP4 131.163.72.4 s=lecture c=IN IP4 131.164.74.2 t=0 0 m=video 52886 RTP/AVP 31 a=rtpmap:31 H261/90000 a=content:slides m=video 53334 RTP/AVP 31 a=rtpmap:31 H261/90000 a=content:main Identity •Browser is configured with identity provider(s) for the user OAuth or 4 1. User “logs on” using protocol OpenID Connect downloaded from identity provider in JavaScript/HTML 2 33Calling Website 2.Browser get an assertion from identity provider which binds the JS App JS App DTLS fingerprint to the identity such as [email protected] Alice Browser DTLS/SRTP Media Bob Browser 1 3.The calling JavaScript passes the assertion to far side 4.Bob’s browser verifies the assertion with identity provider and check DTLS fingerprint matches the assertion 5.Browser display "secure to [email protected]" WebRTC: What’s inside? Audio & video technology in the browser

• Media capture – Device (mic & camera) selection – Driver interaction – High quality capture and rendering • Audio/video enhancements – Echo cancellation – Noise suppression • Coding (compression) • Network handling – Compensate for packet loss and jitter • Effects ‐ Web Audio API, , CSS, WebGL, etc Impact of IP Networks

• Delay – Major effect is “stepping on each other’s talk” – Usage scenario affects annoyance factor • Higher delay can be tolerated for mobile devices • Packet loss – Smooth concealment necessary • Network jitter – Jitter buffer necessary to ensure continuous playout – Trade‐off between delay and quality Impact of Delay

Data from ITU-T G.114 4

3

2 Mean Opinion ScoreMean Opinion

1 0 250 500 750 One-w ay transmission time [ms]

• ITU‐T (G.114) recommends: – Less than 150 ms one‐way audiodelay for most applications (up to 400 ms acceptable in special cases) • Users have got used to longer delays – Still, low delay very important for high quality • Audio and video in sync is preferred (may delay audio) Choosing the right

• Many conflicting parameters affect choice of codec

Complexity Packet-loss robustness

Delay Memory

Codec Input robustness Bit-rate Sampling rate/ Quality • Interoperability Frame rate – At least one Mandatory to Implement (MTI) codec • IPR issues a significant concern • Hardware support (especially for video codecs) Audio Spectrum vs. Speech Quality

Speech Quality

Super CD Wideband Wideband Speech Speech Speech

Narrowband Speech (PSTN) Frequency 4 kHz 8 kHz10 kHz 16 kHz 22.1 kHz Codec selections for WebRTC

• Audio –Narrowband audio & legacy interop: G.711 –Wideband adaptive audio: –Browsers are likely to support other codecs as well

• Video –Huge debate over VP8 vs. H.264 AVC –Primary issue is Royalty Free status of IPR Opus

• State‐of‐the‐art royalty free codec • Standardized in the IETF – RFC 6716 • Bit rates from – From 6 kb/s (speech) – To 512 kb/s (audio) • Low latency – As low as 5 ms possible – 22.5 ms typical • Narrow‐ AND wide‐band Quality of Service (QoS)

• Based on Differentiated Service Code Point markings set on media packets – JS Application can provide hints about relative priority of media streams – Browser knows media type of packets – Browser sets the DSCP appropriately – Network may take DSCP into account when prioritizing packets Congestion Control & Rate Adaptation

• Variety of algorithms combined: – Losing too many packets, slow down – Not losing many packets, speed up – Packet delay starts going up, slow down – If up shifted, then promptly downshifted, wait awhile for next upshift

• Goals: – Be “fair” with TCP ‐ i.e.. don’t push TCP traffic to floor and don’t be pushed to floor by TCP – Minimize latency – React to changing network conditions quickly – Provide a consistent flow of data Standards: WebRTC and RTCWeb IETF & W3C

• Main IETF work is done in the • W3C work is done in WebRTC RTCWeb working group working group • IETF is defining the over wire • Key documents are: protocol – http://dev.w3.org/2011/webrtc/editor/webrtc .html • Mostly these are selected – http://dev.w3.org/2011/webrtc/editor/getuse rmeda.html from existing protocols – ://dvcs.w3.org/hg/dap/raw‐ • Key documents are file/tip/media‐stream‐capture/scenarios.html – draft‐ietf‐rtcweb‐data‐channel – draft‐ietf‐rtcweb‐jsep – draft‐ietf‐rtcweb‐overview – draft‐ietf‐rtcweb‐rtp‐usage – draft‐ietf‐rtcweb‐security‐arch – draft‐ietf‐rtcweb‐security – draft‐ietf‐rtcweb‐use‐cases‐and‐ requirements Implementations

• ‐ Firefox • Apple ‐ – Actively working on – Maintaining strict secrecy implementation • ‐ IE – Has shipped early version – Very active in contributing to developers can play with standards – Expect significant working – Released a plugin that can provide functionality in 2013 limited functionality via polyfill • Google ‐ Chrome – Actively working on implementation – Has shipped early version developers can play with – Expect significant working functionality in 2013 Digging Deeper

• Read the specification at : http://www.w3.org/2011/04/webrtc/ http://tools.ietf.org/wg/rtcweb/

• Read the book: –"WebRTC: and RTCWEB Protocols of the HTML5 Real‐Time Web" by Alan Johnston and Daniel Burnett (See http://webrtcbook.com/ )

• Join the community mailing lists: –W3C: Send email with "subscribe" to public‐webrtc‐ [email protected] –IETF: https://www.ietf.org/mailman/listinfo/rtcweb Credits and Usage

• If you want to use these slides, please contact us at [email protected] or [email protected] and we will be happy to get you some slides you can use • Thanks to many people for contributions to these slides including Ethan Hugg, Suhas Nandakumar, Darin Dunlap, Eric Rescorla, and Justin Uberti