Transfer Protocols for Digital Media Workflows Contents 2 Introduction Introduction to Protocols Introduction to Protocols 2 1 User Datagram Protocol (UDP) 3 With the many advances that have been made in digital media file formats, compression, and distribution – the RUDP 6 technology for the transfer of rich media has remained relatively unchanged for over 40 years. UDPS 6 New lightweight codecs and file choices are regularly introduced to provide content viewing and a downloadable proxy for production. However, with a growing influx of content makers looking to maintain the value of a File Transfer Protocol (FTP) 6 2 large, complex master file for productions, the need to transfer, share and move these large files has various FTPS 8 challenges in speed, bandwidth and security. SFTP 8 Here we take a look at some of the transfer building blocks commonly used throughout the video industry. For the purpose of this document we are going to focus on the three main protocols in practice today – UDP, Hypertext Transfer Protocol (HTTP) 9 3 FTP and HTTP. HTTP 1.1 11 4 Key Considerations 12 11 Transfer Speed Times 13 The Future 14 5 About Aframe 15 Share this... Transfer Protocols for Digital Media Workflows Transfer Protocols for Digital Media Workflows 1 UDP User Datagram Protocol 4 UDP User Datagram Protocol (UDP) Essentials • Typically used for streaming media assets over IP. • UDP will optimize your transfer rates for large data sets when there is high latency or packet loss over the network. Background Developed and released by David P Reed in 1980, thanks to its fast transfer rates and capability of handling large files, UDP is quite often the method to be leveraged for video conferencing and video streaming needs. Compared to some standard Transmission Control Protocol (TCP) communications, it offers a simple lightweight messaging format, where the stream of data is lacking formality. UDP is commonly leveraged along with compression algorithms for improved speed with video file transfers and in local storage systems for speed of access and playback from a Storage Area Network (SAN). The principle advantage of UDP is speed – particularly when transferring large files across wide geographic expanses. The protocol allows for the sending of multiple data streams in parallel, and does not require acknowledgement from the recipient in order to progress. Compare this with TCP-based protocols such as FTP, where each packet requires acknowledgement before the next is sent – and factor in the hundreds, if not thousands of miles of network infrastructure these requests are sent back and forth across - and you can begin to see the advantages of UDP for file transfer. Share this... Transfer Protocols for Digital Media Workflows Part 1 5 UDP These advantages are further emphasized in the world of media, where files are typically many times larger than text-based documents, or even image-based documents. The fact that video production is increasingly becoming a geographically dispersed environment, and the implications this has on collaboration and access for project stakeholders, add further weight to the argument in favor of making UDP the de facto file transfer protocol of the media and entertainment industry. Did you know ? Can you depend on it? Because of its unprecedented speed, The transmissions consist of a header, the datagram for the message and a checksum (optional) for protection against tampering or corruption. The datagram can handle payloads up to 65,535 bytes, (8 of which are for UDP is the preferred the header information). transfer protocol of the London Stock Exchange In the past, UDP has been called out as unreliable due to its non-transactional method for packet checking and the optional use of check sums on the messages in Internet Protocol version 4 networking. The datagram is sent without checking to see if the server side host is ready to receive. It is a stateless protocol, where a socket is created from the client to the server and messages are sent as quickly as the bandwidth of the network connection will allow. There is no overriding control as to which packets in the transfer are sent first. This process can drop packets, which can be lost without application controls. Share this... Transfer Protocols for Digital Media Workflows 6 UDP RUDP - Reliability User Datagram Protocol To answer concerns about UDP’s reliability, RUDP (conceived by Bell Labs), has been adopted by some networking and application companies to specifically address the following concerns: • Acknowledgment of received packets. • Flow control. • Retransmission of lost packets. • Over buffering – or compressing for faster than real time transmissions. UDPS - Extension UDPS is an extension to the UDP standard, where client side applications request that the UDP session be encrypted. UDPS relies on Secure Socket Layer (SSL) and a Transmission Layer Security (TLS) for long public and unrelated session keys, which encrypts the transmission or message for the length of the transmission session. Share this... Transfer Protocols for Digital Media Workflows Transfer Protocols for Digital Media Workflows 2 FTP File Transfer Protocol 8 FTP File Transfer Protocol (FTP) Essentials • FTP may run in active or passive mode, which determines how the data connection is established. • Typically used for point-to-point media movement. • Fast and compatible with most networks. Background File Transfer Protocol, or more commonly FTP, was developed in 1971 by Abhay Bhushan. As a pre–Internet protocol, FTP continues to be the primary way that users talk about digital file transfers. It was originally used for moving large text files from point to point with a simple username and password or anonymous logins for access to publically shared files - FTP was not designed with file sizes in the high gigabytes or even terabytes in mind. This makes something of a square peg for the video industry, where file sizes continue to rise exponentially. As is the case with each protocol discussed here, there will always a be a handoff to TCP at the end of the transfer process. Based on how that is configured on the network it can have an impact on the transfers regarding what’s allowed to pass and not pass through customer network. An example being a TCP policy that doesn’t allow encrypted transmissions on the network. Share this... Transfer Protocols for Digital Media Workflows Part 2 9 FTP Can you depend on it? FTP inherits TCP’s robustness and is generally very reliable for transferring files. Security and the ability to resist brute force attacks (like DOS) on FTP servers, has been a key point for both vendors and customers to examine because of the impact on performance within a datacenter or FTP server. Adding security procedures within the client application and server responses can mitigate these threats. Did you know ? FTPS - Extension TCP guarantees the This increasingly popular protocol version handles client side application requests that the FTP session be delivery of data between encrypted. The server side has the option of allowing or denying connections that do not request TLS. This two hosts and also boasts method uses other ports as part of the encryption for the data transfer. that packets are delivered in the same order in which they were sent SFTP - Secure File Transfer Protocol This uses the secure shell to transfer files, offering encryption for both commands and data, as well as preventing passwords and sensitive information from being transmitted openly over the network. However, SFTP does not interoperate well with common FTP software. This makes collaboration and setup of sharing pools more difficult to administer and run with a wide set of partners. Share this... Transfer Protocols for Digital Media Workflows Transfer Protocols for Digital Media Workflows 3 HTTP Hypertext Transfer Protocol 11 HTTP Hypertext Transfer Protocol (HTTP) Essentials • HTTP is the foundation of data communication for the World Wide Web. • The protocol defines how messages are formatted and transmitted. • HTTP is slower in comparison to FTP & UDP but more likely to work on all networks. Background This protocol is the basis for communications on the World Wide Web and was developed in 1989 by Sir Tim Berners-Lee et al. The function is based on client request and server response for text or other media depending on the application. Unsurprisingly, as the language of the Internet, it’s also the most commonly leveraged transfer communication. HTTP provides logging, caching of repeated content and has been leveraged to provide persistent connections for multiple transfers in the recent improvements to the standard. As with FTP and UDP, it can also utilize acceleration “over buffering” with applications designed for that purpose, but can be seen as “best effort” based upon the request-response and stateless architecture. Its chatty in that regard and there is a larger overhead for instructions compared with FTP and UDP. Share this... Transfer Protocols for Digital Media Workflows Part 3 12 HTTP Can you depend on it? As the use of the Internet has grown exponentially for the consumer, business and enterprise, so have the number of security attacks against web servers. Media applications that require logins, encrypt the transfer Did you know sessions and have unique identifiers (user and application cookies) are more successful in keeping legitimate ? transfers and transactions to take place between the client and server. HTTP is called a stateless protocol because each HTTP 1.1 command is executed independently, without any The latest version of Hypertext Transfer Protocol (HTTP 1.1) has streamlined some of the archaic processes knowledge of the commands of previous models. It now provides a persistent connection that allows multiple requests to be batched or that came before it pipelined to an output buffer. The Transmission Control Protocol layer can put multiple requests into one TCP segment that gets forwarded to the Internet Protocol layer for packet transmission.