Media Gateway Transfer Analysis
Total Page:16
File Type:pdf, Size:1020Kb
EXAMENSARBETE INOM TEKNIK, GRUNDNIVÅ, 15 HP STOCKHOLM, SVERIGE 2020 Media Gateway Transfer Analysis ARON HANSEN BERGGREN KTH SKOLAN FÖR ELEKTROTEKNIK OCH DATAVETENSKAP Abstract There are many protocols available to transfer files where some are faster than others. With ever increasing network speeds and amounts of media needing to be transferred, the need to minimize the amount of data sent is increasing. The amount of available protocols which makes this possible is also increasing, making it harder and harder for teams to decide which one to implement in their particular case. This thesis aims to give a overview on the most suitable protocols for direct media transfer over short to long distance WAN connections. In order to come to any conclusions, a lot of theoretical work on transfer solutions is made to make the comparison only describe useable solutions with reasoning as to why they make the cut for this type of application. Then, a handful of suitable solutions were tested on geographically distributed virtual machines in order to give realistic net- work conditions. The result is that the choice matters on the needs of the organisation, as the commercially available solutions are in general superior to the freely ones not only in speed, but in support and documentation. However, the open source solutions to perform very well for being free to use. In order to say which solution is the ultimate one, a lot more resources would be needed to complete wan transfers in excess of 10Gb/s speeds over many more network conditions. The results quickly showed that the underlying protocol might not be the determining factor for speed. Instead, the use of efficient multi-stream transfers is what actually makes a difference. 2 Sammanfattning Det finns många olika protokoll tillgängliga för att föra över filer och med det sagt är vissa snabbare än andra. Med kontinuerligt ökande nätverkshastigheter och större mängder media som behöver överföras, så gör även behovet att minimera mängden data. Mängden tillgängliga överföringspro- tokoll vilket möjliggör detta fortsätter också att öka, vilket gör det svårare och svårare att för team att bestämma vilket protokoll de ska implementera gällande deras behov. Denna uppsats syftar till att ge en överblick över lämpade protokoll för direktöverföring av media över både långa och korta geografiska avstånd över WAN anslutningar. För att kunna komma till slutsatser i denna rapport, har en stor andel teoretiskt arbete inom överföringsprotokoll genomförts för att reducera mängden kandidater till dem som är lämpliga för denna typ av överföring. Vidare har den resterande mäng- den protokoll testats mellan geografiskt skilda virtuella maskiner för att återspegla en realistisk bild av hur de presterar. Resultatet är blandat, det beror mycket på vad som krävs i varje fall. Kommersiella lösningar presterar bäst, både i prestanda, dokumentation och support. Med det sagt så presterar de med öppen källkod väldigt väl med, framför allt med fördelen av att inte kosta något att använda. Däremot kommer vi inte fram till någon ultimat lösning, då en mycket större mängd resurser behövs för att genomföra längre experiment över ännu högra hastigheter än 10Gb/s över ännu fler nätverksförhållanden. Slutsatsen av experimenten visar att det underliggande pro- tokollet inte spelar särskilt stor roll för den slutgiltiga hastigheten. Det som istället spelar roll är valet av ett protokoll som har multi-stream möjligheter som avgör hur snabbt du kan gå. Authors Aron Hansen Berggren [email protected] Information and Communication Technology KTH Royal Institute of Technology Examiner Peter Sjödin Kista KTH Royal Institute of Technology Supervisor Markus Hidell Kista KTH Royal Institute of Technology Contents 1 Introduction 7 1.1 Background........................................7 1.2 Problem..........................................8 1.3 Purpose..........................................8 1.4 Commissioned Work...................................9 1.5 Target Audience.....................................9 1.6 Ethics and Sustainability................................9 1.7 Methods..........................................9 1.7.1 Literature Studies................................9 1.7.2 Testing and Benchmarking...........................9 1.8 Delimitations....................................... 10 1.9 Disposition........................................ 10 2 Background 11 2.1 Concepts......................................... 11 2.1.1 Protocol...................................... 11 2.1.2 Buckets...................................... 11 2.1.3 Hybrid Cloud Solution.............................. 11 2.1.4 Open Systems Interconnection (OSI) Model.................. 11 2.1.5 REST and Application Interfaces........................ 12 2.1.6 Network Congestion............................... 12 2.1.7 User and Kernel space.............................. 12 2.1.8 TCP........................................ 13 2.1.9 UDP........................................ 13 2.1.10 Multistream.................................... 13 2.1.11 Firewall...................................... 13 2.1.12 Encryption.................................... 13 2.1.13 Metadata..................................... 14 2.2 Iconik Storage Gateways (ISG) current setup..................... 14 2.3 Candidates........................................ 15 2.3.1 HTTPS...................................... 15 2.3.2 FTPS....................................... 15 2.3.3 SFTP....................................... 15 2.3.4 SCP........................................ 16 2.3.5 WDT....................................... 16 2.3.6 QUIC....................................... 16 2.3.7 PA-UDP...................................... 16 2.3.8 UDT........................................ 17 2.3.9 UFTP....................................... 17 2.3.10 FileCatalyst Direct................................ 17 2.3.11 TIXstream.................................... 17 2.3.12 FASP....................................... 18 5 6 CONTENTS 3 Approach 19 3.1 Literature Studies.................................... 19 3.2 Test Runs......................................... 19 3.3 Final Candidates..................................... 19 3.3.1 Experimental................................... 20 3.3.2 Speed Matters.................................. 20 3.3.3 Ports Matters................................... 20 3.3.4 Support Matters................................. 20 3.3.5 Disqualified Protocols.............................. 20 3.3.6 Final Protocols Investigated........................... 21 4 Test Environment 23 4.1 Test Suite......................................... 23 4.1.1 HTTPS...................................... 23 4.1.2 SCP........................................ 23 4.1.3 SFTP....................................... 24 4.1.4 UFTP....................................... 24 4.1.5 WDT....................................... 24 4.1.6 TIXStream MFT................................. 24 4.1.7 Notes....................................... 24 5 Performance Evaluation 25 5.1 Single Stream Results.................................. 25 5.2 Multistream Results................................... 26 5.3 SCP............................................ 26 5.4 SFTP........................................... 26 5.5 UFTP........................................... 27 5.6 WDT........................................... 27 5.7 TIXStream........................................ 27 6 Discussions 29 6.1 Experiment Discussion.................................. 29 6.2 Data Discussion...................................... 30 7 Conclusions and Further Work 31 7.1 Conclusion........................................ 31 7.2 Future Work....................................... 32 References 33 Chapter 1 Introduction 1.1 Background There are many ways to transfer files where some ways are faster than others. The Iconik Stor- age Gateway (ISG) allow users to have local access points to their media and reach it from any location, even if the remote gateway is on another continent. The gateways can also transcode media to different qualities and resolutions, analyze a set of storages for media, index them based on its attributes and create cloud assets corresponding to the media. It takes these orders from the Iconik service running in the cloud[1]. If a users available ISG do not possess media the user wants to work on, the user can request that media. This will cause the ISG which do have access to that media to upload it to the cloud, and then trigger the ISG available to the user to download it. This is very inefficient, as it introduces delay and unnecessary workloads on servers and networks. Both customers and product owners at Iconik.io are asking for a better solution, but the resources to perform a field study on the topic to find which transfer solution(s) are suitable for such a task have not been available. Having this solution in place would reduce the amount of traffic generated by the gateways over WAN, especially on local networks where the gateways would still have to go through the cloud instead of staying on the local network. The time difference between sending media directly and using a intermediate cloud is significant, as the entire project needs to be transferred on to a intermediate cloud before the destination gateway can download it from there. The purpose of the ISGs is to index files and synchronize this online and make them available once requested by a remote site, to benefit from the hybrid cloud approach. The requirement of the intermediate cloud as storage medium