HTTP/WebDAV synchronization protocol optimizations. Piotr Mrowczynski HTTP/WebDAV synchronization protocol optimizations. - HTTP2 (https://github.com/owncloud/client/compare/http2) - Bundling Scope of (https://github.com/owncloud/client/pull/5319) this talk - Request Scheduling (https://github.com/owncloud/client/pull/5440) - Dynamic Chunking for new chunking algorithm (https://github.com/owncloud/client/pull/5368) - Prioritize by modification time (https://github.com/owncloud/client/pull/5349) Current ownCloud WebDAV / HTTP1.1 implementation - HTTP/1.1 without pipelining – head of line blocking - HTTP/1.1 without pipelining – head of line blocking - max 3-6 parallel connections (as in web) - each file is single PUT / GET / DELETE / MKDIR / MOVE request within a single persistent (Keep-Alive Header) connection e m i ... T L CO UT UT MK P P Con. 1 Con. 2 Con. 6 Server can handle 100 requests in parallel at specific moment → client will use anyways max 6 - server can handle 100 requests in parallel at specific moment → client will bind to max 6 - server with concurrent syncs is overloaded with x6 opened connections (usually SSL) wastetime on latency (usually 15-300 ms) - Latency: Each fileinseparate connection has to with openedconnections - server with many concurrentsyncs is overloaded specificmoment →client bind will tomax 6 - server can handle100 requests parallel in at 1000 files / 6 parallel = 167 lines of blocking → ofblocking lines = 167 /files 6parallel 1000 Time L a t e n c L y a a t n e d n c t y ms = x 320 ms =x 53 r a n s f e r 9s Server-side Operations 53s HTTP1, HTTP2 and BUNDLING HTTP/1.1 e m i T MKCOL Con. 1 ... PUT Con. 2 PUT Con. 6 BUNDLING e m i T Reduced latency gain MKCOL Con. 1 ... BUNDLE Con. 2 BUNDLE HTTP/1.1 Con. 6 e m i T MKCOL Con. 1 ... PUT Con. 2 PUT Con. 6 HTTP 2 with ownCloud's requests limitation BUNDLING e e m i m i T Bandwidth/Time T Max Parallel Gap Reduced latency Max Parallel { gain MKCOL Con. 1 ... BUNDLE Con. 2 PUT Con. 1 BUNDLE Optimization target – HTTP/1.1 Con. 6 scheduling e (will talk later) m request i T MKCOL Con. 1 ... PUT Con. 2 PUT Con. 6 HTTP2 with ownCloud request limitation If optimized by pumping more requests or binary data transfers: - might hide request-response latency as in bundling - might utilize bandwidth - only limited by server/database capability of accepting parallel files HTTP2 - possible benefits for ownCloud - The parallel multiplexed requests and response do not block each other. HTTP2 - possible benefits for ownCloud - The parallel multiplexed requests and response do not block each other. - Optimized and faster encryption HTTP2 - possible benefits for ownCloud - The parallel multiplexed requests and response do not block each other. - Optimized and faster encryption - Binary framing – less errors, overhead and more - Header compression HTTP2 - possible benefits for ownCloud - The parallel multiplexed requests and response do not block each other. - Optimized and faster encryption - Binary framing – less errors, overhead and more - Header compression - Flow control (separate from TCP flow control) BUNDLING - possible benefits for ownCloud - Files packed in group of requests, send over the network and single response is returned - Above results in latency reduction and possible better network utilization BUNDLING - possible benefits for ownCloud - Files packed in group of requests, send over the network and single response is returned - Above results in latency reduction and possible better network utilization - Reduces PHP overhead (script is fired up once for whole the group instead of per file) - PHP overhead – can be also reduced by optimizing server side for single requests HTTP1 vs HTTP2 tests HTTP1 vs HTTP2 tests Files in parallel limitation 1000 files 1kB – total 1MB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Berlin, Germany SSD, 8GB RAM, CERNBOX bit/s 79 M 4x2,4GHz, Dow Geneva, Switzerland bit/s, pl 76M ncy, U s late 53 m Melbourne, Australia WiFi, Openstack, 12GB RAM, 4x2.5Ghz Ethernet, 320 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s HTTP1 vs HTTP2 tests Files in parallel limitation 1000 files 1kB – total 1MB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Synchronization to CERNBox (EOS), Geneva, Switzerland Protocol Parallel Limit Location Upload Time [s] Download Time [s] HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9 HTTP2 98.8 +/- 34.8 53.8 +/- 19.2 -17s HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1 HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?) -43s HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7 HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?) Latency and WAN impact HTTP1 vs HTTP2 tests Files in parallel limitation 1000 files 1kB – total 1MB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Synchronization to CERNBox (EOS), Geneva, Switzerland Protocol Parallel Limit Location Upload Time [s] Download Time [s] HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9 HTTP2 98.8 +/- 34.8 53.8 +/- 19.2 HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1 HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?) -170s HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7 HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?) Client still allows 1. Head of line blocking HTTP2 Pipelining only max 6 2. Server bookkeeping blocks upload connections HTTP1 vs HTTP2 tests Files in parallel limitation 1000 files 1kB – total 1MB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Synchronization to CERNBox (EOS), Geneva, Switzerland Protocol Parallel Limit Location Upload Time [s] Download Time [s] HTTP1 6 53 ms, DE 115.9 +/- 39.6 58.7 +/- 17.9 HTTP2 98.8 +/- 34.8 53.8 +/- 19.2 HTTP1 6 320 ms, AU 230.1 +/- 27.2 151.4 +/- 44.1 HTTP2 186.7 +/- 27.6 213.0 +/- 24.1 (?) HTTP1 100 320 ms, AU 209.8 +/- 21.0 129.4 +/- 21.7 HTTP2 39.6 +/- 11.7 42.6 +/ 9.2 (?) On latency 50ms from Berlin to Geneva, we got even 20s →50 Hz (files per second) HTTP1 vs HTTP2 tests SSL Overhead 12 files 1kB – total 12kB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Synchronization to CERNBox (EOS), Geneva, Switzerland Protocol Files Synced Location Upload Time [s] Download Time [s] HTTP1 6.7 +/- 1.4 5.0 +/- 1.3 12 320 ms, AU HTTP2 5.9 +/- 2.2 4.4 +/- 0.4 -1.8s 3-way-handshake and SSL optimization gain, connection reuse limit (only ?) HTTP1 vs BUNDLING tests HTTP1 vs BUNDLING tests 1000 files 1kB – total 1MB of data Measurement repeated 10 times using Smashbox Benchmarking Tool Berlin, Germany SSD, 8GB RAM, DAMKEN CLOUD bit/s 79 M 4x2,4GHz, Dow Nuremberg, Germany bit/s, pl 76M ncy, U s late 37 m Melbourne, Australia WiFi, Openstack, 12GB RAM, 4x2.5Ghz Ethernet, 279 ms latency, Upl 220 Mbit/s, Dow 1448 Mbit/s HTTP1 vs BUNDLING tests Files in parallel limitation 1000 files 1kB – total 1MB of data Measurement repeated 3 times using Smashbox Benchmarking Tool Synchronization to Damken Cloud, Nuremberg, Germany Protocol Bundled Files Location Upload Time [s] Download Time [s] HTTP1 - 155.1 +/- 0.4 41.0 +/- 0.8 37 ms, DE -6s Bundling 100 149.1 +/- 0.8 38.0 +/- 3.5 HTTP1 - 187.8 +/- 5.6 67.6 +/- 0.2 -28s Bundling 100 279 ms, AU 158.4 +/- 3.2 66.0 +/- 1.3 Bundling 10 152.6 +/- 1.2 67.9 +/- 0.1 Latency Influence on HTTP1 Bundling in this prototype works only for upload HTTP1 vs BUNDLING tests Files in parallel limitation 100 files 1kB – total 100kB of data 1000 files to bundle →10 requests needed 100 files to bundle→6 requests needed (as number of connections) It occurred that for 100 files sync time reduced: 20s → 16s in upload for 37ms latency Requests Scheduling “Wide and Narrow Pipe” problem with max 3-6 connections 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 2 MB/s available “Wide and Narrow Pipe” problem with max 3-6 connections 5 MB 5 MB 5 MB 2 MB/s available Better solution utilizing 6 connections 5 MB 5 MB 10 kB 10 kB 10 kB 10 kB 2 MB/s available Even better solution using HTTP/2 for fast and idle servers Boosted using http2 prioritization? (E. Bocchi, Politecnico di Torino) 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 5 MB 5 MB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 10 kB 2 MB/s available TU Berlin → TU Berlin Test Measurement repeated 10 times using Smashbox Benchmarking Tool Folder A Folder B 50 x 100kB 20 x 5MB TU Berlin → TU Berlin Test Old implementation Folder-wise, first folder with small, then folder with big TU Berlin → TU Berlin Test New implementation Cross-folder, big files and small files at the same time TU Berlin → TU Berlin Test Measurement repeated 10 times using Smashbox Benchmarking Tool 20 x 5MB 50 x 100kB - Big files don't block smaller ones - Small files don't block bigger ones Upload: ~23s → ~20s Download: ~16s → ~13s Future using HTTP2, Dynamic Chunking and Scheduling? - Max Parallel Negotiation Sending you 3 files Ok, but I can handle 50 files at this moment My dynamic chunk is now 20 MB.