A Deep Deterministic Policy Gradient Based Network Scheduler For Deadline-Driven Data Transfers Gaurav R. Ghosal1, Dipak Ghosal2,3, Alex Sim3, Aditya V. Thakur2, and Kesheng Wu3 1University of California, Berkeley, CA, U.S.A. 2University of California, Davis, CA, U.S.A. 3Lawrence Berkeley National Laboratory, Berkeley, CA, U.S.A. Abstract—We consider data sources connected to a software amount of data that is generated by the instruments, it is also defined network (SDN) with heterogeneous link access rates. important that the interconnecting network is run at a high Deadline-driven data transfer requests are made to a centralized utilization [3]. network controller that schedules pacing rates of sources and meeting the request deadline has a pre-assigned value. The Next generation science workflows will require complex goal of the scheduler is to maximize the aggregate value. We processing pipelines that may be processed at different HPC design a scheduler (RL-Agent) based on Deep Deterministic facilities [4]. More and more these workflows must be com- Policy Gradient (DDPG). We compare our approach with three pleted within a deadline. For example, the Large Synoptic Sur- heuristics: (i) P FAIR, which shares the bottleneck capacity in vey Telescope (LSST) [5] will take more than 800 panoramic proportion to the access rates, (ii) V D Ratio, which prioritizes flows with high value-to-demand ratio, and (iii) V B EDF, images each night with its 3.2 billion-pixel camera, recording which prioritizes flows with high value-to-deadline ratio. For the entire visible sky twice each week. The new data will be equally valued requests and homogeneous access rates, P FAIR compared with previous images to detect changes in bright- is the same as an idealized TCP algorithm, while V B EDF ness and position of objects as big as distant galaxy clusters and V D Ratio reduce to the Earliest Deadline First (EDF) and as small as near-by asteroids. The processing of each 30 and the Shortest Job First (SJF) algorithms, respectively. In this scenario, we show that RL-Agent performs significantly second image must be completed within 7.5 seconds so that better than P FAIR and V D Ratio and matches and in over- necessary alerts can be generated for down-stream processing. loaded scenarios out-performs V B EDF. When access rates are The deadlines in processing the flows impose deadlines in the heterogeneous, we show that the RL-Agent performs as well data transfer from the instrument facilities (that are typically as V B EDF even though the RL-Agent has no knowledge of in remote locations) to the HPC facilities. Furthermore, the the heterogeneity to start with. For the value maximization problems, we show that the RL-Agent out-performs the heuristics data transfer deadlines are becoming increasingly tight. For for both homogeneous and heterogeneous access networks. For example, in the Linac Coherent Light Source (LCLS) [6], the the general case of heterogeneity with different values, the RL- location of next image is a function of the current image. Agent performs the best despite having no prior knowledge of Deadline driven data transfer is not unique to large scien- the heterogeneity and the values, whereas the heuristics have full tific workflows and is becoming important for cloud service knowledge of the heterogeneity and V D Ratio and V B EDF have partial knowledge of the values through the ratios of value providers with geo-distributed data centers. In order to keep to demand and value to deadline, respectively. the distributed locations synchronized, large data sets must be Index Terms—Deadline-driven data transfers, Software- periodically transferred within a deadline [7], [8] while requir- defined Networking (SDN), Reinforcement Learning, DDPG, ing that the network is operating at a very high utilization [3]. Scheduling heuristics, EDF, TCP, Value maximization This requires fine grain control of the transmission rate to eliminate contention and resulting packet losses. I. INTRODUCTION To support deadline-driven data transfers, ESnet sets up on- Large distributed science is moving to the SuperFacility demand circuits [9] supporting packet priority allowing the model [1] in which distributed instrument facilities, HPC circuit to be used by other traffic when the deadline flow is systems, storage, and researchers are viewed as one integrated inactive. For cloud service providers, the deadline transfers are facility. This is achieved by interconnecting these resources scheduled over private/dedicated WANs that interconnect the using a very high-speed network, such as the ESnet [2], that geo-distributed data centers. These private dedicated WANs must provide performance guarantees on the data transfers are less noisy and more predictable compared to the gen- required by complex science workflows deployed on the eral Internet. Nevertheless, the deadline transfers must be SuperFacility. Furthermore, due to exponential growth in the scheduled in the presence of other background and interactive flows [7], [8]. ISBN 978-3-903176-28-7© 2020 IFIP In this paper, we consider an SDN-enabled pri- 253 vate/dedicated WAN with a centralized controller. There is rates of the last-hop links from R2 to the destination nodes also in-network telemetry (INT) that can provide real-time Di are greater than B and, hence, are never the bottleneck. fine-grained information about network state including router buffer lengths and packet drops [10]. Deadline-aware data transfer requests are made to a central network controller, which schedules the flows by setting pacing rates at the sources of the deadline flows. This work extends the Q-table based network scheduler pre- sented in [11], which demonstrated the feasibility of using a Reinforcement Learning based approach to schedule deadline- driven flows. In particular, the Q-table based approach cannot be applied to large networks with heterogeneous links. In this paper, we design and implement a network scheduler for deadline-driven flows based on Deep Deterministic Policy Gradient (DDPG) and an Actor-Critic model, which is able to handle larger networks with heterogeneous links. Furthermore, this paper tackles a generalization of the scheduling problem that assigns a utility value to each flow if the deadline is met. The scheduler task is to schedule the flows so as to Fig. 1: A dumbbell network with heterogeneous access link maximize the aggregated utility value. We design a scheduler rates. The thicknesses of the links represent different rates. (RL-Agent) based on DDPG and compare its performance Note that the link rates from Router R2 to destination nodes with three heuristics: (i) P FAIR, which shares the bottleneck are same as the link between routers R1 and R2 and hence capacity in proportion to the access rates, (ii) V D Ratio, are never the bottleneck. which prioritizes flows with high value-to-demand ratio, and (iii) V B EDF, which prioritizes flows with high value-to- Requests for file transfers are made to the network con- deadline ratio. troller. Each request j is a five-tuple (sn ; dn ; f ; d ; v ) The main contributions of this paper are: j j j j j where snj denotes the source node, dnj the destination node, 1) We propose a DDPG based network controller which fj is the filesize, dj is the deadline, and vj is a value awarded a) scales with respect to the size of the network (number if the data transfer is completed within the deadline. Each of sources and the bottleneck link capacity), b) can be request corresponds to a single TCP/IP flow. We assume only applied to heterogeneous networks, and c) can be applied a single request from each source at any given time. to a general value maximization problem. The basic unit of time is a scheduling interval; the network 2) For equally valued requests and homogeneous access controller assigns a pacing rate to each active source at the rates, we show that the RL-Agent performs as well as start of each scheduling interval. We consider an episodic and in some cases out-performs V B EDF (which is the (batch) model in which the scheduler receives a request from same as the Earliest Deadline First (EDF) algorithm) and each of the sources at the beginning of an episode [11]. significantly better than P FAIR (same as an idealized Each request is to a different destination. The episode ends TCP algorithm). when all the requests have completed successfully (within the 3) For equally valued requests and heterogeneous access deadline) or not, after which a new episode begins. An episode rates, we show that the RL-Agent performs as well consists of an integer number of scheduling intervals denoted as V B EDF even though it has no knowledge of the as 1;:::;T . heterogeneity to start with. At the beginning of each scheduling interval, for each 4) For the value maximization problem, we show that the request j, we define Rminj(t) as [11]: RL-Agent out-performs the heuristics for both homoge- remaining file size neous and heterogeneous access networks. This is despite Rminj(t) = (1) the fact that the RL-Agent has no prior knowledge of the time until deadline heterogeneity and the values, whereas the heuristics have The value Rminj(t) denotes the minimum rate that is required full knowledge of the heterogeneity and V D Ratio and at every subsequent scheduling interval for request j to meet fj its deadline. We let Rminj(1) = , the initial minimum rate V B EDF have partial knowledge of the values. dj for request j, be denoted by Rminj. II. SYSTEM MODEL Pn We use Sum Rmin = j=1 Rminj to denote the sum of the In this study we consider a heterogeneous dumbbell net- initial Rmin of each request.