Makespan Minimization of Time-Triggered Traffic on A

CZECH INSTITUTE OF INFORMATICS ROBOTICS AND CYBERNETICS

INDUSTRIAL INFORMATICS DEPARTMENT

Makespan minimization of Time-Triggered traffic on a TTEthernet network Jan Dvoˇrák,Martin Heller and ZdenˇekHanzálek arXiv:2006.16863v1 [cs.NI] 20 Jun 2020 DOI: https://doi.org/10.1109/WFCS.2017.7991955 Cite as: J. Dvoˇrák,M. Heller, and Z. Hanzálek.Makespan minimization of time-triggered traffic on a ttethernet network. In 2017 IEEE 13th International Workshop on Factory Communication Systems (WFCS), pages 1–10, 2017 Source code: https://github.com/CTU-IIG/TTEthernetScheduler

c 2017 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works. Makespan minimization of Time-Triggered traﬃc on a TTEthernet network

Jan Dvoˇr´ak Department of Control Engineering, FEE Czech Technical University in Prague Prague, Czech Republic Email: [email protected] Martin Heller ZdenˇekHanz´alek Department of Control Engineering, FEE DCE FEE and CIIRC Czech Technical University Czech Technical University in Prague Prague, Czech Republic Prague, Czech Republic Email: [email protected] Email: [email protected]

Abstract sized instances. The universality of the provided method allows easily modify or extend the prob- The reliability of the increasing number of modern lem statement according to particular industrial de- applications and systems strongly depends on in- mands. Finally, the studied concept of makespan terconnecting technology. Complex systems which minimization is justified through the concept of usually need to exchange, among other things, mul- scheduling with porosity according to the worst- timedia data together with safety-related informa- case delay analysis of Event-Triggered traffic. tion, as in the automotive or avionic industry, for example, make demands on both the high bandwidth and the deterministic behavior of the com- 1 Introduction munication. TTEthernet is a protocol that has been developed to face these requirements while This paper focuses on the problem of creating providing the generous bandwidth of Ethernet up schedules for Time-Triggered traffic on the TTEth- to 1 Gbit/s and enhancing its determinism by the ernet network. The objective is to minimize Time-Triggered message transmission which follows the makespan of the Time-Triggered traffic and, the predetermined schedule. Therefore, synthesiz- thus, maximize the continuous gap for the Event- ing a good schedule which meets all the real-time Triggered traffic. The aim of the paper is to de- requirements is essential for the performance of the velop the scheduling algorithm and, consequently, whole system. to compare the concept of the makespan minimiza- In this paper, we study the concept of creat- tion with the concept of the porosity optimiza- ing the communication schedules for the Time- tion proposed by Steiner in [1]. The TTEthernet Triggered traffic while minimizing its makespan. network is a network protocol which offers Time- The aim is to maximize the uninterrupted gap for Triggered communication and preserves the back- remaining traffic classes in each integration cycle. ward compatibility with Ethernet. The provided scheduling algorithm, based on the Ethernet (IEEE 802.3) has been the prevalent Resource-Constrained Project Scheduling Problem technology for home and office networks over the formulation and the load balancing heuristic, ob- last few decades. Thanks to its widespread adop- tains near-optimal (within 15% of non-tight lower tion, it has developed into a mature technology of- bound) solutions in 5 minutes even for industrial fering high bandwidth with cheap and readily avail-

2 ECU been a trend of using Ethernet-based networks for s industrial applications as well. This trend has be- kn,s come even more pronounced with the ever increasing amounts of data transfers necessary to facilitate k n n,r ECU features like real-time image processing and recog- ECU r nition or communication among individual units in km,n k t o,n a smart system. Therefore, extensions of Ethernet kt,m

1 are being developed to meet the demands of indus- 2

4 m km,o o trial applications. An overview of the development ko,p 1

2 k 3 of industrial networks is given by [4]. ECU l,m 4

1 1 2 TTEthernet is a promising extension of Eth- l 2 3 p 3 4 4 kp,u ernet, which provides determinism and fault- kp,q tolerance while being compatible with standard ECU ECU u q Ethernet. Besides the TTEthernet standard, there has been ongoing effort on the standardization Figure 1: Example of the TTEthernet network in- of extensions to Ethernet for scheduled traffic by frastructure with routing and scheduling of message the IEEE 802.1 Time-Sensitive Networking Task m1 from node l to node q Group. Determinism with the strictest guarantees is achieved through a fixed schedule for the traffic. able hardware. Therefore, synthesizing a good (exactly what this Conventional computer networks are mainly used means will be discussed later) schedule which meets for on-demand data transfer without an immediate all the requirements and deadlines is essential for physical impact on the real world. In case of failure, the performance of the network. Because TTEth- the transmission can usually be repeated without ernet allows for complex topologies, the scheduling causing major difficulties. Therefore, the focus is involves additional complexity compared to bus or on the bandwidth and efficiency with only moder- passive star topologies of networks like FlexRay [5] ate demands on reliability. or CAN. In contrast, in industrial applications various control loops of physical devices are realized by the 1.1 TTEthernet Overview network, e.g. data from sensors are transferred to a processing unit which then sends commands to TTEthernet (TT stands for Time-Triggered) is an actuators. Any disturbance can, thus, have imme- extension of Ethernet for deterministic communica- diate effects on the real world with possible severe tion developed as joint project among the Vienna consequences. As jitter (variance in transmission University of Technology [6], TTTech and Honey- times) is detrimental to the function of the control well, and standardized as SAE AS 6802 [7] in 2011. loops, determinism is often required. In addition, TTEthernet operates at Level 2 of the ISO/OSI individual devices in the network are often limited model, above the physical layer of Ethernet. It re- in hardware and need to operate in demanding en- quires a switched network with fully duplex phys- vironments. ical links, such as Fast Ethernet physical link Due to these differences, different technologies 100BASE-TX or Automotive Ethernet standard and protocols were traditionally used for conven- 100BASE-T1, so that unpredictable conflicts, while tional computer networks and industrial networks. accessing a shared medium, are avoided. An exam- Ethernet has been used for home and office net- ple of the TTEthernet infrastructure is depicted in works while various Fieldbus networks (such as Fig. 1. CAN [2] or Profinet [3]) have been used for indus- TTEthernet specifies a protocol for clock syn- trial applications. However, with the increasing in- chronization and the rules for managing the traffic tegration of industrial systems, increasing demands on the network. After an initial startup phase when on the volume of data transferred and also the ma- the clocks of the devices in the network are synchro- turing and development of the Ethernet, there has nized for the first time, the operation of TTEther-

3 makespan does not participate in the optimization process.

ic1 TT1 RC2 The schedule also provides temporal isolation 5 10 and enables fault tolerance. This is due to the critical gap ic2 TT2 TT3 TT4 fact that not only the sending of a frame is sched- 15 20 uled, but its reception is scheduled as well. If a TT ic 3 TT1 RC2 RC3 acceptance window 25 30 frame arrives outside the (i.e. Integration cycle Integration the time the frame is supposed to arrive consider- ic4 RC1 TT3 TT4 35 time [ms] 40 ing the synchronization inaccuracy), it is discarded by the receiver. This mechanism is called the tem- Figure 2: Example of communication on a link in poral firewall. one cluster cycle For traffic with less strict precision requirements, the RC traffic class can be used. This traffic class conforms to the ARINC 664p7 specification net is periodic. The clocks are being periodically [9] (also called AFDX). It offers greater flexibility synchronized to counter any possible clock drift because only the frame routing needs to be deter- when in steady operation. This period is called mined offline. The messages themselves are event- the integration cycle. The messages which follow a driven within some limitations. deterministic schedule are also periodic. The least The RC traffic represents the event-triggered common multiple of their period is called the clus- communication. Thus, it does not follow any sched- ter cycle. ule known in advance. It is organized in so-called TTEthernet integrates traffic of different time- virtual links. As stated in [10], a virtual link is an criticality levels into one physical network. There analogy to the ARINC 429 [11] single-source multi- are three traffic classes in TTEthernet. These drop bus. The virtual link determines the routing classes, ordered by decreasing priority, are Time- of the messages associated with it. Furthermore, Triggered (TT), Rate-Constrained (RC) and Best- there are two parameters: the maximum allowed Effort (BE) traffic. frame size and the bandwidth allocation gap, asso- The TT traffic class has the highest priority, and ciated with each virtual link. The bandwidth allo- sub-µs jitter can be achieved (depending on the net- cation gap represents the minimum allowed length work devices). The TT messages are periodic. We of an interval between consecutive frames on the assume that they are strictly periodic (i.e. no jitter virtual link. This effectively limits the bandwidth is allowed) in agreement with [8]. Their schedule of the virtual link. In return for this limitation, the is calculated offline and then loaded into the indi- maximum possible delay of any RC message can be vidual devices. The schedule is repetitive with a calculated offline. hyperperiod of the cluster cycle. Standard Ethernet traffic can also be transmit- A simple example of the TT traffic together with ted through the network. Even standard Ethernet the RC traffic on one direction of a physical link is devices, unaware of TTEthernet, can communicate presented in Fig. 2. In the figure, the integration through the network. Such traffic is called the Best- cycles are situated in rows and x-axis represents Effort (BE) traffic and has the lowest priority. the time instants in the particular integration cy- When the TT traffic is used together with other cle. The figure shows that messages TT1, TT3 and traffic classes, a TT frame could be delayed by an- TT4 have the same period (twice the integration other RC or BE frame. This happens when a TT cycle) and TT2 has another period (four times the frame arrives while an RC or BE frame is in trans- integration cycle). The length of the cluster cycle mission. Then, unless the transmission of the RC is equal to four times the length of the integra- or BE frame can be interrupted, the TT frame has tion cycle. The dark message at the beginning of to wait until the transmission is finished. There- each integration cycle is the synchronization mes- fore, a method of handling such situations, called sage. The synchronization message is used for pe- the traffic integration policy, is needed. There are riodical clock synchronization among nodes. This three policies for the integration of different traffic message represents the time allocated for the syn- classes as mentioned by [12]: Pre-emption, Shuf- chronization mechanism of the TTEthernet and it fling and Timely block. The Timely block integra-

4 tion policy, which causes no extra delay of the TT for scheduling the TTEthernet communication by traffic, is used in this paper. In this case, an RC TabuSearch was proposed in [12] and further ex- frame can only be transmitted if there is enough tended and more intensely evaluated in [16]. The time for the transmission of the entire frame before operators of the TabuSearch [17] algorithm take the the next TT frame is scheduled. If there is insuf- RC worst-case delay calculation into account and ficient time, the transmission of the RC frame is try to change the current schedule in such a way postponed until after the TT frame is transmitted. that all the real-time constraints imposed on the It additionally means, that the TT traffic follows RC traffic are satisfied. the schedule without any delays. Craciunas et al. took the given TTEthernet communication schedule into account while schedul- 1.2 Related works ing the tasks on the communication endpoints in [18]. They extended the work and introduced Many recent publications focus on the scheduling the combined network-level and task-level schedul- of Time-Triggered communication on the Ethernet ing in [19]. The CPU tasks are modeled in a very network. The most significant research in this area similar way to the network transmission tasks. A was done by Steiner et al. Their first paper [13] CPU is modeled as another physical link in the net- described the basic constraints for scheduling the work. A task running on this CPU is modeled as TT traffic. The scheduling problem was solved by a frame which needs to be transmitted over this the reformulation of the constraints to the SMT physical link in one direction, and its transmission (Satisfiability modulo theories) model. The native time is equal to the required CPU processing time. SMT formulation was used just for small cases with We have formulated our problem as the Re- up to 100 message instances. The second approach source Constrained Project Scheduling Problem presented allows one to find a schedule for bigger (RCPSP [20]), as will be described later, and its instances. It uses the SMT solver as a backend Multi-Mode version (MMRCPSP) [21]. RCPSP is for scheduling smaller groups of frames, into which already a well-studied optimization problem. The the frames are divided. After a group of frames survey for the problem and its related variants can is scheduled, the positions of the corresponding be found in [22]. The Multi-Mode version, where frame instances are fixed, and the next group is activities have several alternative modes with dif- scheduled iteratively. The paper does not consider ferent parameters, was studied by Schnell et al. other traffic classes, and the only criterion is to in [23]. The authors developed the exact algo- find any feasible schedule. The SMT formulation rithm by extending the SCIP Solver [24]. Added of the Time-Trigged message scheduling problem constraint handlers (i.e. functions for constraints has been adapted to 802.1Qbv [14] in 2016. In [1], propagation) allowed to directly cooperate between which builds on and extends [13], traffic from other a low-level constraint integer program solver and classes than just the TT class is considered. To re- high-level MMRCPSP constraints and objective. serve capacity for rate-constrained traffic, the con- Another exact approach is described in [25] where cept of schedule porosity, i.e. inserting blank slots the constraint-based modeling tool CP Optimizer reserved for the RC traffic into the schedule, is in- was used to solve the RCPSP problem. The ex- troduced. The pessimistic time analysis for the RC pressive power and universality of the CP Opti- traffic is proposed to evaluate the concept. Tamas- mizer allows one to easily extend or modify the Selicean et al. published a new method for cal- RCPSP model, which is used for justification of the culating the worst-case delay [8] which promises makespan and porosity optimization in the paper. much tighter estimates compared to the previous work. The improved precision comes, however, at 1.3 Paper outline and contribution the cost of much more expensive computation. As noted by [15], porosity scheduling has the disad- The paper is organized as follows: Section 2 de- vantage that gaps introduced at the beginning of scribes the studied problem of the TT message the scheduling process do not consider the profile scheduling in the TTEthernet network. In Sec- of the RC traffic. tion 3, the proposed method is described consisting The idea of using the metaheuristic approach of a message routing algorithm, a load-balancing

5 heuristic and an RCPSP based formulation of the words, all the periods ti has to be an integer multi- scheduling problem. The method is evaluated from ple of the length of integration cycle ic. The sched- the resulting schedule length point of view and also ule is so-called strictly periodic, which means that the RC traffic worst-case delay point of view in Sec- the next occurrence of message mi in a particular tion 4. Section 5 concludes the paper. link (further called message occurrence - see Fig. 3) The main contribution of the paper is the inves- appears in the schedule exactly ti time units af- tigation of a new concept for creating schedules of ter the current one. The positions of all message the TT traffic in the TTEthernet network so that, occurrences of message mi in the strictly periodic in opposite to the preceding studies, it focuses on schedule can be deduced from the position of the shortening the makespan (the latest completion of first message occurrence. any message transmission among all integration cy- The transmission time of message mi has to be cles and links - see Fig. 2) of the TT traffic instead smaller than or equal to the integration cycle and of introducing porosity (blank slots in the TT traf- its length ci does not exceed the maximal Ethernet fic schedules that are reserved for RC and BE traf- frame length of 1530 bytes. It would not be pos- fic). The makespan serves as a good measurement sible to send a synchronization message otherwise. for the schedule quality evaluation. The proposed Deadline di and release date ri are assumed to have idea has been inspired by the FlexRay communica- the value in the range 0 ri di ti. ≤ ≤ ≤ tion scheme, where the Time-Triggered and Event- Each node ei in the network has its identifier Triggered segments are separated. A further contri- assigned. The nodes are divided into two classes: bution is the novel formulation of the TTEthernet redistribution nodes and communication endpoints. scheduling problem as an RCPSP problem. Both The communication endpoints are nodes that gen- contributions are evaluated and discussed. erate or process the data (e.g. sensors, actuators, control units and other ECUs). Thus, the identifier of any communication endpoint can be assigned to 2 Problem statement message mi as transmitter qi or one of the receivers from set Qi. The redistribution nodes, on the other This paper aims to design a method for finding fea- side, are switches without any own data to trans- sible periodic schedules for Time-Triggered commu- mit and serve as intermediary nodes for communi- nication on the TTEthernet network such that the cation. maximal part of the remaining bandwidth can be The TTEthernet infrastructure consists of nodes preserved for RC and BE messages. and links which interconnect them. Each link ki,j m Each message i from a set of the TT messages from a set of links K connects two nodes ei and M has assigned: ej. This connection covers just one direction of the full-duplex communication. Therefore, two links ti - period • ki,j and kj,i model one physical link between nodes e e ci - message length in the number of bits con- i and j. These two links are independent from • sidering headers and interframe gap the scheduling point of view. A feasible schedule has to fulfill the following di - deadline hard constraints: • Completeness constraint: Each message mi ri - release date ∈ • M has to be scheduled. Contention-free constraint: Any link is capable qi - identifier of the transmitting node • of transferring at most one message at a time. Precedence constraint: A sequence of the links Qi - set of the receiving nodes q • Si = (kl,m, km,o, ..., kp,q) represents the path that The length of the resulting schedule is determined message mi has to go from transmitter qi = el by the cluster cycle cc (40 ms in Fig. 2). The length to receiver eq Qi through redistribution nodes ∈ of the integration cycle ic (10 ms in Fig. 2) is as- em, ..., ep. An example of such a message transmis- sumed to be the greatest common divisor of mes- sion is presented in Fig. 1 where communication

sage periods (i.e. ic = gcdi|mi∈M (ti)). In other endpoints are titled by ”ECU” and the redistri-

6 ko,m and BE traffic. This idea follows the practice from m km,o o the FlexRay bus or Profinet where the TT traffic km,l has its communication segment. The TT traffic can k 1 m1 ECU l,m 2 be scheduled at the beginning of the integration cy- 3 m l 1 m1 1 cle, and the remaining TT-free gap (coherent gap 2 4

3 m1 in integration cycle without the TT traffic) is pre- 4 served for the RC and BE traffic. The shortest TT- the same free gap among all links ki,j is denoted as a critical the same message instance message occurrence gap (see Fig. 2). Considering the constraints above, the goal of the scheduling is to find a schedule for Figure 3: Visualization of the difference between TT messages which maximizes the critical gap and, message occurrence and message instance thus, minimizes its makespan.

bution nodes has arrows painted on the top side. 3 Algorithm The front side of all nodes is labeled by its name. Only one direction of each physical link is labeled The algorithm proposed in the paper is divided into in Fig. 1 for the sake of simplicity. three stages. In the first stage (Sec. 3.1), the rout- The instance of message mi in link kl,m is de- ing of the messages is established. In the second l,m noted as message instance mi . Thus, all the stage (Sec. 3.2), the algorithm finds the assignment transmissions of some message mi in one partic- of the messages to the particular integration cy- ular link represents the same message instance. cles. The transmission times for each message oc- The message occurrence, on the other hand, rep- currence in each link are scheduled in the last stage resents all the transmissions of some message mi (Sec. 3.3). in one particular integration cycle. The difference between message instance and message occurrence 3.1 Determination of Time- is graphically explained in Fig. 3. The figure shows the detail view on the sub-segment of the network Triggered messages routing infrastructure from Fig. 1 with node ei, em and eo The network topology is often a tree in industrial only. The both links of any physical link are labeled networks. It means that there are no cycles and, here already. therefore, there exists only one possible path from q The path Si is not known in advance if the infras- a communication endpoint to any other endpoint. tructure does not have a tree topology. Therefore, Thus, the routing determination is trivial in such part of the optimization process is to find an appro- a case. However, the TTEthernet does not restrict priate path. The message transition from transmit- the network topology, according to the specifica- ting node qi to all the receivers from Qi has to be tion, to the tree. The cycles introduce new re- accomplished in one integration cycle. In the real dundant paths for messages. The redundant paths multi-hop networks, each hop introduces a tech- can relieve busy links and serve as a backup dur- nical delay caused by queuing in the ingress and ing a partial network malfunction. However, the egress port, etc. Such a delay in a switch is repre- TT messages have to know which path they are sented by parameter τ for TT messages. The value routed through in advance. Therefore, the first of τ can be in range from 1 µs to 2.4 µs according stage of the algorithm finds the routing. Each hop q to the network configuration [26]. Sequence Si en- the message has to pass implies the increase in over- tails the generalized precedence constraints because all traffic of the network. The long path also causes message mi has to be scheduled in link km,o τ time the prolongation of the end-to-end delay because units after it is scheduled in kl,m if kl,m precedes the message is delayed in the ingress and egress q km,o in Si . buffer of each redistribution node in the path. This The coherent TT traffic segment should be com- means that utilizing the shortest path algorithm pressed as much as possible to preserve the maxi- provides an efficient routing with the minimal num- mum part of the remaining bandwidth for the RC ber of hops. The network topology is transformed

7 to graph where edges represent the physical links same message varies among the links. The first con- and its weight is set to one. The Floyd-Warshall al- straint assures that the first message occurrence ap- gorithm [27] is used, consequently, to find a routing pears in exactly one of the possible integration cy- among all nodes in the network. All the messages cles. Thus, it satisfies the completeness constraint. follow the routing consequently. The second constraint makes the variable z to have Considering that the aim of this routing is to the value equal to or greater than the time needed minimize delay caused by the switch hoping, the to exchange all messages in any integration cycle of routing over the shortest path can cause unbal- any link in the network. The constraint is evaluated anced load among links. Thus, there are cases for each resource (each link and each integration where it is better to choose another routing strat- cycle in the cluster cycle) such that the transmis- egy. However, this default routing strategy can be sion time of all message occurrences assigned to the substituted by any other routing that follows de- particular resource is summed up. The resulting to- mands of the particular application. Other routing tal must be less than or equal to variable z. The strategies can be found for example in [28] or [29]. aim of the ILP model is to find such an assignment The routing defines the links in which the mes- that minimizes z. Thus, the maximal time needed sages are to be scheduled and specify the prece- for message exchange among all resources is mini- dence relations among message instances of each mized. The last two constraints force messages to message. be assigned to the integration cycle which can sat- isfy the release date and deadline constraints. 3.2 Integration cycle assignment The resulting assignment balances the load problem among the resources, follows the routing of the messages and satisfies the release date and deadline The used idea how to distribute messages among constraints. the integration cycles comes from the multiprocessor scheduling area. In the multiprocessor schedul- 3.3 Link schedule creation problem ing area, if all the workload of tasks is distributed among the processors evenly, then the schedule The RCPSP formulation, inspired by the RCPSP makespan has a good chance to be minimal. Fol- formulation of the traffic scheduling on Profine- lowing that, the algorithm tries to distribute the tIRT [3], is used for the message scheduling prob- messages among integration cycles evenly. All the lem. Thus, let us provide a brief overview for precedence constraints, time lags imposed by the RCPSP first. switch delay τ and real-time constraints are relaxed here. The integration cycle assignment problem is 3.3.1 RCPSP Overview formulated as the following ILP model: The RCPSP problem used in the paper is classi- min z xi,j fied as PSm,1,1 temp Cmax by Graham’s notation. X | | s.t. xi,j = 1 i Translated to common language, the problem is a ∀ j Project Scheduling problem with m resources, one X l,m cc unit of each resource available, and each activity de- c xi,j mod t z j, l, m j 0 ... i · i ≤ ∀ | ∈ { ic } mi∈kl,m mands at most one unit of the resource. The prob-

xi,j = 0 i, j di < j ic lem is constrained by temporal constraints (time ∀ | · lags among activities), and the objective is to min- xi,j = 0 i, j ri > j ic + ic ∀ | · imize the makespan Cmax. xi,j 0, 1 ; z R i, j ∈ { } ∈ ∀ The problem is defined as a sextuplet The binary variable xi,j = 1 if message mi is (V, p, E, R, B, b). V = A0,A1, ..., An,An+1 { } scheduled to the integration cycle j 0 ... ti and is a set of non-preemptive activities that should be l,m ∈ { } zero otherwise. The parameter ci represents the scheduled. A0 and An+1 are special dummy activ- transmission time of message mi on line ll,m. Note, ities where A0 represents the start of the schedule that if the links are configured to have a differ- and An+1 the end of the schedule, respectively. ent bandwidth then the transmission time of the p = 0, p1, ..., pn, 0 is the vector of the activities { }

8 l,m l,m l,m duration. The dummy activities have zero du- pi = ci of activity Ai expresses the trans- ration. The resulting schedule would be shifted mission time of message mi on link kl,m. How- otherwise. E is a set of pairs representing time ever, we assume that the bandwidth of all physical constraints. If activity Ai has a temporal relation links is the same for the sake of simplicity in this l,m to activity Aj then (Ai,Aj) E. Each pair in E paper. The resulting start time of activity A ∈ i is valued by the start-start time lag li,j. Note that will, consequently, represent the offset of message l,m the time lag between any activity Ai and dummy instance mi in the particular integration cycles activity An+1 is equal to the activity duration pi. in link kl,m. Recall that the integration cycles, in The temporal constraints are often visualized by which the message appears, are determined by the the so-called activity-on-node graph G(V,E). The first message occurrence. set R = R , ..., R is a set of q resources consid- 1 q The set of resources R is obtained from I - the ered in the{ problem.} B, consequently, represents set of integration cycles in the cluster cycle and K the number of units available in the resources. All - the set of links in the network. The number of the resources are unary (Bk = 1 k R) in our ∀ ∈ resources used in the model is then R = I K . case. Parameter b is a set of activity demands l,m | | | | · | | Each resource Ri represents the usage of link kl,m where bi,k represents the amount of resource Rk in the integration cycle ici. The resources have demanded by the execution of activity Ai. The l,m unary availability Bi = 1. Correspondingly, the value of bi,k is binary (bi,k 0, 1 i V, k R) ∈ { } ∀ ∈ ∈ message demands are unary too, which ensures that in the PSm,1,1 temp Cmax problem. | | the contention-free constraint is satisfied. Consid- The goal is to assign the start time to the ac- ering that the given conditions are met, the activities such that resource demands and time con- tivities demand can be directly derived from the straints are satisfied and the start time of activity routing and the message to the integration cycles A is minimal. An interested reader is referred n+1 assignment. The notation bl,m, which represents to [30] for more detailed information on the RCPSP i,j l,m problem. the amount of resource Rj demanded by activity l,m Ai , is used from this point further. If message 3.3.2 Formulation of the link schedule cre- mi is to be routed through link kl,m, among oth- ation problem to RCPSP ers, and it is known that the message appears, for l,m l,m example, in icx, icy and icz then bi,x = 1, bi,y = 1, The stated TTEthernet scheduling problem can bl,m = 1. All other demands for the activity repre- be formulated as the PSm,1,1 temp Cmax problem, i,z | | l,m considering the following conditions are met: senting message instance mi will be equal to zero. All the real-time constraints are modeled by set 1. Routing: It is known in which links each mes- E and time lags. In order to model release date r sage appears i and deadline di, it is necessary to know their rela- ˜ 2. Integration cycle assignment: It is known in tive valuesr ˜i and di with respect to the integration which integration cycles each message appears cycle in which the message is transmitted. The values are obtained by subtracting the integration The first condition is satisfied by the first stage of cycle start time from ri or di respectively. The re- the algorithm where the messages are assigned to lease date of the message mi is transformed to the links according to the routing. The second condi- positive time lag led from the dummy activity A0 to l,m tion is also met and the assignment of each message activity Ai where kl,m is the first link in the path to the integration cycles is obtained from the sec- the message traverses. The time lag is equal to the ond stage of the algorithm. relative release dater ˜i. The deadline, on the other In the RCPSP model [20], the message instances hand, is transformed to a negative time lag led from p,q are represented by the activities. Thus, the set of activity Ai , where kp,q is the last link before re- message instances is translated to the set of ac- ceiving the communication endpoint, to the dummy l,m tivities V where activity Ai corresponds to mes- activity A0. The value of the deadline time lag is l,m sage instance mi . The dummy activities A0 and equal to the negative value of the relative deadline p,q p,q An+1 are artificially added to V . The duration d˜i extended by duration p (i.e. (d˜i + p )). i − i

9 -5 000 000 straints). However, these constraints are out of the scope of our problem statement and are not used l,m in the paper. A1 With a given conﬁguration of the RCPSP model, 67 200 the objective of the RCPSP (i.e. to minimize the

10 000 68 200 maximal makespan) corresponds to the objective of m,o A1 our problem statement. In other words, the mini-

67 200 mal RCPSP makespan assures the maximal critical A0 An+1 gap. 68 200 0 0 The scheduling problem can be formulated as o,p A1 MMRCPSP as well. In this case, the integration

67 200 67 200 -4 432 800 67 200 cycle assignment is not necessary to be solved in

68 200 68 200 advance, because it can be formulated as part of the

p,u p,q MMRCPSP [21]. However, the complexity of the A A 1 1 resulting MMRCPSP model is signiﬁcantly higher

67 200 67 200 and, thus, the solver needs more time to ﬁnd a good or even any solution (see Sec. 4.2).

-4 432 800

Figure 4: Example of activity-on-node graph for 4 Experimental results outlined message m1 from Fig. 1 The proposed scheduling method was tested on a PC with Intel R CoreTM i7-4610M CPU (two cores Note that if there is more receivers in set Qi then it with 3 GHz and hyper-threading) and 8 GB RAM. is necessary to add the deadline time lag for each re- The algorithm uses the CPLEX ILP Solver for solv- ceiver separately. Technical delays and precedence ing the Integration cycle assignment problem. The constraints are represented by the positive prece- RCPSP problem was solved by both the CP Opti- dence time lags between two consecutive message l,m m,n mizer and SCIP Solver. However, the results ob- instances mi and mi . The precedence time lag l,m tained by the CP Optimizer were significantly bet- consists of duration pi and redistribution nodes ter in all the cases. Thus, the results of the SCIP delay parameter τ in store-and-forward networks. Solver are omitted here. In Fig. 4, the example of the activity-on-node graph for message m1 is presented. Message m1 can be also observed in Fig. 1 or in its crop in 4.1 Benchmark instance sets Fig. 3 - the one with a thick outline. The links as- Eight different benchmark sets were used for testing sumed here have a bandwidth of 10 Mbit/s and the purposes. Seven of them were generated by our message consists of 672 bits (64 octets of Ethernet instance generator. They represent the artificial frame + 12 octets interframe gap + 8 octets pream- instances. The last instance is the real problem ble and start of frame delimiter). The message is instance obtained from our industrial partner. transmitted by node el and received by nodes eq The synthetic instances are divided into seven and eu. It traverses the redistribution nodes em, eo groups according to the number of the used TT and ep and the redistribution node delay τ is 1 µs. messages. The sets are called according to that The relative release dater ˜1, expressed by the edge - from Set 20TT (set with twenty TT messages) l,m ˜ from A0 to A1 , is 10 µs. The relative deadline d1, to Set 2000TT. The length of the integration cy- p,u p,q represented by the edge from A1 and A1 to A0, cle equals thousand times the number of the TT is 4.5 ms. messages, which approximately caused the utiliza- The time lags allow one to limit the latency tion of half of the bandwidth. Each such set con- between the message transmission and reception tains thirty independently generated benchmark in- or the precedence constraints between the mes- stances. During the instance creation, the algo- sages (e.g. derived from the application level con- rithm for generating the network topology is se-

10 lected randomly. The generator provides four pos- Table 1: Quality comparison of different ap- sible algorithms - the star topology generator, the proaches snowflake topology generator, the Barabási-Albert ICAP Multi-mode algorithm for random tree generation and a random N [-] LB [ns] Cmax [ns] Cmax [ns] topology generator (the Barabási-Albert algorithm Set 20TT 124 9626 13430 13255 with additional redundant links). The Barabási- Set 50TT 312 19468 22908 23678 Albert algorithm is slightly modified. The nodes of Set 100TT 624 35603 40450 42291 Set 200TT 1243 66594 74663 79733 the network with degree 2 are eliminated to remove Set 500TT 3107 157317 182185 181913 long linear network segments which are uninterest- Set 1000TT 6244 318017 354577 354950 ing from the scheduling point of view. The net- Set 2000TT 12460 616446 663532 - work consists of twenty communication endpoints Set Industrial 5844 59888 63568 - for all the topologies. Each link has the bandwidth Average 177429 194708 - of 1 Gbit/s. The messages have a randomly chosen transmitter and set of receivers from the set of communica- mode RCPSP instead of the general RCPSP model. tion endpoints. Thus, any communication endpoint In the Multi-mode RCPSP, the alternative modes can send a message to an arbitrary subset of other can be assigned to the activities, and the resulting communication endpoints. The payload of the mes- schedule uses just one of the provided activities. sages is taken randomly from interval of 46 to 256 The TTEthernet problem can be formulated as a bytes. The message periods should be harmonic or Multi-mode RCPSP without the necessity to decide close to harmonic to keep the length of the cluster in which integration cycles each message appears cycle reasonable. Thus, the periods are from the in advance. Thus, it is not necessary to solve the n m domain ti = 2 3 ic where n N and m 0, 1 . integration cycle assignment problem heuristically The release times and deadlines∈ are also generated∈ { } by ILP. The Multi-mode RCPSP can return better randomly such that ri < di ti. solution than our described algorithm denoted as The last benchmark set ≤Set Indust consists of ICAP in Table 1 that uses heuristic integration cy- one benchmark instance given to us by our in- cle assignment, but the computational complexity dustrial partner. The instance evaluated the be- increases significantly. havior of the proposed method on instances from Table 1 represents the results obtained by the a natural environment which often does not di- methods. The captions of the benchmark sets rectly correspond to the artificial ones. The are situated in the first column. The second col- network consists of 6 redistribution nodes and umn presents the average number of message in- 59 communication endpoints and contains al- stances/activities in the set (N). Recall that each ternative paths. The instance has 1018 mes- message has one message instance for each link the sages to exchange with periods from domain message passes. Therefore, the number of message ti 12.5ms, 25ms, 50ms, 100ms, 200ms, 1000ms instances is often much higher than the number of and∈ efficient { payloads (payload that contain use-} messages. All the remaining columns shows the re- ful information) from domain ci [12, 9224] bits. sults for the particular algorithm. The values are There are no hard real-time constraints∈ imposed averaged over thirty independent instances in each on the instance. Each message has from one up to set. The third column shows the lower bound (LB) forty-two receivers. for the problem. The lower bound is obtained from the ILP solution of the integration cycle assignment 4.2 Quality and Performance evalu- problem. The makespan of our problem can never ation of the solver be shorter than the sum of the messages transmission time exchanged in any link if the message to The provided method for the TT message schedul- the integration cycle assignment is balanced. That ing has been evaluated on the described benchmark is why the ILP solution can be used as a lower set and compared with the Multi-mode method. bound for the whole problem too. However, it is The Multi-mode method developed by us is similar often not possible to obtain a makespan equal to to the described algorithm, but it uses the Multi- the lower bound because time lags are not consid-

11 ered in the ILP model. Therefore, it is not-tight tion does not separate the TT traffic and RC traffic lower bound. The fourth column contains the av- so strictly. It means that even if the makespan is erage makespan value of the instances in the given e.g. 6 ms (just like in the Fig. 2) the RC message set for our provided algorithm (ICAP) in nanosec- can be transmitted sooner if there is no TT traffic onds. In comparison, the fifth column contains the scheduled in the particular integration cycle (e.g. makespan for the Multi-mode method. The time message RC2 in Fig. 2). Moreover, the RC traffic limit for the computation of each instance was set can be transmitted even in the gaps of the schedule to 300 seconds for both methods. where they can fit (e.g. message RC1 in Fig. 2). The small theoretical instances with 20 TT messages were solved to an optimum by RCPSP solver by both methods in time. The final results for these small instances are better for the Multi-mode ver- 500 Porosity optimization sion because it is not heuristically guided by the Makespan minimization

ILP solution and, therefore, no states of the search s] 400 space are pruned. One can observe the gap between µ the optimal solution of the Multi-mode method 300 and the non-tight lower bound. The Multi-mode RCPSP solver was not able to solve bigger instances 200 to an optimum in time. This leads one to the fact Worst-Case delay [ that the guided ICAP method obtains better solu- 100 tions (except the set with 500 TT messages where 20TT 50TT 100TT 200TT 500TT 1000TT relaxing the time-lags in the ICAP formulation led Set to the weaker integration cycle assignment) from this point further. Furthermore, no feasible solution was found by the Multi-mode RCPSP solver Figure 5: Worst-Case delays for the RC traffic for instances with two thousand messages in 300 seconds. Note that the solutions obtained by the ICAP method are not more than 15% distant from To evaluate the difference between the makespan the lower bound for instances with more than 50 optimization and the porosity optimization in an TT messages. objective way, the Worst-Case delay calculation for the RC messages taken from [8] was employed. 4.3 Worst-case RC traffic delay evaluation In Fig. 5, the average delays among the instances in the particular sets and its standard deviations To justify the used scheduling method, the solver are presented. The evaluation was made for all the based on the porosity idea from [1] was imple- artificial sets. However, Set 2000TT has been omit- mented to compare the makespan optimization ted from the graph because it caused graph scaling method with the porosity optimization method which worsens its clarity. The worst-case RC traffic with respect to the RC traffic. The porosity is delay of Set 2000TT was 797 18µs for makespan ± represented by the set of gaps introduced into the optimization and it was 799 18µs for porosity op- schedule. Consequently, the optimization objective timization. ± is to maximize the length of these gaps. Compared to the makespan optimization, which aims to intro- According to the results, the makespan optimiza- duce just one big gap at the end of the schedule, tion behaves comparable and even slightly better the porosity optimization distributes the free band- than the porosity optimization with respect to the width through the schedule. RC traffic. The standard deviation is higher in the Note that unlike in other protocols that has strict case of Set 500TT because there were a few sig- separations of time-triggered and event-triggered nificant outliers. Overall, the obtained Worst-Case segments, the introduced porosity gaps as well as delays are more stable in the case of the makespan one critical gap introduced by makespan optimiza- optimization.

12 5 Conclusion Acknowledgment

This work was supported by the Grant Agency of The paper in hand focuses on the problem of the the Czech Republic under the Project FOREST Time-Triggered communication scheduling on the GACR P103-16-23509S. TTEthernet network. To incorporate the Event- Triggered Rate-Constrained traffic with TT traffic is very important for mixed-criticality appli- References cations [31]. The majority of communication is [1] W. Steiner, “Synthesis of static commu- Event-Triggered in industrial applications nowa- nication schedules for mixed-criticality days. However, the necessity to incorporate safety- systems,” in 14th IEEE Int. Symp. related communication, e.g. in automotive or Object/Component/Service-Oriented Real- avionics industry, pushes system developers to use Time Distributed Computing Workshops the TT traffic because it can be more easily verified (ISORCW), (Newport Beach, CA, USA), and certified. pp. 11–18, 2011. We have investigated the idea of separating the [2] R. I. Davis, S. Kollmann, V. Pollex, and Time-Triggered traffic and Event-Triggered traffic, F. Slomka, “Schedulability analysis for Con- which was inspired by the scheme of the FlexRay troller Area Network (CAN) with FIFO queues bus communication cycle. The aim has been to priority queues and gateways,” Real-Time minimize the length of the Time-Triggered sched- Syst., vol. 49, no. 1, pp. 73–116, 2013. ule and leave the rest of the communication bandwidth free for the Rate-Constrained traffic. For [3] Z. Hanzálek, P. Burget, and P. S˚ucha,ˇ such a problem statement, we have designed the “Profinet IO IRT Message Scheduling With algorithm based on the ILP formulation of the in- Temporal Constraints,” IEEE Trans. Ind. In- tegration cycle assignment problem (load balancing format., vol. 6, pp. 369–380, Aug 2010. among integration cycles) and the RCPSP formulation of the message scheduling problem. [4] B. Galloway and G. P. Hancke, “Introduc- tion to Industrial Control Networks,” IEEE The experiments performed on several artificial Communications Surveys Tutorials, vol. 15, benchmark sets and one real-case instance show pp. 860–880, July 2013. that the designed method is able to obtain near- optimal results (for bigger instances in 15% of the [5] International Organization for Standardiza- non-tight lower bound) concerning the proposed tion, “ISO 17458 - FlexRay communications criterion. Moreover, the RC traffic worst-case de- system,” 2015. lay calculations suggest that the proposed con- [6] H. Kopetz, A. Ademaj, P. Grillinger, and cept (minimization of the schedule makespan) can K. Steinhammer, “The time-triggered ether- provide slightly better delays in average than the net (TTE) design,” in 8h IEEE Int. Symp. porosity imposed to the schedule by gaps. Object-Oriented Real-Time Distributed Com- For the future work, we would like to study the puting (ISORC), pp. 22–33, IEEE, 2005. behavior of the provided method in the environ- [7] SAE International, “AS6802: Time-Triggered ment of the 802.1Qbv [32] network. The Ether- Ethernet,” tech. rep., SAE International, net technology is used in the automotive, avionics, 2011. etc. industry nowadays, where the development cycle of the products is specific - the new product [8] D. Tamas-Selicean, P. Pop, and W. Steiner, models are created taking a backward compatibil- “Timing Analysis of Rate Constrained Traf- ity with the previous ones into account. Thus, we fic for the TTEthernet Communication Pro- are also going to study the incremental scheduling tocol,” in 18th IEEE Int. Symp. on Real- case where backward compatibility needs to be pre- Time Distributed Computing, (Auckland, New served. Zealand), pp. 119–126, 2015.

13 [9] ARINC (Aeronautical Radio, Inc.), “ARINC on distributed time-triggered networked sys- 664P7: Aircraft Data Network, Part 7, Avion- tems,” in Proc. IEEE Emerging Technology ics Full-Duplex Switched Ethernet Network,” and Factory Automation (ETFA), (Barcelona, tech. rep., ARINC (Aeronautical Radio, Inc.), Spain), pp. 1–8, 2014. 2009. [19] S. S. Craciunas and R. S. Oliver, “Combined [10] Condor Engineering, Inc., “AFDX / ARINC task- and network-level scheduling for dis- 664 Tutorial (1500-049),” tech. rep., Condor tributed time-triggered systems,” Real-Time Engineering, Inc., 2005. Syst., vol. 52, pp. 161–200, Mar 2016. [11] ARINC Industry Activities, “ARINC 429 [20] K. Neumann, C. Schwindt, and J. Zimmer- Standard,” 2017. mann, Project Scheduling with Time Windows and Scarce Resources. Boston, MA: Springer [12] D. Tamas-Selicean, P. Pop, and W. Steiner, US, 2012. “Synthesis of Communication Schedules for TTEthernet-based Mixed-criticality Sys- [21] M. Heller, “Scheduling of the TTEthernet tems,” in Proc. 8th IEEE/ACM/IFIP Int. communication,” Master’s thesis, Czech Tech- Conf. Hardware/Software Codesign and nical University in Prague, Czech Republic, Syst. Synthesis (CODES+ISSS), (Tampere, 2016. Finland), pp. 473–482, 2012. [22] S. Hartmann and D. Briskorn, “A survey [13] W. Steiner, “An Evaluation of SMT-Based of variants and extensions of the resource- Schedule Synthesis for Time-Triggered Multi- constrained project scheduling problem,” Eu- hop Networks,” in 31st IEEE Real-Time ropean J. of Operational Research, vol. 207, Syst. Symp. (RTSS), (San Diego, CA, USA), pp. 1 – 14, Nov 2010. pp. 375–384, 2010. [23] A. Schnell and R. F. Hartl, “On the eﬃ- [14] S. S. Craciunas, R. S. Oliver, M. Chmel´ık,and cient modeling and solution of the multi-mode W. Steiner, “Scheduling real-time communi- resource-constrained project scheduling prob- cation in IEEE 802.1Qbv time sensitive net- lem with generalized precedence relations,” works,” in Proc. 24th Int. Conf. on Real-Time OR Spectrum, vol. 38, pp. 283–303, Mar 2016. Networks and Syst. (RTNS), (Brest, France), pp. 183–192, 2016. [24] T. Achterberg, “SCIP: Solving constraint integer programs,” Math. Programming Compu- [15] W. Steiner, M. Gutirrez, Z. Matyas, F. Pozo, tation, vol. 1, pp. 1–41, July 2009. and G. Rodriguez-Navas, “Current techniques, trends and new horizons in avionics networks [25] P. Sitek and J. Wikarek, “A constraint-based conﬁguration,” in 34th IEEE/AIAA Digital approach to modeling and solving resource- Avionics Syst. Conf. (DASC), (Prague, Czech constrained scheduling problems,” in Proc. Republic), pp. 1–26, 2015. 8th Int. Con. Computational Collective Intell. (ICCCI), (Halkidiki, Greece), pp. 423–433, [16] D. Tamas-Selicean, P. Pop, and W. Steiner, 2016. “Design optimization of TTEthernet-based distributed real-time systems,” Real-Time [26] T. Steinbach, F. Korf, and T. C. Schmidt, Syst., vol. 51, pp. 1–35, Jan 2015. “Comparing time-triggered Ethernet with FlexRay: An evaluation of competing ap- [17] M. Baghel, S. Agrawal, and S. Silakari, “Sur- proaches to real-time for in-vehicle networks,” vey of metaheuristic algorithms for combinato- in Proc. 8th IEEE Int. Workshop on Factory rial optimization,” Int. J. Comput. Applicat., Commun. Syst., (Nancy, France), pp. 199–202, vol. 58, pp. 21–31, Nov 2012. 2010.

[18] S. S. Craciunas, R. S. Oliver, and V. Ecker, [27] R. W. Floyd, “Algorithm 97: Shortest path,” “Optimal static scheduling of real-time tasks Commun. ACM, vol. 5, p. 345, June 1962.

14 [28] G. E. de Andrade, L. A. de Paula Lima, A. Calsavara, J. A. de Oliveira, and G. Mich- elon, “Message routing in vehicular delay- tolerant networks based on human behavior,” in 10th Int. Symp. Commun. Syst., Net- works and Digital Signal Process. (CSNDSP), (Prague, Czech Republic), pp. 1–6, 2016. [29] A. Jouy, J. Yao, and G. Zhu, “Optimal bandwidth allocation with dynamic multi- path routing for non-critical traﬃc in afdx networks,” in 20th IEEE Int. Conf. on Paral- lel and Distributed Syst. (ICPADS), (Hsinchu, Taiwan), pp. 600–607, 2014.

[30] P. Brucker, A. Drexl, R. Mhring, K. Neumann, and E. Pesch, “Resource-constrained project scheduling: Notation, classiﬁcation, models, and methods,” European J. of Operational Re- search, vol. 112, pp. 3 – 41, Jan 1999.

[31] A. Novak, P. Sucha, and Z. Hanzalek, “Eﬃ- cient algorithm for jitter minimization in time- triggered periodic mixed-criticality message scheduling problem,” in Proc. 24th Int. Conf. on Real-Time Networks and Syst. (RTNS), (Brest, France), pp. 23–31, 2016.

[32] WG802.1 - Higher Layer LAN Protocols Work- ing Group, “802.1Qbv-2015 - IEEE Standard for Local and metropolitan area networks – Bridges and Bridged Networks - Amendment 25: Enhancements for Scheduled Traﬃc,” tech. rep., IEEE Standards Association, 2016.