Clusterable Task Scheduler

Masaryk University Faculty of Informatics

Bachelor’s Thesis

Ján Michalov

Brno, Fall 2019

Masaryk University Faculty of Informatics

Clusterable Task Scheduler

Bachelor’s Thesis

Ján Michalov

Brno, Fall 2019

This is where a copy of the official signed thesis assignment and a copy ofthe Statement of an Author is located in the printed version of the document.

Declaration

Hereby I declare that this paper is my original authorial work, which I have worked out on my own. All sources, references, and literature used or excerpted during elaboration of this work are properly cited and listed in complete reference to the due source.

Ján Michalov

Advisor: RNDr. Adam Rambousek, Ph.D.

Acknowledgements

I would like to sincerely thank my advisor RNDr. Adam Rambousek, Ph.D. for his guidance, patience and precious advice. I am also grateful to my consultant Bc. Matej Lazar, who directed me and helped with the design across countless hours of meetings. I wish to thank my friends and family for their support during stressful days.

iii Abstract

The purpose of this thesis is to create a microservice that would schedule tasks happening on other devices. The tasks can have different tasks declared as dependencies, and the microservice must execute them in the correct order. Additionally, the microservice must be able to be deployed in a clustered environment, which means ensuring data consistency and preventing duplicate execution of a task. The chosen platform for the microservice is Java.

iv Keywords microservice, dependency resolution, scheduling, Java, data consistency, cluster

Contents

Introduction 1

1 Theoretical background 3 1.1 Scheduling ...... 3 1.2 Applicaton clustering ...... 3 1.3 Microservice architecture ...... 4

2 JBoss MSC 5 2.1 Architecture ...... 5 2.1.1 Service ...... 5 2.1.2 ServiceController ...... 6 2.1.3 ServiceRegistrationImpl ...... 7 2.1.4 Dependency and Dependent ...... 7 2.1.5 ServiceRegistry ...... 8 2.1.6 ServiceTarget ...... 8 2.1.7 ServiceContainer ...... 9 2.2 Transitional period of a ServiceController ...... 9 2.3 Concurrency and synchronization ...... 11 2.4 Pros and cons ...... 12 2.5 Conclusion ...... 13

3 Infinispan 15 3.1 Client-server mode ...... 15 3.1.1 Network protocols ...... 16 3.1.2 Server ...... 16 3.1.3 Hot Rod Java client ...... 17

4 Application platform 21 4.1 JBoss EAP ...... 21 4.2 Thorntail ...... 22 4.3 Quarkus ...... 22 4.4 Decision ...... 23

5 Design 25 5.1 Requirements ...... 25 5.1.1 Must have ...... 25

vii 5.1.2 Should have ...... 25 5.1.3 Could have ...... 26 5.1.4 Remote entity requirements ...... 26 5.2 Differences and similarities to JBoss MSC ...... 26 5.2.1 Naming changes ...... 26 5.2.2 What stayed ...... 27 5.2.3 What changed ...... 27 5.3 States, Transitions, Modes and Jobs ...... 28 5.3.1 Modes ...... 29 5.3.2 Jobs ...... 29 5.3.3 StageGroups ...... 29 5.3.4 States ...... 30 5.4 Modules ...... 31

6 Implementation 33 6.1 Context dependency injection ...... 33 6.1.1 Maven module problem ...... 33 6.2 Transactions ...... 34 6.2.1 Partial updates ...... 34 6.2.2 Prevention of duplicate execution ...... 34 6.3 Mapping ...... 35 6.4 REST ...... 35 6.5 Installation ...... 36 6.5.1 Prerequisites ...... 36 6.5.2 Setting up an Infinispan server ...... 37 6.5.3 Compilation and execution ...... 38

7 Testing 39 7.1 Local integration testing ...... 39 7.2 Clustered testing ...... 40

8 Conclusion 43 8.1 Future improvements ...... 43

Bibliography 45

A Attached files 47

viii List of Figures

2.1 State-diagram of JBoss MSC. Source: [6] 6 5.1 State-machine diagram of a Task 28 5.2 The diagram of package dependencies in the scheduler 31

Introduction

This thesis was created as an effort from company Red Hat, to improve the scalability of a product called Project Newcastle. Nowadays, scalability is a common problem across products. There are two ways to scale a product. Vertically with an addition of power in the form of memory and CPU cores or horizontally with clustering. However, some products suffer from a sophisticated monolithic design. This problem also persists in Project Newcastle. One of the techniques that solve this dilemma is microservice architecture. Microservice architecture aims to dissect these monoliths into smaller parts, each with their function, that communicates with each other. These parts are simple, therefore easier to maintain, and should be designed in a way to scale in a cluster. One of the functions of Project Newcastle is to schedule tasks, which are executed remotely. Additionally, these tasks have defined dependencies and therefore have to be scheduled in the correct order. The goal of this thesis is to create an open-source remote scheduler with an ability to scale in the cluster and microservice architecture in mind. The thesis is made of seven chapters excluding conclusion. The first chapter focuses on the theoretical aspect of the thesis and introduces the reader to complexities of scheduling, clustering and microservice architecture. The following chapter regards to an analysis of JBoss MSC library. The library implements a scheduling solution with a different use-case but flexible implementation, which concepts are used in the design. Chapter three concentrates on a Red Hat developed datastore solution Infinispan, which intent is to be used for enable- ment of clustering for a variety of applications. Next chapter is briefly introducing available Red Hat application platforms, their strong and weak aspects, and which is the most suitable for the scheduler. Chapter five is the design of the application. The chapter defines and explains the requirements, points out the major distinctions against JBoss MSC and defines the states of a task and other essential models. Chapter six delves into the problematics of the implementation part of the thesis. This chapter points out some flaws of used libraries/frameworks, describes how is data consistency guaranteed and concludes with

1 a guide for compiling from source and subsequent execution of the scheduler. The last chapter before the conclusion is focused on testing. The testing includes local integration tests and clustered tests, which the chapter describes in detail.

2 1 Theoretical background

1.1 Scheduling

In a scheduling problem, there is a set of tasks and a set of constraints. These constraints state that executing a specific task could depend on other tasks being completed beforehand. These sets can be mapped into a directed graph, with the tasks as the nodes and the direct pre- requisite constraints as the edges. If this graph has a cycle, then it implies that there exists a task, which is transitively dependent on itself. Therefore, this task has to complete before it can start, which doesn’t make sense. Hence, the graph can not have cycles and can be expressed as a directed acyclic graph (DAG). To schedule a set of tasks an ordering is needed. The order has to respect dependency constraints. For DAG, this ordering is called topological sorting. Every finite DAG has a topological sort. However, there can be more than one possible topological sort.[1] A topological sort can be found by iteratively marking a task that has either no dependencies or all of its dependencies are marked. The ordering of marks suggests a topological sorting. This algorithm can produce a different order if it has more than one task available to mark. For parallel task scheduling, the algorithm mentioned above can be modified. Instead of marking one task each iteration, it marks all tasks ready for marking. All tasks marked in one iteration are independent of each other and can execute concurrently. A further modification could be to only allow a certain number of marks in one iteration, which simulates a situation where resources are limited.

1.2 Applicaton clustering

Application clustering typically refers to a method of grouping multiple computer servers into an entity that behaves as a single system. A server in a cluster is referred to as a node. Typically, each node runs the same copy of an application that is usually deployed on an application server which provides clustering features (Wildfly). For

3 1. Theoretical background instance, Wildfly1 application servers can discover each other on a network and replicate the state of a deployed application [2]. Benefits of clustering include [3]:

1. Load Balancing (Scalability): Processing requests are distributed across the cluster nodes. The main objective of load balancing is to limit nodes getting overloaded and possibly shutting down. Adding more nodes to a cluster increases the cluster’s whole computing capabilities.

2. Fail-over (High Availability): Clusters enable services to last for longer periods. Singular servers have a single point of failure. A server can fail unexpectedly due to unforeseen causes such as infrastructure issues, networking problems or software crash- ing. On the other hand, a cluster is more resilient. If one node crashes, there are still other nodes that can handle incoming requests.

A direct method of developing a clusterable application is without keeping a state. A stateless application does not retain data for later use. However, the state can be stored in a database instead. Each stateless application can connect to the database where they keep all information. Stateless applications are easily scalable.[4]

1.3 Microservice architecture

Microservice is architectural style motivated by a service-oriented architecture (SOA) that appeared due to a need for flexible and conve- niently scalable applications as opposed to monolithic style, which is challenging to use in distributed systems. Microservices handle gradu- ally increasing complexity of large systems by decomposing them into a set of independent services. These services are loosely-coupled, and each should provide specific functionality. Moreover, microservices can be developed independently, which simplifies creating features and code maintenance.[5]

1. https://wildfly.org/

4 2 JBoss MSC

JBoss MSC1 (JBoss Modular Service Container) is a “lightweight highly concurrent open-source dependency injection container”2 developed by Red Hat and written in Java. It is internally used by JBoss Enterprise Application Platform3 (JBoss EAP) which is Red Hat supported version of WildFly4. JBoss EAP consists of a hundreds of components that are dependant on each other and have to start in the correct order. JBoss MSC solves this complexity by the ability to define these relations while also providing features such as concurrent execution, failure handling or retry mechanisms. This chapter is based on an analysis of the source code and documentation[6] of the JBoss MSC. It addresses internal architecture, design and positives along with negatives of the implementation against requirements of this thesis.

2.1 Architecture

This section analyzes specific classes and interfaces, which serve a significant role in the library’s logic. The analysis is based on 1.3.2.Fi- nal5 version JBoss MSC, which is less complicated. Version 1.4.0.Final6 and later deprecate most of the library and provide new API to add support for Services, to return multiple values.

2.1.1 Service

Service is the central abstraction of the MSC. Definition of a Service is something that can be started and stopped. To create a Service, a user has to create a class implementing the Service interface and define the start, stop and getValue methods. The getValue method serves as

1. https://github.com/jboss-msc/jboss-msc 2. https://jboss-msc.github.io/jboss-msc/manual/ 3. https://www.redhat.com/en/technologies/jboss-middleware/ application-platform/ 4. https://wildfly.org/ 5. https://github.com/jboss-msc/jboss-msc/tree/1.3/ 6. https://github.com/jboss-msc/jboss-msc/tree/1.4/

5 2. JBoss MSC

Figure 2.1: State-diagram of JBoss MSC. Source: [6] a way for a Service to provide a value. Because, for a Service to start, it may need more information or specific value that another Service produces. For example, HTTP server Service may need a value from a Service providing host and port. Therefore, Services create a scheduling problem defined by a directed acyclic graph, where the node isa Service, and vertex signify a dependency.

2.1.2 ServiceController

ServiceController is an interface used to manipulate metadata required for scheduling operations of the Service. Its implementation (Service- ControllerImpl) is both owner and manipulator of the metadata. Each ServiceController has a unique identifier, and an instance of Service associated. Essential metadata:

6 2. JBoss MSC

• ServiceName name: unique identifier for ServiceController, that can be imagined as a string in a format similar to Java package names (e.g. “org.jboss.msc.identifier”)

• Enumeration State and Substate: Represents a state of a Service- Controller in a scheduling process. Substate is a fine-grained representation of State. For a visualization, see Figure 2.1.

• Dependency[] dependencies: An array of dependencies that Controller uses for sending messages to dependencies.

• ServiceRegistrationImpl registration: Registration used for retrieving dependents.

• Additional integers that keep track of information about dependencies/dependents. Example being, runningDependents and stoppingDependencies. Purpose of a ServiceController is to handle transitions between states, handle communication to dependents and dependencies, provide features for removing Services from ServiceContainer.

2.1.3 ServiceRegistrationImpl

ServiceRegistrationImpl is a class that encapsulates ServiceController and implements further functionality. Namely adding dependents at runtime and read/write locking support through Lockable class. Notable properties: • ServiceControllerImpl instance: An instance of ServiceController- Impl that registration owns.

• Set dependents: A set of dependents on this registration.

2.1.4 Dependency and Dependent

Dependency and Dependent interfaces serve as a medium for commu- nicating between ServiceController and its dependencies/dependents. Suppose ServiceController needs to alert its dependents that it stopped. Firstly, the Controller gets a collection of Dependents (available through

7 2. JBoss MSC a property of ServiceRegistrationImpl). Secondly, for each dependent, it invokes dependencyDown() method. Due to ServiceControllerImpl implementing the Dependent interface, dependencyDown() method is invoked on the ServiceController of the dependent. The method changes the metadata (in this particular case, increasing stoppingDependents integer by one), which may cause a transition between states. More- over, ServiceRegistrationImpl implements Dependency interface, but the implementation is mostly delegated to ServiceControllerImpl.

2.1.5 ServiceRegistry

ServiceRegistry is an interface that acts as a registry for Services and their Controllers. This interface exposes an API for retrieving installed ServiceControllers. To retrieve a specific ServiceController, a user needs to know the ServiceName that uniquely identifies it. There are two methods to retrieve a ServiceController. They differ in handling cases where the ServiceController is missing. One returns null, and other raises a ServiceNotFoundException. The implementation of ServiceReg- istry uses ConcurrentHashMap as a data store, where the identifying key is ServiceName and value ServiceRegistrationImpl.

2.1.6 ServiceTarget

ServiceTarget is a target for Service installations. To install a Service, the user has to invoke addService(ServiceName name) method which returns a ServiceBuilder. ServiceBuilder, with the usage of the builder pat- tern7, is used to define all information that ServiceController requires to function in the container. ServiceBuilder API:

• setInitialMode(Mode mode): ServiceController’s starting mode.

• addDependency(ServiceName dependency): Declares a dependency to created Service.

• setInstance(Service service): Sets the instance of a Service to execute.

7. For additional information: https://www.journaldev.com/1425/ builder-design-pattern-in-java

8 2. JBoss MSC

• install(): Invoked at the end to finalize the declaration of Service and its relations.

Firstly, installing ServiceBuilder causes Target to create new unique ServiceRegistrationImpl and ServiceControllerImpl. Secondly, it informs existing dependencies about new dependents, initializes initial Mode and State for the Controller and checks for circular dependencies. In the end, it adds new ServiceRegistrationImpl to a registry.

2.1.7 ServiceContainer

ServiceContainer extends ServiceTarget and ServiceRegistry. Therefore, the implementation of ServiceContainer operates both as a registry and as a target for Service installations. Moreover, the interface provides a method for shutting down the ServiceContainer. To shut down, the ServiceContainer changes Mode for each registered ServiceController to Mode.REMOVE. Furthermore, the shutdown can be initiated by re- ceiving a SIGINT signal to the JVM process running the container. To achieve this, ServiceContainer registers a shutdown hook to Java Run- time8 , which executes on JVM termination. Furthermore, ServiceCon- tainer provides an ExecutorService9 that is needed by ServiceController during transition periods.

2.2 Transitional period of a ServiceController

ServiceController’s decision to transition is solely based on its own metadata. Hence, the metadata has to be modified before a transition can start. Transitions between States can initiate in two ways:

1. Different ServiceController/Registration invokes a method through Dependent interface with the reason to inform the invoked Con- troller that something occurred. Based on this information, the Controller adjusts the metadata and checks if the change causes a transition.

8. https://docs.oracle.com/javase/8/docs/api/java/lang/Runtime.html# addShutdownHook-java.lang.Thread- 9. https://docs.oracle.com/javase/8/docs/api/java/util/concurrent/ ExecutorService.html

9 2. JBoss MSC

2. Changing the Mode through the setMode() method. All possible transitions are pre-defined and reflected in an enumeration Transition. Therefore, an invalid transition cannot arise. If a transition does occur, ServiceController enters a transition period. During this period, no other transition can occur until the period completes. Each transition has a pre-defined list of ControllerTasks which are run after a Transition has been chosen. A ControllerTask is a class that acts as a unit of execution. They implement Runnable10 interface and are executed asynchronously in the ExecutorService provided by Service- Container. A notable mention is that ControllerTasks have direct access to data of a ServiceControllerImpl they are instantiated in, due to being a nested inner class11 of ServiceControllerImpl. Therefore, each Con- trollerTask is bound to some ServiceControllerImpl. There are numerous types of ControllerTasks: • StartTask: Executes a Service through invocation of Service.start() • StopTask: Stops the execution of a Service through invocation of Service.stop()

• DependentsControllerTask: Abstract class which implements a way to send message to all dependents through Dependents interface. • DependenciesControllerTask: Abstract class which implements a way to send message to all dependencies through Dependency interface. • DependencyStartedTask: Implements DependentsControllerTasks and informs all dependents that their dependency has started through Dependent.dependencyUp() method. • ... A transitional period concludes with all bound ControllerTasks completed. In conclusion, the whole workflow can be divided into several parts:

10. https://docs.oracle.com/javase/8/docs/api/java/lang/Runnable.html 11. For additional information: https://docs.oracle.com/javase/tutorial/ java/javaOO/nested.html

10 2. JBoss MSC

1. Invoke setMode or Dependency/Dependant interface method. 2. Alter metadata. 3. Check no bound ControllerTasks are running. 4. Get the current State. 5. Check for transitions from the current State according to altered metadata. 6. If affirmative, get ControllerTasks associated with the transition. If negative, return. 7. Set new State. 8. Enter the transition period. 9. Get an ExecutorService from ServiceContainer. 10. Execute ControllerTasks. 11. Exit the transition period.

2.3 Concurrency and synchronization

JBoss MSC is a highly concurrent library, where asynchronous Con- trollerTasks are the leading cause. Hence we can assume that the data will be accessed from more than one thread. Example being, two Services with common dependent have started at the same time. Both enter a transition period to the UP state and want to inform the common dependent about it. The ControllerTasks (De- pendencyStartedTask) invoke dependencyUp() on the ServiceController of the common dependent. In the method, the stoppingDependencies attribute is read (2) at the same time. Therefore it has the same value for both dependencies. Both reduce the variable’s value by one (1) and save. The variable now has an inconsistent value (1), due to dependencies not having serialized access to the variable. If one of the dependencies waited until the other one saved, the inconsistency would not occur. One of the resolutions is defining borders in the code, which can be accessed by one thread at the same time. Java has an answer in the form of synchronized blocks. The synchronized block construct takes a parameter of any instantiated Java Object. When a thread enters a synchronized block, it acquires a lock based on the passed parameter. Until it exits the block, no other thread can enter the synchronized block with the same parameter. Therefore for synchronization to take effect, both threads have to enter

11 2. JBoss MSC the same parameter. In this case, the instance of ServiceControllerImpl is the solution. JBoss MSC uses synchronized blocks in every ControllerTask and implementations of Dependency/Dependent methods. To avoid entering methods without synchronization, the library uses Thread.holdsLock()12 method, which takes a parameter of an Object instance. If the parameter was also used as a parameter in a synchronized block, it returns true.

2.4 Pros and cons

Purpose of the analysis of JBoss MSC is to investigate whether it is deploy-able in a cluster and, if modified, to meet the conditions defined in the requirements of this thesis.

Positives • Well-thought-out architecture with every component clearly defined. • Highly flexible design that allows full modification of States, Transitions and ControllerTasks to accommodate every demand. • Adding functionality is a matter of creating new ControllerTask and adding it to desired Transitions.

Negatives • Using synchronized blocks hinders the ability to run in a cluster, due to them being bound to JVM, which clusters do not share. • In-memory datastore with ConcurrentMap. • Very complicated code base resulting from supporting multiple features, where most of them are not necessary for the thesis (retry mechanism, value injections). Consequently, modifying the library would add unnecessary complexity. • Services run locally, whereas the intention is running remotely. • Missing feature for submitting multiple Services at once.

12. https://docs.oracle.com/javase/8/docs/api/java/lang/Thread.html# holdsLock-java.lang.Object-

12 2. JBoss MSC 2.5 Conclusion

Directly using the codebase for a clustered solution is not desirable due to in-memory datastore and JVM-bound locking. With the support for all functionality, modifying the codebase would prove complicated because each modification would require a more significant effort to implement. Therefore, the decision is writing new codebase from the beginning. The codebase will be written with an architecture close to JBoss MSC, but adjusted for remote scheduling and native clustering.

3 Infinispan

Infinispan is an open-source in-memory, distributed NoSQL key/value datastore solution. Its primary focus is to be used in conjunction with clusters, where Infinispan offers high-speed access with replication and distribution of data. Infinispan is designed to work in a highly concurrent environment, while also providing functionality such as transactions, clustered locks, querying or distributed processing. Apart from that, Infinispan implements and integrates with a myr- iad of known frameworks like CDI, Hibernate, Apache Lucene, Apache Spark, Quarkus and Wildfly. The supported version of Infinispan is Red Hat Data Grid1 or RHDG. Infinispan provides two modes:

1. Embedded: In the embedded version, Infinispan co-exists with the application in the same JVM. To enable clustering and horizontal scaling of the application, Infinispan automatically dis- covers neighbouring instances of the application. Therefore, creating a cluster is a matter of creating multiple nodes and verifying they can interact.

2. Client-server: In the client/server version, Infinispan is separated from the application and runs as a remote server. The application uses Infinispan client to connect to the server using a network protocol.

This chapter focuses on the client-server version of Infinispan as it was the one chosen in the implementation. Reason being, limited support of embedded mode in Quarkus application server2. The information used in this chapter was inspired by Infinispan user guide[7].

3.1 Client-server mode

In client-server mode, applications access the data stored in a remote Infinispan server through a network protocol. The Infinispan server

1. https://www.redhat.com/en/technologies/jboss-middleware/data-grid 2. More in chapter Application servers

15 3. Infinispan itself is capable of horizontal scaling in a cluster and can support hundreds of nodes. Therefore, the client-server mode achieves elasticity due to the independent scaling of server and client.

3.1.1 Network protocols

Network protocols used by Infinispan are language-neutral. Conse- quently, clients can use other programming languages like C++, C#, Perl, Javascript, Ruby and Python3. Infinispan supports three distinct network protocols for interacting with a server:

• Hot Rod protocol: It is a binary protocol directly developed by Infinispan developers. It is a recommended protocol to useif running Java.

• REST endpoint: Infinispan server exposes RESTful HTTP interface to access caches. It is recommended for an environment where HTTP port is the only one allowed.

• Memcached protocol: It is a text-based protocol with an application for users, that desire failover capabilities of their Mem- cached server, which is not natively clusterable.

Hot Rod protocol is the protocol with the most significant support and functionality with features such as topology awareness, partial transaction support, bulk operations, listeners and server queries.

3.1.2 Server

Infinispan server is a standalone server that provides caching ability through a variety of network protocols. The server exposes additional services for management, persistence, logging, security and transactions. Moreover, the server can be extensively configured. The documentation for configuring the server can be found in the server guide

3. https://infinispan.org/hotrod-clients/

16 3. Infinispan

on Infinispan website4, and all available configuration options are accessible here5.

3.1.3 Hot Rod Java client

This section describes the configuration of the Hot Rod client, how to access and manipulate a cache, data marshallers, transactions and their limitations and querying through Query DSL. Additionally, Hot Rod protocol supports authentication, data encryption and version interoperability (different client and server versions).

Configuration Before accessing a cache, Hot Rod client requires configuration. The client can be configured either programmatically or through a configuration file. A user has to specify at least the host and port ofthe Infinispan server. Furthermore, the user can define marshallers, transaction modes and managers, balancing strategies, a maximum number of near cache entries and much more6. The configuration is then passed into RemoteCacheManager7, which can retrieve RemoteCaches.

Remote cache All caches in Infinispan are named. Therefore, to retrieve a cache, the user has to know its name. Remote caches implement ConcurrentMap interface, which is the primary API for interacting with Remote cache. However, each invocation causes communication with a server. There- fore, to improve performance, the client can be configured to use near caching. Near cache is a local cache that stores recently used data. If another client modifies the data, the local cache is invalidated, and the next access causes communication with the server.

4. https://infinispan.org/docs/9.4.x/server_guide/server_guide.html# infinispan_subsystem_configuration 5. https://docs.jboss.org/infinispan/9.4/configdocs/ infinispan-config-9.4.html 6. https://docs.jboss.org/infinispan/9.4/apidocs/org/infinispan/ client/hotrod/configuration/ConfigurationBuilder.html 7. https://docs.jboss.org/infinispan/9.4/apidocs/org/infinispan/ client/hotrod/RemoteCacheManager.html

17 3. Infinispan

Versioned API Remote cache provides additional API which is similar to the primary API. The difference is that methods retrieve or take extra parameter in the form of version. For each key/value pair, a version is an integer that uniquely identifies each modification. Versioned API can prevent data inconsistencies. If a client saves a value, but he has specified an older version, the save will fail, and the client has to react appropriately.

Marshalling Marshalling is a process of mapping Java Objects to a format that can transfer over a wire. Infinispan client supports some marshalling libraries natively such as JBoss Marshalling8 or Infinispan Protostream marshaller9 that is based on Protobuf serialization format developed by Google10.

Transactions Hot Rod clients are able to participate in Java Transaction API (JTA) transactions11. However, the support is limited: • Isolation level has to be REPEATABLE_READ, which provides a guarantee that data will not change for the duration of the transaction once it has been read for the first time. • Locking must be PESSIMISTIC. Locks are obtained on a write operation, while in OPTIMISTIC setting the lock is received at the end of a transaction, where transaction manager checks for data consistency. • Transaction mode can be either NON_XA or NON_DURABLE_XA. NON_XA transactions are local and specific to one resource or database. On the other hand, NON_DURABLE_XA transactions are global and can span across multiple resources.

8. https://jbossmarshalling.jboss.org 9. https://github.com/infinispan/protostream 10. https://developers.google.com/protocol-buffers 11. https://javaee.github.io/javaee-spec/javadocs/javax/transaction/ TransactionManager.html

18 3. Infinispan

Querying In client/server mode, Infinispan supports querying through their query language called Infinispan Query DSL. Example: import org.infinispan.query.dsl.*;

// get the DSL query factory from the cache: QueryFactory qf= org.infinispan.query.Search.getQueryFactory(cache);

// create a query for all the books with a title containing \say{engine}: org.infinispan.query.dsl.Query query= qf.from(Book.class) .having("title").like("%engine%") .build();

// get the results: List list= query.list();

To enable indexing of entities, the server has to know the structure of data. Therefore, Infinispan encourages users to utilise Protostream- Marshaller, which forces users to create entity schemas in the form of “.proto” files with an option to specify indexed fields. These schemas are sent to the server on client initialization. Moreover, Infinispan provides an automatic way to generate schemas through an annotation processor.

4 Application platform

The application server is a framework that provides an environment for creating enterprise web applications. Java application servers are based on the Java Enterprise Edition(Java EE) standard, with the latest version being Java EE 8. For an application server to be Java EE com- pliant, it has to provide implementations for numerous specifications. For instance, the most notable being: • Java Persistence API (JPA):interacts with the database, provides object-relational mapping with annotations • Java Transaction API (JTA): adds functionality for transactions that can span multiple resources • Java API for RESTful Web Service (JAX-RS): provides the functionality to create web services corresponding to RESTful architectural style • Context Dependency Injection (CDI): framework enabling the code to follow SOLID principle1 This chapter analyses all application servers provided by Red Hat one by one and ends with a decision which one to utilise for implementation.

4.1 JBoss EAP

JBoss Enterprise Application Platform is Red Hat supported application server of Wildfly that provides modular, cloud-ready architecture and robust tools for management and administration. Additionally, JBoss EAP Java EE 8 certified and offers tools for high-availability clustering, distributed caching, messaging and transactions. Moreover, JBoss EAP provides a command-line interface for run-time modification of configuration[8]. It is a mature and thoroughly tested product with a history spanning at least 17 years2.

1. https://itnext.io/solid-principles-explanation-and-examples-715b975dcad4 2. Archive - https://jbossas.jboss.org/downloads

21 4. Application platform 4.2 Thorntail

Thorntail3 is an application server primarily tailored for development of microservices. The core of Thorntail is also Wildfly, but Thorntail offers users to choose which parts they need for their service instead of getting the full unit. Therefore, the service is lesser in size and generates a smaller memory footprint. Furthermore, Thorntail implements MicroProfile4specification, which includes standards optimized for microservices such as health checks, metrics, reactive messaging, OpenApi5 generation, fault tolerance and more.

4.3 Quarkus

Quarkus6 is a new and innovative application platform similar to Thorntail in the basic concepts. These include marketing towards microservices and implementing most of Java EE 8 and MicroProfile specifications. Quarkus is revolutionary in the sense that it enables native Java compilation, which is possible due to a new universal virtual machine called GraalVM7. GraalVM allows for immensely better boot times, even smaller memory footprints and overall better performance. However, GraalVM has a considerable number of limitations. Java libraries and frameworks that use reflection8 will not compile, unless specific classes that use reflection are registered ahead of compilation. Additionally, dynamic class loading is not supported, which is a routine approach in Java to achieve modularity. The full list is posted here9. To counter, Quarkus offers a vast amount of libraries that suffice most cases and compilation for regular JVM.

3. https://thorntail.io/ 4. https://microprofile.io/ 5. https://swagger.io/docs/specification/about/ 6. https://quarkus.io/ 7. https://www.graalvm.org/ 8. https://www.oracle.com/technical-resources/articles/java/ javareflection.html 9. https://github.com/oracle/graal/blob/master/substratevm/ LIMITATIONS.md

22 4. Application platform

Another innovation that Quarkus brings is live coding with development mode, background compilation10 and overall more comfort- able support for integration testing.

4.4 Decision

As one of the goals of microservice architecture is small and fast deployment, JBoss EAP is the least suitable due to providing all functionality of Java EE 8 in contrast to both Thorntail and Quarkus. Thorntail has the advantage of being much more mature with more comprehensive documentation and fewer potential bugs. However, Thorntail ceased proactive development in support of Quarkus.[9] Therefore the decision is to use Quarkus.

10. https://quarkus.io/guides/getting-started#development-mode

5 Design

5.1 Requirements

The requirements were defined through a cooperation with possible future users and the aid of the MoSCoW method.1

5.1.1 Must have

• Dependency resolution: For a task to start, it has to wait for dependencies. (implemented)

• Remote execution: The scheduler has to have the ability to invoke execution on a remote entity. (Remote entity could be as a server where task executes) (implemented)

• Circle detection: Prevent requests that would create a circle in a dependency graph. (implemented)

• Ability to run in cluster (verified)

• Task cancellation: User has to have an option to cancel a task. The scheduler needs to inform the remote entity if the task is running and prevent the execution of dependent tasks. (implemented)

• Public RESTful API: The scheduler has to provide a public API that users can use to submit, cancel and get information about tasks. (implemented)

• Data consistency: In a state of failure, data can’t be left in an inconsistent state and have to be rolled back. (verified)

5.1.2 Should have

• Grouping: User should have an option to schedule multiple tasks in one request. (implemented)

1. https://www.productplan.com/glossary/moscow-prioritization/

25 5. Design

• Notification on state updates: Send a message when a taskhas transitioned to a new state. (not implemented)

• Prevention against duplicate execution: The scheduler has to invoke remote entity exactly once per task. (implemented)

5.1.3 Could have

• Graceful shutdown: If the scheduler is shutdown, it should stop accepting new tasks and wait for existing ones to complete. (not implemented)

5.1.4 Remote entity requirements

• REST endpoints for starting and cancelling the task.

• On the invocation to start a task,

– if the task starts successfully, return positive HTTP status code (200) – if the task fails to start, return negative HTTP status code (400)

• Inform the scheduler on successful and failed execution by callback given by scheduler on the first request.

5.2 Differences and similarities to JBoss MSC

The core of the implementation is inspired by design from the analysis of JBoss MSC, which is considerably adjusted for working with transactions, remote entities and clusterable cache.

5.2.1 Naming changes

The naming changes are introduced due to an unsuitable meaning occurring when transitioning from MSC to the scheduler. In MSC Services can be compared to processes where the goal is to have all dependencies in a state of execution to use their resources. In contrast, the scheduler’s goal is to have all dependencies finished. This leads

26 5. Design

to a decision to change naming from Services to Tasks. As Task had a meaning in MSC’s context, another transition was needed. The ade- quate candidate is Job.

Changes:

• Service -> Task

• Task -> Job

5.2.2 What stayed

• API for Target, Registry, Container, Dependent and Controller is mostly the same.

• Principles of States, Modes, Transitions and ControllerTasks.

• Identification with ServiceName

5.2.3 What changed

• TaskController data moved to a separate entity (Task).

• Data consistency is assured by transactions instead of synchronized blocks.

• The TaskRegistry carries data in Infinispan RemoteCache instead of ConcurrentMap

• Dependency interface is removed. MSC used this interface to place demands on dependencies to start, whereas this situation does not occur for the scheduler. (users place the demand)

• Removed ServiceRegistrationImpl.

• Removed Service interface as that interface served for local execution. (moved to remote entities)

• The Target supports requests for multiple Tasks.

• StartTask renamed to InvokeStartJob, which invokes starting endpoint of remote entity.

27 5. Design

• StopTask renamed to InvokeStopJob, which invokes cancelling endpoint of remote entity.

• Jobs are not async due to a limitation placed by transactions.

• The Controller has two new methods accept() and fail() which signify positive or negative response from a remote entity.

• States, Jobs, Modes and Transitions have been remodelled.

FINAL

STOPPED accept() RUNNING entry / DependencyCancelledJob or DependencyStoppedJob

accept() SUCCESSFUL entry / DependencySuccessfulJob

STOPPING STOP FAILED entry / InvokeStopJob fail() entry / DependencyStoppedJob

setMode (Mode.CANCEL)

IDLE UP FAILED setMode fail() entry / DependencyStoppedJob (Mode.CANCEL)

WAITING

setMode(Mode.Active), dependencySucceeded() accept() [unfinishedDeps <= 0 && STARTING START FAILED setMode(Mode.ACTIVE) mode = Mode.Active] entry / InvokeStartJob fail() entry / DependencyStoppedJob [unfinishedDeps > 0]

NEW entry / setMode(Mode.NEVER)

setMode(Mode.Cancel), dependencyStopped()

Figure 5.1: State-machine diagram of a Task

5.3 States, Transitions, Modes and Jobs

States, Transitions and Jobs described are taken from Figure 5.1.

28 5. Design

5.3.1 Modes

• IDLE: Controller does not attempt to start a Task.

• ACTIVE: Controller is actively trying to start a Task.

• CANCEL: Signal to a Controller to cancel the Task and its dependents recursively.

5.3.2 Jobs

• InvokeStartJob: Invokes start endpoint on a remote entity. Ac- cording to the status code in response, Job calls accept() or fail().

• InvokeStopJob: Invokes stop endpoint on a remote entity. Accord- ing to the status code in response, Job calls accept() or fail().

• DependencyStoppedJob: Calls Dependent.dependencyStopped() on each dependent. The method sets the StopFlag to DEPEN- DENCY_FAILED. Additionally,calling dependencyStopped() will cause a transition to STOPPED state, which will force the dependent Task to create another DependencyStoppedJob. Therefore, executing one DependencyStoppedJob will recursively stop every dependent.

• DependencyCancelledJob: Calls Dependent.dependencyCancelled() on each dependent. The method sets the StopFlag to CAN- CELLED. It has the same propagating mechanism as Dependen- cyStoppedJob.

• DependencySuccessfulJob: Calls Dependent.dependencySucceeded() on each dependent. The method decreases the number of un- finishedDependencies by one. If the resulting number is 0, the dependent will attempt to start.

5.3.3 StageGroups

• IDLE: Task is unproductive and either waiting for dependencies or is in IDLE Mode.

• RUNNING: Task is remotely active.

29 5. Design

• FINAL: Task is in a final state, and it cannot transition further.

5.3.4 States

• NEW: Initial state.

• WAITING: Task is waiting for a dependency to succeed.

• STARTING: Task is attempting to start on a remote entity. Tran- sition into this state causes the invocation of start endpoint on a remote entity.

• START FAILED: Start on the remote entity failed. The Task has to inform its dependents it failed, therefore it runs Dependen- cyStoppedJob.

• UP: Task is running on a remote entity.

• STOPPING: User has requested a cancellation of a task. The Task is remotely active, therefore the scheduler has to inform the remote entity to terminate the Task. That means that the scheduler has to execute InvokeStopJob.

• STOP FAILED: Stopping on the remote entity failed. Therefore the scheduler runs DependencyStoppedJob.

• FAILED: The Task failed to complete on the remote entity. The remote entity invoked internal endpoint of the scheduler with a negative message. Therefore the scheduler runs Dependen- cyStoppedJob.

• SUCCESSFUL: The Task completes on the remote entity. The remote entity invoked internal endpoint of the scheduler with a positive message. The Task needs to inform its dependents it has succeeded.

• STOPPED: The Task was informed to stop. Its direct dependency was either cancelled or failed. In case the dependency failed, it runs DependencyStoppedJob, and if it was cancelled, it runs DependencyCancelledJob. (the decision is based on StopFlag)

30 5. Design

Figure 5.2: The diagram of package dependencies in the scheduler

5.4 Modules

Project hierarchy:

• common: Classes and enums shared across the whole project.

• core: The package that contains whole business logic and datastore communication. It exposes API in the form of interfaces for other packages to use.

• model: Contents of the module are data models of entities, that are saved in the datastore.

31 5. Design

• dto: Holds Data transfer object2 entities, used by REST for responses and requests.

• facade: Connects rest and core modules. Additionally, it includes mappers from the data model to the dto model and vice versa.

• rest: Exposes RESTful endpoints for users.

2. https://www.javaguides.net/2018/08/data-transfer-object-design-pattern-in- java.html

32 6 Implementation

This chapter goes through details of working with the libraries and frameworks such as Quarkus, CDI, Infinispan, JTA, JAX-RS and Map- struct. Furthermore, it mentions problems, obstacles and difficulties encountered during implementation with a little fallback due to lack- ing documentation. It concludes with a guide on how to start up the scheduler.

6.1 Context dependency injection

Context dependency injection (CDI) is a standard from Java EE. De- pendency injection is a programming technique that makes a class independent of its dependencies through delegation of object creation to a container. The container makes sure the dependencies are created beforehand and injects it to the dependent class. Therefore, the class has available dependencies, and additionally, it enables the class to be loosely-coupled. Quarkus provides its own implementation of the CDI framework by the name of ArC. The usual implementations of CDI like Weld, Google Guice and Spring are computing the dependencies in run-time. However, Quarkus shifts the computation to compile-time, which introduces a couple of difficulties. Firstly, the CDI standard is not implemented to the fullest1. Secondly, the injection is quirky across Maven modules.

6.1.1 Maven module problem

The first version of scheduler was designed with a rest, facade and endpoint packages being in separate maven modules. Modules were linked by dependency injection. As Quarkus needs every dependency defined during build-time, it prevents any misconfiguration. Nonethe- less, after a successful build, the scheduler would get stuck on initialization without any logs displayed. Therefore, debugging the problem was complicated and ultimately led to a decision to merge the modules into one.

1. Limitations: https://quarkus.io/guides/cdi-reference#limitations

33 6. Implementation 6.2 Transactions

Transactions are vital to ensure data consistency across clusters. Each REST endpoint invocation is handled by a separate thread, which means that even with a single node, there is a possibility for concurrent updates of the same data. An example being, there are two running tasks with a common dependent. Both complete at the same time, therefore their associated remote entities send a notification. The notifications are accepted at the same time by a single node, and a distinct thread handles each request. The common dependency is loaded from datastore and updated by each thread. This produces a data inconsistency due to each thread modifying data of the same origin and version. With transactions, one of the threads would get rolled back, because datastore can de- tect these inconsistencies. However, the rolled back notification about completion of a Task is necessary for the common dependency ever to start. Therefore, there needs to be a mechanism which would retry the operation again. The retried operation would retrieve updated data and successfully commit. The most significant benefit of solving transaction conflicts ona thread level is that it does not matter from where a thread originates. It would be the same situation if the notifications were sent to different nodes. One of the transactions would get rolled back, and the thread would have to retry the operation.

6.2.1 Partial updates

The transaction boundaries have to be set in a way to avoid partial updates. For that reason, every request to the scheduler has to be atomic, which implies that either everything happens or nothing happens.

6.2.2 Prevention of duplicate execution

Duplicate execution could occur if a transaction that invoked a start on a remote entity failed and was consequently retried. As retrying repeats all operations, it would invoke again. To prevent the incident from happening, the invocation has to be separated from the transaction and only take place if the transaction succeeds. The solution

34 6. Implementation

is to implement a unique type of a Job. This Job executes in a new thread and creates a separate transaction, but only if the transaction from a thread that this Job originated from succeeds. Propagating status of the transaction to another thread was possible due to apply- ing Microprofile-context-propagation2 specification. Therefore, for an invocation to happen, the underlying transaction has to succeed.

6.3 Mapping

Mapping fields from an object to another object can be done by handor by using a mapping framework. Writing mappers by hand is tedious and error-prone. Additionally, if the schema of the object changes, the number of places that a programmer has to modify increases. It is typically more beneficial to leave the work to another library. Most mapping frameworks like Dozer3 and ModelMapper4 use reflection to inspect the fields of an object, which is problematic with Quarkus. However, libraries like Mapstruct5 use an annotation processor to generate injectible mappers with just a standard Java. Mapstruct defines a set of annotations by which you specify which field from one object maps to the other. If a field has the samename and type on both sides, Mapstruct creates mapping automatically. Additionally, Mapstruct automatically maps around collections and nested classes, if there is a mapper available.

6.4 REST

The scheduler exposes two major endpoints, one is internal and serves as a mechanism for remote entities to inform the scheduler about Task completion or failure, and the second is for users of the scheduler to request scheduling, cancel Tasks and retrieve information about them. The endpoints are implemented with JAX-RS standard. Additionally, the OpenAPI document is generated for the whole public API. It is possible due to an extension called smallrye-open-

2. https://github.com/eclipse/microprofile-context-propagation/ 3. https://github.com/DozerMapper/dozer 4. http://modelmapper.org/ 5. https://mapstruct.org/

35 6. Implementation api6 that Quarkus provides. It enables the user to use annotations on their endpoints, from which the OpenAPI document is produced. Furthermore, Quarkus exposes a Swagger UI7, that enables users to visualize and use the endpoint directly through the browser. Internal endpoint: • POST “/rest/internal/{TaskID}/finish”: The endpoint requires a path parameter “TaskID” to identify, which Task is in context, and request body with a boolean signifying completion or failure. Task endpoint: • POST “/rest/tasks”: The purpose of the endpoint is to schedule Tasks. The body of the request contains an array of TaskDTOs. Additionally, the scheduler can return 404 Bad Request if the array creates a circle. • GET “/rest/tasks”: The endpoint returns all tasks it has in the datastore. There is an optional body, where the user can specify filters by StateGroup. • GET “/rest/tasks/{TaskID}”: The endpoint returns information about specific Task identified by path parameter “TaskID”. • PUT “/rest/tasks/{TaskID}”: The endpoint cancels the execution of a Task specified in the path parameter.

6.5 Installation

This section describes necessary prerequisites, the process of setting up and configuring an Infinispan cache-store, and how to compile and consequently run the scheduler.

6.5.1 Prerequisites

1. Apache Maven 3.5.3+ (to compile and build from the source code).

6. https://github.com/smallrye/smallrye-open-api 7. https://swagger.io/tools/swagger-ui/

36 6. Implementation

2. OpenJDK 1.8 or OpenJDK 11 of any distribution (OpenJDK 11 prefered).

3. Infinispan Server of version 9.4.16.Final available for download from8.

6.5.2 Setting up an Infinispan server

To start up the scheduler, it has to have available datastore. Addition- ally, the data store has to be pre-configured with a definition of cache for Tasks. To set up the server, follow these steps:

1. Download the infinispan-server-9.4.16.Final.zip. wget

2. Unzip the file. unzip infinispan-server-9.4.16.Final.zip

3. Start the server infinispan-server-9.4.16.Final/bin/standalone.sh -c clustered.xml

(to make it available on LAN, supply the host/ip-address with an additional option -b="ip-address", otherwise it defaults to localhost)

4. Wait until the server is operational.

5. Open another terminal on the same path.

6. Use the configuration script available in the scheduler’s repository. The script configures the needed cache, avoids potential port conflicts with Infinispan REST server and restarts it. infinispan-server-9.4.16.Final/bin/ispn-cli.sh --file="path/to/scheduler/server-config.cli"

8. https://downloads.jboss.org/infinispan/9.4.16.Final/infinispan-server- 9.4.16.Final.zip

37 6. Implementation 6.5.3 Compilation and execution

During the Maven packaging phase, Quarkus creates a runner jar file which is executable with a java -jar command. To run the scheduler, follow these steps:

1. Open the scheduler repository directory in terminal.

2. Compile the scheduler without running the tests. mvn clean install -DskipTests

3. Execute the scheduler. java[-options] -jar core/target/core-0.1-SNAPSHOT-runner.jar

Additional options:

(a) “-Dquarkus.infinispan-client.server-list= :11222”: Host/ip-address of Infinispan Hot-Rod server. Default value is “localhost:11222”. (b) “-Dscheduler.baseUrl=”: URL address of the scheduler which is used for callbacks sent to remote entities. Default value is “http://localhost:8080/”. (c) “-Dquarkus.http.port=”: Port where the application is deployed on. Default value is “8080”.

38 7 Testing

Testing is a crucial part of software development. Tests ensure the implementation is behaving as expected, meets the requirements and prevents unexpected regression from appearing. This chapter describes local integration testing and tests done in a simulated cluster by hand.

7.1 Local integration testing

Integration tests are the second level of software testing behind unit tests, which test individual components of the system. Integration testing is used to determine whether separately developed components work together as expected. To define the tests, JUnit1 5 testing framework is utilized. JUnit allows users to set the boundaries of the test and define common pre-conditions and post-conditions. Occasionally, tests tend to be complex and confusing, therefore to achieve fluent and more comprehensible syntax, AssertJ library is used. Examples of AssertJ usage can be found here2. Furthermore, Quarkus supports dependency injection in integration tests, which makes the testing very convenient. The tests made for the scheduler verify cases such as: • simple datastore retrieval and store • correct Task format • dependencies triggering the execution of dependent tasks • Infinispan queries • cycles in a complex graph of Tasks • cancellation propagation to all dependents • execution of sophisticated Task graph ending up with all Tasks in a successful state To run the integration tests, these actions are needed: 1. Start a configured Infinispan Server as described in subsection 6.5.2 2. Open a new terminal.

1. https://junit.org/junit5/ 2. Examples: https://assertj.github.io/doc/ 39 7. Testing

3. Change the directory to the repository with the scheduler. 4. Build the scheduler and run the tests. mvn clean install

7.2 Clustered testing

The first step for clustered testing is to set up a simulated cluster of nodes made of the scheduler. The nodes can be either on one or multiple machines sharing a LAN. In the case of one machine, the tester needs to ensure that the schedulers are deployed on different ports. Furthermore, a separate project with an example of a remote entity was created3. The entity has two endpoints, one notifies back the scheduler through a callback, and the other does not. Therefore, we are able to test the communication between the services and simulate the scheduling of Tasks. These steps were taken to simulate a cluster: 1. Start a configured Infinispan Server as described in subsection 6.5.2 and set the ip-address on server corresponding to the machine’s local network address. 2. Start several instances (on single or multiple machines) of the scheduler as described in subsection 6.4.4 and specify baseUrl, address of Infinispan server and HTTP port if required. 3. Choose a machine for a mocked remote entity. 4. Clone the repository with the entity. git clone https://github.com/michalovjan/mock-remote-entity.git

5. Enter the directory. cd mock-remote-entity

6. Compile the entity. mvn clean install

3. https://github.com/michalovjan/mock-remote-entity.git

40 7. Testing

7. Run the entity. (the entity deploys on port 8090 by default) java -jar target/mockremoteentity-1.0.0-SNAPSHOT-runner.jar

Concurrent updates on a single node

Testing of concurrent updates on a single node is done with a request to schedule three Tasks, where one Task has two dependencies. The request must include the mocked remote entity. The invocations to start the Tasks and consequent responses are performed at the same time. The responses create concurrent transactions. One of them fails due to a write to the same place (common dependant). The failed transaction is rolled back and retried to ensure data consistency.

Concurrent updates on distinct nodes

Testing of concurrent updates on different nodes is done with the same request mentioned above. The difference is that the remote entity does not inform the scheduler about the completion of a Task. The notifications are sent manually to different nodes at the same time, which can be done with "curl" command and two terminals. The behaviour is the same as on a single node.

8 Conclusion

The goal of this thesis was to create a general solution for remote task dependency resolution with an ability to run in a cluster. The solution has to be able to prevent data inconsistencies in a concurrent environment, cancel a remote task and prevent duplicate execution. Users have to have an option to access the scheduler through publicly available REST API. Additionally, the implementation should have been published as an open- source project.

To implement the solution, I’ve made thorough and detailed research of existing or partial solutions in the form of analyzes of JBossMSC, Infinispan and application platforms, which are described in chapters two, three and four. According to the knowledge gained, I’ve designed a solution and subsequently created an implementation. The implementation was tested and verified, that it conforms the requirements set by future users.Afterwards, the solution was published as an open-source project on GitHub1.

8.1 Future improvements

Upper limit of running concurrent Tasks

To further configure the scheduler, there could be a cluster- wide upper limit for running Tasks. To implement this feature, I would use a clustered counter provided by Infinispan, that provides an atomic API for use in a concurrent environment. Next task to execute, could be chosen by randomly, by creation date or by a custom algorithm with a heuristic.

1. https://github.com/michalovjan/remote-scheduler

43 8. Conclusion

Graceful shutdown

A graceful shutdown was mentioned in the requirements as could-have. The feature could be implemented by a cluster- wide flag that could be triggered by any of the nodes inthe cluster. The trigger would block incoming requests and prevent further Tasks from executing.

Notification on state updates

It was mentioned as should-have in the requirements. For each transition, there would be a new Job executed, which would sent the information into message queue or web-sockets. It would be up to users, to intercept the message and act accordingly. Additionally, the mentioned Job would have to have the same prevention against duplicate execution as the InvokeStartJob had.

44 Bibliography

1. LEHMAN, E.; LEIGHTON, F.T.; MEYER, A.R. Mathematics for Computer Science. 2018. Available also from: https://courses. csail.mit.edu/6.042/spring18/mcs.pdf. Chapter 10.5 Directed Acyclic Graphs and Scheduling. 2. Wildfly 18 - High Availability guide [online] [visited on 2019-12- 12]. Available from: https : / / docs . wildfly . org / 18 / High _ Availability_Guide.html. 3. Application Clustering For Scalability and High Availability [online] [visited on 2019-12-12]. Available from: https://dzone.com/ articles/application-clustering. 4. Stateless Over Stateful Applications [online] [visited on 2019-12- 12]. Available from: https://medium.com/@rachna3singhal/ stateless-over-stateful-applications-73cbe025f07. 5. DRAGONI, Nicola; GIALLORENZO, Saverio; LAFUENTE, Al- berto Lluch; MAZZARA, Manuel; MONTESI, Fabrizio; MUSTAFIN, Ruslan; SAFINA, Larisa. Microservices: Yesterday, Today, and To- morrow. In: Present and Ulterior Software Engineering. Ed. by MAZ- ZARA, Manuel; MEYER, Bertrand. Cham: Springer International Publishing, 2017, pp. 195–216. ISBN 978-3-319-67425-4. Available from DOI: 10.1007/978-3-319-67425-4_12. 6. API Documentation of JBoss MSC [online] [visited on 2019-12-09]. Available from: https://jboss- msc.github.io/jboss- msc/ apidocs/. 7. User and reference guide of Infinispan [online] [visited on 2019-12- 10]. Available from: https://infinispan.org/docs/9.4.x/ user_guide/user_guide.html. 8. Red Hat JBoss Enterprise Application Platform technology overview [online] [visited on 2019-12-11]. Available from: https://www. redhat.com/en/resources/resources-red-hat-jboss-enterprise- application-platform-technology-overview-html. 9. Thorntail Community Announcement on Quarkus [online] [visited on 2019-12-11]. Available from: https://thorntail.io/posts/ thorntail-community-announcement-on-quarkus/.

A Attached files

scheduler.zip

• A zip file containing maven-based source code of the scheduler and README.md file with instructions for installation.

• The source code open-source and also available on https:// github.com/michalovjan/remote-scheduler.