Composite Event Recognition for Maritime Monitoring

NATIONAL AND KAPODISTRIAN UNIVERSITY OF ATHENS

SCHOOL OF SCIENCE DEPARTMENT OF INFORMATICS AND TELECOMMUNICATIONS

BSc THESIS

Composite event recognition for maritime monitoring

Manolis N. Pitsikalis

Supervisors: Alexander Artikis Researcher, NCSR “Demokritos” Assistant Professor, UNIPI Panagiotis Stamatopoulos Assistant Professor, NKUA

ATHENS JUNE 2018 ΕΘΝΙΚΟ ΚΑΙ ΚΑΠΟΔΙΣΤΡΙΑΚΟ ΠΑΝΕΠΙΣΤΗΜΙΟ ΑΘΗΝΩΝ

ΣΧΟΛΗ ΘΕΤΙΚΩΝ ΕΠΙΣΤΗΜΩΝ ΤΜΗΜΑ ΠΛΗΡΟΦΟΡΙΚΗΣ ΚΑΙ ΤΗΛΕΠΙΚΟΙΝΩΝΙΩΝ

ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ

Αναγνώριση σύνθετων γεγονότων για την παρακολούθηση της ναυτιλιακής δραστηριότητας

Μανώλης Ν. Πιτσικάλης

Επιβλέποντες: Αλέξανδρος Αρτίκης, Ερευνητής, ΕΚΕΦΕ «∆ημόκριτος» Επίκουρος Καθηγητής, ΠΑΠΕΙ Παναγιώτης Σταματόπουλος, Επίκουρος Καθηγητής, ΕΚΠΑ

ΑΘΗΝΑ ΙΟΥΝΙΟΣ 2018 BSc THESIS

Composite event recognition for maritime monitoring

Manolis N. Pitsikalis R.N.: 1115201300143

SUPERVISORS: Alexander Artikis Researcher, NCSR “Demokritos” Assistant Professor, UNIPI Panagiotis Stamatopoulos Assistant Professor, NKUA ΠΤΥΧΙΑΚΗ ΕΡΓΑΣΙΑ

Αναγνώριση σύνθετων γεγονότων για την παρακολούθηση της ναυτιλιακής δραστηριότητας

Μανώλης Ν. Πιτσικάλης Α.Μ.: 1115201300143

ΕΠΙΒΛΕΠΟΝΤΕΣ: Αλέξανδρος Αρτίκης, Ερευνητής, ΕΚΕΦΕ «∆ημόκριτος» Επίκουρος Καθηγητής, ΠΑΠΕΙ Παναγιώτης Σταματόπουλος, Επίκουρος Καθηγητής, ΕΚΠΑ ABSTRACT

Maritime monitoring systems support safe shipping as they allow for the real-time detection of dangerous, suspicious and illegal vessel activities. The intent of this thesis was the development of a composite event recognition engine for maritime monitoring and the construction of a set of patterns expressing effectively maritime activities in the Event Calculus. In this work, we use the Run-Time Event Calculus, a modern Prolog implementation of the Event Calculus along with tools allowing the compression of data streams, and the spatio-temporal link discovery. Additionally, to further improve the performance of recognition engine we extended the Run-Time Event Calculus with a dynamic grounding mechanism. Moreover, to increase the accuracy of the proposed system, we have been collaborating with domain experts in order to construct effective patterns of maritime activity. We evaluated our system in terms of predictive accuracy and efficiency using real kinematic vessel data.

SUBJECT AREA: Artificial Intelligence

KEYWORDS: event recognition, pattern matching, event calculus, grounding, prolog ΠΕΡΙΛΗΨΗ

Τα συστήματα θαλάσσιας επιτήρησης υποστηρίζουν την ασφαλέστερη ναυτιλία, καθώς επιτρέπουν την ανίχνευση σε πραγματικό χρόνο, επικίνδυνες, ύποπτες και παράνομες δραστηριοτήτες σκαφών. Η πρόθεση αυτής της πτυχιακής είναι η ανάπτυξη μίας αρχι- τεκτονικής συστημάτων εστιασμένη στην θαλάσσια επιτήρηση, καθώς και ενός συνόλου “μοτίβων”, ικανά να εφράσουν αποτελεσματικά ναυτιλιακές δραστηριότητες και συμβάντα. Σε αυτή την δουλεία χρησιμοποιούμε ως μήχανη αναγνωρίσης γεγονότων τον Λογισμό Γε- γονότων Πραγματικού Χρόνου, μία σύγχρονη υλοποιήση σε γλώσσα Λογικού Προγραμ- ματισμού, του Λογισμού Γεγονότων, καθώς επίσης ένα εργαλείο συμπίεσης τροχιών και ένα εργαλείο ευρέσης χωρικών σχέσεων. Για να βελτιώσουμε περαιτέρω την απόδοση της μηχανής αναγνωρίσης γεγονότων, δημιουργήσαμε ένα γενικό μηχανισμό δυναμικής θεμε- λίωσης ο οποίος φαίνεται να είναι αποτελεσματικός στα ναυτιλιακά δεδομένα. Επιπλεόν, μέσω της συνεργάσιας μας με τους ειδικούς του δημιουργήσαμε ένα σύνολο από μοτι- βά ναυτιλιακής δραστηριότητας, τα οποία και χρησιμοποιούμε στην πειραματική ανάλυση του συστήματος. Για την αξιολόγηση της προτεινόμενης αρχιτεκτονικής εστιάζουμε σε α- πόδοση και σε ακρίβεια, χρησιμοποιώντας δύο μορφές ροών πραγματικών δεδομένων πλοιών.

ΘΕΜΑΤΙΚΗ ΠΕΡΙΟΧΗ: Τεχνητή Νοημοσύνη

ΛΕΞΕΙΣ ΚΛΕΙΔΙΑ: αναγνώριση γεγονότων, αντιστοίχηση προτύπων, λογισμός γεγονό- των, θεμελίωση, λογικός προγραμματισμός To my family. ACKNOWLEDGMENTS

Firstly, I would like to express my sincere gratitude to my supervisor, Dr. Alexander Artikis for giving me the opportunity to work on this subject, and most importantly his support and deep knowledge on the field, during my research for my B.Sc. thesis. I would also like to thank Cyril Ray, Richard Dreo, Elena Camossi and Anne-Laure Jousselme for their valuable comments during my research on the field. Moreover, I would like to thank Paul Delaunay and Jules-Edouard Pouessel for their insights regarding maritime activity. Finally I would like to thank, assistant Professor Panagiotis Stamatopoulos, for his contribution on my thesis and for being inspiration in my academic course. This work was supported by the project datACRON, which has received funding from the European Union’s Horizon 2020 research and innovation programme under grant agreement No 687591. June 2018 CONTENTS

PREFACE ...... 14

1. INTRODUCTION ...... 15

1.1 Problem description ...... 15

1.2 Contribution ...... 15

1.3 Thesis structure ...... 16

2. RELATED WORK ...... 17

3. BACKGROUND ...... 20

3.1 Event Calculus ...... 20

3.2 Event Calculus for Run-Time reasoning ...... 22 3.2.1 Fluents ...... 23 3.2.2 Window mechanism ...... 25 3.2.3 Semantics ...... 26 3.2.4 Deadlines ...... 26

3.3 Maritime Monitoring ...... 27

3.4 Automatic Identification System - AIS ...... 28

3.5 Trajectory Synopses Generator ...... 28

3.6 Data streams ...... 28

4. MARITIME ACTIVITY PATTERNS ...... 30

4.1 Building blocks ...... 32 4.1.1 Communication gap ...... 32 4.1.2 Stopped vessel ...... 32 4.1.3 Vessel moving with low speed ...... 33 4.1.4 Vessel changing Speed ...... 34 4.1.5 Vessel in area of interest ...... 34

4.2 Maritime Situational Indicators ...... 35 4.2.1 Vessel under way ...... 35 4.2.2 Vessel with high speed near coast ...... 36 4.2.3 Vessel aground ...... 37 4.2.4 Vessel at anchor or moored ...... 38 4.2.5 Vessel with travelling speed ...... 40 4.2.6 Vessel’s movement ability is affected ...... 40 4.2.7 Vessel has speed incompatible with its type ...... 42 4.2.8 Vessel drifting ...... 43 4.2.9 Fishing ...... 44 4.2.10 Vessel engaged in search and rescue operation ...... 45 4.2.11 Vessels in rendezvous ...... 47 4.2.12 Vessel loitering ...... 48 4.2.13 Tugging ...... 48

5. DYNAMIC GROUNDING ...... 51

5.1 Building Domains ...... 51

5.2 Updating domains ...... 51

5.3 Algorithm ...... 52

6. EMPIRICAL ANALYSIS ...... 55

6.1 Experimental setup ...... 55

6.2 Experimental results ...... 56 6.2.1 Efficiency ...... 56 6.2.2 Accuracy ...... 59

7. SUMMARY AND FURTHER WORK ...... 62

ACRONYMS AND ABBREVIATIONS ...... 63

REFERENCES ...... 64 LIST OF FIGURES

Figure 1: A visual illustration of the three interval manipulation constructs of

RTEC. In this example, there are two input fluent streams, I1 and I2. The output of each interval manipulation construct is colored light blue...... 25

Figure 2: An illustration of deadlines. Arrows pointing up denote initiation points caused by initiatedAt rules while arrows pointing down indicate termination points caused by deadlines, lines ending in arrows indicate the intervals the erraticMovement fluent holds, lines ending in empty circles indicate that the deadline has been extended and lines ending in black circles denote that the deadline has been met...... 27

Figure 3: Processing steps prior to CER...... 29

Figure 4: A dependency graph of the recognised CEs...... 30

Figure 5: Vessel under way leaving the port of Brest, France. Yellow circles denote AIS messages, marked circles indicate critical points and the line represents the vessel trajectory reconstructed by the transmitted AIS signals. The absence of position signals is not always classified as a communication gap, since it may not be long enough according to the threshold of the synopsis generator...... 36

Figure 6: Vessel, near the port of Brest, France, with speed above the 5 knots limit near coast...... 37

Figure 7: Anchored vessel. Yellow circles denote AIS messages, marked circles indicate critical points, the line represents the vessel trajectory and the green dotted area is an anchorage area...... 39

Figure 8: General cargo vessel travelling with speed ≈ 10 knots. As illustrated on this Figure, it is common for vessels to travel long distances with travelling speed without changing direction...... 40

Figure 9: Container ship moving with speed 1 to 4 knots in the open sea, while its service speed is 19 knots...... 41

Figure 10: A vessel drifting. Yellow arrows denote AIS messages with their direction matching the course over ground value, similarly the direction of green arrows matches the true heading value, marked arrows are critical points...... 43 Figure 11: Illustration of a detected fishing vessels engaged in trawling. Dotted area expressed a fishing area...... 45

Figure 12: A confirmed case of a vessel in distress and a SAR vessel. Orange and blue circles denote the position signals of the two vessels and marked circles denote critical points...... 46

Figure 13: Rendez-vous between two fishing vessels...... 47

Figure 14: A vessel loitering...... 48

Figure 15: Illustration of a detected tugging activity...... 50

Figure 16: Illustration of input events (lower part), recognised intervals (middle part) and the vessel domain (upper part) for two recognition queries. Squares denote initiation/termination points, while lines not ending in a black square denote open intervals. In the left part, it is illustrated how the values of the coord input event are collected in order to build the domains, while in the left part, is additionally illustrated

how intervals computed at Q1 affect the updating process of the domains...... 52

Figure 17: Coverage for the Brest dataset...... 55

Figure 18: Coverage for the European dataset...... 56

Figure 19: Average recognition times with and without the dynamic grounding mechanism, when using as input the Enriched AIS Stream (left) and the Critical Point Stream (right) of the Brest dataset...... 57

Figure 20: Average recognition time when using the European dataset. . . . . 58

Figure 21: Average recognition times for each CE, using the enriched AIS stream and the critical point stream of the Brest dataset...... 58

Figure 22: Illustration of False Positives (FP) and False Negatives (FN) using the ‘underWay’ CE. Orange circles indicate AIS messages while marked orange circles denote critical points. Red (yellow) lines denote the CE intervals computed when consuming the enhanced AIS (critical point) stream. The two numbers attached to each AIS message express the time-point of the message and the speed of the vessel...... 59 LIST OF TABLES

Table 1: Main predicates of RTEC...... 23

Table 2: RTEC Deadline treatment...... 27

Table 3: Trajectory synopsis parameters...... 29

Table 4: Composite Event/Activity Recognition: Input events are presented above the double horizontal line, while the output stream is presented below this line. The input events above the single horizontal line are detected at the spatial preprocessing step while the remaining input events are detected by the trajectory synopsis generator (critical events). All output activities are durative...... 31

Table 5: Characteristics of the datasets used in our experiments...... 56

Table 6: Recognition Accuracy...... 61 PREFACE

The work presented henceforth was conducted during the period September 2017 to July 2018, at the NCSR “Demokritos”. We have created a Composite Event Recognition system for Maritime Monitoring. At the initial stages of this work, it was important to get familiar with maritime situations and terms in order to effectively address the challenges of Mar- itime Monitoring, as well as the Event Calculus, which is extensively used for the formalisation and the detection of maritime activities. In what follows, we present and evaluate the system we have created, along with the necessary background. Composite event patterns for maritime monitoring

1. INTRODUCTION

1.1 Problem description

Currently, one of the most important industries is the shipping industry. The International Marine Organization (IMO)1 states that 90% of the global trade is handled by the shipping industry. In order to ensure safe and secure shipping there has been an increased need for maritime surveillance. Maritime surveillance can be applied through maritime monitoring systems, these support safe and secure shipping as they detect, in real-time, dangerous, suspicious and illegal vessel activities. Such systems, typically use automatic tracking technologies such as the Automatic Identification System (AIS)2 which relies on messages transmitted by ships in an automatic way, or Synthetic Aperture Radar images which are usually provided by satellites or aircrafts. The system we present in this work uses AIS transmitted messages as its main source of data along with spatial and other static information.

1.2 Contribution

We have created a Composite Event Recognition (CER) system focusing on maritime surveillance. A CER system can derive Composite Events (CE)s from an input stream of instantaneous or durative simple, derived events (SDE)s (e.g., timestamped coordinates of a vessel). Our proposed system consists of three modern tools, a link discovery tool designed to create spatial relations between vessels with areas and other vessels [36], a trajectory synopsis generator capable of identifying the most important transmitted messages characterising the trajectory of a moving object [16], and an Event-Calculus based CER engine, the Run-Time Event Calculus (RTEC), that performs efficiently CER using patterns written in a logic programming language [8]. In collaboration with the domain experts of the datAcron project3 we have constructed a set of effective patterns for maritime monitoring, also known as ‘maritime situational indicators’ [21]. This set of patterns is an extension of some previous work by Delaunay, Paul and Pouessel, Jules-Edouard [14]. Moreover, we extend our CER system with “dynamic grounding” a mechanism that further increases the efficiency of our recognition engine. In order to evaluate our system, we focus on the efficiency of RTEC and its ability to handle large streams of transmitted vessel messages, and second we evaluate the quality of the trajectory synopses produced by the trajectory synopsis generator.

A part of this thesis, has been published and has been presented in the Hellenic Confer- ence of Artificial Intelligence [29].

1http://www.imo.org/en/OurWork/Environment/Pages/Default.aspx 2http://www.imo.org/en/OurWork/Safety/Navigation/Pages/AIS.aspx 3http://datacron-project.eu/

M. Pitsikalis 15 Composite event patterns for maritime monitoring

1.3 Thesis structure

The remainder of this thesis is organized as follows. In Chapter 2 we present some work on CER systems, and research related to maritime monitoring. In Chapter 3 we present the background of this work. Then, in Chapter 4 we present the maritime patterns in the language of RTEC. Next, in Chapter 5 we introduce the dynamic grounding and explain its algorithm. The evaluation of the system is presented in Chapter 6, while in Chapter 7 we summarise the work and discuss further research directions.

M. Pitsikalis 16 Composite event patterns for maritime monitoring

2. RELATED WORK

Efficient stream processing engines are necessary for the big data challenges. In [2] Affetti et al. analyse the execution semantics of five modern streaming processing engines Flink, Storm, Spark 1.6, Google Cloud Dataflow and Azure Stream Analytics. They focus on the notions of windowing and time, which are the most important factors for characterising the behavior of streaming processing engines. For their analysis, Affetti et al. use and evaluate the SECRET model, a model introduced in 2010 capable of analyzing window semantics of streaming processing engines. In their analysis, they aim to determine the strategy a window uses to make its content visible for downstream processing i.e, ready for the next processing stage if any, and the conditions that could trigger a report, conditions can be either time related or tuple related. In the case of time-related conditions a trigger is caused by the progress of time (time windows), while in the case of tuple-related conditions a trigger could be caused by a tuple addition (count windows). Flink, is an Apache project providing a Dataset and a DataStream API. Regarding the SECRET analysis Affetti et al. target on the DataStream API which is used for the processing of static and streaming data. Storm is a framework used for distributed real-time computation, and has been open- sourced after being acquired by Twitter. Storm uses connected operators forming a directly acyclic graph topology, where each operator’s behavior is defined using imperative code. Apache Spark is a general purpose cluster computing system used widely in batch and stream processing applications. Spark stores stores data in resilient distributed datasets (RDDs). An RDD is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. Google Cloud Dataflow is a stream processing engine for transforming and enriching data in stream and batch modes. The Microsoft Azure Stream analytics is a cloud service in which the developer defines the processing tasks in a SQL- like language, the Stream Analytics Query language. Their analysis points out that both Flink and Storm can operate with either time windows or count windows, while Spark 1.6 Google Cloud Dataflow and Azure use only time windows. All of the aforementioned stream processing engines report when a window close when the window content is not empty except Spark which reports regardless the window content. Cugola G. et al. in [13] survey a set of Information Flow Processing Systems and they propose a general, unifying model to capture the different aspects of an Information Flow Processing System. Cugola G. et al. define Information Flow Processing as an application domain in which users need to collect information produced by multiple, distributed sources, to process it in a timely way, in order to extract new knowledge as soon as the rel- evant information is collected. Results from their analysis point out that different systems fit different domains of applications, since the expressiveness of the rule language or the performance when used on large scale scenarios differ from system to system. Regard- ing the topology of Information Flow Systems, Cugola et al. state that some Information Flow Systems focus on an efficient centralized clustered implementation while other focus

M. Pitsikalis 17 Composite event patterns for maritime monitoring on minimizing communication costs by employing techniques such as filtering, correlation etc. Artikis et al. in [7] present a summary of some well known models and algorithms used in complex event recognition systems. Firstly, they analyse automata based models. Au- tomata can be used both for modelling and recognition. Some automata based systems are the Cayuga [9], SASE [25], SASE+ [45] and TESLA [12]. What makes automata suitable for complex event recognition modelling is that by definition they are able to represent sequences. Since automata are inherently non-deterministic the same input can produce different results in each run. Therefore, all candidate runs must be taken into consideration. Storing all possible runs can be very expensive since the set of runs can rapidly become extremely large. For this reason some systems represent runs in a compressed form. Artikis et al. present tree-based models, which similar to the automata-based models can be used both for modelling and recognition. A system using a tree-based model both for modelling and recognition is the ZStream. In this system patterns are defined by connecting primitives or complex events in a tree form using a defined set of operators. In order to perform recognition, ZStream loads a batch of events on the leaf nodes and evaluates potential conditions between their associated data-values, then it computes, propagates and verifies time constraints between events at all leaf nodes. Finally it as- sembles match results in a bottom up manner based on the semantics of the operators in the tree. Artikis et al. present logic-based models. In logic complex event recognition systems, semantics are expressed in some form of logic. They present two logic based for- malisms, the chronicle recognition and the event calculus. In chronicle recognition, events are formalised using predicates that define the content of the event and the time of the occurrence, while the Event calculus builds on fluents, a fluent is time varying fact, that its value may change from time to time. In the Event Calculus, a set of rules defines the event occurrence, the effects of events and the values of fluents. Regarding the maritime domain, maritime monitoring systems have been attracting con- siderable attention for economic as well as environmental reasons [20, 43, 18, 40, 41, 33, 21]. Terosso-Saenz et al. [41], presented a system detecting abnormally high or low vessel speed, as well as when two vessels are in danger of colliding. Their approach, uses event- based processing of AIS data, which as already mentioned is information transmitted by vessels, using a set of patterns and has been tested on both synthetic and real-world data. Van Laere et al. [42] evaluated a workshop aiming at the identification of potential vessel anomalies, such as tampering, rendez-vous between vessels, unusual routing, and so on. A framework focusing on anomally detection and classification of vessels interactions is presented in [39] by Hamed Yaghoubi Shahir et al. Their framework uses patterns represented by left-to-right Hidden Markov Models and classify them using Support Vector Machines. Another important work is the Search for Unidentified Maritime Objects (SUMO) automatic ship detector presented by Santamaria et al. [17]. The SUMO tool, developed by the Joint Research Program (JRC), is a set of algorithms, used for finding ships auto-

M. Pitsikalis 18 Composite event patterns for maritime monitoring matically or semi-automatically when human interaction is available, in synthetic aperture radar images and has contributed in maritime security by helping at the identification of illegal oil dumping, piracy and unsustainable fishing.

Another aspect of maritime monitoring concerns fishing activities [26]. Mills et al. [24], for instance, described a method for identifying trawling and steaming using speed and directionality rules, thus helping in studies of how fishing impacts on species, habitats and the ecosystem. Their research shows that during trawling, fishing vessels tend to have highly variable movements. Another work focusing on fishing activity detection using deep learning approaches, is presented by Jiang et al. at [19]. To deal with irregularities of AIS messages, such as the time interval between two messages, they under-sample the original data and additionally use linear interpolation techniques for missing trajectory components. In addition to fishing activities, there has been an interest on identifying Search and Rescue (SAR) operations. Chatzikokolakis et al. at [10] automatically detect SAR activities using transmitted AIS information. Their approach, combines attributes of AIS messages and additional calculated voyage related information such as port departure etc. which are later used as a training dataset, for their Random Forest model. Experimental results of their method on real world data, point out that during SAR voyages, vessels tend to have more changes in speed and heading than non-SAR voyages.

M. Pitsikalis 19 Composite event patterns for maritime monitoring

3. BACKGROUND

In this Chapter, first we describe briefly event calculus and its clauses. Second, we describe our recognition engine, Event Calculus for Run-Time reasoning (RTEC) and present an RTEC extension supporting ‘deadlines’. Finally we describe the tools we use and the creation process of the data-streams we use in our experiments.

3.1 Event Calculus

The Event Calculus is a logic formalism designed for representing and reasoning about events and their effects [22]. In contrast to the situation calculus, the Event Calculus instead of situations uses events and instead of global states it uses a local state for each time period a fluent, that is a time varying fact [38], holds continuously. Fluents are initiated and terminated by the happening of events. The Event Calculus formalisation is a set of clauses augmented by negation-by-failure and thus is a normal logic program. In this Chapter, we will be presenting the Event Calculus with Context Dependency as presented in [23]. Given a narrative—a narrative is a set of possibly incomplete specification of a set of actual event occurrences [35]—Event Calculus is able to infer the maximal validity intervals over which properties hold continuously. Using the notions of event, time- point and property as primitives Event Calculus defines a model of change where events happen at time-points and their occurrence can cause the initiation or/and termination of time intervals over which a property—similarly a fluent—holds. Context dependency in Event Calculus makes it possible to constrain the initiation and termination of properties to the validity of some given conditions at the time of events’ occurrences. In Event Calculus it is assumed that a property has persistence, in other words this means that a property holds until an event occurs that terminates it. The occurrence of an event is represented using the happens_at(Event, T imeP oint) clause, and a property is initiated or terminated using the two following domain-dependent clauses relating events and properties:

initiates_at(Event1 , P, T ): − happensAt(Event1 , T ). (1) terminates_At(Event2 , P, T ): − happensAt(Event2 , T ).

The initiates_at clause describes that the occurrence of an event initiates a period at which a property P holds, while in a similar manner the terminates_at clause describes that the occurrence of an event terminates a period at which a property P holds. The Event Calculus model of time and change is defined using the following domain independent

M. Pitsikalis 20 Composite event patterns for maritime monitoring predicates:

holds_for(P, [Start, End]) : −

initiates_at(Ei , P, Start),

terminates_at(Et , P, End), (2) End > Start,

not brokend uring(P, [Start, End]).

Rule (2) describes that a property holds continuously between time-point Start and time- point End, if an initiation event occurs at Start, a termination event occurs later in time at time-point End and the property hasn’t been broken in the meantime. The two following rules (3),(4) treat cases of persistence in the future and the past respectively.

holds_for(P, [Start, inf ]) : −

initiates_at(Ei , P, Start), (3) not broken_During(P, [Start, inf ]).

Rule (3) describes that a property P holds to infinity if there is an event initiating P at time-point Start and the property is not broken at any time later.

holds_for(P, [−inf , End]) : −

terminates_at(Et , P, End), (4) not brokenDuring(P, [−inf , End]).

Similarly to rule (3), rule (4) describes that a property holds from -infinity if there is a terminating event at time-point End, and the property is not broken until that time. The broken_during predicate is defined as follows:

broken_during(P, [Start, End]) : − (terminates_at(E, P, T ); initiates_at(E, P, T )), (5) Start < T , End > T . broken_during describes that a property P is broken within the period (Start, End) if there is an initiation or termination event occurring in that period. With this definition Event Calcu- lus takes into account only initiations of properties conforming to the strong interpretation of the iniatiates_at predicate, where a property holds for a period [Start,End] if there is an initiation event at Start and a termination event at End, and that property is not initi-

M. Pitsikalis 21 Composite event patterns for maritime monitoring ated or terminated in the meantime. In other words, in a sequence of initiating events and a termination event occurring after, the period at which P holds is defined by the latest initiation event and the termination event occurring after. However, this interpretation is not appropriate for all applications. For instance, in the maritime domain, where typically a narrative consists of messages containing speed and location information transmitted at a relative constant rate, if we want to recognize the period a vessel has speed above a threshold, a strong interpretation would give a false period since consecutive initiation events would not be taken into account. Properties expressing CEs of such type, should hold for the period [Start, End] if there is an initiation event at Start, a termination event at End and there isn’t a termination event in the meantime. Defining the periods a property holds this way, is called the weak interpretation of the initiates_at predicate. To apply the weak interpretation, the broken_during rule must take into account only termination events and, additionally rule (4) has to be omitted, since an initiation event occurring in the period (−inf, end) would not be taken into account.

holds_at(P, T ): − holds_for(P, [Start, End]), (6) Start < T , End > T .

Finally the holds_at clause expresses that a property P holds at time-point T if T is between the holding period [Start, End] of P . The holds_at clause can be additionally used in initiates_at/terminates_at rules, thus relating properties with other properties, and therefore allowing the creation of context dependent rules.

As you can see, Event Calculus allows with a set of rules the detection of the periods at which events hold. However, not all implementations of Event Calculus are suitable for Big-Data applications such as Maritime Monitoring. To this end, we chose to use Event Calculus for Run-Time reasoning (RTEC), an Event Calculus implementation that has already been used in applications such as Maritime Monitoring that uses several optimisation techniques making it suitable for real-time event detection.

3.2 Event Calculus for Run-Time reasoning

We perform CER using the ‘Event Calculus for Run-Time reasoning’ (RTEC) [8], a Pro- log implementation of the Event Calculus [22]. RTEC computes continuous narrative as- similation queries for pattern matching on data streams. It has already been used with success in CER applications such as public space surveillance [8], city transport & traffic management [8, 6, 37] and maritime monitoring [4, 28]. RTEC has a formal, declarative semantics—composite (vessel) activity formalisations are (locally) stratified logic

M. Pitsikalis 22 Composite event patterns for maritime monitoring

Table 1: Main predicates of RTEC.

Predicate Meaning happensAt(E,T ) Event E occurs at time T holdsAt(F = V,T ) The value of fluent F is V at time T holdsFor(F = V,I) I is the list of the maximal intervals for which F = V holds continuously initiatedAt(F = V,T ) At time T a period of time for which F = V is initiated terminatedAt(F = V,T ) At time T a period of time for which F = V is terminated union_all(L, I ) I is the list of maximal intervals produced by the union of the lists of maximal intervals of list L intersect_all(L, I ) I is the list of maximal intervals produced by the intersection of the lists of maximal intervals of list L relative_complement_all(I ′, L, I ) I is the list of maximal intervals produced by the relative complement of the list of maximal intervals I′ with respect to every list of maximal intervals of list L programs [30]. Moreover, RTEC includes various optimisation techniques for efficient pattern matching, such as ‘caching’, a technique that helps in avoiding unnecessary re- computations and ‘windowing’, whereby all input events that took place prior to the current window are discarded/‘forgotten’. The time model in RTEC is linear and includes integer time-points. Variables start with an upper-case letter, while predicates and constants start with a lower-case letter. Where F is a fluent—a property that is allowed to have different values at different points in time— the term F = V denotes that fluent F has value V . An event description in RTEC includes rules that define the event instances with the use of the happensAt predicate, the effects of events on fluents with the use of the initiatedAt and terminatedAt predicates, and the values of the fluents with the use of the holdsAt and holdsFor predicates. Table 1 summarises the main predicates of RTEC.

3.2.1 Fluents A fluent is a property that is allowed to have different values at different points in time. The holdsAt(F = V , T ) predicate of RTEC denotes that fluent F has value V at the time-point T . The holdsFor(F = V , I ) represents that a fluent F has value V continuously for all the maximal intervals in list I. Fluents are either ‘simple’ or ‘statically determined’. The two following Chapters describe in brief simple fluents and statically determined fluents.

Simple Fluents. For a simple fluent F , F = V holds at a time-point T if F = V has been initiated at some time-point before T , and has not been terminated in the meantime. As you might see, this is an implementation of the law of inertia. To compute maximal intervals I, we compute the initiation points of F = V through the use of initiatedAt rules and for each initiation point we find the first termination point which is calculated respectively with

M. Pitsikalis 23 Composite event patterns for maritime monitoring the use of a terminatedAt rule. initiatedAt (resp. terminatedAt) rules have the following form:

initiatedAt(F = V , T ) ← happensAt(E, T ), (7) conditions[T ].

The conditions[T ] set may include further constraints on time-point T, expressed as follows: • a (possibly empty) set of happensAt predicates, expressing the (non-)occurrence of events.

• a (possibly empty) set of holdsAt predicates, expressing constraints on fluents. • a (possibly empty) set of atemporal predicates, expressing non time-related constraints. An example of a rule-set defining the initiation and termination of a fluent expressing the time periods a vessel sails with low speed is the following:

initiatedAt(lowSpeed(Vessel) = true, T ) ← happensAt(slow_motion_start(Vessel), T ). (8) terminatedAt(lowSpeed(Vessel) = true, T ) ← happensAt(slow_motion_end(Vessel), T ). lowSpeed is a simple fluent, and slow_motion_start, slow_motion_end are events denoting that a vessel started or respectively stopped moving with low speed. Rule-set (8) describes that the fluent lowSpeed is initiated when a slow_motion_start event occurs and terminated when slow_motion_end event occurs. Statically Determined Fluents. Apart from domain independent definitions, events can be defined by means of application-dependent holds- For rules along with the interval manipulation constructs of RTEC union_all, intersect_all and relative_complement_all (see Table 1). For instance a statically determined fluent expressing the time periods a vessel is stopped or sails with low speed would be formalised as follows:

holdsFor(stoppedOrLowSpeed(Vessel) = true, I ) ←

holdsFor(lowSpeed(Vessel) = true, Il ), (9) holdsFor(stopped(Vessel) = true, Is ),

union_all([Il , Is ], I ).

In definition (9) lowSpeed, stopped are fluents expressing the time periods a vessel sails with low speed or it is stopped respectively and union_all is the interval manipulation con-

M. Pitsikalis 24 Composite event patterns for maritime monitoring

I1 I1 I1 I2 I2 I2 time time time (a) Union. (b) Intersection. (c) Relative Complement.

Figure 1: A visual illustration of the three interval manipulation constructs of RTEC. In this example, there are two input fluent streams, I1 and I2. The output of each interval manipulation construct is colored light blue. struct used for calculating the union of intervals. Below we present in detail all interval manipulation constructs of RTEC: union_all(+L, −I) computes the list I of maximal intervals representing the union of maximal intervals of the lists of list L (see Figure 1a). Consider the following example:

union_all([[(5, 20), (26, 40)], [(15, 35)]], [(5, 40)]). intersect_all(+L, −I) computes the list I of maximal intervals representing the intersection of maximal intervals of a set of lists (L) (see Figure 1b). For instance:

intersect_all([[(5, 20), (26, 40)], [(15, 35)]], [(15, 20), (26, 35)]). relative_complement_all(+I′, +L, −I) computes the list I of maximal intervals representing the relative complements of the list of maximal intervals I′ with respect to the union of the maximal intervals of the lists of list L. (Figure 1c illustrates an example). An example of the relative_complement_all is given below:

relative_complement_all([(10, 20), (30, 40)], [[(15, 20)], [(36, inf)]], [(10, 15), (30, 36)]).

A term of the form (Ts,Te) in the RTEC representation expresses the closed-open interval

[Ts,Te). Additionally, as illustrated in the above example, Te can be sometimes equal to inf, this means that a fluent currently holds, but we don’t know yet the time it will terminate —we will refer to intervals of this type as open intervals. On the whole, the use of interval manipulation constructs, leads to a concise definition of statically determined fluents.

3.2.2 Window mechanism As mentioned before, RTEC performs recognition using a window mechanism. A window is in fact the current working memory (WM) containing SDEs. CE recognition takes place at query times Q1,Q2,...,Qi,...,Qn where Qi − Qi−1 is a constant value. Both Qi − Qi−1 and WM are user defined values. At each query time RTEC computes the maximal inter-

M. Pitsikalis 25 Composite event patterns for maritime monitoring vals of CEs using SDEs that fall within the interval (Qi−WM ,Qi]. SDEs that occur before

Qi−WM are discarded. This makes the cost of CE recognition dependent only to the WM size and not to the whole narrative.

3.2.3 Semantics CE definitions in RTEC are (locally) stratified logic programs [30]. We restrict attention to hierarchical definitions, those where it is possible to define a function level that maps all fluent-values F = V and all events to the non-negative integers as follows. Events and statically determined fluent-values F = V of level 0 are those whose happensAt and holdsFor definitions do not depend on any other events or fluents. In CE recognition, they represent the input SDEs. There are no fluent-values F = V of simple fluents F in level 0. Events and simple fluent-values of level n are defined in terms of at least one event or fluent-value of level n−1 and a possibly empty set of events and fluent-values from levels lower than n−1. Statically determined fluent-values of level n are defined in terms of at least one fluent- value of level n−1 and a possibly empty set of fluent-values from levels lower than n−1.

Note that fluent-values F = Vi and F = Vj for Vi≠ Vj could be mapped to different levels. For simplicity however, and without loss of generality, a fluent F itself is either simple or statically determined but not both.

3.2.4 Deadlines

In several applications, there are CEs that require to be automatically terminated after a period of time. Such CEs could for example express the validity period of a contract, or a deadline period of some task. In the maritime domain, there are CEs described by SDEs occurring in a sequence within a period of time. For example, consider the following simplified definition for vessels moving erratically: A vessel is considered moving erratically, if there are consecutive changes in heading,

with time distance between two consecutive changes being less than Tmax. For the treatment of such definitions there is a mechanism called deadlines. The deadline mechanism allows the automatic termination of a fluent without a terminating event after a user-defined period from its latest initiation. In the language of RTEC the above definition is written as follows:

initiatedAt(erraticMovement(Vessel) = true, T ) ← happensAt(change_in_heading(Vessel), T ). (10)

maxDurationUE(erraticMovement(Vessel) = true, Tmax ). erraticMovement is a simple fluent, change_in_heading is an input event denoting that a vessel changes heading, and maxDurationUE describes that the fluent erraticMovement

M. Pitsikalis 26 Composite event patterns for maritime monitoring

t1 t2 t3 t4 t5 t6 Figure 2: An illustration of deadlines. Arrows pointing up denote initiation points caused by initiatedAt rules while arrows pointing down indicate termination points caused by deadlines, lines ending in arrows indicate the intervals the erraticMovement fluent holds, lines ending in empty circles indicate that the deadline has been extended and lines ending in black circles denote that the deadline has been met.

Table 2: RTEC Deadline treatment.

Predicate Meaning maxDuration(F,T ) Fluent F is terminated after T timepoints from its first initiation, if it is not terminated in the meantime maxDurationUE(F,T ) Fluent F is terminated after T timepoints from its latest initiation, if it is not terminated in the meantime

will terminate after Tmax time-points from its latest initiation. Therefore, erraticMovement is initiated when there is a change in heading, and terminated if there hasn’t been a change in heading for a time period equal to Tmax. An example of the recognized intervals is illustrated in Figure 2. As you can see, initiations occur at time-points {t1, t3, t4, t5} and terminations at time-points {t2, t6}. Time point t2 is a termination point because the deadline has been met t2 − t1 = tmax, similarly at t6 the deadline is met because t6 − t5 = tmax. Hence the recognized intervals are [(t1, t2), (t3, t6)]. Notice, that in the second interval the initial deadline period of t3 has been extended at time-points t4 and t5. We will discuss a similar fluent used for fishing CEs in further detail in Chapter 4. In addition to the maxDurationUE predicate, there is also the maxDuration predicate which similarly defines a deadline period but in contrast with maxDurationUE the deadline period can not be extended. In other words, maxDuration(F , T ) expresses that a fluent F will terminate after T time-points from its first initiation, if it is not terminated in the meantime.

3.3 Maritime Monitoring

Maritime monitoring is essential for maritime awareness, which is the knowledge of what and when is happening at sea. With Maritime Monitoring maritime traffic supervision is facilitated, environmental disasters can be prevented, law is better enforced and safety is improved. With Maritime Monitoring Systems it is able to detect in real-time a wide range of vessel activities and scenarios. Detected activities can either signify normal vessel be-

M. Pitsikalis 27 Composite event patterns for maritime monitoring havior such as anchoring or fishing, or they could possibly indicate activities or scenarios requiring immediate actions. The latter involves unlawful activities such as smuggling, illegal fishing, irregular migration etc. and unsafe scenarios such as a vessel in distress, two vessels in danger of collision etc.

3.4 Automatic Identification System - AIS

The Automatic Identification System (AIS) is a tracking technology for identifying and lo- cating vessels at sea through data exchange. AIS integrates a VHF transceiver with a positioning device (e.g., GPS), and other electronic navigation sensors, such as a gy- rocompass or rate of turn indicator, thus producing a wealth of valuable data regarding the vessel and its current status. The acquisition of positional data is achieved either by AIS base stations along coastlines, or even by satellites when out of range of terrestrial networks. Transmitted AIS messages contain either static information or dynamic information. In the case of static information, messages are transmitted every 6 minutes containing identity related information, such as the IMO number, the name of the vessel, its type, its draught etc. while in the case of dynamic information, messages are transmitted every 2 to 10 seconds depending on the speed of the vessel, or every 6 minutes when it is anchored. Dynamic information provided by these messages contains the Maritime Mobile Service Identity (MMSI) number, which is a number uniquely identifying each vessel, rate of turn, speed over ground, that is the actual speed calculated by the GPS signals, course over ground, heading etc.

3.5 Trajectory Synopses Generator

The Trajectory Synopses Generator is a tool used for providing a trajectory synopses of a moving object. This operation is performed by identifying, annotating and retaining only the most important transmitted positions, or as we call them ‘critical points’, of a moving object that can be used later for the recreation of the original trajectory. The annotation labels are described in the middle part of Table 4. Empirical results have indicated that at least 70- 80% of the input data may be discarded as redundant, while compression ratio can be up to 99% when the frequency of position updates is high [16]. In the maritime domain, where vessels tend to travel in long straight predictable routes, using the trajectory synopses can offer huge space savings, making it an important tool for efficient maritime monitoring.

3.6 Data streams

CER on vessel position signals, as defined in [27], requires, at the very least, two steps: (a) computing a set of spatial relations among vessels (such as proximity), and among vessels and areas of interest (e.g., fishing areas), and (b) labelling some of the position

M. Pitsikalis 28 Composite event patterns for maritime monitoring

Figure 3: Processing steps prior to CER.

Table 3: Trajectory synopsis parameters.

Parameter Value

Minimum speed vmin (knots) for asserting movement (stopped) 0.5

Maximum speed vθ (knots) for asserting slow motion 5 Minimum rate α for asserting speed change between locations 0.25 Minimum period ∆T (minutes) for asserting communication gap 1800 Angle threshold ∆θ (degrees)for asserting change in heading 4 signals as ‘critical’—such as when a vessel changes its speed, turns, stops, moves in a slow motion, or stops transmitting its position [16]. Figure 3 illustrates these steps. AIS position signals are streamed into the system, and go through a spatial preprocessing step, for the computation of the spatial relations required by the CE patterns. These relations are displayed in the top three lines of Table 4.

Then, some of the position signals are annotated as critical (see the middle part of Ta- ble 4). Subsequently, the position signals may be consumed by the CER component either directly (see ‘enriched AIS stream’ in Figure 3), or after being compressed, i.e. after removing all signals that have not been labelled as critical (see ‘critical point stream’). In Chapter 6, we present, through an experimental analysis, the effects of such a compression both on the efficiency and the accuracy of the system.

Critical point labeling is performed as part of trajectory synopsis generation, whereby major changes along each vessel’s movement are tracked. This process can instantly identify critical points along each trajectory, such as a stop, a turn, or slow motion. Using the retained movement features (critical points), we may reconstruct the trajectory of a vessel with small acceptable deviations from the original one. The parameters used for trajectory synopsis generation are presented in Table 3.

M. Pitsikalis 29 Composite event patterns for maritime monitoring

4. MARITIME ACTIVITY PATTERNS

In previous works, there has been developed a system for maritime surveillance focusing mostly on efficiency [27, 4]. Formalising accurately maritime activities, requires experts knowledge, to this end we have been collaborating with the domain experts of the datAcron project4 and we have constructed a set of patterns expressing activities of interest that can be used for efficient and effective maritime monitoring. With the detection of these activities, also known as ‘Maritime Situational Indicators’ [21] certain scenarios such as fishing pressure, illegal migration, illicit activities etc. can be detected. A dependency graph of the CEs we detect, created using the simplEC compiler [44] is illustrated in Figure 4. In what follows, we present a formalisation of these patterns in the language of RTEC.

Figure 4: A dependency graph of the recognised CEs.

4http://datacron-project.eu/

M. Pitsikalis 30 Composite event patterns for maritime monitoring

Table 4: Composite Event/Activity Recognition: Input events are presented above the double horizontal line, while the output stream is presented below this line. The input events above the single horizontal line are detected at the spatial preprocessing step while the remaining input events are detected by the trajectory synopsis generator (critical events). All output activities are durative.

Event/Activity Description entersArea(V , A) A vessel V enters an area A leavesArea(V , A) A vessel V leaves area A

Spatial proximity(V1 , V2 ) Two vessels V1 and V2 are close gap_start(V ) A vessel V stopped sending position signals gap_end(V ) A vessel V resumed sending position signals slow_motion_start(V ) A vessel started moving at a low speed slow_motion_end(V ) A vessel stopped moving at a low speed stop_start(V ) A vessel started being idle stop_end(V ) A vessel stopped being idle

Critical change_in_speed_start(V ) A vessel started changing its speed change_in_speed_end(V ) A vessel stopped changing its speed change_in_heading(V ) A vessel changed its heading gap(V ) A vessel V has a communication gap stopped(V ) A vessel V is stopped lowSpeed(V ) A vessel V moves slowly changingSpeed(V ) A vessel V changes speed withinArea(V ) A vessel V is in a specific area type underWay(V ) A vessel V is under way highSpeedNC (V ) A vessel V has high speed near coast

Composite aground(V ) A vessel V is aground atAnchor(V ) A vessel V is anchored moored(V ) A vessel V is moored atAnchorOrMoored(V ) A vessel V is anchored or moored travelSpeed(V ) A vessel V has travelling speed speedLtTr(V ) A vessel V has speed lower than travelling speed speedGtTr(V ) A vessel V has speed greater than travelling speed maa(V ) A vessel V has its movement ability affected vesselIST (V ) A vessel V has incompatible speed the its type adrift(V ) A vessel V is adrift trawlSpeed(V ) A fishing vessel V has trawling speed trawlingCourse(V ) A fishing vessel V has trawling course trawling(V ) A fishing vessel V is engaged in trawling activity

tugging(V1 , V2 ) Two vessels V1 and V2 are engaged in tugging activity

rendezVous(V1 , V2 ) Two vessels V1 and V2 are having a rendez-vous loitering(V ) A vessel V is loitering sarCourse(V ) Course of a SAR vessel V has SAR chararacteristics sarSpeed(V ) A SAR vessel V has speed indicating SAR activity activity inSAR(V ) A SAR vessel V is engaged in a SAR operation

M. Pitsikalis 31 Composite event patterns for maritime monitoring

4.1 Building blocks

We begin by presenting a set of building blocks that will be later used for the construction of more involved patterns.

4.1.1 Communication gap A communication gap takes place when no message has been received from a vessel for a long period of time. This may occur when the vessel sails in an area with no AIS receiving station nearby, or because the transmission power of its transceiver allows broadcasting in a shorter range, or when the transceiver is deliberately turned off. The rules below present a formalisation of communication gap:

initiatedAt(gap(Vessel) = Status, T ) ← happensAt(gap_start(Vessel), T ), happensAt(coord(Vessel, Lon, Lat), T ), (11) portDistance(Lon, Lat, Status). terminatedAt(gap(Vessel) = Status, T ) ← happensAt(gap_end(Vessel), T ). gap is a multi-valued simple fluent, gap_start and gap_end are critical events (see the middle part of Table 4), while coord is contextual information—the vessel’s coordinates— attached to the position signals. portDistance is an atemporal predicate that states whether a vessel is close to some port. We chose to distinguish between communication gaps occurring near ports from those occurring in the open sea, as the first ones usually do not have a significant role in maritime monitoring. Accordingly, Status in the above formalisation may be either ‘far from ports’ or ‘close to a port’. According to rule-set (11), a communication gap is said to be initiated when the synopsis generator emits a ‘gap start’ event, and terminated when a ‘gap end’ is detected. Given this rule-set, RTEC computes the time-points at which the gap fluent is initiated and terminated, and then computes the maximal intervals for which gap(vessel) = Status holds continuously.

4.1.2 Stopped vessel A vessel is stopped when it remains stationary (i.e., not moving) over a period of time. Vessels in general are stopped when they are anchored, moored or aground, there are though several scenarios when a vessel remaining stationary may indicate an engine fault

M. Pitsikalis 32 Composite event patterns for maritime monitoring or an illegal activity.

initiatedAt(stopped(Vessel) = status, T ) ← happensAt(stop_start(Vessel), T ), happensAt(coord(Vessel, Lon, Lat), T ), not happensAt(gap_start(Vessel), T ), portDistance(Lon, Lat, Status). (12) terminatedAt(stopped(Vessel) = _PortStatus, T ) ← happensAt(stop_end(Vessel), T ). terminatedAt(stopped(Vessel) = _PortStatus, T ) ← happensAt(start(gap(Vessel) = _GapStatus), T ). stopped is a multi-valued simple fluent, stop_start and stop_end are critical events (see the middle part of Table 4), and not is ‘negation-by-failure’ [11]. Similar to the gap fluent, in order to facilitate distinction between normal vessel behavior from illegal activities or distress scenarios, we categorize ‘stopped’ incidents by the vessel’s location (‘far from ports’ or ‘close to a port’). Therefore, the stopped fluent holds when a vessel is stopped (i.e, has speed lower than 0.5 knots) near ports or far from ports. Moreover, we make sure that for every detected activity the calculated intervals do not contain communication gaps as we do not have sufficient information about the vessel during that period. Thus we add a termination condition when a communication gap starts and respectively a constraint at the initiation rule. To avoid repetitions and a lengthy representation we will omit presentation of these constraints in following formalisations.

4.1.3 Vessel moving with low speed Typically a vessel moves with low speed when it is engaged in an activity that does not allow high speed such as tugging. Additionally, vessels sail with low speed in areas requiring great attention such as near ports or areas submitting to speed regulations.

initiatedAt(lowSpeed(Vessel) = true, T ) ← happensAt(slow_motion_start(Vessel), T ). (13) terminatedAt(lowSpeed(Vessel) = true, T ) ← happensAt(slow_motion_end(Vessel), T ). lowSpeed is a simple fluent, slow_motion_start and slow_motion_end are critical events. A slow_motion_start is emitted when a vessel starts moving with a speed lower than the vθ threshold of the trajectory synopses module and a slow_motion_end when it exceeds

M. Pitsikalis 33 Composite event patterns for maritime monitoring that threshold. Of course, the chosen threshold is relative to the general speed ranges of moving vessels and not their type.

4.1.4 Vessel changing Speed

Ships, in general sail with a constant speed, however there are periods where this is not the case. For example, until normal travelling speed is reached, speed constantly increases, similarly when entering a slow speed zone, such as a port or when it is turning, there is slowing down phase.

initiatedAt(changingSpeed(Vessel) = true, T ) ← happensAt(speed_change_start(Vessel), T ). (14) terminatedAt(changingSpeed(Vessel) = true, T ) ← happensAt(speed_change_end(Vessel), T ). changingSpeed is a simple fluent, and speed_change_start, speed_change_end are critical events produced during the synopsis generation step. Maximal intervals recognised with the changingSpeed fluent correspond to the time periods a vessel changes speed continuously.

4.1.5 Vessel in area of interest

Knowing the time periods during which a vessel is in an area of a specific type is particularly useful in location related patterns (e.g, fishing patterns). Consider the formalisation below:

initiatedAt(withinArea(Vessel, AreaType) = true, T ) ← happensAt(entersArea(Vessel, AreaName), T ). areaType(AreaName, AreaType). (15) terminatedAt(withinArea(Vessel, AreaType) = true, T ) ← happensAt(leavesArea(Vessel, AreaName), T ), areaType(AreaName, AreaType). withinArea(Vessel, AreaType) = true is a Boolean simple fluent indicating that Vessel is in an area of some type, such as Natura 2000, fishing, anchorage or shallow area. In case the area is of shallow type, AreaType is a touple containing additionally the depth of the area. entersArea and leavesArea are events computed at the spatial preprocessing step (see the top part of Table 4). areaType is an atemporal predicate storing the areas of a given dataset. With the use of rule-set (15), RTEC can compute the maximal intervals for which a vessel is in an area of some type.

M. Pitsikalis 34 Composite event patterns for maritime monitoring

4.2 Maritime Situational Indicators

Maritime Situational Indicators (MSIs) as defined in [21], formalise maritime activities of interest, that can be used for detecting certain scenarios. The MSIs list is the outcome of different workshops held in Sweden [5], Canada [34] and a NATO standard [1].

4.2.1 Vessel under way A vessel is considered ‘under way’ when it is not anchored, aground, or, in general, has not been made fast to a dock, the shore, or some other stationary object. Figure 5 illustrates this activity. Note that a vessel under way is not necessarily propelled or pushed—a vessel may be under way because of the wind or the sea current. So in case of an engine failure, a vessel may be under way due to drifting. A formalisation of this activity is given below:

initiatedAt(underWay(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, CoG, TrHeading), T ),

thresholds(vuwMin , VuwMin ),

thresholds(vuwMax , VuwMax ),

inRange(Speed, VuwMin , VuwMax ). (16) terminatedAt(underWay(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, CoG, TrHeading), T ),

thresholds(vuwMin , VuwMin ),

thresholds(vuwMax , VuwMax ),

not inRange(Speed, VuwMin , VuwMax ). underWay is a Boolean simple fluent, while velocity is contextual information expressing the speed, course over ground (CoG) and true heading (TrHeading) of a vessel. thresholds is an atemporal predicate storing the values of all the variables describing thresholds in patterns. inRange is an auxiliary predicate checking whether the given value is within some lower and upper limit. According to rule-set (16), therefore, underWay(Vessel) is initiated when the Vessel has speed within VuwMin and VuwMax , and terminated otherwise. Note that the speed reported in an AIS message is the speed calculated by the GPS system. For the experiments presented in Chapter 6, we set VuwMin to 2.7 knots, and VuwMax to 48.6 knots to avoid outliers such as data received by search and rescue aircrafts. All numerical thresholds in the patterns, however, may be tuned (by machine learning algorithms and/or human experts) to meet the requirements of the application under consideration.

M. Pitsikalis 35 Composite event patterns for maritime monitoring

500 0 500 1000 1500 2000 m

Critical points AIS messages Trajectory

Figure 5: Vessel under way leaving the port of Brest, France. Yellow circles denote AIS messages, marked circles indicate critical points and the line represents the vessel trajectory reconstructed by the transmitted AIS signals. The absence of position signals is not always classified as a communication gap, since it may not be long enough according to the threshold of the synopsis generator.

4.2.2 Vessel with high speed near coast

In general there aren’t speed limits in the open sea, however depending on the country there are several regulated zones. In French territorial waters there is a 5 knots speed limit for vessels or watercrafts within 300 m from coast. One of the causes of marine accidents near coast is vessels sailing with high speed, thus the early detection of violators ensures safety by improving the efficiency of law enforcement. Figure 6 illustrates a vessel not conforming to the above regulations. Consider the following formalisation for this scenario:

initiatedAt(highSpeedNC (Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, CoG, TrHeading), T ), holdsAt(withinArea(Vessel, nearCoast) = true, T ),

thresholds(vhcMax , VhcMax ),

not inRange(Speed, 0 , VhcMax ). terminatedAt(highSpeedNC (Vessel) = true, T ) ← (17) happensAt(velocity(Vessel, Speed, CoG, TrHeading), T ), holdsAt(withinArea(Vessel, nearCoast) = true, T ),

thresholds(vhcMax , VhcMax ),

inRange(Speed, 0 , VhcMax ). terminatedAt(highSpeedNC (Vessel) = true, T ) ← happensAt(end(withinArea(Vessel, nearCoast) = true)), T ). highSpeedNC is a Boolean simple fluent, and holdsAt(F = V , T ) is a built-in predicate of RTEC that is true if the value of a fluent F equals V at time T . Recall that withinArea is

M. Pitsikalis 36 Composite event patterns for maritime monitoring

100 0 100 200 300 400 m

Critical points AIS messages Trajectory Area within 300m from coast

Figure 6: Vessel, near the port of Brest, France, with speed above the 5 knots limit near coast. a simple fluent that expresses the time periods a vessel is in an area of some type (see rule-set (15)), which in this case is the area within 300 meters from coast constructed, by creating a “buffer” polygon around the polygon of France coastline. Intervals recognised by the highSpeedNC fluent correspond to the time periods a vessel sails with speed greater than VhcMax within 300 meters from coast. In the experiments presented in Chapter 6 we have set VhcMax=5 knots.

4.2.3 Vessel aground Ship grounding is the impact of a ship on seabed or waterway side. This may be uninten- tional as in a marine accident (due to an engine failure or a heavy storm), or intentional as in beaching to land crew or cargo, and careening for maintenance or repair. Consider the specification below:

holdsFor(aground(Vessel) = Status, I ) ← holdsFor(stopped(Vessel) = Status, Is), holdsFor(withinArea(Vessel, (shallow, Depth)) = true, Id), draught(Vessel, Draught), thresholds(depthDiffThr, DepthDiffThr), (18) Diff = |Draught − Depth|, inRange(Diff , 0 , DepthDiffThr), intersect_all([Is, Id], Ii), thresholds(agroundDuration, AgroundDuration), intDurGreater(Ii, AgroundDuration, I ) aground is a multi-valued statically determined fluent, that is, it is specified by means of

M. Pitsikalis 37 Composite event patterns for maritime monitoring a domain-dependent holdsFor predicate and interval manipulation construct—intersect_all in this case, that computes the intersection of a set of lists of maximal intervals (see Ta- ble 1), draught is an atemporal predicate returning the draught of the vessel based on static AIS information and intDurGreater is an auxiliary predicate filtering a list of intervals by removing the short ones according to some threshold (half hour, in the experiments presented here). Similar to communication gaps, we chose to distinguish between ‘aground’ incidents close to ports and those in the open sea (see Status in the above formalisation). According to rule 18, a vessel is considered aground when it is stopped in a shallow area and the absolute difference between the draught of the vessel and the current depth is lower than DepthDiffT hr, in our experiments we have set DepthDiffT hr to 1 meter.

4.2.4 Vessel at anchor or moored A vessel lowers its anchor in specific locations, for waiting to enter into a port, or for taking on cargo or passengers where insufficient port facilities exist (see Figure 7 for an example of an anchored vessel in an anchorage area). In addition to anchoring, vessels can also be moored. A vessel is considered moored when it is secured with ropes in any kind of permanent fixture. A permanent fixture for example may be a quay or a dock. Since anchoring and mooring are similar but not the same activities, we formalise them differently:

holdsFor(atAnchor(Vessel) = Status, I ) ←

holdsFor(stopped(Vessel) = Status, Ist ),

holdsFor(withinArea(Vessel, anchorage) = true, Ia ), (19) intersect_all([Ist , Ia ], Ii ), thresholds(anchorTime, AnchorTime),

intDurGreater(Ii , AnchorTime, I ). atAnchor = Status is a multi-valued statically determined fluent that holds when a vessel is stopped in an anchorage area for a time period greater than AnchorTime (for the experiments presented in Chapter 6 we have used AnchorTime = 3600 sec). Since an anchorage area can be either close to a port or far from ports, the definition of atAnchor

M. Pitsikalis 38 Composite event patterns for maritime monitoring

10 0 10 20 30 40 m

Critical points AIS messages Trajectory Anchorage area

Figure 7: Anchored vessel. Yellow circles denote AIS messages, marked circles indicate critical points, the line represents the vessel trajectory and the green dotted area is an anchorage area. uses both values of the stopped fluent.

holdsFor(moored(Vessel) = true, I ) ← holdsFor(stopped(Vessel) = nearPort, Isn),

holdsFor(atAnchor(Vessel) = nearPort, Ian ), (20) relative_complement_all(Isn , [Ian ], Ii ), thresholds(mooredDuration, MooredDuration),

intDurGreater(Ii , MooredDuration, I ). moored is a Boolean statically determined fluent. Note that we search for only for stopped vessels near ports, as stopped vessels far from ports are not required for the recognition of this CE. Intervals recognised by the moored fluent denote that a vessel is stopped near ports for a period larger than MooredDuration, in our experiments we used a value of 1800 sec. In order to simplify definitions of other patterns that involve anchoring and mooring we have additionally created the following rule:

holdsFor(atAnchorOrMoored(Vessel) = true, I ) ←

holdsFor(atAnchor(Vessel) = nearPort, Inp),

holdsFor(atAnchor(Vessel) = farFromPorts, Iﬀp), (21)

holdsFor(moored(Vessel), Im ),

union_all([Inp, Iﬀp, Im ], I ). atAnchorOrMoored is a Boolean statically determined fluent uniting the intervals of the anchored and the moored fluents.

M. Pitsikalis 39 Composite event patterns for maritime monitoring

1 0 1 2 3 4 km

Critical points AIS messages Trajectory

Figure 8: General cargo vessel travelling with speed ≈ 10 knots. As illustrated on this Figure, it is common for vessels to travel long distances with travelling speed without changing direction.

4.2.5 Vessel with travelling speed When ships travel in the open sea (see Figure 8), they have higher speed relatively to the speed they have near coast or ports. It is evident, that normal service speed varies between vessels. To tackle this issue, we formalised this activity as follows:

initiatedAt(travelSpeed(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, CoG, TrHead), T ), vesselType(Vessel, Type), typeSpeed(Type, Min, Max, Avg), inRange(Speed, Min, Max). (22) terminatedAt(travelSpeed(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, Heading), T ), vesselType(Vessel, Type), typeSpeed(Type, Min, Max, Avg), not inRange(Speed, Min, Max). travelSpeed is a Boolean simple fluent, while the vesselType atemporal predicate returns the type of the ship using the MMSI of the Vessel from the ship type field of AIS static messages. The typeSpeed predicate returns the average, the minimum and the maximum recorded service speed for the given vessel type. Intervals recognised by the travelSpeed fluent express the maximal intervals of vessels moving with travelling speed.

4.2.6 Vessel’s movement ability is affected

Another scenario, that in some cases requires urgent actions, is when a vessel has its movement ability affected. This situation is most commonly caused by an engine problem

M. Pitsikalis 40 Composite event patterns for maritime monitoring

250 0 250 500 750 1000 m

Critical points AIS messages Trajectory

Figure 9: Container ship moving with speed 1 to 4 knots in the open sea, while its service speed is 19 knots. or due to heavy weather. In both cases the vessel moves with lower speed than the travelling speed (see an example of this behavior in Figure 9). Taking the above into account, consider the following specification:

initiatedAt(speedLtTr(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, _CoG, _TrHeading), T ), vesselType(Vessel, Type), typeSpeed(Type, Min, Max, Avg), inRange(Speed, 0 , Min). (23) terminatedAt(speedLtTr(Vessel) = true, T ) ← happensAt(velocity(Vessel, Speed, _CoG, _TrHeading), T ), vesselType(Vessel, Type), typeSpeed(Type, Min, Max, Avg), not inRange(Speed, 0 , Min). speedLtTr is a Boolean simple fluent, that holds for the time periods a vessel has speed lower than its type minimum travelling speed. However, this alone is not enough for the recognition of the time periods a vessel has its movement ability affected, since vessels move with decreased speed in regulated areas, or don’t have speed at all when they are anchored or moored. Consequently, to refine the definition we additionally use the

M. Pitsikalis 41 Composite event patterns for maritime monitoring following rule:

holdsFor(maa(Vessel) = true, I ) ←

holdsFor(speedLtTr(Vessel) = true, Isl ),

holdsFor(atAnchorOrMoored(Vessel) = true, Iam ),

holdsFor(withinArea(Vessel, nearCoast) = true, Inc), (24)

relative_complement_all(Isl , [Iam , Inc], Ii ), thresholds(maaDuration, MaaDuration),

intDurGreater(Ii , MaaDuration, I ). maa is a statically determined fluent and relative_complement_all computes the relative complement of a maximal interval list with the union of a set of lists of maximal intervals (see Table 1). Therefore, maa filters out the time periods a vessel has decreased speed due to irrelevant factors and keep only those of interest. Again, we keep only intervals greater than a time threshold MaaDuration, for the experimental analysis we used MaaDuration = 600 sec.

4.2.7 Vessel has speed incompatible with its type

As already mentioned, vessels also transmit identity related information such as the name, the type, the draught etc. However, this information is not always correct, vessels engaged in illegal activities usually have fraudulent values, for instance most vessels involved in illegal fishing report a false type by spoofing identity related information [32]. Another cause of incorrect information can be the neglection of updating the transmitted static information to match the vessel’s current state. In such cases, it is often observed that the vessel sails with a speed not matching its reported type. For this reason, we created the following rule:

holdsFor(vesselIST (Vessel) = true, I ) ←

holdsFor(underWay(Vessel) = true, Iuw ),

holdsFor(travelSpeed(Vessel) = true, Itr ), (25) holdsFor(withinArea(Vessel, nearCoast) = true, Inc),

relative_complement_all(Iuw , [Itr , Inc], Ii ),

intDurGreater(Ii , ISTDuration, I ). vesselIST is a statically determined fluent, travelSpeed as described in rule-set (22) expresses the intervals in which a vessel has travelling speed, underWay (rule-set (16)) expresses the intervals in which a vessel is under way and withinArea (rule-set (15)) is true when a vessel sails in an area of some type. The intervals recognised with the vesselIST

M. Pitsikalis 42 Composite event patterns for maritime monitoring fluent denote that a vessel is under way, far from coast, and it does not sail with travelling speed for a period greater than ISTDuration, for the experiments presented in Chapter 4, we used a threshold of 1800 sec. Note that for speed lower than the minimum travelling speed of the vessel type, a more refined pattern would include weather information, since a lower speed could be the effect of heavy weather. In future work, additional data sources such as the weather information will be taken into account with our current patterns.

4.2.8 Vessel drifting A vessel is drifting when its course over ground i.e. the direction calculated by the GPS signal is heavily affected by the stream of the sea or harsh weather conditions. Typically, as illustrated in Figure 10 (you can find a video of the presented vessel drifting here5), if course over ground deviates from the true heading of a vessel, that is the direction of the ship’s bow, then the vessel is considered adrift.

25 0 25 50 75 100 m

Critical Points

True Heading

Course over ground / AIS messages Adrift trajectory

Figure 10: A vessel drifting. Yellow arrows denote AIS messages with their direction matching the course over ground value, similarly the direction of green arrows matches the true heading value, marked arrows are critical points.

5https://owncloud.skel.iit.demokritos.gr/index.php/s/cKdlbeoXO0ee36a

M. Pitsikalis 43 Composite event patterns for maritime monitoring

initiatedAt(adrift(Vessel) = true, T ) ← happensAt(velocity(Vessel, _Speed, CoG, TrHeading), T ), holdsAt(underWay(Vessel) = true, T ),

not angleDiﬀLess(CoG, TrueHeading, Diﬀad ). terminatedAt(adrift(Vessel) = true, T ) ←

happensAt(velocity(Vessel, _Speed, CoG, TrHeading), T ), (26) holdsAt(underWay(Vessel) = true, T ),

thresholds(diﬀad , Diﬀad ),

angleDiﬀLess(CoG, TrHeading, Diﬀad ).

terminatedAt(adrift(Vessel) = true, T ) ← happensAt(end(underWay(Vessel) = true), T ). adrift is a Boolean simple fluent, angleDiﬀLess is an auxilliary predicate that is true if the absolute minimum difference between two angles is less than a given threshold and end is a built-in predicate of RTEC denoting the end of a period a fluent holds, for instance the end time-point of the interval [a, b) is b−1. Note that only moving vessels can be considered adrift, to express this constraint rule-set 26 additionally requires that the vessel is under way.

4.2.9 Fishing Fishing is an activity that exploits common natural resources, and thus needs to be regulated to safeguard fair access and sustainability [21]. Maritime monitoring enables better regulation and monitoring of fishing activities. A common fishing method is trawling, that involves a boat (trawler) pulling a fishing net through the water behind it. The trawler has steady—‘trawling’—speed and a wide heading angle distribution. See Figure 11 for an illustration.

A formalisation expressing the trawling movement may be found below:

initiatedAt(trawlingMovement(Vessel) = true, T ) ← vesselType(Vessel, ﬁshing), happensAt(change_in_heading(Vessel), T ), (27) holdsAt(withinArea(Vessel, ﬁshing) = true), T ). maxDurationUE(trawlingMovement(Vessel) = true, TrawlingMDTime).

M. Pitsikalis 44 Composite event patterns for maritime monitoring

1 0 1 2 3 4 km

Critical points AIS messages Trajectory Fishing area

Figure 11: Illustration of a detected fishing vessels engaged in trawling. Dotted area expressed a fishing area. trawlingMovement is a simple fluent and maxDurationUE is a mechanism of RTEC handling deadlines (see Chapter 3.2.4). According to rule-set (27) trawlingMovment holds for the time periods that there are consequent changes in heading with the time distance between two consecutive change_in_heading events not exceeding TrawlingMDTime, for the experiments presented in Chapter 6 we used TrawlingMDTime = 600 sec.

holdsFor(trawling(Vessel) = true, I ) ←

holdsFor(trawlSpeed(Vessel) = true, It ),

holdsFor(trawlingMovement(Vessel) = true, Itc),

holdsFor(withinArea(Vessel, ﬁshing) = true, Iw ), (28)

intersect_all([It , Iw , Itc], Ii ), thresholds(trawlingTime, TrawlingTime),

intDurGreater(Ii , TrawlingTime, I ). trawling is expressed as a statically determined fluent, trawlSpeed is a simple fluent record- ing the intervals in which a vessel has trawling speed. According to rule (28), therefore, a vessel is said to be trawling if it is a fishing vessel, has trawling movement and it sails in trawling speed within a fishing area for a period of time greater than TrawlingTime = 1 hour.

4.2.10 Vessel engaged in search and rescue operation Search and rescue (SAR) operations aim to provide aid to people who are in distress or imminent danger (see figure 12). Research has indicated that vessels engaged in SAR operations, change speed and heading more often compared to other voyages [10]. Taking this into consideration we formalised this event as follows:

M. Pitsikalis 45 Composite event patterns for maritime monitoring

250 0 250 500 750 1000 m

Vessel in distress critical points SAR vessel critical points Vessel in distress AIS messages SAR vessel AIS messages Vessel in distress trajectory SAR vessel trajectory

Figure 12: A confirmed case of a vessel in distress and a SAR vessel. Orange and blue circles denote the position signals of the two vessels and marked circles denote critical points.

initiatedAt(sarCourse(Vessel) = true, T ) ← happensAt(change_in_heading(Vessel), T ), vesselType(Vessel, sar). initiatedAt(sarCourse(Vessel) = true, T ) ← (29) happensAt(start(changingSpeed(Vessel) = true), T ), vesselType(Vessel, sar). maxDurationUE(sarCourse(Vessel) = true, sarMDTime).

Similar to trawlingMovement (see rule (27)) sarCourse is a Boolean simple fluent that is initiated when there is a change in heading or a change in speed phase starts and terminated by the deadline mechanism of RTEC. This rule, doesn’t take into consideration the value of speed. To this end, we additionally created the following rule:

holdsFor(inSARoperation(Vessel) = true, I ) ←

holdsFor(sarSpeed(Vessel) = true, Iu ),

holdsFor(sarCourse(Vessel) = true, Is ), (30)

intersect_all([Iu , Is ], Ii ),

intDurGreater(Ii , SarTime, I ). inSARoperation is a statically determined fluent and sarSpeed is a simple fluent expressing the time periods a SAR vessel has speed above a threshold Vsar. sarSpeed is defined similar to the underWay (see rule-set (16)), however an upper speed limit is not set, since SAR operations can be performed by aircrafts. Therefore, rule (30) describes that the inSARoperation holds when a SAR vessel has SAR speed and its course has SAR characteristics.

M. Pitsikalis 46 Composite event patterns for maritime monitoring

5 0 5 10 15 20 m

Critical points Vessel 1 AIS messages Vessel 2 AIS messages Vessel 1 trajectory Vessel 2 trajectory

Figure 13: Rendez-vous between two fishing vessels.

4.2.11 Vessels in rendezvous

A maritime scenario that may indicate illegal activities, such as illegal cargo transfer, is when two vessels are nearby in the open sea, stopped or sailing at a very low speed (see Figure 13). A specification of rendezvous may be found below:

holdsFor(rendezVous(Vessel1 , Vessel2 ) = true, I ) ←

holdsFor(proximity(Vessel1 , Vessel2 ) = true, Ip),

not vesselType(Vessel1 , tug),

not vesselType(Vessel2 , tug),

holdsFor(lowSpeed(Vessel1 ) = true, Il1 ),

holdsFor(stopped(Vessel1 ) = farFromPorts, Is1 ),

union_all([Il1 , Is1 ], I1 ), (31)

holdsFor(lowSpeed(Vessel2 ) = true, Il2 ),

holdsFor(stopped(Vessel2 ) = farFromPorts, Is2 ),

union_all([Il2 , Is2 ], I2 ),

intersect_all([I1 , I2 , Ip], If ),

thresholds(rVtime , RVtime ),

intDurGreater(If , RVtime , I ). rendezVous is a relational statically determined fluent, as opposed, to the fluents presented so far that concern a single vessel, referring to a pair of vessels. Note that we only check for rendez-vous only if both vessels are not tug boats and they are far from ports, as two slow moving or stopped vessels near ports would probably be marooned or leave the port, thus not indicating a rendez-vous.

M. Pitsikalis 47 Composite event patterns for maritime monitoring

7.5 0 7.5 15 22.5 30 m

Critical points AIS messages Trajectory

Figure 14: A vessel loitering.

4.2.12 Vessel loitering

In general, loitering is the act of remaining in a particular area for a long period without purpose (see Figure 14 for an illustration). In sea, this behaviour could be an indicator of an unlawful activity. Below we formalise this activity:

holdsFor(loitering(Vessel) = true, I ) ←

holdsFor(lowSpeed(Vessel) = true, Il ),

holdsFor(stopped(Vessel) = farFromPorts, Is ),

holdsFor(withinArea(Vessel, Area) = true, Iwa ), Area ≠ nearCoast, (32) holdsFor(atAnchorOrMoored(Vessel) = true, Iam ),

relative_complement_all(Ilsaa , [Iam ], Ils ),

intersect_all([Ils , Iwa ], Ii ), thresholds(ltrTime, LtrTime),

intDurGreater(Ii , LtrTime, I ).

Rule (32) defines loitering as a statically determined fluent. Since we don’t want to take into consideration vessels sailing with low speed in regulated areas, and anchored or moored vessels we filter those intervals out. Therefore, intervals recognised by the loitering fluent correspond to the time periods a vessel is stopped or moves with low speed in the same not speed regulated area for a period greater than LtrT ime and the vessel is not anchored or moored. For the experiments presented in Chapter 6 we have set LtrT ime to 1800 sec.

4.2.13 Tugging

A ship in a crowded harbor or a narrow canal—in general, a vessel that should not move by itself—is typically pulled or towed by a tugboat. Figure 15 shows an example. It is expected

M. Pitsikalis 48 Composite event patterns for maritime monitoring that during tugging, the speed and heading difference between the tugged vessel and the tugboat to be relatively small. Below we present a formalisation of tugging:

initiatedAt(tugging(Vessel1 , Vessel2 ) = true, T ) ←

(vesselType(Vessel1 , tug); VesselType(Vessel2 , tug)),

happensAt(velocity(Vessel1 , Speed1 , CoG1 , _TrHeading1 ), T ),

holdsAt(proximity(Vessel1 , Vessel2 ) = true, T ),

thresholds(w, W ), thresholds(dυmax , ∆υmax ), thresholds(dθmax , ∆θmax ),

lastTimepointOf (Vessel2 , T2 , T , W ),

happensAt(velocity(Vessel2 , Speed2 , CoG2 , _TrHeading2 ), T2 ),

not inRange(Speed1 , 0 , VtugMin ),

not inRange(Speed2 , 0 , VtugMin ),

Diﬀ = |Speed1 − Speed2 |,

inRange(Diﬀ , 0 , ∆υmax ),

angleDiﬀLess(CoG1 , CoG2 , ∆θmax ).

terminatedAt(tugging(Vessel1 , Vessel2 ) = true, T ) ← (33) (vesselType(Vessel1 , tug); VesselType(Vessel2 , tug))

happensAt(velocity(Vessel1 , Speed1 , CoG1 , _TrHeading1 ), T ),

holdsAt(proximity(Vessel1 , Vessel2 ) = true, T ),

thresholds(w, W ), thresholds(dυmax , ∆υmax ), thresholds(dθmax , ∆θmax ),

lastTimepointOf (Vessel2 , T2 , T , W ),

happensAt(velocity(Vessel2 , Speed2 , CoG2 , _TrHeading2 ), T2 ),

Diﬀ = |Speed1 − Speed2 |,

( not inRange(Diﬀ , 0 , ∆υmax );

not angleDiﬀLess(CoG1 , CoG2 , AngleDiﬀ )).

terminatedAt(tugging(Vessel1 , Vessel2 ) = true, T ) ←

( happensAt(end(proximity(Vessel1 , Vessel2 ) = true), T );

happensAt(start(stopped(Vessel1 ) = _StopStatus), T );

happensAt(start(stopped(Vessel2 ) = _StopStatus), T )). tugging is a Boolean relational simple fluent, proximity is a durative input event computed at the spatial preprocessing step, expressing the time periods during which two vessels are ‘close’ (in the presented experiments, their distance is less than 100 m). The

M. Pitsikalis 49 Composite event patterns for maritime monitoring

100 0 100 200 300 400 m

Critical points Bulk Carrier AIS positions Tug AIS positions Bulk Carrier trajectory Tug trajectory

Figure 15: Illustration of a detected tugging activity. lastTimepointOf (Vessel, T , T ′, W ) predicate returns the latest message broadcasted (or the last critical event in the case of synopses) by Vessel at time T ′ in a W time period before

T . We search for the latest message transmitted by Vessel2 in order to have an accurate comparison between the two vessels at specific timepoint T , however the most accurate approach would also consider a time period after T , this approach though proved to be computationally expensive. According to rule-set (33) the tugging fluent is initiated when there is proximity between two vessels, they have speed greater than VtugMin , whereas the speed and angle difference between them must be less than ∆υmax and ∆θmax respectively, for our experiments we used W = 600 sec,VtugMin = 1 .8 knots, ∆υmax = 3 knots and ∆θ = 60 ◦. The tugging fluent terminates when:

• two vessels are no longer close. • the speed/heading difference exceeds the chosen thresholds.

• one of the two vessels stops.

M. Pitsikalis 50 Composite event patterns for maritime monitoring

5. DYNAMIC GROUNDING

To perform query computation, RTEC grounds every output entity. An output entity can be either an event, a simple fluent and/or a statically determined fluent. At each query-time RTEC attempts finding intervals for all possible instances of an output entity created using the variable domains. This operation can be expensive. For example, in maritime monitoring, the domain of vessels i.e., all different MMSI values, is not always the same at each query time. This is because, there are new vessels arriving at some point in the monitored area and respectively there are vessels leaving the monitored area. Consequently, only a subset of MMSI values is required for the recognition process at each query-time. Taking this into consideration, to perform event recognition efficiently we have extended RTEC to support dynamic grounding. Dynamic grounding relies on two operations, building the domains and updating the domains. The first one is used for collecting the domain elements from input events, and the second one is used for discarding or keeping elements of the domains based on the needs of the current recognition period. Below we describe the algorithm of dynamic grounding.

5.1 Building Domains

Each variable of an output entity F = V that is grounded takes values from some domain { i i } i Dn = d1, . . . , dz , where dx is an element of domain Dn, x ranges in [1, z] and z is the number of elements in Dn. An element of a domain Dn contains a value expressed as a Prolog term, that can be used later as a source for an output entity’s variables. In our approach, i the domain elements dx of a domain Di are collected from input events (SDEs). For each i domain Di the set of input entities from which the domain elements dx are collected, must be defined by the user. For instance, in our system, the elements of the domain containing all the vessels in the dataset are collected from the coord input entity. Figure 16a illustrates an example.

5.2 Updating domains

Applying dynamic grounding, requires the execution of several operations in each recognition query time Qi. First, we must collect for all domains their elements from SDEs that fall within (Qi − WM,Qi] then we must discard all redundant elements. An element is redundant if it does not exist in SDEs in current window and there isn’t an output entity bound by that element with an open interval. Checking whether an element is redundant, and removing the unnecessary ones is performed during the step of updating domains.

The ‘updating’ step is illustrated in Figure 16b. At t = Q1, there are 4 intervals recognised, where from two are open intervals. Consequently we have to keep the MMSIs of 227705102 and 228051000 since those have open intervals and possibly remove 245257000

M. Pitsikalis 51 Composite event patterns for maritime monitoring

vessel(245257000) vessel(245257000) vessel(227705102) vessel(227705102) vessel(227705102) vessel(228131600) vessel(228131600) vessel(228131600) vessel(228051000) vessel(228051000) vessel(228051000) vessel(227574020)

245257000 245257000 227705102 227705102 228131600 228131600 228131600 228051000 228051000 227574020 Q Q Q Q Q o 1 o 1 2 WM WM

coord(245257000,...) coord(245257000,...) coord(228131600,...) coord(227705102,...) coord(227705102,...) coord(228051000,...) coord(227574020,...) coord(228131600,...) coord(228131600,...) coord(228051000,...) coord(228051000,...)

(a) Recognition query at t = Q1. (b) Recognition query at t = Q2.

Figure 16: Illustration of input events (lower part), recognised intervals (middle part) and the vessel domain (upper part) for two recognition queries. Squares denote initiation/termination points, while lines not ending in a black square denote open intervals. In the left part, it is illustrated how the values of the coord input event are collected in order to build the domains, while in the left part, is additionally illustrated how intervals computed at Q1 affect the updating process of the domains.

and 228131600. Before performing the recognition query at Q2, we collect vessel IDs from input events that fall within Q2 −WM, as you can see 245257000 does not appear in a coord event therefore it is redundant and must be retracted from the vessel domain, 227705102 although not appearing in a coord event within (Q2 − WM,Q2] is retained since it is involved in an open interval, 228131600, 228051000 are also retained since they both appear in input events, and 227574020 is asserted in the vessel domain since it appears in a coord event.

5.3 Algorithm

Applying dynamic grounding in RTEC requires certain operations at each query time Qi. Consequently, the dynamicGrounding Algorithm (1) must be executed at each query time before recognition starts. Information about the variable domains is acquired using the two types of declarations facts collectElements, dGrounded. Each collectElements fact, has the list of input entities from which each domain is built and the dGrounded facts are used for declaring that an output entity draws values from a specific domain. First, we iterate all input entities in order to collect the elements of the domains. Checking whether an input entity is a source for a domain is performed using the declared collectElements facts (line 2). If an input entity is indeed a source for a domain, we iterate all the instances of that input entity that fall within (Qi−WM ,Qi], create the appropriate terms and store them in a set data structure called inputTerms (lines 3-6).

M. Pitsikalis 52 Composite event patterns for maritime monitoring

Algorithm 1 dynamicGrounding Input: collectElements(L, Domain): set of all declared collectElements facts; dGrounded(OutputEntity, Domain): set of all declared dGrounded facts; Domains: The variable domains; SDEs ∈ (Qi − WM,Qi]: Input events that fall within (Qi − WM,Qi]; CEs computed at Qi−1: CEs computed at previous query time; Output: Domains: Updated domains for the current recognition query; Requires: terms(Domainx ): set of asserted elements of Domainx; setName(Domainx ): a structure capable of holding a set of elements of Domainx in a term form (setName can be for example oldTerms)

1: for all entityi ∈ inputEntities do 2: for all (Ln , Domainn ) ∈ collectElements(L, Domain) do 3: if entityi ∈ Ln then 4: for all instancem ∈ instances(entityi ) do 5: createGroundingTerm(instancem , Domainn , GroundingTerm) 6: inputTerms(Domainn ).add(GroundingTerm) 7: for all entityi ∈ outputEntities do 8: for all Domainn ∈ Domains do 9: if dGrounded(entityi , Domainn ) then 10: for all instancek ∈ instances(entityj ) do 11: if hasOpenIntervals(instancek ) then 12: createGroundingTerm(instancek , Domainn , term) 13: remainingTerms(Domainn ).add(term) 14: for all Domainn ∈ Domains do 15: oldTerms(Domainn ) = terms(Domainn ) 16: allNewTerms(Domainn ) = inputTerms(Domainn ) ∪ remainingTerms(Domainn ) 17: terms2retract(Domainn ) = oldTerms(Domainn ) \ allNewTerms(Domainn ) 18: terms2add(Domainn ) = allNewTerms(Domainn ) \ oldTerms(Domainn ) 19: assertAll(terms2add(Domainn )) 20: retractAll(terms2retract(Domainn ))

Next, for each output entity we check if it draws values from a specific domain using the dGrounded facts. If an output entity draws values from a domain, we iterate all the instances of that entity and check if there are open intervals (lines 9-11). If there are, this means that the values of that entity’s instance, are needed for the next recognition query, and therefore are stored in the set data structure remainingTerms in the appropriate term form (lines 12,13).

In order to prepare the domains for the recognition query at Qi, for each domain we perform the following operations. First, we store all grounding terms asserted at Qi−1 in a set data-structure called oldTerms (line 15). Then, we store in a set data structure called allNewTerms the grounding terms needed at Qi by computing the union of inputTerms and remainingTerms (line 16). Since a number of grounding terms asserted at Qi−1 will be also required at Qi we only need to assert the ones not existing at Qi−1, those terms are cal-

M. Pitsikalis 53 Composite event patterns for maritime monitoring culated by computing the set difference between oldTerms and allNewTerms, in a similar manner we calculate the redundant grounding terms by computing the reversed difference between the afforementioned sets (lines 17,18). Finally we retract the unnecessary grounding terms and assert the required new ones (lines 19,20).

M. Pitsikalis 54 Composite event patterns for maritime monitoring

6. EMPIRICAL ANALYSIS

We evaluate our engine in terms of efficiency—e.g., average recognition time. However, we cannot evaluate the patterns in terms of predictive accuracy, since the dataset does not include the ground truth—e.g., annotation of trawling, rendez-vous, etc, by experts. Therefore, we can only evaluate the accuracy achieved when consuming the compressed trajectories: we consider as reference the CEs detected when consuming the complete AIS stream with the critical point labels—see the ‘enriched AIS stream’ in Figure 3—and then compare those CEs against the ones detected when consuming the compressed AIS stream—see the ‘critical point stream’ in Figure 3. This way, we assess the impact of trajectory compression on CER.

6.1 Experimental setup

In order to evaluate our system we used real world data from two monitoring areas the first one is around the port of Brest in france while the other one covers Europe, Figures 17, 18 illustrate the coverage of the two datasets [3]. Brest: We used 18M position signals from 5K vessels sailing in the Atlantic Ocean around the port of Brest, France, between October 1st 2015 and 31st March 2016 [31]6. The spatial preprocessing module produced 811K spatio-temporal events linking vessels with 0,5K fishing areas, 1K Natura 2000 areas, 10 anchorage areas, 132 near coast areas and 22k shallow areas extracted using the GMRT tool [15] and vessels with other vessels. The trajectory synopsis generator labelled 5M position signals as critical.

Figure 17: Coverage for the Brest dataset.

Europe: Regarding the area of Europe, the dataset consists of 55M position signals from 34K vessels in the European seas in January 2016. The spatial preprocessing module produced 7M spatial events linking vessels with other vessels, 3K fishing areas and 6K Natura 2000 areas.

6https://zenodo.org/record/1167595

M. Pitsikalis 55 Composite event patterns for maritime monitoring

Figure 18: Coverage for the European dataset.

The characteristics of the two datasets are summarised in Table 5. The CEs that were tested and recognised are shown in the lower part—below the double horizontal line—of Table 4. In the case of the Europe, since anchorage areas, speed regulated areas and draught information were not available, the affected CEs are not detected. The experiments were conducted using RTEC7 under YAP-6 Prolog, in a machine running Ubuntu 16.04.3 LTS, with an Intel Core i7-i7700 CPU @ 3.60GHz x 8 and 16 GB 2133 MHz RAM.

Table 5: Characteristics of the datasets used in our experiments.

Attribute Brest Europe

Period (months) 6 1 Vessels 5K 34K Position signals 18M 55M Critical position signals 4.6M 17M Fishing areas 0,5K 3K Natura 2000 areas 1K 6K Anchorage areas 10 - Near coast areas 132 - Shallow areas 22K - Spatio-temporal events 811K 7M

6.2 Experimental results

6.2.1 Efficiency RTEC performs CER over a sliding window [8]. Figure 19 presents the average CER times for windows ranging from 2 to 16 hours and step set to 2 hours, with and without the dynamic grounding mechanism when using the Enriched AIS stream and the critical point stream of the Brest dataset. Dynamic grounding proves to be an important extension of RTEC that further improves its efficiency. Therefore, all the following experiments were

7https://github.com/aartikis/RTEC

M. Pitsikalis 56 Composite event patterns for maritime monitoring

10 10 With DG With DG 8 Without DG 8 Without DG

6 6

4 4

2 2

Average Recognition Time (sec) 0 Average Recognition Time (sec) 0 2 4 8 16 2 4 8 16 Window size (hours) Window size (hours)

When used in the European dataset average recognition times are higher (see Figure 20), though they are still acceptable for real-time monitoring considering that the highest average recognition time is 15 minutes when using a window of 16 hours.

Similarly Figure 21 presents the average CER times per CE. Average recognition time for each CE, corresponds to the average recognition time when performing CER for the CE we want to study along with all the other CEs it depends. Since, there are CEs depending on the same lower in hierarchy CEs, average recognition time for all CEs is lower than the sum of the presented average recognition times per CE. Experimental results show that when windows size is 2 to 4 hours all CEs have similar average recognition times in both streams. However, with window size greater than 4 hours when using the enriched AIS stream the most expensive CEs are tugging, trawling and inSAR with tugging being the most expensive. On the other hand, when using the critical point stream tugging has smaller recognition times than the trawling and inSAR CEs. The cause of this behaviour, is that the average recognition time of the tugging CE depends on the density of the transmitted signals due to the use of the lastTimepointOf (Vessel, Tv , T , W ) predicate which searches for the last transmitted message of V essel inside the period (T − WM,T ]. Therefore, when using the critical point stream the cost of this operation is far less than when using the enriched AIS stream. Concerning the trawling and inSAR CEs, which have higher average recognition times comparing to the other CEs, both use the deadline mechanism of RTEC which adds computational cost to the recognition process. Com- paring average recognition times between the enriched AIS stream and the critical point stream, results point out that for all CEs average recognition times are smaller when using the critical point stream.

M. Pitsikalis 57 Composite event patterns for maritime monitoring

1,500 Enr. AIS Stream Cr. Point Stream

1,000

500

Average Recognition Time (sec) 0 2 4 8 16 Window size (hours)

Figure 20: Average recognition time when using the European dataset.

8 8 Enr. Crit. Enr. Crit. underW ay atAnchor 6 highSpeedNC 6 moored travelSpeed aground 4 4 adrift

2 2

Average Recognition Time (sec) 0 Average Recognition Time (sec) 0 0 2 4 8 16 18 0 2 4 8 16 18 Window size (hours) Window size (hours) (a) Speed related patterns. (b) Vessel status patterns.

8 8 Enr. Crit. Enr. Crit. trawling rendezV ous 6 tugging 6 loitering inSAR vesselIST 4 4

2 2 Average Recognition Time (sec) 0 Average Recognition Time (sec) 0 0 2 4 8 16 18 0 2 4 8 16 18 Window size (hours) Window size (hours) (c) Vessel(s) engaged in a specific activity. (d) Possible illegal activities.

Figure 21: Average recognition times for each CE, using the enriched AIS stream and the critical point stream of the Brest dataset.

M. Pitsikalis 58 Composite event patterns for maritime monitoring

Critical points AIS messages FN Under way using Enr. AIS stream (60,2.69) Under way using Cr. point stream

(50,2.73)

(39,2.72) FP

(29,2.65)

TP (19,2.68)

(8,2.72)

(0,2.73)

Figure 22: Illustration of False Positives (FP) and False Negatives (FN) using the ‘underWay’ CE. Orange circles indicate AIS messages while marked orange circles denote critical points. Red (yellow) lines denote the CE intervals computed when consuming the enhanced AIS (critical point) stream. The two numbers attached to each AIS message express the time-point of the message and the speed of the vessel.

6.2.2 Accuracy

As mentioned above, to perform accuracy experiments, we considered as reference the detected CEs detected when consuming the enriched AIS stream. Then, we compared those against the CEs recognised when consuming the (compressed) critical point stream.

Table 6 presents the results in terms of Precision, Recall and F1-score. The set of True Positives expresses the time-points (seconds) in which a CE is recognised when consuming the enriched AIS stream and the critical point stream. Similarly, the set of False Posi- tives (respectively, False Negatives) expresses the seconds in which a CE is recognised when consuming the critical point (respectively, enriched AIS) stream but not detected when consuming the enriched AIS (critical point) stream. Table 6 shows that there are perfect scores for some CEs. This is not surprising since their patterns are defined only in terms of critical points and spatial events, that are part of both data streams. Fot the remaining CEs, experiments show that the scores are close to perfect with the lowest being the one for the adrift fluent.

In both datasets adrift has the lowest F1 score. This is because it has various False Pos- itives and False Negatives, thus compromising Precision and Recall. False Positives are created when a CE is terminated later when consuming the critical point stream than when consuming the enriched AIS stream. Similarly, False Negatives are created when a CE is initiated later when consuming the critical point stream than when consuming the enriched AIS stream. The low score of the adrift fluent is attributable to the fact that the trajectory synopses generator when creating the synopses, regarding the direction of the vessel, takes into consideration only the course over ground and omits the true heading value, which is a value needed for the recognition of the adrift fluent. Figure 22 offers an illustration of False Positives and False Negatives using the underWay CE as an example. Recall that this CE is initiated when the speed is less higher than 2.7

M. Pitsikalis 59 Composite event patterns for maritime monitoring knots, and terminated otherwise (see rule-set (16)). According to this example, (using the enhanced AIS stream) ‘underWay’ is initiated at time-point t = 0 as speed is 2.73 knots, and terminated at t = 19 when speed becomes 2.68 knots. Similarly, it is initiated at t = 39 and terminated t = 60 . When consuming the critical point stream, however, different intervals are computed, since some of the AIS messages have been removed due to compression. For example, the AIS message at t = 19 has been removed, delaying the termination of ‘underWay’ and thus creating False Positives. Moreover, the AIS messages from t = 39 to t = 60 have also been removed, thus not triggering the initiation of ‘underWay’ and consequently creating False Negatives.

M. Pitsikalis 60 Composite event patterns for maritime monitoring

Table 6: Recognition Accuracy.

Brest Europe

Event Precision Recall F1-Score Precision Recall F1-Score

gap 1 1 1 1 1 1 stopped 1 1 1 1 1 1 lowSpeed 1 1 1 1 1 1 changingSpeed 1 1 1 1 1 1 withinArea 1 1 1 1 1 1 underWay 0.999 0.995 0.997 0.996 0.992 0.994 highSpeedNC 1 0.911 0.953 - - - aground 1 1 1 - - - atAnchor 1 1 1 - - - moored 1 1 1 1 1 1 atAnchorOrMoored 1 1 1 1 1 1 travelSpeed 0.988 0.981 0.985 0.928 0.922 0.925 speedLtTr 0.988 0.990 0.989 0.996 0.893 0.942 speedGtTr 0.894 0.858 0.876 0.907 0.929 0.918 maa 0.987 0.989 0.988 0.986 0.716 0.830 vesselIST 0.938 0.936 0.937 0.975 0.973 0.974 adrift 0.847 0.895 0.870 0.406 0.949 0.568 trawlSpeed 0.940 0.957 0.949 0.888 0.904 0.896 trawlingCourse 1 1 1 1 1 1 trawling 0.992 0.997 0.994 0.953 0.996 0.974 tugging 0.983 0.879 0.928 0.993 0.881 0.934 rendezVous 1 1 1 1 1 1 loitering 1 1 1 1 1 1 sarSpeed 0.998 0.994 0.996 0.995 0.988 0.991 sarCourse 1 1 1 1 1 1 inSAR 0.998 0.999 0.999 0.993 0.998 0.993

M. Pitsikalis 61 Composite event patterns for maritime monitoring

7. SUMMARY AND FURTHER WORK

We developed a CER system for maritime monitoring. For acceptable predictive accuracy, we collaborated with domain experts for pattern construction. For scalable recognition, we extended RTEC by supporting dynamic grounding. Our thorough evaluation on two real, large maritime datasets illustrated the benefits of our system. For future work we are extending RTEC, our CER engine, for handling cyclic events, thus permitting the creation of more general patterns capable of modelling voyage activity. Additionally, we aim to compare RTEC with other CER systems such as the FlinkCEP. Finally, we will investigate the effects of additional data sources, such as weather information on CER.

M. Pitsikalis 62 Composite event patterns for maritime monitoring

ACRONYMS AND ABBREVIATIONS

AI Artificial Intelligence AIS Automatic Identification System CE Composite Event CER Composite Event Recognition MMSI Maritime Mobility Service Identity RTEC Run-Time Event Calculus SDE Simple Derived Events SAR Search And Rescue

M. Pitsikalis 63 Composite event patterns for maritime monitoring

REFERENCES

[1] STANAG 4162. Identification Data Combining Process (edition 3) - AIDPP-01 Edition 1, Tech. Rep. Technical report, NATO Standardization Agency, 2015. NATO UNCLASSIFIED. [2] Lorenzo Affetti, Riccardo Tommasini, Alessandro Margara, Gianpaolo Cugola, and Emanuele Della Valle. Defining the execution semantics of stream processing engines. Journal of Big Data, 4(1):12, Apr 2017. [3] Elias Alevizos, Alexander Artikis, Georgios Paliouras, Evangelos Michelioudakis, Ehab Qadah, Michael Mock, and Georg Fuchs. datacron: D3.5 complex event forecasting. 2018. [4] Elias Alevizos, Alexander Artikis, Kostas Patroumpas, Marios Vodas, Yannis Theodoridis, and Nikos Pelekis. How not to drown in a sea of information: An event recognition approach. In 2015 IEEE International Conference on Big Data, Big Data 2015, Santa Clara, CA, USA, October 29 - November 1, 2015, pages 984–990, 2015. [5] Sten Andler, Mikael Fredin, Per Gustavsson, Joeri Laere, Maria Nilsson, and Pontus Svenson. Smartracin–a concept for spoof resistant tracking of vessels and detection of adverse intentions, 05 2009. [6] A. Artikis, M. Weidlich, A. Gal, V. Kalogeraki, and D. Gunopulos. Self-adaptive event recognition for intelligent transport management. In 2013 IEEE International Conference on Big Data, pages 319–325, Oct 2013. [7] Alexander Artikis, Alessandro Margara, Martin Ugarte, Stijn Vansummeren, and Matthias Weidlich. Complex event recognition languages: Tutorial. In Proceedings of the 11th ACM International Confer- ence on Distributed and Event-based Systems, DEBS ’17, pages 7–10, New York, NY, USA, 2017. ACM. [8] Alexander Artikis, Marek J. Sergot, and Georgios Paliouras. An event calculus for event recognition. IEEE Trans. Knowl. Data Eng., 27(4):895–908, 2015. [9] Lars Brenna, Alan Demers, Johannes Gehrke, Mingsheng Hong, Joel Ossher, Biswanath Panda, Mirek Riedewald, Mohit Thatte, and Walker White. Cayuga: A high-performance event processing engine, 01 2007. [10] Konstantinos Chatzikokolakis, Dimitrios Zissis, Giannis Spiliopoulos, and Konstantinos Tserpes. Min- ing vessel trajectory data for patterns of search and rescue. Zenodo, March 2018. [11] Keith L. Clark. Negation as failure. In H. Gallaire and J. Minker, editors, Logic and Databases, pages 293–322. Plenum Press, 1978. [12] Gianpaolo Cugola and Alessandro Margara. Tesla: a formally defined event specification language. In DEBS, 2010. [13] Gianpaolo Cugola and Alessandro Margara. Processing flows of information: From data stream to complex event processing. ACM Comput. Surv., 44(3):15:1–15:62, June 2012. [14] Paul Delaunay and Jules-Edouard Pouessel. Complex event recognition support in the context of eu-h2020 datacron. Ecole Navale, 2018. [15] Ryan William B. F., Carbotte Suzanne M., Coplan Justin O., O’Hara Suzanne, Melkonian Andrew, Arko Robert, Weissel Rose Anne, Ferrini Vicki, Goodwillie Andrew, Nitsche Frank, Bonczkowski Juliet, and Zemsky Richard. Global multi-resolution topography synthesis. Geochemistry, Geophysics, Geosys-

M. Pitsikalis 64 Composite event patterns for maritime monitoring

tems, 10(3). [16] Harris Georgiou, Sophia Karagiorgou, Kostas Patroumpas, Nikos Pelekis, Petros Petrou, Stylianos Sideridis, Eugenia Stoufi, and Yannis Theodoridis. datacron: D2.1 cross-streaming, real-time detection of moving object trajectories. 2017. [17] Harm Greidanus, Marlene Alvarez, Carlos Santamaria, François-Xavier Thoorens, Naouma Kourti, and Pietro Argentieri. The sumo ship detector algorithm for satellite radar images. Remote Sensing, 9(3), 2017. [18] Bilal Idiri and Aldo Napoli. The automatic identification system of maritime accident risk using rule- based reasoning. In 2012 7th International Conference on System of Systems Engineering (SoSE), pages 125–130, July 2012. [19] Xiang Jiang, Daniel Silver, Baifan Hu, Erico N. de Souza, and Stan Matwin. Fishing activity detection from ais data using autoencoders, 05 2016. [20] Anne-Laure Jousselme, Elena Camossi, Melita Hadzagic, Cyril Ray, Christophe Claramunt, Eric Rear- don, Karna Bryan, and Michael Ilteris. A Fishing Monitoring Use Case in Support to Collaborative Research. In In Proceedings of Maritime Knowledge Discovery and Anomaly Detection Workshop, pages 57–61, 2016. [21] Anne-Laure Jousselme, Cyril Ray, Elena Camossi, Melita Hadzagic, Christophe Claramunt, Karna Bryan, Eric Reardon, and Michael Ilteris. Maritime use case description, h2020 datacron project deliv- erable d5.1. 2016. [22] Robert A. Kowalski and Marek J. Sergot. A logic-based calculus of events. New Generation Comput., 4(1):67–95, 1986. [23] Chittaro L. and Montanari1 A. Efficient temporal reasoning in the cached event calculus. Computational Intelligence, 12(3):359–382. [24] Craig M. Mills, Sunny E. Townsend, Simon Jennings, Paul D. Eastwood, and Carla A. Houghton. Estimating high resolution trawl fishing effort from satellite-based vessel monitoring system data. ICES Journal of Marine Science, 64(2):248–255, 2007. [25] Barzan Mozafari, Kai Zeng, Loris D’Antoni, and Carlo Zaniolo. High-performance complex event processing over hierarchical data. 38:1–39, 11 2013. [26] Ferdinand Oberle, Curt Storlazzi, and Till Hanebuth. What a drag: Quantifying the global impact of chronic bottom trawling on continental shelf sediment. 159, 06 2016. [27] Kostas Patroumpas, Elias Alevizos, Alexander Artikis, Marios Vodas, Nikos Pelekis, and Yannis Theodoridis. Online event recognition from moving vessel trajectories. GeoInformatica, 21(2):389– 427, 2017. [28] Kostas Patroumpas, Alexander Artikis, Nikos Katzouris, Marios Vodas, Yannis Theodoridis, and Nikos Pelekis. Event recognition for maritime surveillance. In EDBT, 2015. [29] Manolis Pitsikalis, Ioannis Kontopoulos, Alexander Artikis, Elias Alevizos, Paul Delaunay, Jules- Edouard Pouessel, Richard Dreo, Cyril Ray, Elena Camossi, Anne-Laure Jousselme, and Melita Hadzagic. Composite event patterns for maritime monitoring. In Proceedings of the 10th Hellenic Conference on Artificial Intelligence, SETN ’18, pages 29:1–29:4, New York, NY, USA, 2018. ACM. [30] Teodor Przymusinski. On the declarative semantics of stratified deductive databases and logic programs. In Foundations of Deductive Databases and Logic Programming. Morgan, 1987.

M. Pitsikalis 65 Composite event patterns for maritime monitoring

[31] Cyril RAY, Richard DRÉO, Elena CAMOSSI, and Anne-Laure JOUSSELME. Heterogeneous Inte- grated Dataset for Maritime Intelligence, Surveillance, and Reconnaissance, February 2018. [32] Cyril Ray, Aldo Napoli, Alain Bouju, and Pierre-Yves Martin. Detection of faked AIS messages and Resulting Risks. In IF&GIS 2015 - 7th International Workshop on Information Fusion and Geographic Information Systems , Grenoble, France, May 2015. [33] Maria Riveiro. Visual Analytics for Maritime Anomaly Detection. PhD thesis, University of SkövdeUni- versity of Skövde, School of Humanities and Informatics, The Informatics Research Centre, 2011. [34] J. Roy and M. Davenport. Exploitation of maritime domain ontologies for anomaly detection and threat analysis. In 2010 International WaterSide Security Conference, pages 1–8, Nov 2010. [35] Erik Sandewall. M.shanahan, solving the frame problem*. 123, 01 2000. [36] Georgios M. Santipantakis, Akrivi Vlachou, Christos Doulkeridis, Alexander Artikis, Ioannis Kontopou- los, and George A. Vouros. A stream reasoning system for maritime monitoring. TIME 2018. [37] François Schnitzler, Alexander Artikis, Matthias Weidlich, Ioannis Boutsis, Thomas Liebig, Nico Pi- atkowski, Christian Bockermann, Katharina Morik, Vana Kalogeraki, Jakub Marecek, Avigdor Gal, Shie Mannor, Dermot Kinane, and Dimitrios Gunopulos. Heterogeneous stream processing and crowd- sourcing for traffic monitoring: Highlights. In Toon Calders, Floriana Esposito, Eyke Hüllermeier, and Rosa Meo, editors, Machine Learning and Knowledge Discovery in Databases, pages 520–523, Berlin, Heidelberg, 2014. Springer Berlin Heidelberg. [38] Marek Sergot. Knowledge Representation - Event Calculus. 2005. [39] Hamed Yaghoubi Shahir, Uwe Glasser, Amir Yaghoubi Shahir, and Hans Wehn. Maritime situation analysis framework: Vessel interaction classification and anomaly detection. In 2015 IEEE International Conference on Big Data (Big Data), pages 1279–1289, Oct 2015. [40] Lauro Snidaro, Ingrid Visentini, and Karna Bryan. Fusing uncertain knowledge and evidence for maritime situational awareness via markov logic networks. Inf. Fusion, 21:159–172, January 2015. [41] Fernando Terroso-Saenz, Mercedes Valdes-Vela, and Antonio F. Skarmeta-Gomez. A complex event processing approach to detect abnormal behaviours in the marine environment. Information Systems Frontiers, 18(4):765–780, Aug 2016. [42] Joeri van Laere and Maria Nilsson. Evaluation of a workshop to capture knowledge from subject matter experts in maritime surveillance. In Proceedings of FUSION, pages 171–178, 2009. [43] Arnaud Vandecasteele and Aldo Napoli. An enhanced spatial reasoning ontology for maritime anomaly detection. In 2012 7th International Conference on System of Systems Engineering (SoSE), pages 1–6, 2012. [44] Christos Vlassopoulos and Alexander Artikis. Towards a simple event calculus for run-time reasoning. In COMMONSENSE, 2017. [45] Haopeng Zhang, Yanlei Diao, and Neil Immerman. On complexity and optimization of expensive queries in complex event processing. In Proceedings of the 2014 ACM SIGMOD International Confer- ence on Management of Data, SIGMOD ’14, pages 217–228, New York, NY, USA, 2014. ACM.

M. Pitsikalis 66