The Complex Event Recognition Group
Total Page:16
File Type:pdf, Size:1020Kb
The Complex Event Recognition Group Elias Alevizos Alexander Artikis Nikos Katzouris Evangelos Michelioudakis Georgios Paliouras Institute of Informatics & Telecommunications, National Centre for Scientific Research (NCSR) Demokritos, Athens, Greece alevizos.elias, a.artikis, nkatz, vagmcs, paliourg @iit.demokritos.gr f g ABSTRACT and, possibly, a set of spatial constraints, expressing The Complex Event Recognition (CER) group is a re- a composite or complex event of special significance search team, affiliated with the National Centre of Sci- for a given application. The system must then be entific Research “Demokritos” in Greece. The CER efficient enough so that instances of pattern satisfac- group works towards advanced and efficient methods tion can be reported to a user with minimal latency. for the recognition of complex events in a multitude of Such systems are called Complex Event Recognition large, heterogeneous and interdependent data streams. (CER) systems [6, 7, 2]. Its research covers multiple aspects of complex event CER systems are widely adopted in contempo- recognition, from efficient detection of patterns on event rary applications. Such applications are the recog- streams to handling uncertainty and noise in streams, nition of attacks in computer network nodes, hu- and machine learning techniques for inferring interest- man activities on video content, emerging stories ing patterns. Lately, it has expanded to methods for fore- and trends on the Social Web, traffic and transport casting the occurrence of events. It was founded in 2009 incidents in smart cities, fraud in electronic market- and currently hosts 3 senior researchers, 5 PhD students places, cardiac arrhythmias and epidemic spread. and works regularly with under-graduate students. Moreover, Big Data frameworks, such as Apache Storm, Spark Streaming and Flink, have been ex- tending their stream processing functionality by in- 1. INTRODUCTION cluding implementations for CER. The proliferation of devices that work in real- There are multiple issues that arise for a CER sys- time, constantly producing data streams, has led to tem. As already mentioned, one issue is the require- a paradigm shift with respect to what is expected ment for minimal latency. Therefore, a CER sys- from a system working with massive amounts of tem has to employ highly efficient reasoning mecha- data. The dominant model for processing large- nisms, scalable to high-velocity streams. Moreover, scale data was one that assumed a relatively fixed pre-processing steps, like data cleaning, have to be database/knowledge base, i.e., it assumed that the equally efficient, otherwise they constitute a \lux- operations of updating existing records/facts and ury" that a CER system cannot afford. In this case, inserting new ones were infrequent. The user of such the system must be able to handle noise. This may a system would then pose queries to the database, be a requirement, even if perfectly clean input data arXiv:1802.04086v1 [cs.AI] 12 Feb 2018 without very strict requirements in terms of latency. is assumed, since domain knowledge is often insuf- While this model is far from being rendered ob- ficient or incomplete. Hence, the patterns defined solete (on the contrary), a system aiming to ex- by the users may themselves carry a certain degree tract actionable knowledge from continuously evolv- of uncertainty. Moreover, it is quite often the case ing streams of data has to address a new set of chal- that such patterns cannot be provided at all, even lenges and satisfy a new set of requirements. The by domain experts. This poses a further challenge of basic idea behind such a system is that it is not how to apply machine learning techniques in order always possible, or even desirable, to store every to extract patterns from streams before a CER sys- bit of the incoming data, so that it can be later tem can actually run with them. Standard machine processed. Rather, the goal is to make sense out of learning techniques are not always directly applica- these streams of data, without having to store them. ble, due to the size and variability of the training This is done by defining a set of queries/patterns, set. As a result, machine learning techniques must continuously applied to the data streams. Each work in an online fashion. Finally, one often needs such pattern includes a set of temporal constraints to move beyond detecting instances of pattern sat- 1 ) 0.9 ECcrisp isfaction into forecasting when a pattern is likely to 0.8 MLN –EC 0.7 be satisfied in the future. 0.6 1 0.5 Our CER group at the National Centre for Sci- input data | 0.4 entific Research (NCSR) Demokritos, in Athens, 0.3 CE Greece, has been conducting research on CER for ( 0.2 P 0.1 0 the past decade, and has developed a number of 0 3 10 20 time I novel algorithms and publicly available software tools. initiation initiation termination In what follows, we sketch the approaches that we have proposed and present some indicative results. Figure 1: CE probability estimation in the Event Calculus. The solid line concerns a probabilistic 2. COMPLEX EVENT RECOGNITION Event Calculus, such as MLN-EC, while the dashed line corresponds to a crisp (non-probabilistic) ver- Numerous CER systems have been proposed in sion of the Event Calculus. Due to the law of in- the literature [6, 7]. Recognition systems with a ertia, the CE probability remains constant in the logic-based representation of complex event (CE) absence of input data. Each time the initiation con- patterns, in particular, have been attracting atten- ditions are satisfied (e.g., in time-points 3 and 10), tion since they exhibit a formal, declarative seman- the CE probability increases. Conversely, when the tics [2]. We have been developing an efficient di- termination conditions are satisfied (e.g., in time- alect of the Event Calculus, called `Event Calculus point 20), the CE probability decreases. for Run-Time reasoning' (RTEC) [4]. The Event Calculus is a logic programming formalism for rep- resenting and reasoning about events and their ef- 3. UNCERTAINTY HANDLING fects [14]. CE patterns in RTEC identify the con- CER applications exhibit various types of uncer- ditions in which a CE is initiated and terminated. tainty, ranging from incomplete and erroneous data Then, according to the law of inertia, a CE holds at streams to imperfect CE patterns [2]. We have been a time-point T if it has been initiated at some time- developing techniques for handling uncertainty in point earlier than T , and has not been terminated CER by extending the Event Calculus with proba- in the meantime. bilistic reasoning. Prob-EC [21] is a logic program- RTEC has been optimised for CER, in order to ming implementation of the Event Calculus using be scalable to high-velocity data streams. A form of the ProbLog engine [13], that incorporates proba- caching stores the results of subcomputations in the bilistic semantics into logic programming. Prob-EC computer memory to avoid unnecessary recomputa- is the first Event Calculus dialect able to deal with tions. A set of interval manipulation constructs sim- uncertainty in the input data streams. For exam- plify CE patterns and improve reasoning efficiency. ple, Prob-EC is more resilient to spurious data than A simple indexing mechanism makes RTEC robust the standard (crisp) Event Calculus. to events that are irrelevant to the patterns we want MLN-EC [22] is an Event Calculus implementa- to match and so RTEC can operate without data fil- tion based on Markov Logic Networks (MLN)s [20], tering modules. Finally, a `windowing' mechanism a framework that combines first-order logic with supports real-time CER. One main motivation for graphical models, in order to enable probabilistic RTEC is that it should remain efficient and scalable inference and learning. CE patterns may be associ- in applications where events arrive with a (variable) ated with weight values, indicating our confidence delay from, or are revised by, the underlying sen- in them. Inference can then be performed regard- sors: RTEC can update the intervals of the already ing the time intervals during which CEs of inter- recognised CEs, and recognise new CEs, when data est hold. Like Prob-EC, MLN-EC increases the arrive with a delay or following revision. probability of a CE every time its initiating con- RTEC has been analysed theoretically, through a ditions are satisfied, and decreases this probability complexity analysis, and assessed experimentally in whenever its terminating conditions are satisfied, several application domains, including city trans- as shown in Figure 1. Moreover, in MLN-EC the port and traffic management [5], activity recogni- domain-independent Event Calculus rules, express- tion on video feeds [4], and maritime monitoring ing the law of inertia, may be associated with weight [18]. In all of these applications, RTEC has proven values, introducing probabilistic inertia. This way, capable of performing real-time CER, scaling to large the model is highly customisable, by tuning appro- data streams and highly complex event patterns. priately the weight values with the use of machine 1http://cer.iit.demokritos.gr/ learning techniques, and thus achieves high predic- 1 paths. Then, for all incorrectly predicted CEs, the 0.9 hypergraph is searched using relational pathfind- 0.8 ing, for clauses supporting the recognition of these 0.7 CEs. The paths discovered during the search are 0.6 generalised into first-order clauses. Subsequently, score 0.5 the weights of the clauses that pass the evaluation 1 F 0.4 stage are optimised using off-the-shelf online weight 0.3 learners. Then, the weighted clauses are appended 0.2 MLN –EC l–CRF to the hypothesis and the procedure is repeated for 0.1 the next set of training examples t+1.