Darko Anicic Event Processing and Stream Reasoning with ETALIS Zur Erlangung des akademischen Grades eines Doktors der Wirtschaftswissenschaften (Dr. rer. pol.) von der Fakultät für Wirtschaftswissenschaften des Karlsruher Instituts für Technologie (KIT) genehmigte Dissertation. Event Processing and Stream Reasoning with ETALIS Dipl.-Ing. Darko Aniciˇ c´ Referent: Prof. Dr. Rudi Studer Korreferent: Prof. Dr. Opher Etzion Prüfer: Prof. Dr. Detlef Seese Vorsitzender der Prüfungskommission: Prof. Dr. Andreas Geyer-Schulz Tag der mündlichen Prüfung: 09. November 2011. Institut für Angewandte Informatik und Formale Beschreibungsverfahren (AIFB) Fakultät Wirtschaftswissenschaften Karlsruher Institut für Technologie (KIT) Karlsruhe 2011 i To Vedrana and Andre Abstract Event Processing (EP) is concerned with detection of situations under time constraints that are of a particular business interest. We face today a paradigm shift toward the real time information processing, and EP has therefore spawned significant attention in science and technology. Due to omnipresence of events, EP is becoming a central as- pect of new distributed systems such as cloud computing and grid systems, mobile and sensor-based systems, as well as a number of application areas including financial ser- vices, business intelligence, social and collaborative networking, click stream analysis and many others. However, there are a number of issues to be considered in order to enable effective event-based computation. A language for describing event patterns needs to feature a well-defined semantics. It also needs to be rich enough to express important classes of event patterns. Pattern matching should be supported in both, query-driven and event- driven modes. A number of other event operations, such as event aggregation, filtering, translation, enrichment and splitting, should be supported too. Since EP is a real time processing task, an EP language needs to feature an efficient execution model. Finally, processing only events is not sufficient in many applications. To detect complex situa- tions of interest, EP needs to be enhanced by background knowledge. This knowledge captures the domain of interest. Its purpose is to be evaluated during detection of events in order to on the fly enrich events with relevant background information; to detect more complex situations; to reason about events and propose certain intelligent recommen- dations; or to accomplish event classification, clustering, filtering and so forth. The ETALIS Language for Events (ELE) is a declarative rule-based language for EP. It supports the above mentioned features, and goes beyond the state of the art by providing stream reasoning capabilities. In this thesis, we first review related literature and extract requirements for modern EP systems. Then we present ELE as a novel expressive formalism that fulfils these requirements. Further on, we show how deductive stream reasoning capabilities of ELE, together with its EP capabilities, have the potential to provide powerful real time intelligence. We give a few extensions of the core ELE. A number of examples and use case scenarios are developed to show the power of the proposed EP framework. We provide a prototype implementation of the language, and present evaluation results for implemented scenarios. Finally, we summarise the results of this thesis and outline our view of the emerging future work. Acknowledgements On my journey to complete this thesis many people have contributed. Foremost, I would like to sincerely thank to my supervisor, Prof. Dr. Rudi Studer, who gave me the opportunity to work in his research group and continuously supported my work. Prof. Dr. Rudi Studer has created a unique research environment that made my research both possible and enjoyable. I am grateful to Dr. Opher Etzion who was willing to serve as a second reviewer of this work. I wish to sincerely thank him also for inviting me to participate in many initiatives and collaborations within the event processing community – most notably the joint work on The Event Processing Manifesto, and participation in the implementation of Fast Flower Delivery use case in ETALIS. I am indebted to my supervisor, Dr. Nenad Stojanovic, for his unmeasurable enthusiasm and inspiration throughout my PhD studies. Ha gave me the necessary freedom to pursue my ideas, and has continuously encouraged me to always further my work. The work presented in this thesis would not be possible without my colleagues Dr. Sebastian Rudolph, Dr. Paul Fodor, Jia Ding, Ahmed Khalil Hafsi and Vesko Georgiev. Without Sebastian this thesis would not have the same level of technical profoundness. Since the time I started the work on ETALIS, Paul has been providing immeasurable help in implementation of ETALIS. Jia Ding, Ahmed Khalil Hafsi and Vesko Georgiev have also greatly contributed to successful development and testing of ETALIS. I would like to thank all the (former and present) colleagues in the Rudiverse for pro- viding such a friendly working atmosphere. I specially need to thank my colleagues from the iCEP group: Sinan Sen, Roland Stühmer, Jun Ma, Yongchun Xu and Dominik Riemer. Roland Stühmer has also generously helped with the early work on ETALIS. My parents and sister receive my sincerest gratitude and love for giving me continuous support and encouragement in my work. I thank my wife for giving me inspiration and energy throughout the time it has taken to complete this thesis. Darko Aniciˇ c´ Karlsruhe, June 2011 Contents List of Figures xiii List of Tables xv List of Algorithms xvii I Introduction 1 1 Thesis Overview 3 1.1 Introduction ................................ 3 1.2 Shifting Event Processing Toward More Intelligent Event Processing . 5 1.3 Shifting Reasoning Toward Stream Reasoning .............. 7 1.4 Aims and Objectives ........................... 11 1.5 Thesis Organization ............................ 14 1.6 Relation to Previous Publications ..................... 15 2 Introduction to Event Processing 17 2.1 What is an Event? ............................. 17 2.2 Event Programming Principles ...................... 18 2.2.1 Events as a Means to Declare Changes .............. 19 2.2.2 Information Push Versus Information Pull ............ 19 2.3 Event Processing Architecture ...................... 20 3 Introduction to Logic Programming 25 3.1 Background of Logic Programming ................... 25 3.2 The Logic in Logic Programs ....................... 26 3.2.1 Syntax of Definite Programs ................... 26 3.2.2 Semantics of Definite Programs ................. 30 3.3 The Control in Logic Programs ...................... 33 3.3.1 Immediate Consequence Operator ................ 34 3.3.2 SLD Resolution .......................... 35 4 State of the Art 39 4.1 Active Databases ............................. 39 viii Contents 4.2 Event Processing Systems ........................ 42 4.3 Approaches for Retraction in Event Processing ............. 46 4.4 Approaches for Out-of-Order Event Processing ............. 49 4.5 Logic-Based Approaches in Event Processing .............. 50 4.6 Semantic-Based Approaches ....................... 52 4.6.1 Temporal RDF .......................... 52 4.6.2 Stream Reasoning Approaches .................. 53 II ETALIS Language for Events 55 5 Logic-Based Event Processing: Design Principles and Requirements 57 5.1 Formal Declarative Semantics ...................... 57 5.2 Point-Based Versus Interval-Based Temporal Semantics ......... 59 5.3 Seamless Integration of Events with Queries and Domain Knowledge ............................ 60 5.3.1 Query Processing ......................... 60 5.3.2 Knowledge Processing ...................... 60 5.4 Event-Driven Incremental Reasoning ................... 61 5.5 Expressivity ................................ 61 5.6 Set at a Time Versus Event at a Time Processing ............. 62 5.7 Simplicity and Ease-of-Use ........................ 63 5.8 Extensibility ................................ 65 6 ETALIS: A Rule-Based Language for Event Processing and Reasoning 67 6.1 Introduction ................................ 67 6.2 Syntax of the Language .......................... 69 6.3 Declarative Semantics of the Language .................. 73 6.3.1 Complexity Properties ...................... 75 6.4 Examples ................................. 75 6.4.1 An Example Application ..................... 77 7 Operational Semantics of the Language 79 7.1 Overview ................................. 79 7.2 Execution Model for ETALIS ....................... 80 7.2.1 Sequence ............................. 80 7.2.2 Conjunction ............................ 84 7.2.3 Concurrency ........................... 87 7.2.4 Disjunction ............................ 87 7.2.5 Negation ............................. 88 7.2.6 Interval-Based Operations .................... 88 7.2.6.1 Duration ........................ 88 7.2.6.2 Start .......................... 89 7.2.6.3 Equal .......................... 89 7.2.6.4 Finish ......................... 89 Contents ix 7.2.6.5 Meet .......................... 90 7.3 Iterative and Aggregative Patterns .................... 90 7.3.1 From Event Rules to Event Iterative Rules ............ 90 7.3.1.1 An Example Application with Iterative Rules ..... 91 7.3.2 Implementation of Aggregative Patterns ............. 92 7.4 Consumption Policies ........................... 96 7.4.1 Consumption Policies Defined on Time Points ......... 97 7.4.2 Consumption Policies Defined on Time Intervals ........ 98 8 The Event Processing Network in ETALIS 101 8.1 Filtering