Distributed Event Based Systems – Complex Event Processing Netzprogrammierung (Algorithmen und Programmierung V) Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 Our topics last week Descriptive models for distributed system design Physical Architectural model Interaction model model Architectural elements Communicating Communication paradigm Roles and res- Placement Interaction entities ponsibilities model Processes Inter-process communication Architectural styles Multiple server UDP TCP Multi- Objects sockets sockets cast Client-server Proxy/Cache Failure Components model Web Services Indirect Remote Peer-to-peer Mobile code communication invocation Architectural patterns Security Vertical distribution Horizontal distribution model Multi-tier Thin/Fat Client Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 2 Our topics today • Distributed Event Based Systems • Complex Event Processing • CEP Languages • CEP Reference Architecture and CEP Functions Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 3 Event Processing vs. Databases Ex-Post Data Queries Real Time Data Processing Database Queries Event ? Subscriptions ? Incoming Events ? Query Processor ? ? Time past Future Processing moment of events Database Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 4 Knowledge Value of Events Value of Events Proactive actions Real-Time Post-Processing Late reaction or Long term report Historical Event Before the event At event Some time after event e.g. 1 hour Time Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 5 Complex Events – What are they? Complex Events are aggregates, derivations, etc. of Simple Events Data Realt Time Simple Events Simple Events Event Complex Events Data Simple Events Patterns Processing Simple Events Complex Event Processing (CEP) will enable, e.g. – Detection of state changes based on observations – Prediction of future states based on past behaviours Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 6 Complex Event Processing – What is it? CEP is about complex event detection and reaction to complex events - Efficient (near real-time) processing of large numbers of events - Detection, prediction and exploitation of relevant complex events - Supports situation awareness, track & trace, sense & respond new auto pay account login Continuous Event Queries logout deposit withdrawal new auto pay transfer INSERT INTO OrdersMatch_s book logout account login SELECT B.ID, S.ID, B.Price deposit FROM BuyOrders_s B, Trades_s T, SellOrders_s S, enquiry Event Cloud MATCHING [30 SECONDS: B, !T, S] (unordered events) enquiry ON B.Price = S.Price = T.Price; account balance request incident new auto pay enquiry account login deposit withdrawal new auto pay Patterns, Rules transfer account login logout enquiry deposit Complex Events activity history B Event Streams A C (ordered events) Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 7 Complex Event Processing – What is it? Complex Event Processing (CEP) is a Abstract View discipline that deals with event-driven behavior Selection, aggregation, and event abstraction for Complex Event Processing Media Processing Event Complex Technical View generating higher level complex events of interest Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 8 CEP – Example Application Domains Monitoring, BAM, RTE ITSM, Monitor and detect Quick decisions, exceptional IT and reactions to service and threats and business behavior opportunities from occurred according to events events in business transactions CEP Media Detect Decide Respond Enterprise Decision Valuable Management Information at the Right Time Expert Systems to the Right Recipient Information Expert Decision Dissemination Management Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 9 Core CEP Life Cycle 4. Event Aggregation 5. Event Handling 3. Event Selection CEP Media 6. Event 2. Event Consumption Definition 1. Event Production Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 10 Basic Definition (1) Atomic Event (also raw event or primitive event) - An atomic event (also raw event or primitive event) is defined as an instantaneous (at a specific point in time), significant (relates to a context), indivisible (cannot be further decomposed and happens completely or not at all) occurrence of a happening. Complex Event - composed (composite event) or derived (derived event) from occurred atomic or other complex event instances, e.g. according to the operators of an event algebra or as a result of applying an algorithmic function or process to one or more other events - included events are called components while the resulting complex event is the parent event - first event instance contributing to the detection of a complex event is called initiator, where the last is the terminator; all others are called interiors Event Pattern - An event pattern (also event class, event definition, event schema, or event type) describes the structure and properties of an (atomic or complex) event - It describes on an abstract level the essential factors that uniquely identify the occurrence of an event of that type, i.e. its detection condition(s) and its properties with respect to instantiation, selection and consumption. Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 11 Basic Definition (2) Event Instance - A concrete instantiation of an event pattern is a specific event instance (also event object). Situation - A situation is initiated or terminated by one or more (complex) events, i.e. the effect of a complex event (complex event + conditional context) Situation Individuals - Situation individuals are discrete individuals of situations with a fixed context allocation of time, date, location, participant, etc. They are not repeatable and are temporally connected. Complex Action - Compound action from occurred atomic or other complex action instances Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 12 Complex Event Patterns – How? Example Snoop Event Algebra Operators: •Sequence Operator (;): (E1;E2) •Disjunction Operator (∨): (E1 ∨ E2 ) , at least one •Conjunction Operator (∧): (E1 ∧ E2) •Simultaneous Operator (=): (E1 = E2) •Negation Operator (¬): (E1 ∧ ¬ Ε2 ) •Quantification (Any): Any(n) E1, when n events of type E1 occurs •Aperiodic Operator (Ap): Ap(E2, E1, E3) , E2 Within E1 & E3 •Periodic Operator (Per): Per(t, E1, E2) , every t time-steps in between E1 and E2 Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 13 Example – Interval-based Sequence –Example: D = A;(B;C) T1 T2 T3 T4 A B C Composition [A,A] [B,C] [[A,A],[B,C]] Consolidation D = [A,C] Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 14 Event Detection Operators & Windowing Processing time Time Window [t-5 <–> t] Future t-8 t-7 t-6 t-5 t-4 t-3 t-2 t-1 t t+1 t+2 t+3 t+4 t+5 t+6 t+7 Events A1 B1 C1 A2 B2 D1 A3 B3 C3 t-1 A1 B1 C1 A2 B2 D1 A3 B3 C3 t-2 A1 B1 C1 A2 B2 D1 A3 B3 C3 t-3 A1 B1 C1 A2 B2 D1 A3 B3 C3 Event Detection Pattern [A ; B] Matches Time t → 2 or 3 (based on event consumption policy) Time t-1 → 1 Time t-2 → 1 Time t-3 → 2 or 3 Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 15 Event Processing Agent and Event Processing Network Event Processing Network is a collection of Event Processing Agents. The EPN describes the p1 p5 c1 Filter: Diabetes c5 admittance p6 “programming in the large”, EPN Example p12 Pattern detection: Alert physician p7 while each individual agent Blood pressure reading p11 p8 p10 Enrich: c2 c6 describes the “programming p2 p9 Is diabetic? p12 Aggregator: Max. c7 Fever C p14 in the small”. Translate: p13 c3 C2F fever p19 Fever F p3 p15 Fever F Pattern Detection: p4 c8 Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 IBM Haifa Research Lab – Event Processing © 2008 IBM Corporation Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 16 What is the idea ? The EPN (event processing network) represents the event processing application as a directed graph: - The nodes represent event processing agents (and states in some models) - The edges represent either individual events or event streams (depends on the processing type). A generalized EPN may support both. The coverage of EPN varies - It may cover some language operator (e.g. SQL like language, rule language, …) - It may take a broader approach and cover also routing decisions, pub/sub etc… - EPN can have a specific programming model, or an hybrid programming model, where one or more languages represent the agents. Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 17 p1 p5 c1 Filter: Diabetes c5 admittance p6 EPN example Pattern detection: p12 Alert physician p7 Blood pressure reading p11 p8 p10 Enrich: c2 c6 p2 p9 Is diabetic? p12 Aggregator: Max. c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 p15 Fever F Pattern Detection: p4 c8 Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever p17 c9 Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 18 Example Event Processing Languages Reaction Inference TIBCO WBE Rules RuleML XChangeEQ Prova ECA / Reaction Rules RuleCore AMiT Starview Agent Agent Logic Oriented EventZero Spade SQL Aleri extension Streambase Coral8 State Oracle oriented Esper Imperative/ Netcool Impact Script Based Apama Barry Linnert, [email protected], Netzprogrammierung