Complex Event Processing

Distributed Event Based Systems – Complex Event Processing

Netzprogrammierung (Algorithmen und Programmierung V)

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 Our topics last week

Descriptive models for distributed system design

Physical Architectural model Interaction model model Architectural elements

Communicating Communication paradigm Roles and res- Placement Interaction entities ponsibilities model

Processes Inter-process communication Architectural styles Multiple server UDP TCP Multi- Objects sockets sockets cast Client-server Proxy/Cache Failure Components model Web Services Indirect Remote Peer-to-peer Mobile code communication invocation

Architectural patterns Security Vertical distribution Horizontal distribution model Multi-tier Thin/Fat Client

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 2 Our topics today • Distributed Event Based Systems • Complex Event Processing • CEP Languages • CEP Reference Architecture and CEP Functions

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 3 Event Processing vs. Databases

Ex-Post Data Queries Real Time Data Processing

Database Queries Event ? Subscriptions ? Incoming Events ?

Query Processor ? ?

Time past Future Processing moment of events Database

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 4 Knowledge Value of Events

Value of Events

Proactive actions

Real-Time Post-Processing Late reaction or Long term report Historical Event

Before the event At event Some time after event e.g. 1 hour Time

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 5 Complex Events – What are they?

Complex Events are aggregates, derivations, etc. of Simple Events

Data Realt Time Simple Events Simple Events Event Complex Events Data Simple Events Patterns Processing Simple Events

 Complex Event Processing (CEP) will enable, e.g. – Detection of state changes based on observations – Prediction of future states based on past behaviours

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 6 Complex Event Processing – What is it? CEP is about complex event detection and reaction to complex events - Efficient (near real-time) processing of large numbers of events - Detection, prediction and exploitation of relevant complex events - Supports situation awareness, track & trace, sense & respond

new auto pay account login Continuous Event Queries logout deposit withdrawal new auto pay transfer INSERT INTO OrdersMatch_s book logout account login SELECT B.ID, S.ID, B.Price deposit FROM BuyOrders_s B, Trades_s T, SellOrders_s S, enquiry Event Cloud MATCHING [30 SECONDS: B, !T, S] (unordered events) enquiry ON B.Price = S.Price = T.Price; account balance request incident new auto pay enquiry account login deposit withdrawal new auto pay Patterns, Rules transfer account login logout enquiry deposit Complex Events activity history B

Event Streams A C (ordered events) Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 7 Complex Event Processing – What is it? Complex Event Processing (CEP) is a

Abstract View discipline that deals with event-driven behavior Selection, aggregation, and event abstraction for

Media Processing Event Complex Technical View generating higher level complex events of interest

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 8 CEP – Example Application Domains

Monitoring, BAM, RTE ITSM, Monitor and detect Quick decisions, exceptional IT and reactions to service and threats and business behavior opportunities from occurred according to events events in business transactions

CEP Media Detect Decide Respond Enterprise Decision Valuable Management Information at

the Right Time Expert Systems to the Right Recipient Information Expert Decision Dissemination Management Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 9 Core CEP Life Cycle

4. Event Aggregation

5. Event Handling 3. Event Selection CEP Media 6. Event 2. Event Consumption Definition

1. Event Production

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 10 Basic Definition (1) Atomic Event (also raw event or primitive event) - An atomic event (also raw event or primitive event) is defined as an instantaneous (at a specific point in time), significant (relates to a context), indivisible (cannot be further decomposed and happens completely or not at all) occurrence of a happening.

Complex Event - composed (composite event) or derived (derived event) from occurred atomic or other complex event instances, e.g. according to the operators of an event algebra or as a result of applying an algorithmic function or process to one or more other events - included events are called components while the resulting complex event is the parent event - first event instance contributing to the detection of a complex event is called initiator, where the last is the terminator; all others are called interiors

Event Pattern - An event pattern (also event class, event definition, event schema, or event type) describes the structure and properties of an (atomic or complex) event - It describes on an abstract level the essential factors that uniquely identify the occurrence of an event of that type, i.e. its detection condition(s) and its properties with respect to instantiation, selection and consumption.

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 11 Basic Definition (2) Event Instance - A concrete instantiation of an event pattern is a specific event instance (also event object).

Situation - A situation is initiated or terminated by one or more (complex) events, i.e. the effect of a complex event (complex event + conditional context)

Situation Individuals - Situation individuals are discrete individuals of situations with a fixed context allocation of time, date, location, participant, etc. They are not repeatable and are temporally connected.

Complex Action - Compound action from occurred atomic or other complex action instances

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 12 Complex Event Patterns – How?

Example Snoop Event Algebra Operators:

•Sequence Operator (;): (E1;E2) •Disjunction Operator (∨): (E1 ∨ E2 ) , at least one •Conjunction Operator (∧): (E1 ∧ E2) •Simultaneous Operator (=): (E1 = E2) •Negation Operator (¬): (E1 ∧ ¬ Ε2 ) •Quantification (Any): Any(n) E1, when n events of type E1 occurs •Aperiodic Operator (Ap): Ap(E2, E1, E3) , E2 Within E1 & E3 •Periodic Operator (Per): Per(t, E1, E2) , every t time-steps in between E1 and E2

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 13 Example – Interval-based Sequence

–Example: D = A;(B;C)

T1 T2 T3 T4

A B C Composition [A,A] [B,C]

[[A,A],[B,C]] Consolidation D = [A,C]

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 14 Event Detection Operators & Windowing Processing time Time Window [t-5 <–> t] Future

t-8 t-7 t-6 t-5 t-4 t-3 t-2 t-1 t t+1 t+2 t+3 t+4 t+5 t+6 t+7 Events A1 B1 C1 A2 B2 D1 A3 B3 C3

t-1 A1 B1 C1 A2 B2 D1 A3 B3 C3

t-2 A1 B1 C1 A2 B2 D1 A3 B3 C3

t-3 A1 B1 C1 A2 B2 D1 A3 B3 C3

Event Detection Pattern [A ; B] Matches Time t → 2 or 3 (based on event consumption policy) Time t-1 → 1 Time t-2 → 1 Time t-3 → 2 or 3

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 15 Event Processing Agent and Event Processing Network

Event Processing Network is a collection of Event Processing Agents.

The EPN describes the p1 p5 c1 Filter: Diabetes c5 admittance p6 “programming in the large”, EPN Example p12 Pattern detection: Alert physician p7

while each individual agent Blood pressure reading p11 p8 p10 Enrich: c2 c6 describes the “programming p2 p9 Is diabetic? p12 Aggregator: Max. c7 Fever C p14 in the small”. Translate: p13 c3 C2F fever p19 Fever F p3 p15 Fever F Pattern Detection: p4 c8 Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever

p17 c9

The EPN (event processing network) represents the event processing application as a directed graph: - The nodes represent event processing agents (and states in some models) - The edges represent either individual events or event streams (depends on the processing type). A generalized EPN may support both.

The coverage of EPN varies - It may cover some language operator (e.g. SQL like language, rule language, …) - It may take a broader approach and cover also routing decisions, pub/sub etc… - EPN can have a specific programming model, or an hybrid programming model, where one or more languages represent the agents. Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 17 p1 p5 c1 Filter: Diabetes c5 admittance p6 EPN example Pattern detection: p12 Alert physician p7

Blood pressure reading p11 p8 p10 Enrich: c2 c6 p2 p9 Is diabetic? p12 Aggregator: Max. c7 Fever C p14 Translate: p13 c3 C2F fever p19 Fever F p3 p15 Fever F Pattern Detection: p4 c8 Alert Nurse p16 p20 Pattern Detection: p18 Continuous Fever

p17 c9

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 18 Example Event Processing Languages

Reaction Inference TIBCO WBE Rules RuleML XChangeEQ Prova ECA / Reaction Rules RuleCore AMiT Starview Agent Agent Logic Oriented EventZero Spade SQL Aleri extension Streambase Coral8 State Oracle oriented Esper

Imperative/ Netcool Impact Script Based Apama

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 19 Some EPN, event flow and data flow oriented languages

Agent Logic Event Zero Oracle/BEA

Spade (IBM System S) Streambase IBM InfoSphere Streams

A SPADE program builds a data-flow network out of operators

source stream processing sink operator operator stream operator

stream

processing stream processing

operator operator stream

stream source sink operator operator

•Finite State Automata, e.g.: –ADAM, ACOOD, ODE, SAMOS, COMPOSE, SNOOP (active database research in 80s & 90s) –SASE, Cayuga, Esper

•Rule-Based CEP Engines, e.g.: –Prolog, Prova, Drools, ETALIS

•Data Stream Engines, e.g.: –Stream CQL, PIPE, TelegraphCQ, Gigascope, Aurora Borealis, SPADE

•Stream Reasoning, e.g.: –C-SPARQL, EP-SPARQL, SCEPter, SPARQLStream, CQELS Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 21

Commercial CEP Market

Source: Tibco Software Inc. 2010 Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 22 Implementations - Example Systems: TIBCO BusinessEvents

TIBCO BusinessEvents “event server” / event processing engines Multiple languages / engines in various packaging options - UML-based event and concept (object) class hierarchical models - Java-based language - Eclipse-based IDE

Distributed execution model: - Inference agents, Query agents, Datagrid agents - In-memory, memory+grid, explicit grid operational modes - “Grid Options” of TIBCO ActiveSpaces / Oracle Coherence - eXtreme Event Processing use cases: Fedex, US Govt, …

Transactional model option: - TIBCO AS Transactions - eXtreme Event Processing use cases: Telco Extensible: eg OWL import, RDF import, etc

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 23 Implementations - Example Systems: TIBCO BusinessEvents (ctd)

Events Event and Data Information Structures

Inference Rules

States and Transitions Continuously process events Sets and using procedural Queries Stored and declarative Events and event processing Temporal Data Patterns elements

Decision Tables

via JMS, RV, MQ, Event Event TCP/IP, etc… Processing Storage

Barry Linnert, [email protected] -berlin.de, Netzprogrammierung WS 2015/16 24

Implementations - Example Systems: Oracle Event Processing (formerly Oracle CEP) Development platform for event processing applications Application model based on EPN (event processing network) abstraction running on top of OSGi-based Java container. Language is an extension of SQL with stream and pattern matching extensions

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 25 Implementations - Example Systems: Oracle Event Processing (formerly Oracle CEP)

CQL: Continuous Query Language Leverages SQL, extended with Stream, Pattern-matching, and Java. Continuous: driven by time, and events Highly-sophisticated push-down technology to RDBMS, Cache, Hadoop

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 26 Implementations - Example Systems: Prova (http://prova.ws) Strong Coupling • Java JVM based, open source rule direct API language for reactive agents and event implementation processing Prova Computer • Leverages declarative ISO Prolog Loose Coupling Agent / Service standard extended with (event,message) Interface

reaction logic, type systems (Java, Message Interchange Prova Ontologies), query built-ins, dynamic Java Agent Request Rule Service integration. 0.0000 … Consumer Response

• Combines declarative, imperative (object- 0.0000 Prova Agent oriented) and functional programming

styles Decoupled Prova Event • Designed to work in distributed Enterprise Event Streams Processing Service Bus and OSGi environments (ordered events) Network new auto pay account login logoutdeposit withdrawalnew auto pay Prova Prova • Supports strong, loose and decoupled transfer logout deposit Eventbook account login Agent Agent enquiry enquiry interaction account balance Cloudincident request new(unordered auto pay accountevents) login enquiry • Compatible with rule interchange depositwithdrawal new auto pay transfer account login Prova enquiry deposit logout standards such as Reaction RuleML activity history Agent Prova Agent Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 27 Implementations - Example Systems: IBM WODM Decision Server Events Decision Server Events component of IBM WebSphere Operational Decision Management (WODM) Manages business events flowing across systems and people to provide timely insight and responses Detect, evaluate, and respond to events Discover event patterns and initiate actions

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 28 Implementations - Example Systems: IBM CICS Transaction Server for z/OS

CICS TS

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 29 Implementations - Example Systems: IBM InfoSphere Streams – Stream Processing Language IBM InfoSphere Streams: platform for analyzing big data in motion i.e. high- volume, continuous data streamsSPL IBM Streams Processing Language (SPL) - Programming language for InfoSphere Streams - Exposes a simple graph-of-operators view - Provides a powerful code-generation interface to C++ and Java

Streams Processing Runtime Tools and

Language and IDE Environment Technology

Integration

Front Office 3.0

Streams Console & Monitoring, Streams Studio Highly Scalable Built-in Stream Relational Analytics, Eclipse IDE for SPL stream processing runtime Adapters, Toolkits

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 30

Functional View source: definitions of EP

Event Processing, Complex Event Processing, Event Stream Processing

Event Abstraction, Event Pattern Detection, Event Composition etc. etc.

Events Derived Events Event Management Event

Event Type Definitions, Type Event Event Producer Event Consumer Event Event Processing Rules, etc. (Event Source, (Event sink, event Event Emitter) handler, event listener)

Design time Run time Administration Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 31 EPTS Reference Architecture - Functional View Architect and Developer perspective - includes the 3 main functions (development, run-time and administration), - targets primarily the automated event processing operations

Run-time functions in 2 main groups: - the event infrastructure (sources and consumers) external to the event processor under consideration, - the event processor.

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 32

Reference Architecture: Functional View

Event Reaction Assessment, Routing, Prediction,

Discovery, Learning Control Monitoring, Process Event

0..* Management State Complex Event Detection Consolidation, Composition, Aggregation 0..* Event Analysis Analytics, Transforms, Tracking, Scoring, Rating, Classification 0..* Event Preparation Identification, Selection, Filtering, Monitoring, Enrichment

Event andComplex Event 0..*

Event Production Event Consumption (Pattern, Control, Rule, Query, RegEx.etc) RegEx.etc) Rule, Query, Control, (Pattern, Publication, Dashboard, Apps, Retrieval External Reaction Definition, Modeling, (continuous)Definition, Modeling, Improvement

Design time Run time Administration

Barry Linnert, [email protected], Netzprogrammierung WS 2015/16 33