Complex Event Processing Tutorial

Paul Vincent TIBCO Software Inc. Presenter workshop: DOC4RTES

Presenter: Paul Vincent, CTO Business Rules and CEP, TIBCO Software Member OMG PRR and W3C RIF rules standards bodies Co-author CEP Blog http://tibcoblogs.com/cep

TIBCO Software Inc.: Provides enterprise software that helps companies achieve service-oriented architecture (SOA) and business process management (BPM) success Headquartered in Palo Alto, California Over 3,000 customers and offices in 40 countries CEP product is TIBCO BusinessEvents Developed from a customer solution and launched 2005 Currently at Release 3.0

Introducing CEP

Customer New Customer Checks Order Logon Fed “Close Account” Base Rate Web Page Increase Contract Mobile Call Submitted Production from CT New Item @11.13 Liability Arrives at Contract Added Store Returned Rental Rental thru EDI Car Car Returned Crashed

Customer New Customer Checks Order Logon Fed Close Account Base Rate Web Page Increase Contract Mobile Call Submitted Production from CT New Item @11.13 Liability Arrives at Contract Added Store Returned Rental Rental thru EDI Car Car Returned Crashed

Change in Fraud Customer Product Sales Risk! CrossSell Trend Risk of Opportunity Customer Defection Contract Cell phone Validated Employee Compliance fraud alert Over hours Limit Approached Contract Customer Valid Rental now rated Contract Gold Complete

workshop: DOC4RTES

New (LotsNew of Order Synchronous Events)Order Events

Simple event processing EventEvent----atatatat----aaaa----timetime

Aggregation as data

IT Services: Business Processes: SOA & traditional Persistence BPM & Workflow Processes Data Processing Services

Based on “human workflow”: one thing at a time Processes handle cases 1 at a time office clerk Use database and refer to it where necessary card index Provide some service flexibility with middleware internal mail Use BPM to document / manage / automate processes Use SOA to distribute / manage / automate services

This model does NOT exploit ALL the information / data / events ALL the time

Behaviour (and business logic) is silo’d T h e r e i s a b e t t e r w a y !

workshop: DOC4RTES

New High performance (LotsNew of Order Events)Order patternpattern----matchingmatching Asynchronous via rules, states, Events & queries

continuous Complex Patterns Event Event of Events Store Processing High performance persistence

IT Services: Business Processes: SOA & traditional Persistence BPM & Workflow Processes Data Processing Services

Event Cloud

Event Streams

CEP (technology) applies pattern detection & filtering to the event clouds & streams and their histories Multiple modelling / execution paradigms are available for pattern detection

CEP

“CEP applies to a very broad spectrum of challenges in information systems. A short list includes:” Business process automation Computer systems to automate scheduling and control network-based processes and processing Identifying when complex contracts are fulfilled Detection intrusion, fraud and other network attacks C3I

The Power of Events , Addison Wesley, ISBN: 0-201-72789-7, 2002

Usually: warnings precede threats

CEP raison ddd’d’’’etreetreetre:: quickly respond to

Value of action of Value detected variances

“Situational Awareness”

“Sense and Respond”

“Track and Trace”

History

Command and Control Correlate all available information Determine tactics based on strategy and up-to-date information

-- from RAF Battle of Britain Fighter Control System 1940 http://www.raf.mod.uk

EVENT SOURCES Data Fusion Domain EXTERNAL

Level 0 Level 1 Level 2 Level 3 DISTRIBUTED Processing Processing Processing Processing LOCAL Sub-Object Object Situation Impact Assessment Assessment Assessment Assessment Sensors

Documents Human/ Computer . People. Interface .. . Database Management . Level 4 System Data Sources Processing Support Fusion Process Database Database Refinement

-- Revised JDL data fusion model, 1998 Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Monitored System

Well informed Decision Fusion Reasoning Classification Feature Extraction Feature

History Mission

-- from “Data Fusion for Developing Predictive Diagnostics for Electromechanical Systems” Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

Events and CEP

Fraud / Theft Relevant event of interest Thousands-to-millions of high-value small-size product items or transactions Product location How do you identify known patterns of “suspicious” change behavior? Logistics / Scheduling Raw material, production & delivery scheduling and resources are complex and prone to change Resource, requirement How do we reallocate resources to handle business and change production changes? Activity Monitoring Complex production and supply process with multiple actors How to measure and action Key Performance Indicators? Process data change

19 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Associated Events workshop: DOC4RTES Positive Events Product item X arrives at Production station S from Store T Production worker Y arrives at Production station S Production contract for item Z by time T is posted Negative Events Product item X has been in transit to Store T for >15 minutes Subcomponent Y hasn’t arrived at the Production station by the ETA Delivery of contract Z has not taken place Sets of Events 5+ items of Product item type Y failed to arrive at destination Supplier Y was 5 mins late for 1 delivery, but made it early to the next Return rate on component Z exceeds SLA %

Time Sensitivity A thief may leave the building at the same time as stolen product 15:39:26.413 A product should take 40 minutes to travel a given production line segment Dec 24 Distributed Event Sources A series of produced items fails at various QA stages, and their common attribute was a storage location Multiple suppliers for a subcomponent are reporting delivery delays

ERP Supplier

21 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. What *is* an “event”? workshop: DOC4RTES Change of state in some entity “Happening” Customer call Bank debit Aircraft movement Observation of some entity Observation CRM record of a customer call ATM report of debit transaction success Radar plot update of an aircraft IT Message Queued point-to-point message IT Message Publish / subscribe message

General Event Channel

Inherits from Timestamp Default Destination Destination TimeToLive Simple Event ExpiryAction

Event contains data + Custom Attributes timestamp Various classifiers (int/ext, transport, lifecycle, …) XML

Complex Inform / multiple events Event Event Processing Sources

Event Processing Agents Simple Invoke / single event Event Processing Event Consumers Business Process Event Store Event Management Bus Event Bus Event

Services

Data Services

S O A E D A

Request ---Reply TransmitTransmit----ListenListen

Synchronous Asynchronous Events Events

Assumptions…

S O A E D A Continuous Event Monitoring

Complex Event Processing

Synchronous Asynchronous Events Events

Assumptions…

workshop: DOC4RTES Sense and Respond / Track and Trace / Situational AwarenessAwareness

Processing type:

Decision Processing

Event Processing

TIBCO Reference Architecture

Decision Latency

Automated Predictive Decisions Human Predictive Decisions Anti-Money Laundering Trade Desk Monitoring Credit-Card Fraud

Database Monitoring Exchange Compliance Customer Interaction Algorithmic Trading --- Bloor Report on Event Processing Automated Operational Decisions Human Operational Decisions Nov 2006

Liquidity Management Order Routing

Baggage Handling Search & Rescue

RFID Rail Networks Tariff Look-Up Process Complexity Process Pattern Matching andPattern Inferencing

Event Complexity

Detecting event patterns across multiple event types + time is difficult for simple event processing solutions Computers can correlate across large volumes of events at high speed, identifying patterns that are not conventionally visible The architecture pattern of “continuous event processing” applies to many business domains such as BAM Examples in use: Track and Trace of RFID data Situation Assessment of airline operational delays (+ their causal events) Sense and Respond to fraud indicators in internet transactions

CEP Technologies

workshop: DOC4RTES

EVENT Proposed for EPTS Reference Architecture SOURCES EVENT PRE- Event-Decision Architecture EXTERNAL PROCESSING

Event Behavior LEVEL ONE LEVEL TWO LEVEL THREE DISTRIBUTED Event Tracking Situation Impact LOCAL & Refinement Detection & Assessment Refinement & Predictive EVENT Event Event Analysis Event SERVICES Behavior Behavior Behavior User . . Interface EVENT. & PROFILES. Systems .. . DATA MANAGEMENT DATA BASES . Historical Profiles & LEVEL FOUR . Data Patterns OTHER DATA Process Refinement &

Adaptation Event Behavior -- Adapted from JDL Data Fusion Model by Tim Bass Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

31 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Requirements for CEP Technology workshop: DOC4RTES Access and Monitor the “Event Cloud” Event Bus or Source JMS, RV, MQ, TCP/IP, etc… Timers to detect lack of events Low-latency reliable message delivery Determine event state changes

Match Patterns, Event and Data Model

Apply Business Logic for information modeling Detect events Detect event patterns Temporal Model for determining

Maintain State and Facts over time and Cache History time-dependent info Update Detection algorithms as events change Pattern Detection Model

for recognizing patterns, defining actions

JMS, (DDS) Event Bus or Source

Low-latency reliable message delivery OMG EMP (RFP in progress)

Event and Data Model OMG UML2 for information modeling Class Models OMG UML2 Temporal Model State Models

for determining time-dependent info OMG PRR, W3C RIF Production Rules

Pattern Detection Model

for recognizing patterns, ANSI SQL-based defining actions continuous queries

History and Cache

Event Services Types of CEP Processing

Rule-Driven CEP Event Sources Inference Rulebase Rule Engine

Neural Net CEP

Neural Net Network

Event Bus

Event Bus Event Event Stream Query CEP

Event Query Engine Stream Queries Consumers

Orchestrated Event Stream CEP

Flow Engine Flow Model

General Behavior External Inherits from Event Input Internal Event-driven Behavior Output Timer Data

Inherits from State Chg

Rule State Query Procedural Event Process Manager Process Process TimeOut

Event Bus or Source Low-latency reliable New message delivery ConditionNew / OrderNew TimeEventOrder Order Event and Data Model

for information modeling

Temporal Model

for determining time-dependent info State 1 State 2

Pattern Detection Model for recognizing patterns, State Model defining actions

History and Cache

36 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. State Model / Process Flow CEP Agent features workshop: DOC4RTES 1. Visual modeling metaphor State diagram / flow diagram is simple to follow 2. State / flow transitions can be time-related Can model missing events through time-outs etc

Event Bus or Source New (LotsNew of Low-latency reliable OrderNew message delivery Events)Order Order

Event and Data Model

for information modeling

Temporal Model

for determining Complex time-dependent info Reaction Patterns

Pattern Detection Model for recognizing patterns, Rule defining actions

History and Cache

38 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Inference Rule CEP Agent features workshop: DOC4RTES 1. High performance pattern matching Rete algorithm determines rules that are executable based on underlying data changes 2. Declarative + Inferencing Rules defined in terms of classes: can be relevant for any # instances Rules’ actions can cause other rules to fire automatically 3. In-memory Limited only by JVM / process memory

Event Bus or Source New (LotsNew of Low-latency reliable OrderNew message delivery Events)Order Order

Event and Data Model

for information modeling

Temporal Model

for determining Complex time-dependent info Results Event Query

Pattern Detection Model for recognizing patterns, Query defining actions

History and Cache

40 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Query CEP Agent features workshop: DOC4RTES 1. Common query language Usually SQL-based – widely used language May be in-memory, in-file or both Can include query optimizers 2. Continuous Extensions usually support time windows for the query to operate over

Examples

workshop: DOC4RTES

Detected Business Situation Resulting Situation-Decision

User X is behaving suspiciously (high Investigate for fraud manually likelihood of fraud)

Subcomponent delivery Y is slightly Issued an automated reminder to late supplier

Customer Y payment for policy P is Alert Customer Agent very late

Orders for product Z are up >20% Alert manufacturing and marketing

Algorithmic Trading CEP Application

Query Querybase Engine

Front Office Middle Office Back Office Compliance CEP Application

State Engine State Model

Data Feeds SOA Archival etc (events) Services Database Inference Rulebase Rule Engine

Event Cache Other Data Update CEP & Track FSA compliance New Classification States check required? Order channelled & filter (Re)Assess Order vs other orders Rulebase State Model

1. Order event arrives over JMS / 4. Comparison rules run against EMS or RV on a particular order by rule engine, new channel information inferred 2. Order event classified into 5. State transition rules executed to appropriate class, stored in cache check for any state change due to for future reference new inferred data. 3. Order state assessed and 6. If order can be completed, run updated due to event details FSA compliance rules

Cache Event Rule Agent(s) Rulesets Preprocessor Node(s)

State Models

Backup DB Function Functions

Rete Working Conceptual Model Memory Events Destinations Channels Query Agent(s) Conceptual Model

ESB / Message Bus / Event Bus /Bus Event / ESB Message Queries Distributed cache CEP engine(s) of events etc

“Advanced” CEP

Intelligent CEP Adaptive Learning Logic AI Semantic CEP Ontologies + Logic Text / language interpretation Multiple CEP Including all types of data processing paradigm (transactional, CLP, inference, mathematical methods, …)

Event Processing CEP Agents Event Sources State Engine State Model

Inference Rule Engine Rulebase

Event Streaming Consumers Query Engine Queries

BI / Analytics tools Event Statistical Business User Interface Bus Pattern Detection CEP Graphical Editors Event Bus / Store Event Dashboard Pattern Detection

Graphical Reports Optimization & Machine Learning

CLP Engine Constraints

BPM / SOA Ontologies, Logic MetaRules

Many EP apps fit the standard CEP patterns: Filter interesting rules Detect predefined patterns / state changes Update data / invoke processes and services based on business rules and high level events Advanced EP: Apply interesting statistical functions to event data to detect new / complex trends Apply different algorithms to event data Modify parameters used in other rules (“metarules”)

EVENT SOURCES EVENT PRE- Event-Decision Architecture EXTERNAL PROCESSING LEVEL ONE LEVEL TWO LEVEL THREE DISTRIBUTED Event Tracking Situation Impact LOCAL & Refinement Detection & Assessment Refinement & Predictive EVENT Analysis SERVICES User . . Interface EVENT. & PROFILES. Systems .. . DB MANAGEMENT DATA BASES . Historical Profiles & LEVEL FOUR . Data Patterns OTHER DATA Process Refinement & Adaptation

-- Adapted from JDL Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001

EVENT PRE- Event-Decision Architecture PROCESSING What are LEVEL ONE LEVEL TWO LEVEL THREE Event Tracking Situation Impact the variables & Refinement Detection & Assessment Refinement & Predictive that can Analysis be adjusted in real-time to optimize system DB MANAGEMENT performance? Historical Profiles & LEVEL FOUR Data Patterns

Process Refinement & Adaptation

workshop: DOC4RTES If RFID event for EVENT PRE- Event-Decision Architecture product X PROCESSING Then LEVEL ONE LEVEL TWO LEVEL THREE monitor, else Event Tracking Situation Impact & Refinement Detection & Assessment ignore Refinement & Predictive Analysis

Becomes

If RFID event for DB MANAGEMENT

product in list Y Historical Profiles & where cost > Z LEVEL FOUR Data Patterns Then monitor, else Process Refinement & ignore Adaptation If average loss increase Updated by for all products in Y > 2% Then reduce Z by 5% 53 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. 24 Event Tracking and Refinement Rules

workshop: DOC4RTES

If drug class X EVENT PRE- Event-Decision Architecture and dose > 200ml PROCESSING Then LEVEL ONE LEVEL TWO LEVEL THREE move to monitored Event Tracking Situation Impact & Refinement Detection & Assessment drug state Refinement & Predictive Analysis

Becomes

If drug class X DB MANAGEMENT and dose > Y ml Historical Profiles & LEVEL FOUR Data Patterns Then Process move to monitored Refinement & drug state Adaptation If clinical negative events for drug class X increase Updated by Then reduce Z by 10ml 54 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. 24 Event Tracking and Refinement Rules

workshop: DOC4RTES

If bag X is not on EVENT PRE- Event-Decision Architecture prescribed flight PROCESSING at (depart – 20) LEVEL ONE LEVEL TWO LEVEL THREE Then Event Tracking Situation Impact & Refinement Detection & Assessment move X state to Refinement & Predictive MissedFlight Analysis Becomes If bag X is not on

prescribed flight DB MANAGEMENT at lastBagTime Historical Profiles & Then LEVEL FOUR Data Patterns move X state to Process Refinement & MissedFlight Adaptation If flight NOT international Updated by Then set lastBagTime to carrier’s min( DoorCloseTime) 55 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. 24 Event Tracking and Refinement Rules

workshop: DOC4RTES

If EVENT PRE- Event-Decision Architecture product.ShipDelay PROCESSING > 1 days LEVEL ONE LEVEL TWO LEVEL THREE Then Event Tracking Situation Impact & Refinement Detection & Assessment contactLegal Refinement & Predictive Analysis Becomes If product.ShipDelay

> contract.SLA DB MANAGEMENT MaxDelay - Historical Profiles & AllowedShipLag LEVEL FOUR Data Patterns Then Process Refinement & warnLegal Adaptation If contract.customer.status = Hi Updated by Then set AllowedShipLag to 2 days 56 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. 24 Issues workshop: DOC4RTES

Needs constraints Eg Cannot reduce discount to <0 or increase above>25 Can handle as “change events” and rules to test… Difficult to test May be based on statistical functions – implies complex test regimes (or test- specific rules) Complex to prove ROI / value End-user may not be able to source or validate the advanced rules Requires statistical function libraries / analytics

Uncertainty Scoring Generating rules Machine learning Predictive Analytics Reasoning + Ontologies Other types of rules Constraint Logic Programming

Simple technique to handle “variable” decisions Rules update a score Example: insurance scoring Typically handled in a special ruleset Object Condition Score (or decision table) Property Effect Good as a KPI in a scorecard Age <18 -10 Typically used with an aggregation rule Age 19 to 26 -15 Age 27 to 49 +5

Age 50 to 69 0 Age 70+ -5

Given a set of data, deduce data result classification patterns and hence rules Requires sophisticated A A A A 1 algorithms

A B A B 2

B A B A 3

other other other other 4

Analyze data to deduce segmentation breaks for tasks like customer classifications Eg: Which customers should be offered what interest rate to maximise profit? Typically using specialist data mining tools Exports decision tree, rules etc in varieties of PMML Overlaps with BI (eg custom reports on historic data) Analytical functions may also be mapped to a ruleset in CEP for real-time analytics

61 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Reasoning + Ontologies workshop: DOC4RTES “Semantic Event Processing” Use Semantic Web technologies to augment CEP Textual news etc analysis Use of deeper ontology relationships Example components OMG Ontology Definition Metamodel ODM joins W3C OWL to UML concept models OWL, RDF, RDFS for terminology, relationships Logic languages/rules to reason about truth over event types and metadata

Semantic Networking Event Monitoring Architecture workshop: DOC4RTES

SPARQL/Pellet/OWL + Probabilistic & Special Purpose Reasoning Pattern & + Description Logics Trend Reasoning Longer-Term Reasoning KB Trend Analysis, Contextual Learning

Notifications of Anomalies, Course Corrections, SPARQL/Pellet/OWL Instructions for Retraining & Recharacterization, event + Description Logics + Production Rule Actions, Queries Event Reasoning Reasoning KB Event Classification & Aggregation Sense events CEP Engine Assess conditions + Production Rules take action CEP + Production Rule reasoning Operational Decisions

• Features include:

event … event … event … • Production rule reasoning can • report up (situation assessment Event Source / Bus & current course of action) • report laterally (situation assessment, & sensed changes, etc.) • report down (initiating actions, querying) -- courtesy of Sandpiper Software • Adaptive capabilities are possible at all levels

Constraint rules for systems Constraint solver to find best values (eg optimize price) With response time as a system constraint! Goal-driven Uses: Maximizing value of inventory Scheduling the best routes for trucks Maximizing probability for SLA achievement

The End

Q & A

A. Appendices & Back-up Information

workshop: DOC4RTES

CEP Development

Business Event Process Model Editors

Business Model User Interface Repository Event Sources Event Processing Application / Agent

EP Control & Update Distributed

Event Event Store Engines Event

Bus Processing CEP Bus

Event Bus Event Event Processing components Control BusControl

CEP Test Event Event History Verification Validation Consumers Warehouse Tools Tools

CEP Monitor Event Event Event Monitor Visualization Analytics

Event Processing Technical Society EPTS www.ep-ts.com

Luckham’s web site complexevents.com

Various vendor blogs (reference from complexevents.com)