Complex Event Processing Tutorial Paul Vincent TIBCO Software Inc. Presenter workshop: DOC4RTES Presenter: Paul Vincent, CTO Business Rules and CEP, TIBCO Software Member OMG PRR and W3C RIF rules standards bodies Co-author CEP Blog http://tibcoblogs.com/cep TIBCO Software Inc.: Provides enterprise software that helps companies achieve service-oriented architecture (SOA) and business process management (BPM) success Headquartered in Palo Alto, California Over 3,000 customers and offices in 40 countries CEP product is TIBCO BusinessEvents Developed from a customer solution and launched 2005 Currently at Release 3.0 2 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Agenda workshop: DOC4RTES Introducing CEP 3 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Real-world Events workshop: DOC4RTES Customer New Customer Checks Order Logon Fed “Close Account” Base Rate Web Page Increase Contract Mobile Call Submitted Production from CT New Item @11.13 Liability Arrives at Contract Added Store Returned Rental Rental thru EDI Car Car Returned Crashed 4 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Where is the information? workshop: DOC4RTES Customer New Customer Checks Order Logon Fed Close Account Base Rate Web Page Increase Contract Mobile Call Submitted Production from CT New Item @11.13 Liability Arrives at Contract Added Store Returned Rental Rental thru EDI Car Car Returned Crashed 5 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Where is the information? workshop: DOC4RTES Change in Fraud Customer Product Sales Risk! CrossSell Trend Risk of Opportunity Customer Defection Contract Cell phone Validated Employee Compliance fraud alert Over hours Limit Approached Contract Customer Valid Rental now rated Contract Gold Complete 6 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Conventional Event Processing workshop: DOC4RTES New (LotsNew of Order Synchronous Events)Order Events Simple event processing EventEvent----atatatat----aaaa----timetime Aggregation as data IT Services: Business Processes: SOA & traditional Persistence BPM & Workflow Processes Data Processing Services 7 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Simple EP = default IT Model, 1950-now workshop: DOC4RTES Based on “human workflow”: one thing at a time Processes handle cases 1 at a time office clerk Use database and refer to it where necessary card index Provide some service flexibility with middleware internal mail Use BPM to document / manage / automate processes Use SOA to distribute / manage / automate services This model does NOT exploit ALL the information / data / events ALL the time Behaviour (and business logic) is silo’d T h e r e i s a b e t t e r w a y ! 8 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Complex Event Processing workshop: DOC4RTES New High performance (LotsNew of Order Events)Order patternpattern----matchingmatching Asynchronous via rules, states, Events & queries continuous Complex Patterns Event Event of Events Store Processing High performance persistence IT Services: Business Processes: SOA & traditional Persistence BPM & Workflow Processes Data Processing Services 9 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. CEP Terminology workshop: DOC4RTES Event Cloud Event Streams CEP (technology) applies pattern detection & filtering to the event clouds & streams and their histories Multiple modelling / execution paradigms are available for pattern detection CEP 10 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. What does CEP cover? workshop: DOC4RTES “CEP applies to a very broad spectrum of challenges in information systems. A short list includes:” Business process automation Computer systems to automate scheduling and control network-based processes and processing Identifying when complex contracts are fulfilled Detection intrusion, fraud and other network attacks C3I The Power of Events , Addison Wesley, ISBN: 0-201-72789-7, 2002 11 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. What does CEP Solve? workshop: DOC4RTES Usually: warnings precede threats CEP raison ddd’d’’’etreetreetre:: quickly respond to Value of action of Value detected variances the “““ Latency Problem “““ 12 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. What CEP provides workshop: DOC4RTES “Situational Awareness” “Sense and Respond” “Track and Trace” 13 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Agenda workshop: DOC4RTES History 14 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Command and Control workshop: DOC4RTES Command and Control Correlate all available information Determine tactics based on strategy and up-to-date information -- from RAF Battle of Britain Fighter Control System 1940 http://www.raf.mod.uk 15 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Data Fusion workshop: DOC4RTES EVENT SOURCES Data Fusion Domain EXTERNAL Level 0 Level 1 Level 2 Level 3 DISTRIBUTED Processing Processing Processing Processing LOCAL Sub-Object Object Situation Impact Assessment Assessment Assessment Assessment Sensors Documents Human/ Computer . People. Interface .. Database Management . Level 4 System Data Sources Processing Support Fusion Process Database Database Refinement -- Revised JDL data fusion model, 1998 Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001 16 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. 24 Condition Based Maintenance workshop: DOC4RTES Monitored System Well informed Decision Fusion Reasoning Classification Feature Extraction Feature History Mission -- from “Data Fusion for Developing Predictive Diagnostics for Electromechanical Systems” Steinberg, A., & Bowman, C., Handbook of Multisensor Data Fusion, CRC Press, 2001 17 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Agenda workshop: DOC4RTES Events and CEP 18 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Complex Business Problems workshop: DOC4RTES Fraud / Theft Relevant event of interest Thousands-to-millions of high-value small-size product items or transactions Product location How do you identify known patterns of “suspicious” change behavior? Logistics / Scheduling Raw material, production & delivery scheduling and resources are complex and prone to change Resource, requirement How do we reallocate resources to handle business and change production changes? Activity Monitoring Complex production and supply process with multiple actors How to measure and action Key Performance Indicators? Process data change 19 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Associated Events workshop: DOC4RTES Positive Events Product item X arrives at Production station S from Store T Production worker Y arrives at Production station S Production contract for item Z by time T is posted Negative Events Product item X has been in transit to Store T for >15 minutes Subcomponent Y hasn’t arrived at the Production station by the ETA Delivery of contract Z has not taken place Sets of Events 5+ items of Product item type Y failed to arrive at destination Supplier Y was 5 mins late for 1 delivery, but made it early to the next Return rate on component Z exceeds SLA % 20 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Significant features of these Events workshop: DOC4RTES Time Sensitivity A thief may leave the building at the same time as stolen product 15:39:26.413 A product should take 40 minutes to travel a given production line segment Dec 24 Distributed Event Sources A series of produced items fails at various QA stages, and their common attribute was a storage location Multiple suppliers for a subcomponent are reporting delivery delays ERP Supplier 21 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. What *is* an “event”? workshop: DOC4RTES Change of state in some entity “Happening” Customer call Bank debit Aircraft movement Observation of some entity Observation CRM record of a customer call ATM report of debit transaction success Radar plot update of an aircraft IT Message Queued point-to-point message IT Message Publish / subscribe message 22 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Sample Event Metamodel workshop: DOC4RTES General Event Channel Inherits from Timestamp Default Destination Destination TimeToLive Simple Event ExpiryAction Event contains data + Custom Attributes timestamp Various classifiers (int/ext, transport, lifecycle, …) XML 23 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Event-driven vs Event Processing workshop: DOC4RTES Complex Inform / multiple events Event Event Processing Sources Event Processing Agents Simple Invoke / single event Event Processing Event Consumers Business Process Event Store Event Management Bus Event Bus Event Services Data Services 24 © 2008 TIBCO Software Inc. All Rights Reserved. Confidential and Proprietary. Event Driven Architecture workshop: DOC4RTES S O A E D A Request ---Reply TransmitTransmit----ListenListen Synchronous Asynchronous Events Events Assumptions… 25 © 2008 TIBCO Software