"Web Age Speaks!" Webinar Series Enterprise Integration Patterns and Solutions for Architects Introduction !Brad8Brown "Managing8Director,8Red8Shift8Inc. "Web8Age8consultant8and8instructor "[email protected] !Web8Age8Solutions "Providing8a8broad8spectrum8of8regular8and8 customized8training8classes8in8 programming,8systems8administration,8and8 architecture8to8our8clients8across8the8world8 for8over8ten8years ©WebAgeSolutions.com 2 Overview of Talk Where and why SOA failed Some SOA and Enterprise Application integration patterns ESB and Service-Oriented integration value proposition API Management Cloud Services Integration ©WebAgeSolutions.com 3 Enterprise Integration Patterns and Solutions for Architects SOA Failed? Where and Why? Where and Why SOA Failed Forgetting that SOA is an Architecture Evolution, not Revolution Many SOA projects failed due to underestimating its technical complexity Attempts to SOA-ize everything based on the perceived simplicity of the technology aspect For many architects and designers SOA boils down to formula: SOA == SOAP Web Services (everywhere) In many cases development was challenged with over- engineered solutions based on UDDI, Advanced WS-* (-ReliableMessaging, etc.) specs , etc. Lack of technical leadership ©WebAgeSolutions.com 5 SOA is Not a Silver Bullet SOA is not a one-size-fits-all solution for Enterprise Integration Problems Depending upon the technology, not every aspect of the enterprise is a good candidate for service orientation, for example: High-performance/critical real-time systems B2B interactions involving intricate transaction semantics Batch processes where vast amounts of data is moved, such as, data warehouses. Systems or business units where tight coupling and/or monolithic integration is required. ©WebAgeSolutions.com 6 Success When Done Right Source: Sastry Malladi, Distinguished Architect. eBay, Inc. ©WebAgeSolutions.com 7 The Modern SOA Infrastructure Source: Dr Mark Little. CTO JBoss, a division of Red Hat ©WebAgeSolutions.com 8 Pragmatic SOA Recommendations Stay unswayed by buzzwords, marketing hype, and technology affinity In practice, many WS-* spec implementations are not fully interoperable, some are not used at all (WS-Coordination, etc.) Start moving towards RESTful HTTP for web services channel Do not try to over-engineer your solutions (KI$$) Try out open source ESB/SOI solutions (more on that later …) ©WebAgeSolutions.com 9 The Time-tested Rules Data is King Data outlives applications Applications outlive integrations ©WebAgeSolutions.com 10 Enterprise Integration Patterns and Solutions for Architects Enterprise Integration Patterns and Solutions A Pattern Definition A design pattern is a proven design solution for a common design problem that is formally documented in a consistent manner “Fight-or-Flight” is a behavioural pattern ©WebAgeSolutions.com 12 SOA Data Service Patterns Executing SOA: A Practical Guide for the Service -Oriented Architect By Norbert Bieberstein; Robert G. Laird; Dr. Keith Jones; Tilak Mitra Publisher: IBM Press Contains SOA patterns related to data services ©WebAgeSolutions.com 13 A Not So Hypothetical Problem Data resides in multiple sources and must be integrated Mergers and acquisitions along with legacy data system integration are two common causes of the problem Data replication or copying is not desirable Support must be provided for SOA and non- SOA applications What is needed: A unified, virtual view of multiple diverse data sources ©WebAgeSolutions.com 14 Data Federation Pattern Solution Define a Data Federation Server (DFS) that Acts as a real-time, synchronous data integrator Performs query partitioning and rewrite, applying sub- queries to the target data sources Provides both standard relational APIs (for example, SQL) and service-oriented interfaces (for example, WSDL) Acts as a service provider (of integrated information services) and consumer (of multiple, dispersed data sources) Provides additional QoS features such as data synchronization, transactional support, access control and management, performance optimization ©WebAgeSolutions.com 15 Data Federation Pattern Solution ©WebAgeSolutions.com 16 Data Federation Pattern: Considerations Single point of failure Keep in mind that if the DFS and data sources are unavailable then an integrated query will fail Transactions Concurrent requests from clients may generate and forward concurrent sub-queries to the same data source Performance will depend on how this data source handles concurrency. For example, its data isolation policy and transactional capabilities High availability This pattern is not well-suited to high-availability scenarios due to its dependence on multiple, real- time access to data sources ©WebAgeSolutions.com 17 SOA Patterns SOA Design Patterns By Thomas Erl This book was written with one primary goal in mind: to provide a master pattern catalog and pattern language for SOA and service- orientation. ©WebAgeSolutions.com 18 SOA Pattern Inventory Fragment Pattern Name Summary Asynchronous Queuing A service can exchange messages with its consumers via an intermediary buffer, allowing service and consumers to process messages independently by remaining temporally decoupled. Canonical Schema Data models for common information sets are standardized across service contracts within an inventory boundary. Canonical Protocol The architecture establishes a single communications technology as the sole or primary medium by which services can interact. Data Format Transformation Intermediary data format transformation logic needs to be introduced in order to dynamically translate one data format into another. Decoupled Contract The service contract is physically decoupled from its implementation. Event-Driven Messaging The consumer establishes itself as a subscriber of the service. The service, in turn, automatically issues notifications of relevant events to this and any of its subscribers. ©WebAgeSolutions.com 19 Decoupled Contract Pattern ©WebAgeSolutions.com 20 Decoupled Contract Pattern: Dé jà vu ©WebAgeSolutions.com 21 Enterprise Application Integration (EAI) Patterns Enterprise Integration Patterns By Hohpe & Woolf, 2003 This book identifies and defines 65 integration patterns, organized into categories ©WebAgeSolutions.com 22 EAI Patterns Summary ©WebAgeSolutions.com 23 Enterprise Integration Patterns and Solutions for Architects ESB and SOI What is an ESB? Enterprise Service Bus (ESB) is an Architectural Pattern From the SOA perspective, ESB can be used as an integration platform that enables existing IT assets and applications to be exposed as services Important part of SOI (Service-Oriented Infrastructure) ©WebAgeSolutions.com 25 ESB Distilled The Service Provider is completely decoupled from the Consumer ESB transparently connects consumer(s) with provider(s) ESB can introduce such enterprise services as security, audit, throttling, HA, etc., protecting the Service Provider ©WebAgeSolutions.com 26 ESB Core Functionality Message routing (message header/content based; rules- based) Location transparency (service consumer is decoupled from provider) Transport protocol mediation (HTTP, JMS, TCP, File, etc.) Connectivity and interoperability (a/synchronous; p2p, pub/sub) Message mediation (via adapters, protocol transformation and service mapping) QoS (security: authentication authorization, encryption; audit; reliability, extensibility; HA; transactional support; etc.) Message manipulation (transformation from one format into another) and enhancement Monitoring & Management ©WebAgeSolutions.com 27 ESB and SOI ESB is the back-bone of SOI (Service-oriented Infrastructure) Source: Dr Mark Little. CTO JBoss, a division of Red Hat ©WebAgeSolutions.com 28 ESB Implement EAI Patterns! ESB vendors, de facto, strive to implement most of EAI patterns Mule ESB Studio Data Integration Primitives: ©WebAgeSolutions.com 29 ESB vs. EAI Most ESB vendors in the past were also (surprise!) enterprise application integration (EAI) solution providers (IBM WebSphere Message Broker, TIBCO Business Works, and Sonic XQ)) Two main differences between ESB and EAI: ESB implements the bus-based architecture while EI solutions use hub-and-spoke architecture For the most part, EAI solutions used proprietary technologies and data formats. ESB products use open standards for communication and data exchange (WSDL, XSD, XML, SOAP, REST, JMS, JEE Connector Architecture, etc.) JBossESB positions itself as the next generation of EAI (without vendor lock-in) ©WebAgeSolutions.com 30 Open Source ESBs JBoss SOA-P Mule WSO2 Apache ServiceMix and Camel OpenESB Petals ©WebAgeSolutions.com 31 Enterprise Integration Patterns and Solutions for Architects API Management APIs Proliferation Netflix recommends movies to its clients based on the history of previously watched movies (done through its internal APIs) Amazon makes product recommendations based on user’s browsing history (also done through an API) ©WebAgeSolutions.com 33 API Management Defined API Management is an integration solution that helps expose internal system APIs for external/public consumption in a secure and controlled fashion as a set of Web APIs (REST or SOAP services). Normally includes: Mediation Engine (Service Gateway) • QoS enforcement • Security • Threat and Trust protection APIs meta information store • API authoring (publish/update/promote) • Searchable It help you build, execute, monitor and monetize your APIs ©WebAgeSolutions.com 34 We need a solution to answer these questions