Enterprise Integration Patterns and Solutions for Architects Introduction

"Web Age Speaks!" Webinar Series

!Brad8Brown "Managing8Director,8Red8Shift8Inc. "Web8Age8consultant8and8instructor "[email protected] !Web8Age8Solutions "Providing8a8broad8spectrum8of8regular8and8 customized8training8classes8in8 programming,8systems8administration,8and8 architecture8to8our8clients8across8the8world8 for8over8ten8years

©WebAgeSolutions.com 2 Overview of Talk

Where and why SOA failed Some SOA and Enterprise Application integration patterns ESB and Service-Oriented integration value proposition API Management Cloud Services Integration

©WebAgeSolutions.com 3 Enterprise Integration Patterns and Solutions for Architects

SOA Failed? Where and Why? Where and Why SOA Failed

 Forgetting that SOA is an Architecture Evolution, not Revolution  Many SOA projects failed due to underestimating its technical complexity  Attempts to SOA-ize everything based on the perceived simplicity of the technology aspect  For many architects and designers SOA boils down to formula: SOA == SOAP Web Services (everywhere)  In many cases development was challenged with over- engineered solutions based on UDDI, Advanced WS-* (-ReliableMessaging, etc.) specs , etc.  Lack of technical leadership

©WebAgeSolutions.com 5 SOA is Not a Silver Bullet SOA is not a one-size-fits-all solution for Enterprise Integration Problems Depending upon the technology, not every aspect of the enterprise is a good candidate for service orientation, for example: High-performance/critical real-time systems B2B interactions involving intricate transaction semantics Batch processes where vast amounts of data is moved, such as, data warehouses. Systems or business units where tight coupling and/or monolithic integration is required.

©WebAgeSolutions.com 6 Success When Done Right

Source: Sastry Malladi, Distinguished Architect. eBay, Inc.

©WebAgeSolutions.com 7 The Modern SOA Infrastructure

Source: Dr Mark Little. CTO JBoss, a division of Red Hat

©WebAgeSolutions.com 8 Pragmatic SOA Recommendations

Stay unswayed by buzzwords, marketing hype, and technology affinity In practice, many WS-* spec implementations are not fully interoperable, some are not used at all (WS-Coordination, etc.) Start moving towards RESTful HTTP for web services channel Do not try to over-engineer your solutions (KI$$) Try out open source ESB/SOI solutions (more on that later …)

©WebAgeSolutions.com 9 The Time-tested Rules

Data is King Data outlives applications Applications outlive integrations

©WebAgeSolutions.com 10 Enterprise Integration Patterns and Solutions for Architects

Enterprise Integration Patterns and Solutions A Pattern Definition

A design pattern is a proven design solution for a common design problem that is formally documented in a consistent manner “Fight-or-Flight” is a behavioural pattern

©WebAgeSolutions.com 12

SOA Data Service Patterns

Executing SOA: A Practical Guide for the Service -Oriented Architect

By Norbert Bieberstein;

Robert G. Laird; Dr. Keith

Jones; Tilak Mitra

Publisher: IBM Press  Contains SOA patterns related to data services

©WebAgeSolutions.com 13 A Not So Hypothetical Problem

Data resides in multiple sources and must be integrated Mergers and acquisitions along with legacy data system integration are two common causes of the problem Data replication or copying is not desirable Support must be provided for SOA and non- SOA applications

What is needed: A unified, virtual view of multiple diverse data sources

©WebAgeSolutions.com 14 Data Federation Pattern Solution

 Define a Data Federation Server (DFS) that  Acts as a real-time, synchronous data integrator  Performs query partitioning and rewrite, applying sub- queries to the target data sources  Provides both standard relational APIs (for example, SQL) and service-oriented interfaces (for example, WSDL)  Acts as a service provider (of integrated information services) and consumer (of multiple, dispersed data sources)  Provides additional QoS features such as data synchronization, transactional support, access control and management, performance optimization

©WebAgeSolutions.com 15 Data Federation Pattern Solution

©WebAgeSolutions.com 16 Data Federation Pattern: Considerations Single point of failure Keep in mind that if the DFS and data sources are unavailable then an integrated query will fail Transactions Concurrent requests from clients may generate and forward concurrent sub-queries to the same data source Performance will depend on how this data source handles concurrency. For example, its data isolation policy and transactional capabilities High availability This pattern is not well-suited to high-availability scenarios due to its dependence on multiple, real- time access to data sources ©WebAgeSolutions.com 17 SOA Patterns

SOA Design Patterns

By Thomas Erl This book was written with one primary goal in mind: to provide a master pattern catalog and pattern language for SOA and service- orientation.

©WebAgeSolutions.com 18 SOA Pattern Inventory Fragment

Pattern Name Summary Asynchronous Queuing A service can exchange messages with its consumers via an intermediary buffer, allowing service and consumers to process messages independently by remaining temporally decoupled. Canonical Schema Data models for common information sets are standardized across service contracts within an inventory boundary. Canonical Protocol The architecture establishes a single communications technology as the sole or primary medium by which services can interact. Data Format Transformation Intermediary data format transformation logic needs to be introduced in order to dynamically translate one data format into another.

Decoupled Contract The service contract is physically decoupled from its implementation. Event-Driven Messaging The consumer establishes itself as a subscriber of the service. The service, in turn, automatically issues notifications of relevant events to this and any of its subscribers.

©WebAgeSolutions.com 19 Decoupled Contract Pattern

©WebAgeSolutions.com 20 Decoupled Contract Pattern: Dé jà vu

©WebAgeSolutions.com 21 Enterprise Application Integration (EAI) Patterns

Enterprise Integration Patterns

By Hohpe & Woolf, 2003 This book identifies and defines 65 integration patterns, organized into categories

©WebAgeSolutions.com 22 EAI Patterns Summary

©WebAgeSolutions.com 23 Enterprise Integration Patterns and Solutions for Architects

ESB and SOI What is an ESB?

Enterprise Service Bus (ESB) is an Architectural Pattern From the SOA perspective, ESB can be used as an integration platform that enables existing IT assets and applications to be exposed as services Important part of SOI (Service-Oriented Infrastructure)

©WebAgeSolutions.com 25 ESB Distilled

The Service Provider is completely decoupled from the Consumer ESB transparently connects consumer(s) with provider(s) ESB can introduce such enterprise services as security, audit, throttling, HA, etc., protecting the Service Provider

©WebAgeSolutions.com 26 ESB Core Functionality  Message routing (message header/content based; rules- based)  Location transparency (service consumer is decoupled from provider)  Transport protocol mediation (HTTP, JMS, TCP, File, etc.)  Connectivity and interoperability (a/synchronous; p2p, pub/sub)  Message mediation (via adapters, protocol transformation and service mapping)  QoS (security: authentication authorization, encryption; audit; reliability, extensibility; HA; transactional support; etc.)  Message manipulation (transformation from one format into another) and enhancement  Monitoring & Management

©WebAgeSolutions.com 27 ESB and SOI

 ESB is the back-bone of SOI (Service-oriented Infrastructure)

Source: Dr Mark Little. CTO JBoss, a division of Red Hat

©WebAgeSolutions.com 28 ESB Implement EAI Patterns!

 ESB vendors, de facto, strive to implement most of EAI patterns  Mule ESB Studio Data Integration Primitives:

©WebAgeSolutions.com 29 ESB vs. EAI

 Most ESB vendors in the past were also (surprise!) enterprise application integration (EAI) solution providers (IBM WebSphere Message Broker, TIBCO Business Works, and Sonic XQ))  Two main differences between ESB and EAI:  ESB implements the bus-based architecture while EI solutions use hub-and-spoke architecture  For the most part, EAI solutions used proprietary technologies and data formats. ESB products use open standards for communication and data exchange (WSDL, XSD, XML, SOAP, REST, JMS, JEE Connector Architecture, etc.)  JBossESB positions itself as the next generation of EAI (without vendor lock-in)

©WebAgeSolutions.com 30 Open Source ESBs

 JBoss SOA-P  Mule

 WSO2  Apache ServiceMix and Camel  OpenESB  Petals

©WebAgeSolutions.com 31 Enterprise Integration Patterns and Solutions for Architects

API Management APIs Proliferation

Netflix recommends movies to its clients based on the history of previously watched movies (done through its internal APIs) Amazon makes product recommendations based on user’s browsing history (also done through an API)

©WebAgeSolutions.com 33 API Management Defined  API Management is an integration solution that helps expose internal system APIs for external/public consumption in a secure and controlled fashion as a set of Web APIs (REST or SOAP services). Normally includes: Mediation Engine (Service Gateway) • QoS enforcement • Security • Threat and Trust protection APIs meta information store • API authoring (publish/update/promote) • Searchable  It help you build, execute, monitor and monetize your APIs

We need a solution to answer these questions

 Does any of those APIs enforce secure access?  Do they provide protection against malicious attacks?  Are they scalable?  How is versioning handled?  Who owns it and where can I find the Service meta information  More questions to follow …

API life cycle management (publishing, versioning) Store for APIs and related artifacts Facilitates APIs discovery and use Promotes system integration and service orchestration Security / Access Control API access key management and distribution Runtime activity / SLA monitoring Analytics / Reporting

 Grow Business  Make yourself more easily discoverable by potential clients (which UDDI failed to deliver)  Internal Services can generate income if exposed for public consumption  Harness ubiquitous communication (mobility, cloud)  Reduce cost  Offload security, threat protection, etc. from back-end systems  Find cost-effective way to distribute digital assets  Innovate (your boss will love it)  E.g. with Web services mash-ups for greater user experience

Driving Forces

50% of the cost of new application development will be integration 75% of the Fortune 500 will have Web APIs by 2014

Source: Gartner Technology Research

 Mashery  The Inventor of API Management  Leader in the space (Gartner Research)  Strong partnership with Intel  Multi-tenant SaaS (Software as a Service) solution  APIs are managed like products (not just like technical integration points)

 Intel SOA Expressway (Service Gateway) software Appliance  Direct competition: IBM DataPower SOA Appliance

Mashery API Management Platform

Community API Portal Branded service catalog for API meta data & docs, forums/blogs API Management Publishing APIs as products suitable for access by internal or external developer communities with full lifecycle versioning and API management API Security Threat protection, OAuth, identity token mapping, data loss prevention, and API key access against enterprise identity infrastructure

©WebAgeSolutions.com 42 Service Gateway Functions  Policy Enforcement Point:  High performance mediation (between different services using mediation engine optimized for Intel multi-core)  Quality of Service Enforcement  Threat and Trust protection • Query Parameter Checks • SQL / Script Injection • Rate Limiting, DoS throttling • Raw XML security • HTTP Basic Authentication • Credential Mediation • LDAP / AD / Database Integration  Custom encryption is a drop-in module  Can take on security concern from back-end systems

Cloud Services Integration Cloud Services

Our Offerings Our Offerings

 WA2233 Enterprise Integration Patterns and Solutions for Architects

In this webinar, we reviewed: Where and why SOA failed Some SOA, Data and Enterprise Application integration patterns ESB and SOI integration value proposition API Management systems Cloud services integration