Apache Camel: A Framework for Enterprise Integration Patterns Mark Irish [email protected] Staff Software Developer IBM April 27, 2021 POW3R Digital Agenda ● Enterprise Integration Patterns – Enterprise Messaging – Routes and Patterns ● Apache Camel – What is it? – Installing Apache Camel – Integrating IBM i Resources © 2021 IBM Corporation Enterprise Integration Patterns © 2021 IBM Corporation The Problem: ● No enterprise infrastructure is fully homogeneous: © 2021 IBM Corporation Traditional Enterprise Messaging ● Applications often need to send information back and forth in a normalized way ● Before messaging systems were invented, it was difficult (or impossible) to coordinate applications written in: – Different languages – Existing on different platforms © 2021 IBM Corporation Enterprise Service Bus ● Most popular solution to heterogeneous systems integration is the Enterprise Service Bus (ESB) pattern ● An ESB provides coordination and federation between systems using a common runtime ● The ESB will handle payload and protocol normalization to facilitate communication ● Endpoints are loosely coupled, so that one service is unaware of the other services’ languages, frameworks, and platforms © 2021 IBM Corporation Enterprise Service Bus Capabilities ● An Enterprise Service Bus should provide: – Transport invocation: Protocols and data binding for sending information – Data routing and transformation: message-routing patterns – Platform mediation: adapters for language/platform mapping – Messaging: message-oriented middleware patterns – Orchestration: coordinating business processes – Qualities of service: security, logging, transaction support – Platform agnosticism: loose coupling of endpoints – Data Validation: schema and/or canonical data validation © 2021 IBM Corporation Enterprise Service Bus Capabilities © 2021 IBM Corporation Examples of Enterprise Service Buses © 2021 IBM Corporation Enterprise Integration Patterns ● 2003 book by Gregor Hohpe and Bobby Woolf ● Formalizes 65 patterns for enterprise integration where before there were no explicit patterns or definitions ● Organized in: – Message Channels – Message Construction – Message Routing – Message Transformation – Messaging Endpoints – Systems Management © 2021 IBM Corporation Enterprise Integration Patterns © 2021 IBM Corporation EIP Example: Wire Tap ● Q: How do you inspect messages that travel on a point-to-point channel? ● A: Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel Images from: https://www.enterpriseintegrationpatterns.com/patterns/messaging/WireTap.html © 2021 IBM Corporation EIP Example: Content Filter ● Q: How do you simplify dealing with a large message, when you are interested only in a few data items? ● A: Use a Content Filter to remove unimportant data tiems from a message leaving only important items. Images from: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ContentFilter.html © 2021 IBM Corporation Defining some terms ● producers: applications that provide data in the form of a message to the broker ● consumers: applications that receive data in the form of a message from the broker ● broker: messaging system for exchanging and marshaling data between processes, applications, servers, etc. consumer producer broker consumer © 2021 IBM Corporation Messaging Patterns ● A messaging pattern defines: – How producers and consumers connect – How producers and consumers communicate – The network-oriented architecture implemented by the broker consumer producer broker consumer © 2021 IBM Corporation How do I make a broker? ● A broker has a lot of work to do to implement message patterns ● A broker has to know a lot of things about a lot of technologies to send messages correctly ● A broker sounds like a lot of work ● How am I ever going to make a broker? consumer producer ????? consumer © 2021 IBM Corporation Apache Camel © 2021 IBM Corporation Apache Camel ● Apache Camel is a mature integration framework that lives on your broker server ● Camel defines integration patterns used by the broker to route messages ● With camel, you define routes that integrate all sorts of technologies ● “Components” can be downloaded that know the ins and outs of the technologies you want to glue together consumer producer consumer © 2021 IBM Corporation Apache Camel ● Apache Camel is written in Java, and programming routes is done in Java ● Because Java runs on IBM i, Apache Camel can run on IBM i ● Can also run Apache Camel on another server and talk remotely to IBM i © 2021 IBM Corporation Camel Exchanges ● Apache Camel uses a concept called an “exchange”, that is composed of: – An in message – A processor – An out message in processor out © 2021 IBM Corporation Camel Routes ● Exchanges can be chained together to form a Camel Route ● The out of the previous exchange becomes the in of the next exchange processor processor inin procesprocessor ouout inin proces ouout inin proces ouout sor t sor t sor t © 2021 IBM Corporation Exchanges and Components ● Components enable out-of-the-box exchanges that you can configure for your own resources ● Just use the right exchange, point it to the right resource, and go! © 2021 IBM Corporation Camel Components ● Components in Camel contain logic to talk to a particular language/framework/service ● 25 Core Components, 300+ Components – List at https://camel.apache.org/components/3.4.x/index.html © 2021 IBM Corporation JT400 Component (camel-jt400) ● IBM i has its own component that wraps around JT400 (Java toolkit for interacting with IBM i resources) ● Has exchanges for: – Writing to a data queue (DTAQ) – Reading from a data queue – Writing to a message queue (MSGQ) – Reading from a message queue – Calling a remote program © 2021 IBM Corporation Camel Route Example ● Get data from IBM i, log it, then send it to Kafka – Exchange 1 gets data from an IBM i message queue – Exchange 2 takes the data, logs it, then passes it on – Exchange 3 takes the data, sends it through kafka to another system in procescamel- out in procescamel-log out in procescamel- out in jt400 ou in ou in kafka ou sor t sor t sor t © 2021 IBM Corporation Apache Camel and IBM i: How To © 2021 IBM Corporation Where to Run Apache Camel? ● Apache Camel can be run on IBM i or on another system ● Where you run it will depend on the topography of your enterprise IT infrastructure: – Are you IBM i-centric? Running on IBM i would be a good choice – Is IBM i one of many types of servers/technologies? Running on another system might be better ● This demo will focus on running on IBM i © 2021 IBM Corporation Installing ● Using yum (the open-source package manager), install OpenJDK: yum install openjdk-11 ● OR use system Java, ensuring you have the appropriate JV1 option installed: – https://www.ibm.com/support/pages/node/1117869 © 2021 IBM Corporation Installing ● Set up the environment variables: JAVA_HOME=/QOpenSys/pkgs/lib/jvm/openjdk-11 export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH JAVA_TOOL_OPTIONS="-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true" export JAVA_TOOL_OPTIONS © 2021 IBM Corporation Installing ● Install maven and ca-certificates-mozilla yum install maven ca-certificates-mozilla © 2021 IBM Corporation Configure pom.xml for Maven ● POM is a Project Object Model, which is an XML file that contains info about our project ● Maven uses this file to download the correct dependencies ● Minimal POM requires: – project – modelVersion – groupId – artifactId – version © 2021 IBM Corporation Configure pom.xml for Maven <project> <modelVersion>4.0.0</modelVersion> <groupId>com.mycompany.app</groupId> <artifactId>test-app</artifactId> <version>1</version> </project> © 2021 IBM Corporation Configure pom.xml for Maven ● In our POM, we also need to declare the dependencies of our application, namely: – camel-parent – camel-core – camel-main – camel-mail – camel-jt400 © 2021 IBM Corporation Pull in Camel © 2021 IBM Corporation Run the code ● Running mvn compile from our directory will pull down Apache Camel, allowing us to use it in our program. ● The code to get Apache Camel running is a bit verbose, and would be more suited to a workshop or a lab... © 2021 IBM Corporation Run the code © 2021 IBM Corporation Run the code © 2021 IBM Corporation Example Exchange URIs ● Send or receive from data queue: – jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/QUEUE.DTAQ ● Send or receive from message queue: – jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/QUEUE.MSGQ ● Call a remote program – jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/program.PGM ● For any URI, options can be appended to the end with: – ?option=value&option=value&... © 2021 IBM Corporation Db2 Enhancements for Apache Camel ● Conversion of data to JSON before going to a message queue can help structure data to be more easily consumed by certain consumers: ● Send data to data queues (and therefore Apache Camel) can be done directly from the database: © 2021 IBM Corporation Conclusions © 2021 IBM Corporation Conclusions ● It can be difficult to integrate different parts of your enterprise infrastructure ● Enterprise Integration Patterns (EIPs) were developed to codify best practices for passing and manipulating data between systems ● Apache Camel implements EIPs and defines Components that you can use out of the box to connect different technologies/frameworks/systems: – IBM i – Email – Kafka – hundreds more! © 2021 IBM Corporation Resources ● IBM i Apache Camel examples: – https://github.com/IBM/ibmi-oss-examples/tree/master/camel ● Apache Camel documentation: – https://camel.apache.org/docs/ ● Apache Camel JT400 documentation: – https://camel.apache.org/components/latest/jt400-component.html © 2021 IBM Corporation.