Apache Camel: a Framework for Enterprise Integration Patterns

Apache Camel: A Framework for Enterprise Integration Patterns

Mark Irish [email protected] Staff Software Developer IBM

April 27, 2021 POW3R Digital Agenda

● Enterprise Integration Patterns

– Enterprise Messaging

– Routes and Patterns ● Apache Camel

– What is it?

– Installing Apache Camel

– Integrating IBM i Resources

● No enterprise infrastructure is fully homogeneous:

● Applications often need to send information back and forth in a normalized way ● Before messaging systems were invented, it was difficult (or impossible) to coordinate applications written in:

– Different languages

– Existing on different platforms

● Most popular solution to heterogeneous systems integration is the Enterprise Service Bus (ESB) pattern ● An ESB provides coordination and federation between systems using a common runtime ● The ESB will handle payload and protocol normalization to facilitate communication ● Endpoints are loosely coupled, so that one service is unaware of the other services’ languages, frameworks, and platforms

● An Enterprise Service Bus should provide:

– Transport invocation: Protocols and data binding for sending information

– Data routing and transformation: message-routing patterns

– Platform mediation: adapters for language/platform mapping

– Messaging: message-oriented middleware patterns

– Orchestration: coordinating business processes

– Qualities of service: security, logging, transaction support

– Platform agnosticism: loose coupling of endpoints

– Data Validation: schema and/or canonical data validation

● 2003 book by Gregor Hohpe and Bobby Woolf ● Formalizes 65 patterns for enterprise integration where before there were no explicit patterns or definitions ● Organized in:

– Message Channels

– Message Construction

– Message Routing

– Message Transformation

– Messaging Endpoints

– Systems Management

● Q: How do you inspect messages that travel on a point-to-point channel? ● A: Insert a simple Recipient List into the channel that publishes each incoming message to the main channel and a secondary channel

Images from: https://www.enterpriseintegrationpatterns.com/patterns/messaging/WireTap.html

● Q: How do you simplify dealing with a large message, when you are interested only in a few data items? ● A: Use a Content Filter to remove unimportant data tiems from a message leaving only important items.

Images from: https://www.enterpriseintegrationpatterns.com/patterns/messaging/ContentFilter.html

● producers: applications that provide data in the form of a message to the broker ● consumers: applications that receive data in the form of a message from the broker ● broker: messaging system for exchanging and marshaling data between processes, applications, servers, etc.

consumer producer broker

consumer

● A messaging pattern defines:

– How producers and consumers connect

– How producers and consumers communicate

– The network-oriented architecture implemented by the broker

consumer producer broker

consumer

● A broker has a lot of work to do to implement message patterns ● A broker has to know a lot of things about a lot of technologies to send messages correctly ● A broker sounds like a lot of work ● How am I ever going to make a broker?

consumer producer ?????

consumer

● Apache Camel is a mature integration framework that lives on your broker server ● Camel defines integration patterns used by the broker to route messages ● With camel, you define routes that integrate all sorts of technologies ● “Components” can be downloaded that know the ins and outs of the technologies you want to glue together

consumer producer

consumer

● Apache Camel is written in Java, and programming routes is done in Java ● Because Java runs on IBM i, Apache Camel can run on IBM i ● Can also run Apache Camel on another server and talk remotely to IBM i

● Apache Camel uses a concept called an “exchange”, that is composed of:

– An in message

– A processor

– An out message

in processor out

● Exchanges can be chained together to form a Camel Route ● The out of the previous exchange becomes the in of the next exchange

processor processor inin procesprocessor ouout inin proces ouout inin proces ouout sor t sor t sor t

● Components enable out-of-the-box exchanges that you can configure for your own resources ● Just use the right exchange, point it to the right resource, and go!

● Components in Camel contain logic to talk to a particular language/framework/service ● 25 Core Components, 300+ Components

– List at https://camel.apache.org/components/3.4.x/index.html

● IBM i has its own component that wraps around JT400 (Java toolkit for interacting with IBM i resources) ● Has exchanges for:

– Writing to a data queue (DTAQ)

– Reading from a data queue

– Writing to a message queue (MSGQ)

– Reading from a message queue

– Calling a remote program

● Get data from IBM i, log it, then send it to Kafka

– Exchange 1 gets data from an IBM i message queue

– Exchange 2 takes the data, logs it, then passes it on

– Exchange 3 takes the data, sends it through kafka to another system

in procescamel- out in procescamel-log out in procescamel- out in jt400 ou in ou in kafka ou sor t sor t sor t

● Apache Camel can be run on IBM i or on another system ● Where you run it will depend on the topography of your enterprise IT infrastructure:

– Are you IBM i-centric? Running on IBM i would be a good choice

– Is IBM i one of many types of servers/technologies? Running on another system might be better ● This demo will focus on running on IBM i

● Using yum (the open-source package manager), install OpenJDK:

yum install openjdk-11

● OR use system Java, ensuring you have the appropriate JV1 option installed:

– https://www.ibm.com/support/pages/node/1117869

● Set up the environment variables:

JAVA_HOME=/QOpenSys/pkgs/lib/jvm/openjdk-11 export JAVA_HOME PATH=$JAVA_HOME/bin:$PATH export PATH JAVA_TOOL_OPTIONS="-Djava.net.preferIPv4Stack=true -Djava.awt.headless=true" export JAVA_TOOL_OPTIONS

● Install maven and ca-certificates-mozilla

yum install maven ca-certificates-mozilla

● POM is a Project Object Model, which is an XML file that contains info about our project ● Maven uses this file to download the correct dependencies ● Minimal POM requires:

– project

– modelVersion

– groupId

– artifactId

– version

4.0.0

com.mycompany.app test-app 1

● In our POM, we also need to declare the dependencies of our application, namely:

– camel-parent

– camel-core

– camel-main

– camel-mail

– camel-jt400

● Running mvn compile from our directory will pull down Apache Camel, allowing us to use it in our program. ● The code to get Apache Camel running is a bit verbose, and would be more suited to a workshop or a lab...

● Send or receive from data queue:

– jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/QUEUE.DTAQ ● Send or receive from message queue:

– jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/QUEUE.MSGQ ● Call a remote program

– jt400://user:password@system/QSYS.LIB/LIBRARY.LIB/program.PGM ● For any URI, options can be appended to the end with:

– ?option=value&option=value&...

● Conversion of data to JSON before going to a message queue can help structure data to be more easily consumed by certain consumers:

● Send data to data queues (and therefore Apache Camel) can be done directly from the database:

● It can be difficult to integrate different parts of your enterprise infrastructure ● Enterprise Integration Patterns (EIPs) were developed to codify best practices for passing and manipulating data between systems ● Apache Camel implements EIPs and defines Components that you can use out of the box to connect different technologies/frameworks/systems:

– IBM i

– Email

– Kafka

– hundreds more!

● IBM i Apache Camel examples:

– https://github.com/IBM/ibmi-oss-examples/tree/master/camel

● Apache Camel documentation:

– https://camel.apache.org/docs/

● Apache Camel JT400 documentation:

– https://camel.apache.org/components/latest/jt400-component.html