Large-Scale Deployments with Servicemix 4 a Fusesource White Paper

WHITE PAPER

Large-Scale Deployments with Servicemix 4 A FuseSource White Paper

Andreas Gies, Principal Consultant, FuseSource March 2009

Large Scale Deployments with Servicemix 4

www.

TABLE OF CONTENTS fusesource

Introduction ...... 2

A Detailed Look at the Case ...... 4 .com Initial Scoping of the Project ...... 5

Software Installation and Configuration ...... 5 Functional Requirements ...... 5 Performance Requirements ...... 6 High-level Application Design ...... 6 The Shop Registry ...... 8 The Server Endpoints Component ...... 10 The Shop Server Router Component ...... 12 The Shop Endpoints Module ...... 14 Benefits of Using OSGi ...... 15 Possible Extensions ...... 15 Deployment Considerations ...... 16 FuseSource: Open Source for the Enterprise ...... 21

Large Scale Deployments with Servicemix 4

www. INTRODUCTION

fusesource One of the more difficult challenges of building an enterprise system is designing for scalability. It is often skipped in the proof-of-concept stage of development, and it requires product-specific knowledge

.com and experience. To help the development of extremely large deployments, the FuseSource team is sharing the experiences and results of a specific, real-world deployment at a large retailer with stores throughout Europe.

This company has over 2000 retail shops spread out across six European countries. This white paper is using the experience we gained in this project with this customer and suggests a blueprint for how to deploy large systems based on Apache ServiceMix . The code discussed here is available on the FuseSource wiki at http://fusesource.com/wiki/display/ProdInfo/Large+Scale+Deploym ent+Demo.

The company has a central ERP system (based on SAP) that serves as the master database for all product and pricing information and performs all disposition tasks required to optimize the delivery of goods to each retail outlet. Data collected by the shops is then used by the ERP system to perform the disposition tasks, and pricing decisions are disseminated to the shops.

The company was using a typical end-of-day batch file transfer pattern. Market data was transferred once a day to the data center, where batch jobs did all the central processing, and the new information for the markets was then transferred back in batch. With a growing number of shops and a growing palette of goods the company offered, soon there was not enough processing time at night to accommodate the growth of the company. In addition, the data was always stale.

Large Scale Deployments with Servicemix 4

www. The solution was to migrate to an architecture that could support

real-time delivery of the shop data in the data center for processing fusesource and real-time communication of pricing events to the shops. Given the huge number of shops and the limited number of IT personnel in

the individual countries, the development team followed two key .com principles:

. To limit the resources needed for development, the operations in each shop standardized from an organizational point of view to limit the number of different services needed in each shop. . To accommodate the uniqueness of each shop, services needed to be parameterized. The parameters would reflect the location and identity of each shop.

Proper design and implementation only brought the team half the way; the provisioning infrastructure was also key for a distributed project of this size. The provisioning infrastructure included:

. A single installation package to be used as an installation base for each shop . An installation package to determine the environment it is running in (e.g., the unique network address of the shop host or the machine name) and derive the service customization parameters from that unique key . Service parameters applied during installation so that the service instances are ,in fact, associated with a specific shop . A mechanism in the installation package to establish the network connectivity between the data center and the freshly installed shop

Large Scale Deployments with Servicemix 4

www. This white paper is using the experience we gained in this project

with this customer and suggests a blueprint for how to deploy large fusesource systems based on Apache ServiceMix. The FuseSource team has decades of experience building mission-critical, IT applications for

the Fortune 500 and employs many of the key committers and .com leaders at Apache.

A DETAILED LOOK AT THE USE CASE

The project consisted of integrating the systems in each of the individual retail outlets (shops) with the central ERP system at headquarters. In each shop, two applications are present: a point-of- sale (POS) system and a warehousing system (WH).

The POS system is the backend system for the cash desks. It needs accurate pricing information from the data center and provides data about the revenues and which goods have been sold to the data center. The system originally sent accumulated data at the end of day to the data center via a traditional batch-oriented file transfer. At some point during the night the POS server received pricing updates from the data center and forwarded them to the cash desks.

The WH system is a dispositioning system that generates and transfers the orders to the central ERP system (SAP). Any state changes for theses orders are communicated back to the WH server via batch-oriented file transfer.

Both the POS and WH systems had a file-based interface in each shop to share information. The SAP Business Connector (BC) collected the files to transmit to the central ERP system and dropped data files into the file system to be imported into the local POS and WH systems.

At headquarters, a mapping tool was used to transform all data going into or coming out of the central ERP system. For simplicity, assume that the mapping tool communicates through a JMS.

Large Scale Deployments with Servicemix 4

www. The existing installation was working, albeit with stale data, but

headquarters imposed a new requirement on the system that fusesource threatened to break the proverbial camel’s back: the data from the POS systems also had to be shared with a data warehouse

component to be analyzed for cost-efficiency, fraud detection of .com cash desk operators, etc.

INITIAL SCOPING OF THE PROJECT

The FuseSource team reviewed the existing infrastructure, future requirements, and the scale of operations and scoped the project into three separate, functional areas.

Software Installation and Configuration

The solution would be a distributed application with decoupled components communicating asynchronously through a JMS middleware component. This would replace the company’s batch file transfer with an event-driven architecture.

The components would be independent from each other and communicate using well-defined interfaces. Each shop would receive a deployment package that contains the necessary components to connect the shop to the data center, and the number of different deployment packages would be minimal. An optimal solution would deliver the same deployment package to each shop and only use different configuration files for the setup.

The components must run in different profiles such as development, QA and production and must be adaptable during deployment.

Functional Requirements

All data must be transferred without data loss.

The data must be transmitted in near real time despite low bandwidth networks. Required changes in the participating applications to move from batch processing to event-driven data processing are not in the scope of this document.

Large Scale Deployments with Servicemix 4

www. The design must be flexible and future-proofed to support future

requirements. fusesource

Performance Requirements

A normal market performs two-to-three cash operations a minute, .com With 300 markets online in phase 1 this translates to roughly 1000

messages per minute. In phase 2 with about 600 markets online the traffic goes up to 2500-3000 messages per minute. Phase 2 includes shops with more cash decks and, therefore, more cash operations per minute than the shops included in phase 1.

HIGH-LEVEL APPLICATION DESIGN

The retail scenario has two major functional components. The first component is run in the data center and contains the following modules:

. Server endpoints module—provides the communication interface to the central ERP system. This module collects the data transmitted from the central ERP system regardless of the underlying transport and creates a shop request document in a defined JMS queue. . Content-based routing module—is responsible for reading the shop request documents from the JMS queue and determining which shop receives the request. This module uses the shop registry module to facilitate the routing decisions. . Shop registry—provides a central registry of shops that are currently installed within the infrastructure. This information is currently used in the shop routing module, but could be enhanced to become a shop maintenance module to provide operational access to the shops at runtime.

Large Scale Deployments with Servicemix 4

www. . Shop registry interface —provides remote access to the shop

registry for registering and deregistering individual shops. fusesource The shops use this interface to register themselves when they first come online, allowing the infrastructure to grow as the

number of shops grows. New shops are automatically taken .com into account by the router as soon as they are registered.

(Note: In our mock application the remote interface is not defined, and it operates with a fixed list of shops. This will be addressed in the next version of the demo.)

The second component runs locally in the shops. This component contains the modules interfacing with the local shop applications. These modules listen on a local JMS queue and process the data from there.

Figure 1: High-level Application Design

Figure 1 shows the high-level design. Note that the central component and each of the shop components are hosted in its own instance of the Fuse ESB® as the components are geographically distributed. Fuse ESB is the productized and supported distribution of Apache ServiceMix from FuseSource.

Large Scale Deployments with Servicemix 4

www. The communication is realized by the Fuse Message Broker®

(productized and supported distribution of Apache ActiveMQ) that is fusesource included by default in a Fuse ESB setup. The installation includes a network of brokers, and the central hub has visibility into all of the

shops’ message brokers, but the shop broker has visibility only into .com the central message broker. To avoid having to change the central broker configuration each time a shop is added to the network, the shop’s message broker initiates the network connection.

The Shop Registry

The shop registry is implemented as a Spring-based OSGi service. First it defines a Shop interface: public interface Shop {

String getUniqueId(); String getCountry(); String getSystemType(); String getShopNr(); String getShopQueueName(); String getServerQueueName(); void setProperty(String propName, String propValue); Properties getProperties(); }

Each shop component is identified by a composite key:

. The country property denotes the country where the shop is located. . The systemType property denotes the local application the component supports. . The shopNr property is a unique identifier for the physical location of a shop within a country.

Using a composite key over a single value identifier makes it easy to group shops for common configurations. For example, all “country wide” or “application wide” shops could be treated similarly.

Large Scale Deployments with Servicemix 4

www. Note the uniqueId property. Imagine that a new shop component comes online for the first time and without knowledge of what the fusesource composite key should be. The component can determine a unique key that identifies where the location is hosted; in this application

the public network address of the shop node is used for this purpose .com because it is unique within a TCP/IP network. This is safe as long as only a single shop component is hosted on a physical node.

With a unique identifier, the shop registry can determine the composite key and any additional properties of importance from that key. This information is stored in a database or in a simple XML file in the data center and, therefore, can be maintained centrally. A sample XML fragment is shown below:

A ShopFactory is exposed as OSGi service and creates Shop objects upon request. This is reused by the Shop routing component and also the remote shop registry interface. public class ShopFactory {

public static Shop createShop(String country, String systemType, String shopNr) { return new ShopImpl(country, systemType, shopNr); }

// TODO: implement creation of shops based on the uniqueId and the xml mapping public static Shop createShop(String uniqueId) throws Exception {

Large Scale Deployments with Servicemix 4

www. throw new OperationNotSupportedException("This is not yet implemented"); fusesource } }

.com Finally, the shop registry maintains a list of shops that are currently

available. The registry is also exposed as an OSGi service and is therefore available to other components. public interface ShopRegistry {

void addShop(Shop s); void removeShop(Shop s); boolean containsShop(Shop s); void clear(); List getShops(); }

Note: the source tar ball is available for download in the fusesource.com wiki: http://fusesource.com/wiki/display/ProdInfo/Large+Scale+Deploym ent+Demo.

The Server Endpoints Component

The server endpoints module leverages Camel, a mediation router that is embedded in the Fuse ESB. This is a simple Camel route that collects data from the ERP system and places shop request documents into a queue. In this mock application the ERP system is replaced by the file system at the central node.

The important piece of code to set up the Camel route is: private String inputDir; private String queueName;

public void configure() throws Exception { from("file:" + inputDir) .convertBodyTo(String.class) .to("activemq:queue:" + queueName); }

Large Scale Deployments with Servicemix 4

www. Note that the input directory for picking up files and destination

name where the requests are placed is configurable via Spring. This fusesource is the first step to externalize these settings. Simple properties files or a provisioning infrastructure can provide external settings later.

The final steps are within the module’s Spring configuration files. The .com properties are referenced within the Spring configuration of the route builder class. Note that the property values are not hard coded, but given as standard Spring replacement tokens.

Then the replacement tokens are defined using a property placeholder:

/tmp/server/inbound ServerRequests

This means that the Fuse ESB looks for a file named com.progress.pso.samples.retail.cfg within its etc directory, which is a normal properties file. If the file contains a property value for one of the replacement tokens, that value is used; otherwise the default value given in the XML file is taken.

Large Scale Deployments with Servicemix 4

www. The files, which we can drop into the pickup directory for our

mockup application, have the following format: fusesource

De,POS,01234,Hello from Server Cz,WRH,12345,Test

.com

Each line represents a single message that is sent to a dedicated shop. Each line has four elements. The first three elements identify the target shop, and the last element specifies the payload sent to the shop. Obviously this is a very simple message format and would be more sophisticated in a real application. The important points are:

. The ERP data format is decoupled from the data format used in the shops so messages for an arbitrary number of shops can be sent within a single file. . Each shop message carries the information about its destination so that content-based routing can be applied properly.

This represents a flexible connection to the central systems that can be instantiated in the Fuse ESB as a normal OSGi bundle. Being a bundle, the module is maintained independently from other modules. It would also be very easy to replace the file system connectivity with something else— a direct JMS connection, for example. It is important to note that shop request documents are generated and placed in the queue for the shop routing module. The Shop Server Router Component

The shop router module is also implemented as a Camel route and is packaged as an OSGi bundle. This route is different than a normal content-based routing module in that the possible destinations are variable as shops register and deregister with the infrastructure.

Below is the Java code that defines the route: public void configure() throws Exception { from("activemq:queue:" + queueName)

.splitter(body(String.class).tokenize("\n"))

Large Scale Deployments with Servicemix 4

www. .process(new Processor() {

private Endpoint fusesource getShopEndpoint(Shop s) throws Exception { Endpoint result = null; String uri =

"activemq:queue:" + s.getShopQueueName(); .com try { result = getContext().getEndpoint(uri); } catch (NoSuchEndpointException nsee) { logger.debug("Trying to create Endpoint : " + uri); result = getContext().getComponent("activemq").createEndpoin t(uri); getContext().addEndpoint(uri, result); } return result; }

public void process(Exchange ex) throws Exception { String line = (String)ex.getIn().getBody(); String data[] = line.split(","); if (data.length != 4) { logger.warn("Ignoring malformatted line: " + line); return; } Shop s = ShopFactory.createShop(data[0], data[1], data[2]); logger.debug("Trying to direct message to shop: " + s);

if (shopRegistry.containsShop(s)) { Endpoint ep = getShopEndpoint(s); ProducerTemplate pt = getContext().createProducerTemplate(); pt.sendBody(ep, data[3]); } else {

Large Scale Deployments with Servicemix 4

www. logger.warn("Ignoring message for non-registered Shop: " + s); } fusesource } }); } .com

A Camel endpoint is used to consume shop request documents. The payload is transformed into a string, which is, in turn, split into individual lines. Each line is run through the processor-defined inline. The processor extracts the composite key for the shop and uses the shop registry to determine whether a registered shop is addressed. If so, it creates a message and sends it to the Camel endpoint associated with the shop. The getEndpoint method registers an endpoint if a newly registered shop is addressed for the first time.

This simple piece of code defines a content-based router with a flexible number of outgoing endpoints. It is important to note how the shop registry API is leveraged to get the correct name for the queue at the remote broker.

The Shop Endpoints Module

The shop endpoints module is a simple Camel route that is deployed in each shop instance. The module registers a listener on the shop queue, and when the local broker has a network connection to the data center defined, the messages sent to that queue are passed to the shop using a store-and-forward mechanism.

The important section in the broker configuration of the Fuse Message Broker configuration in the shop is:

log4j appender that transmits log messages via JMS and implement a log4j collector module that retrieves the log messages and reproduces the log files at the central location.

The mock application explored here does not address the important issue of operations. The shop registry, however, could be the cornerstone for automation utilities used in operations. Examples of those utilities include a simple monitoring Web page displaying the online/offline state of the shop nodes or whether any messages are stuck in transition. The JMX APIs of the Fuse ESB can be used to gather and display that information.

DEPLOYMENT CONSIDERATIONS

For the deployment two parts need to be considered—the server- side and the shop-side nodes. As the server resides in a data center, it is easy to install it manually. The shop node installation is automated from a deployment package.

For the server side, the installation is straightforward. Simply download Fuse ESB from http://fusesource.com/downloads and install it.

Large Scale Deployments with Servicemix 4

www. Before starting, edit the file ${servicemix.home}/etc/ org.apache.servicemix.features.cfg to minimize the fusesource number of bundles that are installed at startup. The file below shows the minimum set of features required to run the retail mock

application. .com

################################################### ##### # # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # ################################################### #####

# # Comma separated list of features repositories to register by default #

Large Scale Deployments with Servicemix 4

www. featuresRepositories=http://svn.apache.org/repos/as f/servicemix/smx4/obr- repo/features.xml,mvn:org.apache.servicemix. fusesource nmr/apache-servicemix-nmr/1.0.0.4- fuse/xml/features,mvn:org.apache.servicemix.feature s/apache-servicemix/4.0.0.4-fuse /xml/features

.com

# # Comma separated list of features to install at startup # featuresBoot=management,transaction,activemq,camel, cxf

In addition to the installed bundles you will need to install camel-jms and activemq-camel from the Fuse ESB shell as follows: osgi install -s mvn:org.apache.activemq/activemq- camel/5.2.0.2-fuse osgi install -s mvn:org.apache.camel/camel- jms/1.5.4.0-fuse

Now you are ready to install the Retail packages: osgi install –s mvn:com.progress.pso.samples.retail/ShopRegistry osgi install -s mvn:com.progress.pso.samples.retail/ServerEndPoints osgi install -s mvn:com.progress.pso.samples.retail/ShopServerRoute r The modules are defined in a way that they do only debug logging. To increase the log level for the retail components enter log set DEBUG com.progress.pso.retail

To build the shop package, install another instance of Fuse ESB. The retail bundles are osgi install –s mvn:com.progress.pso.samples.retail/ShopRegistry osgi install -s mvn:com.progress.pso.samples.retail/ShopEndpoints

Large Scale Deployments with Servicemix 4

www. The variable piece in the deployment due to the design above is a file

fusesource ${servicemix.home}/etc/com.progress.pso.sample s.retail.cfg. This file needs to contain the properties shop.country, shop.system and shop.shopnr. By distributing a file machine.xml with the package as we have .com

shown above, a piece of Java determines the IP addresses of the shop node and matches those against the XML file. The properties file is then generated from the code. The install package would then simply extract the package and apply the stylesheet afterwards.

As a reminder here is an example machine.xml that is distributed to the nodes:

The next piece to add is a piece of Java determining the network addresses of the host and put them into an XML file: while(nwInterfaces.hasMoreElements()) { NetworkInterface nwInterface = (NetworkInterface)(nwInterfaces.nextElement()); Enumeration addresses = nwInterface.getInetAddresses(); while (addresses.hasMoreElements()) { InetAddress addr = (InetAddress)(addresses.nextElement()); System.out.println ("" + addr.getHostAddress() + ""); } }

Large Scale Deployments with Servicemix 4

www. Once that is done, adding XSL code maps the IP addresses against

the XML file (only the important pieces are shown below): fusesource

.com

To summarize, a Fuse ESB instance is prebuilt with all the required bundles, including the machinemapping.xml and the stylesheet. The installation simply extracts the archive and generates the property file that is required to customize the installed bundles.

Large Scale Deployments with Servicemix 4

www. FUSESOURCE: OPEN SOURCE FOR THE ENTERPRISE fusesource

Apache SOA projects are built for IT developers, and the FuseSource distributions are intended for a subset of those users. The .com FuseSource products are targeted at large enterprise environments

running mission-critical applications. The team at FuseSource, a wholly owned subsidiary of Progress Software, has over 15 years of experience working with complex infrastructure at Fortune 2000 companies and tests, certifies and supports distributions of Apache ServiceMix, ActiveMQ, CXF and Camel to meet the needs of this audience. FuseSource provides enterprise-class services and support to ensure that customers are successful with these products.

FuseSource products are a family of components for SOA development that are distributions of Apache projects. FuseSource products are always open, always free, and many of the committers are a part of the FuseSource team. The components include

. Fuse ESB—Based on Apache ServiceMix, Fuse ESB provides a standardized methodology, server, and tools to deploy integration components. Fuse ESB was built from the ground up to support the JBI specification (JSR 208) and provides a structured environment to manage and deploy the components that developers create using Fuse Services Framework® and Fuse Mediation Router® as well as additional JBI-compliant components like BPEL. . Fuse Message Broker— Based on Apache ActiveMQ, Fuse Message Broker is a cost-effective and flexible messaging platform for reliably executing transactions and moving data, efficiently scaling operations, and connecting processes across heterogeneous database and application environments.

Large Scale Deployments with Servicemix 4

www. . Fuse Services Framework—Based on Apache CXF, Fuse

Services Framework provides the ability to create Web fusesource services on an existing or new application. Specifically, Java developers can use JAX-WS, JavaScript, REST, or POJOs to

create web services for a client or server endpoint. .com

. Fuse Mediation Router—Based on Apache Camel, Fuse Mediation Router makes it easy for Java developers to quickly implement integration patterns using a code-first approach using simple POJOs. The API maps to easy-to-understand Enterprise Integration Patterns so a developer with minimal integration skills can quickly create integration components with routing, mediation, and other integration capabilities. . Fuse™ HQ— SOA management and monitoring system available as a part of any FuseSource support subscription, Fuse HQ is integrated with the Fuse product family for real- time administration and control of FuseSource SOA infrastructure.

www.

fusesource

.com

Contact FuseSource FuseSource, 14 Oak Park, Bedford, MA 01730 USA On the Web: fusesource.com Email: [email protected] Toll Free: 888.280.5380 Tel: 781.382.4002 Outside US: +31 10 235 11 22

FuseSource FuseSource, a wholly owned subsidiary of Progress Software, is a community of open source experts that provide software, support, training, and consulting for the most popular Apache-licensed open source integration projects including Apache ServiceMix, ActiveMQ, Camel and CXF. The FuseSource team includes key committers and the leaders at Apache who know the code the best to help FuseSource customers build reliable and scalable software integration infrastructure.