Large-Scale Deployments with Servicemix 4 a Fusesource White Paper
Total Page：16
File Type：pdf, Size：1020Kb
WHITE PAPER Large-Scale Deployments with Servicemix 4 A FuseSource White Paper Andreas Gies, Principal Consultant, FuseSource March 2009 Large Scale Deployments with Servicemix 4 www. TABLE OF CONTENTS fusesource Introduction .......................................................................................2 A Detailed Look at the Case ...............................................................4 .com Initial Scoping of the Project ..............................................................5 Software Installation and Configuration ......................................5 Functional Requirements .............................................................5 Performance Requirements .........................................................6 High-level Application Design ............................................................6 The Shop Registry ........................................................................8 The Server Endpoints Component ...............................................10 The Shop Server Router Component ..........................................12 The Shop Endpoints Module ........................................................14 Benefits of Using OSGi ................................................................15 Possible Extensions ......................................................................15 Deployment Considerations ..............................................................16 FuseSource: Open Source for the Enterprise ....................................21 1 Large Scale Deployments with Servicemix 4 www. INTRODUCTION fusesource One of the more difficult challenges of building an enterprise system is designing for scalability. It is often skipped in the proof-of-concept stage of development, and it requires product-specific knowledge .com and experience. To help the development of extremely large deployments, the FuseSource team is sharing the experiences and results of a specific, real-world deployment at a large retailer with stores throughout Europe. This company has over 2000 retail shops spread out across six European countries. This white paper is using the experience we gained in this project with this customer and suggests a blueprint for how to deploy large systems based on Apache ServiceMix . The code discussed here is available on the FuseSource wiki at http://fusesource.com/wiki/display/ProdInfo/Large+Scale+Deploym ent+Demo. The company has a central ERP system (based on SAP) that serves as the master database for all product and pricing information and performs all disposition tasks required to optimize the delivery of goods to each retail outlet. Data collected by the shops is then used by the ERP system to perform the disposition tasks, and pricing decisions are disseminated to the shops. The company was using a typical end-of-day batch file transfer pattern. Market data was transferred once a day to the data center, where batch jobs did all the central processing, and the new information for the markets was then transferred back in batch. With a growing number of shops and a growing palette of goods the company offered, soon there was not enough processing time at night to accommodate the growth of the company. In addition, the data was always stale. 2 Large Scale Deployments with Servicemix 4 www. The solution was to migrate to an architecture that could support real-time delivery of the shop data in the data center for processing fusesource and real-time communication of pricing events to the shops. Given the huge number of shops and the limited number of IT personnel in the individual countries, the development team followed two key .com principles: . To limit the resources needed for development, the operations in each shop standardized from an organizational point of view to limit the number of different services needed in each shop. To accommodate the uniqueness of each shop, services needed to be parameterized. The parameters would reflect the location and identity of each shop. Proper design and implementation only brought the team half the way; the provisioning infrastructure was also key for a distributed project of this size. The provisioning infrastructure included: . A single installation package to be used as an installation base for each shop . An installation package to determine the environment it is running in (e.g., the unique network address of the shop host or the machine name) and derive the service customization parameters from that unique key . Service parameters applied during installation so that the service instances are ,in fact, associated with a specific shop . A mechanism in the installation package to establish the network connectivity between the data center and the freshly installed shop 3 Large Scale Deployments with Servicemix 4 www. This white paper is using the experience we gained in this project with this customer and suggests a blueprint for how to deploy large fusesource systems based on Apache ServiceMix. The FuseSource team has decades of experience building mission-critical, IT applications for the Fortune 500 and employs many of the key committers and .com leaders at Apache. A DETAILED LOOK AT THE USE CASE The project consisted of integrating the systems in each of the individual retail outlets (shops) with the central ERP system at headquarters. In each shop, two applications are present: a point-of- sale (POS) system and a warehousing system (WH). The POS system is the backend system for the cash desks. It needs accurate pricing information from the data center and provides data about the revenues and which goods have been sold to the data center. The system originally sent accumulated data at the end of day to the data center via a traditional batch-oriented file transfer. At some point during the night the POS server received pricing updates from the data center and forwarded them to the cash desks. The WH system is a dispositioning system that generates and transfers the orders to the central ERP system (SAP). Any state changes for theses orders are communicated back to the WH server via batch-oriented file transfer. Both the POS and WH systems had a file-based interface in each shop to share information. The SAP Business Connector (BC) collected the files to transmit to the central ERP system and dropped data files into the file system to be imported into the local POS and WH systems. At headquarters, a mapping tool was used to transform all data going into or coming out of the central ERP system. For simplicity, assume that the mapping tool communicates through a JMS. 4 Large Scale Deployments with Servicemix 4 www. The existing installation was working, albeit with stale data, but headquarters imposed a new requirement on the system that fusesource threatened to break the proverbial camel’s back: the data from the POS systems also had to be shared with a data warehouse component to be analyzed for cost-efficiency, fraud detection of .com cash desk operators, etc. INITIAL SCOPING OF THE PROJECT The FuseSource team reviewed the existing infrastructure, future requirements, and the scale of operations and scoped the project into three separate, functional areas. Software Installation and Configuration The solution would be a distributed application with decoupled components communicating asynchronously through a JMS middleware component. This would replace the company’s batch file transfer with an event-driven architecture. The components would be independent from each other and communicate using well-defined interfaces. Each shop would receive a deployment package that contains the necessary components to connect the shop to the data center, and the number of different deployment packages would be minimal. An optimal solution would deliver the same deployment package to each shop and only use different configuration files for the setup. The components must run in different profiles such as development, QA and production and must be adaptable during deployment. Functional Requirements All data must be transferred without data loss. The data must be transmitted in near real time despite low bandwidth networks. Required changes in the participating applications to move from batch processing to event-driven data processing are not in the scope of this document. 5 Large Scale Deployments with Servicemix 4 www. The design must be flexible and future-proofed to support future requirements. fusesource Performance Requirements A normal market performs two-to-three cash operations a minute, .com With 300 markets online in phase 1 this translates to roughly 1000 messages per minute. In phase 2 with about 600 markets online the traffic goes up to 2500-3000 messages per minute. Phase 2 includes shops with more cash decks and, therefore, more cash operations per minute than the shops included in phase 1. HIGH-LEVEL APPLICATION DESIGN The retail scenario has two major functional components. The first component is run in the data center and contains the following modules: . Server endpoints module—provides the communication interface to the central ERP system. This module collects the data transmitted from the central ERP system regardless of the underlying transport and creates a shop request document in a defined JMS queue. Content-based routing module—is responsible for reading the shop request documents from the JMS queue and determining which shop receives the request. This module uses the shop registry module to facilitate the routing decisions. Shop registry—provides a central registry of shops that