BizTalk Value

August 2006 Alan Smith About Alan Smith

BizTalk Consultant • Long history of development • Working for KnowIT Consulting in Stockholm • Architect, developer, and mentor roles BizTalk Trainer • Delivering BizTalk Server courses at Informator • Delivering advanced “BizTalk Deep Dive” courses for QuickLearn BizTalk MVP • Active in BizTalk Server developer community • Publish “The Bloggers Guide to BizTalk” • Run BizTalk User Group Sweden (BUGS) BizTalk Server Value

BizTalk Server Introduction

• BizTalk server overview • BizTalk in the Microsoft product line

BizTalk Feature Set

• Publish subscribe message hub • Integration server • Business process workflow engine • Business intelligence tools

System Integration Overview

• The importance of integration • Problems implementing integration

Integration Strategies

• Batch oriented • Message oriented • Service oriented

Case Study

• FRAS Integration hub BizTalk Server Timeline

•Artifacts developed using BizTalk Server numerous tools 2000

•Improvements BizTalk Server made to several 2002 features

•Complete rewrite of product BizTalk Server •Built on .net 1.1 2004 •Visual Studio Development Tools •Massive improvements in all areas

•Built on .net 2.0 BizTalk Server Big improvements in massaging engine •New Administration console 2006 •BAM Improvements

•Integration with WCF and WF BizTalk Server •Integrated support for RFID devices •Support for EDI documents and AS2 transactions 2006 R2 •Integrated support for office 2007 and SharePoint

•Includes“BizTalk Solution Designer? BizTalk Server •Includes new BizTalk mapper V Next interface? BizTalk Server 2006 Licensing Options

Enterprise Edition Standard Edition Developer Edition • $29,999 (210,000 SEK) • $8,499 (60,000 SEK) per • $499 (3,500 SEK) per per processor processor user • Complete EAI, B2B, and • Complete EAI, B2B, and • Complete EAI, B2B, and Business Process Business Process Business Process Management Management Management functionality functionality functionality • Includes all current and • Includes all current and • Includes all current and new application and new application and new application and technology adapters technology adapters technology adapters • Unlimited scale-out, • Limited to two CPUs on • Included in MSDN scale-up and clustering a single server Universal Subscription • Includes Host • Limited to five internal • Limited to designing, Integration Server applications development and Enterprise Edition • Includes Host testing solutions Integration Server Enterprise Edition BizTalk Server in the Microsoft Product Line BizTalk Server Development

VisualStudio.net 2005 is used as the BizTalk development environment BizTalk artifact development is typically performed using graphical interfaces As BizTalk Server is built on .net, the development process shares many similarities with a .net project BizTalk Server functionality can easily be extended using C# or VB.net

BizTalk Sever Administration Tools

BizTalk Server 2006 provides greatly improved server administration

The BizTalk Server Administration console can be used to administer a multi server BizTalk deployment

Powerful command line tools are available for automated script driven deployment

BizTalk Server Feature Set

Business Intelligence Tools

• Business Activity Monitoring (BAM) provides real time and aggregated views of business data • Business Rules Engine allows complex business decisions to be abstracted form the business process

Business Process Workflow Engine

• BizTalk Orchestration Engine supports complex business processes • Persistence of orchestration state enables long running transactions and robust recovery from system failures

Integration Server

• Adapters provide communication with other systems • Protocols (FTP, SMTP, POP3), databases (SQL Server, Oracle, DB2), applications (SAP, Tibco, SharePoint, Siebel)

Publish Subscribe Messaging Hub

• Message box database implemented on SQL Server • Strong support for guaranteed delivery • Configurable message tracking Publish Subscribe Messaging Hub Publish Subscribe Messaging Hub Publish Subscribe Architecture • All messages that are received are published into the message box database • Each message can have zero, one, or more subscribers • Subscriptions can be based on message context (e.g. where the message was received from) or message content (e.g. the name of the company supplied in a message body field) “GuarenteedDelivery” • All messages are persisted in MessageBox database, ensuring recovery in the event of severe failures • Retry logic and backup transports are provided out of the box when sending messages Highly Scalable • A BizTalk group can easily be scaled to multiple servers • Server group can be tuned for high throughput, or low latency Highly Available • Multi-server environments can feature redundant components to provide high availability Error Handling • Handling of errors in message processing is configurable, and supplied out of the box Integration Server

Adapters • Adapters provide communication between BizTalk Server and external systems • Using adapters abstracts the complexity of communicating with a system from the developer • BizTalk Server ships with a wide range of basic adapters out of the box • A number of enterprise application adapters are included within the BizTalk Server license • Third party companies deliver a wide range of adapters for connecting to line of business (LOB) applications • BizTalk Adapter Framework allows for the creation of custom adapters BizTalk Server 2006 R2 • Adapter framework to be based on Windows Communication Framework (WCF), formally known as Indigo BizTalk Adapters

Included in License, Out of the Box Included in License Available Soon • File • SAP • Host Applications • FTP • PeopleSoft • IBM DB2 • HTTP Enterprise • Host Files • SOAP • JD Edwards One • POP3 World XE • SMTP • ODBC Adapter for • Windows SharePoint Services • Siebel eBusiness • SQL Server Applications • Base EDI • TIBCO Rendezvous • MSMQ • TIBCO Enterprise • Websphere MQ Message Service Business Process Workflow Engine

Long running business processes can be implemented using Orchestrations

Graphical design environment provides for rapid development

Visual representation of process aids in communication

Orchestrations are persisted to SQL Server database to provide high availability

Orchestration engine provides a sophisticated transaction ad error handling model

Seamless integration to .net code Business Intelligence Tools

BAM BAS Business Rules • Business Activity • Business Activity • Provides a forward Monitoring Services chaining inference • Provides powerful • Allows management engine views of information of configurable • Business decisions processed by BizTalk trading partners can be created and • Information in • Partners can be modified whilst intercepted, business added, and modified system is running in processes are not using a SharePoint production affected interface • Delivers high • Information can be • Custom agreements performance, and viewed in real time, between partners allows generation of or aggregated using can be created complex rule sets SQL Server Analysis Services Business Activity Monitoring

Decide which Review business Decide how that Business information is process and information should relevant to message contents be viewed Analyst business process

Develop BAM Developer artifacts

IT Deploy and configure BAM Professional artifacts

Business View data provided Configure and Analyst by BAM receive alerts

What is Integration?

”Aset of technologies that enable organizations to control the quality of information by ensuring that it remains complete, consistent, and current across the entire portfolio of applications that define the operationsof the business” - IDC Integration Adds Value

Integration might provide substantialadded values … • Support for processes vs. /just/ functional support • Previous investments are secured • Increased revenue • Lead time & cost reductions • Better focus on the core business • New business models The Importance of System Integration

Cost Savings • Reuse of existing architecture and systems • Reduced IT, personnel, and training costs Information Flow • Transfer of information between systems • Maintaining up to date information across diverse applications in an enterprise • Automate manual processes Acquisitions and Mergers • When companies merge or acquire other companies, information must be shared • Integration provides an unobtrusive method of achieving this

Problems with System Integration

Cost • Integration systems are expensive to purchase • Custom written solutions can be expensive to develop Complexity • Integration is, by nature, a complex discipline • Knowledge of multiple platforms, applications, and protocols is required • Often consultants from multiple companies will be required Project Overruns • Complex development can often lead to overruns • This affects the companies budget, and business operations How BizTalk Server Addresses These Issues Cost • BizTalk Server compares very favorably with other integration systems • BizTalk Server Standard Edition is just $8499 (60,000 SEK) Complexity • BizTalk Server brings integration development within the reach of .net developer • The use of adapters abstracts much of the communication implementation from the developer • Windows Server and SQL Server admin and maintenance skills can be leveraged to maintain a BizTalk deployment Project Overruns • With adequate knowledge, BizTalk Server provides a highly productive development environment • Powerful graphical tools can lead to rapid application development Integration Strategies

Batch Oriented Message Oriented Service Oriented • Systems can • Systems exchange • Web services are output and input messages with exposed by batches of each other systems information • Messages contain • Other systems call • Batches are individual objects, these web services usually text based e.g. one order, • Standard files one stock quote protocols are used • Batches are • Messages are transferred often transferred according to a using fixed time interval asynchronous (daily, weekly etc.) message queues Batch Oriented Integration

The source system will output a batch of information to a file The batch exports are performed according to a specific time schedule, typically daily, weekly etc.

The batch files are typically text based flat files

The target system will import and process these files to update its information Batch Oriented Integration Information is present in System A Batch Oriented Integration A batch file is exported from System A Batch Oriented Integration Batch Oriented Integration The batch file is imported into System B Batch Oriented Integration Information is present in System B Disadvantages with Batch Oriented Integration Information is only updated during each batch, so there are often times when it is out of date Batch files can become very large, and take time and resources to process The flat file format of batches can be complex when a hierarchical structure is required Robust error handling is required to ensure information consistency Day 1

Information present in System A Day 1

Information present in System A Day 1

Information present in System A Day 1

Information transferred to first intermediary system Day 2

Information transferred to first intermediary system Day 2

Information transferred to first intermediary system Day 2

Information transferred to second intermediary system Day 3

Information transferred to second intermediary system Day 3

Information transferred to second intermediary system Day 3

Information present in System B BizTalk Server Batch Support

BizTalk Server converts text based files to and from XML

Batch files can be split into individual messages as they enter BizTalk Server

Individual XML messages can be aggregated to produce a batch file

BizTalk provides powerful error handling and robust recoverable interchange processing to handle errors in a message batch

Much of this functionality can be configured at run time

The Flat File Schema Wizard provides a powerful graphical tool to convert complex flat file structures to and from XML Splitting a Batch File into XML Messages Aggregating XML Messages to Send as a Batch File Demo: Batch Processing in BizTalk Server In this demo you will see: • Converting a text batch to XML messages • Handling errors that occur in a batch Message Oriented Architecture

Systems communicate in real time by sending messages asynchronously M essagescontain “Byte Sized” pieces of inform ation,e.g. an order, an invoice payment Messages are often transferred using asynchronous queuing applications (MSMQ, WebSphere MQ, BizTalk Messaging) Messages are often sent as XML, and based upon industry standards Advantages of Message Oriented Integration Speed of Data Replication • Messages are typically delivered across multiple application boundaries in seconds, rather than days • Errors can be detected and fixed more quickly Ease of Implementation • Byte sized pieces of information are easier to process than batches Use of Standards • XML standards are clearly defined • XSD schemas can be used as message definitions for multiple systems Enterprise Integration Patterns

Industry standard pattern language for message based systems Provides detailed descriptions and implementation guidelines Downloadable appendix describes BizTalk Server 2004 implementations

Enterprise Pattern Implementation in BizTalk Server

Out of the Box Easy to Implement Harder to Implement • Message Endpoint • Message Translator • Aggregator • Publish-Subscribe • Message Router • Resequencer Channel • Point-to-Point Channel • Scatter-Gather • Guaranteed Delivery • Invalid Message Channel • Test Message • Process Manager • Dead Letter Channel • Polling Consumer • Channel Adapter • Event-Driven Consumer • Correlation Identifier • Content-Based Router • Message Filter • Dynamic Router • Recipient List • Splitter • Envelope Wrapper • Content Enricher • Content Filter • Normalizer • Wire Tap Service Oriented Architecture (SOA)

Platform independent Based on open industry standards Massive vender support Message based communication Brings reusability to systems and applications Flexibility for designing system integrations Problems with SOA

Legacy systems may not provide SOA support

Implementing SOA for these systems may be difficult and expensive Lack of real SOA architecture and development knowledge Adding SOA to Legacy Systems

Build a service layer onto a legacy system

• Interfacing with a legacy system can be complex and require special knowledge • Management of security of authentication can be complex • Long running maintenance can be required

Use BizTalk Server to add a SOA layer

• BizTalk Server exposes a SOA interface to clients • Adapters can often provide interface to a legacy system • Security can be managed using Enterprise Single Sign On BizTalk and SOA

BizTalk Server easily integrates with SOA

Web services can be consumed or exposed by BizTalk Server BizTalk Server provides a rich range of adapters, making it an ideal bridge between SOA and legacy systems BizTalk Server provides highly reliable messaging and message tracking out of the box BizTalk and SOA

Integration Enabler pattern taken from “TheArch Hackers BizTalkBlog”

BizTalk Server can act as an Integration Enabler exposing the functionality of legacy systems as a Communication between BizTalk and legacy systems is implemented using BizTalk adapters Business Process Engine

Often, communication between systems is more complex than sending a message Business processes can be required involving complex rules, and long running transactions The BizTalk Server Orchestration Engine provides a platform for implementing complex workflows The Business Rules Engine allows configuration, management, and deployment of complex business rules in real time Business Activity Monitoring (BAM) can provide real time feedback of business process execution BPM + Integration

Message Oriented Integration • “BizTalkServer is passive with respect to the business processes, and acts as a service to route messages between systems. Messages may be tracked, but no business relevant processes are implementedin the message engine.” Business Process Management • “BizTalkServer is active with respect to the business process. Business processes are implemented within BizTalk Server, and consume the services provided by other systems. Business process decisions are implemented using the Business Rules engine, and business activity monitoring is used store business relevant information and provide users with a view of the business processes.” Demo: Business Process and Workflow

In this demo you will see: • Implementation of human based workflow in BizTalk Server • Integration with Microsoft Office InfoPath • Integration with Windows SharePoint Services Human Workflow in BizTalk Server

BizTalk Server

The combination of using BizTalk Server, InfoPath and SharePoint allows basic human workflow to be implemented in the InfoPath business process SharePoint “InfoPathis to BizTalk what Outlook is to Exchange” Case Study - FRAS

Creating new opportunities in the Dairy business FRAS – Business Objectives

The Swedish dairies launched a new business model in 2005 that needed to support: •Using a common transport capacity (logistics) •Selling dairy products nation wide •Each company assuming at least two different roles •Connectingeach dairy’s ERP system in a reliable and secure fashion •Exchanging business information with a maintained customer integrity

This called for:

•An integration solution •Support for various message formats •Support for various business messages •Support for future changes without a complete rewrite

The solution:

•Relies on Microsoft BizTalk Server 2004 as the integration mechanism •Provides a communications infrastructure that allows for future growth •Provides web-access to operational status for critical information (stock, invoice, order etc.) Processes and operations Distribution – Supply Chain

Brand owner Carrier Customer

Dairy Distribution Store Warehouse Warehouse FRAS – Process overview

Planning Planning

Refill

Order Delivery

Support processes FRAS generellt

Fakturering på vanligt sätt Kunden ”Hubben” möjliggör transportsamarbetet mellan mejeriföretagen gen om överföring av information och sammanknytning av ERP-systemen:

ERP Från T Hubb Från V ERP

1 Uppdatering av artikelregister med VM-ägarens artiklar.

Fakturor/ 2 Uppdatering av kundregister med Transportörskunder Samlings- 3 15 fakturor Uppdatering av VM-ägarens artikelsortiment per kund. (Central- 14 fakturering)/ 13 4 Överföring av förhandsorder från säljväskor Gemensam Order faktura Fsg- 5 Orderinfo VM-artiklar statFsg- stat

6 Följesedelinfo VM-artiklar

VM-ägarrollen 7 Påfyllnad av lager Transportörsrollen

8 Uttag från lager

9 Inventering

10 Reklamation

11 Godsmottagning

12 Hanteringssvinn 3 FRAS – Technical profile • Hub – BizTalk Server 2004 – Development tools BTS / VS C# – two physical servers for BizTalk, two servers for SQL Server 2000 • VPN tunnels • Exchange – IBM/ MQ Series – FTP • Asynchronous connections with a high overall throughput / performance • Functionality provided: – Logging – Transformation – Synchronization – Routing – Persistence (customer & line items) Sveriges Mejerier

The Net Result?

• A new business model • An efficient re-use of existing logistics- and IT-solutions • An option to scale out, and up, without having to refrainto a com plete“rew rite” – a flexible and dynamic solution • A comparatively low cost for the IT part of the solution (approximately 13 MSEK including maintenance until 2007) Lessons learned - Experiences - Pitfalls - Recommendations • Integration requires many ”disciplines”… – Project manager – Test experts – O perations’experts – IT Infrastructure – Developers – Integration platform experts – … • Relying on an integration platform is almost always a profitableinvestm ent… – But there are alternatives: • Data exchange (via DTS, SSIS or equivalent) • Custom developed integration The road to successful integration?

We recommend – using an EAI-tool for • Process support and development • Business activity monitoring (BAM) • Application integration – relying on • O penstandards (if possible… ) • AnIntegration Factory™ • Proof-of-concept in the early phases • Best practices • Composite & Enterprise Integration Patterns Inthe Near Future…

BizTalk Server 2006 R2 • Integration with .net 3.0 technologies (Windows Workflow Foundation, Windows Communication Foundation) • Integrated support for RFID devices • Support for EDI documents and AS2 transactions RFID Services Platform • "We are committed to this space because it's an important technology for our customers and our partners, We're committed to providing the technology that our customers want and that our partners need to RFID-enable their solutions. You'll see us bringing innovation to this space." - Javed Sikander - Microsoft program manager for industry solutions GoingFurther…

On the web

• www.microsoft.com/biztalk - BizTalk homepage • msdn.microsoft.com/biztalk – BizTalk developer center • “UnderstandingBizTalk Server 2006” – Dave Chappell - introductory white paper on BizTalk Server 2006 available on msdn.microsoft.com/biztalk • www.biztalkgurus.com – BizTalk community site • BizTalk Server 2006 Virtual Labs on MSDN • “TheBloggers Guide to BizTalk” – Free e-book containing over 350 articles

Courses

• 5 day developer course (MOC) • 2 day administration course (MOC) • 5day advanced “BizTalk Deep Dive” (QuickLearn) • 5 day BizTalk & SOA course (QuickLearn)

Conferances

• SOA & Business Process Conference - October 3rd to 6th - Redmond Contact Information

• Blog: geekswithblogs.net/asmith