SOA Applications, Infrastructure and Management in Windows Hpcserver 2008R 2

SOA Applications, Infrastructure and Management in Windows HPC Server 2008 R2

Microsoft Corporation Published: January 2012

Abstract

Windows® HPC Server 2008 R2 provides a scalable, reliable, and secure platform for interactive applications that are based on the service-oriented architecture (SOA) programming model. Developers can rapidly develop and easily modify interactive applications for HPC clusters.

This article provides a technical overview of application development. The HPC Class Library includes the APIs that a client application can use to create sessions on the cluster. Developers can use the APIs to create reliable sessions and client interaction models such as file and recollect, batch, and shared session clients. This article also addresses the service configuration file, the Windows HPC infrastructure that supports SOA workloads, and how to monitor and manage broker nodes and sessions. This is a preliminary document and may be changed substantially prior to final commercial release of the software described herein.

The information contained in this document represents the current view of Microsoft Corporation on the issues discussed as of the date of publication. Because Microsoft must respond to changing market conditions, it should not be interpreted to be a commitment on the part of Microsoft, and Microsoft cannot guarantee the accuracy of any information presented after the date of publication.

This White Paper is for informational purposes only. MICROSOFT MAKES NO WARRANTIES, EXPRESS, IMPLIED OR STATUTORY, AS TO THE INFORMATION IN THIS DOCUMENT.

Complying with all applicable copyright laws is the responsibility of the user. Without limiting the rights under copyright, no part of this document may be reproduced, stored in or introduced into a retrieval system, or transmitted in any form or by any means (electronic, mechanical, photocopying, recording, or otherwise), or for any purpose, without the express written permission of Microsoft Corporation.

Microsoft may have patents, patent applications, trademarks, copyrights, or other intellectual property rights covering subject matter in this document. Except as expressly provided in any written license agreement from Microsoft, the furnishing of this document does not give you any license to these patents, trademarks, copyrights, or other intellectual property.

Unless otherwise noted, the example companies, organizations, products, domain names, e-mail addresses, logos, people, places and events depicted herein are fictitious, and no association with any real company, organization, product, domain name, email address, logo, person, place or event is intended or should be inferred.

Microsoft, Excel, SQL Server, Visual Basic, Visual Studio, Windows, and Windows Vista are registered trademarks of Microsoft Corporation in the United States and/or other countries.

The names of actual companies and products mentioned herein may be the trademarks of their respective owners. Table of Contents What’s new for SOA in HPC Server 2008 R2 5 Overview of SOA on an HPC cluster 6 What is SOA?...... 6 Why SOA on an HPC cluster?...... 7 Target applications for cluster-SOA...... 7 Implementing and deploying a cluster-SOA solution...... 8 Running SOA sessions on Azure Nodes 10 Burst to Azure from an on-premises head node...... 10 Windows Azure deployment with the Windows Azure HPC Scheduler...... 11 Developing a cluster-SOA Application 13 SOA and WCF...... 13 Roadmap for creating a cluster-SOA client application...... 14 The HPC Class Library...... 19 Programming a client that uses the file and recollect model...... 22 Using common data with DataClient...... 27 Enabling session pooling...... 30 Handling exceptions...... 32 Programming a service to exit gracefully...... 33 Development and debugging support...... 36 Understanding the service configuration file 38 Service registration settings...... 38 Advanced service configuration...... 40 Backward compatibility with HPC Server 2008 registration files...... 47 Understanding the HPC Infrastructure for SOA 49 Key terms...... 49 Interaction details...... 50 Broker nodes and session behavior...... 52 The HPC Job Scheduler Service and SOA sessions...... 53 Broker node failover...... 54 Managing WCF broker nodes and SOA sessions on the cluster 56 Node roles...... 56 Setting up a WCF broker node...... 57 Deploying and registering services...... 59 Diagnostic tests...... 63 Monitoring broker nodes and sessions...... 65 Managing Message Queuing...... 67 Scheduling policies...... 68 Additional References 71 What’s new for SOA in HPC Server 2008 R2 HPC Server 2008 R2 provides increased support for service-oriented architecture (SOA) solutions for clusters. The following table provides an overview of the SOA features that are available with HPC Server 2008 and the additions that are available with HPC Server 2008 R2.

Areas HPC Server 2008 Additions in HPC Server 2008 R2

Application  Volatile sessions  Support for file and recollect clients  Asynchronous method with durable session and broker client invocation plus callback APIs  Debugging support with the  Messages can be persisted by the Cluster-SOA Debugger for broker node using Message Queuing Visual Studio 2008 (also known as MSMQ) when durable sessions are created  Re-attachable sessions with durable session and broker client APIs  Decoupled request and response calls in the client proxy with the broker client API  Ability to perform session finalization by coding an OnExiting function in the service  Ability to cancel requests  Debugging support with the Cluster- SOA Debugger for Visual Studio 2008 and Visual Studio 2010  Runtime handles throttle behavior through the API

Added in SP2  Common-data APIs for SOA workloads to support staging and accessing data that is required by all calculations within one or more sessions  In-process broker option to help reduce communication overhead  Generic service APIs to facilitate porting SOA services  Service pooling options to enable faster first response to client Areas HPC Server 2008 Additions in HPC Server 2008 R2

Infrastructure  Standalone broker  Duplex Message Exchange Pattern  Request and reply  Message persistence using Message Queuing  Automated broker restart or fail over Added in SP1  Support for SOA workloads in burst to Azure Added in SP2  File share for user runtime data configured out of the box (to support SOA workloads that use the common data APIs)

Manageability  Simple monitoring,  Ease of setup diagnostics, and reporting  Single-point of configuration (service  A session maps to two jobs: a registration and session handling broker job and a service job information are all specified in one  Configuration managed in file) the service registration file,  Session handling can be configured HpcWcfBroker.exe.config, differently for each service and  Enhanced monitoring of sessions serviceAssembly.dll.config  One-to-one correspondence between a session and a service job  Scheduling mode can be optimized for interactive workloads by configuring the job scheduler to run in Balanced mode  Enhanced diagnostics  Efficient, correlated tracing

Overview of SOA on an HPC cluster

What is SOA? Service-oriented architecture (SOA) is an approach to building distributed, loosely coupled systems. With SOA, distinct computational functions are packaged as software modules called services. You can distribute services over a network so that users, systems, or other services can access them. This approach helps you reuse, extend, and distribute core functionality.

Services expose their functionality through a well-defined message exchange, which allows any application or process on the network to access the functionality of the service. The service interface allows other applications to call the service independently of the calling application’s underlying programming language or operating system. Developers can implement service functionality by using any programming language, and can write original services or package existing dynamic-link libraries (DLLs) and IT investments as services. Why SOA on an HPC cluster? Windows HPC Server 2008 R2 provides a scalable, reliable, and secure platform for interactive applications that are based on the service-oriented architecture (SOA) programming model. Developers can rapidly develop and easily modify interactive applications for HPC clusters.

HPC applications are typically classified as “message intensive” or “embarrassingly parallel.” Message- intensive applications comprise sequential tasks. Embarrassingly parallel applications can be easily divided into large numbers of parallel tasks, with no dependency or communication between them. If your applications perform repeated parallel calculations, you can package the core calculations as services and deploy the services to a cluster. This provides the following advantages:

 Speed of development: You can solve embarrassingly parallel calculations without rewriting the low-level code.

 Acceleration: You can run your applications faster by distributing core calculations across multiple service hosts. Services can be launched once and then used many times, diminishing the overhead of start-up time, particularly for sub-second calculations.

 Scalability: The service hosts that run your calculations are allocated dynamically, based on a scheduling policy and the number of incoming requests.

 Grid layer logic handled by the WCF broker node: The Windows Communication Foundation (WCF) broker node manages and allocates resources, balances workload, dynamically provisions services, and maintains message queues for durable sessions.

 Reliable calculations and failover: Windows HPC Server 2008 R2 supports automated failover with high-availability WCF broker nodes, and it provides message resilience with durable sessions. With high-availability broker nodes and durable sessions, all requests are processed if there are failures on the cluster, if a client stops responding, or if network connections are lost. When service jobs are created with durable sessions, the HPC Broker Service persists calculation requests and results. When the cluster is configured by using Message Queuing (also known as MSMQ) on remote storage and failover WCF broker nodes, the cluster migrates active sessions on failed WCF broker nodes to healthy ones. Sessions can continue without lost requests or results after the cluster recovers and the broker nodes restart.

Target applications for cluster-SOA Windows HPC Server 2008 supports embarrassingly parallel applications that use the SOA programming model. Cluster-SOA applications interactively use HPC clusters to provide near real- time calculations of complex algorithms. The following table lists examples of applications, related tasks, and the corresponding units of functionality that can be packaged as services and distributed to a cluster.

Examples of SOA applications Example Application Example Task Units of Functionality Monte Carlo simulations Predicting the price of a The pricing of each security financial instrument

BLAST searches Gene matching Individual matching of genes

Genetic algorithms Evolutionary Computational steps computational meta- heuristics

Ray tracing Computational physics Each pixel to be rendered and rendering

Calculation of user defined Calling UDF functions Each UDF function call functions (UDFs) in Microsoft Excel

Monte Carlo simulations The Monte Carlo method is a type of computational algorithm that relies on repeated random samplings. It is often used to simulate the behavior of various mathematical or physical systems. The Monte Carlo method can be used when it is not feasible to compute an exact result with a deterministic algorithm. Because the Monte Carlo method relies on repeated computation and random or pseudo-random numbers, this method is well-suited for cluster-SOA applications.

Many financial corporations use the Monte Carlo method to make investment decisions or to value mergers and acquisitions. For example, financial corporations may need to formulate a trading strategy based on historical market data, complete a risk analysis with a Monte Carlo simulation in near real time, and then price new derivative instruments.

BLAST searches The Basic Local Alignment Search Tool (BLAST) is a computer program that identifies homologous genes (genes in different species that share similar structures and functions) in different organisms. For example, there may be a gene in mice that is related to liking (or not liking) the consumption of alcohol. By using BLAST, scientists can search the human genome for a homologous gene. Because of the many iterations required, BLAST is well suited for SOA and HPC.

Calculation of user defined functions For more information about accelerating Excel spreadsheet calculations, see HPC Service for Excel.

Implementing and deploying a cluster-SOA solution To implement a cluster-SOA solution, developers create a service DLL, a service configuration file, and a client application. Then administrators copy the service DLLs to compute nodes on the cluster and register the service by placing the service configuration file on the head node. Administrators or end- users can then install the client application on end-user’s computers.

The client application provides an interface for the end-users. When end-users run the client application on their computers, it submits a service job to the cluster, initiates a session with the broker node, sends service requests, and receives responses.

The following diagram provides an overview of the process for implementing a cluster-SOA solution.

The following sections discuss the developer and administrator roles in the implementation of cluster- SOA solutions:

Running SOA sessions on Azure Nodes

Managing WCF broker nodes and SOA sessions on the cluster Running SOA sessions on Azure Nodes Using resources in Windows Azure allows you to scale out your computing resources on an on-demand basis. You can run your cluster-SOA applications on Azure Nodes that are part of an on-premises cluster (burst to Azure), or you can create a deployment entirely in Windows Azure that includes the HPC Scheduler (Windows Azure HPC Scheduler).

Some SOA services can be hosted on Azure Nodes with no changes to the service or the client. Review the following considerations to determine if your service can run on Azure Nodes, or if it requires some changes.

 Azure nodes cannot access on-premises nodes or shares directly.

 Services that get data through message requests can run on Azure with no changes to the service or the client. Services that require access to databases or other external data sources must include code that uses the Azure APIs to access data.

 Tasks executing on an Azure node cannot instantiate a connection with an on-premises head node (without performing additional configurations and setup).

 Sending and receiving messages to and from Azure Nodes can take longer than sending and receiving messages to and from on-premises nodes. Compute intensive, longer running service calls or batch-style processing might be more suitable for running on Azure Nodes.

 Azure nodes do not support DataClient, durable sessions, or in-process brokers.

Burst to Azure from an on-premises head node As of Service Pack 1 of Microsoft HPC Pack 2008 R2, administrators can supplement an on-premise cluster as needed by deploying additional Azure Nodes. You can stage your SOA service DLLs and configuration files to an Azure storage account so that every new Azure Node that is added to the cluster automatically has the DLLs installed. Requests that are submitted to the cluster can be processed by service hosts on-premises or in the cloud.

The basic steps for running cluster-SOA applications with Azure burst are as follows (assuming you already have an on-premises Windows HPC cluster, or at least an on-premises head node):

1. Obtain a Windows Azure subscription and create a new hosted service and storage account (in the Windows Azure Management Portal).

2. Create an Azure Node template (in the HPC Cluster Manager console). The Azure Node template specifies information about your Windows Azure subscription. You can then use the template to deploy new Azure Node instances directly from the HPC Cluster Manager console. Note: Detailed procedures for these steps are available in the guide for Deploying Windows Azure Nodes in Windows HPC Server 2008 R2 or in the video .

3. Use hpcpack create command line tool to package your service DLL and configuration file.

4. Use hpcpack upload to upload the package to the Windows Azure storage account.

5. If the SOA service is not already registered and deployed to the on-premises cluster, register the SOA service by placing a copy of the service configuration file in the service registration folder on the head node (typically this is C:\Program Files\Microsoft HPC Pack 2008 R2\ServiceRegistration). For more information, see Deploy and Edit the Service Configuration File.

6. Deploy new Azure Nodes from the HPC Cluster Manager Console. The staged packages containing the service DLLs will be automatically deployed to the new nodes.

Note: To deploy staged packages to Azure Nodes that are already deployed, run the hpcsync command on the Azure Nodes. This command copies OPC packages from the Windows Azure storage account to each Azure Node. SOA services should be copied to the default location (run hpcsync without specifying the optional package root argument).

7. Run the cluster-SOA client as usual. Optionally, set the session properties to specifically use nodes in the Azure Nodes group.

For more information about bursting to Windows Azure, see the following resources:

 Windows HPC Server & Windows Azure or view online

 Deploying Windows Azure Nodes in Windows HPC Server 2008 R2 or see the video

 Deploying Applications to Azure Nodes in Windows HPC Server 2008 R2

 Windows HPC with Burst to Windows Azure: Application Models and Data Considerations

Windows Azure deployment with the Windows Azure HPC Scheduler The other option for cluster-SOA is to scale out directly into a Windows Azure service. That means you don’t need hardware investments on premises – you can bring up resources in Windows Azure as needed. To get the necessary infrastructure for cluster-SOA in a Windows Azure service, you define a service that includes the Windows Azure HPC Scheduler. The Windows Azure HPC Scheduler SDK lets you create a Windows Azure deployment that includes the HPC scheduling and session management functionality so that you can run your cluster-SOA applications (and other parallel and HPC applications) entirely in Windows Azure. The easiest way to get started with the Windows Azure HPC Scheduler is to build and publish the sample HPC Azure Service that we provide (http://go.microsoft.com/fwlink/?LinkId=231894). When you publish the sample, you will have 5 role instances deployed in Azure that are already configured to run cluster- SOA workloads. The deployment will include 1 head node that is also configured as a WCF broker node, 3 compute nodes, and 1 front-end node to provide web access to your deployment.

The basic steps for running cluster-SOA applications within a Windows Azure HPC Scheduler deployment are as follows:

1. Obtain a Windows Azure subscription (in the Windows Azure Management Portal).

2. Publish a Windows Azure HPC Scheduler deployment in Windows Azure.

a. Download the code sample, Windows Azure HPC Scheduler: Sample Service.

b. Open the solution in Visual Studio 2010.

c. To publish the sample service, follow the steps in the guide for Getting Started with Application Deployment with the Windows Azure HPC Scheduler, or watch the video about how to publish the service.

Note: Optionally, before publishing the service, you can delete the sample applications that are included in the project. You can also disable the LINQ to HPC features. For more information, see Disable support for LINQ to HPC applications.

3. Deploy the SOA service to the nodes:

a. Use hpcpack create command line tool to package your service DLL and configuration file.

b. Use hpcpack upload to upload the package to the Windows Azure storage account (the account gets created automatically when you publish the sample Azure service).

c. In the Windows Azure Management portal, open a remote desktop connection to the head node in the deployment. For more information about making a remote desktop connection to a node in your deployment, see Verify the deployment section of the getting started guide.

d. On the head node, open a command prompt window.

e. Type the following command to deploy the package to all compute nodes:

clusrun hpcsync Note: Hpcsync runs automatically when nodes are provisioned. This means that any packages that you have staged to the default container on Windows Azure storage will be automatically deployed to the nodes as they are started. This ensures that if nodes are reprovisioned during a Windows Azure maintenance cycle, the SOA service (or other OPC files in the default storage container) will be automatically re-deployed to those nodes. If you stage packages to storage before publishing your Azure service, you don’t need to run “clusrun hpcsync” manually. For more information, see hpcsync documentation online.

4. To run the cluster-SOA client from your development computer (or another on-premises computer):

a. In the SOA client, specify the scheduler name as “.cloudapp.net”.

b. In the SessionStartInfo properties, set the transport scheme to use WebAPI. This means that in the background, rather than contacting the scheduler and the broker over a network, such as your enterprise network, the communication happens over the internet.

For example:

Note: In a Windows Azure HPC Scheduler deployment, features for cluster-SOA are enabled through the EnableSOA() method of the Microsoft.Hpc.Azure.ClusterConfig class. In the sample application, this method is called in the AppConfigure project.

Note: Depending on the size and throughput of your Windows Azure HPC Scheduler service, you can configure additional broker and web front end roles in your deployment to scale out and handle more sessions and client connections. You can use the HpcBrokerNode and HpcWebFrontEnd plug-ins to configure these roles.

For more information about the Windows Azure HPC Scheduler, see the following resources:

 Windows Azure HPC Scheduler SDK (download)  Windows Azure HPC Scheduler code samples (download)  Getting Started with Application Deployment with the Windows Azure HPC Scheduler (guide)  Windows Azure HPC Scheduler code sample: Overview (video)  Windows Azure HPC Scheduler code sample: Publish the service (video) Developing a cluster-SOA Application This section provides an overview of how to develop a cluster-SOA application for a Windows HPC Server 2008 R2 cluster.

SOA and WCF SOA provides the specifications and open technologies to help developers write service programs and client programs that use the Windows Communication Foundation (WCF) platform. The Microsoft Visual Studio development platform provides WCF service templates and service referencing utilities that help developers to prototype, debug, and test applications.

The following list describes the basic elements of a SOA application:

 Service: A service exposes one or more endpoints, and each endpoint exposes one or more service operations. Endpoints are places where messages are sent or received (or both), and they define all the information that is required for the message exchange.

Developers can write original services in any programming language or package existing DLLs and IT investments as services.

A service can publish metadata that describes the characteristics of the service. The metadata is used to generate a client proxy and the accompanying configuration that a client application uses to interact with the service.

 Client proxy (also known as a WCF client): A client proxy exposes the service operations as methods in the programming language of your choice in the .NET Framework, for example, Visual Basic or Visual C#. When you add a service reference to a project in Visual Studio, the client proxy objects are created automatically. You can also generate a client proxy by pointing the ServiceModel Metadata Utility Tool to a service that is publishing metadata For more information, see ServiceModel Metadata Utility Tool (Svcutil.exe) (http://go.microsoft.com/fwlink/?LinkID=139432).

 Client application: The client application provides an interface so that end-users can access the functionality of one or more services. For a cluster-SOA client, when end-users run the application on their computers, the application starts a session on the cluster, and it creates an instance of the client proxy to call the service operations and get responses.

For more information about WCF, see:

 Fundamental Windows Communication Foundation Concepts http://msdn.microsoft.com/en- us/library/ms731079.aspx

 Windows Communication Foundation http://msdn.microsoft.com/en- us/library/ms735119(v=VS.90).aspx Roadmap for creating a cluster-SOA client application A basic cluster-SOA client application must do the following:

 Create a session on the cluster. This is how the client application connects to a specific cluster and obtains cluster resources.

 Call service operations and make requests. This can be done through a client proxy, which can optionally be wrapped in one of the HPC classes for additional functionality, or through a messaging protocol such as SOAP.

 Get responses from the service. Responses can be retrieved synchronously or asynchronously.

SOA applications can work with data in various ways. Some applications require small amounts of data for each request that can be passed along in the message. Other applications rely on larger amounts of data, or common data sets that are required by all requests. This section provides a roadmap for navigating the options that the HPC SOA model provides for creating a session, calling service operations, retrieving responses, and working with data.

Creating a session To initiate a session on the cluster, the client application calls the CreateSession method (synchronous) or the BeginCreateSession method (asynchronous). When calling one of these methods, the client application specifies the head node name and the service name. Optionally, the client application can specify information regarding resources allocation such as the job template to use, priority level, number of required resources, and resource type. This information is specified in the SessionStartInfo object. When a session is started, a job is automatically submittedto the head node. The job includes a command to start the service host and load the specified service on each allocated resource.

The CreateSession and BeginCreateSession methods are available in both the DurableSession and Session classes. These two session options are described and compared in the following table:

Session Option Pros Cons DurableSession  Messages are automatically  Persist to disk means lower Used for reliable persisted on the broker node throughput than non-durable computations and for file so that messages are not lost sessions. and recollect style clients. in the event of intentional or  Service does not begin processing unintentional disconnection until client flushes requests, (such as network failure). potentially slowing down the first  Failed messages are retried response. automatically (up to the Job  Might leave stale data on the disk Scheduler retry count). if client does not purge explicitly  The cluster can be configured (SOA runtime periodically cleans for high availability, allowing stale data according to cluster- durable sessions to migrate wide settings). to different nodes.  If client persists messages itself,  All messages are stored to then durable session might disk, requiring less memory unnecessarily store messages than for non-durable again. For example, some clients sessions. generate requests from a persistent store. A non-durable session might be more appropriate. Session (interactive)  Better performance. Higher  Volatile. Messages are not Used for fast responses throughput than durable persisted in event of intentional or and interactive style sessions because messages unintentional disconnect. clients. are not written to disk.  Client must resend failed messages  Service processing begins as (restart from last received soon as first request is response). received.  All messages stored in memory (not to disk), requiring more memory than for durable sessions.  Memory usage can be mitigated by adding more broker nodes or by setting throttle limits. Client must fetch responses continuously to avoid being blocked by throttling.

Advanced session options You can specify additional properties in the SessionStartInfo object to enable advance options such as session sharing, using an in-process broker, or to override some default broker settings such as timeouts and message throttling.

Session sharing or session pooling

Session sharing or pooling can be useful if you need multiple clients to share a resource allocation due to scheduling and cluster usage policies, or if you want to reduce start-up time overhead. Depending on how busy the cluster is, a job might have to wait in the queue before resources become available. Attaching to a session that is already running, or to a session pool, means that the resources are already allocated to the service and that the service is already loaded.

You can opt in to sharing or pooling by setting the ShareSession property or both the ShareSession and the UseSessionPool properties (of the SessionStartInfo object). The following table describes and compares these options:

Sharing Option Pros Cons ShareSession  Supported since version 2 of HPC  Client must have way to get Pack (HPC Server 2008). the session ID if joining a  Client can explicitly control how shared session. many clients share a session.  Need a primary client to provide controls that manage all sessions and close when appropriate. ShareSession  Fast access to service.  No fine control over session +  Simple to configure and run. sharing. UseSessionPool Minimal change to client (set  Only one session is used, so (added in SP2) UseSessionPool to true). does not scale across broker nodes if many clients are sharing a pool.

In-process broker

The in-process broker option was added in SP2 to help reduce communication overhead for SOA sessions. The in-process broker runs in the client process, and thereby eliminates the need for a broker node, reduces session creation time, and reduces the number of hops for each message. For example, one usage pattern for the in-process broker is as follows: Instead of running the client application on a client computer, you submit the client application to the cluster as a single-task job. The client application creates a session on the cluster, and instead of passing messages through a broker node, the client sends requests and receives responses directly from the service hosts (compute nodes).

You can enable the in-process broker by setting the UseInProcessBroker property of the SessionStartInfo object.

The following table describes and compares broker options:

Broker Option Pros Cons WCF broker node  Full feature set.  One extra hop per message  Supports burst to Windows between client and broker Azure. node, potentially increasing latency.  Broker node must be configured by the cluster administrator. In-process broker  Client communicates with  Limited feature set. Durable service host directly, less session and multiple sessions latency. per process are not  Does not require a broker supported. node on the cluster.  Does not support burst to  No public network required Windows Azure. if the client runs on the  Client must have direct cluster. access to compute nodes.

Configuring broker behavior

Each service is registered on the cluster through a service configuration file. The configuration file can define how the broker should manage the service in terms of monitoring, message throttling, and timeouts. A subset of these broker settings can be specified when you create a new session on the cluster. Configurations that are specified at session creation override the default settings in the service configuration file.

To fine tune broker behavior for a particular session, you can set the BrokerSettings property of the SessionStartInfo object. See the API reference topic for the BrokerSettingsInfo class about what broker settings can be configured through this property.

Calling service operations The client application can call service operations through a client proxy or through a messaging protocol such as SOAP. A client proxy can be wrapped in the HPC BrokerClient class for additional functionality and built-in error handling.

The following table describes and compares the supported options:

Communication options Pros Cons Client proxy (WCF client)  Standard method for  No advanced SOA support interacting with a WCF for features such as durable service. sessions, file and recollect  Do not need to use HPC style clients, and high session APIs to interact with availability. service.  Not supported for burst to Windows Azure.  Deprecated. BrokerClient  Supports all client features  Must use HPC session APIs to such as durable sessions, file interact with service. and recollect style clients, and high availability.  Simple modifications can be made to client to enable burst to Windows Azure. SOAP  Currently the only option if  Not supported for burst to the client is running on a Windows Azure. different platform or written  Protocol is complicated. in Java or C++.  Deprecated.  Supports all client features such as durable sessions, file and recollect style clients, and high availability.

Processing responses You can get responses back from the service by calling the GetResponse method of your client proxy or through a messaging protocol such as SOAP. The HPC session APIs provide another option for receiving response through a broker response handler.

The following table describes and compares the GetResponse and the broker response handler options:

Response options Pros Cons GetResponse  All responses come back in  Need to explicitly create a one foreach loop. Easy to thread if the main thread use in many cases. should not be blocked. Broker response handler  Non-blocking, asynchronous  Programming model is more callback. Uses the app complex. threadpool, so need to  Thread safety is not create thread explicitly. guaranteed in case of timeout exception. Client needs to handle multi- entrance.  For interactive sessions, the last response is held until EndRequest() or Flush() is called.

Working with data SOA applications can work with data in various ways. Some applications require small amounts of data for each request that can be passed along in the message. Other applications rely on larger amounts of data, or common data sets that are required by all requests.

The following table describes and compares some data options:

Data option Pro Con Data in message  Efficient for small amounts  Inefficient if data is common of data that is different for to every request. each request.  Inefficient for transporting  Currently, the only way to messages greater than 1 MB. transport data to Azure Nodes on the fly. DataClient  Good for common data,  Currently not supported for which replicates to all burst to Azure. necessary compute nodes.  Data can be bound to a job, so that it is automatically cleaned up after job deletion.  Works out of the box. Minimal or no configuration work for the cluster administrator.  The access control list for the data share is configured automatically during SP2 install on the cluster. It’s secure out of box. Custom file share or Velocity  If your current code uses  Not supported for burst to solution that, keep using it. No need Azure. to code against HPC APIs.  Does not work out of the box. Requires extra configuration from the cluster administrator.  No automatic cleanup.

The HPC Class Library The HPC Class Library includes APIs that a client application can use to create sessions on the cluster and interact with services that are deployed to the cluster. The HPC Pack 2008 R2 SDK includes support for creating reliable sessions and client interaction models such as file and recollect, batch, shared session, common-data, and in-process broker. These scenarios are supported by APIs in the Microsoft.Hpc.Scheduler.Session namespace. You can use the APIs to create an interactive or a durable session with the cluster, and you can either use your WCF client proxy directly to call the service, or wrap the proxy in one of the HPC client objects for additional functionality and error handling mechanism that are built into the runtime. Many of these models do not require changes in the service code, only to the client code.

This section provides more information about some of the new classes in the Microsoft HPC Pack 2008 R2 SDK (and later SP releases).

DurableSession class The DurableSession class supports reliable computations. When a durable session is created, the HPC Broker Service persists request and response messages, so if the client or network fails, the requests and responses are not lost. Client applications use the session ID to reattach to a session and retrieve responses.

Note: For durable sessions, the message size limit is 1 GB because of Message Queuing.

The DurableSession class provides the following method to retrieve responses:

AttachSession method: Provides a non-blocking call that retrieves responses, but it does not cause the broker node to release the responses (so they can be retrieved again in the case of failure). This allows the client to attach to a failed session to retrieve responses or to restart a file and recollect session to retrieve responses.

Note: You can configure high-availability WCF broker nodes to ensure resilience if the WCF broker node fails.

BrokerClient class In a standard WCF client, a single call sends and receives responses. With the BrokerClient class, sending and receiving are separate calls. The BrokerClient constructor wraps the WCF client to override standard call behavior and create a message-based proxy.

The BrokerClient class provides the following methods for sending and receiving messages:  SendRequests method: Sends a batch of requests to the WCF broker node. In non-durable sessions, requests start processing when they are received. In durable sessions, the WCF broker node does not start processing requests until EndRequests is called. The client application can break a large batch of requests into separate BrokerClient instances, so all the requests do not have to be sent before the processing begins.

 EndRequests method: Confirms that all messages have been sent and ends the enumerator for the responses. In durable sessions, this call signals that the WCF broker node should start processing requests. If this call fails during a durable session, the client application must resend the batch of requests.

 GetResponses method: Returns a response enumerator so the client application can loop through to get the responses.

You can create multiple BrokerClient instances in one session. You can use each BrokerClient instance to send a batch of requests and receive the corresponding responses. When you create multiple instances of a BrokerClient object, you must give each instance a unique ID to identify the responses. You can create multiple BrokerClient instances to perform the following tasks:

 Send a batch of requests with one parameter set, change the parameter set, and then send a new set of requests in a different BrokerClient instance within the same session.

 Run multiple models at the same time with different parameters by using different BrokerClient instances.

 Break large batches of requests into separate BrokerClient instances in durable sessions. This means that the WCF broker node can start processing requests before the entire batch is sent.

 Share one session with multiple client applications by creating BrokerClient instances with unique identifiers. Each batch of computations is tagged with the BrokerClient ID to identify requests and retrieve results.

Notes The BrokerClient class is not thread safe, so the client should not call SendRequests or GetResponses in multiple threads.

Only one user at a time can call GetResponses with a particular client ID. If a second user tries to get responses for the same client ID at the same time, the first user will get an error message and the second user will gain control of the client.

DataClient class SP2 includes the DataClient class that support staging and accessing common data that is required by all calculation requests within one or more sessions. The data client includes methods to stage data to the shared folder for runtime user data and to read data from that share. Note: One of the configuration steps when you install Microsoft HPC Pack 2008 R2 SP2 is to create a shared folder for runtime user data. The DataClient APIs use this folder. If you are using the common data features in a development or test cluster, or if you are setting up a small cluster, you can accept the default runtime data configuration during setup, which creates a hidden share on the head node. In a production cluster, the administrator might configure the runtime data share on a separate file server.

DataClient includes the following methods for working with shared data.

To create or connect to a data client:

 Create method: Creates a new data client on the specified cluster and an associated subfolder in the runtime user data share. One of the required input parameters is the ID for the data client. The ID is a string value that you assign to the client at creation time, and can be a descriptive, friendly name that helps you recognize the data.

By default, only the user who ran this method has access to the new share. If you want the data to be available to multiple cluster users, you can specify the list of users as an overloaded input parameter.

The data client can be created by the same application that initiates a session and calls a service, or it can be created by a separate application independently of any sessions that will use the data.

 Open method: Opens an existing data client on the specified cluster.

To delete a data client:

 SetDataLifeCycle method: If the data client only needs to be used within a single session, you can associate the data to the session lifecycle. When the specified session ends, the data is automatically deleted from the share.

 Delete method: Deletes the data client and deletes the associated data from the share.

To stage data to the file share (data can only be staged on time per data client):

 WriteAll method: Writes the specified data to the file share. You must specify the data type and the data to stage. For this method, the data must be serializable.

 WriteRawBytesAll method: Writes the specified data to the file share. This method does not serialize the object and will write the byte stream directly. Use this when you want to transport a file or if you are using a programming language that cannot handle .NET serialization. For example, to transport a file you can use call this method as follows: “myDataClient.WriteRawBytesAll (File.ReadAllBytes(“DataFile.txt”));” Note: Optionally, you can include an additional input parameter for the write methods if you want the data to be compressed before it is written. If you enable compression, ensure that the data is compressible. Attempting to compress uncompressible data actually increases the data size.

To read data from the file share:

 ReadAll method: Reads all the data in the data client and returns it as the specified type.

 ReadRawBytesAll method: Reads all the data in the data client and returns it as a byte array.

Programming a client that uses the file and recollect model A client application that uses the file and recollect model allows users to send requests, close the client, and then connect at a later time to receive the results. This is particularly useful in the case of long- running sessions. To implement the file and recollect model, the client does the following:

 Uses a durable session to connect to the broker.

 Uses a BrokerClient proxy to access the service functionality.

 Separates the request and response calls into different functions.

o The request function sends the requests and then calls the EndRequest method to inform the broker node that this instance of the BrokerClient is finished sending request messages and that the broker node should commit all request messages.

o The response function uses the AttachSession method to re-attach to the session and retrieve the responses, and then uses the Close method to clean up the session.

Generate the WCF client When you write a client application, the first step is to generate the WCF client (client proxy) for the service. A WCF client exposes the service operations as methods in the programming language of your choice in the .NET Framework, for example, Visual Basic or Visual C#. Your client application will use the methods in the WCF client to invoke the service. You can automatically generate the WCF client by adding a service reference to the client application project in Visual Studio. To support asynchronous calls and the BrokerClient class, you must select the following advanced options for the service reference in Visual Studio:

 Generate asynchronous operations. This option generates methods such as BeginGetData and EndGetData in the resulting WCF client object.

 Always generate message contracts. This option implements the GetDataRequest and GetDataResponse data types for the BrokerClient object. The following figure illustrates the process of adding a service reference for a service on the local host named “Service1”:

Alternately, you can generate a WCF client by pointing the ServiceModel Metadata Utility tool to a running service that is publishing metadata. For more information, see ServiceModel Metadata Utility Tool (Svcutil.exe) (http://go.microsoft.com/fwlink/?LinkID=139432). Create a BrokerClient instance In your client code, you can create a BrokerClient instance from your WCF client. The following code snippet illustrates how to use the BrokerClient constructor to wrap a WCF client (referred to as IService1).

BrokerClient client = new BrokerClient(session, binding)

The resulting BrokerClient instance includes decoupled send and receive methods, and it can return responses that are mapped to requests. The following file and recollect code sample shows how to use the BrokerClient class.

Code sample: File and recollect The following code sample demonstrates a basic file and recollect model:

// THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A // PARTICULAR PURPOSE. // // Copyright (c) Microsoft Corporation. All rights reserved. using System; using System.Collections.Generic; using System.Linq; using System.Text; using Microsoft.Hpc.Scheduler.Session; using FireNRecollect.EchoService; using System.ServiceModel; namespace FireNRecollect { class Program { static void Main(string[] args) { const string headnode = "yourHeadNode"; const string serviceName = "EchoService";

if (args.Length == 1) { // attach to the session int sessionId = Int32.Parse(args[0]); SessionAttachInfo info = new SessionAttachInfo(headnode, sessionId);

Console.Write("Attaching to session {0}...", sessionId); // Create attach to a session using (DurableSession session = DurableSession.AttachSession(info)) { Console.WriteLine("done."); // Create a client proxy using (BrokerClient client = new BrokerClient(session)) { Console.WriteLine("Retrieving results..."); // Get all the results foreach (BrokerResponse response in client.GetResponses()) { string reply = response.Result.EchoResult; Console.WriteLine("\tReceived response for request {0}: {1}", response.GetUserData(), reply); } Console.WriteLine("Done retrieving results."); }

// Close the session to reclaim the system storage // used to store the results. After the session is closed // you cannot attatch to the same session again session.Close(); } } else { // Create a durable session, fire the requests and exit SessionStartInfo info = new SessionStartInfo(headnode, serviceName); Console.Write("Creating a session..."); using (DurableSession session = DurableSession.CreateSession(info)) { Console.WriteLine("done session id = {0}.", session.Id); NetTcpBinding binding = new NetTcpBinding(SecurityMode.Transport);

using (BrokerClient client = new BrokerClient(session, binding)) { Console.Write("Sending requests..."); for (int i = 0; i < 12; i++) { EchoRequest request = new EchoRequest("hello world!"); client.SendRequest(request, i); } client.EndRequests(); Console.WriteLine("done"); }

Console.WriteLine("Type \"FileNRecollect.exe {0}\" to collect the results", session.Id); } }

} } } Using common data with DataClient The DataClient class helps you share common data across all service requests. The following code samples show how a service and a client can use DataClient.

Code sample: CommonDataService The following code sample demonstrates a service that uses DataClient. In the sample, the service checks to see if the client communicated the data ID through an environment variable. The service can also get the data ID through the service context. using System; using System.Collections.Generic; using System.Linq; // THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A // PARTICULAR PURPOSE. // // Copyright (c) Microsoft Corporation. All rights reserved. using System.Runtime.Serialization; using System.ServiceModel; using System.Text; using Microsoft.Hpc.Scheduler.Session.Data; using Microsoft.Hpc.Scheduler.Session; namespace CommonDataService { // NOTE: If you change the class name "CommonDataService" here, you must also update the reference to "CommonDataService" in App.config. public class CommonDataService : ICommonDataService { private static Dictionary objects; static CommonDataService() { string dictionary_data_id = Environment.GetEnvironmentVariable("DICTIONARY_DATA_ID"); if (String.IsNullOrEmpty(dictionary_data_id) == false) { // Use ServiceContext to obtain DataClient using (DataClient client = ServiceContext.GetDataClient(dictionary_data_id)) { // Using static constructor and objects to share the data across all GetData calls saves memory and reading/deserialization cost. objects = client.ReadAll>(); } } else { objects = new Dictionary(); } }

// One can also read and initialize the data in each service request. // Notice the IDisposable pattern here. Also dispose dataclient after using it. using (DataClient client = ServiceContext.GetDataClient(raw_data_id)) { byte[] bytes = client.ReadRawBytesAll(); result = bytes.Length.ToString() + "\n"; }

// Process data obtained in service static constructor foreach (KeyValuePair obj in objects) { result += string.Format("{0}:{1}\n", obj.Key, obj.Value); } return result; }

Code sample: CommonDataClient The following code sample demonstrates a client that uses DataClient and then calls methods in a service called CommonDataService (from the previous sample):

// THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A // PARTICULAR PURPOSE. // // Copyright (c) Microsoft Corporation. All rights reserved. using System; using System.IO; using System.Collections.Generic; using System.Linq; using System.Text; using System.Reflection; // This namespace is defined in the HPC Server 2008 R2 SDK using Microsoft.Hpc.Scheduler.Session.Data; // This namespace is defined in the HPC Server 2008 R2 SDK using Microsoft.Hpc.Scheduler.Session; // This namespace is defined in the "CommonDataService" service reference using CommonDataClient.CommonDataService; namespace CommonDataClient { class Program { static void Main(string[] args) { string hostname = "[headnode]";

// DataClient Id used to identify the data, this should be unique across the cluster string raw_data_id = "raw_data_id"; string dictionary_data_id = "dictionary_data_id";

Console.WriteLine("Start. " + DateTime.Now.ToLongTimeString());

// Create a DataClient to store a Dictionary data using (DataClient client = DataClient.Create(hostname, dictionary_data_id)) { Console.WriteLine("Data {0} Created. {1} ", dictionary_data_id, DateTime.Now.ToLongTimeString()); // Here we have a DataClient whose life cycle is not managed by SOA Dictionary objects = new Dictionary(); objects.Add("key1", "value1"); objects.Add("key2", "value2"); // WriteAll() can only be called once on a data client. client.WriteAll>(objects); }

SessionStartInfo sessionStartInfo = new SessionStartInfo(hostname, "CommonDataService"); // Pass DataClient Id in SOA session's environment variable so that it could be read from service code sessionStartInfo.Environments.Add("DICTIONARY_DATA_ID", dictionary_data_id);

using (Session session = Session.CreateSession(sessionStartInfo)) { Console.WriteLine("Session {0} Created. {1} ", session.Id, DateTime.Now.ToLongTimeString());

// Create a DataClient to store the raw data read from the file using (DataClient client = DataClient.Create(hostname, raw_data_id)) { Console.WriteLine("Data {0} Created {1}. ", client.Id, DateTime.Now.ToLongTimeString()); // Add a data life cycle management so that it'll be deleted when session is done // Otherwise, the data will have to be cleaned up by client client.SetDataLifeCycle(new DataLifeCycle(session.Id)); // WriteRawBytesAll() doesn't serialize the object and will write the byte stream directly. // Use this when you want to transport a file or have non-.Net code that cannot handle .net serialization. client.WriteRawBytesAll(File.ReadAllBytes("DataFile.txt")); }

// Send/Receive SOA requests using (BrokerClient client = new BrokerClient(session)) { Console.WriteLine("Send Requests. " + DateTime.Now.ToLongTimeString()); client.SendRequest(new GetDataRequest(raw_data_id)); client.EndRequests(); Console.WriteLine("Get Response. " + DateTime.Now.ToLongTimeString()); foreach (BrokerResponse resp in client.GetResponses()) { string result = resp.Result.GetDataResult; Console.WriteLine(result); } } Console.WriteLine("Start closing session. " + DateTime.Now.ToLongTimeString()); } Console.WriteLine("Session Closed. " + DateTime.Now.ToLongTimeString());

Console.WriteLine("Start cleaning up the Data {0}. {1} ", dictionary_data_id, DateTime.Now.ToLongTimeString()); // We should delete the DataClient "dictionary_data_id" here since it's not managed by SOA DataClient.Delete(hostname, dictionary_data_id);

Console.WriteLine("Press any key to exit."); Console.ReadKey(); } } }

Enabling session pooling As of Service Pack 2, you can enable and use the session pooling option for you SOA applications. Session pooling can be useful if you want to reduce start-up time overhead. Depending on how busy the cluster is, a job might have to wait in the queue before resources become available. Attaching to a session that is already running, or to a session pool, means that the resources are already allocated to the service and that the service is already loaded.

You can opt in to sharing or pooling by setting the ShareSession and UseSessionPool properties of the SessionStartInfo object.

Code sample: A client that uses a session pool The following code sample demonstrates a basic client that uses a session pool: namespace HelloWorldR2SessionPool { class Program { static void Main(string[] args) { //change the headnode name here const string headnode = "[headnode]"; const string serviceName = "EchoService"; const int numRequests = 12; SessionStartInfo info = new SessionStartInfo(headnode, serviceName); //session in the session pool should be a shared session info.ShareSession = true; info.UseSessionPool = true; Console.Write("Creating a session using session pool for EchoService...");

using (DurableSession session = DurableSession.CreateSession(info)) { Console.WriteLine("done session id = {0}", session.Id); NetTcpBinding binding = new NetTcpBinding(SecurityMode.Transport);

//to make sure the client id is unique among the sessions string clientId = Guid.NewGuid().ToString();

using (BrokerClient client = new BrokerClient(clientId, session, binding)) { Console.Write("Sending {0} requests...", numRequests); for (int i = 0; i < numRequests; i++) { EchoRequest request = new EchoRequest("hello world!"); client.SendRequest(request, i); }

client.EndRequests(); Console.WriteLine("done");

Console.WriteLine("Retrieving responses..."); foreach (var response in client.GetResponses()) { try { string reply = response.Result.EchoResult; Console.WriteLine("\tReceived response for request {0}: {1}", response.GetUserData(), reply); } catch (Exception ex) { Console.WriteLine("Error occured while processing {0}-th request: {1}", response.GetUserData(), ex.Message); } }

Console.WriteLine("Done retrieving {0} responses", numRequests); }

//should not purge the session if the session is expected to stay in the session pool //the shared session is kept in the session pool session.Close(false);

Console.WriteLine("Press any key to exit."); Console.ReadKey(); } } } Handling exceptions You can add error handling logic to the file and recollect model. For example, you can add check pointing behavior to the client when it sends and receives requests, or you can wrap the requests and responses in a try/catch statement. It is also a good practice to include the EndRequests method in the try/catch statement because EndRequests commits the requests.

Code sample: Try/catch statements in the client The following code snippets demonstrate how to handle exceptions in the SendRequests, EndRequests, and GetResponses methods by using a try/catch statement. If the EndRequests call throws an exception, the batch of requests failed and the client should resend them.

using (BrokerClient client = new BrokerClient(session, binding)) { Console.Write("Sending {0} requests...", numRequests); try { for (int i = 0; i < numRequests; i++) { EchoRequest request = new EchoRequest("hello world!"); client.SendRequest(request, i); }

client.EndRequests(); Console.WriteLine("done"); } catch (TimeoutException) { // something goes wrong, should restart from the beginning try { client.Close(true); } catch { } goto start; goto start; } catch (SessionException) { // something goes wrong, should restart from the beginning try { client.Close(true); } catch { } goto start; }

Console.WriteLine("Retrieving responses..."); }

using (BrokerClient client = new BrokerClient(guid, session, binding)) { Console.WriteLine("Retrieving responses..."); foreach (var response in client.GetResponses()) { try { string reply = response.Result.EchoResult; Console.WriteLine("\tReceived response for request {0}: {1}", response.GetUserData(), reply); } catch (Exception ex) { Console.WriteLine("Error occured while processing {0}-th request: {1}", response.GetUserData(), ex.Message); } }

Programming a service to exit gracefully In HPC Server 2008 R2, an administrator can configure time for services to finish their current service call or perform exit steps if a running task is stopped. The default setting in the Task Cancelation Grace Period cluster property is 15 seconds.

To use Task Cancelation Grace Period, the service code should include an OnExiting function. When the service host receives the signal to cancel the task, the service host triggers the ServiceContext.OnExiting event. If the service does not respond to this event, the service exits immediately on cancelation.

Code sample: OnExiting function in the service To allow services to exit gracefully, the service code must respond to the ServiceContext.OnExiting event. In the following code sample, an event handler is created that calls the service’s OnExiting function. The event handler is registered to the ServiceContext.OnExiting event. If the event is raised, the OnExiting function is triggered. To prevent new requests from being sent, the OnExiting function sets a flag named onExitCalled. The service checks the onExitCalled flag before each iteration. If the flag is True, the service does not process the next request or response.

Note: You can include check points or exit steps in your OnExiting function.

// THIS CODE AND INFORMATION IS PROVIDED "AS IS" WITHOUT WARRANTY OF // ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO // THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A // PARTICULAR PURPOSE. // // Copyright (c) Microsoft Corporation. All rights reserved. using System; using System.Collections.Generic; using System.Linq; using System.Runtime.Serialization; using System.ServiceModel; using System.Text; using System.Diagnostics; using System.Threading; using System.IO; using Microsoft.Hpc.Scheduler.Session; namespace EchoService { // NOTE: If you change the class name "Service1" here, you must also update the reference to "Service1" in Web.config and in the associated .svc file. public class Service1 : IService1 { private static bool onExitCalled = false; private static EventHandler eventHandler = new EventHandler(OnExiting); private static StreamWriter sw = null;

private static string computerName = string.Empty; private static string jobId = string.Empty; private static string taskId = string.Empty;

const string ComputerNameEnvVar = "COMPUTERNAME"; const string JobIDEnvVar = "CCP_JOBID"; const string TaskIDEnvVar = "CCP_TASKSYSTEMID";

static Service1() { computerName = Environment.GetEnvironmentVariable(ComputerNameEnvVar); jobId = Environment.GetEnvironmentVariable(JobIDEnvVar); taskId = Environment.GetEnvironmentVariable(TaskIDEnvVar); //add the OnExiting handler ServiceContext.OnExiting += eventHandler; // this is an example of shared resources, set the file buffer size to 1M bytes and disable the autoflush sw = new StreamWriter(Environment.CurrentDirectory + "\\sample." + jobId + "." + taskId + ".txt", false, Encoding.Unicode, 1000 * 1000); sw.AutoFlush = false; }

return input; }

Console.WriteLine("Global OnExit flag before the delay : {0}", onExitCalled);

// to check if the OnExit is called on this service host // if so, suspend the service host to wait for the OnExit handler and the task exit if (onExitCalled) { Console.WriteLine("The service task is exiting gracefully. Suspend this request to wait the task to exit."); Thread.Sleep(Int32.MaxValue); }

//start the delay time and meanwhile checking the global flag if OnExiting is called sw.WriteLine("Start to delay : {0}.", delay.ToString());

Stopwatch watch = new Stopwatch(); watch.Start(); while (watch.Elapsed < delay) { Thread.Sleep(1);

} watch.Stop();

Console.WriteLine("{0}-{1}-{2} Leave EchoOnExit", computerName, jobId, taskId); sw.WriteLine("{0}-{1}-{2} Leave EchoOnExit", computerName, jobId, taskId);

return string.Format("{0}-{1}-{2} EchoOnExit", computerName, jobId, taskId);

private static void OnExiting(object o, EventArgs e) { //release the resource here in the set timeout for 15 sec Console.WriteLine("{0}-{1}-{2} OnExiting is called. Releasing the resources in 3 seconds.", computerName, jobId, taskId); onExitCalled = true;

//flush the file buffer and close the stream sw.Flush(); sw.Close();

Console.WriteLine("{0}-{1}-{2} The resources are released.The service task gracefully exit here.", computerName, jobId, taskId);

return; } } }

Development and debugging support This section provides information about development and debugging support.

Additional sample code

The HPC Server 2008 R2 SDK samples include .NET and Java code samples for client applications. For example, you can find code for multibatch clients and code to request cancelation and set up exit behavior.

To download the SDK and the code samples, see HPC Pack 2008 R2 SP2 Software Development Kit in the Microsoft Download Center. The SOA samples are in the SOA folder.

Basic test environment on a single server

When you install HPC Pack 2008 R2 on a server to create a head node, the head node is configured by default to act as a head node, a compute node, and a WCF broker node.

To download the evaluation versions of the Windows Server® 2008 R2 HPC Edition operating system and HPC Pack 2008 R2, see Windows HPC Server 2008 R2 Suite Evaluation in the Microsoft Download Center.

Logging and tracing for debugging services

In HPC Cluster Manager, the Services view lets you manage and edit the service registration files from a single location. You can also enable and configure service tracing. When you enable tracing, the trace information is logged to a file on the compute nodes.

The log files trace steps from the service call and the intermediate results on the cluster. You can collect and remove traces by using the Job Management view in HPC Cluster Manager or HPC PowerShell cmdlets. You can view the trace log files in the WCF Service Trace Viewer.

For more information, see Set Event Logging Level and Collect Trace Logs. Cluster-SOA Debugger for Visual Studio 2008 and 2010

The Cluster-SOA Debugger for Visual Studio simplifies the debugging process for applications based on SOA that are running on a Windows HPC Server 2008 R2 cluster. The Cluster-SOA Debugger is an add-in to the C# project system in Visual Studio, and it extends the remote debugger functionality.

The Cluster-SOA Debugger includes two client templates: an interactive client and a client that uses a durable session. The templates include the outline of the code that is used to start a session on a Windows HPC Server cluster and references to the necessary HPC namespaces.

When you start the debugging process, the Cluster-SOA Debugger deploys your service files and any additional files that you specify to the cluster, and then it launches the remote debugger on the specified nodes. The debugger cleans up when the session ends.

For more information, see HPC Cluster Debugger (http://msdn.microsoft.com/en- us/library/ee945373.aspx). Understanding the service configuration file The service configuration file is an XML file that is used to register the service on the cluster and configure the behavior of the service. For example, broker behavior such as monitoring, message throttling, and load balancing can be defined at the individual service level.

The service configuration file must be named servicename.config, where the servicename is the same as the name that is passed to the SessionStartInfo constructor.

At a minimum, the file must specify the registration information for the service, such as the path to the DLL for the service.

Microsoft® HPC Pack 2008 R2 includes a sample service that is named CcpEchoSvc. CcpEchoSvc is a simple, built-in service that can be used with the diagnostics tests to verify SOA functionality on the cluster. You can use the service configuration file for CcpEchoSvc as an example when you create or modify a configuration file.

Service registration settings You provide the registration information in the section which contains the element. The following table describes the service registration attributes that you can specify:

Attribute Description Default value assembly The full path to the service DLL. This can be a path None to a shared folder on the cluster, or it can be a local path on each compute node. For example:

C:\Services\yourServiceName.dll

Required. contract The interface of the service (WCF contract). None Optional if there is only one interface in the service DLL. type The class that implements the WCF contract. None Optional if there is only one interface in the service DLL. architecture The architecture on which your WCF service can X64 run. The possible values are x86 and x64 (any other value is ignored, and the default is used). Optional. includeExceptionDetailInFaults If true, returns managed exception information false to the client. This information can be helpful during development to troubleshoot a service. Setting this property to true is not recommended for production environments.

Warning

Returning managed exception information to clients can be a security risk because exception details expose information about the internal service implementation that could be used by unauthorized clients. For more information, see IncludeExceptionDetailInFaults Property. maxConcurrentCalls The maximum number of messages actively 0 processing across a service host. A value of 0 indicates that the maximum should be automatically calculated based on the service capacity (number of CPU cores) of each service host.

Optional. maxMessageSize The maximum size of the message file, in bytes. 65,536 bytes For durable sessions, the upper limit for message size is 4MB because of the limitation of MSMQ.

Optional. serviceInitializationTimeout The maximum length of time that the broker 60,000 attempts to load the service on a service host. milliseconds If an EndPointNotFound exception is returned when connecting to the service host, the broker waits the amount of time specified by endpointNotFoundRetryPeriod (in the broker settings) and then tries again.

If the broker cannot connect successfully to the service, the broker adds the node to the excluded nodes list for the job. When the excluded nodes limit is reached, the session fails.

Optional. environmentVariables Environment variables used by your service. None Optional.

For example, the following XML code shows how the service registration settings are defined in the service configuration file for CcpEchoSvc (the built-in sample service in Windows HPC Server 2008 R2):

Advanced service configuration You can add the following elements to the service configuration file (only the broker settings are described in this docuement):

 : Configure how the WCF broker node interacts with your service (monitoring, throttle behavior, service address, and load balancing settings).

 : Configure WCF, NetTcp and HTTP binding settings. For durable sessions, the upper limit for message size is 4MB because of the limitations of Message Queuing.  : Configure custom application information such as database connection strings, file paths, and XML Web service URLs.

Note: In HPC Server 2008, these sections were defined in the broker configuration file. For information about updating service registration files for HPC Server 2008 R2, see the Backward compatibility with HPC Server 2008 registration files section later in this document.

Broker monitoring attributes The section contains the element. The following configuration attributes in the element determine how the broker handles sessions for the service:

Default Attribute Description values messageThrottleStartThreshold messageThrottleStartThreshold Throttle start: messageThrottleStopThreshold specifies the number of requests that the broker 4,096 accepts before the broker stops listening and Throttle stop: waits for space to become available in the 3,072 broker queue (as requests are calculated). messageThrottleStopThreshold specifies the number of requests that in the queue to make the broker stop throttling. This number is lower than the start threshold because the broker must have adequate space in its queue before it starts listening again. The amount of memory that the broker queue requires is (messageThrottleStartThreshold) * (sizeOfRequest). If you have throttling limits set too high and your messages are too big, then the broker node can run out of memory. For example, if you set the limit to 400,000, and you expect 64 KB messages, the broker would require 25.6 GB of memory to hold the request queue. If the throttle settings are too low, nodes can be underutilized due to insufficient message throughput. loadSamplingInterval The amount of time between load sampling 1,000 passes. During a load sampling pass, the broker milliseconds checks the request queue to update (1 second) performance counters for a session, including the number of waiting requests. You can increase this number to minimize management overhead. You might want to decrease this number if the service processes very short-running requests. Unit: milliseconds allocationAdjustInterval The amount of time between allocation 30,000 adjustment passes. During an adjustment pass, milliseconds the broker updates job properties and evaluates (30 seconds) the Target Resource Count for a session based on the load sampling metrics gathered within the adjust interval. The job scheduler uses the Target Resource Count property to help determine how many service hosts to allocate to the session. Automatic allocation adjustment can help increase cluster utilization. For example, the job scheduler can shrink an idle session to its minimum resources. However, it can take time for the resources to grow when the session becomes active again. To disable allocation adjustment, you can set this property to a value of -1. Unit: milliseconds clientIdleTimeout After a client application connects to a session, if 300,000 the client does not have any activity and there milliseconds are no pending requests for the client within the (5 minutes) specified amount of time, the broker closes the connection to the client, but it does not close the session. Unit: milliseconds sessionIdleTimeout When all the clients for the session are idle 300,000 (timed out), if no more client applications are milliseconds connected within this timeout period, the broker (5 minutes) closes the session. You can extend this period if users want to keep a session open and send several batches of requests within the same session. This avoids the start-up time of creating a new session and receiving new resources. With an HTTP binding, sessionIdleTimeout does not start counting until after clientIdleTimout triggers. Unit: milliseconds statusUpdateInterval The timer interval for the broker to publish 15,000 service statistics to the job. milliseconds Unit: milliseconds (15 seconds) clientBrokerHeartbeatInterval These setting apply to clients of type Interval: clientBrokerHeartbeatRetryCount BrokerClient. 20,000 The instance of BrokerClient monitors the milliseconds connection to the HPC Broker Service on the (20 seconds) broker node. The broker client sends regular Retry count: 3 heartbeat probes to the broker service. The clientBrokerHeartbeatInterval specifies the number of milliseconds between probes. If the broker service does not respond, the client retries up to the number of times specified by the clientBrokerHeartbeatRetryCount. After the retry count is reached, a SessionException is thrown. In some cases, the client might try to create a new session (up to the job and task retry counts that are set for the HPC Job Scheduler Service).

Note: A subset of the broker monitoring settings can also be defined by using the SessionStartInfo.BrokerSettingsInfo class when the client application creates a session. Values provided in the SessionStartInfo.BrokerSettingsInfo class override the values that are specified in the service configuration file. For more information, see SessionStartInfo.BrokerSettingsInfo Class (http://msdn.microsoft.com/en- us/library/microsoft.hpc.scheduler.session.sessionstartinfo.brokersettingsinfo(VS.85).aspx).

Load balancing settings The following table describes the broker load balancing attributes and their default values:

Default Attribute Description values messageResendLimit The number of times that the broker will resend a 3 message. serviceRequestPrefetchCount The number of requests that can be pending for each 1 service instance. For example, if the prefetch count is set to 1, two requests are sent to each service instance. One request starts processing immediately, and the second ‘prefetched’ request starts processing when the first requests is finished. If the count is too low, nodes will be idle as they wait for incoming requests (for example, 0 or a small number for very short running requests). If the count is too high, then some nodes might become idle while there are unprocessed requests waiting on other nodes. serviceOperationTimeout The length of time that the broker waits for the service 86,400,000 to finish processing the message. milliseconds (24 hours) endpointNotFoundRetryPeriod If an EndPointNotFound exception is returned 30,000 when connecting to the service host, the broker waits milliseconds the amount of time specified by (30 seconds) endpointNotFoundRetryPeriod and then tries again.

Code sample: Service configuration file The following XML code sample shows the service configuration file for CcpEchoSvc. This sample demonstrates how to use the and the sections of the service configuration file. Note that additional elements are included to support the advanced configurations.

Backward compatibility with HPC Server 2008 registration files In Windows HPC Server 2008, configuration settings that applied to an SOA service were defined in three separate configuration files. In Windows HPC Server 2008 R2, all of the settings are defined in a single file: the service configuration file (servicename.config). Even the broker settings, which define how the broker handles service sessions, are defined at the individual service level.

In Windows HPC Server 2008 R2, the service configuration file can include the , , , and sections. (In Windows HPC Server 2008, these sections were defined in the HpcWcfBroker.exe.config (in the %CCP_HOME%Bin\ folder) and in the [serviceAssembly].dll.config files.)

The following table describes the configuration files for these settings in Windows HPC Server 2008 and in Windows HPC Server 2008 R2: Configuration section Configuration file in Windows HPC Configuration file in Server 2008 Windows HPC Server 2008 R2 microsoft.Hpc.Session.ServiceRegistra serviceName.config serviceName.config tion microsoft.Hpc.Broker HpcWcfBroker.exe.config serviceName.config (in the %CCP_HOME%Bin\ folder) system.serviceModel [serviceAssembly].dll.config serviceName.config system.Diagnostics [serviceAssembly].dll.config serviceName.config appSettings [serviceAssembly].dll.config serviceName.config

The service configuration files (servicename.config) that you used to register services on a HPC Server 2008 cluster are compatible with an HPC Server 2008 R2 cluster. You can use the same service registration files to register the service on an R2 cluster.

If you made any changes to the and the sections in the HpcWcfBroker.exe.config configuration file in Windows HPC Server 2008, you can copy these settings into the individual service configuration files for Windows HPC Server 2008 R2.

Note: In Windows HPC Server 2008 R2, the following broker monitor attributes are not supported:

 allocationGrowLoadRatioThreshold

 allocationShrinkLoadRatioThreshold Understanding the HPC Infrastructure for SOA A Windows HPC Server cluster that is configured to support SOA jobs consists minimally of a head node, a WCF broker node, and one or more compute nodes. This section describes how the features of HPC work together to run SOA applications.

Key terms The following table defines the key terms that are associated with running SOA applications on a Windows HPC Server 2008 R2 cluster:

Head node The head node is connected to the enterprise network and to the private and application networks for the cluster if they exist. The head node performs the following tasks:  Monitors the cluster  Coordinates job scheduling through the HPC Job Scheduler Service  Listens for session requests from client applications across the enterprise network  Maintains a list of WCF broker nodes and Message Queuing resources, and  Uses the HPC Session Service to balance work across the available WCF broker nodes  Provides the client with information for connecting to a broker node. HPC Job Scheduler Service Runs on the head node and performs the following tasks:  Maintains the job queue  Allocates cluster resources (nodes, sockets or cores) to jobs  Starts tasks on the nodes HPC Session Service Runs on the head node and performs the following tasks:  Maintains a list of the available WCF broker nodes and Message Queuing resources  Balances workload across the broker nodes.  Coordinates communication between the HPC Job Scheduler and the HPC Broker Service  Sends the client a list of available broker nodes when a client initiates a session WCF broker node  Communicates with the client application across the enterprise network  Monitors and manages sessions  Uses the HPC Broker Service to coordinate service requests and responses between the client and the service hosts HPC Broker Service Runs on each WCF broker node and performs the following tasks:  Listens for broker requests from client applications  Creates a broker instance (worker process) for each session. The broker instance performs the following tasks:  Manages and monitors the session  Receives requests from the client  Distributes requests to the service hosts on the cluster  Collects responses and sends them to the client  Stores the messages to Message Queuing if the session is a durable session Note: The session handling behavior, such as timeouts and error handling, can be adjusted through the settings in the service configuration file. HPC Service Host Runs on the compute nodes and performs the following tasks:  Loads the service DLLs  Communicates with the broker instance on the WCF broker node  Raises the ServiceContext.OnExiting event if the request is canceled  Collects tracing information from the service and logs it to a file if tracing is enabled Networks  Enterprise network: Handles traffic between the cluster and its clients. The head node and the WCF broker nodes must be connected to this network.  Application network (optional): If present, this is the default network for message traffic between the broker node and the compute nodes. If the application network is not available, the private network is used for application traffic. This can be changed with the WCF_NETWORKPREFIX environment variable.  Private network (optional): If present, this is the default network for traffic between the head node and the WCF broker node (otherwise this traffic goes through the enterprise network). Client application The client application provides an interface for the end user to access service functionality. The client application must be running on a computer that is in the same domain as the cluster head node, or be in a domain that has a trust relationship with the head node domain. The client application performs the following tasks:  Creates sessions on the cluster  Sends and receives service requests Service task  The job scheduler runs one instance of a Service task on each resource that is allocated to the job. Service task is one of the options in the Task Type job property. Interaction details To initiate a session on the cluster, the client application calls the CreateSession method. When calling this method, the client application specifies the head node name and the service name. Optionally, the client application can specify information such as the job template to use, the priority level, the number of required resources, and the resource type (node, socket, or core). This information can be specified in the SessionStartInfo object. With this information, CreateSession submits a job to the head node that includes a Service task. The Service task includes a command to start the service host and load the specified service.

When the head node receives the SOA job from the client, the following interactions occur:

 The HPC Job Scheduler Service validates the job, applies default values for job properties as defined by the job template, and adds the job to the queue.

 When resources are available, the HPC Job Scheduler Service allocates resources to the job, and it runs an instance of the Service task on each allocated resource. This starts the service host and loads the service DLL on the allocated compute nodes, so the service is ready to accept requests.

 The HPC Session Service sends the client a list of available WCF broker nodes. The CreateSession method attempts to connect to a broker node on the list. A broker node accepts the connection from the client, and the HPC Broker Service creates a broker instance (also known as worker process) to manage the session.

 The client sends requests directly to the broker instance on the WCF broker node.

 The broker instance detects the allocated service hosts from the running service task and begins sending service requests.

 The broker instance passes requests and responses between the client and the service hosts.

 If the client created a durable session, the broker uses Message Queuing to store all messages. Responses that are stored by the broker can be retrieved by the client at any time, even after intentional or unintentional disconnect.

Note: The HPC Job Scheduler Service and the broker instance work together to adjust the number of allocated service hosts periodically based on resource availability and the number of outstanding requests.

The following diagram illustrates this interaction. Broker nodes and session behavior The WCF broker nodes manage sessions on the cluster. The HPC Broker Service runs on the WCF broker nodes and listens for requests from client applications. HPC Broker Service creates a broker instance for each session. The broker instance manages and monitors the session as follows:

 Creates an end point for the application traffic for that session.  Finds out from HPC Session Service which service hosts are allocated to the job.  Accepts requests from the client.

 Distributes requests across the allocated service hosts.  Collects responses from the service hosts and sends them to the client.

 Stores requests and responses using Message Queuing if the session is a durable session, so that work is not lost in case of interrupted connections with the client.  Monitors the number of incoming requests and helps adjust resource allocation (grow/shrink) by setting and maintaining the Target Resource Count property for the job.

 Monitors service loading failures, and maintains the Excluded Nodes property for the job.

 Retries up to the specified number of times in the case of message failure.

The session handling behavior (such as timeouts and error handling) can be adjusted through the settings in the service configuration file. This means that the broker nodes can handle different services in different ways. A subset of the broker monitoring settings can also be defined by using the SessionStartInfo.BrokerSettingsInfo class when the client application creates a session. Values provided in the SessionStartInfo.BrokerSettingsInfo class override the values that are specified in the service configuration file. For more information, see SessionStartInfo.BrokerSettingsInfo (http://msdn.microsoft.com/en- us/library/microsoft.hpc.scheduler.session.sessionstartinfo.brokersettingsinfo(VS.85).aspx).

The HPC Job Scheduler Service and SOA sessions The HPC Job Scheduler Service runs on the head node. This service maintains the job queue, allocates cluster resources to jobs, and starts tasks on the head node. The HPC Job Scheduler Service can run in one of two modes: Queued or Balanced. Queued mode is optimized for MPI and batch jobs, Balanced mode is optimized for interactive workloads. For more information, see the Job scheduler configuration section in this document.

The cluster administrator can define job templates to control the allowable job properties for job types or user sets. Job properties include priority level and requested resources. When you are creating a client application, it is a good practice to check with the cluster administrator regarding specific job templates that should be used to start a session on the cluster, or provide a way that the end-user can specify an appropriate job template. For more information, see the Job templates section later in this document.

When an SOA client application calls the CreateSession method, a job that includes a Service task is submitted to the head node. The Service task runs a command or service on all resources that are allocated to the job. New instances of the command start when new resources are added to the job or if a previously running instance exits and the resource is still allocated to the job. A service task can create up to 1,000,000 subtasks (instances). Service tasks continue to start instances until the job is canceled or stopped, the maximum run time expires, or the maximum number of instances is reached.

The broker instance detects the allocated service hosts from the running service task and begins sending service requests. The broker instance manages the session, and sets and maintains the following job properties:

 Target Resource Count – For SOA jobs, this property acts as a dynamic maximum resource count, so the HPC Job Scheduler Service does not allocate more service hosts than can be used by the job. The broker node sets and maintains the Target Resource Count job property based on the number of outstanding messages for the session.

 Excluded Nodes List – This job property lists nodes that should not be used to run a particular job, for example, if the tasks in a job keep failing on a particular node. For SOA jobs, the broker instance sets and maintains the list of excluded nodes for a session. The broker instance adds a node to this list if the service cannot load on that node. The EndPointNotFoundRetryPeriod setting (in the service configuration file) specifies how long the service host should retry loading the service and how long the broker should wait for a connection. If this time elapses, the broker adds the node (service host) to the Excluded Nodes list. Broker node failover The servers in a failover clusters work together, so if one server in the cluster fails, another server in that cluster automatically begins providing service (in a process known as failover). To increase SOA runtime reliability, you can configure the head node in a failover cluster and then configure WCF broker nodes in separate failover clusters.

The following diagram shows a head node configured in a failover cluster that contains two nodes, plus a set of WCF broker nodes configured in a failover cluster that contains four nodes, three active and one passive (a passive node is inactive but ready to provide service if needed).

MSMQ MSMQ MSMQ MSMQ Quorum Disk Disk Disk Disk Disk

SQL Quorum MSMQ Tx MSMQ Tx MSMQ Tx MSMQ Tx Database Disk Disk Disk Disk Disk Disk SCSI, iSCSI, FibreChannel

Active HN Passive HN Active BN Active BN Active BN Passive BN Active CN Active CN Active CN

Application Network

Private Network

Enterprise Network

Client

For a failover cluster that supports the head node, the storage server contains the job scheduler and management databases. The HPC Job Scheduler Service can fail over to another server if a problem occurs. For the WCF broker nodes, the storage disk contains Message Queuing queues for durable sessions.

A cluster can also include WCF broker nodes that are not configured for failover. If the cluster has failover broker nodes and non-failover broker nodes, the broker nodes that are not configured for failover are only used to host non-durable sessions, and they are not connected to Message Queuing disks. One instance of Message Queuing runs on each active WCF broker node, and each instance requires its own disk in storage. Optimally, each Message Queuing instance has access to two disks: one for Message Queuing data file and one for Message Queuing transaction file. You can store the data and transaction files on a single disk, but that can affect performance. The Message Queuing instance can use a virtual hard disk (VHD) as its shared disk, so you can further consolidate by storing multiple VHDs on a single shared disk. The broker node can access shared disks through SCSI, iSCSI or FibreChannel networks. Managing WCF broker nodes and SOA sessions on the cluster This section provides an overview of how a cluster administrator can deploy WCF broker nodes and SOA services to the cluster, monitor nodes and sessions, and configure job scheduling policies.

Node roles When you install HPC Pack 2008 R2 on a server to create a head node, the head node includes the features and services for the broker node and compute node role. If the head node is not configured in a failover cluster (for high-availability), it can perform one or both of these additional roles. By default, the head node has all three roles enabled, so if you bring it to the Online node state, it will start performing all three roles. Running all three roles on the head node allows you to set up a development ‘cluster’ with a single server. Two roles on the head node can be useful to maximize resource usage in smaller clusters if you do not need an entire server dedicated to management and job scheduling.

Broker nodes include the features and services for the compute node role, so you can quickly switch between roles. For larger clusters that support SOA workloads, this allows you to scale your compute nodes and broker nodes as your workload requires. For example, the number of SOA sessions that your cluster supports might vary according to weekly or monthly business cycles, or might slow down at night and on weekends. You can enable more broker nodes when demand for sessions is high, and then convert them to compute nodes when demand slows down.

For information about how to change node roles, see multiple roles and broker scaling in the TechNet library.

Note: When the compute node role is enabled on a node, and the node is Online, the HPC Job Scheduler Service considers every core on the node as a potential job resource. The HPC Job Scheduler Service tries to keep the cluster as busy as possible, so it will attempt to allocate tasks to every core. If you enable compute node as an additional role, the head node and/or broker node functionality will have to share CPU time with running tasks. The feasibility of adding the compute node role depends on how busy the head node or broker node are, and how much work the tasks are doing.

Service Pack 2 includes the ability to specify the number of cores on a node that you want to make available to the HPC Job Scheduler Service. If you want to enable multiple roles on a node, you can use this feature to essentially limit the number of cores that will be used for the compute node role. For more information, see Over-subscribe or under-subscribe core or socket counts on cluster nodes.

The following table describes the features for each node role:

Node role Role features Head node  HPC Management Service  HPC Job Scheduler Service  HPC SDM Store Service  HPC Session Service  SQL Server  DHCP  Windows Deployment Server Service

Compute node  HPC Management Service  HPC Node Manager Service  HPC MPI Service  HPC Service Host WCF broker node  HPC Broker Service  Message Queuing

Setting up a WCF broker node You can use the head node as a WCF broker node, or you can deploy one or more nodes in the cluster as broker nodes. Before you add a new broker node to the cluster, you need to create a broker node template. When the head node detects new nodes on the network, the nodes appear in HPC Cluster Manager in the Node Management view. The cluster administrator approves the nodes and joins them to the cluster by assigning a node template.

Creating and using node templates Administrators use node templates to join nodes to a cluster. Node templates can define the necessary tasks for adding, configuring, and maintaining compute nodes in your cluster. With a node template, you can deploy an operating system image, add specific drivers and software to nodes, add a preconfigured node to your HPC cluster, or apply software updates to your nodes.

You can use the node template wizard in HPC Cluster Manger to create broker node, compute node, or workstation node templates. When the template is assigned, the corresponding node role is enabled automatically.

If you want to deploy a WCF broker node from bare metal, you can create a broker node template that includes an associated operating system image. When you assign the template, the Windows Deployment Server Service will install the operating system on the node, and any additional deployment steps that you have specified in the template will be performed. If you want to add a server that has already been configured as a WCF broker node, you can create a broker node template that does not include an associated operating system image. When you assign the template, any deployment steps that you specified in the template will be performed.

Preconfiguring a broker node You can install HPC Pack 2008 R2 directly on a server to create a preconfigured broker node. To perform as a WCF broker node, a server must meet the following requirements:

 Have Windows Server 2008 R2 HPC Edition or another edition of the Windows Server 2008 R2 operating system installed

 Be connected to the enterprise network

When you install HPC Pack 2008 R2 on a server, you have the option to set up a head node, a compute node, or a broker node. The selection that you make in the wizard determines the features that are installed on the node. When you select the option to join a cluster by creating a WCF broker node, the installation wizard includes a step to specify the head node of the cluster. When the installation completes, the new node appears in the Node Management list view as WCF Broker Node in the Unknown state. You can then approve the node and add it to the cluster by assigning a broker node template that does not include an operating system image.

Deploying and registering services There are two components to services that you deploy to your cluster: dynamic-link libraries (DLLs) and the service registration file .

Installing DLLs The following table describes the three options that are available for installing service DLLs:

Deployment Description Option

Copy the service DLL to any folder on the root directory of each node in the cluster (for Local example, C:\Services). This option yields the best performance, but updating the DLLs deployment can be time-consuming in a large cluster, especially if all the nodes are not online at the same time. Copy the service DLL to a shared folder in the cluster. This option makes it easy to update the DLLs; however, it may result in longer load times if the DLLs are large. Central deployment You also need to set up the .NET security permissions. For more information, see the Code Access Security Policy Tool (Caspol.exe) (http://go.microsoft.com/fwlink/? LinkId=122770) on MSDN.

Hybrid Copy large DLLS that are not updated often to the local nodes, and copy small or more deployment frequently updated services to a shared folder.

Package together the service DLLs, the service configuration file, and any dependent DLLs in a format that can be uploaded to a Windows Azure storage account, and then upload the package to the Windows Azure storage account (you can use the hpcpack command line utility to do this). When you provision a set of Windows Azure nodes from HPC Cluster Manager, any packages that you staged to the storage account are automatically deployed to the Azure nodes.

SOA services on Azure nodes must be deployed to the default package deployment Windows location. This location is partially determined by the %CCP_PACKAGE_ROOT% Azure node environment variable, which is set on Windows Azure nodes during the provisioning deployment process. The full path is determined as follows: %CCP_PACKAGE_ROOT %\\.

Important: When you create the package, ensure that the package name is the same as the SOA service name (that is, the service name that the SOA client specifies in the SessionStartInfo constructor). This ensures that the service host can find the service DLLs.

For more information, see Deploying Applications to Azure Nodes.

The service registration file for the deployed service specifies the location of the DLL (this tells the service host where to find the DLL). Ensure that the assembly attribute of the service element points to the location where you installed the DLL. This can be a path to a shared folder on the cluster, or it can be a local path on each compute node. For deployment to Windows Azure nodes, you do not need to modify the assembly attribute. On Windows Azure nodes, the service host looks for the DLL in the default package deployment location (determined by the %CCP_PACKAGE_ROOT% environment variable and the package name).

Registering the service After you have installed your DLLs, you can install the service configuration file in a shared folder, or you can register the service locally on each node. Central service registration lets you to maintain a list of services that are deployed on the cluster, and manage configuration and troubleshooting settings from a single location.

For detailed information, see the section in this paper.

The following table describes the options that are available to register the service: Registration Options Registration Methods

Central registration Services configuration files that are deployed to a central location can be managed in HPC Cluster Manager. You can use the default Service Registration folder on the head node, for example:

C:\Program Files\Microsoft HPC Pack 2008 R2\ServiceRegistration

This folder is shared as:

\\\HPCServiceRegistration\

The central service registration folder can be any shared folder that the head node, broker nodes, and compute nodes can access and read. Additionally, the cluster administrators must have full control permissions on the shared folder so that configuration files can be installed to that path. You can change the default location of the Service Registration folder by setting the cluster environment variable, CCP_SERVICEREGISTRATION_PATH. You can specify multiple paths separated by a semicolon (;). This variable works the same as PATHS works in cmd.exe. For example, at a command prompt, type the following command:

cluscfg setenvs CCP_SERVICEREGISTRATION_PATH=\\\\

The central service registration folder can be any shared folder that the head node, broker nodes, and compute nodes can access and read. The cluster administrators must have full control permissions on the shared folder so that configuration files can be installed to that path

Local registration To register the DLL locally on each node, install the completed registration file in the service registration folder located at %CCP_HOME %\ServiceRegistration.

If you are using local deployment, the registration files must also be copied to the WCF broker nodes.

Registration on Windows To register a service that will run on Windows Azure nodes, you must Azure nodes place a copy of the service configuration file on the on-premises cluster in a location that is accessible to the head node and the WCF broker nodes (such as the Service Registration folder on the head node).

Additionally, you must package a copy of the service configuration file along with the service assembly and any dependent DLLs that you deploy to the Windows Azure storage account.

Note: To change service configuration settings, you only need to update the on-premises copy of the service configuration file.

Centrally managing and troubleshooting the service configuration file In HPC Cluster Manager, in Services, you can review a list of all the SOA services that have centrally deployed configuration files. By default, the Services view displays the services that have configuration files in the Service Registration folder on the head node (by default, C:\Program Files\Microsoft HPC Pack 2008 R2\ServiceRegistration). The Services list view does not display service configuration files that are locally deployed on compute nodes.

When you centrally install the service configuration files, the Services management pane provides the following actions to help you manage and troubleshoot the files:

 Open an XML file editor to edit the configuration files for the SOA services (for example, edit or verify the location of the service DLL)

 Configure the type and level of the events that are logged for each service (this action is intended for development clusters, and it is not recommended for production clusters)

 Specify a folder where job errors will be saved to a text file

The service configuration file opens in the default XML editor on your head node. Depending on the software that is installed on your head node, the default XML editor can be the WCF configuration file editor or a program such as Notepad, Visual Studio, or XML Notepad 2007. To download XML Notepad 2007, see XML Notepad 2007 in the Microsoft Download Center (http://go.microsoft.com/fwlink/?LinkId=139128).

Cluster administrators must have write permissions to the service configuration files to make changes to the service registration and configuration settings.

Important: If you use the WCF configuration file editor on a computer that has the Windows 7 operating system to edit the service configuration file, the extendedProtectionPolicy property is added to the file. This property is supported only on the Windows 7 operating system (Windows HPC Server 2008 R2). If your compute nodes have Windows HPC Server 2008 installed, you should remove this property from the service configuration file.

The following screenshot shows the Services view in HPC Cluster Manager (version 3.2):

Note: CcpEchoSvc is a simple, built-in service that can be used with the diagnostics tests to verify SOA functionality on the cluster. The other services you see in the screenshot are the HPC Services for Excel. These services support the various solutions for accelerating Excel workbooks with a Windows HPC cluster and the SOA framework. For more information, see HPC Services for Excel (http://technet.microsoft.com/en-us/library/ff877820(WS.10).aspx).

Diagnostic tests Windows HPC Pack 2008 R2 includes a built-in diagnostic test named SOA Service Loading Diagnostic Test. When you run this test, you can specify a set of nodes and a service. The test verifies that the DLLs for the specified service can be loaded on the specified nodes, and that any detected dependencies for the DLL are present on the compute nodes.

The following screenshot shows the Choose Tests and Nodes tab in the Run Diagnostic Tests dialog box. When you right-click a service name to open the Run Diagnostics Tests dialog box, that service is automatically specified in the test parameters. If you open the dialog box from the Node Management or Diagnostics views, you can specify a service name in the test parameters. If you do not specify a service, the built-in sample service CcpEchoSvc is used to verify SOA functionality on the specified nodes. The following screenshot shows the Configure Test Parameters tab in the Run Diagnostic Tests dialog box: Note: Windows HPC Server 2008 R2 provides an extensible diagnostics framework that lets you add custom diagnostic tests to your cluster. This allows you to run your own tests and the built-in tests from a single location by using HCP Cluster Manager or HPC PowerShell.

Monitoring broker nodes and sessions HPC Cluster Manager provides customizable dashboards that you can use to monitor cluster nodes. The following table describes the available metrics that apply to WCF broker nodes:

Name Description HPC SOA Calculations/Sec Current calculating calls from the broker node. This is a sliding average of the past N seconds. Updated by broker node. This value can be significantly higher than the number of cores due to caching on the service host. HPC SOA Faults/Sec Faulted calls on the node per second. Updated by broker node. HPC SOA Requests/Sec Number of requests to the broker node per second. Updated by broker node. HPC SOA Responses/Sec Number of responses from the broker node. This is a sliding average of the past N seconds. Updated by broker node. Durable Requests Queue Number of total requests stored in local Message Queuing. Length Durable Responses Queue Number of total responses stored in local Message Queuing. Length Durable Queues Total Total number of bytes of Message Queuing messages on the broker node. Bytes Durable Queues Total Total number of Message Queuing messages on the broker node. Messages Free Disk Space (%) Percentage of total usable space on the local disk.

You can monitor sessions in the Job Management view in HPC Cluster Manager. The job ID is the same as the session ID. You can choose the visible columns in the list view to display job information such as the number of incoming calls, the number of calls per second, the number of outstanding calls, and the current allocation.

In the job Progress column, you can see the percentage of the jobs that are complete. The Progress information is useful if you have a specific number of requests. However, if requests are continuously coming in and being processed, the progress percentage is less informative.

The following screenshot shows the Job List view in HPC Cluster Manager:

You can double-click a particular session to get more details. The following screenshot shows the information on the Session Progress tab in the Job View dialog box: Managing Message Queuing When an SOA client creates sessions on the cluster by using durable sessions, the WCF broker node persists messages by using Message Queuing. By default, Message Queuing storage quota limit is 1 GB. Depending on the message size and the number of durable sessions that are running on the cluster, you might need to increase this limit. The amount of disk space that messages for durable sessions use can be calculated as follows:

Required disk space = (number of messages) x (number of active sessions) x (message size) During normal operations, messages are deleted from Message Queuing in the following ways:

 Clients can cause Message Queuing to delete the messages associated with their session when they close the session.

 Messages associated with finished sessions are deleted automatically from Message Queuing after the number of days specified in the Job History settings in the job scheduler configuration options. By default, records for completed jobs are kept for five days.

You can configure Message Queuing in Server Manager under Features:

Scheduling policies The cluster administrator defines scheduling policies by:

 Configuring the Job Scheduler properties such as backfilling, preemption, and scheduling mode. Scheduling mode can be optimized for interactive jobs (Balanced mode) or for batch and MPI jobs (Batch mode).

 Creating Job Templates that define default and permissible job property values for different types of jobs and the users who are allowed to submit jobs with that template.

 Creating custom programs called Job Submission or Job Activation filters to further control jobs that are submitted and started on the cluster. For example, a Job Activation filter can be used to check for license availability before a job is started. Job scheduler configuration The job scheduler policy configuration settings control how resources are allocated to queued jobs. The Scheduling Mode lets you optimize resource allocation for large batch and MPI workloads or for service workloads. The following table summarizes the Scheduling Mode options and their default configurations:

Queued mode (default) Balanced mode

 Interactive workloads, such as service-oriented architecture  Large MPI and batch jobs (SOA) jobs and service-type Optimized  Long running tasks applications for  Stateful tasks (rely on state)  Short running tasks  Parametric sweeps  Stateless tasks (do not rely on state)  Parametric sweeps  Starts all incoming jobs as soon as  Starts jobs in queue order possible  Finishes highest priority jobs as soon as  Gives jobs their minimum possible requested resources  Gives jobs their maximum requested  Allocates additional resources to resources jobs based on priority  Minimizes job run time Description  Minimizes wait time in the job  Causes a longer wait in the job queue for queue lower priority jobs  Takes jobs longer to complete  Uses available resources to first meet the  Can take resources from running maximum resource request of running jobs to start new jobs, even if the jobs, and then to start new jobs (when running job has a higher priority automatic growth is enabled)

 Preemption Default: Graceful preemption  Priority bias  Adaptive resource allocation Default: Medium Bias Additional (grow/shrink)  Rebalancing interval settings Default: Increase and decrease resources Default: 10 seconds automatically both enabled

If your cluster is going to be running primarily interactive workloads, you can change the Scheduling Mode to Balanced mode. In Balanced mode , the HPC Job Scheduler periodically rebalances resources according to the following settings:

Priority Bias setting: Controls how additional resources are allocated to jobs. In Balanced mode, “additional resources” refers to cluster resources above the total minimum resources for all running jobs. Tasks that are running on additional resources can be canceled with immediate preemption to accommodate new jobs or to converge on the desired allocation pattern. Priority Bias has the following options:

 High Bias: All additional resources are allocated to higher priority jobs.

 Medium Bias (default): Each priority band is given a higher proportion of additional resources than the band below it. The priority bands are Highest, Above Normal, Normal, Below Normal, and Lowest.

 No Bias: Additional resources are allocated equally across the job queue.

Note: Job priority can be defined by the client application when the job is submitted, and it can be regulated through job templates that are defined by the cluster administrator.

Rebalancing Interval setting: Represents the time, in seconds, between job scheduler rebalancing passes. The default value is 10 seconds. A longer interval can improve scheduler performance, but it can take longer to respond to new jobs and converge on the desired allocation pattern. Longer intervals are good if you do not need instant growing and shrinking. If your cluster has a high turnaround rate (jobs are submitted frequently and finish quickly), you might want a longer interval to avoid excessive growing and shrinking. A shorter rebalancing interval provides a faster response when new jobs are submitted, at the cost of additional load on the head node. The other settings that you can adjust if you need faster responses are the Task Cancel Grace Period and the Release Task Timeout, which can cause it to take longer for running work to get pushed out of the way.

Job templates Job templates are required to submit jobs. A default job template is provided. However, if you want to define custom submission policies for your cluster, you can create additional job templates. For example, you can specify the priority levels for different types of jobs or for different groups of users.

A job template contains the following:

 An access control list (ACL) that defines the users who have permission to submit jobs using the job template.

 A list of job properties and associated value settings. For each property, the administrator can define:

 Default values

 Value constraints

A job template allows you to associate a set of default values and value constraints for job properties with a particular set of users. Examples of policies that you can enforce with job templates include:

 Enforce run time and size limits on jobs  Limit job priority for different users or job types

 Send different types of jobs to different sets of nodes

 Partition the cluster for different user groups

 Define which types of jobs can or cannot be preempted

 Define which types of jobs can enable job exclusivity

 Simplify job submission for the user

 Customize the job submission process to support different user scenarios

If you create specific job templates to regulate SOA job scheduling, you can work with the application developer to ensure that the correct job template is specified when the session is created. If you cannot have the appropriate job template specified directly by the client application, you can write a job submission filter program to evaluate incoming jobs and assign templates or modify job properties according to criteria of your choosing.

After the job is submitted, the HPC Job Scheduler Service applies the job template as follows:

 Confirms that the user has the Submit jobs permission for the specified job template. If not, the job is rejected and the submission fails. This enables an administrator to control the types of jobs that a select group of users can submit.

 Applies default settings from the job template for any job properties that the user did not specify.

 Validates each job property against the constraints that are defined in the job template. If any job property is not within the valid range that is specified in the template, the job is rejected and the submission fails.

If the job passes all checks, it is added to the job queue.

Additional References  Technical library for Windows HPC Server 2008 R2

 White papers for Windows HPC Server 2008 R2

 Developer Resources for Windows HPC Server 2008 R2  HPC Class Library

 HPC Cluster Debugger for Visual Studio 2008 and Visual Studio 2010

 Accelerating Excel 2010 with Windows HPC Server 2008 R2: Technical Overview  Diagnostics Extensibility in Windows HPC Server 2008 R2 Step-by-Step Guide

 Fundamental Windows Communication Foundation Concepts

 Windows Communication Foundation