Optimizing Borland® StarTeam® for distributed teams
The best techniques for supporting distributed teams
A Borland White Paper
By Randy Guck, Chief Scientist Borland Software Corporation
January 2005
Optimizing Borland® StarTeam® for distributed teams
Contents
Overview ...... 3
The distributed team dilemma: Centralize or replicate? ...... 3
Distributed teams and network concerns ...... 3 Replication: Advantages and concerns ...... 4 Centralized repositories: Advantages and concerns...... 5 Is replication dead? ...... 7
Distributed teams the StarTeam way ...... 7
Standard client/server features for distributed teams ...... 8
Command API compression ...... 9 Delta checkout ...... 11 Auto-reconnect...... 13
StarTeamMPX: Turning the network inside-out ...... 15
StarTeamMPX Cache Agent: Distributed file caching...... 20
Basic Cache Agent architecture ...... 20 Stacked Cache Agents...... 23 Cache Agent-aware StarTeam clients ...... 25 Using Cache Agents with the Cross-Platform Client...... 26 Using Cache Agents with the bulk checkout utility...... 27
StarTeam Import/Export Manager ...... 28
Summary: The world according to StarTeam...... 32
Glossary ...... 34
References ...... 36
2 Optimizing Borland® StarTeam® for distributed teams
Overview
Software development teams are increasingly becoming spread around the globe. Organizations are leveraging new talents, time zones, and tools made possible by a networked world. If you work on a distributed team, how should you manage lifecycle tools with shared repositories such as Borland® StarTeam®? If you centralize files, change requests, and other application lifecycle management (ALM) artifacts, how can you address performance and reliability? If you replicate artifacts to distributed sites, how do you handle synchronization and conflicts?
This white paper describes the best techniques for supporting distributed teams with StarTeam. The problems with replication are discussed, and the case for supporting your organization with centralized StarTeam repositories is made. Features with which you can optimize StarTeam for distributed teams, including new features introduced with StarTeam 2005, are described. These features address responsiveness, limited network bandwidth, and even unreliable networks.
The distributed team dilemma: Centralize or replicate?
This section examines the basic dilemma that organizations face when deploying a repository- centric client/server application such as StarTeam. Basically, the question boils down to: Can you centralize the repository and provide adequate performance to remote teams? Or should you deploy multiple repositories and attempt to replicate and synchronize between them?
Distributed teams and network concerns
So, your organization has decided on StarTeam to manage core development processes. Congratulations! But now you have to figure out how to make StarTeam work for your distributed team. You have one group in Chicago, another in Europe, and a third group in
3 Optimizing Borland® StarTeam® for distributed teams
India. All of your developers require access to the same projects, views, files, change requests, and tasks. And, to complicate matters, the transatlantic VPN connection between sites isn’t highly reliable; there are occasional outages ranging from minutes to hours. How do you deploy StarTeam to maximize distributed team productivity while avoiding administrative headaches?
Geography often creates its own challenges, but sometimes network issues arise even when distance is not a factor. In sufficiently large organizations, LANs and WANs can experience periods of congestion or instability, raising similar issues as those experienced by geographically dispersed teams. In both cases, team members need access to the same application lifecycle artifacts but are faced with network latency, bandwidth, and reliability issues.
Would you rather centralize your repository or distribute multiple copies and synchronize them? Let’s examine both approaches in more detail.
Replication: Advantages and concerns
Some repository-centric systems recommend that the way to address distributed teams is to replicate the repository between remote sites and continuously synchronize them. By providing a “network near” repository instance, remote teams experience two primary benefits:
• Performance: Accessing a local repository over a local, high-speed LAN generally performs better than accessing the same information over a slower, long-distance network.
• Accessibility: Except for the inability to synchronize, a local repository will be unaffected by the unavailability of long-distance access. Hence, local teams can access artifacts even when the WAN, VPN, or ISP connection is down.
However, organizations that attempt to implement replication eventually discover that there are many costs, some of which are not obvious at the outset:
4 Optimizing Borland® StarTeam® for distributed teams
• Administrative costs: Each replicated server requires its own administration, backup, monitoring, and other maintenance. You might find that you need repository, security, database, and other expertise at each remote site.
• Artificial merge conditions: Replication forces artificial merge conditions in cases in which you wouldn’t otherwise want it. Two developers working on the same file might introduce conflicting changes. Whereas they might otherwise avoid merge conflicts via locking, with separated repositories, they will not realize that they are introducing a conflict. This creates more work during synchronization, lengthening the development cycle.
• Network demands: Replication creates its own bandwidth and reliability issues. As more replicated repositories are deployed and project sizes grow, network demands increase (which is the reason that replication was deployed in the first place!) And if available bandwidth can’t keep up with the constant replication…think house of cards in a strong wind.
• Other costs: Replication means extra server hardware, licenses, redundancy, backup media, and more.
But suppose a centralized repository performed well even over long-distance, bandwidth- constrained networks? What if developers could use locally cached data, with less reliance on online access to the repository? What if you could get replication’s benefits without the cost?
Centralized repositories: Advantages and concerns
Most organizations prefer centralized repositories. Why? Among the advantages of centralized repositories:
• Overall cost: With a centralized repository, your most expensive infrastructure costs—servers, high-capacity storage, high availability/failover support—are deployed in one place. Furthermore, your administrative expertise can be concentrated in one place.
5 Optimizing Borland® StarTeam® for distributed teams
• Security control: With a centralized repository, it is much easier to perform administrative security tasks such as user and group management.
• Process control: Workflow rules and repository customizations can be managed in one location. Artifacts can be locked in one place, when needed, to synchronize updates across the entire enterprise.
• Backups: Comprehensive backup procedures can be more easily managed when they occur in a central location. Recovery and disaster-recovery plans are easier to manage with fewer “moving parts” in the overall infrastructure.
• Upgrade control: New product versions can be installed and the repository can be upgraded at a single installation.
At the same time, centralized repositories introduce challenges for distributed teams. There are two primary concerns:
• Performance: How can you provide adequate performance to remote teams, which might be constrained by network bandwidth? The more users and the more lifecycle artifacts you have, the higher the demands on long-distance network segments.
• Accessibility: The longer the “wire” between two sites, the less reliable it becomes. During a “network brownout”, how do you keep remote teams productive until connectivity is restored?
We’ve tipped our hand that StarTeam promotes centralized repositories over replication. Just to whet your appetite a bit, here’s an interesting fact: real customer benchmarks have shown that StarTeam’s performance over a global network can be an order of magnitude faster than competing products performance over a local network. How can StarTeam do that? Read on…
6 Optimizing Borland® StarTeam® for distributed teams
Is replication dead?
Almost. Other products recommend or require replication to address product scalability and performance shortcomings that do not exist with StarTeam.
However, there are valid use cases why you might need to copy all or a portion of a StarTeam repository to another location, where it is loaded into another repository. StarTeam provides a tool for this purpose, known as the StarTeam Import/Export Manager. Later, we examine this tool and the use cases for which it is suited. But for normal StarTeam usage scenarios, replication will not be a term in your vocabulary.
Distributed teams the StarTeam way
Behold the fool saith, “put not all thine eggs in one basket” — which is but a manner of saying, “scatter your money and your attention”; but the wise man saith, “put all your eggs in the one basket and—WATCH THAT BASKET.” — Mark Twain
The StarTeam approach for supporting distributed teams is to promote centralized repositories and attack the challenges of distributed teams with innovative technologies. For example:
• StarTeam provides options that minimize the amount of network bandwidth used by clients.
• StarTeam auto-reconnect silently handles minor network glitches and outages.
• StarTeamMPX uses publish/subscribe messaging to asynchronously push updates to clients before they are requested, reducing the need for clients to poll, refresh, or otherwise pull updates from the server.
• New for StarTeam 2005, StarTeamMPX Cache Agents provide a new method for pushing file revisions to distributed teams, providing alternate checkout sources and superior performance regardless of geography.
7 Optimizing Borland® StarTeam® for distributed teams
These features are discribed more detail in the next sections.
Standard client/server features for distributed teams
StarTeam uses a standard client/server architecture as shown below:
StarTeam StarTeam Client Server Command API
StarTeam VauVaulltt DB Client
StarTeam Client
Figure 1: StarTeam client/server archietcture
A StarTeam repository instance, also known as a configuration, consists of a vault and database, managed by a single StarTeam Server process. StarTeam clients access repository artifacts by connecting to the StarTeam Server using a TCP/IP protocol known as the command API. Clients send command requests to the server and receive command replies in response. As with any client/server architecture, several issues arise as the number of clients or the distance between clients and the server grows:
• Network latency: If a client requires frequent interaction with the server, responsiveness drops as network latency increases.
8 Optimizing Borland® StarTeam® for distributed teams
• Network bandwidth and congestion: As more clients access the server over the same network path, responsiveness drops. Client/server interactions that require large message sizes amplify bandwidth requirements. As available network bandwidth becomes constrained, the more likely that network congestion will occur, causing a backlog of outstanding messages and a further drop in client responsiveness.
• Network outages: The more a client depends on continuous network connectivity, the more susceptible the client becomes to network outages. Even short-term outages (“glitches”) can force clients to reconnect, reestablish work context, and restart previous operations. Longer-term outages can render a remote team completely unproductive.
The first measure with which StarTeam addresses network issues is in the design of the command API. As a binary TCP/IP protocol, the command API uses smaller request/reply messages than comparable text-based protocols. Additionally, you can reduce the network traffic for specific clients or mitigate network glitches for all clients with the options defined next.
Command API compression
You can select command API compression on a per-client basis, which reduces the size of all command API messages for that client. For graphical clients, you select the compression option for each configuration via the Server Properties dialog box:
9 Optimizing Borland® StarTeam® for distributed teams
Figure 2: StarTeam compression option
Other clients such as the StarTeam Command-Line Utility (stcmd) have comparable options to enable compression. When you enable compression, message sizes are condensed up to 80%, significantly reducing network bandwidth requirements.
When should you enable command API compression? Compression obviously requires more CPU time from both the server and the client. Consequently, the total “wall clock” time of a command might not be reduced if the savings in network transfer time do not compensate for the increased CPU time. As a result, command compression is somtimes not beneficial to users on high-speed networks (e.g., 100 megabits or better). However, consider these factors:
• Modern CPUs are faster than ever before, especially for computationally-intense algorithms. New optimizations within Intel® Pentium®-class processors have significantly reduced the CPU time required for compression and decompression routines.
• Even when the compression results in a negligible overall impact on an individual client, the overall reduction in network traffic benefits the greater network community.
• StarTeam 2005 uses a new intelligent compression strategy for file checkout. With the 2005 StarTeam Server, the new vault format used for storing files automatically chooses the best storage and checkout transfer format based on each file’s
10 Optimizing Borland® StarTeam® for distributed teams
compressibility. Files that compress well are stored and transferred in compressed format during checkout and decompressed by each client, regardless of the client’s command API compression option. Similarly, files that do not compress well are stored and transferred uncompressed during file checkout, regardless of the client’s compression option. The new file check-out strategy for StarTeam 2005 has two positive consequences:
1. Files are never “double compressed”, and poorly compressible files are never compressed, regardless of client options. This eliminates wasted compression/decompression cycles at both the client and server.
2. Files that compress well are always sent in compressed format with no compression overhead incurred by the server. The client must incur decompression overhead, but decompression is faster than compression. Hence, file checkout minimizes both network bandwidth and overall CPU time.
Note that the new file checkout strategy is used only when both the StarTeam client and server are version 2005 or higher. (Pre-2005 clients can use a StarTeam 2005 server, but will use the pre-2005 checkout strategy.)
The bottom line? If you are using an older client machine (e.g., Pentium III class) and are accessing the StarTeam server on a high-speed network (100BaseT or higher), don’t enable command API compression. If you are using a newer client machine (e.g., Pentium IV class) or are accessing the StarTeam server over a more limited bandwidth network connection, enable compression. You should see a significant boost in responsiveness.
Delta checkout
Not surprisingly, one of the most bandwidth-demanding operations is file checkout. Even with the StarTeam 2005 intelligent compression strategy described above, checking out a large number of files can require a lot of data transfer.
Most StarTeam users maintain a working set of files on their local disks, allowing them to check out only the new file revisions they need to get caught up. Suppose, for example, that
11 Optimizing Borland® StarTeam® for distributed teams
you are working on a project for which StarTeam tells you that you have 75 “out of date” files. This means that you have an older revision of each file on your local drive, but you need a newer revision to get caught up. Normally, when you select those files and perform a check- out, StarTeam sends you each new file as a “full” revision. For high-speed to broadband network connections, normal file checkout will give you satisfactory performance. (If you’re a new StarTeam user and have doubts based on your experience with other SCM systems, you might be very surprised by how well StarTeam file checkouts perform, even over long distances.)
But what if network bandwidth is very tight or congestion is a constant concern? StarTeam provides an option to check out newer file revisions with a different strategy called “delta checkout”. Instead of sending full file revisions, delta checkout causes “delta” records to be sent by the StarTeam server that allow the existing revision of each file to be “patched” to the revision you’re checking out. Because delta records are usually much smaller than the whole file revision, they require even smaller file checkout messages.
Delta checkout is enabled individually at the client level. Via the graphical StarTeam clients, select Tools Æ Personal Options Æ File, which brings up the following dialog box:
Figure 3: StarTeam slow connection optimization option
Check the box labeled “Optimize for slow connections” to enable delta checkouts. Note that before the StarTeam 2005 release, this option was available only for the StarTeam Win32
12 Optimizing Borland® StarTeam® for distributed teams
client. With the 2005 release, you can select this option for the StarTeam Cross-Platform client as well.
Here are some other notes to consider to decide if delta check-outs will help you:
• Delta checkout helps only when you have an older version of a file on disk. It does not help for files that are missing. If you select “Optimize for slow connections”, StarTeam will use it when it can, but it will send full file revisions if delta checkout is not possible.
• Delta checkout is possible only with text-based files. Binary files are always checked out by sending full file revisions.
• Delta checkout requires more CPU time and disk I/O than normal (full version) checkout because the existing work file must be “patched” up to the selected version. Hence, you should use this option only when network bandwidth is highly constrained (think modem kind of bandwidth).
• Delta checkouts are not used, nor are they necessary, when you are checking out from a StarTeamMPX Cache Agent (described later).
Auto-reconnect
With modern network topographies, brief network outages are common. A TCP/IP network connection can be lost for a variety of reasons:
• Firewalls: You’ve been a good network citizen by using StarTeam, minimizing network bandwidth, and, if you’re using StarTeamMPX, eliminating it completely in some cases. What is your reward for your contribution to the network ecology? The firewall closes your connection because you haven’t sent any messages for a while. No good deed goes unpunished.
• Wi-Fi: Wireless networking (i.e., 802.11 or “Wi-Fi”) has sufficient throughput and has become so inexpensive that it’s become widely used. If you don’t have Wi-Fi in
13 Optimizing Borland® StarTeam® for distributed teams
your home yet, just wait a few weeks and your kids will set it up for you. And then you get to discover what happens when someone turns on the microwave oven: you lose your Wi-Fi connection and your StarTeam connection along with it. (“Do you have to heat up a burrito at 11 p.m.?”)
• Poltergeists: Let’s face it, the network guys can’t even explain why, but every once in a while connections just get dropped. It’s the router, or the ISP, or the VPN, or the fact that someone ran a network cable too close to the elevator motor (true story!) Whatever the reason, these glitches are temporary but annoying.
StarTeam can silently recover from dropped network connections by enabling auto reconnect. Enable this option at the server level, and it becomes available for all users of that server. The option can be found via the StarTeam Server Administrator: Configure Server Æ General tab:
Figure 4: StarTeam auto reconnect setting
When you enable it, you must also enter a reconnect timeout value. This tells the StarTeam server how long to remember session state for clients whose connections were ungracefully lost (i.e., they didn’t log off). If the client reconnects within the specified time, the old session will be reestablished automatically. If the client does not reconnect within the specified timeframe, the session state is tossed, and the client will have to log on again the next time it uses the server.
14 Optimizing Borland® StarTeam® for distributed teams
What do you see from the client? In most cases, nothing. When a StarTeam client connects to a StarTeam server, it detects that auto-reconnect is enabled. Thereafter, if the connection is lost, it silently reconnects as needed. If you attempt an operation and the connection has been lost, you may see a slight pause while the connection is reestablished. If the connection can not be reestablished, you will receive an error message to that effect, and you will have to log on again. Note that auto-reconnect was available to the StarTeam Win32 client in the 6.0 release. With StarTeam 2005, auto-reconnect is available to the Cross-Platform Client, the StarTeam SDK, and all applications that use the SDK.
StarTeamMPX: Turning the network inside-out
So far, we have discussed techniques for minimizing network traffic and recovering from minor network traffic. However, we are still in the realm of request/reply communication, wherein each client pulls information from the server as it is needed. What if there were a way to push information to the client before it even asks for it, thereby dramatically reducing the number of requests made? This is the purpose of StarTeamMPX.
15 Optimizing Borland® StarTeam® for distributed teams
The basic StarTeamMPX architecture is shown below:
StarTeam StarTeam Client Server
Event publish stream
StarTeam Message VaultVault DB Client Broker
StarTeam Client
Figure 5: StarTeamMPX architecture
The major component added by StarTeamMPX is the Message Broker, which acts as a publish/subscribe communication gateway. StarTeam clients still establish a TCP/IP command API connection to the StarTeam Server in order to logon and retrieve initial objects. During logon, clients detect that MPX functionality is available and consequently “subscribe” to update event topics of interest. In turn, the Message Broker receives update events from the StarTeam Server and publishes them to all interested clients.
How does MPX benefit remote teams?
• Updated objects are asynchronously pushed to clients at update time. Clients receive relevant updates faster than they would if they used polling or refresh strategies. When multiple Message Brokers are used (e.g., one in each geographic region), each update event traverses long-distance wires only once, instead of being pulled by multiple clients, potentially multiple times. As a result, overall network usage is decreased.
16 Optimizing Borland® StarTeam® for distributed teams
• When a StarTeam client is running with MPX enabled, it eliminates certain client/server requests. For example, because it knows that new objects are automatically received, a client will no longer perform certain look-up and refresh commands. Try this experiment: press F5 when a StarTeam client is running in MPX mode and watch your network activity icon: you won’t see any network traffic at all.
• In MPX mode, the overall request/reply (command) count is cut by up to 50% per client. By decreasing the per-client cost, MPX increases server scalability, providing better overall response without changing hardware.
• Studies have shown that MPX “flattens out” request storms and demand peaks. Consequently, network congestion is reduced, and responsiveness is more predictable.
Your StarTeamMPX deployment strategy should follow StarTeam’s centralized repository strategy: deploy a “core” Message Broker at the centralized StarTeam repository site. Then, deploy one remote Message Broker in each major geographic area, chained to the core Message Broker. What constitutes a “major geographic” area? Any remote team with five to 10 or more users who access the central repository (keeping in mind that you can have a maximum of ten Message Brokers in a single messaging “cloud”.) This hub-and-spoke MPX blueprint is illustrated below:
17 Optimizing Borland® StarTeam® for distributed teams
MB
MB
ST Server and MB MB
MB
Figure 6: StarTeamMPX deployment example
To use MPX from a client, you must first enable it. In the graphical StarTeam clients, the option is found via Tools Æ Personal Options Æ StarTeamMPX Server, as shown below:
Figure 7: StarTeamMPX option
See the MPX references cited at the end of this article to learn more about the benefits of (automatic refresh.)
18 Optimizing Borland® StarTeam® for distributed teams
Normally, the StarTeam administrator creates a default MPX profile that directs clients to the appropriate Message Broker. If you’re using a client that will benefit from using a Message Broker that is geographically closer to you, you should select the appropriate alternate MPX profile set up by the administrator. In the graphical clients, do this via the Server Properties dialog box:
Figure 8: StarTeamMPX profile
You need to do this only once. StarTeam remembers the selected profile for each StarTeam Server. The StarTeam SDK also has options to enable MPX and select the default or an alternate profile. (By the way, the SDK provides a rich event-based interface so you can write applications that tap into the MPX event bus.)
Message Brokers have a variety of configuration options to support external users, VPNs, failover, and other issues. Multicast broadcasting is also available to reduce network traffic even further on local networks. More extensive information on StarTeamMPX can be found in the sources cited at the end of this article.
We say that MPX turns the network inside-out because it evolves client/server communication from synchronous, bandwidth-sensitive pull messaging toward more asynchronous, bandwidth-sparing push messaging. The trend can be characterized with this analogy: Instead of everyone phoning into the newsroom to see if anything is new, the newsroom installs a radio transmitter so that everyone can get the news as it happens.
19 Optimizing Borland® StarTeam® for distributed teams
StarTeamMPX Cache Agent: Distributed file caching
Through the 6.0 release, MPX broadcasts updated metadata and database objects: change requests, tasks, filters, etc. When a file is modified, its object properties (modified by, revision number, etc.) are broadcast, allowing clients to know when a new file is available. However, a client still must check out the new file via the command API in order to receive its contents.
With the StarTeam 2005 release, the MPX Cache Agent is introduced that allows file contents to be broadcast via the MPX framework and cached at geographic regions. The operation of the Cache Agent are described next.
Basic Cache Agent architecture
The relationship of the MPX Cache Agent to the overall StarTeam architecture is shown below:
StarTeam StarTeam Client Server
Check-out requests
Cache Message VaVaulultt DB Agent Broker
EnEncrcryypptteded File publish stream CaCacheche
StarTeam Client
Figure 9: StarTeamMPX Cache Agent architecture
20 Optimizing Borland® StarTeam® for distributed teams
The Cache Agent subscribes to a file publish stream that is received and broadcast by the Message Broker. Contents of new files are encrypted and stored persistently by the Cache Agent, where they can be retrieved by Cache Agent-aware clients. Any number of Cache Agents can be deployed throughout an organization, providing multiple places from which files can be checked out.
What does the Cache Agent do for your distributed team? Lots:
• Performance: Remote team members who check out from a Cache Agent receive network-near performance, even when the StarTeam server is on the other side of the world. (In fact, Cache Agent users will receive even better performance when checking out large numbers of files since the Cache Agent uses new multifile and parallel checkout strategies.)
• Bandwidth conservation: Each new file is broadcast once from the StarTeam Server to each Cache Agent. Consequently, when remote teams use the Cache Agent, the same file is not pulled repeatedly over the long-wire network. You will see a drop in overall network.
• Not your father’s caching: Cache Agents are trickle-charged, which means they asynchronously receive files in the background. In most cases, new files arrive at Cache Agents within a few seconds to a few minutes after they are checked in. Consequently, a client will receive very high “hit” ratios when using a Cache Agent, even when it is the first client to request a file. This strategy is known as push caching, which is much more advanced from traditional demand caching. In the case where a client requests a file from a Cache Agent that is not available, the client reverts to normal command API checkout for that file. (See also the discussion on forwarding described later.)
• Security: All files stored by the Cache Agent are encrypted. Clients can request only files for which they have permission to access, and files are decrypted only as they are streamed to the client’s work files.
21 Optimizing Borland® StarTeam® for distributed teams
• Less long-distance network dependency: Because StarTeam uses work files for local file access, you normally do not need StarTeam during edit, build, debug, and other lifecycle tasks. However, your StarTeam client needs server access to update work file status, check out new files, and perform updates. The time to check out files is often the factor that most affects the “window” of accessibility that you need, especially when you are working remotely over constrained bandwidth. When you check out from a local Cache Agent, the long-wire network is not used at all. Consequently, the accessibility window is dramatically reduced, minimizing the susceptibility of your tasks to network outages. It’s not 100% autonomy, but it’s more autonomy than ever before.
• Server scalability: Studies of real customer databases have shown that file check- outs constitute up to 33% of all commands and up to 98% of all outbound network traffic from the StarTeam server. The more team members use Cache Agents for checkouts, the more capacity is freed at the server, enabling it to support more users and provide better overall performance.
OK, it sounds good, but what’s the catch? Cache Agents are probably hard to install, configure, monitor, or use, right? Nope. If there is a catch, ease-of-use isn’t it. Here are some features that make Cache Agents easy to get along with:
• A Cache Agent can cache content from any number of StarTeam servers. The size of each Cache Agent’s cache can be configured, and the most recently stored and used files will be kept, as older, unrequested files are deleted.
• Cache Agent-aware clients can be configured to use a specific Cache Agent, or they can be configured to “auto-locate” the Cache Agent that is network nearest to them. This means that new Cache Agents can be installed, and clients will immediately begin to find and use them.
• Cache Agents do not have to be backed up, nor do they require special high- availability or failover measures. Because Cache Agents are optional and contain
22 Optimizing Borland® StarTeam® for distributed teams
only read-only copies of files that are stored on StarTeam servers, they do not represent critical operational units.
• Cache Agents use HTTP for communication. They can be monitored from anywhere with a standard Web browser.
Sold? But wait! There’s more!
Stacked Cache Agents
Cache Agents can be stacked to provide a hierarchical structure. A simple case of stacked Cache Agents is illustrated below:
StarTeam StarTeam Client Server
Remote Message VauVaulltt Cache DB Broker Agent
EEnncrycrypptedted CaCacheche
Catch-up/forwarded requests Root StarTeam Cache Client Agent Alternate check-out path
Figure 10: StarTeamMPX stacked Cache Agent example
To create hierarchical Cache Agents, a special root Cache Agent is deployed for each participating StarTeam Server. A root Cache Agent is different from all other Cache Agents
23 Optimizing Borland® StarTeam® for distributed teams
only in configuration; it is given access to the StarTeam Server’s vault, allowing it to fulfill check-out requests directly from the vault. A root Cache Agent can run on the same or a different server than the StarTeam Server that it services. Non-root or remote Cache Agents use one or more root Cache Agents as catch-up and forwarding points.
What do stacked Cache Agents do for your distributed teams?
• A Root Cache Agent provides an alternate checkout path for StarTeam clients when a remote Cache Agent is not available or as simply another path to draw checkout traffic away from the StarTeam Server.
• A newly deployed remote Cache Agent can use a root Cache Agent to “precharge” with the latest content, allowing it to provide effective caching quickly after it is installed.
• When a remote Cache Agents can not fulfill a checkout request, it can forward the request to a root Cache Agent, which will fulfill the request, “charging” the remote Cache Agent with the missing file in the process. In a sense, stacking allows Cache Agents to revert to traditional demand caching when needed.
• Remote Cache Agents are designed to be resilient to network outages. After an outage, a remote Cache Agent will use root Cache Agents to “catch up” with content that they might have missed during the outage. Hence, remote Cache Agents automatically stay up-to-date with the latest content.
• Remote Cache Agents can be configured to auto-locate each required root Cache Agent, simplifying configuration requirements.
• With auto-locate, precharging, and forwarding capabilities, Cache Agents can be dynamically added to the network and effectively used with little configuration or administration.
The previous diagram shows one remote and one root Cache Agent, representing a two-level hierarchy. In practice, you can configure Cache Agents into as many levels as you like. Cache
24 Optimizing Borland® StarTeam® for distributed teams
Agents can be configured as “public”, allowing themselves to be auto-located, or “private” and hence dedicated to specific applications. For example, a distributed team with a checkout intensive build utility could deploy the following three-level Cache Agent hierarchy:
1. A public root Cache Agent located network-near to the StarTeam Server that it serves. As a public Cache Agent, both Cache Agent-aware clients and downstream Cache Agents can auto-locate it.
2. A public remote Cache Agent located at a remote development site, tiered to the root Cache Agent. As a public Cache Agent, the remote Cache Agent can be auto-located by Cache Agent-aware clients.
3. A private remote Cache Agent located on a build machine at a remote development site, tiered to the public remote Cache Agent. As a private Cache Agent, it can be dedicated for use by the build application, providing it with superior performance for bulk checkouts.
Keep in mind that Cache Agents provide read-only file checkout functionality; hence updates still must be directed to the appropriate StarTeam Server. This prevents artificial merge conditions and prevents the need for replication and synchronization (and the associated headaches). In most environments, updates constitute a relatively small percentage of transactions, whereas file checkouts constitute a larger percentage of outbound network traffic. In short, StarTeamMPX Cache Agents free up long-distance network traffic where it is needed most.
Cache Agent-aware StarTeam clients
With the StarTeam 2005 introduction of MPX Cache Agents, included are two StarTeam clients that are Cache Agent-aware: the Cross-Platform Client and a new Bulk Checkout Utility.
25 Optimizing Borland® StarTeam® for distributed teams
Using Cache Agents with the Cross-Platform Client To enable Cache Agent usage in the Cross-Platform Client, you’ll notice that the StarTeamMPX options tab (found in Tools Æ Personal Options) has been extended:
Figure 11: StarTeamMPX Cross-Platform client options
Check the “Enable StarTeamMPX Cache Agent” box to allow the Cross-Platform Client to use a Cache Agent for file checkouts. Next, select “Automatically locate…” to use the public Cache Agent that is network-nearest to the client, or select “Use Cache Agent at” and enter the address and port of the Cache Agent that you want to use.
You’ll notice one other change when you check out files with the Cross-Platform Client. The checkout dialog box provides a new option for displaying checkout statistics:
26 Optimizing Borland® StarTeam® for distributed teams
Figure 12: StarTeam Cross-Platform client check out options
This option allows you to review the effectiveness (e.g., number of files checked out) and performance (bytes transferred per second) of the Cache Agent. If one or more files cannot be checked out through the configured or auto-located Cache Agent, the Cross-Platform Client automatically switches to regular StarTeam command API. When requested, the statistics dialog box shown at the conclusion of checkout will tell you how many files it retrieved with each method.
Using Cache Agents with the bulk checkout utility The Bulk Checkout utility (BCO) is new in StarTeam 2005. It is similar to the StarTeam Command-Line utility’s “checkout” command (i.e., “stcmd co …”). In fact, BCO provides the same functionality as “stcmd co” with the same parameter names and options, with a few differences:
• BCO provides a “-useca” option that enables use of a Cache Agent for file checkout. An explicit Cache Agent address and port can be provided, or “autolocate” can be used to find and use the closest public Cache Agent.
27 Optimizing Borland® StarTeam® for distributed teams
• A “-t” parameter allows file checkout metrics to be displayed, similar to the Cross- Platform Client’s “Show Checkout Statistics” option. Like the Cross-Platform Client, BCO will attempt to check out all files from the Cache Agent, but it will switch to the regular command API checkout as needed.
• Whereas the StarTeam Command Line utility provides many functions, BCO is dedicated to large file set (bulk) checkouts and hence is optimized for this purpose. Buildmeisters and other users who need to check out large sets of files from a command-line interface will find BCO perfect for the job.
• BCO has slight option variations from the “stcmd co” command. See the help text and user’s guide for more details.
An example use of BCO is shown below:
bco -p "user:pw@prod1:49201/Project1/View1/srcfiles" -useCA autolocate -cfgl "6.0.1" -is -filter IO "*.java"
This sample run causes BCO to logon with user “user” (password “pw”) to the StarTeam Server named “prod1” (port 49201) and use project “Project1” and view “View1”. Files are checked out using an autolocated Cache Agent (“-useCA autolocate”) as of the view label “6.0.1”. All Java files (“*.java”) from the base folder “srcfiles” and its descendents (“-is”) are checked out whose status is “out-of-date” or “missing” (“-filter IO”).
StarTeam Import/Export Manager
Are there cases where you legitimately need replication? When you consider using replication, you are usually facing one of the following issues:
28 Optimizing Borland® StarTeam® for distributed teams
1. Scalability: Your repository has hit a performance wall based on its current hardware and number of users, and your hardware is maxed out and you can’t decrease users.
2. Distributed teams: You have a distributed team that requires access to shared artifacts, and remote sites have limited and/or unreliable bandwidth.
3. Security issues: You have an offshore development team, third-party contractor, or other business partner that needs access to your team’s artifacts, but you have security concerns and want to use replication as a functional “firewall”.
4. Connectivity: You have a business partner or customer who requires access to your project’s artifacts, but network access is not possible. This occurs, for example, with government and military projects.
With issue #1, some ALM systems hit the scalability wall with as few as 100 users. With StarTeam, extensive performance and scalability techniques allow a single StarTeam server to support thousands of users and hundreds of concurrent sessions. Replication is not needed to scale StarTeam repositories. See the references cited at the end of this article for more information on scalability techniques.
Issue #2 is the focus of this article. As we have shown, numerous features such as the StarTeamMPX Cache Agent are available to optimize the experience of distributed teams. No replication needed here.
Issue #3 can also be addressed with available StarTeam features. StarTeam has powerful, ACL-based security controls that allow you to grant or deny access to artifacts, with granularity ranging from projects to individual objects. Access for users and groups can be specified for specific verbs such as “create label”, “check out file”, or “modify folder”. Combined with StarTeam’s flexible “view” facility, StarTeam security allows you to isolate access for external groups without using replication. See the references at the end of this article and the StarTeam Administrator’s guide for more information on StarTeam’s security capabilities.
29 Optimizing Borland® StarTeam® for distributed teams
Issue #4 is the scenario that most justifiably requires replication. Sometimes, replication is the only solution to deliver StarTeam artifacts to external groups. Examples where direct access to StarTeam artifacts is not possible are summarized below:
• Embedding: One or more StarTeam projects are part of an application’s deliverable. Recipient clients must load these artifacts into their own repository and cannot access yours.
• No physical access: For security reasons such as military contracts, participating team members must operate within closed systems for which no network access is possible. Repository artifacts must be transferred into (and occasionally out of) such installations on removable media only.
• Contractual: Even if physical network access is possible, contractual issues may force team members to participate as if none were available. Despite StarTeam’s security controls, security regulations may prohibit two teams from sharing a common repository.
In these and similar cases, you need a way to transfer StarTeam artifacts from one location to another, potentially with no interconnecting network. These are the cases for which the StarTeam Import/Export Manager was designed.
StarTeam Import/Export Manager allows artifacts, including history revisions and file content, to be transferred from one StarTeam repository to another. An overview of the transfer process is shown below:
30 Optimizing Borland® StarTeam® for distributed teams
StarTeam Import/Export Manager
Read-only Online or Local update process offline transfer process
Export Import
Source Target StarTeam XML+ XML+ StarTeam Server archive archive Server
image image
Figure 13: StarTeam Import/Export Manager overview
StarTeam Import/Export Manager allows you to transfer all or a portion of a “source” StarTeam repository’s artifacts to another “target” repository in three distinct phases:
1. Export: In this phase, the desired artifacts are extracted and persisted as a transfer “image”, consisting of XML and archive files. The export process is run on the source StarTeam server machine or a machine network near to it. Export reads only the source repository. It does not perform any updates.
2. Transfer: The generated image is transferred to the target environment with any online or offline transfer technique: network copy, FTP, removable media, “sneaker net”, etc.
3. Import: The transferred image is then loaded into the target machine. The import is performed on the target server or a machine network near to it.
With StarTeam Import/Export Manager, you can perform an initial “bulk” export/import followed by periodic “incremental” runs, which allow synchronization with minimal transfer size. You can also use “scoping” to transfer all or a portion of a repository’s artifacts such as a
31 Optimizing Borland® StarTeam® for distributed teams
single project, view, or folder. Many other options exist to control the level of metadata that is included in the transfer. The import/transfer/export process can also be performed in the reverse direction so that, for example, updated artifacts can be loaded back into the original source server.
When replication is truly needed, StarTeam Import/Export Manager is your solution.
Summary: The world according to StarTeam
In this article, the case has been made that the optimal use of StarTeam for distributed teams is to deploy centrally-managed repositories. Performance for distributed teams can be optimized by using compression and delta checkouts. Resiliency from minor network glitches can be achieved by using StarTeam’s auto-reconnect feature.
At the next level of optimization, StarTeamMPX can be used to minimize network usage while improving the responsiveness of remote StarTeam clients. In the StarTeam 2005 release, the introduction of the MPX Cache Agent allows distributed file caching, dramatically improving the performance of file checkouts for remote teams while further reducing global network usage.
32 Optimizing Borland® StarTeam® for distributed teams
MB and CA
MB and CA
ST Server, MB, and root CA MB and CA
MB and CA
Figure 14: StarTeam global useage example
In the optimized StarTeam community, the StarTeam Server, base Message Broker, and root Cache Agent are deployed at a centralized location. High availability strategies, backup procedures, and administrative expertise are concentrated at this facility. Each remote development site is supported with a remote Message Broker and remote Cache Agent. By using compression, StarTeamMPX, and a network-near Cache Agent, each developer receives superior responsiveness while being buffered from common network issues. At a global level, an organization can provide ample accessibility and performance without enduring the headaches of replication. In cases where network access is not an option, StarTeam Import/Export Manager can provide the needed replication functions.
Over the past several years, StarTeam has continually evolved to improve scalability, reliability, and performance for distributed teams. So, with the introduction of the Cache Agent in the 2005 release, is this the end of the road for features that support distributed teams? Absolutely not! Want a glimpse of the future? Here’s a peek at where we’re headed:
33 Optimizing Borland® StarTeam® for distributed teams
• In its initial form, the MPX Cache Agent caches only files’ contents. However, it is possible to persistently cache any kind of lifecycle artifact: change requests, tasks, requirements, etc. As we enable more artifacts for distributed caching, we move closer to allowing entire projects to be cached for distributed teams.
• For clients that are mobile, occasionally connected, or otherwise require offline processing, it is possible to persistent client contexts similar to office-style “auto save” features. Imagine, for example, that you open a project, and your client silently preserves a snapshot of that project’s structure (folder hierarchy, labels, item IDs, etc.) Reconstructing the hierarchy from a saved snapshot and retrieving needed artifacts from a network-near (possibly same-machine) Cache Agent would be possible. Together, these features constitute complete offline processing capabilities.
• Once offline read-only operation is supported, the next step would be to provide offline updates. How could we do that? Think email or PDA metaphor: Updates could be stored in a local “outbox” and resynchronized when your clients establishes connectivity. Conflicts would be flagged as now, via local status values that must be resolved before final “check-in” is allowed.
In short, global teams are moving towards a networked yet mobile, occasionally connected world, and StarTeam will continue to move with it.
Glossary
The StarTeamMPX component that provides file-caching functions. The Cache Agent Cache Agent provides distributed teams a high-performance alternate source for file checkout operations. The basic client/server API used by StarTeam. Each client request and command API corresponding server response constitutes a single “command”. A special form of file checkout that uses less network traffic by sending delta information to the client instead of full file revisions. With the delta checkout graphical StarTeam clients, delta checkout is activated via the “Optimize for slow connections” option.
34 Optimizing Borland® StarTeam® for distributed teams
The traditional method for charging a cache wherein a client requests information, which is fetched from a server and returned to the client demand caching while being added to the cache at the same time. Demand caching causes the first request for specific information to result in a cache “miss”. Contrast with push caching. A term that describes an MPX Cache Agent’s hierarchical location with downstream respect to another Cache Agent. A Cache Agent is “downstream” of any Cache Agent Cache Agent to which it sends forwarding and catch-up requests. The file compression strategy used by the new StarTeam 2005 Native II vault. As new files are checked in, they are tested for compressibility. intelligent Thereafter, they are stored and transferred during checkout in a manner compression that optimizes compression and decompression usage. Intelligent compression also results in more efficient network usage during checkouts. A configuration defined by the StarTeam administrator for a specific StarTeamMPX Message Broker or Multicast Service. A default MPX MPX profile profile is specified that most clients will use, but remote clients can select an alternate profile to use a Message Broker or Multicast Service closer to their location. An advanced caching method wherein a cache is charged by pushing new content to it before it is requested by clients. Compared to demand push caching caching, push caching improves the cache request “hit” ratio. Push caching is used by MPX Cache Agents. An MPX Cache Agent that is not a root Cache Agent, typically used to remote Cache provide file caching for a remote team. A remote Cache Agent will Agent usually also be downstream Cache Agent, connected to an upstream Cache Agent for forwarding and catch-up functions. A special MPX Cache Agent that serves as the forwarding and catch-up root Cache headwater for a specific StarTeam server. A root Cache Agent is typically Agent an upstream Cache Agent for one or more remote Cache Agents. replication A method for copying artifacts from one repository to another. StarTeam A single deployment of a StarTeam repository. A StarTeam configuration configuration has a database, which houses metadata and artifact objects, and a vault, (or instance) which houses the contents of file revisions. An add-on product for StarTeam that adds a publish/subscribe framework to the client/server architecture. StarTeamMPX, also referred to as simply StarTeamMPX MPX, boosts server scalability and client responsiveness while reducing overall network traffic. The StarTeam component that provides server processing for a single StarTeam Server StarTeam configuration.
35 Optimizing Borland® StarTeam® for distributed teams
A term that describes an MPX Cache Agent’s hierarchical location with upstream Cache respect to another Cache Agent. A Cache Agent is “upstream” of all Agent Cache Agents which are sending it catch-up and forwarding functions. Short for “wireless fidelity”, the set of wireless communications protocols Wi-Fi based on the family of IEEE 802.11 standards.
References
• 24 x 7 StarTeam, September 2004. Presented at BorCon 2004. (Watch http://bdn.borland.com for forthcoming download instructions.) This article describes techniques and approaches you can use to develop a comprehensive high availability plan for your StarTeam repositories.
• Borland StarTeam MPX Server, May 2003. Available at: http://www.borland.com/products/white_papers/starteam_mpx_server.html. This paper provides a high-level overview of StarTeamMPX.
• StarTeam Configuration Best Practices, December 2003. Presented at BorCon 2003. Available at: http://bdn.borland.com/article/0,1410,31869,00.html. This articles presents best practices for creating StarTeam server configurations, defining and managing projects, and using StarTeam views. Do’s and Don’ts, sample development scenarios, and general tips and techniques are presented for using StarTeam effectively.
• StarTeam Performance and Scalability Techniques, December 2003. Presented at BorCon 2003. Available at: http://bdn.borland.com/article/0,1410,31868,00.html. This article describes techniques for maximizing the performance and scalability of StarTeam servers. Hardware and capacity planning, StarTeamMPX configurations, and performance tuning are all discussed.
• StarTeam Security Explained!, September 2004. Presented at BorCon 2004. (Watch http://bdn.borland.com for forthcoming download instructions.) This article explains StarTeam ACLs, ACEs, and other StarTeam security concepts and how to configure security for your StarTeam projects.
36 Optimizing Borland® StarTeam® for distributed teams
Made in Borland® Copyright © 2005 Borland Software Corporation. All rights reserved. All Borland brand and product names are trademarks or registered trademarks of Borland Software Corporation in the United States and other countries. Corporate Headquarters: 100 Enterprise Way, Scotts Valley, CA 95066-3249 • 831-431-1000 • www.borland.com • Offices in: Australia, Brazil, Canada, China, Czech Republic, Finland, France, Germany, Hong Kong, Hungary, India, Ireland, Italy, Japan, Korea, Mexico, the Netherlands, New Zealand, Russia, Singapore, Spain, Sweden, Taiwan, the United Kingdom, and the United States. • 23255
37