PTAGIS Web-App Build Process

Using Ant and Subversion

Version 1.3, 29 Aug 2005 Doug Clough, SYNERGETICS

We have a lot of great open-source tools at our disposal. Many of them come with good tutorials and usage examples.

This document pulls together ideas from Ant and Subversion (abbreviated SVN) on-line documentation, the O’Reilly book on Ant, Pragmatic Version Control Using Subversion, the SpringFramework MVC Step-By-Step tutorial by Thomas Risberg, and the on-line manual for Log4J.

The objective is to establish a practical, standardized approach to web-application development and deployment that incorporates recommended practices of the Extreme Programming community - e.g. Java Tools for Extreme Programming, Wiley, 2004 - while addressing specific features of the PTAGIS environment.

But before diving in, a few words about the tools:

SpringFramework – General purpose Java application development framework – Compared to Struts, SpringMVC substantially simplifies the development of Java-based web applications. Using the process described herein, applications constructed with SpringMVC and incorporating Log4j will be configured and provisioned by Ant, archived in SVN, tested with JUnit, and packaged by Ant as war-files for deployment to WORKstation, BETA-test, and PRODuction environments.

SVN – ‘Subversion’ Version Control System – Files imported into an SVN repository may be checked out and edited concurrently by any number of remote users. When users commit their work, SVN inspects the changes and notifies users of any conflicts it has detected. According to the Subversion web-site: “The goal of the Subversion project is to build a version control system that is a compelling replacement for CVS in the open source community.” As this document is being written, we’re experimenting with SVN as an alternative to CVS. SVN offers some significant advantages over CVS – for example, truly atomic ‘commits’. The build process described herein will be pretty much the same whichever version control system is adopted. If you happen to spot a left-over ‘CVS’ in any of the slides, mentally replace it with ‘SVN’.

Ant – “Another Neat Tool” – Flexible Java program, driven by an XML configuration file, performing the full range of operations required to build and deploy Java applications. Fulfills the same role as “make”, but is much more intuitive and easier to use.

Log4j – “Logging Utility For Java” – Modular, configurable logging utility for incorporation into Java programs. Logging verbosity can be adjusted by editing a configuration file, without re-compiling the code.

JUnit – “Java Unit Test Utility” – Framework for easy development of unit tests. Although not covered in this document, JUnit is an integral part of the PTAGIS Web-App Build Process (and will be covered in another document).

1 Challenges Tackled by Automated Build Process

sebastes

Multiple concurrent Multiple deployment blueback developers environments

- Third-party ‘jar’ files pitblade - Re-usable locally-developed classes sockeye - Avoid conflicts with other web-apps

What features of the PTAGIS environment need to be addressed? How can Ant and SVN help?

SVN provides the means for reconciling the work of multiple concurrent developers.

Before “development” work on a new project begins, Ant enables automated creation of the project directory tree, and ‘provisioning’ the application with required third-party jar-files.

At build-time, Ant can automatically tailor the build process to the deployment environment; for example, setting log-file path names, JDBC connection parameters, and other things that vary between a developer’s workstation, the beta- test environment on ‘pitblade’, and the production environment on ‘sebastes’.

Much of what follows is a detailed examination of the means by which this customization will be achieved for PTAGIS web applications.

2 New Project Setup

• Automatically create project directory tree • Establish properties that vary by environment • Provide copies of required 3rd-party JAR-files • Import project into SVN • Check project out of SVN, build, and deploy to verify setup

Let’s walk through the process, as if we were setting up a new project.

The first step is to create a scratch directory to contain temporary copies of the various files and sub-directories required by a typical SpringMVC web application. The scratch directory and its content will be deleted as soon as the new project has been imported to the SVN repository.

NOTE: Not all of this has been implemented yet – I’m “thinking out loud” here …

As ‘ptagdev’ on ‘sockeye’ … > web_proj_temp Å Unix ‘alias’ that cd’s to appropriate directory > UTIL_ProjSetup.pl Å Runs Ant using specially constructed build.xml, to create the directory tree shown in the next slide

3 Automatically Create Project Directory Tree

What are these?

In addition to creating the directory tree in our temporary work area, Ant copies basic versions of files required by SpringMVC web-apps into the war/WEB-INF directory

It also populates the top-level directory with

build.xml Parameterized Ant build script for the new project build.template Generic template for creating build.properties; contains placeholder “tokens” for parameter values that vary by environment log4j.template Generic template for creating log4j.properties; contains placeholder “tokens” for parameter values that vary by environment

Examples of these files are shown in the following slides. Don’t worry about every detail – just note the tokens – e.g. @JDBC_URL@ – in the template files. These are the values that must be set differently, depending on where the app is to be deployed.

Appropriate values for BETA-test (‘pitblade’) and PRODuction (‘sebastes’) and the mechanism for replacing the tokens with suitable values are detailed in later slides.

4 build.template Æ build.properties

# # Source : Generic build.template file with replaceable tokens. # For Deployment To: @DEPLOYMENT@ environment #

# These are the same for all environments ... lib.root=/home/ptagdev/ptagis3/java_lib

lib.spring=spring-framework-1.2.2 lib.db=ca_jdbc_type_2 lib.log4j=log4j-1.2.9 lib.commons-logging=commons-logging-1.0.4 lib.tags_jstl=jstl-1.0.3 lib.tags_standard=jstl-1.0.6

lib.servlet=servlet-2.4/weblogic-8.1

db.driver=ca.edbc.jdbc.EdbcDriver

# These differ between deployment environments ... db.url=@JDBC_URL@ db.user=@JDBC_USER@ db.pw=@JDBC_PW@

Here’s the template that produces the build.properties file used by Ant.

5 log4j.template Æ log4j.properties

# # Source : Generic log4j.template file with replaceable tokens. # For Deployment To: @DEPLOYMENT@ environment #

# This establishes the logging level and varies by deployment environment ... log4j.rootLogger=@ROOT_LOGGER@

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n

log4j.appender.logfile=org.apache.log4j.RollingFileAppender log4j.appender.logfile.MaxFileSize=512KB

# This establishes the log-file location and varies by deployment environment ... log4j.appender.logfile.File=@LOG4J_LOG_PATH@

# Keep three backup files. log4j.appender.logfile.MaxBackupIndex=3

# Pattern to output: date priority [category] - message log4j.appender.logfile.layout=org.apache.log4j.PatternLayout log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

This template produces the Log4j.properties file, that controls logging behavior of the deployed application.

6 New Project Setup

Now that we’ve had a look at the template files and their replaceable tokens, let’s see where the corresponding values are established.

7 Establish Properties That Vary by Environment

Manually edit template.properties files

Created by developer (later on) to suit workstation environment

Created at project initiation to suit PRODuction and BETA-test environments

As shown in this slide, Ant has created BETA and PROD directories in the project directory tree.

These are used to store files that pertain specifically to the BETA-test and PRODuction environments.

At the time of initial project setup, each of these directories contains one file – template.properties – defining environment-specific replacement values for the tokens mentioned in the previous slides.

Initial versions of these files provided by the Ant procedure will probably be suitable for most projects. Once a project has been imported to SVN, changes to these files will be tracked along with everything else in the repository.

Note, however, that the slide shows an additional copy of template.properties, in the top-level project directory.

The purpose of this file is to establish values for use by a developer on an individual WORKstation. Its values will most probably differ from developer to developer. Accordingly, this file is not placed into the project directory during the initial setup procedure – neither is it registered with SVN when the project is imported into the repository.

Further, developers must not add their individual template.properties files to the project’s SVN repository – since their individual changes would be incompatible – rather, they should archive the files locally by any convenient means.

Typical WORKstation, BETA-test, and PRODuction versions of template.properties are shown in the next slide.

8 template.properties – For Different Environments

for_deployment_to=WORK

jdbc_url=jdbc:edbc://localhost:II7/sockeye::g2_ptagis3 jdbc_user=ptagdev WORK jdbc_pw=my_secret

root_logger=DEBUG, stdout, logfile log4j_log_path=/home/ptagdev/ptagis3/source/SpringFramework/Logs/springapp.log for_deployment_to=BETA

jdbc_url=jdbc:edbc://localhost:SG7/g2_ptagis3 jdbc_user=ptagdev BETA jdbc_pw=my_secret

root_logger=INFO, stdout, logfile log4j_log_path=/dsk2/local/bea81/user_projects/mydomain/springapp.log for_deployment_to=PROD

jdbc_url=jdbc:edbc://localhost:II7/blueback::g2_ptagis3 jdbc_user=pittag PROD jdbc_pw=my_secret

root_logger=ERROR, stdout, logfile log4j_log_path=/usr/local/bea81/user_projects/mydomain2/springapp.log

As may be seen in the slide, the use of distinct template.properties files allows JDBC connection parameters, logging verbosity, and other variables to be set appropriately for each environment.

In the next slide we’ll review what we’ve covered so far …

9 Ant ‘filterset’ Æ Customization By Token Substitution

Developers must track local changes to template.properties on their respective workstations Tracked in SVN

BETA & PROD Tracked in SVN

Not Tracked in SVN

build.template Generic configuration files, with placeholder tokens for things log4j.template that vary by environment

Substitution values for tokens in template.properties generic configuration files

build.properties Environment-specific properties files produced by ‘interpolate’ log4j.properties target of Ant build.xml file

Remember: The purpose of all the “arm waving” is to build WAR-files (web- application archive files) automatically customized for deployment to WORKstation, BETA-test, and PRODuction environments.

This is accomplished by a set of generic, tokenized template-files and one specific template.properties file for each deployment environment:

XXX.template combined with template.properties produces a customized XXX.properties file

Let’s step back from the detail and have a look at the big picture, in the next slide.

10 Establish Properties That Vary by Environment

When ready to build, we run Ant ‘interpolate’ target to generate customized ‘properties’ files.

PROD

WORK Interpolation Ant Templates

BETA

SVN Ant

Fortunately, we won’t have to concern ourselves with these details once the project has been set up and imported to SVN.

Whether serving the needs of an individual developer or someone tasked with building war-files for deployment to BETA-test or PRODuction, Ant – executing the project’s build.xml file – will replace the tokens with the appropriate parameter values.

In case you’re interested, the next slide shows the portion of build.xml – the interpolate task – that does this job.

11 build.xml –“interpolate” Target

e.g. For deployment to ‘pitblade’ > cd $PROJ_ROOT/BETA > ant –buildfile ../build.xml interpolate

Relative to location of build.xml

‘cd’ to WORK, PROD, or BETA directory before running Ant

build.xml’s many other ‘targets’omitted

You may have wondered how Ant knows which environment to build for: Actually, it doesn’t!! Remember the template.properties files discussed at the beginning?

During initial project setup, Ant provided copies of template.properties in the BETA and PROD directories. It was stated in an early slide that each developer – after checking the project out of the SVN repository – must create a template.properties file suitable for their particular workstation, and place it in the top-level project directory.

Ant knows nothing about this – it just knows how to execute build.xml. So – we need to make sure it finds build.xml; build.xml will take care of the rest!

As highlighted in the slide, file path-names in build.xml are interpreted relative to basedir – which is set to the directory where build.xml resides. In contrast, build.xml locates template.properties in ${user.dir} – the currently selected directory, which could be the top-level project directory or the BETA or PROD sub-directory.

This means that a developer can simply “cd” to the project root directory and issue the following command: > ant interpolate Ant will find the developer’s template.properties file and generate build.properties in the project root directory. It will also place an appropriately customized instance of log4j.properties in the project’s war/WEB-INF/classes directory. When this has been accomplished, the makeWar target can be executed, producing output that will work in the developer’s environment.

On the other hand, an administrator preparing a build for deployment to BETA or PROD must “cd” to the project’s BETA or PROD directory, then: >ant –buildfile ../build.xml interpolate In this case, Ant finds the copy of template.properties appropriate to the selected deployment environment. It places the generated build.properties file in the environment-specific (BETA or PROD) directory. In contrast, however, it places the generated log4j.properties file in the project’s war/WEB-INF/classes directory, as before.

In actual usage, it isn’t necessary to run the interpolate target manually: it’s run automatically, as one of the preparatory steps in the makeWar target. The important thing to remember is to “cd” to the appropriate directory (as described above) before running Ant. Also, as emphasized previously – files generated by the various Ant targets, such as build.properties and log4j.properties, are not to be stored in SVN!!

Sample output for the PROD environment is shown in the following two slides …

12 build.properties – For PROD Environment

Sample output from > ant –buildfile ../build.xml interpolate

# # Source : Generic build.template file with replaceable tokens. # For Deployment To: PROD environment #

# These are the same for all environments ... lib.root=/home/ptagdev/ptagis3/java_lib

lib.spring=spring-framework-1.2.2 lib.db=ca_jdbc_type_2 lib.log4j=log4j-1.2.9 lib.commons-logging=commons-logging-1.0.4 lib.tags_jstl=jstl-1.0.3 lib.tags_standard=jstl-1.0.6

lib.servlet=servlet-2.4/weblogic-8.1

db.driver=ca.edbc.jdbc.EdbcDriver

# These differ between deployment environments ... db.url=jdbc:edbc://localhost:II7/blueback::g2_ptagis3 db.user=pittag db.pw=my_secret

Here we see the result of PROD/template.properties interpolated into build.template

Where is the output file located? That’s right – it’s in PROD/build.properties!!

13 war/WEB-INF/classes/log4j.properties – For PROD Environment

Sample output from > ant –buildfile ../build.xml interpolate

# # Source : Generic log4j.template file with replaceable tokens. # For Deployment To: PROD environment #

# This establishes the logging level and varies by deployment environment ... log4j.rootLogger=INFO, stdout, logfile

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d %p [%c] - <%m>%n

log4j.appender.logfile=org.apache.log4j.RollingFileAppender log4j.appender.logfile.MaxFileSize=512KB

# This establishes the log-file location and varies by deployment environment ... log4j.appender.logfile.File=/usr/local/bea81/user_projects/mydomain2/springapp.log

# Keep three backup files. log4j.appender.logfile.MaxBackupIndex=3

# Pattern to output: date priority [category] - message log4j.appender.logfile.layout=org.apache.log4j.PatternLayout log4j.appender.logfile.layout.ConversionPattern=%d %p [%c] - %m%n

Similarly, here is the result of interpolating values from PROD/template.properties into log4j.template.

Where is this file located? You’re right! I thought I could trip you up, but it is in fact located in war/WEB-INF/classes – not in an environment-specific directory.

14 New Project Setup

15 Provide copies of required 3rd-party JAR-files

• How can we identify different versions of foobar.jar? Successive versions have the same name! MANIFEST.MF may give us a clue • Put each jar-file in its own versioned subdirectory Name the subdirectory after the jar’s version Include manifest info in MANIFEST_foobar_jar.txt • util_JarLib_Register.pl does the heavy lifting

Not only do we want to provide all the necessary “external” jar files needed by the new project, we also want to track which version of each jar is used for every build throughout the project’s life. This poses a problem, because many jar-files – e.g. struts.jar, commons-logging.jar – contain no version information in their name. How, then, can we distinguish successive versions from one another – let alone know the version number of a given jar?

In some cases, the answer will be contained in the jar’s internal META-INF/MANIFEST.MF file. In other cases, we may never know the version number in the context of the vendor’s build process. In any case, though, we can solve the problem by storing each jar file – or compatible set of interdependent jars - in a distinctly named subdirectory within a jar library.

A tool – util_JarLib_Register.pl – has been developed to facilitate this scheme. Given the full path- name of a jar-file, the tool extracts the jar’s MANIFEST.MF and displays its content to the user. It then prompts for the name of the subdirectory within the $JAR_LIB directory where the jar-file is to be stored. The user examines the manifest information, looking for Implementation Version: X.Y.Z. If version info is present in the manifest, the user types in a subdirectory name like foobar-X.Y.Z. The tool then creates the subdirectory, copies foobar.jar to it, and writes MANIFEST_foobar_jar.txt, thereby making the manifest information readily available for future reference.

If version information is not available in the MANIFEST.MF file, the user must concoct an identifying string – for example, based on the jar-file’s time-stamp.

Several of the possible scenarios are illustrated in the next slide …

16 Provide copies of required 3rd-party JAR-files

Version supplied by “vendor”

Related jar-files grouped in subdirectory

Version extracted from MANIFEST.MF

In the simplest case, version information will have been provided by the “vendor”. For example, the SpringFramework distribution is packaged as a zip-file, with the release version indicated in the file name. The distribution contains numerous jar-files, which have been extracted to a versioned subdirectory within $JAR_LIB. As shown at the top of the slide, all jar-files related to the SpringFramework have been loaded into the spring-framework-1.2.2 subdirectory. We could have used util_JarLib_Register.pl to extract version information from each of the individual jar-files, but that isn’t necessary. When we install a new version of SpringFramework, we’ll just create a new, appropriately named top-level subdirectory of $JAR_LIB, and put the new jar files into it.

In other cases, version information may not be externally evident but may be available within the jar, in the META-INF/MANIFEST.MF file. As shown at the bottom of the slide, the MANIFEST.MF of our current version of commons-logging.jar indicates that the version is 1.0.4. Accordingly, the jar-file has been stored in a subdirectory named commons-logging-1.0.4.

Another interesting scenario is illustrated toward the middle of the slide: Vendors may create different implementations of the same spec; names of the associated jar-files may not indicate which implementation they contain. For example, developers of the WebLogic and Tomcat servlet containers both implemented the servlet-2.4 specification. The resulting jar-files are servlet.jar and servlet-api.jar, respectively. But the file names give no clue as to which is which. Of course, the necessary information is available in the jar-files’ respective MANIFEST.MF files, and this has been used to establish appropriately named subdirectories under servlet-2.4 for the two implementations.

OK, now we have a bunch of distinctly named subdirectories inside of $JAR_LIB – so, how does that help?

Well, have a look at the next slide …

17 build.template Æ build.properties

# # Source : Generic build.template file with replaceable tokens. # For Deployment To: @DEPLOYMENT@ environment #

# These are the same for all environments ... lib.root=/home/ptagdev/ptagis3/java_lib

lib.spring=spring-framework-1.2.2 lib.db=ca_jdbc_type_2 lib.log4j=log4j-1.2.9 lib.commons-logging=commons-logging-1.0.4 lib.tags_jstl=jstl-1.0.3 lib.tags_standard=jstl-1.0.6

lib.servlet=servlet-2.4/weblogic-8.1

db.driver=ca.edbc.jdbc.EdbcDriver

# These differ between deployment environments ... db.url=@JDBC_URL@ db.user=@JDBC_USER@ db.pw=@JDBC_PW@

Remember this from a few slides back? It’s the template that produces the build.properties file used by Ant.

It contains a set of name-value pairs, associating a logical-jar name with a physical-jar version – making use of the $JAR_LIB subdirectory names established by util_JarLib_Register.pl.

This file is tracked by SVN, so it provides a permanent record of the distinct version of every external jar used in every build of the project.

Further – using the build.properties file for the current build environment – Ant retrieves a full set of the specified jar-files whenever the ‘provision’ target of build.xml is executed. As may be seen in the following slide, build.xml refers to the jar’s by their logical-jar names, but the copy operation makes use of the corresponding physical-jar version

18 build.xml –“provision” Target

build.xml’s many other ‘targets’omitted for brevity

It is envisioned that the provision target would be run during initial project setup, to populate the project’s war/WEB-INF/lib with all jar-files needed for building. These would be imported into the project’s SVN repository and would subsequently be available to all of the project’s developers.

Only if a new version of a jar-file - or an additional jar-file – were needed, would the provision target be run again. Typically, this would not be done by a developer, rather by the person or persons responsible for building and deployment on the BETA-test and PRODuction environments.

19 New Project Setup

20 New Project Setup

21 22 23 24