17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

A Web Portal for CMS Grid Job Submission and Management

David Braun1 and Norbert Neumeister2 1 Department of Physics, Purdue University, W. Lafayette, IN 47907, USA 2 Rosen Center for Advanced Computing, Purdue University, W. Lafayette, IN 47907, USA E-mail: [email protected]

Abstract. We present a Web portal for CMS Grid submission and management. The portal is built using a JBoss application server. It has a three tier architecture; presentation, business logic and data. Bean based business logic interacts with the underlying Grid infrastructure and pre-existing external applications, while the presentation layer uses AJAX to offer an intuitive, functional interface to the back-end. Application data aggregating information from the portal as well as the external applications is persisted to the server memory cache and then to a back- end database. We describe how the portal exploits standard, off-the-shelf commodity software together with existing Grid infrastructures in order to facilitate job submission and monitoring for the CMS collaboration. This paper describes the design, development, current functionality and plans for future enhancements of the portal.

1. Introduction The experiments at the Large Hadron Collider (LHC) at CERN will start producing event data when the collider begins operating in late 2009. As the moment when real LHC data will start to be collected approaches, it is important to provide to the community of LHC physicists powerful and user-friendly tools to analyse data using distributed computing resources. One way to achieve this is by providing access to these resources via scientific portals that clients, LHC physicists, will find attractive and easy to use. Grid portals can deliver complex Grid solutions to users without the need to download, install and maintain specialized software, or worrying about setting up site-specific components. The goal is to reduce the complexity of the user Grid experience and to bring the full power of the Grid to physicists engaged in LHC analysis through a standard web GUI. Currently users are exposed to different flavors of grid middle ware and the installation and maintenance of experiment specific software is still very complex for most physicists. The available Grid tools are not so user friendly at times and the Grid requires even more software and services to be installed and maintained. Moreover, currently only few platforms are supported and the handling of Grid user certificates is still a problem for many users. We have evaluated different technologies to provide a science portal in order to reduce the complexity of the user Grid experience. Because this is a missing functionality of the official tool set of the CMS collaboration, we developed a prototype of a CMS Grid submission portal and we currently provide this service to a local CMS Tier-2 community at Purdue University. The goal of the CMS Grid submission portal is to hide and integrate the complex infrastructure details that can hinder a user’s ability to do science. A rich AJAX user interface

c 2010 IOP Publishing Ltd 1 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012 provides users the functionality to create, submit, share and monitor Grid submissions. The Grid portal is built on J2EE [1] architecture employing enterprise technologies powered by a JBoss [2] application server. This technology has been used for many years in industry to provide enterprise class application deployments. The architecture is comprised of three tiers; presentation, business logic and data persistence. The presentation layer currently consists of a Java Server Faces (JSF) [3] web interface developed with NetBeans [4] Visual Web Page development tools. The business logic layer provides interfaces to existing Grid infrastructure such as VOMS [5], Globus [6], and CMS specific Grid submission tools.

2. Technology Integrated authentication systems and the easy integration of multiple data sources are key considerations when designing an interactive web-based Grid application. These requirements naturally led to a design where the architecture of the application is based upon J2EE components such as Java Server Faces (JSF), Java Server Pages (JSP), Servlets and Enterprise Java Beans (EJB) [7]. JBoss was used as the application server of choice because it combines both a web container and an EJB container to provide a unified platform for development. It has been industry proven to scale quite well using distributed caching technology and can be further broken in smaller components as needed to match the needs of the application [8]. State management is provided at the request, session and application levels. The life cycle management built into JBoss provides the mechanism to persist state items and control the overall rendering pipeline. Making the portal web session interactive and auto-updating the data displayed to the end user via JavaScript and AJAX was important to the use case, but made the browser to server interactions increasingly complex. The development of a complex web application meant that technology decisions for this project needed to be made in a way that allowed us to best leverage the available programming resources in the research project. In order to exponentially increase the amount of development a single professional programmer could accomplish, the NetBeans IDE was chosen to automate much of the GUI interface code generation. Of particular interest when selecting development tools was the use of graphical tools to create web pages. For this project, NetBeans was chosen as the integrated development environment and employed the visual web pages plugin, which meant we were able to generate complex AJAX interfaces with minimal JavaScript having to be hand written. Visual web pages are based on the use of JSF to supply a component based framework that is similar to the long existing tools that are used to layout Java swing user interfaces. This project was built on the Woodstock framework [9], which is the reference framework developed by Sun and the Java community. The motivation for this choice was the fact that the NetBeans visual web page development environment supported all the features in this framework. The advantage of using frameworks like this is that most of the cross browser issues are handled within the component or the life cycle management and do not have to be hand coded by the programmer. JSF is a framework that is composed of three major functions; GUI component, state management, and life cycle management. The GUI component items, such as drop down menus, include encapsulated code that specifies how the component should be rendered. Depending on the target, browser components can choose the correct JavaScript to send to the browser which eliminates the need for the typical testing of multiple browser targets. JSF employs the Expression Language (EL) [10] which allows the components to dynamically consume evaluated expressions that are used to bind component properties to state management items. Within the JSF specification state management is implemented as three distinct scopes: request, session, and application. Each scope will persist data for different periods of time ranging from page to page navigation, user sessions and global application. The lifecyle management provides support for how the component tree is restored, associated state is updated and completing the request

2 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012 with a generated response, as well as mechanisms to handle complex user interactions such as browser navigation.

3. Architecture The architecture of the CMS Grid Submission Portal is divided into three main pieces: presentation, logic and data layers. The presentation and logic layers are within a JBoss application server and the data layer is MySQL [11] and file store. Within the JBoss container, we deploy beans which wrap the CMS Remote Analysis Builder (CRAB) [12], allowing users to execute the CRAB functions they need from the portal web page. Figure. 1 shows a diagram of the chosen architecture. The presentation layer is implemented with JSF technology. The JSF servlets are managed within a JBoss managed Tomcat [13] container. The advantage of doing this is that the container is configured to support a unified security model such that all members of the architecture within the application server exist under one security domain. There are two main backing beans that the application relies upon, the application and user session bean. The application bean is used to hold application-wide state. In this case the state of the application bean is also being backed up in the lower layers. The state that is being managed at the application level is an in memory state that is required for fast response for polling queries, or static elements choice lists. The general use case of the application has three main steps: the user defines a project space, configures and loads required data, and then submits and monitors jobs using CRAB. The user’s project is based on a release of CMSSW and may contain private code/libraries and configuration files. Data is extracted from the CRAB submission directory about the project and cached in a database. These actions all require that a valid proxy exists within the user’s directory as an additional security measure. There are two active components within the architecture that will act upon the user’s data autonomously: proxy expiration checking and project monitoring. A periodic proxy check will validate that the proxy has not expired. If an expiration is detected, the proxy is deleted and the user is notified. Project monitoring occurs when a user selects a project for monitoring; this will schedule background processing for that user to update the status of the CRAB project. As portals become more heavily used and traffic increases on the server, managed lateral scaling is required. JBoss can easily be configured as a collection of servers in a cluster fashion to meet this requirement, making it the best architectural choice, not only for current needs, but with an eye towards future growth. The developer does not have to develop code that will implement the clustering functionality as long as the specifications of the EJB interfaces are upheld. There are some use cases, for example persistent open files, which fall outside the specification. In this case depending on the application server, managed pinning can be employed. Pinning within a cluster means that there is exactly one resource defined. In the case of JBoss what this means is that at the Java Naming Directory Interface (JNDI) [14] layer requests for a server are dynamically mapped to the cluster member that has the resource. So why all of this complexity with beans, beans and more beans? This is where the features and services of the application server play the biggest role. The data associated with a project is implemented using Enterprise Java Beans (EJB) technology. EJB technology is a way to encapsulate both logic and data. There are two major types of EJBs, Session and Entity. The advantage of using EJB technology for data is that data can be encapsulated as an annotated Java class which the application server maps and for which it manages the database transactions based upon the presented annotations. This type of mechanism provides insulation from the underlying data layer changes. If changes are required that are beyond what is specified in the code, the application server can change the mapping without a required code change with additional specifications. Session beans encapsulate the logic of an application. In the current architecture, session

3 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

Figure 1. Architecture of the portal beans are used as a facade to an application server specific bean called a Service Bean. The application was implemented this way because during development core functions could be evaluated via a Java Management Extension (JMX) [15] interface. In subsequent revisions, the functions will be migrated to the Session Bean layer.

4. Grid security Security is a necessary evil within the Grid world. Users who are not used to the current certificate model can find it challenging to manage certificates. Users have become more and more accustomed to using the nearest computer at hand, but this requires users to install keys on foreign machines in order to interact with the Grid environment. One popular solution is to install keys on USB drives and use them as necessary on foreign machines. This method is sufficient for the certificate savvy user, there is a growing population of users that would be better served if the keys could be managed for them. Remote Grid proxy generation is not a new concept. Currently the TeraGrid [16] employs a single sign on system that allows users to retrieve a proxy based upon the TeraGrid user portal name and password. The proxy that is returned is limited in the fact that it cannot be used in a delegation situation. At the time there was not a solution like this for OSG [17] that was able to use VOMS proxy init. In order to demonstrate one possible solution to remotely managing keys and proxies, we implemented a solution that allows a user to upload their keys to the server in much the same way that we would normally do if we were logging into a cluster head node. Handling keys is a very sensitive issue because of all the trust issues that arise. The keys are uploaded to a Java keystore [18] that is located on the server. The requirement that the keystore have an additional passphrase insures that only the user can retrieve the keys from the keystore. As part of the upload process keys are checked to make sure that they too have a passphrase. At no time are the passphrases stored or cached within the system. When the user needs to generate a proxy, the user must supply both passphrases to retrieve the keys that are locally held in memory during the proxy generation using Globus Cogkit [19]. The generated Grid proxy is stored within the user’s directory – again protected by the system.

4 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

Figure 2 shows a diagram of security implementation where the keys are protected by three layers: frontal access , encrypted keystore, and system level security. The portal users must provide credentials for access. Access can be a user name password or the presence of a registered client certificate. When the users upload their keys to the system the keys are stored in a keystore which resides on the local file system.

Figure 2. Security diagram

The portal implements a client certificate fallback strategy for user authentication. When a connection is established, the server asks the browser for a client certificate. If the client fails to present a certificate, authentication falls back to a user name password authentication system. This technology removes the need to have a separate link provided to users to determine if the browser they are currently using has their certificates installed or not.

5. User experience 5.1. Home page The home page of the portal contains a Google RSS reader [20] that can aggregate many RSS feeds. The intent here is to provide users with additional information about what is going on within the community at large or information about changes to the portal environment. Figure 3 is a screen shot of the home page. The user can be presented with additional information about the portal and other aggregated information. The homepage can also be configured to stand alone, allowing for easy redirects in the case of server migration or maintenance.

5.2. Registration User registration allows users to define their identity within the application. At the time of the portal’s implementation where wasn’t a unified identity management system. User registration in the CMS Grid Submission Portal has been constructed in such a way that this is possible

5 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

Figure 3. Home page of the portal to implement. The user registration is protected by a user certificate fallback security domain that has been modified to acquire information and assume that all users are authenticated. The secure check provides a mechanism to acquire the user’s Distinguished Name (DN) from their presented x509 certificate. If no certificate is presented then the option of registering the DN is not visible to the user.

5.3. Login When users click on the start link, they are directed to the main application page. During this time the security domain requests the client certificate. If the certificate is valid, the user is allowed to pass to the main application. If no certificate is presented or the certificate is invalid, then the user is redirected to the login page for a user name/password authentication.

5.4. Profile A user can access its profile information and personalize some of the basic setups for CRAB job submission. Users can modify the various profile data items, then use these items as macros within the CRAB and CMSSW submission templates. When a user defines a submission template, the template will be used as the default when a new project is created.

5.5. Projects Projects are the combination of user CMSSW code/libraries, CRAB and CMSSW configuration files and the CRAB working directory. When users create a new project they specify the CMSSW version that will be used for the project. If they choose to upload the required configuration files they can do so at this time, or the default configurations can be used instead. If a user has defined a user-specific template then it will be used. If they have not defined a template, the system default template will be used. At this point the user can choose to go to a configuration wizard that has a tabbed pane where the parameters of the crab.cfg has been broken out in a way that is easier to define in the default values. Figure 4 shows the main project area that users

6 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012 are presented with at login. This area provides a list of the user’s defined projects. The user can choose to monitor, share or delete a project. Additional information about the project (e.g. version) is displayed in this view. Clicking on the project name directs the user to the project editor where the user can further work with the details of the project and issues commands. The user can create a new project by clicking on the new project button, which will direct them to a project creator. The user can specify a project name, CMSSW version, and upload configuration files. At this point the more experienced user can just continue to the project editor page. Users who need more guidance can navigate to the configuration wizard which will guide them through the process. The project editor allows the user to view and edit the configuration files, view the file system and submit CRAB commands. Once the user has defined a project, it can be shared with other portal users. Users can click on the project name to retrieve additional information before they decide to clone a project. Sharing projects allows users to easily collaborate within the portal environment, as shared projects can be cloned by all users of the portal community. Users have the ability to submit to the CRAB server from two different points, using the command line version of CRAB and through the portal. To support requirements for transportable projects, the portal can clone projects that have been submitted to the local Purdue CRAB server as well as projects submitted through this portal.

Figure 4. Project overview

Once a project is submitted it can be selected to be monitored. Periodically the status of the project is updated by an active server side component and the status is merged with the current state of the project and displayed to the user. Monitoring will continue until all the jobs are in a terminal state or the user’s proxy expires.

7 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

5.6. User proxies One of the most important parts of implementing a keystore management system was to increase the usability for the non-expert. We will describe what the users are presented with as they define a keystore and generate a proxy. When users log in, they start at the main entry point. Contained within the header bar of the main CMS Grid Submission Portal page there are two icons, a key with a red X and a blinking red LED. The key is the icon for the keystore and the red LED is the icon indicating that there is either no proxy or the proxy has expired. The user clicks on the key, and is taken to the keystore manager where they can either destroy an existing keystore or load their keys to create a new one. If a keystore is created they are taken back to the main project area where a key icon without the X replaces the icon with an X. The user can now click on the LED and is taken to the proxy generation page. On the left they are given choices either to init a proxy from the keystore if it’s defined or to fetch a proxy from a remote myproxy server. Choosing to init the proxy from the keystore will take them to a page that prompts for the keystore passphrase, the key passphrase and a lifetime. If all the fields are correct, a proxy is generated, the LED now turns to a green state. During this process the proxy is also stored on the myproxy server at CERN. As the proxy’s lifetime expires, the LED is changed to a yellow state indicated a warning condition to the user. At any time the user can click on the LED and is taken a proxy information page that will display details about the proxy. At the present time, the proxy generation is based upon the glite-security-voms.jar [21]package. The package has been wrapped in a secondary API called ProxyTool which contains a collection of initialization routines for Globus CogKit and VOMS as well as static methods to manipulate the keystores, Globus proxies, and myproxy interfaces.

5.7. File upload Sometimes a project requires many files to be uploaded. Typically these files are either configuration files or additional libraries that are linked into the CMSSW analysis framework during runtime. Users have the ability to upload files in bulk through the use of a modified JUpload [22] applet. Users can select single files or an entire directory of files that will be uploaded to the defined project directory. The files are uploaded to the correct directories within a project area to ensure that CRAB will function correctly. In addition to uploading, the user can browse the project file tree to ensure that all files have correctly been uploaded and remove extra files that may have been included in the bulk upload process. Figure 5 shows the applet uploading files to the project area.

5.8. External tool integration Integration will become very important in the future as portals like this one continue to sprout up to support the users’ demands for simple web based tools that allow them to perform their job functions. This portal project has demonstrated three forms of external tool integration, direct, interface and inclusion. The CRAB and CRAB server integration demonstrate direct integration where either a command line process was spawned or where the file system was accessed. This type of integration requires the development of process management and methods of interaction with the spawning process. The portal integrates CRAB via a command line interface, which required the addition of a synchronization method to ensure that the automatic monitoring processes and the user issued commands did not collide. The cloning of CRAB server projects uses an internal Java tar package [23] to access the project archive rather than spawning an additional process which would have to be managed. Interface integrations are centered around the idea that there is a published interface of some kind that allows an application to use an external service without a change in the technology

8 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

Figure 5. File upload base. The VOMS integration is a demonstration of interface integration where a remote server is accessed. This type of integration will be the most important type of integration for the future projects because it reduces the complexity for the portal developer. In the best case the use of standards further reduces the complexity of the integration efforts. Inclusion is where external applications are included into an application with no modification. An example of inclusion was the use of frames to include the SiteDB web page. Though inclusion works, it is not ideal because it simply places the external component within the current environment in much the same way we would reference a web page. The included content, though aggregated, is subject to its one implementation on security and typically has no method of further interaction such as setting field parameters.

6. Future developments With most of the core functionality implemented, future development may include a SRM [24] file browser, a Google Gadget [25] job status monitor, or further JSON-RPC [26] integration with CRAB server. Integration with external tools will mainly be focused around using published interfaces to include more services within this user environment. We also plan on moving to a different user interface framework such as ICEFaces [27] or GWT [28] to increase the amount of dynamic user interaction, as these alternative frameworks will better support an increased load of interactive components.

9 17th International Conference on Computing in High Energy and Nuclear Physics (CHEP09) IOP Publishing Journal of Physics: Conference Series 219 (2010) 072012 doi:10.1088/1742-6596/219/7/072012

Acknowledgments We would like to thank Haiying Xu and the local Purdue CMS Tier-2 users for testing the portal and giving us valuable feedback.

References [1] J2EE, http://java.sun.com/javaee/ [2] JBoss http://www.jboss.org [3] JSF, http://java.sun.com/javaee/javaserverfaces/ [4] Netbeans, http;//www.netbeans.org [5] VOMS, http://infnforge.cnaf.infn.it/projects/voms [6] Globus, http://www.globus.org [7] EJB, http://java.sun.com/products/ejb/ [8] JBoss Scaling, http://www.infoq.com/news/2008/06/jboss-cache-interview [9] Woodstock, https://woodstock.dev.java.net/index.html [10] Unified Expression Lanaguage (EL), http://java.sun.com/products/jsp/reference/techart/unifiedEL.html [11] MySQL, http://www.mysql.com [12] Spiga S 2007, CRAB (CMS Remote Analysis Builder), Proc. of Int. Conf. CHEP 2007 [13] Tomcat, http://tomcat.apache.org [14] JNDI, http://java.sun.com/products/jndi/ [15] JMX, http://java.sun.com/javase/technologies/core/mntr-mgmt/javamanagement/ [16] TeraGrid Science Gateways, http://www.teragrid.org/gateways/ [17] Open Science Grid, http://www.opensciencegrid.org/ [18] Java Keystore, http://java.sun.com/javase/technologies/security/ [19] Globus Cogkit, http://www.cogkit.org [20] Google RSS Reader, http://www.google.com/reader/ [21] GLITE, http://glite.web.cern.ch/ [22] JUpload Applet, http://jupload.sourceforge.net/ [23] Java Tar Package, http://www.trustice.com/java/tar/ [24] SRM, http://sdm.lbl.gov/srm-wg/ [25] Google Gadget, http://code.google.com/apis/gadgets/ [26] JSON RPC, http://oss.metaparadigm.com/jsonrpc/ [27] ICEFaces, http://www.icefaces.org [28] Google Web Toolkit (GWT), http://code.google.com/webtoolkit/

10