Task 3.3 Grid Monitoring M15 Deliverable

W P 3 . 3 G R I D M O N I T O R I N G M 1 5 D E L I V E R A B L E S O F T W A R E E V A L U A T I O N A N D T E S T I N G

WP3 New Grid Services and Tools

Document Filename: CG3.3-D3.4-v1.4-TCD016-M15Deliverable.doc

Work package: WP3 New Grid Services and Tools

Partner(s): TCD, CYFRONET, ICM

Lead Partner: TCD

Config ID: CG3.3-D3.4-v1.4-TCD016-M15Deliverable

Document classification: Confidential

Abstract: This document is an internal progress report on WP3, Task 3.3 grid monitoring, software evaluation and testing.

CG3.3-D3.4-v1.4-TCD016-M15Deliverable Confidential 1 / 61 TASK 3.3 GRID MONITORING M15 DELIVERABLE

Delivery Slip

Name Partner Date Signature From WP3, Subtask 3.3 TCD May 30th, 2003 Brian Coghlan

Verified by

Approved by

Document Log

Version Date Summary of changes Author Bartosz Baliś, Brian Coghlan, Stuart Kenny, Krzysztof Nawrocki, Adam 1-0 10/04/2003 Draft version Padee, Marcin Radecki, Tomasz Szepieniec, Slawomir Zielinski Bartosz Baliś, Brian Coghlan, Stuart Draft version. Updated after TAT Kenny, Krzysztof Nawrocki, Adam 1-1 02/05/2003 review. Padee, Marcin Radecki, Tomasz Szepieniec, Slawomir Zielinski Bartosz Baliś, Brian Coghlan, Stuart Draft version. Updated after further Kenny, Krzysztof Nawrocki, Adam 1-2 08/05/2003 comments. Padee, Marcin Radecki, Tomasz Szepieniec, Slawomir Zielinski Bartosz Baliś, Brian Coghlan, Stuart Draft version. Updated risk Kenny, Krzysztof Nawrocki, Adam 1-3 26/05/2003 assessment. Padee, Marcin Radecki, Tomasz Szepieniec, Slawomir Zielinski Bartosz Baliś, Brian Coghlan, Stuart Final version. Updated after internal Kenny, Krzysztof Nawrocki, Adam 1-4 30/05/2003 review. Padee, Marcin Radecki, Tomasz Szepieniec, Slawomir Zielinski

CONTENTS 1 REFERENCES...... 6

2 EXECUTIVE SUMMARY...... 8

3 INTRODUCTION...... 9

3.1 DEFINITIONS, ABBREVIATIONS, ACRONYMS...... 11 4 STATE OF THE ART...... 12

4.1 APPLICATIONS MONITORING...... 12 4.1.1 OCM-G...... 12 4.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 14 4.2.1 Instruments: SANTA-G...... 14 4.2.2 Infrastructure: JIMS...... 15 4.2.3 Derived Results: Postprocessing...... 17 5 CONTRIBUTIONS TO GRID TECHNOLOGY...... 19

5.1 APPLICATIONS MONITORING...... 19 5.1.1 OCM-G...... 19 5.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 20 5.2.1 Instruments: SANTA-G...... 20 5.2.2 Infrastructure: JIMS...... 21 5.2.3 Derived Results: Postprocessing...... 22 6 BRIEF DESCRIPTION OF THE SOFTWARE...... 23

6.1 APPLICATION MONITORING...... 23 6.1.1 OCM-G...... 23 6.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 23 6.2.1 Instruments: SANTA-G...... 23 6.2.2 Infrastructure: JIMS...... 24 6.2.3 Derived Results: Postprocessing...... 25 7 AIMS, TESTS AND EVALUATION, NEW REQUIREMENTS...... 26

7.1 APPLICATIONS MONITORING...... 26 7.1.1 OCM-G...... 26 7.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 27 7.2.1 Instruments: SANTA-G...... 27 7.2.2 Infrastructure: JIMS...... 28 7.2.3 Derived Results: Postprocessing...... 28 8 RESULTS OF THE TESTS AND EVALUATION...... 29

8.1 APPLICATIONS MONITORING...... 29 8.1.1 OCM-G...... 29 8.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 29 8.2.1 Instruments: SANTA-G...... 29 8.2.2 Infrastructure: JIMS...... 29 8.2.3 Derived Results: Postprocessing...... 29 9 PROBLEMS AND ISSUES...... 30

9.1 APPLICATIONS MONITORING...... 30 9.1.1 OCM-G...... 30 9.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 30 9.2.1 Instruments: SANTA-G...... 30 9.2.2 Infrastructure: JIMS...... 31 9.2.3 Derived Results: Postprocessing...... 31

10 FUTURE PLANS...... 32

10.1 APPLICATIONS MONITORING...... 32 10.1.1 OCM-G...... 32 10.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 32 10.2.1 Instruments: SANTA-G...... 32 10.2.2 Infrastructure: JIMS...... 33 10.2.3 Derived Results: Postprocessing...... 35 11 TOWARDS OGSA...... 36

11.1 APPLICATIONS MONITORING...... 36 11.1.1 OCM-G...... 36 11.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 36 11.2.1 Instruments: SANTA-G...... 36 11.2.2 Infrastructure: JIMS...... 36 11.2.3 Derived Results: Postprocessing...... 37 12 RISK ASSESSMENT...... 38

12.1 APPLICATIONS MONITORING...... 38 12.1.1 OCM-G...... 38 12.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS...... 43 12.2.1 Instruments: SANTA-G...... 43 12.2.2 Infrastructure: JIMS...... 46 12.2.3 Derived Results: Postprocessing...... 50 13 CONCLUDING REMARKS...... 55

14 APPENDIX A...... 56

1 REFERENCES Bal2000 Z. Balaton, P. Kacsuk, N. Podhorszki, F. Vajda, Comparison of Representative Grid Monitoring Tools, http://www.lpds.sztaki.hu/publications/reports/lpds-2-2000.pdf CrossGrid CrossGrid Project Technical Annex, http://www.eu-crossgrid.org/CrossGridAnnex1_v31.pdf DataGrid DataGrid Project Technical Annex DataGridPart_B_V2_51.doc Ganglia http://ganglia.sourceforge.net/ GRIDLAB GridLab Project Home Page http://www.gridlab.org GRADS Grid Application Development Software Project Home Page http://nhse2.cs.rice.edu/grads/ Jiro http://www.jiro.org/ Jiro D3.3 Jiro Based Grid Infrastructure Monitoring System, First Prototype Description, part of D3.3 http://www.eu-crossgrid.org/Deliverables/M12pdf/CG3.3.3-CYF-D3.3-v1.1-Jiro.pdf Jiro D3.2 Jiro Software Design Document http://tulip.man.poznan.pl/doc/dd-04-09-2002/3.3/CG3.3.3-D3.2-v1.1-CYF022- JiroDesign.pdf Jiro Tech Jiro Technology Installation and Configuration Guide, © Sun Microsystems JMX Spec Java Management Extension Specification, © Sun Microsystems, http://java.sun.com/products/JavaManagement/ KaTools P. Augerat, C. Martin, B. Stein, Scalable monitoring and configuration tools for grids and clusters, http://ka-tools.sourceforge.net/publications/ka-admin-pdp02.pdf NWS http://nws.cs.ucsb.edu/ OCM A Monitoring System for Interoperable Tools http://wwwbode.cs.tum.edu/~omis/Docs/spdt98.ps.gz OCMGD3.3 Description of the OCM-G first prototype, part of CrossGrid deliverable 3.3. http://www.eu-crossgrid.org/Deliverables/M12pdf/CG-3.3.1-CYF-D3.3-v1.0-OCM- G.pdf OGSA The Physiology of the Grid: An Open Grid Services Architecture for Distributed Systems Integration. I. Foster, C. Kesselman, J. Nick, S. Tuecke, January 2002. http://www.globus.org/research/papers/ogsa.pdf OGSA-DAI http://www.ogsadai.org.uk/ OMIS OMIS – On-line Monitoring Interface Specification. Version 2.0. Lehrstuhl für Rechnertechnik und Rechnerorganisation Institut für Informatik (LRR-TUM), Technische Universität München. http://wwwbode.informatik.tu-muenchen.de/~omis/ RFC 1213 K. McCloghrie, M. Rose (ed.), Management Information Base for Network Management of TCP/IP-based Internets: MIB-II RGIS-RG Relational Grid Information Services Research Group http://hepunx.rl.ac.uk/ggf/rgis-rg/ R-GMA R-GMA: A Relational Grid Information and Monitoring System

http://hepunx.rl.ac.uk/edg/wp3/documentation R-GMA ARCH DataGrid Project Deliverable 3.2 DataGrid-03-D3.2-0101-1-0 http://hepunx.rl.ac/edg/wp3/documentation/ RGMA-OGSA A presentation describing the R-GMA transformation to web-services https://edms.cern.ch/file/353606/1/GGF5-2.pdf SANTA Prototype Trace Probe and Probe Adaptor and Prototype Trace Software, http://www.cs.tcd.ie/Brian.Coghlan/scieuro/scieuro.html, Esprit Project 25257, SCI Europe, Deliverables, Trinity College Dublin, 1999 SCITRAC B. Skalli, I. Birkeli, B. Nossum, and D. Wormald, Scitrac – an lsa preprocessor for sci link tracing, in Scalable Coherent Interface: Technology and Application, 1998. SMiLE Jie Tao, A low-level Software Infrastructure for the SMiLE Monitoring Approach http://wwwbode.cs.tum.edu/archive/smile/scieur2001/scieuro2001-s2-p2.pdf SNORT Open-source network intrusion detection system, http://www.snort.org/ Spitfire Grid enabled middleware service for access to relational databases, http://spitfire.web.cern.ch/Spitfire Task3.3 SRS Task3.3 Grid Monitoring Software Requirements Specification Task3.3-SRS.pdf http://www.eu-crossgrid.org/Deliverables/M3pdf/Task3.3-SRS.pdf Task2.4 SRS Task2.4 Interactive and semiautomatic performance evaluation tools CG-2.4-DOC-CYFRONET001-SRS.pdf http://www.eu-crossgrid.org/Deliverables/M3pdf/CG-2.4-DOC-CYFRONET001- SRS.pdf TCPDump http://www.tcpdump.org/ TEK http://www.tek.com TopoMon M. den Burger, T. Kielmann, H. E. Bal, TopoMon: A Monitoring Tool for Grid Network Topology, http://www.gridlab.org/Resources/Papers/iccs02.pdf WP3 Inst WP3 Installation Guide http://alpha.ific.uv.es/~sgonzale/integration/WP3-Installation-guide.doc

2 EXECUTIVE SUMMARY This document is the month 15 deliverable for Task 3.3, Grid Monitoring. It forms part of an internal progress report on WP3 software evaluation and testing. Section 3 is the introduction, and also provides definitions, abbreviations and acronyms. Section 4 describes the current state of the art, while Section 5 provides a description of the contribution of Task 3.3 to current grid technology. Section 6 provides a brief description of the software. The aims of the prototypes and the tests are described in Section 7, and the results of these tests are given in Section 8. Any problems and issues discovered with the prototypes are described in Section 9. The plans for Task 3.3 for the next year are detailed in Section 10. Section 11 describes the plans of Task 3.3 regarding preparing for OGSA. A detailed risk analysis for the task is given in Section 12. Section 13 contains concluding remarks.

3 INTRODUCTION According to the technical annex [CrossGrid], it was proposed that within Task 3.3 a prototype infrastructure for the needs of monitoring-related activities for automatic extraction of high-level performance properties and for tool support of performance analysis would be developed. The aim of the Grid monitoring facilities provided by the Grid Monitoring task is to underpin the functionality of various tools by delivering the low-level data intended for the above. During the design phase system scalability, flexibility and ease of configuration were the guidelines. These are still the guidelines now during the implementation phase.

For the Grid Monitoring task we deliberately restricted our scope. We chose to both add new services (for applications monitoring) and to extend existing services (for monitoring instruments and infrastructure, and for generating derived-results), see Figure 3.1. Analysis has shown that the new applications monitoring was substantially different from monitoring infrastructure and instruments - they needed two separate approaches:

 Infrastructure monitoring collects static and dynamic information about Grid components, such as hosts or network connections; this information is indispensable for basic Grid activities as resource allocation or load balancing; often this type of information has not only immediate, but also historic value, thus it is often stored in a database for a later analysis (e.g., statistical, forecasting, etc.),  Application monitoring aims at observing a particular execution of an application; the collected data is useful for tools for application development support, which are used to detect bugs, bottlenecks or just visualize the application's behaviour; this kind of information in principle does not have historic value – it is meaningful only in the context of a particular execution.

Infrastructure monitoring

R-GMA/OGSA info

Application monitoring Figure 3.1: The Grid Monitoring System

For applications monitoring we chose to:  proceed in an OCM-compliant route, for compatibility with WP2  implement direct low-latency communications

For extending existing grid monitoring services we chose to:  deliberately avoid a hierarchical (e.g. LDAP) approach  proceed in an OGSA-compliant route, using R-GMA as the interim web-based service  implement instrument monitoring as part of R-GMA  implement infrastructure monitoring using Jiro tools that already support infrastructure protocols and dynamic system configuration, with import/export from/to other services  explore the potential of Jiro techniques for higher levels of grid monitoring  consider results derived from monitoring data as just another stream of monitoring data  produce data for external consumption via a single web-based API (initially R-GMA)  closely monitor OGSA developments

3.1 DEFINITIONS, ABBREVIATIONS, ACRONYMS CrossGrid The EU CrossGrid Project IST-2001-32243 DataGrid The EU DataGrid Project IST-2000-25182 FMA Federated Management Architecture, defined by Sun Microsystems GUI Graphical User Interface JIMS Jiro/JMX-based Grid Infrastructure Monitoring System Jiro SUN Jiro, implementation of the FMA specification LDAP Lightweight Directory Access Protocol LDAP DIT LDAP Directory Information tree MDS Monitoring and Discovery Service NWS Network Weather Service OCM OMIS-Compliant Monitor OCM-G Grid-enabled OMIS-Compliant Monitor OGSA Open Grid Services Architecture OGSA-DAI OGSA Data Access and Integration OMIS On-line Monitoring Interface Specification RDBMS Relational Database Management System RGIS-RG Relational Grid Information Services – Research Group R-GMA DataGrid relational Grid monitoring architecture

SANTA System Area Network Trace Analysis SANTA-G Grid-enabled System Area Network Trace Analysis SMiLE Shared Memory in a LAN-like Environment SNORT Open source network intrusion detection system SOAP Simple Object Access Protocol SRS Software Requirements Specification WSDL Web Services Description Language

4 STATE OF THE ART

4.1 APPLICATIONS MONITORING

4.1.1 OCM-G In this section, we provide an overview of three grid application monitoring approaches currently being developed, which are similar to the OCM-G approach. We try to compare the presented approaches to our own – OMIS/OCM-G based. The mentioned projects/systems are as follows: GrADS (Au- topilot), GridLab, DataGrid (GRM).

 GrADS

The Grid Application Development Software (GrADS) project develops a software architecture designed to support application adaptation and performance monitoring. The GrADSoft architecture (program preparation and execution system), replaces the discrete steps of application creation, compilation, execution, and post-mortem analysis with a continuous process of adapting applications to both a changing Grid and a specific problem instance.

The GrADSoft monitoring infrastructure is based on Autopilot, a toolkit for application and resource monitoring and control. The monitoring is based on sensors, which may be put directly into source code or embedded in the application library. The sensors register in Autopilot Manag- er and can then be accessed by sensor clients to collect information. The clients can be located anywhere on the Grid. Modification of the application behavior is achieved by executing actua- tors, which are implemented by the user in the source code.

There are predefined sets of sensors and sensor clients, but the user can also implement them man - ually with an API provided by Autopilot. This is, however, not very convenient. Sensors can include user-defined code for data pre-processing. Although it is flexible, the processing is fixed at compile time, while in the OCM-G it is defined at run-time.

More important is that the Autopilot toolkit works within an enclosed framework, where definition of measurement, compilation and performance tuning is done automatically, thus the users do not have a detailed insight into their applications. In our approach, we focus on providing the user with exact knowledge of application performance in a particular execution, e.g., the user can flexi- bly specify what information he needs.

Although the Autopilot toolkit is a mature and interesting application monitoring approach, it is oriented more towards automatic steering, than to providing feedback to the programmer. It gives a rather general view of application and environment, e.g., to explore patterns in behavior instead of particular performance loss. Based on those patterns and external knowledge (e.g. user’s experi- ence), a performance optimization action is taken. It suits well the situation of a run-time system, where a special compiler creates a program, which is then automatically reconfigured at run-time depending on discovered patterns, e.g., the I/O buffer is resized for certain operations to improve performance.

The GrADS project differs from ours in its goal: it aims to provide tools that will free the user from many low-level concerns, permitting greater focus on the high-level design and tuning of

programs for a heterogeneous distributed computing environment, while our goals are to provide the user with exact, low-level knowledge of the application internals and behavior.

 Gridlab

The application monitoring system developed within the GridLab project implements on-line steering guided by performance prediction routines deriving results from low level, infrastructure- related sensors (CPU, network load). GridLab proposed a protocol for high-level producer-consumer communication. The protocol has three types of messages: commands, command responses and metric values. A consumer can authenticate, initiate a measurement, and collect data. Addi- tionally, the GridLab team proposed a standard set of metrics along with their semantics.

However, this approach is limited in its functionality and flexibility in comparison with OMIS/OCM-G. First, it does not allow for manipulations on the target application. Second, the approach seems to rely only on full traces, at least the currently available documents do not mention anything about the possibility of data preprocessing. Finally, the predefined semantics of all metrics requires all Grid users to agree on this, which is rather restrictive.

In contrast, the OCM-G provides a wide variety of manipulation services. It also supports advanced mechanisms for data buffering and preprocessing, which can be controlled directly by the user in a flexible way. For example, in the GridLab approach, buffering of data is just turned on by a proper command, while in the OCM-G the user can explicitly create and manipulate, e.g., counters, and has a better control over the buffering policy. Finally, the OMIS/OCM-G approach does not impose any predefined semantics on metrics. Instead, it provides a large set of services via which the user can construct the metrics in a flexible and easy way. The latter feature is essential for the High Level Component Architecture (HLAC) component, being developed as part of the G-PM tool (Crossgrid Task 2.4), since it allows the user to define his own metrics with the semantics he needs.

 DataGrid

The GRM monitor and PROVE visualization tool, being developed in the Datagrid project, origi- nate from the cluster environment, in which they work as part of P-GRADE, an environment for parallel program development. These tools have been separated from P-GRADE and modified to work on the Grid. The GRM is a semi-on-line monitor which collects information about applications and delivers this information to the PROVE visualisation tool. According to authors, semi- on-line mode means that only observation but not manipulation or data processing is possible, thus it is not a real on-line monitor. On the other hand though, traces are not stored in local files as in off-line approaches, but are just buffered locally.

While the GRM/PROVE environment is well suited for the DataGrid project, where only batch processing is supported, it is less usable for the monitoring of interactive applications.

First, monitoring in the GRM/PROVE is mainly based on event tracing. However, event tracing has some serious drawbacks. Most important is that achieving low latency and low intrusion at the same time is basically impossible when monitoring is based on trace data. If the traces are buffered, the latency increases, if not, the overhead for transmitting the events is too high. This means that a tool will not be able to provide an update rate that is high enough to meet the requirements of an on-line visualization, especially for interactive applications.

Second, in order to perform an accurate measurement, it is especially important that the tool is provided with information as soon as possible, thus we should not use any high-latency communi-

cation infrastructure like R-GMA's Java servlets. Moreover, low latency is essential for interactive applications such as those developed in CrossGrid, since they need a short response time. This again requires lightweight communication solutions such as the direct TCP/IP, which is used in the OCM-G.

Third, the GRM does not support programmable, distributed data processing, that is, it has a fixed functionality. With the OCM-G, one can tell it at run-time what should be done if an event occurs. This is extremely important as it enables user-defined metrics (see also the GridLab paragraph).

A fourth disadvantage of the GRM/PROVE is instrumentation, which is not automatic but requires the user to instrument the whole program manually. In the OCM-G, we provide pre-instrumented libraries, thus the user does not have to care about it, and he is not bound to any specific tool that performs the instrumentation of the source code. More important, however, is that the instrumentation in the OCM-G is selective: initially all instrumentation is inactive. The overhead in this case is practically zero since only one additional memory read and a compare operation is performed per event. The user at run-time can activate selected events and deactivate them again. In contrast to the source code instrumentation, this approach allows for dynamic run-time selection and modification of interesting data / measurements.

4.2 INSTRUMENTS, INFRASTRUCTURE, DERIVED RESULTS

4.2.1 Instruments: SANTA-G There are a number of instruments available that provide the ability to perform hardware monitoring of resources, for example, the instruments SCITRAC and SCIview , delivered by SCILAB Technology for non-invasive tracing on SCI links [SCITRAC]. The SANTA project uses these instruments to collect deep traces of high speed interconnect traffic into a relational database [SANTA]. The SMiLE project at LRR-TUM uses a complimentary monitoring approach designed to observe the network transactions and to provide detailed information about the run time memory access behaviour of parallel applications; their prototype hardware monitor is implemented through FPGAs and a hardware object technology [SMiLE]. At a more general level, a number of manufacturers supply a variety of acquisition instruments, for example, the Tektronix 1503C, the Tektronix TS100, the Tektronix SD24, the Tektronix TFP2A and TFS3031, and the Tektronix TOP300 [TEK], also from Hewlett Packard the HP54753 and their HP54750A, the Hewlett Packard HP83480A, HP8147 and HPE6000A. None of the above however are grid enabled. SANTA-G’s generic framework, based upon the Canonical Producer, allows for information captured by a hardware monitor to be introduced into the grid information system. This is a new approach. The Canonical Producer, an integral part of the DataGrid R-GMA [R-GMA], is the generic user customizable R-GMA producer. R-GMA is relational. R-GMA was chosen over other information systems as we felt it compared favourably to the state of the art. This includes grid information systems such as MDS and an OGSA-based system. MDS is based on a hierarchical LDAP structure. OGSA is web-services oriented and uses standard web protocols such as SOAP and WSDL. The OGSA-DAI [OGSA-DAI], which has just been released, is semi-structured, using XML. It is only partially relational. One of the main drawbacks with the LDAP model is its lack of support for queries that combine information across objects of different class. In relational terms it does not support the join concept. This means it is only possible to answer queries for which the tree structure has been defined. Also the LDAP DIT is normally centrally organised and there is no provision for a user being able to define and publish their own data [R-GMA]. The most obvious drawback with OGSA systems was that there was previously no implementation available. The Globus Toolkit version 3, which is

based on these concepts, was only released as an alpha version in January of this year. The R-GMA, however, was available and they are currently preparing to release the second version of their software. EDG is preparing to deploy R-GMA on all sites and discontinue use of MDS. Furthermore R-GMA will be rewritten over the next twelve months to be OGSA compliant, using the same web based protocols. Therefore by using R-GMA we had a working and available Grid information system, close interaction with the DataGrid monitoring task, and the added benefit that we will be OGSA compliant in the future.

4.2.2 Infrastructure: JIMS There are several infrastructure monitoring systems that provide similar functionality. Although the Jiro-based monitoring system (JIMS) design shares some concepts with them, there are many differences. The most important are pointed out in this section. We stress the following concepts:  using standardized interfaces between system entities,  reducing the volume of reported events,  using multicast communication where applicable. We also intend for this section to provide a base for future comparisons between JIMS and others by reusing the approach presented in [Bal2000]. A description of the set of parameters observed by JIMS agents in comparison with the Globus MDS 2.2 service is provided in Appendix A.

 Using standardized interfaces between system entities The article [Bal2000] points out the following aims of gathering monitoring data: (a) performance analysis, (b) performance tuning, (c) performance prediction, (d) scheduling, (e) fault detection and diagnosis. Infrastructure-related data are important for all of these goals. This is because most of the monitoring systems make use of host or network sensors that gather information about the parameters of a selected device. The Network Weather Service, NWS, is an example of such a system. The NWS’s main goal is to predict quality of service parameters related to communication between two remote processes. In order to accomplish this, the NWS measures network parameters (such as round-trip time) continuously, and produces forecasts based on these. The JIMS instrumentation layer does not measure such parameters at the moment, but because of its extensibility, such functionality could be added seamlessly (it could be implemented as another type of MBean and deployed dynamically to the hosts that the user would need to provide such reports). Moreover, the NWS takes into account only point-to-point communication and does not deal with data streams colliding on shared links. An extension of the service [TopoMon] is developed in the EU’s GridLab Project. The TopoMon system is implemented as a set of Java- based sensors, which makes it portable. However, it uses proprietary communication protocols (based on sockets) and therefore will be less extensible, than systems relying on industry standards, such as JMX.

 Reducing the volume of reported events

The concept of providing the management layer is similar to the Ka-tools filters [KaTools]. The filters can aggregate, remove, and transform data when passing it to external applications. The management layer will provide that functionality as well. The main functional difference is that the management layer entities will be able to launch pre-programmed actions, which will not be limited to filtering data or producing notifications about failures, but will include execution of a declared piece of code (even remotely). JIMS will provide a way of programming such actions.

 Using multicast communication where applicable Jiro by its nature utilizes multicast communication to keep many instances of the Lookup Service (the main functionality of which is to keep track of the system entities). The infrastructure monitoring system naturally reuses that approach. Moreover, because the system entities are required to renew their entries in the service, the Lookup Service can detect entities that have failed. That is in contrast to centralized registry services, which commonly form single points of failure. Because the system entities are not expected to observe the monitoring data directly, the monitoring agents do not communicate the values of monitored parameters via multicast, which is an approach of some monitoring systems, such as Ganglia [Ganglia].

 JIMS capabilities This subsection provides a list of the JIMS capabilities following the list of comparison criteria presented in [Bal2000]. Note that the upper layers of the system are not implemented yet. Therefore, the table refers mostly to foreseen features. Scalability number of users number of resources number of events depends on the upper yes yes (database) layer (additional event filters (volume of and other entities can be communication will be added on demand – reduced by the filters) even on the monitored host) Intrusiveness monitored systems network can sensors control their intrusiveness? network devices: low low there are two (queried via SNMP) mechanisms to control hosts: presumably low, network intrusiveness: but depending on user setting time period demands between notifications and setting threshold values Validity of information freshness (time to live accuracy (error bounds) information) no no Data format easy to use compactness standard portability

yes depending on database no yes (via web service query (XML-based) interface) Extendibility new sensors new type of data yes yes Communication comm. model internal interfaces external interfaces push JMX, Java RMI web services Security authentication authorization encrypted communication yes yes possible Measurement metrics application performance visualisation tool achievable time support resolution comprehensive no yes One second archive handling tool availability yes not yet (development)

4.2.3 Derived Results: Postprocessing Providing forecasts about future Grid status - the goal of the Postprocessing Analysis tool is similar to the goal of the Network Weather Service, [NWS]. However these two tools differ in some critical aspects. As stated on the NWS web page: "The Network Weather Service is a distributed system that periodically monitors, and dynamically forecasts, the performance various network and computational resources can deliver over a given time interval. The service operates a distributed set of performance sensors (network monitors, CPU monitors, etc.) from which it gathers readings of the instantaneous conditions. It then uses numerical models to generate forecasts of what the conditions will be for a given time frame.” [NWS] Up to now the NWS supports the following forecasting methods:  mean-based methods that use some estimate of the sample mean as a forecast,  median-based methods that use a median estimator,  autoregressive methods. As can be seen NWS's forecasters are based on deterministic methods.

When designing the Postprocessing Analysis tool the following topics were assumed to be the most important in order to make it a useful part of the Crossgrid software:  forecasts should be as reliable as possible. To achieve this goal it has been decided to use the best tools available, such as Neural Networks and Kalman Filters, to predict the state of the Grid in the future. These are, as opposed to the methods used by NWS, non-deterministic

methods, which we believe are better suited to predicting the state of the Grid, which is not expected to behave as a totally deterministic system.  The system, as opposed to the NWS, will provide information that will reflect the structure of the CrossGrid (and DataGrid). It will be achieved by making forecasts for every CrossGrid cluster entity, such as individual hosts and whole clusters. The data acquistion will also reflect the Grid testbed structure by installing main data acquisition systems on Computing elements and using Globus services to exchange the data.

It is expected that by implementing advanced non-deterministic algorithms to prepare forecasts, and using CrossGrid optimised data collection methods, the Postprocessing Analysis tool will be useful for allocation of data intensive, interactive applications on those clusters that will run them, thereby giving best performance and response time.

5 CONTRIBUTIONS TO GRID TECHNOLOGY

5.1.1 OCM-G While the current Grid technology is oriented more towards batch processing, the CrossGrid project is focused on interactive applications, where there is a person ``in the computing loop’’. First, we assem- ble a list of requirements imposed on a system for application monitoring in a Grid environment, valid for any monitoring system, but with a special focus on the support for interactive applications.

 Efficiency – this is very important for any monitoring system, but especially for interactive applications. Several issues play an important role here: (1) Intrusiveness. The overhead of the monitoring system due to the probe effect must be as low as possible. For example, we should minimize the amount of instrumentation in the application. (2) Data rate. The communication due to monitoring (mainly monitoring data collection) should induce minimum load on the network. This can be achieved by adjusting the rate at which performance data is generated using such techniques as buffering, pre-processing and selective gathering of data (only data interesting at the given moment). (3) Low latency. Performance data and especially data concerning interactive applications are relevant for only a short period of time. Therefore, it must be delivered to the consumer(s) with minimum delays.  Interoperability – the Grid area is intensively developing. Other large-scale Grid computing ef- forts are under way and new tools are going to emerge. This suggests that support for interoperability between these tools is strongly recommended. For example, it would be nice to have a possibility to apply multiple tools to one application, e.g., a performance analyzer and a load balancer. Multiple examples have shown that such cooperation could be useful. The prerequisites essential for interoperability are: a standardized monitoring protocol and a common monitoring infrastructure.  Security – there are two aspects of security in an application monitoring system. First, we must ensure privacy, and integrity of monitoring data. This implies proper authentication and authorization of users requesting operations on an application, encryption if necessary, etc. Second, the components of the monitoring system themselves must be mutually authenticated to protect against a forged-components attack.  Scalability – because potentially there may be many users running many applications, the monitoring system should be scalable in terms of users, applications, and size of a particular application, as well as being capable of handling a high data rate of performance data, which may occur even though we use data reduction.  Portability – since the Grid is heterogeneous by nature, it is important for monitoring system to be portable across many architectures. Moreover, support for common parallel programming environments like PVM or especially MPI is recommended.  Flexibility – the monitoring interface should be robust and flexible. For example, it is better to provide a low level services with which the user can define the metrics in the way he likes than impose a fixed semantics on the metrics. Additionally, the user should have a possibility to change his measurements while the application is still running. It is especially significant for long-time running applications, because resubmitting the job would be too expensive.

Our approach to application monitoring is distinguishable in several aspects. First, it provides the on- line mode of monitoring. Second, it allows for manipulations. Third, it provides a flexible set of monitoring services which allow the user to better control his monitoring session and define metrics with

a semantics he needs. Fourth, the services are defined in the form of a flexible, standard interface – OMIS. Fifth, the OCM-G is designed as a permanent monitoring infrastructure working as a Grid service. Other approaches have either different goals (GrADS), do not support manipulations (DataGrid, GridLab) or their performance is (most likely) insufficient for interactive applications due to support for tracing only or inefficient communication technology (DataGrid, GridLab).

Although monitoring of interactive applications is possible in off-line mode, this approach has many disadvantages. First, besides information about the application’s behavior, also user’s interactions must be monitored and stored in a trace file, so that a further relation between the interactions and performance is possible. This increases monitoring intrusiveness and trace sizes. It also increases the implementation effort; since the interactions must be properly saved, and subsequently visualized, in a way smart enough to properly show the relation between interactions and performance. Second, manipulations are not possible in off-line mode, and thus, for example, steering is impossible. Third, information of interest cannot be specified at run-time. Specifically, we cannot change measurements at run-time as a result of already available information. This again increases the amount of data that has to be gathered and monitoring intrusiveness. Fourth, it is more convenient for the user when he can immediately see the impact of his interactions on performance.

Capability of manipulations also gives many benefits. Some of them are as follows. First, it may be useful to stop a long-time running application to set up new measurements based on the already available results. Second, it reduces the data rate produced by monitoring since we can at runtime focus on the very data we need at the given moment. Third, manipulations are essential for steering of the application.

The OMIS/OCM-G approach is designed in such a way that it provides a large set of monitoring services, which allow the user to perform the monitoring session in a very flexible way. For example, OMIS does not impose any predefined semantics on metrics. Instead, using the large set of services, the user can construct the metrics in a flexible and easy way. The latter feature is essential for the High-Level Analysis Component (HLAC), being developed as part of the G-PM tool (CrossGrid Task 2.4), since it allows the user to define his own metrics with the semantics he needs. By contrast, existing approaches, even if they define a simple monitoring protocol (GridLab), also specify a list of available metrics, with a fixed syntax and semantics. Furthermore, the OMIS protocol provides even more flexibility, not only with regard to available metrics. For example, in the GridLab approach, buffering of data is just turned on by a proper protocol command, while in the OCM-G, the user, by means of a set of services, has a direct control over creation and use of data storage objects, e.g., counters and timers, and thus has a better control over the buffering policy. It is the user who decides whether to store the events in a trace or just increment a counter.

5.2.1 Instruments: SANTA-G SANTA-G adds new capability:  Extension to the DataGrid R-GMA. SANTA-G’s CanonicalProducer was an extension to the DataGrid R-GMA Grid information system (it is now an integral component). As stated above, it is the generic user customizable R-GMA producer. It enables the introduction of non-invasive monitoring information from external instruments into the R-GMA, specifically to support ad-hoc rather than routine monitoring. The R-GMA is based on a relational model.

There are currently a number of activities underway within the Global Grid Forum in this area. For example the Relational Grid Information Services Research Group [RGIS-RG].  Non-invasive monitoring. There are currently many Grid projects involved in the area of grid monitoring. They have produced several tools to allow for this activity. For example, GRM and Prove, NetLogger, Globus Heartbeat Monitor, Network Weather service, Autopilot and Virtue, and OCM-G, which is being developed within Task 3.3. What these monitoring tools all have in common is that they are software monitors. These types of monitors have drawbacks. Monitors embedded in the application address space will consume some of the host system resources. This can obviously affect the monitoring data obtained. For this reason software monitors are referred to as invasive monitoring systems. A monitoring system that does not affect the system under study is referred to as a non-invasive monitoring system. In general non-invasive monitoring tools are external instruments, referred to as hardware monitors; these do not perturb the system under study. Because of this, hardware monitors are often used in order to calibrate and validate software monitors. It is this class of tools that SANTA-G provides support for. Hardware monitors often store collected data in raw log files. As described in the State of the Art Section above, there are a number of these types of monitors available. None of these however are grid enabled. The SANTA-G system provides a framework for allowing access to this raw data through the Grid information system. It has been carefully designed to support very large log files that are rapidly generated in near-real- time environments. Through the use of the Canonical Producer SANTA-G also provides support for both immediate and historical querying of instruments monitoring interactive or batch application environments. The NetTracer (see CG3.3.2-TCD-D3.3-v1.1-SANTAG.doc, CG3.3.2-D3.2-v1.2-SantaGDesign.doc) is a demonstrator of this concept, providing access to raw log files obtained by the network-monitoring program TCPdump.

5.2.2 Infrastructure: JIMS What is innovative in the Jiro Based Grid Infrastructure Monitoring System is that by reusing the JMX – which is an industry standard – for monitoring the grid infrastructure, it forms an open environment. As an example, consider the plans for integrating the system with SANTA-G, which – being R-GMA- based – relies on a completely different technology but nonetheless will be able to input the Jiro-based monitoring data. Moreover, many software programs expose JMX-based interfaces. This trend will eventually result in development of tools that would use the two types of information sources in a uniform, JMX-based, way. The values reported by the system could be observed in many ways, as the JMX environment could be extended by so called protocol connectors. The dynamic character of the system, currently manifested in dynamic agent deployment, instantiation and revocation, although resulting from application of modern technologies (such as Jini and Jiro), seems to be new in the grid software. Although in the first prototype the deployment of monitoring agents is performed from an administrator’s console, it is quite easy to equip, for example, intelligent failure tracking tools with such functionality. The system architecture allows for using legacy standards, such as SNMP, for monitoring devices, which are not capable of running other monitoring agents, thus enabling grid applications to use data regarding the state of the underlying network, i.e. traffic routes, link loads etc. That information could be useful for schedulers, especially if they are equipped with some automatic management mechanisms. As an example, consider a management application reconfiguring routers based on the schedule in order not to cause link overloading in the future. Such functionality could be implemented in the form of Jiro services and bound to the monitoring system at the management level. Finally, the data gathered by the JMX-based agents can be relatively easily exposed in the form of web services, which makes the integration of the system with OGSA easier.

5.2.3 Derived Results: Postprocessing We plan to deliver a consistent monitoring environment for providing aggregated data from various sources about the Grid and based on that we will make more accurate predictions about its future status than currently available tools such as Network Weather Service. The main novelty in our approach is the implementation of Neural Networks and recursive Kalman filters instead of typical time-series analysis, which is most commonly used in similar software. We believe that this method could achieve much better results with that kind of data. For this reason intensive tests are now being carried out on an ICM cluster using the HEP application prototype (Task 1.3) and our own set of local monitoring tools. Development of an accurate forecasting engine is very important, as the postprocessing tool will be one of the main information providers for the Performance Prediction software that is being developed in the scope of Task 2.4.2. The necessity for providing accurate, short term predictions about future state of the Grid is especially important for running interactive applications, which require fast and reliable resource allocation. We believe that an iterative Kalman filter or Neural Network will perform much better in this case than the algorithms used in similar projects. Also, as it recursively updates it’s internal state, each iteration should be relatively fast. The modular construction of our forecasters will also make it easier to reuse them in other applications, not necessarily connected with CrossGrid.

6 BRIEF DESCRIPTION OF THE SOFTWARE

6.1 APPLICATION MONITORING

6.1.1 OCM-G Task 3.3.1 is focused on development of an infrastructure for monitoring application running on the Grid. This infrastructure is responsible for gathering performance data from an application and providing it to performance analysis tools, which can then be used by application developers in order to understand an application’s behavior and improve its performance. The OCM-G is composed of a collection of monitors that observe and manipulate application processes. Information gathered from each application process is transferred to higher-level managers, which expose the information to tools. A base for the OCM-G is a monitoring system for parallel applications running on clusters – the OCM. This system was extended to make it Grid-enabled, which includes, for example, a new start-up, new monitoring services, etc. The tool-monitor interface is OMIS (On-line Monitoring Interface Specification), which is a specification of a generic, flexible on-line monitoring interface. For a detailed description of the architecture of the OCM-G, refer to [OCMGD3.3].

6.2.1 Instruments: SANTA-G SANTA-G services are a specialized non-invasive complement to other more intrusive monitoring services. One application of these services would be in validation and calibration of both intrusive monitoring systems and systemic models, and also for performance analysis. The objectives are to: (a) allow information captured by external monitoring instruments to be introduced into the Grid information system, (b) support analysis of performance using this information.

Most of the aspects of this overall functionality have been implemented in the first prototype, at least in a limited way. The prototype of the SANTA-G system is made up of two main modules, a publishing module and a viewer module (see Figure 6.2.1.1). The publishing module has a further three components, the CanonicalProducer and Servlet, the Query Engine, and the Sensor. The following describes the basic functionality of the prototype.

Grid Information Publishing Module Circular System (R-GMA) Viewer Module Queue Of Trace Files

Figure 6.2.1.1: The SANTA-G System Modules

In order to obtain data from the Grid Information System a user may use the SANTA-G Viewer. In the spirit of R-GMA, a SQL query statement is entered into the GUI in order to define the data that is to be acquired. The Viewer, by using its own Consumer interface, will contact a Consumer Servlet, which in turn contacts a R-GMA registry in order to locate the required producers of the information. Once found the query is forwarded to these Producers. The CanonicalProducer servlet will receive the query and in turn forward it to the CanonicalProducer that holds the information needed. The CanonicalProducer now uses the QueryEngine to collect the data from the raw TCPDump log file by performing seek operations. Once the required data is found it is returned to the QueryEngine. The information is then returned through the same mechanisms to the Viewer in the form of a ResultSet. The individual fields of the result set are then extracted and displayed by the Viewer as a table. The user may also use the graphical display panel, which provides a number of controls for navigating a trace. Selected packets can then be displayed graphically in the Viewer GUI.

The Sensor maintains two tables of information relating to the trace available. The first is the Trace Information table. It stores a trace ID, and the path to the directory used to store the trace log files. The second table, the File Information table contains information relating to the files used to store the raw trace data within the trace directory. This table stores again, the path of the directory used to store the trace circular queue, a file ID, which identifies an individual file within the trace, and the filename. These two tables are stored, and subsequently published, by using a DBProducer.

The work carried out on the first prototype is in-line with our aims. The intended functionality of the first prototype was achieved and functioned as expected.

As stated in previous sections much was gained during the first year from our close collaboration with the EDG WP3. SANTA-G makes use of the DataGrid R-GMA, but also contributed to it. The Canonical Producer implemented as part of SANTA-G now forms an integral part of the R-GMA.

6.2.2 Infrastructure: JIMS The purpose of the JIMS system is to: (a) gather and expose information concerning the state of devices used to build a grid environment. (b) notify the user not only about simple events, but derived ones as well, and (c) take managerial actions in cases of failures.

The idea is to reuse a standardised approach to instrument the monitored devices in order to expose their parameters to a database system in a unified way. The first prototype provides the means of instrumentation for computational and storage nodes and SNMP-capable devices.

 Gathering data from hosts The JMX agents used for monitoring hosts use interfaces exposed by system kernels. The agent consists of two parts: native – for interaction with operating system, and Java-based (gathering data from the native part using JNI). From the programmer’s point of view, the agents present the JMX MBean Server interface. Since that, introspection on the agent capabilities is quite straightforward, thus making JIMS usable even in environments that make different sets of parameters visible to the user.

 Gathering data from networking devices The first prototype of JIMS makes use of Sun’s Java Dynamic Management Kit mainly in order to gather data from SNMP-capable networking devices. This way, the parameters exposed by routers, switches, etc. can also be observed. Although future releases of JIMS will be based on a publicly available implementation of JMX that does not provide this functionality, there is an effort to provide similar agents.

6.2.3 Derived Results: Postprocessing The information collected from instruments and infrastructure represents a valuable asset for both immediate use and historical analysis. Postprocessing is focused on two topics: (a) Gathering information from other monitoring tools and creation of one consistent user interface. (b) Generation of forecasts of future grid state using Kalman Filters and neural networks.

7 AIMS, TESTS AND EVALUATION, NEW REQUIREMENTS

7.1.1 OCM-G The implementation of the first OCM-G prototype is described as follows.  The core structure of the monitor was successfully ported from the existing monitoring system for clusters, the OCM.  A new, Grid-enabled start-up of the monitoring infrastructure has been implemented.  The concept of application objects was introduced.  The concept of probes – support for user-defined events was implemented.  New Grid-specific services have been added – to handle the application objects and to handle probes.  Integration with the G-PM tool (CrossGrid Task 2.4) was successfully achieved. The main limitations of the first prototype in comparison to the final version’s planned functionality are as follows: it handles only one site, the secure communication and authentication is not yet incorporated, several services are yet to be implemented. The tests necessary to verify the correct functioning of the OCM-G were as follows.  correctness of the new start-up procedure  correctness of the functionality of new monitoring services  verification of data returned by the monitoring system. Since the core part of the system was ported from the OCM, which has been an existing monitoring system for clusters for several years now, some parts of the OCM-G were already tested (such as algorithms for request distribution and assembly, infrastructure for event services, etc.). Apart from individually testing new modules, the most important test was the general functionality of the system, especially the correctness of data returned. This was checked in integration with the G-PM tool using an application with a well-defined behaviour.

The only direct external connection of Task 3.3.1 is to Task 2.4.1 (performance measurement), thus all requirements come from this Task, though indirectly they are from other tasks (including WP1 applications). In the course of the recent development, the following issues emerged in addition to the original requirements.  Though the start-up of the OCM-G already requires little effort from the user, it still could be made easier. At present, some environment variables must be set to run the OCM-G properly. However, part of this will be eliminated when the OCM-G is permanently installed on the testbed and the proper executables/libraries are in the system PATH.  Clean-up should be improved. At the moment, some of the OCM-G and application processes hang occasionally.

 A call to the ocmg_register function must be placed after MPI_Init, otherwise only the master process will receive proper command line arguments. It would be convenient to make the ocmg_register call from an already instrumented MPI library. This will be done soon.

7.2.1 Instruments: SANTA-G The SANTA-G system provides a framework for allowing access to raw trace data through the Grid information system. The first prototype illustrates these concepts with a NetTracer, a demonstrator that allows network trace information obtained by the program TCPdump to be accessed through the Grid information system. In reality the underlying concepts and resulting software framework are designed to have wider applicability; they allow information from a great variety of hardware or software instruments to be accessed through the Grid information system (for example, with very little additional work the demonstrator itself could act as a grid-enabled front-end to the SNORT security intrusion monitor). Testing was of a purely functional nature. Following the integration meeting in Santiago, a number of new requirements were received. These were mostly in the form of requests for increased functionality. The following table summarises these and shows the requirements status, either done, indicating that it has been completed, or 2nd prototype, indicating that it will be completed in time for the second prototype.

Requirement Status Directory and name of trace files should not be hardwired Done Use environment variables for port number, registry and Done schema machine etc. Make one daemon which sources rgma-setup.sh, starts the 2nd prototype Query Engine and the Sensor. Automatic detection by the QueryEngine when a new file 2nd prototype is added to trace directory. Wrapper around TCPdump command to produce log files 2nd prototype in the correct directory and with correct filename. Viewer should have more SQL support. 2nd prototype Viewer should have an ‘easy way’ to construct complex Done queries. Viewer should have automatic translation of IP’s to DNS. Done Possibility to start and end a trace on a given element. 2nd prototype Ability to view SNORT log files. 2nd prototype Clean-up of old trace files. 2nd prototype

7.2.2 Infrastructure: JIMS The first prototype consists of the first implementation of the instrumentation and agent layers of the system’s architecture (see [Jiro D3.2] for detailed description of the architecture) as well as of some preliminary user interfaces. Although currently the system is able to present only a single node state, which is one of five use cases declared in [WP3.3 SRS], many important features, like dynamic deployment of monitoring agents are already implemented.

7.2.3 Derived Results: Postprocessing Postprocessing application prototype is a demonstration of the basic concepts of the systems internal structure. Most the prototypes building blocks will be replaced by the final versions in the future – the current release was aimed at identifying bottlenecks of the system and making the necessary adjustments in the application schema. The main objective of the first prototype test was to run the application in the working environment of the local cluster, write the monitoring data (using a very simple tool called SYSSTAT) to the database and run the Neural Network forecaster module on the data. We also designed a viewer module to visualize the results of the forecast.

8 RESULTS OF THE TESTS AND EVALUATION

8.1.1 OCM-G The results of the tests were successful. The new modules of the OCM-G worked correctly. The general functionality of the OCM-G was tested with a simple ping-pong application, which had a well- defined behaviour that could be described in concrete values (volumes of data sent, delays due to sleeping, etc.). The OCM-G was used to monitor volumes of data sent and delays between sends and receives. The result shown on the G-PM displays proved the functionality was correct and verified the correctness of data returned by the OCM-G. The values read from the G-PM displays showed only a statistic deviation from the expected results.

8.2.1 Instruments: SANTA-G All planned functionality worked, and its operation between Spain and Ireland was demonstrated at a workshop in Santiago in February 2003, and during the first EU review in March 2003. The areas that require improvement are in general related to increasing functionality, particularly of the Sensor. Also some work is required in order to make the Viewer more user friendly. This is discussed further in the following section Problems and Issues, Section 9.2.1.

8.2.2 Infrastructure: JIMS Tests were successfully conducted in a local fabric only.

8.2.3 Derived Results: Postprocessing Tests were completed successfully and produced valuable conclusions about the necessary adjustments to the application model. The main change will concern the forecasters’ architecture, and is described below in more detail.

9 PROBLEMS AND ISSUES Please note that problems in installing the software are outlined in the installation guide [WP3 Inst]

9.1.1 OCM-G We have encountered the following problems and issues during the development of the first prototype of the OCM-G.  In [OCMGD3.3], we planned to use the MDS to store the information if a Service Manager is running or not. Due to problems with updating MDS frequently, we decided to store in MDS only information about the SM’s location (which is more static). Testing if the SM is running or not will be done simply by trying to connect to it.  In the design document we did not explain the case where a single site is shared by two or more Virtual Organisations. While sharing the monitoring data by many VO may not be acceptable, it should be possible to run many SM instances on a single host (each for one VO). In consequence, we might not be able to apply the `well know port’ technique to locate SMs, but we can still assign a default port during the installation of the package. This issue is still to be discussed and recognized in detail.  It was planned to partially implement the software in C++, but finally we decided to leave the whole project in plain C. The reasons for this are the following: (a) We reused most of the existing OCM code, which was plain C. Due to a high complexity of existing code, migration of this to C++ turned out to be unrealistic in the framework of the project. (b) There are parts of the system, that must be (or are better) written in pure C, e.g., the parts of the monitoring system that are linked to the application (which is usually written in C). (c) The benefits of implementing new modules in C++ are outweighed by the effort required to integrate the old C parts with the new C++ parts.

9.2.1 Instruments: SANTA-G SANTA-G is made up of two main modules, a publishing module and a viewer module. The publishing module has a further three components, the CanonicalProducer and Servlet, the Query Engine, and the Sensor. Some of the functionality of the components has yet to be implemented. The Sensor application is one of these. The purposes of the Sensor is to allow a user to start and stop TCPDump, and also maintain tables of information relating to the resulting log files by using a R- GMA DataBaseProducer (a standard R-GMA component). For the first prototype the Sensor provides only limited tables of information using a DataBaseProducer, and it does not allow a user to control TCPdump. The Sensor therefore only provides information on static pre-existing log files. Because of this the prototype can only use log files generated off-line by TCPdump. Also the prototype can only handle a subset of the potential schema for the trace files, and can only handle a more limited subset of SQL than will eventually be supported.

9.2.2 Infrastructure: JIMS The main problem with the developed software is that it was implemented with Sun’s Java Dynamic Management Kit™, which is commercial software. Therefore, the tests were conducted only on the local Cyfronet cluster. In the future (second prototype) the underlying implementation of Java Management Extensions will be changed to the Reference Implementation, which is available publicly. Since the functionality of the free implementation is significantly limited, that change will introduce some new problems, such as the need to implement from scratch the MBeans meant for monitoring networking devices through SNMP. A number of problems arise from incompatibilities between Jiro 1.5.1 (currently the latest) release and some implementations of the Java Virtual Machine. Although the system works properly with the Java Development Kit 1.3 for Linux, the incompatibility introduces some problems when running the system with the 1.4 release.

9.2.3 Derived Results: Postprocessing There were several major problems and limitations affecting our first approach to the creation of the post-processing application. The first problem was the very preliminary system of data acquisition, which worked only on local cluster. This will be addressed in the future in several ways, first by using a simple GSI-ftp solution, and in the next version by providing a fully OGSA compliant input interface which will read data using R-GMA. The next problem was the pattern generation for training the forecasters. At the time of prototype creation there were no working Crossgrid application prototypes, so we used a program called QSbench instead. Therefore the load characteristics may be significantly different than in Grid applications. This may, though unlikely, affect the forecasters architecture. The third problem directly concerned the architecture of the first Neural Network forecaster. We assumed that it will work “on demand”, i.e. full learning cycle is carried out upon receiving a request. Unfortunately, this approach was quite CPU consuming, so will be rather ineffective for serving larger structures (like the whole of Crossgrid). Therefore we decided to change the forecasting paradigm, and use recursive-learning algorithms. This was a very valuable conclusion from the first prototype tests, and we hope that we will achieve a much more scalable system using this approach.

10 FUTURE PLANS

10.1.1 OCM-G Our plans for the next year are as follows.  Add user contexts. The current version of the OCM-G handles only one user. In the next version, we plan to introduce user contexts so that all monitoring actions are always performed on behalf of a particular user. The description of a user is also required to introduce security mechanisms into the system.  Enable security in OCM-G. This is the main issue for the second prototype. The security is important at two levels: (a) GSI security for component-component connections. We plan to replace the raw socket communication with Globus I/O. (b) Secure protocol to join a Virtual Monitoring System – for monitoring the chosen application.  Integration with external Grid service. Location of Service Managers should be available in an information service so that other entities can discover it. Thus, we should put it in a Grid information service. The current OCM-G version temporarily assumes a shared file system and stores this information in a file located in the user’s home directory.

10.2.1 Instruments: SANTA-G In the next year the following will be implemented:  The schema and SQL-support of the Query Engine to be extended. Currently the Query Engine only provides a limited schema of the possible information that can be obtained from the TCPdump log files. It also only supports simple SQL select statements.  On-line data acquisition. It is intended that the Sensor should allow a user to control TCPdump and therefore provide access to dynamically obtained network trace data.  Integrate Sensor and Query Engine. Currently the Sensor and Query Engine operate independently. Although the Sensor provides limited tables of information relating to the available log files the Query Engine does not make use of these. In order to implement dynamic data acquisition there will need to be co-operation between the Sensor and Query Engine modules.  Viewer to be extended. Additional functionality will be added to the Viewer, for example SQL query construction support. It will also be changed to reflect the additional schema extensions in the Query Engine, in order to allow the Viewer to display a larger subset of packets. It is intended that this work will be carried out opportunistically.  Integration with other Task 3.3 components. It is intended within the next year to integrate the ‘non-applications-monitoring’ services by providing a common interface. It is intended

that this interface will be R-GMA based. The network trace information obtained from SANTA-G and published through the R-GMA, will be combined with the Grid infrastructure monitoring information obtained by the JIMS, similarly published to the R-GMA. This information will then be used by the Post-Processing component, which will again publish its results to the R-GMA.  Prepare for OGSA. See section 11.2.1.

10.2.2 Infrastructure: JIMS  Implementation of the management layer. The management layer, able to interpret events is going to be implemented in the second prototype of the software (by M18). The entities of that layer, which are going to be programmable in a declarative way, will serve the user in two ways by: (a) reducing the volume of events, (b) generating derived events. The underlying technology will be an implementation of the Java Rule Engine, which – for now – is at the Java Specification Request stage. It will enable the system user (presumably, an administrator) to program actions initiated by the system in response to detected states. Although the Rule Engine would allow any actions, it is expected mainly to report compound events based on the elementary ones reported by the instrumentation and agent layers.

11 Integration with SANTA-G. The second prototype will contain an additional agent layer component, which will gather all the basic events from a cluster and deliver them to the SANTA- G team. This prototype will expose a web services based interface in order to integrate with external systems easily. Although the component will gather the information just as the management layer entities do, it is considered an agent layer component, because it will perform no interpretation of the received data, but instead will serve as a gateway for the low-level, detailed communications. The SANTA-G team will wrap the component in a way that is appropriate for that system (see Figure 10.2.2.1). Another form of integration with SANTA-G is gathering the data about network communication and feeding JIMS with it. The SANTA-G team will provide a component exposing a JMX-based interface. In this way, the data taken from SANTA-G can be submitted to the management layer of the infrastructure monitoring system. Explicit mechanisms will be employed to suppress cyclic effects.

Agent layer web service Collector interface To SANTA-G

MBeanServer MBeanServer

Instrumentation layer MBean MBean MBean MBean

Management layer

Agent layer MBeanServer SANTA-G agent

Instrumentation layer MBean MBean

SANTA-G info Figure 10.2.2.1: Integration with the SANTA-G monitoring system. The results of the integration will be at least as follows: (a) ability for SANTA-G users to query for hardware devices’ state using SQL, or SQL-like queries, (b) ability for JIMS users to derive compound events from more elementary ones, that could result, for example, in easier detection of failures, or even in automatic recovery from foreseen ones.

 Porting the software to public domain libraries. Currently, the instrumentation and agent layers are implemented with Sun’s Java Dynamic Management Kit 4.2 software, which is not freely available. The second prototype will not rely on that library, using the JMX Reference Implementation instead.  Development of more sophisticated monitoring agents. The instrumentation and agent layers will undergo many implementation changes. The Linux monitoring agent will expose more monitored parameters and the system will use different sources for some information (e.g. network interfaces related). Moreover, advanced monitoring agents (using e.g. iptables/ipchains filtering functionality) are also under consideration.  Further plans. The next stage of development (after M18’s second prototype) will be concerned mainly with the implementation of the database and presentation layers. Moreover, there will be significant effort to expose system’s services in an OGSA-compliant way. It is expected that the user of the system would have three ways of using it: (a) by extending its functionality (programming actions) this use case will be ready for M18. (b) by observing data reported by the management layer using a web services’ based interface (also a part of the M18’s second prototype). SANTA-G will make use of this feature. (c) by querying the database layer for more aggregated information.

11.1.1 Derived Results: Postprocessing The first prototype presented in CG3.3.4-v0.5-ICM-PrototypeDoc was an initial approach to define the application's internal structure. It was limited in functionality, especially in the input/output interfaces. The reason for this was the absence of prototype implementations of the other monitoring tools. In the current phase of development it has been decided to focus on the internal postprocessing engine and additional data sources. The main goals of the application development in the next year can be described as follows:  Kalman filters and neural network forecasters. Presently a completely new set of forecasting modules is being developed. Using the first Crossgrid application prototypes, the developers are trying to identify the nature of typical grid usage patterns and design the most accurate algorithm for load forecasting. The idea is to use non-deterministic methodologies such as Kalman filters and Neural Networks for construction of these modules.  Integration with other monitoring tasks when they provide the final versions of their APIs.  Building an alternative solution for data input. We plan to develop a simple, but effective system for collecting data about activity on Crossgrid clusters and their components such as: CPU load, memory usage, network traffic, etc. The tool will be based on the standard Linux system tool SAR (System Activity Reporter). Data will be transferred from clusters to the Postprocessing Server using GSI-ftp. In the future this approach will be enhanced by using the extended grid monitoring tools (SANTA-G and JIMS).  Providing data for Task 2.4.2. The software for performance analysis and prediction is vitally interested in aggregated information about current and future Grid status. Therefore we will work on interfacing our tool with the Task 2.4.2 software, which will be done using R- GMA/Spitfire.

12 TOWARDS OGSA

12.1.1 OCM-G A recent trend in Grid technology is to implement services as web services, one of the frameworks supporting this approach being OGSA [OGSA]. However, in the current state of development, OGSA does not fit/support our approach. First, OGSA does not yet support C/C++. Since our system is written in C (which is important for efficiency reasons), we cannot implement OGSA servers exposing the OCM-G services. Second, though OGSA already provides an infrastructure for secure communication in the Grid, the communication layer of the OCM-G, based on an efficient direct TCP/IP solution, is already implemented, and the secure communication mechanisms are currently in the final stage of development. Another reason to develop our own communication layer is that the efficiency provided by OGSA is unclear. Nevertheless, our current research includes a feasibility study and the design of an interface to OMIS monitoring services based on OGSA. The preliminary results of this research are expected soon.

12.2.1 Instruments: SANTA-G The DataGrid R-GMA is aware of the expected change to OGSA, and is currently investigating a move to this architecture. Because R-GMA is conceptually already quite close to web-services this move should be straightforward. As OGSA progresses the R-GMA will naturally evolve to this structure. A consequence of this is that by SANTA-G making use of the R-GMA we are ensuring that we will be fully OGSA compliant when the changeover to this architecture occurs. There have been a number of presentations given on R-GMA plans regarding OGSA, for example, see [RGMA-OGSA].

12.2.2 Infrastructure: JIMS The Jiro-based infrastructure monitoring system will expose interfaces to external systems on two layers: agent layer (described in the “Future Plans”) section and database layer. The interface on the agent layer will be implemented as a web service, as this will mean integration with other OGSA- based components should be straightforward. The collector will form a producer in the GMA terminology and will communicate with its sensors via Java RMI. However, as no direct communication will take place between external systems and the sensors that would not impose any problems with integration. The database layer will also expose web services wrapping Enterprise Java Beans that will access the database directly. Similarly, integration with other web services is not expected to introduce any problems.

12.2.3 Derived Results: Postprocessing All interfaces to external modules, with respect to the Postprocessing Analysis tool, will be implemented using the R-GMA. As the result of the fact that the R-GMA will evolve to an OGSA compatible architecture in the near future, OGSIfication of the Postprocessing Analysis module will be quite seamless.

13 RISK ASSESSMENT Please note that the risk analyses presented in this section are based on the risk model described in http://www.newproductdynamics.com/Risk/C2M9-02.pdf.

We use the following 5-degree scale for specifying the probability of risk events: 1) very unlikely (10%) 2) unlikely (30%) 3) possible (50%) 4) likely (70%) 5) very likely (90%)

The risk events, their impact and related total and estimated loss are represented in the following tables:

A. Risk event Description of the event B. Probability Probability of the occurrence of the event C. Impact Description of impact due to the occurrence of the risk event D. Prob. of impact Probability of impact E. Total loss Maximum possible loss due to the risk event in time or money F. Estimated loss Estimated loss, which is B * D * E. G. Mitigation plan Actions planned in order to avoid the risk event H. Responsible person Person responsible for the mitigation strategy on the appropriate level (WP, Task)

13.1.1 OCM-G Below we present detailed risk analysis and loss estimation for several risk events we have identified to be possible.

 Inter-site communication

The current concept of inter-site communication in the OCM-G relies on the assumption that the system administrators will agree to open one or a pool of a few ports on firewalls for Service Managers. The currently being developed communication layer is suited for this case. However, if this turns out to be un-realistic, an additional design and implementation effort will be necessary to develop a new mechanism for communication across firewalls. The effort could then be very high (approximately two months of time), and it is not clear what could be an alternative solution. For example, for ssh tunneling the user would have to be able to log into remote hosts via ssh, which is not permitted. On the other hand, the probability of such an event is very low (if not 0).

This is because if there will be MPI applications that work across sites, the system administrators will have to open additional ports anyway for MPICH-G2. We can then just use one of these ports. We also hope that design of a strong security mechanism can convince the system administrators that opening additional port(s) for SMs is safe.

Risk event The system administrators will not agree to open a port for SMs Probability very unlikely (10%) Impact Delay due to the necessity of designing a new way to communicate across firewalls Prob. of impact 100% Total loss Two months due to the implementation Estimated loss Six days Mitigation plan Design of strategies for security and reduction of the risk for the system due to an additional open port so that sys-administrators are likely to be convinced about safety of the solution. Responsible person Bartosz Balis

 Using the OCM-G as a permanent Grid service

The OCM-G is designed to work as a permanent Grid service which in practice means that the Service Managers run permanently as additional system daemons with unprivileged user’s rights (nobody). There is a possibility that the system administrators will disallow to run the SMs permanently. However, there is a quick solution for that though a bit inconvenient for the user. The Service Managers can just be run by the user by hand on all sites he plans to run the application on.

Risk event The system administrators will not agree to run SMs permanently Probability unlikely (30%) Impact Necessity to run the SMs by hand Prob. of impact 100% Total loss 0, just some inconvenience for the users Estimated loss 0 Mitigation plan Design of strategies for security and reduction of the risk for the system due to an additional system service so that sys-administrators are likely to be convinced about safety of the solution. Responsible person Bartosz Balis

 Linux kernel patches

In the original design of the OCM-G, two kernel patches were assumed: one to enable the FIFO queues to issue SIGIO when written, the other one to enable using the processor’s hardware counters. It is very likely, though, that the administrators will not permit installation of the patches

on the Linux kernel. It does not matter for the FIFO/SIGIO anymore, since we have already provided for that case and implemented a workaround solution that eliminates the need for patches. However, in the hardware counters case the impact will be simply that they cannot be used, which means that the user cannot take advantage of additional information coming from the hardware counters data.

Risk event The system administrators will not agree to apply Linux kernel patches Probability very likely (90%) Impact Cannot use hardware counters Prob. of impact 100% Total loss User cannot take advantage of additional information Estimated loss User cannot take advantage of additional information Mitigation plan Inform the Linux kernel development community of the need for the FIFO/SIGIO functionality in the mainstream kernel release. Recognize the possibility of movement of CrossGrid testbed to a new kernel version. Investigate the conditions under which the installation of patches or movement to a new kernel version would be possible. Responsible person Bartosz Balis

 Non-availability of MDS

The OCM-G needs an information service mainly to publish the location of Service Managers so that other entities can find them. The current implementation plans are to use MDS for this. However, it is quite likely that MDS will be discontinued in the future and the Grid community will move to another information service (such as R-GMA). However, the impact of such an event should not be very high if there is a well separated module for using an external information service and this is already designed in such a way in the OCM-G. Assuming this, the impact will just be a few days for re-implementation of a few functions.

Risk event Information service will change Probability very likely (90%) Impact Delay due to the movement to a new information service Prob. of impact 100% Total loss One week Estimated loss Six days Mitigation plan Investigation and design of possible alternative solutions. Responsible person Bartosz Baliś

 Losing a team member

Though we consider the probability of this risk event to be very low, the resulting impact could be very high due to the lack of manpower. On the other hand, the exact impact is really hard to

estimate, since it depends on many factors (e.g., at which point of the project a team member is lost). Below we give an estimation of a maximum impact, however, it is still a rough estimation.

Risk event Loss of a team member Probability very unlikely (10%) Impact Delay due to the lack of manpower Prob. of impact 100% Total loss 4 months due to the implementation Estimated loss Two weeks Mitigation plan Use of external manpower (e.g., students’ projects) for initial design, investigation of technologies, and possibly part of implementation of the future tasks (e.g., well-defined modules). Responsible person Bartosz Balis

 No stable version of OGSA is available

The OCM-G does not rely on the OGSA technology, so we consider this event to be not applicable.

Risk event No stable version of OGSA is available Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 DataGrid does not provide declared functionality required by specific CG task

The OCM-G does not rely on the external technology provided by DataGrid, so we consider this event to be not applicable.

Risk event DataGrid does not provide declared functionality required by specific CG task Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A

Mitigation plan N/A Responsible person N/A

 GT2.x is not supported

The GT2.x functionality currently available is sufficient for the needs of the OCM-G, and thus the risk probability is 0, so the event can be considered as not applicable.

Risk event GT2.x is not supported Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 CrossGrid moves to GT3

The event of a migration of CrossGrid to Globus Toolkit 3 may imply several changes in the OCM-G, the most important being the necessity to move to Web Services (OGSA). This event is connected with several issues and some other risk events play an important role here whose occurrence may greatly increase the total loss. Here we will estimate the loss in the worst case in which all the worst risk events occur in conjunction. They are as following. 1) CrossGrid decides to move to Globus Toolkit 3 (OGSA) 2) There is still no support for OGSA servers for C/C++ 3) We will have to provide an OGSA interface for the OCM-G 4) We will need to use SOAP/WSDL as the communication layer

All of the above events are possible and imply some impact. OGSA currently supports Java only and if this will continue, we will have to use JNI encapsulation or at least partially move to Java. Furthermore, GT3 seems not to support globus_io, so it may be necessary to move to SOAP/WSDL as the communication layer. Overall, the impact will be very high, but we estimate the probability of all the events in conjunction to be very low. One additional factor which can in fact further lower the estimated loss, but is hard to measure, is that the necessary condition for all the risk events to be valid is that CrossGrid must decide to move to OGSA, which is the event we can to some extent influence.

Risk event CrossGrid moves to GT3 Probability unlikely (30%) Impact Delay due to the migration to OGSA Prob. of impact 100%

Total loss 4 months due to the implementation Estimated loss Five weeks Mitigation plan Discussion with CrossGrid community about the possibility of the migration to GT3, and implications. Investigation of the plans of OGSA community wrt the support for C++. Investigation and design of alternative solutions even with GT3. Responsible person Bartosz Balis

 CrossGrid does not move to GT3

This event is not applicable, since the OCM-G does not rely on GT3 technology.

Risk event CrossGrid does not move to GT3 Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

13.2.1 Instruments: SANTA-G Below we present detailed risk analysis and loss estimation for several risk events we have identified to be possible.

 CrossGrid moves to GT3

As described above (see Section 11.2.1), by using R-GMA we are ensuring that we will be fully OGSA compliant in the future. This is because the R-GMA will evolve to an OGSA compliant structure. Therefore this risk will have no impact.

Risk event CrossGrid moves to GT3 Probability unlikely (30%) Impact None Prob. of impact 100% Total loss 0 Estimated loss 0

Mitigation plan Discussion with CrossGrid community about the possibility of the migration to GT3, and implications. Responsible person Stuart Kenny

 CrossGrid moves to OGSA before R-GMA

It is not clear what is the probability that CrossGrid will move to GT3, but let’s assume that it will. As described above by using R-GMA we are ensuring that we will be fully OGSA compliant in the future. The problem with this is that we are tied to the DataGrid R-GMA schedule and therefore will not be fully OGSA compliant until the R-GMA changeover is complete. So therefore the total loss would be the time between CrossGrid requiring all services to be OGSA compliant and the time that the R-GMA becomes OGSA compliant. There is obviously a further risk involved in this, which is explained below.

Risk event CG moves to OGSA before R-GMA moves to OGSA Probability possible (50%) Impact Delay due to the R-GMA moving to OGSA Prob. of impact 100% Total loss 0 Estimated loss 0 Mitigation plan As the risk is based on external factors, over which we have no control, there is no plan to avoid the risk. The delay would be unavoidable. Responsible Person Stuart Kenny

 R-GMA fails to move to OGSA

It is a possible risk that the R-GMA will not move to an OGSA compliant structure. This is highly unlikely as work is already under way in this area. The impact of this would again not be overly serious. As the CanonicalProducer is already web and java based it could be refactored into a web service to meet with OGSA requirements. The only delay would be in this work.

Risk event R-GMA fails to move to OGSA Probability very unlikely (10%) Impact Delay due to need to refactor CanonicalProducer to web service Prob. of impact 100% Total loss 1 month Estimated loss 3 days Mitigation plan Investigation into methods of refactoring CanonicalProducer to OGSA compliant web service. Responsible Person Stuart Kenny

This event is highly unlikely. The impact would be quite high in terms of increased workload for the remaining members.

Risk event Losing a team member Probability very unlikely (10%) Impact Delay to re-allocating work, and due to familiarising other members with task of member. Prob. of impact 100% Total loss 2 weeks Estimated loss 2 days Mitigation plan Ensuring all members are familiar with current status and development Responsible person Stuart Kenny

 DataGrid does not provide declared functionality required by the task

This risk does not apply. We have already provided the required functionality to the DataGrid R- GMA in the form of the CanonicalProducer. This producer is now an integral part of the DataGrid R-GMA.

Risk event DataGrid does not provide declared functionality required by the task. Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 No stable version of OGSA is available

SANTA-G does not rely on the OGSA technology, so we consider this event to be not applicable.

Risk event No stable version of OGSA is available Probability N/A Impact N/A Prob. of impact N/A Total loss N/A

Estimated loss N/A Mitigation plan N/A Responsible person N/A

 GT2.x is not supported

This event is not applicable, since SANTA-G does not rely on GT2 technology.

Risk event GT2.x is not supported Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 CrossGrid does not move to GT3

This event is not applicable, since SANTA-G does not rely on GT3 technology.

Risk event CrossGrid does not move to GT3 Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

13.2.2 Infrastructure: JIMS Below we present detailed risk analysis and loss estimation for several risk events we have identified to be possible.

The design of the JIMS is component-oriented, so even the loss of a key team member will not result in failure of all development. The modules that are in the early stages of development would be affected more severely by such an event.

Risk event Loss of a key team member Probability very unlikely (10%) Impact Delay in development of a module Prob. of impact 100% Total loss About three months (depending on the affected module) Estimated loss Six days Mitigation plan Introducing another person Responsible person Sławomir Zieliński

 No stable version of OGSA available

Since this subtask does not depend on OGSA, this event is not applicable to JIMS.

Risk event Unavailability of any stable version of OGSA Probability possible (50%) Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 DataGrid does not provide declared functionality required by specific CrossGrid task

Since this subtask does not need any input from DataGrid, this event is not applicable to JIMS

Risk event Unavailability of functionality declared by DataGrid Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 Globus Toolkit 2.x is not supported

Since GT 2.x is not used in the development, this event is not applicable to JIMS.

Risk event No support for GT 2.x Probability N/A Impact N/A Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person N/A

 CrossGrid does not move to Globus Toolkit 3.0

From the functional point of view that event is not really a threat. However, it would make presenting a common interface with other monitoring systems (esp. SANTA-G) harder.

Risk event CrossGrid does not move to GT 3 Probability 70% Impact In the worst case, the currently developed bridge to SANTA-G could not be used; more likely – a need to reimplement some of its code Prob. of impact 30% Total loss Two months Estimated loss One week Mitigation plan Even if CG does not move to GT 3, integration inside T3.3 with SANTA-G using web services should be possible. If not, there will be a need to reimplement the interface part of the bridge. The data collection part would remain the same. Responsible person Sławomir Zieliński

 Inter-site communication

The inter-site communication will occur at the database level and could require a few ports to be opened for execution of queries.

Risk event The system administrators will not agree to open ports for executing queries Probability very unlikely (10%) Impact Delay due to the necessity of employing query proxies in order to

communicate across firewalls Prob. of impact 100% Total loss Two months due to the implementation Estimated loss Six days Mitigation plan There will be a need to implement some proxies based on protocols that typically traverse firewalls (e.g. HTTP) Responsible person Sławomir Zieliński

 Using JIMS as a permanent service

Although the monitoring agents are designed to be dynamically deployed and revoked, there are some components required to run permanently, such as RMI daemon and Jiro Deployment Station that hosts both monitoring agents and other dynamic system components (such as the management layer declaratively programmed entities).

Risk event The system administrators will not agree to run Deployment Stations permanently Probability unlikely (30%) Impact Necessity to run the system entities by hand using some administrator’s interface Prob. of impact 100% Total loss None from developers’ point of view, inconvenience for the users Estimated loss 0 Mitigation plan N/A Responsible person N/A

 Unavailability of required software

The main problem with the development of software is the lack of a freely available implementation of the Java Rule Engine Specification. Although there are some software packages with similar functionality (such as ILOG’s JRules or IBM’s CommonRules), they are available publicly only under trial licenses. The lack of free rule engine implementation will make the development harder. However, it is possible to build the management layer entities (maybe with limited functionality) based on some other technology. Such entities could be, for example, capable of interpreting some imperative language, instead of the declarative one. Although they would be considerably harder to use, still they could deliver the same functionality to the upper layers. Since the risk of not having the implementation of rule engine is high, another solution must be considered seriously. The environment for programming actions has to be able to: (a) integrate with existing software easily, (b) be programmed by the user. There exist some technologies that fulfill these requirements, such as BeanShell that is currently used to deploy and instantiate the instrumentation layer entities. The BeanShell is capable of

interpreting scripts in a language similar to Java. It can be integrated with a graphical tool, such as ExtFinder, and is equipped with an editor for scripts. The instrumentation layer entities can be bound to local BeanShell variables, that makes it possible to call their methods from inside the scripts. The notification of compound events could be launched by simple Jiro services also bound to BeanShell variables.

Risk event No free implementation of the Java Rule Engine is available Probability M18: very likely (90%), M24 or later: unlikely (30%) Impact Cannot use declarative language Prob. of impact 100% Total loss Harder programming of managerial actions and event filters Estimated loss Harder programming of managerial actions and event filters Mitigation plan Implementation of required functionality with different user programming language; work on BeanShell based interface has already started Responsible person Sławomir Zieliński

13.2.3 Derived Results: Postprocessing Below we present detailed risk analysis and loss estimation for several risk events we have identified to be possible.

 General

Risk event Losing a team member |Probability very unlikely (10%) Impact Problems with support and further development of some portions of application (intertask communication, forecasters) Prob. of impact 70% Total loss Two months due to finding appropriate person to continue the work and rewriting some parts of the code. Estimated loss Six days Mitigation plan Hiring a new person with some expertise in the field in question. Responsible person Task 3.3.4 Coordinator

Risk event No stable version of OGSA is available Probability possible (50%) Impact No impact Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan Our interface is not directly connected with OGSA architecture. In this

case we could continue with current solutions. Responsible person Task 3.3.4 Coordinator

Risk event DataGrid does not provide declared functionality required by T3.3.4 Probability very unlikely (10%) Impact Problem with construction of intertask communication Prob. of impact 70% Total loss Two months due to the implementation of alternative communication interfaces. Estimated loss Six days Mitigation plan Agreement between us and consumers of our data on some other output interface, adaptation of our software to that interface. Responsible person Task 3.3.4 Coordinator

Risk event GT2.x is not supported Probability very unlikely (10%) Impact No impact Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan Our task does not directly use native GT2.x tools and interfaces except of GridFTP, which will be supported in GT3 Responsible person Task 3.3.4 Coordinator

Risk event CrossGrid moves to GT3 Probability unlikely (30%) Impact No impact Prob. of impact N/A Total loss N/A Estimated loss N/A Mitigation plan N/A Responsible person Task 3.3.4 Coordinator

Risk event CrossGrid does not move to GT3 Probability likely (70%) Impact No impact Prob. of impact N/A Total loss N/A Estimated loss N/A

Mitigation plan N/A Responsible person Task 3.3.4 Coordinator

 Data sources

Because the Postprocessing task in its final version uses input data from other monitoring tasks (SANTA-G and JIMS), one of the main issues is availability of their data using a common API. We have chosen to use R-GMA both to exchange data between 3.3 subtasks and to publish the Postprocessing task results. This approach is preferred because of the expected migration of the R- GMA API to OGSA. It will also allow us to maintain consistency within task 3.3 parts. We see the following risks in this solution: the R-GMA functionality will not enable the provision of the required information, the R-GMA will not be available on the monitored sites, the R-GMA will not migrate to OGSA. These risks are summarized in the tables below.

Risk event The R-GMA functionality will not enable the provision of the required information Probability very unlikely (10%) Impact Delay due to the necessity of designing alternative solution Prob. of impact 70% Total loss Two months due to the implementation Estimated loss Six days Mitigation plan Using Spitfire to provide T3.3.4 interface Responsible person Task 3.3.4 Coordinator

Risk event The R-GMA will not be available on the monitored sites Probability very unlikely (10%) Impact Problem with construction of intertask communication Prob. of impact 70% Total loss Two months due to the implementation Estimated loss Six days Mitigation plan Using Spitfire to provide T3.3.4 interface Responsible person Task 3.3.4 Coordinator

Risk event R-GMA fails to move to OGSA Probability very unlikely (10%) Impact Delay due to need to refactoring of CanonicalProducer to web service Prob. of impact 100% Total loss 1 month Estimated loss 3 days Mitigation plan Using Spitfire to provide T3.3.4 interface

Responsible person Task 3.3.4 Coordinator

Risk event T3.3.2 and T3.3.3 functionalities will not enable the provision of the required information Probability very unlikely (10%) Impact Delay due to the necessity of designing alternative solution Prob. of impact 70% Total loss Two months due to the implementation Estimated loss Six days Mitigation plan Using a dedicated tool based on SAR (System Activity Reporter) to gather information necessary to construct forecasts Responsible person Task 3.3.4 Coordinator

To minimize the impact of the presented risks on our task we decided to construct an additional tool for collecting data from remote clusters using standard Linux and Globus tools (SAR - System Activity Reporter and GSI-ftp). This solution is very simple with respect to the R-GMA – OGSA approach, but turned out to be sufficient at the current state of the development. Two risks of this approach that we are taking into account are: unavailability of the SAR tool at remote clusters and lack of important system information that can be obtained using the SAR tool (which will for example be provided in the future by JIMS). In case the R-GMA is not ready or appropriate to provide interfaces between our task and our task’s data recipients, we consider as an alternative solution using Spitfire [Spitfire], a tool being developed in the EDG project serving as Grid enabled middleware service for access to relational databases. Installation of Spitfire on remote clusters by administrators should not pose a problem, as Spitfire exists as EDG package ready for installation.

 Development of forecasters basing on real Grid enabled applications

The forecasts are being constructed using methods (Kalman Filters (KF) and neural networks (NN)) that were never before applied to predict the status of the Grid. To verify the usefulness of this approach, data from real Grid enabled applications (ex. Grid enabled MPI applications) are necessary. Both the applications and tools to gather information about how the applications use Grid resources are being developed. The risk is that the power of forecasters to predict the state of the Grid will turn out not to be as effective as we expect. Since tools that are going to be used to construct forecasts are very advanced it is expected that they will give more reliable forecasts than obtained using deterministic methods. In the unlikely case it turns out not to be true it will be possible to use simpler methodology used for example by NWS, but anyhow our approach will be more tuned to the Crossgrid’s testbed infrastructure.

Risk event Bad effectiveness of (KF & NN) forecasters to predict the state of the Grid Probability unlikely (30%) Impact Delay due to the necessity of implementing alternative solution Prob. of impact 100% Total loss Two months due to the implementation

Estimated loss 18 days Mitigation plan Using deterministic algorithms to construct forecasts Responsible person Task 3.3.4 Coordinator

14 CONCLUDING REMARKS Task 3.3, Grid Monitoring provides both new services and extensions to existing services. The new service is aimed at applications monitoring whereas the extension to existing services provides the ability to monitor both infrastructure and instruments. The benefits of these approaches and a comparison to the current state of the art are discussed in Sections 4 and 5.

An important point in this deliverable is that of integration within Task 3.3. It is intended to provide a common interface to integrate the tools. This interface and further integration is covered in Section 10, Future Plans.

15 APPENDIX A The set of parameters that could be monitored using JIMS partially overlaps with such set defined in MDS schema (http://www.globus.org/mds/Schema.html). The following table presents the set of parameters exposed by the agent layer of the first JIMS prototype and corresponding (i.e. providing the same or similar information) MDS parameters. Note that:  the names of the attributes are JIMS internals, referring only to the agent layer and they are subject to change at the database layer,  the first prototype identifies SNMP attributes by using appropriate sequences of numbers (defined in RFC 1213),  the list of parameters available from SNMP agents is abbreviated (the table contains only the parameters important from the comparison point of view); full list is provided in RFC 1213.

JIMS description corresponding attribute MDS attribute Linux operating system attributes (simple,static) hname hostname of the monitored machine Mds-Host-hn model specific model of the hardware platform Mds-Computer-platform type instruction set architecture Mds-Computer-isa ncpus number of processors Mds-Cpu-Total-count ndisks number of local disks - maxmem maximum RAM memory available to users Mds-Memory-Ram-sizeMB maxswp maximum available swap space Mds-Memory-VM-sizeMB cline command line passed to kernel at boot time - ver version of the system Mds-Os-version iomap IO memory map of the system - cpuinf system CPU information Mds-Cpu-* Linux operating system attributes (simple,dynamic) l1m system load average (1 minute) Mds-Cpu-Total-Free- 1minX100 l5m system load average (5 minutes) Mds-Cpu-Total-Free- 5minX100 l15m system load average (15 minutes) Mds-Cpu-Total-Free- 15minX100 pg paging activity (pages in+pages out per second) - it cpu utilization – idle time (average on MP systems) Mds-Cpu-Total-Free-* ut cpu utilization – user time (average on MP systems) - utn cpu utilization – low priority (nice) user time - (average on MP systems) st cpu utilization – system time (average on MP systems) - swp available swap memory Mds-Memory-Vm-freeMB mem available (free) memory Mds-Memory-Ram-freeMB

memsh shared memory - membuf memory used for file buffers - memch memory used as cache memory - uptime system up time since boot - itime system idle time since boot - nproc number of processes - rproc number of running processes - Linux operating system attributes (compound,dynamic) fileSystemS  pathname the system is mounted on, Mds-Fs-mount, tatistics[]  device or server for filesystem, -  file system type (name, number), -  mount options, -  total data blocks in filesystem, Mds-Fs-sizeMB,  free blocks in filesystem, Mds-Fs-freeMB  free blocks available to non-superuser, -  optimal transfer block size, -  total file nodes in filesystem, -  free file nodes in filesystem -  maximum length of file names - Solaris 8,9 operating system attributes (simple, static) hname hostname of the monitored machine Mds-Host-hn model specific model of the hardware platform Mds-Computer-platform type instruction set architecture Mds-Computer-isa ncpus number of processors Mds-Cpu-Total-count ndisks number of local disks - maxmem maximum RAM memory available to users Mds-Memory-Ram-sizeMB maxswp maximum available swap space Mds-Memory-VM-sizeMB maxtmp maximum available space in temporary file system - ps page size - Solaris 8,9 operating system attributes (simple, dynamic) r1m run queue length (1 minute) - r5m run queue length (5 minutes) - r15m run queue length (15 minutes) - ls number of users logged in a system - pg paging activity (pages in+pages out per second) - it cpu utilization – idle time (average on MP systems) - ut cpu utilization – user time (average on MP systems) - st cpu utilization – system time (average on MP systems) -

swp available swap memory Mds-Memory-Vm-freeMB swpall allocated swap memory - swpres reserved swap memory - mem available (free) memory Mds-Memory-Ram-freeMB tmp available space in temporary filesystem - Solaris 8,9 operating system attributes (compound, dynamic) fileSystemS  pathname the system is mounted on, Mds-Fs-mount, tatistics[]  device or server for filesystem, -  file system type (name, number), -  mount options, -  total data blocks in filesystem, Mds-Fs-sizeMB,  free blocks in filesystem, Mds-Fs-freeMB  free blocks available to non-superuser, -  fundamental file system block size, -  optimal transfer block size, -  total file nodes in filesystem, -  free file nodes in filesystem, -  file nodes available to non-superuser, -  maximum length of file names - loggedUser  user login name, - sInformatio  remote host name, - n[]  name of the tty, -  time the user logged on, -  user process id - operatingSy  name of the implementation of the OS, Mds-Os-name, stemInform  name of the machine (hostname), Mds-Host-hn, ation  OS release, Mds-Os-release,  OS version, Mds-Os-version,  machine, -  instruction set architecture, Mds-Computer-isa  specific model of the hardware platform, -  variant instruction set architectures executable on the - current system,  hardware manufacturer, -  hardware-specific serial number - numberOfP  number of processors configured, Mds-Cpu-Total-count rocessors  number of processors online -

loadAverag  load average (1minute), Mds-Cpu-Total-Free- e 1minX100  load average (5minutes), Mds-Cpu-Total-Free- 5minX100  load average (15minutes) Mds-Cpu-Total-Free- 15minX100 memorySta  page size, - tistics  total pages, -  available pages - cpusStatisti  cpu id, - cs[]  page ins, -  page outs, -  pages paged in, -  pages paged out, -  page reclaims, -  page reclaims from free list, -  pages freed by daemon or auto, -  minor page faults, -  major page faults, -  cpu utilization – idle time, -  cpu utilization – user time, -  cpu utilization – system time, -  device interrupts, -  system calls, -  cpu context switches, -  thread migrations - diskStatisti  data read, - cs[]  data written, -  read operations, -  write operations - Selected parameters available from SNMP agents located on routers, switches, etc. group:  system name, - system  description (textual description of the entity, including - hardware type, operating system, networking software),  contact person, -  uptime, -  physical location of the node, -  services -

group:  number of network interfaces, - interfaces  interface descriptions, -  interface type, -  interface MTU, -  estimated interface speed, -  interface status (administrative/operational), -  octets in/out, -  unicast packets in/out, -  non-unicast packets in/out, -  errors counters - group: ip  forwarding (gateways: true, others: false), -  default TTL, -  datagrams in/out, -  datagrams in/out requests, -  errors counters, -  fragmentation/reassembly counters, -  routing table: - - destination, - interface index, - route metrics, - next hop, - ... group:  in/out counters for distinct ICMP message types - icmp group: tcp  retransmission-related counters, -  active/passive connection open counters, -  connections parameters/state table - group: udp  datagrams in/out, -  information about UDP listeners - group:  snmp packets in/out, - snmp  operations counters, -  error counters -

The following table contains a list of MDS parameters not supported or partially supported by the agent layer of JIMS first prototype: MDS attribute(s) explanation / future plans Mds-validfrom, these attributes refer to the validity period of a specified record; such Mds-validto, functionality is to be implemented at the database layer

Mds-keepto class MdsHostNodeGroup this class of attributes refers more to monitoring of a clusters configuration, than to infrastructure monitoring; however, it can be incorporated in future releases class this class is definitely related to configuration monitoring; therefore it is MdsSoftwareDeployment not included in the first prototype classes MdsCpuSmp, to be included in future releases (probably after second prototype) MdsCpuCache classes attributes of these classes can be computed from atomic values; the MdsMemoryRamTotal, database layer of JIMS is expect to provide this MdsMemoryVmTotal, MdsFsTotal class MdsNet currently the user is required to use SNMP agents in order to get values for the attributes of this class; future releases would use operating system API’s for this purpose classes MdsService, not applicable MdsServiceLdap

CG3.3-D3.4-v1.4-TCD016-M15Deliverable Confidential 61 / 61