GWD-TYPE 4/7/2003
Accounting Interchange Natural Language Description (Requirements)
Status of This Memo
This memo provides information to the Grid community regarding the Usage Record format requirements. Distribution is unlimited.
Copyright Notice
Copyright © Global Grid Forum (2002). All Rights Reserved.
Abstract For resources to be shared, sites must be able to exchange basic accounting and usage data in a common format. This format must encompass both job level accounting, and aggregate accounting. This document describes the requirements in natural language form for a Usage Record standard.
Abstract...... 1 1. Base Properties...... 2 1.1 Charge...... 2 1.2 Status...... 2 1.3 Jobname...... 2 1.4 Network...... 3 1.5 Disk...... 3 1.6 Memory...... 3 1.7 Walltime...... 3 1.8 Cputime...... 4 1.9 Numnodes...... 4 1.10 Processors...... 4 1.11 NonCompute...... 4 1.12 Factor...... 5 1.13 End Time...... 5 1.14 Start Time...... 5 1.15 Exec Host...... 5 1.16 From Host...... 5 1.17 Queue...... 5 1.18 Jobid...... 6 1.19 Project name...... 6 1.20 Extension...... 6 1.21 Factor...... 6 1.22 End Time...... 7 1.23 Start Time...... 7 1.24 Exec Host...... 7 1.25 From Host...... 7 1.26 Queue...... 7 1.27 Jobid...... 7 1.28 Username...... 8 1.29 Resource...... 8 1.30 Units...... 8 2. Job Level Accounting...... 8 3. Aggregate Accounting...... 9 4. Fully Qualified Domain Names (FQDN)...... 9 5. Other Requirements...... 9
[email protected] 1 GWD-TYPE 4/7/2003
6. Units Definition...... 9 7. Security Considerations...... 11 Author Information...... 11 Intellectual Property Statement...... 11 Full Copyright Notice...... 11 References...... 12
1. Base Properties
The following is a list of "base" properties that compose the Usage Record requirements for both job level and aggregate properties. The definitions for the type of data (integer, string) are the base data types. The naming convention and naming style of these properties are implementation specific, and should use the referring name as a guideline for any naming.
Properties may carry meta-properties that can assist in establishing semantic meaning from the actual property value, such as “metric” and “units”.
1.1 Charge
This property represents the total charge of the job in the system’s allocation unit. For example 100, 200, or 3000. The meaning of this charge will be site dependent.
This property SHOULD be referred to as charge It MUST contain data of type integer This property MAY be present There SHOULD be no restriction on the data values or length The metric MAY be present. The metric may be Global or Local. The default is Local. A local charge would be a charge internal to the site, while the global charge may a grid charge.
1.2 Status
This property will represent the completion status of the job. This is typically the exit status of the running process. For example, in the Portable Batch System (PBS) this is encoded as Exit_status in the accounting record. The meaning of the Status will be site dependent
This property SHOULD be referred to as status This property MUST contain data of type integer This property MAY not be present There SHOULD be no restriction on the data values The metric MAY be specified. There is no default metric
1.3 Jobname
The job or application name. For example, this could be the name of the executable that was ran, or the name of the batch queuing systems name for the job
This property SHOULD be referred to as jobname This property MUST contain data of type string This property MAY not be present There SHOULD be no restriction on the length, or restrictions on available characters
[email protected] 2 GWD-TYPE 4/7/2003
The metric MAY be specified. The supported metrics are Global and Local. A local metric may indicate the jobname assigned by a local queuing system. A Global metric may indicate the jobname as assigned by a grid scheduler. The default is local.
1.4 Network
The network used by the job. Could be Average, Total, or Max usage
This property SHOULD be referred to as network This property MUST contain data of type integer This property MAY not be present Units SHOULD be specified, possible values SHOULD be restricted to "KB", "MB", or "GB" Default unit MUST be MB This metric MAY be present. The default is “Total”. The metric is limited to “Total”, “Average”, “Min”, and “Max”
1.5 Disk
Disk storage used
This property SHOULD be referred to as disk This property MUST contain data of type integer This property MAY not be present Units MUST be specified, possible values SHOULD be restricted to "KB", "MB", “GB”, or “blocks” Default unit MUST be MB Metric MAY be present. The default is “Total”. The metric is limited to Average, Total, Max, Min, Requested.
1.6 Memory
The maximum amount of virtual memory used by all concurrent processes in the job
This property SHOULD be referred to as memory This property MUST contain data of type integer This property MAY not be present Units MUST be specified, possible values SHOULD be restricted to "KB", "MB", “GB” Default unit MUST be MB This metric MAY be present. The default is “Total”. The metric is limited to “Total”, “Average”, “Min”, and “Max”
1.7 Walltime
Wall clock time which elapsed while the job was running
This property SHOULD be referred to as walltime This property MUST contain data of type integer This property MAY not be present The default unit MUST be seconds The value MUST be greater than or equal to zero Metric MAY be present. The default is “Granted”. Possible values are “Granted” and “Requested”
[email protected] 3 GWD-TYPE 4/7/2003
1.8 Cputime
CPU time used, summed over all processes in the job
2. This property SHOULD be referred to as cputime This property MUST contain data of type integer This property MAY not be present The default unit MUST be in seconds The value MUST be greater than or equal to zero Metric MAY be present. The default is “Granted”. Possible values are “Granted” and “Requested”
2.1 Numnodes
Number of nodes used. A node definition may be dependent on the architecture, but typically a node is a physical machine. For example an IBM SP2 of 16 machines with one processor each is a 16 “node” machine, each with one “processor”. A 16 processor Origin however, is 1 physical node (machine) with 16 processors.
This property SHOULD be referred to as numnodes This property MUST contain data of type integer This property MAY not be present The value MUST be greater than zero Metric MAY be present. The default is “Granted”. Possible values are “Granted” and “Requested”
2.2 Processors
The number of processors used or requested. A processor definition may be dependent on the machine architecture. Typically processor is equivalent to the number of physical CPU’s used. For example, if a job uses two SP2 “nodes”, each node having 16 cpu’s each, the number of total processors would be 32.
This property SHOULD be referred to as processors This property MUST contain data of type integer This property MAY not be present The value MUST be greater than zero Metric MAY be present. The default is “Granted”. Possible values are “Granted” and “Requested”
2.3 NonCompute
A noncomputing resource used by a job. For example, consulting hours are considered a non- compute resource
This property SHOULD be referred to as noncompute This property MUST contain data of type String This property MAY not be present Metric MAY be present. There is no default
[email protected] 4 GWD-TYPE 4/7/2003
2.4 Factor
A prorata that can be used in conjunction with the reported quantity to represent consumption conditions such as premiums, discounts, node splitting and consumption rates.
This property SHOULD be named factor This property MUST contain data of type float This property MAY be omitted. Metric MAY be present. No default is specified
2.5 End Time
The date the job completed, in UTC time zone format (ISO 8601) 3. This property SHOULD be referred to as endttime This property MUST contain data of type date This property MAY not be present This property data MUST conform to UTC time zone format as described by ISO 8601.
3.1 Start Time
The date the job started, in UTC time zone format (ISO 8601). Start time may be dependent on the queue system. For example some systems include time to stage files, others do not.
The name of This property MUST be starttime This property MUST contain data of type date This property MAY not be present This property data MUST conform to UTC time zone format as described by ISO 8601
3.2 Exec Host
The system hostname where the job ran on.
This property SHOULD be referred to as exechost This property MUST contain data of type string This property MAY not be present The data MUST be a (FQDN). See the section on FQDN
3.3 From Host
The system name where the job was submitted from.
This property SHOULD be referred to as fromhost This property MUST contain data of type string This property MAY not be present The data MUST be a (FQDN). See the section on FQDN
3.4 Queue
The name of the queue where the job was either executed or submitted from.
This property SHOULD be referred to as queue
[email protected] 5 GWD-TYPE 4/7/2003
This property MUST have data of type string This property MAY not be present There SHOULD be no restriction on the length, or restrictions on available characters
3.5 Jobid
The job id as assigned by the batch queue, or PID of the job for interactive jobs.
This property SHOULD be referred to as jobid This property MUST have data of type integer This property MAY not be present The data SHOULD not have any restrictions
3.6 Project name
The users account where the usage which be charged as determined by the accounting system. Some accounting systems define this is the ACID, which is often the same as the users Unix group from /etc/passwd. The project is also referred to as the effective gid under which the job consumed resources on some systems.
This property MUST be called projectname This property MUST have data of type string This property MAY not be present There SHOULD be no restriction on the length or available characters
3.7 Extension
For sites that may want to exchange data that is not one of the base properties (for example, perhaps grid telescope power ) the Extension property can be used to encode any type of usage information. The sites can agree on the name, metric, units to be supported by the extension
This property MUST be called extension This property MUST have data of type string This property MAY not be present There SHOULD be no restriction on the length, or restrictions on available characters There MAY be an undefined metric There MAY be an undefined unit There MAY be an undefined String name, for example “TelescopePowerUsage”.
3.8 Factor
A prorata that can be used in conjunction with the reported quantity to represent consumption conditions such as premiums, discounts, node splitting and consumption rates.
This property SHOULD be named factor This property MUST contain data of type float This property MAY be omitted. Metric MAY be present. No default is specified
[email protected] 6 GWD-TYPE 4/7/2003
3.9 End Time
The date the job completed, in UTC time zone format (ISO 8601) 4. This property SHOULD be referred to as endttime This property MUST contain data of type date This property MAY not be present This property data MUST conform to UTC time zone format as described by ISO 8601.
4.1 Start Time
The date the job started, in UTC time zone format (ISO 8601). Start time may be dependent on the queue system. For example some systems include time to stage files, others do not.
The name of This property MUST be starttime This property MUST contain data of type date This property MAY not be present This property data MUST conform to UTC time zone format as described by ISO 8601
4.2 Exec Host
The system hostname where the job ran on.
This property SHOULD be referred to as exechost This property MUST contain data of type string This property MAY not be present The data MUST be a (FQDN). See the section on FQDN
4.3 From Host
The system name where the job was submitted from.
This property SHOULD be referred to as fromhost This property MUST contain data of type string This property MAY not be present The data MUST be a (FQDN). See the section on FQDN
4.4 Queue
The name of the queue where the job was either executed or submitted from.
This property SHOULD be referred to as queue This property MUST have data of type string This property MAY not be present There SHOULD be no restriction on the length, or restrictions on available characters
4.5 Jobid
The job id as assigned by the batch queue, or PID of the job for interactive jobs.
This property SHOULD be referred to as jobid [email protected] 7 GWD-TYPE 4/7/2003
This property MUST have data of type integer This property MAY not be present The data SHOULD not have any restrictions Metric MAY be present. The supported metrics are local for a local jobid that may have be assigned by the local scheduler, and global for a jobid that may have be assigned by a grid type scheduler. The default metric is local.
4.6 Username
The user’s login name corresponding to the users id in the /etc/passwd file on Unix systems for a local metric, or the DN for a global metric. The meaning will be site specific.
This property MUST be called username This property MUST have data of type string This property MAY not be present There SHOULD be no restriction on the length, or restrictions on available characters There MAY be a metric. The supported metrics are local and global. The default is local.
4.7 Resource
This property represents consumable resources associated with the execution of a job. Resources are defined categorically with several common pieces of data associated with the resource detailed by this property. Each resource property may include the following base properties, described previously.
Network Disk Memory Walltime CPUtime NumNodes Processors NonCompute
4.8 Units
The unit of measurement used for quantifying the resource consumption. The legal values for this meta property depend on the type of resource property, and are defined in the section for the resource property and summarized in table 1.
This property SHOULD be named unit This property MUST contain data of type string This property MAY be omitted. If omitted, the default unit MAY be inferred.
5. Job Level Accounting Job level accounting reports accounting data at the job level. PBS and LoadLeveller for example, use job level accounting where each job is assigned a unique id and the accounting for that job is
[email protected] 8 GWD-TYPE 4/7/2003 reported for each id. The properties listed in this document are not required to appear within the usage record structure in the order .
Job Level accounting MAY contain any of the base properties listed in this document, which appear exactly once per job. Job Level properties SHOULD be contained within a “job” property for each job
6. Aggregate Accounting
Aggregate accounting reports the accounting data in aggregate (summarized form). The properties listed in this document are not required to appear within the usage record structure in the order .
Aggregate accounting MAY contain any of base properties listed in this document, which appear exactly once per job. Aggregate properties SHOULD be contained within an “aggregate” property.
7. Fully Qualified Domain Names (FQDN) Any fully qualified domain names that appear (like ultramega.engine.umich.edu) MUST follow the following naming convention as defined by RFC 1034, summarized here:
FQDN’s can be up to 255 characters long and can contain alphabetic and numeric characters and the "-" and "." characters.
8. Other Requirements The XML implementation must have the root property of the document be UsageRecord
9. Units Definition
The units for processor storage, real and virtual memory use the abbreviations KB, MB, and GB. a megabyte (abbreviated MB) is 2 to the 20th power bytes, or 1,048,576 bytes in decimal notation. KB is 2 to the tenth power bytes, and gigabyte is 2 to the 30 power bytes.
Table 1 Base Properties Units and Default Units
Property Name Supported Units Default Unit charge NONE NONE status NONE NONE Jobname NONE NONE Network KB”, “MB”, “GB” MB Disk KB”, “MB”, “GB” MB Memory KB”, “MB”, “GB” MB Walltime Seconds, minutes, hours Seconds
[email protected] 9 GWD-TYPE 4/7/2003
Cputime Seconds, minutes, hours Seconds NumNodes NONE NONE Processors NONE NONE NonCompute NONE NONE Factor NONE NONE EndTime Seconds, minutes, hours Seconds StartTime Seconds, minutes, hours Seconds Exec Host NONE NONE From Host NONE NONE Distinguished Name NONE NONE Queue NONE NONE Jobid Global, Local Local Project Name NONE NONE Username Global, Local Local Extension NONE NONE
Table 2 Base Properties Metric and Default Metric
Property Name Supported Metric Default Metric charge Global, Local Global status NONE NONE Jobname Global, Local NONE Network Average, Total, Max, Min, Requested Total Disk Average, Total, Max, Min, Requested Total Memory Average, Total, Max, Min, Requested Total Walltime Requested, Granted Granted Cputime Requested, Granted Granted NumNodes Requested, Granted Granted Processors Requested, Granted Granted NonCompute Requested, Granted Granted Factor NONE NONE EndTime NONE NONE StartTime NONE NONE Exec Host NONE NONE From Host NONE NONE Distinguished Name NONE NONE Queue NONE NONE
[email protected] 10 GWD-TYPE 4/7/2003
Jobid NONE NONE Project Name NONE NONE Username NONE NONE Extension NONE NONE
10. Security Considerations
This is no security considerations for the requirements.
Author Information
Rodney Mach, Center for Advanced Computing
.
Intellectual Property Statement
The GGF takes no position regarding the validity or scope of any intellectual property or other rights that might be claimed to pertain to the implementation or use of the technology described in this document or the extent to which any license under such rights might or might not be available; neither does it represent that it has made any effort to identify any such rights. Copies of claims of rights made available for publication and any assurances of licenses to be made available, or the result of an attempt made to obtain a general license or permission for the use of such proprietary rights by implementers or users of this specification can be obtained from the GGF Secretariat.
The GGF invites any interested party to bring to its attention any copyrights, patents or patent applications, or other proprietary rights which may cover technology that may be required to practice this recommendation. Please address the information to the GGF Executive Director.
Full Copyright Notice
Copyright (C) Global Grid Forum (1/18/2003). All Rights Reserved.
This document and translations of it may be copied and furnished to others, and derivative works that comment on or otherwise explain it or assist in its implementation may be prepared, copied, published and distributed, in whole or in part, without restriction of any kind, provided that the above copyright notice and this paragraph are included on all such copies and derivative works. However, this document itself may not be modified in any way, such as by removing the copyright notice or references to the GGF or other organizations, except as needed for the purpose of developing Grid Recommendations in which case the procedures for copyrights defined in the GGF Document process must be followed, or as required to translate it into languages other than English.
The limited permissions granted above are perpetual and will not be revoked by the GGF or its successors or assigns.
[email protected] 11 GWD-TYPE 4/7/2003
This document and the information contained herein is provided on an "AS IS" basis and THE GLOBAL GRID FORUM DISCLAIMS ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE."
References
KEYWORDS RFC 2119. Key words for use in RFCs to Indicate Requirement Levels. S. Bradner. March 1997. http://www.ietf.org/rfc/rfc2119.txt
Resource Survey DRAFT READY FOR GGF6