The Advanced Resource Connector User Guide

Total Page：16

File Type：pdf, Size：1020Kb

The Advanced Resource Connector User Guide

NORDUGRID NORDUGRID-MANUAL-6 7/12/2010 The NorduGrid/ARC User Guide Advanced Resource Connector (ARC) usage manual 2 Contents 1 Preface 11 2 Roadmap 13 3 Client Installation 15 3.1 Beginner Installation . 15 3.1.1 Download . 15 3.1.2 Unpack . 15 3.1.3 Configure . 16 3.2 System-wide installation . 16 3.2.1 Installation via repositories . 17 3.2.2 Globus Toolkit . 18 3.2.3 Download the client . 18 3.2.4 Build . 19 3.2.5 Install the client . 20 3.3 Client configuration files . 21 3.3.1 client.conf . 21 3.3.2 srms.conf . 23 3.3.3 Deprecated configuration files . 24 4 Grid Certificates 25 4.1 Quick start with certificates . 25 4.1.1 Certificates in local standalone client . 25 4.1.2 Certificates for system-wide client installation . 27 4.1.3 Obtain a personal certificate . 27 4.2 Grid Authentication And Certificate Authorities . 28 4.2.1 Grid Login, Proxies . 28 4.2.2 Certificate Authorities . 29 4.2.3 Globus Grid certificates . 29 4.2.4 Associating yourself with a proper CA . 30 4.2.5 Friendly CAs . 31 4.2.6 Certificate Request . 32 4.2.7 Working with certificates: examples . 33 3 4 CONTENTS 5 Getting Access to Grid Resources: Virtual Organisations 35 5.1 NorduGrid Virtual Organisation . 35 5.2 Other Virtual Organisations . 36 6 Grid Session 37 6.1 Logging Into The Grid . 37 6.1.1 Proxy Handling Tips . 38 6.2 First Grid test . 38 6.3 Logging Out . 39 7 Describing Grid Tasks 41 7.1 Task Description Language: xRSL . 41 7.1.1 URLs . 42 7.1.2 Task Description: Attributes . 45 executable . 46 arguments . 46 inputFiles . 46 executables . 47 cache.................................................. 47 outputFiles . 48 cpuTime . 49 wallTime . 50 gridTime . 50 benchmarks . 51 memory . 51 disk .................................................. 51 runTimeEnvironment . 52 middleware . 52 opsys.................................................. 53 stdin.................................................. 53 stdout . 53 stderr . 54 join................................................... 54 gmlog ................................................. 54 jobName . 54 ftpThreads . 55 acl................................................... 55 cluster . 55 queue ................................................. 56 startTime . 56 lifeTime . 57 notify . 57 CONTENTS 5 replicaCollection . 57 rerun.................................................. 58 architecture . 58 nodeAccess . 58 dryRun . 58 rsl substitution . 59 environment . 59 count.................................................. 60 jobreport . 60 credentialserver . 60 7.2 Examples . 60 7.2.1 Hello World . 61 7.2.2 An Own Executable . 62 7.2.3 Many Executables . 62 8 Job Management 65 8.1 First steps: the test suite . 65 8.1.1 ngtest . 65 8.2 Working with jobs . 66 8.2.1 ngsync . 66 8.2.2 ngsub . 67 8.2.3 ngstat . 71 8.2.4 ngcat . 72 8.2.5 ngget . 73 8.2.6 ngkill . 75 8.2.7 ngresub . 76 8.2.8 ngclean . ..

123

Copy Link

Recommended publications

Grid Computing: What Is It, and Why Do I Care?*
Grid Computing: What Is It, and Why Do I Care?* Ken MacInnis <[email protected]> * Or, “Mi caja es su caja!” (c) Ken MacInnis 2004 1 Outline Introduction and Motivation Examples Architecture, Components, Tools Lessons Learned and The Future Questions? (c) Ken MacInnis 2004 2 What is “grid computing”? Many different definitions: Utility computing Cycles for sale Distributed computing distributed.net RC5, SETI@Home High-performance resource sharing Clusters, storage, visualization, networking “We will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.” Len Kleinrock (1969) The word “grid” doesn’t equal Grid Computing: Sun Grid Engine is a mere scheduler! (c) Ken MacInnis 2004 3 Better definitions: Common protocols allowing large problems to be solved in a distributed multi-resource multi-user environment. “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities.” Kesselman & Foster (1998) “…coordinated resource sharing and problem solving in dynamic, multi- institutional virtual organizations.” Kesselman, Foster, Tuecke (2000) (c) Ken MacInnis 2004 4 New Challenges for Computing Grid computing evolved out of a need to share resources Flexible, ever-changing “virtual organizations” High-energy physics, astronomy, more Differing site policies with common needs Disparate computing needs
[Show full text]
The Translational Journey of the Htcondor-CE
Journal of Computational Science xxx (xxxx) xxx Contents lists available at ScienceDirect Journal of Computational Science journal homepage: www.elsevier.com/locate/jocs Principles, technologies, and time: The translational journey of the HTCondor-CE Brian Bockelman a,*, Miron Livny a,b, Brian Lin b, Francesco Prelz c a Morgridge Institute for Research, Madison, USA b Department of Computer Sciences, University of Wisconsin-Madison, Madison, USA c INFN Milan, Milan, Italy ARTICLE INFO ABSTRACT Keywords: Mechanisms for remote execution of computational tasks enable a distributed system to effectively utilize all Distributed high throughput computing available resources. This ability is essential to attaining the objectives of high availability, system reliability, and High throughput computing graceful degradation and directly contribute to flexibility, adaptability, and incremental growth. As part of a Translational computing national fabric of Distributed High Throughput Computing (dHTC) services, remote execution is a cornerstone of Distributed computing the Open Science Grid (OSG) Compute Federation. Most of the organizations that harness the computing capacity provided by the OSG also deploy HTCondor pools on resources acquired from the OSG. The HTCondor Compute Entrypoint (CE) facilitates the remote acquisition of resources by all organizations. The HTCondor-CE is the product of a most recent translational cycle that is part of a multidecade translational process. The process is rooted in a partnership, between members of the High Energy Physics community and computer scientists, that evolved over three decades and involved testing and evaluation with active users and production infrastructures. Through several translational cycles that involved researchers from different organizations and continents, principles, ideas, frameworks and technologies were translated into a widely adopted software artifact that isresponsible for provisioning of approximately 9 million core hours per day across 170 endpoints.
[Show full text]
Usage Statistics and Usage Patterns on the Nordugrid: Analyzing the Logging Information Collected on One of the Largest Production Grids of the World
CHEP04 Contribution ID: 102 Type: poster Usage statistics and usage patterns on the NorduGrid: Analyzing the logging information collected on one of the largest production Grids of the world. Wednesday, 29 September 2004 10:00 (0 minutes) The Nordic Grid facility (NorduGrid) came into production operation during the summer of 2002 when the Scandinavian Atlas HEP group started to use the Grid for the Atlas Data Challenges and was thus the first Grid ever contributing to an Atlas production. Since then, the Grid facility has been in continuous 24/7 operation offering an increasing number of resources to a growing set of active users coming from various scientific areas including chemistry, biology, informatics. As of today the Grid has grown into one of the largest production Grids of the world continuously running Grid jobs on the more than 30 Grid-connected sites which offer over 2000 CPUs. This article will start with a short overview of the design and implementation of the Advanced Resource Connector (ARC), the NorduGrid middleware, which delivers reliable Grid services to the NorduGrid production facility. This will be followed by a presentation of the logging facility of NorduGrid, describing the logging service and the collected information. The main part of the talk will focus onthe analysis of the collected logging information: usage statistics, usage patterns (what is a typical grid job on the NorduGrid looks like?). Use cases from different application domains will also be discussed. References: -NorduGrid live: www.nordugrid.org -> Grid Monitor -Atlas Data-Challenge 1 on NorduGrid: http://arxiv.org/abs/physics/0306013 Primary authors: KONSTANTINOV, A.
[Show full text]
Performance of the Nordugrid ARC and the Dulcinea Executor in ATLAS Data Challenge 2
Performance of the NorduGrid ARC and the Dulcinea Executor in ATLAS Data Challenge 2 Sturrock, R.; Eerola, Paula; Konya, Balazs; Smirnova, Oxana; Lindemann, Jonas; et, al. Published in: CERN-2005-002 2005 Link to publication Citation for published version (APA): Sturrock, R., Eerola, P., Konya, B., Smirnova, O., Lindemann, J., & et, A. (2005). Performance of the NorduGrid ARC and the Dulcinea Executor in ATLAS Data Challenge 2. In A. Aimar, J. Harvey, & N. Knoors (Eds.), CERN- 2005-002 (Vol. 2, pp. 1095-1098). CERN. Total number of authors: 6 General rights Unless other specific re-use rights are stated the following general rights apply: Copyright and moral rights for the publications made accessible in the public portal are retained by the authors and/or other copyright owners and it is a condition of accessing publications that users recognise and abide by the legal requirements associated with these rights. • Users may download and print one copy of any publication from the public portal for the purpose of private study or research. • You may not further distribute the material or use it for any profit-making activity or commercial gain • You may freely distribute the URL identifying the publication in the public portal Read more about Creative commons licenses: https://creativecommons.org/licenses/ Take down policy If you believe that this document breaches copyright please contact us providing details, and we will remove access to the work immediately and investigate your claim. LUND UNIVERSITY PO Box 117 221 00 Lund +46 46-222 00
[Show full text]
GT 4.0 RLS GT 4.0 RLS Table of Contents
GT 4.0 RLS GT 4.0 RLS Table of Contents 1. Key Concepts .................................................................................................................................. 1 1. Overview of Data Management in GT4 ........................................................................................ 1 2. Data movement ........................................................................................................................ 1 3. Data replication ....................................................................................................................... 2 4. Higher level data services .......................................................................................................... 4 2. 4.0.0 Release Note ............................................................................................................................ 5 1. Component Overview ............................................................................................................... 5 2. Feature Summary ..................................................................................................................... 5 3. Bug Fixes ............................................................................................................................... 5 4. Known Problems ...................................................................................................................... 6 5. Technology Dependencies .........................................................................................................
[Show full text]
A Globus Primer Or, Everything You Wanted to Know About Globus, but Were Afraid to Ask Describing Globus Toolkit Version 4
A Globus Primer Or, Everything You Wanted to Know about Globus, but Were Afraid To Ask Describing Globus Toolkit Version 4 An Early and Incomplete Draft Please send comments, criticisms, and suggestions to: [email protected] Preface The Globus Toolkit (GT) has been developed since the late 1990s to support the development of service-oriented distributed computing applications and infrastructures. Core GT components address basic issues relating to security, resource access and management, data movement and management, resource discovery, and so forth. A broader “Globus universe” comprises numerous tools and components that build on core GT4 functionality to provide many useful application- level functions. These tools have been used to develop many Grid systems and applications. Version 4 of the Globus Toolkit, GT4, released in April 2005, represents a significant advance relative to the GT3 implementation of Web services functionality in terms of the range of components provided, functionality, standards conformance, usability, and quality of documentation. This document is intended to provide a first introduction to key features of both GT4 and associated tools, and the ways in which these components can be used to develop Grid infrastructures and applications. Its focus is on the user’s view of the technology and its application, and the practical techniques that should be employed to develop GT4-based applications. We discuss in turn the applications that motivate the development of GT4 and related tools; the four tasks involved in building Grids: design, deployment, application, operations; GT4 structure, including its Web services (WS) and pre-WS components; the Globus universe and its various components; GT4 performance, scalability, and functionality testing; porting to GT4 from earlier versions of GT; who’s using GT4; likely future directions, and a few words of history.
[Show full text]
Status Report
Nordic Testbed for Wide Area Computing and Data Handling 9/01/01 NORDIC TESTBED FOR WIDE AREA COMPUTING AND DATA HANDLING (NORDUGRID) Status Report Introduction The purpose of the project is to create the grid computing infrastructure in Nordic countries. Project participants include universities and research centers in Denmark, Sweden, Finland and Norway. The active phase of the project started in May 2001, and involves the Niels Bohr Institute (Denmark), Lund and Uppsala Universities (Sweden), University of Oslo (Norway) and Helsinki Institute of Physics (Finland). From the very beginning, the NorduGrid testbed became an integral part of the EU DataGrid project, initiated at CERN and aimed at creation of the computing infrastructure for the future high−energy physics experiments. This report overviews the status of the NorduGrid project as of September 1, 2001. 1. Organization Due to the considerable geographical spread of the project member sites, the organization and management heavily relies on the Internet and teleconferences. The project Web site (http://www.quark.lu.se/grid) opened in December 2000, and is regularly updated with the most detailed information. 1.1. Responsibilities 1.1.1. Steering Panel Prior to the project’s start, the steering panel was appointed to provide adequate management. It consists of the coordinator, John Renner Hansen, representing Niels Bohr Institute, Denmark; Tord Ekelöf, Paula Eerola and Sten Hellman, representing Uppsala, Lund and Stockholm Universities of Sweden; Farid Ould−Saada from the University of Oslo, Norway, and Matti Heikkurinen, representing Helsinki Institute of Physics, Finland. The group holds regular phone meetings. 1.1.2. New positions Three new postdoctoral positions, funded by the project, were created.
[Show full text]
The Nordugrid Architecture and Middleware for Scientific Applications
The NorduGrid Architecture And Middleware for Scientific Applications O. Smirnova1, P. Eerola1, T. Ekelöf2, M. Ellert2, J. R. Hansen3, A. Konstantinov4, B. Kónya1, J. L. Nielsen3, F. Ould-Saada5, and A. Wäänänen3 1 Particle Physics, Institute of Physics, Lund University, Box 118, 22100 Lund, Sweden 2 Department of Radiation Sciences, Uppsala University, Box 535, 75121 Uppsala, Sweden 3 Niels Bohr Institutet for Astronomi, Fysik og Geofysik, Blegdamsvej 17, DK-2100, Copenhagen Ø, Denmark 4 Vilnius University, Institute of Material Science and Applied Research, Saul˙etekio al. 9, Vilnius 2040, Lithuania 5 University of Oslo, Department of Physics, P. O. Box 1048, Blindern, 0316 Oslo, Norway Abstract. The NorduGrid project operates a production Grid infras- tructure in Scandinavia and Finland using own innovative middleware solutions. The resources range from small test clusters at academic in- stitutions to large farms at several supercomputer centers, and are used for various scientific applications. This talk reviews the architecture and describes the Grid services, implemented via the NorduGrid middleware. 1 Introduction The NorduGrid project [1] started in May 2001, initiated by academic institutes in Scandinavia and Finland, aiming to build a Nordic testbed for wide area com- puting and data handling. After evaluating the existing Grid solutions (such as the Globus ToolkitTM [2] and the EU Datagrid (EDG) [3]), the NorduGrid devel- opers came up with an original architecture and implementation proposal [4, 5]. The NorduGrid testbed was set up accordingly in May 2002, and is in continuous operation and development since August 2002. By now it has grown into one of the largest operational Grids in the world, including not only test sites, but also several large production clusters.
[Show full text]
Nordugrid ARC Tutorial
ARARCC TTututoorriialal Aleksandr Konstantinov Vilnius University/Lithuania and University of Oslo/Norway CoContnteentntss What is Grid? – NorduGrid – What changes? ARC • How it works Misunderstandings. • Job's definition Applications • Preparation Middleware • Simple job – Legion • Sophisticated job – Unicore – Globus Grid users – Authentication – Authorization 2005-03-17 www.nordugrid.org 2 WhatWhat iiss GGrriidd?? Relatively new term – definition is fuzzy. – Many users, many definitions – Common misunderstandings – Standardization just http://www.globalgridforum.org/. Old idea – Uniform and safe access to geographically remote and inhomogeneous computing resources. – Dynamic pool of users and resources. – Distributed management. – Resources belonging to different institutions are linked into system. – Collaboration and social networking are as important as technology Grid environment is usually formed a layer over operating system installed on participating resources and other services. – Term used - “middleware”. I. Foster and C. Kesselmann, The Grid: Blueprint for a New Computing Infrastructure. Morgan Kaufman Publishers, 1998 2005-03-17 www.nordugrid.org 3 WhatWhat changechangess?? Physical location of computing and data resources not important. Unification is important – Usernames and passwords are not used – Unification of resources: virtualization or unified interfaces. • A lot of unsolved problems. • Standardization just started. • Maybe virtual computers can help. • What should adapt, user's application or Grid environment? 2005-03-17 www.nordugrid.org 4 CoCommmmoonn mmiissundundeerrssttandandiingsngs Grid increases resources – One computing resource won't be able to compute more if put on a Grid. – But resources can be used more effectively. Grid is black box and everything inside it happens automatically. – So far in future plans only – In reality users still have to learn about peculiarities of implementation.
[Show full text]
Parallelism of Machine Learning Algorithms
10/9/2013 Outline Condor Presented by : Walid Budgaga The Anatomy of the Grid Globus Toolkit CS 655 – Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 2 Motivation Motivation High Throughput Computing (HTC)? HTC is suitable for scientific research Large amounts of computing capacity over long Example(Parameter sweep): periods of time. Testing parameter combinations to Measured: operations per month or per year High Performance Computing (HPC)? keep temp. at particular level Large amounts of computing capacity for short periods of time op(x,y,z) takes 10 hours, 500(MB) memory, I/O 100(MB) Measured: FLOPS x(100), y(50), z(25)=> 100x50x25=125,000(145 years) 3 4 Motivation HTC Environment Fort Collins Large amounts of processing capacity? Exploiting computers on the network Science Center Utilizing heterogeneous resources Uses Condor for Overcoming differences of the platforms By building portable solution scientific projects Including resource management framework Over long periods of time? System must be reliable and maintainable Surviving failures (software & hardware) Allowing leaving and joining of resources at any time Upgrading and configuring without significant downtimes Source: http://www.fort.usgs.gov/Condor/ComputingTimes.asp 5 6 1 10/9/2013 HTC Environment HTC Also, the system must meet the needs of: Other considerations: Resource owners The distributive owned resources lead to: Rights respected Decentralized maintenance and configuration of resources Policies enforced Resource availability Customers Benefit of additional processing capacity outweigh complexity of Applications preempted at any time usage Adds an additional degree of resource heterogeneity System administrators Real benefit provided to users outweigh the maintenance cost 7 8 Condor Overview Open-source high-throughput computing framework for computing intensive tasks.
[Show full text]
Facilitating E-Science Discovery Using Scientific Workflows on the Grid
Chapter 13 Facilitating e-Science Discovery Using Scientific Workflows on the Grid Jianwu Wang, Prakashan Korambath, Seonah Kim, Scott Johnson, Kejian Jin, Daniel Crawl, Ilkay Altintas, Shava Smallen, Bill Labate, and Kendall N. Houk Abstract e-Science has been greatly enhanced from the developing capability and usability of cyberinfrastructure. This chapter explains how scientific workflow systems can facilitate e-Science discovery in Grid environments by providing features including scientific process automation, resource consolidation, parallelism, provenance tracking, fault tolerance, and workflow reuse. We first overview the core services to support e-Science discovery. To demonstrate how these services can be seamlessly assembled, an open source scientific workflow system, called Kepler, is integrated into the University of California Grid. This architecture is being applied to a computational enzyme design process, which is a formidable and collaborative problem in computational chemistry that challenges our knowledge of protein chemistry. Our implementation and experiments validate how the Kepler workflow system can make the scientific computation process automated, pipe- lined, efficient, extensible, stable, and easy-to-use. 13.1 Introduction “e-Science is about global collaboration in key areas of science and the next genera- tion of infrastructure that will enable it.”1 Grid computing “coordinates resources that are not subject to centralized control by using standard, open, general-purpose 1 John Taylor, Director General of Research Councils, Office of Science and Technology, UK. J. Wang (*), D. Crawl, I. Altintas, and S. Smallen San Diego Supercomputer Center, UCSD, 9500 Gilman Drive, MC 0505, La Jolla, CA 92093, USA e-mail: [email protected] P. Korambath, K.
[Show full text]
Pos(ACAT08)046
The Advanced Resource Connector for Distributed LHC Computing PoS(ACAT08)046 David Cameron∗ab, Aleksandr Konstantinova, Farid Ould-Saadaa, Katarina Pajchela, Alexander Reada, Bjørn Samseta, Adrian Tagaa † aDepartment of Physics, University of Oslo, P.b. 1048 Blindern, N-0316 Oslo, Norway bNordic Data Grid Facility, Kastruplundgade 22, DK-2770 Kastrup, Denmark E-mail: [email protected] The NorduGrid collaboration and its middleware product, ARC (the Advanced Resource Con- nector), span institutions in Scandinavia and several other countries in Europe and the rest of the world. The innovative nature of the ARC design and flexible, lightweight distribution make it an ideal choice to connect heterogeneous distributed resources for use by HEP and non-HEP appli- cations alike. ARC has been used by scientific projects for many years and through experience it has been hardened and refined to a reliable, efficient software product. In this paper we present the architecture and design of ARC and show how ARC’s simplicity eases application integration and facilitates taking advantage of distributed resources. Example applications are shown along with some results from one particular application, simulation production and analysis for the ATLAS experiment, as an illustration of ARC’s common usage today. These results demonstrate ARC’s ability to handle significant fractions of the computing needs of the LHC experiments today and well into the future. XII Advanced Computing and Analysis Techniques in Physics Research November 3-7, 2008 Erice, Italy ∗Speaker. †This work was partly supported by the Nordic Data Grid Facility and EU KnowARC (contract nr. 032691). c Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence.
[Show full text]