DOCSLIB.ORG

14.SCHULZ Grid.Pdf

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
14.SCHULZ Grid.Pdf Grid Computing ESI 2011 Markus Schulz IT Grid Technology Group, CERN WLCG [email protected] Overview • Grid Computing – Definition, History, Fundamental Problems, Technology – EMI/gLite • WLCG – Challenge – Infrastructure – Usage • Grid Computing – Who? – How? • What’s Next? Markus Schulz 2 Focus • Understanding concepts • Understanding the current usage • Understanding whether you can profit from grid computing • Not much about projects, history, details… • If you want practical examples: • https://edms.cern.ch/file/7223 98/1.4/gLite-3-UserGuide.pdf Markus Schulz 3 What is a Computing Grid? • There are many conflicting definitions – Has been used for several years for marketing… • Marketing moved recently to “Cloud-computing” • Ian Foster and Karl Kesselman – “coordinated resource sharing and problem solving in dynamic, multi- institutional virtual organizations. “ – These are the people who started globus, the first grid middleware project • From the user’s perspective: – I want to be able to use computing resources as I need – I don’t care who owns resources, or where they are – Have to be secure – My programs have to run there • The owners of computing resources (CPU cycles, storage, bandwidth) – My resources can be used by any authorized person (not for free) – Authorization is not tied to my administrative organization • – NO centralized control of resources or users Markus Schulz 4 Other Problems? • The world is a fairly heterogeneous place – Computing services are extremely heterogeneous • Examples: – Batch Systems (controlling the execution of your jobs ) • LSF, PBS, TorQue, Condor, SUN-GridEngine, BQS, ….. • Each comes with its own commands and status messages – Storage: Xroot, CASTOR, dCache, DPM, STORM,+++ – Operating Systems: • Windows, Linux ( 5 popular flavors), Solaris, MacOS,…. • All come in several versions – Site managers • Highly experienced professionals • Scientists forced to do it ( or volunteering ) • Summer students doing it for 3 months……. Markus Schulz 5 What is a Virtual Organization? • A Virtual Organization is a group of people that agree to share resources for solving a common problem – The members often belong to different organizations – The organizations are often in different countries – High Energy Physics Collaborations are a good example Markus Schulz 6 Fundamental Problems • Provide security without central control • Hide and manage heterogeneity • Facilitate communication between users and providers • Not only a technical problem! • Grid Middleware is the software to address these problems Markus Schulz, CERN, IT Department Short History • 1998 The GRID by Ian Foster & Carl Kesselman – Made the idea popular • 1998 Globus-1 first middleware widely available – Proof of concept – www.globus.org evolved to gt-5 ( 2010 ) • Since 1998 several hundred middleware solutions • OpenGridForum works towards standardization – Progress is slow….. • LHC experiments use: – gLite, ARC, OSG (globus, VDT), Alien Markus Schulz 8 Software Approach • Identify an AAA system that all can agree on • Authentication, Authorization, Auditing – That doesn’t require local user registration – That delegates “details” to the users ( Virtual Organizations) • Define and implement abstraction layers for resources – Computing, Storage, etc. • Define and implement a way to announce your resources (Information System) • Build high level services to optimize the usage • Interface your applications to the system Markus Schulz 9 gLite as an example Markus Schulz 10 gLite middleware Job Information & Security Access Data Services Management Monitoring Services Services Services Services Computing Storage Element Information Element System Authorization User Interface Worker Node File and Job Monitoring Replica Catalog Workload Management Authentication API Metadata Accounting Catalog Job Provenance CERN, IT Department The Big Picture 12 Security • Authentication – Who you are • Authorization – What you can do • Auditing/Accounting – What you have done • How to establish trust? Markus Schulz 13 Authentication • Authentication is based on X.509 PKI infrastructure ( Public Key) – Certificate Authorities (CA) issue (long lived) certificates identifying individuals (much like a passport) • Commonly used in web browsers to authenticate to sites – Trust between CAs and sites is established (offline) – In order to reduce vulnerability, on the Grid user identification is done by using (short lived) proxies of their certificates • Short-Lived Credential Services (SLCS) – issue short lived certificates or proxies to its local users • e.g. from Kerberos or from Shibboleth credentials • Proxies can – Be delegated to a service such that it can act on the user’s behalf – Be stored in an external proxy store (MyProxy) – Be renewed (in case they are about to expire) – Include additional attributes - Authorization 14 gLite The EGEE Middleware Distribution Public Key Based Security • How to exchange secret keys? – 340 Sites ( global) • With hundreds of nodes each? – 200 User Communities ( non local) – 10000 Users (global) • And keep them secret!!! Markus Schulz 15 Authorization • VOMS is now a de-facto standard – Attribute Certificates provide users with additional capabilities defined by the VO. – Allows group and role based authorization – Basis for the authorization process • Authorization: currently via mapping to a local user on the resource ( or ACLs) – glexec changes the local identity (based on suexec from Apache) • Designing an authorization service with a common interface agreed with multiple partners – Uniform implementation of authorization in gLite services – Easier interoperability with other infrastructures – ARGUS Security - overview Certification Authority 1 per country/region/la b 17 gLite The EGEE Middleware Distribution Common AuthZ interface Pilot job on CREAM OSG Worker Node EGEE/EGI (both EGEE and pre-WS GT4 gk, GT4 OSG) gridftp, opensshd gatekeeper,g ridftp, gt4-interface (opensshd) edg-gk dCache glexec edg-gridftp Prima + gPlazma: LCAS + LCMAPS SAML-XACML L&L plug-in: SAML-XACML Common SAML XACML library SAML-XACML Query Q: map.user.to.some.pool R: Oblg: user001, somegrp <other obligations> SAML-XACML interface Common SAML XACML library GPBox Site Central: GUMS (+ SAZ) Site Central: LCAS + LCMAPS LCMAPS L&L plug-ins plug-in Trust • You can setup a Certification Authority and several Registration Authorities – Using openssl – 1h work • No one will trust your certificates • Trust is based on: – Common policies – Common infrastructure to follow up on security problems Markus Schulz 19 Security groups • Joint Security Policy Group: TAGPMA EUGridPMA APGridPMA – Joint with WLCG, OSG, and others – Focus on policy issues – Strong input to e-IRG EUGridPMA • The Asia- – Pan-European trust federation of CAs Americas European Pacific Grid PMA Grid PMA Grid PMA – Included in IGTF (and was model for it) – Success: most grid projects now subscribe to the IGTF • Grid Security Vulnerability Group Incident Certification Response Audit – Looking at how to manage vulnerabilities Authorities Requirements – Risk analysis is fundamental – Balance between openness and security Usage Security & Availability • Operational Security Coordination Team VO Rules Policy Security – Main day-to-day operational work – Incident response and follow up – Members in all NGI and sites User Registration Application Development & VO Management & Network Admin Guide – Frequent tests (Security Challenges) Computing Access Computing Elements (CE) • CE CE – gateways to farms LFS Site CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU CPU • Workload Management – WMS/LB – Matches resources and requests Including data location • UI – Handles failures (resubmission) WM S – Manages complex workflows UI – Tracks job status 21 Workload Management (compact) Desktops A few ~50 nodes 1-20 per site 1-24000 per site Job Description Language • [ • Executable = “my_exe”; • StdOutput = “out”; • StdError = “err”; • Arguments = “a b c”; • InputSandbox = {“/home/giaco/my_exe”}; • OutputSandbox = {“out”, “err”}; • Requirements = Member( • other.GlueHostApplicationSoftwareRunTimeEnvironment, • "ALICE3.07.01“ • ); • Rank = -other.GlueCEStateEstimatedResponseTime; • RetryCount = 3 • ] 23 ECSAC'09 - Veli Lošinj, Croatia, 25-29 August 2009 Pilot Jobs • All WLCG experiments use a form of pilot jobs • They have given a number of advantages • Responsive scheduling • Knowledge of environment • They have also faced some resistance • Security • Traceability 17/05/2011 Markus Schulz - ESI 2011 24 Information System • BDII = Yellow Pages – realtime • Light weight Database • LDAP protocol • GLUE 1.3 (2) Schema – Describes resources and their state – Approx 100MB – Update 2min • Several hundred instances 25 Data Management • Storage Elements (SEs) – External interfaces based on SRM 2.2 and gridFTP – Many implementations: • CASTOR, Storm, DPM, dCache, BestMan…. Many local interfaces: – Grid rfio CPU • POSIX, dcap, secure rfio, rfio, xrootd FTP CPU xrootd SRM SE CPU • Catalogue: LFC (local and global) Site CPU • File Transfer Service (FTS) • Data management clients gfal/LCG-Utils 26 Data Management VO User Tools Frameworks lcg_utils Data Management FTS GFAL Cataloging Storage Data transfer Information System/Environment Variables Vendor (Classic gridftp Specific (RLS) LFC SRM SE) RFIO APIs General Storage Element • Storage Resource Manager (SRM) – hides the storage system implementation (disk or tape) – handles authorization – translates SURLs (Storage
Load more

Recommended publications
  • The Translational Journey of the Htcondor-CE
    Journal of Computational Science xxx (xxxx) xxx Contents lists available at ScienceDirect Journal of Computational Science journal homepage: www.elsevier.com/locate/jocs Principles, technologies, and time: The translational journey of the HTCondor-CE Brian Bockelman a,*, Miron Livny a,b, Brian Lin b, Francesco Prelz c a Morgridge Institute for Research, Madison, USA b Department of Computer Sciences, University of Wisconsin-Madison, Madison, USA c INFN Milan, Milan, Italy ARTICLE INFO ABSTRACT Keywords: Mechanisms for remote execution of computational tasks enable a distributed system to effectively utilize all Distributed high throughput computing available resources. This ability is essential to attaining the objectives of high availability, system reliability, and High throughput computing graceful degradation and directly contribute to flexibility, adaptability, and incremental growth. As part of a Translational computing national fabric of Distributed High Throughput Computing (dHTC) services, remote execution is a cornerstone of Distributed computing the Open Science Grid (OSG) Compute Federation. Most of the organizations that harness the computing capacity provided by the OSG also deploy HTCondor pools on resources acquired from the OSG. The HTCondor Compute Entrypoint (CE) facilitates the remote acquisition of resources by all organizations. The HTCondor-CE is the product of a most recent translational cycle that is part of a multidecade translational process. The process is rooted in a partnership, between members of the High Energy Physics community and computer scientists, that evolved over three decades and involved testing and evaluation with active users and production infrastructures. Through several translational cycles that involved researchers from different organizations and continents, principles, ideas, frameworks and technologies were translated into a widely adopted software artifact that isresponsible for provisioning of approximately 9 million core hours per day across 170 endpoints.
    [Show full text]
  • The Advanced Resource Connector User Guide
    NORDUGRID NORDUGRID-MANUAL-6 7/12/2010 The NorduGrid/ARC User Guide Advanced Resource Connector (ARC) usage manual 2 Contents 1 Preface 11 2 Roadmap 13 3 Client Installation 15 3.1 Beginner Installation . 15 3.1.1 Download . 15 3.1.2 Unpack . 15 3.1.3 Configure . 16 3.2 System-wide installation . 16 3.2.1 Installation via repositories . 17 3.2.2 Globus Toolkit . 18 3.2.3 Download the client . 18 3.2.4 Build . 19 3.2.5 Install the client . 20 3.3 Client configuration files . 21 3.3.1 client.conf . 21 3.3.2 srms.conf . 23 3.3.3 Deprecated configuration files . 24 4 Grid Certificates 25 4.1 Quick start with certificates . 25 4.1.1 Certificates in local standalone client . 25 4.1.2 Certificates for system-wide client installation . 27 4.1.3 Obtain a personal certificate . 27 4.2 Grid Authentication And Certificate Authorities . 28 4.2.1 Grid Login, Proxies . 28 4.2.2 Certificate Authorities . 29 4.2.3 Globus Grid certificates . 29 4.2.4 Associating yourself with a proper CA . 30 4.2.5 Friendly CAs . 31 4.2.6 Certificate Request . 32 4.2.7 Working with certificates: examples . 33 3 4 CONTENTS 5 Getting Access to Grid Resources: Virtual Organisations 35 5.1 NorduGrid Virtual Organisation . 35 5.2 Other Virtual Organisations . 36 6 Grid Session 37 6.1 Logging Into The Grid . 37 6.1.1 Proxy Handling Tips . 38 6.2 First Grid test . 38 6.3 Logging Out .
    [Show full text]
  • Facilitating E-Science Discovery Using Scientific Workflows on the Grid
    Chapter 13 Facilitating e-Science Discovery Using Scientific Workflows on the Grid Jianwu Wang, Prakashan Korambath, Seonah Kim, Scott Johnson, Kejian Jin, Daniel Crawl, Ilkay Altintas, Shava Smallen, Bill Labate, and Kendall N. Houk Abstract e-Science has been greatly enhanced from the developing capability and usability of cyberinfrastructure. This chapter explains how scientific workflow systems can facilitate e-Science discovery in Grid environments by providing features including scientific process automation, resource consolidation, parallelism, provenance tracking, fault tolerance, and workflow reuse. We first overview the core services to support e-Science discovery. To demonstrate how these services can be seamlessly assembled, an open source scientific workflow system, called Kepler, is integrated into the University of California Grid. This architecture is being applied to a computational enzyme design process, which is a formidable and collaborative problem in computational chemistry that challenges our knowledge of protein chemistry. Our implementation and experiments validate how the Kepler workflow system can make the scientific computation process automated, pipe- lined, efficient, extensible, stable, and easy-to-use. 13.1 Introduction “e-Science is about global collaboration in key areas of science and the next genera- tion of infrastructure that will enable it.”1 Grid computing “coordinates resources that are not subject to centralized control by using standard, open, general-purpose 1 John Taylor, Director General of Research Councils, Office of Science and Technology, UK. J. Wang (*), D. Crawl, I. Altintas, and S. Smallen San Diego Supercomputer Center, UCSD, 9500 Gilman Drive, MC 0505, La Jolla, CA 92093, USA e-mail: [email protected] P. Korambath, K.
    [Show full text]
  • Glite Middleware
    Enabling Grids for E-sciencE Overview of the EGEE project and the gLite middleware www. eu-egee.org EGEE-III INFSO-RI-222667 Outline Enabling Grids for E-sciencE • What is EGEE? – The project – The infrastructure • gLite middleware • EGEE applications • Sources of further information EGEE-III INFSO-RI-222667 2 Defining the Grid Enabling Grids for E-sciencE • A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user. EGEE-III INFSO-RI-222667 Providing a Production Grid Infrastructure for Collaborative Science 3 The EGEE Project Enabling Grids for E-sciencE • Aim of EGEE: “to establish a seamless European Grid infrastructure for the support of the European Research Area (ERA)” • EGEE – 1 April 2004 – 31 March 2006 – 71 partners in 27 countries, federated in regional Grids • EGEE-II – 1 April 2006 – 30 April 2008 – EddtiExpanded consortium • EGEE-III – 1 May 2008 – 30 April 2010 – Transition to sustainable model EGEE-III INFSO-RI-222667 Providing a Production Grid Infrastructure for Collaborative Science 4 Defining the Grid Enabling Grids for E-sciencE • A Grid is the combination of networked resources and the corresponding middleware, which provides services for the user. EGEE-III INFSO-RI-222667 Providing a Production Grid Infrastructure for Collaborative Science 5 EGEE working with related Enabling Grids for E-sciencE infrastructure projects GIN EGEE-III INFSO-RI-222667 Providing a Production Grid Infrastructure for Collaborative Science 6 What is happening now? Enabling Grids
    [Show full text]
  • Pos(EGICF12-EMITC2)050
    A Science Gateway Getting Ready for Serving the International Molecular Simulation Community PoS(EGICF12-EMITC2)050 Sandra Gesing Center for Bioinformatics & Department of Computer Science, University of Tübingen Sand 14, 72076 Tübingen, Germany E-mail: [email protected] Sonja Herres-Pawlis1 Department of Chemistry, Ludwig-Maximilians-University Munich Butenandtstr. 5-13, 81377 München, Germany E-mail: [email protected] Georg Birkenheuer Paderborn Center for Parallel Computing, University of Paderborn Warburger Str. 100, 33089 Paderborn, Germany E-mail: [email protected] André Brinkmann Johannes Gutenberg-University Mainz 55099 Mainz, Germany E-mail: [email protected] Richard Grunzke Center for Information Services and High Performance Computing, Technische Universität Dresden Zellescher Weg 12-14, Germany E-mail: [email protected] Peter Kacsuk Laboratory of Parallel and Distributed Systems, MTA SZTAKI Kende Street 13-17, 1111 Budapest, Hungary E-mail: [email protected] Oliver Kohlbacher Center for Bioinformatics & Department of Computer Science, University of Tübingen Sand 14, 72076 Tübingen, Germany E-mail: [email protected] 1 Speaker Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it Miklos Kozlovszky Laboratory of Parallel and Distributed Systems, MTA SZTAKI Kende Street 13-17, 1111 Budapest, Hungary E-mail: [email protected] Jens Krüger Center for Bioinformatics & Department
    [Show full text]
  • Workload Management in the EMI Project
    International Conference on Computing in High Energy and Nuclear Physics 2012 (CHEP2012) IOP Publishing Journal of Physics: Conference Series 396 (2012) 032021 doi:10.1088/1742-6596/396/3/032021 Workload management in the EMI project Paolo Andreettoa, Sara Bertoccoa, Fabio Capanninib, Marco Cecchib, Alvise Dorigoa, Eric Frizzieroa, Alessio Gianellea, Aristotelis Kretsise, Massimo Mezzadric, Salvatore Monforted, Francesco Prelzc David Rebattoc, Massimo Sgaravattoa and Luigi Zangrandoa a INFN Padova, Via Marzolo 8, I-35131 Padova, Italy b INFN CNAF, Viale Berti Pichat 6/2, I-40127 Bologna, Italy c INFN Milano, Via Celoria 16, I-20133 Milano, Italy d INFN Catania, Via Santa Sofia, 64, I-95123 Catania, Italy d Research Academic Computer and Technology Institute (RACTI), Patras, Greece Abstract. The EU-funded project EMI, now at its second year, aims at providing a unified, high quality middleware distribution for e-Science communities. Several aspects about workload management over diverse distributed computing environments are being challenged by the EMI roadmap: enabling seamless access to both HTC and HPC computing services, implementing a commonly agreed framework for the execution of parallel computations and supporting interoperability models between Grids and Clouds. Besides, a rigourous requirements collection process, involving the WLCG and various NGIs across Europe, assures that the EMI stack is always committed to serving actual needs. With this background, the gLite Workload Management System (WMS), the meta-scheduler service delivered by EMI, is augmenting its functionality and scheduling models according to the aforementioned project roadmap and the numerous requirements collected over the first project year. This paper is about present and future work of the EMI WMS, reporting on design changes, implementation choices and long- term vision.
    [Show full text]
  • A Review of Medical Grids and Their Direction - a Swiss/Japanese Perspective
    International Journal of Research Studies in Computing 2015 April, Volume 4 Number 1, 15-23 A review of medical grids and their direction - A Swiss/Japanese perspective Nakai, Toshiharu NeuroImaging and Informatics, National Center for Geriatrics & Gerontology, Ohbu, Japan ([email protected] ) ISSN: 2243-772X Müller, Henning Online ISSN: 2243-7797 Business Information Systems, University of Applied Sciences Western Switzerland, Sierre, Switzerland ([email protected] ) OPEN ACCESS Bagarinao, Epifanio Brain and Mind Research Center, Nagoya University, Nagoya, Japan ([email protected] ) Tomida, Koji Nagoya Institute of Technology, Graduate School of Engineering, Nagoya, Japan ([email protected] ) Shiraishi, Yoshiaki Kobe University, Graduate School of Engineering, Kobe, Japan ([email protected] ) Niinimaki, Marko CERN, Helsinki Institute of Physics, Geneva, Switzerland ([email protected] ) Received : 15 February 2015 Revised : 7 March 2015 Accepted : 4 May 2015 Available Online : 15 June 2015 DOI: 10.5861/ijrsc.2015.1109 Abstract This paper presents a general and comparative review of the advances of grid technologies in medical sciences since 2000, with a special emphasis on Europe and Japan. The EU has over the years funded several big projects in the Grid and now the cloud area, covering besides the high energy physics domain also bioinformatics and medical applications as in image analysis. Bioinformatics and pharmaceutical design have been the major targets of grid applications in Japan. Grids and clouds share many of the same goals and techniques, although clouds are more commercially oriented and privately provided services and grids are rather publicly initiated resource sharing platforms between institutions without strong economic objectives.
    [Show full text]
  • ARIS and EGIIS Installation, Configuration and Usage Manual
    NORDUGRID NORDUGRID-MANUAL-21 2/5/2017 ARIS and EGIIS Installation, Configuration and Usage Manual O. Smirnova, M. Ellert, D. Johansson Abstract In order to create a Grid infrastructure using ARC-enabled computing resources, information descrip- tion and aggregation services need to be deployed. ARIS is coupled to a computing resource and collects information about it. EGIIS keeps a list of ARIS instances, and eventually, of other EGIIS instances lower down in hierarchy. Top-level EGIIS instances thus serve as an entry point to the Grid, allowing to discover all the resources. 1 Introduction In a distributed computing world, discovery of necessary resources and services is a key to enabling functional infrastructures. In order to be discovered and consequently used, a Grid resource or service must accomplish these two tasks: 1. advertise itself to potential clients in a standard manner, 2. expose relevant information about itself via a standard interface. In ARC, these functionalities are implemented in the ARIS component (Section 3). Given the potentially very large number of Grid resources and services advertising themselves, a hierarchical structure of organising information published by them is necessary in order to avoid bottlenecks. Such an hierarchy is often implemented by using information aggregating or indexing services, kind of \super-peers". Such an indexing service in ARC is implemented in EGIIS (Section 4). While ARIS is coupled to a resource, EGIIS is an independent service. A typical Grid resource owner always has to deploy ARIS∗. EGIIS servers, on the other hand, are normally deployed by the overall Grid infrastructure operators. A system effectively created by ARIS and EGIIS services is called the ARC Information System.
    [Show full text]
  • EGI Grid Middleware and Distributed Computing Infrastructures Integration
    EGI grid middleware and distributed computing infrastructures integration PoS(ISGC 2013)010 PoS(ISGC 2013)010 Emir Imamagic1 University Computing Centre, University of Zagreb J. Marohnica 5, Zagreb, Croatia E-mail: [email protected] Tiziana Ferrari EGI.eu Science Park 140, Amsterdam, The Netherlands E-mail: [email protected] The European Grid Infrastructure (EGI) is a distributed computing infrastructure (DCI) that provides more than 320,000 logical CPUs and 152 PB of disk space and data. EGI provides a set of basic operational interfaces that enable efficient management of the distributed computing environment and coordination of all stakeholders. This paper describes the tools, interfaces and procedures that allow the integration of resources based on different software stacks into EGI. The integration relies on the operational interfaces of the tools that support daily operations: configuration management, monitoring, the operations dashboard, support and accounting. In parallel the operational interfaces are being extended in order to enable the integration with other resource infrastructures such as PRACE and EUDAT. 1 Speaker Copyright owned by the author(s) under the terms of the Creative Commons Attribution-NonCommercial-ShareAlike Licence. http://pos.sissa.it PoS(ISGC 2013)010 The International Symposium on Grids and Clouds (ISGC) 2013 March 17-22, 2013 PoS(ISGC 2013)010 Academia Sinica, Taipei, Taiwan 1. Introduction The European Grid Infrastructure (EGI) [8] is a publicly funded Distributed Computing Infrastructure (DCI) that provides more than 320,000 logical CPUs and 152 PB of disk space and data. EGI provides a set of basic operational interfaces that enable efficient and sustainable management of the distributed computing environment and coordination of all stakeholders (resource and infrastructure providers, technology providers, users).
    [Show full text]
  • HEP-FCE Working Group on Libraries and Tools
    HEP-FCE Working Group on Libraries and Tools Anders Borgland1, Peter Elmer2, Michael Kirby3, Simon Patton4, Maxim Potekhin5, Brett Viren6, and Brian Yanny7 1SLAC 2Princeton 3FNAL 4LBNL 5BNL 6BNL 7FNAL December 19, 2014 Contents 1 Introduction 3 2 Vision Statement 3 2.1 HEP Software Foundation . .3 2.2 Community . .4 2.3 Support . .4 3 Prioritized Recommendations 5 3.1 Cross-Experiment Effort . .5 3.2 Effort by Experiments . .5 4 Survey of Current Landscape 7 4.1 Forces Counter to Cross-experiment Software . .7 4.2 Best Practices for Experiments . .9 4.3 Areas of Opportunity . 10 5 Event Processing Software Frameworks 11 arXiv:1506.01309v1 [hep-ex] 3 Jun 2015 5.1 Description . 11 5.2 Gaudi . 11 5.3 CMSSW and art . 12 5.4 IceTray . 13 5.5 Opportunity for improvement . 13 6 Software Development 15 6.1 Description . 15 6.2 Follow Free Software . 16 6.3 Category Integration . 16 6.4 Distributed Software Tools . 17 6.5 Automate . 18 6.6 Opportunity for improvement . 18 1 7 Data Management 19 7.1 Definition . 19 7.2 Moving Data . 19 7.3 Metadata, Data Catalogs and Levels of Data Aggregation . 21 7.4 Small and Medium Scale Experiments . 22 7.5 Opportunities For Improvement . 23 8 Workflow and Workload Management 24 8.1 The Challenge of the Three Domains . 24 8.2 Description . 24 8.3 Examples . 27 8.4 Common Features . 28 8.5 Intelligent Networks and Network Intelligence . 30 8.6 Opportunities for improvement . 30 9 Geometry Information Management 32 9.1 Description .
    [Show full text]
  • Glite 3.2 User Guide
    WORLDWIDE LHC COMPUTING GRID GLITE 3.2 USER GUIDE MANUALS SERIES Document identifier: CERN-LCG-GDEIS-722398 EDMS id: 722398 Version: 1.4.2 Date: July 18, 2012 Section: Experiment Integration and Distributed Analysis Document status: RELEASED Author(s): Stephen Burke, Simone Campana, Elisa Lanciotti, Maarten Litmaath, Patricia Mendez´ Lorenzo, Vincenzo Miccio, Christopher Nater, Roberto Santinelli, Andrea Sciaba` Editor: Andrea Sciaba` File: gLite-3-UserGuide Abstract: This guide is an introduction to the WLCG/EGI Grid and to the gLite 3.2 middleware from a user’s point of view. Document Change Record Issue Item Reason for Change 17/07/12 v1.4.2 Minor corrections 31/07/11 v1.4.1 Improved version for gLite 3.2 15/12/10 v1.4 Updated to gLite 3.2 15/01/10 v1.3 Improved version for gLite 3.1 15/01/08 v1.2 First version for gLite 3.1 17/01/07 v1.1 Revised version 20/04/06 v1.0 First draft Files Software Products User files PDF https://edms.cern.ch/file/722398/1.4/gLite-3-UserGuide.pdf CERN-LCG-GDEIS-722398 Manuals Series Page 2 CONTENTS 1 INTRODUCTION . 8 1.1 ACKNOWLEDGMENTS . 8 1.2 OBJECTIVES OF THIS DOCUMENT . 8 1.3 APPLICATION AREA . 8 1.4 DOCUMENT EVOLUTION PROCEDURE . 8 1.5 REFERENCE AND APPLICABLE DOCUMENTS . 9 1.6 TERMINOLOGY . 12 1.6.1 Acronyms. 12 2 EXECUTIVE SUMMARY . 15 3 OVERVIEW . 16 3.1 PRELIMINARY MATTERS. 17 3.1.1 Code Development . 17 3.1.2 Troubleshooting . 17 3.1.3 User and VO utilities .
    [Show full text]
  • Provisioning of Grid Middleware for EGI in the Framework of EGI-Inspire
    Provisioning of Grid Middleware for EGI in the framework of EGI-InSPIRE M. David1, G. Borges1 , J. Gomes1, I. Campos2, A. Lopez2, P. Orviz2 J. L´opez Cacheiro3, C. Fernandez3 and A. Simon3 1 Laborat´orio de Instrumenta¸c~aoe F´ısicaExperimental de Part´ıculasLisboa, Portugal [email protected] 2 Instituto de Fisica de Cantabria, CSIC, Santander, Spain 3 Fundacion Centro de Supercomputacion de Galicia, Santiago de Compostela, Spain Abstract. The European Grid Initiative (EGI) will provide a sustainable pan-European Grid computing infrastructure for e-Science based on a net- work of regional and national Grids. Although EGI will emerge from the structure established by the project Enabling Grids for E-SciencE (EGEE), the organizational and operational aspects will be deeply changed. The transition to the new model will also impact on the middleware develop- ment and deployment issues. Under the new model, Ibergrid is now coordi- nating the middleware deployment processes in EGI. This paper describes the EGI middleware strategy and the middleware rollout workflow being planned by Ibergrid for the whole European infrastructure. 1 Introduction The European Grid Initiative (EGI.eu) [1, 3] aims to setup an organization that will enable the sustainable provision of grid computing services to the research community in Europe. EGI will provide a mechanism whereby national resources will be fully integrated into a pan-European infrastructure. The mechanism and interfaces for resource integration and software provision for users and application support will form the basis of the EGI coordination work. To this end EGI will provide the necessary middleware by partnering with several external software providers.
    [Show full text]