DOCSLIB.ORG

Grid Computing for Developers Limited Warranty and Disclaimer of Liability

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Grid Computing for Developers Limited Warranty and Disclaimer of Liability GRID COMPUTING FOR DEVELOPERS LIMITED WARRANTY AND DISCLAIMER OF LIABILITY THE CD-ROM THAT ACCOMPANIES THE BOOK MAY BE USED ON A SINGLE PC ONLY. THE LICENSE DOES NOT PERMIT THE USE ON A NETWORK (OF ANY KIND). YOU FURTHER AGREE THAT THIS LICENSE GRANTS PERMISSION TO USE THE PRODUCTS CONTAINED HEREIN, BUT DOES NOT GIVE YOU RIGHT OF OWNERSHIP TO ANY OF THE CONTENT OR PRODUCT CONTAINED ON THIS CD-ROM. USE OF THIRD-PARTY SOFTWARE CONTAINED ON THIS CD-ROM IS LIMITED TO AND SUBJECT TO LICENSING TERMS FOR THE RESPECTIVE PRODUCTS. CHARLES RIVER MEDIA, INC. (“CRM”) AND/OR ANYONE WHO HAS BEEN INVOLVED IN THE WRITING, CREATION, OR PRODUCTION OF THE ACCOMPA- NYING CODE (“THE SOFTWARE”) OR THE THIRD-PARTY PRODUCTS CON- TAINED ON THE CD-ROM OR TEXTUAL MATERIAL IN THE BOOK, CANNOT AND DO NOT WARRANT THE PERFORMANCE OR RESULTS THAT MAY BE OBTAINED BY USING THE SOFTWARE OR CONTENTS OF THE BOOK. THE AUTHOR AND PUBLISHER HAVE USED THEIR BEST EFFORTS TO ENSURE THE ACCURACY AND FUNCTIONALITY OF THE TEXTUAL MATERIAL AND PROGRAMS CONTAINED HEREIN. WE HOWEVER, MAKE NO WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, REGARDING THE PERFORMANCE OF THESE PROGRAMS OR CON- TENTS. THE SOFTWARE IS SOLD “AS IS” WITHOUT WARRANTY (EXCEPT FOR DEFECTIVE MATERIALS USED IN MANUFACTURING THE DISK OR DUE TO FAULTY WORKMANSHIP). THE AUTHOR, THE PUBLISHER, DEVELOPERS OF THIRD-PARTY SOFTWARE, AND ANYONE INVOLVED IN THE PRODUCTION AND MANUFACTURING OF THIS WORK SHALL NOT BE LIABLE FOR DAMAGES OF ANY KIND ARISING OUT OF THE USE OF (OR THE INABILITY TO USE) THE PROGRAMS, SOURCE CODE, OR TEXTUAL MATERIAL CONTAINED IN THIS PUBLICATION. THIS INCLUDES, BUT IS NOT LIMITED TO, LOSS OF REVENUE OR PROFIT, OR OTHER INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OF THE PRODUCT. THE SOLE REMEDY IN THE EVENT OF A CLAIM OF ANY KIND IS EXPRESSLY LIMITED TO REPLACEMENT OF THE BOOK AND/OR CD-ROM, AND ONLY AT THE DISCRETION OF CRM. THE USE OF “IMPLIED WARRANTY” AND CERTAIN “EXCLUSIONS” VARIES FROM STATE TO STATE, AND MAY NOT APPLY TO THE PURCHASER OF THIS PRODUCT. GRID COMPUTING FOR DEVELOPERS VLADIMIR SILVA CHARLES RIVER MEDIA, INC. Hingham, Massachusetts Copyright 2006 by THOMSON DELMAR LEARNING. Published by CHARLES RIVER MEDIA, INC. All rights reserved. No part of this publication may be reproduced in any way, stored in a retrieval system of any type, or transmitted by any means or media, electronic or mechanical, including, but not limited to, photocopy, recording, or scanning, without prior permission in writing from the publisher. Cover Design: Tyler Creative CHARLES RIVER MEDIA, INC. 10 Downer Avenue Hingham, Massachusetts 02043 781-740-0400 781-740-8816 (FAX) [email protected] www.charlesriver.com This book is printed on acid-free paper. Vladimir Silva. Grid Computing for Developers. ISBN: 1-58450-424-2 eISBN: 1-58450-651-2 All brand names and product names mentioned in this book are trademarks or service marks of their respective companies. Any omission or misuse (of any kind) of service marks or trade- marks should not be regarded as intent to infringe on the property of others. The publisher recognizes and respects all marks used by companies, manufacturers, and developers as a means to distinguish their products. Library of Congress Cataloging-in-Publication Data Silva, Vladimir, 1969- Grid computing for developers / Vladimir Silva. p. cm. Includes index. ISBN 1-58450-424-2 (alk. paper) 1. Computational grids (Computer systems) 2. Computer software--Development. I. Title. QA76.9.C58S56 2005 004'.36--dc22 2005026627 Printed in the United States of America 05 7 6 5 4 3 2 First Edition CHARLES RIVER MEDIA titles are available for site license or bulk purchase by institutions, user groups, corporations, etc. For additional information, please contact the Special Sales Depart- ment at 781-740-0400. Requests for replacement of a defective CD-ROM must be accompanied by the original disc, your mailing address, telephone number, date of purchase and purchase price. Please state the nature of the problem, and send the information to CHARLES RIVER MEDIA, INC., 10 Downer Avenue, Hingham, Massachusetts 02043. CRM’s sole obligation to the purchaser is to replace the disc, based on defective materials or faulty workmanship, but not on the operation or functionality of the product. To my dear parents Manuel and Annissia who raised me with love and values; and to my brothers Natasha, Alfredo, Ivan, and Sonia from whom I drew inspiration for this project. This page intentionally left blank Contents Dedication v Preface xxi Part I Theory and Foundations 1 1 The Roadmap to High-Performance Computing 3 1.1 Evolution of Grid Technologies 3 1.2 The Grid in a Nutshell 5 1.2.1 What Can Be Called a Grid Application 6 1.3 Distributed Computing Models 6 1.3.1 Internet Computing 6 1.3.2 Peer-to-Peer (P2P) 7 1.3.3 Grid Architectures 8 1.4 Computing on Demand (COD) 11 1.4.1 How It Works 11 1.4.2 User Authorization 12 1.4.3 Limitation 12 1.5 Grids in Science and Technology 12 1.5.1 Grid Consortiums and Open Forums 12 1.5.2 Grids in Science and Engineering 13 1.6 Summary 15 2 Enterprise Computing 17 2.1 Existing Enterprise Infrastructures 18 vii viii Contents 2.2 Integration with Grid Architectures 19 2.2.1 Workload and Policy Management 20 2.2.2 Shared Services 21 2.2.3 Stateless Business Components 21 2.2.4 Client State Management 21 2.2.5 Pooled Resources and Lightweight Server Processes 22 2.2.6 Database Concurrency Maximization 22 2.3 Grid-Enabled Resource Management Models 23 2.3.1 Hierarchical 24 2.3.2 Abstract Owner Model (AO) 24 2.3.3 Economy/Market Model 25 2.4 Enter Open Grid Services 25 2.4.1 Grids on the Enterprise 26 2.4.2 Challenges Facing Grid Computing on the Enterprise 28 2.4.3 The Jump to Web Services Resource Framework (WSRF) 29 2.5 The Competitive IT Organization 29 2.6 Computational Economy 30 2.6.1 Why Computational Economy? 30 2.6.2 Economy-Driven Grid Resource Management 32 2.6.3 Resource Management Models 33 2.6.4 Grid Architecture for Computational Economy (GRACE) 34 2.7 Summary 35 3 Core Grid Middleware 39 3.1 P2P 41 3.1.1 Definition 41 3.1.2 Advantages and Disadvantages 41 3.1.3 P2P vs. the Client-Server Model 42 3.1.4 P2P Architectures 43 3.2 Globus 45 3.3 Grid Computing and Business Technologies (GRIDBUS) 46 Contents ix 3.4 Summary 46 Part II Grid Middleware 49 4 Grid Portal Development 51 4.1 Resource Management 52 4.1.1 Apache Jetspeed 53 4.1.2 A Resource Management JSP Portlet for Jetspeed 53 4.1.3 WebSphere® Portal Server v5.0 (WPS) 89 4.1.4 IBM Portlet API 89 4.1.5 Job Submission Portlet for WPS 91 4.1.6 Portlet Program 91 4.1.7 Portlet View JSP 95 4.1.8 Testing 98 4.2 Data Management Portlets 99 4.2.1 Global Access to Secondary Storage (GASS) 99 4.2.2 GridFTP 100 4.2.3 Remote I/O 100 4.2.4 Globus Executable Management (GEM) 100 4.2.5 Remote Data Access Portlet for Jetspeed 100 4.2.6 Remote Data Access Portlet for IBM WPS 107 4.3 Troubleshooting 115 4.3.1 Certificate Not Yet Valid Errors 115 4.3.2 Defective Credential Errors 115 4.4 Summary 116 5 Schedulers 119 5.1 An Overview of Popular Schedulers and Grid Middleware Integration 120 5.1.1 Job Schedulers versus Job Managers 121 5.2 Portable Batch System—OpenPBS 121 5.2.1 Overview 121 5.2.2 OpenPBS in Red Hat Linux (RHL) 9 122 x Contents 5.2.3 Configuring PBS on a Single Machine Cluster 123 5.2.4 Tweaking PBS GUIs for RHL 9 124 5.2.5 Configuring the Globus Toolkit 3.2 PBS Job Scheduler 125 5.2.6 Customizing the PBS Job Manager 126 5.2.7 Troubleshooting PBS MMJFS Runtime Errors 127 5.2.8 Customizing PBS Scheduler Execution Parameters with RSL 131 5.3 Silver/Maui Metascheduler 134 5.3.1 Reservations 135 5.3.2 Fairshare 135 5.3.3 Backfill 135 5.3.4 Job Prioritization 136 5.3.5 Throttling Policies 136 5.3.6 Quality of Service (QoS) 136 5.3.7 Node Allocation 136 5.4 Sun Grid Engine (SGE)—N1 Grid Engine 6 137 5.4.1 Workload Management 137 5.4.2 Architecture 137 5.4.3 Step-by-Step SGE Installation Transcript 138 5.4.4 Installation Troubleshooting Tips 140 5.4.5 Installation Verification 142 5.5 Condor-G 143 5.5.1 Condor Features 143 5.5.2 Condor-MMJFS Installation Transcript 144 5.6 MMJFS Integration with Other Job Schedulers 147 5.6.1 Step 1: Create a Job Manager Script 148 5.6.2 Step 2: Update the GT 3.2 Deployment Descriptors 161 5.6.3 Step 3: Verification 165 5.6.4 Integration Troubleshooting 170 5.6.5 Packaging Advice 176 5.7 Factors for Choosing the Right Scheduler 178 5.7.1 Features 178 Contents xi 5.7.2 Installation, Configuration, and Usability 178 5.7.3 User Interfaces 178 5.7.4 Support for Open Standards 178 5.7.5 Interoperability 179 5.7.6 Organization’s Requirements 179 5.7.7 Support for Grid Middleware 179 5.8 Summary 180 6 Open Grid Services Architecture (OGSA) 181 6.1 Web Services versus Grid Services 182 6.2 OGSA Service Model 184 6.3 Standard Interfaces 184 6.4 Other Service Models 185 6.5 Service Factories 186 6.6 Lifetime Management 187 6.7 Handles and References 187 6.8 Service Information and Discovery 187 6.9 Notifications 188 6.10 Network Protocol Bindings 188 6.11 Higher-Level services 189 6.12 Other Interfaces 189 6.13 Hosting Environments 189 6.14 Virtual Organizations (VOs) 190 6.15 The Globus Toolkit 191 6.15.1 Core Services 192 6.15.2 Security 192 6.15.3 Data Management 192 6.16 Sample Grid Services: Large Integer Factorization 193 6.16.1 Overview: Large Integer Factorization and Public Key Cryptosystems 193 6.16.2 Service Implementation 196 6.16.3 Factorization with the Quadratic Sieve 196 xiiContents 6.16.4 Service Architecture 197 6.16.5 Step 1: Obtaining the Factorization code 197
Load more

Recommended publications
  • Grid Computing: What Is It, and Why Do I Care?*
    Grid Computing: What Is It, and Why Do I Care?* Ken MacInnis <[email protected]> * Or, “Mi caja es su caja!” (c) Ken MacInnis 2004 1 Outline Introduction and Motivation Examples Architecture, Components, Tools Lessons Learned and The Future Questions? (c) Ken MacInnis 2004 2 What is “grid computing”? Many different definitions: Utility computing Cycles for sale Distributed computing distributed.net RC5, SETI@Home High-performance resource sharing Clusters, storage, visualization, networking “We will probably see the spread of ‘computer utilities’, which, like present electric and telephone utilities, will service individual homes and offices across the country.” Len Kleinrock (1969) The word “grid” doesn’t equal Grid Computing: Sun Grid Engine is a mere scheduler! (c) Ken MacInnis 2004 3 Better definitions: Common protocols allowing large problems to be solved in a distributed multi-resource multi-user environment. “A computational grid is a hardware and software infrastructure that provides dependable, consistent, pervasive, and inexpensive access to high-end computational capabilities.” Kesselman & Foster (1998) “…coordinated resource sharing and problem solving in dynamic, multi- institutional virtual organizations.” Kesselman, Foster, Tuecke (2000) (c) Ken MacInnis 2004 4 New Challenges for Computing Grid computing evolved out of a need to share resources Flexible, ever-changing “virtual organizations” High-energy physics, astronomy, more Differing site policies with common needs Disparate computing needs
    [Show full text]
  • GT 4.0 RLS GT 4.0 RLS Table of Contents
    GT 4.0 RLS GT 4.0 RLS Table of Contents 1. Key Concepts .................................................................................................................................. 1 1. Overview of Data Management in GT4 ........................................................................................ 1 2. Data movement ........................................................................................................................ 1 3. Data replication ....................................................................................................................... 2 4. Higher level data services .......................................................................................................... 4 2. 4.0.0 Release Note ............................................................................................................................ 5 1. Component Overview ............................................................................................................... 5 2. Feature Summary ..................................................................................................................... 5 3. Bug Fixes ............................................................................................................................... 5 4. Known Problems ...................................................................................................................... 6 5. Technology Dependencies .........................................................................................................
    [Show full text]
  • A Globus Primer Or, Everything You Wanted to Know About Globus, but Were Afraid to Ask Describing Globus Toolkit Version 4
    A Globus Primer Or, Everything You Wanted to Know about Globus, but Were Afraid To Ask Describing Globus Toolkit Version 4 An Early and Incomplete Draft Please send comments, criticisms, and suggestions to: [email protected] Preface The Globus Toolkit (GT) has been developed since the late 1990s to support the development of service-oriented distributed computing applications and infrastructures. Core GT components address basic issues relating to security, resource access and management, data movement and management, resource discovery, and so forth. A broader “Globus universe” comprises numerous tools and components that build on core GT4 functionality to provide many useful application- level functions. These tools have been used to develop many Grid systems and applications. Version 4 of the Globus Toolkit, GT4, released in April 2005, represents a significant advance relative to the GT3 implementation of Web services functionality in terms of the range of components provided, functionality, standards conformance, usability, and quality of documentation. This document is intended to provide a first introduction to key features of both GT4 and associated tools, and the ways in which these components can be used to develop Grid infrastructures and applications. Its focus is on the user’s view of the technology and its application, and the practical techniques that should be employed to develop GT4-based applications. We discuss in turn the applications that motivate the development of GT4 and related tools; the four tasks involved in building Grids: design, deployment, application, operations; GT4 structure, including its Web services (WS) and pre-WS components; the Globus universe and its various components; GT4 performance, scalability, and functionality testing; porting to GT4 from earlier versions of GT; who’s using GT4; likely future directions, and a few words of history.
    [Show full text]
  • Parallelism of Machine Learning Algorithms
    10/9/2013 Outline Condor Presented by : Walid Budgaga The Anatomy of the Grid Globus Toolkit CS 655 – Advanced Topics in Distributed Systems Computer Science Department Colorado State University 1 2 Motivation Motivation High Throughput Computing (HTC)? HTC is suitable for scientific research Large amounts of computing capacity over long Example(Parameter sweep): periods of time. Testing parameter combinations to Measured: operations per month or per year High Performance Computing (HPC)? keep temp. at particular level Large amounts of computing capacity for short periods of time op(x,y,z) takes 10 hours, 500(MB) memory, I/O 100(MB) Measured: FLOPS x(100), y(50), z(25)=> 100x50x25=125,000(145 years) 3 4 Motivation HTC Environment Fort Collins Large amounts of processing capacity? Exploiting computers on the network Science Center Utilizing heterogeneous resources Uses Condor for Overcoming differences of the platforms By building portable solution scientific projects Including resource management framework Over long periods of time? System must be reliable and maintainable Surviving failures (software & hardware) Allowing leaving and joining of resources at any time Upgrading and configuring without significant downtimes Source: http://www.fort.usgs.gov/Condor/ComputingTimes.asp 5 6 1 10/9/2013 HTC Environment HTC Also, the system must meet the needs of: Other considerations: Resource owners The distributive owned resources lead to: Rights respected Decentralized maintenance and configuration of resources Policies enforced Resource availability Customers Benefit of additional processing capacity outweigh complexity of Applications preempted at any time usage Adds an additional degree of resource heterogeneity System administrators Real benefit provided to users outweigh the maintenance cost 7 8 Condor Overview Open-source high-throughput computing framework for computing intensive tasks.
    [Show full text]
  • The Advanced Resource Connector User Guide
    NORDUGRID NORDUGRID-MANUAL-6 7/12/2010 The NorduGrid/ARC User Guide Advanced Resource Connector (ARC) usage manual 2 Contents 1 Preface 11 2 Roadmap 13 3 Client Installation 15 3.1 Beginner Installation . 15 3.1.1 Download . 15 3.1.2 Unpack . 15 3.1.3 Configure . 16 3.2 System-wide installation . 16 3.2.1 Installation via repositories . 17 3.2.2 Globus Toolkit . 18 3.2.3 Download the client . 18 3.2.4 Build . 19 3.2.5 Install the client . 20 3.3 Client configuration files . 21 3.3.1 client.conf . 21 3.3.2 srms.conf . 23 3.3.3 Deprecated configuration files . 24 4 Grid Certificates 25 4.1 Quick start with certificates . 25 4.1.1 Certificates in local standalone client . 25 4.1.2 Certificates for system-wide client installation . 27 4.1.3 Obtain a personal certificate . 27 4.2 Grid Authentication And Certificate Authorities . 28 4.2.1 Grid Login, Proxies . 28 4.2.2 Certificate Authorities . 29 4.2.3 Globus Grid certificates . 29 4.2.4 Associating yourself with a proper CA . 30 4.2.5 Friendly CAs . 31 4.2.6 Certificate Request . 32 4.2.7 Working with certificates: examples . 33 3 4 CONTENTS 5 Getting Access to Grid Resources: Virtual Organisations 35 5.1 NorduGrid Virtual Organisation . 35 5.2 Other Virtual Organisations . 36 6 Grid Session 37 6.1 Logging Into The Grid . 37 6.1.1 Proxy Handling Tips . 38 6.2 First Grid test . 38 6.3 Logging Out .
    [Show full text]
  • Contributions to Desktop Grid Computing Gilles Fedak
    Contributions to Desktop Grid Computing Gilles Fedak To cite this version: Gilles Fedak. Contributions to Desktop Grid Computing : From High Throughput Computing to Data-Intensive Sciences on Hybrid Distributed Computing Infrastructures. Calcul parallèle, distribué et partagé [cs.DC]. Ecole Normale Supérieure de Lyon, 2015. tel-01158462v2 HAL Id: tel-01158462 https://hal.inria.fr/tel-01158462v2 Submitted on 7 Dec 2015 HAL is a multi-disciplinary open access L’archive ouverte pluridisciplinaire HAL, est archive for the deposit and dissemination of sci- destinée au dépôt et à la diffusion de documents entific research documents, whether they are pub- scientifiques de niveau recherche, publiés ou non, lished or not. The documents may come from émanant des établissements d’enseignement et de teaching and research institutions in France or recherche français ou étrangers, des laboratoires abroad, or from public or private research centers. publics ou privés. Copyright University of Lyon Habilitation `aDiriger des Recherches Contributions to Desktop Grid Computing From High Throughput Computing to Data-Intensive Sciences on Hybrid Distributed Computing Infrastructures Gilles Fedak 28 Mai 2015 Apr`esavis de : Christine Morin Directeur de Recherche, INRIA Pierre Sens Professeur, Universit´eParis VI Domenico Talia Professeur, University of Calabria Devant la commission d'examen form´eede : Vincent Breton Directeur de Recherche, CNRS Christine Morin Directeur de Recherche, INRIA Manish Parashar Professeur, Rutgers University Christian Perez Directeur de Recherche, INRIA Pierre Sens Professeur, Universit´eParis VI Domenico Talia Professeur, University of Calabria Contents 1 Introduction 9 1.1 Historical Context . .9 1.2 Contributions . 13 1.3 Summary of the Document . 16 2 Evolution of Desktop Grid Computing 19 2.1 A Brief History of Desktop Grid and Volunteer Computing .
    [Show full text]
  • Resume Ivo Janssen
    Ivo J. Janssen 13266 Kerrville Folkway Austin, TX 78729 (512) 750-9455 / [email protected] Objective To be part of a successful professional services team in a pre-sales and post-sales architecture and deployment role. Summary I'm an experienced, enthusiastic and skilled professional services consultant with experience in architecting and supporting large-scale enterprise software, with extensive experience in designing and implementing custom solutions as part of pre- and post sales support activities in diverse industry verticals across five continents. I'm as comfortable in designing and coding in an engineering role as I am in on-site planning, executing and troubleshooting customer deployments in a professional services role, as well as in participating in sales calls and drafting proposals in a sales engineering role. I'm able to adapt to new environments quickly and rapidly become proficient with new systems, tools and skills. I'm a good team player who thinks beyond the problem at hand instead of following the established paths in order to find a better solution that meets both current and future needs. Professional experience • Virtual Bridges , Austin, TX, USA (Nov 2010 – present) Senior Solution Architect o Solution architect for VDI product line. • Initiate Systems / IBM , Austin, TX, USA (Sep 2008 – Nov 2010) Senior Consultant o Senior implementation engineer for Healthcare sector of Master Data Management product line. o Responsibilities include requirements gathering, product installation and configuration, custom coding, connectivity scripting on Windows and Unix, database performance tuning, customer training workshops, pre/post-sales support. o Successfully architected, implemented and brought live custom implementation projects for major hospital systems and labs in the US, including customers with up to 250 million records (1TB database) connected to 20 auxiliary systems.
    [Show full text]
  • Grid Computing in Research and Business Status and Direction
    Unicore Summit, Sophia Antipolis Oct 11 – 12, 2005 Grid Computing In Research and Business Status and Direction Wolfgang Gentzsch, D-Grid and RENCI Unicore Summit Sophia Antipolis, October 11 – 12, 2005 Topics of the Day What is a Grid ? Why should we care about Grid ? Grid Examples: Research, Industry, Community 12 good reasons why grids are ready for research and early adopters in business Grid challenges and opportunities in business The future of Grid Computing Unicore Summit Sophia Antipolis, October 11 – 12, 2005 What is a Grid ? 1001 Definitions Distributed, networked computing & data resources Networking and computing infrastructure for utility computing Distributed platform for sharing scientific experiments and instruments The next generation of enterprise IT architecture The next generation of the Internet and the WWW Computing from the wall socket The Advanced Network …and 994 more… Unicore Summit Sophia Antipolis, October 11 – 12, 2005 What is a Grid ? 1001 Definitions It’s all about distributed, networked, shared resources in (virtual) organizations Unicore Summit Sophia Antipolis, October 11 – 12, 2005 Industry is on a Journey Old World New World Static Dynamic Silo Shared Physical Virtual Courtesy Mark Linesch, GGF Manual Automated Application Service Transitioning from Silo Oriented Architecture to Service Oriented Architecture Unicore Summit Sophia Antipolis, October 11 – 12, 2005 Example: Tele-Science Grid coordinated problem solving on dynamic and heterogeneous resource assemblies DATA ADVANCED ACQUISITION VISUALIZATION ,ANALYSIS QuickTime™ and a decompressor are needed to see this picture. COMPUTATIONAL RESOURCES IMAGING INSTRUMENTS LARGE-SCALE DATABASES Courtesy of Ellisman & Berman /UCSD&NPACI Unicore Summit Sophia Antipolis, October 11 – 12, 2005 Why should we care about Grids ? Three Examples Research Grid Industry Grid Community Grid Unicore Summit Sophia Antipolis, October 11 – 12, 2005 Grids for Research: NEES, Network for Earthquake Engineering Simulation NEESgrid links earthquake researchers across the U.S.
    [Show full text]
  • Reflections on Building a High-Performance Computing Cluster Using Freebsd
    Reflections on Building a High-performance Computing Cluster Using FreeBSD NYCBUG March 2008 Brooks Davis The Aerospace Corporation El Segundo, California, USA http://people.freebsd.org/~brooks/pubs/meetbsd2007/ © Copyright 2007 The Aerospace Corporation Outline ● Fellowship Overview ● Evaluation of Design Issues ● Lessons Learned ● Thoughts on Future Clusters ● Conclusions A Brief History of Fellowship ● Started in 2001 ● Primarily motivated by the needs of the GPS program office ● Intended to be a corporate resource – Diverse set of users Fellowship Hardware ● 352 dual-processor nodes – 64 Quad-core Woodcrest Xeons – 288 Opterons (152 dual-core) – 1-4GB RAM, 80-250GB disk ● 6 core systems – fellowship – shell server – fellowship64 – amd64 shell server – arwen – node netboot server, scheduler, NIS, DNS, Mathematica License Server – elrond – /scratch – moria – NetApp FAS250 /home, /usr/aero Fellowship Circa February, 2007 Fellowship Composition by processor count Fellowship Composition by core count Fellowship Network r01n01 r01n02 fellowship r01n03 ... frodo r02n01 10.5.0.0/16 Aerospace gamgee r02n02 r02n03 Network arwen Cat6509 ... elrond r03n01 r03n01 r03n02 moria r03n02 r03n03 r03n03 ... ... Fellowship Software ● FreeBSD 6.2-RELEASE-p8 ● Sun Grid Engine (SGE) 6.0u11 ● Message Passing Interface (MPI) – MPICH – MPICH2 – OpenMPI ● PVM ● Mathematica ● Matlab Design Issues ● The Issue ● What We Did ● How it worked Operating System ● FreeBSD 4.x initially – Many years of service – Userland threading forced some application design choices
    [Show full text]
  • The Grid: Past, Present, Future
    1 The Grid: past, present, future Fran Berman,1 Geoffrey Fox,2 and Tony Hey3,4 1University of California, San Diego, California, 2Indiana University, Bloomington, Indiana, 3EPSRC, Swindon, United Kingdom, 4University of Southampton, Southampton, United Kingdom 1.1 THE GRID The Grid is the computing and data management infrastructure that will provide the elec- tronic underpinning for a global society in business, government, research, science and entertainment [1–5]. Grids idealized in Figure 1.1 integrate networking, communication, computation and information to provide a virtual platform for computation and data man- agement in the same way that the Internet integrates resources to form a virtual platform for information. The Grid is transforming science, business, health and society. In this book we consider the Grid in depth, describing its immense promise, potential and com- plexity from the perspective of the community of individuals working hard to make the Grid vision a reality. The Grid infrastructure will provide us with the ability to dynamically link together resources as an ensemble to support the execution of large-scale, resource-intensive, and distributed applications. Grid Computing – Making the Global Infrastructure a Reality, edited by F. Berman, G. Fox and T. Hey. 2002 John Wiley & Sons, Ltd. 10 FRAN BERMAN, GEOFFREY FOX, AND TONY HEY Data Advanced acquisition visualization Analysis Computational resources Imaging instruments Large-scale databases Figure 1.1 Managed digital resources as building blocks of the Grid. Large-scale Grids are intrinsically distributed, heterogeneous and dynamic. They pro- mise effectively infinite cycles and storage, as well as access to instruments, visualization devices and so on without regard to geographic location.
    [Show full text]
  • Experimental Study of Remote Job Submission and Execution on LRM Through Grid Computing Mechanisms
    Experimental Study of Remote Job Submission and Execution on LRM through Grid Computing Mechanisms Harshadkumar B. Prajapati Vipul A. Shah Information Technology Department Instrumentation & Control Engineering Dharmsinh Desai University Department Nadiad, INDIA Dharmsinh Desai University e-mail: [email protected], Nadiad, INDIA [email protected] e-mail: [email protected], [email protected] Abstract —Remote job submission and execution is The fundamental unit of work-done in Grid fundamental requirement of distributed computing done computing is successful execution of a job. A job in using Cluster computing. However, Cluster computing Grid is generally a batch-job, which does not interact limits usage within a single organization. Grid computing with user while it is running. Higher-level Grid environment can allow use of resources for remote job services and applications use job submission and execution that are available in other organizations. This execution facilities that are supported by a Grid paper discusses concepts of batch-job execution using LRM and using Grid. The paper discusses two ways of middleware. Therefore, it is very important to preparing test Grid computing environment that we use understand how a job is submitted, executed, and for experimental testing of concepts. This paper presents monitored in Grid computing environment. experimental testing of remote job submission and Researchers and scientists who are working in higher- execution mechanisms through LRM specific way and level Grid services and applications do not pay much Grid computing ways. Moreover, the paper also discusses attention to the underlying Grid infrastructure in their various problems faced while working with Grid published work.
    [Show full text]
  • Introduction to Grid Computing with the Globus Toolkit Version 3 Page 1 Colorado Software Summit: October 26 – 31, 2003 © Copyright 2003, IBM Corporation
    Colorado Software Summit: October 26 – 31, 2003 © Copyright 2003, IBM Corporation Introduction to Grid Programming with the Globus Toolkit Version 3 Michael Brown IBM Linux Integration Center Austin, Texas Michael Brown – Introduction to Grid Computing with the Globus Toolkit Version 3 Page 1 Colorado Software Summit: October 26 – 31, 2003 © Copyright 2003, IBM Corporation Session Introduction Who am I? ➢ [email protected] ➢ Team Leader for Americas Projects ➢ IBM Linux Integration Center, Austin ➢ Author on 2 IBM Redbooks about Globus 3 What will we talk about ? ➢ Recent history of grid computing ➢ Intro to the galaxy of Globus ➢ Run a Globus V3 example (fingers crossed) Michael Brown – Introduction to Grid Computing with the Globus Toolkit Version 3 Page 2 Colorado Software Summit: October 26 – 31, 2003 © Copyright 2003, IBM Corporation Introduction to Grid Computing Don't panic: it's just the new word for distributed computing We want to exploit resources on a wide, loose network ➢ Not a cluster, but heterogeneous systems ➢ CPU, storage, software, devices, all w/quality-of-service We started small in the early days with file copying, printing Later we moved to program-program communication We have to solve the same problems every time ➢ Comms protocol, data format, security, management Why can't we just focus on the high-level tasks? Please? ➢ Leverage existing standards ➢ Follow accepted patterns Michael Brown – Introduction to Grid Computing with the Globus Toolkit Version 3 Page 3 Colorado Software Summit: October 26 – 31,
    [Show full text]