DOCSLIB.ORG

Scheduling in Metacomputing Systems

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
Scheduling in Metacomputing Systems SCHEDULING IN METACOMPUTING SYSTEMS By Heath A. James, B.Sc.(Ma. & Comp' Sc')(Hons) July 1999 A THESIS SUBMITTED FOR THE DEGREE OF Docton oF PHILoSoPHY IN THE DEPARÎMENT OF COMPUTER SCIENCE UNwPRstrY oF AoPl¡'lPP Contents lx Abstract xl Declaration xllt Acknowledgements 1 Introduction 1 1.1 Project Motivation 2 4 7.2 Why MetacomPuting? . ' 5 1.3 Focus and Contributions of this Thesis 7.4 Organisation of this Thesis 6 , Cluster Computing I 2.L Resource Managers 10 2.1.1 Batch SYstems 72 2.L.2 Extended Batch SYstems 16 2.2 Middleware and Metacomputing Systems 19 2.2.L Middleware Tools 22 2.2.2 Globus 26 2.2-3 Harness 28 2.2.4 Infospheres 30 2.2.5 Legion 31 2.2.6 DOCT 33 2.2.7 WebFlow 34 2.3 DISCWorld 34 2.4 Discussion and comparison of Metacomputing systems 37 2.5 Conclusion 39 I 43 3 A Review of Scheduling 46 3.1 Scheduling Models 47 3.1.1 Static Scheduling 48 3.7.2 DYnamic Scheduling 49 3.1.3 Hybrid Static-Dynamic Scheduling 50 3.2 IndePendent Tasks 51 3.3 DePendent Tasks 51 3.3.1 List Schedules ' 53 3.3.2 Clustering Algorithms 53 3.4 System State Information 57 3.5 Historical Review of Scheduling 63 3.6 Discussion 64 3.7 Conclusion 67 4 Scheduling IndePendent Tasks 4.I Introduction 67 68 4.2 Scheduling IndePendent Jobs 70 4.3 Scheduling Algorithms 4.4 Algorithms Test Results 75 4.5 Discussion and Conclusion 81 5 Modeling Scheduling and Placement 85 86 5.1 Terminology . 5.2 Features and Constraints of Model 86 5.2,7 Distributed Scheduling with Partial System State 87 5.2.2 Model InPuts 88 o1 5.2.3 Characteristics oi System Componenis JL 5.2.4 Restricted Placement 96 5.2.5 Heuristic Optimal Schedules 97 5.2.6 Duplication of Services and Data 98 5.2.7 Clustering of Services 98 5.2.8 Non-PreemPtive Scheduling 99 5.3 A Distributed Job Placement Language 99 5.4 Scheduling Model 104 5.4.1 Cost-Minimisation Function 105 II 5.4.2 Schedule Creation and Processor Assignment ' 109 5.5 Discussion 1L2 113 5.6 Conclusion 6 Implementing Scheduling in DISC'World 115 116 6.1 Schedule Execution Models 6.2 DISCWortd Global Naming Strategy 118 r20 6.3 DISCWorld Remote Access Mechanism L24 6.3.1 OPerations on DRAMs ' t29 6.4 DRAM Futures 130 6.5 Execution in DISCWorld ' t32 6.5.1 Schedule Creation 133 6.5.2 Global Execution 148 6.5.3 LocalisedExecution 156 6.6 SummarY 159 7 DISC\Morld Performance AnalYsis 160 7.I ExamPIe Services 165 7.2 A Detailed ExamPIe L70 7.3 Performance Considerations L72 7.4 Conclusion 175 8 Conclusions and Future Work L75 8.1 Conclusions 178 8.2 Future Work A Distributed Geographic Information Systems 181 181 4.1 Introduction 183 A.2 Online Geospatial Data and Services Standards and APIs 184 ^.2.tA.2.2 Implementation of a standardised Geospatial Data Archive ' 185 4.3 Distributed Systems for Decision Support 187 193 4.4 Decision support Applications requiring Distributed GIS 4.4.1 Land Management and Crop Yield Forecasting' ' ' 193 Defence and C3I 193 ^.4.2 llt 4.4.3 Emergency Services 194 4.5 Conclusions 195 B ERIC 20L 8.1 Introduction 20L 8.2 Repository Design 202 203 8.3 Imagery OPerations . 8.4 Eric Architecture 205 8.5 Performance Issues 208 8.6 Discussion and Conclusions 209 Bibliography zlt IV List of Tables 40 1 Metacomputingsystemcharacteristicssummary 4t 2 Middleware tools characteristics summary 58 3 comparison of models found in scheduling literature ' 69 4 Simulation job duration times jobs 76 5 Execution time and assignment latency for independent 106 6 Description and examples of node characteristics L25 7 Formal definitions of DRAM operations ' 8 comparison of service execution overheads for different execution 161 environments services 163 9 Performance of DISCWorld prototype using multiple L64 10 Performance of DISCWorld prototype using single service images 208 11 Single image compressed HDF to JPEG conversion times for 209 12 Times to compute/retrieve cloud cover for Australia ' v List of Figures XU 1 Inspiration and creativitY . 10 2 Cluster example 11 3 An idealised batch queueing system T2 4 An idealised wide-area manager 2L 5 Metacomputing system architecture 36 6 High-level DISCWorld architecture 36 7 DISCWorld daemon architecture . programs 37 8 Level of integration of cluster computing packages with user 45 9 Graphical representation of scheduling precedence 1 0 List scheduling algorithms rely on task execution times and 52 relationships . 54 11 Clustering together nodes of a task graph L2 Complete system state information 55 13 Consequences of partial system state information 56 T4 dploader program structure. 70 72 15 Round-robin scheduling . 16 Round-robin scheduling with clustering 73 nr) t7 Minimai aciaptive sche<iuiing tù 18 Continual adaptive scheduling 74 19 First-come first-serve scheduling 75 20 Simulation execution times for 1-50 and 50-1000 processes 77 2L Job turnaround data from ANU Supercomputer Facility 80 22 Partial system state information 88 23 Relationship between DISCWortd processing request, services and data dependencies 89 24 DJPL XML document template definition (DTD) L02 vl 103 25 Example DJPL scriPt 111 26 Service and data placement pseudo-algorithm 27 SymbolsusedtoexplaintheDlsCWorldRemoteAccessMechanism L22 (DRAM) notation. t22 28 DRAMs are more than Pointers L24 29 DRAM Java base class L25 30 Service DRAM (DRAMS) Java class L25 31 Data DRAM (DRAMD) Java class 726 32 Example contents of a DRAMS 727 33 Example contents of a DRAMD ' ' 728 34 DRAM operations 131 35 Future DRAM (DRAMF) Java class 132 36 Example contents of a DRAMF' ' ' qF, 135 JI Example state of the DISCWorld environment 136 38 Exampleun-annotaúedDlsCWorldprocessingrequest.'. t37 39 Example annotated DISCWorld processing request 138 40 DISCWorld execution, steps i)-iv) ' 139 4t DISCWorld execution, steps v)-viii) 140 42 DISCWorld execution, steps ix)-xii) 747 43 A graphical user interface to DISCWorld 151 44 Selected components of DISCWorld architecture 151 45 DISCWorld Java classes and inner classes 153 46 DISCWorld thread hierarchY 166 47 Pictorial representation of a complex DJPL request 169 48 Complex processing request event timing sequence L97 49 Screen dump from GIAS implementation 198 50 Dedicated decision support system 198 51 Value-adding feeding chain 198 52 Hierarchical relationships between vaiue-adders 198 53 Value-adding to data 199 54 Benefit of caching intermediate and final data 199 55 Special products can be developed for users parts 199 56 Breaking up a complex query into smaller, computable bitmask 204 57 GMS-5 satellite image and corresponding cloud cover VII 58 Eric architecture 206 59 ERIC: Single image query form. 207 207 60 ERIC: Single image results screen. vlu Abstract environment is that of The problem of scheduling in a distributed heterogeneous execution of the programs' The deciding where to place programs? and when to start on jobs consisting of both general problem of scheduling is investigated, with focus program execution within the independent and dependent programs. We consider being able to make context of metacomputing environments, and the benefits of restricted predictions on the performance of programs. using the constraint of produces heuristically placement of programs, we present a scheduling system that good execution schedules in the absence of complete global system state information' naming strategy The scheduling system is reliant on a processor-independent global and a single-assignment restriction for data' Cluster computing is an abstraction that treats a collection of interconnected parallel or distributed computers as a single resource. This abstraction is commonly of queueing used to refer to the scope of resource managels' most often in the context and systems. To express greater complexity in a cluster computing environment, is now the programs that are run on the environment, the tetm metacomputing [74] being widely adopted. This may be defined as a collection of possibly heterogeneous for the computational nodes which have the appearance of a single, virtual computer purposes of resource management and remote execution. with We review current technologies for cluster computing and metacomputing, focus on their resource management and scheduling capabilities. we critically World review these technologies against our Distributed Information Systems Control (DISCWorld). program We develop novel mechanisms that enable and support scheduling and placement in the DISCWorld prototype. we also discuss a mechanism by which a high-level job's internal structure can be represented, and processing requests controlled. we formulate this using extended markup language (xML). lx To enable processing requests, which consist of a directed acyclic graph of programs, we develop a mechanism for their composition and scheduling placement' This rich data pointer and a complimentary futures mechanism are implemented as the basis part of the DISCWoTId remote access mechanism (DRAM). They also form for a model of wide-area computation independent of DISCWorld. simple RMI and \Me have implemented geospatial imagery applications using both thesis' After common gateway interface, and the novel mechanisms deveioped in this scalability analysing and measuring our system, we find significant performance and benefits. x Declaration for the award of any other This work contains no material which has been accepted institution and' to the best of degree or diploma in any university or other tertiary published or written by my knowledge and belief, contains no material previously made in the text' another person, except where due reference has been in the university Library' I give consent to this copy of my thesis, when deposited being available for loan and photocopying' High Although DISCWorld is a joint development of the Distributed k' work reported here Performance Computing Project group and other students, the on scheduling in DISCWoTId is my own work' Heath A.
Load more

Recommended publications
  • A Flexible Middleware for Metacomputing a Flexible
    IJCSNS International Journal of Computer Science and Network Security, VOL.6 No.2B, February 2006 92 A flexible middleware for metacomputing in multimulti----domaindomain networks Franco Frattolillo Research Centre on Software Technology Department of Engineering, University of Sannio Benevento, Italy Summary technologies, but they require further and significant Middlewares are software infrastructures able to harness the advances in mechanisms, techniques, and tools to bring enormous, but often poorly utilized, computing resources together computational resources distributed over the existing on the Internet in order to solve large-scale problems. Internet to efficiently solve a single large-scale problem However, most of such resources, particularly those existing according to the new scenario introduced by grid within “departmental” organizations, cannot often be considered computing [2, 3, 4]. In fact, in such a scenario, cluster actually available to run large-scale applications, in that they cannot be easily exploited by most of the currently used computing still remains a valid and actual support to high middlewares for grid computing. In fact, such resources are performance parallel computing, since clusters of usually represented by computing nodes belonging to non- workstations represent high performance/cost ratio routable, private networks and connected to the Internet through computing platforms and are widely available within publicly addressable IP front-end nodes. This paper presents a “departmental” organizations, such
    [Show full text]
  • Four-Dimensional Model for Describing the Status of Peers in Peer-To-Peer Distributed Systems
    Turkish Journal of Electrical Engineering & Computer Sciences Turk J Elec Eng & Comp Sci (2013) 21: 1646 { 1664 http://journals.tubitak.gov.tr/elektrik/ ⃝c TUB¨ ITAK_ Research Article doi:10.3906/elk-1108-27 Four-dimensional model for describing the status of peers in peer-to-peer distributed systems Seyedeh Leili MIRTAHERI,1;∗ Ehsan Mousavi KHANEGHAH,1 Mohsen SHARIFI,1 Behrouz MINAEI-BIDGOLI,1 Bijan RAAHEMI,2 Mohammad Norouzi ARAB,1 Abbas Saleh ARDESTANI3 1School of Computer Engineering, Iran University of Science and Technology, Tehran, Iran 2School of Information Technology and Engineering, University of Ottawa, Ottawa, Ontario, Canada 3Department of Business Management, Faculty of Management, Islamic Azad University Central Tehran Branch, Tehran, Iran Received: 09.08.2011 • Accepted: 15.03.2012 • Published Online: 02.10.2013 • Printed: 28.10.2013 Abstract: One of the important aspects of decision making and management in distributed systems is collecting accurate information about the available resources of the peers. The previously proposed approaches for collecting such information completely depend on the system's architecture. In the server-oriented architecture, servers assume the main role of collecting comprehensive information from the peers and the system. Next, based on the information about the features of the basic activities and the system, an exact description of the peers' status is produced. Accurate decisions are then made using this description. However, the amount of information gathered in this architecture is too large, and it requires massive processing. On the other hand, updating the information takes time, causing delays and undermining the validity of the information. In addition, due to the limitations imposed by the servers, such architecture is not scalable and dynamic enough.
    [Show full text]
  • From Computational Science to Internetics: Integration of Science with Computer Science
    Syracuse University SURFACE Northeast Parallel Architecture Center College of Engineering and Computer Science 2002 From Computational Science to Internetics: Integration of Science with Computer Science Geoffrey C. Fox Syracuse University, Northeast Parallel Architectures Center Follow this and additional works at: https://surface.syr.edu/npac Part of the Computer Sciences Commons, and the Curriculum and Instruction Commons Recommended Citation Fox, Geoffrey C., "From Computational Science to Internetics: Integration of Science with Computer Science" (2002). Northeast Parallel Architecture Center. 80. https://surface.syr.edu/npac/80 This Article is brought to you for free and open access by the College of Engineering and Computer Science at SURFACE. It has been accepted for inclusion in Northeast Parallel Architecture Center by an authorized administrator of SURFACE. For more information, please contact [email protected]. From Computational Science to Internetics Integration of Science with Computer Science Geoffrey C. Fox Syracuse University Northeast Parallel Architectures Center 111 College Place Syracuse, New York 13244-4100 [email protected] http://www.npac.syr.edu In honor of John Rice at Purdue University Abstract We describe how our world dominated by Science and Scientists has been changed and will be revolutionized by technologies moving with Internet time. Computers have always been well-used tools but in the beginning only the science counted and little credit or significance was attached to any computing activities associated with scientific research. Some 20 years ago, this started to change and the area of computational science gathered support with the NSF Supercomputer centers playing a critical role. However this vision has stalled over the last 5 years with information technology increasing in importance.
    [Show full text]
  • Globus: a Metacomputing Infrastructure Toolkit
    Globus A Metacomputing Infrastructure Toolkit y Ian Foster Carl Kesselman httpwwwglobusorg Abstract Emerging highp erformance applications require the ability to exploit diverse ge ographically distributed resources These applications use highsp eed networks to in tegrate sup ercomputers large databases archival storage devices advanced visualiza tion devices andor scientic instruments to form networked virtual supercomputers or metacomputers While the physical infrastructure to build such systems is b ecoming widespread the heterogeneous and dynamic nature of the metacomputing environment p oses new challenges for developers of system software parallel to ols and applications In this article we introduce Globus a system that we are developing to address these challenges The Globus system is intended to achieve a vertically integrated treatment of application middleware and network A lowlevel toolkit provides basic mechanisms such as communication authentication network information and data access These mechanisms are used to construct various higherlevel metacomputing services such as parallel programming to ols and schedulers Our longterm goal is to build an Adaptive Wide Area Resource Environment AWARE an integrated set of higherlevel services that enable applications to adapt to heterogeneous and dynamically changing meta computing environments Preliminary versions of Globus comp onents were deployed successfully as part of the IWAY networking exp eriment Introduction New classes of highp erformance applications are b eing developed
    [Show full text]
  • Using Metacomputing Tools to Facilitate Large Scale Analyses Of
    Pacific Symposium on Biocomputing 6:360-371 (2001) USING METACOMPUTING TOOLS TO FACILITATE LARGE- SCALE ANALYSES OF BIOLOGICAL DATABASES Allison Waugh1, Glenn A. Williams1, Liping Wei2, and Russ B. Altman1 1Stanford Medical Informatics 251 Campus Drive, MSOB X-215, Stanford, CA 94305-5479 {waugh, gaw, altman} @ smi.stanford.edu 2Exelixis, Inc. 260 Littlefield Ave, South San Francisco, CA 94080 [email protected] ABSTRACT Given the high rate at which biological data are being collected and made public, it is essential that computational tools be developed that are capable of efficiently accessing and analyzing these data. High-performance distributed computing resources can play a key role in enabling large-scale analyses of biological databases. We use a distributed computing environment, LEGION, to enable large-scale computations on the Protein Data Bank (PDB). In particular, we employ the FEATURE program to scan all protein structures in the PDB in search for unrecognized potential cation binding sites. We evaluate the efficiency of LEGION’s parallel execution capabilities and analyze the initial biological implications that result from having a site annotation scan of the entire PDB. We discuss four interesting proteins with unannotated, high-scoring candidate cation binding sites. INTRODUCTION In the last three years the Protein Data Bank (PDB) has increased from 5,811 released atomic coordinate entries to 12,1101. The growth of the PDB is typical of many other biological databases. As these databases increase in size, analyzing their contents in a systematic manner becomes important. Two common analyses are (1) linear scans that examine each entry (such as looking through all DNA sequences for the presence of a new motif), and (2) "all-versus-all" comparisons in which each element is compared with all other elements (such as clustering of structures, as reported in Shindyalov and Bourne2).
    [Show full text]
  • Metacomputing Across Intercontinental Networks S.M
    Future Generation Computer Systems 17 (2001) 911–918 Metacomputing across intercontinental networks S.M. Pickles a,∗, J.M. Brooke a, F.C. Costen a, Edgar Gabriel b, Matthias Müller b, Michael Resch b, S.M. Ord c a Manchester Computing, The University of Manchester, Oxford Road, Manchester M13 9PL, UK b High Performance Computing Center, Stuttgart, Allmandring 30, D-70550 Stuttgart, Germany c NRAL, Jodrell Bank, Macclesfield, Cheshire SK11 9DL, UK Abstract An intercontinental network of supercomputers spanning more than 10 000 miles and running challenging scientific appli- cations was realized at the Supercomputing ’99 (SC99) conference in Portland, OR using PACX-MPI and ATM PVCs. In this paper, we describe how we constructed the heterogeneous cluster of supercomputers, the problems we confronted in terms of multi-architecture and the way several applications handled the specific requirements of a metacomputer. © 2001 Elsevier Science B.V. All rights reserved. Keywords: Metacomputing; High-performance networks; PACX-MPI 1. Overview of the SC99 global network • national high speed networks in each partner coun- try, During SC99 a network connection was set up that • the European high speed research network connected systems in Europe, the US and Japan. For TEN-155, the experiments described in this paper, four super- • a transatlantic ATM connection between the Ger- computers were linked together. man research network (DFN) and the US research network Abilene, • A Hitachi SR8000 at Tsukuba/Japan with 512 pro- • a transpacific ATM research network (TransPAC) cessors (64 nodes). that was connecting Japanese research networks to • A Cray T3E-900/512 at Pittsburgh/USA with 512 STAR-TAP at Chicago.
    [Show full text]
  • G2-P2P: a Fully Decentralised Fault-Tolerant Cycle-Stealing Framework
    G2-P2P: A Fully Decentralised Fault-Tolerant Cycle-Stealing Framework Richard Mason Wayne Kelly Centre for Information Technology Innovation Queensland University of Technology Brisbane, Australia Email: {r.mason, w.kelly}@qut.edu.au Abstract present in this paper is the first that we know of that attempts to support generic cycle stealing over a fully Existing cycle-stealing frameworks are generally decentralised P2P network. based on simple client-server or hierarchical style ar- The benefits of utilizing a fully decentralised P2P chitectures. G2:P2P moves cycle-stealing into the network are obvious – unlimited scalability and no “pure” peer-to-peer (P2P), or fully decentralised single point of failure; however the challenges to arena, removing the bottleneck and single point of achieving this goal are also great. The basic problem failure that centralised systems suffer from. Addition- is that the peers that make up a cycle stealing net- ally, by utilising direct P2P communication, G2:P2P work may vary considerably during the lifetime of a supports a far broader range of applications than the computation running on the network. In a file sharing master-worker style that most cycle-stealing frame- environment, this is not a major problem; it simply works offer. means that searching for a particular song may tem- G2:P2P moves away from the task based program- porarily return no hits. With a long running parallel ming model typical of cycle-stealing systems to a dis- application, however, the consequences of losing even tributed object abstraction which simplifies commu- part of the applications state in most cases will be nication.
    [Show full text]
  • Web Based Metacomputing
    Web based metacomputing Tomasz Haupt, Erol Akarsu, Geoffrey Fox, Wojtek Furmanski Northeast Parallel Architecture Center at Syracuse University Abstract Programming tools that are simultaneously sustainable, highly functional, robust and easy to use have been hard to come by in the HPCC arena. This is partially due to the difficulty in developing sophisticated customized systems for what is a relatively small part of the worldwide computing enterprise. Thus we have developed a new strategy - termed HPcc High Performance Commodity Computing[1] - which builds HPCC programming tools on top of the remarkable new software infrastructure being built for the commercial web and distributed object areas. We add high performance to commodity systems using multi tier architecture with GLOBUS metacomputing toolkit as the backend of a middle tier of commodity web and object servers. We have demonstrated the fully functional prototype of WebFlow during Alliance'98 meeting. Introduction Programming tools that are simultaneously sustainable, highly functional, robust and easy to use have been hard to come by in the HPCC arena. This is partially due to the difficulty in developing sophisticated customized systems for what is a relatively small part of the worldwide computing enterprise. Thus we have developed a new strategy - termed HPcc High Performance Commodity Computing[1] - which builds HPCC programming tools on top of the remarkable new software infrastructure being built for the commercial web and distributed object areas. This leverage of a huge industry investment naturally delivers tools with the desired properties with the one (albeit critical) exception that high performance is not guaranteed. Our approach automatically gives the user access to the full range of commercial capabilities (e.g.
    [Show full text]
  • Decision Support System on the Grid 1 Introduction
    The University of Sheffield Decision Support System on The Grid Published at the 2004 Int’l Conference on Knowledge-Based Intelligent Information & Engineering Systems (KES2004), New Zealand. Decision Support System on The Grid M. Ong, X. Ren, G. Allan, V. Kadirkamanathan, HA Thompson and PJ Fleming Rolls-Royce Supported University Technology Centre in Control and Systems Engineering, University of Sheffield, Department of Automatic Control and Systems Engineering, Mappin Street, Sheffield, S1 3JD, United Kingdom. [email protected] Abstract. Aero engines are extremely reliable machines and operational failures are rare. However, currently great effort is being put into reducing the number of in-flight engine shutdowns, aborted take-offs and flight delays through the use of advanced engine health monitoring technology. This represents a benefit to society through reduced delays, reduced anxiety and reduced cost of ownership of aircraft. This is reflected in a change of emphasis within aero engine companies where, instead of selling engines to customers, there is a fundamental shift to adoption of power-by-the-hour contracts. In these contracts, airlines make fixed regular payments based on the hours flown and the engine manufacturer retains responsibility for maintaining the engine. To support this new approach, improvements in in-flight monitoring of engines are being introduced with the collection of much more detailed data on the operation of the engine. At the same time advances have been made in Internet technologies providing a worldwide network of computers that can be used to access and process that data. The explosion of available knowledge within those large datasets also presents its own problems and here it is necessary to work on advanced decision support systems to identify the useful information in the data and provide knowledge-based advice between individual aircraft, airline repair and overhaul bases, world-wide data warehouses and the engine manufacturer.
    [Show full text]
  • Kenneth A. Hawick PD Coddington HA James
    Student: Vidar Tulinius Email: [email protected] Distributed frameworks and parallel algorithms for processing large-scale geographic data Kenneth A. Hawick P. D. Coddington H. A. James Overview Increasing power PC Workstations Mono-processors Geographic information systems (GIS) provide a resource-hungry application domain. Environmental and defense applications need to deal with really large data sets. Outline Metacomputing and grid computing Metacomputing middleware DISCWorld Transparent parallelism Image processing applications Parallel spatial data interpolation Kriging interpolation Spatial data grids Conclusion Parallel techniques Satellite imagery Increased size Improved resolution More frequency channels Data stored as lists of objects Locations Vectors for roads Objects allocated for processing among processors in a load balanced fashion Metacomputing and grid computing Number of resources distributed over a network. High-performance computers Clusters of workstations Disk arrays Tape silos Data archives Data visualization Scientific instruments Metacomputing and grid computing GIS applications adhere well to grid computing Require access to multiple databases Large data sets Require significant computational resources. Metacomputing and grid computing cont. ‘bleeding-edge’ technology Globus Grid Toolkit Low-level Hard to use Requires substantial technical expertise OLDA developed a prototype computational grids. Distributed data archives Distributed computational resources Metacomputing
    [Show full text]
  • Sorcer: Computing and Metacomputing Intergrid
    SORCER: COMPUTING AND METACOMPUTING INTERGRID Michael Soblewski Texas Tech University, Lubbock, TX, U.S.A. [email protected] Keywords: Metacomputing, metaprogramming, grid computing, service-oriented architectures, service-oriented programming. Abstract: This paper investigates grid computing from the point of view three basic computing platforms. Each platform considered consists of virtual compute resources, a programming environment allowing for the development of grid applications, and a grid operating system to execute user programs and to make solving complex user problems easier. Three platforms are discussed: compute grid, metacompute grid and intergrid. Service protocol-oriented architectures are contrasted with service object-oriented architectures, then the current SORCER metacompute grid based on a service object-oriented architecture and a new metacomputing paradigm is described and analyzed. Finally, we explain how SORCER, with its core services and federated file system, can also be used as a traditional compute grid and an intergrid—a hybrid of compute and metacompute grids. 1 INTRODUCTION leveraging the dynamically federating resources. We can distinguish three generic grid platforms, The term “grid computing” originated in the early which are described below. 1990s as a metaphor for accessing computer power Programmers use abstractions all the time. The as easy as an electric power grid. Today there are source code written in programming language is an many definitions of grid computing with a varying abstraction of the machine language. From machine focus on architectures, resource management, language to object-oriented programming, layers of access, virtualization, provisioning, and sharing abstractions have accumulated like geological strata. between heterogeneous compute domains. Thus, Every generation of programmers uses its era’s diverse compute resources across different programming languages and tools to build programs administrative domains form a grid for the shared of next generation.
    [Show full text]
  • Distributed Frameworks and Parallel Algorithms for Processing Large-Scale Geographic Data
    Parallel Computing 29 (2003) 1297–1333 www.elsevier.com/locate/parco Distributed frameworks and parallel algorithms for processing large-scale geographic data Kenneth A. Hawick a,*, P.D. Coddington b,1, H.A. James a a Computer Science Division, School of Informatics, University of Wales, Bangor, North Wales LL57 1UT, UK b Department of Computer Science, University of Adelaide, SA 5005, Australia Received 13 August 2002; accepted 23 April 2003 Abstract The number of applications that require parallel and high-performance computing tech- niques has diminished in recent years due to to the continuing increase in power of PC, work- station and mono-processor systems. However, Geographic information systems (GIS) still provide a resource-hungry application domain that can make good use of parallel techniques. We describe our work with geographical systems for environmental and defence applications and some of the algorithms and techniques we have deployed to deliver high-performance pro- totype systems that can deal with large data sets. GIS applications are often run operationally as part of decision support systems with both a human interactive component as well as large scale batch or server-based components. Parallel computing technology embedded in a distri- buted system therefore provides an ideal and practical solution for multi-site organisations and especially government agencies who need to extract the best value from bulk geographic data. We describe the distributed computing approaches we have used to integrate bulk data and metadata sources and the grid computing techniques we have used to embed parallel services in an operational infrastructure. We describe some of the parallel techniques we have used: for data assimilation; for image and map data processing; for data cluster analysis; and for data mining.
    [Show full text]