DOCSLIB.ORG

KAAPI: a Thread Scheduling Runtime System for Data Flow Computations

Total Page：16

File Type：pdf, Size：1020Kb

Download full-text PDF Read full-text
KAAPI: a Thread Scheduling Runtime System for Data Flow Computations KAAPI: A thread scheduling runtime system for data flow computations on cluster of multi-processors Thierry Gautier, Xavier Besseron, Laurent Pigeon INRIA, projet MOAIS LIG, Batiment^ ENSIMAG 51 avenue Jean-Kuntzmann F-38330 Montbonnot St Martin, France [email protected], [email protected], [email protected] ABSTRACT among the processors. In the work-stealing scheduling [7], The high availability of multiprocessor clusters for com- each processor gets its own stack implemented by a deque puter science seems to be very attractive to the engineer of frames. When a thread is created, a new frame is ini- because, at a first level, such computers aggregate high per- tialized and pushed at the bottom of the deque. When a formances. Nevertheless, obtaining peak performances on processor become idle, it tries to steal work from a victim irregular applications such as computer algebra problems re- processor with a non-empty deque; then it takes the top- mains a challenging problem. The delay to access memory is most frame (the least recently pushed frame) and places it non uniform and the irregularity of computations requires to in its own deque which is linked to the victim deque. use scheduling algorithms in order to automatically balance Such scheduling has been proven to be efficient for fully- the workload among the processors. strict multithreaded computations [7, 12] while requiring a This paper focuses on the runtime support implementa- bounded memory space with respect to a depth first sequen- tion to exploit with great efficiency the computation re- tial execution [8, 29]. sources of a multiprocessor cluster. The originality of our In Cilk [7], the implementation is highly optimized and approach relies on the implementation of an efficient work- follow the work first principle that aims at moving most of stealing algorithm for a macro data flow computation based the scheduling cost of from the work of the execution to the on minor extension of POSIX thread interface. rare case during work-stealing operations. Lock free algo- rithm manages the access to the deque allowing to reach good efficiency even for fine grain application. Cilk is one Categories and Subject Descriptors of the best known system that allows developers to focus on D.1.3 [Programming Techniques]: Concurrent Program- the algorithm rather than the hardware. Nevertheless, Cilk ming—distributed programming, parallel programming targets are shared memory machines. Although a past ver- sion of Cilk, called Cilk-NOW [6], was designed for network General Terms of workstations, the current highly optimized Cilk cannot runs on multiprocessor cluster. The underlaying DAG con- Performance, Algorithms, Experimentation sistency memory model remains challenging to implement for large scale clusters. Keywords Data flow based languages do not suffer from penalty due Work-stealing, dataflow, multi-core, multi-processor, cluster to memory coherence protocol, mainly because data move- ment between instructions are explicit in the model. Since 1. INTRODUCTION the 70s, data flow models have moved to multi-threaded architecture where key points of original model have been Multithreaded languages have been proposed as a gen- conserved in order to hide latency of (remote) memory ac- eral approach to model dynamic, unstructured parallelism. cess. They include data parallel ones – e.g. NESL [4] –, data flow – The EARTH [19, 28] Threaded-C language exposes two ID [9] –, macro dataflow – Athapascan [15] –, languages with level of threads (standard threads and fiber threads) that fork-join based constructs – Cilk [7] – or with additional syn- allows a fine control over the effective parallelism. Fibers chronization primitives – Jade [34], EARTH [19] –. are executed using a data flow approach: a synchronization Efficient execution of a multithreaded computation on unit detects and schedules fibers that have all their inputs a parallel computer relies on the schedule of the threads produced. In [37], the authors present a thread partitioning and scheduling algorithm to statically gather instructions into threads. Permission to make digital or hard copies of all or part of this work for The paper focuses on the implementation of KAAPI [25], personal or classroom use is granted without fee provided that copies are a runtime support for scheduling irregular macro data flow not made or distributed for profit or commercial advantage and that copies programs on a cluster of multi-processors using a work- bear this notice and the full citation on the first page. To copy otherwise, to stealing algorithm [10, 17]. Section 2 presents the high level republish, to post on servers or to redistribute to lists, requires prior specific and low level programming interfaces. We show how the in- permission and/or a fee. PASCO’07, July 27–28, 2007, London, Ontario, Canada. ternal data flow representation is built at runtime with low Copyright 2007 ACM 978-1-59593-741-4/07/0007 ...$5.00. 15 overhead. Moreover, we explain our lazy evaluation of readi- include <athapascan-1> ness properties of the data flow graph following the work first void sum(shared r int res1, shared r int res2, shared w int res) { } principle of Cilk. This is similar to hybrid data flow archi- res = res1+res2; tecture which allows to group together tasks for a sequential void fibonacci( int n, shared w int res ) execution in order to reduce scheduling overhead. Neverthe- { less, KAAPI permits to reconsider at runtime group of tasks if (n < 2) res = n; in order to extract more parallelism. Section 3 deals with { the runtime system and the scheduling of KAAPI threads else shared int res1; on the processors. The work-stealing algorithm relies on an shared extension of POSIX thread interface that allows work steal- int res2; fork fibonacci(n-1, res1 ); ing. Next, we present the execution of data flow graph as fork an application of the general work-stealing algorithm when fibonacci(n-2, res2 ); fork sum( res1, res2, res ); threads are structured as lists of tasks. This section ends } with the presentation of the thread partitioning to distribute } work among processors prior to the execution. The interac- tion with the communication sub-system to send messages over the network concludes this section. Section 4 reports Figure 1: Fibonacci algorithm with Athapascan in- experiments on multi-core / multi-processor architectures, terface. shared w (resp. shared r) means that the clusters and grids up to 1400 processors. parameter is written (resp. read). 2. THE KAAPI PROGRAMMING MODEL 2.2 Low level interface and internal data flow This section describes the high level instructions used to graph representation express parallel execution as a dynamic data flow graph. Our instructions extend the ones proposed in Cilk [14] in The KAAPI low level interface allows to build the macro order to take into account data dependencies. Following data flow graph between tasks during the execution of Atha- Jade [34], parallel computation is modelled from three con- pascan program. This interface defines several objects. A cepts: tasks, shared objects and access specification. How- closure object represents a function call with a state. An ever, while Jade is restricted to iterative computations, here access object represents the binding of an formal parameter nested recursive parallelism is considered to take benefit in the global memory. The state of a closure can be created, from the work-stealing scheduling performances [8, 15, 17]. running, stolen or terminated and represents the evolution of the closure during the execution. The state of the access 2.1 High level interface object is one of the following: created, ready, destroyed.An access is created when a task is forked. The KAAPI interface is based on the Athapascan inter- Once it is created, a closure should be bound to each ef- face described in [15, 16]. The programming model is based fective parameter of the call. Binding of parameter by value on a global address space called global memory and allows makes copy. Binding of shared object in the global mem- to describe data dependencies between tasks that access ob- ory links together the last access to the object and the new jects in the global memory. The language extends C++ 1 access. The link between accesses is called shared link and with two keywords . The shared keyword is a type qualifier represents the sequence of accesses on a same shared vari- to declare objects in the global memory. The fork keyword able. Figure 2 illustrates the internal data flow graph after creates a new task that may be executed in concurrence one execution of the task’s body fibonacci of the figure 1. with other tasks. Figure 1 sketches the folk algorithm for Let us note that creation of objects closure and access are computing the n-th Fibonacci number in a recursive way. pushed in a stack in order to improve allocation performance A task is a function call: a function that should return no (O(1) allocation time). Moreover, a stack is associated to value except through the global memory and its effective pa- each control flow, so that there is no contention on stack. rameters. The function signature should specify the mode When a closure is being executed, an activation frame is first of access (read, write or access) of the formal parameters in pushed in the stack in order to restore the activation of the the global memory. For instance the function sum of the fig- caller’ stack environment after the execution. There is no ure 1 is specified to have read accesses (res1 and res2)and a priori limit on the stack size: it is managed as a linked to have write access (res). The effective parameters in the list of memory blocks of fixed size, except for some request. global memory are passed by reference and other parameters The system guarantees that an allocation request in a stack are passed by value.
Load more

Recommended publications
  • CSE421 Midterm Solutions —SOLUTION SET— 09 Mar 2012
    CSE421 Midterm Solutions —SOLUTION SET— 09 Mar 2012 This midterm exam consists of three types of questions: 1. 10 multiple choice questions worth 1 point each. These are drawn directly from lecture slides and intended to be easy. 2. 6 short answer questions worth 5 points each. You can answer as many as you want, but we will give you credit for your best four answers for a total of up to 20 points. You should be able to answer the short answer questions in four or five sentences. 3. 2 long answer questions worth 20 points each. Please answer only one long answer question. If you answer both, we will only grade one. Your answer to the long answer should span a page or two. Please answer each question as clearly and succinctly as possible. Feel free to draw pic- tures or diagrams if they help you to do so. No aids of any kind are permitted. The point value assigned to each question is intended to suggest how to allocate your time. So you should work on a 5 point question for roughly 5 minutes. CSE421 Midterm Solutions 09 Mar 2012 Multiple Choice 1. (10 points) Answer all ten of the following questions. Each is worth one point. (a) In the story that GWA (Geoff) began class with on Monday, March 4th, why was the Harvard student concerned about his grade? p He never attended class. He never arrived at class on time. He usually fell asleep in class. He was using drugs. (b) All of the following are inter-process (IPC) communication mechanisms except p shared files.
    [Show full text]
  • Precise Garbage Collection for C
    Precise Garbage Collection for C Jon Rafkind Adam Wick John Regehr Matthew Flatt University of Utah Galois, Inc. University of Utah University of Utah [email protected] [email protected] [email protected] mfl[email protected] Abstract no-ops. The resulting program usually runs about as well as before, Magpie is a source-to-source transformation for C programs that but without the ongoing maintenance burden of manual memory enables precise garbage collection, where precise means that inte- management. In our experience, however, conservative GC works gers are not confused with pointers, and the liveness of a pointer poorly for long-running programs, such as a web server, a program- is apparent at the source level. Precise GC is primarily useful for ming environment, or an operating system kernel. For such pro- long-running programs and programs that interact with untrusted grams, conservative GC can trigger unbounded memory use due to components. In particular, we have successfully deployed precise linked lists [Boehm 2002] that manage threads and continuations; GC in the C implementation of a language run-time system that was this problem is usually due to liveness imprecision [Hirzel et al. originally designed to use conservative GC. We also report on our 2002], rather than type imprecision. Furthermore, the programs experience in transforming parts of the Linux kernel to use precise are susceptible to memory-exhaustion attack from malicious code GC instead of manual memory management. (e.g., user programs or untrusted servlets) that might be otherwise restricted through a sandbox [Wick and Flatt 2004]. Categories and Subject Descriptors D.4.2 [Storage Manage- This paper describes our design of and experience with a GC ment]: Garbage Collection for C that is less conservative.
    [Show full text]
  • Man Pages Section 3 Library Interfaces and Headers
    man pages section 3: Library Interfaces and Headers Part No: 816–5173–16 September 2010 Copyright © 2010, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related software documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT RIGHTS Programs, software, databases, and related documentation and technical data delivered to U.S. Government customers are “commercial computer software” or “commercial technical data” pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, duplication, disclosure, modification, and adaptation shall be subject to the restrictions and license terms setforth in the applicable Government contract, and, to the extent applicable by the terms of the Government contract, the additional rights set forth in FAR 52.227-19, Commercial Computer Software License (December 2007). Oracle America, Inc., 500 Oracle Parkway, Redwood City, CA 94065.
    [Show full text]
  • Part I: Unix Signals
    Part I: Unix Signals 1 Stopping a Program What if you run this program? int main () { while (1); printf("bye\n"); return 0; } What happens if you hit Ctl-C? Could you make Ctl-C print ªbyeº before exiting? 2-4 Signals A shell handles Ctl-C by sending the SIGINT signal to a process The sigaction() function can be used to install a signal handler See bye.c and bye2.c 5 Some Other Signals SIGHUP terminal is gone SIGQUIT please quit SIGKILL force quit (cannot handle) SIGSEGV seg fault SIGALRM timer expired SIGPIPE write to pipe with closed read end SIGCHLD child completed 6 Timers Use setitimer() to start a timer See timer.c and timer2.c... 7 Signal Handlers and Races Beware! Ð a signal handler is practically a thread Use sigprocmask() to (un)block signals See timer3.c 8 Part II: Deadlock · Conditions · Prevention · Detection 9 Deadlock is when two or more threads are waiting for an event that can only be generated by these same threads printer->Wait(); disk->Wait(); disk->Wait(); printer->Wait(); // copy from disk // copy from disk // to printer // to printer printer->Signal(); printer->Signal(); disk->Signal(); disk->Signal(); Deadlock can occur anytime threads acquire multiple resources (printers, disks, etc.), perform work, and then release their resources 10 Deadlock 11 Deadlock 12 Deadlock Examples · Linux: In-kernel memory allocator runs out of pages, causing an ªout of memory handlerº to run, which calls a function that tries to allocate a page. · Windows 2000: The OS keeps a pool of ªworker threadsº waiting to do work; one worker thread submits a job to another worker thread, but there are no free worker-thread slots.
    [Show full text]
  • Automatic Handling of Global Variables for Multi-Threaded MPI Programs
    Automatic Handling of Global Variables for Multi-threaded MPI Programs Gengbin Zheng, Stas Negara, Celso L. Mendes, Laxmikant V. Kale´ Eduardo R. Rodrigues Department of Computer Science Institute of Informatics University of Illinois at Urbana-Champaign Federal University of Rio Grande do Sul Urbana, IL 61801, USA Porto Alegre, Brazil fgzheng,snegara2,cmendes,[email protected] [email protected] Abstract— necessary to privatize global and static variables to ensure Hybrid programming models, such as MPI combined with the thread-safety of an application. threads, are one of the most efficient ways to write parallel ap- In high-performance computing, one way to exploit multi- plications for current machines comprising multi-socket/multi- core nodes and an interconnection network. Global variables in core platforms is to adopt hybrid programming models that legacy MPI applications, however, present a challenge because combine MPI with threads, where different programming they may be accessed by multiple MPI threads simultaneously. models are used to address issues such as multiple threads Thus, transforming legacy MPI applications to be thread-safe per core, decreasing amount of memory per core, load in order to exploit multi-core architectures requires proper imbalance, etc. When porting legacy MPI applications to handling of global variables. In this paper, we present three approaches to eliminate these hybrid models that involve multiple threads, thread- global variables to ensure thread-safety for an MPI program. safety of the applications needs to be addressed again. These approaches include: (a) a compiler-based refactoring Global variables cause no problem with traditional MPI technique, using a Photran-based tool as an example, which implementations, since each process image contains a sep- automates the source-to-source transformation for programs arate instance of the variable.
    [Show full text]
  • Threading and GUI Issues for R
    Threading and GUI Issues for R Luke Tierney School of Statistics University of Minnesota March 5, 2001 Contents 1 Introduction 2 2 Concurrency and Parallelism 2 3 Concurrency and Dynamic State 3 3.1 Options Settings . 3 3.2 User Defined Options . 5 3.3 Devices and Par Settings . 5 3.4 Standard Connections . 6 3.5 The Context Stack . 6 3.5.1 Synchronization . 6 4 GUI Events And Blocking IO 6 4.1 UNIX Issues . 7 4.2 Win32 Issues . 7 4.3 Classic MacOS Issues . 8 4.4 Implementations To Consider . 8 4.5 A Note On Java . 8 4.6 A Strategy for GUI/IO Management . 9 4.7 A Sample Implementation . 9 5 Threads and GUI’s 10 6 Threading Design Space 11 6.1 Parallelism Through HL Threads: The MXM Options . 12 6.2 Light-Weight Threads: The XMX Options . 12 6.3 Multiple OS Threads Running One At A Time: MSS . 14 6.4 Variations on OS Threads . 14 6.5 SMS or MXS: Which To Choose? . 14 7 Light-Weight Thread Implementation 14 1 March 5, 2001 2 8 Other Issues 15 8.1 High-Level GUI Interfaces . 16 8.2 High-Level Thread Interfaces . 16 8.3 High-Level Streams Interfaces . 16 8.4 Completely Random Stuff . 16 1 Introduction This document collects some random thoughts on runtime issues relating to concurrency, threads, GUI’s and the like. Some of this is extracted from recent R-core email threads. I’ve tried to provide lots of references that might be of use.
    [Show full text]
  • Brocade Fabric OS Administration Guide, 8.2.1
    ADMINISTRATION GUIDE Brocade Fabric OS Administration Guide, 8.2.1 Supporting Fabric OS 8.2.1 FOS-821-AG101 28 September 2018 Copyright © 2018 Brocade Communications Systems LLC. All Rights Reserved. Brocade and the stylized B logo are among the trademarks of Brocade Communications Systems LLC. Broadcom, the pulse logo, and Connecting everything are among the trademarks of Broadcom. The term "Broadcom" refers to Broadcom Inc. and/or its subsidiaries. Brocade, a Broadcom Inc. Company, reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design. Information furnished by Brocade is believed to be accurate and reliable. However, Brocade does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under its patent rights nor the rights of others. The product described by this document may contain open source software covered by the GNU General Public License or other open source license agreements. To find out which open source software is included in Brocade products, view the licensing terms applicable to the open source software, and obtain a copy of the programming source code, please visit https://www.broadcom.com/support/fibre-channel-networking/tools/oscd. Brocade Fabric OS Administration Guide, 8.2.1 2 FOS-821-AG101 Contents Introduction........................................................................................................................................................................................................................18
    [Show full text]
  • ELC PLATFORM SPECIFICATION Version 1.0
    ! ! ! ! !"# $"%&'()*+,$!#-'-#%&-(. /012345+678 "##$%&'()*'+',-'$).//. &90+!:;0<<0<+"35=>+#45241?3=: ! ! CONTENTS 1 Copyright and Licensing Terms...........................................................................................1 2 Acknowledgements.............................................................................................................1 3 Introduction.................................................................................................................. .......2 3.1 Purpose .......................................................................................................................2 3.2 Relationship to Other Industry Standards.....................................................................3 3.3 How To Use This Specification ...................................................................................4 3.4 Definitions...................................................................................................................4 3.4.1 ELCPS.................................................................................................................4 3.4.2 ELCPS-Compliant Application............................................................................4 3.4.3 ELCPS-Conforming Implementation ...................................................................4 3.4.4 Non-ELCPS-Compliant Application....................................................................5 3.4.5 ELCPS Implementation Conformance .................................................................5 3.4.6
    [Show full text]
  • A Debugging API for User-Level Thread Libraries
    User Level DB: a Debugging API for User-Level Thread Libraries Kevin Pouget∗ , Marc Perache´ y , Patrick Carribaulty and Herve´ Jourdreny ∗[email protected]; yfmarc.perache, patrick.carribault, [email protected] CEA, DAM, DIF F-91297 Arpajon France Abstract—With the advent of the multicore era, parallel THREAD DB, which lets them be controlled by common programming is becoming ubiquitous. Multithreading is a debuggers like GDB [6], DBX [7] or TOTALVIEW [8]. common approach to benefit from these architectures. Hybrid The use of user-level or hybrid thread libraries provides M:N libraries like MultiProcessor Communication (MPC) good performances but most of these thread libraries do or MARCEL reach high performance expressing fine-grain parallelism by mapping M user-level threads onto N kernel- not implement the THREAD DB API. Without this accurate level threads. However, such implementations skew the debug- support, debuggers can only handle LWPs (Light Weight gers’ ability to distinguish one thread from another, because Process) distinction, rather than end-user level threads. only kernel threads can be handled. SUN MICROSYSTEMS’ LWPs are related to the underlying OS. Their events can THREAD DB API is an interface between the debugger and ptrace the thread library allowing the debugger to inquire for thread be monitored through the system call [9]. Also semantics details. In this paper we introduce the USER LEVEL this function offers memories and registers access primitives. DB (ULDB) library, an implementation of the THREAD DB However, user threads only exist within their library. The interface abstracting the common features of user-level thread kernel is not aware that user threads are handled on a LWP libraries.
    [Show full text]
  • Accurate Recovery of Functions in COTS Binaries a Dissertation Presented by Rui Qiao to the Graduate School in Partial Fulfillme
    Accurate Recovery of Functions in COTS Binaries A Dissertation presented by Rui Qiao to The Graduate School in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Computer Science Stony Brook University May 2017 Copyright by Rui Qiao 2017 Stony Brook University The Graduate School Rui Qiao We, the dissertation committe for the above candidate for the Doctor of Philosophy degree, hereby recommend acceptance of this dissertation Dr. R. Sekar - Dissertation Advisor Professor, Computer Science Department Dr. Michalis Polychronakis - Chairperson of Defense Assistant Professor, Computer Science Department Dr. Nick Nikiforakis - Committee Member Assistant Professor, Computer Science Department Dr. Aravind Prakash - External Committee Member Assistant Professor, Computer Science Department Binghamton University This dissertation is accepted by the Graduate School Charles Taber Dean of the Graduate School ii Abstract of the Dissertation Accurate Recovery of Functions in COTS Binaries by Rui Qiao Doctor of Philosophy in Computer Science Stony Brook University 2017 Binary analysis and instrumentation play a central role in COTS software security. They can be used to detect and prevent vulnerabilities, mitigate exploits, enforce security policies, and so on. Many security instrumentations work at the granularity of functions. How- ever, unlike high-level languages, functions in binaries are not clearly demar- cated. To complicate matters further, functions in binaries may have multiple entry points and/or exit points. Some of these entries or exits may not be determined simply by instruction syntax or code patterns. Moreover, many functions are reachable only through indirect control transfers, while some may be altogether unreachable. In this dissertation, we present an approach that overcomes these challenges to accurately identify function boundaries, as well as calls and returns.
    [Show full text]
  • Download Cert
    man pages section 3: Library Interfaces and Headers Part No: E29033 October 2012 Copyright © 1993, 2012, Oracle and/or its affiliates. All rights reserved. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, the following notice is applicable: U.S. GOVERNMENT END USERS. Oracle programs, including any operating system, integrated software, any programs installed on the hardware, and/or documentation, delivered to U.S. Government end users are "commercial computer software" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, use, duplication, disclosure, modification, and adaptation of the programs, including anyoperating system, integrated software, any programs installed on the hardware, and/or documentation, shall be subject to license terms and license restrictions applicable to the programs. No other rights are granted to the U.S. Government. This software or hardware is developed for general use in a variety of information management applications.
    [Show full text]
  • Oracle® Linux Ksplice User's Guide
    Oracle® Linux Ksplice User's Guide E39380-53 September 2021 Oracle Legal Notices Copyright © 2013, 2021, Oracle and/or its affiliates. This software and related documentation are provided under a license agreement containing restrictions on use and disclosure and are protected by intellectual property laws. Except as expressly permitted in your license agreement or allowed by law, you may not use, copy, reproduce, translate, broadcast, modify, license, transmit, distribute, exhibit, perform, publish, or display any part, in any form, or by any means. Reverse engineering, disassembly, or decompilation of this software, unless required by law for interoperability, is prohibited. The information contained herein is subject to change without notice and is not warranted to be error-free. If you find any errors, please report them to us in writing. If this is software or related documentation that is delivered to the U.S. Government or anyone licensing it on behalf of the U.S. Government, then the following notice is applicable: U.S. GOVERNMENT END USERS: Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs) and Oracle computer documentation or other Oracle data delivered to or accessed by U.S. Government end users are "commercial computer software" or "commercial computer software documentation" pursuant to the applicable Federal Acquisition Regulation and agency-specific supplemental regulations. As such, the use, reproduction, duplication, release, display, disclosure, modification, preparation of derivative works, and/or adaptation of i) Oracle programs (including any operating system, integrated software, any programs embedded, installed or activated on delivered hardware, and modifications of such programs), ii) Oracle computer documentation and/or iii) other Oracle data, is subject to the rights and limitations specified in the license contained in the applicable contract.
    [Show full text]