CAF) Unified Parallel C (UPC

PGAS Partitioned Global Address Space Languages

Coarray Fortran (CAF) Unified Parallel C (UPC)

Dr. R. Bader Dr. A. Block

May 2010 Applying PGAS to classical HPC languages

Design target for PGAS extensions: smallest changes required to convert Fortran and C into robust and efficient parallel languages

add only a few new rules to the languages provide mechanisms to allow

explicitly parallel execution: SPMD style programming model data distribution : partitioned memory model synchronization vs. race conditions memory management for dynamic sharable entities

Standardization efforts: Fortran 2008 draft standard (now in DIS stage, publication targeted for August 2010) separately standardized C extension (work in progress; existing document is somewhat informal)

Going from single to multiple Replicate single program a execution contexts fixed number of times CAF - images : set number of replicates at compile time or at execution time asynchronous execution – loose 1 coupling unless program-control- 2 led synchronization occurs 3 Separate set of entities on 4 time each replicate program-controlled exchange of data UPC uses zero-based may necessitate synchronization counting UPC uses the term thread where CAF has images

One-to-one: each image / thread executed by a single physical processor core Many-to-one: some (or all) images / threads are executed by multiple cores each (e.g., socket could support OpenMP multi-threading within an image) One-to-many: fewer cores are available to the program than images / threads scheduling issues useful typically only for algorithms which do not require the bulk of CPU resources on one image Many-to-many Note: startup mechanism and resource assignment method are implementation-dependent

CAF – intrinsic integer functions for orientation program hello implicit none write(*, '(''Hello from image '',i0, '' of '',i0)') & this_image() , num_images() end program between 1 and num_images() UPC non-repeatably unsorted output uses integer expressions for the same purpose if multiple images /threads used

#include #include #include

int main (void) { printf(“Hello from thread %d of %d \n”, \ MYTHREAD , THREADS ); return 0; } between 0 and THREADS - 1

All entities belong to one of two classes: execute on any image global entities not explicitly shown: s1 s2 s3 s4 purely local accesses (fastest ) global memory execute on image where address (e.g. ,128 bit) „right“ x is located impossible local entities x x x x

per-image address physical memory on local (private) entities: only accessible to the image/thread core execu- which „owns“ them this is what we get from conventional ting image 4 language semantics global ( shared ) entities in partitioned global memory: objects declared on and physically assigned to one image/thread the term „shared“: may be accessed by any other one different semantics allows implementation for distributed memory systems than in OpenMP (esp. for CAF)

CAF: UPC: coarray notation with explicit uses shared attribute indication of location (coindex) implicit locality; various blocking symmetry is enforced strategies (asymmetric data must use asymmetry – threads may have derived types – see later) uneven share of data integer, & shared [1] int A[10]; codimension [*] :: a(3) shared int A[10]; integer a(3) [*] Image 1 2 3 4 Thread 0 1 2 3

A(1)[1] A(1)[2] A(1)[3] A(1)[4] A[0] A[1] A[2] A[3] A(2)[1] A(2)[2] A(2)[3] A(2)[4] A[4] A[5] A[6] A[7] A(3)[1] A(3)[2] A(3)[3] A(3)[4] A[8] A[9] more images additional more threads e.g., A[4] moves coindex value to another physical memory

General syntax Some examples: for a one-dimensional array shared [block size ] type \ shared [N] float A[N][N]; var_name [total size ]; complete matrix rows on each thread (≥1 per thread) scalars and multi-dimensional arrays also possible Values for block size shared [*] int \ omitted default value is 1 A[ THREADS ][3]; integer constant (maximum storage sequence matches with value UPC_MAX_BLOCK_SIZE ) rank 1 coarray from previous [*] one block on each slide ( symmetry restored) thread, as large as possible, size static THREADS environment depends on number of threads may be required (compile-time [] or [0] all elements on thread number determination) one thread

© 2010 LRZ PGAS Languages: Coarray Fortran/Unified Parallel C 8 CAF shared data: coindex to image mapping Image 1 2 3 4 Coarrays S[-2] S[-1] S[0] S[1] may be scalars integer s [-2:*] and/or have corank > 1 real z(10,10) [0:3,3:*] Intrinsic functions query lower and upper cobounds lower cobound upper cobound of codimension 1 of last codimension mapping to image index: lcobound (coarray [,dim,kind ]) ucobound (coarray [,dim,kind ]) 3 4 5 z(:,:)[2,4] 0 1 5 9 return an integer array or scalar 10 images available here scalar if optional dim argument (/4,3/) 1 2 6 10 ragged rectangular pattern is present 2 3 7 z(:,:)[3,5] example: 3 4 8 invalid write(*, *) lcobound (z) coshape = programmer‘s responsibility for will produce the output 0 3 valid coindex reference (see later for additional intrinsic support)

Simplest example CAF syntax: collective scatter: each integer :: b(3), a(3)[*] thread/image executes one statement implying data transfer b = a(:) [q] a b 1 q: same value on all images/threads (shared) (private) UPC syntax:

int b[3]; remember enforced symmetry q shared [*] int a[THREADS][3]; q+1 for (i=0; i<3; i++) { n b[i] = a[ q][i]; time } Note: one-sided semantics: initializations of a and q omitted – b = a (from image/thread q) there‘s a catch here … „Pull“

CAF: UPC: local accesses are fastest and trivial implicit locality cross-thread integer :: a(3)[*] accesses easier to write a(:) = (/ … /) ensuring local access: explicit mapping of array index to thread may same as a(:)[this_image()] be required (see next slide) but probably faster supporting intrinsics: coarray ↔ coindexed object explicit coindex: usually a visual indication of communication shared [B] int A[N]; a block of A supporting intrinsics: size_t img, pos; on thread 4 real :: z(10,10)[0:3,3*] img = \ A[4*B] integer :: cindx(2), m1, img 0 10 images upc_threadof (&A[4*B+2]); 1 A[4*B+1] on any thread, returns 4 2 A[4*B+2] 3 4 5 cindx = this_image(z) on image 7, returns (/2,4/) pos = \ … … 0 1 5 9 m1 = this_image(z,1) upc_phaseof (&A[4*B+2]); B-1 A[5*B-1] on any thread, returns 2 1 2 6 10 on image 7, returns 2 2 img = image_index (z,(/2,4/)) 3 7 on all images, returns 7 A[5*B] 3 4 8 img = image_index (z, (/2,5/) phase on any image, returns 0

Typical case conditional may be inefficient loop or iterator execution cyclic distribution may be slow CAF: upc_forall index transformations between integrates affinity with loop local and global construct integer :: a(nlocal)[*] shared int a[N]; do i=1, nlocal upc_forall (i=0; i

integer :: b(3), a(3)[*] Serial semantics

a = c execution sequence b = a(:) [q] Parallel semantics CAF terminology Focus on image pair q, p: relaxed consistency unordered segments of p, q three scenarios explicit synchronization by user c required to prevent races q a Imposed by algorithm : RaU („reference after update“) is correct b p global barrier: enforce ordering of segments – on all images RaU time a = c for (…) a[n][i]= …; sync all upc_barrier; b = a(:)[q] for (…) b[i]=a[q][i];

How are shared entities accessed? relaxed mode program assumes no concurrent accesses from different threads strict mode program ensures that accesses from different threads are separated, and prevents code movement across these implicit barriers relaxed is default; strict may have large performance penalty Options for synchronization mode selection variable level: strict shared int flag = 0; (at declaration ) relaxed shared [*] int c[THREADS][3];

q has same example for c[q][i] = …; while (!flag) {…}; value on a spin lock flag = 1; … = c[q][j]; thread p as Thread q Thread Thread p Thread on thread q code section level: program level

{ // start of block #include #pragma upc strict // or upc_relaxed.h … // block statements } consistency mode on variable declaration overrides // return to default mode code section or program level specification

Synchronizing subsets Critical regions

code / time code / time

only one thread at a time if (this_image() < 3) then executes sync images ( (/ 1, 2 /) ) end if order is unspecified

all images against one: critical if (this_image() == 1) then : ! statements in region : ! send data end critical sync images ( * ) else images 2 etc. sync images ( 1 ) don‘t mind can have a name, but this has no : ! use data stragglers specific semantics end if

© 2010 LRZ PGAS Languages: Coarray Fortran/Unified Parallel C 15 Memory fences and atomic subroutines – user-defined light-weight synchronization atomic entities are exempt from the synchronization rules programmer‘s responsibility for proper handling CAF: spin-lock example UPC: memory fence is defined by logical(ATOMIC_LOGICAL_KIND ),save :: & ready[*] = .false. upc_fence; logical :: val atomic functions: extension me = THIS_IMAGE() segment Pi ends supported by Berkeley UPC if (me == p) then : ! produce Remarks sync memory ! A memory fence: implied by many call ATOMIC_DEFINE (ready[q], .true.) else if (me == q) other synchronization constructs val = .false. atomic operations: do while (.not. val) call ATOMIC_REF (val, ready) guarantee undivided state change, end do but not a particular ordering or sync memory ! B appearance : ! consume light-weight – if hardware supports end if segment Qj starts atomic operations, better- memory fence: prevents reordering performing than the big global of statements (A), enforces memory barrier hammer loads (for coarrays, B)

Coordinate access to shared (=sensitive) data sensitive data represented as “red balls” Use a coarray/shared lock variable modified atomically consistency across images/threads

CAF: UPC: simplest examples single pointer lock variable

use, intrinsic :: iso_fortran_env #include

type( lock_type ) :: lock [*] upc_lock_t *lock ; // local pointer ! default initialized to unlocked // to a shared entity logical :: got_it like critical , but lock = upc_all_lock_alloc (); lock (lock[1]) more flexible collective call : ! play with red balls upc_lock (lock); same result on unlock (lock[1]) : // play with red balls each thread upc_unlock (lock); do lock(lock[ 2], acquired_lock=got_it) for (;;) { if (got_it) exit if (upc_lock_attempt (lock)) break; : ! do other stuff : // do other stuff end do } : ! play with other red balls : // play with red balls unlock(lock[ 2]) upc_unlock(lock); upc_lock_free (lock); lock must be a coarray as many locks as there are images thread-individual lock generation is lock/unlock: no memory fence, only also possible (non-collective) one-way segment ordering lock/unlock imply memory fence

Separate barrier completion point from waiting point this allows threads to perform other computations before they are required to wait

for (…) a[n][i]= …; upc_notify; // do work not // involving a upc_wait ; for (…) b[i]=a[q][i]; completion point waiting point code / time completion of upc_wait implies synchronization collective – all threads must execute sequence CAF: presently does not have this facility in statement form can define using locks

Remember pointer semantics different between C and Fortran , [dimension (: [,:,… ])], pointer :: ptr no pointer arithmetic ptr => target ! ptr is an alias for target type and rank matching

* ptr; pointer arithmetic rank irrelevant pointer-to-pointer ptr = & var ; ! ptr holds address of var pointer -to -void

Pointers and PGAS memory categorization both pointer entity and pointee might be private or shared 4 combinations possible UPC: three of these combinations are realized

CAF: only two of them allowed, and only in a limited manner  aliasing only to local entities

CAF: UPC:

integer, target :: i1[*] int *p1; integer, pointer :: p1 shared int *p2; a coarray cannot have the shared int *shared p3; type :: ctr pointer attribute int *shared pdep; problem: integer, pointer :: p2(:) where does end type int a[N]; pdep point? type(ctr) :: o[*] all other threads shared int b[N]; may not reference integer, target :: i2(3)

entity „o“: typically asymmetric pointer to shared: addressing overhead p1 => i1 ix=o[1]%p2 p3 pdep o[1]%p2 i1[3] i1[4] b[1] p2 = &b[1]; pdep ref./def.

i2 i2 ix p1 p1 a[0] p2 p2 a [0]

(alias+coindexing) vs. address

collective allocation facilities which synchronize all images/threads CAF: UPC:

deferred shape and coshape shared [100] int *id; integer, & id = (shared [100] int *) \ allocatable :: id(:) [:] upc_all_alloc ( \ THREADS,100*sizeof(int)); allocate(id(100) [2:*] ) number bytes of blocks per block symmetric allocation required : layout equivalent to coarray on the same type, type parameters, left (note compile time constants) bounds and cobounds on every arguments of type size_t image result is a pointer to shared (same value on each thread) deallocate(id) deallocation

deallocation: synchronizes upc_barrier; if (MYTHREAD==0) upc_free(id); before carried out is not collective

Assume 4 threads: shared [2] int A[10]; shared int *p1; shared [2] int *p2; int *ploc; Thread 0 1 2 3 if (MYTHREAD == 1) { p1 = &A[0]; p2 = &A[0]; A[0] A[2] A[4] A[6] p1 += 4; p2 += 4; A[1] A[3] A[5] A[7] ploc = (int *) &A[2]; A[8 ] ploc += 1; A[9] }

ploc p2 Block size: is a property of the shared type used p1 can cast between different block sizes pointer arithmetic follows blocking of pointer! Cast to a pointer to local: must point to data with affinity to current thread

Declaration:

type :: shared_stuff real, allocatable :: r(:) [:] : ! other (maybe non-coarray) components end type

component must be allocatable type extension : base type must already have a coarray component Usage / Semantics much like allocatable coarrays entities must be scalar, may not be allocatable or pointers

type(shared_stuff) :: o allocate(o%r(100)[-4:*])) ! synchronizes : ! use o%r deallocate(o%r)

CAF: UPC: allocatable type components two routines, both called by individual threads … type :: ragged integer, allocatable :: a(:) shared void *upc_global_alloc ( NBLOCKS,NBYTES); end type dynamic allocation: type(ragged) :: o[*] could be in shared space per-thread pointer to first element allocate (o%a (10* this_image ())) of multiple distributed blocks

shared void *upc_alloc (NBYTES); must be local can be asymmetric memory allocated in shared remote accesses require space on calling thread ( → single synchronization block with affinity)

sync all pointer to first element of alloca- b(1:size(o[p]%a)) = o[p]%a ted memory require shared pointer to handle … assuming b is large enough data transfers (“directory”)

Important cases: Case 3: 1. coarray dummy arguments copy-in/out will usually occur 2. local coarray entities additional synchronization 3. non-coarray dummy arguments rule needed associated with a coindexed object Case 1: restrictions ensure that no copy- in/out can occur for allocatable entities, synchro- p nization can occur inside subpro- gram, symmetric call is required Case 2: q a SAVE attribute required, no automatic entities r allocatable is allowed a[q] = … synchronization is implied code / time

Separate include file Example:

#include void upc all reduce T( shared void *restrict dst, Two types: shared const void *restrict src, data redistribution (e.g., scatter, upc_op_t op, size_t nelems, gather) size_t blk size, T(*func)( T, T), upc_flag_t flags); computation operations (reduce, prefix, sort) T is one of the following types: Synchronization mode : C/UC – signed/unsigned L/UL – signed/unsigned constants of type upc_flag_t char long

NOSYNC S/US – signed/unsigned F/D/LD – UPC_ IN _ MYSYNC short float/double/long double OUT ALLSYNC I/UI – signed/unsigned int

entry or exit point, synchronize op is one of the following not at all / wrt data on entered operations: UPC_ADD, UPC_MULT, threads / wrt all threads allow UPC_AND, UPC_OR, UPC_XOR, UPC_LOGAND, function to read/write data UPC_LOGOR, UPC_MIN,UPC_MAX, UPC_FUNC, UPC_NONCOMM_FUNC can combine using „|“

Extension to UPC otherwise weak semantics defined in upc_io.h (upc_all_fclose(), upc_all_fsync() ) provide collective I/O functions Shared file pointer local and shared reads and one pointer shared by all writes (individual vs. shared threads file pointer) cannot use in conjunction Individual file pointer with pointers to local buffers read thread-specific sections consistency requirements for of a file arguments write thread-specific sections I/O on shared vs. private data of a file can do both using individual special flag for atomicity and file pointers consistency semantics (writes from multiple threads)

Teams load imbalanced problems (partial synchronization) recursive algorithms Asyncs and Places memory and function shipping support for accelerator devices? Collective calls in CAF maybe even asynchronous? Process topologies in CAF more general abstraction than multiple coindices Global variables and co-pointers in CAF increase programming flexibility Split-phase barrier in CAF Parallel I/O in CAF

Any questions ?