Introduction to Parallel Programming in OpenMP
David Colignon, ULg
CÉCI - Consortium des Équipements de Calcul Intensif http://www.ceci-hpc.be Main References
• “Parallel Programming with GCC”, Diego Novillo, Red Hat Red Hat Summit, Nashville, May 2006 http://www.airs.com/dnovillo/Papers/rhs2006.pdf • "An Overview of OpenMP", Ruud van der Pas, Oracle IWOMP 2010, Tsukuba, 14-16 June 2010 http://www.compunity.org/training/tutorials/3 Overview_OpenMP.pdf and http://openmp.org/wp/2010/07/iwomp-2010-material-available/ More References:
Specification
OpenMP, The OpenMP API specification for parallel programming http://openmp.org/
Articles
Wikipedia (good summary) http://en.wikipedia.org/wiki/Openmp 32 OpenMP traps for C++ developers http://software.intel.com/en-us/articles/32-openmp-traps-for-c-developers/ Common Mistakes in OpenMP and How To Avoid Them http://www.michaelsuess.net/.../suess_leopold_common_mistakes_06.pdf IWOMP 2009, The 2009 International Workshop on OpenMP (Slides) http://openmp.org/wp/2009/06/iwomp2009/ IWOMP 2010, The 2010 International Workshop on OpenMP (Slides) http://openmp.org/wp/2010/07/iwomp-2010-material-available/ Avoiding and Identifying False Sharing Among Threads http://software.intel.com/en-us/articles/avoiding-and-identifying-false-sharing
Tutorials
Parallel Programming with GCC, D. Novillo, Red Hat Summit, Nashville, May 2006 http://www.airs.com/dnovillo/Papers/rhs2006.pdf An Overview of OpenMP, IWOMP 2010, Ruud van der Pas, Oracle http://www.compunity.org/training/tutorials/3 Overview_OpenMP.pdf A "Hands-on" Introduction to OpenMP, SC08, Mattsonand Meadows, Intel http://www.openmp.org/mp-documents/omp-hands-on-SC08.pdf Cours OpenMP (en français !) de l'IDRIS http://www.idris.fr/data/cours/parallel/openmp/ Using OpenMP, SC09, Hartman-Baker R., ORNL, NCCS http://www.greatlakesconsortium.org/events/scaling/files/openmp09.pdf OpenMP Tutorial, Barney B., LLNL https://computing.llnl.gov/tutorials/openMP/
Books
Using OpenMP - Portable Shared Memory Parallel Programming, by Chapman et al. https://mitpress.mit.edu/books/using-openmp (Download Book Examples and Discuss) http://openmp.org/wp/2009/04/download-book-examples-and-discuss/
Parallel Programming in OpenMP, by Rohit Chandra et al. http://www.elsevier.com/wps/find/bookdescription.cws_home/677929/description (Google Preview) http://books.google.be/books?id=18CmnqIhbhUC Outline
• Introduction to parallel computing • Parallel programming models - Shared memory - Message passing • OpenMP - Guided tour - In depth overview +","--.-$/01234567
! !"#$%&'()&'#$*+,*-''#,*.$/+'$0,*'#&"#( " 1#'/+'2&,*# " 1'+34#2$"05# ! 6)+$2&0,$2+(#4" " 7%&'#($2#2+'. " 80"9'03-9#($2#2+'. ! :&9-'#$+/$;'+34#2$(0*9&9#" " <+2;-9&90+,=*+22-,0*&90+,$'&90+ " >&'()&'#$'#?-0'#2#,9"
!"#$%&'$())* % ,-./0-12/34$!3560# =.,)'#(..)#,
>?@ >?@ >?@ >?@ >?@ >?@
A)0('1 A)0('1 A)0('1 A)0('1 A)0('1 A)0('1
! !"#$%&'(#)**('%$"*%+,*%(-.%&'+/",)%0)0('1 ! !2&3+#+,%#(004.+#",+(. ! !2&3+#+,%*1.#$'(.+5",+(. ! 6+77+#43,%,(%&'(8'"0%94,%.(:7)-%$+;;).%*+;)<)77)#,*
!"#$%&'$())* + ,-"./0$!/12.#
!"#$%&
'()"%*$(("*)
+,- +,- +,- +,- +,- +,-
! ,%$*"..$%./.01%"/*$##$(/#"#$%& ! '#234*4)/*$##5(4*1)4$( ! 67234*4)/.&(*0%$(481)4$( ! 94#23"/)$/2%$:%1#/;5)/04<<"(/.4<"=">>"*).
!"#$%&'$())* + +,-.,"//01.$!-2345
! !"#$%&'()*+$),-+%&./%/0$1.02+%3.40/,)3%& " 5%+60$7*.02./-8+)940$%.30&%* " :#$#88%8)*/.#;#)8#,8%.#+.;#$)0-*.8%;%8* ! <&&)+)03#8.$%=-)$%/%3+*.0;%$.*%=-%3+)#8 " >#*7.4$%#+)03 " ?0//-3)4#+)03 " !134"$03)@#+)03 ! A06.&0.6%.B$0C$#/.+"%*%.*1*+%/*D
!"#$%&'$())* * ,-./0102$3"4"//5/067
! !"#$%&'()$'*"+%,-"."/%&'001(2&-)2'(%-(3% "4(&5$'(26-)2'( ! 7(&$#-"#3%8$'9$-002(9%&'08*#:2)4 " ;<)#(%$#=12$#%32<<#$#()%-*9'$2)50" ! >-(4%32<<#$#()%-88$'-&5#" " ?-$-**#*%*-(91-9#"%'$%*-(91-9#%#:)#("2'("+%@?A/% ;&&-0/%B-C- " D'082*#$%-((')-)2'("+%;8#(>? " E2F$-$2#"+%?)5$#-3"/%>?7
!"#$%&'$())* + ,"-"../.012$03$455 !""#$%&&'()$#*'%(#+',+%((-,+.#/'0-1$
4433$$.. 2233((00
##00"" ((&&))**++,,--..//**--++'' $$%%&&''!!"" !"!"##
! 33%%))''//33))55++ ! 33%%))''//33))55++ ! /3/3,,%%3311 ! //33,,%%3311 ! ,,''##%%$$--((##++'',,))((''11 ! ++''//&&51-51-((##''&&))5'5',, ! ++''//&&51-(51-(##005(5(--++))5566--$$ ! $$&&--++55331#1#159159((33((5-5-$$ ! ,,''))##5,5,))((%%$$5656-- ! ,,''))##5,5,))((%%$$5656-- ! $$''//--778833))##55,,))((%$%$5656-- ! 66--((..##55,,))((%%$$5656--
11//22&&33++44335522&&33''++**33''&&))&&2222//,,--667733,,&&66//**&&8833**++3322%%&&&&8855%%2299
!"#$%&'$())* + !+,,"-+$.",,/0-
! !"#$%&'&%()%*+,-,()+-.&/ ! 0").$&1*-%)1"#$*%&,).2$$",'),&32*,&/ ! 45&)6-..&#+%()%-782-8&9)":)$-,-%%&%) $,"8,-##*78 " ;%'*#-'&)1"7',"% " ;%'*#-'&)$-*7)<5&7)'5*78.)8")<,"78 " !"#$2'-'*"7=1"##27*1-'*"7),-'*")#2.')+&)5*85 ! >&..-8&)?-..*78)@7'&,:-1&)A>?@B)#".') $"$2%-,)#"/&%
!"#$%&'$())* %* !,--".,$/"--01.
! !"#$%$&"'$((%"))')#$*") " +&',$-'$.)/'0"'1)"('/2'$')3$%"(',",/%-' ,$*342" ! 5"$6-'7"483&'#%/*"))") ! 9/,,124*$&4/2'":#.4*4&'64$'2"&7/%;' ,"))$8") " <)"%'%")#/2)40."'=/%',$%)3$..428>')"2(428'$2(' %"*"46428
!"#$%&'$())* %+ ,-./!0$1$2/345678395/
! !"#$%"$&'&()&#*+,#*'-,.'*/".&0'1&1,.2' 3,#3%.. ! 4%55,.)*'67'688'"#0'9,.)."# ! :1;&00&0'0+.&3)+<&*'*5&3+-2 " =".">>&>+*1 " ?")"'*/".+#$'*&1"#)+3* " @,.A'*/".+#$'*&1"#)+3* ! 4)"#0".0'"#0'+#3.&"*+#$>2'5,5%>".
!"#$%&'$())* &+ ,-./!0$1$02342"556/4$!37.8
! !"#$%&'(&)'*+,-'.(&#$/"(0.1# " 2"#0$*&03*$"%"5(#&0$"/#&')&13.6%*$(&03*$"%# " 766&03*$"%#"*$&1'//'(&/$/'*8 ! 766'5#&#$9:$(0."6&"(%&4"*"66$6&$;$1:0.'(
2"#0$*& 03*$"%
)'*+ -'.(
<"*"66$6&*$=.'(
!"#$%&'$())* &+ +,-.!/$0$/1231"445.3$!26-7
! !"#$%&'()*%('+,%-'.)-%/)$(/0#/.)1!2)!334)"() +"##'5,.)16"(,(/547 ! !"#$%&'()('$&/+'.)*%('+,%-'.)8%,9)+/&&.),") (:5,%#')&%;(/(<)1!"#$%&'4 ! =:5,%#')+"5,("&.)/-/%&/;&')-%/)&%;(/(<)>?@) /5*)'5-%("5#'5,)-/(%/;&'. ! A5-%("5#'5,)-/(%/;&'.)+"5,("&)$/(/&&'&%.# !"#$%&"$'()*+,- !"#$-.(*,&/* !"#$,0%+"1. !"#$%*-'*,
!"#$%&'$())* &* ,-./!0$1$02342"556/4$!37.8
! !"#$%&%'()*+,%-.(+-/()0-&*,1-%2+'%1- ! 3*,4+/)(%-'4,+&'(5%+()*+,4/(5+,%+6$4) " 7454,+$(8+0)(91,()#4&%90%-.()*+,4/(/+'+ " 7*+,%-.(+$8+0)(+'('*4(5+,%+6$4($454$ ! :,1.,+;;4,(,4)#1-)%6$4(91,()0-&*,1-%2+'%1- " <-%-'4-/4/()*+,%-.($4+/)('1(=/+'+(,+&4)> " <)4()0-&*,1-%2+'%1-(/%,4&'%54)(+-/($%6,+,0(?:@ " 70-&*,1-%2+'%1-(%)(4"#4-)%54
!"#$%&'$())* &+
! !"#$%&'%(&)#*+#$,&"-. B99C'0<.'C12'&<3
+1./62'+4<77'D.E/.112 $2 "#$%&'()!) **+,'-./0123/45'67'839:9;< 839:9;<,'=<><. =9.1'!?@!A,'()!) RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ! Getting Started with OpenMP RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 " http://www.openmp.org http://www.compunity.org RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 # $%%&'(()))*+&,-.&*+/0 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1 Shameless Plug - “Using OpenMP” !"#$%&'()*%+,- ,./0123*'451/*6'+*7./8' ,1/133*3',/.&/177$%& 951)71%:';.#0:'<1%'6*/',1# 23456/,77859::; 3<=>?@:' :?919?#!!:9?9 3<=>?@!' AB;?:?919?#!!:9?B CD7%5&/DE,'5!#5FG< RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ; What is OpenMP? ! De-facto standard Application Programming Interface (API) to write shared memory parallel applications in C, C++, and Fortran ! Consists of: " Compiler directives " Run time routines " Environment variables ! Specification maintained by the OpenMP Architecture Review Board (http://www.openmp.org) ! Version 3.0 has been released May 2008 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 A When to consider OpenMP? ! Using an automatically parallelizing compiler: " It can not find the parallelism # The data dependence analysis is not able to determine whether it is safe to parallelize or not " The granularity is not high enough # The compiler lacks information to parallelize at the highest possible level ! Not using an automatically parallelizing compiler: " No choice than doing it yourself RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @: Advantages of OpenMP ! Good performance and scalability " If you do it right .... ! De-facto and mature standard ! An OpenMP program is portable " Supported by a large number of compilers ! Requires little programming effort ! Allows the program to be parallelized incrementally RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @@ OpenMP and Multicore OpenMP is ideally suited for multicore architectures Memory and threading model map naturally Lightweight Mature Widely available and used RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 The OpenMP Execution Model !" !"#$%&'(%)"*'%+"(,- Master Thread Parallel region Worker Threads Synchronization Worker Parallel region Threads Synchronization RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 *+,#&--) A process is created by the operating system, and requires a fair amount of "overhead". Processes contain information about program resources and program execution state, including: !"#$%&'((")*+",$%&'(("-$%.,")*+".('$")*+"/01"-$%.,")*"" !"2034$%05'06"" !"7%$840-"14$'&6%$9:"" !"#$%-$/5"40(6$.&64%0("" !";'-4(6'$(""" !"<6/&8"" !"='/,"" !">4?'"1'(&$4,6%$("" !"<4-0/?"/&64%0("" !"<@/$'1"?4A$/$4'("" !")06'$!,$%&'(("&%55.04&/64%0"6%%?("B(.&@"/("5'((/-'"C.'.'(+",4,'(+" ('5/,@%$'(+"%$"(@/$'1"5'5%$9D:"" !!!"#$%&#'"$() !" *+,&'-) A thread is defined as an independent stream of instructions that can be scheduled to run as such by the operating system. Threads use and exist within the process resources - are able to be scheduled by the operating system - run as independent entities - they duplicate only the bare essential resources that enable them to exist as executable code. This independent flow of control is accomplished because a thread maintains its own: - Stack pointer - Registers - Scheduling properties (such as policy or priority) - Set of pending and blocked signals - Thread specific data. Threads may share the process resources with other threads that act equally independently (and dependently) Reading and writing to the same memory locations is possible, and therefore requires explicit synchronization by the programmer. Thread die if the parent process dies Thread Iis "lightweight" because most of the overhead has already been accomplished through the creation of its process. !!!"#$%&#'"$() !" @9 The OpenMP Memory Model T T # All threads have access to the same, globally shared, memory private private # Data can be shared or private # Shared data is accessible by all Shared threads # Private data can only be Memory accessed by the thread that T owns it # Data transfer is transparent to private the programmer private T # Synchronization takes place, T but it is mostly implicit private RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @" Data-sharing Attributes ! In an OpenMP program, data needs to be “labeled” ! Essentially there are two basic types: " Shared - There is only one instance of the data # All threads can read and write the data simultaneously, unless protected through a specific OpenMP construct # All changes made are visible to all threads $ But not necessarily immediately, unless enforced ...... " Private - Each thread has a copy of the data # No other thread can access this data # Changes only visible to the thread owning the data RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @# The private and shared clauses private (list) # No storage association with original object # All references are to the local object # Values are undefi ned on entry and exit shared (list) # Data is accessible by all threads in the team # All threads access the same address space RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" What is a Data Race? ! Two different threads in a multi-threaded shared memory program ! Access the same (=shared) memory location " Asynchronously and " Without holding any common exclusive locks and " At least one of the accesses is a write/store RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Example of a data race #pragma omp parallel shared(n) {n = omp_get_thread_num();} T # T # private private n Shared Memory RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Another example #pragma omp parallel shared(x) {x = x + 1;} T #$% T #$% private private x Shared Memory RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" About Data Races ! Loosely described, a data race means that the update of a shared variable is not well protected ! A data race tends to show up in a nasty way: " Numerical results are (somewhat) different from run to run " Especially with Floating-Point data difficult to distinguish from a numerical side-effect " Changing the number of threads can cause the problem to seemingly (dis)appear # May also depend on the load on the system " May only show up using many threads RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" A parallel loop Every iteration in this !"#$%&'()$&*+)$&,,- loop is independent of $$$./&0$'$./&0$,$1/&0) the other iterations Thread 1 Thread 2 ./(0'./(0,1/(0 ./20'./20,1/20 ./30'./30,1/30 ./40'./40,1/40 ./50'./50,1/50 ./60'./60,1/60 Time ./70'./70,1/70 ./80'./80,1/80 RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Not a parallel loop !"#$%&'()$&*+)$&,,- The result is not deterministic when $$$./&0$'$./&,10$,$2/&0) run in parallel ! Thread 1 Thread 2 ./(0'./10,2/(0 ./30'./40,2/30 ./10'./50,2/10 ./40'./60,2/40 ./50'./70,2/50 ./60'./80,2/60 Time ./70'./30,2/70 ./80'./+0,2/80 RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !! About the experiment ! We manually parallelized the previous loop " The compiler detects the data dependence and does not parallelize the loop ! Vectors a and b are of type integer ! We use the checksum of a as a measure for correctness: " checksum += a[i] for i = 0, 1, 2, ...., n-2 ! The correct, sequential, checksum result is computed as a reference ! We ran the program using 1, 2, 4, 32 and 48 threads " Each of these experiments was repeated 4 times RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Numerical results threads: 1 checksum 1953 correct value 1953 #$%$&'$()& threads: 1 checksum 1953 correct value 1953 threads: 1 checksum 1953 correct value 1953 threads: 1 checksum 1953 correct value 1953 *+&,(%-.+&/ threads: 2 checksum 1953 correct value 1953 threads: 2 checksum 1953 correct value 1953 threads: 2 checksum 1953 correct value 1953 threads: 2 checksum 1953 correct value 1953 threads: 4 checksum 1905 correct value 1953 threads: 4 checksum 1905 correct value 1953 threads: 4 checksum 1953 correct value 1953 threads: 4 checksum 1937 correct value 1953 threads: 32 checksum 1525 correct value 1953 threads: 32 checksum 1473 correct value 1953 threads: 32 checksum 1489 correct value 1953 threads: 32 checksum 1513 correct value 1953 threads: 48 checksum 936 correct value 1953 threads: 48 checksum 1007 correct value 1953 threads: 48 checksum 887 correct value 1953 threads: 48 checksum 822 correct value 1953 RvdP/V1 Basic Concepts in Parallelization Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @1 An OpenMP example H+/?I++&5)D%$5D-J,&,-J,-%5 H+/?I++&5&K/KII,IDL,J5M7D-05 D%,/K%D+-7 K-5N&,-265&/K0.K for (int i=0; i $ cc -xopenmp source.c $ export OMP_NUM_THREADS=5 $ ./a.out RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 Example Parallel Execution @B 4$/,KJ5: 4$/,KJ5@ 4$/,KJ59 4$/,KJ5! 4$/,KJ5" DO:?@AA DO9::?!AA DO"::?#AA DO1::?BAA DO;::?AAA a[i] a[i] a[i] a[i] a[i] + + + + + b[i] b[i] b[i] b[i] b[i] = = = = = c[i] c[i] c[i] c[i] c[i] RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @; Defining Parallelism in OpenMP ! OpenMP Team := Master + Workers ! A Parallel Region is a block of code executed by all threads simultaneously % The master thread always has thread ID 0 % Thread adjustment (if enabled) is only done before entering a parallel region % Parallel regions can be nested, but support for this is implementation dependent % An "if" clause can be used to guard the parallel region; in case the condition evaluates to "false", the code is executed serially ! A work-sharing construct divides the execution of the enclosed code region among the members of the team; in other words: they split the work RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Components of OpenMP Directives Runtime Environment environment variables !! ParallelParallel regionregion !! NumberNumber ofof threadsthreads !! NumberNumber ofof threadsthreads ! Thread ID !! WorksharingWorksharing ! Thread ID !! SchedulingScheduling typetype constructsconstructs !! DynamicDynamic threadthread ! Dynamic thread adjustmentadjustment ! Dynamic thread !! TaskingTasking adjustmentadjustment !! NestedNested parallelismparallelism !! SynchronizationSynchronization !! NestedNested parallelismparallelism !! ScheduleSchedule !! Data-sharingData-sharing !! ActiveActive levelslevels !! StacksizeStacksize attributesattributes !! ThreadThread limitlimit !! IdleIdle threadsthreads !! NestingNesting levellevel !! ActiveActive levelslevels !! AncestorAncestor threadthread !! ThreadThread limitlimit !! TeamTeam sizesize !! WallclockWallclock timertimer !! LockingLocking RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @A Directive format ! C: directives are case sensitive " Syntax: #pragma omp directive [clause [clause] ...] ! Continuation: use \ in pragma ! Conditional compilation: _OPENMP macro is set ! Fortran: directives are case insensitive " Syntax: sentinel directive [clause [[,] clause]...] " The sentinel is one of the following: # !$OMP or C$OMP or *$OMP (fixed format) # !$OMP (free format) ! Continuation: follows the language syntax ! Conditional compilation: !$ or C$ -> 2 spaces RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9: OpenMP clauses ! Many OpenMP directives support clauses " These clauses are used to provide additional information with the directive ! For example, private(a) is a clause to the “for” directive: " #pragma omp for private(a) ! The specific clause(s) that can be used, depend on the directive RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9@ Example 2 - Matrix times vector $;<#9=+9$"+<$<9#9>>?>$!"#$@?!90>A%3"3?-$B $$$$$$$$$$$$$<#&C9A?%&D2D/0+-$/E9#?@%+D3D9D4D8- !"#$%&'()$&*+)$&,,- j . $$$/0+$'$(1() $$$!"#$%2'()$2*3)$2,,- = * $$$$$/0+$,'$45&652678526) $$$95&6$'$/0+) i $$: TID = 0 TID = 1 for (i=0,1,2,3,4) for (i=5,6,7,8,9) i = 0 i = 5 sum = b[i=0][j]*c[j] sum = b[i=5][j]*c[j] a[0] = sum a[5] = sum i = 1 i = 6 sum = b[i=1][j]*c[j] sum = b[i=6][j]*c[j] a[1] = sum a[6] = sum ... etc ... RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 99 OpenMP Performance Example $#"" !%&'()*+ $%&'()*+, -%&'()*+, ) s / $""" p o fl M ( !#"" e Matrix too c n small * a m !""" r o f s r c e a P #"" l e s " ! !" !"" !""" !"""" !""""" !"""""" Memory Footprint (KByte) *) With the IF-clause in OpenMP this performance degradation can be avoided RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9! The if clause if (scalar expression) # Only execute in parallel if expression evaluates to true # Otherwise, execute serially #pragma omp parallel if (n > some_threshold) \ shared(n,x,y) private(i) { #pragma omp for for (i=0; i RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9" Barrier/1 Suppose we run each of these two loops in parallel over i: !"#$%&'()$&$*$F)$&,,- $$$95&6$'$45&6$,$85&6) !"#$%&'()$&$*$F)$&,,- $$$@5&6$'$95&6$,$45&6) This may give us a wrong answer (one day) Why ? RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9# Barrier/2 We need to have updated all of a[ ] first, before using a[ ] * !"#$%&'()$&$*$F)$&,,- $$$95&6$'$45&6$,$85&6) wait ! barrier !"#$%&'()$&$*$F)$&,,- $$$@5&6$'$95&6$,$45&6) All threads wait at the barrier point and only continue when all threads have reached the barrier point =>'?@'05*/*'$#'05*'&A1/1%0**'0510'05*'71))$%&'.@'$0*/10$.%#'.%0.'05/*16#' $#'$6*%0$B13'@./'2.05'3..)#:'05*/*'C$33'%.0'2*'1'6101'/1B*'$%'05$#'B1#* RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 91 Barrier/3 Barrier Region idle idle idle time Barrier syntax in OpenMP: ;<#9=+9$"+<$49##&?# GH"+<$49##&?# RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9B When to use barriers ? ! If data is updated asynchronously and data integrity is at risk ! Examples: " Between parts in the code that read and write the same section of memory " After one timestep/iteration in a solver ! Unfortunately, barriers tend to be expensive and also may not scale to a large number of processors ! Therefore, use them with care RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9; The nowait clause ! To minimize synchronization, some OpenMP directives/pragmas support the optional nowait clause ! If present, threads do not synchronize/wait at the end of that particular construct ! In Fortran the nowait clause is appended at the closing part of the construct ! In C, it is one of the clauses on the pragma ;<#9=+9$"+<$!"#$3"J9&A GH"+<$@"$ .$ $$$$$$$I $$$$$$$I $$$$$$$I : GH"+<$?3@$@"$3"J9&A RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 9A A more elaborate example ;<#9=+9$"+<$<9#9>>?>$&!$%3O>&+&A-$@?!90>A%3"3?-$B $$$$$$$$/E9#?@%3D9D4D8DLDMDK-$<#&C9A?%!D&D/89>?- . !$'$N1() Statement is executed by all threads ;<#9=+9$"+<$!"#$3"J9&A &K/KII,I5I++& !"#$%&'()$&*3)$&,,- P)+/Q5D75JD7%/DRM%,JS p $$$K5&6$'$L5&6$,$M5&6) a r a l l e l ;<#9=+9$"+<$!"#$3"J9&A r e &K/KII,I5I++& g i !"#$%&'()$&*3)$&,,- o $$$95&6$'$45&6$,$85&6) P)+/Q5D75JD7%/DRM%,JS n $$$$$$1111 ;<#9=+9$"+<$49##&?# synchronization Statement is executed /89>?$'$/0+%9D(D3-$,$/0+%KD(D3-$,$!) by all threads $$$$$$1111 :$P7QQ$R3@$"!$<9#9>>?>$#?=&"3$QQ7P RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !: Components of OpenMP Directives Runtime Environment environment variables $$ ParallelParallel regionregion $$ NumberNumber ofof threadsthreads $$ NumberNumber ofof threadsthreads $ Thread ID $$ WorksharingWorksharing $ Thread ID $$ SchedulingScheduling typetype constructsconstructs $$ DynamicDynamic threadthread $ Dynamic thread adjustmentadjustment $ Dynamic thread $$ TaskingTasking adjustmentadjustment $$ NestedNested parallelismparallelism $$ SynchronizationSynchronization $$ NestedNested parallelismparallelism $$ ScheduleSchedule $$ Data-sharingData-sharing $$ ActiveActive levelslevels $$ StacksizeStacksize attributesattributes $$ ThreadThread limitlimit $$ IdleIdle threadsthreads $$ NestingNesting levellevel $$ ActiveActive levelslevels $$ AncestorAncestor threadthread $$ ThreadThread limitlimit $$ TeamTeam sizesize $$ WallclockWallclock timertimer $$ LockingLocking RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !@ The Parallel Region A parallel region is a block of code executed by multiple threads simultaneously ;<#9=+9$"+<$<9#9>>?>$58>90/?55D6$8>90/?6$1116 . $$$!"#$%&'()*&$%&*+*',"*)&$-&./0/11*1! :&2$3.1$*)&4/00$*05 GH"+<$<9#9>>?>$58>90/?55D6$8>90/?6$1116 $$$!"#$%&'()*&$%&*+*',"*)&$-&./0/11*1! GH"+<$?3@$<9#9>>?>&2$3.1$*)&4/00$*05 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !"#$%&' !(#)(*#+' 01' &2)233#3'.#4*5$'6'7$'892:"3#;-' #include int main(int argc, char *argv[]) { printf("Hello World\n"); return(0); } ,-' .(/&' !"#$%&'%(&)#*+#$,&"-.# !"#$%&' !(#)(*#+' 01' &2)233#3'.#4*5$'6'7$'892:"3#;0' #include int main(int argc, char *argv[]) { #pragma omp parallel { printf("Hello World\n"); } // End of parallel region return(0); } ,-' .(/&' !"#$%&'%(&)#*+#$,&"-.# !"#$%&' !(#)(*#+' 01' &2)233#3'.#4*5$'6'7$'892:"3#;<' $ cc -xopenmp -fast hello.c $ export OMP_NUM_THREADS=2 $ ./a.out Hello World Hello World $ export OMP_NUM_THREADS=4 $ ./a.out Hello World Hello World Hello World Hello World $ ,-' .(/&' !"#$%&'%(&)#*+#$,&"-.# !9 The Worksharing Constructs The OpenMP worksharing constructs ;<#9=+9$"+<$!"#;<#9=+9$"+<$!"# ;<#9=+9$"+<$/?8A&"3/;<#9=+9$"+<$/?8A&"3/ ;<#9=+9$"+<$/&3=>?;<#9=+9$"+<$/&3=>? ...... $$$$$$11111111 $$$$$$$$$$$$11111111 $$$$$$$$$$$$11111111 :: :: :: GHSTU$VSGHSTU$VS GHSTU$WRXYZSFWGHSTU$WRXYZSFW GHSTU$WZF[\RGHSTU$WZF[\R $$$$$$11111111 $$$$$$$$$$$$11111111 $$$$$$$$$$$$11111111 GHSTU$RFV$VSGHSTU$RFV$VS GHSTU$RFV$WRXYZSFWGHSTU$RFV$WRXYZSFW GHSTU$RFV$WZF[\RGHSTU$RFV$WZF[\R % The work is distributed over the threads % Must be enclosed in a parallel region % Must be encountered by all threads in the team, or none at all % No implied barrier on entry; implied barrier on exit (unless nowait is specified) % A work-sharing construct does not launch any new threads RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !! The Workshare construct Fortran has a fourth worksharing construct: GHSTU$]S^_W`a^RGHSTU$]S^_W`a^R $$$$$$$$*9##9M$/M3A9LO*9##9M$/M3A9LO GHSTU$RFV$]S^_W`a^R$5FS]aZY6GHSTU$RFV$]S^_W`a^R$5FS]aZY6 Example: GHSTU$]S^_W`a^RGHSTU$]S^_W`a^R $$$$$$$$a%NIT-$'$a%NIT-$,$b%NIT-a%NIT-$'$a%NIT-$,$b%NIT- GHSTU$RFV$]S^_W`a^R$FS]aZYGHSTU$RFV$]S^_W`a^R$FS]aZY RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" The omp for directive - Example ;<#9=+9$"+<$<9#9>>?>$@?!90>A%3"3?-B $$$$$$$$/E9#?@%3D9D4D8D@-$<#&C9A?%&- $$. $$$$;<#9=+9$"+<$!"#$3"J9&A $$$$$ !"#$%&'()$&*3QN)$&,,- $$$$45&6$'$%95&6$,$95&,N6-Pc) $$$$ $$$$;<#9=+9$"+<$!"#$3"J9&A $$$$$ !"#$%&'()$&*3)$&,,- $$ $$$$@5&6$'$N1(P85&6) $$ $$:$P7QQ$R3@$"!$<9#9>>?>$#?=&"3$QQ7P (implied barrier) RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !B The sections directive - Example ;<#9=+9$"+<$<9#9>>?>$@?!90>A%3"3?-B $$$$$$$$/E9#?@%3D9D4D8D@-$<#&C9A?%&- $$. $$$$;<#9=+9$"+<$/?8A&"3/$3"J9&A $$$$. $$$$$$;<#9=+9$"+<$/?8A&"3 $$$$$$$ !"#$%&'()$&*3QN)$&,,- $$$$45&6$'$%95&6$,$95&,N6-Pc) $$$$$$ $$$$$$;<#9=+9$"+<$/?8A&"3 $$$$$$$ !"#$%&'()$&*3)$&,,- $$$$@5&6$'$N1(P85&6) $$$$ $$$$:$P7QQ$R3@$"!$/?8A&"3/$QQ7P $$:$P7QQ$R3@$"!$<9#9>>?>$#?=&"3$QQ7P RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !; Overlap I/O and Processing/1 !"#$%&'()*+, 4)56*778"9&'()*+,:7; -$%#$%&'()*+, . / . * 0 / . < 8 1 0 / ' 2 1 0 3 2 1 3 2 3 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !A Overlap I/O and Processing/2 ;<#9=+9$"+<$<9#9>>?>$/?8A&"3/ . $$$;<#9=+9$"+<$/?8A&"3 $$$. $$$$$!"#$%&3A$&'()$&*F)$&,,-$. $$$$$$$$%C"&@-$#?9@d&3<0A%&-) $$$$$$$$%C"&@-$/&=39>d#?9@%&-) !"#$%&'()*+, $$$$$: $$$: $$$;<#9=+9$"+<$/?8A&"3 $$$. $$$$$!"#$%&3A$&'()$&*F)$&,,-$. $$$$$$$$%C"&@-$J9&Ad#?9@%&-) 4)56*778"9&'()*+,:7; $$$$$$$$%C"&@-$<#"8?//d@9A9%&-) $$$$$$$$%C"&@-$/&=39>d<#"8?//?@%&-) $$$$$: $$$: $$$;<#9=+9$"+<$/?8A&"3 $$$. $$$$$!"#$%&3A$&'()$&*F)$&,,-$. $$$$$$$$%C"&@-$J9&Ad<#"8?//?@%&-) $$$$$$$$%C"&@-$J#&A?d"0A<0A%&-) -$%#$%&'()*+, $$$$$: $$$: :$P7QQ$R3@$"!$<9#9>>?>$/?8A&"3/$QQ7P RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ": Single processor region/1 This construct is ideally suited for I/O or initializations Original Code $$$$$11111 $$$e#?9@$95(11FQN6e) $$$$$11111 D6*B31/*'E'0.'2*'2*'#51/*6D ;<#9=+9$"+<$<9#9>>?> . $$$$$11111 one volunteer requested $$$e#?9@$95(11FQN6e) thanks, we're done $$$$$11111 May have to insert a : barrier here Parallel Version RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "@ Single processor region/2 single processor region time Threads wait in the barrier RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "9 The Single Directive Only one thread in the team executes the code enclosed ;<#9=+9$"+<$/&3=>?$5<#&C9A?65!&#/A<#&C9A?6$B $$$$$$$$$$$$$$$$$$$58" GH"+<$/&3=>?$5<#&C9A?65!&#/A<#&C9A?6 6'()*741('89 GH"+<$?3@$/&3=>?$58" RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "! Combined work-sharing constructs ;<#9=+9$"+<$<9#9>>?> ;<#9=+9$"+<$<9#9>>?>$!"# ;<#9=+9$"+<$!"# !"#$%1111- &&&:(0&2;;;5 Single PARALLEL loop GH"+<$<9#9>>?> GH"+<$@" GH"+<$<9#9>>?>$@" &&&&&&&&;;; &&&&&&&&;;; GH"+<$?3@$@" GH"+<$?3@$<9#9>>?>$@" GH"+<$?3@$<9#9>>?> GH"+<$<9#9>>?> Single WORKSHARE loop GH"+<$J"#f/E9#? GH"+<$<9#9>>?>$J"#f/E9#? &&&&&&&&;;; &&&&&&&&;;; GH"+<$?3@$J"#f/E9#? GH"+<$?3@$<9#9>>?>$J"#f/E9#? GH"+<$?3@$<9#9>>?> ;<#9=+9$"+<$<9#9>>?> ;<#9=+9$"+<$/?8A&"3/ ;<#9=+9$"+<$<9#9>>?>$/?8A&"3/$ .$;;;: <&;;;&: Single PARALLEL sections GH"+<$<9#9>>?>$ GH"+<$/?8A&"3/ GH"+<$<9#9>>?>$/?8A&"3/ &&&&&&&&;;; &&&&&&&&;;; GH"+<$?3@$/?8A&"3/ GH"+<$?3@$<9#9>>?>$/?8A&"3/ GH"+<$?3@$<9#9>>?> RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "" Orphaning $$$$$$$$I +/&$K-,J ;<#9=+9$"+<$<9#9>>?>$ C"&@$@"J"#f%- )+/Q?7$K/D-0 .$$$$$ JD/,E%DT, . $$$$$$$$I $$$$$$$$I $$;<#9=+9$"+<$!"# $$$$$%C"&@-$@"J"#f%-) $$$!"#$%&3A$&'()&*3)&,,- $$$$$$$$I $$$. : $$$$$$$$I $$$$$$$$I $$$: $$$$$$$$I : & The OpenMP specification does not restrict worksharing and synchronization directives (omp for, omp single, critical, barrier, etc.) to be within the lexical extent of a parallel region. These directives can be orphaned & That is, they can appear outside the lexical extent of a parallel region RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "# More on orphaning C"&@$@"J"#f%- $$$%C"&@-$@"J"#f%-)$GQ$W?g0?3A&9>$hS^ . ;<#9=+9$"+<$!"# $$$;<#9=+9$"+<$<9#9>>?> $$$!"#$%&'()1111- $$$. $$$. $$$$$%C"&@-$@"J"#f%-)$GQ$U9#9>>?>$hS^ $$$$$$I $$$: $$$: : & When an orphaned worksharing or synchronization directive is encountered in the sequential part of the program (outside the dynamic extent of any parallel region), it is executed by the master thread only. In effect, the directive will be ignored RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 "A OpenMP Runtime Routines RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 #: OpenMP Runtime Functions/1 Name Functionality omp_set_num_threads Set number of threads omp_get_num_threads Number of threads in team omp_get_max_threads Max num of threads for parallel region omp_get_thread_num Get thread ID omp_get_num_procs Maximum number of processors omp_in_parallel Check whether in parallel region omp_set_dynamic Activate dynamic thread adjustment (but implementation is free to ignore this) omp_get_dynamic Check for dynamic thread adjustment omp_set_nested Activate nested parallelism (but implementation is free to ignore this) omp_get_nested Check for nested parallelism omp_get_wtime Returns wall clock time omp_get_wtick Number of seconds between clock ticks C/C++ : Need to include file RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 #@ OpenMP Runtime Functions/2 !"#$ %&'()*+'",*)- +#./0$)/0(1$2&,$ 3$)40(1$2&,$45*6478&')*#$94*04&0$2: +#./;$)/0(1$2&,$ <$)&8'04)1$40(1$2&,$4*'4&0$ +#./;$)/)18$"2/,*#*) =">4'&#?$84+64)18$"2046+84.8+;8"# +#./0$)/#">/"()*@$/,$@$,0 3$)4'&#?$84+64"()*@$4."8",,$,48$;*+'0 +#./;$)/#">/"()*@$/,$@$,0 !&#?$84+64"()*@$4."8",,$,48$;*+'0 +#./;$)/,$@$, !&#?$84+64'$0)$24."8",,$,48$;*+'0 +#./;$)/"()*@$/,$@$, !&#?$84+64'$0)$24"()*@$4."8A48$;*+'0 +#./;$)/"'($0)+8/)18$"2/'&# B18$"24*24+64"'($0)+84)18$"2 +#./;$)/)$"#/0*C$45,$@$,: 3*C$4+64)1$4)18$"24)$"#4")4)1*04,$@$, =>=?? @ A**,&%5&8"6B$,*&C8B*&D5<#E(F G5)%)+" @ H,,&I$7*&5<#JB8KL&5)&8"6B$,*&C8B*&I5<#JB8KE(L RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 #B OpenMP Environment Variables RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 #; OpenMP Environment Variables N&,-265,-TD/+-.,-%5TK/DKRI, U,VKMI%5V+/5N/KEI,5<+IK/D75<%MJD+ N26Y>G2Y4j^_[U<5- @ N26Y<\j_UGC_5W7E$,JMI,8dE$M-QfX 7%K%DE85W>(6X N26YUZ>[23\5]54^G_5`5H[C<_5a 4^G_ N26Y>_<4_U5]54^G_5`5H[C<_5a H[C<_ N26Y<4[\b<3c_57DL,5d=`b`2`ef "52=5P!95RD%S5(5;52=5P1"?RD%S N26Yg[34Y6NC3\Z55d[\43h_5`56[<<3h_f 6[<<3h_ N26Y2[iY[\43h_YC_h_C< " N26Y4j^_[UYC3234 @:9" Note: The names are in uppercase, the values are case insensitive RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1" Using OpenMP RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1# Using OpenMP ! We have already seen many features of OpenMP ! We will now cover " Additional language constructs " Features that may be useful or needed when running an OpenMP application ! The tasking concept is covered in separate section RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 11 About storage association ! Private variables are undefined on entry and exit of the parallel region ! A private variable within a parallel region has no storage association with the same variable outside of the region ! Use the first/last private clause to override this behavior ! We illustrate these concepts with an example RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1B Example private variables +9&3%- . $$a$'$N() ;<#9=+9$"+<$<9#9>>?> . $$;<#9=+9$"+<$!"#$<#&C9A?%&-$!&#/A<#&C9A?%a-$>9/A<#&C9A?%b-111;<#9=+9$"+<$!"#$<#&C9A?%&Db-$!&#/A<#&C9A?%a-$111;<#9=+9$"+<$!"#$<#&C9A?%&DaDb-$111 $$!"#$%&'()$&*3)$&,,- $$. $$$$$$1111 $$$$$$b$'$a$,$&)$$$$$ P7QQ$a$03@?!&3?@D$03>?//$@?8>9#?@$ $$$$$!&#/A<#&C9A?$QQ7P $$$$$$1111$$ $$: P7QQ$b$03@?!&3?@D$03>?//$@?8>9#?@$ $$X$'$b)$$$$$$$$$$$$ $$$$$>9/A<#&C9A?$QQ7P :$P7QQ$R3@$"!$S>?>$#?=&"3$QQ7P : F$#B31$7*/G'H5$#'B.6*'@/1&7*%0'$#'%.0'<*/8'7*1%$%&@A3'1%6'.%38'#*/<*#'0.' 6*7.%#0/10*'05*'B31A#*# RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1; The first/last private clauses firstprivate (list) # All variables in the list are initialized with the value the original object had before entering the parallel construct lastprivate (list) # The thread that executes the sequentially last iteration or section updates the value of the objects in the list RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1A The default clause default (none | shared | private | threadprivate ) Fortran default ( none | shared ) C/C++ none # No implicit defaults; have to scope all variables explicitly shared # All variables are shared # The default in absence of an explicit "default" clause private # All variables are private to the thread # Includes common block data, unless THREADPRIVATE firstprivate # All variables are private to the thread; pre-initialized RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B: The reduction clause - Example !!!!!!/0+$'$(1( GH"+<$<9#9>>?>$@?!90>A%3"3?-$i GH"+<$/E9#?@%3DL-$<#&C9A?%&- Variable SUM is a GH"+<$@"$#?@08A&"3$%,I/0+- shared variable $$$$$$@"$&$'$ND$3 $$$$$$$$$/0+$'$/0+$,$L%&- $$$$$$?3@$@" GH"+<$?3@$@" GH"+<$?3@$<9#9>>?> $$$$$$<#&3A$7D/0+ % Care needs to be taken when updating shared variable SUM % With the reduction clause, the OpenMP compiler generates code such that a race condition is avoided RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B@ The reduction clause reduction ( [operator | intrinsic] ) : list ) Fortran reduction ( operator : list ) C/C++ # Reduction variable(s) must be shared variables # A reduction is defined as: Check the docs Fortran C/C++ for details L$'$L$"&/A- L,,D$,,LD$LQQD$QQL L$'$&3A#&3/&8$%?L<#d>&/AD$L- L$*4&3" # The reduction can be hidden in a function call RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B9 Fortran - Allocatable Arrays ! Fortran allocatable arrays whose status is “currently allocated” are allowed to be specified as private, lastprivate, firstprivate, reduction, or copyprivate integer, allocatable,dimension (:) :: A integer i allocate (A(n)) !$omp parallel private (A) do i = 1, n A(i) = i end do ... !$omp end parallel RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B! The schedule clause/1 schedule ( static | dynamic | guided | auto [, chunk] ) schedule (runtime) static [, chunk] # Distribute iterations in blocks of size "chunk" over the threads in a round-robin fashion # In absence of "chunk", each thread executes approx. N/P chunks for a loop of length N and P threads " Details are implementation defined # Under certain conditions, the assignment of iterations to threads is the same across multiple loops in the same parallel region RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B" The schedule clause/2 Example static schedule Loop of length 16, 4 threads: Thread 0 1 2 3 -(&'#,-8= NQj kQl mQNc NnQNo '#,-8&>&? NQc nQj kQo pQl mQN( NNQNc NnQNj NkQNo *) The precise distribution is implementation defined RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B# The schedule clause/3 dynamic [, chunk] # Fixed portions of work; size is controlled by the value of chunk # When a thread finishes, it starts on the next portion of work guided [, chunk] # Same dynamic behavior as "dynamic", but size of the portion of work decreases exponentially auto # The compiler (or runtime system) decides what is best to use; choice could be implementation dependent runtime # Iteration scheduling scheme is set at runtime through environment variable OMP_SCHEDULE RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 B1 The Experiment 500 iterations using 4 threads 3 2 guided, 5 1 0 3 D 2 I d 1 a e 0 r h 3 T dynamic, 5 2 1 static 0 : #: @:: @#: 9:: 9#: !:: !#: ":: "#: #:: Iteration Number RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Nested Parallelism Master Thread 3-way parallel Outer parallel region 9-way parallel Nested parallel region 3-way parallel Outer parallel region Note: nesting level can be arbitrarily deep RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !" Loop Collapse ! !""#$%&'()(""*"+,(-+#.&#/&'*)/*0-"1&.*%-*2&"##'%&$+-3#4-& 4%+.5&.*%-*2&'()(""*"+%6 ! #$%%&'()&0"(4%*&#.&/#)72#&"##'&+.2+0(-*%&3#$&6(.1&"##'%& %3#4"2&8*&0#""('%*2 ! 9#6'+"*)&/#)6%&(&%+.5"*&"##'&(.2&-3*.&'()(""*"+,*%&-3+% !$omp parallel do collapse(2) ... do i = il, iu, is do j = jl. ju. js do k = kl, ku, ks ..... end do end do end do !$omp end parallel do RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;@ Additional Directives/1 ;<#9=+9$"+<$+9/A?# .6'()*741('89: GH"+<$+9/A?# 6'()*741('89 GH"+<$?3@$+9/A?# ;<#9=+9$"+<$8#&A&89>$5%39+?-6 .6'()*741('89: GH"+<$8#&A&89>$5%39+?-6 6'()*741('89 GH"+<$?3@$8#&A&89>$5%39+?-6 ;<#9=+9$"+<$9A"+&8 GH"+<$9A"+&8 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;9 Additional Directives/2 ;<#9=+9$"+<$"#@?#?@ .6'()*741('89: GH"+<$"#@?#?@ 6'()*741('89 GH"+<$?3@$"#@?#?@ ;<#9=+9$"+<$!>0/E$5%>&/A-6 GH"+<$!>0/E$5%>&/A-6 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;! The Master Directive Only the master thread executes the code block: ;<#9=+9$"+<$+9/A?# .6'()*741('89: There is no implied barrier on entry or GH"+<$+9/A?# exit ! 6'()*741('89 GH"+<$?3@$+9/A?# RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;" Critical Region/1 If sum is a shared variable, this loop can not run in parallel !"#$%&'()$&$*$3)$&,,-. $$$$$11111 $$$/0+$,'$95&6) $$$$$11111 : We can use a critical region for this: !"#$%&'()$&$*$3)$&,,-. $$$$$11111 one at a time can proceed $$$/0+$,'$95&6) $$$$$11111 next in line, please : RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;# Critical Region/2 ! Useful to avoid a race condition, or to perform I/O (but that still has random order) ! Be aware that there is a cost associated with a critical region critical region time RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;1 Critical and Atomic constructs Critical: All threads execute the code, but only one at a time: ;<#9=+9$"+<$8#&A&89>$5%39+?-6 .6'()*741('89: There is no implied GH"+<$8#&A&89>$5%39+?-6 barrier on entry or 6'()*741('89 exit ! GH"+<$?3@$8#&A&89>$5%39+?-6 Atomic: only the loads and store are atomic .... ;<#9=+9$"+<$9A"+&8 GH"+<$9A"+&8 */A9A?+?3AO */A9A?+?3AO This is a lightweight, special ;<#9=+9$"+<$9A"+&8 form of a critical section $$$$95&3@L5&66$,'$45&6) RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;B More synchronization constructs The enclosed block of code is executed in the order in which iterations would be executed sequentially: ;<#9=+9$"+<$"#@?#?@ May introduce .6'()*741('89: serialization (could be expensive) GH"+<$"#@?#?@ 6'()*741('89 GH"+<$?3@$"#@?#?@ Ensure that all threads in a team have a consistent view of certain objects in memory: ;<#9=+9$"+<$!>0/E$5%>&/A-6 In the absence of a list, all visible variables are flushed GH"+<$!>0/E$5%>&/A-6 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ;; Implied flush regions ! During a barrier region ! At exit from worksharing regions, unless a nowait is present ! At entry to and exit from parallel, critical, ordered and parallel worksharing regions ! During omp_set_lock and omp_unset_lock regions ! During omp_test_lock, omp_set_nest_lock, omp_unset _nest_lock and omp_test_nest_lock regions, if the region causes the lock to be set or unset ! Immediately before and after every task scheduling point ! At entry to and exit from atomic regions, where the list contains only the variable updated in the atomic construct ! A fl ush region is not implied at the following locations: " At entry to a worksharing region " At entry to or exit from a master region RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @1@ Summary OpenMP ! OpenMP provides for a small, but yet powerful, programming model ! It can be used on a shared memory system of any size " This includes a single socket multicore system ! Compilers with OpenMP support are widely available ! The tasking concept opens up opportunities to parallelize a wider range of applications ! Oracle Solaris Studio has extensive support for OpenMP developers RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 Example: Serial PI Program static long num_steps = 100000; double step; void main () { int i; double x, pi, sum = 0.0; step = 1.0/(double) num_steps; for (i=0;i< num_steps; i++){ x = (i+0.5)*step; sum = sum + 4.0/(1.0+x*x); } pi = step * sum; } 32 Example: A simple Parallel pi program !"#$%&'()*+,-./0) VL+,+2()1$3%3L)2+)3#) 1232"$)%+#4)#&,512(-1)6)7888889)))))))))'+&:%()12(-9) 3LL3W)'",(#1"+#('):W) !'(;"#()<=>5?@ABCDE)F) #&,:(L)+;)2/L(3'1)2+) G+"'),3"#)HI) 3G+"')L3$()$+#'"2"+#.) J) ))"#2)"K)#2/L(3'19))'+&:%()-"K)1&,M<=>5?@ABCDEN9) ) ))12(-)6)7.8OH'+&:%(I)#&,512(-19) ) ))+,-51(25#&,52/L(3'1H<=>5?@ABCDEI9) ))))!-L34,3)+,-)-3L3%%(%) ))))J) )))) ) )"#2)"K)"'K#2/L'19) Y#%W)+#()2/L(3')1/+&%')$+-W) ))))))))))))))'+&:%()P9) 2/()#&,:(L)+;)2/L(3'1)2+)2/() 4%+:3%)G3%&()2+),3X()1&L() ))))))))))))))"')6)+,-54(252/L(3'5#&,HI9) ,&%2"-%()2/L(3'1)ZL"2"#4)2+)2/() ))))))))))))))#2/L'1)6)+,-54(25#&,52/L(3'1HI9) 13,()3''L(11)'+# 2)$+#;%"$2.))) ))))))))))))))";)H"')66)8I)))#2/L(3'1)6)#2/L'19) ) ));+L)H"6"'K)1&,M"'N68.89"*)#&,512(-19)"6"Q#2/L'1I)J) ) ) ))P)6)H"Q8.RIS12(-9) ?/"1)"1)3)$+,,+#) 2L"$X)"#)EV>D) ) ) ))1&,M"'N)Q6)T.8OH7.8QPSPI9) -L+4L3,1)2+)$L(32() ) ))U) 3)$W$%"$)'"12L":&2"+#) )))))U) +;)%++-)"2(L32"+#1) ) ));+LH"68K)-"68.89"*#2/L(3'19"QQI-")Q6)1&,M"N)S)12(-9) U) 33 SPMD: Single Program Mulitple Data Run the same program on P processing elements where P can be arbitrarily large. - between a set of tasks and to manage any shared data structures. This pattern is very general and has been used to support most (if not all) the algorithm strategy patterns. probably the most commonly used pattern in the history of parallel programming. 34 Results* Original Serial pi program with 100000000 steps ran in 1.83 seconds. threads 1st SPMD 1 1.86 2 1.03 3 1.08 4 0.97 *Intel compiler (icpc) with no optimization on Apple OS X 10.7.3 with a dual core (four HW thread) Intel® CoreTM i5 processor at 1.7 Ghz and 4 Gbyte DDR3 memory at 1.333 Ghz. !"# Why such poor scaling? False sharing If independent data elements happen to sit on the same cache line, each . HW thrd. 0 HW thrd. 1 HW thrd. 2 HW thrd. 3 L1 $ lines L1 $ lines Sum[0] Sum[1] Sum[2] Sum[3] Sum[0] Sum[1] Sum[2] Sum[3] Core 0 Core 1 Shared last level cache and connection to I/O and DRAM If you promote scalars to an array to support creation of an SPMD program, the array elements are contiguous in memory and hence share cache lines Solution: Pad arrays so elements you use are on distinct cache lines. 36 Example: eliminate False sharing by padding the sum array !"#$%&'()*+,-./0) 1232"$)%+#4)#&,512(-1)6)7888889)))))))))'+&:%()12(-9) !'(;"#())))<=>))))))?))))))))))))@@)311&,()AB):C2()D7)$3$/()%"#()1"E() !'(;"#()FGH5IJKL=>M)N) O+"'),3"#)PQ) R) ))"#2)"S)#2/T(3'19))'+&:%()-"S)1&,UFGH5IJKL=>MVU<=>W9) ) ))12(-)6)7.8@P'+&:%(Q)#&,512(-19) ) ))+,-51(25#&,52/T(3'1PFGH5IJKL=>MQ9) ))))!-T34,3)+,-)-3T3%%(%) <3')2/()3TT3C) ))))R) )"#2)"S)"'S#2/T'19) 1+)(3$/)1&,) ))))))))))))))'+&:%()X9) O3%&()"1)"#)3) '";;(T(#2) ))))))))))))))"')6)+,-54(252/T(3'5#&,PQ9) $3$/()%"#() ))))))))))))))#2/T'1)6)+,-54(25#&,52/T(3'1PQ9) ))))))))))))))";)P"')66)8Q)))#2/T(3'1)6)#2/T'19) ) ));+T)P"6"'S)1&,U"'V68.89"*)#&,512(-19)"6"Y#2/T'1Q)R) ) ) ))X)6)P"Y8.ZQ[12(-9) ) ) ))1&,U"'VU8V)Y6)[email protected][XQ9) ) ))W) )))))W) ) ));+TP"68S)-"68.89"*#2/T(3'19"YYQ-")Y6)1&,U"VU8V)[)12(-9) W) 37 Results*: pi program padded accumulator Original Serial pi program with 100000000 steps ran in 1.83 seconds. threads 1st 1st SPMD SPMD padded 1 1.86 1.86 2 1.03 1.01 3 1.08 0.69 4 0.97 0.53 *Intel compiler (icpc) with no optimization on Apple OS X 10.7.3 with a dual core (four HW thread) Intel® CoreTM i5 processor at 1.7 Ghz and 4 Gbyte DDR3 memory at 1.333 Ghz. !"# ! !"##$%&'()"%*+',)'-.'/)""0 B99C'0<.'C12'&<3 +1./62'+4<77'D.E/.112 $2 "#$%&'()!) **+,'-./0123/45'67'839:9;< 839:9;<,'=<><. =9.1'!?@!A,'()!) RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ( Outline ! The Myth ! Deep Trouble ! Get Real ! The Wrapping RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 12'34#56'$%').)789:'7;"6'$;';.3"#5$%&' I #59#'$;'<$0"84'="8$">"0'=7#'?98;"@A'@@@@@@ B;.7:C"D'<<<@:"?":"%C"@C.3E RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ? The Myth “OpenMP Does Not Scale” RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 J Hmmm .... What Does That Really Mean ? RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 A Some Questions I Could Ask 1F.'4.7'3"9%'4.7'<:.#"'9')9:988"8'):.&:936'7;$%&'()"%*+' 9%0'$#'0.";%G#')":?.:3HA' 1I';""@'F$0'4.7'39J"';7:"'#5"'):.&:93'<9;'?9$:84'<"88' .)#$3$K"0'$%';"L7"%#$98'3.0"HA' 1(5@'M.7'0$0%G#@'Q4'#5"'<946'<54'0.'4.7'"R)"C#'#5"'):.&:93' #.';C98"HA 1(5@'M.7'N7;#'#5$%J'$#';5.780'9%0'4.7'7;"0'988'#5"'C.:";@' O9>"'4.7'";#$39#"0'#5"';)""0'7)'7;$%&'230958G;'P9 RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 K Some More Questions I Could Ask 1(5@'M.7'0$0%G#@'M.7'N7;#')9:988"8$K"0'988'8..);'$%'#5"' ):.&:93@'F$0'4.7'#:4'#.'9>.$0')9:988"8$K$%&'$%%":3.;#'8..);' $%'9'8..)'%";#HA' 1(5@'M.7'0$0%G#@'F$0'4.7'3$%$3$K"'#5"'%73=":'.?')9:988"8' :"&$.%;'#5"%HA' 1(5@'M.7'0$0%G#@'I#'N7;#'<.:J"0'?$%"'#5"'<94'$#'<9;@ 1F$0'4.7'9#'8"9;#'7;"'#5"'%.<9$#'C897;"'#.'3$%$3$K"'#5"'7;"' .?'=9::$":;HA' 1(5@'M.7G>"'%">":'5"9:0'.?'9'=9::$":@'*$&5#'="'<.:#5'#.':"90' 7)'.%@A 1F.'988'):.C";;.:;':.7&584')":?.:3'#5"';93"'93.7%#'.?' <.:JHA 1M.7'0.%G#'J%.<6'=7#'#5$%J'$#'$;'.J94@'I'5.)"'4.7G:"':$&5#@A RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 L I Don't Give Up That Easily 1F$0'4.7'39J"'.)#$398'7;"'.?'):$>9#"'09#96'.:'0$0'4.7';59:"' 3.;#'.?'$#HA' 1(5@'M.7'0$0%G#@'/59:$%&'$;'N7;#'"9;$":@'I';""@ 1M.7';""3'#.'="'7;$%&'9'CCVS,*2';4;#"3@'F$0'4.7'#9J"'#59#' $%#.'9CC.7%#HA' 1M.7G>"'%">":'5"9:0'.?'#59#'"$#5":@'O.<'7%?.:#7%9#"@'U.780' #5":"')":59);'="'9%4'?98;"';59:$%&'9??"C#$%&')":?.:39%C"HA' 1(5@'S">":'5"9:0'.?'#59#'"$#5":@'*94'C.3"'59%04'#.'8"9:%'9' 8$##8"'3.:"'9=.7#'=.#5@A' 1/.6'<59#'0$0'4.7'0.'%"R#'#.'900:";;'#5"')":?.:39%C"'HA' 1/<$#C5"0'#.'*+I@'F.";'#59#')":?.:3'9%4'="##":'#5"%HA' 1(5@'M.7'0.%G#'J%.<@'M.7G:"';#$88'0"=7&&$%&'#5"'C.0"@A RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 M Going Into Pedantic Mode 1W5$8"'4.7G:"'<9$#$%&'?.:'4.7:'*+I'0"=7&':7%'#.'?$%$;5'B9:"' 4.7';7:"'$#'0.";%G#'59%&'=4'#5"'<94E6')8"9;"'988.<'3"'#.'#98J' 9'8$##8"'3.:"'9=.7#'()"%*+'9%0'+":?.:39%C"@A RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !) Deep Trouble RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !! OpenMP and Performance ! The transparency of OpenMP is a mixed blessing " Makes things pretty easy " May mask performance bottlenecks ! In the ideal world, an OpenMP application just performs well ! Unfortunately, this is not the case ! Two of the more obscure effects that can negatively impact performance are cc-NUMA behavior and False Sharing ! Neither of these are restricted to OpenMP, but they are important enough to cover in some detail here RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !( False Sharing RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !I False Sharing A store into a shared cache line invalidates the other copies of that line: CPUs Caches Memory The system is not able to distinguish between changes within one individual line RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !? False Sharing Red Flags # Be alert, when all of these three conditions are met: " Shared data is modified by multiple processors " Multiple threads operate on the same cache line(s) " Update occurs simultaneously and very frequently # Use local data where possible # Shared read-only data does not lead to false sharing RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !J Considerations for cc-NUMA RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !A A generic cc-NUMA architecture % 5 1 2 O 6 6 O &26F13362 &26F13362 2 1 5 % P.C98'2CC";; X"3.#"'2CC";; B?9;#E * *9$%'I;;7"D'O.<'-.'F$;#:$=7#"'-5"'F9#9'H RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !K About Data Distribution ! Important aspect on a cc-NUMA system " If not optimal - longer access times, memory hotspots ! OpenMP does not provide support for cc-NUMA ! Placement comes from the Operating System " This is therefore Operating System dependent ! Solaris, Linux and Windows use “First Touch” to place data RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !L About “First Touch” placement/1 % 5 1 a[0] 2 O 6 : 6 O &26F13362 &26F13362 2 1 a[99] 5 % * for (i=0; i<100; i++) a[i] = 0; First Touch All array elements are in the memory of the processor executing this thread RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 !M About “First Touch” placement/2 % 5 1 a[0] 2 a[50] O 6 6 : O &26F13362 &26F13362 : 2 1 a[49] 5 a[99] % * #pragma omp parallel for num_threads(2) for (i=0; i<100; i++) a[i] = 0; First Touch Both memories each have “their half” of the array RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 () Get Real RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (! Block Matrix Update RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (( A 3D matrix update !!!!!!"#!$!%!&'!( !!!!!!!!!"#!)!%!&'!( *+#,-!-./.0010!"#!"12.304567./1"8!-/9:.41598!; *+#,-!6<71"301564.49<8 !!!!!!!!!!!!"#!9!%!='!, !!!!!!!!!!!!!!!>59')'$8!%!>59')'$?=8!@!>59')?='$8A6<.01 !!!!!!!!!!!!1("!"# *+#,-!1("!-./.0010!"# !!!!!!!!!1("!"# !!!!!!1("!"# ! The loops are correctly nested for K Data Dependency Graph serial performance k ! Due to a data dependency on J and K, only the inner loop can be k-1 parallelized ! This will cause the barrier to be executed (N-1) 2 times J I j-1 j RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (I The performance IJ) Scaling is very poor I)) Inner loop over I has (as to be expected) ) s / been parallelized p (J) o fl M ()) ( e c !J) n a m r !)) o f r e J) P ) ) !) () I) ?) J) A) Number of threads Dimensions : M=7,500 N=20 Footprint : ~24 MByte RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (? Performance Analyzer data !B.,1!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!C><0D!E61/!!!!F(<0D!!!!!C><0D!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!Using 10 threads GHE!!!!!!!!!!!E61/!GHE!!I.00!!! !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!61 !\0#<$QW"Q!??!ZH!"#.00!2/#,!09(1!=U!]Q+"=[=UD\0#<$QW"Q^!!!ND=TN!!!=DY!!!ODP=N!!!!ND=TN!l s a !\0#<$QW"Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!NDNUN!!!NDU!!!YDUYN!!!!NDNUN!t c t a !,1,614!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!NDNWN!!!NDW!!!NDNWN!!!!NDNXNo a l n e s e l o s B.,1!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!C><0D!E61/!!!!F(<0D!!!!!C><0D!!a d Using 20 threads o !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!c GHE!!!!!!!!!!!E61/!GHE!!I.00!!! m !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!s 61 w !QQ,4QC("R2L.6$QS.//91/Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!&ODTNN!!OUDO!!&ODTNN!!!!NDPXN! h !QQ,4QI.94V#/I#/$Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=PDXXN!!U&D&!!=PDXXN!!!!ND!!!! a !QQ,4QZ.641/V3(<49#(Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!=D=NN!!!&DW!!!=DW&N!!!!=D=NN! t !Z[FBQ!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!ND=PN!!!NDU!!!&DO&N!!!!NDUTN! !\0#<$QW"Q!??!ZH!"#.00!2/#,!09(1!=U!]Q+"=[=UD\0#<$QW"Q^!!!ND=NN!!!ND&!!&ODXNN!!!!ND=NN! !QQ,4Q6143-Q"#_#\Q9(4Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!NDNXN!!!ND&!!!NDNXN!!!!NDNXN! !QQ,4Q6143-Q)#\Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!NDN&N!!!NDN!!!NDN&N!!!!NDN&N! !\0#<$QW"Q!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!NDN=N!!!NDN!!&TDN&N!!!!NDN=N! Question: Why is __mt_WaitForWork so high in the profile ? RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (J The Analyzer Timeline overview __mt_EndOfTask_Barrier __mt_WaitForWork RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (A This is False Sharing at work ! *+#,-!-./.0010!"#!"12.304567./1"8!-/9:.41598!; *+#,-!6<71"301564.49<8 !!!!!!!!!!!!"#!9!%!='!, !!!!!!!!!!!!!!!>59')'$8!%!>59')'$?=8!@!>59')?='$8A6<.01 !!!!!!!!!!!!1("!"# *+#,-!1("!-./.0010!"# P=1 P=2 P=4 P=8 False sharing increases as we increase the number of g n i threads r a h s o n RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (K Sanity Check: Setting M=75000* Only a very few barrier calls now *) Increasing the length of the loop should decrease false sharing RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (L Performance comparison K)) ) s / A)) p o M = 75,000 fl J)) M ( e ?)) c n a I)) m r o f r ()) e P !)) M = 7,500 ) ) !) () I) ?) J) A) Number of threads For a higher value of M, the program scales better RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 (M Observation K ! No data dependency on 'I' ! Therefore we can split the 3D matrix in larger blocks and process these in parallel J I "#!$!%!&'!( !!!"#!)!%!&'!( !!!!!!"#!9!%!='!, !!!!!!!!!>59')'$8!%!>59')'$?=8!@!>59')?='$8A6<.01 !!!!!!1("!"# !!!1("!"# 1("!"# RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 I) The Idea K ! We need to distribute the M iterations over the number of processors ie is ! We do this by controlling the start (IS) and end (IE) I value of the inner loop J ! Each thread will calculate these values for it's portion of the work "#!$!%!&'!( !!!"#!)!%!&'!( !!!!!!"#!9!%!96'!91 !!!!!!!!!>59')'$8!%!>59')'$?=8!@!>59')?='$8A6<.01 !!!!!!1("!"# !!!1("!"# 1("!"# RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 I! The first implementation !!!!!361!#,-Q09\ 63\/#349(1!$1/(10596'91','('>'6<.018 !!!!!!!!!DDDDDDDD !!!!!!!!!DDDDDDD "#!$!%!&'!( !!!!!(/1,!!!%!,#"5,'(47/1."68 !"#!)!%!&'!( !!!!!(<73($!%!5,?(/1,8`(47/1."6 !!!!"#!9!%!96'!91 !!!!!!>59')'$8%>59')'$?=8@>59')?='$8A6<.01 !!!!1("!"# *+#,-!-./.0010!"12.304!5(#(18; !!1("!"# *+#,-!-/9:.41!5H'96'918!!!!!!; 1("!"# *+#,-!67./1"!!5(/1,'(<73($','('>'6<.018 !!!!!!H!%!#,-Qa14Q47/1."Q(3,58 !!!!!!92!5!H!K!(/1,!8!471( !!!!!!!!96!%!=!@!HA5(<73($!@!=8 !!!!!!!!91!%!96!@!(<73($ !!!!!!1061 !!!!!!!!96!%!=!@!HA(<73($@!(/1, !!!!!!!!91!%!96!@!(<73($!?!= !!!!!!1("!92 !!!!!!<.00!$1/(10596'91','('>'6<.018 *+#,-!1("!-./.0010 RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 I( Another Idea: Use OpenMP ! !!!!!!361!#,-Q09\ !!!!!!9,-09<94!(#(1 !!!!!!9(41a1/!!!!!bb!96'!91'!,'!( !!!!!!/1.05$9("%X8bb!>5,'('(8'!6<.01 !!!!!!9(41a1/!!!!!bb!9'!)'!$ *+#,-!-./.0010!"12.3045(#(18!; *+#,-!-/9:.4159')'$8!67./1"5,'('6<.01'>8 !!!!!!"#!$!%!&'!( !!!!!!!!!"#!)!%!&'!( *+#,-!"#!6<71"301564.49<8 !!!!!!!!!!!!"#!9!%!='!, !!!!!!!!!!!!!!!>59')'$8!%!>59')'$?=8!@!>59')?='$8A6<.01 !!!!!!!!!!!!1("!"# *+#,-!1("!"#!(#c.94 !!!!!!!!!1("!"# !!!!!!1("!"# *+#,-!1("!-./.0010 RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 II How this works on 2 threads L7/1."!N!C>1<3416b L7/1."!=!C>1<3416b $%& parallel region $%& )%& )%& "#!9!%!=',`& "#!9!%!,`&@=', !!!>59'&'&8!%!DDD work sharing !!!>59'&'&8!%!DDD 1("!"# 1("!"# $%& parallel region $%& )%W )%W "#!9!%!=',`& "#!9!%!,`&@=', !!!>59'W'&8!%!DDD work sharing !!!>59'W'&8!%!DDD 1("!"# 1("!"# DDD!14 RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 I? Performance ! We have set M=7500 N=20 " This problem size does not scale at all when we explicitly parallelized the inner loop over 'I' ! We have have tested 4 versions of this program " Inner Loop Over 'I' - Our first OpenMP version " AutoPar - The automatically parallelized version of 'kernel' " OMP_Chunks - The manually parallelized version with our explicit calculation of the chunks " OMP_DO - The version with the OpenMP parallel region and work-sharing DO RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 IJ The performance (M=7,500) (J)) OMP Chunks ) ())) OMP DO s / p o fl M !J)) ( e The auto-parallelizing c n a !))) compiler does really well ! m r o f r e J)) P Innerloop ) ) !) () I) ?) J) A) Number of threads Dimensions : M=7,500 N=20 Footprint : ~24 MByte RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 IA Matrix Times Vector RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 IK The Sequential Source 2#/!59%Nd!9K,d!9@@8 j e !!.]9^!%!NDNd = * !!2#/!5)%Nd!)K(d!)@@8 !!!!!.]9^!@%!\]9^])^A<])^d i f RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 IL The OpenMP Source g-/.a,.!#,-!-./.0010!2#/!"12.3045(#(18!h !!!!!!!!-/9:.4159')8!67./1"5,'('.'\'<8 2#/!59%Nd!9K,d!9@@8 j e !!.]9^!%!NDNd = * !!2#/!5)%Nd!)K(d!)@@8 !!!!!.]9^!@%!\]9^])^A<])^d i !f RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 IM Performance - 2 Socket Nehalem &$!!! #'()*+,- &!!!! %'()*+,-. /'()*+,-. 0'()*+,-. > %$!!! . C #1'()*+,-. 8 4 B @ %!!!! 2 ; ' + A !"#$%"%&#'($)*%$+#,% : , #$!!! -.)/"$#-'%#,%+#0+12% 3 * 4 4.))56(.%#,%7839:% @ ."/"11)1%33333 * #!!!! + -'12 ? $!!! ! !"# # #! #!! #!!! #!!!! #!!!!! #!!!!!! 2+34*5'64478*9:7';<=57+> RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?) A Two Socket Nehalem System ?*4A+..4*'IJ3K+* )D'7)*+,-'! ! A,A)+. A4*+'! 0 + )D'7)*+,-'# . ) 5 4 A )D'7)*+,-'! # * , A,A)+. A4*+'# A 4 L A E ' )D'7)*+,-'# + 7 - 3 7 + ' A % + )D'7)*+,-'! ! * A,A)+. A4*+'% + , 3 )D'7)*+,-'# #! ) : . : )D'7)*+,-'! & A,A)+. A4*+'& 4 )D'7)*+,-'# ## A * + 7 )D'7)*+,-'! / A,A)+. A4*+'! : G #% + )D'7)*+,-'# ' G . ) 5 4 A )D'7)*+,-'! $ * ? , A,A)+. A4*+'# A 4 A E #& ' )D'7)*+,-'# F + - 3 7 + ' 1 + )D'7)*+,-'! # * A,A)+. A4*+'% , 3 )D'7)*+,-'# #/ ) . )D'7)*+,-'! H A,A)+. A4*+'& )D'7)*+,-'# #$ RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?! Data Initialization g-/.a,.!#,-!-./.0010!"12.3045(#(18!h !!!!!!!!67./1"5,'('.'\'<8!-/9:.4159')8 e g-/.a,.!#,-!2#/ j !!!2#/!5)%Nd!)K(d!)@@8 !!!!!!<])^!%!=DNd = g-/.a,.!#,-!2#/ * !!!2#/!59%Nd!9K,d!9@@8 !!!e i !!!!!!.]9^!%!?=POTDNd !!!!!!2#/!5)%Nd!)K(d!)@@8 !!!!!!!!!\]9]^)^!%!9d !!!f!`A??!C("!#2!#,-!2#/!??A` f!`A??!C("!#2!-./.0010!/1a9#(!??A` RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?( Exploit First Touch &$!!! #'()*+,- &!!!! %'()*+,-. /'()*+,-. > %$!!! . C 0'()*+,-. 8 4 B @ %!!!! #1'()*+,-. 2 ; ' + A >+)%-'12%?+"'0)%#,%$+)%@"2% : , #$!!! $+)%5"$"%#,%5#,$/#A($)5%-B)/% 3 ;":%,.))5%(.% * 4 @ $+)%,2,$)& * #!!!! #,%7<3=: + ? $!!! ! !"# # #! #!! #!!! #!!!! #!!!!! #!!!!!! 2+34*5'64478*9:7';<=57+> RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?I Summary Case Studies ! There are several important basic aspects to consider when it comes to writing an efficient OpenMP program ! Moreover, there are also obscure additional aspects: " cc-NUMA " False Sharing ! Key problem is that most developers are not aware of these rules and .... blaming OpenMP is all that easy " In some cases it is a trade-off between ease of use and performance " OpenMP typically goes for the former, but ..... $ With some extra effort can be made to scale well in many cases RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?? The Wrapping RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?J Wrapping Things Up 1W5$8"'<"G:"';#$88'<9$#$%&'?.:'4.7:'*+I'0"=7&':7%'#.'?$%$;56'I' <9%#'#.'9;J'4.7'<5"#5":'4.7'?.7%0'34'$%?.:39#$.%'7;"?78@A 1M";6'$#'$;'.>":<5"83$%&@'I'J%.<@A 12%0'()"%*+'$;';.3"<59#'.=;C7:"'$%'C":#9$%'9:"9;@'I'J%.<' #59#'9;'<"88@A 1I'7%0":;#9%0@'M.7G:"'%.#'9'U.3)7#":'/C$"%#$;#'9%0'N7;#' %""0'#.'&"#'4.7:';C$"%#$?$C':";"9:C5'0.%"@A 1I'9&:""'#5$;'$;'%.#'9'&..0';$#79#$.%6'=7#'$#'$;'988'9=.7#' F9:<$%6'4.7'J%.<@'IG3';.::46'$#'$;'9'#.7&5'<.:80'.7#'#5":"@A RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?A It Never Ends 1(56'4.7:'*+I'N.='N7;#'?$%$;5"0Y'!:"9#@A 1M.7:'):.&:93'0.";'%.#'<:$#"'9'?$8"'C988"0'GC.:"G'9%0'$#' <9;%G#'#5":"'<5"%'4.7';#9:#"0'#5"'):.&:93HA 1M.7'<.%0":'<5":"';7C5'9'?$8"'C.3";'?:.3H'P"#G;'&"#'9'=$&' 9%0';#:.%&'C.??""'?$:;#@A RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ?K That's It Thank You and ..... Stay Tuned ! Ruud van der Pas [email protected] RvdP/V1 Getting OpenMP Up To Speed Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010