Introduction to Parallel Programming in OpenMP David Colignon, ULg CÉCI - Consortium des Équipements de Calcul Intensif http://www.ceci-hpc.be Main References • “Parallel Programming with GCC”, Diego Novillo, Red Hat Red Hat Summit, Nashville, May 2006 http://www.airs.com/dnovillo/Papers/rhs2006.pdf • "An Overview of OpenMP", Ruud van der Pas, Oracle IWOMP 2010, Tsukuba, 14-16 June 2010 http://www.compunity.org/training/tutorials/3 Overview_OpenMP.pdf and http://openmp.org/wp/2010/07/iwomp-2010-material-available/ More References: Specification OpenMP, The OpenMP API specification for parallel programming http://openmp.org/ Articles Wikipedia (good summary) http://en.wikipedia.org/wiki/Openmp 32 OpenMP traps for C++ developers http://software.intel.com/en-us/articles/32-openmp-traps-for-c-developers/ Common Mistakes in OpenMP and How To Avoid Them http://www.michaelsuess.net/.../suess_leopold_common_mistakes_06.pdf IWOMP 2009, The 2009 International Workshop on OpenMP (Slides) http://openmp.org/wp/2009/06/iwomp2009/ IWOMP 2010, The 2010 International Workshop on OpenMP (Slides) http://openmp.org/wp/2010/07/iwomp-2010-material-available/ Avoiding and Identifying False Sharing Among Threads http://software.intel.com/en-us/articles/avoiding-and-identifying-false-sharing Tutorials Parallel Programming with GCC, D. Novillo, Red Hat Summit, Nashville, May 2006 http://www.airs.com/dnovillo/Papers/rhs2006.pdf An Overview of OpenMP, IWOMP 2010, Ruud van der Pas, Oracle http://www.compunity.org/training/tutorials/3 Overview_OpenMP.pdf A "Hands-on" Introduction to OpenMP, SC08, Mattsonand Meadows, Intel http://www.openmp.org/mp-documents/omp-hands-on-SC08.pdf Cours OpenMP (en français !) de l'IDRIS http://www.idris.fr/data/cours/parallel/openmp/ Using OpenMP, SC09, Hartman-Baker R., ORNL, NCCS http://www.greatlakesconsortium.org/events/scaling/files/openmp09.pdf OpenMP Tutorial, Barney B., LLNL https://computing.llnl.gov/tutorials/openMP/ Books Using OpenMP - Portable Shared Memory Parallel Programming, by Chapman et al. https://mitpress.mit.edu/books/using-openmp (Download Book Examples and Discuss) http://openmp.org/wp/2009/04/download-book-examples-and-discuss/ Parallel Programming in OpenMP, by Rohit Chandra et al. http://www.elsevier.com/wps/find/bookdescription.cws_home/677929/description (Google Preview) http://books.google.be/books?id=18CmnqIhbhUC Outline • Introduction to parallel computing • Parallel programming models - Shared memory - Message passing • OpenMP - Guided tour - In depth overview +","--.-$/01234567 ! !"#$%&'()&'#$*+,*-''#,*.$/+'$0,*'#&"#( " 1#'/+'2&,*# " 1'+34#2$"05# ! 6)+$2&0,$2+(#4" " 7%&'#($2#2+'. " 80"9'03-9#($2#2+'. ! :&9-'#$+/$;'+34#2$(0*9&9#" " <+2;-9&90+,=*+22-,0*&90+,$'&90+ " >&'()&'#$'#?-0'#2#,9" !"#$%&'$())* % ,-./0-12/34$!3560# =.,)'#(..)#, >?@ >?@ >?@ >?@ >?@ >?@ A)0('1 A)0('1 A)0('1 A)0('1 A)0('1 A)0('1 ! !"#$%&'(#)**('%$"*%+,*%(-.%&'+/",)%0)0('1 ! !2&3+#+,%#(004.+#",+(. ! !2&3+#+,%*1.#$'(.+5",+(. ! 6+77+#43,%,(%&'(8'"0%94,%.(:7)-%$+;;).%*+;)<)77)#,* !"#$%&'$())* + ,-"./0$!/12.# !"#$%& '()"%*$(("*) +,- +,- +,- +,- +,- +,- ! ,%$*"..$%./.01%"/*$##$(/#"#$%& ! '#234*4)/*$##5(4*1)4$( ! 67234*4)/.&(*0%$(481)4$( ! 94#23"/)$/2%$:%1#/;5)/04<<"(/.4<"=">>"*). !"#$%&'$())* + +,-.,"//01.$!-2345 ! !"#$%&'()*+$),-+%&./%/0$1.02+%3.40/,)3%& " 5%+60$7*.02./-8+)940$%.30&%* " :#$#88%8)*/.#;#)8#,8%.#+.;#$)0-*.8%;%8* ! <&&)+)03#8.$%=-)$%/%3+*.0;%$.*%=-%3+)#8 " >#*7.4$%#+)03 " ?0//-3)4#+)03 " !134"$03)@#+)03 ! A06.&0.6%.B$0C$#/.+"%*%.*1*+%/*D !"#$%&'$())* * ,-./0102$3"4"//5/067 ! !"#$%&'()$'*"+%,-"."/%&'001(2&-)2'(%-(3% "4(&5$'(26-)2'( ! 7(&$#-"#3%8$'9$-002(9%&'08*#:2)4 " ;<)#(%$#=12$#%32<<#$#()%-*9'$2)50" ! >-(4%32<<#$#()%-88$'-&5#" " ?-$-**#*%*-(91-9#"%'$%*-(91-9#%#:)#("2'("+%@?A/% ;&&-0/%B-C- " D'082*#$%-((')-)2'("+%;8#(>? " E2F$-$2#"+%?)5$#-3"/%>?7 !"#$%&'$())* + ,"-"../.012$03$455 !""#$%&&'()$#*'%(#+',+%((-,+.#/'0-1$ 4433$$.. 2233((00 ##00"" ((&&))**++,,--..//**--++'' $$%%&&''!!"" !"!"## ! 33%%))''//33))55++ ! 33%%))''//33))55++ ! /3/3,,%%3311 ! //33,,%%3311 ! ,,''##%%$$--((##++'',,))((''11 ! ++''//&&51-51-((##''&&))5'5',, ! ++''//&&51-(51-(##005(5(--++))5566--$$ ! $$&&--++55331#1#159159((33((5-5-$$ ! ,,''))##5,5,))((%%$$5656-- ! ,,''))##5,5,))((%%$$5656-- ! $$''//--778833))##55,,))((%$%$5656-- ! 66--((..##55,,))((%%$$5656-- 11//22&&33++44335522&&33''++**33''&&))&&2222//,,--667733,,&&66//**&&8833**++3322%%&&&&8855%%2299 !"#$%&'$())* + !+,,"-+$.",,/0- ! !"#$%&'&%()%*+,-,()+-.&/ ! 0").$&1*-%)1"#$*%&,).2$$",'),&32*,&/ ! 45&)6-..&#+%()%-782-8&9)":)$-,-%%&%) $,"8,-##*78 " ;%'*#-'&)1"7',"% " ;%'*#-'&)$-*7)<5&7)'5*78.)8")<,"78 " !"#$2'-'*"7=1"##27*1-'*"7),-'*")#2.')+&)5*85 ! >&..-8&)?-..*78)@7'&,:-1&)A>?@B)#".') $"$2%-,)#"/&% !"#$%&'$())* %* !,--".,$/"--01. ! !"#$%$&"'$((%"))')#$*") " +&',$-'$.)/'0"'1)"('/2'$')3$%"(',",/%-' ,$*342" ! 5"$6-'7"483&'#%/*"))") ! 9/,,124*$&4/2'":#.4*4&'64$'2"&7/%;' ,"))$8") " <)"%'%")#/2)40."'=/%',$%)3$..428>')"2(428'$2(' %"*"46428 !"#$%&'$())* %+ ,-./!0$1$2/345678395/ ! !"#$%"$&'&()&#*+,#*'-,.'*/".&0'1&1,.2' 3,#3%..&#32 ! 4%55,.)*'67'688'"#0'9,.)."# ! :1;&00&0'0+.&3)+<&*'*5&3+-2 " =".">>&>+*1 " ?")"'*/".+#$'*&1"#)+3* " @,.A'*/".+#$'*&1"#)+3* ! 4)"#0".0'"#0'+#3.&"*+#$>2'5,5%>". !"#$%&'$())* &+ ,-./!0$1$02342"556/4$!37.8 ! !"#$%&'(&)'*+,-'.(&#$/"(0.1# " 2"#0$*&03*$"%&#4"5(#&0$"/#&')&13.6%*$(&03*$"%# " 766&03*$"%#&#3"*$&1'//'(&/$/'*8 ! 766'5#&#$9:$(0."6&"(%&4"*"66$6&$;$1:0.'( 2"#0$*& 03*$"% )'*+ -'.( <"*"66$6&*$=.'( !"#$%&'$())* &+ +,-.!/$0$/1231"445.3$!26-7 ! !"#$%&'()*%('+,%-'.)-%/)$(/0#/.)1!2)!334)"() +"##'5,.)16"(,(/547 ! !"#$%&'()('$&/+'.)*%('+,%-'.)8%,9)+/&&.),") (:5,%#')&%;(/(<)1!"#$%&'4 ! =:5,%#')+"5,("&.)/-/%&/;&')-%/)&%;(/(<)>?@) /5*)'5-%("5#'5,)-/(%/;&'. ! A5-%("5#'5,)-/(%/;&'.)+"5,("&)$/(/&&'&%.# !"#$%&"$'()*+,- !"#$-.(*,&/* !"#$,0%+"1. !"#$%*-'*, !"#$%&'$())* &* ,-./!0$1$02342"556/4$!37.8 ! !"#$%&%'()*+,%-.(+-/()0-&*,1-%2+'%1- ! 3*,4+/)(%-'4,+&'(5%+()*+,4/(5+,%+6$4) " 7454,+$(8+0)(91,()#4&%90%-.()*+,4/(/+'+ " 7*+,%-.(+$8+0)(+'('*4(5+,%+6$4($454$ ! :,1.,+;;4,(,4)#1-)%6$4(91,()0-&*,1-%2+'%1- " <-%-'4-/4/()*+,%-.($4+/)('1(=/+'+(,+&4)> " <)4()0-&*,1-%2+'%1-(/%,4&'%54)(+-/($%6,+,0(?:@ " 70-&*,1-%2+'%1-(%)(4"#4-)%54 !"#$%&'$())* &+ ! !"#$%&'%(&)#*+#$,&"-. B99C'0<.'C12'&<3 +1./62'+4<77'D.E/.112 $2<FG1'+6G<2/3'+49C/6 $2<FG1 %1.G6'&<2:,'*H,'-+H "#$%&'()!) **+,'-./0123/45'67'839:9;< 839:9;<,'=<><. =9.1'!?@!A,'()!) RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ! Getting Started with OpenMP RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 " http://www.openmp.org http://www.compunity.org RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 # $%%&'(()))*+&,-.&*+/0 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 1 Shameless Plug - “Using OpenMP” !"#$%&'()*%+,- ,./0123*'451/*6'+*7./8' ,1/133*3',/.&/177$%& 951)71%:';.#0:'<1%'6*/',1# 23456/,77859::; 3<=>?@:' :?919?#!!:9?9 3<=>?@!' AB;?:?919?#!!:9?B CD7%5&/DE,'5!#5FG< RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 ; What is OpenMP? ! De-facto standard Application Programming Interface (API) to write shared memory parallel applications in C, C++, and Fortran ! Consists of: " Compiler directives " Run time routines " Environment variables ! Specification maintained by the OpenMP Architecture Review Board (http://www.openmp.org) ! Version 3.0 has been released May 2008 RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 A When to consider OpenMP? ! Using an automatically parallelizing compiler: " It can not find the parallelism # The data dependence analysis is not able to determine whether it is safe to parallelize or not " The granularity is not high enough # The compiler lacks information to parallelize at the highest possible level ! Not using an automatically parallelizing compiler: " No choice than doing it yourself RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @: Advantages of OpenMP ! Good performance and scalability " If you do it right .... ! De-facto and mature standard ! An OpenMP program is portable " Supported by a large number of compilers ! Requires little programming effort ! Allows the program to be parallelized incrementally RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 @@ OpenMP and Multicore OpenMP is ideally suited for multicore architectures Memory and threading model map naturally Lightweight Mature Widely available and used RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 The OpenMP Execution Model !" !"#$%&'(%)"*'%+"(,- Master Thread Parallel region Worker Threads Synchronization Worker Parallel region Threads Synchronization RvdP/V1 An Overview of OpenMP Tutorial IWOMP 2010 – CCS Un. of Tsukuba, June 14, 2010 *+,#&--) A process is created by the operating system, and requires a fair amount of "overhead". Processes contain information about program resources and program execution state, including: !"#$%&'((")*+",$%&'(("-$%.,")*+".('$")*+"/01"-$%.,")*"" !"2034$%05'06"" !"7%$840-"14$'&6%$9:"" !"#$%-$/5"40(6$.&64%0("" !";'-4(6'$(""" !"<6/&8"" !"='/,"" !">4?'"1'(&$4,6%$("" !"<4-0/?"/&64%0("" !"<@/$'1"?4A$/$4'("" !")06'$!,$%&'(("&%55.04&/64%0"6%%?("B(.&@"/("5'((/-'"C.'.'(+",4,'(+" ('5/,@%$'(+"%$"(@/$'1"5'5%$9D:"" !!!"#$%&#'"$() !" *+,&'-) A thread is defined as an independent stream of instructions that can be scheduled to run as such by the operating system. Threads use and exist within the process