Patterns of Parallel Programming
Total Page：16
File Type：pdf, Size：1020Kb
PATTERNS OF PARALLEL PROGRAMMING UNDERSTANDING AND APPLYING PARALLEL PATTERNS WITH THE .NET FRAMEWORK 4 AND VISUAL C# Stephen Toub Parallel Computing Platform Microsoft Corporation Abstract This document provides an in-depth tour of support in the Microsoft® .NET Framework 4 for parallel programming. This includes an examination of common parallel patterns and how they’re implemented without and with this new support, as well as best practices for developing parallel components utilizing parallel patterns. Last Updated: July 1, 2010 This material is provided for informational purposes only. Microsoft makes no warranties, express or implied. ©2010 Microsoft Corporation. TABLE OF CONTENTS Introduction ................................................................................................................................................................... 3 Delightfully Parallel Loops ............................................................................................................................................. 4 Fork/Join ...................................................................................................................................................................... 36 Passing Data ................................................................................................................................................................. 49 Producer/Consumer .................................................................................................................................................... 53 Aggregations ................................................................................................................................................................ 67 MapReduce .................................................................................................................................................................. 75 Dependencies .............................................................................................................................................................. 77 Data Sets of Unknown Size .......................................................................................................................................... 88 Speculative Processing ................................................................................................................................................ 94 Laziness ........................................................................................................................................................................ 97 Shared State .............................................................................................................................................................. 105 Conclusion ................................................................................................................................................................. 118 Patterns of Parallel Programming Page 2 INTRODUCTION Patterns are everywhere, yielding software development best practices and helping to seed new generations of developers with immediate knowledge of established directions on a wide array of problem spaces. Patterns represent successful (or in the case of anti-patterns, unsuccessful) repeated and common solutions developers have applied time and again in particular architectural and programming domains. Over time, these tried and true practices find themselves with names, stature, and variations, helping further to proliferate their application and to jumpstart many a project. Patterns don’t just manifest at the macro level. Whereas design patterns typically cover architectural structure or methodologies, coding patterns and building blocks also emerge, representing typical ways of implementing a specific mechanism. Such patterns typically become ingrained in our psyche, and we code with them on a daily basis without even thinking about it. These patterns represent solutions to common tasks we encounter repeatedly. Of course, finding good patterns can happen only after many successful and failed attempts at solutions. Thus for new problem spaces, it can take some time for them to gain a reputation. Such is where our industry lies today with regards to patterns for parallel programming. While developers in high-performance computing have had to develop solutions for supercomputers and clusters for decades, the need for such experiences has only recently found its way to personal computing, as multi-core machines have become the norm for everyday users. As we move forward with multi-core into the manycore era, ensuring that all software is written with as much parallelism and scalability in mind is crucial to the future of the computing industry. This makes patterns in the parallel computing space critical to that same future. “In general, a ‘multi-core’ chip refers to eight or fewer homogeneous cores in one microprocessor package, whereas a ‘manycore’ chip has more than eight possibly heterogeneous cores in one microprocessor package. In a manycore system, all cores share the resources and services, including memory and disk access, provided by the operating system.” –The Manycore Shift, (Microsoft Corp., 2007) In the .NET Framework 4, a slew of new support has been added to handle common needs in parallel programming, to help developers tackle the difficult problem that is programming for multi-core and manycore. Parallel programming is difficult for many reasons and is fraught with perils most developers haven’t had to experience. Issues of races, deadlocks, livelocks, priority inversions, two-step dances, and lock convoys typically have no place in a sequential world, and avoiding such issues makes quality patterns all the more important. This new support in the .NET Framework 4 provides support for key parallel patterns along with building blocks to help enable implementations of new ones that arise. To that end, this document provides an in-depth tour of support in the .NET Framework 4 for parallel programming, common parallel patterns and how they’re implemented without and with this new support, and best practices for developing parallel components in this brave new world. This document only minimally covers the subject of asynchrony for scalable, I/O-bound applications: instead, it focuses predominantly on applications of CPU-bound workloads and of workloads with a balance of both CPU and I/O activity. This document also does not cover Visual F# in Visual Studio 2010, which includes language-based support for several key parallel patterns. Patterns of Parallel Programming Page 3 DELIGHTFULLY PARALLEL LOOPS Arguably the most well-known parallel pattern is that befitting “Embarrassingly Parallel” algorithms. Programs that fit this pattern are able to run well in parallel because the many individual operations being performed may operate in relative independence, with few or no dependencies between operations such that they can be carried out in parallel efficiently. It’s unfortunate that the “embarrassing” moniker has been applied to such programs, as there’s nothing at all embarrassing about them. In fact, if more algorithms and problem domains mapped to the embarrassing parallel domain, the software industry would be in a much better state of affairs. For this reason, many folks have started using alternative names for this pattern, such as “conveniently parallel,” “pleasantly parallel,” and “delightfully parallel,” in order to exemplify the true nature of these problems. If you find yourself trying to parallelize a problem that fits this pattern, consider yourself fortunate, and expect that your parallelization job will be much easier than it otherwise could have been, potentially even a “delightful” activity. A significant majority of the work in many applications and algorithms is done through loop control constructs. Loops, after all, often enable the application to execute a set of instructions over and over, applying logic to discrete entities, whether those entities are integral values, such as in the case of a for loop, or sets of data, such as in the case of a for each loop. Many languages have built-in control constructs for these kinds of loops, Microsoft Visual C#® and Microsoft Visual Basic® being among them, the former with for and foreach keywords, and the latter with For and For Each keywords. For problems that may be considered delightfully parallel, the entities to be processed by individual iterations of the loops may execute concurrently: thus, we need a mechanism to enable such parallel processing. IMPLEMENTING A PARALLEL LOOPING CONSTRUCT As delightfully parallel loops are such a predominant pattern, it’s really important to understand the ins and outs of how they work, and all of the tradeoffs implicit to the pattern. To understand these concepts further, we’ll build a simple parallelized loop using support in the .NET Framework 3.5, prior to the inclusion of the more comprehensive parallelization support introduced in the .NET Framework 4. First, we need a signature. To parallelize a for loop, we’ll implement a method that takes three parameters: a lower-bound, an upper-bound, and a delegate for the loop body that accepts as a parameter an integral value to represent the current iteration index (that delegate will be invoked once for each iteration). Note that we have several options for the behavior of these parameters. With C# and Visual Basic, the vast majority of for loops are written in a manner similar to the following: