PDL | Scientific Programming in Perl Karl Glazebrook Christian S¨oller Tuomas J. Lukka Marc Lehmann Jarle Brinchmann Doug Hunt John Cerney Robin Williams Tim Pickering August 12, 2001 Contents 0 PDL | Year Zero 5 0.1 Who this book is for. 6 0.2 The case for a high-level approach. 7 0.3 The case for a free Data Language. 8 0.4 So why Perl? . 9 0.5 What this book is about. 11 0.6 What this book is NOT about. 11 0.7 Conventions used in this book . 12 1 A Whirlwind tour through PDL 13 1.1 Alright let's do something. 13 1.2 Whirling through the Whirlpool . 16 1.2.1 Measuring the brightness of M51 . 20 1.2.2 Twinkle, twinkle, little star. 21 1.2.3 Getting Complex with M51 . 28 1.3 Roundoff . 29 2 The Basics of PDL 31 2.1 Basic manipulations of piddles . 31 2.1.1 What is a piddle? . 31 2.1.2 Creating piddles . 32 2.1.3 Elementary arithmetic. 33 2.1.4 In-place arithmetic . 34 2.1.5 Using = and .= . 34 2.2 Piddles are NOT Perl `arrays' . 35 1 CONTENTS 2 2.2.1 The benefits of piddles. 37 2.2.2 Lists and Hashes of piddles . 37 2.3 PDL Datatypes . 38 2.3.1 Type conversion and creation . 39 2.3.2 Complex types . 39 2.4 PDL projection operators . 40 2.4.1 What about projecting along other dimensions? . 41 2.5 `Threading' over extra dimensions. 41 2.6 Example Problem . 43 3 Slicing, Dicing and Threading with PDL dimensions 44 3.1 Finding piddle dimensions. 45 3.2 The slice function | regular subsets along axes . 45 3.2.1 The basic slicing specification. 45 3.2.2 Modifying slices. 46 3.2.3 Does a slice consume memory? . 47 3.2.4 Advanced slice syntax . 48 3.2.5 PDL's Method notation . 48 3.3 The dice and dice_axis functions | irregular subsets along axes 49 3.4 Using mv, xchg and reorder | transposing dimensions . 50 3.5 Combining dimensions with clump . 50 3.6 Adding dimensions with dummy ................... 52 3.7 Completely general subsets of data with index, which and where 53 3.8 Example Problem . 55 3.9 PDL threading and signatures . 55 3.9.1 Threading . 56 3.9.2 A simple example . 56 3.9.3 Why bother . 58 3.9.4 More examples . 58 3.9.5 Why threading and why call it threading? . 60 3.9.6 The general case: PDL functions and their signature . 63 3.9.7 You can write your own threading routines . 67 3.9.8 Matching threading dimensions . 68 3.9.9 Promotion: When a piddle has less dimensions than required 72 CONTENTS 3 3.9.10 Calling conventions and tight loops . 72 3.9.11 Manipulating indices | generalised slicing . 72 3.9.12 Memory and speed implications | Vaffines . 72 3.9.13 writing your own threading aware function | at the perl level! . 72 3.9.14 A worked example . 72 3.9.15 explicit threading for complex cases . 73 4 Graphics with PDL 74 4.1 Introduction . 74 4.2 2-dimensional graphics using PGPLOT . 75 4.3 Introducing PDL::Graphics::PGPLOT . 75 4.4 An overview of 2D plotting commands . 77 4.4.1 Setting up the plot area . 77 4.4.2 Drawing lines and plotting points . 78 4.4.3 Displaying images . 79 4.4.4 Contour plots and vector fields . 81 4.4.5 Using colour . 82 4.4.6 Using PGPLOT commands directly . 83 4.4.7 Plotting commands . 84 4.5 Plot command reference . 85 4.5.1 General options . 85 4.5.2 Graphics commands in PDL::Graphics::PGPLOT . 87 4.6 3D Graphics with OpenGL . 90 4.6.1 Introduction . 90 4.6.2 Parametric Graphics . 91 4.6.3 Types of 3D Graphical Objects . 93 4.6.4 More than one Image . 95 4.6.5 Animation . 97 4.6.6 Putting it all together | cool hacks . 97 4.6.7 The VRML backend . 99 4.7 Pixel manipulations using the Gimp . 100 4.7.1 The components of Gimp . 100 4.7.2 Direct pixel access . 100 CONTENTS 4 4.7.3 Gimp Plug-Ins . 100 4.7.4 More Information . 101 6 Advanced Examples of Using PDL 102 6.1 A PDL weather map . 102 6.1.1 Drawing the map . 102 6.1.2 Getting the weather data . 104 6.1.3 Plotting the contour map . 105 6.1.4 What happened? . 106 6.2 Objects using PDL - defining your own data types . 109 6.2.1 Implementing derived classes: The Complex module . 110 6.3 Numerical simulations in PDL . 116 6.3.1 The wave equation . 116 6.3.2 Multiple dimensions and higher order . 120 6.3.3 Gas dynamics . 126 6.3.4 Ordinary Differential Equations . 126 Chapter 0 PDL | Year Zero "Why is it that we entertain the belief that for every purpose odd numbers are the most effectual?” - Pliny the Elder. The PDL project began in February 1996, when I decided to experiment with writing my own `Data Language'. I am an astronomer. My day job involves a lot of analysis of digital data accumualated on many nights observing on telescopes around the world. Such data might for example be images containing millions of pixels and thousands of images of distant stars and galaxies. Or more abstrusely, many hundreds of digital spectral revealing the secrets of the composition and propertues of these distant objects. Obviously many astronomers before have dealt with these problems, and a large amount of software has been constructed to facilitate their analysis. However, like many of my colleagues, I was constantly frustrated by the lack of generality and flexibility of these programs and the difficulty of doing anything out of the ordinary quickly and easily. What I wanted had a name: `Data Language', i.e. a language which allowed the manipulation of large amounts of data with simple arithmetic expressions. In fact some commericial software worked like this, and I was impressed with the capabilities but not with the price tag. And I thought I could do better. As a fairly computer literate astronomer (read `nerd' or `geek' according to your local argot) I was very familiar with `Perl', a computer language which now seems to fill the shelves of many bookstores around the world. I was impressed by it's power and flexibility, and especially it's ease of use. I had even explored the depths of it's internals and written an interface to allow graphics (the PGPLOT module1). The ease with which I could then create charts and graphs, for my papers, was refreshing. 1The PGPLOT module for perl is an interface to the pgplot graphics library (written in C and Fortran) created by Tim Pearson of Caltech. More information about this library can be obtained from: http://astro.caltech.edu/~tjp/pgplot/ 5 CHAPTER 0. PDL | YEAR ZERO 6 Version 5 of Perl had just been released, and I was fascinated by the new fea- tures available. Especially the support of arbitrary data structures (or `objects' in modern parlance) and the ability to `overload' operators | i.e. make math- ematical symbols like +-*/ do whatever you felt like. It seemed to me it ought to be possible to write an extension to Perl where I could play with my data in a general way: for example using the maths operators manipulate whole images at once. Well one slow night at an observatory I just thought I would try a little exper- iment. In a bored moment I fired up a text editor and started to create a file called `PDL.xs' | a Perl extension module to manipulate data vectors. A few hours later I actually had something half decent working, where I could add two images in the Perl language, fast! This was something I could not let rest, and it probably cost me one or two scientific papers worth of productivity. A few weeks later the Perl Data Language version 1.0 was born. It was a pretty bare infant: very little was there apart from the basic arithmetic operators. But encouraged I made it available on the Internet to see what people thought. Well people were fairly critical | among the most vocal were Tuomas Lukka and Christian Soeller. Unfortunately for them they were both Perl enthusiasts too and soon found themselves improving my code to implement all the features they thought PDL ought to have and I had heinously neglected. PDL is a prime example of that modern phenomenon of authoring large free software packages via the Internet. Large numbers of people, most of whom have never met, have made contributions ranging for core functionality to large modules to the smallest of bug patches. PDL version 2.0 is now here (though it should perhaps have been called version 10 to reflect the amount of growth in size and functionality) and the phenomenon continues. I firmly believe that PDL is a great tool for tackling general problems of data analysis. It is powerful, fast, easy to add too and freely available to anyone. I wish I had had it when I was a graduate student! I hope you too will find it of immense value, I hope it will save you from heaps of time and frustration in solving complex problems. Of course it can't do everything, but it provides the framework, the hammers and the nails for building solutions without having to reinvent wheels or levers.