Extending and Embedding Perl Extending and Embedding Perl TIM JENNESS SIMON COZENS MANNING Greenwich (74° w. long.) For online information and ordering of this and other Manning books, go to www.manning.com. The publisher offers discounts on this book when ordered in quantity. For more information, please contact: Special Sales Department Manning Publications Co. 209 Bruce Park Avenue Fax: (203) 661-9018 Greenwich, CT 06830 email: [email protected] ©2003 by Manning Publications Co. All rights reserved. No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by means electronic, mechanical, photocopying, or otherwise, without prior written permission of the publisher. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in the book, and Manning Publications was aware of a trademark claim, the designations have been printed in initial caps or all caps. Recognizing the importance of preserving what has been written, it is Manning’s policy to have the books we publish printed on acid-free paper, and we exert our best efforts to that end. Manning Publications Co. Copyeditor: Tiffany Taylor 209 Bruce Park Avenue Typesetter: Dottie Marsico Greenwich, CT 06830 Cover designer: Leslie Haimes ISBN 1930110820 Printed in the United States of America 12345678910–VHG–0605040302 To the Perl open source community contents preface xiii acknowledgments xv about this book xvi author online xix about the cover illustration xx 1 C for Perl programmers 1 1.1 Hello, world 1 1.2 The C compiler 2 1.3 Header files 3 1.4 The main function 4 1.5 Variables and functions 6 Function parameters 6 ✦ Automatic variables 7 Global variables 8 ✦ Static variables 9 1.6 Data types 10 C types 11 ✦ Types defined in Perl 15 1.7 Casting 16 1.8 Control constructs 17 Statements and blocks 17 ✦ The break and continue statements 18 ✦ The switch statement 19 1.9 Macros and the C preprocessor 20 1.10 Library functions 23 1.11 Summary 23 2 Extending Perl: an introduction 24 2.1 Perl modules 24 Module distributions 26 vii 2.2 Interfacing to another language: C from XS 30 The Perl module 30 ✦ The XS file 31 ✦ Example: “Hello, world” 32 Return values 36 ✦ Arguments and return values 37 2.3 XS and C: taking things further 38 Modifying input variables 38 ✦ Output arguments 39 Compiler constants 40 2.4 What about Makefile.PL? 44 It really is a Perl program 47 2.5 Interface design: part 1 47 Status and multiple return arguments 48 ✦ Don’t supply what is already known 48 ✦ Don’t export everything 49 Use namespaces 49 ✦ Use double precision 49 2.6 Further reading 50 2.7 Summary 50 3Advanced C51 3.1 Arrays 51 3.2 Pointers 53 Pointers and arrays 55 ✦ Pointers to functions 57 3.3 Strings 58 Arrays of strings 59 3.4 Structures 60 3.5 File I/O 62 3.6 Memory management 63 Allocating memory at runtime 64 ✦ Altering the size of memory 65 Manipulating memory 65 ✦ Memory manipulation and Perl 67 3.7 C Traps for the Perl programmer 68 3.8 Further reading 69 3.9 Summary 69 4Perl’s variable types70 4.1 General concepts 70 Reference counting 71 ✦ Looking inside: Devel::Peek 71 ✦ The flag system 72 4.2 Scalar variables 74 The SvNULL type 74 ✦ SvRV: references 76 ✦ SvPV: string values 76 ✦ SvPVIV: integers 78 ✦ SvPVNV: floating-point numbers 79 ✦ SvIV and SvNV 80 ✦ SvOOK: offset strings 80 4.3 Magic variables: SvPVMG 81 viii CONTENTS 4.4 Array variables 85 4.5 Hashes 87 4.6 Globs 91 4.7 Namespaces and stashes 94 4.8 Lexical “my” variables 95 4.9 Code blocks 96 Important CV flags 97 4.10 Further reading 99 4.11 Summary 99 5 The Perl 5 API 100 5.1 Sample entry 101 5.2 SV functions 101 Special SVs 101 ✦ Creating SVs 103 ✦ Accessing data 110 Manipulating data 119 ✦ String functions 124 ✦ References 129 5.3 AV functions 132 Creation and destruction 132 ✦ Manipulating elements 136 Testing and changing array size 142 5.4 HV functions 144 Creation and destruction 144 ✦ Manipulating elements 146 5.5 Miscellaneous functions 150 Memory management 150 ✦ Unicode data handling 155 Everything else 158 5.6 Summary 162 6 Advanced XS programming 163 6.1 Pointers and things 164 6.2 Filehandles 166 6.3 Typemaps 167 6.4 The argument stack 169 6.5 C structures 170 C structures as black boxes 170 ✦ C structures as objects 176 C structures as hashes 179 6.6 Arrays 183 Passing numeric arrays from Perl to C 183 ✦ Passing numeric arrays from C to Perl 190 ✦ The Perl Data Language 192 Benchmarks 198 ✦ Character strings 199 CONTENTS ix 6.7 Callbacks 202 Immediate callbacks 203 ✦ Deferred callbacks 206 ✦ Multiple callbacks 207 6.8 Other languages 209 Linking Perl to C++ 209 ✦ Linking Perl to Fortran 216 Linking Perl to Java 223 6.9 Interface design: part 2 223 6.10 Older Perls 224 6.11 What’s really going on? 225 What does xsubpp generate? 226 6.12 Further reading 230 6.13 Summary 230 7 Alternatives to XS 231 7.1 The h2xs program 232 7.2 SWIG 233 Data structures 236 7.3 The Inline module 238 What is going on? 239 ✦ Additional Inline examples 240 Inline and CPAN 245 ✦ Inline module summary 246 7.4 The PDL::PP module 247 The .pd file 248 ✦ The Makefile.PL file 249 ✦ Pure PDL 251 7.5 Earlier alternatives 251 7.6 Further reading 252 7.7 Summary 253 8 Embedding Perl in C 254 8.1 When to embed 254 8.2 When not to embed 255 8.3 Things to think about 255 8.4 “Hello C” from Perl 255 Compiling embedded programs 257 8.5 Passing data 257 8.6 Calling Perl routines 259 Stack manipulation 261 ✦ Context 263 ✦ Trapping errors with eval 263 ✦ Calling Perl methods in C 264 ✦ Calling Perl statements 265 8.7 Using C in Perl in C 265 8.8 Embedding wisdom 266 8.9 Summary 267 x CONTENTS 9 Embedding case study 268 9.1 Goals 268 9.2 Preparing the ground 269 9.3 Configuration options 270 9.4 Testing options 273 Binary options 273 ✦ Quad-state options 274 ✦ String options 275 9.5 Summary 276 10 Introduction to Perl internals 277 10.1 The source tree 277 The Perl library 277 ✦ The XS library 278 ✦ The I/O subsystem 278 The Regexp engine 278 ✦ The parser and tokenizer 278 ✦ Variable handling 279 ✦ Runtime execution 279 10.2 The parser 279 BNF and parsing 279 ✦ Parse actions and token values 281 Parsing some Perl 281 10.3 The tokenizer 282 Basic tokenizing 282 ✦ Sublexing 284 ✦ Tokenizer summary 285 10.4 Op code trees 285 The basic op 285 ✦ The different operations 286 ✦ Different flavors of ops 286 ✦ Tying it all together 288 ✦ PP Code 290 The opcode table and opcodes.pl 293 ✦ Scratchpads and targets 293 The optimizer 294 ✦ Op code trees summary 294 10.5 Execution 295 10.6 The Perl compiler 295 What is the Perl compiler? 296 ✦ B:: modules 296 ✦ What B and O provide 299 ✦ Using B for simple tasks 300 10.7 Further reading 303 10.8 Summary 303 11 Hacking Perl 304 11.1 The development process 304 Perl versioning 304 ✦ The development tracks 305 ✦ The perl5-porters mailing list 305 ✦ Pumpkins and pumpkings 305 ✦ The Perl repository 306 11.2 Debugging aids 306 Debugging modules 307 ✦ The built-in debugger: perl -D 307 Debugging functions 310 ✦ External debuggers 310 11.3 Creating a patch 317 How to solve problems 317 ✦ Autogenerated files 318 ✦ The patch itself 319 Documentation 320 ✦ Testing 320 ✦ Submitting your patch 320 CONTENTS xi 11.4 Perl 6: the future of Perl 321 A history 321 ✦ Design and implementation 322 What happens next 323 ✦ The future for Perl 5 323 11.5 Further reading 323 11.6 Summary 323 A: Perl’s typemaps 324 B: Further reading 348 C: Perl API index 350 index 355 xii CONTENTS preface Perl is a wonderful language. We, along with at least a million other programmers, love it dearly. It’s great for all kinds of applications: text processing, network programming, system administra- tion, and much more. But there are times when we need to go beyond the core of the language and do something not provided by Perl. Sometimes we do this without noticing it: several modules that ship with Perl call out to C routines to get their work done, as do some of the most commonly used CPAN modules. Other times, we do it deliberately: the various modules for building graphical applications in Perl almost all, directly or indirectly, use external C libraries. Either way, writing extensions to Perl has his- torically been a bit of a black art. We don’t believe this situation is fair, so we’ve written this book to attempt to demystify the process of relating Perl and C. That’s not to say we’re fed up with writing modules in Perl. Both of us write many of our mod- ules in Perl, and although sometimes it might be easier to interface to C, we’ve decided to stick with Perl. In fact, writing a module in Perl has a number of advantages over using other languages: • It is far easier to write portable cross-platform code in Perl than in C. One of the successes of Perl has been its support of varied operating systems.