SECOND EDITION Mastering Perl brian d foy Mastering Perl, Second Edition by brian d foy Copyright © 2014 brian d foy. All rights reserved. Printed in the United States of America. Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472. O’Reilly books may be purchased for educational, business, or sales promotional use. Online editions are also available for most titles (http://my.safaribooksonline.com). For more information, contact our corporate/ institutional sales department: 800-998-9938 or [email protected]. Editor: Rachel Roumeliotis Indexer: Lucie Haskins Production Editor: Kara Ebrahim Cover Designer: Randy Comer Copyeditor: Becca Freed Interior Designer: David Futato Proofreader: Charles Roumeliotis Illustrator: Rebecca Demarest January 2014: Second Edition Revision History for the Second Edition: 2014-01-08: First release See http://oreilly.com/catalog/errata.csp?isbn=9781449393113 for release details. Nutshell Handbook, the Nutshell Handbook logo, and the O’Reilly logo are registered trademarks of O’Reilly Media, Inc. Mastering Perl, Second Edition, the image of a vicuña and her young, and related trade dress are trademarks of O’Reilly Media, Inc. Many of the designations used by manufacturers and sellers to distinguish their products are claimed as trademarks. Where those designations appear in this book, and O’Reilly Media, Inc., was aware of a trade‐ mark claim, the designations have been printed in caps or initial caps. While every precaution has been taken in the preparation of this book, the publisher and authors assume no responsibility for errors or omissions, or for damages resulting from the use of the information contained herein. ISBN: 978-1-449-39311-3 [LSI] Table of Contents Preface. xi 1. Advanced Regular Expressions. 1 Readable Regexes, /x and (?#…) 1 Global Matching 3 Global Match Anchors 5 Recursive Regular Expressions 7 Repeating a Subpattern 7 Lookarounds 18 Lookahead Assertions, (?=PATTERN) and (?!PATTERN) 18 Lookbehind Assertions, (?<!PATTERN) and (?<=PATTERN) 22 Debugging Regular Expressions 25 The -D Switch 25 Summary 29 Further Reading 29 2. Secure Programming Techniques. 31 Bad Data Can Ruin Your Day 31 Taint Checking 32 Warnings Instead of Fatal Errors 34 Automatic Taint Mode 35 mod_perl 35 Tainted Data 35 Side Effects of Taint Checking 36 Untainting Data 37 IO::Handle::untaint 39 Hash Keys 40 Taint::Util 41 Choosing Untainted Data with Tainted Data 41 iii Symbolic References 42 Defensive Database Programming with DBI 45 List Forms of system and exec 47 Three-Argument open 48 sysopen 49 Limit Special Privileges 49 Safe Compartments 50 Safe Limitations 56 A Little Fun 56 Summary 57 Further Reading 58 3. Perl Debuggers. 59 Before You Waste Too Much Time 59 The Best Debugger in the World 60 Safely Changing Modules 61 Wrapping Subroutines 62 The Perl Debugger 65 Alternative Debuggers 66 Using a Different Debugger with -d 66 Devel::ptkdb 66 Devel::ebug 68 Devel::hdb 69 IDE Debuggers 70 EPIC 70 Komodo 70 Summary 70 Further Reading 71 4. Profiling Perl. 73 Finding the Culprit 73 The General Approach 77 Profiling DBI 79 Other DBI::Profile Reports 83 Making It Even Easier 84 Switching Databases 85 Devel::NYTProf 87 Writing My Own Profiler 88 Devel::LineCounter 88 Profiling Test Suites 89 Devel::Cover 89 Summary 91 iv | Table of Contents Further Reading 91 5. Benchmarking Perl. 93 Benchmarking Theory 93 Benchmarking Time 95 Comparing Code 98 Don’t Turn Off Your Thinking Cap 101 Isolating the Environment 105 Handling Outliers 107 Memory Use 109 The perlbench Tool 114 Summary 116 Further Reading 116 6. Cleaning Up Perl. 119 Good Style 119 perltidy 120 Deobfuscation 122 De-encoding Hidden Source 122 Unparsing Code with B::Deparse 125 Perl::Critic 127 Creating My Own Perl::Critic Policy 130 Summary 131 Further Reading 132 7. Symbol Tables and Typeglobs. 133 Package and Lexical Variables 133 Getting the Package Version 135 The Symbol Table 137 Typeglobs 139 Aliasing 142 Filehandle Arguments in Older Code 144 Naming Anonymous Subroutines 145 The Easy Way 146 Summary 147 Further Reading 148 8. Dynamic Subroutines. 149 Subroutines as Data 149 Creating and Replacing Named Subroutines 153 Symbolic References 155 Iterating Through Subroutine Lists 157 Table of Contents | v Processing Pipelines 159 Self-Referencing Anonymous Subroutines 160 Method Lists 160 Subroutines as Arguments 161 Autoloaded Methods 165 Hashes as Objects 167 AutoSplit 168 Summary 169 Further Reading 169 9. Modifying and Jury-Rigging Modules. 171 Choosing the Right Solution 171 Sending Patches to the Author 171 Local Patches 173 Taking Over a Module 173 Forking 174 Starting Over on My Own 174 Replacing Module Parts 174 Subclassing 177 An ExtUtils::MakeMaker Example 179 Other Examples 182 Wrapping Subroutines 182 Summary 184 Further Reading 184 10. Configuring Perl Programs. 185 Things Not to Do 185 Code in a Separate File 187 Better Ways 188 Environment Variables 188 Special Environment Variables 189 Turning on Extra Output 189 Command-Line Switches 191 The -s Switch 192 Getopt Modules 193 Configuration Files 198 ConfigReader::Simple 198 Config::IniFiles 199 Config::Scoped 199 Other Configuration Formats 200 Scripts with a Different Name 200 Interactive and Noninteractive Programs 201 vi | Table of Contents perl’s Config 202 Different Operating Systems 203 Summary 204 Further Reading 204 11. Detecting and Reporting Errors. 205 Perl Error Basics 205 Operating System Errors 206 Child Process Errors 208 Errors Specific to the Operating System 210 Reporting Module Errors 211 Separation of Concerns 211 Exceptions 213 eval 214 Multiple Levels of die 215 die with a Reference 216 Propagating Objects with die 218 Clobbering $@ 220 autodie 222 Reporting the Culprit 223 Catching Exceptions 227 Try::Tiny 227 TryCatch 229 Polymorphic Return Values 230 Summary 231 Further Reading 231 12. Logging. 233 Recording Errors and Other Information 233 Log4perl 234 Subroutine Arguments 236 Configuring Log4perl 237 Persistent Configuration 241 Logging Categories 241 Other Log::Log4perl Features 243 Summary 245 Further Reading 245 13. Data Persistence. 247 Perl-Specific Formats 247 pack 247 Fixed-Length Records 249 Table of Contents | vii Unpacking Binary Formats 249 Data::Dumper 250 Similar Modules 254 Storable 256 Freezing Data 257 Storable’s Security Problem 260 Sereal 262 DBM Files 267 dbmopen 268 DBM::Deep 268 Perl-Agnostic Formats 270 JSON 270 YAML 272 MessagePack 274 Summary 275 Further Reading 275 14. Working with Pod. 277 The Pod Format 277 Directives 277 Encoding 279 Body Elements 279 Translating Pod 279 Pod Translators 280 Pod::Perldoc::ToToc 281 Pod::Simple 283 Subclassing Pod::Simple 286 Pod in Your Web Server 286 Testing Pod 286 Checking Pod 287 Pod Coverage 287 Hiding and Ignoring Functions 289 Summary 290 Further Reading 290 15. Working with Bits. 291 Binary Numbers 291 Writing in Binary 292 Bit Operators 293 Unary NOT (~) 294 Bitwise AND (&) 296 Binary OR (|) 297 viii | Table of Contents Exclusive OR (^) 298 Left << and Right >> Shift Operators 300 Bit Vectors 300 The vec Function 302 Bit String Storage 304 Storing DNA 306 Checking Primes 307 Keeping Track of Things 309 Summary 310 Further Reading 310 16. The Magic of Tied Variables. 311 They Look Like Normal Variables 311 At the User Level 312 Behind the Curtain 313 Scalars 314 Tie::Cycle 314 Bounded Integers 317 Self-Destructing Values 318 Arrays 319 Reinventing Arrays 320 Something a Bit More Realistic 323 Hashes 328 Filehandles 331 Summary 333 Further Reading 333 17. Modules as Programs. 335 The main Thing 335 Backing Up 336 Who’s Calling? 337 Testing the Program 338 Modules as Tests 338 Creating a Program Distribution 343 Adding to the Script 345 Distributing the Programs 350 Summary 350 Further Reading 351 A. Further Reading. 353 B. brian’s Guide to Solving Any Perl Problem. 357 Table of Contents | ix Index of Perl Modules in This Book. 364 Index. 367 x | Table of Contents Preface Mastering Perl is the third book in the series starting with Learning Perl, which taught you the basics of Perl syntax, progressing to Intermediate Perl, which taught you how to create reusable Perl software, and finally this book, which pulls everything together to show you how to bend Perl to your will. This isn’t a collection of clever tricks, but a way of thinking about Perl programming so you integrate the real-life problems of debugging, maintenance, configuration, and other tasks you’ll encounter as a working programmer. This book starts you on your path to becoming the person with the an‐ swers, and, failing that, the person who knows how to find the answers or discover the problem. Becoming a Master This book isn’t going to make you a Perl master; you have to do that for yourself by programming a lot of Perl, trying a lot of new things, and making a lot of mistakes. I’m going to help you get on the right path, but the road to mastery is one of self-reliance and independence. As a Perl master, you’ll be able to answer your own questions as well as those of others. In the golden age of guilds, craftsmen followed a certain path, both literally and figu‐ ratively, as they mastered their craft. They started as apprentices and would do the boring bits of work until they had enough skill to become the more trusted journeyman. The journeyman had greater responsibility but still worked under a recognized master. When they had learned enough of the craft, the journeymen would produce a “master work” to prove their skill. If other masters deemed it adequately masterful, the jour‐ neyman became a recognized master himself. The journeymen and masters also traveled (although people dispute if that’s where the “journey” part of the name came from) to other masters, where they would learn new techniques and skills.