How Ruby Works Ruby Under a Under the Hood a Microscope Under Ruby Ruby is a powerful programming language with How Ruby employs algorithms originally a focus on simplicity, but beneath its elegant developed for Lisp syntax it performs countless unseen tasks. How Ruby uses grammar rules to parse Microscope Ruby Under a Microscope gives you a and understand your code hands-on look at Ruby’s core, using extensive How your Ruby code is translated into a diagrams and thorough explanations to show different language by a compiler you how Ruby is implemented (no C skills required). Author Pat Shaughnessy takes No programming language needs to be a An Illustrated Guide a scientific approach, laying out a series of black box. Whether you’re already intrigued by experiments with Ruby code to take you behind language implementation or just want to dig to Ruby Internals the scenes of how programming languages deeper into Ruby, you’ll findRuby Under a work. You’ll even find information on JRuby Microscope a fascinating way to become and Rubinius (two alternative implementations a better programmer. of Ruby), as well as in-depth explorations of Pat Shaughnessy Ruby’s garbage collection algorithm. About the Author Ruby Under a Microscope will teach you: Well known for his coding expertise and passion for the Ruby programming language, How a few computer science concepts Pat Shaughnessy blogs and writes tutorials underpin Ruby’s complex implementation at http://patshaughnessy.net/. He also How Ruby executes your code using a develops Ruby applications at management virtual machine consulting firm McKinsey & Co. Shaughnessy is a regular presenter on the Ruby conference 10.times do |n| How classes and modules are the same circuit, and his articles and presentations have inside Ruby been featured in the Ruby Weekly newsletter, the Ruby5 podcast, and The Ruby Show. puts n Shaughnessy Covers Ruby 2.x, 1.9, and 1.8 end “I LIE FLAT.” This book uses RepKover — a durable binding that won’t snap shut. THE FINEST IN GEEK ENTERTAINMENT™ www.nostarch.com $39.95 ($41.95 CDN) Shelve In: Programming Languages/Ruby ADVANCE PRAISE FOR RUBY UNDER A MICROSCOPE “Many people have dug into the Ruby source code, but few make it back out and tell the tale as elegantly as Pat does in Ruby Under a Microscope! I particularly love the diagrams—and there are lots of them—as they make many opaque implementation topics a lot easier to understand, especially when coupled with Pat’s gentle narrative. This book is a delight for language implementation geeks and Rubyists with a penchant for dig- ging into the guts of their tools.” —PETER COOPER (@PETERC), EDITOR OF RUBY INSIDE AND RUBY WEEKLY “Man, this book was missing in the Ruby landscape—awesome content.” —XAVIER NORIA (@FXN), RUBY HERO, RUBY ON RAILS CORE TEAM MEMBER “Pat Shaughnessy did a tremendous job writing THE book about Ruby internals. Definitely a must read—you won’t find information like this anywhere else.” —SANTIAGO PASTORINO (@SPASTORINO), WYEWORKS CO-FOUNDER, RUBY ON RAILS CORE TEAM MEMBER “I really enjoyed the book and now have a far better understanding of both Ruby and CS. The writing made very complex topics (at least for me) very accessible, and I found the book hard to put down. Diagrams were awesome and are already popping in my head as I code. This is by far one of my top 3 favourite Ruby books.” —VLAD IVANOVIC (@VLADIIM), DIGITAL STRATEGIST AT HOLLER SYDNEY “While I’m not usually digging into Ruby Internals, this book was an absolutely awesome read.” —DAVID DERYL DOWNEY (@DAVIDDWDOWNEY), FOUNDER OF CYBERSPACE TEchNOLOGIES GROUP Ruby Under a Microscope An Illustrated Guide to Ruby Internals Pat Shaughnessy RUBY UNDER A MICROSCOPE. Copyright © 2014 by Patrick Shaughnessy. All rights reserved. No part of this work may be reproduced or transmitted in any form or by any means, elec- tronic or mechanical, including photocopying, recording, or by any information storage or retrieval system, without the prior written permission of the copyright owner and the publisher. Printed in USA First printing 17 16 15 14 13 1 2 3 4 5 6 7 8 9 ISBN-10: 1-59327-527-7 ISBN-13: 978-1-59327-527-3 Publisher: William Pollock Production Editor: Riley Hoffman Cover Illustration: Charlie Wylie Interior Design: Octopod Studios Developmental Editor: William Pollock Technical Reviewer: Aaron Patterson Copyeditor: Julianne Jigour Compositors: Susan Glinert Stevens and Riley Hoffman Proofreader: Elaine Merrill For information on distribution, translations, or bulk sales, please contact No Starch Press, Inc. directly: No Starch Press, Inc. 245 8th Street, San Francisco, CA 94103 phone: 415.863.9900; fax: 415.863.9950; [email protected]; www.nostarch.com Library of Congress Cataloging-in-Publication Data Shaughnessy, Pat. Ruby under a microscope : an illustrated guide to Ruby internals / by Pat Shaughnessy. pages cm Summary: "An under-the-hood look at how the Ruby programming language runs code. Extensively illustrated with complete explanations and hands-on experiments. Covers Ruby 2.x"-- Provided by publisher. ISBN 978-1-59327-527-3 (paperback) -- ISBN 1-59327-527-7 (paperback) 1. Ruby (Computer program language) I. Title. QA76.73.R83S53 2013 005.1'17--dc23 2013030614 No Starch Press and the No Starch Press logo are registered trademarks of No Starch Press, Inc. Other product and company names mentioned herein may be the trademarks of their respective owners. Rather than use a trademark symbol with every occurrence of a trademarked name, we are using the names only in an editorial fashion and to the benefit of the trademark owner, with no intention of infringement of the trademark. The information in this book is distributed on an “As Is” basis, without warranty. While every precaution has been taken in the preparation of this work, neither the author nor No Starch Press, Inc. shall have any liability to any person or entity with respect to any loss or damage caused or alleged to be caused directly or indirectly by the information contained in it. To my wife, Cristina; my daughter, Ana; and my son, Liam— thanks for supporting me all along. ABOUT THE AUTHOR Pat Shaughnessy is a Ruby developer working at McKinsey & Co., a management consulting firm. Pat was originally trained as a physicist at MIT, but later spent more than 20 years working as a software developer using C, Java, PHP, and Ruby, among other languages. Writing Ruby Under a Microscope has given him an excuse to reuse bits of his scientific training while studying Ruby. A fluent Spanish speaker, Pat frequently visits his wife’s family in northern Spain. He lives outside of Boston with his wife and two children. BRIEF CONTENTS Foreword by Aaron Patterson . xv Acknowledgments . xvii Introduction . xix Chapter 1: Tokenization and Parsing . 3 Chapter 2: Compilation . 31 Chapter 3: How Ruby Executes Your Code . 55 Chapter 4: Control Structures and Method Dispatch . 83 Chapter 5: Objects and Classes . 105 Chapter 6: Method Lookup and Constant Lookup . 133 Chapter 7: The Hash Table: The Workhorse of Ruby Internals . 167 Chapter 8: How Ruby Borrowed a Decades-Old Idea from Lisp . 191 Chapter 9: Metaprogramming . 219 Chapter 10: JRuby: Ruby on the JVM . 251 Chapter 11: Rubinius: Ruby Implemented with Ruby . 273 Chapter 12: Garbage Collection in MRI, JRuby, and Rubinius . 295 Index . 327 CONTENTS IN DETAIL FOREWORD by Aaron Patterson xv ACKNOWLEDGMENTS xvii INTRODUCTION xix Who This Book Is For . xx Using Ruby to Test Itself . xx Which Implementation of Ruby? . xxi Overview . xxi 1 TOKENIZATION AND PARSING 3 Tokens: The Words That Make Up the Ruby Language . 4 The parser_yylex Function . 8 Experiment 1-1: Using Ripper to Tokenize Different Ruby Scripts ................... 9 Parsing: How Ruby Understands Your Code . 12 Understanding the LALR Parse Algorithm . 13 Some Actual Ruby Grammar Rules . 20 Reading a Bison Grammar Rule . 22 Experiment 1-2: Using Ripper to Parse Different Ruby Scripts ..................... 23 Summary . 29 2 COMPILATION 31 No Compiler for Ruby 1 .8 . 32 Ruby 1 9. and 2 .0 Introduce a Compiler . 33 How Ruby Compiles a Simple Script . 34 Compiling a Call to a Block . 38 How Ruby Iterates Through the AST . 42 Experiment 2-1: Displaying YARV Instructions ................................ 44 The Local Table . 46 Compiling Optional Arguments . 48 Compiling Keyword Arguments . 49 Experiment 2-2: Displaying the Local Table .................................. 51 Summary . 53 3 HOW RUBY EXECUTES YOUR CODE 55 YARV’s Internal Stack and Your Ruby Stack . 56 Stepping Through How Ruby Executes a Simple Script . 58 Executing a Call to a Block . 61 Taking a Close Look at a YARV Instruction . 63 Experiment 3-1: Benchmarking Ruby 2.0 and Ruby 1.9 vs. Ruby 1.8 ............... 65 Local and Dynamic Access of Ruby Variables . 67 Local Variable Access . 67 Method Arguments Are Treated Like Local Variables . 70 Dynamic Variable Access . 71 Climbing the Environment Pointer Ladder in C . 74 Experiment 3-2: Exploring Special Variables ................................. 75 A Definitive List of Special Variables . 79 Summary . 81 4 CONTROL STRUCTURES AND METHOD DISPATCH 83 How Ruby Executes an if Statement . 84 Jumping from One Scope to Another . 86 Catch Tables . 88 Other Uses for Catch Tables . 90 Experiment 4-1: Testing How Ruby Implements for Loops Internally ................ 90 The send Instruction: Ruby’s Most Complex Control Structure . 92 Method Lookup and Method Dispatch . 92 Eleven Types of Ruby Methods . 93 Calling Normal Ruby Methods . 95 Preparing Arguments for Normal Ruby Methods . 95 Calling Built-In Ruby Methods . 97 Calling attr_reader and attr_writer . 97 Method Dispatch Optimizes attr_reader and attr_writer . 98 Experiment 4-2: Exploring How Ruby Implements Keyword Arguments .............. 99 Summary . 103 5 OBJECTS AND CLASSES 105 Inside a Ruby Object .