Expert Python Programming Third Edition Become a master in Python by learning coding best practices and advanced programming concepts in Python 3.7 Michał Jaworski Tarek Ziadé BIRMINGHAM - MUMBAI Expert Python Programming Third Edition Copyright © 2019 Packt Publishing All rights reserved. No part of this book may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, without the prior written permission of the publisher, except in the case of brief quotations embedded in critical articles or reviews. Every effort has been made in the preparation of this book to ensure the accuracy of the information presented. However, the information contained in this book is sold without warranty, either express or implied. Neither the authors, nor Packt Publishing or its dealers and distributors, will be held liable for any damages caused or alleged to have been caused directly or indirectly by this book. Packt Publishing has endeavored to provide trademark information about all of the companies and products mentioned in this book by the appropriate use of capitals. However, Packt Publishing cannot guarantee the accuracy of this information. Commissioning Editor: Kunal Chaudhari Acquisition Editor: Chaitanya Nair Content Development Editor: Zeeyan Pinheiro Technical Editor: Ketan Kamble Copy Editor: Safis Editing Project Coordinator: Vaidehi Sawant Proofreader: Safis Editing Indexer: Priyanka Dhadke Graphics: Alishon Mendonsa Production Coordinator: Shraddha Falebhai First published: September 2008 Second edition: May 2016 Third edition: April 2019 Production reference: 1270419 Published by Packt Publishing Ltd. Livery Place 35 Livery Street Birmingham B3 2PB, UK. ISBN 978-1-78980-889-6 www.packtpub.com To my beloved wife, Oliwia, for her love, inspiration, and her endless patience. To my loyal friends, Piotr, Daniel, and Paweł, for their support. To my mother, for introducing me to the amazing world of programming. – Michał Jaworski mapt.io Mapt is an online digital library that gives you full access to over 5,000 books and videos, as well as industry leading tools to help you plan your personal development and advance your career. For more information, please visit our website. Why subscribe? Spend less time learning and more time coding with practical eBooks and Videos from over 4,000 industry professionals Improve your learning with Skill Plans built especially for you Get a free eBook or video every month Mapt is fully searchable Copy and paste, print, and bookmark content Packt.com Did you know that Packt offers eBook versions of every book published, with PDF and ePub files available? You can upgrade to the eBook version at www.packt.com and as a print book customer, you are entitled to a discount on the eBook copy. Get in touch with us at [email protected] for more details. At www.packt.com, you can also read a collection of free technical articles, sign up for a range of free newsletters, and receive exclusive discounts and offers on Packt books and eBooks. Contributors About the authors Michał Jaworski has 10 years' of professional experience in Python. He has been in various roles at different companies, from an ordinary full-stack developer, through software architect, to VP of engineering in a fast-paced start-up company. He is currently a senior backend engineer at Showpad. He is highly experienced in designing high-performance distributed services. He is also an active contributor to many open source Python projects. Tarek Ziadé is a Python developer located in the countryside near Dijon, France. He works at Mozilla in the services team. He founded a French Python user group called Afpy, and has written several books about Python in French and English. When he is not hacking on his computer or hanging out with his family, he's spending time between his two passions, running and playing the trumpet. You can visit his personal blog (Fetchez le Python) and follow him on Twitter (tarek_ziade). About the reviewer Cody Jackson is a disabled military veteran, the founder of Socius Consulting, an IT and business management consulting company in San Antonio, and a co-founder of Top Men Technologies. He is currently employed at CACI International as the lead ICS/SCADA modeling and simulations engineer. He has been involved in the tech industry since 1994, when he joined the Navy as a nuclear chemist and radcon technician. Prior to CACI, he worked at ECPI University as a computer information systems adjunct professor. A self- taught Python programmer, he is the author of Learning to Program Using Python and Secret Recipes of the Python Ninja. He holds an Associate in Science degree, a Bachelor of Science degree, and a Master of Science degree. Packt is searching for authors like you If you're interested in becoming an author for Packt, please visit authors.packtpub.com and apply today. We have worked with thousands of developers and tech professionals, just like you, to help them share their insight with the global tech community. You can make a general application, apply for a specific hot topic that we are recruiting an author for, or submit your own idea. Table of Contents Preface 1 Section 1: Before You Start Chapter 1: Current Status of Python 8 Technical requirements 9 Where are we now and where we are going to? 9 Why and how Python changes 10 Being up-to-date with changes by following PEP documents 10 Python 3 adoption at the time of writing this book 11 The main differences between Python 3 and Python 2 13 Why should I care? 13 The main syntax differences and common pitfalls 13 Syntax changes 14 Changes in the standard library 15 Changes in data types and collections and string literals 16 The popular tools and techniques used for maintaining cross-version compatibility 16 Not only CPython 20 Why should I care? 20 Stackless Python 21 Jython 22 IronPython 22 PyPy 23 MicroPython 24 Useful resources 25 Summary 26 Chapter 2: Modern Python Development Environments 27 Technical requirements 28 Installing additional Python packages using pip 28 Isolating the runtime environment 30 Application-level isolation versus system-level isolation 31 Python's venv 32 venv versus virtualenv 34 System-level environment isolation 35 Virtual development environments using Vagrant 37 Virtual environments using Docker 39 Containerization versus virtualization 39 Writing your first Dockerfile 40 Table of Contents Running containers 43 Setting up complex environments 44 Useful Docker recipes for Python 45 Reducing the size of containers 46 Addressing services inside of a Compose environment 47 Communicating between multiple Compose environments 48 Popular productivity tools 50 Custom Python shells – ipython, bpython, ptpython, and so on 50 Setting up the PYTHONSTARTUP environment variable 52 IPython 52 bpython 52 ptpython 53 Incorporating shells in your own scripts and programs 53 Interactive debuggers 54 Summary 55 Section 2: Python Craftsmanship Chapter 3: Modern Syntax Elements - Below the Class Level 57 Technical requirements 57 Python's built-in types 58 Strings and bytes 58 Implementation details 61 String concatenation 61 Constant folding, the peephole optimizer, and the AST optimizer 63 String formatting with f-strings 63 Containers 65 Lists and tuples 65 Implementation details 66 List comprehensions 67 Other idioms 68 Dictionaries 71 Implementation details 72 Weaknesses and alternatives 73 Sets 75 Implementation details 76 Supplemental data types and containers 77 Specialized data containers from the collections module 77 Symbolic enumeration with the enum module 78 Advanced syntax 81 Iterators 81 Generators and yield statements 84 Decorators 88 General syntax and possible implementations 88 As a function 89 As a class 90 Parametrizing decorators 90 Introspection preserving decorators 91 Usage and useful examples 93 Argument checking 93 [ ii ] Table of Contents Caching 95 Proxy 98 Context provider 99 Context managers – the with statement 100 The general syntax and possible implementations 101 As a class 102 As a function – the contextlib module 104 Functional-style features of Python 105 What is functional programming? 106 Lambda functions 107 map(), filter(), and reduce() 108 Partial objects and partial() functions 111 Generator expressions 112 Function and variable annotations 113 The general syntax 113 The possible uses 114 Static type checking with mypy 115 Other syntax elements you may not know of yet 116 The for ... else ... statement 116 Keyword-only arguments 117 Summary 119 Chapter 4: Modern Syntax Elements - Above the Class Level 120 Technical requirements 121 The protocols of the Python language – dunder methods and attributes 121 Reducing boilerplate with data classes 123 Subclassing built-in types 126 MRO and accessing methods from superclasses 129 Old-style classes and super in Python 2 131 Understanding Python's Method Resolution Order 132 Super pitfalls 137 Mixing super and explicit class calls 137 Heterogeneous arguments 138 Best practices 140 Advanced attribute access patterns 140 Descriptors 141 Real-life example – lazily evaluated attributes 144 Properties 147 Slots 150 Summary 151 Chapter 5: Elements of Metaprogramming 152 Technical requirements 152 What is metaprogramming? 153 Decorators – a method of metaprogramming 153 Class decorators 154 [ iii ] Table of Contents Using __new__() for overriding the instance creation process 156 Metaclasses 159 The general syntax 160 New Python 3 syntax for metaclasses 163 Metaclass usage 166 Metaclass pitfalls 166 Code generation 167 exec,