Introduction to Python for Econometrics, Statistics and Data Analysis 4Th Edition
Total Page:16
File Type:pdf, Size:1020Kb
Introduction to Python for Econometrics, Statistics and Data Analysis 4th Edition Kevin Sheppard University of Oxford Thursday 31st December, 2020 2 - ©2020 Kevin Sheppard Solutions and Other Material Solutions Solutions for exercises and some extended examples are available on GitHub. https://github.com/bashtage/python-for-econometrics-statistics-data-analysis Introductory Course A self-paced introductory course is available on GitHub in the course/introduction folder. Solutions are avail- able in the solutions/introduction folder. https://github.com/bashtage/python-introduction/ Video Demonstrations The introductory course is accompanied by video demonstrations of each lesson on YouTube. https://www.youtube.com/playlist?list=PLVR_rJLcetzkqoeuhpIXmG9uQCtSoGBz1 Using Python for Financial Econometrics A self-paced course that shows how Python can be used in econometric analysis, with an emphasis on financial econometrics, is also available on GitHub in the course/autumn and course/winter folders. https://github.com/bashtage/python-introduction/ ii Changes Changes since the Fourth Edition • Added a discussion of context managers using the with statement. • Switched examples to prefer the context manager syntax to reflect best practices. iv Notes to the Fourth Edition Changes in the Fourth Edition • Python 3.8 is the recommended version. The notes require Python 3.6 or later, and all references to Python 2.7 have been removed. • Removed references to NumPy’s matrix class and clarified that it should not be used. • Verified that all code and examples work correctly against 2020 versions of modules. The notable pack- ages and their versions are: – Python 3.8 (Preferred version), 3.6 (Minimum version) – NumPy: 1.19.1 – SciPy: 1.5.3 – pandas: 1.1 – matplotlib: 3.3 • Expanded description of model classes and statistical tests in statsmodels that are most relevant for econo- metrics. TODO • Expanded the list of packages of interest to researchers working in statistics, econometrics and machine learning. TODO • Introduced f-Strings in Section 21.3.3 as the preferred way to format strings using modern Python. • Added minimize as the preferred interface for non-linear function optimization in Chapter 20. TODO Changes since the Third Edition • Verified that all code and examples work correctly against 2019 versions of modules. The notable pack- ages and their versions are: – Python 3.7 (Preferred version) – NumPy: 1.16 – SciPy: 1.3 – pandas: 0.25 – matplotlib: 3.1 • Python 2.7 support has been officially dropped, although most examples continue to work with 2.7. Do not Python 2.7 in 2019 for numerical code. vi • Small typo fixes, thanks to Marton Huebler. • Fixed direct download of FRED data due to API changes, thanks to Jesper Termansen. • Thanks for Bill Tubbs for a detailed read and multiple typo reports. • Updated to changes in line profiler (see Ch. 23) • Updated deprecations in pandas. • Removed hold from plotting chapter since this is no longer required. • Thanks for Gen Li for multiple typo reports. • Tested all code on Pyton 3.6. Code has been tested against the current set of modules installed by conda as of February 2018. The notable packages and their versions are: – NumPy: 1.13 – Pandas: 0.22 Notes to the Third Edition This edition includes the following changes from the second edition (August 2014). Changes in the Third Edition • Rewritten installation section focused exclusively on using Continuum’s Anaconda. • Python 3.5 is the default version of Python instead of 2.7. Python 3.5 (or newer) is well supported by the Python packages required to analyze data and perform statistical analysis, and bring some new useful features, such as a new operator for matrix multiplication (@). • Removed distinction between integers and longs in built-in data types chapter. This distinction is only relevant for Python 2.7. • dot has been removed from most examples and replaced with @ to produce more readable code. • Split Cython and Numba into separate chapters to highlight the improved capabilities of Numba. • Verified all code working on current versions of core libraries using Python 3.5. • pandas – Updated syntax of pandas functions such as resample. – Added pandas Categorical. – Expanded coverage of pandas groupby. – Expanded coverage of date and time data types and functions. • New chapter introducing statsmodels, a package that facilitates statistical analysis of data. statsmodels includes regression analysis, Generalized Linear Models (GLM) and time-series analysis using ARIMA models. Changes since the Second Edition • Fixed typos reported by a reader – thanks to Ilya Sorvachev • Code verified against Anaconda 2.0.1. • Added diagnostic tools and a simple method to use external code in the Cython section. • Updated the Numba section to reflect recent changes. • Fixed some typos in the chapter on Performance and Optimization. viii • Added examples of joblib and IPython’s cluster to the chapter on running code in parallel. • New chapter introducing object-oriented programming as a method to provide structure and organization to related code. • Added seaborn to the recommended package list, and have included it be default in the graphics chapter. • Based on experience teaching Python to economics students, the recommended installation has been simplified by removing the suggestion to use virtual environment. The discussion of virtual environments as been moved to the appendix. • Rewrote parts of the pandas chapter. • Changed the Anaconda install to use both create and install, which shows how to install additional pack- ages. • Fixed some missing packages in the direct install. • Changed the configuration of IPython to reflect best practices. • Added subsection covering IPython profiles. • Small section about Spyder as a good starting IDE. Notes to the Second Edition This edition includes the following changes from the first edition (March 2012). Changes in the Second Edition • The preferred installation method is now Continuum Analytics’ Anaconda. Anaconda is a complete scientific stack and is available for all major platforms. • New chapter on pandas. pandas provides a simple but powerful tool to manage data and perform prelim- inary analysis. It also greatly simplifies importing and exporting data. • New chapter on advanced selection of elements from an array. • Numba provides just-in-time compilation for numeric Python code which often produces large perfor- mance gains when pure NumPy solutions are not available (e.g. looping code). • Dictionary, set and tuple comprehensions • Numerous typos • All code has been verified working against Anaconda 1.7.0. x Contents 1 Introduction 1 1.1 Background ............................................ 1 1.2 Conventions ........................................... 2 1.3 Important Components of the Python Scientific Stack ..................... 3 1.4 Setup ............................................... 4 1.5 Using Python ........................................... 5 1.6 Exercises ............................................. 12 1.A Additional Installation Issues .................................. 12 2 Built-in Data Types 15 2.1 Variable Names .......................................... 15 2.2 Core Native Data Types ..................................... 16 2.3 Additional Container Data Types in the Standard Library ................... 24 2.4 Python and Memory Management ............................... 26 2.5 Exercises ............................................. 27 3 Arrays 29 3.1 Array ............................................... 29 3.2 1-dimensional Arrays ....................................... 30 3.3 2-dimensional Arrays ....................................... 31 3.4 Multidimensional Arrays ..................................... 31 3.5 Concatenation .......................................... 32 3.6 Accessing Elements of an Array ................................. 33 3.7 Slicing and Memory Management ................................ 37 3.8 import and Modules ...................................... 39 3.9 Calling Functions ......................................... 40 3.10 Exercises ............................................. 41 4 Basic Math 43 4.1 Operators ............................................. 43 4.2 Broadcasting ........................................... 43 4.3 Addition (+) and Subtraction (-) ................................. 45 4.4 Multiplication (⁎) ......................................... 45 4.5 Matrix Multiplication (@) ..................................... 45 4.6 Array and Matrix Division (=) ................................... 46 xii CONTENTS 4.7 Exponentiation (**) ........................................ 46 4.8 Parentheses ........................................... 46 4.9 Transpose ............................................. 46 4.10 Operator Precedence ...................................... 46 4.11 Exercises ............................................. 47 5 Basic Functions and Numerical Indexing 49 5.1 Generating Arrays ........................................ 49 5.2 Rounding ............................................. 52 5.3 Mathematics ........................................... 52 5.4 Complex Values ......................................... 54 5.5 Set Functions ........................................... 54 5.6 Sorting and Extreme Values ................................... 56 5.7 Nan Functions .......................................... 57 5.8 Functions and Methods/Properties ............................... 58 5.9 Exercises ............................................. 58 6 Special