PYTHON PROGRAMMING Notes by Michael Brothers http://titaniumventures.net/library/ Table of Contents INTRODUCTION ...... 5 What is Python? ...... 5 Ways to Code Python ...... 5 Using Anaconda ...... 5 Using Jupyter Notebooks ...... 5 Python 2 vs Python 3 ...... 5 PEP 8 -- Style Guide for Python Code...... 5 Using this guide: ...... 5 PYTHON PROGRAMMING ...... 6 Variables ...... 6 Multiple Declaration ...... 6 Multiple Assignment ...... 6 Data Types ...... 6 Operators ...... 6 Comparison Operators: ...... 6 Chained Comparison Operators: ...... 6 Strings: ...... 7 Lists: ...... 7 Tuples: ...... 7 Dictionaries: ...... 7 Sets:...... 7 Comments:...... 7 WORKING WITH STRINGS ...... 8 Built-in String Functions: ...... 8 Built-in String Methods: ...... 8 Splitting Strings: ...... 8 Joining Strings: ...... 8 Turning Objects Into Strings ...... 8 String formatting with formatted string literals (f-strings) ...... 9 String Formatting with .format() ...... 10 String Formatting with Placeholders: ...... 11 Escape Characters: ...... 11 WORKING WITH LISTS: ...... 12 Built-in List Functions: ...... 12 Built-in List Methods: ...... 12 Adding & Removing List Items ...... 12 List Index Method ...... 12 Sorting Lists...... 13 Making a List of Lists: ...... 13 LIST COMPREHENSIONS ...... 14 WORKING WITH TUPLES ...... 15 WORKING WITH DICTIONARIES ...... 15 Dictionary Comprehensions: ...... 15 WORKING WITH SETS: ...... 16 Set Operators ...... 16

1 REV 1020

Built-in Set Methods ...... 16 Frozensets ...... 16 RANGE ...... 16 CONDITIONAL STATEMENTS & LOOPS ...... 17 If / Elif / Else statements: ...... 17 For Loops ...... 17 While Loops ...... 17 Nested Loops ...... 17 Loop Control Statements (Break, Continue, Pass & Else) ...... 18 Try and Except ...... 18 INPUT ...... 19 UNPACKING ...... 19 Tuple Unpacking ...... 19 Dictionary Unpacking ...... 19 FUNCTIONS ...... 20 Default Parameter Values ...... 20 Positional Arguments *args and **kwargs ...... 20 PRE-DEFINED FUNCTIONS ...... 21 LAMBDA EXPRESSIONS ...... 21 MORE USEFUL FUNCTIONS ...... 22 MAP ...... 22 FILTER ...... 22 REDUCE ...... 22 ZIP ...... 22 ENUMERATE ...... 23 ALL & ANY ...... 23 COMPLEX ...... 23 PYTHON THEORY & DEFINITIONS ...... 24 SCOPE ...... 24 LEGB Rule: ...... 24 In place ...... 24 Sequenced ...... 24 Iterable...... 24 Statement ...... 24 Stack ...... 24 FUNCTIONS AS OBJECTS & ASSIGNING VARIABLES ...... 25 FUNCTIONS AS ARGUMENTS ...... 25 DECORATORS: ...... 26 GENERATORS & ITERATORS ...... 27 NEXT & ITER built-in functions: ...... 27 GENERATOR COMPREHENSIONS ...... 27 WORKING WITH FILES ...... 28 READING AND WRITING TO FILES ...... 28 RENAMING & COPYING FILES with the OS MODULE ...... 28 CREATING BINARY FILES with the SHELVE MODULE ...... 28 ORGANIZING FILES with the SHUTIL MODULE ...... 29 DELETING FOLDERS AND FILES ...... 29

2 REV 1020

Using the OS module: ...... 29 Using the SHUTIL module: ...... 29 Using the Send2Trash module: ...... 29 WALKING A DIRECTORY TREE ...... 29 MANAGING EXCEPTIONS ...... 30 Raising Exceptions ...... 30 Raising Your Own Exception ...... 30 Tracebacks ...... 30 Assertions ...... 30 Logging ...... 31 Logging to a text file ...... 31 Create your own Exception class: ...... 31 OBJECT ORIENTED PROGRAMMING – Classes, Attributes & Methods ...... 32 Classes: ...... 32 Built-in Types: ...... 32 Instances: ...... 32 Attributes: ...... 32 Methods: ...... 32 Inheritance: ...... 32 Special Methods (aka Magic Methods): ...... 32 For Further Reading: ...... 32 Example 1: ...... 33 Example 2 – Methods: ...... 33 Example 3 – Inheritance: ...... 34 Example 4 – Special Methods: ...... 35 MODULES & PACKAGES ...... 36 COLLECTIONS Module: ...... 37 Counter ...... 37 defaultdict ...... 37 OrderedDict ...... 38 namedtuple ...... 38 RANDOM Module ...... 38 DATETIME Module ...... 39 TIMEIT Module ...... 39 PYTHON DEBUGGER – the pdb Module ...... 40 IDLE'S BUILT-IN DEBUGGER ...... 40 REGULAR EXPRESSIONS – the re Module ...... 41 Searching for Patterns in Text ...... 41 Finding all matches ...... 41 Split on multiple delimeters with regular expressions ...... 41 Using metacharacters ...... 42 STYLE AND READABILITY (PEP 8) ...... 43 APPENDIX I: GOING DEEPER: ...... 45 Reserved words ...... 45 The '_' variable ...... 46 Print on the same line: ...... 46 Print multiple objects using the * unpacking operator ...... 46 Some more (& obscure) built-in string methods:...... 47

3 REV 1020

Some more (& obscure) built-in set methods: ...... 47 Common Errors & Exceptions: ...... 47 Using dir() to identify available names and functions ...... 48 Adding a username & password ...... 48 Picking a random rock/paper/scissors ...... 48 Referential Arrays ...... 48 Deep and shallow copies ...... 48 Dynamic Arrays ...... 49 More ways to break out of loops (in "Goto" fashion) ...... 49 Bitwise Operators: ...... 49 How to take an input as a list , tuple or dictionary ...... 50 Unpacking a nested list using sum() ...... 50 Adding an incremental counter as a class attribute ...... 50 APPENDIX II: PYTHON "GOTCHAS" ...... 51 1. Default Arguments ...... 51 2. Rounding issues in Python ...... 51 3. Some methods permanently affect the objects they act on AND RETURN A NONE VALUE ...... 51 4. print() returns a None value ...... 51 APPENDIX III: CODE SAMPLES ...... 52 Try and Except ...... 52 Capture Tracebacks to a File ...... 52 Assertions: The Stoplight Example ...... 52 Logging with a Buggy Factorial program ...... 53 Find every target item in a nested list using recursion ...... 53 APPENDIX IV: DIFFERENCES BETWEEN PYTHON 2 and PYTHON 3 ...... 54 APPENDIX V: OTHER RESOURCES ...... 56 Docs, Tutorials and Popular Forums ...... 56 Sharing Code and Version Control ...... 56 For More Practice ...... 56

4 REV 1020

The following courses and resources aided in the creation of this document: Introduction To Python Programming Instructed by Avinash Jain https://www.udemy.com/pythonforbeginnersintro/ The Python Tutorial http://docs.python.org/3/tutorial/ Introducing Python: Modern Computing in Simple Packages, 1st ed by Bill Lubanovic (paperback) Complete Python Bootcamp Instructed by Jose Portilla https://www.udemy.com/complete-python-bootcamp/ Automate the Boring Stuff with Python Instructed by Al Sweigart https://www.udemy.com/automate/

INTRODUCTION

What is Python? From Wikipedia: "Python is an interpreted, high-level and general-purpose programming language. Python's design philosophy emphasizes code readability with its notable use of significant whitespace. Its language constructs and object-oriented approach aim to help programmers write clear, logical code for small and large-scale projects." It was created by Guido van Rossum and first released in 1991.

Ways to Code Python Python can be run from a command prompt using IDLE (runs an editor & a python shell side-by-side), though it is far more common to write Python programs using a text editor (Notepad, Notepad++, Atom, Sublime Text) or an Integrated Development Environment (IDE) (WingWare, Komodo, PyCharm, Spyder, Jupyter)

Using Anaconda https://www.anaconda.com/products/individual Anaconda is a free Python distribution that includes many of the most popular Python packages including packages for science, math, engineering and data analysis. In addition, Anaconda makes it easy to create and maintain virtual environments. Anaconda comes bundled with two popular IDEs: Jupyter and Spyder.

Using Jupyter Notebooks Jupyter is an IDE that facilitates writing code cell-by-cell, running cells individually, reordering, interrupting kernels, etc. Additionally, it acts like a cloud in that you can store/share/download other users’ notebooks Shift+Enter will run a cell (shift+enter on a blank cell will clear any residual output in the cell) Tab will show a pop-up of the methods on an object Shift+Tab will show a help pop-up (or docstring) for a function pwd will print the current working directory

Python 2 vs Python 3 Version 3 of Python made significant changes to the language, breaking some backward compatibility. Python 2 reached its End of Life status in January 2020, and is no longer supported. For more info, refer to Appendix IV, or visit https://wiki.python.org/moin/Python2orPython3

PEP 8 -- Style Guide for Python Code https://www.python.org/dev/peps/pep-0008/ Written by Guido van Rossum himself, provides guidelines for writing clean, consistent code

Using this guide: Code is written in a fixed-width font: Red indicates required syntax/new concepts Green represents arbitrary variable assignments Blue indicates program output

5 REV 1020

PYTHON PROGRAMMING

Variables: reserved memory space. Can hold any value, assigned to a term. Case-sensitive. Can’t contain spaces. Variable Names 1. Names can not start with a number 2. There can be no spaces in the name, use _ instead 3. Can’t use any of these symbols: ' " , < > / ? | \ ( ) ! @ # $ % ^ & * ~ - + 4. It’s considered best practice (PEP8) that variable names are lowercase with underscores 5. Don't use these reserved words: and assert break class continue def del elif else except exec finally for from global if import in is lambda not or pass print raise return try while 6. Avoid these words: dict, int, len, list, str 7. From PEP8, "Never use the characters 'l' (lowercase letter el), 'O' (uppercase letter oh), or 'I' (uppercase letter eye) as single character variable names." For a full list of variable names to avoid, jump here 8. In production, use names that clearly identify their purpose, ie "firstname" is better than "fn"

Multiple Declaration: var1, var2, var3 = 'apples','oranges','pears' Multiple Assignment: var1 = var2 = var3 = 'apples' (spaces/no spaces doesn’t matter)

Data Types: number (integer, float or complex), string (text), list, tuple, dictionary, set, Boolean (True/False), None Operators: + - * / addition, subtraction, multiplication, division % modulo = gives the remainder after division // floor divisor = discards the fraction without rounding ** exponentiator 5/2 returns 2.5 5//2 returns 2 5%2 returns 1 5**3 returns 125 Note: 2 is an int type number, while 2.5 is a float. Division returns a float. (6/3 returns 2.0)

Comparison Operators: (aka Relational Operators) > greater than >= greater than or equal to < less than <= less than or equal to == equal to Note: use == when comparing objects (it's a comparison operator) != not equal to Use = to assign names to objects (it's an assignment operator)

Chained Comparison Operators: 1 < 2 < 3 returns True this is shorthand for 1 < 2 and 2 < 3

6 REV 1020

Strings: str1 = 'This is a string' anything between a set of quotation marks (single or double) Embedded quotes are handled with the other type of quote: "This won't cause problems"

Lists: list1 = ['apples','oranges','pears'] created using square brackets

Tuples: tuple1 = (1,2,3) created using parentheses max(tuple1) returns 3, min(tuple1) returns 1

Strings, lists and tuples are sequences. Their contents are indexed as (0,1,2…) str1[1] returns 'h' list1[1] returns 'oranges' tuple1[1] returns 2

Strings are immutable. You can’t modify a string once it's created. Tuples are also immutable. You can't add, remove, replace or reorder tuple elements once created. However, if a tuple contains a mutable element like a list, then that element is still mutable.

Dictionaries: contain a set of key/value pairs, created using curly braces { } and colons dict1 = {'Tom':4, 'Dick':7, 'Harry':23} here the keys are 'Tom', 'Dick' and 'Harry' and the values are 4, 7 and 23 dict1['Harry'] returns 23 Dictionaries are mappings, not sequences. dict1[1] would return an error as dictionaries are unordered objects.* Dictionary keys must be unique, although two keys can share the same value. Dictionaries are mutable – keys can be added, removed, and have their values reassigned. * Dictionaries do retain insertion order. From the docs: Changed in version 3.7: LIFO order is now guaranteed. In prior versions, popitem() would return an arbitrary key/value pair.

Sets: behave like dictionaries, but only contain unique keys. Sets are unordered (not sequenced). set1 = set([1,1,2,2,3]) this is called "casting a list as a set" set1 returns {1,2,3}

Comments: # (one octothorpe) provides quick one-liners """ (a set of triple double quotes) allows multiline full text comments, called docstrings """

7 REV 1020

WORKING WITH STRINGS Indexes: var[10] returns the 11th character in the string (indexing in Python starts at 0) Slices: var[2:8] returns everything from index 2 up to but not including index 8 var[2:] returns everything after the second character (ie, it skips the first two elements) var[:3] returns everything up to the third character (ie, the first three elements) var[1:-1] returns everything between the first and last character Steps: var[::2] returns every other character starting with the first (0,2,4,6…) var[::-1] returns the string backwards [aka Reversing a String] Concatenate: var = var + ‘ more text’ Multiply: var*10 returns the var string 10 times Reverse: var[::-1] (there is no built-in reverse function or method) var[9:3:-2] start is greater than stop when traversing backward Shift: var = var[2:]+var[:2] moves the first two characters to the end

Built-in String Functions: len(string) returns the length of the string (including spaces) str(object) converts objects (int, float, etc.) into strings

Built-in String Methods: .upper s.upper() returns a copy of the string converted to uppercase. .lower s.lower() returns a copy of the string converted to lowercase. .count s.count('string'[start,[stop]]) adds up the number of times a character or sequence of characters appears in a string Case-sensitive, does not overlap If s='hahahah' then s.count('hah') returns only 2. .isupper s.isupper() returns True if all cased characters in the string are uppercase. There must be at least one cased character. It returns False otherwise. .islower s.islower() returns True if all cased characters in the string are lowercase. There must be at least one cased character. It returns False otherwise. .find s.find(value[,start,end]) finds the index position of the first occurrence of a character/phrase in a string Returns -1 if value not found .replace s.replace(old,new[,count]) returns a copy of the string with substitution, the original string is unchanged optional "count" argument limits the number of replacements In Jupyter, hit Tab to see a list of available methods for that object. Hit Shift+Tab for more information on the selected method - equivalent to help(s.method)

Splitting Strings: greeting = 'Hello, how are you?' greeting.split() returns ['Hello,', 'how', 'are', 'you?'] Note: the default delimiter is whitespace fruit = 'Apple' fruit.split('p') returns ['a', '', 'le'] note the additional null value fruit.partition('p') returns ('a','p','ple') (head, sep, tail) Note: methods work on objects, so 'The quick brown fox'.split() is also valid

Joining Strings: delimeter.join(list) joins a list of strings together, connected by a start string (delimeter) list1 = ['Ready', 'aim', 'fire!'] ', '.join(list1) returns 'Ready, aim, fire!'

Turning Objects Into Strings : str() aka "casting objects as strings" testnum = 3 print('You have just completed test ' + str(testnum) + '.') You have just completed test 3. Note the extra space after test above

8 REV 1020

String formatting with formatted string literals (f-strings) Introduced in Python 3.6, f-strings offer several benefits over the older .format() string method described in the next section. For one, you can bring outside variables immediately into to the string rather than pass them as arguments through .format(var). name = 'Fred' print(f"My grandfather's name was {name}.") My grandfather's name was Fred. pi = 3.14159265 print(f"The ratio of a circle's circumference to its diameter is {pi:{10}.{5}}") The ratio of a circle's circumference to its diameter is 3.1416 Here the number is given a width of 10 characters in the string, with a precision of 5 digits (this includes the 3).

In the precision, an f tells python to set the precision to the right of the decimal. Without it, precision is set to the total number of significant digits, as used in scientific notation. x = 567.12345 print(f'x = {x:.6f}') x = 567.123450 print(f'x = {x:.6}') x = 567.123

For more info visit https://docs.python.org/3/reference/lexical_analysis.html#f-strings

9 REV 1020

String Formatting with .format() An older style of string formatting involves the .format() string method. Double curly-braces serve as positional placeholders and eliminate need for str() print('I prefer Python version {} to {}.'.format(3.8, 2.7)) Note the lack of quotes I prefer Python version 3.8 to 2.7. around 3.8 and 2.7

You can change the order of variables inside the function: print('I prefer Python version {1} to {0}.'.format(3.8, 2.7)) I prefer Python version 2.7 to 3.8.

You can assign local variable names to placeholders: print('First: {x}, Second: {y}, Third: {z}.'.format(x=1., z='B', y=5)) First: 1.0, Second: 5, Third: B. Note that variables x, y and z are not defined outside of the function, and format handles the different object types. Format variables may be used more than once in a string, and stored in any order.

Within the braces you can assign field lengths, left/right alignments, rounding parameters and more print('{0:8} | {1:9}'.format('Fruit', 'Quantity')) print('{0:8} | {1:9}'.format('Apples', 3.)) print('{0:8} | {1:9}'.format('Oranges', 10)) Fruit | Quantity the 0 parameter takes the first object encountered Apples | 3.0 the 8 parameter sets the minimum field length to 8 characters Oranges | 10 By default, .format aligns text to the left, numbers to the right print('{0:8} | {1:<8}'.format('Fruit', 'Quantity')) print('{0:8} | {1:<8.2f}'.format('Apples', 3.66667)) print('{0:8} | {1:<8.2f}'.format('Oranges', 10)) Fruit | Quantity < sets a left-align (^ for center, > for right) Apples | 3.67 .2f converts the variable to a float with 2 decimal places Oranges | 10.00

These two statements are equivalent: print('This is my ten-character, two-decimal number:%10.2f' %13.579) print('This is my ten-character, two-decimal number:{0:10.2f}'.format(13.579)) This is my ten-character, two-decimal number: 13.58 Note that there are 5 spaces following the colon, and 5 characters taken up by 13.58, for a total of ten characters.

You can assign field lengths as arguments: print('{:<{}} goal'.format('field', 9)) field goal With manual field specification this becomes {0:<{1}s}

You can choose the padding character: print('{:-<9} goal'.format('field')) field---- goal

You can truncate (the opposite of padding): …and by argument: print('{:.5}'.format('xylophone')) print('{:.{}}'.format('xylophone',7)) xylop xylopho

Conversion tags enable output in either str, repr, or (in Python3) ASCII: { !s} { !r} { !a}

Format supports named placeholders (**kwargs), signed numbers, Getitem/Getattr, Datetime and custom objects. For more info: https://pyformat.info

10 REV 1020

String Formatting with Placeholders: (%s, %f et al) The oldest style of string formatting involves modulos as placeholders, and it is still supported in Python 3. The .format() method is usually preferable. Unlike %s placeholders, format variables may be used more than once in a string, and stored in any order. See above.

Placeholders: %s acts as a placeholder for a string, %d for a number print('Place my variable here: %s' %(string_name)) Note that %s converts whatever it's given into a string. print('Floating point number: %1.2f' %(13.145)) Floating point number: 13.14 where in 1.2, 1 is the minimum number of digits to return, and 2 is the number of digits to return past the decimal point. print('Floating point number: %11.4f' %(13.145)) Floating point number: 13.1450 There are 4 extra spaces (11 total characters including decimal)

NOTE: %s replicates the str() function, while %r replicates the repr() function

Passing multiple objects: print('First: %s, Second: %s, Third: %s' %('hi','two',3)) First: hi, Second: two, Third: 3 Variables are passed in the order they appear in the tuple. Not very pythonic because to pass the same variable twice means repeating it in the tuple. Use .format() instead (see above!)

Omitting the argument at the end causes the placeholder to print explicitly: print('To round 15.45 to 15.5 use %1.1f') To round 15.45 to 15.5 use %1.1f

…as does using %% (python sees this as a literal %) print('To round 15.45 to %1.1f use %%1.1f') %(15.45) To round 15.45 to 15.5 use %1.1f

Python has a known issue when rounding float 5's (up/down seem arbitrary) From https://docs.python.org/3/library/functions.html#round: "The behavior of round() for floats can be surprising: for example, round(2.675, 2) gives 2.67 instead of the expected 2.68. This is not a bug: it’s a result of the fact that most decimal fractions can’t be represented exactly as a float." For better performance, use the decimal module.

For more info on string formatting visit https://docs.python.org/2/library/stdtypes.html#string-formatting

Escape Characters: mystring = 'It's a nice day' returns an error mystring = 'It\'s a nice day' handles the embedded apostrophe \' = single quote, \" = double quote, \t = tab, \n = newline, \\ = backslash \ can also break code up into multiline statements for clarity

Note: embedded apostrophes are also handled by changing the apostrophe type mystring = "It's a nice day" is also valid

11 REV 1020

WORKING WITH LISTS:

Built-in List Functions: del mylist[1] removes the second item from the list len(mylist1) returns the number of objects in the list len(mylist1[-2]) returns the number of characters in the second-to-last string in the list, including spaces sorted(mylist1) returns a sorted copy of the list Remember: never use "list" as a variable name! Doing so breaks Python's built-in list() function. If you do inadvertently assign "list" as a variable name you can correct it by running del list.

Built-in List Methods: .append L.append(object) -- append object to end in place .count L.count(value) -> integer -- return number of occurrences of value .extend L.extend(iterable) -- extend list in place by appending elements from the iterable .index L.index(value, [start, [stop]]) -> integer -- return first index of value. Raises ValueError if the value is not present. .insert L.insert(index, object) -- insert object before index .pop L.pop([index]) -> item – remove and return item at index (default last). .remove L.remove(value) -- remove first occurrence of a value. Raises ValueError if the value is not present. .reverse L.reverse() -- reverses *IN PLACE* .sort L.sort(key=None, reverse=False) -- stable sort *IN PLACE*; In Jupyter, hit Tab to see a list of available methods for that object. Hit Shift+Tab for more information on the selected method - equivalent to help(L.method)

Adding & Removing List Items Adding objects to a list: list1.append('rhubarb') Adding multiple objects to a list: list1.extend('turnips','squash') Adding contents of one list to another: list1.extend(list2) adds the contents of list2 to list1 To add a list to another list as one object, use append: list1 = ['a','b'] list1.extend(['c','d']) returns ['a', 'b', 'c', 'd'] list1.append(['c','d']) returns ['a', 'b', ['c', 'd']] Note that .append() and .extend() change a list in place. These methods return a None value, so an assignment like newlist = list1.append(list2) is not valid code.

Inserting items into a list: list1.insert(3,'beets') puts 'beets' in the fourth position

Remove items from a list: list1.pop() returns the last (-1) item and permanently removes it list1.pop(1) returns the second item and permanently removes it

To capture a popped object: x = list1.pop()

List Index Method list.index(object) returns the index position of the first occurrence of an object in a list list1 = ['a','p','p','l','e'] list1.index('p') returns 1

To check the existence of a value in a list: object in list returns True/False as appropriate (names work too)

To join items use the string method: 'potato'.join(['one ',', two ','.']) returns 'one potato, two potato.'

12 REV 1020

Sorting Lists Lists can be sorted using either the sorted() built-in function, or the .sort() method. list1.sort() rewrites the list in alphabetical order IN PLACE list2 = sorted(list1) creates a new list while retaining the original list1.sort(reverse=True) reverse sorts the list list1.reverse() reverses the order of items in a list IN PLACE

Python sorts "ASCII-betically", which puts the uppercase alphabet ahead of lowercase: list2 = ['aaron','Abe','Tom','5'] sorted(list2) ['5', 'Abe', 'Tom', 'aaron']

Use the keyword argument key=str.lower to avoid this: sorted(list2, key=str.lower) or list2.sort(key=str.lower) ['5', 'aaron', 'Abe', 'Tom']

Making a List of Lists: list1, list2, list3 = [1,2,3], [4,5,6], [7,8,9] python supports multiple assignment! matrix = [list1,list2,list3] matrix [[1, 2, 3], [4, 5, 6], [7, 8, 9]] Note: “matrix” now points to the same list objects that the three variable names point to. If you later change one of the lists, matrix will ALSO be affected.

Slicing: matrix[0] returns [1,2,3] matrix[0][0] returns 1 (the first object inside the first object)

Reversing: matrix[1].reverse() modifies list2 in place. matrix becomes [[1, 2, 3], [6, 5, 4], [7, 8, 9]]

Slicing with a list comprehension (see next section for an explanation of list comprehensions): first_col = [row[0] for row in matrix] first_col returns [1,4,7]

13 REV 1020

LIST COMPREHENSIONS [expression for item in iterable (if condition)] – always returns a list Allows you to perform for loops within one set of brackets. [Jump to section on for loops] As a for loop: As a comprehension: chars = [] chars = [letter for letter in 'word'] for letter in 'word': print(chars) chars.append(letter) ['w', 'o', 'r', 'd'] print(chars) ['w', 'o', 'r', 'd'] list_of_squares = [x**2 for x in range(6)] list_of_squares [0, 1, 4, 9, 16, 25] even_numbers = [num for num in range(7) if num%2==0] even_numbers [0, 2, 4, 6]

Convert Celsius to Fahrenheit: celsius = [0,10,20.1,34.5] fahrenheit = [(temp*(9/5)+32) for temp in celsius] fahrenheit [32.0, 50.0, 68.18, 94.1]

Nested list comprehensions: fourth_power = [x**2 for x in [x**2 for x in range(6)]] fourth_power [0, 1, 16, 81, 256, 625]

14 REV 1020

WORKING WITH TUPLES Remember: Tuple elements cannot be modified once assigned. Tuples are immutable. tuple1 = (1,2,3) max(tuple1) returns 3, min(tuple1) returns 1 Note: commas define tuples, not parentheses. hank = 1,2 assigns the tuple (1,2) to hank

Built-in Tuple Methods (there are only 2) .count T.count(value) -> integer -- returns number of occurrences of value .index T.index(value, [start, [stop]]) -> integer -- returns first index of value Raises ValueError if the value is not present

WORKING WITH DICTIONARIES dict1 = {'Tom':7, 'Dick':4, 'Harry':23}

To update a value: dict1['Harry'] = 25 To increase a value: dict1['Harry'] += 100 the pythonic way to add/subtract/etc. value To add a new key: dict1['Fred'] = 42 To clear a dictionary: dict1.clear() keeps the dictionary, but now it’s empty of values To delete a dictionary: del dict1 technically deletes only the name – the dictionary object gets deleted once all references to it have been removed dict1.keys() returns an iterable object of 'Tom', 'Dick', 'Harry' list(dict1.keys()) returns the list ['Tom', 'Dick', 'Harry'] list(dict1.values()) returns [7,4,23] Remember, dictionaries retain insertion order! list(dict1.items()) returns [('Tom',7),('Dick',4),('Harry',23)] a list of tuples!

To add one dictionary to another: dict1.update(dict2) Nesting dictionaries: dict3 = {'topkey':{'nestkey':{'subkey':'fred'}}} dict3['topkey']['nestkey']['subkey'].upper() returns 'FRED' dict1.get(k[,d]) returns a value if a key exists, or a default value otherwise. d defaults to None (as opposed to throwing a KeyError)

To change a dictionary key from 'k1' to 'key1': d['key1'] = d.pop('k1')

Import the pprint module to display dictionaries in key order: import pprint pprint.pprint(dict1) displays {'Dick':4, 'Harry':23, 'Tom':7}

Dictionary Comprehensions: {key:value for key,value in iterable} used to create a dictionary {key:value for value,key in iterable} used if k,v appear in v,k order in iterable

Example: bikes = [('unicycle',1),('bicycle',2),('tricycle',3)] bikes_dict = {name:num for name,num in bikes} bikes_dict {'unicycle': 1, 'bicycle': 2, 'tricycle': 3}

15 REV 1020

WORKING WITH SETS: To declare an empty set: set1=set() because set1={} creates an empty dictionary

A list can be cast as a set to remove duplicates: set([2,1,2,1,3,3,4]) returns {1,2,3,4} items appear in order, though sets do not support indexing

A string can be cast as a set to isolate every character (case matters!): set('Monday 3:00am') returns {' ', '0', '3', ':', 'M', 'a', 'd', 'm', 'n', 'o', 'y'}

A dictionary may use sets to store values (consider an example of cocktail names and their ingredients)

Set Operators a = {1,2,3} b = {3,4,5} c = {2,3} 1 in a returns True set a contains 1 Intersection & .intersection() a&b returns {3} Union | .union() a|b returns {1,2,3,4,5} Difference - .difference() a-b returns {1,2} items in a but not in b Exclusive ^ .symmetric_difference() a^b returns {1,2,4,5} items unique to each set Subset <= .issubset() c<=a returns True Proper subset < c= .issuperset() a>=a returns True Proper superset < a>a returns False

Built-in Set Methods (Not for frozensets): S.add(x), S.remove(x), S.discard(x), S.pop(), S.clear() adds an item, removes an item by value, removes an item if present, removes and returns an arbitrary item, removes all items. Note: .remove throws a KeyError if the value is not found. .discard doesn't.

Frozensets Sets are mutable — their contents can be changed using methods like add() and remove(). Since it is mutable, a set has no hash value and cannot be used as either a dictionary key or as an element of another set. The frozenset type is immutable and hashable — its contents cannot be altered after it is created; it can therefore be used as a dictionary key or as an element of another set. set1 = {1,2,3} set3 set2 = {4,5,6} {frozenset({1, 2, 3}), frozenset({4, 5, 6})} set3 = {frozenset(set1)} set1 in set3 set3.add(frozenset(set2)} True

For more info visit https://docs.python.org/3/library/stdtypes.html#set-types-set-frozenset

RANGE is a generator in Python 3 – it returns values stepwise (see the section on Generators) range([start,] stop [,step]) range(10) generates values from 0 up to but not including 10 0, 1, 2, 3, 4, 5, 6, 7, 8, 9 range(1,10,2) 1, 3, 5, 7, 9

To create an actual list from a range: list(range(6)) [0, 1, 2, 3, 4, 5]

16 REV 1020

CONDITIONAL STATEMENTS & LOOPS If / Elif / Else statements: day = 'Tuesday' this step is called initializing a variable if day == 'Monday': print('Sunny') elif day == 'Tuesday': this step runs only when the above step returns False print('Cloudy') else: this step runs if ALL above steps return False print('Rainy') Cloudy you can also nest and & or in the statement

For Loops iterate through an entire set of data for i in range(0,3): here i is an arbitrary name that can only be used inside the for loop print(i) returns 0 1 2 Note that range(a,b) iterates from (a) to (b-1) for i in range(2,7,2): print(i) returns 2 4 6 list1 = [1,2,3,4,5] list_sum = 0 for num in list1: list_sum += num print(list_sum) note this "print" is outside the for loop 15 This is a clunky way to add items in a list!

While Loops run continuously until come criteria is met counter = 7 while counter < 10: BEWARE OF INFINITE LOOPS! print(counter) counter = counter + 1 alternatively, counter += 1 returns: 7 8 9

Note: while True: is a way to run a loop forever until some criteria is satisfied.

Nested Loops Prime number generator: for i in range (2,30): j = 2 j is assigned as a “divisor variable” counter = 0 while j < i: if i % j == 0: counter = 1 j = j + 1 else: j = j + 1 if counter == 0: print(str(i) + ' is a prime number') else: the “str” function converts integer i to a string counter = 0

17 REV 1020

Loop Control Statements (Break, Continue, Pass & Else) counter = 0 while counter < 7: counter += 1 if counter == 4: break print(counter) returns 1 2 3 then breaks out of the loop counter = 0 while counter < 7: counter += 1 if counter == 4: continue print(counter) returns 1 2 3 5 6 when counter=4, continue skips the rest of the loop and returns to the top counter = 0 while counter < 7: counter += 1 if counter == 4: pass print(counter) returns 1 2 3 4 5 6 pass is used as a placeholder for something you haven’t written yet for i in range(7): for i in range(7): if i == 4: if i == 8: break break print(i) print(i) else: else: print('finished') print('finished') returns 0 1 2 3 returns 0 1 2 3 4 5 6 finished Here, the else clause is part of the loop. It only executes if a for loop finishes, or if a while condition becomes false. If you break out of the loop, or if an exception is raised, it won't be executed.

Try and Except [Jump to Code Sample] You can circumnavigate entire batches of code without crashing/invoking Python’s error engine: try: code… except Exception as ex: captures the particular exception (requires the alias assignment) print(ex) except NamedException: if a particular exception occurs, do this print('Uh oh!') except: if any exception occurs, do this print('Uh oh!') else: if no exception occurs, then do this print('All good!') finally: do this regardless of exceptions print('Moving on!')

When try & except are run inside a loop, and a continue, break or return statement is issued, the finally clause still runs before affecting the behavior of the rest of the loop!

For more information, jump to Managing Exceptions

18 REV 1020

INPUT Asks for input from the user and converts whatever is typed into a string: x = input('What would you like to input? ') What would you like to input? If the user enters the number 4, then x equals the string '4' To convert input back to an integer use x = int(input('Enter input: ')) Use try & except to handle exceptions. To input a list, tuple or dictionary in its native type, jump here.

UNPACKING Tuple Unpacking Straightforward printing of tuples from a list of tuples: list1 = [(2,4),(6,8),(10,12)] for tup in list1: print(tup) (2, 4) (6, 8) (10, 12)

Pulling apart tuples: a technique used when working with coordinates coor = (3,8) x,y = coor x type(x), type(y) return "int", type(coor) returns "tuple" 3 y 8 y,x technically, tuples are defined by commas, not by parentheses (8, 3)

Unpacking the first item inside a list of tuples: Perform arithmetic on items inside tuples: for t1,t2 in list1: for t1,t2 in list1: print(t1) print(t1+t2) 2 6 6 14 10 22 t1 and t2 are arbitrary variable names that disappear once the for loops shut down

Dictionary Unpacking The .items() method creates a generator to separate keys & values d = {'k1':1,'k2':2,'k3':3} for k,v in d.items(): for k in d.items(): print(k) print(k) print(v) ('k1', 1) k1 ('k2', 2) 1 ('k3', 3) k2 2 k3 3 k and v are arbitrary variable names that disappear once the for loops shut down

19 REV 1020

FUNCTIONS DRY = “Don’t Repeat Yourself” def funcName(myname): NOTE: the variable "myname" is only used within the print('Hello,', myname) function – we have not initialized it as a global variable, funcName('Michael') so it can’t be called outside the function Hello, Michael def funcName(fname, lname): Functions accept multiple arguments """ It's always good practice to include a docstring This function returns a simple greeting. """ print('Hello,', fname, lname) funcName('Bilbo', 'Baggins') Hello, Bilbo Baggins def Amtwtax(cost): With Python, you don’t need to declare variable types. return cost * 1.0625 When passing "x + y", numbers are added, strings are concatenated print('spam') print(Amtwtax(7)) 7.4375 Note that "spam" did NOT print. Once a function returns something it shuts down

Default Parameter Values You can set a default value that is overridden if when another argument is presented to the function def funcName(fname='John', lname='Smith'): print('Hello,', fname, lname) Will print whatever is passed to the function, otherwise prints Hello, John Smith Never use mutable objects as default parameter values! (don't use a list, dictionary, set, etc.)

Positional Arguments *args and **kwargs Use *args to pass an open number of arguments to a function: def func1(*args): print(sum(args)) *args builds a tuple of arguments named 'args' func1(2,3,4,5) 14

Use **kwargs to pass keyworded arguments to a function: def func2(**kwargs): for key, value in kwargs.items(): dictionary unpacking! print(f'{key} == {value}') func2(happy='smile',sad='frown') happy == smile **kwargs builds a temporary dictionary named 'kwargs' sad == frown

Alternatively: def func3(**kwargs): for key in kwargs: print(f'{key} == {kwargs[key]}')

Note: only the asterisks * and ** are required. "args" and "kwargs" are just conventions.

20 REV 1020

PRE-DEFINED FUNCTIONS For a list of pre-defined functions, see https://docs.python.org/3/library/functions.html DON'T USE EXISTING NAMES WHEN CREATING FUNCTIONS! len() returns the number of items in an iterable (list, tuple, etc) or the number of characters in a string bool() returns “True” if populated, “False” if zero or empty abs() returns absolute value pow() is an exponentiator. pow(2,4) returns 16. hex() and bin() convert numbers to hexadecimal and binary, respectively. hex(43) returns '0x2b' bin(43) returns '0b101011' round() rounds to a specified number of digits (default=0). 5 rounds to the nearest even digit (usually.) print() in Python 3, print is a function. It accepts keyword arguments like sep (to change the default space character) and end (to change the default newline). print() always returns a None value. dir() returns every applicable function for that object dir([]) returns all functions that can be used with lists & arrays help() returns help on the application of a specific function against a specific object help(list1.count) tells you that count returns the number of occurrences of value (where list1 is a variable previously set up in our program)

LAMBDA EXPRESSIONS – used for writing ad hoc, anonymous functions, without the overhead of def • lambda's body is a single one-line expression (not a block of statements) • lambda is well-suited for coding simple functions (def handles the larger tasks)

Converting def to lambda: DEF LAMBDA def square(num): lambda num: num**2 return num**2 square(num)

With def, you have to assign a name to the function, and call it explicitly. Lambda expressions can be assigned a name (eg. square = lambda num: num**2), but usually they're just embedded. Lambdas work well inside of functions like map(), filter() and reduce()

Example: a lambda expression to check if a number is even num = 9 lambda num: num%2==0 Note: lambdas need to return something (here it returns False)

For further reading: Iterating with Python Lambdas http://caisbalderas.com/blog/iterating-with-python-lambdas/ Describes use of the map() function with lambdas.

For further reading: Python Conquers the Universe https://pythonconquerstheuniverse.wordpress.com/2011/08/29/lambda_tutorial/ Does a nice job of explaining how expressions (that return something) differ from assignment statements (that don't) Lambdas can include functions [including print() ], list comprehensions, conditional expressions: lambda x: 'big' if x > 100 else 'small'

21 REV 1020

MORE USEFUL FUNCTIONS

MAP map() is a generator; use list(map( to immediately see its output map(function, sequence) applies a function to elements in the sequence, and yields new values temp = [0, 22.5, 40, 100] def fahrenheit(T): return (9/5)*T + 32) map(fahrenheit, temp) don't put parentheses after "fahrenheit" – you're calling 32.0, 72.5, 104.0, 212.0 the fahrenheit function object, not its output map(lambda T: (9/5)*T+32, temp) use lambda in place of declared functions 32.0, 72.5, 104.0, 212.0 a,b,c = [1,2,3],[4,5,6],[7,8,9] list(map(lambda x,y,z: x+y+z, a,b,c)) you can pass multiple arguments to lambda so long as the same [12, 15, 18] number of arguments are passed in sequence

FILTER filter() is a generator; use list(filter( to immediately see its output filter(function, sequence) returns only those elements for which a function returns True. list1 = range(10) filter(lambda x: x%2==0, list1) 0 … 2 … 4 … 6 … 8

REDUCE reduce() was deprecated in Python 3, but is still available from functools reduce(function, sequence) continually applies a function to a sequence and returns a single value list1 = [47,11,42,13] from functools import reduce reduce(lambda x,y: x+y, list1) the math: (47+11=58) … (58+42=100) … (100+13=113) 113 reduce(lambda a,b: a if (a>b) else b, list1) works like max(list1)

ZIP zip() makes an iterator that aggregates elements from each of the iterables. It stops after the shortest input iterable is exhausted. With no arguments it returns an empty iterator. x,y = [1,2,3],[4,5,6,7] list(zip(x,y)) [(1, 4), (2, 5), (3, 6)]

22 REV 1020

ENUMERATE enumerate(sequence,[start=]) returns a tuple in the form (position, item) list1 = ['a','p','p','l','e'] for x,y in enumerate(list1): print(x,y) 0 a 1 p 2 p 3 l 4 e

Alternatively: list(enumerate(list1)) [(0, 'a'), (1, 'p'), (2, 'p'), (3, 'l'), (4, 'e')] list(enumerate(list1, start=2)) [(2, 'a'), (3, 'p'), (4, 'p'), (5, 'l'), (6, 'e')]

ALL & ANY all(iterable) returns True if every element is true. any(iterable) returns True if any element is true.

COMPLEX complex() accepts either a string or a pair of numbers, returns a complex number complex(2,3) returns (2+3j) complex('4+5j') returns (4+5j)

23 REV 1020

PYTHON THEORY & DEFINITIONS SCOPE Variable names are stored in a namespace Variable names have a scope that determines their visibility to other parts of code. Variables declared inside functions and loops have a local scope – they can be called from within the function or loop, but disappear once the function or loop shuts down. Variables declared in the body of the program (outside of functions and loops) have a global scope. They can be called by loops, and by functions if certain criteria are met. Whenever a variable name is invoked, Python follows the LEGB rule to determine which scope applies. LEGB Rule: L: Local — Names assigned in any way within a function (def or lambda), and not declared global in that function. E: Enclosing function locals — Names in the local scope of any and all enclosing functions (def or lambda), from inner to outer. G: Global (module) — Names assigned at the top-level of a module file, or declared global in a def within the file. B: Built-in (Python) — Names preassigned in the built-in names module: open, range, SyntaxError, etc. x = 25 def printer(): x = 50 return x print(printer()) 50 x inside the function is local (x=50) print(x) 25 x outside the function is global, and is unchanged by the function (x=25) x = 25 def printer(): global x this calls global x into the function! x = 50 this reassigns the global x return x print(printer()) 50 print(x) 50

Use globals() and locals() to see current global & local variables Return variable names only with globals().keys()

In place – "Strings are immutable; you can't change a string in place." (string[0]='n' doesn't work) String (not mutable in place) List (mutable in place) a='crackerjack' b= ['joe', 'ted'] a.replace('cr','sn') b.reverse() 'snackerjack' b a ['ted', 'joe'] 'crackerjack' Note that a.replace('cr','sn') returned 'snackerjack' without prompting, but a is unchanged. Sequenced – object elements have an established order (offset), and can be sliced Iterable – object contains any series of elements that can be called one-at-a-time. Sets are iterable, but not sequenced. Mutable – most containers (lists, dictionaries and sets) can be changed. Most scalars (integers, floats and strings) are immutable. Tuples are also immutable, although mutable objects contained in tuples can still be changed. For a good discussion visit https://codehabitude.com/python-objects-mutable-vs-immutable/ Statement - del is a python statement, not a function or method. It's sort of the reverse of assignment (=): it detaches a name from a python object and can free up the object's memory if that name was the last reference to it. Stack – using .append() to add items to the end of a list and .pop() to remove them from the same end creates a data structure known as a LIFO queue. using .pop(0) to remove items from the starting end is a FIFO queue. These types of queues are called stacks.

24 REV 1020

FUNCTIONS AS OBJECTS & ASSIGNING VARIABLES If you define a function and then assign another name to that function (output is in blue): def hello(name='Fred'): return 'Hello '+name hello() 'Hello Fred'

greet = hello greet greet() 'Hello Fred' Note that the assignment points "greet" to the original function. If we delete "hello", "greet" still works! Also, if we define a new function named "hello", greet will work the old way, and hello will work the new way.

Returning functions inside of functions – consider the following: def hello(name='Fred'): def greet(): print('This is inside the greet() function') def welcome(): print('This is inside the welcome() function') if name == 'Fred': return greet note no parentheses – return the function, not its output else: return welcome x = hello() we could call hello()() instead x() This is inside the greet() function x x is assigned to greet because name == 'Fred' x = hello('notFred') pass any name except 'Fred' x() This is inside the welcome() function x x is assigned to welcome because name != 'Fred'

When x was assigned to hello(), Python ran hello() and followed its instructions – it said "return this function's greet function to x". As soon as hello() finished, greet, welcome & name were cleared from memory! x remains as a global variable, and (until it's reassigned) it still has a copy of the embedded greet function as its object. x is unchanged even if we change hello and run hello() elsewhere in our program. The only way to change x is to run x=hello('notFred'), run x=hello() on a changed hello, or assign x to a new object entirely.

FUNCTIONS AS ARGUMENTS def hello(): return 'Hi Fred!' we carefully said "return" here, to return a string when called def other(func): Here other is expecting an argument before it executes print('Other code would go here') print(func()) print whatever the executed function returns other(hello) Here the hello function was passed as an argument to other Other code would go here Hi Fred!

25 REV 1020

DECORATORS: Decorators can be thought of as functions which modify the functionality of another function. They help to make your code shorter and more "pythonic". Useful when working with web frameworks like Django and Flask. def new_decorator(func): new_decorator is looking for a function as an argument def wrap_func(): print('Code could be here, before executing the function') func() print('Code here will execute after the function') return wrap_func returns the decorator's function, not its output

@new_decorator the @ symbol invokes the decorator def func_needs_decorator(): print(' This function is in need of a Decorator') func_needs_decorator() Code could be here, before executing the function This function is in need of a Decorator Code here will execute after the function

In the above code the decorator returned a function - it added functionality to the func_needs_decorator function we defined in the second step. Calling func_needs_decorator() ran both the function and the decorator code.

Alternatively we can write a decorator without wrap_func: def new_decorator(func): print('Code could be here, before executing the function') func() print('Code here will execute after the function')

@new_decorator def func_needs_decorator(): print(' This function is in need of a Decorator') Code could be here, before executing the function This function is in need of a Decorator Code here will execute after the function this decorator returns output immediately! Whenever a function is assigned to a variable, the function runs, and whatever it returns is passed to the variable.

For further reading: http://simeonfranklin.com/blog/2012/jul/1/python-decorators-in-12-steps/

26 REV 1020

GENERATORS & ITERATORS Rather than return a fully constructed iterable, generators yield values one at a time. When called they don't actually return a value and then exit. Instead, the generator functions will automatically suspend and later resume their execution and state around the last point of value generation. This feature is known as state suspension, and it saves memory space in for loops. Functions become generators by using yield in place of return.

Example: Generate a Fibonnaci sequence up to n

GENERATOR: ITERATOR: def genfibon(n): def fibon(n): a = 1 a = 1 b = 1 b = 1 output = [] for i in range(n): for i in range(n): yield a output.append(a) a,b = b,a+b a,b = b,a+b return output for num in genfibon(10): fibon(10) print(num) returns: 1 1 2 3 5 8 13 21 34 55 returns: [1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Notice that if we call some huge value of n (like 100000) the iterator function will have to keep track of every single result, when in our case we actually only care about the previous result to generate the next one!

NEXT & ITER built-in functions: next() is used to walk through a generator: def simple_gen(): next(g) returns 0 for x in range(3): next(g) returns 1 yield x next(g) returns 2 g = simple_gen() next(g) raises a StopIteration exception iter() turns an iterable into a iterator: s = 'hello' next(s_iter) returns 'h' s_iter = iter(s) next(s_iter) returns 'e' next(s_iter) returns 'l' and so on

GENERATOR COMPREHENSIONS Used just like list comprehensions, except they don't retain values. Use parentheses. my_list = [1, 3, 5, 9, 2, 6] filtered_gen = (item for item in my_list if item > 3) next(filtered_gen) 5 next(filtered_gen) 9 next(filtered_gen) 6

27 REV 1020

WORKING WITH FILES testFile = open('c:\\test.txt') testFile is now a "file" object that holds the contents of our text file. The open function opens test.txt in "plaintext read mode", you can't write to or modify the source file in any way testFile.read() returns the contents as a single string in quotes. \n appears in place of line breaks. NOTE: the pointer is now at the END of our text file. Repeating the read function returns nothing. testFile.readlines() returns each line as a separate string inside a list (with linebreaks) testFile.tell() returns the current character position of the pointer in our file testFile.seek(0,0) repositions 0 bytes of data from our pointer to the 0 position (beginning) of the file testFile.seek(0) does the same thing. testFile.close() closes the test.txt file

READING AND WRITING TO FILES The open function takes 3 parameters: filename, access mode & buffer size testFile = open('test.txt','w') opens the file in "write mode" and destroys the original! testFile.write('new text') REPLACES the contents of the file with the string 'new text' testFile = open('test.txt','a') allows appending to the file (a plus is needed to read the file) testFile.write('\nnew text') ADDS the words 'new text' to the end of the existing file (with a line break)

Note: 'w' and 'a' are write-only conditions, use 'w+' and 'a+' for reading and writing. 'w' and 'w+' truncate the original file to zero length. A file is created if it doesn't exist. 'a' and 'a+' open the file and put the pointer at the end (so anything written is appended). A file is created if it doesn't exist. 'r+' opens the file in read/write, but places the pointer at the beginning. Anything written REPLACES the beginning of the file, and leaves the pointer where it stopped writing.

RENAMING & COPYING FILES with the OS MODULE testFile = open('test.txt') import os imports the operating system module os.rename('test.txt','test2.txt') test.txt has been renamed to text2.txt testFile.close() testFile = open('test2.txt') newFile = open('test3.txt','w') creates a new file test3.txt and allows writing to the file (note: 'newFile' is just an arbitrary name for our variable) newFile.write(testFile.read())

CREATING BINARY FILES with the SHELVE MODULE import shelve shelfFile = shelve.open('mydata') creates a set of files called "mydata" Note that in Windows, mydata exists as three binary files in the working directory: myadata.bak, mydata.dat, mydata.dir shelfFile['dogs'] = The "shelf" object behaves like a dictionary with keys and values, and has similar methods: list(shelfFile.keys()) creates a list of key names list(shelfFile.values()) creates a list of object values

28 REV 1020

ORGANIZING FILES with the SHUTIL MODULE The Shell Utilities module lets you copy, move, rename and delete files in Python programs. import shutil shutil.copy('c:\\file.txt', 'c:\\folder') copies a file from the root into a folder 'c:\\folder\\file.txt' …and returns the path of the new file shutil.copy('c:\\file.txt', 'c:\\folder\\file2.txt') copies and renames in one step 'c:\\folder\\file2.txt' shutil.copytree('c:\\folder', 'c:\\backup_folder') copies a folder and all of its contents shutil.move('c:\\file.txt', 'c:\\folder') moves a file from the root into a folder shutil.move('c:\\file.txt', 'c:\\file2.txt') RENAMES a file in place

DELETING FOLDERS AND FILES Using the OS module: import os os.unlink('file.txt') deletes a file from the current working directory os.rmdir('c:\\folder') removes the "folder" directory ONLY IF EMPTY Using the SHUTIL module: import shutil shutil.rmtree('c:\\folder') deletes the "folder" directory AND ALL CONTENTS Note that these are permanent deletes, not moves to a recycle bin! TIP: Perform a dry run by commenting out the delete code (# os.unlink…) and printing the target files instead.

Using the Send2Trash module: import send2trash may require pip install send2trash from the command line send2trash.send2trash('c:\\file.txt') moves a file to the recycle bin

WALKING A DIRECTORY TREE import os for folder, subfolders, filenames in os.walk('c:\\folder'): The variable names are arbitrary. The walk method is a generator that returns three values on each iteration: the current folder's name, a list of strings for this folder's subfolders, and a list of strings for this folder's files. The generator returns an empty string if it finds no subfolders or files. This is useful for performing some task across all the files & folders throughout the tree.

29 REV 1020

MANAGING EXCEPTIONS [Jump to Common Exceptions] Raising Exceptions Python raises a specific exception when it can't execute code, and provides a Traceback: >>> 1/0 Traceback (most recent call last): File "", line 1, in 1/0 ZeroDivisionError: division by zero

In Jupyter: In [1]: 1/0 ------ZeroDivisionError Traceback (most recent call last) in () ----> 1 1/0

ZeroDivisionError: division by zero

Raising Your Own Exception >>> raise Exception('This is an error message.') Traceback (most recent call last): File "", line 1, in raise Exception('This is an error message.') Exception: This is an error message.

In Jupyter: In [1]: raise Exception('This is an error message.') ------Exception Traceback (most recent call last) in () ----> 1 raise Exception('This is an error message.')

Exception: This is an error message.

Tracebacks (aka "call stacks") provide detail on where a function was called, and where the exception occurred. The traceback can be captured as a string value using the traceback module: import traceback try: raise Exception('This is an error message') except: print('The traceback info is:\n'+traceback.format_exc()) This example isn't very useful – it does what raise would have done anyway. For more info visit https://docs.python.org/3/library/traceback.html For a code sample that captures tracebacks to a file, [Jump to the Tracebacks code sample]

Assertions [Jump to the Stoplight code sample] Use an assert statement to run a logical test. If False, raise an AssertionError with text. [code] assert [logical test], 'Text if test returns False'+other data Traceback (most recent call last): File info AssertionError: Text if test returns False NOTE: Use assertions to trace programming errors (as occasional sanity checks). User errors should be handled by an exception.

30 REV 1020

Logging [Jump to the Buggy Factorial code sample] import logging logging.basicConfig(level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s') this sets up logging logging.debug('Text message et al') this generates text strings as code is run logging.disable(logging.CRITICAL) disables the automatic printing of ALL logging levels

NOTE: There are five built-in logging levels: logging.debug() (lowest) logging.info() logging.warning() logging.error() logging.critical() Logging to a text file logging.basicConfig(filename='myProgramLog.txt', level=logging.DEBUG, format='%(asctime)s - %(levelname)s - %(message)s')

For more info: https://docs.python.org/3/library/logging.html

Create your own Exception class: class MyException(Exception): pass ...and then, in the code: if thing meets criteria: raise MyException(thing)

For more info on built-in exceptions: https://docs.python.org/3/library/exceptions.html

31 REV 1020

OBJECT ORIENTED PROGRAMMING – Classes, Attributes & Methods Classes: Python lets us define our own object classes and methods using the class keyword An attribute is a characteristic of an object. A method is an operation we can perform on the object. Built-in Types: we've already seen Python's built-in objects: int, float, str, list, tuple, dict, set, function Instances: the number 1 is an instance of the int class. When we assign a name to an object, we're said to have intantiated the object class.

Create a new object type: class Sample: by convention, class names start with a capital letter thing1 = 'abc' "thing1" is a class variable shared by all instances def __init__(self,thing2): self.thing2 = thing2 "thing2" is an instance variable unique to each instance def method1(self): "method1" is a public method print(self.thing2) Note: we can't say "print(thing2)" because thing2 is not an object, it's an attribute attached to an object def _method2(self): _method2 is a private method (note the single underscore) print('Hi there!') Public methods are visible by hitting Tab. Private methods aren't. x = Sample('def') here we "instantiate" the class by giving it an instance

Attributes: class attributes are accessed using standard obj.name syntax: x.thing1 returns 'abc' every Sample object will have a thing1 attribute Note we don't use parentheses when calling attributes x.thing2 returns 'def' this is the argument we passed when creating x

Attributes can be modified by assignment: x.thing2 = 'xyz'

Methods: Methods are functions defined inside the body of a class. They are used to perform operations with the attributes of our objects. Methods are an essential encapsulation concept of the OOP paradigm. This is essential in dividing responsibilities in programming, especially in large applications. You can basically think of methods as functions acting on an Object that take the Object itself into account through its self argument. Methods that start with a single underscore are private methods; they can't be seen with the Tab key, although they can be seen with dir() for debugging purposes. x.method1() returns def x._method2() returns Hi there!

Inheritance: Inheritance is a way to form new classes using classes that have already been defined. The newly formed classes are called derived classes, the classes that we derive from are called base classes. Important benefits of inheritance are code reuse and reduction of complexity of a program. The derived classes (descendants) override or extend the functionality of base classes (ancestors).

Special Methods (aka Magic Methods): Classes in Python can implement certain operations with special method names. These methods are not actually called directly but by Python specific language syntax (double underscore). They allow us to use specific functions on objects in our class. For more info: https://github.com/RafeKettler/magicmethods/blob/master/magicmethods.markdown

For Further Reading: Jeff Knupp's Post Tutorials Point Official Documentation

32 REV 1020

Example 1: class Dog: species = 'mammal' here we assign a Class Object Attribute (all instances share this attribute) def __init__(self,breed): here we initialize an attribute "breed" self.breed = breed this calls for the attribute "breed" anytime we create a "Dog" object def bark(self): here we define a method ".bark()" print("Woof!") sam = Dog(breed='Lab') frank = Dog(breed='Huskie') sam.breed | frank.breed Note: there aren't any parentheses after ".breed" because 'Lab' | 'Huskie' it is an attribute and doesn't take any arguments sam.species | frank.species species is also an attribute (no parentheses) shared by all instances 'mammal' | 'mammal' sam.bark() Woof!

Example 2 – Methods: class Circle: pi = 3.14

def __init__(self, radius=1): self.radius = radius Note: by setting radius=1 in the __init__, we don't require an argument when creating a Circle. x=Circle() creates a Circle object with radius 1, y=Circle(2) creates a Circle object with radius 2.

def area(self): return self.radius * self.radius * Circle.pi We can't return radius * radius * pi because radius and pi aren't objects, they're attributes Also, area is not an attribute! Circle.area will return an error. Only Circle.area() will return a value

def setRadius(self, radius): self.radius = radius This turns the assignment statement .radius=2 into a method .setRadius(2)

def getRadius(self): return self.radius This turns the attribute call .radius into a method .getRadius() c = Circle() Since radius was assigned a default=1, it's not a required argument here c.setRadius(2) print('Radius is: ', c.getRadius()) Radius is: 2 print('Area is: ', c.area()) Area is: 12.56

33 REV 1020

Example 3 – Inheritance: class Animal: def __init__(self): print("Animal created") anytime an Animal is created, print "Animal created" note that we didn't need to initialize any attributes def whoAmI(self): print("Animal") def eat(self): print("Eating") class Dog(Animal): here we inherit all of Animal's attributes and methods def __init__(self): here we say "whenever a Dog is created.. Animal.__init__(self) …initialize whatever Animal would have initialized… print("Dog created") …and print "Dog created" def whoAmI(self): here we override the Animal .whoAmI method print("Dog") def bark(self): here we introduce a new method unique to Dog print("Woof!") d = Dog() Animal created Dog created d.whoAmI() Dog d.eat() the .eat method was inherited from Animal Eating d.bark() Woof!

In this example, we have two classes: Animal and Dog. The Animal is the base class, the Dog is the derived class.

The derived class inherits the functionality of the base class (as shown by the eat() method). The derived class modifies existing behaviour of the base class (as shown by the whoAmI() method). Finally, the derived class extends the functionality of the base class, by defining a new bark() method.

In Python 2 you often saw class definitions with an (object) argument, as in class Dog(object): This made sure the class followed the "new style object" rules. In Python 3 all objects are new style, so (object) can be omitted.

34 REV 1020

Example 4 – Special Methods: class Book(object): def __init__(self, title, author, pages=0): 0 is a default pages value print("A book is created") self.title = title self.author = author self.pages = pages

def __str__(self): return "Title: {}, author: {}, pages: {}"\ \ allows multiline entry .format(self.title, self.author, self.pages)

def __len__(self): return self.pages

def __del__(self): print("A book is destroyed") book1 = Book("Python Rocks!", "Jose Portilla", 159) A book is created print(book1) "print" works now because the __str__ method enabled it and we told it what to return Title: Python Rocks!, author: Jose Portilla, pages: 159 print(len(book1)) note that just len(book) doesn't do anything visibly 159 del book1 this deletes the book object, then prints something A book is destroyed book2 = Book("Writer's Block", "Michael Brothers") we don't have to pass a "pages" value A book is created print(book2) Title: Writer's Block, author: Michael Brothers, pages: 0

The __init__(), __str__(), __len__() and the __del__() methods: These special methods are defined by their use of double underscores. They allow us to use Python specific functions on objects created through our class.

35 REV 1020

MODULES & PACKAGES A module is a file containing Python definitions and statements. The file name is the module name with the suffix .py A package is a set of modules organized into folders and subfolders. Any folder with an __init__.py file is considered a package.

 shuffle __init__.py  numbers __init__.py revsort.py rndshffle.py  strings __init__.py revsort.py rndshffle.py

Here we have a package called "shuffle". To use it in our Python program we would say import shuffle Then, to use any submodule in our program we would say shuffle.numbers.revsort(597) which might return 975 I made this up, don’t try it at home!

Alternatively, if we know we'll only need a particular submodule (or to save keystrokes) we can say: from shuffle.numbers import revsort Then to call this module we could say simply revsort(597)

But be careful! In our example, some submodules share the same name. To avoid collisions use an alias: from shuffle.numbers import rndshffle as nr from shuffle.strings import rndshffle as sr so that nr() and sr() access different submodules

As a real-life example, you can import the math module using import math To access a specific function: math.sqrt(4) In Jupyter, hit Tab after math. to see a list of available functions for the module.

Alternatively, you can import specific functions from a module: from math import sqrt Then to access a function it's just: sqrt(4) For more information: http://docs.python.org/3/tutorial/modules.html#packages

36 REV 1020

COLLECTIONS Module: The collections module is a built-in module that implements specialized container datatypes providing alternatives to Python’s general purpose built-in containers. We've already gone over the basics: dict, list, set, and tuple.

Counter is a dict subclass which helps count hashable objects. Inside of it elements are stored as dictionary keys and the counts of the objects are stored as the value.

Counter() with lists: from collections import Counter note the uppercase "C" in Counter list1 = [1,2,2,2,2,3,3,3,2,2,1,12,3,2,32,1,21,1,223,1] Counter(list1) Counter({2: 7, 1: 5, 3: 4, 32: 1, 12: 1, 21: 1, 223: 1}) returns a Counter object

Counter() with strings: Counter('aabsbsbsbhshhbbsbs') Counter({'b': 7, 's': 6, 'h': 3, 'a': 2})

Counter(s.split()) counts words in a sentence (be careful with case and punctuation)

Counter('If you read you are well read'.split()) Counter({'read': 2, 'you': 2, 'well': 1, 'are': 1, 'If': 1})

Counter methods c = Counter('abacab') print(c) Counter({'a': 3, 'b': 2, 'c': 1}) c.most_common(2) note that methods act on objects. Counter('abacab').most_common(2) also works. [('a', 3), ('b', 2)]

Common patterns when using the Counter() object: sum(c.values()) total of all counts c.clear() reset all counts list(c) list unique elements set(c) convert to a set dict(c) convert to a regular dictionary c.items() convert to a list of (elem, cnt) pairs Counter(dict(list_of_pairs)) convert from a list of (elem, cnt) pairs c.most_common()[:-n-1:-1] n least common elements (using string notation) c += Counter() remove zero and negative counts defaultdict is a dictionary-like object which provides all methods provided by dictionary but takes a first argument (default_factory) as default data type for the dictionary. In other words, a defaultdict will never raise a KeyError. Any key that does not exist gets the value returned by the default factory. Using defaultdict is faster than doing the same using the dict.set_default method. from collections import defaultdict d = defaultdict(object) d['key1']