Generators & Coroutines Part 1: Iterators
Total Page:16
File Type:pdf, Size:1020Kb
Generators & Coroutines Part 1: Iterators Edited Version of Slides by David Beazely http://www.dabeaz.comPart I Introduction to Iterators and Generators Monday, May 16, 2011 Monday, May 16, 2011 Copyright (C) 2008, http://www.dabeaz.com 1- 11 Iteration Iterating over a Dict As you know, Python has a "for" statement • • If you loop over a dictionary you get keys You use it to loop over a collection of items • >>> prices = { 'GOOG' : 490.10, >>> for x in [1,4,5,10]: ... 'AAPL' : 145.23, ... print x, ... 'YHOO' : 21.71 } ... ... 1 4 5 10 >>> for key in prices: >>> ... print key ... And, as you have probably noticed, you can YHOO • GOOG iterate over many different kinds of objects AAPL (not just lists) >>> Copyright (C) 2008, http://www.dabeaz.com 1- 12 Copyright (C) 2008, http://www.dabeaz.com 1- 13 Iterating over a String • If you loop over a string, you get characters >>> s = "Yow!" >>> for c in s: ... print c ... Y o w ! >>> Copyright (C) 2008, http://www.dabeaz.com 1- 14 Iterating over a Dict • If you loop over a dictionary you get keys >>> prices = { 'GOOG' : 490.10, ... 'AAPL' : 145.23, ... 'YHOO' : 21.71 } ... >>> for key in prices: ... print key ... YHOO GOOG AAPL >>> Copyright (C) 2008, http://www.dabeaz.com 1- 13 Iterating over a String Iterating over a File • If you loop over a file you get lines If you loop over a string, you get characters >>> for line in open("real.txt"): • ... print line, ... >>> s = "Yow!" Iterating over a File Real Programmers write in FORTRAN >>> for c in s: ... print c Maybe they do now, ... If you loop over a file you get lines in this decadent era of • Y Lite beer, hand calculators, and "user-friendly" software o >>> for line in open("real.txt"): but back in the Good Old Days, w ... print line, when the term "software" sounded funny ! ... and Real Computers were made out of drums and vacuum tubes, >>> Real Programmers write in FORTRAN Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Maybe they do now, Machine Code. in this decadent era of Raw, unadorned, inscrutable hexadecimal numbers. Lite beer, hand calculators, and "user-friendly" software Directly. but back in the Good Old Days, when the term "software" sounded funny 1- 14 Copyright (C) 2008, http://www.dabeaz.com 1- 15 Copyright (C) 2008, http://www.dabeaz.com and Real Computers were made out of drums and vacuum tubes, Real Programmers wrote in machine code. Not FORTRAN. Not RATFOR. Not, even, assembly language. Machine Code. Raw, unadorned, inscrutable hexadecimal numbers. Directly. 1- 15 Copyright (C) 2008, http://www.dabeaz.com Consuming Iterables • Many functions consume an "iterable" object • Reductions: Consuming Iterables sum(s),Iteration min(s), max(s) Protocol TheConstructors reason why you can iterate over different Many functions consume an "iterable" object •• • objectslist(s), is thattuple(s), there set(s), is a specific dict(s) protocol Reductions: • >>>in operatoritems = [1, 4, 5] • >>> it = iter(items) sum(s), min(s), max(s) >>>item it.next() in s Constructors 1 • >>>Many it.next() others in the library • 4 list(s), tuple(s), set(s), dict(s) >>> it.next() in operator 5 • Copyright (C) 2008, http://www.dabeaz.com>>> it.next() 1- 16 Traceback (most recent call last): item in s File "<stdin>", line 1, in <module> StopIteration • Many others in the library >>> Copyright (C) 2008, http://www.dabeaz.com 1- 16 Copyright (C) 2008, http://www.dabeaz.com 1- 17 Iteration Protocol • An inside look at the for statement for x in obj: # statements • Underneath the covers _iter = iter(obj) # Get iterator object while 1: try: x = _iter.next() # Get next item except StopIteration: # No more items break # statements ... • Any object that supports iter() and next() is said to be "iterable." Copyright (C) 2008, http://www.dabeaz.com 1-18 Iteration Protocol • The reason why you can iterate over different objects is that there is a specific protocol >>> items = [1, 4, 5] >>> it = iter(items) >>> it.next() 1 >>> it.next() 4 >>> it.next() 5 >>> it.next() Traceback (most recent call last): File "<stdin>", line 1, in <module> StopIteration >>> Copyright (C) 2008, http://www.dabeaz.com 1- 17 Iteration Protocol Supporting Iteration • An inside look at the for statement for x in obj: • User-defined objects can support iteration # statements Example: Counting down... Underneath the covers • • Supporting Iteration >>> for x in countdown(10): _iter = iter(obj) # Get iterator object ... print x, while 1: ... User-defined try: objects can support iteration 10 9 8 7 6 5 4 3 2 1 • x = _iter.next() # Get next item >>> except StopIteration: # No more items • Example: break Counting down... To do this, you just have to make the object # statements • >>> ...for x in countdown(10): ... print x, implement __iter__() and next() Any... object that supports iter() and next() is • 10 9 8 7 6 5 4 3 2 1 said>>> to be "iterable." To do this, you just have to make the object Copyright (C) 2008,• http://www.dabeaz.com 1-18 Copyright (C) 2008, http://www.dabeaz.com 1-19 implement __iter__() and next() Copyright (C) 2008, http://www.dabeaz.com 1-19 Supporting Iteration • Sample implementation class countdown(object): Supporting Iteration Iteration def __init__(self,start): Example self.count = start def __iter__(self): return self def next(self): • Sample implementation if self.count <= 0: raise StopIteration class countdown(object): Example r use:= self.count def __init__(self,start): • >>> c =self.count countdown(5) -= 1 self.count = start return r def __iter__(self): >>> for i in c: return self ... print i, def next(self): ... if self.count <= 0: 5 4 3 2 1 raise StopIteration >>> r = self.count Copyright (C) 2008, http://www.dabeaz.com 1-20 self.count -= 1 return r Copyright (C) 2008, http://www.dabeaz.com 1-20 Copyright (C) 2008, http://www.dabeaz.com 1-21 Iteration Commentary • There are many subtle details involving the design of iterators for various objects • However, we're not going to cover that • This isn't a tutorial on "iterators" • We're talking about generators... Copyright (C) 2008, http://www.dabeaz.com 1-22 Iteration Example • Example use: >>> c = countdown(5) >>> for i in c: ... print i, ... 5 4 3 2 1 >>> Copyright (C) 2008, http://www.dabeaz.com 1-21 Iteration Commentary • There are many subtle details involving the design of iterators for various objects • However, we're not going to cover that Part 2: Generators • This isn't a tutorial on "iterators" • We're talking about generators... Copyright (C) 2008, http://www.dabeaz.com 1-22 Monday, May 16, 2011 Generators Generator Functions • A generator is a function that produces a • The function only executes on next() sequence of results instead of a single value >>> x = countdown(10) >>> x def countdown(n): <generator object at 0x58490> while n > 0: >>> x.next() Function starts yield n Counting down from 10 executing here n -= 1 10 >>> for i in countdown(5): >>> ... print i, yield produces a value, but suspends the function ... • 5 4 3 2 1 Function resumes on next call to next() >>> • >>> x.next() Instead of returning a value, you generate a 9 • >>> x.next() series of values (using the yield statement) 8 >>> Copyright (C) 2008, http://www.dabeaz.com 1-23 Copyright (C) 2008, http://www.dabeaz.com 1-25 Generators Generator Functions • Behavior is quite different than normal func • Calling a generator function creates an generator object. However, it does not start • When the generator returns, iteration stops running the function. >>> x.next() 1 def countdown(n): >>> x.next() print "Counting down from", n Traceback (most recent call last): while n > 0: File "<stdin>", line 1, in ? yield n StopIteration n -= 1 Notice that no >>> output was >>> x = countdown(10) produced >>> x <generator object at 0x58490> >>> Copyright (C) 2008, http://www.dabeaz.com 1-24 Copyright (C) 2008, http://www.dabeaz.com 1-26 Generator Functions • The function only executes on next() >>> x = countdown(10) >>> x <generator object at 0x58490> >>> x.next() Counting down from 10 Function starts 10 executing here >>> • yield produces a value, but suspends the function • Function resumes on next call to next() >>> x.next() 9 >>> x.next() 8 >>> Copyright (C) 2008, http://www.dabeaz.com 1-25 Generator Functions Generator Functions When the generator returns, iteration stops • Generator Functions A generator function is mainly a more >>> x.next() • 1 convenient way of writing an iterator >>> x.next() Traceback (most recent call last): You don't have to worry about the iterator File "<stdin>", line 1, in ? • StopIteration protocol (.next, .__iter__, etc.) A>>> generator function is mainly a more • It just works convenient way of writing an iterator • • You don't have to worry about the iterator protocol (.next, .__iter__, etc.) Copyright (C) 2008, http://www.dabeaz.com 1-26 Copyright (C) 2008, http://www.dabeaz.com 1-27 • It just works Copyright (C) 2008, http://www.dabeaz.com 1-27 Generators vs. Iterators • A generator function is slightly different Generators vs. Iterators Generatorthan an object that supports Expressions iteration • A generator is a one-time operation. You •canA generatediterate over version the generated of a list comprehension data once, A generator function is slightly different but>>> if ayou = [1,2,3,4] want to do it again, you have to • >>> b = (2*x for x in a) than an object that supports iteration call>>> the b generator function again. <generator object at 0x58760> A generator is a one-time operation. You This>>> isfor different i in b: printthan b,a list (which you can • • ... can iterate over the generated data once, iterate2 4 6 8over as many times as you want) but if you want to do it again, you have to >>> call the generator function again.