Change the future

Sunday 1:10 p.m.–1:40 p.m.

Iteration & Generators: the Python Way

Luciano Ramalho

Audience level:
Novice
Category:
Best Practices/Patterns

Description

Did you know that "for a, (b, c) in s:" is a valid Python line? From the elegant for statement through list/set/dict comprehensions and generator functions, this talk shows how the Iterator pattern is so deeply embedded in the syntax of Python, and so widely supported by its libraries, that some of its most powerful applications can be overlooked by programmers coming from other languages.

Abstract

UPDATE: Preview slides for this talk can be seen at Speakerdeck Please check them out. Suggestions for improvement are welcome. Some minor fixes are already planned.

UPDATE #2: This talk was selected for a keynote slot at the RuPy Strongly Dynamic Conference in Brazil, December 2012

UPDATE #3: This talk was voted by the community for a slot at PythonBrasil (our PyCon), November 2012

The core of the talk is refactoring a simple iterable class from the classic Iterator design pattern (as implemented in the GoF book) to compatible but less verbose implementations using generators. This provides a meaningful context to understand the value of generators. Along the way the behavior of the iter function, the Sequence protocol and the Iterable interface are presented.

Iteration provides a theme to explore core ideas in the design of Python: the data model, special methods, classic protocols, the new Collections ABC, and why it makes sense that len, reversed and iter are not methods, but built-in functions. Each of these ideas is briefly discussed in the talk, to provide insight and encourage deeper exploration of the language by the attendees.

Talk Outline (30'-45')

Topics will be presented roughly in the order shown below. Section times are given for 30'-45' versions.

Motivation (2'-3')

  • iterating over a file, a Django QuerySet and a simple user-defined class
  • iterating to build: list, dict and set comprehensions
  • syntatic support: tuple unpacking and exploding function arguments

Implementation (6'-11')

  • what is a Python iterable really: the role of the iter function
  • special methods and the Python object model
  • the legacy sequence protocol
  • the Iterable interface, as defined in the Collections ABC
  • implementing the Iterator design pattern: the way of the GoF

Generators (15'-19')

  • motivating example: enumerate, the "missing piece" of the for loop
  • the generator concept: more general than an iterator
  • building a generator function
  • the behaviour of the simplest generator function: one yield, no loop
  • refactoring the Iterator design pattern with a generator method
  • genexps: generator expressions
  • refactoring the Iterator design pattern by returning a genexp
  • conclusion: generators preserve an iteration context

Library support (5'-8')

  • built-in iterable types
  • built-in functions that consume or produce iterables
  • the itertools functions and recipes

Case study (2'-4')

  • using generator functions to decouple reading and writing logic in a database conversion utility

Sponsors