Coroutines, event loops, and the history of Python generators

Type:
Talk
Audience level:
Experienced
Category:
Concurrency
March 10th 1:20 p.m. – 2:15 p.m.

Description

This talk traces lightweight concurrency from Python 2.2's generators, which enabled semi-coroutines as a mechanism for scheduling "weightless" threads; to PEP 342, which created true coroutines, and hence made event-driven programming easier; to 3rd party libraries built around coroutines, from older GTasklet and peak.events to the current Greenlet/gevent and Twisted Reactor.

Abstract

This talk aims to provide both a practical guide and theoretical underpinnings to the use of generator-based lightweight concurrency in Python.

  1. Lightning tour of generator constructs. Why generator-based scheduling is particularly useful for event-based programming.
  2. Simple example of a "trampoline" or scheduler.
  3. Slightly fleshed out example of scheduler with discussion of data-passing issues.
  4. Examples using GTasklet to make coroutine code look more like familiar sequential code (the framework is based on greenlets rather than generators, but accomplishes similar purpose).
  5. Brief examples of Twisted Reactors and Deferreds.
  6. Limits of generator-based concurrency (i.e. doesn't help with multiple cores and multiple servers). "Throw at the wall" list of ways to generalize to larger scales than single cores.