Friday 5:10 p.m.–5:40 p.m.

Fan-in and Fan-out: The crucial components of concurrency

Brett Slatkin

Audience level:
Intermediate
Category:
Best Practices & Patterns

Description

The most important yet underappreciated parts of concurrent APIs are good constructs for fan-out (one thread spawns others) and fan-in (many threads join together). This talk will show examples of fan-in and fan-out. It will cover the successes and shortcomings of APIs that provide concurrency. Finally, we'll discuss how the async model of Tulip/PEP3156 is the ultimate API. The future is bright!

Abstract

Over the past 8 years of programing Python professionally, I've used a bunch of concurrent/parallel APIs. I've written at least three concurrent APIs myself and influenced a few others. By far the most important part of these APIs is fan-out and fan-in. But these crucial components are not very much appreciated. Most people don't even realize they're there.

The goal of this talk is to make you care about fan-out and fan-in, and see how the future of Python is bright in this regard.

I'll explain what fan-out and fan-in are, and why they're useful. I'll go over a bunch of examples of fan-out and fan-in, including data constructs (ORMs like SQLAlchemy, Map Reduce, real-time aggregation), mutiple threads (subprocesses, queues, and multiprocessing), and "futuristic" asynchronous APIs (App Engine NDB, Tulip/PEP3156).

Attendees will learn the value of fan-in and fan-out in API design. They'll have a name to give something they've used many times in their concurrent programs without realizing what the magical mechanisms are called. They'll get excited about the future of Python because Tulip/PEP3156 is such a great API for concurrency.