Saturday 3:15 p.m.–3:45 p.m.

Pushing Python: Building a High Throughput, Low Latency System

Kevin Ballard

Audience level:
Intermediate
Category:
Best Practices & Patterns

Description

Taba is a distributed metrics aggregator, similar in concept to statsd. Built with Python using Redis, gevent, and Cython, it currently handles over 6M events/sec with strong consistency guarantees. This talk will present an overview of its design, and discuss the challenges and solutions encountered in the process of building a high throughput, low latency distributed service.

Abstract

Taba is a distributed aggregation service, currently handling over 6M events per second, with low end-to-end latency. Similar in concept to statsd, Taba can handle orders of magnitude more data, with stronger consistency guarantees. It is built atop Redis using Python with gevent and Cython. We will discuss the data model used, and how its design impacts process flows, the database schema, network usage, cacheability, etc. We will also describe the cluster architecture, which is based on many identical distributed processes, and uses master-less coordination to automatically distribute load across the cluster. Finally, we will discuss some of the interesting challenges and solutions encountered. These include: - Memory fragmentation and ‘ratcheting’ in CPython - Using generators, greenlets, and queues to great effect in reducing memory usage without sacrificing performance. - Using Cython to optimize critical sections - Optimizing CPU intensive operations (e.g. distributed reservoir sampling) - Hybrid manual / automatic memory management techniques