Friday 4:15 p.m.–5 p.m.

Distributed Computing Is Hard, Lets Go Shopping

Lewis Franklin

Audience level:
Intermediate
Category:
Best Practices & Patterns

Description

Distributed computing is challenging. The network can be spotty. The servers can be pushed too hard. Race conditions can show up. Tasks can seem to disappear. Deployment can be messy. Celery is an excellent tool for writing distributed applications and this talk focuses on real-world challenges faced when doing those tasks.

Abstract

I work for a small company that specializes in analytics. Our business is data and the availability of that data is crucial. We were running into problems where all of our servers were not being fully utilized, while others were overloaded. I realized that distributed computing via Celery could be an answer to a number of problems. Over the past year we have integrated Celery into a number of our systems and use it for nearly everything except for presenting the analytics to the customer. The intention of this talk is to provide a how-to guide for using Celery across a number of systems, from long-running tasks to IO-intensive applications. The process of getting our systems stable and running consistently has been a learning process and so it is my hope that this talk will help other developers avoid the mistakes I made in the process.