Thursday 1:20 p.m.–4:40 p.m.

Distributed task processing using Celery

Narahari Allamraju

Audience level:
Novice
Category:
Other

Description

Celery is a distributed task queue for Python that is used by a large number of web applications to off load tasks that can be executed asynchronously. This workshop will demonstrate how we can use Celery for a use case that isn't a web site or a web application and is something which could happen in a hospital or a bank or any other such large organization.

Abstract

Celery is a distributed task queue for Python. It is different from MapReduce because instead of applying a given mapper function on a large set of data and then aggregating (reducing) the results, in Celery you define small self contained tasks and then execute them in large number across a set of worker nodes. Each task can work on just a set of arguments which will maybe later form a part of a larger solution. Celery is written in Python and is very easy to use with minimal setup required for the workers or for the process creating these tasks that are executed on the workers. For more in depth details on Celery you can visit www.celeryproject.com. For this workshop we will be looking at - how to setup Celery, how to use it for simple cases and how we can do some complex workflows with Celery. The assumption is that you know Python, any level of expertise, and can understand a little bit about how and why we would like to distribute task execution - never the less we will start with a brief introduction of why we would like to distribute task execution. We will be looking at Celery, using Redis as broker, and we will be looking at a Celery monitoring tool called Flower. Celery can work with any backend for passing around the tasks and task results - which broker backend we use will ideally depend on the use case and the performance requirements that we have. However for the sake of this tutorial we will use Redis because it is very simple to setup and use.

Student Handout

No handouts have been provided yet for this tutorial