How we scaled Celery for our Django app

The scenario

call some_task.delay(), it returns immediately and executes asynchronously

Need to process all of this (source: https://unsplash.com/photos/jOqJbvo1P9g)

Little about Celery

Are tasks function calls?

Only primitives are allowed as task arguments

Updating a function will break old tasks

Make tasks changes backward compatible, like APIs.

Django-celery-results is a time bomb

Standard logging is a much better solution for any of your debugging needs.

Pro tip: Log task params, in addition to task name

Unit Testing

1. Task invocation

2. Task execution

Ignore celery tasks in the test suite

Scaling up 🚀

Source: https://unsplash.com/photos/TV2gg2kZD1o

Prioritisation

Isolation

Solution: Use different Queues

Some of the queues at apna.co

Bonus: Rate limiting

   x / 0.2 = 500
=> x = 100

Redis Broker does not scale

Celery was duplicating

Delayed Tasks

It starts off simple and has enough features to scale for many different workloads!

I feel we will continue to use it in the years to come.

Engineering Manager at apna.co | Startup enthusiast, Public Speaker, Tech Blogger | 1 startup was acqui-hired.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store