heisel.org

September 11, 2009

A short introduction to Django and Gearman

Note: This is a recap of a light­ning talk I gave at Djan­go­Con 2009. Here are the slides

Prob­lem: You’ve got resource or time inten­sive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetch­ing remote resources, resiz­ing images, expen­sive data­base queries, etc. are good candidates.

At work, when­ever we have a prob­lem with scal­ing, or really any prob­lem at all, Johnny always asks “What would Brad Fitz­patrick do?”

What he did was build Gear­man. It is a work queu­ing and dis­tri­b­u­tion system. From your client code you send a request off for work to be done, either syn­chro­nously or asyn­chro­nously. Worker instances pull jobs off the stack and per­form them.

The Python bind­ings are stu­pidly simple to use.

# client.py
from gearman import GearmanClient
client = GearmanClient(["127.0.0.1"])
client.dispatch_background_task("echo", "foo")
 
# worker.py
worker = GearmanWorker(["127.0.0.1"])
worker.register_function("echo", lambda job:job.arg)
worker.work()

It’s con­cep­tu­ally sim­i­lar to Rab­bitMQ. Both are mes­sage queues but Gear­man feels a little more spe­cial­ized for use as a task queue, while Rab­bitMQ is more a gen­eral mes­sag­ing framework.

Rab­bitMQ is nice as well but I’ve got one par­tic­u­lar pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much “Use the power of Erlang”.

When you ask “How do I scale Gearman?” the answer is “just like mem­cached, add more servers to the clus­ter and spec­ify them in your connection.”

from gearman import GearmanClient
 
client = GearmanClient(["workserver.yourdomain.com",
"procrastinate.yourdomain.com"])

If you’re look­ing for a task queue, I highly rec­om­mend Gear­man — it’s easy to use and easy to scale!

Filed under: Django, Programming, Python

Next: Memcached and WordPress MU
Previous: Directory support in pywatch 0.4

Related

• Setup Django with mod_wsgi on your Mac
• Django testing tip: don’t test template output
• The job hunt(ed)