September 11, 2009
A short introduction to Django and Gearman
Note: This is a recap of a lightning talk I gave at DjangoCon 2009. Here are the slides
Problem: You’ve got resource or time intensive work to be done but you don’t want to do it within the Django request/response cycle. Jobs like fetching remote resources, resizing images, expensive database queries, etc. are good candidates.
At work, whenever we have a problem with scaling, or really any problem at all, Johnny always asks “What would Brad Fitzpatrick do?”
What he did was build Gearman. It is a work queuing and distribution system. From your client code you send a request off for work to be done, either synchronously or asynchronously. Worker instances pull jobs off the stack and perform them.
The Python bindings are stupidly simple to use.
# client.py from gearman import GearmanClient client = GearmanClient(["127.0.0.1"]) client.dispatch_background_task("echo", "foo") # worker.py worker = GearmanWorker(["127.0.0.1"]) worker.register_function("echo", lambda job:job.arg) worker.work()
It’s conceptually similar to RabbitMQ. Both are message queues but Gearman feels a little more specialized for use as a task queue, while RabbitMQ is more a general messaging framework.
RabbitMQ is nice as well but I’ve got one particular pet peeve with it. When you ask “How do I scale RabbitMQ?” the answer is pretty much “Use the power of Erlang”.
When you ask “How do I scale Gearman?” the answer is “just like memcached, add more servers to the cluster and specify them in your connection.”
from gearman import GearmanClient client = GearmanClient(["workserver.yourdomain.com", "procrastinate.yourdomain.com"])
If you’re looking for a task queue, I highly recommend Gearman — it’s easy to use and easy to scale!
Filed under: Django,Programming,Python
Next: Memcached and WordPress MU
Previous: Directory support in pywatch 0.4
Related
- Setup Django with mod_wsgi on your Mac
- Django testing tip: don’t test template output
- The job hunt(ed)
-
http://www.zellyn.com/ Zellyn
-
http://heisel.org Chris
