Tuning gunicorn and Django performance on Heroku with blitz.io

Before starting to promote your startup to the public you really need to make sure that your landing page can handle any kind of traffic that will come your way. Blitz.io is my tool of choice. Heroku integrates with blitz and sets you up for a free account right from the start. Very nice.

Here's how Flawless.QA's landing page performance looked like, serving a page directly from memcached:

Note that we're not tuning Django here, or database performance. Our Postgres database is running on a different EC2 instance behind pgbouncer and should not have any effect on the results. This guide will show you how to set up your own Postgres instance. I suspect that using Amazon's RDS will be even easier than setting up Postgres ourselves, but that's something to look into at a later stage.

Based on Heroku's recommendations, the setup above is using gevent so we can have asynchronous workers. This seems to be a point of controversy, as Heroku's docs mention gevent but don't actually show you how to install it and set it up in your Procfile. Here's how our initial setup looks like:

web: python manage.py run_gunicorn -b 0.0.0.0:\$PORT -w 4 -k geventĀ --max-requests=500 --preload

The easiest thing to tune seems to be the number of workers, so let's try 9 workers instead of 4:

Well, that went to shit real fast. Performance is worse than with 4 workers, and if there's many concurrent requests they all fail, not just a small portion of the requests, which is what happened when we had 4 workers.

Now let's see what happens if we turn gevent off:

Perfect! Response times fluctuate a bit around the 200 requests per second limit, but every single requests is served quickly and correctly.

So why does gevent not give better performance on Heroku? In my test case there was nothing infrastructure-related that should have posed a problem, as the response was fetched from memcached directly, and the database was on a different instance. I'm guessing it might be because the overhead of gevent is a bit too much for a Heroku dyno to handle, especially when you're using a large number of worker processes (in gunicorn). I also wonder if this result might be more in favor of gevent if we count web page requests that include loading the static files references on the page. In our case this is a non-issue though, as our static files are hosted directly on Amazon S3, so I can't see any reason to turn gevent on when you're running a similar setup on Heroku.

But don't believe anything you read on the internet. Benchmark it yourself for your particular use case.

Posted in Tech | Tagged , , , ,