Date: 2011-01-16 14:07:00
Tags: minilink, python
minilink performance improvement
My minilink/lnk.nu link shortener has been getting increasingly popular lately. There was a sharp jump up in activity in mid-October, which has now put the request rate at around 20k requests per day. There have been spikes up to 40k per day, too.

This increase in activity has taken a toll on my server, since minilink had been implemented as a simple Python CGI program. This was the original implementation I started with and I haven't needed to change it. However, CGI is only appropriate for low performance applications, and minilink is probably edging into medium performance at this point. A CGI program has to start up an entire Python runtime environment for every request.

Over the last two days, there have been big enough activity spikes that the load on my server goes through the roof and it can't really handle the requests anymore. This can cause things to crash and run out of memory and other impolite behaviours. This happened once last March, and twice in the last two days.

Today I did something I should have done long ago: I converted minilink from a CGI program to a WSGI program. WSGI improves performance because it keeps the Python runtime loaded in memory, and with Apache's mod_wsgi, the calls into Python code are much faster.

I also implemented a simple test suite so that I could refactor the implementation while ensuring that the behaviour stayed the same. That's something I should have done long ago, too.

You may see an increase in performance when using minilink starting today. If you see anything go wrong though, please let me know.
[info]sdodson : results
2011-02-02T05:11:19Z
So how'd it go?
[info]ghewgill : Re: results
2011-02-09T18:11:16Z
Looks like it went well. The request rate has maintained an average of about 20k-30k requests per day since mid-January, with no noticeable ill effects since I switched over to WSGI.
[info]openmynd
2011-02-22T20:46:34Z
I probably shouldn't attach this to this post, as it's kinda unrelated, but...

Are y'all OK?!?!
[info]ghewgill
2011-02-22T20:52:21Z
Yes Amy and I are both safe at home, thanks for checking!
Greg Hewgill <greg@hewgill.com>