![[lang]](/images/eye.png)
quick hacks are hard |
2009-12-01
|
The good: in around 5 hours of coding (on two train and one plane trips) I knocked together a proof-of-concept traffic redirector for our platform, which understands incoming requests, maps them into client groups, then makes routing decisions by group to any of our sites based on a policy (least cost, geographically closest, ...) In addition, I wrote a simulator that replays actual request log lines, and is able to then track the bandwidth used by each client group and each site, and even able to dump it to an .rrd file for later graphing. Not bad at all for a quick hack. Python is awesome when it needs to be.
The bad: some binary tree algorithm I use to be able to get a sorted list of simulated events seems to crap out currently at about 1000 request lines with a 'Maximum recursion depth exceeded'. Last time I asked our database guy, I think we had about 10 million lines a day.
I don't think my simulator is ready for prime-time just yet...
While algorithm XYZ may not need to be recursive, the “about 1000” is probably “exactly 1000”, because of this:
http://www.google.com/search?q=sys.setrecursionlimit
Comment by Michael DeHaan — 2009-12-01 @ 01:01
Sounds like you need to change the logic to use iteration (probably using a stack) instead of recursion. Hey, it’s Python so it will be easy :)
Comment by matt harrison — 2009-12-01 @ 01:26
Indeed, Python is a very good language for this kind of thing.
Incidentally, do you have some special behavior on this blog relating to referrer? Any time I open pages on your site from Google Reader, the article and comment text aren’t present. If I then go to the location bar and hit enter, the page loads correctly…
Comment by Simon — 2009-12-01 @ 02:10
If you’re sorting a list of events from your log files, you probably only want the next one (or the earliest time). You don’t need to sort the whole sequence for that – use the heapq module. This module maintaines a partially sorted heap, which can guarantee to give the the min (or max) value in a sequence when needed, without having to do a full resort on every pop/insert.
Comment by simon davy — 2009-12-01 @ 09:22
Why not use heapq?
Comment by Niki — 2009-12-01 @ 10:18
I have the same problem with Gogole Reader as Simon.
Comment by fons — 2009-12-02 @ 16:48