more thoughts on unbearably slow rails3

So as mentioned before, my specific app converted to Rails3 is horribly slow (300% to 400% slower than it was before conversion in Rails2), and the slowness seems to be attributable to ruby Garbage Collection.  And in fact, contrary to my initial report, even using REE and messing with GC tuning hasn’t been enough to reliably make things right again.

Continuing to look around on the web for anyone with anything to say about this, I find this post mentioning similar Rails performance problems — in a comment on that post, Jared Mehle points us to Aaron Paterson’s keynote at the recent RailsConf.   (thanks very much to Luke, Jared, and the rails blogosophere in general for sharing their notes and links making this info more findable than it might be otherwise).

Starting at about 34:10, Aaron talks about Rails3 performance loss compared to Rails2.  In whatever sample application used, he’s talking about ~30% performance loss, not the more like 300% I’m seeing.  But he also attributes it to ruby garbage collection.

And to what does he attribute the extra ruby garbage collection? Increased size of the call stack.  That is, if you were to raise an exception and look at the call stack backtrace from a controller action method, in Rails2.3, according to Aaron,  you’d get a call stack of 51 lines, and in Rails3.0 60 lines, and in Rails 3.1 (which my app isn’t actually on yet), 67 lines.

Aaron says “The way that ruby’s garbage collector works, it has to scan the stack for objects to collect, so obviously as our stack gets larger, the garbage collector has to do more work.”  This isn’t entirely ‘obvious’ to me, I don’t really understand why deeper call stack (as opposed to the more straightforward ‘creating more objects’) leads to more GC (something about closures or similar?), but I trust Aaron knows what he’s talking about, he’s both a ruby and Rails core team member.

Now if I was just getting a 30% or even 40% slowdown, like Aaron’s benchmarked case, that would be one thing. But I’m getting a 300, 400, 500% slowdown, incapacitating my app. So I’d been wondering, okay, what is my app specifically doing to put itself in this pathological case, what makes my app different than anyone elses?  And I hadn’t come up with any hypothesis that ended up verified by experiment. (Ie, cutting out the part of my app I suspected to see if that did it).

But here’s a scary thought: Maybe my app has a deeper call stack than other peoples?  Maybe it always did, but then when you add my own code’s deeper call stack onto Rails3’s deeper call stack, you get a really big call stack which crosses some line to make the ruby GC really pathological?  That I might have a deeper call stack than everyone else seems plausible to me — I am in the habit of creating some fairly abstract multi-level OO designs to make things reusable and seperated, which would lead to increased depth of call stack.

But this is a really disconcerting thought, because I (like most of us) am really not used to thinking in terms of how deep the call stack is, that is not a kind of analysis that is intuitive for me or anything but very challenging to analyze, or take account of in my architecture and programming.  The way we (or at least I) have learned to program, all about encapsulation and abstraction and seperation of concerns and testing in isolation, the size of the call stack is an implementation detail at a lower level of abstraction than I generally ever think about.  And it’s very unclear what I could possibly do to reduce the size of the call stack in my own code, aside from ripping out all my elegant OO abstraction and turning it into a mess of unshareable non-DRY unique-to-the-project spaghetti instead. Nor do I really have any idea how to test this hypothesis that a deep call stack is giving me pathological performance. (Patterson has some ideas for reducing the size of the call stack in Rails3 itself, having identified a particular part of the architecture that has led to the stack size increase, and some ideas for refactoring to reduce it — the middleware stack).

I am awfully frustrated with ruby right now.

I think that ruby 1.9.3, predicted in final release in a couple months, has some new GC strategies, which if I’m lucky might address some of these issues. I haven’t gotten my particular app to run under 1.9.x yet, but perhaps that’s what I’m going to have to do sooner rather than later. Very frustrated with ruby/rails right now, the amount of time I am spending on under-the-hood infrastructure and migration and remediation is leaving not enough time for actually developing the features that make the customers happy.