I was interested in
- MRI 1.9.3 (yes, MRI, with the GIL),
- Rails 3.2 with
config.threadsafe!set, so the app can have requests dispatched with multi-threaded concurrency. The equivalent to
config.threadsafe!is always on in Rails 4.
- Testing a Rails app that is heavily I/O bound rather than CPU bound -- most of my web apps are such.
- Testing performance under relatively heavy concurrent load, to get at the limits of how much a dyno can handle under alternate scenarios.
- On a single free heroku dyno (cause I didn't want to pay for time for this experiment)
It appears that Puma provides the best performance characteristics. By far. Especially under puma 2.x "clustered" hybrid multi-process/multi-threaded mode, but even under single-process multi-threaded mode. Yes, even for MRI with the GIL, for an IO-bound app.
- For IO-bound apps, you can likely get a lot more out of your heroku web dynos with multi-threaded request dispatch -- that is, puma and (rails3) config.threadsafe! , using puma's new clustered mode for multiple workers as well. The more I/O-waiting your app does as a proportion of it's response time, the higher the likely advantage; advantage doesn't dissappear unless you have really minimal iowait.
Multi-threaded request dispatch is not just for jruby. I am hoping this discussion increases interest and experimentation with multi-threading under MRI, especially multi-threaded request dispatch of web apps.
Aside from the results themselves -- which I think are interesting, and for which you should go read the complete report, or the Hacker News discussion -- I can say a few things about publication methods here too.
I published in a github repo, that contains all the code tools/instruments I used to do the benchmarking too. The idea is to aid transparency and (attempts at) reproducibility. The interested reader can see exactly what I did, to identify any flaws in my methods. The interested reader can also re-run the benchmarks/experiments using my same tools; or even easily modify my tools/methods (by forking the repo and editing) in order to try to ameliorate perceived shortcomings, or just try alternative experiments to test alternative hypotheses or setups.
At least that's the idea. In this tiny insignificant personal experiment in open research, I discovered it's a lot of work to keep your notes sufficiently detailed, to really put a usable 'research notebook' on the web. Although certainly easier with github. I'm enticed by the possibility of using github to really expose a scientists entire research notebook. For software-instrument-centric experiments (of which more and more science is composed, from astronomy to bioinformatics), one could even imagine tools which captured everything the researcher did automatically, to form the basis of a 'research notebook.'
Additional tools built on top of github could lessen the technical/implementation burden of open research, although not neccesarily lessen the fear of airing your dirty laundry. I know there are various folks thinking and working in those directions, it seems an interesting field of work. Those actual commercial apps focus on the 'collaboration' benefit more than the 'transparency' benefit in their brochureware, which probably makes a lot sense for the actual market of researchers, who still have a lot of ambivelence about transparency.