About a year ago Vincent Landgraf did a very useful performance comparison of different ruby http client libraries. Even better, he shared his code on github, so you could see exactly what was being tested, and fork it to do things differently if you want.
My changes and different approaches:
- I’ll admit I’ve got a horse in this race. I have previously evaluated HTTPClient and liked it for it’s API, featureset, and implementation. But does it do okay performancewise? In Landgraf’s original implementation, HTTPClient did very poorly. However, nahi the HTTPClient author noted that the test was creating an HTTPClient object in it’s inner loop, while not doing that for other libraries that also need a client object created. I wanted to test HTTPClient without an init in inner loop. I admit I’ve got a horse in the race upfront, but all my code is transparent and on github, feel free to let me know if you think I’ve done something unfair.
- `patron` is out, I couldn’t get it to install on my machine. Presumably I’m missing some C dev libraries or have the wrong versions. I didn’t feel like fighting it.
- net-http-persistent is in, I was curious.
- Landgraf, as far as I can tell, intentionally wanted to purely test actual http network mechanics, isolating out everything else. On the other hand I want to test something a little bit closer to actual likely use cases.
- With and without a server providing HTTP 1.1 persistent connections.
- Under a couple different scenarios of multi-threaded concurrency, because I do that. Added some features to the test suite for this.
- With and without https/SSL, because I do that.
- I do have my tests requesting from an HTTP server on a different machine; it is a machine on my local network — I’m not actually sure if they’re on the same subnet; it’s quite possible (without me knowing or investigating) they’re both VM’s on the same physical host even, not sure. (When I tried a ‘real’ real world example, running the test from consumer broadband at home, to a server at my place of work, the I/O was so slow I couldn’t run many iterations, but I might try that again later.)
- I don’t really care about comparing to Java or apache bench. And I’m adding some more dimensions (concurrency, persistent connections), so there’s enough to consider without including those. I don’t run Landgraf’s Makefile, I just run the test.rb ruby tester — and just under ruby 1.9.3, I don’t care enough about ruby 1.8.7 anymore to confuse things with more numbers.
The most basic test
I used the built-in “create_test_data” script to create the test.json file. It’s being served by apache — I’ve intentionally set `KeepAlive off` in Apache, so it won’t maintain persistent HTTP 1.1 connections.
This will keep any library that supports these from having an advantage, and might even give em a disadvantage, as they’ve got a bit of overhead for checking for persistent connections.
I did fix my apache to properly send “application/json” for a test.json file back, to keep from disturbing some of the gems. Sorry, I’m not making pretty graph’s like Vincent did, you just get the raw output.
$ ruby test.rb 10000 http://u*.l*.j*.edu/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 10000 requests (10000 iterations with concurrency of 0, ) in each test... user system total real testing httparty 48.450000 2.290000 50.740000 ( 92.785885) testing righttp 4.560000 1.560000 6.120000 ( 29.852759) testing net/http 9.000000 1.910000 10.910000 ( 33.486526) testing curl 4.440000 21.490000 95.360000 (202.245988) testing rufus-jig 9.490000 2.180000 11.670000 ( 33.423559) testing httprb 9.310000 2.010000 11.320000 ( 34.270971) testing RestClient 11.000000 2.060000 13.060000 ( 35.791577) testing curb 4.150000 1.310000 5.460000 ( 12.322419) testing net-http-persistent 8.850000 1.880000 10.730000 ( 33.104290) testing httpclient 7.590000 3.460000 11.050000 ( 32.785188)
(Don’t ask me why I’m still using 1.9.3-p0 on this machine. Hopefully it won’t make a difference.)
First, we see there’s a big difference between CPU time (user+system=total), and real/wall time. Presumably because http requests involve a lot of waiting on I/O, that’s where the ‘real’ time comes from? Now, I can’t explain why the difference between CPU time and wall time varies so much between alternatives; some have more efficient network code than others? Not sure. (I did run these tests a few times at different times of day, and the rankings, although not the absolute times, seemed to be fairly consistent, so I don’t think it’s one of them just getting unlucky on the network). I’ll mostly pay attention to ‘real’/wall time in my analysis, because, well, that’s what matters to my app, right? But I’ll keep giving you the complete results, so you can look at what you like.
We see that curb is way ahead of the pack. httparty and curl are way behind the pack. curl’s just a shell out to command line curl, not surprising it’s slow with the overhead of that. I did check that httpparty isn’t subject to the “creation in inner loop” problem, it ain’t, at least as far as it’s external api.
Most of the rest are in a middle-of-the-pack group, with righttp taking a clear lead in there (rightttp is pure ruby even I think, nice job). httpclient takes second in wall time, just barely ahead of net-http-persistent. I don’t know if the difference is meaningful, but it was consistent running this test multiple times as I got it right.
Add in HTTP 1.1 Persistent Connections
Okay, set “KeepAlive on” again on the apache, apache is now allowing persistent HTTP connections. We’ll expect the gems that re-use persistent connections to start doing better — I know HTTPClient and net-http-persistent do, not sure about the others.
$ ruby test.rb 10000 http://u*.l*.j*.e*/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 10000 requests (10000 iterations with concurrency of 0, ) in each test... user system total real testing httparty 45.780000 2.090000 47.870000 ( 92.226495) testing righttp 4.330000 1.540000 5.870000 ( 29.753645) testing net/http 8.590000 1.900000 10.490000 ( 32.755722) testing curl 4.260000 20.900000 93.110000 (200.474332) testing rufus-jig 8.820000 2.070000 10.890000 ( 32.604847) testing httprb 8.800000 1.890000 10.690000 ( 39.891161) testing RestClient 10.570000 1.940000 12.510000 ( 35.257572) testing curb 3.600000 0.600000 4.200000 ( 8.399786) testing net-http-persistent 11.370000 0.850000 12.220000 ( 16.436428) testing httpclient 7.650000 2.090000 9.740000 ( 13.416148)
Indeed the ones that can maintain persistent connections see a huge benefit. curb is still the leader (and looks like it may re-use persistent connections too?), but net-http-persistent and httpclient have really pulled ahead of the pack. rightttp, the former leader, looks like it doesn’t do persistent connections, and has been eclipsed. httpclient is pretty significantly ahead of net-http-persistent.
Add in Threads
I use multi-threading in some of my apps.
You may sometimes too even if you don’t realize it — for instance, in a Rails app, depending on your web server architecture, each request may end up in it’s own thread, or not. (My understanding is mongrel, seperate threads per request; thin, everything same thread; passenger, not sure, and it may change in future versions of passenger. setting `config.threadsafe!` in your Rails app may or may not change this depending on your web server infrastructure).
But I care about threads. Let’s do the simplest possible test involving threads, we’re going to run each request in a seperate thread, but only one thread at a time. We’ve still got our server supporting persistent HTTP connections, and will for subsequent tests unless otherwise noted. I’m going to leave out httparty and shell-out curl, becuase, let’s face it, they’re out of the running, and I’m sick of waiting on them when I run the benchmark.
$ SKIP=test_curl.rb,test_httparty.rb CONCURRENCY=1 ruby test.rb 10000 http://u*.l*.j*.e*/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 10000 requests (10000 iterations with concurrency of 1, 1 requests per-thread) in each test... user system total real testing righttp 4.550000 1.640000 6.190000 ( 31.268725) testing net/http 8.980000 2.240000 11.220000 ( 34.092804) testing rufus-jig 8.960000 2.610000 11.570000 ( 34.164368) testing httprb 8.340000 2.060000 10.400000 ( 38.728446) testing RestClient 10.970000 2.630000 13.600000 ( 37.003828) testing curb 3.540000 0.900000 4.440000 ( 9.214666) testing net-http-persistent 9.250000 2.550000 11.800000 ( 42.206206) testing httpclient 7.750000 2.120000 9.870000 ( 13.890384)
First, because we’re only running one thread a time, even those gems that may not actually be multi-threaded safe managed not to reveal that.
Everything is slightly slower, probably because of thread overhead, but mostly retains the same basic rankings.
Except, net-http-persistent gets treated really unfairly, because it’s persistent connection re-use is only within the same thread — by putting everything in a seperate thread, it’s got all the overhead of threading, all the overhead of checking for persistent connections, but never actually gets to re-use any persistent connections. Clearly, don’t use net-http-persistent like this. We’ll give net-http-persistent a better chance in a later test.
Add in threads with real concurrency
Okay, let’s do some real concurrency, requests in threads but 20 threads at time. This matches a real use case I have, although others may or may not, threading was kind of unpopular in ruby for a while, although I think it’s starting to catch on again.
Using a feature I added to test.rb to run 20 concurrent threads at a time. I’m choosing to still run 10000 total requests, but that’s only 500 actual iterations, cause it’s doing 20 requests in parallel in each iteration, we expect results to be a lot faster.
- Sadly, we had to exclude previous front-runner curb from this test entirely. Not only did it raise under multi-threaded concurrency, but it segfaulted my interpreter! Maybe I installed or compiled it or it’s C libraries wrong, I dunno.
- rufus-jig doesn’t raise an exception under multi-threaded concurrency, and performs okay… but if it’s included in the test, every subsequent test raises a “Cannot assign requested address – connect(2).” I’m guessing under multi-threaded concurrency, rufus-jig is leaving a whole mess of sockets open and our process is running out of file descriptors or what have you. So gotta leave rufus-jig out too.
$ SKIP=test_curl.rb,test_httparty.rb,test_curb.rb,test_rufus_jig.rb CONCURRENCY=20 ruby test.rb 10000 http://u*.l*.j*.e*/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 10000 requests (500 iterations with concurrency of 20, 1 requests per-thread) in each test... user system total real testing righttp 7.400000 4.820000 12.220000 ( 12.550614) testing net/http 11.290000 4.690000 15.980000 ( 17.994148) testing httprb --> failed attempt to read body out of block testing RestClient 12.410000 4.190000 16.600000 ( 19.890307) testing net-http-persistent 11.910000 4.310000 16.220000 ( 24.661706) testing httpclient 6.860000 2.830000 9.690000 ( 12.428816)
httprb can’t handle multi-threading, the threads are stepping on each other and getting confused about what request goes with call.
Somehow righttp has caught up again? Got me. rightttp doesn’t mention anything about multi-threading in it’s README. I don’t see how it can have performed so much worse in one-thread-at-a-time, but caught up under true concurrency, I kind of suspect it’s not being thread-safe here, even though it’s not raising any exceptions, I am suspicious, something weird is going on.
net-http-persistent is still penalized for not sharing persistent connections between threads. We’ll ameliorate that a bit in the next test.
A slightly more ‘realistic’ threaded concurrency scenario
Okay, but even if you’re using threads (or your framework is under the covers), how often are you going to run just one http request in each thread? Well, maybe sometimes. But often you’re going to run a few instead. Let’s use another feature I added to test.rb, still doing 20 concurrent threads, but doing 20 http requests in each thread (instead of each request in it’s own thread). This will give net-http-persistent a better go of it again.
$ SKIP=test_curl.rb,test_httparty.rb,test_curb.rb,test_rufus_jig.rb CONCURRENCY=20 PER_THREAD=20 ruby test.rb 10000 http://u*.l*.j*.e*/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 10000 requests (25 iterations with concurrency of 20, 20 requests per-thread) in each test... user system total real testing righttp 7.170000 4.350000 11.520000 ( 11.242854) testing net/http 11.110000 4.130000 15.240000 ( 17.174888) testing httprb --> failed undefined method `first' for nil:NilClass testing RestClient 12.870000 3.520000 16.390000 ( 19.699471) testing net-http-persistent 9.210000 1.120000 10.330000 ( 14.290732) testing httpclient 6.380000 2.950000 9.330000 ( 12.479392)
Yep, net-http-persistent caught up a bit, but still not to httpclient.
I am still suspicious of rightttp’s results, and think it’s doing something unthreadsafe that would cause problems in reality, even though no exceptions are raised.
Okay, let’s add in SSL
An SSL connection is more expensive to create than an ordinary http connection. This could let those gems that re-use http persistent connections do a heck of a lot better than the competition. We’ll take concurrency out again, but make sure our server is still supporting persistent http connections.
Our gems that couldn’t hack concurrency are back! And oh yeah, does SSL slow things down a LOT. So much I didn’t want to wait for 10000 iterations, just doing 1000.
$ ruby test.rb 1000 https://u*.l*.j*.e*/test.json Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux] doing 1000 requests (1000 iterations with concurrency of 0, ) in each test... user system total real testing httparty 6.490000 0.130000 6.620000 ( 63.834384) testing righttp --> failed undefined method `' for nil:NilClass testing net/http --> failed wrong status line: "" testing curl 0.300000 1.370000 21.800000 ( 57.444003) testing rufus-jig --> failed wrong status line: "" testing httprb 2.940000 0.090000 3.030000 ( 71.498656) testing RestClient 3.080000 0.270000 3.350000 ( 60.274534) testing curb 0.460000 0.050000 0.510000 ( 2.216336) testing net-http-persistent 0.840000 0.070000 0.910000 ( 2.257812) testing httpclient 0.750000 0.170000 0.920000 ( 2.160168)
Indeed, as expected, the gems that can handle re-using persistent http connections win huge here, avoiding the over-head of creating 1000 SSL connections. And apparently that includes curb?
But now we get a new set of gems that can’t handle SSL https connections: rightttp, net/http, rufus-jig. Some of those definitely could if you use a different special API, I don’t know, I didn’t investigate, I don’t want a special different API, I want https to Just Work.
Some of the gems that do handle SSL with no changes to API… seem to perform even worse than shell out to command-line curl. Not sure how they manage that, or if it’s a testing artifact of some kind.
Just for completeness, I was gonna do SSL under multi-threaded concurrency, but, you know, I’m sick of waiting for these tests, you’re sick of reading this, we all know how it would turn out.
Caveats: Benchmarks are not reality
There are a few things that can lead one to mistakenly conclude more from Benchmarks than one ought to.
Artificial benchmarks are not real world scenarios
In the real world, you’re software is never going to be simply making as many requests as it possibly can in a given amount of time and doing nothing else. It’s never going to make 1000 HTTP requests in 10 seconds, very few applications will have that profile.
This particularly might effect the tests involving HTTP 1.1 persistent connections. Our benchmarks got to make a lot of requests (probably all of them) in a single persistent connection, if the gem supported it. In the real world, servers would be closing these connections, they’d have to be reopened, etc.
This artificial benchmark scenario may also have different effects on ruby GC (which can mess with your benchmarks even in the best of cases), different effects on network activity and remote server load (oh yeah, this benchmark wasn’t real world network activity if you ever talk to servers accross the internet instead of a local subnet). Etc.
‘Wall time’ is unreliable
I don’t want to just look at CPU time, because in the case of HTTP requests, I/O time actually does matter. Some libraries use different I/O routines than others, and this can matter. For instance, the size of the read buffer used has been shown to matter in real world use cases. I don’t want to ‘factor that out’. Plus, in multi-threaded uses, different blocking/concurrency design decisions could seriously effect ‘wall time’ without effecting CPU time (if threads spend more time waiting on locks); don’t want to ‘factor that out’ either.
But the “real”/clock/wall time can be affected by things out of our control that may vary between tests. Load on the machine running the tests, load on the machine running the apache, network congestion, disk activity on the apache machine serving files from disk. Both the machine running the tests and the machine running the apache were doing pretty much nothing else during these tests—except for the important disclosure that both ‘machines’ are xen VMs. They’ve got dedicated CPU’s in the xen environment, but I can’t say the physical host was doing nothing else.
I can say that in the course of getting these tests working, I ran em a buncha times over the course of a couple days, and the relative wall time rankings between contenders seemed consistent (although the absolute times changed a bit every test). But no, I didn’t actually capture all those outputs and analyze em statistically. Finishing what i’ve got here took quite a bit longer than i expected already.
Running benchmarks with actual validity is hard and time consuming. For all the reasons in this ‘Caveats’ section. For myself, I feel confident that this has given me sufficiently more information than I had before to make sufficiently better decisions than I did before. But you can decide differently; you can go look at CPU times above and ignore the wall times; you can fork and run the tests differently, spending more time on statistical analysis, you can do what you like!
(thanks to Bill Dueber for pointing out the need for this caveat, to be clear.)
Averages aren’t in fact enough
Most people benchmarking just look at an average, which is what we’re doing by just looking at total times, in fact we’re specifically looking at the mean average in the end.
But that’s not actually all the info you really want in a benchmark. It doesn’t capture variation. Two runs may have had the same average, but in one of them the individual iterations all took about the same amount of time, but in others a minority of the longest iterations took a heck of a lot of time — but it averaged out to the same.
Ideally you want at least median (50th percentile) and not mean, but also standard deviation as a measure of variation, or at least looking at the 10th and 90th and 99th percentile maybe. But I didn’t do that either, too much trouble for me too. (It would be sweet if someone added utilities to ruby stdlib Benchmark to help you do this).
This level of performance may simply not matter to you
We’re testing http request performance by doing 10000 of them in a row as fast we can. Your real app probably doesn’t do this. The time it takes to perform the http request, even in the worst case, may be such a small percentage of your actual app’s run time or response time, it simply may not matter. Even the slowest clients in this test may not cause any problem at all for your app, who knows.
Don’t pick something just because it’s the fastest, while sacrificing other things (say, ease of use of API), unless you actually have reason to believe your app has a meaningful bottleneck in this area of functionality.
But it’s still good to know performance, with a grain of salt from what you can know from benchmarks, and all things being equal, I’d certainly avoid the very slow ones, and prefer one that performs well. If nothing else, good performance suggests the developers know what they’re doing, which will likely show in other places too.
Let’s face it, in reality, http requests is probably a small part of your app’s end-to-end time or even CPU. You can probably get away with using even the pathologically slow ones in reality. (like httparty, or httpclient creating a new client for every request). But if you do care about performance:
- curb is super fast, if you don’t mind a C extension, and can get it compiled. Unless you need multi-threaded use, which curb failed hard at for me. Maybe you can get curb compiled differently to handle it.
- Otherwise, if the servers you talk to don’t support persistent HTTP 1.1 connections, msot of what’s tested is pretty decent (stay away from HTTParty and shell out command line curl!)
- If they do, and you want to take advantage of this, both net-http-persistent, and httpclient do pretty fine — so long as all your requests are in the same thread. Personally, I like httpclient’s API and complete featureset better.
- Some of this speed-up from persistent connections may not carry over to real world use cases though, where you aren’t doing hundreds of connections a second. But if you’re doing SSL/https, it probably really does pay to use a library that can reuse persistent http connections, the savings of not having to renegotiate the SSL each time are pretty huge.
- If you want to take advantage of persistent http connections to speed things up, and you want persistent http connections to be shared between threads for maximum efficiency, httpclient definitely wins.
- If you are writing a web app, and want persistent http connections to be shared between different request actions, and don’t want to have to think about whether different requests will end up different threads in your particular infrastructure — you can use httpclient and it Just Works.
- righttp is a mysterious one, it doens’t seem to do as well as the persistent-connection-handling gems with persistent connections, but mysteriously catches up under multi-threaded concurrency. I suspect something is going wrong under the covers under multi-threading, if you plan to use it in any environment where multi-threading might be possible, I’d be very cautious and investigate more.
I set out really liking httpclient’s API and featureset and wanting to make sure it performs okay. I conclude that it does. It performs pretty close to as well as anything else (but curb), if not better, even in simple non-persistent-connection single-threaded use. And it performs best of all under multi-threaded use. It’s good enough all around in whatever cases I threw at it, that I’m comfortable using it as my work horse of an http client, I don’t need to think “But does it still work/perform for this usage pattern or architecture”, it Just Works.
The important caveat is that httpclient performs well if you re-use your HTTPClient object, not create a new one per-request. (That probably applies to net-http-persistent too, which has a similar api of first-create-a-client-object-then-use it, but I didn’t test it.)
If you care about efficiency from persistent http connections, you’d want to make sure to re-use the client object anyway — in order to make sure you’re reusing persistent connections! And fortunately, HTTPClient is perfectly thread-safe, so you can even create a shared client in global class-variable state, and re-use it wherever you want. You might need to be careful with thread-safety during initialization of the HTTPClient itslef, and issues with forking web servers like passenger. I would like to write a little mixin to give you a thread-safe-init and passenger-fork-safe class-level HTTPClient for these purposes (although if somebody beat me to it I wouldn’t mind!).