ruby HTTP client performance shootout redux

About a year ago Vincent Landgraf did a very useful performance comparison of different ruby http client libraries.  Even better, he shared his code on github, so you could see exactly what was being tested, and fork it to do things differently if you want.

I do want! 

My changes and different approaches:

  • I’ll admit I’ve got a horse in this race. I have previously evaluated HTTPClient and liked it for it’s API, featureset, and implementation.  But does it do okay performancewise?  In Landgraf’s original implementation, HTTPClient did very poorly. However, nahi the HTTPClient author noted that the test was creating an HTTPClient object in it’s inner loop, while not doing that for other libraries that also need a client object created.  I wanted to test HTTPClient without an init in inner loop.  I admit I’ve got a horse in the race upfront, but all my code is transparent and on github, feel free to let me know if you think I’ve done something unfair.
  • `patron` is out, I couldn’t get it to install on my machine. Presumably I’m missing some C dev libraries or have the wrong versions. I didn’t feel like fighting it.
  • net-http-persistent is in, I was curious.
  • Landgraf, as far as I can tell,  intentionally wanted to purely test actual http network mechanics, isolating out everything else. On the other hand I want to test something a little bit closer to actual likely use cases.
    • With and without a server providing HTTP 1.1 persistent connections.
    • Under a couple different scenarios of multi-threaded concurrency, because I do that. Added some features to the test suite for this.
    • With and without https/SSL, because I do that.
    • I do have my tests requesting from an HTTP server on a different machine; it is a machine on my local network — I’m not actually sure if they’re on the same subnet; it’s quite possible (without me knowing or investigating) they’re both VM’s on the same physical host even, not sure. (When I tried a ‘real’ real world example, running the test from consumer broadband at home, to a server at my place of work, the I/O was so slow I couldn’t run many iterations, but I might try that again later.)
  • I don’t really care about comparing to Java or apache bench. And I’m adding some more dimensions (concurrency, persistent connections), so there’s enough to consider without including those. I don’t run Landgraf’s Makefile, I just run the test.rb ruby tester — and just  under ruby 1.9.3, I don’t care enough about ruby 1.8.7 anymore to confuse things with more numbers.

The most basic test

I used the built-in “create_test_data” script to create the test.json file. It’s being served by apache — I’ve intentionally set `KeepAlive off` in Apache, so it won’t maintain persistent HTTP 1.1 connections.

This will keep any library that supports these from having an advantage, and might even give em a disadvantage, as they’ve got a bit of overhead for checking for persistent connections.

I did fix my apache to properly send “application/json” for a test.json file back, to keep from disturbing some of the gems. Sorry, I’m not making pretty graph’s like Vincent did, you just get the raw output.

$ ruby test.rb 10000 http://u*.l*.j*.edu/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 10000 requests (10000 iterations with concurrency of 0, ) in each test...
                                   user     system      total        real
testing httparty              48.450000   2.290000  50.740000 ( 92.785885)
testing righttp                4.560000   1.560000   6.120000 ( 29.852759)
testing net/http               9.000000   1.910000  10.910000 ( 33.486526)
testing curl                   4.440000  21.490000  95.360000 (202.245988)
testing rufus-jig              9.490000   2.180000  11.670000 ( 33.423559)
testing httprb                 9.310000   2.010000  11.320000 ( 34.270971)
testing RestClient            11.000000   2.060000  13.060000 ( 35.791577)
testing curb                   4.150000   1.310000   5.460000 ( 12.322419)
testing net-http-persistent    8.850000   1.880000  10.730000 ( 33.104290)
testing httpclient             7.590000   3.460000  11.050000 ( 32.785188)

(Don’t ask me why I’m still using 1.9.3-p0 on this machine. Hopefully it won’t make a difference.)

First, we see there’s a big difference between CPU time (user+system=total), and real/wall time. Presumably because http requests involve a lot of waiting on I/O, that’s where the ‘real’ time comes from?  Now, I can’t explain why the difference between CPU time and wall time varies so much between alternatives; some have more efficient network code than others?  Not sure. (I did run these tests a few times at different times of day, and the rankings, although not the absolute times, seemed to be fairly consistent, so I don’t think it’s one of them just getting unlucky on the network).  I’ll mostly pay attention to ‘real’/wall time in my analysis, because, well, that’s what matters to my app, right?  But I’ll keep giving you the complete results, so you can look at what you like.

We see that curb is way ahead of the pack.  httparty and curl are way behind the pack.  curl’s just a shell out to command line curl, not surprising it’s slow with the overhead of that. I did check that httpparty isn’t subject to the “creation in inner loop” problem, it ain’t, at least as far as it’s external api.

Most of the rest are in a middle-of-the-pack group, with righttp taking a clear lead in there (rightttp is pure ruby even I think, nice job). httpclient takes second in wall time, just barely ahead of net-http-persistent. I don’t know if the difference is meaningful, but it was consistent running this test multiple times as I got it right.

Add in HTTP 1.1 Persistent Connections

Okay, set “KeepAlive on” again on the apache, apache is now allowing persistent HTTP connections. We’ll expect the gems that re-use persistent connections to start doing better — I know HTTPClient and net-http-persistent do, not sure about the others.

$ ruby test.rb 10000 http://u*.l*.j*.e*/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 10000 requests (10000 iterations with concurrency of 0, ) in each test...
                                   user     system      total        real
testing httparty              45.780000   2.090000  47.870000 ( 92.226495)
testing righttp                4.330000   1.540000   5.870000 ( 29.753645)
testing net/http               8.590000   1.900000  10.490000 ( 32.755722)
testing curl                   4.260000  20.900000  93.110000 (200.474332)
testing rufus-jig              8.820000   2.070000  10.890000 ( 32.604847)
testing httprb                 8.800000   1.890000  10.690000 ( 39.891161)
testing RestClient            10.570000   1.940000  12.510000 ( 35.257572)
testing curb                   3.600000   0.600000   4.200000 (  8.399786)
testing net-http-persistent   11.370000   0.850000  12.220000 ( 16.436428)
testing httpclient             7.650000   2.090000   9.740000 ( 13.416148)

Indeed the ones that can maintain persistent connections see a huge benefit. curb is still the leader (and looks like it may re-use persistent connections too?), but net-http-persistent and httpclient have really pulled ahead of the pack. rightttp, the former leader, looks like it doesn’t do persistent connections, and has been eclipsed. httpclient is pretty significantly ahead of net-http-persistent.

Add in Threads

I use multi-threading in some of my apps.

You may sometimes too even if you don’t realize it — for instance, in a Rails app, depending on your web server architecture, each request may end up in it’s own thread, or not. (My understanding is mongrel, seperate threads per request; thin, everything same thread; passenger, not sure, and it may change in future versions of passenger. setting `config.threadsafe!` in your Rails app may or may not change this depending on your web server infrastructure).

But I care about threads. Let’s do the simplest possible test involving threads, we’re going to run each request in a seperate thread, but only one thread at a time. We’ve still got our server supporting persistent HTTP connections, and will for subsequent tests unless otherwise noted. I’m going to leave out httparty and shell-out curl, becuase, let’s face it, they’re out of the running, and I’m sick of waiting on them when I run the benchmark.

$ SKIP=test_curl.rb,test_httparty.rb CONCURRENCY=1 ruby test.rb 10000 http://u*.l*.j*.e*/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 10000 requests (10000 iterations with concurrency of 1, 1 requests per-thread) in each test...
                                   user     system      total        real
testing righttp                4.550000   1.640000   6.190000 ( 31.268725)
testing net/http               8.980000   2.240000  11.220000 ( 34.092804)
testing rufus-jig              8.960000   2.610000  11.570000 ( 34.164368)
testing httprb                 8.340000   2.060000  10.400000 ( 38.728446)
testing RestClient            10.970000   2.630000  13.600000 ( 37.003828)
testing curb                   3.540000   0.900000   4.440000 (  9.214666)
testing net-http-persistent    9.250000   2.550000  11.800000 ( 42.206206)
testing httpclient             7.750000   2.120000   9.870000 ( 13.890384)

First, because we’re only running one thread a time, even those gems that may not actually be multi-threaded safe managed not to reveal that.

Everything is slightly slower, probably because of thread overhead, but mostly retains the same basic rankings.

Except, net-http-persistent gets treated really unfairly, because it’s persistent connection re-use is only within the same thread — by putting everything in a seperate thread, it’s got all the overhead of threading, all the overhead of checking for persistent connections, but never actually gets to re-use any persistent connections. Clearly, don’t use net-http-persistent like this. We’ll give net-http-persistent a better chance in a later test.

Add in threads with real concurrency

Okay, let’s do some real concurrency, requests in threads but 20 threads at time. This matches a real use case I have, although others may or may not, threading was kind of unpopular in ruby for a while, although I think it’s starting to catch on again. 

Using a feature I added to test.rb to run 20 concurrent threads at a time. I’m choosing to still run 10000 total requests, but that’s only 500 actual iterations, cause it’s doing 20 requests in parallel in each iteration, we expect results to be a lot faster.

  • Sadly, we had to exclude previous front-runner curb from this test entirely. Not only did it raise under multi-threaded concurrency, but it segfaulted my interpreter! Maybe I installed or compiled it or it’s C libraries wrong, I dunno.
  • rufus-jig doesn’t raise an exception under multi-threaded concurrency, and performs okay… but if it’s included in the test, every subsequent test raises a “Cannot assign requested address – connect(2).” I’m guessing under multi-threaded concurrency, rufus-jig is leaving a whole mess of sockets open and our process is running out of file descriptors or what have you. So gotta leave rufus-jig out too.
$ SKIP=test_curl.rb,test_httparty.rb,test_curb.rb,test_rufus_jig.rb CONCURRENCY=20 ruby test.rb 10000 http://u*.l*.j*.e*/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 10000 requests (500 iterations with concurrency of 20, 1 requests per-thread) in each test...
                                   user     system      total        real
testing righttp                7.400000   4.820000  12.220000 ( 12.550614)
testing net/http              11.290000   4.690000  15.980000 ( 17.994148)
testing httprb                --> failed attempt to read body out of block
testing RestClient            12.410000   4.190000  16.600000 ( 19.890307)
testing net-http-persistent   11.910000   4.310000  16.220000 ( 24.661706)
testing httpclient             6.860000   2.830000   9.690000 ( 12.428816)

httprb can’t handle multi-threading, the threads are stepping on each other and getting confused about what request goes with call.

Somehow righttp has caught up again? Got me. rightttp doesn’t mention anything about multi-threading in it’s README. I don’t see how it can have performed so much worse in one-thread-at-a-time, but caught up under true concurrency, I kind of suspect it’s not being thread-safe here, even though it’s not raising any exceptions, I am suspicious, something weird is going on.

net-http-persistent is still penalized for not sharing persistent connections between threads. We’ll ameliorate that a bit in the next test.

A slightly more ‘realistic’ threaded concurrency scenario

Okay, but even if you’re using threads (or your framework is under the covers), how often are you going to run just one http request in each thread?  Well, maybe sometimes. But often you’re going to run a few instead.  Let’s use another feature I added to test.rb, still doing 20 concurrent threads, but doing 20 http requests in each thread (instead of each request in it’s own thread).  This will give net-http-persistent a better go of it again.

$ SKIP=test_curl.rb,test_httparty.rb,test_curb.rb,test_rufus_jig.rb CONCURRENCY=20 PER_THREAD=20 ruby test.rb 10000 http://u*.l*.j*.e*/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 10000 requests (25 iterations with concurrency of 20, 20 requests per-thread) in each test...
                                   user     system      total        real
testing righttp                7.170000   4.350000  11.520000 ( 11.242854)
testing net/http              11.110000   4.130000  15.240000 ( 17.174888)
testing httprb                --> failed undefined method `first' for nil:NilClass
testing RestClient            12.870000   3.520000  16.390000 ( 19.699471)
testing net-http-persistent    9.210000   1.120000  10.330000 ( 14.290732)
testing httpclient             6.380000   2.950000   9.330000 ( 12.479392)

Yep, net-http-persistent caught up a bit, but still not to httpclient.

I am still suspicious of rightttp’s results, and think it’s doing something unthreadsafe that would cause problems in reality, even though no exceptions are raised.

Okay, let’s add in SSL

An SSL connection is more expensive to create than an ordinary http connection. This could let those gems that re-use http persistent connections do a heck of a  lot better than the competition. We’ll take concurrency out again, but make sure our server is still supporting persistent http connections.

Our gems that couldn’t hack concurrency are back!  And oh yeah, does SSL slow things down a LOT. So much I didn’t want to wait for 10000 iterations, just doing 1000.

$ ruby test.rb 1000 https://u*.l*.j*.e*/test.json
Execute http performance test using ruby ruby 1.9.3p0 (2011-10-30 revision 33570) [x86_64-linux]
  doing 1000 requests (1000 iterations with concurrency of 0, ) in each test...
                                   user     system      total        real
testing httparty               6.490000   0.130000   6.620000 ( 63.834384)
testing righttp               --> failed undefined method `[]' for nil:NilClass
testing net/http              --> failed wrong status line: ""
testing curl                   0.300000   1.370000  21.800000 ( 57.444003)
testing rufus-jig             --> failed wrong status line: ""
testing httprb                 2.940000   0.090000   3.030000 ( 71.498656)
testing RestClient             3.080000   0.270000   3.350000 ( 60.274534)
testing curb                   0.460000   0.050000   0.510000 (  2.216336)
testing net-http-persistent    0.840000   0.070000   0.910000 (  2.257812)
testing httpclient             0.750000   0.170000   0.920000 (  2.160168)

Indeed, as expected, the gems that can handle re-using persistent http connections win huge here, avoiding the over-head of creating 1000 SSL connections. And apparently that includes curb?

But now we get a new set of gems that can’t handle SSL https connections: rightttp, net/http, rufus-jig.   Some of those definitely could if you use a different special API, I don’t know, I didn’t investigate, I don’t want a special different API, I want https to Just Work.

Some of the gems that do handle SSL with no changes to API… seem to perform even worse than shell out to command-line curl. Not sure how they manage that, or if it’s a testing artifact of some kind.

Just for completeness, I was gonna do SSL under multi-threaded concurrency, but, you know, I’m sick of waiting for these tests, you’re sick of reading this, we all know how it would turn out.

Caveats: Benchmarks are not reality

There are a few things that can lead one to mistakenly conclude more from Benchmarks than one ought to.

Artificial benchmarks are not real world scenarios

In the real world, you’re software is never going to be simply making as many requests as it possibly can in a given amount of time and doing nothing else. It’s never going to make 1000 HTTP requests in 10 seconds, very few applications will have that profile.

This particularly might effect the tests involving HTTP 1.1 persistent connections. Our benchmarks got to make a lot of requests (probably all of them) in a single persistent connection, if the gem supported it. In the real world, servers would be closing these connections, they’d have to be reopened, etc.

This artificial benchmark scenario may also have different effects on ruby GC (which can mess with your benchmarks even in the best of cases), different effects on network activity and remote server load (oh yeah, this benchmark wasn’t real world network activity if you ever talk to servers accross the internet instead of a local subnet). Etc.

‘Wall time’ is unreliable

I don’t want to just look at CPU time, because in the case of HTTP requests, I/O time actually does matter. Some libraries use different I/O routines than others, and this can matter. For instance, the size of the read buffer used has been shown to matter in real world use cases. I don’t want to ‘factor that out’.  Plus, in multi-threaded uses, different blocking/concurrency design decisions could seriously effect ‘wall time’ without effecting CPU time (if threads spend more time waiting on locks); don’t want to ‘factor that out’ either.

But the “real”/clock/wall time can be affected by things out of our control that may vary between tests. Load on the machine running the tests, load on the machine running the apache, network congestion, disk activity on the apache machine serving files from disk.  Both the machine running the tests and the machine running the apache were doing pretty much nothing else during these tests—except for the important disclosure that both ‘machines’ are xen VMs. They’ve got dedicated CPU’s in the xen environment, but I can’t say the physical host was doing nothing else.

I can say that in the course of getting these tests working, I ran em a buncha times over the course of a couple days, and the relative wall time rankings between contenders seemed consistent (although the absolute times changed a bit every test).  But no, I didn’t actually capture all those outputs and analyze em statistically.  Finishing what i’ve got here took quite a bit longer than i expected already.

Running benchmarks with actual validity is hard and time consuming. For all the reasons in this ‘Caveats’ section. For myself, I feel confident that this has given me sufficiently more information than I had before to make sufficiently better decisions than I did before. But you can decide differently; you can go look at CPU times above and ignore the wall times; you can fork and run the tests differently, spending more time on statistical analysis, you can do what you like!
(thanks to Bill Dueber for pointing out the need for this caveat, to be clear.)

Averages aren’t in fact enough

Most people benchmarking just look at an average, which is what we’re doing by just looking at total times, in fact we’re specifically looking at the mean average in the end.

But that’s not actually all the info you really want in a benchmark. It doesn’t capture variation.  Two runs may have had the same average, but in one of them the individual iterations all took about the same amount of time, but in others a minority of the  longest iterations took a heck of a lot of time — but it averaged out to the same.

Ideally you want at least median (50th percentile) and not mean, but also standard deviation as a measure of variation, or at least looking at the 10th and 90th and 99th percentile maybe. But I didn’t do that either, too much trouble for me too. (It would be sweet if someone added utilities to ruby stdlib Benchmark to help you do this).

This level of performance may simply not matter to you

We’re testing http request performance by doing 10000 of them in a row as fast we can. Your real app probably doesn’t do this. The time it takes to perform the http request, even in the worst case, may be such a small percentage of your actual app’s run time or response time, it simply may not matter. Even the slowest clients in this test may not cause any problem at all for your app, who knows.

Don’t pick something just because it’s the fastest, while sacrificing other things (say, ease of use of API), unless you actually have reason to believe your app has a meaningful bottleneck in this area of functionality.

But it’s still good to know performance, with a grain of salt from what you can know from benchmarks, and all things being equal, I’d certainly avoid the very slow ones, and prefer one that performs well. If nothing else, good performance suggests the developers know what they’re doing, which will likely show in other places too.

Conclusion

Let’s face it, in reality, http requests is probably a small part of your app’s end-to-end time or even CPU.  You can probably get away with using even the pathologically slow ones in reality. (like httparty, or httpclient creating a new client for every request).  But if you do care about performance:

  • curb is super fast, if you don’t mind a C extension, and can get it compiled. Unless you need multi-threaded use, which curb failed hard at for me. Maybe you can get curb compiled differently to handle it.
  • Otherwise, if the servers you talk to don’t support persistent HTTP 1.1 connections, msot of what’s tested is pretty decent (stay away from HTTParty and shell out command line curl!)
  • If they do, and you want to take advantage of this, both  net-http-persistent, and httpclient do pretty fine — so long as all your requests are in the same thread.  Personally, I like httpclient’s API and complete featureset better.
  • Some of this speed-up from persistent connections may not carry over to real world use cases though, where you aren’t doing hundreds of connections a second. But if you’re doing SSL/https, it probably really does pay to use a library that can reuse persistent http connections, the savings of not having to renegotiate the SSL each time are pretty huge.
  • If you want to take advantage of persistent http connections to speed things up, and you want persistent http connections to be shared between threads for maximum efficiency, httpclient definitely wins.
    • If you are writing a web app, and want persistent http connections to be shared between different request actions, and don’t want to have to think about whether different requests will end up different threads in your particular infrastructure — you can use httpclient and it Just Works.
  • righttp is a mysterious one, it doens’t seem to do as well as the persistent-connection-handling gems with persistent connections, but mysteriously catches up under multi-threaded concurrency. I suspect something is going wrong under the covers under multi-threading, if you plan to use it in any environment where multi-threading might be possible, I’d be very cautious and investigate more.

I set out really liking httpclient’s API and featureset and wanting to make sure it performs okay.  I conclude that it does. It performs pretty close to as well as anything else (but curb), if not better, even in simple non-persistent-connection single-threaded use. And it performs best of all under multi-threaded use.  It’s good enough all around in whatever cases I threw at it, that I’m comfortable using it as my work horse of an http client, I don’t need to think “But does it still work/perform for this usage pattern or architecture”, it Just Works.

The important caveat is that httpclient performs well if you re-use your HTTPClient object, not create a new one per-request. (That probably applies to net-http-persistent too, which has a similar api of first-create-a-client-object-then-use it, but I didn’t test it.)

If you care about efficiency from persistent http connections, you’d want to make sure to re-use the client object anyway — in order to make sure you’re reusing persistent connections!  And fortunately, HTTPClient is perfectly thread-safe, so you can even create a shared client in global class-variable state, and re-use it wherever you want. You might need to be careful with thread-safety during initialization of the HTTPClient itslef, and issues with forking web servers like passenger. I would like to write a little mixin to give you a thread-safe-init and passenger-fork-safe class-level HTTPClient for these purposes (although if somebody beat me to it I wouldn’t mind!).

About these ads
This entry was posted in General. Bookmark the permalink.

14 Responses to ruby HTTP client performance shootout redux

  1. Pingback: Gem review: HTTPClient is a nice http client for ruby | Bibliographic Wilderness

  2. Alexander says:

    You left out Excon (https://github.com/geemus/excon). Excon actually comes with a benchmarking tool (which compares Excon to other libs like Curb) that you could have used.

    In my testing, Excon’s performance is roughly on par with Curb. Its code is also very clean, as is its API, and it supports streaming nicely. The only issue I have with Excon is the lack of async I/O support, but that applies to pretty much all the current clients.

    Most of the clients you tested actually use the old net/http library internally.

  3. drbrain says:

    I’m the author of net-http-persistent and I think this is a wonderful write-up. It shows the cases I designed my library for (many requests/thread) and didn’t design for (few requests/thread, non-persistent servers). As you’ve shown, my library works best in a producer/consumer chain or under a rails app run atop unicorn or thin which don’t create new threads.

    A note on implementation, my library uses thread-local variables to store connection state so there’s no locking overhead, but as you’ve shown my library suffers as connections can’t be shared across threads. In the future I may add a connection pool, but I didn’t want to bother with evaluating extra dependencies (or implementing a pool) as it wasn’t useful for the task at work I extracted it for. Hopefully ruby will get a resource pool implementation I can reuse in 2.0

    You should report your multithreading problems as a curb bug. When I wrote net-http-persistent I benchmarked it against curb and found a bug which was fixed in curb 0.7.3: http://blog.segment7.net/2010/05/07/net-http-is-not-slow

    I’d like to mention one other test that would favor net-http-persistent. For SSL reconnection, net-http-persistent reuses the SSL session parameters from the previous request to reduce handshake time. A single-threaded SSL test where the server hung up every 10 or so responses should show net-http-persistent doing especially well.

  4. jrochkind says:

    Thanks drbrain!

    Even in the cases your library is designed for and does well in, HTTPClient seems to do just as well. But it would be interesting to test the SSL reconnection case you suggest it might shine in.

    Personally, I don’t want to choose a different client library for different use cases, or have to redo everything with a different library if my use cases end up veering off in originally unexpected directions. I want a client library that just plain works sufficiently well no matter what you throw at it, and has the API at the right level of abstraction to do most anything I’d want to do with it.

    I kind of wish instead of creating yet more http client alternatives, more of the ruby http client experts could pool their efforts and get behind the same codebase.

    But for me, HTTPClient is looking pretty good.

  5. drbrain says:

    I’m working towards taking net-http-persistent and HTTP-only parts of mechanize and combining them into a new HTTP user-agent that can ship with ruby atop net/http. By releasing gems to prove the various concepts first I have a better idea of how to write a library that will work for multiple uses. Hopefully, in the end, we’ll have something that provides all the goodness of HTTPClient and net-http-persistent and mechanize without having to look for extra libraries.

  6. jrochkind says:

    Cool. On the subject of cross-thread connection pooling, looking into how HTTPClient did it (to make sure I could trust it, and just cause i was curious), i was delightfully surprised to see that it doesn’t use a complicated connection pool at all, it’s implementation is “so stupid it just might work”, it just removes in-use connections from it’s list of available connections (in a synchronized block), and puts em back again when they’re done (again in a synchronized block of course). Super simple, but works out just fine.

    I realized you only need more complicated connection pool implementations when you want to enforce a _maximum_ number of connections — and especially when you want clients to block waiting for an avail connection if maximum # are already checked out. That gets tricky in multi-threaded scenario, but if you don’t need to enforce max connections, a dirt simple implementation is possible. It’s unclear to me enforcing max connections is useful in the http case.

    Obviously I’ve become a partisan of httpclient at this point myself, but I think it’s justified! I’d encourage you to check it out for other ideas for a stdlib httpclient. Some of other really handy features it has that I’ve actually wanted before but not had in other clients include basic _and_ digest auth (and ntlm, which i’ve never needed but I bet someone has), ability to easily set my own https client key, transparent uncompress, and correctly setting 1.9 char encoding based on http headers. It really does it all. AND performs great too. If it were me, I’d just plain USE httpclient for a future ruby stdlib http client or at least base it on httpclient, fixing up it’s API a bit, and changing what seems inappropriate for a stdlib component (mainly using it’s own packaged trusted cert store as a default; just about everything else seems great to me).

    My complete review of HTTPClient is in the previous blog post. http://bibwild.wordpress.com/2012/04/30/httpclient-is-a-nice-http-client-forin-ruby/

  7. jrochkind says:

    Please, everyone that wants to see more options compared, feel free to fork either my copy or the original, and DIY!

    That’s what I did here, from what Vincent Landgraf started with, because I wanted to see a couple options he didn’t include (net-http-persistent, and httpclient done right), and under a couple different scenarios he didn’t test (persistent connections and multi-threading).

    I spent a buncha hours getting what I needed out of this and documenting it for y’all, I think I’m probably done with this for a while, but please feel free to post in comments here with links to your own findings if you want to take it further!

    Landgraf’s original: https://github.com/threez/test-http-clients
    My fork with MT options for the ruby test, and some new test candidates: https://github.com/jrochkind/test-http-clients

  8. tonywok says:

    No love for Faraday? https://github.com/technoweenie/faraday

    I’ve been using it a bit, and I’ve found it to be awesome. The middleware feature is killer. I’m curious how it would match up against your other benchmarks.

  9. On SSL and Net::HTTP, you need to say explicit that the site uses SSL, like:
    adapter = Net::HTTP.new
    adapter.use_ssl = true

  10. @tonywok: Like HTTParty, Faraday is just a lib on top of other HTTP clients. I don’t think it should be included in this comparison. However, we’re committed to making Faraday fast. We hope its overhead is negligible compared to the actual HTTP request, but we’ve yet to confirm that experimentally.

    @Jonathan: great article. I will try to adapt lessons learned here in a wiki page on Faraday that will offer advice how to choose an adapter based on specific needs.

  11. jrochkind says:

    Thanks Mislav.

    Personally I am interested in these aspects of HTTPClient, which I think are unique to it:

    * You create an HTTPClient object, and once you do so persistent connections are re-used by that client, but NOT shared with other clients
    * And, they are re-used accross threads, in a thread safe way, for a particular client object.

    Would it be possible to write an HTTPClient adapter for faraday that respects both these architectural points? I am not too familiar with faraday, does it also use a “instantiate a client object first, then ask it to do things” model?

  12. jrochkind says:

    PS: It would be relatively straightforward to take Landgraf’s skeleton, like I did, and use it to compare Faraday with a particular adapter vs that particular adapter ‘bare’, in various scenarios, to verify that it’s overhead is low. Sounds like a good project for someone.

  13. Pingback: The Mega Ruby News and Release Roundup for May 2012

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s