Optional gem dependencies

I’ve sometimes wanted to release a gem with an optional dependency. In one case I can recall, it was an optional dependency on Celluloid (although I’d update that to use concurrent-ruby if I did it over again now) in bento_search.

I didn’t want to include (eg) Celluloid in the gemspec, because not all or even most uses of bento_search use Celluloid. Including it in the gemspec, bundler/rubygems would insist on installing Celluloid for all users of my gem — and in some setups the app would also actually require Celluloid on boot too. Requiring celluloid on boot will also do some somewhat expensive setup code, run some background threads, and possibly give you strange warnings on app exit (all of those things at least in some versions of Celluloid, like the one I was developing against at the time; not sure if it’s still true). I didn’t want any of those things to happen for most people who didn’t need Celluloid with bento_search.

But rubygems/bundler has no way to specify an optional gem dependency. So I resorted to not including the desired optional dependency in my gemspec, but just providing documentation saying “If you want to use feature X, which uses Celluloid, you must add Celluloid to your Gemfile yourself.”

What I didn’t like was there was no way, other than documentation, to include a version specification for what versions of Celluloid my own gem demanded, as an optional dependency. You don’t need to use Celluloid at all, but if you do, then it must be a certain version we know we work with. Not too old (lacking features or having bugs), but also not too new (may have backwards breaking changes; assuming the optional dependency uses semver so that’s predictable on version number).

I thought there was no way to include a version specification for this kind of optional dependency. To be sure, an optional gem dependency is not a good idea. Don’t do it unless you really have to, it complicates things. But I think sometimes it really does make sense to do so, and if you have to, it turns out there is one way to deal with specifying version requirements too.

Because it turns out Rails agrees with me that sometimes an optional dependency really is the lesser evil. The ActiveRecord database adapters are included with Rails, but they often depend on a lower-level database-specific gem, which is not included as an actual gemspec dependency.

They provide a best-of-dealing-with-a-bad-situation pattern to specify version constraints too: use the runtime (not Bundler) `gem` method that rubygems provides, as at:

https://github.com/rails/rails/blob/661731c4c83f7d60f6b97c77f008e2f08441e1a1/activerecord/lib/active_record/connection_adapters/mysql2_adapter.rb#L3

This will get executed only if you are requiring the mysql2_adapter. If you are, it’ll try to load the `mysql2` gem with that version spec, in that version of mysql2_adapter `~> 0.3.13‘` If the “optional dependency” is not loaded at all (because you didn’t include it in your Gemfile), or a version is loaded that doesn’t match those version requirements, it’ll raise a Gem::LoadError, which Rails catches and re-raises with a somewhat better message:

https://github.com/rails/rails/blob/dfb89c9ff2628da9edda7d95fba8657d2fc16d3b/activerecord/lib/active_record/connection_adapters/connection_specification.rb#L175

Of course, this leads to a problem many of us have run into over the past two days since mysql2 was released.  The generated (or recommended) Gemfile for a Rails app using mysql2 includes an unconstrained `gem “mysql2″`. So Bundler is willing to install and use the newly released mysql2 0.4.0.  But then the mysql2_adapter is not willing to use 0.4.0, and ends up raising the somewhat confusing error message:

Specified ‘mysql2’ for database adapter, but the gem is not loaded. Add `gem ‘mysql2’` to your Gemfile (and ensure its version is at the minimum required by ActiveRecord).

In this case, mysql2 0.4.0 would in fact work fine, but mysql2_adapter isn’t willing to use it. (As an aside, why the heck isn’t the mysql2 gem at 1.0 yet and using semver?)  As another aside, if you run into this, until Rails fixes things up, you need to modify your app Gemfile to say `gem ‘mysql2’, “< 0.4.0″`, since Rails 4.2.4 won’t use 0.4.0.

The error message is confusing, because the problem was not a minimum specified by ActiveRecord, but a maximum.  And why not have the error message more clearly tell you exactly what you need?

Leaving aside the complexities of what Rails is trying to do and the right fix on Rails’ end, if I need an optional dependency in the future, I think I’d follow Rails lead, but improve upon the error message:

begin
   gem 'some_gem', "~&gt; 1.4.5"
rescue Gem::LoadError =&gt; e
   raise Gem::LoadError, "You are using functionality requiring 
     the optional gem dependency `#{e.name}`, but the gem is not
     loaded, or is not using an acceptable version. Add 
     `gem '#{e.name}'` to your Gemfile. Version #{MyGem::VERSION}
     of my_gem_name requires #{e.name} that matches #{e.requirement}"
end

Note that the Gem::LoadError includes a requirement attribute that tells you exactly what the version requirements were that failed. Why not include this in the message too, somewhat less confusing?

Except I realize we’re still creating a new Gem::LoadError, without those super useful `name` and `requirement` fields filled out. Our newly raised exception probably ought to copy those over properly too. Left as an exersize to the reader.

I may try to submit a PR to Rails to include a better error message here.

Optional dependencies are still not a good idea. They lead to confusingness like Rails ran into here. But sometimes you really do want to do it anyway, it’s not as bad as the alternatives. Doing what Rails does seems like the least worst pattern available for this kind of optional dependency: Use the runtime `gem` method to specify version constraints for the optional dependency; catch the `Gem::LoadError`;  and provide a better error message for it (either re-raising or writing to log or other place developer will see an error).

This entry was posted in General. Bookmark the permalink.

8 Responses to Optional gem dependencies

  1. It would be nice if this sort of thing were more Built In, though, esp. for allowing (but not requiring) faster — often native-code — versions of built-in libraries. As it is, it seems like one would have to:

    * Have a list of acceptable libraries (which is fine), grouped by “goodness”
    * Check to see if any of the gems in the best group are already loaded by looking for constants
    * if so, check the version to see if it’s ok
    * if not, move on
    * Try to load (using `gem`) stuff in the best group
    * On failure, move to the next group.

    I suppose this whole process itself could be moved into a gem with some configuration, but that’s just re-inventing big swaths of rubygems and/or bundler.

  2. jrochkind says:

    rubygems and bundler are pretty complicated already, with some kinda kludgey code. I doubt we’ll see optional dependencies as a built-in feature anytime soon. I’m not sure the semantics of optional dependencies are entirely clear, in a way that actually works out well, especially at the dependency resolution phase. I think rubygems and bundler are focusing on work on merging, and that’s going fairly slowly.

    I do think an optional dependencies are probably bad idea that should be used only as a last resort — but yeah, there are times you need it anyway. Maybe we can work out the least evil way to do that.

  3. Tom says:

    How about creating a separate gem to encapsulate the functionality that depends on the optional dependency? Then you can have a strict dependency on whatever it is. If you want you could still stub out the method in the original gem, but instead of saying you need to add the mysql2 gem, you say you need to add mygem-mysql gem, and bundler will automatically be able to find the correct version of that gem to use, and the the correct version of the mysql2 gem.

  4. jrochkind says:

    Tom, I’m not seeing how that improves things.

    Either mygem-mysql itself strictly depends on the original mysql gem — in which case you are in the same situation as if the original gem had just strictly depended on it in the first place; pushing the dependency down to be indirect via mygem-mysql doesn’t get around whatever reasons you didn’t want to include it as a strict dependency in the first place, you’ve still got it as a strict dependency.

    Or mygem-mysql does not strictly depend on the original mysql gem — in which case you still have the same optional dependency issue, just pushed down into mygem_mysql, and I still don’t see that you’ve gained.

    Am I missing something?

  5. I disagree with you that optional dependencies are a code smell. For better or for worse, Ruby gems are a pretty heterogenous group, with lots of gems providing similar functionality. Things like MultiJson are one attempt to deal with it, but that goes badly. There should be a canonical way for me to say, “Hey, my code will work fine with nokogiri or ox or oga or rexml; whichever one is loaded will be used. I won’t try to load more than one, and you don’t need to install more than one.” Ditto for json, http clients, etc.

    Some gems will require, say, complex SSL/keepalive HTTP requests, in which case the author will likely carefully pick a gem to use. But say my stuff just needs ‘get_conent(url)’, which any of them can fulfill. Should my gem require Faraday? Or should I write it using Net::HTTP, in which case people who have already loaded a kick-ass http library for other code will have to put up without any decent error handling, two different sets of error messages, etc.?

    What we end up with is either gems that lock you into a particular serializer or database or whatever, or gems that require every possible thing that might work. Dependency injection could get us through one or two layers or indirection, but my top-level code shouldn’t have to configure what unicode normalizer some random json library five gems deep will be using.

    I’m guess I’m arguing both for optional gems, and for common interfaces for some of these things in the way that, for example, YAJL provides “yajl/json_gem” to make it a drop-in replacement for the stock JSON class. That might work for things defined in the standard library. But going beyond that would require a mature interface ecosystem and a level of coordination that I don’t think we have yet.

  6. Tom says:

    What you’ve gained is that the dependencies are listed where a user expects them, rather than hidden in an error message, and bundler can resolve them for you. Of course, you still need some documentation that you’ve got functionality that works with mysql, but now, you just need to tell your users to add mygem-mysql to their gemspec, and bundler can find the correct version of mysql based on the dependencies in that gem.

    Imagine if you used two gems with optional dependencies on the same thing, but with slightly different versions. Now the user has to find that information from both gems, and make sure to install the correct version. If they both had *-mysql gems, the user would just add those, and bundler would find the version that satisfies both (or give you an error at install time that they aren’t going to work together).

    You also gain the ability to upgrade your ‘core’ gem separately from the code that depends on mysql (assuming the the core gem stays backwards compatible enough). You could have versions of mygem-mysql that work with 0.4.0 and 1.0.0, and the core gem can receive updates without having to maintain separate release branches for all the permutations of optional gems you support.

  7. jrochkind says:

    Tom, I guess I’m not understanding how that would work, I don’t understand how the architecture you outline produces the results you claim, where bundler can find the version of mysql that satisfies both or give you an error, but mysql is still an optional dependency. Or how telling the user to add mygem_mysql to their Gemfile is any better than just telling them to add mysql to their Gemfile in the first place.

    I must be missing something. If it would work that way, it does sound like it could be a good way to go. If you ever mock it up in code to demonstrate, please feel free to share here!

  8. kbrock says:

    Thanks for giving examples on how to work around this. It seems that this idea hasn’t gotten much traction in the past.

    You know, the gemspec does have something like optional dependencies:

    – `add_development_dependency` allows a gem author to help users run their specs.
    – `add_runtime_dependency` allows a gem author to help users run their main code.

    `add_optional_dependency` would allow a gem author to help users run their optional features.
    Rails would use for database drivers, and bundler would use for graphviz.

    Bundler has something like optional dependencies, with the `groups`. These optional dependencies can be excluded using `–without`. Do keep in mind, that when resolving the `Gemfile` it takes these dependencies into account, they just aren’t installed.

    I had thought `add_optional_dependency` would work much the same way. When resolving the `Gemfile`, both runtime and optional dependencies would be taken into account. And this does mean that there may be some conflicts in optional dependencies across gems that may not even be required, but this seems like a reasonable concession.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s