loosely coupled components, developer joy, and dependency hell

The ruby open source community’s behavior makes two choices with interesting consequences:

  1. The ruby open source community tries very hard to never re-write what someone else has written, instead composing gems and applications out of smaller pieces of de-coupled re-usable code. People try to share their code in composable decoupled pieces, and people try to write code using such pieces.
  2. The ruby open source community strongly prefers innovation to stability, spending a lot more energy on improving things to be cooler, easier, faster, better — than on providing backwards compatibility or supporting old code.

I think both of these are actually consequences of a couple other very related things:

  1. The ruby community is very volunteer-collaboration driven. Most code is not written by people being paid specifically to write that code.   Although lots of code is written by people who are using it for work they get paid for, relatively little code is written by people who have writing that code, as open source,  as an assigned task. (Compare to Java open source community).  Most ruby open source code is a labor of love (for the code, for the other developers you are sharing with).
  2. The ruby community cares a lot about ‘developer joy’. Going back to matz’s original intention for ruby: “Ruby is designed to make programming not only easy but also fun”.  But this is also related to the point above — a mostly volunteer developer community only happens when developing is fun, and the more fun something is, the more likely there will be sufficient volunteer developer capacity to accomplish it.

Likewise, a mostly volunteer developer community kind of needs to develop based on loosely coupled small units developed by other people, to get anything done. We need to absolutely maximize our efficiency, because we don’t have armies of programmers getting paid to write what we need, we’ve got to take maximum advantage of what we’ve got.

But there’s a downside. Dependency hell.  When your product is built from dependencies on a bunch of things written by other people (and each of those has it’s own dependencies) — and especially when everything is constantly changing as people try to make things better, caring less about backwards compat and stability — it means things can be  constantly breaking.

After having spent a few years writing production ruby/rails code now, and still working at the same place where I wrote that code — I lately spend a huge amount of time upgrading my stuff to work with new versions of dependencies, or abandoning one now not-sufficiently-maintained dependency for another, etc.

Each individual upgrade may not be that hard, but they add up. It’s a huge portion of my time these days. And this is not a very joyful state, running on the treadmill just to stay in place instead of creating awesome new things. Ironically, optimizing for developer joy in the upstream dependencies (those developers don’t find it joyful to work on backwards-compat or refrain from releasing new ways of doing things because it’ll require people to change) — has made the joy of the downstream developer using all those dependencies very non-optimal.

Another interesting example is the way the recent Rails release had to downgrade sprockets to fix a regression. Rails too is built from a bunch of building blocks many developed by others (it always was to some extent, but Rails 3.0 pushed it even more in this direction). Which means those dependencies can easily break Rails too.  A minor-version-level release is supposed to be strictly backwards compatible.  But, well, making it reliably so is actually hard to do, it takes time and effort, that isn’t as fun as adding cool new stuff, and takes developer resources that an all-volunteer project simply may not have.

So Rails ends up having to release a 3.1.3 release of Rails that requires sprockets 2.0.3 instead of allowing 2.1.0, because neither the Rails developers nor the sprockets developer(s) have time to figure out what’s really going on. (And it’s not very joyful work to do so!).  But someone’s probably going to have to do that eventually, Rails can’t stay at an old version of sprockets forever, because there will be bugfixes and improvements that Rails and it’s users want — or because Rails users will want to use a gem that has a dependency on a more recent version of (eg) sprockets.

There’s no magic answer to this issue that I know of. No “if only people would do this” or “if only we had this.”  Before bundler, there was a tooling gap. But bundler greatly improves things, giving you the right control over your dependencies when you’ve got a complicated dependency tree. But it doesn’t really solve the problem.  And I would not prefer if the ruby community went to another extreme of focusing on stability and maintenance and backwards compatibility at the expense of awesome code. Ideally for me, I guess, more open source ruby developers would start moderating more toward the stability/maintenance direction, which I think has started to happen a bit, as more mature and legacy ruby code is out there and the ruby community gets bigger.

In my own projects though, I’ve realized that assembling small re-useable de-coupled components is not always more efficient — whether the small pieces are developed by me or someone else.  Yeah, you saved some initial time not re-inventing the wheel, but that time can easily be lost debugging component interactions when something goes wrong, patching dependencies, arguing with developers of dependencies on whether the change you need to support your use is really the right one for the dependency, etc.  Sometimes it really is more efficient to build and maintain your own more tightly coupled wheel.


4 thoughts on “loosely coupled components, developer joy, and dependency hell

  1. Good article, thanks for sharing your thoughts. I’d only yesterday tweeted about being tangled up in a web of dependencies in my current rails 2.3 project.

    It seems to me like the longer you try to hold onto older versions of particular groups of tools (rails, rspec and cucumber for example), the harder it becomes when you eventually are forced into upgrading.

    If anyone has a good way of handling this I’m certain that we’d all like to hear it!

  2. Same here: I have an app on Rails 2.3, Hobo 1.0. No time to do the dependency management hell, and no time to test. I’m hobbling along on what’s there now. Not a happy state of affairs.

  3. It’s true that if you hold onto old versions for a while, you’ll have a big confusing job to do when you finally upgrade (unless you somehow never need to). — but the alternative, in rubyland where things evolve quickly, is _constantly_ spending time on upgrading everything. So I’m not sure if it’s easily predictable which approach will be most efficient use of your time in the long term, on any given project.

    But I certainly know that trying to upgrade a Rails 1.0 app (that uses some external gems and plugins) to Rails 3.0 is a terrible, terrible, thing to have to do. And the 1.0 app (actually, my real app was a rails 2.2 app at this point although it started as 1.0, currently moving it to 3.1) was “only” written a few years ago, we’re not talking ancient history!

  4. It also just occured to me that this is one thing you get with a NEW language/environment, like ruby was for most of it’s community not too long ago — when starting in a new field, you automatically get to ignore backwards compat, maintenance, dealing with old code, and make everything as great as you can, from your experience with the LAST language you used.

    This is perhaps one reason why new languages with small communities seem to attractive to some people — it’s actually in part not related to any features of the language or environment itself, it’s in fact it’s actual _newness_, giving the developers the opportunity to do things “right”, not like the crap they did in the last language they abandoned.

    Wait a few years, and that new language/environment will now have a bunch of open source libraries that aren’t as good as people want, and you have to either abandon them for something new (causing backwards-compat problems for people who are using em), or worry about backwards-compat and stabiltiy (slowing down your ability to replace it with what you now realize would be better).

    Easier to abandon the whole language/environment/community for a new one where you can start from scratch!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s