loosely coupled components, developer joy, and dependency hell

The ruby open source community’s behavior makes two choices with interesting consequences:

1. The ruby open source community tries very hard to never re-write what someone else has written, instead composing gems and applications out of smaller pieces of de-coupled re-usable code. People try to share their code in composable decoupled pieces, and people try to write code using such pieces.
2. The ruby open source community strongly prefers innovation to stability, spending a lot more energy on improving things to be cooler, easier, faster, better — than on providing backwards compatibility or supporting old code.

I think both of these are actually consequences of a couple other very related things:

1. The ruby community is very volunteer-collaboration driven. Most code is not written by people being paid specifically to write that code.   Although lots of code is written by people who are using it for work they get paid for, relatively little code is written by people who have writing that code, as open source,  as an assigned task. (Compare to Java open source community).  Most ruby open source code is a labor of love (for the code, for the other developers you are sharing with).
2. The ruby community cares a lot about ‘developer joy’. Going back to matz’s original intention for ruby: “Ruby is designed to make programming not only easy but also fun”.  But this is also related to the point above — a mostly volunteer developer community only happens when developing is fun, and the more fun something is, the more likely there will be sufficient volunteer developer capacity to accomplish it.

Likewise, a mostly volunteer developer community kind of needs to develop based on loosely coupled small units developed by other people, to get anything done. We need to absolutely maximize our efficiency, because we don’t have armies of programmers getting paid to write what we need, we’ve got to take maximum advantage of what we’ve got.

But there’s a downside. Dependency hell.  When your product is built from dependencies on a bunch of things written by other people (and each of those has it’s own dependencies) — and especially when everything is constantly changing as people try to make things better, caring less about backwards compat and stability — it means things can be  constantly breaking.

After having spent a few years writing production ruby/rails code now, and still working at the same place where I wrote that code — I lately spend a huge amount of time upgrading my stuff to work with new versions of dependencies, or abandoning one now not-sufficiently-maintained dependency for another, etc.

Each individual upgrade may not be that hard, but they add up. It’s a huge portion of my time these days. And this is not a very joyful state, running on the treadmill just to stay in place instead of creating awesome new things. Ironically, optimizing for developer joy in the upstream dependencies (those developers don’t find it joyful to work on backwards-compat or refrain from releasing new ways of doing things because it’ll require people to change) — has made the joy of the downstream developer using all those dependencies very non-optimal.

Another interesting example is the way the recent Rails release had to downgrade sprockets to fix a regression. Rails too is built from a bunch of building blocks many developed by others (it always was to some extent, but Rails 3.0 pushed it even more in this direction). Which means those dependencies can easily break Rails too.  A minor-version-level release is supposed to be strictly backwards compatible.  But, well, making it reliably so is actually hard to do, it takes time and effort, that isn’t as fun as adding cool new stuff, and takes developer resources that an all-volunteer project simply may not have.

So Rails ends up having to release a 3.1.3 release of Rails that requires sprockets 2.0.3 instead of allowing 2.1.0, because neither the Rails developers nor the sprockets developer(s) have time to figure out what’s really going on. (And it’s not very joyful work to do so!).  But someone’s probably going to have to do that eventually, Rails can’t stay at an old version of sprockets forever, because there will be bugfixes and improvements that Rails and it’s users want — or because Rails users will want to use a gem that has a dependency on a more recent version of (eg) sprockets.

There’s no magic answer to this issue that I know of. No “if only people would do this” or “if only we had this.”  Before bundler, there was a tooling gap. But bundler greatly improves things, giving you the right control over your dependencies when you’ve got a complicated dependency tree. But it doesn’t really solve the problem.  And I would not prefer if the ruby community went to another extreme of focusing on stability and maintenance and backwards compatibility at the expense of awesome code. Ideally for me, I guess, more open source ruby developers would start moderating more toward the stability/maintenance direction, which I think has started to happen a bit, as more mature and legacy ruby code is out there and the ruby community gets bigger.

In my own projects though, I’ve realized that assembling small re-useable de-coupled components is not always more efficient — whether the small pieces are developed by me or someone else.  Yeah, you saved some initial time not re-inventing the wheel, but that time can easily be lost debugging component interactions when something goes wrong, patching dependencies, arguing with developers of dependencies on whether the change you need to support your use is really the right one for the dependency, etc.  Sometimes it really is more efficient to build and maintain your own more tightly coupled wheel.

Advertisement