Blacklight Strengths, Weaknesses, Health, and Future

My Own Personal Opinion Analysis of Blacklight Strength, Weaknesses, Health, and Future

My reflections on the Blacklight Community Survey results, and my own experiences with BL.

What people like about BL is it’s flexibility; what people don’t like is it’s complexity and backwards/forwards compatibility issues.

Developing any software, especially shared library/gem software, it is difficult to create a package which is on the one hand very flexible, extensible, and customizable; and on the other maintains a simple and consistent codebase, backwards compatibility with easy upgrades, and simple installation with a shallow learning curve for common use cases.

In my software engineering career, I see these tensions as one of the fundamental challenges in developing shared software. It’s not unique to Blacklight, but I think Blacklight is having some difficulties in weathering that challenge.

I think the diversity of Blacklight versions in use is a negative indicator for community health. People on old unsupported versions of BL (or Rails) can run into bugs which nobody can fix for them; and even if they put in work on debugging and fixing them themselves, it’s less likely to lead to a patch that can be of use to the larger BL community, since they’re working on an old version. It reduces the potential size of our collaborative development community. And it puts those running old versions of BL (or Rails) in a difficult spot eventually after much deferred upgrading, when they find themselves on unmaintained software with a very challenging upgrade path across many versions.

Also, if when a new BL release is dropped it’s not actually put into production by anyone (not even core committers?) for many months, that increases the chances that severe bugs are present but not yet found in even months-old releases (we have seen this happen), which can be a vicious circle that makes people even more reluctant to upgrade.

And we have some idea why BL applications aren’t being upgraded: Even though only a bare minority of respondents reported going through a major BL upgrade, issues with difficulty of upgrading are a very major represented theme in reported biggest challenges with Blacklight.  I know I personally have found that maintaining a BL app responsibly (which to me means keeping up with Rails and BL releases without too much lag) has had a much higher “total cost of ownership” than I expected or desire; you can maybe guess that part of my motivation in releasing this survey was to see if I was alone, I see I am not.

I think these pain points are likely to get worse: many existing BL deployments may have been originally written for BL 5.x and not yet had to deal with them but will; and many people currently using a “release and forget and never upgrade” practice may come to realize this is untenable. (“Software is a growing organism”, Ranganathan’s fifth law. Wait, Ranganathan wasn’t talking about software?)

To be fair, Blacklight core developers have gotten much better at backwards compatibility — in BL 4.x and especially 5.x — in the sense that backwards-incompatible changes within a major BL version are attempted, with much success, to be kept minimal to non-existent (in keeping with semver‘s requirements for release labelling).  This is a pretty major accomplishment.

But the backwards compatibility is not accomplished by minimizing code or architectural churn or change. Rather the changes are still pretty fast and furious, but the old behavior is left in and marked deprecated. Ironically, this has the effect of making the BL codebase even more complicated and hard to understand, with multiple duplicative or incompatible architectural elements co-existing and sometimes never fully disappearing. (More tensions between different software quality values, inherent challenges to any large software project.)  In BL 5.x, the focus on maintaining backwards compat was fierce — but we sometimes got deprecated behavior in one 5.x release, with suggested new behavior, where that suggested new behavior was sometimes itself deprecated in a future 5.x release in favor of yet newer behavior.  Backwards compatibility is strictly enforced, but the developer’s burden of keeping up with churn may not be as lightened as one would expect.

Don’t get me wrong, I think some of the 5.x changes are great designs. I like the new SearchBuilder architecture. But it was dropped in bits and pieces over multiple 5.x releases, without much documentation for much of it, making it hard to keep up with as a non-core developer not participating in writing it.  And the current implementation still has, to my mind, some inconsistencies or non-optimal choices (like `!` on the end of a method or lack thereof being inconsistently used to signal a method mutates the receiver vs returns-a-dup) — which now that they are in a release, need to be maintained for backwards compatibility (or if changed in a major version drop, still cause backwards compat challenges for existing app maintainers; just labeling it a major version doesn’t reduce these challenges, only reducing the velocity of such changes does).

In my own personal opinion, Blacklight’s biggest weakness and biggest challenge for continued and increased success is figuring out ways to maintain the flexibility, while significantly reducing code complexity, architectural complexity, code churn, and backwards incompatibility/deprecation velocity.

What can be done (in my own personal opinion)?

These challenges are not unique to Blacklight, they are tensions and challenges, in my observation/opinion/experience with nearly any shared non-trivial codebase.  But Blacklight can, perhaps, choose to take a different tack to approaching them, focus on different priorities in code evolution, think of practices to adopt to strike a better balance.

The first step is consensus on the nature of the problem (which we may not have, this is just my own opinion on the nature of the problem; I’m hoping this survey can help people think about BL’s strengths and weaknesses and build consensus).

In own brainstorming about possible approaches, I come up with a few, tentative, brainstorm-quality, possibilities:

  • Require documentation for major architectural components. We’ve built a culture in BL (and much of open source world) that a feature isn’t done and ready to merge until it’s covered by tests; I think we should have a similar culture around documentation, a feature isn’t done and ready to merge until documented. Which we lack in BL, and much of the open source world. But this can add a challenge, in a codebase with high churn, you now have to make sure to update the docs lest they become out of date and inaccurate too (something BL also hasn’t always kept up with)….
  • Accept less refactoring of internal architecture to make the code cleaner and more elegant.  Sometimes you’ve just got to stick with what you’ve got, for longer, even if the change would improve code architecture, as many of them have.  There’s an irony here. Often the motivation for an internal architectural refactoring is to better support something one wants to do. You can do that in the current codebase, but in a hacky not really supported way, that’s likely to break in future BL versions. You want to introduce the architecture that will let you do what you want in a ‘safer’ way, for forwards compatibility. But the irony is that the constant refactorings to introduce these better architectures actually have a net reduction on forwards compatibility, as they are always breaking some existing code.
  • Be cautious of the desire to expel functionality to external plugins. External BL plugins generally receive less attention, they are likely to not be up to date with current BL, and it has been difficult to figure out what version of an external plugin actually is compatible with what version of BL. If you’re always on the bleeding edge, you don’t notice, but if you have an older version of BL and are maybe trying to upgrade to a new one, figuring out plugin compatibility in BL can be a major nightmare. Expelling code to plugins makes core BL easier to maintain, but at the cost of making the plugins harder to maintain, less likely to receive maintainance, and harder to use for BL installers.  If the plugin code is an edge case not used by many people that may make sense. But I continue to worry about the expulsion of MARC support to a plugin. MARC is not used by as many BL implementers as it used to be, but “library catalog/discovery” is still BL use for a third of survey respondents, and MARC is still used by nearly half.
  • Do major refactorings in segregated branches, only merging into master (and including in releases) when they are “fully baked”. What does fully baked mean? I guess maybe it means understanding the use cases that will need to be supported; having a ‘story’ about how to use the architecture for those use cases; having a few people actually looked over the code and tried it out and given feedback.In BL 5.x, there were a couple major architectural refactorings, but they were released in dribs and drabs over multiple BL releases, sometimes reversing themselves, sometimes after realizing there were important use cases which couldn’t be supported. This adds TCO/maintenance burden to BL implementers, and adds backwards-compat-maintaining burder to BL core developers when they realize something already released should have been done differently.If I understand right, the primary motivation for some of the major 5.x-6.0 architectural refactorings was to support ElasticSearch as an alternate back-end. But, while these refactorings have already been released, there has actually been no demonstration of using ElasticSearch as a front-end, it’s not done yet. Without such demonstrations trying and testing the architecture, how confident can we be that these refactorings will actually be sufficient or the right direction for the goal? Yet more redesigns may be needed before we get there.

When I brought up this last point with a core BL developer, he said that it was unrealistic to expect this could be possible, because of the limited developer resources available to BL.

It’s true that there are very few developers making non-trivial commits to BL, and that BL does function in an environment of limited developer resources, which is a challenge. However, in fact, studies have shown that most succesful open source projects have the vast majority of commits contributed by only 1-3 developers. (Darnit, I can’t find the cite now).

I wonder if beyond developer resources as a ‘quantity’, the nature of the developers and their external constraints matters. Are many of the core BL developers working for vendors, where most hours need to be billable to clients on particular projects which need to be completed as quickly as practical?  Or working for universities that have a similar ‘entrepeneurial’ approach where most developer hours are spent on ‘sprints’ for particular features on particular projects?

Is anyone given time to steward the overall direction and architectural soundness of Blacklight?  If nobody really has such time, it’s actually a significant accomplishment that BL’s architecture has continued to evolve and improve regardless. But it’s not a surprise that it’s done so in a fairly chaotic and high-churn way, where people need to get just enough to accomplish the project in front of them into BL, and into a release asap.

I suspect that BL may, at this point in it’s development, need a bit more formality and transparency in who makes major decisions. (Eg, who decided that supporting ElasticSearch was a top priority, and how?) (And I say this as someone that, five years ago at the beginning of BL, didn’t think we needed any more formality there then a bunch of involved developers reaching consensus on a weekly phone call (that I don’t think happens anymore?). But I’ve learned from experience, and BL is at a different point in it’s life cycle now.)

In software projects where I do have some say (I haven’t made major commits to BL in at least 2-3 years), where they are projects that are expected to have long lives, I’ve come to try to push for a sort of “slow programming” (compare to ‘slow food’ etc) approach. Consider changes carefully, even at the cost of reducing velocity of improvements, release nothing into master before it’s time, prioritize backwards compatibility over time (not just over major releases, but actual calendar time). Treat your code like a bonsai tree, not a last-minute term paper.  But sometimes you can get away with this sometimes you can’t, sometimes your stakeholders will let you sometimes they won’t, and sometimes it isn’t really the right decision.

Software design is hard!


5 thoughts on “Blacklight Strengths, Weaknesses, Health, and Future

  1. (I’m not a Blacklight developer – I help maintain a production Blacklight app.) I like your idea of slower releases, and making sure new designs are fully-baked before being released (to reduce the churn you’re describing). We are currently using Blacklight 5.13.1, but I *think* that’s more from a desire to keep things up-to-date and maintained than from a need for any new features (I think our app may be fine with an older version of Blacklight). Sort of related point, we’re still on Rails 4.1.x, and don’t have a need to move to the newer 4.2.x.

  2. I think that Blacklight needs an administrative interface that allows library administrators some ability to customize things. These could be just the location of the Solr servers, collections and/or cores, but there could also be places to add CSS and/or JavaScript to specific pages.

    If you want to get *really* fancy, you could use CodeMirror to allow CSS, JavaScript, and even Ruby helpers and ERB code for fields to be customized directly through the interface. The problem of course with this is that it can also work against proper software engineering, preventing developers from keeping their code in a git repository. Products often stumble on this one and make it easy to “customize” while making it hard to “maintain”, by which I mean “evolve”.

    On the word “evolve” – Ranganathan’s fifth law is about libraries, but Lehman’s Laws of Software Evolution back-up your highlight on maintenance. Lehman states that 90% of software maintenance is really evolution, by which he mean’t the adaptation to new/changed requirements. This has been at various times refreshed to expand the definition of evolution to include update of 3rd party dependencies, and also to validate the result on 3rd and 4th generation languages.

  3. Another thought – because it is a “gem”, blacklight developers think of blacklight like a plugin to a rails app, but your users likely think of blacklight as a system that needs to support plugins of its own.

  4. > I think that Blacklight needs an administrative interface that allows library administrators…

    I don’t think there’s anyone who would be displeased to see that, but it takes an enormous amount of developer skill and hours to pull off well. I don’t think it’s realistic to expect a Blacklight that can actually be run by non-developers any time soon, so I think even attempts in that direction would be a false promise, not implemented robustly, that would likely lead to an increase rather than a decrease in dissatisfaction.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s