open source, engineering professional ethics, complicity, and chef

So an open topic of controversy in open source philosophy/ideology/practice (/theology), among those involved in controversing on such things, has been “field of endeavor” restrictions. If I release software I own the copyright to as (quasi-)open source, but I try to say that legally you can’t use it for certain things, or the license suggests I have the legal right to withdraw permission for certain entities to be named later… is this truly “open source”? Is it practical at all, can we as developers get what we want out of shared collaborative gift-economy-esque software if everyone starts doing that? GPL/rms says it’s not workable to try it,  and the Open Source Initiative says it’s not “open source” if you try it. Both the GPL/”viral”/free-as-in-libre and the Apache/MIT-style/unencumbered/”corporate” sides of open source theology seem to agree on this one, so maybe the controversy hasn’t been all that open, but it comes up in internet arguments.

I’m honestly not sure how to work it all out in legal/licensing or social/practice-of-engineering systems, I don’t think there’s a pat answer, but I know I wouldn’t be happy about software I wrote and shared open source with “gift economy” intentions, to find it was being used — with no interaction with me personally — by, say, the Nazis in Nazi Germany, or, just another of course unrelated example, ICE/CBP. It would lead me to question how I had directed my labor, based on the results.

But that basic situation is NOT, in fact, quite what’s going on here, or at least all that’s going on here, in this article from Vice’s Motherboard, ‘Everyone Should Have a Moral Code’ Says Developer Who Deleted Code Sold to ICE, by Joseph Cox.

Rather than releasing open source software and discovering that someone had chosen to use it for unpleasant purposes on their own, Chef, Inc. instead seems to have a $100,000 contract with ICE of some kind, where Chef makes money helping or providing software to help ICE manage their information systems in some way (using the chef software).

And Seth Vargo used to work for Chef, Inc., but no longer does… but apparently still had admin permissions to code repos and release artifacts for some open source parts of chef. And maybe kept making open source code writing/reviewing/releasing contributions after he was no longer an employee? Not sure. The Motherboard article is short on the details we curious software engineers would want on the social/business/licensing aspects, and I haven’t done the research to track it all down yet, sorry; I don’t believe the specific nature of Chef Inc’s business with ICE is publicly known.

Personally, I was aware of chef-the-software but my own experience with it has not gone beyond skimming docs to get a basic idea of what it does. I had been under the (mistaken?) impression the whole thing was open source, which left me confused by what code Chef Inc “sold” to ICE (in the Motherboard headline) how… but googled and discovered it had been “open core”, but in April 2019 all the code was released with an apache license… still a bit confused what’s going on.

At any rate, Seth Vargo apparently was kinda furious that code he wrote was being used to help ICE manage their information systems, for organizing, you know, concentration camps and child abuse and fundamental violations of human rights and dignity and stuff like that.  (And if it were me, I’d be especially enraged that someone was making money off doing that with the code I wrote, not sure how that reaction fits into a moral philosophy, but I know I’d have it).  And Vargo did some things he could to disrupt it, at least a bit, (basically deleting and misconfiguring things that can, ultimately, still be fairly easily/quickly restored). I think he deserves support for doing so, and for bringing more attention to the case in part by doing so.

Meanwhile, these quotes from Chef CEO Barry Crist are just ridiculous. 

“While I understand that many of you and many of our community members would prefer we had no business relationship with DHS-ICE, I have made a principled decision, with the support of the Chef executive team, to work with the institutions of our government, regardless of whether or not we personally agree with their various policies,” Crist wrote, who added that Chef’s work with ICE started during the previous administration.

“My goal is to continue growing Chef as a company that transcends numerous U.S. presidential administrations. And to be clear: I also find policies such as separating families and detaining children wrong and contrary to the best interests of our country,” he wrote.

This is the statement of a moral coward. He does not seem to realize he’s essentially telling us “I want you to know, I have values, I’m not a monster! It’s just that I’m willing to sacrifice all of them for the right price, like anyone would be, right?”

He even suggests there is something “principled” about the decision “to work with the institutions of our government, regardless of whether or not we personally agree with their various policies.” While 1930s IBM agreed with the “principle” of aiding efforts of any government whose money was good, say, maybe in Germany, “whether or not anyone personally agreed” with the efforts they were aiding… this is a self-serving sociopathic Ayn Rand-ian “principle”.

These comments kept burning me up, I couldn’t get them out of my head… and then I realized this is basically the conversation in Boots Riley’s batshit political parody(?) 2018 film Sorry to Bother You (SPOILERS AHEAD) , in what was for me the most genius gut-punchingly horribly hilarious moment in a movie that has plenty of them. A scene which doesn’t come across nearly as well in just text transcript without the context and body language/tone and exceptional delivery of the actors, but I’m gonna give it to you anyway.

So at one point in Sorry To Bother You, just after Cash has discovered the results of the rich CEO’s secret plan for engineering horse-human hybrids out of kidnapped conscripts, the CEO has shown the terrified and confused Cash an in-house promotional video explaining the, uh, well-thought-out business model for horse-human slave labor. The video ends, and:

CEO: See? It’s all just a big misunderstanding.

Cash: This ain’t no fucking ‘misunderstanding’, man.
So, you making half-human half-horse fucking things so you can make more money?

CEO: Yeah, basically. I just didn’t want you to think I was crazy. That I was doing this for no reason. Because this isn’t irrational.

Cash: Oh…. Cool. Alright. Cool…. No, I understand. I just, I just got to leave now, man. So, please get the fuck out of my way.

Of course we don’t agree with what ICE is doing, we don’t want you to think we’re crazy… it’s just that the principle of being able to “grow Chef as a company” by helping them do those things wins out, right?

With what I know now, I would  never work for Chef Inc. or contribute any code to any chef projects, and will be using any power or sway I have to dissuade anyone i work for or with from using chef. (I don’t think any do at present, so it’s not much of a sacrifice/risk to me at present, to be sure).  Engineering ethics matter. These are not good times, it’s not always clear to me either what to do about it, anyone who sees somewhere they can afford to intervene should take the opportunity, we can’t afford to skip any, large or small.

Incidentally I found out about this story by  seeing this cryptic post on the rubygems blog, noticing it via the Rubyland News aggregator I run, and then googling to figure out what weirdness was going on with chef to prompt that, and finding the Motherboard article by Joseph Cox. Also credit to @shanley for, apparently, discovering and publicizing the Chef/ICE contract. And to rubygems/Evan Phoenix for transparently posting evidence they had forcibly changed gem ownership, rather than do it silently.

I probably wouldn’t have noticed at all if Vargo hadn’t made it a story by engaging in some relatively easy low-risk direct action, which is really the least any of us should do in such a situation, but Vargo deserves credit and support because so many of us engineers maybe wouldn’t have, but it’s time for us to figure out how to step up.

In some more welcome news from here in Baltimore, Johns Hopkins University/Medical Institutions is reported to have recently declined to renew some ICE contracts — including one for “tactical medical training” to agents in the Homeland Security Investigations unit of ICE, which carries out workplace raids — occurring after a Hopkins student-led but community coalition campaign of public pressure on Hopkins to stop profiting from supporting and facilitating ICE/CBP human rights violations. While the ~$1.7 million ICE contracts were relatively small money in Hopkins terms, Hopkins as an institution has previously shown itself to be quite dedicated to that same “principle” of never, ever, turning down a buck; may this breach of profiteering “principle” lead to many more.

Advertisements

Card Catalogs: “Paper Machines”

A book I just became aware of that I am very excited about (thanks to Jessamyn West for posting a screenshot of her ‘summer reading’ on facebook, bringing it to my attention!)

Paper Machines: About Cards & Catalogs, 1548-1929
by Krajewski PhD, Markus (Author), Peter Krapp (Translator)

Why the card catalog―a “paper machine” with rearrangeable elements―can be regarded as a precursor of the computer.

Today on almost every desk in every office sits a computer. Eighty years ago, desktops were equipped with a nonelectronic data processing machine: a card file. In Paper Machines, Markus Krajewski traces the evolution of this proto-computer of rearrangeable parts (file cards) that became ubiquitous in offices between the world wars.

The story begins with Konrad Gessner, a sixteenth-century Swiss polymath who described a new method of processing data: to cut up a sheet of handwritten notes into slips of paper, with one fact or topic per slip, and arrange as desired. In the late eighteenth century, the card catalog became the librarian’s answer to the threat of information overload. Then, at the turn of the twentieth century, business adopted the technology of the card catalog as a bookkeeping tool. Krajewski explores this conceptual development and casts the card file as a “universal paper machine” that accomplishes the basic operations of Turing’s universal discrete machine: storing, processing, and transferring data. In telling his story, Krajewski takes the reader on a number of illuminating detours, telling us, for example, that the card catalog and the numbered street address emerged at the same time in the same city (Vienna), and that Harvard University’s home-grown cataloging system grew out of a librarian’s laziness; and that Melvil Dewey (originator of the Dewey Decimal System) helped bring about the technology transfer of card files to business.

I haven’t read it yet myself.

But I’ve thought for a while about how card catalogs were pre-computer information processing systems (with some nostalgia-for-a-time-i-didn’t-experience-myself of when library science was at the forefront of practically-focused information processing system theory and practice).

And I’ve realized for a while that most of our legacy data was designed for these pre-computer information processing systems. And by “legacy” data, I mean the bulk of data we have :) MARC, AACR2, LCSH, even call number systems like DDC or LCC.

If you want to understand this data, you have to understand the systems it was designed for — their affordances and constraints, how they evolved over time — and thinking of them as information processing machines is the best way to understand it, and understand how to make use of it in the present digital environment, or how to change it to get the most benefit from the different constraints and affordances of a computerized environment.

So I can’t quite recommend the book, cause I haven’t read it myself yet — but I recommend it anyway. :)

Dealing with legacy and externally loaded code in webpack(er)

I’ve been mostly a ruby and Rails dev for a while now, and I’ve been a ‘full-stack web dev’ since that was the only kind of web dev. I’ve always been just comfortable enough in Javascript to get by — well, until recently.

The, I don’t know what you call it, “modern JS” (?) advances and (especially) tooling have left me a bit bewildered. (And I know I’m not alone there).  But lately I’ve been pulled (maybe a bit kicking and screaming) into Webpacker with Rails, because you really need modern npm-based tooling to get some JS dependencies you want — and Webpacker will be the default JS toolchain in Rails 6 (and I think you won’t have access to sprockets for JS at all by default).

If the tooling wasn’t already confusing enough — and Webpacker makes webpack a little bit easier to use with Rails-friendly conventions over configuration, but also adds another layer of indirection on understanding your tools — I frequently have to deal with projects where not all the code is managed with webpacker.

I might have dependencies to JS provided only via Rails engine gems (no npm package). I might have legacy projects where not all the code that could be transitioned to webpack(er) control has been yet. And other reasons.  So I might have some code being included via a webpack and javascript_pack_tag, but some code being included in a separate compiled JS via sprockets and javascript_include_tag, and maybe other code doing other odd things.

  • Might need webpacker-packed code that uses dependencies loaded via external mechanisms (sprockets, or a raw <script> tag to a CDN host).
  • Might need non-webpacker-packed code (ie, sprockets-managed usually) that uses a dependency that is loaded by webpacker (because npm/yarn is the best way to get a lot of JS dependencies)
  • Might have “vendored” third-party code that is old and doesn’t play well with ES6 import/export.

So I decided to take some time and understand the webpack(er) patterns and features relevant here. Some webpack documentation calls these techniques for “shimming”, but I think they are relevant for cases beyond what I would consider “shimming”. These techniques are generally available in webpack, but my configuration examples will be Webpacker, cause lack of webpacker examples was a barrier to newbie me in figuring this out.

I am not an expert in this stuff and appreciate any corrections!

“Externals” — webpack code depending on a library loaded via other means

Let’s say we load Uppy via a manual script tag to CDN (so it’s available old-style via window.Uppy after load), but have webpacker-packed code that needs to refer to it. (Why are we loading Uppy that way? I dunno, we might have Reasons, or it might just be legacy in mid-point in being migrated to webpacker).

You want to use the webpack externals feature.

In your config/webpack/environment.js: (after “const { environment } = require(‘@rails/webpacker’)” and before “module.exports = environment”)

environment.config.externals = {
  uppy: 'Uppy'
}

And now you can import Uppy from 'uppy'; in a webpacker source just like you would if Uppy was a local yarn/npm dependency.

The typical examples do this with jQuery:

  externals: {
    jquery: 'jQuery'
  }

Note: In my experimentations, I found that I can apparently just use Uppy (when it’s loaded in window.Uppy by non-webpacker sources) in my webpacker sources without doing the externals setup and the import. I’m not sure why or if this is expected, but externals seems like better practice.

Note: Every time the pack is loaded, if window.Uppy is not available when you have that external, you’ll get a complaint in console “Uncaught ReferenceError: Uppy is not defined”, and your whole JS pack won’t be loaded due to aborting on the error — this tripped me up when I was trying to conditionally load Uppy from CDN only on pages that needed it, but other pages had the pack loaded. I guess the right way to do this would be having separate pack files, and only register the externals with the pack file that actually uses Uppy.

Note: I don’t have Uppy in my package.json/yarn.lock at ALL, so I know webpacker isn’t compiling it into the pack. If I did, but for some reason still wanted to rely on it from an ‘external’ instead of compiling it into the pack, I’d want to do more investigative work to make sure it wans’t in my pack too, resulting in a double-load in the browser since it was already being loaded via CDN.

“Expose” — make a webpack(er) loaded dependency available to external-to-webpack JS

Let’s say you have openseadragon being controlled by webpacker and included in your pack. (Because how else are you going to get the dependency? The old method of creating a rails engine gem with a vendored asset, and keeping it up to date with third-party releases, is a REAL DRAG).

But let’s say the code that uses openseadragon is not controlled by webpacker and included in your pack. It’s still being managed and delivered with sprockets. (Why? Maybe it’s just one step along a migration to webpacker, in which you want to keep everything working step by step.)

So even though OpenSeadragon is being included in your pack, you want it available at window.OpenSeadragon “old-style”, so the other code that expects it there old-style can access it. This is a task for the webpack expose-loader.

You’ll need to yarn add expose-loader — it doesn’t come with webpack/webpacker by default. (You don’t seem to need any configuration to make it available to webpack, once you’ve added it to your package).

So you’ve already yarn add openseadragon-ed. Now in your config/webpack/environment.js: (after “const { environment } = require(‘@rails/webpacker’)” and before “module.exports = environment”)

environment.loaders.append('expose', {
  test: require.resolve('openseadragon'),
  use: [{
          loader: 'expose-loader',
          options: 'OpenSeadragon'
  }]
})

Now window.OpenSeadragon will be set, and available to JS sources that came from somewhere else (like sprockets) (or just accessed as OpenSeadragon in that code).

That is, as long as openseadragon is included in your pack. The “expose” loader directive alone won’t put it in your pack, and if it’s not in your pack, it can’t be exposed at window. (and webpacker won’t complain).

So if you aren’t already including it in your pack, over in (eg) your app/javascript/packs/application.js, add one of these:

// You don't need all of these lines, any one will do:
import 'openseadragon'
import OpenSeadragon from 'openseadragon'
require('openseadragon')

Now OpenSeadragon is included in your pack file, and exposed at window.OpenSeadragon for non-packed JS (say in a sprockets-compiled file) to access.

If you are loading jQuery in your pack, and want to make it available to “external” JS at both jQuery and $ to support either as ordinary JQuery, you want:

environment.loaders.append('expose', {
  test: require.resolve('jquery'),
  use: [{
    loader: 'expose-loader',
    options: 'jQuery'
  }, {
    loader: 'expose-loader',
    options: '$'
  }]
})

“Provide” — automatic “import” for legacy code that doesn’t

Let’s say you are including jQuery with webpacker, in your pack. Great!

And you have some legacy code in sprockets you want to move over to webpacker. This legacy code, as legacy code does, just refers to $ in it, expecting it to be available in window.$ . Or maybe it refers to jQuery. Or a little bit of both.

The “right” way to handle this would be to add import jQuery from 'jquery' at the top of every file as you move it into webpacker. Or maybe import $ from 'jquery'. Or if you want both… do you do two imports? I’m not totally sure.

Or, you can use the webpack ProvidePlugin to avoid having to add import statements, and have $ and jQuery still available (and their use triggering an implicit ‘import’ so jQuery is included in your pack).

In the middle of your config/webpack/environment.js:

const webpack = require('webpack');
environment.plugins.append('Provide', new webpack.ProvidePlugin({
  $: 'jquery',
  jQuery: 'jquery'
}));

Now you can just refer to $ and jQuery in a webpacker source, and it’ll just magically be as if you had imported it.

For code you control, this may be just a convenience, there ought to be a way to get it working without ProvidePlugin, with the proper import statements in every file. But maybe it’s vendored third-party code that was written for a “pre-modern” JS world, and you don’t want to to editing it. ProvidePlugin magically makes it automatically “import” what it needs without having to go adding the right ‘import’ statements everywhere.

Other WebPack plugins of note for legacy code

The ImportsLoader works very much like the ProvidePlugin above. But while the ProvidePlugin makes it magic happen globally — any file in the pack that references an “auto-imported” constant will trigger an import — the ImportsLoader lets you scope that behavior to only specific files.

That seems better overall — avoid accidentally using automatic import in some non-legacy code where you intend to be doing things “right” — but for whatever reason ImportsLoader is discussed a lot less on the web than ProvidePlugin, and I didn’t discover it until later, and I haven’t tried it out.

The ExportsLoader seems to be a way of magically getting legacy code to do ES6 exports (so they can be imported by other webpack sources), without actually having to edit the code to have exports statements. I haven’t played with it.

More Sources

In addition to the docs linked to above for each feature, there are a couple WebPack guides on ‘shimming’ that try to cover this material. I’m not sure why I found two of them and they don’t quite match in their recommendations, not sure which is more up to date. 1) Shimming in Webpack docs. 2) “Shimming modules” in webpack github wiki

My favorite blog post covering converting a sprockets-based Rails app to be webpacker-based instead is “Goodbye Sprockets. Welcome Webpacker” by Alessandro Rodi, although there are other blog posts covering the same goal written since I discovered Rodi’s.

In Rails6, you may not have sprockets available at all for managing Javascript, unless you hack (or politely just “configure”) it back in. This reddit comment claims to have instructions for doing so in Rails6, although I haven’t tried it yet (nor have I confirmed that in RC2 you indeed need it to get sprockets to handle JS; leaving this in part for myself, when I get to it). See also this diff.

Bootstrap 3 to 4: Changes in how font size, line-height, and spacing is done. Or “what happened to $line-height-computed.”

Bootstrap 4 (I am writing this in the age of 4.3.0) changes some significant things about how it handles font-size, line-height, and spacer variables in SASS.

In particular, changing font-size calculations from px units to rem units; with some implications for line-heights as handled in bootstrap; and changes to how whitespace is calculated to be in terms of font-size.

I have a custom stylesheet built on top of Bootstrap 3, and am migrating it to Bootstrap 4, and I was getting confused about what’s going on. And googling, some things are written about “Bootstrap 4” that are really about a Bootstrap 4 alpha, and in some cases things changed majorly before the final.

So I decided to just figure it out looking at the code and what docs I could find, and write it up as a learning exersize for myself, perhaps useful to others.

Bootstrap 3

In Bootstrap 3, the variable $font-size-base is the basic default font size. It defaults to 14px, and is expected to be expressed in pixel units.

CSS line-height is given to the browser as a unit-less number. MDN says “Desktop browsers (including Firefox) use a default value of roughly 1.2, depending on the element’s font-family.” Bootstrap sets the CSS line-height to a larger than ‘typical’ browser default value, having decided that is better typography at least for the default Bootstrap fonts.

In Bootstrap 3, the unit-less $line-height-base variable defaults to the unusual value of 1.428571429. This is to make it equivalent to a nice round value of “20px” for a font-size-base of 14px, when the unit-less line-height is multiplied by the font-size-base. And there is a line-height-computed value that’s defined as exactly that by default, it’s defined in terms of $line-height-base.  So line-height-base is a unit-less value you can supply to the CSS line-height property (which _scaffolding does on body), and line-height-computed is a value in pixels that should be the same size, just converted to pixels.

 

As a whitespace measure, in bootstrap 3

Bootstrap wants to make everything scale depending on font-size, so tries to define various paddings and margins based on your selected line height in pixels.

For instance, an alerts, breadcrumbs, and tables, all have a margin-bottom of $line-height-computed (default 20px, with the default 14px font size and default unit-less line-height). h1, h2, and h3 all have a margin-top of $line-height-computed.

h1, h2, and h3 all have a margin-bottom of $line-height-computed/2 (half a line heigh tin pixels; 10px by default). And ($line-height-computed / 2) is both margin-bottom and margin-top for a p tag.

You can redefine the size of your font or line-height in variables, but bootstrap 3 tries to express lots of whitespace values in terms of “the height of a line on the page in pixels” (or half of one) — which is line-height-computed, which is by default 20px.

On the other hand, other kinds of whitespace are expressed in hard-coded values, unrelated to the font-size, and only sometimes changeable by bootstrap variables either.  Often using the specific fixed values 30px and 15px.

$grid-gutter-width is set to 30px.  So is $jumbotron-padding, You can change these variables yourself, but they don’t automatically change “responsively” if you change the base font-size in $font-size-base. They aren’t expressed in terms of font-size.

A .list-group has a margin-bottom set to 20px, and a .list-group-item has a padding of 10px 15px, and there’s no way to change either of these with a bootstrap variable, they are truly hard-coded into the SCSS. (You could of course try to override them with additional CSS).

So some white-space in Bootstrap 3 does not scale proportionately when you change $font-size-baseand/or $line-height-base.

Bootstrap 4

In Bootstrap 4, the fundamental starting font-size variable is still $font-size-base, but it’s defined in terms of rem, it is by default defined to 1rem.

You can’t set $font-size-base to a value in px units, without bootstrap’s sass complaining as it tries to do things with it that are dimensionally incompatible with px. You can change it to something other than 1rem, but bootstrap 4 wants $font-size-base in rem units.

1rem means “same as the font-size value on the html element.”  Most browsers (at least most desktop browsers?) default to 16px, so it will usually by default mean 16px. But this isn’t required, and some browsers may choose other defaults.

Some users may set their browser default to something other than 16px, perhaps because they want ‘large print’. (Although you can also set default ‘zoom level’ instead in a browser; what a browser offers and how it effects rendering can differ between browsers). This is, I think, the main justification for Bootstrap changing to rem, accessibility improvements respecting browser default stylesheets.

Bootstrap docs say not much to explain the change, but I did find this:

No base font-size is declared on the <html>, but 16px is assumed (the browser default). font-size: 1rem is applied on the <body> for easy responsive type-scaling via media queries while respecting user preferences and ensuring a more accessible approach.

https://getbootstrap.com/docs/4.3/content/reboot/#page-defaults

Perhaps for these reasons of accessibility, Bootstrap itself does not define a font-size on the html element, it just takes the browser default. But in your custom stylesheet, you could insist html { font-size: 16px } to get consistent 1rem=16px regardless of browser (and possibly with accessibility concerns — although you can find a lot of people debating this if you google, and I haven’t found much that goes into detail and is actually informed by user-testing or communication with relevant communities/experts).  If you don’t do this, your bootstrap default font-size will usually be 16px, but may depend on browser, although the big ones seem to default to 16px.

(So note, Bootstrap 3 defaulted to 14px base-font-size, Bootstrap 4 defaults to what will usually be 16px). 

Likewise, when they say “responsive type-scaling via media queries”, I guess they mean that based on media queries, you could set font-size on html to something like 1.8​, meaning “1.8 times as large as ordinary browser default font-size.”  Bootstrap itself doesn’t seem to supply any examples of this, but I think it’s what it’s meant to support. (You wouldn’t want to set the font-size in px based on a media-query, if you believe respecting default browser font-size is good for accessibility).

Line-height in Bootstrap 4

The variable line-height-base is still in Bootstrap 4, and defaults to 1.5.  So in the same ballpark as Bootstrap 3’s 1.428571429, although slightly larger — Bootstrap is no longer worried about making it a round number in pixels when multiplied against a pixel-unit font-size-base.  line-height-base is still set as default line-height for body, now in _reboot.scss (_scaffolding.scss no longer exists).

$line-height-computed, which in Bootstrap 3 was “height in pixel units”, no longer exists in Bootstrap 4. In part because at CSS-writing/compile time, we can’t be sure what it will be in pixels, because it’s up to the browser’s default size.

If we assume browser default size of 16px, the “computed” line-height it’s now 24px, which is still a nice round number after all.

But by doing everything in terms of rem, it can also change based on media query of course. So if the point of Bootstrap 3 line-height-computed was often to use for whitespace and other page-size calculations based on base font-size, if we want to let base-font-size fluctuate based on a media query, we can’t know the value in terms of pixels at CSS writing time.

Bootstrap docs say:

For easier scaling across device sizes, block elements should use rems for margins.

https://getbootstrap.com/docs/4.3/content/reboot/#approach

Font-size dependent whitespace in Bootstrap 4

In Bootstrap 3, line-height-computed ) (20px for 14px base font; one line height) was often used for a margin-bottom.

In Bootstrap 4, we have a new variable $spacer that is often used. For instance, table now uses $spacer as margin bottom.  And spacer defaults to… 1rem. (Just like font-size-base1, but it’s not defined in terms of it, if you want them to match and you change one, you’d have to change the other to match).

alert and breadcrumbs both have their own new variables for margin-bottom, which also both default to: 1rem. Again not in terms of font-size-base, just happen to default to the same thing.

So one notable thing is that Bootstrap 3, as related to base font size, is putting less whitespace in margin-bottom on these elements. In Bootstrap 3, they got the line-height as margin (roughly 1.5 times the font size, 20px for a 14px font-size). In Bootstrap 4, they get 1rem which is the same as the default font-size, so in pixels that’s 16px for the default 16px font-size. Not sure why Bootstrap 4 decided to slightly reduce the separator whitespace here. 

All h1-h6 have a margin-bottom of $headings-margin-bottom, which defaults to half a $spacer. –default 1rem. (bootstrap 3 gave h1-h2 ‘double’ margin-bottom).

p uses $paragraph-margin-bottom, now in _reboot.scss. Which defaults to, you guessed it, 1rem.  (note that paragraph spacing in bootstrap 3 was ($line-height-computed / 2), half of a lot of other block element spacing. Now it’s 1rem, same as the rest).

grid-gutter-width is still in pixels, and still 30px, it is not responsive to font size.

list-groups look like the use padding rather than margin now, but it is defined in terms of rem .75rem in the vertical direction.

So a bunch of white-space separator values that used to be ‘size of line-height’ are now the (smaller) ‘size of font’ (and now expressed in rems).

If you wanted to make them bigger, the same relation to font/line-height they had in bootstrap 3, you might want to set them to 1rem * $line-height-base, or to actually respond properly to any resets to font-size-base, $font-size-base * $line-height-base. You’d have a whole bunch of variables to reset this way, as every component uses it’s own variable, which aren’t in terms of each other.

The only thing in Bootstrap 4 that still uses $font-size-base * $line-height-base (actual line height expressed in units, in this case rem units) seems to in custom_forms for custom checkbox/radio button styling. 

For your own stuff? $spacer and associated multiples

$spacer is probably a good variable to use where before you might have used $line-height-computed, for “standard vertical whitespace used most other places” — but beware it’s now equal to font-size-base, not (the larger) line-height-base.

There are additional spacing utilities, to let you get standard spaces of various sizes as margin or padding, whose values are by default defined as multiples of $spacer. I don’t believe these $spacer values are used internally to bootstrap though, even if the comments suggest they will be. Internally, bootstrap sometimes manually does things like $spacer / 2, ignoring your settings for $spacers.

If you need to do arithmetic with something expressed in rem (like $spacer), and a value expressed in pixels… you can let the browser do it with calc. calc($spacer - 15px)" actually delivered to the browser should work in any recent browser.

One more weird thing: Responsive font-sizes?

While off by default, Bootstrap gives you an option to enable “responsive font sizes”, which change themselves based on the viewport size. Not totally sure of the implications of this on whitespace defined in terms of font-size (will that end up responsive too?), it’s enough to make the head spin.

What happened to $grid-float-breakpoint in Bootstrap 4. And screen size breakpoint shift from 3 -> 4.

I have an app that customizes Bootstrap 3 stylesheets, by re-using Bootstrap variables and mixins.

My app used the Bootstrap 3 $grid-float-breakpoint and $grid-float-breakpoint-max variables in @media queries, to have ‘complex’ layout ‘collapse’ to something compact and small on a small screen.

This variable isn’t available in bootstrap 4 anymore.  This post is about Bootstrap 4.3.0, and probably applies to Bootstrap 4.0.0 final too. But googling to try to figure out changes between Bootstrap 3 and 4, I find a lot of things written for one of the Bootstrap 4 alphas, sometimes just calling it “Bootstrap 4” — and in some cases things changed pretty substantially between alphas and final. So it’s confusing, although I’m not sure if this is one of those cases. I don’t think people writing “what’s changed in Bootstrap 4” blogs about an alpha release were expecting as many changes as there were before final.

Quick answer

If in Bootstrap 3 you were doing:

// Bootstrap 3
@media(max-width: $grid-float-breakpoint-max) {
  // CSS rules
}

Then in Bootstrap 4, you want to use this mixin instead:

// Bootstrap 4
@include media-breakpoint-down(md) {
  // CSS rules
}

In in Bootstrap 3, you were doing:

// Bootstrap 3
@media (min-width: $grid-float-breakpoint) {
  // CSS rules
}  

Then in Bootstrap 4, you want to do:

@include media-breakpoint-up(lg) {
  // CSS rules
}

If you were doing anything else in Bootstrap 3 with media queries and $grid_float_breakpoint, like doing (min-width: $grid-float-breakpoint-max`) or (max-width: $grid-float-breakpoint), or doing any + 1 or - 1 yourself — you probably didn’t mean to be doing that, were doing the wrong thing, and meant to be doing one of these things. 

One of the advantage of the new mix-in style, is that it makes it a little bit more clear what you are doing, how to apply a style to “just when it’s collapsed” vs “just when it’s not collapsed”.

What’s going on

Bootstrap 3

In Bootstrap 3,   there is a variable `$grid-float-breakpoint`, documented in comments as “Point at which the navbar becomes uncollapsed.”  It is by default set to equal the Bootstrap 3 variable `$screen-sm-min` — so we have an uncollapsed navbar at “sm” screen size and above, and a collapsed navbar at smaller than ‘sm’ screen size.  screen-sm-min in Bootstrap 3 defaults to 768px. 

For convenience, there was also a $grid-float-breakpoint-max, documented as “Point at which the navbar begins collapsing” — which is a bit confusing to my programmer brain, it’s more accurate to say it’s the largest size at which the navbar is uncollapsed. (I would say it begins collapsing at $grid-float-breakpoint, one higher than $grid-float-breakpoint-max).

$grid-float-breakpoint-maxis defined as ($grid-float-breakpoint - 1) to make that so. So, yeah, $grid-float-breakpoint-max is confusingly one pixel less than $grid-float-breakpoint — kind of easy to get confused.

While documented as applying to the navbar, it was also used in default Bootstrap 3 styles in at least one other place, dropdown.scss,  where I don’t totally understand what it’s doing, but is somehow changing alignment to something suitable for ‘small screen’ at the same place navbars break — smaller than ‘screen-sm’.

If you wanted to change the point of ‘breakdown’ navbars, dropdowns, and anything else you may have re-used this variable for — you could just reset the $grid-float-breakpoint variable, it would now be unrelated to $screen-sm size. Or you could reset $screen-sm size. In either case, the change is now global to all navbars, dropdowns, etc.

Bootstrap 4

In Bootstrap 4, instead of just one breakpoint for navbar collapsing, hard-coded at the screen-sm boundary, you can choose to have your navbar break at any of bootstrap’s screen size boundaries, using classes ‘.navbar-expand-sm’, ‘.navbar-expand-lg’, etc. ‘navbar-expand-sm’. You can now choose different breakpoints for different navbars using the same stylesheet, so long as they correspond to one of the bootstrap defined breakpoints.

‘.navbar-expand-sm` means “be expanded at size ‘sm’ and above’, collapsed below that.”

If you don’t put any ‘.navbar-expand-*’ class on your navar — it will always be collapsed, always have the ‘hamburger’ button, no matter how small the screen size.

And instead of all dropdowns breaking at the same point as all navbars at ‘grid-float-break, there are similar differently-sized responsive classes for dropdowns.  (I still don’t entirely understand how dropdowns change at their breakpoint, have to experiment).

In support of bootstrap’s own code creating all these breakpoints for navbars and dropdowns, there is a new set of breakpoint utility mixins.  These also handily make explicit in their names “do you want this size and smaller” or “do you want this size and larger”, to try to avoid the easy “off by one” errors using Bootstrap 3 variables, where a variable name sometimes left it confusing whether it was the high-end of (eg) md or the low-end of md.

You can also use these utility mixins yourself of course!  breakpoint-min(md) will be the lowest value in pixels that is still “md” size. breakpoint-min(xs) will return sass null value (which often converts to an empty string), because “xs” goes all the way to 0.

breakpoint-max(md) will return a value with px units, that is the largest pixel value that’s within “md” size. breakpoint-max(xl) will return null/””, because “xl” has no max value, it goes all the way up to infinity.

Or you can use the mixins that generate the actual media queries you want, like media-breakpoint-up(sm) (size “sm” and up), or media-breakpoint-down(md) (size ‘md’ and down). Or even the handy media-breakpoint-between(sm, lg) (small to large, inclusive; does not include xs or xl.)

Some Bootstrap 4 components still have breakpoints hard-coded to a certain responsive size, rather than the flexible array of responsive breakpoint classes. For instance a card has a collapse breakpoint at the bottom of ‘sm’ size, and there’s no built-in way to choose a different collapse breakpoint.  Note how the Bootstrap source uses the media-breakpoint-up utility to style the ‘card’ collapse breakpoint.

Bootstrap 4 responsive sizes shift by one from Bootstrap 3!

To make things more confusing, ‘sm’ in bootstrap 3 is actually ‘md’ in bootstrap 4.

  • Added a new sm grid tier below 768px for more granular control. We now have xs, sm, md, lg, and xl. This also means every tier has been bumped up one level (so .col-md-6 in v3 is now .col-lg-6 in v4)

https://getbootstrap.com/docs/4.0/migration/

In Bootstrap 3, ‘sm’ began at 768px. In Bootstrap 4, it’s md that by default begins at 768px. And there’s a new ‘sm’ inserted below 768 — in Bootstrap 4 sm by default begins at 576px. 

So that’s why to get the equivalent of Bootstrap 3(max-width: $grid-float-breakpoint-max), where $grid-float-breakpoint was defined  based on “screen-sm-min” in Bootstrap 3 (smaller than ‘sm’) — in bootstrap 4 we need to use md instead — media-breakpoint-down(md).

Customizing breakpoints in Bootstrap 4

The responsive size breakpoints in bootstrap 4 are defined in a SASS ‘map’ variable called grid-breakpoints. You can change these breakpoints, taking some care to mutate the ‘map;’ without removing default values, if you that is your goal.

If you change them there, you will change all the relevant breakpoints, including the grid utility classes like col-lg-2, as well as the collapse points for responsive classes for navbars and dropdowns. If you change the sm breakpoint, you’ll change the collapse breakpoint for card for instance too.

There’s no way to only change the navbar/dropdown collapse breakpoint, as you could in Bootstrap 3 with $grid-float-breakpoint. On the other hand, you can at least hypothetically (I haven’t tried it or seen it documented) add additional breakpoints if you want, maybe you want something in between md and large, called, uh, I don’t know what you’d call it that wouldn’t be confusing. But in theory all the responsive utilities should work with it, the various built-in *-md-* etc classes should now be joined by classes for your new one (since the built-in ones are generated dynamically), etc. I don’t know if this is really a good idea.

Blacklight 7: current_user or other request context in SearchBuilder solr query builder

In Blacklight, the “SearchBuilder” is an object responsible for creating a Solr query. A template is generated into your app for customization, and you can write a kind of “plugin” to customize how the query is generated.

You might need some “request context” to do this. One common example is the current_user, for various kinds of axis control. For instance, to hide certain objects from returning in Solr query depending on user’s permissions, or perhaps to keep certain Solr fields from searched (in qf or pf params) unless a user is authorized to see/search them.

The way you can do this changed between Blacklight 6 and Blacklight 7. The way to do it in Blacklight 7.1 is relatively straightforward, but I’m not sure if it’s documented, so I’ll explain it here. (Anyone wanting to try to update the blacklight-access_controls or hydra-access-controls gems to work with Blacklight 7 will need to know this).

I was going to start by describing how this worked in Blacklight 6… but I realized I didn’t understand it, and got lost figuring it out. So we’ll skip that. But I believe that in BL 6, controllers interacted directly with a SearchBuilder. I can also say that the way a SearchBuilder got “context” like a current_user in BL6 and previous was a bit ad hoc and messy, without a clear API, and had evolved over time in a kind of “legacy” way.

Blacklight 7 introduces a new abstraction, the somewhat generically named “search service”, normally an instance of Blacklight::SearchService. (I don’t think this is mentioned in the BL 7 Release Notes, but is a somewhat significant architectural change that can break things trying to hook into BL).

Now, controllers don’t interact with the SearchBuilder, but with a “search service”, which itself instantiates and uses a SearchBuilder “under the hood”. In Blacklight 7.0, there was no good way to get “context” to the SearchBuilder, but 7.1.0.alpha has a feature that’s pretty easy to use.

In your CatalogController, define a search_service_context method which returns a hash of whatever context you need available:

class CatalogController < ApplicationController
  include Blacklight::Catalog

  def search_service_context
    { current_user: current_user }
  end

# ...
end

OK, now the Blacklight code will automatically add that to the "search service" context. But how does your SearchBuilder get it?

Turns out, in Blacklight 7, the somewhat confusingly named scope attribute in a SearchBuilder will hold the acting SearchService instance, so in a search builder or mix-in to a search_builder…

def some_search_builder_method
  if scope.context[:current_user]
    # we have a current_user!
  end
end

And that’s pretty much it.

I believe in BL 7, the scope attribute in a SearchBuilder will always be a “search service”, perhaps it would make sense to alias it as “search_service”. To avoid the somewhat ugly scope.context[:current_user], you could put a method in your SearchBuilder that covers that as current_user, but that would introduce some coupling between that method existing in SearchBuilder, and a SearchBuilder extension that needs to use it, so I didn’t go that route.

For a PR in our local app that supplies a very simple local SearchBuilder extension, puts it into use, and makes the current_user available in a context, see this PR. 

A terrible Github UI — accidentally shadow a tag with a branch

So we generally like to tag our releases in git, like v1.0.0 or what have you.

Github Web UI has a “tag/branch switcher” widget, which lets you look at a particular branch or tag in the Web UI.

Screenshot 2019-04-30 12.27.17

You can see it has separate tabs for “branches” and “tags”. Let’s say you get confused, and type “v1.0.0” (a tag) while the “branches” tab is selected (under the text box).

Screenshot 2019-04-30 12.30.55

It found no auto-complete for “v1.0.0” in “branches” (although there is a tag with that name it would have found if “tags” tab had been selected), and it “helpfully” offers to create a branch with that name.

Now, if you do that, you’re going to have a new branch, created off master, with the same name as a tag. Which is going to be really confusing. And not what you wanted.

Maybe your muscle memory makes your fingers hit “enter” and you wind up there — but at least it is very clearly identified, it says in fairly big and bold text “Create branch: v1.0.0 (from master)”, at least it warned you, although it’d be easy to miss in a hurry from muscle memory thinking you know what you’re doing.

That’s not the really evil UI yet.

Now let’s go to git’s “compare” UI. At https://github.com/someorg/someproject/compare

A fairly common thing I at least want to do is look at the compare between two releases, or from last release to master. But the ‘compare’ UI doesn’t have the tabs, it will only list or auto-complete from branches.

Screenshot 2019-04-30 12.34.55

In a hurry, going from muscle memory, you type in “v1.0.0” anyway.

Screenshot 2019-04-30 12.35.37

It does say “nothing to show”. But “v1.0.0” shows up in the list anyway. With a pretty obscure icon I’ve never seen before. Do you know what that icon means? It turns out, apparently, it means “Create branch: v1.0.0 (from master)”.

If confused, or in a hurry, or with your muscle memory outpacing your brain, you click on that line — that’s what happens.

Now you’ve got a branch called “v1.0.0”, created off current master, along with a tag “v1.0.0” pointing at a different SHA.  Because many UI’s treat branches and tags somewhat interchangeably, this is confusing. If you do a git checkout v1.0.0, are you going to get the branch or the tag?

It turns out if you go to a github compare UI, like `https://github.com/someorg/someproject/compare/v1.0.0..master`, Github is going to compare the new branch you accidentally made, not the existing tag (showing nothing in the diff, if master hasn’t changed yet). There is no no way to get Github to compare the tag. If you didn’t realize exactly what you did, you’re going to be awfully confused about what the heck is going on.

You’re going to need to figure it out, and delete the branch you just made, which it turns out you can do from the command line with the confusing and dangerous command: ` git push origin :refs/heads/v1.0.0`

And that’s how I lost a couple hours to figuring out “what the heck is going on here?”

What should you do if you want github ‘compare’ web UI for a tag rather than a branch? Turns out, as far as I know, you just need to manually enter the the URL https://github.com/org/project/compare/v1.0.0..v1.0.1 or what have you. The actual UI widgets will not get you there. They’ll just get you to a mess.

Am I missing something? That seems like github web UI is not only not providing for what I would think is a pretty common use (comparing tags), but leading you down a path to disaster when you look for it, no?

Solr Indexing in Kithe

So you may recall the kithe toolkit we are building in concert with our new digital collections app, which I introduced here.

I have completed some Solr Indexing support in kithe. It’s just about indexing, getting your data into Solr. It doesn’t assume Blacklight, but should work fine with Blacklight; there isn’t currently any support in kithe for what you do to provide UX for your Solr index.  You can look at the kithe guide documentation for the indexing features for a walk-through.

The kithe indexing support is based on ActiveRecord callbacks, in particular the after_commit callback. While callbacks get a bad rap, I think they are appropriate here, and note that both the popular sunspot gem (Solr/Rails integration, currently looking for new maintainers) and the popular searchkick gem (ElasticSearch/Rails integration) base their indexing synchronization on AR callbacks too. (There are various ways in kithe’s feature to turn off automatic callbacks temporarily or permanently in your code, like there are in those other two gems too). I spent some time looking at API’s, features, and implementation of the indexing-related functionality in sunspot, and searchkick, as well as other “prior art”, before/while developing kithe’s support.

The kithe indexing support is also based on traject for defining your mappings.

I am very happy with how it turned out, I think the implementation and public API both ended up pretty decent. (I am often reminded of the quote of uncertain attribution “I didn’t have time to write a short letter, so I wrote a long one instead” — it can take a lot of work to make nice concise code).

The kithe indexing support is independent of any other kithe features and doesn’t depend on them. I think it might be worth looking at for anyone writing a an app whose persistence is based on ActiveRecord. (If something ActiveModel-like but not ActiveRecord, it probably doesn’t have after_commit callbacks, but if it has after_save callbacks, we could make the kithe feature optionally use those instead; sunspot and searchkick can both do that).

Again, here’s the kithe documentation giving a tour of the indexing features. 

Note on traject

The part of the architecture I’m least happy with is traject, actually.

Traject was written for a different use case — command-line executed high-volume bulk/batch indexing from file serializations. And it was built for that basic domain and context at the time, with some YAGNI thoughts.

So why try to use it for a different case of event-based few-or-one object sync’ing, integrated into an app?  Well, hopefully it was not just because I already had traject and was the maintainer (‘when all you have is a hammer’), although that’s a risk. Partially because traject’s mapping DSL/API has proven to work well for many existing users. And it did at least lead me to a nice architecture where the indexing code is separate and fairly decoupled from the ActiveRecord model.

And the Traject SolrJsonWriter already had nice batching functionality (and thread-safety, although didn’t end up using it in current kithe architecture), which made it convenient to implement batching features in a de-coupled way (just send to a writer that’s batching, the other code doesn’t need to know about it, except for maybe flushing at the end).

And, well, maybe I just wanted to try it. And I think it worked out pretty well, although there are some oddities in there due to traject’s current basic architectural decisions. (Like, instantiating a Traject “Indexer” can be slow, so we use a global singleton in the kithe architecture, which is weird.)  I have some ideas for possible refactors of traject (some backwards compat some not) that would make it seem more polished for this kind of use case, but in the meantime, it really does work out fine.

Note on times to index, legacy sufia app vs our kithe-based app

Our collection, currently in a sufia app, is relatively small. We have about 7,000 Works (some of which are “child works”), 23,000 “FileSets” (which in kithe we call “Assets”), and 50 Collections.

In our existing Sufia-based app, it takes about 6 hours to reindex to Solr on an empty index.

  • Except actually, on an empty index it might take two re-index operations, because of the way sufia indexing is reliant on getting things out of the index to figure out the proper way to index a thing at hand. (We spent a lot of work trying to reorganize the indexing to not require an index to index, but I’m not sure if we succeeded, and may ironically have made performance issues with fedora worse with the new patterns?) So maybe 12 hours.
  • Except that 6 hours is just a guess from memory. I tried to do a bulk reindex-everything in our sufia app to reconfirm it — but we can’t actually currently do a bulk reindex at all, because it triggers an HTTP timeout from Fedora taking too long to respond to some API request.
    • If we upgraded to ActiveFedora 12, we could increase the timeout that ActiveFedora is willing to wait for a fedora response for. If we upgraded to ActiveFedora 12.1, it would include this PR, which I believe is intended to eliminate those super long fedora responses. I don’t think it would significantly change our end-to-end indexing time, the bulk of it is not in those initial very long fedora API calls. But I could be wrong. And not sure how realistic it is to upgrade our sufia app to AF 12 anyway.
    • To be fair, if we already had an existing index, but needed to reindex our actual works/collections/filesets because of a Solr config change, we had another routine which could do so in only ~25 minutes.

In our new app, we can run our complete reindexing routine in currently… 30 seconds. (That’s about 300 records/second throughput — only indexing Works and Collections. In past versions as I was building out the indexing I was getting up to 1000 records/second, but I haven’t taken time to investigate what changed, cause 30s is still just fine).

In our sufia app we are backing up our on-disk Solr indexes, because we didn’t want to risk the downtime it would take to rebuild (possibly including fighting with the code to get it to reindex).  In addition to just being more bytes to sling, this leads to ongoing developer time on such things as “did we back up the solr data files in a consistent state? Sync’d with our postgres backup?”, and “turns out we just noticed an error in the backup routine means the backup actually wasn’t happening.” (As anyone who deals with backups of any sort knows can be A Thing).

In the new system, we can just… not do that.  We know we can easily and quickly regenerate the Solr index whenever, from the data in postgres. (And if we upgrade to a new Solr version that requires an index rebuild, no need to figure out how to do so without downtime in a complicated way).

Why is the new system so much faster? I’ve identified three areas I believe are likely, but haven’t actually tried to do much profiling to determine which of these (if any?) are the predominant factors, so couldn’t say.

  1. Getting things out of fedora (at least under sufia’s usage patterns) is slow. Getting things out of postgres is fast.
  2. We are now only indexing what we need to support search.
    • The only things that show up in our search results are Works and Collections, so that’s all we’re indexing. (Sufia indexes not only FileSets too, but some ancillary objects such as one or two kinds of permission objects, and possibly a variety of other things I’m not familiar with. Sufia is trying to put pretty much everything that’s in fedora in Solr. For Reasons, mainly that it’s hard to query your things in Fedora with Fedora).
    • And we are only indexing the fields we actually need in Solr for those objects. Sufia tries to index a more or less round-trippable representation to Solr, with every property in it’s own stored solr field, etc. We aren’t doing that anymore. We could put all text in one “text” field, if we didn’t want to boost some higher than others. So we only index to as many fields as need different boosts, plus fields for facets, etc. Only what need to support the Solr functionality we want.
      • If you want to render your results from only Solr stored fields (as sufia/hyrax do, and blacklight kind of wants you to) you’d also need those stored fields, sufficiently independently addressable to render what you want (or perhaps just in one big serialized JSON?). We are hoping to not use solr stored fields for rendering at all, but even if we end up with Solr stored fields for rendering, it will be just enough that we need for rendering. (For instance, some people using Blacklight are using solr stored fields for the “index”/search results/hits page, but not for the individual record ‘show’ page).
  3. The indexing routines in new thing send updates to Solr in an efficient way, both batching record updates into fewer Solr HTTP update requests, and not sending synchronous Solr “hard commits” at all. (the bulk reindex, like the after_commit indexing, currently sends a softCommit per update request, although this could be configured differently).

So

Check out the kithe guide on indexing support! Maybe you want to use kithe, maybe you’re writing an ActiveRecord-based apps and want to consider kithe’s solr indexing support in isolation, or maybe you just want to look at it for API and implementation ideas in your own thing(s).

very rough benchmarking of Solr update batching performance characteristics

In figuring out how I want to integrate a synchronized Solr index into my Rails application, I am doing some very rough profiling/benchmarking of batching Solr adds vs not, just to get a general sense of it.

(This is all _very rough estimates_ and may depend a lot on your environment and Solr setup, including how many records you have in Solr, if Solr is being simultaneously used for queries, etc).

One thing some Solr (or ElasticSearch) integration packages sometimes end up concentrating on is batching multiple index-change-needed events into fewer Solr update requests.

Based on my observations, I think it’s not actually the separate HTTP requests that are expensive. (although I’m benchmarking with a solr on localhost).

But the commits are — if you are doing them. In my benchmarks reindexing a whole bunch of things, if I’m not doing any commits, whether I batch into fewer HTTP update requests to Solr or not has no appreciable effect on speed.

But sending a softCommit per record/update makes it around 2.5x slower.

Sending a (hard) commit per record makes it around 4x slower.

Even without explicit commit directives, if you have your solr setup to autocommit (soft or hard), it may of course occasionally pause to do some commits, so your measured time may depend on if you hit one of those.

So if you don’t care about realtime/near-realtime, you may not have to care about batching. I had already gotten the sense from Solr’s documentation that Solr will really like it better if the client never sends commits, but just lets Solr’s autoCommit/autoSoftCommit/commitWithin configuration to make sure updates become visible within a certain amount of maximum time. The reason to have the client send commits is generally because you need to guarantee that the updates will be visible to queries as soon as your code doing the update is finished.

The reason so many end up caring about batching updates might not because individual http requests to solr are a problem, but because too many _commits_ are. So if for some reason it was more convenient, only sending a commit per X records might be just as good as actually batching http requests — if you have to send commits from the client at all.

What “search engine” to use in a digital collections Rails app?

Traditional samvera apps have Blacklight, and it’s Solr index, very tightly integrated into many parts of persistence and discovery functionality, including management interfaces.

In rewriting our digital collections app , we have the opportunity to make other choices. Which of course is both a blessing and a curse, who wants choices?

One thing I know I don’t want is as tight and coupled an integration to Solr as a typical sufia or hyrax app.

We should be able to at least find persisted model items by id (or iterate through all of them), make some automated changes (say correcting a typo), and persist them to storage — without a Solr index existing at all. To the extent a Solr (or other text-search-engine) index exists at all, discrepencies between what’s in the index and what’s in our “real” store should not cause any usual mutation-and-persistence APIs to fail (either with an error or with wrong side effect outcome).

Really, I want a back-end interface that can do most if not all things a staff user needs to do in managing the repo, without any Solr index existing at all.  Just plain postgres ‘like’ search may sometimes be enough, when it’s not using pg’s full text indexing features likely are. These sorts of features are not as powerful as a ‘text search engine’ product like lucene or Solr — they aren’t going to do the complicated stemming that Solr does, or probably features like “phrase boosting”. They can give you filters/limits, but not facets in the Solr sense (telling you what terms are present in a given already restricted search, with term counts).

So we almost certainly still want Solr or a similar search engine providing user-facing front-end discovery, for this powerful search experience. We just want it sitting loosely on top or our app, not tightly integrated into every persistence and retrieval operation like it ends up in a typical sufia or hyrax app.

And part of this, for me, is I only want to have to index in Solr (or similar) what is neccesary for discovery/retrieval features, for search. This is how Solr works best. I don’t want to have to store a complete representation of the model instance in Solr, with every field addressable from a Solr result. So, that means, even in the implementation of the front-end UX search experience, i want display of the results to be from my ordinary ActiveRecord model objects (even on the ‘index’ search results page, and certainly on the ‘item detail’ page).  This is in fact how sunspot works — after solr returns a hit list, take the db pk’s (and model names) from the solr response, and then just fetch the results again from the database.  In a nice efficient SQL, using pre-loading (via SQL joins) etc. This is how one attempt at at elasticsearch-rails integration works too.

Yeah, it’s doing an “extra” fetch from the db, when it theoretically could have gotten everything it needed to display from Solr.  But properly optimized fetches from the db to display one page of search results are pretty quick, certainly faster than what was going on with ActiveFedora in our sufia app anyway, and the developer pain (and subsequent bugs) that can come from trying to duplicate everything in Solr just aren’t worth trying to optimize away the db fetch. There’s a reason popular Solr or ElasticSearch integrations with Rails do the “extra” fetch.

OK, I know what I want (and what I don’t), but what am I going to do? There’s still some choices, curses!

1. Blacklight, intervened in to return actual ActiveRecord models to views?

Blacklight was originally written for “library catalog” use cases where you might not be indexing from a local rdbms at all, you might be indexing from a third party data API, and you might not have representations in a local rdbms, solr is all you’ve got. So it was written to find everything it needs to display the results found in the Solr response.

But can we intervene in Blacklight to instead take the Solr responses, use them to get model names and pks to then fetch from a local rdbms instead? Like sunspot does?

This was my initial plan, and at first I thought I could easily. In fact, years ago, when writing a Blacklight catalog app, I had to do something in some ways analagous. We wanted our library catalog to show live checked in/out status for things returned by Solr. But we had no good way to get this into our Solr index in realtime. So, okay, we can’t provide a filter/facet by this value without it in the index, but can we still display it with realtime accuracy?

We could! We wanted to hook into the solr search results process in Blacklight, take the unique IDs from the solr response, and use them to make API calls to our ILS to figure out item status. (checked out, etc). And we wanted to do this in one bulk query (with all the IDs that are on the page of results), not one request per search hit, which would have been a performance problem. (We won’t talk about how our ILS didn’t actually have an API; let’s just say we gave it one).

So I had already done that, and thought the hook points and functions were pretty similar (just look up ‘extra info’ differently, this time the ‘extra info’ is an actual ActiveRecord model associated with each hit, instead of item status info). So I figured I could do it again!

The Blacklight method I had overridden to do that (back in maybe Blacklight 2.x days), was the search_results method called by Catalog#index action among other places. Overriding this got every place Blacklight got ‘results’, so we could make sure to hook in to get ‘extra stuff’ on every results fetching. it returned the @response itself, so we could hook into it to enhance the SolrDocument‘s returned to have extra stuff! Or hey, it’s a controller method, we can even have it set other iVars if we want. A nice clean intervention.

But alas! While I had used this in many-years-ago Blacklight, and it made it all the way to Blacklight 6… this method no longer exists in Blacklight 7, and there’s no great obvious place to override to do similar. It looks like it actually went away in a BL commit a couple years ago, but that commit didn’t make it into a BL release until 7.0. The BL 7 def index action method… doesn’t really have any clear point to intervene to do similar.

Maybe I could do something over in the ‘search_service.search_result’  method, I guess in a custom search_service. It’s not a controller method, so couldn’t provide it’s own iVar, but it could still modify the @response to enhance the SolrDocuments in it.  There are some more layers of architecture to deal with (and possibly break with future BL releases), and I haven’t yet really figured out what the search_service is and where it comes from! But it could be done.

Or I could try to get search_results cover method back in BL. (Not sure how ammenable BL would be to such a PR).

Or I could… override even more? The whole catalog index method? Don’t even use the blacklight Catalog controller at all but write my own? Both of those, my intuition based on experience with BL says, there be dragons.

2. Use Solr, but not Blacklight

So as I contemplated the danger of overriding big pieces of BL, I thought, ok, wait, why am I using BL at all, actually?

A couple senior developers at a couple institutions I talked to (I won’t reveal their names in case I’m accidentally misrepresenting them, and to not bring down the heat of bucking the consensus on them) said they were considering just writing ruby code to interact with Solr. (We’re talking on search/discovery, indexing — getting the data into Solr in the first place — is another topic). They said, gee, what we need in our UI for Solr results just isn’t that complicated, we think maybe it’s not actually that hard to just write the code to do it, maybe easier than fighting with BL, which in the words of one developer has a tendency to “infect” your application making everything more complicated when you try doing things the “Blacklight way”.

And it’s true we spend a lot of time overriding or configuring Blacklight to turn off features we didn’t think worked right, or we just don’t want in our UX. (Sufia/hyrax have traditionally tried to override Blacklight to make the ‘saved searches’ feature go away for instance). And there’s more features we just don’t use.

Could we just write our own code for issuing queries to BL, and displaying facets and hits from the results? Maybe. Off-hand, I can think of a couple things we get from BL that are non-trivial.

  1. The “back to search” link on the item detail page. Supplying a state-ful UX on top of state-less HTTP web app (while leaving the URLs clean) is a pain.  The Blacklight implementation has gone through various iterations and probably still has some weirdness, but my stakeholders have told me this feature is important.
  2. The date range facet with little mini-histogram, provided by blacklight_range_limit. This feature is also implemented kind of crazily (I should know, I wrote the first version, although I’m not currently really the maintainer) — if all you want is a date range limit where you enter a start and end year, without the little bar-graph-ish histogram, that’s actually easy, and I think some people are using blacklight_range_limit when that’s all they want, and could be doing it a lot simpler. But the histogram, with the nice calculated (for the particular result set!) human-friendly range endpoints (it’ll be 10s or 5s or whatever, at the right order of magnitude for your current facetted results!), kind of a pain, and it just works with blacklight_range_limit (although I don’t actually know if blacklight_range_limit works for BL7, it may not).

Probably a few more things I’m not thinking of that I’d run into.

On the plus side, wouldn’t have to fight with Blacklight to turn off the things I don’t want, or to get it to have the retrieval behavior I want retrieving hits my actual rdbms for display.

Hmm.

(While I keep looking at sunspot for ideas — it is/was somewhat popular, so must have at least gotten some developer APIs right for some use cases involving rdbms data searched in Solr — it’s got some awfully complicated implementation, is assuming certain “basic search” use cases as the golden path and definitely has some things I’d have to fight with, and has a “Looking for maintainers” line on it’s README, so I’m not really considering actually using it).

3. Should we use ElasticSearch?

Hypothetically, I think Blacklight was abstracted at one point to support ElasticSearch. I’m not totally sure how that went, if anyone is using BL with ES in production or whatever.

But if I wanted to use ElasticSearch, I think I probably wouldn’t try to use it with BL, but as an extension of the “2” part. If I’m going to be writing it myself anyway, might we want to use ElasticSearch instead?

ElasticSearch, like Solr, is an HTTP-api search engine app built on top of lucene. In some ways, I think Solr is a victim of being first. It’s got a lot more… legacy.  And different kinds of deployments it supports. (SolrCloud or not cloud? Managed schema or not? What?) Solr can sometimes seem to me like it gives you a billion ways to do whatever you want to do, but you’ve got to figure out which one to do (and whatever you choose may break some other feature). Whereas ElasticSearch just seems to be more straighfforrward. Or maybe that’s just that it seems to have better clearer documentation. It just seems less overwhelming, and I theoretically am familiar with Solr from years of use (but I always learned just enough to get by).

For whatever reasons, ElasticSearch seems to have possibly overtaken Solr in popularity, be, seems to be easier to pay someone else for a cloud-hosted PaaS instance at an affordable price, and seems to just generally be considered a bit easier to get started with.

I’ve talked to some other people in samvera space who are hypothetically considering ElasticSearch too, if they could do whatever they wanted (although I know of nobody actually moving forward with plans).

ElasticSearch at least historically didn’t have all the features and flexiblity of Solr, but it’s caught up a lot. Might it have everything we actually need for this digital collections app?

I’m not sure. Historically ES had problems with facets, and while it seems to have caught up a lot… they don’t work quite like Solr’s, and looking around to see if they do everything I need, it seems like there are a couple different ES features that approach Solr’s “facets”, and I’m not totally sure either does what I actually need (ordinary Solr facets: exact term counts, sorted by most-represented-term-first, within a result set).

It might! But really ES’s unfamiliarity is the biggest barrier. I’d have to figure out how to do things with slightly different featureset, and sometimes might find myself up against a brick wall, and am not sure I’d know that for sure until I’m in the middle of it. I have a pretty good sense of what Solr can do at this point, I know what I’m getting into.

(ES also maybe exposes different semantics around lucene ‘commits’? If you need synchronous “realtime” commits immediately visible on next query, I think maybe you can get that from ES, but I’m not 100% confident, it’s definitely not ES’s “happy path”. Historically samvera apps have believed they needed this; I’m not sure I do if I succesfully have search engine functionality resting more lightly on the app. But I’m not sure I don’t).

So what will I do?

I’m actually not sure, I’m a bit stumped.

I think going to ElasticSearch is probably too much for me right now, there’s too many balls in the air in rewriting this app to add in search engine software I’m not familiar with that may not have quite the same featureset.

But between using BL and doing it myself… I’m not sure, both offer some risks and possible rewards.

The fact that I can’t use the override-point in BL I was planning to, cause it’s gone from BL 7, annoys me and pushes me a bit more to consider a DIY approach. But I’m not sure if I’m going to regret that later. I might start out trying it out and seeing where it gets me… or I might just figure out how to hack in the rdbms-retrieval pattern I want into BL, even if it’s not pretty. I know want to write my display logic in terms of my ActiveRecord models, and with full access to ActiveRecord eager-loading to load any associated records I need (ala sunspot), instead of trying to jam it all into a Solr record in a denormalized fashion. Being able to get out of that by escaping from sufia/hyrax was one of the main attractions of doing so!