I see a couple interesting points here.

Harvard has a famously byzantine library system comprising over forty libraries, and administratively divided into two separate library systems (confusingly called the Harvard University Library or HUL, and the Harvard College Library, or HCL) has changed very little in terms of organizational structure since the late 19th century.

Harvard is not alone here. In fact, I’d suggest that the oldest academic libraries, and ironically especially the old ones that really excelled 80+ years ago, are most likely to have completely dysfunctional organizational structures and organizational behavior today.

Libraries today aren’t the same as libraries 80+ years ago, especially with regard to electronic content we purchase, which has different workflows to manage and different economies to purchase; and in terms of metadata maintenance as well, something which the blog author rightly points out libraries realized the benefits of cooperating/coordinating/sharing many years ago — but sharing cards (or data to print cards) through LC is a different beast than than modern metadata control needs.

I also generally agree with the blogger’s conclusion — but with less optimism:

But second, the importance of catalogers, and more broadly speaking, librarians is not necessarily diminishing into nothingness.  The environment has changed radically, and there are sure to be plenty of future “massacre-like” events that will painfully remind us of these changes.  But librarians do have a future, and I think it may even be a bright one: they just need to accept that it won’t be quite the same as the past.

I fully agree that there is still as much of a need for the tasks librarians have always done as ever — most definitely and even especially including cataloging/metadata control.

However, despite agreeing with that, I am actually not optimistic, like that blogger is.  We are running out of time to demonstrate that our profession, community, and industry is capable of meeting the metadata control needs of the 21st century.  We are not doing a good job of it. We do not seem to be capable of changing our priorities, expertise, organizational structures, and inter-organizational collaborative infrastructures, to deal with it.

The traditional goals of libraries have traditionally are still useful and needed just as much as ever, but with different ways of accomplishing them. There is still a great need for an organization specializing in information management on behalf of a user community, and without trying to make a profit off that user community.  But I am, sadly, no longer particularly optimistic that libraries as they are are actually capable of accomplishing those goals.  However, even in the best of cases, trying will result in some painful organization reorgs — nobody likes change. (It’s of course also possible for painful reorgs to end up entirely useless or even counter-productive, or simply admissions of defeat as libraries slowly die).

Hint: If you or your organization thinks if we can just put all our metadata into RDF as quickly as possible and therefore be “doing linked data”, that this is necessary and sufficient to handle modern metadata control needs — you have not only missed the boat, you are on the wrong boat.   I have lately been seeing a worrying increase of people suggesting “oh, we just need linked data to solve that problem”, with “linked data” meaning “the data we’ve already got expressed in RDF”, with a worrying ignorance/disregard for what good data actually entails in the 21st century systems environment.

I worry that there isn’t enough patron education on this issue. Patrons need to know that it’s publishers standing in the way, not library traditionalism or incompetence (well, there might be some of that too).

I was heartened to see a popular press article highlighting publisher resistance to library ebook lending, and the barriers publishers put in place. I hope this starts getting more coverage (and wish the ALA’s advocacy and popular education wings would work on it; what’s the ALA for, anyway?)

As demand for e-books soars, libraries struggle to stock their virtual shelves. By Christian Davenport, Published: January 14 . Washington Post.

And in a very related topic, an Amazon press release (blogged and analyzed here) suggests that making a title available through the Kindle lending program increases ebook sales for that title compared to if it were not available for lending, as well as resulting in royalties from Amazon’s library program.  Perhaps the publishers don’t need to be scared of library lending?

Of course, Amazon (as well as perhaps the publishers), would like to see an Amazon-controlled platform take control with no intermediation by pesky non-profit public libraries.  And with a new per-use royalty payment model. (which the US First Sale Doctrine makes unenforceable for print, but not ebooks).  Remember, not-for-profit libraries (public, academic, and sometimes special use)  are pretty much the only institutions in the publishing chain/universe, whose only interest is the benefit of their patrons/customers, rather than squeezing as much profit as possible out of their customers.

(In fact, there are some places which don’t contain useful Rails content I’d expect it to, it seems to mostly be a general text!  The chapter on HTML Injection oddly doesn’t mention Rails 3.x auto-escaping in ERB <%= %>, and #html_escape, etc. )

But anyhow, recommended if you want to brush up on your knowledge of categories of attacks and types of defenses for web apps.

It is quite easy for a user to get around this. For instance, with a simple javascript bookmarklet that deletes or modifies the cookies the site uses to track how many views you’ve had.

Being a programmer, I started thinking of ways the nytimes site could try defend against this. Right now it has no defenses at all. I thought of possible defenses that would at least require much more complicated javascript, and possibly ways that would require an actual browser plugin (say, to modify an spoof the Referer header) to defeat.

The nytimes doesn’t seem too interested in this game of cat and mouse, their defenses haven’t changed much since it was deployed. Perhaps because it’s enough that many/most users are ‘honest’, and because they realize that unless they change their usage policies to be much more locked down (perhaps requiring a login to view any articles at all, if not ending the lack of limits from external referrers), it will always be possible to defeat.

Mobile lockdown

But then I also thought of iOS.  I haven’t checked to see if the bookmarklet approach to delete cookies works on iOS. It is possible to install clickable js bookmarklets on iOS (iPhone or iPad) Mobile Safari, although it’s a pain, requiring either manual copy-paste/editing of JS code, or syncing with your desktop safari bookmarks.  It’s accessible to many fewer users than saving a bookmarklet on a desktop device.

It is not possible to spoof Referer headers on an iPad or an iPhone.

Unlike a desktop computer where you can install any software you want (including, say,  an open source browser with a feature to spoof referers, or a browser that takes plugins, and a plugin that does this), on iOS you can only install software approved by Apple.

[Unless you 'jailbreak' your device, which at least temporarily for the moment is not actually illegal (Apple would surely like it to be, does it surprise you that it quite likely would be without the special exemption from the Librarian of Congress?), but is something that typical users won't want to do, for various reasons.]

The App Store rules prohibit any alternate non-Safari/built-in-Webkit browsers.  Apple sometimes approves alternate browsers — only when they use the built in Webkit.  I am confident that part of the approval process for any app that involves a browser component is ensuring that it doesn’t let users do ‘untoward’ things like spoof referer headers or other parts of an http request.

The reasons for these restrictions are not just (or even mostly) about a consistent UI experience. They are about making sure websites that want to have DRM-like restrictions like the nytimes (or netflix, or hulu), have those restrictions be airtight on the iOS.  (These restrictions implemented by a website may or may not technically be ‘DRM’, under the DMCA etc. The actual implementation by the website probably is not, although it serves the same ends as DRM. The restrictions on the device itself or it’s built in software to try to keep you from an end-run around the website’s implementation probably would count as DRM, thus the LoC’s specific exemption for jailbreaking your phone).

The nytimes may or may not have yet implemented paywall protection that is impossible to get around on an iPhone.  But hulu already has. You can’t watch hulu on an iPhone for free, although you can on a desktop.  If the iPhone were a platform that gave users control, it would be easy to install a browser or browser plugin that hid from the hulu servers the fact that it was an iPhone.

But instead, the iPhone is a platform where users can only install software that Apple approves, and Apple’s policies and approval processes are in part designed to protect and enforce content provide restrictions.  Note that not all content owner technical restrictions simply enforce the law — DRM keeps users from doing things that would be legal, for fair use or other reasons, too. 

A future where most people have a mobile device as their main or only web browsing computer seems quite plausible.  If the iOS ‘closed-shop’ platform model becomes prevalent (as also seems quite plausible, as it’s been quite succesful — and I wouldn’t be shocked to see larger form factor non-mobile OSs adopt this model too, perhaps the Apple desktop app store is an exploratory shot) — This could be the end of the era where computer owners have the freedom to install whatever they want on their computers, and the beginning of an era where computer owners can only install what the platform vendors say they can install.  And their permission to install will be subject to their own business models and interests, and the business models and interests of their business partners.  This is not a welcome course.

(Note the paucity of open source software on the iOS app store — can anyone find me any examples?  I don’t think the app store rules actually prohibit open source, but the nature of the ecosystem discourages it or makes it less attractive to developers for several reasons. It’s pretty difficult to hack on a fork of an open source project for iOS, let alone distribute your mods to others.)

Umlaut is an “open source front-end for a link resolver”, or:

Umlaut is a just-in-time aggregator of “last mile” specific citation services, taking input as OpenURL, and providing an HTML UI as well as an api suite for embedding Umlaut services in other applications.

What the heck does this mean? Read more.

The 3.0 release of Umlaut will not add any new features, but instead modernizes Umlaut’s architecture to be based on Rails 3.1+ as an engine gem, and work on modern ruby versions. Lots of unsupported cruft was also removed from the codebase. (Umlaut actually began as a Rails 1.x application!).

Why this matters to you is that Umlaut should be easier to install and maintain than it ever was before. See Installation/Getting Started instructions. 

This is still an alpha release at present. It likely has some not yet discovered bugs, missing features, or performance issues. But it should be much easier to work with than Umlaut 2.x, if you are looking to get started with Umlaut, definitely start with the 3.x alpha.  Alpha tester feedback very welcome, please let me know of any difficulties you have with it, suggestions, questions, etc.

Umlaut 3.x source code is available in the umlaut3dev branch in the github project (eventually it will move to master).

A shift to ebooks has been predicted for a while, and seems to be happening. I’ve talked to many people who wonder why their public libraries don’t offer more ebooks they can download on their e-reader of choice, assuming it’s because the public libraries don’t want to, or are not technically competent to. The first is, i think, definitely no longer true — libraries want to. The second, technical competence, may be a barrier, but it’s not the prime one.

The prime barrier is that publishers by and large don’t want libraries to. I don’t think most library patrons realize the threat to libraries here — I think it’s high-time library organizations like the ALA start educating them. Most people like public libraries and want them to continue; a public that realizes they may be threatened will be more likely to support policy to make them do so, and that’s what’s needed.

With a print book, a library can buy a book and loan it to as many borrowers as they like, without any permission from the publisher at all. In the US, the right to do this is protected by the first sale doctrine. 

That does not apply to ebooks. I do not have the right to buy an ebook and loan it out again. I need the publisher’s permission — I may also need enabling technology to make it possible, vs Digital Rights Management that actively seeks to prevent it (and the DMCA which makes it illegal to violate that DRM technology).  A publisher can charge dearly for that permission, or withhold it entirely. With a print book, a publisher might still be worried that being able to borrow for free from a library can hurt sales — but they can’t do anything about it. With an ebook, they can do something about it, the balance of powers has shifted tectonically in favor of publishers.

Here’s a recent new york times article about this battle between publishers and libraries: Publishers vs. Libraries: An E-Book Tug of War

Here’s an account by Patrick Berry of California State University Chico on the difficulties in running a kindle lending program.   Patrick investigates the feasibility of loaning out actual physical kindles loaded with titles purchased by the library — this used to be somewhat feasible, but recently Amazon made technical changes that require a purchased title to be registered to a particular kindle device when purchased, and only readable on that device. Actually, to be fair, Amazon allows up to six devices to be registered for a title. But if a library wants to purchase more than six physical kindle devices, then a given title they purchase they can only load on six of them, unless they buy multiple copies. Note that this isn’t only six at once , it’s only six specific physical devices, pretty much ever.

What if you instead of loaning out actual physical kindles, a library wanted to loan out kindle titles for patrons to load on their own kindles?  Well, that same restriction makes it impossible for a library to simply buy the book normally at a normal rate and loan it out themselves directly to patrons (as libraries do with print books).  Now, you may have heard that Amazon recently announced a program with Overdrive to support library lending. The details of this program are vague and don’t seem available on the open internet. It seems likely to me that not every title available for kindle is in the Overdrive library program, probably only titles that publishers opted in to. (Compare to ordinary print books, where a library can buy any book at all and lend it out).  Likewise, it seems likely that libraries pay more over the lifetime of use for a kindle Overdrive title than they would for a print title. I don’t know what the library pricing model is here, and would be very very curious if anyone does — does a library pay an up front ‘purchase’ fee, is it more than the usual kindle purchase fee? Does the library pay a per-checkout fee as well?  (something a library does not do with a print book, and there’d be no way for a publisher to require it).

Libraries can offer ebooks, unlike print books, only at the sufferance of publishers, and publishers may charge whatever they like for this ‘privilege’. Publishers, not liking the idea of libraries much, are not providing that permission in some cases, and are providing pricing models in other cases which make it much more expensive for a library to offer an ebook than a print book.  If reader preferences continue to shift to ebooks as we expect, we may very well see the end of libraries as book lending institutions. (That’s of course not all a library does, and they may continue in other roles). Not because patrons don’t want to borrow ebooks from a library same as they did print ebooks, not because libraries don’t want to loan ebooks or can’t figure out the technology, but because publishers simply don’t want it to happen, and our laws give them the right to prevent it. 

• EBSCO EDS (Amanda Clay Powers, Mississippi State University)
• OCLC WorldCat (Zinthia C. Briceño-Rosales, Washington State University) [Haven't figured out if this is the WorldCat Local product, or simply the free-to-any-OCLC-member use of worldcat.org]
•  Ex Libris Primo (Rebecca Fernandez, Midwestern State University), and
• Serial Solutions Summon (Ken Varnum, University of Michigan).

Discovering What Works: Librarians Compare Discovery Interface Experiences

A very useful overview of what’s out there, why institutions are choosing these products, and what their experiences are.

One negative critique I have is that, like many such articles, these are kind of short on people sharing the negatives, what didn’t work as expected, what they wished worked better, what they could have done differently.  Surely any product and any implementation process has some negatives too. I think people are still too reluctant to admit that not everything went perfectly (nothing ever does), that there were unexpected c consequences of choices made by the library (there always are) or to criticize their vendors in public (library customer/vendor relationships are kind of sado-masochistic).  But making informed decisions requires knowing upsides and downsides, things that worked and things that didn’t, and we provide tremendous value to our community when we share the full picture. Still, just sharing the experiences as they are in this article is still a great step, we need more of this.

Thanks to local colleague Agnes Flannery-Denner  for alerting me to this article. 

update: (15:33) Ken Chad on the Code4Lib listserv draws our attention to  The Higher Education Library Technology (HELibTech) wiki ‘Discovery’ entry with some actual compare and contrast between products. Although it seems to me to mostly just be marketting brochure text for the commercial products. Still useful to have em all in one place, I guess.

And Peter Murray tells about the Unified Resource Discovery Comparison chart on Google Docs.

Neither of these resources look all that useful to me, honestly.

…A federal agency ruled on Monday that a set of important features commonly found in smartphones are protected by an Apple patent, a decision that could force changes in how Google’s Android phones function….

…For example, the case decided Monday involves the technology that lets you tap your finger once on the touch screen to call a phone number that is written inside an e-mail or text message. It also involves the technology that allows you to schedule a calendar appointment, again with a single tap of the finger, for a date mentioned in an e-mail.

http://www.nytimes.com/2011/12/20/technology/apple-wins-partial-victory-on-patent-claim-over-android-features.html?ref=technology

“Clicking” (tapping) on a phone number is a novel idea protected by an apple patent?

…Apple applied for one of the patents at issue in the HTC case — for detecting phone numbers and other forms of data — in 1996, 11 years before the iPhone was released….

I wonder if Skype is currently paying Apple royalties for use of essentially the same idea in their browser plugin, or if the patent is somehow unique to mobile, or to “tap” instead of “click”.

The routine granting of patents for techniques that would be obvious to any software engineer confronted with a particular context of technological ‘affordances’ is deeply harmful to software innovation, competition, and freedom.

• Someone starts the project in an publicly accessible repo.
• If she works for a company, in the best case she got permission with her employer (who may or may not own copyright to code she writes) to release it as open source.
• She sticks some open source License file in the repo saying “copying Carrie Coder” and/or the the name of the employer.

Okay, so far so good, but then:

• She adds someone else as a committer, who starts committing code. And/or accepts pull requests on github etc, committing code by other authors.
• Never even thinks about licensing/intellectual property issues.

What can go wrong?

•  Well, the license file probably still says ‘copyright Carrie Coder’ or ‘copyright Acme Inc’, even though the code by other authors has copyright held by them (or their employers). So right away something seems not all on the up and up.
• One of those contributors can later be like “Wait, I didn’t mean to release that open source, and I own the copyright, you don’t have my permission to use it, take it out.”
• Or worse, one of the contributors employers can assert they own the copyright and did not give permission for it to be released open source and you don’t have permission to use it (and neither does anyone else that’s copied or forked it from you).

Heavy weight solutions

So there’s a really heavy-weight solution to this, like Apache Foundation uses in their Contributor License Agreement.  This is something people have to actually print out and sign and mail in. Some agreements like this actually transfer the copyright to some corporate entity, presumably so the project can easily re-license under a different license later. (I thought Apache did this, but apparently not).

This is kind of too much over-head for a simple non-corporate-sponsored open source project. Who’s going to receive all this mail, and where are they going to keep the contracts? There is no corporate entity to be granted a non-exclusive license to do anything. (And the hypothetical project isn’t nearly so important or popular to justify trying to get umbrella stewardship from Apache or the Software Freedom Conservancy or whatever.(If it were, the Software Freedom Conservancy is a good option, but still too much overhead for the dozens of different tiny-to-medium sized projects anyone may be involved in. )

Even so far as individuals, over the life of the project who the committers are may very well change, and not include the original author(s) anymore.

And you don’t want to make someone print out sign and wait for you to receive something before accepting their commits, that’s not internet-speed.

Best practices for a simpler solution that’s not nothing?

So doing it ‘right’ with that heavy-weight solution is just way too much trouble, so most of us just keep ignoring it.

But is there some lighter-weight better-than-nothing probably-good-enough approach?  I am curious if anyone can provide examples, ideally lawyer-vetted examples, of doing this much simpler. 

Most of my projects are MIT-style licensed, which already says “do whatever the heck you want with this code”, so I don’t really care about being able to re-license under a different license later (I don’t think I do? Or maybe even the MIT license would already allow anyone to do that). So I definitely don’t need  and can’t really can’t handle paper print-outs.

I’m imagining something where each contributor/accepted-pull-request-submitter basically just puts a digital file in the repo, once,  that says something like “All the code I’ve contributed to this repo in past or future, I have the legal ability to release under license X, and I have done so.” And then I guess in the License file, instead of saying ‘copyright Original Author’, it would be like ‘copyright by various contributors, see files in ./contributors to see who.’

Does something along those lines end up working legally, or is it worthless, no better than just continuing to ignore the problem, so you might as well just continue to ignore the problem?  Or if it is potentially workable, does anyone have examples of projects using such a system, ideally with some evidence some lawyer has said it’s worthwhile, including a lawyer-vetted digital contributor agreement (that is itself licensed such that someone else can re-use it, ha)?

Any ideas?

For Rails 3.1 (3.1.3 is the latest release as I write this).

If you have JQuery-UI CSS generated by the theme roller, say for the ‘cupertino’ theme. One option is exploding the contents a bit, put the `jquery-ui-1.8.16.custom.css` file directly at ./app/assets/stylesheets/jquery-ui-1.8.16.custom.css, and put the corresponding jquery-ui theme ./images subdir directly at `./app/assets/stylesheets/images.`  (Or ./vendor/assets or ./lib/assets). Now just include the .css file in your application.css, “ *= require jquery-ui-1.8.16.custom.css “.

If instead you really do want to leave the JQuery-UI theme source in it’s own subdir by itself, say at `./app/assets/stylesheets/jquery-ui/cupertino` (or ./vendor/assets or ./lib/assets), then you’ve got to add this to your config/application.rb, matching the file path you’ve chosen to put it at:

initializer :after_append_asset_paths,
            :group => :all,
            :after => :append_assets_path do
   config.assets.paths.unshift Rails.root.join("app", "assets", "stylesheets", "jquery-ui", "cupertino").to_s
end

Your JQuery-UI theme images will now work, both in standard development config, standard production, as well as other combinations of asset piepline configuration.  Works well as far as I can tell in Rails 3.1.3, probably will continue working in the future. (can put in ./vendor/assets or ./lib/assets  instead of ./app/assets, same thing).

For more info and a couple other options, read on.

The problem

The jquery-rails gem, which is added to a new app’s Gemfile by default in rails 3.1, comes with JS for JQuery-UI.  You can include JQuery-UI JS just by adding this to your application.js:

//= require jquery-ui

Okay, but what about the JQuery-UI theme, which is CSS and some images referenced by that CSS?  The jquery-rails README kind of unhelpfully suggests:

In order to use the themed parts of jQuery UI, you will also need to supply your own theme CSS. See jqueryui.com for more information.

Okay, but if you do this the most straightforward way, you’ll find out it doesn’t work. I download a theme, say for the theme ‘cupertino’, and I get a zipfile `jquery-ui-1.8.16.custom.zip`.  Inside there, is a ./css directory.

Okay, I take this ./css directory and actually rename it into my app, say at: `./app/assets/stylesheets/jquery-ui`.  (Can put it in ./vendor/assets too, or even ./lib/assets, all will have the same outcome).

So now inside my local `./app/assets/stylesheets/jquery-ui`, I’ve got `./cupertino`, `./cupertino/jquery-ui-1.8.16.custom.css`, and `./cupertino/images`

So I go into my ./app/assets/application.css, and add:

*= require jquery-ui/cupertino/jquery-ui-1.8.16.custom.css

Start up my app, with some JQuery-UI code activated, and everything works fine in standard ‘development’ mode configuration. But in ‘production’ mode, with pre-compiled assets, none of the JQuery-UI theme images show up. (Which in some cases degrades usably but not as aesthetic, in other cases is a usability problem like lack of a ‘close’ button on a dialog).

What happened? Well, when compiled under sprockets, the contents of jquery-ui-1.8.16.custom.css is actually combined into the application.css file, which lives at the URL (for an app mounted at host root URL): `/assets/application.css`.   But that JQuery-UI CSS contains relative URLs to image assets, of the form `url(images/something.png)`

And relative to `/assets/application.css`, that means `/assets/images/something.png`.  But sprockets actually compiled the images to file path `./public/assets/jquery-ui/cupertino/images/something.png`, where it’ll be at url `/assets/jquery-ui/cupertino/images/something.png`  Not the right place for the CSS.

If I look around on the web for solutions to this, you get a lot of conflicting and confusing answers, perhaps because they were written for different versions of the asset pipeline (inc. some pre-release versions).  I’m not even going to link you to the various reddits, stackoverflows, etc, because it’s just too confusing and depressing. The most popular one seems to involve making symbolic links in your source. Come on really? Symbolic links, that tend to go wrong if you don’t do things perfectly when zipping up your source or copying it to another place or on Windows? I figured there’s got to be a better way.

In fact, I’m going to give you not just one alternate/better way, but four (count them, one, two, three, four) alternate/better ways.  Well, that’s kind of a lie, I’ll give you two better way that actually work with the asset pipeline, and two more that don’t use the pipeline for jquery-ui theme, but you can still use in your app that otherwise uses the pipeline.  For typical cases, any of em could work, but weird cases (say, wanting to have more than one theme available at runtime) each has trade-offs.

One way that works with the pipeline, take apart the subdir structure

Just take the files out of the jquery-ui generated directory stucture, and  put `jquery-ui-1.8.16.custom.css` directly  at ./app/assets/stylesheets/jquery-ui-1.8.16.custom.css, and put the corresponding jquery-ui theme ./images subdir _directly_ at `./app/assets/stylesheets/images.`

(Or put em in ./vendor/assets or ./lib/assets, will work identically in any of those. Maybe vendor is best, I dunno).

Now in your application.css, just

 *= require jquery-ui-1.8.16.custom.css

Now the relative paths in the CSS will resolve correctly both with and without pre-compiled assets, as well as even with `config.assets.debug = false` in dev.

Great, that’s pretty simple. I have to admit I didn’t think of this until I had spent quite a bit of time figuring out the next way, which will work with whatever subdirectory structure you want. Hey, maybe you really want to keep the subdir structure to keep your jquery-ui theme stuff nicely segregated. Or maybe you just want to learn a bit more about how Rails and Sprockets work. Then read, on.

Second way that works with the pipeline, whatever subdirs you want

So maybe you really still want to keep your jquery-ui theme all nicely grouped together in a subdir,  `./(app|vendor|lib)/assets/stylesheets/jquery-ui/themename`.  Just cause it’s tidier, or becuase you need multiple themes installed in your source (although that latter gets trickier).

First I thought, okay, well, maybe we can just add `./app/vendor/assets/stylesheets/themename`  to config.assets.paths in application.rb, now we’re telling sprockets to treat that as a ‘base’ dir, and compile the stuff (images in this case) in there directly to ./public/assets without the parent dirs, keeping the relative paths working.

Turns out I was on the right track, but that doesn’t work. Because  it turns out the order of paths matters to the Sprockets device that compiles assets. And Rails insists on adding ./(app|lib|vendor)/assets to the paths before anything you add yourself in config.assets.paths .  And Sprockets will then compile your stuff at that path, and when it sees a different subsequent path that also matches those same files, Sprockets is smart enough to know it already compiled that stuff from the previous path, and not compile it again.  So your addition to config.assets.paths of a subdir of one of the paths listed earlier essentially does nothing.

Turns out the place Rails insists on adding it’s own default paths first is in an append_assets_path initializer.    (Yes, despite the name “append_”, it actually prepends em with Array#unshift. Maybe it used to append but was changed to prepend to fix some other desired behavior?). Okay, great, the Rails 3.x initializer framework let’s us hook in right after Rails does this, to put what we want first in config.assets.paths instead.

Say we have our jquery-ui theme at `./vendor/assets/stylesheets/jquery-ui/cupertino`.  Add that path to the beginning of config.assets.paths, even before the stuff Rails prepended, by putting this in your config/application.rb:

initializer :after_append_asset_paths,
            :group => :all,
            :after => :append_assets_path do
   config.assets.paths.unshift Rails.root.join("vendor", "assets", "stylesheets", "jquery-ui", "cupertino").to_s
end

This works, to get the theme css’s relative URLs to images working,  in default (as installed by Rails) development mode asset config. It works in default production mode asset config, with precompiled assets.  It even works in development mode but with `config.assets.debug = false`. Each of these setups winds up with different ways of serving assets and potentially different path relationships between assets, but this works in every combo I tried. (Some of them probably work because config.assets.paths, depending on assets config, is sometimes used to create routing to assets too).

Will this keep working in the future? I dunno, I’m not technically using any internal API or anything, but I doing something not documented or popular, and counting on internal implementation not changing that much, the :append_assets_path initiailizer remaining there and named the same and used in the same way, etc. Guess it depends on how much Rails changes, like usual.

How the heck did I figure that out

I was not previously familiar with the inner workings of Sprockets or the Rails 3.1 asset pipeline. Instead, figuring that out took me about 5 hours of iteratively doing a bunch of this stuff:

• Looking at source code for both Rails and Sprockets in like 6 open tabs of github source, trying to understand how they are interacting where.
• Using github search function (wish it worked better), or grep’ing through checked out source to try and figure out where a particular method or module or class is defined. (Rails, why do you namespace classes in Rails source with ::Sprockets , fooling me into thinking I’d find em in Sprockets source? Couldn’t you have namespaced em Rails::Sprockets instead?)
• Using ruby-debug to investigate live source, and put breakpoints at Rails or Sprockets source lines to investigate em there. One reason to investigate is to figure out what the heck class a given variable is (Rails.application.assets anyone?). Another trick I like to do is drop into debug, call #freeze on an object, continue, and then wait for the stacktrace to figure out exactly who/where tried to modify that object next. (It is a travesty that ruby 1.9.3 does not work with ruby-debug, a travesty I say)
• Looking at the Rails config and asset pipeline guides to try and understand at least the big picture design.
• Googling for hints from other people, or sometimes just hints as to what source file a particular method was actually defined in when I was having trouble finding it.
• Wash, rinse, repeat, do these steps in a different order, go down blind alleys and realize that and backtrack in my investigation, etc.

Rails 3.x is really complicated code. Composed of multiple independent decoupled modules which interact with each other in complicated ways, occasionally even monkey-patching each other. (And Sprockets, would it kill you to add some one-two liner rdoc-style comments above your methods and classes, even non-public ones, telling the reader what they do?)

On the other hand, 4-6 hours ain’t that much time for getting to the bottom of fairly complicated functionality. And the solution I arrived at is fairly tidy, Rails 3.x is designed/factored pretty nicely these days to support surgical interventions (thanks Rails for just modifying config.assets.path once in an initiailizer instead of hard-coding it into the places Rails.application.assets is init’d and routes are init’d), and the 3.x initializer chain system is pretty sweet. (But Rails can’t decide whether to use strings or symbols for initializer names, it uses both, but you need to match the form used originally for the initializer :after or :before stuff. Grr.)

Third way, why host the theme yourself at all?

If you are taking a stock JQuery-UI theme and not customizing it at all, did you know that the Google ajax/open source tools CDN hosts all the stock JQuery-UI themes? (Although it’s not documented well, they’re there).  Why host it yourself at all, just link to the Google CDN. Sure it’ll be one more browser request since you haven’t combined the jquery-ui CSS into your application.css with the pipeline, but just one more request, to the Google CDN which is presumably relatively reliable and high performance, to save you the trouble of dealing with this crap or hosting it yourself at all?

Put in your layout:

<%= stylesheet_link_tag "http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.16/themes/cupertino/jquery-ui.css" %>

Replace ‘cupertino’ with whatever theme you want. Some legacy versions of themes also available, replace “1.8.16″ with “1.7.2″ for instance.

Or, do an import in your application.css instead:

@import url(http://ajax.googleapis.com/ajax/libs/jqueryui/1.8.16/themes/cupertino/jquery-ui.css)

It’ll still be an extra browser request, but it seems pleasant to keep all your CSS stuff, internal and external,  in your application.css, rather than splitting some of it out to the layout and then getting confused later.  CSS @import seems to be supported in all non-ancient browsers including IE6.

Fourth way, host it but skip the asset pipeline

Just cause you’re using the Rails 3.1 asset pipeline doesn’t mean you have to use it for all your assets. The Rails asset pipeline guide even says:

This is not to say that assets can (or should) no longer be placed in public; they still can be and will be served as static files by the application or web server. You would only use app/assets if you wish your files to undergo some pre-processing before they are served.

While we don’t really want any pre-processing of the source itself, if we just put the assets into public we do give up a couple things. We’d give up combining the CSS itself into assets.css to save a browser request (no big deal to save a headache in this special case, sez me. I ain’t running netflix here.) And we’d give up the cache-busting fingerprinting (again, seems tolerable to me — you aren’t going to be changing your jquery-ui theme very often, when you do, just make the changed theme a new theme in a new directory location in ./public, seems okay to me for most cases).

Now, the rails asset pipeline guide says you can do this, but it ends up being a pain, because when you have the asset pipeline turned on, you don’t have any helper methods anymore to generate URLs to assets in ./public, all the helper methods assume they’ll want an asset-pipeline-controlled asset. (Unless there’s a helper method I can’t find somewhere? Come on Rails, if you’re going to tell us it’s an option, can’t you support us doing it?)

You could I suppose just write hard-coded literal URLs — but that breaks if you are deploying an app at a sub-dir of your webserver, as Rails supports, and as I like to be able to do sometimes.

Okay, so create your own helper method, here you go, stick it in a helper somewhere:

# Much like usual rails route helpers,
# whether you realized it or not, you can pass
# option :only_path => false to get a
# full URL instead of a
# relative path.
 def public_static_path(path, options = nil)
   root_path(options) + path
 end

Now stick the jquery-UI theme at ./public/jquery-ui/cupertino, and in your layout:

<%= stylesheet_link_tag public_static_path("jquery-ui/cupertino/jquery-ui-1.8.16") %>

It’s not in the asset pipeline, but no problem.

What if you want to stick it in application.css with an “@import” instead? Make it into application.css.erb so we can use ERB to call out to a helper to generate the URL… Oops, again we run into the problem of having no way to generate the URL to the asset in public.  A helper method you add to your usual app helpers isn’t available in an erb asset. Yeah, there is a way to monkey-patch helpers into the asset context too, but I went down that path and it started getting more complicated there were a couple other tricks, so I just gave up. Oh well.

The end

The end. Any improvements, feedback, alternate methods, welcome. I wasn’t actually finding any of these suggestions on google myself (maybe i’m just a bad googler), so hopefully having em here now will make it easier for others trying to figure out what’s going on. Symbolic links, geez, you don’t have to do that.

The center’s own records — kept in a restricted section of its Web site, but found by The New York Times in an unsecured archived version of the site…

http://www.nytimes.com/2011/11/30/us/politics/gingrich-gave-push-to-clients-not-just-ideas.html

The internet version of going through the garbage for unshredded papers?  Somehow it reminds me of wikileaks too, so many journalistic information sources these days are effectively disintermediated information sources, not even people. (Not to discount the heroic intermediation of Bradley Manning).

1. The ruby open source community tries very hard to never re-write what someone else has written, instead composing gems and applications out of smaller pieces of de-coupled re-usable code. People try to share their code in composable decoupled pieces, and people try to write code using such pieces.
2. The ruby open source community strongly prefers innovation to stability, spending a lot more energy on improving things to be cooler, easier, faster, better — than on providing backwards compatibility or supporting old code.

I think both of these are actually consequences of a couple other very related things:

1. The ruby community is very volunteer-collaboration driven. Most code is not written by people being paid specifically to write that code.   Although lots of code is written by people who are using it for work they get paid for, relatively little code is written by people who have writing that code, as open source,  as an assigned task. (Compare to Java open source community).  Most ruby open source code is a labor of love (for the code, for the other developers you are sharing with).
2. The ruby community cares a lot about ‘developer joy’. Going back to matz’s original intention for ruby: “Ruby is designed to make programming not only easy but also fun”.  But this is also related to the point above — a mostly volunteer developer community only happens when developing is fun, and the more fun something is, the more likely there will be sufficient volunteer developer capacity to accomplish it.

Likewise, a mostly volunteer developer community kind of needs to develop based on loosely coupled small units developed by other people, to get anything done. We need to absolutely maximize our efficiency, because we don’t have armies of programmers getting paid to write what we need, we’ve got to take maximum advantage of what we’ve got.

But there’s a downside. Dependency hell.  When your product is built from dependencies on a bunch of things written by other people (and each of those has it’s own dependencies) — and especially when everything is constantly changing as people try to make things better, caring less about backwards compat and stability — it means things can be  constantly breaking.

After having spent a few years writing production ruby/rails code now, and still working at the same place where I wrote that code — I lately spend a huge amount of time upgrading my stuff to work with new versions of dependencies, or abandoning one now not-sufficiently-maintained dependency for another, etc.

Each individual upgrade may not be that hard, but they add up. It’s a huge portion of my time these days. And this is not a very joyful state, running on the treadmill just to stay in place instead of creating awesome new things. Ironically, optimizing for developer joy in the upstream dependencies (those developers don’t find it joyful to work on backwards-compat or refrain from releasing new ways of doing things because it’ll require people to change) — has made the joy of the downstream developer using all those dependencies very non-optimal.

Another interesting example is the way the recent Rails release had to downgrade sprockets to fix a regression. Rails too is built from a bunch of building blocks many developed by others (it always was to some extent, but Rails 3.0 pushed it even more in this direction). Which means those dependencies can easily break Rails too.  A minor-version-level release is supposed to be strictly backwards compatible.  But, well, making it reliably so is actually hard to do, it takes time and effort, that isn’t as fun as adding cool new stuff, and takes developer resources that an all-volunteer project simply may not have.

So Rails ends up having to release a 3.1.3 release of Rails that requires sprockets 2.0.3 instead of allowing 2.1.0, because neither the Rails developers nor the sprockets developer(s) have time to figure out what’s really going on. (And it’s not very joyful work to do so!).  But someone’s probably going to have to do that eventually, Rails can’t stay at an old version of sprockets forever, because there will be bugfixes and improvements that Rails and it’s users want — or because Rails users will want to use a gem that has a dependency on a more recent version of (eg) sprockets.

There’s no magic answer to this issue that I know of. No “if only people would do this” or “if only we had this.”  Before bundler, there was a tooling gap. But bundler greatly improves things, giving you the right control over your dependencies when you’ve got a complicated dependency tree. But it doesn’t really solve the problem.  And I would not prefer if the ruby community went to another extreme of focusing on stability and maintenance and backwards compatibility at the expense of awesome code. Ideally for me, I guess, more open source ruby developers would start moderating more toward the stability/maintenance direction, which I think has started to happen a bit, as more mature and legacy ruby code is out there and the ruby community gets bigger.

In my own projects though, I’ve realized that assembling small re-useable de-coupled components is not always more efficient — whether the small pieces are developed by me or someone else.  Yeah, you saved some initial time not re-inventing the wheel, but that time can easily be lost debugging component interactions when something goes wrong, patching dependencies, arguing with developers of dependencies on whether the change you need to support your use is really the right one for the dependency, etc.  Sometimes it really is more efficient to build and maintain your own more tightly coupled wheel.

Until now. Hpricot conflicts with Rails 3.1 — both of them monkey patch a #to_xs method into String, but they have conflicting semantics/arrity. (This goes to show this is probably a bad technique for a gem to use, and we could consider them both at fault, but as the biggest gorilla in the match, I don’t expect Rails to blink and change, esp when hpricot isn’t so popular anymore anyway).

This manifests as a `ArgumentError: wrong number of arguments (1 for 0)` error when you have hpricot gem loaded in Rails 3.1, and you try to use Rails (XML) ‘builder’ or call the #to_xml method on an ActiveRecord. (Hpricot redefined the #to_xs method differently than Rails expected it).

I wouldn’t necessarily call Hpricot ‘abandoned’, at the time of this writing it’s got a regular commit history , mostly nicksieger committing pull requests from outside developers.  However, there aren’t any comments from nicksieger or any other maintainers/committers on the ticket for this particular #to_xs Rails 3.1 conflict issue, filed two months ago.  It appears hpricot no longer has a developer community with the interest/capacity in dealing with this issue, anyhow.

From Hpricot to Nokogiri

So switching from Hpricot to Nokogiri is not too bad, Hpricot  is mostly api-compatible with nokogiri.

One difference that I ran into right away, and you might too, is how they handle xpaths from node objects.

 
<root>
 <parent>
   <child>
     <grandchild>
     </grandchild>
   </child>

   <child>
   </child>
  </parent>

  <parent>
  </parent>
</root>

hp_parent = Hpricot::XML(xml).at("//parent")
nk_parent = Nokogiri::XML(xml).at("//parent")

Now, if we do an xpath on the ruby object representing parent that begins with “/”, Hpricot interprets the xpath in a context where current node is considered root, but nokogiri still interprets the xpath in the context of the entire document, even though you’re calling it on a particular node.

3 : hp_parent.at('/child')
==> <child>

4 : nk_parent.at('/child')
==> nil

5 : hp_parent.at('/root/parent')
==> nil

6 : nk_parent.at('/root/parent')
==> <parent>

They’ll both behave the same, starting from current node, if you leave out the initial ‘/’.

7 : hp_parent.at('child')
==> <child>

8 : nk_parent.at('child')
==> <child>

You might think you could use “./child” too, xpath-style. You can in nokogiri, and it does the same thing. But Hpricot is broken with respect to “./” xpaths, no matter what comes after “./”, Hpricot will return the starting node.

11 : hp_parent.at("./child")
==> <parent>

12 : nk_parent.at("./child")
==> <child>

13 : hp_parent.at("./child/grandchild")
==> <parent>

14 : nk_parent.at("./child/grandchild")
==> <grandchild>

15 : hp_parent.at("./child/wrong")
==> <parent>

16 : nk_parent.at("./child/wrong")
==> nil

Here’s a little test script I used to demonstrate all this, which also demonstrates a cool hacky technique to write such demonstration scripts which output the code and result.

Special Hpricot xpath predicates

Continuing to find things that break when switching from hpricot to nokogiri (there aren’t _exactly_ api compatible), I’ll try to continue documenting em.

Hpricot supports some custom things in Xpath, that nokogiri does not. See “Supported Predicates, but differently” on https://github.com/hpricot/hpricot/wiki/Supported-XPath-Expressions .

For instance, i was using the “:eq(n)” one. For that one at least, nokogiri seems to supports the standard xpath predicate just fine: “[n+1]” (hpricot’s custom one is 0-based, actual xpath is 1-based indexing)

To be fair, designing a good API is hard (meaning ‘expensive’ in developer/designer time if nothing else), and while being able to say you have an API seems to have become neccesary for marketting your product, it’s unclear how many customers will actually use that API, and whether having an actual good API will get you more business. Of course, there’s a chicken and egg thing here too; the better designed and supported an API is, the more likely it will be used.  But, like with many aspects of software development, if you really want a good API, you’ve got to have experienced capable people developing it — with input from potential users, but you can’t expect potential users/customers to design it for you, or expect you’ll wind up with something good if you just do exactly what a couple customers ask for.

Anyhow, here’s a brief blog post with some principles of good API design.  Vendors serious about providing a good API would do well to think about these issues.

Interestingly, one of the examples in the article is from the library domain, I am curious what experience the blog author has working in this market — and I can guess what legacy back-end poor architecture leads to the particular issue he hints at, and makes it difficult to provide a good API.  From the provider’s point of view, it is definitely difficult to provide a good API on top of poorly architected software.

Mostly what you’ll find is digitized bound journals from libraries (that is, digitization of some volumes of the journal, probably not all of them, which may or may not have full text access).  Sometimes things that physically look like monographs but are published serially also get ISSNs, you might get some of those too, not sure. Has to be in GBS, and GBS has to have ISSN metadata for the record, not sure how often that happens.

http://www.google.com/search?hl=en&tbo=1&tbm=bks&q=issn%3A0008543X

with or without hyphen, both seem to work.

http://www.google.com/search?hl=en&tbo=1&tbm=bks&q=issn%3A0098-7484

http://www.google.com/search?hl=en&tbo=1&tbm=bks&q=issn%3A00987484

But don’t try to put phrase quotes around the value, that (oddly?) doesn’t work, with or without hyphen.

http://www.google.com/search?hl=en&tbo=1&tbm=bks&q=issn%3A”00987484″

I’ve recently been updating the application that uses threading with ActiveRecord to 3.0 or 3.1, so it’s time for an update.

The good news is that multi-threaded use of ActiveRecord is in many ways more supported and clearly documented; however, it’s still tricky writing for multi-threaded concurrency, and there are still some gotchas.  There isn’t a lot of general overview material on this on google, so I hope this post can be of value, I’ve tried to provide a general overview of various issues and gotchas.

When I say “old rails”  I mean Rails/Active Record 2.1, before the introduction of the ConnectionPool.  When I say “new” or “current” Rails, I mean Rails 3.0 or 3.1  In fact, the ConnectionPool was introduced in Rails 2.2, so the general principles may apply starting then, however there have been continual tweaks to ConnectionPool for bugs, robustness, and performance, so using Rails 3.0 or 3.1 is recommended.  Rails 3.2 does not yet exist at time of this writing, and may completely change the game yet again, if Aaron tenderlove goes forward with his ideas for a major refactor.

My use case

I have a Rails app. I do not want/need concurrent request handling. You can enable that with config.threadsafe!, which triggers config.allow_concurrency along with some other config values meant to make concurrency safer. But I’m not doing that, that’s not my interest, the usual Rails multi-process scaling model is fine for me.

What I do need though, is to kick off several threads which which will use ActiveRecord. I need to do this within a request-response loop, waiting on each of these threads with Thread#join before returning the response. I also want to kick off some threads that will happen in the ‘background’ — kick off threads in a controller action method, and not wait on them with Thread#join before rendering response. Those ‘background’ threads will store results in ActiveRecord, and my web pages poll with AJAX to get updated results from ‘background’.

These threads will be doing a bunch of HTTP calls to third-party servers, and also some ActiveRecord.  Each thread is unique code, each one is a different ‘plugin’ for contacting external services via HTTP apis and storing the results in the database. The reason why I want threads here, is so when a thread is waiting on I/O (database, or more likely, a slow HTTP api), it can be scheduled out and another thread can do it’s thing. The ruby Global Interpreter Lock isn’t, in theory,  a problem for my case, I’m fine with a process sticking to a single CPU core, I just need a thread waiting on external HTTP api (or database) to not lock up that CPU core while waiting.

I do need relatively low ‘latency’, I need all these ‘foreground’ async-ish tasks to start immediately and finish as fast as they can (sharing a CPU), and I need the ‘background’ tasks to also start pretty darn quick. (That is one thing that makes various ‘background job’ gems seem unsuitable. Another is that this is an app (soon to be Rails3 engine gem) that is used by third parties, and I’d really like to avoid requiring extra daemons to be started up other than typical rails app deployment).

So if your use case is at all like mine, this may be helpful to you. If your use case is very different than mine, a non-threaded solution may be better. If you want actual concurrent request handling, then that’s actually even a bit more complicated than what I’m describing here, although Rails theoretically supports it, I haven’t tried it.

My app is working. In fact, it was working even in Rails 2.1, but it works somewhat better in Rails 3. Here’s what I’ve discovered:

Use MRI 1.9.x, not MRI 1.8.x

MRI 1.8.x uses “green threads”, simulated threading entirely within the VM. While it sounds like an okay idea, experience (mine and others) show that ruby 1.8.x green threads behave poorly. Not only is thread-switching kind of expensive, but the threads do not share CPU very well, even with multiple threads waiting for CPU time, one thread could end up holding onto it for a couple seconds without switching out, in mysteriously unpredictable ways.

Ruby 1.9.x uses OS native threads, and the threads should behave more like you expect threads to.

Besides, ruby 1.9.x is the future, 1.8.7 will be end-of-lifed in the medium term, next major rails release will support only 1.9.x, just use 1.9.x.

Yes, even with MRI ruby 1.9.x, there is still the ruby “global interpreter lock”, which prevents a single ruby process from taking advantage of multiple CPU cores simultaneously, even with threads.  However, I don’t think this means “concurrency is a myth” — there are still many use cases that threading will work for. If a thread is waiting on I/O (an http request, a database call), despite the GIL, ruby should still be able to switch it out and another thread in, returning the first thread to the CPU when it’s I/O is ready. This is sufficient for my use case, and probably many use cases.  (Note “should”, see below on mysql vs mysql2).

jruby may be even better for threading, with possibly less overhead to thread creation and context switching and no GIL. For reasons not yet determined, i can’t get my app to run under jruby, but it’s definitely worth considering.

(The GIL does significantly lessen the attraction of using actual concurrent request handling in rails (config.threadsafe!, config.allow_concurrency), which is one of the reasons I don’t. The multi-process request-handling strategy typically used by rails applications is also quite sufficient for me, I have no problem with it. )

Use ‘mysql2′ adapter, never ‘mysql’

It turns out the original ‘mysql’ adapter would hold on to the ruby global interpreter lock (GIL) even when waiting for a database response. Even with the ruby GIL, in theory when one thread is waiting on a response from database, another thread ought to be switched in. However, the implementation of the ‘mysql’ adapter prevented this, making multi-threaded use of ActiveRecord a lot let useful with mysql.

Fortunately, the ‘mysql2′ adapter was written, fixing this problem. If planning on using concurrency with ActiveRecord and mysql, always use the ‘mysql2′ adapter, never use the ‘mysql’ adapter. ‘mysql2′ is now the recommended ‘standard’ in Rails, although it’s still called ‘mysql2′, and is a seperate adapter from ‘mysql2′.

If your database is not mysql, you might want to figure out if the ruby/rails database adapter is written in such a way to be concurrency friendly. It took a while for the community to catch on this was a problem with the mysql adapter code, could it still be a problem with other common adapters? I don’t really know. (It should not be a problem if you are using jruby with JDBC adapters).

Here is a little test script, cribbed from slide 4 in this presentation by Ilya Grigorik. It reveals that ‘mysql’ adapter indeed exhibits the behavior Grigork notes; however ‘mysql2′ adapter (based on code from Grigorik, I believe), does not — even using ‘mysql2′ adapter, however, we get a total elapsed real time of 2.1 seconds, which is quite a bit more than the 1-second-and-change one would expect.  Apparently there is still quite a bit of overhead here, perhaps from thread context switching, or ActiveRecord::ConnectionPool checkout mutex, or the implementation of ‘mysql2′ itself,  or something else I am not guessing. So multi-threaded ActiveRecord database access still may not perform as well as you’d like. But it’s good enough for my purposes, and way better than the pathological ‘mysql’ adapter.

sqlite3? Probably not. As sqlite3 doesn’t support SQL “sleep” function, this brain-dead script couldn’t be used to test it, you’d need an actual slow SQL query I haven’t had time to set up an environment for.  However, googling leaves me very confused about sqlite3′s stance toward concurrency even without taking the ruby/rails adapter into account. It may depend on what options sqlite3 was compiled with (no idea what options ‘gem install sqlite3′ uses), as well as exactly how the ruby/rails adapters are written. Trying to use multi-threaded ActiveRecord concurrency with sqlite3 is probably not a great idea.

Pay attention to the ConnectionPool contract

In all rails, threads can not share ActiveRecord Connection objects — or share the underlying network connection encapsulated by the Connection object. Each thread needs it’s own connection to the database.

In Rails 2.1, each thread would end up opening up a new network connection to the database (and you had to take some extreme measures to make sure that connection ended up getting closed when the thread was finished with it). That’s obviously not great for performance.  If you created lots of threads, you could also end up with a pretty huge number of open network connections to your database.

Current Rails introduces the ConnectionPool , which will pool open connections instead.  Even better, the ConnectionPool has a clearly documented ‘contract’ for use. Pay attention to it in the documentation.

In ordinary Rails code, ActionPack kind of takes care of this all behind the scenes for you. But if you’re manually creating threads, you’ve got to think about when/how connections get checked out and checked back in to the ConnectionPool. Seriously, go read the documentation and consider the three methods/strategies it provides.

For manually creating threads in a use case resembling mine, the with_connection method seems just right. Make sure all ActiveRecord database access is wrapped in a with_connection block; this will make sure a database connection gets checked out from the pool, then checked back in at end of with_connection block.

Note, the documentation is a bit confusing, but in fact you don’t really need to deal with the yielded connection directly, you can just do ordinary ActiveRecord actions inside the block:

ActiveRecord::Base.connection_pool.with_connection do
   m = SomeModel.find(something)
   m.foo = "bar"
   m.save!
   etc
   # and at end of block, connection will be checked
   # back into pool.
end

Great, not that hard to work with, this is a reasonable API.

Note also that, while the documentation doesn’t mention it, you can do nested  with_connection calls, and still only one connection will be used by the thread, without much extra overhead. This means that if you’re writing code in a context where you’re not sure if a parent scope already wrapped in with_connection or used some other means for checkout out a connection — you can still simply use #with_connection, and it’ll work fine either way.

Trick #1: Avoiding accidentally using first strategy

So you’ve chosen ConnectionPool strategy 3, using the with_connection method for all database activity in your thread.

But what happens if you accidentally execute some database-communicating ActiveRecord logic in a thread outside of a with_connection block? Maybe you just missed a place, but this is also very easy to do because of the way ActiveRecord lazily loads association content and does other kinds of database access “on demand.”

If you accidentally trigger some database access outside of a with_connection block, now you’re using method #1 instead. And a connection will be automatically checked out of the ConnectionPool for you — and it’ll never be checked in, unless you do something about it. You’ll quickly run out of connections.

It would be nice if you could tell ActiveRecord, perhaps on a per-thread basis, not to allow this kind of implicit connection checkout (strategy 1). Here’s a really hacky monkey patch to ActiveRecord to support that. With this monkey patch applied, call ActiveRecord::base.forbid_implicit_checkout_for_thread! from a thread, and if that thread later tries to access an active record connection without explicit checkout (#with_connection, or #checkout), an ImplicitConnectionForbiddenError will be raised.

I don’t trust this hack enough to use it in production  (I wrote it), but I use it in development/test just to try and find any accidental implicit checkouts. Apply monkey patch, call #forbid_implicit_checkout_for_thread! from each manually created thread, run your tests, you should get an ImplicitConnectionForbiddenError if you’re accidentally implicitly checking out a connection.

Trick #2: Clean up any accidental checkouts just in case

Just in case and out of paranoia, I periodically call:

ActiveRecord::Base.connection_pool.clear_stale_cached_connections!

This will look through the open database connection list, and clear any that were associated with Threads that are no longer alive. So if a thread accidentally does check out a connection without checking it in, this will clean it up.  clear_stale_cached_connections! is in theory a bit expensive, as it needs to go through the complete list of threads in the VM to see which threads are still alive.  However, it’s better than leaking connections, and I haven’t found it to be a problem, there are probably much bigger performance bottlenecks to multi-threaded access to ActiveRecord that make this one a non-issue.

Note that in Rails 2.1, there was advice to call a method #verify_active_connections! to do that same thing. Do not not  not ever call verify_active_connections! in current Rails.  verify_active_connections! is not thread-safe in current rails.  I can tell you from experience that if you do call it while you have multi-threaded ActiveRecord use going on, you will get weird errors, including (for the sake of people Googling, let’s list em):

• Mysql2::Error: MySQL server has gone away
• Mysql2::Error: Lost connection to MySQL server during query
• ActiveRecord::StatementInvalid (Mysql2::Error: Malformed packet
• threads and processes hanging in apparent deadlocks of some kind.
• Occasionally an actual segmentation fault.

I can’t think of any time you’d both want to call verify_active_connections! and it would actually be safe to call, I’m not sure why this method even exists. But perhaps there are use cases I am not thinking of; but they won’t be when any kind of multi-threaded ActiveRecord use is going on.

If you are seeing errors like this, but you aren’t using verify_active_connections!, then you’re probably doing something else that violates ActiveRecord::ConnectionPool’s contract in a way that two threads end up sharing the same connection (which again, AR is not designed to support).

If I understand things right, you do not want to call clear_active_connections! in an app that makes multi-threaded use of AR.

Note on ConnectionPool size

The default ConnectionPool size is 5. You can, however, set it to whatever you’d like with the “pool:” key in your connection dictionary in your database.yml.

You’ll almost certainly need a larger pool size than that, and may very well need a larger ConnectionPool than you expect. I did.

It seems like if we wrap all our ActiveRecord access in with_connection, we ought to be able to have a lot more active threads than we have connections in the pool.  Each thread will check out a connection just to do it’s work, and then check it back in.  If a thread needs to work and no connection is available, it’ll wait for one, and since each thread is just checking out a connection temporarily for a relatively quick database call, it should get a connection long before the 5 second timeout.   If you don’t have quite enough connections, things may slow down with threads spending time waiting for connections, but I wouldn’t expect a ConnectionTimeoutError, unless you have like a couple orders of magnitude more active threads than connections in the pool.

However, that wasn’t my experience. With 10 connections in pool and only 20 active threads using ActiveRecord via with_connection, I was still getting ConnectionTimeoutErrors. Why? I am not sure.  Even in ruby 1.9.x, is ruby still terrible at thread context switching, or are ruby mutex’s used by ConnectionPool#checkout still crazy expensive?   Even following ConnectionPool’s contract, was I somehow creating a deadlock, which ought not to be possible?  Or is my code accidentally not following the contract correctly somehow?  I really have no idea.  But using 30 or 40 connections in my connection pool seems to be sufficient for 100 or more threads, but 10 connections is not for 20. Although take these exact numbers with a grain of salt, all I can say is you may need more connections than you expect.

Note on errors raised in a thread

This is not really ActiveRecord specific, but about ruby threads in general.

If a thread raises an exception, that thread will stop executing, and the exception will be ‘lost’ — until/unless another thread waits on that aborted thread with Thread#join.  When Thread#join is called on a thread that raised an uncaught exception, the Thread#join directive will itself raise. The exception it raises will have the same class as the original exception, but it’s backtrace will be from the Thread#join call.

This is a bit annoying, you want that backtrace to debug the problem. With my hacky monkey patch for ImplicitConnectionForbiddenError  you really need the backtrace to fix the problem; and I usually want to see the backtrace for a ConnectionTimeoutError too.

To deal with that, I wrap all thread logic in a rescue, and use Thread-storage to store the exception, retrieving it in the ‘master’ thread, like so:

threads = []
1.upto.whatever do
  threads << Thread.new do
    begin
      # whatever stuff
    rescue Exception => e
      Thread.current[:exception] = e
    end
  end
end 

threads.each do |thread|
   thread.join
   if thread[:exception]
     # log it somehow, or even re-raise it if you
     # really want, it's got it's original backtrace.
   end
end

config.cache_classes

If you start threads that will keep going after a response is returned from a request-response loop, then you need to set “config.cache_classes = true”, even in development.   Rails development-mode class reloading will unload/reload all your reloadable classes at the end/beginning of every request-response loop.  But if you have code still working across request-response loop boundaries, that’s obviously a problem, a class will get unloaded while a thread is in the middle of trying to do stuff with instances of that class, bad.

So you need to give up on development-mode auto-reloading, and set `config.cache_classes = true` even in development.

If you are starting threads in an action, but waiting on them all with #join before returning the response, then it makes sense that you might be able to get away with `config.cache_classes = false` still, but I haven’t tried it.

None of the other concurrency-related config options that go along with config.threadsafe! seem to be required. Although I wouldn’t be surprised if there is an occasional edge case race condition involving autoloading with the default values for config.dependency_loading and config.preload_frameworks.  But changing these to be ‘thread safe’ means more significant changes to your app, so I’ve decided just to risk it. So far so good.

thread.priority

In MRI 1.8.7, I found that I needed to set Thread.priority on threads that were not the main Rails request-response thread to a priority less than default, or the actual request-response thread would not actually ever get scheduled for the CPU.  Theoretically, ruby 1.9.x (or jruby) with native threads should do a better job of thread scheduling, but I’ve left in my Thread.priority setters anyway, out of fear. :)

Consider NOT using multi-threaded access to ActiveRecord

So, you know, multi-threaded ActiveRecord use is a bit better and cleaner in current Rails than it was back in the Rails 2.1 days.

But it’s still kind of a pain; there are tricks and gotchas (not even counting the usual gotchas with any kind of concurrent programming, regardless of ActiveRecord), performance is somewhat unpredictable.

You probably want to consider other solutions than multi-threading.

The state of “async background job” processing options has gotten better than in the old days when BackgrounDRb was the best option. delayed_job seems to be a current popular choice, and is reportedly relatively easy to use and robust.  rabbitmq was recommended in comments on another thread-related blog post, but I’m not sure what the best easy to use ruby glue for it is.

For my use case, I’m not sure background task approach would work out so well — I need a large number of concurrent workers, such that either I’d need a huge number of worker processes, or I’d need my worker processes to deal with mutli-threading in ActiveRecord anyway. Plus I have an app (now an engine gem) that’s distributed to other end-users, and I’d like to avoid dependency on extra external daemons if possible. But it’s quite possible things would work much better and I could save myself trouble by using a background task solution instead of threads. If it’ll work for your use case, it’ll probably be a lot less headache than threading.

And the new hot thing is of course EventMachine. I haven’t quite been able to wrap my head around how to use EventMachine for concurrency without totally re-writing my fairly large legacy app, which I don’t really want to use. But em-syncrony looks like perhaps it can be used to incorporate some event-machine based async operations in a Rails app that is otherwise fairly standard, without having to go crazy with callback-style code. lthough I still can’t quite wrap my head around how I’d use it in my architecture (in which third-party plugins need to be runnable in a concurrent context too).  I suspect that an em-syncrony/fiber-based solution would perform a lot better than multi-threaded AR use. But it might be just as tricky to work with, you’ve still got to deal with the AR ConnectionPool, which em-syncrony activerecord actually patches to be fiber-aware (so one connection per fiber, instead of per-thread).  It possibly could end up even more confusing, what with the extra abstraction to deal with.  But if you’re interested in being cutting edge and can figure out how to make EM or em-syncrony work for you, definitely consider it.

The Future?

So Aaron tenderlove Patterson has expressed interest in a fairly major re-architecture of connection management in AR, possibly for Rails 3.2.

Current ConnectionPool is kind of weird, it’s got some weird hacks in it, it’s kind of tricky to understand how to use it in a multi-threaded environment (although so much better than pre-connectionpool days), it’s hard to use with fibers instead of threads (although em-syncrony seems to think it’s solved that, I definitely haven’t tried it, and am unclear what assumptions it may make about your use patterns), it’s performance seems unpredictable (if that’s the fault of my code, then we could say it makes it hard to understand how to write performant multi-threaded AR code).

So in theory I’m pleased about Aaron’s interest in re-working it. But I’ve spent so much time at this point figuring out how to get things to work barely-good-enough with the current architecture, if there’s a new architecture I really hope it’s not that different, or if it is, that it has enough real significant improvement to justify the cost of reworking my app again.

Acknowledge

Thanks very much to the author of coderr.wordpress.com, who is almost the only person one can find on the net apparently interested in multi-threaded ActiveRecord access, for his useful blog posts and surgically targetted rails patches. He has a couple blog posts, somewhat difficult to find on google actually, which were incredibly helpful to me, and some of the only things I can find explaining what’s going on and how to deal with it.

And of course, the ActiveRecord source. Thanks to api.rubyonrails.com for providing links directly to github for all documented methods. Without that, in the old days, it could be hard to even figure out what file a given method was implemented in; now with a click on a hyperlink, you’re looking at it (and can even use GitHub features to see how it’s changed over multiple Rails versions).  And thanks to GitHub for providing an html interface that makes source code a first class citizen on the web, so web pages can link to such a powerful and useable interface for looking at source code.   The ecology we’ve got based around github makes dealing with someone elses open source code so much easier than it used to be.

Does anyone know the pricing model to libraries?

An important thing to realize about how the rise of ebooks effects libraries, is the legal difference between ebook or print books for library lending.

A print book, a library (or in fact anyone else) who purchases a copy has the right to lend it to anyone they like (or to resell it to anyone they like), without any special permission from the copyright holder or vendor.   I know some publishers have a special higher “library price”; it’s never been entirely clear to me why libraries would pay this, pretty sure there’s no way to enforce it or prevent a library (or anyone else) from buying the ordinary retail copy at an ordinary retail price and lending.

That is generally not true of ebooks. Because of DRM; because of license agreements and the general model of ‘licensing’ software rather than selling it; and because the fact that transfering an ebook to a new device neccesarily involves ‘making a copy’ complicates the application of the ‘first sale doctrine’.  Note that Amazon only recently allowed libraries to lend ebooks under presumably a different pricing model than simply buying a single retail copy and lending it out to as many patrons as you want (one at a time).  For print books, no such “permission” is needed, you can simply do it.

Readers and library patrons generally unaware of threat

When I bring up this issue (not at my workplace, but among friends), many library patrons are unaware of it, and indeed somewhat hostile to even believing me. Perhaps assuming I’m simply being a luddite or hating on ebooks for aesthetic reasons, or some sort of judgement against those who prefer ebooks — this is not the case, I’ve got nothing against ebooks as a technology. 

But many patrons figure, after all, libraries have started to lend ebooks; patrons assume that the limited availability is simply the beginning of a technological adoption curve, and that libraries carry ebooks at all means that libraries will transition to ebooks along with their readers.

Patrons do not realize that libraries are probably paying a lot more per-use for ebooks than they did for print books (I would be really interested in some actual data on this if anyone has it, for a particular provider or averaged accross providers common among public or academic libraries), and that it’s not at all clear that this will change, and if it does not, it could make a library transition to ebook lending entirely unfeasible.  Because libraries aren’t telling them, perhaps out of a desire to prove continued relevancy, libraries are trying to make it look like they will have no problem transitioning to ebooks.  The cost of this, however, is suppressing awareness of the threat to the actual viability of lending libraries in an ebook world — and therefore suppressing the potential of a demand for legal or marketplace changes to ensure lending library viability in an eobok world.  It may time to sound the alarm, instead.

(I know that ALA expressed some concern over a particular publishers ebook lending policy; but I’m somewhat surprised ALA hasn’t been sounding a more general alarm about the serious threat to the right to lend at all, in an ebook world.  Complaining about one particular publisher’s ‘policy’ is to implicitly concede the fact that it’s up to the publisher’s discretion to make such ‘policies’ giving us permission to lend at all, which they did not in a print book world. This is such a shift in the socio-economics of reading that we ought not to be conceding.)

While I do love traditional printed and bound books, I will emphasize yet again that I do not  neccesarily find the prospect of a transition in reader preferences to ebooks threatening, considered purely in terms of technology/medium. Changes in the technology of reading and information sharing have happened before and will continue to happen, it’s still (a kind of) reading and (a kind of) book.

What is more frightening is the differences in intellectual property rights and coresponding differences in how the marketplace works —  not just for libraries, but in general looking at a world where books must be individually purchased by each reader and can’t be shared or loaned among friends.  This is of course what copyright holders and publishers want; what’s a library but a pre-digital form of ‘file sharing’, after all?  But it’s actually rather terrifying to this (e)book  lover.

I think it does apply to any library that does any in-house software development at all (even little tiny widgets and integration mash-ups): It is our job as professional writers of software to write good software — or to do the best we can (I, like everyone else, look at software I wrote three years ago and think, god, how could I have written such crap? I suspect this happens to even the best programmers; if it doesn’t happen to a programmer, it’s probably because they in fact aren’t very good and never improve!)

It’s our job to write good software — maintainable, well-architected, properly designed software — because it serves the interests of the organizations we work for. A library essentially is an IT organization these days, software is incredibly important to it — if you write terrible unmaintainable software, the organization will pay for it down the line, this ‘technical debt‘ will have to be paid. Either in huge amounts of development time to fix what you didn’t do right, or in having a really crappy software environment for our users that never gets fixed. I know we’re all familiar with both phenomenon.

(Yes, this also means that if a library is doing any in-house programming, and many many are these days, it is terribly short-sighted of them if they don’t hire professional programmers, or get training for the non-technical staff that ‘just fell into it’ — the cost is worth it, do it right or don’t do it).

Anyhow, here’s some of the ideas Mandavilli identifies as (in my interpretation) barriers to excellent software. I’ve highlighted the ones I know will be familiar to anyone that writes software in a library environment.

• “There isn’t enough time to do it right.”
• “Management has not prioritized it” In our context, this might be “the committee”, or “the reference librarians”, not management. Mandavilli says “what exactly hasn’t management prioritized — making a good product?”
  
• “No one is asking for it” Yes, we have to be user-centered, and in the library world we’re still trying to do that, but being user-centered isn’t just doing whatever people explicitly ask for and nothing else. Mandavilli says “This reminds me of Henry Ford’s wry comment “If I’d asked people what they wanted, they would have said ‘a faster horse’.”   Yes, this applies to reference librarians too.
• “We have to have consensus” Mandavilli says ”Someone has spent years studying, learning and working in a specific field, and to not defer to that person for the final decision is to waste all that expertise, not to mention deliver a bad product, demoralize the expert, adopt the safest and most timid way and most insidious of all,diffuse accountability…. A group decision is a way to duck responsibility for the outcome. ‘We all decided together’ is a way of saying ‘No one is responsible’”.  jrochkind says: Oh boy, sound familiar? I suppose it’s somewhat comforting to learn this disease is not unique to library organizations.

Read the original article, there are some more in there.

And if you write software in a library, commit to software excellence. Commit to fighting for software excellence against the other forces in our organization (and in our heads) that work against it, to convincing managers and committees that software excellence matters, and that to get it you have to give the software experts responsibility and authority to do it (as well as sufficient resources to support the desired outcome). No matter whether you learned programming in school or on the job, commit to continuously improving your craft by looking at others code, by reading, by taking some courses if needed, by peer-reviewing your peers code and getting your code peer reviewed (by a peer at another institution if you don’t have one at your organization) and by reflecting and evaluating on your own code.

Do it right or don’t do it.  If you can’t afford to do it right, you can’t afford to do it at all, doing it also crappy is something we can not afford. Now, it’s true that there is an opposing danger of over-engineering and micro-tweaking and never getting anything done.  Finding the balance is perhaps something that only comes from experienced developer intuition.  But I think libraries tend to err on the side of ‘crappy’, for various reasons.  If you’re a programmer at a library, it’s your job to develop that intuition and experience, not to settle for crappy.

At any rate, the plus operator is gone now.

I’m not entirely sure that the quotes will actually insist on the quoted word being present in the page?  Can anyone find a counter-example?

I had actually noticed a while ago that the google advanced search page had stopped providing any fields that resulted in “+”, and was suggesting double quotes for “exactly this form of word” (not variants), rather than “phrase”.   Exactly what given operators (and bare searches) do has continually evolved over time, and isn’t always documented or reflected in the “search tips” page or “advanced search” screen.

But let’s take a look in the Internet Archive to see if we can document the evolution.  Sadly, we can’t look at what a live google search at various historic points did. Nor can we look at what an ‘advanced search’ page actually did — the advanced search page has used different implementations at different times to convert your entry into a google search (un-mediated form submit to google search; javascript;  an intermediate redirection), but most of them aren’t amenable to interpretation from what’s in the Archive.

But we can look at the ‘search tips’ page, and the ‘advanced search’ screen itself when needed.  Let’s take a look at a snapshot from October (or as close as we can get) from each year from 2004 to now.

Google search tips over the years

In 2004, “+” is recommended to force inclusion of “common words” that would otherwise be excluded. But “another method for doing this is conducting a phrase search, which simply means putting quotation marks around 2 or more words. ”  Note, “2 or more words”.

In 2005, “+” is still recommended to force inclusion of ‘common’ words and phrase search is for the moment not mentioned at all in the narrative. But the screen shot of the multi-box advanced search page shows an ‘exact phrase’ box.

In 2006,  and 2007, largely the same.

In 2008, the entry box that used to say “exact phrase” now says “exact wording or phrase”, hinting that perhaps around here was the change where double quotes could be used on a single term to avoid alternate forms, as well as a multi-word phrase.  The “+” search guidance still talks about how “Google ignores common words and characters”, and the “+” can be used to search on em anyhow.

In 2009,  the search hints page has moved to a new URL, with new layout.   The “+” is now described differently: No more mention of Google normally ignoring common words (around then is perhaps when they started indexing even common words?), the “+” is now described as excluding variant forms:  ”sometimes Google helps out a little too much and gives you a synonym when you don’t really want it. By attaching a + immediately before a word….you are telling Google to match that word precisely as you typed it.”   The phrase search is again described narratively “the exact words in that exact order without any change”, hinting at both the phrase and “non-variant form” functions.  In 2009, we don’t have a screenshot of the advanced search page itself, so let’s check out the cached version:  yep, still got ‘exact word or phrase’ text box, which presumably results in double quotes.

2010, almost exactly the same, although the URL of the page is at help/refinesearch.html now. (Google’s 3xx redirects at the time the Archive crawled, and the Archive’s smarts in following them at crawl time and retrieval time, keeps us on track). And yeah, 2010 advanced search page still has ‘exact word or phrase’ box.

And it seems to stay that way until June 2011, the last crawl Archive seems to have at present.

Until present day, when the page is also much the same except no longer instructs on “+”, and has a search tip in upper right telling you to use double quotes instead.

So you wanna make a word mandatory, do you?

Interesting.  Many of us know that “+” used to result in “insist on this word or phrase being in the page”, since Google has (at most times, including up to present) sometimes showed you pages (even high up in your results) without including every term you entered. But the Google ‘search hints’ has actually never described it that way — it’s described it as either forcing search for common words otherwise not searched (no longer applicable, as Google currently indexes all words and even some punctuation), or as turning off synonyms (which a double-quoted single term has probably also done for some time).

It’s unclear to me if any functionality still exists to “insist”/require a word — I’m not sure I trust that double quotes do it.

(In other news, Wow, is the Internet Archive awesome, and their UX refresh a year or so ago so helpful for doing this kind of research). 

In the past, they’ve sent out mass email sometimes when this API changes to all registered users, but this time I only found out about it because, my application just started erroring. Like just now, within the past couple hours, I think. Even though it’s only October 18, not October 25.

So be warned!

Okay then, guess I better get an Affiliate ID.  This is a bit annoying, because I certainly don’t want to receive any money from an Affiliate ID — this use is at work at an academic library, I couldn’t take any money myself, and it’s really not worth the beaurocracy to figure out how to have this money go to library, when it probably won’t be very much money, and we don’t really want a profit motive to send things to Amazon. We just want to use the Amazon API to get details on books and Amazon availability.

But okay then!

I don’t think any of the other API changes will effect me, although I’m a bit worried about “Selected ItemAttributes that are not relevant to the broader product advertising use case will be deprecated”, not sure if that even applies to my use case, but if it does, and means that some data that I use will be going away (cover image? Citation metadata?), that would be inconvenient.