Non-digested asset names in Rails 4: Your Options

Rails 4 removes the ability to produce non-digest-named assets in addition to digest-named-assets. (ie ‘application.js’ in addition to ‘application-810e09b66b226e9982f63c48d8b7b366.css’).

There are a variety of ways to work around this by extending asset compilation. After researching and considering them all, I chose to use a custom Rake task that uses the sprockets manifest.json file. In this post, I’ll explain the situation and the options.

The Background

The Rails asset pipeline, powered by sprockets, compiles (sass, coffeescript, others), aggregates (combines multiple source files into one file for performance purposes), and post-processes (minimization, gzip’ing) your assets.

It produces assets to be delivered to the client that are fingerprinted with a digest hash based on the contents of the file — such as ‘application-810e09b66b226e9982f63c48d8b7b366.css’.  People (and configuration) often refer to this filename-fingerprinting as “digested assets”.

The benefit of this is that because the asset filenames are guaranteed to change if their content changes, the individual files can be cached indefinitely, which is great. (You still probably need to adjust your web server configuration to take advantage of this, which you may not be doing).

In Rails3, a ‘straight’ named copy of the assets (eg `application.js`) were also produced, alongside the fingerprinted digest-named assets.

Rails4 stopped doing this by default, and also took away any ability to do this even as a configurable option. While I can’t find the thread now, I recall seeing discussion that in Rails3, the production of non-digest-named assets was accomplished through actually asking sprockets to compile everything twice, which made asset compilation take roughly twice as long as it should.   Which is indeed a problem.

Rather than looking to fix Sprockets api to make it possible to compile the file once but simply write it twice, Rails devs decided there was no need for the straight-named files at all, and simply removed the feature.

Why would you need straight-named assets?

Extensive and combative discussion on this feature change occurred in sprockets-rails issue #49.

The title of this issue reveals one reason people wanted the non-digest-named assets: “breaks compatibility with bad gems”.   This mainly applies to gems that supply javascript, which may need to generate links to assets, and not be produced to look up the current digest-named URLs.  It’s really about javascript, not ‘gems’, it can apply to javascript you’ve included without gemifying it too.

The Rails devs expression opinion on this issue believed (at least initially) that these ‘bad gems’ should simply be fixed, accomodating them was the wrong thing to do, as it eliminates the ability to cache-forever assets they refer to.

I think they under-estimate the amount of work it can take to fix these ‘bad’ JS dependencies, which often are included through multi-level dependency trees (requiring getting patches accepted by multiple upstreams) — and also basically requires wrapping all JS assets in rubygems that apply sprockets/rails-specific patches on top, instead of, say, just using bower.

I think there’s a good argument for accommodating JS assets which the community has not yet had the time/resources to make respect the sprockets fingerprinting. Still, it is definitely preferable, and always at least theoretically possible, to make all your JS respect sprockets asset fingerprinting — and in most of my apps, I’ve done that.

But there’s other use cases: like mine!

I have an application that needs to offer a Javascript file at a particular stable URL, as part of it’s API — think JS “widgets”.

I want it to go through the asset pipeline, for source control, release management, aggregation, SASS, minimization, etc. The suggestion to just “put it in /public as a static asset” is no good at all. But I need the current version available at a persistent  URL.

Rails 3, this Just Worked, since the asset pipeline created a non-digested name. In Rails 4, we need a workaround.  I don’t need every asset to have a non-digest-named version, but I do need a whitelist of a few that are part of my public API.

I think this is a pretty legitimate use case, and not one that can be solved by ‘fixing bad gems’. I have no idea if Rails devs recognize it or not.

(It’s been suggested that HTML emails linking to CSS stylesheets (or JS?) is another use case. I haven’t done that and don’t understand it well enough to comment. Oh, and other people want em for their static 500 error pages.)

Possible Workaround Options

So that giant Github Issue thread? At first it looks like just one of those annoying ones with continual argument by uninformed people that will never die, and eventually @rafaelfranca locked it. But it’s also got a bunch of comments with people offering their solutions, and is the best aggregation of possible workarounds to consider — I’m glad it wasn’t locked sooner. Another example of how GitHub qualitatively improves open source development — finding this stuff on a listserv would have been a lot harder.

The Basic Rake Task

Early in the thread, Rails core team member @guilleiguaran suggested a Rake task, which simply looks in the file system for fingerprinted assets and copies them over to the un-digest-named version. Rails core team member @rafaelfranca later endorsed this approach too. 

The problem is it won’t work. I’ve got nothing against a rake task solution. It’s easy to wire things up so your new rake task automatically gets called every time after `rake assets:precompile’, no problem!

The problem is that a deployed Rails app may have multiple fingerprinted versions of a particular asset file around, representing multiple releases. And really you should set things up this way —  because right after you do a release, there may be cached copies of HTML (in browser caches, or proxying caches including a CDN) still around, still referencing the old version with the old digest fingerprint. You’ve got to keep it around for a while.

(How long? Depends on the cache headers on the HTML that might reference it. The fact that sprockets only supports keeping around a certain number of releases, and not releases made within a certain time window, is a different discussion. But, yeah, you need to keep around some old versions).

So it’s unpredictable which of the several versions you’ve got hanging around the rake task is going to copy to the non-digest-named version, there’s no guarantee it’ll be the latest one. (Maybe it depends on their lexographic sort?). That’s no good.

Enhance the core-team-suggested rake task?

Before I realized this problem, I had already spent some time trying to implement the basic rake task, add a whitelist parameter, etc. So I tried to keep going with it after realizing this problem.

I figured, okay, there are multiple versions of the asset around, but sprockets and rails have to know which one is the current one (to serve it to the current application), so I must be able to use sprockets ruby API in the rake task to figure it out and copy that one.

  • It was kind of challenging to figure out how to get sprockets to do this, but eventually it was sort of working.
  • Except i started to get worried that I might be triggering the double-compilation that Rails3 did, which I didn’t want to do, and got confused about even figuring out if I was doing it.
  • And I wasn’t really sure if I was using sprockets API meant to be public or internal. It didn’t seem to be clearly documented, and sprockets and sprockets-rails have been pretty churny, I thought I was taking a significant risk of it breaking in future sprockets/rails version(s) and needing continual maintenance.

Verdict: Nope, not so simple, even though it seems to be the rails-core-endorsed solution. 

Monkey-patch sprockets: non-stupid-digest-assets

Okay, so maybe we need to monkey-patch sprockets I figured.

@alexspeller provides a gem to monkey-patch Sprockets to support non-digested-asset creation, the unfortunately combatively named non-stupid-digest-assets.

If someone else has already figured it out and packaged it in a gem, great! Maybe they’ll even take on the maintenance burden of keeping it working with churny sprockets updates!

But non-stupid-digest-assets just takes the same kind logic from that basic rake task, another pass through all the assets post-compilation, but implements it with a sprockets monkeypatch instead of a rake task. It does add a white list.  I can’t quite figure out if it’s still subject to the same might-end-up-with-older-version-of-asset problem.

There’s really no benefit just to using a monkey patch instead of a rake task doing the same thing, and it has increased risk of breaking with new Rails releases. Some have already reported it not working with the Rails 4.2.betas — I haven’t investigated myself to see what’s up with that, and @alexspeller doesn’t seem to be in any hurry to either.

Verdict: Nope. non-stupid-digest-assets ain’t as smart as it thinks it is. 

Monkey-patch sprockets: The right way?

If you’re going to monkey-patch sprockets and take on forwards-compat risk, why not actually do it right, and make sprockets simply write the compiled file to two different file locations (and/or use symlinks) at the point of compilation?

@ryana  suggested such code. I’m not sure how tested it is, and I’d want to add the whitelist feature.

At this point, I was too scared of the forwards-compatibility-maintenance risks of monkey patching sprockets, and realized there was another solution I liked better…

Verdict: It’s the right way to do it, but carries some forwards-compat maintenance risk as an unsupported monkey patch

Use the Manifest, Luke, erm, Rake!

I had tried and given up on using the sprockets ruby api to determine ‘current digest-named asset’.  But as I was going back and reading through the Monster Issue looking for ideas again, I noticed @drojas suggested using the manifest.json file that sprockets creates, in a rake task.

Yep, this is where sprockets actually stores info on the current digest-named-assets. Forget the sprockets ruby api, we can just get it from there, and make sure we’re making a copy (or symlinking) the current digested version to the non-digested name.

But are we still using private api that may carry maintenance risk with future sprockets versions?  Hey, look, in a source code comment Sprockets tells us “The JSON is part of the public API and should be considered stable.” Sweet!

Now, even if sprockets devs  remember one of them once said this was public API (I hope this blog post helps), and even if sprockets is committed to semantic versioning, that still doesn’t mean it can never change. In fact, the way some of rubydom treats semver, it doesn’t even mean it can’t change soon and frequently; it just means they’ve got to update the sprockets major version number when it changes. Hey, at least that’d be a clue.

But note that changes can happen in between Rails major releases. Rails 4.1 uses sprockets-rails 2.x which uses sprockets 2.x. Rails 4.2 — no Rails major version number change — will use sprockets-rails 3.x which, oh, still uses sprockets 2.x, but clearly there’s no commitment on Rails not to change sprockets-rails/sprockets major versions without a Rails major version change.

Anyway, what can you do, you pays your money and you takes your chances. This solution seems pretty good to me.

Here’s my rake task, just a couple dozen lines of code, no problem.

 Verdict: Pretty decent option, best of our current choices

The Redirect

One more option is using a redirect to take requests for the non-digest-named asset, and redirect it to the current digest-named asset.

@Intrepidd suggests using rack middleware to do that.   I think it would also work to just use a Rails route redirect, with lambda. (I’m kind of allergic to middleware.) Same difference either way as far as what your app is doing.

I didn’t really notice this one until I had settled on The Manifest.  It requires two HTTP requests every time a client wants the asset at the persistent URL though. The first one will touch your app and needs short cache time, that will then redirect to the digest-named asset that will be served directly by the web server and can be cached forever. I’m not really sure if the performance implications are significant, probably depends on your use cases and request volume. @will-r suggests it won’t work well with CDN’s though. 

Verdict: Meh, maybe, I dunno, but it doesn’t feel right to introduce the extra latency

The Future

@rafaelfranca says Rails core has changed their mind and are going to deal with “this issue” “in some way”. Although I don’t think it made it into Rails 4.2 after all.

But what’s “this issue” exactly? I dunno, they are not sharing what they see as the legitimate use cases to handle, and requirements on legitimate ways to handle em.

I kinda suspect they might just be dealing with the “non-Rails JS that needs to know asset URLs” issue, and considering some complicated way to automatically make it use digest-named assets without having to repackage it for Rails.  Which might be a useful feature, although also a complicated enough one to have some bug risks (ah, the story of the asset pipeline).

And it’s not what I need, anyway, there are other uses cases than the “non-Rails JS” one that need non-digest-named assets.

I just need sprockets to produce parallel non-digested asset filenames for certain whitelisted assets. That really is the right way to handle it for my use case. Yes, it means you need to know the implications and how to use cache headers responsibly. If you don’t give me enough rope to hang myself, I don’t have enough rope to climb the rock face either. I thought Rails target audience was people who know what they’re doing?

It doesn’t seem like this would be a difficult feature for sprockets to implement (without double compilation!).  @ryana’s monkeypatch seems like pretty simple code that is most of the way there.  It’s the feature what I need.

I considered making a pull request to sprockets (the first step, then probably sprockets-rails, needs to support passing on the config settings).  But you know what, I don’t have the time or psychic energy to get in an argument about it in a PR; the Rails/sprockets devs seem opposed to this feature for some reason.  Heck, I just spent hours figuring out how to make my app work now, and writing it all up for you instead!

But, yeah, just add that feature to sprockets, pretty please.

So, if you’re reading this post in the future, maybe things will have changed, I dunno.

This entry was posted in General. Bookmark the permalink.

17 Responses to Non-digested asset names in Rails 4: Your Options

  1. Mat says:

    Thanks for this timely article. We’re looking to do the exact same thing. Rails app to host various JS libraries which need to have static links. Seems crazy it’s not just a config option to also generate the non-digest files. I’m going with your rake task, post-assets-precompile solution.

  2. jrochkind says:

    Thanks for the comment Matt, I always wonder if anyone’s seeing and finding useful my long posts!

  3. Martin says:

    My use case: I generate static html pages from a Rails app. I store those HTMLs in S3 and serve them through Cloudfront. I need a non digest version, since I need the already generated static HTMLs to keep working after new releases. This is one option. Another option was to make sprockets not to remove old digested assets (it says it keeps the last 3).

  4. roninonrails says:

    Thanks so much for this. It helped me understand why my upgraded webservers work fine, but a brand new one doesn’t serve assets properly! I’ll be tweaking your rake task to create undigested copies of assets to try and get back to a working system.

  5. Rohit Joshi says:

    Thanks for this write up. You saved my day. I had similar issue, I needed some images at a particular stable URL, and I had my app running in subdirectory, on heroku. I slightly modified your rake task and worked like a charm. Thanks.

  6. Just used this for a Shopify app where I have a little snippet of JS that you include in your product template to add some functionality. I’d much rather have people include “example.com/assets/notify.js” than “example.com/assets/notify-829ed0770bc33d37e9989cd2a1152387.js” Thanks much!

  7. jmwind says:

    This saved my ass for the same reasons as @canninkin mentioned, a Shopify app. I can understand the desire to make versioning the default, but it’s not practical in all cases.

  8. laziman says:

    For the use-case of a JavaScript file at a static URL as part of an API, I think I would be inclined to create a Rails route that does a 302 (or possibly 303) redirect to the fingerprinted asset. Most browsers I have used will follow this redirect just fine. The redirect itself would not be cached (at least not aggressively). But the fingerprinted URL redirected to would be cached very aggressively.

  9. jrochkind says:

    I just don’t like the extra non-cached request that will be made on every asset request — will it actually make a difference? I’m not sure. On mobile it could, that extra round-trip. Probably some testing of some kind would be called for, although hard to know how to test for various mobile experiences.

  10. Yo L says:

    About the semver version break in minor rails versions: The change Rails 4.0 to Rails 4.1 is a semver major version change,

    “Minor Y: New features, may contain API changes (Serve as major versions of Semver). Breaking changes are paired with deprecation notices in the previous minor or major release.” (http://guides.rubyonrails.org/maintenance_policy.html)

  11. jrochkind says:

    Yeah, thanks for pointing that out Yo L.

    While Rails reserves the right to release breaking changes in minor versions, they seem much less likely to do so than in major versions. There were more breaking changes, and more substantial (rather than trivial to handle with a search-and-replace) from 3 to 4 than there have been from 4.0 to 4.1 or 4.1 to 4.2. Although Rails team has been a bit more careful with breaking changes in general since the great 2.x to 3.x disruption, thankfully. But it’s true you are not guaranteed backwards compatibility in a Rails minor release.

    While I think semver is a major step forward, it’s contribution is mainly to make backwards compatibility _predictable_ from the version number, so you can be confident that certain upgrades will involve absolutely zero (intended!) backwards incompat. Just from the version numbers, without reading the docs. Which means ‘upgrade only to guaranteed backwards compat’ can be more easily automated (bundler!), even across a large transitive dependendency tree.

    While this is important, the _velocity_ of backwards breaking changes is still pretty important for actual developer pain, happiness, efficiency. In some ways semver, I think, means to discourage frequent backwards incompat releases by requiring a major version update, thinking people wouldn’t want so frequent major version increments. Of course, you can just accomodate yourself to rapidly increasing major version increments, but the developer pain of frequent backwards incompat will still be there. Or you can do what Rails has done, and decide you want frequent backwards breaking changes, but still don’t want to increment the major version, so you’ll just peg the minor version to what semver should be major instead. Okay then. (But again, I am grateful that Rails backwards incompat changes, especially major hard to deal with ones, have slowed down regardless).

  12. Yo L says:

    And FULL ACK to your conclusion.
    This whole https://github.com/rails/sprockets-rails/issues/49 is such a mess. So hard to now see what is actually the way to enable use of non digest css/js/logo assets for a static 500/404 page.
    And to have css js on a static 404/500 isn’t that a pretty common use case that almost affects every app? Also the trade off that the 500 page might be served with outdated assets: Not that big of a deal after all. The developers should be able to decide if they accept the outdated cached file trade-off or not.

    Thanks a lot for your post which wraps it up pretty nicely.

  13. S H says:

    Used a modified version of your rake task on Heroku -worked great! One thing I had to add was copying the gzipped version of the assets (if they exist). Otherwise, I was creating application.css, but not application.css.gz, and the server was being a good boy and serving the gzipped version (which was stale!).

    Essentially, the addition looks like this:

    gz_digested_path = “#{full_digested_path}.gz”
    gz_nondigested_path = “#{full_nondigested_path}.gz”
    FileUtils.copy_file(gz_digested_path, gz_nondigested_path, true) if File.exist?(gz_digested_path)

  14. jrochkind says:

    Thanks S H. I _think_ at the time I wrote that, the rails asset pipeline wasn’t creating .gz’s at all, at least not by default. So hard to keep track of what the heck the rails asset pipeline is doing.

  15. gamov says:

    Oh boy, thank you for this long explanation. What a clusterfuck this asset pipeline is! I’ve wasted so much time migrating to it on 3.2 and now, I’m doing it all over again for Rails 4!!!
    I just REFUSE to edit 3rd party JS code AND to place it in my app/assets, it’s plain wrong!
    I learn the .enhance for rails task from you, excellent!

  16. Jerome says:

    We wanted to use asset pipeline to transform some of our error pages and this blog helped immensely. BIG thankyou

  17. Thanks for the task enhancement. Got it working in 4.2.7, the only difference is that the manifest path has changed to ‘public/assets/.sprockets-manifest-*.json’

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s