design for including Rails engine assets into pipeline manifest

Phew, how’s that for a jargon-filled title.

I have a Rails engine I help develop. It’s not suitable for new 3.1 “isolation”, it really does want to interact with the main app that’s hosting it, injecting behavior into it and such. Think kind of like ‘devise’ but more so. (Whether this is a good idea or not, it’s increasingly feasible in modern Rails. I happen to think it’s a good idea in the right circumstances).

One thing it needs to do is provide some JS and CSS assets to the hosting app. The Rails 3.1 asset pipeline provides some infrastructure that makes it easier to do this more cleanly than it was previously, but there are still some tricks to good design, at least if you have the requirements I have.

  • I want to use the asset pipeline, so ‘require’ statements adding my assets will be added to their local application.js or application.css manifests.
  • I want to provide a generator which actually adds these lines, and I want to make it as idiot proof as possible. The engine’s generator likely does a few things other than adding assets to the manifest too, keep in mind.
  • I want the generator to be ‘idempotent’ — run it as many times as you want, it won’t add the same line a million times. This is useful if you’ve forgotten if you’ve run the generator; it’s also useful because you can say the ‘upgrade’ path is just to update the gem, and then re-run the generator — if there’s anything new the generator wants to give to you, it’ll give it to you, and it’ll be okay with the things from the old version it previously generated.
  • I want power-users to be able to move around or disable what was generated into their manifest, but STILL be able to subsequently run the generator again without messing things up or un-doing what they’ve done.

I think there are a few tricks to making this work reliably. Here’s what I’ve figured out (just now, so haven’t field-tested it too much yet), curious for feedback, also curious if people like way of doing things. (Considering abstracting out into some helper Thor methods for some of these techniques, possibly putting them in a gem.)

1. Have ONE engine manifest that can be included in local manifest

The engine might have a bunch of, say, seperate JS files (same for CSS):


Don’t add separate ‘require’ lines for each of those to the local application.js manifest. We want one line added to local manifest that says “add whatever the engine’s got to give me” — to keep the local app manifest nice and clean, and to be ‘forwards compatible’ so it’ll keep working if later version of the engine add or remove assets.

You might at first think you can add to local application.js manifest:

//= require_tree './my_engine'
// or instead:
//= require_tree 'my_engine'

You can’t. the asset pipeline’s require_tree and require_directory  both require a directory relative to the actual current file’s location, not located at a different asset path (in the engine in this case). But it turns out you can do multi-level asset pipeline ‘require’:

In your engine:


The contents of app/assets/javascripts/my_engine.js is simply:

//= require_tree 'my_engine'

Which will then bring in my_engine/first_js.js, etc.

So the line we want to add to local manifest is simply: “require ‘my_engine'”, which will bring in my_engine/app/assets/javascripts/my_engine, which will then in turn bring in the ./my_engine subdirectories contents.  This is better than if we could do require_tree/require_directory in the first place, because it insulates the host app from those implementation details, and allows the gem to later change it’s mind and realize it’s got to include each asset specifically to get a desired order, etc.

Be careful where you have the generator add it, for robustness

We’re going to use the rails generator/thor method ‘insert_into_file‘, but insert after or before what?

For Javascripts, mine are almost always dependent on ‘jquery’, and I want to add it after the manifest line that brings in jquery.  But even if the rails generator generates “//= require jquery”, putting jquery in single or double quotes is also legal, let’s be a little bit more robust and use a regexp:

insert_into_file "app/assets/javascripts/application.js", :after => %r{//= require +['"]?jquery['"]?} do
  "\n//= require 'my_engine'\n\n"

For stylesheets, it’s not exactly clear where to put it, the easiest thing to do is put it at the end of the existing manifest, becuase my engine stylesheets often are meant to over-ride styles from other places. The end-user developer can always open up the manifest and move it around — we’ll make sure our generator is resilient to that later. The application.css manifest usually has all it’s ‘require’s in a big /* */ block, putting it before the first ‘*/’ in the file seems more or less safe enough:

 insert_into_file "app/assets/stylesheets/application.css", :before => "*/" do
   "\n *= require 'my_engine'\n\n"

But run your own check for existence before insert_into_file

Insert_into_file tries to be ‘idempotent’ to some extent already. If you run the exact same insert_into_file twice, it will only add one copy of your text. (Unless you ran the generator with the “–force” flag; turns out the “–force” flag kind of means different things to different actions, kind of dangerous). But this only works if the text to be inserted is still right before the “before” argument. (or after the ‘after’ argument).

But we want to let a power user move around the order of things in their manifest, but still have our generator be ‘idempotent’-ly runnable more than once, without adding more than one copy. So we’re going to want to check the file ourself to see if our ‘require my_engine’ line is already in it, no matter where it is.

Another thing we want to do though, what if the end-user developer actually decides they don’t want our engines assets at all? (Or wants to include various assets one by one picking and choosing, not include our engine sub-manifest as a whole).   We still want to let them re-run the generator (which may do things other than add assets to the manifest) without it going and adding it again each time requiring them to remove it again. If we make our check for existence liberal enough, it’ll let them ‘comment out’ the require line, in such a way it’s still there to prevent re-generation.

original_js = File.binread("app/assets/javascripts/application.js")
if original_js.include?("require 'my_engine'")
   say_status("skipped", "insert into app/assets/javascripts/application.js", :yellow)
   insert_into_file "app/assets/javascripts/application.js", :after => %r{//= require ['"]?jquery['"]?} do
      "\n//= require 'my_engine'\n\n"

Now the generator won’t re-add the line if it exists _anywhere_ in the file, even if it’s not still right after the ‘require jquery’ line.  Also, the generator won’t re-add the line if you’ve “commented it out” like this (kind of weird, since it already is a comment with a pre-preocessor directive, but we want to ruin the pre-processor syntax while leaving it in for the generator to notice) :

// If we wanted my_engine js, we'd enable:
// do not:  require 'my_engine'

And it even prints out a ‘skipped’ message in the generator execution, to just be transparent about what it’s doing.

Note that it’s the fact that we only have one line we add to the manifest that makes this more feasible — if we had more than one, and the end-user developer deleted just some of them or re-ordered them, it gets harder to manage. (Although probably do-able if you dry/extract out some stuff into a helper method, see below).

(Note, the default insert_into_file, if the :before or :after aren’t found in the file at all, will NOT raise OR print out an error, but will silently avoid inserting the text, while printing out the exact same status message as if it had. Something we’d probably want to fix for maximum robustness, but I haven’t yet.)

Extract into a helper or even gem?

So now we’ve turned a one-liner insert_into_file into sadly a several liner, with some boilerplate that has to be copy and pasted each time you do it. Ugh.

In theory, this seems like a nice thing to extract into it’s own more powerful thor/generator method. I haven’t done it yet, but I’d imagine something like:

safe_insert_into_file(filepath, :unless => "require 'my_engine'",
           :after => %r{//= require ['"]?jquery['"]?} do
   "\n//= require 'my_engine'\n\n"

Maybe add in the status warning if the before/after isn’t found too, instead of the default silent failure. Maybe some options for custom ‘skipped’ messages on unless or before/after not found.

I’m more likely to do this if anyone else is interested, but I’m not sure how many other people are writing these kind of heavy-duty engines where this kind of thing becomes useful.

One thought on “design for including Rails engine assets into pipeline manifest

  1. All this is a sad testament to the fact that engines are still not all that thought out for the more advanced cases. I like to split up a Rail app into multiple smaller full-stack engines, that can be developed and tested independently. The manifest structure is a strange beast, in that it uses special syntax in the comment sections, much like Java Doc, which ended up becoming a a meta language of its own, hopefully not the way we should end up in Rails…

    I plan to shortly develop a small gem with an asset generator template that can be slotted into any engine in order to manage these concerns for now, such as automatically copying assets to the dummy app for testing. Thanks for the tips!


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s