Bibliographic Wilderness

One year of the aggregator


It’s been a year since I launched, my sort of modern take on a “planet” style aggregator of ruby news and blog RSS/atom feeds.

Is there still a place for an RSS feed aggregator in a social media world? I think I like it, and find it a fun hobby/side project regardless. And I’m a librarian by training and trade, and just feel an inner urge to collect, aggregate, and distribute information, heh. But do other people find it useful? Not sure!  You can (you may or may not have known) follow on twitter instead, and it’s currently got 86 followers, that’s probably a good sign. I don’t currently track analytics on visits to the http page. It’s also possible to follow through it’s own aggregated RSS feed, which would be additionally hard to track.

Do you use it or like it? I’d love for you to let me know.

Thoughts on a year of developing/maintaining

I haven’t actually done too much maintenance, it kind of just keeps on chugging. Which is great.  I had originally planned to add a bunch of features, mainly including an online form to submit suggested feeds to include, and an online admin interface for me to approve and otherwise manage feeds. Never got to it, haven’t really needed it — it would take a lot of work over the no-login-no-admin-screen thing that’s there now, and adding feeds with a rake task has worked out fine. heroku run rake feeds:add[http://some/feed.rss], no problem.  So just keep feeling free to email me if you have a suggestion please. So far, I don’t get too many such suggestions, but I myself keep an eye on /r/reddit and add blogs when I see an interesting post from one of them there. I haven’t yet removed any feeds, but maybe I should; inactivity doesn’t matter too much, but feeds sometimes drift to no longer be so much about ruby.

If I was going to do anything at this point, it’d probably trying to abstract the code a bit so I can use it for other aggregators, with their own names and CSS etc.

It’s kind of fun to have a very simple Rails app for a change. I’m not regretting using Rails here, I know Rails, and it works fine here (no performance problems, I’m just caching everything aggressively with Rails fragment caching, I don’t even bother with a CDN. Unless I set up cloudflare and forgot? I forget. The site only has like 4 pages!). I can do things like my first upgrade of an app to Rails 5.1 in a very simple but real testbed. (It was surprisingly not quite as trivial as I thought even to upgrade this very simple app from rails 5.0 to 5.1. Of course, that ended up not being just Rails 5.1, but doing things like switching to heroku’s supported free-for-hobby-dyno SSL endpoint (the hacky way it was doing it before no longer worked with rails 5.1), and other minor deferred maintenance.  Took a couple hours probably.

It’s fun working with RSS/Atom feeds, I enjoy it. Remember that dream of a “Web 2.0” world that was all about open information sharing through APIs?  We didn’t really get that, we got walled garden social media instead. (More like gated plantations than walled gardens actually, a walled garden sounds kind of nice and peaceful).

But somehow we’ve still got RSS and Atom, and they are still in fairly widespread use. So I get to kind of pretend I’m still in that world. They are in fairly widespread use… but usually as a sort of forgotten unmaintained stepchild.  There are lacks of specification in the specifications that will never be filled in, and we get to deal with it. (Can a ‘title’ be HTML, or must it be plain text?  If it’s HTML, is there any way to know it is? Nope, not really). I run into all kinds of weirdness — can links in a feed be relative urls? If so, they are supposed to be… relative to what? You might think the feed url… but that’s not always how they go. I get to try to work around them all, which is kinda fun. Or sometimes ‘fun’.

I wish people would offer more tagged/subsection feeds, those seem pretty rare still. I wish medium would offer feeds that worked at all, they don’t really — medium has feeds for a person, but they include both posts and comments with no ways to distinguish, and are thus pretty useless for an aggregator. (I don’t want your out of context two-line comments in my aggregator).

I also get to do fun HTTP/REST kind of stuff — one of the reasons I chose to use Rails with a database as a backend, so I can keep state, is so I can actually do conditional GET requests of feeds and only fetch if a feed has changed. Around 66% of the feed URLs actually provide etags or last-modified so I can try. Then every once in a while I see a feed which reports “304 Not Modified” but it’s a lie, there is new content, the server is just broken. I usually just ignore em.

Keeping state also lets me refuse to let a site post-date it’s entries to keep em at the top of the list, and generally lets me keep the aggregated list in a consistent and non-changing order even if people change their dates on their posts. Oh, dealing with dates is another ‘fun’ thing, people deliver dates in all sorts of formats, with and without timezones, with and without times (just dates), I got to try to normalize them all somewhat to keep things in a somewhat expected and persistent newest-on-top order. (in which state is also helpful, because I can know when I last fetched a feed, and what entries are actually new since then, to help me guess a “real” timestamp for screwy or timestamp-missing entries).

Anyway, it’s both fun and “fun”.

Modest Sponsorship from Honeybadger is hosted on heroku, cause it’s easy, and even fun, and this is a side project. It’s costs are low (one hobby dyno, a free postgres that I might upgrade to the lowest tier paid one at some point). Costs are low, but there are costs.

Fortunately covered by a modest $20/month sponsorship from Honeybadger. I think it’s important to be open about exactly how much they are paying, so you can decide for yourself if it’s likely influencing’s editorial decisions or whatever, and just everything is transparent. I don’t think it is, I do include honeybadger’s Developer Blog in the aggregator, but I think I’d stop if it started looking spammy.

When they first offered the modest sponsorship, I had no experience with honeybadger. But since then I’ve been using it both for (which has very few approaching zero uncaught exceptions) and a day job project (which has plenty). I’ve liked using it, I definitely recommend checking it out.  Honeybadger definitely keeps developing, adding and refining features, if there’s any justice I think it’ll be as successful in the market as bugsnag.  I think I like it better than bugsnag, although it’s been a while since I used bugsnag now. I think honeybadger pricing tends to be better than bugsnag’s, although it depends on your needs and sizes. They also offer a free “micro” plan for projects that are non-commercial open source, although you gotta email them to ask for it. Check em out!