Code4Lib Journal and “Peer Review”

People periodically ask if the Code4Lib Journal is “peer reviewed”. I believe that 100% of these queries come from people who have to publish in “peer reviewed” journals for tenure processes, not from people wanting that information to judge the quality or reliability of what we publish.

These academic tenure processes that care about ‘peer reviewed’ often aren’t using a plain language or  general meaning of ‘peer reviewed‘, they’re particularly wanting “(double) blind refereed“.

I will argue that Code4Lib Journal is peer-reviewed, intentionally — but it is not and shouldn’t become  blind-referreed.

Plain language: Code4Lib Journal articles are  in fact ‘peer reviewed’

Our entire editorial board (which we call an ‘editorial committee’) is composed of practicing library technologists — that is, peers in the field.   There’s nobody who makes editorial or management decisions for the Journal except this editorial committee.

Every article comes to us first in the form of a proposal or draft — and every proposal is looked over by the editorial committee, which then, by a vote, provisionally expresses interest (or not) in the article.

Article proposals which have been provisionally approved are assigned an editor from the comittee — and these days, a second editor as well.  That assigned editor (a practicing library technologist, a peer) works with the author to finalize the article for publication, trying to ensure the final published article is as high qualtiy as possible. Anyone else from the editorial committee is welcome to make suggestions as well, but we assign an editor to make sure someone (a  peer) has the responsibility of making sure the article is of sufficient  quality to publish before we publish it.

When the assigned editor (a peer), working with the author, decides the article is finished and of sufficient quality, she or he recommends it for publication to the editorial committee — the entire editorial committee, collectively (by vote), then has to sign-off and agree the article is publication-worthy before it is published. (Very occasionally, when we think we don’t have the neccesary expertise on the committee,  we have asked an external reviewer for an opinion/advice).

Here’s the published summary of our overall process and workflow. 

I say this is peer-review.  By working with the author and making several rounds of suggestions and requirements — it is in fact more peer review than I think many articles get in traditionally ‘peer reviewed’ publications — where I suspect one or two ‘blind reviewers’ read a finished draft, and say “yes” or “no”, or at best give a couple bullet pointed suggestions.

The wikipedia article on ‘peer review’ recognizes enough diversity of peer review processes, that I think even ours would fit — the wikipedia article doesn’t neccesarily equate ‘peer review’ with ‘blind refereed’.  I think the peer review process we supply is important in ensuring high-quality output, and think we are reasonably succesful at it.

What we’re not is “blind refereed”.

Why the Code4Lib Journal should not become ‘blind refereed’

Not even as an ‘option’. Basically, I think not only would it not improve the quality of what we publish, but would actually, in practice, end up decreasing the quality of what we publish, by putting more barriers in the way to actual review by peers — while adding unsustainable workload to our all-volunteer editorial committee.

Since many of us are computer programmers who work on open source, let’s illustrate by analogy (it is just an analogy) to open source software development, patch/pull requests, and code review.  Let’s analogize our current workflow to a distributed collaborative open source software production process, and similarly analogize the ‘double blind referee’ process.

Our current process

Before you write a giant patch (article), you talk to the committers (editors) and make sure the general approach you’re thinking about makes sense with their understanding/vision of the architcture/direction of the software (general priorities, focii, and standards of the Journal).  They say, yeah, this sounds great, please give us a patch. Or they say, yeah, okay, but we think you should do THIS part a bit differently. (Or they say, no, that’s not the direction we want for this software, don’t bother). So you work hard on the patch and submit it.

One or two committers (editors) are assigned to code review your patch (article) — actually they usually volunteer, based on interest/knowledge in the area of the software you are patching, and yes, sometimes existing working relationships with you the code author. They review your code, and carry on a possibly extended back-and-forth with you, helping you make the patch as good as possible/feasible, and good enough to commit (publish).

Unlike most open source, our actual Journal ‘issue tracker’ isn’t public, it’s restricted to other ‘committers’ (editors) (for various reasons).  But, restricted to this audience,  the other committers (editors) can see the conversation, and chime in if they want, helping with the code (article) review. We make sure two committers are assigned as code reviewers to make sure someone’s doing it (and centralize communication rather than have the author get a cacophony of conflicting advice/demands), but advice from any of the committers (editors) is welcome.

After the code reviewers decide it’s good enough, just as a check, before actually committing, they let the rest of the committers know “Yeah, we think it’s good enough, if you want to take one more once over and let us know if we’re making a disastrous mistake to commit this.”  (Our final ‘sign off’ by the whole committee).

A blind referee process, analogized to software development code review

A patch-submitter pretty much has to start out by submitting a complete patch. They submit it.

Two committers are assigned to code review. The patch author doesn’t know who the code reviewers are, the code reviewers don’t know who the patch author is.

Some system needs to be in place to pass communications between them anonymously.  Either it’s a human system or a software system, even if it’s a software system, it’s more work for the committers, someone’s got to maintain that software system, troubleshoot it when it’s not working, etc, all while maintaining anonymity.

The code reviewers don’t know who you are, so a patch from Linus is treated pretty much the same as a patch from Joe Schmoe. The ostensible benefit of this is that Joe Schmoe gets treated ‘fairly’, since it’s quite possible someone you’ve never heard of can write code as good as Linus, and it’s quite possible that Linus messes up sometimes and commits bad code — this is true.  But the other result is that code reviewers have to spend as much time  reviewing every line of Linus’s code as they do of Joe Schmoe’s.  The result is that they actually spend not a whole lot of time reviewing either of their patches, because they’re busy people; Linus’s gets a bit more attention than it probably needed, and Joe Schmoe’s gets a lot less.

It’s pretty much just those two code reviewers who do code review. It’s too confusing to preserve anonymity otherwise.  In fact, not being able to talk directly to the patch author, say in an issue tracker, makes things just cumbersome enough that the code reviewers probably  just give a few bullet points of advice one-time-only (or reject it outright). (That’s how most blind referee works). The bullet points are sent to the patch author, who tries to take account of them. When the patch author says they’ve done it, it pretty much just gets committed.

Really?

Doesn’t that second one sound crazy?  For software development, or for publishing? A bunch of work and beurocracy for not only any increase in quality, but a likely actual decrease in quality? But hey, you treated all patch submitters ‘fairly’, the incompetent as well as the competent both have a chance to get patches submitted! The first one, what we do now, is actually more or less like volunteer distributed open source works.  The second one is how ‘blind referee’ works, and is insanity, and no software development process would ever work that way.

And there’s increasing critique of whether it actually results in high quality for publishing human-language articles either.  The main thing double-blind refereering is supposed to do is prevent prejudice against unknown scholars from unknown institutions — I really don’t think this is much a problem in our Journal or in our (fairly low academic prestige) field of library technology in general.  I could be wrong, I’m certainly not a neutral party — but this is not the reason people are asking or wanting us to do double-blind reviews, nobody’s alleging we’re being unfair or could be more fair. Rather, potential authors  need to meet the requirements of their local processes, whether reasonable and rational or not.

What is the Code4Lib Journal For

Our mission is entirely focused on providing value to an audience by what we publish.

Our mission is not to aid academic librarians in a tenure-track process. Our mission is to provide quality content that aids practitioners in improving their practice.

While some potential authors may not be able to publish with us because of their own local tenure process requirements (which may or may not be reasonable or rational), that has not kept us from getting tons of quality content.  We have no shortage of quality content to publish, we do not need to meet anyone’s local institutional tenure requirements in order to  receive enough quality content.  You can judge for yourself the quality of what we’ve published.

In fact, I think the Journal could be quite a bit more selective than it’s been — if we only published 4 articles per issue, but those were all very high quality articles that addressed issues and topics of high significance to our practice and resulted in improved collective practice — that’d be a success.

Instead, spending our limited attention and resources (reminder: all-volunteer editorial committee which runs the whole thing) on meeting (possibly irrational or unreasonable) local tenure policies would reduce the time and attention we have to attend to our primary goal: ensuring the highest possible quality of content we publish.  And the strictures of a blind refereering process would make it more difficult and less convenient for editors (peers) to be engaged and closely involved in working with authors in an iterative review process to work towards as high quality as possible. A process which I suggest has been succesful in ensuring high quality — through what is in fact a very intensive peer review process, just not a ‘blind refereed’ process.

I also don’t think it has limited the audience, distribution,  or impact of our publication. Some of our articles have been quite widely distributed, talked about, and influential.  I think this is because those were high quality articles, well-written, on topics of impact — I also think it’s because our journal is open access, and formatted/styled readably.  And because our editorial process tries to make sure authors write well, concisely, and readably.  I think the articles quality speaks for itself (or doesn’t) — nobody’s thinking “Yeah, it seems to be saying something useful, but it’s not blind-refereed so I’m going to ignore it.” It just isn’t happening, it’s not a problem.

If potential authors have a problem with their institutions policies that prevent them from publishing in the Journal, we’ll miss their contributions (if they were good :) ), but they should take that up with their institutions, get their institutions to recognize that ‘peer review’ comes in not only double-blind and single-blind forms, but non-blinded forms — rather than take up with us their dissatisfaction with their institutions policies.

Oh, and we also provide even more post-publication peer review — every published article has a comments section. We welcome your comments, readers!