More on structural problems with academic research

Not to get too cynical, but I do think it is important for librarians to be aware of ongoing research into structural (not just ‘bad actor’) problems with academic research and the validity or quality of published research.

I think it’s part of our jobs to understand these issues — and part of our jobs to educate our patrons about them.  Paying attention to the growing research into these structural problems will also keep us from over-simplifying the task of separating ‘good’ from ‘bad’ research, in our own work and in our education of our patrons.

From Nature, US behavioural research studies skew positive: Scientists speculate ‘US effect’ is a result of publish-or-perish mentality.

US behavioural researchers have been handed a dubious distinction — they are more likely than their colleagues in other parts of the world to exaggerate findings, according to a study published today.

“The take-home here is that the ‘bad guy/good guy’ narrative — the idea that we only need to worry about the monsters out there who are making up data — is naive,” Martinson says….

…The researchers then determined how well the strength of an observed result or effect reported in a given study agreed with that of the meta-analysis in which the study was included. They found that, worldwide, behavioural studies were more likely than non-behavioural studies to report ‘extreme effects’ — findings that deviated from the overall effects reported by the meta-analyses.
 And US-based behavioural researchers were more likely than behavioural researchers elsewhere to report extreme effects that deviated in favour of their starting hypotheses….

…Fanelli and Ioannidis do not explain why that might be. They found that the ‘small-study effect’, in which overall results are biased towards positive, extreme findings because negative findings from small studies are not published, did not explain their results.

“It has to be because of methodological choices made before the study is submitted,” Fanelli says, possibly under pressure from the ‘publish or perish’ mentality that takes hold when career progress depends on high-profile publications….

There is a growing literature of research into structural problems with academic research (especially biomed), this is just one recent article. I am not aware of any textbook or good article or essay summing it all up. Or for that matter of any good bibliography. Preparing such a bibliography might be a great project for an interested librarian or library student — bibliographies are a much less significant part of librarians jobs than they used to be, but it occurs to me this is an area where a bibliography would be useful. (I wonder if there are any good (free?) platforms for collaborative construction of an annotated bibliography? Mendeley?)


[Hacker News comment thread on the Nature blog post.]


2 thoughts on “More on structural problems with academic research”

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )


Connecting to %s