Skip to content
August 23, 2018 / jpschimel

“Evidence-based review”?

I got an e–mail this morning from Environmental Research Letters (ERL) proudly announcing that they now publish “evidence-based reviews.”

Screen Shot 2018-08-23 at 10.33.43 AM

I was initially stunned, then horrified by their choice of language. If their reviews are “evidence-based” what are everyone else’s? I always understood that for something to be science, it had to be based on evidence! The alternative to an “evidence-based review” is a review not based in evidence? But by definition, that would not be science—it would be science fiction.

It seems that what ERL may be emphasizing is more along the lines of meta-analysis, in which the review is a formal quantitative analysis of specific data-sets. If so, yes, that is different than qualitative or conceptual analysis of existing knowledge and understanding. If you want to know how much the Earth’s temperature has increased over the last 50 years, there are many datasets to synthesize, and a conclusion must use a formal analytical structure that provides clear rules for what is included or excluded. But that is no more “evidence-based” than a “traditional” review that synthesizes existing understanding of a topic. I’ve written a number of such reviews and I maintain that they are deeply “evidence-based;” I’m sure that the reviewers and editors who handled those papers would agree.

So why did the ERL Editor’s choose the term “evidence-based review”? A term so loaded that I’ve been stewing over it for hours and that it motivated me to write a blog post?

I can postulate three, not mutually exclusive, hypotheses. First, but I suspect least likely, is that they did intend to disparage the more traditional conceptual approach to synthesizing knowledge and literature. Perhaps the editors feel that this approach is too subject to individual interpretation. But all datasets are subject to interpretation and that is what peer review is for: to ensure that contributions are robust, sound, and accurately reflect the evidence.

More likely would be that they simply fell into a “Curse of Knowledge” trap—they knew what they meant by “evidence-based,” and did not see that it might be viewed differently by others. Such problems plague communication and are hard to avoid because it is hard to know what others know and think.

I have more sympathy for this explanation, but only a little because this should have been easy to foresee and avoid. If you create a new category of “evidence-based” review, you obviously and explicitly suggest the existence of “non-evidence-based” reviews—something I never dreamed could exist until I got ERL’s e-mail. This is a form of “othering” that I find very problematic. I can only hope that the Editors of ERL were looking for a simple, positive term to define a new category of reviews, and didn’t adequately consider the implications of their language choice.

My third hypothesis recognizes that ERLs Editor-in-Chief is Dr. Daniel Kammen. Dr. Kammen is an eminent scientist who works extensively at the interface of environmental policy. In the U.S., there is increasing focus in policy decisions to distinguish inputs that are based on real evidence vs. those based on the pure opinion. ERL is a journal that aims to publish science that will be relevant to environmental policy decisions. Hence, perhaps there is a need to more effectively identify science as being evidence-based. So voilà: “evidence-based reviews”! In the Journal of Public Policy, I wouldn’t object to this, because in policy the distinction between data-based vs. expert opinion based input is important.

But if that hypothesis is correct, the appropriate response for ERL, a pure science journal, should not be to flag some publications as being “evidence-based,” and so to suggest that there is an alternative (are they going to have evidence-based research papers?), but to more effectively highlight that “If it isn’t evidence-based, it isn’t science” and that ERL only publishes science.

I can believe that the decision to use the term “evidence-based” might reflect Dr. Kammen’s experience at the science-policy interface in the era of “Fake News.” If this is true, though, I am still deeply disappointed in the journal’s choice of terminology. I very much hope that ERL will find a better, more suitable term to describe what they are looking for.



Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: