Skip to content
December 12, 2011 / jpschimel

Revisionist History: how to handle a study that goes in unexpected directions

If you are doing research that is novel and creative (which I hope you are), then sometimes the data, and the story that develops from them, won’t be what you’d anticipated. So how do you write the paper?

You could lay out your starting questions and then discuss how the data don’t address those questions, but rather, different ones. Unfortunately, that would probably produce a paper that is incoherent. That is bad. But, if you rewrite the paper to make it seem like the story you ultimately found was the one you started out looking for, you may have been dishonest. That is worse. So, how do tell the story that emerged without being dishonest about your earlier thinking? Can you write a good paper while treading the knife edge between bad and worse?

The simplest case would be if you falsified your initial hypothesis. This means you get to tell the story you were aiming for, just slightly twisted and perhaps stronger. After all, Philosophy of Science dictates that we are supposed to try to falsify our hypotheses. Because one can never prove a hypothesis, falsifying is absolute while supporting can never be. This also makes a good story; one of the classic storylines is overcoming established authority to change the world. So, if your initial hypothesis reflects the conventional wisdom, and you overturn it, you fit directly into a classic and powerful storyline.

When you do this, though, you must address the real, scientific hypothesis you were testing. It should not be a statistical null hypothesis. Saying that “our null hypothesis was that there would be no effect, but, Eureka, there was” does not make a good story. Equally, setting up an arbitrary “straw man” just so you can knock down is tacky and feels forced. Readers know how they think and what the overall state of thinking in the field is, so trying to impose ideas upon them—“This is how the field thinks, but it’s wrong”—is likely to annoy, rather than educate. Anne Lamott argues that “Characters should not, conversely, serve as pawns for some plot you’ve dreamed up [1].” When those characters are your reviewers, expect trouble.

The real challenge is when the story goes off in a direction you had not anticipated. Then you need to tread carefully. You need to find ways to A) be accurate in what you say about your initial thinking, while B) finding ways to ground the “new” story in extant understanding and literature, but C) without stating that you’d been thinking that way before you started the work. This is tricky, but doable.

Let me illustrate one approach to solving this dilemma with a paper that I co-authored [2]. The paper was about how resource availability affects soil microbes’ response to the stress of repeated drying and rewetting. We had hypothesized that microbes from resource-rich environments would be tolerant of stress because they had the resources to pay the physiological costs necessary to acclimate and survive. Conversely, we had hypothesized that microbes from resource-poor environments would be sensitive because they couldn’t afford to pay for stress-tolerance. We tested this using surface (rich) and deep (poor) soils and repeatedly stressing them by drying and then rapidly rewetting. As it turned out, dry/wet cycles mobilized otherwise unavailable resources (biodegradable organic compounds) and stimulated microbial growth.

The story we needed to emphasize was about dry/wet cycles mobilizing resources rather than just about stress effects on microbes. To make an effective story about resource mobilization we needed to integrate that idea into the paper’s Introduction, even though it wasn’t part of our specific hypothesis when we designed the experiment. Here is a paragraph from the Introduction:

There is a major challenge, however, in developing a better understanding of how dry/wet
cycles affect soil microbial and carbon cycling processes. The challenge is that the specific
mechanism underlying the Birch effect [pulse of respiration following rewetting a dry soil]
is still unclear. Two fundamentally different mechanisms have been postulated, and they
would likely have opposite long-term effects on soil C dynamics. The first is the “microbial
stress” mechanism, while the second is the “substrate supply” mechanism. …
The alternative, substrate supply, mechanism argues that physical processes associated with
rewetting (aggregate disruption, organic matter redistribution, desorption, etc.) supply pulses
of substrate to microbes.

We hadn’t thought about the substrate-supply mechanism in the immediate context of planning this project, so this is revisionist history. We had thought about it, however, for other related work and it was well established in the literature. So, we grounded the paper in the balance of possible mechanisms, which we could have done with what we knew before the work. We carefully did not say that we had thought about this specific issue before we did the work—rather we discussed what was known at the time. Note that we even used the passive voice “have been postulated” to avoid stating who had postulated it. Thus, I consider this opening to be a good example of balancing honesty and the need to tell the right story. We built the new story by following Anne Lamott’s advice (See Chap. 2 in Writing Science): we listened to our characters and told their story, rather than stuffing them into the plot line we had anticipated.

There are a number of flag-words that often indicate revisionist history in writing a paper. When authors write things like “We used these data to test the hypothesis that…” you can be pretty confident that wasn’t the hypothesis they thought they were going to test when the planned the research! But that is no criticism—quite the opposite, it is a complement. Those authors were being honest about the paper and in avoiding misrepresenting their previous thinking. For the paragraph I wish I’d written differently, look to a future Blog post.

[1] Anne Lamott. Bird by Bird. See Writing Science, pg. 9.

[2] Xiang et al. 2008. Soil Biology & Biochemistry. 40: 2281-2289

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: