The sharp minded among you might remember last Autumn's Junior Researcher Grants initative. We asked early career researchers to submit preregistered study plans, and we gave away $5000 to enable their awesome research on Prolific!

This blog post is written by the first of our grant winners, Emma Henderson of Kingston University, UK. Take it away Emma!

In December 2018, I was lucky enough to receive a Junior Researcher Grant from Prolific in order to fund a close replication study. This was the first study of my PhD in experimental social psychology, where I'm investigating how people use non-probative cues, such as repetition, to make truth judgements in the absence of information or knowledge.

The study I set out to replicate investigated truth judgements. The authors found that participants judged concretely worded trivia statements as subjectively more true than abstractly worded statements containing the same content. I wanted to replicate this study in order to test the scientific ground I planned to build on.

As I began my research I was conscious of recent efforts in the psychological sciences to conduct more rigorous research, and I was keen to eliminate bias as much as I could from my work. Registered Reports stood out to me as the format that would help me to avoid doing bad, biased science. The purpose of this post is to briefly introduce Registered Reports, explain how the process works, and describe what it was like to write one!

Registered Reports?

Registered Reports are a type of journal article in which the Introduction, Methods and planned analyses are peer-reviewed and preregistered prior to the research being conducted (stage 1). Based on peer review of the stage 1 protocol the journal offers in principle acceptance (IPA) – a guarantee of publication, regardless of outcome, provided that the stage 1 protocol is followed. This model breaks the traditional outcome-based selection system because results are unknown when the decision to publish is made. This means that Registered Reports prevent many questionable research practices, such as HARKing, low statistical power, selective reporting, and publication bias, while still allowing exploratory analyses to be conducted.

Initially I found the idea of writing a Registered Report pretty daunting, especially as a first year PhD student. But, as with most things, once I got started everything was better than I had imagined. It might sound like a lot of work at the start, but Registered Reports don't really require more work: you just do it in a different order.

Everybody’s excited at the beginning of the study design process: there are so many decisions to make and so much unknown to be discovered. With a Registered Report, you get to plan and write half your paper while you’re still in that honeymoon period. Because the work is front-loaded, Registered Reports help you maintain that level of enthusiasm while working through a much more secure publication process.

Writing a registered report:

Stage 1

A Registered Report has two stages: stage 1 involves writing the Introduction and Methods along with a detailed analysis plan. These sections remain unchanged in the final paper (except a shift from future to past tense). In our paper, we decided to go one step further and use contingent language to describe the conclusions we'd draw from various results. The picture below shows an example of what this looks like in practice:

Contingent language is not a requirement of Registered Reports, but it did mean that once we’d analysed the results, all we had to do was delete the irrelevant language. This might sound like overkill, but it really made us consider the conclusions we would draw from the various combinations of results that could occur. Thinking through every decision was tricky, but Registered Reports gave us the structure to do it much more easily.

Stage 1 Review

We submitted the stage 1 Registered Report to Collabra: Psychology. Stage 1 review is about evaluating the aspects of research we can control and ignoring those that we can’t. So at this stage, the focus is on the importance of the research question, the rigour of the methods and the soundness of the analysis plan. Reviewers can't be swayed by results because the study hasn’t been run yet!

The reviews we received were both helpful and collaborative. Importantly, because reviews were received before we ran the experiments, we could actually implement reviewer suggestions and improve our study! We responded quickly and received IPA within a few days. This meant that as long as we followed our plans, the eventual paper would be published regardless of the findings. It took six weeks between stage 1 submission and receiving the IPA.

Data Collection

With an IPA under our belts, we began data collection with confidence. We conducted the same experiment twice: with students in the classroom and online, using Prolific to collect data from a broader sample. We piloted our study carefully on Prolific (because data collection happens so quickly, it’s important there are no mistakes!). The pilot went smoothly, and I started the main data collection process one evening to find it was finished by the time I woke up! In contrast, data collection in the classroom was slower and more labour intensive. Thankfully, knowing that we had IPA kept us motivated.

Stage 2

Given that we’d already written the plan, data analysis was very quick. We didn’t replicate the original finding - our primary confirmatory hypothesis was non-significant - but with IPA independent of the results, that didn’t phase us. It was then trivial to delete the non-contingent language in the Results section, write a short Discussion, and we were ready for stage 2 submission.

Stage 2 Review

Stage 2 review was quite different from the traditional review process. The scope is limited to checking that you followed your method and analysis plan, and that the conclusions you’ve drawn reflect the data. Importantly, reviewers cannot comment or suggest changes to any aspect of the study that has already passed stage 1 review. It took six weeks from stage 2 submission to full acceptance (it would have been quicker, were it not for some ill-timed holidays on our part) and the article was published a few weeks later - hooray!

Conclusion

Overall, I enjoyed every stage of writing a Registered Report. Writing stage 1 meant I had to think ahead and make more detailed, non-arbitrary, upfront decisions. Then, after IPA, I felt confident in conducting the study because it had already gone through peer review. Needless to say, the experience of writing the stage 2 manuscript knowing it would be published was, well, great!

I would definately recommend the Registered Report route for your next study: any initial fears I had were completely blown away by the many positives of the format. In fact, the only difficulty I experienced was figuring out when to celebrate our publication, because it happened in stages!

If you’re curious to see the difference between a stage 1 and stage 2 Registered Report, you can find both versions of ours on the OSF here. Our final paper is published open access here. And, you can find out more information about Registered Reports here including information about the (over 200) participating journals and a template to guide you through the writing process. I wish you the best of luck if you decide to write one yourself!

Click here to learn more about Emma Henderson

Come discuss this blog post with our community!