Reproducibility and open science

I began grad school as a Master’s student in 2012 when the replication crisis was starting and damn was that a shock. Things definitely changed very quickly in those two years!! Just my brief two cents on this in an attempt to say what’s not already been said. I’m going to separate them into what’s great and what’s not so great.

What’s great:

• It’s made me a better researcher. I think more about my research design.

• Pre-registrations in writing are great pieces of documents to remind you of what exactly you’re doing when you’re doing more than 5 projects at the same time or just extremely stressed.

• It has reduced my temptation to jam pack everything into one study.

• It’s not an all or nothing. There is still room for exploratory analyses.

• It’s helped identify things I thought I knew, but actually didn’t.

What’s not-so-great:

• The gap is widening. Your stats needs to be better, design needs to be better.

• There is a great deal of resignation that this is way it’s going to be forever which of course affects morale. [Sidebar: I think we’re in the middle of sweeping changes and it feels that way, especially if you’re future is that of an uncertain grad student].

• Lots of material is on blogs, personal websites and not peer-reviewed. Great for open science and increased access, but bad for filtering out the junk information, especially if you’re a noob.

• The implications are often unclear when it comes to the nuts and blots of literature review writing. Should pre-2010s studies and findings be taken seriously if they haven't been replicated, have small effect sizes, or seem to be p-hacking?

To sum it up, it's been a lot of changes for sure, but my observation of people in my academic networks is that they mostly seem positive about what's to come. In some ways, perhaps working on addressing reproducibility has opened certain doors that were previously shut to those in lower levels of the academic hierarchies. That, I see, as a great start.