Lecture "Observational Open Science: An Application to the Literature on Irrelevant Events and Voting Behavior"
Cecilia Mo (University of California, Berkeley)
Date: March 26, 2019 (Tuesday) 11:00~12:30 (10:30 Open)

Venue: Room 549, Akamon General Research Building, Hongo Campus, University of Tokyo
Access Map

Speaker: Cecilia Mo (Department of Political Science, UC Berkeley)
About Professor Mo

Abstract: Replication and transparency have become increasingly important in bolstering the credibility of political science research, yet open science tools are typically designed for experiments. For observational studies, which make up the bulk of political science, current practice suffers from an important pathology: just as researchers can often ``p-hack’’ their way to initial findings, it is often possible to ``null hack’’ a finding away through specification and case search. We propose an observational open science (OOS) framework that consists of (1) leveraging the passage of time to add new out-of-sample cases, (2) independent collection of all data, (3) pre-registration of alternative specifications, (4) multiple simultaneous replications, and (5) collaboration between original authors and skeptics. We apply the approach to three highly cited studies in the literature on ``irrelevant’’ events and voting behavior. On some dimensions, the studies all replicated quite well. On other dimensions, they replicated quite poorly. Had we sought to debunk any of these three studies in a conventional approach to replication with ex post specification search, we could have easily done so. However, given that we conducted pre-registered analyses, we have a more accurate sense of the full and complicated picture. We conclude with suggestions for future refinements to our approach.