Moving beyond unwise replication practices: The case of romantic motivation

Replication research holds an increasingly important place in modern psychological science. If such work is to improve the state of knowledge rather than add confusion, however, replication attempts must be held to high standards of rigor. As an example of how replication attempts can add confusion rather than clarity, we consider an article by Shanks and colleagues (2015). They conducted a meta-analysis of studies examining romantic motivation, using problematic criteria for the inclusion of effects and reached conclusions of a null effect that were unwarranted. A more rigorous and defensible approach, relying on a representative analysis of effects and p-curves, suggests a different, more positive conclusion with no evidence of p-hacking. Shanks et al. also conducted several experiments that suffered from numerous issues, such as relying on inappropriate subject samples (e.g., older adults likely to be less sensitive to mating manipulations than college students used in previous research), altered research methods, and demonstrably weak manipulations, among other problems. We discuss the broader implications of this case, to illustrate both the opportunities and the pitfalls inherent in attempts to replicate contextually sensitive research.