Psychology professors participate in National Study Replicability Project

Project included 20 universities nationwide

Study replicability has been making the rounds in the media recently, ranging from The New York Times to the HBO series “Last Week Tonight with John Oliver.” Five professors from NSU’s College of Psychology worked on the issue as part of a nationwide research effort.

The NSU professors contributed to the nonprofit Center for Open Science’s Many Labs 3 project to test the replicability of psychology studies. Weylin Sternglanz, Ph.D., heard about the project because the center was founded by his former professor, Brian Nosek of the University of Virginia. Separately, Jonathan Banks, Ph.D., heard about the project from a former student who was involved as a coordinator.

“It’s an important issue for all science, especially psychology,” Banks said.

Sternglanz said that the project is an attempt to crowdsource replicability research. Twenty universities participated in the effort, which produced a manuscript with 64 authors. In addition to Banks, Sternglanz recruited three other colleagues to participate in Many Labs 3: Tim Razza, Psy.D., and Madhavi Menon and Leanne Boucher, both Ph.D.

“They have the advantage of a large and diverse sample because there’s so many people in so many different places,” Sternglanz said.

Boucher was drawn to the project’s focus on replicability. “I think lots of researchers think about whether their studies will replicate. I was excited to do it because I could contribute to a larger database and answer questions I had been curious about.”

Menon appreciated the opportunity to collaborate with colleagues who have different areas of expertise. “All of us are looking at questions through different lenses,” she said.

With the pressure to publish novel research, replicating a study does not receive as much attention.

“A lot of flashy studies get published, even if they have small sample sizes,” Banks said.

The pressure to publish can lead some researchers to falsify data, or to massage it and attempt to pull a statistically significant result from insignificant data.

“We all need to be strong consumers of research,” Razza said.

Each group member had heard the conventional wisdom that college students participating in studies are less attentive at the end of the semester and produce worse results. In fall 2014, the group examined 10 studies and gathered data from students at the beginning, middle, and end of the semester. The studies included the importance of weight and whether holding an object like a heavy clipboard influenced the participants’ perception of their importance. Another focused on if reading a description of a person as “warm” could prime participants in estimating a room’s ambient temperature. The well-known Stroop effect was also tested, in which participants read the name of a color, such as blue, but the word was printed in red ink. The group found virtually no change in the data gathered early in the semester versus at the end. Altogether, seven out of the 10 original effects were not replicated.

However, Sternglanz noted that not replicating the original study’s effect did not mean that there was no absolute proof for it. “There’s no absolute proof in science, only varying degrees of probability,” he said.

Six students who participated in the project received an honorable mention at the 2015 NSU Undergraduate Student Symposium. The Many Labs 3 manuscript is currently in the process of receiving responses from the original researchers. Publication of the manuscript is in the works.

Because of his participation in the project, Sternglanz said he now spends more time in class discussing replication when teaching methodology. The group members said they would be willing to participate in a similar effort in the future.

“Participating in this project made me more sensitive to that students should not take things at face value. It sort of made it more real to the students, more approachable,” Menon said.

“I think it’s excellent for the field,” Boucher said. “I find it intellectually stimulating.”