On the Replication Crisis and Forms of Knowledge

Psychology fails its own internal criteria for truth. As such, we should reject it until the replication crisis is solved and instead rely on other disciplines to understand the mind.

Psychology seems to dominate our view of the world. The subject in general, and evolutionary psychology in particular, have become the go-to tool for understanding why our friend chose a particular subject, or why they decided to date that horrible guy. Other traditional sources of explanation - class, genetics, the favour of God, and so on - are no longer employed to the extent that they were. Given its domineering place in our way of thinking, it should be questioned.

There are two main reasons why psychology shouldn’t be as dominating as it is. First, as I’ll argue in this post, until the replication crisis is solved, psychology does not fulfill the requirements its stipulation as a form of knowledge. As such, we should turn to other forms that don’t use the scientific method as a claim to knowledge. Second, as I’ll argue in a forthcoming post, we turn to psychology to satisfy our need for forgiveness, instead of the posited reason of “scientific knowledge of the mind”. This is a type of self-deceit, and should be rejected.

Some Caveats

Psychology is a broad subject. When I say psychology, I mean what pops into your head when you think about psychology: clinical psychology, which is the study of mental disorders and how the mind functions, as well as behavioural psychology, which is the study of how the mind reacts to different situations. The arguments presented here could probably be extended to other areas of psychology as well, but I’ll restrict myself for the sake of clarity.

I’m also taking quite an aggressive stance on the subject for the sake of argument. I’m not an academic psychologists, and the replication crisis is still in flux. But this fluxness is a problem in itself, as I’ll argue below.

The Replication Crisis

Psychology has during the 2010’s been rocked by something called the replication crisis. It turns out that a lot of fundamental experiments are impossible to replicate, i.e., perform again and get the same results. That is a massive problem. As Popper famously stated, “non-replicable single occurrences are of no significance to science”. Two different people, using the same theory, method, and material, should get the same results. If not, it doesn’t qualify as science. And a lot of experiments in psychology are not replicable. Most of the information here has been taken from Felipe Romero’s excellent article “Philosophy of science and the replicability crisis”.

The crisis started with some notable papers in the early 2010’s, like with Bem’s extra-sensory perception studies. The author apparently found evidence of extra-sensory powers to predict the future. This result was quite reasonably widely critiqued and the results were not found to be replicable. That in itself is not a problem - faulty studies are made all the time. But the problem was rather that most of the statistical techniques used in the study to achieve the results were common practice in the field of psychology. The obvious follow-up question is then: “If this study was able to produce statistically significant results with the common techniques, how can we be certain that all the other studies that use the same techniques are trustworthy?”.

To answer that question, meta-studies (i.e., studies of studies) were conducted. Perhaps the most notable is the Reproducibility project that takes random samples of studies in psychology and attempts to replicate them. Here comes the shocker: only about one third of all studies were replicated! Likewise, other meta-studies have found alarmingly low levels of significance.

There are many reasons why this situation has occurred. Romero points at p-hacking, the reward structure of modern science that promotes publication bias and discourages replication testing through the “publish or perish” culture, as well as a misunderstanding of the philosophical logic underlying the statistical techniques. If you’re interested, you can read more about it here and here. I’ll instead turn to the philosophical implications.

Forms of Knowledge

Why, then, is this a problem for psychology? Sure, the experiments are impossible to replicate, but so are a bunch of other things. You wouldn’t be able to write the same book two times but literature and art still tell us things about the world. This is a good point. To understand why this poses problems for psychology and not art, we should think about something I’ll call forms of knowledge, and idea that bears resemblance to the paradigms of Kuhn.

Forms of knowledge are the forms of truth. Different disciplines have had different forms of knowledge, and different criteria (or methods) to determine what is true. The most common in the modern era is science and the scientific method. Science is a very robust form of knowledge and indubitably tells us true things. But other disciplines have different forms of knowledge. Art and literature are the main alternative in the modern world. They, equally indubitably, can provide us with insights about the world and tell us true things. I, for example, have learned true things about the human condition through, say, Kafka. But that form of knowledge is different from what I’ve learned about Newton’s laws of motion. I would even posit that different forms of knowledge are incommensurable, which is a fancy philosophical word for saying that two things are so different that you can’t compare them. Asking which is better, or more true, is like asking whether kilos or liters are larger.

So I then classify science as the form of knowledge that uses the scientific method, where replicability is a pivotal constituent part.

At this point, some people might reasonably say that lots of scientific disciplines aren’t perfectly replicable. Most hermeunistic social science is not replicable. Does this mean that hermeunisitcs do not qualify as science? My answer would be no, it doesn’t. But hermeunistic scholars, usually, argue that they use a different scientific method, with different critieria for what constitutes good science. In my language, a different form of knowledge.

Psychology, however, does not do this same maneuver. It still uses the scientific method as its truth criteria. When a large part and especially fundamental building blocks of a discipline fails its own truth critieria for its form of knowledge, we should be very careful. We can’t simply say that it’s just a few rotten planks that we won’t use when building the structure. A discipline is interconnected, and every result relies on other, older results, and those rely on still older results, etc. etc. The replication crisis means that the discipline of psychology becomes a old, dilapidated ruin. You really shouldn’t run around and play in one of those.

What Then?

If we then abandon psychology until the replication crisis is solved, where should we turn? To say that we simply cannot know anything about the human mind would be very boring. Thankfully, there a plenty of disciplines that tell us plenty about how humans function, that do not employ the same form of knowledge. I’ve already mentioned literature and art, of course. Crime and Punishment by Dostoevsky, The Year of Magical Thinking by Didion, or Inferno by Strindberg, all serve as excellent sources of knowledge. We might even return to psychoanalysis. I’ll leave the choice of alternative disciplines to others. But whatever we choose, we should make sure that it is consistent with the critieria of truth within its form of knowledge.