Theories and models are not the only fruit
Daniel Nettle
Final post of the 6-part series, Modeling for metascientists (and other interesting people).
Poor old psychology: looking silly again.
It wanted to be seen as a real science, hypothetico-deductive model and all. First it looked a bit silly when most of the ‘effects’ that constituted the alleged support for its theories turned out not to replicate. And now it’s looking silly for a second time, as people point out that those theories themselves are remarkably weak (actually this problem has been known from much longer, since before the beginning of the current replication crisis).¹
It’s often not clear what, precisely, the theories predict, where they come from, what the alternatives are, what patterns of data would put them in jeopardy, and whether they actually explain (as opposed to merely describe) the patterns taken to support them. Thoughtful researchers have tried to shut the barn door of non-replicability with better experimental practices, such as pre-registration. And now, people are wondering if formalising theories with formal models could be another way of improving our discipline. Leo’s blog series is part of the mini-movement to encourage better theory in this way (and the series does a nice job explaining why, and dispelling some misconceptions).
I approve of the move to make formal models from our theories where possible, just as I approve of the move to improve experimental practices. If you claim to have a fully identified theory, you should be required or encouraged to formalise it, to make its assumptions and predictions completely transparent.
But the point I’d like to make here is a different one: you shouldn’t have to claim to have a fully identified theory (especially if you haven’t actually got one, which I think is true of much psychological research, including plenty that is well worth doing). Although the hypothetico-deductive model is an important part of science, there is a great deal of valuable science that precedes it. Biology had hundreds of years of taxonomy and natural history before it had formal phylogenetic models; geology had detailed maps of rocks before it accepted plate tectonics; people had been studying the motion of planets long before Newtonian mechanics, and so on.
A lot of what we do falls into one or both of two categories, both of which precede the development of theory proper (or formal theory) in theory construction methodology.² We might be doing natural history of psychological phenomena (what psychological processes seem to exist, which ones are distinct and which the same, how should they be grouped, and so on). So, this is the stage of simply identifying the phenomena. Or, we might also be doing proto-theory (what kinds of levers can we pull to change the output of a process, and what kinds can we not pull? What kind of manipulations make people’s intuitions or behaviours different, and what kinds leave them the same, even if we don’t quite know why at this stage?). You might well do experiments to answer these kinds of questions, without anything as grand as a fully-identified theory. We can draw on what has seemed to work elsewhere at this point; or, being human ourselves, we can use our own introspection, intuition and observation of social life.
For me, a problem with much of psychology is that we have had to pretend to have theories even when we were actually just describing phenomena or “proto-theorising.” That’s why our discipline is full of silly-sounding theories like five-factor theory, which is the theory that there are often five factors, or social identity theory, which is the theory that people have social identities that are important to them; or statements like ‘life history theory predicts…’ where the actual empirical regularity seems to bear no relation to the cited body of formal theory.
The underlying research in these cases is not silly: there is nothing silly about mapping out an area or trying a few levers to see what ends up changing and in which directions. But for years, because we have been trying so hard to look like ‘proper’ science, we’ve had to dress up what we were doing as ‘testing something-something’ theory, as opposed to ‘some stuff that we noticed or tried.’ It’s not surprising that many of these theories turn out not to be strong — they were only nods at theory, proto-theories at best. It’s like the old Soviet joke: the bosses pretend to have theories, and we pretend to test them.
I would like people to formally model their theories when they are ready to do so (and to be honest, we’ll be there a lot sooner in psychophysics and parts of cognitive psychology than we will in social psychology). And all of Leo’s advice is helpful for understanding why we might do that.
But, another reform I would like to see is people not feeling the need to claim that they have a ‘theory’ when they’re really just doing natural history or proto-theorizing. At present, people are as guilty of CITEing (Calling It ‘Theory’ for Effect) as they are of HARKing (Hypothesizing After the Results are Known). It ought to be quite fine to say that we are testing some intuitions about what kind of factors affect this process, or investigating different ways of measuring something, with the cheerful admission that there’s no real theory yet. Maybe this could be part of psychology’s recovery plan: just as a paper shouldn’t need a significant p-value to be publishable, it shouldn’t necessarily need to claim to have used a theory either.³
- Meehl, P. E. (1990). Why Summaries of Research on Psychological Theories are Often Uninterpretable. Psychological Reports, 66, 195–244. https://doi.org/10.2466/pr0.1990.66.1.195
- Borsboom, D. et al. (2021) ‘Theory Construction Methodology: A Practical Framework for Building Theories in Psychology’, Perspectives on Psychological Science. https://doi.org/10.1177%2F1745691620969647
- Scheel, A. M. et al. (2020) ‘Why hypothesis testers should spend less time testing hypotheses’, Perspectives on Psychological Science. https://doi.org/10.1177%2F1745691620966795
Edit, May 17 2021: “explain” added after “whether they actually…”
Edit, May 19 2021: bracket added after “current replication crisis.” Eiko Fried can now finally get some sleep : )
