Why don’t we evaluate scientists based on their potential?

I’ve been thinking that it’s a bit strange that we hire scientists for permanent positions based on their past contributions, as opposed to their potential to make contributions in the future.

In plenty of other domains, we don’t make this mistake. When we decide whether to buy a house, we don’t really care about what the house was like in the past or assume that the house will be worth the same amount in the future. Instead, we’re reasonable. Maybe it’s not our “dream” house in its current state, but we see that it could be great if we put in a bit of work, or might be worth more if the neighborhood perks up.

When we decide whether to build a life with a relationship partner, we (hopefully) don’t just judge them based on their past relationships, whether their life is perfectly sorted at the moment, or whether they eat soup like a normal person or have this ridiculous habit of holding the spoon with the palm of their hand (hypothetically). No, instead we make our decision based on their potential — whether we see a possibility of a future where we can live a happy life together and where they might learn to eat soup properly someday (again, hypothetically).

It’s a bit weird then that in academia, we make decisions about whether to fund or hire people based on the work they’ve done in the past, as opposed to their potential for producing important work in the future.

My guess is not that there’s some deep logic as to why we do this, but rather that evaluating based on past contributions is easier. It’s simple enough to count up a scientist’s publications, the number of times their work has been cited, the impact of the journals in which they have published, and so on. On the other hand, predicting future impact is a lot harder (e.g., predicting future citations from past ones).

But ignoring practical difficulties for a moment, it seems to me that, in theory, there are a few approaches to this problem. Here I’ll throw out two that might be worth thinking about.

Consider the slope of scientists’ contributions.

Knowing a bit about the slope of someone’s career might be nice. By ignoring the slope, we miss out on those scientists who spend the early parts of their careers developing skills and learning things that will improve their ability to make future contributions (i.e., trading off early specialization and productivity for an investment in their capacity to make contributions at a later point).

The below figure might help to illustrate why slope matters.

Imagine two hypothetical scientists, Privileged and Hard-Working…

Privileged has an easier start: they begin their career at a prestigious university with many resources and can produce impactful work early on. Because of this, Privileged doesn’t have to work that hard to achieve success, and so they develop an average work ethic.

Hard-working has a tougher trajectory: they begin their career at a mid-tier university, without much funding and with substantial teaching obligations. This makes it harder for Hard-Working to produce impactful work early on. But, the tougher circumstances motivate Hard-Working to develop a fierce work ethic and dedicate themselves to self-improvement. As a result, their skills grow at a faster rate.

Early on, Privileged has the upper hand. But in the long run, Hard-Working will beat out Privileged and produce more impactful scientific work (ok, technically you’d need to integrate the productivity of both scientists over their entire careers, but you get the idea). Judging future success by past accomplishments would erroneously favor Privileged, whereas judging future success based on slope or “trajectory” would accurately favor Hard-Working.

Consider the functional form that underlies scientists’ contributions.

This is a bit more nuanced than the slope approach, because for non-linear functions the slope varies depending on where in the function you look. But if it can be done, a function-centered approach would be even better than a slope-based approach.

In this example, early on, Privileged produces more impactful research and has a steeper slope than Hard-Working. But if you knew the functional form of their contributions, you might still want to pick Hard-Working over Privileged. Privileged produces research at a constant rate, whereas Hard-Working starts off slow but will exceed Privileged, given enough time.

The exponential function for Hard-Working could also represent a scientist who invests in reading widely early on in their career. Such reading would allow them to explore ideas and methods from a range of disciplines, developing into more of a generalist. This scientist wouldn’t be very productive early on, but would be in a position to later make connections between otherwise isolated disciplines, to see “big questions” that span several fields, and to deal with “wicked” problems characterized by complexity and unpredictability. The book Range provides a case for why such generalist skill-sets are useful (but often undervalued).

Conclusion

We probably shouldn’t hire and fund scientists based on their past contributions, but rather on their potential for future contributions. This post proposed a few ideas for how to approach this. Curious to hear your thoughts as to whether there’s any reasonable way to implement these in practice.

*One thing I didn’t talk about is that of preferential attachment, otherwise known as cumulative advantage, the rich get richer, or The Matthew Effect. Sometimes being successful early on increases people’s chance of being successful later on. As just one example, in the Netherlands, there’s evidence that if you get a big grant early on, you’ll have a better chance of acquiring more funding later in your career compared to a scientist who had a similar track record but wasn’t lucky enough to get the grant.

Using the examples in this post, preferential attachment would mean that Privileged has an advantage over Hard-Working, because they get rewarded for their early productivity. Such an advantage could be arbitrary (e.g., future scientists cite Privileged because past scientists have cited Privileged), or less arbitrary (e.g., getting a grant gives Privileged the time and money to develop into a better scientist).

This generally seems like another reason not to judge too heavily based on past success.

That said, it depends on whether the goal is to figure out who is best scientist OR which scientist has the highest chance of being “successful” in the future. You can imagine the case of a not-as-good scientist who gets lucky early on, and then gets cited and gets grants because of their early luck. So, if you all you cared about was predicting future success (e.g., in terms of grants and citations), then it would indeed make sense to judge based on past success.

**Adding another reason to not judge too much based on past success: success depends on the context in which scientists have found themselves throughout their career. Some scientists may have had toxic supervisors, unsupportive peers, lacked good collaborators, not been in a “good place” during their PhDs, and so on. What you “really want to know” (Don’t kill me, Daniel) when hiring this scientist is how they’ll perform in the place where they’ll be hired. Are they someone who has a more-or-less similar level of scientific output regardless of their context? Then it makes more sense to look at their past success. Or are they more sensitive? Maybe they need to be around certain types of people or in a certain department to do their best work, but when in that context, they flourish? In that case, their past track record will probably have been an underperformance, and they might be a “hidden gem” if you can provide them with the right working environment.