It's a throbbing question, but I have to ask it
For years, I have defended the university system against critics on the political right. For example, in 2017, I wrote this in response to a Republican plan to raise taxes on certain activities at universities.
The university system in the United States is still one of the most important systems with economic advantages in this country. Despite manufacturing moving to China, the United States maintains its superiority in higher education. The research and technology contributions brought by universities across America and the highly skilled graduate workers are important factors in the continued accumulation of knowledge industries in this country. Just to name three, one reason why Silicon Valley, the pharmaceutical industry, and the oil service industry remain in the United States instead of fleeing abroad for lower labor costs is the presence of such highly skilled workers. If higher education deteriorates, the attractiveness of the United States for cutting-edge industries will decrease, and its importance to the global economy will also diminish.
This point is still valid. However, seven years have passed since then, and I have become even more convinced of the following point. The American university system needs greater external oversight, and criticism is also necessary to address various issues. Without doing so, the above advantages will no longer be valid.
Trust in American universities continues to decline - not only among Republican supporters, but also among Democratic supporters.
Source: Gallup
In the plagiarism scandal that recently occurred at Harvard University, this debate is heating up. In a provocative article titled Universities Are Not on the Level, Josh Barro argues that the problem lies in universities accepting the spread of dishonesty.
In my view, the problem starts with a crisis of reproducibility (...). The research in social psychology that I studied at university is being constantly refuted. Replication experiments continue to fail, p-hacking is often carried out in research, and sometimes studies are even conducted based on fraudulent data. (...) I am not the only one wondering how long Dan Ariely intends to remain a professor at Duke University (...).
A few weeks ago, Matt Yglesias wrote an article about a paper by historian Jenny Balfour-Paul (...). Balfour-Paul claimed that a somewhat important metallurgical technology developed in late 18th-century Britain was actually stolen from a black Jamaican metallurgist (...). The weakness of Balfour-Paul's paper lies in the complete lack of credible evidence to support her claim.
Furthermore, Barrow discusses various aspects of campus life, such as employment practices and student acceptance, where dishonesty can be observed. However, the point he makes about dishonest research hits the nail on the head. When defending the university system against conservative critics, I have always relied on the research conducted at universities. While universities are often thought of in terms of their educational function, as most Americans only interact with the university system when receiving education, perhaps the research function of universities is even more important for our country. Over the past 40 years, the responsibility of research undertaken by universities across the country has become increasingly heavy:
Source: Arora et al. (2019)
Such research is extremely important for increasing human knowledge, but it is also essential for strengthening the private sector. According to the research by Tartari & Stern (2021), for every $1.6 million in research grants to universities by the government, a startup value of up to $50 million is eventually generated!
However, at the same time, there is something happening in parallel: productivity of research measured by total factor productivity growth compared to research expenditure is steadily declining.
Source: Bloom et al. (2020)
So, it's not that there is no longer any value in investing money in research (there is value). Besides, this trend is not solely the fault of universities. This trend has been happening long before universities became the main venue for research. Perhaps, it's largely due to the fact that it has become more difficult to simply discover various ideas than it was in the past, or maybe it's because more money is being invested in research projects in America, which may have less significance (but still have value), or perhaps it's because research results are not being commercially utilized.
However, considering the pivotal role that universities play in the research ecosystem today, as well as the crisis of replicability in fields such as psychology, medicine, and other empirical disciplines, it is reasonable to question whether American universities are wasting the country's resources by producing research that is less meaningful than before or focusing on research with little significance for the future.
This is a throbbing question. There are too many people who have experienced the university system, work at universities, or have friends and family working at universities (including myself). Considering the lurking threat of conservative forces wanting to destroy the university internally due to culture wars unrelated to research productivity, many progressives feel uncomfortable. Nevertheless, this question needs to be asked. Because the talent gathered in research universities across the country is an exceptionally valuable resource, even in America. It is also necessary to consider whether the incentive system we have created effectively utilizes such resources.
It's really difficult to measure waste
It is extremely difficult to quantify the waste of research. One way to think about it is to consider each stage of the research output process from publishing papers to applications. Marc Andreessen is considering such an analysis proposal:
Academic paper quality funnel
- Total number of written papers
- Proportion of papers with reproducible research results out of (1)
- Proportion of papers read out of (2)
- Proportion of papers cited out of (3)
- Proportion of papers not consumed by radical politics out of (4)
- Proportion of papers without plagiarism out of (5)
- Proportion of papers with meaningful research results out of (6) What will be the final number?
To be honest, the only thing I really care about is the last item he mentioned. Let me explain the reason for that.
I'm not worried about the impact of radical political situations on academic papers. Certainly, politics can corrode research. However, even if that's the case, any harm or distortion should be revealed through other filters [evaluating research]. Also, I'm not concerned about plagiarism as a major issue. While cases of plagiarism can expose unethical standards and should be punished, unless the research results themselves are directly copied from other papers (even if only the words used to explain the results are copied), it is not a sign of wasted research effort.
Moreover, measuring whether a paper is cited does not necessarily require actually measuring which papers are being read without being cited (anyway, the latter is difficult to measure). It wouldn't be surprising if there were papers being used in industry without being cited, but such cases are likely to be quite rare.
I wonder how many papers are being cited? It is often heard that '50% of papers end up without being cited.' There are even stories that 'it's 90%.' In fact, this is a quite exaggerated estimate, but it used to be true. Moreover, it seems that this is still true in humanities.
Source: Lariviere et al. (2008)
Self-citations by the lead author are not counted here (although there is a risk of data contamination from self-citations by other co-authors). Looking at more recent analyses focusing on natural sciences, the number of uncited papers becomes even fewer if questionable open-access academic journals are not included.
So, it seems that the number of scientific papers that are not cited is small and decreasing. However, considering this, it would be strange not to be cautious about whether the number of citations is a useful measure of success. With the citation rate in science increasing, can we assume that papers are getting better? There are various reasons to doubt this, especially when considering the overall research productivity is falling apart. Above all, with a significant increase in the number of researchers, there are far more people citing papers than before. Therefore, the probability of any paper being cited at least once increases, right?
Second, there is a possibility that a culture of extensive citation is gradually developing in these fields. There is a concern that a culture where everyone is expected to cite everyone else is emerging. The most obvious mechanism is the peer review system. In peer review, it is very common for reviewers to request 'more citations' before granting publication permission (including the reviewer's own papers). Peer review became a common practice in the 1970s, not that long ago. One possible reason for the very low citation rates in the humanities is that this culture has not yet fully developed. If humanities scholars were to start citing each other excessively, their citation rates would likely skyrocket.
Finally, how many research papers [experimental results] have been replicated? It varies depending on the field, academic journal, and time period, but the best available estimate now is slightly below half. (The numbers are even lower for clinical trials.)
This needs to be a little higher. It is a natural move for psychologists and biologists to try various methods to reduce the failure rate of [reproducibility experiments] - such as preregistration, code transparency, blind review of results [review method that evaluates only the validity of research methods and data analysis], raising the criteria for statistical significance, etc. However, even if the proportion of papers with unreproducible experimental results is halved, it would only be relatively small in reversing the decline in research productivity to one-eighth of what it was in the 1960s. Moreover, it is expected that papers with unreproducible results will receive less attention compared to those with reproducible results. This means that the harm of such papers may not be as significant as everyone thinks - most of the papers with unreproducible results are probably not well-known. Alzheimer's disease research by Marc Tessier-Lavigne and research on dishonesty by Dan Ariely and Francesca Gino are different from widely publicized studies.
Moreover, in many research fields, reproducing research is simply impossible. How do you reproduce theoretical research? What about reproducing philosophical papers or ethnographic studies? Even with research based on observational data, only part of it can be reproduced. Why? Because you can only collect a specific dataset once.
The existence of a large number of unread papers and the crisis of reproducibility are both concerns, but neither seems to be the main worry. What I am mainly concerned about in the academic industry—and what many scholars are mainly concerned about—is not there. The core issue that should be questioned is 'how many valuable research results are there,' and data on citations and reproducibility have little to do with alleviating this concern.
Or rather, measuring the usefulness of scientific papers is very, very difficult. It's not impossible - you can track the course of a study being cited by another study, leading to a patent being taken out, and then the product being commercialized and succeeding in the market. However, no one has managed to do such a large-scale endeavor so far. Moreover, many studies are not aiming for 'commercial success' in the first place - much of the research in social sciences aims at changing policies, and research in humanities aims at enriching our entire culture. In other studies, such as cosmology, the only output is satisfying humanity's curiosity about the universe.
So, I don't think we can get a clear quantitative answer to how much of the published research is useless. However, we can think about some of the strange incentives existing in the university system, and it is possible to consider that research that is useless may be increasing due to such incentives.
Who decides which research is excellent?
I would like to mention an example this time, albeit disrespectful to researchers of string theory and real business cycle theory.
Physicists once hoped that 'string theory' would become the 'theory of everything' that unifies gravity and quantum mechanics. However, by the early 2000s when I was majoring in physics as an undergraduate, string theory was already losing momentum among many. Shortly after graduating, two books arguing that string theory had reached a dead end by failing to make testable predictions were published - Lee Smolin's 'The Trouble with Physics' and Peter Woit's 'Not Even Wrong: The Failure of String Theory and the Search for Unity in Physical Law'. Over the next 20 years, string theory researchers continued to defend string theory against critics, with the typical defense being that 'critics are not studying string theory itself, so they are not adequately prepared to criticize it.'
Over the years, Woit has patiently documented such attacks on his blog. Thanks to this, an entertaining and enlightening chronicle capturing the dynamics of society unfolding at the most profound levels of the elites' academic field has emerged. Today, physicists rarely get upset and come to fiercely criticize someone just for saying something mean about string theory. However, criticism these days is often in the form of 'There's practically no one doing (actual) string theory nowadays, right?' as seen here or in this manner
This incident was an enlightening experience for me because while reading Waite's blog, I had witnessed something similar in macroeconomics at that time. The Real Business Cycle (RBC) theory, which won the Nobel Prize in 2004, was losing popularity in mainstream macroeconomics, being replaced by the New Keynesian theory. This was because the RBC theory was based on assumptions that seemed unrealistic and did not fit well with the data. However, there were researchers in a few university departments who published RBC-type papers, and there were academic journals that enthusiastically featured such papers. Supporters of the RBC theory responded to critics with harsh rhetorical attacks, and I myself have experienced such attacks more than once.
Through such incidents, I learned that scholars tend to retreat to their defensive positions when their theories or fields are challenged. When faced with a challenge, scholars often say things like, 'Outsiders do not understand this field. Therefore, they have no basis for criticism.' The role entrusted to outsiders is simply to keep sending checks to scholars in the form of salaries or research grants.
However, from this way of being, immediately obvious problems arise - people who make a living working in a certain research field lack the incentive to criticize that field themselves. Their salaries, and even more importantly, their reputation, depend on whether they can convince the public that they are exploring important topics and achieving great success. Therefore, if only those within a field are allowed to criticize it, it is very clear that criticism will tend to be biased towards the less likely to occur.
Or rather, I think similar dynamics are at work throughout the academic world. For example, if an economist somewhere writes a paper on the impact of corporate mergers, who would take on the job of evaluating whether the paper is worthy of being published in an academic journal? Other economists writing about the impact of corporate mergers, right? If the paper were to be submitted to a journal, the journal would likely ask scholars researching in the same subfield to review it. Sometimes, they may also try to find external reviewers outside that subfield (often stereotyped as 'Reviewer 3'). However, 'Reviewer 3' usually does not criticize the paper too harshly because they are outsiders who are not well-versed in the subfield. Learning about the field would take up a lot of time, and since there is no compensation for the time spent on reviewing, they tend not to be too critical.
In the worst case scenario, there may be de facto citation rings formed. Citation rings are a type of fraud where a group of researchers cite each other's papers to boost their reputations. Even without explicit collusion, similar situations can arise due to the incentive structure of peer review. For example, let's say Joe and Sarah are both researching the impact of corporate mergers. Joe knows that Sarah often reviews his papers, and Sarah knows that Joe frequently reviews hers. As a result, they have an incentive to cite each other favorably. This dynamic can occur without authors explicitly proposing to cite the reviewer's papers.
The smaller the lower field, the weaker it becomes in this type of quasi-quotation. When there are only 10 to 15 researchers studying a particular topic, it is usually clear who will review their paper. As the academic industry advances in ultra-specialization, research literature becomes scarcer and narrower, making such mutual praise among insiders even more likely.
Even in the absence of collusion, there is a gatekeeper effect. Imagine a newcomer researcher arriving and saying, 'Hey everyone, you're all wrong, this is how corporate mergers actually function.' Even if their argument was 100% correct, the reviewers would likely reject this newcomer's paper. Indeed, there is evidence that this gatekeeper effect is strong. In a study by Azoulay et al. (2019), it was found that when a prominent biologist dies, their research field becomes more vibrant, allowing research to progress vigorously:
This paper examines how the vitality of a particular field changes when outstanding life scientists die prematurely. After the death of a star scientist, the influx of papers authored by [collaborators who had worked with the scientist] decreases, while the influx of papers authored by [other researchers who did not have a direct connection with the star scientist] significantly increases. The papers by these newly increased outsiders reference different literature than before and are overwhelmingly more likely to be cited. While during the lifetime of the star scientist, outsiders are observed not to actively challenge the leadership structure of the field, when the leader passes away, opportunities for the field to develop in new directions arise, and the frontier of knowledge advances in those directions.
In other words, to promote scientific progress, it is probably effective to forcibly retire famous elderly scientists early.
To a certain extent, whenever specialized knowledge matters, this issue always arises. If the Department of Defense warns that the United States needs to increase defense spending to deter enemy countries, what should we do? Should we assume that the Department of Defense is just asking for money and risk losing the war? Or should we assume that the Department of Defense understands things well and take the risk of inflating the budget year after year?
This is the problem of 'expert capture' (expert capture). This always becomes a tricky issue. Usually, the best solution is to utilize an outsider who has some relevant expertise. For example, in the case of the U.S. Department of Defense, it would be good to hire advisors from neutral countries' military to evaluate whether it is better not to inflate defense spending. To assess whether macroeconomists are using the correct theories, it would be appropriate to have trade economists do it (Paul Krugman did exactly this). The same applies in other fields.
That being said, there is no systematic way to do this for the university system. As specialized differentiation continues and academic fields further divide into finer factions, it is increasingly important to develop countermeasures to periodically inspect whether researchers in each subfield are colluding to give each other accreditation stamps for useless garbage.
What happens when a certain research field becomes obsolete?
In an article written in 2021, I wrote a hypothesis that science might be a bit like searching for a gold vein:
In any field where discoveries and technological innovations are made, it's like a gold vein. The closest to the surface, the easiest to mine. The deeper you dig, the harder it gets to mine. However, occasionally, while doing so, you may stumble upon a new gold vein - then mining progresses rapidly and effortlessly again. But the direction of digging changes.
By following this kind of process, the progress of science resembles a series of S-shaped curves [repeating the acceleration and deceleration of discoveries and inventions]. The curves may partially overlap, but the aspects where discoveries are made vary from time to time.
As an example, the field of artificial intelligence was cited. Twenty-five years ago, artificial intelligence was a small corner of the field. Now, it is #expanding at an incredible pace.
The emergence of new fields is essential for the progress of science to be made in this way. When viewed over a sufficiently long span, progress in any specific field will inevitably slow down. This is because easily achievable discoveries and inventions are exhausted. The reason this 'machine' of [scientific research] continues to advance is solely due to the successive emergence of new fields.
However, what about the old field that has been completely mined out? Perhaps, if there is an efficient method, researchers should move from such fields to new ones. Some may welcome this transfer and enthusiastically embrace new challenges, while others may resist the need for change by fortifying the defensive positions mentioned in the previous section.
There seems to be another very powerful factor resisting these changes. That factor is the inertia of the system. With the tenure system in place, professors do not have to change their field if they do not want to. Academic journals in each field are likely to continue publishing endless papers on the same topic. Powerful factions within various university departments will continue to argue for the continued hiring of intellectual heirs to carry on their research.
In high-energy physics, physicist Sabine Hossenfelder believes that such processes are at work. According to her argument, physicists keep inventing theories predicting new undiscovered particles to justify government spending on ever-expanding particle accelerator facilities aimed at exploring these particles. Yet, no new particles have been found in these accelerators.
Some physicists are calling this the 'nightmare scenario'. This is because in this situation, there is no clear guide on where to go next. 'But if theory can explain all the experimental observations, where is the nightmare? Just declare victory and switch to working on quantum information theory, solid-state physics, or nuclear fusion?' In fact, many scholars have a strong desire to continue working on things they were not hired for, rather than what society needs most.
When it becomes difficult to bring a lower field to a close, it would be far more challenging to bring an entire department to a close. For example, let's assume that the entire field of sociology has been largely mined out - with most of the interesting insights and empirical discoveries already exhausted, and any remaining important questions could be tackled by relatively few empirical economists, anthropological ethnographers, or social psychologists. (To my sociologist colleagues, please understand that this is purely a hypothetical scenario before coming to burn down my house). In this case, how would the transition proceed? It would be necessary to convince the university administration that there is no longer any purpose for the sociology department as a whole. This would be an impossible task.
In addition, students will need to be convinced that 'there is no value in majoring in sociology.' In an article written in 2021, I introduced some research that young professors are actually engaging in signaling. It is a signaling to prove that they know enough about their research field to be entrusted with educating undergraduate students.
The number of professors to be hired for research is not determined by the actual need for research, but by the demand for undergraduate education. Furthermore, this demand varies by department - if there is an increase in undergraduate students majoring in economics, the university will likely hire more economics professors.
This means that the American university system - and roughly the university systems of other countries that mimic it - is essentially full of professors employed as teachers, but these professors prove their suitability for the professorship through research. They are pressured to publish papers or perish, regardless of whether they have anything interesting to publish. Additionally, academic journals are aware that driving researchers out of work would be detrimental to the academic system, so they help young researchers by publishing enough papers to be hired by departments with a national tenure track.
As mentioned in this article, this is becoming a further headwind for research productivity. This is because many ambitious professors are incentivized to conduct safe and incremental research. However, beyond that, there is a situation where undergraduate students' demand is influencing research efforts to some extent, much like 'the tail wagging the dog.' If a large number of students decide they want to study sociology, then many smart Americans will spend time researching sociology. Whether there is a real need for this is irrelevant. (Here, I would like to ask my fellow sociologists to understand that this is a fictional story)
In short, in a healthy academic industry, it would rapidly move towards new promising fields, and I don't think it would just keep increasing academic fields endlessly, believing that all fields will continue to grow forever.
Now, I don't think there is direct evidence that the academic industry is investing more resources in unproductive research pursuits, but on the other hand, there is evidence that the decline in research productivity is concerning, and there are clear signs that institutional incentives favor maintaining the status quo and gradual progress. Universities and the research conducted there are incredibly important for our country and future prosperity. However, instead of treating universities and research as sacred cows that are untouchable, there is a need to push and poke them to continue improving.
[Featured image: JF Martin on Unsplash] Noah Smith, “How much of modern academia is waste?” Noahpinion, January 7, 2024