Neuro-hype: the claim is in the headline. The proof is in the methods.

Neuroscience sells. Put a brain scan next to a claim and people will tend to believe it. Why? I'm still unsure, and I'm interested in figuring it out. Maybe because it's a complicated field, the study of the human brain. Or maybe because it carries credibility without questioning. Neuro-hype lives in this gap between what a headline says and what the study behind it can actually support.

In 2008, a study reported that articles with a brain image received higher ratings of scientific reasoning than the same articles with a bar graph or no image ^[1]. Five years later, ten replication experiments with more than 2,000 participants found that the image had little to no effect ^[2]. The words may matter more than the picture. Another 2008 study found that adding irrelevant neuroscience language made weak explanations more convincing to non-experts ^[3].

From 2018 to 2022 I analyzed clinical MRI for traumatic brain injury cases going for lawsuits. Each case asked a plain question: does this scan show injury? The answer rarely came from the image. It came from how the image was made. Which scanner, which sequence, how many diffusion directions, whether the person moved, how the data were cleaned, what they were compared against, how this comparison was made, and how it was interpreted. Change one of these factors, and a small finding can appear or disappear.

In this market where neuroscience claims sell because they sound smart and sexy, what we need more of is what I call applied neuroscience methodologists. Auditors of how the conclusion was reached, what the methods can and cannot tell you, and whether or not they support the claim and results.

Where results go wrong

Most errors in this field are not fraud. They are ordinary choices that nobody questioned.

Children move. In pediatric diffusion MRI, head motion can shift the same measures that are read as signs of white matter damage ^[4]. A study that compares restless children to still children can report a brain difference that is a motion difference.

Small samples produce big effects. The typical brain imaging study includes about 25 people. When researchers tested links between brain measures and behavior in data from around 50,000 people, they found the true effects are small, and that samples of a few dozen produce inflated results that fail to replicate. Reliable findings of this kind need thousands of participants ^[5]. A striking result from a small study is often striking because the study was small.

The same data can give different answers. Seventy research teams received the same brain scans and tested the same nine hypotheses. Their analysis choices differed, and so did their conclusions on several of the hypotheses ^[6].

How a claim leaves the lab

Brain and cognitive measures now leave the lab and enter settings where the people using them cannot verify them, or much more often, choose not to verify them.

Most people meet a finding through a press release, written by the university's press office, or a headline, written by a journalist, not through the paper written by the scientists. A study of 462 health press releases from 20 UK universities found that 40% gave more direct advice than a published paper (that the general public does not read) they described. 33% turned a correlation into causation, and 36% applied animal findings to humans ^[7].

And when a first finding is later overturned, the public rarely hears about it. Researchers took the ten ADHD studies that newspapers covered most in the 1990s and checked what later research found. Many of these first findings were later contradicted or shown to be weaker, yet newspapers almost never reported the studies that corrected them ^[8].

By the time a claim reaches a product, it can arrive with almost no evidence at all. Of 73 top-ranked mental health apps, 64% claimed to diagnose a condition or improve symptoms, mood or self-management. Only one of the 73 cited published research ^[9].

The money follows the claim, not the evidence.
Here are two examples, years apart.
In 1993, a study of 36 college students found that ten minutes of Mozart improved performance on one spatial reasoning task, for about ten to fifteen minutes ^[10]. In 1998, the governor of Georgia proposed $105,000 a year to give every newborn in the state a classical music CD ^[10]. A meta-analysis later found little evidence for any specific Mozart effect ^[11].
In 2016, the maker of the brain-training app Lumosity, backed by $67.5 million in venture funding, paid $2 million to settle Federal Trade Commission charges that it lacked the science for its claims that its games improved performance at work and school and could delay cognitive decline ^[12].

Put these together and a pattern appears. Small studies tend to overestimate effects ^[5], and first findings are often weakened or overturned by later work ^[8]. Yet the first findings are the ones that get covered, while the later studies that correct them rarely are ^[8]. On average, then, the results that reach the public are less reliable than the ones that stay in the journals.

Outside academia, those are the results that decisions get built on. Nothing in the result itself signals the problem: a motion artifact looks like a brain difference ^[4]. A result cannot tell you whether it is reliable. Only the way it was produced can. That is why the checking has to happen at the level of method, and before the decision rather than after it.

The incentives rarely push the other way. Under FDA policy, a low-risk product that claims to support focus, relaxation or stress management, without claiming to diagnose or treat a condition, can be sold as a general wellness product without FDA review ^[13].
Many consumer neurotechnology products are sold as wellness products, and investors have said publicly that they would find it hard to fund them if they required FDA approval ^[14]. The cheapest route to market is the one that needs the least evidence, and the money follows that route. And the little evidence we have gets overblown for marketing effect.

And so, this is the way it goes. An investor reads that a startup's brain wearable measures focus. A parent reads that an app will improve her child's attention. A school district reads that a reading program is based on brain science. Validated how, against which outcome, in which population, and with how many people?

Who asks these questions?

A public company cannot audit its own books. Before investors see the numbers, an independent firm checks them. Commercial neuroscience rarely works that way ^[14]. The same company can design the study, run it, analyze it and announce the result to sell the product.

And while every claim in this field arrives with a result, few arrive with their methods open to inspection. The question worth asking of each one is short: is this neuroscience claim defensible?