"Evidence-first" only means something if we're honest about a genuinely uncomfortable finding: expecting a supplement to work changes how it feels to work, sometimes almost as much as the ingredient itself does. This isn't a reason to distrust every positive experience with a product — but understanding the mechanism makes you a much harder person to market to. It is also the argument underneath everything else we publish: when we say a finding is weak, this is why.
The experiment that makes this concrete
A study published in Frontiers in Psychiatry gave participants an identical inert nasal spray, but told one group it was a cognitive enhancer and another group it was a cognitive suppressant.1 Both groups got exactly the same substance. The group told it was an enhancer rated their own performance significantly higher (a large effect) and reported feeling notably less tired than the group told it was a suppressant. But on the actual standardized attention tests — reaction time, accuracy — there was no measurable difference between the groups at all.
Same substance, opposite instructions: the group told it was an enhancer felt sharper and less tired. Their actual test scores didn't move at all.
Subjective outcomes move. Objective ones often don't.
That split isn't a quirk of one study. It is the central result of the largest systematic attempt to measure placebo effects at all. Hróbjartsson and Gøtzsche's Cochrane review compared placebo against no treatment — not against a drug — across 202 trials with extractable data.2 For binary outcomes, across 44 trials and 6,041 patients, the pooled relative risk was 0.93 (95% CI 0.88 to 0.99): barely anything. For continuous outcomes, across 158 trials and 10,525 patients, the standardised mean difference was −0.23 (95% CI −0.28 to −0.17) — a small effect.
Then the crucial breakdown. Patient-reported continuous outcomes moved by −0.26; observer-reported outcomes moved by −0.13, half as much. Pain moved by −0.28, but that figure was driven substantially by a cluster of German acupuncture trials, with other pain trials showing negligible effects. The reviewers' own conclusion is the sentence worth memorising: it is difficult to distinguish patient-reported effects of placebo from biased reporting.
This is why we treat "participants felt better" and "participants performed better" as different classes of evidence, and why a product whose case rests on self-reported energy or focus scores gets discounted here. The same discount applies to the phone-and-attention literature in what screen time actually does to focus.
Open-label placebo: it still works when you're told it's a placebo
The strangest finding in this field is that deception doesn't appear to be required. In a 2010 randomised trial, 80 patients with irritable bowel syndrome were assigned either to no treatment or to placebo pills explicitly and repeatedly described as "placebo pills made of an inert substance, like sugar pills" — delivered with a roughly 15-minute script explaining that placebo effects are powerful and that taking the pills faithfully matters.3 After three weeks, the open-label placebo group scored 5.0 ± 1.5 on the IBS Global Improvement Scale against 3.9 ± 1.3 for controls (p = 0.002, d = 0.79).
A 2021 meta-analysis pooled the field: 13 studies and 834 participants in the review, 11 studies and 654 participants in the meta-analysis, across back pain, cancer-related fatigue, ADHD, allergic rhinitis, depression, IBS and menopausal hot flushes.4 The pooled standardised mean difference was 0.72 (95% CI 0.39 to 1.05); restricted to lower-risk studies it fell to 0.49 (0.32 to 0.66).
Now the part that is usually left out. 31% of those studies were rated at high risk of bias and the other 69% raised "some concerns" — which is all of them. Samples were small and follow-up short. Every primary outcome was self-reported. Exactly one study measured an objective outcome, and that result was not significant. The funnel plot was asymmetric. The IBS authors' own caveats were similar: a proof-of-principle pilot, patients recruited because they were drawn to a mind-body study, and a no-treatment arm who may simply have been disappointed. Open-label placebo is a real and fascinating effect on how people rate their symptoms. There is still essentially no evidence that it changes anything you can measure without asking.
Two things that aren't placebo, and get mistaken for it
Practice effects. Cognitive tests get easier the second time you sit them, and this has been quantified precisely. A meta-analysis of 122 studies, 174 samples and 153,185 participants found a mean retest gain of 0.33 standard deviations from the first to the second administration, rising to 0.50 by the third, with no further gains after that.5 Sit still with that number: 0.33 is larger than the effect most nootropic ingredients report in properly controlled trials. Any before-and-after study without a control group will manufacture an improvement of roughly that size out of nothing but the retest.
Regression to the mean. People start a supplement at their worst — the week the brain fog is bad, the month sleep collapsed. Symptoms that fluctuate tend to drift back toward their own average whether or not you intervene, and whoever intervened gets the credit. A bad week followed by a better week is not evidence of anything, which is why our write-ups insist on design rather than anecdote: see what a new magnesium and sleep trial actually found.
Why a placebo arm is the whole argument
Put those four together — expectation, biased self-report, practice effects, regression to the mean — and a single-arm supplement study cannot tell you anything. A control group taking an identical-looking capsule subtracts all four at once, which is precisely why the arm exists, and why a trial without one is not weak evidence but closer to no evidence.
Two consequences follow. First: a testimonial cannot clear this bar, and neither can your own experience, however genuine. Second: when a properly controlled trial is finally run on a popular ingredient, it often fails. The Ginkgo Evaluation of Memory study — the largest trial ever run on ginkgo, following older adults over years on 240mg a day — found it ineffective at reducing dementia incidence.6 Ginkgo had decades of enthusiastic uncontrolled literature behind it.
The publication and funding picture compounds this. An analysis of 206 nutrition publications found the odds of a favourable conclusion were four to eight times higher when the study was funded by the relevant manufacturer; among interventional studies with industry as sole sponsor, 16 of 16 reported favourable outcomes, against 12 of 19 where funding was mixed or absent.7 The author notes that association isn't proof of bias. It is still the best reason to check who paid before you read the abstract. Our working rules for that are in how to read a supplement label and what a certificate of analysis actually tells you.
What this means for how you use a supplement
It doesn't mean your experience with a product is fake or meaningless — feeling sharper is a real, valid outcome even when part of it is expectation-driven. It does mean "I felt something within a day" isn't evidence for ingredients like Bacopa monnieri, where every trial in the pooled analysis required at least 12 weeks of continuous use before testing. Some things in this category do act same-day and have the trial base to say so — the caffeine and L-theanine combination is the clearest example. The distinction between those two cases is the whole point.
Practically: judge a formula over weeks rather than days, expect the honest effect to be modest, and be suspicious of any product whose pitch is same-day testimonials rather than a citable placebo-controlled trial. Hold us to it too — Cochrane found omega-3 moved depression scores by about 2.5 points, short of the 3-point shift researchers treat as clinically meaningful, and we print that rather than the friendlier reviews. The standard we apply before stocking anything is in why we don't sell "brain blends", and where the best-evidenced answer isn't a supplement at all we say so — as in CBT-I.
References
- Faasse K, et al. "Placebo- and Nocebo-Effects in Cognitive Neuroenhancement: When Expectation Shapes Perception." Frontiers in Psychiatry, 2019. frontiersin.org
- Hróbjartsson A, Gøtzsche PC. "Placebo interventions for all clinical conditions." Cochrane Database of Systematic Reviews, 2010;(1):CD003974. 202 trials with extractable data. cochranelibrary.com
- Kaptchuk TJ, et al. "Placebos without Deception: A Randomized Controlled Trial in Irritable Bowel Syndrome." PLoS ONE, 2010;5(12):e15591. n = 80. journals.plos.org
- von Wernsdorff M, Loef M, Tuschen-Caffier B, Schmidt S. "Effects of open-label placebos in clinical trials: a systematic review and meta-analysis." Scientific Reports, 2021. nature.com
- Scharfen J, Peters JM, Holling H. "Retest effects in cognitive ability tests: A meta-analysis." Intelligence, 2018;67:44-66. 122 studies, 153,185 participants. sciencedirect.com
- "The Ginkgo Evaluation of Memory (GEM) Study." National Center for Complementary and Integrative Health. nccih.nih.gov
- Lesser LI, Ebbeling CB, Goozner M, Wypij D, Ludwig DS. "Relationship between Funding Source and Conclusion among Nutrition-Related Scientific Articles." PLoS Medicine, 2007;4(1):e5. 206 publications. journals.plos.org
This article is for informational purposes only and has not been evaluated by the FDA. It is not intended to diagnose, treat, cure, or prevent any disease.
Related reading: How to Read a Supplement Label: A Practical Guide to Proprietary Blends and Underdosing
See the full evidence behind every Proco ingredient at procohq.com.