One view I hold - one I know many people here will be skeptical of - is that the future is partially predictable in a systematic way. Not in a deterministic or oracular sense, but in the limited, Tetlock-style sense of assigning calibrated probabilities to uncertain events and doing so better than baseline forecasters over time.
I’ve spent roughly the last 15 years trying to formalize and stress-test my own forecasting process. During that period, I’ve made public, timestamped predictions about events such as COVID, the Ukraine war, and various market movements. Some of these forecasts were wrong, some were directionally correct, and many were correct with meaningful lead time. Taken together, I think they at least suggest that forecasting can be treated as a learnable, improvable skill rather than an exercise in narrative hindsight.
When I’ve raised versions of this argument in the past (including in The Motte’s earlier Reddit incarnation), I’ve consistently encountered a few objections. I think these objections reflect reasonable priors, so I want to address them explicitly.
1 - “If prediction is possible, why aren’t the experts already doing it?”
My claim is not that expertise is useless, but that many expert institutions are poorly optimized for predictive accuracy. Incentives matter. Academia, media, and policy organizations tend to reward coherence, confidence, and alignment with prevailing narratives more than calibration or long-term scoring.
One reason I became interested in forecasting is that I appear to have unusually strong priors and pattern-recognition ability by objective measures. I’ve scored in the top 1% on multiple standardized exams (SAT, SHSAT, GMAT) on first attempts, which at least suggests above-average ability to reason under uncertainty and time pressure. That doesn’t make me infallible, but it does affect my prior that this might be a domain where individual skill differences matter.
Tetlock’s work also suggests that elite forecasting performance correlates less with formal credentials and more with specific cognitive habits: base-rate awareness, decomposition, active updating, and comfort expressing uncertainty numerically. These traits are not especially rewarded by most expert pipelines, which may explain why high-status experts often underperform trained forecasters.
My suspicion - very much a hypothesis, not a conclusion - is that many people in communities like this one are already better forecasters than credentialed experts, even if they don’t label what they’re doing as forecasting.
2 - “If you can forecast, why not just make money in markets?”
This is a fair question, since markets are one of the few environments where forecasts are continuously scored.
I have used forecasting methods in investing. Over the past five years, my average annual return has been approximately 40%, substantially outperforming major indices and comparable to or better than many elite hedge funds over the same period. This is net of mistakes, drawdowns, and revisions—not a cherry-picked subset.
That said, markets are noisy, capital-constrained, and adversarial. Forecasting ability helps, but translating probabilistic beliefs into portfolio construction, position sizing, and risk management is its own discipline. Forecasting is a necessary input, not a sufficient condition for success.
More importantly, I don’t think markets are the only - or even the most interesting - application. Forecasting is at least as relevant to geopolitics, institutional risk, public health, and personal decision-making, where feedback is slower but the stakes are often higher.
3 - “Where are the receipts?”
That’s a reasonable demand. I’ve tried to make at least some predictions public and timestamped so they can be evaluated ex ante rather than reconstructed after the fact.
Here are a few examples where I laid out forecasts and reasoning in advance:
https://questioner.substack.com/p/more-stock-advice
https://questioner.substack.com/p/superforecasting-for-dummies-9a5
I don’t claim these constitute definitive proof. At best, they are auditable data points that can be examined, criticized, or falsified.
What I’m Actually Interested in Discussing
I’m not asking anyone to defer to my forecasts, and I’m not claiming prediction is easy or universally applicable. What I am interested in is whether superforecasting should be treated as a legitimate applied discipline—and, if so:
Where does it work reliably, and where does it fail?
How should forecasting skill be evaluated outside of markets?
What selection effects or survivorship biases should we worry about?
Can forecasting methods be exploited or weaponized?
What institutional designs would actually reward calibration over narrative?
If your view is that forecasting success is mostly an artifact of hindsight bias or selective memory, I’d be genuinely interested in stress-testing that claim. Likewise, if you think forecasting works only in narrow domains, I’d like to understand where you’d draw those boundaries and why.
I’m less interested in persuading anyone than in subjecting the model itself to adversarial scrutiny. Looking forwards to hearing your thoughts.
Jump in the discussion.
No email address required.
Notes -
This is an excellent comment, and I largely agree with your taxonomy and framing. In particular, I think you’re exactly right that reference-class forecasting shines most when you have (a) stable baselines and (b) a well-posed question to begin with. Your distinction between known unknowns and unknown unknowns maps very cleanly onto where forecasting techniques feel powerful versus where they feel brittle in practice.
Your intelligence-analysis perspective also rings true to me. Using the outside view as a stabilizer against excited inside-view narratives is, in my experience, one of the highest-leverage applications of forecasting. In most real-world settings, the dominant failure mode isn’t underreaction but overreaction to new, salient information, and reference classes are a very effective corrective.
Where I’d push back slightly—and I mean this as a nuance rather than a rejection—is on COVID as an example of a true black swan in the Taleb sense.
I agree completely with your café-owner framing: for many individuals, COVID was effectively unaskable ex ante, and therefore indistinguishable from an unknown unknown. At the decision-maker level, it absolutely behaved like a black swan. That’s an important and underappreciated point.
However, at the system level, I’m less convinced it was unforeseeable. A number of people did, in fact, raise the specific risk in advance:
Bill Gates publicly warned in 2015 that global pandemic preparedness was dangerously inadequate and that a fast-moving virus was a more realistic threat than many conventional disaster scenarios.
The Wuhan Institute of Virology had been criticized multiple times prior to 2020 for operating at biosafety levels below what many thought appropriate for the research being conducted.
More broadly, pandemic risk had a nontrivial base rate in the epidemiology and biosecurity literature, even if the exact trigger and timing were unknown.
On a more personal note (and not meant as special pleading), I discussed viral and memetic contagion risks repeatedly in The Dark Arts of Rationality: Updated for the Digital Age, which was printed several months before COVID.
All of which is to say: COVID may not have been a black swan so much as a gray rhino—a high-impact risk that was visible to some, articulated by a few, but ignored by most institutions and individuals because it didn’t map cleanly onto their local decision models.
I think this distinction matters for forecasting as a discipline. It suggests that one of the core failures isn’t predictive ability per se, but attention allocation: which warnings get surfaced, amplified, and translated into actionable questions for the people whose decisions hinge on them. In that sense, I think you’re exactly right that Tetlock’s next frontier—teaching people how to ask better questions—is the crux.
So I’d summarize my position as: Forecasting works best in domains with history and well-posed questions, struggles at the edges, and fails catastrophically when important questions never get asked. But some events we label “unpredictable” may actually be predictable but institutionally invisible—which is a slightly different (and potentially more tractable) failure mode.
Curious whether that distinction resonates with your experience in intelligence work, or if you think I’m still underestimating the true weight of the unknown-unknown problem.
I'm fine with this interpretation, but realistically I don't think you can meaningfully operationalise it. Obviously a cafe owner can't be expected to include pandemics, nuclear war, general IA risks, alien invasions etc in their risk assessment. Sure, in an ideal world they would be able to click on a holistic risk portal website, punch in their circumstances and know that on aggregate they have a 20% chance of a life alterning goepolitical incident ruining their business in the next 30 years. But...
Nassim Taleb famously used the example of a airliner crashing into his building as an example of a black swan. This was pre-9/11. People immediately used this example as a rebuttal of his wider claims about black swans, as you could argue he had imagined exactly the kind world changing incident that was about to alter the course of US history, end the 1990s and shape global international relations for X years.
But there's a huge difference in functional use of these predictions and simply saying the words. Approximately zero cafe owners around the world had a "pandemic survival fund" in preparation for COVID 19. So it's fair to say that the industry as a whole, functionally, had no sense that this was a danger. Therefore it was a black swan to them, even though we have high profile guys saying "this is a possibility."
You have a fundamental problem that most intelligence analysts will describe on excruciating detail: we looked at this problem, we knew it was a possibility, but the operational team responsible for the investigation/operation/follow up didn't take the problem seriously. In a world with limited resources, we really can't expect ops teams to chase up every low likelihood problem. If a steel mill in Vietnam blows up tomorrow, the manufacturer that makes my dad's decorative embellishments for his roulette wheel company is going to raise prices. My dad doesn't really run any analysis on the health and safety standards of a random steel mill in Vietnam. But he does look into e.g. risks to the gambling industry where his products are sold. There's a level of reasonableness that we can expect, and then a level that is so far flung we just can't. Even if in hindsight it is clear that my dad's business hinges on a Vietnamese steel mill checking their power sockets every three months.
We live in a world with an infinity of risks. And every risk has another associated risk. With limited resources these risks quickly become black swans. Peter Theil might be the type of guy with the resources to analyse a lot of these and prepare a subterranean refuge under a New Zealand mountain, but for the rest of us, they're functionally black swans.
I'm even at odds with multiple federal and congressional investigations re: 9/11. My opinion is that they could not have possibly predicted 9/11 with the information they had. They would, by definition, be wrong to have predicted planes flying into the trade towers as the most likely scenario. Far more likely was a hijacking or a hostage scenario. This goes back to the limitations of reference class forecasting, as discussed. We can enumerate flood and fire risks and insure ourselves for the eventuality. But if an asteroid hit a major city, it's a black swan despite some astro physicist having warned us for the last 10 years that we're under prepared for this eventuality.
The point is that there's two problems with forecasting the future. Too little information means we can't form an accurate baseline, even if we know to ask the question to begin with. And too much information which results in us being functionally incapable of doing analysis on it due to limited resources.
Tetlock really focuses on accurately forecasting e.g. flood risks. Taleb focuses on the asteroid hitting the middle of NYC. These concepts both dovetail very nicely and both need to be understood by super forecasters or professional analysts.
It does and I agree with what you're saying overall.
I think identifying intelligence gaps is a massively under explored area. Like extremely under explored, and frighteningly so. As a field, we are very good at counting tanks, tracking submarines, analysing how to win battles. And as a field we're pretty good at filling existing information or intelligence gaps. Anybody can send a spy into an economic forum to take a picture of a tank through their pocket.
But nobody I know of is developing that unknown/unknown question, to determine which intelligence gaps we're not aware of. Intelligence is very reactive to new problems, but not very proactive in getting ahead of them before they become a problem. This is bad because those unidentified intelligence problems have a disproportionate effect on world events.
Ponderously telling people that there's a 5% chance of China invading Taiwan is useful, but right now the entire Danish political corps is chain smoking outside the government offices in Denmark (https://youtube.com/shorts/L-Rr9F9g_VM) because they weren't positioned for this possibility by their intelligence services. This is essentially a black swan by all meaningful definitions.
More options
Context Copy link
Cov19 the disease was eminently predictable -- we have novel seasonal respiratory virus outbreaks all the time, something like a 10%/a prediction would be not too bad.
The Cov19 response (which is the risk that bit cafe buyers' collective ass) would be pretty hard to predict, given that it was completely unimaginable in 2019 -- maybe a generalized "massive four horseman-related social disruption in my area" @ 1%/a would work, but this doesn't exactly seem like what we are after when talking about forecasting skillz.
More options
Context Copy link
More options
Context Copy link