One view I hold - one I know many people here will be skeptical of - is that the future is partially predictable in a systematic way. Not in a deterministic or oracular sense, but in the limited, Tetlock-style sense of assigning calibrated probabilities to uncertain events and doing so better than baseline forecasters over time.
I’ve spent roughly the last 15 years trying to formalize and stress-test my own forecasting process. During that period, I’ve made public, timestamped predictions about events such as COVID, the Ukraine war, and various market movements. Some of these forecasts were wrong, some were directionally correct, and many were correct with meaningful lead time. Taken together, I think they at least suggest that forecasting can be treated as a learnable, improvable skill rather than an exercise in narrative hindsight.
When I’ve raised versions of this argument in the past (including in The Motte’s earlier Reddit incarnation), I’ve consistently encountered a few objections. I think these objections reflect reasonable priors, so I want to address them explicitly.
1 - “If prediction is possible, why aren’t the experts already doing it?”
My claim is not that expertise is useless, but that many expert institutions are poorly optimized for predictive accuracy. Incentives matter. Academia, media, and policy organizations tend to reward coherence, confidence, and alignment with prevailing narratives more than calibration or long-term scoring.
One reason I became interested in forecasting is that I appear to have unusually strong priors and pattern-recognition ability by objective measures. I’ve scored in the top 1% on multiple standardized exams (SAT, SHSAT, GMAT) on first attempts, which at least suggests above-average ability to reason under uncertainty and time pressure. That doesn’t make me infallible, but it does affect my prior that this might be a domain where individual skill differences matter.
Tetlock’s work also suggests that elite forecasting performance correlates less with formal credentials and more with specific cognitive habits: base-rate awareness, decomposition, active updating, and comfort expressing uncertainty numerically. These traits are not especially rewarded by most expert pipelines, which may explain why high-status experts often underperform trained forecasters.
My suspicion - very much a hypothesis, not a conclusion - is that many people in communities like this one are already better forecasters than credentialed experts, even if they don’t label what they’re doing as forecasting.
2 - “If you can forecast, why not just make money in markets?”
This is a fair question, since markets are one of the few environments where forecasts are continuously scored.
I have used forecasting methods in investing. Over the past five years, my average annual return has been approximately 40%, substantially outperforming major indices and comparable to or better than many elite hedge funds over the same period. This is net of mistakes, drawdowns, and revisions—not a cherry-picked subset.
That said, markets are noisy, capital-constrained, and adversarial. Forecasting ability helps, but translating probabilistic beliefs into portfolio construction, position sizing, and risk management is its own discipline. Forecasting is a necessary input, not a sufficient condition for success.
More importantly, I don’t think markets are the only - or even the most interesting - application. Forecasting is at least as relevant to geopolitics, institutional risk, public health, and personal decision-making, where feedback is slower but the stakes are often higher.
3 - “Where are the receipts?”
That’s a reasonable demand. I’ve tried to make at least some predictions public and timestamped so they can be evaluated ex ante rather than reconstructed after the fact.
Here are a few examples where I laid out forecasts and reasoning in advance:
https://questioner.substack.com/p/more-stock-advice
https://questioner.substack.com/p/superforecasting-for-dummies-9a5
I don’t claim these constitute definitive proof. At best, they are auditable data points that can be examined, criticized, or falsified.
What I’m Actually Interested in Discussing
I’m not asking anyone to defer to my forecasts, and I’m not claiming prediction is easy or universally applicable. What I am interested in is whether superforecasting should be treated as a legitimate applied discipline—and, if so:
Where does it work reliably, and where does it fail?
How should forecasting skill be evaluated outside of markets?
What selection effects or survivorship biases should we worry about?
Can forecasting methods be exploited or weaponized?
What institutional designs would actually reward calibration over narrative?
If your view is that forecasting success is mostly an artifact of hindsight bias or selective memory, I’d be genuinely interested in stress-testing that claim. Likewise, if you think forecasting works only in narrow domains, I’d like to understand where you’d draw those boundaries and why.
I’m less interested in persuading anyone than in subjecting the model itself to adversarial scrutiny. Looking forwards to hearing your thoughts.
Jump in the discussion.
No email address required.
Notes -
I'm fine with this interpretation, but realistically I don't think you can meaningfully operationalise it. Obviously a cafe owner can't be expected to include pandemics, nuclear war, general IA risks, alien invasions etc in their risk assessment. Sure, in an ideal world they would be able to click on a holistic risk portal website, punch in their circumstances and know that on aggregate they have a 20% chance of a life alterning goepolitical incident ruining their business in the next 30 years. But...
Nassim Taleb famously used the example of a airliner crashing into his building as an example of a black swan. This was pre-9/11. People immediately used this example as a rebuttal of his wider claims about black swans, as you could argue he had imagined exactly the kind world changing incident that was about to alter the course of US history, end the 1990s and shape global international relations for X years.
But there's a huge difference in functional use of these predictions and simply saying the words. Approximately zero cafe owners around the world had a "pandemic survival fund" in preparation for COVID 19. So it's fair to say that the industry as a whole, functionally, had no sense that this was a danger. Therefore it was a black swan to them, even though we have high profile guys saying "this is a possibility."
You have a fundamental problem that most intelligence analysts will describe on excruciating detail: we looked at this problem, we knew it was a possibility, but the operational team responsible for the investigation/operation/follow up didn't take the problem seriously. In a world with limited resources, we really can't expect ops teams to chase up every low likelihood problem. If a steel mill in Vietnam blows up tomorrow, the manufacturer that makes my dad's decorative embellishments for his roulette wheel company is going to raise prices. My dad doesn't really run any analysis on the health and safety standards of a random steel mill in Vietnam. But he does look into e.g. risks to the gambling industry where his products are sold. There's a level of reasonableness that we can expect, and then a level that is so far flung we just can't. Even if in hindsight it is clear that my dad's business hinges on a Vietnamese steel mill checking their power sockets every three months.
We live in a world with an infinity of risks. And every risk has another associated risk. With limited resources these risks quickly become black swans. Peter Theil might be the type of guy with the resources to analyse a lot of these and prepare a subterranean refuge under a New Zealand mountain, but for the rest of us, they're functionally black swans.
I'm even at odds with multiple federal and congressional investigations re: 9/11. My opinion is that they could not have possibly predicted 9/11 with the information they had. They would, by definition, be wrong to have predicted planes flying into the trade towers as the most likely scenario. Far more likely was a hijacking or a hostage scenario. This goes back to the limitations of reference class forecasting, as discussed. We can enumerate flood and fire risks and insure ourselves for the eventuality. But if an asteroid hit a major city, it's a black swan despite some astro physicist having warned us for the last 10 years that we're under prepared for this eventuality.
The point is that there's two problems with forecasting the future. Too little information means we can't form an accurate baseline, even if we know to ask the question to begin with. And too much information which results in us being functionally incapable of doing analysis on it due to limited resources.
Tetlock really focuses on accurately forecasting e.g. flood risks. Taleb focuses on the asteroid hitting the middle of NYC. These concepts both dovetail very nicely and both need to be understood by super forecasters or professional analysts.
It does and I agree with what you're saying overall.
I think identifying intelligence gaps is a massively under explored area. Like extremely under explored, and frighteningly so. As a field, we are very good at counting tanks, tracking submarines, analysing how to win battles. And as a field we're pretty good at filling existing information or intelligence gaps. Anybody can send a spy into an economic forum to take a picture of a tank through their pocket.
But nobody I know of is developing that unknown/unknown question, to determine which intelligence gaps we're not aware of. Intelligence is very reactive to new problems, but not very proactive in getting ahead of them before they become a problem. This is bad because those unidentified intelligence problems have a disproportionate effect on world events.
Ponderously telling people that there's a 5% chance of China invading Taiwan is useful, but right now the entire Danish political corps is chain smoking outside the government offices in Denmark (https://youtube.com/shorts/L-Rr9F9g_VM) because they weren't positioned for this possibility by their intelligence services. This is essentially a black swan by all meaningful definitions.
More options
Context Copy link