One view I hold - one I know many people here will be skeptical of - is that the future is partially predictable in a systematic way. Not in a deterministic or oracular sense, but in the limited, Tetlock-style sense of assigning calibrated probabilities to uncertain events and doing so better than baseline forecasters over time.
I’ve spent roughly the last 15 years trying to formalize and stress-test my own forecasting process. During that period, I’ve made public, timestamped predictions about events such as COVID, the Ukraine war, and various market movements. Some of these forecasts were wrong, some were directionally correct, and many were correct with meaningful lead time. Taken together, I think they at least suggest that forecasting can be treated as a learnable, improvable skill rather than an exercise in narrative hindsight.
When I’ve raised versions of this argument in the past (including in The Motte’s earlier Reddit incarnation), I’ve consistently encountered a few objections. I think these objections reflect reasonable priors, so I want to address them explicitly.
1 - “If prediction is possible, why aren’t the experts already doing it?”
My claim is not that expertise is useless, but that many expert institutions are poorly optimized for predictive accuracy. Incentives matter. Academia, media, and policy organizations tend to reward coherence, confidence, and alignment with prevailing narratives more than calibration or long-term scoring.
One reason I became interested in forecasting is that I appear to have unusually strong priors and pattern-recognition ability by objective measures. I’ve scored in the top 1% on multiple standardized exams (SAT, SHSAT, GMAT) on first attempts, which at least suggests above-average ability to reason under uncertainty and time pressure. That doesn’t make me infallible, but it does affect my prior that this might be a domain where individual skill differences matter.
Tetlock’s work also suggests that elite forecasting performance correlates less with formal credentials and more with specific cognitive habits: base-rate awareness, decomposition, active updating, and comfort expressing uncertainty numerically. These traits are not especially rewarded by most expert pipelines, which may explain why high-status experts often underperform trained forecasters.
My suspicion - very much a hypothesis, not a conclusion - is that many people in communities like this one are already better forecasters than credentialed experts, even if they don’t label what they’re doing as forecasting.
2 - “If you can forecast, why not just make money in markets?”
This is a fair question, since markets are one of the few environments where forecasts are continuously scored.
I have used forecasting methods in investing. Over the past five years, my average annual return has been approximately 40%, substantially outperforming major indices and comparable to or better than many elite hedge funds over the same period. This is net of mistakes, drawdowns, and revisions—not a cherry-picked subset.
That said, markets are noisy, capital-constrained, and adversarial. Forecasting ability helps, but translating probabilistic beliefs into portfolio construction, position sizing, and risk management is its own discipline. Forecasting is a necessary input, not a sufficient condition for success.
More importantly, I don’t think markets are the only - or even the most interesting - application. Forecasting is at least as relevant to geopolitics, institutional risk, public health, and personal decision-making, where feedback is slower but the stakes are often higher.
3 - “Where are the receipts?”
That’s a reasonable demand. I’ve tried to make at least some predictions public and timestamped so they can be evaluated ex ante rather than reconstructed after the fact.
Here are a few examples where I laid out forecasts and reasoning in advance:
https://questioner.substack.com/p/more-stock-advice
https://questioner.substack.com/p/superforecasting-for-dummies-9a5
I don’t claim these constitute definitive proof. At best, they are auditable data points that can be examined, criticized, or falsified.
What I’m Actually Interested in Discussing
I’m not asking anyone to defer to my forecasts, and I’m not claiming prediction is easy or universally applicable. What I am interested in is whether superforecasting should be treated as a legitimate applied discipline—and, if so:
Where does it work reliably, and where does it fail?
How should forecasting skill be evaluated outside of markets?
What selection effects or survivorship biases should we worry about?
Can forecasting methods be exploited or weaponized?
What institutional designs would actually reward calibration over narrative?
If your view is that forecasting success is mostly an artifact of hindsight bias or selective memory, I’d be genuinely interested in stress-testing that claim. Likewise, if you think forecasting works only in narrow domains, I’d like to understand where you’d draw those boundaries and why.
I’m less interested in persuading anyone than in subjecting the model itself to adversarial scrutiny. Looking forwards to hearing your thoughts.
Jump in the discussion.
No email address required.
Notes -
I would be surprise if anyone here doesn’t believe that forecasting is possible. My gut says there would be nearly 100% agreement that psychohistory (Foundation Series) is most likely true.
I would assume 50% of posters here could generate 40% annual returns on modest amounts of capital if they focused on earning their living in trading. It’s not an impressive number especially when you made zero mention of your risks system.
Jane Street literally recruited straight out of SSC message boards. And they do like $25 billion a year in pnl now (far less when they began recruiting from SSC).
More options
Context Copy link
I can reliably say that personal vehicles will be an important part of society in 2080 with a high degree of confidence. This is the type of question Tetlock/Kahneman's reference class forecasting does well to predict.
You also have the Nate Silver application of categorising and data analyses. So in a data rich environment (sports, weather, crime) you can generate very accurate assessments about the future.
But you also have the Nasim Taleb category of forecasting. E.g. if you can think to ask the question in the first place, it's probably not that useful. He empahsises the importance of the unknown unknowns and thinks we over invest in facts that are obvious about the future.
Said another way, we have known knowns (facts), known unknowns (intelligence gaps) and unknown unknowns (intelligence gaps we haven't identified yet). We can do a good job talking about the first two, and determine a base rate for whether a nuclear bomb is going to be detonated in the next 10 years. We do a very bad job at answering the questions we don't know we need to ask, for obvious reasons.
Tetlock has brought this up too. He says the next major research question is how to teach people to ask good questions to begin with. He doesn't use the intelligence gap terminology, but that's what professional intelligence analysts interpret this point as.
To put a point on this question you kind of need to understand the problem with identifying these intelligence questions to begin with. In 2019, probably a lot of experts on diseases and pandemics would have rated the possibility of a global outbreak quite highly in relative terms. By 2021 they would have known their predictions had been correct about the standing risk. But if I was opening a cafe in 2019, I would not have been able to even ask the question about a global pandemic which would kill my business in 2 years. I did not have the information required to know what i didn't know, and all of the business analysis I did was useless as I saw the city shut down society for months at a time.
The main gap in the reference class forecasting technique is that foundational problem. Knowing which questions are relevant to me in an environment of incomplete information.
It's one of the best tools we have. I've worked in intelligence for 15 years and can consistently give intuitive reactions just by having reference class forecasting in the back of my mind. A question will be thrown at us in a meeting and I intuitively stabilise thr inside view against a baseline outside view. Most of the time, in most applications, the answer is to moderate the claims of people who are getting excited about all this new information coming in. You can do this with Iran today, you can do it with the ICE raids, you can do it with the likely GDP per annum.
But it is much harder to be pointed at extreme edge cases. Tetlock resists that his technique is better used in environments with a lot of history to form the baseline. But in practical applications, I've found this to be the case.
Again, I dont think this is too much of a flaw. The real problem is the initial questions. If there's something really important that's going to happen next year, but nobody has the method or ability to identify that there are quesrions about it that need to be asked, the technique is useless. Everybody can argue over the arrival date of general AI milestones. But it is incredibly difficult to identify black swans ahead of time. And I think most important, world changing things hit people more like cafe owners in 2019 who have no idea their whole business hinges on this lab in China not fucking up one day.
This is an excellent comment, and I largely agree with your taxonomy and framing. In particular, I think you’re exactly right that reference-class forecasting shines most when you have (a) stable baselines and (b) a well-posed question to begin with. Your distinction between known unknowns and unknown unknowns maps very cleanly onto where forecasting techniques feel powerful versus where they feel brittle in practice.
Your intelligence-analysis perspective also rings true to me. Using the outside view as a stabilizer against excited inside-view narratives is, in my experience, one of the highest-leverage applications of forecasting. In most real-world settings, the dominant failure mode isn’t underreaction but overreaction to new, salient information, and reference classes are a very effective corrective.
Where I’d push back slightly—and I mean this as a nuance rather than a rejection—is on COVID as an example of a true black swan in the Taleb sense.
I agree completely with your café-owner framing: for many individuals, COVID was effectively unaskable ex ante, and therefore indistinguishable from an unknown unknown. At the decision-maker level, it absolutely behaved like a black swan. That’s an important and underappreciated point.
However, at the system level, I’m less convinced it was unforeseeable. A number of people did, in fact, raise the specific risk in advance:
Bill Gates publicly warned in 2015 that global pandemic preparedness was dangerously inadequate and that a fast-moving virus was a more realistic threat than many conventional disaster scenarios.
The Wuhan Institute of Virology had been criticized multiple times prior to 2020 for operating at biosafety levels below what many thought appropriate for the research being conducted.
More broadly, pandemic risk had a nontrivial base rate in the epidemiology and biosecurity literature, even if the exact trigger and timing were unknown.
On a more personal note (and not meant as special pleading), I discussed viral and memetic contagion risks repeatedly in The Dark Arts of Rationality: Updated for the Digital Age, which was printed several months before COVID.
All of which is to say: COVID may not have been a black swan so much as a gray rhino—a high-impact risk that was visible to some, articulated by a few, but ignored by most institutions and individuals because it didn’t map cleanly onto their local decision models.
I think this distinction matters for forecasting as a discipline. It suggests that one of the core failures isn’t predictive ability per se, but attention allocation: which warnings get surfaced, amplified, and translated into actionable questions for the people whose decisions hinge on them. In that sense, I think you’re exactly right that Tetlock’s next frontier—teaching people how to ask better questions—is the crux.
So I’d summarize my position as: Forecasting works best in domains with history and well-posed questions, struggles at the edges, and fails catastrophically when important questions never get asked. But some events we label “unpredictable” may actually be predictable but institutionally invisible—which is a slightly different (and potentially more tractable) failure mode.
Curious whether that distinction resonates with your experience in intelligence work, or if you think I’m still underestimating the true weight of the unknown-unknown problem.
More options
Context Copy link
More options
Context Copy link
I will answer your questions with two questions of my own. (The questions are semi-rhetorical in that I think they shed light on the answers to your questions, but also I would genuinely really like answers and I haven't seen any good answers.)
Theoretical Q.
I overall like the forecasting trend in the rationalist community. I find the idea of quantifying bias and uncertainty to be a valuable exercise that I have benefited from personally. I have a theory-level concern, however, that I've never seen properly addressed.
I internally model forecasting as: there exists a probability distribution over all possible futures, and the job of the forecaster is to approximate this distribution. In practice, forecasters do this by assigning probabilities to a bunch of events and then scoring themselves based on what actually happens (like you describe in your OP).
So here's my question: How confident can we actually be that your scoring algorithms are stable and consistent? I'm using these words in the technical sense from statistics. To see an example of how everything can go bad: Let's say you're trying to predict the number of people who die in 2026. If the true distribution of deaths/year is gaussian, you can use standard formulas for computing the mean and get a good estimate with error bars. But if the true distribution is Cauchy, the mean is undefined, and there is provably no way to accurately estimate this mean because it doesn't exist. The Cauchy distribution looks essentially identical to the Gaussian distribution, and it is extremely difficult to determine whether you are actually sampling from one or the other in practice. In practice, people who work under the Gaussian assumption will look like they're doing very well by the metrics superforecasters use until suddenly they have a disaster (see e.g. the 2008 financial collapse). Similarly, a 40% return over 5 years is "trivial" to achieve if you allow yourself to have a very high risk of ruin. Just invest in the S&P500 with 5x leverage.
So what are the actual, philosophical and statistical assumptions about the universe that superforecasters are relying on?
Practical Q
I work professionally with North Korea. I put in a lot of time studying their culture, geopolitics, language, etc in order to make my professional work more effective. I've long thought about how to quantify this work both to make my work even more effective and to convince other people that I am an expert on this topic. How do I go about as a practical matter starting to forecast on a very niche topic like this?
My impression is that most forecasters work very generally and basically try to eek out an edge over the general populace by (like you mention) not being fooled by basic statistical fallacies. This lets forecasters make more level-headed judgements about a wide range of topics, most of which are well-established questions that normies also think about (who will win the election? will an epidemic cause a downturn in the economy? etc.)
But I am interested only in a very narrow domain where there are basically no established questions to ask. With regards to North Korea, the basic questions might be:
But these are all super basic questions that anyone moderately politically aware could reasonably answer. There's no opportunity for me to develop my skill with questions like this, and there's not a "large enough n" for me to meaningfully test my skill. So I need to develop more detailed questions if I want to really improve my forecasting ability. But how? Some more detailed questions could be:
But how do I go about actually creating good questions like this? You especially want the questions to be correlated with the "basic"/important questions above, but it's not at all clear to me that the ability to predict food prices is at all related to the ability to predict whether and how large of a military conflict there will be.
One last aside: You don't mention the intelligence community at all. This is where calibrated predictions are rewarded more than narrative, and this is where people who actually want to work as superforecasters work. Some "fun" reading if you haven't already seen them are:
"Psychology of Intelligence Analysis"
"A Tradecraft Primer: Structured Analytic Techniques for Improving Intelligence Analysis"
"Analytic Culture in the U.S. Intelligence Community"
These are all declassified CIA publications you can get from cia.gov. Most of my questions/frustrations expressed above are things that I've thought about from reading these works and talking to the people who use them professionally.
This is a very thoughtful comment—thank you for taking the time to lay it out so clearly. Also, thanks for the reading recommendations; I’m familiar with Psychology of Intelligence Analysis, but I haven’t read all three you listed, and I appreciate the pointers. The intelligence-community framing is very much adjacent to how I think about this problem.
Let me try to respond to both the theoretical and practical questions in turn.
Theoretical question: what assumptions are superforecasters actually making?
I think your concern is a real one, and I don’t think there’s a fully satisfying, formally rigorous answer yet.
You’re right that most forecasting implicitly assumes something like: there exists a stable-enough probability distribution over futures that can be approximated and scored. And you’re also right that if the underlying distribution is heavy-tailed, discontinuous, or adversarial in the wrong ways, then many common scoring and evaluation methods can look “good” right up until they catastrophically fail. Finance is full of examples of exactly this dynamic.
Two clarifications about my own claims:
I did not use leverage. The 40% average annual return I mentioned was achieved without leverage. I agree completely that high apparent performance with hidden ruin risk is trivial to generate, and I’m very wary of arguments that don’t control for that.
I don’t have a clean statistical confidence interval for my forecasting ability. I wish I did. What I can say—without pretending it’s a theorem—is that when I pitched this approach to VCs last year, several were interested in investing on the order of ~$2M. That’s not proof of correctness, but it does suggest that sophisticated actors found the combination of reasoning and track record at least plausible. (For the record, I embarrassed myself by not having the proper licenses lined up before pitching a hedge fund idea, which is a lesson I learned the hard way.)
More broadly, I think the honest answer is that superforecasting rests on a weak ontological assumption rather than a strong one: not that the world is well-behaved, but that some environments are predictable enough, often enough, to beat naive baselines. The goal isn’t asymptotic optimality; it’s persistent edge.
Where I personally diverge from the “pure scoring-rule” framing is that I don’t think of forecasting as approximating a single global distribution. Instead, I think of it as model selection under uncertainty, where the models themselves are provisional and frequently discarded. That doesn’t fully resolve the Cauchy-vs-Gaussian problem you raise—but it does mean I’m less committed to any single assumed distribution than the formalism might suggest.
Practical question: forecasting in a narrow, expert domain
Your North Korea example is excellent, and I agree with your diagnosis of the problem. If all you ask are first-order, low-entropy questions (“Will war break out this year?”), you get almost no learning signal, even if your answers are technically correct.
This is where my approach probably diverges from how most superforecasters would describe their own methods, and I want to be clear that I’m not claiming this is canonical.
Very roughly, my technique is to lean heavily on macro-level regularities and treat individuals as if they were particles—subject to incentives, constraints, and flows—rather than as unique narrative agents. At that level of abstraction, societies start to behave less like chess games and more like fluid systems. You can’t predict the motion of a single molecule, but you can often predict pressure gradients, bottlenecks, and phase transitions.
Applied to your case, that suggests focusing less on isolated facts (rice prices, phones) and more on questions that proxy for stress, throughput, and constraint relaxation. The exact phrasing matters less than whether the question sits on a causal pathway that connects to higher-level outcomes you care about.
You’re also right that the skill of asking good questions is the real bottleneck. My (imperfect) heuristic is to ask:
Does this variable aggregate many micro-decisions?
Is it constrained by hard resources or incentives?
Would a large deviation here force updates elsewhere?
Those questions won’t necessarily predict war directly—but they can tell you when the system is moving into a regime where war becomes more or less likely.
Finally, I agree with you that the intelligence community is one of the few places where calibration is actually rewarded rather than punished. In many ways, I think superforecasting is a partial rediscovery—by civilians—of techniques analysts have been developing for decades, albeit with better scoring and feedback loops.
I don’t think your concerns undermine forecasting as a practice. I think they correctly point out that it’s a tool with sharp edges, and that the hardest problems aren’t about probability math but about question selection, regime change, and institutional attention.
If you’re open to it, I’d actually be very interested in how you decide which NK-related variables are worth tracking at all—that feels like exactly the frontier Tetlock is gesturing at.
More options
Context Copy link
I've been working in intelligence for 15 years and have read all of these books, and others from the canon.
I can say that none of these should be recommended after Tetlock's work has been published. Some techniques from the structured analytical technique toolbox don't work. Some definitely do more harm than good. And the continued teaching of these techniques in place of e.g. reference class forecasting is so baffling to me that I can't express my frustration.
Heuer seems like a good guy who was doing his best to fix the terrible problems in the CIA at the time. But he's been totally replaced as an authority on this subject since Kahneman came along. Kahneman obviously did experiments and knew the research. Heuer was going off his personal observations from his career, and suggested analytical techniques that were a little better than the gut feels of the ivy league scotch swizzling guys in the agency through the cold war. He has been very accepting of new research and basically says "I did my best to formalise analysis, if others can come along and do better that's fantastic."
Since that time, we've come a long way.
R Pherson on the other hand is just scum. He refuses to acknowledge that the techniques in his books don't have any scientific basis. They've been directly measured across various studies, and he essentially argues that they do, in fact, work. Despite them definitely not working on any meaningful metric. He's made a career in the lecture circuit and knows that backing down will undermine his financial basis.
The real indicator that these techniques don't work is that nobody in the planet really uses them. If they improved outcomes, people would. But instead everybody goes to these week long courses, learns how to do a bullshit mind map or analysis of competing hypothesis, gets signed off as certified, and never looks at the techniques again.
More options
Context Copy link
More options
Context Copy link