forecasting.substack.com
In short…
- Forecasting platforms and prediction markets are partially making the pie bigger together, and partially undercutting each other.
- The forecasting ecosystem adjusted after the loss of plentiful FTX money.
- Dustin Moskovitz’s foundation (Open Philanthropy) is increasing their presence in the forecasting space, but my sense is that chasing its funding can sometimes be a bad move.
- As AI systems improve, they become more relevant for judgmental forecasting practice.
- Betting with real money is still frowned upon by the US powers that be–but the US isn’t willing to institute the oversight regime that would keep people from making bets over the internet in practice.
- Forecasting hasn’t taken over the world yet, but I’m hoping that as people try out different iterations, someone will find a formula to produce lots of value in a way that scales.
Jump in the discussion.
No email address required.
Notes -
Good writeup I read your article.
The biggest gap I see in prediction markets is a lack of actual large stakes betting without a house spread. It is either play money, or amounts too small to matter, or it is rigged against you by spreads in Vegas.
I would say the biggest betting markets you've missed are the actual financial, governmental and business markets. That is where the rubber hits the road and huge fortunes are gambled on directional bets. If every country that can is investing in nukes then nukes are probably important on the world stage; if every company that can invest in AI is doing so, the AI is probably key to business going forward.
There is another problem with forecasting that I'm sure people have written doctorial dissertations on. If it gets really good and really common then how would one extract any value from it? It would lack the informational asymmetry that might bring you value today if you had the only good predictive model for something important.
The final issue is that if it is common and good then it will alter the very things it is trying to predict. Does predicting it make it true when we trust predictions at a 99.9% confidence ratio? Is there then a rebound effect where they become worthless and you need a meta meta meta meta meta prediction market to determine the accuracy of the prediction market you're trusting to verify the accuracy of prediction market that you're using to make the initial prediction?
Postscriptum for my deterministic peeps out there-
Eventually everyone will have to accept total hard determinism, because after all, what are scientific physical laws but predictions that are right 100% of the time. But until then, there are hurdles to jump over in finding value in this space.
If there's one answer that everyone agrees on and no one betting against each other, all the value would come from whomever posted the question and sponsored it so that anyone who guesses correctly gets proceeds. If people disagree on the answer, then people who're right also get value from whomever's wrong losing their money to the people who are right.
If people disagree on the answer then they need better forecasting.
Sure, but I don't think we're going to ever get to that point anytime soon, because some questions are real hard to forecast. I don't think anyone today knows when say the Israel-Palestine conflict will next have a ceasefire, but that's something a lot of people would like to know. If we get to a point where we can forecast so well everyone agrees on when it's most likely to end, then that's mission accomplished.
If we get to that point then the conflict would never have happened in the first place.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Thanks for the comment. Some points:
To quantify this, there are some markets in which you can bet >100k, particularly around US elections. Kalshi is also trying to change this in the US. But yeah.
Different niche, though. One important difference is that in normal markets "the market can stay irrational longer than you can stay solvent". Not so in prediction markets/forecasting questions: there is a definite date.
Not all goods are rival, not all games are zero sum. E.g., people can and do get value from weather forecasting.
Sure. You do have fixed point problems. You can also make predictions conditional on a level of investment. It's still a consideration, though.
You can make large stakes bets, but as I said they are handicapped to the point where your odds are less than 50% of winning because of spreads.
Is the market irrational? Or are you?
Fair point there are a lot of positives to be had from certain predictive algorithms.
I don't know what this last one means.
The last one is: I agree that sometimes predictions influence what happens. A few cases people have studied is alarmist Ebola predictions making Ebola spread less because people invested more early on, and optimistic predictions about Hillary Clinton leading to lower turnout.
You can solve these problems in various ways. For the Ebola one, instead of giving one probability, you could give a probability for every "level of effort" to prevent it early on. For the Hillary Clinton one, you could find the fixed point, the probability which takes into account that it lowers turnout a little bit (https://en.wikipedia.org/wiki/Fixed_point_(mathematics)).
Unfortunately, the level of effort option can still lead people to think it'll be fine if they just take the measures, then not get concerned and don't take the measures.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
Nah, I think the issue that precedes and largely supercedes this is the oracle question. Do people trust that whatever entity is reporting the final results is doing so accurately and isn't fudging numbers to give an edge to its allies or to cover up some other outcome that TBTB are trying to disguise?
Do we trust that ambiguous results will be resolved in good faith and correctly more often than not?
Who do we actually rely on to be the final arbiter of 'truth' such that these markets can continue to settle reliably where there's incentive to capture such institutions to divert them from the purpose of accurate reporting.
In other words I personally doubt we'll ever reach 99.9% confidence in prediction markets if only because we can't reach that confidence in the platforming hosting the markets or the entities producing the results which are deemed as 'truth,' and I don't believe these are easily tractable issues.
This is a big problem on Manifold Markets and on Polymarket. On Manifold, there are a lot of market creators who write ambiguous resolution criteria or they even change the resolution criteria after people have placed bets. The resolution criteria often describe something quite different than what the question is literally asking. For example, there was a question that straightforwardly asked whether Israel blew up a certain hospital in Gaza, and then when it turned out the hospital hadn't been blown up at all and that the bomb had exploded in the parking lot, the question was changed to whether Israel was responsible for the explosion.
It has become common to resolve in favour of some nebulous, undefined "spirit" of the question, rather than the actual meaning of the question that was asked. A lot of markets become mainly bets on how the creator will decide to resolve it rather than on what the question is purportedly about.
On Polymarket, the resolution mechanism for disputed questions relies on a Keynesian beauty contest that has settled on an equilibrium where everyone assumes the simplest and stupidest possible interpretation, and now people are even contesting uncontroversial resolutions in order to take advantage of this broken system. There will be a question that resolves and everyone agrees that it was resolved correctly, but then the resolution will be contested and everyone knows the vote will go in favour of some hypothetical interpretation that would only work if everyone was retarded, so they vote that way. No one agrees with the interpretation, but everyone is incentivized to vote how they think everyone else will vote. And everyone knows the winning vote is expected to be the one that doesn't involve reading the full resolution description and doesn't involve using any sort of complex thought.
Yes. I've seen problems arise even with fairly 'objective' markets because even if you can measure a given phenomenon with precision, people might still mistrust the sensor doing the measuring. The market asks "what will be the high temperature in Miami on [date]" and we have to consider whose thermometer? Is it calibrated correctly? Are there any conditions that might throw it into an unexpected/error state?
So now the question is somewhat less about climate conditions and more about the quirks of the measurement system.
In theory you could solve this by attaching a reputation market to the system, so that a given resolution source can have their 'trustworthiness' rating impacted if enough people suspect they're fudging numbers or intentionally writing ambiguous questions/resolution criteria.
But that's just yet another system that is susceptible to gaming.
Augur had a seemingly solid system for avoiding this, but probably couldn't handle the volume, being dependent on Ethereum.
I am literally a practicing attorney and I have had my mind blown at some of the rules-lawyering/munchkin behavior that has come out of the space.
Ironically this perhaps goes to show why sports betting is so popular, because sports rules are uniformly understood, well-defined, and the bets are set on easily determinable outcomes like "Who won" and "what was the score", outcomes which are rarely ever walked back after the fact.
I speculate that we'll see some kind of AI-based solution arise and different markets will become popular with different segments of the population based on the quirks of how, say, Kalshi's AI resolves questions vs. Polymarket's vs. Manifold's.
In this case prediction markets might not actually 'solve' the issue of people having different reality bubbles, but at least there'll be some competition.
This is not what I recall. Invalid markets resolved to 50/50, so you had users, chiefly someone who went by the moniker of Poyo, create markets that appeared to be legit but e.g., had the wrong date, so that people would bet & he'd win money when they resolved 50/50
Yes, and the Augur 2.0 solution was to add in an option for people to bet on whether a market was invalid in the market itself.
More options
Context Copy link
More options
Context Copy link
I have a lot of lawyers in my family, one of whom is close to me and the main part of his job is to write legal documents in clear, precise, unambiguous language, so I'm used to thinking about language and rules in a certain way (I also have a STEM background, where things have precise definitions). I've been blown away by how bad otherwise intelligent people are at writing and interpreting resolution criteria. They throw out basic principles which I would have thought were necessary for there to be any hope being able to decide these things in a consistent and predictable manner. I even explained one of the resolution disputes on Polymarket to these family members, one that was ambiguous due to a blatant self-contradiction in the resolution criteria, and they said it should definitely be resolved one way, which ended up being resolved the other way (essentially on the principle of most people wouldn't read that far into the description of the resolution criteria).
One possible solution is that you have people pay to have questions answered, and as part of that payment, they pay people to act as oracles who have good reputations. So the incentive is to decide things in a way that most closely matches what the question asker intended and also most closely matches what bettors think the question is about so that they are willing to bet on it, since this improves the market's accuracy.
Yeah, this was part of how Augur's system worked. Reward people who end up on the 'right' side of a final resolution question consistently AND anyone who is answering the question has to stake some portion of their reputation on the outcome they're judging. Eventually 'bad actors' (who are either malicious or are too stupid to reliably interpret contracts) lose out and the correct/consistent oracles accumulate more wealth so they can have more influence over future resolutions.
It helped settle into an equilibrium where it was usually not worthwhile to try to exploit an apparent ambiguity, while knowing that wealthier oracles will ignore said ambiguity and you'll lose money directly by trying to challenge them.
Yep. There are plenty of bright line rules for resolving ambiguity in legal contracts, and it can be permissible to pull in outside evidence to interpret them, but you have to think about the ENTIRE document in a systematic way, you can't just glance it over and interpret it based on vibes.
And glancing at things and going with your gut is how so, so many humans operate.
The problem is there's always a tradeoff when you try to get as precise as possible with your wording, in that it both makes it harder for laypeople to easily understand what the terms say (and less likely to read it all) and, paradoxically, can open up a greater attack surface because there's more places where ambiguities can arise.
This is where I imagine LLMs would have a role, if they are given a set of 'rules' by which all contracts are to be interpreted, and they can explain the contracts they read to laypeople, and everyone agrees that the AI's interpretation is final, then you at least make it more challenging to play games with the wording.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link
I mean, in practice you don't need 99.9, you need better than alternatives in at least some cases.
Agreed. And thus I strongly support prediction markets as a concept for making personal decisions, hedging risks, and predicting important events.
Just noticing that centralized prediction markets are yet another sort of institution that can be captured and/or sabotaged if they become important to guiding/controlling society.
Would really hope we have robust competition between them to ensure no player ever becomes fully dominant in the space.
More options
Context Copy link
More options
Context Copy link
Well if you walk out your door and a little box on your wrist literally tells you what you'll do next, even if you change your mind, or more importantly, what others will do. If it is always right you'll trust it. Then that might be good enough to get people to 99.9% yah? It is possible, yet not likely.
More options
Context Copy link
More options
Context Copy link
More options
Context Copy link