Prediction market perspective on collective intelligence
From Handbook of Collective Intelligence
Set $wgShowExceptionDetails = true; at the bottom of LocalSettings.php to show detailed debugging information.
Prediction markets (and more generally, information markets) are exchanges in which people buy and sell their predictions about the future. In a standard set up, those who bet correctly profit, and those who do not, lose.
Prediction markets have been used in a number of settings:
|What's predicted or traded||Actors||Resources||Actions||Results||Evaluation and measures||Facilitator/owner, notes|
|Product sales||Sales executives||Information, money||Votes, cash reward up to $250 at HP||Voting result||Count of votes, variances||A number of firms, such as Hewlett Packard (called Behaviorally Robust Aggregation of Information Networks, or BRAIN), IBM, Microsoft, and Ford|
|Product sales||Employees||Information, money||Money contributed by employer||Trading in a market||market prices||Microsoft, Google, HP, many others|
|Product sales||Employees||Information, play money||Buy or sell contracts, bet on outcomes||Trading and/or betting in a market||market prices||Henkel (European consumer goods firm with $13bn revenue p. a.), Tchibo (European specialist retailer with 2000+ outlets) , facilitated by CrowdWorx S & OP demand forecasting solution |
|Terrorism||Public||Money||Buy or sell contracts||Trading in a market||market prices||Policy Analysis Market (PAM) (by DARPA and Net Exhange), in extreme form, see also assassination markets|
|Success of pharma products in pipeline||Employees||Information||Buy or sell contracts||Voting result||Count of votes||Incentive Markets, closed in Feb 2005|
|Prices of leased cars||Customer leasee||Information about leased car||Manheim trial leasees willing to pay 9 percent higher prices||Voting result||Count of votes||Manheim, Ford |
|Variety||Subscribers||Play or real money||Buy or sell contracts||Voting result||Arcelor gave weekend luxury trips as prizes to winners among the 30-40 traders, and Microsoft gives Xbox 360 consoles, Business Week (2006)||Intrade , Tradesports50K subscribers, 4M trades/mth, real money, NewsFutures(whose clients include Arcelor Mittal, Corning Digitial, Dentsu, HVG Hungary, Lilly, MasterFoods, Pfizer, RAND, SAIC, SCA Corrugated, Siemens, Texas Dept of Transportation, Thomson, de Volkskrant Netherlands, World Economic Forum, Yahoo! (which has a currency called Yootle), source), Foresight Exchange , Owise|
|Horse racing outcome||Subscribers||Buy or sell contracts||Voting result||Count of votes||Parimutuel bet at TradeHorseRacing |
|Mostly election and economic outcomes||Open to all||Real money||Buy or sell contracts||Trading in a market||market prices||Univ of Iowa |
|Economic outcomes||US residents||Buy or sell contracts||Trading in a market||market prices||HedgeStreet, regulated by the CFTC, with Susquehanna International Group and DRW Trading Group as market makers|
|Movie and related outcomes||Subscribers||Play money||Buy or sell contracts||Trading in a market||market prices||Hollywood Stock Exchange, which is used by MGM and Lions Gate Entertainment, BusinessWeek (2006)(discussion site)|
|Tech concepts, trends||Open||Play money||Buy or sell contracts||Trading in a market||market prices||Yahoo! Buzz Game |
|Tech concepts, trends||Employees||Play money||Buy or sell contracts||Trading in a market||market prices||France Telecom's Destiny |
|Macroeconomic forecasts||Traders||Bets on economic derivatives||Trading in a market||market prices||Goldman Sachs and Deutsche Bank|
|Macroeconomic forecasts||Employees||Information, play money||Buy or sell contracts||Trading in a market||market prices||Syngenta (Global agri-business firm with $11bn revenue p. a.) , facilitated by CrowdWorx Prediction Markets|
|Video games sales||Members||Contracts with infinite life||Trading in a market||market prices||SimExchange|
|Media publishing, esp. last mile (finalists in run up to selection for publication)||Members||Play money||Contracts||Voting result||Count of votes||Touchstone/Simon & Schuster with Media Predict|
|Capacity allocation||1 manager and 5 sales reps||Play money||Contracts for production volumes||1st round simulation: 86.6% as efficient as optimum; 3rd: 99%||Voting result||Malone (2004)|
|Emission rights||Business units||Real money||Contracts on emission rights||Trading in a market||market prices||BP, cited in Malone (2004)|
|Revenues and profits||Employees||Play money||Contracts||Trading in a market||market prices||HP project, led by Bernado Huberman, cited in Malone (2004)|
|New products||MBA students||Play money||Contracts||Voting result||Count of votes||MIT experiment described by Chan et al. (2002)|
|New products||Employees||Information, play money||Buy or sell contracts, bet on outcomes||Trading and/or betting in a market||market prices||Deutsche Telekom (Global telco firm with $82bn revenue p. a.) , facilitated by Crowdsourcing with CrowdWorx |
|Product launch dates, new office openings, etc.||Over a thousand Googlers, 146 events, 43 different subject areas||Play money||Contracts||Trading in a market||market prices||Google|
|Weather||Play money||Open||Contracts||Voting result||Count of votes||WeatherBill|
|Truth of various claims||Play money||Open||Contracts||Voting result||Count of votes||Truth Markets|
In some cases, there is a quasi-market, in which information is available in the open, but decisions are taken by a centralized entity. For example, Malone (2004) describes how HP employees can propose projects onto an equivalent of a bulletin board. HP's VC Cafe (board of senior managers) decides which proposed projects to fund, and funded projects that get staffing, again using the electronic bulletin board.
There are also studies in the reverse direction, on the impact of event probabilities on asset market prices:
- Effect of Steve Forbes being elected president, on municipal bond prices: Slemrod and Greimel (1999)
- Effect of an Iraq war, on equities prices: Leigh, Wolfers, Zitzewitz (2003)
- Effect of a Republican win in the presidential elections, on equities prices: Snowberg, Wolfers, and Zitzewitz (2007)
There's a related table at CommerceNet
One way to classify prediction markets is along different choices in the design parameters above.
- What's predicted or traded. We discuss only design dimensions with interesting implications:
- Contract families. A point contract predicts one outcome (e.g., Q4 sales exceeds $40M). A family of contracts could predict multiple outcomes (e.g., several contracts predicting Q4 sales, one for sales up to $10M, another for between $10M and up to $20M, etc.) The latter reveals the market's view of the distribution of sales, not just a point estimate.
- Payoffs. Common types are (see Spann and Skiera (2003), Wolfers and Zitzewitz (2004)):
- Fixed payment--e.g., a trader pays the market price for a contract that pays $1 if Bush wins presidency, $0 otherwise. The contract price then reflects the probability of Bush winning, if traders are risk-neutral.
- Indexed payment--e.g., a trader pays the market price for a contract that pays $p if Bush gets p% of the vote. The contract price now reflects the market's view of the mean percent of votes.
- Fixed price--e.g., a trader pays $1 for a contract and specifies that Bush will get more than p% of the vote, and the contract pays some fixed $x if the outcome occurs. If $x=$2, then p represents the market's view of the median percent of votes.
- Tournament, or betting pool--e.g., a trader pays to a market maker a fixed price for a contract to predict whether Bush will win presidency; the contract pays $x/y to traders who predicted right, $0 otherwise, where $x could be either pre-specified fixed amount, or is a function of the number of traders who made bets, and y the number of winners.
- Non-linear combinations of some of the above.
- Actors. A key question here is consider how open the participation should be. Preliminary evidence (and theory) from experiments and from other fields such as finance suggest that opening the market to as many as possible, and to allow non-experts or irrational ones (noise traders in finance) in the market.
- Resources. The obvious resource traders have is information that might not be available to all. Another dimension is money, whether real or play money. The implication is that both seem to have unique advantages. Servan-Schreiber, et al. (2004) study predictions for the 2003-04 NFL season at TradeSports and NewsFutures. The former uses real money, and the latter, play money. They find that real money provides stronger incentives for information discovery, while play money yield more efficient information aggregation (because unlike real money, the exogenous wealth distribution of players does not affect the weighting of the bets).
- Actions. The main choice of what traders do depends on the payoff design. On other dimensions, there seems to be general agreement that it is good to have as many market makers as possible, and to have the exchange take on the role as counter-party, both of which are design choices made in financial stock exchanges.
- Results. The design choices here are tied to contract design (under what's predicted or traded above).
- Evaluation and measures. In the case of a tournament, there is a design choice of how the opinions are aggregated. In an actual Prediction Market, buying and selling of rights to conditional payoffs sets relative prices, which can be interpreted as probabilities. Other institutions use an average, weigh guesses by characteristics of the traders (see BRAIN, e.g.) treat inputs strictly as votes, etc. (see the interesting ways of doing this for TripAdvisor reviews in Keates (2007)), etc. This question also seems under-studied.
Do prediction markets work, and where?
We begin by listing research that shows that market prices are empirically unbiased estimates of actual event probabilities:
- TradeSports: Tetlock (2004), Borghesi (2006), Zitzewitz (2006)
- Macroeconomic forecasts: Wolfers and Gurkaynak (2005)
- Hollywood Stock Exchange (HSX): Pennock (2001), Wolfers and Zitzewitz (2004)
- Iowa Market: Berg and Rietz (2003)
- Horseracing: Thaler and Ziemba (1988), Snowberg and Wolfers (2006)
- CBOE Options: Zitzewitz (2006)
Nevertheless, Ali (1977) and Manski (2005) model risk-neutral traders and show that prices may be biased aggregates of traders' beliefs. Max Keiser, former co-chairman of HSX, argues that markets are horrible predictors of anything. This is along the lines of the random walk theory of efficient markets in finance--e.g., Malkiel (2003). Gjerstad (2003) and Wolfers and Zitzewitz (2007) refine the model to show that under common empirical conditions (such as risk-averse traders and smooth distribution of their beliefs), prices are tend to traders' mean beliefs.
A stronger result is that market prices are weakly more accurate than expert forecasters:
- Orange juice futures improve weather forecasts by experts, see Roll (1984)
- Horseracing markets beat horseracing experts, Figlewski (1979)
- Prices of contracts for Oscar, Emmy, and Grammy awards
- Derivative prices on natural gas storage at the US Department of Energy are off 3.9% from the next-day announced storage number, compared with 5.8 to 12.3% for 6 of the best known gas gurus and the average 7.1% off by three dozen analysts and traders, Jakab (2004)
- Stock market able to pinpoint Morton Thiokol as the primary contributor to the Challenger crash, and not the other 3 companies (Lockheed, Martin Marietta, Rockwell) the period immediately following the crash while the esteemed panel...took several months, Maloney and Mulherin (2003)
- Iowa Election Market beats opinion polls, Berg and Rietz (2003)
- Forecasts by a relatively small number of people...selected specifically from different parts of the business operation beat official forecasts, Chen and Plott (2002). The latter might have faltered because these forecasts were used not only as forecasts, but as management compensation targets and sales quotas.
- HSX predictions beat those by expert Brandon Gray, of Box Office Mojo, Spann and Skiera (2003)
- Predictions by 20 sales and marketing employees at a German mobile phone operator beat those obtained by standard extrapolations (such as linear, exponential), Spann and Skiera (2003)
- Forecasts of video game sales by SimExchange members were 15% off for sales of Nintendo's DS and Wii, Sony's PlayStation 3 and PlayStation Portable and Microsoft's Xbox 360, while Wedbush Morgan's Michael Pachter, one of the most quoted industry analysts is off by 10.6%. But the previous month, SimExchange is off by 15.8% and Pacher by 34.9%, Stuart (2007)
Why do prediction markets work?
Aggregation of private signals
This is first stated as early as Hayek (1948). More recently, Hanson (2007) points out that it is hard to find information that such market prices do not embody, because anyone who finds such neglected information can profit by trading on it...Speculative markets work well not only because they reward accuracy and punish error, but also because they encourage self-selection of traders; people who realize they are not as well-informed as average traders stay away. People who do not realize they are not well-informed lose and then go away. (pg. 6)
Finally, and [e]ven more important, the traders can see whatever other data they have (even if it's just their instincts) to judge whether a given prediction at a given price is a good buy or not (Malone (2004), pg. 3).
Furthermore, the aggregated information tends to be free of the biases associated with getting information from the field. For example, when getting information from front-line sales staff, [o]ne salesperson may want to make his number look big to keep his boss happy until performance reviews are completed; a sales manager may want to make her number look small so she can argue that she needs more staff. By contrast, in the market, salespeople are motivated to trade based on what they actually think will happen--not what they want to happen or what they want others to think will happen (Malone (2004), pg. 3). This suggests of course, that the market somehow finds ways to mitigate the earlier political concerns when salespeople reveal their bets in the market, perhaps with higher-powered incentives to reveal the truth during trading.
Benefits of prediction markets
More accurate predictions could be an end in itself, but in the process of producing more accurate predictions, markets also have other properties, such as uncovering more information. All these have their benefits (see Malone (2004), pp. 5-7).
- Traders can correct their own biases, if they can see how others are voting
- Traders get a bigger picture that has a high signal-to-noise ratio. The aggregated prediction is a reasonably good summary statistic of many people's reading of the situation.
- More agile organizations, since agility depends partly on being to better anticipate the future.
- Internal pricing leads to more precise asset allocation (the invisible hand). As one example, it could lead to more individualized service if, for example, frontline sales staff now knows how much it would cost to accelerate orders to satisfy a particularly important customer.
- Contingent contracts can aid decision making--see 'Hanson (1999). However, one needs to be very cautious about interpreting contract prices as probabilities for contingencies, since it is easy to mistaken correlation for causation.
Issues with prediction markets
- Will trading reduce team spirit? How can group trading mitigate this?
- What are the other impact on people's motivation (someone who loses, for example, or a self-fulfilling prophecy in which team members jump off a sinking ship project)
- Disclosure dilemma. Do firms have to disclose to outsider stakeholders (such as financial markets) what they found?
- How substantially more accurate are prediction markets? In small experiments from 1996 to 96 at HP, prediction markets did better in 6 out of 8 cases, but a few percentage points...The accuracy improvement was not high enough to be adopted BusinessWeek (2006)
- Is it legal? It is conceivable that prediction markets are exempt from anti-gambling laws, since gambling is legally defined as an activity that involves consideration, prize, and chance (see Bell (2006)) and prediction markets do not have the last. However, US law do regulate exchanges, so some prediction markets that use real money with many traders, like the Iowa exchange, have found it safe to get a no action letter from the Commodities Futures Trading Commission. There have been suggestions for the government to grant safe harbor to prediction markets; see Arrow, et al. (2007).
- Endowment effect. In games where traders are initially endowed with play money, traders' willingness to accept a contract might greatly exceed their willingness to pay, see Kahneman, et al. (1991).
The areas where prediction markets have not worked are especially interesting to study. The failures, however, seem to be either too obvious or are understudied:
- Policy Analysis Market (PAM)
- Incentive Markets, which closed in Feb 2005
- Shrinking markets or when drastic change is needed. Malone (2004) hypothesize that when a company is shrinking, for example, the ability to resolve conflicts through centralized authority is usually more important than encouraging creativity and independence (pg. 7).
- Market failures (see below)
Factors facilitating collective intelligence
- Learning. Adams (2006) and Ottaviani and Sorensen (2006) show that with multiperiod learning, the market price converges to traders' mean beliefs, aggregating all private signals.
- Design choices. These (along the lines outlined above) seem important, but this area seems under-studied.
Factors inhibiting collective intelligence
- Overly restrictive or unclear specification of event to be predicted. For example, a 2006 Tradesports contract on whether North Korea conducts a missile test specifies that the US Department of Defense as a confirmation source. But on this event, the DOD does not confirm the incident, even though it has been widely reported on in the media. In another case, a user alleges that Tradesports announces an outcome of a game in a random number of seconds after the specified time of the event. Wolfers and Zitzewitz (2004) conjecture that play money might facilitate the introduction of loose definitions, enlarging the set of questions that might be amenable to prediction markets.
- System downtime during critical betting periods. For example, one user alleges that Tradesports' site is down during the last one minute of a crucial SMC/USF game on Feb 19, 2007.
- Involvement of biased parties, and cornering the market. Rhode and Strumpf (2003) studied bets on outcomes of presidential elections between 1868 and 1940, and conclude that there is little evidence in the historical record that wagering compromised the integrity of elections despite the active involvement of political parties (pg. 1). In contrast, Friedrich Dürrenmatt's The Visit (1956, 62) is a play about rich lady Claire Zachanassian who visits the town of Güllen. She puts a big bet that a resident Alfred Ill, a former lover, will be killed. The beneficiary of the bet is the impoverished town. Although the townsfolks initially found the bet repulsive, one by one, they begin to spend beyond their means, as if the town will someday be rich. In the end, Alfred Ill dies mysteriously.
- Predictions of extreme events. The favoriate-longshot bias has been documented in TradeSport, Chen et al. (2006) and Fair (2006)
- Long-lived contracts? SimExchange uses infinite-life contracts (mimicking equities in the stock market) in a prediction market for video game sales. Some observers suggest that this might work only for play money.
- Affiliation bias. Koleman (2004) shows evidence that New York betters in the Iowa prediction market favor the Yankees. Forsythe, et al. (1999) show that traders favor bets of their own political parties.
We also conjecture that many factors that inhibit healthy development and functioning of traditional asset markets, such as the stock market, might inhibit CI. Such factors include:
- manipulation. However, Hanson, et al. (2006) argue that at least in an experimental market they studied, manipulation did not last and the non-manipulators compensate for the bias in offers from manipulators by setting a different threshold at which they are willing to accept trades.
- insider trading. Hanson (2007) also argues that there are ways to curb excessive insider trading, such as requiring elite traders (insiders) to disclose their trades ahead of time.
- thin markets--e.g., O'Hara (2003)
- herding--e.g., Banerjee (1992). Plott and Sunder (1982) and Plott and Sunder (1988) show that in experiments, bubbles seem to occur. On the other hand,
- poor rule of law--e.g., Shleifer and Vishny (1997)
- limits to arbitrage, if traders are agents for other principals--e.g., Shleifer and Vishny (1997)
Techniques for enhancing collective intelligence
- Real or play money? Both seem to have unique advantages. Servan-Schreiber, et al. (2004) study predictions for the 2003-04 NFL season at TradeSports and NewsFutures. The former uses real money, and the latter, play money. They find that real money provides stronger incentives for information discovery, while play money yield more efficient information aggregation (because unlike real money, the exogenous wealth distribution of players does not affect the weighting of the bets).
What we would most like to understand in future research
- Impact of design choices on performance. Some early attempts have been made (e.g., Spann and Skiera (2003)), but we seem very far from even a rudimentary understanding.
- How do we incorporate computer agents and quantitative techniques into prediction markets?