Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. IARPA’s Geopolitical Forecasting (GF) Challenge invites individuals or teams to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts of geopolitical events. Solvers will be competing against each other and in parallel with a similarly structured research program funded by IARPA. Over the course of approximately seven months, the GF Challenge solvers will be asked to develop solutions capable of producing forecasts to a series of questions like:
- Who will win the upcoming presidential election in Egypt?
- or What will the spot price of Brent Crude oil be on [date]?
In order to answer these questions, solvers will be given access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters and will be allowed to use other data sources to produce their solutions. The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that successfully forecast of a wide variety of geopolitical events, such as political elections, international conflict, disease outbreaks, and macro-economic indicators.
Solvers will be invited to the GF Challenge Platform to compete. Each solver or team will be assigned a unique application programming interface (API) token and login account for the platform. A steady stream of questions (roughly 25 per month) will be released for solvers to produce probabilistic forecasts against (specific requirements will be described in the rules). Questions may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the questions, solvers will receive access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters. Solvers are encouraged to create solutions that use this stream of human judgments alongside other publicly available data streams and information to create their forecasting solutions.
- Questions will be released on the platform and can be pulled through the API
- Each forecast question will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a day to many months in the future. Challenge participants will be asked to produce forecasts until the question resolves.
- Examples of potential questions*:
- Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
- Before March 2018, with South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
- All forecasts will need to be submitted through the API
- Forecasts may be updated and will be scored on a daily basis for each question that is open.
Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, a quadratic measure of the forecaster's closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence. The score will be calculated on a daily basis over the duration of the question; those daily accuracy scores will then be averaged. The more accurate and timely the forecast, the better score.
As questions will range in duration, milestone check-ins for questions resolved during a certain period will be used to award progress-based incentives. This means solvers who start after the challenge has launched are still able to compete for some of the prizes. Finalists who wish to compete for monetary prizes will need to provide the solution package for review the GF Challenge Team at the end of the challenge, or as appropriate, if requested. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. Please refer to guidelines for a description of scoring.
Solvers will be able to compete on their own or form teams to compete for a total prize purse of $200,000. The prize amounts shown are dependent upon certain criteria and/or participation requirements. Challenge results and solutions will be reviewed at an IARPA forecasting Workshop in Washington, D.C. For additional details and prize requirements, please refer to the guidelines. Prizes will be distributed for the following criteria:
- Overall Performance Prizes
- 1st Place - $20,000
- 1st Place could be eligible to receive the Performance Bonus: Ultimate Forecaster, an additional $40,000*
- 2nd Place - $15,000
- 3rd Place - $10,000
- 4th Place - $7,000
- 5th Place - $4,000
- Second through Fifth Place may be eligible for a share of Performance Bonus: Star Forecasters, an additional $30,000*
- Additional Incentives
- Best in Domain / Region Pair (X5) - $25,000 total
- Best Undergraduate - $4,000
- Milestone Performance (X3) - $30,000 total
- Interim Prizes - up to $15,000
*Bonuses have additional eligibility requirements. Please see the challenge rules for details.
Rules and Eligibility
Individuals and teams, 18 years of age or older will be able to compete for cash prizes. Some individuals or organizations may not be eligible for monetary prizes. However, these solvers may, upon IARPA approval, participate in the challenge and be eligible for ranking on the leaderboard. Approval will need to be obtained in advance of participation by emailing: email@example.com.
What Can You Do Right Now?