38,822
Geopolitical Forecasting [GF] Challenge

Geopolitical Forecasting [GF] Challenge

Can you create a method to predict future geopolitical outcomes that is better than current state-of-the-art methods? Read Overview...
Overview

The Problem

Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. IARPA’s Geopolitical Forecasting (GF) Challenge invites individuals or teams to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts of geopolitical events. Solvers will be competing against each other and in parallel with a similarly structured research program funded by IARPA. Over the course of approximately seven months, the GF Challenge solvers will be asked to develop solutions capable of producing forecasts to a series of questions like:

  • Who will win the upcoming presidential election in Egypt?
  • or What will the spot price of Brent Crude oil be on [date]?

In order to answer these questions, solvers will be given access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters and will be allowed to use other data sources to produce their solutions. The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that successfully forecast of a wide variety of geopolitical events, such as political elections, international conflict, disease outbreaks, and macro-economic indicators. 

Challenge Details

Solvers will be invited to the GF Challenge Platform to compete. Each solver or team will be assigned a unique application programming interface (API) token and login account for the platform. A steady stream of questions (roughly 25 per month) will be released for solvers to produce probabilistic forecasts against (specific requirements will be described in the rules). Questions may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the questions, solvers will receive access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters. Solvers are encouraged to create solutions that use this stream of human judgments alongside other publicly available data streams and information to create their forecasting solutions.

  • Questions will be released on the platform and can be pulled through the API
  • Each forecast question will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a day to many months in the future. Challenge participants will be asked to produce forecasts until the question resolves. 
  • Examples of potential questions*:
  • Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
  • Before March 2018, with South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
  • All forecasts will need to be submitted through the API
  • Forecasts may be updated and will be scored on a daily basis for each question that is open.

Scoring

Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, a quadratic measure of the forecaster's closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence. The score will be calculated on a daily basis over the duration of the question; those daily accuracy scores will then be averaged. The more accurate and timely the forecast, the better score.

As questions will range in duration, milestone check-ins for questions resolved during a certain period will be used to award progress-based incentives. This means solvers who start after the challenge has launched are still able to compete for some of the prizes. Finalists who wish to compete for monetary prizes will need to provide the solution package for review the GF Challenge Team at the end of the challenge, or as appropriate, if requested. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. Please refer to guidelines for a description of scoring.

Prizes

Solvers will be able to compete on their own or form teams to compete for a total prize purse of $200,000. The prize amounts shown are dependent upon certain criteria and/or participation requirements. Challenge results and solutions will be reviewed at an IARPA forecasting Workshop in Washington, D.C. For additional details and prize requirements, please refer to the guidelines. Prizes will be distributed for the following criteria:

  • Overall Performance Prizes
    • 1st Place - $20,000
      • 1st Place could be eligible to receive the Performance Bonus: Ultimate Forecaster, an additional $40,000*
    • 2nd Place - $15,000
    • 3rd Place - $10,000
    • 4th Place - $7,000
    • 5th Place - $4,000
      • Second through Fifth Place may be eligible for a share of Performance Bonus: Star Forecasters, an additional $30,000*
  • Additional Incentives
    • Best in Domain / Region Pair (X5) - $25,000 total
    • Best Undergraduate - $4,000
    • Milestone Performance (X3) - $30,000 total
    • Interim Prizes - up to $15,000

*Bonuses have additional eligibility requirements. Please see the challenge rules for details.

Rules and Eligibility

Individuals and teams, 18 years of age or older will be able to compete for cash prizes. Some individuals or organizations may not be eligible for monetary prizes. However, these solvers may, upon IARPA approval, participate in the challenge and be eligible for ranking on the leaderboard. Approval will need to be obtained in advance of participation by emailing: gfchallenge@iarpa.gov.

 

What Can You Do Right Now?

  • Click “Follow” to receive challenge updates and follow its progress!

  • Register for the challenge by clicking "Accept the Challenge" and agreeing to the challenge rules and IP agreement.
  • Share this challenge on social media using the icons above. Show your friends, your family, or anyone you know who has a passion for discovery.
  • Start a conversation in our Forum to join the conversation, ask questions or connect with other innovators.
Guidelines
Timeline
Updates 29

IARPA announces Geopolitical Forecasting Challenge 2 (GF Challenge 2) – Are you ready?

April 4, 2019, 11:01 a.m. PDT by Despina Maliaka

Are you ready?

Whether you were an active Solver or followed the first GF Challenge, we wanted to share this exciting news... GF Challenge 2 starts in May 2019!  

Check out the GF Challenge 2 page to learn more and begin engaging with this challenge.  https://www.herox.com/IARPAGFChallenge2

 

-The GF Challenge Team


Announcing the GF Challenge Winners

Oct. 1, 2018, 5:50 p.m. PDT by Kyla Jeffrey

Over the past several months, you have participated in a unique challenge hosted by the Intelligence Advanced Research Projects Activity (IARPA). Together, we’ve watched multiple world events unfold before us and celebrated your work along the way with numerous Milestone awards. We thank you for your efforts.  

 

After completing the final calculations, we are thrilled to announce the overall winners of the Geopolitical Forecasting Challenge. 

1st Place: DigitalDelphi 

2nd Place: BEEFERS 

3rd Place: Catskills Research Company 

4th Place: truthLover 

5th Place: SISLers 

Ultimate Forecaster: no winner 

Star Forecaster: BEEFERS 
 

And here are the Additional Prize Winners in specific categories.  Congratulations! 

Election Forecaster: DigitalDelphi 

Domain Region Prizes 

  • Politics/International Relations & Middle East: truthLover 
  • Geopolitical Grab Bag & Asia: Catskills Research Company 
  • Macroeconomics/Finance & Europe: LIA 
  • Politics/International Relations & Europe: BEEFERS 
  • Politics/International Relations & Asia: truthLover

Announcing the Winners of the Workshop Presenter Prize

Sept. 25, 2018, 7:20 a.m. PDT by Kyla Jeffrey

Dear Solvers, 

It’s our pleasure to announce the winners of the GF Challenge Final Workshop.  Per the rules, these teams were selected based on Leaderboard Standings as of September 23rd.  These teams and individuals have received a presentation slot at the Final Workshop.  They will receive a $2,500 prize payment upon completion of their presentation.  

  1. DigitalDelphi
  2. BEEFERS
  3. catskills
  4. LIA
  5. SISLers

Congratulations – we can’t wait to meet you in the Washington, D.C. area!  

Solvers, as a reminder: While the winners of the final Workshop will receive a prize award to help with travel and accommodations, we invite all GF Solvers to join us at this special event in celebration of your participation in this unique challenge. It’s a great opportunity to network and learn from your fellow Solvers. Please keep an eye out in your email for an invitation to the workshop.  

Cheers! 

GF Challenge Team


Milestone 3 Awards

Sept. 12, 2018, 6 a.m. PDT by Kyla Jeffrey

Dear Solvers,

It’s time to announce the winners of our third—and final—milestone! This award prize totals $12,500 and is split among the top 10 teams with 80% participation. Congratulations to the following teams and Solvers!

  1. BEEFERS
  2. Catskills
  3. CitizensBand
  4. DigitalDelphi
  5. LIA
  6. SISlers
  7. Seb
  8. dobreovidiu
  9. pb2pv
  10. truthLover

We’ve got more award announcements on the way, so stay tuned! You can expect to hear the final results around the end of September. Thank you so much for your participation!Cheers!

GF Challenge Team


Announcing Second Chance Awards

Aug. 31, 2018, 8:42 a.m. PDT by Liz Treadwell

As the GF Challenge enters its final weeks, we are watching the leaderboard with excitement. The Solver community has been doing an incredible job producing forecasts against a very broad portfolio of individual forecasting problems (IFPs). We are pleased to announce a new tier of “Second Chance” Prizes that may be awarded in the event that fewer than 5 methods qualify for Overall Prizes. As described in the GF Challenge Rules Document, the 1st through 5th Place Overall Prizes will be awarded for those methods with the highest Net Brier Points (NBPs), conditional on participating in at least 70% of IFPs and receiving positive NBPs on more than half of attempted IFPs. (Only each Solver’s best scoring method will be considered).

If, however, fewer than 5 winners receive Overall Prizes, IARPA will award “Second Chance” Prizes to recognize methods that achieved high NBPs, while participating in at least 70% of IFPs, but did not meet the 50% positive NBP minimum. The Second Chance award tier will be awarded as follows:

  • Methods that qualify for 1st through 5th Place Overall Prizes (and Ultimate Forecaster and Star Forecaster) will be awarded according to the originally stated criteria.
  • If fewer than 5 winning Solver-methods receive Overall Prizes, the remaining slots will become available in the Second Chance tier (e.g., if only 1st and 2nd place Overall Prizes are awarded, then 3rd through 5th place slots will be eligible for Second Chance award).
  • Second Chance prizes will be awarded using the same standards as Overall Prizes (i.e., highest NBPs, at least 70% participation) with the exception that methods will need to have positive NBPs on a minimum of only 33.3333…% of attempted IFPs.
  • Second Chance prizes will be worth half as much as their equivalent Overall Prize amounts, as shown in the table below, and will not be eligible for Ultimate Forecaster or Star Forecaster awards.
  • Power Forecaster(s) Award: Only awarded if both the Ultimate and Star Forecaster Overall Prizes are not awarded. Must participate in at least 70% of the IFPs.  Must have positive NBPs on a minimum of one third (33.3333…%) of the IFPs you attempt.  Must have a lower Brier score than the Benchmark on the Beat HFC Leaderboard.  Have a lower Brier score than all HFC Performers on the Beat HFC Leaderboard.  Uses the best scoring method, regardless of the Solver’s overall method used in final scoring. This will be awarded at $25,000 and is split amongst all eligible Solvers / teams.
  • Each Solver will only be eligible for one prize (either Overall or Second Chance) based on their single best performing method.
  • Please see the updated Rules Document for full eligibility requirements and prize breakdowns

Position

Overall Prize

Second Chance Prizes (if applicable)

1st

$20,000

$10,000

Ultimate Forecaster

$40,000

N/A

2nd

$12,000

$6,000

3rd

$10,000

$5,000

4th

$7,000

$3,500

5th

$4,000

$2,000

Star Forecaster(s)

$30,000

N/A

Power Forecaster(s)

N/A

$25,000 (split amongst all eligible Solvers / teams)

 

How were the percentage and prize allocations determined?

The team examined the challenge from many different perspectives—trends across time, attempt rates, success rates, as well as comparison evaluation with HFC. We recognize that we set an extremely high bar and are appreciative of the community’s achievements to date. An evaluation of the leaderboard and IFPs assisted in setting the “Second Chance” Prize percentage, which then informed the “Second Chance” prize amount awards.


Forum 222
Community 798
FAQ
Resources