41,295
Geopolitical Forecasting [GF] Challenge

Geopolitical Forecasting [GF] Challenge

Can you create a method to predict future geopolitical outcomes that is better than current state-of-the-art methods?
stage:
Fall 2018: Winner Announcement
prize:
$200,000
Overview

Challenge Overview

The Problem

Decision makers rely on the Intelligence Community to provide accurate and relevant forecasts, and IARPA is working to identify methods to maximize the quality of these forecasts. IARPA’s Geopolitical Forecasting (GF) Challenge invites individuals or teams to develop innovative solutions and methods for integrating crowdsourced forecasts and other data into accurate, timely forecasts of geopolitical events. Solvers will be competing against each other and in parallel with a similarly structured research program funded by IARPA. Over the course of approximately seven months, the GF Challenge solvers will be asked to develop solutions capable of producing forecasts to a series of questions like:

  • Who will win the upcoming presidential election in Egypt?
  • or What will the spot price of Brent Crude oil be on [date]?

In order to answer these questions, solvers will be given access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters and will be allowed to use other data sources to produce their solutions. The challenge presents an opportunity for individuals and teams to earn prizes by creating methods that successfully forecast of a wide variety of geopolitical events, such as political elections, international conflict, disease outbreaks, and macro-economic indicators. 

Challenge Details

Solvers will be invited to the GF Challenge Platform to compete. Each solver or team will be assigned a unique application programming interface (API) token and login account for the platform. A steady stream of questions (roughly 25 per month) will be released for solvers to produce probabilistic forecasts against (specific requirements will be described in the rules). Questions may be binary (yes/no), multiple choice, or ordered outcome (binned quantity/date). In addition to the questions, solvers will receive access to a continuously updated stream of forecast judgments produced by a crowd of human forecasters. Solvers are encouraged to create solutions that use this stream of human judgments alongside other publicly available data streams and information to create their forecasting solutions.

  • Questions will be released on the platform and can be pulled through the API
  • Each forecast question will include a set of exhaustive, and mutually exclusive response options, and will have a final resolution date that may range from a day to many months in the future. Challenge participants will be asked to produce forecasts until the question resolves. 
  • Examples of potential questions*:
  • Will the WHO confirm more than 10 cases of Marburg in 2018? (Yes/No)
  • Before March 2018, with South Korea file a World Trade Organization dispute against the United States related to solar panels? (Yes/No)
  • All forecasts will need to be submitted through the API
  • Forecasts may be updated and will be scored on a daily basis for each question that is open.

Scoring

Scores will be calculated using a metric which incorporates forecast accuracy and confidence. This metric is based on the Brier score, a quadratic measure of the forecaster's closeness to ground truth for the outcome that occurs. For example, an accurate forecast with 90% confidence for an event that occurs will score four times better than a forecast with 80% confidence. The score will be calculated on a daily basis over the duration of the question; those daily accuracy scores will then be averaged. The more accurate and timely the forecast, the better score.

As questions will range in duration, milestone check-ins for questions resolved during a certain period will be used to award progress-based incentives. This means solvers who start after the challenge has launched are still able to compete for some of the prizes. Finalists who wish to compete for monetary prizes will need to provide the solution package for review the GF Challenge Team at the end of the challenge, or as appropriate, if requested. Solutions will need to be documented and all data streams used in prize-winning solutions will need to be outlined and verified by the GF Challenge Team. Please refer to guidelines for a description of scoring.

Prizes

Solvers will be able to compete on their own or form teams to compete for a total prize purse of $200,000. The prize amounts shown are dependent upon certain criteria and/or participation requirements. Challenge results and solutions will be reviewed at an IARPA forecasting Workshop in Washington, D.C. For additional details and prize requirements, please refer to the guidelines. Prizes will be distributed for the following criteria:

  • Overall Performance Prizes
    • 1st Place - $20,000
      • 1st Place could be eligible to receive the Performance Bonus: Ultimate Forecaster, an additional $40,000*
    • 2nd Place - $15,000
    • 3rd Place - $10,000
    • 4th Place - $7,000
    • 5th Place - $4,000
      • Second through Fifth Place may be eligible for a share of Performance Bonus: Star Forecasters, an additional $30,000*
  • Additional Incentives
    • Best in Domain / Region Pair (X5) - $25,000 total
    • Best Undergraduate - $4,000
    • Milestone Performance (X3) - $30,000 total
    • Interim Prizes - up to $15,000

*Bonuses have additional eligibility requirements. Please see the challenge rules for details.

Rules and Eligibility

Individuals and teams, 18 years of age or older will be able to compete for cash prizes. Some individuals or organizations may not be eligible for monetary prizes. However, these solvers may, upon IARPA approval, participate in the challenge and be eligible for ranking on the leaderboard. Approval will need to be obtained in advance of participation by emailing: gfchallenge@iarpa.gov.

 

What Can You Do Right Now?

  • Click “Follow” to receive challenge updates and follow its progress!

  • Register for the challenge by clicking "Accept the Challenge" and agreeing to the challenge rules and IP agreement.
  • Share this challenge on social media using the icons above. Show your friends, your family, or anyone you know who has a passion for discovery.
  • Start a conversation in our Forum to join the conversation, ask questions or connect with other innovators.
Guidelines
Timeline
Updates 29
Forum 222
Community 802
FAQ

Frequently Asked Questions

I don’t have an approach that I wish to enter in the GF Challenge. Is there another way that I can participate?

IARPA is interested in any method that advances the science of forecasting. This may include investigating how a combination of computer models and human judgments can be used to make accurate forecasts, but it need not. If indeed a very good human forecaster (or a small group thereof), can do just as well as hybrid solutions or the ACE baseline, then we have learned something. However, we believe this is extremely unlikely due to the number of IFPs in the Challenge; and the fact that methods will be compared, for many of the prizes, against a benchmark crowd of human forecasters using the validated state-of-the-art method from the ACE program. The administration of the Challenge is automated and secured by use of an API for IFP and baseline data distribution and forecast submission, so some programming/data science skills are necessary. The GitHub repository is our way of sharing documentation, some data, and example API code with the Solver community.

All teams must complete a prize distribution form prior to their first submission in accordance with the challenge rules. Here is the link to the distribution form. Once completed and signed by all team members, you will need to email the completed form to: .

Yes, this is a government funded project. The Intelligence Advanced Research Projects Activity (IARPA) envisions and leads high-risk, high-payoff research that delivers innovative technology for future overwhelming intelligence advantage.

In order to be fully registered for the Challenge, you will need to log into the HeroX GF Challenge page and “Accept the Challenge”. This includes accepting the Challenge rules and acknowledge that you adhere to the eligibility requirements for the Challenge, in addition to accepting the Intellectual Property (IP) Agreement. Once you have completed that step, you will be sent a link to the Cultivate Labs platform to register for your, or your team’s, API token.

HFC is an Intelligence Advanced Research Projects Activity (IARPA) program which is funding research teams to develop what we call “hybrid forecasting systems” to integrate human forecasters and statistical models in novel ways. The program is trying to harness the best of both worlds to produce forecasts that are better than either method could produce on their own. You can read more about the HFC program here: www.hybridforecasting.com

The GF Challenge is looking for forecasting methods; solutions that are capable of processing data and making forecasts. While this may include investigating how a combination of both computer models and human judgments can be used to make accurate forecasts, it need not do so. And while your solution need not be fully algorithmic in nature, you will need to submit your forecasts throughout the Challenge through an API, and be able to describe the solution for your forecasts at the end of the Challenge, so some data science literacy is likely required.

On the other hand, if you are instead more interested in making geopolitical forecasts using your intuition, you may find participating as a forecaster in HFC (www.hybridforecasting.com) to be more enjoyable. HFC is currently recruiting forecasters at the aforementioned URL.

Please note that you cannot participate in both the GF Challenge and HFC.

HFC teams are developing systems at a significant expense to IARPA. These teams are looking for honest experimental tests of their methodologies. IARPA is simultaneously looking for honest original solutions in the GF Challenge and we don’t want to see solutions that we are seeing in HFC. If we allowed dual-registration, there would be a possibility of Solvers accessing HFC performers’ intellectual property and using that in the Challenge. You will not be eligible for any prizes if you are registered for HFC. In order to remove yourself from participating in HFC, you need to send an email to stating your intentions and then register for the GF Challenge. By registering for the GF Challenge, you will be acknowledging that you are not concurrently participating in HFC; those who are awarded prizes will be asked to certify that they have not participated as a forecaster in HFC. 

This is a prize Challenge. This is not a traditional IARPA program. The Solvers are the ones doing the development. You are competing for prize money, not for contracting opportunities. 

For the GF Challenge, we are interested in seeing the solutions that individuals or teams can produce that are capable of intaking data and generating forecasts. To participate, you should use your product or tool to submit forecasts in response to the Individual Forecasting Problems (IFPs) that are issued during the Challenge. If you’d like more information on doing business with IARPA outside of this Challenge, you can visit the following link: https://www.iarpa.gov/index.php/working-with-iarpa

We anticipate asking between 125 to 175 IFPs over the course of the Challenge. All IFPs will be designed to resolve by the end of the competition.

You do not need to forecast against every IFP, but in order to be prize eligible, you will need to meet certain minimum targets. You can read more about the minimum participation requirements for each prize category in the prize section within the Challenge Rules Document, as well as finding information on how your responses, or non-responses, may impact your score in the Scoring section of the Challenge Rules Document.

Each prize category will have some minimum participation percentage requirements of IFPs. For the Overall Performance Prizes, it is 70% participation of all IFPs, for the milestone prizes it is 80% of the IFPs within that milestone period. Of the IFPs launched within that period of time, you must meet the minimum participation percentage associated with that prize category. When it comes to which IFPs you provide forecasts for, that is up to you. If you are thinking of using just human judgments, we discourage you from doing that. In order to submit forecasts, you must have the ability to interact with the API. If you would like to use your judgments or purely human judgments, then we recommend that you register for participation as a forecaster in IARPA’s HFC program at http://www.hybridforecasting.com.

Yes, every day is a scored day. The Challenge Rules Document describes what forecast days are.

Each day, there is a reporting deadline: 14:00:59 U.S Eastern Time. You will be scored by what you have as a forecast in the system at that time. We will impute or assume your forecast is the control group forecast. Once you initially submit, it will take the place of the control group forecast. 

There will be some variation between IFPs. We might ask a question on the possible outbreak of some conflict that will only be a day long. There may be some other questions where the maximum possible duration could be up to six months. The time period that a forecast will be open is a wide variation of time dependent upon the type of IFP.

Your score will be tracked on a leaderboard on the Cultivate Labs Platform. You can log into your account or Team account on the platform to see how well you are doing in the overall Challenge, or on the individual prizes/ milestones throughout the Challenge.

Once it has been determined that you have won a prize, you will be contacted via email with instructions on where to send your tax and payment information in order to receive your funds. As a reminder, all taxes that may be incurred based on the award of prize money is the sole responsibility of the Solver.

You can still participate in the Challenge! As long as you meet the minimum participation requirements for the prizes, you will still eligible to win at least some of the prizes. Note that some of the prizes have lower IFP participation requirements, and others are based on success across discrete periods of time. All is not lost if you join later in the Challenge!

You will retain rights to whatever methods you use to generate your forecasts. IARPA is supportive of further scientific research per the Office of Science and Technology Policy (OSTP) memorandum of 2013 and we would ask that you send us a courtesy copy of the publication if you cite the Challenge or IARPA within it, but you retain your IP rights, including the solutions you developed or used to make your forecasts.

The Government does not own the methods that you develop for this Challenge. We retain a copy of the forecasts that you provide and a description of your method but do not own the IP. You should have no expectation that the Government will use or others will use your methods. It is possible if you create something potentially valuable, that you may be approached from people who are interested in teaming with you on various types of efforts.

Yes, with a small number of exceptions, international Solvers are allowed to participate in this Challenge. Within the Challenge Rules Document, there is a section outlining the guidelines as to which countries are allowed to participate, and additional guidance. If you still need clarification on your eligibility to participate, please send an email to: .

No, you will need to select one of the events to participate in. IARPA’s HFC Performer team members and HFC Good Judgment Forecasters, as well as members of such persons’ immediate families (spouses, children, siblings, parents), and persons living in the same household as such persons, whether or not related, are not eligible to participate in the competition.

Yes, with a few stipulations. First, they must forego the prize money for the Challenge. Second, the entity, along with anyone working on the Challenge, must meet the other eligibility requirements in the Challenge Rules Document. Third, they may not have supported IARPA’s HFC Program in any manner. Please see the Challenge Rules Document for complete information on how to proceed.

Once the Challenge launches on HeroX, you will have an opportunity to register as a team and invite members to join the competition and join your team. However, if you decide to create a team after you have already registered, you have the ability to create a team within the community tab of the HeroX GF Challenge page. All team members must meet the general eligibility requirements, register and acknowledge the Challenge rules through the HeroX GF Challenge page. Upon registering for the Challenge, the team captain will need to provide a breakdown of all team members along with the agreed upon prize percentage allocation between all team members. Each team will need to determine who will be setting up their team’s API token, as there can only be one API token granted per Solver/Team. You are only permitted to participate in the Challenge as an individual or as part of one team (membership on multiple teams is not allowed).

No, participants can only join a single team in the Challenge and can either be on a team or be a lone Solver, but not both.

No, once you have registered for a team and your team has begun making forecasts, you cannot switch teams, or join a new team. Similarly, if you are a lone Solver and have begun making forecasts, you can no longer join a team.

Yes, individuals can form teams after the first IFP is issued, but you cannot form a team after you have submitted a forecast as an individual or the intended team has submitted a forecast. The IFPs will come out on March 7 — do not start forecasting until you have your team solidified. If you are thinking about teaming with someone, please wait to submit your forecast until you determine how you would like to participate in this Challenge (as part of a team or as an individual.)

We suggest that you post to the HeroX forum in order to seek out other team members.

Teaming presents an opportunity to divide and conquer the labor. However, many individuals prefer to wotk independently. It is up to your personal preference and working style.

These IFPs are developed by an experienced team working for IARPA who developed similar questions for the earlier IARPA Aggregative Contingent Estimation (ACE) program, which spawned Phil Tetlock’s book Superforecasting. Each IFP includes an authoritative source and/or procedure for determining the outcome. In our experience, we have found these IFPs to be well posed and to have unambiguous outcomes.

The IFPs have already been segmented into domains and regions by the IARPA team. The IFPs affect both the GF Challenge and HFC so there is not an opportunity for requests/nominations from Solvers.

Challenge Solvers will have access to a stream of de-identified individual-level human forecast data. From this, an aggregate will be constructed using a method based on the state-of-the art from IARPA’s ACE program—Solvers will also be given access to this aggregate. The aggregate measure will serve as the baseline against which NBPs will be calculated. This baseline (and the associated de-identified individual forecast data) will be available to Solvers via the Cultivate Labs API.

Each Solver is given an authentication code in the form of an API token upon registration. Forecast submissions are authenticated with this code and are not visible to others. If you keep your API token secure, you should have nothing to worry about.

All submissions must be made through the API provided by the Cultivate Labs Platform. Those wishing to make smaller numbers of forecasts using human judgment are encouraged to register for participation as a forecaster in IARPA’s HFC program at http://www.hybridforecasting.com.

Once you accept the Challenge Rules and IP Agreement for the Challenge on the HeroX GF Challenge page, you will be sent a link to register for your API token and access to the Cultivate Labs Platform. Please note that only one API account will be allowed for each lone Solver or team of Solvers.

No. Each team is only allowed to register for a single API account for the Challenge.

Log into your API account on the Cultivate Labs Platform and from the Home Page click the “View API Token Management” button to retrieve or re-generate an API token.

The complete API documentation will be available on the Cultivate Labs Platform upon registering for the Challenge on HeroX.

The Challenge will only provide all solvers with a single data set, which is the human forecaster data. We are not providing any additional data beyond that, as we believe this data is state-of-the-art for these types of questions. We understand that not all Solvers may want to use this data. Solvers are able to use additional or alternative data sources that they deem useful to their solutions.

The data from the ACE program is available for download in DataVerse. The ACE data - the benchmark method as well as the data that we are providing you is based on systems from IARPA’s ACE program - there will be a leader board in the Cultivate Labs site where this Challenge is being hosted. 

The final score will be based solely on the accuracy of your forecasts — you will have to turn over a document that describes your method. While the Government will gain rights to the forecasts, you will retain the rights to your IP. Your score is based on the quality of your forecasts.

We will make the exact requirements clear, but the expectation is that the document that describes your solution will be no more than four pages long. We may request more information to gain a better understanding of your method or to provide clarity. Any parts of your solution that are proprietary need to be clearly marked as such. In the event that you are a prize winner, we need to be able to describe your methodology in greater detail to a public audience so that the public can have confidence that the prize funds were awarded properly.

The results of this Challenge will be publicized through public communication channels and updated on the microsite.

Yes, but it’s quick and easy. Just click the “Join Us” button at the top of the page and follow the instructions to complete your registration. All you need to provide is your name and email address.

If you have a question not answered in the FAQ, we recommend that you post it in the Forum where someone will respond to you. This way, others who may have the same question will be able to see it.

The challenge is expected to officially launch in February 21, 2018. Please follow this challenge to get updates for when the challenge launches and more information will become available.

Once the challenge has officially launched, you will be directed to the challenge platform where you can download the API key.

For questions involving topics and events that are widely reported in the media, we consult credible open source media sources (e.g., Reuters, BBC, AP) to determine the outcome of a question. Some questions may specify that the resolution will be determined by specific sources (e.g., UNHCR data, Moody's). When in doubt, please check the information provided in the “More Info” section of each question for guidance specific to that question.

Resources