menu
Kyla Jeffrey HeroX team
Sept. 4, 2018
10:30 a.m. PDT

Announcing the GF Challenge Final Workshop

Dear Solvers,

As noted in the rules update last week, all Solvers are invited to attend the GF Challenge Final Workshop taking place in the Northern Virginia area from October 23-24; this workshop runs in conjunction with the HFC Principal Investigators’ meeting.

The top 5 Solvers on the leaderboard on September 23rd will be invited to present their solution at this Final Workshop. These Solvers will receive a $2,500 Workshop Presenter Prize, as outlined in the updated Rules Document. Please keep an eye out for an email announcing the winners of this Workshop Presenter Award.

While we only have a few presenter slots, we invite all GF Solvers to join us at this special event in celebration of your participation in the GF Challenge. It’s a great opportunity to network with HFC performers, IARPA officials, and fellow Solvers. More information on the event, schedule, and invitations to come.

Cheers!
GF Challenge Team
24 Replies

Lars Ericson
Sept. 4, 2018
11:13 a.m. PDT
@Kyla Jeffrey
Which Leaderboard? Depending on whether you pick GFC or Beat HFC with 70% or 80% and with or without %positive you'll get a different set of presenters.

Since there are only 7 who beat the benchmark at the 70% participation level, and only 10 in total above 70% level participation, I think you'd get more bang for your buck by inviting everybody who has participated 70% or more to present.

Lars Ericson
Sept. 4, 2018
6:13 p.m. PDT
To give you a sense of this, here's GFC Leaderboard with 70% participation. Top 5 are sudo, Cititzens, truth, delphi and catskills:
Attachments

Lars Ericson
Sept. 4, 2018
6:16 p.m. PDT
Now if we add the 50% positive requirement, the order changes to delphi, Beefers, sudo, truth and catskills. So we add Beefers and we lose Citizens:
Attachments

Lars Ericson
Sept. 4, 2018
6:18 p.m. PDT
Switching to Beat HFC Leaderboard, for 70% participation, the order is seb, Citizens, sudo, Beefers and truth. In this case you lose Delphi and catskills:
Attachments

Lars Ericson
Sept. 4, 2018
6:20 p.m. PDT
But then if you add the 50% positive requirement, the order is Beefers, delphi, sudo, catskills and truth, and you lose seb:
Attachments

Lars Ericson
Sept. 4, 2018
6:32 p.m. PDT
The different rankings result from luck of the draw but could also be interpreted as variously optimizing Net Brier Points, % Positive and Average Brier Score. If you take an inclusive view and take the union of the top 5 in each case, the speakers would be BEEFERS, DigitalDelphi, sudo-bang-bang, catskills, truthLover, seb, CitizensBand. Give 7 teams 15 minutes each with a few coffee breaks and that's just 2 hours to spend covering what I hope would be a lot of different thinking on this topic. Adding 3 more slots gets you the whole crew. That would be LIA, pb2pv and dobreovidiu who had participation rates of 80%, 90% and 98%. Their scores aren't really that much different from everybody elses, and everybody beats Performer 2 from HFC. So given that all 10 GFers beat at least one HFCer, I really think that we should hear from everybody that stayed till the end, because they may have unique insights that are worth hearing: at least as worthy of workshop time as Performer 2. The total cost for inviting all finalists is $25,000. This is a very small fraction of the cost allotted to the HFC Performers. It's not clear from this picture that they are bringing any magic to the table that the GFers have not:
Attachments

Kyla Jeffrey HeroX team
Sept. 5, 2018
12:42 p.m. PDT
To help clarify things… we’ll use the same guidelines applied to the Overall Prizes to determine the Top 5. As a memory refresher, that means an examination of 70% participation and 50% positive results. Let’s say fewer than 5 people hit that criteria, then we will apply the Second Chance Tier metrics—70% Participation and 33.3333… % Positive to reach the 5 slots.

Calculations will be based on the current state of the leaderboard at 5pm on September 23rd as the leaderboard may continue to evolve with IFPs that resolve or update after that time.

Looking further down the line at the workshop. We appreciate everyone’s participation in the challenge, as well as their creative submissions. The workshop itself is open to the GF Solvers, so they can connect and exchange experiences. However, in the interest of time, presentation opportunities are limited to top leaderboard winners, and will be sent direct invitations to share their solutions in the near future. Both guest and presenter-level participation are important to continuing our strong challenge information exchange.

Lars Ericson
Sept. 5, 2018
3:46 p.m. PDT
@Kyla Jeffrey
There are two Leaderboards, GFC and Beat HFC. Which one is being used for Overall Prizes? Also, for the 70% participation 50% positive, does it matter if we get 50% positive but our NBP is below the Benchmark? This is the current situation for all solvers in the following picture on the GFC Leaderboard:
Attachments

Ian Pitchford
Sept. 7, 2018
8:09 a.m. PDT
@Kyla Jeffrey
Is there any chance of a livestream and/or upload of key presentations to the ODNI YouTube site?

EagleWings
Sept. 11, 2018
1:03 p.m. PDT
@Lars Ericson
Lars, I think to get bonus (Ultimate, Star or Power forecasters), you need to beat the benchmark, otherwise, you do not have to.

Lars Ericson
Sept. 11, 2018
2:32 p.m. PDT
@truthLover
I concur. Currently I think only Delphi will be above the benchmark at the 2nd chance level (33%). So zero 1st chance winners, 1 second chance winner, and nobody for the Ultimate, Star or Power prizes.

My opinion is that you and all of the 10 teams that participated at 70% level or above should be heard as speakers at their 2-day therapy session, for the reasons I outlined above. Since we all beat the HFC performers under one metric or another, what we have to say is as technically interesting as anything the HFC performers might say. I guess they don't agree. Kyla's response above was still unclear on the details for the 5 they will pick. Based on that, I wrote this post, so far unanswered: https://www.herox.com/IARPAGFChallenge/forum/thread/3436

EagleWings
Sept. 11, 2018
6:19 p.m. PDT
Lars, we have slightly different understandings. I think on the final leaderboard, those top five satisfying 70% &50% will get first chance prizes, but not likely to get Ultimate or Star bonus; those satisfying 70% & 33.33% will get second chance prizes (if less than 5 satisfying 70% & 50%), and get Power forecasters bonus if beating benchmark and human performers (if Ultimate and Star bonus go to no one) .

Lars Ericson
Sept. 11, 2018
7:20 p.m. PDT
I guess you are right, looking at the Rules document (https://www.dropbox.com/s/d8r5u2mdui02z04/GF%20Challenge%20Rules%20%28Updated%208.30.18%29.pdf?dl=0), it appears that only the bonus awards have the extra condition "Must beat the GF Challenge Baseline", plus "Must beat the HFC top score". I don't think the bonuses will get paid.

NG
Sept. 12, 2018
4:17 a.m. PDT
are there more IFPs that have not been resolved yet ? Or the current leaderboard position is more or less final ?

Lars Ericson
Sept. 12, 2018
4:42 a.m. PDT
I see 2 open questions from the download, 1055 Australia Flu and 1397 Brent Crude 06SEP. What's your team name? If you're on the bubble these last two resolutions should be exciting.

Ian Pitchford
Sept. 12, 2018
4:44 a.m. PDT
We’ll have the oil data later today and the flu data on 24 September. They are going to give us “human data stream” folks our overall Brier scores by the end of September.

Kyla Jeffrey HeroX team
Sept. 14, 2018
8:54 a.m. PDT
Hi there, 

There is one IFP left to resolve. The last IFP will come through by the end of September. Please note the leaderboard is subject to change based on the resolution of IFPs until the final IFP has been resolved.   The Invitation to the Workshop will be based on the GFC Leaderboard.  https://www.herox.com/IARPAGFChallenge/forum/thread/3444

Thank you!
GF Challenge Team

NG
Sept. 14, 2018
3:01 p.m. PDT
@Lars Ericson Thanks for the information. Yes, the last resolution will be exciting indeed. I am team LIA.

Lars Ericson
Sept. 14, 2018
4:26 p.m. PDT
@NG are you coming to the Workshop? You're on the bubble now but you've got a good shot at staying in the top 5. Even if you miss the bubble, they've opened up Lightning Talk spots so you'll be able to talk anyway if you feel like sharing. Also it looks like you'll be winning the Econ/Europe domain pair (https://bit.ly/2tGueNk ), which is extra motivation.

Ian Pitchford
Sept. 15, 2018
1:43 a.m. PDT
The Australian flu count for IFP 1055 is still only 255 though. The notes say “This question will be resolved by consulting the source between 10:00 AM and 4:00 PM ET starting on 24 September 2018 and each Monday thereafter until data are posted. Resolution will be based on the value observed at the time of source consultation.” Full data for weeks 32 and 33 isn’t available yet but it will be very surprising if the number climbs above 1000.