menu
Kyla Jeffrey HeroX team
Sept. 14, 2018
8:53 a.m. PDT

Locking the Leaderboard

Dear Solvers,

Please note that the final scores after the last IFP is resolved are final. That will be the end of the challenge calculations, even if new information is surfaced which changes the outcome of the IFPs after that date.  Currently, as we’ve seen in a few instances throughout the challenge, it is possible that an IFP may be re-resolved if new information leads to a different resolution than was initially used.  Once the last IFP is resolved, we will lock the leaderboard at that time and those will be the final standings for the end of the challenge, even if the database/information sources are updated post-challenge.

Thank you,
GF Challenge Team
22 Replies

anna buczak
Sept. 24, 2018
12:28 p.m. PDT
So when will the final standing be known?

Lars Ericson
Sept. 24, 2018
2:07 p.m. PDT
We are locked in this state: https://youtu.be/4NwP3wes4M8
Attachments

Lars Ericson
Sept. 24, 2018
5:04 p.m. PDT
And...bingo. TruthLover got their extra bump, LIA got pushed off the bubble:
Attachments

Lars Ericson
Sept. 24, 2018
5:17 p.m. PDT
It looks like BEEFERS grabbed the Ultimate Forecaster award, and nobody got Star Forecaster:
Attachments

Lars Ericson
Sept. 24, 2018
5:21 p.m. PDT
If they decide to pay Power Forecaster awards (because nobody as far as I can tell got Star Forecaster), then there are more candidates under 33.333% positive rule:
Attachments

Lars Ericson
Sept. 24, 2018
5:25 p.m. PDT
I wish there was an 100% Participation award (there isn't), because I'd win that one hands down (but don't rely on my forecasts, they're twice as bad as a coin toss):
Attachments

EagleWings
Sept. 24, 2018
6:07 p.m. PDT
Thanks, Lars. :)

EagleWings
Sept. 24, 2018
6:12 p.m. PDT
I think to get the Ultimate Forecaster bonus, one has to be the number one on the overall leaderboard. Wish Power Forecaster awards be split. :)

EagleWings
Sept. 24, 2018
6:41 p.m. PDT
My understanding is that whether a method beats benchmark or not, the GFC leaderboard should be used, in that case, only my team and DigitalDelphi have the chance to share the Power Forecaster awards. Wish I am right on this one. :) Of course, the GF Challenge team has the final say.

Lars Ericson
Sept. 24, 2018
6:46 p.m. PDT
@truthLover, the rules document: https://www.dropbox.com/s/d8r5u2mdui02z04/GF%20Challenge%20Rules%20%28Updated%208.30.18%29.pdf?dl=0
says Ultimate Forecaster is (a) 1st place, (b) beats Benchmark and (c) beats the HFC Top Score. So that references HFC Leaderboard.
The 2nd Chance update: https://www.herox.com/IARPAGFChallenge/update/2233
for Power Forecaster also references HFC Leaderboard, it just lowers the %positive requirement to 33 1/3rd %.

NG
Sept. 24, 2018
10:11 p.m. PDT
@truthLover congratulations !! and to all other top 5 teams :-)

Ian Pitchford
Sept. 24, 2018
11:30 p.m. PDT
Why is the leaderboard based on only 165 of the 187 questions resolved? Also, is the 1-WeightedLogit score based on Carbon plus Mechanical Turk forecasters? I’m trying to get an overview for a comparison between HFC and the original ACE tournaments based on the published papers and the book by Tetlock and Gardner, but it’s difficult: even the number of IFPs is not clear. During ACE there were 104 questions in the first year and 199 over two years, which suggests there were 95 in year 2. There were 140 questions in year 3 and “almost 500” in total, which suggests ~160 in year 4. In year 1 the combined Brier score was 0.37 for “all other forecasters” and 0.25 for superforecasters. The latter group were the top 2% in any one year. In year 1 the best score was 0.19. In year 2 the best score was 0.14, by Doug Lorch, and he was the only person to beat the Logit algorithm. In year 3 Tim MInto came top with a Brier score of 0.15. I haven’t found a top score for year 4. Basically, I don’t know what a superforecaster is and I can’t work out what is being compared to what.

EagleWings
Sept. 25, 2018
6:01 a.m. PDT
@NG, thanks!

EagleWings
Sept. 25, 2018
6:03 a.m. PDT
@Lars, maybe you are right. Let's see how the GF Challenge team hands out the Ultimate Forecaster bonus, and/or the "leftover". :)

EagleWings
Sept. 25, 2018
6:20 a.m. PDT
@Lars, after reading the rule again, I think my understanding was correct. -- 1. Must achieve "the 1st Place Overall Prize" (which is based on GFC Leaderboard). 2. Must beat the GF Challenge baseline (it does not tell which Leaderboard to use). 3. Must beat the HFC top score (This one is clear)

EagleWings
Sept. 25, 2018
6:22 a.m. PDT
Note it says "the 1st Place Overall Prize", not simply "the 1st place" which could refer to either Leaderboard.

Lars Ericson
Sept. 25, 2018
6:45 a.m. PDT
@truthLover
Yes since top GFC winner is below the benchmark, I don't think anybody gets bonus prizes so we will have to wait and see whether they fudge the rules again to pay out their award budget or roll the budget forward to next season. But they haven't said at all that there will be a next season for GF. There is definitely a next round for HFC, it's got a couple of years to go.

Lars Ericson
Sept. 25, 2018
8:21 a.m. PDT
@NG congratulations! They made the Presenters Award cut based on 9/23 levels before the last IFP was resolved on 9/24: https://www.herox.com/IARPAGFChallenge/update/2312 I look forward to your talk!

NG
Sept. 25, 2018
9:40 a.m. PDT
@Lars Ericson thanks Lars. I look forward to meeting you but I am a bit far (Switzerland). In any case, someone from our team will surely be there to give the talk.

Lars Ericson
Sept. 25, 2018
10:20 a.m. PDT
@NG
So can I guess that you are with EPFL LIA? According to Google Flights, United will get you from Geneva to Dulles for $1300 round trip. That leaves $1200 over for hotel and coffee (IARPA requires you to sign a form acknowledging that they won't serve coffee; apparently someone close to them criticized the flavor, and now they refuse to bring it out at gatherings). Don't you want to get out of the house and see the cherry blossoms on the Washington Mall?
Attachments