menu
Lars Ericson
April 19, 2018
11:17 a.m. PDT

Position in Leaderboards versus Logit

@Ben Roesch, I have 0 methods in GFC Leaderboard better than consensus 1-WeightedLogit-PercentRecent, which is at position 45. However, on Beat HFC Leaderboard, I have 2 methods, at positions 49 and 71, better than 1-WeightedLogit-PercentRecent which is at position 72.

How does the math work on that?
8 Replies

Lars Ericson
April 22, 2018
6:32 p.m. PDT
@IARPA, come on guys, this is an easy technical question. You've got A>B and then B>A, please explain.

Lars Ericson
April 23, 2018
11:12 a.m. PDT
Hello hello.....

Anna Buczak
April 23, 2018
1:21 p.m. PDT
@IARPA
I have a similar question. Definitely there is no clarity how the BeatHFCLeaderboard is computed vs the GFCLeaderboard.
We have our method 22 ranked as 21 on GFCLeaderboard and this is our method that has the highest rank on GFCLeaderboard. On BeatHFCLeaderboard our method6 has the highest rank (19). If I look for method6 on GFCLeaderboard it has rank 42 (well below our method 22). This does not seem to make any sense.
It would be great if we could get an explanation how exactly methods are scored on BeatHFCLeaderboard, and why the order is different than on GFCLeaderboard.

Kyla Jeffrey HeroX team
April 23, 2018
2:41 p.m. PDT
@Lars Ericson @ Anna Buczak
The GFC Leaderboard uses Net Brier Points. The Beat HFC leaderboard uses Mean Daily Brier scores, as referenced in the guidelines under Judging and Scoring #4 (Mean Daily Brier score referenced in Appendix A.3 of challenge rules.)

Lars Ericson
April 23, 2018
2:43 p.m. PDT
@Kyla Jeffrey
What criterion is used for prizes, MDB or NBP?

Lars Ericson
April 24, 2018
5:59 a.m. PDT
@Kyla Jeffrey
Put another way, my reading of NBP is that NBP(SolverMethod)=MDB(Consensus)-MDB(SolverMethod). GFC Leaderboard reports MDB(x). GFC Leaderboad reports NBP(x). Beat HFC Leaderboard reports MDB(x). If MDB(SolverMethod) > MDB(Consensus) then NBP(SolverMethod) < 0 and SolverMethod will appear below Consensus on GFC Leaderboard. Also SolverMethod will appear below Consensus on Beat HFC Leaderboard. The order relationship is the same. How do we end up switching order?

Kyla Jeffrey HeroX team
April 24, 2018
11:49 a.m. PDT
@Lars Ericson
This depends on the specific prize, as described in points 3 and 4 under "Judging and Scoring" in the Challenge rules. "Beat the Best of HFC" uses mean MDB and the associated leaderboard reflects this, all other prizes use Net Brier Points.

For the question regarding "order relationship between the two leaderboards": The mean MDB metric treats all IFPs as equal, regardless of duration. The NBP metric is computed on an IFP-day basis. A longer duration IFP provides more opportunities to earn NBP, especially with an early accurate forecast that differs from the benchmark. Performance under the two metrics is somewhat correlated but not perfectly so. Both reward accurate and timely forecasts.

Lars Ericson
April 24, 2018
12:10 p.m. PDT
@Kyla Jeffrey
That's a little non-obvious because Brier Score is already an average over time. So mean daily Brier Score is a mean of a mean. Sometimes means of means are just means, but I guess not in this case. If you're saying that questions that last 20 days count twice as much in the scoring as 10 day questions, OK, but that doesn't jump out at me in the scoring document. https://s3-us-west-2.amazonaws.com/herox-production/clientspecific/iarpa/GF+Challenge+Rules_.pdf

It's OK, I'm not fussy, I pretty much get it. It would be much better if the website had prize-specific leaderboards and some automated report showing exactly how a score is calculated. I guess you don't have budget for that so I'll let it go. However I and I guess other solvers would certainly appreciate more objective performance/scoring information rather than less. Feedback is motivating and retaining.