menu
Lars Ericson
Sept. 10, 2018
4:30 p.m. PDT

Please reconfirm the definition of "The Leaderboard"

@Kyla @IARPA, in this post: https://www.herox.com/IARPAGFChallenge/forum/thread/3419 it says "The top 5 Solvers on the leaderboard on September 23rd will be invited to present their solution at this Final Workshop." In this post: https://www.herox.com/IARPAGFChallenge/update/2109 it says "We've updated the Leaderboard to show who is at the top of the Leaderboard. This means those at the top have participated in and achieved: All resolved IFPs with 70% or (50% participation for undergrad) and positive NBPs on over half of the IFPs attempted. Please view the overall Leaderboard here: https://bit.ly/2KnAroA" The current view of The Leaderboard (as https://bit.ly/2KnAroA") has Delphi, Beefers, catskills, sudo and truth as the top 5 solvers with 70% participation and %positive > 50 (as seen in the picture below). Please confirm that this is a correct reading of the rules and communications and that this is indeed The Leaderboard.
Attachments
10 Replies

Kyla Jeffrey HeroX team
Sept. 14, 2018
8:55 a.m. PDT
@Lars Ericson
Hi Lars,

Those links show a minimum of 50% positive.   However, it is important to note that a method needs to be over 50% positive, so the leaderboard link can be adjusted to this link (https://bit.ly/2QuIumY) in order to see who is currently in the leaderboard slots.  It’s important to note that the leaderboard may shift if any of the IFPs re-resolve prior to the final IFP resolution at the end of the month.  See forum post about the final leaderboard here: https://www.herox.com/IARPAGFChallenge/forum/thread/3444

Thank you!
GF Challenge Team

Lars Ericson
Sept. 14, 2018
9:05 a.m. PDT
@Kyla Jeffrey
Are you 100% certain about your link https://bit.ly/2Md9PHc ? When I follow it I get the picture below. Which I don't mind at all, I'm in 2nd place, but it's not what I expected. It is constrained to an April-June period and there are no participation requirements.
Attachments

Kyla Jeffrey HeroX team
Sept. 14, 2018
9:57 a.m. PDT
@Lars Ericson
Good catch -- here is the correct link: https://bit.ly/2QuIumY. Thanks, Lars.

EagleWings
Sept. 14, 2018
10:42 a.m. PDT
One question, can you explain the different results between GFC Leaderboard and Beat HFC Leaderboard with same parameter 70% &30%? There are two methods better than Benchmark on the former, while 13 better than Benchmark on the latter.
https://discover.iarpagfchallenge.com/leaderboards/relative_brier_scores?best_method=false&min_questions_attempted=70&min_percent_positive=30
https://discover.iarpagfchallenge.com/leaderboards/brier_scores?min_percent_positive=30&min_questions_attempted=70&best_method=false
Thanks!

Lars Ericson
Sept. 14, 2018
11:30 a.m. PDT
@truthLover
One is sorting on net brier points and the other on average daily brier points. You may be expecting

GFC.NetBrierPoints(Method) = HFC.Brier(Benchmark)-HFC.Brier(Method)

However, I think this is not the case, because one leaderboard uses Mean Daily Brier and the other uses Average Daily Brier or something like that, which are like oranges and grapefruit. There was some discussion of this very early on here: https://www.herox.com/IARPAGFChallenge/forum/thread/2870?page=1#post-11124 If that ends up making any sense to you, maybe you can post a summary recap here for the rest of us!

Also, I see you are currently getting burned by the 50.01% thing Kyla announced above in the thread. Good luck on the last question! Here the value judgment is whether 98%/50.00%/-8.14 is lower performance than 85%/53.62%/-12.24. Hard to say! Clearly it seems like where there is a multi-axis metric, it might be easier on people's nerves to combine it into a single number with some formula.

EagleWings
Sept. 14, 2018
11:46 a.m. PDT
Thanks, Lars, I will take a look at the link. Also, thank you for your kind words!

Lars Ericson
Sept. 14, 2018
12:01 p.m. PDT
@truthLover
I hope to see you at the Workshop anyway even if you get burned. They are offering Lightning Talk spots for people out of the top 5. You're at the top of the board for a couple of Domain awards (https://www.herox.com/IARPAGFChallenge/update/2109), so you should have enough cash for a luxury tour of Fairfax County.

EagleWings
Sept. 14, 2018
12:16 p.m. PDT
Again, thank you, Lars. Due to schedule, my team may not be able to attend this Workshop.

Lars Ericson
Sept. 14, 2018
2:19 p.m. PDT
@truthLover, would it alter the attendance utility function if we could prevail on IARPA to bring in an ample supply of Duck Donuts each morning? https://www.duckdonuts.com/location/alexandria-va/

[*] And footnote: https://english.stackexchange.com/questions/464358/prevail-upon-or-prevail-on

EagleWings
Sept. 14, 2018
4:14 p.m. PDT
@Lars, perhaps, who knows? :)