Yes, that's exactly my point, I've been saying it over and over. Last year, you got to pick and choose somewhat: you had to answer at least 70% of questions in the highest scoring method. It happened to be the case that, last year, if you scored people against 100% of questions, I won. Against everybody, including all the HFC Performers. This year, it happens to be the case (and I wouldn't mind taking a little credit), that they are going with the 100% rule: you are scored against every question, whether you answered it or not. For all methods. And they're leaving the Performers out of it this time. (For the Performers that are left, I think one or two got knocked out. ) "Relevant" means that the question is in the live portal (not staging or training) and hasn't been voided. If you're not answering every question, y'all got work to do.
The origin (in my mind) of this idea is to imagine yourself as the analytic component of an intelligence agency: You don't get to pick and choose what questions come in. You have to work on every question that comes in. So the fair measure of a system is not how well it does on the questions it thinks it can handle. The fair measure is against all the questions it is told to handle.
I'm not boasting for this year, by the way. I'm near the bottom of the Leaderboard. But I'm doing my homework. Maybe I'll get back up towards the top when I'm done with it: https://www.linkedin.com/pulse/parsing-ifps-iarpa-gfc2-lars-ericson/
Modified on July 5, 2019, 4:06 a.m. PDT