James Thorne
Oct. 21, 2016
4:56 a.m. PDT

Accuracy and Scoring

Hi

I have question relating to how the accuracy of systems will be scored for this challenge.

Currently, on the practise quiz, there are four answers for truthfulness. However, in testing some of these, I've found that it isn't always possible to accurately fact check the statements. Take for example: "There are 3 hospital beds per 1,000 people", while you state this factoid is true, this assumes that you know which country/hospital/location is being discussed.

Because of this, the system has two options. Either guess an answer or to return "I don't know". If our system is unable to confidently fact check a fact, is it possible to skip or pass this question without harming the overall accuracy score?

Best,
James
3 Replies

Ovidiu Dobre
Oct. 21, 2016
6:49 a.m. PDT
@James Thorne
Great question!

Mevan Babakar moderator
Oct. 24, 2016
5:24 a.m. PDT
Hey James,

That's a really great question. We joined the process late in the game and would have liked to see an "I don't know" or "not applicable" option. But as that wasn't possible, what we decided on was to make sure that the context was encapsulated in the claims themselves. You're right to point that one out. I've had a look through more thoroughly and spotted some others. This is the offending list and they will all be updated at the end of today:

From: "There are 3 hospital beds per 1,000 people"
To: "In the USA there are 3 hospitals beds per 1,000 people"

From: 5.0% of labor force is unemployed
To: 5.0% of the US labor force is unemployed

From: White women are nearly half as likely to be unemployed than black women
To: In America white women are nearly half as likely to be unemployed than black women

Hope that helps! Good luck.

Mevan

Kyla Jeffrey HeroX team
Oct. 24, 2016
12:22 p.m. PDT
Hi Everyone,

I've updated the Challenge Page to include the extra context provided by Mevan.

Thanks so much!
Kyla
Post Your Reply
Let these people know about your message