explore
launch challenge
how it works
COVID-19
search

IARPA

 19,170
Credibility Assessment Standardized Evaluation (CASE) Prize Challenge

Credibility Assessment Standardized Evaluation (CASE) Prize Challenge

Develop a ground-breaking and innovative solution to evaluate credibility tools and techniques
Government Technology
stage:
Won
prize:
$125,000
Government Technology
more
Overview Guidelines Timeline Updates 12 Forum 11 Community 366 Press FAQ Rules
Overview

Challenge Overview

Every day we make decisions about whether the people and information sources around us are reliable, honest, and trustworthy – the person, their actions, what they say, a particular news source, or the actual information being conveyed. Often, the only tool to help us make those decisions are our own judgments based on current or past experiences.

For some in-person and virtual interactions there are tools to aid our judgments. These might include listening to the way someone tells a story, asking specific questions, looking at a user badge or rating system, asking for confirming information from other people - or in more formal settings, verifying biometrics or recording someone’s physiological responses, such as is the case with the polygraph. Each of these examples uses a very different type of tool to augment our ability to evaluate credibility. Yet there are no standardized and rigorous tests to evaluate how accurate such tools really are.

Countless studies have tested a variety of credibility assessment techniques and have attempted to use them to rigorously determine when a source and/or a message is credible and, more specifically, when a person is lying or telling the truth. Despite the large and lengthy investment in such research, a rigorous set of valid methods that are useful in determining the credibility of a source or their information across different applications remains difficult to achieve.

 

Challenge Focus

This challenge is focused on the methods used to evaluate credibility assessment techniques or technologies, rather than on the techniques or technologies themselves. In this context, a method is a detailed plan or set of actions that can be easily followed and replicated. 

In this challenge, we ask that your solution is a method for conducting a study, which includes background information, the objectives of the research, study design, the logistics and means for running the study, and details about what data would be collected if your solution were implemented. 

The Intelligence Advanced Research Projects Activity (IARPA) invests in high-risk, high-payoff research programs to tackle some of the most difficult challenges facing the US Government’s Intelligence Community (IC). An important challenge facing the IC is knowing who, or what, is credible. By sharing this challenge with the Hero-X community, IARPA seeks to motivate good ideas from people with diverse backgrounds. Successful solutions could be used to inform future research efforts, to help the Government evaluate new tools, and to develop a deeper understanding of what it means to be credible and how we can evaluate credibility across diverse domains – in person, in virtual spaces, and in the information and media we consume. 

The challenge of developing a useful evaluation of credibility assessment techniques and technologies lies in the method’s design for drawing out real behavior, credible or not, from an individual or other source and then having a valid means to test it. This can be difficult as many current techniques involve actors or games where individuals may not feel that they need to be honest or do not truly act as they would in a real-life scenario. How to Get Involved

 

  • Click FOLLOW above to receive an update once the CASE Challenge is fully launched.
  • Share this CASE Challenge on social media using the Share button above. Show your colleagues, friends, family, or anyone you know who has a passion for discovery!
  • Start a conversation in our Forum, ask questions or connect with other innovators.
View legal agreement
Guidelines
Timeline
Updates 12

Challenge Updates

CASE Challenge Stage 2 Winners

Aug. 6, 2019, 6:29 a.m. PDT by Natan Getahun

 

We are excited to announce the overall winners of the Pitch Presentation and the end of the Intelligence Advanced Research Projects Activity’s (IARPA) Credibility Assessment Standardized Evaluation (CASE) Challenge!

The Pitch Presentation, Stage 2, of The CASE Challenge was held during the Challenge Workshop on July 18, 2019 in Washington, D.C. The Workshop was a full day of presentations and panel discussions from subject matter experts across industry, academia, and government around credibility assessment. During the event, the Credibility Champions from Stage 1 pitched their Solution to a live panel of Judges and audience members.

Here are the winners of the Stage 2 of The CASE Challenge!

 

 

1st Place

Computer Augmented Social Engagement Lab team (Taylan Sen (Captain), Ehsan Hoque, and Kurtis Haut) - SPIDER2

SPIDER2 is an interview-based evaluation in which participants are required to construct a device and then are subjected to both an Interview and a Debrief. The Interview and Debriefing sessions are then used as sets of data for credibility assessment.

 

2nd Place

Jeremy Martinez, M.D. - Credibility: Assessing the Assessment (C2A)

C2A adapts the popular social game for use in evaluating credibility assessment techniques. The methodology applies of an engaging, socially embedded game, with natural elements of deception for use in credibility assessment evaluation.

3rd Place

Dr. Joseph R. Stainback IV - Strategic Cognitive and Mobility Room (SCAMR)

SCAMR employs an Escape Room concept and an Espionage scenario to incentivize participant deception and elicit reactions through a series of puzzles, riddles, and strategy.

 

 

Congratulations to the winners of The CASE Challenge! Each team presented a thoughtful and engaging solution. We are eager to see the impact of their Solutions to the future of credibility assessment.

 

Thanks for reading! 

-The CASE Challenge Team 

 

If you have any questions, please contact the challenge team at CASEChallenge@iarpa.gov.

 

leave a comment

Register Today! Credibility Assessment Standardized Evaluation (CASE) Challenge Workshop on July 18, 2019

June 25, 2019, 8:51 a.m. PDT by M M

Greetings,

 

There is still time to register to attend the Credibility Assessment Standardized Evaluation (CASE) Challenge Workshop on July 18, 2019 in Washington, D.C., hosted by the Intelligence Advanced Research Projects Activity (IARPA)

IARPA’s CASE Challenge crowd-sourced the development of innovative methods to evaluate credibility assessment techniques and technologies. The culmination of this challenge is a workshop aimed at fostering a rich exchange of ideas that will help to inform potential ways ahead for future investments in credibility assessment research and development, as well as practical applications, education, and training. 

During the workshop we will also highlight the winners of the CASE Challenge and take stock of past, present and potential future developments in credibility assessment. Participants will include CASE Challenge solvers, industry representatives, academic researchers, and Government stakeholders.

Please join us in this engaging opportunity to learn from and network with experts in the credibility field!

For more information on the CASE Challenge please check out the following:

https://www.iarpa.gov/challenges/casechallenge.html and https://www.herox.com/CASEchallenge.   

Please share this invitation with interested colleagues!

 

Workshop Details

When:  Thursday, July 18, 2019

8:00 AM - 5:00 PM

Where: WeWork Universal North

1875 Connecticut Ave NW

10th Floor

Washington DC, District of Columbia 20009

 

Register online at the link below, by Wednesday, July 10, 2019 (no fee to register or attend):

Site: https://eventmanagement.cvent.com/CASEChallengeFinalWorkshop

Password: CASE829CFW

 

The registration site also includes the agenda, directions, food options, and other visitor information.

Please contact CASE Challenge Team with any additional questions at CASEChallenge@iarpa.gov.

 

Thank you!

 

CASE Challenge Team

 

1 comment

CASE Challenge Stage 1 Winners

June 11, 2019, 1:30 p.m. PDT by Liz Treadwell

 

We are excited to announce the winners from Stage 1 of the Intelligence Advanced Research Projects Activity’s (IARPA) Credibility Assessment Standardized Evaluation (CASE) Challenge!

The CASE Challenge received high quality solutions from a range of solvers from across the globe. In total, we received a total of 27 solutions from competitors, offering creative approaches to evaluate the validity of current and future credibility assessment techniques and technologies. The submissions ranged from low-tech solutions, such as a clever variation on a popular party game, to high-tech solutions using neural networks for testing protocols.

During the months of April and May our esteemed panel of judges took the time to deliberate on the scores of the submissions. Following a rigorous evaluation process, we have come to a decision as to the winners of the Stage 1 Prizes.

 

Credibility Champions

We are honored to announce three Credibility Champions. These champions are Dr. Joseph R. Stainback IV for his solution of Strategic Cognitive and Mobility Room (SCAMR), Jeremy Martinez, M.D. for his solution of Credibility: Assessing the Assessment (C2A) and the Computer Augmented Social Engagement Lab team (which includes Taylan Sen (Captain), Ehsan Hoque, and Kurtis Haut) for their solution of SPIDER2, congratulations! Each of the solvers provided intriguing and well-crafted solutions.

Being recognized as Credibility Champions means that these solutions had the best overall performance in Stage 1.  As part of their winnings, they move onto Stage 2 of The CASE Challenge where they will pitch their solutions for an opportunity to win additional prizes at the CASE Challenge Workshop on July 18th.

Novelty prizes

Creative Technology: Winner is Lex Fridman for a method with outstanding and innovative use of technology in their solution,Credibility Meta-Assessment with Neural Networks.

Ground Truth:  Winner is Charles McElroy for providing the best method of objectively determining the true state of credibility in their solution, Checkmate Revisited.

Realistic Reflection: Winner is Ilona Palmer for a method with outstanding performance reflecting the parameters of the real-world in their solution, The Purrrfect CAT.

Although Stage 1 of the competition is over, we will continue to provide updates on The CASE Challenge.

Be sure to stay tuned for information about upcoming workshop, exciting news in the field of credibility, details on the solutions presented in Stage 1 and much more! 

 

Thanks for reading! 

 

-The CASE Challenge Team 

 

If you have any questions, please contact the challenge team at CASEChallenge@iarpa.gov.

1 comment

CASE Challenge Workshop in Washington, DC

June 4, 2019, 1:38 p.m. PDT by Liz Treadwell

Greetings,

 

You are invited to join the Intelligence Advanced Research Projects Activity (IARPA) at the Credibility Assessment Standardized Evaluation (CASE) Challenge Workshop on July 18, 2019 in Washington, DC.

IARPA’s CASE Challenge crowd-sourced the development of innovative methods to evaluate credibility assessment techniques and technologies. The culmination of this challenge is a workshop aimed at fostering a rich exchange of ideas that will help to inform potential ways ahead for future investments in credibility assessment research and development, as well as practical applications, education, and training. 

During the workshop we will also highlight the winners of the CASE Challenge and take stock of past, present and potential future developments in credibility assessment. Participants will include CASE Challenge solvers, industry representatives, academic researchers, and Government stakeholders.

Please join us in this engaging opportunity to learn from and network with experts in the credibility field!

 

For more information on the CASE Challenge please check out the following:

https://www.iarpa.gov/challenges/casechallenge.html and https://www.herox.com/CASEchallenge.   

1 comment

Just the Weekend Left to Submit your Solutions for the CASE Challenge!

April 12, 2019, 11:30 a.m. PDT by Liz Treadwell

 

Dear CASE Challengers,

There are only a few days left to finalize your solutions! The submission form will be CLOSED on 8:59 p.m. PDT Sunday, April 14th for the CASE Challenge. The Challenge Team looks forward to reviewing your submissions!

Feel free to reach out to  if you have any questions.

 

Good luck!

-The CASE Challenge Team

leave a comment
Forum 11
Community 366
Press
FAQ
Rules
We've successfully collaborated winning and fulfilling on NASA contracts through their Center of Excellence for Collaborative Innovation. HeroX is positioned as the social network for innovation and therefore has a broader and more uncredentialed crowd, which pairs well with our Topcoder crowd of experts in development and design. Their team is great to work with, we've had the pleasure of hosting webinars and other joint initiatives.
Clinton Bonner | VP Marketing | TopCoder
about herox
community
support
sign in