Note: Other Government Agencies, Federally Funded Research and Development Centers (FFRDCs), University Affiliated Research Centers (UARCs), or any other similar organizations that have a special relationship with the Government, that gives them access to privileged or proprietary information, or access to Government equipment or real property, will not be eligible to participate in the prize challenge.
Why is this Challenging?
The methodology you present in your solution, that is the way the solution’s method is implemented, will impact the type of information or personal attributes being evaluated. Depending on the approach, the motivations to be credible, as well as subsequent penalties, will vary. In some cases the ground truth about an individual’s credibility or the credibility of their information will be difficult, if not impossible, to establish.
This highlights one of the roadblocks to testing a new credibility assessment method for use in practical applications - there is no universally accepted method by which to establish the performance of a new technique or compare across methods.
A key difficulty in validating that a new method can be used for practical, everyday purposes is in the attempt to move research findings into the real world. Several limitations exist for current methods including, but not limited to:
- Simplistic Procedures: In order to achieve clear results, research studies are often simplistic and highly controlled, which does not reflect the complexity of the conditions under which credibility is often assessed in real-world scenarios.
- Artificial Motivations: Decisions to be credible or not credible are often assigned, or otherwise controlled, and limit a participant’s ability to determine for themselves if they want to be credible, when, and why.
- Low Psychological Realism: There is very little personal investment of the participant in the outcomes of the study, leading to low motivation to behave in natural and authentic ways. And related to this, studies typically lack significant penalties if an individual fails to be judged as credible. The lack of meaningful incentives further exacerbates low intrinsic motivation in participants.
These limitations are particularly impactful when considering applications for national security, where an individual may feel that their livelihood, core values and beliefs, safety, or freedom are in jeopardy if someone doesn’t believe that they are credible. It is difficult to build such motivation and jeopardy into a replicable methodology that is safe, ethical, AND could be used as a common standard across some or all credibility assessment applications.
Another limitation of methods used today is the large amount of work devoted to retrospective evaluations, or situations in which a participant is evaluated based on past experiences. This is in contrast to prospective evaluations that require credibility to be based on future intent or behaviors, which is very challenging to objectively measure. While prospective screening represents the majority of uses of credibility assessment methods by the US Government, for example as screening during employee hiring, it is relatively underrepresented in the research done on credibility assessment.
Prizes
Stage 1 Prizes
- Five (5) Credibility Champions – Following the first round of evaluations, five prizes of $5,000 each will be awarded to the top five overall solutions. Solution submitters/teams will be invited to attend a live judging event, where they will pitch their solution to a panel of experts.
- One (1) Prospective Perspective Prize – Following the first round of evaluations a special $10,000 prize will be awarded to the solution which best addresses current limitations in finding a true prospective methodology. That is, a method which requires credibility to be based on future intent or behaviors.
- Five (5) Novelty Prizes – Following the first round of evaluations five prizes of $1,000 each will be awarded to solutions which do the best job at addressing novelty-specific evaluation criteria.
- One (1) Innovative Methodology Award - Award given for the most innovative methodology
- One (1) Outstanding Participant Motivation Award - Award given for the most inspirational motivation to participant
- One (1) Realistic Reflection Award - Award given for the method that best reflects the parameters of the real-world
- One (1) Creative Technology Award - Award given for the method with the most innovative use of technology
- One (1) Ground Truth Award - Award given for the method that best establishes ground truths.
Stage 2 Prizes
- Five (5) Final Prizes – Following the second and final round of evaluations one of the below prizes will be awarded to each of the finalists /participating teams.
- 1st Prize – $40,000
- 2nd Prize – $25,000
- 3rd Prize – $10,000
- 4th Prize – $5,000
- 5th Prize – $5,000
How Do I Win?
All solvers will submit their solutions in the prescribed format, via the HeroX CASE Challenge site. You may find the CASE Challenge Solution Submission Template here. A Word version of the Solution Submission Template can be found here.
Judging Criteria
CASE Challenge solution scoring will consist of three phases: Compliance Review, Stage 1 Scoring, and Stage 2 Scoring. When a solution submission is received, the ‘Solver Names’ field and associated content will be removed and each submission will be given a unique ID. Additionally, throughout the scoring process each solution will be evaluated separately (i.e., multiple solutions submitted by the same respondent will be separated and evaluated individually rather than as a group).
- Compliance Review Panel
- The Compliance Review Panel will conduct an initial screening of all submissions. During this phase, the submission will be compared against the specified CASE Challenge minimum requirements and disqualifiers listed above. Submissions that do not comply with the minimum requirements and/or that meet any of the disqualification criteria will not be further evaluated.
- Stage 1 Scoring Panel
- The Stage 1 Scoring Panel will review all submissions that pass the Compliance Review screening.
- The panel will at a minimum consider the following scoring criteria, discussed in more detail below: Scientific Support, Realism, Novelty, and Participant Considerations. Up to five winners will be selected to move on to the 2nd Round.
- Stage 2 Scoring Panel
- Winners from the 1st stage will be provided travel awards to attend and present their solution to a panel of judges at a public event for people interested in credibility assessment. Here the judging panel will evaluate the presentation and solution feasibility, alongside the scoring criteria from the Stage 1.
Please see CASE Challenge Rules for additional details
The CASE Challenge is the first concerted effort to invite interested individuals to develop credibility assessment evaluation methods that can be used to objectively evaluate both existing and future credibility assessment techniques/technologies. In doing this the CASE Challenge strives to incentivize a broad range of new ideas, while still ensuring their utility to real-world applications. To meet this goal, a scoring panel of experts will evaluate each solution based on the background and strength of the methods, how well it reflects realistic conditions, how creative and clever it is compared to currently used methods like the mock crime scenario, and how well it ensures the responsible care and consideration of participants.
Please see CASE Challenge Microsite for more information.