DeID2 - A Better Meter Stick for Differential Privacy

Contest Background

Public safety use-cases of temporal map data include emergency planning, epidemiologic analysis, and policy setting. High-quality data is required to perform sound analyses in these areas. Both time and space segments may be sparsely populated yet critically important. Further, these sparsely populated segments have an inherently greater risk to linkage attack, where auxiliary and possibly completely unrelated datasets, in combination with records in the dataset that contain sensitive information, can be used to determine uniquely identifiable individuals. Although differential privacy has a formal guarantee against linkage attacks, there is no guarantee of accuracy in output data.

In this contest, NIST PSCR seeks novel metrics by which to assess the quality of differentially private algorithms on temporal map data. Submissions should provide robust metrics that can be applied to a wide variety of data sets involving temporal and spatial data. Solvers are encouraged to provide examples of how their proposed metrics will improve use-case outcomes. 

Better Meter Stick for Differential Privacy Contest Guidelines

NIST PSCR is interested in creative, effective and insightful approaches to evaluating the outputs from differential privacy algorithms, especially those involving temporal map data. The area of data privatization is growing rapidly, as is our understanding of the quality of privatized data.

NIST PSCR invites solvers to develop metrics that best assess the accuracy of the data output by the algorithms that de-identify temporal map data. In particular, methods are sought that:

• Measure the quality of data with respect to temporal or geographic accuracy/utility, or both.
• Evaluate data quality in contexts beyond this challenge.
• Are clearly explained, and straightforward to correctly implement and use.

As you propose your evaluation metrics, be prepared to explain their relevance and how they would be used. These metrics may be your original content, based on existing work, or any combination thereof. If your proposed metrics are based on existing work or techniques, please provide citations. Participants will be required to submit both a broad overview of proposed approaches and specific details about the metric definition and usage. Additionally, we are interested in how easily an approach can accommodate large data sets (scalability) and how well it can translate to different use cases (generalizability).

In order to help the community propose better metrics, NIST PSCR will review the executive summary section for any metric submitted by November 30, 2020 and provide high level feedback.  NIST PSCR will only review and comment on the executive summary section.  This feedback is intended to help participants develop more complete and thoughtful metrics.  All feedback will be provided by December 3, 2020.  Participation in this step is optional, but it is a good opportunity that will allow you to check if you are on the right track.

Each submission should contain only one metric.  Each individual/team lead is allowed one submission.  Organizations may have multiple teams, but each team must have a different team lead.  All submissions are due by January 5, 2021, 10pm EST and should include a title, brief description of the proposed metric, introduction to the team, and a completed submission template (found here).

Competitors’ Resources

Please carefully review the contents of the Competitors’ Resources tab.  This tab includes the data sets described below, a sample submission to this contest, a submission template, and a document sharing “tips and tricks” for developing a generalized metric that effectively measures the practical utility of the privatized data.

Each submission should include a demonstration of the metric on at least one data set. The data may be real or synthesized.  NIST PSCR is providing one example of temporal map data that may be used, including 4 data sets: 1) a ground truth data set; 2) a privatized data set of poor quality; 3) a privatized data set of moderate quality; and 4) a supplementary data set with demographic characteristics of map segments. These data sets can be downloaded from the Competitors’ Resources tab. Additional information about the data can be found here.

Participants may use these provided data sets, or they may use or create their own. Any data sets used must be freely and publicly available (which included the provided data above), or created by the submitting participants. Preference is also given to data sets with usefulness for public safety.

Prize

This A Better Meter Stick for Differential Privacy Contest will award up to $29,000 to the top-ranked submissions and to submitted metrics receiving the most votes from public voting as follows:

Timeline 

Judging

Submissions to the Metric Paper Contest will undergo initial filtering to ensure they meet minimum criteria before they are reviewed and evaluated by members of the expert judging panel.  These minimum criteria include:

• Submitter or submitting team meets eligibility requirements,
• All required sections of the submission form are completed,
• Proposed metric is coherently presented and plausible,
• Each submission should contain only one metric.  Each individual/team lead is allowed one submission.

Submissions that have passed the initial filtering step will be reviewed for Technical Merit by members of the expert judging panel, evaluated against the evaluation criteria listed below, and scored based on the relative weightings shown.  

• Clarity (30/100 points)
  • Metric explanation is clear and well written, defines jargon and does not assume any specific area of technical expertise. Pseudocode is clearly defined and easily understood.
  • Participants clearly address whether the proposed metric provides snapshot evaluation (quickly computable summary score) and/or deep dive evaluation (generates reports locating significant points of disparity between the real and synthetic data distributions), and explain how to apply it.
  • Participants thoroughly answer the questions, and provide clear guidance on metric limitations.
• Utility (40/100 points)
  • The metric effectively distinguishes between real and synthetic data.
  • The metric represents a breadth of use cases for the data.
  • Motivating examples are clearly explained and fit the abstract problem definition.
  • Metric is innovative, unique, and likely to lead to greater, future improvements compared with other proposed metrics.
• Robustness (30/100 points)
  • Metric is feasible to use for large volume use cases.
  • The metric has flexible parameters that control the focus, breadth, and rigor of evaluation.
  • The proposed metric is relevant in many different data applications that fit the abstract problem definition.

Submission Form

Each metric must be submitted as a separate submission. Metrics may be oriented towards map data, temporal sequence data, or combined temporal map data. Successful submissions to the Metric Paper Contest will include:

• Submission Title
• A brief description of the proposed metric,  (Note that this will be included with your title when identifying your metric during the public voting stage)
• An introduction to the submitter or submitting team that includes brief background and expertise, and optional explanation of the author’s interest in the problem,
• A PDF document, using the provided template, with a minimum length of 2 pages that thoughtfully and clearly addresses what the metric is, why it works, and how it addresses the needs of data users, drawing on results from application on data. The document must include the following three sections and address the points outlined below.
  
  
  • Executive Summary  (1-2 pages)
    Please provide a 1-2 page, easily readable review of the main ideas. This is likely to be especially useful for people reading multiple submissions during the public voting phase. The executive summary should be readily understood by a technical layperson and include:
    • The high-level explanation of the proposed metric, reasoning and rationale for why it works
    • An example use case
       
  • Metric Definition
    • Any technical background information needed to understand the metric. (Note that these metric write-ups should be accessible to technical experts from a diverse variety of disciplines.  Please provide clear definitions of any terms/tools that are specific to your field, and provide a clear explanation for any properties that will be relevant to your metric definition or defense.)
    • A written definition of the metric, including English explanation and pseudocode that has been clearly written and annotated with comments. Code can also be included (optionally) with the submission.
    • Explanation of parameters and configurations.  Note that this includes feature-specific configurations.  For instance, a metric could reference “demographic features” or “financial features” for specific treatment, and given a new data set with a new schema, the appropriate features could be specified in a configuration file without loss of generalizability.
    • Walk-through examples of metric use in snapshot mode (quickly computable summary score) and/or deep dive mode (generates reports locating significant points of disparity between the real and synthetic data distributions) as applicable to the metric.
       
  • Metric Defense
    • Describe the metric’s tuning properties that control the focus, breadth, and rigor of evaluation
    • Describe the discriminative power of the proposed metric: how well it identifies points of disparity between the ground truth and privatized data
    • Describe the coverage properties of the proposed metric: how well it abstracts/covers a breadth of uses for the data
    • Address the feasibility of implementing the proposed metric. For instance,  what is the computation time and resource requirements for the metric when running on data? How does the metric scale with an increase in variables, map segments, time segments, and records? This information may include empirical results (e.g. runtime) or theoretical results (e.g. mathematical properties). Feel free to provide assumptions about hardware (e.g. CPU model, memory, operating system) and feature constraints.
    • Provide an example of 2-3 very different data applications where the metric can be used.

Eligibility

All participants 18 years or older are invited to register to participate except for individuals from entities or countries sanctioned by the United States Government. 

A Participant (whether an individual, team, or legal entity) must have registered to participate in order to be an eligible Participant. 

Cash prizes are restricted to eligible Participants who have complied with all of the requirements under section 3719 of title 15, United States Code as contained herein. At the time of entry, the Official Representative (individual or team lead, in the case of a group project) must be age 18 or older and a U.S. citizen or permanent resident of the United States or its territories. In the case of NIST PSCR: Differential Privacy Temporal Map Challenge, Official Rules Page 18 of 24 a private entity, the business shall be incorporated in and maintain a place of business in the United States or its territories. 

Employees, contractors, directors and officers (including their spouses, parents, and/or children) of HeroX and DrivenData, Inc. and each of their respective parent companies, subsidiaries and affiliated companies, distributors, web design, advertising, fulfillment, judging and agencies involved in the administration, development, fulfillment and execution of this Challenge will not be eligible to compete in this Challenge. 

Participants may not be a Federal entity or Federal employee acting within the scope of their employment. Current and former NIST PSCR Federal employees or Associates are not eligible to compete in a prize challenge within one year from their exit date. Individuals currently receiving PSCR funding through a grant or cooperative agreement are eligible to compete but may not utilize the previous NIST funding for competing in this challenge. Previous and current PSCR prize challenge participants are eligible to compete. Non-NIST Federal employees acting in their personal capacities should consult with their respective agency ethics officials to determine whether their participation in this competition is permissible. A Participant shall not be deemed ineligible because the Participant consulted with Federal employees or used Federal facilities in preparing its entry to the Challenge if the Federal employees and facilities are made available to all Participants on an equitable basis. 

Participants, including individuals and private entities, must not have been convicted of a felony criminal violation under any Federal law within the preceding 24 months and must not have any unpaid Federal tax liability that has been assessed, for which all judicial and administrative remedies have been exhausted or have lapsed, and that is not being paid in a timely manner pursuant to an agreement with the authority responsible for collecting the tax liability. Participants must not be suspended, debarred, or otherwise excluded from doing business with the Federal Government. 

Multiple individuals and/or legal entities may collaborate as a group to submit a single entry and a single individual from the group must be designated as an Official Representative for each entry. That designated individual will be responsible for meeting all entry and evaluation requirements.

Rules

Please see the Official Rules on challenge.gov for rules and complete terms and conditions.