Hi everyone,

We wanted to let you know that we have updated the scoring system for Task 1. We would like to shift the focus on the tasks that are important  to SAP. Hence, these are the following changes we are going to make:

  • Merge the classes of handwritten and signatures into the class signature.
  • Remove invoices with personal addresses or personal telephone numbers from the test set.
  • You can download the updated test file list here.

We’ve also added the possibility to label an invoice as not anonymizable due to bad quality. If an invoice is labeled as such it will count with a score of 0.35. This corresponds roughly to the score of predicting no boxes for each invoice.

To label an invoice as bad quality just add "bad_quality": true to the label, e.g., {"id": "imagesl_lgq30c00_ti01410925.png", "bad_quality": true}

In addition, we updated the zip file containing the scoring algorithm. You can download it here. We made sure that predictions to the previous scoring will still work with the current algorithm.We also submitted an empty label set flagging every invoice as bad quality to the leaderboard. This gives a baseline prediction of 0.35.

If you have already submitted an entry to the leaderboard, you may see an update to your score based on this new scoring mechanism.

The motivation behind the above changes is that we wanted to simplify the challenge while still maintaining the full anonymization invoices. In more detail, the merging change is just a direct simplification. The removal of invoices with personal addresses and phone numbers is due to those labels only occurring in less than 10% of the cases. Lastly, the addition of a “bad quality” label is motivated by the fact that in practical applications those invoices can be handled manually.

We hope these updates will help as you continue to develop your solutions. Remember, you can submit an entry to the leaderboard at any time to see how it performs. You can also update your leaderboard entry once per day with any improvements you’ve made. If you have any questions regarding the above changes, please comment directly in the challenge forum.