Five times more deadly than the flu, COVID-19 causes significant morbidity and mortality. Like other pneumonias, pulmonary infection with COVID-19 results in inflammation and fluid in the lungs. COVID-19 looks very similar to other viral and bacterial pneumonias on chest radiographs, which makes it difficult to diagnose. Your computer vision model to detect and localize COVID-19 would help doctors provide a quick and confident diagnosis. As a result, patients could get the right treatment before the most severe effects of the virus take hold.
Currently, COVID-19 can be diagnosed via polymerase chain reaction to detect genetic material from the virus or chest radiograph. However, it can take a few hours and sometimes days before the molecular test results are back. By contrast, chest radiographs can be obtained in minutes. While guidelines exist to help radiologists differentiate COVID-19 from other types of infection, their assessments vary. In addition, non-radiologists could be supported with better localization of the disease, such as with a visual bounding box.
As the leading healthcare organization in their field, the Society for Imaging Informatics in Medicine (SIIM)'s mission is to advance medical imaging informatics through education, research, and innovation. SIIM has partnered with the Foundation for the Promotion of Health and Biomedical Research of Valencia Region (FISABIO), Medical Imaging Databank of the Valencia Region (BIMCV) and the Radiological Society of North America (RSNA) for this competition.
In this competition, you’ll identify and localize COVID-19 abnormalities on chest radiographs. In particular, you'll categorize the radiographs as negative for pneumonia or typical, indeterminate, or atypical for COVID-19. You and your model will work with imaging data and annotations from a group of radiologists.
If successful, you'll help radiologists diagnose the millions of COVID-19 patients more confidently and quickly. This will also enable doctors to see the extent of the disease and help them make decisions regarding treatment. Depending upon severity, affected patients may need hospitalization, admission into an intensive care unit, or supportive therapies like mechanical ventilation. As a result of better diagnosis, more patients will quickly receive the best care for their condition, which could mitigate the most severe effects of the virus.
This is a Code Competition. Refer to Code Requirements for details.
FISABIO, The Foundation for the Promotion of Health and Biomedical Research of Valencia Region
The Foundation for the Promotion of Health and Biomedical Research of Valencia Region, FISABIO, is a non-profit scientific and healthcare entity, whose primary purpose is to encourage, to promote and to develop scientific and technical health and biomedical research in Valencia Region. FISABIO integrates and manages the Health Research Map of the Centre for Public Health Research, Dr. Peset University Hospital Foundation, Alicante University General Hospital Foundation, Elche University General Hospital Foundation, and the Mediterranean Ophthalmological Foundation. The BIMCV facility is connected with a multi-level vendor neutral archive (VNA). The imaging population facility is storing data from the Valencia Region, which accounts for more than 5.1 million habitants.
Radiological Society of North America (RSNA)
The Radiological Society of North America (RSNA) is a non-profit organization that represents 31 radiologic subspecialties from 145 countries around the world. RSNA promotes excellence in patient care and health care delivery through education, research and technological innovation.
RSNA provides high-quality educational resources, publishes five top peer-reviewed journals, hosts the world’s largest radiology conference and is dedicated to building the future of the profession through the RSNA Research & Education (R&E) Foundation, which has funded $66 million in grants since its inception. RSNA also supports and facilitates artificial intelligence (AI) research in medical imaging by sponsoring an ongoing series of AI challenge competitions.
The challenge uses the standard PASCAL VOC 2010 mean Average Precision (mAP) at IoU > 0.5. Note that the linked document describes VOC 2012, which differs in some minor ways (e.g. there is no concept of "difficult" classes in VOC 2010). The P/R curve and AP calculations remain the same.
In this competition, we are making predictions at both a study (multi-image) and image level.
Study-level labels
Studies in the test set may contain more than one label. They are as follows:
"negative", "typical", "indeterminate", "atypical"
Please see the Data page for further details.
For each study in the test set, you should predict at least one of the above labels. The format for a given label's prediction would be a class ID from the above list, a confidence score, and 0 0 1 1 is a one-pixel bounding box.
Image-level labels
Images in the test set may contain more than one object. For each object in a given test image, you must predict a class ID of "opacity", a confidence score, and bounding box in format xmin ymin xmax ymax. If you predict that there are NO objects in a given image, you should predict none 1.0 0 0 1 1, where none is the class ID for "No finding", 1.0 is the confidence, and 0 0 1 1 is a one-pixel bounding box.
Submission File
The submission file should contain a header and have the following format:
Id,PredictionString 2b95d54e4be65_study,negative 1 0 0 1 1 2b95d54e4be66_study,typical 1 0 0 1 1 2b95d54e4be67_study,indeterminate 1 0 0 1 1 atypical 1 0 0 1 1 2b95d54e4be68_image,none 1 0 0 1 1 2b95d54e4be69_image,opacity 0.5 100 100 200 200 opacity 0.7 10 10 20 20 etc.
May 17, 2021 - Start Date.
August 2, 2021 - Entry Deadline. You must accept the competition rules before this date in order to compete.
August 2, 2021 - Team Merger Deadline. This is the last day participants may join or merge teams.
August 9, 2021 - Final Submission Deadline.
September 19-20, 2021 - Winners' Showcase at SIIM CMIMI 2021
All deadlines are at 11:59 PM UTC on the corresponding day unless otherwise noted. The competition organizers reserve the right to update the contest timeline if they deem it necessary.
Leaderboard Prizes: Awarded on the basis of private leaderboard rank. Only selected submissions will be ranked on the private leaderboard.
Student Team Prize: Awarded to the top-scoring eligible Student Team on the basis of private leaderboard rank. To be fulfilled by HP's Fulfillment Agency
One HP ZBook Studio G8 data science workstation will be awarded to each member of the winning Student Team. Each ZBook is valued at $6,000.
Important Requirements
As this competition is aimed at the advancement of research using machine learning in medical imaging, the hosts require that winners fulfill the obligations stated in the Competition Rules, Section A.3. in order to retain their leaderboard position. If a team is unwilling to fulfill these obligations and declines their prize, they will also be disqualified from the competition and removed from the Competition leaderboard.
In addition to the standard Kaggle Winners' Obligations (open-source licensing requirements, solution packaging/delivery), the host team also asks that you:
(i) provide a video presentation to be shared with the SIIM community,
(ii) publish a link to your open sourced code on the competition forum, and
(iii) [optional] create a publicly available demo version of the model for more hands-on testing purposes. As an example of a hosted algorithm, please see http://demos.md.ai/#/bone-age.
To be eligible to earn the Student Team Prize, in addition to meeting the eligibility rules specified in Section 2 (Eligibility) of the Competition Rules, 50% or more of the members of a Student Team (i.e., 3 out of 5) must also be:
(i) currently enrolled as a full-time student (undergraduate or graduate) at a college, university, high school, secondary school or equivalent educational institution
(ii) able to provide proof of your current enrollment before you may receive a prize. If you cannot provide documentary proof that the required percentage of student members of a Student Team are currently enrolled as a full-time students, the Student Team may be disqualified or the team member without documentation may be disqualified, at the discretion of the Competition Sponsor.
We will open up a submission form to self-nominate your eligibility for the Student Team Prize at the end of the competition. It is possible to win both the Student Team Prize and a Leaderboard Prize. But to be considered for the Student Prize, you must self-nominate through the submission form.
All teams, regardless of place, are also strongly encouraged and invited to open source their code to the Model Zoo. Contribute to the Model Zoo through this link. Your contributions will greatly advance the diagnosis and treatment of COVID-19.
This is a Code Competition
Submissions to this competition must be made through Notebooks. In order for the "Submit" button to be active after a commit, the following conditions must be met:
Please see the Code Competition FAQ for more information on how to submit. And review the code debugging doc if you are encountering submission errors.
Support Research - Call for Open Source AI Models
Reproducible science requires reproducible review. AI is poised to transform the medical imaging process from diagnosis through intervention, but these technologies are not reaching patients due to complex, non-scalable, and non-verifiable validation.
This challenge is brought to life thanks to the NSF Convergence Accelerator Grant, on which SIIM is a Co-Principal Investigator. One of the main deliverables for Phase 1 of the grant was to create a prototype of a Model Zoo, that would allow for search, discovery, sharing, and 3rd party validation of AI models in medical imaging.
To populate the Model Zoo, SIIM has issued its first-ever Call for AI Models - the winning solutions will be presented at the SIIM21 Annual Meeting on May 24-27.
In addition, SIIM has partnered with FISABIO and RSNA, as well as HP and Intel who are providing $100,000 in prizes to organize this competition, in hopes to receive as many open source models as possible to add to the Model Zoo.
If SIIM + Partners are granted Phase 2 of this NSF Convergence Accelerator, the primary deliverable will be clinical research testing of collaborative model-centric AI platform to meet the urgent needs of scalable validation and translation of model-centric AI in medical imaging. Thus, the more verified models we are able to add to the Model Zoo, the better it is for the advancement of science and betterment of healthcare.
In the spirit of advancing research on this topic, we invite you to contribute your open sourced models for this competition to be included in the Model Zoo. This offer is extended to any participant. Submit your open source model via the Strait I3 Platform.
