Machine Learning Challenge Cancer Image Segmentation
CDI and Vector Institute Collaboration
Challenge details
Congratulations to Team Fight Tumor, and Team McIntosh Lab on placing in 1st and 2nd place respectively in our first ever Machine Learning Challenge. We want to thank all teams and participants for their hard work and dedication during the challenge. We look forward to hearing our first-place winners present their findings at the Machine Learning Summit later this year!
1st Place Winning Team
Team: Fight Tumor
Bo Wang- UHN, Vector Institute
Jun Ma- UHN
Ronald Xie- UHN, Vector Institute
Shihao (Rex) Ma- UHN, Vector Institute
2nd Place Winning Team:
Team: McIntosh Lab
Chris McIntosh- UHN, Vector Institute
Elyar Abbasi Bavil- UHN
Sangwook Kim- UHN
Siham Belgadi- UHN
Tom Purdie- UHN
Testimonials from 2022 Teams
“The nature and the tasks of the challenge were very well designed. It was just so interesting.”
|
“I found the team to be very responsive and helpful- they were very quick to answer questions. I also really enjoyed the talk from the medical physicist. It was very educational.”
|
“The challenge did a great job of representing the real challenges of working with this type of data. The team was very quick to answer any questions we had along the way.”
|
Thank you to everyone who registered for CDI and Vector Institute’s Machine Learning Challenge. Miss the deadline to apply? We will be announcing similar exciting opportunities and partnerships in the future!
Table of Contents:
Challenge details
The Machine Learning (ML) challenge will engage scientists and clinicians in an open competition to explore their computational limitations in the space of object detection and contouring as it relates to medical imaging.
- Connect AI applications with clinical expertise
- Optimize the future impact of healthcare delivery
- Work with high quality, high dimensional, Canadian data
- Take advantage of an optimized computing environment
CDI collects clinical data to optimize disease insights and gain a deeper understanding of data nuances and clinical considerations which can empower the development of novel artificial intelligence (AI) models and applications. Clinicians often do not have sufficient knowledge of AI to improve their clinical workflows. This presents an exciting opportunity for collaboration.
Vector faculty and affiliates will gain direct access to expert clinical teams and their domain-specific expertise. The ML challenge encourages and welcomes all UHN and Vector-affiliated AI researchers, regardless of previous experience, to apply AI in the health domain.
The Proposal:
A competition to challenge ML experts to develop accurate auto-segmentation models in the space of medical (3D radiological) imaging
The Objective/Question:
This challenge will tackle the issue of region of interest (ROI) segmentation for radiation therapy (RT) in head and neck cancer (HNC) and is divided into two tasks:
- Organ-at-risk (OAR) segmentation
- Gross Tumor Volume (GTV) Segmentation
The Dataset (i.e. RADCURE):
A retrospective dataset of 2,552 patients with HNC treated at the PM Cancer Centre. Currently, this is the largest HNC dataset of its kind.
Evaluation:
- A hold-out dataset of 500 patients from the original dataset
Your model will be assessed based on two key factors:
- The accuracy of your contours relative to the clinical gold-standard contours;
- The model complexity (estimated by the number of model parameters) and inference time
Details of the metric, including the full equation can be found here: https://docs.google.com/document/d/1-D5ZH51hB03yCryF9uA95Dh49Z8kvS36D29nif8Ye-U/edit?usp=sharing
This document also contains the full list of ROIs that your models will be evaluated on and the baseline evaluation metric.
A complete list of ROIs that will be used to evaluate model performance:
Category | ROIs (n=34) |
Long & Simple | Brain, Brainstem; SpinalCord
Eye (Lt/Rt); Lacrimal Glnd (Rt/Lt) Esophagus; Larynx; Mandible; Lips Parotid Glnd (Rt/Lt); Submandibular Glnd (Rt/Lt); Pharyngeal constrictor (S/M/I)*, CTVn(Rt/Lt) (Not an OAR but essential anatomy-based treatment volume), |
Short & Simple | Lens (Rt/Lt);
Optic Nerve (Rt/Lt); Cochlea (Rt/Lt); Cricoid, Oral Cavity |
Long & Complicated | Brachial Plexus (Rt/Lt) |
Short & Complicated | Optic Chiasm, SPC Retropharngeal (Rt/Lt), (GTVp, not an OAR but essential in identifying gross extent of primary tumour) |
Important details
Eligibility criteria
- Participants must have an affiliation with Vector or UHN
- Students and post-doctoral fellows who are not affiliated with Vector or UHN must have a supervisor who is affiliated with Vector or UHN
Vector Affiliation includes those in a Vector Faculty, Vector Affiliate Faculty, and Vector Postdoc role.
Award
The winning teams will receive the following:
First place
- $1000
- First authorship on a manuscript about findings
- Oral presentation at The Machine Learning Society (TMLS)
- Listing on the CDI website and Twitter as the first place winner
Second place
- $500
All decisions related to the judging of entries are at the sole discretion of the organizers and are final and binding. The award will be given to the team lead of the winning team. The organizers have no responsibility for how the award is subsequently used.
Information for participants
Evaluation metrics
The submitted models will be evaluated using the following criteria:
- 95 Hausdorf Distance
- DICE
- Average Surface Distance (ASD)
- Applied Path Length (APL)
- Total inference time
- Training time until convergence
- Total number of optimizable parameters
Ranking Scheme:
Step 1. Compute metrics 1-5 for each testing case ( 95HD, DICE, ASD, APL and inference time)
Step 2. Rank participants for each of the testing cases (n) and each metric, such that each participants will have nx5 rankings
Step 3. Average all the rankings and weight by the inverse of model complexity * total time until convergence
Acknowledgments
We would like to thank Dr. Tony Tadic and Tirth Patel for helping to curate, support, develop and make the RADCURE dataset available for participants to use in the challenge. We would also like to thank the Head and Neck site group at Princess Margaret for their collaboration. Without their help, the success and the great learning opportunities of the challenge would not be possible.
About the Vector Institute
The Vector Institute is a not-for-profit organization with a vision to drive excellence and leadership in Canada’s knowledge, creation, and use of artificial intelligence (AI) to foster economic growth and improve lives. By working with partners in the health and academic sectors, Vector is supporting and enabling AI research and innovation across a wide range of topics to transform and improve health. Vector is home to over 500 active researchers with broad expertise in AI including faculty members, affiliate faculty members, post-graduate affiliates, postdoctoral fellows, and Ph.D. and Master’s students who are world-class machine learning researchers. Learn more about Vector Institute and their work here.