Machine Learning Challenge Cancer Image Segmentation

CDI and Vector Institute Collaboration

Challenge details

Congratulations to Team Fight Tumor, and Team McIntosh Lab on placing in 1st and 2nd place respectively in our first ever Machine Learning Challenge. We want to thank all teams and participants for their hard work and dedication during the challenge. We look forward to hearing our first-place winners present their findings at the Machine Learning Summit later this year!

1st Place Winning Team

Team: Fight Tumor

Bo Wang- UHN, Vector Institute 

Jun Ma- UHN

Ronald Xie- UHN, Vector Institute 

Shihao (Rex) Ma- UHN, Vector Institute 

2nd Place Winning Team:

Team: McIntosh Lab

Chris McIntosh- UHN, Vector Institute

Elyar Abbasi Bavil- UHN

Sangwook Kim- UHN

Siham Belgadi- UHN

Tom Purdie- UHN

Testimonials from 2022 Teams

“The nature and the tasks of the challenge were very well designed. It was just so interesting.” 

 

“I found the team to be very responsive and helpful- they were very quick to answer questions. I also really enjoyed the talk from the medical physicist. It was very educational.” 

 

“The challenge did a great job of representing the real challenges of working with this type of data. The team was very quick to answer any questions we had along the way.” 

 

 

Thank you to everyone who registered for CDI and Vector Institute’s Machine Learning Challenge. Miss the deadline to apply? We will be announcing similar exciting opportunities and partnerships in the future!

Table of Contents:

Challenge details

The Machine Learning (ML) challenge will engage scientists and clinicians in an open competition to explore their computational limitations in the space of object detection and contouring as it relates to medical imaging.

  • Connect AI applications with clinical expertise
  • Optimize the future impact of healthcare delivery
  • Work with high quality, high dimensional, Canadian data
  • Take advantage of an optimized computing environment

CDI collects clinical data to optimize disease insights and gain a deeper understanding of data nuances and clinical considerations which can empower the development of novel artificial intelligence (AI) models and applications. Clinicians often do not have sufficient knowledge of AI to improve their clinical workflows. This presents an exciting opportunity for collaboration.

Vector faculty and affiliates will gain direct access to expert clinical teams and their domain-specific expertise. The ML challenge encourages and welcomes all UHN and Vector-affiliated AI researchers, regardless of previous experience, to apply AI in the health domain.

The Proposal: 
A competition to challenge ML experts to develop accurate auto-segmentation models in the space of medical (3D radiological) imaging

The Objective/Question:
This challenge will tackle the issue of region of interest (ROI) segmentation for radiation therapy (RT) in head and neck cancer (HNC) and is divided into two tasks:

  • Organ-at-risk (OAR) segmentation
  • Gross Tumor Volume (GTV) Segmentation

The Dataset (i.e. RADCURE):
A retrospective dataset of 2,552 patients with HNC treated at the PM Cancer Centre. Currently, this is the largest HNC dataset of its kind.

Evaluation:

  • A hold-out dataset of 500 patients from the original dataset

Your model will be assessed based on two key factors:

  1. The accuracy of your contours relative to the clinical gold-standard contours;
  2. The model complexity (estimated by the number of model parameters) and inference time

Details of the metric, including the full equation can be found here: https://docs.google.com/document/d/1-D5ZH51hB03yCryF9uA95Dh49Z8kvS36D29nif8Ye-U/edit?usp=sharing

This document also contains the full list of ROIs that your models will be evaluated on and the baseline evaluation metric.

A complete list of ROIs that will be used to evaluate model performance:

Category ROIs (n=34)
Long & Simple Brain, Brainstem; SpinalCord

Eye (Lt/Rt); Lacrimal Glnd (Rt/Lt)

Esophagus; Larynx; Mandible; Lips

Parotid Glnd (Rt/Lt); Submandibular Glnd (Rt/Lt); Pharyngeal constrictor (S/M/I)*, CTVn(Rt/Lt) (Not an OAR but essential anatomy-based treatment volume), 

Short & Simple Lens (Rt/Lt);

Optic Nerve (Rt/Lt);

Cochlea (Rt/Lt);

Cricoid, Oral Cavity

Long & Complicated Brachial Plexus (Rt/Lt)
Short & Complicated Optic Chiasm, SPC Retropharngeal (Rt/Lt), (GTVp, not an OAR but essential in identifying gross extent of primary tumour)

Important details

Eligibility criteria

  • Participants must have an affiliation with Vector or UHN
  • Students and post-doctoral fellows who are not affiliated with Vector or UHN must have a supervisor who is affiliated with Vector or UHN

Vector Affiliation includes those in a Vector Faculty, Vector Affiliate Faculty, and Vector Postdoc role.

Award

The winning teams will receive the following:

First place

  • $1000
  • First authorship on a manuscript about findings
  • Oral presentation at The Machine Learning Society (TMLS)
  • Listing on the CDI website and Twitter as the first place winner

Second place

  • $500

All decisions related to the judging of entries are at the sole discretion of the organizers and are final and binding.  The award will be given to the team lead of the winning team. The organizers have no responsibility for how the award is subsequently used.

Information for participants

Evaluation metrics

The submitted models will be evaluated using the following criteria:

  • 95 Hausdorf Distance
  • DICE
  • Average Surface Distance (ASD)
  • Applied Path Length (APL)
  • Total inference time
  • Training time until convergence
  • Total number of optimizable parameters

Ranking Scheme:

Step 1. Compute metrics 1-5 for each testing case ( 95HD, DICE, ASD, APL and inference time)

Step 2. Rank participants for each of the testing cases (n) and each metric, such that each participants will have nx5 rankings

Step 3. Average all the rankings and weight by the inverse of model complexity * total time until convergence

Acknowledgments

We would like to thank Dr. Tony Tadic and Tirth Patel for helping to curate, support, develop and make the RADCURE dataset available for participants to use in the challenge. We would also like to thank the Head and Neck site group at Princess Margaret for their collaboration. Without their help, the success and the great learning opportunities of the challenge would not be possible.

About the Vector Institute

The Vector Institute is a not-for-profit organization with a vision to drive excellence and leadership in Canada’s knowledge, creation, and use of artificial intelligence (AI) to foster economic growth and improve lives. By working with partners in the health and academic sectors, Vector is supporting and enabling AI research and innovation across a wide range of topics to transform and improve health. Vector is home to over 500 active researchers with broad expertise in AI including faculty members, affiliate faculty members, post-graduate affiliates, postdoctoral fellows, and Ph.D. and Master’s students who are world-class machine learning researchers. Learn more about Vector Institute and their work here.