SPIE-AAPM-NCI BreastPathQ: Cancer Cellularity Challenge 2019

Organized by organizing.committee - Current server time: Aug. 25, 2019, 12:53 a.m. UTC


Dec. 6, 2018, midnight UTC


On-going Challenge
Jan. 18, 2019, 1:41 p.m. UTC


Competition Ends
Jan. 1, 2020, 5 a.m. UTC

SPIE-AAPM-NCI BreastPathQ: Cancer Cellularity Challenge

Test Performance Results as of 1/02/2019

The challenge has been closed as of 12/28/2018. The results for the test submissions have been emailed to the participants. The histogram of the submitted test results are shown below for reference.

Challenge Overview

The SPIE (the international society for optics and photonics), along with the American Association of Physicists in Medicine (AAPM), and the National Cancer Institute (NCI), will conduct a “Grand Challenge” on the development of quantitative biomarkers for the determination of cancer cellularity from whole slide images (WSI) of breast cancer hematoxylin and eosin (H&E) stained pathological slides. As part of the 2019 SPIE Medical Imaging Conference, the BreastPathQ Challenge will provide a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets.

This year, we have introduced a cancer cellularity scoring challenge for tumor burden assessment in breast pathology. Participants will be tasked to develop an automated method for analyzing histology patches extracted from whole slide images and assign a score reflecting cancer cellularity in each. Currently, this task is performed manually and relies upon expert interpretation of complex tissue structures. Furthermore, reproducibility of cancer cellularity scores is a concern in current practice, therefore a fully automated method holds great promise for increasing throughput and reducing inter- and intra-observer variability.

Important Dates

Release date of validation set cases without truth October 15, 2018
Release date of test set cases without truth and validation truth December 1, 2018
Final date for participants’ to submit test set cellularity scores December 28, 2018
Challenge results released to participants January 4, 2019
BreastPathQ Workshop (2019 SPIE Medical Imaging) February 16, 2019


A joint computer-aided diagnosis (CAD) and digital pathology session at the 2019 SPIE Medical Imaging Conference will focus on the BreastPathQ Challenge; an individual from each of the three top-performing teams on the test set of the challenge will receive a waiver of the meeting registration fee in order to present their methods during this session. After the completion of the BreastPathQ challenge, all participants will be invited to submit their work in a collaborative journal paper and each participating team will be allowed up to two co-authorships.


Neoadjuvant treatment (NAT) of breast cancer (BCa) is an option for patients with the locally advanced disease. Moreover, tumor response to the therapy provides useful information for patient management. In addition to the treatment effect on tumor size, NAT may alter the tumor cellularity. Tumor size many not decrease, but the overall cellularity may be markedly reduced, making residual tumor cellularity an important factor in assessing response. The pathological examination of the tissue sections after surgery is the gold-standard to estimate the residual tumor and the assessment of cellularity is an important component of tumor burden assessment. Cellularity within the tumor bed is defined as the percentage area of the overall tumor bed that is comprised of tumor cells (invasive or in situ). The most acceptable methods of assessing residual cancer burden in ongoing clinical trials and in clinical practice in some centers follows the algorithm proposed by an international working group. This algorithm takes into account several parameters including tumor cellularity. In the current clinical practice, tumor cellularity is manually estimated by pathologists on hematoxylin and eosin (H&E) stained slides, the quality, and reliability of which might be impaired by inter-observer variability which potentially affects prognostic power assessment in NAT trials.


The dataset for this challenge was collected at the Sunnybrook Health Sciences Centre, Toronto. It comprises 96 whole slide images (WSI) which have been stained with haematoxylin and eosin (H&E), a commonly used stain in pathology to highlight cellular and connective tissue. WSIs were extracted from 64 patients with residual invasive breast cancer on resection specimens following neoadjuvant therapy. The specimens were handled according to routine clinical protocols and WSIs were scanned at 20X magnification (0.5 lm/pixel).

In this challenge, you will be provided with a training/validation set of 2,579 patches extracted from the above WSIs. Each patch in the training set has been assigned a tumor cellularity score by 1 expert pathologist, while the test set has reference standards from 2 pathologists. Coordinates for each patch and original WSIs may be provided upon request for additional context. You may choose to use the provided WSIs but it is not a requirement.

The Challenge

Participants are required to automatically score each patch on a continuous scale from 0 to 1. Scores must be provided as a floating-point number and must be between 0 and 1 inclusive.

Participants may use the training set cases in any manner they would like for the purpose of training their systems; there will be no restrictions on the use of the data or the advice sought from local experts for training purposes. The test set cases, however, are to be manipulated, processed, and analyzed without human intervention.

Training/Validation Sets The training/validation set has been prepared and contains 2,579 patches extracted from 69 WSIs. Labels will be provided for each training case to be used in algorithm development. Validation labels for 185 patches from 6 WSIs will originally be withheld for initial validation of developed algorithms but subsequently, the validation labels will be released when the test data becomes available.

Test Set The test set has been prepared in an identical manner as the training set and contains 1,121 patches extracted from 25 WSIs. No labels will be provided for the test set until after the completion of the challenge. You will be asked to run your developed solution on the test set and submit a CSV file per task with your final submission.

Additional Dataset

In addition to image patches extracted from whole slide images, we will also provide annotations of lymphocytes, malignant epithelial and normal epithelial cell nuclei in 153 regions-of-interest (ROI) from the same dataset. These annotations may be used in addition to the dataset described above, to distinguish between malignant and healthy structures. Cell nuclei have been marked and x-y co-ordinates stored in an .xml file for each ROI. Usage of this particular data has been described in:

Peikari, M., Salama, S., Nofech-Mozes, S. and Martel, A.L., 2017. Automatic cellularity assessment from post-treated breast surgical specimens. Cytometry Part A, 91(11), pp.1078-1087.

Participants are free to download the training set and, subsequently, the test set when these datasets become available. It is important to note that once participants submit their test set cellularity output to the challenge organizers, they will be considered fully vested in the Challenge, so that their performance results (without links to the identity of the participant) will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers. The truth associated with the test set cases is expected to be made publicly available after publication of the BreastPathQ Challenge.

Participation in the BreastPathQ Challenge acknowledges the educational, friendly competition, and community-building nature of this challenge and commits to conduct consistent with this spirit for the advancement of the medical imaging research community. See this link for a discussion of lessons learned from the 2015 LUNGx Challenge, also sponsored by SPIE, AAPM, and NCI.

Evaluation Metric

Method performance will be assessed using prediction probability (P_K) (Smith W.D, Dutton R.C, Smith N.T. (1996) A measure of association for assessing prediction accuracy that is a generalization of non-parametric ROC area. Stat Med. Jun 15;15(11):1199-215). The prediction probability will be calculated for each method for each reference standard (pathologist 1 and pathologist 2), then averaged to determine a final overall prediction probability value. If the prediction probability values across multiple methods are the same (a tie), the prediction probability value for the subset of cases with mean reference values below 50% will be used as a tie breaking subset of cases. The same approach described above will be used to estimate a prediction probability value for this subset of cases. If the methods are still tied in performance, a coin flip will be used to determine a winner.

The definition of prediction probability that is used is the following:

p_k = (((P - Q) / (P + Q + T)) + 1)/2

where P is the number of concordant pairs, Q the number of discordant pairs, and T the number of ties only in the submitted labels.

A python implementation of prediction probability can be downloaded here

Preparing Your Submission

Method Description

Participants must submit a brief two-page description of their solution for the challenge. The description manuscript should mention and describe whether any of the following was used:

  1. Training with cases other than the ones provided
  2. Any human/semi-supervised components
  3. Image pre-processing and post-processing
  4. Data augmentation

You may provide figures and tables but they must fit within your two-page summary. Any references are allowed beyond the two-page limit.

If you are submitting a trained neural network, we ask you to also submit a copy of the network architecture by filling out the network_template.docx file which can be found in the download repository.

We recommend you use the SPIE proceeding template to prepare your manuscript. An abstract is not required for your submission. You may provide figures and tables but they must fit within your two-page manuscript. Any references are allowed beyond the two-page limit. Templates are provided within the download repository but can also be found on the SPIE website: https://spie.org/x5258.xml

Please also include a sentence indicating that the participants are abiding by the Conflict of Interest statement.


To facilitate performance assessment across participants for the test cases in the BreastPathQ Challenge, please follow these guidelines: we require you to provide a CSV file containing your results for the test set. Please use the following naming convention: “[TeamName]_Results.csv”.

The CSV results file must contain three columns with the following headings: “slide”, “rid” and “p”. The “slide” column must be populated with the name of the WSI from which each patch originated from and “rid” is a unique number assigned to each patch. "p" is the algorithm output for the patch (a floating-point number between 0 and 1.0, where 0 corresponds to 0% cellularity, and 1.0 corresponds to 100% cellularity). If you are using the downloaded dataset, each TIFF file is named “slide_rid.tif”.

Using your method, calculate a cellularity score for each patch as a floating-point value on the range [0, 1.0]. The submitted output must be a floating-poing value between 0 and 1.0, inclusive.

Record your scores for all cases in a CSV file containing three columns with

  1. The first column being the whole slide image number (slide).
  2. The second column being the region number (rid).
  3. The third column being your method's assigned cellularity score (p).

Please report your results in the same order as lesions appear in the BreastPathQ-Test.csv file. Also note that the system expects a file with 1,122 rows with the first row being the header row followed by your output for the 1,121 test cases (in the format described above)

All files to be submitted need to be placed in a zip file, which contains only 1 score file in the format “[TeamName]_Results.csv”. The zip file is submitted to the challenge through the "Participant" tab.

Number of Submissions

Each participant is limited to successful submissions of testing results from at most 3 algorithms to the Challenge.


This challenge is organized in the spirit of cooperative scientific progress. We therefore ask all participants to respect the rules below.

The following rules apply to those who register a team and download the data:


  • The downloaded repository or any data derived from these datasets with the exception of datasets that are already publicly available, may not be given or redistributed under any circumstances to persons not belonging to the registered team;

  • Downloads from this website and TCIA must primarily be used for preparing an entry to be submitted to this challenge. The datasets may not be used for other purposes in scientific studies and may not be used to train or develop other algorithms, including but not limited to algorithms used in commercial products, without prior participation in this challenge. After the SPIE BreastPathQ workshop in February, the teams are free to use the datasets for other purposes provided that the source of the data is correctly acknowledged (see below) ;

  • Use of other datasets during the development of a method that will be submitted to the challenge is allowed. The teams must name all data sources in the method description when submitting their final solution;

  • The test dataset must not be used in any way during the method development (for example, semi-supervised learning with the testing dataset is not allowed);


  • All information entered when registering a team, including the name of the contact person, the affiliation (institute, organization or company the team's contact person works for) and the e-mail address must be complete and correct. In other words, anonymous registration is not allowed. The data provided will not be used for any purposes other than the challenge;

  • Evaluation of results uploaded to this website will be made publicly available on this site, and by submitting results, you grant us permission to publish our evaluation.

  • Participating teams maintain full ownership and rights to their method. We do not claim any ownership or rights to the algorithms;


  • We ask participants to not publish any material (including workshop, journal, conference, ArXiv publications) or preliminary work during the length of the BreastPathQ challenge. Two months after the completion of the BreastPathQ challenge, publications are permitted with appropriate citation (see acknowledgements).

Feel free to contact us if you have any questions or need clarification of the rules of the challenge.

Conflict of Interest:

All participants must attest to having no advanced knowledge regarding the Challenge development, data, and design and are prohibited from collaborating on any aspect of a challenge submission with a member of the Challenge Organizing Committee.

Individual Organizing Committee members, students and staff from Sunnybrook Research Institute, University of Toronto, and students and staff from FDA/CDRH/OSEL/DIDSR are ineligible to participate in the test phase of the Challenge and are ineligible to receive a prize (free registration and presentation of their methods at SPIE Medical Imaging 2019) due to having direct access to the challenge data. Other Organizing Committee members did not have prior access to any of the challenge data and will not be provided access to the data beyond what is available to all participants.


Anyone wishing to use these datasets for research, presentation, or publication purposes outside of the BreastPathQ Challenge should acknowledge the Canadian Cancer Society, SPIE, the AAPM, the NCI, the Sunnybrook Research Institute (“Data used in this research was acquired from Sunnybrook Health Sciences Centre with funding from the Canadian Cancer Society and was made available for the BreastPathQ challenge, sponsored by the SPIE, NCI/NIH, AAPM, Sunnybrook Research Institute.”) and cite the following article:

Peikari, M., Salama, S., Nofech-Mozes, S. and Martel, A.L., 2017. Automatic cellularity assessment from post-treated breast surgical specimens. Cytometry Part A, 91(11), pp.1078-1087.

Organizers and Major Contributors:

  • Anne Martel Sunnybrook Research Institute, University of Toronto (anne.martel@sunnybrook.ca)
  • Shazia Akbar, University of Toronto (shazia.akbar@sunnybrook.ca)
  • Sharon Nofech-Mozes, Sunnybrook Health Sciences Centre (sharon.nofech-mozes@sunnybrook.ca)
  • Nick Petrick, U.S. FDA (nicholas.petrick@fda.hhs.gov)
  • Marios Gavrielides, U.S. FDA (marios.gavrielides@fda.hhs.gov)
  • Berkman Sahiner, U.S. FDA (berkman.sahiner@fda.hhs.gov)
  • Kenny Cha, U.S. FDA (kenny.cha@fda.hhs.gov)
  • Sam Armato, University of Chicago (s-armato@uchicago.edu)
  • Karen Drukker, University of Chicago (kdrukker@uchicago.edu)
  • Lubomir Hadjiiski, University of Michigan (lhadjisk@umich.edu)
  • Keyvan Farahani, NIH/NCI (farahank@mail.nih.gov)
  • Jayashree Kalpathy-Cramer, Harvard University (kalpathy@nmr.mgh.harvard.edu)
  • Diane Cline, SPIE (diane@spie.org)
  • Joel Saltz, Stony Brook University (joel.saltz@stonybrookmedicine.edu)
  • John Tomaszewski, Kaleida Health (jtomaszewski@KaleidaHealth.org)
  • Aaron Ward, Western University (aaron.ward@uwo.ca)
  • Horst Hahn, Fraunhofer MEVIS (horst.hahn@mevis.fraunhofer.de)
  • Kensaku Mori, Nagoya University (mori@nuie.nagoya-u.ac.jp )
  • SPIE-MI CAD Technical committee team
  • SPIE-MI Pathology Technical committee team


Start: Oct. 15, 2018, midnight

Description: Results from the training phase based on testing on the training set. All files to be submitted need to be placed in a zip file, which contains only 1 score file in the format “[TeamName]_Results.csv”. This is only for the training subset, not including the validation set.


Start: Nov. 28, 2018, midnight

Description: Results from the validation phase based on testing on the validation set. All files to be submitted need to be placed in a zip file, which contains only 1 score file in the format “[TeamName]_Results.csv”.


Start: Dec. 6, 2018, midnight

Description: Results from the test phase based on testing on the test set. All files to be submitted need to be placed in a zip file, which contains only 1 score file in the format “[TeamName]_Results.csv”. The submission must also include algorithm descriptions as described under the submission rules for the submission to be a valid submission.

On-going Challenge

Start: Jan. 18, 2019, 1:41 p.m.

Description: Results from the on-going challenge phase based on testing on the test set. All files to be submitted need to be placed in a zip file, which contains only 1 score file in the format “[TeamName]_Results.csv”.

Competition Ends

Jan. 1, 2020, 5 a.m.

You must be logged in to participate in competitions.

Sign In