SPIE-AAPM-NCI Prostate MR Gleason Grade Group Challenge

Organized by organizing.committee - Current server time: Sept. 23, 2017, 7:58 p.m. UTC

First phase

May 15, 2017, 6 p.m. UTC


Competition Ends
June 26, 2017, 7 p.m. UTC

SPIE-AAPM-NCI Prostate MR Gleason Grade Group Challenge

The American Association of Physicists in Medicine (AAPM), along with the SPIE (the international society for optics and photonics) and the National Cancer Institute (NCI), will conduct a part 2 “Grand Challenge” on the development of quantitative multi-parametric magnetic resonance imaging (MRI) biomarkers for the determination of Gleason Grade Group in prostate cancer. As part of the 2017 AAPM Annual Meeting, the PROSTATEx-2 Challenge will provide a unique opportunity for participants to compare their algorithms with those of others from academia, industry, and government in a structured, direct way using the same data sets.

A session at the 2017 AAPM Annual Meeting will focus on the PROSTATEx-2 Challenge; an individual from each of the two top-performing teams will receive a waiver of the meeting registration fee in order to present their methods during this session. Challenge participants are encouraged to submit their work for peer review to the AAPM’s scientific journal Medical Physics.

Release date of training set cases with truth: May 15, 2017
Release date of test set cases without truth: June 5, 2017
Submission date for participants’ test set Gleason Grade Group output: June 23, 2017
Challenge results released to participants: June 30, 2017
AAPM Annual Meeting: July 30 – August 3, 2017 (PROSTATEx-2 session: August 1)

PROSTATEx-2 Challenge Format

The database for this challenge contains a total of 162 MRI cases, each from a single examination from a distinct patient, with each case consisting of four sets of MRI scan data: two sets of T2-weighted images (transaxial and sagittal; DICOM format), Ktrans images (computed from dynamic contrast-enhanced (DCE) images; mhd format), and apparent diffusion coefficient (ADC) images (computed from diffusion-weighted (DWI) imaging; DICOM format). These cases contain a total of 182 findings (lesions): the training set contains 112 findings, and the test set contains 70 findings. Location and a reference thumbnail image will be provided for each lesion.

Each lesion will have known pathology-defined Gleason Grade Group as defined in the reference:

Epstein JI et al. The 2014 International Society of Urologic Pathology (ISUP) Consensus Conference on Gleason Grading of Prostatic Carcinoma: Definition of grading patterns and proposal for a new grading system. Am J Surg Pathol, (40)244-252, 2016).

Grade Group 1 (Gleason score <=6): Only individual discrete well-formed glands

Grade Group 2 (Gleason score 3+4=7): Predominantly well-formed glands with lesser component of poorly-formed/fused/cribriform glands

Grade Group 3 (Gleason score 4+3=7): Predominantly poorly formed/fused/cribriform glands with lesser component of well-formed glands

Grade Group 4 (Gleason score 4+4=8; 3+5=8; 5+3=8): (1) Only poorly-formed/fused/cribriform glands or (2) predominantly well-formed glands and lesser component lacking glands or (3) predominantly lacking glands and lesser component of well-formed glands

Grade Group 5 (Gleason scores 9 or 10): Lacks gland formation (or with necrosis) with or without poorly formed/fused/cribriform glands

Participants may use the training set cases in any manner they would like for the purpose of training their systems; there will be no restrictions on the use of the data or the advice sought from local experts for training purposes. The test set cases, however, are to be manipulated, processed, and analyzed without human intervention (although human-supervised delineation of the prostate gland border or the gross lesion margin will be allowed, if the operation of a computerized system requires such input).

The set of training cases being used for PROSTATEx-2 is a subset of the training cases that were used for PROSTATEx (which was held in conjunction with SPIE Medical Imaging 2017), and the set of test cases being used for PROSTATEx-2 is a subset of the test cases that were used for PROSTATEx. Please note that the findings from PROSTATEx that are not included in PROSTATEx-2 are findings for which Gleason Grade Group assignment is not possible (they are benign or non-tumor).

Participants are free to download the training set and, subsequently, the test set when these datasets become available. It is important to note that once participants submit their test set Gleason Grade Group output to the challenge organizers, they will be considered fully vested in the Challenge, so that their performance results (without links to the identity of the participant) will become part of any presentations, publications, or subsequent analyses derived from the Challenge at the discretion of the organizers. The truth associated with the test set cases is expected to be made publicly available after publication of the PROSTATEx-2 Challenge and its predecessor PROSTATEx Challenge (http://www.spie.org/PROSTATEx/).

Participation in the PROSTATEx-2 Challenge acknowledges the educational, friendly competition, and community-building nature of this challenge and commits to conduct consistent with this spirit for the advancement of the medical imaging research community. See http://medicalimaging.spiedigitallibrary.org/article.aspx?articleid=2375472 for a discussion of lessons learned from the 2015 LUNGx Challenge, also sponsored by SPIE, AAPM, and NCI.

Output format for the PROSTATEx-2 Challenge test set results:

Method performance will be assessed using a quadratic-weighted Cohen’s kappa coefficient of agreement with the reference standard Gleason Grade Group for each lesion. If the kappa coefficient across multiple methods is the same, the positive predictive value of each method’s ability to identify Gleason Grade Group greater than 1 will be used to break the tie. To facilitate performance assessment across participants for the test cases in the PROSTATEx-2 Challenge, please follow these guidelines:

  1. Using your method, calculate a Gleason Grade Group for each case (lesion) as an integer value on the range [1, 5]. The submitted output must be an integer between 1 and 5, inclusive.
  2. Record your scores for all cases in a CSV file containing three columns with
    1. The first column being the case number (ProxID).
    2. The second column being the lesion number (fid).
    3. The third column being your method's assigned Gleason Grade Group.
  3. Please report your results in the same order as lesions appear in the ProstateX-2-Findings-Test.csv file. Also note that the system expects a file with 71 rows with the first row being the header row followed by your output for the 70 test cases (in the format described above)

Conflict of Interest:

All participants must attest that they are not directly affiliated with the labs of any of the PROSTATEx-2 organizers or major contributors.


Anyone wishing to use these datasets for research, presentation, or publication purposes outside of the PROSTATEx-2 Challenge should acknowledge the SPIE, the AAPM, the NCI, and Radboud University (“Data used in this research were obtained from The Cancer Imaging Archive (TCIA) sponsored by the SPIE, NCI/NIH, AAPM, and Radboud University.”) and cite the following article:

Litjens G, Debats O, Barentsz J, Karssemeijer N, Huisman H. Computer-aided detection of prostate cancer in MRI. IEEE Transactions on Medical Imaging, (33)1083-92, 2014.

Organizers and Major Contributors:

  • Sam Armato, University of Chicago (s-armato@uchicago.edu)
  • Henkjan Huisman, Radboud University (Henkjan.Huisman@radboudumc.nl)
  • Karen Drukker, University of Chicago (kdrukker@uchicago.edu)
  • Lubomir Hadjiiski, University of Michigan (lhadjisk@umich.edu)
  • Keyvan Farahani, NIH/NCI (farahank@mail.nih.gov)
  • Justin Kirby, NIH/NCI (kirbyju@mail.nih.gov)
  • Nicholas Petrick, FDA (nicholas.petrick@fda.hhs.gov)
  • George Redmond, NIH/NCI (gr34m@nih.gov)
  • Maryellen Giger, University of Chicago (m-giger@uchicago.edu)
  • Kenny Cha, University of Michigan (heekon@med.umich.edu)
  • Jayashree Kalpathy-Cramer, Harvard University (kalpathy@nmr.mgh.harvard.edu)
  • Artem Mamonov, Harvard University (amamonov@mgh.harvard.edu)
  • Robert Nordstrom, NIH/NCI (nordstrr@mail.nih.gov)
  • Shayna Knazik, AAPM (shayna@aapm.org)
  • Diane Cline, SPIE (diane@spie.org)


Start: May 15, 2017, 6 p.m.


Start: June 5, 2017, 6 p.m.

Competition Ends

June 26, 2017, 7 p.m.

You must be logged in to participate in competitions.

Sign In