Assessment of the performance of a deep learning system at identifying breast cancer on screening mammograms using de-identified historic data.
| ISRCTN | ISRCTN18056078 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN18056078 |
| Secondary identifying numbers | AUX-07-2018-KMT |
- Submission date
- 06/03/2019
- Registration date
- 20/03/2019
- Last edited
- 07/04/2025
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Cancer
Plain English Summary
Current plain English summary as of 25/05/2021:
Background and study aims:
Breast cancer is a leading cause of cancer-related mortality among women worldwide, accounting for approximately 600,000 deaths annually.
There is a need for rigorous large-scale studies to assess the performance of AI for mammography in double reading. This should be done on diverse cohorts of women across multiple screening sites and programmes, and on unenriched data representative of a true screening population.
The aim of this study is to evaluate the ability of a novel AI system to act as a reliable independent reader in a double reading workflow, as well as demonstrating its standalone performance compared to the historical results.
Who can participate?
Being a retrospective study, no participants are directly involved in the study, and there will be no effect or change to any participant’s care. The study will evaluate the AI system based on its analysis of historical, de-identified cases from study sites where outcomes data (e.g. biopsy, histopathology results, follow-up information) is also collected.
What does the study involve?
Eligible cases will be presented to the AI system for analysis.
What are the possible benefits and risks of participating?
No benefits or risks of participating are anticipated.
Where is the study run from?
Selected study sites in the UK and Hungary
When is the study starting and how long is it expected to run for?
From September 2018 to January 2021
Who is funding the study?
1. Innovate UK (UK)
2. Kheiron Medical Technologies (UK)
Who is the main contact?
Dr. Annie Ng
annie@kheironmed.com
Previous plain English summary:
Background and study aims
Breast cancer is one of the most frequently occurring cancer types in women worldwide. Breast screening is considered to be the world-wide gold standard for early detection and control of breast cancer. Mammography reading for breast screening is known to be a laboriously repetitive and meticulous task, and many sites struggle to meet required performance targets (NHS, EU), Radiologists currently have no effective practical support tools for reading mammography images; however, with the application of leading-edge deep learning techniques, the sponsor has developed software for the accurate analysis of mammograms to support the diagnosis of breast cancer.
This study aims to calibrate and validate the software's performance on retrospective (historic) data from NHS trusts and mammography units. By doing this study, we will learn how effective and generalisable the software can be in supporting radiologists in breast screening in the UK.
Who can participate?
Being a retrospective study, no patients will be directly involved in the study, and there will be no effect or change to any patient's care. The study will evaluate the software based on its analysis of validated de-identified historical cases from investigational sites where the outcomes (i.e. biopsy results, normal follow-up) are already known.
What does the study involve?
Historic data from NHS trusts and mammography units will be fed into the software to test functionality and performance.
What are the possible benefits and risks of participating?
Not applicable
Where is the study run from?
Kheiron Medical Technologies, London
When is the study starting and how long is it expected to run for?
September 2018 to October 2020 (updated 24/02/2021, previously: July 2020)
Who is funding the study?
1. Innovate UK
2. Kheiron Medical Technologies
Who is the main contact?
Dr Annie Ng
annie@kheironmed.com
Contact information
Scientific
Kheiron Medical Technologies
London
EC1V 9BG
United Kingdom
| Phone | +44 (0)7379467701 |
|---|---|
| annie@kheironmed.com |
Scientific
Leeds Teaching Hospital NHS Trust
Leeds
LS9 7TF
United Kingdom
| Phone | 0113 243 3144 |
|---|---|
| nisha.sharma2@nhs.net |
Study information
| Study design | Multi-national multi-site retrospective study |
|---|---|
| Primary study design | Observational |
| Secondary study design | Case-control study |
| Study setting(s) | Hospital |
| Study type | Screening |
| Participant information sheet | None available |
| Scientific title | A retrospective multi-centre clinical investigation of a novel medical technology solution in the assessment of mammography images |
| Study hypothesis | Current study hypothesis as of 25/05/2021: The purpose of the study is to evaluate the performance of a novel AI system in detecting breast cancer. The study aims to evaluate the AI system's ability to act as a reliable independent reader in a double reading workflow, as well as demonstrating its standalone performance compared to the historical results. Previous study hypothesis: The purpose of the study is to calibrate and validate a deep learning software system that provides breast radiologists with recall decision support in a breast screening setting. The performance of this software in supporting recall decisions in breast screening is validated on retrospective data from multiple sites from a cohort of women who have undergone routine mammographic screening for breast cancer and have sufficient follow-up imaging or biopsy data. |
| Ethics approval(s) | Current ethics approval as of 25/05/2021: 1. Approved 03/10/2018, HRA and Health and Care Research Wales (HCRW) (Castle Bridge 4, 15-19 Cowbridge Rd E, Cardiff CF11 9AB; 029 2023 0457; hra.approval@nhs.net), Ref: 19/HRA/0376 2. Approved 03/07/2018, ETT-TUKEB Medical Research Council, Scientific and Research Ethics Committee, Hungary (1051 Budapest, Széchenyi István tér 7-8; (+36 1) 795-1197; tukeb@emmi.gov.hu), ref: OGYÉI/46651-4/2020 Previous ethics approval: Approved 03/10/2018, HRA and Health and Care Research Wales (HCRW) (Castle Bridge 4, 15-19 Cowbridge Rd E, Cardiff CF11 9AB; 029 2023 0457; hra.approval@nhs.net), Ref: 19/HRA/0376 |
| Condition | Breast cancer |
| Intervention | The intervention is the sponsor's deep learning software, assessed on de-identified randomised retrospective breast screening cases and outcomes. Comparison is made against the control arm of existing reference outcomes within the retrospective dataset where the deep learning software was not in use. |
| Intervention type | Device |
| Pharmaceutical study type(s) | |
| Phase | Not Applicable |
| Drug / device / biological / vaccine name(s) | - |
| Primary outcome measure | Current primary outcome measure as of 25/05/2021: Standalone sensitivity and specificity performance of the AI system. Previous primary outcome measure: Rate of detection of malignancy of the Sponsor’s deep learning software measured using patient notes. |
| Secondary outcome measures | Current secondary outcome measures as of 25/05/2021: 1. Non-inferiority and superiority testing of the AI system as an independent reader in a double reading workflow compared to national guidelines and historical double reading in terms of recall rate, cancer detection rate, sensitivity, specificity, and interval cancer rate. 2. Comparing the AI system’s standalone performance to the historical first reader. 3. The AI system’s standalone performance in terms of recall rate, cancer detection rate, interval cancer rate, positive predictive value, arbitration rate, and AUC. Previous secondary outcome measures: Secondary aims will assess the software's performance (sensitivity, specificity, percent indeterminate) at varying settings as well as measure its recall rate (percentage of screening mammograms recalled for further assessment). Accuracy is measured in terms of sensitivity (true positive rate), specificity (true negative rate) as compared to a defined Reference Standard, plotted onto Receiver Operator Curves, and an Area Under the Curve (AUC) calculated. Recall rate is the rate of positive reported findings in a given sample. |
| Overall study start date | 01/06/2018 |
| Overall study end date | 31/01/2021 |
Eligibility
| Participant type(s) | Healthy volunteer |
|---|---|
| Age group | Adult |
| Sex | Female |
| Target number of participants | Up to 1,000,000 |
| Participant inclusion criteria | Current participant inclusion criteria as of 25/05/2021: 1. Female participants 2. 4-view mammography cases (with exactly one of each: MLO-R, MLO-L, CC-R, CC-L of the four standard views) produced by certified digital mammography hardware and taken for screening purposes Previous participant inclusion criteria: 1. Female patients 2. Mammography cases for screening purposes, i.e. cases from: a) patients involved in the national breast screening program (depending on the jurisdiction includes women of age 45-73 who are called for examination via a letter by the national health authorities based on the population database), and b) women outside the national breast screening program who decided on their own to participate as per standard of care 3. Cases with images in DICOM format 4. Cases with images produced by certified digital mammography hardware 5. Cases with one set of all of the 4 standard mammography images (i.e. exactly one of each: MLO-R, MLO-L, CC-R, CC-L) present (no images missing and no extra images) 6. Cases with available historical outcome information as specified below*: (Outcome information: Confirmed positive case: malignancy is confirmed by a decisive biopsy, cytology or histology of the surgical specimen within 250 days after the time of the image acquisition date. Confirmed negative case: a negative follow-up result is available at least 34 months after the image acquisition date (with no malignant operation and no malignancy indication in that period.) *This inclusion criteria only applies to sensitivity/specificity analysis (not recall rate analysis) |
| Participant exclusion criteria | Current participant exclusion criteria as of 25/05/2021: 1. Male participants 2. Participants from whom any image data is used during training, calibration, or testing for the technology development of the deep learning model 3. Non-original, magnified, or spot-compressed images Previous participant exclusion criteria: 1. Male patients 2. Images that are non-original images (e.g. post-processed images) 3. Magnified images (in the DICOM file the View Modifier Code Sequence (0054, 0222) has either of the values: R- 102D6, “Magnification” or R-102D7, “Spot compression”) 4. Cases with indication of a breast operation due to malignancy in the past medical history 5. Cases dated after a breast cancer confirmed by biopsy, cytology or histology 6. All patients of whom any image data was used during training, calibration or testing during the technology development of the deep learning model. Note: hormone replacement therapy in the past medical history is not an exclusion criterion. |
| Recruitment start date | 03/10/2018 |
| Recruitment end date | 01/09/2019 |
Locations
Countries of recruitment
- England
- Hungary
- United Kingdom
Study participating centres
Beckett Street
Leeds
LS9 7TF
United Kingdom
Derby Road
Nottingham
NG7 2UH
United Kingdom
Greetwell Road
Lincoln
LN2 4AX
United Kingdom
Budapest
1125
Hungary
Budapest
7100
Hungary
Budapest
6000
Hungary
Budapest
5000
Hungary
Sponsor information
Industry
Stylus Building
112-116 Old Street
London
EC1V 9BG
United Kingdom
| Phone | +44 7379467701 |
|---|---|
| annie@kheironmed.com | |
| Website | http://www.kheironmed.com |
"ROR" |
https://ror.org/01r3ct535 |
Funders
Funder type
Government
Government organisation / National government
- Alternative name(s)
- innovateuk
- Location
- United Kingdom
No information available
Results and Publications
| Intention to publish date | 30/12/2021 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Not expected to be made available |
| Publication and dissemination plan | Peer reviewed publication is anticipated, alongside academic conference scientific presentations. Results will be submitted to regulatory authorities for the purposes of medical device certification. |
| IPD sharing plan | Current IPD sharing statement as of 07/06/2021: The data generated or analysed during the study cannot be shared at this time due to contractual agreements with study sites. Previous IPD sharing statement: The data sharing plans for the current study are unknown and will be made available at a later date. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Results article | 19/05/2023 | 22/05/2023 | Yes | No | |
| Results article | 06/06/2023 | 07/04/2025 | Yes | No |
Editorial Notes
07/04/2025: Publication reference added.
22/05/2023: Publication reference added.
22/10/2021: Internal review.
21/10/2021: The overall trial end date has been changed from 31/01/2020 to 31/01/2021 and the plain English summary has been updated accordingly.
08/06/2021: The following changes have been made:
1. The sponsor contact has been updated.
2. Internal review.
07/06/2021: The following changes have been made:
1. The scientific contact has been updated.
2. The participant level data has been changed from "To be made available at a later date" to "Not expected to be made available" and the IPD sharing statement has been updated.
3. The intention to publish date has been changed from 30/12/2020 to 30/12/2021.
4. The recruitment start date has been changed from 01/09/2018 to 03/10/2018.
5. The plain English summary has been updated to reflect the changes above.
25/05/2021: The following changes have been made:
1. The study hypothesis has been updated.
2. The ethics approval has been updated.
3. The has been changed from "Multi-site retrospective cohort study" to "Multi-national, multi-site retrospective study".
4. The primary outcome measure has been updated.
5. The secondary outcome measures have been updated.
6. The participant inclusion criteria have been updated.
7. The participant exclusion critera have been updated.
8. The country of recruitment Hungary has been added.
9. The trial participating centres "MaMMa Klinika Budapest & Mozgó Emlőszűrő Állomás", "MaMMa Klinika Szekszárd", "MaMMa Klinika Kecskemét", and "MaMMa Klinika Szolnok" been added.
101. The plain English summary has been updated.
24/02/2021: The following changes were made to the trial record:
1. The overall end date was changed from 30/09/2020 to 31/01/2020.
2. The plain English summary was updated to reflect these changes.
05/08/2020: The following changes were made to the trial record:
1. The overall end date was changed from 31/07/2020 to 30/09/2020.
2. The intention to publish date was changed from 01/09/2020 to 30/12/2020.
08/07/2020: The following changes were made to the trial record:
1. The overall trial end date was changed from 30/06/2020 to 31/07/2020.
2. The intention to publish date was changed from 01/06/2020 to 01/09/2020.
07/05/2020: The following changes have been made:
1. The overall trial end date has been changed from 31/03/2021 to 30/06/2020.
2. A trial contact has been updated.
3. The plain English summary has been updated accordingly.
13/03/2020: The following changes have been made:
1. The sponsor address has been updated.
2. The overall trial end date has been changed from 01/09/2019 to 31/03/2021.
3. The intention to publish date has been changed from 01/12/2019 to 01/06/2020.
4. The trial participating centre "Manchester University NHS Foundation Trust" has been removed.
5. The plain English summary has been updated to reflect the changes above.
21/06/2019: The plain English summary was added.
05/04/2019: Internal review.
07/03/2019: Trial’s existence confirmed by IRB
