Can artificial intelligence, applied on ultrasound images, discriminate benign and malignant ovarian tumours, and thus be used in the triage of women with these lesions? An external international multicentre validation study by the Ovarian Tumour Machine Learning Collaboration (OMLC)
| ISRCTN | ISRCTN51927471 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN51927471 |
| ClinicalTrials.gov (NCT) | Nil known |
| Clinical Trials Information System (CTIS) | Nil known |
| Protocol serial number | Nil known |
| Sponsors | Stockholm County Council, Stockholm County Council, ALF medicine |
| Funder | SLL: Innovations fonden, ALF-medicin |
- Submission date
- 16/07/2020
- Registration date
- 24/07/2020
- Last edited
- 02/09/2024
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Cancer
Plain English summary of protocol
Background and study aims
Expert ultrasound examination has become the main imaging technique for assessing ovarian lesions. While the diagnostic accuracy is higher in experts than in less experienced doctors, there is a shortage of expert examiners. Every year approximately 10,000 ovarian surgical procedures are performed in Sweden. We believe that up to a quarter of these are unnecessary procedures that could be avoided if expert ultrasound assessment would be available. AI approaches have gained interest in several medical fields where experts visually assess images. Automated imaging AI tools have matched or even surpassed experts. Our own recent data show that artificial intelligence (AI), using deep neural networks (DNN), can discriminating between benign and malignant ovarian tumors with performance on par with ultrasound experts.
Aim: To externally validate our DNN models, and to compare the results to the assessment made by expert ultrasound examiners, in a large international multicentre setting.
Who can participate?
Any secondary/tertiary gynecological/gyneoncological ultrasound referral centre using high-end ultrasound systems (GE Voluson E8, GE Voluson E10, Philips IU22, Philips EPIQ, or similar), that can provide at least 100 consecutive cases (50 benign and 50 malignant) with at least 3 good quality, representative ultrasound images per case.
What does the study involve?
This study involves the validation and the comparison of machine learning models to human experts with regard to assessing ovarian tumours as benign or malignant.
What are the possible benefits and risks of participating?
None
Where is the study run from?
Karolinska Institutet (Sweden)
When is the study starting and how long is it expected to run for?
July 2020 to December 2020
Who is funding the study?
SLL: Innovations fonden, ALF-medicin (Sweden)
Who is the main contact?
1. Elisabeth Epstein, Department of Clinical Science and Education, Karolinska Institutet, Stockholm, Sweden,
Elisabeth.epstein@ki.se
2. Filip Christiansen, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden,
filipchr@kth.se
3. Elliot Epstein, School of Engineering Sciences, KTH Royal Institute of Technology, Stockholm, Sweden,
elliotepstein14@gmail.com
Contact information
Public
Department of Clinical Science and Education, Södersjukhuset, Karolinska Institutet
Sjukusbacken 10
Stockholm
11883
Sweden
 ORCID ID |
0000-0003-2298-7785 |
|---|---|
| Phone | +46 706699019 |
| Elisabeth.epstein@ki.se |
Study information
| Primary study design | Observational |
|---|---|
| Study design | Observational retrospective study |
| Secondary study design | Cross sectional study |
| Study type | Participant information sheet |
| Scientific title | External validation of the deep learning models Ovry-Dx1 and Ovry-Dx2, applied on ultrasound images, to discriminate benign and malignant ovarian tumours. An external international multicentre validation study by the Ovarian Tumour Machine Learning Collaboration (OMLC) |
| Study acronym | OMLC validation study |
| Study objectives | Based on our preliminary findings we hypothesize that DNN models can discriminate between benign and malignant ovarian tumors with performance similar to ultrasound experts, and this performance generalizes to a large scale multicenter setting including images of varying quality. We anticipate that DNN models can be used in the triage of women with ovarian tumours, aiding and improving clinical decision making. Especially in the case of non-expert examiners, an autonomous AI clinical decision support tool is expected to result in higher detection of ovarian cancer, at a lower rate of false positives, and thus a more cost-effective utilization of healthcare resources and reduced morbidity among patients. |
| Ethics approval(s) | Approved 10/11/2020, Swedish Ethical Review Authority (Etikprövningsmyndigheten, Box 2110, 750 02, Uppsala, Sweden; +46 10-475 08 00; registrator@etikprovning.se), ref: DNR 2020-04090 |
| Health condition(s) or problem(s) studied | Ovarian tumours |
| Intervention | Observational study: Multi-centre (n=22) study, including at least 6,000 images from at least 2,000 cases (1,000 benign and 1,000 malignant) of adnexal lesions, with known histological outcome from surgery. Subjective classification of tumours prior to surgery; benign or malignant and the certainty in the assessment will be used for comparative analysis. All cases will also undergo external review by 3 experts from other centres, evaluating tumours as benign or malignant based on the available images from each case. Images and questionnaires will be made available on a web-based platform. |
| Intervention type | Other |
| Primary outcome measure(s) |
Diagnostic performance of the previously developed deep learning models (Ovry-Dx1 and Ovry-Dx2) in discriminating benign and malignant lesions. These models were created by transfer learning on three pre-trained DNNs: VGG16, ResNet50 and MobileNet. Each model was trained, and the outputs calibrated using temperature scaling. An ensemble of the three models was then used to estimate the probability of malignancy based on all images from a given case. Using DNNs, tumours were classified as benign or malignant (Ovry-Dx1); or benign, inconclusive or malignant (Ovry-Dx2). |
| Key secondary outcome measure(s) |
Data collected from patient records: |
| Completion date | 31/12/2022 |
Eligibility
| Participant type(s) | Patient |
|---|---|
| Age group | All |
| Sex | Female |
| Target sample size at registration | 1600 |
| Total final enrolment | 3657 |
| Key inclusion criteria | 1. Women with adnexal lesions undergoing structured ultrasound examination prior to surgery 2. At least 3 good quality, representative ultrasound images per case 3. Histological outcome form surgery available |
| Key exclusion criteria | Does not meet inclusion criteria |
| Date of first enrolment | 31/07/2020 |
| Date of final enrolment | 30/04/2021 |
Locations
Countries of recruitment
- Belgium
- Czech Republic
- Greece
- Italy
- Lithuania
- Philippines
- Poland
- Spain
- Sweden
Study participating centres
Stockholm
11883
Sweden
Division of Gynaecology
Milan
20141
Italy
Department of Obstetrics and Gynecology
First Faculty of Medicine
Prague
50005
Czech Republic
Athens
115 28
Greece
Trieste
34137
Italy
Milan
20157
Italy
Pamplona
31008
Spain
Monza
20900
Italy
Monserrato
Cagliari
09042
Italy
Bologna
40138
Italy
Katowice
40-055
Poland
Lund
22185
Sweden
Vilnius
44307
Lithuania
Prague
100 00
Czech Republic
Barcelona
08028
Spain
Lublin
20-059
Poland
Manila
1000
Philippines
Ancona
60123
Italy
Olbia
07026
Italy
Brescia
25124
Italy
Results and Publications
| Individual participant data (IPD) Intention to share | Yes |
|---|---|
| IPD sharing plan summary | Stored in repository |
| IPD sharing plan | The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Other publications | development and testing ultrasound image analysis using deep neural networks | 03/11/2020 | 02/09/2024 | Yes | No |
| Participant information sheet | Participant information sheet | 11/11/2025 | 11/11/2025 | No | Yes |
| Protocol file | version V4 | 11/12/2020 | No | No |
Additional files
- ISRCTN51927471_PROTOCOL_V4.pdf
- uploaded 11/12/2020
Editorial Notes
02/09/2024: Publication reference added.
12/08/2024: The contact details were updated.
12/12/2022: The total final enrolment was changed from 2495 to 3657.
22/11/2021: The trial participating centres Skåne University Hospital and Ziekenhuis Oost-Limburg were removed.
17/11/2021: The trial participating centre Fondazione Poliambulanza was added.
26/08/2021: The following changes were made to the trial record:
1. The total final enrolment was added.
2. The trial participating centre Dipartimento Scienze della Salute della Donna, del Bambino e di Sanità Pubblica was removed.
24/08/2021: The trial participating centres "New York University School of Medicine, University College London Hospitals" were removed. The trial participating centres "Clinica Ostetrica e Ginecologica, Ospedale “G.Salesi, Mater Olbia Hospital, Gynaecology and Breast care centre" were added.
14/12/2020: The ethics approval was added.
11/12/2020: The following changes were made to the trial record:
1. The recruitment end date was changed from 31/12/2020 to 30/04/2021.
2. Uploaded protocol (not peer reviewed) Version 4, no date.
17/07/2020: Trial’s existence confirmed by SLL: Innovations fonden
