Condition category
Cancer
Date applied
16/07/2020
Date assigned
24/07/2020
Last edited
24/07/2020
Prospective/Retrospective
Prospectively registered
Overall trial status
Ongoing
Recruitment status
Recruiting

Plain English Summary

Background and study aims
Expert ultrasound examination has become the main imaging technique for assessing ovarian lesions. While the diagnostic accuracy is higher in experts than in less experienced doctors, there is a shortage of expert examiners. Every year approximately 10,000 ovarian surgical procedures are performed in Sweden. We believe that up to a quarter of these are unnecessary procedures that could be avoided if expert ultrasound assessment would be available. AI approaches have gained interest in several medical fields where experts visually assess images. Automated imaging AI tools have matched or even surpassed experts. Our own recent data show that artificial intelligence (AI), using deep neural networks (DNN), can discriminating between benign and malignant ovarian tumors with performance on par with ultrasound experts.
Aim: To externally validate our DNN models, and to compare the results to the assessment made by expert ultrasound examiners, in a large international multicentre setting.

Who can participate?
Any secondary/tertiary gynecological/gyneoncological ultrasound referral centre using high-end ultrasound systems (GE Voluson E8, GE Voluson E10, Philips IU22, Philips EPIQ, or similar), that can provide at least 100 consecutive cases (50 benign and 50 malignant) with at least 3 good quality, representative ultrasound images per case.

What does the study involve?
This study involves the validation and the comparison of machine learning models to human experts with regard to assessing ovarian tumours as benign or malignant.

What are the possible benefits and risks of participating?
None

Where is the study run from?
Karolinska Institutet (Sweden)

When is the study starting and how long is it expected to run for?
July 2020 to December 2020

Who is funding the study?
SLL: Innovations fonden, ALF-medicin (Sweden)

Who is the main contact?
1. Elisabeth Epstein, Department of Clinical Science and Education, Karolinska Institutet, Stockholm, Sweden,
elisabeth.epstein@sll.se
2. Filip Christiansen, School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden,
filipchr@kth.se
Elliot Epstein, School of Engineering Sciences, KTH Royal Institute of Technology, Stockholm, Sweden,
elliotepstein14@gmail.com

Trial website

Contact information

Type

Public

Primary contact

Prof Elisabeth Epstein

ORCID ID

http://orcid.org/0000-0003-2298-7785

Contact details

Department of Obstetrics and Gynecology
Södersjukhuset
Stockholm
18883
Sweden
+46 706699019
elisabeth.epstein@sll.se

Additional identifiers

EudraCT number

Nil known

ClinicalTrials.gov number

Nil known

Protocol/serial number

Nil known

Study information

Scientific title

External validation of the deep learning models Ovry-Dx1 and Ovry-Dx2, applied on ultrasound images, to discriminate benign and malignant ovarian tumours. An external international multicentre validation study by the Ovarian Tumour Machine Learning Collaboration (OMLC)

Acronym

OMLC validation study

Study hypothesis

Based on our preliminary findings we hypothesize that DNN models can discriminate between benign and malignant ovarian tumors with performance similar to ultrasound experts, and this performance generalizes to a large scale multicenter setting including images of varying quality. We anticipate that DNN models can be used in the triage of women with ovarian tumours, aiding and improving clinical decision making. Especially in the case of non-expert examiners, an autonomous AI clinical decision support tool is expected to result in higher detection of ovarian cancer, at a lower rate of false positives, and thus a more cost-effective utilization of healthcare resources and reduced morbidity among patients.

Ethics approval

Approval pending, Swedish Ethical Review Authority (Etikprövningsmyndigheten, Box 2110,
750 02, Uppsala, Sweden; +46 10-475 08 00; registrator@etikprovning.se)

Study design

Observational retrospective study

Primary study design

Observational

Secondary study design

Cross sectional study

Trial setting

Other

Trial type

Diagnostic

Patient information sheet

No participant information sheet available

Condition

Ovarian tumours

Intervention

Observational study: Multi-centre (n=22) study, including at least 6,000 images from at least 2,000 cases (1,000 benign and 1,000 malignant) of adnexal lesions, with known histological outcome from surgery. Subjective classification of tumours prior to surgery; benign or malignant and the certainty in the assessment will be used for comparative analysis.
All cases will also undergo external review by 3 experts from other centres, evaluating tumours as benign or malignant based on the available images from each case. Images and questionnaires will be made available on a web-based platform.

Intervention type

Other

Phase

Drug names

Primary outcome measure

Diagnostic performance of the previously developed deep learning models (Ovry-Dx1 and Ovry-Dx2) in discriminating benign and malignant lesions. These models were created by transfer learning on three pre-trained DNNs: VGG16, ResNet50 and MobileNet. Each model was trained, and the outputs calibrated using temperature scaling. An ensemble of the three models was then used to estimate the probability of malignancy based on all images from a given case. Using DNNs, tumours were classified as benign or malignant (Ovry-Dx1); or benign, inconclusive or malignant (Ovry-Dx2).

Secondary outcome measures

Data collected from patient records:
1. Case ID
2. Subjective expert assessment prior to surgery
3. Classification of tumours (benign, borderline or malignant)
4. The certainty in the assessment (uncertain vs. certain)
5. Histological outcome (benign/malignant)
6. Specific histological diagnosis form surgery
7. Date of examination
8. Ultrasound system used

Overall trial start date

16/07/2020

Overall trial end date

31/12/2022

Reason abandoned (if study stopped)

Eligibility

Participant inclusion criteria

1. Women with adnexal lesions undergoing structured ultrasound examination prior to surgery
2. At least 3 good quality, representative ultrasound images per case
3. Histological outcome form surgery available

Participant type

Patient

Age group

All

Gender

Female

Target number of participants

at least 1600

Participant exclusion criteria

Does not meet inclusion criteria

Recruitment start date

31/07/2020

Recruitment end date

31/12/2020

Locations

Countries of recruitment

Belgium, Czech Republic, Greece, Italy, Lithuania, Philippines, Poland, Spain, Sweden, United Kingdom, United States of America

Trial participating centre

Södersjukhuset
Department of Obstetrics and Gynecology
Stockholm
11883
Sweden

Trial participating centre

Dipartimento Scienze della Salute della Donna, del Bambino e di Sanità Pubblica
Fondazione Policlinico Universitario Agostino Gemelli Istituto di Ricovero e Cura a Carattere Scientifico
Rome
00168
Italy

Trial participating centre

Skåne University Hospital
Department of Obstetrics and Gynaecology
Malmö
205 02
Sweden

Trial participating centre

European Institute of Oncology IRCCS
Preventive Gynaecology Unit Division of Gynaecology
Milan
20141
Italy

Trial participating centre

Charles University and General University Hospital
Gynaecological Oncology Centre Department of Obstetrics and Gynecology First Faculty of Medicine
Prague
50005
Czech Republic

Trial participating centre

Alexandra Hospital
First Department of Obstetrics and Gynaecology
Athens
115 28
Greece

Trial participating centre

IRCCS “Burlo Garofolo”
Institute for Maternal and Child Health
Trieste
34137
Italy

Trial participating centre

Biomedical and Clinical Sciences Institute L. Sacco
Department of Obstetrics and Gynaecology
Milan
20157
Italy

Trial participating centre

Clinica Universidad de Navarra
Department of Obstetrics and Gynaecology
Pamplona
31008
Spain

Trial participating centre

San Gerardo Hospital
Clinic of Obstetrics and Gynaecology
Monza
20900
Italy

Trial participating centre

Policlinico Universitario Duilio Casula
Department of Obstetrics and Gynaecology Monserrato
Cagliari
09042
Italy

Trial participating centre

S Orsola-Malpighi Hospital
Gynecology and Reproductive Medicine Unit
Bologna
40138
Italy

Trial participating centre

S Orsola-Malpighi Hospital
Gynecology and Reproductive Medicine Unit
Bologna
40138
Italy

Trial participating centre

Ziekenhuis Oost-Limburg
Department of Obstetrics and Gynaecology
Genk
3600
Belgium

Trial participating centre

School of Health Sciences in Katowice
Department of Perinatology and Oncological Gynaecology
Katowice
40-055
Poland

Trial participating centre

Skåne University Hospital Lund
Department of Obstetrics and Gynaecology
Lund
22185
Sweden

Trial participating centre

Kaunas Medical University Hospital
Department of Obstetrics and Gynecology
Vilnius
44307
Lithuania

Trial participating centre

Third Faculty of Medicine, Charles University
Institute for the Care of Mother and Child
Prague
100 00
Czech Republic

Trial participating centre

Hospital Universitario Dexeus
Department of Obstetrics, Gynecology, and Reproduction
Barcelona
08028
Spain

Trial participating centre

Medical University of Lublin
First Department of Gynaecological Oncology and Gynaecology
Lublin
20-059
Poland

Trial participating centre

New York University School of Medicine
Department of Obstetrics and Gynecology
New York
10016
United States of America

Trial participating centre

St Luke´s Medical Centre
Department of Obstetrics and Gynecology
Manila
1000
Philippines

Trial participating centre

University College London Hospitals
Department of Gynaecology 235 Euston Road
London
NW1 2BU
United Kingdom

Sponsor information

Organisation

Stockholm County Council

Sponsor details

Box 225 50
Stockholm
104 22
Sweden
+46 72569 41 15
annette.alkebo@sll.se

Sponsor type

Government

Website

https://forskningsstod.vmi.se/Ansokan/start.asp

Organisation

Stockholm County Council, ALF medicine

Sponsor details

Box 225 50
Stockholm
104 22
Sweden
+4672598 12 65
kristin.blidberg@sll.se

Sponsor type

Government

Website

https://forskningsstod.vmi.se/Ansokan/start.asp

Funders

Funder type

Government

Funder name

SLL: Innovations fonden, ALF-medicin

Alternative name(s)

Funding Body Type

Funding Body Subtype

Location

Results and Publications

Publication and dissemination plan

Planned publication in high-impact peer-revewed journal within 1-1.5 years.
OMLC collaborators will be offered to use the image data set to validate their own AI-models.

IPD sharing statement:
The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository.

Intention to publish date

31/12/2023

Participant level data

Stored in repository

Basic results (scientific)

Publication list

Publication citations

Additional files

Editorial Notes

17/07/2020: Trial’s existence confirmed by SLL: Innovations fonden