The impact of an AI tool on diagnostic accuracy, review time, and diagnostic confidence in detecting acute abnormalities in non-contrast CT head scans: a study among emergency medicine clinicians, general radiologists, and radiographers
| ISRCTN | ISRCTN17560291 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN17560291 |
| IRAS number | 310995 |
| ClinicalTrials.gov number | NCT06018545 |
| Secondary identifying numbers | IRAS 310995 |
- Submission date
- 07/09/2023
- Registration date
- 16/10/2023
- Last edited
- 16/10/2023
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Injury, Occupational Diseases, Poisoning
Plain English Summary
Background and study aims
The study aims to evaluate the impact of an AI tool called qER 2.0 EU on the diagnostic accuracy, speed, and confidence of healthcare professionals who review non-contrast CT head scans. The study will involve 30 readers, including general radiologists, emergency medicine clinicians, and CT radiographers, who will interpret 150 non-contrast CT head scans, first without and then with the assistance of the AI tool. The scans will include 60 control cases and 90 abnormal cases with intracranial haemorrhage, brain infarct, midline shift, or skull fracture. The study will assess the stand-alone performance of the AI tool and its impact on the readers' performance.
Who can participate?
Emergency medicine consultants and registrars, general radiologist consultants and registrars, and CT radiographers who review CT head scans as part of their clinical practice.
What does the study involve?
30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen EM clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the qER 2.0 EU AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers’ performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence.
What are the possible benefits and risks of participating?
None
Where is the study run from?
Oxford University Hospitals NHS Trust (UK)
When is the study starting and how long is it expected to run for?
October 2022 to October 2024
Who is funding the study?
This work was supported by the NHSX AI in Health and Care Award (UK)
Who is the main contact?
Prof. Alex Novak, alex.novak@ouh.nhs.uk
Contact information
Principal Investigator
Headley Way
Headington
Oxford
OX3 9DU
United Kingdom
 ORCID ID |
0009-0006-4086-3152 |
|---|---|
| Phone | +44 7944 653970 |
| alex.novak@ouh.nhs.uk |
Study information
| Study design | Observational cohort study that is retrospective multicenter and multireader |
|---|---|
| Primary study design | Observational |
| Secondary study design | Cohort study |
| Study setting(s) | Hospital, Medical and other records |
| Study type | Other, Efficacy |
| Participant information sheet | 44254 STEDI2 AI-REACT study PIS V1.0 29Nov2022.pdf |
| Scientific title | AI Assisted Reader Evaluation in Acute Computed Tomography (CT) head interpretation (AI-REACT) |
| Study acronym | AI-REACT |
| Study hypothesis | The purpose of the study is to assess the impact of an Artificial Intelligence (AI) tool called qER 2.0 EU on the performance of readers, including general radiologists, emergency medicine clinicians, and radiographers, in interpreting non-contrast CT head scans. The study aims to evaluate the changes in accuracy, review time, and diagnostic confidence when using the AI tool. It also seeks to provide evidence on the diagnostic performance of the AI tool and its potential to improve efficiency and patient care in the context of the National Health Service (NHS). The study will use a dataset of 150 CT head scans, including both control cases and abnormal cases with specific abnormalities. The results of this study will inform larger follow-up studies in real-life Emergency Department (ED) settings. |
| Ethics approval(s) |
Approved 13/12/2022, Research Ethics Committee of the Central University (CUREC) and the Interdivisional Research Ethics Committee of Medical Sciences (IDREC) (Churchill Drive, Headington, Oxford, OX3 7GB, United Kingdom; +44 1865 (6)16577; ethics@medsci.ox.ac.uk), ref: R80145/RE002 |
| Condition | Extradural haemorrhage, Subdural haemorrhage, Subarachnoid haemorrhage, Intraparenchymal haemorrhage, Intraventricular haemorrhage, Brain infarct or stroke, Intracraneal Mass effect, Skull fractures |
| Intervention | A retrospective dataset of 150 non-contrast CT head scans will be compiled, to include 60 control cases and 90 abnormal cases with intracranial haemorrhage, brain infarct, midline shift or skull fracture. The intracranial haemorrhage cases will be sub-classified into extradural, subdural, subarachnoid, intraparenchymal, and intraventricular. 30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen EM clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the qER 2.0 EU AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers’ performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty. |
| Intervention type | Other |
| Primary outcome measure | Reader and qER performance will be evaluated as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Area Under Receiver Operating Characteristic Curve (AUC). Reader performance will be evaluated with and without AI assistance. Reader speed will be evaluated as the mean review time per scan, with and without AI assistance. |
| Secondary outcome measures | Reader confidence will be evaluated as self-reported diagnostic confidence on a 10 point visual analogue scale, with 0 being not completely confident, and 10 being totally confident, evaluated with and without AI assistance. |
| Overall study start date | 01/10/2022 |
| Overall study end date | 02/10/2024 |
Eligibility
| Participant type(s) | Health professional |
|---|---|
| Age group | Adult |
| Lower age limit | 18 Years |
| Sex | Both |
| Target number of participants | 30 |
| Total final enrolment | 30 |
| Participant inclusion criteria | Emergency medicine consultants and registrars, general radiologist consultants and registrars, and CT radiographers who review CT head scans as part of their clinical practice |
| Participant exclusion criteria | 1. Neuroradiologists 2. (Non-radiologist groups) Clinicians with previous formal postgraduate CT reporting training 3. (Emergency Medicine group) Clinicians with previous career in radiology/neurosurgery to registrar level |
| Recruitment start date | 27/06/2023 |
| Recruitment end date | 04/08/2023 |
Locations
Countries of recruitment
- England
- Scotland
- United Kingdom
Study participating centres
Headley Way
Headington
Oxford
OX3 9DU
United Kingdom
Westminster Bridge Road
London
SE1 7EH
United Kingdom
North Shields
NE29 8NH
United Kingdom
Gartnavel Royal Hospital
1055 Great Western Road Glasgow
Glasgow
G12 0XH
United Kingdom
Sponsor information
Hospital/treatment centre
John Radcliffe Hospital
Headley Way
Headington
Oxford
OX3 9DU
England
United Kingdom
| Phone | +44 300 304 7777 |
|---|---|
| palsjr@ouh.nhs.uk | |
| Website | https://www.ouh.nhs.uk |
"ROR" |
https://ror.org/03h2bh287 |
Funders
Funder type
Government
No information available
Results and Publications
| Intention to publish date | 01/10/2025 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Published as a supplement to the results publication |
| Publication and dissemination plan | Planned publication in a high-impact peer-reviewed journal |
| IPD sharing plan | All data generated or analysed during this study will be included in the subsequent results publication |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Participant information sheet | version 1.0 | 29/11/2022 | 12/09/2023 | No | Yes |
| Protocol file | 12/09/2023 | No | No |
Additional files
Editorial Notes
12/09/2023: Trial's existence confirmed by NHS HRA.
