The impact of an AI tool on diagnostic accuracy, review time, and diagnostic confidence in detecting acute abnormalities in non-contrast CT head scans: a study among emergency medicine clinicians, general radiologists, and radiographers
| ISRCTN | ISRCTN17560291 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN17560291 |
| ClinicalTrials.gov (NCT) | NCT06018545 |
| Clinical Trials Information System (CTIS) | Nil known |
| Integrated Research Application System (IRAS) | 310995 |
| Protocol serial number | IRAS 310995 |
| Sponsor | Oxford University Hospitals NHS Trust |
| Funder | NHS transformation directorate (NHSX) |
- Submission date
- 07/09/2023
- Registration date
- 16/10/2023
- Last edited
- 12/11/2025
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Injury, Occupational Diseases, Poisoning
Plain English summary of protocol
Background and study aims
The study aims to evaluate the impact of an AI tool called qER 2.0 EU on the diagnostic accuracy, speed, and confidence of healthcare professionals who review non-contrast CT head scans. The study will involve 30 readers, including general radiologists, emergency medicine clinicians, and CT radiographers, who will interpret 150 non-contrast CT head scans, first without and then with the assistance of the AI tool. The scans will include 60 control cases and 90 abnormal cases with intracranial haemorrhage, brain infarct, midline shift, or skull fracture. The study will assess the stand-alone performance of the AI tool and its impact on the readers' performance.
Who can participate?
Emergency medicine consultants and registrars, general radiologist consultants and registrars, and CT radiographers who review CT head scans as part of their clinical practice.
What does the study involve?
30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen EM clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the qER 2.0 EU AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers’ performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence.
What are the possible benefits and risks of participating?
None
Where is the study run from?
Oxford University Hospitals NHS Trust (UK)
When is the study starting and how long is it expected to run for?
October 2022 to October 2024
Who is funding the study?
This work was supported by the NHSX AI in Health and Care Award (UK)
Who is the main contact?
Prof. Alex Novak, alex.novak@ouh.nhs.uk
Contact information
Principal investigator
Headley Way
Headington
Oxford
OX3 9DU
United Kingdom
 ORCID ID |
0009-0006-4086-3152 |
|---|---|
| Phone | +44 7944 653970 |
| alex.novak@ouh.nhs.uk |
Study information
| Primary study design | Observational |
|---|---|
| Study design | Observational cohort study that is retrospective multicenter and multireader |
| Secondary study design | Cohort study |
| Participant information sheet | 44254 STEDI2 AI-REACT study PIS V1.0 29Nov2022.pdf |
| Scientific title | AI Assisted Reader Evaluation in Acute Computed Tomography (CT) head interpretation (AI-REACT) |
| Study acronym | AI-REACT |
| Study objectives | The purpose of the study is to assess the impact of an Artificial Intelligence (AI) tool called qER 2.0 EU on the performance of readers, including general radiologists, emergency medicine clinicians, and radiographers, in interpreting non-contrast CT head scans. The study aims to evaluate the changes in accuracy, review time, and diagnostic confidence when using the AI tool. It also seeks to provide evidence on the diagnostic performance of the AI tool and its potential to improve efficiency and patient care in the context of the National Health Service (NHS). The study will use a dataset of 150 CT head scans, including both control cases and abnormal cases with specific abnormalities. The results of this study will inform larger follow-up studies in real-life Emergency Department (ED) settings. |
| Ethics approval(s) |
Approved 13/12/2022, Research Ethics Committee of the Central University (CUREC) and the Interdivisional Research Ethics Committee of Medical Sciences (IDREC) (Churchill Drive, Headington, Oxford, OX3 7GB, United Kingdom; +44 1865 (6)16577; ethics@medsci.ox.ac.uk), ref: R80145/RE002 |
| Health condition(s) or problem(s) studied | Extradural haemorrhage, Subdural haemorrhage, Subarachnoid haemorrhage, Intraparenchymal haemorrhage, Intraventricular haemorrhage, Brain infarct or stroke, Intracraneal Mass effect, Skull fractures |
| Intervention | A retrospective dataset of 150 non-contrast CT head scans will be compiled, to include 60 control cases and 90 abnormal cases with intracranial haemorrhage, brain infarct, midline shift or skull fracture. The intracranial haemorrhage cases will be sub-classified into extradural, subdural, subarachnoid, intraparenchymal, and intraventricular. 30 readers will be recruited across four NHS trusts including ten general radiologists, fifteen EM clinicians, and five CT radiographers of varying seniority. Readers will interpret each scan first without, then with, the assistance of the qER 2.0 EU AI tool, with an intervening 4-week washout period. Using a panel of neuroradiologists as ground truth, the stand-alone performance of qER will be assessed, and its impact on the readers’ performance will be analysed as change in accuracy, mean review time per scan, and self-reported diagnostic confidence. Subgroup analyses will be performed by reader professional group, reader seniority, pathological finding, and neuroradiologist-rated difficulty. |
| Intervention type | Other |
| Primary outcome measure(s) |
Reader and qER performance will be evaluated as sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and Area Under Receiver Operating Characteristic Curve (AUC). Reader performance will be evaluated with and without AI assistance. Reader speed will be evaluated as the mean review time per scan, with and without AI assistance. |
| Key secondary outcome measure(s) |
Reader confidence will be evaluated as self-reported diagnostic confidence on a 10 point visual analogue scale, with 0 being not completely confident, and 10 being totally confident, evaluated with and without AI assistance. |
| Completion date | 02/10/2024 |
Eligibility
| Participant type(s) | Health professional |
|---|---|
| Age group | Mixed |
| Lower age limit | 18 Years |
| Upper age limit | 99 Years |
| Sex | All |
| Target sample size at registration | 30 |
| Total final enrolment | 30 |
| Key inclusion criteria | Emergency medicine consultants and registrars, general radiologist consultants and registrars, and CT radiographers who review CT head scans as part of their clinical practice |
| Key exclusion criteria | 1. Neuroradiologists 2. (Non-radiologist groups) Clinicians with previous formal postgraduate CT reporting training 3. (Emergency Medicine group) Clinicians with previous career in radiology/neurosurgery to registrar level |
| Date of first enrolment | 27/06/2023 |
| Date of final enrolment | 04/08/2023 |
Locations
Countries of recruitment
- United Kingdom
- England
- Scotland
Study participating centres
Headley Way
Headington
Oxford
OX3 9DU
England
Westminster Bridge Road
London
SE1 7EH
England
North Shields
NE29 8NH
England
Gartnavel Royal Hospital
1055 Great Western Road Glasgow
Glasgow
G12 0XH
Scotland
Results and Publications
| Individual participant data (IPD) Intention to share | Yes |
|---|---|
| IPD sharing plan summary | Published as a supplement to the results publication |
| IPD sharing plan | All data generated or analysed during this study will be included in the subsequent results publication |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Protocol article | 12/02/2024 | 12/11/2025 | Yes | No | |
| Other unpublished results | version 0.1 | 12/11/2025 | No | No | |
| Other unpublished results | Supplementary materials version 2.4 |
12/11/2025 | No | No | |
| Participant information sheet | version 1.0 | 29/11/2022 | 12/09/2023 | No | Yes |
| Protocol file | 12/09/2023 | No | No | ||
| Statistical Analysis Plan | version 0.1 | 12/11/2025 | No | No |
Additional files
- 44254 STEDI2 AI-REACT study PIS V1.0 29Nov2022.pdf
- Participant information sheet
- 44254 AI-REACT Protocol.pdf
- Protocol file
- ISRCTN17560291 AI-REACT Statistical Plan v0.1.pdf
- Statistical Analysis Plan
- ISRCTN17560291 AI-REACT Results v0.1.pdf
- Other unpublished results
- ISRCTN17560291 Novak et al AI-REACT Supplementary Materials v2.4.pdf
- Supplementary materials
Editorial Notes
12/11/2025: The following changes were made to the trial record:
1. The statistical analysis plan was uploaded as an additional file.
2. Publication reference added.
3. A file of results was uploaded as an additional file.
4. A file of supplementary materials was uploaded as an additional file.
12/09/2023: Trial's existence confirmed by NHS HRA.
