Testing an artificial intelligence tool for childhood fracture detection on X-rays
| ISRCTN | ISRCTN12921105 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN12921105 |
| IRAS number | 274278 |
| Secondary identifying numbers | IRAS 274278 |
- Submission date
- 10/11/2023
- Registration date
- 28/12/2023
- Last edited
- 30/08/2024
- Recruitment status
- No longer recruiting
- Overall study status
- Ongoing
- Condition category
- Musculoskeletal Diseases
Plain English summary of protocol
Background and study aims
The study aims to evaluate the impact of an AI tool called BoneView (provided by vendor: Gleamer) on the diagnostic accuracy, confidence and potential change in management plans of healthcare professionals who routinely review bone radiographs of children. The study will involve a minimum of 40 readers, including general radiologists, emergency medicine clinicians, reporting radiographers and orthopaedic surgeons who will interpret 500 paediatric limb radiographs (across 4 body parts - ankle, wrist, elbow, knee) without and with the assistance of the AI tool. The scans will include approximately 35% abnormal (fractured) cases and the rest normal to simulate the normal prevalence of injuries in clinical practice. The study will assess the stand-alone performance of the AI tool and its impact on the readers' performance.
Who can participate?
Radiologists, emergency medicine, orthopaedic surgical consultants and registrars, reporting radiographers and senior triage nurses who review paediatric limb radiographs as part of their clinical practice.
What does the study involve?
40 readers will be recruited as stated above. Readers will interpret each of the 500 paediatric radiographs both without and with AI assistance (BoneView tool). Each reader will provide an opinion on presence/absence of fracture (and location of fracture where relevant), confidence score (scale of 1 to 5, 5 = very confident) and their management plan (based on a drop down menu checklist, tailored for each specialty type).
Using a panel of two consultant radiologists as setting the ground truth, the stand-alone performance of BoneView will be assessed, and its impact on the readers’ performance will be analysed as change in accuracy and changes in self-reported diagnostic confidence.
What are the possible benefits and risks of participating?
None
Where is the study run from?
Great Ormond Street Hospital for Children NHS Foundation Trust (UK)
When is the study starting and how long is it expected to run for?
September 2021 to August 2026
Who is funding the study?
National Institute for Health and Care Research (NIHR) (UK).
Who is the main contact?
Dr. Susan Shelmerdine, susan.shelmerdine@gosh.nhs.uk
Contact information
Public, Scientific, Principal Investigator
Great Ormond Street Hospital NHS Foundation Trust
London
WC1N 3JH
United Kingdom
 ORCID ID |
0000-0001-6642-9967 |
|---|---|
| Phone | +44 2074059200 |
| susan.shelmerdine@gosh.nhs.uk |
Study information
| Study design | Observational cohort study that is retrospective multicenter and multireader |
|---|---|
| Primary study design | Observational |
| Secondary study design | Cohort study |
| Study setting(s) | Hospital, Medical and other records |
| Study type | Diagnostic, Safety, Efficacy |
| Participant information sheet | Not applicable (retrospective study) |
| Scientific title | External validation of an artificial intelligence tool for paediatric fracture detection |
| Study acronym | FRACTURE Study |
| Study objectives | A commercially available AI algorithm can be used for accurate paediatric fracture detection, and potentially improve clinical decision making in a simulated implementation study. |
| Ethics approval(s) | Ethics approval not required |
| Ethics approval additional information | HRA approval has already been granted and REC approval waived for the collection of the retrospective multicentric dataset. Ethical approval is not required for the multi-reader case study of healthcare professionals |
| Health condition(s) or problem(s) studied | Acute fractures in otherwise healthy children (i.e. no underlying skeletal dysplasia, metabolic bone disease) |
| Intervention | Current interventions as of 30/08/2024: A retrospective dataset of 500 scans will be compiled, to include fractures across 4 body parts in children (older than 2 years old, but less than 16 years old; both genders). The body parts include ankles, wrists, elbows and knees. There will therefore be 125 scans per 4 body parts, with each body part being approximately 35% abnormal (i.e. each body part = 81 normal and 44 abnormal (fractured)). This balance of normal to abnormal is intended to better mimic clinical practice whilst still being statistically powered. 40 readers will be recruited across all NHS sites to include at least 6 radiologists (both general, paediatric and musculoskeletal radiologists – of any experience level), 6 reporting radiographers, 6 emergency department staff (i.e. physicians), 6 senior triage nurses and 6 orthopaedic surgeons - each group comprising of staff of varying seniority. Readers will interpret each of the 500 scans twice in a random order during two different reading sessions i.e. the first without AI assistance, and the second with AI assistance. The image viewer platform will randomise the order of the studies so that each reader at each session will be viewing the images in completely random order by abnormality and body part. There will be a washout period of 4 weeks in between the two reading sessions to minimise reader memory of the radiographs reviewed The ground truth (reference standard) will be set by two consultant paediatric radiologists. The stand-alone performance of BoneView will be assessed, and its impact on the readers’ performance will be analysed as changes in accuracy, self-reported diagnostic confidence and changes in diagnostic decision-making. Subgroup analyses will be performed by the reader professional group and reader seniority. Inter- and intra-observer variability will be evaluated. Previous interventions: A retrospective dataset of 500 scans will be compiled, to include fractures across 4 body parts in children (older than 2 years old, but less than 16 years old; both genders). The body parts include ankles, wrists, elbows and knees. There will therefore be 125 scans per 4 body parts, with each body part being approximately 35% abnormal (i.e. each body part = 81 normal and 44 abnormal (fractured)). This balance of normal to abnormal is intended to better mimic clinical practice whilst still being statistically powered. 30 readers will be recruited across all NHS sites to include at least 6 radiologists (both general, paediatric and musculoskeletal radiologists – of any experience level), 6 reporting radiographers, 6 emergency department staff (i.e. physicians), 6 senior triage nurses and 6 orthopaedic surgeons - each group comprising of staff of varying seniority. Readers will interpret each of the 500 scans twice in a random order during two different reading sessions i.e. the first without AI assistance, and the second with AI assistance. The image viewer platform will randomise the order of the studies so that each reader at each session will be viewing the images in completely random order by abnormality and body part. There will be a washout period of 4 weeks in between the two reading sessions to minimise reader memory of the radiographs reviewed The ground truth (reference standard) will be set by two consultant paediatric radiologists. The stand-alone performance of BoneView will be assessed, and its impact on the readers’ performance will be analysed as changes in accuracy, self-reported diagnostic confidence and changes in diagnostic decision-making. Subgroup analyses will be performed by the reader professional group and reader seniority. |
| Intervention type | Other |
| Primary outcome measure | Reader and AI performance of the paediatric X-rays will be evaluated using measures of sensitivity, specificity, positive predictive value, negative predictive value and accuracy, where each correctly identified fracture on an Xray (where one exists) will be counted as a true positive, and each incorrectly identified fracture on an Xray (i.e. an overcall) will be counted as a false positive. Where fractures are present but not identified by the reader, this will constitute a false negative. Where no fracture exists, and none is identified by the reader, this will count as a true negative. The performance measures listed above will be compared for each reader before and after using AI assistance in interpretation of the X-rays. The performance of the AI tool alone will also be evaluated (without a human in the loop) for comparative measure. |
| Secondary outcome measures | 1. The reader confidence in their diagnostic ability to identify or confirm the absence of a fracture per Xray will be measured using a survey provided at the time of reviewing each Xray on the image viewer platform using a 5 point Likert scale (1 = not confident, 5 = very confident). Differences will be compared in these scores before and after the use of the AI tool. 2. The readers’ intended management plan (for the patient) based on the Xray will be provided in a drop down menu (7 options available) provided on the image viewer platform next to each Xray the reader has to interpret. The reader will need to select the single best option they would follow. The differences in theoretical management choices will be compared before and after the use of the AI tool. |
| Overall study start date | 01/09/2021 |
| Completion date | 31/08/2026 |
Eligibility
| Participant type(s) | Health professional |
|---|---|
| Age group | Adult |
| Lower age limit | 18 Years |
| Sex | Both |
| Target number of participants | 40 |
| Key inclusion criteria | Radiographic 'readers' will include radiology consultants and registrars (either general, musculoskeletal or paediatric subspecialty interests), emergency medicine consultants and registrars, orthopaedic surgical consultants and registrars and reporting radiographers who review paediatric limb radiographs as part of their clinical practice |
| Key exclusion criteria | Any doctor, nurse, radiographer who does not routinely review paediatric radiographs in their clinical practice or for their job. |
| Date of first enrolment | 01/12/2023 |
| Date of final enrolment | 31/03/2024 |
Locations
Countries of recruitment
- England
- Northern Ireland
- Scotland
- United Kingdom
- Wales
Study participating centres
London
WC1N 3JH
United Kingdom
London
SW17 0QT
United Kingdom
London
SE5 9RS
United Kingdom
Sponsor information
Hospital/treatment centre
Great Ormond Street, London
London
WC1N 3JH
England
United Kingdom
| Phone | +44 2079052700 |
|---|---|
| research.governance@gosh.nhs.uk | |
| Website | http://www.gosh.nhs.uk/ |
"ROR" |
https://ror.org/03zydm450 |
Funders
Funder type
Government
Government organisation / National government
- Alternative name(s)
- National Institute for Health Research, NIHR Research, NIHRresearch, NIHR - National Institute for Health Research, NIHR (The National Institute for Health and Care Research), NIHR
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 31/12/2025 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Published as a supplement to the results publication |
| Publication and dissemination plan | Planned publication in a high-impact peer-reviewed journal |
| IPD sharing plan | All data generated or analysed during this study will be included in the subsequent results publication |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Protocol file | version 1.4 | 26/12/2023 | 28/12/2023 | No | No |
Additional files
Editorial Notes
30/08/2024: The following changes were made:
1. The target number of participants was changed from 30 to 40 and the interventions and plain English summary were updated to reflect that change.
2. The intention to publish date was changed from 31/12/2024 to 31/12/2025.
09/01/2024: The following changes were made to the study record:
1. The recruitment end date was changed from 01/01/2024 to 31/03/2024.
2. The intention to publish date was changed from 30/09/2024 to 31/12/2024.
28/12/2023: Trial's existence confirmed by NHS HRA.
