Evaluation of autoSCORE: an artificial intelligence based algorithm for EEG classification versus human experts
| ISRCTN | ISRCTN14307038 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN14307038 |
| Secondary identifying numbers | 24084-01 |
- Submission date
- 18/03/2022
- Registration date
- 25/03/2022
- Last edited
- 05/09/2023
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Nervous System Diseases
Plain English Summary
Background and study aims
Electroencephalography (EEG) measures electric brain activity using electrodes attached to the scalp. This is used to investigate brain disease, most commonly epilepsy, coma, and dementia. The clinical interpretation of EEGs is until now mainly based on expert visual analysis, and there are indications that EEG reviewers are under increasing time pressure. A large, anonymized dataset consisting of EEGs evaluated and annotated by experts with a dedicated software package (SCORE EEG) is used to train an algorithm (autoSCORE) to automatically assess EEGs. autoSCORE is trained to separate normal from abnormal EEGs. When autoSCORE assesses the EEG as abnormal it will further sub-classify abnormalities into four sub-groups, which provide important information for medical decisions on patient management. This is a human expert validation study to validate the algorithm.
Who can participate?
This study involves analysis of recorded electroencephalography data. Direct participation of new patients is not required.
What does the study involve?
Doctors will assess 100 EEGs and the algorithm will assess the same EEGs. A large independent EEG dataset from Oslo University Hospital will be used to compare autoSCORE with the clinical scorings of the experts who evaluated the EEGs. The goal is to prove that the autoSCORE algorithm is non-inferior to human experts. The researchers will compare autoSCORE with a commercially available EEG analysis software package (ENCEVIS).
What are the possible benefits and risks of participating?
As the study involves analysis of the EEG data of patients who were referred to the investigation as part of their diagnostic work-up, there are no additional risks for the patients. EEG is a non-invasive procedure without any risks. The benefits are EEG diagnostics will become available also in underserved areas where EEG experts are not available, and it will assist physicians in reducing their workload in places where the expertise is available.
Where is the study run from?
1. Danish Epilepsy Centre Filadelfia (Denmark)
2. Haukeland University Hospital (Norway)
3. Oslo University Hospital (Norway)
4. Mayo Clinic (USA)
When is the study starting and how long is it expected to run for?
June 2020 to March 2022
Who is funding the study?
Holberg EEG (Norway)
Who is the main contact?
Prof. Sandor Beniczky
sbz@filadelfia.dk
Contact information
Principal Investigator
Visby Allé 5
Dianalund
4293
Denmark
 ORCID ID |
0000-0002-6035-6581 |
|---|---|
| Phone | +45 (0)26981536 |
| sbz@filadelfia.dk |
Study information
| Study design | Cross-sectional validation study |
|---|---|
| Primary study design | Observational |
| Secondary study design | Cross sectional study |
| Study setting(s) | Hospital |
| Study type | Diagnostic |
| Scientific title | Accuracy of EEG classification by autoSCORE algorithm compared with human experts |
| Study acronym | autoSCORE |
| Study hypothesis | The autoSCORE algorithm has an accuracy similar to human experts in distinguishing abnormal from normal electroencephalography (EEG) recordings and classifying the abnormal recordings: focal-epileptiform, generalized-epileptiform, focal-slowing, diffuse-slowing. AutoSCORE is an algorithm developed using artificial intelligence, based on a large SCORE dataset. The validation process is prospective, i.e. after the development of the algorithm, using a fixed algorithm and threshold (cut-off) value. |
| Ethics approval(s) | This study uses anonymized EEG datasets and does not require ethics approval. The study has been reviewed on 07/07/2020 by the institutional review board and the data safety officer at the institution of the Principal Investigator, the Danish Epilepsy Centre, Filadelfia (Kolonivej 1, 4293, Dianalund, Denmark; +45 (0)58264200; pwo@filadelfia.dk), ref: Sagsnr. 0100256 |
| Condition | Patients suspected of epilepsy or other conditions with impaired consciousness or cognition |
| Intervention | EEGs will be automatically assessed by the previously developed autoSCORE algorithm using pre-defined detection thresholds. The algorithm first distinguishes between normal and abnormal recordings. Then, it classifies the abnormal EEGs into four categories: focal-epileptiform, generalized-epileptiform, focal-slowing, diffuse-slowing. The performance of autoSCORE will be compared with the evaluation of the EEGs by a panel of experts on an independent dataset of a balanced sample of 100 randomly selected EEGs and form a large independent dataset from a hospital that did not participate in the development of the algorithm. |
| Intervention type | Other |
| Primary outcome measure | Sensitivity, specificity, accuracy, positive predictive value, negative predictive value of autoSCORE compared with the majority-consensus scoring of the human experts, calculated using a balanced sample of 100 randomly selected EEGs at a single timepoint |
| Secondary outcome measures | Calculated using a balanced sample of 100 randomly selected EEGs at a single timepoint: 1. Inter-test agreement (autoSCORE vs human experts) in the large, independent dataset 2. Performance of autoSCORE at identifying recordings with epileptiform abnormalities (both focal and generalized) compared with the commercially available spike-detector software packages |
| Overall study start date | 01/06/2020 |
| Overall study end date | 18/03/2022 |
Eligibility
| Participant type(s) | Patient |
|---|---|
| Age group | Mixed |
| Sex | Both |
| Target number of participants | 100 |
| Total final enrolment | 100 |
| Participant inclusion criteria | The EEGs to be selected for this study have not been part of the training dataset to develop the autoSCORE. The datasets are distributed between the EEGs arriving from Haukeland University Hospital, Danish Epilepsy Centre Filadelfia and Mayo Clinic. Although there is no scientific reason to consider that ethnicities or geographical origin of the EEG, nor the software used for acquisition influences the results, in order to avoid any such potential bias, the study design has addressed this by using EEGs from different geographies. Age range: 35% under 16 years (pediatric population), 65% over 16 years (adult population). |
| Participant exclusion criteria | 1. Neonatal 2. EEGs reported with rhythmic and periodic patterns in critically ill patients |
| Recruitment start date | 01/06/2021 |
| Recruitment end date | 18/03/2022 |
Locations
Countries of recruitment
- Denmark
- Norway
- United States of America
Study participating centres
Dianalund
4293
Denmark
Bergen
5009
Norway
Jacksonville
FL 32224
United States of America
Sponsor information
Industry
Fjøsangerveien 70 A
Bergen
5068
Norway
| Phone | +47 (0)926 44 261 |
|---|---|
| shelley.cam@holbergeeg.com | |
| Website | https://www.holbergeeg.com/ |
Funders
Funder type
Industry
No information available
Results and Publications
| Intention to publish date | 09/09/2023 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Available on request |
| Publication and dissemination plan | Planned publication in a high-impact peer-reviewed journal. |
| IPD sharing plan | Additional information, including raw data, is available on request, pending IRB approval for the intended use. Please contact Prof. Sandor Beniczky (sbz@filadelfia.dk). Type of data: anonymised EEG, diagnostic gold standard; demographics (age, gender), output of the algorithm. Data will be available upon request for 10 years from the publication for scientific non-commercial use. As the dataset is de-identified, there is no need for consent from the participants. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Protocol file | version 4 | 28/02/2022 | 24/03/2022 | No | No |
| Results article | 01/08/2023 | 05/09/2023 | Yes | No |
Additional files
Editorial Notes
05/09/2023: Publication reference added.
06/04/2023: The intention to publish date was changed from 01/06/2023 to 09/09/2023.
04/01/2023: The intention to publish date was changed from 01/01/2023 to 01/06/2023.
02/09/2022: The intention to publish date was changed from 01/09/2022 to 01/01/2023.
24/03/2022: Trial's existence confirmed by the Danish Epilepsy Centre.
