Piloting automated testing of different reminder messages to increase attendance in a Botswana vision screening programme
| ISRCTN | ISRCTN96528723 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN96528723 |
| Secondary identifying numbers | 26480 |
- Submission date
- 05/01/2022
- Registration date
- 06/01/2022
- Last edited
- 16/05/2025
- Recruitment status
- No longer recruiting
- Overall study status
- Suspended
- Condition category
- Eye Diseases
Plain English summary of protocol
Background and study aims
This study aims to test whether an algorithm (lines of code) can be used to automatically test which form of mobile phone-based reminder message is most effective at encouraging parents and guardians to take their children to receive spectacles. The study will compare the algorithm’s performance and costs against that of human statisticians performing the same tasks. The underlying aim is to try and automate complex statistical testing so that global health programmes don’t have to spend so much time and money on optimising their projects.
Who can participate?
The main aim of the study is to test the algorithm’s performance as it runs a small trial. The participants in this underlying trial will be parents/guardians of children identified as needing referral for refractive services in Botswanan vision screening programmes run by ‘Peek Vision’ in 2022.
What does the study involve?
Parents/guardians will be randomly allocated to receive three standard reminder SMS messages (the control group); the same three standard messages with a new pre-recorded voice reminder (intervention arm 1); three new SMS messages with the new voice reminder (intervention arm 2); or just the three new SMS messages (intervention group 3).
What are the possible benefits and risks of participating?
There are no direct benefits to participants. The negligible risks include experiencing distress at receiving the reminder messages.
Where is the study run from?
The London School of Hygiene and Tropical Medicine (UK) and the University of Botswana (Botswana)
When is the study starting and how long is it expected to run for?
July 2021 to April 2024
Who is funding the study?
1. Wellcome Trust (UK)
2. National Institute for Health Research (UK)
Who is the main contact?
Dr Luke Allen
luke.allen@lshtm.ac.uk
Contact information
Principal Investigator
London School of Hygiene & Tropical Medicine
London
WC1E 7HT
United Kingdom
 ORCID ID |
0000-0003-2750-3575 |
|---|---|
| Phone | +44 (0)20 7636 8636 |
| luke.allen@lshtm.ac.uk |
Study information
| Study design | Health economics performance evaluation with a nested randomized controlled trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Randomised controlled trial |
| Study setting(s) | Community |
| Study type | Other |
| Participant information sheet | No participant information sheet available |
| Scientific title | Semi-automated Allocation For Equitable Research: automated adaptive allocation and hypothesis testing to increase attendance in a Botswana vision screening programme |
| Study acronym | SAFER |
| Study objectives | Automated interim review, randomisation, allocation, and hypothesis testing will produce the same results as human statisticians performing the same tasks. |
| Ethics approval(s) | 1. Approved 22/12/2021, London School of Hygiene & Tropical Medicine Ethics Committee (Room LG36, Keppel Street, London WC1E 7HT, UK; +44 (0)20 7927 2221; ethics@lshtm.ac.uk), ref: 26480 2. Approval pending, Ministry of Health in Botswana |
| Health condition(s) or problem(s) studied | Attendance at ophthalmology clinic among schoolchildren identified as requiring correction of visual impairment during a vision screening programme |
| Intervention | This is an evaluation of an algorithm's performance. The researchers have developed lines of code that can run a pilot embedded, pragmatic, four-arm, adaptive randomised controlled trial (RCT) whereby parents of children identified as requiring follow-up will be sent either the standard three SMS reminder messages (control), or one of three (intervention) alternative reminder messages. The primary aim is to test the ability of the algorithm to autonomously interpret outcome data and adjust the allocation ratio in line with a pre-specified Bayesian equation; and then to identify when the stopping criteria have been met (one arm is found to be best with >95% probability, or there is a >95% probability that the difference between the arms is <1%). Human statisticians will interrogate the data and the 'decisions' made by the algorithm. Differences between the algorithm and human statisticians will be reported. The researchers will also quantify the hours spent by each team member in setting up and running the algorithm. Primary intervention: the SAFER algorithm: lines of code embedded in a screening programme that can randomise and allocate participants to receive one of four different reminder messages, and then perform interim review, adaptive allocation, and mean difference testing to identify the best performing arm. Primary control: Human statisticians performing the same tasks using the same data and underlying equations. Nested RCT control: Three SMS reminder messages will be sent to the phone number provided by the child's registered guardian. Nested RCT Intervention 1: Three SMS reminder messages with different wording will be sent. The wording of these alternate messages will be developed by a Botswana technical working group. Nested RCT Intervention 2: The control SMS reminders will be sent. In addition, a voice message will be sent once. The wording of the voice message will be developed by a Botswana technical working group. Nested RCT Intervention 3: The new SMS messages and the voice message. Participants will initially be randomly allocated into four arms using computer-generated blocks of 12. As intervention delivery (sending SMS messages) is fully automated, there is no need for any of the investigators to see participant allocation status. Once the first participants attend refractive services the algorithm will begin adjusting the allocation ratio to favour the best-performing arms. There is no need for the investigators to see allocation status at this stage either. The data safety monitoring committee will be fully unmasked to allocation status and all outcome data and will have the power to stop the trial or suspend any arm. |
| Intervention type | Other |
| Primary outcome measure | 1. Differences in the timing and magnitude of allocation adjustments, dropping of arms, and closure of the trial - comparing algorithm performance against human statisticians who will perform real-time review using the same data and underlying equations and criteria: 1.1. Daily assessment of the number of participants allocated to each arm by the algorithm in comparison to the number of participants allocated to each arm by human statisticians using the same outcome data 1.2. Date at which the algorithm concluded that a stopping rule has been satisfied (this includes dropping arms and stopping the trial) vs date at which human statisticians concluded that a stopping rule has been satisfied; report difference in days |
| Secondary outcome measures | 1. Direct costs of setting up and running the software, compared to the estimated direct costs of using human statisticians to perform the same work. Costs will be calculated at the end of the study once the embedded RCT study has closed: 1.1. The researchers will collect information on two main costs components: the staff involved and the IT hardware and software equipment used in the trial. They will register a map of interrelations between all the activities performed by all the staff included in the trial (statisticians, epidemiologists, database developer and clinical trial managers) and the resources consumed measured in working hours. Activities include, for example, team meetings to conceptualise and plan the trial or algorithm design, and testing and monitoring conducted by the statistician. Staff wages and overheads will be obtained from LSHTM and Peek Payroll services. A list of all IT hardware and software equipment used during the clinical trial will also be recorded including workstation towers, laptops, monitors and printers. IT equipment characteristics, acquisition costs and annual depreciation will be collected from Peek and the International Centre for Eye Health. 1.2. All other costs, not directly related to a specific activity, will be included in an overhead cost category estimated at 15% of the cost of all activities performed, including costs connected with infrastructures and the general operation of the organisation, such as depreciation of buildings, water/gas/electricity, maintenances, insurances, supplies and office equipment, communication and connection costs, and costs connected with general services such as administrative and financial management, human resources, training, legal advice, and documentation. 1.3. To estimate the costs associated with future amendments/adaptations, a list of all the technical requirements will be produced by the statisticians and epidemiologists. Based on these requirements the researchers will estimate the number of working hours that are needed to adapt the algorithm and therefore estimate the associated costs. 2. The primary outcome for the underlying trial is attendance at refractive services on the appointed day (yes/no). The researchers will report attendance rates for each arm (number of participants who attended clinic within 3 weeks of being referred / all those referred). |
| Overall study start date | 01/07/2021 |
| Completion date | 01/04/2024 |
Eligibility
| Participant type(s) | Carer |
|---|---|
| Age group | All |
| Sex | Both |
| Target number of participants | This is an adaptive RCT so there is not a pre-specified target number. The researchers have set a ceiling at 3,000 but may be able to stop before this. |
| Key inclusion criteria | Parents/guardians of children identified as needing referral for refractive services in Botswanan vision screening programmes run by ‘Peek Vision’ in 2022 |
| Key exclusion criteria | 1. Participant does not speak English or Setswana 2. Participant does not have access to a mobile phone |
| Date of first enrolment | 15/07/2022 |
| Date of final enrolment | 15/03/2024 |
Locations
Countries of recruitment
- Botswana
Study participating centre
Gaborone
-
Botswana
Sponsor information
University/education
Keppel St
London
WC1E 7HT
England
United Kingdom
| Phone | +44 (0)207 927 2626 |
|---|---|
| rgio@lshtm.ac.uk | |
| Website | http://www.lshtm.ac.uk/ |
"ROR" |
https://ror.org/00a0jsq62 |
Funders
Funder type
Research organisation
Private sector organisation / Trusts, charities, foundations (both public and private)
- Alternative name(s)
- Wellcome, WT
- Location
- United Kingdom
Government organisation / National government
- Alternative name(s)
- National Institute for Health Research, NIHR Research, NIHRresearch, NIHR - National Institute for Health Research, NIHR (The National Institute for Health and Care Research), NIHR
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 01/12/2024 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Available on request |
| Publication and dissemination plan | The researchers will publish their findings in an open access, peer-reviewed article. |
| IPD sharing plan | The datasets generated during and/or analysed during the current study are/will be available upon request from Dr Luke Allen (luke.allen@lshtm.ac.uk). Data type: pseudo-anonymised attendance data by study arm. Available from the end of study (01/09/2022) until 01/09/2032 for any type of analysis. In line with the UK concordat on open research data (2016), anonymised data from this trial will be made available to bona fide research groups (evidenced via CVs and the involvement of a qualified statistician), and in line with the trial’s publicly available data sharing policy, following review and approval from the trial’s data monitoring committee. No reasonable request will be turned down, and the appropriate data will be made available within 1 month of receiving the request. Mechanism: csv, r or STATA file sent via encrypted email Consent has not been obtained from participants. As such, there may be multiple levels of permission required in-country before data can be shared, including national ministry of health approval +- local implementation partner approval. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Protocol file | version 2.0 | 01/09/2021 | 06/01/2022 | No | No |
| Protocol article | 15/08/2022 | 16/08/2022 | Yes | No |
Additional files
Editorial Notes
16/05/2025: The trial was embedded within the ministry of health national screening programme that stopped running. The trial has therefore been indefinitely suspended.
02/03/2023: The intention to publish date was changed from 01/12/2022 to 01/12/2024.
01/03/2023: The following changes have been made:
1. The recruitment end date has been changed from 15/03/2023 to 15/03/2024.
2. The overall trial end date has been changed from 01/04/2023 to 01/04/2024 and the plain English summary updated accordingly.
19/12/2022: The following changes have been made:
1. The recruitment end date has been changed from 15/01/2023 to 15/03/2023.
2. The overall trial end date has been changed from 01/12/2022 to 01/04/2023 and the plain English summary updated accordingly.
16/08/2022: Publication reference added.
05/07/2022: The following changes were made to the trial record:
1. The recruitment start date was changed from 01/04/2022 to 15/07/2022.
2. The recruitment end date was changed from 01/07/2022 to 15/01/2023.
02/03/2022: The following changes have been made:
1. The recruitment start date has been changed from 01/03/2022 to 01/04/2022.
2. The recruitment end date has been changed from 01/06/2022 to 01/07/2022.
3. The overall trial end date has been changed from 01/09/2022 to 01/12/2022 and the plain English summary has been updated accordingly.
06/01/2022: Trial's existence confirmed by the London School of Hygiene & Tropical Medicine Ethics Committee.
