NHS App evaluation
| ISRCTN | ISRCTN72729780 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN72729780 |
| IRAS number | 291627 |
| Secondary identifying numbers | CPMS 48568, NIHR128285, IRAS 291627 |
- Submission date
- 24/12/2021
- Registration date
- 03/02/2022
- Last edited
- 06/02/2025
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English summary of protocol
Background and study aims
The National Health Service (NHS) in England has introduced a new (smartphone or tablet) app for patients, called ‘the NHS App’ (https://www.nhs.uk/using-the-nhs/nhs-services/the-nhs-app/). This app has been available in England from July 2019. People are able to use the app to see their medical records, prove their Covid-19 vaccination status, book appointments, order repeat prescriptions, add their name to the organ donation register, have online consultations and undertake other tasks related to their health and care.
This project was developed in 2019 to evaluate the newly developed NHS App. It is important to note that this is not the Covid-19 App used for contact tracing (which has received much publicity of late) or a commercial app which has been branded as an NHS App.
This research (https://fundingawards.nihr.ac.uk/award/NIHR128285) will look at what different groups think about the NHS App, how they use it, whether they find the app helpful, and whether it changes how people use the NHS. We are working with patients, carers, clinical and non-clinical staff, commissioners and policy-makers to explore the use, roll out of the NHS App.
Who can participate?
People who do and do not use the app, staff working in the NHS (e.g. doctors, nurses, receptionists, practice managers) and those working in healthcare planning or technology development for the NHS.
What does the study involve?
The research will primarily use interviews and discussion groups (either virtually or face-to-face, if possible) with people who do and do not use the app, with staff working in the NHS (e.g. doctors, nurses, receptionists, practice managers) and with those working in healthcare planning or technology development for the NHS. These interviews and discussion groups will try to understand what people think about the app, and whether they find it helpful. We will also sit (either virtually or face-to-face, if possible) with people and watch how they use the app in practice.
We will work closely with NHSX and NHS Digital teams responsible for the development and implementation of the app so that we can learn from their experience, and we can feedback our findings to help improve the app.
We will also communicate the findings by going to events where we can talk about the work, preparing presentations about the study and findings, and also by writing and blogging about it on our department and the NHS App’s web pages. In addition, we plan to write academic papers reporting on the outcomes of this study as well as providing lay summaries to participants of the study.
We work closely with our patient and public involvement (PPI) group to design and undertake the study. For example, we work with the group to refine what questions to ask in the interviews and discussion groups, and how to ask these questions when we talk to patients and NHS Staff. We will also ask our PPI members to help with understanding what we have found out from the interviews and discussions groups and how to share these findings with others. There will also be opportunities for writing up the work in different formats. We will work with our PPI group to help communicate our findings to NHS staff, patients, policymakers and the public.
What are the possible benefits and risks of participating?
None
Where is the study run from?
The project is run jointly by Imperial College and Oxford University (UK)
When is the study starting and how long is it expected to run for?
October 2020 to June 2023
Who is funding the study?
National Institute for Health Research (NIHR) (UK).
Who is the main contact?
Prof John Powell, john.powell@phc.ox.ac.uk
Contact information
Scientific, Principal Investigator
Nuffield Department of Primary Care Health Sciences
University of Oxford
Radcliffe Primary Care Building
Radcliffe Observatory Quarter
Woodstock Road
Oxford
OX2 6GG
United Kingdom
 ORCID ID |
0000-0002-1456-4857 |
|---|---|
| john.powell@phc.ox.ac.uk |
Principal Investigator
Imperial College London
South Kensington Campus
London
SW7 2AZ
United Kingdom
| felix.greaves08@imperial.ac.uk |
Study information
| Study design | Observational qualitative study |
|---|---|
| Primary study design | Observational |
| Secondary study design | Cross sectional study |
| Study setting(s) | Internet/virtual |
| Study type | Other |
| Participant information sheet | 40869 PIS NHS Staff v1.0 05Feb2021.pdf |
| Scientific title | Qualitative evaluation of the national rollout of the NHS App in England |
| Study objectives | The NHS in England have introduced a new smartphone app for patients, called ‘the NHS App’. This app has been available to everyone in England from July 2019. People are able to use the App to see their medical records, book appointments, order repeat prescriptions, set their organ donation preferences register, have online consultations and undertake other tasks related to their health and care. This research will look at what people think about the NHS App, whether they use it to do the things it was designed for, and whether it changes how people use the NHS. |
| Ethics approval(s) | Approved 21/12/2021, West of Scotland Research Ethics Committee 3 (West of Scotland Research Ethics Service, Ward 11, Dykebar Hospital, Grahamston Road, Paisley, PA2 7DE, UK; +44 (0)141 3140213; WoSREC3@ggc.scot.nhs.uk), ref: 21/WS/0031 |
| Health condition(s) or problem(s) studied | NHS App evaluation |
| Intervention | Given current COVID-19 related circumstances, our primary methods will initially include remote (on the phone or online) semi-structured interviews and focus groups, as well as the participant demonstrating their use of the technology (think aloud interviews) as part of video interviews or virtual platforms, where feasible. At later stages and where Covid-19 restrictions allow, we will also carry out face-to-face interviews and focus groups, as well as ethnographic observation (of patients and organisational settings). SAMPLING STRATEGY We will follow a staged approach in the recruitment of GP practices to ensure we cover a wide range of characteristics and patterns of technology engagement. We have been liaising with the delivery team at NHS Digital to develop our recruitment strategy drawing on their learning so far and have identified potential sites for recruitment. We will identify participants via our professional networks and via recruited surgeries, and potentially ‘snowballing’ from our initial contacts (especially staff participants, commissioners and NHS App development and delivery teams) to others in their organisations. We will cover a number of criteria across GP practices including location (urban, rural, coastal), ethnic diversity, socioeconomic status, digital maturity and practice size. We believe that by carefully and purposive selecting, we should be able to identify practices that capture variation across these criteria. DATA COLLECTION: NHS patients, carers and members of the public We will carry out semi-structured, qualitative interviews with patients, carers and members of the public to explore their views and experiences. Interviews will be carried out virtually using an appropriate remote platform (MS Teams) where Covid-19 restrictions do not allow face-to-face contact or where participants prefer to be interviewed online. A subset of participants will be invited to a follow-up interview 12 months later to reflect on longitudinal changes. Interviews will follow a flexible topic guide and will last 45-60mins. Alongside qualitative interviews, we will carry out ethnographic observations to better understand the wider context of participants’ lives; how they cope with own healthcare needs or their caring responsibilities, and wider engagement with technology (where relevant). This means we will spend time engaging with participants in their homes (if feasible), or we will ensure to have more time when engaging people on interviews through video, so that we can ask them to tell us more about their situation, given we will not be able to observe their surroundings as we would do in a face-to-face interview. We appreciate the potential ethical implications of this so we will explain to participants that if we are using video interviews, to consider if members of their household (or other setting) may be captured on video as well, and we will acquire consent from said household members who may otherwise also be involved in the interview. With a small sample of patients and/or carers and/or members of the public (up to 5), we are planning to develop video diaries (e.g. using the camera on their smartphones or bespoke technology provided by the project) to record how they manage their everyday activities, how they engage with the app, or how they are accessing healthcare. As part of our interactions with participants, we will employ interactive methods, such as think aloud protocols, where we will observe patients booking their appointments on the app, ordering repeat prescriptions, checking their symptoms or looking at their GP records, and will ask them to articulate how they accomplish each of these tasks and any particular challenges they encounter. These tasks will be undertaken over telephone, video call or video diary if we are not able to observe face-to-face. We will capture these interactions in field notes, photos, screenshots, and videos, with participant permission. While interviews and observations will explore individual patient, carer and members of the public experiences in depth, we will additionally use focus groups to allow participants to collectively debate their views and perceptions about the NHS App. Half of the focus groups will involve technology users and half will involve non-users. Discussions will take place on virtual discussion platforms (Microsoft Teams), or face-to-face at GP practices or voluntary/community organisations (at later stages if this becomes feasible). The discussion will last 60 minutes using a topic guide to maintain focus on predetermined questions, while allowing flexibility. NHS Staff We will gather feedback from NHS staff: health professionals (GPs, trainees, practice nurses, healthcare assistants), practice and IT managers, and reception staff. Interviews will last 30-45mins and will be carried out either virtually or face-to-face. An indicative topic guide will be followed, tailored to different types of NHS staff. We will also carry out face-to-face (when feasible) ethnographic observation on back-end operational and technical processes required to integrate the app in the service, to promote the app with patients and to manage competing priorities. Observations will be planned so that they cover every day routine work but also relevant internal activities (e.g. appointment planning sessions, operational meetings, troubleshooting with NHS Digital etc.). Acknowledging that NHS staff are under pressure, especially in the Covid-19 context, we aim to align our research with existing activities to minimise any extra burden on their work. Stakeholders: Local Commissioners We will use qualitative methods to draw on this learning and to consolidate our findings from research with patients, carers, the public, and NHS staff in the 5 main case studies. We will carry out semi-structured interviews with commissioners. Interviews will last 45-60mins and will be carried out either face-to-face (if feasible) or by telephone/video. Commissioning insights will allow us to understand the wider impact on service planning and access. Given the move towards Primary Care Networks, we will closely follow developments and adjust our recruitment plans accordingly in this changing environment. NHS App development and delivery teams We will draw on specific qualitative data collection methods, including semi-structured interviews with the development and delivery teams at NHS Digital, NHS England, and other organisations as relevant. These will last 45-60mins and will be carried out either face-to-face (if feasible) or by telephone/video. SUBSEQUENT VISITS Most participants will have one study visit/engagement (an interview or focus group), while a sub-set of patient/carer/public participants will receive between two and three engagements if they take part in a semi-structured interview, a think aloud interview or observation (if the additional think aloud interview/observation are part of a separate interview or on a different day/time) as well as a follow up semi-structured interview approximately 12 months on from the original semi-structured interview. NHS staff will be involved in between one and four study visits/engagements which could include an original interview, a follow-up interview 12 months on and may be involved in up to two observations. Stakeholders (Commissioners or Delivery and Development team) participants will have one study visit/engagement in the form of a semi-structured interview. |
| Intervention type | Other |
| Primary outcome measure | The following research questions will be measured using qualitative interviews and focus groups at baseline and 12 month follow up: 1. How and why do patients, carers and members of the public use (or not use) the NHS App and what are their experiences with technology-enabled access to primary care? 2. What are the experiences of healthcare staff with the NHS App and its integration in the service? 3. What background work and ongoing adaptations are needed to accommodate the impact of technology-enabled access to primary care? 4. What is the role of local commissioning groups and NHS App development and delivery teams in successfully mainstreaming technology-enabled options for accessing primary care and other services? 5. What are the perceived implications of the NHS App for access, efficiency, safety and overall experience in primary care over time? 6. What transferable learning can we draw from the example of the NHS App to inform the implementation, national roll-out and routine use of health technologies? |
| Secondary outcome measures | There are no secondary outcome measures |
| Overall study start date | 01/10/2020 |
| Completion date | 30/06/2023 |
Eligibility
| Participant type(s) | Patient |
|---|---|
| Age group | Adult |
| Lower age limit | 18 Years |
| Sex | Both |
| Target number of participants | Planned Sample Size: 79; UK Sample Size: 79 |
| Total final enrolment | 88 |
| Key inclusion criteria | All participants 1. Participants must be 18 years or older 2. Willing and able to give informed consent for participation NHS Staff 1. NHS professionals from different areas of NHS General Practice such as doctors, receptionists, and practice managers Stakeholders 1. Local Commissioners: Professionals involved in the commissioning of NHS services 2. NHS App development and delivery teams: Professionals involved in the planning, development, roll out, policy making or communication of the NHS App |
| Key exclusion criteria | Does not meet inclusion criteria |
| Date of first enrolment | 19/03/2021 |
| Date of final enrolment | 30/04/2023 |
Locations
Countries of recruitment
- United Kingdom
Study participating centre
-
-
United Kingdom
Sponsor information
University/education
Joint Research Office 1st floor
Boundary Brook House Churchill Drive
Headington
Oxford
OX1 2JD
England
United Kingdom
| ctrg@admin.ox.ac.uk | |
| Website | http://www.ox.ac.uk/ |
"ROR" |
https://ror.org/052gg0110 |
Funders
Funder type
Government
No information available
Government organisation / National government
- Alternative name(s)
- National Institute for Health Research, NIHR Research, NIHRresearch, NIHR - National Institute for Health Research, NIHR (The National Institute for Health and Care Research), NIHR
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 30/09/2023 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Not expected to be made available |
| Publication and dissemination plan | Anticipated outputs include: Regular briefing on emerging findings to NHSX A final evaluation report for the NHS App development and delivery teams and policymakers Academic papers and the final academic report/monograph for the NIHR Journals Library. Conference outputs - at events for the HSR, digital health and primary care communities Public information material for GP practice patient groups, social media and the NHS website |
| IPD sharing plan | The datasets generated during and/or analysed during the current study are not expected to be made available due to confidentiality. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Participant information sheet | NHS staff version 1.0 |
05/02/2021 | 04/01/2022 | No | Yes |
| Participant information sheet | Patients and carers version 1.0 |
05/02/2021 | 04/01/2022 | No | Yes |
| Participant information sheet | Stakeholders version 1.0 |
05/02/2021 | 04/01/2022 | No | Yes |
| HRA research summary | 28/06/2023 | No | No | ||
| Results article | 21/01/2025 | 27/01/2025 | Yes | No | |
| Plain English results | 06/02/2025 | No | Yes |
Additional files
- 40869 PIS NHS Staff v1.0 05Feb2021.pdf
- NHS staff
- 40869 PIS patients and carers v1.0 05Feb2021.pdf
- Patients and carers
- 40869 PIS stakeholders v1.0 05Feb2021.pdf
- Stakeholders
- ISRCTN72729780 Plain English results.pdf
Editorial Notes
06/02/2025: The plain English results were uploaded as an additional file.
27/01/2025: Publication reference added.
02/10/2023: The scientific contact was replaced.
15/05/2023: Total final enrolment added. The recruitment end date was changed from 31/07/2022 to 30/04/2023.
15/05/2023: The following changes were made to the trial record:
1. The overall end date was changed from 30/11/2022 to 30/06/2023.
2. The plain English summary was updated to reflect these changes.
18/11/2022: The following changes were made to the trial record:
1. The overall end date was changed from 30/09/2022 to 30/11/2022.
2. The plain English summary was updated to reflect these changes.
06/05/2022: The recruitment end date has been changed from 30/05/2022 to 31/07/2022.
24/12/2021: Trial's existence confirmed by the National Institute for Health Research (NIHR) (UK).
