Evaluation of Primary Science Quality Mark educational accreditation programme
| ISRCTN | ISRCTN50771738 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN50771738 |
| Secondary identifying numbers | 18165 |
- Submission date
- 16/05/2019
- Registration date
- 07/06/2019
- Last edited
- 02/05/2020
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English summary of protocol
Background and study aims
RAND Europe was commissioned by the Education Endowment Foundation to conduct a randomised controlled trial (RCT) evaluating the impact of Primary Science Quality Mark (PSQM). The study aims to understand the impact of the intervention on pupils and teachers and will also examine how the intervention is implemented. PSQM is an accreditation programme aimed at improving science teaching and raising the profile of science in primary schools. PSQM was initiated in 2008 at the University of Hertfordshire in response to the low profile of science in primary schools and the low uptake of science-specific continuing professional development for teachers.
PSQM unfolds over one academic year. The intervention involves the equivalent of two days of training for teachers/science subject leaders. The training informs the creation and implementation of an action plan to improve a school's science provision. At the end of the year, the school submits a portfolio of evidence demonstrating that they have met the criteria covering leadership in science, teaching, learning and enrichment. If successful, the school will receive a three-year PSQM accreditation, after which the school can reapply for reaccreditation. Qualitative research suggests that PSQM accreditation has a positive impact on the profile of science in school, the monitoring and development of science teaching, teacher behaviours, and children’s attitudes and engagement.
Who can participate?
Teachers and pupils at primary schools in England can participate.
What does the study involve?
The evaluation will be conducted through a two-arm Randomised Controlled Trial, with 140 primary schools randomly allocated to PSQM or business-as-usual. Schools will be recruited from a number of geographical areas in England. In each school, one science subject leader and one Year 5 teacher will participate in the training. The knowledge of and attitudes towards science of the Year 5 pupils in the PSQM schools will be compared to those in the control group at the end of the PSQM implementation year (2019-2020). The following cohort of Year 5 pupils will have an assessment at the end of next year (2020-2021) to assess longer-term intervention effects. Teacher practices and attitudes in relation to science and science teaching in the intervention group will be compared to that in the control group.
What are the possible benefits and risks of participating?
Benefits for teachers include an opportunity to improve their teaching skills and in turn, for pupils to improve their knowledge and have a more engaging experience in science classes. We do not anticipate any risks for participation.
Where is the study run from?
The University of Hertfordshire, UK
When is the study starting and how long is it expected to run for?
February 2019 to June 2023
Who is funding the study?
Education Endowment Foundation, UK
Who is the main contact?
1. Mrs Elena Rosa Brown (public),
erbrown@rand.org
2. Ms Miriam Broeks (public),
mbroeks@randeurope.org
Contact information
Public
RAND Europe
Westbrook Centre, Milton Road
Cambridge
CB4 1YG
United Kingdom
| Phone | +44 1223 353 329 Extension: 2575 |
|---|---|
| erbrown@rand.org |
Scientific
RAND Europe
Westbrook Centre, Milton Road
Cambridge
CB4 1 YG
United Kingdom
 ORCID ID |
0000-0001-5549-6451 |
|---|---|
| Phone | +44 1223 353 329 Extension: 2556 |
| aharshfi@rand.org |
Scientific
RAND Europe
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
| mbroeks@randeurope.org |
Study information
| Study design | Two-arm cluster randomised controlled trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Other |
| Participant information sheet | Not available in web format, please use the contact details to request a participant information sheet |
| Scientific title | The efficacy of Primary Science Quality Mark programme: a two-arm cluster-randomised controlled trial |
| Study acronym | PSQM |
| Study objectives | Hypothesis 1: Year 5 pupils in randomly allocated primary schools participating in PSQM (intervention schools) will have higher levels of science attainment than the pupils in the comparison schools one year following the end of PSQM implementation, 2020/21 (Summer 2021; primary outcome). Hypothesis 2: Year 5 pupils in primary schools participating in PSQM (intervention schools) will have higher levels of science attainment than the pupils in the comparison schools at the end of the school year when the intervention takes place, 2019/20 (Summer 2020; secondary outcome). Hypothesis 3: Year 5 pupils in primary schools participating in PSQM (intervention schools) will be report higher levels of enjoying science than the pupils in the comparison schools in 2019/20 and 2020/21 (Summer 2020 and 2021; secondary outcomes). |
| Ethics approval(s) | Approved 30/05/2019, RAND Corporation's Human Subjects Protection Committee (HSPC), (no postal address; berry@rand.org; +1 310-393-0411 x7779; Tebow@rand.org); ref: 2018-0843; FWA00003425 |
| Health condition(s) or problem(s) studied | Science attainment in pupils, pupil science knowledge in study year 1, pupil attitudes to science and science teaching |
| Intervention | PSQM - Primary Science Quality Mark PSQM is a developmental accreditation programme aiming to improve science education in primary schools through providing teachers and school science leaders with a framework for self-assessment, reflection and development as well as relevant training. PSQM is delivered within hubs of schools (with 4-6 schools in a hub), supported by an experienced hub leader. Hub leaders have backgrounds such as Local Authority advisers, consultants, university lecturers and teachers who have achieved PSQM awards in the past. Schools can apply for one of three awards – PSQM, PSQM Gilt and PSQM Outreach. PSQM award is for “schools which demonstrate how effective science leadership is beginning to have an impact on science teaching and learning across the school”, whereas PSQM Gilt award requires demonstrating a “sustained impact”, and PSQM Outreach is for schools that meet Gilt criteria and also impact science leadership and teaching in other schools. Over the course of one academic year, PSQM involves the following activities: • Staff training completed over two full days or four half-days (topics: introduction to PSQM, creating and executing an action plan, and writing a reflective submission with evidence on the completed work). • The subject leader works with colleagues across the school to audit existing provision in science and agree to an appropriate level of award to work towards. • The subject leader creates an action plan to develop all aspects of science teaching and works with colleagues to implement it. • The subject leader is supported by the hub leader, with ongoing online mentoring provided via the PSQM Virtual Learning Environment (VLE), and access to resources such as the PSQM handbook and information on relevant Continuing Professional Development (CPD) offers. • The subject leader collates and submits the evidence for the relevant PSQM award, which is assessed by a hub leader from another hub. • Hub leaders use PSQM evaluative criteria to consider a school for the relevant award. Awards are made to schools following an analysis of a series of documents that detail how the activities implemented during the intervention year have impacted on the science teaching and learning across the school and how the school meets the PSQM criteria. There are 13 PSQM criteria covering (1) primary school science leadership, (2) teaching and learning, and (3) science enrichment. Rather than the award itself being central, the focus of the programme is on the process of self-assessment, reflection and development. All schools must meet the same criteria, ensure that the subject leaders (and another member of staff if possible) attend standardised training and submit common core documents. However, each school’s action plan, implementation and final submission is relevant to its own context. In the current trial, PSQM will be delivered in up to 140 primary schools, with approximately half of the schools within each hub allocated to PSQM. In the current evaluation, the programme will focus on the school’s science subject leader and Year 5 teacher from each school (and a Key Stage (KS) 1 teacher, if the Y5 teacher is the subject leader). Randomisation will be conducted in Stata by the Evaluation Team’s Primary Investigator. Hub will be the main stratifying variable, with around 10-12 hubs expected to be recruited. In addition, we plan to stratify on school size (single-entry versus multiple-entry school), as reported by the school. The trial allocation will be recorded and communicated to the implementation team and the EEF in an Excel file that has been password protected to prevent editing. Initial outcome analyses will be conducted blind to allocation. Baseline equivalence will be examined based on the initial randomisation. Well-conducted randomisation will, in expectation, yield groups that are equivalent at baseline. Because schools are randomly allocated to the control and intervention conditions, any imbalance at baseline will have occurred by chance. To assess imbalance at baseline, we will compare groups at school and pupil levels, by means of cross-tabulations and histograms that assess the distribution of each characteristic within the control and intervention groups. Process of data collection: Outcome data will be collected within schools post-intervention via pen and paper. Data collection for implementation and process evaluation of the study in detail will commence as follows: PRE-INTERVENTION Non-enrolment and drop-outs While it is not possible to identify the characteristics of the schools that do not participate in the trial, the PSQM team will monitor the contact with schools and the general sign-up rate in order to get a sense of non-enrolment. If a school drops out from the programme, the Hub will be responsible for notifying the PSQM team with accompanying reasons for why the drop-out occurred. DURING INTERVENTION PHASE Motivations for joining the study The baseline headteacher online surveys (June – July 2019) will aim to examine the motivations for joining the trial and the current practice related to science teaching. Completion of intervention activities by intervention schools We will assess closer to the time the merits of observing PSQM teacher training sessions. Where we do observe, we will attempt to attend the second of the two training sessions that involve the case study schools. PSQM will make the training materials available to RAND prior to observations so that there is a clear indicate of what should be observed. Attendance of trainings by teachers is mandatory and will be tracked by the PSQM team through attendance logs. Based on information provided by the PSQM team, we will also report on the number of schools that are successful in gaining the award they aimed for, and numbers of where a submission was sent for a second review, a school was asked to submit additional evidence to get the award, and had a deferral/extension. POST-INTERVENTION Exploring programme implementation and changes in practice Online surveys One of the aims of the post-intervention headteacher surveys (in June – July 2020 and again in June – July 2021) will be to capture any potential changes in science practice (in both intervention and control schools). Any such identified changes will be highlighted in the evaluation report. Hub leader surveys will focus on their experiences of working with schools and any barriers and facilitators to implementation. We expect that it will take no more than 10-15 minutes to complete the online surveys. The text for the survey will be prepared and compiled by RAND Europe and disseminated to schools by PSQM. Paper surveys To facilitate higher response rates, paper surveys will be distributed to two staff members per school in June-July 2020 and again in 2021. The focus will be on usual practice, attitudes, perceptions and science-related activities in the classroom. The majority of the items will be the same in both control and intervention schools, examining practices and attitudes around teaching of science and science-specific CPD activities. The questions will be based on the Wellcome State of the Nation surveys for teachers and subject leaders. In addition, in the intervention arm, intervention-specific questions will be included. Interviews in case study schools Four schools among those assigned to the Intervention (PSQM), representing a diverse set of characteristics, will be approached for in-depth case studies. These will involve interviews with school teaching staff and headteachers, as well as parents, and – where possible – school governors, as well as visits to PSQM training sessions attended by the year 5 teacher and science subject leader from those schools. These interviews will allow the evaluation team to gain a more in-depth understanding of what PSQM involves in practice for participating teachers, subject leaders, and schools, and explore the mechanisms of change as a result of the intervention. Examining continuity of Year 5 teachers We would expect programme effects to be strongest for Year 5 pupils in those schools where Year 5 teachers trained in PSQM continue teaching the next Year 5 cohort (Cohort B). To examine this, we will descriptively compare programme outcomes for those schools where the same teachers are working in Year 5 in 2020/21. The information on continuity will be based on self-report by teacher in Year 2 as part of their surveys. Compliance measure To enable a non-compliance analysis, compliance will be defined at the school level, based on completion of programme activities, as recorded by the PSQM team. Data collection plan: Data collection mode Participants When Online Survey Headteacher/SLT Year 1, autumn term Online Survey Headteacher/SLT Year 1, summer term Online Survey Headteacher/SLT Year 2, summer term Online Survey Hub leader Year 1, summer term (first 2 weeks of July) Paper Survey Y5 Teachers (teacher selected for PSQM) Year 1, summer term (first 2 weeks of July) Paper Survey Science subject leaders Year 1, summer term (first 2 weeks of July) Paper Survey Teacher and subject leaders Year 2, summer term (first 2 weeks of July) Feedback from training events Teachers Year 1 Data on previous rounds of PSQM Schools Set-up year Monitoring data from PSQM Schools Year 1 Interviews Teacher, headteacher, parent, governor (4 people per school) + hub leader Year 1 and 2, summer term (first 2 weeks of July) Interviews Hub leader Year 1, summer term |
| Intervention type | Behavioural |
| Primary outcome measure | Science attainment will be measured as in Hanley et al., (2015) (potentially modified to ensure fit with the National Curriculum), science assessment with scores ranging from 0 to 41 points |
| Secondary outcome measures | Pupil science knowledge in study year 1, pupil attitudes to science and science teaching will be measured using the equivalent scales from the "Trends in International Mathematics and Science Study (TIMMS)" |
| Overall study start date | 01/08/2018 |
| Completion date | 30/09/2022 |
Eligibility
| Participant type(s) | Other |
|---|---|
| Age group | All |
| Sex | Both |
| Target number of participants | Primary school science teachers in ~140 schools; ~5,600 - 7,000 pupils |
| Key inclusion criteria | Teachers: Primary school science teachers Pupils: Pupils in the intervention group will receive schooling from teachers who have been trained, all pupils will be subject to outcome testing |
| Key exclusion criteria | 1. School cannot have received a PSQM award in the last 3 years (i.e., a school has not participated in PSQM in 2017, 2018 or 2019) 2. School must be a state primary, junior or all-through school 3. Schools with mixed Year 5/6 or another combination are eligible if they have Year 5 pupils taught separately by one teacher for the core subjects, including science 4. Infant or first schools, private schools, special schools, Pupil Referral Units (PRUs) or middle schools are not eligible |
| Date of first enrolment | 11/02/2019 |
| Date of final enrolment | 31/07/2019 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centre
Hatfield
AL10 9AB
United Kingdom
Sponsor information
Research organisation
Westbrook Centre
Milton Road
Cambridge
CB4 1 YG
United Kingdom
| Phone | +44 1223 353 329 |
|---|---|
| psqm@rand.org | |
| Website | https://www.rand.org/randeurope.html |
"ROR" |
https://ror.org/037pk1914 |
Funders
Funder type
Charity
Private sector organisation / Trusts, charities, foundations (both public and private)
- Alternative name(s)
- EducEndowFoundn, Education Endowment Foundation | London, EEF
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 30/09/2022 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Data sharing statement to be made available at a later date |
| Publication and dissemination plan | May 2019 - Protocol September 2020 - Statistical Analysis Plan September 2022 - Final Report |
| IPD sharing plan | The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository. Aggregated outcome data will be sent to the EEF archive, managed by the Fischer Family Trust (FFT), together with UPN’s. This is to allow for long term follow up investigations in the future. |
Editorial Notes
02/05/2020: The following changes have been made:
1. The recruitment end date has been changed from 31/07/2021 to 31/07/2019.
2. A trial contact has been updated and the plain English summary has been updated accordingly.
04/06/2019: Trial’s existence confirmed by Education Endowment Foundation
