Plain English Summary
Background and study aims
RAND Europe was commissioned by the Education Endowment Foundation to conduct a randomised controlled trial (RCT) evaluating the impact of Primary Science Quality Mark (PSQM). The study aims to understand the impact of the intervention on pupils and teachers and will also examine how the intervention is implemented. PSQM is an accreditation programme aimed at improving science teaching and raising the profile of science in primary schools. PSQM was initiated in 2008 at the University of Hertfordshire in response to the low profile of science in primary schools and the low uptake of science-specific continuing professional development for teachers.
PSQM unfolds over one academic year. The intervention involves the equivalent of two days of training for teachers/science subject leaders. The training informs the creation and implementation of an action plan to improve a school's science provision. At the end of the year, the school submits a portfolio of evidence demonstrating that they have met the criteria covering leadership in science, teaching, learning and enrichment. If successful, the school will receive a three-year PSQM accreditation, after which the school can reapply for reaccreditation. Qualitative research suggests that PSQM accreditation has a positive impact on the profile of science in school, the monitoring and development of science teaching, teacher behaviours, and children’s attitudes and engagement.
Who can participate?
Teachers and pupils at primary schools in England can participate.
What does the study involve?
The evaluation will be conducted through a two-arm Randomised Controlled Trial, with 140 primary schools randomly allocated to PSQM or business-as-usual. Schools will be recruited from a number of geographical areas in England. In each school, one science subject leader and one Year 5 teacher will participate in the training. The knowledge of and attitudes towards science of the Year 5 pupils in the PSQM schools will be compared to those in the control group at the end of the PSQM implementation year (2019-2020). The following cohort of Year 5 pupils will have an assessment at the end of next year (2020-2021) to assess longer-term intervention effects. Teacher practices and attitudes in relation to science and science teaching in the intervention group will be compared to that in the control group.
What are the possible benefits and risks of participating?
Benefits for teachers include an opportunity to improve their teaching skills and in turn, for pupils to improve their knowledge and have a more engaging experience in science classes. We do not anticipate any risks for participation.
Where is the study run from?
The University of Hertfordshire, UK
When is the study starting and how long is it expected to run for?
February 2019 to June 2023
Who is funding the study?
Education Endowment Foundation, UK
Who is the main contact?
1. Mrs Elena Rosa Brown (public),
erbrown@rand.org
2. Ms Amelia Harshfield (scientific),
aharshfi@rand.org
3. Mr Hannes Jarke (scientific),
hjarke@rand.org
Trial website
https://www.rand.org/randeurope/research/projects/primary-science-quality-mark.html
Contact information
Type
Public
Primary contact
Mrs Elena Rosa Brown
ORCID ID
Contact details
RAND Europe
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
+44 1223 353 329 Extension: 2575
erbrown@rand.org
Type
Scientific
Additional contact
Ms Amelia Harshfield
ORCID ID
http://orcid.org/0000-0001-5549-6451
Contact details
RAND Europe
Westbrook Centre
Milton Road
Cambridge
CB4 1 YG
United Kingdom
+44 1223 353 329 Extension: 2556
aharshfi@rand.org
Type
Scientific
Additional contact
Mr Hannes Jarke
ORCID ID
http://orcid.org/0000-0002-6022-6381
Contact details
RAND Europe
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
+44 1223 353 329 Extension: 2597
hjarke@rand.org
Additional identifiers
EudraCT number
Nil known
ClinicalTrials.gov number
Nil known
Protocol/serial number
18165
Study information
Scientific title
The efficacy of Primary Science Quality Mark programme: a two-arm cluster-randomised controlled trial
Acronym
PSQM
Study hypothesis
Hypothesis 1: Year 5 pupils in randomly allocated primary schools participating in PSQM (intervention schools) will have higher levels of science attainment than the pupils in the comparison schools one year following the end of PSQM implementation, 2020/21 (Summer 2021; primary outcome).
Hypothesis 2: Year 5 pupils in primary schools participating in PSQM (intervention schools) will have higher levels of science attainment than the pupils in the comparison schools at the end of the school year when the intervention takes place, 2019/20 (Summer 2020; secondary outcome).
Hypothesis 3: Year 5 pupils in primary schools participating in PSQM (intervention schools) will be report higher levels of enjoying science than the pupils in the comparison schools in 2019/20 and 2020/21 (Summer 2020 and 2021; secondary outcomes).
Ethics approval
Approved 30/05/2019, RAND Corporation's Human Subjects Protection Committee (HSPC), (no postal address; berry@rand.org; +1 310-393-0411 x7779; Tebow@rand.org); ref: 2018-0843; FWA00003425
Study design
Two-arm cluster randomised controlled trial
Primary study design
Interventional
Secondary study design
Cluster randomised trial
Trial setting
Schools
Trial type
Other
Patient information sheet
Not available in web format, please use the contact details to request a participant information sheet
Condition
Science attainment in pupils, pupil science knowledge in study year 1, pupil attitudes to science and science teaching
Intervention
PSQM - Primary Science Quality Mark
PSQM is a developmental accreditation programme aiming to improve science education in primary schools through providing teachers and school science leaders with a framework for self-assessment, reflection and development as well as relevant training.
PSQM is delivered within hubs of schools (with 4-6 schools in a hub), supported by an experienced hub leader. Hub leaders have backgrounds such as Local Authority advisers, consultants, university lecturers and teachers who have achieved PSQM awards in the past. Schools can apply for one of three awards – PSQM, PSQM Gilt and PSQM Outreach. PSQM award is for “schools which demonstrate how effective science leadership is beginning to have an impact on science teaching and learning across the school”, whereas PSQM Gilt award requires demonstrating a “sustained impact”, and PSQM Outreach is for schools that meet Gilt criteria and also impact science leadership and teaching in other schools.
Over the course of one academic year, PSQM involves the following activities:
• Staff training completed over two full days or four half-days (topics: introduction to PSQM, creating and executing an action plan, and writing a reflective submission with evidence on the completed work).
• The subject leader works with colleagues across the school to audit existing provision in science and agree to an appropriate level of award to work towards.
• The subject leader creates an action plan to develop all aspects of science teaching and works with colleagues to implement it.
• The subject leader is supported by the hub leader, with ongoing online mentoring provided via the PSQM Virtual Learning Environment (VLE), and access to resources such as the PSQM handbook and information on relevant Continuing Professional Development (CPD) offers.
• The subject leader collates and submits the evidence for the relevant PSQM award, which is assessed by a hub leader from another hub.
• Hub leaders use PSQM evaluative criteria to consider a school for the relevant award.
Awards are made to schools following an analysis of a series of documents that detail how the activities implemented during the intervention year have impacted on the science teaching and learning across the school and how the school meets the PSQM criteria. There are 13 PSQM criteria covering (1) primary school science leadership, (2) teaching and learning, and (3) science enrichment. Rather than the award itself being central, the focus of the programme is on the process of self-assessment, reflection and development.
All schools must meet the same criteria, ensure that the subject leaders (and another member of staff if possible) attend standardised training and submit common core documents. However, each school’s action plan, implementation and final submission is relevant to its own context.
In the current trial, PSQM will be delivered in up to 140 primary schools, with approximately half of the schools within each hub allocated to PSQM. In the current evaluation, the programme will focus on the school’s science subject leader and Year 5 teacher from each school (and a Key Stage (KS) 1 teacher, if the Y5 teacher is the subject leader).
Randomisation will be conducted in Stata by the Evaluation Team’s Primary Investigator. Hub will be the main stratifying variable, with around 10-12 hubs expected to be recruited. In addition, we plan to stratify on school size (single-entry versus multiple-entry school), as reported by the school. The trial allocation will be recorded and communicated to the implementation team and the EEF in an Excel file that has been password protected to prevent editing. Initial outcome analyses will be conducted blind to allocation.
Baseline equivalence will be examined based on the initial randomisation. Well-conducted randomisation will, in expectation, yield groups that are equivalent at baseline. Because schools are randomly allocated to the control and intervention conditions, any imbalance at baseline will have occurred by chance. To assess imbalance at baseline, we will compare groups at school and pupil levels, by means of cross-tabulations and histograms that assess the distribution of each characteristic within the control and intervention groups.
Process of data collection:
Outcome data will be collected within schools post-intervention via pen and paper. Data collection for implementation and process evaluation of the study in detail will commence as follows:
PRE-INTERVENTION
Non-enrolment and drop-outs
While it is not possible to identify the characteristics of the schools that do not participate in the trial, the PSQM team will monitor the contact with schools and the general sign-up rate in order to get a sense of non-enrolment. If a school drops out from the programme, the Hub will be responsible for notifying the PSQM team with accompanying reasons for why the drop-out occurred.
DURING INTERVENTION PHASE
Motivations for joining the study
The baseline headteacher online surveys (June – July 2019) will aim to examine the motivations for joining the trial and the current practice related to science teaching.
Completion of intervention activities by intervention schools
We will assess closer to the time the merits of observing PSQM teacher training sessions. Where we do observe, we will attempt to attend the second of the two training sessions that involve the case study schools. PSQM will make the training materials available to RAND prior to observations so that there is a clear indicate of what should be observed.
Attendance of trainings by teachers is mandatory and will be tracked by the PSQM team through attendance logs.
Based on information provided by the PSQM team, we will also report on the number of schools that are successful in gaining the award they aimed for, and numbers of where a submission was sent for a second review, a school was asked to submit additional evidence to get the award, and had a deferral/extension.
POST-INTERVENTION
Exploring programme implementation and changes in practice
Online surveys
One of the aims of the post-intervention headteacher surveys (in June – July 2020 and again in June – July 2021) will be to capture any potential changes in science practice (in both intervention and control schools). Any such identified changes will be highlighted in the evaluation report.
Hub leader surveys will focus on their experiences of working with schools and any barriers and facilitators to implementation. We expect that it will take no more than 10-15 minutes to complete the online surveys. The text for the survey will be prepared and compiled by RAND Europe and disseminated to schools by PSQM.
Paper surveys
To facilitate higher response rates, paper surveys will be distributed to two staff members per school in June-July 2020 and again in 2021. The focus will be on usual practice, attitudes, perceptions and science-related activities in the classroom.
The majority of the items will be the same in both control and intervention schools, examining practices and attitudes around teaching of science and science-specific CPD activities. The questions will be based on the Wellcome State of the Nation surveys for teachers and subject leaders. In addition, in the intervention arm, intervention-specific questions will be included.
Interviews in case study schools
Four schools among those assigned to the Intervention (PSQM), representing a diverse set of characteristics, will be approached for in-depth case studies. These will involve interviews with school teaching staff and headteachers, as well as parents, and – where possible – school governors, as well as visits to PSQM training sessions attended by the year 5 teacher and science subject leader from those schools.
These interviews will allow the evaluation team to gain a more in-depth understanding of what PSQM involves in practice for participating teachers, subject leaders, and schools, and explore the mechanisms of change as a result of the intervention.
Examining continuity of Year 5 teachers
We would expect programme effects to be strongest for Year 5 pupils in those schools where Year 5 teachers trained in PSQM continue teaching the next Year 5 cohort (Cohort B). To examine this, we will descriptively compare programme outcomes for those schools where the same teachers are working in Year 5 in 2020/21. The information on continuity will be based on self-report by teacher in Year 2 as part of their surveys.
Compliance measure
To enable a non-compliance analysis, compliance will be defined at the school level, based on completion of programme activities, as recorded by the PSQM team.
Data collection plan:
Data collection mode Participants When
Online Survey Headteacher/SLT Year 1, autumn term
Online Survey Headteacher/SLT Year 1, summer term
Online Survey Headteacher/SLT Year 2, summer term
Online Survey Hub leader Year 1, summer term (first 2 weeks of July)
Paper Survey Y5 Teachers (teacher selected for PSQM) Year 1, summer term (first 2 weeks of July)
Paper Survey Science subject leaders Year 1, summer term (first 2 weeks of July)
Paper Survey Teacher and subject leaders Year 2, summer term (first 2 weeks of July)
Feedback from training events Teachers Year 1
Data on previous rounds of PSQM Schools Set-up year
Monitoring data from PSQM Schools Year 1
Interviews Teacher, headteacher, parent, governor (4 people per school) + hub leader Year 1 and 2, summer term (first 2 weeks of July)
Interviews Hub leader Year 1, summer term
Intervention type
Behavioural
Phase
Drug names
Primary outcome measure
Science attainment will be measured as in Hanley et al., (2015) (potentially modified to ensure fit with the National Curriculum), science assessment with scores ranging from 0 to 41 points
Secondary outcome measures
Pupil science knowledge in study year 1, pupil attitudes to science and science teaching will be measured using the equivalent scales from the "Trends in International Mathematics and Science Study (TIMMS)"
Overall trial start date
01/08/2018
Overall trial end date
30/09/2022
Reason abandoned (if study stopped)
Eligibility
Participant inclusion criteria
Teachers:
Primary school science teachers
Pupils:
Pupils in the intervention group will receive schooling from teachers who have been trained, all pupils will be subject to outcome testing
Participant type
Other
Age group
All
Gender
Both
Target number of participants
Primary school science teachers in ~140 schools; ~5,600 - 7,000 pupils
Participant exclusion criteria
1. School cannot have received a PSQM award in the last 3 years (i.e., a school has not participated in PSQM in 2017, 2018 or 2019)
2. School must be a state primary, junior or all-through school
3. Schools with mixed Year 5/6 or another combination are eligible if they have Year 5 pupils taught separately by one teacher for the core subjects, including science
4. Infant or first schools, private schools, special schools, Pupil Referral Units (PRUs) or middle schools are not eligible
Recruitment start date
11/02/2019
Recruitment end date
31/07/2021
Locations
Countries of recruitment
United Kingdom
Trial participating centre
PSQM
University of Hertfordshire
Hatfield
AL10 9AB
United Kingdom
Sponsor information
Organisation
RAND Europe
Sponsor details
Westbrook Centre
Milton Road
Cambridge
CB4 1 YG
United Kingdom
+44 1223 353 329
psqm@rand.org
Sponsor type
Research organisation
Website
Funders
Funder type
Charity
Funder name
Education Endowment Foundation
Alternative name(s)
EEF
Funding Body Type
private sector organisation
Funding Body Subtype
Trusts, charities, foundations (both publically funded and privately funded)
Location
United Kingdom
Results and Publications
Publication and dissemination plan
May 2019 - Protocol
September 2020 - Statistical Analysis Plan
September 2022 - Final Report
IPD sharing statement:
The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository. Aggregated outcome data will be sent to the EEF archive, managed by the Fischer Family Trust (FFT), together with UPN’s. This is to allow for long term follow up investigations in the future.
Intention to publish date
30/09/2022
Participant level data
To be made available at a later date
Basic results (scientific)
Publication list