Plain English Summary
Background and study aims
The aim of this study is to find out whether monetary incentives and coaching for teachers motivates them to improve their pupils’ attainment. Although some recent research has demonstrated that performance-based compensation holds promise for improving pupil achievement, in three studies researchers found small, non-significant impacts on pupil achievement. However, research from developing nations indicates that teacher performance pay can be effective. One study found that providing teachers with bonuses upfront, and then asking for money back if pupil achievement does not improve sufficiently, was an effective way to improve pupil achievement. Coaching is also included as part of this study because some teachers may be motivated by pay incentives, but may not know what to do to improve their performance. Coaching is intended to fill this gap and may increase teacher performance compared to the use of incentive pay alone. There is evidence that coaching can improve both teacher performance and pupil outcomes.
Who can participate?
Year 2-6 primary school academy teachers and Year 7-8 secondary academy mathematics teachers
What does the study involve?
Participating teachers are randomly allocated to either the intervention or the control group for financial incentives. Teachers allocated to the intervention group receive a “loss-framed” financial incentive of £4000 at the beginning of the school year. This is defined as “loss-framed” because a percentage of this amount may be recouped at the end of the year depending on the teacher’s pupils’ progress relative to similar pupils. Teachers allocated to the control group continue with 'business as usual'. A group of teachers are also randomly selected to receive coaching. This excludes teachers who act as coaches and teachers who definitely need to receive coaching as indicated by the head teacher. Teachers allocated to the coaching condition, as well as teachers who definitely need coaching, receive personalised coaching throughout the year during which lessons are observed weekly and small changes are recommended by the coach during short 30-minute sessions. At the end of the 2017-2018 school year the mathematics test results of the two groups are compared.
What are the possible benefits and risks of participating?
For pupils in participating classes the study may improve learning outcomes, and no real risks are anticipated. Teachers allocated to receive financial incentives may experience stress. The loss aversion setup, whereby teachers receive a sum of money at the start of the year that may have to be repaid, can carry the following risks of stress: financial stress following a fear of spending the money over the course of the year without being able to repay; lack of understanding of how the incentive repayment works, leading to false expectations about financial gains (or losses); and stress about teaching practices to ensure that pupils achieve higher grades in order to retain the financial incentive.
Where is the study run from?
RAND Europe (UK)
When is the study starting and how long is it expected to run for?
September 2016 to December 2019
Who is funding the study?
Education Endowment Foundation (UK)
Who is the main contact?
Dr Alex Sutherland
Trial website
Contact information
Type
Scientific
Primary contact
Dr Alex Sutherland
ORCID ID
http://orcid.org/0000-0003-0276-9705
Contact details
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
Additional identifiers
EudraCT number
ClinicalTrials.gov number
Protocol/serial number
N/A
Study information
Scientific title
A randomised controlled trial of the 'Incentives and Coaching Research' intervention for pupils in Primary Years 2-6 and Secondary Maths pupils in Year 7 and 8 in schools which are part of the United Learning Academies in the UK
Acronym
Payment by Results in Education (PbREd)
Study hypothesis
1. Hypothesis 1: financial incentives (loss-framed) will have a positive effect on pupil outcomes (in this trial mathematics) who are in randomly assigned intervention classes.
2. Hypothesis 2: coaching of teachers will have a positive effect on pupil outcomes (in this trial mathematics) who are in randomly assigned intervention classes.
3. Hypothesis 3: financial incentives (loss-framed) combined with coaching will have a greater positive effect on pupil outcomes (in this trial mathematics) than those who are randomly assigned to control classes or to incentives alone.
Ethics approval
RAND Human Subjects Protection Committee, 26/01/2017, ref: RAND HSPC: 2016-0475-AM01
Study design
Two-stage randomised controlled trial
Primary study design
Interventional
Secondary study design
Randomised controlled trial
Trial setting
Schools
Trial type
Other
Patient information sheet
Not available in web format, please use the contact details to request a participant information sheet
Condition
Mathematics comprehension in Year 2-6 Primary school pupils and Year 7-8 Secondary school pupils
Intervention
Participating teachers will be allocated to either the intervention or the control group for financial incentives. Teachers randomised to the financial incentive condition will receive a “loss-framed” financial incentive of £4000 at the beginning of the school year. This is defined as “loss-framed” because a percentage of this amount may be recouped at the end of the year depending on the teacher’s pupils’ progress relative to similar pupils. Control groups will proceed with 'business as usual'.
In a second round of randomisation, a group of teachers will be randomised to receive coaching. Two groups of teachers will not be randomised to coaching, namely teachers who act as a coach, and teachers who definitely need to receive coaching as indicated by the head teacher. Teachers randomised to the coaching condition, as well as teachers who will definitely receive coaching, will receive personalised coaching throughout the year during which lessons are observed weekly and small changes recommended by the coach during short 30-minute sessions.
Duration of the intervention: 1 year (school year 2017-2018)
Follow-up: outcome testing is at the end of the 2017-2018 school year, with no further follow-up
Intervention type
Behavioural
Phase
Drug names
Primary outcome measure
The trial will be measuring primary outcomes for different year groups:
Year 3, 4, 5, 7 and 8: the primary outcome measure at the end of Years 3, 4, 5, 7 and 8 will be a stand-alone measure of mathematics administered in the summer term of 2018, one year after the intervention begins. The measure of mathematics will be administered and collected by the Australian Council for Educational Research (ACER). The study will use ACERs Essential Learning Metric (ELM) for Mathematics for Years 3, 4, 5, 7 and 8. The tests will be digital and will take 45-50 minutes to complete. The tests consist of two sets of measures: mathematical content (number, algebra, measurement, geometry, statistics and probability); and mathematical processes (understanding, fluency, problem solving and reasoning).
Year 2 and 6: the primary outcome measure at the end of Years 2 and 6 will be the Key Stage 1 and Key Stage 2 test results for mathematics, administered in the summer term of 2018.
Secondary outcome measures
1. Cost effectiveness of the intervention through collecting data on intervention costs. Cost data will be collected through surveys of head teachers and teachers. Questions on costs will be included in surveys administered at midline (3 months after the start of the intervention, around December 2017), and at endline (in the summer term of 2018). Cost data will focus on prerequisite costs required for the implementation of the intervention, additional paid staff time required, paid staff cover required and unpaid overtime required
2. Compliance with allocation, through monitoring of the payout of financial incentives and monitoring of the coaching
Overall trial start date
01/09/2016
Overall trial end date
31/12/2019
Reason abandoned (if study stopped)
Participant recruitment issue
Eligibility
Participant inclusion criteria
All Year 2-6 Primary school pupils in participating schools in classes in which the teacher has agreed to participate and all Year 7-8 Secondary school pupils in participating schools in mathematics classes in which the teacher has agreed to participate.
Participant type
Other
Age group
Child
Gender
Both
Target number of participants
The target number of schools is 42; the target number of teachers is 400 (200 financial incentives, 200 control); the expected number of pupils is 13,000, depending on school size
Total final enrolment
46
Participant exclusion criteria
1. Special education facilities for pupils with physical and/or mental and/or behavioural disorders
2. Pupil Referral Units
Recruitment start date
01/01/2017
Recruitment end date
01/05/2017
Locations
Countries of recruitment
United Kingdom
Trial participating centre
RAND Europe
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
Sponsor information
Organisation
RAND Europe (UK)
Sponsor details
Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom
Sponsor type
Research organisation
Website
Funders
Funder type
Charity
Funder name
Education Endowment Foundation (UK)
Alternative name(s)
Funding Body Type
Funding Body Subtype
Location
Results and Publications
Publication and dissemination plan
The main report will be for the Education Endowment Foundation, which will be published in the Spring of 2019. In addition, the University of California at San Diego and RAND Europe will separately develop peer-reviewed papers.
IPD sharing plan
The full dataset will not be made publicly available as it contains pupil names and teacher identifiers that could compromise confidentiality. The data will be held with the EEF’s data contractor FFT Education and in an anonymised form to the UK Data Archive.
The Education Endowment Foundation, the funder of the evaluation, uses data gathered from the research it funds to look at the longer-term benefits of the approaches being tried out in schools. This information is passed – in anonymous form – to the EEF’s partner FFT Education, who conduct follow ups via the National Pupil Database. Pupil names will only be used for to match pupils to the National Pupil Database.
Intention to publish date
01/03/2019
Participant level data
Not expected to be available
Basic results (scientific)
Publication list