The effect of incentive payments for teachers on pupil attainment

Plain English Summary

Background and study aims
The aim of this study is to find out whether monetary incentives and coaching for teachers motivates them to improve their pupils’ attainment. Although some recent research has demonstrated that performance-based compensation holds promise for improving pupil achievement, in three studies researchers found small, non-significant impacts on pupil achievement. However, research from developing nations indicates that teacher performance pay can be effective. One study found that providing teachers with bonuses upfront, and then asking for money back if pupil achievement does not improve sufficiently, was an effective way to improve pupil achievement. Coaching is also included as part of this study because some teachers may be motivated by pay incentives, but may not know what to do to improve their performance. Coaching is intended to fill this gap and may increase teacher performance compared to the use of incentive pay alone. There is evidence that coaching can improve both teacher performance and pupil outcomes.

Who can participate?
Year 2-6 primary school academy teachers and Year 7-8 secondary academy mathematics teachers

What does the study involve?
Participating teachers are randomly allocated to either the intervention or the control group for financial incentives. Teachers allocated to the intervention group receive a “loss-framed” financial incentive of £4000 at the beginning of the school year. This is defined as “loss-framed” because a percentage of this amount may be recouped at the end of the year depending on the teacher’s pupils’ progress relative to similar pupils. Teachers allocated to the control group continue with 'business as usual'. A group of teachers are also randomly selected to receive coaching. This excludes teachers who act as coaches and teachers who definitely need to receive coaching as indicated by the head teacher. Teachers allocated to the coaching condition, as well as teachers who definitely need coaching, receive personalised coaching throughout the year during which lessons are observed weekly and small changes are recommended by the coach during short 30-minute sessions. At the end of the 2017-2018 school year the mathematics test results of the two groups are compared.

What are the possible benefits and risks of participating?
For pupils in participating classes the study may improve learning outcomes, and no real risks are anticipated. Teachers allocated to receive financial incentives may experience stress. The loss aversion setup, whereby teachers receive a sum of money at the start of the year that may have to be repaid, can carry the following risks of stress: financial stress following a fear of spending the money over the course of the year without being able to repay; lack of understanding of how the incentive repayment works, leading to false expectations about financial gains (or losses); and stress about teaching practices to ensure that pupils achieve higher grades in order to retain the financial incentive.

Where is the study run from?
RAND Europe (UK)

When is the study starting and how long is it expected to run for?
September 2016 to December 2019

Who is funding the study?
Education Endowment Foundation (UK)

Who is the main contact?
Dr Alex Sutherland

Trial website

Type

Scientific

Primary contact

Dr Alex Sutherland

ORCID ID

http://orcid.org/0000-0003-0276-9705

Contact details

Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom

EudraCT number

ClinicalTrials.gov number

Protocol/serial number

N/A

Scientific title

A randomised controlled trial of the 'Incentives and Coaching Research' intervention for pupils in Primary Years 2-6 and Secondary Maths pupils in Year 7 and 8 in schools which are part of the United Learning Academies in the UK

Acronym

Payment by Results in Education (PbREd)

Study hypothesis

1. Hypothesis 1: financial incentives (loss-framed) will have a positive effect on pupil outcomes (in this trial mathematics) who are in randomly assigned intervention classes.
2. Hypothesis 2: coaching of teachers will have a positive effect on pupil outcomes (in this trial mathematics) who are in randomly assigned intervention classes.
3. Hypothesis 3: financial incentives (loss-framed) combined with coaching will have a greater positive effect on pupil outcomes (in this trial mathematics) than those who are randomly assigned to control classes or to incentives alone.

Ethics approval

RAND Human Subjects Protection Committee, 26/01/2017, ref: RAND HSPC: 2016-0475-AM01

Study design

Two-stage randomised controlled trial

Primary study design

Interventional

Secondary study design

Randomised controlled trial

Trial setting

Schools

Trial type

Other

Patient information sheet

Not available in web format, please use the contact details to request a participant information sheet

Condition

Mathematics comprehension in Year 2-6 Primary school pupils and Year 7-8 Secondary school pupils

Intervention

Participating teachers will be allocated to either the intervention or the control group for financial incentives. Teachers randomised to the financial incentive condition will receive a “loss-framed” financial incentive of £4000 at the beginning of the school year. This is defined as “loss-framed” because a percentage of this amount may be recouped at the end of the year depending on the teacher’s pupils’ progress relative to similar pupils. Control groups will proceed with 'business as usual'.

In a second round of randomisation, a group of teachers will be randomised to receive coaching. Two groups of teachers will not be randomised to coaching, namely teachers who act as a coach, and teachers who definitely need to receive coaching as indicated by the head teacher. Teachers randomised to the coaching condition, as well as teachers who will definitely receive coaching, will receive personalised coaching throughout the year during which lessons are observed weekly and small changes recommended by the coach during short 30-minute sessions.

Duration of the intervention: 1 year (school year 2017-2018)
Follow-up: outcome testing is at the end of the 2017-2018 school year, with no further follow-up

Intervention type

Behavioural

Phase

Drug names

Primary outcome measure

The trial will be measuring primary outcomes for different year groups:
Year 3, 4, 5, 7 and 8: the primary outcome measure at the end of Years 3, 4, 5, 7 and 8 will be a stand-alone measure of mathematics administered in the summer term of 2018, one year after the intervention begins. The measure of mathematics will be administered and collected by the Australian Council for Educational Research (ACER). The study will use ACERs Essential Learning Metric (ELM) for Mathematics for Years 3, 4, 5, 7 and 8. The tests will be digital and will take 45-50 minutes to complete. The tests consist of two sets of measures: mathematical content (number, algebra, measurement, geometry, statistics and probability); and mathematical processes (understanding, fluency, problem solving and reasoning).
Year 2 and 6: the primary outcome measure at the end of Years 2 and 6 will be the Key Stage 1 and Key Stage 2 test results for mathematics, administered in the summer term of 2018.

Secondary outcome measures

1. Cost effectiveness of the intervention through collecting data on intervention costs. Cost data will be collected through surveys of head teachers and teachers. Questions on costs will be included in surveys administered at midline (3 months after the start of the intervention, around December 2017), and at endline (in the summer term of 2018). Cost data will focus on prerequisite costs required for the implementation of the intervention, additional paid staff time required, paid staff cover required and unpaid overtime required
2. Compliance with allocation, through monitoring of the payout of financial incentives and monitoring of the coaching

Overall trial start date

01/09/2016

Overall trial end date

31/12/2019

Reason abandoned (if study stopped)

Participant inclusion criteria

All Year 2-6 Primary school pupils in participating schools in classes in which the teacher has agreed to participate and all Year 7-8 Secondary school pupils in participating schools in mathematics classes in which the teacher has agreed to participate.

Participant type

Other

Age group

Child

Gender

Both

Target number of participants

The target number of schools is 42; the target number of teachers is 400 (200 financial incentives, 200 control); the expected number of pupils is 13,000, depending on school size

Participant exclusion criteria

1. Special education facilities for pupils with physical and/or mental and/or behavioural disorders
2. Pupil Referral Units

Recruitment start date

01/01/2017

Recruitment end date

01/05/2017

Countries of recruitment

United Kingdom

Trial participating centre

RAND Europe
Westbrook Centre Milton Road
Cambridge
CB4 1YG
United Kingdom

Organisation

RAND Europe (UK)

Sponsor details

Westbrook Centre
Milton Road
Cambridge
CB4 1YG
United Kingdom

Sponsor type

Research organisation

Website

http://www.rand.org/randeurope.html

Funder type

Charity

Funder name

Education Endowment Foundation (UK)

Alternative name(s)

Funding Body Type

Funding Body Subtype

Location

Publication and dissemination plan

The main report will be for the Education Endowment Foundation, which will be published in the Spring of 2019. In addition, the University of California at San Diego and RAND Europe will separately develop peer-reviewed papers.

IPD sharing plan
The full dataset will not be made publicly available as it contains pupil names and teacher identifiers that could compromise confidentiality. The data will be held with the EEF’s data contractor FFT Education and in an anonymised form to the UK Data Archive.
The Education Endowment Foundation, the funder of the evaluation, uses data gathered from the research it funds to look at the longer-term benefits of the approaches being tried out in schools. This information is passed – in anonymous form – to the EEF’s partner FFT Education, who conduct follow ups via the National Pupil Database. Pupil names will only be used for to match pupils to the National Pupil Database.

Intention to publish date

01/03/2019

Participant level data

Not expected to be available

Basic results (scientific)

Publication list

Publication citations