Condition category
Not Applicable
Date applied
25/10/2019
Date assigned
28/10/2019
Last edited
25/10/2019
Prospective/Retrospective
Retrospectively registered
Overall trial status
Ongoing
Recruitment status
No longer recruiting

Plain English Summary

Background and study aims
The Basic Maths Premium (BMP) Pilot is an intervention developed by the UK Department for Education (DfE) that provides additional funding to support 16-18-year-old students studying maths after GCSE. The premium, worth up to £500 per eligible student, will be made available to post-16 educational settings in the most disadvantaged areas of England. The pilot will run from autumn 2018 for two years. Settings are randomly allocated to three funding models. Model T1 consists of a guaranteed payment of £500 for every student. Model T2 consists of a guaranteed payment of £250 for every student and a further £250 in the academic year 2020 to 2021 for every student who achieves in maths. Model T3 consists of a payment of £500 for every student who achieves in maths. The aims of this study are to assess the impact of different funding models on maths outcomes for students, to identify how the additional funding is used by institutions to understand the underlying mechanisms of change, and to build up an evidence base on which activities lead to improvements in teaching and learning.

Who can participate?
The intervention was offered to all post-16 educational institutions in Category 5 and 6 Achieving Excellence Areas (AEA). All institutions that have agreed to be part of the pilot and had at least one eligible student from cohort 2018/19 enrolled by November 2018 received the intervention and were included in the evaluation. Students enrolled in a 16 to 19 study programme for the first time without prior attainment of a maths GCSE grade 4 or above for 2018/2019 and 2019/2020 academic years were eligible for the additional funding attached to this pilot.

What does the study involve?
Schools are randomly allocated to funding model T1, T2 or T3. The impact of the intervention on maths GCSE is measured, compared with a matched control group. The study also looks at how the programme is implemented across different settings and how the funding is spent.

What are the possible benefits and risks of participating?
The expected benefits are higher attainment in maths. The main risk of participating is a displacement effect (i.e. higher attainment in maths is achieved at the cost of lower attainment in other subjects such as English).

Where is the study run from?
1. National Centre for Social Research (NatCen) (UK)
2. Schools in England

When is the study starting and how long is it expected to run for?
May 2018 to August 2022

Who is funding the study?
Education Endowment Foundation (UK)

Who is the main contact?
Dr Arnaud Vaganay
arnaud.vaganay@natcen.ac.uk

Trial website

https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/basic-maths-premium-pilot/

Contact information

Type

Public

Primary contact

Dr Arnaud Vaganay

ORCID ID

http://orcid.org/0000-0001-5584-9050

Contact details

35 Northampton Square
London
EC1V 0AX
United Kingdom
+44 (0)207 250 1866
arnaud.vaganay@natcen.ac.uk

Additional identifiers

EudraCT number

Nil known

ClinicalTrials.gov number

Nil known

Protocol/serial number

P13021

Study information

Scientific title

Effect of additional funding on maths attainment in post-16 institutions: a three-armed cluster-randomised trial with matched comparison group

Acronym

BMP

Study hypothesis

Question 1: What is the impact of the different funding models on maths attainment of eligible students in post-16 settings in England compared to business as usual?
Hypothesis 1a: The 2018/2019 cohort of students (hereafter 'Cohort 18/19') in participating institutions that have been randomly assigned to model T1 will on average have better maths outcomes than comparison students in non-participating institutions.
Hypothesis 1b: Cohort 18/19 students in participating institutions that have been randomly assigned to model T2 will on average have better maths outcomes than comparison students in non-participating institutions.
Hypothesis 1c: Cohort 18/19 students in participating institutions that have been randomly assigned to Model T3 will on average have better maths outcomes than comparison students in non-participating institutions.
H1d: Same as hypothesis H1a but for the 2019/2020 cohort of students (hereafter 'Cohort 19/20)
H1e: Same as hypothesis H1b but for Cohort 19/20 students
H1f: Same as hypothesis H1c but for Cohort 19/20 students
H1g: Same as hypothesis H1a but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H1h: Same as hypothesis H1b but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H1i: Same as hypothesis H1c but for all participating students (Cohort 18/19 and Cohort 19/20 students)

Question 2: What is the effectiveness of the different funding models relative to each other?
H2a: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T1 compared to those in institutions assigned to model T3.
H2b: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T2 compared to those in institutions assigned to model T3.
H2c: There will be a different (higher or lower) impact on maths outcomes of Cohort 18/19 students in institutions assigned to model T1 compared to those in institutions assigned to model T2.
H2d: Same as hypothesis H2a but for Cohort 19/20 students
H2e: Same as hypothesis H2b but for Cohort 19/20 students
H2f: Same as hypothesis H2c but for Cohort 19/20 students
H2g: Same as hypothesis H2a but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H2h: Same as hypothesis H2b but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H2i: Same as hypothesis H2c but for all participating students (Cohort 18/19 and Cohort 19/20 students)

Question 3: What are the effects of the different funding models on GCSE maths exam retake?
H3a: Being assigned to model T1 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions.
H3b: Being assigned to model T2 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions.
H3c: Being assigned to model T3 will have a different (higher or lower) impact on the chances of Cohort 18/19 students attempting a GCSE maths re-take exam compared to comparison students in non-participating institutions.
H3d: Same as hypothesis H3a but for Cohort 19/20 students
H3e: Same as hypothesis H3b but for Cohort 19/20 students
H3f: Same as hypothesis H3c but for Cohort 19/20 students
H3g: Same as hypothesis H3a but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H3h: Same as hypothesis H3b but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H3i: Same as hypothesis H3c but for all participating students (Cohort 18/19 and Cohort 19/20 students)

Question 4: How do these effects differ by student disadvantage status, prior attainment in GCSE maths and number of eligible students per institution?
H4a: The effect of the T1, T2, and T3 combined25 on student attainment for Cohort 18/19 will differ (be higher or lower) for those students that have ever been eligible for free school meals compared to comparison students in non-participating institutions.
H4b: The effect of the T1, T2 and T3 combined on student attainment for Cohort 18/19 participants will differ (be higher or lower) depending on the number of eligible students that institutions have.
H4c: The effect of the T1, T2 and T3 combined on student attainment and attempt to retake GCSE maths for Cohort 18/19 participants will differ (be higher or lower) depending on student prior attainment in GCSE maths.
H4d: Same as hypothesis H4a but for Cohort 19/20 students
H4e: Same as hypothesis H4b but for Cohort 19/20 students
H4f: Same as hypothesis H4c but for Cohort 19/20 students
H4g: Same as hypothesis H4a but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H4h: Same as hypothesis H4b but for all participating students (Cohort 18/19 and Cohort 19/20 students)
H4i: Same as hypothesis H4c but for all participating students (Cohort 18/19 and Cohort 19/20 students)

Ethics approval

Approved 11/05/2018, NatCen's Research Ethics Committee (REC) (35 Northampton Square, London, EC1V 0AX, UK; Tel: +44 (0)20 7250 1866; Email: joanne.mcbride@natcen.ac.uk), ref: P13021

Study design

Multi-site three-armed cluster-randomised trial with matched comparison group

Primary study design

Interventional

Secondary study design

Cluster randomised trial

Trial setting

Schools

Trial type

Other

Patient information sheet

http://www.natcen.ac.uk/taking-part/studies-in-field/maths-premium-pilot/

Condition

Attainment in maths; inequalities in education

Intervention

The Basic Maths Premium (BMP) Pilot is an intervention developed by the Department for Education (DfE) that provides additional funding to support 16-18-year-old students studying maths after GCSE. The premium, worth up to £500 per eligible student, will be made available to post-16 educational settings in the most disadvantaged areas of England. The pilot will run from autumn 2018 for two years.

The BMP Pilot will test three alternative funding models:
Model T1: Consists of a guaranteed payment of £500 for every student enrolled by November 2018 without a Grade 4 or above in GCSE maths (only for Cohort 18/19 funding-eligible students);
Model T2: Consists of a guaranteed payment of £250 for every student enrolled by November 2018 without a Grade 4 or above in GCSE maths, and a further £250 in the academic year 2020 to 2021 for every student who achieves in maths by Summer 2020 (only for Cohort 18/19 funding-eligible students);
Model T3: Consists of a payment of £500 in the academic year 2020 to 2021 for every student enrolled by November 2018 who goes on to achieve in maths by Summer 2020 (only for Cohort 18/19 funding-eligible students).

Institutions can choose how to utilise their additional funding. Regardless of how resources were assigned, institutions will be free to invest their resources in Cohort 18/19 and/or Cohort 19/20 students as they see fit. This implies that eligible students (Cohort 18/19) will not necessarily be the only beneficiaries of the intervention.

Institutions were randomly allocated to model T1, T2 or T3 using Stata software.

Given DfE’s commitment to offer all institutions in the pilot areas some form of intervention, it was not possible to randomly assign a proportion of the institutions in the pilot areas to a control group. Consequently, the estimation of the impact of the three treatment conditions compared to business as usual will need to rely on the establishment of a counterfactual through a quasi-experimental design (QED). The researchers will use the doubly-robust method to create a control group. In order to construct a control group, they will use a pool of institutions similar to those in the treatment group. The doubly robust-method mixes propensity score estimation with regression analysis. To estimate the impact of the programme, the researchers will initially estimate propensity scores modelling the selection process into treatment. Later on, a regression model will be constructed to identify the causal effect of the treatment in the outcome of interest. Their preferred regression models will include student-level characteristics as well as institution-level characteristics and institution random effects, accounting for the clustered nature of the data. Finally, and to produce their preferred estimate of the impact of the intervention (the doubly robust-estimator) the researchers will make use of both the estimated regression coefficients as well as the estimated propensity scores for all individuals in the sample.

Intervention type

Behavioural

Phase

Drug names

Primary outcome measure

Level 2 attainment in maths, defined as achievement of a grade 4 or above in GCSE maths or attainment of a pass in a Level 2 Functional Skills (depending on the student’s prior attainment). This data will be taken from the National Pupil Database (NPD).

All baseline data for the 18/19 cohort – collected from National Pupil Dataset in November 2018
All baseline data for the 19/20 cohort – collected from National Pupil Dataset in November 2019
All endline data for the 18/19 cohort – collected from National Pupil Dataset in November 2020
All endline data for the 19/20 cohort – collected from National Pupil Dataset in November 2021

Secondary outcome measures

Whether participants attempted to resit their GCSE maths exam. This data will be taken from the National Pupil Database (NPD).

All baseline data for the 18/19 cohort – collected from National Pupil Dataset in November 2018
All baseline data for the 19/20 cohort – collected from National Pupil Dataset in November 2019
All endline data for the 18/19 cohort – collected from National Pupil Dataset in November 2020
All endline data for the 19/20 cohort – collected from National Pupil Dataset in November 2021

Overall trial start date

01/05/2018

Overall trial end date

30/07/2022

Reason abandoned (if study stopped)

Eligibility

Participant inclusion criteria

In order to be eligible to take part, institutions must meet the following criteria:
1. Post-16 institutions in Category 5 and 6 AEAs that have signed up for the pilot and have at least one student eligible to receive BMP funding by November 2018

In order to be eligible to take part, students must meet the following criteria:
1. Students without prior attainment of at least grade 4 in GCSE maths that enrol in their post-16 studies in the participating institutions in the 2018/2019 academic year (the “eligible students” for this pilot). These students are hereafter referred to as “Cohort 18/19”
2. Students without prior attainment of at least grade 4 in GCSE maths that will enrol in their post-16 studies in the participating institutions in the 2019/2020 academic year. These students are hereafter referred to as “Cohort 19/20”

Participant type

Mixed

Age group

Other

Gender

Both

Target number of participants

50,000

Participant exclusion criteria

Does not meet inclusion criteria

Recruitment start date

15/05/2018

Recruitment end date

31/07/2018

Locations

Countries of recruitment

United Kingdom

Trial participating centre

National Centre for Social Research (NatCen)
35 Northampton Square
London
EC1V 0AX
United Kingdom

Sponsor information

Organisation

Education Endowment Foundation

Sponsor details

5th floor
Millbank Tower
21-24 Millbank
London
SW1P 4QP
United Kingdom
+44 (0)207 802 1676
info@eefoundation.org.uk

Sponsor type

Charity

Website

https://educationendowmentfoundation.org.uk

Funders

Funder type

Charity

Funder name

Education Endowment Foundation

Alternative name(s)

EEF

Funding Body Type

private sector organisation

Funding Body Subtype

Trusts, charities, foundations (both publically funded and privately funded)

Location

United Kingdom

Results and Publications

Publication and dissemination plan

1. A protocol and analysis plan is available on the website of the Education Endowment Foundation. See: https://educationendowmentfoundation.org.uk/projects-and-evaluation/projects/basic-maths-premium-pilot/
2. Results will be published as a report to the EEF on the same webpage.

IPD sharing statement
The dataset(s) used in the above-mentioned analyses will be anonymised and archived in a repository managed by FFT Education Datalab: https://ffteducationdatalab.org.uk/. Data and results will be available around May 2023.

Intention to publish date

05/05/2023

Participant level data

Stored in repository

Basic results (scientific)

Publication list

Publication citations

Additional files

Editorial Notes

25/10/2019: Trial's existence confirmed by ethics committee.