Condition category
Not Applicable
Date applied
27/03/2018
Date assigned
28/11/2018
Last edited
28/11/2018
Prospective/Retrospective
Retrospectively registered
Overall trial status
Ongoing
Recruitment status
No longer recruiting

Plain English Summary

Background and study aims
The study aims to test whether students using an online formative question setting and marking programme known as ED have higher grades at GCSE Mathematics than students that do not have access to ED, and test whether teachers using ED to set, assess and provide feedback to students have lower workloads than those without access to it.

Who can participate?
Students entering Year 10 from September 2018, in 90 participating English secondary schools (intervention schools) and their teachers are encouraged to use ED. The students are aged 14-15 years at the commencement of the study and will be followed for two years.

What does the study involve?
Outcomes for students and teachers using ED in intervention schools are compared to outcomes for students and teachers following ‘business as usual’ instruction in mathematics in control schools. Access to ED is provided to all teachers and students in intervention schools on an equal basis. Students and teachers in control schools are prevented from accessing ED during the study.

What are the possible benefits and risks of participating?
Students using ED receive weekly quizzes in mathematics related to their course of study. Teachers set quizzes but the ED system automatically marks them and identifies areas of weakness for students that teachers can then address. ED provides students with timely and accurate feedback that is formative in nature. For teachers ED reduces their workload through automatically scoring quizzes and assessing performance thus reducing time otherwise spent marking and providing feedback.

Where is the study run from?
The intervention is delivered by Eedi (a commercial enterprise) and the Behavioural Insights Team, who together recruit schools and providing training in the use of ED.

When is the study starting and how long is it expected to run for?
The study started in January 2018 and is expected to run around three years

How long will the trial be recruiting participants for?
Schools will be recruited to the study between February and June 2018. The study is funded by the Education Endowment Foundation

Who is the main contact?
Professor Stephen Morris, s.morris@mmu.ac.uk






Trial website

n/a

Contact information

Type

Scientific

Primary contact

Prof Stephen Morris

ORCID ID

Contact details

Manchester Metropolitan University
Geoffrey Manton Building
4 Rosamond Street West
Off Oxford Road
Manchester
M15 6LL
United Kingdom

Additional identifiers

EudraCT number

ClinicalTrials.gov number

Protocol/serial number

00001

Study information

Scientific title

A cluster randomised trial evaluating the effectiveness of ED formative assessment programme on attainment in GCSE mathematics among a cohort of Year 11 pupils in English schools

Acronym

n/a

Study hypothesis

Exposure to ED formative assessment platform for mathematics raises average attainment at GCSE mathematics among Year 11 pupils in English secondary schools compared to instruction in mathematics without access to ED

Ethics approval

Manchester Metropolitan University Arts & Humanities Faculty Research Ethics and Governance Committee, 14/02/2018, A&H1718-40

Study design

Two-arm parallel randomised controlled cluster trial

Primary study design

Interventional

Secondary study design

Cluster randomised trial

Trial setting

Schools

Trial type

Other

Patient information sheet

Not available in web format, please use contact details to request participant information sheet

Condition

Low attainment in GCSE Mathematics and excessive teacher workload

Intervention

This study seeks to examine whether pupils’ exposure to ED formative question setting platform and marking system for mathematics in secondary schools raises attainment in mathematics at GCSE. It also seeks to understand whether teacher workload, for teachers using ED for mathematics instruction at Years 10 and 11, is reduced. Whole schools will be assigned at random to either intervention (exposure to ED) or control conditions (non-exposure to ED) on a 1:1 basis.

The intervention is an online question setting and diagnostic platform for instruction in mathematics at GCSE known as ED (previously known as diagnostic questions), developed by Eedi (https://www.eedi.co.uk/). The core components of the programme are as follows:
1. Schools assign formative assessment ‘quizzes’ aligned to their exam board’s Scheme of Work. Eedi instantly populates a calendar of multiple choice quizzes for the entire year. If they wish, teachers can manually adjust the ordering/scheduling of the quizzes. There are two quizzes per topic in the Scheme of Work.
2. Students complete quizzes each week according to the Scheme of Work set by their teacher. Each quiz comprises ten questions and each question has four multiple choice answers. Each incorrect answer is designed to diagnose a specific ‘misunderstanding’. Quizzes are marked through Eedi. Students are then prompted to review their answers with feedback targeting their specific misunderstanding. Students are also given targeted learning materials from Oxford University Press (OUP).
3. Teachers do not have to mark quizzes, Eedi marks them automatically. Teachers can review their class’s scores on Eedi, identify common misunderstandings and if necessary, send additional feedback to all students making that mistake.
4. Parents can receive automated text messages about new quizzes, students not completing quizzes as required, or what topics are being covered in class. More detailed reports are available by logging in to the Eedi website.

The intervention will be delivered to school pupils entering Year 10 from September 2018 studying mathematics at GCSE, in around 90 English state secondary schools (intervention schools). The intervention will be delivered over two school years (Years 10 and 11 for this cohort).
Training for schools is delivered by Eedi. Each school appoints a project lead to liaise with Eedi. Eedi works with the school project lead to assist them in setting up the school’s scheme of work on Eedi. Eedi provides two hours of training in person to the school’s maths department during the spring/summer terms 2018. This addresses the following:
Context for this study and the importance of formative assessment
Monitoring quizzes and whole class performance
Giving feedback and setting further quizzes
Parental alerts
How to provide online access for all students
How to use Eedi’s technical support

Eedi provide additional training to school project leads, which covers:
Troubleshooting and minor problems
Access to online support
Further support for ensuring access to Eedi for all students
Liaison with Eedi’s account manager

Schools have access to online technical support available 8am to 8pm, Monday to Friday. Eedi monitors usage of the Eedi platform and aims to proactively approach schools with the offer of support should problems become apparent. Parents, pupils and schools are given access to OUP materials that address misconceptions identified in Eedi throughout the life of the study (this is usually an additional paid for service, but for the trial free access has been granted to all intervention schools). Schools are provided with a grant of £1000 to cover the costs of text messaging associated with the operation of Eedi. Control schools are also offered an incentive of £1000 to take part and adhere to their responsibilities in the trial (as set out in the MOU), this will be paid at the end of trial period (July 2020).

Intervention type

Behavioural

Phase

Drug names

Primary outcome measure

Attainment in GCSE Mathematics at the end of Year 11

Secondary outcome measures

Teacher workload assessed by surveys of teachers responsible for delivering instruction in mathematics to the focal cohort of school pupils described above in both intervention and control schools. The survey will be administered online via email.
Surveys will be administered online to teachers in both intervention and control schools at four points in time. The first of these occasions will be prior to the commencement of the intervention in the summer term 2018. Post randomisation teacher surveys will be administered in December 2018, March 2019 and March 2020.
At each measurement occasion teachers will be asked to provide an estimate of the total amount of time they were engaged in the following tasks during a specified reference week:
Setting maths homework
Marking maths homework
Giving individual feedback to students in mathematics
Giving group feedback to students
Lesson planning in mathematics
From this information, the total mathematics related workload will be calculated as the total amount of time spent across these five dimensions per teacher per week.

Overall trial start date

01/01/2018

Overall trial end date

31/07/2020

Reason abandoned (if study stopped)

Eligibility

Participant inclusion criteria

Schools identified in UK Education Endowment Foundation databases, AQA and EdExcel examining board databases and through Eedi’s own data base of teachers are approached in order to gauge their interest in participating in the study. It was felt important to only admit schools to the trial where existing use of ED was either completely absent or minimal. However, ED is available to schools, teachers and pupils across England, making it difficult to find schools in which there is no existing use of the system.

Once a school has signalled their interest, Eedi (the developers and implementers) assess existing usage of Eedi at the school through examining their administrative data records. These records indicate which schools, teachers and pupils have existing accounts on Eedi and the extent of their current usage. Only schools with 30 or fewer existing pupil accounts were considered for inclusion in the trial. This threshold was devised by the developers based upon their knowledge of existing usage and how best to measure it and is considered to be low existing usage. All schools with more intensive usage according to this criterion are excluded from the trial.

It is important to note that schools can only enter the trial if they agree that access to Eedi is disabled completely for the duration of study should they be allocated to control conditions. This step is taken to avoid non-compliance among control schools.

In each school, all pupils currently in Year 9 during the school year 2017/18, whose parents do not withdraw their child from the study are considered to be in range of intervention. Pupils in intervention schools will receive instruction in mathematics using Eedi for two school years – 2018/19 (Year 10) and 2019/20 (Year 11).

Participant type

Other

Age group

Child

Gender

Both

Target number of participants

A total of 180 schools will be recruited to the trial comprising approximately 30,000 pupils

Participant exclusion criteria

Pupils whose parents opt to withdraw their child from the study are excluded from the study sample

Recruitment start date

01/02/2018

Recruitment end date

31/07/2018

Locations

Countries of recruitment

United Kingdom

Trial participating centre

Eedi Ltd
Unit 4.2.2 The Leather Market, Weston Street
London
SE1 3ER
United Kingdom

Trial participating centre

Behavioural Insights Team
4 Matthew Parker Street
London
SW1H 9NP
United Kingdom

Trial participating centre

Policy Evaluation Research Unit
Department of Sociology, Manchester Metropolitan University Geoffrey Manton Building Off Oxford Road
Manchester
M15 6LL
United Kingdom

Trial participating centre

AlphaPlus Consultancy Ltd.
Unit 109, Albert Mill 10 Hulme Hall Road
Manchester
M15 4LY
United Kingdom

Sponsor information

Organisation

Education Endowment Foundation

Sponsor details

9th Floor
Millbank Tower
21-24 Millbank
London
SW1P 4QP
United Kingdom
020 7802 1676
info@eefoundation.org.uk

Sponsor type

Charity

Website

https://educationendowmentfoundation.org.uk/

Funders

Funder type

Not defined

Funder name

Education Endowment Foundation

Alternative name(s)

EEF

Funding Body Type

private sector organisation

Funding Body Subtype

foundation

Location

United Kingdom

Results and Publications

Publication and dissemination plan

A main evaluation report and summary will be published by study sponsor the UK Education Endowment Foundation in its report series.

IPD sharing statement
The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository known as the EEF Data Archive (https://educationendowmentfoundation.org.uk/projects-and-evaluation/evaluating-projects/evaluator-resources/submitting-your-data-to-the-fft-archive/). Evaluators will deposit project data with pupil identifiers so that further data linking can take place. Access to data is controlled by approval from the Education Endowment Foundation (EEF) and the Department for Education. Only anonymised data will be provided for secondary or meta-analysis from the EEF Data Archive. Retention of the data is determined by EEF as the data controller and where data include records from the UK National Pupil Database the UK Department for Education.

Intention to publish date

31/12/2021

Participant level data

Stored in repository

Basic results (scientific)

Publication list

Publication citations

Additional files

Editorial Notes