Evaluating the effectiveness of ED (previously Diagnostic Questions or DQ) formative assessment programme on raising attainment in mathematics at GCSE
| ISRCTN | ISRCTN62362872 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN62362872 |
| Secondary identifying numbers | 00001 |
- Submission date
- 27/03/2018
- Registration date
- 28/11/2018
- Last edited
- 28/10/2022
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English Summary
Background and study aims
The study aims to test whether students using an online formative question setting and marking programme known as ED have higher grades at GCSE Mathematics than students that do not have access to ED, and test whether teachers using ED to set, assess and provide feedback to students have lower workloads than those without access to it.
Who can participate?
Students entering Year 10 from September 2018, in 90 participating English secondary schools (intervention schools) and their teachers are encouraged to use ED. The students are aged 14-15 years at the commencement of the study and will be followed for two years.
What does the study involve?
Outcomes for students and teachers using ED in intervention schools are compared to outcomes for students and teachers following ‘business as usual’ instruction in mathematics in control schools. Access to ED is provided to all teachers and students in intervention schools on an equal basis. Students and teachers in control schools are prevented from accessing ED during the study.
What are the possible benefits and risks of participating?
Students using ED receive weekly quizzes in mathematics related to their course of study. Teachers set quizzes but the ED system automatically marks them and identifies areas of weakness for students that teachers can then address. ED provides students with timely and accurate feedback that is formative in nature. For teachers ED reduces their workload through automatically scoring quizzes and assessing performance thus reducing time otherwise spent marking and providing feedback.
Where is the study run from?
The intervention is delivered by Eedi (a commercial enterprise) and the Behavioural Insights Team, who together recruit schools and providing training in the use of ED.
When is the study starting and how long is it expected to run for?
The study started in January 2018 and is expected to run around three years
How long will the trial be recruiting participants for?
Schools will be recruited to the study between February and June 2018. The study is funded by the Education Endowment Foundation
Who is the main contact?
Professor Stephen Morris, s.morris@mmu.ac.uk
Contact information
Scientific
Manchester Metropolitan University
Geoffrey Manton Building
4 Rosamond Street West
Off Oxford Road
Manchester
M15 6LL
United Kingdom
Study information
| Study design | Two-arm parallel randomised controlled cluster trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Other |
| Participant information sheet | Not available in web format, please use contact details to request participant information sheet |
| Scientific title | A cluster randomised trial evaluating the effectiveness of ED formative assessment programme on attainment in GCSE mathematics among a cohort of Year 11 pupils in English schools |
| Study acronym | n/a |
| Study hypothesis | Exposure to ED formative assessment platform for mathematics raises average attainment at GCSE mathematics among Year 11 pupils in English secondary schools compared to instruction in mathematics without access to ED |
| Ethics approval(s) | Manchester Metropolitan University Arts & Humanities Faculty Research Ethics and Governance Committee, 14/02/2018, A&H1718-40 |
| Condition | Low attainment in GCSE Mathematics and excessive teacher workload |
| Intervention | This study seeks to examine whether pupils’ exposure to ED formative question setting platform and marking system for mathematics in secondary schools raises attainment in mathematics at GCSE. It also seeks to understand whether teacher workload, for teachers using ED for mathematics instruction at Years 10 and 11, is reduced. Whole schools will be assigned at random to either intervention (exposure to ED) or control conditions (non-exposure to ED) on a 1:1 basis. The intervention is an online question setting and diagnostic platform for instruction in mathematics at GCSE known as ED (previously known as diagnostic questions), developed by Eedi (https://www.eedi.co.uk/). The core components of the programme are as follows: 1. Schools assign formative assessment ‘quizzes’ aligned to their exam board’s Scheme of Work. Eedi instantly populates a calendar of multiple choice quizzes for the entire year. If they wish, teachers can manually adjust the ordering/scheduling of the quizzes. There are two quizzes per topic in the Scheme of Work. 2. Students complete quizzes each week according to the Scheme of Work set by their teacher. Each quiz comprises ten questions and each question has four multiple choice answers. Each incorrect answer is designed to diagnose a specific ‘misunderstanding’. Quizzes are marked through Eedi. Students are then prompted to review their answers with feedback targeting their specific misunderstanding. Students are also given targeted learning materials from Oxford University Press (OUP). 3. Teachers do not have to mark quizzes, Eedi marks them automatically. Teachers can review their class’s scores on Eedi, identify common misunderstandings and if necessary, send additional feedback to all students making that mistake. 4. Parents can receive automated text messages about new quizzes, students not completing quizzes as required, or what topics are being covered in class. More detailed reports are available by logging in to the Eedi website. The intervention will be delivered to school pupils entering Year 10 from September 2018 studying mathematics at GCSE, in around 90 English state secondary schools (intervention schools). The intervention will be delivered over two school years (Years 10 and 11 for this cohort). Training for schools is delivered by Eedi. Each school appoints a project lead to liaise with Eedi. Eedi works with the school project lead to assist them in setting up the school’s scheme of work on Eedi. Eedi provides two hours of training in person to the school’s maths department during the spring/summer terms 2018. This addresses the following: Context for this study and the importance of formative assessment Monitoring quizzes and whole class performance Giving feedback and setting further quizzes Parental alerts How to provide online access for all students How to use Eedi’s technical support Eedi provide additional training to school project leads, which covers: Troubleshooting and minor problems Access to online support Further support for ensuring access to Eedi for all students Liaison with Eedi’s account manager Schools have access to online technical support available 8am to 8pm, Monday to Friday. Eedi monitors usage of the Eedi platform and aims to proactively approach schools with the offer of support should problems become apparent. Parents, pupils and schools are given access to OUP materials that address misconceptions identified in Eedi throughout the life of the study (this is usually an additional paid for service, but for the trial free access has been granted to all intervention schools). Schools are provided with a grant of £1000 to cover the costs of text messaging associated with the operation of Eedi. Control schools are also offered an incentive of £1000 to take part and adhere to their responsibilities in the trial (as set out in the MOU), this will be paid at the end of trial period (July 2020). |
| Intervention type | Behavioural |
| Primary outcome measure | Attainment in GCSE Mathematics at the end of Year 11 |
| Secondary outcome measures | Teacher workload assessed by surveys of teachers responsible for delivering instruction in mathematics to the focal cohort of school pupils described above in both intervention and control schools. The survey will be administered online via email. Surveys will be administered online to teachers in both intervention and control schools at four points in time. The first of these occasions will be prior to the commencement of the intervention in the summer term 2018. Post randomisation teacher surveys will be administered in December 2018, March 2019 and March 2020. At each measurement occasion teachers will be asked to provide an estimate of the total amount of time they were engaged in the following tasks during a specified reference week: Setting maths homework Marking maths homework Giving individual feedback to students in mathematics Giving group feedback to students Lesson planning in mathematics From this information, the total mathematics related workload will be calculated as the total amount of time spent across these five dimensions per teacher per week. |
| Overall study start date | 01/01/2018 |
| Overall study end date | 31/07/2020 |
Eligibility
| Participant type(s) | Other |
|---|---|
| Age group | Child |
| Sex | Both |
| Target number of participants | A total of 180 schools will be recruited to the trial comprising approximately 30,000 pupils |
| Total final enrolment | 28930 |
| Participant inclusion criteria | Schools identified in UK Education Endowment Foundation databases, AQA and EdExcel examining board databases and through Eedi’s own data base of teachers are approached in order to gauge their interest in participating in the study. It was felt important to only admit schools to the trial where existing use of ED was either completely absent or minimal. However, ED is available to schools, teachers and pupils across England, making it difficult to find schools in which there is no existing use of the system. Once a school has signalled their interest, Eedi (the developers and implementers) assess existing usage of Eedi at the school through examining their administrative data records. These records indicate which schools, teachers and pupils have existing accounts on Eedi and the extent of their current usage. Only schools with 30 or fewer existing pupil accounts were considered for inclusion in the trial. This threshold was devised by the developers based upon their knowledge of existing usage and how best to measure it and is considered to be low existing usage. All schools with more intensive usage according to this criterion are excluded from the trial. It is important to note that schools can only enter the trial if they agree that access to Eedi is disabled completely for the duration of study should they be allocated to control conditions. This step is taken to avoid non-compliance among control schools. In each school, all pupils currently in Year 9 during the school year 2017/18, whose parents do not withdraw their child from the study are considered to be in range of intervention. Pupils in intervention schools will receive instruction in mathematics using Eedi for two school years – 2018/19 (Year 10) and 2019/20 (Year 11). |
| Participant exclusion criteria | Pupils whose parents opt to withdraw their child from the study are excluded from the study sample |
| Recruitment start date | 01/02/2018 |
| Recruitment end date | 31/07/2018 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centres
London
SE1 3ER
United Kingdom
London
SW1H 9NP
United Kingdom
Geoffrey Manton Building
Off Oxford Road
Manchester
M15 6LL
United Kingdom
10 Hulme Hall Road
Manchester
M15 4LY
United Kingdom
Sponsor information
Charity
9th Floor, Millbank Tower
21-24 Millbank
London
SW1P 4QP
United Kingdom
| Phone | 020 7802 1676 |
|---|---|
| info@eefoundation.org.uk | |
| Website | https://educationendowmentfoundation.org.uk/ |
"ROR" |
https://ror.org/03bhd6288 |
Funders
Funder type
Charity
Private sector organisation / Trusts, charities, foundations (both public and private)
- Alternative name(s)
- EducEndowFoundn, Education Endowment Foundation | London, EEF
- Location
- United Kingdom
Results and Publications
| Intention to publish date | 31/12/2021 |
|---|---|
| Individual participant data (IPD) Intention to share | Yes |
| IPD sharing plan summary | Stored in non-publicly available repository |
| Publication and dissemination plan | A main evaluation report and summary will be published by study sponsor the UK Education Endowment Foundation in its report series. |
| IPD sharing plan | The datasets generated during and/or analysed during the current study will be stored in a non-publically available repository known as the EEF Data Archive (https://educationendowmentfoundation.org.uk/projects-and-evaluation/evaluating-projects/evaluator-resources/submitting-your-data-to-the-fft-archive/). Evaluators will deposit project data with pupil identifiers so that further data linking can take place. Access to data is controlled by approval from the Education Endowment Foundation (EEF) and the Department for Education. Only anonymised data will be provided for secondary or meta-analysis from the EEF Data Archive. Retention of the data is determined by EEF as the data controller and where data include records from the UK National Pupil Database the UK Department for Education. |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Funder report results | 16/12/2021 | 28/10/2022 | No | No |
Editorial Notes
28/10/2022: Funder report reference corrected.
20/12/2021: The following changes have been made:
1. Funder report reference added.
2. The total final enrolment number has been added.
19/12/2018: Internal review.
