Condition category
Not Applicable
Date applied
05/10/2016
Date assigned
11/10/2016
Last edited
17/10/2016
Prospective/Retrospective
Retrospectively registered
Overall trial status
Ongoing
Recruitment status
No longer recruiting

Plain English Summary

Background and study aims
This study will evaluate the impact of teaching pupils programming with ‘Scratch’, a free online programming environment developed by MIT Media Lab. It enables children to programme by dragging and dropping code elements instead of typing them. Teaching coding enables children to learn through experimentation, mastering concepts such as computational thinking and logic. This study aims to develop pupil material and assessments for teachers to guide the teaching of the Scratch programme.

Who can participate?
Pupils in Year 5 in 2015/16 (aged 9-10)

What does the study involve?
Participating primary schools are randomly allocated to one of two groups. One group receive the 'Scratch' programme and the other group continue with their usual curriculum and are offered the Scratch programme after the study has ended. In the ‘Stratch’ programme teachers receive training and are provided with teaching materials and assessments to use during classes in Year 5 during the 2015/16 academic year and in Year 6 classes during the 2016/17 academic year. The pupils receive 'Scratch' lessons within their normal curriculum in Year 5 and Year 6. The impact of 'Scratch' is assessed using a computational thinking test at the end of 2015/16, and the Key Stage 2 mathematics results at the end of 2016/17.

What are the possible benefits and risks of participating?
The main benefits of participating will be for teachers and pupils to access and experience the 'Scratch' programme. This may have a positive impact on their mathematical understanding. This could then potentially benefit pupils more widely in other primary schools in England. The main risks relate to if no impact or a negative impact is observed. If this is the case, the curriculum time could have been used more fruitfully for the pupils involved in the Scratch programme.

Where is the study run from?
UCL Knowledge Lab (UK)

When is the study starting and how long is it expected to run for?
March 2014 to June 2017

Who is funding the study?
Education Endowment Foundation (EEF)

Who is the main contact?
Mark Boylan

Trial website

Contact information

Type

Public

Primary contact

Mr Mark Boylan

ORCID ID

http://orcid.org/0000-0002-8581-1886

Contact details

Arundel Level 1
Sheffield
S1 1WB
United Kingdom

Additional identifiers

EudraCT number

ClinicalTrials.gov number

Protocol/serial number

N/A

Study information

Scientific title

Impact and process evaluation of 'ScratchMaths': a clustered randomised controlled trial to measure the impact of 'ScratchMaths' on computational thinking and mathematics attainment, and a process evaluation on how schools, teachers and pupils engage with 'ScratchMaths' to produce the measured impact

Acronym

Study hypothesis

Training teachers in using Scratch Maths will lead to improved KS2 mathematics attainment in pupils.

Ethics approval

Sheffield Hallam University faculty research ethics committee, 27/10/2014

Study design

Interventional cluster randomised controlled trial

Primary study design

Interventional

Secondary study design

Cluster randomised trial

Trial setting

Schools

Trial type

Other

Patient information sheet

Not available in web format, please use the contact details to request a patient information sheet

Condition

Computational Thinking and KS2 Mathematics attainment

Intervention

Between March 2014 and April 2015, a total of 110 primary schools were recruited to the trial. As part of this recruitment, for a school to become part of the trial sample they were required to provide details on teachers and pupils. Specifically, schools were required to provide detail on two named teachers who would be taking a Y5 class in the 2015/16 academic year and would participate in the 'ScratchMaths' CPD programme if the school was selected. Further, schools provided class lists for pupils who the two named teachers would teach in Y5 during the 2015/16 academic year. Using this pupil-level detail, baseline data on Key Stage 1 attainment, gender and Free School Meal status was obtained from the National Pupil Database (NPD).
Using details from the 2013/14 Annual School Census (ASC) for all schools recruited to the Scratch evaluation, a stratification scheme was developed using propensity scores. Propensity scores provide a practical way of drawing on a greater number of explanatory variables within a stratification scheme. A logistic regression model was used to generate (predicted probability or propensity) scores based on a KS2 attainment outcome variable and seven explanatory variables. Within each of the hub areas, the propensity scores were used to group schools into their 'nearest statistical neighbour' pairs. One school from each pair was then randomly selected into the 'ScratchMaths' intervention (or wave 1) group, the remaining school was allocated to the control group.
In all, 55 schools were randomly selected to receive the Scratch programming intervention within the trial period (wave 1) and their 55 pairs were allocated into the business as usual control group, being offered the Scratch programming intervention after the trial has ended (wave 2).
Following randomisation, teachers in the 55 'ScratchMaths' schools began to receive the 'ScratchMaths' CPD programme. This involved attending training events and being provided with teaching materials and access to an online support resource. Teachers then drew on this training and materials within their Y5 classes in 2015/16 and their Y6 classes in 2016/17.
Prior to the start of the trial, the secondary computational thinking outcome was developed in consultation with the team delivering 'ScratchMaths' (the Institute of Education). The computational thinking measure was developed from an international online project 'Beaver Computing'. The resulting measure comprised of 10 items which combine to create a measure that validly captured key aspects of computational thinking with good internal consistency (Cronbach Alpha = 0.77).
The main impact analyses for the primary and secondary outcomes will adopt an intention to treat approach.
Within the 55 intervention schools, a process evaluation collected quantitative and qualitative detail on school, teacher and pupil-level engagement with the 'ScratchMaths' intervention. Within the 55 control schools, the process evaluation collected quantitative and qualitative detail on any changes to the Y5 and Y6 curricula that related to computational thinking and programming.
Additional impact analyses will draw on the data collected through the process evaluation. These exploratory analyses will focus on pupil-level impact in schools identified as having a specified minimum level of teacher and class level engagement with 'ScratchMaths' (i.e. on treatment analyses). Additionally, if the process evaluation reveals that some control schools introduced similar computational thinking/programming to Y5/Y6 pupils during the trial period, exploratory sensitivity analyses will be run excluding pupils from such schools.

Intervention type

Other

Phase

Drug names

Primary outcome measures

Pupils' attainment in KS2 mathematics, measured using KS2 mathematics test scores at the end of Y6 in 2016/17

Secondary outcome measures

Pupils' attainment on an online Computational Thinking test developed by the evaluators at the end of Y5 in 2015/16

Overall trial start date

11/03/2014

Overall trial end date

30/06/2017

Reason abandoned

Eligibility

Participant inclusion criteria

1. Primary schools
2. Ideally with a two form entry
3. Provided teacher and pupil details before 30/04/2015
2. Pupils who were in Y5 in 2015/16 (aged 9-10)

Participant type

Other

Age group

Child

Gender

Both

Target number of participants

4,400 pupils are clustered into 110 schools (55 intervention schools, 55 control schools). Two teachers from each school (220 overall, 110 in intervention schools, 110 in control schools). 4,400 pupils overall (40 pupils per school; 2,200 in intervention schools, 2,200 in control schools)

Participant exclusion criteria

Single gender schools

Recruitment start date

01/12/2014

Recruitment end date

30/04/2015

Locations

Countries of recruitment

United Kingdom

Trial participating centre

UCL Knowledge Lab
UCL Institute of Education University College London 23-29 Emerald Street
London
WC1N 3QS
United Kingdom

Trial participating centre

Sheffield Hallam University
Arundel level 1
Sheffield
S1 1WB
United Kingdom

Sponsor information

Organisation

Education Endowment Foundation

Sponsor details

Millbank Tower
21-24 Millbank
london
SW1P 4QP
United Kingdom
+44 (0)207 802 1676
Anneka.Dawson@eefoundation.org.uk

Sponsor type

Charity

Website

Funders

Funder type

Charity

Funder name

Education Endowment Foundation

Alternative name(s)

Funding Body Type

Funding Body Subtype

Location

Results and Publications

Publication and dissemination plan

The trial results will be published on the Educational Endowment Foundation website (https://educationendowmentfoundation.org.uk/). This is expected to be in summer 2018.

IPD sharing plan
All data will be stored on a secure server (Jalap) which only named researchers have access to – there is no weblink. Outcome data from the Computational Thinking Test will be matched with data extracted from the National Pupil Database (NPD); all data will be held in an anonymised format so no individual pupil can be identified. Opt-out consent was obtained from pupils/parents for data collection, linkage and storage. Outcome data is extracted from the NPD and provided in an anonymised format (no pupil names are stored in these data files).

Intention to publish date

01/06/2018

Participant level data

Stored in repository

Results - basic reporting

Publication summary

Publication citations

Additional files

Editorial Notes