Plain English Summary
Background and study aims
This study will evaluate the impact of teaching pupils programming with ‘Scratch’, a free online programming environment developed by MIT Media Lab. It enables children to programme by dragging and dropping code elements instead of typing them. Teaching coding enables children to learn through experimentation, mastering concepts such as computational thinking and logic. This study aims to develop pupil material and assessments for teachers to guide the teaching of the Scratch programme.
Who can participate?
Pupils in Year 5 in 2015/16 (aged 9-10)
What does the study involve?
Participating primary schools are randomly allocated to one of two groups. One group receive the 'Scratch' programme and the other group continue with their usual curriculum and are offered the Scratch programme after the study has ended. In the ‘Stratch’ programme teachers receive training and are provided with teaching materials and assessments to use during classes in Year 5 during the 2015/16 academic year and in Year 6 classes during the 2016/17 academic year. The pupils receive 'Scratch' lessons within their normal curriculum in Year 5 and Year 6. The impact of 'Scratch' is assessed using a computational thinking test at the end of 2015/16, and the Key Stage 2 mathematics results at the end of 2016/17.
What are the possible benefits and risks of participating?
The main benefits of participating will be for teachers and pupils to access and experience the 'Scratch' programme. This may have a positive impact on their mathematical understanding. This could then potentially benefit pupils more widely in other primary schools in England. The main risks relate to if no impact or a negative impact is observed. If this is the case, the curriculum time could have been used more fruitfully for the pupils involved in the Scratch programme.
Where is the study run from?
UCL Knowledge Lab (UK)
When is the study starting and how long is it expected to run for?
March 2014 to June 2017
Who is funding the study?
Education Endowment Foundation (EEF)
Who is the main contact?
Mark Boylan
Trial website
Contact information
Type
Public
Primary contact
Mr Mark Boylan
ORCID ID
http://orcid.org/0000-0002-8581-1886
Contact details
Arundel Level 1
Sheffield
S1 1WB
United Kingdom
Additional identifiers
EudraCT number
ClinicalTrials.gov number
Protocol/serial number
N/A
Study information
Scientific title
Impact and process evaluation of 'ScratchMaths': a clustered randomised controlled trial to measure the impact of 'ScratchMaths' on computational thinking and mathematics attainment, and a process evaluation on how schools, teachers and pupils engage with 'ScratchMaths' to produce the measured impact
Acronym
Study hypothesis
Training teachers in using Scratch Maths will lead to improved KS2 mathematics attainment in pupils.
Ethics approval
Sheffield Hallam University faculty research ethics committee, 27/10/2014
Study design
Interventional cluster randomised controlled trial
Primary study design
Interventional
Secondary study design
Cluster randomised trial
Trial setting
Schools
Trial type
Other
Patient information sheet
Not available in web format, please use the contact details to request a patient information sheet
Condition
Computational Thinking and KS2 Mathematics attainment
Intervention
Between March 2014 and April 2015, a total of 110 primary schools were recruited to the trial. As part of this recruitment, for a school to become part of the trial sample they were required to provide details on teachers and pupils. Specifically, schools were required to provide detail on two named teachers who would be taking a Y5 class in the 2015/16 academic year and would participate in the 'ScratchMaths' CPD programme if the school was selected. Further, schools provided class lists for pupils who the two named teachers would teach in Y5 during the 2015/16 academic year. Using this pupil-level detail, baseline data on Key Stage 1 attainment, gender and Free School Meal status was obtained from the National Pupil Database (NPD).
Using details from the 2013/14 Annual School Census (ASC) for all schools recruited to the Scratch evaluation, a stratification scheme was developed using propensity scores. Propensity scores provide a practical way of drawing on a greater number of explanatory variables within a stratification scheme. A logistic regression model was used to generate (predicted probability or propensity) scores based on a KS2 attainment outcome variable and seven explanatory variables. Within each of the hub areas, the propensity scores were used to group schools into their 'nearest statistical neighbour' pairs. One school from each pair was then randomly selected into the 'ScratchMaths' intervention (or wave 1) group, the remaining school was allocated to the control group.
In all, 55 schools were randomly selected to receive the Scratch programming intervention within the trial period (wave 1) and their 55 pairs were allocated into the business as usual control group, being offered the Scratch programming intervention after the trial has ended (wave 2).
Following randomisation, teachers in the 55 'ScratchMaths' schools began to receive the 'ScratchMaths' CPD programme. This involved attending training events and being provided with teaching materials and access to an online support resource. Teachers then drew on this training and materials within their Y5 classes in 2015/16 and their Y6 classes in 2016/17.
Prior to the start of the trial, the secondary computational thinking outcome was developed in consultation with the team delivering 'ScratchMaths' (the Institute of Education). The computational thinking measure was developed from an international online project 'Beaver Computing'. The resulting measure comprised of 10 items which combine to create a measure that validly captured key aspects of computational thinking with good internal consistency (Cronbach Alpha = 0.77).
The main impact analyses for the primary and secondary outcomes will adopt an intention to treat approach.
Within the 55 intervention schools, a process evaluation collected quantitative and qualitative detail on school, teacher and pupil-level engagement with the 'ScratchMaths' intervention. Within the 55 control schools, the process evaluation collected quantitative and qualitative detail on any changes to the Y5 and Y6 curricula that related to computational thinking and programming.
Additional impact analyses will draw on the data collected through the process evaluation. These exploratory analyses will focus on pupil-level impact in schools identified as having a specified minimum level of teacher and class level engagement with 'ScratchMaths' (i.e. on treatment analyses). Additionally, if the process evaluation reveals that some control schools introduced similar computational thinking/programming to Y5/Y6 pupils during the trial period, exploratory sensitivity analyses will be run excluding pupils from such schools.
Intervention type
Other
Phase
Drug names
Primary outcome measure
Pupils' attainment in KS2 mathematics, measured using KS2 mathematics test scores at the end of Y6 in 2016/17
Secondary outcome measures
Pupils' attainment on an online Computational Thinking test developed by the evaluators at the end of Y5 in 2015/16
Overall trial start date
11/03/2014
Overall trial end date
30/06/2017
Reason abandoned (if study stopped)
Eligibility
Participant inclusion criteria
1. Primary schools
2. Ideally with a two form entry
3. Provided teacher and pupil details before 30/04/2015
2. Pupils who were in Y5 in 2015/16 (aged 9-10)
Participant type
Other
Age group
Child
Gender
Both
Target number of participants
4,400 pupils are clustered into 110 schools (55 intervention schools, 55 control schools). Two teachers from each school (220 overall, 110 in intervention schools, 110 in control schools). 4,400 pupils overall (40 pupils per school; 2,200 in intervention schools, 2,200 in control schools)
Participant exclusion criteria
Single gender schools
Recruitment start date
01/12/2014
Recruitment end date
30/04/2015
Locations
Countries of recruitment
United Kingdom
Trial participating centre
UCL Knowledge Lab
UCL Institute of Education
University College London
23-29 Emerald Street
London
WC1N 3QS
United Kingdom
Trial participating centre
Sheffield Hallam University
Arundel level 1
Sheffield
S1 1WB
United Kingdom
Sponsor information
Organisation
Education Endowment Foundation
Sponsor details
Millbank Tower
21-24 Millbank
london
SW1P 4QP
United Kingdom
+44 (0)207 802 1676
Anneka.Dawson@eefoundation.org.uk
Sponsor type
Charity
Website
Funders
Funder type
Charity
Funder name
Education Endowment Foundation
Alternative name(s)
Funding Body Type
Funding Body Subtype
Location
Results and Publications
Publication and dissemination plan
The trial results will be published on the Educational Endowment Foundation website (https://educationendowmentfoundation.org.uk/). This is expected to be in summer 2018.
IPD sharing plan
All data will be stored on a secure server (Jalap) which only named researchers have access to – there is no weblink. Outcome data from the Computational Thinking Test will be matched with data extracted from the National Pupil Database (NPD); all data will be held in an anonymised format so no individual pupil can be identified. Opt-out consent was obtained from pupils/parents for data collection, linkage and storage. Outcome data is extracted from the NPD and provided in an anonymised format (no pupil names are stored in these data files).
Intention to publish date
01/06/2018
Participant level data
Stored in repository
Basic results (scientific)
Publication list
2018 results in https://educationendowmentfoundation.org.uk/public/files/Projects/Evaluation_Reports/ScratchMaths.pdf