Evaluation of the 'ScratchMaths' computer programming educational intervention involving 110 primary schools in England
| ISRCTN | ISRCTN10189078 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN10189078 |
| Secondary identifying numbers | N/A |
- Submission date
- 05/10/2016
- Registration date
- 11/10/2016
- Last edited
- 19/12/2018
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English summary of protocol
Background and study aims
This study will evaluate the impact of teaching pupils programming with ‘Scratch’, a free online programming environment developed by MIT Media Lab. It enables children to programme by dragging and dropping code elements instead of typing them. Teaching coding enables children to learn through experimentation, mastering concepts such as computational thinking and logic. This study aims to develop pupil material and assessments for teachers to guide the teaching of the Scratch programme.
Who can participate?
Pupils in Year 5 in 2015/16 (aged 9-10)
What does the study involve?
Participating primary schools are randomly allocated to one of two groups. One group receive the 'Scratch' programme and the other group continue with their usual curriculum and are offered the Scratch programme after the study has ended. In the ‘Stratch’ programme teachers receive training and are provided with teaching materials and assessments to use during classes in Year 5 during the 2015/16 academic year and in Year 6 classes during the 2016/17 academic year. The pupils receive 'Scratch' lessons within their normal curriculum in Year 5 and Year 6. The impact of 'Scratch' is assessed using a computational thinking test at the end of 2015/16, and the Key Stage 2 mathematics results at the end of 2016/17.
What are the possible benefits and risks of participating?
The main benefits of participating will be for teachers and pupils to access and experience the 'Scratch' programme. This may have a positive impact on their mathematical understanding. This could then potentially benefit pupils more widely in other primary schools in England. The main risks relate to if no impact or a negative impact is observed. If this is the case, the curriculum time could have been used more fruitfully for the pupils involved in the Scratch programme.
Where is the study run from?
UCL Knowledge Lab (UK)
When is the study starting and how long is it expected to run for?
March 2014 to June 2017
Who is funding the study?
Education Endowment Foundation (EEF)
Who is the main contact?
Mark Boylan
Contact information
Public
Arundel Level 1
Sheffield
S1 1WB
United Kingdom
 ORCID ID |
0000-0002-8581-1886 |
|---|
Study information
| Study design | Interventional cluster randomised controlled trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Other |
| Participant information sheet | Not available in web format, please use the contact details to request a patient information sheet |
| Scientific title | Impact and process evaluation of 'ScratchMaths': a clustered randomised controlled trial to measure the impact of 'ScratchMaths' on computational thinking and mathematics attainment, and a process evaluation on how schools, teachers and pupils engage with 'ScratchMaths' to produce the measured impact |
| Study objectives | Training teachers in using Scratch Maths will lead to improved KS2 mathematics attainment in pupils. |
| Ethics approval(s) | Sheffield Hallam University faculty research ethics committee, 27/10/2014 |
| Health condition(s) or problem(s) studied | Computational Thinking and KS2 Mathematics attainment |
| Intervention | Between March 2014 and April 2015, a total of 110 primary schools were recruited to the trial. As part of this recruitment, for a school to become part of the trial sample they were required to provide details on teachers and pupils. Specifically, schools were required to provide detail on two named teachers who would be taking a Y5 class in the 2015/16 academic year and would participate in the 'ScratchMaths' CPD programme if the school was selected. Further, schools provided class lists for pupils who the two named teachers would teach in Y5 during the 2015/16 academic year. Using this pupil-level detail, baseline data on Key Stage 1 attainment, gender and Free School Meal status was obtained from the National Pupil Database (NPD). Using details from the 2013/14 Annual School Census (ASC) for all schools recruited to the Scratch evaluation, a stratification scheme was developed using propensity scores. Propensity scores provide a practical way of drawing on a greater number of explanatory variables within a stratification scheme. A logistic regression model was used to generate (predicted probability or propensity) scores based on a KS2 attainment outcome variable and seven explanatory variables. Within each of the hub areas, the propensity scores were used to group schools into their 'nearest statistical neighbour' pairs. One school from each pair was then randomly selected into the 'ScratchMaths' intervention (or wave 1) group, the remaining school was allocated to the control group. In all, 55 schools were randomly selected to receive the Scratch programming intervention within the trial period (wave 1) and their 55 pairs were allocated into the business as usual control group, being offered the Scratch programming intervention after the trial has ended (wave 2). Following randomisation, teachers in the 55 'ScratchMaths' schools began to receive the 'ScratchMaths' CPD programme. This involved attending training events and being provided with teaching materials and access to an online support resource. Teachers then drew on this training and materials within their Y5 classes in 2015/16 and their Y6 classes in 2016/17. Prior to the start of the trial, the secondary computational thinking outcome was developed in consultation with the team delivering 'ScratchMaths' (the Institute of Education). The computational thinking measure was developed from an international online project 'Beaver Computing'. The resulting measure comprised of 10 items which combine to create a measure that validly captured key aspects of computational thinking with good internal consistency (Cronbach Alpha = 0.77). The main impact analyses for the primary and secondary outcomes will adopt an intention to treat approach. Within the 55 intervention schools, a process evaluation collected quantitative and qualitative detail on school, teacher and pupil-level engagement with the 'ScratchMaths' intervention. Within the 55 control schools, the process evaluation collected quantitative and qualitative detail on any changes to the Y5 and Y6 curricula that related to computational thinking and programming. Additional impact analyses will draw on the data collected through the process evaluation. These exploratory analyses will focus on pupil-level impact in schools identified as having a specified minimum level of teacher and class level engagement with 'ScratchMaths' (i.e. on treatment analyses). Additionally, if the process evaluation reveals that some control schools introduced similar computational thinking/programming to Y5/Y6 pupils during the trial period, exploratory sensitivity analyses will be run excluding pupils from such schools. |
| Intervention type | Other |
| Primary outcome measure | Pupils' attainment in KS2 mathematics, measured using KS2 mathematics test scores at the end of Y6 in 2016/17 |
| Secondary outcome measures | Pupils' attainment on an online Computational Thinking test developed by the evaluators at the end of Y5 in 2015/16 |
| Overall study start date | 11/03/2014 |
| Completion date | 30/06/2017 |
Eligibility
| Participant type(s) | Other |
|---|---|
| Age group | Child |
| Lower age limit | 9 Years |
| Upper age limit | 10 Years |
| Sex | Both |
| Target number of participants | 4,400 pupils are clustered into 110 schools (55 intervention schools, 55 control schools). Two teachers from each school (220 overall, 110 in intervention schools, 110 in control schools). 4,400 pupils overall (40 pupils per school; 2,200 in intervention schools, 2,200 in control schools) |
| Key inclusion criteria | 1. Primary schools 2. Ideally with a two form entry 3. Provided teacher and pupil details before 30/04/2015 2. Pupils who were in Y5 in 2015/16 (aged 9-10) |
| Key exclusion criteria | Single gender schools |
| Date of first enrolment | 01/12/2014 |
| Date of final enrolment | 30/04/2015 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centres
University College London
23-29 Emerald Street
London
WC1N 3QS
United Kingdom
Sheffield
S1 1WB
United Kingdom
Sponsor information
Charity
Millbank Tower
21-24 Millbank
london
SW1P 4QP
United Kingdom
| Phone | +44 (0)207 802 1676 |
|---|---|
| Anneka.Dawson@eefoundation.org.uk | |
"ROR" |
https://ror.org/03bhd6288 |
Funders
Funder type
Charity
No information available
Results and Publications
| Intention to publish date | 01/06/2018 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Stored in repository |
| Publication and dissemination plan | The trial results will be published on the Educational Endowment Foundation website (https://educationendowmentfoundation.org.uk/). This is expected to be in summer 2018. |
| IPD sharing plan | All data will be stored on a secure server (Jalap) which only named researchers have access to – there is no weblink. Outcome data from the Computational Thinking Test will be matched with data extracted from the National Pupil Database (NPD); all data will be held in an anonymised format so no individual pupil can be identified. Opt-out consent was obtained from pupils/parents for data collection, linkage and storage. Outcome data is extracted from the NPD and provided in an anonymised format (no pupil names are stored in these data files). |
Study outputs
| Output type | Details | Date created | Date added | Peer reviewed? | Patient-facing? |
|---|---|---|---|---|---|
| Funder report results | results | No | No |
Editorial Notes
19/12/2018: Publication reference added.
