Visible classroom evaluation
| ISRCTN | ISRCTN14774597 |
|---|---|
| DOI | https://doi.org/10.1186/ISRCTN14774597 |
| Secondary identifying numbers | N/A |
- Submission date
- 05/11/2016
- Registration date
- 30/11/2016
- Last edited
- 04/04/2023
- Recruitment status
- No longer recruiting
- Overall study status
- Completed
- Condition category
- Other
Plain English Summary
Background and study aims
There is a significant amount of evidence to suggest that timely provision of feedback improves learning. ‘Visible Classroom’ (VC) is an app and training programme that aims to encourage teachers to reflect on their teaching and develop their classroom practice. The app aims to improve student achievement by supporting teachers’ professional practice development by using personalised feedback on teaching. This feedback is designed to encourage teachers to reflect critically on their teaching and develop their classroom practice, which would be expected to have flow on effects for their pupils’ learning and attainment. The aim of the study is to test whether the VC application improves student attainment in English and Maths.
Who can participate?
Up to 140 primary schools in the UK that have at least one year 5 and one year 6 class, as long as they have not previously used the VC application.
What does the study involve?
Schools participating in the trial are randomly allocated to one of two groups. In schools in the first group, every term, primary school teachers record their normal lesson practice, and upload audio files to a website. After every upload, Ai Media, a captioning company, provides a verbatim transcript of the teacher’s talk. Alongside the word-for-word transcript will be simple analytics (e.g. speed of instruction, quantity of teacher vs student talk) a few hours after the lesson. After several hours of lessons have been uploaded, teachers receive a more detailed report, written by experts at the University of Melbourne. This will identify areas for improvement in their teaching, based on their self-selected teaching goals and the evidence from their transcripts. Teachers are expected to discuss the feedback with a mentor – another teacher at their school. In schools in the second group are expected to continue with business as usual. At the end of the school year, pupils results in KS2 exams are examined to compare attainment between the groups.
What are the possible benefits and risks of participating?
The main potential benefit of participating in the trial is the opportunity to improve pupil’s academic attainment. Even for those schools allocated to the control group, these schools still have the opportunity to benefit from the programme once the study is complete. There are no notable risks involved in participating.
Where is the study run from?
The study is being run by The Schools, Students and Teachers network (SSAT) and takes place in 140 schools from across England (UK)
When is the study starting and how long is it expected to run for?
January 2016 to November 2018
Who is funding the study?
Education Endowment Foundation (UK)
Who is the main contact?
Mr Daniel Carr
Contact information
Public
4 Matthew Parker Street
Behavioural Insights Team
London
SW1H 9NP
United Kingdom
Study information
| Study design | Single-blind cluster randomised controlled trial |
|---|---|
| Primary study design | Interventional |
| Secondary study design | Cluster randomised trial |
| Study setting(s) | School |
| Study type | Other |
| Participant information sheet | https://drive.google.com/file/d/0B6asyPNMbWTySWZ3NmU1MmRtMlE/view |
| Scientific title | Does providing feedback to teachers on their audio recorded lessons has an impact on student outcomes for KS2 results in maths and English? |
| Study acronym | VC |
| Study hypothesis | Use of the Visible Classroom app and associated mentory increases the educational attainment of Y5 and Y6 students (age 9-11) in combined KS2 maths and English scores. |
| Ethics approval(s) | University of Melbourne Humanities and Applied Sciences Human Ethics Sub-Committee, 15/09/2016, ref: 1647332 |
| Condition | Academic attainment |
| Intervention | The trial is a two-arm, school-level cluster randomised trial. The two arms are (1) ‘business as usual’ control arm, and (2) the Visible Classroom intervention. Approximately 140 schools will be recruited to this trial. Following recruitment of schools, schools will be randomised to either the control or treatment arm. This will be conducted using data analysis and statistical software Stata. The randomisation will follow a two-stage process: 1. The schools will be stratified on the basis of the proportion of FSM students (split across the median sample proportion), 2010-11 Key Stage 1 (KS1) Average Point Score (split across the median sample score), and whether schools were offered entry into the trial on a capped or uncapped basis. This last stratification covariate is included due to a decision after school recruitment began to begin capping the number of teachers per school who could be involved in the trial. This decision was made to avoid a cost overrun in the delivery team’s budget. As capped schools may select which teachers can be involved in this trial on the basis of merit or some other element correlated with student attainment, we made the decision to include this covariate in our stratification. 2. A random number will be generated within each block to ensure that the proportion of FSM students, KS1 results, and number of and capped and uncapped schools are balanced across trial arms. We will use data from DfE’s Performance Tables to determine the blocking characteristics, and SSAT will note which schools are offered capped entry during recruitment. The randomisation will occur following recruitment and after baseline data collection. The Stata code used for the randomisation will be recorded in the final report. Control group: The control or ‘business as usual’ arm will not have any intervention provided. Intervention group: The intervention, or Visible Classroom (VC) arm, involves teachers audio recording lessons and receiving detailed feedback on their teaching practices. Every term, primary school teachers will record their normal lesson practice, and upload audio files to a website. After every upload, Ai Media, a captioning company, will provide a verbatim transcript of the teacher’s talk. Alongside the word-for-word transcript will be simple analytics (e.g. speed of instruction, quantity of teacher vs student talk) a few hours after the lesson. After several hours of lessons have been uploaded, teachers will receive a more detailed report, written by experts at the University of Melbourne. This will identify areas for improvement in their teaching, based on their self-selected teaching goals and the evidence from their transcripts. Teachers will be expected to discuss the feedback with a mentor – another teacher at their school. The intervention programme will be delivered over the school academic year (approximately 7 months), with follow up for outcome measure collection occurring at approximately 8 months post randomisation. |
| Intervention type | Behavioural |
| Primary outcome measure | Combined maths and English KS 2 results for Year 5 and 6 is measured using results from national Key Stage 2 examinations. If results are consistent, the analysis will be done on the combined Y5 & Y6 data. Taking into account potential issues of implementation for Y6, as it is a ‘high-stakes’ year, we will alternatively report two headline findings recorded one year apart for Y5 and Y6 individually. For the year 6 cohort the outcome measure collection will come 7 months after randomisation and the intervention begins. For the year 5 cohort, who sit their KS2 exams one year later, outcome measures will be collected 19 months after randomisation. Analysis of both outcomes will incorporate a pre-randomisation record of student results utilising KS1 teacher assessments, which are held on record for each participant. |
| Secondary outcome measures | These secondary measures assess student results in KS2 exams on a year level and subject specific basis (for a total of 4 outcomes). The time of measurement is the same as for the primary outcome measures. 1. KS2 English results for Y5 2. KS2 English results for Y6 3. KS2 maths results for Y5 4. KS2 maths results for Y6 2. FSM and teacher subgroup analysis For FSM students and for the teacher subgroup analysis, the empirical strategy (and outcome measures) will remain the same with the addition of interaction analysis considering the interaction between the intervention and the FSM students and teachers who are deemed high on adherence to the rubric at baseline 3. Dosage analysis (IV approach) of the combined time the teacher spends recording lessons and using the online feedback which is available for teachers on their online dashboard is measured using survey responses collected in May 2017 4. Way that self-reported changes in teaching practice moderate any effect on attainment are assessed using the survey of teacher practice in May 2017 |
| Overall study start date | 06/01/2016 |
| Overall study end date | 30/11/2018 |
Eligibility
| Participant type(s) | Other |
|---|---|
| Age group | Child |
| Sex | Both |
| Target number of participants | Target recruitment number for this study is 140 schools, with a predicted average student cohort of 42 students per year level. As this trial will involve both 2016-17 year 5 and 6 students, this would mean an average of 84 students per school, for a total of 11,760 participants. |
| Participant inclusion criteria | School inclusion criteria: 1. Upfront transfer of data to the evaluators including eligible student UPNs 2. A completed MOU 3. Agreement that teachers in both the VC intervention and control arm complete a survey at the end of the trial period 4. Inclusion of both Y5 & Y6 in the intervention, with a minimum of two teachers providing baseline data and attending the training session 5. Access to an iPad/smartphone and sufficient internet connection to upload recording (internet connection to be verified at baseline data gathering) 6. The schools must not be using Visible Learning plus, a similar intervention designed by the University of Melbourne Project Team |
| Participant exclusion criteria | 1. School has less than two teachers across years 5 and 6 2. School is not able to send staff to the proposed training dates should they be assigned to the treatment group 3. School staff have access to a tablet or mobile device that is capable of downloading the ‘Visible Classroom’ application 4. The school must not have previous experience using the ‘Visible Classroom’ application |
| Recruitment start date | 01/06/2016 |
| Recruitment end date | 28/10/2016 |
Locations
Countries of recruitment
- England
- United Kingdom
Study participating centre
142 Central Street
London
EC1V 8AR
United Kingdom
Sponsor information
Research organisation
4 Matthew Parker Street
London
SW1H 9NP
United Kingdom
| Website | behaviouralinsights.co.uk |
|---|---|
"ROR" |
https://ror.org/03mk5b468 |
Funders
Funder type
Charity
No information available
Results and Publications
| Intention to publish date | 30/11/2018 |
|---|---|
| Individual participant data (IPD) Intention to share | No |
| IPD sharing plan summary | Not expected to be made available |
| Publication and dissemination plan | Planned publication of an evaluation report. |
| IPD sharing plan | Participant level data is quite sensitive, as it contains personal and academic information that could be used to identify trial participants. As such, this data will not be made publicly available. However, participant level data will be transferred and stored with the Education Endowment Foundation (the project funder) and (in an anonymised form) the UK Data Archive. |
Editorial Notes
04/04/2023: Internal review.
