| Abstract | The National Tutoring Programme (NTP) Academic Mentoring (AM) programme (2020/21) was designed to help disadvantaged pupils ‘catch up’ on missed learning by providing trained academic mentors to deliver one to one and small group tutoring in schools. This evaluation covers year 1 of the AM programme as delivered by Teach First from November 2020 to July 2021 (delivery was in three waves starting 26th October 2020, 15th January 2021 and 22nd February 2021). AM was one arm of the NTP. The NTP aimed to support teachers and schools in providing a sustained response to the Covid-19 pandemic and to provide a longer -term contribution to closing the attainment gap between disadvantaged pupils and their peers. The NTP was part of a wider government response to the pandemic, funded by the Department for Education (DfE) and was originally developed by the Education Endowment Foundation (EEF), Nesta, Impetus, The Sutton Trust, Teach First, and with the support of the KPMG Foundation. The DfE appointed Teach First to manage the provision of mentors (referred to as ‘academic mentors’) to schools; recruiting, training and placing them in schools. The mentor worked in the school setting as an employee of the school. It was expected that each academic mentor would work with at least 50 pupils between the date they started in school and the end of the academic year. Mentoring was provided online and/or face-to-face; and was one to one, or in groups of 2-4 pupils; and available in English/literacy, maths, science, humanities, and modern foreign languages. Mentoring was expected to be delivered in schools during normal teaching time, as well as before or after school. In certain circumstances, mentoring could be delivered online with pupil(s) at home. The AM programme was targeted at state-maintained primary and secondary schools serving disadvantaged populations. 89% of the schools met Teach First’s priority criteria, which is based on the proportion of children living in income deprived families (IDACI) and whether the school is in an area of chronic and persistent underperformance (AEA). The remaining 11% of schools had an above average proportion of pupils eligible for Pupil Premium (Teach First, 2021). Participating schools could decide which pupils received support from academic mentors. However, the programme encouraged them to select pupils from disadvantaged households or those whose education had been disproportionately impacted by Covid-19. Pupils in Years 1–11 were eligible (5–16 years old). The programme aimed to reach a minimum of 900 schools and 50,000 children, with 1,000 academic mentors. By the end of February 2021, it had surpassed targets having trained and placed 1,124 academic mentors in 946 schools and delivered mentoring sessions to 103,862 pupils, 49% of whom were identified by mentors as being eligible for Pupil Premium of Free School Meals (FSM), and 23% of whom were identified as having a special educational need or disability. The AM programme was initiated and delivered at a time of great pressure for schools when the education system had been disrupted by a series of school closures to most pupils and was contending with ongoing widespread pupil and staff absences. Covid-19 related issues disrupted the anticipated operation of academic mentoring during the year. The AM programme involved initial training and ongoing support from Teach First as intended but there was greater variation in schools’ deployment of mentors during the latter stages of the Autumn Term 2020/21, and during the January to March 2021 period of school closures to most pupils.
This evaluation report presents the analysis of the impact of the AM programme on maths and English attainment outcomes for Year 11 pupils only—who represent a very small proportion of individuals targeted by the AM programme. Originally, it was planned to evaluate impact across all year groups (Years 1 – 11) at primary and secondary level using schools’ standardised assessment data from Renaissance Learning (RL) assessments and, in addition, to evaluate the impact for Year 6 pupils using Key Stage (KS) 2 data. However, these analyses could not go ahead as KS2 assessments were cancelled in summer 2021 (related to the ongoing Covid-19 pandemic) and because the number of schools providing agreement to use their RL data was insufficient to warrant impact analyses. Data was only available for pupils in Year 11. Since GCSEs could not go ahead as planned in 2021, the data was in the form of Teacher Assessed Grades (TAGs), which had not previously been used as an outcome measurement tool. Checks were therefore undertaken to explore if TAGs would be suitable as an outcome measure. The only analysis that could proceed was therefore exploratory. The evaluation uses a quasi-experimental design (QED), in which a group of secondary schools and Year 11 pupils who did not receive the AM programme were selected for comparison with schools and pupils who received the AM programme. Comparison schools were selected by matching schools that were similar in important, observable regards to the schools that participated in AM. The evaluation included analysis on the availability of AM for pupils who were eligible for Pupil Premium (a key focus of the overall NTP), and all pupils, as these groups could be identified for both the AM and non-AM schools. In addition, the evaluation aimed to analyse the impact on pupils who received AM by predicting their participation and identifying a comparison group of pupils with similar characteristics. Analysis was based on data about Year 11 pupils’ attainment and characteristics from the National Pupil Database (NPD) merged with data provided by Teach First about pupils’ participation in AM. In total, 159 AM schools (8,977 Year 11 pupils eligible for Pupil Premium) and an equal number of comparison schools (8,419 Year 11 pupils eligible for Pupil Premium) were included in the final analysis. The evaluation assessed impact in English and maths using Teacher Assessed Grades (TAGs) from 2021. Where appropriate, this impact evaluation refers to important implementation features from the implementation and process evaluation (IPE) conducted by Teach First themselves. However, there is no independent IPE data to draw on in the interpretation of the impact results. Of the Year 11 pupils selected for Academic Mentoring in this evaluation, 46% of them were eligible for Pupil Premium, however, despite this it is important to note that the number of Year 11 Pupil Premium-eligible pupils selected for AM in AM schools was small as a proportion of all Year 11 Pupil Premium-eligible pupils, and the number of these Year 11 Pupil Premium-eligible pupils receiving AM in maths and/or English (as opposed to other subjects), was smaller still. The same is the case when considering the whole year group of Year 11 pupils – the number receiving AM was small as a proportion of all Year 11 pupils. This means that in the analysis, the number of Year 11 pupils who actually received AM in maths and/or English was heavily ‘diluted’ by the number of pupils who did not. The primary impact findings must be therefore treated with a high degree of caution. The analysis was subject to very high dilution; a large proportion of the pupils eligible for Pupil Premium included in the analysis in AM schools were not selected for AM. This was due to limited programme reach and a tendency for teachers to allocate both non-Pupil Premium and Pupil Premium eligible pupils to the programme. This dilution means that, in order to detect an effect, either the effect would need to be very strong amongst the very small proportion of Year 11 pupils eligible for Pupil Premium who were selected for mentoring (and there was no indication that this was the case elsewhere in our analysis), and/or there would need to be strong spillover effects amongst the rest of the Year 11 pupils eligible for Pupil Premium. Although the programme Theory of Change includes such a mechanism, it is unlikely to be relevant at the dilution levels seen. With such high dilution, it is hard to detect whether AM had an effect on those who received mentoring in the analyses focusing on pupils eligible for Pupil Premium and on all pupils. It is not possible to conclude whether a lack of observed impact is due to the small proportion of disadvantaged pupils who received mentoring, or because AM did not work for those who received it. An additional challenge was that it was not possible to construct a comparison group of similar Year 11 pupils in nonAM to schools to those who received mentoring in AM schools, based on observable, pupil-level characteristics, and this impact analysis did not go ahead. Schools used information such as classroom assessments to select pupils into the programme that was not observable in the available datasets, suggesting that pupil-level selection was driven by unobserved dimensions. These constraints, both of very high dilution and not being able to identify a comparison group with similar pupil characteristics, mean that the evaluation is unable to conclude, with any certainty, whether or not AM had an impact on the English or mathematics attainment outcomes of those pupils who received it. The report must be considered in the light of these caveats.
Year 11 pupils eligible for Pupil Premium in schools that received AM made, on average, similar progress in English compared to Year 11 pupils eligible for Pupil Premium in comparison schools (there was no evidence of an effect). In maths, Year 11 pupils eligible for Pupil Premium in schools that received AM made, on average, slightly more progress (equivalent to 1 months’ additional progress) compared to Year 11 pupils eligible for Pupil Premium in comparison schools. However, there is uncertainty around this result; it is also consistent with a null (0 months) effect or an effect of slightly larger than 1 month’s additional progress. A particular challenge in interpretation is that, on average, only 13% of Year 11 pupils eligible for Pupil Premium were selected for mentoring by schools, and only 4.2% of Year 11 pupils eligible for Pupil Premium were selected for mentoring in maths and 2.9% in English, meaning that the vast majority of pupils eligible for Pupil Premium included in the analysis did not receive mentoring. Therefore, this estimated impact of AM is severely diluted and it is unlikely any of these differences were due to AM. When looking at all Year 11 pupils, pupils in schools that received AM made, on average, similar progress in English and maths compared to all Year 11 pupils in comparison schools (there was no evidence of an effect). However, this finding was similarly subject to severe dilution: on average only 10% of Year 11 pupils in the analysed schools were selected for mentoring, with 3.4% in maths and 2.1% in English, and therefore it is hard to detect any effect that may (or may not) have been present. Within schools that offered AM to Year 11 pupils, there was no association between the number of completed mentoring sessions in maths and Year 11 outcomes in maths, or between the number of completed mentoring sessions in English and Year 11 outcomes in English. These results are associations and not necessarily causal. |
|---|