Significant Change to Progress 8 for 2018; Outliers are Out

In January 2018, the Department for Education released their latest update on Secondary School Accountability measures.  It seems a number of people missed the significant change to Progress 8 that a number of groups, including Headteachers’ Roundtable, had been calling for. 

Whilst all schools have their Progress 8 score affected by negative outliers; those with particularly small cohorts are disproportionately affected.  The text in italics below is taken straight from the January 2018 publication.

Pupils with extremely negative progress scores – change from 2018

“The department has received some feedback about the disproportionate effect that a small number of extremely negative scores can have on a school’s average. Such extreme cases tend to occur where a pupil was a middle or high achiever at key stage 2 and goes on to achieve much worse at key stage 4, for reasons beyond the control of the school (e.g. long term illness), than the national average for others with similar prior attainment. This has created concerns that in a small number of cases some progress scores calculated for individual pupils can be so largely negative that they can distort the overall picture of performance for a school.

The department has listened to this feedback and is refining the methodology for 2018 in order to reduce the disproportionate impact of the most extreme pupil level progress scores only. The refinement we are making introduces a limit on how negative a pupil’s progress score can be when calculating the school average. These pupil progress scores will still be large negative scores (to reflect that the pupils have made much less progress than other pupils in the same prior attainment group as them), but the disproportionate effect these have on a school’s score will be reduced.

We are not setting a maximum limit on how positive a pupil’s progress score can be as there are much smaller numbers of extremely positive progress scores that have a disproportionate impact than extremely negative ones progress score can be when calculating the school average.

The limit will mean that there is a minimum progress score that is assigned to pupils within the prior attainment groups (PAGs) where extremely negative scores exist. Where a pupil’s score is more negative than the minimum score, the minimum score will replace the pupil’s original progress score when calculating a school’s Progress 8 average. The minimum score for each PAG will be determined based on the variation in pupil progress scores for pupils across the country within that PAG (as measured by the standard deviation). The minimum scores will be fixed at a set number of standard deviations below the mean so that approximately 1% of pupils are identified nationally (we anticipate this will normally be no more than 1 or 2 pupils in any school). As such, predicting which pupils will, and will not, have their score affected by this methodology change, in advance of Progress 8 scores being made available, will not be possible. The exact minimum progress scores will be confirmed in the autumn, once we have the 2018 progress data.

Rather confusingly the School Performance Tables will display two Progress 8 figures; one with the negative outliers removed and one with the negative outliers in.  I’m not convinced this is necessary, helpful or useful in terms of transparency.  It will probably confuse more people that it helps.  Despite this the limits placed on negative outliers is to be welcomed; not often I do this but *tips his hat to the DfE*.  We appreciate you listening and making the necessary changes.

Now we need to maintain the dialogue to ensure off rolled pupils are proportionately placed back onto a school’s roll and a three year contextualised rolling average is agreed as the headline school effectiveness measure.