Improving Progress 8; Avoid Perverse Incentives

The first unvalidated Progress 8 scores will be finding their way into schools shortly.  They will trigger a series of emotions.  Progress 8 will always be imperfect; no single number or grade can ever hope to fully express everything a school does for its pupils.

Despite its imperfection, I think it is one of the better whole school accountability measures we have had.  There is now the gradual process of working with the measure to help drive the kinds of positive, inclusive behaviour that sits with its “every pupil, every grade” philosophy.  It is already evident that some schools will work hard to manipulate any performance measure to make themselves look good or outstanding; it would be foolish to think we all share an identical moral compass.

From a school with 70% Pupil Premium (Disadvantaged) Pupils. Contextualise for disadvantage and the Progress 8 is circa -0.08 (not bad at all)

Addressing Outliers

There is already a promise to look at the Progress 8 outliers; the few children whose extreme variation from what they statistically should attain to what they actually do has a disproportionate impact on the overall measure.  These children’s extreme lack of progress is rarely a product of their schooling; at one extreme life may have imploded at secondary school and at the other a much lower baseline than expected, due to circumstances at the time.  Education Datalab’s sensible proposal to remove all pupils whose Progress 8 score is +/-2.5 is one such suggestion.

Just to illustrate the point of perverse behaviours I want to imagine how this may be manipulated by an unscrupulous school.  At the higher end you don’t want a pupil to make so much progress that they attain 25 points or more above expectation; 26 points and they are removed from the calculation.  However, 24 points and they add a 2.4 to the Progress 8 score.  This would be very difficult for a schools and seems unlikely but imagine the other end of the spectrum where a pupil is beginning to significantly underachieve; there is a perverse incentive for them to do as badly as possible.  If they miss their expected attainment point score by 25 points or more they can then be removed from the calculation; I’m saddened to admit I can see some schools considering this reasonable collateral damage in their pursuit of excellence.  I wonder whether there is another way?

This method would also mean that low attaining pupils at the end of primary school would almost certainly be included in the calculation as their expected points total at GCSE is less than 25 points.  Some schools have so many more of these pupils.

Countering Off-rolling

The removing of pupils from a school’s roll prior to the census in January of Year 11, used as the basis for the number of pupils in the year group on performance tables, has long been a concern.  Alternative data tables suggest that certain multi academy trusts and schools have long used this unacceptable practice as a way to artificially inflate their results.

Headteachers’ Roundtable picked up Education Datalab’s suggestion of proportionate accountability; if a pupil is on your roll for three out of five years then 60% of their results you remain accountable for.  This adjustment could help drive inclusive behaviour and see the greater re-integration of pupils from Alternative Provision; when you take a pupil in Year 11 you aren’t held accountable for all their results.

However, a perverse behaviour would see schools permanently excluding pupils or moving them to elective home education earlier, to reduce their level of accountability.  The less time a pupil is in the school the less accountable it is.  It may be the proportionate accountability model needs to come with a few more teeth; you will be proportionately accountable for the whole key stage of any child who moves off your roll; any child permanently excluded or moved to elective home education you will be 100% accountable for.  Too harsh?  No system will produce a perfect Progress 8 but would this help align some leaders’ thinking with a greater moral purpose?  Any method used to counter off-rolling would need to be considered alongside how to deal with outliers.

Recognising Effectiveness; Contextualise Data

We currently have two standards for schools in England; one for advantaged pupils who statistically get a positive Progress 8 and one for disadvantaged pupils who collectively get a negative Progress 8.  Depending what your intake is like you’re either on Easier Street or Hard Miles Way.  Progress 8 seeks to be a school effectiveness measure and so context matters and needs to be taken into account.

Smoothing Out Volatility

The sooner we move to the headline figure being a three year rolling average the better in my books.  It could start as early as next year when three years worth of Progress 8 data becomes available for each school.  It may be argues that it should be delayed a year as schools should have due notice; I can live with that.  I can only see the positive in this.