//
you're reading...
Accountability, Assessment

Improving Progress 8; Avoid Perverse Incentives

The first unvalidated Progress 8 scores will be finding their way into schools shortly.  They will trigger a series of emotions.  Progress 8 will always be imperfect; no single number or grade can ever hope to fully express everything a school does for its pupils.

Despite its imperfection, I think it is one of the better whole school accountability measures we have had.  There is now the gradual process of working with the measure to help drive the kinds of positive, inclusive behaviour that sits with its “every pupil, every grade” philosophy.  It is already evident that some schools will work hard to manipulate any performance measure to make themselves look good or outstanding; it would be foolish to think we all share an identical moral compass.

From a school with 70% Pupil Premium (Disadvantaged) Pupils. Contextualise for disadvantage and the Progress 8 is circa -0.08 (not bad at all)

Addressing Outliers

There is already a promise to look at the Progress 8 outliers; the few children whose extreme variation from what they statistically should attain to what they actually do has a disproportionate impact on the overall measure.  These children’s extreme lack of progress is rarely a product of their schooling; at one extreme life may have imploded at secondary school and at the other a much lower baseline than expected, due to circumstances at the time.  Education Datalab’s sensible proposal to remove all pupils whose Progress 8 score is +/-2.5 is one such suggestion.

Just to illustrate the point of perverse behaviours I want to imagine how this may be manipulated by an unscrupulous school.  At the higher end you don’t want a pupil to make so much progress that they attain 25 points or more above expectation; 26 points and they are removed from the calculation.  However, 24 points and they add a 2.4 to the Progress 8 score.  This would be very difficult for a schools and seems unlikely but imagine the other end of the spectrum where a pupil is beginning to significantly underachieve; there is a perverse incentive for them to do as badly as possible.  If they miss their expected attainment point score by 25 points or more they can then be removed from the calculation; I’m saddened to admit I can see some schools considering this reasonable collateral damage in their pursuit of excellence.  I wonder whether there is another way?

This method would also mean that low attaining pupils at the end of primary school would almost certainly be included in the calculation as their expected points total at GCSE is less than 25 points.  Some schools have so many more of these pupils.

Countering Off-rolling

The removing of pupils from a school’s roll prior to the census in January of Year 11, used as the basis for the number of pupils in the year group on performance tables, has long been a concern.  Alternative data tables suggest that certain multi academy trusts and schools have long used this unacceptable practice as a way to artificially inflate their results.

Headteachers’ Roundtable picked up Education Datalab’s suggestion of proportionate accountability; if a pupil is on your roll for three out of five years then 60% of their results you remain accountable for.  This adjustment could help drive inclusive behaviour and see the greater re-integration of pupils from Alternative Provision; when you take a pupil in Year 11 you aren’t held accountable for all their results.

However, a perverse behaviour would see schools permanently excluding pupils or moving them to elective home education earlier, to reduce their level of accountability.  The less time a pupil is in the school the less accountable it is.  It may be the proportionate accountability model needs to come with a few more teeth; you will be proportionately accountable for the whole key stage of any child who moves off your roll; any child permanently excluded or moved to elective home education you will be 100% accountable for.  Too harsh?  No system will produce a perfect Progress 8 but would this help align some leaders’ thinking with a greater moral purpose?  Any method used to counter off-rolling would need to be considered alongside how to deal with outliers.

Recognising Effectiveness; Contextualise Data

We currently have two standards for schools in England; one for advantaged pupils who statistically get a positive Progress 8 and one for disadvantaged pupils who collectively get a negative Progress 8.  Depending what your intake is like you’re either on Easier Street or Hard Miles Way.  Progress 8 seeks to be a school effectiveness measure and so context matters and needs to be taken into account.

Smoothing Out Volatility

The sooner we move to the headline figure being a three year rolling average the better in my books.  It could start as early as next year when three years worth of Progress 8 data becomes available for each school.  It may be argues that it should be delayed a year as schools should have due notice; I can live with that.  I can only see the positive in this.

Advertisements

Discussion

5 thoughts on “Improving Progress 8; Avoid Perverse Incentives

  1. If the problem is removing these students why not cap it? Just say you can’t contribute more than 2.5 or less than -2.5…?

    Posted by Adam | September 24, 2017, 6:34 pm
    • I’d better just re-check the Education DataLab idea; maybe that’s what they were saying and I’ve misread it. Like this idea, thanks for adding.

      Posted by LeadingLearner | September 24, 2017, 6:36 pm
      • Yup I remember reading their piece and think this is what they said but can’t quite remember. To be fair the problem at the low end still remains in a theoretical sense. If the bulk of their (negative) progress is not due to the school then I guess it’s fair enough to remove completely. But the proof for that would have to be more than just their p8 score.

        Posted by Adam | September 24, 2017, 6:38 pm
  2. Hi,
    Can I just ask where you copied the table from- ASP?

    Posted by ancient-mariner | September 27, 2017, 11:53 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

New Book: Liminal Leadership

Enter your email address to follow this blog and receive notifications of new posts by email.

Join 21,269 other followers

Follow @LeadingLearner on WordPress.com

If you're interested in finding out more about how Mangahigh can support your Maths Curriculum please click me

The Teacher Development Trust a national charity supporting effective professional development (please click to find out more)

Blog Stats

  • 886,473 hits

COPYRIGHT LICENCE

#5MinMarkingPlan

It won't get your marking done in five minutes but will help your structure your marking so you and the students get the most out of it.

#5MinBehaviourPlan

Getting the behaviour of students right, in the class room, is the challenge for every teacher. This simple planner will take you through the main stages required to help you achieve a purposeful learning envionment

#Outstandingin10Plus10

Ideas and the thinking behind a professional development to help improve the quality of teaching and learning. Keep the learning tight, the lesson plan loose and focus on the learners

Planning to Get Behaviour Right: Research Plus Experience

The thinking behind the #5MinBehaviourPlan and our whole school approach to managing behaviour.

Education for Wisdom

Why the four knowledge dimensions (covering knowledge and skills) and a moral compass are at the heart of an outstanding education.

When Feedback Met Bloom

A look at the power of Feedback to students using Bloom's four knowledge dimensions

%d bloggers like this: