Progressing Progress 8

Progress 8 has been designed as a way of holding schools accountable for the quality of education they offer; it’s about school effectiveness and shouldn’t be viewed at an individual pupil level outside of this context. 

To illustrate this point; imagine a child is way below the standard of other pupils when they leave primary school.  They make the expected progress at secondary schools; however, leave secondary school still way below the expected standard.  The school has arguably done a reasonable job but for the individual pupil we need to aspire to do so much more.  Attainment outcomes are what matter to individual pupils.  These are their passport to future education and career opportunities.  Sadly, frustrated by a world of comparable outcomes; one child attaining highly means another must attain lowly.

Acknowledgement: Lots of my thinking is informed by reading Education Datalab’s publications. Thanks to Dr Becky Allen and her team.

Tweaking Progress 8

Progress 8, with its focus on every pupil, every grade across a reasonably broad curriculum is the best secondary school headline accountability measure, I have experienced as a school leader.  Not perfect but  miles better than 5+A*-CEM.  We should continue to use Progress 8 as the main metric for a secondary school’s effectiveness but here are a few tweaks that I’d like to see:

We need to contextualise Progress 8 otherwise we are artificially inflating the effectiveness of schools with more affluent intakes and inappropriately damning schools that have more disadvantaged ones.  You’d need to contextualise for ethnicity and English as an additional language (EAL) plus potentially other factors to get a true contextualised value added score.

If we don’t contextualise progress we run the danger of someone (puts hand up) establishing a Free School for Chinese Girls from affluent families who enter the country in Year 4 with no English.  The school’s Progress 8 score would be unbelievable; its leader would undeservedly stand in the reflected glory.  The progress would not simply be due to the quality of the education provided by the school: EAL pupils’ progress accelerates as their grasp of the English language improves; girls make more progress than boys; the highest ethnic performing group is Chinese and the affluent outperform the disadvantaged in the English School system.  It’s not the same for every child but statistically this holds true.

Tweak 1: Contextualise the Progress 8 Score (even better have a three year rolling contextualised P8; irons out normal statistical variation from one year to the next)

The issue of outliers, pupils whose actual attainment is way below that of what was predicted given their Key Stage 2 SATs, is increasingly well documented.  The pupil whose Attainment 8 score is 30 points below expectation has to be counterbalanced by a class of thirty pupils all gaining one grade higher than expected; that’s a big ask.  Outliers, those well outside of the expected range, are arguably seldom due to the school effect; more likely a life or family imploding or at the other extreme a bad Key Stage 2 SATs work or radically improved family circumstances.  Education Datalab have suggested putting a cap on the maximum and minimum Progress 8 score any individual achieves.  An alternative would be to discount the top and bottom 5-10% of pupils from a school’s Progress 8 score.  Removing the outliers would give a truer assessment of a school’s effectiveness.

Tweak 2: Remove outliers from school’s Progress 8 scores

One of the concerns with Progress 8 and outliers is that unethical school leaders will seek to remove possible outliers at the earliest opportunity.  Nicking an idea from Education Datalab (again), Headteachers’ Roundtable has proposed that a school is proportionally accountable for a time that a pupil attends it.  For example, a pupil who joins you in Year 7 and leaves in Year 9 (Headteachers’ Roundtable suggested adding a year) would be on your school statistics for 80% of their eventual outcome (4 out of 5 years).  This is not a perfect proposal, far from.  Imagine a school serving an army battalion or pupils naturally moving as their parents relocate for work or family reasons.  Schools have no control over this.  However, given the impact it would have on the few schools who try to manipulate their pupil population I’d prefer, on balance, to live with the side effects.

Tweak 3: You’re proportionally accountable for all pupils who you’ve educated

The fourth tweak is required to stop some schools trying to game the system.  Schools must focus on doing the best for their pupils not manipulating Progress 8.  I’ll never forget being sat in a room of school leaders a decade or so ago being told to put more pupils on the SEN register and forget to designate your EAL pupils as such on your census return.  Along with ridiculous equivalences for some vocational courses it undermined the previous contextualised value added measure.

Tweak 4: Make public the general principles of contextualising and removing outliers from Progress 8 but keep the methodology secret and vary it over time.

The ECDL Effect needs to be addressed.  There is nothing wrong with ECDL; it’s a great course.  However, it doesn’t seem to be equivalent to other GCSEs.  There is quite a detailed analysis of all subjects by Education Datalab on this matter; be warned, it’s complicated.  I’d be happy just to remove the few subjects which are hugely out of sync in the first instant.

Acknowledgement: Education Datalab

Tweak 5: Limit gaming by qualification; the curriculum must be right for the young person

Most of what I’m suggesting above also applies to any primary school value added system.  The problem of trying to make any sensible value added proposals for the primary phase is the assessment system – no acceptable and standard baseline and use of teacher assessment for accountability purposes – is a bit of a basket case.

Here are a few more Education Datalab posts you might be interested in: one on outliers and the other on comparing schools with different intakes.  Told you I was and Education Datalab Groupie.