Feeding the Data Monster

It’s been an odd twelve months with respect to data.  I’ve spent the last two decades as a deputy and head teacher chucking as much aggregated data into the Data Monster, similar beasts exist in schools up and down the country, only to have now become an aggregated data minimalist.

My photograph taken October 2014 in Melbourne. Original art by @jacklesdouglas

The post-level World has sent many School Data Managers into a tailspin.   A number of schools have tended to gather data which predicted a pupils’ Key Stage 2, GCSE or A-level grade from an indeterminate base or determined a current level, grade or performance via dubious means.  The process of predicting an individual pupil’s level at the end of a key stage or GCSE grade has a weak statistical basis which can lead to significant variance between the predictions made and the actual outcomes.  These statistical models need a sample size greater than one to be of much value.  Teacher bias, different value systems and beliefs and inappropriate processes for determining a pupil’s current performance all add to the variance alongside the uncertainty of future efforts and performance.  The questions, “Why bother at all, is it really worth the effort or is there a better way to spend the limited time we have available?” were too rarely asked.  Accountability is inexorable or is it?

Frustrated by teacher assessment data which suggested our Year 6 cohorts had taken a massive step backwards since the end of Year 5 or inaccurate GCSE predictions we changed our methodology.

The Current Grade Methodology

Last September all three academies, one 11-18 and two primaries, moved to a current grade reporting system.  We possibly didn’t get off to the greatest start with confusion over current grade, current performance and some subjects still using predictions leaving many leaders scratching their heads and pulling their hair out in fairly equal measure.  In truth we needed far greater clarity for middle leaders than merely stating we wanted current grades and then expecting everyone to become mind readers.  The following set of criteria is aimed at GCSE but the same thinking applies to current mark/grade assessment for Key Stage 2 or A-level:

1. In a subject pupils will complete all the examination papers and associated controlled assessments/coursework, from a single examination season.  Where an element is not taken a mark of zero will be given.
2. The papers and associated controlled assessments/coursework will be marked using the examination board’s mark scheme or evaluation criteria/assessment objectives as appropriate.
3. Teacher’s marking will be internally moderated and where possible and appropriate externally moderated.
4. A final mark will be determined using the examination board’s methodology and converted into a GCSE grade using the grade boundaries published by the examination board for the specific examination season.

I would be interested in comments (please add to the bottom of the post) about a more or equally accurate way to determine a current grade.  If you don’t use the national/examination board published grade boundaries how can you reasonably ascribe a grade?  If you are using the grade boundaries from a particular year does it predispose that the pupils have completed the papers and associated controlled assessments for that particular set of grade boundaries, that is, for a given examination season.  Can you accurately produce a GCSE grade (or others) via another methodology?

Not Too Early, Not Too Often

One of the challenges of using a current grade data system is that there is very little point collecting data too early in the key stage or course.  Pupils simply haven’t covered sufficient material for the standardised mark/grade to be of much value.  There is a need to hold your nerve but also put in place a different type of assessment which simply looks at what pupils do and don’t know without worrying about the standardised mark/grade.  If you want to know more about our approach to this aspect of assessment it’s in the post Data and Feedback Informed Teaching and Learning.

As well as resisting the temptation to collect current grades to early, our first collection will be at the end of Year 10 for GCSE and Year 5 for Key Stage 2, there is a need for leaders to resist the temptation to collect grade data too often.  Pupils aren’t going to be moving a whole grade every few weeks and our data drops are the end of November and beginning of March in Year 6 & 11.  It’s much more minimalist in approach than anything we have done before.  As well as making sense to collect whole school aggregated data less often the issue of workload for teachers in marking and analysing each set of papers is significant.

Formative Use of Summative Assessment

Or put more simply find out what they don’t know and teach them it.  Leaders will need to consider at what level assessments, for example the mock examination papers, should be analysed.  A granular level analysis with the opportunity for reteach needs to be planned for in the scheme of learning; our growing experience is suggesting that we’ve underestimated the time needed to reteach quite significantly.

Photo Credit g-neato via Flickr cc

Whilst as a governor, head teacher or senior leader lots of data can provide a comfort blanket the most important assessment element of these mock examinations is determining where to next?  It is how a teacher and pupil respond to the gaps in learning and address them that really matters.  The only grade which has any true significance is the one gained at the end of the key stage or course in the national tests or public examinations. 

In our academies there are now hundreds of examples of spreadsheets, all different sizes as deemed appropriate by the subject leader, to help with the analysis of what pupils do and don’t know and some increasingly sophisticated ways of re-teaching or revising aspects of work previously taught.  An interesting spin off is the extent to which pupils can actually answer, with varying degrees of success, parts of the curriculum which have not yet been taught.  It provides a great assessment of prior learning on which to plan future teaching.

Sharing with Pupils

 One of the debates around using current grades is the potentially demotivating effect of getting a low grade early in the course.  If I’m honest I don’t think it is a strong argument; it doesn’t require a sophisticated level of understanding by pupils to appreciate they haven’t covered all the course material and once they have, if they are willing to learn it properly or at least retain it for the examination, then they will do much better next time and their grade will improve.  It also starts an honest debate with teachers and pupils about the standards being reached.  If improvement is needed let’s not wait until after the results are published.  Improvement is always wanted if you are ambitious enough for your pupils.  The challenge then is providing great professional development for teachers and programmes to build the resilience and academic buoyancy required by our pupils, if they are to succeed.

Different students know different things – their personal gaps in learning need teaching too

Similarly, if pupils’ attainment is low there are likely to be significant gaps in their learning helpfully exposed by the assessments.  You may need to trace back years to the previous key stage or even the one before to get to the root cause of the problem and address the gaps in learning which are preventing current and future learning.  Alternatively, if you are still concerned you don’t need to share current grades with pupils but instead just focus on the learning now required as a product of the forensic analysis completed.

I’m finding that I’m blogging a lot about assessment at the moment.  I’m genuinely enthused about some of the work we are doing as our assessment is increasingly focussed on its impact on teaching, learning and the learner rather than collecting less useful aggregated data for leaders.  Don’t feed the data monster too much it’s not healthy for it or education; look for a more balanced assessment diet.