It depends on what you mean by…. Defining our terms in assessment

I have been worrying a lot recently – and not so recently in fact – about the meanings of words. In particular these sets of words have been giving me sleepless nights:

Set 1:

Set 2:

Set 3:

When Jonathan Savage and I wrote our “A to Z of Teaching” book we wrote about some of these words, and so I have spent some considerable time thinking about what these words mean. And JS and I were very thorough and careful in our planning, and so I don’t want to revisit that too much, save to say that we were writing for teachers of all subjects, and what I am worrying about here is the very specific application of these words to music education.

Now I know that a popular perception of academics is that we sit in our ivory towers deliberating upon how many angels can dance upon the head of a pin, but in reality meanings matter, as words can also be used as weapons:

“Your pupils have not made the required amount of progress”
“The attainment of your pupils is not at the national standard”
“Measurement of your pupils in tests show that they are falling behind”

And saying, “ah – it depends on what you mean by…” probably won’t help your case!

So what do these words mean? Well, let’s be even more academic, and ask what we mean by another word: “meaning”. I am rather fond of Ogden and Richards (1927) work entitled The Meaning of Meaning. But even that doesn’t help us much, as O&R weren’t writing for music teachers. So, lets take the list one set at a time, as that is how I have presented them.

There is a problem doing this, which I recognise, and that is they are interdependent on each other for meanings, thus in defining one I often invoke another, without having defined that first. Sorry. (Still this is a blog, and I hope no-one is grading it!)

Set 1: Assessment; Evaluation; Measurement
For me, all of these words have something to do with judgements, either formal or informal. Assessment troubles me, as I think formative assessment is very different from summative. I have cited Dylan Wiliam on this before:

“”The big mistake that Paul Black and I made was calling this stuff ‘assessment’,” he said. “Because when you use the word assessment, people think about tests and exams. For me, AfL is all about better teaching”” (Stewart, 2012).

Quite. So formative assessment, (and I’ve given up on distinguishing between it and AfL) is about working with children, in the moment, helping them know what to do next; whereas summative assessment ascribes a ‘summing up’ grade, mark or level.

Evaluation seems to have different meanings depending on where you are in the world. Here is Wynne Harlen:

“The terms ‘evaluation’ and ‘assessment’ in education are sometimes used with different meanings, but also interchangeably. In some countries, including the USA, the term ‘evaluation’ is often used to refer to individual student achievement, which in other countries including the UK is described as ‘assessment’. In the UK ‘evaluation’ is more often used to denote the process of collecting evidence and making judgments about programmes… The processes of assessment and evaluation are similar, but the kinds of evidence, the purpose and the basis on which judgments are made, differ.” (Harlen, 2007 p.12)

And now Saville Kushner:

“Unlike the USA, Britain does not have a well-defined professional community of evaluators. Programme evaluation, particularly, has, in the USA, been the subject of much meta-evaluation and critical review. This has helped spawn a range of accredited university courses in evaluation around a robust evidence base and scholarly discourse. Issues in the conduct and use of programme evaluation are well rehearsed. This has not been the case in the UK.” (Kushner, 2005 p.111)

So, depending on where you are… and I am well aware that readers of this blog are all over the world, I tend to use ‘evaluation’ in the way Harlen describes above, to describe the evaluation of programmes of study, and so on, although I have noticed here in the UK we are using it more and more to describe individual attainments too.

Measurement really troubles me in music education, again especially in UK usage. We can measure some things, like the speed of playing a scale, but the moment any sort of subjective judgement comes in I want to turn into a positivist, and say that that isn’t measurement! Now this isn’t to diss the ABRSM, who are experts at measuring within instrumental music exams, but they undertake comprehensive standardisation procedures to endeavour to ensure accuracy. I am not convinced that, say, A-level examnation composing, is ‘measured’ in any way accurately at the moment.

Set 2: Attainment; Achievement
Once again, for me – and again my view might be different from others – attainment is a response to assessment, it’s that which results in a mark/grade/level. Achievement is more about reaching potential, and usually involves some evaluation of the progress involved. Thus, for me, running a mile would be a very significant achievement, but for some of my colleagues this is something they easily dash off before breakfast! The differences matter in assessment terms as although the attainment is the same, a mile has been run, the differences in the work put into it have been very significant. Ofsted have a very specific view: “Achievement takes account of pupils’ attainment and their rate of progress” (Ofsted, 2012 p.6). This clearly has implications for the ways in which we assess pupils, as in order to maximise their potential attainment levels we need to have a view as to where they started, in other words we need to take into account the progress that they have made.

Set 3: Progress; Progression; Development
Now I am happy to link the first two! Progress, for me, is moving through attainments, resulting in grades/marks/levels. Progression is the process of doing this. We can add an element of speed to this, ‘rapid progress’ is different from ‘slow progress’. The end result may be the same, but one has taken longer to get there than the other.

Development, for me, is somehow psychologically or physiologically related. Cognitive development occurs, as does physical development. We plan our curricula for the former, and take account of the latter (eg in singing, especially for boys).

So, how many angels are dancing on the head of this particular pin? Well, for me, I worry that sometimes people can say one thing, and the listener hears another. A while ago I published an article entitled “Lost in Translation” (Fautley, 2007), and the matters in this blog are another example of this, only here there is no translation, we think we are speaking the same language, but are we? And if nothing else, music teachers thinking about these issues, and discussing them with colleagues might help clear up some other confusions which have crept in, especially with regards to assessment, recording, and reporting. If we think we are doing something, but actually aren’t, this matters! And if our colleagues use the words differently, then the fact that my trousers are his pants ceases to be a matter of amusement!

Fautley, M. (2007) Lost in translation – The changed language of assessment in music education. NAME (National Association of Music Educators) Journal, 2-4.
Harlen, W. (2007) Assessment of Learning, London, Sage.
Kushner, S. (2005) Qualitative Control A Review of the Framework for Assessing Qualitative Evaluation. Evaluation, 11, 1, 111-22.
Ofsted (2012) ‘The evaluation schedule for schools 2012’. London, Oftsed.
Ogden, C. & Richards, I. A. (1927) The Meaning of Meaning, London, K. Paul Trench, Trubner & Co.
Stewart, W. (2012) ‘Think you’ve implemented Assessment for Learning?’. Times Educational Supplement, London.

This entry was posted in Assessment, Music Education, Progression and tagged , , , , . Bookmark the permalink.

One Response to It depends on what you mean by…. Defining our terms in assessment

  1. terryloane says:

    Thank you, Martin, for another thoughtful and thought-provoking post. I think you are absolutely right about the importance of thinking carefully about the words that are used to talk about institutional (e.g. school) learning and what these words actually mean.

    But I wonder if the reason you are losing sleep is that you are concentrating too much on trying to ‘define’ these words rather than on understanding how and why they are in common use. One problem with trying to define words (a problem you acknowledge when you say that “they are interdependent on each other for meanings”) is that we end up using one word to define another and going around in circles – which is a very good way of staying awake at night! There are a couple of examples of this in your post. One is from Ofsted, who manage to combine this circularity with vagueness in their statement that “achievement takes account of pupils’ attainment and their rate of progress”. The other example is your own definition of progression: “Progress, for me, is moving through attainments, resulting in grades/marks/levels. Progression is the process of doing this.”

    You touch on the notion of the meaning of meaning so let me touch on Wittgenstein. He talks of ‘meaning-in-use’: the idea that the meaning of a word is nothing more or less than how it is used in language. In his ‘Philosophical Investigations’ he tells us: “Don’t think, but look!” This is, of course, exactly your approach when you consider the differences between the ways the word ‘evaluation’ is used in the USA and the UK. If we do adopt Wittgenstein’s approach of ‘looking’ rather than ‘thinking’ then we will surely focus on the unstated assumptions behind how words are used in the dominant educational narrative. And I think that a fundamental assumption behind contemporary use of all the words you list at the start of your post is that human learning/achievement/capability etc. can be measured numerically.

    But this is an erroneous assumption when it comes to complex activities like the creation/performance of music because, as you explain Martin, judgements are inevitably subjective. By the way, I like your example of the speed of playing a scale as being one of the few things in music that can be measured objectively, but even here things are not straightforward. We all know that as we try to play scales faster things can start to go wrong, so subjective judgement does come into play if we try to make a balanced judgement taking account of speed as well as, say, deteriorating intonation for a string player or unevenness of touch for a pianist – or perhaps we could use an oscilloscope rather than an ABRSM examiner to judge how well a scale is played!

    But if we move on from scales to real music then simplistic measurement just isn’t possible. When you refer to yourself as a positivist and state that “I am not convinced that, say, A-level examination composing is ‘measured’ in any way accurately AT THE MOMENT” [my emphasis] are you suggesting that it might be possible to ‘measure’ the quality of composition in the future if only we try harder? Surely not. Each act of musical creation is unique and creates its own criteria of success and failure (as I pointed out in an earlier comment in your blog with reference to Beethoven’s 3rd symphony). So judgements about different compositions are simply not commensurable. To make the point let’s consider two composers, both of whom started to write music in the same country and during the same decade: John Lennon and John Casken (some of whose early works I performed in when we were students together in Birmingham in the late 1960s). Now notwithstanding their similar roots in post-war industrial northern England the compositions of Lennon and Casken are so utterly different in style, context and ‘purpose’ that any numerical comparison (as required for the A-level music exam) would simply be nonsensical.

    Surely we must have the courage and imagination to develop meaningful non-quantitative ways of assessing, discussing and reporting on acts of musical creation. And if we do so then perhaps music educators might actually be able to lead the way in developing helpful systems and language for assessing, discussing and reporting educational achievement rather than lamely tagging along behind the educational managerialists, trying in vain to match their inappropriate and empty vocabulary to the valuable and creative work we and our students do – and perhaps you would have fewer sleepless nights, Martin:-)

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s