Principles of Evaluation

Evaluation may be defined as "a disciplined attempt to find out if a taught programme makes a difference". I am here (very) roughly following the Kirkpatrick model; it's not the only one, but since it is business oriented (rather than purely academic) it does emphasise the impact of taught programmes rather than simply their internal characteristics.

• a disciplined attempt; Eric Morecambe's catch-phrase "What do you think of it so far?" is the essence of evaluation. Unfortunately, that question only ever led to one response; "Rubbish!" (Apologies to those readers who do not get the allusions to a major British cultural tradition of 1961-83, but still current.) Professional evaluation aspires to something a little more organised, and providing more constructive feedback.
• a taught programme; evaluation feedback is only useful if it can lead to improvements in a further iteration of the programme. It presupposes a degree of control over the conditions of learning. We can only usefully evaluate a programme of experiential learning in respect of those factors which can be managed.
  • The original sick joke is, "Apart from that, Mrs Lincoln, what did you think of the play?" (More edgily, because more recent, "Apart from that, how was the World Trade Center as a place to work?") There may be things to learn from one-off experiences, but systematic evaluation can't really touch them.
• makes a difference; the Kirkpatrick model suggests evaluation is ultimately about the capacity of a course to effect change.
  Kirkpatrick, D. (2006) Evaluating Training Programs: The Four Levels. (3rd edn.) Berrett-Koehler Publishers.
  This is a broadly positivistic perspective, which may not be applicable to all academic courses, but it poses the critical question whether the course itself has made more impact than, say, the accumulation of experience over a similar number of hours. In my own area of teacher training, this is a big question (albeit rarely asked, because of vested interests); would a student learn more by spending a year teaching, than by spending it on a PGCE course? (Since those courses in the UK are now 80% practice and only 20% in college, the question is implicitly answered.)

The model was developed as part of his Ph.D by Don Kirkpatrick in 1954—rarely can a PhD have paid off so directly and for so long! It was modified in minor respects in the "New World" version in 2010. In brief, it identifies four levels at which training programmes need to work: (Kirkpatrick's own points are italicised; my notes follow.)

Level 1: Reaction

"To what degree participants react favorably to the training."

It is of course all too easy to be seduced by this—particularly if you are a freelance trainer looking to be re-engaged, perhaps. It's not unknown for people to report that they thoroughly enjoyed a course—whether academic or technical—but did not learn anything. You could say this is the "entertainment" level.

Level 2: Learning

"To what degree participants acquire the intended knowledge, skills, attitudes, confidence and commitment based on their participation in a training event."

This is what you hope to discover from your assessment of participants' learning; and feedback to you about how successful you have been in "making learning happen" (Race, 2010). And this is as far as it goes, for educational institutions. For them, the award of their qualifications is what matters—not much more.

Level 3: Behaviour

"To what degree participants apply what they learned during training when they are back on the job."

Here we step outside the teaching/training system itself, and this is where this approach to evaluation was distinctive. Naturally, as a sponsor of or contributor to training, we want it to be effective, don't we? Actually, clearly not. It is often clear that offering training is a backside-covering tactic, particularly in health and safety areas, for example. It is sometimes merely a tick-box exercise. Indeed, it can be counter-productive; it seems to have declined somewhat now, but a popular form of trade union "industrial action" used to be the "work to rule", when workers knew that actually practising as they had been trained would bring production to a virtual halt.

But there are also other factors involved in working practice, which mean that application of what has been learned is not always straightforward. See also this piece on how behaviour might change, but not always in the intended direction.

Level 4: Results

"To what degree targeted outcomes occur as a result of the training event and subsequent reinforcement."

A big question—was a training intervention an effective way of bringing about change in the organisation? This question may go beyond the pay-grade of many education and training practitioners of course, but it is the key test.

And what about unintended and unexpected outcomes? The Kirkpatrick focus does not allow for the basic IT skills course which leads to "transformative learning" for example, and it assumes that direct intervention has intended and only intended consequences...

Up-dated 3 August 2013