Methods of evaluation

Some almost random ideas about how you can evaluate...

Brookfield's Critical Incident Questionnaire (more here from Brookfield's own site). This is intended to get behind the superficial aspects of the session to those elements which make a difference to the learning experience. Note that the teacher does not process the questionnaires—a student does so and introduces a discussion on the results at the start of the next session.

Post-it notes: Make use of different colours to ask different questions. Perhaps pink for "Something I still don't understand"; yellow for "Can we have some more on this?"; green for "I finally got this point!" Or green for "I like this kind of teaching (exercise, for example)"; pink for "You are going too fast!" You can decide on the questions at the last minute to test out your hunches on what may be going well and not so well. Get them filled in at the end of the class—students stick them on designated white-board as they leave. You can sort and group them roughly (if you have time)—in any case, take a photo of them before pulling them down, so you can read at leisure. I post the pictures on the course blog. (Why are blue post-its so rare?) This is a variant on the "One-minute paper".

You can using voting to rate the importance of comments or significance of action points, this is called the nominal-group technique.

You can also get students to signal to you by holding up coloured post-its (or better still, coloured pages in their course handbooks) in answer to your questions at intervals during the session (this may be combined with a quick chat to a neighbour about the topic). Effectively this is a low-tech substitute for the use of clicker technology. The answers are of course not anonymous, but it takes only seconds to set the exercise up, and you can get a good-enough impression of the distribution of responses in a glance.

Action-research; this is an approach of disciplined examination of your own practice with a view to improving it. Evaluation is an early part of the process, but for present purposes look at action research as a framework which will help to focus your evaluation questions. (Go here for an accessible introduction from one of the principal figures in the field, Jean McNiff)

Get quick feedback about the level of understanding of a topic by encouraging students to take notes by concept- or mind-mapping. Why? Because the shape of the map itself (and its level of complexity) can tell you a lot at a glance about the author's understanding (see Kinchin and Hay, 2000).

If you like using technology, set up a twitter hashtag and invite students to live-tweet the lecture. This does assume that they can be trusted not to go off-task or start trolling, and it does help to have an assistant to keep track of the tweets and to relay themes to you as the session develops.

Inviting peer observation; student teachers get an enormous amount of participating in an observation chain, which has no agenda other than furthering the learning of both observer and observed—and there's no reason why that can't carry on for ever (see here for an example). Of course you need to be prepared to return the compliment, and it's time-consuming, but very worth-while.

Recordings; One downside of recording used to be the need for a technician and bulky equipment. That is no longer the case; video can be recorded on a phone or camcorder in HD, either from a tripod or other mount (I have one which attaches to a window with a heavy-duty suction cup) or which can track a teacher automatically; such equipment is routinely used for micro-teaching. The other downside is that reviewing the material can be time-consuming—if a student or colleague can keep time and note when issues come up which could yield useful material, then it is much easier to find them. (Tip; run another camera on the class group—after asking them, of course—so you can get reactions. It helps if both cameras have the same clock in shot, but it is not essential—it is easy to synch them in an editing package, matching the wave-forms of the soundtracks.)

Use the data; Courses nowadays generate enormous amounts of "big data" all by themselves, and most of it is digital and thus searchable. Written assignments, for example, can be run through Wordle, for a quick impression of how much key words appear. In a few seconds you can generally see whether the whole set of assignments gives evidence of whether material has been grasped. It may be superficial, but it identifies points of interest easily (and you can show the word-cloud back to the students as a guide to your overall feedback on how an assignment has been handled.) Incidentally, this article looks at another way in which "waste" data can be used. But should it be?

Further ideas welcome!

(6 August 2013)