3 Evaluating the effectiveness of seminars

Seminars may be evaluated according to their objectives or according to whether the processes in the seminar might be expected to achieve the objectives. Evaluating a seminar according to whether it has achieved its stated objectives is perfectly sensible in principle, but there are limitations to carrying out this out in practice. First, many objectives cannot be achieved in a single session. Second, it may be difficult fully to assess within the session whether the objectives have been achieved. Evaluation is, therefore, unlikely to rely solely on assessing outcomes. Evaluating the way in which the seminar was conducted is, in any case, central to developing practice.

Evaluation may also be conducted by the seminar leader, an academic colleague, an external evaluator or the students. In principle, the criteria for evaluation ought to be the same whoever is carrying out the evaluation. The usefulness to lecturers of evaluations carried out by external agencies and by students will depend in large part on the measure to which there is agreement about the criteria used in the evaluation. For example, students may judge that a session is uninteresting even if it has contributed to longer-term goals of the programme. However, other things being equal, we would expect the level of students’ interest to contribute to the quality of their learning.

3.1 Criteria for evaluation

The QAA (2001) Subject Review Handbook requires observers to summarise each session’s ‘overall quality in relation to the learning objectives’ and to evaluate the processes of a seminar according to: clarity of objectives; planning and organisation; methods/approach; delivery and pace; content (currency, accuracy, relevance, use of examples, level, match to student needs); student participation; and use of accommodation and learning resources. This provides a starting point for a review of the criteria that might be used in self-evaluation. Given the discussion in section 2, objectives should be clear in terms of the description of what students will know, understand or be able to do. Objectives that are written in terms of the content that will be covered are not compatible with an intention to encourage ‘deep learning’. Examples of objectives that might encourage deep learning include:

  • to understand the problems associated with the quantitative evaluation of the development experiences of different countries;
  • to understand the relation of Marxian perspective to political, social and philosophical assumptions;
  • to synthesise and analyse key developments in the monetary sector.

The criterion ‘clarity of objectives’ is ambiguous with regard to ‘clarity to whom?’ – the seminar leader, an academic peer, the students? The implication from section 2 is that objectives should be clear to students and, to that end, it is a good idea to state objectives explicitly at the beginning of the session. It is often useful to give a short explanation of the format of the seminar and how that format is expected to help students’ learning. This practice should help students to understand the purpose of the activity and should increase their ability to participate effectively. It should also help students to draw together the different aspects of the session and this, in turn, should promote greater understanding.

Other criteria in the QAA list (planning, organisation and methods, and student participation) might be judged according to the nature and variety of activity for students. Does the organisation of the seminar provide appropriate stimulus to students’ thinking and does it provide sufficient space for them to develop their ideas? Does it allow the seminar leader to meet the needs of students who are working at different levels of understanding? Does it allow for different modes of student participation? In evaluating student participation, it is useful to bear in mind the discussion of learning styles in section 2. The willingness of students to participate will depend on the match between the style of activity and their ‘preferred learning style’. The degree of students’ participation can be evaluated only alongside the opportunities that have been provided for different types of participation.

3.2 Student evaluations

It is usual for students to be asked to evaluate their experience by completing a questionnaire at the end of each module. Questions may solicit students’ judgements on the clarity of the unit description, the style of teaching, the quality of handouts and the level of interest that they maintained throughout the unit. The collation of such evaluations may provide award leaders and university administrators with an overall sense of whether students tend to be happy or unhappy with the teaching on a module. However, they provide limited information for a lecturer seeking to use students’ evaluations as a means of improving teaching and learning. It is more useful to spend part of one seminar discussing one of the evaluation criteria with students. In general, higher-quality information will be gathered through dialogue with students.

This dialogue can be a routine part of weekly seminars, particularly when the seminar leader is working with small groups of students as they are engaged on a seminar task. Students may be asked whether they find the exercise helpful to their learning needs, whether they have any suggestions about how the exercises could be changed to provide greater benefit, and whether they feel that the accumulation of exercises has helped them to gain further understanding of the material and its application. Alternatively, evaluation of individual sessions can be gathered from brief questions at the end of a seminar (e.g. ‘Was this a useful exercise?’, ‘Did this exercise help you to enhance your understanding of the subject matter?’, ‘What changes would make this exercise more useful in the future?’).

3.3 Formal module evaluations

University policy at most institutions requires staff to critically evaluate unit delivery, student response and assessment strategy as a matter of course at the end of each semester. In addition to this process, it is very useful to incorporate a system of personal reflective practice into seminar programmes in order to keep a continuous record of successful teaching practices and the changes that need to be made to individual sessions. When introducing a new approach or technique in teaching, it is useful to draw up a session plan that outlines objectives for the session, how the session will be organised and what you are expecting the students to do in each part of the session. This kind of plan is useful for future reference and also provides a good starting point for personal evaluation of whether the new approach is worth continuing and whether there are improvements that could be made.

It may be useful to focus personal evaluation around some leading questions that provide a structure for reflecting on the value of the session and why the session worked in the way it did. Some questions to consider are:

  • Did I meet my objectives?
  • Did I deviate from my original lesson plan and, if so, why?
  • What were the strong points of the session?
  • What were the weak points of the session?
  • What modifications should I make to the session for future delivery?
  • What does the feedback from students tell me?
  • How did I assess student understanding?