Porter and Riley (1994) quote Castellan as saying "The goal of instructional technology is to increase the effectiveness of learning. This may involve engaging students' attention, teaching difficult concepts, or improving the efficiency of learning." Indeed, one of the stated aims of the Teaching and Learning Technology Programme is to make teaching and learning more productive and efficient by harnessing modern technology. As Porter and Riley (1992) point out "The need to show the superiority of CAI over other teaching techniques is probably the most critical issue" but "..little work has been done, particularly in economics, to show that using CAI materials really improves students' comprehension of the material."
The WinEcon evaluation project was set up to try and gauge whether the aim of TLTP has been met and whether computer based learning methods really do represent an improvement over traditional methods, both from the point of view of the student and the lecturer. Essentially, as Draper et al (1994) point out "There are three kinds of things to measure: learning outcomes for students, the state of all factors in the situation with a significant impact on learning, and the costs (e.g. in time) for all participants." This article is not an attempt to answer all the questions that have been asked, it is merely intended to provide an overview of the evaluation that has taken place so far and to outline the future objectives of the evaluation project.
Throughout the development phase of WinEcon there has been a strong quality assurance mechanism which has formed the crux of the formative evaluation. Prototype versions of the software were trialled with students and feedback was provided to the developers to be incorporated into later versions of the software. Feedback was obtained using a standard Consortium questionnaire which covers five areas, `The "Look and Feel" of the Screens', `The Economics Content', `Thoughts on Computer Based Learning in Economics', `Personal Effect' and `General Impression'. Students are also invited to comment on areas which are perhaps not covered in the questionnaire.
Since October 1994 evaluation of WinEcon has been taking place to attempt to assess the impact that this method of learning has on students. Student trials of the software are taking place at a number of institutions and all students who use the software are asked to complete the standard questionnaire. In addition, in trials with first year students at Portsmouth, time has been spent in the computer labs observing the students using the software and some students have been asked to complete log books after every WinEcon session to pick up any comments or observations that the questionnaire may miss. The final method of collecting data is via informal interviews and discussion sessions. The results gained from these methods of evaluation are obviously subjective in nature, but as Blecha (1992) points out "A subjective measure can convey valuable information as long as its limits are understood."
So, what are the initial findings of the project? Responses from the students have generally been favourable with all students finding navigation through the package easy. The Economics content of the material was thought to be pitched at an appropriate level and the majority of students felt that those with less knowledge of Economics than themselves would benefit from using the system. However, the crucial question is whether students felt that using this system helped their understanding of a topic - no students felt that the system worsened their knowledge and only one claimed not to have had his knowledge enhanced. As was pointed out earlier, this measure of learning gain is subjective but, to refer back to Blecha (1992) "The subjective measure used here has the strength of reflecting a student's own judgement about whether or not the software was helpful in learning course material."
Of the students who completed the questionnaire 89% felt that the system would be useful to them if they were new to a topic. Interestingly, one of the main comments to come out of discussions about WinEcon was that the students would like more test questions to work through. At the moment the package is seen as a substitute for textbooks/tutorials and as a backup to lectures and seminars. On the whole, 74% of the students found the programme interesting but one of the main complaints was about limited access to computers, often leading to the query "Where can I buy a copy?"
While the responses gained from the students so far have been encouraging, more work needs to be done to formulate more objective measures of the learning gain experienced by students. Initial attempts have been made to identify students learning styles using Kolb's Learning Style Inventory. It is an accepted fact that different students learn in different ways and some students may benefit more from computer based methods than others. The aim is to attempt to identify those characteristics which have an impact on students' learning gain when using computer based learning methods. Research in this area will be continuing in Portsmouth next semester, when WinEcon is to be trialled with more students.
The evaluation project will also be attempting to assess the impact that computer based teaching methods such as WinEcon have on the teachers of Economics principles courses. Lecturers will not be persuaded to use WinEcon, or other computer based learning techniques, unless it can be proved that the package holds some benefit for them, for example in terms of time, student assessment and demonstrable learning gains for students. The project will aim to provide recommendations as to how best to incorporate WinEcon into Economics courses. Looking to the future, WinEcon may not be limited to replacing seminars and tutorials and providing a method of student assessment, as the advent of HyperLecturing, a term first introduced by David Gillette at the American Economics Association Meeting in New Orleans 1992 and expanded upon in Gillette (1994), provides an ideal vehicle for the use of WinEcon.
To conclude, the evaluation project has a number of aims and objectives which will be tackled in the course of the next few months. While it is hoped that the results of the evaluation will add credence to WinEcon's claims to be an innovative teaching technique, it is also anticipated that it will be possible to generalise some of the findings to other forms of computer based learning.