[Note from Web Editor: in conjunction with the electronic version of this article, we are launching a service which allows users to create simple quizzes easily using a web form. Read on for details.]
As part of the process of establishing a set of web pages for each of the courses that I teach I have recently considered the idea of incorporating on-line quizzes to give the students immediate feedback on their understanding of course materials. Some economics course web pages that I have seen already do this, and it is an idea which was supported by Jane Leuthold in her recent article about how to build a Homepage for economics classes (Leuthold (1998)).
However, having considered the idea my next problem was how to go about it. Looking at examples of such web-based quizzes both in economics, and in other subjects, I could see that there are a variety of ways of going about the task. I decided to investigate the matter more systematically. This paper reports on some of the things I have discovered. I sought to answer the following questions:
There are comparatively few on-line economics quizzes on the web that I could find. Other subject areas seem to have many more such quizzes. It may be that economists are sceptical of their value, are not aware of the tools, which are now available to produce them, or just consider that the opportunity cost of spending time producing them is too high.
Some of the quizzes, which I did come across, are linked to textbooks. Publishers are increasingly creating Web sites for their textbooks, making available supporting materials such as datafiles, slide presentations and quizzes through this medium. Here again economics is not at the forefront of these developments, but there are some to look at. For example, the publishers of Manfred Gärtner’s new European Macroeconomics text have set up a web site with quiz questions to go with each chapter.
Sometimes authors create their own web sites to go with their books. For example, Steve Hackett has produced four on-line quizzes to go with his textbook “Environmental and Natural Resources Economics”
Roger McCain has lots of quiz questions as part of his on-line text “Essential Principles of Economics: A Hypermedia Text” [No longer on open access- Wed Editor]. There is also a large collection of multiple choice economics questions at the Amos World Economic Testing System site.
As I come across other examples of on-line quizzes in economics I shall place links to them on a special web page I have set up for this purpose.
On-line quizzes can be used as an instrument for providing feedback to students on the degree of their understanding of course material. Such quizzes can be used at the beginning of a course for diagnostic purposes to indicate any areas where prerequisite knowledge may be inadequate, during the course to measure progress in understanding, or at the end of a course to assist in revision.
Byrnes and Debreceny (1995) cite Ramsden (1992) in stressing the importance of high quality feedback on student work, where students are not only given an indication of how well they have done (by grade or mark) but also an indication of how they have gone wrong and explanations which can help them improve their understanding of the material.
When large numbers of students are taking a course it becomes impossible to mark and return regular tests and exercises. Properly designed on-line quizzes however allow students to test out their knowledge of a topic and get immediate feedback. Students appreciate this feature, and although there may be limits to the types of questions, which can be constructed for such quizzes, they can have important motivational role. There are of course dangers, in that students may mistake good performances on these quizzes for knowledge of a sufficient depth to deal with examination questions and coursework assignments. Students should be warned that while they might see such quizzes as a way of confirming that they have sufficient understanding of a topic (when they get all the answers right) it would be better if they thought of them more as a way of revealing areas of weakness where they have an incomplete understanding of the material (shown when their answers are wrong).
Multiple choice quizzes have, of course, been available for many years, very often even in computerized form to allow automatic marking. By placing such quizzes on the Web, however, a lecturer can potentially take advantage of all its facilities, expanding the types of questions which can be asked (possibly including not only on-line graphics but also full multi-media audio and video clips as the basis for the questions posed). Direct links to further material for the student to work through, depending upon the outcome of the test, can also be given as part of the feedback. With the quizzes placed on the Web students can try them whenever and where ever they like (provided of course that they have access to the Web).
Putting material on the Web also allows lecturers to benefit from sharing resources with each other. This is as true for quizzes as for other types of on-line material and lecturers can provide pointers to quizzes produced by other economists and placed on the Web (provided they have not restricted them to students at their own institution through password and user id screening). I should certainly like to encourage more sharing of on-line resources of this kind. One of the aims of the Netquest project reported on in CHEER Volume 11 Issue 2 (see Williams et al (1997)) is to construct fully indexed and searchable "questionbanks" from which questions can be drawn. This approach requires that lecturers donating questions conform to common standards.
For lecturers, if the system they are using contains mechanisms for tracking and analyzing student responses, it can help to identify students who are having problems, areas of common concern for students where further explanation is needed, or where badly designed questions need to be rewritten.
If the system also has built into it password and id screening, together with secure archiving of test results, it can be used for on-line end of topic or end of course examinations. Although people at a number of universities are piloting the use of such electronic examination systems, this has not been my primary interest in this study. My focus is more on informal on-line quizzes to facilitate the learning process.
Quizzes on the Web can, in principle, include questions of each of the following types (although it may not be possible to produce some types of questions using some development tools, and some question types may require special hardware or software on the student’s computer):
Some quiz setting systems will allow for the random selection of questions from a database, or in the case of numerical questions set randomized parameter values (different every time). Some will allow users to retry quizzes as many times as they wish and will deliver detailed explanations when an incorrect response is given. In such cases if the quiz is not used as a part of the assessment students might intentionally give an incorrect answer in order to see the full solution, using the test to elicit as much information as possible. Other systems can prevent users from taking a test more than once or restrict the amount of time available for them to answer each question.
Essentially there are two approaches to authoring on-line quizzes for the Web: (i) program them yourself, using a scripting language such as Perl or Java, or (ii) make use of a software tool which can automatically generate the code for you. An intermediate position is to cut, paste and edit code which has been produced by someone else, and which you can alter to suit your needs. Which approach you choose can depend upon the quantity, variety and purpose of the questions you want to produce, whether you want to integrate the quizzes directly into your web page or keep them as separate resources to which students may be directed, your experience and confidence with programming languages, whether you are working on your own or as part of a team, and the availability of funds to pay for software or programming time.
If you plan to code the questions yourself it makes sense to take a look at the way that other people have done it. Some quiz authors have gone out of their way to make their code accessible to others.
For example, Brian Tissue of Virginia Tech, who together with some of his colleagues works on the Chemistry Hypermedia Project, has made available scripts (both JavaScripts and Perl scripts) for placing interactive questions on Web pages. You can copy, paste and edit scripts to create multiple choice or numerical questions. The scripts are made available freely for non-commercial use, provided that Brian Tissue’s copyright is acknowledged.
To illustrate how it works I have taken Tissue’s JavaScript for creating a multiple-choice question with a single correct answer, edited it and produced a single question asking about the use of the Durbin-Watson statistic (see below). The code for generating this question is given in the Appendix. Text values are assigned to the variables containing the question choices. The final variable for the question indicates which of these responses is the correct answer - in this case answer number 4. When the user clicks on the Check Answer button they are given an indication as to whether or not their answer is correct. Here just the brief message “Correct Answer!” or “Please Try Again” is given, but longer feedback messages could easily be used. Adding hints or further explanation of the correct answer would be straightforward using this approach by including a box to display the hint, or by activating a hypertext link to another web page.
Sample multiple choice question produced using Brian Tissue’s
code: Try the question View the source of the question Create more Javascript questions using our on-line toolbox |
If you want to link your question to a graphical image this too is reasonably straightforward. For example you could show a labelled diagram or a screen grab of an image you want the students to refer to by incorporating a link to the image source. Then you could pose the question and ask the user to choose the correct response from a list of choices. Making such a graphic a “Hot Image”, so that the user instead responds by pointing to a particular part of the image, is a little more complicated and requires more advanced programming skills.
If you are providing a set of questions to be answered you may wish to keep a tally of the student’s score. This can be achieved by adding in a scoring variable whose value is cumulated as the quiz progresses. The simplest scoring system would allocate one mark for every correct answer and a zero for each incorrect response. However, alternative scoring systems could be used, possibly even with negative scores for wrong answers to deter guessing. At the end of the quiz the student can be given a message to reflect their overall performance on the quiz; high marks could generate a congratulatory message and a suggestion that the student moves on to the next quiz. Poor marks could call forth a different message, which also points the student towards material, which should be studied before the test is tried again.
A possible weakness of this approach is that the source file contains the information about which response is correct. As it can be viewed on-line using the browser software, students could cheat and find out the right answer that way. This may not be an issue if, like me, your purpose is to provide quizzes as an interactive learning tool rather than for assessment. However, one way to retain such information on the server is to make use of a CGI (Common Gateway Interface) program. This program will link the remote user to the database of your questions, answers and scoring information held in a directory on your server. When the user requests the question it will be displayed on their screen. When they choose a response the program will check back with the database and display an appropriate message for the user. The student never has direct access to the full set of questions and answers. This is the approach which is used in the NetQuest project (see their site for more details).
The NetQuest questions themselves have been produced using the Tutorial Mark-up Language (TML) which is a super-set of HTML. As Williams et al (1997) noted, this language was developed by Joel Crisp at the ILRT, building on some initial work undertaken by Neil Holtz at the University of Carleton in Canada.
Figure 1. Sample multiple choice question produced using TML.
Click above to go to the question on the NetQuest site.
Figure 1 shows a multiple choice question produced by biz/ed staff using this system to illustrate its potential for the construction of tutorial questions for A level economics. Instead of using radio buttons to select an answer, the user must use the mouse to point at the letter which corresponds to their choice. When the mouse is clicked the software finds and displays the message, which is linked to the relevant URL.
Although the TML software which was used to create the question illustrated in Figure 1 (TML version 4.0) can still be downloaded from the NetQuest web site, the project team has recently been concentrating on ensuring that their system is consistent with the standards that have been agreed by the World-Wide Web Consortium (WC3). As a result, a new version of the language TML 5.0 is being developed (and TML is now taken to stand for Tutorial Modelling Language). For more information contact Dan Brickley at daniel.brickley@bristol.ac.uk.
A number of other projects of this type have been going on around the world, including the CASTLE Project at the University of Leicester and TRIADS (Tripartite Assessment Delivery System), a joint project involving the University of Derby, the University of Liverpool and the Open University. Links to sites for these and other projects can also be found on my special quiz links web page.
An alternative approach, which can be used to create on-line quizzes, is to purchase one of the growing numbers of commercial software tools. Several departments at Portsmouth are using Question Mark Perception to develop and deliver both self-assessment tests and formal computer-mediated exams. A variety of standard question types are supported including numeric response and diagram/hotspot response as well as multiple choice, multiple response and text based questions. Questions are authored off line using Question Manager and stored in a database. Questions can be enriched by linking them to multimedia (graphics, sound and video) files. You can also integrate your own customised question types through the Question Markup Language (QML) used by the program.
Session Manager enables you organise these questions into tests. You can select questions individually or have them extracted randomly from one or more topic groups. You then use Perception Server to deliver the tests over the Web (or an Intranet) to authorized individuals. Perception Manager provides extensive on-line security management of both assessments and users. A final component called Reporter can be used to analyze the answers and results.
Note: In order to install the Web-based server components of Question Mark Perception you must have an ISAPI compliant server. ISAPI (the Internet Server Application Programming Interface) is for Microsoft’s IIS (Internet Information Server). Several Web servers from companies other than Microsoft also support ISAPI. ISAPI enables programmers to develop Web-based applications that run much faster than CGI programs because they are more closely integrated with the Web server.
You can see examples of the kind of tests that can be produced using Question Mark Perception at Question Mark Computing’s web site. You can also download a fully working 30 day evaluation copy of the program. For further information contact Philippa Bean, the UK Sales Manager, Question Mark Computing Ltd. [Tel +44 (0) 171 263 7575; Fax +44 (0) 171 263 7555; E-mail philippa@qmark.co.uk]
Another on-line testing system now available commercially is inQsit (Integrated Network Quizzing Surveying and Interactive Testing system). Developed at Ball State University, Indiana, a site licence for inQsit is only $499 for an educational or non-profit organization. The program supports a variety of question types (but not hot-spot graphics), uses cgi scripts, allows interfaces to other applications (including Excel), ensures password protected security and provides automatic analysis of results. A 30 day free trial version is available. For further details go to http://www.bsu.edu/inqsit/.
Web@ssessor from ComputerPREP is another recent product designed with Java to run on the Web. In addition to the usual range of question types, links to graphics, video clips, audio files, and animations, Web@ssessor allows tests to be authored or taken in any language. The web site gives examples of tests in both French and Spanish.
Other tools of this type which are available include QuizCenter, developed at the university of Hawaii; Top Class Test Assistant ; WebCT from the University of British Columbia and WinAsks Professional from SmartLite Software.
Last, but certainly not least, may I draw your attention to Quia! [http://www.quia.com/]. Like the commercial tools described above, Quia enables you to set up your own Web based quizzes without the need to know anything about programming. Once you have registered to use the service you can just log in, go to the Activity Manager part of the Quia web site and author your questions on-line. The system is quite easy to use. You can create quizzes of a variety of types, although for some types you must have a Java-enabled browser.
Quia differs from the other commercial tools described above in two important ways. First, authors do not store their questions and solutions on their own servers, but on Quia’s. This means that even if you don’t have access to a server and your own Homepage you can still set up quizzes at Quia and then let your students know where to go for them. Even if you have your own server, you can save the space on it for other things when you store your quizzes at Quia. Secondly, and importantly, the service is free to quiz authors and quiz users. Paul Mishkin from Quia says “Our goal is to provide powerful yet easy-to-use Internet tools for educators and learners at all levels. The services on Quia are all free, and will at some point be supported by advertising banners.”
Quia are keen to encourage the sharing of quizzes. When you have created a quiz you are invited to have it added to the subject directory so that visitors to the Quia site can find it. If you just want to use the quiz with your own students you don’t have to include it in the directory - but you will have to provide the students with the full URL. For example, I have created a brief simple regression quiz which I haven’t yet added to the directory. (I do plan to construct a number of these quizzes and then add them all to the directory but I want to try them out with my students first). In the meantime you can find this quiz through this link.
If you want to track the performance of your students as they take the quizzes you can create special “quiz sessions” based on your quiz questions. Students go to the session login page and enter your session name (a unique name that you picked when creating the quiz session). When the students answer the quiz questions and press submit, their scores are stored in a database. You can then examine the students’ performance, either individually, or as a group, obtaining information about:
You can also view the results by question, revealing information like:
Quiz questions are independent of quizzes, so you can create multiple questions for a single quiz if you have several different groups of students that you want to track separately. You can also create sessions for quizzes that others have authored. For example, suppose someone has created a series of quizzes to go with chapters in a specific textbook. If those quizzes are added into the Quia directory, other lecturers around the world who are using the same textbook can find them and create quiz sessions based on them to track their own students’ scores. Quiz sessions are private, so the original quiz author will not be able to see any sessions that others have created.
So far economists have not made much use of Quia. Currently there are only two economics quizzes in the directory, and they are on very basic material. However several other economists have experimented with Quia quizzes. Trudy Ann Cameron, for example, has some flash cards on Applied Regression Analysis. Quia are very keen to expand their economics section. Indeed they hope to be able to have specific micro, macro, econometrics etc. sections in their directory. This is a cheap (free!) and easy way of setting up web quizzes and I hope that CHEER readers will consider using the tools provided by Quia to author and store more quizzes in our subject area.
As I discovered when I started to try to create my own on-line web quizzes, the hardest part of the process is creating the questions themselves, not the stage where you put them up on the web. You should definitely not try to do this on-line, but think carefully about what it is you require students to know, and what common misconceptions there might be. Questions should be piloted with a small group of students first, or you should get a colleague to look at them to check that they are suitably phrased.
Although not specifically related to on-line quizzes there is a literature on the problems of using quizzes and multiple choice questions in economics; see, inter alia, Bresnock et al (1989), Bruno (1989), Walstead and Robson (1997). These papers examine whether or not gender, ethnic background or learning styles can affect student performance on multiple choice quizzes. Carlson and Ostrosky (1992) present evidence that student scores on a micro principles test may be affected by the order in which content is presented in the questions.
The Appendix to this article is on a separate page.
Bicanich, E., Slivinski, T., Hardwicke, S.B., and Kapes, J.T. (1997) Internet-Based Testing: A Vision or Reality? Technological Horizons in Education. September 1997. [http://www.thejournal.com/magazine/vault/A1918.cfm]
Bresnock, A.E., Graves, P.E. and White, N. (1989) Multiple-Choice Testing: Question and Response Position. Journal of Economic Education Vol 20 No3 (Summer) pp239-245.
Bruno, J.E. (1989) Using MCW-APM Test Scoring to Evaluate Economics Curricicula. Journal of Economic Education Vol 20 No1 (Winter) pp5-22.
Byrnes, R. and R. Debreceny (1995) The Development of a Multiple-Choice and True-False Testing Environment on the Web. Paper presented at AusWeb95. The First Australian WorldWide Web Conference. [http://ausweb.scu.edu.au/aw95/education3/byrnes/index.html]
Carlson, J.L. and Ostrosky, A.L. (1992) Item Sequence and Student performance on Multiple-Choice Exams: Further Evidence”. Journal of Economic Education Vol 23 No3 (Summer) pp232-235.
Gartner, M. (1998) A Primer in European Economics. Prentice-Hall Europe.
Hackett, S. (1998) Environmental and Natural Resources Economics: Theory, Policy, and the Sustainable Society. M.E. Sharpe, Publisher.
Leuthold, J.H. (1998) Building a Homepage for your Economics Class. Journal of Economic Education, Volume (Summer 1998) pp247-261
Ramsden, P. (1992) Learning to Teach in Higher Education. London: Routledge.
Walstead, W.B. and Robson, D. (1997) Differential Item Functioning and Male-Female Differences on Multiple-Choice tests in Economics. Journal of Economic Education Vol 28 No2 (Spring) pp 155-172.
Williams, J., Browning, P., Brickley, D. and Missou, H. (1997) The NetQuest Project: Question Delivery over the Web using TML Computers in Higher Education Economics Review Volume 11 Issue 2 pp 9 [http://www.economics.ltsn.ac.uk/cheer/ch11_2/ch11_2p9.htm]