The Economics Network

Improving economics teaching and learning for over 20 years

Creative uses of in-class technology

1. Introduction

‘E-learning can be defined as “learning facilitated and supported through the use of information and communications technology”. It can cover a spectrum of activities from the use of technology to support learning as part of a “blended” approach (a combination of traditional and e-learning approaches), to learning that is delivered entirely online. Whatever the technology, however, learning is the vital element.’[1]

1.1 Summary of the chapter

Evidence of several decades suggests that reliance on ‘chalk and talk’ is not the best pedagogic practice but nonetheless one that typically economists tend to rely on.[2] Our experience suggests that we can add value to both our teaching and students’ learning through thoughtfully and creatively employing technology which can complement active learning methods.

The focus of this handbook chapter is to share our ‘practitioner perspective’ and in particular the experiences we have of using a wide range of learning technologies. This chapter draws upon many presentations and workshops that we have run (in particular for the Economics Network) and is guided by positive participant feedback.

Both of us are Senior Teaching Fellows within our Economics department, committed to developing excellence in teaching and enhancing the learning opportunities of our students. We recently participated in a JISC funded project that specifically encouraged us to experiment and trial new technologies.[3] We are now at the consolidation stage, which includes embedding and disseminating our e-learning experiences. The main part of this chapter therefore focuses on these experiences and in particular the technologies that we have adopted and integrated into our ‘in-class’ economics teaching and learning; we also include some ‘out-of-class’ technologies that directly support or enhance the classroom experience. In each case we consider the key features of the technology, include applications across a range of Economics modules and discuss their strengths/weaknesses.

There is considerable variation in terms used to describe e-learning as well as overlaps between the technologies, their applications and accompanying methodologies (Rushby, 2011). This is particularly true for blended (also known as ‘hybrid’) learning and distance (or ‘off-line’) learning. The former, which is defined here as integrating ‘face to face’ with on-line activities, is the primary focus of this chapter; whereas the latter typically involves complete geographic separation between teaching and learning and therefore lies outside the scope of this chapter.

1.2 Motivations for using e-learning in economics

‘The key principles behind effective learning and teaching still apply: whatever the technology or mode of delivery, learning should be the key objective, and pedagogy rather than technology should drive the decision-making.’ (JISC, 2009)

‘Few other disciplines are poised to have their pedagogies so radically affected by technology.’ (Raymond et al., 2008)

Enhancing learning

Technology has the potential to enable interactivity, providing immediate feedback on learning as well as reinforcing and extending classroom-based learning. Evidence suggests that active and collaborative learning enhanced by technology can improve students’ performance and almost certainly engagement and motivation.[4] Such learning encourages greater engagement with concepts and material, facilitating both acquisition and retention, as students assume a more active role (Ghosh and Renna, 2009). Furthermore active in-class activities in particular enhance learning: students reach a deeper understanding of concepts working at higher cognitive levels. Salemi (2009) also observes that interaction with peers fosters both critical listening and appreciating diversity of opinions; he gives a number of examples of in-class activities, many of which can be adapted and run more inclusively with technology in lectures.[5]

We also believe that many technologies can be used to enhance learning whether in a lecture group of 50 or 500. The latter presents a familiar optimisation problem: how to maximise excellence in teaching and learning given the constraints of increasingly large number of students. Research suggests that using technology such as audience response systems with large enrolments on a first year Principles course does lead to increased student engagement (Salemi, 2009).

Improving understanding

An Editorial in 2010 in the International Review of Economics Education (IREE) observed that ‘typically evaluations of teaching in universities currently rely more on students’ views than on evidence of change in students’ understanding’. Our evaluations are in this sense then, typical: although we have observed throughout this three-year period improvements in students’ performance, our research to confirm this is not yet complete. Nevertheless, a number of recent studies seem to clearly suggest that different types of teaching strategies, enhanced by technology, affect the development of critical thinking skills of different groups of students in a diverse way and as such catering for the average student by relying on one teaching model might not lead to the best outcome (Siriopoulos and Pomonis, 2009)

Furthermore, Manochehr (2006) investigated the impact of e-learning according to the different learning styles of students, finding that in some cases learners performed significantly better with e-learning methods. Olczak (2011) evaluated the impact of web-based resources, focusing on student learning and performance, also finding significant positive effects while Raymond et al. (2008) find the use of technology in the classroom for teaching mathematical concepts has a positive impact on knowledge retention and student performance. However they caution that it is essential that lecturers have clear objectives and the technology must be well integrated within the teaching. It is worth emphasising that it is really about how rather than if a technology is used.

Addressing concerns

A common concern of lecturers is the belief that using technologies will be costly, specifically in terms of time. However many of the technologies are very low cost and others, whilst requiring an initial investment, have a rapid pay-off. Furthermore, what we are not proposing is the use of technology and e-learning tools to replace lecturers. We would rather advocate parallel teaching than simply video-streaming lectures (although this can be useful in the short term given unforeseen illness or unexpected buy-outs). Despite numerous initiatives – including in the US nearly two decades of AEA annual conference sessions focused on economics teaching and educational research and in the UK the excellent support and innovations of The Economics Network – the reluctance of lecturers to engage with more ‘complex innovations’ continues. Addressing the question of why this is the case suggests insufficient incentives, high sunk costs, incomplete understanding of the benefits as well as perceptions of high start-up costs (Raymond et al., 2008). Watts and Schaur (2011) conclude that ‘there is some evidence of change possibly due to the availability of newer technologies along with younger faculty, with more emphasis on teaching and learning as well as greater resources and training’.

Our motivation throughout is pedagogy-pull not technology-push. Whether digital technology is ‘changing patterns of learning and leading to a new learning paradigm’ or rather represents the ‘evolution of 40 years of educational theory and practice’[6] we believe that ‘the art of the practitioner remains key to the process of learning’(JISC, 2009). Finally, and importantly, using technologies can potentially make learning and teaching more enjoyable for both lecturers and students.

(See Appendix 2 for Background to e-learning developments within Higher Education.)

2. Main content

The purpose of this section is briefly to present and discuss the main e-learning technologies that can enhance ‘in-class’ teaching and learning as well as some ‘out-of-class’ technologies that directly support or enhance the classroom experience and student participation. As mentioned before, the list of e-learning technologies discussed here is not exhaustive as our focus is mainly on teaching strategies that increase in-class participation and learning.[1]

Table 1 summarises the main advantages and disadvantages of the e-learning technologies covered in this section.

Table 1: Summary of e-learning technologies

E-Learning TechnologyIn-Class (I) / Out-Class (O) Use?Easily used for Group Work?Main AdvantagesMain DisadvantagesFreeware/ Open source available?
Lecture CaptureOXAllows for revision of previous lectures at students leisure 24h a dayPossible drop in student attendanceX
Audience Response SystemsI

Allows for interaction with students in class

Software generates a whole array of automatic very rich activity reports (per class, per session, per student, etc.)

Allows for immediate feedback of students answers

Does not allow for the inclusion of 'open ended' questions

Cost

X
Mobile PhonesI, O

Low cost for University

Allows for both closed and open questions to students

Students must be willing to pay for texts used in classroom context

Feedback may require editing before shown in class

Requires students to have mobile phones to be able to participate in activities

Most mobile providers have free software to manage messages in real time.

Online version also possible, e.g. Poll Everywhere

Portable Digital Video CamerasI, O

Allows students to review their own presentations and improve their presentation skills

Allows students to review other presentations to analyse skills of others

Allows for pre-recording of student assignment/lecturer feedback outside the classroom which might bring flexibility in the use of limited resources.

Cost

Requires large storing capacity in the server as files tend to be very large in size.

Smartphones might be used to some extent but without the same quality and ease of use.
Turnitin and GradeMarkOX

Very powerfull tool to check for plagiarism and collusion in students assignments

Grademark can greatly save time in providing feedback to students and in ensuring that markers follow similar marking criteria

Simplifies the stages of collecting assignments, handing back feedback and publishing the marks.

Cost

As it is a Web-based service it is dependent on broadband speeds and can be rather slow at times.

X
WikisO

Ideal for group work as a wikis were devised to be used collaboratively by multiple users.

Allows to track each individual student's contribution to group work and thus fairer grading of group work.

Technical issues can be a major problem and very time consuming.

As it is an alternative to face-to-face meetings it might not be able to capture all the benefits of group work.

Some institutions have their own systems (e.g. Confluence) but a large number of external websites are available
Learning Logs/BlogsOX

Encourages students to keep their own personal notes and to be reflective on the materials they are studying.

It encourages individual, independent student work.

Allows student to record their work on a weekly basis, to interact with others and to share their thoughts on a specific topic.

Can be very time consuming for lecturers, especially in large classes as it requires the setting up of and the checking up on individual student's work on a weekly basis.Most VLEs allow for Learning Logs to be created. Outside alternatives include blog sites like Blogger.
ScreencastingI,OX

Easily transforms presentations into eLearning content

Engages learners with application simulations, "show me" movies and "try it" scenarios.

Allows to assess learner performance (more expensive versions only)

Cheaper versions are very limited and require to use applications through the web.

Best software solutions quite expensive

Best free resource is Jing
Web-based, textbook-linked resourcesOX

Access to large databanks of questions and materials which have been pre-tested and improved over a period of years

Facilitates the application and management of continuous assessment, homework tasks and study plans

Cost

System based outside university's servers which might create some dependency on a specific textbook

X

2.1 Lecture capture

One of the common difficulties students have in traditional lectures, especially the ones with large number of students where interaction between lecturer and student is more problematic, is the inability to listen carefully to the lecture and simultaneously to take notes. This is especially true for international students, where language and cultural barriers often impede students from following the material being covered and contribute to poor performance.

Recent technological developments have created the opportunity to record lectures so that students can replay them at their leisure on their computers or other electronic devices. Although there are several types of apparatus that allow for the recording of lectures (the portable digital video cameras described in section 2.4 are but one example), the focus here will be on technologies devised specifically for a classroom environment.

The most sophisticated systems for lecture capture are capable of automatically capturing audio, video (from a fixed camera) and any output from the computer (including slides) and other devices connected to the projector such as visualisers, interactive whiteboards or tablet PCs.[1] The video files are stored on a media streaming server and can be made available in various ways including via Virtual Learning Environments (VLEs).

The lecture capture system integrates with existing technologies and lecturers are not required to do anything in order to have their lectures captured as the whole system is totally automated. At the end of the lecture, the recording is automatically remotely stored and a link to the recording is made almost immediately in chronological order within the appropriate unit in the VLE, and an announcement made to students. The lecturer has, however, control on when the lecture recordings are made available to students and while most lecturers make their lecture recordings available immediately, others decide to limit the viewing to the period immediately before the final exam for revision purposes. Alternatively, one could make the materials available for a fixed period after the lecture encouraging students to view them early and keeping them ‘on track’ with the sequence.

Students can watch the synchronised recorded lecture including PowerPoint slides and movie clips as frequently as they like and choose whether they watch using high or low speed connection and whether they view lectures using their ‘smartphones’, tablet PCs, MP4 players, laptops or desktop PCs.

An example of a streamed recording available online is presented in Figure 1.

Figure 1: Example of lecture capture (Echo360)

The use of this technology is quite controversial among academics. The detractors cite a number of potential problems with its use and make their resistance quite clear. Among the reasons commonly put forward are the possibilities that the technology will discourage students to come to class and that the lecture recordings will make lecturers redundant.

Our experience to date is that lecture recordings have no significant impact on student attendance.[2] A student-designed questionnaire involving more than 200 students on the use of ECHO360 in 2011 showed 70% of students stating that the lecture recordings had no impact on their attendance as they felt that they were ‘missing out’ if they did not attend the lectures. Most students (87%) stated that attendance was much more a function of the quality of the lecturer and the module content. This suggests that lecture recordings will not make lecturers redundant. Students like to be in the classroom and take part in the activities; ‘they feel they are missing out if they do not attend, irrespective of video recording taking place’ (Lowe, 2011).

The survey also showed most students using the recordings to go over material that has troubled them and to write up notes. Most students felt the recordings were particularly useful in modules with large mathematics content, in lectures which included case studies and in lectures with large student numbers.

Finally, a very different model for using lecture recording is being increasingly practised and explored in the US. This is known as the Flipped Classroom Model where students watch and listen to recorded lectures for ‘homework’ (asynchronously) and class time is used instead for collaborative, active and experiential learning. Partly this is driven by the desire to optimise use of precious class time (therefore this is less relevant when lectures are supported by tutorials) but largely by the emphasis on active, and often personalised, learning and engagement.

2.2 Audience Response Systems (ARSs)

Audience response systems (also known as ‘clickers’ – see Figure 2) are individual handheld devices that allow interaction between a presenter and his/her audience. Audience response systems normally combine wireless hardware with presentation software (most commonly Microsoft PowerPoint).

Figure 2: An example of an ARS or ‘clicker’

Most audience response systems integrate into Microsoft PowerPoint and allow audiences and students to participate in lectures by submitting responses to interactive questions using an individual clicker. This is done by downloading a free software package that adds an extra tab on PowerPoint, as depicted in Figure 3 below.

Figure 3: TurningPoint Tab in Microsoft PowerPoint

By using an ARS, the PowerPoint presentations become powerful data collection and assessment tools that collect real-time audience responses and dramatically improve student engagement and motivation in the classroom. The advantages of using clickers in the classroom are many and have been well documented before. These include:[1]

  1. improved knowledge retention;
  2. anonymous pooling (as opposed to show of hands) encourages truthful answers and higher participation rates among students;
  3. tracking of individual student performance (and attendance) over time;
  4. immediate results encourage students to learn better and the lecturer to incorporate the results in the class;
  5. promotes discussion and collaboration among students during class with group exercises;
  6. encourages a more interactive and fun learning environment;
  7. diagnostically: confirms audience understanding of key points immediately, signalling the need to go over the areas students did not fully comprehend;
  8. results are gathered in a database that can be used for reporting, analysis and research.

There are a large number of different providers of ARSs and the associated software. The leading companies include Meridia, Reply, TurningPoint, Qwizdom, Interwrite, Poll Everywhere (ARS using mobile phones in place of clicker keypads) and eInstruction.

Lecturers can create new slides in seconds or adapt their existing PowerPoint slides to include questions for students to respond in class. Students then select one of the options available in their handsets and the results are presented instantly in the form of a table or graph, an example of which is shown in Figure 4.

Figure 4: An example of a TurningPoint slide

All entries in each session can be saved in a database and a large number of reports can then be generated by selecting from a large menu of options as shown in Figure 5.

Figure 5: List of reports in TurningPoint

The reports are very comprehensive and are created automatically in Excel spreadsheets. Figure 6 shows an example of a ‘results by question report’ which describes the aggregate results of the class on a question by question basis.

Figure 6: Example of a ‘results by question’ report in TurningPoint

A whole array of other possible reports can easily be generated after each session. Figure 7 shows an example of a ‘result by participant report’ where the individual performance of each individual student is presented in a manner that can easily be sent as feedback to each individual student just after each class.

Figure 7: Example of a result by participant report in TurningPoint

Despite the wide range of different possible applications,[2] clickers do not allow for ‘open questions’ in the classroom. For lectures where open questions that allow for a variety of essay type answers are desired, other alternatives must be sought, including the use of mobile phones which will discussed in the next section.[3]

Audience response systems can also present some challenges for lectures. First, using clickers takes up class time, especially if students do not keep their devices between classes.[4] Next, some technical problems may arise, especially when the technology is first adopted. For example, if students use clickers in different classrooms and settings, there will be a need to synchronise each clicker to the correct frequency.

Also, by finding out what the students do not understand instantly, places more demands on the lecturer to adjust his/her teaching on-the-spot[5] thus demanding a more flexible approach to teaching. Many lecturers also use clickers to encourage class-wide or group discussions. This might prove quite challenging for lecturers who are used to a more traditional lecturing style, as it requires a higher degree of planning and adaptability.

2.3 Mobile phones

The motivation for using this technology was two-fold. First we wanted to use a response technology that allowed greater functionality, specifically enabling textual narrative responses, especially for modules where developing students’ critical thinking skills was seen as fundamental; the clicker system which we use constrains us with a numeric key pad whereas mobile phones enable us to ask open questions. Secondly, the almost ubiquitous ownership of this technology by students suggested it would be useful to explore its potential.

As all our technologies develop and evolve there will be devices that enable all these functions; however the application to learning and the underlying pedagogy remains the same. An attractive feature of using this technology is that it is low cost: all the providers supply freely downloadable packages that allow for the managing of text messages in real time. Also, rapid developments in the mobile phone market have brought about changes in pricing structures where typically texts are now unlimited and free (addressing some of the earlier student concerns).

We have used the mobile phone technology within lectures to ask questions and prompt discussion, with both diagnostic intent and as a formative exercise for students. Other possibilities include using the device to enable students to ask the questions, which can then be addressed by the lecturer. One case study (Jones et al., 2009) focused on using mobile devices for both academic and personal development particularly during transition to HE, including administrative uses such as reminding students of upcoming deadlines as well as recent changes in lecture venues, cancellations, etc.

Example 1:

  • The Philosophy of Economics UG Level 1 (15 credit elective module)
  • Student numbers approx 100+
  • Economics, Economics and Politics as well as PPE students

The first part of the course introduces ideas from within the Philosophy of Science before focusing more specifically on methodological questions arising in economics. An early question, given to students for discussion (along with the mobile phone number to text in the response), was ‘What is science?’ In this case the answers were not viewed in real time but presented back in the following lecture; first by creating visual representations using a word cloud generator such as Wordle[1] as shown in Figure 8 and then as a table (into which the responses were pasted) as seen in Figure 9. The depth and length of responses as well as the lack of ‘text language’ was truly surprising, as was the student’s eagerness in participating in the activity. The responses fell conceptually into different categories which presented a basis for further discussion as well as reviewing and reflecting on ideas from the previous lectures. The follow-up lecture allowed for analysis and discussion of results and responses.

Figure 8: An example of a Wordle cloud

Figure 9: Student text responses to the question “What is science?”
Universally accepted rules that explain why and how things happenI think it is the process of discoveries about the world and new findings, learning about how the world works and operates and the things within it, creating theories and laws etc.The desire to know about the world following a strict methodological framework.

Science is about explaining.

Scientists try to find regularities, establish axioms, through observation and experimentation, in order to analyse, understate and predict.

Science is the process of gathering data in order to make sound analysis so as to be able to make predictions and facilitate decision making when faced with future problemsScience is the ability to falsify a statement / theory through empirical testing
A method where you can isolate and test variables. Scientific theory is without any bias or assumptions that are not founded on evidence.Using observed and proved facts to try and gain a reliable theory that explains those facts.

Science is about explaining.

Scientists try to find regularities, establish axioms, through observation and experimentation, in order to analyse, understate and predict.

Example 2:

  • The Business of Climate Change Level 3 UG (15 credit module)
  • 120 students (from across the Business School particularly Business Economists)
  • This course is assessed by final examination (70%) and written assignment 1500 words (30%)

The aim of this module is to provide students with the opportunity to study how the challenge of climate change can be addressed. At the midpoint of a two-hour introductory lecture students were asked for their ideas on strategies and policies to deal with climate change. We chose to do this before we started the next lectures on mitigation policy instruments in order for us to have some idea of the background to this group (of self-selecting students); also to encourage students to freely share ideas and perhaps be more innovative. We put the question ‘What suggestions do you have for addressing climate change?’ and then viewed the responses in real time which stimulated a lively discussion. We revisited these ideas later in the module, when students could consider them within conceptual frameworks they were developing. This also provided an opportunity for assessment as an assignment question. From a choice of five questions one included: ‘Choose one of the suggestions for reducing climate change risk, made in the first lecture, and explore its feasibility.’ Students were given a brief emphasising rigour and reasonableness, discouraging them from straying too far.

Table 2: Example of student suggestions for reducing climate change risk
Reduce deforestation, switch power production away from fossil fuels to renewable sources; subsidies for use of solar panels, wind turbines and other renewable sources such as less tax.Make businesses liable for the pollution they create. Greener methods of production. Subsidising alternative energy sources.
Price carbon through cap and trade. Have a carbon neutral electricity grid. Incentivise firms for small-scale research and development. Invest in renewable energy.
  1. Carbon credits.
  2. Persuasive measures to encourage consumers to change their behaviour (financial incentives).
  3. R&D in new fuels. Huge investments for R&D into alternatives for both domestic users and business and incentives.
More government money in development of hybrid cars. Tighter restrictions on cars. More alternative transport initiatives like the virus bikes. Raised awareness of immediate issues (scare tactics). Enforced recycling. Solar panels should become more available and awareness raised. More emphasis on energy saving on a local level.Encourage, climate change on the national syllabus. Encourage government investment in sustainability projects and research. Encourage corporations to reduce carbon footprint in more proactive ways, e.g. rooftop gardens on corporate buildings. Nuclear, solar, wind, wave power. Educate LEDC on more efficient farming methods, and cleaner power.
Tax for going over targets and subsides for below. More education/awareness. More effective use of renewable energy resources. Events such as days when everyone tries not to use electricity, unless it is essential.Plants and grasses on roofs to absorb carbon emissions, greater incentives from government or businesses to share cars/provide shuttle buses, generate energy from the use of gym equipment, incentives to buy local produce to reduce food miles.
A general policy of education and economic encouragement; don’t just tell people their actions are bad and set the argument in an apocalyptic tone. Educate the public and encourage it to lead a more environmentally friendly existence instead of tax rises and new laws.Nuclear power. Recycling compulsion. Limit on car size (smart cars). Hybrid car compulsion. Start taxing aviation fuel. Tax on carbon. Higher fuel duty. Ban tumble dryers. Wind farms. Stricter environmental regulations on industry in places like China. Better insulation.
  1. Carbon pricing including tradable pollution permits with exemptions for developing countries through to a certain date.
  2. Grants and subsidies to promote production and consumption of solar, hydrogen cars; ground heat source pumps, etc.
  3. Taxing high emitters.
  4. Educate the population as to the causes, likely consequences and it role in reducing emissions.
  5. Legislate against certain activity and introduce climate acts like the UK 2008 Act.
Carbon pricing – tariffs (re-sellable) perhaps different for developing countries. Government incentive schemes – grants and subsidies for example for solar panels, tax consumer for use of high-emission products. Education on the subject – causes, consequences and solutions. International agreements. Weighted allowances of CO2 emissions against developed countries to take into account the first world’s previous pollution and to allow for developing countries to gain economic growth.
Auction the right to own a car with set quota of licences a year. Double public transport investment and encourage cycling ‐ i.e. grade separate cycle paths. Demolish energy inefficient palaces/mansions/castles. Smaller abodes for less energy consumption. More apartments/urban density for shorter commutes.International carbon markets, cut meat consumption, genetically modify crops to increase crop yields, educate children at a young age to lead low carbon lifestyles, more nuclear power plants, incentives against deforestation, higher CO2 caps for developed counties compared to developing countries, various reengineering strategies, invest lots of money in nuclear fusion

One of the key conclusions of both this chapter and of a large number of previous studies is the need to use a variety of technologies which complement each other. The use of mobile phones in the classroom has the advantage of opening up discussions in situations where most students would not otherwise participate. Students’ anonymity and the ability to have more time to think about the question encourages larger numbers of students to participate. Students tend to enjoy the interaction and the activity helps to maintain concentration and focus. Students particularly liked to see their responses in the context of other students. Initial student feedback on the use of SMS in the classroom suggested that most students were enthusiastic and their responses were very similar to those when asked to review use of the ARS. The students also stated that they benefited from the use of a variety of different systems and teaching methods.[2]

2.4 Portable digital cameras

There are many uses and applications of filming with (and by) students; video, as one type of visual learning tool, can be used to accelerate thinking, understanding and learning.[1] The original motivation for using mobile video devices (in the case study presented in this section) was to facilitate the summative assessment of group work. It became apparent very quickly, however, that there were many additional advantages to using this technology which in turn further developed practice. The very act of filming students’ presentations instils high expectations of professionalism which are shared by all participants.

‘There is nowhere to hide when the camera is running’ (Student comment)

‘It gives students who want to be really good permission to shine.’ (Lecturer comment)

Furthermore, the medium has an immediacy which integrates well with peer review processes, involving active learning activities designed to encourage critical engagement and reflection. This can improve the quality of feedback that students provide in the role of assessors, which then correlates positively with the quality of the students’ final projects.

Many types of devices can be used but there are advantages to dedicated mobile digital video cameras as they combine long-lasting battery life, high-definition 1280 x 720 resolution, a pop-out USB connector that slots directly to a PC and a large memory (rather than using other equipment with video cameras included as a feature). Mobile phone cameras for example raise issues of ownership and privacy plus challenges of variety, compatibility, quality and data storage capabilities. Specific digital video cameras, issued for the duration of the group work projects and with nominated students responsible for them, are highly recommended.

Example 1

Using this technology, to film group work presentations, provides an invaluable record for the lecturer to revisit as well as evidence for moderation and second marking. The use of these devices to record weekly group work in a large first year ‘Management Concepts and Practices’ module[2] showed a notable increase in the standard of the student projects: qualitatively in terms of the depth of critical analysis evident in the written outputs as well as the videoed content. Students invested a lot more in the project both in terms of the breadth and depth of relevant research and presented more substantial pieces of work (the challenge now being word limits) that included an impressive range of presentational techniques. In addition, requiring students to prepare questions to ask following the presentations, which are also filmed, encourages all students in the group to engage more deeply with the concepts and material.

Another particular benefit was to support and develop the collaborative ethos which is integral to successful group work.[3] This is important as a common challenge is ‘free riding’ – or at least the perception of this occurring. Knowing their contribution will be recorded and then reviewed, both by other students as well as the lecturer, we find that all group members are keen to get involved – we observe an improvement in attendance as well as contributions. These are large and diverse cohorts of students who come with different prior experiences of learning; in order to get them to engage with the module the lecturer was concerned to foster a sense of community and strong group relationships.

‘When I was presenting my first presentation, I was too nervous even to make a sound… I am now able to contribute ideas and discuss opinions with my group.’ (Chinese student)

Each group was also required to engage in a formal peer review process of the presentation (a weighted element of the final assessment); it is essential that all elements are well documented and the video is critical. Feedback and ‘feed forward’ comments are received and evidence of action is later recorded. Marking criteria are defined to include the presentation element as well as engagement with audience to encourage students to take the presentation aspect seriously (typically additional academics should also be invited to attend). This technology seems particularly relevant when developing professional skills as managers is the goal, whilst maintaining academic content.

It was an exhilarating experience as it provided us with a unique opportunity of looking back and reviewing the mistakes… Despite the difference in culture and language I have been fully supported by all of the group.’ (Vietnamese student)

In the context of this Management module, with its focus on theory and practice, the use of portable video cameras for group work is integral to the assessment design. It enhances the learning process through deepening engagement as well as formatively providing input. For the technology to support the learning (and not distract) it needs to work smoothly, which necessitates careful planning and organisation. This was managed by designating willing students as the ‘e-learning champions’ (see Learners’ perspectives, below), responsible for managing the portable video camera and the uploading of the videos for the semester. This degree of engagement greatly facilitates the management of the process and encourages student participation and motivation.

One of the challenges of the extensive use of this technology (apart from the cost) arises from the need for ever-larger server space to store the video clips. When testing this technology for formative rather than summative assessment, student engagement was not found to be as high and thus careful consideration needs to be taken in deciding what uses (and in what type of modules) this technology can be successfully applied.

A recent student research project suggested that international students indicated a stronger preference for video technologies than UK/EU students. Although the reasons for this difference are not yet totally clear, one possible explanation might be that international students value the ability to revisit video materials to enhance understanding of discipline specific language.

2.5 Turnitin (plus GradeMark and PeerMark)

Turnitin is a well-known and market-leading web-based, subscription-only suite of tools that includes plagiarism detection, management of formative and summative assignments, digital (paperless) grading and feedback, and peer commenting. Turnitin is most famous for its plagiarism checking service as it is a very powerful tool in detecting collusion and plagiarism. When a student submits a paper to Turnitin, it is checked against a database of previously submitted student work around the world and an extensive variety of online and print research sources. If the system finds passages of identical text, it flags them for the instructor to review. An example of a Turnitin originality report is presented in Figure 10.

Figure 10: Example of a Turnitin originality report

Less well known features of Turnitin are the GradeMark and PeerMark features.

The GradeMark component of Turnitin gives instructors a fully-featured digital environment for grading and commenting on student work.[1] After grades are posted by the instructor, students can access GradeMark to review comments and print or save a copy of the graded files.

An example of GradeMark view is presented in Figure 11.

Figure 11: An example of GradeMark view

After the assignment has been marked using the rubric scorecard (see Figure 12), automated feedback is generated that explains to the student the reasons for his/her grade. In addition, specific feedback can be added, either throughout the assignment (comments will appear in a different colour and can be typed over the original text) or at the end of the feedback report.

Figure 12: An example of rubric score card-based feedback

PeerMark is a functionality that allows the assignments to be peer reviewed so that students can evaluate each other's work and learn from their classmates. This will develop students' critical thinking skills and empower them to share and help other students in the classroom. An example of the PeerMark view is presented in Figure 13.

Figure 13: An example of PeerMark view

Source: Turnitin.com

2.6 Wikis

Wikis first appeared in 1994 when Ward Cunningham invented the concept and gave it its name. A wiki is a website whose content any authorised person can modify, without requiring any knowledge of web publishing software. Wikis are widely known and used and have, in the past few years, been more widely adopted in the academic world.

The philosophy of wikis is to be an open collaborative space and to encourage collective intelligence. For these reasons, wikis can be an important tool for group work where students can share their reflections and learning from a lecture/text/assignment, and undertake group assignments by collaborating on shared documents and submitting a final joint report/assignment.

The most common problems of group work can be minimised by using a wiki. Common problems in group work include misunderstanding about individual member’s responsibilities, perceived lack of commitment of some members, the issue of free-riders, issues about fair grading of each individual member and personality and timetable clashes.[1]

The organisation of group work through a wiki allows for the tracking of each individual member’s contribution to the final assignment, thus greatly eliminating the incentive to free-ride on the other members of the group. Also, being a digital meeting room, students can work at a time of their own choice and do not always have to meet physically in order to work on the common project. As noted on the JISC infoNet website, ‘wikis give an opportunity for users with different types of knowledge, confidence and communication to contribute equally to a joint publication. Collaborators who lack confidence to argue a case in a “live” face-to-face debate can feel more comfortable making the same points in a wiki environment where the pace of discussion is slower and the quality of the thinking is more significant than force of personality. Wikis can be particularly helpful at including people who are shy, or who have hearing or communication difficulties.’[2]

As with blogging, there are many different wiki creating and hosting services available. Wikis can either be created and managed by external, mostly open service providers (e.g. Wikispaces) or internally in the university’s IT systems. The latter tends to be the preferred option of most universities where there may be a greater emphasis on features like access control (and data protection), integration with other internal software, and document management. Most VLEs provide basic tools for the creation and management of wikis but many prefer to house their wikis on a dedicated server and use specialist software that allows for a higher number of functionalities and uses (Confluence, Socialtext, Jive SBS, SamePage, and Traction TeamPage). Figure 14 presents an example of a Confluence Wiki system.

Figure 14: An example of a Confluence wiki system

A separate wiki can be created for each group and students’ access can be limited to viewing their own group’s wiki. Inside each group’s wiki, sections for webpages, blogs, labels, attachments, activities and others are available to group members and lecturers for guidance. An example of a group’s wiki page is presented in Figure 15.

Figure 15: Example of a wiki for group work

Group members will work together, editing the same comments and adding their input over time to the file containing the final report. As all students work in the same document, only one version of each document exists at any one time. Figure 16 shows an example of a final group report.

Figure 16: Example of a final group report

As the wiki management system is integrated in the university’s IT system, the final report can be checked for collusion and plagiarism with the Turnitin software (see section 2.5 above).

Even though the use of wikis is now quite widespread and its proponents are many, a number of challenges also exist. Technical issues can be a major problem and often require a lot of technical assistance. The creation and management of a large number of wikis (groups) requires a large input by the lecturer(s). Monitoring each group’s work and assessing each student’s contribution to the final report is very time consuming despite all the tools available. An alternative is to use a single wiki and then ask groups to create their own pages. As all changes are recorded, it is still easy to determine the individual’s contributions and, by comparing time stamps, to see if students are taking ideas from other groups. Also, wikis often generate competition between members of the team but do not necessarily lead to better work.

2.7 Learning logs

Learning logs are diaries students keep that record their reflections about what they are learning and how they are going about learning it. Learning logs are useful because they promote reflective learning. They are also useful tools for educators because they can reveal students' perceptions (and misperceptions) of the information, as well as reveal how they are reacting to the way the material is being taught.

Learning logs can be considered identical to individual blogs (whilst group or common blogs are similar to wikis, discussed in the previous section). Learning logs/blogs can be easily created and integrated in the university’s VLE or created and administrated in an external website, much like wikis. Again, most institutions tend to prefer to host the learning logs internally as it makes access control simpler and data management more straightforward.

An example of a Learning Log is shown in Figure 17.

Figure 17: Example of a learning log

Students respond best to learning log activities when provided with some structure. For example, the lecturer can provide a set of ‘guiding questions’ that students can select from and respond to on a regular basis. Learning logs with prompts can guide students by asking questions which in turn will help students to activate background knowledge, i.e., to bring to mind and state, write down or otherwise record what they know. For this reason, learning logs can be an effective tool to promote continuous and reflective learning. An example of an entry in a learning log is presented in Figure 18.

Figure 18: Example of an entry in a learning log

Learning logs (and blogs) share the same disadvantages with wikis. However, as learning logs are individual by nature, the amount of work required of the lecturer is even larger here, especially if learning logs are to be used in summative assessment – although one strategy might be to ask students to create a small portfolio of their posts. As mentioned above, for best results the lecturer should regularly prompt students by adding new questions and comment on individual students’ entries which will logically be higher in number for learning logs than wikis.

Finally, there is the question of the degree of openness of these websites: should they be open to all others in the class or open only to the lecturer and student? There are potential benefits to some degree of accessibility but whether they are used formatively or summatively may be an essential consideration.

2.8 Screencasting

A screencast is a digital recording of computer screen output, also known as a video screen capture that can be accompanied by audio narration and music and written descriptions. A screencast is essentially a movie of the changes over time that a user sees on a computer screen, enhanced with audio narration.

A number of companies provide software solutions for creating screencasts with varying degrees of functionality and price, including Adobe Captivate, Camtasia and Jing.[1]

The most comprehensive screencasting software is provided by Adobe Captivate. This solution allows not only the creation of screencasts but also a whole variety of other interactive e-learning content. With Captivate, existing Microsoft PowerPoint presentations can be easily adapted and made more engaging with rich media, application simulations, branching scenarios and quizzes. Furthermore, it allows monitoring student progress by embedding a wide variety of interactive ‘test me’ quizzes in the content.

After selecting the type of content to create and the audio features, the recording starts and every action within the selected windows will be recorded and a movie be created. An example of the options available is presented in Figure 19.

Figure 19: Adobe Captivate project options

After selecting the type of content to be created, the software builds and edits interactive software demonstrations, simulations, podcasts, screencasts, program demos and lessons with no programming or multimedia skills required. An example of a demonstration is presented in Figure 20.

Figure 20: Example of a Screencast demonstration

One of the main advantages of screencasts is the ability to answer common students’ queries in a dynamic, step-by-step fashion that avoids having to repeat the same explanation a large number of times. Students can see the screencast as many times as they wish and decide on the ideal pace to follow the explanation. The screencasts are also a powerful tool for revision and for catch-up if a student misses a lecture or tutorial.

2.9 Web-based, textbook-linked electronic resources

The use of web-based, textbook-linked electronic resources allows lecturers to save time when preparing resources for a module and when giving feedback to students. This will be particularly useful for first year modules that typically have large numbers of students and are composed of introductory modules whose content does not need to be updated from year to year in an major way. Adopting these resources also allows the possible use of resources that are cleverly linked among the lectures, the course work and the textbook.

Most of the same resources can be created by the lecturer and incorporated in the institution’s VLE. This would, however, require a big commitment in terms of time and well developed IT skills. Furthermore, the publishers’ resources tend to be more advanced than the resources available to us in the VLEs. This is particularly true in terms of the type and quality of feedback provided to students and the dynamic aspects of the electronic resources (being the possibility to create algorithm-generated questions with unique answers for each student or problems that allow students to respond graphically and get automatic feedback if they get it wrong).

Nowadays, most publishers invest heavily in online assignment and assessment systems, customised to particular textbooks that are available on option and at an extra cost to students as this is seen as a strong selling point. Typically, these systems use the exercises from the textbook both in a static as well as dynamic fashion. In the first instance, we can find randomly ordered multiple choice quizzes where questions, answers and feedback are fixed and the same for all students. The second type allows for the creation of problems that are unique for each student where pre-set algorithms generate multiple data possibilities, answers and feedback. Lecturers can in general, have a large degree of control over the resources at their disposal. This includes, not only the possibility to edit and adapt existing questions but also to add original content to the system.

The most notable examples in the Economics area are Pearson’s MyEconLab, Mc-Graw Hill’s Connect, Cengage’s Aplia and Wiley’s Wiley Plus. The University of Exeter Business School has used the first of these in two large first-year modules, Economic Principles and Economics for Business, with a combined intake of around 1000 students, for a number of years. Therefore, this section will focus on this system (but as stated above most publishers have equivalent products).

In the first lecture of the year, students are encouraged to purchase the textbook which contains the access code for MyEconLab. Although the use of this system is not compulsory, around 90% of the students enrolled on the course used MyEconLab. The module coordinator sets homework and a test every week, which are automatically marked. Students are encouraged to complete their homework tasks (which are formative assessment) as these serve as preparation for the weekly tests and the final exam. While in completing the homework students can go back over the questions and get feedback and a step-by-step breakdown of the problems when they get questions wrong; when completing the weekly tests (which are part of the summative assessment) students only have one chance. An example of a MyEconLab course home page is presented in Figure 21.

Figure 21: MyEconLab course homepage

The homework and tests appear in the students’ study plans, which allow students to follow their progress during the course and the lecturers to monitor student progress online and so gauge the level of understanding of the majority of students on each topic, and thus signal areas that require further explanation and practice. Although no formal empirical analysis has been completed on the results of using MyEconLab (and therefore no causality can be established), the evidence to date shows that students that made the best use of MyEconLab, who completed the homework and tests, and used the ‘ask my instructor’ link, were those who achieved better results in the final exam. An example of a student’s individual study plan is shown in Figure 22.

Figure 22: MyEconlab individual study plan

The use of textbook-based resources has caveats. As access to these systems requires the purchase of an individual access code per student per term, the access to it has to be either voluntary (and therefore cannot easily be used for summative assessment) or the university has to purchase the access codes for all students. In a context of rising fees for UK students, we must all ask ourselves whether post-2012 students will be as willing as those before them to part with additional funds to pay for these systems for their assessment and feedback. Furthermore, as these platforms tend to be ‘stand-alone’ technologies, functioning outside the existing VLEs, lecturers do not have full control of contents (even if publishers allow some degree of flexibility by allowing lecturers to customise the resources) and are dependent on the publishers to update contents and to provide timely access to all students. For those reasons, we believe these systems are more relevant for first-year modules whose content does not change frequently, as typical introductory first-year modules tend to be.

3. Learners' perspectives

Turning now from our practitioner experiences, it is important and often illuminating to consider the learners’ perspectives on using technology to enhance the educational experience. There is an emphasis in current pedagogic research on characterising and understanding the ‘student experience of learning’.

The Higher Education Funding Council for England (HEFCE) recently commissioned a report, ‘Student perspectives on technology – demand, perceptions and training needs’ (NUS, 2011). The research highlighted a number of trends including that student opinions are divided over the benefits of e-learning. Students felt that technology should be integrated and complementary to their learning; they do not want technology to replace traditional teaching (in particular as a cost-cutting measure). Further, students are concerned about the ICT competency of lecturers, thus identifying training needs. Students’ expectations are certainly increasing and continue to do so in the light of increased tuition fees.

‘These new technologies not only give an opportunity to refresh teaching methods, but also act as a starting point for more flexible, personal and improved learning environments.’ (NUS, 2011)

Case study: ‘Students as change agents in technology’

‘The concept of “listening to the student voice” – implicitly if not deliberately – supports the perspective of student as “consumer’, whereas “students as change agents” explicitly supports a view of the student as “active collaborator” and “co-producer”, with the potential for transformation.’ (Dunne et al., 2011)

Evolving from the ethos and framework of The University of Exeter ‘Students as Change Agents’ project,[1] partnerships between Business School staff and students emerged to address the challenges of using technology with large and diverse cohorts. The intention of this framework is to move beyond responding to student feedback and evaluations to the direct involvement of students in actually bringing about change.

Students carried out research (Lowe, 2011) into the views and perceptions of students which led to recommendations and solutions for practice. The need for greater and more consistent use of available technologies to facilitate their learning is highlighted (expressing frustration if this does not happen). This has led to proposals that the Business School adopts a policy which requires lecturers to use ‘lecture streaming’ when it is available. Students also identified the need for institutional support in terms of investment as well as training programmes for staff. This led in turn to the ‘Technology Enhanced Learning Champions’ project where a number of recommendations are being followed through including the use of student mentors to support lecturers with their adoption of e-learning technologies.

‘I think the most exciting part of this project has been that it is possible to make a change even at such a large institution. Before this project I did not expect it to be possible to make a change, but this has shown me that with a little work and dedication you can make things happen’. (Student)

Figure 23: A theoretical model for students as change agents

Source: Dunne et al., (2011)

Notes

Section 1

[1] See the JISC website for publications, programmes and projects http://www.jisc.ac.uk/whatwedo/themes/elearning.aspx.

[2] Watts and Schaur (2011), analysing the Fourth National Quinquennial Survey (US), are struck by the consistency of these results over time.

[3] For more details on the JISC project see Appendix 1.

[4] Mann and Robinson (2009) show evidence that students also find such lectures less boring.

[5] The etc project (link now broken) propose principles for teaching involving the use of reflective, problem-focused and threshold exercises which also can be enhanced with technology.

[6] See for example the views of Christine Redecker (2009) compared with Paul Nicholson et al. (2007).


Section 2

[1] We will not, therefore, discuss here typical out-of-class, stand-alone technologies that, despite the fact they might have a role in teaching, were not specially designed for teaching (e.g. Skype, Flickr, Twitter, Facebook, etc).


Section 2.1

[1] The Echo360 system is the market leader for lecture capturing systems. Although Echo360 is a proprietary product, there are free (but less sophisticated) open source video streaming tools available.

[2] See White (2009) for similar experience to ours plus using ‘clickers’ (see section 2.2) also motivated attendance.


Section 2.2

[1] For detailed examples of clicker applications in the specific case of economics degrees see for example Ghosh and Renna (2009) and Johnson and Robson (2008) (who also introduce a cautionary note). Plus clickers can be used to facilitate classroom experiments, especially with large groups. See http://projects.exeter.ac.uk/feele/

[2] Derek Bruff of Vanderbilt University, for example, lists the following type of applications: Recall Questions; Conceptual Understanding Questions; Application Questions; Critical Thinking Questions; Student Perspective Questions; Confidence Level Questions; Monitoring Questions; Classroom Experiments. For more details go to https://cft.vanderbilt.edu/guides-sub-pages/clickers/.

[3] Some of the more advanced (and expensive) clickers include LCD screens that allow for the input of text as well. As the cost is much higher, however, this is not an easy option to consider, especially for large classrooms. Other alternatives include web-based polling applications (e.g. ResponseWare or PollEverywhere) but as these require that students have mobile phones (and especially smartphones), this is an option that does not guarantee the whole class participation and might lead to the alienation of some students. These options, therefore, will not be formally considered here.

[4] At Exeter University, all first year students are given a clicker in their welcome pack, which they will carry with them for the period they study. This saves precious time and encourages lecturers to use clickers more often, which in turn encourages students to always carry their devices with them.


Section 2.3

[2] There is increasing interest in many forms of ‘mobile-learning’ or ‘m-learning’ (i.e. using any type of mobile or handheld device), including research and debate on the value of ‘m-learning’ in developing countries where the proliferation of mobile technology is having a huge impact.


Section 2.4

[1] See the discussion in the Special Issue on Visual Learning in Higher Education (Bligh et al., 2010) and the work of the Visual Learning Lab, University of Nottingham.

[2] Core module for the University of Exeter Business Economics programme.

[3] There was one instance of an international student who (for family reasons) had returned home; she was keen to continue participating and therefore she filmed, recorded and uploaded her contribution which was then included in the final presentation.


Section 2.5

[1] Some Learning Management Systems (see, for example, Instructure Canvas) provide this functionality. Canvas automatically sends both instructor comments and graded files to students.


Section 2.6

[1] For a more detailed discussion on the common problems of group work, see for example Davies (2009).


Section 2.8

[1] There is a shorter, free version of Jing which might be useful for the uninitiated to explore.


Section 3

[1] See Dunne, L., Zandstra, R. and ESCalate (2011) Students as Change Agents – New ways of engaging with learning and teaching in Higher Education, ESCalate; other outputs can be found at http://projects.exeter.ac.uk/integrate/saca.html

References

Beetham, H. (2008) Review: Design for Learning programme phase 2, JISC commissioned report, online at  
http://www.jisc.ac.uk/media/documents/programmes/elearningpedagogy/helenbeethamreviewphase2.doc

Browne, T., Hewitt, R., Jenkins, M., Voce, J., Walker, R. and Yip, H. (eds) (2010) Survey of Technology Enhanced Learning for higher education in the UK, Universities and Colleges Information Systems Association, University of Oxford, Oxford.

Bligh, B., Wiesemes, R. and Murphy, R. (eds) (2010) Special Issue on Visual Learning in Higher Education, International Journal of Media, Technology and Lifelong Learning, Vol. 6, Issue 1.

Davies, W. M. (2009) "Groupwork as a form of assessment: common problems and recommended solutions" Higher Education, Vol. 58, Issue 4, pp. 563-584. https://doi.org/10.1007/s10734-009-9216-y

Falconer, I., Beetham, H., Oliver, R., Lockyer, L. and Littlejohn, A. (2007) Mod4L Final Report: Representing Learning Designs, JISC Design For Learning Programme.

Ghosh, S., Renna, F. (2009) "Using Electronic Response Systems in Economics Classes" The Journal of Economic Education, Vol. 40, Issue 4, pp. 354-365.  
https://doi.org/10.1080/00220480903297651

HEFCE (2009) Enhancing learning and teaching through the use of technology: A revised approach to HEFCE’s strategy for e-learning, typescript available at http://www.hefce.ac.uk/learning/techlearn/.

JISC (2009) Effective Practice in a Digital Age: A guide to technology-enhanced learning and teaching

Johnson, M. and Robson, D. (2008) ‘Clickers, Student Engagement and Performance in an Introductory Economics Course: a Cautionary Tale’, Computers in Higher Education Economics Review, Vol. 20, Issue 1, pp. 4–12.

Jones, G., Edwards, G. and Reid, A. (2009) "How can mobile SMS communication support and enhance a first year undergraduate learning environment?" ALT-J: Research in Learning Technology, Vol. 17, No. 3, pp. 201–218  
https://doi.org/10.1080/09687760903247625

Lowe, T. (2011) ‘Technology in Learning & Teaching – Desirable or Destructive?’, Unpublished dissertation. Interview available http://projects.exeter.ac.uk/integrate/

Mann, S. and Robinson, A. (2009) ‘Boredom in the Lecture Theatre: An Investigation into the Contributors, Moderators and Outcomes of Boredom Amongst University Students’, British Educational Research Journal, Vol. 35, Issue 2, pp. 243–258.  
https://doi.org/10.1080/01411920802042911

Manochehr, N-N. (2006) ‘The Influence of Learning Styles on Learners in E-Learning Environments: an Empirical Study’, Computers in Higher Education Economics Review, Vol. 18, Issue 1, pp. 10–14.

Mayes, T., Morrison, D., Mellar, H., Bullen, P. and Oliver, M. (eds) (2009) Transforming higher education through technology-enhanced learning, The Higher Education Academy, York.

Mayes, T. and de Freitas, S. (2004) JISC e-Learning Models Desk Study – Stage 2: Review of e-learning theories, frameworks and models, typescript available at http://www.jisc.ac.uk/uploaded_documents/Stage%202%20Learning%20Models%20%28Version%201%29.pdf.

Nicholson, P., Fernandez-Majon, B., Sanchez-Perez, J.M. and Bravo-Rodiguez, J. (eds) (2007) ‘A History of E-Learning: Echoes of the Pioneers’, Computers and Education, XI.  
https://doi.org/10.1007/978-1-4020-4914-9_1

NUS (2011) "ICT: Students and Technology"

Olczak, M. (2011) Evaluating the impact of a web-based resource on student learning, April, typescript available at http://ssrn.com/abstract=1822665.

Raymond, F., Raymond, A. and McCrickard, M. (2008) ‘Stuck Behind the Math: Just How Helpful Can One Expect Technology to be in the Economics Classroom?’, International Review of Economics Education, Vol. 7, pp. 63–102.

Redecker, C. (2009) Review of Learning 2.0 Practices, JRC Scientific and Technical Reports, typescript available at https://publications.jrc.ec.europa.eu/repository/handle/JRC49108.

Rushby, N. (2011) ‘Editorial: Trends in learning technologies’, British Journal of Educational Technology, Vol. 42, Issue 6, pp. 885–888.  
https://doi.org/10.1111/j.1467-8535.2011.01240.x

Salemi, M.K. (2009) ‘Clickenomics: Using a Classroom Response System to Increase Student Engagement in a Large-Enrollment Principles of Economics Course’, The Journal of Economic Education, Vol. 40, Issue 4, pp. 385–404.  
https://doi.org/10.1080/00220480903237950

San Diego, J., Laurillard, D., Boyle, T., Bradley, C., Llubojevic D., Neumann, T. and Pearce, D. (2008) 'Toward a user-oriented analytical approach to learning design', ALT-J: Research in Learning Technology, Vol. 16, Issue 1, pp. 15–29.  
https://doi.org/10.1080/09687760701850174

Siriopoulos, C., Pomonis, G. (2009) "Selecting Strategies to Foster Economists’ Critical Thinking Skills: A Quantile Regression Approach" International Review of Economics Education, Vol. 8, Issue 1, pp. 106-131

Watts, M. and Schaur, G. (2011) ‘Teaching and Assessment Methods in Undergraduate Economics: A Fourth National Quinquennial Survey’, The Journal of Economic Education, Vol. 42, Issue 3, pp. 294–309.  
https://doi.org/10.1080/00220485.2011.581956

White, B. (2009) ‘Analysis of Students’ Downloading of Online Audio Lecture Recordings in a Large Biology Lecture Course’, Journal of College Science Teaching, Vol. 38, Issue 3, pp. 23–27.

Appendices