Assessments in the Brave New World: A Reflection on 24/48h Assignments

For most UK universities, the exam session takes place in the period May-July. This year, due to the COVID19 pandemic and following lockdown measures, we all had to move our assessments online.

For two of my modules, I opted to go from invigilated 2-hours exams to 24/48h assessments. In this article, I describe the experience in setting these assessments and compare this year results with previous years. I reflect on the experience and why I plan to adopt this assessment in the next academic year when assessments are likely to happen remotely again.

The discussion is structured as follows: section 1 explains the structure of both assessments and provide basic considerations for assessment design using this year’s assessments as an example. Section 2 summarises some of the results of the survey and shows students’ comments on the various assessments. In Section 3 I briefly comment on poor academic practice and conclude.

1. Assessment structure

Before moving to remote assessment in March 2020, the final assessments for the two modules I coordinate were a 2-hour exam worth 70%. Each module is taught by two lecturers. There are around 70 students in module A (Year 2) and nearly 30 students in module B (Year 3).

When moving the assessments online, we tried to maintain a similar structure to the invigilated exams. For module A, the original exam had three sections. Section A (30%) had three compulsory, short-answer questions. In Section B and Section C (35% each) students had to pick one long-answer question from each section (there were two to choose from). In the online version, we maintained the three sections, but Section A had two short-answer questions only, and there was one question for Section B and one for Section C both required longer answers (i.e. we removed the choice). From the time when the questions were released, students had 24h to submit their answers.

In module B there were two sections in the original exam (one for each lecturer’s material), students had to answer two out of three questions in each section, all carrying equal weight (25 marks each). In the online version, students still had to answer two questions out of three in Section A (25 marks each), and one question (out of two) in Section B (50 marks). Students had 48 hours to submit their answers.

For the academic year 2020/21, the structure of the 24/48h assessments will be maintained. First, we believed this structure worked very well this year; second, this will offer some continuity. Students use past exams to practice during revision, and therefore, they will find useful to have use this year assessments to familiarise themselves with the structure and as practice.

The main consideration when setting an assessment is how this allows students to demonstrate their engagement with the module’s learning outcomes. Assessments should assess that all skills, knowledge and attitudes are measured, and nothing is over/under-valued. Therefore, before deciding for a 24/48h assessment, we need to consider their link with the learning outcomes.

Once we have decided that a 24/48h assessment is suitable for the module’s learning outcomes, we need to design the questions. Using this year’s assessments, I offer some reflections on how I set the questions. Example 1 shows two of the questions in the 24h assessment.

a. Clear word limit

As you can see in Example 1, all questions had word limits. Setting clear word limits is very important as it helps students and markers. Students have a clear idea on how much they have to write for that specific question and avoid copying and pasting all the material they may think it is relevant. Markers on the other hand, do not have to deal with very long answers, likely to be unfocused and unstructured.

Setting a word limit also means that students have to be selective on what they write.The first two questions had a very short word limit, which can be very challenging to stick to if you are not familiar with the topic. In order to ensure that this was feasible, I wrote at least two versions of an answer under 100 words. In general, students did not seem to find problems with the word limits, but the quality of the answers varied quite a lot.

b. Google your questions!

Given that it is likely that students use search engines to look at information, you may as well google your question (and/or keywords) and see what is possible for them to find. When looking in the search engines for Question 3 in Example 1, I found some reports from the European Central Bank on negative interest rates which could be very useful to answer the question. However, students had to relate this to the UK experience, which means they could not just copy and paste from these documents.

In general, remote assessments are open book. However, even if students look for information on internet, those who have a better understanding of the topic will select the information in a way that provides a clear answer. Those less familiar with the topic are less likely to be critical of the work they find using search engines. Remember that they need to be selective and critical given the word limit they have.

c. Time frame: 24 or 48h?

How long to submit? The choice between 24 and 48 hours was not arbitrary. This should depend on what it is expected from students to answer the questions. In my specific cases, the difference between 24h and 48h was based on how much research the students had to make to complete the assessment. Example 2 shows one question in Section B. Unlike questions in Module A, in this assessment students were invited to provide some evidence for their policy report (instructions on how to write a policy report were sent in advanced). Therefore, in order to allow for enough time to look for evidence, understand how this can help to support their policy recommendations, and write down the report, we decided to give 48h rather than 24h.

More in general, it is important to understand that 24/48h is not the time that students should work on these assessments. The assessments were designed to be completed in 2-3h if students worked on these non-stop (replicating what they would do in an invigilated room). However, we strongly encourage students to take breaks, think about their answers and their structure, and critically analyse the information they want to include in their answer.

Assessments that are too long may affect how students engage with the assessments and may encourage more rather than less engagement in poor academic practice as students may not be able to finish the assessment on time and may just copy from others.

d. Open questions, no one answer

All the questions were designed in a way that there is no a unique answer(especially in the long answer ones). Students had to explain what considerations they made in order to provide their answer. This fosters critical thinking as they have to be selective on how to propose their argument.

One of the common concerns among lecturers is how to limit students’ collaboration. For these assessments, I did not encourage or discourage collaboration. I provided guidelines on what was required to do well in the assessment and explained the positive and negative aspects of collaborating with others. One of the positive aspects is that working together can help to see different aspects that they may not have considered on their own. However, they needed to be careful not to share any final work, as if this was copied by another student, they both will get penalised. All assessments go through the plagiarism software (this is the case for all submissions), students were aware of this. Therefore, copying verbatim from each other or other sources was strongly discouraged as they would get a penalty (that may be a mark of zero in the exam).

I also explained that they could consult documents that they found via search engines or any lecture material, however I asked them to cite the sources. For short answer questions, I strongly advice against using and citing external sources as they may not have enough words to show their own understanding. Despite this, some students still cited external sources (these tended to be weaker answers).

e. Assessment forum

I set up a forum for students to post their questions about the exam.This was open during the whole revision period and during the exam session. I did not answer any exam related question using private emails, and all students were advised to put their questions in the forum, so everybody had access to these.

The forum was generally well-used, but mostly to ask clarification questions.The aim of the forum was also to facilitate communication among students, but this did not happen and most of the interaction happened between students and lecturer. Despite this, all of them could see my answers to their colleagues’ queries meaning no one had extra information.

I cannot say how much collaboration there was in general among students (e.g. using other means). I assume there was some sharing of information and ideas, however the variety of answers make me think that there was a lot of individual work too. The answers differed a lot from one student to another. In Module B, the policy advice provided by students was so diverse that I believe any communication among students was mostly positive and was a real exchange of ideas and opinions than then students developed on their won, rather than students copying each other’s ideas.

2. Students’ perception of the assessments

The academic year 2019/20 was a big challenge for students and staff alike. Despite all our efforts on trying to smooth their assessment experience, the disruption caused by COVID19 was significant and it is difficult to quantify how much this affected student’s performance.

At the end of the assessments (but before the results were released), I run a short student survey on students’ views on the assessments. Out of 76 registered to sit these two modules, 40 students filled the survey (I did not ask to specify which of the two modules they took).

Table 1 summarises the results for two questions on the perceived difficulty and level of stress of remote assessments. Unlike we may expect, it is not clear that students find remote assessments less stressful and easier than invigilated exams. Very few of them (8%) said remote assessments were easier and found these assessments to be less stressful. There is a correlation between the level of difficulty and stress: 22% of respondents found that compared to invigilated exams,remote assessments were more difficult/same difficulty and more stressful, 18% said they were same level of stress and difficulty.

The question also allowed students to add their comments under “depends”. Several comments emphasised IT issues as one of the main challenges. This was also highlighted when asking students their preferred assessment method (Table 2). Invigilated exams were the preferred method over online/remote assessments (28% against 19%), however a very large proportion of students selected “it depends”. A common explanation for “depends” was due to the challenges faced when uploading equations and formulas into the online exams, especially in those exams with a short time for completion (2-3 hours). For instance, students commented:

Comment 1: “I think they[remote assessments]were similar in difficulty in terms of the questions asked, but more difficult and stressful in terms of the procedure of answering the questions, inserting the pics, keeping the time yourself, technology issues, etc.”

Comment 2: “More stressful in some aspects because a lot of time is spent trying to figure out the best way of adding equations to the word document. However having notes available lessened the stress.”

Comment 3: “Modules which are maths-intensive are difficult to do on a computer, however essay-based modules are fine”

Comment 4: “The modules with some mathematics and some longer style questions (e.g. [name of the module]) were much more inconvenient due to having to swap between pen and paper constant in order to complete the exams. Modules where it was mainly writing or mainly maths this issue was not as prominent.”

Logistics in fact, has been one of the main issues in remote assessments, and it is something that we need to work on for the future academic year, especially for those exams in which the timeframe is much shorter (2-3 hours).

Figure 1: Rank the following assessments by preference (1 the most preferred –6 the least)

Note: There was also “group project” & “other” which were the least popular choices and are not added in this graph.

If we look more in detail the different type of remote assessments, it is clear in fact that the reason why students prefer invigilated assessments to remote ones is due to the challenges uploading figures and equations in the 2-3h online exams. When students were asked to rank their preferred assessment types, 33% of students chose invigilated exams as their top preference, with 24/48h assessments following very close (28%). Very few students (8%) chose online exams as their favourite (see Figure 1). However, if we consider 24/48h and essay-type assessments together 48% of students pick either of these as their preferred choice. Therefore, the type of remote assessment matters for students’ preferences, especially if we consider potential interaction with other aspects e.g. IT facilities.

Students in fact were very positive about 24/48h assessments and commented on how these help them to alleviate some of the stress caused by the new online environment:

Comment 5: “I found the 24 hour format of the exam very helpful compared to the 2-3 hour format, it alleviated a lot of the stress surrounding the online assessment and I believe it will have helped me to perform better as a result. I think the format really meant that my understanding of the content was able to be examined fully.”

Students also felt that 24/48h assessments provide them with more opportunities to critically analyse their answers and also to demonstrate their understanding, while at the same time adding some (but not too much) time pressure:

Comment 6: “I think a 24 h/48 hour exam would work better in eliminating the stress and pressure of submitting in time, technology issues, etc. as well as testing the students’ knowledge more broadly. If the questions are designed correctly, like in [module A] for example, it is still possible for a lecturer to test the students’ knowledge and preparation, without concerns that the student was not able to retain much information and just copied everything from the internet. Such questions involve the student already knowing the material quite well and the different aspects of a single question, which cannot be answered well if the student relies only on the 24 hour time frame to find the answers in the material already provided. In addition, such assessments would provide certain modules like [name of module] or [name of module] to be a bit more applied, more case study focused on some of the questions, which could better test our understanding of the material and also be easier to check for plagiarism than just having math based problems. In [name of module] or [name of module] we could be asked to do some STATA analysis as well, which is something that we usually do not get to do as part of exams for this module, but would also test our understanding better.”

Comment 7: “I like the concept of a 24/48 hour exam but I also like essay-type assessments. The great thing about the 24 hour assessment is it gives you the to create a complete response and actually think about the questions yet you are still under some time pressure which I actually find beneficial personally. This is probably most useful for people like me who procrastinate put ideas into full writing so the time pressure of 24 is nice in that sense.”

Of course, students also recognise that different module may require different assessment types. For example:

Comment 8: “For [Module B]’s applied components, the 48h format made perfect sense! For other traditional essays or empirical modules, the time exam makes sense.”

Comment 9: “For exams with maths/short answer questions a limited time exam would be better, but if it's an essay writing exam then the 24/48h one would be better as there is much more time to formulate.”

Comment 10: “For economics I prefer 24h assessments, but for other essay-based subjects like politics, I prefer essay-type assessments”

Comment 11: “I feel empirical modules are better suited to invigiliation, as you practise applying the techniques learned and then have to apply them under pressure which allows for greater cohort differentiation. For theory based modules, open-book timed exams are better as it allows focusing on argumentation and analysis, rather than rote-learning author names, dates, and results.”

Students' comments and insights were very helpful. In the academic year 2020/21, I will maintain 24/48h assessments for both modules (other lecturers teaching in the module agreed too). Designing the questions does take some time as they need to be open and allow students to shape their answer according to their understanding of the material and engagement with the resources provided, in addition of doing some basic research. The results this year are encouraging. These are in line with previous years as shown in Table 3, in which I compare some basic statistics for one of the modules with previous years. The variation in average, median and standard deviation in the remote 24/48h assessment is almost zero compared to the previous two academic years in which students sat an invigilated 2h exam.

However, this does not mean that there are no challenges. Much of the engagement with students next academic year will happen online. We need to rethink assessments in order for students to engage with the material. While these high-stake assessments will remain the same, their weight on the final mark will be lower. Smaller stake assessments will be used to promote engagement with the subject. It is important that we motivate students during the academic year.

Some of the open comments from the survey emphasise the importance of engagement. Some students found more difficulty to get the right incentives to prepare for the exams, and in general to engage with the subject. For example:

Comment 12: “With remote exams you don't have the incentive to study as well as during an invigilated exam, therefore for timed remote exams it becomes more difficult to write the correct answer while in 24 hour remote exams it’s easier.”

Comment 13: “There were upsides and downsides. The most noticeable upside was that, in most modules comprehending the material was enough and studying many equations and formulas by heart was not needed. The downside is that I prefer the handwritten format, and it is easier to switch our mindset to exams if it's on-campus.”

The biggest challenge for next year’s assessments (and teaching material in general could indeed be to foster engagement.

3. Final Thoughts

The 2019/20 exam session was challenging for all academics and students. This does not mean that we cannot learn something from this experience and take the positive aspects and carry them in the future.

The shift to 24/48h assessments showed an arguably better way to assess students in these two modules. Students had the opportunity to show their engagement with the learning outcomes in a less stressful setting than online exams. These assessments also have the advantage to allow for some more engaging questions e.g. students looking for some basic data to support their policy recommendations. Unlike longer deadlines, 24/48h assessments do add some time-pressure which is beneficial for engagement without negatively affecting performance. I do not exclude the possibility to maintain these assessments once we go back to “normal”, but this will depend on institutional constraints.

Of course, one of the main concerns for lecturers is students cheating in open-book exams. I briefly discussed on lack of evidence on students engaging in poor academic practice in these two assessments: answers did differ among each other, the plagiarism software did not detect plagiarism cases, and students provided a list of resources consulted for each question. The mark distribution was very similar to what have been in past academic years (no extra adjustments were required during moderation process). I talk more poor academic practice in our case study “Assessment in the Time of Pandemic” (joint with Tim Burnett). In this document, I also discuss more in detail on the benefits of 24/48h assessments.

Poor academic practice is not just a lecturer’s concern, but students are concerned about this too. From the survey:

Comment 14: “Indifferent for my own performance, but I strongly suspect a lot of collaboration will have taken place and not been caught, pushing grade boundaries up a lot. I think this could be a big issue in maths heavy exams, where plagiarism and collaboration is almost undetectable.”

Comment 15: “If the questions on the exam can be directly copied from lecture notes or other module material, then online exams seem very useless.”

Comment 16: “I am a bit sceptical as to how university can prevent cheating in the online exam format. While I understand it is a very difficult matter, those who may use external help, will get a relatively better grade than those who not, thus distorting the distribution.”

In brief, I do not think cheating should be our main worry. We should focus on engaging students with the topics discussed in the classes, designing assessments that truly assess learning outcomes and clearly communicating to students on our strategy to minimise poor academic practice. These are going to be the main challenges in the new academic year.