AI and Higher Education

This is an overview of AI tools, including Large Language Models (LLMs) such as ChatGPT, and their relevance to teaching and assessment in higher education.

Midjourney's response to "A profile photo of an Economics professor" (click to enlarge). Midjourney is a generative AI program trained on hundreds of millions of images from the web. Large datasets can have biases or gaps which are reflected in stereotypical features in its output.

Why AI is relevant to university education

• The capabilities of LLMs are changing rapidly, even week by week.
• Although one family of LLMs, ChatGPT, is attracting most attention, there are multiple LLMs with similar capabilities. LLMs can drive other software and devices and are rapidly being integrated into tools like search engines, Microsoft Office applications,(ref 8) or text messaging apps. Hence one can be "using" an LLM in increasingly many situations.
• Although the initial output of an LLM has a recognisable style, the default behaviour can be altered in several ways. One can give the model conversational feedback, tell it to adopt a persona, or feed its output into other AI tools.
• LLMs have prompted a lot of discussion about whether and how Higher Education needs to adapt. In the context of assessment, there are concerns about sophisticated cheating and the generation of pseudo-information, (termed "hallucination"). There is also discussion about whether LLMs will assist learning (for students in general or for some kinds of student), whether they can assist educators, and about whether universities need to prepare students for workplaces that in many cases will use LLMs.
• Some LLM "detectors" are available, but suffer from false positives, variations in LLM output and the availability of tools that re-write text. (ref 6)

Image created by Midjourney from the prompt "How to worry wisely about intelligent machines", via The Economist, June 2022

Capabilities

Not all LLMs have the same capabilities. Even the functionality of ChatGPT differs between the paid and free versions and depends on what plugins are available to each user.

In at least some cases, LLMs can:

• Generate writing in a given style, including writing at a specific educational level or introducing deliberate errors.
• Convert between writing styles, e.g. from bullet-point list to narrative essay or vice versa; casual to academic style or vice versa. (ref 9)
• Generate code, including comments, from a verbal description, in languages including R, Matlab, Python, and Excel macros. (ref 1) It has difficulty writing Stata code but is much more capable with other languages. (ref 2)
• Look up information on the internet by connecting to external services, for example using Wolfram|Alpha. (using the Plugins feature of ChatGPT 4) (ref 3)
• Analyse an Excel data set, visualise the data, suggest hypotheses that can be tested with the data, conduct regressions and report the results in natural language (using GPT-4's Code Interpreter plugin). (ref 7)
• Score more highly than most human students on some exams. GPT-4 performs very differently on different kinds of exam involving mathematics. It gets a 5 (the highest score) on Advanced Placement exams in Microeconomics, Macroeconomics, and Statistics. On the SAT Math test used in the US, it scores above 89% of students. (ref 4) On American Mathematics Competition exams, it scores around the median on the AMC12 and in the bottom 12% on the AMC10. (ref 5)

Ethical concerns

Users of LLMs should be aware that:

• Training of the ChatGPT model involved African workers in conditions in which they have been described as "underpaid and exploited". (ref 13)
• LLMs require a lot of computations, which in turn require power and cooling. The data centres running these models have a large water footprint. (ref 10)
• There are also concerns about the carbon impact of the data centres. (ref 11)
• LLMs are trained on huge sets of text created by human beings. A training set may include the entirety of Wikipedia, StackOverflow, or GitHub. Image generators are trained on art and photography created by human beings. Since these creators are not credited, there are ethical questions around exploitation, copyright, and consent. ChatGPT can potentially infringe copyright by reproducing an existing piece of text, and it itself cannot tell when it is doing so. (ref 12)
• Since the training data are drawn from the digital world, they reflect the dominance of English and the comparative absence of many indigenous languages. Values and assumptions of the people who contribute most online text shape the default output of the models. LLMs thus perpetuate languages, attitudes, and values of one group of humanity at the expense of the rest. (ref 14)

Similar concerns are also raised about other online services — not to mention other features of the modern workplace — but ethical concerns are part of the current debate about the use of LLMs and may even form part of the classroom discussion about whether and how they should be used.

Micro-blog (most recent links first)

• My students are using AI to cheat. Here’s why it’s a teachable moment by Siva Vaidhyanathan, The Guardian, 19 May 2023
  • "I am excited about the instant popularity of large-language models in our lives. As long as they remain terrible at what they purport to do, they are perfect for study. They reveal so many of the issues we too often let lurk below our frenetic attention."
• A Generative AI Primer, Jisc National Centre for AI in Tertiary Education, 11 May 2023
  • "Generative AI is progressing rapidly and is likely to have a significant impact on education for the foreseeable future. [...] Nonetheless, with care and an increase in staff and student knowledge, there are substantial gains to be made."
•   
  Some suggested uses of ChatGPT in education in the UNESCO guide

  ChatGPT and Artificial Intelligence in higher education: Quick start guide by UNESCO International Institute for Higher Education in Latin America and the Caribbean, April 2023.

  • "Due to its ability to generate and assess information, ChatGPT can play a range of roles in teaching and learning processes. Together with other forms of AI, ChatGPT could improve the process and experience of learning for students."
  • "There are concerns that existing tools to detect plagiarism may not be effective in the face of writing done by ChatGPT."
• ChatGPT has Aced the Test of Understanding in College Economics: Now What? by Wayne Geerling, G. Dirk Mateer, Jadrian Wooten, and Nikhil Damodoran, The American Economist. April 2023 (testing ChatGPT 3)
  • "While ChatGPT-generated papers have received good grades, they lack the depth of understanding that is expected in higher education."
  • "Tools like ChatGPT are likely to become a common part of the writing process, just as calculators and computers have become essential tools for learning mathematics and science. The challenge of universities is to adapt their curriculum to this new reality."
• ChatGPT as a teaching tool, not a cheating tool by Jennifer Rose, Times Higher Education, 21 February 2023
  • "One way that ChatGPT answers can be used in class is by asking students to compare what they have written with a ChatGPT answer. [...] This dialogic approach develops the higher-order thinking skills that will keep our students ahead of AI technology."
• ChatGPT, assessment and cheating – have we tried trusting students? WONKHE, 20 February 2023
  • "We’ve seen some efforts to employ the “detection tool” approach used for other forms of academic malpractices – but every single one of them has been beaten in practice, and many flag the work of humans as AI derived."
  • "One of fundamental shifts in assessment is therefore likely to be around defining the level of creativity and originality lecturers expect from students, and what these terms will mean."
• Navigating the Use of ChatGPT in Education Education Directorate, University of Kent, 19 February 2023
  • Summary of a webinar held in February.
  • "Only through open and honest dialogue with our students can we highlight the benefits and limitations of using AI technologies and support students to use them responsibly and ethically."
• ChatGPT for students with Dyslexia? DyStIncT magazine, 19 February 2023
  • "ChatGPT also has amazing potential for supporting people with learning difficulties. [...] However, the difficulty and potentially huge drawback to ChatGPT is this use. The function that makes it helpful is the function that can be incredibly problematic."
• The Rise of Artificial Intelligence Software and Potential Risks for Academic Integrity: Briefing Paper for Higher Education Providers QAA, 30 January 2023
  • "Assessments generated by the software tools used by LLMs may take the form of coursework such as essays and dissertations, but also projects, presentations, computer source code and other forms"
  • "[W]ork created in this way can be difficult to identify and cannot be picked up by more traditional plagiarism detection tools."
• Some initial lessons from using ChatGPT and what I will tell my Macroeconomics students by Stefania Paredes Fuentes, University of Warwick, January 2023
  • "Artificial Intelligence tools are not going to disappear, and they are going to change the way we learn (and hopefully teach)."
  • "Rather than “banning” the use of ChatGPT [...], let’s engage with a conversation with students regarding the limitations of this technology but also on ways to use it."
• Nearly 1 in 3 college students have used ChatGPT on written assignments Intelligent.com, 23 January 2023 (reporting a survey of US students)
  • "Twenty-eight percent of survey respondents also believe that their professors are ‘probably’ (23%) or ‘definitely’ (5%) not aware that they have used the tool on their assignments."
  • "3 in 4 ChatGPT users believe it is cheating, but use it anyway."
• Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course by Christian Terwiesch, University of Pennsylvania, January 2023
  • "Chat GPT3 does an amazing job at basic operations management and process analysis questions 
    including those that are based on case studies. Not only are the answers correct, but the explanations are 
    excellent. [...] Chat GPT3 at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math. These mistakes can be massive in magnitude."

Perspectives

Padlet from the Digitally Enhanced Education Webinars on which participants share their ideas and concerns about ChatGPT in education.

References

1. Artificial Intelligence and Signal Processing, Tom O'Haver, University of Maryland at College Park, March 2023

2. Can AI write your Stata code? Owen Ozier, the World Bank, 1 February 2023

3. ChatGPT Gets its Wolfram Super-powers, Stephen Wolfram, 23 March 2023

4. List: Here Are the Exams ChatGPT Has Passed so Far, Lakshmi Varanasi, Business Insider, 21 March 2023

5. GPT-4 is Amazing but Still Struggles at High School Math Competitions, Russell Lim, 24 March 2023

6. How to detect ChatGPT plagiarism — and why it’s becoming so difficult, Aaron Leong, Digital Trends, 20 January 2023

7. It is starting to get strange, Ethan Mollick, 2 May 2023

8. Introducing Microsoft 365 Copilot, Microsoft 365, 16 March 2023

9. 10 Strategies to Alter ChatGPT's Writing Style, Luke Skyward, PlainEnglish, 16 January 2023

10. ChatGPT needs to 'drink' a water bottle's worth of fresh water for every 20 to 50 questions you ask, researchers say, Will Gendron, Business Insider, 14 April 2023

11. Artificial Intelligence Is Booming—So Is Its Carbon Footprint, Josh Saul and Dina Bass, Bloomberg UK, 9 March 2023. See also ChatGPT’s Carbon Footprint Tanushree Kain, SigmaEarth, 12 April 2023

12. Copyright and ChatGPT, Kirsty Stewart and Hannah Smethurst, Thorntons Law, 1 March 2023 "ChatGPT could subsequently produce material in response to any question by a user, which directly infringes an existing copyright holder’s work. Unfortunately, there is no easy way for users to tell what, if any, of ChatGPT’s responses have been pulled directly from an existing (and protected by copyright) work, nor who the author of this original work is."

13. Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic, Billy Perrigo, TIME, 18 January 2023

14. Don’t fret about students using ChatGPT to cheat – AI is a bigger threat to educational equality, Collin Bjork, The Conversation, 5 April 2023; Artificial generative intelligence risks a return to cultural colonialism, Songyee Yoon, VentureBeat, 25 April 2023