AI and Higher Education

This is an overview of AI tools, including Large Language Models (LLMs) such as ChatGPT, and their relevance to teaching and assessment in higher education. It is necessarily in flux, as the capabilities of LLMs are changing rapidly and new controversies are emerging.

This page is curated manually, without the use of any LLMs.

The Economics Network hosted an online seminar on Assessment and AI on 29 February 2024, chaired by Prof. Dimitra Petropoulou. Prof. Alvin Birdi gave an overview of AI's capabilities and the controversy about its use, and Prof. Carlos Cortinhas reported on a survey on use of AI in higher education. Follow the link for lightly edited video as well as slides and links.

Midjourney's response to "A profile photo of an Economics professor" (click to enlarge). Midjourney is a generative AI program trained on hundreds of millions of images from the web. Large datasets can have biases or gaps which are reflected in stereotypical features in its output.

Background

• Although one family of LLMs, ChatGPT, is attracting most attention, there are multiple LLMs with similar capabilities. LLMs can drive other software and devices and are rapidly being integrated into tools like search engines, Microsoft Office applications,(ref 8) or text messaging apps. Hence one can be "using" an LLM in increasingly many situations.
• Although the initial output of an LLM has a recognisable style, the default behaviour can be altered in several ways. One can give the model conversational feedback, tell it to adopt a persona, or feed its output into other AI tools.
• LLMs have prompted a lot of discussion about whether and how Higher Education needs to adapt. In the context of assessment, there are concerns about sophisticated cheating and the generation of pseudo-information, (termed "hallucination"). There is also discussion about whether LLMs will assist learning (for students in general or for some kinds of student), whether they can assist educators, and about whether universities need to prepare students for workplaces that in many cases will use LLMs.

Capabilities

GPT-4's code interpreter retrieving and plotting GDP data from a public interface

Not all LLMs have the same capabilities. Even the functionality of ChatGPT differs between the paid and free versions and depends on what plugins are available to each user.

There are presently three major chatbots that are significantly more capable than the rest: ChatGPT-4, Google Gemini Advanced, and Claude 3 Opus, each requiring a paid subscription. Each can create and manipulate text, images, and code but there are differences between them. (ref 18)

The brand name "ChatGPT", "Gemini", or "Claude" is not on its own a good indicator of the capabilities of the chatbot. Each of these exists in multiple versions with widely varying capabilities.

In at least some cases, LLMs can:

• Generate writing in a given style, including writing at a specific educational level or introducing deliberate errors.
• Convert between writing styles, e.g. from bullet-point list to narrative essay or vice versa; casual to academic style or vice versa. (ref 9)
• Generate code, including comments, from a verbal description, in languages including R, Matlab, Python, and Excel macros. (ref 1) It has difficulty writing Stata code but is much more capable with other languages. (ref 2)
• Look up information on the internet by connecting to external services, for example using Wolfram|Alpha. (using the Plugins feature of ChatGPT 4). (ref 3)
• Analyse an Excel data set, visualise the data, suggest hypotheses that can be tested with the data, conduct regressions and report the results in natural language. (ref 7) Create static charts or animated or interactive visualisations to summarise a data set. (ref 15) (both using GPT-4's Code Interpreter plugin).
• Score more highly than most human students on some exams. GPT-4 performs very differently on different kinds of exam involving mathematics. It gets a 5 (the highest score) on Advanced Placement exams in Microeconomics, Macroeconomics, and Statistics. On the SAT Math test used in the US, it scores above 89% of students. (ref 4) On American Mathematics Competition exams, it scores around the median on the AMC12 and in the bottom 12% on the AMC10. (ref 5)
• Help students with mathematical questions from the SAT exam (e.g. Solving for two unknowns with two constraints; Identifying the missing measurement from an average) by generating custom explanations.
• Watch a screen-capture video of someone working at a computer, describe what they did, and advise on more efficient ways of working. (ref 20)

Prompting and custom bots

Khanmigo bot giving feedback to a learner about a mistaken step in an algebra problem. Click to expand. Via the Mathworlds Substack

• Chatbots can be given customised instructions and "personas" to prepare them for a specific task. These custom bots do not have additional functionality, but are tailored for a particular task such as talking a learner through a mathematical exercise. They are "re-skinned" versions of existing chatbots rather than a distinct technology, so they have the same pattern of strengths and errors. Khanmigo is Khan Academy's customised ChatGPT for giving feedback to learners. Other custom bots help write code, summarise research papers, or prepare slide presentations. New custom bots appear each week.
• Members of the Network have found students creating customised bots to help with specific courses or aspects of learning.
• Telling a chatbot to "think step by step" of "solve this problem step-by-step" seems to be improve answers in some cases. There is a very unpredictable relationship between the way a bot is instructed ("prompted") and its performance. Some academic users have found that telling the bot it will earn tips for good performance actually improves results. Others report benefits from telling the bot that it has taken an adderall tablet.
• Some of the predictability of chatbots — in their text style and in the ideas they suggest — can be mitigated by custom instructions. (ref 19) Some LLMs have a "temperature" setting, controlling the amount of randomness in its input, which can be set high to get less predictable responses.
• Ethan Mollick's Prompt Library is a set of example prompts for Higher Education contexts, identifying the bots with which they have been used. These illustrate how much tailoring can go into a prompt to adapt LLMs for a specific purpose, and they can be starting points for subject customisation.

Performance in economics tests and exams

• ChatGPT has Aced the Test of Understanding in College Economics: Now What? by Wayne Geerling, G. Dirk Mateer, Jadrian Wooten, and Nikhil Damodoran, The American Economist, April 2023 (testing ChatGPT 3)
  • "While ChatGPT-generated papers have received good grades, they lack the depth of understanding that is expected in higher education."
  • "Tools like ChatGPT are likely to become a common part of the writing process, just as calculators and computers have become essential tools for learning mathematics and science. The challenge of universities is to adapt their curriculum to this new reality."
• How to Learn and Teach Economics with Large Language Models, Including GPT by Tyler Cowen and Alexander T. Tabarrok, GMU Working Paper in Economics No. 23-18, 27 March 2023
  • "GPTs have not yet fully mastered long chains of abstract reasoning; they cannot "think through" a complex economic problem from beginning to end and provide a comprehensive answer with multiple cause-and-effect relationships."
  • "Chat GPT is very good at writing exam questions throughout the curriculum. [...] ChatGPT and Bing Chat will also create very credible syllabi for a variety of courses including readings, course policies, and grading procedures."
• Would Chat GPT3 Get a Wharton MBA? A Prediction Based on Its Performance in the Operations Management Course by Christian Terwiesch, University of Pennsylvania, January 2023
  • "Chat GPT3 does an amazing job at basic operations management and process analysis questions including those that are based on case studies. Not only are the answers correct, but the explanations are excellent. [...] Chat GPT3 at times makes surprising mistakes in relatively simple calculations at the level of 6th grade Math. These mistakes can be massive in magnitude."

Plagiarism detection and AI

Image created by Midjourney from the prompt "The gradual robotification of polite society"

Section 4 of the Handbook for Economics Lecturers chapter on Prevention and Detection of Plagiarism in Higher Education addresses the implications of AI for plagiarism and different ways in which universities can respond.

Some LLM "detectors" are available, but suffer from false positives, variations in LLM output and the availability of tools that re-write text. (ref 6)

• Why AI detectors think the US Constitution was written by AI by Benj Edwards, Ars Technica, 14 July 2023
  • "Due to false positives, AI writing detectors such as GPTZero, ZeroGPT, and OpenAI's Text Classifier cannot be trusted to detect text composed by large language models (LLMs) like ChatGPT. [...] Perhaps the most damaging result of people using these inaccurate and imperfect tools is the personal cost of false accusations."
• ChatGPT, assessment and cheating – have we tried trusting students? WONKHE, 20 February 2023
  • "We’ve seen some efforts to employ the “detection tool” approach used for other forms of academic malpractices – but every single one of them has been beaten in practice, and many flag the work of humans as AI derived."
  • "One of fundamental shifts in assessment is therefore likely to be around defining the level of creativity and originality lecturers expect from students, and what these terms will mean."
• The Rise of Artificial Intelligence Software and Potential Risks for Academic Integrity: Briefing Paper for Higher Education Providers QAA, 30 January 2023
  • "Assessments generated by the software tools used by LLMs may take the form of coursework such as essays and dissertations, but also projects, presentations, computer source code and other forms"
  • "[W]ork created in this way can be difficult to identify and cannot be picked up by more traditional plagiarism detection tools."
• Nearly 1 in 3 college students have used ChatGPT on written assignments Intelligent.com, 23 January 2023 (reporting a survey of US students)
  • "Twenty-eight percent of survey respondents also believe that their professors are ‘probably’ (23%) or ‘definitely’ (5%) not aware that they have used the tool on their assignments."
  • "3 in 4 ChatGPT users believe it is cheating, but use it anyway."

Positive uses of AI in education

Some suggestions of "What ChatGPT is good at" from Alvin Birdi's presentation:

• Feedback on paper drafts
• Providing counterarguments
• Improving writing
• Synthesising text from bullet points
• Editing text
• Evaluating text (lack of clarity, passive voice, structure etc)  -> feedback 
• Acting as a tutor for concepts
• Brainstorming ideas and examples related to a theme -> lead to homogeneity?
• Drafting assessments that do or do not make use of ChatGPT/AI
• Drafting assessments that maintain integrity
• Initial drafts for teaching plans/lectures

AI in education (actionable insights for educators), dated 16 July 2023, is one of a suite of reports produced by Warwick University.

UCL's guidance for tutors and students on AI distinguishes tasks where AI is forbidden from those where it is allowed as an assistive tool and those where it can be integral to assessment tasks.

• Creative Storytelling in Economics with Lego and AI by Swati Virmani, published February 2024
  • "This drove me to experiment with the concept of composing a complete storyboard, directing an AI tool to define a term, write the scenes/sequences, and subsequently transform each scene into a descriptive image."
• Language models and AI in economic education: Unpacking the risks and opportunities presentation slides from DEE Conference 2023 by Tomasz Kopczewski and Ewa Weychert
  • "Change narratives about AI: Passive use of generative AI is the first step to unemployment. You must be a 'critical miner' of generative AI. Don't stop at acquiring the ore (information) - turn it into knowledge and share it with others. Your even imperfect interpretation of information is needed to enhance diversity and, thus, collective knowledge."
• Assigning AI: Seven Approaches for Students, with Prompts by Ethan R. Mollick and Lilach Mollick, updated 23 September 2023
  • "The authors propose seven approaches for utilizing AI in classrooms: AI-tutor, AI-coach, AI-mentor, AI-teammate, AI-tool, AI-simulator, and AI-student, each with distinct pedagogical benefits and risks. The aim is to help students learn with and about AI, with practical strategies designed to mitigate risks such as complacency about the AI’s output, errors, and biases."
• What Should Data Science Education Do with Large Language Models? by Xinming Tu et al., 7 July 2023 (preprint)
  • "With the assistance of LLMs, data scientists can shift their focus towards higher-level tasks, such as designing questions and managing projects, effectively transitioning into roles similar to product managers."
  • "LLMs can assist educators in designing dynamic and engaging curricula, generating contextually relevant examples, exercises, and explanations that help students grasp complex concepts with greater ease."
•    
  Some suggested uses of ChatGPT in education in the UNESCO guide

  ChatGPT and Artificial Intelligence in higher education: Quick start guide by UNESCO International Institute for Higher Education in Latin America and the Caribbean, April 2023.

  • "Due to its ability to generate and assess information, ChatGPT can play a range of roles in teaching and learning processes. Together with other forms of AI, ChatGPT could improve the process and experience of learning for students."
  • See also the UNESCO document "Guidance for generative AI in education and research", issued 7 September 2023
• 
  Five examples of opportunity cost generated by ChatGPT 4 via Mollick and Mollick (click to expand)

  Using AI to Implement Effective Teaching Strategies in Classrooms: Five Strategies, Including Prompts by Ethan R. Mollick and Lilach Mollick, 17 March 2023

  • "Many teaching techniques have proven value but are hard to put into practice because they are time-consuming for overworked instructors to apply. With the help of AI, however, these techniques are more accessible."
  • "[I]ntentionally implementing teaching strategies with the help of an LLM can be a force multiplier for instructors and provide students with extremely useful material that is hard to generate."
• ChatGPT is the push higher education needs to rethink assessment by Sioux McKenna et al., The Conversation, 12 March 2023
  • "ChatGPT can be used to support essay writing and to help foster a sense of mastery and autonomy. Students can analyse ChatGPT responses to note how the software has drawn from multiple sources and to identify flaws in the ChatGPT responses which would need their attention."
• ChatGPT as a teaching tool, not a cheating tool by Jennifer Rose, Times Higher Education, 21 February 2023
  • "One way that ChatGPT answers can be used in class is by asking students to compare what they have written with a ChatGPT answer. [...] This dialogic approach develops the higher-order thinking skills that will keep our students ahead of AI technology."
• Some initial lessons from using ChatGPT and what I will tell my Macroeconomics students by Stefania Paredes Fuentes, University of Warwick, January 2023
  • "Artificial Intelligence tools are not going to disappear, and they are going to change the way we learn (and hopefully teach)."
  • "Rather than 'banning' the use of ChatGPT [...], let’s engage with a conversation with students regarding the limitations of this technology but also on ways to use it."

Perspectives from the Higher Education sector

• Padlet from the Economics Network Virtual Symposium session on Assessment and AI, 29 February 2024
• New principles on use of AI in education, The Russell Group, 4 July 2023
  • "Our universities will develop resources and training opportunities, so that staff are able to provide students with clear guidance on how to use generative AI to support their learning, assignments, and research."
  • "Engagement and dialogue between academic staff and students will be important to establish a shared understanding of the appropriate use of generative AI tools."
• A Generative AI Primer, Jisc National Centre for AI in Tertiary Education, 11 May 2023
  • "Generative AI is progressing rapidly and is likely to have a significant impact on education for the foreseeable future. [...] Nonetheless, with care and an increase in staff and student knowledge, there are substantial gains to be made."
• Navigating the Use of ChatGPT in Education Education Directorate, University of Kent, 19 February 2023
  • Summary of a webinar held in February.
  • "Only through open and honest dialogue with our students can we highlight the benefits and limitations of using AI technologies and support students to use them responsibly and ethically."
• Padlet from the Digitally Enhanced Education Webinars on which participants share their ideas and concerns about ChatGPT in education.
• ChatGPT for students with Dyslexia? DyStIncT magazine, 19 February 2023
  • "ChatGPT also has amazing potential for supporting people with learning difficulties. [...] However, the difficulty and potentially huge drawback to ChatGPT is this use. The function that makes it helpful is the function that can be incredibly problematic."

Ethical concerns

Users of LLMs should be aware that:

• Training of the ChatGPT model involved African workers in conditions in which they have been described as "underpaid and exploited". (ref 13)
• LLMs require a lot of computations, which in turn require power and cooling. The data centres running these models have a large water footprint. (ref 10)
• There are also concerns about the carbon impact of the data centres. (ref 11) Attempts to compare the carbon impact of AI-generated content with other ways of creating similar content run into problems of data quality and relevance. (ref 16)
• LLMs are trained on huge sets of text created by human beings. A training set may include the entirety of Wikipedia, StackOverflow, or GitHub. Image generators are trained on art and photography created by human beings. Since these creators are not credited, there are ethical questions around exploitation, copyright, and consent. ChatGPT can potentially infringe copyright by reproducing an existing piece of text, and it itself cannot tell when it is doing so. (ref 12)
• Since the training data are drawn from the digital world, they reflect the dominance of English and the comparative absence of many indigenous languages. Values and assumptions of the people who contribute most online text shape the default output of the models. LLMs thus perpetuate languages, attitudes, and values of one group of humanity at the expense of the rest. (ref 14)

Similar concerns are also raised about other online services — not to mention other features of the modern workplace — but ethical concerns are part of the current debate about the use of LLMs and may even form part of the classroom discussion about whether and how they should be used.

References

1. Artificial Intelligence and Signal Processing, Tom O'Haver, University of Maryland at College Park, March 2023

2. Can AI write your Stata code? Owen Ozier, the World Bank, 1 February 2023

3. ChatGPT Gets its Wolfram Super-powers, Stephen Wolfram, 23 March 2023

4. List: Here Are the Exams ChatGPT Has Passed so Far, Lakshmi Varanasi, Business Insider, 21 March 2023

5. GPT-4 is Amazing but Still Struggles at High School Math Competitions, Russell Lim, 24 March 2023

6. How to detect ChatGPT plagiarism — and why it’s becoming so difficult, Aaron Leong, Digital Trends, 20 January 2023

7. It is starting to get strange, Ethan Mollick, 2 May 2023 / What AI can do with a toolbox... Getting started with Code Interpreter, Ethan Mollick, 7 July 2023

8. Introducing Microsoft 365 Copilot, Microsoft 365, 16 March 2023

9. 10 Strategies to Alter ChatGPT's Writing Style, Luke Skyward, PlainEnglish, 16 January 2023

10. ChatGPT needs to 'drink' a water bottle's worth of fresh water for every 20 to 50 questions you ask, researchers say, Will Gendron, Business Insider, 14 April 2023

11. Artificial Intelligence Is Booming—So Is Its Carbon Footprint, Josh Saul and Dina Bass, Bloomberg UK, 9 March 2023. See also ChatGPT’s Carbon Footprint Tanushree Kain, SigmaEarth, 12 April 2023

12. Copyright and ChatGPT, Kirsty Stewart and Hannah Smethurst, Thorntons Law, 1 March 2023 "ChatGPT could subsequently produce material in response to any question by a user, which directly infringes an existing copyright holder’s work. Unfortunately, there is no easy way for users to tell what, if any, of ChatGPT’s responses have been pulled directly from an existing (and protected by copyright) work, nor who the author of this original work is."

13. Exclusive: OpenAI Used Kenyan Workers on Less Than $2 Per Hour to Make ChatGPT Less Toxic, Billy Perrigo, TIME, 18 January 2023

14. Don’t fret about students using ChatGPT to cheat – AI is a bigger threat to educational equality, Collin Bjork, The Conversation, 5 April 2023; Artificial generative intelligence risks a return to cultural colonialism, Songyee Yoon, VentureBeat, 25 April 2023

15. ChatGPT’s Code Interpreter – Top 6 uses, ChatGPTGuide, 10 July 2023 (Use 3)

16. How much energy does AI use compared to humans? Surprising study ignites controversy, Bryson Masse, VentureBeat, 22 September 2023 (summarising this preprint)

17. Kumar, Harsh and Rothschild, David M. and Goldstein, Daniel G. and Hofman, Jake, "Math Education with Large Language Models: Peril or Promise?" (November 22, 2023). http://dx.doi.org/10.2139/ssrn.4641653

18. "Google's Gemini Advanced: Tasting Notes and Implications", Ethan Mollick 8 February 2024

19. Meincke, Lennart; Mollick, Ethan R. and Terwiesch, Christian, "Prompting Diverse Ideas: Increasing AI Idea Variance" (January 27, 2024). http://dx.doi.org/10.2139/ssrn.4708466

20. "Which AI should I use? Superpowers and the State of Play", Ethan Mollick 18 March 2024