Countless stories are emerging where students have been accused of using ChatGPT to write essays for classes.   But is ChatGPT output that easy to detect?  The answer is not reliably.  The accuracy of these tools on the market for detection is about 80% right now.   Statistically speaking, that is a pretty high false positive rate of 20% to accuse someone of academic dishonesty.

As an example of the difficulty to detect a ChatGPT response to a sample essay question, I prepared two different prompts for ChatGPT.   The first of which is a general prompt for the question and the second has ChatGPT adjusts the grade level of the writing. The idea being a high school student responding to the essay question.

Essay Question:

In a paragraph, describe the conditions leading up to the Great Recession.

Prompt #1:  Write a short paragraph on conditions leading up to the Great Recession

The Great Recession, which began in 2007, was triggered by a mix of financial issues. In the U.S., a housing bubble saw home prices soar unsustainably, and when they suddenly plummeted, many homeowners owed more than their homes were worth. Banks, having given out risky loans and invested in complex financial products tied to these mortgages, faced significant losses. This created a domino effect of financial instability, with banks distrusting each other and consumers losing confidence, ultimately leading to a global economic downturn.

Prompt #2: Write a short paragraph on conditions leading up to the Great Recession at a 6th grade level.

The Great Recession, which began in 2007, was like a big money problem that affected many countries. One big reason was that in the U.S., many people were buying houses with money they borrowed, thinking the house prices would keep going up forever. But, like a balloon popping, the prices suddenly dropped. This meant a lot of people had houses that were worth less than the money they borrowed. At the same time, some banks were making risky decisions with their money. When the house prices dropped, these banks got into trouble too. Because of these issues, many people lost money, some lost their jobs, and others lost their homes.

Unless you knew the prior work of the student, could you reasonably accuse the student of cheating? Even then, if the student were smart enough to add some typos and change the grade level of the writing, it would be difficult at best to even suspect.

What can be done?

There are some indicators and methods that educators can employ to raise suspicions, most of which depend on knowing the prior output of a student or analysis of the essay in response to the question:

Current techniques an educator can use:

1. Inconsistencies in Writing Style: If a student’s assignment or response suddenly shifts in style, complexity, or vocabulary, it might be a red flag. For instance, if a student who typically writes in simple sentences suddenly submits an essay with complex structures and advanced vocabulary, it might warrant a closer look.
2. Overly General or Off-topic Responses: AI tools like ChatGPT can provide vast amounts of information, but they might not always nail the specific context or nuance of a given assignment. If a student’s response seems too general or strays from the topic, it might indicate AI assistance.
3. Immediate and Complex Responses: In real-time settings, like online quizzes or exams, if a student provides immediate and intricate answers to open-ended questions, it could be a sign of using an AI tool.
4. In-depth Knowledge Beyond Course Content: If a student provides information or references concepts that haven’t been covered in the course or are too advanced for the curriculum, it might be a sign they sourced information from external tools.
5. Questions Designed for Critical Thinking: Designing questions that require students to apply knowledge in unique ways or reflect on personal experiences can make it harder for generic AI-generated responses to be relevant.
6. Open Conversations: Educators can have open dialogues with students about the use of AI tools. Understanding why students might turn to these tools (e.g., feeling overwhelmed, not understanding the material) can provide insights and allow educators to address the root causes.
7. Understanding AI Capabilities: Being aware of the capabilities and limitations of tools like ChatGPT can help educators devise strategies to detect or even counteract their unauthorized use.

It’s important to note that while these methods can raise suspicions, they aren’t foolproof in definitively proving AI tool usage. As AI becomes more prevalent, the challenge will be in striking a balance between leveraging these tools for genuine learning while discouraging undue reliance or unethical use in academic settings.

Like all new tools in any industry, fear of changing the status quo and a lack of learning is driving the rejection of using ChatGPT in an education setting.  To that end however, educators and students alike should learn to incorporate these tools into their toolbelt.