Over the past month there’s been a fair amount of panic surrounding the rise of AI chatbots and how they could have an impact on testing in schools. Well, a potential solution could be DetectGPT.
DetectGPT is a model that has been developed with a method of detecting text generated by popular AI text apps like ChatGPT. According to the announcement post, DetectGPT detects samples from pre-trained large language models (LLMs like ChatGPT) “using the local curvature of the model’s log probability function”.
Let’s translate what this means: the model recognises patterns that may indicate a piece of AI-generated text, based on what is presented. Recognising these patterns, DetectGPT should be able to detect if a piece of text has been generated by an AI.
The model apparently doesn’t require a separate dataset to inform its own knowledge database, but instead relies on the “log probabilities computed by the model of interest [for example, ChatGPT] and random perturbations of the passage from another generic pre-trained language model [for example, a ChatGPT alternative like T5]”, according to the announcement. What this means is that the AI is trained to recognise patterns in content structure from AIs and flag when it detects them.
@gizmodoau Please don’t cheat on your test using an AI. #tech #chatgpt #tech #university
As an example, the team said that DetectGPT is “more discriminative” than existing similar methods, and has noticeably improved assessments when tasked with detecting fake news articles.
At the moment, this is just an announcement. The page reads “code, data and more coming soon” at the footer, so in time we might be able to publically use this AI detection tool.
In which case, it will be very welcome. While the CEO of ChatGPT thinks schools should just get over AI cheating concerns, it’s proving to be a big problem for assessing student knowledge in exams or tests.
Something sophisticated like DetectGPT will be very valuable in the academic world.
To state the obvious: don’t cheat using an AI. At best, you’re getting by in your test dishonestly and at worst you could face academic penalties.
And going forward, it may be more likely that you’ll get caught.