In the past couple of years I have been in a lot of meetings centered about the topic of "OMG students are using genAI in assessments what do we doooo?"
After marking a lot of assessments of different type from different courses in different years of study and two different undergraduate programmes, here are my conclusions, some of which I have no way of proving, I know, but that's fine.
1. A lot of students use genAI. In many cases I cannot prove that, it's just a feeling (nobody writes like this, especially not Year 1 non-native speakers), but I had students telling me directly and I do believe them.
2. Looking at marks in the cases above where I have that feeling, I see a wide spread from fails to high A.
3. Following from 2, overall marks have not changed significantly and systematically in any of our courses from the past 5 years. There are of course year-to-year fluctuations, but that's cohort-dependent there seems to be no overall trend.
4. The conclusion you might draw from 2 is that we're rewarding students who are good at using genAI. Possibly, and I have not made up my mind up about this. My answer is that we should change our way of assessing and teaching, rather than trying to "catch" students using genAI. We're currently redesigning our Programmes and that's what we're trying to do. I am in the process of designing a new course and I have tried to do that assuming students will use genAI, but making it so that using it will not be an advantage, and might actually make it more cumbersome to do the assessments.
5. We looked at final year dissertations and plotted marks against %AI writing detected by our submission system. Taking this with a pinch of salt, given that AI detectors are biased and unreliable, there is only a small negative correlation but it's such a small effect size as to be essentially negligible. Again, the distribution of marks is the same as in previous years.
So in conclusion, just like anything else AI related, there's a lot of hype on how this is disruptive and it's terrible or game-changing depending on which side you're on. And yet, in practice...
Has anyone had similar observations? I'd love to hear your thoughts.
@WestLawns On the surface, I totally agree with you. There are, however, several more complex issues.
1. Pedagogically, genAI is problematic in the context of traditional assessments (eg because of cognitive offload).
2. There are many ethical issues surrounding genAI usage (copyright, privacy, ecological, modern slavery etc)
3. There is a question of fairness. Our University policy is that genAI use is not allowed in assessment unless specified on a course by course basis. Not every student will want to use it (it's academic misconduct) and given in 90% of cases it is essentially impossible to reliably detect it, students not using it might be at a disadvantage although they are doing the right thing. Even then, richer students who can afford to pay for better models will be more advantaged.
So, yes, it's the same story as with calculators, but at the same time it is not!
