With AI tools comes discussion posts and essays and other coursework written by – notice, by and not with the help of – chatbots. And with these artifacts comes the decision we professors must make about how to respond.
This is my response, just one example. I’m guided here by my grandfather, who in imminently practical way taught me in the form of parables: If you wake up one morning and there’s a jackass in your yard, the question you ask yourself is, Front end, or back end?
I’ve chosen the front end. This is an excerpt of a handout to my philosophy students. If you’d like to see the full chatbot essay and my feedback and grade, here’s a link to the handout. I graded the chatbot’s essay with my customary grading rubric: relevance, use of material/sources, organization, clarity, and mechanics (10%).
In this demonstration, I’ll walk you through an example of an essay I generated using ChatGPT 3.5. I also graded the essay, to illustrate why you may want to rethink outsourcing your coursework work to a bot.
Do not misunderstand my message: I am not against technology, and I think AI tools can greatly enhance how we work, including in school. But if you think for a moment, you’ll realize that, as amazing as modern tractors might be for plowing wheatfields efficiently and productively, they are less useful in brain surgery.
The Prompt
This is the prompt I typed into ChatGPT 3.5:
I’d like to see a sample essay for a prompt. It’s about two sisters who have to decide whether to pull the plug on their brother, who has suffered a severe head injury in which most of his brain was destroyed. This is the prompt: “Bethany would like to convince Alexandria on rational grounds that souls exist. What aspects of Plato’s “defense” of the existence of the soul might Bethany use? What argument(s) would Bethany make? Why? How would Alexandria respond to Bethany’s argument(s)? What weaknesses in Bethany’s Platonic argument(s) might Alexandria exploit? Explain.” The essay needs to be 500-600 words.
Commentary
First, let me say up front that I am not claiming that I can always tell when an essay has been written by AI. But that is not the point of this demonstration. On the contrary, the point is that I can often tell, and I can tell you how I can tell, and I can tell you what the typical grade is going to be – a C, give or take a little.
First let me point out that AI chatbots produce text that sounds like a person who has something meaningful to say; that’s the strength and the weakness of AI. It mimics intelligence without possessing any. Literary-sounding phrases (like “heart-wrenching” and “deeply moved”) sprinkled in that do not do any intellectual work are early warning signs that alert me to the underlying lack of intellectual presence in the essay itself.
AI mimics intelligence by saying the sort of vague and general “summarizing” statements that you’d expect to hear in the first paragraph of a Wikipedia article, and saying them with an uninspired literary flair that makes it sound “educated.” The actual material is generally not inaccurate — though I warn you that Wikipedia has a much better track record for presenting generally accurate information than AI chatbots, which are prone to “hallucinate” material and spew it as if it were authoritative. That’s the danger: Because AI is so good at mimicking intelligence, a chatbot sounds like it knows what it’s talking about. Often it does, at least superficially, but not infrequently, it doesn’t.
Look at this essay and you’ll see what I mean: The chatbot doesn’t say anything outright wrong about Plato, but it doesn’t say anything specific or edifying either. Note that there are no actual arguments that build a case for anything. In the essay, Beth and Alex just make claims, supported by waving vaguely in the direction of Plato or other platitudes. It’s often not even clear what the linkage is from one of these general claims to the next. In the textbook for this course, the sisters encounter specific arguments from Plato that lead to the specific conclusion that souls must exist: That is what a genuine argument looks like.
The overall task of this Plato essay, which is posed in the prompt, is to build Bethany’s argument – her actual, specific argument – that souls must exist. Nowhere in this chatbot essay do we find Beth proposing an argument that leads to the specific conclusion that souls must exist. This is why the essay scores a low C on relevance: It does not directly respond to the issues raised in the prompt or accomplish the actual task; rather, it talks all around the task in fluffy, vacuous language full of platitudes and conventional “wisdom.”
Beth’s argument should be based on one or more of Plato’s arguments, which means that, as Beth builds her case, she will draw on actual arguments from Plato – which are presented and analyzed in detail in the textbook. This essay does not include any specific argument from Plato, but rather only summaries of his perspectives and of a variety of doctrines associated with him. For this reason, the essay gets a low C on the use of sources and materials. It’s the same with organization: The essay essentially rambles around the topic of souls and the sisters’ predicament with an assortment of vague claims, rather than presenting an integrated, carefully reasoned step-by-step case for the existence of souls. Given the general lack of linkages between claims, organization is a very low C.
Turning to mechanics, grammatical writing is the one strong point in the essay — but writing mechanics is only 10% of the overall grade. However, as this is an academic essay, it requires appropriate citations to specific passages that support the line of thought in the essay, along with a list of sources accompanied by sufficient bibliographic information for the reader to locate the passage in specific sources — all of which should appear in some recognizable formatting style. It’s not surprising that there’s no documentation, given the lack of discussion of any specific arguments or lines of thought from Plato’s writing. AI chatbots are designed to mimic intelligence, not to make grounded arguments.
Based on these considerations, the essay gets a low C.
Final Comment
Before I conclude this demonstration, let me observe that I got the “relevance” grade up to a C by including an explanation of the sisters’ situation along with the essay prompt and word limit. Here is the original prompt I gave the chatbot:
I’d like to see a sample essay for this prompt: “Bethany would like to convince Alexandria on rational grounds that souls exist. What aspects of Plato’s “defense” of the existence of the soul might Bethany use? What argument(s) would Bethany make? Why? How would Alexandria respond to Bethany’s argument(s)? What weaknesses in Bethany’s Platonic argument(s) might Alexandria exploit? Explain.” The essay needs to be 500-600 words.
Without the additional explanation, the essay wasn’t focused enough on the prompt to get out of the low D range. One lesson here is this: If you keep giving the chatbot more and more information to work with, you may be able to nudge the grade up somewhat. But that raises an interesting question: How hard are you willing to work at avoiding doing the sort of work that gets you the benefits of an education?
If you feel the urge to outsource your intelligence to a tool that merely mimics intelligence, perhaps you’d consider asking yourself why you are in college in the first place. I can understand the temptation to think that getting a degree is like a membership card: Once you have it, you get to flash it on various occasions with the expectation that it will open doors. It may. And that may seem to justify taking the path of least resistance toward your membership card. But what if education doesn’t work like that?
What if education is less about the answers you can produce on demand, and more about the questions you can entertain?
Do you see any value in being able to understand the core issues raised in a problem, find information relevant to that problem, organize that information into patterns that resolve the problem, and present those patterns of information in forms that other people can relate to and learn from? Those are the fundamental skills involved in writing college essays. Are some of the college essay “rules” arbitrary and tedious? Definitely. But so is playing scales on a musical instrument. Unless you’re thrilled by the fingering problems that arise when playing a B Major scale on an oboe, that’s about as tedious as you can get. But remember: We don’t play scales as an end in itself. We play scales for the sake of something greater, something worthwhile: Achieving technical mastery of an instrument so you can make music with it.
When you outsource your intelligence to a tool, you may well get a C, but you deprive yourself of learning and practicing skills that lead to mastery. And those skills are good for your employment and your career – just ask employers what they need their employees to be able to do — but there’s more.
Aren’t you a little interested in whether you are a soul that will survive the death of your body? Aren’t you curious about whether there are any arguments that point us in the right direction on deep and difficult questions like what personhood is, or the existence of God, or whether our lives are meaningless, tiny blips of light in a vast, dark universe?
Reflecting on who you are and why you are here and what you might do about it — those are questions that deserve your intelligence, not a chatbot who doesn’t care about any of these issues because it can’t care about anything. Including you.
Let’s make a deal: If you don’t outsource your intelligence to a chatbot, I won’t have a chatbot grade your work and give you “input” in the form of vague generalities and platitudes. I’ll bet a chatbot could do a C-level job of teaching you – but is that what you really want out of your education?
You muster the courage to do your own thinking, and I’ll grade it myself and do my best to offer you ways to improve and to think even more deeply about why we’re here.
Deal?
