Generative AI and the Anxieties of Academic Writing

I’ve been a blogger for as long as I’ve been an academic writer, even if I’ve been a writer for longer than I’ve been a blogger. After two decades of regular blogging, on a succession of strange and deeply personal spaces before launching my current blog in 2010, it was difficult for me to untangle the relationship between blogging and writing. I’d written on many occasions about the role of blogging in my enjoyment of writing, suggesting that it provided a forum in which ideas could be worked out in a public relationship to a slightly nebulous audience (Carrigan 2019). If I return to the end of my part-time PhD I can see that I understood this relationship in terms of a freedom from constraint, reflecting in Carrigan (2014) that “Blogging was a release from the all the structure pressures corroding the creative impulse” which “helped me make my peace with the jumping through hoops that a modern academic career unavoidably entails”. The fact that “I can write whatever the hell I want here” helped me “feel better about subjugating what I want to write to instrumental considerations elsewhere”.

In other words, it helped me find a particular way of trying together my internal and external motivations. It provided a forum for craft writing, passionate writing motivated purely by my own interests, as opposed to the extrinsically motivated writing which I imagined defined the priorities of the working academic. It left me with a stark opposition between what I wanted to do and what I had to do, treating the former as a palliative which made the latter more bearable. Ten years later at a mid-career stage, this compromise no longer seems tenable to me and I find it strange that it ever did. It suggests to me a difficulty in reconciling oppositions, as if something could be done entirely for my own reasons or entirely to please others but the two clusters of motivations could never meet.

This tension between writing for ourselves and writing for others sits at the heart of many academic anxieties. It’s also precisely the space where generative AI now intervenes, promising to smooth over the difficulties and frictions that define our relationship with writing. Are you present when you are writing? Or are you somewhere else? Are you feeling an energy to the words as you are writing them? Or are you watching the clock, literally or figuratively, waiting to meet your target or for the time you’ve carved out to elapse? These questions about presence and engagement become even more pressing when AI tools offer to take over the aspects of writing we find most challenging. The parts where we struggle, where we feel most distant from our words, are exactly where the temptation to outsource becomes strongest.

I’ve drawn attention throughout this book to the audience we are addressing (or failing to) through our writing. For many academic writers, this sense of audience can be overwhelming as a vector of expectation. How will I please them? What if they don’t like what I’ve written? What if I’m not taken seriously? These expectations are filtered through real encounters from the notorious reviewer two, through to encouraging supervisor or the dismissive colleague at a seminar. These encounters might be mediated or predicated upon inaction, such as the paper which goes determinedly uncited by others, even as the view count slowly ratchets up on the journal’s page. However they are often defined by an anticipatory anxiety in which these experiences mutate into a diffuse sense of what our professional community expects from us and what we feel we are able (or unable) to deliver to them through our writing. Even the functional writing which fills our days has an audience implicit within it. It’s not just that our emails, reports and forms will have readers, rather we are trying to influence or bring about an effect in them through what and how we write (Jones 2022: 9).

Often these intentions are so familiar and mundane that they operate beneath the surface, only becoming apparent to us when when we realise our email has been misconstrued or our form rejected for what is perceived as some mistake. But this doesn’t diminish the role of the audience, as much as it shows how these dynamics can be folded into the functional routines of the bureaucracies within which we work. If you see machine writing as a means to an end, you’re unlikely to enter into this dynamic. Instead you will approach this software as a way of producing something as quickly as possible, whether that’s a section of a document to ‘fill in the blanks’ or a complete text. As the philosopher Gillian Rose (1995) once described writing: “that mix of discipline and miracle, which leaves you in control, even when what appears on the page has emerged from regions beyond your control”. If we use conversational agents purely to expand our control, to enact our aspiration in ever more effective ways, we imperil our access to those ‘regions beyond our control’ from which inspiration emerges.

Generative AI and the challenge of unbidden thoughts

The novelist Anne Mallot (1991: 138) describes how “unbidden, seemingly out of nowhere, a thought or image arrives” which will “float into your head like a goldfish, lovely, bright orange, and weightless, and you follow them like a child looking at an aquarium that was thought to be without fish. They’re often so rich, these unbidden thoughts, and so clear that they feel indelible. But I say write them all down anyway.”

Where do you write them down? Early in my research career I carried ornate notebooks around with me in which I recorded these thoughts as they emerged. At that stage they felt precious to me as the repositories in which my ideas would slowly gestate. I imagined that to lose them would be to lose the ideas, the precious elements through which I would eventually cobble together a thesis in a process which still seemed unsettlingly opaque to me.

So I’ve written it down. Now what? At the time of writing I have 6,303 posts on my research blog over a fourteen year period. I often recall something, usually a quote or an idea, which I then search for to find. I occasionally feel moved to read back through old posts, usually the things I have written in the last few weeks, though every so often going back to a particular year when I recall something I was working on but which I have since dropped. Otherwise the ideas sit inert on my blog, with the potential to make themselves known if someone else stumbles across them. There’s a value in having written them, in the sense that it embeds the idea into my psyche in a similar way to writing them in a physical notebook. I become someone who has thought that thought by virtue of having written it, in a way that simply registering the thought fails to do. But for the most part it simply gets the thought out of my head, into what the technologist Cory Doctorow (2008) describes as an ‘outboard brain’, before leaving it there.

Obviously a blog is not the only way in which academics might keep a file, to use the term favoured by C. Wright Mills (2000). It’s just an interesting one by virtue of its public and iterative character. The growth of knowledge management systems like Roam and Obsidian in recent years has been a fringe but notable feature of academic practice, enabling complex systems of notes to emerge which can be queried in a much more efficient way than I can work with my blog.

But collecting thoughts is only the beginning. How do we transform these fragments into coherent writing? This question has taken on new significance with the emergence of AI writing tools. The relationship between capturing ideas and developing them has been fundamentally altered by technologies that promise to help us articulate what we’re struggling to express.

The writing scholar Elbow (1981:26) recommends a practice of speedily recording everything “you can think of that might belong or pertain to your writing task: incidents that come to mind for your story, images for your poem, ideas and facts for your essay or report”. His advice is that you not “waste any time or energy on how to organize it, what to start with, paragraphing, wording, spelling, grammar, or any other matters of presentation”. Instead if “you can’t find the right word just leave a blank” and if “you can’t say it the way you want to say it, say it the wrong way”.

Using machine writing in this way involves plentiful writing. If you agonise over each word or sentence of a prompt, it will be a time consuming and difficult process which still fails to provide the model with sufficient guidance to reach its full capacity. In contrast if you can write quickly and plentifully, prioritising content over form, you provide the context which ensures that machine writing revolves around your intended project. It can be liberating to realise that this doesn’t have to be clear and coherent in the manner necessary for a human writer to comprehend you. Elbow (1981: 24) observes how the predictable misspeaking of his two year old son doesn’t cause a problem for the parents because “we hear through the externals of his language to the meanings and intentions that lie behind”. In most cases your adult interlocutors will not be able to do that, creating the need to optimise the form of your writing to ensure the substance is communicated effectively. This need is less pronounced with models because they are ultimately engaging with the network of your words rather than the linear sense making process of a human reader.

But this raises a deeper question: what happens to our relationship with difficult ideas when AI can smooth away the struggle?

What happens when writing is hard? It’s difficult to find what I’m trying to say. I’m embodying the argument I’m trying to make. I feel like I’m circling around my argument, with a sense of what I mean but does anyone else understand what I mean? Perhaps these are platitudes but I really feel that I’m on to something here. That I’ve gone far enough into this that there’s something I’ve intuited about how LLMs can change writing which really needs to be expressed. But it’s about the difficulty of expressing things and what that means for how we look to models. I wonder if the model can help me.

When I shared this with Claude 3.5 in the context of my ongoing project (with the associated knowledge base) it immediately offered an incredibly succinct statement of my core argument: “writing is inherently difficult precisely because it involves wrestling with partially-formed thoughts and bringing them into clear articulation” and that “LLMs present themselves as a solution to this difficulty, but in doing so they risk short-circuiting the very process that makes writing valuable – both as an intellectual endeavour and as a source of satisfaction”. This is my argument, offered in an oddly satisfying pastiche of my writing style, but the formulation is Claude’s. It distilled what I’ve been circling around, over tens of thousands of words, into a couple of sentences. It even offered ‘key themes’ and ‘suggestions for development’ which connected this short passage of free writing to overarching themes within the knowledge base I had provided it with.

Oddly it was able to do something nearly comparable when I offered the same passage to an instance of the model which didn’t have this significant amount of my existing writing as a context:

“Your message resonates deeply with that universal writer’s struggle – the feeling of circling an insight that feels just out of reach. It seems you’re wrestling with something profound about the relationship between the difficulty of writing and how AI language models fit into that process.

Perhaps the very difficulty you’re describing – that tension between intuition and expression – is central to your emerging argument? The struggle to articulate our thoughts isn’t just an obstacle to writing, but an essential part of how we come to understand what we think.

When you say you’re ’embodying the argument,’ it strikes me that you’re living the very phenomenon you’re trying to describe – the complex relationship between thought, expression and the role AI might play in that space.”

Note the self-anthropomorphising language used here: “it strikes me”. Obviously it doesn’t strike Claude in the sense of an insight occurring in a flash. But it discerns a connection latent within the tokens I have provided it with, leading to a way of organising these connections which produces an effect within me.

This experience illuminates a fascinating paradox. The very tools I’m using to understand how AI affects writing are themselves demonstrating the phenomenon I’m trying to describe. When Claude articulated my half-formed thoughts with such clarity, I felt both relief and a subtle unease. Relief at seeing my intuition confirmed and shaped into words; unease at how easily the difficult work of articulation was handed to me.

These “unbidden thoughts” that Mallot describes, the goldfish swimming into view, still require our active engagement to become meaningful. Our outboard brains, whether notebooks, blogs, or digital gardens, serve not just as storage but as spaces for thinking to develop. AI can now join these thinking tools, offering remarkable assistance in crystallising ideas, but perhaps at a cost.

The question becomes not whether to use these tools, but how to preserve the valuable struggle of bringing thoughts into being while still benefiting from AI’s capacity to help us see connections and formulations we might otherwise miss.

How the GAI Assessment Debate Has Led Us in the Wrong Direction

The common thread running through these debates is a fixation on outputs. GAI captured attention because of the ease with which eerily human outputs could be produced in response to natural language prompts. They may have been profoundly mediocre, at least initially. But it was still a remarkable discovery liable to unsettle the self-conception of those who work with text for a living. In fact it’s hard not to suspect this often underpinned the determination exhibited by many to explain away these automated texts as mediocre. The concern about academic integrity, what I describe in Carrigan (2024: ch 1) as the great assessment panic, rested on a similar assumption. It was now possible for students to substitute a human generated response to an assessment task with a machine generated response, creating an urgent need to find a way to distinguish between them.

There has thankfully been a widespread recognition that detection tools are unable to authoritatively identify machine generated text. They might detect patterns in writing which could be statistically associated with the use of GAI systems but they might also be associated with the style of those who are writing in a second or third language. Their demonstrated propensity to trigger false positives needs to be taken extremely seriously, because assuming a student of trying to cheat who has done no such thing must surely be weighed up against letting students through the net who have relied on GAI to produce their work.

The fact that one of the major detection tools has pivoted towards a water-marking model, effectively asking students to self-surveil the writing process in order to demonstrate their contribution to it, stands as a tacit admission to these limitations. There are certainly flags featuring in texts which should lead to a request for an explanation from a student, ranging from leaving what appears to be part of the prompt in the text or functional parts of the conversational agent’s response though to departures from assigned reading, odd stylistic fluctuations or hallucinated references. But the idea there is some conclusive means by which we could determine machine generated text, as opposed to a balance of evidence informing the decision to ask the student about their authorship, has rapidly unravelled in ways that can still feel troubling even to those who struggled to see how such an outcome could be possible from the outset.

It is slightly too easy to respond to this scenario by suggesting that assessment practice has to change and that perhaps the essay questions most vulnerable to machine generated responses always had their limitations. As I’ve argued previously, we’re now being forced to face limitations that were exposed by an essay mill industry which itself was facilitated by an earlier wave of digitalisation, in the sense that the scale it reached depended on platforms which could bring together buyers and sellers from across the globe (Carrigan 2024: ch 1).

But the idea that problematic use of GAI can be avoided entirely through shifting to more authentic and processual forms of assessment, involving ‘real world’ tasks in which students make things with assessment potentially informed by contextual knowledge of their progress through a module, should be treated carefully. Not least of all because there’s probably only so many podcasts or posters you should ask a student to do each year. But it goes deeper than a lack of creativity about what these new assessments should entail and the corresponding problem of student boredom.

Kay, Husbands and Tangen (2024) suggested plausibly that the rapidly emerging “cottage industry … promising to translate traditional assessments into supposedly generative-AI proof formats … greatly underestimate the power of current LLMs”. Not only current LLMs but the ones which will be released in the near future. Since they published this article in early 2024, ChatGPT released their latest GPT-4o, Anthropic released Claude 3 Opus which was rapidly superseded by Claude 3.5 Sonnet and Alphabet released further iterations of their Gemini model. I’m writing these words in July 2024. By the time I finish this book, let alone by the time you read it, there will be further models with expanded capacities. This is unlikely to be an indefinite technological expansion of the kind claimed by evangelists who believe that ever increasing model sizes will eventually lead us to the fabled artificial general intelligence (AGI). But the expansion of practical capabilities is likely to continue, as these systems are refined as software, even if the technological development of LLMs eventually stalls.

The problem is that, as Mollick (2024) has pointed out repeatedly, a working knowledge of these capabilities is limited to those who are heavily engaged with them in an applied context. I’ve been using Anthropic’s Claude on a daily basis for well over a year at the time of writing, with a specific commitment to exploring how it can be used as part of my work and identifying new uses which might be valuable to other academics. Inevitably I’m more sensitive to the changes in the successive models than someone who occasionally uses it for specific purposes or who has experimented for a period of time then set it down. In fact I’m slightly embarrassed to admit what an exciting event it is for me at this stage when Anthropic release a new Claude.

There is a rapidly developing knowledge gap between those who are using LLMs in a sustained hands-on way and those casual users or non-users who are simply following at a distance. This means that decision making in higher education risks lagging behind the current state of practice, including our sense of what LLMs could and could not be used for by students. It would be a mistake to imagine our students are ‘digital natives’ who are intrinsically attuned to these developments in a way that leaves them much better able to exploit the new capabilities which come with each successive model (Carrigan 2021). But there clearly are students who are doing this and there are student user communities, found in places like Reddit, where knowledge of this is being shared.

It is difficult to estimate how wide such a student group is and my experience has been that most students, even on the educational technology master programme I teach, restrict themselves to using ChatGPT 3.5. Even so this knowledge gap means that we should treat our assumptions about what LLMs can do, as well as what at least some of our students can do with them, with a lot of caution. It probably isn’t possible to design an assessment which a student couldn’t in principle complete with the support of an LLM. It might be that most students couldn’t or wouldn’t do this. But the idea that we can definitely exclude LLMs from assessment through design ingenuity is likely to distract us from the real challenge here.

Instead of thinking in terms of outputs we need to think about process. This will help us move away from an unhelpful dichotomy in which entirely human-generated outputs are counterpoised to entirely machine-generated output. Rather than conceiving of the technology as somehow polluting the work of students, we could instead consider how the process through which they have crafted the work leads to certain kinds of outcomes which embody certain kinds of learning. This leaves us in a murky grey zone in which hybrid work, combining human and technological influences, should be taken as our starting point. Authentic forms of assessment are valuable because they provide opportunities for students to demonstrate their thinking process, not just their final product—showing how they approach problems, make decisions, and integrate various sources of information including, potentially, AI tools.

One of the first things I did when tentatively exploring ChatGPT 3.5 was present it with a title from a recent essay I had marked, in which a student critically reflected on their experience of using Microsoft Teams with a view to understanding the opportunities and limitations of the platform in an educational setting. It immediately produced a piece of writing which was blandly competent, self-reflecting in a way which was coherent but likely to receive a low B at most. I remember sending a message to a friend confidently stating that the work it produced was mediocre. I’m struck in retrospect at how loaded this confidence was, as if immediately feeling I could place it in a marking scheme gave me agency over what was portended by this development.

It’s a reaction I could see through the immediate discourse within the sector, as a frenetic anxiety which imagined the teaching and learning bureaucracy being near immediately washed away was combined with a confident dismissal of the mediocrity of the writing which was produced. As Riskin (2023) put it in a reflection on what she believed to be AI-student generated essays, these words often read like the “literary equivalent of fluorescent lighting”.

To confidently dismiss these outputs as mediocre tended to gloss over the troubling fact of how predominant mediocre writing is from our students, as well as in wider society. The bland competence which tended to characterise these early experiments with ChatGPT, the rhythm and resonance which betrayed a lack of human engagement with the task, was perhaps uncomfortably familiar to many academics. It’s the style of writing which results when the primary motivation of the writer is meeting requirements they have been set, jumping through hoops in order to achieve a specific goal. It’s the writing which ensues when extrinsic motivations predominate over intrinsic motivations, accomplishing something in the world by following a procedure rather than a creative process in which something is expressed.

It would be naive to imagine students might exclusively be motivated by the latter, in a context where studying involves taking on an enormous amount of debt and a potentially bleak social and occupational future awaits them at the end of the line. But the poetics of instrumentality are troubling, amongst other reasons, because they confront us with our own role within that system. They remind us of students not enjoying their writing, much as we ourselves often do not enjoy it. The fluorescent lighting of mechanically written work, intended to serve a purpose rather than exhibit meaning, is not confined to our students.

What concerns me is how many academics seem to have left matters here. Either experimenting with ChatGPT 3.5 themselves, or seeing the experiments of fellow academics, the matter seemed settled. The problem with such a view is not only the radical expansion in capabilities with successive waves of models. This is often expressed in terms of the parameters involved: escalating billions, through to trillions with Claude 3, which might have rhetorical effects if we could grasp what either the billions or the parameters mean concretely. This quantification offers little insight for most potential users about how the models are advancing.

In reality it is difficult to understand the expanding horizons of what they can do unless you are immersed in their daily use within a specific context. This is still such an unusual position to be in that it can be difficult to convey this understanding to colleagues who are undertaking their work without the benefit of a GAI assistant. In fact it might be prudent to keep this strange collaboration to yourself because there remains a lack of consensus in any profession as to what constitutes appropriate or inappropriate use of these technologies. Furthermore, there are such vast number of regulatory issues related to their use within workplaces that many employers would prefer to ensure they are not used, even if there is little way for them to enforce this decision when employees routinely work from home and use their own devices.