"In many situations, we’re interested in reading something because we think a human wrote it." This is definitely the issue for me when reading Substack or LinkedIn posts, because of the corollary that follows: if a human wrote it, they might care about what I say about it. Whatever intelligence I might grant LLMs right now, caring about what I think about what they said is not among them. For now, responses to comments often make clear what was originally human-authored and personal and what was not.
Good point! I agree, and I think that's also why some people (including myself) might feel that sense of betrayal upon realizing that something was LLM-generated. If you spend time not only reading something but crafting a response, it's presumably because (as you say) you care, at least marginally, about what they think about what you think.
It's really interesting to read some normative thoughts about AI use from this research-informed perspective! I think I largely agree with the "job" vs "gym" sort of model; however, what worries me is that (1) I think, in practice, very few things would or should be considered "jobs" in Thompson's sense, and (2) the general direction many people seem to be going in involves classifying more and more things as "jobs".
I think research is a great example, where plenty of people have already been incorporating LLMs at nearly every stage. I've seen AI to generate research protocol proposals, AI to run stats on your data, AI to generate a manuscript. I don't think any of this is inherently bad! In fact, I think it's likely that AI models are quickly outpacing humans in important things like, say, interpreting radiological images. This will probably be great for patient outcomes and so on. On the other hand, I like to think that at least part of the point of research is for the researchers (and people in general) to better understand things. So if I let an LLM do all these "job" tasks for me (and I don't also ask it to explain what it's doing), it seems like my actual job is incomplete in some way.
A separate, but related thought: in my experience, many people -- especially young people -- are also fearful of AI precisely *because* it threatens to do their jobs (I mean it in both senses here). In the first sense, I'm referring of course to the more pragmatic worry that current grads might be unable to find work as roles get filled by AI. In the second sense, though, I'm referring to the idea that people might desire both "job" and "gym." That is, one might want to spend effort on something *because* the outcome is important. Having put the effort in (rather than having automated it) makes one feel, maybe, like they meaningfully contributed to something. I think this notion is more widely discussed and advocated for in creative disciplines like art, as you mention in this article, but I suspect it could be relevant to a lot of other disciplines as well (whether people acknowledge it or not).
> That is, one might want to spend effort on something *because* the outcome is important. Having put the effort in (rather than having automated it) makes one feel, maybe, like they meaningfully contributed to something. I think this notion is more widely discussed and advocated for in creative disciplines like art, as you mention in this article, but I suspect it could be relevant to a lot of other disciplines as well (whether people acknowledge it or not).
I agree. I focused on things like art here, in part because I think it's the clearest case where people (usually) value the process behind an outcome, but on an individual level I do think a lot of people derive meaning from the act of doing work that also leads to a good outcome, i.e., it's hard to decouple the "good feeling" one gets from having produced a good outcome from the fact that one directly participated in bringing about that outcome.
You might be interested in this essay by Harvey Lederman on "ChatGPT and the meaning of life", which takes seriously the possibility of systems that could actually supersede human competence on a range of tasks and asks what sources of meaning we might find in such a world: https://scottaaronson.blog/?p=9030
Stop pretending this is about whether the prose has machine fingerprints. That's a temporary detection game. The deeper question is whether the writer used the machine to sharpen their own thought or to avoid having one.
"A long and complex train of thought can no more be carried on without the aid of words, whether spoken or silent, than a long calculation without the use of figures or algebra."
— Charles Darwin, The Descent of Man, 1871
You mention that it's becoming a cliche that the "writing process is deeply intertwined with the thinking process". What's interesting to note is that this intuition runs counter to the arguably dominant position in the cognitive sciences—that "Language is primarily a tool for communication rather than thought" (Fedorenko, et al., 2024).
So which is it? Is the production of words central to certain kinds of "thought," or merely output for communicative purposes? Here is my attempt at a computational argument in support of your and Darwin's intuition that words are in fact necessary for some kinds of thought: https://philarchive.org/rec/HUDTOS
I'll check out your paper later today, but I *do* think language is a kind of "cognitive tool", at least in part, though I don't think that's mutually exclusive with it being evolved for communication (see below). Re: its use as a device for thinking, I'm influenced pretty strongly here by the work of Gary Lupyan and others, e.g., the idea that words help anchor / stabilize conceptual representations, and also facilitate a kind of conceptual abstraction and compositionality that's harder to imagine without such symbols. See, e.g.,
> Lupyan, G. (2016). The centrality of language in human cognition. Language Learning, 66(3), 516-553.
> Dove, G. (2020). More than a scaffold: Language is a neuroenhancement. Cognitive neuropsychology, 37(5-6), 288-311.
Perhaps more controversially, I even think the idea that language has a statistical structure, which bears out certain conceptual regularities—as evidenced by the fact that LLMs work at all—could be seen as a kind of cognitive scaffold. I don't know about you, but I've had times where I feel like I'm just producing words without planning my utterance or even having a clear idea of what I'm going to say, but somehow the sequence of words just "walks" me towards some conceptual destination. Presumably in the context of effective communication this process is harnessed by our broader cognitive system (i.e., in the service of some communicative intent). But I think we can equally see these as an example of linguistic structure/statistics working as a cognitive scaffold.
At the same time, I think this question can be separated to some extent (though not entirely) from the debate taken up by Fedorenko et al. (2024), which is positioned as contrasting with a different (Chomskyan) intuition that the communicative function of language is a kind of "spandrel". As I understand them, Fedorenko et al. (2024) are pushing back against the idea (championed by Chomsky and others) that communication is *not* the primary purpose of language. A conceptually similar argument is made in Piantadosi et al. (2012), which pushes back against a more specific claim by Chomsky, namely that the presence of ambiguity in language is evidence that it's *not* well-designed for communication. (Piantadosi et al. suggest that ambiguity is actually a design feature of an efficient communication system).
Now, I don't entirely agree with this interpretation of the evidence re: ambiguity (see Trott & Bergen, 2020). But I do tend to align with the theoretical view that language is adapted at least in part for effective communication, and that we can understand this communicative system by looking to the communicative systems of our evolutionary ancestors. I don't think that's incompatible with the view that it *also* serves a cognitive function. For example, just as Chomsky suggests that language is the result of some biological adaptation specialized for certain cognitive processes (e.g., the "merge" operation), and the use of language for communication is a spandrel, we could suggest the opposite: maybe language evolved "for" communication, but turns out to be a pretty effective cognitive tool too. I think the success of language models is actually a kind of proof of concept that the structure and statistics of language is sufficient to produce something *like* thinking, even if one disagrees that it's constitutive of thought. (See, again, Lupyan et al. 2026).
> Piantadosi, S. T., Tily, H., & Gibson, E. (2012). The communicative function of ambiguity in language. Cognition, 122(3), 280-291.
> Trott, S., & Bergen, B. (2020). Why do human languages have homophones?. Cognition, 205, 104449.
> Lupyan, G., Gentry, H., & Zettersten, M. (2026). How important is language for human-like intelligence?. Perspectives on Psychological Science, 17456916251398539.
"Perhaps more controversially, I even think the idea that language has a statistical structure, which bears out certain conceptual regularities—as evidenced by the fact that LLMs work at all—could be seen as a kind of cognitive scaffold. I don't know about you, but I've had times where I feel like I'm just producing words without planning my utterance or even having a clear idea of what I'm going to say, but somehow the sequence of words just "walks" me towards some conceptual destination."
This gets very close to summarizing my thesis! However, "scaffold" would seem to imply something less essential and constitutive than I argue for (or than Darwin insists in the quote above). What I attempt to do is identify a category of cognitive tasks that are indeed not possible without the use of something that has the particular properties language does. I use displaced causal reasoning to motivate the argument, but it extends to any reasoning that requires navigating dependency structures (multi-step planning, mathematical derivation, logical proof). The associative learning mechanisms that all other animals rely on are, I argue, structurally incapable of handling these problems. But the "prediction" of each next word during speech/writing utilizes the statistics of everything you've processed previously to guide—or "walk"—you toward constructions applicable to your current goals.
> However, "scaffold" would seem to imply something less essential and constitutive than I argue for (or than Darwin insists in the quote above). What I attempt to do is identify a category of cognitive tasks that are indeed not possible without the use of something that has the particular properties language does.
That's fair, the Guy Dove paper I mentioned above also makes a similar case (the title is literally "more than a scaffold"), and I'm definitely open to the argument that language is in fact necessary to some tasks.
BTW, I just started your paper and was struck by this sentence:
> For the first time, however, we now have a nonbiological entity with human-like language capabilities. Might large language models (LLMs) offer the experimental proxy required to empirically isolate language’s
contribution to cognition?
This is really similar to how I often frame my own research in talks these days—I use LLMs as "model organisms" to isolate what kinds of behaviors you can expect from exposure to linguistic input alone. Looking forward to reading the rest of the paper!
The first answer from an LLM can set the rhythm of the next thought so fast. Once that cadence gets in your head, it’s weirdly hard to hear your own draft again.
"In many situations, we’re interested in reading something because we think a human wrote it." This is definitely the issue for me when reading Substack or LinkedIn posts, because of the corollary that follows: if a human wrote it, they might care about what I say about it. Whatever intelligence I might grant LLMs right now, caring about what I think about what they said is not among them. For now, responses to comments often make clear what was originally human-authored and personal and what was not.
Good point! I agree, and I think that's also why some people (including myself) might feel that sense of betrayal upon realizing that something was LLM-generated. If you spend time not only reading something but crafting a response, it's presumably because (as you say) you care, at least marginally, about what they think about what you think.
Case in point! :)
It's really interesting to read some normative thoughts about AI use from this research-informed perspective! I think I largely agree with the "job" vs "gym" sort of model; however, what worries me is that (1) I think, in practice, very few things would or should be considered "jobs" in Thompson's sense, and (2) the general direction many people seem to be going in involves classifying more and more things as "jobs".
I think research is a great example, where plenty of people have already been incorporating LLMs at nearly every stage. I've seen AI to generate research protocol proposals, AI to run stats on your data, AI to generate a manuscript. I don't think any of this is inherently bad! In fact, I think it's likely that AI models are quickly outpacing humans in important things like, say, interpreting radiological images. This will probably be great for patient outcomes and so on. On the other hand, I like to think that at least part of the point of research is for the researchers (and people in general) to better understand things. So if I let an LLM do all these "job" tasks for me (and I don't also ask it to explain what it's doing), it seems like my actual job is incomplete in some way.
A separate, but related thought: in my experience, many people -- especially young people -- are also fearful of AI precisely *because* it threatens to do their jobs (I mean it in both senses here). In the first sense, I'm referring of course to the more pragmatic worry that current grads might be unable to find work as roles get filled by AI. In the second sense, though, I'm referring to the idea that people might desire both "job" and "gym." That is, one might want to spend effort on something *because* the outcome is important. Having put the effort in (rather than having automated it) makes one feel, maybe, like they meaningfully contributed to something. I think this notion is more widely discussed and advocated for in creative disciplines like art, as you mention in this article, but I suspect it could be relevant to a lot of other disciplines as well (whether people acknowledge it or not).
Thanks, Sam!
Regarding your last point:
> That is, one might want to spend effort on something *because* the outcome is important. Having put the effort in (rather than having automated it) makes one feel, maybe, like they meaningfully contributed to something. I think this notion is more widely discussed and advocated for in creative disciplines like art, as you mention in this article, but I suspect it could be relevant to a lot of other disciplines as well (whether people acknowledge it or not).
I agree. I focused on things like art here, in part because I think it's the clearest case where people (usually) value the process behind an outcome, but on an individual level I do think a lot of people derive meaning from the act of doing work that also leads to a good outcome, i.e., it's hard to decouple the "good feeling" one gets from having produced a good outcome from the fact that one directly participated in bringing about that outcome.
You might be interested in this essay by Harvey Lederman on "ChatGPT and the meaning of life", which takes seriously the possibility of systems that could actually supersede human competence on a range of tasks and asks what sources of meaning we might find in such a world: https://scottaaronson.blog/?p=9030
That looks like a great read - thanks for sharing!
Stop pretending this is about whether the prose has machine fingerprints. That's a temporary detection game. The deeper question is whether the writer used the machine to sharpen their own thought or to avoid having one.
Yes, that's consistent with the point I'm making, particularly when it comes to using an LLM in one's own work.
"A long and complex train of thought can no more be carried on without the aid of words, whether spoken or silent, than a long calculation without the use of figures or algebra."
— Charles Darwin, The Descent of Man, 1871
You mention that it's becoming a cliche that the "writing process is deeply intertwined with the thinking process". What's interesting to note is that this intuition runs counter to the arguably dominant position in the cognitive sciences—that "Language is primarily a tool for communication rather than thought" (Fedorenko, et al., 2024).
So which is it? Is the production of words central to certain kinds of "thought," or merely output for communicative purposes? Here is my attempt at a computational argument in support of your and Darwin's intuition that words are in fact necessary for some kinds of thought: https://philarchive.org/rec/HUDTOS
Great question!
I'll check out your paper later today, but I *do* think language is a kind of "cognitive tool", at least in part, though I don't think that's mutually exclusive with it being evolved for communication (see below). Re: its use as a device for thinking, I'm influenced pretty strongly here by the work of Gary Lupyan and others, e.g., the idea that words help anchor / stabilize conceptual representations, and also facilitate a kind of conceptual abstraction and compositionality that's harder to imagine without such symbols. See, e.g.,
> Lupyan, G. (2016). The centrality of language in human cognition. Language Learning, 66(3), 516-553.
> Dove, G. (2020). More than a scaffold: Language is a neuroenhancement. Cognitive neuropsychology, 37(5-6), 288-311.
Perhaps more controversially, I even think the idea that language has a statistical structure, which bears out certain conceptual regularities—as evidenced by the fact that LLMs work at all—could be seen as a kind of cognitive scaffold. I don't know about you, but I've had times where I feel like I'm just producing words without planning my utterance or even having a clear idea of what I'm going to say, but somehow the sequence of words just "walks" me towards some conceptual destination. Presumably in the context of effective communication this process is harnessed by our broader cognitive system (i.e., in the service of some communicative intent). But I think we can equally see these as an example of linguistic structure/statistics working as a cognitive scaffold.
At the same time, I think this question can be separated to some extent (though not entirely) from the debate taken up by Fedorenko et al. (2024), which is positioned as contrasting with a different (Chomskyan) intuition that the communicative function of language is a kind of "spandrel". As I understand them, Fedorenko et al. (2024) are pushing back against the idea (championed by Chomsky and others) that communication is *not* the primary purpose of language. A conceptually similar argument is made in Piantadosi et al. (2012), which pushes back against a more specific claim by Chomsky, namely that the presence of ambiguity in language is evidence that it's *not* well-designed for communication. (Piantadosi et al. suggest that ambiguity is actually a design feature of an efficient communication system).
Now, I don't entirely agree with this interpretation of the evidence re: ambiguity (see Trott & Bergen, 2020). But I do tend to align with the theoretical view that language is adapted at least in part for effective communication, and that we can understand this communicative system by looking to the communicative systems of our evolutionary ancestors. I don't think that's incompatible with the view that it *also* serves a cognitive function. For example, just as Chomsky suggests that language is the result of some biological adaptation specialized for certain cognitive processes (e.g., the "merge" operation), and the use of language for communication is a spandrel, we could suggest the opposite: maybe language evolved "for" communication, but turns out to be a pretty effective cognitive tool too. I think the success of language models is actually a kind of proof of concept that the structure and statistics of language is sufficient to produce something *like* thinking, even if one disagrees that it's constitutive of thought. (See, again, Lupyan et al. 2026).
> Piantadosi, S. T., Tily, H., & Gibson, E. (2012). The communicative function of ambiguity in language. Cognition, 122(3), 280-291.
> Trott, S., & Bergen, B. (2020). Why do human languages have homophones?. Cognition, 205, 104449.
> Lupyan, G., Gentry, H., & Zettersten, M. (2026). How important is language for human-like intelligence?. Perspectives on Psychological Science, 17456916251398539.
"Perhaps more controversially, I even think the idea that language has a statistical structure, which bears out certain conceptual regularities—as evidenced by the fact that LLMs work at all—could be seen as a kind of cognitive scaffold. I don't know about you, but I've had times where I feel like I'm just producing words without planning my utterance or even having a clear idea of what I'm going to say, but somehow the sequence of words just "walks" me towards some conceptual destination."
This gets very close to summarizing my thesis! However, "scaffold" would seem to imply something less essential and constitutive than I argue for (or than Darwin insists in the quote above). What I attempt to do is identify a category of cognitive tasks that are indeed not possible without the use of something that has the particular properties language does. I use displaced causal reasoning to motivate the argument, but it extends to any reasoning that requires navigating dependency structures (multi-step planning, mathematical derivation, logical proof). The associative learning mechanisms that all other animals rely on are, I argue, structurally incapable of handling these problems. But the "prediction" of each next word during speech/writing utilizes the statistics of everything you've processed previously to guide—or "walk"—you toward constructions applicable to your current goals.
> However, "scaffold" would seem to imply something less essential and constitutive than I argue for (or than Darwin insists in the quote above). What I attempt to do is identify a category of cognitive tasks that are indeed not possible without the use of something that has the particular properties language does.
That's fair, the Guy Dove paper I mentioned above also makes a similar case (the title is literally "more than a scaffold"), and I'm definitely open to the argument that language is in fact necessary to some tasks.
BTW, I just started your paper and was struck by this sentence:
> For the first time, however, we now have a nonbiological entity with human-like language capabilities. Might large language models (LLMs) offer the experimental proxy required to empirically isolate language’s
contribution to cognition?
This is really similar to how I often frame my own research in talks these days—I use LLMs as "model organisms" to isolate what kinds of behaviors you can expect from exposure to linguistic input alone. Looking forward to reading the rest of the paper!
The first answer from an LLM can set the rhythm of the next thought so fast. Once that cadence gets in your head, it’s weirdly hard to hear your own draft again.