Language models and language change, redux

GPT-3 contributes to the conversation.

Over a month ago, I wrote a post enumerating the possible ways in which widespread adoption of language models could shape the evolution of human language. I described three possible futures:

1. Language models have no effect on language or language change (i.e., the null hypothesis).

2. Language models slow down language change, freezing current practices in place.

3. Language models speed up language change, and possibly take it to some very strange places.

The impetus for that original post came from thinking about (2) specifically, in the context of technologies homogenizing our reality. When you ask a model to predict the future based on the past, and then give that model the tools to actively shape the future, it’s reasonable to expect that a future created by that model will start to look a lot like the past.

Just now, I was playing around with GPT-3 for a different project, and decided––just for fun––to see how GPT-3 would respond to this issue.

GPT-3’s Contribution

I gave GPT-3 the following prompt:

How might the widespread use of predictive text shape our language? List three hypotheses and say which one is most likely.

This is how GPT-3 responded

First, I continue to be impressed by how fluently and naturally GPT-3 responds to prompts along the lines of “list N reasons”. I didn’t give GPT-3 guidelines in terms of how to format the response, nor did I provide any other context besides that initial question (“How might the widespread use of predictive text shape our language"?). Nonetheless, GPT-3 produced a reasonably-formatted response, which was further divided into a list of hypotheses and then it’s judgment of which hypothesis was most likely. Content aside, I think it’s fascinating how little GPT-3 sometimes needs to “get” what’s being asked. (Remember that this is, on some level, just a very sophisticated predictive text engine.)

Turning to the hypotheses themselves, you’ll note that they’re also pretty reasonable. In fact, they’re not too different from what I wrote in my original post. (3) is the most similar to my hypothesis that language models could fossilize certain linguistic practices in place, i.e., leading to “standardized communication”. (2) is more similar to my proposal that language models could take us into linguistically uncharted waters, i.e., “more creative communication”.

What about (1)? Is “concise communication” more like standardization or more like unforeseen change? In the limit, an increasingly concise communication system could converge on the Zipfian ideal of a speaker-centric language, in which speakers produce only a single word (e.g., “ba”) to convey every possible meaning. This would be quite different from any language we’ve ever seen before. Notably, this interpretation is also similar to what I labeled “Very Weird”, under the broader scenario of language models speeding up language change.

Here’s the relevant section:

Or to make things even more extreme: we don’t simply go around saying “ba ba ba”, because even if we knew that what we intended was “Yesterday I read an excellent paper about sound change”, our comprehender has no way of knowing. Human language is subject to the constraint of being interpretable by human comprehenders.

But the advent of PLMs might remove that pressure. At the very least, any given linguistic expression just needs to be comprehensible by a PLM, such that the PLM can tailor the expression to its human partner––thus reducing the need for human speakers to engage in audience design. And this might make language very odd indeed6.

This scenario assumed that a Personalized Language Model (PLM) would also be capable of summarizing linguistic input, thereby weakening the need for speakers to tailor their communication to a particular audience.

Thus, if I had to choose, I’d say the “concise communication” outcome also fits into the bucket of “language models taking language to weird places”.

What does this all mean?

This raises some questions.

Does GPT-3 understand what it wrote? Should we put stock in its responses? How should we feel about the fact that it produced a response that wouldn’t be out of place in my own essay, despite never having read my essay?

I’ll start with the first question. The issue of whether GPT-3 “understands” its input and output has been hotly debated, and as far as I’m concerned, we’re still in the Defining Terms stage of the debate. I’ve yet to see a satisfactory account of “understanding” that makes me think this question could be answered empirically rather than appealing to an a priori assumption. So for now I’ll just say: it’s unclear, though GPT-3 behaves as if it understands, and I’ll be revisiting this question in the future.

Now for the second question. If GPT-3 may not truly understand its input/output, should we put any stock in it at all? It depends on what “put stock in it” means. I don’t think we should trust GPT-3’s output, given its propensity to make up “facts” altogether.¹ But this response did make me consider––or at least flesh out––a scenario I'd neglected somewhat in my initial essay. Perhaps it's better to think of GPT-3 (at least for now) as providing some kind of stochastic inspiration––like a drunk but occasionally very interesting friend at the bar.²

As for the third question: how should we feel about the apparent sensibility of GPT-3’s response? I’m not sure there’s any particular way one should feel. Personally, I feel a mixture of impressed, skeptical, and amused. I’m impressed because the response so closely mirrors my own essay. I’m skeptical because I know humans are prone to imagining intelligence where it doesn’t exist, at least not in the way we might initially think, and so I’m trying consciously to exercise caution here. And finally, I’m amused because––again––GPT-3’s response wouldn’t look out of place as a summary of my own essay.

This means one of two things, perhaps both:

1. The ideas in that essay have a certain inevitability––logically, there’s only one way that essay could’ve turned out––which I suppose grants them with a certain legitimacy.

2. The ideas in that essay are so uncreative that something as simple as predictive text could have (and did!) come up with them.

1. I’ll be writing a post soon on whether GPT-3 could plausibly be used to “cheat” in academic classes.

2. For anyone who’s concerned by this sentence, I’m not saying it’s sentient. It’s an analogy.