Results from poll #2

"Modifying readability with LLMs" is a winner

This month’s poll for paying subscribers presented four options:

1. A short follow-up to my empirical study looking at how well large language models (LLMs) estimate the readability of different texts.

2. A longer (>1 month) follow-up to that same readability study asking how well LLM can modify the readability of a text.

3. An empirical test of the so-called “winter break hypothesis”.

4. An empirical study on whether movies these days are more frequently about the past than they used to be.

The winner—by a narrow margin—was (2): a more intensive follow-up looking at how well LLMs can modify the readability of a piece of text. I’m excited to work on this!

Results of February’s poll.

This was a pretty close one. Sorry to those of you who were rooting for the winter break hypothesis. Because that one was close behind, I’ll likely include that option in a future poll.

What to expect

Like I mentioned in the original poll, this readability study will be a little more involved. The primary reason for this is that I’d like to collect human judgments to assess how successfully LLMs modify the readability of a piece of text. I could also calculate this using readability formulas or another LLM (and I will), but I’m not quite ready to give up on humans as the “gold standard” just yet, since “readability” is ultimately a human construct.

But that does mean that I’ll run into complications that wouldn’t arise if I were just using LLMs. Just to give you a sense of what I’m envisioning, here’s what I wrote in the original description of the project:

• I’d select a subset of texts (e.g., 100-200 texts) from the CLEAR corpus as my experimental materials.

• Then, I’d ask GPT-4 Turbo to generate an “easier” version of each of these texts (or possibly a few easier versions); I might also try the opposite direction too (e.g., create a less accessible version).

• Then, I’d run an online human study (e.g., on Prolific) asking people to judge the readability of those new texts. Realistically this would probably involve comparing the original text to the new one and asking which is easier to read.

• Optimistically, I’d also like to run a parallel study asking people to judge whether the modified version loses any information or whether it’s still just as informative as the original.

• Just for fun, I might ask GPT-4 to do the same task as the humans and see how well its judgments stack up against the human judgments.

• Finally, I’d ask whether the modified texts are, on average, judged to be easier than the originals—and also whether they’re judged as losing any information.

There are a number of decision points throughout, but the studies with human participants will definitely be the biggest investment. To keep it tractable (and affordable), I’ll need to limit the number of texts I consider in my sample—particularly since I’d like to get multiple people to make judgments about each text to improve reliability.

My rough estimate is that this will take at least 2 months from start to finish (i.e., including the write-up), and possibly a third. As I mentioned in the February poll, that means there won’t be an additional poll in March. But that doesn’t mean there won’t be any posts! Here’s what you can expect:

• Periodic (~1x per month) updates while this project is still ongoing. That includes: planning decisions, exploratory data analysis, and more.

• Plenty of the usual content, including some upcoming posts on external validity, negation in LLMs, and so-called “good-enough processing” in human language comprehension.

As always, thanks for reading and for the support.