Results from poll #3 (and updates)
Tokenization's a winner, plus some administrative updates.
This month’s poll for paying subscribers presented three options:
A project empirically testing the “winter break hypothesis”.
A project asking about visual reasoning in multimodal large language models, using the famous “tangram” task.
An explainer on tokenization in large language models.
The winner was tokenization, which means you’ll be getting an in-depth explainer on what tokenization is, how it works (and why we do it), and what some of its consequences for how LLM process words are. I’ve been getting interested in this subject anyway, so I’m excited to work on an accessible review!
The visual reasoning task was a close second, so I’ll probably include that option in May’s poll. (The winter break hypothesis received no votes this time, despite being a runner-up in the previous poll.)
Additional newsletter updates
I also wanted to take this opportunity to share a few other updates about the newsletter and related topics.
First, I had a months-long email exchange with Benjamin Riley of Cognitive Resonance about LLM-ology and whether or not we need a different scientific paradigm to study how LLMs work. The first part of that exchange is now available on Ben’s Substack. I really enjoyed this discussion, and if you like the kinds of topics I write about on the Counterfactual I think you’ll enjoy it as well—check it out, along with Ben’s other posts on minds, machines, and the study of AI.
Second, one of my papers was recently published in the journal Behavior Research Methods. It is open access, so it should be available to anyone who wants to read it. I’ll probably be writing a post about that paper and a follow-up (which will hopefully be published soon). But if you enjoyed either of my posts on using LLMs to capture psycholinguistic judgments (the original post, and its follow-up), I think you’ll enjoy the paper.
Finally, in addition to my upcoming review of tokenization, you can expect some other posts, including: a reflection on the undergraduate course I just taught on LLMs, a close look at some recent results in chain-of-thought “reasoning”, and a discussion of learning across timescales.
As always, thanks for reading and for the support.