The Counterfactual’s 2024 in review

Wrapping up another year.

2024 is coming to a close, so I thought it was a nice time for a brief “year in review” post.

Thank you!

First off, I want to thank the 3000+ subscribers who now read The Counterfactual. I started this newsletter a couple of years ago as a way to jot down and consolidate random thoughts (research-related and otherwise), and I genuinely didn’t expect to have anything like the readership I have now. I am very grateful to everyone who subscribes and/or reads these posts, and I’m also grateful to people like Timothy Lee over at Understanding AI for having promoted this newsletter.

As you can see below, a big inflection point occurred with the publication of the LLM explainer I wrote with Timothy Lee. But since then, I’ve been pleased to see continual growth throughout 2024. I hope that means people are enjoying the kind of stuff I write here.

A new system

This was also the first year I tried out a new system: about once a month, paying subscribers could vote on a topic of their choice, which I’d then write a post about. Sometimes these posts were novel empirical studies, and sometimes they were deep dives or explainers on a complex topic like mechanistic interpretability. I like this model because it gives people real control over the direction of the newsletter and in some cases actual scientific research.

Overall, I’m really happy with how this went! It was illuminating to see what topics voters were most interested in reading about, and the process also served as a useful motivation for me to actually work on posts that I might otherwise procrastinate on. Explainers, for example, take a fair bit of time to write—I’ve got to read enough about the topic to make sure I understand the basics, then write about it in a way that hopefully conveys key concepts without being overly technical. I’ve been grateful for this motivation every time, as I always learn something along the way.

In at least one case, this has also been useful for my own research. My posts on measuring and modifying readability, for example, were the seeds for a more formal paper I published with my wife at this year’s workshop on text simplification and readability at EMNLP.

In other cases, these posts are useful artifacts I can share with students or interested researchers, such as those interested in LLMology or learning about vision-language models (VLMs). In the spring quarter, I’ll be teaching my class on LLMs and Cognitive Science again, and I plan to incorporate some of these posts as additional materials or resources for students.

I’m looking forward to continuing this system in the new year, and I plan to do so insofar as it remains sustainable along with my other obligations. I have a couple of posts I owe subscribers currently in the pipeline already—a “sequel” to last month’s VLM explainer, along with an empirical study on synthetic text detection—and I also have some ideas for topics to vote on in future polls. If any readers have thoughts about how this process has worked (good or bad), please feel free to reach out.

Assorted announcements

I also have a few other general announcements.

I don’t talk much about my personal life here, but as I’ve alluded to previously on this newsletter, my wife and I welcomed our daughter into the world this May—that was (and continues to be) the highlight of the year and indeed our lives thus far. It’s possible I’ll write more about it, though I’m not sure I’m skilled enough at writing to convey my experience in a way that people would find interesting to read. More likely, I’ll draw on the experience as inspiration to revisit some of the child language acquisition literature and its relevance to language model research.

Another, much less pleasant personal detail is that I’ve been suffering from pretty severe lower back and leg pain (sciatica) over the past month and a half. I hope it will resolve in 2025, but an ancillary benefit of sorts is that I’ve now learned more than I ever cared to about spinal anatomy and the research on back pain—it’s an interesting topic, in my view, and quite relevant to lots of people given the prevalence of sciatic pain, so it’s likely I’ll write a review on this as well.

I’m also in the midst of several exciting research projects, including one involving “developmental” mechanistic interpretability: tracing the evolution of LLM representations over the course of pre-training. I hope to wrap up the first stages soon, and you can expect a write-up here.

Finally, I’ve been meaning for quite some time to write up a few book reviews. I’m currently writing one on Richard Wrangham’s “Catching Fire”, but my intention is to also review a few related books, including Brillat-Savarin’s “Physiology of Taste”, as well as unrelated ones, such as William James’s “Varieties of Religious Experience”. I’m debating hosting some of the more esoteric reviews on a separate newsletter for the same of thematic consistency, but I haven’t made up my mind yet. Either way, readers can expect to see some of my thoughts on culinary evolution and disenchantment (unrelated topics) in the following year.

As always, thank you for reading!