The Counterfactual's year in review, and an announcement

Trying out a new model for community participation in empirical research.

2023 has been a big year for the Counterfactual. I wanted to write this year in review post to summarize some of the major things that have happened, highlight specific posts, and make an announcement about what’s to come.

Thanks to all the readers!

First, I’m very grateful to everyone who reads this newsletter. I’ve been blogging for years but never really made much effort to circulate what I wrote widely. When I started The Counterfactual about a year and a half ago, one of my hopes was that I would reach a wider readership—though to be honest, I was skeptical I’d reach more than a handful of people.

As you can see in the figure below, the subscriber base grew slowly but steadily over that first year. Then, in late July of this year, I co-authored an explainer piece on LLMs with Timothy Lee of Understanding AI. In a matter of days, the number of subscribers more than quadrupled, and then it just kept growing.

An example of very non-linear growth.

I continue to be both very surprised and very grateful for this readership. I’m also thankful to Timothy Lee for inviting me to be a co-author on that piece in the first place (and if you’re not already a subscriber, you should check out his newsletter here).

Some key posts

When I started the Substack, my intent was to write posts about various topics at the intersection of Cognitive Science, statistics, and Artificial Intelligence. I think I’ve mostly adhered to that, but the last year has seen a particularly sharp focus on Large Language Models (LLMs).

With that in mind, here are some of my favorite posts from the last year.

1. The LLMs explainer piece was obviously the most successful in terms of driving subscriptions. I’m also very proud that it’s ended up on at least a couple course syllabi. My day job is a college professor, so I care a lot about education—one of my goals with this newsletter is to share some of that educational content more widely.

2. My piece on construct validity has many fewer views, but it’s one of my favorites because, as I mention in the piece, the concept of construct validity is just so important when it comes to debates about what LLMs can and can’t do. I find myself referencing it frequently in other posts (like in my more recent post on using human tests to evaluate LLMs), so I’m glad I wrote it.

3. I’m also happy with the post I co-wrote with Cameron Jones on whether ChatGPT has a Theory of Mind. Because my day-to-day work involves a lot of empirical research on LLMs, I try to bring that perspective to the articles here—i.e., diving into the weeds of what a research paper (or set of papers) did.

What to expect in 2024

First off, you can expect more of the same: I’m aiming for approximately 2 posts a month, covering a mix of recent empirical work on LLMs (like the Theory of Mind piece) and theoretical arguments (like the one on using human tests to evaluate LLMs).

But I’m also going to try something new, which brings me to my announcement: I’m going to turn on paid subscriptions ($5 per month, or $50 per year). The typical ~2 posts a month are still free and always will be—my goal is not to paywall that content. I’ve thought a lot about what I want paid subscriptions to be “for”—what are you getting for your money—and I’ve reached a tentative (and hopefully exciting) conclusion.

Here’s the deal: at minimum, paying subscribers will be able to vote, at least once per month, on which empirical research project I should tackle next and write about here in the Counterfactual. You can think of this as a version of citizen science: I’ll provide a set of tangible research ideas (“proposals”), and you’ll vote on which one you’d like to see research on. Alternatively, you could construe paying subscribers as a kind of distributed funding agency: it does cost money to run some of the studies I talk about here, and I’m interested in exploring alternative sources of funding (i.e., besides traditional granting agencies)—particularly for research that’s adjacent to but not exactly in the “academic” sphere.

These will mostly be empirical projects. Basically, there are a bunch of things I’d like to do that aren’t explicitly rewarded in the academic system, i.e., they’re interesting but hard to publish in an academic journal. Examples of empirical projects might include: asking LLMs to play different games (like the board game “Taboo”); evaluating the ability of an LLM to create psycholinguistic stimuli (like I propose in my science factory post); and various other small empirical projects (mostly using LLMs).

Occasionally, I might also include the option to vote for a review of empirical research on some topic (e.g., what do we know about how LLMs learn grammar) or an explainer piece of some sort (e.g., different probing techniques).

The results of these monthly votes will decide which research project I prioritize and write about in the following month. Note that the posts themselves will continue to be public—especially the results of any empirical work, as I generally don’t like hiding that stuff behind a paywall. I may occasionally include an extra post for paying subscribers only, but only if I’ve already reached 2 free posts that month.

I plan to turn on paying subscriptions early next week—and soon after I’ll post the first poll of 2024. Thanks again to everyone who read the Counterfactual in 2023!