Falsifications are important in observational research

Falsifications are important in observational research

scott cunningham

Nov 3

The challenges of observational data are obvious. The treatment was not instituted in a controlled environment and therefore confounders lurk everywhere. We build as a result these designs around their existence, but we really aren’t sure that they’ve all been neutralized. They’re like ghosts haunting our studies, and sometimes we cannot get rid of them so much as we can hope.

The falsification, as I’ve said before on here and also my new book, is unique in observational research because it is not a design to eliminate a confounder. Methods designed to eliminate a confounder, such as instrumental variables or difference-in-differences, are of course a vital part of research. Falsifications by contrast don’t “do” anything. They are not used for identification at all. They are simply verifications that what you’re finding isn’t caused by a confounder, and they’re a bit scary because when you do them, you open yourself to all kinds of cognitive biases including rationalizing their existence as meaning something other than what you thought they meant before you did them.

A friend of mine has advocated that we should be pre-registering falsifications. It’s just too easy to skip them altogether, or simply endogenously cherry pick which one you’ll report.

But you know what really convinced me again this morning of its value? Reading a paper that uses a falsification and made the entire paper about that. This new paper by Sarah Bana (now at LSE — congratulations Sarah!) and coauthors looked at a common methodology and finding — looking at post ChatGPT exposure to AI exposed sectors as the treatment and some measure of employment as the outcome — but earlier before ChatGPT was released. Here’s the abstract.

I think one of the things that makes people worried about falsifications is that they take so long to do. By the time you do them, your study may be nearly finished. I mean when you think about it, it does seem like the falsification comes last. Way after you already have promising results. The sunk cost fallacy is very prominent at that point because you’ve sunk tons and tons of time into your main results by then, and your design, that to do something that might make it seem like a house of cards is scary, understandably.

But what if we were to do them first, not last? What if falsifications were a design stage not a robustness stage? We often present robustness stuff as things we do to perturb our results, relaxing this or that assumption, changes specs this way or that way. But true falsifications aren’t really that at all — they are completely consistent with no looking at treated outcomes, which makes them design activities. If it is possible to do them without ever having run your main causally identified model, then they are things you could do without ever having peeked at your outcomes.

Falsifications aren’t where you know there is no treatment therefore there should be no treatment effects. They do not get “used” for identification, they do not help you with power, they are not useful for estimation whatsoever. They simply tell you whether you should proceed that intended way or not.

In my checklists I emphasize to people in this AI agent day and age, I do put falsifications after main estimation, but I am wondering if they always be considered a design stage and therefore done before. Doesn’t mean you order the paper that way, but it does mean wait to do your main stuff until you’ve convinced yourself the design is solid and falsifications are one of the best ways to do that. But you should do them without having seen the results — which is again one of the frustrating things about diff-in-diff. We tend to produce event studies by violating that principle in that we produce event study plots for post and pre treatment. What I like about the new robust diff in diff estimators these days is that you can in principle not do that since each pre treatment coefficient is technically its own separate regression (“2x2”) that you could do and report and get confidence intervals without having ever touched a treated outcome. But I think that’s rare to do. Most simply do them all at once and once you see those treated outcomes processed by your estimator, you’re opening yourself up to cognitive biases.

But you know what? With agents, you totally could do that. And you know what else about agents? You also get to that state faster. And i Think one of the better things about using agents for research is that youre less likely to be pot committed to some design or approach if you can get the agent to get there asap.

Today is my birthday, n a different note. 51yo. I’m in Boston this week presenting at Tufts and APPAM conference. The fall in New England couldn’t be more beautiful. The leaves are changing colors, and the weather is just gorgeous. I entered the second third of my life last year when I turned 50, and now I’m 1/50th of the way through the second act. Looking forward to what awaits. The last year was filled with newfound joys and experiences, life changing ones. We buried my father on November 1 a year ago, which is also interestingly the day of the Dead. So it was almost like the end of one chapter of my life. Mom said she often feels like he comes to visit her now. She is doing well. She has many friends and stays busy. She cooks for her church, and she still paints. She sold two big paintings this last month.

So best of luck to you. We never know where we are on our journeys when we are there. Try not to let the intrusive thoughts swamp you.

© 2026 scott cunningham
n/a