I stress-tested Laya, an open decision model, on pharma surveillance

I stress-tested Laya, an open decision model, on pharma surveillance

One prompt tweak took my detector from 0/38 to 30/38, and 20 minutes of fine-tuning turned a demo into a triage system

Eric J. Ma

Oct 22

Hello fellow datanistas,

Ever shipped a model that was confidently, catastrophically wrong, and only found out after it was routing real work?

That's the fear that keeps us benching good models instead of deploying them. So when an open decision model called Laya landed (the Apache 2.0 answer to a closed competitor), I spent a week of evenings trying to break it on a pharma surveillance workload: email triage, multilingual adverse event detection, document classification, fine-tuning, calibration, and a serverless deployment.

Two things happened that I think you'll want to see.

First, the failure. On 50 social posts in 13 languages, my adverse event detector caught exactly zero of 38 real events. The model was fine, my question was broken. One negatively-phrased exclusion clause had inverted its yes/no judgment. I cut the question down to a short, positively-phrased sentence, and detection jumped to 30 of 38 with zero false alarms. Your prompts are code; tiny edits flip outcomes.

Second, the win. After just 20 minutes of fine-tuning on 212 labeled documents on my laptop, held-out accuracy jumped from 0.04 to 0.89-0.96 across every question, and the probabilities became ones I'd actually trust (a P=0.9 threshold now means 90% precision, not vibes).

There were also cautionary tales. The English checkpoint gave a machine-translated Khmer email 0.952 confidence at 0.000 accuracy. It didn't even know it was lost. And in every failure I found, the model was just as confident when wrong as when right. That's why the post ends with a simple rule: use it as a fast first pass, keep a human as the second.

I also ran the deployment math: a serverless CPU profile that scales to zero costs pennies for sporadic traffic, while a GPU cuts inference to 49 milliseconds when latency matters.

Everything ships as seven reproducible marimo notebooks, so you can rerun every figure yourself.

The full story, what worked, what broke, and the exact numbers, is in this post.

Happy stress-testing,
Eric

P.S. This whole week came from walking into a field I don't own, pharmacovigilance, and using the model as a sparring partner, which is exactly the skill I'd want any of us to practice. Daniel Chen and I are organizing a retreat on how to learn anything with AI, February 2027. Applications open to the waitlist first: https://learn-anything.nonlinearlabs.ai/

If you liked this post from Data Science Programming Newsletter, share it around!

© 2026 Eric J. Ma
548 Market Street PMB 72296, San Francisco, CA 94104