I build AI agents, and the small models that check their work.

If you run an AI agent in production, you want something watching it: did it follow the refund policy, did it call the right tool, does its reply actually match what the tool returned? The usual answer is to ask a big LLM to grade it. That works, but it's slow and expensive for what are mostly yes-or-no questions, and it gives you no confidence score you can act on. Someone raised exactly this in AWS Strands Agents #4585, and I wanted to see how small the judge could get.

What I built

TinyJudge is ModernBERT-base, about 150M parameters, fine-tuned to answer three questions about an agent's turn: was the policy followed, was the right tool chosen, and is the reply faithful to the tool results. The training data comes from rule-labelled agent trajectories plus LLM-written replies, and the dataset is versioned with a SHA-256 manifest so a run can always be traced back to the exact data it saw.

TinyJudge 150M

88.0%

LLM judge 27B

81.2%

0.20 → 0.10calibration error (ECE) on the grounding task after Platt scaling

74% → 84%grounding accuracy after calibration

89.9% · 35%cascade accuracy, sending only 35% of cases to the LLM

Making the confidence mean something

A judge that says 0.9 should be right about 90% of the time. Out of the box it wasn't, so I fitted Platt calibration per task on a separate half of the real data. Calibration error on the grounding task halved, and once the scores were honest I could build a cascade: TinyJudge handles the cases it's sure about and only hands the uncertain ones to the big LLM. That got to 89.9% while sending about a third of the traffic upstream.

Shipping it

Runs are tracked in MLflow and models are versioned on the Hugging Face Hub. The FastAPI service exposes Prometheus metrics and simple drift signals. In CI, GitHub Actions builds the Docker image, smoke-tests it, benchmarks it, and refuses to publish if accuracy drops below 85%. That gate has caught me more than once.

Haan is Hindi for "yes", the little word you say on a call to show you're still listening. Voice agents need the opposite skill too: knowing when you've actually finished. They decide that with end-of-turn models, and the open ones are trained on clean 16 kHz audio. Phone calls are 8 kHz and squashed into a narrow band, so the model hears a pause, guesses you're done, and talks over you. People asked about it in pipecat-ai/smart-turn #42, so I measured how bad it was and tried to fix it.

studio-trained model cuts in at the pause phone-tuned model waits for the real end

What I did

I started from Smart Turn v3.2. Only the ONNX export was released, so I recovered PyTorch weights from it, exact to 6e-8, so I could fine-tune. Then I trained with telephony augmentation (8 kHz resampling, μ-law, the 300 to 3400 Hz band) against a control trained identically without it.

Everything is reported with paired bootstrap 95% confidence intervals and per-language tables, because a single average across 23 languages hides a lot.

The honest bit

I also put haan through LiveKit's public turn-detection benchmark in eot-bench #18. On their clean benchmark audio it ties the base model. Their maintainer reproduced my numbers and confirmed the phone-line gains hold, but on clean audio no metric moved beyond noise, so the PR wasn't merged. That's the right call: the fix only matters on degraded audio, and the benchmark doesn't have real phone calls yet. I learned more from that review than from my own results table.

Agents re-derive the same procedure in every conversation and skip steps along the way: they issue a refund before checking the order, or forget to confirm the email. Google ADK #5398 asked for a way to learn those procedures from what the agent has already done. SkillMiner is my working answer.

How it works

An ADK plugin records traces. To have something to mine without real customers, I wrote an LLM customer simulator with personas and hidden labels, so I always know the true intent behind each conversation. The intent miner groups conversations with an LLM and reached purity 1.00 and ARI 0.93, against 0.48 for a TF-IDF clustering baseline. The repeated workflows are written out as Agent Skills (SKILL.md files) that ADK loads directly.

Before a skill ships, a paired A/B validator runs the agent with and without it and compares them with an exact McNemar test. There's also PII redaction on the traces and a drift check for when tools change underneath a skill.

In a pilot A/B test with 8 pairs, procedure compliance went from 75% to 100% and policy violations from 1 to 0. Eight pairs is too few to be statistically significant, and I say so in the README.