How to know your AI actually works

Every team building with LLMs hits the same wall. It feels like it's working, until it isn't, and no one can tell you why. For a CIO or CTO, that gap has a price. One team's code got stuck in a continuous loop while still in QA, quietly burning money by the day before anyone noticed. No engineer flagged it. The company-wide AI tracing system caught it, running quietly underneath everything.

Martin Serrano has been working that wall for three years. As the principal engineer who's run evals at HubSpot since before "evals" was a category, he shipped his own team's natural-language report generator, then built the second version of a company-wide framework, with one aborted attempt in between. He's built the version that gets used and broken a few that didn't. He spoke with A.Team's AI Guild to walk through what actually holds up in production.

His framing: stop treating this as one problem. It's three questions, each needing a different tool. What happened, what's happening, and what will happen.

The mental model: What happened, what's happening, what will happen

What happened is pure observability. You can't answer it without seeing exactly what went into the model, what came out, and every tool call and skill load in between. What's happening is the health of your system right now, sampled from live traffic. What will happen is a bet: if you change a prompt, a model, or an agent step, does the outcome get better or worse, and can you prove it before you ship?

Underneath all three is the idea most teams get wrong. An AI feature is a system, not a model. It's the code loading configuration, the prompt, how user input gets assembled, and only then the model call. Bugs hide in any of those steps. Test the model in isolation and you'll miss most of what actually breaks in production.

zoom_in

What happened: Trace everything, through a layer of your own

HubSpot standardized on OpenTelemetry's Gen AI semantic conventions, and Martin's advice to anyone starting fresh is to do the same. It's an interchangeable format, so you're not locked into one observability vendor's opinion of what a trace should look like. For a CTO, that's the difference between a data asset you own and one you rent.

The architectural decision underneath it is stricter than most teams expect. Nobody at HubSpot calls the OpenAI or Anthropic clients directly. Every call goes through an internal wrapper that implements the same interface but captures every detail, streaming and non-streaming, as it happens. That keeps them independent of any one framework's instrumentation (LangChain wraps its own clients, so does Braintrust) and buys fine-grained token accounting: what was consumed, when, and for which problem. The one gap is Gemini, which ships sealed classes instead of interfaces, so you build a facade in front of it.

This isn't a production-only concern. A modified span exporter routes traces from a developer's machine into the same system, so pre-prod bugs surface in the same place as everything else.

Feedback is data, even below a 1% response rate

Explicit feedback is the thumbs up and thumbs down you already ship. Martin's point: you don't need much volume to make it useful. Even a response rate under 1% gives you signal. Implicit feedback counts too. A generated result that gets edited and saved, or reused later, is telling you something without anyone clicking a button.

The cheapest place to start is a pattern any team can copy. One of HubSpot's internal agents collects a steady stream of thumbs up and thumbs down. Every thumbs-up interaction becomes a positive test case automatically. Nobody has to agree on what the "correct" answer was, they just accept what already happened as good enough. Run that same set again after a change, and if the pass rate drops, you likely broke something that mattered, with no hand-labeled golden answer required.

HubSpot doesn't fine-tune on any of this. Martin's read is that base models are good enough now that fine-tuning isn't worth the engineering time, and he doesn't expect that to change. Positive feedback scales because noise averages out at volume. Negative feedback is harder, and there's no clean automated pipeline for it. What works is human review, specifically getting a negative example in front of a product manager. Generative AI changed which problems get solved and who solves them. At HubSpot's scale of hundreds of artifacts, features, and teams, no engineer is an expert on sales processes, marketing workflows, and everything else customers ask for. PMs and individual TLs are the subject-matter experts on what the feature should have done. Show them the failure, ask what should have happened, and that answer becomes the next test case.

What's happening: Judge live traffic, one aspect at a time

Sampling live traffic matters because drift happens outside your control. Martin's example: a lower-level schema-generation library, shared across several teams, changed in a way that was still technically valid but made the JSON schemas sent to OpenAI less clean. OpenAI has historically been finicky about schema shape for structured outputs. Nothing failed outright. An online judge scoring live traffic surfaced that an unchanged configuration had quietly started performing worse. No alert would have fired, because nothing was broken. It was just worse.

Three rules held up in practice for building the judges. Keep each judge scoped to one aspect of the output, because a single 1-to-10 "how good is this" score collapses toward the middle and stops discriminating. Keep sample sizes small enough that you can afford a top-tier model as the judge (Martin's example was Opus-class). And calibrate every judge against a hand-scored golden set before trusting it. Skip that last step and you've traded "shipping without evals" for "judging without evals," one level removed. HubSpot also uses dynamic judging: each test case carries its own sentence describing what that judge should check for, fed in as context alongside the output.

What will happen: Every change is a bet

A prompt edit, a model swap, a new agent step. Each one is a bet, and an online judge can't grade a bet you haven't shipped. That's what a fixed, offline eval set is for. Score every case against an ideal answer instead of pass/fail, so you can track aspirational or beta capabilities you're not ready to put in front of every customer.

Wherever the output is structured, score it deterministically first. For some of Martin's evals, the code walks two JSON trees, the actual output and the ideal one, in a hierarchical comparison, rolling accuracy up the tree while keeping a record of exactly where it diverged. Only the generated-text leaves get a judge, or something like cosine similarity. Everything normalizes to a 0-to-1 range, and the score you act on is the mean. Martin is upfront that this is simplistic from a machine-learning standpoint, but a single unitless number is something a non-engineer can reason about, and it answers exactly one question: is this better or not. The catch everyone gets wrong at first is that a 0.8 is not "80% correct." It means nothing on its own. It only means something next to another run's score.

zoom_in

That framing is what makes cost decisions defensible. The standard sequence: solve the problem with the strongest model you can get, to prove it's solvable at all, then start substituting cheaper models, less context, different configurations, while watching the mean score. Losing five points of score to save a million dollars a year is a trade Martin's team will actually make, but only because they can see the five points.

Over time the eval set becomes the product spec. Every bug, failure, or incident that gets reported turns into a new test case, and the set grows as production teaches you what it has to handle. The discipline that's easy to skip and shouldn't be: weight test cases by how often you actually see them in the real world, so one exotic edge case doesn't carry the same vote as your most common path.

The data model, and where it's headed

Underneath all three stages is one simple chain. A trace (spans, tokens, cost, timing) can become a test case. Feedback tags which test cases are good or bad. A group of test cases becomes a dataset. A dataset plus a configuration is an evaluation, the thing you compare against another evaluation. HubSpot keeps these dynamic and ephemeral rather than fixed: take a chunk of cases, tag a subset as a dataset for this comparison, run it, done.

zoom_in

Where this is heading matters. Give an agent read, write, and execute access to that model and it can query traces, compare one run's trends against a prior one, generate a batch of new test cases on request, and run something close to an automated hill-climb on a prompt or tool description. The more of this that gets formally modeled, instead of handed to the agent as a raw JSON trace dump, the cheaper each run gets, which is what makes doing more of them possible.

Build or buy, and who actually runs it

HubSpot built in-house, and Martin is candid about why that's viable for them: a large number of HubSpot's engineers sit in infrastructure or developer experience. That's the precondition most companies don't have. He looked hard at buying. Braintrust is SDK-first and a good fit if you're comfortable with your data living in someone else's cloud (they do have some options for running the data plane within your own VPC). Langfuse was the other serious look. His caution for anyone evaluating tools: watch for platforms that bake too much opinion into the workflow. At this stage you want composable primitives, and a platform that hard-codes how evals should work takes that control away.

A few more patterns from the fireside worth carrying into your own program:

Turn-level and conversation-level evals, both. HubSpot's agentic flows can run dozens of turns across tools and skills, sometimes resuming hours later. They link every turn into one conversation so they can evaluate at either level. Single turns are easier to build and reason about. Full-conversation runs can use a simulated "conversational agent" persona (say, an inexperienced marketer trying to send a newsletter) to replay a realistic scenario end to end and check whether the outcome actually lands.

Tool calls get deterministic checks, not vibes. Every tool call, its arguments, and every skill load gets tracked. That surfaces a tool firing without its skill loaded, a sign the skill is misconfigured or unnecessary, and lets them check whether arguments were correct.

Reasoning as part of the solution, not a separate grading target. In some of HubSpot's agents, a second-stage judge reads the first stage's reasoning trace to sanity-check the decision, "like a colleague double-checking someone else's output." They capture every reasoning delta in the trace, but grading chain-of-thought as an end in itself isn't common practice yet, at least as far as Martin has seen.

Write the eval as you write the feature, not after. The moment you catch yourself manually re-running something to check it still works, that's the test case. Capture it right there. A test case doesn't need infrastructure behind it, it can be a single sentence of intent scored by a judge.

PII lives in traces, not in test cases. Raw traces keep real content, PII included, access-restricted to the team debugging that case. Promote a trace into a shared test case and it gets scrubbed first, itself a quick job for an agent. The most sensitive fields, billing and payment data, should be encrypted so they're invisible even to engineers debugging the system.

The real bottleneck isn't tooling

Martin didn't start with a three-stage mental model or a company-wide framework. He started because he refused to ship code he hadn't tested, and evals turned out to be that same discipline applied to prompts and agents instead of functions. The OTel conventions, the dynamic judges, the unitless scoring all got layered on top of that one refusal, over three years, with a false start along the way.

That's the part worth sitting with if you own an AI roadmap. Teams that skip evals aren't behind on tooling. They're behind on the discipline, and on having the right people close enough to the problem to own quality. The question Martin gets asked most is whether to build or buy the full eval data model. His answer: that matters far less than whether the habit and the eval mindset exist at all.

If you're running evals at your own company and want to compare notes, that conversation is happening in A.Team's AI Guild, where A.Team's senior AI practitioners show each other what's actually running in production.

Want engineers who build this discipline into your production AI? Hire senior AI talent from A.Team →