Agent Wars, Episode 3: Revenge of the Traces

With contributions from Richard Yan (richard_yan@tech.gov.sg), Raymond Low, and Joel Foo (joel_foo@tech.gov.sg).

tl;dr: We've spent the last couple of year(s) making agents smarter, more capable and better informed. What most teams still can't do is say, after the fact, what their agent actually did. Observability is the layer that makes every other improvement (evals, guardrails, cost control, governance) possible, and so it belongs in the platform by default, not in each team's backlog. OpenTelemetry is the vehicle for it, not the destination. And telemetry without context is just data: you need shared semantic conventions to make it legible, and a risk-based framework (like our ORF playbook) to decide what is worth collecting in the first place.

Near the end of Revenge of the Sith, Obi-Wan and Yoda go back into the ruined Jedi Temple. Yoda warns him off: "If into the security recordings you go, only pain will you find." Obi-Wan goes anyway, and the hologram shows him what nobody wanted to believe about Anakin. Up to that point every Jedi had a theory about what was happening. The recording showed what actually happened.

At least the Temple had recordings. A lot of agents in production today don't. The Jedi Order had many problems, but logging wasn't one of them.

Think about what you'd actually have if one of your agents did something strange last Tuesday. Most teams would have an application log with a few lines in it, a bill from the model provider, and (maybe) a user who remembers it differently from how the agent would describe it. You'd know the agent was called. You probably wouldn't know which tools it chose, what it retrieved, which version of the prompt it was running, how many times it retried, or why it stopped where it did.

Logic, harness, brain, and then what?

If you've been following this series, here's where we are so far.

We built agents with logic. The ReAct agent loop and the graph-shaped versions of it you see in frameworks like LangGraph gave us a way to turn a model that answers questions into something that works through a task step by step.

We then made them capable with harnesses. That was Episode 1: the tools, the permissions, the sandboxes and the plumbing that let an agent actually do work in a real environment rather than describe the work it would do.

We made them better with brains and context. That was Episode 2: the memory and context that let an agent start from what the organisation already knows instead of from a blank whiteboard.

Each of those layers made agents more useful, but they also made agents harder to reason about. A loop that decides its own next step is less predictable than a function. A harness with thirty tools has thirty more ways to go wrong than one with none. A shared brain means the agent's behaviour could change based on what someone else wrote into that shared memory without you knowing. Which brings us to this episode: now that the agent can do all this, how do we know it's doing what we want it to do? It's not adding "make no mistakes" to the prompt.

Why don't we have observability?

To be fair, there are good reasons people skip an observability platform as an upfront choice.

When you're building an agent, the feedback loop is your terminal. You run it, you read the output, you tweak the prompt, you run it again. You can see everything because you're sitting right next to it. Instrumentation feels like overhead at that stage, like something you'll get to once the thing works. Most frameworks print a reasonable amount to the console already. Your cloud provider has logs. If you use a hosted model, the provider dashboard tells you how many tokens you burned. And there's a whole category of LLM observability vendors happy to show you a pretty trace view if you install their SDK. So you might tell yourself: we'll add proper observability later, when we have users.

The problem is that "later" tends to arrive as an incident, and by then the evidence you need was never captured. This is like how we used to say - "it worked on my laptop".

In a normal service, the code is the behaviour. If you want to know what happened on a given request, you can read the code path, look at the inputs, and reconstruct it with reasonable confidence; logs and metrics are there to confirm and to alert. In an agent, the code is mostly a loop. The behaviour (which tool, in what order, with what arguments, stopping when) is decided at runtime by a model, and it changes with the prompt, the model version, the retrieved context, the memory contents, and the phase of the moon. Reading the code tells you what the agent could do. Only the trace tells you what it did.

So the trace is the only real record of what ran. And a lot of things depend on it:

• Evals need real traces. The most useful evaluation datasets come from production behaviour, and you can't sample what you didn't record.
• Guardrails need a record of their own decisions. A guardrail that blocks something and leaves no trace is Schrödinger's guardrail.
• Cost control needs token usage attributed to the agent, the task and the step that spent it, not a single number on a monthly invoice.
• Debugging needs the sequence of decisions, not just the final output.
• Governance and audit need to be able to answer "who (or what) did this, on whose behalf, and why" months later.

Everyone wants to build these. Very few of them can be built properly without the trace underneath. That's why I think observability is the critical part of agentic infrastructure that most people miss: the rest of the list sits on top of it.

What central observability has done for us

Harness (GovTech's platform for agents in government; yes, we can debate on naming) provides central observability out of the box for agencies. It's OpenTelemetry-native: a team drops in one SDK, points it at Harness, and their agent runs show up as traces (what it did, what it called, how long each step took, how many tokens it used). Nobody has to procure, deploy or operate their own trace store to get started. And agents built on Harness itself don't even need the SDK: they're traced from the moment they're deployed.

We built this up front, and a lot of things ended up depending on that decision.

The first and most obvious one was debugging. Central observability is what let us debug a lot of what went wrong in Desk (our AI assistant for public officers), and, more importantly, improve it. "Why did it do that?" is a question for the trace. The alternative is reproducing it locally and hoping it misbehaves the same way twice. Improving it gets easier for the same reason. It's much easier to argue for a prompt change, a tool description rewrite or a different retrieval setting when you can show the before and after on real runs, rather than on the handful of examples you happened to try.

The second one was Aperture, which derives insights on AI usage and adoption. Aperture uses more than traces: it also pulls in usage data from other sources, and what's available varies a lot from one source to the next. What made it work was the same decision underneath. Every source, our traces included, lands its data (redacted) in one central place in an agreed shape, and Aperture builds on that. So a question like "who is actually using this, for what, and is it sticking?" is answered from data that already exists, rather than from a new data collection exercise (and a new set of forms for every agency to fill in).

The third one is Metis, our agentic evaluation platform built in collaboration with our Responsible AI team, which is still a work in progress. Metis relies on OpenTelemetry to work. Evaluating an agent properly means evaluating the trajectory (did it pick sensible tools, did it loop, did it give up too early, did it recover from a failed call), not just the final answer, and the trajectory only exists if something recorded it in a shape an evaluator can read. Without consistent traces there is nothing for Metis to evaluate.

None of those three were the reason we built central observability (it's a story for another time). They're things that became possible, or much cheaper, because the observability was already there. I think that's the general pattern, and also why we should be doing it early.

Aperture is also a good reminder that central observability, at least the way we've ended up doing it, isn't just OTel and a trace viewer. Traces are one input into a wider pool of data that lands centrally. This piece is about the traces, because for agents that's the part most teams are missing, but the bigger decision was agreeing that everything lands in one place in a shape someone else can build on.

Why it has to be on by default

If observability is foundational, the next question is who's responsible for it. In my view the answer is the platform, and always-on.

Observability that each team has to opt into means observability that nobody is opting in to: the quick prototype that ended up in production, the team that was too busy shipping to wire up tracing, the agent that was "only internal". Asking every team to instrument their agents properly, choose a backend, operate it, secure it and keep it consistent with everyone else is asking for a lot of slightly different, partially complete answers. We made a similar argument in The Road under the Harness: the requirement is that traces are "queryable after the fact", not that every team builds a bespoke observability platform. Retrofitting traces onto systems that were never designed to emit them is slow and painful, and it's usually done under pressure.

So any good platform should have observability built in out of the box. When you deploy an agent onto it, it gets traced, and you don't have to do anything extra. The easiest path should also be the instrumented one.

How does a platform do that without becoming a bottleneck?

OpenTelemetry is a car, not a product

A common question about observability for agents is some version of "which tool should we use?". Should it be Langfuse, or one of the many LLM observability vendors, or the monitoring tool we already use for everything else (Grafana and the like), or the cloud provider's native offering? That does matter eventually, but I think it's the second question, not the first.

The first question is how your agents will emit telemetry, and I'd suggest using OpenTelemetry (OTel). OTel is a vendor-neutral standard and set of SDKs for producing traces, metrics and logs, and a collector for moving them around. It graduated from the CNCF this year, so it's about as mature and widely adopted as open source infrastructure gets.

I think it's also useful to compare it to Kubernetes. If you think about it, nobody's goal is "run Kubernetes". Kubernetes is how you get your workloads deployed, scaled, healed and moved between environments without rewriting them each time. It's a car, not a destination. People who treat it as the destination end up with a beautifully configured cluster and nothing useful running on it.

OTel is the same kind of thing. On its own it gives you no dashboard, no eval score, no guardrail, no audit report. What it gives you is a single, standard way to get telemetry from your agents to wherever it needs to go. The destinations are the products: a trace viewer for debugging, an evaluation platform, a guardrail service that wants to see decisions in near real time, a cost dashboard, an adoption analytics tool, a security monitoring pipeline, an audit store. For us, Harness and Metis read the traces directly, and the same traces also land in our central store, where they serve as an audit log and feed into Aperture alongside everything else.

Two things follow from treating OTel as the car.

First, you instrument once and keep your options open. If your agents emit standard OTel, switching backends or adding a new destination is a collector configuration change, not a re-instrumentation project across every team. That matters a lot here, because the tooling on the destination side changes quickly. The backend you pick this year might not be the one you want in two years.

Second, you can send the same telemetry to more than one place, with different treatment for each. The collector can fan out: operational metadata to the engineering trace store, sensitive content to a restricted store, security-relevant events to the security team's tooling. The agent doesn't need to know about any of that.

But a car only helps if everyone agrees on which side of the road to drive on. That's the job of semantic conventions.

Semantic conventions, or agreeing on what to call things

OTel on its own just carries spans and attributes. If one framework records token usage as llm.tokens.prompt, another as input_token_count, and a third as usage.in, your telemetry is standard on paper but not very useful, since every dashboard, eval and alert has to be rewritten for each framework, and anything that spans two frameworks (which is most real agent systems) breaks at the boundary.

Semantic conventions fix that by agreeing on names and meanings. Engineers are famously good at agreeing on names, so this was quick. OTel has a set of GenAI semantic conventions that define, among other things, what a model call, a tool execution and an agent invocation look like as spans, and which attributes they carry. A few that matter most for agents:

• gen_ai.operation.name says what kind of operation a span is. Well-known values include chat, embeddings, retrieval, execute_tool, invoke_agent, create_agent, invoke_workflow, plan, and a family of memory operations (search_memory, create_memory, update_memory, delete_memory). That last group is new and ties back to Episode 2: memory reads and writes are becoming first-class things to trace.
• gen_ai.agent.id, gen_ai.agent.name and gen_ai.agent.version identify which agent (and which version of it) did the work.
• gen_ai.conversation.id ties the spans of a session together.
• gen_ai.provider.name, gen_ai.request.model and gen_ai.response.model record who served the call and which model actually answered (which isn't always the one you asked for).
• gen_ai.usage.input_tokens and gen_ai.usage.output_tokens record what it cost.
• gen_ai.tool.name and gen_ai.tool.call.id identify tool calls, so you can tell a retry of the same logical call from a new one.

There's also an event for evaluation results (gen_ai.evaluation.result, with a name, a score and an explanation), which means an evaluator can write its verdict back onto the trace it judged, in the same vocabulary.

Put together, a single agent run starts to look like a tree that anyone (or any tool) can read (an illustrative example):

invoke_agent  policy-research-assistant v3     conversation=…  agent.version=3.2.0
├── chat           model=…  input_tokens=1,812  output_tokens=164
├── execute_tool   hansard.search   tool.call.id=call_01
├── retrieval      data_source=circulars-index
├── chat           model=…  input_tokens=4,377  output_tokens=622
└── execute_tool   gmail.draft      tool.call.id=call_02

You get the same shape whether the agent was built with one framework or another, and whether the trace ends up in one backend or several.

A couple of caveats. The GenAI conventions are still marked as in development, and they have recently moved out of the main semantic conventions repository into a dedicated GenAI repository. Names have changed before and will probably change again. The practical advice (which ORF also gives) is to pin the convention version you've adopted, record it in your telemetry design, and treat upgrades as deliberate changes rather than something that happens when someone bumps a dependency.

The conventions also cover MCP, which doesn't get talked about much. Plain HTTP trace propagation only covers the HTTP request, not the individual messages exchanged within an MCP session, so the conventions recommend carrying the trace context inside the MCP request itself (in params._meta). If your agents call tools over MCP (as agents on Harness do through its MCP registry), this is what stops the trace from going dark the moment the agent hands work to a tool server. The same logic applies to agents delegating to other agents: if the trace context doesn't travel with the task, you get two disconnected traces and no way to say which one caused the other.

The OTel Operator, or how "out of the box" actually happens

Conventions solve the vocabulary problem. They don't solve the adoption problem: someone still has to add the SDK, configure the exporter and point it at the right collector, for every service, in every team. That's the kind of thing that gets skipped.

If you run your workloads on Kubernetes, the OpenTelemetry Operator is the piece that turns observability from a request into a default. It does two jobs.

It manages collectors. You declare an OpenTelemetryCollector resource and the operator runs it in whichever shape makes sense (as a central deployment, a per-node daemon, or a sidecar next to each workload), with the platform team owning the pipeline configuration: where data goes, what gets redacted, what gets sampled.

It also injects auto-instrumentation. You declare an Instrumentation resource describing the exporter endpoint, propagators and sampling, and then annotate a workload, or an entire namespace, with something like:

instrumentation.opentelemetry.io/inject-python: "true"

From then on, pods in that namespace start with the OTel SDK and the relevant instrumentation libraries injected and pre-configured to send to the platform's collector. Java, Node.js, Python, .NET, Go and a couple of web servers are supported. The application team didn't have to write anything.

That's what "observability out of the box" means in practice. The platform decides once, centrally, how telemetry is produced and where it goes, and every team inherits it by deploying into the platform. Changing the backend, tightening redaction or adjusting sampling is a change in one place.

Observability without context is just data

Telemetry on its own is just data. Usually more of it than anyone can read. What turns it into evidence is context, and there are two kinds.

The first kind is structural. This is what the conventions and correlation give you: spans that mean the same thing everywhere, and trace context that's created at the entry point and carried through every model call, retrieval, tool, sub-agent and human review step. Without it you have a pile of spans that individually look fine and collectively tell you nothing, because you can't tell which ones belong to the same user's request.

The second kind is purpose. Which signals actually matter for this system, given what it does and what could go wrong? An internal drafting assistant whose every output is reviewed by a person has very different evidence needs from an agent that takes actions on external systems on behalf of the public. Collecting everything for both is expensive, noisy and (because telemetry can contain the same sensitive information as the system itself) a second data risk surface you now have to secure, classify, retain and delete properly. Collecting the same minimal set for both leaves the high-risk one unaccountable.

And some of the most important questions can't be answered by generic telemetry at all, however much of it you have. As we also highlighted in the ORF playbook: "A successful tool call and a successful user outcome are not the same thing", and "Do not interpret the presence of a human as proof that effective oversight occurred." A dashboard full of green tool calls says nothing about whether the user got what they needed. And if a human-in-the-loop step recorded nothing about what the reviewer saw or decided, you can't show that any oversight happened. You have to decide which questions you'll need to answer before you need to answer them, and no standard can do that for you.

That's what ORF was written for.

Getting started with ORF

The Observability Requirements Framework (ORF) is a risk-tiered set of logs, traces, metrics and review evidence for generative and agentic AI systems, and the playbook turns it into a practical implementation path for product, engineering, risk and operations teams.

It's about evidence, not about collecting everything. ORF defines 61 telemetry items across six operational categories, but the framework is explicit that it isn't a mandate to collect every possible field, and it isn't permission to store prompts or personal data without safeguards. Which items apply depends on the system's risk tier. For a low-risk pilot most of it is good to have; for a high-risk system, most of it is mandatory.

It's technology-agnostic, but OTel-friendly. ORF doesn't require you to buy any particular product. The playbook's integration guidance is to use the current OTel GenAI attributes where they fit, record the convention version you used, and add an organisation-specific namespace only for things the conventions don't cover yet (session and task identifiers, policy versions, data classification, human review decisions). That's the same "conventions first, extensions only where needed" approach described above, and it means ORF sits on top of the OTel investment rather than competing with it.

There's also a reference implementation if you'd rather start from working code than from a requirements catalogue.

Back to the Temple

Think about what the security recordings in Revenge of the Sith actually did. They didn't prevent anything; by the time Obi-Wan watched them, the damage was done. What they did was replace a lot of guessing with what actually happened. Before watching, Obi-Wan didn't want to believe Anakin was involved. After watching, he knew, and he and Yoda split up to go after Anakin and Sidious. So they used a record of what happened to decide what to do next.

I think that's where observability for agents is heading. Today it's mostly used after the fact, to debug and explain. Increasingly it's what evaluations are computed from, what guardrails act on, what cost and adoption decisions are made with, and what governance relies on when someone asks who did what. None of those work without the trace, and the trace is far easier to get right if the platform makes it the default from day one.

We've built agents that reason, act and remember. In my view, the next bit of work is making sure we can see what they did, in a format consistent enough to build on and scoped tightly enough to be safe. OTel and its conventions get us most of the way on the first part. Frameworks like ORF help with the second. The rest is just deciding to do it before you need to go looking for the recordings.