In this article, we are sharing GovWork-Bench, our latest internal benchmark for evaluating commercial and in-house AI assistants on Public Service work in Singapore. We will walk through how we built it: from constructing representative task baskets and designing task-specific metrics, to running browser-based evaluations and scaling scoring with LLM judges. Lastly, we will share some practical challenges and lessons we picked up along the way.

Why are we doing this?

As we enter the last quarter of 2026, the share of public servants who are actively engaging AI assistants in their work has grown steadily, according to data from our internal usage analytics platform. Since the beginning of this year, the Public Service has been rolling out several enterprise AI assistants, alongside newer, in-house ones like Desk for public servants to integrate into their daily work. Naturally, this leads us to answer an important question: How effective are the AI assistants in helping public servants accomplish public sector tasks in their day-to-day work? 

To answer that properly, we needed a benchmark, and specifically, one grounded in Public Service work.

A reasonable question to ask is, what value add is our benchmark providing. There already exist many benchmarks in the wild that assess reasoning capabilities like GPQA Diamond, MMMU Pro, ones that assess agentic capabilities like 𝜏² Bench, Humanity’s Last Exam or even composite indexes like the Artificial Analysis Intelligence Index. Following this line of logic, one might be tempted to question whether those benchmarks can already be a sufficient and good proxy for public sector work.

Our response to that is yes, but not really. These benchmarks are useful for gauging underlying model capabilities and, depending on the benchmark, particular agentic capabilities. Some of those capabilities assessments can certainly transfer to public sector tasks. However, public sector tasks also have their own workflow constraints and idiosyncrasies that can be integral to task success, but are not necessarily captured by generic capability benchmarks.

In particular, such tasks can involve nuances in the decision-making process that may seem unintuitive in a conventional capabilities benchmark. For example, a regular benchmark may place heavy emphasis on whether the final answer or outcome reached is correct.

Yet, we imagine that it would not work in a hypothetical use case involving citizen appeals, where the accept/reject decision itself may not always be the most salient measure of success – there can be room for discretion, and the recommendation may not even be the final deciding factor. 

What matters more in this hypothetical scenario is whether the AI assistant has correctly understood the case and can produce a “good enough” draft rationale that meaningfully supports the officer’s workflow, rather than trying to replace their judgement.

With this background framing in mind, we set out to create our own Public Sector Benchmark.

Scope 

Before we dive into the methodology, it is important to understand what we are actually benchmarking. GovWork-Bench targets the product layer, rather than evaluating the underlying foundation model in isolation.

We made this choice because, for most public servants (excluding software engineers and some data specialists) the product is the primary interface through which they experience an AI assistant. Their effective capability is therefore not just determined by whether GPT-X or Claude-Y sits underneath it. It also depends on the surrounding harness: file handling, code execution, connectors and how these components are orchestrated.

This distinction affects how we design the evaluation. Some task baskets can be completed using capabilities exposed natively by the product, while others require product-specific connectors or MCPs. Our unit of evaluation is therefore the AI assistant as experienced by the user, not simply its underlying model.

Methodology 

We will break the methodology section down into four main parts:

1. Task Basket Construction, where we cover what a “representative task basket” means and walk through an example of how we synthetically generated realistic datasets for Public Service work.
2. Mode of Evaluation, where we compare different evaluation modes and explain why we chose the approach we did.
3. Metrics & Scoring Design, where we cover why we created a metric hierarchy and how we use LLM judges to scale scoring in a reliable and auditable way.
4. Statistical Significance Testing, where we explain our statistical approach to improve robustness of the interpretation of our results

Task Basket Construction

An important question in constructing GovWork-Bench was what a “representative” Public Service task actually means.

Public Service work covers a very wide range of tasks, many of which also overlap with generic knowledge work such as summarisation, drafting and data analysis. With so many possible tasks to choose from, we needed a principled way of deciding what should make it into the benchmark.

Grounding task selection in observed usage

To make the selection more representative, we decided to ground it in observed usage patterns.

For this, we referenced Aperture, our internal usage analytics platform, which analyses all PII-removed conversations from all usage of the rolled out enterprise AI assistants across 94 Public Service agencies. One of its features classifies conversations into predefined Public Service task categories and ranks them by their observed frequency. 

Usage frequency alone was not enough. We also had to consider whether a task was practical to evaluate: could we construct a sufficiently realistic task, establish a meaningful ground truth, and score the resulting output reliably? This inevitably meant making trade-offs. GovWork-Bench is therefore not intended to cover every possible Public Service workflow; rather, it focuses on key use cases that are feasible to evaluate.

From this process, we arrived at five broad task baskets:

1. Knowledge Base Q&A
2. Case Management and Decision Support
3. Numerical Data Analysis
4. Document Summarisation
5. Policy and Briefing Work

Together, these baskets give us a structured way to sample different types of Public Service work.

Designing for different modes of usage

The task baskets were also designed to exercise different ways in which public officers might actually use an AI assistant, rather than assuming every task happens within the same harness setup.

Broadly, we wanted the benchmark to cover three modes:

1. Native product capabilities — tasks that rely only on the tools and harness already available within each assistant, such as file handling, code interpretation or specific agent skills.
2. Whole-of-Government (WOG) harness capabilities — tasks where assistants connect to shared government services or knowledge sources, such as Hansard or other internal systems.
3. Custom MCP servers — tasks where the user or agency provides a purpose-built MCP server exposing task-specific data or capabilities.

This distinction matters because an AI assistant’s real-world usefulness depends on more than the underlying model. Two assistants may have access to similarly capable models, but expose very different tools, connectors and integration paths around them. 

By designing tasks across these different modes, we can evaluate not just what the model knows, but how effectively the overall product can operate within the kinds of workflows users may actually encounter.

From task baskets to evaluable datasets

With the task baskets and modes of usage defined, the next step was to turn each broad workflow into a concrete, repeatable evaluation dataset with known ground truths.

For some baskets, suitable third-party datasets for benchmarking already existed and could be adapted to our evaluation setup. For example, we adapted the following third-party datasets: GovReport for document summarisation and SpreadsheetBench for numerical data analysis. For workflows that are more specific to the Singapore Public Service however, suitable datasets were harder to come by, which meant we had to construct our own. 

This is where things get a little more interesting.

Constructing the Citizen Appeals dataset

One of the more interesting datasets we constructed was Citizen Appeals, which evaluates an AI assistant under the Case Management and Decision Support task basket. Here, the AI assistant looks over the case details and compares it against official policies before presenting a recommendation and its rationale for officers to make the final decision.

Note that this is not a real use case that is deployed with AI assistants but rather an imagined scenario of what it could be like if we do decide to deploy AI assistants. We do not insinuate in any manner that this is a reality or that there exists a concrete plan to do so.

We started with real, PII-removed citizen appeal emails and the corresponding responses from agencies. However, directly replaying these examples would make the task relatively trivial: the original appeal and response already contain much of the information needed to infer the outcome. Instead, we used them as the basis for constructing a more realistic evaluation environment.

As shown above, the construction process has two main branches.

First, we use an LLM to extract and structure the case details from the original appeal and agency response. These are then stored as case records in a mock Case Database.

In parallel, an agent equipped with Web Search and SearchSG identifies the policy documents relevant to the case. These documents are ingested into a Policy Knowledge Base.

Together, the Case Database and Policy Knowledge Base form the information environment exposed to the AI assistant during evaluation. The assistant is then given the citizen's appeal and must retrieve the relevant case information and policy context before producing a recommendation and rationale.

The original agency response serves another important purpose: together with the citizen appeal, it is used to construct the ground truth, which the generated recommendation and rationale can be evaluated against.

Mode of Evaluation

Once we had the task baskets and their respective datasets, the next question was where and how to run the evaluations.

For our purposes, there were two main ways to run the evaluation: through the product's browser interface or through LLM Application Programming Interfaces (APIs).

Browser-based evaluation means interacting with the assistant through the same browser-based product interface available to users, automating user actions such as clicking and uploading files. API-based evaluation, by contrast, interacts programmatically with models and tools through exposed APIs.

Why we chose browser-based evaluation

For GovWork-Bench, we chose to run the evaluations primarily through the browser.

This follows naturally from the modes of usage described earlier: if we want to evaluate the product capabilities available to public officers, we should ideally interact with the assistants through the same interface they use.

One trade-off was realism versus comparability. We wanted to preserve how each product actually works, while keeping enough of the evaluation setup consistent to make comparisons meaningful.

Our approach was therefore to standardise the testing conditions, rather than the products themselves. Evaluations were run in each assistant's production browser interface, while controlling for which tools or MCPs were available for each task.

Automating the browser runs with Playwright

Of course, manually running hundreds of tasks through three different browser interfaces would not have been particularly fun.

We automated almost all browser interactions using Playwright. This let us scale evaluation while preserving the real product interface; for third-party datasets, we also disabled web search where appropriate to reduce benchmark contamination.

Metrics & Scoring Design

Once the tasks were defined, the next question was how to score them in a way that is both meaningful and interpretable.

Different task baskets have very different notions of success. For a spreadsheet task, there may be a relatively clear answer: did the assistant produce all the required cell values correctly? For summarisation or decision-support tasks, however, quality is inherently multi-dimensional. A response can be factually faithful but incomplete, concise but uninformative, or arrive at the expected decision with a weak rationale.

Rather than forcing every task into a single generic metric, we designed the metrics around the success criteria of each task.

A hierarchy of metrics

For each task basket, we also distinguish between:

• a headline metric, intended to capture the primary measure of task success; and
• supporting metrics, which provide additional context on quality, risk or failure modes.

For example, in document summarisation, the headline metric is claim coverage: whether the assistant captured the important information from the reference summary. Supporting metrics then capture other dimensions such as faithfulness, conciseness and information density. For SpreadsheetBench, by contrast, the headline metric is simply whether all required cell values are correct. 

The key principle is simple: we want to provide a concise view of the most important outcome, without throwing away the detail needed to understand what sits behind that score.

Deterministic where possible, LLM judges where necessary

The next design choice was how each metric should actually be scored.

Wherever there is a clear, machine-checkable ground truth, we use deterministic scoring. This works well for tasks such as spreadsheet cell value matching, exact-answer Q&A, decision accuracy, where the evaluation can be reduced to an explicit match or rule. 

But many of the dimensions we care about are not easily captured with exact matching and would require some degree of semantic judgement. We use LLM-as-a-judge for those metrics.

Making LLM judging more constrained via atomic claim-level evaluations

We did not want to simply ask an LLM to assign a broad quality score such as 1–5.

Instead, the judges evaluate against predefined, atomic claims. These claims were constructed by using a frontier LLM to decompose the in-prose golden answer into multiple, non-overlapping atomic propositions. These claims were then subsequently frozen. During evaluation, the judges are constrained to return a binary outcome. For example, supported or unsupported, together with a short reasoning note.

This makes the scoring more structured and easier to audit. It also reduces the amount of discretion given to the judge model, since it is deciding whether a specific claim is satisfied rather than making an open-ended judgement of overall quality.

To further reduce dependence on any single judge, we employ a panel of three LLMs from different families. In the current setup, the jury comprises Opus 5, GPT-5.6 Sol and Qwen 3.7 Max. 

For selected tasks, we also track inter-judge agreement. Where the judges disagree materially, the case is manually reviewed and reconciled. This gives us a targeted human-review layer without requiring every evaluation item to be checked manually. 

On Composite Indexes

While task-level metrics give us the most meaningful view of performance, we also wanted a single comparative measure that summarises performance across the benchmark. 

Because headline metrics operate on different scales and task types are not equal in importance, we developed a comparative composite index called Usage-Weighted Normalised Performance (UWNP), which normalises performance within each benchmark dataset before weighting it by observed usage frequency. 

The derivation of UWNP is provided below using illustrative figures.

Statistical Significance Testing

How we tested whether the observed differences were robust to sampling variation

After we obtained the results, the next consideration was whether the observed score differences between assistants were robust to sampling variation. Put simply, we wanted to ask: if we had sampled a different but equally plausible set of tasks, would we still observe the same performance differences?

Because our benchmark only samples a finite number of tasks from a much larger universe of Public Service work, different task samples could produce different score gaps or even change the relative ordering of assistants.

To account for this, we used a two-stage statistical significance testing pipeline, applying different statistical tests according to the metric type. However, our analysis does not account for run-to-run variability arising from LLM stochasticity. All tests used an alpha threshold of 0.05.

The flowchart below describes the pipeline in greater detail.

Lessons Learnt

Building GovWork-Bench surfaced a few lessons that were less obvious when we started.

Representativeness is a moving target. Grounding tasks in observed usage gave us a defensible starting point, but no finite benchmark can capture the full spectrum of Public Service work. The benchmark will need to evolve alongside actual usage.

Fair does not mean identical. For product-level evals, we found it more useful to standardise the testing conditions than to artificially standardise the products themselves.

Automation is never really automatic. Browser-based evals gave us greater product fidelity, but Playwright workflows remained susceptible to frontend changes. This made us realise that we needed to treat evaluation harnesses as first class citizens and invest early in reusable automation in order to scale the benchmark.

Closing Remarks & Next Steps

GovWork-Bench is still a work in progress. For our next steps, we plan to expand task coverage, incorporate qualitative feedback from public officers and continue refining the evaluation methodology. 

More broadly, one thing we took away from this project is that building a useful domain-specific benchmark is as much about defining what “good” looks like for the domain as it is about deciding how to run the eval itself.

Credits and Acknowledgements

Watson Chua, Zijing Chak –  for co-authoring this benchmark with me and being involved in every aspect from benchmark construction to evaluation execution

Charles Low – for handcrafting the Hansard benchmark dataset, and brainstorming on public service workflows to add to the benchmark

Barry Tng – for providing policy-related guidance when constructing first-party datasets which aided our understanding of how realistic public sector tasks looks like

Zheng Feng Toh, Lu Li – for auditing SpreadSheetBench and surfacing concerns early on that improved the robustness of our SpreadSheetBench results

Shangru Ng, Yangbo Xu, Keng Ji Chow – for providing access to key platforms and helping to integrate the MCPs/relevant custom connectors