The Employee Digital HR Concierge: A blueprint for agentic AI in functional domains

The value we need: One seamless experience

Imagine you want to take leave to attend your child’s school orientation. Which type of leave is applicable? Do you have enough balance? And how can you submit the application for that date?

Instead of navigating different sources and systems, you would want a tool that brings the answers and actions together in one experience. This tool would connect policy information from Instruction Manuals and information such as leave balances from HR systems, allowing officers to move seamlessly from question to answer to action. This would reduce everyday friction and make HR support faster for every officer.

The idea: An orchestrator that serves as a concierge

Our answer is an orchestrator. An orchestrator is a single agent that sits in front of existing systems (be they tools or bots). It understands what the user wants, breaks the request into parts, decides which system should handle each part, calls those systems, keeps track of the conversation, and combines everything into one reply.

The value of an orchestrator is not in replacing the downstream bots. What the orchestrator adds is coordination. The user asks once, in plain language, and never has to know which system holds the answer.

Think of it as similar to a hotel concierge. You do not need to know whether your request belongs to housekeeping, the kitchen, or the front desk. You ask the concierge, and they work out who to call.

With this idea in mind, Singapore’s Public Service Division (PSD) partnered GovTech’s AI Practice to build the Employee Digital HR Concierge (EDC). We began with a prototype focusing on leave-related queries. The prototype can, however, be extended to other HR queries, and we plan to introduce further integrations and capabilities via the agentic architecture as development progresses.

How we built the EDC

Let us follow how a single question flows through the system:

• A public officer submits their request in plain language through Desk (GovTech’s ChatGPT-like platform where officers use approved assistants)
• The request reaches the EDC agent, which sits at the centre and coordinates the work. On each turn, the agent assembles a prompt for a large language model (LLM). The prompt is built from a few ingredients:
  • The agent’s base instructions,
  • The user’s profile (separately retrieved upon initialisation),
  • The list of tools available along with their descriptions,
  • The conversation so far, and
  • Rules for how to write its reply.
• The model reads this and decides whether to answer directly or to call one or more tools.
• If the model asks for one or more tools, the agent makes those calls through an MCP gateway to the right HR systems. MCP, the Model Context Protocol, is a common standard that lets an agent discover and call tools.
• The results come back to the agent, which assembles a new prompt for the model, and the loop repeats until the task is done or it reaches a limit on the number of tool calls.
• The final answer is then sent back to Desk and shown to the user. The user may follow up with additional messages.

One design choice that differentiates an orchestrator from a “menu” is worth highlighting. We do not hard-code a table of keywords that maps a phrase to a tool. Instead, we give the model a filtered set of tool descriptions and let it choose which tool to call and in what order. For better access control, we only surface the tools that the officer’s agency is entitled to, on a least-privilege basis.

All of this runs on GovTech’s ‘Harness’, a platform for building and hosting agents. Utilising Harness greatly assists the development work, because it provides the heavy machinery so we can concentrate on the HR logic. Harness gives us a managed agent runtime, access to large language models, a gateway for tools, and observability so we can see what the agent is doing. We only need to build the parts that are specific to HR: the orchestration logic, the prompts, the connections to HR systems, and the evaluation framework.

Another benefit of using Harness is that getting code into production is “unexciting”, which is the point. We simply push to a GitLab repository, and a CI/CD pipeline builds the container, publishes it, and deploys the new image to the managed runtime. This lets a small team ship reliably.

It is worth emphasising that this same template (one orchestrator in front of many specialised tools) can be reused for Finance, Procurement, or IT support. Only the tools and the domain knowledge change.

How we evaluated the EDC

Evaluating the EDC prototype was a critical part of this project. We evaluated it for three reasons.

• First, to diagnose where the system could improve, so we knew what to fix.
• Second, to provide concrete and specific data for decision-making on whether the EDC was good enough to roll out.
• Third, to be honest about what the EDC could and could not do well, so that expectations were set correctly.

Our approach followed the four main steps of the core evaluation loop, as described in our Evaluation Playbook: define what “good” means, design how to measure “good” and design test scenarios that match real use, execute the evaluation, and improve based on our diagnosis of the test results.

Defining “good” and designing how to measure it

“Good” for the EDC meant that the user’s request was addressed, routed to the right systems, and answered completely and truthfully. We turned that into four metrics.

For each test conversation, we computed the following binary scores using an LLM judge:

1. Intent completion asked whether the orchestrator resolved everything the user had set out to do. A single request could hold several intents, and all of them had to reach a valid outcome for the conversation to pass. A successful action was not the only way to score a valid outcome. A clearly explained “I cannot do that”, a legitimate refusal, or a correct denial of a leave request for a genuine business reason all counted as valid outcomes, because in each case the user received an honest, correct result.
2. Query decomposition checked whether the orchestrator had broken the request into the right parts and sent each to the correct system. For instance, “Check my remaining childcare leave and what I can use it for” had to become two sub-queries: a balance check and a policy question.

For each turn of the test conversation, we computed the following binary scores (or NA if there was nothing to evaluate) using an LLM judge:

3. Reply completeness checked whether the reply surfaced every critical piece of information the tools had returned, rather than quietly dropping part of it.
4. Grounding checked that the reply did not invent anything and stuck to what the tools actually said.

One difficulty we faced when designing the metrics was that the “goodness” of the replies depended on systems we do not own. For example, if a reply was incomplete or contained hallucinated facts, was it due to the orchestrator layer or the downstream bots giving incomplete or wrong answers? We went to some lengths to isolate the performance of our orchestrator layer. Specifically, (i) we measured reply completeness against the set of information that the tools had returned; and (ii) we measured grounding with the tool responses taken as absolute truth. The reason for this design decision was to make it clearer what our orchestrator needed to improve on; the downstream tools were expected to run their own evaluation and improvement cycles.

We also tracked two operational qualities on every call:

5. Latency measured the time taken to process a conversation turn. Latency was reported not just as an average but also as the median and the slow tail, so we could see both typical and worst-case speed. We also isolated the latency incurred by the orchestrator layer and each downstream tool (whenever called) for more detailed diagnosis.
6. Tool reliability measured the rate at which each downstream system or tool responded successfully. From time to time, an operational failure could happen due to a timeout, an authorisation error, or a malformed response. We collated and reported how often such failures occurred for each downstream tool, and provided this information to the system owners for further action.

Designing and executing realistic test scenarios

Before user acceptance testing (UAT), we first tested our agent by generating our test cases synthetically. One LLM played the employee, following a defined persona and profile, and held a live, multi-turn conversation with the real deployed orchestrator. A separate LLM acted as a judge and scored the conversation against the metrics above. Each conversation ran for up to ten turns.

Some thought went into designing each conversation (which we called a “scenario”). We started from a set of intents, not from a script. Each scenario sampled one or more base intents, such as “apply for leave” or “check eligibility for paternity leave”, and then layered on realistic modifiers such as leave type, dates, ambiguity, or an insufficient balance. From there, we generated a persona, their context, and their way of speaking around that fixed set of intents.

Anchoring on intents gave us two things at once. It was a concrete lever for diversity, because we could deliberately vary the intents and conditions under test. And it was a concrete checklist for intent completion, because we already knew exactly what the user was trying to achieve and could tick each item off.

The intents themselves were not invented in a vacuum. We drew them from the existing test cases, so our scenarios reflected the questions the EDC could see in practice. In addition, a subject matter expert had given us a set of complex, realistic leave questions, such as planning leave across a year-end boundary or planning the optimal combination of leave for childbirth. We also seeded synthetic conversations from these, and scored intent completion against the list of base intents provided by the subject matter expert.

To keep the conversations lifelike, we gave personas one of six tone presets, for example “neutral and concise” or “casual and conversational”, and half of them stated their full request upfront while the other half let it unfold gradually.

For safety testing, we also generated out-of-scope requests to check that the orchestrator stayed within its remit. Some were wholly out of scope, such as asking it to plan a holiday, where the right behaviour was to decline politely and explain why. Others mixed relevant with irrelevant requests, and the EDC had to refuse at least the irrelevant ones. The metric here was the block rate, i.e. the percentage of conversations with out-of-scope content that were successfully refused. We also onboarded the EDC to Litmus and Sentinel for further safety testing.

Improving the EDC based on error analysis

Through a thorough analysis of the evaluation results, we uncovered some actionable ways to improve the EDC. For example:

• Evaluation revealed cases where a connected service interpreted a user’s request incorrectly. The issue was shared with the relevant service owner and addressed, demonstrating how end-to-end testing could improve the wider service ecosystem.
• Latency analysis paid off too. We found that the very first message in a conversation was consistently slow. The cause was that the orchestrator had to load its tool registry (the list of available tools) before it could act. The team used these findings to explore platform-level optimisations.

What comes next

The EDC is being prepared for a phased rollout, beginning with simpler services and subject to the necessary readiness and approvals. This work has been made possible through close collaboration across policy, product, engineering and platform teams:

• GovTech’s AI Innovation squad in AI Practice: Barry Tng, Darrius Ng
• Our partners in PSD: Yong Zhun Hung, Victor Ong, Jeanette Khong
• GovTech’s AI Programme for support on Harness and Desk: Ng Shangru, Xu Yangbo, Chow Keng Ji
• Partners from HR System teams

We built the EDC for HR, but the pattern it demonstrates goes beyond HR. We hope it provides a useful example of how to build an agentic orchestrator on shared infrastructure and evaluate it rigorously enough to support responsible adoption. Teams exploring similar approaches can adapt the same principles to their own functional domains.