A Message Misfires

In January 2021, amid the COVID lockdown, the UK government released the following poster:

[UK Covid lockdown poster from January 2021]

At first glance, the message was a standard one, which advised people to stay home and save lives. A closer look, however, revealed something more contentious. The poster depicted women busy with household chores and babysitting, while a man lounged on the sofa. The public did not take well to this sexist subtext, and the poster was pulled within days. 

The UK government had set out to deliver a public health message. Instead, it ignited a debate over gender roles. How can the government ensure that its actions and intentions are conveyed in the right manner, rather than being blindsided by the ways the public might construe them?

The Cost of Getting It Right

Policy communication in a society as multi-racial, multi-religious, and multi-faceted as Singapore is especially tricky. To get it right, government agencies conduct rigorous focus group discussions (FGDs) across multiple demographics to better understand and address ground sentiment.

However, FGDs are both costly and time-consuming. Finding the right people to survey, coordinating the sessions, writing up reports, and finally translating the findings into policy decisions and announcements - all of it demands considerable money and effort.

The length of this process also impedes the government's ability to iterate quickly. A single round of FGDs can take months, leaving less time for follow-up consultations to refine a policy further. Moreover, FGDs are often run by different teams, and their findings remain siloed from other public officers, which can lead to duplicated consultations on similar topics and further delays.

Stress-Testing the Message

To address this, our team developed an app to red-team policy communications. The term comes from cybersecurity, where a "red team" probes a system for vulnerabilities before a real attacker can find them. Here, we do the same for policy communications, such as surfacing blind spots in an announcement before it goes public.

As a trial, we consolidated historical FGD transcripts. We then used the data to predict how different Singaporean personas might respond to a given policy or announcement. Instead of opening with a lengthy FGD, policy and communications officers can now run a quick dry-run against the app to surface blind spots and iterate, then conclude with a more thorough FGD in the refinement phase. 

In this post, we'll walk through the technical methodology behind two key features of the policy/comms red-team app:

1. Grounding reactions in real voices. Predicted reactions have to reflect what actual Singaporeans have said, not stereotypes hallucinated by the AI model. We draw on real FGD transcripts to predict how different personas would respond. This involved two steps:
   1. How we processed the raw transcript data
   2. How we retrieved the data relevant to each persona's voice
2. Multimodality. Our app evaluates both words and images because words aren't the only thing that can land badly. As we have seen earlier, a background design or an image can miss just as easily.

Grounding reactions in real voices

To ensure that we reflect the views and opinions of real Singaporeans, we used FGD transcripts to ground each persona's voice. This is an example set of personas we have:

 [example set of personas]

Let’s compare how different personas might react to the 2026 Budget announcement on the $2 billion Progressive Wage Credit Scheme (PWCS) top-up. Below is the announcement we fed into the app:

“The Government will implement a new Progressive Wage Credit Scheme (PWCS) top-up of $2 billion over the next three years to co-fund wage increases for lower-wage workers earning up to $3,000/month. Employers who raise wages by at least 5% annually will receive co-funding of up to 50% of the wage increase in the first year, tapering to 30% in subsequent years. This builds on existing progressive wage efforts and is aimed at uplifting incomes at the bottom while keeping businesses competitive.”

On our app, the SME owner persona pushed back with the same gripes employers had raised previously in an FGD, with a focus on costs, regulatory burden, and manpower issues:

[SME owner’s reactions to PWCS top-up Budget announcment]

Running it past the critical intellectual persona gave a different angle altogether, that subsidising employers changes nothing structural once the three years are up. Between the two, the app grounded its reactions accurately and surfaced the different viewpoints real Singaporeans actually hold.

[critical intellectual’s reactions to PWCS top-up Budget announcment]

Once all the feedback is gathered, we use AI to take it all in and revise the messaging, producing a tailored announcement that highlights which persona voice drove each change.

[AI-revised message addressing targeted personas]

It took several tries to successfully extract these real, unique persona voices from raw FGD transcripts. We describe our techniques in the following sections.

Data Processing

Chunking

We started with the naïve approach of chunking raw FGD transcripts directly into searchable passages. This worked poorly, largely because of how much the transcripts varied in format. Chunks came back truncated mid-sentence, or captured the moderator's words instead of the speaker's. Manually recorded transcripts were riddled with abbreviations, while AI-transcribed ones were filled with errors from mishearing the Singaporean accent.

We tried a number of fixes: adjusting chunk sizes and the overlap between them, segmenting by speaker turn, and using sliding windows. None of it meaningfully improved results.

After brooding over the problem for a while, we realised that what mattered was not the raw text but each speaker's voice and the opinion they were trying to get across. So instead of chunking the transcripts as they were, we added a first-pass LLM call to extract the point each speaker was driving at, and chunked that instead. Inconsistent formatting became far less of a concern, since the resulting chunks were clean and self-contained.

The next step was deduplication. The same opinion often surfaced many times within a single transcript, so we added a pass to ensure each distinct point is stored only once, preventing the same opinion from being retrieved over and over.

[visual representation of opinion chunks stored in the Qdrant vector database]

Tagging

After extracting opinion chunks, we tested the retrieval of chunks via semantic similarity, but the results were off. For example, the persona for an SME owner was quoting opinions from heartlanders.

This was when we realised that semantic similarity alone is not enough. Different personas can have overlapping traits and opinions, so an SME owner can also be a heartlander. Therefore, when we used similarity alone, the wrong voices kept bleeding into the wrong personas.

To fix this, we began adding demographic tags to the opinion chunks. The signal comes from three main places, and we draw on all three because each is strong where the others are weak.

The filename is the most reliable, but also the coarsest. Each FGD is recruited against a defined profile, so the filename often carries the defining trait of the whole group. There's no guesswork involved, but it describes only the group, not the individual speaker.

The opinion text itself is the most specific. As we extract each opinion, we also capture any demographics the speaker reveals in the very lines they are speaking, like "I am 42 and a single parent." Because it's tied to the exact chunk, it's reliable. However, it only helps when the speaker actually says something about themselves.

The per-speaker roster fills the gaps between. A separate LLM pass reads the opening of each transcript and works out who is in the room and their demographics, keyed to the speaker labels. It reaches every speaker, even the quiet ones who never disclose anything, though because it's inferred, it's the softest of the three.

We combine these signals via a layering approach. We start from the roster, let what the speaker disclosed override it, and stamp the deterministic filename trait on top. Should two different signals land on the same field, the more reliable one wins, which is why the filename trait is applied last.

After opinion chunks have been tagged with demographic signals, we built a payload filter, enabled by our Qdrant vector database. This final step narrowed the pool of opinion chunks to the relevant ones before retrieval. 

With the tags in place, the payload filter narrows each persona's search to matching voices before ranking ever happens. The Young Graduate persona, for instance, only ever sees young-graduate opinions. Anything off-profile drops out of the candidate pool up front rather than being filtered out after the fact. But the filter is only as good as the tags behind it, which is why getting the tagging right turned out to be the hard part.

Data Retrieval

To retrieve the chunks that enrich each persona's voice, we chose a hybrid approach called reciprocal rank fusion (RRF), which combines dense and sparse embeddings. Dense embeddings capture meaning, or how semantically close a chunk is to the query, which is scored by cosine similarity between 0 and 1. Sparse embeddings, computed with BM25, capture keywords by scoring chunks on exact term matches via TF-IDF, and their scores are unbounded.

We chose a hybrid method because dense embeddings alone, good as they are at meaning, kept missing something important: our documents are full of abbreviations that are themselves the keywords that matter. A semantic model glosses over an acronym like CDC voucher or BTO, because it does not understand what these terms mean. That is where sparse BM25 earns its keep, by matching the exact keyword.

The catch was how to combine the two techniques. Their scores sit on completely different scales, so we cannot just add a 0.82 cosine score to a 14.3 BM25 score and expect the sum to mean anything. We could try to normalise them onto a shared range, but that is fragile, since one outlier score can throw the whole thing off. RRF sidesteps the problem entirely. It discards the raw scores and keeps only each chunk's rank position in its list. A chunk that comes first in the dense list and one that comes first in the sparse list both count as a top result and contribute equally, no matter how far apart their raw scores were.

Each chunk is then scored by adding up its rank-based contribution across both lists:

score(d) = 1 / (k + rank_dense(d)) + 1 / (k + rank_sparse(d)),

where rank_dense(d) and rank_sparse(d) are the chunk's positions in each ranked list, and k is a smoothing constant that controls how steeply rank matters. We use Qdrant's hybrid search, where k defaults to 2, far smaller than the 60 used in the original RRF paper. A low k makes the contributions fall off fast. With ranks indexed from 0, the top result is worth 1/2, the next 1/3, and a few positions down it's already a small fraction of that. So the strongest hits from each method carry most of the weight, while anything further down barely moves the needle. That suits us, since we would rather surface each method's best matches than smooth everything out across long, noisy lists.

This is still very much a work in progress. We currently weight dense and sparse embeddings equally, but we'd like to try giving one more weight than the other to see how that shifts the results. We're also looking at other fusion methods, such as distribution-based fusion, which normalises each side by its own score distribution rather than going purely by rank, to see how they compare.

Multimodality

Government communications are often accompanied by visuals, so we wanted to capture how different personas react not just to the words, but to the imagery as well. Our visual reviewer reads artwork the way a designer would, weighing both how it looks and how clearly it carries the message.

In critiquing a piece, it blends the two things real designers never separate: craft and message. Craft covers layout, typography, colour, whitespace, and visual hierarchy; message covers clarity, the call-to-action, audience fit, and readability. It has to point at real elements, like "the grey 10pt body on a white background," or "that stock photo of office workers", to offer concrete fixes. It is also trained to catch bias and prejudices.

Furthermore, any FGDs conducted on visual communication materials would inform the visual reviewer, keeping its feedback grounded.

The output is a short sentiment label followed by a 100-to-200-word reaction that reads like candid feedback to the team that made the piece, then three to five prioritised issues, each framed as the problem, the confusion it creates for the audience, and the specific fix. It also watches for when an image does more harm than good, or unsettles people in a way nothing about the layout or wording obviously explains.

Let’s see how the visual reviewer would react to the COVID ad we shared at the beginning of the post:

[visual reviewer’s reaction on the UK covid lockdown poster]

Similarly, the visual reviewer echoed the accusations of sexism that the UK Covid lockdown poster had attracted.

For an example closer to home, we also tested the visual reviewer on a 2019 sexual assault awareness poster that was posted in Singapore's MRT stations:

[2019 sexual assault awareness poster published in Singapore MRT stations]

As shown below, the visual reviewer landed on the very problem the public had raised at the time: the framing made sexual assault sound like a cost-benefit calculation rather than the serious harm it is. The critical intellectual persona caught the same thing from the message side. Neither was told what the controversy had been, and they surfaced it on their own.

[design fixes identified by the visual reviewer]

[critical intellectual’s feedback highlighting layout and messaging issues]

To make its design fixes concrete, the visual reviewer lays out a numbered set of recommendations beside the original material. Beyond the surfaced reactions and design fixes, we can also regenerate a whole new infographic based on specific brand guidelines. If an agency doesn’t have such guidelines to provide, the model can also infer them from the image. The regeneration is powered by Gemini 3.1 Flash Image.

We demonstrate the visual reviewer’s feedback and generation capabilities below using the Ministry of Manpower’s Committee of Supply (COS) 2026 infographic:

[design recommendations and image regeneration interface]

 [original vs. generated infographics]

 [regenerated MOM COS 2026 infographic]

Next steps

Web Search

The next technical feature we’d like to enhance is web search. We launched this as an exploration, and today it powers the critical intellectual, the persona that has to back its critique with real, verifiable citations.

With the rise of personal agents, web search APIs are in demand, and providers like Brave Search, Exa, and SerpAPI are currently costly. Instead of paying per query, we chose to self-host a SearXNG meta-search mirror, which is more cost effective and lets us control which backend engines are queried.

However, this approach is not ideal because established search engines, like Google and DuckDuckGo, are rate-limited. Alternative search engines yielded poor results, as they tended to fixate on the first word of the query and return weak matches. To keep results relevant, we have scoped the web search to government website sources for now.

The web-search path is wrapped with a Claude Sonnet 4.6 model that refines the query each turn until it surfaces something relevant. Safe search is turned on, and soft-failing is enabled, so any error returns an empty result set rather than crashing the whole run.

Here’s an example of the web search is being cited in the critical intellectual persona:

[web search citations by the Critical Intellectual persona]

Continuous Evaluation

We've launched this red-team app in alpha to gather feedback on how it performs. So far, we've evaluated its accuracy by validating the provenance of each persona's reactions, by confirming that they trace back to relevant chunks.

We're now working with interested agencies to assess how closely the personas' reactions match real voices from FGDs. The goal is a more rigorous setup that tests the app's predictions on fresh announcements, ones with newly collected FGD transcripts the bot has never drawn on, judged through a mix of human review and automated metrics. If your agency would like to evaluate the app for your own use, we'd love to hear from you.

With an evaluation pipeline in place, we'll start exploring ways to push performance further, such as testing different embedding models and tuning the parameters in our rank-based retrieval.

Conclusion

Government communications risk carrying meanings their authors never intended. The UK poster that opened this piece wasn't the product of carelessness. It was made by people doing their best to be helpful, who simply couldn't see their own work the way the public would. That gap between intent and reception is the hard, permanent part of the job, and no tool removes it entirely.

What our app does is make the gap cheaper to find. By grounding its personas in what real Singaporeans have actually said, it lets a policy/comms officer rehearse the public's reaction. A dry run that once took weeks and thousands of dollars now takes minutes, across both the words and the images that carry them.

We're careful not to overclaim. The app is not a replacement for talking to real people, and it never will be. Simulated personas are strongest at generating hypotheses and surfacing blind spots early, not at standing in for a representative sample. The aim is the opposite: fewer, sharper FGDs, where policy/comms officers can walk into already knowing the right questions to ask.

Every announcement that lands as intended is a small deposit in public trust, while every one that misfires is a withdrawal that costs far more to repair. If this tool helps officers catch even a handful of those misfires before they happen — before the rest of Singapore does — it will have been worth building.

Government officers on the Government Enterprise Network (GEN) can access the app at: go.gov.sg/comms-redteam

Credits

Technical contributors to this app include Annalyn Ng, who built the initial prototype, and Celeste Neo, who has co-developed the app with me during her internship.

A special shout-out to Jason, Kimberly, Kathi, Kylie and the team, who are our business users that built this with us and gave us constant feedback and support throughout.