Before We Generate 1,000 Learning Missions, We're Building the Tests First

Imagine we took the obvious shortcut.

We connect an LLM to our curriculum outlines, prompt it to generate Learning Missions, and let it run. By the end of the afternoon, we have a thousand new missions sitting in our content library, ready to publish. The team celebrates.

Then someone actually starts reading them.

Mission 41 is flagged: a grammar exercise set at a void deck where two students confess their feelings for each other. Technically correct. Completely inappropriate for a Primary 4 classroom.

Mission 212 uses 鱼丸 (yú wán) for “fish ball,” when 鱼圆 (yú yuán) is the term more commonly used by Chinese speakers in Singapore.

Mission 380 has a student ordering beer during a conversation at a kopitiam, a traditional Singapore coffee shop. This is hardly a suitable scenario for a school activity.

Mission 614 says 我坐巴士 (wǒ zuò bā shì), which literally means “I sit on the bus.” It is understandable, but students in Singapore would more naturally say 搭巴士 (dā bā shì), meaning “take the bus,” or 搭 MRT, meaning “take the train.”

None of these are catastrophic failures. That's exactly the problem.

Each issue is subtle enough to slip past a quick review. But there are now a thousand missions to review. By the time the pattern emerges, the content is already live, students have already worked through it, and "fixing it later" means auditing an entire content library, one mission at a time for problems nobody ever defined clearly in the first place.

This is exactly the failure mode we're trying to avoid at LangBuddy, an AI-powered Mother Tongue conversation and oral practice platform for students in Singapore.

Rather than generating thousands of Learning Missions first and worrying about quality later, we're taking the opposite approach: defining the tests before we generate the content.

Where the current process runs out of room

Every Learning Mission on Langbuddy (the bite-sized exercises that guide a student through a piece of vocabulary, a grammar pattern, or a cultural context) is handcrafted by our language specialists today. That's a deliberate choice. Mother Tongue content is sensitive. Tone, age-appropriateness, cultural nuance, and pedagogical correctness all matter, and machines have historically been bad at all four.

But handcrafting doesn't scale. As we look to grow Langbuddy's content library, three problems keep resurfacing:

1. Quality can vary. With specialists writing missions independently and limited time for cross-review, the definition of “good” may differ between batches.
2. A small team has limited capacity. Supporting more topics, grade levels, and language variants requires more content than a small team can reasonably produce by hand.
3. Governance becomes harder. As we explore AI-assisted drafting to increase our capacity, a new risk emerges: how do we ensure AI-generated content is safe, accurate, and aligned with the curriculum before students see it?

We cannot solve the third challenge simply by producing missions faster. We first need to define what “good” means and build a system that checks every mission against those standards before generating content at scale.

Why we're not starting with generation

We could generate 1,000 missions today. We're choosing not to, until we can tell you, mission by mission, exactly why each one passed or failed, before a single student ever sees it.

In a previous article, I wrote about the test pyramid: unit tests check a small piece of logic in isolation, integration tests check that components work together correctly, and end-to-end tests check the whole system the way a real user would experience it. The same shape applies here. We're deliberately not skipping straight to end-to-end, a thousand missions live in front of real students, with content that's never been tested at any smaller scope first.

It's the same discipline as defining success criteria before running a performance test, response time thresholds, error rate ceilings, resource utilisation limits, just with a different unit of measurement. Instead of latency and throughput, our criteria are about curriculum accuracy, linguistic correctness, tone, and safety.

We're not the first language learning team to learn this the hard way. In their write-up on scaling DuoRadio, Duolingo described how their first attempts at AI script generation produced content that needed extensive manual editing to meet curriculum and quality standards. Their fix was to build what they called "comprehensive evaluators": AI-powered filters that assessed every generated script for naturalness, grammaticality, coherence, and logic before it reached learners. Only after those filters were in place did generation at scale become viable. They went from 300 episodes to over 15,000 across 25 courses in under two quarters. The evaluators came before the scale, not after.

So before generating missions at scale, we asked our language specialists a deceptively simple question: "When you reject a mission, why?"

Their answers are becoming our test suite.

Turning specialist judgment into a rubric

We sat with our language specialists and catalogued the reasons a mission has ever been sent back for rework. The list converged into a small number of evaluation dimensions:

• Curriculum alignment. Does this mission teach what it claims to teach, at the right proficiency level?
• Linguistic accuracy. Is the grammar, vocabulary, and usage correct for the target Mother Tongue Language?
• Age and tone appropriateness. Is the content suitable for the intended age group, free of anything jarring or inappropriate?
• Pedagogical structure. Does the mission scaffold learning correctly (clear instructions, fair difficulty, a sensible answer key)?
• Safety and policy compliance. Does the content avoid anything that violates our content policies, including AI safety considerations specific to generated text?

Each dimension is becoming a rubric with concrete pass/fail criteria, not vague guidance like "make sure it's good," but specific, checkable statements a judge (human or machine) can apply consistently. This is the same discipline as defining success criteria for a load test: vague goals produce vague results, but "P95 response time under 3 seconds" produces a clear pass or fail.

This rubric is the contract we're building between our specialists and any system, human or AI, that wants to produce a Learning Mission.

The architecture we're building: unit test, integration test, then live

With the rubric taking shape, we can design a pipeline meant to scale mission creation without scaling quality risk. The core idea borrows directly from the test pyramid: don't send anything straight to production. Test it at the smallest useful scope first, then build up.

Here's how it's designed to work end to end.

A language specialist still sits at the start of the pipeline, providing curriculum input and maintaining the rubric, the same way a team agrees on requirements before a feature is built. From there:

1. AI mission generation drafts a batch of missions based on the curriculum input. This is the raw output: fast and cheap to produce, not yet proven.
2. An LLM-as-judge evaluator scores every draft against the rubric, checking curriculum alignment, linguistic accuracy, tone, structure, and safety automatically. This is the unit test layer: a single mission checked in isolation against a fixed standard, before anyone else gets involved.
3. Drafts that fail are sent back for automatic regeneration, re-prompted with the specific rubric criteria they missed, then re-scored. A failing draft gets reworked and re-checked, not waved through to the next stage.
4. Drafts that pass the LLM judge move to human review, so the specialist reviews pre-filtered, pre-scored content instead of starting from a blank page. This is the integration layer: checking that the mission doesn't just pass in isolation, but actually fits the broader curriculum and the learner's journey.
5. Approved missions get published, joining the live curriculum that real students learn from. This is the end-to-end layer. Rejections at the human stage feed back into the rubric, sharpening the judge for the next batch.

The result is a loop, not a one-way pipeline. Every rejection, whether caught by the LLM judge or by a human, is meant to become information that improves the system, instead of a dead end that only the rejecting specialist learns from.

We've built and are actively testing the evaluator against this rubric. Full-scale generation, the 1,000 missions in the title, is our end-to-end stage. We're not running it in production until the unit and integration layers both hold up on their own.

Why Human Review Remains Essential

We have made a deliberate design choice: every mission will still be reviewed by a human before it is published.

The LLM-as-judge acts as a pre-filter. It does not replace specialist judgement. Passing an automated check does not mean a mission is ready for students. It means the specialist has a stronger starting point for review.

This matters for two reasons:

• Mother Tongue content carries cultural meaning that is difficult to capture fully in a rubric. An evaluator can check grammar, structure, and other defined criteria consistently. But a language specialist is better placed to judge whether an expression feels natural or whether a cultural reference is appropriate.
• Safety and governance are part of the problem we are trying to solve. Removing human review would replace the limitations of a fully manual process with the risks of ungoverned AI-generated content.

The evaluator’s real job is to make specialist time more effective. It catches routine issues early, allowing specialists to focus on the nuanced decisions that genuinely require human judgement.

What We’ll Be Watching Once Generation Starts

We do not have results at scale yet. That is the point of this article, not a gap in it. But we already know what we will measure once the pipeline moves from “built” to “running”:

• Pass rate at the LLM-as-judge stage. How much generated content meets the quality bar without rework.
• Agreement between the LLM judge and human reviewers. Whether its decisions align with specialist judgement.
• Specialist review time per mission. Whether automated checks genuinely reduce the team’s workload.
• Defect rate reaching students. The metric that matters most, and one we are not willing to guess.
• Teacher and student feedback. Whether missions are clear, engaging, relevant, and useful in real learning environments.

Automated checks are only the beginning. Over time, we want our evaluations to capture specialist judgement while remaining grounded in the experiences of teachers and students. Their feedback and annotations can help us improve both our quality standards and the design of future missions.

If we publish a follow-up with real results, it will mean the evaluation system has earned enough trust to operate at scale. If the results reveal gaps, that will be valuable information too.

Where this leaves us

Scaling content generation is relatively straightforward. Scaling confidence in the quality of that content is much harder.

For LangBuddy, that means treating evaluation as part of the foundation, not a final checkpoint.

Our immediate objective is demonstrating that the evaluation system can reliably distinguish content that meets our standards from content that needs further work.

We have not generated our 1,000 missions yet. First, we built a system that will tell us whether they are any good, much like writing unit and integration tests before releasing software.

In our experience, that is the order that works.

That is the engineering problem we need to solve before we scale, and one that extends well beyond language learning.

How do you scale AI-generated content without compromising quality? Share your thoughts in the comments or reach out. I’d love to exchange ideas.