Teaching an AI marking assistant to remember: Building and evaluating Markly’s memory layer

Why Markly needs memory

Markly is the Singapore Ministry of Education's (MOE) AI-assisted marking tool for long-form, rubric-based essays, currently piloted in Geography, English, History and Social Studies. Teachers upload student scripts, attach a rubric and instructions, and Markly drafts inline comments, overall feedback and suggested levels. The teacher stays in control: each suggestion can be accepted, edited or rejected, and a built-in chat lets them instruct and steer the assistant.

Imagine marking an assignment with Markly. Script by script, you steer it: comments become shorter and more encouraging, feedback begins to target the reasoning gaps you care about, and by the final scripts the drafts read almost like your own. Then the next assignment arrives, and none of it carries over. The teaching starts again from the beginning.

That steering carries real teaching knowledge. A teacher might want students to explain the mechanism behind a claim, define their evaluative criteria before making a judgement, or work on a few important weaknesses before polishing every sentence. Another teacher might place greater emphasis on recognising sound reasoning before suggesting improvements.

A rubric establishes assessment expectations, but it cannot capture every decision about how to help a particular student learn. Teachers supply that context through their instructions and revisions. Without persistent memory, much of what they teach Markly stays within the assignment where the interaction happened.  

Together with MOE's Markly team, GovTech's AI Innovation squad in AI Practice set out to preserve this teaching by building a memory layer. The aim is for Markly to learn each teacher's preferences from their ordinary marking activity, so that useful guidance carries into later assignments.

That requires judgement about what to remember. A request about one paragraph should not automatically become a permanent rule, and an established preference may later change. The memory layer needs to retain useful evidence, recognise its scope, and make it available when it matters.

How agentic memory works

An agentic memory layer typically connects two processes: writing useful information into memory after each human interaction and retrieving relevant information from memory before a subsequent model response. The model receives the retrieved material as part of its context, so it does not need to be retrained whenever a teacher expresses a preference. The documentation of Mem0, a memory framework for AI agents, describes this extraction-and-retrieval pattern.

Turning interactions into memories

Writing memory begins with capturing evidence. In Markly, this can include a teacher's chat instructions and their revisions to proposed comments. An LLM distils that evidence into concise memory units: statements describing something the assistant may need to apply again.

For example, several requests to break feedback into individual suggestions might support a memory that the teacher prefers giving feedback in bullet points over continuous prose. Beyond the statement itself, a useful unit can retain a pointer to the source interaction, along with information such as its type, confidence and importance.

The system then reconciles the new units with existing memory. Duplicate observations may be merged, repeated evidence may strengthen a preference, and conflicting observations may require an update. Markly's design stores the reconciled units in a separate pool for each teacher and subject, giving retrieval a defined boundary.

Bringing the right memory back

When Markly prepares a chat reply or an inline comment, it forms a query from the current task. The memory layer ranks stored memory units by relevance to this query and returns a limited selection. Depending on the framework, ranking may use keyword overlap, semantic similarity, or additional signals such as importance and recency.

These retrieved memory units join the assignment instructions, rubric and student script in the model's context. The model can then apply a remembered preference to the task in front of it: for example, prompting a student to connect evidence to an argument in the concise, encouraging style their teacher expects.

Different frameworks, different choices

We explored several memory frameworks, each making different choices about how memory is distilled, stored and retrieved:

• Mem0 provides a memory layer that extracts reusable information from interactions and retrieves it for later requests. It offers configurable components for extraction, storage and search. Mem0 overview.
• SimpleMem compresses dialogue into structured memory units and supports retrieval over that representation. Our experiments included a keyword-focused configuration and an evolving-memory variant that reinforces or revises stored evidence. These are configurations we evaluated, rather than a claim that all SimpleMem deployments behave identically. SimpleMem repository.
• Graphiti represents information through entities and relationships in a temporal knowledge graph. This makes relationships and changes over time explicit parts of the stored representation. Graphiti repository.
• The Markly variant lets us make application-specific choices about distillation prompts, embeddings, memory resolution and retrieval methods, all within a service that fits Markly's workflow.

We also retained a no-memory control. A memory system earns its place by improving what the teacher receives, compared with the same task handled without remembered context. A larger store or a more elaborate architecture is not evidence of that improvement on its own.

How we evaluated memory for Markly

Our central question was whether Markly learns preferences from earlier interactions and applies them appropriately to new work. A single overall score would not tell us much, because writing, retrieval and application can each perform poorly in its own way: the system might infer the wrong preference, store it correctly but retrieve it for the wrong task, or retrieve it successfully and still fail to apply it. We therefore examined the full path from memory formation to generated feedback, measuring each stage separately so that the frameworks could be compared step by step.

Simulating teacher interactions across assignments

Our benchmark used ten simulated teachers with distinct preference profiles. Their interactions took place around sampled Secondary 4 Geography assignments and student scripts. The simulated teachers responded to what Markly produced: they could chat, request comments, review and edit feedback, and end the session.

Each teacher progressed through six main blocks and a separate distractor branch. A block contained two assignments for building memory, followed by one held-out assignment for testing. Each assignment contained ten scripts. Memory accumulated through the build sessions, while test sessions did not add new memories.

This separation was essential: we wanted to measure whether earlier evidence helps on new work. For the same reason, the memory systems being evaluated did not receive the simulated teacher's private preference profile or the reference answers used for scoring.

The main sequence began with preference set A. After three blocks, the teacher changed to preference set B, retaining half the preferences and replacing the rest. This tested whether memory could preserve still-relevant guidance while adapting to a sustained change.

The distractor branch started from a separate copy of the memory state saved just before that change. One build session introduced contradictory directives; the next restored the original preferences. We then tested against the original profile, retaining the memory affected by the contradictory session. This tested robustness against isolated contradictory preferences and recovery through further evidence, without simply rolling memory back.

Following the evidence from storage to output

We organised the evaluation around four questions, one for each stage.

Did the system capture the teacher's preferences? Memory-formation checks examined whether stored units preserved the intended guidance. Our framework included preference-reconstruction comparisons using Arena-based evaluation, where an LLM judge assessed which memory collection better represented a known profile. We also inspected duplication, contradictions and how units were consolidated or superseded.

Did it retrieve useful evidence? Retrieval evaluation labelled memories for relevance to a task and examined both relevant-memory retrieval (using recall@K) and ranking quality, using Normalised Discounted Cumulative Gain (NDCG). Ranking matters because the model receives a limited memory budget: useful evidence buried below irrelevant units may never reach the prompt. Saved retrieval selections also let us inspect what the model actually saw.

Did Markly apply preferences accurately and reasonably when responding in chat? We compared each generated chat response with a reference response, checking whether the expected points were covered and whether the reply aligned with the teacher's applicable preferences. Point coverage distinguishes a fully expressed point from a partial, missing or contradicted one. This gives us more detail than a single judgement that a reply looks helpful.

Did essay feedback improve? Markly generates comments by selecting snippets from the script and writing feedback specific to each one. We tested comments in two complementary settings. In the fixed-passage setting, Markly received the snippet to comment on, isolating the quality of the feedback it wrote. In the whole-essay setting, Markly chose which snippets to comment on as well, testing snippet selection and feedback generation together.

In the whole-essay setting, we first matched generated comments to reference comments by their passage snippets and measured which reference targets were found or missed. Matched comments, and all comments in the fixed-passage setting, were then scored in the same way as chat responses: against the reference comment and for alignment with the teacher's preferences.

Alongside these comparisons, we used a separate general quality score to investigate whether adding retrieved memories affected the overall quality of Markly’s chat responses and comments. The specific definition of “quality” in this context was designed through consultation with MOE. Personalisation should preserve accuracy, relevance and usefulness, while respecting the rubric and the core instructions in Markly’s system prompt.

Assessing how memory is managed over time

As teachers work through more assignments, the memory layer receives repeated instructions, new preferences and occasional contradictions. We assessed how it managed this accumulating evidence, alongside the quality of the responses it produced.

We tracked memory-pool growth at each block: how many units had ever been stored, how many remained active and available for retrieval, and how these counts changed across assignments. Comparing the active pool with the cumulative total shows how much stored material has been superseded. We also distinguished memories originating from chat and comment interactions to understand how each contributed to the pool.

Tracing the lifecycle of memory units helped us understand what was driving pool growth and how often each resolution step was applied. Consolidation should keep the pool compact, but a smaller pool alone does not establish better memory: useful distinctions could have been lost along with repetition.

Finally, we measured the proportion of active memories retrieved across the evaluation tasks to understand how much of the pool was being used. Memories that were never retrieved may still be useful in other contexts, so this measures utilisation within the tested tasks and retrieval queries rather than the pool’s overall utility. Read alongside retrieval relevance and response quality, it helped us assess whether the memory pool remains useful as it grows.

Encouraging results and the teacher pilot

Our benchmark comparisons showed encouraging improvements over the no-memory control. With memory, Markly's chat replies better reflected the teacher's preferences and covered more of the expected points. General chat quality also improved in the separate assessment. Comments showed stronger preference alignment in both the fixed-passage and whole-essay settings.

These findings support the purpose of the memory layer: carrying teachers' guidance into later tasks can make Markly's output more consistent with their expectations. The gains were not uniform across every dimension, however. General comment quality did not show the same improvement, giving us a clear reason to keep evaluating personalisation and feedback quality separately.

The Markly variant is now deployed as an independent memory service in MOE's stack, with a teacher pilot involving Geography and English. Teachers continue using the existing marking interface. The service boundary also lets the team evolve the memory implementation without tightly coupling it to the core application.

The live pilot examines whether the benchmark improvements translate into everyday marking. We are monitoring comment acceptance and editing, how much steering teachers need to provide in chat, and the memory lifecycle itself. These observations will help establish whether the remembered guidance is useful in practice; the benchmark findings above should not be read as measured reductions in teachers' workload.

What we learned about evaluating memory

For teams building a memory layer of their own, three lessons from this work are useful for us to share:

• Test memory across multiple sessions. How the pool grows, how often stored units are superseded, and whether the system adapts when preferences change only became clear across blocks of assignments. A single-session test would have missed these patterns.
• Score personalisation and general quality separately. Memory strengthened preference alignment in comments without improving their general quality. A single blended score would have hidden that gap.
• Measure each stage on its own. Because formation, retrieval and application were scored separately, we were able to trace a weak response to the step that caused it.

This work has been made possible through close collaboration across product, engineering and platform teams:

• GovTech’s AI Innovation squad in AI Practice: Barry Tng, Watson Chua, Darrius Ng
• MOE Markly team: Leo Li, Leonard Siar, Janice Tan, Goh Sheen An, Justin Ng, Richelle Tan

While we analyse the results from the pilot, we are also running further studies to better understand when memory helps, when it becomes stale, and when a teacher's instruction should stay local to an assignment. We are also exploring future work on capturing student learning patterns and on lossless memory management techniques such as memory dreaming. If any of these ideas sound interesting to you, feel free to leave a comment!