Bounded-Memory Long-Horizon Conversations: MGPT on BABILong

A transformer's context window is its short-term memory, and it is both finite and expensive. As a conversation gets longer, the usual options are to grow the window — which costs more compute and memory every turn — or to throw old turns away and lose the thread. Neither scales to the kind of long-horizon interaction we care about.

MGPT — Mask-Generative Pretrained Transformer — takes a different route: the model governs its own memory instead of relying on an ever-growing window.

The result

On BABILong, a long-context question-answering benchmark, a 4B-parameter MGPT reaches 99% token accuracy on the QA1 task with constant memory usage, outperforming substantially larger systems that have to keep growing their context to keep up — enough to answer questions about information that left its effective context many turns earlier.

This is the proof we needed for the architecture thesis: capability does not have to come from a bigger model or a bigger window. A small model that manages its own memory can sustain long-horizon interaction at a fixed cost.

The method

The training recipe and memory mechanism behind this result are part of Continual MI's proprietary MGPT work and are not published. The capability ships as a product through the MGPT API.