Teaching a language model to stop thinking out loud

The problem: reasoning traces in the assistant response

Our AI assistant uses a language model to generate the assistant’s responses and to make tool calls. For latency reasons we run the model in non-reasoning mode, so it answers directly instead of writing out a reasoning trace first.

After we moved some of our deployments onto GLM-5.2, we started seeing new issues in the conversations between the AI assistant and the user. GLM-5.2 had started adding its reasoning traces to the actual assistant response. The model would write out what the user was asking, which rule applied and what it was about to do, and only then give the reply. Here are a few representative samples of such leaks.

These issues showed up in about 1% of conversations. GPT-5.2, which we had used earlier for the same assistants, had them in about 0.03% of conversations. The leaks were also easy to reproduce. When we replayed one logged turn through GLM-5.2 forty times, it leaked on 26 of them. So this was not a rare event that happened by chance. For those turns it was the model’s default behaviour.

What prompting could and could not fix

We started with prompt engineering. Every leak we observed happened at a point where the model had to make a decision while writing its reply: whether to offer a transfer, whether the other side was a voicemail system, whether to switch language. At each of these points we added an instruction to the prompt that told the model exactly what its reply should look like. For example, at the point where the assistant decides whether to offer a transfer, the prompt said that the entire reply should be one short confirmation question, and gave an example of one. This fixed the leak at that point almost completely. We also added a short section to the prompt that stated the rule that everything the model writes is shown to the user, with a pair of WRONG and RIGHT example replies, and we repeated that rule in one sentence at the very end of the prompt as a reminder. The table below lists the prompt changes we tried, the leak rate before and after each one, and what it taught us.

Prompting was patching each issue we encountered, and it did not provide a general fix for the problem. A leak that fired on 165 of 165 replays in one conversation state went to zero once we scripted that state, and then a neighbouring state started leaking. After every fix there was still a floor of roughly half a percent to one percent of conversations, and the remaining leaks were always at a decision point we had not scripted yet, or on users who explicitly asked the model to explain its reasoning, or on language-switch decisions.

Finding the root cause

To understand what was going on, we turned reasoning mode on and replayed the same leak-prone contexts. With reasoning on, the leaks disappeared (0 of 36 replays, compared with 15 of 29 with reasoning off), the tool syntax leaks disappeared too, and tool calls became more reliable. The cost was latency: time to first token went up by 1 to 4 seconds on decision turns. Our assistants have to answer in real time, so we could not simply turn reasoning on for every turn.

This experiment also explained the defect. The model has learned to reason before it answers. The non-reasoning chat template asks it to skip that step by pre-filling an empty think block at the start of the response. When the context is dense with rules and the user’s intent is vague, the model still reasons, and that reasoning ends up in the only place left, which is the AI assistant reply. Our hypothesis was that GLM-5.2’s post-training was dominated by reasoning-mode data, and that the non-reasoning path received much less attention. One observation supporting our hypothesis is that GLM-5.3, the next model in the same family, no longer offers a non-reasoning mode in its API. The lowest setting is minimal reasoning, not none.

We tried three ways to mitigate the issue without paying the latency cost of reasoning mode.

• A fixed thought. The template ends the prompt with an empty think block. We tried filling that block with a fixed sentence such as “I have already worked out what this turn needs”, so that the model starts writing the answer as if it had already reasoned. This worked on the habitual leaks: narration on the Spanish-language turns went from 2 of 5 to 0, and the policy deliberation leaks went from 3 of 10 to 0, with no latency cost. It did not work on turns that needed real deliberation. When the model had to classify an unusual voicemail message, it still narrated on 2 of 30 samples and produced malformed tool syntax on 3 of 30, while real reasoning gave 0 of 30. So the fixed thought helped where the model was narrating out of habit, but not where the turn genuinely needed some reasoning.
• A regex guard at runtime. We can deterministically try to strip reasoning out of the response with a regex search before the response reaches the user. This is only a patch. It catches the obvious cases, such as a sentence that starts with “The user is asking”, but it does not address the cases where the reasoning appears in a more subtle way that the regex misses. Removing a sentence also does not give the user the right sentence in its place.
• Other models. We looked at DeepSeek V4 Flash and saw leaks there as well. We looked at GLM-5.3 Flash, but it only runs in reasoning mode, which adds the same extra latency.

That left one option: fine-tune the model.

Why DPO, and why not just SFT

The obvious recipe for fine-tuning is supervised fine-tuning (SFT): collect a set of clean conversations where the assistant did not leak, and train the model to reproduce them. We did not think it would work here, for two reasons. First, clean conversations carry no signal about the leak. The model already assigns high probability to a clean reply, and making that probability slightly higher does nothing to push the leaky version of the same reply down. Second, SFT pulls the model toward the dataset. Whatever tone, reply length, tool-argument habits and mix of users the clean conversations happen to have becomes the model’s new default, and that is exactly the kind of regression we could not afford across a dozen assistants with different prompts.

Direct preference optimization (DPO) fits the problem better because it trains on the difference between two responses to the same prompt: a rejected response and a chosen response. If the rejected response is the leak followed by the reply, and the chosen response is the same reply without the leak, then the only difference between the two is the leak, and that is the only thing the model is pushed to change. Every pair in our training set has this shape, and both sides of every pair are the model’s own text, sampled with the same prompts and settings as production.

Two additions to plain DPO mattered.

A likelihood term on the chosen response. When the rejected response is this easy to tell apart from the chosen one, the DPO objective is satisfied as long as the rejected response becomes less likely than the chosen one. It does not care whether the chosen response itself stays likely, and in practice the probability of both responses can fall as long as the rejected one falls faster. This is a known failure mode of DPO, and on a production model it shows up as drift in behaviour everywhere, including on the turns we were not targeting. To prevent it we added the negative log-likelihood of the chosen response to the loss, so the model is also trained to keep producing the chosen response. This is the RPO formulation (regularized preference optimization), which is DPO plus a standard supervised loss on the chosen response, and we gave the two terms equal weight. The training curves further down show the effect: the chosen responses stay where they were, within about one nat of log-probability of the base model, while the rejected responses drop by about 87 nats.

A length correction. In 95% of our pairs the rejected response is longer than the chosen one, because it is the chosen response with a leak in front of it. Without a correction, the model could satisfy the training objective simply by learning that shorter responses are better, which is not what we want. We used the length-desensitized variant of DPO (LD-DPO). It splits the longer response of each pair into the part that overlaps in length with the shorter response and the part that goes beyond it, and it gives the tokens in the extra part half the weight of the others. This reduces the reward the model can earn just from a length difference. It was on from the first step of training.

Building the DPO pairs

The pairs came from five kinds of context. A context is one point in a conversation where the assistant has to reply, and each one was rendered the way the production deployment renders a request: the deployment’s system prompt, the conversation history up to that turn, and the tool schemas, passed through the model’s own chat template with reasoning disabled. The conversations themselves are synthetic. They were built from production conversations without any end-user data, as described in the section after this one.

Here is how we built the DPO pairs from these contexts.

1. Sample the model. For each context we sampled the production model eight times at temperature 1.0 with reasoning off. These samples are the candidates for both sides of a pair. For the leak-prone contexts and for the ordinary turns we also sampled the model five times with reasoning on. We had seen in the root-cause experiment that the model makes the right decision and produces a clean reply once it is allowed to reason, so these reasoning-on samples give us three things for the same context: a reliable answer to which action is correct, a clean reply for contexts where every reasoning-off sample leaked, and a reasoning trace in the model’s own words that we can use to build a rejected response. The steps below use each of these.
2. Label every sample. A labeler checked each sample for leaks. Deterministic detectors handled the classes with a clear textual signature, such as narration, tool syntax and bracketed asides. A language-model judge handled the classes that a regex gets wrong, such as the model deliberating over policy from the state of the conversation, language switches and repeated prompt rules. The judge had to quote the leaking text word for word, and if the quote could not be found in the sample the label was discarded.
3. Pick the rejected response. A sample flagged as a leak became the rejected side of a pair.
4. Decide the correct action for the context. Before choosing the other side of the pair, we needed to know what the right action was at that point in the conversation: a specific tool call, or a plain reply with no tool call. We first looked at the clean reasoning-off samples. If at least 60% of them took the same action, that became the reference action. If the clean samples disagreed with each other, we used the reasoning-on samples as the referee and took their majority action, because the model decides more reliably when it reasons.
5. Pick the chosen response. The chosen side had to be a clean sample that takes the reference action: the same tool call, with the same arguments for tools that take a fixed set of values such as language and end-conversation, or no tool call if that was the reference action. We never paired a leak that ended in a tool call with a clean reply that made no tool call, because that would teach the model to skip the tool call rather than to skip the narration.
6. Handle contexts with no clean sample. For some leak-prone contexts every one of the eight reasoning-off samples leaked, so there was no clean sample to use as the chosen side. We first sampled the model eight more times. If there was still no clean sample with the reference action, we took a reasoning-on sample, removed its reasoning trace, and used the reply that followed the trace as the chosen response, since the model produces a clean reply once it has reasoned. Contexts that still had no chosen response were dropped. We never wrote a chosen response by hand.
7. Build a rejected response for the ordinary turns. The ordinary turns had no leak to start from. The assistant’s clean reply at that point in the conversation became the chosen side. To make a rejected side, we took the reasoning trace from a reasoning-on sample of the same context, removed the think tags, and placed the trace in front of the clean reply. The result is a leak in the model’s own words for that exact context, with the same shape as the sampled leaks. Where no reasoning-on sample was available, we instead placed a narration opener, taken from a leak the model had produced on another context of the same deployment, in front of the reply. We also used the reasoning-trace construction on the leak-prone contexts, as a second pair beside the sampled leak. In the final set, 1,504 rejected sides were sampled leaks, 1,547 were a reasoning trace placed in front of a clean reply, and 437 were a narration opener placed in front of a clean reply.

The leaks on the rejected side fall into eight classes, listed below. A rejected response often carries more than one class, so the counts add up to more than the number of pairs.

We also applied a few rules to keep the set balanced. No context contributed more than two pairs and no conversation more than three. Contexts were deduplicated by a hash of the system prompt plus the last six messages. We capped the share of any one deployment at a quarter of the set and the share of any one system prompt at 15%. Neither cap was reached. The largest deployment ended at 23% of the set, and the largest single system prompt at 1.6%, which is much smaller than the deployment share because a deployment’s prompt changes from version to version and carries fields that differ from one conversation to the next. The hold-out sets were frozen before any sampling and split by conversation, never by sample, so that no conversation appears in both training and evaluation. We also held out ten synthetic businesses in industries and languages that never appear in training, to measure the one thing prompting could not deliver: generalization to prompts the model has never seen.

Prompts are long. The median pair carries 22,400 prompt tokens, the 90th percentile 31,700, and the longest 59,600. We did not truncate. Capping the history at 32k tokens would have cut roughly one pair in ten, and these are mostly the long histories that trigger the leak in the first place.

Building synthetic training data from production conversations

The agent prompts, which are customized for each enterprise customer, stay in the training data by design. They never leave hardware we rent, and the adapter is ours alone. End-user data is a different matter, and none of it is in the training set. Every context the model was trained on is a synthetic conversation, built from a production conversation so that it keeps everything that makes the leak happen without carrying any of the user’s data.

Each synthetic conversation follows the same flow as the production conversation it was built from: the same system prompt, the same sequence of turns and the same tool calls at the same points. The user side is played by a simulated user, and every user response is generated, including every specific value the simulated user gives: names, phone numbers, account numbers, passcodes, emails, addresses and dates of birth. Every GLM-5.2 response is generated as well, so the assistant’s read-backs and tool calls follow from what the simulated user said rather than from the original conversation. The same generated values also fill the tool results and the user-specific fields that the deployment injects into the system prompt. A generated value is used consistently across the whole conversation, so a phone number that the simulated user gives, that the assistant reads back digit by digit and that a tool receives in normalized form is the same number in all three places. This consistency matters because these assistants repeat values back to the simulated user and pass them to tools, and a conversation whose values disagreed with each other would teach the model the wrong thing.

Leak-prone turns tend to sit deep in a conversation, after the user has been verified, so a synthetic conversation has to reproduce the whole history up to that point, not just the turn itself, for the leak trigger to be present. Both sides of every pair were sampled from these synthetic contexts, and the synthetic context is what the model was trained on.

Training

We serve an NVFP4-quantized checkpoint of GLM-5.2, and the serving engine compiles the routed experts into CUDA graphs. A LoRA adapter on the experts would have forced a re-quantization step for every candidate adapter. The checkpoint does, however, keep a specific set of modules in bf16: the attention projections on all 78 layers, the shared expert, and the three dense layers at the bottom of the network. That is 16.4B parameters, enough to change the model’s behaviour, and an adapter on exactly those modules merges into the served checkpoint with no quantization step. Only 33 GB of tensors change, and the engine configuration is byte-identical.

We wrote a small trainer for this rather than using an off-the-shelf one, because of the quantized base model and the very long prompts. It runs as a pipeline over the eight B200 GPUs, with one process per slice of layers, and each pair moves through the slices in turn. A small Triton kernel dequantizes the NVFP4 experts on the fly, so the base model runs from the served checkpoint directly. To fit a 60k-token pair into memory, the prompt is run once without gradient and its activations are cached; only the response tokens and the last eight prompt tokens are scored with gradient. This is the one deliberate approximation in the setup. Exact DPO would backpropagate through the whole prompt, which for 20k to 60k tokens per pair would not fit on one node.

Before training, we checked that the trainer sees exactly what the serving endpoint sees. The trainer renders the prompt locally from the raw messages, so a difference between our rendering and the endpoint’s would mean training against the wrong reference. We checked two things: the locally rendered prompt had to tokenize to exactly the token count the serving endpoint reports, and the base model’s log-probabilities for a sample served by the endpoint had to match the endpoint’s own log-probabilities within 0.02 per token. This check found two differences, and getting either of them wrong would have silently trained the model against the wrong reference. The first was a one-token difference in how the endpoint counts the prompt. The endpoint reported exactly one token more than our rendering on every one of the 2,943 contexts we compared, so this is a constant token that the serving engine adds at the start of the prompt, not a difference in how the conversation is rendered. We could not identify which token it was, and adding a guessed token made the log-probabilities worse, so we trained without it after confirming that the log-probabilities matched without it. The second difference was the end-of-turn token. The served model ends a response that makes a tool call with a different token from a response that is a plain reply. We confirmed which token the base model itself predicts at the end of each kind of response, and made the exporter end every tool-call response and every plain reply with the matching token.

The reward accuracy, which is the fraction of held-out pairs where the model prefers the chosen response, passes 95% by step 25. The reward margin keeps growing after that because the rejected responses keep getting less likely. The chosen responses do not move: their per-token loss stays at the base model’s level throughout training, thanks to the likelihood term, and their total log-probability changes by less than one nat while the rejected responses lose about 87.

Evaluation

We evaluated the fine-tuned model in four ways: a held-out suite of leak-prone contexts, replays of real conversations, public benchmarks, and live simulations on the production serving stack.

Held-out suite

The held-out suite is a set of contexts that we set aside before we sampled any training data. The split is by conversation, so no conversation in the suite has any turn in the training set. For each context we sampled the model eight times at temperature 1.0, so that leak rates are comparable between the base model and the fine-tuned model, plus once at the production sampling parameters. The samples were scored by the same labeler we used to build the pairs, and by a judge that did not know which model produced the sample.

The four sets cover different kinds of context. The reference set is the real leak turns that started the investigation. The recent set is the most recent real leaks at the time the training data was cut. The probe suite is a simulation suite of 223 leak-prone scenarios with long histories. The synthetic set is ten invented businesses in industries and languages that do not appear in the training data. The synthetic set is the generalization test that prompting could never pass, and the fine-tuned model brings its leak rate down as well.

This figure pools the three held-out sets that were built from leak-prone turns of real assistants, which are the reference, recent and probe sets described above, and leaves out the synthetic set. Every class went down and none went up. The remaining leaks are mostly language switches, in Spanish conversations where a single English clause survives in the reply.

We also checked on the same suite that behaviour outside the leaks did not regress. On ordinary turns the fine-tuned model takes the same action as the base model, and agrees with a reasoning-on reference just as often. The arguments it passes to tools with a fixed set of values are unchanged. Replies became slightly shorter. The model still switches language when the user asks, and it drifts into English on Spanish conversations less often than before.

Replays of real conversations

The strongest test is to replay real conversations. We took the logged requests from five deployments, 44,986 assistant turns over 4,359 conversations, and generated one reply per turn from each model at the production sampling parameters. Every reply flagged by the labeler was then reviewed by a second judge to separate real leaks from false positives such as a persona-style opening line. 88% of these turns come from conversations that are not in the training set.

The production logs row is what the base model actually said in the live conversations. The replayed row is the base model run again on the same requests, which is the right control because the served deployment is not deterministic. The fine-tuned model produced no real leaks of any class.

The fine-tuned model matches the production reply slightly less often than the base model does. A swing of one to two percent here is not surprising. The DPO model is a fine-tuned version of the base model, so some difference in its choices is expected, and the replay judge marks nearly every action mismatch as worse, for the base model too.

Public benchmarks

We also ran the model on public datasets, in both reasoning and non-reasoning mode, to make sure it had not regressed on general capability. GSM8K, run with reasoning on, checks that the reasoning mode we never trained on is intact. IFEval, run with reasoning off, checks instruction following in the mode we serve.

Live simulations

Finally, we ran whole-conversation simulation suites against the merged model on the production serving stack: 639 simulated conversations on one assistant’s core suite and 459 on another’s, graded by rubric and interleaved with a control run of the base model on the same serving stack. The fine-tuned model did not regress against the control run in either suite, and no leaks were spotted in the fine-tuned model.

Conclusion

GLM-5.2 in non-reasoning mode was leaking its reasoning into the AI assistant’s responses in about 1% of conversations. Prompt changes fixed each case we could see but never the next one, because the leak is a property of the model rather than of any one prompt. Turning reasoning on removed the leak but cost too much latency. A small LoRA adapter, trained with DPO on 3,526 pairs where the only difference between the two responses is the leak, with a likelihood term to keep the rest of the model’s behaviour in place, cut the leak rate sharply on the held-out suite and to zero on 44,986 replayed turns from real conversations, with no measurable regression on our behaviour checks, on public benchmarks or in live simulations. Most of the engineering effort went into the data: building pairs that differ only in the leak, and building every training context as a synthetic conversation so that no end-user data reached the model. The fine-tuned model is now serving production traffic.

All transcripts and preference pairs shown are synthetic reconstructions with invented businesses and users. Figures are drawn from the training logs and evaluation scorecards of the DPO run. Replay counts are judge-adjudicated real leaks; base-model rates are from replaying the same requests, not from the live logs, except where labelled.