We rewrote one weight matrix to teach a 0.8B model conversation classification

Claude Fable 5.1 took Qwen3.5-0.8B, a small open model, and doubled its accuracy on a classification task without fine-tuning it: no gradient steps, no adapters, no new parameters. Fable measured the model’s own activations and rewrote one weight matrix in closed form. Accuracy on held-out conversations went from 0.41 to 0.83.

The task

Fable was tasked with classifying customer-support conversations into predefined scenarios. For example, if a customer contacts a retailer to return a purchased item, the model should classify the conversation as “Return request”. If the customer is asking about the whereabouts of a parcel, the correct label is “Order status and delivery tracking”.

The system prompt provides a list of scenario names, each accompanied by a one-line description, followed by the conversation transcript. The model must identify the scenario that best matches the customer’s intent and respond with its exact name.

The task spans five domains: retail, telecom, banking, airlines, and SaaS, with 12 scenarios per domain (60 in total). We considered both voice-style and chat transcripts of varying lengths, as well as challenging cases in which the customer’s underlying intent only becomes clear late in the conversation.

The data

We considered a synthetic dataset of 3,600 conversations, each labelled with one scenario, split three ways:

• 1,800 (fit): the only conversations Fable used to compute the change.
• 600 (val): used by Fable to evaluate each iteration of the change and select the best-performing version.
• 1,200 (test): untouched until the end. Every number below comes from this set.

What Fable found inside the model

A quick reminder of how the model produces its answer. The prompt is split into tokens. For every token position the model keeps a vector of 1,024 numbers, and each of its 24 layers reads that vector and adds something to it. At the last position of the prompt, the vector that comes out of layer 24 goes through a final norm and the unembedding matrix, which gives a score for every token in the vocabulary. The highest-scoring token is the first token of the answer, so for this task the model’s answer is whichever of the 12 listed scenario names scores highest.

Step 1: record what the model computes. Fable ran the unchanged model on the 1,800 fit conversations. At the answer position of each one it saved the vector coming out of every layer (24 vectors per conversation) and the 3,584 hidden activations of the last layer’s MLP, which are the numbers the down-projection matrix multiplies.

Step 2: check whether the scenario is already in those vectors. For this Fable built a nearest-prototype probe. A prototype is just the average vector of one scenario. Take the 30 fit conversations labelled “Return request”, take the vector each of them produced after layer 24, and average those 30 vectors. The result is a single vector that stands for what a “Return request” conversation typically looks like inside the model at that layer. Doing the same for the other 11 scenarios of the domain gives 12 prototypes. To classify a conversation the probe has never seen, take its vector after the same layer, measure its distance to each of the 12 prototypes, and answer with the nearest one. Nothing is trained: the probe consists of 12 averages and nothing else.

What the probe showed. On the 600 val conversations the probe is right 75% of the time with the vector after layer 24, and 79% with the vector after layer 19. The model’s own answer, computed from the very same layer-24 vector, is right only 41% of the time.

Step 3: locate the fault. The model has already worked out the scenario, and that knowledge is still present in its final vector. What goes wrong is the last step, which turns the final vector into a scenario name: it leans toward a few names in each domain. On the airline test conversations, for instance, 184 of the base model’s 240 answers are “Flight cancelled by the airline” or “Flight change or rebooking”, although each of the 12 scenarios appears only 20 times.

Why rewrite the MLP rather than the unembedding? The unembedding is a fixed linear readout: it scores each vocabulary token by how far the final vector points along that token’s direction. The probe tells us the scenario is in the final vector, but not along the directions the unembedding reads for the first tokens of the scenario names; the vector points partly toward a few habitual names whatever the conversation. There are two ways to repair that: turn the readout toward where the information is, or move the vector toward where the readout looks. Fable chose the second. The unembedding is shared by every token the model ever writes, in every prompt and at every position, and in this model it is tied to the input embeddings as well, so any change there leaks into everything the model does. The last layer’s MLP, in contrast, is a context-specific map. Its 3,584 hidden activations are computed from this particular conversation at the answer position, and they already identify the scenario. Fable checked this with the same nearest-prototype probe, run on the activations instead of the final vector: using all 3,584 of them it is right 74% of the time. Each activation is one number, so Fable also ranked them by how differently they respond to one scenario than to the other scenarios of the same domain, kept only the 32 most selective, and ran the probe on those 32 numbers alone: still right 69% of the time.

The down-projection matrix W decides what gets added to the final vector for each pattern of those activations. Rewriting W therefore lets Fable steer the final vector toward the correct name’s direction only when the conversation looks like that scenario, and leave it alone otherwise, while the norm and the unembedding stay exactly as trained. A further advantage is that nothing comes after layer 24, so the effect of a change to W on the scores can be computed exactly instead of estimated.

The change

1. Measure. Run the unchanged model on the 1,800 fit conversations and save, at the answer position, each one’s 3,584 MLP activations (the input to W) and its final vector (what the layer hands to the final norm and unembedding).
2. Decide where each final vector should move. For each conversation, compute the smallest shift of the final vector that would put the correct scenario name ahead of every other listed name by a fixed margin in the scores. Conversations the model already gets right by that margin need no shift.
3. Find one change to W that produces those shifts. The MLP output is W applied to the activations, so changing W by dW adds dW applied to those same activations. Fable solved for the single dW that, applied to each conversation’s activations, comes closest to producing that conversation’s shift, for all 1,800 conversations at once, with a penalty for changing W’s output on ordinary text, so that the model’s behaviour elsewhere stays put. This is a ridge regression, which has an exact closed-form answer: one solve of a 3,584 by 3,584 linear system gives the best dW in a single step, where fine-tuning would take thousands of gradient steps with a learning rate, batch size and epoch count to tune.
4. Apply and repeat. Replace W by W + dW. Because layer 24 is the last layer, the new final vector is known exactly (the old one plus dW applied to the activations); nothing downstream has to be re-estimated. Fable recomputed the scores and margins with the new matrix and repeated steps 2 to 4, each round correcting what the previous one left short. The final version used three rounds, so the whole change came out of three linear solves.
5. Choose and verify. The procedure has three knobs: how far ahead the correct name must be in step 2 (the margin size), how strongly step 3 resists changing W’s behaviour on other text (the penalty strength), and how many times steps 2 to 4 are repeated (the number of rounds). Fable tried a handful of settings for each, computed the change from the fit conversations for every combination, and measured each result’s accuracy on the 600 val conversations. The combination with the best val accuracy was kept; the test conversations played no part in this choice. The 1,200 test conversations were then run once, with the new matrix loaded into the model, to produce the numbers in the next section.

Steps 2 to 4 are cheap: given the activations and final vectors saved in step 1, computing the change for one combination of settings (all three rounds) takes under a minute on a laptop. The expensive parts are step 1 and the final verification, which need the model to run on a GPU; a complete refit, from recording the fit conversations to the test numbers, takes about 45 minutes.

Results on the unseen test set

What each row measures:

• Pick from list, first two tokens. The model is not allowed to write freely. For each conversation Fable reads off the score the model gives to each of the 12 listed names, using the first two tokens of each name, which is enough to tell every name apart, and counts the conversation correct when the right name has the highest score. This isolates the question the edit was meant to settle: does the final vector now point toward the right name?
• Pick from list, full name. The same, except that the score of a name is summed over all of its tokens, so long names are judged on their complete wording. It agrees with the first row, so the result does not depend on how the names are scored.
• Free generation, exact name. The model generates its answer on its own, exactly as it would in production, and is counted correct only if it writes the right name word for word. This is the number that matters in deployment.
• General-English perplexity. Perplexity measures how surprised the model is by ordinary English text (here a standard held-out set of Wikipedia articles); lower means the model predicts the text better. The edited model’s perplexity rises from 17.4 to 19.6, a 12% increase, so it is slightly worse at general English. This is expected: the rewritten matrix now spends part of its output on steering conversations toward scenario names, and the penalty in step 3 limits its effect on the English capability. We are fine with the loss, because the goal was a classifier, not a general-purpose assistant.

Conclusion

Everything above came out of a small budget: 1,800 training conversations, a single rented GPU for under an hour per refit, a laptop for the solve itself, and about 4 hours of Fable’s time from the first measurement to a verified checkpoint. With that, Fable found where the model keeps what it knows, worked out why it fails to say it, and rewrote one matrix to fix it, doubling accuracy without a single gradient step. Nothing in the approach is tied to this model, this task or this scale. If this is what Fable can do with one GPU and a few thousand examples, it is worth imagining what it could do with a large model, real data and the compute of an ordinary training run.