Our Approach: A Hot-Swappable LoRA System Built on Qwen

For our base model, we chose Qwen3-8B, a modern ~8B-parameter open-weight instruction model. There were two primary reasons why Qwen3-8B was the ideal choice:

• Capacity and latency balance: 8B parameters are enough to handle long contexts and subtle label boundaries. It’s also small enough to hit aggressive real-time benchmarks by optimizing prefill.

• Ecosystem: Qwen3-8B has a great tokenizer and instruction obedience out of the box. It also supports fine-tuning, quantization, and modern inference stacks.

We chose to keep Qwen3-8B as a frozen base; we do not fine-tune the model directly, keeping all of the original weights. Instead, we use lightweight LoRA (low-rank adaptation) adapters to create an efficient low-dimensional representation of fine-tuning updates. If you are interested in the details surrounding this approach, we highly recommend this Thinking Machines post.

There are many benefits of using LoRA adapters that closely align with our design criteria:

• Isolated by design: Each adapter has its own set of weights, each tuned to a single customer success policy. By changing (or breaking) one adapter, we don’t affect the behavior of another. We could have hundreds of available adapters at scale.

• Faster: Compared to full fine-tuning, LoRA tuning takes hours rather than days due to the significantly smaller vector space. LoRA is similar to tailoring a standard suit with a few precise alterations instead of sewing an entirely new one.

• Footprint: Adapters are very small. The significantly smaller memory footprint, often dipping under 175MB per adapter, allows us to load multiple in parallel.

• Safer practices: Adapters can easily be rolled back if metrics are moving in the wrong direction. Additionally, they could be versioned, cached, and hot-swapped. This allows us to easily A/B test, loading both in parallel.

Our Specific Classification Problem

We trained our classifier LLM on a few anonymized data sources pulled from Giga:

• Long-form interaction logs: These are logs detailing conversations between the AI voice agent and our customers’ customers. Our classifier LLM served as context management, ensuring agents had the right guidelines to answer questions without hallucinating.

• System messages: System messages are the prompts for each LLM request. This is a constant prefix for every prompt.

• Policy/configuration snippets: Policies are instructions specific to a customer or use case that the voice agent needs to follow. For example, an airliner might have a policy for issuing ticket changes.

• Templated edge cases for tricky boundaries: For most customer service operations, there are repeatable edge cases that could be over-sampled during training to teach the adapter the fine-grained distinctions.

For the actual ingestion, we slotted our data into a fairly simple schema:

{
	context: string, //long-horizon background, including all metadata + configs
	input_window: string, //a short snippet of text we're classifying
	labels: [string] //an allowed subset of labels from a fixed catalog 
}

Additionally, to protect the quality of our data, we:

• Performed inter-rater agreement checks, where we ensured multiple human raters would assign the same labels

• Created label-consistency tests for near-duplicate contexts, where we ensured that near-duplicate or duplicate data was marked with the same labels

• Scanned for leakage between train and validation data, where we ensured that training and validation data were discrete sets to avoid overfitting.

Training Strategy

Our training strategy was to use a teacher model, where a larger LLM with high-precision prompts generates labels in a structured JSON. Then, humans reviewed certain things, such as label frequency, often confused label pairs, and other edge cases. We performed adversarial training to target near-miss phrasing, code-switching, input channel quirks, and formatting noise. All fine-tuning ran on a single H200 GPU hosted on Runpod.

There were a few other unsuccessful trials. We originally tried Together AI, but they lacked the flexibility to modify chat templates (thereby making it impossible to disable thinking). We also tried directly training FP8 weights on a Qwen3-8B model, but this proved to be significantly inefficient and just comparable on accuracy to LoRA adapters. Additionally, we tried building on a Qwen3 30B A3B model, but we also achieved only comparable accuracy but with worse latency due to its Mixture of Experts (MoE) architecture.