Ekka: Automated Diagnosis of Silent Errors in LLM Inference

Yile Gu, Zhen Zhang, Shaowei Zhu, Xinwei Fu, Jun Wu, Yida Wang, Baris Kasikci

June 29, 2026

Read the paper (arXiv) · ICML 2026 page

TL;DR: LLM serving frameworks ship fast, and sometimes ship silent errors — bugs that degrade output quality without raising any error. Ekka diagnoses them automatically by treating diagnosis as differential debugging: it aligns and compares the intermediate execution states of a buggy framework against a trusted reference, and pinpoints the component where they diverge. On 17 real-world bugs from vLLM and SGLang, Ekka hits 80% pass@1 diagnosis accuracy at ~$30/case, and has already found 4 new bugs confirmed by developers.

Overview

Modern LLM serving frameworks like vLLM and SGLang are remarkable pieces of AI infra engineering: paged attention, radix attention, custom CUDA kernels, continuous batching, and a steady stream of new optimizations. But that velocity has a cost. The more complex and fast-moving the stack, the easier it is to introduce a silent error: a bug where the framework keeps running, returns a perfectly well-formed response, and quietly produces the wrong answer.

Unlike a crash or an assertion failure, a silent error gives you no stack trace and no error message: just output that looks plausible but is subtly, or badly, wrong. Diagnosing these bugs is notoriously hard because of the huge semantic gap between the symptom, say a dropped benchmark score, and the root cause, which might be a misconfigured attention window buried deep in the model stack. Today, tracking one down is a manual, sometimes months-long slog of toggling configs, dumping tensors, and comparing against a reference implementation by hand.

Ekka is an automated diagnosis system that takes on exactly this problem.

A motivating bug

Here is a real silent error from vLLM (#17689). Serving Gemma 3 with vLLM, accuracy on the HellaSwag benchmark dropped by nearly 30% compared to HuggingFace — with no runtime error or warning. Both frameworks were given the same GSM8K prompt:

Prompt

Rory orders 2 subs for $7.50 each, 2 bags of chips for $1.50 each and 2 cookies for $1.00 each for delivery. There’s a 20% delivery fee and a $5.00 tip. What will her order cost?

and produced very different answers:

HuggingFace (reference) → Final Answer: $29 ✓

Subs: 2 × $7.50 = $15. Chips: 2 × $1.50 = $3. Cookies: 2 × $1.00 = $2. Subtotal: 15 + 3 + 2 = 20. Delivery fee: 20% of 20 = 4. Total: $29

vLLM (buggy) → Final Answer: $100 ✗

Subs: 2 × $7.50 = $15. Chips: 2 × $1.50 = $3. Cookies: 2 × $1.00 = $2. The total cost of the food is 15 + 15 + 15 = 45. Delivery fee: 45 × 0.20 = 9. Total: $100

The generation finishes without error, but the math silently goes off the rails. It took developers three months to trace this to the root cause: a misconfiguration that incorrectly enforced sliding-window attention on all layers instead of 5 out of every 6. From the output alone, there is almost no signal pointing to where in the computation that error originated.

What do silent errors actually look like?

To ground the design, we conducted an empirical study of 90 real-world silent errors collected from vLLM and SGLang GitHub issues. We used the 70 closed issues to understand root causes and diagnosis workflows, and kept the open ones to evaluate Ekka. Three findings shaped Ekka’s design.

Figure 1. Symptom distribution (left) and root-cause distribution (right) of silent errors in vLLM and SGLang.

1. The symptom rarely points to the cause. The most common symptom by far is accuracy regression at 43.8%: superficially valid text that scores worse on benchmarks. The rest are bogus output such as gibberish and repetition loops, plus inconsistent output. A single degraded answer tells you almost nothing about where the computation went wrong.

2. Root causes are spread across the entire stack. Framework implementation at 30.6%, model implementation at 25.5%, and kernel backends at 24.5% all contribute, and about 19.4% of issues are pure numerical precision — floating-point instability with no logical defect at all. Crucially, roughly half of all silent errors originate inside the model stack of model and kernel implementation, so a useful diagnosis tool cannot treat the model as a black box. It has to be model-aware.

3. Developers already do differential debugging — by hand. The most common diagnosis actions are configuration toggling and minimal reproduction, each in over 60% of issues, and comparing against another framework appears in about half of cases, often paired with manually dumping and inspecting activations. This is exactly the workflow Ekka automates.

Figure 2. Diagnosis actions developers perform when resolving silent errors. Comparing against a reference framework and inspecting activations are common — but manual and laborious.

The catch: manually comparing tensors across two frameworks is painful. Optimized serving engines use completely different internal module structures, naming, memory layouts, and fused operations than a reference model. A direct tensor comparison is usually impossible without significant manual alignment effort. Ekka’s job is to make that alignment automatic.

How Ekka works

Given the model, an example prompt, the two frameworks’ configs, and which is the buggy target vs. the trusted reference, Ekka outputs a ranked report of the components most likely responsible for the divergence. It first collects context — it parses both codebases into a code index, compresses each model’s architecture into a Model Tree, and uses PyTorch forward hooks to record activations and the call sequence while reproducing the bug — then runs an agentic diagnosis in three steps.

Figure 3. Ekka’s architecture. The top row collects static and dynamic context; the bottom row runs the three-step agentic diagnosis, backed by a toolset and a knowledge base, and emits a ranked diagnosis report.

1. Component Mapping finds semantically equivalent components across the two frameworks. This is harder than name matching: vLLM fuses Q/K/V into one QKVProjection where HuggingFace keeps three modules, and even matching names can diverge (HuggingFace’s RotaryEmbedding.forward computes cos/sin, while vLLM’s applies the rotation). Working from the compressed Model Tree, the agent maps nodes and uses a get_class_definition tool to inspect implementations when names are ambiguous. Mappings are built incrementally and checked by a validator that removes confirmed pairs and requires every component to be either mapped or explained.

2. Activation Alignment makes mapped components actually comparable, since their outputs can differ in shape, dtype, layout, or ordering. Aligning HuggingFace’s separate Q/K/V with vLLM’s fused output, for instance, means concatenating three tensors and dropping a batch dimension. Ekka frames this as code generation: the agent writes small Python postprocessing functions, with a fixed template handling the boilerplate. Helper tools (get_tensor_sum, infer_tensor_match) help match tensors, a knowledge base of validated alignments from known-correct models provides in-context examples, and a validator checks the generated code.

3. Error Analysis separates real bugs from numerical noise. Because models run in low precision such as BF16 or FP8, floating-point error accumulates across layers and a fixed L2 threshold doesn’t work. Ekka uses a robust error ratio that normalizes the target’s deviation by the reference’s own low-vs-full-precision gap:

where X_T and X_R are the low-precision outputs from the target and reference, and Y_R is the reference’s full-precision FP32 output. A correct component stays near 1; a buggy one spikes well above it. Since errors propagate downstream, Ekka walks the call sequence and applies change-point analysis to flag the earliest component whose error ratio stays persistently elevated — a sustained shift, not a one-off spike from alignment noise — then reports the ranked suspects with reasoning.

Does it work?

We built a benchmark of 17 real-world silent errors from vLLM and SGLang — 12 from our bug study and 5 reported afterward — packaged as Docker containers with runnable reproduction scripts. The bugs span a wide range of models, including Llama 4, Gemma 3, Qwen 3, ERNIE 4.5, and MambaMixer 2, and root causes, from RMSNorm dimension mismatches and incorrect tensor-parallel sharding to FlashInfer kernel bugs and BF16 accumulation errors. Each case is labeled with the ground-truth buggy component.

We compare against two state-of-the-art software-engineering agents — OpenCode and Mini-SWE-Agent — each given the original bug report, the codebase, the model path, and a reproduction script. All systems use Claude Sonnet 4.5 as the backend and output their top-3 suspected components, scored with the pass@$k$ metric.

Figure 4. End-to-end diagnosis accuracy. Ekka reaches 80% pass@1 and 88% pass@5, well ahead of general-purpose coding agents.

Ekka reaches 80% pass@1 and 88% pass@5, a 28% pass@1 improvement over Mini-SWE-Agent and 34% over OpenCode. The takeaway: general-purpose coding agents lack the domain-specific scaffolding — model-aware mapping, tensor alignment, precision-robust error analysis — needed to navigate the semantic gap, and that scaffolding is exactly what drives Ekka’s accuracy.

Ablations confirm each piece matters. For component mapping, adding the validation loop and the code-inspection tool lifts mapping accuracy from ~60% to ~90%, and raises end-to-end pass@1 from 0.47 without validation and 0.67 without the tool up to 0.84. For activation alignment, the code-validation loop plus helper tools and knowledge base push alignment accuracy from ~7% to ~88%, and pass@1 drops to 0.39 once tools and the knowledge base are removed. Diagnosis accuracy is also stable across error thresholds from 1.5 to 2.5: correct components cluster well below 1.5 across both BF16 and FP8, while ground-truth buggy components range from 3.1 all the way up to 1093.

It’s cheap. Using Claude Sonnet 4.5 without prompt caching, Ekka averages ~4.0M input and ~304K output tokens per case — under $30 per diagnosis even in the worst case. That makes it practical to fold into a serving framework’s existing testing workflow.

Finding new bugs in the wild

The real test is whether Ekka generalizes beyond a curated benchmark. Over the course of a month, Ekka diagnosed 4 new silent errors in vLLM and SGLang — all confirmed by the developers:

Issue	Symptom	Root cause
vLLM-28539	Gemma 3 accuracy regression on GSM8K	BOS token not applied
vLLM-30777	whisper-large-v3 produces incorrect output	CUDA graph capture
SGLang-13044	Qwen 2 output inconsistent with HF	FlashInfer kernel
SGLang-16132	Qwen-Image generates reduced-quality images	Normalization after denoising

These span text, audio, and image models — including SGLang-16132, a diffusion-model bug well outside the autoregressive text path Ekka was originally designed around — which speaks to the generality of the differential-debugging approach.

Takeaways and what’s next

LLM serving frameworks will keep moving fast, and silent errors are an inevitable side effect. Ekka shows that the manual, hours-to-months differential-debugging workflow developers already rely on can be automated: give an LLM agent the right scaffolding to map components across frameworks, align their intermediate activations, and reason about divergence with a precision- and propagation-aware error metric, and it diagnoses silent errors at 80% pass@1 and roughly $30 per case, with 4 confirmed new bugs and counting.

Ekka localizes at the PyTorch module level today; pushing to finer function- or line-level granularity, supporting non-PyTorch stacks such as JAX/Flax, and driving cost down with smaller backbones and stronger caching are the natural next steps. The core pipeline of component mapping, activation alignment, and error analysis stays the same.