r9k: a from-scratch inference engine for Qwen3.8-27B on two Radeon AI PRO R9700.

Qwen3.8-27B is a hybrid model: 48 Gated DeltaNet layers and 16 full-attention layers, 27 billion parameters, with FP8 E4M3 weights in 128 × 128 scale blocks. On a pair of R9700 cards each forward pass reads about 13.5 GB of weights per GPU, so single-request decoding is bound by memory bandwidth and by how many tokens each pass can produce.

r9k keeps the whole decode step on the GPU as one HIP graph, splits the model over the two cards with its own PCIe all-reduce, verifies several draft tokens per pass with lossless speculative decoding, and batches up to eight requests. Prefill runs through a custom block-scaled FP8 GEMM. The engine is about 8,100 lines of C++ and HIP with 70 GPU kernels, plus an OpenAI-compatible HTTP server.

Against a tuned vLLM 0.31 deployment on the same machine and harness, r9k v8 decodes one request 1.9× faster on a 2K prompt and 2.6× faster on an 8K code prompt, reaches first token 16% sooner on a 2K prompt, and produces 1.45–1.5× the total throughput at four and eight concurrent requests.

v9, serving since 26 October, adds lookup drafting from the conversation's own history with 16-row verification, and a host-memory prefix cache. Edits that reproduce most of their input (renaming a function, proofreading, adding a field to every JSON record) decode 63–73% faster than v8, at 14–15 tokens per step, and follow-up turns of a conversation reach their first token in about 95 ms instead of 760–790 ms. The standard harness stays within 1% of v8, with identical outputs.

The engine described here serves Qwen3.8-27B to the public through unassuming.ai: a browser chat and an OpenAI-compatible API. Both need an access key, which you can request from contact@unassuming.ai.

Benchmark method

All rows in the next section were measured with one harness on this machine. Each request streams over HTTP from a local client with fixed token IDs, temperature=0, ignore_eos and 256 output tokens; every scenario gets one warm-up round and three measured rounds, and the medians are reported. The scenarios are a 2,048-token English prompt with one and with four concurrent requests, an 8,192-token code prompt with one request, and a 2,048-token code prompt with eight concurrent requests. Decode speed is output tokens after the first divided by the time after the first token. Total throughput counts all output tokens over the wall time, including prefill. Prefill speed is prompt length divided by time to first token.

The sampled variant of the harness differs only in its sampling parameters (temperature 1.0, top-p 0.95, top-k 20, the model's own defaults).

Greedy decoding, same harness and hardware for every row. Decode columns are tokens per second for one request; total columns are output tokens per second across all concurrent requests.

vLLM was tuned during an earlier optimization pass on this machine. The DFlash2 row for vLLM includes a fix that keeps the drafter's convolution projections in BF16; without it, acceptance falls to nearly zero. The Radiance row uses its built-in MTP head, which is the configuration comparable to r9k v7. r9k v7 and v8 differ in drafter and prefill path; both run the same FP8 target weights.

Figures reported by others for the same GPU, with their own models, prompts and harnesses. They describe the state of the ecosystem but are not like-for-like with the table above: prompt lengths, output lengths, model versions, quantization and GPU counts differ, as each row states. The last card shows r9k results from our records with the closest workload we have measured: six tasks (a 2K English prompt and five short chat prompts) and 256 output tokens, run in-process with the engine's command-line driver and averaged over two runs.

C=1 figures are output tokens per second for one request; C=4 and C=8 are aggregate output throughput; TTFT is for one request. A dash means the source did not report the value. Puget Systems tested Qwen3.6-27B, the predecessor model with the same architecture class.

Each version was benchmarked with the same harness when it was completed. v1 through v4 decoded one request at a time, so their concurrent totals reflect queued requests. Continuous batching arrived in v5.

v1–v8 were measured directly against the engine. v9 was measured on 26 October through the production gateway, under the memory limits that let two smaller models share the GPUs; v8 measured the same way on the same day gave 441 ms, 129.7, 198.4 and 277.9, so the differences from the v8 row come from the newer setup, not from v9. The charts stop at v8 for that reason.

Chat deployments sample from the model rather than decode greedily. Speculative decoding usually loses acceptance under sampling, so we measure this path separately. In v8 the drafter samples its proposals from its own distribution and the engine accepts each with probability min(1, p/q), resampling from the residual on rejection; the output distribution is unchanged.

By v8, edits that reproduce most of their input, such as renaming a function and returning the file, were already accepting 7.5–8 tokens per step: the ceiling of an eight-row verification block. v9 doubles the block to 16 rows and fills the extra rows from the sequence's own history. It also keeps snapshots of processed prompts in host memory, so the next turn of a conversation computes only its new tokens. v9 has served production traffic since 26 October 2026.

One request, thinking off, greedy, up to 1,536 output tokens; prompts of 0.9K–2.5K tokens. Sampled at temperature 0.7, top-p 0.8, top-k 20, the first three tasks run at the same speed. The control task writes new text, so lookup rarely finds a match there.

Each conversation has a reference document in its system prompt and three questions, with up to 220 output tokens per turn. A cache miss costs the same as in v8.

The harness from the comparison above (greedy, 256 output tokens, medians of three rounds), run through the production gateway on 26 October for both versions. All 66 outputs, 42 from these scenarios and 24 from eight natural-task prompts, are identical to v8 with the same tokens per step; the natural tasks change by −0.1% to −1.0%. The remaining cost is the lookup kernel, about 0.1 ms per step, and the occasional wide step that is not accepted.

Measured effects on this machine. Each change was benchmarked in isolation before being kept.

• A float32 CPU reference of the full model checks every kernel change. With BF16-activation prefill, greedy output matches the reference token for token on the three reference prompts (40 / 40, 64 / 64 and 64 / 64 tokens), and the DFlash2 drafter's first block matches its own float32 reference exactly.
• Speculative decoding is lossless: greedy output with MTP and with DFlash2 is identical, and so is output with batched and unbatched prefill and with either all-reduce path.
• Sampling was checked statistically: first-token distributions against exact distributions from full logits (χ² p = 0.15–0.89 across temperature, top-p, min-p and repetition settings), and speculative against non-speculative sampling over 2,000 samples each (two-sample tests on tokens 2–6 and their joint prefixes, p = 0.17–0.97).
• Both GPUs draw identical counter-based random numbers, so every sampling and acceptance decision agrees across ranks without extra communication.
• v9 requests that miss the prefix cache match v8 word for word: 42 of 42 standard-harness outputs, 24 of 24 natural-task outputs and the cold turns of the multi-turn test. The lookup kernel matches a host reference implementation on 2,400 cases.
• On a cache hit only the rest of the prompt is prefilled, as a separate group, and GEMM rounding depends on the row count; one turn of the multi-turn test diverged from the cold computation at a point where two continuations were almost equally likely. Wide verification evaluates the output head with fp16 activations, so greedy output can differ at near-ties when it is used, as it already could in v8 with two or more concurrent requests.

• Snapshots at message boundaries, so conversations that share a system prompt also share its cache entry.
• Wide verification at three and four concurrent requests.
• Long contexts: an FP8 KV cache and more attention splits. At 61K tokens of context a decode step takes 6.8 ms (about 18%) longer than at 2K.
• Chunked Gated DeltaNet prefill. The prefill recurrence is serial per token today, about 1.5–2.3 ms per layer for a 2K prompt; a chunked delta-rule formulation turns most of it into matrix products.
• Mixed prefill and decode batches, so running requests keep decoding while new prompts are processed.
• Vendor block-scaled FP8 GEMMs for prefill once hipBLASLt supports 128-element scale modes on gfx12.
• More models and more ROCm targets on the same engine structure.