A deliberately narrow, self-contained C inference engine — not a generic GGUF runner, not a wrapper around another runtime. Built for one model, tested against official logits, and ready for coding agents.
137 commits over 11 days — from a bare initial release to a polished multi-backend inference engine with tool streaming, disk KV cache, eval harness, and agent APIs.
Salvatore Sanfilippo (antirez) releases the first version of ds4 — a standalone C inference engine for DeepSeek V4 Flash. The project begins with a CPU reference backend, Metal GPU support, and a basic interactive CLI.
Fixes routing of OpenAI thinking/reasoning content
through the streaming path so agent clients receive
structured reasoning_content events
instead of raw text.
Adds visible curl progress bars while
downloading model GGUFs, making the 80+ GB download
experience less opaque.
Adds backend-independent token dump debugging — a crucial tool for aligning local inference output with official DeepSeek logits during regression testing.
Critical innovation. DeepSeek models emit tool calls as DSML text, but agent clients send back JSON tool-call objects. If the server re-rendered those objects differently, the KV cache prefix would mismatch and force a full rebuild. This series adds exact replay memory backed by radix trees: every tool call's original DSML bytes are stored and reused verbatim on the next turn. The map survives in KV cache files across server restarts.
Fixes a session cache miss on every turn when operating in toolless thinking mode — a subtle KV checkpoint boundary bug that cost performance in agent sessions.
Uses rendered text prefixes (not just token IDs) to decide KV cache reuse — makes the cache hit rate dramatically better when prompt structures change slightly between turns.
Major milestone. NVIDIA CUDA backend joins Metal. Includes CUDA-specific kernels, memory management, and graph scheduling. The DGX Spark (NVIDIA's local AI machine) becomes a first-class target.
Refactors the backend abstraction layer — Metal, CUDA, and CPU now share a unified interface. Makes adding new backends (like ROCm) tractable.
From "ds4" to "DwarfStar 4" — a proper name that reflects the project's identity as a compact but powerful inference engine, like a dwarf star that burns hot and dense.
Documents that ds4 runs well with 96 GB of system memory — enabling MacBook Pro users with the M3 Max 96 GB config to run the 2-bit quant at reasonable context windows.
Benchmarks reorganized: renamed to
speed-bench for clarity, CSV files
unified into a single directory, and a Python SVG
rendering script added for visual benchmark charts.
Community benchmark results for the Mac Studio M2 Ultra with 192 GB — showing 84 t/s prefill and 36 t/s generation on the 2-bit quant.
/v1/responses)
bb697b3
Major feature. Adds the OpenAI
Responses API endpoint — the preferred interface for
Codex CLI. Supports input,
instructions, tools,
reasoning, and streaming. Continuations
bound to live state when possible.
Guards CUDA q8 fp16 cache memory use, allocates
graph tensors in device memory, zeroes compressor
state with memset. Core CUDA
performance and correctness work.
Imports CUDA long-context fixes to handle the 1M token context window correctly. Reduces long-context prefill slope for faster chunked processing. Adds a formal long-context regression test.
Adds the gguf-tools suite: offline GGUF
generation, importance matrix collection,
quantization, and quality checks. Enables the
community to build their own GGUFs.
MoE down-projection block16 path made opt-in then cleaned up. Live MTP correctness fixes kept for the speculative decoding path. CUDA q8 fp16 cache restored for large-memory devices.
Broader and better calibration prompts for the importance matrix — directly improves quantization quality for the imatrix-tuned GGUFs.
CUDA backend now usable for quality scoring in the gguf-tools pipeline — enables GPU-accelerated validation of custom GGUFs against official DeepSeek continuations.
Adds a long-context story fact-recall regression test — the model must retrieve spelled-out person-number assignments from a long prose prompt. Essential for validating the 1M context window.
Harden the Responses API tool replay: tighter
tool_search replay, fix tool checkpoint
cache reuse, fix live continuation. Makes Codex CLI
sessions robust across turns.
Metal backend covers q4 expert tensors in model views. Skips tool checkpoint canonicalization for exact DSML replay. The Responses API branch is merged into main.
Fixes the order in which Anthropic tool replay parses DSML blocks — critical for correct multi-tool call handling in Claude Code sessions.
CUDA backend uses managed memory for the KV cache when contexts are huge (>100k tokens) — enables the 1M context window on GPU without exhausting device memory.
Major feature. The
ds4-eval benchmark harness is added —
questions drawn from GPQA Diamond, SuperGPQA, AIME
2025, and COMPSEC. Model card synopsis document
added. KV disk cache eviction thrash fixed.
OpenAI cached token counts kept read-only in API responses. KV cache hit decay made the default eviction strategy. Opt-in CORS support for browser-based clients.
Benchmark items cleaned up, prompts audited for quality, and ds4-eval controls improved (auto-sizing context, reporting polished).
Auto-sizing added to the eval context window. COMPSEC category introduced — 17 single-function C/C++ security reasoning items derived from public CVE writeups. Localization cases refined for clarity.
Default sampling changed to
min-p filtering (relative probability,
not nucleus mass). Tool calls emitted inside
thinking blocks are ignored. Server gets a
--chdir working directory option.
Applies correctness fixes to the Flash computation graph — addresses numerical accuracy issues in the forward pass that could cause subtle output divergence from the reference implementation.
Fixes RoPE (Rotary Position Embedding) position indices during compressed prefill on CUDA — a correctness bug where position assignments were incorrect for compressed attention rows, causing degraded output quality on long contexts.
How ds4.c is structured — a single-model inference engine with three backends and a narrow public API.
GGUF files are memory-mapped, not eagerly copied. The 80+ GB model sits on disk and is accessed via page faults. This keeps startup fast and memory usage proportional to active working set.
DeepSeek V4 uses CSA + HCA: a 128-token sliding window (raw), ratio-4 compressed rows with indexer (even layers, top-k=512), and ratio-128 compressed rows (odd layers). DS4 implements this mixed-attention computation natively.
The KV cache is designed for disk persistence, not just
RAM. --kv-disk-dir stores compressed KV
state as files on SSD. The server can cold-start from
disk checkpoints, making long agent sessions practical
without keeping everything in memory.
Metal (macOS, primary target), CUDA (NVIDIA, DGX Spark focus), CPU (reference/diagnostics). Unified backend interface with backend-specific graph scheduling and kernel implementations.
Exact replay memory backed by radix trees: every tool call's original DSML bytes are stored and reused verbatim. Falls back to deterministic canonicalization. The map survives in KV cache files across server restarts.
OpenAI Chat Completions
(/v1/chat/completions), OpenAI Responses
(/v1/responses), and Anthropic Messages
(/v1/messages). All support SSE streaming,
tool calls, and thinking mode.
What makes ds4.c unique — and why it exists as a standalone engine rather than a GGML plugin.
Not a generic GGUF runner. Every optimization, kernel, and memory policy is tailored for DeepSeek V4 Flash's specific tensor layout, quantization mix, and compressed attention pattern. This narrowness is the source of its speed and correctness.
The KV cache is a first-class disk citizen, not a RAM-only structure. Compressed KV state is stored as files on fast SSDs. This changes the assumption that long-context inference requires enormous RAM — the disk absorbs the history, and only the active window lives in memory.
The model emits tool calls as DSML text, but agents send back JSON. Without exact replay, every turn would force a full KV cache rebuild. DS4 stores the original DSML bytes verbatim and replays them — making multi-turn agent sessions efficient.
DeepSeek V4 Flash produces a thinking section that is
proportional to problem complexity —
often 1/5 the length of other models. DS4 exposes
--think, --think-max,
--nothink controls and streams reasoning
content in the native API format.
The model's 1 million token context window is fully supported. Compressed KV cache (indexer + ratio-4/ratio-128 rows) keeps memory manageable: ~26 GB for the full 1M context, making it feasible on 128 GB machines at smaller windows.
Every GGUF is tested against official DeepSeek
continuation vectors. The ds4_test suite
compares token bytes and logit slices against the
reference implementation. No "works on my machine" —
correctness is verified by ground truth.
Three complete API implementations: OpenAI Chat, OpenAI Responses (for Codex CLI), and Anthropic Messages (for Claude Code). All support streaming, tool calls, thinking mode, and KV cache reuse. Tool schemas are prepended to the system prompt automatically.
The 2-bit quants are not a joke: they use asymmetrical quantization where only routed MoE experts are quantized (up/gate at IQ2_XXS, down at Q2_K), while shared experts, projections, and routing are left untouched. The result is remarkable quality at 81 GB model size.
Real-world throughput numbers across hardware configurations. Short prompts, long prompts, and the DGX Spark.
| Machine | Quant | Prompt Type | Prefill (t/s) | Generation (t/s) |
|---|---|---|---|---|
| MacBook Pro M3 Max, 128 GB | q2 | short | 58.52 | 26.68 |
| MacBook Pro M3 Max, 128 GB | q2 | 11,709 tokens | 250.11 | 21.47 |
| Mac Studio M3 Ultra, 512 GB | q2 | short | 84.43 | 36.86 |
| Mac Studio M3 Ultra, 512 GB | q2 | 11,709 tokens | 468.03 | 27.39 |
| Mac Studio M3 Ultra, 512 GB | q4 | short | 78.95 | 35.50 |
| Mac Studio M3 Ultra, 512 GB | q4 | 12,018 tokens | 448.82 | 26.62 |
| DGX Spark GB10, 128 GB | q2 | 7,047 tokens | 343.81 | 13.75 |
ds4-eval — a 92-question regression suite, not a
leaderboard runner.
Graduate-level science questions (physics, chemistry, biology) with multiple-choice answers. Tests whether the model can still reason correctly after kernel or quantization changes.
Broad specialist knowledge and domain-transfer questions. Curated from the upstream dataset — rows with wrong keys or underspecified prompts are replaced.
Exact-answer contest math problems. No multiple-choice prior, no partial credit — a single arithmetic slip changes the grade. The most unforgiving items in the set.
Single-function C/C++ security reasoning items reduced from public CVE writeups. Identify the best source line where the defensive code flaw is introduced.
To understand ds4.c and the DeepSeek V4 Flash model, start here.
README.md (full)The primary documentation — motivations, architecture, CLI, server setup, agent configuration, and tool-call handling. Read this first.
MODEL_CARD.mdSynopsis of the official DeepSeek-V4-Flash Hugging Face model card. Covers model family, architecture (CSA/HCA), quantization strategy, and benchmark interpretation.
CONTRIBUTING.md
Correctness and speed regression testing guide.
Describes the ds4_test suite: logprob
vectors, long-context story recall, tool-call quality,
and server tests.
AGENT.mdInternal developer notes: goals, quality rules, safety constraints, and code layout. Explains why the project is structured the way it is.
gguf-tools/README.mdOffline GGUF generation, imatrix collection, quantization tooling, and quality checks. For building custom GGUFs.
speed-bench/README.md
Benchmark methodology, CSV files, and SVG graph
generation. Explains how ds4-bench measures
incremental throughput at context frontiers.
dir-steering/README.mdDirectional steering data, vector generation, and usage — how to bias model outputs by injecting steering vectors during generation.
tests/test-vectors/README.mdOfficial continuation vectors used for regression checks. Critical for understanding the correctness validation pipeline.
The official model card: architecture details, training data, benchmarks, and the DSML encoding specification used for tool calls.
DS4's spiritual parent. Understanding GGML's approach to tensor scheduling, quantization, and Metal/CUDA kernels provides context for DS4's design decisions.
DS4 exists because DeepSeek V4 Flash is special: 13B active parameters from 284B total, compressed KV cache, 1M context, remarkably short thinking traces, and excellent 2-bit quantization behavior. A generic GGUF runner can't optimize for all these properties — DS4 is deliberately narrow.
Start with ds4.h (the public API), then
ds4.c (model loading + sessions), then
ds4_server.c (API orchestration), then
ds4_metal.m (GPU scheduling). The code
is organized around this boundary — CLI and server
code don't know tensor internals.
The commit history in this page tells a story: initial release → tool streaming → KV cache replay → CUDA support → Responses API → eval harness → correctness fixes. Each phase added a critical capability. Understanding the order reveals why features were built.
make test runs the full regression
suite.
./ds4_test --logprob-vectors compares
against official continuations.
./ds4_test --long-context checks story
recall at 100k+ tokens. Tests are the ground truth
for correctness.
Each subdirectory has its own README:
gguf-tools/ for quantization,
speed-bench/ for benchmarking,
dir-steering/ for steering vectors,
tests/test-vectors/ for regression
data. The main README links to them all.
DS4 speaks three API protocols. The key innovation is exact DSML tool replay — the bridge between the model's native DSML tool format and the agent's JSON tool-call objects. This is what makes multi-turn agent sessions efficient.