A model that loads successfully is only the beginning of an inference deployment. It still has to fit the intended context window, handle simultaneous requests, speak the client’s API, and deliver acceptable quality within a useful latency budget.

I built Inference God Mode to put those decisions into a reusable coding-agent skill. Its inference-optimizer workflow takes a specific model, hardware setup, and workload through planning, deployment, benchmarking, and tuning. The same process extends from a single GPU to Kubernetes serving, with the evidence for each decision kept visible.

Fit the full context, verify APIs and quality, measure real traffic, then change one thing and test again. The loop keeps the workload requirements fixed.

Define the deployment requirements

The first step is an inventory: the exact model and tokenizer, GPU memory and interconnect, host RAM, storage, and the client features that must work. Then come the workload questions: how many generations are active at once, how long the prompts and outputs are, and how quickly users need the first and subsequent tokens.

Registered users do not tell us the concurrency. A few agents working through long documents can put more pressure on a server than many occasional chat users. When the workload is unspecified, the skill provisionally assumes one active user and prioritizes interactive latency. It also starts with the model’s full native context, including output headroom. If that cannot fit, it exposes the capacity gap before changing the requirement.

Weights and KV cache are separate budgets

A 4-bit checkpoint is a starting candidate, not a complete memory plan. The deployment also needs room for the KV cache, runtime allocations, graph capture, and workspace. Those costs depend on the model, context length, concurrency, and engine. A GGUF file size does not establish that the same model fits in vLLM or SGLang.

The skill evaluates weight precision separately from KV precision. It prefers an FP8 KV candidate where the device, model, and attention backend support it, while retaining a BF16 or FP16 control. Promotion requires checks on quality, long-context behavior, memory, and performance. Likewise, an NVFP4 label is not proof of native FP4 execution: the checkpoint metadata and the launched kernel path both need verification.

Weights, KV cache, runtime allocations, and headroom share the GPU memory budget. Weight precision and KV precision are separate choices; segment sizes are schematic, not measured.

Choose a runtime, then test the hypothesis

The routing guidance covers vLLM, SGLang, TensorRT-LLM, and llama.cpp, with MLX-LM and vLLM-Metal considered for Apple Silicon. On supported NVIDIA hardware, vLLM is the initial candidate for concurrent serving and SGLang for one active user. These are project starting preferences; a comparable run on the actual workload decides which configuration to keep.

Compatibility can change the route before performance tuning begins. Older hardware may call for a llama.cpp or Vulkan candidate. A required API can favor another engine or a tested adapter. Community builds and plugins are candidates too, but their source revisions, changes, and actual kernel behavior have to be recorded. A successful startup is one check in that process.

An API endpoint is a behavioral contract

Coding agents need more than a plain-text completion. The workflow tests the required OpenAI Chat Completions, OpenAI Responses, or Anthropic Messages routes, including streaming, tool-call round trips, structured output, reasoning fields, and usage reporting where needed. The tokenizer, chat template, and model-specific parsers are part of that verification.

A server returning HTTP 200 does not establish that an agent can continue a tool exchange or interpret its stream. If a gateway or adapter translates the protocol, the tests run through that component as well. The point is to verify the path the real client uses before measuring it as a working deployment.

Measure the workload before adding optimizations

The benchmark plan includes short, medium, and near-native-context prompts at both interactive and peak concurrency. Repeated-prefix applications get cold-cache and warm-cache runs. The report tracks time to first token, inter-token latency, per-user and aggregate output throughput, latency percentiles, errors, and peak memory. Enough samples are needed before interpreting tail latency.

Tuning then changes one dimension at a time: weights and kernels, FP8 KV, batching and scheduling, graph capture, prefix reuse, parallelism, and speculative decoding where supported. Correctness and quality are rechecked after changes. A throughput gain that misses the latency target does not win automatically, and a faster synthetic microbenchmark does not establish a faster serving system.

Extend the same discipline to multiple GPUs

For multi-GPU deployments, the skill evaluates feasible tensor, pipeline, data, and expert parallel layouts sequentially. Expert parallelism applies to supported mixture-of-experts models. Per-GPU fit, rank placement, and interconnect costs matter; adding devices does not guarantee lower latency for one active stream.

At the fleet level, llm-d enters as an orchestration and routing layer above the serving engine. The comparison starts with ordinary replicas, then cache- and load-aware routing, and finally separate prefill and decode pools when measured interference justifies them. KV transfer time, network contention, pool balance, and gateway latency remain part of the measurement. The aim is the simplest architecture that meets the workload with a demonstrated benefit.

Workers A and C hold the same reusable prefix. When A becomes busy, routing to C preserves cache reuse while avoiding extra load on A. The worker loads are illustrative.

Use it with your coding agent

The canonical skill lives in the repository’s Codex skill folder. Claude Code and OpenCode paths point to that same folder through relative symlinks. For another project, copy the complete inference-optimizer folder into your harness’s supported skill directory and keep its references folder alongside SKILL.md.

git clone https://github.com/abhiram1809/inference-god-mode.git

Then give the agent a concrete request. Replace the placeholders with your deployment requirements:

Use inference-optimizer to serve <model repository>
on <GPU model, count, and memory>.

Required APIs: <client protocols and features>.
Peak active generations: <concurrency>.
Latency target: <first-token and inter-token budget>.

Keep the full native context.
Evaluate 4-bit weights and FP8 KV.
Benchmark my workload and report the best verified setup.

What the project establishes today

Inference God Mode provides a structured workflow and reference guides. Its published validation covers skill structure, local reference links, and harness symlinks. It does not establish a universal speedup or a benchmark-winning configuration. Those results still have to be measured for the chosen model, hardware, runtime, and traffic.

The useful output is a reproducible configuration or a concrete next experiment: pinned revisions, selected kernels, tested APIs, context and concurrency limits, measurements, quality observations, and the remaining bottleneck. That makes the skill a way to organize inference engineering around evidence and to carry that evidence into the next deployment decision.

View Inference God Mode on GitHub Read the skill workflow

This article describes the repository as reviewed on October 2, 2026. The linked skill revision records the workflow; deployment commands and compatibility checks should use current upstream documentation.