Configuration reference

One place for poot-serve command-line options. Every setting is a flag: no environment variable selects a mode or a limit, and a value that does not parse stops startup instead of falling back to a default. The batch size and context length are not options: the engine plan fits them to the device. Where the model runs is one flag, --placement; pooled-MoE serving is not implemented. See Serving for what it does.

CLI

Flag / argument	Default	Notes
MODEL (positional)	developer-local default path	Safetensors directory or .gguf file. Always pass it.
ADDR (positional)	127.0.0.1:8080	Bind address.
--backend	auto	wgpu, rocm, ptx, vulkan or auto. Every backend is in every build, so --backend rocm needs no build flag; a backend whose device is absent is refused at startup. auto opens PTX if a CUDA device opens, else ROCm if an HSA GPU agent opens, else wgpu, and never falls back after a refusal. vulkan (raw Vulkan) fits the workload to the free memory VK_EXT_memory_budget reports, and is refused at startup with a typed capacity error on a driver without it. See Picking a backend.
--placement	single	single (the backend's default device), pipeline:ORD,ORD,... (one layer stage per GPU ordinal, in the order given, the cut balanced over each GPU's free memory) or tensor:ORD,ORD,... (one tensor-parallel rank per ordinal). A group needs at least two distinct ordinals and the ptx backend (the other backends cannot copy between their devices) and is refused otherwise at startup. Each GPU is fitted against its own memory; one that cannot hold its share stops startup naming it. --device-timing is refused over a group. See Tensor parallelism.
--draft-lookup NGRAM:K	off	Speculate by prompt lookup: match the last NGRAM tokens of a sequence against its earlier text and propose up to K following tokens. Both numbers are positive integers; the value is split at the first colon. Exclusive with --draft-model and --draft-mtp. See Speculative decoding.
--draft-model PATH:K	off	Speculate with a draft model loaded from PATH, K tokens a round. The value is split at the last colon, so PATH may contain colons. A path that does not load, or a draft the target cannot verify (larger vocabulary, different token ids, fewer positions), stops startup before the listener binds. Exclusive with --draft-lookup and --draft-mtp.
--draft-mtp K	off	Speculate with the served model's own multi-token-prediction layer, K tokens a round. K is a positive integer and has no default. Exclusive with --draft-lookup and --draft-model. Startup is refused, with the driver's typed error, for a model with no MTP layer, one whose MTP layer carries recurrent state, adapters, beam search, and a device group (--placement pipeline: or tensor:). See Speculative decoding.
--chat-template JINJA	off	Render chats through this jinja template instead of the model's own. A value with no jinja syntax names a built-in template, which poot lacks, and is refused at startup, as is a template that does not compile. A template that fails to render a request's conversation answers that request with an error; it never falls back to another format. Exclusive with --chat-template-file. Not supported for a vision-language checkpoint, whose image chats render through the checkpoint's own template: startup refuses the flag there.
--chat-template-file FILE	off	Read the jinja chat template from FILE, with the same refusals as --chat-template.
--rows N	4 (decoder), 32 (encoder)	Rows the engine is fitted for: concurrent sequences of a generative model, texts per request of an embedding or reranking model (a decoder's embeddings take the same rows). At least 1. The memory fit halves it until the workload fits, and /health reports the rows it settled on.
--max-seq N	4096	Positions per sequence of a generative model, or tokens per text of an embedding or reranking model, capped at the model's own limit. At least 1. The memory fit halves it (to a floor of 128) until the workload fits, and /health reports the value it settled on.
--expert-pool ROWS	off	Page every routed expert table through a pool of ROWS resident rows per layer: the experts a step selects are made resident before they are read, least recently used rows making room. ROWS is at least the experts one token selects and below the model's expert count; a dense model is refused.

Any -- option not listed here is refused at startup. The three draft flags are mutually exclusive. A model whose recurrent state advances in granules of more than one token cannot speculate, and neither can a draft model with recurrent state, so a draft flag on either is refused at startup; --draft-mtp has the further refusals in its row above (see Speculative decoding).

The listener opens once the model is prepared on the device, so a client gets connection refused while the server starts. /health is then 200 with "status": "serving" and the backend, rows and max_seq the engine plan settled on. serving means the device plan was admitted and the engine thread is running. An embedding or reranking model (an embedding encoder or cross-encoder, or any checkpoint whose output is hidden states or scores) is planned the same way as a generative one, and a decoder's embedding pass is planned beside its decode shapes: the encode shapes are fitted and prepared before serving, and a request never compiles or allocates on the device. One that does not fit stops startup with the capacity error.

Logging uses RUST_LOG (default info). SIGINT/SIGTERM stop accepting connections and drain in-flight requests.

Example:

cargo run -p poot-serve --release -- ~/models/bge-small-en 0.0.0.0:8080

HTTP and administration

Flag	Default	Purpose
--rate-limit-rpm N	0 (disabled)	Per client IP fixed 60s window on every POST (/v1/completions, /v1/chat/completions, /v1/embeddings, /v1/rerank). Keyed on peer IP. /health, /metrics and other GETs are not limited.
--lora-admin-key-file F	unset	File holding the bearer token required for the /v1/lora_adapters routes (including on loopback). One token on its own line; a trailing newline is ignored. A file keeps the key out of ps. Inference stays unauthenticated. See LoRA adapters.
--max-connections N	4096 (0 = unlimited)	Cap on simultaneous connection-handler threads.
--max-pending-requests N	256	Model requests admitted at once, running or waiting for the engine. One more is answered 429 with Retry-After before its body is read. At least 1.
--max-receiving-requests N	64	Admitted requests that may be reading their body at once. One more is answered 429 with Retry-After. A request that has been admitted but is still sending holds a place of --max-pending-requests, so this smaller bound keeps slow senders from filling the whole queue before any work reaches the engine; the trade-off is that more than this many simultaneous large uploads wait their turn. At least 1.
--max-queue-wait-secs N	60	Seconds an admitted generation request may wait for the engine to open it. A request still queued then is answered 503; running requests are not affected. At least 1.
--max-body-mib N	4	Largest request body, in MiB, of every route but a chat on a server that takes images. A larger Content-Length is answered 413 before the body is read.
--max-image-body-mib N	64	Largest request body, in MiB, of a chat on a server that takes images (a vision-language model).
--request-timeout-secs N	30	Seconds from accept in which the whole request, head and body, must arrive; a slower client is answered 408 and closed however it paces its bytes. At least 1.
--write-timeout-secs N	30	Seconds one response write (a whole response or one stream frame) may take, however slowly the client reads it. Must be a positive integer.
--drain-timeout-secs N	30 (1 to 86400)	Graceful-shutdown drain time before in-flight requests are cancelled. 0 or a value above one day stops startup.

Request headers are capped at 64 KiB. A POST must carry Content-Type: application/json, a Content-Length and a Host that names the listener (localhost, its loopback address, or its bound address; a wildcard listener checks only the port); a request with Transfer-Encoding, a malformed or repeated Content-Length or a body that is not UTF-8 is refused with a 4xx before it reaches the engine. The /v1/lora_adapters routes get the same Host check. poot does not validate API keys for inference unless you add policy in front of the server.

Device timing

By default the server keeps cumulative counters only: step counts and the host wall of each step cost nothing beyond a clock read, and no device timestamp is requested. --device-timing asks the device for per-dispatch timestamps and keeps bounded detail. It changes how work is submitted, so a run with it is diagnostic and not throughput evidence. A backend that cannot time its dispatches (raw Vulkan) reports the device time as unknown.

Flag	Default	Purpose
--device-timing	off	Request device timestamps and retain bounded per-dispatch detail.
--timing-max-steps N	4096	Most steps whose detail is retained (1 to 16384).
--timing-max-dispatch-records N	65536	Most per-dispatch records retained across those steps (1 to 262144).
--timing-max-in-flight-queries N	64	Most timestamp queries in flight at once (1 to 4096).
--timing-sample-every N	1	Collect detail for one of every N steps.

The bound flags need --device-timing; zero and values over a bound stop startup. /metrics reports the result under device_timing: the exact step count, the host wall, the device time (null when no device measured it, never 0), whether the retained detail is complete, sampled, truncated or unavailable, and whether collecting it perturbed submission. /metrics/prometheus carries the same figures; its device-time series is absent when the device time is unknown.