Serving

poot-serve is the OpenAI-compatible HTTP server. It loads a registered model, fits it to the device and serves /v1/completions and /v1/chat/completions from one scheduling loop over the driver. An embedding or reranking checkpoint serves /v1/embeddings and /v1/rerank, which a decoder's embedding pass also serves. LoRA administration runs on the same engine thread between steps. For embedding poot as a library, see Develop; the design behind these pages is under Serving design. Which families, quantizations and backends are served is listed per row in the feature matrix, and measured speed on the performance page.

Getting started

• Serving your first model: first HTTP responses from a checkpoint.
• Picking a backend: the --backend option and how auto chooses.
• Using quantized checkpoints: packed GGUF and safetensors formats.
• Configuration reference: every command-line flag.

API

• OpenAI-compatible API: endpoints, error classes, bounds and metrics.
• Guided decoding: constrained output (regex, grammar, JSON schema, tools).
• Embeddings and reranking: /v1/embeddings and /v1/rerank.
• Multimodal: image input for a checkpoint with a vision front end.
• Using LoRA adapters: load, select and unload adapters.

Scheduling and placement

• Continuous batching and concurrency: the scheduling loop, rows and bounds.
• Speculative decoding: prompt lookup, draft model and the model's own MTP layer.
• Tensor parallelism: weights and the paged KV pool divided over GPU ranks.