Skip to main content

Picking a backend

poot-serve --backend wgpu|rocm|ptx|vulkan|auto (default auto) picks the device. The choice is made once, at startup, before the server listens:

  • auto opens the first device that is present, in one fixed order: ptx if a CUDA device opens, else rocm if an HSA GPU agent opens, else wgpu. It decides by device presence only. It never picks vulkan.
  • A named backend that cannot open, or a model the chosen backend refuses (an operation the device lacks a capability for), stops startup with the refusal. poot-serve never tries another backend after a refusal: a model that does not run on the device you chose is an error you can read, not a silent move to different hardware.
  • --placement pipeline:ORD,ORD,... (or tensor:...) runs one model over several GPUs, and only the ptx backend can: the other backends cannot copy between their devices, so a group on them is refused at startup. --device-timing is refused over a group.
  • Before it serves, the server fits the workload to the device memory: it computes the exact footprint of each candidate (weights once, every retained arena, state) and takes the largest candidate that fits, then checks what the device actually allocated before /health reports serving.

--backend vulkan serves too: raw Vulkan reads the free device memory from VK_EXT_memory_budget, so the engine plan fits the workload to it like any other backend. A Vulkan driver without that extension reports no budget, and startup then stops with a typed capacity error (an unknown budget is never guessed).

The backends are libraries that share the graph IR and model definitions: wgpu (Vulkan/SPIR-V, the default), rocm (AMD raw HSA), ptx (NVIDIA CUDA) and vulkan (the same SPIR-V kernels as wgpu, dispatched through raw Vulkan with no wgpu). Only codegen and dispatch differ. Pick one in library code through the poot-llm API. auto never picks vulkan, since it runs on the same hardware as wgpu. The feature matrix lists what each supports, including the raw Vulkan section, and Architecture: Backends describes the design.

The AMD XDNA2 and Intel NPU4 NPUs are not serving backends: they are experimental library devices (Architecture: NPU Offload).

Checking that the toolchain sees your GPU​

Whatever backend you use, confirm the toolchain sees your GPU before troubleshooting:

llc --version | grep -iE 'spirv|nvptx'   # both codegen targets present
vulkaninfo | grep -i 'deviceName'         # your GPU shows up to Vulkan (needed even for rocm/ptx builds)