Glossary

All terms used in the architecture section, alphabetically. Search or use the letter jumps to navigate.

AQL (Architected Queue Language)The packet format AMD HSA runtimes use to submit work to a GPU queue. poot-rocm-runtime records AQL kernel-dispatch packets and replays them for capture/replay decode.see backends

ashA Rust crate of low-level Vulkan API bindings (unsafe, close to the C API). poot-vulkan-runtime uses ash to record and resubmit command buffers without going through wgpu.see backends

BackendA specific GPU target that poot can lower kernels to. The GPU backends are wgpu (SPIR-V/Vulkan), raw Vulkan (SPIR-V through ash), ROCm (AMD HSA) and PTX (NVIDIA). A backend consists of a codegen target, a runtime, and an executor.see backends

Block tableA host-side per-sequence table that maps logical KV block indices to physical block indices in the paged KV pool. The host flattens it into the Slot::SlotMap graph input (logical position to physical slot), which the paged decode graph uses to gather K/V, and into the Slot::WriteMap input (each new token to its physical slot), which the graph writes K/V through.see attention

Builder / Tracedpoot_graph_ir::Builder records equations into a Graph. A Traced value is a type-only handle (a ValueId plus a TensorType) returned by each builder call. Model tracers combine them with the composite helpers in poot_graph_ir::ops.see tracing

Causal maskAn additive attention mask (shape [cap]) that is 0 for valid key positions t <= pos and a large negative value for later ones, added to the scores before softmax. It also hides KV cache positions not yet written, enabling a fixed-capacity KV allocation. It is a graph computation over Slot::Pos; no slot carries a mask.see attention

ConstA weight or table value in the graph IR (Storage::Const). Consts are bound by name from the checkpoint, cached on the device, and never reallocated between tokens. Weights, embedding tables, and normalization weights are all Consts.see graph ir

Continuous batchingA serving strategy where in-flight requests are combined into one batch per token rather than at request boundaries. When a request finishes, its batch slot is freed immediately. New requests claim free slots. `poot-serve --draft-lookup NGRAM:K` and `--draft-model PATH:K` serve it through the one scheduling loop.see serving design

Cooperative kernelA kernel launched so that all workgroups are co-resident (cuLaunchCooperativeKernel on PTX, an HSA cooperative queue on ROCm). A persistent-grid kernel relies on this to coordinate across workgroups with event counters.see megakernel

CPU oracleThe graph reference evaluator in poot-eval: one walk, eval(&graph, &inputs, EvalOptions). Device float results use stated tolerances; packed decode and integer semantics have exact checks.see execution

CUDA graphA NVIDIA API that records a sequence of GPU dispatches and their parameters, then replays the entire sequence with a single cuGraphLaunch call. Eliminates per-dispatch driver overhead. poot uses CUDA graphs on the PTX backend as the primary mechanism for reducing decode latency.see execution

cudarcThe Rust crate poot uses to interface with the NVIDIA CUDA driver API (https://github.com/coreylowman/cudarc). Handles JIT compilation of PTX strings, kernel launches, device memory, and CUDA graph capture/replay.see backends

Device (storage class)The storage class for ordinary intermediate values (Storage::Device). The planner derives storage requirements; the executor allocates or aliases GPU buffers and retains them for cached execution. Activations use this class.see graph ir

DriverThe poot-llm token loop over the executor contract. It binds one ModelHandle to one executor, loads the weights once as one executable, adds one entry per (phase, step shape, sampling suffix) it needs, and runs open, step, commit and release. It knows no model family and no backend.see driver

Entry (executor)One compiled specialization on an executable: a program for a (phase, step shape, sampling suffix) key. An entry records on its first step and replays afterwards. Token ids, positions and sampler parameters are bound inputs and never make a new entry.see driver

Eqn (equation)One node in the graph IR: an OpKind (with its parameters), a list of input Operands (a value or an inline literal), and one output ValueId. The graph is an ordered list of Eqns in topological order.see graph ir

Flash attentionA family of attention algorithms that avoid materializing the full [L, L] attention score matrix by computing the softmax online. Peak memory is O(L x head_dim) instead of O(L^2). In poot, a compiler pass rewrites the primitive attention subgraph to the fused kernel automatically.see attention

FusionA compiler pass that groups primitive eqns into regions that become single kernel dispatches. Reduces the number of dispatches and the number of DRAM round-trips.see fusion

GBNFGGML/llama.cpp Backus-Naur Form: a grammar syntax used to constrain model output. poot-serve accepts a grammar in the guided_grammar request field and masks disallowed tokens with it.see serving design

GGUFA file format for storing quantized model weights, used by llama.cpp. poot supports loading GGUF files via poot-load. Supported quantization types include Q8_0, Q4_0, and the common K-quants.

GQA (grouped-query attention)An attention variant where K and V have fewer heads than Q. Multiple Q heads share one K/V head. poot handles GQA via Broadcast ops over the head axis - no special opcode.see attention

Graph IRpoot's core intermediate representation. A flat, ordered list of primitive tensor operation equations in SSA (static single assignment) form. Every other subsystem produces or consumes it.see graph ir

HSAHeterogeneous System Architecture: AMD's low-level runtime model. poot-rocm-runtime talks HSA directly (hsa-runtime64) rather than HIP/rocBLAS.see backends

iGPUIntegrated GPU - built into the same chip as the CPU, sharing system memory (unified memory architecture). Accessed via the wgpu or raw Vulkan backend.

KIR / kernel IRThe kernel intermediate representation in poot-kernel-ir. A tiny CFG-based IR for one kernel function body, between Stable MIR and LLVM IR. Decouples the importer from codegen backends.see kernel import

KV cacheDevice buffers that store keys (K) and values (V) for all previous tokens in a sequence. The attention kernel reads from the KV cache. Without it, K and V would need to be recomputed for every token.see attention

Launch metadataA dispatch's launch metadata (Launch::meta) binds a small read-only u32 metadata buffer to an imported kernel: shape and parameter information such as head counts, capacity, and the scale as f32 bits. Allows one kernel binary to serve multiple configurations.see fusion

LDS (local data store)GPU hardware shared memory within a workgroup (called "shared memory" in CUDA). In imported poot kernels it is accessed with wg_write and wg_read on a literal array id. Required for flash attention accumulators.see kernel import

LLVM IRThe intermediate representation used by the LLVM compiler. poot-codegen emits LLVM IR from kernel IR, then lowers it with llc (SPIR-V, PTX) or ROCm clang (AMDGCN).see kernel import

LoRALow-Rank Adaptation (Hu et al., 2021): fine-tune small rank adapters instead of full weights. The driver keeps adapter sets; poot-serve loads, selects and unloads them through its LoRA administration routes.see serving design

MegakernelA persistent device launch combining work from several kernels. poot has no megakernel. A bounded single-step implementation needs compiler lowering and a co-residency proof; an entire token loop needs additional contracts. The required grid-wide synchronization is unavailable on the SPIR-V backends (wgpu and raw Vulkan).see megakernel

MegatronMegatron-LM style tensor parallelism: shard attention/MLP weights across GPUs with collectives. poot-serve does not implement tensor parallelism.see backends

MHA (multi-head attention)Attention with the same number of heads in Q, K, and V. The standard Transformer attention mechanism.see attention

MLA (multi-head latent attention)Attention that compresses K/V into a latent cache. Can use different head widths for Q/K vs V; poot keeps unequal-width cases on the primitive decomposition rather than flash fusion.see attention

Model (trait)What a model family implements: its ModelConfig, its WeightMap, and trace_into(phase, step shape, builder), which builds the graph of one step under the caller's TraceLimits. A shape or phase the family cannot trace, or a step that exceeds the limits (TraceError::Budget), is a typed TraceError.see driver

ModelHandleA loaded checkpoint: the family's Model, its weight store and its text services (tokenizer, chat template). Loading resolves the family from the config before any weight is read.see driver

MoE (mixture of experts)A model architecture where each token is routed to a subset of expert FFN layers. poot traces routing as ArgTopK plus IndexedMatMul over stacked expert weights. Packed expert paths currently decode and stack those weights into device tables first.see tracing

MQA (multi-query attention)Extreme GQA where K and V have exactly one head. Multiple Q heads all attend to the same K/V. Same mechanism as GQA - Broadcast over the head axis.see attention

Online softmaxA one-pass algorithm to compute softmax without materializing the full score vector. Maintains a running maximum and denominator while streaming over inputs. Flash attention uses this to avoid the O(L^2) score matrix.see attention

OpenVINOIntel's model compiler/runtime stack. poot lowers graph regions to OpenVINO IR text for the Intel NPU, which the NPU runtime consumes. The NPU is under active development, with prefill as the goal.see npu

OpKindThe enum of all operations in the graph IR: primitives (Unary, Binary, Select, Reduce, Broadcast, Cast, Reshape, Transpose, Slice, Concat, Gather, MatMul, ...) and compiler-introduced composites (Fused, FusedRow, ContractionEpilogue, Attention, Rope, packed and dense contractions). No op records a kernel choice; the planner records it with each plan.see graph ir

Packed dequantizationDecoding stored weights according to a PackedWeight descriptor from poot-quant. PackedDequant is the primitive; compiler-only PackedContraction and PackedRowGather decode values while consuming them. A packed MoE expert stack is read by one indexed packed contraction, so no expert table is materialized.see graph ir

Paged KV cacheA KV cache implementation that divides the K/V space into fixed-size blocks from a shared pool. Memory scales with actual sequence length. Enables block sharing across requests with common prefixes.see attention

PlanThe per-eqn output of planning (poot-graph-plan plan_eqn): a compute body (optionally with a metadata buffer), chunked compute, a buffer alias or strided view. The executor maps the Plans of a graph to a sequence of device dispatches.see fusion

pootcA rustc driver that compiles a crate normally, then hooks after_analysis to import every #[kernel] fn's Stable MIR into poot's kernel IR. Output: .kir.json files for each kernel.see kernel import

Prefix cacheA serving optimization where KV blocks from a previous request are reused for a new request that starts with the same prefix tokens. Reduces prefill cost for common system prompts. `poot-serve --draft-lookup NGRAM:K` and `--draft-model PATH:K` serve it through the one scheduling loop.see attention

PTX (Parallel Thread eXecution)NVIDIA's virtual instruction set architecture. poot lowers kernel IR to PTX via LLVM's nvptx64 backend. The PTX text is JIT-compiled by the CUDA driver to the actual GPU ISA at load time.see backends

RADVMesa's open-source Vulkan driver for AMD GPUs. poot's wgpu and raw Vulkan paths are verified against RADV; several SPIR-V subset rules exist because RADV crashes on otherwise valid patterns.see backends

RegistryThe set of model families a driver can load. Each FamilyEntry names the config keys that select the family (a Hugging Face model_type, a GGUF architecture), and how to build it. A key no family holds is a typed unsupported-model error.see driver

Rust kernel subsetThe subset of Rust accepted by pootc for #[kernel] functions. Covers loops, slice indexing, arithmetic, float intrinsics, LDS arrays, barriers, and atomics. Violations produce a named diagnostic.see kernel import

SafetensorsA file format for storing model weights, used by Hugging Face. poot-load reads tensor byte ranges into typed weight owners, preserving their stored dtype or packed format.

SlotA per-step graph input (Storage::Slot) such as Token, Pos, SlotMap, or WriteMap. Stored in a device buffer at a stable address. The captured graph references these addresses; per-token updates write the new value in place.see execution

Speculative decodingA latency optimization where a cheap draft model generates N token proposals, and the full target model verifies all N in one forward pass. Accepted tokens advance the sequence by up to N+1 positions. `poot-serve --draft-lookup NGRAM:K` and `--draft-model PATH:K` serve it through the one scheduling loop.see serving design

SPIR-VA portable binary intermediate language for GPU kernels, used by Vulkan. poot lowers kernel IR to SPIR-V via LLVM's spirv64 backend and loads the binary via wgpu or raw Vulkan.see backends

SSA (static single assignment)A property of the graph IR: each computed ValueId is defined by one Eqn; graph inputs are binders. Makes transforms simple - no ambiguity about which definition a use refers to.see graph ir

Stable MIR (rustc_public)A public API to rustc's Mid-level Intermediate Representation (https://github.com/rust-lang/rustc_public). pootc uses it to read #[kernel] function bodies without depending on rustc's unstable internal APIs.see kernel import

State (storage class)The storage class for values carried across decode steps, such as the KV cache (Storage::State). State buffers are pre-allocated to capacity, and each state_in binder is paired with a state_out value that the executor writes back into the same buffer.see graph ir

StepShapeThe semantic shape of one traced step: rows, new tokens per row, cache capacity, KV layout (contiguous or paged), whether logits return for the last token or all tokens and, when asked, a hidden-state carry.see driver

StructurizationA compiler pass that converts the unstructured goto-based CFG of the kernel IR into structured control flow (proper loops, if/else). Required for SPIR-V emit. Applied once per kernel.see backends

TensorType (aval)A shape + dtype pair with no data. The graph IR describes every value with a TensorType, stored in ValueMeta.aval, without holding any concrete data. Shape inference (OpKind::infer) operates on TensorTypes.see graph ir

TokenA unit of text (word, subword, or character) from the model's vocabulary. In poot, the current token id is a Slot input updated each decode step.

TracingRunning a model function once with type-only Traced handles so each builder call records primitive eqns into the graph IR. Traced values carry no data, so model code cannot branch on activation values.see tracing

ValueIdA dense integer index into the graph's value table. Every tensor in the graph is identified by a ValueId. Every Eqn output and every graph input gets a unique ValueId.see graph ir

ValueMetaThe metadata for a ValueId: its TensorType (aval), storage class (Device, Const, Slot, State, or Computed), and optional name.see graph ir

wgpuA Rust crate providing a safe, portable GPU API over Vulkan, Metal, DX12, and WebGPU. poot uses wgpu as the runtime of the wgpu backend.see backends

WorkgroupThe unit of GPU parallelism in which threads share LDS and can synchronize with a barrier. Called a "thread block" in CUDA, "workgroup" in Vulkan/SPIR-V. poot kernels are written in terms of workgroups.see kernel import

XDNA2AMD's second-generation AI Engine tile array, present in Ryzen AI Max processors. poot compiles to a typed AIE artifact through Peano (an LLVM fork with aie2p target support) and an IRON link step, and the experimental `poot-npu` device (behind its `xdna` feature) runs the linked artifacts on the NPU. It is verified for F32 add, sub and mul only.see npu