Buoyant Releases Kinference: An Open Source Inference Control Plane for Kubernetes and vLLM

7 November 2026 3 min read هدهد تك | Hudhud Tech

Buoyant, known for its work on Linkerd, has released Kinference, an open source project designed for building inference platforms on Kubernetes. Kinference serves as an inference control plane for Kubernetes and vLLM, managing request routing to a fleet of vLLM workers. It implements routing policies, token rate limits, KV-cache-aware load balancing, and per-tenant metrics, targeting the layer above the model server. The project is licensed under Apache 2.0 and is available on GitHub. The current version is v0.1.0-alpha.7.

TL;DR

Size: v0.1.0-alpha.7, written in Rust, Apache 2.0, configured through 5 Kubernetes CRDs.
Runs on: any GPU-enabled Kubernetes cluster running vLLM. Buoyant’s benchmark utilized 3 L40S GPUs per pool.
Performance: significant improvements on prefix-heavy traffic compared to standard Kubernetes Service routing (self-reported).
Best: median TTFT decreased from 6.88 seconds to 0.31 seconds (95%) on a shared-prefix workload.
Worst: no median TTFT improvement on prompts without shared prefixes (6.92 seconds vs 6.83 seconds).

What is Kinference?

Kinference is a policy-driven control plane that transforms Kubernetes and vLLM into a multi-tenant inference platform. According to Buoyant’s FAQ, vLLM acts as the data plane while Kinference serves as the control plane.

How does Kinference work?

The system consists of two components: a traffic gateway and an inference handler. The handler exposes a gRPC interface, processing requests and returning the selected endpoint along with a potentially modified body. It works in conjunction with a proxy, such as the Buoyant Gateway.

Kinference monitors vLLM KV-cache events and engine metrics, currently utilizing NVIDIA’s dynamo-kv-router crate for cache event processing. Configuration is managed through five Custom Resource Definitions (CRDs): GatewayBackend, InferencePolicy, InferencePool, InferenceProvider, and InferenceWorkerSet.

What are inference policies?

Inference policies are declarative rules that dictate how each request is routed or rejected. These rules can match based on tenant, requested model, GPU pool state, and semantic tags. Example policies available on the project site include:

Simple to cheaper: route low-complexity prompts to a smaller model unless cache overlap makes the larger model faster.
Burst to cloud: bypass workers with over two queued prompts, reverting to providers like Bedrock if necessary.
PII stays local: exclude external providers when a PII tag is triggered, failing closed with onEmpty: Reject.
Batch on idle GPUs: direct batch requests only to workers with less than 70% KV-cache usage.
Model aliasing: maintain a single stable name, such as standard-model, backed by multiple models.

Most policies conclude with minimize: PredictedTTFT, where Kinference predicts the time-to-first-token for each candidate endpoint based on cache overlap and queue length. Policies are stored in Git and can be validated offline using the kinfctl CLI.

Kinference represents a significant advancement for developers managing multiple open-weight models for various internal clients, streamlining the inference process on Kubernetes.