From vLLM to SGLang: How the inference engine evolved

As an ML engineer, model inference can feel like a layer that is already handled — something you configure at a very high level once, or hand off to a cloud provider entirely. Often you don't touch infrastructure at all: on Baseten, Together AI, Nebius, AWS Bedrock, or AWS SageMaker JumpStart you pick a model from a catalog, choose a GPU tier that matches your traffic, and get back an endpoint URL. Whatever inference engine the provider runs behind that URL is a black box — and most of the time, we never think about it.

We probably should. Look at what that black box actually produces. This chart from Artificial Analysis benchmarks DeepSeek V3.2 across providers — same model, same 10,000-token input but significantly different output speed. And output speed is just one axis: time-to-first-token, end-to-end response time, cost per million tokens, and throughput all diverge too. Some of that spread is hardware, but a large part is the serving stack itself — the inference engine, the precision it runs at, and how it batches and schedules requests.

This article is a peek inside the serving stack, the inference engine in particular — how it drives latency down, throughput up, and GPU memory waste toward zero — explained on the examples of the major problems that vLLM and SGLang each set out to solve. (And the new problems they created along the way, and then had to solve too :D)

What an inference engine actually does

On its own, a model loaded onto a GPU handles one request at a time. That wastes most of the GPU. The inference engine is, in simple terms, the software layer that turns it into a system serving thousands of requests at once.

Most LLMs with a transformer architecture generate text in two very different phases, prefill and decode. Prefill is a single pass over the whole prompt: it reads all the input tokens in one request at once and builds up the model's working memory of them (the KV cache). Decode then produces the answer one token at a time — each step reuses everything cached so far and adds the single new token it just generated.

With that in mind, the anatomy of vLLM lays out what the engine does:

• Scheduling — the engine keeps a queue of incoming requests and decides which to work on at each step. Requests already generating take priority over new ones, because keeping the GPU busy beats starting fresh work.
• Managing GPU memory — every active request needs space to hold what the model has already computed for it. The engine hands out and tracks that memory, and when it runs short, pauses lower-priority requests and reclaims their space for more urgent work.
• Packing requests together — rather than handle requests one at a time, the engine runs many in the same pass, mixing new prompts with in-progress generations (performance engineers call this continuous batching). It packs them so each request only ever sees its own tokens — no idle waiting for a fixed batch to fill up.
• Running the model efficiently — the engine coordinates the actual computation: splitting a model that is too big to fit on one GPU across several (tensor parallelism), and trimming the overhead between steps so the GPU never sits idle.
• Choosing the next token — after each step the engine picks the next token, from "always the most likely word" to something more varied and creative (the temperature and top-p are one of the parameters that control that).
• Returning the answer — it turns the model's raw output back into text, checks whether the response should stop, and streams it back to you. The instant a request finishes, its memory is freed for the next one.

Every one of these carries a price tag in latency, throughput, or GPU memory — and the engine is where that price gets set. Which is one of the reasons why, back at that benchmark chart, the same model ran far faster on one provider than another.

Initial problem: wasted GPU memory

The biggest problem early inference engines faced was memory waste. It was a critical issue especially when the demand on GPUs started to grow insanely fast. To see why, we first have to understand how the GPU storage is used at the high level.

Most of the GPU memory during the inference is occupied mostly by model weights and intermediate results for every token the model has seen — its working memory for one request. This is the KV cache (short for key-value cache). It is built during prefill from the prompt, then grows by one entry on every decode step as new tokens are produced. Inference is still lighter than training, which also stores gradients and activations.

That working memory is enormous. For Llama 3.3 70B handling a 32,000-token request, the KV cache for that single request is about 9.77 GB. The model's own weights already take roughly 140 GB — more than fits on one GPU. An NVIDIA H100 has 80 GB of memory, so how much KV cache you can hold is the main limit on how many requests run at once — and running more at once is almost the only way to push throughput up.

So the engine's central job is to pack as many requests' working memory into the GPU as possible. The naive approach was really terrible at this. When a request arrives, the engine cannot know how long the answer will be — the model keeps going until it decides to stop, which might be 10 or 5,000 tokens. To stay safe, early engines reserved enough memory for the longest possible answer, up front, every time. The image below illustrates that: a request that finished after 10 tokens still held a block for the unused part — everything past "for" — sat locked and idle for the request's entire life.

The Berkeley team that built vLLM measured the memory waste: in older engines, only 20–38% of the reserved KV cache memory ever held real data. The rest was wasted, in three ways (see the chart from PagedAttention paper (Kwon et al., 2023) below):

• Over-reservation (reservation) — space held for tokens the request might generate later but has not yet.
• Wasted tail (internal fragmentation) — a request reserved for 2,048 tokens that finishes in 10 leaves the other ~2,038 slots locked and useless until it is done.
• Scattered gaps (external fragmentation) — as requests come and go, the leftover free memory breaks into small, oddly-sized holes. Plenty is technically free, but no single piece is big enough to fit a new request.

The three bars on the left are older engines, burning 60–80% of their KV cache memory on waste. Fine in the easy days. But then the GPU drought hit... vLLM, on the right, went straight for it, putting 96.3% of the same memory to work on identical hardware. And the payoff chains directly: less waste means more requests fit at once, and more requests at once means higher throughput.

Where LLM infrastructure missed a lesson from operating systems

The root cause of all three waste types is one assumption: that a request's KV cache must live in a single contiguous block of memory. Contiguous here means — every byte of the cache laid out in a single unbroken run of adjacent GPU memory addresses. Contiguous layout was the natural default, so the prior inference engines inherited that constraint — and with it, all the fragmentation.

Operating systems solved this exact problem decades ago with virtual memory and paging. A process doesn't need to own a contiguous block of physical RAM; the OS gives it a clean logical address space and silently maps it to scattered physical pages. The process never knows or cares.

vLLM used this idea for PagedAttention. It stores the KV cache in fixed-size blocks (usually of 16-token size) that do not have to sit next to each other in memory. Each block holds the keys and values for a fixed number of tokens and fills left to right; when a sequence needs more room, it grabs the next free block from anywhere. Attention for each new token is computed across all the scattered blocks.

This solution kills all three problems at once. Internal fragmentation shrinks to at most one partially-filled block per sequence. External fragmentation disappears entirely because every block is the same size. And because blocks are independent, they can be shared across sequences and requests, unlocking further latency improvements.

The next move: prefix caching

Once the KV cache lives in independent, shareable blocks, a second opportunity opens up. If two requests start with the same prompt, why compute that prompt's KV cache twice?

vLLM's answer is prefix caching. It gives each block an identity built from its own tokens and every token before it — a chained hash. Two prompts that begin the same way produce the same identities, so when a new request arrives, vLLM reuses any blocks it already holds and prefills only the part that is new.

One detail matters: only complete blocks can be cached. A block is hashed once it fills up, so any leftover tokens that don't fill a whole block are never cached — they get recomputed every time. The diagram below uses a block size of 4 and one word≈one token for clarity; in practice the block size is more like 16. So a 100-token prompt has 100 // 16 = 6 full, cacheable blocks (96 tokens), and the trailing 4 tokens sit in a partial block that misses on every request.

In the diagram above, Request 2 shares its first two blocks with Request 1 — both produce the same chained hashes, so both are cache hits. Only the final block, where the prompts diverge, is a miss that still has to be prefilled. For workloads with heavy prompt overlap — a shared system prompt, a common few-shot preamble, a long document queried repeatedly — this is a large, almost-free win.

Where block hashing starts to strain

Block hashing was built for one pattern: the same run of tokens showing up over and over. However, the application of LLMs is switching from simple chat-bots to long-running agents. In these workloads, one shared context spawns many continuations, and each of those spawns more. What the cache actually sees is not a flat list of repeated strings but a tree.

Imagine a travel-planning agent. One user request fans out into "search flights," then several parallel "search hotels" calls, each spawning "compare options" calls, each spawning "book hotel" calls. The shared context — the original request, the plan so far — is the trunk; the divergent tips are the leaves.

vLLM's flat, hash-addressed cache handles this, but with friction:

• The cache is block-granular. Reuse only kicks in at exact block boundaries, so prefixes that share most-but-not-all of a block get a miss.
• It can't tell a trunk from a leaf. Agents may form prefix trees, not just repeated strings — a shared trunk many branches depend on, plus disposable leaf tips. To a flat hash cache those are just anonymous, equal-weight blocks; it has no notion of which prefix is the valuable shared one.
• Eviction is per-block LRU. A shared trunk about to be reused by hundreds of branches can be evicted just because it looks "least recently used" in isolation — the cache doesn't know it's the most valuable thing in memory. (You can explore this idea in more detail in Fun part: DIY section)
• Scheduling is cache-blind. A request that would hit a 10,000-token cached prefix waits in line behind one with zero overlap, because the scheduler is unaware of cache structure when it admits work.

The flat cache holds no representation of the tree it is storing. That structure exists only implicitly, encoded across separate hashes.

SGLang's bet: make the prefix tree first-class

This is the gap SGLang's RadixAttention is built for. Instead of a flat table of hashed blocks, SGLang stores the KV cache as a radix tree over token sequences. Shared trunks are stored exactly once. Branches fork at precisely the token where sequences diverge. The tree is the primary data structure, guiding prefix matching, eviction, and scheduling directly.

The diagram above shows the idea in miniature. The shared prefix "Plan a weekend trip to" lives in a single trunk node; "Paris" and "Rome" hang off it as separate branches that fork exactly where the sequences diverge. The trunk's KV is stored once and shared by both, while each leaf holds only the KV for the tokens that make it unique — and each node keeps its own KV slots, so a later request matching "Plan a weekend trip to" reuses the trunk and only computes its own ending.

The shift can be summarized as follows:

• Matching becomes a tree walk instead of a block-by-block hash lookup, so reuse follows the true structure of the workload.
• Eviction can favor trunks over leaves — keep the shared prefix that hundreds of branches depend on; drop the disposable tips.
• Scheduling can prioritize cache hits — admit the request that lands on a large cached prefix ahead of the one starting from scratch.

Which one to use?

Both engines are production-grade. The right answer depends on the shape of your traffic. The best approach is always to benchmark.

Key takeaway

Inference engines keep evolving to keep pace with how LLMs are actually used — longer-running agents, rapidly growing user counts, and relentless pressure on scarce GPUs. vLLM and SGLang are the two leading open-source options today, and neither is universally "best": the right choice depends on the shape of your workload, so let your own use case — or a well-curated benchmark — guide your decision. They aren't the only options, either. NVIDIA's TensorRT-LLM is a strong alternative, and several inference providers run their own proprietary engines — the engine is, after all, one of the key differentiators to pay attention to.

Fun part: DIY

If you want these ideas to stick, build them. The three mechanisms in this article — paged KV memory, prefix caching, and continuous batching — are small enough to implement as a state machine on a laptop, with no GPU required.

In the repository, you can find the homework 3 used for one of the modules in the "AI Performance Engineering" course at Nebius Academy.

The setup. A toy one-layer transformer with random weights stands in for a real model — it does no real language modeling, just enough to exercise the memory-and-scheduling engine. It runs a fixed-size pool of 512 blocks (16 tokens each, so ~8,000 tokens of KV in flight at once), admits requests that arrive over time, and where 80% of them share a common prefix.

What you build.

• Cache Manager — owns the memory. It hands out blocks on demand, saves the cache, and evicts the least-recently-used cached blocks first when the block pool fills.
• Scheduler — decides who runs. It admits requests only when their blocks can be found, preempts a running one as a last resort, and batches work under one of two policies: prefill-first (favor new prompts) or decode-first (favor finishing in-flight requests).

On memory pressure the response escalates in order: use free blocks → evict cached prefixes → preempt a running request.

The experiment. Two workloads, built to stress opposite phases:

• Prefill-heavy — most of the requests share 256-token prefix and result in short outputs.
• Decode-heavy — small prompts, a tiny 32-token shared prefix, long outputs.

Each runs with prefix caching off and then on, and the harness measures total steps, time-to-first-token, and cache hit rate.

The results on my machine.

The takeaway is simple: when a long prefix is shared across many requests and kept in cache, reusing it instead of recomputing reduces the latency (time-to-first-token and end-to-end time). Moreover and also logically, caching lowers how much memory you need, that is why turning caching on cut preemptions sharply (91→18 on the prefill-heavy run, 273→159 on decode-heavy).