Multimodal on-premise: serving a VLM with vLLM (vision + language)

· By José Manuel Román
Contents

Third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM. Its sibling pieces in this batch are FinOps and GPU multi-tenancy with LiteLLM, which puts a price on every token, and a visual token costs the same as a text one, and Speeding up cold start with Tensorizer, which matters twice as much here because a VLM loads two models (the vision encoder and the LLM). If the VLM ends up feeding a document assistant, that end-to-end setup with LibreChat and RAG is another story (in preparation).

TL;DR

A vision-language model (VLM) is not magic: it is a normal LLM with a sensory organ bolted on the front. Three pieces in series: a vision encoder (a ViT) that looks at the image, a projector or connector (typically an MLP) that translates the encoder’s output into the LLM’s embedding space, and the usual LLM, which receives those embeddings as if they were just more tokens. The consequence that governs the entire operational design is brutal in its simplicity: an image becomes visual tokens, and those tokens cost exactly the same as text tokens. A high-resolution document page is not “a couple of image tokens”: it is hundreds or thousands of tokens entering the context, inflating the prefill (compute-bound) and occupying KV cache (memory-bound) just as if you had pasted a thousand words. In Qwen2.5-VL the arithmetic is literal: the number of visual tokens is $H\times W / (14\times14\times4)$, so an $896\times896$ image is 1,024 tokens and an A4 page scanned at a decent resolution easily reaches 1,500–2,500 tokens for the image alone. This post opens up the anatomy, works through the token-cost maths and translates it into TTFT and VRAM on an 80 GB H100, explains vLLM’s multimodal support (--limit-mm-per-prompt, pixel budget via mm_processor_kwargs, chat template) and answers the question that really matters: classic OCR or VLM. The short answer: use Tesseract/PaddleOCR for the bulk of documents with clean text, and reserve the VLM, expensive in tokens and in GPU, for the subset that genuinely needs it: complex layout, nested tables, handwriting, stamps, charts. On the generic 4×H100 SXM cluster, with FP8 so it fits.

The analogy: the reader who sees the page versus the one who only hears the transcript

Picture two experts you ask to review a scanned contract. To the first you read the contract aloud, a blind transcript, word by word. It is fast, efficient, and for 90 % of the clauses it is enough: the text is the text. But there are things you cannot read out: that the signature at the end is handwritten and does not match the typed name; that there is a “PAID” stamp struck diagonally across the amount; that the table on page 3 has a merged cell that changes who each row applies to; that there is a pen annotation in the margin. The blind transcript loses all of that or flattens it into gibberish.

To the second expert you hand the whole page, to look at. They see the signature, they see the stamp, they understand the structure of the table because they perceive the lines and the cells, they read the marginal note. They grasp the document as a visual object with layout, not as a river of characters. They are strictly more capable.

Why not always use the second one, then? Because seeing costs more mental bandwidth. The first expert receives the contract as a flow of words: cheap, linear. The second receives it as an image that their brain has to parcel up, attend to region by region, and reconstruct. They process a great deal more information per page, most of which, in a clean text document, is redundant with what the transcript would already have given them. If you only need to read the clauses, hiring the expert-who-sees for every page is overpaying for capability you do not use.

That is the entire thesis of this post. The VLM is the expert who sees; OCR is the blind transcript; and “mental bandwidth” is tokens. The engineering consists of sending the expert-who-sees only the pages where seeing changes the answer, and resolving the rest with the cheap transcript.

Anatomy of a VLM: encoder, projector, LLM

A modern VLM of the Qwen-VL family (Qwen2.5-VL model card; Qwen3-VL Technical Report) is made up of three blocks in series:

  1. Vision encoder (ViT). A Vision Transformer that receives the image, cuts it into patches and produces an embedding per patch (or per group of patches). In Qwen2.5-VL it is a dynamic-resolution ViT: it accepts images of arbitrary size, resizing them to multiples of 28 and splitting them into 14×14 pixel patches, with window attention to speed up the processing of large images. There is no “canonical size” everything is squashed to: a large image produces more patches than a small one.

  2. Projector / connector. The ViT’s embedding space is not the LLM’s. The projector acts as the translator. In Qwen2.5-VL it is elegant and cheap: it groups every 2×2 block of adjacent patches (four ViT tokens), concatenates them and projects them with a two-layer MLP into a single fused token in the LLM’s dimension. That fusion factor of 4 is the reason for the $4$ that appears in the denominator of the token formula below: four ViT patches collapse into one visual token that the LLM sees.

  3. The LLM. The usual language transformer. It receives a sequence of embeddings that is a mixture of text tokens (from your prompt) and visual tokens (from the image), all in the same space, and does what it knows how to do: attention over the complete sequence and autoregressive generation. To the LLM, a visual token and a text token are indistinguishable in cost: both go through the same QKV projections, both occupy an entry in the KV cache, both take part in the $O(C^2)$ attention.

The VLM pipeline: where the token budget explodesImageA4 page~1100×1500 pxViT encoder14×14 patcheswindow attnProjector2×2 fusion → MLP÷4 tokensVisual tokens~1500–2500enter the contextLLMattention overtext + visionpixels~8400 patches÷4Here the budget explodes: the image is already text to the LLMprompt ~80 tokvisual tokens from the image ~1500–2500 tokprefill = process ALL of this (compute-bound) · KV cache = store ALL of this (memory-bound)in Qwen2.5-VL: visual tokens = H × W / (14 × 14 × 4)896×896 → 1024 tok · one image ≈ a long paragraph… or five pages of text

The idea to internalise: the encoder and the projector are the sensory organ, but the inference cost lives in the LLM, and that cost is paid in tokens. The ViT adds a fixed prefill overhead (processing the image once), but the recurring bill, the LLM prefill and the KV cache throughout the whole generation, is set by the number of visual tokens the projector spits out. That is why the control lever is not “which encoder”, but how many pixels you let through.

The maths: what it costs to see a page

Let us put numbers on “an image costs many tokens”, with the real Qwen2.5-VL formula (Qwen team, Qwen2.5-VL blog):

$$N_{\text{tok}} = \frac{H \times W}{14 \times 14 \times 4} = \frac{H \times W}{784}$$

where $H$ and $W$ are height and width in pixels (rounded to the multiple of 28 the ViT imposes). The $14\times14$ is the patch size; the $\times4$ is the projector’s 2×2 fusion. A sanity check against the model card figure: an $896\times896$ image gives

$$N_{\text{tok}} = \frac{896 \times 896}{784} = \frac{802{,}816}{784} = 1{,}024 \text{ tokens.}$$

It checks out. Now an A4 page scanned at a resolution reasonable for reading small text, say $1120 \times 1568$ px (approximate height $\times$ width, already rounded to multiples of 28):

$$N_{\text{tok}} = \frac{1120 \times 1568}{784} = \frac{1{,}756{,}160}{784} \approx 2{,}240 \text{ tokens.}$$

A single page is ~2,240 visual tokens before you write a word of prompt. For context: that is roughly what 1,700 words of Spanish text would take up (at ~1.3 tokens/word). If that A4 page contains 600 words of actual text, the VLM is paying almost 4× in tokens to see it rather than read its transcript. If the page is a dense table or a handwritten document that OCR cannot read, that 4× is justified. If it is a paragraph of clean text, it is pure waste.

Qwen2.5-VL lets you control this with min_pixels / max_pixels: the number of tokens per image is dynamic and ranges from about 4 up to 16,384 by default, adjustable with those parameters (HF Qwen2.5-VL docs). Lowering max_pixels cuts tokens at the expense of resolution, and of the ability to read the fine print. It is exactly the lever of “how much mental bandwidth do I give the expert”.

From token cost to TTFT

Visual tokens are not free in latency. The prefill, processing the whole context before the first token, is compute-bound and its cost grows with the context length $C$ (linear in the projections, quadratic in the attention); this is worked through in detail in vLLM’s attention backend. Every visual token enters that $C$.

Let us model the TTFT as the time to process the prefill tokens at a given prefill throughput. Take a VLM of around 7–8B on an H100 with an illustrative prefill throughput of ~12,000 tok/s (an example figure; measure it, do not assume it, and it depends on how much the ViT forward pass weighs on top). For a request with an 80-token text prompt plus one page at 2,240 visual tokens:

$$\text{TTFT} \approx \frac{80 + 2240}{12{,}000 \text{ tok/s}} \approx \frac{2320}{12{,}000} \approx 0{.}19 \text{ s}$$

plus the fixed cost of the vision encoder’s forward pass over the ~9,000 ViT patches (another compute-bound slice of prefill). Now five pages in the same request (a short PDF):

$$\text{TTFT} \approx \frac{80 + 5 \times 2240}{12{,}000} = \frac{11{,}280}{12{,}000} \approx 0{.}94 \text{ s}$$

The TTFT multiplies almost fivefold, because the visual tokens completely dominate the context. And attention scales as $O(C^2)$: five pages are not “5× more expensive” in the attention component, but ~25× relative to one. The operational lesson: the number of images and their resolution are your TTFT budget, not a detail. Lowering max_pixels from 2,240 to, say, 1,100 tokens/page (lower resolution, enough for large text) cuts the prefill almost in half.

VRAM: the VLM’s weights and what is left for KV

On an 80 GB H100 SXM, let us budget a VLM of the Qwen-VL family. Take an 8B (LLM) plus the vision encoder (a ViT of a few hundred million parameters; call the whole thing ~8.5B effective parameters for weight purposes). In BF16 (2 bytes/parameter):

$$M_{\text{BF16}} \approx 8{.}5 \times 10^9 \times 2 \text{ B} \approx 17 \text{ GB.}$$

In FP8 (1 byte/parameter), the weights halve (Qwen3-VL-8B-Instruct-FP8, with reported quality almost identical to BF16):

$$M_{\text{FP8}} \approx 8{.}5 \times 10^9 \times 1 \text{ B} \approx 8{.}5 \text{ GB.}$$

For an 8B model this fits comfortably in BF16 on an H100; FP8 matters more for leaving VRAM free for KV cache, and here the KV is the problem, because visual tokens inflate it. The detail of end-to-end FP8 (weights and KV) is in FP8 end to end. What is relevant: one page is ~2,240 KV entries; serving several sessions that send long documents consumes KV at a rate a text-only case would never see. With a 32B VLM in FP8 (~32 GB of weights) on an 80 GB H100 you are left with ~45 GB for KV and activations, which with long multimodal contexts run out sooner than “text-only” intuition suggests. The rule: with a VLM, plan the VRAM counting the visual tokens in the KV, not just the weights.

Serving a VLM with vLLM

vLLM supports image multimodal input natively for the Qwen-VL family and many other VLMs (Multimodal Inputs, vLLM docs). The gears that matter:

--limit-mm-per-prompt. Limits how many multimodal items each prompt accepts, per modality. By default the limit is generous (999 per modality), but in production it is worth setting it low, --limit-mm-per-prompt '{"image": 2}', because every accepted image reserves token and KV budget. If you set a limit of 8 images and someone sends 8 A4 pages, that is ~18,000 visual tokens in a single request: it blows up max_model_len or the KV. The limit is a budget firebreak, not a cosmetic restriction. (A curious case: setting it to 0 for a modality lets you pass precomputed embeddings without loading the encoder module, saving VRAM, useful if you do the vision encoding outside vLLM.)

Pixel budget via mm_processor_kwargs. Here the token formula connects with the configuration. When serving Qwen2.5-VL you pass, for example, --mm-processor-kwargs '{"min_pixels": 12544, "max_pixels": 254016}' (vLLM Qwen2.5-VL discussion). With $254{,}016$ maximum pixels per image, the token ceiling per image is $254{,}016 / 784 \approx 324$ tokens, an aggressive trim that keeps the cost in check at the price of not being able to read very small print. This is the dial that really governs your token bill: raise it for dense documents, lower it for images where fine detail does not matter.

The chat template and the image placeholder. The model’s processor (loaded via AutoProcessor) inserts special tokens marking where the image goes in the sequence: in Qwen2.5-VL the sequence <|vision_start|><|image_pad|><|vision_end|>, where <|image_pad|> expands to the exact number of visual tokens the image produced. vLLM applies this template automatically; it is worth knowing there is a documented quirk, the image block tends to be placed before the user’s text in the sequence order, regardless of where you put it in the content (issue #15125, vllm-project/vllm), which rarely causes trouble but is worth bearing in mind if the image/text order is semantically important for your prompt.

Current models. As of June 2026, the reference open family is Qwen3-VL (dense 2B/4B/8B/32B variants and MoE 30B-A3B / 235B-A22B, interleaved contexts of up to 256K tokens combining text, image and video), with published FP8 checkpoints of almost identical quality to BF16 (Qwen3-VL Technical Report, arXiv 2511.21631; Qwen3-VL Usage Guide, vLLM recipes). Qwen2.5-VL remains perfectly serviceable and its token formula is the one we have used because it is public and clean; the cost figures are of the same order in both generations.

Classic OCR vs VLM: when to use which

Here is the decision that pays the bill. Parsing documents to feed them into a document ingestion pipeline has two paths, and choosing the split between them well is the difference between a healthy cluster and a choking one.

Classic OCR (Tesseract, PaddleOCR). Text-recognition engines that run on CPU, are cheap, fast and mature. Tesseract is the portable workhorse (edge, offline kiosks, embedded scanners). PaddleOCR is more capable on layout and Asian scripts, with its PP-Structure module for preserving structure, extracting tables and detecting key-value fields (PaddleOCR vs Tesseract, CodeSOTA 2026). On clean, well-scanned text, classic OCR is hard to beat on cost per page: zero GPU, millisecond latency, very high throughput. Its weakness is the “hard” document: complex multi-column layout, tables with merged cells, handwriting, stamps, charts, poor scan quality. There, pipeline-based OCR either fails or returns gibberish that destroys the structure.

VLM. It takes the image of the document and generates structured markdown or HTML in a single end-to-end pass, understanding the layout because it sees it. It shines exactly where classic OCR struggles: nested tables, reading charts, handwriting, understanding the spatial relationship between elements. The price is the one we have been calculating all through this post: expensive in tokens and in GPU. Worth noting a 2026 nuance: the frontier has moved. VLM models specialised in document parsing have appeared, compact (~1–3B), open source, scoring very high on benchmarks such as OmniDocBench (PaddleOCR-VL in the 94–96 point band; dots.ocr 3B at ~88 with full markdown including tables) (LLM-OCR vs Traditional OCR, Parsli 2026; Open-Source OCR Models, E2E 2025). In other words: for pure parsing, today you may not want a 32B generalist VLM, but a small specialised VLM-OCR. But it is still GPU and it still pays tokens per image: the cost calculation does not disappear, the model just gets cheaper.

Classic OCR or VLM? Per-document triageIs the text clean andthe layout simple?YES → classic OCR (CPU)Tesseract / PaddleOCR · cheap · ms · no GPU80–90 % of documents land hereNO → VLM (GPU)tables, handwriting, stamps, charts, layouttoken-expensive · save it for the hard subsetyesnoThe optimal split: cheap triage first, VLM only for what needs ita light classifier (any tables? low OCR confidence? handwriting?) picks the routeso the VLM only touches 10–20 % of pages, and the GPU does not drown in clean text

The split rule. It is not “OCR or VLM” as a global choice, but per-document triage. Run everything through a cheap first filter: classic OCR already returns a confidence score; if it is high and the document has no tables or visual elements, the OCR transcript will do and you have spent CPU. If confidence is low, there are tables, there is handwriting or there are graphical elements, escalate that page to the VLM. In a typical corpus, 80–90 % of pages are resolved with cheap OCR and only the rest consumes GPU. This is what makes an on-premise VLM sustainable: you do not serve everything with it, you use it as the expensive specialist you send only the hard cases.

Applied to the generic 4×H100 cluster

How this fits into a cluster of 4×H100 SXM 80GB:

Where to place the VLM. A VLM deserves a whole H100 or a large slice, for a VLM-specific reason that text-only intuition does not anticipate: the token cost per image means not many concurrent sessions fit. If each request brings one or several pages, each consumes thousands of KV tokens; the KV cache runs out with few simultaneous sessions. Compared with a text-only chat service where dozens of short conversations fit per GPU, a document service saturates the KV with a handful of heavy requests. That is why it makes no sense to carve the GPU finely for the VLM with aggressive MIG (GPU sharing is covered in sharing a GPU: time-slicing, MPS and MIG): the bottleneck is not idle compute but the KV, and a small slice runs out of KV before it makes use of the compute.

FP8 so it fits and to free up KV. Serve the VLM in FP8: at equal reported quality, the weights take up half and you leave the maximum VRAM for the KV cache, which is the scarce resource with multimodal contexts. For a 32B in FP8 (~32 GB) on an H100, those ~45 GB free for KV are what determine how many concurrent pages you can handle.

The VLM only for the subset that needs it. The most important architectural piece: do not put the VLM on the critical path of every document. The triage from the previous section lives upstream; the bulk (clean text) is resolved with OCR/PaddleOCR on CPU, and in a cluster with 4×H100 there is probably spare CPU to run it in parallel with the GPUs, and only the hard subset reaches the VLM. This frees the H100s for what really needs them and keeps a reasonable cost per document. It connects directly with FinOps and GPU multi-tenancy with LiteLLM: when you put a price on every token, a visual token costs the same as a text one, so a page through the VLM can cost 4× its transcript, and that should show up in the chargeback of the team that decides to send everything to the VLM for convenience.

Double cold start. An operational detail that bites: a VLM loads two models into memory, the vision encoder and the LLM, so its cold start is heavier than that of an equivalent LLM. If you are going to do model swapping or on-demand scaling, fast loading matters twice as much; that is where speeding up cold start with Tensorizer comes in. If instead you serve several models on one GPU by rotating them, the swap+sleep pattern (serving several models on one GPU) has to account for the VLM taking up more room and loading more slowly.

Consumer GPU, for reference. If someone wants to prototype outside the cluster, an RTX 5090 (Blackwell, 32 GB) comfortably serves a 7–8B VLM in FP8 (~8.5 GB of weights) with KV to spare for a few pages per request; a 32B in FP8 (~32 GB) no longer fits with useful KV on that card and calls for the H100. The 5090 is good for validating the triage pipeline and the prompt; the real service belongs on the H100s.

What we have not covered

  • Video. Qwen3-VL handles video with dynamic FPS sampling; each frame is an image and the token cost multiplies by the number of sampled frames. One minute of video can be tens of thousands of tokens. It is the most expensive multimodal beast and deserves its own analysis.
  • Prefix caching of images. If the same image (a logo, a template) appears in many requests, can its KV be cached? It depends on the backend and on the stability of the placeholder; it connects with the attention backend.
  • Evaluation. How to measure that the VLM extracts the table correctly (it is not enough for it to “look right”): faithfulness against a set with structured ground truth.
  • Fine-tuning the VLM for a specific document domain (invoices, particular forms), which can put part of the domain’s “knowing how to see” into the weights and reduce what the prompt has to explain.

See also

References