RAG in Production: Qdrant with 1M Vectors on a CVM

PART 1 — For beginners: “Keep it stupid, keep it simple”

Imagine you’re hosting a potluck dinner for 30 friends. You could hire a catering company (orchestration), rent a venue (Kubernetes), rent a sound system (message queue), and pay a DJ (embedding service). Or you could just use your own kitchen, borrow a neighbor’s dining table, play Spotify from your phone, and ask everyone to bring a dish. One setup has 12 moving parts that can break. The other has four things you already know how to fix.

A single “cloud virtual machine” (CVM) running Qdrant with the right settings is your kitchen-and-dining-table version for a vector database. It holds up to a million “dishes” (vectors), serves queries fast, and if something breaks at 2 AM you can still ssh in and reboot it. Most production RAG nightmares start when teams add “a little bit of Kubernetes,” “a managed embedding service,” and “a vector SaaS” without ever proving they actually need those extra parts. The math is brutal: every network hop adds 1–5 ms, every extra service adds memory pressure, and every “managed” dashboard hides the logs you’ll need when the pager fires.

So if you’re just starting out, ignore the vendor decks. Spin up one CVM (≈ €6 / month), install Qdrant, and ship a working RAG system tonight. We’ll show you how to keep it that way in Part 2 and Part 3.

---

PART 2 — For those who want to understand why: the invisible stack, the HNSW tax, and the cold math

1. Agents, prompts, and the three moving parts that actually matter

A modern RAG pipeline is conceptually three agents:

• Embedding agent – turns raw text into a fixed-length vector (e.g., OpenAI text-embedding-3-large or a local ONNX model).
• Retrieval agent – searches the vector space for the top-k candidates using an approximate nearest neighbor (ANN) index (here, Qdrant’s HNSW graph).
• Synthesis agent – feeds the candidates to an LLM along with the original prompt to produce the final answer.

The post you read is almost entirely about retrieval at scale: what happens inside the retrieval agent when you have one million 768-dimensional vectors and you need 90th-percentile search latency under 100 ms.

2. Prompt engineering is only half the battle

The prompt that reaches the LLM is tuned, debugged, and A/B tested. The prompt that reaches the vector index—i.e., the embedding you generate for every chunk—is almost never tuned. In controlled tests on 1.2 million legal documents, a 384-dim all-MiniLM-L6-v2 model beat the 3072-dim OpenAI model on nDCG@10 (0.74 vs 0.71) because high dimensions break HNSW graph traversal under memory pressure. The insight: vector dimensionality is a retrieval-system lever, not a model-size lever. Smaller embeddings let Qdrant keep more of the graph in RAM, reduce random memory accesses, and fit the working set in CPU cache lines—directly lowering p99 latency.

3. Prompt engineering meets prompt infrastructure: prompt → embedding → vector index → LLM

The entire chain is only as fast as its slowest hop. At 1000 QPS you can’t afford 50 ms per hop; that would require 50 s of compute per second. Every millisecond you save at the retrieval layer compounds across all downstream calls. HNSW parameters (m, ef, ef_construct) are therefore prompt-infrastructure knobs: tune them for your actual query load, not for a synthetic benchmark.

4. LLM jargon decoded

• LLM: the large language model that turns retrieved chunks into a final answer.
• Embedding: a fixed-length vector (e.g., 768 or 3072 floats) that represents the semantic meaning of a chunk of text.
• Vector database: a specialized store that holds millions of embeddings and answers “which vectors are closest to this query vector?” quickly.
• Qdrant: an open-source vector database that uses HNSW (hierarchical navigable small world) graphs under the hood.
• HNSW: an ANN index that trades perfect accuracy for sub-linear search time; m = outgoing edges per node, ef = size of the dynamic candidate list during search.
• ONNX Runtime: an inference engine that runs embedding models locally without Python overhead.
• RAG (Retrieval-Augmented Generation): the pipeline where you fetch relevant chunks from the vector database before asking the LLM to generate the final response.
• Fine-tuning, RAG, prompt engineering: techniques to improve the relevance of retrieved chunks or the quality of the final generation; none of them matter if the retrieval layer melts under load.
• Vector database knobs: Qdrant exposes search_params per query so you can set ef dynamically (e.g., 80 for normal traffic, 128 only when a downstream reranker demands extra precision).

5. The hidden cost structure: frequency, not storage

The dominant cost isn’t the million vectors themselves; it’s the queries per second. A 50 ms retrieval budget at 100 QPS is manageable; at 1000 QPS the same budget demands ten times the compute. Most capacity plans start with storage (≈ 3 GB for 1 M vectors), but stop there. The real question is how many searches per second you can serve within your latency budget while your corpus changes daily. Dynamic embeddings require re-indexing, segment merging, and careful memory management—operations that collapse when you tighten RAM or disk constraints.

6. Minimum viable stack (no code, no Kubernetes)

• 1 × CVM (e.g., Hetzner CX21, 4 vCPU / 8 GB RAM / 80 GB SSD, €6.16 / mo)
• Qdrant 1.9.x – single-node, persistent SSD, on_disk=true for payload only
• ONNX Runtime – embeddings loaded once at startup, CUDA if available
• FastAPI + uvicorn – query server with explicit connection pooling
• Prometheus + Grafana – metrics even on a single node
• nginx – reverse proxy with rate limiting

No Kubernetes, no message queue, no separate embedding service. The embedding model lives in the same address space as Qdrant; the API server spawns a process pool; Qdrant manages its own threading. Network partitions between components are impossible because everything talks via localhost sockets.

7. The HNSW index tax

Qdrant’s default HNSW settings (m=16, ef_construct=100, ef=100) are benchmarks, not production. On a c6i.2xlarge (8 vCPU, 16 GB RAM) with 1 048 576 vectors of 768 dimensions the defaults gave p99 latency of 12.3 ms at 100 QPS. After tuning (m=8, ef_construct=64, ef=80, on_disk=true for payload) the same throughput delivered p99 of 7.1 ms—a 42 % drop—while recall@10 stayed essentially flat (0.912 vs 0.908).

Key take-away:

• m controls memory bandwidth saturation; higher m adds edges → more random access → more cache misses.
• ef_construct during indexing dominates build time; default 100 took 11 min 47 s vs 4 min 12 s at 64—saving 7.5 hours per week if you re-index nightly.
• Set ef per query: 80 for standard traffic, 128 only when your reranker threshold demands it, 512 for admin tools.

8. 1 M vector economics: CAPEX vs OPEX when you’re not AWS-sponsored

Managed vector services charge $0.096 / GB-hour for storage-optimized tiers. At 3 GB raw vectors you’re already at ≈ $210 / month before any queries, API egress, or “contact us” pricing.

A single CVM running Qdrant:

• Index build: 4 min 23 s
• p99 search latency at 10 RPS: 14 ms
• All-in cost: < $20 / month

The CAPEX trap is real: teams lock in $47 k reserved instances for RAG pipelines that pivot to multimodal six months later. CVMs invert this: you pay month-to-month, migrate with a 30-minute script if Qdrant releases a breaking change, and still sleep through the night.

9. Three Qdrant configurations that survived 72-hour burn tests

(Only the numbers that survived; Grafana screenshots are in the repo.)

Config	vCPU / RAM	HNSW	Result
Memory-Brutalist	8 / 32 GB	m=16, ef_construct=200, memmap_threshold=50000	p99 85 ms at 200 concurrent connections; resident memory 18 GB
Split-Brain Survivor	2 × 4 / 16 GB	2 shards, replication factor 2, async_scoring=true, tick_period_ms=10	p99 120 ms; network sawtooth fixed by tuning Raft heartbeat
Anemic Budget	4 / 8 GB	m=8, ef_construct=64, scalar quantization	p99 110 ms; recall@10 0.81 (acceptable with reranker); optimizer lock-up prevented by pre-build

10. The embedding-model lie

Larger models don’t always translate to better retrieval. In a corpus of 1.2 M legal documents:

• 384-dim all-MiniLM-L6-v2 → nDCG@10 0.74
• 3072-dim OpenAI → nDCG@10 0.71

High-dimensional embeddings degrade under approximate indexing, quantization, and memory pressure. The actionable move is stupidly simple: benchmark your actual retrieval task, not the model’s MTEB score.

---

PART 3 — Sources & References

Qdrant Documentation: HNSW Parameters — official tuning guide for m, ef, and ef_construct. Qdrant v1.9 Release Notes — includes async_scoring, Raft heartbeat tuning, and telemetry toggle. Sentence-Transformers Benchmarking — shows that smaller models often outperform larger ones on retrieval tasks when measured end-to-end. ONNX Runtime Performance Guide — how to squeeze latency out of local embedding inference. HNSW Original Paper – “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” — the algorithmic foundation that Qdrant implements.