Vector database vs fine-tuning: how to choose the RAG architecture without overpaying on embeddings

Your company wants to use an LLM with its own documentation. The typical conversation starts with ‘we need to fine-tune the model with our data’ and ends 6 months and $200k USD later with a model that still hallucinates about the same topics as in your first attempt. The pragmatic alternative — Retrieval Augmented Generation (RAG) with a vector database — solves 80% of the cases at a fraction of the cost. But not every case. This is the real decision behind the decision.

What each architecture solves

Fine-tuning adjusts the model’s internal weights so it ‘learns’ the knowledge from your documents. Once trained, the model answers without consulting anything external. Vector RAG leaves the model intact and, on every question, searches a vector database for the most relevant fragments of your documents and injects them into the prompt. These are fundamentally different approaches.

When fine-tuning is the right choice

Fine-tuning makes sense in three very specific scenarios:

1. Changing the response style or format. For example, you want the model to respond in a specific tone, with a particular structure, or following a strict JSON format. That is feasible with fine-tuning and very difficult with RAG.
2. Teaching the model a completely new domain with curated synthetic data. For example, training a technical support model in proprietary jargon that does not exist in public internet data. This is what vertical models like BloombergGPT do, or the support models from hyperscalers.
3. Reducing latency at the cost of quality. A smaller fine-tuned model (7B parameters) can respond in 200 ms with acceptable accuracy, while RAG over a large model (70B+) takes 1-3 seconds due to retrieval + generation.

When vector RAG is the right choice

RAG is the right option in most commercial cases:

1. Documentation that changes frequently. With RAG you update the vector database when a new document arrives and the model ‘sees’ it immediately. With fine-tuning, you have to retrain, which takes weeks.
2. You need source attribution. RAG can return ‘this answer is based on paragraphs X, Y, Z of document Q’ with citations. Fine-tuning loses attribution because the knowledge is baked into the weights.
3. Large documentation (>1M pages). Fine-tuning has a practical limit on how many tokens the model can absorb. RAG scales to millions of documents without issue.
4. Tight budget and timeline. An RAG MVP can be built in 2-4 weeks with a team of 2 engineers. A fine-tuning project takes 3-6 months and requires data engineers, ML engineers, and training GPUs.

How to choose your vector database

Once the RAG route is decided, the operating cost depends heavily on the vector database. There are three main families:

• Postgres with pgvector. The cheapest option if you already have Postgres. Embeddings limited by RAM (typically 1-10M practical vectors). Cost: add an extension + a bit more RAM. No per-query licensing.
• Dedicated Vector DBs (Pinecone, Weaviate, Qdrant, Milvus). Optimized for search at scale. Handle 100M-1B+ vectors with consistent latency. Cost: $0.10-0.50 USD per million vectors stored + cost per query. Requires operating additional infrastructure.
• Hybrid solutions (pgvector + replicas, OpenSearch, Elasticsearch with kNN). If you already have OpenSearch or Elasticsearch, you can add vector retrieval without a new operational component.

When embedding cost eats your ROI

The most common hidden cost of RAG is generating embeddings. To index 1M single-page documents, you need ~3M embeddings (each document is split into chunks of 200-500 tokens). At $0.10 USD per million tokens with OpenAI text-embedding-3-small, that is ~$30 USD. If you do it with Cohere or with an open-source model on your own GPU, the cost drops to ~$5 USD or free on in-house compute. But if you have to reindex everything every time you update the documentation, the cost multiplies. The typical optimization is: embeddings for stable content + incremental reindexing for content that changes.

The mistake that kills RAG projects

The most common mistake is not one of technology, it is one of expectation. RAG does not ‘think’ about your documents: it searches for the most semantically similar ones and injects them into the prompt. If your question requires crossing information between 3 different documents, or if the answer depends on logical reasoning over the data (not just search), RAG will disappoint you. The solution is to decompose the problem: retrieval for factual search + tool calling for reasoning + LLM for the final synthesis. That is no longer pure RAG, it is ‘agentic RAG’, and it is its own topic.

Practical recommendation

Start with RAG in Postgres with pgvector. If you have fewer than 1M vectors, you do not need anything else. If it grows, migrate to Qdrant or Weaviate self-hosted. Avoid Pinecone until you have a validated use case and approved budget, because the per-query cost scales poorly when you have real adoption. And above all: do not fine-tune until you have evidence that RAG does not solve your case. In 80% of projects, RAG is enough.

Sources

1. pgvector — GitHub repository (open source vector extension for Postgres) — https://github.com/pgvector/pgvector
2. Pinecone — RAG learning series — https://www.pinecone.io/learn/series/rag/
3. Weaviate — Developer documentation — https://weaviate.io/developers/weaviate