Building a RAG Chatbot with Spring AI and PGVector

You already run PostgreSQL. So before you reach for a dedicated vector database, know this: with the pgvector extension and Spring AI, Postgres is a perfectly good vector store for most RAG workloads — one less system to operate. This guide builds a grounded RAG chatbot end to end: ingest documents, embed and store them in pgvector, retrieve the relevant chunks, and let the model answer using only your data.

The architecture

Two flows: an ingestion pipeline (offline: docs → chunks → embeddings → pgvector) and a query pipeline (online: question → retrieve → prompt → answer).

Dependencies

Add the chat model, the embedding model, and the pgvector store starter:

<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-model-anthropic</artifactId>
</dependency>
<dependency>
  <groupId>org.springframework.ai</groupId>
  <artifactId>spring-ai-starter-vector-store-pgvector</artifactId>
</dependency>

Configure the datasource and let Spring AI initialise the schema:

spring:
  datasource:
    url: jdbc:postgresql://localhost:5432/rag
    username: rag
    password: rag
  ai:
    vectorstore:
      pgvector:
        initialize-schema: true
        dimensions: 1024   # match your embedding model
        index-type: HNSW

Enable pgvector

In your Postgres database, the extension must exist (Spring AI creates the table/index, but the extension is a one-time DB step):

CREATE EXTENSION IF NOT EXISTS vector;

The ingestion pipeline

Read your source documents, split them into retrieval-sized chunks, and store them — Spring AI embeds each chunk automatically as it writes to the VectorStore:

@Service
public class IngestionService {

    private final VectorStore vectorStore;

    public IngestionService(VectorStore vectorStore) {
        this.vectorStore = vectorStore;
    }

    public void ingest(Resource file) {
        // 1. Read (Tika reader handles PDF, DOCX, HTML, etc.)
        var reader = new TikaDocumentReader(file);
        List<Document> docs = reader.get();

        // 2. Split into chunks
        var splitter = new TokenTextSplitter();
        List<Document> chunks = splitter.apply(docs);

        // 3. Embed + store (embedding happens inside add())
        vectorStore.add(chunks);
    }
}

Chunking quality caps retrieval quality — if answers are vague or miss context, revisit your chunk size and overlap before touching the model. See RAG chunking strategies for the details.

The query pipeline

The clean way to do retrieval-augmented answers in Spring AI is the QuestionAnswerAdvisor. It retrieves the top matching chunks and injects them into the prompt for you:

@Service
public class RagService {

    private final ChatClient chatClient;

    public RagService(ChatClient.Builder builder, VectorStore vectorStore) {
        this.chatClient = builder
                .defaultAdvisors(new QuestionAnswerAdvisor(vectorStore))
                .build();
    }

    public String ask(String question) {
        return chatClient.prompt()
                .user(question)
                .call()
                .content();
    }
}

Want to tune how much context is retrieved? Configure the advisor with a search request — topK (how many chunks) and a similarity threshold (filter weak matches):

new QuestionAnswerAdvisor(vectorStore, SearchRequest.builder()
        .topK(5)
        .similarityThreshold(0.7)
        .build());

Expose it

@RestController
public class RagController {

    private final RagService ragService;

    public RagController(RagService ragService) {
        this.ragService = ragService;
    }

    @PostMapping("/api/ask")
    public String ask(@RequestBody AskRequest req) {
        return ragService.ask(req.question());
    }
}

Combine this with streaming and you have a responsive, grounded chatbot.

Production gotchas

• Dimension mismatch — the embedding model's output size must equal the configured dimensions. Set it explicitly.
• topK too high — stuffing 20 chunks wastes tokens and dilutes the answer. Start at 4–6 and tune.
• No similarity threshold — without one, you always retrieve something, even for off-topic questions. A threshold lets the bot say "I don't know."
• Metadata filtering — store tenant/source metadata on each chunk so you can scope retrieval (critical for multi-tenant apps).
• Re-embedding on model change — if you switch embedding models, you must re-embed the whole corpus; vectors from different models aren't comparable.

FAQ

Is pgvector good enough for production RAG? For most workloads (up to millions of chunks with HNSW indexing), yes — and it saves you operating a separate vector DB. Reach for a dedicated store only when you hit its scaling or feature limits.

How big should chunks be? Start around 300–500 tokens with light overlap, then tune based on retrieval quality. See RAG chunking strategies.

Can I stream the RAG answer? Yes — swap .call() for .stream(); retrieval happens first, then the answer streams. See streaming responses.

How do I stop it from answering outside my data? Set a similarityThreshold and instruct the model (via system prompt) to answer only from the provided context and say when it doesn't know.

That's a complete, grounded RAG chatbot on infrastructure you already run. The framework work is small; your retrieval quality — chunking, topK, thresholds — is what makes or breaks it.

More: the Spring AI overview, RAG systems explained, and evaluating RAG with Ragas.

Building enterprise RAG and want the architecture reviewed? Let's talk.