How to Stream LLM Responses in Spring AI (Server-Sent Events)

@RestController
public class ChatController {

    private final ChatService chatService;

    public ChatController(ChatService chatService) {
        this.chatService = chatService;
    }

    @GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> stream(@RequestParam String message) {
        return chatService.stream(message);
    }
}

Each emitted chunk is sent to the client as its own data: event the moment the model produces it. If you want explicit SSE events (with ids, event names, or heartbeats), map to ServerSentEvent:

@GetMapping(value = "/api/chat/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
public Flux<ServerSentEvent<String>> stream(@RequestParam String message) {
    return chatService.stream(message)
            .map(chunk -> ServerSentEvent.builder(chunk).build());
}

No WebFlux? Use SseEmitter in Spring MVC

Plenty of enterprise apps are still on the classic servlet stack. You don't need to migrate to WebFlux — Spring MVC's SseEmitter streams fine. You subscribe to the Flux and push each chunk onto the emitter, completing (or erroring) it when the stream ends:

@GetMapping("/api/chat/stream")
public SseEmitter stream(@RequestParam String message) {
    SseEmitter emitter = new SseEmitter(Duration.ofMinutes(2).toMillis());

    chatService.stream(message).subscribe(
            chunk -> {
                try {
                    emitter.send(SseEmitter.event().data(chunk));
                } catch (IOException e) {
                    emitter.completeWithError(e);
                }
            },
            emitter::completeWithError,   // onError
            emitter::complete             // onComplete
    );

    return emitter;
}

Consume the stream on the frontend

For a GET endpoint, the browser's built-in EventSource is the simplest consumer. Concatenate the chunks as they arrive:

function streamChat(message, onToken, onDone) {
  const url = `/api/chat/stream?message=${encodeURIComponent(message)}`;
  const source = new EventSource(url);

  source.onmessage = (event) => onToken(event.data);

  source.onerror = () => {
    // SSE closes with an error event when the server finishes the stream
    source.close();
    onDone();
  };
}

// Usage
let answer = "";
streamChat("Explain vector databases in one paragraph.",
  (token) => { answer += token; render(answer); },
  () => console.log("done"),
);

If you need POST (to send a large body or headers), EventSource won't work — use fetch with a streaming reader instead:

const res = await fetch("/api/chat/stream", {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ message }),
});
const reader = res.body.getReader();
const decoder = new TextDecoder();
let answer = "";
while (true) {
  const { value, done } = await reader.read();
  if (done) break;
  answer += decoder.decode(value, { stream: true });
  render(answer); // you'll get raw SSE `data:` frames here — strip the prefixes
}

The flow, end to end

Production details that actually bite

Streaming works in a demo on the first try. These are the things that break it in production:

• Proxy buffering. Nginx and many API gateways buffer responses by default, which defeats streaming — the client gets everything at once. Disable it for the stream route (proxy_buffering off; in Nginx) and send X-Accel-Buffering: no.
• Compression. Response compression can also buffer the stream. Exclude text/event-stream from gzip.
• Cancellation. When the user navigates away, the browser closes the connection. In WebFlux this cancels the Flux automatically (and stops the upstream LLM call); with SseEmitter, wire an onCompletion/onTimeout callback to dispose the subscription so you're not billed for tokens nobody will read.
• Errors mid-stream. Once the first byte is sent you can't change the HTTP status. Emit a final SSE event with an error payload (e.g. event: error) so the client can distinguish "done" from "failed."
• Resilience. LLM calls are flaky network calls. Apply timeouts and a fallback path — see AI system resilience for the patterns.
• CORS. If your frontend is on a different origin, SSE still needs the usual CORS headers on the streaming route.

FAQ

Does streaming reduce token cost? No — you're billed for the same input/output tokens. Streaming only changes when they arrive, improving perceived latency.

Do I have to switch to WebFlux to stream? No. SseEmitter on classic Spring MVC streams a Flux fine. WebFlux is cleaner if you're already reactive, but it's not required.

Why does the client get everything at once instead of token-by-token? Almost always a buffering proxy (Nginx/gateway) or response compression. Disable buffering and exclude text/event-stream from gzip.

Can I stream structured output (JSON) too? You can stream the raw text, but partial JSON isn't parseable until it's complete. For structured responses, prefer .call().entity(...) unless you specifically want to render tokens as they arrive.

How do I stop the LLM call when the user cancels? In WebFlux, closing the connection cancels the Flux and the upstream request automatically. With SseEmitter, dispose the reactive subscription in an onCompletion/onTimeout handler.

Streaming is the highest-leverage UX change you can make to an LLM feature, and Spring AI reduces it to swapping .call() for .stream(). Get the endpoint right, disable proxy buffering, handle cancellation, and your app will feel dramatically faster for zero extra cost.

New to Spring AI? Start with the Spring AI overview, then see tool calling and the enterprise Spring AI guide. Planning capacity and cost for a streaming feature? Estimate prompt size with the free LLM Token Counter.

Building a production AI feature and want a second pair of eyes on the architecture? Let's talk.