Skip to content

Why there are two backends

rtb-chat presents one AiClient and, behind it, runs two entirely different pieces of machinery: the genai crate for OpenAI, Gemini, Ollama and OpenAI-compatible endpoints, and its own reqwest calls against the Anthropic Messages API.

That looks like an inconsistency worth removing. It is the central design decision, and removing it would cost more than it saves.

A unified interface can only expose the common subset

The reason to want one client is obvious enough. One user has an Anthropic key, another an OpenAI key, someone is on Gemini, someone runs Ollama locally because their data cannot leave the building. A tool that hard-wires one vendor is a tool that half its users cannot run.

The catch is what "unified" means. An interface over several providers can only offer what all of them have in common, and the common subset is plain chat: messages in, text out, optionally streamed. That is genuinely useful, and it is also the floor.

The features that make a particular provider worth choosing are never on the floor. They are the things only that provider does. Prompt caching, extended thinking and citations are Anthropic's. Reduce the interface to the intersection and you have built something that talks to five providers and gets the best out of none of them.

The direct path buys three specific features

The Anthropic-direct backend exists for three capabilities, and it would not exist without them:

  • Prompt caching. Marking the system prompt and the first user block as ephemeral so a long, stable prefix is not re-billed on every turn. It shows up in Usage::cache_read_input_tokens, which is the only place the saving is visible.
  • Extended thinking. A token budget for reasoning the model does before it answers, surfaced separately as ChatStreamEvent::ThinkingToken so a caller can show it, hide it, or bill it differently.
  • Citations. Spans of source text the model quotes, with offsets, returned as structured Citation values rather than left in the prose to be regexed out.

Each of those is a wire-format feature. Wrapping a library that does not send the field cannot produce it, so the choice was between doing without and writing the HTTP by hand for one vendor.

Why not hand-write all of them

Because the other four do not pay for it. Speaking OpenAI, Gemini and Ollama natively means four more request builders, four more response parsers, four more sets of streaming quirks, and a standing obligation to track four vendors' API changes — in exchange for features rtb-chat does not currently expose.

genai already carries that maintenance. Delegating it is worth the seam.

What the seam costs

The seam is not free, and the honest accounting is that it leaks.

ChatRequest has fields that only one backend honours. Some of that is inherent — cache_control cannot mean anything to Ollama. Some of it is not: temperature, max_tokens, base_url and timeout are ordinary settings that any provider could honour, and today they are dropped on the genai path. Those are defects, and they are listed in What rtb-chat does not do.

The error surface leaks the same way. AiError::Auth and AiError::RateLimited exist because the Anthropic-direct path maps status codes itself; genai reports failures through its own error type, and everything it says arrives as AiError::Provider.

Reading a behaviour question as "which backend was that?" answers it most of the time. That is the seam being visible from the outside, which is preferable to it being invisible and surprising.

Why the crate is named after chat, not AI

It was rtb-ai inside the rust-tool-base monorepo, and was renamed to rtb-chat when it was extracted into its own repository, to converge with the Go toolkit's chat module. The two are siblings by intent: the same idea, one bet per language.

Only the crate identity changed. The types, the fields and the behaviour came across as they were — which is why the miette diagnostic codes still read rtb::ai::*. See Errors.