Skip to content

What rtb-chat does not do

Documentation that only describes what software does leaves you to infer the rest from silence. This page states the gaps.

Two kinds of gap are mixed here on purpose, but they are labelled: deliberate omissions, which are not going to change soon, and defects, where the crate does not do what its own API implies. The behaviour described is what v0.7.2 actually does either way.

A custom base_url is ignored on every genai-backed provider

Defect.

Config::base_url is validated for Provider::OpenAi, Provider::OpenAiCompatible, Provider::Gemini and Provider::Ollama, and then never used. rtb-chat builds a genai::Client::default() and passes it no endpoint, so genai resolves its own.

Which means the headline use case for Provider::OpenAiCompatible — pointing at Together, Fireworks, vLLM or an in-house gateway — does not work. The request goes to whatever endpoint genai picks from the model name, and if the name matches nothing genai recognises, that endpoint is http://localhost:11434/ because Ollama is its fallback adapter.

Configured with a custom base URL and the model llama-3.1-70b-instruct, the request lands on localhost:

POST /api/chat HTTP/1.1        ← received by a listener on 127.0.0.1:11434

If something is listening there, there is no error and no warning — the only signal is the endpoint the request turns up at. If nothing is, the call fails against localhost rather than against the endpoint you configured:

provider error: Web call failed for model 'llama-3.1-70b-instruct (adapter: Ollama)'.
Cause: Reqwest error: error sending request for url (http://localhost:11434/api/chat)

What to do instead. Use Provider::AnthropicLocal with base_url set if your gateway speaks the Anthropic Messages format — that path honours the override. If it speaks OpenAI's format, there is no way to reach it through rtb-chat today.

Config::timeout is ignored on every genai-backed provider

Defect, and the same cause. The timeout is applied to the reqwest client that the Anthropic-direct backend builds. genai builds its own HTTP client and is given no timeout, so Config::timeout has no effect on four of the six providers.

temperature and max_tokens are silently dropped on genai providers

Defect. ChatRequest::temperature and ChatRequest::max_tokens are read only by the Anthropic-direct request builder. The genai request builder copies across the system prompt and the messages, and nothing else.

Set both on an OpenAI-compatible call and the wire body contains neither:

{"messages":[{"content":"hi","role":"user"}],"model":"llama-3.1-70b-instruct","stream":false}

The call succeeds, at the provider's default temperature and length limit. If determinism matters to you — a classifier, a structured extraction — that temperature: Some(0.0) is doing nothing on those providers.

Streaming token counts are always zero

Defect, on both backends, for two separate reasons.

On the Anthropic-direct path, ChatStreamEvent::Done reads usage from the message_stop SSE event, which does not carry any. The Messages API reports input tokens on message_start and output tokens on message_delta, both of which rtb-chat parses and discards.

On the genai path, Done reads genai's captured_usage, which genai only populates when capture_usage is switched on in its chat options. rtb-chat passes no chat options, and genai's default is off.

Either way, a streamed call ends with Done(Usage::default()) — four zeros — however many tokens were really used. Use the non-streaming chat call when you need to account for usage.

Only text content is supported

Deliberate, for now. ContentBlock has one variant, Text. Images, audio, tool use and function calling have no representation in the request, and any non-text block in a response is dropped during parsing rather than surfaced.

ContentBlock and Provider are both #[non_exhaustive] so that adding those variants later is not a breaking change.

No tool use, no function calling, no agent loop

Deliberate. rtb-chat sends a request and returns a reply. It does not execute tools, does not loop, and does not manage conversation state — the messages vector you pass is the whole history, and keeping it is your job.

genai's tool-call stream chunks are received and thrown away, which is what makes this a hard boundary rather than a partial implementation.

No retries, no backoff, no circuit breaker

Deliberate. A 429 becomes AiError::RateLimited carrying retry_after when the provider sent a parseable Retry-After header, and that is the extent of it. Nothing sleeps and nothing retries.

Retry policy belongs to the caller because the right policy depends on whether a human is waiting. Note that AiError::RateLimited never occurs on a genai provider — see below — so a retry loop keyed on that variant covers only the two Anthropic providers.

genai providers report every failure as a provider error

Deliberate consequence of the two-backend split, worth knowing about. The status-code mapping that produces AiError::Auth and AiError::RateLimited lives in rtb-chat's own Anthropic path. genai surfaces failures through its own error type, and all of them are wrapped as AiError::Provider.

A bad OpenAI key does not give you AiError::Auth. It gives you AiError::Provider with genai's description inside.

The genai backend puts your key in the process environment

Deliberate, and the crate's one unsafe block. genai resolves provider credentials from environment variables. To hand it the key from Config, AiClient::new calls std::env::set_var — OPENAI_API_KEY for Provider::OpenAi and Provider::OpenAiCompatible, GEMINI_API_KEY for Provider::Gemini.

Three consequences follow, none of them obvious from the type signature:

  • The key outlives the client. It stays in the environment for the life of the process and is readable by anything in it, including code that dumps the environment into a crash report.
  • The last client wins. Build a second AiClient for the same provider with a different key and the first client starts using the second key, because both read the same variable at call time.
  • Constructing clients concurrently is a data race. std::env::set_var is unsafe on modern Rust editions for exactly this reason. The crate assumes the common pattern of one client built once per process; building clients from several threads while other threads read the environment is undefined behaviour.

The Anthropic-direct path does not do this — it sends the key as a header and touches no environment variable. Neither does Provider::Ollama, which sets nothing.

An API key is required even when the provider does not need one

Deliberate, and awkward. AiClient::new rejects an empty api_key for every provider, Provider::Ollama included. A local Ollama needs no credential, so you have to invent one:

api_key: SecretString::from("unused".to_string()),

The check is one rule applied uniformly rather than a per-provider table. It costs a placeholder string; leaving it out would let a genuine missing-key mistake reach the provider as a confusing 401.

Config cannot be deserialised

Deliberate. Config derives Debug and Clone and nothing else. There is no Deserialize, so a config file cannot be mapped onto it directly — the tool layer reads its own settings type and constructs a Config in code.

That is what keeps allow_insecure_base_url out of reach of anything but Rust source, which is the property that makes the HTTPS enforcement worth having. See Why the endpoint is validated.

One model per client, and no fallback

Deliberate. Config::model is fixed at construction. There is no per-request override, no fallback chain, and no "try the cheap model first" behaviour. Build a second AiClient for a second model — bearing in mind the environment-variable caveat above if both are genai-backed.

anthropic-version cannot be changed

Deliberate. The Anthropic-direct path sends anthropic-version: 2023-06-01 from a compile-time constant. A Messages API feature gated behind a newer version string needs a crate release, not a config change.

Structured output is one attempt, with no repair

Deliberate. chat_structured asks for bare JSON, parses what comes back, validates it against the schema and deserialises it. If any of those fails, you get the error — there is no retry with a corrective prompt, no fence-stripping, no partial extraction.

The most common failure is a reply truncated by the 1024-token default max_tokens, which arrives as AiError::Deserialize rather than anything that mentions length.

Where the defects are recorded

The four defects above are documented here because they are the crate's current behaviour and a reader needs to know about them. They are not accepted design. They were found while writing these pages and are reported against rust/chat#1; this page should shrink as they are fixed.