Requests¶

ChatRequest is the argument to chat, chat_stream and chat_structured. It derives Default, so the usual shape is to set the fields you care about:

use rtb_chat::{ChatRequest, Message, ThinkingMode};

let req = ChatRequest {
    system: Some("You summarise changelogs.".into()),
    messages: vec![Message::user("Summarise this changelog in one line.")],
    max_tokens: Some(256),
    cache_control: true,
    thinking: Some(ThinkingMode::budget(2048)),
    ..ChatRequest::default()
};

ChatRequest fields at a glance¶

max_tokens defaults to 1024 when unset¶

On the Anthropic-direct path the Messages API requires max_tokens, so rtb-chat supplies 1024 when the field is None. That default is invisible until a long reply gets cut off mid-sentence.

It bites hardest in chat_structured, where a truncated reply is not valid JSON and comes back as AiError::Deserialize rather than anything that says "too long". If a structured type has more than a handful of fields, set max_tokens explicitly.

On the genai path max_tokens is not sent at all, and the provider's own default applies.

temperature and max_tokens are dropped on genai providers¶

Neither field reaches the wire on the genai backend. A request built with temperature: Some(0.2) and max_tokens: Some(64) against Provider::OpenAiCompatible produces this body:

{"messages":[{"content":"hi","role":"user"}],"model":"llama-3.1-70b-instruct","stream":false}

No error, no warning — the model answers at the provider's defaults. This is a defect rather than a design decision; see What rtb-chat does not do.

Message, Role and ContentBlock¶

pub struct Message {
    pub role: Role,               // System | User | Assistant
    pub content: Vec<ContentBlock>,
}

Message::user, Message::system and Message::assistant each build a message with a single ContentBlock::Text.

ContentBlock has exactly one variant today, Text(String), and is #[non_exhaustive] so image and tool-use blocks can be added without a breaking change. ContentBlock::as_text() returns Option<&str> for that reason; today it is always Some.

Multiple blocks in one message are preserved on the Anthropic-direct path as separate {"type":"text"} entries. On the genai path they are joined with newlines into a single string, because genai's message type takes one body.

A System role inside messages is not a system prompt¶

The two backends handle it differently, and neither does what the name suggests.

• Anthropic-direct — the Messages API has no system role in its messages array, so a Role::System message is sent with "role": "user". Its text reaches the model as user input.
• genai — a Role::System message is applied as genai's system prompt, which replaces anything set through ChatRequest::system or by an earlier Role::System message. Last one wins.

Put the system prompt in ChatRequest::system and keep messages to User and Assistant turns.

cache_control marks two breakpoints¶

Setting cache_control: true adds "cache_control": {"type": "ephemeral"} to:

1. the system block, when system is Some; and
2. the first text block of the first message in messages.

Nothing else is marked. Later messages cache off the implicit prefix rather than carrying their own breakpoint.

The resulting body:

{
  "model": "claude-opus-4-7",
  "max_tokens": 1024,
  "system": [{"type": "text", "text": "you are helpful",
              "cache_control": {"type": "ephemeral"}}],
  "messages": [{"role": "user", "content": [
      {"type": "text", "text": "hi", "cache_control": {"type": "ephemeral"}}]}]
}

Cache hits and writes come back in Usage::cache_read_input_tokens and Usage::cache_creation_input_tokens. Both are 0 on every genai provider, whether or not there was a cache.

ThinkingMode sets a token budget¶

pub enum ThinkingMode {
    Budget { max_tokens: u32 },
}

ThinkingMode::budget(2048) is the constructor. The enum is #[non_exhaustive] and has one variant today.

It serialises to {"type": "enabled", "budget_tokens": <max_tokens>} on the Anthropic-direct path. Anthropic recommends at least 1024 tokens for the mode to be worth enabling; rtb-chat does not enforce a floor, so a budget of 1 is sent as-is and rejected by the provider.

Thinking output arrives as ChatStreamEvent::ThinkingToken on the streaming path. On the non-streaming path there is nowhere for it to go: parse_chat_response keeps only "type": "text" blocks, so thinking blocks in the response body are discarded and ChatResponse carries the reply alone.

Set max_tokens above the thinking budget¶

rtb-chat sends max_tokens and budget_tokens as independent numbers and does not check that one fits inside the other. Leaving max_tokens unset means the reply cap is 1024, so asking for a 2048-token thinking budget builds a request whose budget exceeds its own cap — the provider rejects it, and the failure arrives as AiError::Provider with the provider's own wording. Set max_tokens comfortably above the budget whenever thinking is Some.

chat_structured rewrites the system prompt¶

chat_structured::<T>() takes any T: DeserializeOwned + JsonSchema. Before sending, it generates T's JSON Schema and appends an instruction block to ChatRequest::system:

<your system prompt>

You MUST respond with a single JSON value matching this schema. No prose, no code fences:
{ ...the generated schema... }

Your own system prompt is kept as the prefix. If system was None, the instruction becomes the whole system prompt.

The reply is then parsed as JSON, validated against the schema, and deserialised into T. Because the instruction asks for bare JSON, a model that wraps its answer in a ```json fence fails at the parse step with AiError::Deserialize — the schema never gets a look in.

Nothing strips a fence or retries. If a model keeps fencing its output, say so again in your own system prompt — it is prepended, so it is the first thing the model reads.