Cache a prompt and enable thinking¶

Both features are Anthropic-direct only. On Provider::OpenAi, Provider::OpenAiCompatible, Provider::Gemini and Provider::Ollama the fields compile and do nothing.

Cache a long, stable prompt¶

Set cache_control: true:

use rtb_chat::{ChatRequest, Message};

fn build(style_guide: &str, question: &str) -> ChatRequest {
    ChatRequest {
        system: Some(format!("You answer questions about this style guide.\n\n{style_guide}")),
        messages: vec![Message::user(question)],
        cache_control: true,
        ..ChatRequest::default()
    }
}

That marks two cache breakpoints: the system block, and the first text block of the first message. Nothing else is marked.

Which means the layout of your request decides whether caching helps. Put the large, unchanging material — a style guide, a schema, a document being asked about — in system, and keep the varying part in the user message. Reverse them and you are caching the part that changes on every call.

Confirm the cache is being used¶

Usage reports it, on the non-streaming chat call:

async fn run(client: &rtb_chat::AiClient, req: rtb_chat::ChatRequest)
    -> Result<(), rtb_chat::AiError> {
    let resp = client.chat(req).await?;
    println!(
        "cache write {} / cache read {}",
        resp.usage.cache_creation_input_tokens,
        resp.usage.cache_read_input_tokens,
    );
    Ok(())
}

Expect a non-zero cache_creation_input_tokens on the first call and a non-zero cache_read_input_tokens on the ones after it. Two zeros mean nothing was cached — the usual causes are a prompt below the provider's minimum cacheable length, a prefix that changed between calls, or a gap long enough for the entry to expire.

Both fields are hard-coded to 0 on every genai provider, so this check tells you nothing there. Streaming reports zeros too, on every provider — see Streaming token counts are always zero.

Give the model room to think¶

use rtb_chat::{ChatRequest, Message, ThinkingMode};

fn build(question: &str) -> ChatRequest {
    ChatRequest {
        messages: vec![Message::user(question)],
        thinking: Some(ThinkingMode::budget(4096)),
        max_tokens: Some(8192),
        ..ChatRequest::default()
    }
}

Set max_tokens above the budget. rtb-chat sends the two numbers independently and does not check one against the other, and an unset max_tokens is 1024 — so ThinkingMode::budget(4096) on its own builds a request the provider rejects.

Anthropic recommends at least 1024 thinking tokens for the mode to be worth enabling. rtb-chat enforces no floor: a budget of 1 is sent as-is.

Read the thinking output¶

Only the streaming path surfaces it, as ChatStreamEvent::ThinkingToken. The non-streaming chat call keeps text blocks only, so thinking content is discarded before you see it.

If you want both the reasoning and a complete answer, stream and accumulate:

use futures_util::StreamExt;
use rtb_chat::ChatStreamEvent;
async fn drain(mut stream: rtb_chat::ChatStream) -> (String, String) {
    let (mut thoughts, mut answer) = (String::new(), String::new());
    while let Some(event) = stream.next().await {
        match event {
            ChatStreamEvent::ThinkingToken(t) => thoughts.push_str(&t),
            ChatStreamEvent::Token(t) => answer.push_str(&t),
            ChatStreamEvent::Done(_) => break,
            _ => {}
        }
    }
    (thoughts, answer)
}

Use both together¶

They compose. Caching applies to the request, thinking to the response, and neither touches the other's fields:

use rtb_chat::{ChatRequest, Message, ThinkingMode};
fn build(corpus: &str, question: &str) -> ChatRequest {
    ChatRequest {
        system: Some(corpus.to_string()),
        messages: vec![Message::user(question)],
        cache_control: true,
        thinking: Some(ThinkingMode::budget(4096)),
        max_tokens: Some(8192),
        ..ChatRequest::default()
    }
}