Responses and streaming¶

ChatResponse¶

pub struct ChatResponse {
    pub message: Message,
    pub usage: Usage,
    pub citations: Vec<Citation>,
}

message always has role: Role::Assistant and exactly one ContentBlock::Text. Where the provider returned several text blocks they are concatenated, with no separator. Blocks of any other type (images, tool use, thinking) are dropped rather than surfaced.

A response body with no content array at all is AiError::Provider:

provider error: missing `content` array on response

Usage¶

pub struct Usage {
    pub input_tokens: u32,
    pub output_tokens: u32,
    pub cache_creation_input_tokens: u32,
    pub cache_read_input_tokens: u32,
}

Any field the provider omits reads 0. There is no "not reported" state, so 0 means either zero or unknown.

• Anthropic-direct, non-streaming: all four fields come from the response usage object.
• genai: input_tokens and output_tokens come from genai's prompt and completion counts. The two cache fields are hard-coded to 0, regardless of provider.
• Streaming, either backend: see Streaming usage arrives on Done.

Counts arrive from the provider as u64 and are narrowed to u32 with an as cast, which truncates rather than saturating. No real token count comes close to u32::MAX, so this is theory rather than practice.

Citation¶

pub struct Citation {
    pub cited_text: String,
    pub source: String,
    pub start_index: Option<u32>,
    pub end_index: Option<u32>,
}

Populated only on the Anthropic-direct backend, and only when the model emits citations. Every genai provider returns an empty vector.

source is filled from the first of document_title, file_path, source that the provider supplies; if none is present it is an empty string rather than None. start_index and end_index come from start_char_index and end_char_index and stay None when absent.

ChatStream and ChatStreamEvent¶

chat_stream returns a ChatStream, which implements futures_util::Stream<Item = ChatStreamEvent>. It is Send, so it can move into tokio::spawn, and it is not Sync.

pub enum ChatStreamEvent {
    Token(String),
    ThinkingToken(String),
    Done(Usage),
    Error(AiError),
}

#[non_exhaustive], so consumers need a _ arm.

Events rtb-chat does not model (message_start, message_delta, genai's start and tool-call chunks) consume their bytes and produce nothing. Empty Token events are filtered out on the genai path, so a genuinely empty text chunk is dropped too.

Done is not the end of the stream¶

Done is emitted when the terminating event arrives, but the stream keeps yielding until the underlying byte stream closes. while let Some(event) = stream.next().await will sit there until the connection ends.

Break out of the loop when you see Done:

use futures_util::StreamExt;
use rtb_chat::ChatStreamEvent;

async fn drain(mut stream: rtb_chat::ChatStream) -> String {
    let mut answer = String::new();
    while let Some(event) = stream.next().await {
        match event {
            ChatStreamEvent::Token(t) => answer.push_str(&t),
            ChatStreamEvent::ThinkingToken(_) => {}
            ChatStreamEvent::Done(_usage) => break,
            ChatStreamEvent::Error(e) => { eprintln!("stream failed: {e}"); break }
            _ => {}
        }
    }
    answer
}

An Error event ends the exchange in practice, since nothing useful follows it, but rtb-chat does not close the stream for you.

Streaming usage arrives on Done¶

Done(usage) carries the counts the provider reported during the stream.

On the Anthropic-direct path the Messages API spreads them over two events. message_start carries the input and cache counts, and message_delta carries the running output count. rtb-chat keeps the latest of each and puts them on Done when message_stop arrives.

On the genai path rtb-chat switches on genai's capture_usage for every stream, and Done carries what genai captured. For the OpenAI adapter that means the request asks for stream_options: {"include_usage": true}, so an OpenAI-compatible server must accept that field. As on the non-streaming genai path, the two cache fields are always 0.

A provider that reports nothing during the stream still gives you four zeros.

Errors during a stream¶

Errors split by when they happen:

• Before the stream starts: connection failures and non-2xx status codes are returned from chat_stream itself, as a Result::Err. On the Anthropic-direct backend the same status mapping as chat applies, so a 429 here is a real AiError::RateLimited; on genai it is AiError::Provider, as it is for chat.
• During the stream: surfaced as ChatStreamEvent::Error inside the stream. These are always AiError::Provider (an SSE error event) or AiError::Transport (the byte stream failed).