Responses and streaming¶

ChatResponse¶

pub struct ChatResponse {
    pub message: Message,
    pub usage: Usage,
    pub citations: Vec<Citation>,
}

message always has role: Role::Assistant and exactly one ContentBlock::Text. Where the provider returned several text blocks they are concatenated, with no separator. Blocks of any other type — images, tool use, thinking — are dropped rather than surfaced.

A response body with no content array at all is AiError::Provider:

provider error: missing `content` array on response

Usage¶

pub struct Usage {
    pub input_tokens: u32,
    pub output_tokens: u32,
    pub cache_creation_input_tokens: u32,
    pub cache_read_input_tokens: u32,
}

Any field the provider omits reads 0 — there is no "not reported" state, so 0 means either zero or unknown.

• Anthropic-direct, non-streaming — all four fields come from the response usage object.
• genai — input_tokens and output_tokens come from genai's prompt and completion counts. The two cache fields are hard-coded to 0, regardless of provider.
• Streaming, either backend — see Streaming usage is reported as zero.

Counts arrive from the provider as u64 and are narrowed to u32 with an as cast, which truncates rather than saturating. No real token count comes close to u32::MAX, so this is theory rather than practice.

Citation¶

pub struct Citation {
    pub cited_text: String,
    pub source: String,
    pub start_index: Option<u32>,
    pub end_index: Option<u32>,
}

Populated only on the Anthropic-direct backend, and only when the model emits citations. Every genai provider returns an empty vector.

source is filled from the first of document_title, file_path, source that the provider supplies; if none is present it is an empty string rather than None. start_index and end_index come from start_char_index and end_char_index and stay None when absent.

ChatStream and ChatStreamEvent¶

chat_stream returns a ChatStream, which implements futures_util::Stream<Item = ChatStreamEvent>. It is Send — safe to move into tokio::spawn — and not Sync.

pub enum ChatStreamEvent {
    Token(String),
    ThinkingToken(String),
    Done(Usage),
    Error(AiError),
}

#[non_exhaustive], so consumers need a _ arm.

Events rtb-chat does not model — message_start, message_delta, genai's start and tool-call chunks — consume their bytes and produce nothing. Empty Token events are filtered out on the genai path, so a genuinely empty text chunk is dropped too.

Done is not the end of the stream¶

Done is emitted when the terminating event arrives, but the stream keeps yielding until the underlying byte stream closes. while let Some(event) = stream.next().await will sit there until the connection ends.

Break out of the loop when you see Done:

use futures_util::StreamExt;
use rtb_chat::ChatStreamEvent;

async fn drain(mut stream: rtb_chat::ChatStream) -> String {
    let mut answer = String::new();
    while let Some(event) = stream.next().await {
        match event {
            ChatStreamEvent::Token(t) => answer.push_str(&t),
            ChatStreamEvent::ThinkingToken(_) => {}
            ChatStreamEvent::Done(_usage) => break,
            ChatStreamEvent::Error(e) => { eprintln!("stream failed: {e}"); break }
            _ => {}
        }
    }
    answer
}

An Error event ends the exchange in practice — nothing useful follows it — but rtb-chat does not close the stream for you.

Streaming usage is reported as zero¶

Done on the Anthropic-direct path reads the token counts from the message_stop SSE event. The Messages API does not put usage there: input counts arrive on message_start and output counts on message_delta, both of which rtb-chat parses as "no event" and discards.

The genai path lands in the same place by a different route — it reads genai's captured_usage, which genai fills in only when capture_usage is enabled in its chat options, and rtb-chat passes none.

The result is that Done(usage) on a real streaming call carries Usage::default() — four zeros — even when the provider reported real numbers earlier in the stream. Confirmed against a stream carrying message_start with input_tokens: 11 and message_delta with output_tokens: 42:

Done usage: Usage { input_tokens: 0, output_tokens: 0,
                    cache_creation_input_tokens: 0, cache_read_input_tokens: 0 }

This is a defect, and it is tracked as such — see What rtb-chat does not do. Until it is fixed, use the non-streaming chat call when you need to account for tokens, or count them yourself.

Errors during a stream¶

Errors split by when they happen:

• Before the stream starts — connection failures and non-2xx status codes are returned from chat_stream itself, as a Result::Err. The same status mapping as chat applies, so a 429 here is a real AiError::RateLimited.
• During the stream — surfaced as ChatStreamEvent::Error inside the stream. These are always AiError::Provider (an SSE error event) or AiError::Transport (the byte stream failed).