llm_context_summarization

Utility for context summarization in LLM services.

This module provides reusable functionality for automatically compressing conversation context when token limits are reached, enabling efficient long-running conversations.

class pipecat.utils.context.llm_context_summarization.LLMContextSummaryConfig(target_context_tokens: int = 6000, min_messages_after_summary: int = 4, summarization_prompt: str | None = None, summary_message_template: str = 'Conversation summary: {summary}', llm: LLMService | None = None, summarization_timeout: float = 120.0)[source]

Bases: object

Configuration for summary generation parameters.

Contains settings that control how a summary is generated. Used by both automatic and manual summarization modes.

Parameters:

target_context_tokens – Maximum token size for the generated summary. This value is passed directly to the LLM as the max_tokens parameter when generating the summary. Should be sized appropriately to allow the summary plus recent preserved messages to fit within reasonable context limits.
min_messages_after_summary – Number of recent messages to preserve uncompressed after each summarization. These messages maintain immediate conversational context.
summarization_prompt – Custom prompt for the LLM to use when generating summaries. If None, uses DEFAULT_SUMMARIZATION_PROMPT.
summary_message_template – Template for formatting the summary when injected into context. Must contain {summary} as a placeholder for the generated summary text. Allows applications to wrap the summary in custom delimiters (e.g., XML tags) so that system prompts can distinguish summaries from live conversation.
llm – Optional separate LLM service for generating summaries. When set, summarization requests are sent to this service instead of the pipeline’s primary LLM. Useful for routing summarization to a cheaper/faster model (e.g., Gemini Flash) while keeping an expensive model for conversation. If None, uses the pipeline LLM.
summarization_timeout – Maximum time in seconds to wait for the LLM to generate a summary. If the call exceeds this timeout, summarization is aborted with an error and future summarizations are unblocked.

target_context_tokens: int = 6000

min_messages_after_summary: int = 4

summarization_prompt: str | None = None

summary_message_template: str = 'Conversation summary: {summary}'

llm: LLMService | None = None

summarization_timeout: float = 120.0

property summary_prompt: str

Get the summarization prompt to use.

Returns:: The custom prompt if set, otherwise the default summarization prompt.

class pipecat.utils.context.llm_context_summarization.LLMAutoContextSummarizationConfig(max_context_tokens: int | None = 8000, max_unsummarized_messages: int | None = 20, summary_config: LLMContextSummaryConfig = <factory>)[source]

Bases: object

Configuration for automatic context summarization.

Controls when conversation context is automatically compressed and how that summary is generated. Summarization is triggered when either the token limit or the unsummarized message count threshold is exceeded.

At least one of max_context_tokens and max_unsummarized_messages must be set. Set the other to None to disable that threshold.

Parameters:

max_context_tokens – Maximum allowed context size in tokens. When this limit is reached, summarization is triggered to compress the context. The tokens are calculated using the industry-standard approximation of 1 token ≈ 4 characters. Set to None to disable token-based triggering.
max_unsummarized_messages – Maximum number of new messages that can accumulate since the last summary before triggering a new summarization. This ensures regular compression even if token limits are not reached. Set to None to disable message-count triggering.
summary_config – Configuration for summary generation parameters (prompt, token budget, messages to keep). If not provided, uses default LLMContextSummaryConfig values.

max_context_tokens: int | None = 8000

max_unsummarized_messages: int | None = 20

summary_config: LLMContextSummaryConfig

class pipecat.utils.context.llm_context_summarization.LLMContextSummarizationConfig(max_context_tokens: int | None = 8000, target_context_tokens: int = 6000, max_unsummarized_messages: int | None = 20, min_messages_after_summary: int = 4, summarization_prompt: str | None = None, summary_message_template: str = 'Conversation summary: {summary}', llm: LLMService | None = None, summarization_timeout: float = 120.0)[source]

Bases: object

Configuration for context summarization behavior.

Deprecated since version 0.0.104: Use LLMAutoContextSummarizationConfig with a nested LLMContextSummaryConfig instead.

Example:

LLMAutoContextSummarizationConfig(
    max_context_tokens=8000,
    max_unsummarized_messages=20,
    summary_config=LLMContextSummaryConfig(
        target_context_tokens=6000,
        min_messages_after_summary=4,
    ),
)

Parameters:

max_context_tokens – Maximum allowed context size in tokens. Set to None to disable token-based triggering.
target_context_tokens – Maximum token size for the generated summary.
max_unsummarized_messages – Maximum new messages before triggering summarization. Set to None to disable message-count triggering.
min_messages_after_summary – Number of recent messages to preserve.
summarization_prompt – Custom prompt for summary generation.

max_context_tokens: int | None = 8000

target_context_tokens: int = 6000

max_unsummarized_messages: int | None = 20

min_messages_after_summary: int = 4

summarization_prompt: str | None = None

summary_message_template: str = 'Conversation summary: {summary}'

llm: LLMService | None = None

summarization_timeout: float = 120.0

property summary_prompt: str

Get the summarization prompt to use.

Returns:: The custom prompt if set, otherwise the default summarization prompt.

to_auto_config() → LLMAutoContextSummarizationConfig[source]

Convert to the new LLMAutoContextSummarizationConfig.

Returns:: An equivalent LLMAutoContextSummarizationConfig instance.

class pipecat.utils.context.llm_context_summarization.LLMMessagesToSummarize(messages: list[ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage], last_summarized_index: int)[source]

Bases: object

Result of get_messages_to_summarize operation.

Parameters:

messages – Messages to include in the summary
last_summarized_index – Index of the last message being summarized

messages: list[ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage]

last_summarized_index: int

class pipecat.utils.context.llm_context_summarization.LLMContextSummarizationUtil[source]

Bases: object

Utility providing context summarization capabilities for LLM processing.

This utility enables automatic conversation context compression when token limits are reached. It provides functionality for both aggregators (which decide when to summarize) and LLM services (which generate summaries).

Key features: - Token estimation using character-count heuristics (chars // 4) - Smart message selection (preserves system messages and recent context) - Function call awareness (avoids summarizing incomplete tool interactions) - Flexible transcript formatting for summarization - Maximum summary token calculation with safety buffers

Usage:

Use the static methods directly on the class:

tokens = LLMContextSummarizationUtil.estimate_context_tokens(context) result = LLMContextSummarizationUtil.get_messages_to_summarize(context, 4) transcript = LLMContextSummarizationUtil.format_messages_for_summary(messages)

Note

Token estimation uses the industry-standard heuristic of 1 token ≈ 4 characters.

static estimate_tokens(text: str) → int[source]

Estimate token count for text using character count heuristic.

Uses the industry-standard approximation of 1 token ≈ 4 characters. This works well across different content types (prose, code, etc.) and languages.

Note

For more accurate token counts, use the model’s official tokenizer. This is a rough estimate suitable for threshold checks and budgeting.

Parameters:: text – Text to estimate tokens for
Returns:: Estimated token count (characters // 4)

static estimate_context_tokens(context: LLMContext) → int[source]

Estimate total token count for a context.

Calculates an approximate token count by analyzing all messages, including text content, tool calls, and structural overhead.

Parameters:

context – LLM context to estimate.

Returns:

Message content (text, images)
Tool calls and their arguments
Tool results
Structural overhead (TOKEN_OVERHEAD_PER_MESSAGE per message)

Return type:

Estimated total token count including

static get_messages_to_summarize(context: LLMContext, min_messages_to_keep: int) → LLMMessagesToSummarize[source]

Determine which messages should be included in summarization.

Intelligently selects messages for summarization while preserving: - The first system message (defines assistant behavior) - The last N messages (maintains immediate conversation context) - Incomplete function call sequences (preserves tool interaction integrity)

Parameters:

context – The LLM context containing all messages.
min_messages_to_keep – Number of recent messages to exclude from summarization.

Returns:

LLMMessagesToSummarize containing the messages to summarize and the index of the last message included.

static format_messages_for_summary(messages: list[dict]) → str[source]

Format messages as a transcript for summarization.

Parameters:: messages – Messages to format
Returns:: Formatted transcript string