llm_context

Universal LLM context management for LLM services in Pipecat.

Context contents are represented in a universal format (based on OpenAI) that supports a union of known Pipecat LLM service functionality.

Whenever an LLM service needs to access context, it does a just-in-time translation from this universal context into whatever format it needs, using a service-specific adapter.

pipecat.processors.aggregators.llm_context.is_given(value: _T | NotGiven) → TypeGuard[_T][source]

Check whether a value was explicitly provided.

Typically used when checking whether a NotGiven-valued field or parameter was set:

if is_given(context.tools):
    ...

Also acts as a type guard: inside a true branch, the value is narrowed to exclude NotGiven (e.g. ToolsSchema | NotGiven becomes ToolsSchema).

Parameters:: value – The value to check.
Returns:: True if value is anything other than NOT_GIVEN.

class pipecat.processors.aggregators.llm_context.LLMSpecificMessage(llm: str, message: Any)[source]

Bases: object

A container for a context message that is specific to a particular LLM service.

Enables the use of service-specific message types while maintaining compatibility with the universal LLM context format.

llm: str

message: Any

class pipecat.processors.aggregators.llm_context.LLMContext(messages: list[ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage] | None = None, tools: ToolsSchema | NotGiven = NOT_GIVEN, tool_choice: Literal['none', 'auto', 'required'] | ChatCompletionAllowedToolChoiceParam | ChatCompletionNamedToolChoiceParam | ChatCompletionNamedToolChoiceCustomParam | NotGiven = NOT_GIVEN)[source]

Bases: object

Manages conversation context for LLM interactions.

Handles message history, tool definitions, tool choices, and multimedia content for LLM conversations. Provides methods for message manipulation, and content formatting.

Initialize the LLM context.

Parameters:

messages – Initial list of conversation messages.
tools – Available tools for the LLM to use.
tool_choice – Tool selection strategy for the LLM.

Create a context message containing an image URL.

Parameters:

role – The role of this message (defaults to “user”).
url – The URL of the image.
text – Optional text to include with the image.

async static create_image_message(*, role: str = 'user', format: str, size: tuple[int, int], image: bytes, text: str | None = None) → ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage[source]

Create a context message containing an image.

Parameters:

role – The role of this message (defaults to “user”).
format – Image format (e.g., ‘RGB’, ‘RGBA’, or, if already encoded, the MIME type like ‘image/jpeg’).
size – Image dimensions as (width, height) tuple.
image – Raw image bytes.
text – Optional text to include with the image.

async static create_audio_message(*, role: str = 'user', audio_frames: list[AudioRawFrame], text: str = 'Audio follows') → ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage[source]

Create a context message containing audio.

Parameters:

role – The role of this message (defaults to “user”).
audio_frames – List of audio frame objects to include.
text – Optional text to include with the audio.

Get the current messages list.

NOTE: This is equivalent to calling get_messages() with no filter. If you want to filter out LLM-specific messages that don’t pertain to your LLM, use get_messages() directly.

Returns:: List of conversation messages.

get_messages(llm_specific_filter: str | None = None, *, truncate_large_values: bool = False) → list[ChatCompletionDeveloperMessageParam | ChatCompletionSystemMessageParam | ChatCompletionUserMessageParam | ChatCompletionAssistantMessageParam | ChatCompletionToolMessageParam | ChatCompletionFunctionMessageParam | LLMSpecificMessage][source]

Get the current messages list.

Parameters:

llm_specific_filter – Optional filter to return LLM-specific messages for the given LLM, in addition to the standard messages. If messages end up being filtered, an error will be logged; this is intended to catch accidental use of incompatible LLM-specific messages.
truncate_large_values – If True, return deep copies of messages with large values shortened. For standard messages, known binary data (base64-encoded images, audio) is replaced with short placeholders. For LLM-specific messages, long string values are truncated.

Returns:

List of conversation messages.

property tools: ToolsSchema | NotGiven

Get the tools list.

Returns:: Tools list.

property tool_choice: Literal['none', 'auto', 'required'] | ChatCompletionAllowedToolChoiceParam | ChatCompletionNamedToolChoiceParam | ChatCompletionNamedToolChoiceCustomParam | NotGiven

Get the current tool choice setting.

Returns:: The tool choice configuration.

Add a single message to the context.

Parameters:: message – The message to add to the conversation history.

Add multiple messages to the context.

Parameters:: messages – List of messages to add to the conversation history.

Replace all messages in the context.

Parameters:: messages – New list of messages to replace the current history.

Transform the current messages using the provided function.

Parameters:: transform – A function that takes the current list of messages and returns a modified list of messages to set in the context.

set_tools(tools: ToolsSchema | NotGiven = NOT_GIVEN)[source]

Set the available tools for the LLM.

Parameters:: tools – A ToolsSchema or NOT_GIVEN to disable tools.

set_tool_choice(tool_choice: Literal['none', 'auto', 'required'] | ChatCompletionAllowedToolChoiceParam | ChatCompletionNamedToolChoiceParam | ChatCompletionNamedToolChoiceCustomParam | NotGiven)[source]

Set the tool choice configuration.

Parameters:: tool_choice – Tool selection strategy for the LLM.

async add_image_frame_message(*, format: str, size: tuple[int, int], image: bytes, text: str | None = None, role: str = 'user')[source]

Add a message containing an image frame.

Parameters:

format – Image format (e.g., ‘RGB’, ‘RGBA’, or, if already encoded, the MIME type like ‘image/jpeg’).
size – Image dimensions as (width, height) tuple.
image – Raw image bytes.
text – Optional text to include with the image.
role – The role of this message (defaults to “user”).

async add_audio_frames_message(*, audio_frames: list[AudioRawFrame], text: str = 'Audio follows')[source]

Add a message containing audio frames.

Parameters:

audio_frames – List of audio frame objects to include.
text – Optional text to include with the audio.