transport

Small WebRTC transport implementation for Pipecat.

This module provides a WebRTC transport implementation using aiortc for real-time audio and video communication. It supports bidirectional media streaming, application messaging, and client connection management.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCCallbacks(*, on_app_message: Callable[[Any, str], Awaitable[None]], on_client_connected: Callable[[SmallWebRTCConnection], Awaitable[None]], on_client_disconnected: Callable[[SmallWebRTCConnection], Awaitable[None]])

Bases: BaseModel

Callback handlers for SmallWebRTC events.

Parameters:

• on_app_message – Called when an application message is received.

• on_client_connected – Called when a client establishes connection.

• on_client_disconnected – Called when a client disconnects.

on_app_message: Callable[[Any, str], Awaitable[None]]

on_client_connected: Callable[[SmallWebRTCConnection], Awaitable[None]]

on_client_disconnected: Callable[[SmallWebRTCConnection], Awaitable[None]]

class pipecat.transports.smallwebrtc.transport.RawAudioTrack(sample_rate: int, auto_silence: bool = True)

Bases: AudioStreamTrack

Custom audio stream track for WebRTC output.

Handles audio frame generation and timing for WebRTC transmission, supporting queued audio data with proper synchronization.

__init__(sample_rate: int, auto_silence: bool = True)

Initialize the raw audio track.

Parameters:

• sample_rate – The audio sample rate in Hz.

• auto_silence – If True, emit silence when the queue is empty. If False, wait until audio data is available.

add_audio_bytes(audio_bytes: bytes)

Add audio bytes to the buffer for transmission.

Parameters:

audio_bytes – Raw audio data to queue for transmission.

Returns:

A Future that completes when the data is processed.

Raises:

ValueError – If audio bytes are not a multiple of 10ms size.

async recv()

Return the next audio frame for WebRTC transmission.

Returns:

An AudioFrame containing the next audio data, or silence if the queue is empty and auto_silence is True.

class pipecat.transports.smallwebrtc.transport.RawVideoTrack(width, height)

Bases: VideoStreamTrack

Custom video stream track for WebRTC output.

Handles video frame queuing and conversion for WebRTC transmission.

__init__(width, height)

Initialize the raw video track.

Parameters:

• width – Video frame width in pixels.

• height – Video frame height in pixels.

add_video_frame(frame)

Add a video frame to the transmission buffer.

Parameters:

frame – The video frame to queue for transmission.

async recv()

Return the next video frame for WebRTC transmission.

Returns:

A VideoFrame ready for WebRTC transmission.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCClient(webrtc_connection: SmallWebRTCConnection, callbacks: SmallWebRTCCallbacks)

Bases: object

WebRTC client implementation for handling connections and media streams.

Manages WebRTC peer connections, audio/video streaming, and application messaging through the SmallWebRTCConnection interface.

FORMAT_CONVERSIONS = {'gray': cv2.COLOR_GRAY2RGB, 'nv12': cv2.COLOR_YUV2RGB_NV12, 'yuv420p': cv2.COLOR_YUV2RGB_I420, 'yuvj420p': cv2.COLOR_YUV2RGB_I420}

__init__(webrtc_connection: SmallWebRTCConnection, callbacks: SmallWebRTCCallbacks)

Initialize the WebRTC client.

Parameters:

• webrtc_connection – The underlying WebRTC connection handler.

• callbacks – Event callbacks for connection and message handling.

async read_video_frame(video_source: str)

Read video frames from the WebRTC connection.

Reads a video frame from the given MediaStreamTrack, converts it to RGB, and creates an InputImageRawFrame.

Parameters:

video_source – Video source to capture (“camera” or “screenVideo”).

Yields:

UserImageRawFrame objects containing video data from the peer.

async read_audio_frame()

Read audio frames from the WebRTC connection.

Reads 20ms of audio from the given MediaStreamTrack and creates an InputAudioRawFrame.

Yields:

InputAudioRawFrame objects containing audio data from the peer.

async write_audio_frame(frame: OutputAudioRawFrame) → bool

Write an audio frame to the WebRTC connection.

Parameters:

frame – The audio frame to transmit.

Returns:

True if the audio frame was written successfully, False otherwise.

async write_video_frame(frame: OutputImageRawFrame) → bool

Write a video frame to the WebRTC connection.

Parameters:

frame – The video frame to transmit.

Returns:

True if the video frame was written successfully, False otherwise.

async setup(_params: TransportParams, frame)

Set up the client with transport parameters.

Parameters:

• _params – Transport configuration parameters.

• frame – The initialization frame containing setup data.

async connect()

Establish the WebRTC connection.

async disconnect()

Disconnect from the WebRTC peer.

async send_message(frame: OutputTransportMessageFrame | OutputTransportMessageUrgentFrame)

Send an application message through the WebRTC connection.

Parameters:

frame – The message frame to send.

property is_connected: bool

Check if the WebRTC connection is established.

Returns:

True if connected to the peer.

property is_closing: bool

Check if the connection is in the process of closing.

Returns:

True if the connection is closing.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCInputTransport(client: SmallWebRTCClient, params: TransportParams, **kwargs)

Bases: BaseInputTransport

Input transport implementation for SmallWebRTC.

Handles incoming audio and video streams from WebRTC peers, including user image requests and application message handling.

__init__(client: SmallWebRTCClient, params: TransportParams, **kwargs)

Initialize the WebRTC input transport.

Parameters:

• client – The WebRTC client instance.

• params – Transport configuration parameters.

• **kwargs – Additional arguments passed to parent class.

async process_frame(frame: Frame, direction: FrameDirection)

Process incoming frames including user image requests.

Parameters:

• frame – The frame to process.

• direction – The direction of frame flow in the pipeline.

async start(frame: StartFrame)

Start the input transport and establish WebRTC connection.

Parameters:

frame – The start frame containing initialization parameters.

async stop(frame: EndFrame)

Stop the input transport and disconnect from WebRTC.

Parameters:

frame – The end frame signaling transport shutdown.

async cancel(frame: CancelFrame)

Cancel the input transport and disconnect immediately.

Parameters:

frame – The cancel frame signaling immediate cancellation.

async push_app_message(message: Any)

Push an application message into the pipeline.

Parameters:

message – The application message to process.

async request_participant_image(frame: UserImageRequestFrame)

Request an image frame from the participant’s video stream.

When a UserImageRequestFrame is received, this method will store the request and the next video frame received will be converted to a UserImageRawFrame.

Parameters:

frame – The user image request frame.

async capture_participant_media(source: str = 'camera')

Capture media from a specific participant.

Parameters:

source – Media source to capture from. (“camera”, “microphone”, or “screenVideo”)

class pipecat.transports.smallwebrtc.transport.SmallWebRTCOutputTransport(client: SmallWebRTCClient, params: TransportParams, **kwargs)

Bases: BaseOutputTransport

Output transport implementation for SmallWebRTC.

Handles outgoing audio and video streams to WebRTC peers, including transport message sending.

__init__(client: SmallWebRTCClient, params: TransportParams, **kwargs)

Initialize the WebRTC output transport.

Parameters:

• client – The WebRTC client instance.

• params – Transport configuration parameters.

• **kwargs – Additional arguments passed to parent class.

async start(frame: StartFrame)

Start the output transport and establish WebRTC connection.

Parameters:

frame – The start frame containing initialization parameters.

async stop(frame: EndFrame)

Stop the output transport and disconnect from WebRTC.

Parameters:

frame – The end frame signaling transport shutdown.

async cancel(frame: CancelFrame)

Cancel the output transport and disconnect immediately.

Parameters:

frame – The cancel frame signaling immediate cancellation.

async send_message(frame: OutputTransportMessageFrame | OutputTransportMessageUrgentFrame)

Send a transport message through the WebRTC connection.

Parameters:

frame – The transport message frame to send.

async write_audio_frame(frame: OutputAudioRawFrame) → bool

Write an audio frame to the WebRTC connection.

Parameters:

frame – The output audio frame to transmit.

Returns:

True if the audio frame was written successfully, False otherwise.

async write_video_frame(frame: OutputImageRawFrame) → bool

Write a video frame to the WebRTC connection.

Parameters:

frame – The output video frame to transmit.

Returns:

True if the video frame was written successfully, False otherwise.

class pipecat.transports.smallwebrtc.transport.SmallWebRTCTransport(webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: str | None = None, output_name: str | None = None)

Bases: BaseTransport

WebRTC transport implementation for real-time communication.

Provides bidirectional audio and video streaming over WebRTC connections with support for application messaging and connection event handling.

Event handlers available:

• on_client_connected(transport, client): Client connected to WebRTC session

• on_client_disconnected(transport, client): Client disconnected from WebRTC session

• on_client_message(transport, message, client): Received a data channel message

Example:

@transport.event_handler("on_client_connected")
async def on_client_connected(transport, client):
    ...

__init__(webrtc_connection: SmallWebRTCConnection, params: TransportParams, input_name: str | None = None, output_name: str | None = None)

Initialize the WebRTC transport.

Parameters:

• webrtc_connection – The underlying WebRTC connection handler.

• params – Transport configuration parameters.

• input_name – Optional name for the input processor.

• output_name – Optional name for the output processor.

input() → SmallWebRTCInputTransport

Get the input transport processor.

Returns:

The input transport for handling incoming media streams.

output() → SmallWebRTCOutputTransport

Get the output transport processor.

Returns:

The output transport for handling outgoing media streams.

async send_image(frame: OutputImageRawFrame | SpriteFrame)

Send an image frame through the transport.

Parameters:

frame – The image frame to send.

async send_audio(frame: OutputAudioRawFrame)

Send an audio frame through the transport.

Parameters:

frame – The audio frame to send.

async capture_participant_video(video_source: str = 'camera')

Capture video from a specific participant.

Parameters:

video_source – Video source to capture from (“camera” or “screenVideo”).

async capture_participant_audio(audio_source: str = 'microphone')

Capture audio from a specific participant.

Parameters:

audio_source – Audio source to capture from. (currently, “microphone” is the only supported option)