Media sinks

Media sinks provide APIs for extracting media data from input files. Like media sources, sinks come at different abstraction levels, letting you choose between convenience and control.

Overview of media sinks

Media sinks can be organized into three abstraction levels:

High-level sinks

CanvasSink, AudioBufferSinkEasy to use, provides processed output ready for display/playback.

Mid-level sinks

VideoSampleSink, AudioSampleSinkAccess to decoded samples, handles decoding internally.

Low-level sinks

EncodedPacketSinkDirect packet access, you handle decoding.

When to use each sink

EncodedPacketSink

Best for: Metadata extraction, remuxing without decoding, custom decoding pipelines EncodedPacketSink provides direct access to encoded packets without decoding. Use this when you only need packet metadata or want to implement custom decoding.

src/media-sink.ts

const sink = new EncodedPacketSink(videoTrack);

// Get packet at specific timestamp
const packet = await sink.getPacket(5.0);

// Get key frame at timestamp
const keyPacket = await sink.getKeyPacket(5.0);

// Iterate over all packets
for await (const packet of sink.packets()) {
  console.log(packet.timestamp, packet.type);
}

Advantages:

No decoding overhead
Access to packet metadata
Perfect for remuxing operations
Fast iteration over packet structure

Source code: src/media-sink.ts:120-375

VideoSampleSink

Best for: Frame-by-frame processing, video analysis, custom rendering VideoSampleSink decodes video packets into raw VideoFrames, giving you access to decoded pixel data.

src/media-sink.ts

const sink = new VideoSampleSink(videoTrack);

// Get frame at specific timestamp
const sample = await sink.getSample(5.0);

// Iterate over all frames
for await (const sample of sink.samples()) {
  // Process the VideoFrame
  console.log(sample.codedWidth, sample.codedHeight);
  sample.close(); // Don't forget to close!
}

// Get frames at specific timestamps (efficient)
for await (const sample of sink.samplesAtTimestamps([0, 1, 2, 3, 4])) {
  // Process sample
  sample?.close();
}

Advantages:

Direct access to decoded frames
Integration with VideoFrame API
Efficient sparse sampling
Automatic decoding pipeline

Important: Always call close() on VideoSamples when done to free memory. Source code: src/media-sink.ts:1361-1440

CanvasSink

Best for: Thumbnail generation, video preview, frame export, display in browser CanvasSink provides the most convenient way to extract video frames as canvases, with built-in support for resizing, rotation, and cropping.

src/media-sink.ts

const sink = new CanvasSink(videoTrack, {
  width: 1280,
  height: 720,
  fit: 'contain',
  rotation: 90,
  poolSize: 3, // Reuse canvases for efficiency
});

// Get canvas at timestamp
const { canvas, timestamp, duration } = await sink.getCanvas(5.0);

// Iterate over frames
for await (const { canvas, timestamp } of sink.canvases()) {
  // Canvas is ready to display or export
  document.body.appendChild(canvas);
}

Advantages:

Ready-to-display canvases
Built-in resizing, rotation, cropping
Canvas pooling for memory efficiency
Perfect for thumbnails and previews

Source code: src/media-sink.ts:1516-1719

AudioSampleSink

Best for: Audio analysis, waveform generation, custom audio processing AudioSampleSink decodes audio packets into raw AudioData, giving you access to decoded audio samples.

src/media-sink.ts

const sink = new AudioSampleSink(audioTrack);

// Get sample at timestamp
const sample = await sink.getSample(5.0);

// Iterate over all samples
for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Process audio data
  sample.close();
}

Advantages:

Access to raw audio samples
Integration with AudioData API
Precise sample-level control
Automatic decoding

Source code: src/media-sink.ts:2034-2092

AudioBufferSink

Best for: Web Audio API integration, playback, audio processing with Web Audio AudioBufferSink provides decoded audio as AudioBuffers, ready for use with the Web Audio API.

src/media-sink.ts

const sink = new AudioBufferSink(audioTrack);
const audioContext = new AudioContext();

// Play audio from timestamp
for await (const { buffer, timestamp } of sink.buffers(10.0, 20.0)) {
  const source = audioContext.createBufferSource();
  source.buffer = buffer;
  source.connect(audioContext.destination);
  source.start(audioContext.currentTime + timestamp - 10.0);
}

Advantages:

Direct AudioBuffer support
Perfect for Web Audio API
Ready for playback
Easy audio processing

Source code: src/media-sink.ts:2094+

Advanced usage patterns

Efficient sparse sampling

When you need samples at specific timestamps, use samplesAtTimestamps instead of multiple getSample calls:

// Inefficient - decodes same packets multiple times
const sample1 = await sink.getSample(1.0);
const sample2 = await sink.getSample(2.0);
const sample3 = await sink.getSample(3.0);

// Efficient - optimized decode pipeline
for await (const sample of sink.samplesAtTimestamps([1.0, 2.0, 3.0])) {
  // Process sample
  sample?.close();
}

Generating thumbnails

Generate evenly-spaced thumbnails efficiently:

const sink = new CanvasSink(videoTrack, {
  width: 160,
  height: 90,
  fit: 'cover',
  poolSize: 1, // Only need one canvas at a time
});

const duration = await videoTrack.computeDuration();
const thumbnailCount = 10;
const timestamps = Array.from(
  { length: thumbnailCount },
  (_, i) => (duration * i) / (thumbnailCount - 1)
);

const thumbnails = [];
for await (const { canvas } of sink.canvasesAtTimestamps(timestamps)) {
  // Export canvas to blob
  const blob = await new Promise(resolve => 
    canvas.toBlob(resolve, 'image/jpeg', 0.9)
  );
  thumbnails.push(blob);
}

Extracting key frames only

Combine EncodedPacketSink with VideoSampleSink to extract only key frames:

const packetSink = new EncodedPacketSink(videoTrack);
const sampleSink = new VideoSampleSink(videoTrack);

const keyFrameTimestamps = (async function* () {
  let packet = await packetSink.getFirstPacket();
  
  while (packet) {
    if (packet.type === 'key') {
      yield packet.timestamp;
    }
    packet = await packetSink.getNextPacket(packet);
  }
})();

for await (const sample of sampleSink.samplesAtTimestamps(keyFrameTimestamps)) {
  // Process only key frames
  sample?.close();
}

Audio waveform generation

Generate a waveform visualization:

const sink = new AudioSampleSink(audioTrack);
const waveformData = [];

for await (const sample of sink.samples()) {
  const bytesNeeded = sample.allocationSize({ format: 'f32', planeIndex: 0 });
  const floats = new Float32Array(bytesNeeded / 4);
  sample.copyTo(floats, { format: 'f32', planeIndex: 0 });
  
  // Calculate RMS for this chunk
  let sum = 0;
  for (let i = 0; i < floats.length; i++) {
    sum += floats[i] ** 2;
  }
  const rms = Math.sqrt(sum / floats.length);
  waveformData.push(rms);
  
  sample.close();
}

Range iteration with break

Exit iteration early while ensuring proper cleanup:

let frameCount = 0;
for await (const sample of sink.samples()) {
  // Process frame
  sample.close();
  
  // Stop after 100 frames
  if (++frameCount >= 100) {
    break; // Automatically cleans up decoder
  }
}

Canvas pool optimization

Use canvas pooling to minimize memory allocation:

No pool (default)
With pool

// Creates new canvas for each frame - can be slow
const sink = new CanvasSink(videoTrack);

for await (const { canvas } of sink.canvases()) {
  // New canvas every iteration
}

// Reuses canvases from pool - much faster
const sink = new CanvasSink(videoTrack, { poolSize: 3 });

for await (const { canvas } of sink.canvases()) {
  // Canvas reused from pool
}

For sequential iteration, poolSize: 1 is sufficient and optimal.

Verifying key packets

Some files incorrectly mark packet types. Verify key packets to ensure decoder compatibility:

const sink = new EncodedPacketSink(videoTrack);

// Without verification (faster, but may be incorrect)
const packet1 = await sink.getKeyPacket(5.0);

// With verification (slower, but guaranteed correct)
const packet2 = await sink.getKeyPacket(5.0, { verifyKeyPackets: true });

Metadata-only packet retrieval

When you only need packet metadata, avoid loading packet data:

const sink = new EncodedPacketSink(videoTrack);

// Get metadata without loading packet data
const packet = await sink.getPacket(5.0, { metadataOnly: true });

console.log(packet.timestamp); // Available
console.log(packet.type);      // Available
console.log(packet.data);      // Empty Uint8Array

Decode vs. presentation order

Understanding the difference between decode and presentation order is crucial:

Presentation order: The order in which frames are displayed (sorted by timestamp)
Decode order: The order in which packets must be decoded (may differ due to B-frames)

B-frames example

Consider frames with B-frames:

Presentation order: Frame1(0.0s) → Frame2(0.1s) → Frame3(0.2s)
Decode order:       Frame1(0.0s) → Frame3(0.2s) → Frame2(0.1s)

EncodedPacketSink methods:

packets() - Returns packets in decode order
getPacket(timestamp) - Searches by presentation timestamp

VideoSampleSink and CanvasSink methods:

All methods use presentation order

Best practices

Choose the right abstraction

Use high-level sinks (CanvasSink, AudioBufferSink) unless you need sample-level control.

Close samples and frames

Always call close() on VideoSamples and AudioSamples to prevent memory leaks.

Use sparse sampling wisely

Use samplesAtTimestamps() instead of multiple getSample() calls for efficiency.

Enable canvas pooling

Use poolSize option in CanvasSink to reduce memory allocation overhead.

Break early when needed

Use break in for-await loops to exit early while ensuring proper cleanup.

Verify key packets when needed

Enable verifyKeyPackets if you encounter decoder errors with key frames.

Performance considerations

Memory management

Always close VideoSamples and AudioSamples
Use canvas pooling for CanvasSink
Use metadata-only packets when possible
Break out of iterations early if possible

Decoding efficiency

Use sparse sampling for non-sequential access
Prefer range iteration for sequential access
Use EncodedPacketSink to skip decoding entirely
Consider decoder queue sizes

Get Started

Core Concepts

Guides

Advanced

Extensions

Overview of media sinks

High-level sinks

Mid-level sinks

Low-level sinks

When to use each sink

EncodedPacketSink

VideoSampleSink

CanvasSink

AudioSampleSink

AudioBufferSink

Advanced usage patterns

Efficient sparse sampling

Generating thumbnails

Extracting key frames only

Audio waveform generation

Range iteration with break

Canvas pool optimization

Verifying key packets

Metadata-only packet retrieval

Decode vs. presentation order

Best practices

Performance considerations

Memory management

Decoding efficiency

See also

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Advanced

Extensions

Documentation Index

​Overview of media sinks

High-level sinks

Mid-level sinks

Low-level sinks

​When to use each sink

​EncodedPacketSink

​VideoSampleSink

​CanvasSink

​AudioSampleSink

​AudioBufferSink

​Advanced usage patterns

​Efficient sparse sampling

​Generating thumbnails

​Extracting key frames only

​Audio waveform generation

​Range iteration with break

​Canvas pool optimization

​Verifying key packets

​Metadata-only packet retrieval

​Decode vs. presentation order

​Best practices

​Performance considerations

Memory management

Decoding efficiency

​See also

Build docs developers (and LLMs) love

Overview of media sinks

When to use each sink

EncodedPacketSink

VideoSampleSink

CanvasSink

AudioSampleSink

AudioBufferSink

Advanced usage patterns

Efficient sparse sampling

Generating thumbnails

Extracting key frames only

Audio waveform generation

Range iteration with break

Canvas pool optimization

Verifying key packets

Metadata-only packet retrieval

Decode vs. presentation order

Best practices

Performance considerations

See also