Packets and Samples

In Mediabunny, media data exists in two fundamental forms: packets (compressed data) and samples (raw data). Understanding when to use each is essential for effective media processing.

The fundamental difference

Packets

Compressed, encoded media data as stored in files. Small, efficient, but not directly usable.

Samples

Raw, decoded media data ready for processing. Large, uncompressed, directly manipulable.

Think of it like this:

Packet: A JPEG image file (compressed, efficient to store)
Sample: Raw pixel data in memory (uncompressed, ready to edit)

EncodedPacket

The EncodedPacket class represents a chunk of compressed media data - either video or audio.

Structure

An EncodedPacket contains:

packet.ts:47-87

export class EncodedPacket {
  constructor(
    public readonly data: Uint8Array,        // The compressed bytes
    public readonly type: PacketType,         // 'key' or 'delta'
    public readonly timestamp: number,        // Presentation time (seconds)
    public readonly duration: number,         // Duration (seconds)
    public readonly sequenceNumber = -1,      // Decode order
    byteLength?: number,
    sideData?: EncodedPacketSideData,
  ) { /* ... */ }
}

Field descriptions

data: The actual compressed bytes in codec-specific format
type: 'key' (can decode independently) or 'delta' (depends on previous frames)
timestamp: When this packet should be presented, in seconds
duration: How long this packet lasts, in seconds
sequenceNumber: Decode order (lower numbers decode first)
byteLength: Size of the data (useful for metadata-only packets)
sideData: Additional data like alpha channel information

Creating packets

You typically create packets from encoded data or WebCodecs API chunks:

const packet = new EncodedPacket(
  encodedData,    // Uint8Array of compressed data
  'key',          // Key frame
  1.5,            // Timestamp: 1.5 seconds
  0.033,          // Duration: ~30fps
  42              // Sequence number
);

Using packets

Packets are what you read from input files and write to output files:

import { Input, MP4, BlobSource } from 'mediabunny';

const input = new Input({
  formats: [MP4],
  source: new BlobSource(file)
});

const videoTrack = await input.getPrimaryVideoTrack();

// Read packets from the file
for await (const packet of videoTrack.readPackets()) {
  console.log('Packet:', {
    type: packet.type,
    timestamp: packet.timestamp,
    duration: packet.duration,
    size: packet.byteLength
  });
  
  // Packets are compressed - can't directly access pixel data!
}

Packet types

Key packets
Delta packets

Key packets (also called I-frames or keyframes) can be decoded independently:

if (packet.type === 'key') {
  // This packet can be decoded without any previous packets
  // Perfect for seeking, splitting, or starting playback
}

Key packets are larger but essential for random access and seeking in media files.

Delta packets (P-frames, B-frames) depend on other packets for decoding:

if (packet.type === 'delta') {
  // This packet needs previous (and possibly future) packets to decode
  // Smaller but requires proper decode order
}

Delta packets must be decoded in sequence order, not presentation order. Use sequenceNumber to determine decode order.

Converting to WebCodecs

Packets can be converted to WebCodecs API types:

const videoChunk = packet.toEncodedVideoChunk();
await videoDecoder.decode(videoChunk);

VideoSample

The VideoSample class represents a raw, unencoded video frame with direct pixel data access.

Structure

A VideoSample provides:

sample.ts:171-210

export class VideoSample implements Disposable {
  readonly format: VideoSamplePixelFormat | null;  // Pixel format (I420, RGBA, etc.)
  readonly visibleRect: Rectangle;                 // Visible region
  readonly codedWidth: number;                     // Frame width
  readonly codedHeight: number;                    // Frame height
  readonly rotation: Rotation;                     // 0, 90, 180, or 270 degrees
  readonly timestamp: number;                      // Presentation time (seconds)
  readonly duration: number;                       // Duration (seconds)
  readonly colorSpace: VideoSampleColorSpace;      // Color space info
}

Creating video samples

You can create video samples from various sources:

const sample = new VideoSample(videoFrame, {
  timestamp: 1.5,
  duration: 0.033,
  rotation: 90
});

Pixel formats

Mediabunny supports 21 pixel formats:

sample.ts:86-121

export const VIDEO_SAMPLE_PIXEL_FORMATS = [
  // 4:2:0 Y, U, V
  'I420', 'I420P10', 'I420P12',
  // 4:2:0 Y, U, V, A (with alpha)
  'I420A', 'I420AP10', 'I420AP12',
  // 4:2:2 Y, U, V
  'I422', 'I422P10', 'I422P12',
  // 4:2:2 Y, U, V, A
  'I422A', 'I422AP10', 'I422AP12',
  // 4:4:4 Y, U, V
  'I444', 'I444P10', 'I444P12',
  // 4:4:4 Y, U, V, A
  'I444A', 'I444AP10', 'I444AP12',
  // 4:2:0 Y, UV
  'NV12',
  // 4:4:4 RGBA
  'RGBA', 'RGBX', 'BGRA', 'BGRX',
] as const;

Understanding pixel formats

I420: Most common format for video encoding (4:2:0 subsampling)
I420P10/P12: 10-bit and 12-bit variants for HDR content
I420A: I420 with alpha channel for transparency
RGBA/BGRA: Full-color formats with alpha, useful for graphics
RGBX/BGRX: Opaque RGB formats
NV12: Efficient format used by many hardware decoders

Working with video samples

Reading pixels
Drawing to canvas
Advanced drawing

Access raw pixel data from a sample:

const sample = await videoTrack.readSample();

// Get buffer size needed
const size = sample.allocationSize();

// Copy pixels to buffer
const buffer = new Uint8Array(size);
const layout = await sample.copyTo(buffer);

console.log('Pixel data copied:', buffer.length, 'bytes');
console.log('Plane layout:', layout);

Draw a video sample to a 2D canvas:

const canvas = document.createElement('canvas');
const ctx = canvas.getContext('2d');

canvas.width = sample.displayWidth;
canvas.height = sample.displayHeight;

// Draw the sample (rotation is handled automatically)
sample.draw(ctx, 0, 0);

Use advanced drawing with fit modes:

sample.drawWithFit(ctx, {
  fit: 'contain',  // or 'cover', 'fill'
  rotation: 90,    // Override rotation
  crop: {          // Crop region
    left: 100,
    top: 100,
    width: 800,
    height: 600
  }
});

Converting to VideoFrame

Convert a sample to WebCodecs VideoFrame:

const videoFrame = sample.toVideoFrame();
await videoEncoder.encode(videoFrame);
videoFrame.close(); // Don't forget to close!

Resource management

Video samples hold resources that must be explicitly freed:

const sample = await videoTrack.readSample();

// Use the sample
processSample(sample);

// ALWAYS close when done
sample.close();

// Or use explicit resource management
using sample = await videoTrack.readSample();
// Automatically closed at end of scope

AudioSample

The AudioSample class represents raw, unencoded audio data with direct PCM access.

Structure

An AudioSample provides:

sample.ts:1366-1391

export class AudioSample implements Disposable {
  readonly format: AudioSampleFormat;          // Sample format (f32, s16, etc.)
  readonly sampleRate: number;                 // Sample rate in Hz
  readonly numberOfFrames: number;             // Length in frames
  readonly numberOfChannels: number;           // Channel count
  readonly duration: number;                   // Duration (seconds)
  readonly timestamp: number;                  // Presentation time (seconds)
}

Creating audio samples

const sample = new AudioSample(audioData);

Audio formats

Supported audio sample formats:

type AudioSampleFormat = 
  | 'f32' | 'f32-planar'       // 32-bit float
  | 's16' | 's16-planar'       // 16-bit signed int
  | 's32' | 's32-planar'       // 32-bit signed int  
  | 'u8'  | 'u8-planar';       // 8-bit unsigned int

Planar formats store each channel separately, while non-planar formats interleave channels.

Working with audio samples

Reading audio data
Converting formats

const sample = await audioTrack.readSample();

// Get buffer size for stereo output
const size = sample.allocationSize({
  planeIndex: 0,
  format: 'f32'  // Convert to float32
});

// Copy audio data
const buffer = new ArrayBuffer(size);
sample.copyTo(buffer, {
  planeIndex: 0,
  format: 'f32',
  frameOffset: 0,
  frameCount: sample.numberOfFrames
});

// Read as interleaved float32
sample.copyTo(buffer, {
  planeIndex: 0,
  format: 'f32'
});

// Read as planar int16, left channel only
sample.copyTo(buffer, {
  planeIndex: 0,  // Left channel
  format: 's16-planar'
});

Converting to AudioData

const audioData = sample.toAudioData();
await audioEncoder.encode(audioData);
audioData.close();

Resource management

Like video samples, audio samples must be closed:

const sample = await audioTrack.readSample();
processAudio(sample);
sample.close();

// Or use explicit resource management
using sample = await audioTrack.readSample();

When to use packets vs samples

Choose the right data type for your use case:

Use Packets
Use Samples

Use EncodedPacket when you:✅ Copy/remux media without re-encoding
✅ Need efficient storage and transfer
✅ Don’t need to manipulate pixel/audio data
✅ Want to preserve original encoding quality

// Remuxing: packets in, packets out (fast)
for await (const packet of inputTrack.readPackets()) {
  outputSource.addPacket(packet);
}

Use VideoSample/AudioSample when you:✅ Need to decode and process raw media data
✅ Apply filters, effects, or transformations
✅ Draw video frames to canvas
✅ Analyze audio waveforms
✅ Transcode to different codecs

// Processing: decode to samples, manipulate, re-encode
for await (const sample of inputTrack.readSamples()) {
  const processed = applyFilter(sample);
  await encoder.encode(processed.toVideoFrame());
  sample.close();
  processed.close();
}

Performance considerations

Packets

Very fast (no encoding/decoding)
Low memory usage
Perfect for remuxing
Limited processing options

Samples

Slower (requires decode/encode)
High memory usage
Full pixel/audio access
Enables rich processing

Example: Remuxing (fast)

// Copy MP4 to WebM without re-encoding
for await (const packet of videoTrack.readPackets()) {
  outputVideoSource.addPacket(packet);  // Direct packet copy
}

Example: Transcoding (slower)

// Convert H.264 to VP9 (requires decode + encode)
const decoder = new VideoDecoder({ /* ... */ });
const encoder = new VideoEncoder({ /* ... */ });

for await (const sample of videoTrack.readSamples()) {
  const frame = sample.toVideoFrame();
  await encoder.encode(frame);  // Re-encode to VP9
  frame.close();
  sample.close();
}

Get Started

Core Concepts

Guides

Advanced

Extensions

The fundamental difference

Packets

Samples

EncodedPacket

Structure

Creating packets

Using packets

Packet types

Converting to WebCodecs

VideoSample

Structure

Creating video samples

Pixel formats

Working with video samples

Converting to VideoFrame

Resource management

AudioSample

Structure

Creating audio samples

Audio formats

Working with audio samples

Converting to AudioData

Resource management

When to use packets vs samples

Performance considerations

Packets

Samples

Example: Remuxing (fast)

Example: Transcoding (slower)

Next steps

Input and Output

Encoding and Decoding

Build docs developers (and LLMs) love

Get Started

Core Concepts

Guides

Advanced

Extensions

Documentation Index

​The fundamental difference

Packets

Samples

​EncodedPacket

​Structure

​Creating packets

​Using packets

​Packet types

​Converting to WebCodecs

​VideoSample

​Structure

​Creating video samples

​Pixel formats

​Working with video samples

​Converting to VideoFrame

​Resource management

​AudioSample

​Structure

​Creating audio samples

​Audio formats

​Working with audio samples

​Converting to AudioData

​Resource management

​When to use packets vs samples

​Performance considerations

Packets

Samples

​Example: Remuxing (fast)

​Example: Transcoding (slower)

​Next steps

Input and Output

Encoding and Decoding

Build docs developers (and LLMs) love

The fundamental difference

EncodedPacket

Structure

Creating packets

Using packets

Packet types

Converting to WebCodecs

VideoSample

Structure

Creating video samples

Pixel formats

Working with video samples

Converting to VideoFrame

Resource management

AudioSample

Structure

Creating audio samples

Audio formats

Working with audio samples

Converting to AudioData

Resource management

When to use packets vs samples

Performance considerations

Example: Remuxing (fast)

Example: Transcoding (slower)

Next steps