Media Types, Readers, and Codec Utilities

The WebRTS Client exports a set of lower-level media primitives for advanced use cases: codec detection and string encoding, CMAF/fMP4 container demuxing and writing, RTS binary format parsing, MSE buffer management, and screen-resolution helpers. Most application developers only need Player, Source, and Metadata — but these types become essential when building custom renderers, testing pipeline components in isolation, or integrating the library into non-standard playback architectures.

// Named namespace imports — Media and AVC are exported as named namespace exports
import { Media, AVC } from '@ceeblue/wrts-client';

// Named exports (readers, writers, buffer classes)
import {
  Reader, ReaderError,
  CMAFReader,
  RTSReader,
  CMAFWriter, CMAFWriterError,
  MediaBuffer, MediaBufferError,
  MediaPlayback, MediaPlaybackError,
  MediaTrack
} from '@ceeblue/wrts-client';

Media and AVC are re-exported as named namespace exports from the package root via export * as Media and export * as AVC. Import them by name: import { Media, AVC } from '@ceeblue/wrts-client'. You then access members as Media.Type.VIDEO, Media.Codec.H264, AVC.readCodecString, etc.

Media Module

The Media module (from src/media/Media.ts) defines the core enums and types shared across the entire library.

Media.Type

Identifies the kind of data carried by a track or sample.

Value	Number	Description
`Media.Type.DATA`	`0`	Data or subtitle track (ID3 tags, JSON metadata, WebVTT, etc.)
`Media.Type.AUDIO`	`1`	Audio elementary stream
`Media.Type.VIDEO`	`2`	Video elementary stream

Media.Codec

Identifies a specific codec. Used to filter tracks in Metadata.filterAudios/filterVideos/filterDatas and to drive MSE codec strings.

Value	String	Description
`Media.Codec.UNKNOWN`	`''`	Codec not yet identified
`Media.Codec.H264`	`'H264'`	H.264 / AVC video
`Media.Codec.HEVC`	`'HEVC'`	H.265 / HEVC video
`Media.Codec.VP8`	`'VP8'`	VP8 video
`Media.Codec.MP3`	`'MP3'`	MPEG-1/2 Audio Layer III
`Media.Codec.AAC`	`'AAC'`	Advanced Audio Coding
`Media.Codec.OPUS`	`'OPUS'`	Opus audio
`Media.Codec.ID3`	`'ID3'`	ID3 timed metadata (data track)
`Media.Codec.JSON`	`'JSON'`	JSON metadata (data track)
`Media.Codec.SUBTITLE`	`'SUBTITLE'`	Subtitle / closed-caption data

Media.Sample

Represents a single decoded media sample produced by a Reader and consumed by a MediaBuffer.

type Media.Sample = {
  time: number;              // Decode timestamp in milliseconds
  duration: number;          // Sample duration in milliseconds
  data: Uint8Array;          // Raw sample payload
  compositionOffset?: number; // PTS – DTS offset in milliseconds (B-frame reordering)
  isKeyFrame?: boolean;      // true for IDR/key frames
  subSamples?: Array<{       // DRM SENC sub-sample map
    clearBytes: number;
    encryptedBytes: number;
  }>;
  iv?: Uint8Array;           // Per-sample IV when ContentProtection.ivMode === 'sample'
};

Media.Resolution

A simple width/height pair used for video track resolution and screen size comparisons.

type Media.Resolution = {
  width: number;
  height: number;
};

Media.Tracks

Selects which tracks are active. Passed to onInitTracks and used internally for ABR switching.

type Media.Tracks = {
  audio?: number;        // Audio track ID; undefined = Multi-Bitrate (MBR), -1 = disable audio
  video?: number;        // Video track ID; undefined = MBR, -1 = disable video
  data?: Set<number>;    // Set of data track IDs to receive; undefined = receive all
};

Media Utility Functions

Media.typeToString(type)

function

Converts a Media.Type enum value to a lowercase string.

function typeToString(type: Media.Type): string
// Media.Type.AUDIO  → 'audio'
// Media.Type.VIDEO  → 'video'
// Media.Type.DATA   → 'data'
// (unknown)         → 'unknown'

Media.screenResolution()

function

Returns the physical screen resolution in pixels, accounting for window.devicePixelRatio. Automatically swaps width/height on portrait (smartphone) screens so the larger dimension is always width. Returns undefined in non-browser environments.

function screenResolution(): Media.Resolution | undefined

Media.overScreenSize(resolution, screen?)

function

Returns true if resolution exceeds both the width and height of screen. Useful for filtering out video tracks that the display cannot render at full quality.

function overScreenSize(resolution: Media.Resolution, screen?: Media.Resolution): boolean

Media.MAX_GOP_DURATION

number

Maximum Group of Pictures duration constant, set to 10000 ms (10 seconds). Used internally to bound buffer window calculations.

Reader Abstract Class

Reader is the abstract base class for all container-format demuxers. Concrete implementations (CMAFReader, RTSReader) extend it and implement the parse() method. Feed binary data incrementally via read(); the reader buffers incomplete frames automatically.

Constructor

new Reader()

Methods

read(data)

method

Feeds a chunk of binary data into the reader. Internally accumulates incomplete frames between calls.

read(data: BufferSource): void

Show Parameters

data

BufferSource

required

An ArrayBuffer, ArrayBufferView, or any BufferSource containing the next chunk of the stream.

reset()

method

Clears any internally buffered incomplete data. Call this when seeking or after a track change.

reset(): void

Events

onSample(type, trackId, sample)

event

Fired for each decoded media sample.

onSample(type: Media.Type, trackId: number, sample: Media.Sample): void

onMetadata(metadata)

event

Fired when a metadata packet is found embedded in the stream (e.g. an RTS metadata frame or a CMAF emsg box carrying a JSON manifest).

onMetadata(metadata: Metadata): void

onInitTracks(tracks)

event

Fired once after the container initialization segment is parsed and track IDs are known.

onInitTracks(tracks: Media.Tracks): void

onError(error)

event

Fired on parse errors. The default implementation logs the error.

onError(error: ReaderError): void

ReaderError Type

Invalid payload

The binary data does not conform to the expected format at a specific point in the parse.

{ type: 'ReaderError'; name: 'Invalid payload'; detail: string }

Unknown format

An unrecognized box type or packet type was encountered.

{ type: 'ReaderError'; name: 'Unknown format'; format: string | number }

Unsupported format

A recognized but unimplemented format variant was encountered (e.g. an unsupported senc version).

{ type: 'ReaderError'; name: 'Unsupported format'; format: string | number }

Unfound track

A fragment header (tfhd) referenced a track ID that was never declared in the moov box.

{ type: 'ReaderError'; name: 'Unfound track'; track: number }

CMAFReader

CMAFReader extends Reader and parses CMAF/fMP4 (Fragmented MP4) container segments. It handles moov, moof, mdat, DRM boxes (tenc, senc, sinf/schi), and emsg event message boxes. It is used internally by HTTPAdaptiveSource and WSSource when receiving CMAF-segmented streams.

Constructor

new CMAFReader(passthrough?: boolean)

passthrough

boolean

When true, the raw CMAF bytes are collected and passed as sample.data for each sample rather than the unwrapped elementary stream payload. Defaults to false.

Additional Event

onMessage(name, time, duration, data)

event

Fired when an emsg (Event Message Box) is found in the stream, such as an ID3 timed metadata event.

onMessage(name: string, time: number, duration: number, data: Uint8Array): void

Static Method

CMAFReader.parseSinfTrack(initData)

static method

Parses a raw sinf/schi payload (as received in a FairPlay encrypted event) and returns the track encryption descriptor, including KID and IV configuration.

static parseSinfTrack(initData: Uint8Array): TrackEncryption | undefined

RTSReader

RTSReader extends Reader and parses the RTS binary container format used by WebRTS WebSocket streams. The RTS format uses 7-bit variable-length integers for compact framing and carries audio, video, data, metadata, and init-tracks commands in a single multiplexed stream.

Constructor

new RTSReader(options?: { withSize?: boolean })

options.withSize

boolean

When true, each packet in the stream is preceded by a 7-bit variable-length size field. When false (default), the reader assumes a frame-oriented transport such as WebSocket where each message is already a complete packet.

CMAFWriter

CMAFWriter writes CMAF/fMP4 output optimized for live streaming. It is used internally by MediaBuffer to wrap elementary stream samples in proper moov/moof/mdat boxes before they are appended to an MSE SourceBuffer. It currently supports H.264 video and AAC/MP3 audio, with optional CENC/CBCS content protection.

Constructor

new CMAFWriter()

Methods

init(track, contentProtection?)

method

Writes the ftyp + moov initialization segment for the given track. Returns a CMAFWriterError if the codec or track type is unsupported, undefined on success.

init(track: MediaTrack, contentProtection?: ContentProtection): CMAFWriterError | undefined

write(sample, contentProtection?)

method

Wraps a single Media.Sample in moof + mdat boxes and fires onWrite.

write(sample: Media.Sample, contentProtection?: ContentProtection): void

Event

onWrite(packet)

event

Fired with each encoded CMAF packet ready to be appended to a SourceBuffer.

onWrite(packet: Uint8Array): void

CMAFWriterError Type

type CMAFWriterError =
  | { type: 'CMAFWriterError'; name: 'Unsupported codec'; codec: Media.Codec }
  | { type: 'CMAFWriterError'; name: 'Unsupported track type'; trackType: Media.Type };

AVC Module

The AVC module (src/media/AVC.ts) provides H.264/AVC-centric codec string parsing, SPS decoding, and audio config serialization utilities. These are used internally by CMAFReader and Metadata to populate MediaTrack fields from binary codec configuration data.

AVC.readCodecString(codecString, out)

function

Parses an RFC 6381 codec descriptor string and writes the detected codec and type into out. Returns true on a successful match.

function readCodecString(
  codecString: string,
  out: { codec: Media.Codec; type: Media.Type }
): boolean

Recognized prefixes include avc1/avc2/avc3/avc4/h264/x264/264, mp4a.40 (AAC), mp4a.40.34 (MP3), opus, vp8, id3, json, subtitle.

AVC.writeCodecString(codec, config?)

function

Builds an RFC 6381 codec descriptor string from a Media.Codec enum and optional binary config data (e.g. avcC bytes for H.264 profile/level encoding).

function writeCodecString(codec: Media.Codec, config?: Uint8Array): string
// e.g. Media.Codec.H264 + avcC bytes → 'avc1.640028'
// Media.Codec.AAC + esds bytes       → 'mp4a.40.2'
// Media.Codec.OPUS                   → 'opus'

AVC.readVideoConfig(data)

function

Parses an avcC or hvcC binary blob and returns a VideoConfig object containing the raw sps and optional pps byte arrays.

function readVideoConfig(data: Uint8Array): VideoConfig
// VideoConfig = { sps: Uint8Array; pps?: Uint8Array }

AVC.writeVideoConfig(writer, config)

function

Serializes a VideoConfig (SPS + PPS) into an avcC binary blob using the provided BinaryWriter.

function writeVideoConfig(writer: BinaryWriter, config: VideoConfig): BinaryWriter

AVC.parseSPS(sps, out)

function

Decodes an H.264 Sequence Parameter Set to extract resolution and frame rate, writing them into out only if not already populated.

function parseSPS(
  sps: Uint8Array,
  out: { resolution: Media.Resolution; rate: number }
): boolean

AVC.writeAudioConfig(type, rate, channels)

function

Encodes MPEG-4 Audio Specific Config (ASC) bytes for an AAC stream.

function writeAudioConfig(type: number, rate: number, channels: number): Uint8Array

AVC.readAudioConfig(data)

function

Decodes MPEG-4 ASC bytes and returns { rate, channels }.

function readAudioConfig(data: Uint8Array): { rate: number; channels: number } | undefined

AVC.nalType(byte)

function

Extracts the 5-bit NAL unit type from a NAL header byte.

function nalType(byte: number): NAL

MediaBuffer

MediaBuffer wraps a single MSE SourceBuffer (audio or video) and manages the full lifecycle of appending, codec-type changes, buffer trimming, and quota-exceeded recovery. It uses CMAFWriter internally to convert Media.Sample objects into fMP4 data before appending.

Constructor

new MediaBuffer(
  mediaSource: MediaSource,
  mimeType: string,           // e.g. 'video/mp4' or 'audio/mp4'
  isAlreadyCMAF?: boolean     // true = samples are already fMP4 segments; skip CMAFWriter
)

Key Properties

isVideo

boolean

true if the MIME type starts with 'video'.

startTime

number (get/set)

The playback start boundary in seconds. Setting this triggers a flush() that removes buffered data before the new start time.

endTime

number

The end of the currently buffered range in seconds (read-only).

Key Methods

append(metadata, trackId, sample)

method

Appends a single decoded sample to the buffer. Automatically initializes the SourceBuffer codec on first call or on track ID change.

append(metadata: Metadata, trackId: number, sample: Media.Sample): this | undefined

flush(fixHole?)

method

Drains the internal packet queue into the SourceBuffer. Pass fixHole = true to force removal of timeline discontinuities.

abort()

method

Cancels all pending operations and releases the SourceBuffer.

Events

onUpdate()

event

Fired after each successful buffer update cycle.

onDataAppended(data)

event

Fired with each raw Uint8Array appended to the SourceBuffer. Useful for debugging MSE ingestion.

onError(error)

event

Fired on any buffer error (see MediaBufferError).

MediaBufferError Type

type MediaBufferError =
  | { type: 'MediaBufferError'; name: 'SourceBuffer aborted'; mimeType: string }
  | { type: 'MediaBufferError'; name: 'Track without metadata'; track: number }
  | { type: 'MediaBufferError'; name: 'Append buffer issue'; detail: string }
  | { type: 'MediaBufferError'; name: 'Exceeds buffer size' }
  | CMAFWriterError;

MediaPlayback

MediaPlayback coordinates one audio MediaBuffer and one video MediaBuffer over a shared MediaSource. It exposes a unified startTime/endTime that spans both tracks, handles MSE endOfStream, and fires onProgress as the buffers advance.

Constructor

new MediaPlayback(
  mediaSource: MediaSource,
  passthroughCMAF?: boolean // pass-through CMAF mode; forwarded to MediaBuffer
)

Key Properties

closed

boolean

true when both the audio and video buffers have been released.

startTime

number (get/set)

The earliest common buffered time across both tracks (in seconds). Setting this trims both buffers.

endTime

number

The latest common sync point across both tracks (in seconds).

audioEnabled

boolean (get/set)

Enables the audio track (creates a new audio/mp4 MediaBuffer). Disabling audio is not supported; attempting to set false closes the playback with an error.

videoEnabled

boolean (get/set)

Enables the video track (creates a new video/mp4 MediaBuffer). Disabling video is not supported; attempting to set false closes the playback with an error.

Key Methods

appendAudio(metadata, trackId, sample)

method

Appends an audio sample to the audio buffer.

appendVideo(metadata, trackId, sample)

method

Appends a video sample to the video buffer.

flush(fixHole?)

method

Flushes both audio and video buffers.

close(error?)

method

Ends the MediaSource stream, aborts both buffers, and fires onClose.

Events

onClose(error?)

event

Fired when the playback is closed, with an optional error if the closure was abnormal.

onBufferOverflow()

event

Fired when a buffer quota is exceeded. Call player.play() or advance startTime to recover.

onProgress()

event

Fired each time the combined buffer end time advances.

onAudioAppended(data)

event

Raw audio bytes appended to MSE (for debugging).

onVideoAppended(data)

event

Raw video bytes appended to MSE (for debugging).

MediaPlaybackError Type

type MediaPlaybackError =
  | { type: 'MediaPlaybackError'; name: 'Media buffer init error'; mimeType: string; detail: string }
  | { type: 'MediaPlaybackError'; name: 'Cannot create a new track'; trackType: Media.Type }
  | { type: 'MediaPlaybackError'; name: 'Cannot remove a track'; trackType: Media.Type }
  | MediaBufferError;

MediaTrack

MediaTrack is a plain data class representing a single track descriptor. Its properties are documented in full on the Metadata reference page under MediaTrack Properties, since MediaTrack instances are always obtained from Metadata.tracks, Metadata.audioTracks, Metadata.videoTracks, or Metadata.dataTracks. The class also provides a convenience toString() method that builds a human-readable label:

// e.g. "H264 1920x1080 30fps 4000kbps"
console.log(metadata.videoTracks[0].toString());

Reader, CMAFReader, RTSReader, CMAFWriter, MediaBuffer, and MediaPlayback are all designed for advanced or internal use. Most application developers building a standard live stream viewer only need Player, Source/HTTPAdaptiveSource/WSSource, and Metadata.

Core

Sources

Media

Documentation Index

​Media Module

​Media.Type

​Media.Codec

​Media.Sample

​Media.Resolution

​Media.Tracks

​Media Utility Functions

​Reader Abstract Class

​Constructor

​Methods

​Events

​ReaderError Type

​CMAFReader

​Constructor

​Additional Event

​Static Method

​RTSReader

​Constructor

​CMAFWriter

​Constructor

​Methods

​Event

​CMAFWriterError Type

​AVC Module

​MediaBuffer

​Constructor

​Key Properties

​Key Methods

​Events

​MediaBufferError Type

​MediaPlayback

​Constructor

​Key Properties

​Key Methods

​Events

​MediaPlaybackError Type

​MediaTrack

Build docs developers (and LLMs) love

Media Module

Media.Type

Media.Codec

Media.Sample

Media.Resolution

Media.Tracks

Media Utility Functions

Reader Abstract Class

Constructor

Methods

Events

ReaderError Type

CMAFReader

Constructor

Additional Event

Static Method

RTSReader

Constructor

CMAFWriter

Constructor

Methods

Event

CMAFWriterError Type

AVC Module

MediaBuffer

Constructor

Key Properties

Key Methods

Events

MediaBufferError Type

MediaPlayback

Constructor

Key Properties

Key Methods

Events

MediaPlaybackError Type

MediaTrack