Skip to content
tedconfPublic

About

Streaming parser for m3u8 files in Node.js

Topics

Resources

Stars

192 stars

Watchers

34 watching

Forks

Latest commit

 

History

94 Commits

Folders and files

Repository files navigation

@tedconf/m3u8

A streaming parser and writer for HTTP Live Streaming (HLS) playlists.

It reads master playlists and media playlists into an M3U object. You can rewrite segment URIs, merge playlists, and write the result back out. TED's hls service uses it to build the playlists behind hls.ted.com.

  • Requires Node.js 22 or later.
  • Has no runtime dependencies.

Package name. This repo used to publish m3u8 on npm. That package is frozen at 0.0.10. From 1.0.0 onward, releases go to GitHub Packages as @tedconf/m3u8. Installing it under the alias m3u8 keeps require('m3u8') working.

Contents

Install

GitHub Packages needs a token to install packages, even public ones.

  1. Point the @tedconf scope at GitHub Packages. Add this to the project's .npmrc:

    @tedconf:registry=https://npm.pkg.github.com
    
  2. Log in once with a GitHub personal access token that has read:packages:

    npm login --scope=@tedconf --registry=https://npm.pkg.github.com
    
  3. Install under the m3u8 alias:

    npm install m3u8@npm:@tedconf/m3u8@1.0.0-beta.1
    

In GitHub Actions:

  • Give the job packages: read.
  • Pass registry-url: https://npm.pkg.github.com and scope: '@tedconf' to actions/setup-node.
  • Set NODE_AUTH_TOKEN: ${{ secrets.GITHUB_TOKEN }} on the install step.
  • In the package's settings, under Manage Actions access, give the consuming repository read access.

Quick start

const fs   = require('fs');
const m3u8 = require('m3u8');

const parser = m3u8.createStream();

parser.on('item', (item) => {
  // Runs for each PlaylistItem, StreamItem, IframeStreamItem and MediaItem.
  item.set('uri', 'https://cdn.example.com/' + item.get('uri'));
});

parser.on('m3u', (m3u) => {
  console.log(m3u.totalDuration());
  process.stdout.write(m3u.toString());
});

parser.on('error', (error) => {
  // The first line was not #EXTM3U.
});

fs.createReadStream('index.m3u8').pipe(parser);

To parse a string, call parser.end(text) instead of piping.

Parser

m3u8.createStream() returns a writable stream. Pipe a playlist into it, or call write() and end(). It accepts strings and Buffers. Lines may end in LF or CRLF, and multi-byte UTF-8 characters can be split across chunks.

Event Payload When
item the item Once per item. A PlaylistItem or StreamItem fires when its URI line is read, so its tags are already set. MediaItem and IframeStreamItem fire when their tag is read.
m3u M3U After the input ends. Fires on the tick after end().
end none Right after m3u.
error Error Once, when the first line is not #EXTM3U or a line can't be parsed (for example an #EXTINF without a duration, or a URI before any tag). The stream is then destroyed and m3u does not fire.

Prefer stream.pipeline(source, parser, callback) to source.pipe(parser). When the parser errors, pipe() unpipes and pauses a modern Readable source, which then never ends (an HTTP response stays open). pipeline() destroys the source.

M3U

m3u8.M3U is also available as require('m3u8').M3U.

const m3u = m3u8.M3U.create();
m3u.set('version', 3);
m3u.set('targetDuration', 10);
m3u.set('playlistType', 'VOD');
m3u.addPlaylistItem({ duration: 9.009, uri: 'segment-1.ts' });
m3u.toString();

Playlist properties

get(key) and set(key, value) read and write playlist-level tags. Known tags are stored under a camelCase key. set() also accepts the tag name.

Key Tag Type
allowCache EXT-X-ALLOW-CACHE string
discontinuitySequence EXT-X-DISCONTINUITY-SEQUENCE integer
endlist EXT-X-ENDLIST flag, written after the last segment
iframesOnly EXT-X-I-FRAMES-ONLY flag
independentSegments EXT-X-INDEPENDENT-SEGMENTS flag
mediaSequence EXT-X-MEDIA-SEQUENCE integer
playlistType EXT-X-PLAYLIST-TYPE string
targetDuration EXT-X-TARGETDURATION integer
version EXT-X-VERSION integer

Any other tag is stored under its own name, for example m3u.get('EXT-X-START'). A tag with no value is stored as null and written back as a bare tag. See Limitations.

Methods

Method Description
M3U.create() Returns an empty playlist.
items Arrays of each item type: items.PlaylistItem, items.StreamItem, items.IframeStreamItem, items.MediaItem.
addItem(item) Appends an item object to the array for its type.
addPlaylistItem(data), addStreamItem(data), addIframeStreamItem(data), addMediaItem(data) Create an item from a plain object (see Items) and append it.
removePlaylistItem(index) Removes a segment. Throws RangeError if the index is out of range.
totalDuration() Sum of segment durations, in seconds.
domainDurations() Durations split at each discontinuity, e.g. [intro, talk].
merge(other) Appends another media playlist's segments. Marks the first appended segment as a discontinuity and keeps the larger targetDuration and version. endlist comes from other, since its segments end the result. Raises version to 6 if any segment has an init segment. Mutates and returns this.
hasMap() true if any segment has an EXT-X-MAP init segment.
toString() Writes the playlist. Media playlists end with #EXT-X-ENDLIST when playlistType is VOD.
serialize() Returns a plain JSON-safe object, e.g. for a cache.
M3U.unserialize(object) Rebuilds an M3U from serialize() output.

Items

All items share get(key), set(key, value), setData(object), serialize() and toString(). A key that is one of the item's properties reads or writes that property. Any other key reads or writes an attribute from the tag's attribute list, such as bandwidth, codecs or resolution.

PlaylistItem: a media segment

Property Source tag Notes
uri URI line
duration, title EXTINF Written with four decimal places.
byteRange EXT-X-BYTERANGE String, e.g. "1000@0".
discontinuity EXT-X-DISCONTINUITY true on the segment after the tag.
date EXT-X-PROGRAM-DATE-TIME String as parsed. A Date is written with toISOString().
map, mapByteRange EXT-X-MAP See fMP4 / CMAF init segments.
daiPlacementOpportunity none Written only: set it to true to write EXT-X-PLACEMENT-OPPORTUNITY.

StreamItem, IframeStreamItem, MediaItem: master playlist entries

These come from EXT-X-STREAM-INF, EXT-X-I-FRAME-STREAM-INF and EXT-X-MEDIA. Their attributes are read and written by lowercase name:

item.get('bandwidth');                 // 1104484
item.get('resolution');                // [640, 480]
item.get('codecs');                    // 'avc1.64001f,mp4a.40.2'
item.set('uri', 'https://cdn.example.com/' + item.get('uri'));

Each attribute is parsed and written according to its HLS type: quoted string, integer, decimal, resolution, YES/NO or enumerated string. Attributes the library doesn't know are kept as they are and logged with console.error.

fMP4 / CMAF init segments

EXT-X-MAP applies to every segment after it until the next EXT-X-MAP. The parser copies it onto each of those PlaylistItems:

  • map holds the URI.
  • mapByteRange holds the BYTERANGE value, or null.

Because each segment holds its own copy, rewriting it in an item listener works the same way as rewriting uri:

parser.on('item', (item) => {
  item.set('uri', resolve(item.get('uri')));
  if (item.get('map')) {
    item.set('map', resolve(item.get('map')));
  }
});

toString():

  • writes EXT-X-MAP before a segment's EXTINF, on the first segment, after every EXT-X-DISCONTINUITY, and wherever the map changes;
  • writes at least EXT-X-VERSION:6 when any segment has a map.

merge() keeps each playlist's own init segment. An intro and a talk merged into one playlist look like this:

#EXT-X-MAP:URI="https://cdn.example.com/intro/init.mp4"
#EXTINF:6.0060,
https://cdn.example.com/intro/segment-1.m4s
...
#EXT-X-DISCONTINUITY
#EXT-X-MAP:URI="https://cdn.example.com/talk/init.mp4"
#EXTINF:6.0060,
https://cdn.example.com/talk/segment-1.m4s

Supported tags

Tag Stored as
EXTINF, EXT-X-BYTERANGE, EXT-X-DISCONTINUITY, EXT-X-PROGRAM-DATE-TIME, EXT-X-MAP, EXT-X-KEY PlaylistItem properties. keys is an array of the raw attribute lists of the EXT-X-KEY tags in effect for the segment.
EXT-X-STREAM-INF StreamItem
EXT-X-I-FRAME-STREAM-INF IframeStreamItem
EXT-X-MEDIA MediaItem
EXT-X-ENDLIST sets playlistType to VOD
Playlist properties table above M3U properties

Limitations

Known limitations:

  • Unknown tags are playlist-level. Any tag not listed above is stored as a playlist property and written back in the header. Only its last value is kept, and it is not tied to a segment. This affects segment-level tags such as EXT-X-CUE-OUT/EXT-X-CUE-IN, EXT-X-DATERANGE and EXT-X-GAP. Don't round-trip playlists with per-segment ad markers through this library.
  • Don't mix TS and fMP4 in one playlist. HLS has no tag that clears an init segment. If a segment without a map follows one with a map, players apply the earlier map to it. So merge a CMAF intro only with a CMAF talk, never with a TS one.

Upgrading from 0.0.10

1.0.0 is a breaking release:

  • The package is @tedconf/m3u8 on GitHub Packages, not m3u8 on npm (see Install).
  • It requires Node.js 22 or later.
  • Unknown tags with no value, such as #EXT-X-PLACEMENT-OPPORTUNITY, are written back as bare tags. 0.0.10 wrote them as #TAG:undefined.
  • The parser is a Writable stream only. It no longer emits a data event per line, and can't be piped onward. It still emits end.
  • Parse errors emit error once and destroy the stream. 0.0.10 emitted error for every line after an invalid first line and then still emitted m3u, and threw from write() on other malformed input.
  • EXT-X-PROGRAM-DATE-TIME, EXT-X-MAP and EXT-X-DISCONTINUITY written between EXTINF and the URI apply to that segment.
  • merge() keeps the higher version of the two playlists.
  • EXT-X-ENDLIST sets the endlist flag, and toString() writes it when endlist is set or playlistType is VOD. A playlist with no EXT-X-PLAYLIST-TYPE still gets VOD, as in 0.0.10, but an EVENT playlist that ends stays EVENT instead of becoming VOD.
  • EXTINF titles keep their commas. 0.0.10 cut them at the first comma.
  • A URI line that follows another URI line starts a new segment. 0.0.10 overwrote the previous segment's URI.
  • EXT-X-KEY is parsed per segment into keys, and written before the first segment and wherever it changes. It is no longer a playlist header property, so key rotation round-trips. merge() writes #EXT-X-KEY:METHOD=NONE when an unencrypted playlist follows an encrypted one.
  • Empty input and #EXTINF:Infinity are parse errors.
  • The m3u event fires on the tick after end(), not synchronously inside it.
  • EXT-X-PROGRAM-DATE-TIME sets date on the next segment. It is no longer a playlist header property.
  • EXT-X-MAP is parsed per segment (see above). It is no longer a playlist header property, so m3u.get('EXT-X-MAP') is undefined. If you patched PlaylistItem.prototype.toString to write the map yourself, remove the patch, or the tag is written twice.
  • EXT-X-DISCONTINUITY-SEQUENCE is the integer property discontinuitySequence.
  • M3U.unserialize fills in properties missing from older cached data, such as map.
  • Attributes without a value, such as a bare CHARACTERISTICS in EXT-X-STREAM-INF, are kept and written back as bare keys.

The module's exports, classes and method names are unchanged, so code that only calls them needs no changes beyond the install.

On every playlist TED serves today (TS master, media and iframe playlists, and merged intro + talk playlists), the output is byte-for-byte the same as 0.0.10. None of them carry EXT-X-PROGRAM-DATE-TIME or EXT-X-KEY; a playlist that does will differ, because those tags now move to their segments.

Development

nvm use
npm ci
npm test

Tests use mocha, should and sinon:

  • Unit tests are in test/*.test.js.
  • Fixture-based tests are in test/acceptance/.
  • Fixture playlists are in test/fixtures/.

CI runs npm test on Node 22, 24 and 26 for every pull request and every push to main. Merging to main needs passing CI and a review from a code owner.

Releasing

  1. Bump version in package.json and package-lock.json in a pull request, e.g. npm version 1.0.0-beta.2 --no-git-tag-version.
  2. After it merges, create a GitHub release on main tagged v<version>. Mark prereleases as pre-release.
  3. The publish workflow runs the tests, checks that the tag matches package.json, and publishes to GitHub Packages.
    • Versions with a hyphen (1.0.0-beta.1) get the next dist-tag.
    • Other versions get latest.

License

MIT. See LICENSE.txt.

About

Streaming parser for m3u8 files in Node.js

Topics

Resources

Stars

192 stars

Watchers

34 watching

Forks

Releases

Packages

Used by

Contributors

Languages