Building a live HLS packager in Go

At work, I'm one of the two core backend engineers on a video streaming platform. It handles uploads, video on demand, premieres, live streams and live chat.

Video on demand was already solid. Every upload goes through an FFmpeg HLS ladder, which encodes the video at several quality levels. The output lands in S3 and is served through CloudFront with signed URLs. When we added live streaming, the goal was obvious: live video should travel the same road. Same bucket, same CDN, same signed-URL logic, same player.

This post is about the piece that makes that possible, a process we call the packager. My first design didn't work at all. The second worked on a laptop, then failed in a handful of interesting ways once I tried to break it. Each failure taught me something about media, processes or concurrency in Go.

The shape of the system

Creators stream from OBS or from the browser. A teammate built the first version of the browser path. Both land on MediaMTX, an open-source media server that accepts RTMP, SRT, RTSP and WebRTC publishes. Those are just different protocols for sending a live stream. MediaMTX is where streams come in. It's not where viewers watch.

The packager is a separate process, started with worker --packager. Its job:

• Ask MediaMTX, on a timer, which streams are live right now.
• For each live stream, run one session that turns it into plain HLS: a playlist plus short video segments.
• Upload those files to S3 under the broadcast's id, so viewers hit the same CDN path as on-demand video.
• Own the database flag is_live, which the rest of the product reads.

HLS is simple once you see it. The video is cut into small files called segments. A text playlist (master.m3u8) lists the most recent few, and the player keeps re-fetching the playlist to find new ones. Segments never change once written, but the playlist changes every few seconds. That difference matters later.

Attempt one: copy what MediaMTX already serves

MediaMTX can serve HLS itself. So my first packager just fetched it over HTTP and copied the files to S3:

// FetchHLS GETs a file from the MediaMTX HLS origin for the given stream key
func (c *MediaMTXClient) FetchHLS(ctx context.Context, streamKey, file string) ([]byte, error) {
	url := c.hlsBase + "/" + streamKey + "/" + file
	// ...
	if resp.StatusCode != http.StatusOK {
		return nil, fmt.Errorf("fetch %s: status %d", file, resp.StatusCode)
	}
	return io.ReadAll(resp.Body)
}

When I tested it end to end locally, nothing usable ever showed up in S3. That's because MediaMTX's HLS output is session-based. A request goes through a 302 cookie handshake and gets a master playlist. That playlist points at variant URLs that carry a short-lived ?session= token. There's no stable playlist sitting at a fixed path for you to copy. The endpoint is built for a player talking to it directly, not for a second server copying it.

Attempt two: pull the stream and remux it

So instead of copying someone else's HLS, the packager now pulls the raw stream over internal RTMP. Then it lets ffmpeg write HLS to a temp directory:

cmd := exec.CommandContext(ctx, "ffmpeg",
	"-nostdin", "-loglevel", "warning", "-rw_timeout", "15000000",
	"-i", input, // rtmp://mediamtx:1935/<stream_key>
	"-c", "copy",
	"-f", "hls",
	"-hls_time", "4",
	"-hls_list_size", "6",
	"-hls_flags", "delete_segments+append_list+independent_segments",
	"-hls_segment_filename", filepath.Join(dir, "seg_%05d.ts"),
	master,
)

The key flag is -c copy. OBS already sends H.264 video and AAC audio. That's exactly what HLS with MPEG-TS segments expects. So ffmpeg only has to remux: it rewraps the same compressed data in a new container, without decoding anything. That's cheap enough to run one per stream.

An upload loop then pushes new files to S3. Two details made the output safe to serve:

// uploadOnce pushes any not-yet-uploaded TS segments, then the master
// playlist (last, so it never references a segment that isn't in S3 yet).
if err := p.putFile(ctx, dir, name, id, segmentContentType, "public, max-age=31536000"); err != nil {
	// ...
}
if err := p.putFile(ctx, dir, masterPlaylist, id, playlistContentType, "no-cache"); err != nil {
	// ...
}

• Order: segments first, playlist last. If the playlist went up first, a player could read it and ask for a segment that isn't there yet.
• Cache headers: segments never change, so the CDN can keep them for a year. The playlist changes every few seconds, so it's no-cache.

I also made the copy function swappable, so I could unit-test the session lifecycle without starting ffmpeg. The commit records an end-to-end check against MediaMTX and MinIO. It covered publish auth, is_live turning on, the ffmpeg remux, HLS landing in S3, playable output (ffprobe reported h264 and aac), stopping, and is_live being cleared.

Hardening: six bugs in one session loop

Working on a laptop isn't the same as working. So I went looking for ways it could fail: dropped publishers, crashes, slow databases, long streams. Almost every one of them broke something.

ffmpeg exited and nobody noticed

When a publisher dropped, the session was supposed to end. The loop checked whether ffmpeg had exited like this:

case <-ticker.C:
	// If ffmpeg died (publisher dropped, etc.) stop the loop.
	if cmd.ProcessState != nil {
		return nil
	}

That condition can never be true. In Go, exec.Cmd.ProcessState is only filled in after Wait() returns. And nothing was calling Wait() while the loop ran. So a dropped publisher left behind a zombie upload loop and a stream stuck on is_live.

The fix is to watch the process in its own goroutine and signal its exit with a channel:

// Supervise ffmpeg in a goroutine: cmd.ProcessState is only populated
// after Wait() returns, so polling it inline never detects exit.
exited := make(chan struct{})
var waitErr error
go func() { waitErr = cmd.Wait(); close(exited) }()

Now the select loop can react to <-exited like any other event.

The end of every stream was missing

The old shutdown path uploaded the last segments using the session's context:

case <-ctx.Done():
	_ = cmd.Wait()
	p.uploadOnce(ctx, dir, id, uploaded) // final flush (best-effort)
	return nil

But ctx.Done() firing is exactly why we're in this branch. The context is already cancelled. So every S3 PUT inside uploadOnce returned right away, and the last few seconds of every stream never reached S3. The new version does the final upload with its own fresh context:

case <-ctx.Done():
	// ctx is already dead, so reusing it would no-op every S3 put
	// and lose the tail of the stream.
	<-exited
	p.flushFinal(dir, id, uploaded) // fresh context.WithTimeout(Background, 10s)
	return nil
case <-exited:
	p.flushFinal(dir, id, uploaded)
	return nil

One slow call froze every stream

A reconcile pass compares MediaMTX's list of live streams with the sessions that are running. It held the packager's mutex for the whole pass. That included a database round trip (MarkLive) and waiting for ffmpeg to drain (session.stop()):

p.mu.Lock(); defer p.mu.Unlock()
for k := range live { if _, running := p.sessions[k]; !running { p.startSession(ctx, k) } } // DB call under lock
for k, s := range p.sessions { if _, stillLive := live[k]; !stillLive { s.stop(); delete(p.sessions, k) } } // ffmpeg drain under lock

So one slow database call, or one slow drain, held up every other stream. Now the lock only protects the map. You decide under the lock and act outside it.

p.mu.Lock()
// ... collect toStart / toStop from the session map ...
p.mu.Unlock()
for _, k := range toStart { p.startSession(ctx, k) }
for _, s := range toStop { s.stop() }

stopAll also stops sessions in parallel, and stop() only waits so long. I wrote the timeouts down together, because they depend on each other. The drain (15s) must last longer than the final upload (10s), and MarkEnded gets 5s.

Small numbers that break long streams

Two problems only show up on streams that run for days:

• Segment names used %05d. With 4-second segments, that runs out after about 4.6 days, and names start to collide on a 24/7 stream. %09d lasts about 127 years.
• The set of already-uploaded segment names grew forever. Now it's trimmed to match what's actually on disk, since ffmpeg deletes old segments itself (delete_segments).

LIVE forever after a crash

If the packager crashed, the broadcasts it was handling stayed is_live in the database. No process was left to clear them. Now it runs a recovery pass on startup. Any stream the database says is live, but MediaMTX doesn't list, gets ended.

Two writers, and a reconnect that was refused

Two things wrote is_live: MediaMTX's ready/not-ready webhooks and the packager. Two writers on one flag will disagree sooner or later. I removed the hooks. Now the packager is the only owner of is_live.

The publish-auth check also refused a publish when is_live was already true. The idea was to block duplicate publishers. But when OBS reconnects after a network blip, the flag is still true. So the real creator got locked out of their own stream. MediaMTX already allows only one publisher per path, so the extra check only caused harm:

// NOTE: we deliberately do NOT reject when row.IsLive is already true.
// MediaMTX enforces single-publisher-per-path natively ... Gating here instead would
// lock out the legitimate publisher on reconnect

Browser streams with no sound

Once browser publishing (WebRTC, via WHIP) was in, we got a report: video played, but there was no audio. OBS streams were fine.

The cause is a container rule. A container is the file format that wraps the audio and video. RTMP and MPEG-TS HLS can carry AAC audio, but not Opus. Browsers send WebRTC audio as Opus, and video as H.264 or VP8. On the -c copy path, ffmpeg can't put Opus into the output container, so it quietly drops the audio track. No error, just silence. VP8 video has a similar problem: it can't travel over RTMP at all.

The first version of the fix treated the whole stream as either "copy" or "transcode" (decode and re-encode):

if transcode {
	input = strings.TrimRight(p.rtspBase, "/") + "/" + streamKey
	args = append(args,
		"-rtsp_transport", "tcp", "-i", input,
		"-c:v", "libx264", "-preset", "veryfast", "-tune", "zerolatency",
		"-pix_fmt", "yuv420p", "-g", "60",
		"-c:a", "aac", "-ar", "44100",
	)
} else {
	input = strings.TrimRight(p.rtmpBase, "/") + "/" + streamKey
	args = append(args, "-rw_timeout", "15000000", "-i", input, "-c", "copy")
}

That works, but it re-encodes the video for a stream whose only problem is the audio. The final version decides per track, based on the codecs MediaMTX reports. If any track needs re-encoding, the packager pulls over RTSP instead, because RTSP can carry Opus and VP8. Then it transcodes only that track:

// RTMP can only carry H.264/AAC. Any track that needs re-encoding (VP8
// video, or the common browser-WebRTC case, Opus audio) means we must
// pull over RTSP, which carries those codecs, and transcode just that track.
needRTSP := transcodeVideo || transcodeAudio
// ...
if transcodeVideo {
	args = append(args, "-c:v", "libx264", "-preset", "veryfast", "-tune", "zerolatency", "-pix_fmt", "yuv420p", "-g", "60")
} else {
	args = append(args, "-c:v", "copy")
}
if transcodeAudio {
	args = append(args, "-c:a", "aac", "-ar", "44100")
} else {
	args = append(args, "-c:a", "copy")
}

Now a webcam sending H.264 plus Opus only pays for a cheap audio encode. The commit records a VP8/Opus RTSP publish being detected, transcoded and served as h264/aac HLS.

A cap that refuses isn't a cap that slows down

Video transcoding is expensive. My commit message guessed it at roughly 20 to 100 times the CPU of a copy. That's an estimate, not a benchmark. So video transcodes sit behind a limit on how many can run at once. It was hard-coded to 2.

Think about what kind of limit that is. When it's full, it doesn't slow streams down. It refuses them. And a slot is held for the whole broadcast, not a few seconds. So with two VP8 broadcasts running, a third creator goes live and just never shows up for viewers. That's much worse than a slow page.

Now the default scales with the machine, and an environment variable can override it:

func defaultTranscodeCap() int {
	if n := runtime.NumCPU() / 2; n > 2 {
		return n
	}
	return 2
}

What I took away

• Copying a media server's player-facing HLS is fragile. Pull the stream and package it yourself.
• Upload segments before the playlist that points to them. Give files that never change and files that do different cache headers.
• exec.Cmd.ProcessState is nil until Wait() returns. Watch processes with a goroutine and a channel.
• Use a fresh context for cleanup and final uploads.
• Never hold a mutex while doing I/O. Take a snapshot under the lock, then act outside it.
• Size counters and sets for streams that run for weeks.
• Give each piece of state exactly one writer. Don't stack a stale check on top of a rule another system already enforces.
• -c copy quietly drops tracks the container can't carry, so decide per track.
• Know whether your limits refuse work or slow it down, size them to the hardware, and write down what happens when they're hit.