Live Stream Troubleshooting Runbook for AV Teams
More live video workflow notes and product updates.
Live-stream incident runbook
Find the first broken boundary before changing settings
A live-stream problem becomes expensive when several people change unrelated controls at once. The useful question is rarely “why is the stream broken?” It is “what is the last point where we can still prove that picture, sound, timing, and transport were healthy?” Start there, move one boundary downstream, and preserve the evidence.
The five-minute triage rule
Freeze nonessential configuration changes. Confirm the audience symptom from an independent device, note the time, and divide the path into source, encode, contribution, processing, and delivery. Assign one operator to each active boundary and one incident lead to record findings. Change a single variable only after you have a hypothesis and a way to tell whether the change helped.
These animated bars illustrate correlation, not live telemetry. A real diagnosis compares time-aligned evidence from the source, encoder, transport, platform, and audience.
Build an evidence line from camera to viewer
A green connection badge proves only that one session exists. It does not prove the camera is in focus, that audio channels are mapped correctly, that timestamps are monotonic, or that the public player receives a usable rendition. For every important event, decide how each boundary will be observed before the event starts.
| Boundary | Useful evidence | Misleading shortcut |
|---|---|---|
| Camera and audio | Local confidence monitor, waveform or meters, focus, exposure, source standard, channel identification. | Assuming a connected cable means the intended source is present. |
| Encoder | Input lock, actual output bitrate, dropped or late frames, CPU/GPU headroom, timestamps, codec profile. | Reading only the configured bitrate or preset name. |
| Contribution | Session identity, packet loss or retransmission signals where available, RTT trend, received bitrate, reconnects. | Running a generic speed test after the incident and treating it as path history. |
| Processing | Correct source selected, decoded media, transcoder health, route state, recording state, active/backup identity. | Restarting the whole platform before finding which module lost the signal. |
| Delivery and player | Destination health, manifest and segments, rendition selection, playback on a separate network, audience reports by region/device. | Watching only the local production return. |
When the stream is offline
- Confirm the scope. Is every viewer offline, one destination offline, one rendition missing, or one operator preview black? Capture the exact URL, destination, device, region, and timestamp.
- Check the source locally. Verify the physical or software source before touching network settings. A camera battery, capture-card input, muted program bus, or stopped scene can look like an ingest failure downstream.
- Inspect encoder output. Confirm that it is running, encoding the expected input, and sending to the intended address. Look for resource saturation and output-frame counters, not just the “streaming” state.
- Inspect the first receiver. Confirm whether a session arrived and whether bytes represent valid moving media. For SRT, compare role, address, UDP port, identity, passphrase, latency contract, and firewall path. For RTMP, recheck application and stream-key ownership.
- Follow the selected route. If ingest is healthy, inspect the exact restream, transcoder, failover route, or player feeding the affected destination. Avoid restarting healthy branches.
- Verify recovery from the audience side. A recovered encoder preview is not enough. Confirm the destination and a real playback device, then record the interval affected and the action that restored service.
When video freezes, blocks, or drops frames
First separate capture defects from encoder overload and network loss. If the local source monitor already freezes, the contribution network is not the first suspect. If the source is clean but encoded-frame counters stall under detailed motion, inspect encoder capacity and profile complexity. If encoded output is healthy while the receiver reports changing transport conditions, investigate the path.
Do not tune from a single speed test
An upload test measures one service over a short interval. It does not reproduce the route, packet-loss pattern, jitter, or sustained traffic of the production. Use it as one observation, then run a representative stream to the real receiver and watch it through motion and network variation.
Leave operating headroom
A bitrate that briefly fits an uplink may still fail when capacity changes or protocol overhead is considered. Select a profile from measured conditions and the destination's accepted range. If two independent paths are required, qualify each path rather than assuming their headline rates can be added.
Correlate bitrate with picture complexity
A talking head, a noisy low-light sensor, confetti, and a fast camera pan stress an encoder differently. Rehearse with the hardest scene the event is likely to produce. A stable slate is useful for identity but poor evidence of sustained encode behavior.
Separate viewer buffering from contribution loss
If the server receives clean media but one audience cohort buffers, examine manifests, segments, CDN delivery, rendition availability, device decode support, and the viewer network. Changing the venue's SRT latency cannot repair a missing delivery rendition.
When audio is wrong but video looks healthy
Audio failures are often mapping or timing problems rather than total transport failures. Identify channels with a safe spoken or tone sequence at the source. Follow the same channels through the encoder, contribution decoder, mixer, transcoder, recording, and public output. Write down whether the fault is absence, distortion, level, channel swap, intermittent loss, or sync drift; those symptoms point to different boundaries.
- No audio anywhere downstream: inspect source selection, mute state, embedded-audio mapping, encoder input, and the codec carried by the output.
- Audio on one output only: compare the exact processing and destination profiles. Do not disturb the healthy branch.
- Constant lip-sync offset: measure the offset with identifiable picture and sound, then correct it once at the responsible processing boundary.
- Offset that grows: investigate clocks, sample-rate conversion, timestamps, frame-rate interpretation, and long-duration processing. A fixed delay control will not cure continuing drift.
- Pops or clipping: inspect gain structure, overload, clocking, and codec transitions. A limiter can contain peaks but cannot recover already clipped input.
When the stream is online but looks poor
Do not use bitrate as a universal quality knob. Check the source first: focus, shutter, exposure, noise, scaling, deinterlacing, and graphics can determine the result before compression begins. Then compare the encoder's actual codec, resolution, frame rate, rate-control behavior, and keyframe policy with the destination's current requirements.
A hardware encoder is not automatically better than a software encoder, and a software encoder is not automatically more flexible under load. The useful comparison is the exact device or host, firmware or software version, input format, codec implementation, target profile, and recovery behavior. Run a reproducible test rather than changing platform category during an incident.
Use recovery without hiding the root cause
Automatic failover can protect the audience while engineers diagnose the primary path. It should also make the active source obvious and preserve a record of why the switch occurred. Rehearse loss of the primary feed, confirm the backup is genuinely independent, and define how the operation returns to primary. An automatic switch that nobody notices until the backup fails is not observability.
Callaba can bring SRT or RTMP inputs into a common operating surface, show them in Multiview, route or restream them, attach recording, and configure live-video failover. Use those controls to expose the boundary being tested. The product does not remove the need to validate cameras, encoders, destination services, or external networks.
Prepare the runbook before call time
- Name every primary and backup source consistently across the encoder, receiver, Multiview, routes, recordings, and operator notes.
- Record approved profiles and protect them from casual event-day edits.
- List destination owners, credential owners, escalation contacts, and the exact evidence each person can inspect.
- Synchronize logs and operator clocks well enough to correlate an audience report with transport and module events.
- Run controlled disconnects for source, network, receiver, processing, and destination. Document expected recovery and the safe manual action.
- After the event, preserve a short sample, incident timestamps, and the actual configuration. Replace vague lessons with one testable change for the next rehearsal.
Give the operator one place to see the live boundary
Build a small representative workflow, label the inputs by production role, and rehearse a failure before scaling it. A useful control surface makes the first missing signal easier to find without pretending every external system lives inside it.