Skip to content

feat(webrtc): allow switching to an audio-only rendition in a video session - #2212

Open
naanlizard wants to merge 1 commit into
OvenMediaLabs:masterfrom
naanlizard:feature/webrtc-audio-only-rendition
Open

feat(webrtc): allow switching to an audio-only rendition in a video session#2212
naanlizard wants to merge 1 commit into
OvenMediaLabs:masterfrom
naanlizard:feature/webrtc-audio-only-rendition

Conversation

@naanlizard

@naanlizard naanlizard commented Jun 19, 2026

Copy link
Copy Markdown
Contributor

One more PR for now, again full disclosure all coded by claude, this is a much smaller change though.

The very short version is that this lets us downgrade background-tab WebRTC streams to audio only, saving a bunch of bandwidth

Problem

RtcMasterPlaylist buckets renditions by (video_codec_id, audio_codec_id), so an
audio-only rendition (video codec None) lands in its own (None, audio) playlist that
a session which negotiated video never sees. A WebRTC video session therefore cannot
switch to an audio-only rendition on the same PeerConnection — which is useful for
keeping audio while dropping all video traffic without renegotiating, for example when a
player is backgrounded or hidden.

Change

A new playlist option WebRtcAudioOnlyFallback (off by default) enables the behaviour.
When it is set:

  • Each audio-only rendition is cross-added into every video playlist that shares its
    audio codec, and back-filled into video playlists created after the audio-only
    rendition is first seen (so config declaration order does not matter).
  • The triggering video rendition is added before any cross-added audio-only rendition, so
    it stays the playlist default (GetFirstRendition) — a session always starts on video.

Because the option defaults to off, existing configurations are unaffected.

ABR semantics

Within a video playlist, automatic bitrate ABR never moves a session onto, or off of, the
audio-only rendition:

  • GetNextHigherBitrateRendition / GetNextLowerBitrateRendition skip audio-only
    candidates, so congestion never drops a video session down to audio-only.
  • They also pin a session whose current rendition is audio-only (return nullptr in both
    directions), so a session that switched to audio-only is not auto-resumed to video even
    if the client left auto-ABR enabled.

So the audio-only rendition is reachable only by an explicit change_rendition, and the
session stays there until the client switches back. Both behaviours are gated on the
playlist being a video playlist, so an audio-only playlist still performs normal bitrate
ABR among its own renditions.

No SDP or renegotiation changes are needed: rtc_session already supports switching to a
video-less rendition on an established PeerConnection, and the audio-only rendition is
advertised in the playlist notification so a client can request it by name.

Config example

<Playlist>
    <Name>webrtc</Name>
    <FileName>webrtc</FileName>
    <Options>
        <WebRtcAutoAbr>true</WebRtcAutoAbr>
        <WebRtcAudioOnlyFallback>true</WebRtcAudioOnlyFallback>
    </Options>
    <Rendition>
        <Name>720p</Name>
        <Video>720p</Video>
        <Audio>opus</Audio>
    </Rendition>
    <Rendition>
        <Name>audio</Name>
        <Audio>opus</Audio>
    </Rendition>
</Playlist>

The video and audio-only renditions must share the same audio codec for the cross-add to
apply. The option is documented in docs/streaming/webrtc-publishing.md.

Testing

Adds gtest coverage in src/projects/publishers/webrtc/webrtc_test.cpp for
RtcMasterPlaylist / RtcPlaylist: cross-add and back-fill ordering, the
video-rendition-stays-default invariant, ABR exclusion of audio-only in a video playlist
(including an audio-only rendition whose bitrate sits between two video renditions, and the
pinning of an explicitly-selected audio-only rendition against auto-ABR), normal bitrate
ABR within an audio-only playlist, codec-mismatch and multiple-video-playlist cases,
video-only renditions with no audio track, and the option being disabled by default.

@naanlizard
naanlizard requested a review from a team as a code owner June 19, 2026 16:02
@naanlizard
naanlizard requested review from dimiden and removed request for a team June 19, 2026 16:02
…ession

RtcMasterPlaylist buckets renditions by (video_codec_id, audio_codec_id),
so an audio-only rendition (video codec None) lands in its own playlist that
a session which negotiated video never sees. A video session therefore cannot
switch to an audio-only rendition on the same PeerConnection.

When the WebRtcAudioOnlyFallback playlist option is enabled, cross-add each
audio-only rendition into every video playlist that shares its audio codec, and
back-fill video playlists created after the audio-only rendition is seen. The
triggering video rendition is added before any cross-added audio-only rendition,
so it remains the playlist's default rendition (GetFirstRendition). The option
is off by default, so existing configurations are unaffected.

Within a video playlist, automatic bitrate ABR never moves a session onto, or off
of, an audio-only rendition: GetNext{Higher,Lower}BitrateRendition skip audio-only
candidates so congestion never drops a video session to audio-only, and they pin a
session whose current rendition is audio-only so it is not auto-resumed to video.
Audio-only is therefore reachable only by an explicit change_rendition request and
the session stays there until the client switches back. Both behaviours are gated
on the playlist being a video playlist, so an audio-only playlist still performs
normal bitrate ABR among its own renditions.

Audio-only renditions are advertised in the playlist notification so a client
can request them by name. rtc_session already supports switching to a video-less
rendition without renegotiation.

Add unit tests for RtcMasterPlaylist/RtcPlaylist covering cross-add and back-fill
ordering, the video-rendition-stays-default invariant, ABR exclusion in a video
playlist (including an audio-only rendition placed between two video bitrates and
the pinning of an explicitly-selected audio-only rendition against auto-ABR), ABR
among renditions in an audio-only playlist, codec-mismatch and multiple-video
-playlist cases, video-only renditions with no audio track, and the option being
disabled by default. Document the option in the WebRTC publishing guide.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@naanlizard
naanlizard force-pushed the feature/webrtc-audio-only-rendition branch from 0b07aec to 368fb28 Compare June 19, 2026 21:06
@dimiden
dimiden requested a review from Copilot July 13, 2026 13:05

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR enables a WebRTC video session to explicitly switch to an audio-only rendition (same audio codec) without renegotiation by making audio-only renditions visible inside matching video playlists when a new config option is enabled.

Changes:

  • Add WebRtcAudioOnlyFallback playlist option (default off) and plumb it from config → info::PlaylistRtcStreamRtcMasterPlaylist.
  • Update RtcMasterPlaylist to cross-add/back-fill audio-only renditions into matching video playlists, and update RtcPlaylist ABR to never auto-select (or auto-exit) audio-only within video playlists.
  • Add gtest coverage for cross-add/back-fill ordering and ABR semantics; document the new option.

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
src/projects/publishers/webrtc/webrtc_test.cpp Adds gtests validating cross-add/back-fill behavior and ABR exclusion/pinning semantics.
src/projects/publishers/webrtc/rtc_stream.cpp Plumbs the new playlist option into RtcMasterPlaylist creation.
src/projects/publishers/webrtc/rtc_playlist.h Implements audio-only cross-add/back-fill and ABR guardrails for video playlists.
src/projects/config/items/virtual_hosts/applications/output_profiles/playlist/playlist.h Copies the parsed option into info::Playlist.
src/projects/config/items/virtual_hosts/applications/output_profiles/playlist/options.h Adds the WebRtcAudioOnlyFallback config option and default.
src/projects/base/info/playlist.h Stores, prints, compares, and exposes the new option on info::Playlist.
docs/streaming/webrtc-publishing.md Documents the new option and provides an XML example.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.


If `<Options>/<WebRtcAutoAbr>` is set to true, OvenMediaEngine will measure the bandwidth of the player session and automatically switch to the appropriate rendition.

If `<Options>/<WebRtcAudioOnlyFallback>` is set to true, OvenMediaEngine cross-adds any audio-only rendition of the playlist (a `<Rendition>` with no `<Video>`) into the video renditions that share its audio codec. A WebRTC player can then switch to the audio-only rendition on the same PeerConnection without renegotiation — for example to keep audio while dropping video bandwidth when the player is in the background. The option is off by default. Automatic ABR never switches to the audio-only rendition; it is reachable only by an explicit rendition change, and a session that has switched to it stays there until the player switches back.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants