Fun with ffmpeg filter and transparency

[toots]

Note: This post requires the latest liquidsoap main and ocaml-ffmpeg master branch at time of writting (Aug. 28, 2021)

Had some fun with ffmpeg, video and transparency after a request from a user! Here's the gist of it:

Static image, audio and transparent overlay

The goal of the script is to:

• Take a static background
• Add a playlist of audio files
• Overlay a transparent logo to the video
• Stream it to youtube

We achieve this using ffmpeg filters!

1. Static logo image

The first version of our script will use static images for both background and logo. First, we need our basic sources:

# Basic sources:
background = single("/path/to/background.jpg")
playlist = playlist("/path/to/playlist")
logo = single("/path/to/logo.png")

Next, we write the ffmpeg filter used to overlay the logo on top of the background:

def logo_overlay(background, logo) =
  def mkfilter(graph) =
    background  = ffmpeg.filter.video.input(graph, background)
    logo = ffmpeg.filter.video.input(graph, logo)

    v = ffmpeg.filter.overlay(graph, background, logo,
      y="50",
      x="25"
    )

    ffmpeg.filter.video.output(graph, v)
  end

  ffmpeg.filter.create(mkfilter)
end

And finally, we put it all together:

video = logo_overlay(background, logo)

s = mux_video(video=video, playlist)

url = "rtmp://a.rtmp.youtube.com/live2/xxxxxxxx"

encoder = %ffmpeg(
  format="flv",
  %audio(codec="aac", b="320k", samplerate=48000, q=4),
  %video.raw(codec="libx264",width=1280, height=720,b="2500k", preset="veryfast")
)

output.url(
  fallible=true,
  url=url,
  encoder,
  s
)

The one thing to note is the use of %video.raw for the video encoder. This makes sure that we keep the video frame as ffmpeg frame all along.

The source is decoded as ffmpeg frames, passed to the filter as such and then to the encoder. This way, we are never converting the frame to the internal liquidsoap video frame data. This is good for two reasons:

• This avoids needless data copy
• Images transparency like that does not seem to work as seamlessly with our internal data format at the moment. This is, of course, something that we plan on supporting next.

Time to see the script in action! The whole script:

Full script

def logo_overlay(background, logo) =
  def mkfilter(graph) =
    background  = ffmpeg.filter.video.input(graph, background)
    logo = ffmpeg.filter.video.input(graph, logo)

    v = ffmpeg.filter.overlay(graph, background, logo,
      y="50",
      x="25"
    )

    ffmpeg.filter.video.output(graph, v)
  end

  ffmpeg.filter.create(mkfilter)
end

background = single("/path/to/background.jpg")
playlist = playlist("/path/to/playlist")
logo = single("/path/to/logo.png")

video = logo_overlay(background, logo)

s = mux_video(video=video, playlist)

url = "rtmp://a.rtmp.youtube.com/live2/xxxxxxxx"

encoder = %ffmpeg(
  format="flv",
  %audio(codec="aac", b="320k", samplerate=48000, q=4),
  %video.raw(codec="libx264",width=1280, height=720,b="2500k", preset="veryfast")
)

output.url(
  fallible=true,
  url=url,
  encoder,
  s
)

2. Dynamic transparent video

Next, we're adding a transparent video! This is pretty much the same, except:

• We need to use a vp9 video since this is one of the only codec that supports transparent video..
• Support for decoding transparency with vp9 in ffmpeg is still very new and a bit rough. In particular, we need to force the use of google's libvpx-vp9 vp9 decoder instead of the internal one.

This is easily achieved, though, by adding this to the top of our script:

settings.decoder.ffmpeg.codecs.vp9.set("libvpx-vp9")

Next, we wil scale the logo video to the size we want it to be. Here's our updated ffmpeg filter:

def logo_overlay(background, logo) =
  def mkfilter(graph) =
    background = ffmpeg.filter.video.input(graph, background)
    logo = ffmpeg.filter.video.input(graph, logo)

    logo = ffmpeg.filter.scale(graph, logo,
      w="-1",
      h="75")

    v = ffmpeg.filter.overlay(graph, background, logo,
      y="50",
      x="25"
    )

    ffmpeg.filter.video.output(graph, v)
  end

  ffmpeg.filter.create(mkfilter)
end

Then we only need to change logo.png to logo.webm, our transparent vp9 video. Here we go:

Full script

settings.decoder.ffmpeg.codecs.vp9.set("libvpx-vp9")

def logo_overlay(background, logo) =
  def mkfilter(graph) =
    background  = ffmpeg.filter.video.input(graph, background)
    logo = ffmpeg.filter.video.input(graph, logo)
    
    logo = ffmpeg.filter.scale(graph, logo,
      w="-1",
      h="75")

    v = ffmpeg.filter.overlay(graph, background, logo,
      y="50",
      x="25"
    )

    ffmpeg.filter.video.output(graph, v)
  end

  ffmpeg.filter.create(mkfilter)
end

background = single("/path/to/background.jpg")
playlist = playlist("/path/to/playlist")
logo = single("/path/to/logo.webm")

video = logo_overlay(background, logo)

s = mux_video(video=video, playlist)

url = "rtmp://a.rtmp.youtube.com/live2/xxxxxxxx"

encoder = %ffmpeg(
  format="flv",
  %audio(codec="aac", b="320k", samplerate=48000, q=4),
  %video.raw(codec="libx264",width=1280, height=720,b="2500k", preset="veryfast")
)

output.url(
  fallible=true,
  url=url,
  encoder,
  s
)

And that's it! I'm attaching the script and files to this discussion for anyone to reproduce locally!

transparent.zip

[smimram]

This is great work, thanks! Any chance that you could add pre-written functions in the standard library? Something like video.add_logo.ffmeg?

[realYogiFish]

This is a great example to see. When using larger sized transparent videos, CPU usage creates a lot of buffering/ Is there a way to reduce the CPU usage, or dedicate processes to individual cores for processing?

Also, when you have time, can you show an example for using a Video playlist, with embeded audio, as the background source.

Thanks -

[toots]

Hi @realYogiFish ! Do you have a specific script/setup we could look at? In a situation like the one described here, where everything is in ffmpeg.raw format end-to-end, we're really looking to be on-part with ffmpeg's performances and so, if not, we'd be happy to look at it, for sure.

[realYogiFish]

Hey @toots I have a few scripts, but they're not how I'd want to recreate it in Liquidsoap. Sort of a work around, where I make a composite from the playlist with the station logo, then that output feeds ,as a live stream, into another script to composite with the animation.

I'm still getting up to speed with learning Liquidsoap, so I'm not sure how to best explain for comparison. Keeping everything in (raw) is the challenge.

This is the first part script, but I can't find the second receiving script.. but it's pretty much the same.. with the source being a stream in, with the animation replacing the logo image.

ffmpeg \ -loglevel -8 -stats \ -re \ -stream_loop -1 \ -f concat -safe 0 -segment_time_metadata 1 \ -i /var/www/html/video_playlists/main.pl \ -i /var/www/html/video_images/logo.png \ -filter_complex \ "[1:v] format=rgba, colorchannelmixer=aa=0.30 [logo];[0] [logo] overlay=W-w-40:H-h-40:format=auto,format=yuv420p,split=2[out1][out2]; \ [out1]scale=1920:-2[v0]; \ [out2]scale=640:-2[v1]" \ -map "[v0]" -c:v libx264 -b:v:0 5M -maxrate:v:0 5M -minrate:v:0 5M -bufsize:v:0 10M -preset:v faster -profile:v baseline -g 48 -r 24 -sc_threshold 0 -keyint_min 48 \ -map "[v1]" -c:v libx264 -b:v:0 3M -maxrate:v:0 3M -minrate:v:0 3M -bufsize:v:0 3M -preset:v faster -profile:v baseline -g 48 -r 24 -sc_threshold 0 -keyint_min 48 \ -map 0:a -c:a copy -strict -2 -flvflags no_buffer,no_sequence_end,no_duration_filesize -flags +global_header+low_delay -f tee \ "[select=\'v:0,a\':f=flv]rtmp://xxx.xxx.xx.xxx:1935/hd/broadcast| \ [select=\'v:1,a\':f=flv]rtmp://xxx.xxx.xx.xxx:1935/sd/broadcast"

[toots]

I believe this should be a translation of your filter:

def complex_filter(background, logo) =
  def mkfilter(graph) =
    background = ffmpeg.filter.video.input(graph, background)
    logo = ffmpeg.filter.video.input(graph, logo)

    logo = ffmpeg.filter.format(pix_fmts="rgba", graph, logo)
    logo = ffmpeg.filter.colorchannelmixer(aa=0.30, graph, logo)

    video = ffmpeg.filter.overlay(graph, background, logo,
      y="W-w-40",
      x="H-h-40"
    )

    video = ffmpeg.filter.format(pix_fmts="yuv420p", graph, video)

    let (_, video) = ffmpeg.filter.split(outputs=2, graph, video)
    let [out1, out2] = video

    out1 = ffmpeg.filter.scale(graph, out1,
      width="1920", height="-1"
    )

    out2 = ffmpeg.filter.scale(graph, out2,
      width="640", height="-1"
    )

    (ffmpeg.filter.video.output(graph, out1),
     ffmpeg.filter.video.output(graph, out2))
  end

  ffmpeg.filter.create(mkfilter)
end

You can use it as follows:

let (v1, v2) = complex_filter(logo, background)

[realYogiFish]

Wow, Thanks. I'll work on giving that a run through. Things are coming together nicely. Liquidsoap will be perfect for streaming video for me.

[toots]

Thanks! Yes, we are very much looking for examples like that to finer tweak this new support so we appreciate the tryout as well!