output.url(url="srt://…", input.rtmp("…")): Freezes after two minutes

[copy]

Describe the bug

I'm trying to stream a camera which has rtmp output to srt. The script below freezes after about two minutes after printing We must catchup … seconds many times. Afterwards liquidsoap doesn't terminate cleanly either.

To Reproduce

stream = mksafe(input.rtmp("rtmp://0.0.0.0:53163/"))
url = "srt://server:53162/output"
enc = %ffmpeg(
        format="flv",
        %audio(codec="aac", ar="44100"),
        %video(codec="libx264", preset="veryfast", r="30", g="60", keyint_min="30", b="1500k", minrate="1500k", maxrate="1500k"))
output.url(url=url, enc, stream)

Expected behavior

The stream should not freeze

Version details

Liquidsoap 2.0.2 (see the log for details)

Install method

opam

Logs

2022/01/26 18:18:15 >>> LOG START
2022/01/26 18:18:14 [main:3] Liquidsoap 2.0.2
2022/01/26 18:18:14 [main:3] Using: bytes=[distributed with OCaml 4.02 or above] pcre=7.5.0 sedlex=2.5 menhirLib=20211128 dtools=0.4.4 duppy=0.9.2 mm=0.7.2 dynlink=[distributed with Ocaml] ffmpeg=1.1.1 alsa=0.3.0 camomile=1.0.2 pulseaudio=0.1.4 tsdl=v0.9.8 tsdl-image=0.3.2 srt.constants=0.2.1 srt.types=0.2.1 srt.stubs=0.2.1 srt.stubs.locked=0.2.1 srt=0.2.1
2022/01/26 18:18:14 [dynamic.loader:3] Could not find dynamic module for lame encoder.
2022/01/26 18:18:14 [dynamic.loader:3] Could not find dynamic module for fdkaac encoder.
2022/01/26 18:18:14 [clock:3] Using builtin (low-precision) implementation for latency control
2022/01/26 18:18:15 [frame:3] Using 44100Hz audio, 25Hz video, 44100Hz main.
2022/01/26 18:18:15 [frame:3] Video frame size set to: 1280x720
2022/01/26 18:18:15 [frame:3] Frame size must be a multiple of 1764 ticks = 1764 audio samples = 1 video samples.
2022/01/26 18:18:15 [frame:3] Targeting 'frame.duration': 0.04s = 1764 audio samples = 1764 ticks.
2022/01/26 18:18:15 [frame:3] Frames last 0.04s = 1764 audio samples = 1 video samples = 1764 ticks.
2022/01/26 18:18:15 [sandbox:3] Sandboxing disabled
2022/01/26 18:18:15 [video.converter:3] Using preferred video converter: ffmpeg.
2022/01/26 18:18:15 [audio.converter:3] Using samplerate converter: ffmpeg.
[srt @ 0x7f0f58005a80] Connection to srt://localhost:53162/output failed (Input/output error), trying next address
[libx264 @ 0x7f0f58063f00] VBV maxrate specified, but no bufsize, ignored
2022/01/26 18:18:18 [clock.main:3] Streaming loop starts in auto-sync mode
2022/01/26 18:18:18 [clock.main:3] Delegating synchronisation to CPU clock
2022/01/26 18:18:18 [mksafe:3] Switch to safe_blank.
[rtmp @ 0x7f0f10005a00] App field don't match up: share <-> 
[swscaler @ 0x7f0f106defc0] deprecated pixel format used, make sure you did set range correctly
2022/01/26 18:18:20 [mksafe:3] Switch to input.ffmpeg_0 with transition.
2022/01/26 18:18:37 [clock.main:2] We must catchup 0.21 seconds!
2022/01/26 18:18:41 [clock.main:2] We must catchup 0.27 seconds!
2022/01/26 18:18:45 [clock.main:2] We must catchup 0.35 seconds!
2022/01/26 18:18:49 [clock.main:2] We must catchup 0.48 seconds!
2022/01/26 18:18:53 [clock.main:2] We must catchup 0.57 seconds!
2022/01/26 18:18:57 [clock.main:2] We must catchup 0.73 seconds!
2022/01/26 18:19:01 [clock.main:2] We must catchup 0.80 seconds!
2022/01/26 18:19:03 [clock.main:2] We must catchup 1.13 seconds!
2022/01/26 18:19:04 [clock.main:2] We must catchup 1.09 seconds!
2022/01/26 18:19:05 [clock.main:2] We must catchup 1.52 seconds!
2022/01/26 18:19:06 [clock.main:2] We must catchup 1.49 seconds!
…
2022/01/26 18:21:24 [clock.main:2] We must catchup 9.87 seconds!
2022/01/26 18:21:25 [clock.main:2] We must catchup 9.88 seconds!
2022/01/26 18:21:27 [clock.main:2] We must catchup 9.91 seconds!
2022/01/26 18:21:28 [clock.main:2] We must catchup 9.95 seconds!
2022/01/26 18:21:29 [clock.main:2] We must catchup 9.97 seconds!
[h264 @ 0x7f0f0ee90cc0] mmco: unref short failure
[h264 @ 0x7f0f0ee90cc0] illegal short term buffer state detected
^C2022/01/26 18:26:21 [main:3] Shutdown started!
2022/01/26 18:26:21 [main:3] Waiting for main threads to terminate...

[toots]

Thanks for this report. I agree that the application should shutdown more cleanly. At the heart of the issue, however, is the fact that your machine does not seem to be able to process the video encoding fast enough, which causes liquidsoap to queue up a lot of data which seems to cause a major failure with the ffmpeg library.

Have you tried with the ffmpeg command line utility? Do you get a better outcome?

[toots]

Additionally, if your goal is to pass the media data from the rtmp source to a srt output, you should be able to pass it without reencoding this way:

stream = mksafe(input.rtmp("rtmp://0.0.0.0:53163/"))
url = "srt://server:53162/output"
enc = %ffmpeg(
        format="flv",
        %audio.copy,
        %video.copy)
output.url(url=url, enc, stream)

This will definitively have a much better CPU load on your machine.. 🙂

[copy]

At the heart of the issue, however, is the fact that your machine does not seem to be able to process the video encoding fast enough, which causes liquidsoap to queue up a lot of data which seems to cause a major failure with the ffmpeg library.

I think my CPU should be powerful enough to decode and re-encode h264 in realtime.

I don't know how to exactly translate the liquidsoap to ffmpeg (after all, I'm learning liquidsoap to avoid having to deal with the ffmpeg command line). The following command that re-encodes the rtmp stream and saves it to a file seems to work fine though (with about 150% CPU usage according to htop; for reference liquidsoap shows about the same usage but as two separate processes).

ffmpeg -f flv -listen 1 -i rtmp://0.0.0.0:53163/ -c:v libx264 -preset veryfast -b:v 1500k -maxrate 1500k -pix_fmt yuv420p -g 30 out.mp4

Additionally, if your goal is to pass the media data from the rtmp source to a srt output, you should be able to pass it without reencoding this way

That's a good idea. On the other hand, it would be nice if liquidsoap could handle re-encoding too, since I'd eventually like to receive h265 or VP9 and re-encode it to h264.

[toots]

At the heart of the issue, however, is the fact that your machine does not seem to be able to process the video encoding fast enough, which causes liquidsoap to queue up a lot of data which seems to cause a major failure with the ffmpeg library.

I think my CPU should be powerful enough to decode and re-encode h264 in realtime.

I don't know how to exactly translate the liquidsoap to ffmpeg (after all, I'm learning liquidsoap to avoid having to deal with the ffmpeg command line). The following command that re-encodes the rtmp stream and saves it to a file seems to work fine though (with about 150% CPU usage according to htop; for reference liquidsoap shows about the same usage but as two separate processes).

ffmpeg -f flv -listen 1 -i rtmp://0.0.0.0:53163/ -c:v libx264 -preset veryfast -b:v 1500k -maxrate 1500k -pix_fmt yuv420p -g 30 out.mp4

Additionally, if your goal is to pass the media data from the rtmp source to a srt output, you should be able to pass it without reencoding this way

That's a good idea. On the other hand, it would be nice if liquidsoap could handle re-encoding too, since I'd eventually like to receive h265 or VP9 and re-encode it to h264.

Definitely.

If it's not the CPU, I'd suggest to test the rtmp output. If network communication is too slow, it will also create a catchup with liquidsoap.

For ffmpeg that should be something like:

ffmpeg -re -f flv -listen 1 -i rtmp://0.0.0.0:53163/ -c:v libx264 -preset veryfast -b:v 1500k -maxrate 1500k -pix_fmt yuv420p -g <uri>

The -re option tells ffmpeg to send in real time, like liquidsoap does. Then you should watch the reported speed. It should be around 1x at all time.

[copy]

Interestingly, your ffmpeg commands runs slightly slower than 1x. It reports speed=0.963x after leaving it running for two minutes. However, the delay between the input and output doesn't seem to be growing, and it keeps running stably.

If network communication is too slow, it will also create a catchup with liquidsoap.

That makes sense. Is it possible for liquidsoap to catch up by skipping input frames? Something like ffmpeg without the -re option (which should not be used if the input is a live stream, see https://ffmpeg.org/ffmpeg.html#Advanced-options).

[toots]

Well, if you run under constantly under 1x, you are bound to accumulate a growing delay. We have no mechanism for skipping frames at the moment but that's a good idea & I will think about it.

[toots]

I'm gonna convert this into a discussion and open a feature request for frame dropping.

[kpfa]

I was able able to input.ffmpeg an RTP stream and reboot the source (a Barix box) several times, which normally causes the dreaded: [clock.input.ffmpeg:2] We must catchup x.xx seconds!

• I found adding self_sync=true to have the "source control its own timing" resolved the synchronization issue.
• Additionally, setting ffmpeg string_args to: "reorder_queue_size","0" controls the number of packets to buffer for handling of reordered packets. For me there are no longer timing issues. and network delay isn't fatal, like it once was.

barix_rtp = input.ffmpeg(id="barix_rtp", self_sync=true,
    string_args=[("protocol_whitelist", "file,udp,rtp"),("reorder_queue_size","0")],
    "barix100_rtp.sdp")

[toots]

Thanks @kpfa this looks very useful. The self_sync parameter can indeed be crucial I forgot to mention it..