Problem of video frame skipping when saving RTSP stream as HLS

[zdco]

Hi!

I am using srs (ossrs) 3.0.6, the latest code from the devleop branch. I am using ingest to pull an rtsp stream and output it as rtmp and hls live streams.

The configuration file is as follows:

listen              1935;
pid                 ./objs/srs.pid;
chunk_size          60000;
ff_log_dir          ./objs;
srs_log_tank        file;
srs_log_level       trace;
srs_log_file        ./objs/srs.log;
max_connections     1000;
daemon              on;


http_stream {
    enabled         on;
    listen          8080;
    dir             ./objs/nginx/html;
}


vhost live {
    #gop_cache       off;
    #queue_length    10;


    hls {
        enabled         on;
        hls_path        ./objs/nginx/html;
        hls_fragment    10;
        hls_window      60;
        hls_on_error    ignore;
    }
}


vhost ingest {
    hls {
        enabled         off;
    }


    ingest livestream {
        enabled      on;
        input {
            type    stream;
            url     rtsp://192.168.1.231:554/udp/av0_0;
        }
        ffmpeg      ./objs/ffmpeg/bin/ffmpeg;
        engine {
            enabled          on;
            vcodec          copy;
            acodec          an;
            output          rtmp://127.0.0.1:[port]/live?vhost=live/livestream;
        }
    }
}

The rtmp address for the live stream is rtmp://192.168.1.117:1935/live?vhost=live/livestream.

When starting srs, an error occurred and the log is as follows:

[2015-11-23 15:16:19.691][trace][1500][105] fork process: ./objs/ffmpeg/bin/ffmpeg -i rtsp://192.168.1.231:554/udp/av0_0 -vcodec copy -an -f flv -y rtmp://127.0.0.1:1935/live?vhost=live/livestream 1 > ./objs/ffmpeg-ingest-ingest-
live-livestream.log 2 > ./objs/ffmpeg-ingest-ingest-live-livestream.log
[2015-11-23 15:16:19.691][trace][1500][105] vfored process, pid=1503, bin=./objs/ffmpeg/bin/ffmpeg
[2015-11-23 15:16:19.691][trace][1500][105] -> IGS time=11086, ingesters=1, #0(alive=11s, ingest/livestream)
[2015-11-23 15:16:20.731][trace][1500][119] RTMP client ip=127.0.0.1
[2015-11-23 15:16:20.732][trace][1500][119] complex handshake success
[2015-11-23 15:16:20.772][trace][1500][119] connect app, tcUrl=rtmp://127.0.0.1:1935/live?vhost=live, pageUrl=, swfUrl=, schema=rtmp, vhost=live, port=1935, app=live, args=null
[2015-11-23 15:16:20.772][trace][1500][119] out chunk size to 60000
[2015-11-23 15:16:20.851][trace][1500][119] input chunk size to 60000
[2015-11-23 15:16:20.852][trace][1500][119] client identified, type=fmle-publish, stream_name=livestream, duration=-1.00
[2015-11-23 15:16:20.852][trace][1500][119] source url=live/live/livestream, ip=127.0.0.1, cache=1, is_edge=0, source_id=-1[-1]
[2015-11-23 15:16:20.933][trace][1500][119] hls: win=60.00, frag=10.00, prefix=, path=./objs/nginx/html, m3u8=[app]/[stream].m3u8, ts=[app]/[stream]-[seq].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0
[2015-11-23 15:16:20.933][trace][1500][119] ignore disabled exec for vhost=live
[2015-11-23 15:16:20.944][trace][1500][119] exec thread cid=120, current_cid=119
[2015-11-23 15:16:20.954][trace][1500][119] start publish mr=0/350, p1stpt=20000, pnt=20000, tcp_nodelay=0, rtcid=121
[2015-11-23 15:16:20.973][trace][1500][119] got metadata, width=1920, height=1080, vcodec=7
[2015-11-23 15:16:20.973][error][1500][119][11] sps the seq_scaling_matrix_present_flag invalid, i=0, nb_scmpfs=8. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][error][1500][119][11] source codec demux video failed. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][error][1500][119][11] source process video message failed. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][error][1500][119][11] fmle process publish message failed. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][error][1500][119][11] thread process message failed. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][warn][1500][119][11] thread recv cycle failed, ignored and retry, ret=3001
[2015-11-23 15:16:20.973][error][1500][119][11] recv thread failed. ret=3001(Resource temporarily unavailable)
[2015-11-23 15:16:20.973][trace][1500][119] unpublish drop ts segment, sequence_no=0, uri=livestream-0.ts, duration=0.00, start=0
[2015-11-23 15:16:20.973][trace][1500][119] cleanup when unpublish
[2015-11-23 15:16:20.973][error][1500][119][4] stream service cycle failed. ret=3001(Interrupted system call)
[2015-11-23 15:16:22.693][trace][1500][105] process pid=1503 terminate, restart it.
[2015-11-23 15:16:25.573][trace][1500][104] user terminate program, gracefully quit.

Later, upon reviewing the code, it was found that there is a condition for seq_scaling_matrix_present_flag_i in srs_kernel_codec.cpp.

/*
if (seq_scaling_matrix_present_flag_i) {
    ret = ERROR_HLS_DECODE_ERROR;
    srs_error("sps the seq_scaling_matrix_present_flag invalid, i=%d, nb_scmpfs=%d. ret=%d", i, nb_scmpfs, ret);
    return ret;
}
*/

If it fails, it will exit. I think this condition is unnecessary, so I commented it out. After that, the live streaming can be used normally without any lag!

The log is as follows:

[2015-11-23 16:29:08.673][trace][1672][105] fork process: ./objs/ffmpeg/bin/ffmpeg -i rtsp://192.168.1.231:554/udp/av0_0 -vcodec copy -an -f flv -y rtmp://127.0.0.1:1935/live?vhost=live/livestream 1 > ./objs/ffmpeg-ingest-ingest-live-livestream.log 2 > ./objs/ffmpeg-ingest-ingest-live-livestream.log
[2015-11-23 16:29:08.673][trace][1672][105] vfored process, pid=2145, bin=./objs/ffmpeg/bin/ffmpeg
[2015-11-23 16:29:08.673][trace][1672][105] -> IGS time=4319782, ingesters=1, #0(alive=3s, ingest/livestream)
[2015-11-23 16:29:09.703][trace][1672][161] RTMP client ip=127.0.0.1
[2015-11-23 16:29:09.705][trace][1672][161] complex handshake success
[2015-11-23 16:29:09.745][trace][1672][161] connect app, tcUrl=rtmp://127.0.0.1:1935/live?vhost=live, pageUrl=, swfUrl=, schema=rtmp, vhost=live, port=1935, app=live, args=null
[2015-11-23 16:29:09.745][trace][1672][161] out chunk size to 60000
[2015-11-23 16:29:09.831][trace][1672][161] input chunk size to 60000
[2015-11-23 16:29:09.831][trace][1672][161] client identified, type=fmle-publish, stream_name=livestream, duration=-1.00
[2015-11-23 16:29:09.831][trace][1672][161] source url=live/live/livestream, ip=127.0.0.1, cache=1, is_edge=0, source_id=-1[-1]
[2015-11-23 16:29:09.911][trace][1672][161] hls: win=60.00, frag=10.00, prefix=, path=./objs/nginx/html, m3u8=[app]/[stream].m3u8, ts=[app]/[stream]-[seq].ts, aof=2.00, floor=0, clean=1, waitk=1, dispose=0
[2015-11-23 16:29:09.911][trace][1672][161] ignore disabled exec for vhost=live
[2015-11-23 16:29:09.921][trace][1672][161] exec thread cid=162, current_cid=161
[2015-11-23 16:29:09.931][trace][1672][161] start publish mr=0/350, p1stpt=20000, pnt=20000, tcp_nodelay=0, rtcid=163
[2015-11-23 16:29:09.952][trace][1672][161] got metadata, width=1920, height=1080, vcodec=7
[2015-11-23 16:29:09.953][trace][1672][161] 48B video sh,  codec(7, profile=High, level=Other, 128x2048, 0kbps, 0fps, 0s)
[2015-11-23 16:29:09.953][warn][1672][161][11] hls timestamp jump 388598670=>0
[2015-11-23 16:29:09.954][trace][1672][161] shrink the cache queue, size=1, removed=34, max=1.00
[2015-11-23 16:29:09.958][trace][1672][158] update source_id=161[161]
[2015-11-23 16:29:10.322][trace][1672][158] -> PLA time=38097, msgs=11, okbps=1501,1658,0, ikbps=0,0,0, mw=350
[2015-11-23 16:29:12.782][trace][1672][161] -> HLS time=4323709, stream dts=357030(3967ms), sno=325, ts=livestream-324.ts, dur=3.97, dva=0p
[2015-11-23 16:29:20.221][trace][1672][158] -> PLA time=48113, msgs=11, okbps=1548,1658,0, ikbps=0,0,0, mw=350
[2015-11-23 16:29:20.675][trace][1672][105] -> IGS time=4331798, ingesters=1, #0(alive=15s, ingest/livestream)
[2015-11-23 16:29:22.781][trace][1672][161] -> HLS time=4333723, stream dts=1257030(13967ms), sno=326, ts=livestream-325.ts, dur=3.93, dva=0p
[2015-11-23 16:29:29.931][trace][1672][161] <- CPB time=0, okbps=1,0,0, ikbps=1821,0,0, mr=0/350, p1stpt=20000, pnt=20000
[2015-11-23 16:29:30.480][trace][1672][158] -> PLA time=58119, msgs=11, okbps=1610,1658,0, ikbps=0,0,0, mw=350
[2015-11-23 16:29:32.676][trace][1672][105] -> IGS time=4343806, ingesters=1, #0(alive=27s, ingest/livestream)
[2015-11-23 16:29:32.780][trace][1672][161] -> HLS time=4343729, stream dts=2157120(23968ms), sno=327, ts=livestream-326.ts, dur=0.60, dva=0p
[2015-11-23 16:29:40.380][trace][1672][158] -> PLA time=68128, msgs=11, okbps=1646,1550,0, ikbps=0,0,0, mw=350
[2015-11-23 16:29:42.781][trace][1672][161] -> HLS time=4353752, stream dts=3057120(33968ms), sno=327, ts=livestream-326.ts, dur=10.60, dva=0p
[2015-11-23 16:29:44.677][trace][1672][105] -> IGS time=4355830, ingesters=1, #0(alive=39s, ingest/livestream)
[2015-11-23 16:29:49.920][trace][1672][158] -> PLA time=78147, msgs=0, okbps=1673,1550,0, ikbps=0,0,0, mw=350
[2015-11-23 16:29:49.931][trace][1672][161] <- CPB time=20022, okbps=0,1,0, ikbps=1846,1825,0, mr=0/350, p1stpt=20000, pnt=20000
[2015-11-23 16:29:52.819][trace][1672][161] -> HLS time=4363756, stream dts=3960180(44002ms), sno=328, ts=livestream-327.ts, dur=7.30, dva=0p

When using HLS live streaming, the streaming address is http://192.168.1.117:8080/live/livestream.m3u8. There is no screen flickering when playing, but the video is lagging. The image changes every three seconds, giving a feeling of only playing I-frames. I have uploaded the TS files to Baidu Cloud Drive, you can find them at http://pan.baidu.com/s/1dDwPmG5.

When I enable DVR mode and store the files in FLV format, they can be played normally. Only the TS files generated in HLS mode are not working properly.

Can you please tell me the reason for this?

TRANS_BY_GPT3

[winlinvip]

The failure to decode something in H.264 will result in some information in HLS being incorrect.

TRANS_BY_GPT3

[stylite1024]

Hello, I also encountered a similar problem. Did you solve yours?

TRANS_BY_GPT3

[alphonsetai]

@winlinvip I have some clues here. When pushing the stream to SRS using ffmpeg, it can be confirmed that there is an issue with the RTSP stream from the IPC, whether it is RTP or TCP.

In fact, not only HLS but also RTMP will be affected, but the extent of the impact depends on the player.

First, it is not possible to confirm whether it is a transmission issue in the network protocol or an issue with the ES stream. Currently, it has been observed that the stream output from the IPC is irregular:

1. The ES stream does not completely match the actual video encoding parameters of the IPC.
2. There is a mysterious jitter in the network data (observed in direct connections).

Now, the issue is being resolved by using the transcode method, which involves re-encoding the stream using ffmpeg and then pushing it out. The ffmpeg parameters used are similar to the encoding parameters of the IPC, which effectively solves this problem.

nginx-rtmp also has similar issues.

The above is a rough positioning of the information provided. Due to the loose coupling of SRS and codec, most of the ES stream processing and protocol handling are delegated to ffmpeg. I don't know if SRS has considered the possibility of tight coupling in the future.

My initial consideration is that both the IPC stream and ffmpeg may be related.

TRANS_BY_GPT3

[winlinvip]

Does not support pushing RTSP streams from the camera to SRS, only supports ingesting/FFmpeg pulling RTSP streams from the camera and then forwarding them to SRS. Please refer to: #2304 (comment)

TRANS_BY_GPT3