Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transcript: Noisy introduced in audio stream. #222

Open
winlinvip opened this issue Sep 28, 2024 · 4 comments
Open

Transcript: Noisy introduced in audio stream. #222

winlinvip opened this issue Sep 28, 2024 · 4 comments
Assignees
Labels
bug Something isn't working EnglishNative This issue is conveyed exclusively in English.

Comments

@winlinvip
Copy link
Member

winlinvip commented Sep 28, 2024

Origin stream is OK:

original.zip

Transcript stream has noisy:

transcript.zip

The noisy comes from the transcoding for each piece, so it appears every 10s, when switching to next ts piece.

You can use VLC to play the index.m3u8 and reproduce this issue.

@winlinvip winlinvip added bug Something isn't working EnglishNative This issue is conveyed exclusively in English. labels Sep 28, 2024
@winlinvip winlinvip self-assigned this Sep 28, 2024
@winlinvip
Copy link
Member Author

If change the fps to 25, the stutter is less but still exists.

@suzp1984
Copy link
Contributor

suzp1984 commented Nov 17, 2024

Play transcript.zip by ffplay: ffplay -loglevel debug index.m3u8, the warning message is

[hls @ 0x144e076c0] HLS request for url '1455-overlay-b5c12e9c-53d9-4cac-ac4f-04f08a6c8637.ts', offset 0, playlist 0
[hls @ 0x144e076c0] Opening '1455-overlay-b5c12e9c-53d9-4cac-ac4f-04f08a6c8637.ts' for reading
[mpegts @ 0x144e07c40] Continuity check failed for pid 17 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 0 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 4096 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 256 expected 11 got 0
[mpegts @ 0x144e07c40] Packet corrupt (stream = 0, dts = 1310932497).
[hls @ 0x144e076c0] Packet corrupt (stream = 0, dts = 1310929494).

debug the ffplay by gdb, the Packet corrupt log print in

https://github.com/FFmpeg/FFmpeg/blob/1d55f548466516a6a4ed4cc67e118e1c29c6ae5a/libavformat/demux.c#L577-L581C78

    if (pkt->flags & AV_PKT_FLAG_CORRUPT) {
        av_log(s, AV_LOG_WARNING,
               "Packet corrupt (stream = %d, dts = %s)%s.\n",
               pkt->stream_index, av_ts2str(pkt->dts),
               s->flags & AVFMT_FLAG_DISCARD_CORRUPT ? ", dropping it" : "");

the AV_PKT_FLAG_CORRUPT is set at:
https://github.com/FFmpeg/FFmpeg/blob/1d55f548466516a6a4ed4cc67e118e1c29c6ae5a/libavformat/mpegts.c#L2814-L2822
when demux failed the continuity check.

[mpegts @ 0x144e07c40] Continuity check failed for pid 17 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 0 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 4096 expected 4 got 0
[mpegts @ 0x144e07c40] Continuity check failed for pid 256 expected 11 got 0

  • How TS continuity check works?

https://github.com/ossrs/srs/blob/7951bf3bd63a33ec5bfddfba39f9d6e2d476e381/trunk/src/kernel/srs_kernel_ts.hpp#L406-L420

There is a 4-bit size continuity_counter field in TS packet, in range [0, 15], to keep the order of ts packets per pid.

when ffmpeg transcode the ts segment file, the countinuity_counter recount from 0 per pid. And I didn't find the ffmpeg cmd or filter to config the start counter, [0, 15], per pid for mpeg-ts format yet.
If countinuity_counter config is not the feature of ffmpeg then I need to mux the mpeg-ts packet by code.

  1. first, maintain a map[pid] = previous ts segmet's last countinuity_counter;
  2. add another queue to the oryx transcript task, which demux and mux the ts packets by recount countinuity_counter from map[pid] + 1, and rewrite map[pid] by current ts segment's last packet.
  • Verify this issue

  • dev a countinuity_counter recount app and apply to transcript.zip, then let VLC works well.


Confirmed both the countinuity_counter (cc) and timestamp impact this issue.
Here is the cc fixed transcript sample:

transcript_cc_fixed.zip

The code to account cc manually:
https://gist.github.com/suzp1984/37a356a3fec4e69531ec10fbcb4a627f

@suzp1984
Copy link
Contributor

compare transcript.zip and transcript_cc_fixed.zip in VLC.

The sound loss of "transcript.zip" is very obvious, while the sound of "transcript_cc_fixed.zip" is discontinuous between ts segments. The transcript_cc_fixed.zip only let cc count continually between ts, so I think if the timestamp between ts improved, I believe the sounds can play well.

the timestamp can be improved by choice 25 fps video source.

If change the fps to 25, the stutter is less but still exists.

Next action:

  • More testing with 25 fps video source + cc recount improve.
  • Edit timestamp of TS segment by code.
  • Check ffmpeg doc found possible way to rewrite|config cc count and timestamp.
  • Revalue the solutions whether fixed in ffmpeg or oryx.

@suzp1984
Copy link
Contributor

suzp1984 commented Nov 18, 2024

Another testing I did to verify the sound discontinuous problem, check above comment, the cc count continually between ts can fix the obvious problem, but still sound still not perfect.

I use the original.zip to reproduce the same audio stream problem.

here is the steps to reproduce:

  1. transcode the ts segments in original.zip
    ffmpeg -i 1457-org-cfe71a34-a90d-4e3f-9ec8-b008223cc957.ts -c:v libx264 -profile:v main -preset:v medium -tune zerolatency -bf 0 -c:a copy -copyts -y aac_copy/1457-org-cfe71a34-a90d-4e3f-9ec8-b008223cc957.ts
    transcode the ts segment with similar method used in oryx, but the audio data is copy without transcode, the video data is transcoded without rendering srt file.
    apply above cmd to every ts segments, and copy index.m3u8 to target folder aac_copy/ later.
  2. play aac_copy/index.m3u8 in VLC, the audio stream stutter is reproduced so close to transcript.zip
  3. apply cc count to ts segments, aac_copy/*.ts:
    cp aac_copy/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts aac_copy_cc_fixed/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts
    the first ts need not to fixed.
    run https://gist.github.com/suzp1984/37a356a3fec4e69531ec10fbcb4a627f to find the latest cc number for audio and video stream.
    go run . aac_copy_cc_fixed/1454-org-024c7fd8-9b60-431c-9ef6-236518f00078.ts
    change the video stream pid (256) and audio stream pid (257) new start count number according to above probe result.
    then run
    go run aac_copy/1455-*.ts aac_copy_cc_fixed/1455-*.ts
    redo similar probe for aac_copy_cc_fixed/1455-*.ts: go run . aac_copy_cc_fixed/1455-*.ts.
    then do same job to 1456-*.ts.
  4. play aac_copy_cc_fixed/index.m3u8 in VLC, the stream works really well.

I know those steps are all did manually and so silly. I upload the result.

aac_copy.zip
is the result of step 1, which reproduce the audio stream stutter.
aac_copy_cc_fixed.zip
is the result of step 3, which apply cc fix to step 1, the audio stream stutter fixed.

Conclusion
The cc discontinuously between ts segments and audio stream transcode both contribute the audio stutter.

suzp1984 added a commit to suzp1984/oryx that referenced this issue Nov 21, 2024
1. let continuity_counter continually between overlay ts;
2. copy audio data when transcoding ts;
suzp1984 added a commit to suzp1984/oryx that referenced this issue Nov 21, 2024
1. let continuity_counter continually between overlay ts;
2. copy audio data when transcoding ts;
suzp1984 added a commit to suzp1984/oryx that referenced this issue Nov 24, 2024
1. let continuity_counter continually between overlay ts;
2. copy audio data when transcoding ts;
suzp1984 added a commit to suzp1984/oryx that referenced this issue Nov 25, 2024
1. let continuity_counter continually between overlay ts;
2. copy audio data when transcoding ts;
suzp1984 added a commit to suzp1984/oryx that referenced this issue Nov 25, 2024
1. let continuity_counter continually between overlay ts;
2. copy audio data when transcoding ts;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working EnglishNative This issue is conveyed exclusively in English.
Projects
None yet
Development

No branches or pull requests

2 participants