forked from FFmpeg/FFmpeg
-
Notifications
You must be signed in to change notification settings - Fork 12
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
FFmpeg: Add Whip Muxer support for subsecond latency streaming #1
Open
winlinvip
wants to merge
60
commits into
ossrs:feature/whip
Choose a base branch
from
winlinvip:feature/rtc-muxer
base: feature/whip
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+2,882
−2
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
winlinvip
force-pushed
the
feature/rtc-muxer
branch
7 times, most recently
from
April 21, 2023 23:16
598aecc
to
cf69b01
Compare
winlinvip
changed the title
WHIP: Support FFmpeg WebRTC muxer via WHP protocol.
FFmpeg: Support WebRTC muxer via WHP protocol.
Apr 21, 2023
winlinvip
changed the title
FFmpeg: Support WebRTC muxer via WHP protocol.
FFmpeg: Support WebRTC muxer via WHIP protocol.
Apr 21, 2023
winlinvip
force-pushed
the
feature/rtc-muxer
branch
8 times, most recently
from
April 22, 2023 23:23
48d93eb
to
92c4187
Compare
winlinvip
force-pushed
the
feature/rtc-muxer
branch
13 times, most recently
from
May 2, 2023 01:02
b4da552
to
2ad58fe
Compare
…cessary ClientHello.
1. Fix OpenSSL build error. 2. Support OpenSSL 1.0.1k and newer versions. 3. Support WHIP authorization via Bearer HTTP header. 4. Change the option default value from 1500 to 1200, to make Pion work. 5. Detect the minimum required OpenSSL version, should be 1.0.1k and newer. 6. Quickly check the SDP answer by taking a glance at the first few bytes.
1. Merge ICE and DTLS ARQ max retry options into a single handshake timeout. 2. Utilize DTLS server role to prevent ARQ, as the peer DTLS client will handle ARQ. 3. Replace IO from DTLSContext with a callback function. 4. Measure and analyze the time cost for each step in the process. 5. Implement DTLS BIO callback for packet fragmentation using BIO_set_callback. 6. Generate private key and certificate prior to ICE for faster handshake. 7. Refine DTLS MTU settings using SSL_set_mtu and DTLS_set_link_mtu. 8. Provide callback for DTLS state, returning errors when DTLS encounters issues or closes. 9. Consolidate ICE request/response handling and DTLS handshake into a single function.
1. Refine WHIP muxer name. 1. Refine SRTP key macros. 1. Refine logging context. 1. Refine SSL error messages. 1. Refine DTLS error messages. 1. Refine RTC error messages. 1. Use AV_RB8 to read integer from memory. 1. Update DTLS curve list to X25519:P-256:P-384:P-521. 1. Refine SRTP profile name for FFmpeg and OpenSSL. 1. Replace magic numbers with macros and extract to functions. 1. Alter log levels from INFO to VERBOSE, except for final results. 1. Use typedef SRTPContext. 1. Refine the ICE STUN magic number. 1. Reposition the on_rtp_write_packet function. 1. Refer to Chrome definition of RTP payload types. 1. Replace magic numbers with macros for RTP and RTCP payload types. 1. Rename to WHIP muxer. 1. Add TODO for OPUS timestamp issue. 1. Refine comments, do not hardcode H.264. 1. Define SDP session id and creator IP as macros. 1. Refine fixed frame size 960 to rtc->audio_par->frame_size. 1. Use h264_mp4toannexb to convert MP4/ISOM to annexb. 1. Address occasional inaccuracies in OPUS audio timestamps. 1. Correct marker setting after utilizing BSF. 1. Remove dependency on avc.h after using BSF.
…64_mp4toannexb filter only processing MP4 ISOM format.
1. Change the CommonName from ffmpeg.org to lavf. 2. Rename rtcenc.c to whip.c, rtc to whip. 3. Replace av_get_random_seed by AVLFG. 4. Add TODO to support libtls, mbedtls, and gnutls.
winlinvip
force-pushed
the
feature/rtc-muxer
branch
from
October 17, 2023 02:39
df62b77
to
63e3a55
Compare
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
WHIP stands for the WebRTC-HTTP ingestion protocol, which is a sub-second streaming protocol designed by encoders and publishers. It is widely supported by various tools and media servers, allowing it to interact with other WebRTC clients, ingest streams to media servers, and is compatible with all modern browsers.
Unfortunately, most WHIP implementations are highly complex and require modern C++11 or C++14, or RUST. This complexity makes it impossible to integrate with FFmpeg, which requires C.
However, if FFmpeg were to incorporate WHIP support, it could be a game-changer for both FFmpeg and the WebRTC ecosystem, particularly for certain IoT or small devices that are too small to run modern languages but can run FFmpeg.
To meet FFmpeg's requirements, this PR contains just C code. And we have rewritten the WHIP and WebRTC protocol stack using only around 3k lines of C code.
Content
Usage
Please select a open-source WHIP server to work with FFmpeg.
If you encounter any issues or get stuck, please leave us a message on Discord.
You also have the option to select a WHIP cloud service, which is typically provided by video cloud service providers.
If you encounter any issues or get stuck, please leave us a message on Discord.
Usage: FFmpeg + SRS
To enable FFmpeg to publish a stream, SRS can be used as the WHIP server. It is recommended to use docker.
Alternatively, you can build SRS from its source code.
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing stream to SRS, you can play the WHIP stream in web browser such as Chrome, using srs-player.
The image below shows that the latency is around 150ms.
The RTMP, HTTP-FLV, or HTTP-TS stream remuxed by SRS can be played using ffplay, VLC, or srs-player.
Usage: FFmpeg + Janus
We referred to WISH, WHIP and Janus: Part II to make it possible to use FFmpeg for publishing a stream to Janus via WHIP. We have also created a demo docker image janus-docker for quick testing.
git clone https://github.com/winlinvip/janus-docker.git cd janus-docker
Initially, run the demo Docker that includes Janus server using the following command:
After that, access the URL http://localhost:8081/videoroomtest.html?room=2345 in your browser to join the Janus room.
Next, download and run the Simple WHIP Server for Janus using the following command:
git clone https://github.com/meetecho/simple-whip-server.git cd simple-whip-server npm install npm run build npm run start
Generate a WHIP handler using curl, which will enable ffmpeg to join the same Janus room through WHIP:
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing the stream to the Janus room, you will be able to view it on the previously opened webpage. The image below shows that the latency is around 120ms.
Usage: FFmpeg + Pion
To download the code and build FFmpeg, you can use the following command.
Although WebRTC has the capability to support x264 main and high profiles without B frames, it is advisable to use the baseline profile for better compatibility. If your stream doesn't have these codecs, you can transcode it using FFmpeg.
After publishing stream to pion, you can play the WebRTC stream in web browser such as Chrome
Usage: FFmpeg + Millicast
On the way.
Usage: FFmpeg + TRTC
On the way...
Usage: FFmpeg + Cloudflare
Here's a WHIP URL for testing: https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/67805a89d86d263b95f7056e5b212a0ek99a1686b042e85c3a5f86c38af9d9dad/webRTC/publish
And WHEP to play it back:
https://customer-wi9sckcs7uxt7lh4.cloudflarestream.com/99a1686b042e85c3a5f86c38af9d9dad/webRTC/play
You can use this player: https://wish.chens.link/watch
Known Issues
The current version has some known issues that we plan to fix in the future. You are welcome to help us fix them by sending a pull request.
The current known issues include:
-tune zerolatency
option is enabled andthreads>1
, FFmpeg will encode the frame in multiple slices. This process can cause stuttering in Chrome's decoding, with only the IDR being able to be decoded. Both the IDR and P frames may be encoded to multiple slices and sent by multiple RTP packets with the same timestamp. RTC players, such as Chrome, may have issues decoding frames that have been encoded using multiple slices. As a result, Chrome may only decode some of the slices of the IDR frame and drop the P frames. If this issue occurs, it may appear as if the video player is stuttering and the decoded framerate will be equivalent to the GOP size.Below are the issues that have already been fixed:
Latency
To test the end-to-end latency of a WIHP stream published using FFmpeg, you can capture your desktop using FFmpeg, open a stopwatch or miaobiao in the browser, and compare the player with the original stopwatch.
The test results are incredible! The image below shows that the latency is around 150ms.
OpenSSL
The following OpenSSL versions are supported. A GitHub action here is available to automatically test all major OpenSSL versions for compatibility with FFmpeg WHP.
In short, OpenSSL
1.0.1k
and newer versions should work. However, OpenSSL 1.1.0h and newer verions are highly recommended.Execute the command below to compile OpenSSL.
Load Certificate File
To import a DTLS certificate and private key from a file, you should first generate an SSL certificate and private key file or obtain them from a Certificate Authority (CA) service.
openssl genrsa -out dtls.key 2048 openssl req -new -x509 -key dtls.key -out dtls.crt -days 3650 \ -subj "/C=CN/ST=Beijing/L=Beijing/O=Me/OU=Me/CN=ossrs.net"
Then use
-cert_file
and-key_file
to load it:It works.
Authorization
Set option
-authorization token
to use the authorization of WHIP, please refer to the Authentication and authorization.Contributors
This patch has been created and is maintained by the developers below.
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
This patch has been reviewed by the developers listed below, and we extend our gratitude to them.
The following review comments pertain to the patch/whip/v0 branch.
[email protected]
[email protected]
[email protected]
[email protected]
[email protected]
The following review comments pertain to the patch/whip/v1 branch.
Links