Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rust screamtx crash caused by a null buffer passed to rust callback #56

Closed
realuptime opened this issue Sep 7, 2023 · 3 comments
Closed

Rust screamtx crash caused by a null buffer passed to rust callback #56

realuptime opened this issue Sep 7, 2023 · 3 comments

Comments

@realuptime
Copy link

realuptime commented Sep 7, 2023 •
edited
Loading

Rust code does not like null pointers.
RtpQueue.pop() sometimes extracts null items (when the queue is empty?)

Here are three callstacks when it happened:

1:

rtpQueueDelay 0.237793 too large 1 6693.13 RTP queue 123 packets discarded for SSRC 1 hiSeqTx 27367 hiSeqAckendl 0 seqNrOfNextRtp 27368 seqNrOfLastRtp 27490
diff 123
imp.rs: force_idr rc true enabled 1
Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed
rtpQueueDelay 0.250488 too large 1 6693.39 RTP queue 133 packets discarded for SSRC 1 hiSeqTx 27500 hiSeqAckendl 0 seqNrOfNextRtp 27501 seqNrOfLastRtp 27633 diff 133
thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:358:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80001c894
Obtained 15 stack frames.
#0 0x007f521fd86420 in funlockfile
#1 0x007f521fbc100b in gsignal
#2 0x007f521fba0859 in abort
#3 0x007f50e0082b37 in std::sys::unix::abort_internal::hbf9c556da2861ce5
#4 0x007f50e0081252 in rust_panic
#5 0x007f50e00810da in std::panicking::rust_panic_with_hook::hdb4da1ae79c845a5
#6 0x007f50e0080e19 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::h02b5b35b126d5cf2
#7 0x007f50e007fbec in std::sys_common::backtrace::__rust_end_short_backtrace::h6c6853376cf416d1
#8 0x007f50e0080b22 in rust_begin_unwind
#9 0x007f50e0049f23 in core::panicking::panic_fmt::hfd9e949092070b66
#10 0x007f50e004a353 in core::result::unwrap_failed::h4d34d8346233eb49
#11 0x007f50e004d2b4 in gstscream::screamtx::imp::callback::hdc9f203da2c8fa8e
#12 0x007f50ec04f8b5 in transmitRtpThread(void*)
#13 0x007f521fd7a609 in start_thread
#14 0x007f521fc9d133 in clone

2:

Warning: Unexpected item 0x7f21040c0ea0 dequeued from queue queue1 (refcounting problem?)
0:46:24.098985438 52690 0x2a33cc0 ERROR queue_dataflow gstqueue.c:1486:gst_queue_push_one: exit because we have no item in the queue
rtpQueueDelay 0.497314 too large 1 2782.61 RTP queue 836 packets discarded for SSRC 1 hiSeqTx 58122 hiSeqAckendl 11191 seqNrOfNextRtp 58123 seqNrOfLastRtp 60763 diff 2641
thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80000cdd2
Obtained 17 stack frames.
#0 0x007f22d7c0fcf0 in __sigaction
#1 0x007f22d7c6626b in pthread_kill
#2 0x007f22d7c0fc46 in gsignal
#3 0x007f22d7bf67fc in abort
#4 0x007f2164590327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
#5 0x007f216458ea42 in rust_panic
#6 0x007f216458e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
#7 0x007f216458e609 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
#8 0x007f216458d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
#9 0x007f216458e312 in rust_begin_unwind
#10 0x007f21644d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
#11 0x007f21644d1313 in core::result::unwrap_failed::h62abf61e411aaa08
#12 0x007f21644ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
#13 0x007f21644e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
#14 0x007f216493d297 in transmitRtpThread(void*)
#15 0x007f22d7c64402 in pthread_condattr_setpshared
#16 0x007f22d7cf3590 in __xmknodat

3, exactly when a refresh happened:

refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
refresh 1035.84 RTP queue 1 packets discarded for SSRC 1
Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed
thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e8000068f4
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
refresh 1035.86 RTP queue 1 packets discarded for SSRC 1
Obtained 17 stack frames.
#0 0x007f3c0f4e5cf0 in __sigaction
#1 0x007f3c0f53c26b in pthread_kill
#2 0x007f3c0f4e5c46 in gsignal
#3 0x007f3c0f4cc7fc in abort
#4 0x007f3aa81fe327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
#5 0x007f3aa81fca42 in rust_panic
#6 0x007f3aa81fc8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
#7 0x007f3aa81fc609 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
#8 0x007f3aa81fb3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
#9 0x007f3aa81fc312 in rust_begin_unwind
#10 0x007f3aa813eee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
#11 0x007f3aa813f313 in core::result::unwrap_failed::h62abf61e411aaa08
#12 0x007f3aa814bb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
#13 0x007f3aa815394e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
#14 0x007f3b0803a28e in transmitRtpThread(void*)
#15 0x007f3c0f53a402 in pthread_condattr_setpshared
#16 0x007f3c0f5c9590 in __xmknodat
@realuptime
Copy link
Author

Another over-night crash:

rtpQueueDelay 0.232422 too large 1 13919.3 RTP queue 118 packets discarded for SSRC 1 hiSeqTx 58159 hiSeqAckendl 0 seqNrOfNextRtp 58160 seqNrOfLastRtp 58277 diff 118
imp.rs: force_idr rc true enabled 1
rtpQueueDelay 0.253906 too large 1 Warning: Unexpected item 0x7f85e402b7e0 dequeued from queue queue1 (refcounting problem?)
13919.6 RTP queue 133 packets discarded for SSRC 1 hiSeqTx 58283 hiSeqAckendl 0 seqNrOfNextRtp 58284 seqNrOfLastRtp 58416 diff 133
3:52:01.238217250 ESC[36m130979ESC[00m 0x25fb8c0 ESC[31;01mERROR ESC[00m ESC[00m queue_dataflow gstqueue.c:1486:gst_queue_push_one:ESC[00m exit because we have no item in the queue
thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e80001ffa3
Obtained 17 stack frames.
#0 0x007f87d066ecf0 in __sigaction
#1 0x007f87d06c526b in pthread_kill
#2 0x007f87d066ec46 in gsignal
#3 0x007f87d06557fc in abort
#4 0x007f864c990327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
#5 0x007f864c98ea42 in rust_panic
#6 0x007f864c98e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
#7 0x007f864c98e609 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
#8 0x007f864c98d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
#9 0x007f864c98e312 in rust_begin_unwind
#10 0x007f864c8d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
#11 0x007f864c8d1313 in core::result::unwrap_failed::h62abf61e411aaa08
#12 0x007f864c8ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
#13 0x007f864c8e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
#14 0x007f86900b8297 in transmitRtpThread(void*)
#15 0x007f87d06c3402 in pthread_condattr_setpshared
#16 0x007f87d0752590 in __xmknodat

@realuptime
Copy link
Author

and another:

summary 1628.9 Transmit rate = 8367kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps
InputServer: Received Heartbeat
InputServer: Received Heartbeat
summary 1630.9 Transmit rate = 8264kbps, PLR = 0.00%( 0.03%), RTT = 0.049s, Queue delay = 0.015s encoder_rate 8000 kbbps
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
refresh 1631 RTP queue 1 packets discarded for SSRC 1
Error: gst_pad_push: assertion 'GST_IS_BUFFER (buffer)' failed
thread '' panicked at 'Screamtx callback srcpad.push failed: Error', src/screamtx/imp.rs:360:22
note: run with RUST_BACKTRACE=1 environment variable to display a backtrace
fatal runtime error: failed to initiate panic, error 5
Caught fatal signal - signo:6 code:-6 errno:0 addr:0x3e800005ba0
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.03 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.05 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.07 RTP queue 1 packets discarded for SSRC 1
refresh 1631.09 RTP queue 1 packets discarded for SSRC 1
refresh 1631.09 RTP queue 1 packets discarded for SSRC 1
refresh 1631.09 RTP queue 1 packets discarded for SSRC 1
refresh 1631.09 RTP queue 1 packets discarded for SSRC 1
refresh 1631.09 RTP queue 1 packets discarded for SSRC 1
Obtained 17 stack frames.
#0 0x007fa46c7d2cf0 in __sigaction
#1 0x007fa46c82926b in pthread_kill
#2 0x007fa46c7d2c46 in gsignal
#3 0x007fa46c7b97fc in abort
#4 0x007fa2f9390327 in std::sys::unix::abort_internal::hf940f5a54d8a80b4
#5 0x007fa2f938ea42 in rust_panic
#6 0x007fa2f938e8ca in std::panicking::rust_panic_with_hook::h5cafdc4b3bfd5528
#7 0x007fa2f938e609 in std::panicking::begin_panic_handler::_$u7b$$u7b$closure$u7d$$u7d$::hf31c60f40775892c
#8 0x007fa2f938d3dc in std::sys_common::backtrace::__rust_end_short_backtrace::h28a5c7be595826cd
#9 0x007fa2f938e312 in rust_begin_unwind
#10 0x007fa2f92d0ee3 in core::panicking::panic_fmt::h8fa27a0b37dd98b7
#11 0x007fa2f92d1313 in core::result::unwrap_failed::h62abf61e411aaa08
#12 0x007fa2f92ddb18 in core::result::Result$LT$T$C$E$GT$::expect::h54fe755c33896c5a
#13 0x007fa2f92e594e in gstscream::screamtx::imp::callback::h11dff52b2a2873a4
#14 0x007fa30c04c297 in transmitRtpThread(void*)
#15 0x007fa46c827402 in pthread_condattr_setpshared
#16 0x007fa46c8b6590 in __xmknodat

@jacobt21
Copy link
Collaborator

Pushed the fix

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@realuptime @jacobt21