Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Crash when using OBS Virtual Camera in Chrome #4615

Closed
randomascii opened this issue Apr 28, 2021 · 13 comments
Closed

Crash when using OBS Virtual Camera in Chrome #4615

randomascii opened this issue Apr 28, 2021 · 13 comments
Labels
Crash Report Feedback required Issue does not contain enough information or is missing the template Windows Affects Windows

Comments

@randomascii
Copy link

randomascii commented Apr 28, 2021

Operating System Info

Windows 10

Other OS

No response

OBS Studio Version

27.0.0-rc2

OBS Studio Version (Other)

No response

OBS Studio Log URL

Not available

OBS Studio Crash Log URL

Not available

Expected Behavior

No crashes

Current Behavior

Some Chrome users hit crashes when using OBS. I am a Chrome developer so I receive and analyze these crashes. I have not reproduced these crashes and I don't know what the users are doing. This was initially reported here:

CatxFish/obs-virtual-cam#141

The tracking bug on the Chromium side is crbug.com/1137308 but this link is just for my purposes - it has restricted viewing.

Since then 27.0 rc2 has become available and I have restricted my analysis to those crashes since PDBs are available.

I did some crash queries and found seven related crashes with the 27.0.0 OBS. It's too early to tell if the crash rate will be affected by the newest version but I don't think so. Here is the data from one of the crashes, with the OBS symbols:

0:016> .ecxr
rax=000000520e5ff618 rbx=000032f4002778e0 rcx=0000000000000000
rdx=0000000000000000 rsi=0000000000000000 rdi=000000520e5ff618
rip=00007ff837d63fe9 rsp=000000520e5ff5e0 rbp=000000520e5ff8e0
r8=0000000000000000 r9=000000520e5ff608 r10=0000000000000000
r11=0000000000000246 r12=000000520e5ff8f0 r13=000000520e5ff8e0
r14=000032f40043a080 r15=000000520e5ff8c8
iopl=0 nv up ei pl nz na po nc
cs=0033 ss=0000 ds=0000 es=0000 fs=0053 gs=002b efl=00010206
chrome!media::VideoCaptureDeviceClient::OnIncomingCapturedBufferExt+0x199:
00007ff837d63fe9 488b06 mov rax,qword ptr [rsi] ds:0000000000000000=????????????????
0:016> k
  *** Stack trace for last set context - .thread/.cxr resets it
 # Child-SP          RetAddr           Call Site
00 00000052`0e5ff5e0 00007ff8`37d62d5d chrome!media::VideoCaptureDeviceClient::OnIncomingCapturedBufferExt+0x199 [c:\b\s\w\ir\cache\builder\src\media\capture\video\video_capture_device_client.cc @ 631] 
01 00000052`0e5ff810 00007ff8`38c4d358 chrome!media::VideoCaptureDeviceClient::OnIncomingCapturedData+0x5bd [c:\b\s\w\ir\cache\builder\src\media\capture\video\video_capture_device_client.cc @ 390] 
02 00000052`0e5ffb50 00007ff8`3964cbe6 chrome!media::VideoCaptureDeviceWin::FrameReceived+0x128 [c:\b\s\w\ir\cache\builder\src\media\capture\video\win\video_capture_device_win.cc @ 875] 
03 00000052`0e5ffc60 00007ff8`3d1192a0 chrome!media::SinkInputPin::Receive+0x106 [c:\b\s\w\ir\cache\builder\src\media\capture\video\win\sink_input_pin_win.cc @ 235] 
04 00000052`0e5ffce0 00007ff8`3d11ad00 obs_virtualcam_module64!DShow::OutputPin::UnlockSampleData+0x90 [D:\obs2\plugins\win-dshow\libdshowcapture\source\output-filter.cpp @ 594] 
05 (Inline Function) --------`-------- obs_virtualcam_module64!DShow::OutputFilter::UnlockSampleData+0xf [D:\obs2\plugins\win-dshow\libdshowcapture\source\output-filter.hpp @ 213] 
06 00000052`0e5ffd20 00007ff8`3d11aeeb obs_virtualcam_module64!VCamFilter::Frame+0x210 [D:\obs2\plugins\win-dshow\virtualcam-module\virtualcam-filter.cpp @ 254] 
07 00000052`0e5ffd80 00007ff8`3d11a3a1 obs_virtualcam_module64!VCamFilter::Thread+0x12b [D:\obs2\plugins\win-dshow\virtualcam-module\virtualcam-filter.cpp @ 185] 
08 (Inline Function) --------`-------- obs_virtualcam_module64!VCamFilter::{ctor}::__l2::<lambda_4e6f779957cbddf8fba6c1db701dac17>::operator()+0x8 [D:\obs2\plugins\win-dshow\virtualcam-module\virtualcam-filter.cpp @ 119] 
09 (Inline Function) --------`-------- obs_virtualcam_module64!std::invoke+0x8 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\include\type_traits @ 1584] 
0a 00000052`0e5ffdd0 00007ff8`3d186460 obs_virtualcam_module64!std::threed::_Invoke<std::tuple<<lambda_4e6f779957cbddf8fba6c1db701dac17> >,0>+0x11 [C:\Program Files (x86)\Microsoft Visual Studio\2019\Professional\VC\Tools\MSVC\14.28.29333\include\thread @ 44] 
0b 00000052`0e5ffe00 00007ff8`7f867034 obs_virtualcam_module64!thread_start<unsigned int (__cdecl*)(void *),1>+0x50 [minkernel\crts\ucrt\src\appcrt\startup\thread.cpp @ 97] 
0c 00000052`0e5ffe30 00007ff8`80762651 KERNEL32!BaseThreadInitThunk+0x14
0d 00000052`0e5ffe60 00000000`00000000 ntdll!RtlUserThreadStart+0x21

(note that I changed the spelling of thread to threed to stop an emoji from being inserted there)

As before it is a NULL dereference crash. The crash is right after a call to media::ReadyFrameInBuffer::ReadyFrameInBuffer and it is clear that receiver_ is NULL when we try to call OnFrameReadyInBuffer. You can see the code right here:

https://source.chromium.org/chromium/chromium/src/+/master:media/capture/video/video_capture_device_client.cc;l=633?q=video_capture_device_client.cc

Does that help?

BTW, one of the crashes had a slightly different build of the DLL. The version number is the same but the build date was April 2nd instead of April 12. FWIW:

0:015> lmv m obs_virtualcam_module64
Browse full module list
start end module name
00007ffcca120000 00007ffcca201000 obs_virtualcam_module64 T (no symbols)
Loaded symbol image file: obs-virtualcam-module64.dll
Image path: C:\Program Files\obs-studio\data\obs-plugins\win-dshow\obs-virtualcam-module64.dll
Image name: obs-virtualcam-module64.dll
Browse all global symbols functions data
Timestamp: Fri Apr 2 08:52:08 2021 (60673DA8)
CheckSum: 00000000
ImageSize: 000E1000
File version: 27.0.0.0

Steps to Reproduce

Unknown.

Anything else we should know?

No response

@notr1ch
Copy link
Member

notr1ch commented Apr 28, 2021

Thanks for the new crash log. This is a strange place to be crashing, as this code path only occurs when all of the following succeed:

	hr = allocator->GetBuffer(&sample, nullptr, nullptr, 0);
	if (FAILED(hr))
		return false;

	if (FAILED(sample->SetActualDataLength((long)bufSize)))
		return false;
	if (FAILED(sample->SetDiscontinuity(false)))
		return false;
	if (FAILED(sample->SetPreroll(false)))
		return false;
	if (FAILED(sample->GetPointer(ptr)))
		return false;

So the pointer for the sample data should be valid. Do you happen to know what Chrome is trying to dereference at OnIncomingCapturedBufferExt+0x199?

@notr1ch notr1ch added Crash Report Windows Affects Windows labels Apr 28, 2021
@randomascii
Copy link
Author

See the source-code link, which takes you to this:

buffer_pool_->HoldForConsumers(buffer.id, 1);
receiver_->OnFrameReadyInBuffer(
ReadyFrameInBuffer(
buffer.id, buffer.frame_feedback_id,
std::make_unique<ScopedBufferPoolReservation>(
buffer_pool_, buffer.id),
std::move(info)),
{});
}

The crash is when dereferencing receiver_ to load the v-table pointer, which is being loaded in order to call OnFrameReadyInBuffer.

From the code-search link you can easily go to the declaration of receiver_, which is this:

// The receiver to which we post events.
const std::unique_ptr receiver_;

@notr1ch
Copy link
Member

notr1ch commented Apr 28, 2021

Sorry, I missed that link initially, I got too excited with the new stack trace :). receiver seems to be private to Chrome, so it's odd that the virtual camera would be making it null. The video thread in the virtual camera module will continue to output samples until the filter is destroyed, so perhaps there is some race condition or case where Chrome is trying to stop the output (and destroying receiver) but still receiving and trying to process samples. AFAIK we don't implement the IMediaFilter::Stop method, so perhaps that is a good place to start looking.

@randomascii
Copy link
Author

When you say that you don't implement IMediaFilter::Stop does that mean that Chrome might be telling you to stop but you keep sending data?

I don't work on the media team so I'm unclear on how all of this ties together or what the contract is. It would be nice to have a repro.

@notr1ch
Copy link
Member

notr1ch commented Apr 30, 2021

That's what I'm thinking might be happening. Our output thread keeps running until the filter is unloaded, so even after a Stop call it may still be calling Receive with a new sample on the connected pin. It's possible Stop is something that's handled by one of the base classes so there's no point implementing it ourselves, I'm unfortunately not very familiar with DirectShow.

Comparing our filter code to the Microsoft SDK samples does make me think we might be missing a few bits and pieces though. A repro case would indeed be very useful. Is there anything else in common between the Chrome crash reports?

@WizardCM WizardCM changed the title Crash when using OBS in Chrome Crash when using OBS Virtual Camera in Chrome May 1, 2021
@ilyanikolaevskiy
Copy link

There's a new type of crash discovered (https://bugs.chromium.org/p/chromium/issues/detail?id=1240085) with the same root cause: the OBSVirtualCamera keeps sending frames at chrome even though it should be stopped.

In the crashlogs I see two threads with the following callstacks:
Crashed thread:

0x00007ff8971cca30 | (ntdll.dll + 0x0006ca30) |   |  
-- | -- | -- | --
  | 0x00007ff80c12b01c | (chrome.dll -video_capture_buffer_pool_impl.cc:199) |   | media::VideoCaptureBufferPoolImpl::HoldForConsumers(int,int)
  | 0x00007ff80c12bd1e | (chrome.dll -video_capture_device_client.cc:634) |   | media::VideoCaptureDeviceClient::OnIncomingCapturedBufferExt(media::VideoCaptureDevice::Client::Buffer,media::VideoCaptureFormat const &,gfx::ColorSpace const &,base::TimeTicks,base::TimeDelta,gfx::Rect,media::VideoFrameMetadata const &)
  | 0x00007ff80c12b85a | (chrome.dll -video_capture_device_client.cc:393) |   | media::VideoCaptureDeviceClient::OnIncomingCapturedData(unsigned char const *,int,media::VideoCaptureFormat const &,gfx::ColorSpace const &,int,bool,base::TimeTicks,base::TimeDelta,int)
  | 0x00007ff80cc66a5e | (chrome.dll -video_capture_device_win.cc:882) |   | media::VideoCaptureDeviceWin::FrameReceived(unsigned char const *,int,media::VideoCaptureFormat const &,base::TimeDelta,bool)
  | 0x00007ff80d46a788 | (chrome.dll -sink_input_pin_win.cc:237) |   | media::SinkInputPin::Receive(IMediaSample *)
  | 0x00007ff811d886df | (obs-virtualcam-module64.dll + 0x000086df) |   |  
  | 0x00007ff811d8a7a1 | (obs-virtualcam-module64.dll + 0x0000a7a1) |   |  
  | 0x00007ff811d8a077 | (obs-virtualcam-module64.dll + 0x0000a077) |   |  
  | 0x00007ff811d8a21e | (obs-virtualcam-module64.dll + 0x0000a21e) |   |  
  | 0x00007ff811d89830 | (obs-virtualcam-module64.dll + 0x00009830) |   |  
  | 0x00007ff811df575f | (obs-virtualcam-module64.dll + 0x0007575f) |   |  
  | 0x00007ff8962154df | (KERNEL32.DLL + 0x000154df) |   |  
  | 0x00007ff89716485a | (ntdll.dll + 0x0000485a) |   |  
  | 0x00007ff89482208f | (KERNELBASE.dll + 0x0014208f)

Second thread:

0x00007ff897203354 | (ntdll.dll + 0x000a3354) |   |  
-- | -- | -- | --
  | 0x00007ff89472292d | (KERNELBASE.dll + 0x0004292d) |   |  
  | 0x00007ff811d8cdf6 | (obs-virtualcam-module64.dll + 0x0000cdf6) |   |  
  | 0x00007ff89472b3fa | (KERNELBASE.dll + 0x0004b3fa) |   |  
  | 0x00007ff811d89dd1 | (obs-virtualcam-module64.dll + 0x00009dd1) |   |  
  | 0x00007ff811d89ea3 | (obs-virtualcam-module64.dll + 0x00009ea3) |   |  
  | 0x00007ff811d88222 | (obs-virtualcam-module64.dll + 0x00008222) |   |  
  | 0x00007ff80cc6588a | (chrome.dll -video_capture_device_win.cc:344) |   | media::VideoCaptureDeviceWin::~VideoCaptureDeviceWin()
  | 0x00007ff80cc66dbf | (chrome.dll -video_capture_device_win.cc:318) |   | media::VideoCaptureDeviceWin::~VideoCaptureDeviceWin
  | 0x00007ff80739ba67 | (chrome.dll -allocator_traits.h:320) |   | std::__1::allocator_traits<std::__1::allocator<std::__1::__tree_node<std::__1::__value_type<std::__1::string,video_capture::DeviceFactoryMediaToMojoAdapter::ActiveDeviceEntry>,void *> > >::destroy<std::__1::pair<const std::__1::string,video_capture::DeviceFactoryMediaToMojoAdapter::ActiveDeviceEntry>,void,void>
  | 0x00007ff80b38f6a3 | (chrome.dll -__tree:2447) |   | std::__1::__tree<std::__1::__value_type<std::__1::string,video_capture::DeviceFactoryMediaToMojoAdapter::ActiveDeviceEntry>,std::__1::__map_value_compare<std::__1::string,std::__1::__value_type<std::__1::string,video_capture::DeviceFactoryMediaToMojoAdapter::ActiveDeviceEntry>,std::__1::less<std::__1::string>,1>,std::__1::allocator<std::__1::__value_type<std::__1::string,video_capture::DeviceFactoryMediaToMojoAdapter::ActiveDeviceEntry> > >::__erase_unique<std::__1::string>
  | 0x00007ff805f4b2c8 | (chrome.dll -interface_endpoint_client.cc:685) |   | mojo::InterfaceEndpointClient::NotifyError(absl::optional<mojo::DisconnectReason> const &)
  | 0x00007ff80a2e6cc2 | (chrome.dll -multiplex_router.cc:930) |   | mojo::internal::MultiplexRouter::ProcessTasks
  | 0x00007ff80a2e636a | (chrome.dll -bind_internal.h:690) |   | base::internal::Invoker<base::internal::BindState<void (mojo::internal::MultiplexRouter::*)(bool),base::internal::UnretainedWrapper<mojo::internal::MultiplexRouter>,bool>,void ()>::RunOnce
  | 0x00007ff8076725f7 | (chrome.dll -bind_internal.h:706) |   | base::internal::Invoker<base::internal::BindState<void (mojo::Connector::*)(unsigned int),base::internal::UnretainedWrapper<mojo::Connector> >,void (unsigned int)>::Run
  | 0x00007ff80a3301ab | (chrome.dll -bind_internal.h:690) |   | base::internal::Invoker<base::internal::BindState<void (mojo::SimpleWatcher::*)(int, unsigned int, const mojo::HandleSignalsState &),base::WeakPtr<mojo::SimpleWatcher>,int,unsigned int,mojo::HandleSignalsState>,void ()>::RunOnce
  | 0x00007ff805a454b3 | (chrome.dll -task_annotator.cc:178) |   | base::TaskAnnotator::RunTask(char const *,base::PendingTask *)
  | 0x00007ff809cf85e6 | (chrome.dll -thread_controller_with_message_pump_impl.cc:260) |   | base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::DoWork()
  | 0x00007ff80781f6c8 | (chrome.dll -message_pump_default.cc:39) |   | base::MessagePumpDefault::Run(base::MessagePump::Delegate *)
  | 0x00007ff806259222 | (chrome.dll -thread_controller_with_message_pump_impl.cc:467) |   | base::sequence_manager::internal::ThreadControllerWithMessagePumpImpl::Run(bool,base::TimeDelta)
  | 0x00007ff806417c85 | (chrome.dll -run_loop.cc:134) |   | base::RunLoop::Run(base::Location const &)
  | 0x00007ff808b65e46 | (chrome.dll -utility_main.cc:235) |   | content::UtilityMain(content::MainFunctionParams const &)
  | 0x00007ff8089c29ef | (chrome.dll -content_main_runner_impl.cc:973) |   | content::ContentMainRunnerImpl::Run(bool)
  | 0x00007ff80764ff52 | (chrome.dll -content_main.cc:418) |   | content::ContentMain(content::ContentMainParams const &)
  | 0x00007ff8084135b1 | (chrome.dll -chrome_main.cc:172) |   | ChromeMain
  | 0x00007ff76c32f3af | (chrome.exe -main_dll_loader_win.cc:169) |   | MainDllLoader::Launch(HINSTANCE__ *,base::TimeTicks)
  | 0x00007ff76c32ef4e | (chrome.exe -chrome_exe_main_win.cc:382) |   | wWinMain
  | 0x00007ff76c394871 | (chrome.exe -exe_common.inl:288) |   | __scrt_common_main_seh
  | 0x00007ff8962154df | (KERNEL32.DLL + 0x000154df) |   |  
  | 0x00007ff89716485a | (ntdll.dll + 0x0000485a)

So, the callback is called on already destroyed/being destroyed instance. In no way can chrome protect against that.

I'm not sure exactly what caused the destruction of the video capture device. The callstack indicates what that's not a regular path.

Still, I suggest OBS software to add some locks and flags which to defensively not call chrome callbacks after it has stopped requesting frames.

@ilyanikolaevskiy
Copy link

Hello, would you please bump the priority of this one? This is a 6th top crash in the utility process in the stable chrome.

@jp9000
Copy link
Member

jp9000 commented Aug 31, 2021

I'll bump the priority on it.

@Fenrirthviti
Copy link
Member

Just to request to anyone watching this thread, this is difficult for us to debug without replication steps. Can someone who is experiencing the issue provide them so we can perform additional debugging on our side? I've spent a little bit of time playing around with various Chrome test/webcam features (enabling/disabling/swapping devices/changing modes/etc.) and I cannot replicate the crash, so there is something missing to what exactly is triggering it.

Are there any specific websites that happen to be causing this crash that can be provided? Is the virtual output active at the time of the crash in OBS, or is this replicated just with the virtual camera device registered to the system?

Distilling down the actual scenarios in which this causes a crash will be helpful.

@Fenrirthviti Fenrirthviti added the Feedback required Issue does not contain enough information or is missing the template label Sep 7, 2021
@ilyanikolaevskiy
Copy link

Sadly, we don't have any reproduction guide. We only see this in automatic crash reports. Seems like the affected users didn't file bug reports to google or chrome.

From the callstack in other threads it looks like some kind of error in ipc communication needs to be happen to trigger the crash in chrome (mojo::InterfaceEndpointClient::NotifyError).

Maybe it's happening on exit from chrome and therefore isn't user visible. No information about Obs configuration is available to our crash reporting systems, so here I also can't say anything.

Could you point me to the code which passes the frames to the Chrome in this repository? I've failed to find it myself.

@Fenrirthviti
Copy link
Member

I suspected as much, but wanted to confirm either way.

Code for our dshow filter is here: https://github.com/obsproject/obs-studio/tree/master/plugins/win-dshow/virtualcam-module

@tylercrompton
Copy link

tylercrompton commented Oct 30, 2021

I believe that issue is affecting me in multiple programs. It kept crashing Firefox Aurora 94.0b9 (at all websites). Some of my crash reports can be found here and here.

VLC seemed to only be able to capture the first frame, with the default DirectShow options (other than changing the video device name of course).

After seeing this GitHub issue, I tried it in Chromium 95.0.4638.54 and it worked flawlessly.

I tried it in Firefox again and didn't encounter any crashes. However, on some sites, the video appeared to be corrupted. See the screenshot below.

Unfortunately, I have no idea how to reproduce the crashes in Firefox. It seems to be quite the Heisenbug.

Edit: Sorry. It seems that there are better issues that I could have reported this in. I'm about to fall asleep at my desk.

corrupted OBS virtual camera stream in Firefox

@notr1ch
Copy link
Member

notr1ch commented Oct 4, 2022

This should be fixed by 0f08432. Please report if you continue to see this behavior in OBS 28 or above.

@notr1ch notr1ch closed this as completed Oct 4, 2022
@RytoEX RytoEX added this to the OBS Studio 28.0 milestone Oct 5, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Crash Report Feedback required Issue does not contain enough information or is missing the template Windows Affects Windows
Projects
None yet
Development

No branches or pull requests

7 participants