Demuxed audio stream off sync with video stream on longer duration Hap video files in Windows #2013
Comments
|
Hi @jsquared90, Thanks for reporting this - which Video API are you using? I've just tried here and see fairly consistent frame drops using your video with Media Foundation (will look into separately) but DirectShow seemed more consistent. Cheers, |
|
Hey there. My initial tests were with Media Foundation. Since then I have conducted more tests and have found success with A/V sync with legacy Hap formats using DirectShow as well. However, Hap-R failed to render anything using DirectShow. I also found that if I have the "Renderer Sync" setting disabled, I can get successful A/V sync when set to the Media Foundation API. However, I have found other issues, including A/V sync issues with NotchLC and all sorts of issues when using DX12 (all my success has been with DX11). I sent a separate email to you all to see how you recommend my correspondence on this platform of my findings. I wasn't sure how much you feel I should include in this topic and how much and when I should generate a new topic. Thx! |
|
After more testing, I discovered I must have been mistaken about the note regarding disabling "Renderer Sync" providing a solution to the issue. However, I am able to resolve the issue by using the Direct Show API. I learned that I just needed to install the latest LAV filters in order to get HapR to work using Direct Show. So I have a fully working solution that involves switching to the Direct Show API from the Media Foundation API (via code) for any Hap playback. |
|
Hey Renderheads,
Sigh… I’m afraid I am still not out of the woods on this one. I had devised a solution where Media Player game objects would be set to use the DirectShow API when there was presence of an audio track in the file, and Media Foundation when there was not. That allowed me to work around the A/V sync issue I had found while still using the Media Foundation API whenever audio was not involved. However, I am experiencing some major computational cost when instantiating game objects using the DirectShow API. The app I am building is founded on many stacked video layers and there is a significant difference in performance when there is a DirectShow layer involved. I’m afraid it is to the point where my solution is not viable.
Do you all have any ideas/clues to this mystery A/V sync issue? Any strategies in the works to correct for in a future version? I am looking to plan accordingly as I will have another usage of my app in January that I need to prepare for. Otherwise - do you have any other helpful hints for me?
I truly do appreciate all the support you have provided.
Thanks,
JJ Myers
Pixel Mosaic | Director
T: +1 (312) 481-7979
M: +1 (847) 924-7983
***@***.*** ***@***.***> | www.pixelmosaic.com <http://www.pixelmosaic.com/>
2415 W. 19th St., Suite 2B Chicago, IL 60608
… On Oct 18, 2024, at 6:58 AM, Chris-RH ***@***.***> wrote:
Closed #2013 <#2013> as completed.
—
Reply to this email directly, view it on GitHub <#2013 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRKV4FOMTG7N3SGSOTQCCLZ4DZXBAVCNFSM6AAAAABO22PFNGVHI2DSMVQWIX3LMV45UABCJFZXG5LFIV3GK3TUJZXXI2LGNFRWC5DJN5XDWMJUG4YTSMRRGU3TANA>.
You are receiving this because you were mentioned.
|
|
Hi @jsquared90, That's a shame - yeah, we're aware of DirectShow blocking the main thread whilst initialising and I'm actually in the process of rewriting the plugin, which includes shifting all video devices to their own threads for this reason. I'm presuming due to the setup of your app there's no way to move the initialisation (/stall) elsewhere, leaving paused then playing at the appropriate point, in which case yeah, a bit of a conundrum! We're pretty slammed at the moment and the progress of the rewrite is contigent on a few factors which make it hard to guarantee it'll be ready for Janurary - although there might be the possibility of a pre-release version. In the meantime I will try and spend some time looking further into what might be causing the delay in MF, in case there are some quick wins.. Cheers, |
|
Sounds good. Thanks for the prompt and thorough response!
JJ Myers
Pixel Mosaic | Director
T: +1 (312) 481-7979
M: +1 (847) 924-7983
***@***.*** ***@***.***> | www.pixelmosaic.com <http://www.pixelmosaic.com/>
2415 W. 19th St., Suite 2B Chicago, IL 60608
… On Oct 24, 2024, at 6:57 AM, Richard Turnbull ***@***.***> wrote:
Hi @jsquared90 <https://github.com/jsquared90>,
That's a shame - yeah, we're aware of DirectShow blocking the main thread whilst initialising and I'm actually in the process of rewriting the plugin, which includes shifting all video devices to their own threads for this reason.
I'm presuming due to the setup of your app there's no way to move the initialisation (/stall) elsewhere, leaving paused then playing at the appropriate point, in which case yeah, a bit of a conundrum!
We're pretty slammed at the moment and the progress of the rewrite is contigent on a few factors which make it hard to guarantee it'll be ready for Janurary - although there might be the possibility of a pre-release version.
In the meantime I will try and spend some time looking further into what might be causing the delay in MF, in case there are some quick wins..
Cheers,
—
Reply to this email directly, view it on GitHub <#2013 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRKV4CBS2UZAHXJH4UBGDDZ5DOAZAVCNFSM6AAAAABO22PFNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVGA3TINBYGQ>.
You are receiving this because you were mentioned.
|
|
One quick thing to check @jsquared90 - with 'Media Foundation API Options' do you have 'Audio Output' set to 'System' or 'Unity'? As I'd expect 'System' to have less latency (and mirrors how DirectShow works) Also, do you get any different results if you increase 'Parallel Frame Count' under 'Use Hap/NotchLC'? |
|
I have only been using the ’System’ setting. I experimented with the ‘Unity’ setting briefly, but could not get any sound output results. I could not ascertain enough from the documentation exactly how that works. I wasn’t sure if I add the component directly to the game object that also has Media Player, or a child component, etc… And then there is the matter of linkage between the various components. I’m sure I could figure it out though, if you think it’s worth a shot. Though, it sounds like from your reply you might expect the opposite. Any further thoughts on that would be greatly appreciated.
Thanks,
JJ Myers
Pixel Mosaic | Director
T: +1 (312) 481-7979
M: +1 (847) 924-7983
***@***.*** ***@***.***> | www.pixelmosaic.com <http://www.pixelmosaic.com/>
2415 W. 19th St., Suite 2B Chicago, IL 60608
… On Oct 24, 2024, at 9:20 AM, Richard Turnbull ***@***.***> wrote:
One quick thing to check @jsquared90 <https://github.com/jsquared90> - with 'Media Foundation API Options' do you have 'Audio Output' set to 'System' or 'Unity'? As I'd expect 'System' to have less latency (and mirrors how DirectShow works)
—
Reply to this email directly, view it on GitHub <#2013 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACRKV4EO7MOWJZJAYD7W4TTZ5D63JAVCNFSM6AAAAABO22PFNGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMZVGQZTCNZWGU>.
You are receiving this because you were mentioned.
|
|
Hi @jsquared90, The 'Unity' audio setting only works with Media Foundation and you should be able to see it set up in our 'Demo_MediaPlayer' - although I don't think it's worth visiting at this juncture as 'System' is most likely to be the best option here. Apologies, I later updated the post but presume you only saw the original by email - note the addition: "Also, do you get any different results if you increase 'Parallel Frame Count' under 'Use Hap/NotchLC'?" Cheers, |
The demuxed audio stream drifts out of sync from the video stream on Hap videos with a duration over 3 minutes. Does not occur in MacOS. Only occurs in Windows (tested in Windows 10).
Load AVPro with the following video file and watch lip sync to the end:
https://www.dropbox.com/scl/fi/2p0pd1r5pgwdvh69q7hjm/davita_tlm_service_091824_7.mov?rlkey=oicn9q7f3a0uigdgzfnqzg4jf&dl=0
Within the final 30 seconds, A/V sync should appear off by several frames.
The text was updated successfully, but these errors were encountered: