New anti-bot blocker on Youtube prevents to get video manifests

[Darki93]

Version

6.3.16

Platform

.NET 8 / Debian

Steps to reproduce

Query and download many videos for a longer time on the same system with static IP address.

Details

I am executing YoutubeExplode on a Debian server with static IP address.
Since a few days I am encountering the following exception on the GetManifestAsync method:

YoutubeExplode.Exceptions.VideoUnplayableException: Video '' is unplayable. Reason: 'Sign in to confirm you’re not a bot'.
at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(VideoId videoId, PlayerResponse playerResponse, CancellationToken cancellationToken) in //YoutubeExplode/Videos/Streams/StreamClient.cs:line 212
at YoutubeExplode.Videos.Streams.StreamClient.GetStreamInfosAsync(VideoId videoId, CancellationToken cancellationToken) in //YoutubeExplode/Videos/Streams/StreamClient.cs:line 276
at YoutubeExplode.Videos.Streams.StreamClient.GetManifestAsync(VideoId videoId, CancellationToken cancellationToken) in /_/YoutubeExplode/Videos/Streams/StreamClient.cs:line 292

Seems like Youtube added a new mechanism to stop multiple videos to be queried/downloaded from the same machine/IP without logging in.

Checklist

• I have looked through existing issues to make sure that this bug has not been reported before
• I have provided a descriptive title for this issue
• I have made sure that this bug is reproducible on the latest version of the package
• I have provided all the information needed to reproduce this bug as efficiently as possible
• I have sponsored this project
• I have not read any of the above and just checked all the boxes to submit the issue

[Tyrrrz]

Seems like Youtube added a new mechanism to stop multiple videos to be queried/downloaded from the same machine/IP without logging in.

I don't think there's much that you can do if that's the case. Would have to sign in.

[ReenigneArcher]

This was kind of random before... it's extremely repeatable now.

[rpm61]

I am also getting "Video 'AfWnMPDM0G4' is not available." from our Azure servers, noit getting it locally. Is it possible to use proxies with Explode?

[Darki93]

There is a YoutubeClient constructor overload taking a HttpClient as argument, which can be configured to use a proxy.
However, it is probably just a matter of time until the proxy's IP will also be flagged as bot.

[rpm61]

Thank you, should have seen that. Using proxies have solved the issue for me.

[Aimeast]

Maybe you could try bringing your authentication cookies or anti-bot cookies

[ReenigneArcher]

According to youtube-dl, cookies probably won't work for long if they even work at all. ytdl-org/youtube-dl#32905 (comment)

[qtlin]

Is there any way to make an authenticated request to satisfy anti-bot protection?
For the record, the problem is easy to reproduce using any browser private/incognito window to trigger "login to view" message from an IP address blocked by YouTube for any reason. At the same time, regular browser window is allowed by YouTube to view videos from the same IP address. Seems like YouTube views YoutubeExplode request like private/incognito window, at least based on YouTube response. May be above will help to find a remedy.

[Tyrrrz]

Somewhat useful observations:

• CI runs have been consistently failing for over a month now
• Tests that are failing are always the same:
  • All optimistic tests in StreamSpecs fail
  • Most optimistic tests in VideoSpecs fail
    • For some reason, tests for videos qs3NZHVM_Ik and rXMX4YJ7Lks do not fail
• Tests run fine locally most of the time, although I have observed rare failures
• Failure doesn't seem to be dependent on request rate/density, rather on origin IP ranges

[Tyrrrz]

Is there any way to make an authenticated request to satisfy anti-bot protection?

99% answers can usually be found in the readme: https://github.com/Tyrrrz/YoutubeExplode#authentication

[Tyrrrz]

When testing locally, it seems that instrumenting a short delay with no parallelism helped avoid the bot flagging mechanism (see #813). Also, when the client does get flagged, the issue automatically goes away after some time (not sure on the exact duration, but definitely less than an hour).

On CI, however, this did not yield any useful results. It seems that the public GitHub Actions runners are pre-identified as suspicious by YouTube and the corresponding requests are pre-emptively banned. I imagine the same would apply to apps running on other public infrastructure as well.

[qtlin]

Is there any way to make an authenticated request to satisfy anti-bot protection?

99% answers can usually be found in the readme: https://github.com/Tyrrrz/YoutubeExplode#authentication

Thank you very much, I am using an application that calls YoutubeExplode, I'll report your answer there.

[pagdot]

I've switched for a personal project to YoutubeDLSharp with yt-dlp and a youtube oauth login plugin

Hadn't had any issues with it yet

[3052]

original post is vague as hell, to the point of being not actionable - can someone please describe an actual test case? I might have a fix but its impossible to say give the current (lack of) information

[ReenigneArcher]

@3052 personally, I have a jellyfin plugin using this library. This unit tests is failing because of YouTube's anti bot measures. My guess is YouTube is blocking the IPs of GitHub runners.

https://github.com/LizardByte/Themerr-jellyfin/blob/1b5d4d71dd7216655ef9aa81991d0a75d908df4c/Jellyfin.Plugin.Themerr.Tests/TestThemerrManager.cs#L103

Edit: Same issue with yt-dlp (saw your comment there) and youtube-dl with other repos.

[Tyrrrz]

original post is vague as hell, to the point of being not actionable - can someone please describe an actual test case? I might have a fix but its impossible to say give the current (lack of) information

I was able to reproduce the issue by running YoutubeExplode's test suite 2-3 times.

It's also consistently reproducible on CI because the agent's IP/range is presumably banned: https://github.com/Tyrrrz/YoutubeExplode/actions/runs/11024028945

[3052]

OK I will continue my work in private then, good luck

[qtlin]

@Tyrrrz you can consistently reproduce at home/office if you connect to almost any VPN.

[rogerfar]

I tried 6.4.2 and it I'm getting much less 403 errors.

[ReenigneArcher]

For me, the result is the same (everything still blocked in CI) LizardByte/Themerr-jellyfin#546

[Tyrrrz]

6.4.2 didn't do anything about this particular issue, it solved other class of 403 errors

[DestroyerDarkNess]

I don't know if I should create a new issue, but GetMuxedStreams is not working. and after so many attempts it asks me to log in.

GetAudioOnlyStreams works correctly.

[ReenigneArcher]

but GetMuxedStreams is not working

That's documented in the release notes: https://github.com/Tyrrrz/YoutubeExplode/releases/tag/6.4.2

[DestroyerDarkNess]

Oh, you're right, I overlooked that. Thanks for replying.

[nemtajo]

When testing locally, it seems that instrumenting a short delay with no parallelism helped avoid the bot flagging mechanism (see #813). Also, when the client does get flagged, the issue automatically goes away after some time (not sure on the exact duration, but definitely less than an hour).

On CI, however, this did not yield any useful results. It seems that the public GitHub Actions runners are pre-identified as suspicious by YouTube and the corresponding requests are pre-emptively banned. I imagine the same would apply to apps running on other public infrastructure as well.

@Tyrrrz is there a way to disable parallelism when making httpRequests? For my app it is more important to be reliable in downloading than performant. Please let me know if I can contribute in any way to avoid youtube flagging a client as a bot.

[Tyrrrz]

There is no parallelism in HTTP requests unless you explicitly instrument it

[nemtajo]

If my app is a website (in-browser), I can't copy cross-site cookies like you would do with a webview in WPF app.
I have OAuth screen on which a user allows access to their videos, once granted my app gets access token associated with the user:

{
	"access_token": "ACCESSTOKEN",
	"token_type": "Bearer",
	"expires_in": 3595,
	"refresh_token": null,
	"scope": "https://www.googleapis.com/auth/youtube.readonly https://www.googleapis.com/auth/youtube.upload",
	"id_token": null,
	"expireTime": null,
	"Issued": "2024-10-01T19:29:53.1009793+03:00",
	"IssuedUtc": "2024-10-01T16:29:53.1009793Z"
}

Is there any way to use this token with YouTubeExplode client to access user's videos (for download purposes ofcourse)?
My OAuth screen and the whole application flow has been approved by YouTube, so it's all legal.
Do you have any recommendation or idea how to achieve download as already authenticated app?
The last thing I want is to get the IP of the server banned...
As I mentioned, I am happy to contribute to the library if there is a way to achieve it.

[Tyrrrz]

To my knowledge, you can't use that access token. Do note that downloading videos isn't something YouTube wants to support, so whichever way you find to make it possible, it's probably not going to be an official way (which OAuth is).

[noammaoz]

Any idea how to fix this issue ("Please sign in") or do I need to use another library?

[qtlin]

FWIW: Open source https://freetubeapp.io/ plays and downloads videos without any login on the same blacklisted IP where Firefox and other download tools are required to login in order to view.
UPD: spoke too soon. Found some addresses where even freetube does not play but iPhone iOS has no problems without login. Possibly uses previous cookies, needs to be tested with clean client.