Invalid Crumb / Yahoo disables API access to quotes data

[AbhigyanBafna]

Bug Report

Describe the bug

Quote and Quote summary stopped working recently due to invalid crumbs. First encountered it on my NextJS server and was able to reproduce the bug in CLI as well.

Minimal Reproduction

npx yahoo-finance2 quote AAPL

Environment

Node
Node version (if applicable):
v21.6.1
Npm version:
9.5.1

Additional Context

[NarendranKT]

I thought, I was only one to get this error. If someone has an solution to this drop it here. It will be useful for us.

[th3l0g4n]

From the Link in the yahoo error message:
We’re sorry for the inconvenience, but API-level access to Yahoo Finance quotes data has been disabled.

Inofficial API Access might be over...

[gadicc]

I would strongly encourage everyone to go to that link and fill out the form. https://bit.ly/yahoo-finance-api-feedback

This project has always done its best to respect Yahoo's policies and while many efforts have been made to keep up-to-date with their technical requirements, we will never blatantly help to side-step their policies. Both for legal reasons but also out of great respect and appreciation for the data they have been providing us at no cost (!) until now.

The above form is an amazing step by Yahoo to try better understand developers who are using their API. Many serious users of this project (myself included) have looked at alternative (paid) options and found them inferior in many areas. Whereas in the past, Yahoo has mostly just ignored developers using their platform, it now looks like they're looking to seriously engage us and (perhaps) better address our needs.

The possibility of having a official API support from Yahoo is a big deal, and would result in much more stable use without us having to constantly ship updates for the frequent changes they make to their existing and basically internal API. I think having a small free plan for personal use with sensible limits would be amazing, as would a paid option which opens up the possibility for developers to monetize their hard work (alongside any local regulatory requirements).

So, again, I really encourage everyone to complete the form and see where this process leads. As users of this library, you may have enjoyed a reasonably stable experience (if you keep the library up-to-date), but as developers, keeping things working (and working correctly) against a constantly changing service is a huge effort and massive energy drain. Something official from Yahoo would be absolutely incredibly for all involved, so again, I encourage you take advantage of this opportunity and keep your fingers crossed.

[gadicc]

Thanks also to everyone above for reporting the issue, and for your input and feedback. I know this is a huge pain and inconvenience, but I really believe that Yahoo asking for developer feedback rather than just blocking us is a very positive step that could lead somewhere good for all of us.

[programmishka]

From the Link in the yahoo error message: _We’re sorry for the inconvenience, but API-level access to Yahoo Finance

This message from yahoo has been provided since many months. The API was still accessible using the node-yahoo-finance2 library, because it deals with cookies. However, the "Invalid Crumb" error is new, as of last week. The API responded last Tuesday, using the version 2.9.0 of the library.

[programmishka]

I am able to get a positive response from the API
https://query2.finance.yahoo.com/v7/finance/quote?symbols=AAPL&crumb=XXXXXXX
Using Postman, with my A1 cookie set.

[programmishka]

The library (v.2.11.1) works from my console:
npx yahoo-finance2 quote AAPL Storing cookies in /Users/xxxxxx/.yf2-cookies.json Retrieved crumb from cookie store: ELG7IOJ9t.1 { language: 'en-US', region: 'US', ...
For my case I need to find out, why the error "Invalid Crumb" occurs when I use the library (v.2.9.0 or v.2.11.1) in my Google Cloud Function. This problem I have after last Tuesday. But this is not the problem of the library, I think.

[watadarkstar]

Just started getting this error today. Sad to see this library not working anymore due to Yahoo disabling their API. Hopefully they open their API up even if its not free. I'd pay for it if needed.

[almeidapaulooliveira]

Confirmed not working on my side too. Hope Yahoo reopens as a paid service.

[programmishka]

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

[anupanup2001]

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

I tried it. doesnt work for me. The library doesn't even work from the console for me.

npx yahoo-finance2 quote AAPL
Storing cookies in /home/neptune/.yf2-cookies.json
Retrieved crumb from cookie store: yRG6Yj2wkbt
Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

[Nathan-Franck]

Welp, I filled out the form... Back to Selenium I go 😔

If anybody has any other simple stock/crypto price feeds you know about, let me know

[Yhprum]

I would strongly encourage everyone to go to that link and fill out the form.

This form first appeared last year around May or so, so it's definitely not a new thing they just made and set out - but who knows about any new API since the new Yahoo Finance interface is public now.

I assume this is just a different cookie/crumb/useragent thing to fix - for me it's working fine on my deployed site but not in dev (maybe because prod has an older crumb?) so I'll be looking into the issue on my end as well.

[watadarkstar]

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

Can you explain this in more detail? Specifically how you retrieved a crumb?

[Finalet]

Getting same issues. Additionally, yahooFinance.options also returns Error: Invalid Crumb

[mnsrulz]

https://github.com/achannarasappa/ticker this package is still standing strong. It's not exactly the same but provide stock relation info using yahoo finance api's only. Any effort to incorporate the implementation detail from that package into this one?

[nocodehummel]

I just ran the tests and all test cases passed. Does it actually access the Yahoo API for any request in DEVEL mode?

[vmetnev]

As mentioned above, the library works from the console. I've modified my Google Cloud function to use axios and make direct requests to the quote API. The API works fine. You need to retrieve a crumb using your YF cookie and use it as an API parameter.

Can you provide the code for you solution? It would be very much appreciated!

[nocodehummel]

When removing some cached results the test cases are passed. Access to the API is working with fetching new results in the test cases.

[programmishka]

Can you explain this in more detail? Specifically how you retrieved a crumb?

Using the URL
https://query2.finance.yahoo.com/v1/test/getcrumb
You need your cookie for YF, named A1 or A3 for both - getcrumb and quote requests (they expire in 6 months, so need to update them sometimes). For the quote request you need to set the crumb as an URL parameter:
https://query2.finance.yahoo.com/v7/finance/quote?crumb=XXXXX

I've just analyzed the network activity when searching for AAPL on the YF site and made the same requests. The API works.

UPD: for getcrumb request you also need to set the User-Agent header, otherwise it will respond with "too many requests".

[nocodehummel]

@programmishka in my logs i see that the crumb is retrieved which indicates that getCrumb is working.

[gadicc]

Hey all. Firstly, thanks everyone for chiming in. And wow, I personally had never seen the form until now, so thanks for letting me know it's been around for some time.

A few comments:

1. Also confirming that my old cookie and crumb, stored by the CLI in ~/.yf2-cookies.json, works great, but if I delete that file, I get the invalid crumb error for the newly retrieved cookie / crumb.

I guess that's encouraging that probably this issue could be worked around, but a) I unfortunately don't have any time to look at this now, and b) we need to be very careful of any legal implications (I'm thinking at least, requiring an extra step from the developer to acknowledge the situation with a link to yahoo's form).

2. @nocodehummel, as you noticed, all the tests used local disk cache for speed, however, if you set env var FETCH_DEVEL="nocache" it will instead make fresh network requests (link to docs). Note, this is very slow, can sometimes trigger a yahoo rate limiting ban (although I think we added our own rate-limiting code to avoid such bans), and also, a lot of tests that pass in the cache can sometimes fail when they added/removed fields we didn't accommodate yet.

Anyway, let's see what we can do as a community. As I said, I won't have a chance to investigate this now, but maybe others in the community might have some luck in the meantime. Thanks again to everyone above who has posted their findings so far - you guys are awesome! - and also to all users for your understanding of the situation and our stance.

P.S. Probably quickest workaround would be to access and accept all terms from your web browser, and then for us to have a good way to import the cookie/crumb for yf2. With the appropriate warnings, acknowledgements and links I mentioned above. Lastly, one big thanks again to this awesome community! 🙏 ❤️

[nocodehummel]

@gadicc thanks for the feedback.
The tests are also passed when deleting a cached file (quoteSummary-summaryProfile-AAPL.json).
The YF API returns the data and it is saved again including the content, but formatted with escaped quotes.
How can the tests be passed when the actual API is not working with invalid crumb?

[gadicc]

@nocodehummel, my pleasure, thanks for your interest in this :)

1. Ah yes, true, deleting a specific test file will re-fetch just for that test.

2. The fact that it's saving as quoted text is mildly worrying, it probably means that Yahoo isn't sending the correct content-type header, but I'd need to look through the code again to be sure.

How can the tests be passed when the actual API is not working with invalid crumb?

3. That sounds impossible `:) In short, I don't have an answer at this stage, until I'll have a chance to look at in more detail :/

[nocodehummel]

Some modules (endpoints) are actually working (e.g. calendar and time-series data), while others are not (e.g. quoteSummary).

3. my assumption how it is possible that all test passed, is because the cached crumb is valid.

[gadicc]

Ah yeah sorry, I thought you meant for new fetches (non-cached). Thanks.

Quick update: had a quick chance to look at this now. Using their /v1/test/getcrumb endpoint we can actually get a different crumb that is valid. It's a slightly different flow than what we used before. I'm on track but ran out of time, presumably I can get something working later this week, BUT, it requires changing the User-Agent, which goes against the guiding policies of this library, so, I need to think more about this.

[NarendranKT]

Can you explain this in more detail? Specifically how you retrieved a crumb?

Using the URL https://query2.finance.yahoo.com/v1/test/getcrumb You need your cookie for YF, named A1 or A3 for both - getcrumb and quote requests (they expire in 6 months, so need to update them sometimes). For the quote request you need to set the crumb as an URL parameter: https://query2.finance.yahoo.com/v7/finance/quote?crumb=XXXXX

I've just analyzed the network activity when searching for AAPL on the YF site and made the same requests. The API works.

UPD: for getcrumb request you also need to set the User-Agent header, otherwise it will respond with "too many requests".

Thanks for the help. After tons of errors finally I got an response as a text and finally I parsed it as JSON.

[gadicc]

Hi all, a fix is building now and there will be an automated message here when the release is published.

Thanks to @AbhigyanBafna who first reported the issue and @nocodehummel, @programmishka, @Yhprum who helped to investigate and work towards a fix.

The endpoint does work with Mozilla/5.0 (compatible; yahoo-finance2/0.0.1).

@nocodehummel, this is a great find! And a great compromise that I'm happy to publish. I wasn't willing to ship something that falsified the user-agent, and had intended to provide some explanations and general directions on manually overriding the user-agent by hand. So, on behalf of all our users, thanks for figuring out something that works out the box!

Thanks again to everyone who contributed above. The magic of open source remains awesome and keeps helping everyone 🙏 ❤️

[gadicc]

🎉 This issue has been resolved in version 2.11.2 🎉

The release is available on:

• npm package (@latest dist-tag)
• GitHub release

Your semantic-release bot 📦🚀

[miliu99]

It appears it's broken again since this morning (5/30/2024):

Storing cookies in C:\Users\mingl\.yf2-cookies.json
Retrieved crumb from cookie store: yRG6Yj2wkbt
Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

[DDushkin]

It appears it's broken again since this morning (5/30/2024):

Storing cookies in C:\Users\mingl\.yf2-cookies.json

Retrieved crumb from cookie store: yRG6Yj2wkbt

Exiting with Error: Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback

Try upgrade to the latest version, it helped me

[miliu99]

OK, the latest version (2.11.3) works. I was using 2.11.2 and it worked until yesterday. Thank you very much, DDushkin.

[raalarcon9705]

version 2.11.3 is not working for me

[Fleker]

I updated the package in the package.json, npm install'd, and now it does seem to work.

[miliu99]

Yes, it started working for me again after I upgraded from 2.11.2 to 2.11.3 in package.json.

However, I just realized I'm still getting the same error when running command line npx yahoo-finance2 quote AAPL. Even with npx [email protected] quote AAPL, and it asks me to confirm to install the package first, it still gave me the same error. Any body can explain why?

[gadicc]

Thanks everyone for your comments and helping each other out 🙏

@miliu99, you might have a bad cookie stored in your .yf2-cookies.json file (used only by the CLI) from the old version. Can you try deleting this file and run the command again and see if it works then?

[miliu99]

After deleting .yf2-cookies.json, I got this:

> npx yahoo-finance2 quote AAPL
Storing cookies in C:\Users\mingl\.yf2-cookies.json
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
Exiting with Error: Unsupported redirect to https://finance.yahoo.com/quote/AAPL/, please report.

[gadicc]

@miliu99 are you sure you're definitely on 2.11.3 ? Because this is exactly what we fixed in that release, #777... actually the string of the error you showed doesn't even exist in this release 😅

what does the version field say in node_modules/yahoo-finance2/package.json ? If you only upped the version in your project's package.json you still need to tell your package manager to do the update.

[miliu99]

I just took a closer look at this...
Strangely, I found two folders in C:\Users\{UserName}\AppData\Local\npm-cache\_npx\ for yahoo-finance2, one is version "2.11.1" (installed 4/15/2024) and another "2.11.3" (installed 5/30/2024). In this state, I still get same error. So it appears it's still using the cache for 2.11.1. I then manually deleted the folder for 2.11.1, run the command again, it reinstalls yahoo-finance2 again (I can see 2.11.3 is already in cache, why reinstall?) and it started working as shown here:

>npx yahoo-finance2 quote AAPL
Need to install the following packages:
  [email protected]
Ok to proceed? (y) y
Storing cookies in C:\Users\mingl\.yf2-cookies.json
Please consider completing the survey at https://bit.ly/yahoo-finance-api-feedback if you haven't already; for more info see https://github.com/gadicc/node-yahoo-finance2/issues/764#issuecomment-2056623851.
Fetching crumb and cookies from https://finance.yahoo.com/quote/AAPL...
We expected a redirect to guce.yahoo.com, but got https://finance.yahoo.com/quote/AAPL/
We'll try to continue anyway - you can safely ignore this if the request succeeds
Success. Cookie expires on Sun Jun 01 2025 16:43:27 GMT-0500 (Central Daylight Time)
fetch https://query1.finance.yahoo.com/v1/test/getcrumb
New crumb: MKI8vq023i9
{
  language: 'en-US',
  region: 'US',
  quoteType: 'EQUITY',
...

When I check the folder at C:\Users\{UserName}\AppData\Local\npm-cache\_npx\ again and see a new folders for yahoo-finance2 (installed today). So, somehow, the one installed on 5/30/2024 is not good but I couldn't see anything wrong when I compare two folders.

[AbhigyanBafna]

@miliu99 What about performing a clean install of yahoo-finance2 entirely with the latest flag?

npm i yahoo-finance2@latest

Or if you are using this in a project, you could just modify the package.json -

"yahoo-finance2": "^2.11.3"

[miliu99]

@AbhigyanBafna As I mentioned earlier, it works fine in my project, but I was having trouble with npx command. Thanks.