Not working for stocks or options since last saturday, Jan 27 2024. Have the latest version. 0.3.2.

[schittor]

Getting this message
401 Client Error: Unauthorized for url: https://query2.finance.yahoo.com/v7/finance/options/TICKER

Looks like that error happened in past but fixed.

[sbraitti]

Yahoo api has changed and now requires "Crumb".

{"finance":{"result":null,"error":{"code":"Unauthorized","description":"Invalid Crumb. For Developers - https://bit.ly/yahoo-finance-api-feedback"}}}

Full explanation here: scheb/yahoo-finance-api#44

Is there anyone working on that already?

[mcdallas]

You need to get a crumb from /getcrumb for which you need a cookie which I’m not sure where it’s coming from. Probably some obfuscated javascript

[bradacsa]

Okay, I just started to write an API in C++ and libcurl. The Brave inspection gave me these:
Steps:

1. ---

Request URL: https://finance.yahoo.com/chart/ZURN.SW
Request Method: GET
Status Code: 307 Temporary Redirect
Remote Address: 87.248.119.252:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

:authority: finance.yahoo.com
:method: GET
:path: /chart/ZURN.SW
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Cache-Control: no-store
Content-Language: en
Content-Length: 0
Content-Type: text/html; charset=utf-8
Date: Mon, 26 Feb 2024 22:06:40 GMT
Expect-Ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
Location: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=d2MLOAk&done=https%3A%2F%2Ffinance.yahoo.com%2Fchart%2FZURN.SW
Server: ATS
Set-Cookie: GUCS=AXdjCzgJ; Max-Age=1800; Domain=.yahoo.com; Path=/; Secure
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Xss-Protection: 1; mode=block

2. ---

Request URL: https://guce.yahoo.com/consent?brandType=nonEu&gcrumb=d2MLOAk&done=https%3A%2F%2Ffinance.yahoo.com%2Fchart%2FZURN.SW
Request Method: GET
Status Code: 302 Found
Remote Address: 34.242.51.9:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Connection: keep-alive
Cookie: GUCS=AXdjCzgJ
Host: guce.yahoo.com
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Connection: keep-alive
Content-Length: 0
Date: Mon, 26 Feb 2024 22:06:41 GMT
Location: https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c
Server: guce
Strict-Transport-Security: max-age=31536000; includeSubDomains

3. ---

Request URL: https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c
Request Method: GET
Status Code: 200 OK
Remote Address: 34.243.184.210:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Connection: keep-alive
Cookie: GUCS=AXdjCzgJ
Host: consent.yahoo.com
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Cache-Control: no-cache, no-store, must-revalidate
Connection: keep-alive
Content-Encoding: gzig
Content-Security-Policy-Report-Only: default-src 'none';
block-all-mixed-content; connect-src 'self';
frame-ancestors 'none';
img-src 'self' https://s.yimg.com;
media-src 'none';
script-src 'self' 'nonce-4iOt96V4jtfS9qna59ePtO+LE+36YEqx' https://s.yimg.com;
style-src 'self' 'nonce-4iOt96V4jtfS9qna59ePtO+LE+36YEqx' https://s.yimg.com;
font-src 'self';
object-src 'none';
frame-src 'none';
report-uri https://csp.yahoo.com/beacon/csp?src=guce
Content-Type: text/html; charset=UTF-8
Date: Mon, 26 Feb 2024 22:06:41 GMT
Expires: 0
Pragma: no-cache
Referrer-Policy: strict-origin-when-cross-origin
Server: guce
Strict-Transport-Security: max-age=31536000; includeSubDomains
Transfer-Encoding: chunked
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Xss-Protection: 1; mode=block

4. ---

Request URL: https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c
Request Method: POST
Status Code: 302 Found
Remote Address: 34.243.184.210:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Cache-Control: max-age=0
Connection: keep-alive
Content-Length: 190
Content-Type: application/x-www-form-urlencoded
Cookie: GUCS=AXdjCzgJ
Host: consent.yahoo.com
Origin: https://consent.yahoo.com
Referer: https://consent.yahoo.com/v2/collectConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-origin
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Connection: keep-alive
Content-Length: 0
Date: Mon, 26 Feb 2024 22:06:41 GMT
Location: https://guce.yahoo.com/copyConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c&lang=hu-HU
Server: guce
Set-Cookie: CFC=AQABCAFl3ldmEEMg8ATe&s=AQAAAJKEgFvP&g=Zd0Lew; Expires=Wed, 26 Feb 2025 10:05:55 GMT; Domain=consent.yahoo.com; Path=/; Secure
Strict-Transport-Security: max-age=31536000; includeSubDomains

Form Data:

csrfToken: d2MLOAk
sessionId: 3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c
originalDoneUrl: https://finance.yahoo.com/chart/ZURN.SW?guccounter=1
namespace: yahoo
reject: reject
reject: reject

5. ---

Request URL: https://guce.yahoo.com/copyConsent?sessionId=3_cc-session_0a35c6d2-d9e1-4d31-af6e-2fa3b396058c&lang=hu-HU
Request Method: GET
Status Code: 302 Found
Remote Address: 34.242.51.9:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Cache-Control: max-age=0
Connection: keep-alive
Cookie: GUCS=AXdjCzgJ
Host: guce.yahoo.com
Referer: https://consent.yahoo.com/
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-site
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Connection: keep-alive
Content-Length: 0
Date: Mon, 26 Feb 2024 22:06:41 GMT
Location https://finance.yahoo.com/chart/ZURN.SW?guccounter=1
Server: guce
Set-Cookie: A1=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk; Expires=Tue, 25 Feb 2025 22:06:41 GMT; Max-Age=31536000; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure; HttpOnly
Set-Cookie: EuConsent=CP6lk0AP6lk0AAOACKHUAoEgAAAAAAAAACiQAAAAAAAA;Version=1;Comment=;Domain=yahoo.com;Path=/;Max-Age=86400
Set-Cookie: GUC=AQABCAFl3lBmCkIg8ATe&s=AQAAAOohMemh&g=Zd0Lew; Expires=Wed, 26 Feb 2025 10:05:55 GMT; Domain=yahoo.com; Path=/; Secure
Set-Cookie: A1S=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk; Domain=.yahoo.com; Path=/; SameSite=Lax; Secure
Strict-Transport-Security: max-age=31536000; includeSubDomains

6. ---

Request URL: https://finance.yahoo.com/chart/ZURN.SW?guccounter=1
Request Method: GET
Status Code: 200 OK
Remote Address: 87.248.119.252:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

:authority: finance.yahoo.com
:method: GET
:path: /chart/ZURN.SW?guccounter=1
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Cache-Control: max-age=0
Cookie: GUCS=AXdjCzgJ;
A1=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk;
EuConsent=CP6lk0AP6lk0AAOACKHUAoEgAAAAAAAAACiQAAAAAAAA;
GUC=AQABCAFl3lBmCkIg8ATe&s=AQAAAOohMemh&g=Zd0Lew;
A1S=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk
Referer: https://consent.yahoo.com/
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: same-site
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Age: 1
Cache-Control: private, no-store, no-cache, max-age=0
Content-Encoding: gzip
Content-Security-Policy: frame-ancestors 'self' https://.yahoo.com https://.engadget.com https://.pnr.ouryahoo.com https://pnr.ouryahoo.com https://.search.aol.com https://.onesearch.com https://.publishing.oath.com https://.aol.com;
sandbox allow-downloads allow-forms allow-modals allow-same-origin allow-scripts allow-popups allow-popups-to-escape-sandbox allow-top-navigation-by-user-activation allow-presentation;
Content-Type: text/html; charset=utf-8
Date: Mon, 26 Feb 2024 22:06:42 GMT
Expect-Ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
Referrer-Policy: no-referrer-when-downgrade
Server: ATS
Set-Cookie: A3=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk; Expires=Wed, 26 Feb 2025 04:06:42 GMT; Max-Age=31557600; Domain=.yahoo.com; Path=/; SameSite=None; Secure; HttpOnly
Strict-Transport-Security: max-age=31536000
Vary: Accept-Encoding
X-Content-Type-Options: nosniff
X-Envoy-Decorator-Operation: finance-nodejs--mtls-production-ir2.finance-k8s.svc.yahoo.local:4080/
X-Envoy-Upstream-Service-Time: 254
X-Xss-Protection: 1; mode=block

7. ---

Request URL: https://query2.finance.yahoo.com/v1/test/getcrumb
Request Method: GET
Status Code: 200 OK
Remote Address: 87.248.119.252:443
Referrer Policy: strict-origin-when-cross-origin

Request header:

:authority: query2.finance.yahoo.com
:method: GET
:path: /v1/test/getcrumb
:scheme: https
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8
Accept-Encoding: gzip, deflate, br
Accept-Language: hu-HU,hu;q=0.5
Cookie: A1=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk; EuConsent=CP6lk0AP6lk0AAOACKHUAoEgAAAAAAAAACiQAAAAAAAA; GUC=AQABCAFl3lBmCkIg8ATe&s=AQAAAOohMemh&g=Zd0Lew;
A1S=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk; A3=d=AQABBHEL3WUCEH9T8cLfRX5gIdOXm64u-3YFEgABCAFQ3mUKZutMb2UBAiAAAAcIcQvdZbRaMF4&S=AQAAAuJxIlHe51IrNvLJY4nesIk
Sec-Ch-Ua: "Chromium";v="122", "Not(A:Brand";v="24", "Brave";v="122"
Sec-Ch-Ua-Mobile: ?0
Sec-Ch-Ua-Platform: "Windows"
Sec-Fetch-Dest: document
Sec-Fetch-Mode: navigate
Sec-Fetch-Site: none
Sec-Fetch-User: ?1
Sec-Gpc: 1
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Response header:

Age: 0
Cache-Control: private
Content-Type: text/plain;charset=utf-8
Date: Mon, 26 Feb 2024 23:36:22 GMT
Expect-Ct: max-age=31536000, report-uri="http://csp.yahoo.com/beacon/csp?src=yahoocom-expect-ct-report-only"
Referrer-Policy: no-referrer-when-downgrade
Server: ATS
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Envoy-Decorator-Operation: finance-yql--mtls-default-production-ir2.finance-k8s.svc.yahoo.local:4080/*
X-Envoy-Upstream-Service-Time: 1
X-Frame-Options: SAMEORIGIN
X-Xss-Protection: 1; mode=block

Response:

Rg/wiUEL7kh <- This is the crumb.

At step 4. there is a POST. Basically here you should get a page where you can click on accept when you are trying to open yahoo finance at the first time. If you open a chart this POST will accept the agreement for you and then the process goes on with and other redirect where you get A1 cookie and so on.
The problem is when I POST the request nothing happens and my curlEasy object is just running on timeout (1 minute was the longest try). Every setup is fine. The POST puts the cookie(s) to the header what I get before with the other GETs. If I don't use 'Content-Length' in the header it runs on 400 - Bad request. I'm trying to figuring out what happens in the background (maybe the requests are tracked in the server, get GUCS, and then a sessionId, then there is a acception from server side, then the user agreement).
Can anybody reproduce the same thing with CURL?
Do you have any experiences?

---

Edit:
I could step one forward. It turned out that the problem was with a libcurl setopt parameter, but I got stuck again at step 6 and 7.
At step 6 I put everything in together, but I get a 404 error. That's not a problem, because the server sends me the A3 cookie as well, so basically I have everything that I needed for go to step 7 and get a crumb.
After packing everything together and make the GET request I get "429 Too Many Requests" error. Have no idea what happens in the background, I will try it again a few days later. Maybe the server has some limitations and I had a few hundred tries in the last couple days...

---

Edit 2:
It works finally!

The problem was with get crumb is that I just tried to give to much parameters in the header. The only thing you will need is the user-agent ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36 Edg/119.0.0.0" <- this works for me, first I use "Chrome/122.0.6261.69" only, but this is not enough for the crumb to get) and the cookies (attached all of them that I got in the process because I collected them into one std::vector). This will give a crumb that could use in the "https://query1.finance.yahoo.com/v7/finance/" request.

So the full process was:

1. Go to https://finance.yahoo.com/chart/ZURN.SW - GET. Take cookie "GUCS" and the redirection URL from the response header. You will need "gcrumb" value from the redirection URL.

2. Go the redirected URL from step 1 with attached GUCS - GET. This will gives you and other redirection URL that contains your sessionId. You will need this too.

3. Go to the redirection URL from step 2 with attached GUCS - GET. I'm not sure what's happening here, maybe some server background thingy...

4. Go to the redirection URL from step 2 with attached GUCS - POST!!!
   Form data will be:

• csrfToken -> gcrumb value what you got in step 1.
• sessionId -> sessionId what you got in step 2.
• originalDoneUrl -> [whatever URL you used in the first step]?guccounter=1
• namespace -> yahoo
• reject -> reject
• reject -> reject

The last two is a bit silly for me, but Brave sended this, so I did the same thing.
Two headers are really important here:

• Content-Length: this will be the full length of the formdata. 'originalDoneUrl' should be urlencoded!
• Content-Type: application/x-www-form-urlencoded
  At POST you will get an other redirection URL + a cookie called "CFC". Not sure we will need this, I collected it.

5. Go to the redirection URL from step 4 - GET. You will get here cookies "A1", "EuConsent" (maybe this is locationdependent), "GUC", "A1S". Redirection URL - again...

6. Go to the redirection URL from step 5 - GET. This will gives you 404, but no problem. You will get an "A3" cookie as well. Take it, love it.

7. Go to https://query2.finance.yahoo.com/v1/test/getcrumb with all the collected cookies attached. I used a few headers until this (Accept, Accept-Language, Accept-Encoding, User-Agent, Cache-Control), but here I used User-Agent with the value I wrote before.You will get crumb value as your content. Save this.

8. With all the cookies attached you could get data from "https://query1.finance.yahoo.com/v7/finance/" or whatever you wanted to use. Don't forget to write the crumb value into the URL. Cookies will be expired around in 2 or 3 days I think so you should do this again in every few days.

[mcdallas]

Thanks @bradacsa I’ll give it a go over the weekend, the other option I’m considering is adding a dependency on https://github.com/ranaroussi/yfinance since they seem to have solved it already

[bradacsa]

Thanks @bradacsa I’ll give it a go over the weekend, the other option I’m considering is adding a dependency on https://github.com/ranaroussi/yfinance since they seem to have solved it already

As I see they did the same thing in data.py.

[mcdallas]

I've published a new release v0.4.0 that uses https://github.com/ranaroussi/yfinance to handle all the cookie magic and fetch the data. Please let me know if there are still any issues

[schittor]

Great ! Thanks, Will try in a day or two and will let you know it is working fine. Suresh Yahoo Mail: Search, Organize, Conquer On Sat, Mar 9, 2024 at 2:51 PM, Mike ***@***.***> wrote: I've published a new release v0.4.0 that uses https://github.com/ranaroussi/yfinance to handle all the cookie magic and fetch the data. Please let me know if there are still any issues — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>

[schittor]

0.4.0 Working !! I use it to download stock and options information including greeks to manage my investments. Had temporarily been using Yahoofin, but like wallstreet. It is a little bit slow in getting first option price requestvs yahoofin, but I can get all the information on options I need.Thanks for fixing. On Saturday, March 9, 2024 at 03:54:47 PM PST, suresh chittor ***@***.***> wrote: Great ! Thanks, Will try in a day or two and will let you know it is working fine. Suresh Yahoo Mail: Search, Organize, Conquer On Sat, Mar 9, 2024 at 2:51 PM, Mike ***@***.***> wrote: I've published a new release v0.4.0 that uses https://github.com/ranaroussi/yfinance to handle all the cookie magic and fetch the data. Please let me know if there are still any issues — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***>