public.api.bsky.app failing to resolve new handles for ~8 hours

[shrink]

Hi, I run Handles.club and as of 8 hours ago, all new handles are failing to resolve via the public.api.bsky.app. I've looked through the codebase to identify where the issue might lie and nothing jumps out.

End users have been reporting the "Invalid Handle" error intermittently for a week-or-so (e.g: bluesky-social/social-app#6338) and it appears to be a symptom of Bluesky being under high-load (i.e: it doesn't have the resources to complete the handle resolution) but since 2024-11-20 01:05:48.27039+00 it has not been possible to resolve any new handles (I have a few dozen examples so far).

The problem appears to be with public.api.bsky.app because earthstar.us-east.host.bsky.network (used when verifying a handle during the change handle process in bsky.app) is capable of resolving the handles, e.g:

GET https://earthstar.us-east.host.bsky.network/xrpc/com.atproto.identity.resolveHandle?handle=person.behind.camera
{"did":"did:plc:7v32gj56c4s4yzoh36zavgz3"}

However, when giving that did to public.api.bsky.app it returns an invalid.handle:

GET https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=did:plc:7v32gj56c4s4yzoh36zavgz3
{"did":"did:plc:7v32gj56c4s4yzoh36zavgz3","handle":"handle.invalid","displayName":...}

Likewise giving the handle to public.api.bsky.app returns an error:

GET https://public.api.bsky.app/xrpc/app.bsky.actor.getProfile?actor=person.behind.camera
{"error":"InvalidRequest","message":"Profile not found"}

Reviewing my request logs for /.well-known/atproto-did I can see that requests are being made when using the Handle change settings (by earthstar.us-east.host.bsky.network) but no requests are being received otherwise, suggesting that public.api.bsky.app is not making any requests to verify handles. I think this explains the invalid.handle issue, i.e: earthstar.us-east.host.bsky.network (used by the frontend of bsky.app) can verify a handle but public.api.bsky.app cannot, so the user interface accepts the handle and then public.api.bsky.app immediately fails to verify it and stores it as invalid.handle.

I guess there's a couple of thoughts / question:

1. Should bsky.app use bsky.network to verify handles in the interface if it's going to use public.api.bsky.app on the backend -- the inconsistency means it is very easy for users to end up with an invalid.handle
2. What is the difference between bsky.network and public.api.bsky.app?
3. What is the configuration used by public.api.bsky.app for caching handles, is it using the redis cache?

Thank you :)

[shrink]

After reading more documentation, I think I have a better understanding of the issue now. I was mistaken about how a new handle is saved, that action appears to go through a relay server too which explains why setting the new handle is accepted.

POST https://earthstar.us-east.host.bsky.network/xrpc/com.atproto.identity.updateHandle
{"handle": "833.on.computer"}

I think this narrows down the source of the problem to be public.api.bsky.app's ability to verify handles. Notably, once a handle becomes invalid.handle it seems to be a permanent state, i.e:

indigo/atproto/identity/handle.go

Lines 166 to 173 in 9198b79

	func (d *BaseDirectory) ResolveHandle(ctx context.Context, handle syntax.Handle) (syntax.DID, error) {
	// TODO: *could* do resolution in parallel, but expecting that sequential is sufficient to start
	var dnsErr error
	var did syntax.DID
	if handle.IsInvalidHandle() {
	return "", fmt.Errorf("invalid handle")
	}

indigo/atproto/identity/cache_directory.go

Lines 94 to 97 in 9198b79

	func (d *CacheDirectory) ResolveHandle(ctx context.Context, h syntax.Handle) (syntax.DID, error) {
	if h.IsInvalidHandle() {
	return "", fmt.Errorf("invalid handle")
	}

I think this means:

1. A user goes to their account settings in bsky.app
2. The user enters their new handle, e.g: 833.on.computer
3. The bsky.network relay instance accepts the new handle as valid and sets it
4. The next time bsky.app synchronises with the relay it receives the new handle, then tries to verify it, but it fails for a reason (e.g: network issues?) and then sets it to invalid.handle
5. bsky.app never tries to resolve the handle again

I think the fix is to remove the if invalid handle condition but I'm unclear if there would be any unintended consequence of doing that.

[shrink]

New handles are resolving again (since around 2024-11-20 14:00 UTC) which puts the total length of the issue at around 11 hours (probably due to these technical difficulties). Notably, handles created during the "outage" have not started resolving which I think confirms that they have ended up in a bad state that cannot be automatically recovered from. I think this leaves an outstanding task of addressing handles that get permanently "stuck" as invalid.handle. I think a user can self-serve fix by "resetting" their handle (change back to @bsky.social, then change to their desired handle again) but I will try to track down the cause and send a Pull Request with a fix.

[ivanbea]

Same case here https://bsky.app/profile/ivanbea.eu stuck with Invalid Handle, I tried several times changing back and forth to .bsky.social with no luck.
The debug page confirms that my DNS is ok https://bsky-debug.app/handle?handle=ivanbea.eu

[ivanbea]

Still with problems, I have changed to random .bsky.social names and all is ok (as you can see on https://clearsky.app/ivanbea.eu), but when I return to @ivanbea.eu (my wanted handle) I can use bsky.app correctly but with invalid handle and problems with other ATproto services like:

https://whtwnd.com/ivanbea.eu/entries/Primers%20passos%20a%20Bluesky
https://blueskyroast.com/roast/ivanbea.eu
https://skylights.my/profile/ivanbea.eu

My DNS records are Ok https://bsky-debug.app/handle?handle=ivanbea.eu

I am out of ideas and I don't want to start again with a new handle, please help @pfrazee