Investigate why success rate gets worse over time

[lidel]

Restarting biforst-gateway on staging produces very close success rate, but over time, it erodes into worse and worse state:

Inspect yourself:

• overview@stg board
• detailed bifrost-gw staging

Summary from the latter:

Some ideas/thoughts why:

• in-memory block cache perf regression is unlikely, cache size is symbolic, aims to limit roundtrips per requests.
  Staging runs with BLOCK_CACHE_SIZE=16k (Adjust size of in-memory block cache #47 (comment)) and the slowness will happens way after that is filled up multiple times, and we see on the next graph the duration increase of CAR fetch happens on Caboose side:

  
  

• Saturn L1 pool health gets worse for some reason:

  

• Saturn per-L1 CAR fetch durations increase while other durations stay the same:

  

• HTTP 499s suggests clients giving up before they get our response, which is aligned with things getting slower over time, and more and more clients giving up waiting for response. This is not specific to Rhea, the old mirrored node that runs Kubo is also seeing more 499s over time, but it is less prominent:

  

Any feedback / thoughts / hypothesis are welcome. 🙏

[lidel]

fyi the problem is still present -- after a reboot it performs well, then, after ~1h things get visibly worse (499s are gone for 1h, and then back, very weird):

[lidel]

Ok, got bit more clarity. Works fine for a while, then erodes, and we see CPUs being capped when it happens:

Looks like a bug either in GRAPH_BACKEND=true or the new Caboose -- non-staging boxes run older version with graph backend disabled and have no CPU issues:

[aarshkshah1992]

@lidel The new Caboose wasn't there when we reported this issue two weeks back. So I'd be more suspicious of something going wrong with GRAPH_BACKEND=true

[lidel]

I've restarted staging with GRAPH_BACKEND=false. If we don't see any regression after ~4h, then we know it can be safely deployed to prod.

[lidel]

GRAPH_BACKEND=false looks solid, was stable on staging for long enough to confirm the problem is limited to GRAPH_BACKEND=true:

[BigLep]

Related PR: #102

[lidel]

Just to wrap this up, tldr

The fix removed CPU load on staging:

and we've had it running for over 12h without the usual success rate going down after ~2h mark: