[release-2.9.x] Cache: correctly check background cache size (#11656)

Backport 88aaa7d from #11654

---

**What this PR does / why we need it**:
The size check was not being performed atomically, which led to
flakiness in the `TestBackgroundSizeLimit` test.

---------

Co-authored-by: Danny Kopping <[email protected]>

CHANGELOG.md

@@ -7,6 +7,7 @@
 ##### Fixes
 
 * [11601](https://github.com/grafana/loki/pull/11601) **dannykopping** Ruler: Fixed a panic that can be caused by concurrent read-write access of tenant configs when there are a large amount of rules.
+* [11654](https://github.com/grafana/loki/pull/11654) **dannykopping** Cache: atomically check background cache size limit correctly. 
 
 ## 2.9.3 (2023-10-16)
 

pkg/storage/chunk/cache/background.go

@@ -147,8 +147,11 @@ func (c *backgroundCache) Store(ctx context.Context, keys []string, bufs [][]byt
 		}
 
 		size := bgWrite.size()
-		newSize := c.size.Load() + int64(size)
+		// prospectively add new size
+		newSize := c.size.Add(int64(size))
 		if newSize > int64(c.sizeLimit) {
+			// subtract it since we've exceeded the limit
+			c.size.Sub(int64(size))
 			c.failStore(ctx, size, num, "queue at byte size limit")
 			return nil
 		}