FIP: Reduce Storage Limits

[Wazzymandias]

Authors

@sanjayprabhu @varunsrin @Wazzymandias

Problem

A hub stores a copy of all data on the Farcaster network. This is about 140GB today and is growing at 1GB/day. This is problematic for a few reasons:

1. Increasing data size will lead to slower startup and sync processes.
2. There is a soft cap at 7 TB — high performance hubs need NVMe disks which tap out here.
3. Farcaster’s growth is unpredictable and could 10x quickly creating new problems.

Proposal

We propose reducing the amount of space that each storage unit provides without changing the price. The new limits for casts, reactions, and links will be:

Type	Old Limit	Proposed Limit	Users Impacted
Casts	5000	4000	~1700
Reactions	2500	2000	~23000
Links	2500	2000	~5700
UserData	50	50	No Change
Verification	25	25	No Change
Username Proof	5	5	No Change

Analysis

• Casts, links, and reactions use roughly 70% of all storage space for users on hubs
• The average user has around 179 casts, 418 reactions, and 236 links

Message Type	Count	Total Bytes	Total Bytes %
Cast	92.0M	23.6B	18.38
Link	121.3M	22.4B	17.47
Reaction	214.7M	43.5B	33.94
Verification	411.8K	118.3M	0.09
UserData	2.0M	406.9M	0.32
Username Proof	10.2K	2.905M	0.00

Impact

These changes should decrease the size of data on the Hub by 5%.

A few accounts (~ 1%) have casts and links that will be pruned by this change. A majority of these accounts are “programmatic” and post automated replies (e.g. @perl). They can easily resolve this by buying another storage unit.

A slightly larger number (~ 5%) have reactions that will be impacted by this. A majority of these expired reactions are not useful to the network. Keeping them around makes hubs more difficult to operate while providing little value to the network. Anyone who really wishes to retain more reactions can always purchase more storage.

Rationale

Why not increase the price of storage units?

Making storage more expensive will reduce the number of users signing up. This isn’t good because real users are price sensitive while airdrop farmers aren’t.

Can Hubs be sharded to scale better?

Yes, but sharding makes sync and availability orders of magnitude more complex for developers. While we will eventually need to do this, we want to avoid it until we are closer to the 7 TB cap.

Can we use compression or other lossless optimizations? We’ve explored compression, removing indices and other such changes. The best ones we’ve found reduce 1% — 2% of storage but come with a lot of implementation complexity. The proposed change is much simpler and saves much more storage.

Do we have to do this now?

Farcaster’s growth is unpredictable — we’d rather propose and implement these changes when we have the time and space for a discussion, instead of doing it in an emergency situation./

Will this impact channel moderation?

Channel moderation depends on likes, which are getting more expensive. If you are running a channel bot that will be affected by this, reach out to @dwr.eth or @v. Warpcast will sponsor a few storage units for channel bots to make this transition simpler.

I use all my reaction storage but very little of my cast storage. Can storage units be flexible?

Flexible storage units are very tricky to implement with a good UX. There are many problems, but the most pressing one is what to do when you’ve exceeded the limit and send a new message. The only solution with our eventually consistent network is to prune the oldest message, irrespective of type. This means that if you run out of space and like something, the first person you followed may get unfollowed, which is really bad. There is no easy solution to this problem.

Release

This change will be part of the next protocol version 2024.07.24 . Since the change is backwards incompatible, the new limits will not be enforced until 2024.08.08, one week after protocol version release.

Pruning happens every two hours so we expect that your hub will enforce the new limits for its users within that period. Hubs may be slightly out of sync for a few hours until the pruning processes are complete.

Users and accounts that want to preserve data should upgrade their storage before this date.

[EoghanH]

This unfortunately just looks like a bandaid solution - and honestly, not a very good one imo.

Decreasing storage allowances per unit by approx. 20% will imo have minimal impact on growth - it's more likely that users affected by the change will (with disgruntlement) just purchase extra storage units. I would be surprised to see it having any real knock-on effect on rate of growth; especially when considering the risks posed by an exponential growth scenario.

Would be interested to see the change in rate of growth over the past few months, and how quickly we can expect storage requirements to double from the current 1GB/d (discounting any exponential/burst growth event).

Concern on my end is that we'd just wind up upsetting users who essentially are losing something they've paid for, while it turns out such an action would only save us a few days or weeks at best before hitting the 7TB soft-cap noted.

[zingerj]

No data should be lost as a result of this change. Existing users deserve to be made whole on storage that they've already paid for without being required to purchase additional storage units if they want to avoid their data being pruned.

[varunsrin]

We're making two changes based on feedback:

1. Storage limit changes will only apply to units purchased after this change. Older storage units will retain their higher limits until they expire 1 year after they were rented.

2. Storage limits will be reduced even further than the 20% proposed. Our plan was to make a series of adjustments to decrease storage units over time, but that's going to create uncertainty for users. Instead, we should reduce the storage unit to the smallest possible size that will give new users a good starting experience while keeping the footprint small. If we end up making them too small and it hurts retention, it will be very easy to adjust back up with a backwards compatible change.

[EoghanH]

@varunsrin I really can't see a situation where 2 results in a positive user experience. The concept of purchasing "storage units" is already alien to people outside of the decentralized world; and ultimately our goal is to onboard these people to farcaster.

Telling them that they need to purchase storage units to keep making casts or "like" someone else's cast is already a hard sell. Especially when it's a recurring fee. We're actively disincentivizing use of the network by scaling through restrictive user limitations, rather than scaling through architectural work.

Just can't see this as being healthy for the protocol at all.

[varunsrin]

Open to counter proposals that:

1. can be implemented in a reasonable amount of time
2. do not affect the decentralization of the network (see this article for problems with federation, the most commonly suggested alternative)
3. are well thought through and have enough detail to consider

[sammdec]

@varunsrin Has this been rolled out in the proposed protocol update on 24.07.2024?

[varunsrin]

nope, we're holding it back to the next release.

there were a few other priorities that came up, and a few other options that people suggested that we want to explore first.

[sammdec]

Ok great! I'll keep an eye out for updates

[compprofi]

Starting from June and 1.14+ version 200Gb storage size is not enough anymore.
I'm constantly failing to install hubble on 200Gb disk storage server when it eats all the space and cannot finish installation (54% sync finishes once it reaches 190Gb consumption). Rocks DB is consuming too much space while it syncing all the messages. Is there any plans to optimise disc space usage?

Thank you

[varunsrin]

we have a known issue where during sync we end up needing a lot more storage space because of unpacking the snapshot. on the roadmap to fix, but not sure when we'll get around to it