Limit server resources for the tracker

[josecelano]

We are running a tracker demo at:

• HTTP Tracker: https://tracker.torrust-demo.com/announce
• UDP Tracker: udp://tracker.torrust-demo.com:6969/announce

We are using a Digital Ocean droplet:

We had to increase it recently because the server received more and more requests.

Currently, the server is processing approximately 500 req/sec, which is a very low load compared with other trackers, such as https://www.gbitt.info/.

The tracker was restarted on the 29th of April. The current stats:

{
  "torrents": 1250905,
  "seeders": 1482241,
  "completed": 27409,
  "leechers": 2729942,
  "tcp4_connections_handled": 0,
  "tcp4_announces_handled": 0,
  "tcp4_scrapes_handled": 0,
  "tcp6_connections_handled": 0,
  "tcp6_announces_handled": 0,
  "tcp6_scrapes_handled": 0,
  "udp4_connections_handled": 24527865,
  "udp4_announces_handled": 59027694,
  "udp4_scrapes_handled": 1507185,
  "udp6_connections_handled": 0,
  "udp6_announces_handled": 0,
  "udp6_scrapes_handled": 0
}

If the tracker continues running, we will have problems again because it will consume more resources. In the past, we tried to limit memory consumption:

#789

However, we did not finish that implementation. Limiting resources is a tricky/complex feature because it can degrade the service (response time or accuracy).

We are considering different strategies for limiting resource consumption. Resources considered are:

• CPU
• Memory
• Bandwidth

We would like to know:

1. Wether it makes sense or not to limit tracker resources.
2. Different alternatives already implemented by other trackers.
3. Extra ideas to implement these limits.

Does it make sense?

If we don't limit resources, the tracker will be restarted periodically depending on the server size, which means downtime anyway. A bigger machine can only prolong the period between restarts.

On the other hand, it seems it would make sense to allow people to run smaller servers depending on their budgets. Some people may want to contribute to the community by running a small $6 tracker server. We don't know if this makes sense since that would make the tracker less efficient, which is the main advantage compared to DHT. Many smaller trackers could lead to an efficiency similar to DHT. Maybe only big tracker servers make sense.

We will add some ideas we have been discussing below. For example, limiting the memory consumption for the torrent repository. We add an "update" datetime to all torrents. That datetime is updated every time an announce request is received for that torrent. When we reach the limit, we remove torrents that have not been updated the longest.

Type of solutions

There are two main solutions to this problem:

• Scalability (horizontally or vertically). Notice that the current tracker implementation does not allow horizontal scaling because the torrent repository is stored in memory in-process.
• Resource limiting.

This discussion is only about the second type.

cc @da2ce7 @Power2All @greatest-ape

[josecelano]

1. MEM: Clean peerless torrents

This is already implemented to save memory. Peers that don't announce in 2 minutes are removed from the torrent entry in the torrent repository. When the last peer is removed from the torrent (peerless torrent), the torrent is also removed from the repo.

[josecelano]

2. MEM LIMIT: Set a memory limit and remove torrents that have not been updated the longest

We have already tried to implement this:

#789

But it was not merged because it was mixed with other features that were not ready to merge and because it was not implemented for all the torrent repository implementations.

The main idea is to keep the last "update" datetime for each torrent and calculate the memory used by the torrent repository. When we reach the limit we remove the torrents that has not been updated the longest.

Pros:

• Limit memory consumption.

Cons:

• Decreases performance because we have to keep an ordered list of torrents by their updated datetime.
• If we have a lot of torrents, a torrent that has not updated the longest could be an active torrent that has received an announce request recently. It could produce "empty peer lists".

[josecelano]

3. MEM LIMIT: Set a memory limit and move less active torrents to disk

This would require counting the number of requests per torrent. Popular torrents remain in memory, and torrents with few requests are moved to disk.

Pros:

• The server does not consume all the memory.

Cons:

• Keep track of the number of requests (it should not be too costly).
• Move torrents from disk to memory depending on demand.
• It requires enabling the database.

This would work if we have a Pareto distribution or similar, otherwise we are just moving torrents from memory to disk.

[greatest-ape]

Torrents and peers can be stored using very little memory. I’ve been told that the explodie.org tracker tends to use around 1.5GB, and it currently tracks 4 million torrents and 10 million peers and serves 120k requests per second. For this reason, I think resource limiting is a bit of a dead end, and that a good first step would instead be to use a memory profiler to see where all the memory is consumed and try to reduce it.

If you do implement memory limit functionality or similar, you should be careful about in practice increasing memory use due to overhead for extra metadata, duplicating data when shuffling it around, and so on, risking pushing peak use into OOM territory.

[greatest-ape]

Ok, nice! It would be interesting to see how it compares to aquatic and torrust-axum in benchmarks, like the one on this page: https://github.com/greatest-ape/aquatic/tree/master/crates/http. I still think it’s a good idea to look for opportunities to save space before trying to limit resources, though.

[Power2All]

Ok, nice! It would be interesting to see how it compares to aquatic and torrust-axum in benchmarks, like the one on this page: https://github.com/greatest-ape/aquatic/tree/master/crates/http. I still think it’s a good idea to look for opportunities to save space before trying to limit resources, though.

Torrust-Axum is renamed to Torrust-Actix because of the issues with it though.
Axum has a problem with Client Read and Write timeouts hence the switch.

Your library is not the issue, in fact, it works neat and good.
I've not went deep in your library, do you use any RWLock or Mutex locking ?
If so, I would suggest you look at Crossbeam SkipMap as alternative, faster and more thread-safe solution.

Saving memory is always a interesting thing.
I am looking in improving memory usage too with a rewrite (currently rewriting Torrust-Actix as version 4.0 with WebTorrent support eventually added too), I might be also looking into applying compression in memory storage, to lower usage as well, but is still something I need to look into.

[greatest-ape]

Ok, I see.

The devil is always in the details, and a lot can be achieved with fine-grained locks. My current implementation is already very fast and does not suffer from deadlock issues. SkipMap has worse time complexity than the standard Rust HashMap and doesn’t allow controlling when memory is deallocated, so I’m not convinced that it would be better in practice than my current aquatic_udp implementation.

[josecelano]

Hi @greatest-ape, memory profiling is a good idea. I was thinking only about the core tracker repository but not considering other structs. I was assuming the biggest consumption is there. Anyway, my point is you eventually will run out of memory no matter how much memory you have if you have a lot of torrents/peers. And I wanted to know if it makes sense to limit it and what other projects are doing.

[greatest-ape]

It likely is the torrent/peer data, but stuff like HTTP connection buffers, various channels etc might matter too. And profiling is useful in any case.

I don’t think opentracker has any resource limiting, but it also stores the bare minimum of peer data and does so very efficiently.

[greatest-ape]

Also, memory use should not increase over time if torrent/peer counts are stable. Have you checked for memory leaks?

[Power2All]

Nah, the problem is, is the RWLock that creates deadlocks, and it's really slow.
Also, the web framework has it's issues, hence why I myself switched to Actix for these reasons, with including preventing slowloris be usedful at all.

[greatest-ape]

Well, the point is, the server should be able to run for months/years without issues if the torrent/peer counts are stable. I don’t see why it would periodically run out of memory, as the first post in this thread indicated.

[Power2All]

Well, the point is, the server should be able to run for months/years without issues if the torrent/peer counts are stable. I don’t see why it would periodically run out of memory, as the first post in this thread indicated.

That's due to RWLock, eventually it deadlocks, and the system keeps on rolling along, until the memory usage is exceeded. It's not so much a memory leak, it's just the threads stop working and hanging, cause they are unable to get a Read Lock from RWLock after it's exhausted. This has been explained a lot on the internet after I've been dealing with this for a long while ago.

[greatest-ape]

Ok, yes, care definitely needs to be taken to avoid deadlocks and to keep critical sections short.

[josecelano]

Torrents and peers are always increasing. I have collected stats manually. That's why I assumed the problem was simply the server running out of memory. I want to generate those graphs automatically. I've started collecting all the info I need with Datadog, at least for debugging, so I can find the real bottleneck:

torrust/torrust-demo#3

But I will do the memory profiling to find out the percentages consumed by:

• The core tracker data structure
• Other structs, like responses
• Memory consumed by threads (potentially increased to undesirable levels by deadlocks)

Regarind memory leaks at the beginning, I thought it was this problem:

• There are a lot of zombie processes torrust-demo#2

But the zombie process issue likely is because the server is busy, and child threads can't connect to the parent one. They are not created until the server starts having problems with swapping.

Thank you, @Power2All @greatest-ape, for your feedback.