[Optimization] AsyncShardFetch response contains DiscoveryNode which takes GBs of memory

[amkhar]

Describe the bug
This issue is a subtask to the project #5098 where we are trying to revamp the flow for AsyncShardFetch, as that takes GBs in memory and impacts cluster manager with JVM spikes.

After analyzing the heap dump carefully, we found that DiscoveryNode object is maintained in memory for 1.5 million times though we know these nodes are limited in cluster like 100 only. As part of this optimization we need to understand the reasoning behind it and improve the current objects so DiscoveryNode is maintained in memory for a limited number of times not in order of number of shards etc.

To Reproduce
Step of repro were same as here: #5098 (comment)

Expected behavior
DiscoveryNode objects should be limited and not take memory in GBs.

[anasalkouz]

@amkhar are you actively working on fixing this?

[amkhar]

@anasalkouz I've not started working on it, currently working on a revamp approach for #5098

I'll come up with proposal for this after 1-2 weeks. It's in my next priority.