[improve][broker] Reduce memory occupation of the delayed message queue

[thetumbled]

Motivation

Delayed queue consume a lot of memory to store the message id. We can reduce the memory occupation with new data structure simillar to #23601.

Modifications

Use new data structure to reduce memory occupation of the delayed message queue, InMemoryDelayedDeliveryTracker.

List some experiment data with following test code:

    static long trimLowerBit(long timestamp, int bits) {
        return timestamp & (-1L << bits);
    }

    public static void main(String[] args) throws IOException {
        TripleLongPriorityQueue queue = new TripleLongPriorityQueue();
        Long2ObjectSortedMap<Long2ObjectMap<Roaring64Bitmap>> map2 = new Long2ObjectAVLTreeMap<>();

        long numMessages = 10000000, numLedgers=1000;
        long numEntries = numMessages/numLedgers;
        long ledgerId, entryId, partitionIndex, timestamp=System.currentTimeMillis();
        for(long i=0; i<numLedgers; i++) {
            ledgerId = 10000+i;
            for(long j=0; j<numEntries; j++) {
                entryId = 10000+j;
                partitionIndex = 0;
                // 1ms per message
                timestamp++;
                queue.add(timestamp, ledgerId, entryId);
                
                timestamp = trimLowerBit(timestamp, 8);
                if (map2.containsKey(timestamp)) {
                    Long2ObjectMap<Roaring64Bitmap> ledgerMap = map2.get(timestamp);
                    if (ledgerMap.containsKey(ledgerId)) {
                        ledgerMap.get(ledgerId).add(entryId);
                    } else {
                        Roaring64Bitmap entrySet = new Roaring64Bitmap();
                        entrySet.add(entryId);
                        ledgerMap.put(ledgerId, entrySet);
                    }
                } else {
                    Long2ObjectMap<Roaring64Bitmap> ledgerMap = new Long2ObjectOpenHashMap<>();
                    Roaring64Bitmap entrySet = new Roaring64Bitmap();
                    entrySet.add(entryId);
                    ledgerMap.put(ledgerId, entrySet);
                    map2.put(timestamp, ledgerMap);
                }
            }
        }
        
        System.out.println("queue size in MB: " + queue.bytesCapacity() / 1024.0 / 1024.0);
        try {
            Thread.sleep(10000000);
        } catch (InterruptedException e) {
            // TODO Auto-generated catch block
            e.printStackTrace();
        }
    }

Produce 1kw entries with 1k ledgers, with the speed of 1 msg/ms.

• Original data structure: TripleLongPriorityQueue
  It store the info in the direct memory, we can call queue.bytesCapacity() to get the size of memory it take.
  
  So, TripleLongPriorityQueue take up 240MB.

• New data structure: Long2ObjectSortedMap<Long2ObjectMap<Roaring64Bitmap>>
  It store the data in heap, so we can use jmap to dump the memory to analyze.
  
  So, it take up only 8MB, down to only 8/240=3.33% of the original size!

Verifying this change

• Make sure that the change passes the CI checks.

(Please pick either of the following options)

This change is already covered by existing tests, such as (please describe tests).

Does this pull request potentially affect one of the following parts:

If the box was checked, please highlight the changes

• Dependencies (add or upgrade a dependency)
• The public API
• The schema
• The default values of configurations
• The threading model
• The binary protocol
• The REST endpoints
• The admin CLI options
• The metrics
• Anything that affects deployment

Documentation

• doc
• doc-required
• doc-not-needed
• doc-complete

Matching PR in forked repository

PR in forked repository: thetumbled#65

[dao-jun]

Good work, and I'm just wandering how much memories can we reduce?

[thetumbled]

Good work, and I'm just wandering how much memories can we reduce?

Updated in the PR.

[lhotari]

LGTM. Great improvement! Thanks for the contribution @thetumbled