fatal error: OutOfMemory encountered: Java heap space after updating from 2021.3 to 2024.7

[DamianBorek]

Expected behavior

Broker does not crash during runtime

Actual behavior

We have been using the HiveMQ broker version 2021.3 Community Edition for several years without any issues.

Recently, we decided to upgrade to version 2024.7. Prior to the upgrade, we conducted tests, and the broker initially worked well after the update. However, issues began after updating our devices that connect to the broker.

We have around 5,000 devices in our system. Occasionally, we perform software updates on these devices, which rely on the broker. Our software is about 3 MB in size, and during an update, each device first downloads the new version and then restarts, resulting in a disconnection and reconnection to the broker.

This process creates a significant load on the broker, and we observe a CPU spike due to the SSL handshake calculation. While this increase in CPU usage is noticeable, it is not currently a critical issue for us (we will do SSL offloading for that using AWS network load balancer),

The main problem is that in version 2024.7, the broker stops functioning due to the following error just after our devices start to update:

A fatal error has been detected by the Java Runtime Environment:
fatal error: OutOfMemory encountered: Java heap space

This error never occurred in version 2021.3.

2024.7 is working fine while we do not perform any devices updates.

Could you help us understand what might be causing this? For the upgrade, I replaced the hivemq bin directory and restarted the broker process without altering any configurations in the conf directory or modifying the contents of the data directory. Java env also did not changed at all.

In version 2021.3, we also did not adjust the heap size, and it was sufficient for our needs. Why is the heap size now insufficient? Is this issue due to HiveMQ’s changes in version 2024.7, or does it relate to our configuration?

• Affected HiveMQ CE version(s):
  2024.7

• NOT Affected HiveMQ CE version(s):
  2021.3

• Used JVM version:
  JAVA 11

I am also attaching the log file which is produced after crash. There is also heap-dump.hprof prodeced but it is 5.6GB
hs_err_pid2461506.log