The maximum number of messages per second that DLT can process

[goodbenya]

Hello everyone,

I'm reaching out to see if anyone knows how many messages the DLT daemon can handle in a second?

For example:
Imagine we're continuously sending different 200-character messages to the DLT daemon non-stop.

Are there any settings – maybe in the config files or the code itself – that we could adjust to incrise the number of messages the daemon can process every second?

---

We're already using DLT in our project for logging and it's working great. But we're planning to start tracing from the system level, which will mean dealing with more than ~100K messages per second.

[svlad-90]

Hi @minminlittleshrimp, could dlt-daemon maintainers provide feedback regarding this point?

@goodbenya and I are working on the same project, so I can be considered a co-author of this question. ))

Despite my attempts to point to the General Rules for Logging article, my management is frequently asking me regarding why we can't send more than 5-6k messages per second? After that point, we start to lose some of the messages.

In this respect, I have the following questions:

• Are there any configuration parameters to get more frequent messaging from the dlt-daemon?
• What would be the consequence of such a configuration change? Higher level of the CPU load? Ram load? IO?
• What is the bottleneck here? Why can we send 10-50k per second using, e.g., perfetto without any issues and with DLT not? Is it due to the design of the DLT protocol, which is quite heavy? Or due to the use of IPC mechanisms? Or due to other reasons?

Please understand me. I know the difference between logging tools and tracing tools. They have different purposes, and my idea is to prove to my management that we should separate the collection of low-level data ( data from IP blocks, processor registers, etc. ) to the Perfetto and not use DLT there.

For our project, we've created a facade on top of the DLT Log API, and we can extend it with some additional calls that will use the Perfetto API under the hood. This topic is more about collecting the arguments for why it should be done this way.

On the other hand, many messages can be received even from the dlt-logd-converter in Android. So, having an understanding of the limitations is essential from multiple perspectives.

I'll appreciate any feedback! I'm looking forward to getting it. ))

[svlad-90]

Hi all,

I've spent some time investigating this topic more deeply on my side.

It occurred that there are no strict limitations, as I've stated before. There is no 5-6k message per second limitation.

---

Here are some test results.

On my local Ubuntu machine, I was able to get the following statistics:

Root | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 125000 | Payload : 145888890 | Payload, % : 100.000 | Payload, b/sec, av. : 18236111
|-DLT Log Message Number | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 125000 | Payload : 145888890 | Payload, % : 100.000 | Payload, b/sec, av. : 18236111

Around 125000 messages per second and 17.4 Mbytes / second.

In the specific simulation environment ( simulation inside Docker that is inside Ubuntu OS ) I got such results:

Root | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 83333 | Payload : 157888890 | Payload, % : 100.000 | Payload, b/sec, av. : 13157407
|-1000000 | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 83333 | Payload : 157888890 | Payload, % : 100.000 | Payload, b/sec, av. : 13157407

Around 83333 messages per second and 12.5 Mbytes / second.

On the project's target board:

Root | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 23809 | Payload : 145888890 | Payload, % : 100.000 | Payload, b/sec, av. : 3473545
|-DLT Log Message Number: | Msg-s : 1000000 | Msg-s, % : 100.000 | Msg-s/sec, av. : 23809 | Payload : 145888890 | Payload, % : 100.000 | Payload, b/sec, av. : 3473545

Around 23809 messages per second and 3.3 Mbytes / second.

My initial results on the target board were:

|-DMS application has started! | Msg-s : 11774 | Msg-s, % : 100.000 | Msg-s/sec, av. : 3924 |

I had much smaller values — only 3924 messages per second. But that was caused by the fact that I was collecting data through the dlt-viewer while using my company's VPN to connect to the remote board. The VPN connection was limiting the maximum bandwidth.

From my understanding, that was causing:

• Slow data consumption by the dlt-daemon from the application's DLT Log API buffer
• Much faster client buffer overflow, which was causing a drop in the messages

---

So, the limitations of sending too many messages in my case were:

• Slow connection between the dlt-viewer and the dlt-daemon
• Usage of the default buffer size - 500000 bytes

If someone's is interested in maximization of the number of printed messages:

• Ensure fast connection to the target or use the offline storage feature - https://github.com/COVESA/dlt-daemon/blob/master/doc/dlt_offline_logstorage.md.

• Extend the client's buffer size with:

  export DLT_USER_BUFFER_MAX=<size_in_bytes>

  In my case, setting 41943040 bytes ( 40 Mb ) was enough to eliminate all such message drops on the client side, with the client sending 1000000 messages in a loop:

  [129219.438993]~DLT~1474616~WARNING ~Buffer full! Messages will be discarded.
  [129220.320842]~DLT~1474616~WARNING ~244997 messages discarded!
  [129220.618055]~DLT~1474616~WARNING ~Buffer full! Messages will be discarded.
  [129221.748309]~DLT~1474616~WARNING ~284605 messages discarded!
  [129222.069520]~DLT~1474616~WARNING ~Buffer full! Messages will be discarded.

---

To be clear, this topic is just an investigation of what is possible, and we should all try to follow the General Rules for Logging.

---

If anyone asks, "Can I use DLT for data tracing?" You can answer, "No!" The main technical reason is that DLT has a time precision limitation. It uses a tenth of a millisecond while tracing requires nanoseconds. You will end up with the following situation:

Many messages with the same timestamp. If you try to create a plot out of such data, you will get many values located at the same time point. That would be a useless visualization.

For our project, we finally concluded that DLT as a logging tool is quite performant, still found an answer as to why we WILL NOT use it for high-frequent data. We are considering using Perfetto to collect tracing data for some of our target OSs. DLT will be used, but only for logging.

This topic can be considered as "Closed". ))

[minminlittleshrimp]

Hello @svlad-90
Your finding is impressive to me. Thanks for your research.
As far as I know, DLT Daemon, and also DLT viewer, are such logging utilities supporting for DLT log format from Autosar standard. Hence, DLT Daemon is implemented also based on Autosar (classic) specification, you can check the pdf online or via link in README md.

Thus, DLT is a COMstack component in Autosar, and I believe for MCU, a very slow and small-sized data - comparing with SoC or other high-load in GPU/NPU, idk, is fetched via some protocols (Vsomeip, spi-inc, tcp4/6, CAN), and is for sure low-freq. To serve that, DLT Daemon must adapt somehow with the specs. That is all I can come up with from my experience with DLT.
Regards

[svlad-90]

Hi @minminlittleshrimp,

I've found this dlt-daemon discussion with your answers that uncovered ( for me ) a feature to support nanoseconds timestamp precision in DLT:
#563

R21-11

[PRS_Dlt_01004] ⌈If the Log and Trace message is a Data Message (Verbose Mode
or Non-Verbose Mode), the TMSP2 (Timestamp) with a nanosecond resolution shall
be added to the Base Header.⌋ (RS_LT_00002, RS_LT_00013, RS_LT_00017)

As far as I know, the dlt-viewer already supports dlt protocol V2.

What about the dlt-daemon? Does it also support working with protocol V2? If not, is there any plan for when it will be supported?

[minminlittleshrimp]

Hello @svlad-90
I could not make sure, and also we have not received any update from COVESA for the plan (at least IMHO I believe). I think we can involve COVESA in this discussion for clarification.

Hello @michael-methner
Do you have any update on this info? I believe we can check with COVESA.
Regards