Add very simple benchmarking against reactor-c #140

erlingrj · 2024-11-28T04:57:23Z

This should be merged AFTER #137

This PR adds the Savina PingPong and a ReactionLatency benchmark, with a C and uC version. Current results on my machine is:

PingPongUc:
 Best Time:      77.571 msec
Worst Time:      87.062 msec
Median Time:     77.455 msec
PingPongC:
 Best Time:      122.188 msec
Worst Time:      130.564 msec
Median Time:     123.531 msec
ReactionLatencyUc:
 Best latency: 17358. 
Median latency: 56424
Worst latency: 255700
ReactionLatencyC:
 Best latency: 4804
Median latency: 53017
Worst latency: 307132

I.e. our runtime has less overhead (PingPong only needs 60-70% of the time needed by reactor-c). When it comes to reaction latency, we are very similar to reactor-c, the Median latency is the most useful number showing that we might have a slightly longer execution path from wakeup to the first reaction. But not significantly. Those last numbers are in nsec btw

…ny call to wait_until.

github-actions · 2024-11-28T04:58:21Z

Memory usage after merging this PR will be:

Memory Report

action_empty_test_c

	from	to	increase (%)
text	59850	59617	-0.39
data	744	744	0.00
bss	10112	10112	0.00
total	70706	70473	-0.33

action_microstep_test_c

	from	to	increase (%)
text	60687	60454	-0.38
data	752	752	0.00
bss	10112	10112	0.00
total	71551	71318	-0.33

action_overwrite_test_c

	from	to	increase (%)
text	60524	60291	-0.38
data	744	744	0.00
bss	10112	10112	0.00
total	71380	71147	-0.33

action_test_c

	from	to	increase (%)
text	60460	60227	-0.39
data	752	752	0.00
bss	10112	10112	0.00
total	71324	71091	-0.33

delayed_conn_test_c

	from	to	increase (%)
text	61712	61479	-0.38
data	744	744	0.00
bss	10112	10112	0.00
total	72568	72335	-0.32

event_payload_pool_test_c

	from	to
text	18330	18330
data	624	624
bss	320	320
total	19274	19274

event_queue_test_c

	from	to
text	27597	27597
data	736	736
bss	480	480
total	28813	28813

nanopb_test_c

	from	to
text	42888	42888
data	904	904
bss	320	320
total	44112	44112

port_test_c

	from	to	increase (%)
text	61660	61427	-0.38
data	744	744	0.00
bss	10112	10112	0.00
total	72516	72283	-0.32

reaction_queue_test_c

	from	to
text	27277	27277
data	736	736
bss	480	480
total	28493	28493

request_shutdown_test_c

	from	to	increase (%)
text	60659	60426	-0.38
data	744	744	0.00
bss	10112	10112	0.00
total	71515	71282	-0.33

startup_test_c

	from	to	increase (%)
text	56010	55777	-0.42
data	752	752	0.00
bss	10752	10752	0.00
total	67514	67281	-0.35

tcp_channel_test_c

	from	to
text	60294	60294
data	1176	1176
bss	11136	11136
total	72606	72606

timer_test_c

	from	to	increase (%)
text	55901	55668	-0.42
data	744	744	0.00
bss	10752	10752	0.00
total	67397	67164	-0.35

github-actions · 2024-11-28T05:02:08Z

Memory usage after merging this PR will be:

Memory Report

action_empty_test_c

	from	to	increase (%)
text	59830	59850	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	70686	70706	0.03

action_microstep_test_c

	from	to	increase (%)
text	60667	60687	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71531	71551	0.03

action_overwrite_test_c

	from	to	increase (%)
text	60504	60524	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71360	71380	0.03

action_test_c

	from	to	increase (%)
text	60440	60460	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71304	71324	0.03

delayed_conn_test_c

	from	to	increase (%)
text	61692	61712	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72548	72568	0.03

event_payload_pool_test_c

	from	to
text	18330	18330
data	624	624
bss	320	320
total	19274	19274

event_queue_test_c

	from	to
text	27597	27597
data	736	736
bss	480	480
total	28813	28813

nanopb_test_c

	from	to
text	42888	42888
data	904	904
bss	320	320
total	44112	44112

port_test_c

	from	to	increase (%)
text	61640	61660	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72496	72516	0.03

reaction_queue_test_c

	from	to
text	27277	27277
data	736	736
bss	480	480
total	28493	28493

request_shutdown_test_c

	from	to	increase (%)
text	60639	60659	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71495	71515	0.03

startup_test_c

	from	to	increase (%)
text	55983	56010	0.05
data	752	752	0.00
bss	10752	10752	0.00
total	67487	67514	0.04

tcp_channel_test_c

	from	to
text	60294	60294
data	1176	1176
bss	11136	11136
total	72606	72606

timer_test_c

	from	to	increase (%)
text	55881	55901	0.04
data	744	744	0.00
bss	10752	10752	0.00
total	67377	67397	0.03

github-actions · 2024-11-28T05:34:27Z

Memory usage after merging this PR will be:

Memory Report

action_empty_test_c

	from	to	increase (%)
text	59830	59850	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	70686	70706	0.03

action_microstep_test_c

	from	to	increase (%)
text	60667	60687	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71531	71551	0.03

action_overwrite_test_c

	from	to	increase (%)
text	60504	60524	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71360	71380	0.03

action_test_c

	from	to	increase (%)
text	60440	60460	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71304	71324	0.03

delayed_conn_test_c

	from	to	increase (%)
text	61692	61712	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72548	72568	0.03

event_payload_pool_test_c

	from	to
text	18330	18330
data	624	624
bss	320	320
total	19274	19274

event_queue_test_c

	from	to
text	27597	27597
data	736	736
bss	480	480
total	28813	28813

nanopb_test_c

	from	to
text	42888	42888
data	904	904
bss	320	320
total	44112	44112

port_test_c

	from	to	increase (%)
text	61640	61660	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72496	72516	0.03

reaction_queue_test_c

	from	to
text	27277	27277
data	736	736
bss	480	480
total	28493	28493

request_shutdown_test_c

	from	to	increase (%)
text	60639	60659	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71495	71515	0.03

startup_test_c

	from	to	increase (%)
text	55983	56010	0.05
data	752	752	0.00
bss	10752	10752	0.00
total	67487	67514	0.04

tcp_channel_test_c

	from	to
text	60294	60294
data	1176	1176
bss	11136	11136
total	72606	72606

timer_test_c

	from	to	increase (%)
text	55881	55901	0.04
data	744	744	0.00
bss	10752	10752	0.00
total	67377	67397	0.03

tanneberger · 2024-11-28T05:37:54Z

vewwy vewwy cool.

can you also post the binary sizes and ram usage? of those two examples?

github-actions · 2024-11-28T05:43:44Z

Memory usage after merging this PR will be:

Memory Report

action_empty_test_c

	from	to	increase (%)
text	59830	59850	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	70686	70706	0.03

action_microstep_test_c

	from	to	increase (%)
text	60667	60687	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71531	71551	0.03

action_overwrite_test_c

	from	to	increase (%)
text	60504	60524	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71360	71380	0.03

action_test_c

	from	to	increase (%)
text	60440	60460	0.03
data	752	752	0.00
bss	10112	10112	0.00
total	71304	71324	0.03

delayed_conn_test_c

	from	to	increase (%)
text	61692	61712	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72548	72568	0.03

event_payload_pool_test_c

	from	to
text	18330	18330
data	624	624
bss	320	320
total	19274	19274

event_queue_test_c

	from	to
text	27597	27597
data	736	736
bss	480	480
total	28813	28813

nanopb_test_c

	from	to
text	42888	42888
data	904	904
bss	320	320
total	44112	44112

port_test_c

	from	to	increase (%)
text	61640	61660	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	72496	72516	0.03

reaction_queue_test_c

	from	to
text	27277	27277
data	736	736
bss	480	480
total	28493	28493

request_shutdown_test_c

	from	to	increase (%)
text	60639	60659	0.03
data	744	744	0.00
bss	10112	10112	0.00
total	71495	71515	0.03

startup_test_c

	from	to	increase (%)
text	55983	56010	0.05
data	752	752	0.00
bss	10752	10752	0.00
total	67487	67514	0.04

tcp_channel_test_c

	from	to
text	60294	60294
data	1176	1176
bss	11136	11136
total	72606	72606

timer_test_c

	from	to	increase (%)
text	55881	55901	0.04
data	744	744	0.00
bss	10752	10752	0.00
total	67377	67397	0.03

github-actions · 2024-11-28T18:42:29Z

Benchmark results after merging this PR:

Benchmark results

Performance:

PingPongUc:
Best Time: 137.471 msec
Worst Time: 168.716 msec
Median Time: 138.521 msec

PingPongC:
Best Time: 168.873 msec
Worst Time: 169.428 msec
Median Time: 168.846 msec

ReactionLatencyUc:
Best latency: 21277 nsec
Median latency: 59943 nsec
Worst latency: 217890 nsec

ReactionLatencyC:
Best latency: 26608 nsec
Median latency: 59889 nsec
Worst latency: 229363 nsec

Memory usage:

PingPongUc:
text data bss dec hex filename
39812 752 8792 49356 c0cc bin/PingPongUc

PingPongC:
text data bss dec hex filename
46044 872 360 47276 b8ac bin/PingPongC

ReactionLatencyUc:
text data bss dec hex filename
29841 736 2080 32657 7f91 bin/ReactionLatencyUc

ReactionLatencyC:
text data bss dec hex filename
41666 840 360 42866 a772 bin/ReactionLatencyC

github-actions · 2024-11-28T21:55:07Z

Coverage after merging add-benchmarks into main will be

67.83%

Coverage Report

File	Stmts	Branches	Funcs	Lines	Uncovered Lines
src
action.c	81.90%	69.23%	100%	85.33%	120–121, 24, 43–46, 49, 51, 53, 56–58, 63–64, 73–74, 85–86
builtin_triggers.c	90.91%	70%	100%	96.77%	14, 18, 40, 43
connection.c	78.52%	51.16%	100%	88.66%	10, 104, 11, 110, 123–124, 136–137, 14, 14, 143, 145, 16–17, 21–22, 22, 22–23, 25, 27–28, 33, 48, 48, 48–49, 55, 60–62, 97
environment.c	78.35%	55.56%	84.62%	83.33%	12–13, 18, 20–21, 31, 35–36, 42–43, 51–52, 55–56, 60–61, 93–95
event.c	95.35%	92.86%	100%	96.15%	14–15
federated.c	0%	0%	0%	0%	100–102, 104, 104, 104–107, 109, 11, 111–112, 112, 112, 112–116, 118–119, 122, 125–126, 126, 126–127, 129, 13, 13, 13, 130, 132, 136–137, 139, 14, 140–141, 144, 146–149, 15, 150–151, 153–155, 158–159, 16, 160, 160, 160–161, 161, 161–163, 165, 168–169, 17, 17, 17, 171–175, 177–179, 18, 180–182, 184, 184, 184–187, 189, 189, 189–191, 191, 191–192, 196–197, 197, 197, 200–201, 205–207, 209, 209, 209, 21, 211–215, 218, 218, 218–219, 22, 220–221, 224–225, 225, 225–226, 228–229, 23, 23, 23, 232–233, 238–239, 239, 239, 24, 240, 242, 244, 244, 244–247, 247, 247, 247, 247–249, 25, 25, 25, 250–259, 26, 263, 266, 266, 266–268, 27, 272, 275–276, 276, 276, 276–279, 28, 280–284, 286, 288–289, 29, 29, 29, 290, 290, 290–291, 293, 293, 293–295, 297–299, 30, 301, 303, 307, 307, 307–308, 308, 308–309, 309, 309, 31, 31, 31, 31, 310–319, 32, 320, 322–323, 326–329, 33, 331, 331, 331–332, 336–337, 337, 337, 339, 34, 341–342, 342, 342–343, 343, 343–344, 344, 344–345, 345, 345–346, 346, 346–347, 347, 347, 349, 349, 349, 35, 350, 350, 350–351, 351, 351–352, 352, 352, 354, 37, 37, 37, 37, 37–38, 40–43, 48, 48, 48, 48, 48–49, 52, 56–57, 59–62, 64, 64, 64–65, 65, 65, 67, 67, 67–69, 69, 69–71, 75–76, 80–81, 83–86, 88, 9, 90, 90, 90–91, 91, 91–92, 92, 92–93, 93, 93, 96–97, 99
logging.c	73.21%	60%	100%	75%	24, 24–27, 37–39, 46, 46–49, 59–60
port.c	81.43%	50%	100%	95.45%	10, 10, 10, 16, 20, 26, 26–28, 38, 38, 38–39
queues.c	89.94%	80.36%	100%	94.06%	108, 113, 119, 21–23, 47–48, 60–61, 84–88, 91–92
reaction.c	69.83%	54.55%	100%	77.61%	15, 17, 21–22, 28–31, 31, 31–32, 42, 45, 47, 52–53, 53, 53–55, 55, 55–56, 73, 89–91, 91, 91–94, 94, 94–95
reactor.c	69.33%	51.52%	100%	82.28%	10, 101–102, 14–19, 22, 28, 30, 32–37, 37, 37–38, 38, 38, 43, 55, 58–59, 59, 59–60, 60, 60–61, 63, 77–78, 81–82, 82, 82–83, 83, 83–84, 86, 91
serialization.c	50%	50%	50%	50%	16–17, 26–27, 33–35, 38–40
tag.c	40.19%	31.48%	60%	47.92%	14, 14–15, 17, 17–18, 23–24, 24, 24, 24, 24–25, 27, 27, 27, 27, 27–28, 30, 30, 30–31, 33–34, 34, 34–35, 37, 37, 37, 37, 37–38, 40, 40, 40, 40, 40–41, 43, 53–54, 63, 63–64, 83–85, 85, 85, 85, 85, 85, 85, 85, 85, 85, 85–87, 89
timer.c	95%	66.67%	100%	100%	14, 25
trigger.c	100%	100%	100%	100%
util.c	0%	0%	0%	0%	10, 3–4, 4, 4–5, 5, 5–6, 8–9
src/platform/posix
posix.c	52.73%	30%	66.67%	56%	100, 100, 100–102, 106, 16, 18, 20–21, 34–36, 38–40, 48–49, 54–59, 59, 59–62, 62, 62–64, 67, 73–74, 78, 81, 92–94, 94, 94–96, 98–99
tcp_ip_channel.c	67.52%	56.69%	94.12%	70.73%	114–116, 121–123, 127–129, 149, 153, 153, 153, 157–158, 171–172, 175–176, 178, 178, 178, 180–181, 183–184, 191–192, 194, 197, 212–213, 217–219, 221, 226–229, 232, 235, 238, 245–248, 250, 250, 250–252, 254–257, 260–261, 291–293, 300, 305–307, 307, 307–308, 310–313, 32, 322, 322–324, 33, 346–349, 349, 349–351, 361–362, 362, 362–363, 363, 363–364, 368–369, 372, 382–383, 400–401,

erlingrj added 13 commits November 26, 2024 20:37

Remove any platform specific code in code-generated CMakeLists

9ca11b8

Fix FlexPRET

e64d35c

Dump working changes

cdcdc99

Merge remote-tracking branch 'origin/main' into lfc-generate-num-events

bfa8a74

Minimum reaction and event queue size is 1

044d6ae

Ironing out the final wrinkles

9f98fb8

Add support for running in fast mode, simply return immediatly from a…

ec566b9

…ny call to wait_until.

Merge remote-tracking branch 'origin/main' into fast-mode

5b5ec84

Typo

575e683

Update examples

cd92846

Fix preamble handling

a11e910

Add two simple benchmarks

da13e6a

Run benchmarks in CI

655908f

erlingrj linked an issue Nov 28, 2024 that may be closed by this pull request

Add preliminary benchmarking versus reactor-c #134

Closed

Report benchmark result units

d107fe3

Post benchmark results to the PR

7aa9553

Try to output something from a step

db3a315

CI

bcc0d2d

erlingrj added 5 commits November 28, 2024 11:06

Fix colorized output

e8be0a5

Merge remote-tracking branch 'origin/main' into add-benchmarks

ad166de

Remove the need for dynamically sized array on the stack

87c1e50

Add a newline

67b5095

Another newline

5172109

erlingrj merged commit a7d9dc4 into main Nov 28, 2024
8 checks passed

erlingrj deleted the add-benchmarks branch November 28, 2024 22:08

Add very simple benchmarking against reactor-c #140

Add very simple benchmarking against reactor-c #140

Conversation

erlingrj commented Nov 28, 2024 • edited Loading

github-actions bot commented Nov 28, 2024 • edited Loading

action_empty_test_c

action_microstep_test_c

action_overwrite_test_c

action_test_c

delayed_conn_test_c

event_payload_pool_test_c

event_queue_test_c

nanopb_test_c

port_test_c

reaction_queue_test_c

request_shutdown_test_c

startup_test_c

tcp_channel_test_c

timer_test_c

github-actions bot commented Nov 28, 2024

action_empty_test_c

action_microstep_test_c

action_overwrite_test_c

action_test_c

delayed_conn_test_c

event_payload_pool_test_c

event_queue_test_c

nanopb_test_c

port_test_c

reaction_queue_test_c

request_shutdown_test_c

startup_test_c

tcp_channel_test_c

timer_test_c

github-actions bot commented Nov 28, 2024

action_empty_test_c

action_microstep_test_c

action_overwrite_test_c

action_test_c

delayed_conn_test_c

event_payload_pool_test_c

event_queue_test_c

nanopb_test_c

port_test_c

reaction_queue_test_c

request_shutdown_test_c

startup_test_c

tcp_channel_test_c

timer_test_c

tanneberger commented Nov 28, 2024

github-actions bot commented Nov 28, 2024

action_empty_test_c

action_microstep_test_c

action_overwrite_test_c

action_test_c

delayed_conn_test_c

event_payload_pool_test_c

event_queue_test_c

nanopb_test_c

port_test_c

reaction_queue_test_c

request_shutdown_test_c

startup_test_c

tcp_channel_test_c

timer_test_c

github-actions bot commented Nov 28, 2024 • edited Loading

Performance:

Memory usage:

github-actions bot commented Nov 28, 2024

erlingrj commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited

Loading

github-actions bot commented Nov 28, 2024 •

edited

Loading