Compilation takes an ungodly amount of RAM now

[NetSysFire]

Describe the bug

I have a system with 8 GB of DDR4 RAM to compile and usually use -j 10 for that (Ryzen 1600af, 12 threads), which is not possible anymore. However it works for basically any other project and used to work in the past.

Steps to reproduce

1. Compile with -j10.
2. Your system freezes because it fills up the RAM so quickly the OOM killer can not even react.

Expected behavior

It does not use this large amount of RAM. It worked with -j10 until the item* files, so the cause may be somewhere around there. I am assuming there might be a recent regression.

I am currently using -j6 which means it peaks at about 4 GB of RAM.

Screenshots

No response

Versions and configuration

• OS: Arch Linux
• Compiler: GCC 12

Additional context

No response

[NetSysFire]

After some discussion on the IRC and additional light debugging, I have come to the conclusion that certain files are simply too big. Examples include cata_tiles.o, certain inventory stuff, activity_actor.o. Considering that I used to be able to compile without memory issues, this is still a regression.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions. Please do not bump or comment on this issue unless you are actively working on it. Stale issues, and stale issues that are closed are still considered.

[NetSysFire]

I have found out why: LTO. I "only" have 8 GB RAM and that also makes debugging with gdb impossible, as loading the symbols will take up more than 8 GB (see https://bbs.archlinux.org/viewtopic.php?pid=2092119).

A gdb dev could confirm the memory usage of trying to load symbols.
CC @kevingranade, as he appears to be interested in optimization.

[anothersimulacrum]

Compiling with 40 cores with gcc takes significantly more memory (30-ish gigs?) than clang (10-ish) on my machine.

[svenstaro]

Is this still relevant?

[kevingranade]

While reducing memory footprint while compiling is a nice to have, we have no target build system requirements and therefore this is not a regression per se.
Workaround, just reduce the argument you pass to -j

I'm not completely hostile to a check along the lines of, "no compilation unit should consume more than X MB of RAM with ", but also I wouldn't expect lto to be on the given compiler options, especially as it makes linking take absolutely ungodly amounts of RAM, and if someone is interested in pushing that forward they need to provide the tooling to support it.