System freezes when setting smp value to 54 or higher with qemu-system-riscv64

[imguoguo]

Description:

When using qemu-system-riscv64 and setting the smp (symmetric multiprocessing) value to 54 or higher, the system freezes and is unable to properly boot into U-Boot. This issue appears to have been introduced in this commit: 87aa306 . On the latest version, this problem still exists.

The root cause seems to be the addition of this line of code:

heap_size += 0x40 * (hart_count) * (hart_count);

After commenting out this line, it is possible to increase the smp value range, but the system still freezes when attempting to set smp to 72 or higher.

And here's some info on the variables. Here, heap_size(old) is used to represent the default value, and heap_size(new) is used to represent the added value, and return_size is the same as well.

smp	54	72
heap_size(old)	143360	180224
return_size(old)	143360	180224
heap_size(new)	329984	512000
return_size(new)	330752	512000

Test Command:

qemu-system-riscv64 -M virt -m 128G -smp 54 -nographic -bios build/platform/generic/firmware/fw_payload.bin

QEMU Log:

OpenSBI v1.3-140-g87aa306
Build time: 2024-04-26 15:58:14 +0800
Build compiler: gcc version 12.3.0 (Buildroot 2021.11-11272-ge2962af)
   ____                    _____ ____ _____
  / __ \                  / ____|  _ \_   _|
 | |  | |_ __   ___ _ __ | (___ | |_) || |
 | |  | | '_ \ / _ \ '_ \ \___ \|  _ < | |
 | |__| | |_) |  __/ | | |____) | |_) || |_
  \____/| .__/ \___|_| |_|_____/|____/_____|
        | |
        |_|

Platform Name             : riscv-virtio,qemu
Platform Features         : medeleg
Platform HART Count       : 54
Platform IPI Device       : aclint-mswi
Platform Timer Device     : aclint-mtimer @ 10000000Hz
Platform Console Device   : uart8250
Platform HSM Device       : ---
Platform PMU Device       : ---
Platform Reboot Device    : syscon-reboot
Platform Shutdown Device  : syscon-poweroff
Platform Suspend Device   : ---
Platform CPPC Device      : ---
Firmware Base             : 0x80000000
Firmware Size             : 1035 KB
Firmware RW Offset        : 0x40000
Firmware RW Size          : 779 KB
Firmware Heap Offset      : 0xb2000
Firmware Heap Size        : 323 KB (total), 20 KB (reserved), 12 KB (used), 290 KB (free)
Firmware Scratch Size     : 4096 B (total), 328 B (used), 3768 B (free)
Runtime SBI Version       : 2.0

Domain0 Name              : root
Domain0 Boot HART         : 45
Domain0 HARTs             : 0*,1*,2*,3*,4*,5*,6*,7*,8*,9*,10*,11*,12*,13*,14*,15*,16*,17*,18*,19*,20*,21*,22*,23*,24*,25*,26*,27*,28
*,29*,30*,31*,32*,33*,34*,35*,36*,37*,38*,39*,40*,41*,42*,43*,44*,45*,46*,47*,48*,49*,50*,51*,52*,53*
Domain0 Region00          : 0x0000000000100000-0x0000000000100fff M: (I,R,W) S/U: (R,W)
Domain0 Region01          : 0x0000000010000000-0x0000000010000fff M: (I,R,W) S/U: (R,W)
Domain0 Region02          : 0x0000000002000000-0x000000000200ffff M: (I,R,W) S/U: ()
Domain0 Region03          : 0x0000000080000000-0x000000008003ffff M: (R,X) S/U: ()
Domain0 Region04          : 0x0000000080000000-0x00000000801fffff M: (R,W) S/U: ()
Domain0 Region05          : 0x0000000000000000-0xffffffffffffffff M: () S/U: (R,W,X)
Domain0 Next Address      : 0x0000000080200000
Domain0 Next Arg1         : 0x0000000082200000
Domain0 Next Mode         : S-mode
Domain0 SysReset          : yes
Domain0 SysSuspend        : yes

Boot HART ID              : 45
Boot HART Domain          : root
Boot HART Priv Version    : v1.12
Boot HART Base ISA        : rv64imafdch
Boot HART ISA Extensions  : sstc,zicntr,zihpm
Boot HART PMP Count       : 16
Boot HART PMP Granularity : 2 bits
Boot HART PMP Address Bits: 54
Boot HART MHPM Info       : 16 (0x0007fff8)
Boot HART MIDELEG         : 0x0000000000001666
Boot HART MEDELEG         : 0x0000000000f0b509

And system freezes here.

Others Environment:

• U-Boot 2024.07-rc1-00069-gd097f9e129
• qemu-8.2.2-1.fc40

[SiFiveHolland]

It looks like with the current master branch (86d2c17), this has been fixed at some point. I can boot with up to 107 harts. At 108 harts, the size of the heap causes the firmware overflow 2 MiB, which is the default FW_PAYLOAD_OFFSET. This makes root.next_arg1 overlap the firmware memory region, which causes sbi_domain_init() to fail. Increasing FW_PAYLOAD_OFFSET allows OpenSBI to boot with 128 harts, which is the maximum supported (SBI_HARTMASK_MAX_BITS).