Why does IOMMU needs to be OFF ?

[ArtificialEU]

NVIDIA Open GPU Kernel Modules Version

X

Please confirm this issue does not happen with the proprietary driver (of the same version). This issue tracker is only for bugs specific to the open kernel driver.

• I confirm that this does not happen with the proprietary driver package.

Operating System and Version

X

Kernel Release

X

Please confirm you are running a stable release kernel (e.g. not a -rc). We do not accept bug reports for unreleased kernels.

• I am running on a stable kernel release.

Hardware: GPU

NVIDIA RtX 4090

Describe the bug

Hi,

I'm using my GPU Cluster inside a proxmox VM and i'm passing through my devices, so IOMMU off is kinda not possible, what's the reason behind it and the consequences ?

Thanks

To Reproduce

X

Bug Incidence

Once

nvidia-bug-report.log.gz

X

More Info

No response

[legraphista]

Through my troubleshooting, I found out that P2P can be enabled with IOMMU on, but it may not work as intended.
When moving data between GPUs via P2P, I was getting IO_PAGE_FAULT and the copied tensor would be allocated but zeroed out.

Disabling IOMMU fixed this behavior in my case.

Note: my SR-IOV was disabled, your mileage may vary