Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Kernel died on CoverageMap with 1080Ti GPU ad Docker #650

Open
lillidith opened this issue Nov 6, 2024 · 6 comments
Open

Kernel died on CoverageMap with 1080Ti GPU ad Docker #650

lillidith opened this issue Nov 6, 2024 · 6 comments

Comments

@lillidith
Copy link

Hi,
I set up the Sionna environment with docker using the associated makefile.
The Notebook Sionna_Ray_Tracing_Coverage_Map.ipynb con load the libraries, and the GPU is correctly configured.
The scene is loaded successfully, but the kernel died when I computed:

cm = scene.coverage_map(max_depth=2,           # Maximum number of ray scene interactions
                        num_samples=int(2e6), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[400, 400],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0]) # Orientation of the coverage map, e.g., could be also vertical

The same configuration works fine and smooth on Server with RTX A5000 and A100 GPUs.

Server configuration

sudo apt-get update
sudo apt-get install -y cuda-drivers-560
sudo apt-get -y install cuda-toolkit-12-6
sudo usermod -aG docker myuser
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
@merlinND
Copy link
Collaborator

merlinND commented Nov 6, 2024

Hello @lillidith,

What error message was shown in the terminal when the kernel died?
The most likely cause is running out of memory. You could try reducing num_samples and trying again.

@Fedomer
Copy link

Fedomer commented Nov 6, 2024

Same issue on a workstation with 1080Ti 11Gb:

scene = load_scene() # Load empty scene

scene.tx_array = PlanarArray(num_rows=1,
                             num_cols=1,
                             vertical_spacing=0.5,  # relative to wavelength
                             horizontal_spacing=0.5,  # relative to wavelength
                             pattern="iso",
                             polarization="V")
scene.rx_array = scene.tx_array

tx0 = Transmitter(name='tx0',
                  position=[15, -10, 20],
                  orientation=[np.pi*5/6, 0, 0],
                  power_dbm=10)
scene.add(tx0)

cm = scene.coverage_map(max_depth=5,           # Maximum number of ray scene interactions
                        num_samples=int(100), # If you increase: less noise, but more memory required
                        cm_cell_size=(5, 5),   # Resolution of the coverage map
                        cm_center=[0, 0, 0],   # Center of the coverage map
                        cm_size=[50, 50],    # Total size of the coverage map
                        cm_orientation=[0, 0, 0])

.... of corse all seem perfect in workstation with A100 40Gb but it's not a memory pb ( num_samples=int(100) )!
Compute capability? Or incompatible GPU ? I dont find a minimun requirement on hardware in the Sionna Doc site.

@merlinND
Copy link
Collaborator

merlinND commented Nov 6, 2024

What error message was shown in the terminal when the kernel died?

@merlinND
Copy link
Collaborator

merlinND commented Nov 6, 2024

By the way, as a workaround you can set the environment variable CUDA_VISIBLE_DEVICE="" before launching the Jupyter server to fall back on the CPU backend. Depending on your hardware, it could be a viable alternative.

@Fedomer
Copy link

Fedomer commented Nov 22, 2024

it's seem that with 2080Ti with 11 Gb it works so the the 1080Ti 11Gb card it's to old for sionna.

@merlinND
Copy link
Collaborator

merlinND commented Jan 3, 2025

We may be able to find out the root cause if you share the error message printed in the terminal when the kernel dies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants