[Bug Report] Data collection failed while running in headless mode

[fan-ziqi]

Describe the bug

Using headless mode can significantly improve the speed of collect data. Headless can be used normally in the matterport environment, but it cannot be directly used in carla and warehouse environment. The script must be run with non-headless mode in carla and warehouse environment once, and then stop the script and then can using headless mode. The problem is that hits is empty in following code:

viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py

Lines 883 to 910 in 7d481c1

	def _raycast_usd_stage(
	self,
	ray_starts: torch.Tensor,
	ray_directions: torch.Tensor,
	max_dist: float = 1e6,
	return_distance: bool = False,
	return_normal: bool = False,
	return_class: bool = False,
	) -> tuple[torch.Tensor, torch.Tensor | None, torch.Tensor | None, list | None]:
	"""
	Perform raycasting over the entire loaded stage.
	Interface is the same as the normal raycast_mesh function without the option to provide specific meshes.
	"""
	hits = [
	get_physx_scene_query_interface().raycast_closest(carb.Float3(ray_single), carb.Float3(ray_dir), max_dist)
	for ray_single, ray_dir in zip(ray_starts.cpu().numpy(), ray_directions.cpu().numpy())
	]
	# get all hit idx
	hit_idx = [idx for idx, single_hit in enumerate(hits) if single_hit["hit"]]
	# hit positions
	hit_positions = torch.zeros_like(ray_starts).fill_(float("inf"))
	hit_positions[hit_idx] = torch.tensor([single_hit["position"] for single_hit in hits if single_hit["hit"]]).to(
	ray_starts.device
	)

error:

/home/ubuntu/anaconda3/envs/viplanner/lib/python3.10/site-packages/torch/functional.py:513: UserWarning: torch.meshgrid: in an upcoming release, it will be required to pass the indexing argument. (Triggered internally at ../aten/src/ATen/native/TensorShape.cpp:3609.)
  return _VF.meshgrid(tensors, **kwargs)  # type: ignore[attr-defined]
Traceback (most recent call last):
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/standalone/data_collect.py", line 136, in <module>
    main()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/standalone/data_collect.py", line 118, in main
    samples = explorer.sample_viewpoints(args_cli.num_samples)
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/viewpoint_sampling.py", line 66, in sample_viewpoints
    self.terrain_analyser.analyse()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 74, in analyse
    self.construct_height_map()
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 770, in construct_height_map
    hit_point = self._raycast_usd_stage(
  File "/home/ubuntu/workspaces/viplanner_ws/viplanner/omniverse/extension/omni.viplanner/omni/viplanner/collectors/terrain_analysis.py", line 908, in _raycast_usd_stage
    hit_positions[hit_idx] = torch.tensor([single_hit["position"] for single_hit in hits if single_hit["hit"]]).to(
RuntimeError: shape mismatch: value tensor of shape [0] cannot be broadcast to indexing result of shape [0, 3]

I don't know how to fix this. After fixing this issue, the data_collect.py script should run in headless mode to accelerate collection.

System Info

Describe the characteristic of your environment:

• Commit: [e.g. 8f3b9ca]
• Isaac Sim Version: 4.2
• OS: Ubuntu 20.04
• GPU: e.g. RTX 4090
• CUDA: 12

Additional context

Add any other context about the problem here.

Checklist

• I have checked that there is no similar issue in the repo (required)
• I have checked that the issue is not in running Isaac Sim itself and is related to the repo

Acceptance Criteria

Add the criteria for which this task is considered done. If not known at issue creation time, you can add this once the issue is assigned.

• Criteria 1
• Criteria 2