Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

robot state publisher crashing #49

Open
ashBabu opened this issue Feb 28, 2024 · 9 comments
Open

robot state publisher crashing #49

ashBabu opened this issue Feb 28, 2024 · 9 comments

Comments

@ashBabu
Copy link

ashBabu commented Feb 28, 2024

Ubuntu 22.04, ROS2 Humble, Gazebo Sim, version 7.7.0.

The iris_runway.launch.py or the maze dont even start whereas the wildthumper_playpen shows something in gazebo although there are error msgs on not able to load sensors. The log is as follows

[~/ros2_ws] - ros2 launch ardupilot_gz_bringup iris_runway.launch.py 
[INFO] [launch]: Default logging verbosity is set to INFO
namespace:        
transport:        udp4
middleware:       dds
verbose:          4
discovery:        7400
port:             2019
refs:             /home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/dds_xrce_profile.xml
command:          arduplane
model:            json
speedup:          1
slave:            0
sim_address:      127.0.0.1
instance:         0
defaults:         /home/ash/ros2_ws/install/ardupilot_gazebo/share/ardupilot_gazebo/config/gazebo-iris-gimbal.parm,/home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/default_params/dds_udp.parm
synthetic_clock:  True
command:          mavproxy.py
master:           tcp:127.0.0.1:5760
sitl:             127.0.0.1:5501
out:              127.0.0.1:14550
[INFO] [ruby $(which ign) gazebo-1]: process started with pid [390291]
[INFO] [ruby $(which ign) gazebo-2]: process started with pid [390294]
[INFO] [micro_ros_agent-3]: process started with pid [390297]
[INFO] [dds_udp.parm --synthetic-clock -4]: process started with pid [390300]
[INFO] [mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]: process started with pid [390303]
[INFO] [robot_state_publisher-6]: process started with pid [390307]
[INFO] [rviz2-7]: process started with pid [390309]
[dds_udp.parm --synthetic-clock -4] Setting SIM_SPEEDUP=1.000000
[dds_udp.parm --synthetic-clock -4] Starting SITL: JSON
[dds_udp.parm --synthetic-clock -4] JSON control interface set to 127.0.0.1:9002
[dds_udp.parm --synthetic-clock -4] Starting sketch 'ArduPlane'
[dds_udp.parm --synthetic-clock -4] Starting SITL input
[dds_udp.parm --synthetic-clock -4] Using Irlock at port : 9005
[dds_udp.parm --synthetic-clock -4] Waiting for connection ....
[dds_udp.parm --synthetic-clock -4] bind port 5760 for SERIAL0
[dds_udp.parm --synthetic-clock -4] SERIAL0 on TCP port 5760
[micro_ros_agent-3] [1709120146.169920] info     | UDPv4AgentLinux.cpp | init                     | running...             | port: 2019
[micro_ros_agent-3] [1709120146.170453] info     | Root.cpp           | set_verbose_level        | logger setup           | verbose_level: 4
[robot_state_publisher-6] [ERROR] [1709120146.316153064] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-6] [ERROR] [1709120146.316222425] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-6] Failed to parse robot description using: sdformat_urdf_plugin/SDFormatURDFParser
[robot_state_publisher-6] terminate called after throwing an instance of 'std::runtime_error'
[robot_state_publisher-6]   what():  Unable to initialize urdf::model from robot description
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 5: Msg: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 5: Msg: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 9: Msg: Failed to load a world.
[dds_udp.parm --synthetic-clock -4] Connection on serial port 5760
[dds_udp.parm --synthetic-clock -4] Loaded defaults from /home/ash/ros2_ws/install/ardupilot_gazebo/share/ardupilot_gazebo/config/gazebo-iris-gimbal.parm,/home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/default_params/dds_udp.parm
[dds_udp.parm --synthetic-clock -4] Home: -35.363262 149.165237 alt=584.000000m hdg=353.000000
[ruby $(which ign) gazebo-1] [Msg] Ignition Gazebo Server v6.16.0
[ruby $(which ign) gazebo-1] [Msg] Loading SDF world file[/home/ash/ros2_ws/install/ardupilot_gz_gazebo/share/ardupilot_gz_gazebo/worlds/iris_runway.sdf].
[ruby $(which ign) gazebo-1] [Dbg] [gz.cc:410] Shutting down ign-gazebo-server
[INFO] [ruby $(which ign) gazebo-1]: process has finished cleanly [pid 390291]
[INFO] [launch]: process[ruby $(which ign) gazebo-1] was required: shutting down launched system
[INFO] [rviz2-7]: sending signal 'SIGINT' to process[rviz2-7]
[INFO] [robot_state_publisher-6]: sending signal 'SIGINT' to process[robot_state_publisher-6]
[INFO] [mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]: sending signal 'SIGINT' to process[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]
[INFO] [dds_udp.parm --synthetic-clock -4]: sending signal 'SIGINT' to process[dds_udp.parm --synthetic-clock -4]
[INFO] [micro_ros_agent-3]: sending signal 'SIGINT' to process[micro_ros_agent-3]
[INFO] [ruby $(which ign) gazebo-2]: sending signal 'SIGINT' to process[ruby $(which ign) gazebo-2]
[rviz2-7] [INFO] [1709120146.436794442] [rclcpp]: signal_handler(signum=2)
[ERROR] [micro_ros_agent-3]: process has died [pid 390297, exit code -2, cmd '/home/ash/ros2_ws/install/micro_ros_agent/lib/micro_ros_agent/micro_ros_agent udp4 --middleware dds --verbose 4 --port 2019 --refs /home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/dds_xrce_profile.xml --ros-args -r __node:=micro_ros_agent -r __ns:=/'].
[rviz2-7] [INFO] [1709120146.573630162] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-7] [INFO] [1709120146.573657832] [rviz2]: OpenGl version: 4.6 (GLSL 4.6)
[rviz2-7] [INFO] [1709120146.586730385] [rviz2]: Stereo is NOT SUPPORTED
[rviz2-7] terminate called after throwing an instance of 'rclcpp::exceptions::RCLError'
[rviz2-7]   what():  failed to create guard condition: the given context is not valid, either rcl_init() was not called or rcl_shutdown() was called., at ./src/rcl/guard_condition.c:67
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] qrc:/qml/StyleDialog.qml:112:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] qrc:/qml/StyleDialog.qml:105:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] qrc:/qml/StyleDialog.qml:98:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] qrc:qml/Main.qml:102:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] qrc:/qml/PluginMenu.qml:27:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ruby $(which ign) gazebo-2] [GUI] [Wrn] [Application.cc:797] [QT] file::/Gazebo/GazeboDrawer.qml:242:3: QML Connections: Implicitly defined onFoo properties in Connections are deprecated. Use this syntax instead: function onFoo(<arguments>) { ... }
[ERROR] [robot_state_publisher-6]: process has died [pid 390307, exit code -6, cmd '/opt/ros/humble/lib/robot_state_publisher/robot_state_publisher --ros-args -r __node:=robot_state_publisher --params-file /tmp/launch_params_0tcs77rn --params-file /tmp/launch_params_lgvabq_y'].
[dds_udp.parm --synthetic-clock -4] No JSON sensor message received, resending servos
[ERROR] [rviz2-7]: process has died [pid 390309, exit code -6, cmd '/opt/ros/humble/lib/rviz2/rviz2 -d /home/ash/ros2_ws/install/ardupilot_gz_bringup/share/ardupilot_gz_bringup/rviz/iris.rviz --ros-args'].
[dds_udp.parm --synthetic-clock -4] No JSON sensor message received, resending servos
[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] Connect tcp:127.0.0.1:5760 source_system=255
[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] Log Directory: 
[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] Telemetry log: mav.tlog
[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] Waiting for heartbeat from tcp:127.0.0.1:5760
[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] link 1 down
[dds_udp.parm --synthetic-clock -4] No JSON sensor message received, resending servos
[dds_udp.parm --synthetic-clock -4] No JSON sensor message received, resending servos
[ERROR] [mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]: process[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[ERROR] [dds_udp.parm --synthetic-clock -4]: process[dds_udp.parm --synthetic-clock -4] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[ERROR] [ruby $(which ign) gazebo-2]: process[ruby $(which ign) gazebo-2] failed to terminate '5' seconds after receiving 'SIGINT', escalating to 'SIGTERM'
[INFO] [mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]: sending signal 'SIGTERM' to process[mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]
[INFO] [dds_udp.parm --synthetic-clock -4]: sending signal 'SIGTERM' to process[dds_udp.parm --synthetic-clock -4]
[INFO] [ruby $(which ign) gazebo-2]: sending signal 'SIGTERM' to process[ruby $(which ign) gazebo-2]
[ERROR] [mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive -5]: process has died [pid 390303, exit code -15, cmd 'mavproxy.py --out 127.0.0.1:14550 --out 127.0.0.1:14551 --master tcp:127.0.0.1:5760 --sitl 127.0.0.1:5501 --non-interactive '].
[ERROR] [dds_udp.parm --synthetic-clock -4]: process has died [pid 390300, exit code -15, cmd 'arduplane --model json --speedup 1 --slave 0 --sim-address=127.0.0.1 --instance 0 --defaults /home/ash/ros2_ws/install/ardupilot_gazebo/share/ardupilot_gazebo/config/gazebo-iris-gimbal.parm,/home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/default_params/dds_udp.parm --synthetic-clock '].
[ERROR] [ruby $(which ign) gazebo-2]: process has died [pid 390294, exit code -15, cmd 'ruby $(which ign) gazebo -v4 -g --force-version 6'].
[INFO] [launch]: process[ruby $(which ign) gazebo-2] was required: shutting down launched system
@Ryanf55
Copy link
Collaborator

Ryanf55 commented Apr 20, 2024

Can you test again on harmonic?

Error from above logs:

[robot_state_publisher-6] [ERROR] [1709120146.316153064] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-6] [ERROR] [1709120146.316222425] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-6] Failed to parse robot description using: sdformat_urdf_plugin/SDFormatURDFParser
[robot_state_publisher-6] terminate called after throwing an instance of 'std::runtime_error'
[robot_state_publisher-6]   what():  Unable to initialize urdf::model from robot description
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 5: Msg: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 5: Msg: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[ruby $(which ign) gazebo-1] [Err] [Server.cc:139] Error Code 9: Msg: Failed to load a world.

@lrabius
Copy link

lrabius commented Jun 27, 2024

From the logs it looks like you build the workspace for ignition gazebo, but the only supported versions are garden or harmonic.
I had similar problems. You must have installed the right gazebo version, like the gz-garden package as described in the wiki. And before doing a clean build of the workspace the same gazebo version must be also set in the terminal with export GZ_VERSION=garden.

Then you should see $(which gz) instead of $(which ign) in your logs.

@EricPedley
Copy link

EricPedley commented Oct 30, 2024

I have the same problem and I've isolated it to just the robot_state_publisher package. In my case I don't even need to be running gazebo in the launch file to have this problem. Here's some terminal history that illustrates the problem:

miller@eros:~/code/ardu_ws/src/uavf_2025
> echo $GZ_VERSION
harmonic
miller@eros:~/code/ardu_ws/src/uavf_2025
> ros2 launch uavf_2025 state_pub.launch.py
[INFO] [launch]: All log files can be found below /home/miller/.ros/log/2024-10-30-16-51-37-927343-eros-45120
[INFO] [launch]: Default logging verbosity is set to INFO
[INFO] [robot_state_publisher-1]: process started with pid [45121]
[robot_state_publisher-1] [ERROR] [1730332298.247053684] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-1] [ERROR] [1730332298.247142213] [sdformat_urdf]: Attempting to load a Sensor, but the provided sensor type is missing or invalid.
[robot_state_publisher-1] Failed to parse robot description using: sdformat_urdf_plugin/SDFormatURDFParser
[robot_state_publisher-1] terminate called after throwing an instance of 'std::runtime_error'
[robot_state_publisher-1]   what():  Unable to initialize urdf::model from robot description
[ERROR] [robot_state_publisher-1]: process has died [pid 45121, exit code -6, cmd '/opt/ros/humble/lib/robot_state_publisher/robot_state_publisher --ros-args -r __node:=robot_state_publisher --params-file /tmp/launch_params_20r5hyw2 --params-file /tmp/launch_params_3lrtqvwy'].
miller@eros:~/code/ardu_ws/src/uavf_2025
> cat launch/state_pub.launch.py 
import os

from ament_index_python.packages import get_package_share_directory

from launch import LaunchDescription

from launch_ros.actions import Node



def generate_launch_description():
    """Generate a launch description for a iris quadcopter."""
    pkg_ardupilot_gazebo = get_package_share_directory("ardupilot_gazebo")

    # Robot description.

    # Ensure `SDF_PATH` is populated as `sdformat_urdf`` uses this rather
    # than `GZ_SIM_RESOURCE_PATH` to locate resources.
    if "GZ_SIM_RESOURCE_PATH" in os.environ:
        gz_sim_resource_path = os.environ["GZ_SIM_RESOURCE_PATH"]

        if "SDF_PATH" in os.environ:
            sdf_path = os.environ["SDF_PATH"]
            os.environ["SDF_PATH"] = sdf_path + ":" + gz_sim_resource_path
        else:
            os.environ["SDF_PATH"] = gz_sim_resource_path

    # Load SDF file.
    sdf_file = os.path.join(
        pkg_ardupilot_gazebo, "models", "iris_with_gimbal", "model.sdf"
    )
    with open(sdf_file, "r") as infp:
        robot_desc = infp.read()
        # print(robot_desc)

    # Publish /tf and /tf_static.
    robot_state_publisher = Node(
        package="robot_state_publisher",
        executable="robot_state_publisher",
        name="robot_state_publisher",
        output="both",
        parameters=[
            {"robot_description": robot_desc},
            {"frame_prefix": ""},
        ],
    )



    return LaunchDescription(
        [
            robot_state_publisher,
        ]
    )

On my other machine where this runs successfully, the output looks like this instead:

[robot_state_publisher-1] [WARN] [1730333179.357768094] [sdformat_urdf]: SDFormat link [base_link] has a <sensor>, but URDF does not support this
[robot_state_publisher-1] [WARN] [1730333179.357815298] [sdformat_urdf]: SDFormat link [imu_link] has a <sensor>, but URDF does not support this
[robot_state_publisher-1] [WARN] [1730333179.357879338] [sdformat_urdf]: SDFormat link [tilt_link] has a <sensor>, but URDF does not support this
[robot_state_publisher-1] [WARN] [1730333179.357905954] [sdformat_urdf]: SDFormat link [avoidance_cam_left_link] has a <sensor>, but URDF does not support this

I tracked down where in the source code of sdformat_urdf that warning comes from, but I can't seem to find where the error message is coming from. https://github.com/ros/sdformat_urdf/blob/4be22a54e48562323f3e9deecbe1bde894eb7508/sdformat_urdf/src/sdformat_urdf.cpp#L465
Searching the string "Attempting" across the entire repo history yields no results (git grep Attempting $(git rev-list --all))

@Ryanf55
Copy link
Collaborator

Ryanf55 commented Oct 31, 2024

What's SDF_PATH and GZ_SIM_RESOURCE_PATH in your shell set to?

env | grep SDF_PATH
env | grep GZ_SIM_RESOURCE_PATH

@EricPedley
Copy link

SDF_PATH is empty on both the computer that it's broken on, and the computer it works on. GZ_SIM_RESOURCE_PATH looks like this on the broken system:

/home/miller/code/ardu_ws/install/ardupilot_sitl_models/share/ardupilot_sitl_models/worlds:/home/miller/code/ardu_ws/install/ardupilot_sitl_models/share/ardupilot_sitl_models/models:/home/miller/code/ardu_ws/install/ardupilot_sitl_models/share:/home/miller/code/ardu_ws/install/ardupilot_gz_gazebo/share/ardupilot_gz_gazebo/worlds:/home/miller/code/ardu_ws/install/ardupilot_gz_description/share/ardupilot_gz_description/models:/home/miller/code/ardu_ws/install/ardupilot_gz_description/share:/home/miller/code/ardu_ws/install/ardupilot_gazebo/share/ardupilot_gazebo/worlds:/home/miller/code/ardu_ws/install/ardupilot_gazebo/share/ardupilot_gazebo/models:/home/miller/code/ardu_ws/install/ardupilot_gazebo/share

@EricPedley
Copy link

EricPedley commented Oct 31, 2024

Truly inexplicable: I just deleted the install and build directories from my ardu_ws folder, did colcon build again, and it gets the warning instead of an error. I guess next time I get a weird bug like this I should move them instead so I can do a diff to see what changed.

@EricPedley
Copy link

EricPedley commented Oct 31, 2024

Before fixing this, QGC was only getting like halfway through the connection progress bar when I launched iris_runway.launch.py and I couldn't make the drone take off, but now it works fine (after a minute of prearm failing because of no position estimate). IDK if robot_state_publisher failing to run has downstream effects on the rest of the stuff in that launch file running correctly, or if my build was just messed up in other ways that got fixed after a clean rebuild.

@srmainwaring
Copy link
Collaborator

srmainwaring commented Oct 31, 2024

[~/ros2_ws] - ros2 launch ardupilot_gz_bringup iris_runway.launch.py 
[INFO] [launch]: Default logging verbosity is set to INFO
namespace:        
transport:        udp4
middleware:       dds
verbose:          4
discovery:        7400
port:             2019
refs:             /home/ash/ros2_ws/install/ardupilot_sitl/share/ardupilot_sitl/config/dds_xrce_profile.xml
command:          arduplane
model:            json

@ashBabu - the original log does not look as if it launching the correct binary. An iris_runway.launch.py should not be using the arduplane command (should be arducopter). Not sure if this is then causing issues downstream for other nodes that get launched in the script, but it indicates something was not correct in configuration.

@srmainwaring
Copy link
Collaborator

srmainwaring commented Oct 31, 2024

Truly inexplicable: I just deleted the install and build directories from my ardu_ws folder, did colcon build again, and it gets the warning instead of an error. I guess next time I get a weird bug like this I should move them instead so I can do a diff to see what changed.

@EricPedley, this may be caused by a version of the iris model not suitable for using with ROS 2 being present in the GZ_SIM_RESOURCE_PATH.

You can check the contents of the SDF file by enabling the print statement in

# Load SDF file.
sdf_file = os.path.join(
pkg_ardupilot_gazebo, "models", "iris_with_gimbal", "model.sdf"
)
with open(sdf_file, "r") as infp:
robot_desc = infp.read()
# print(robot_desc)
# Publish /tf and /tf_static.
robot_state_publisher = Node(
package="robot_state_publisher",
executable="robot_state_publisher",
name="robot_state_publisher",
output="both",
parameters=[
{"robot_description": robot_desc},
{"frame_prefix": ""},
],
)

If it contains model:// prefixes in the <uri> elements, it will not load.

See also: #59 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants