Loading torch script model failed

[DongYanqi1029]

When I tried to load pytorch model in my passive component, my input port handler function can not work normally, and the entire deployment workflow is stuck at this input port handler function when I run fprime-gds. Then I tried to use try&catch to catch exceptions, still nothing show up in my fprime log.
`
void MyVehicleActor::stepIn_handler( const NATIVE_INT_TYPE portNum)
{
this->log_ACTIVITY_LO_STEP_PORT_ACTIVATED();

std::cout << "Loading Model" << std::endl;
try {
  torch::jit::script::Module model = torch::jit::load(std::string("/home/usr/catkin_ws/src/my_vehicle_learning/training_results/actor_LaserNet.pt"));
}
catch (std::exception& e) {
  std::cout << "Error occured" << std::endl;
}

std::cout << "Finish Loading Model" << std::endl;

}
`

[codeflight1]

I don't know anything about torch, but it looks like you should be loading the module in the constructor or init function, and then using it in some way in the input handler.

[DongYanqi1029]

Thanks, I will give it a try. ^_^

[DongYanqi1029]

The problem still exists, I try to load my model in the component init function, it worked. But when I try to use the following code, the whole deployment is stucked again.
`
void MyVehicleActor ::
init(
const NATIVE_INT_TYPE instance
)
{
MyVehicleActorComponentBase::init(instance);

std::vector<float> scan(180, 0);
this->obs.assign(scan.begin(), scan.end());

std::cout << "Loading Model" << std::endl;
this->model = torch::jit::load(std::string("/home/godfather/catkin_ws/src/my_vehicle_learning/training_results/actor_LaserNet.pt"));
this->model.to(torch::kCPU);
std::cout << "Finish Loading Model" << std::endl;

}

void MyVehicleActor ::
stepIn_handler(
const NATIVE_INT_TYPE portNum
)
{
this->log_ACTIVITY_LO_STEP_PORT_ACTIVATED();

auto opts = torch::TensorOptions().dtype(torch::kFloat32);
at::Tensor tensor = torch::from_blob(obs.data(), {int64_t(obs.size())}, opts).clone();

}

`

[codeflight1]

I'm not exactly sure what you are trying to do here, as there are no data inputs to the handler function. Could you explain the scenario a bit more?
Also note that things you log will not show up in the stdout of fprime-gds, you have to check the files in the logs folder.

[DongYanqi1029]

I am using two components "MyVehicleController" and "MyVehicleActor" to control a turtlebot in the Gazebo Simulation. "MyVehicleController" send instructions to "MyVehicleActor" through invoking ports, and the Actor publishes messages to related ROS topics to control the turtlebot.
What I am trying to do here is to load my pytorch model in "MyVehicleActor", and when "MyVehicleController" invokes the "step" port , it means that MyVehicleActor should use the pytorch model to tell the turtlebot what to do.
My problem is that when I am trying to generate a tensor as the pytorch network input, the program stops after sending out event in the "stepIn_handler" function. The same code works just fine outside F Prime.
The code I am using now is as follows.
`namespace Ref {

// ----------------------------------------------------------------------
// Construction, initialization, and destruction
// ----------------------------------------------------------------------

MyVehicleActor ::
MyVehicleActor(
const char *const compName
) : MyVehicleActorComponentBase(compName)
{

}

void MyVehicleActor ::
init(
const NATIVE_INT_TYPE instance
)
{
MyVehicleActorComponentBase::init(instance);

std::vector<float> scan(180, 0);
this->obs.assign(scan.begin(), scan.end());

std::cout << "Loading Model" << std::endl;
this->model = torch::jit::load(std::string("/home/godfather/catkin_ws/src/my_vehicle_learning/training_results/actor_LaserNet.pt"));
this->model.to(torch::kCPU);
std::cout << "Finish Loading Model" << std::endl;

}

MyVehicleActor ::
~MyVehicleActor()
{

}

// ----------------------------------------------------------------------
// Handler implementations for user-defined typed input ports
// ----------------------------------------------------------------------

void MyVehicleActor ::
closeLaunchIn_handler(
const NATIVE_INT_TYPE portNum
)
{
this->log_ACTIVITY_LO_CLOSE_LAUNCH_PORT_ACTIVATED();
}

void MyVehicleActor ::
resetEnvIn_handler(
const NATIVE_INT_TYPE portNum
)
{
this->log_ACTIVITY_LO_RESET_ENV_PORT_ACTIVATED();
}

void MyVehicleActor ::
startLaunchIn_handler(
const NATIVE_INT_TYPE portNum,
I32 sim_count
)
{
this->log_ACTIVITY_LO_START_LAUNCH_PORT_ACTIVATED();
}

void MyVehicleActor ::
stepIn_handler(
const NATIVE_INT_TYPE portNum
)
{
this->log_ACTIVITY_LO_STEP_PORT_ACTIVATED();

auto opts = torch::TensorOptions().dtype(torch::kFloat32);
at::Tensor tensor = torch::from_blob(obs.data(), {int64_t(obs.size())}, opts).clone();
tensor = tensor.unsqueeze(0);
std::vector<torch::jit::IValue> inputs;
inputs.push_back(tensor);  

}

} `

[DongYanqi1029]

the "obs" variable is std::vector

[ThibFrgsGmz]

What operating system/hardware platform are you using? WSL-2? Linux/RPI?

[DongYanqi1029]

I am implementing my components in Linux

[DongYanqi1029]

ubunutu 20.04

[LeStarch]

Does anything appear in the running binary's logs? If you ran the binary using fprime-gds it will be on the Logs tab of the GUI. Otherwise, it will be in standard out of the console where you ran the binary.

[DongYanqi1029]

Nothing appears in the log tab, when the program invoke the port handler where i try to generate a tensor, the log tab just stop publishing any message but the binary kept running.

[DongYanqi1029]

This odd thing occured when I tried to use torch::jit::load() or torch::from_blob() function in the port handler, no error or exception info, but when I move the code to the component class construction, the code worked just fine. I'm confused.

[codeflight1]

Are you sure the handler in being invoked? What is it connected to?

[DongYanqi1029]

I am pretty sure the handler is invoked because I called function this->log_ACTIVITY_LO_STEP_PORT_ACTIVATED();, and I can see the corresponding event in the log tab. This port connects my components MyVehicleController and MyVehicleActor, the Controller invoke this port to tell the Actor to take some actions in the handler.

[DongYanqi1029]

This odd thing happened only when certain functions are called in the port handler or command handler, I'm confused what's exactly happening in those handlers that causes this weird error.

[ThibFrgsGmz]

Are you allowed to share it in open source to push the investigation a bit further? With a dummy model if there is an IP issue maybe?

And maybe configure it to open with GitPod to ease the configuration process and be ready to code/investigate?

PS: an example of GitPod setup for Fprime is available here: #788