F´ GDS/SeqGen performance w.r.t validation/acceptance testing

[ThibFrgsGmz]

Hello,

For a space payload demonstration project, we are considering using F´ and I am wondering if there is a tool in F´ to ensure that the product meets the customer's needs by running scenarios based on use cases.

In these scenarios, we need to define command sequences, set telemetry/events expectations, verify expectations, and generate reports.

This tool should be able to be run automatically.
It would also be interesting to run it in a CI pipeline to ensure non-regression.

It seems to me that you have a part of this in fprime-gds called fprime-seqgen.

From what I've seen, you define sequences of commands to send to the embedded system, which is awesome (if I understand what fprime-seqgen does).

But what about defining the expected data?

Do you have a companion tool to fprime-seqgen that checks that the system has nominally responded according to the product requirements?

[LeStarch]

We have a tool for running what we term "integration tests" against F´ deployments. An integration test is expected to test an integrated system (we could also call them system tests, or acceptance testing). The integration test framework connects to the GDS subsystem when it is attached to a running system (local software execution or remote hardware execution) and dispatches commands through the GDS. It can then assert on the results coming back through the GDS. A basic test could be:

1. Send COMMAND_NO_OP
2. Verify received "COMMAND_DISPATCHED" and "COMMAND_COMPLETE" events back to the GDS
3. Verify telemetry indicates 1 command dispatched

You can then setup these tests to emit commands, check results, and produce output indicating success or failure. If you are planning to execute sequences in flight, you can use the integration test api to send the command "execute sequence" and the validate that. The Ref application has a basic set of integration tests we run on it. We did wire them into CI such that they test the Ref app automatically on each PR.

References:

• Intergration Test API Documentation: https://nasa.github.io/fprime/UsersGuide/dev/testAPI/user_guide.html
• Ref App Examples: https://github.com/nasa/fprime/blob/devel/Ref/test/int/ref_integration_test.py

As a note, you run these tests using pytest and as such:

cd <Ref>
fprime-gds &  #Run the Ref app
pytest # Run tests

[ThibFrgsGmz]

Outch it seems I missed a great feature of the framework!

It seems very complete, the learning curve seems quite steep at first but it seems to do the job.

Do you use it constantly at home? Or is it a tool that few people use?

Note: no in-flight operation as we are delivering equipment to a satellite integrator. Once delivered we don't interact with it anymore (like for a star tracker).

[LeStarch]

This is a tool that has fewer adopters (currently) as it was deliver to F´ between generations of projects. It was designed and developed base on input from developers who used an older variant with the older fprime-gds.

The data plumbing is the same as the HTML gui so that should operate the same way. This just adds on a testable history and commands to assert on and send from python!

As for the learning curve, I'd focus on some examples first as those tend to be easy to understand. Then read the full documentation as it shows all the features.

I expect this tool to be used more heavily as newer projects get to the stage of needing integration tests.

[ThibFrgsGmz]

Do you know if there are any restrictions, bugs, desired improvements? Does the tool meet the needs of your internal users?
(I'm also looking for ways to contribute to improving the framework in open source rather than just using it.)

Can you tell me what time granularity we have for expectations?

For example, an M1 measure of type X expected at T0, an M2 measure of type Y expected at T0+1ms, and so on.
Is this possible?

Based on a quick review of the documentation, I believe this mechanism is enabled by the get_time() function: "These comparisons can be very useful in testing whether FSW meets timing requirements.".

[LeStarch]

The biggest improvement/bug that is needed (that I am aware of) is a cleaner way to set up the GDS connection. You'll see it is hardcoded in the constructor of the class containing tests. It would be awesome to have that moved to command line arguments or settings.ini such that it wasn't hardcoded.

As for time, events and channels come with time stamps (U32 system seconds, U32 microseconds). To test that events occurred at a specific time, you can use that. However, if your goal is to read an event, emit command, read another event, emit another command, with low latency turnaround....this will be tougher because there is a lot of communication within the GDS and so times tend to be more variable.

In this case, it would be better to dispatch a sequence, or a command that emits events at higher speed, collect the whole run on the ground, and then verify the relative timings after the run is done.

[ThibFrgsGmz]

@LeStarch I am starting to use the tool to see what it is worth.
When I run "pytest" or "pytest -s" (to get stdout of the tests), no log file/test report file is generated. Is this normal?