Ref deployment integration test suite does not generate reports

[ThibFrgsGmz]

Hi, I'm experimenting with the integration test feature but I can't get the report generation to work.

From c8672e2 :

1. I built the reference deployment:

(.venv) ThibFrgsGmz@ubuntu:~/Documents/fprime/Ref$ fprime-util generate

...

(.venv) ThibFrgsGmz@ubuntu:~/Documents/fprime/Ref$ fprime-util build -j32

2. I ran the GDS separately from the deployment:

(.venv) ThibFrgsGmz@ubuntu:~/Documents/fprime/Ref$ fprime-gds --ip-address 127.0.0.1 --ip-port 50002 -n -g none

3. I run the deployment:

(.venv) ThibFrgsGmz@ubuntu:~/Documents/fprime/Ref/build-artifacts/Linux/bin$ sudo ./Ref -p 50002 -a 127.0.0.1

4. Finally, I ran the integration test suite using pytest :

(.venv) ThibFrgsGmz@ubuntu:~/Documents/fprime/Ref$ pytest -s
=========================================================================================== test session starts ===========================================================================================
linux platform -- Python 3.8.10, pytest-6.2.4, py-1.11.0, pluggy-0.13.1
rootdir: /home/ThibFrgsGmz/Documents/fprime
collected 6 items                                                                                                                                                                                         

test/int/ref_integration_test.py received channel 19205 update: 2022-06-16 16:04:28: systemResources.CPU_00 = 100.00 percent
received channel 19206 updated: 2022-06-16 16:04:28: systemResources.CPU_01 = 1.02 percent
received channel 19207 updated: 2022-06-16 16:04:28: systemResources.CPU_02 = 1.02 percent
received channel 19208 updated: 2022-06-16 16:04:28: systemResources.CPU_03 = 2.04 percent
received channel 19209 updated: 2022-06-16 16:04:28: systemResources.CPU_04 = 0.00 percent
.....Sequence of 59 bytes with timebase b 'ANY'.
CRC: 1866831267 (0x6F4595A3)
.

====================================================================================== 6 passed in 98.42s (0:01:38) =======================================================================================

But in the end, I have no output generated in the logs directory specified in the python script:

fprime/Ref/test/int/ref_integration_test.py

Line 35 in 7fb554d

logpath = os.path.join(filename, "./logs")

I was expecting a report in Excel/CSV/Text format, but the contents of the directory remains empty.
From the deployment tty, I can see that there is some activity: NO-OP command received/dispatched.

I notice that the two paths given to gdsName and fprimeName are no longer accurate since the creation of fprime-util and fprime-gds it seems. Is there a correlation?

fprime/Ref/test/int/ref_integration_test.py

Line 9 in 7fb554d

gdsName = os.path.join(filename, "../../../Gds/src")

fprime/Ref/test/int/ref_integration_test.py

Line 10 in 7fb554d

fprimeName = os.path.join(filename, "../../../Fw/Python/src")

I see that the integration test suite is executed in the CI but I don't really see that the output logs are analyzed.

fprime/ci/tests/fputil.bash

Line 105 in 7fb554d

${TIMEOUT} --kill-after=10s 180s pytest

Could you please guide me?

[LeStarch]

In order to get test reports in XLS format, you need to pip install openpyxl. Once that is done, reports are generated at the setup path. In the case of ref, the path is set to Ref/test/int/logs/ as it is releative to the .py integration file.

https://github.com/nasa/fprime/blob/devel/docs/UsersGuide/dev/testAPI/user_guide.md#api-test-log

[ThibFrgsGmz]

OK it works once the package is installed, it's great!
It's strange that I didn't get an error saying that I didn't have this package. Is this intended?

What about non-accurate paths? No trouble with the application?

[ThibFrgsGmz]

As it is a dependency of FPrime, wouldn't the openpyxl package be added in the list of dependencies to download when installing the project?

[LeStarch]

This is added as an "optional dependency" since it isn't strictly required for the base functionality of testing. Based on your experience, I'd consider adding:

1. A note/warning when the functionality is turned off
2. Just making it a required package

Thoughts?

[ThibFrgsGmz]

The first one seems important to me, without your help it would have taken me way too long to find the solution.

For the second one, it seems better to have it by default. The autogeneration of reports in Excel (even if I prefer FOSS) seems to me an important feature. It at least facilitates the QA's work to have a synthetic and readable enough information.

[LeStarch]

Good feedback. We'll take that issue.