Phase out runtest files

[metan-ucw]

The runtest files are not flexible enough and maintaining them by hand is error prone. My idea for the future would be to phase out the runtest files eventually and provide more flexible filtering capabilities in the test executing framework based on the test metadata.

The runtest files actually provide two different purposes. Firstly tests are split into different categories, some categories are overlapping and some categories are subset of others. Secondly the runtest files allow us to add test variants, e.g. run test with different sets of parameters.

The idea to replace the test categories is actually quite simple, instead of maintaining list of tests the tests would be filtered by metadata. We would start with the full set of tests and filter out user requested sets. Everything in the syscalls/ directory would be collected as a syscall test. Then we would add lists of syscalls per special purpose, e.g. sysvipc file would contain names of all sysvipc syscalls that could then be again filtered based on the metadata. If we wanted to run CVE tests we would select tests with a CVE tag. All of this should be actually present in the test metadata already, for new library testcases only though. We can easily start to prototype this filtering capability soon enough and it will be fully usable once we are done with test conversions into the new test library.

Second part would be a bit more complicated, we would have to move the test variants somewhere else. I would vote to move them into the tests themselves and into the tst_test structure. That way the test binary would include list of all variants and we could also add shortcuts to running the different variants, e.g. -t 2 would mean run test variant 2 and -t would list all pre-defined variants. This would also be parsed into the metadata and the test runner framework could use that information to schedule tests as well.

[metan-ucw]

@tbird20d FYI

[acerv]

This ticket would make sense in the LTP issues list as well.

[metan-ucw]

@acerv I guess that the part about moving test variants into the test could be tracked as a LTP ticket as well, will you create that one or should I do so?

[acerv]

@metan-ucw you have a better vision on the specific implementation. Thanks!