Adding an option of built-in test variations

[are-scenic]

I've noticed that it became quite common by our developers to run the same test on several variants of certain input.

For instance, let's say we have a storage device, which is composed of physical units and there are several configurations based on capacity, i.e. each capacity has a different amount of this physical units. Each configuration might have impact on our specific test. Now, writing a test for each such configuration might seem a little hideous, not to mention redundant duplication. But if we add a new API in addition to RUN_TEST(_func_) that would be named something like RUN_TEST_VECTOR(_func_, _vector_), where _vector_ field would assume an argument of array of integers, it would help to prevent this duplication. The _func_ field will require a function that receives an integer as a parameter. This parameter can be used in the test function either as a variation value itself or a pointer to structure with proper input arguments. The macro would call the test-function for each value in the vector and display it along properly. The macro also can have a signature that assumes additional array of names for each test-variation to make print-out more readable.

What do you think?

[dmitry-kandiner]

Sounds as a good idea to me!

The parameter can also be an index in the array of input structures.

Regards,
Komrad Dmitry.

[are-scenic]

Thank you, Dmitry.

Could you please elaborate what you mean by 'index in the array of input structures'?

Thank you,
Arseniy

[dmitry-kandiner]

I mean, instead of passing to _func_ a pointer to a structure with the variation parameters, it can receive an index in a _vector_.

Regards,
Dmitry

[are-scenic]

Oh, I see now.
I apologize, I think my initial description wasn't clear enough the function assumed by _func_ will not get the vector, but an item from this vector. Actually, it's more like what you suggest, the function will receive an equivalent of providing a vector and an index.

Alternative A

By the way, alternatively, we can think of another option. The user will call something like RUN_TEST_COMBINATIONS(_func_, _names_), where _names_ would be a static array of pointers to const strings. Each string would represent a name of the combination. The macro itself will assume that _func_ has a signature like this void (*test_combination_func)(const char *combinationName_p, _U32 combinationIndex).
In this variation, instead of operating with vector of values, it will follow a paradigm similar to your suggestion. The developer will maintain his/her own test-vector the way he/she prefers, the framework will only provide an index into it.

Alternative B

We may also suggest the following alternative macro: RUN_TEST_WITH_VARIATIONS(_func_, _amount_of_variations_). This is a simplified version of the Alternative A that requires no prior creation of array of strings or some other form of test-vector.

@dmitry-kandiner what do you think?

[dmitry-kandiner]

@are-scenic , I'm not really sure which option is the best.

In case of Alternative A, raises the need to maintain the list (or the array) of variations and variation names, and check for boundaries somehow (e.g. that the parameters array and names array has the same length).

In case of Alternative B, it is not clear how the parameters array is defined, and where it is present. One option is to maintain it statically within the _func_, but then it is up to the tester to pass the correct array length to the macro RUN_TEST_WITH_VARIATIONS. On the other hand, the variation list can be explicitly defined as NULL-terminated, and the test function will return a special value, something like PASS_BUT_HAS_MORE_VARIANTS if there is still something to test against.

Dmitry.