Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Catch2 integration does not handle Generators inside sections #205

Open
Manu343726 opened this issue Sep 19, 2022 · 1 comment
Open

Catch2 integration does not handle Generators inside sections #205

Manu343726 opened this issue Sep 19, 2022 · 1 comment

Comments

@Manu343726
Copy link

Per Catch2 documentation:

GENERATE can be seen as an implicit SECTION, that goes from the place GENERATE is used, to the end of the scope.
[...]
The fact that GENERATE introduces a virtual SECTION can also be used to make a generator replay only some SECTIONs, without having to explicitly add a SECTION

I looked into the Catch2 integration and it seems like the Catch2 API does not include generator information as part of the sector:

// Approvals/integrations/catch/Cathc2Approvals.h

struct Catch2TestCommitRevert : Catch::TestEventListenerBase
{
    ...
    virtual void sectionStarting(Catch::SectionInfo const& sectionInfo) override
    {
        currentTest.sections.push_back(sectionInfo.name); // Name here does not include the generator value
    }
    ...
};

In fact, if you look at the Catch::SectionInfo type it only includes name and source location. I'm no expert on the Catch2 internals but at first glance, I would say Catch2 generators are handled more like part of the body of a section rather than variants of the parent section (Like generated tests cases in other frameworks).

The result of this behavior is that if you run an approval inside a section with a generator, the approval is uniquely registered for the section rather than each generated variant of the section, resulting in approval failures following the execution of the different variants of the section:

enum class Keyword
{
    Unknown, Foo, Bar, Quux
};

Keyword parseKeyword(const std::string_view word);

SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
     const std::string_view word = GENERATE("foo", "bar", "quux");
     Approvals::verify(parseKeyword(word)); // If we approve "foo", it fails in the next iteration of the generator
}
@Manu343726
Copy link
Author

If I manually generate the sections it works as intended:

SECTION("Verify our parsing algorithm works with the most relevant words in the vocabulary of a programmer")
{
    for(const auto word : {"foo", "bar", "quux"})
    {
        SECTION(fmt::format("works with '{}'", word))
        {
            const std::string_view word = GENERATE("foo", "bar", "quux");
            Approvals::verify(parseKeyword(word)); // Now works, one approval per word
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant