Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Before submission, please check that ...
you updated necessary documentation within retest/docs.Description
Bug
This especially noticeable, when testing the same website multiple times (different browsers, screen resolutions, etc.). Because the same site is tested multiple times, the same differences will occur within the different tests. The example below shows a test performed with two different screen resolutions, executed twice to account for the Golden Master creation.
These changes are present twice and thus are expected to be accepted with a single method call:
However, when accepting this within review, each change has to be accepted individually to be counted for the model and accepted into the Golden Master.
Note that review (develop) already uses a similar technique to identify similar changes, which are not propagated properly to the
GlobalChangeSetApplier
because of this bug.Workaround
Instead of using the
hashCode
, use the reducedidentifier
methods of bothIdentifyingAttributes
andAttributeDifferences
to only take into account important attributes. The former therefore will only take thepath
,type
andsuffix
attribute, while the latter will only takeactual
andexpected
attributes. We might want to change this in the future, but it seems to work for now.I would consider this more of a workaround, since per definition all
IdentifyingAttributes
should only hold stable attributes. Therefore I would consider implementing retest/recheck-web#200, but that might be a major breaking change.Test
To have a test as close to the actual behavior, I introduced an integration test. Since manually putting together a report is cumbersome and we do not have a migration strategy for file based reports to load from, the integration test performs a full recheck lifecycle using the example above to create a report.
Since we are collecting the report within a separate class
SuiteAggregator
it has to be reset before the test begins to prevent other tests to contribute differences to the report. I am aware that this is not a clean solution as it relies on side effects due to the usage of static singletons. However, alternative solutions would be either introducing a new constructor that takes such aSuiteAggregator
or exposing the current test report fromRecheckImpl
which both then would be public API. If the current solution does not suit you, I would be in favor of exposing the current report as that would be the lowest impact.Due to the broken down identifier methods, the contents of the various lookup methods cannot be compared anymore. Therefore I decided to only use simple
hasSize
-checks to test for the validity of the generated lookup tables, as all other possible checks (e.g. accepting changes) seemed to be very cumbersome to properly check.This has been verified manually that it works with review