kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

ljmotta · 2025-01-21T22:08:30Z

Closes apache/incubator-kie-issues#1114
Closes #2839

Description

This PR introduces containerized end-to-end (E2E) Playwright tests to ensure consistent and reliable test execution across environments.

Tests were executed in the following environments:

macOS
- arm64
- Sonoma 14.6.1
- Podman Desktop v1.15
- Podman Engine 5.3.1
- Docker CLI 27.2.1
Windows WSL
- amd64
- Ubuntu 22.04.3 LTS
- Docker CE CLI 25.0.2
- Docker CE Engine 25.0.2

Known Issues

Running the online-editor containerized test-e2e script on macOS requires starting the services in a separate terminal.
Performance: Tests may run slower on macOS due to architecture differences.
Flakiness: Retries are enabled because tests can occasionally fail without valid reasons when running in the restricted containerized environment.
Debugging tests is not currently supported but could be added in the future.
The Docker CLI is required to run the containerized tests.

Trying it out

Install the dependencies.
Build the @kie-tools/playwright-base image with the following command:

KIE_TOOLS_BUILD__buildContainerImages=true pnpm -F @kie-tools/playwright-base image:docker:build

In a package that contains Playwright e2e tests, run the following:

KIE_TOOLS_BUILD__runContainerizedEndToEndTests=true pnpm test-e2e

This command will start the required services and run the tests inside the container.

Updating screenshots

To update screenshots, use the pnpm test-e2e:container:shell script to open a shell inside the container:

pnpm test-e2e:container:shell
# In the container shell, manually call the `test-e2e:run` passing the `-u` or `--update-snapshots` flag
pnpm test-e2e:run -u
# or
pnpm test-e2e:run --update-snapshots

To improve performance, you can filter which tests will run by adding the -g flag:

pnpm test-e2e:run -u -g "test name"

…ne-editor

jomarko · 2025-01-22T09:28:20Z

Thank you @ljmotta . Please, let us know this is ready for a verification. I will try on non-MAC environment.

ljmotta · 2025-01-23T10:51:29Z

@jomarko This PR can be tested for macOS and Linux. I still need to make some adjustments for native Windows (non WSL).

thiagoelg

Awesome PR @ljmotta! This will make our e2e tests more stable and concise!

I've made some comments below about the package.json scripts; let me know what you think.

packages/playwright-base/package.json

packages/online-editor/package.json

Co-authored-by: Thiago Lugli <[email protected]>

ljmotta · 2025-01-23T22:14:49Z

Status update regarding Windows. My current setup with Docker CLI + Docker WSL didn't work, as WSL doesn't handle Windows symlinks [1][2], which is heavily used by pnpm, so mounting the Windows volume in the container doesn't work. I'll try a different approach with Docker CLI + Podman Desktop to check. Podman works a little different, but it also uses WSL behind the scenes so I'm not too confident, but I'll give it a shot.

[1] microsoft/WSL#5118
[2] microsoft/WSL#12250

tiagobento · 2025-01-23T22:33:38Z

Maybe try hard links @ljmotta? I think we had a similar issue with maven-m2-repo-via-http-inage.

ljmotta · 2025-01-24T13:51:01Z

@tiagobento The symlinks are located in the node_modules directory, so switching to hard links might not be straightforward. After experimenting, I made some progress with volumes using Docker Desktop and HyperV. However, I encountered two additional issues:

It appears that playwright Windows browsers cannot be used inside the container. This means the browsers need to be reinstalled. Furthermore, the playwright binary is unavailable, requiring a full reinstallation of playwright, including erasing node_modules [1].
To address the first issue, all untracked files (e.g., node_modules) need to be removed. However, since pnpm creates recursive structures with symlinks, the git clean -fdx command cannot be used [2][3].

Even if the second issue were resolved, having to fix the first issue every time a container is started would be impractical. Considering these challenges, I am leaning toward not supporting Windows development without WSL for this pull request.

[1] microsoft/playwright#23098 (comment)
[2] git-for-windows/git#3358
[3] pnpm/pnpm#4401

tiagobento · 2025-01-24T15:23:55Z

@ljmotta I'm ok with not supporting Windows without WSL here, just please add a new statement to the README making that explicit. And thank you so much for the effort!

packages/playwright-base/Containerfile

packages/playwright-base/README.md

tiagobento · 2025-01-27T15:27:40Z

packages/root-env/env/index.js

+      containerizedEndToEndTests: {
+        run: str2bool(getOrDefault(this.vars.KIE_TOOLS_BUILD__runContainerizedEndToEndTests)),
+      },


So I think the KIE_TOOLS_BUILD__runEndToEndTests env var should still be the one use to toggle the execution of the E2E tests, and this new flag should only toggle the MODE with which the tests run.

KIE_TOOLS_BUILD__runEndToEndTests=false # E2E tests don't run and `KIE_TOOLS_BUILD__runContainerizedEndToEndTests` is ignored.

KIE_TOOLS_BUILD__runEndToEndTests=true KIE_TOOLS_BUILD__runContainerizedEndToEndTests=true # E2E tests run inside containers

KIE_TOOLS_BUILD__runEndToEndTests=true KIE_TOOLS_BUILD__runContainerizedEndToEndTests=false (default) # E2E tests run on host machine (without containers involved)

WDYT?

packages/dmn-editor/tests-e2e/__containerization__/playwright-docker-compose.yml

Co-authored-by: Tiago Bento <[email protected]>

ljmotta added 21 commits April 11, 2024 21:49

initial steps to create a playwright environment with docker

c432d4b

Merge remote-tracking branch 'origin/main' into playwright-docker

6c1ac33

Add container solution to generate playwright screenshots

1d47181

run bash script after up

1a44663

Update container image and scripts

7fad5fa

Use run script os in check script, and use container on all runs

227aac0

Add containerizedEndToEndTests

1fe89ff

Improve scripts and use dynamic host

09eb7c0

Add missing cross-env

f53f4b3

Rollback operator.yml

dd1b1e4

Simplify CI

69fd7c3

package.json adjustments

1a81a4b

Simplify container file

ce08de1

Make it compatible with macOS

5963a30

Use nginx to forward host.docker.internal to localhost - missing onli…

c42ba1e

…ne-editor

Add reverser proxies for other services

fad1d53

Rollback CI

9ff8897

Writting playwright-base README

1f9945a

Oops

5cfabe0

Clean up

3b89a5b

Update README and enable retries

f4cc454

ljmotta added area:dependencies Pull requests that update a dependency file area:tests labels Jan 21, 2025

ljmotta self-assigned this Jan 21, 2025

ljmotta requested a review from tiagobento as a code owner January 21, 2025 22:08

Remove unused dependency

5d7a64e

ljmotta added the pr: wip PR is still under development label Jan 21, 2025

thiagoelg reviewed Jan 23, 2025

View reviewed changes

packages/playwright-base/package.json Outdated Show resolved Hide resolved

packages/online-editor/package.json Outdated Show resolved Hide resolved

Break into build steps to isolate the env var

3b5e2f2

Co-authored-by: Thiago Lugli <[email protected]>

Simplify scripts and improve docker-compose files

7250866

ljmotta added 2 commits January 24, 2025 15:00

Update README

4091abc

Improve docker-compose file

5d4128f

tiagobento reviewed Jan 24, 2025

View reviewed changes

packages/playwright-base/Containerfile Outdated Show resolved Hide resolved

tiagobento reviewed Jan 24, 2025

View reviewed changes

packages/playwright-base/README.md Outdated Show resolved Hide resolved

Bump up to node22

635655b

tiagobento reviewed Jan 27, 2025

View reviewed changes

packages/dmn-editor/tests-e2e/__containerization__/playwright-docker-compose.yml Outdated Show resolved Hide resolved

tiagobento reviewed Jan 27, 2025

View reviewed changes

packages/dmn-editor/tests-e2e/__containerization__/playwright-docker-compose.yml Outdated Show resolved Hide resolved

tiagobento reviewed Jan 27, 2025

View reviewed changes

packages/dmn-editor/tests-e2e/__containerization__/playwright-docker-compose.yml Outdated Show resolved Hide resolved

ljmotta and others added 2 commits January 27, 2025 17:11

Review and add upgrade for nginx

7c3a69d

Fix screenshots typo

baf3ba4

Co-authored-by: Tiago Bento <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

ljmotta commented Jan 21, 2025 •

edited

Loading

jomarko commented Jan 22, 2025

ljmotta commented Jan 23, 2025

thiagoelg left a comment

ljmotta commented Jan 23, 2025

tiagobento commented Jan 23, 2025

ljmotta commented Jan 24, 2025 •

edited

Loading

tiagobento commented Jan 24, 2025

tiagobento Jan 27, 2025

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

Are you sure you want to change the base?

kie-issues#1114: Containerize Playwright end-to-end tests to resolve screenshot comparison issues caused by OS differences #2866

Conversation

ljmotta commented Jan 21, 2025 • edited Loading

Description

Known Issues

Trying it out

Updating screenshots

jomarko commented Jan 22, 2025

ljmotta commented Jan 23, 2025

thiagoelg left a comment

Choose a reason for hiding this comment

ljmotta commented Jan 23, 2025

tiagobento commented Jan 23, 2025

ljmotta commented Jan 24, 2025 • edited Loading

tiagobento commented Jan 24, 2025

tiagobento Jan 27, 2025

Choose a reason for hiding this comment

ljmotta commented Jan 21, 2025 •

edited

Loading

ljmotta commented Jan 24, 2025 •

edited

Loading