Integration with playwright?

[vincerubinetti]

I use https://www.npmjs.com/package/axe-playwright to run my tests, which requires you to do...

import { injectAxe, getViolations} from 'axe-playwright'

await injectAxe(page);
await configureAxe(page, { rules, checks });
const violations = await getViolations(page);

Perhaps it would be enough to export rules and checks from this library so we could use them.

Edit 1

I tried to figure out how to get the rules and checks this plugin has registered back from axe core, like axe.getRules() or axe._audit.data.checks or axe._audit.checks, but I've had no luck.

Edit 2

More info. I tried patching this package in my node_modules to return the rules and checks. When passing them to axe-playwright, I get the following error:

Error: Unexpected value

  53 |     /** setup axe */
  54 |     await injectAxe(page);
> 55 |     await configureAxe(page, { rules, checks });
     |           ^
  56 |
  57 |     /** axe check */
  58 |     const check = async () => {

    at /Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/axe-playwright/dist/index.js:64:16
    at /Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/axe-playwright/dist/index.js:31:71
    at __awaiter (/Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/axe-playwright/dist/index.js:27:12)
    at configureAxe (/Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/axe-playwright/dist/index.js:63:45)
    at /Users/vincerubinetti/Desktop/molevolvr2.0/frontend/tests/accessibility.spec.ts:55:11

Edit 3

I tried switching from axe-playwright to @axe-core/playwright, and am still running into an issue:

Error: Unexpected value

  53 |     /** axe check */
  54 |     const check = async () => {
> 55 |       const { violations } = await new AxeBuilder({ page })
     |                              ^
  56 |         .options({ rules, checks })
  57 |         .analyze();
  58 |       expect(violations).toStrictEqual([]);

    at AxeBuilder.runPartialRecursive (/Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/@axe-core/playwright/dist/index.mjs:283:34)
    at AxeBuilder.analyze (/Users/vincerubinetti/Desktop/molevolvr2.0/frontend/node_modules/@axe-core/playwright/dist/index.mjs:210:28)
    at check (/Users/vincerubinetti/Desktop/molevolvr2.0/frontend/tests/accessibility.spec.ts:55:30)
    at /Users/vincerubinetti/Desktop/molevolvr2.0/frontend/tests/accessibility.spec.ts:61:5

[vincerubinetti]

Okay, there's several issues here.

First, anything you pass to Playwright via page.evaluate must be serializable. That's understandable. So that's why the axe-playwright library doesn't work when passing a custom check with a custom evaluate function, because functions are not (easily/automatically) serializable. And @axe-core/playwright doesn't even provide an option to pass custom checks at all, lol.

So, I tried just basically copying the code from this library right into a page.evaluate statement. It depends on apca-w3 (which itself has dependencies) for the actual calculations, so I had to inject that library too in a vanilla-js fashion way. So I ended up with this absolute monstrosity:

code

await injectAxe(page);

    await page.evaluate(async () => {
      await new Promise((resolve) => {
        const script = document.createElement("script");
        document.body.append(script);
        script.type = "module";
        script.src = "https://cdn.jsdelivr.net/npm/[email protected]/+esm";
        script.addEventListener("load", resolve);
      });

      const apcaRule: Rule = {
        id: `color-contrast-apca-silver`,
        impact: "serious",
        selector: "p",
        // matches: "color-contrast-matches",
        metadata: {
          description: `Ensures the contrast between foreground and background colors meets APCA silver level conformance minimums thresholds`,
          help: "Elements must meet APCA conformance minimums thresholds",
          helpUrl:
            "https://readtech.org/ARC/tests/visual-readability-contrast/?tn=criterion",
        },
        all: [`color-contrast-apca-silver-conformance`],
        tags: ["apca", "wcag3", `apca-silver`],
      };

      const apcaCheck: Check = {
        id: `color-contrast-apca-silver-conformance`,
        metadata: {
          impact: "serious",
          messages: {
            pass: "Element has sufficient APCA silver level lightness contrast (Lc) of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Expected minimum APCA contrast of ${data.apcaThreshold}}",
            fail: {
              default:
                "Element has insufficient APCA silver level contrast of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Expected minimum APCA lightness contrast of ${data.apcaThreshold}Lc",
              increaseFont:
                "Element has insufficient APCA silver level contrast of ${data.apcaContrast}Lc (foreground color: ${data.fgColor}, background color: ${data.bgColor}, font size: ${data.fontSize}, font weight: ${data.fontWeight}). Increase font size and/or font weight to meet APCA conformance minimums",
            },
            incomplete: "Unable to determine APCA lightness contrast (Lc)",
          },
        },

        evaluate(node) {
          const style = window.getComputedStyle(node);
          const fontSize = style.getPropertyValue("font-size");
          const fontWeight = style.getPropertyValue("font-weight");
          const bgColor = axe.commons.color.getBackgroundColor(node);
          const fgColor = axe.commons.color.getForegroundColor(
            node,
            false,
            bgColor,
          );
          console.log(node, bgColor, fgColor, fontSize, fontWeight);
          if (!bgColor || !fgColor || !fontSize || !fontWeight)
            return undefined;
          const toRGBA = (color) =>
            `rgba(${color.red}, ${color.green}, ${color.blue}, ${color.alpha})`;
          const threshold = (fontSize) => {
            const size = parseFloat(fontSize);
            if (size >= 32) return 45;
            if (size >= 16) return 60;
            return 75;
          };
          const apcaContrast = Math.abs(
            calcAPCA(toRGBA(fgColor), toRGBA(bgColor)),
          );
          const apcaThreshold = threshold(fontSize);
          this.data({
            fgColor: fgColor.toHexString(),
            bgColor: bgColor.toHexString(),
            fontSize: `${(parseFloat(fontSize) * (72 / 96)).toFixed(1)}pt (${parseFloat(fontSize)}px)`,
            fontWeight,
            apcaContrast: Math.round(apcaContrast * 100) / 100,
            apcaThreshold,
            messageKey: apcaThreshold === null ? "increaseFont" : "default",
          });
          return apcaThreshold ? apcaContrast >= apcaThreshold : false;
        },
      };

      const config: Spec = {
        checks: [apcaCheck],
        rules: [apcaRule, { id: "color-contrast", enabled: false }],
      };

      await window.axe.configure(config);
      await window.axe.run();
    });

So, one thing that might help is if this apca-check library could export a fully bundled single js file as part of its release process, so I could just inject the whole thing into the Playwright browser like:

const apcaScriptString = readFileSync(resolve("apca-check/apca-check.min.js"));
await page.evaluate(
  (scriptString) => window.eval(scriptString),
  scriptString
);

But, even when I did the above monstrosity, I still ran into issues. It was at least able to register the rules and checks. Running Playwright in --debug mode, and opening the browser's console after injecting the axe script and rules/checks, I could see that color-contrast-apca-silver was in axe.getRules() and axe._audit.rules and axe._audit.checks.

But I got no violations on a test where it definitely should've failed: a paragraph with white text and a light beige background. Note the console.log in my above code. It wasn't even checking the paragraph! For some reason, the matches: "color-contrast-matches" was only checking like 3 elements on the whole page. That is really concerning to me. Like, has Axe this whole time been checking only a fraction of my page elements in the standard tests?

Finally I forced it to selector: "p",, but the rule still failed to report a problem. Yes I'm using the looser bronze threshold function, but making it way more strict still did not report an error.

---

So....!!! I've wasted way too much time on this, so I'm just giving up. Someone else is going to have to figure this out. I'll check color contrast manually with Lighthouse whenever I have the time. Or maybe I'll try moving to the Lighthouse CLI tool.