feat: document/test support for axe-core v3.4.1 and axe-cli v3.2.0 (#175

) #### Description of changes Neither axe-core v3.4.1 nor axe-cli v3.2.0 contain any breaking output format changes, so axe-sarif-converter was already compatible with both of them. This PR updates our test cases to include verification of this compatibility. This PR also checks in code to generate test cases for future axe-core updates, and documents the process for performing such an update in the README under a new subsection of `Contributing`. Review suggestions: * Start from the `README.md` changes; this provides a broad understanding of the changes and the expected usage of the new generator. * For the files under `/src/test-resources/*.{json|sarif}`, review by comparing against the similar files for old axe versions to satisfy yourself that they've been generated consistently and follow the same format. * Review all other code per normal (particularly the new code under `/src/test-resources/generator`) #### Pull request checklist - [x] PR title respects [Conventional Commits](https://www.conventionalcommits.org) (starts with `fix:`, `feat:`, etc, and is suitable for user-facing release notes) - [x] PR contains no breaking changes, **OR** description of both PR **and final merge commit** starts with `BREAKING CHANGE:` - [n/a] (if applicable) Addresses issue: #0000 - [x] Added relevant unit tests for your changes - [x] Ran `yarn precheckin` - [x] Verified code coverage for the changes made
microsoft · Jan 7, 2020 · 97f6a41 · 97f6a41
1 parent b5d72fa
commit 97f6a41
Show file tree

Hide file tree

Showing 33 changed files with 58,564 additions and 46 deletions.
diff --git a/.gitignore b/.gitignore
@@ -62,4 +62,5 @@ typings/
 
 # build output
 /dist/
+/src/test-resources/generator/*.js
 /test-results/
diff --git a/README.md b/README.md
@@ -66,7 +66,7 @@ The [microsoft/axe-pipelines-samples](https://github.com/microsoft/axe-pipelines
 
 The version number of this library is **independent** from the version numbers of the axe-core inputs and SARIF outputs it supports.
 
--   axe-sarif-converter version 2.x supports input from version ^3.2.0 of axe-core (tested with 3.2.2 and 3.3.2) and outputs SARIF v2.1
+-   axe-sarif-converter version 2.x supports input from version ^3.2.0 of axe-core (tested with 3.2.2, 3.3.2, and 3.4.1) and outputs SARIF v2.1
 -   axe-sarif-converter version 1.x supports input from version >= 3.2.0 < 3.3.0 of axe-core (tested with 3.2.2) and outputs SARIF v2.0
 
 Note that the SARIF format _does not use semantic versioning_, and there are breaking changes between the v2.0 and v2.1 SARIF formats. If you need compatibility with a SARIF viewer that only supports v2.0, you should use version 1.x of this library.
@@ -91,6 +91,33 @@ To get started working on the project:
     - `node dist/cli.js`
     - Alternately, register a linked global `axe-sarif-converter` command with `npm install && npm link` (yarn doesn't work for this; see [yarnpkg/yarn#1585](https://github.com/yarnpkg/yarn/issues/1585))
 
+### Updating axe-core version
+
+This package attempts to maintain backwards compatibility with axe-core versions ^3.2.2. We maintain
+test cases using pinned output from multiple axe-core versions under `/src/test-resources/`, so updating
+the version of axe-core we support involves generating new output for the new versions.
+
+Ideally we'd specify axe-core as a peer dependency; unfortunately, changing this now would be a breaking
+change, so we're waiting to change this until we would need to make a breaking change anyway.
+
+To update the package and test cases to account for a new axe-core version:
+
+1. Update the version of axe-core in `yarn.lock` (_not_ `package.json`); usually dependabot will cover this.
+1. Update the versions of axe-cli and axe-core in `src/test-resources/generator/package.json`
+1. Generate test resource files for the new version with:
+
+    ```
+    cd src/test-resources/generator
+    yarn install
+    yarn generate
+    ```
+
+1. Manually compare the diff of `/src/test-resources/basic-axe-vPREVIOUS.sarif` and `/src/test-resources/basic-axe-vNEW.sarif`; the only differences should be the version numbers.
+1. Manually compare the diff of `/src/test-resources/w3citylights-axe-vPREVIOUS.sarif` and `/src/test-resources/w3citylights-axe-vNEW.sarif`; in addition to version number differences, you should see some differences based on new/removed rules between the axe versions.
+1. Add test cases involving the new files to the integration tests in `src/index.test.ts` and `src/cli.test.ts`
+1. Update snapshots (`yarn test -u`)
+1. Update this README's `Version numbers` section to note which versions we've tested against.
+
 ### Contributor License Agreement
 
 This project welcomes contributions and suggestions. Most contributions require you to agree to a

diff --git a/jest.config.js b/jest.config.js
@@ -18,6 +18,7 @@ module.exports = {
     collectCoverageFrom: [
         '<rootDir>/**/*.ts',
         '!<rootDir>/**/*.test.ts',
+        '!<rootDir>/test-resources/generator/**/*.ts',
         // The CLI is tested via integration tests that spawn separate node
         // processes, so coverage information on this file isn't accurate
         '!<rootDir>/cli.ts',

diff --git a/src/__snapshots__/index.test.ts.snap b/src/__snapshots__/index.test.ts.snap
@@ -737,9 +737,9 @@ Object {
       },
       "invocations": Array [
         Object {
-          "endTimeUtc": "2019-03-22T19:12:06.129Z",
+          "endTimeUtc": "2000-01-02T03:04:05.006Z",
           "executionSuccessful": true,
-          "startTimeUtc": "2019-03-22T19:12:06.129Z",
+          "startTimeUtc": "2000-01-02T03:04:05.006Z",
         },
       ],
       "results": Array [],
@@ -1386,15 +1386,15 @@ Object {
       ],
       "tool": Object {
         "driver": Object {
-          "downloadUri": "https://www.npmjs.com/package/axe-core/v/3.3.2",
-          "fullName": "axe for Web v3.3.2",
+          "downloadUri": "https://www.npmjs.com/package/axe-core/v/3.4.1",
+          "fullName": "axe for Web v3.4.1",
           "informationUri": "https://www.deque.com/axe/axe-for-web/",
           "name": "axe-core",
           "properties": Object {
             "microsoft/qualityDomain": "Accessibility",
           },
           "rules": Array [],
-          "semanticVersion": "3.3.2",
+          "semanticVersion": "3.4.1",
           "shortDescription": Object {
             "text": "An open source accessibility rules library for automated testing.",
           },
@@ -1405,7 +1405,7 @@ Object {
               "name": "WCAG",
             },
           ],
-          "version": "3.3.2",
+          "version": "3.4.1",
         },
       },
     },

diff --git a/src/cli.test.ts b/src/cli.test.ts
@@ -42,18 +42,31 @@ describe('axe-sarif-converter CLI', () => {
         }
     });
 
-    it('supports conversion from axe-cli style list of results', async () => {
-        const outputFile = path.join(testResultsDir, 'axe-cli.sarif');
-        await deleteIfExists(outputFile);
-
-        const output = await invokeCliWith(`-i ${axeCliFile} -o ${outputFile}`);
-
-        expect(output.stderr).toBe('');
-        expect(output.stdout).toBe('');
-
-        const outputJson = JSON.parse((await readFile(outputFile)).toString());
-        expect(outputJson.runs.length).toBe(1);
-    });
+    it.each`
+        inputFile
+        ${'w3citylights-axe-v3.3.2.axe-cli-v3.1.1.json'}
+        ${'basic-axe-v3.4.1.axe-cli-v3.2.0.json'}
+        ${'w3citylights-axe-v3.4.1.axe-cli-v3.2.0.json'}
+    `(
+        'supports conversion from axe-cli output $inputFile',
+        async ({ inputFile }) => {
+            const inputFilePath = path.join(testResourcesDir, inputFile);
+            const outputFile = path.join(testResultsDir, `${inputFile}.sarif`);
+            await deleteIfExists(outputFile);
+
+            const output = await invokeCliWith(
+                `-i ${inputFilePath} -o ${outputFile}`,
+            );
+
+            expect(output.stderr).toBe('');
+            expect(output.stdout).toBe('');
+
+            const outputJson = JSON.parse(
+                (await readFile(outputFile)).toString(),
+            );
+            expect(outputJson.runs.length).toBe(1);
+        },
+    );
 
     it('supports basic conversion with short-form i/o args', async () => {
         const outputFile = path.join(testResultsDir, 'basic_short.sarif');
@@ -158,10 +171,6 @@ describe('axe-sarif-converter CLI', () => {
         testResourcesDir,
         'basic-axe-v3.3.2.sarif',
     );
-    const axeCliFile = path.join(
-        testResourcesDir,
-        'w3citylights-axe-v3.3.2.axe-cli-v3.1.1.json',
-    );
 
     const mkdir = promisify(fs.mkdir);
     const writeFile = promisify(fs.writeFile);

diff --git a/src/index.test.ts b/src/index.test.ts
@@ -12,6 +12,7 @@ import * as fs from 'fs';
 import * as path from 'path';
 import { AxeRawResult } from './axe-raw-result';
 import { convertAxeToSarif, SarifLog, sarifReporter } from './index';
+import { testResourceTimestampPlaceholder } from './test-resource-constants';
 
 function readTestResourceJSON(testResourceFileName: string): any {
     const rawFileContents: string = fs.readFileSync(
@@ -52,6 +53,10 @@ describe('public convertAxeToSarif API', () => {
         ${'basic-axe-v3.3.2.reporter-v2.json'}        | ${'basic-axe-v3.3.2.sarif'}
         ${'w3citylights-axe-v3.3.2.reporter-v1.json'} | ${'w3citylights-axe-v3.3.2.sarif'}
         ${'w3citylights-axe-v3.3.2.reporter-v2.json'} | ${'w3citylights-axe-v3.3.2.sarif'}
+        ${'basic-axe-v3.4.1.reporter-v1.json'}        | ${'basic-axe-v3.4.1.sarif'}
+        ${'basic-axe-v3.4.1.reporter-v2.json'}        | ${'basic-axe-v3.4.1.sarif'}
+        ${'w3citylights-axe-v3.4.1.reporter-v1.json'} | ${'w3citylights-axe-v3.4.1.sarif'}
+        ${'w3citylights-axe-v3.4.1.reporter-v2.json'} | ${'w3citylights-axe-v3.4.1.sarif'}
     `(
         'converts pinned v1/v2 input $inputFile to pinned output $outputFile',
         ({ inputFile, outputFile }) => {
@@ -71,11 +76,11 @@ require('axe-core');
 describe('public sarifReporter API', () => {
     const emptyAxeRunOptions = {};
 
-    // Normalized values are the pinned expectations from basic-axe-v3.2.2-sarif-v2.1.2.sarif
+    // Normalized values are the pinned expectations from generated test-resources files
     function normalizeEnvironmentDerivedSarifProperties(sarif: SarifLog): void {
         sarif.runs[0]!.invocations!.forEach(i => {
-            i.endTimeUtc = '2019-03-22T19:12:06.129Z';
-            i.startTimeUtc = '2019-03-22T19:12:06.129Z';
+            i.endTimeUtc = testResourceTimestampPlaceholder;
+            i.startTimeUtc = testResourceTimestampPlaceholder;
         });
     }
 
@@ -96,7 +101,7 @@ describe('public sarifReporter API', () => {
     // it isn't very meaningful to test cases that involve old axe versions here.
     it.each`
         inputFile                               | outputFile
-        ${'basic-axe-v3.3.2.reporter-raw.json'} | ${'basic-axe-v3.3.2.sarif'}
+        ${'basic-axe-v3.4.1.reporter-raw.json'} | ${'basic-axe-v3.4.1.sarif'}
     `(
         'converts pinned raw input $inputFile to pinned output $outputFile',
         async ({ inputFile, outputFile }) => {

diff --git a/src/test-resource-constants.ts b/src/test-resource-constants.ts
@@ -0,0 +1,4 @@
+// Copyright (c) Microsoft Corporation. All rights reserved.
+// Licensed under the MIT License.
+
+export const testResourceTimestampPlaceholder = '2000-01-02T03:04:05.006Z';
diff --git a/src/test-resources/basic-axe-v3.2.2.reporter-v2.json b/src/test-resources/basic-axe-v3.2.2.reporter-v2.json
@@ -13,7 +13,7 @@
         "orientationAngle": 0,
         "orientationType": "landscape-primary"
     },
-    "timestamp": "2019-03-22T19:12:06.129Z",
+    "timestamp": "2000-01-02T03:04:05.006Z",
     "url": "http://localhost/",
     "toolOptions": {
         "reporter": "v2"

diff --git a/src/test-resources/basic-axe-v3.2.2.sarif b/src/test-resources/basic-axe-v3.2.2.sarif
@@ -65,8 +65,8 @@
         },
         "invocations": [
           {
-            "startTimeUtc": "2019-03-22T19:12:06.129Z",
-            "endTimeUtc": "2019-03-22T19:12:06.129Z",
+            "startTimeUtc": "2000-01-02T03:04:05.006Z",
+            "endTimeUtc": "2000-01-02T03:04:05.006Z",
             "executionSuccessful": true
           }
         ],

diff --git a/src/test-resources/basic-axe-v3.3.2.reporter-v1.json b/src/test-resources/basic-axe-v3.3.2.reporter-v1.json
@@ -13,7 +13,7 @@
     "orientationAngle": 0,
     "orientationType": "portrait-primary"
   },
-  "timestamp": "2019-03-22T19:12:06.129Z",
+  "timestamp": "2000-01-02T03:04:05.006Z",
   "url": "http://localhost/",
   "toolOptions": {
     "reporter": "v1"

diff --git a/src/test-resources/basic-axe-v3.3.2.reporter-v2.json b/src/test-resources/basic-axe-v3.3.2.reporter-v2.json
@@ -13,7 +13,7 @@
     "orientationAngle": 0,
     "orientationType": "portrait-primary"
   },
-  "timestamp": "2019-03-22T19:12:06.129Z",
+  "timestamp": "2000-01-02T03:04:05.006Z",
   "url": "http://localhost/",
   "toolOptions": {
     "reporter": "v2"

diff --git a/src/test-resources/basic-axe-v3.3.2.sarif b/src/test-resources/basic-axe-v3.3.2.sarif
@@ -65,8 +65,8 @@
       },
       "invocations": [
         {
-          "startTimeUtc": "2019-03-22T19:12:06.129Z",
-          "endTimeUtc": "2019-03-22T19:12:06.129Z",
+          "startTimeUtc": "2000-01-02T03:04:05.006Z",
+          "endTimeUtc": "2000-01-02T03:04:05.006Z",
           "executionSuccessful": true
         }
       ],

diff --git a/src/test-resources/basic-axe-v3.4.1.axe-cli-v3.2.0.json b/src/test-resources/basic-axe-v3.4.1.axe-cli-v3.2.0.json
@@ -0,0 +1,67 @@
+[
+  {
+    "inapplicable": [],
+    "incomplete": [],
+    "passes": [],
+    "testEngine": {
+      "name": "axe-core",
+      "version": "3.4.1"
+    },
+    "testEnvironment": {
+      "orientationAngle": 0,
+      "orientationType": "landscape-primary",
+      "userAgent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) HeadlessChrome/79.0.3945.88 Safari/537.36",
+      "windowHeight": 600,
+      "windowWidth": 800
+    },
+    "testRunner": {
+      "name": "axe"
+    },
+    "timestamp": "2020-01-07T01:18:34.868Z",
+    "toolOptions": {
+      "reporter": "v1",
+      "runOnly": {
+        "type": "rule",
+        "values": [
+          "document-title"
+        ]
+      }
+    },
+    "url": "file:///Q:/repos/axe-sarif-converter/src/test-resources/basic.html",
+    "violations": [
+      {
+        "description": "Ensures each HTML document contains a non-empty <title> element",
+        "help": "Documents must have <title> element to aid in navigation",
+        "helpUrl": "https://dequeuniversity.com/rules/axe/3.4/document-title?application=webdriverjs",
+        "id": "document-title",
+        "impact": "serious",
+        "nodes": [
+          {
+            "all": [],
+            "any": [
+              {
+                "data": null,
+                "id": "doc-has-title",
+                "impact": "serious",
+                "message": "Document does not have a non-empty <title> element",
+                "relatedNodes": []
+              }
+            ],
+            "failureSummary": "Fix any of the following:\n  Document does not have a non-empty <title> element",
+            "html": "<html class=\"deque-axe-is-ready\"><head></head><body>\n</body><script>document.documentElement.classList.add(\"deque-axe-is-ready\");</script></html>",
+            "impact": "serious",
+            "none": [],
+            "target": [
+              "html"
+            ]
+          }
+        ],
+        "tags": [
+          "cat.text-alternatives",
+          "wcag2a",
+          "wcag242"
+        ]
+      }
+    ]
+  }
+]
diff --git a/src/test-resources/basic-axe-v3.4.1.reporter-raw.json b/src/test-resources/basic-axe-v3.4.1.reporter-raw.json
@@ -0,0 +1,45 @@
+[
+  {
+    "id": "document-title",
+    "result": "failed",
+    "pageLevel": false,
+    "impact": "serious",
+    "tags": [
+      "cat.text-alternatives",
+      "wcag2a",
+      "wcag242"
+    ],
+    "description": "Ensures each HTML document contains a non-empty <title> element",
+    "help": "Documents must have <title> element to aid in navigation",
+    "helpUrl": "https://dequeuniversity.com/rules/axe/3.4/document-title?application=axe-puppeteer",
+    "inapplicable": [],
+    "passes": [],
+    "incomplete": [],
+    "violations": [
+      {
+        "any": [
+          {
+            "id": "doc-has-title",
+            "data": null,
+            "relatedNodes": [],
+            "impact": "serious",
+            "message": "Document does not have a non-empty <title> element"
+          }
+        ],
+        "all": [],
+        "none": [],
+        "node": {
+          "selector": [
+            "html"
+          ],
+          "source": "<html><head></head><body>\n</body></html>",
+          "xpath": [
+            "/html"
+          ]
+        },
+        "impact": "serious",
+        "result": "failed"
+      }
+    ]
+  }
+]