Allow writing JSON reporter to write directly to file

[mkamioner]

Description of the Change

Add feature for reporterOption in JSON reporter to output directly to file system. Method is similar to how xunit is implemented

Alternate Designs

It would be simple to write the JSON to the stdout and then just pipe that into a file. However when using certain tools like mocha-multi-reporters there is no way around writing JSON and the spec reporter to the stdout. Furthermore, with bash and jq JSON is a great way to parse the results while still allowing for human readable output with spec.

Why should this be in core?

There are so many different JSON reporters but non are managed and they all fall through. This is a simple fix that would be a great feature that will help many

Benefits

The ability to write JSON files of your output thereby making analyzation of your tests easier to parse

Possible Drawbacks

None that I can think of right now (happy to hear if someone has some ideas)

Applicable issues

This is an enhancement (minor release) I think

[jsf-clabot]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Mo Kamioner seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.

[coveralls]

Coverage increased (+0.02%) to 90.009% when pulling 55b1f82 on mkamioner:master into f71f347 on mochajs:master.

[boneskull]

Thanks. I'm not convinced Mocha needs this feature, since you can pipe to a file, like you mentioned.

If we do want to be able to write directly to a file (and I'm not sure how necessary that is for the userbase, because pipes), I'd rather make it happen in the general case by swapping the output stream, instead of a reporter-by-reporter basis.

Please don't let this discourage you from contributing to Mocha! There are plenty of issues labeled help wanted that we could use help with.

[plroebuck]

Guess I wasted my time doing basically the same enhancement, except I threw out the existing "json.spec.js" and based my tests on revamped version of XUnit's spec (which did stream checking). I have a use case for our project (which programmatically uses Mocha under the covers for RAML-based test automation) referenced in this issue (for one). If someone adds console.log() statements to his hook files, but specifies (for example) the Mocha json reporter -- assertions will fail due to crossing of the streams and Abao can't salvage Mocha's output. I was actually rather stunned that file output was ONLY available for XUnit reporter -- IMO, output should be trivially redirectable (-o | --output) without having to convert individual reporters as such or involve command shell piping.

[plroebuck]

@boneskull BTW, had two questions left after doing my enhancement -- could you shed some light on them?

1. These two lines. First binds and assigns to a variable. But why is the expectation seemingly referring to result of its invocation.
   

   mocha/test/reporters/xunit.spec.js

   Lines 41 to 42 in d3d7799

   	var boundXUnit = XUnit.bind({}, runner, options);
   	expect(boundXUnit).to.throwException('file output not supported in browser');

2. Reports fake times when used with sinon.useFakeTimers(). #237
   

   mocha/lib/reporters/xunit.js

   Lines 17 to 28 in d3d7799

   	/**
   	* Save timer references to avoid Sinon interfering (see GH-237).
   	*/
   	/* eslint-disable no-unused-vars, no-native-reassign */
   	var Date = global.Date;
   	var setTimeout = global.setTimeout;
   	var setInterval = global.setInterval;
   	var clearTimeout = global.clearTimeout;
   	var clearInterval = global.clearInterval;
   	/* eslint-enable no-unused-vars, no-native-reassign */

   
   

   mocha/lib/reporters/base.js

   Lines 21 to 33 in d3d7799

   	/**
   	* Save timer references to avoid Sinon interfering.
   	* See: https://github.com/mochajs/mocha/issues/237
   	*/
   	/* eslint-disable no-unused-vars, no-native-reassign */
   	var Date = global.Date;
   	var setTimeout = global.setTimeout;
   	var setInterval = global.setInterval;
   	var clearTimeout = global.clearTimeout;
   	var clearInterval = global.clearInterval;
   	/* eslint-enable no-unused-vars, no-native-reassign */

If these variables were exported from Base, would the individual reporters need to have same code repeated?

[boneskull]

1. XUnit.bind() returns a function, and that function is then run, and we assert it throws.
2. I don't think that repetition is necessary.

[loopingz]

Not having a way to redirect the json output to a file can be an issue when launched within a test suite:
nyc mocha --recursive --exit --timeout=10000 -r ./node_modules/ts-node/register src/**/*.spec.ts -R json
As it will collide with nyc output and others. We could argue that we could do two runs: one for coverage and one for unit test result but it is overkilled no?
Am I missing something obvious?

[bryansray]

@boneskull Is it possible to remove the "extra" stuff that comes out when piping? For example, when I run something like:

npx cypress run --headless --reporter json > results.json

My results.json file comes out like this:

====================================================================================================

  (Run Starting)

  ┌────────────────────────────────────────────────────────────────────────────────────────────────┐
  │ Cypress:    3.8.3                                                                              │
  │ Browser:    Electron 78 (headless)                                                             │
  │ Specs:      1 found (test.spec.js)                                                             │
  └────────────────────────────────────────────────────────────────────────────────────────────────┘


────────────────────────────────────────────────────────────────────────────────────────────────────
                                                                                                    
  Running:  test.spec.js                                                                    (1 of 1)
{
  "stats": {
    "suites": 1,
    "tests": 12,
    "passes": 11,
    "pending": 0,
    "failures": 1,
    "start": "2020-05-06T02:31:09.690Z",
    "end": "2020-05-06T02:31:14.497Z",
    "duration": 4807
  },
 ..... removed for brevity ...

Obviously this isn't a valid JSON file because of the initial output. Is it possible to get rid of that and just pipe the JSON result?

[boneskull]

that output is from cypress not mocha

[bryansray]

Oh, shoot. My apologies!

[boneskull]

regarding nyc output, you could suppress that instead and write to file. not ideal...

this being said I would support adding the ability to swap the output stream (generally just process.stdout) in mocha, but it should be generic and applicable to all reporters. someone would need to send that PR though

this would allow these sorts of situations—mocha combined with other tools—to work together more nicely.

implementation would just be something like a “default” stream for each reporter—probably in the Base reporter—which can be swapped out via reporter options.