Add the ability to fail performance test if one version is significantly slower then another.

[lightsofapollo]

No description provided.

[lightsofapollo]

My working theory is something like this:

var versions = [
  ['super-old', ICAL_Y],
  ['previous',  ICAL_X],
  ['latest', ICAL]
];

The new benchmarking framework will run through each and fail if latest is significantly slower then another version.... I will probably make this opt-in something like the following:

perf.suite(function(perf) {
  /* fail tests in this suite if latest is more then 20% slower then any other version */
  perf.failThreshold(0.2);
});

[lightsofapollo]

@kewisch feedback?

[kewisch]

Yes, sounds good to me. It would also be good if there were some perf report that summarizes the difference per test and even better if this were visible on the travis build.

[lightsofapollo]

@kewisch I will see what I can do- for now keep in mind (if you have not seen this already) that we currently dump this information in travis now. We just don't do any fancy error throwing if things are slower ;(

[kewisch]

I've disabled the perf tests on CI to save some cycles, as the new test I have added in c50c765 takes a while to complete.

[kewisch]

I'm going to close this for a bit of cleanup, if someone comes in with a need for perf testing and has smart ideas on how to run them in a reliable fashion on github actions CI, I'm all for it.