Docs on running specs needed

[linusg]

Disclaimer: I'm a complete ruby noob :)

Just wrote my first couple of lines of code for natalie & found several issues (#195, #201, #217) referring to specs. While I did manage to get it to run some tests, the whole process was & still is really unclear to me, as I couldn't find any docs.

Specifically:

1. From various other PR's I noticed that spec files are added on a case by cases basis:

• Are these copied from https://github.com/ruby/spec?
• If so: why not use a submodule or separate clone of that repository? Manually copying files initially & then again for every upstream change seems incredibly cumbersome and error prone. We've had a good experience with using a plain checkout of the upstream test262 repository for Serenity's LibJS, both for local development and CI.

2. Simply doing what the GitHub actions workflow causes thousands of unstaged git changes. Surely that's not the right way for testing locally? Or is the whole idea that you only run a subset of known working tests locally (part of the natalie repo) and all of them on CI?
   https://github.com/seven1m/natalie/blob/d012e0f86304f4d1282dc77f6d588c49e31ecc40/.github/workflows/run_all_specs.yml#L21-L29
3. Is there a way to specify a subdirectory or specific spec file? I had to edit this line manually:
   https://github.com/seven1m/natalie/blob/bc22ef43ae1200eb57f33ac7459b3e3c9209f48f/spec/support/ruby_spec_runner.rb#L20
4. Is there a way of getting detailed test failure output? I only got this:

   Start running specs spec/core/string/lstrip_spec.rb
   Ran spec spec/core/string/lstrip_spec.rb
   {"Successful Examples"=>2, "Failed Examples"=>2, "Errored Examples"=>2, "Compile Errors"=>0, "Crashes"=>0, "Timeouts"=>0}
   

5. How do I install the needed dependencies for the spec runner? The command from the CI workflow (bundle exec ruby spec/support/ruby_spec_runner.rb) failed with:

   spec/support/ruby_spec_runner.rb:1:in `require': cannot load such file -- concurrent (LoadError)
       from spec/support/ruby_spec_runner.rb:1:in `<main>'
   

   and running bundle add concurrent-ruby didn't fix that. gem install concurrent-ruby + ruby spec/support/ruby_spec_runner.rb worked however. Again, no idea what I'm doing. :)

Thanks!

[seven1m]

I'll try to answer your questions here, then we can think about how to best document some of these things...

From various other PR's I noticed that spec files are added on a case by cases basis:
Are these copied from https://github.com/ruby/spec?

Yes!

If so: why not use a submodule or separate clone of that repository?

That might be possible, but I find we often have to comment out not-yet-passing bits of the spec, which would not be possible if it were a submodule. I hope at some point in the future we can just link to the specs as-is without changes, but we're not there yet. 😅

Even in your PR, we had to disable the lstrip!() and rstrip!() specs (by using xdescribe) since you did not implement the mutating versions of those methods. (Which is totally fine!)

Simply doing what the GitHub actions workflow causes thousands of unstaged git changes. Surely that's not the right way for testing locally? Or is the whole idea that you only run a subset of known working tests locally (part of the natalie repo) and all of them on CI?

Yes, that's not clear, but yes (we only run a subset of known working tests locally). We run all the specs on CI so we can get a nightly test report, which is shown on natalie-lang.org. Those are not meant to be run locally.

Only the specs in /test and /spec are to be run locally. Those are the specs we expect to run without fail.

Is there a way to specify a subdirectory or specific spec file? I had to edit this line manually:

Yep! You can just run the ruby file like you would another ruby script:

bin/natalie spec/core/string/strip_spec.rb

There is a little script helper if you want to run more than one file at a time:

bin/natalie test/runner.rb spec/core/string/*strip*

Lastly, you can run rake test to run all the local specs that we expect to pass.

Is there a way of getting detailed test failure output?

Yes. Since you were running the ruby_spec_runner.rb, which is only meant to be run by that nightly CI job, that's why you were getting the machine-readable output. You can run either rake test or bin/natalie path/to/spec to get human-readable output.

How do I install the needed dependencies for the spec runner? The command from the CI workflow (bundle exec ruby spec/support/ruby_spec_runner.rb) failed with cannot load such file -- concurrent...

Normally you would not be running all the specs like that, so you wouldn't need that dependency. But just in case your curious, it's specified in the Gemfile here, which you would have to install with the command:

bundle install --with run_all_specs

...since that group is optional.

---

Anyway! I suspect you didn't actually want all these answers for yourself, but rather for posterity and to help others not have to go through the pain that you did. 😂

I will add some help to the readme for how to get started with specs, rather than leave poor beginners to fend for themselves.

Thanks linus! 🤗

[linusg]

All questions answered, closing. Will work on some more things with this new knowledge 😄 Thanks again!

[seven1m]

I think we should rename ruby_spec_runner.rb to something like nightly_all_the_things_spec_runner.rb or something, just to further clear up any confusion about its purpose.