Use EISOP CF sources

[wmdietl]

This is a follow-up to #116.
Instead of merging into the main branch and breaking CI, this PR merges into a new main-eisop branch. CI for that branch will still fail, but we'll have a clean main branch and can have smaller PRs against main-eisop until that branch passes CI or is considered good enough.

[cpovirk]

We have a SHALLOW environment variable that users can set to enable this. That might be better than hard-coding (which, admittedly, we do for the JDK, but it's truly huge). I don't remember exactly how much thought we gave to all this, though.

[netdpb]

The point of $SHALLOW is to allow CIs like GitHub actions to avoid cloning any commits they don't care about. If you're just running tests for CI you don't need to check any past commits out.

I think if someone has their own fork of eisop we wouldn't want them to clone only one branch. I would suggest continuing to rely on $SHALLOW (defined externally) for checker-framework.

[cpovirk]

Not really related to this PR, but:

Use --filter=blob:none instead of --depth 1

I wonder if that's worth investigating someday.

[wmdietl]

I had a quick look at this post:
https://github.blog/2020-12-21-get-up-to-speed-with-partial-clone-and-shallow-clone/

It seems to generally prefer the blob-less mode. For CI, where one only has a single build, continuing to use --depth 1 seems okay. But as this script is for developer set-up, using blob-less might be better.

[netdpb]

Let's postpone that to a different PR. It's not related to switching to EISOP.

[netdpb]

The point of $SHALLOW is to allow CIs like GitHub actions to avoid cloning any commits they don't care about. If you're just running tests for CI you don't need to check any past commits out.

I think if someone has their own fork of eisop we wouldn't want them to clone only one branch. I would suggest continuing to rely on $SHALLOW (defined externally) for checker-framework.

[netdpb]

Interesting!

tests.ConformanceTest > conformanceTests FAILED
    expected:
        # 10 pass; 6 fail; 16 total; 62.5% score
        ...
    but was:
        # 4 pass; 12 fail; 16 total; 25.0% score

but

tests.ConformanceTest > conformanceTestsOnSamples FAILED
    expected:
        # 75 pass; 498 fail; 573 total; 13.1% score
        ...
    but was:
        # 224 pass; 349 fail; 573 total; 39.1% score

Score went up from 13.1% to 39.1%. Woohoo!

[netdpb]

I see a failure of depending on that zip artifact. Do you need to merge main in again?

Once that's fixed, you can run the following to generate changes to the conformance test reports to reflect the current behavior:

JSPECIFY_CONFORMANCE_TEST_MODE=write ./gradelew conformanceTests

[wmdietl]

I see a failure of depending on that zip artifact. Do you need to merge main in again?

It looks like this PR is even with main and I don't see a failure about a zip file. Can you point me to the failure?
I think I got this error locally when I hadn't updated my local jspecify clone.

I do see this failure:

> Task :conformanceTests

tests.ConformanceTest > conformanceTests FAILED
    java.nio.file.NoSuchFileException: /home/wdietl/Sync/wmdietl/workspaces/jspecify/jspecify-reference-checker-use-eisop/build/conformanceTests/assertions/org/jspecify/conformance/tests

locally and on CI.

I tried cleaning, updating, and rebuilding everything, but I still get this error.
Did you mean this failure?

[netdpb]

Yes, that's the error I mean. That directory should have been created when unzipping.

[wmdietl]

Yes, that's the error I mean. That directory should have been created when unzipping.

This was jspecify/jspecify#436 fixed by jspecify/jspecify#437.