Initial Review

[adityasaky]

I'm jotting down some initial thoughts on the setup here. I might add more as this evolves.

1. Great start! It makes sense to use the demo for this exercise.
2. I wonder if it's possible to make a template for these commands: https://github.com/PradyumnaKrishna/in-toto-testing/blob/main/tests/test_golang.py#L58-L71. The same template can be used in test_python as well with only the command swapped out.
3. Interesting to see the test on docker option. Can you elaborate some of the thinking there? I think it comes down to what we see as the aims of this exercise. If this is purely a CI thing, do we need to test in a bespoke container as well?
4. On the note of the docker environment, can the tests incorporate the use of https://github.com/in-toto/Dockerfiles images?
5. Can we bump up the test environments? In particular I want to ensure we incorporate Windows.
6. On that note, could you explore what this compatibility looks like on ARM? I think we might have some issues with in-toto-python but it's possible that's been cleaned up.

[PradyumnaKrishna]

2. I wonder if it's possible to make a template for these commands: https://github.com/PradyumnaKrishna/in-toto-testing/blob/main/tests/test_golang.py#L58-L71. The same template can be used in test_python as well with only the command swapped out.

I will look into this, the commands have different attributes in python and golang but I will check the similarity and try to build up a template if possible.

3. Interesting to see the test on docker option. Can you elaborate some of the thinking there? I think it comes down to what we see as the aims of this exercise. If this is purely a CI thing, do we need to test in a bespoke container as well?

I saw a project that does something similar, it uses docker build to test using CI. if the container build fails then the testing fails too. In our case this is a little different, we already have a docker image and we execute test under that container. If tests runs successfully then it exits with status 0 else 1.

This way we can add CI to other repositories like in-toto and in-toto-golang. Adding more command line options will make test more usable in case of pull requests. However, we have to rebuild the image if we update the tests or Dockerfile.

4. On the note of the docker environment, can the tests incorporate the use of https://github.com/in-toto/Dockerfiles images?

These images only includes in-toto python implementation, but we need to install all in-toto versions. We can use it as base to build on but have to install another version of in-toto for testing purpose.

5. Can we bump up the test environments? In particular I want to ensure we incorporate Windows.

So, what do we have to run these tests on windows as well? I guess we can do so.

6. On that note, could you explore what this compatibility looks like on ARM? I think we might have some issues with in-toto-python but it's possible that's been cleaned up.

I use M1 mac, and found the tests works well on ARM. Also, Docker tests executed successfully as well.

[adityasaky]

windows, ARM

let's add those as targets in github actions.

[PradyumnaKrishna]

windows is added in github actions. Didn't found ARM runner on github actions.

[adityasaky]

Looks like ARM is only with self hosted runners so that's a no go :(

[adityasaky]

Also, the workflow should run periodically, perhaps once a week.

[PradyumnaKrishna]

Done, I have also added matrix for latest and head versions of in-toto.

[adityasaky]

@lukpueh could you take a look at Pradyumna's work here? I wonder if we want to plan to move this into the in-toto namespace and iterate from there.