Github Release Process and OVAL repository structure questions

[vanderpol]

@OVAL-Community/oval-board-members
We are planning to proceed with using github's release process to make each release. As such we should consider what files should be included in said release. Github includes every directory/file from the entire repository in the "source code.zip" file for each build.

Should we make a OVAL-Board repository contain non-OVAL schema related files such as?

• Area Supervisors
• OVAL- Board (old archive of board meetings prior to sharing via Discussions)
• guidelines (more than just guidelines, contains all of they python code for generating documentation from .xsd files
• tools (just some transforms)

I feel like the release should include

1. oval-schemas
2. oval-specifications
3. oval-documentation (offline HTML generated files from .xsd files, generated from the 'guidelines' data.

Feedback/thoughts?

[solind]

I agree that the schema should be its own project, without the clutter that you are suggesting to pull out.

I assume the documentation will be generated by a script as part of the release process? (i.e., you're not proposing to include generated HTML documentation in the repository itself).

[DeafBDoor]

As I suggested in the other thread, we may use the github pipelines to clean/filter the artifacts which will be released. This would allows us to use the same project, instead of creating additional ones.

[solind]

I think the only benefit to leaving that information in the repo would be if we wanted to make references to it from markdown pages.

You're quite right that the release downloads don't need to resemble a ZIP of the entire repository. Commonly, they're build artifacts. Relevant build artifacts for the OVAL repo would include schemas (potentially stripped of documentation), schematron XSDs, and the aforementioned HTML documentation.

[vanderpol]

Sounds like I may need to do some more learning on github pipelines. Sorry, just a noob here and leading the ship, I'll get there.

I'm trying to learn how the existing documentation was generated, I see the transforms in "tools", and the process for creating HTML files in "guidelines", but I'm just shooting in the dark on how this was all supposed to work, as it doesn't appear to be documented (that I can find).

If anyone wants to help me learn how this should all work, by either documenting the existing process, or by creating a new pipeline, I'm all for it.

[vanderpol]

@DeafBDoor if can point me to what you would recommend for filtering/cleaning the artifacts I'll gladly incorporate it into our processes. I'd like to do a posthumous rebuild of 5.11.2 using the initial commit to the OVAL-Community repository, partly to test the release process, and partly to allow for an easy way for anyone to compare 5.11.2 with 5.12.

And @solind I think we will still want to move some of the OVAL Board directories to a different repository, as there's no reason to have them on the developmental branches for merges later on. The 'guidelines' directory likely needs to remain, as I think that's generated (somehow) on a per revision basis, but we'll want to filter it out, minus the HTML end product from the release.

[A-Biggs]

I've used Github actions that are triggered when creating a release and attaching release artifacts to the release before, so I could possibly help a big here if needed.

[vanderpol]

@A-Biggs that would be great if you could lend a hand here. Maybe you could create a pre-release of what we have now just to work out the kinks? I'm still waiting for CIS to provide the keys to the castle for ReadTheDocs, so for now we can't change anything structurally, or it will all break.

Ideally I'd think our 'release' would contain

1. oval-schema
2. oval-specifications (although these are dated and rarely updated, so may not even be part of it)
3. HTML versions of the documentation we generate for readTheDocs. I've figured out how to manually generate the files on my home computer, but am not sure all of the hooks needed to it part of an action.

[DeafBDoor]

Hi, I added a first version of the PR #178 with the code that retrieves the repo, removes the undesirable directories, commits, tags and pushes it under the tag name which was provided by the operator.

[DeafBDoor]

Note that we can further enhance it by doing the pre-release (or even a release) process automatically. I'm still investigating the gh command to provide this and will drop it in future commits or PRs.

[vanderpol]

@DeafBDoor your PR is merged in and available for testing. Not sure what I messed up on my first attempt to merge it. I may have approved it, but not confirmed or something.

[vanderpol]

@DeafBDoor and @A-Biggs, per our meeting yesterday, below is what I have deduced from the existing guidelines directory. I have been able to manual regenerate the .rst files, which appear to be what is used to generate the readthedocs files. There seems to be some residual, unused .md files in github that are red herrings, and that I think we need to delete, but wanted someone else to confirm.

Require Python Modules:

pip install -U sphinx
pip install sphinx-autobuild
pip install sphinx-rtd-theme
pip install recommonmark

Any XSLT tranform tool (I found this one worked, if you have Java)
https://sourceforge.net/projects/saxon/files/Saxon-HE/10/Java/SaxonHE10-8J.zip/download

Genenerate .rst files from .xsd files (this is a powershell example)

$files = Get-ChildItem "C:\Users\Jack\Documents\GitHub\OVAL\oval-schemas" -Filter *.xsd
foreach ($f in $files){
$outfile = $f.Name + ".rst"
java -jar C:\Users\Jack\Downloads\SaxonHE10-8J\saxon-he-10.8.jar -s:$f.FullName -xsl:C:\Users\Jack\Documents\GitHub\OVAL\tools\oval_xsd2rst.xsl -o:C:\Users\Jack\Documents\GitHub\OVAL\guidelines\oval-schema-documentation\$outfile
}

Run Make.bat

make.bat html

Note: In order to make it work correctly, I had to locally edit my OVAL\guidelines\conf.py source_suffix = '.rst', it appears that at some point the documentation switched from a a combination of .md and .rst to just .rst.
I think we can delete all of the OVAL\guidelines\oval-schema-documentation *.md files as they are no longer used, but wanted someone else to do a sanity check on both the config.py and the .md files

[DeafBDoor]

I was able to build in a debian container, installing python and adding the requirements on top of it. It generated the html files based on the readthedocs.io theme.

Now I'm compiling the instructions into a Dockerfile so we can create our own container and put it into a register.

Note there are sphinx containers already available on docker registry, but I couldn't find one with good procedence. Let me know if anyone have info on them.

[vanderpol]

Sounds promising so far, thanks for your help on this. Unfortunately I can't add anything to this regarding info on any sphinx containers...

[DeafBDoor]

Added the Dockerfile which lets you easily build the documentation. It's available on my local copy of the repo: DeafBDoor#4

[DeafBDoor]

Added a MR which integrates the Docker container with the buildRelease workflow. See #190

[vanderpol]

FYI I just received my credentials to ReadTheDocs, and am learning how it works. I no longer think I'll need to move the documentation to a new directory or repository in order to support multiple versions. We should be able to easily have documentation generated online for 5.11.2, 5.12 and 6.0 by just activating different branches of the OVAL repository. So as long as we have branches with valid data for each it should all work pretty easily. I'll start to play around with the develop branch 'as 5.12' and see if I can get it working as expected.

Although it may still make sense going forward for any branch management to move the readthedocs stuff to a separate repository outside of OVAL, we can ponder that more later.

[vanderpol]

With creating the OVAL-Board repository for votes, I moved the OVAL-Board, Area Supervisors, and Email List Archives out of OVAL and into OVAL-Board. Not sure if this impacts the release/build scripts, I assume not, but wanted to give you a heads up.

[DeafBDoor]

The directories you moved are listed in the clean up script, but their removal is done using the rm -rf command, which doesn't fail when the file does not exist. I created a quick PR to remove these dirs anyway: #187

[DeafBDoor]

I'm not being able to trigger a given action on release. Looked for info in the web and the common approach is to use the tag creation as the event which leds to a release instead. Never did this before on github, so some advice (maybe @A-Biggs) would be helpful.

On the other hand, if we decide to use the approach of creating releases based on the tag creation event, we can have a solution which works roughly as following:

1. Maintainer initiates the github action "Clean and tag" to generate a new cleaned version. The action takes a version as an input.
2. Action commits cleaned repository
3. Action creates a tag using the input version which is related to the abovementioned commit
4. Tag creation triggers action "Build release" (BR)
5. BR action generates the documentation
6. BR action creates a new release based on the associated tag
7. BR action compresses and uploads the documentation to the new release

Any inputs on this flow (@vanderpol )? It does what the requirements demands and is also flexible enough to let the mantainers create a tag manually with other resources, which would also generate a new release automatically.

Maybe there is also the possibility of restricting creation of releases based on the branch containing the tag. Would have to research more for that.
Additionally, I'd like to have a fixed format for the version number. E.g. "v5.12". Trigger would be coded to execute only if the version number format is correct.

[DeafBDoor]

So, I went ahead and created a commit to generate the release based on the tag trigger. Unfortunately that didn't work due to a limitation from github: https://github.com/orgs/community/discussions/27028

New idea is to make the first workflow call the second directly instead of through a trigger.

[vanderpol]

@DeafBDoor sorry for the delayed response, I've been out of the office for a couple days. Your steps above seem very reasonable, and if I can enter in v5.12 and have it create what we need, even with 2 processes, it seems OK to me. Pardon my ignorance, but with the issue above, can we document what is needed to complete the process manually, or having two actions?

[DeafBDoor]

My idea is to make the process as easy as possible, so I'll keep investigating possible solutions. And yes, I'll add a small .md file explaining how to use it.

I'm also registering my chain of reasoning here to make my (and maybe others') life easier to maintain the process.

[DeafBDoor]

In the end I decided to simplify and just have a single workflow with two jobs: one for cleanup and one which generates the docs and create a release. Additionally, there is an action which is responsible for handling the container associated with the doc build process.

[vanderpol]

@DeafBDoor just FYI, if by chance while you are working on documentation, if you happen to figure out why this happens it would be a win, but not a show stopper.

#129

The issue I think is related to the fact that the tests in question exist with the same name in both Unix and Windows schema's, and somehow our generation process links the UNIX versions to Windows documentation.

[DeafBDoor]

Sure, now that we have a functional release process, I'll take a look at the documentation problem.

[vanderpol]

@A-Biggs and @DeafBDoor, I apparently am missing some key point to making the build process work after merging everything with master. I did a pre-release build but I'm not seeing the docs etc..

https://github.com/OVAL-Community/OVAL/releases/tag/v5.12

[DeafBDoor]

It triggers only on release and if you edit a release. It does not trigger if you create a pre-release. Feel free to test it on my forked repo. I fetched A-Biggs branch there. Kind regards, Richard Maciel Costa Em ter., 3 de dez. de 2024 às 14:50, Jack Vander Pol < ***@***.***> escreveu:

…

@A-Biggs <https://github.com/A-Biggs> and @DeafBDoor <https://github.com/DeafBDoor>, I apparently am missing some key point to making the build process work after merging everything with master. I did a pre-release build but I'm not seeing the docs etc.. https://github.com/OVAL-Community/OVAL/releases/tag/v5.12 — Reply to this email directly, view it on GitHub <#167 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACDHXJRSELX7F7DRQXODFG32DWZJ3AVCNFSM6AAAAABPFXP3ASVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCNBUHA3TQMY> . You are receiving this because you were mentioned.Message ID: ***@***.***>