Ensuring code quality, testing, and review

[rafmudaf]

I've had a running list of ideas around encouraging and ensuring that new code is tested and reviewed ideally with semi-automated tools as much as possible. I'm putting it here just for future reference and so others can iterate on it.

Expectations from developers, maintainers, and the repo

Project tasks will fund the software development. Project tasks will also fund the code review in a separate line item. Testing (including unit testing) are implicit in the software development process so this does not require a separate line item. Both the developer and code reviewer are ultimately responsible for validation and verification of the code, and ensuring that the pull request is in a mergable state. The testing infrastructure should provide most of the relevant reporting so that the code reviewer can mostly inspect results rather than having to download and run the code.

Automated systems

These do not exist

GitHub Bot should merge dev into a pull request branch when requested and there are no merge conflicts.

• Something like @openfast-bot merge dev

GitHub Bot should update the regression test baselines, commit to r-test, and make a pull request to the corresponding openfast repo when requested.

• Something like @openfast-bot update rtest baselines

Pull request merge checklist

• Theory reviewed by <@reviewer1, @reviewer2, ...>
• Code implementation reviewed by <@reviewer1, @reviewer2, ...>
• Labels added to indicate which portions of the project are impacted
• Proof of correctness
  • New code is covered by a function-level test; or
  • Describe validation so that others can understand and duplicate results. Scripts to generate data are best, screenshots of figures are good.

Guidelines

• Target branch is dev unless there's a really good reason for it to be main
• Add the PR to a project and milestone
• The release notes are automatically generated based on the PR title - Is the title accurate and meaningful to other people?
• If the commit history is meaningful or its a very large set of changes, use "Merge pull request"; otherwise, if the changes can reasonably be contained in a single commit, use "Squash and merge"