[Feature Request]: Integrate PSRule PoC

[eriqua]

Description

Related issues #1186 #1580

Leveraging this issue as a starting point for a PoC to integrate PSRule with the CARML CI environment pipelines.
The goal is to enable and test it for at least 3 modules already converted to the new dependencies approach (see issue #1791)

Workstreams and goals

Pre-flight - Test before resources are deployed from templates.

• CARML PSRule (sub)set:
  • Identify if any baseline rule does not apply to CARML and a customized set is required.
  • Scope: identify if the test target should be repo/module/test specific or a combination of the 3.
• Pester + PSrule:
  - Integrate current static validation with PSRule, identify any overlap and adjust the set of Pester tests/rules accordingly.

The goals of this workstream are:

• Identify the right set of rules to run
• Integrate related jobs in the pipelines
• Run tests and generate output (see ref)
• Act on results accordingly (e.g. update module)

Stretch goals

Cross platform

• Start with GH workflows and extend to ADO pipelines
• Extend to ARM/JSON converted modules

Custom rules

• Inspect the need for custom rules and develop accordingly

In-flight

• Test after resources are deployed to an Azure subscription.
  • Discuss if useful to add and what would be the added value when using together with pre-flight
  • Identify how to integrate in-flight tests with current deployment validation steps, before resources get cleaned up.

References

• Main docs: https://azure.github.io/PSRule.Rules.Azure/about/
• GH repo: https://github.com/Azure/PSRule.Rules.Azure
• List of rules by resource type: https://azure.github.io/PSRule.Rules.Azure/en/rules/resource/
• Generating output: https://microsoft.github.io/PSRule/v2/analysis-output/ and https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

[eriqua]

Day 1

• Create and protect a dev branch hack/topic6, linked to this issue.
• New self-contained dependencies approach overview. Already converted modules (ResourceGroups, KeyVaults, VirtualNetworks) will be the target of the PoC.
• Individual time for each team member to get familiar with PSRule fundamentals.
• Integrate PSRule action
  • Module validation GH workflows
  • Linter workflow triggered on pull requests
• Divide in 2 groups to analyze and solve the following 2 challenges

Challenges

1. Token replacement: needs to run before the PSRule check, otherwise rules checking correct naming will fail. Token <<namePrefix>> is seen as an intended parameter value and normally not compliant with any resource name.
   1. Group 1 implemented and integrated a fix in the ResourceGroup workflow.
   2. PR merged to the dev branch [Hackathon] Token replacement for the resource group module #2068
2. Run subset of rules: how to exclude specific rules from the default baseline?
   1. Group 2 looked into customizing general settings in ps-rule.yaml. Here we can ignore specific files/patterns, add exclusion of specific rules, add suppression of specific rules for specific resources. Tested in the KeyVault workflow.
   2. PR merged to the dev branch [Hackaton] First test for PSRule exclusions on KeyVault #2067

PSRule feedback

Start collecting possible feedback to discuss with the PSRule team:

• Logs show a WARNING: Target object 'modules/Microsoft.Resources/resourceGroups/.test/common/deploy.test.bicep' has not been processed because no matching rules were found. although the file is then checked successfully
• PSRule VSC extension: cannot be used due to tokenized parameters

Next steps

• Refine/clean up token replacement code
• Define the set of rules we want to run
• Analyze PSRule action options.

  Example: Multiple ps-rule.yaml files can be created and placed in specific folders (e.g. root/module/test scope) and specified in a relative path using workflow variables. We should anyway aim for as less configuration files as possible, leveraging tokens or wildcards (see next point)

• Verify if possible to use wildcards/tokens (e.g. in the root ps-rule.yaml setting file) to skip specific rules on specific resources. Ref SuppressionGroups

  Example: Resource names containing min are the ones specifying only the required set of parameters. They will likely fail on rules suggesting the use of tags for all resources.

• Add output (PSRule action option), discuss and select the best format. Test GH job summaries
• Custom rules

[eriqua]

Day 2

• Start analysis of rules per resource type and identify the multiple ways rules can be ignored (Ref How do I ignore a rule? exclusion vs suppression vs suppression groups)
• Test PSRule action on VirtualNetwork module using dedicated ps-rule.yaml file: [Hackathon] PSRule addition for VNet resource  #2070
• Exempt dependencies from PSRule (but the RG) (general setting for all repo): [Hackathon] Added suppression group for dependency resources #2071
• Refine/clean up token replacement code and align the 3 workflows to use matrix (parallel pre-flight checks for each test file): [Hackaton] Align 3 module workflows to use PSRule test matrix #2065
• PSRule output to GH Job Summaries for VirtualNetwork module: [Hackathon] PSRule output in markdown  #2072

Ref Vnet Workflow run

PSRule feedback

• SuppressionGroups do not allow the use of wildcards for rule names

  Example: skip Azure.KeyVault.*

Next steps

• Refine GH Job Summaries output

  Example format markdown output content to outline failed tests, suppressions, passed tests and count for each category, similar to Pester output
  
  Check also this blog page https://techcommunity.microsoft.com/t5/core-infrastructure-and-security/getting-started-with-testing-infrastructure-code/ba-p/1500276

• Reduce log verbosity

  Example: suppress rule warning

• Custom baselines
• Custom rules
• Extend to ADO
• Update modules

[eriqua]

Day 3

• Research & test the several PSRule output options and combinations: detailed vs summary, different output formats --> CSV + Detailed
  • PR [Hackaton] Psrule output on csv format with detailed results #2089 and [Hackaton] csv output on Keyvault module #2091
• Research & test the several logging options to reduce log verbosity
• Parse csv content and provide output in markdown as a job summary.
  • PR [Hackathon] Set PSRule output script #2093

Ref #2094 running on pull request

PSRule feedback

• Provide better markdown output to the PSRule action already. Ref PSRule issue

Next steps

• Integrate csv parser in all workflows
• Create GitHub composite action to reduce code duplication
• Extend to ADO
• Update module tests to comply with failing rules

[eriqua]

Day 4

• PSRuleValidation GitHub composite action [Hackaton] Added PSRule Output to job summary #2110 [Hackaton] Hack/topic6 virtual network PSrule on GitHub Action #2111 [Hackaton] Fix job summary output in GH action #2112
• Integrate job summaries to the 3 workflows [hackaton]: Hack/topic6 fabmas/psrule test updated RG, KV, VNET #2113
• Failing rules analysis. To be fixed vs To be ignored. Adding exeptions for rules to be ignored (min, dep, purge protection) [Hackathon] Suppress rules where not required #2115
• ADO pipeline draft (PSRule extension required on the ADO organization. See feedback below)

Ref Vnet run workflow

PSRule feedback

• Same as here. The PSRule extension prerequisite may be a blocker to adopting the analyzer in ADO pipelines. A workaround to be discussed for CARML is to replace the action with custom script using PSRule module and cmdlets

Next steps

• Extend to ADO
• Uncomment deployment and publishing steps in GH and test the whole workflow
• Add condition to the deployment job: running even if psrule fails (for the time being)
• Update module tests to comply with failing rules
• CSV parser synopsis and optimization
• Open a draft PR to main to get feedback

[eriqua]

Failing rules and target resources before adding suppression groups:

Next steps

• Collected in issue [Feature Request]: PSRule integration - container #2149

[BernieWhite]

Logs show a WARNING: Target object 'modules/Microsoft.Resources/resourceGroups/.test/common/deploy.test.bicep' has not been processed because no matching rules were found. although the file is then checked successfully

Currently this is by design, although we are aware that it is a common point of confusion and may change this behvaiour for PSRule v3. microsoft/PSRule#1179

However you can disable this warning by setting Execution.NotProcessedWarning.

https://microsoft.github.io/PSRule/v2/concepts/PSRule/en-US/about_PSRule_Options/#executionnotprocessedwarning

[BernieWhite]

PSRule VSC extension: cannot be used due to tokenized parameters.

Interested in finding more about this one.

[BernieWhite]

SuppressionGroups do not allow the use of wildcards for rule names
Example: skip Azure.KeyVault.*

Interested in more detail about this or a feature request.

[AlexanderSehr]

Can be closed in favor of #2151 . Next steps such as the ADO extension should be adressed seperatly as we're blocked until that becomes available.