catch common resource schema issues in cfn validate

[PatMyron]

continuing #663, #668
fixes #395, #459, #414

---

how to run new validations on all existing resource provider schemas

---

• incorrect min/max constraints (other incorrect type-specific keywords in catch common resource schema issues in cfn validate #729)
• inconsistent property casing

curl -s -N --compressed https://d1uauaxba7bl26.cloudfront.net/latest/gzip/CloudFormationResourceSpecification.json | grep -E '"[a-g][i-z]'
        "error": {
        "code": {

• hardcoded aws partition patterns (arn:aws: should instead be something like arn:aws[-a-z]*:)

CloudformationSchemas $ grep 'pattern.*arn:aws:' *
aws-appflow-flow.json:      "pattern" : "arn:aws:appflow:.*:[0-9]+:.*",
aws-appflow-flow.json:      "pattern" : "arn:aws:kms:.*:[0-9]+:.*",
aws-globalaccelerator-endpointgroup.json:      "pattern" : "^arn:aws:(\\w+)::(\\d{12}):(accelerator)/([0-9a-f-]+)/listener/([0-9a-f-]+)$"
aws-ivs-channel.json:      "pattern" : "^arn:aws:ivs:[a-z0-9-]+:[0-9]+:channel/[a-zA-Z0-9-]+$",
aws-ivs-playbackkeypair.json:      "pattern" : "^arn:aws:ivs:[a-z0-9-]+:[0-9]+:playback-key/[a-zA-Z0-9-]+$",
aws-ivs-streamkey.json:      "pattern" : "^arn:aws:ivs:[a-z0-9-]+:[0-9]+:stream-key/[a-zA-Z0-9-]+$",
aws-ivs-streamkey.json:      "pattern" : "^arn:aws:ivs:[a-z0-9-]+:[0-9]+:channel/[a-zA-Z0-9-]+$"
aws-mwaa-environment.json:      "pattern" : "^(((arn:aws(-[a-z]+)?:kms:[a-z]{2}-[a-z]+-\\d:\\d+:)?key\\/)?[a-f0-9]{8}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{4}-[a-f0-9]{12}|(arn:aws:kms:[a-z]{2}-[a-z]+-\\d:\\d+:)?alias/.+)$"
aws-opsworkscm-server.json:      "pattern" : "arn:aws:iam::[0-9]{12}:role/.*",
aws-opsworkscm-server.json:      "pattern" : "arn:aws:iam::[0-9]{12}:instance-profile/.*",
aws-sso-assignment.json:      "pattern" : "arn:aws:sso:::instance/(sso)?ins-[a-zA-Z0-9-.]{16}",
aws-sso-assignment.json:      "pattern" : "arn:aws:sso:::permissionSet/(sso)?ins-[a-zA-Z0-9-.]{16}/ps-[a-zA-Z0-9-./]{16}",
aws-sso-instanceaccesscontrolattributeconfiguration.json:      "pattern" : "arn:aws:sso:::instance/(sso)?ins-[a-zA-Z0-9-.]{16}",
aws-sso-permissionset.json:      "pattern" : "arn:aws:sso:::permissionSet/(sso)?ins-[a-zA-Z0-9-.]{16}/ps-[a-zA-Z0-9-./]{16}",
aws-sso-permissionset.json:      "pattern" : "arn:aws:sso:::instance/(sso)?ins-[a-zA-Z0-9-.]{16}",

• invalid regular expression patterns
• non-ASCII resource schema characters
• hardcoding constantly evolving enums like AMIs, instance types, lambda runtimes, partitions, regions, availability zones, etc. that get outdated quickly

[kddejong]

I like this type of testing. Get rid of some of the java patterns would be helpful. My question is the python re the one we want to standardize on? For instance ^[\d\w-_.+]*$ is technically valid but in python it needs to be ^[\d\w\-_.+]*$.

[PatMyron]

Ideally, resource schema patterns should be valid and equivalent in both Python and Java (and more)

JSON schema itself recommends sticking to a minimal subset of regular expression syntax. Let's encourage the same

Just emitting warnings for patterns not valid in Python seems like a good start in that direction:

  23 Could not validate regular expression: ^([\p{L}\p{Z}\p{N}_.:/=+\-@]*)$
   6 Could not validate regular expression: ^[a-zA-Z0-9_\-\+\./\(\)\$\p{Zs}]+$
   4 Could not validate regular expression: ^[a-zA-Z0-9-]+{1,255}$
   3 Could not validate regular expression: ^([\p{L}\p{Z}\p{N}_.:=+\/\-@%]*)$
   3 Could not validate regular expression: [\u0020-\uD7FF\uE000-\uFFFD\uD800\uDC00-\uDBFF\uDFFF  ]*
   2 Could not validate regular expression: ^[\/]+([^~]*(~[01])*)*{1,512}$
   2 Could not validate regular expression: ^((?!aws:)[\p{L}\p{Z}\p{N}_.:/=+\-@]*)$
   2 Could not validate regular expression: [\p{L}\p{M}\p{Z}\p{S}\p{N}\p{P}]*
   1 Could not validate regular expression: ^[a-zA-Z0-9_\-\+\./\(\)\p{Zs}]*$
   1 Could not validate regular expression: ^[a-zA-Z0-9\s-_()\[\]]+$
   1 Could not validate regular expression: ^[a-zA-Z-0-9-:\/]*{1,1000}$
   1 Could not validate regular expression: ^[a-zA-Z-0-9-:\/.]*{1,1000}$
   1 Could not validate regular expression: ^[\p{L}\p{Z}\p{N}_.:\/=+\-@%]*$
   1 Could not validate regular expression: ^[\p{L}\p{Z}\p{N}_.:/=+\-@]*$|resource-groups:Name
   1 Could not validate regular expression: ^[A-Za-z0-9._\-:\/#\+]+{1,255}$
   1 Could not validate regular expression: ^(?!aws:.*)([\p{L}\p{Z}\p{N}_.:=+\/\-@%]*)$
   1 Could not validate regular expression: ^(?!aws:)[\p{L}\p{Z}\p{N}_.:\/=+\-@%]*$
   1 Could not validate regular expression: [^_\p{Z}][\p{L}\p{M}\p{S}\p{N}\p{P}][^_\p{Z}]+
   1 Could not validate regular expression: [\p{L}\p{Z}\p{N}_.:\/=+\-@]+
   1 Could not validate regular expression: [\p{L}\p{Z}\p{N}_.:\/=+\-@\[\]\{\}\$\\"]*

[kddejong]

JSON schema itself recommends sticking to a minimal subset of regular expression syntax. Let's encourage the same

Agreed completely here.

[PatMyron]

Golang has some interesting regex limitations even within that minimal subset:
hashicorp/terraform-provider-awscc#88
golang/go#7252

[johnttompkins]

I would advise trying to use the schema flattener to check any nested object's properties. The flattener will place all nested schemas in the schema_map for use. iterating over these as well as the top level definitions and properties should allow you to definitively check all objects

[omkhegde]

LGTM, I suggest adding a test case with a large schema file with a lot of nested properties.

[PatMyron]

LGTM, I suggest adding a test case with a large schema file with a lot of nested properties.

ran cfn validate on all the existing resource schemas, which should have quite a few nested properties