Allow unquoted Secret Manager ARNs in config.yaml

[lewismarsden-xft]

Version: 1.8.0

Steps to Reproduce:

• Given the following config.yaml with unquoted Secret Manager ARNs:

inputs:
  - type: cloudwatch-logs
    id: arn:aws:logs:REGION:ACCOUNT:log-group:LOG_GROUP_NAME:*
    outputs:
      - type: elasticsearch
        args:
          api_key: arn:aws:secretsmanager:REGION:ACCOUNT:secret:SECRET_NAME_1
          cloud_id: arn:aws:secretsmanager:REGION:ACCOUNT:secret:SECRET_NAME_2
          es_datastream_name: datastream

• When using a deployed ESF forwarder that uses this config.yaml, the following exception is raised in the lambda logs:

[ERROR] ConfigFileException: Invalid arn format: arn:aws:secretsmanager:REGION:ACCOUNT:secret:SECRET_NAME_1
          cloud_id: arn:aws:secretsmanager:REGION:ACCOUNT:secret:SECRET_NAME_2
          es_datastream_name: datastream

Traceback (most recent call last):
  File "/var/task/main_aws.py", line 17, in handler
    return lambda_handler(lambda_event, lambda_context)
  File "/var/task/handlers/aws/utils.py", line 63, in wrapper
    return func(lambda_event, lambda_context)
  File "/var/task/handlers/aws/utils.py", line 98, in wrapper
    raise e
  File "/var/task/handlers/aws/utils.py", line 82, in wrapper
    return func(lambda_event, lambda_context)
  File "/var/task/handlers/aws/handler.py", line 75, in lambda_handler
    raise ConfigFileException(e)

This happens because the Secrets Manager ARN regex defined at share/secretsmanager.py does not terminate on line endings, only terminating on single- and double-quotes using the following pattern:
re_pattern = r"arn:aws:secretsmanager:(?:[^:]+)?:(?:[^:]+)?:secret:(?:[^\"']+)?"
adding the new line character in the final capturing group should allow unquoted ARNs to parse nicely:
re_pattern = r"arn:aws:secretsmanager:(?:[^:]+)?:(?:[^:]+)?:secret:(?:[^\"'\n]+)?"

We're currently defining our config.yaml programmatically (using the AWS CDK to retrieve ARNs and then build the config file), so while admittedly this isn't much of a problem - we could just update this to wrap the ARNs in quotes - the regex would be useful in our case as Java requires a not-so-clean addition of code such as "\"" + secret.getSecretArn() + "\"" which would be nice to avoid.

Appreciate this is a small problem, and interested to hear any pro-quotation arguments against this suggestion (e.g. protect against special characters) - my understanding is that the YAML is parsed as a string, so this doesn't have an impact on the parser. I'm happy to contribute a PR for the change with a test case if accepted.

[girodav]

Hey @lewismarsden-xft , apologies for the long delay. I added this to our backlog and let you know :).

[keiransteele-phocas]

Will you accept a PR to fix the regex above? I have written a python lambda to update the list of log groups and config with tagged log groups but the pyyaml library doesn't play well with preserving double quotes

[keiransteele-phocas]

I solved the issue in Python using ruamel.yaml and ruamel.yaml.bytes

from ruamel.yaml import YAML, scalarstring

# Initialize ruamel.yaml YAML object
yaml = YAML(typ=['rt', 'bytes'])
yaml.preserve_quotes = True

DQ = scalarstring.DoubleQuotedScalarString
...
'cloud_id': DQ(cloud_id_arn),
...
config_bytes = yaml.dump_to_bytes(config)
bucket.put_object(Key='config.yaml', Body=config_bytes)