Function to find all Resources of a specific type (to use with Fn::ForEach)

[cortexcompiler]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request

An intrinsic function that can find all resources of a specific type. A bonus would be an ability to filter them based upon a Logical Id pattern or even other properties.

Tell us about the problem you are trying to solve. What are you trying to do, and why is it hard?

Example use case:
I would like to add an AWS::Logs::SubscriptionFilter for every AWS::Logs::LogGroup defined in my template. Every property is the same for the SubscriptionFilter except for the Logical Id, and the LogGroupName. This seems like a great use case for Fn::ForEach, if I could just get the list of Logical Ids for the LogGroups. One other note: the LogGroups may have some variability so I cannot drive their creation using a list or map (and so do not have that to drive the creation of the SubscriptionFilter resources). I have been unable to find a satisfactory solution in CloudFormation examples or documentation.

Without this I end up having to maintain and copy/paste a lot of boilerplate configuration and run the risk of missing a SubscriptionFilter for a LogGroup.

Are you currently working around this issue?

I am currently copying and pasting the same AWS::Logs::SubscriptionFilter block and then changing the resource id and LogGroupName.

What is the expected behavior with this new feature

Rather than having to do something like this (simplified example)

AWSTemplateFormatVersion: '2010-09-09'
Transform:
- AWS::LanguageExtensions
- AWS::Serverless-2016-10-31
Description: Example to demonstrate need

Globals:
  Function:
    Runtime: nodejs16.x

Resources:
  FirstFunction:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      MemorySize: 128
      CodeUri: .
      Handler: index.handler

  FirstFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${FirstFunction}"

  FirstFunctionLogSubscriptionFilter: 
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:common-log-ingestion
      FilterPattern: ''
      LogGroupName: !Ref FirstFunctionLogGroup

  SecondFunction:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - x86_64
      MemorySize: 256
      CodeUri: .
      Handler: index.handler

  SecondFunctionLogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub "/aws/lambda/${SecondFunction}"

  SecondFunctionLogSubscriptionFilter: 
    Type: AWS::Logs::SubscriptionFilter
    Properties:
      DestinationArn: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:common-log-ingestion
      FilterPattern: ''
      LogGroupName: !Ref SecondFunctionLogGroup

I am envisioning something like this (in this case I am driving both the LogGroup and the SubscriptionFilter off of the Function resources):

AWSTemplateFormatVersion: '2010-09-09'
Transform:
- AWS::LanguageExtensions
- AWS::Serverless-2016-10-31
Description: Example to demonstrate need

Globals:
  Function:
    Runtime: nodejs16.x

Resources:
  FirstFunction:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - arm64
      MemorySize: 128
      CodeUri: .
      Handler: index.handler

  SecondFunction:
    Type: AWS::Serverless::Function
    Properties:
      Architectures:
        - x86_64
      MemorySize: 256
      CodeUri: .
      Handler: index.handler

  # Iterate over each AWS::Serverless::Function (bonus: Logical Id matching pattern "^.*Function")
  # and for each Function create a LogGroup and SubscriptionFilter for that group
  Fn::ForEach::FunctionLogGroups:
    - FunctionLogicalId
    - Fn::FindResources: 
        Type: AWS::Serverless::Function
        Pattern: ^.*Function
    - "${FunctionLogicalId}LogGroup": 
        Type: AWS::Logs::LogGroup
        Properties:
          LogGroupName: !Sub "/aws/lambda/${FunctionLogicalId}"

      "${FunctionLogicalId}SubscriptionFilter": 
        Type: AWS::Logs::SubscriptionFilter
        Properties:
          DestinationArn: !Sub arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:common-log-ingestion
          FilterPattern: ''
          LogGroupName: !Ref ${FunctionLogicalId}LogGroup

Additional context

This could save time and prevent issues caused by copy/paste errors and omissions.

Attachments

[rhbecker]

I like this idea, but until something like that is supported, you could potentially create log and subscription resources from the same Collection parameter, in a single Fn::ForEach loop.

EDIT: Maybe you've said that approach wouldn't work for your use case? If so, would you mind elaborating on why, maybe with an example? I'm wondering whether it would work if you included some conditions in your loop, to cover variations.

[benbridts]

I think either support for yaml anchors and tags, or an expansion of the SAM globals section (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification-template-anatomy-globals.html) would be a cleaner solution for this

[cortexcompiler]

I like this idea, but until something like that is supported, you could potentially create log and subscription resources from the same Collection parameter, in a single Fn::ForEach loop.

EDIT: Maybe you've said that approach wouldn't work for your use case? If so, would you mind elaborating on why, maybe with an example? I'm wondering whether it would work if you included some conditions in your loop, to cover variations.

I will be giving this a try for my use case to see if it is good enough. My concern about using a collection was that I might miss the name of a Lambda resource or perhaps typo the name (the latter would probably fail fast). I lose the resource "type" connection and the ability to automatically include any new Lambda resources that are added without needing to copy and past the names.

[cortexcompiler]

I think either support for yaml anchors and tags, or an expansion of the SAM globals section (https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/sam-specification-template-anatomy-globals.html) would be a cleaner solution for this

I don't think globals would quite cover the use case, YAML anchors are an interesting idea, though I would like the ability to override certain properties (like names of things), and overriding the whole property might not be what is needed.

[rhbecker]

My concern about using a collection was that I might miss the name of a Lambda resource or perhaps typo the name (the latter would probably fail fast). I lose the resource "type" connection and the ability to automatically include any new Lambda resources that are added without needing to copy and past the names.

Makes sense - I was basing my suggestion on what you said here ...

I would like to add an AWS::Logs::SubscriptionFilter for every AWS::Logs::LogGroup defined in my template.

But I now see you later clarified ...

I am driving both the LogGroup and the SubscriptionFilter off of the Function resources

Either way, I like the concept of the intrinsic function you propose. Please let us know how your testing goes - if you get something working, I'd love to see an example snippet.