AWS Cloud Service Collector

Plugin to collect AWS Cloud Services

SpaceONE's plugin-aws-cloud-service-inven-collector is a convenient tool to get cloud service data from AWS.

Find us also at Dockerhub

Latest stable version : 1.15.8

Please contact us if you need any further information. ([email protected])

---

Collecting Contents

• Table of Contents
  • API Gateway
    • API (REST API / Websocket)
  • Auto Scaling Group
    • Auto Scaling Group
    • Launch Configuration
    • Launch Template
  • Cloud Front
    • Distribution
  • Cloud Trail
    • Trail
  • Direct Connect
    • Connection
    • Direct Connect Gateway
    • Virtual Private Gateway
    • LAG
  • DocumentDB
    • Cluster
    • Subnet Group
    • Parameter Group
  • DynamoDB
    • Table
  • EBS
    • Volume
    • Snapshot
  • EC2
    • Security Group
    • AMI
  • ECR
    • Repository
  • ECS
    • Cluster
  • EFS
    • Filesystem
  • EIP
    • EIP
  • EKS
    • Cluster
    • Node Group
  • ElastiCache
    • Memcached
    • Redis
  • ELB
    • Load Balancer
    • Target Group
  • IAM
    • Group
    • User
    • Role
    • Policy
    • Identity Provider
    • Access Key
  • Kinesis Datastream
    • Data stream
  • Kinesis Firehose
    • Delivery stream
  • KMS
    • Key
  • Lambda
    • Function
    • Layer
  • MSK
    • Cluster
    • Cluster Configuration
  • RDS
    • Database
    • Instance
    • Snapshot
    • Subnet Group
    • Option Group
  • Redshift
    • Cluster
  • Route53
    • Hosted Zone
  • S3
    • Bucket
  • Secrets Manager
    • Secret
  • SNS
    • Topic
  • SQS
    • Queue
  • VPC
    • VPC
    • Subnet
    • Route Table
    • Internet Gateway
    • Egress only internet Gateway
    • NAT Gateway
    • Peer Connection
    • Network ACL
    • Endpoint
    • Transit Gateway
    • Customer Gateway
    • VPN Connection
    • VPN Gateway
  • Lightsail
    • Instance
    • Disk
    • Snapshot
    • Bucket
    • Static IP
    • Database
    • Container
    • Load Balancer
    • Distribution

---

AWS Service Endpoint (in use)

There is an endpoints used to collect AWS resources information. AWS endpoint is a URL consisting of a region and a service code.

https://[service-code].[region-code].amazonaws.com

We use hundreds of endpoints because we collect information from a lots of regions and services.

Region list

Below is the AWS region information. The regions we collect are not all regions supported by AWS. Exactly, we target the regions results returned by describe_regions() of AWS ec2 client.

No.	Region name	Region Code
1	US East (Ohio)	us-east-2
2	US East (N. Virginia)	us-east-1
3	US West (N. California)	us-west-1
4	US West (Oregon)	us-west-2
5	Asia Pacific (Mumbai)	ap-south-1
6	Asia Pacific (Osaka)	ap-northeast-3
7	Asia Pacific (Seoul)	ap-northeast-2
8	Asia Pacific (Singapore)	ap-southeast-1
9	Asia Pacific (Sydney)	ap-southeast-2
10	Asia Pacific (Tokyo)	ap-northeast-1
11	Canada (Central)	ca-central-1
12	Europe (Frankfurt)	eu-central-1
13	Europe (Ireland)	eu-west-1
14	Europe (London)	eu-west-2
15	Europe (Paris)	eu-west-3
16	Europe (Stockholm)	eu-north-1
17	South America (São Paulo)	sa-east-1

Service list

The following is a list of services being collected and service code information.

No.	Service name	Service Code
1	AWS Certifcate Manager	AWSCertificateManager
2	API Gateway (REST API)	AmazonApiGateway
3	API Gateway V2 (Websocket)	AmazonApiGateway
4	Auto Scaling Group	AmazonEC2
5	CloudFront	AmazonCloudFront
6	CloudTrail	AWSCloudTrail
7	Direct Connect	AWSDirectConnect
8	DocumentDB	AmazonDocDB
9	DynamoDB	AmazonDynamoDB
10	Elastic Block Store (EBS)	AmazonEC2
11	EC2 (SecurityGroup, AMI, EIP)	AmazonEC2
12	Elastic Container Registry (ECR)	AmazonECR
13	Elastic Container Service (ECS)	AmazonECS
14	Elastic File System (EFS)	AmazonEFS
15	Elastic Kubernetes Service (EKS)	AmazonEKS
16	Elasticache	AmazonElastiCache
17	Elastic Load Balancer (ELB)	AWSELB
18	Identity Access Management (IAM)	-
19	Kinesis Data Stream	AmazonKinesis
20	Kinesis Firehose	AmazonKinesisFirehose
21	Key Management System (KMS)	awskms
22	Lambda	AWSLambda
21	Managed Streaming for Apache Kafka (MSK)	AmazonMSK
22	Relational Database Service (RDS)	AmazonRDS
23	Redshift	AmazonRedshift
24	Route53	AmazonRoute53
25	Simple Cloud Storage (S3)	AmazonS3
26	Secrets Manager	AWSSecretsManager
27	Simple Notification Service (SNS)	AmazonSNS
28	Simple Queue Service (SQS)	AWSQueueService
29	Virtual Private Cloud (VPC)	AmazonVPC
30	Lightsail	AmazonLightsail

---

Authentication Overview

Registered service account on SpaceONE must have certain permissions to collect cloud service data Please, set authentication privilege for followings:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Action": [
                "acm:Describe*",
                "acm:List*",
                "apigateway:GET",
                "application-autoscaling:Describe*",
                "autoscaling:Describe*",
                "cloudfront:List*",
                "cloudtrail:Describe*",
                "cloudtrail:Get*",
                "cloudtrail:List*",
                "cloudwatch:Describe*",
                "cloudwatch:Get*",
                "cloudwatch:List*",
                "directconnect:Describe*",
                "dynamodb:Describe*",
                "dynamodb:List*",
                "ec2:Describe*",
                "ecr:Describe*",
                "ecr:List*",
                "ecs:Describe*",
                "ecs:List*",
                "eks:Describe*",
                "eks:List*",
                "elasticache:Describe*",
                "elasticache:List*",
                "elasticfilesystem:Describe*",
                "elasticloadbalancing:Describe*",
                "firehose:Describe*",
                "firehose:List*",
                "health:Describe*",
                "iam:Get*",
                "iam:List*",
                "kafka:Describe*",
                "kafka:List*",
                "kinesis:Describe*",
                "kinesis:List*",
                "kms:Describe*",
                "kms:Get*",
                "kms:List*",
                "lambda:List*",
                "lambda:Get*",
                "rds:Describe*",
                "rds:List*",
                "redshift:Describe*",
                "route53:List*",
                "s3:Get*",
                "s3:List*",
                "secretsmanager:List*",
                "sns:Get*",
                "sns:List*",
                "sqs:Get*",
                "sqs:List*",
                "ssm:Describe*",
                "Lightsail:Get*"
            ],
            "Effect": "Allow",
            "Resource": "*"
        }
    ]
}

---

Options

Cloud Service Type : Specify what to collect

If cloud_service_types is added to the list elements in options, only the specified cloud service type is collected. By default, if cloud_service_types is not specified in options, all services are collected.

The cloud_service_types items that can be specified are as follows.

{
    "cloud_service_types": [
        'IAM',          
        'DynamoDB',     
        'Lambda',       
        'CloudFront',
        'RDS',
        'Route53',
        'S3',
        'AutoScalingGroup',
        'ElastiCache',
        'APIGateway',
        'DirectConnect',
        'EFS',
        'DocumentDB',
        'ECS',
        'Redshift',
        'EKS',
        'SQS',
        'KMS',
        'ECR',
        'CloudTrail',
        'SNS',
        'SecretsManager',
        'ELB',
        'EIP',
        'EBS',
        'VPC',
        'EC2',
        'ACM',
        'KinesisDataStream',
        'KinesisFirehose',
        'MSK',
        'Lightsail'
    ]
}

How to update plugin information using spacectl is as follows. First, create a yaml file to set options.

> cat update_collector.yaml
---
collector_id: collector-xxxxxxx
options:
  cloud_service_types:
    - EC2
    - RDS
    - ELB

Update plugin through spacectl command with the created yaml file.

Service Code Mapper : Convert service code in Cloud Service Type what you want.

If service_code_mappers is added in options, You can replace the service code specified in the cloud service type. The service code set by default can be checked in the Service List item of this document.

The service_code_mappers items that can be specified are as follows.

{
    "service_code_mappers": {
        "AmazonEC2": "Amazon Elastic Computing",
        "AmazonRDS": "Amazon Relation Database",
    }
}

Custom Asset URL : Possible to modify icon path of cloud service

If custom_asset_url is added in options, You can replace the path of the icon each cloud service type instead of default path.

The custom_asset_url items that can be specified are as follows.

{
    "custom_asset_url": "https://CUSTOM_ASSET_URL/..."
}

---

Release note