Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

json object not serializable for array of objects datatype #146

Open
sriram9707 opened this issue Dec 18, 2020 · 14 comments
Open

json object not serializable for array of objects datatype #146

sriram9707 opened this issue Dec 18, 2020 · 14 comments
Assignees
Labels

Comments

@sriram9707
Copy link

Hi,
We have a created a custom resource provider type for aws using https://github.com/iann0036/cfn-tf-custom-types .
When trying to launch stack in aws, the datatypes with array of objects throwing an error as json not serializable.

Can some one help me out resolving the issue

@ammokhov
Copy link
Contributor

@sriram9707 could you please provide a bit more details:

  • resource schema
  • sample template
  • log/stack event

@sriram9707
Copy link
Author

@ammokhov i use the following versions.

cloudformation-cli==0.1.14
cloudformation-cli-python-plugin==2.1.2
python 3.8
awscli v2

@sriram9707
Copy link
Author

@ammokhov is there any update on this please ?

@johnttompkins
Copy link
Contributor

Could you provide a stack arn

@ammokhov sure. Thanks for looking in to this

schema.json:-
{
"tfcfnid": "c3d2fabe-5654-4ad0-a71b-22238579c3ed",
"Ami": "ami-0e472933a1395e172",
"Arn": "arn:aws:ec2:us-west-2::instance/i-00db2841dc2dd36b3",
"AssociatePublicIpAddress": true,
"AvailabilityZone": null,
"CpuCoreCount": null,
"CpuThreadsPerCore": null,
"DisableApiTermination": null,
"EbsOptimized": null,
"GetPasswordData": null,
"Hibernation": null,
"HostId": null,
"IamInstanceProfile": null,
"Id": "i-00db2841dc2dd36b3",
"InstanceInitiatedShutdownBehavior": null,
"InstanceState": "running",
"InstanceType": "t2.micro",
"Ipv6AddressCount": null,
"Ipv6Addresses": null,
"KeyName": null,
"Monitoring": null,
"OutpostArn": "",
"PasswordData": "",
"PlacementGroup": null,
"PrimaryNetworkInterfaceId": "eni-0e6b7cc9ae266781c",
"PrivateDns": ".us-west-2.compute.internal",
"PrivateIp": null,
"PublicDns": "ec2-.us-west-2.compute.amazonaws.com",
"PublicIp": "",
"SecondaryPrivateIps": null,
"SecurityGroups": null,
"SourceDestCheck": null,
"SubnetId": "subnet-0e43651c44f1",
"Tags": null,
"Tenancy": null,
"UserData": null,
"UserDataBase64": null,
"VolumeTags": null,
"VpcSecurityGroupIds": [
"sg-04e828fc47b66c01f"
],
"CreditSpecification": null,
"EbsBlockDevice": null,
"EphemeralBlockDevice": null,
"MetadataOptions": null,
"NetworkInterface": null,
"RootBlockDevice": null,
"Timeouts": null
}

sample template:-

AWSTemplateFormatVersion: '2010-09-09'
Resources:
myInstance:
Type: Terraform::AWS::Instance
Properties:
Ami: ami-0e472933a1395e172
InstanceType: t2.micro
AssociatePublicIpAddress: true
SubnetId:
Tags:

  • MapKey: test
    MapValue: one

stackevent:-
UPDATE_FAILED Object of type Tags is not JSON serializable

When i try to update the resource with Tags, i get the following error. The Tags is of array of objects datatype. I even tried with similar properties which has array of objects data type and got the same error as json object not serializable

This doesn't look like a schema file. The schema should conform to the resource provider schema

If you could provide that schema here or provide a stack arn to [email protected] we can take a look at that.

@sriram9707
Copy link
Author

sriram9707 commented Jan 5, 2021

@jotompki Please find the resource schema here.

{
    "typeName": "Terraform::AWS::Instance",
    "description": "Provides an EC2 instance resource. This allows instances to be created, updated,\nand deleted. Instances also support [provisioning](/docs/provisioners/index.html).",
    "sourceUrl": "https://github.com/iann0036/cfn-tf-custom-types.git",
    "documentationUrl": "https://github.com/iann0036/cfn-tf-custom-types/blob/docs/resources/aws/Terraform-AWS-Instance/docs/README.md",
    "definitions": {
        "CreditSpecification": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "CpuCredits": {
                    "type": "string"
                }
            },
            "required": []
        },
        "EbsBlockDevice": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "DeleteOnTermination": {
                    "type": "boolean"
                },
                "DeviceName": {
                    "type": "string"
                },
                "Encrypted": {
                    "type": "boolean"
                },
                "Iops": {
                    "type": "number"
                },
                "KmsKeyId": {
                    "type": "string"
                },
                "SnapshotId": {
                    "type": "string"
                },
                "Throughput": {
                    "type": "number"
                },
                "VolumeSize": {
                    "type": "number"
                },
                "VolumeType": {
                    "type": "string"
                }
            },
            "required": [
                "DeviceName"
            ]
        },
        "EnclaveOptions": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "Enabled": {
                    "type": "boolean"
                }
            },
            "required": []
        },
        "EphemeralBlockDevice": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "DeviceName": {
                    "type": "string"
                },
                "NoDevice": {
                    "type": "boolean"
                },
                "VirtualName": {
                    "type": "string"
                }
            },
            "required": [
                "DeviceName"
            ]
        },
        "MetadataOptions": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "HttpEndpoint": {
                    "type": "string"
                },
                "HttpPutResponseHopLimit": {
                    "type": "number"
                },
                "HttpTokens": {
                    "type": "string"
                }
            },
            "required": []
        },
        "NetworkInterface": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "DeleteOnTermination": {
                    "type": "boolean"
                },
                "DeviceIndex": {
                    "type": "number"
                },
                "NetworkInterfaceId": {
                    "type": "string"
                }
            },
            "required": [
                "DeviceIndex",
                "NetworkInterfaceId"
            ]
        },
        "RootBlockDevice": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "DeleteOnTermination": {
                    "type": "boolean"
                },
                "Encrypted": {
                    "type": "boolean"
                },
                "Iops": {
                    "type": "number"
                },
                "KmsKeyId": {
                    "type": "string"
                },
                "Throughput": {
                    "type": "number"
                },
                "VolumeSize": {
                    "type": "number"
                },
                "VolumeType": {
                    "type": "string"
                }
            },
            "required": []
        },
        "Timeouts": {
            "type": "object",
            "additionalProperties": false,
            "properties": {
                "Create": {
                    "type": "string"
                },
                "Delete": {
                    "type": "string"
                },
                "Update": {
                    "type": "string"
                }
            },
            "required": []
        }
    },
    "properties": {
        "tfcfnid": {
            "description": "Internal identifier for tracking resource changes. Do not use.",
            "type": "string"
        },
        "Ami": {
            "type": "string",
            "description": "The AMI to use for the instance."
        },
        "Arn": {
            "type": "string"
        },
        "AssociatePublicIpAddress": {
            "type": "boolean",
            "description": "Associate a public ip address with an instance in a VPC.  Boolean value."
        },
        "AvailabilityZone": {
            "type": "string",
            "description": "The AZ to start the instance in."
        },
        "CpuCoreCount": {
            "type": "number",
            "description": "Sets the number of CPU cores for an instance. This option is\nonly supported on creation of instance type that support CPU Options\n[CPU Cores and Threads Per CPU Core Per Instance Type](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html#cpu-options-supported-instances-values) - specifying this option for unsupported instance types will return an error from the EC2 API."
        },
        "CpuThreadsPerCore": {
            "type": "number",
            "description": "If set to to 1, hyperthreading is disabled on the launched instance. Defaults to 2 if not set. See [Optimizing CPU Options](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instance-optimize-cpu.html) for more information."
        },
        "DisableApiTermination": {
            "type": "boolean",
            "description": "If true, enables [EC2 Instance\nTermination Protection](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingDisableAPITermination)."
        },
        "EbsOptimized": {
            "type": "boolean",
            "description": "If true, the launched EC2 instance will be EBS-optimized.\nNote that if this is not set on an instance type that is optimized by default then\nthis will show as disabled but if the instance type is optimized by default then\nthere is no need to set this and there is no effect to disabling it.\nSee the [EBS Optimized section](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSOptimized.html) of the AWS User Guide for more information."
        },
        "GetPasswordData": {
            "type": "boolean",
            "description": "If true, wait for password data to become available and retrieve it. Useful for getting the administrator password for instances running Microsoft Windows. The password data is exported to the `password_data` attribute. See [GetPasswordData](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_GetPasswordData.html) for more information."
        },
        "Hibernation": {
            "type": "boolean",
            "description": "If true, the launched EC2 instance will support hibernation."
        },
        "HostId": {
            "type": "string",
            "description": "The Id of a dedicated host that the instance will be assigned to. Use when an instance is to be launched on a specific dedicated host."
        },
        "IamInstanceProfile": {
            "type": "string",
            "description": "The IAM Instance Profile to\nlaunch the instance with. Specified as the name of the Instance Profile. Ensure your credentials have the correct permission to assign the instance profile according to the [EC2 documentation](http://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2.html#roles-usingrole-ec2instance-permissions), notably `iam:PassRole`."
        },
        "Id": {
            "type": "string"
        },
        "InstanceInitiatedShutdownBehavior": {
            "type": "string",
            "description": "Shutdown behavior for the\ninstance. Amazon defaults this to `stop` for EBS-backed instances and\n`terminate` for instance-store instances. Cannot be set on instance-store\ninstances. See [Shutdown Behavior](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/terminating-instances.html#Using_ChangingInstanceInitiatedShutdownBehavior) for more information."
        },
        "InstanceState": {
            "type": "string"
        },
        "InstanceType": {
            "type": "string",
            "description": "The type of instance to start. Updates to this field will trigger a stop/start of the EC2 instance."
        },
        "Ipv6AddressCount": {
            "type": "number",
            "description": "A number of IPv6 addresses to associate with the primary network interface. Amazon EC2 chooses the IPv6 addresses from the range of your subnet."
        },
        "Ipv6Addresses": {
            "type": "array",
            "insertionOrder": false,
            "items": {
                "type": "string"
            },
            "description": "Specify one or more IPv6 addresses from the range of the subnet to associate with the primary network interface."
        },
        "KeyName": {
            "type": "string",
            "description": "The key name of the Key Pair to use for the instance; which can be managed using [the `aws_key_pair` resource](key_pair.html)."
        },
        "Monitoring": {
            "type": "boolean",
            "description": "If true, the launched EC2 instance will have detailed monitoring enabled. (Available since v0.6.0)."
        },
        "OutpostArn": {
            "type": "string"
        },
        "PasswordData": {
            "type": "string"
        },
        "PlacementGroup": {
            "type": "string",
            "description": "The Placement Group to start the instance in."
        },
        "PrimaryNetworkInterfaceId": {
            "type": "string"
        },
        "PrivateDns": {
            "type": "string"
        },
        "PrivateIp": {
            "type": "string",
            "description": "Private IP address to associate with the\ninstance in a VPC."
        },
        "PublicDns": {
            "type": "string"
        },
        "PublicIp": {
            "type": "string"
        },
        "SecondaryPrivateIps": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "type": "string"
            },
            "description": "A list of secondary private IPv4 addresses to assign to the instance's primary network interface (eth0) in a VPC. Can only be assigned to the primary network interface (eth0) attached at instance creation, not a pre-existing network interface i.e. referenced in a `network_interface` block. Refer to the [Elastic network interfaces documentation](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-eni.html#AvailableIpPerENI) to see the maximum number of private IP addresses allowed per instance type."
        },
        "SecurityGroups": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "type": "string"
            },
            "description": "A list of security group names (EC2-Classic) or IDs (default VPC) to associate with."
        },
        "SourceDestCheck": {
            "type": "boolean",
            "description": "Controls if traffic is routed to the instance when\nthe destination address does not match the instance. Used for NAT or VPNs. Defaults true."
        },
        "SubnetId": {
            "type": "string",
            "description": "The VPC Subnet ID to launch in."
        },
        "Tags": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "type": "object",
                "additionalProperties": false,
                "properties": {
                    "MapKey": {
                        "type": "string"
                    },
                    "MapValue": {
                        "type": "string"
                    }
                },
                "required": [
                    "MapKey",
                    "MapValue"
                ]
            },
            "description": "A map of tags to assign to the resource."
        },
        "Tenancy": {
            "type": "string",
            "description": "The tenancy of the instance (if the instance is running in a VPC). An instance with a tenancy of dedicated runs on single-tenant hardware. The host tenancy is not supported for the import-instance command."
        },
        "UserData": {
            "type": "string",
            "description": "The user data to provide when launching the instance. Do not pass gzip-compressed data via this argument; see `user_data_base64` instead."
        },
        "UserDataBase64": {
            "type": "string",
            "description": "Can be used instead of `user_data` to pass base64-encoded binary data directly. Use this instead of `user_data` whenever the value is not a valid UTF-8 string. For example, gzip-encoded user data must be base64-encoded and passed via this argument to avoid corruption."
        },
        "VolumeTags": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "type": "object",
                "additionalProperties": false,
                "properties": {
                    "MapKey": {
                        "type": "string"
                    },
                    "MapValue": {
                        "type": "string"
                    }
                },
                "required": [
                    "MapKey",
                    "MapValue"
                ]
            },
            "description": "A map of tags to assign to the devices created by the instance at launch time."
        },
        "VpcSecurityGroupIds": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "type": "string"
            },
            "description": "A list of security group IDs to associate with."
        },
        "CreditSpecification": {
            "type": "array",
            "insertionOrder": false,
            "items": {
                "$ref": "#/definitions/CreditSpecification"
            },
            "maxItems": 1
        },
        "EbsBlockDevice": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "$ref": "#/definitions/EbsBlockDevice"
            }
        },
        "EnclaveOptions": {
            "type": "array",
            "insertionOrder": false,
            "items": {
                "$ref": "#/definitions/EnclaveOptions"
            },
            "maxItems": 1
        },
        "EphemeralBlockDevice": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "$ref": "#/definitions/EphemeralBlockDevice"
            }
        },
        "MetadataOptions": {
            "type": "array",
            "insertionOrder": false,
            "items": {
                "$ref": "#/definitions/MetadataOptions"
            },
            "maxItems": 1
        },
        "NetworkInterface": {
            "type": "array",
            "insertionOrder": true,
            "items": {
                "$ref": "#/definitions/NetworkInterface"
            }
        },
        "RootBlockDevice": {
            "type": "array",
            "insertionOrder": false,
            "items": {
                "$ref": "#/definitions/RootBlockDevice"
            },
            "maxItems": 1
        },
        "Timeouts": {
            "$ref": "#/definitions/Timeouts"
        }
    },
    "additionalProperties": false,
    "required": [
        "Ami",
        "InstanceType"
    ],
    "readOnlyProperties": [
        "/properties/tfcfnid",
        "/properties/Arn",
        "/properties/Id",
        "/properties/InstanceState",
        "/properties/OutpostArn",
        "/properties/PasswordData",
        "/properties/PrimaryNetworkInterfaceId",
        "/properties/PrivateDns",
        "/properties/PublicDns",
        "/properties/PublicIp"
    ],
    "primaryIdentifier": [
        "/properties/tfcfnid"
    ],
    "handlers": {
        "create": {
            "permissions": [
                "*"
            ]
        },
        "read": {
            "permissions": [
                "*"
            ]
        },
        "update": {
            "permissions": [
                "*"
            ]
        },
        "delete": {
            "permissions": [
                "*"
            ]
        },
        "list": {
            "permissions": [
                "*"
            ]
        }
    }
}

@johnttompkins
Copy link
Contributor

Unable to reproduce with a minimal stack with only tags and a primary identifier:

Schema:

{
    "typeName": "My::Tags::Resource",
    "description": "An example resource schema demonstrating some basic constructs and validation rules.",
    "sourceUrl": "https://github.com/aws-cloudformation/aws-cloudformation-rpdk.git",
    "properties": {
        "Title": {
            "description": "The title of the TPS report is a mandatory element.",
            "type": "string",
            "minLength": 20,
            "maxLength": 250
        },
        "Tags": {
            "description": "An array of key-value pairs to apply to this resource.",
            "type": "array",
            "insertionOrder": true,
            "items": {
                "description": "A key-value pair to associate with a resource.",
                "type": "object",
                "properties": {
                    "MapKey": {
                        "type": "string",
                        "description": "The key name of the tag. You can specify a value that is 1 to 128 Unicode characters in length and cannot be prefixed with aws:. You can use any of the following characters: the set of Unicode letters, digits, whitespace, _, ., /, =, +, and -.",
                        "minLength": 1,
                        "maxLength": 128
                    },
                    "MapValue": {
                        "type": "string",
                        "description": "The value for the tag. You can specify a value that is 0 to 256 Unicode characters in length and cannot be prefixed with aws:. You can use any of the following characters: the set of Unicode letters, digits, whitespace, _, ., /, =, +, and -.",
                        "minLength": 0,
                        "maxLength": 256
                    }
                },
                "required": [
                    "MapKey",
                    "MapValue"
                ],
                "additionalProperties": false
            }
        }
    },
    "additionalProperties": false,
    "required": [
        "Title"
    ],
    "primaryIdentifier": [
        "/properties/Title"
    ],
    "handlers": {
        "create": {
            "permissions": [
            ]
        },
        "read": {
            "permissions": [
            ]
        },
        "update": {
            "permissions": [
            ]
        },
        "delete": {
            "permissions": [
            ]
        },
        "list": {
            "permissions": [
            ]
        }
    }
}

Template for create:

AWSTemplateFormatVersion: "2010-09-09"
Description: AWS SAM template for the My::Tags::Resource resource type

Resources:
  TypeFunction:
    Type: My::Tags::Resource
    Properties:
      Title: This is the Primary Identifier
      Tags:
        - MapKey: TagKey
          MapValue: TagValue

template for update:

AWSTemplateFormatVersion: "2010-09-09"
Description: AWS SAM template for the My::Tags::Resource resource type

Resources:
  TypeFunction:
    Type: My::Tags::Resource
    Properties:
      Title: This is the Primary Identifier
      Tags:
        - MapKey: TagKeys
          MapValue: TagValue

cloudformation-cli-python-lib-2.1.5
cloudformation-cli-python-plgin-2.1.2
cloudformation-cli 0.2.1

Handlers simply returned model input into the template back to cloudformation. it appears to be something with the handler code being submitted misinterpreting the tags on return. could you share the handler code for create/update?

@sriram9707
Copy link
Author

sriram9707 commented Jan 15, 2021

Please find the handlers.py and models.py @jotompki

Handlers.py:-

`import logging
import json
import os
from uuid import uuid4
from typing import Any, MutableMapping, Optional

from cloudformation_cli_python_lib import (
Action,
HandlerErrorCode,
OperationStatus,
ProgressEvent,
Resource,
SessionProxy,
exceptions,
)

from .models import ResourceHandlerRequest, ResourceModel

LOG = logging.getLogger(name)
TYPE_NAME = "Terraform::AWS::Instance"

resource = Resource(TYPE_NAME, ResourceModel)
test_entrypoint = resource.test_entrypoint

def check_progress(operationid, trackingid, progress, session):
LOG.warn("Retrieving existing operation status ({})".format(operationid))

s3client = session.client('s3')
stsclient = session.client('sts')

callerid = stsclient.get_caller_identity()
statebucketname = "cfntf-{}-{}".format(os.environ['AWS_REGION'], callerid.get('Account'))

try:
    result = json.loads(s3client.get_object(Bucket=statebucketname, Key="status/{}.json".format(operationid))['Body'].read())
    s3client.delete_object(Bucket=statebucketname, Key="status/{}.json".format(operationid))

    if result['status'] == 'completed':
        progress.status = OperationStatus.SUCCESS

        # retrieve model
        try:
            model_state = json.loads(s3client.get_object(Bucket=statebucketname, Key="state/{}.model.json".format(trackingid))['Body'].read())
            
            for k,v in model_state.items():
                setattr(progress.resourceModel, k, v)
        except Exception as e:
            LOG.warn(str(e))

        LOG.warn("Action complete")
    else:
        progress.status = OperationStatus.FAILED
        if 'error' in result:
            progress.message = result['error']
except:
    progress.callbackDelaySeconds = 20
    progress.callbackContext = {
        'trackingid': trackingid,
        'operationid': operationid,
    }

return progress

@resource.handler(Action.CREATE)
def create_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)

if callback_context.get('operationid'):
    return check_progress(callback_context.get('operationid'), callback_context.get('trackingid'), progress, session)

LOG.warn("Starting create action")

try:
    lambdaclient = session.client("lambda")

    trackingid = str(uuid4())
    operationid = str(uuid4())

    resolved_model = None
    if model: # potentially no properties set
        resolved_model = {}
        for prop, value in vars(model).items():
            resolved_model[prop] = value
    
    lambdaclient.invoke(
        FunctionName="cfntf-executor",
        InvocationType="Event",
        Payload=json.dumps({
            'action': 'CREATE',
            'trackingId': trackingid,
            'operationId': operationid,
            'model': resolved_model,
            'logicalId': request.logicalResourceIdentifier,
            'providerTypeName': 'aws',
            'terraformTypeName': 'aws_instance',
            'returnValues': ["Arn", "Id", "InstanceState", "OutpostArn", "PasswordData", "PrimaryNetworkInterfaceId", "PrivateDns", "PublicDns", "PublicIp"],
        }).encode(),
    )

    progress.resourceModel.tfcfnid = trackingid
    progress.callbackDelaySeconds = 20
    progress.callbackContext = {
        'trackingid': trackingid,
        'operationid': operationid,
    }
except Exception as e:
    progress.message = str(e)
    progress.status = OperationStatus.FAILED
return progress

@resource.handler(Action.UPDATE)
def update_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)

if callback_context.get('operationid'):
    return check_progress(callback_context.get('operationid'), callback_context.get('trackingid'), progress, session)

LOG.warn("Starting update action")

try:
    lambdaclient = session.client("lambda")

    trackingid = model.tfcfnid
    operationid = str(uuid4())
    
    resolved_model = None
    if model: # potentially no properties set
        resolved_model = {}
        for prop, value in vars(model).items():
            resolved_model[prop] = value
    
    lambdaclient.invoke(
        FunctionName="cfntf-executor",
        InvocationType="Event",
        Payload=json.dumps({
            'action': 'UPDATE',
            'trackingId': trackingid,
            'operationId': operationid,
            'model': resolved_model,
            'logicalId': request.logicalResourceIdentifier,
            'providerTypeName': 'aws',
            'terraformTypeName': 'aws_instance',
            'returnValues': ["Arn", "Id", "InstanceState", "OutpostArn", "PasswordData", "PrimaryNetworkInterfaceId", "PrivateDns", "PublicDns", "PublicIp"],
        }).encode(),
    )

    progress.resourceModel.tfcfnid = trackingid
    progress.callbackDelaySeconds = 20
    progress.callbackContext = {
        'trackingid': trackingid,
        'operationid': operationid,
    }
except Exception as e:
    progress.message = str(e)
    progress.status = OperationStatus.FAILED
return progress

@resource.handler(Action.DELETE)
def delete_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState
progress: ProgressEvent = ProgressEvent(
status=OperationStatus.IN_PROGRESS,
resourceModel=model,
)

if callback_context.get('operationid'):
    return check_progress(callback_context.get('operationid'), callback_context.get('trackingid'), progress, session)

LOG.warn("Starting delete action")

try:
    lambdaclient = session.client("lambda")

    trackingid = model.tfcfnid
    operationid = str(uuid4())
    
    resolved_model = None
    if model: # potentially no properties set
        resolved_model = {}
        for prop, value in vars(model).items():
            resolved_model[prop] = value
    
    lambdaclient.invoke(
        FunctionName="cfntf-executor",
        InvocationType="Event",
        Payload=json.dumps({
            'action': 'DELETE',
            'trackingId': trackingid,
            'operationId': operationid,
            'model': resolved_model,
            'logicalId': request.logicalResourceIdentifier,
            'providerTypeName': 'aws',
            'terraformTypeName': 'aws_instance',
            'returnValues': ["Arn", "Id", "InstanceState", "OutpostArn", "PasswordData", "PrimaryNetworkInterfaceId", "PrivateDns", "PublicDns", "PublicIp"],
        }).encode(),
    )

    progress.resourceModel.tfcfnid = trackingid
    progress.callbackDelaySeconds = 20
    progress.callbackContext = {
        'trackingid': trackingid,
        'operationid': operationid,
    }
except Exception as e:
    progress.message = str(e)
    progress.status = OperationStatus.FAILED
return progress

@resource.handler(Action.READ)
def read_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
model = request.desiredResourceState

s3client = session.client('s3')
stsclient = session.client('sts')

callerid = stsclient.get_caller_identity()
statebucketname = "cfntf-{}-{}".format(os.environ['AWS_REGION'], callerid.get('Account'))

# retrieve model
try:
    model_state = json.loads(s3client.get_object(Bucket=statebucketname, Key="state/{}.model.json".format(model.tfcfnid))['Body'].read())
    
    for k,v in model_state.items():
        setattr(model, k, v)
    
    return ProgressEvent(
        status=OperationStatus.SUCCESS,
        resourceModel=model,
    )
except Exception as e:
    LOG.warn(str(e))

return ProgressEvent(
    status=OperationStatus.FAILED,
    resourceModel=model,
)

@resource.handler(Action.LIST)
def list_handler(
session: Optional[SessionProxy],
request: ResourceHandlerRequest,
callback_context: MutableMapping[str, Any],
) -> ProgressEvent:
# TODO: put code here
return ProgressEvent(
status=OperationStatus.SUCCESS,
resourceModels=[],
)`

@sriram9707
Copy link
Author

sriram9707 commented Jan 15, 2021

Models.py:-

`
import sys
from dataclasses import dataclass
from inspect import getmembers, isclass
from typing import (
AbstractSet,
Any,
Generic,
Mapping,
MutableMapping,
Optional,
Sequence,
Type,
TypeVar,
)

from cloudformation_cli_python_lib.interface import (
BaseModel,
BaseResourceHandlerRequest,
)
from cloudformation_cli_python_lib.recast import recast_object
from cloudformation_cli_python_lib.utils import deserialize_list

T = TypeVar("T")

def set_or_none(value: Optional[Sequence[T]]) -> Optional[AbstractSet[T]]:
if value:
return set(value)
return None

@DataClass
class ResourceHandlerRequest(BaseResourceHandlerRequest):
# pylint: disable=invalid-name
desiredResourceState: Optional["ResourceModel"]
previousResourceState: Optional["ResourceModel"]

@DataClass
class ResourceModel(BaseModel):
tfcfnid: Optional[str]
Ami: Optional[str]
Arn: Optional[str]
AssociatePublicIpAddress: Optional[bool]
AvailabilityZone: Optional[str]
CpuCoreCount: Optional[float]
CpuThreadsPerCore: Optional[float]
DisableApiTermination: Optional[bool]
EbsOptimized: Optional[bool]
GetPasswordData: Optional[bool]
Hibernation: Optional[bool]
HostId: Optional[str]
IamInstanceProfile: Optional[str]
Id: Optional[str]
InstanceInitiatedShutdownBehavior: Optional[str]
InstanceState: Optional[str]
InstanceType: Optional[str]
Ipv6AddressCount: Optional[float]
Ipv6Addresses: Optional[Sequence[str]]
KeyName: Optional[str]
Monitoring: Optional[bool]
OutpostArn: Optional[str]
PasswordData: Optional[str]
PlacementGroup: Optional[str]
PrimaryNetworkInterfaceId: Optional[str]
PrivateDns: Optional[str]
PrivateIp: Optional[str]
PublicDns: Optional[str]
PublicIp: Optional[str]
SecondaryPrivateIps: Optional[Sequence[str]]
SecurityGroups: Optional[Sequence[str]]
SourceDestCheck: Optional[bool]
SubnetId: Optional[str]
Tags: Optional[Sequence["_Tags"]]
Tenancy: Optional[str]
UserData: Optional[str]
UserDataBase64: Optional[str]
VolumeTags: Optional[Sequence["_VolumeTags"]]
VpcSecurityGroupIds: Optional[Sequence[str]]
CreditSpecification: Optional[Sequence["_CreditSpecification"]]
EbsBlockDevice: Optional[Sequence["_EbsBlockDevice"]]
EnclaveOptions: Optional[Sequence["_EnclaveOptions"]]
EphemeralBlockDevice: Optional[Sequence["_EphemeralBlockDevice"]]
MetadataOptions: Optional[Sequence["_MetadataOptions"]]
NetworkInterface: Optional[Sequence["_NetworkInterface"]]
RootBlockDevice: Optional[Sequence["_RootBlockDevice"]]
Timeouts: Optional["_Timeouts"]

@classmethod
def _deserialize(
    cls: Type["_ResourceModel"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_ResourceModel"]:
    if not json_data:
        return None
    dataclasses = {n: o for n, o in getmembers(sys.modules[__name__]) if isclass(o)}
    recast_object(cls, json_data, dataclasses)
    return cls(
        tfcfnid=json_data.get("tfcfnid"),
        Ami=json_data.get("Ami"),
        Arn=json_data.get("Arn"),
        AssociatePublicIpAddress=json_data.get("AssociatePublicIpAddress"),
        AvailabilityZone=json_data.get("AvailabilityZone"),
        CpuCoreCount=json_data.get("CpuCoreCount"),
        CpuThreadsPerCore=json_data.get("CpuThreadsPerCore"),
        DisableApiTermination=json_data.get("DisableApiTermination"),
        EbsOptimized=json_data.get("EbsOptimized"),
        GetPasswordData=json_data.get("GetPasswordData"),
        Hibernation=json_data.get("Hibernation"),
        HostId=json_data.get("HostId"),
        IamInstanceProfile=json_data.get("IamInstanceProfile"),
        Id=json_data.get("Id"),
        InstanceInitiatedShutdownBehavior=json_data.get("InstanceInitiatedShutdownBehavior"),
        InstanceState=json_data.get("InstanceState"),
        InstanceType=json_data.get("InstanceType"),
        Ipv6AddressCount=json_data.get("Ipv6AddressCount"),
        Ipv6Addresses=json_data.get("Ipv6Addresses"),
        KeyName=json_data.get("KeyName"),
        Monitoring=json_data.get("Monitoring"),
        OutpostArn=json_data.get("OutpostArn"),
        PasswordData=json_data.get("PasswordData"),
        PlacementGroup=json_data.get("PlacementGroup"),
        PrimaryNetworkInterfaceId=json_data.get("PrimaryNetworkInterfaceId"),
        PrivateDns=json_data.get("PrivateDns"),
        PrivateIp=json_data.get("PrivateIp"),
        PublicDns=json_data.get("PublicDns"),
        PublicIp=json_data.get("PublicIp"),
        SecondaryPrivateIps=json_data.get("SecondaryPrivateIps"),
        SecurityGroups=json_data.get("SecurityGroups"),
        SourceDestCheck=json_data.get("SourceDestCheck"),
        SubnetId=json_data.get("SubnetId"),
        Tags=deserialize_list(json_data.get("Tags"), Tags),
        Tenancy=json_data.get("Tenancy"),
        UserData=json_data.get("UserData"),
        UserDataBase64=json_data.get("UserDataBase64"),
        VolumeTags=deserialize_list(json_data.get("VolumeTags"), VolumeTags),
        VpcSecurityGroupIds=json_data.get("VpcSecurityGroupIds"),
        CreditSpecification=deserialize_list(json_data.get("CreditSpecification"), CreditSpecification),
        EbsBlockDevice=deserialize_list(json_data.get("EbsBlockDevice"), EbsBlockDevice),
        EnclaveOptions=deserialize_list(json_data.get("EnclaveOptions"), EnclaveOptions),
        EphemeralBlockDevice=deserialize_list(json_data.get("EphemeralBlockDevice"), EphemeralBlockDevice),
        MetadataOptions=deserialize_list(json_data.get("MetadataOptions"), MetadataOptions),
        NetworkInterface=deserialize_list(json_data.get("NetworkInterface"), NetworkInterface),
        RootBlockDevice=deserialize_list(json_data.get("RootBlockDevice"), RootBlockDevice),
        Timeouts=Timeouts._deserialize(json_data.get("Timeouts")),
    )

_ResourceModel = ResourceModel

@DataClass
class Tags(BaseModel):
MapKey: Optional[str]
MapValue: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_Tags"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_Tags"]:
    if not json_data:
        return None
    return cls(
        MapKey=json_data.get("MapKey"),
        MapValue=json_data.get("MapValue"),
    )

_Tags = Tags

@DataClass
class VolumeTags(BaseModel):
MapKey: Optional[str]
MapValue: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_VolumeTags"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_VolumeTags"]:
    if not json_data:
        return None
    return cls(
        MapKey=json_data.get("MapKey"),
        MapValue=json_data.get("MapValue"),
    )

_VolumeTags = VolumeTags

@DataClass
class CreditSpecification(BaseModel):
CpuCredits: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_CreditSpecification"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_CreditSpecification"]:
    if not json_data:
        return None
    return cls(
        CpuCredits=json_data.get("CpuCredits"),
    )

_CreditSpecification = CreditSpecification

@DataClass
class EbsBlockDevice(BaseModel):
DeleteOnTermination: Optional[bool]
DeviceName: Optional[str]
Encrypted: Optional[bool]
Iops: Optional[float]
KmsKeyId: Optional[str]
SnapshotId: Optional[str]
Throughput: Optional[float]
VolumeSize: Optional[float]
VolumeType: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_EbsBlockDevice"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_EbsBlockDevice"]:
    if not json_data:
        return None
    return cls(
        DeleteOnTermination=json_data.get("DeleteOnTermination"),
        DeviceName=json_data.get("DeviceName"),
        Encrypted=json_data.get("Encrypted"),
        Iops=json_data.get("Iops"),
        KmsKeyId=json_data.get("KmsKeyId"),
        SnapshotId=json_data.get("SnapshotId"),
        Throughput=json_data.get("Throughput"),
        VolumeSize=json_data.get("VolumeSize"),
        VolumeType=json_data.get("VolumeType"),
    )

_EbsBlockDevice = EbsBlockDevice

@DataClass
class EnclaveOptions(BaseModel):
Enabled: Optional[bool]

@classmethod
def _deserialize(
    cls: Type["_EnclaveOptions"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_EnclaveOptions"]:
    if not json_data:
        return None
    return cls(
        Enabled=json_data.get("Enabled"),
    )

_EnclaveOptions = EnclaveOptions

@DataClass
class EphemeralBlockDevice(BaseModel):
DeviceName: Optional[str]
NoDevice: Optional[bool]
VirtualName: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_EphemeralBlockDevice"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_EphemeralBlockDevice"]:
    if not json_data:
        return None
    return cls(
        DeviceName=json_data.get("DeviceName"),
        NoDevice=json_data.get("NoDevice"),
        VirtualName=json_data.get("VirtualName"),
    )

_EphemeralBlockDevice = EphemeralBlockDevice

@DataClass
class MetadataOptions(BaseModel):
HttpEndpoint: Optional[str]
HttpPutResponseHopLimit: Optional[float]
HttpTokens: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_MetadataOptions"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_MetadataOptions"]:
    if not json_data:
        return None
    return cls(
        HttpEndpoint=json_data.get("HttpEndpoint"),
        HttpPutResponseHopLimit=json_data.get("HttpPutResponseHopLimit"),
        HttpTokens=json_data.get("HttpTokens"),
    )

_MetadataOptions = MetadataOptions

@DataClass
class NetworkInterface(BaseModel):
DeleteOnTermination: Optional[bool]
DeviceIndex: Optional[float]
NetworkInterfaceId: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_NetworkInterface"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_NetworkInterface"]:
    if not json_data:
        return None
    return cls(
        DeleteOnTermination=json_data.get("DeleteOnTermination"),
        DeviceIndex=json_data.get("DeviceIndex"),
        NetworkInterfaceId=json_data.get("NetworkInterfaceId"),
    )

_NetworkInterface = NetworkInterface

@DataClass
class RootBlockDevice(BaseModel):
DeleteOnTermination: Optional[bool]
Encrypted: Optional[bool]
Iops: Optional[float]
KmsKeyId: Optional[str]
Throughput: Optional[float]
VolumeSize: Optional[float]
VolumeType: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_RootBlockDevice"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_RootBlockDevice"]:
    if not json_data:
        return None
    return cls(
        DeleteOnTermination=json_data.get("DeleteOnTermination"),
        Encrypted=json_data.get("Encrypted"),
        Iops=json_data.get("Iops"),
        KmsKeyId=json_data.get("KmsKeyId"),
        Throughput=json_data.get("Throughput"),
        VolumeSize=json_data.get("VolumeSize"),
        VolumeType=json_data.get("VolumeType"),
    )

_RootBlockDevice = RootBlockDevice

@DataClass
class Timeouts(BaseModel):
Create: Optional[str]
Delete: Optional[str]
Update: Optional[str]

@classmethod
def _deserialize(
    cls: Type["_Timeouts"],
    json_data: Optional[Mapping[str, Any]],
) -> Optional["_Timeouts"]:
    if not json_data:
        return None
    return cls(
        Create=json_data.get("Create"),
        Delete=json_data.get("Delete"),
        Update=json_data.get("Update"),
    )

_Timeouts = Timeouts

`

@sriram9707
Copy link
Author

@jotompki is there any update please ?

@johnttompkins
Copy link
Contributor

@jotompki is there any update please ?

Could you point to a github repo or something a bit more formatted? It is hard to tell how all of these are interacting in an issue comment.

@sriram9707
Copy link
Author

@jotompki sure. please find the github repo https://github.com/iann0036/cfn-tf-custom-types.git

@sriram9707
Copy link
Author

@jotompki is there any update on this pls ?

@sriram9707
Copy link
Author

any update on this please ? @jotompki @ammokhov @wbingli @stilvoid

@JohnPreston
Copy link
Contributor

JohnPreston commented Feb 10, 2022

I am having similar issues with an array of simple objects
I have it down to an IndexError. However, all SAM local tests and aws cloudformation test-type work just fine.
It only is when using "the real CFN" that it fails with the exact same input (i.e, took what's in inputs/input_1_create.json and set that as Properties in the template to try it with).

note: the IndexError is caught by Exception in resources.py.Resource.__call__
I had to modify that file to have the ProgressEvent return the message

@ericzbeard ericzbeard self-assigned this Oct 13, 2022
@ericzbeard ericzbeard added the p2 Low priority label Oct 13, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

5 participants