change_resource_record_sets() DELETE action fails with error

[tdaquino]

Describe the bug

Trying to delete a DNS entry with change_resource_record_sets() fails with error "'NoneType' object has no attribute 'split'."

Expected Behavior

I expected the DNS entry to be deleted without issue.

Current Behavior

The request fails with an error that doesn't provide much insight as to where the problem lies.

This is the full stack trace:

[ERROR] AttributeError: 'NoneType' object has no attribute 'split' Traceback (most recent call last): File "/var/task/lambda_function.py", line 86, in lambda_handler response = action(resource, command, region, deployment_name, user_id, detail, log) File "/var/task/lambda_function.py", line 54, in action call_function = functions[command]() File "/var/task/domains.py", line 332, in delete delete_domain_entry_response = self.delete_domain_entry() File "/var/task/domains.py", line 289, in delete_domain_entry delete_validate_cert_response = self.delete_validate_cert() File "/var/task/domains.py", line 183, in delete_validate_cert self.aws_route53_client.change_resource_record_sets( File "/var/runtime/botocore/client.py", line 530, in _api_call return self._make_api_call(operation_name, kwargs) File "/var/runtime/botocore/client.py", line 919, in _make_api_call request_dict = self._convert_to_request_dict( File "/var/runtime/botocore/client.py", line 987, in _convert_to_request_dict api_params = self._emit_api_params( File "/var/runtime/botocore/client.py", line 1026, in _emit_api_params self.meta.events.emit( File "/var/runtime/botocore/hooks.py", line 412, in emit return self._emitter.emit(aliased_event_name, **kwargs) File "/var/runtime/botocore/hooks.py", line 256, in emit return self._emit(event_name, kwargs) File "/var/runtime/botocore/hooks.py", line 239, in _emit response = handler(**kwargs) File "/var/runtime/botocore/handlers.py", line 650, in fix_route53_ids params[name] = orig_value.split('/')[-1]

This is the debug log:

[DEBUG] 2023-09-27T21:43:53.361Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f601fc108b0> 2023-09-27 21:43:53,361 botocore.hooks [DEBUG] Event choose-service-name: calling handler <function handle_service_name_alias at 0x7f601fc108b0> [DEBUG] 2023-09-27T21:43:53.364Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/service-2.json 2023-09-27 21:43:53,364 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/service-2.json [DEBUG] 2023-09-27T21:43:53.521Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/endpoint-rule-set-1.json.gz 2023-09-27 21:43:53,521 botocore.loaders [DEBUG] Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/endpoint-rule-set-1.json.gz [DEBUG] 2023-09-27T21:43:53.524Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Event creating-client-class.route-53: calling handler <function add_generate_presigned_url at 0x7f601fce9a60> 2023-09-27 21:43:53,524 botocore.hooks [DEBUG] Event creating-client-class.route-53: calling handler <function add_generate_presigned_url at 0x7f601fce9a60> [DEBUG] 2023-09-27T21:43:53.561Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Using partition endpoint for route53, us-west-2: aws-global 2023-09-27 21:43:53,561 botocore.regions [DEBUG] Using partition endpoint for route53, us-west-2: aws-global [DEBUG] 2023-09-27T21:43:53.563Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Setting route53 timeout as (60, 60) 2023-09-27 21:43:53,563 botocore.endpoint [DEBUG] Setting route53 timeout as (60, 60) [DEBUG] 2023-09-27T21:43:53.565Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Registering retry handlers for service: route53 2023-09-27 21:43:53,565 botocore.client [DEBUG] Registering retry handlers for service: route53 [DEBUG] 2023-09-27T21:43:53.622Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Calling endpoint provider with parameters: {'Region': 'us-west-2', 'UseDualStack': False, 'UseFIPS': False} 2023-09-27 21:43:53,622 botocore.regions [DEBUG] Calling endpoint provider with parameters: {'Region': 'us-west-2', 'UseDualStack': False, 'UseFIPS': False} [DEBUG] 2023-09-27T21:43:53.622Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Endpoint provider result: https://route53.amazonaws.com 2023-09-27 21:43:53,622 botocore.regions [DEBUG] Endpoint provider result: https://route53.amazonaws.com [DEBUG] 2023-09-27T21:43:53.622Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Selecting from endpoint provider's list of auth schemes: ""sigv4"". User selected auth scheme is: ""None"" 2023-09-27 21:43:53,622 botocore.regions [DEBUG] Selecting from endpoint provider's list of auth schemes: ""sigv4"". User selected auth scheme is: ""None"" [DEBUG] 2023-09-27T21:43:53.622Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Selected auth type ""v4"" as ""v4"" with signing context params: {'region': 'us-east-1', 'signing_name': 'route53'} 2023-09-27 21:43:53,622 botocore.regions [DEBUG] Selected auth type ""v4"" as ""v4"" with signing context params: {'region': 'us-east-1', 'signing_name': 'route53'} [DEBUG] 2023-09-27T21:43:53.622Z f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Event before-parameter-build.route-53.ChangeResourceRecordSets: calling handler <function fix_route53_ids at 0x7f601fc2eca0> 2023-09-27 21:43:53,622 botocore.hooks [DEBUG] Event before-parameter-build.route-53.ChangeResourceRecordSets: calling handler <function fix_route53_ids at 0x7f601fc2eca0> [ERROR] AttributeError: 'NoneType' object has no attribute 'split' Traceback (most recent call last):   File ""/var/task/lambda_function.py"", line 86, in lambda_handler     response = action(resource, command, region, deployment_name, user_id, detail, log)   File ""/var/task/lambda_function.py"", line 54, in action     call_function = functions[command]()   File ""/var/task/domains.py"", line 332, in delete     delete_domain_entry_response = self.delete_domain_entry()   File ""/var/task/domains.py"", line 289, in delete_domain_entry     delete_validate_cert_response = self.delete_validate_cert()   File ""/var/task/domains.py"", line 183, in delete_validate_cert     self.aws_route53_client.change_resource_record_sets(   File ""/var/runtime/botocore/client.py"", line 530, in _api_call     return self._make_api_call(operation_name, kwargs)   File ""/var/runtime/botocore/client.py"", line 919, in _make_api_call     request_dict = self._convert_to_request_dict(   File ""/var/runtime/botocore/client.py"", line 987, in _convert_to_request_dict     api_params = self._emit_api_params(   File ""/var/runtime/botocore/client.py"", line 1026, in _emit_api_params     self.meta.events.emit(   File ""/var/runtime/botocore/hooks.py"", line 412, in emit     return self._emitter.emit(aliased_event_name, **kwargs)   File ""/var/runtime/botocore/hooks.py"", line 256, in emit     return self._emit(event_name, kwargs)   File ""/var/runtime/botocore/hooks.py"", line 239, in _emit     response = handler(**kwargs)   File ""/var/runtime/botocore/handlers.py"", line 650, in fix_route53_ids     params[name] = orig_value.split('/')[-1]" END RequestId: f9eb3403-0ccb-4eb9-a052-f5ed19977d53 REPORT RequestId: f9eb3403-0ccb-4eb9-a052-f5ed19977d53 Duration: 2287.30 ms Billed Duration: 2288 ms Memory Size: 128 MB Max Memory Used: 74 MB Init Duration: 315.65 ms

Reproduction Steps

This is the relevant code:

`class Domain:

def __init__(self):
    self.hosted_zone = None
    self.validation_record = None
    self.__aws_route53_client = None

@property
def aws_route53_client(self):
    if self.__aws_route53_client is None:
        self.__aws_route53_client = boto3.client('route53')
    return self.__aws_route53_client

def create_validate_cert(self):
    name = self.validation_record[0]
    value = self.validation_record[1]
    try:
        self.aws_route53_client.change_resource_record_sets(
            HostedZoneId = self.hosted_zone,
            ChangeBatch = {
                'Changes': [
                    {
                        'Action': 'UPSERT',
                        'ResourceRecordSet':{
                            'Name': name,
                            'Type': 'CNAME',
                            'TTL': 300,
                            'ResourceRecords': [
                                {
                                    'Value': value
                                }
                            ]
                        }
                    }
                ]
            }
        )
    except botocore.exceptions.ClientError as error:
        return error
    except botocore.exceptions.ParamValidationError as error:
        return error
    return 'cert_validation_record_created'

def delete_validate_cert(self):
    name = self.validation_record[0]
    value = self.validation_record[1]
    try:
        self.aws_route53_client.change_resource_record_sets(
            HostedZoneId = self.hosted_zone,
            ChangeBatch = {
                'Changes': [
                    {
                        'Action': 'DELETE',
                        'ResourceRecordSet': {
                            'Name': name,
                            'Type': 'CNAME',
                            'TTL': 300,
                            'ResourceRecords': [
                                {
                                    'Value': value
                                }
                            ]
                        }
                    }
                ]
            }
        )
    except botocore.exceptions.ClientError as error:
        return error
    except botocore.exceptions.ParamValidationError as error:
        return error
    return 'cert_validation_record_deleted'`

This is along side of a bunch of other code that sets self.hosted_zone = to the hosted zone ID where the CNAME exists and sets self.validation_record = to a list that contains [<record_name>,<record_value>] of the CNAME.

Possible Solution

No response

Additional Information/Context

No response

SDK version used

Whatever version is embedded in the Lambda Python 3.8 runtime. I also tried the 3.10 runtime and got the same result.

Environment details (OS name and version, etc.)

Lambda - Python 3.8 runtime on x86_64 architecture

[tim-finnigan]

Hi @tdaquino thanks for reaching out. Here are the default versions provided in Lambda runtimes. Based on that you are likely using boto3-1.26.90 / botocore-1.29.90. (Those versions are behind the latest versions of Boto3/Botocore, and for Lamdba issues we often refer customers to this article to fix common errors caused by using older versions. Although I don't think that would address the issue you're reporting here.)

I'm not sure that this is directly an issue with the change_resource_record_sets command or the corresponding API.

I think we need a little more information to help isolate the problem. Here is the fix_route_ids method in Botocore where the split() is failing. We'd need to determine why we're getting a None value for one of ResourceId, DelegationSetId, and ChangeId.

[tdaquino]

Hi @tim-finnigan, thanks for your reply and for picking up this issue.

The fix_route53_ids method isn't my code. It's part of botocore in the handlers.py file. Here's a link to the method:

botocore/botocore/handlers.py

Line 629 in fcad884

def fix_route53_ids(params, model, **kwargs):

The stacktrace shows that the method with the issue is being called by the botocore client, which is being called by the change_resource_record_sets method, but it's not clear to me what is being passed to fix_route53_ids.

The debug output shows botocore is loading these JSON files so maybe fix_route53_ids is supposed format something from them:

Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/service-2.json

Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/service-2.json

Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/endpoint-rule-set-1.json.gz

Loading JSON file: /var/runtime/botocore/data/route53/2013-04-01/endpoint-rule-set-1.json.gz

I'll dig into the botocore code tomorrow when I have more time and maybe I'll be able to get a better idea of what that method is supposed to be fixing.

BYW, thanks for the tip on handling outdated boto3 libs.

[tim-finnigan]

Hi @tdaquino thanks for following up — I had updated my comment after a colleague pointed out the same thing regarding fix_route53_ids but it looks like you were replying to the original comment. Just to highlight what was said earlier - we need to determine why one of ResourceId, DelegationSetId, or ChangeId is a None value and causing that method to fail. Please let us know if you have any updates on your end as far as which value might be causing this.

[tdaquino]

@tim-finnigan, I'm not calling the fix_route53_ids method from my code directly (it's not a boto3 documented method so, how would I know how to call it in the first place?) so there's no way for me to know why those values are empty and the debug output isn't any help either. Not sure what more you would like for me to provide but if you can be specific, I'm happy to gather more details for you.

I'm calling the change_resource_record_sets method with all of the required parameters populated per the boto3 documentation. If you see something wrong with how I'm calling the change_resource_record_sets method (my code is included in the original report), I'm happy to make suggested changes but there isn't anything in the stack trace that would indicate that is the case. If you'd like to review my code in it's entirety, you can find it here: https://github.com/havocsh/havoc/blob/main/havoc_control_api/manage/domains.py

This is either a bug in the botocore code or a bug in the docs (in the sense that there are requirements not documented). Either way, please put the bug tag back on this issue so that it gets treated accordingly. And if you're not familiar with how botocore's handlers are called, I would suggest pulling in someone who is familiar. The botocore code is a nightmare to try to follow and I won't be of any help in that regard.

Thanks,
Tom

[tim-finnigan]

Hi @tdauino thanks for following up again and apologies for the confusion. The fix_route53_ids function you linked isn't publicly documented for use but is an internal handler that serves the following purpose:

Check for and split apart Route53 resource IDs, setting
only the last piece. This allows the output of one operation
(e.g. 'foo/1234') to be used as input in another
operation (e.g. it expects just '1234').

I can add the bug label back and we can try to narrow down the source of the issue. In your delete_validate_cert function, can you validate the parameters that get passed to change_resource_record_sets()?

[tdaquino]

@tim-finnigan, I think I've figured out what's happening here. When my manage domains operation creates a certificate validation resource record, it stores the associated DNS name and value in a DynamoDB table as a string set. When a request is made to delete the domain, it pulls the certificate validation DNS details from the DynamoDB table. Since the string set data type doesn't preserve the original order (my mistake for using a string set instead of a list), the DNS name and value parameters are coming back in a different order than how I originally inserted them. Thus when the change_resource_record_sets method is called, the name and value parameters are transposed so the method can't find record set that it is being asked to delete.

And therein lies the bug in botocore. I should be getting some sort of "resource record not found" error returned to the invalid request but instead, it returns the very unhelpful 'NoneType' object has no attribute 'split' error. This seems like a logic error to me. The fix_route53_ids method is being called prior to verifying the resource record's existence.

I can resolve the issue on my end so feel free to close this out but I would still suggest flagging this as an error handling bug that needs resolution.

Thanks,
Tom

[tim-finnigan]

Thanks @tdaquino for sharing your findings! I'll close this out as resolved but I created a new issue for the confusing error: #3033. If others encounter the error hopefully that will help provide some guidance while we consider possible ways to improve the error messaging.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.