Default indentifiers

[LadMartin]

My question is about defaulting indentifiers in Resources.

let's say I have an resource s3 now I want to crete a bucket but I do not want to give it an explicit name because I do not care about the name. I would like to have some meta code that will implicitly generate some specificly prefixed name for me.
So then the code could look like this

bkt = s3.Bucket() # no explicit name needed to set
bkt.create()

My motivation is using it for test where user do not care about the names - but since the names will be prefixed then It would be easy to recognize and remove such buckets in teardown.

I did not manage to use events for this purpose - should it be possible using them or is there any other way?

[aaoswal]

Hi @LadMartin,
You can use events in this case and add a function to add a prefix to the bucket name and then generate the bucket when called.

You mentioned that you did not manage to use events for this purpose, could you please share a sample code for troubleshooting purposes?

[LadMartin]

Needs to be added that the client gets the Bucket argument because the bucket is created but for some reason the boto Resource is not properly created.

[aaoswal]

@LadMartin,
I am unable to understand the second code snippet provided:

s3.meta.client.meta.events.register(
    'provide-client-params.s3.*', add_my_bucket)

I am unable to understand the ask here. Is events not working as intended?

I would appreciate it if you could provide context on the current use-case. Also, could you also share the debug logs by adding boto3.set_stream_logger("") to your sample code?

[LadMartin]

[cloudshell-user@ip-10-0-191-59 ~]$ python3
Python 3.7.10 (default, Jun  3 2021, 00:02:01) 
[GCC 7.3.1 20180712 (Red Hat 7.3.1-13)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import boto3
>>> 
>>> s3 = boto3.resource('s3')
>>> boto3.set_stream_logger("")
>>> 
>>> BUCKET='mlad-testing'
>>> 
>>> def add_my_bucket(params, **kwargs):
...     if 'Bucket' not in params:
...         params['Bucket'] = BUCKET
... 
>>> s3.meta.client.meta.events.register('provide-client-params.s3.*', add_my_bucket)
>>> 
>>> bkt = s3.create_bucket()
2022-04-29 12:14:24,333 boto3.resources.action [DEBUG] Calling s3:create_bucket with {}
2022-04-29 12:14:24,333 botocore.hooks [DEBUG] Event provide-client-params.s3.CreateBucket: calling handler <function add_my_bucket at 0x7f943fd5a290>
2022-04-29 12:14:24,334 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function validate_bucket_name at 0x7f94410b4c20>
2022-04-29 12:14:24,334 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_cache of <botocore.utils.S3RegionRedirector object at 0x7f943fd88cd0>>
2022-04-29 12:14:24,334 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <bound method S3ArnParamHandler.handle_arn of <botocore.utils.S3ArnParamHandler object at 0x7f943fd88c90>>
2022-04-29 12:14:24,334 botocore.hooks [DEBUG] Event before-parameter-build.s3.CreateBucket: calling handler <function generate_idempotent_uuid at 0x7f94410b4a70>
2022-04-29 12:14:24,335 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <function add_expect_header at 0x7f94410b4f80>
2022-04-29 12:14:24,335 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <bound method S3RegionRedirector.set_request_url of <botocore.utils.S3RegionRedirector object at 0x7f943fd88cd0>>
2022-04-29 12:14:24,335 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <function add_recursion_detection_header at 0x7f9441130830>
2022-04-29 12:14:24,335 botocore.hooks [DEBUG] Event before-call.s3.CreateBucket: calling handler <function inject_api_version_header_if_needed at 0x7f94410bd320>
2022-04-29 12:14:24,335 botocore.endpoint [DEBUG] Making request for OperationModel(name=CreateBucket) with params: {'url_path': '/mlad-testing', 'query_string': {}, 'method': 'PUT', 'headers': {'User-Agent': 'Boto3/1.21.9 Python/3.7.10 Linux/4.14.268-205.500.amzn2.x86_64 exec-env/CloudShell Botocore/1.23.54 Resource'}, 'body': b'', 'url': 'https://s3.amazonaws.com/mlad-testing', 'context': {'client_region': 'us-east-1', 'client_config': <botocore.config.Config object at 0x7f944050d490>, 'has_streaming_input': False, 'auth_type': None, 'signing': {'bucket': 'mlad-testing'}}}
2022-04-29 12:14:24,335 botocore.hooks [DEBUG] Event request-created.s3.CreateBucket: calling handler <bound method RequestSigner.handler of <botocore.signers.RequestSigner object at 0x7f94407ea510>>
2022-04-29 12:14:24,336 botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <bound method S3EndpointSetter.set_signer of <botocore.utils.S3EndpointSetter object at 0x7f943fd43cd0>>
2022-04-29 12:14:24,336 botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <bound method ClientCreator._default_s3_presign_to_sigv2 of <botocore.client.ClientCreator object at 0x7f944102ad90>>
2022-04-29 12:14:24,336 botocore.hooks [DEBUG] Event choose-signer.s3.CreateBucket: calling handler <function set_operation_specific_signer at 0x7f94410b4950>
2022-04-29 12:14:24,336 botocore.hooks [DEBUG] Event before-sign.s3.CreateBucket: calling handler <bound method S3EndpointSetter.set_endpoint of <botocore.utils.S3EndpointSetter object at 0x7f943fd43cd0>>
2022-04-29 12:14:24,336 botocore.utils [DEBUG] Defaulting to S3 virtual host style addressing with path style addressing fallback.
2022-04-29 12:14:24,336 botocore.utils [DEBUG] Checking for DNS compatible bucket for: https://s3.amazonaws.com/mlad-testing
2022-04-29 12:14:24,336 botocore.utils [DEBUG] URI updated to: https://mlad-testing.s3.amazonaws.com/
2022-04-29 12:14:24,336 botocore.credentials [DEBUG] Credentials need to be refreshed.
2022-04-29 12:14:24,337 botocore.credentials [DEBUG] Credentials need to be refreshed.
2022-04-29 12:14:24,337 botocore.credentials [DEBUG] Credentials need to be refreshed.
2022-04-29 12:14:24,338 urllib3.connectionpool [DEBUG] http://localhost:1338 "GET /latest/meta-data/container/security-credentials HTTP/1.1" 200 1268
2022-04-29 12:14:24,338 botocore.credentials [DEBUG] Retrieved credentials will expire at: 2022-04-29 12:22:43+00:00
2022-04-29 12:14:24,339 botocore.auth [DEBUG] Calculating signature using v4 auth.
2022-04-29 12:14:24,339 botocore.auth [DEBUG] CanonicalRequest:
PUT
/

host:mlad-testing.s3.amazonaws.com
x-amz-content-sha256:...
x-amz-date:...
x-amz-security-token:...

host;x-amz-content-sha256;x-amz-date;x-amz-security-token
...
2022-04-29 12:14:24,339 botocore.auth [DEBUG] StringToSign:
...
20220429T121424Z
20220429/us-east-1/s3/aws4_request
...
2022-04-29 12:14:24,339 botocore.auth [DEBUG] Signature:
...
2022-04-29 12:14:24,339 botocore.hooks [DEBUG] Event request-created.s3.CreateBucket: calling handler <function add_retry_headers at 0x7f94410bd9e0>
2022-04-29 12:14:24,339 botocore.endpoint [DEBUG] Sending http request: <AWSPreparedRequest stream_output=False, method=PUT, url=https://mlad-testing.s3.amazonaws.com/, headers={'User-Agent': b'Boto3/1.21.9 Python/3.7.10 Linux/4.14.268-205.500.amzn2.x86_64 exec-env/CloudShell Botocore/1.23.54 Resource', 'X-Amz-Date': b'20220429T121424Z', 'X-Amz-Security-Token': b'...', 'X-Amz-Content-SHA256': b'...', 'Authorization': b'.../us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date;x-amz-security-token, Signature=...', 'amz-sdk-invocation-id': b'...', 'amz-sdk-request': b'attempt=1', 'Content-Length': '0'}>
2022-04-29 12:14:24,341 botocore.httpsession [DEBUG] Certificate path: /usr/local/lib/python3.7/site-packages/certifi/cacert.pem
2022-04-29 12:14:24,341 urllib3.connectionpool [DEBUG] Starting new HTTPS connection (1): mlad-testing.s3.amazonaws.com:443
2022-04-29 12:14:24,423 urllib3.connectionpool [DEBUG] https://mlad-testing.s3.amazonaws.com:443 "PUT / HTTP/1.1" 200 0
2022-04-29 12:14:24,424 botocore.parsers [DEBUG] Response headers: {'x-amz-id-2': '...', 'x-amz-request-id': 'PR86SD4EFZQ35HNG', 'Date': 'Fri, 29 Apr 2022 12:14:25 GMT', 'Location': '/mlad-testing', 'Server': 'AmazonS3', 'Content-Length': '0'}
2022-04-29 12:14:24,424 botocore.parsers [DEBUG] Response body:
b''
2022-04-29 12:14:24,424 botocore.hooks [DEBUG] Event needs-retry.s3.CreateBucket: calling handler <botocore.retryhandler.RetryHandler object at 0x7f944102ac10>
2022-04-29 12:14:24,425 botocore.retryhandler [DEBUG] No retry needed.
2022-04-29 12:14:24,425 botocore.hooks [DEBUG] Event needs-retry.s3.CreateBucket: calling handler <bound method S3RegionRedirector.redirect_from_error of <botocore.utils.S3RegionRedirector object at 0x7f943fd88cd0>>
2022-04-29 12:14:24,425 boto3.resources.action [DEBUG] Response: {'ResponseMetadata': {'RequestId': 'PR86SD4EFZQ35HNG', 'HostId': '...', 'HTTPStatusCode': 200, 'HTTPHeaders': {'x-amz-id-2': '...', 'x-amz-request-id': 'PR86SD4EFZQ35HNG', 'date': 'Fri, 29 Apr 2022 12:14:25 GMT', 'location': '/mlad-testing', 'server': 'AmazonS3', 'content-length': '0'}, 'RetryAttempts': 0}, 'Location': '/mlad-testing'}
2022-04-29 12:14:24,425 boto3.resources.factory [DEBUG] Loading s3:Bucket
2022-04-29 12:14:24,425 boto3.resources.model [DEBUG] Renaming Bucket attribute name
2022-04-29 12:14:24,426 botocore.hooks [DEBUG] Event creating-resource-class.s3.Bucket: calling handler <function lazy_call.<locals>._handler at 0x7f9441075200>
>>> 
>>> print(bkt)
{}
>>>

[aaoswal]

@LadMartin,
I am unable to understand the second code snippet provided:

s3.meta.client.meta.events.register(
    'provide-client-params.s3.*', add_my_bucket)

I am unable to understand the ask here. Is events not working as intended?

[LadMartin]

@aaoswal,
Please, see the example, it's whole code that I run.
The restult is that
bkt = s3.create_bucket() shall return s3.Bucket and instead it returns {}
why it does that...I do not know. - the only thing I can see is that the event's seem to work but the create_bucket fails to create and instance of s3.Bucket - why it fails to do that I do not know - and that's what I am trying to find out.

[aaoswal]

@LadMartin
You would need to specify the region that you want to create the bucket in, which might have been causing problems in your case.

This worked for me:

BUCKET='mlad-testing'
s3 = boto3.client('s3')

# Access the event system on the S3 client
event_system = s3.meta.events

def add_bucket(params, **kwargs):
    if 'Bucket' not in params:
        params['Bucket'] = BUCKET

event_system.register('provide-client-params.s3.*', add_bucket)
response = s3.create_bucket(CreateBucketConfiguration={
        'LocationConstraint': 'us-west-2',
    },
)
print(response)

[LadMartin]

@aaoswal yes it works with s3 = boto3.client('s3') but it should work as well with s3 = boto3.resource('s3') right?
I can see that i my case the client works as well, but then the s3.Bucket resource is not created.

I am specifically interested in Resource and its create_bucket that returns s3.Bucket

[aaoswal]

@LadMartin Unfortunately, I am unable to reproduce the issue on my end.
After using resource, a valid response is returned with metadata related to the bucket creation each time it is called instead of an empty response.

Are you making any additional customizations or have any proxies enabled?

Reference Code:

BUCKET='mlad-testing'
s3 = boto3.resource('s3')

# Access the event system on the S3 client
# event_system = s3.meta.events

def add_bucket(params, **kwargs):
    if 'Bucket' not in params:
        params['Bucket'] = BUCKET

s3.meta.client.meta.events.register('provide-client-params.s3.*', add_bucket)
response = s3.create_bucket(CreateBucketConfiguration={
        'LocationConstraint': 'us-west-2',
    },
)
print(response)

[LadMartin]

@aaoswal, I use AWS cloud shell - web interface - I do not think that there is anything special set up. I do not even think that the LocationConstraint is needed because when I explicitly provide the name to s3.create_bucket(Bucket=BUCKET) then it will correcly return the s3.Bucket

following code does not work

>>> import boto3
>>> BUCKET='mlad-testing'
>>> s3 = boto3.resource('s3')
>>> 
>>> def add_bucket(params, **kwargs):
...     if 'Bucket' not in params:
...         params['Bucket'] = BUCKET
... 
>>> s3.meta.client.meta.events.register('provide-client-params.s3.*', add_bucket)
>>> response = s3.create_bucket()
>>> print(response)
{}

but this code with explicit Bucket argument works correctly

>>> import boto3
>>> BUCKET='mlad-testing'
>>> s3 = boto3.resource('s3')
>>> 
>>> def add_bucket(params, **kwargs):
...     if 'Bucket' not in params:
...         params['Bucket'] = BUCKET
... 
>>> s3.meta.client.meta.events.register('provide-client-params.s3.*', add_bucket)
>>> response = s3.create_bucket(Bucket=BUCKET)
>>> print(response)
s3.Bucket(name='mlad-testing')

From this my conclusion is that there is something wrong with passing the arguments between inner client.create_bucket call and s3.Bucket inicialization.

But it's just my guess I do not see into the lib internals.

[aaoswal]

Hi @LadMartin,
On deeper analysis, I found that a better workaround might be to use UUID to generate a random ID and then append it to the bucket explicitly instead when making a function call.

The event system while useful in this case will not communicate the ID of the request back to client.create_bucket() as the BUCKET_NAME is passed implicitly through events, hence resulting in the bucket being created but the response being printed as empty as the create_bucket() method does not have the request ID to track the appropriate response for the same.

Hope this helps!

[LadMartin]

Thx for clarification, @aaoswal.