boto3 Read timeout error when listing snapshots

[gauravkcldcvr]

I'm using below config in script to list snapshots which is expected to be around 200,000 for the region (using pagination here). Have used the below config block to pass timeouts. (tried with 5000 as value too)

ec2_client = boto3.client('ec2',region_name='ap-south-1',config=Config(read_timeout=1000,connect_timeout=1000,retries={'max_attempts': 0}))

And it still returns the error -

raise ReadTimeoutError(endpoint_url=request.url, error=e)
botocore.exceptions.ReadTimeoutError: Read timeout on endpoint URL: "https://ec2.ap-south-1.amazonaws.com/"

anything i'm missing here which is causing this, or is it due to the high number of snapshots? can anyone suggest me how can this be fixed.

Without using pagination here, when used below block it gives the results of the 1000 snapshots, I believe this is happening due having the large number of data and it's failing to parse resulting in timeout error.

response = ec2_client.describe_snapshots(
            MaxResults=1000,
            )

[tim-finnigan]

Hi @gauravkcldcvr have you tried increasing the max_attempts in addition to the read and connect timeout lengths?

[gauravkcldcvr]

Yes @tim-finnigan tried increasing max_attempts as well but that didn't help.

[tim-finnigan]

What are the highest numbers you tried for max_attempts, read_timeout, and connect_timeout? 200k results is a large payload to return. Is there any filtering you can do to get the specific results you're looking for?

[gauravkcldcvr]

max_attempts=5, read_timeout=5000, and connect_timeout=5000 was the highest value I had tried with. Yes, even if it works it's gonna consume high time to return outputs. We were currently looking to get all snapshots list, but will be checking using filters (like Owner as me) but still that'll be around 100k as well.