Inconsistent/poor lambda invoke performance

[mrworkman]

Describe the bug

Despite using a cached lambda client, invocation frequently performs poorly.

Expected Behavior

Lambda client should consistently invoke the same lambda in roughly the same amount of time each time, in the least amount of time.

Current Behavior

When invoking a lambda, the amount of time required to invoke it varies wildly, and generally takes much longer to run than the lambda itself.

In this example, the time to do the invocation fluctuates by an order of magnitude between invocations (each is with 10s delay between tests):

• Total execution time: 697.498ms
• Total execution time: 53.168ms
• Total execution time: 224.166ms
• Total execution time: 46.677ms
• Total execution time: 300.755ms

The lambda itself only actually runs for 3-10ms (very consistently), yet the lambda client's invoke function itself seems to take an excessive amount of time. This can be seen in tracing:

Reproduction Steps

client = boto3.client("lambda")

def invoke_lambda(function_name, payload):
    response = client.invoke(
        FunctionName=function_name,
        LogType="Tail",
        Payload=json.dumps(payload),
        Qualifier="latest",
    )

    return response["Payload"].read().decode("utf-8")

for _ in range(0, 50):
    start = datetime.datetime.now()
    invoke_lambda(
        function_name="arn:aws:lambda:eu-west-1:<ACCT_NO>:function:<FUNC_NAME>",
        payload={
            "something": "something"
        },
    )
    print(f"Total execution time: {(datetime.datetime.now() - start).microseconds / 1000}ms")

Possible Solution

No response

Additional Information/Context

No response

SDK version used

boto3 1.29.6

Environment details (OS name and version, etc.)

AWS Lambda Python 3.9 runtime

[mrworkman]

After doing a considerable amount of investigation into this issue, including invoking the lambda by building my own request by calling https://lambda.eu-west-1.amazonaws.com/2015-03-31/functions/<FUNCTION_NAME>/invocations directly, I'm not convinced this is an issue brought on by boto3/botocore. Alternatives exhibit exactly the same behaviour.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.

[github-actions]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see.
If you need more assistance, please either tag a team member or open a new issue that references this one.
If you wish to keep having a conversation with other community members under this issue feel free to do so.