From 82a2ab947671723d4de526c9fede1f487d863ee5 Mon Sep 17 00:00:00 2001 From: Matas Lauzadis Date: Tue, 26 Mar 2024 10:33:08 -0400 Subject: [PATCH] Update with latest benchmarks baseline --- tests/benchmarks/service-benchmarks/README.md | 68 ++++++++----------- 1 file changed, 29 insertions(+), 39 deletions(-) diff --git a/tests/benchmarks/service-benchmarks/README.md b/tests/benchmarks/service-benchmarks/README.md index e7df911ca14..ccadfc73906 100644 --- a/tests/benchmarks/service-benchmarks/README.md +++ b/tests/benchmarks/service-benchmarks/README.md @@ -15,7 +15,7 @@ To run the benchmarks: * `./gradlew :tests:benchmarks:service-benchmarks:run` This runs the benchmark suite and prints the results to the console formatted as a Markdown table. -## Baseline as of 8/8/2023 +## Baseline as of 3/26/2024 The following benchmark run serves as a baseline for future runs: @@ -23,46 +23,36 @@ The following benchmark run serves as a baseline for future runs: | Hardware type | Operating system | SDK version | |----------------|------------------|-----------------| -| EC2 m5.4xlarge | Amazon Linux 2 | 0.30.0-SNAPSHOT | +| EC2 m5.4xlarge | Amazon Linux 2 | 1.1.4 | ### Results -| | Overhead (ms) | n | min | avg | med | p90 | p99 | max | -|:----------------------|--------------:|-----:|------:|------:|------:|------:|-------:|-------:| -| **S3** | | | | | | | | | -| —HeadObject | | 1715 | 0.334 | 0.561 | 0.379 | 0.521 | 3.149 | 20.071 | -| —PutObject | | 739 | 0.306 | 0.492 | 0.337 | 0.389 | 7.958 | 16.556 | -| **SNS** | | | | | | | | | -| —GetTopicAttributes | | 3041 | 0.235 | 0.494 | 0.354 | 0.461 | 2.964 | 17.129 | -| —Publish | | 1001 | 0.199 | 0.394 | 0.224 | 0.420 | 1.262 | 16.160 | -| **STS** | | | | | | | | | -| —AssumeRole | | 1081 | 0.273 | 0.419 | 0.349 | 0.485 | 0.622 | 14.781 | -| —GetCallerIdentity | | 4705 | 0.157 | 0.242 | 0.184 | 0.217 | 0.414 | 13.459 | -| **CloudWatch** | | | | | | | | | -| —GetMetricData | | 1500 | 0.174 | 1.352 | 0.219 | 3.239 | 13.830 | 15.193 | -| —PutMetricData | | 2452 | 0.133 | 1.194 | 0.144 | 1.911 | 13.007 | 14.862 | -| **CloudWatch Events** | | | | | | | | | -| —DescribeEventBus | | 1500 | 0.156 | 0.290 | 0.187 | 0.238 | 0.530 | 18.934 | -| —PutEvents | | 4577 | 0.152 | 0.293 | 0.176 | 0.378 | 3.921 | 10.022 | -| **DynamoDB** | | | | | | | | | -| —GetItem | | 4223 | 0.135 | 0.154 | 0.148 | 0.164 | 0.216 | 2.415 | -| —PutItem | | 3059 | 0.130 | 0.154 | 0.145 | 0.178 | 0.193 | 1.771 | -| **Pinpoint** | | | | | | | | | -| —GetEndpoint | | 555 | 0.220 | 0.401 | 0.406 | 0.452 | 0.506 | 6.606 | -| —PutEvents | | 415 | 0.242 | 0.400 | 0.420 | 0.466 | 0.619 | 2.762 | - -### S3 Express -S3 Express benchmarks were ran separately. - -| Hardware type | Operating system | SDK version | -|----------------|-------------------|-------------| -| EC2 m5.4xlarge | Amazon Linux 2023 | 1.0.66 | - -| | E2E Duration (ms) | n | min | avg | med | p90 | p99 | max | -|:--------------|------------------:|-----:|------:|------:|------:|------:|------:|-------:| -| **S3Express** | | | | | | | | | -| —PutObject | | 1950 | 7.240 | 7.487 | 7.455 | 7.617 | 7.886 | 21.096 | -| —GetObject | | 3402 | 4.049 | 4.188 | 4.141 | 4.243 | 4.470 | 20.537 | +| | Overhead (ms) | n | min | avg | med | p90 | p99 | max | +| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: | +| **S3** | | | | | | | | | +| —HeadObject | | 1637 | 0.445 | 0.628 | 0.517 | 0.655 | 6.427 | 10.388 | +| —PutObject | | 741 | 0.442 | 0.624 | 0.517 | 0.591 | 5.214 | 11.596 | +| **SNS** | | | | | | | | | +| —GetTopicAttributes | | 4269 | 0.294 | 0.502 | 0.392 | 0.565 | 6.191 | 29.866 | +| —Publish | | 1089 | 0.255 | 0.428 | 0.337 | 0.384 | 3.072 | 9.253 | +| **STS** | | | | | | | | | +| —AssumeRole | | 1273 | 0.290 | 0.444 | 0.402 | 0.524 | 0.596 | 7.902 | +| —GetCallerIdentity | | 7039 | 0.225 | 0.286 | 0.258 | 0.286 | 0.360 | 11.254 | +| **CloudWatch** | | | | | | | | | +| —GetMetricData | | 1500 | 0.245 | 0.869 | 0.325 | 4.129 | 5.988 | 6.793 | +| —PutMetricData | | 3238 | 0.191 | 0.654 | 0.221 | 3.313 | 4.846 | 9.071 | +| **CloudWatch Events** | | | | | | | | | +| —DescribeEventBus | | 1500 | 0.223 | 0.395 | 0.312 | 0.498 | 4.932 | 10.820 | +| —PutEvents | | 7224 | 0.230 | 0.323 | 0.271 | 0.312 | 4.112 | 6.740 | +| **DynamoDB** | | | | | | | | | +| —GetItem | | 5254 | 0.210 | 0.251 | 0.246 | 0.268 | 0.293 | 3.347 | +| —PutItem | | 3361 | 0.205 | 0.268 | 0.263 | 0.301 | 0.323 | 2.693 | +| **S3Express** | | | | | | | | | +| —PutObject | | 2077 | 0.659 | 0.722 | 0.709 | 0.734 | 0.772 | 7.732 | +| —GetObject | | 3458 | 0.275 | 0.316 | 0.301 | 0.328 | 0.363 | 9.233 | +| **Secrets Manager** | | | | | | | | | +| —GetSecretValue | | 1206 | 0.282 | 0.434 | 0.375 | 0.434 | 3.829 | 7.043 | +| —PutSecretValue | | 435 | 0.281 | 0.413 | 0.352 | 0.398 | 3.217 | 6.679 | ## Methodology @@ -72,7 +62,7 @@ This section describes how the benchmarks actually work at a high level: These benchmarks select a handful of services to test against. The selection criterion is the top 7 services by traffic coming from the AWS SDK for Kotlin (i.e., not from other SDKs, console, etc.). As of 7/28, those top 7 services are S3, -SNS, STS, CloudWatch, CloudWatch Events, DynamoDB, and Pinpoint (in descending order). +SNS, STS, CloudWatch, CloudWatch Events, DynamoDB, and Pinpoint (in descending order). However, Pinpoint has strict throttles that make benchmarking impossible, so Secrets Manager is selected instead. For each service, two APIs are selected roughly corresponding to a read and a write operation (e.g., S3::HeadObject is a read operation and S3::PutObject is a write operation). Efforts are made to ensure that the APIs selected are the top