Skip to content

Commit

Permalink
Update with latest benchmarks baseline
Browse files Browse the repository at this point in the history
  • Loading branch information
@lauzadis
lauzadis committed Mar 26, 2024
1 parent a87ecbe commit 82a2ab9
Showing 1 changed file with 29 additions and 39 deletions.
68 changes: 29 additions & 39 deletions tests/benchmarks/service-benchmarks/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -15,54 +15,44 @@ To run the benchmarks:
* `./gradlew :tests:benchmarks:service-benchmarks:run`
This runs the benchmark suite and prints the results to the console formatted as a Markdown table.

## Baseline as of 8/8/2023
## Baseline as of 3/26/2024

The following benchmark run serves as a baseline for future runs:

### Environment

| Hardware type | Operating system | SDK version |
|----------------|------------------|-----------------|
| EC2 m5.4xlarge | Amazon Linux 2 | 0.30.0-SNAPSHOT |
| EC2 m5.4xlarge | Amazon Linux 2 | 1.1.4 |

### Results

| | Overhead (ms) | n | min | avg | med | p90 | p99 | max |
|:----------------------|--------------:|-----:|------:|------:|------:|------:|-------:|-------:|
| **S3** | | | | | | | | |
| —HeadObject | | 1715 | 0.334 | 0.561 | 0.379 | 0.521 | 3.149 | 20.071 |
| —PutObject | | 739 | 0.306 | 0.492 | 0.337 | 0.389 | 7.958 | 16.556 |
| **SNS** | | | | | | | | |
| —GetTopicAttributes | | 3041 | 0.235 | 0.494 | 0.354 | 0.461 | 2.964 | 17.129 |
| —Publish | | 1001 | 0.199 | 0.394 | 0.224 | 0.420 | 1.262 | 16.160 |
| **STS** | | | | | | | | |
| —AssumeRole | | 1081 | 0.273 | 0.419 | 0.349 | 0.485 | 0.622 | 14.781 |
| —GetCallerIdentity | | 4705 | 0.157 | 0.242 | 0.184 | 0.217 | 0.414 | 13.459 |
| **CloudWatch** | | | | | | | | |
| —GetMetricData | | 1500 | 0.174 | 1.352 | 0.219 | 3.239 | 13.830 | 15.193 |
| —PutMetricData | | 2452 | 0.133 | 1.194 | 0.144 | 1.911 | 13.007 | 14.862 |
| **CloudWatch Events** | | | | | | | | |
| —DescribeEventBus | | 1500 | 0.156 | 0.290 | 0.187 | 0.238 | 0.530 | 18.934 |
| —PutEvents | | 4577 | 0.152 | 0.293 | 0.176 | 0.378 | 3.921 | 10.022 |
| **DynamoDB** | | | | | | | | |
| —GetItem | | 4223 | 0.135 | 0.154 | 0.148 | 0.164 | 0.216 | 2.415 |
| —PutItem | | 3059 | 0.130 | 0.154 | 0.145 | 0.178 | 0.193 | 1.771 |
| **Pinpoint** | | | | | | | | |
| —GetEndpoint | | 555 | 0.220 | 0.401 | 0.406 | 0.452 | 0.506 | 6.606 |
| —PutEvents | | 415 | 0.242 | 0.400 | 0.420 | 0.466 | 0.619 | 2.762 |

### S3 Express
S3 Express benchmarks were ran separately.

| Hardware type | Operating system | SDK version |
|----------------|-------------------|-------------|
| EC2 m5.4xlarge | Amazon Linux 2023 | 1.0.66 |

| | E2E Duration (ms) | n | min | avg | med | p90 | p99 | max |
|:--------------|------------------:|-----:|------:|------:|------:|------:|------:|-------:|
| **S3Express** | | | | | | | | |
| —PutObject | | 1950 | 7.240 | 7.487 | 7.455 | 7.617 | 7.886 | 21.096 |
| —GetObject | | 3402 | 4.049 | 4.188 | 4.141 | 4.243 | 4.470 | 20.537 |
| | Overhead (ms) | n | min | avg | med | p90 | p99 | max |
| :--- | ---: | ---: | ---: | ---: | ---: | ---: | ---: | ---: |
| **S3** | | | | | | | | |
| —HeadObject | | 1637 | 0.445 | 0.628 | 0.517 | 0.655 | 6.427 | 10.388 |
| —PutObject | | 741 | 0.442 | 0.624 | 0.517 | 0.591 | 5.214 | 11.596 |
| **SNS** | | | | | | | | |
| —GetTopicAttributes | | 4269 | 0.294 | 0.502 | 0.392 | 0.565 | 6.191 | 29.866 |
| —Publish | | 1089 | 0.255 | 0.428 | 0.337 | 0.384 | 3.072 | 9.253 |
| **STS** | | | | | | | | |
| —AssumeRole | | 1273 | 0.290 | 0.444 | 0.402 | 0.524 | 0.596 | 7.902 |
| —GetCallerIdentity | | 7039 | 0.225 | 0.286 | 0.258 | 0.286 | 0.360 | 11.254 |
| **CloudWatch** | | | | | | | | |
| —GetMetricData | | 1500 | 0.245 | 0.869 | 0.325 | 4.129 | 5.988 | 6.793 |
| —PutMetricData | | 3238 | 0.191 | 0.654 | 0.221 | 3.313 | 4.846 | 9.071 |
| **CloudWatch Events** | | | | | | | | |
| —DescribeEventBus | | 1500 | 0.223 | 0.395 | 0.312 | 0.498 | 4.932 | 10.820 |
| —PutEvents | | 7224 | 0.230 | 0.323 | 0.271 | 0.312 | 4.112 | 6.740 |
| **DynamoDB** | | | | | | | | |
| —GetItem | | 5254 | 0.210 | 0.251 | 0.246 | 0.268 | 0.293 | 3.347 |
| —PutItem | | 3361 | 0.205 | 0.268 | 0.263 | 0.301 | 0.323 | 2.693 |
| **S3Express** | | | | | | | | |
| —PutObject | | 2077 | 0.659 | 0.722 | 0.709 | 0.734 | 0.772 | 7.732 |
| —GetObject | | 3458 | 0.275 | 0.316 | 0.301 | 0.328 | 0.363 | 9.233 |
| **Secrets Manager** | | | | | | | | |
| —GetSecretValue | | 1206 | 0.282 | 0.434 | 0.375 | 0.434 | 3.829 | 7.043 |
| —PutSecretValue | | 435 | 0.281 | 0.413 | 0.352 | 0.398 | 3.217 | 6.679 |

## Methodology

Expand All @@ -72,7 +62,7 @@ This section describes how the benchmarks actually work at a high level:

These benchmarks select a handful of services to test against. The selection criterion is the top 7 services by traffic
coming from the AWS SDK for Kotlin (i.e., not from other SDKs, console, etc.). As of 7/28, those top 7 services are S3,
SNS, STS, CloudWatch, CloudWatch Events, DynamoDB, and Pinpoint (in descending order).
SNS, STS, CloudWatch, CloudWatch Events, DynamoDB, and Pinpoint (in descending order). However, Pinpoint has strict throttles that make benchmarking impossible, so Secrets Manager is selected instead.

For each service, two APIs are selected roughly corresponding to a read and a write operation (e.g., S3::HeadObject is
a read operation and S3::PutObject is a write operation). Efforts are made to ensure that the APIs selected are the top
Expand Down

0 comments on commit 82a2ab9

Please sign in to comment.