Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aidenh6307 patch 1 #212

Open
wants to merge 17 commits into
base: master
Choose a base branch
from
Open
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 22 additions & 0 deletions dockerfiles/aws/nodejs/Dockerfile.buid2023
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
ARG BASE_IMAGE
FROM ${BASE_IMAGE}

# shadow-utils
RUN dnf install -y shadow-utils && dnf clean all

ENV GOSU_VERSION 1.14
# https://github.com/tianon/gosu/releases/tag/1.14
# key https://keys.openpgp.org/search?q=tianon%40debian.org
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-amd64" \
&& chmod +x /usr/local/bin/gosu
Comment on lines +10 to +11
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add checksum verification for gosu download.

To ensure the integrity and authenticity of the gosu binary, consider adding a checksum verification step after downloading it.

RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-amd64" \
    && echo "<CHECKSUM>  /usr/local/bin/gosu" | sha256sum -c - \
    && chmod +x /usr/local/bin/gosu

Replace <CHECKSUM> with the actual checksum value.


RUN mkdir -p /sebs/
COPY dockerfiles/nodejs_installer.sh /sebs/installer.sh
COPY dockerfiles/entrypoint.sh /sebs/entrypoint.sh
RUN chmod +x /sebs/entrypoint.sh

# useradd and groupmod is installed in /usr/sbin which is not in PATH
ENV PATH=/usr/sbin:$PATH
ENV SCRIPT_FILE=/mnt/function/package.sh
CMD /bin/bash /sebs/installer.sh
ENTRYPOINT ["/sebs/entrypoint.sh"]
24 changes: 24 additions & 0 deletions dockerfiles/aws/python/Dockerfile.build2023
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
ARG BASE_IMAGE
FROM ${BASE_IMAGE}
ARG VERSION
ENV PYTHON_VERSION=${VERSION}

# shadow-utils, zip
RUN dnf install -y shadow-utils zip && dnf clean all

ENV GOSU_VERSION 1.14
# https://github.com/tianon/gosu/releases/tag/1.14
# key https://keys.openpgp.org/search?q=tianon%40debian.org
RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-amd64" \
&& chmod +x /usr/local/bin/gosu
Comment on lines +12 to +13
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add checksum verification for gosu download.

To ensure the integrity and authenticity of the gosu binary, consider adding a checksum verification step after downloading it.

RUN curl -o /usr/local/bin/gosu -SL "https://github.com/tianon/gosu/releases/download/${GOSU_VERSION}/gosu-amd64" \
    && echo "<CHECKSUM>  /usr/local/bin/gosu" | sha256sum -c - \
    && chmod +x /usr/local/bin/gosu

Replace <CHECKSUM> with the actual checksum value.


RUN mkdir -p /sebs/
COPY dockerfiles/python_installer.sh /sebs/installer.sh
COPY dockerfiles/entrypoint.sh /sebs/entrypoint.sh
RUN chmod +x /sebs/entrypoint.sh

# useradd and groupmod is installed in /usr/sbin which is not in PATH
ENV PATH=/usr/sbin:$PATH
ENV SCRIPT_FILE=/mnt/function/package.sh
CMD /bin/bash /sebs/installer.sh
ENTRYPOINT ["/sebs/entrypoint.sh"]
77 changes: 77 additions & 0 deletions load_generator/artillery_generator.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
import yaml
import sys
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove unused import.

The sys module is imported but not used in the file.

- import sys
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
import sys
Tools
Ruff

2-2: sys imported but unused

Remove unused import: sys

(F401)

import argparse
import math

def create_yaml_config(max_users, frequency, cycles):
"""
Create a YAML configuration for load testing with a sinusoidal pattern.

:param max_users: Maximum number of concurrent users
:param frequency: Duration of each cycle in seconds
:param cycles: Number of cycles to run
:return: Dictionary containing the YAML configuration
"""
# Define the initial configuration dictionary
config = {
'config': {
'target': 'http://172.17.0.2:9000', # Target URL for the load test
'phases': [], # List to store the different phases of the load test
'ensure': {
'p95': 2000 # Ensure 95% of responses are under 2000ms
}
},
'scenarios': [
{
'flow': [
{
'post': {
'url': '/post',
'json': '{{ payload }}' # JSON payload for the POST request
}
}
]
}
],
'payload': '{{ $processEnvironment.PAYLOAD_FILE }}' # Reference to the environment variable for payload file
}
Comment on lines +6 to +37
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Parameterize the target URL.

The target URL should be passed as a parameter to the function instead of being hardcoded.

- def create_yaml_config(max_users, frequency, cycles):
+ def create_yaml_config(max_users, frequency, cycles, target_url):

- 'target': 'http://172.17.0.2:9000',  # Target URL for the load test
+ 'target': target_url,  # Target URL for the load test
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
def create_yaml_config(max_users, frequency, cycles):
"""
Create a YAML configuration for load testing with a sinusoidal pattern.
:param max_users: Maximum number of concurrent users
:param frequency: Duration of each cycle in seconds
:param cycles: Number of cycles to run
:return: Dictionary containing the YAML configuration
"""
# Define the initial configuration dictionary
config = {
'config': {
'target': 'http://172.17.0.2:9000', # Target URL for the load test
'phases': [], # List to store the different phases of the load test
'ensure': {
'p95': 2000 # Ensure 95% of responses are under 2000ms
}
},
'scenarios': [
{
'flow': [
{
'post': {
'url': '/post',
'json': '{{ payload }}' # JSON payload for the POST request
}
}
]
}
],
'payload': '{{ $processEnvironment.PAYLOAD_FILE }}' # Reference to the environment variable for payload file
}
def create_yaml_config(max_users, frequency, cycles, target_url):
"""
Create a YAML configuration for load testing with a sinusoidal pattern.
:param max_users: Maximum number of concurrent users
:param frequency: Duration of each cycle in seconds
:param cycles: Number of cycles to run
:param target_url: Target URL for the load test
:return: Dictionary containing the YAML configuration
"""
# Define the initial configuration dictionary
config = {
'config': {
'target': target_url, # Target URL for the load test
'phases': [], # List to store the different phases of the load test
'ensure': {
'p95': 2000 # Ensure 95% of responses are under 2000ms
}
},
'scenarios': [
{
'flow': [
{
'post': {
'url': '/post',
'json': '{{ payload }}' # JSON payload for the POST request
}
}
]
}
],
'payload': '{{ $processEnvironment.PAYLOAD_FILE }}' # Reference to the environment variable for payload file
}


# Generate phases for each cycle
for i in range(cycles):
for j in range(10): # 10 phases per cycle for a smoother sinusoidal pattern
phase_duration = max(1, int(frequency / 10)) # Ensure phase duration is at least 1 second
t = j / 10 # Time variable from 0 to 1
users = int(max_users * (math.sin(2 * math.pi * t) + 1) / 2) # Calculate users using sine function

# Append the phase configuration to the phases list
config['config']['phases'].append({
'duration': phase_duration,
'arrivalRate': users,
'name': f'Cycle {i+1}, Phase {j+1}'
})

return config

def main():
"""
Main function to parse command-line arguments and generate the YAML configuration file.
"""
# Set up command-line argument parser
parser = argparse.ArgumentParser(description='Generate YAML config for load testing')
parser.add_argument('max_users', type=int, help='Maximum number of users (1-1000)')
parser.add_argument('frequency', type=int, help='Duration of each cycle in seconds (1-50)')
parser.add_argument('cycles', type=int, help='Number of cycles (1-50)')

args = parser.parse_args()

# Generate YAML configuration using the input parameters
config = create_yaml_config(args.max_users, args.frequency, args.cycles)

# Write YAML configuration to file
with open('load_test_config.yaml', 'w') as f:
yaml.dump(config, f, default_flow_style=False)

print("YAML configuration file 'load_test_config.yaml' has been created.")

if __name__ == '__main__':
main()
50 changes: 50 additions & 0 deletions load_generator/payloads/210_payload.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
{
"functions": [
{
"benchmark": "210.thumbnailer",
"config": {
"architecture": "x86",
"memory": 256,
"runtime": {
"language": "python",
"version": "3.7"
},
"timeout": 60
},
"hash": "442f3e4db7130c83a19c23ecd82efb70",
"instance_id": "67f07668974637517d711c11b9d1c761f7dd571af6ad2173019aec143dcf7a2a",
"name": "210.thumbnailer-python-3.7",
"port": 9000,
"triggers": [],
"url": "172.17.0.3:9000"
}
],
"inputs": [
{
"bucket": {
"bucket": "sebs-benchmarks-local-0cf48662",
"input": "210.thumbnailer-0-input",
"output": "210.thumbnailer-0-output"
},
"object": {
"height": 200,
"key": "6_astronomy-desktop-wallpaper-evening-1624438.jpg",
"width": 200
}
}
],
"storage": {
"access_key": "GNAZaCZQiH0sd-cZfpXN-2DextknZwjN5JezJ2wiciw",
"address": "172.17.0.2:9000",
"input_buckets": [
"210.thumbnailer-0-input"
],
"instance_id": "69161516e1cd11774ba54925d1828a14b6e43ca389468897bc7fa3bedb339f54",
"mapped_port": 9011,
"output_buckets": [
"210.thumbnailer-0-output"
],
"secret_key": "ae8704512eda16b6b5aa0cc267192b0fbdc931eb78acd8dba417c27c2badb735",
"type": "minio"
}
Comment on lines +36 to +49
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove hardcoded access keys.

The access keys should not be hardcoded in the file. Store them in environment variables or a secure vault.

-    "access_key": "GNAZaCZQiH0sd-cZfpXN-2DextknZwjN5JezJ2wiciw",
-    "secret_key": "ae8704512eda16b6b5aa0cc267192b0fbdc931eb78acd8dba417c27c2badb735",
+    "access_key": "${MINIO_ACCESS_KEY}",
+    "secret_key": "${MINIO_SECRET_KEY}",
Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"storage": {
"access_key": "GNAZaCZQiH0sd-cZfpXN-2DextknZwjN5JezJ2wiciw",
"address": "172.17.0.2:9000",
"input_buckets": [
"210.thumbnailer-0-input"
],
"instance_id": "69161516e1cd11774ba54925d1828a14b6e43ca389468897bc7fa3bedb339f54",
"mapped_port": 9011,
"output_buckets": [
"210.thumbnailer-0-output"
],
"secret_key": "ae8704512eda16b6b5aa0cc267192b0fbdc931eb78acd8dba417c27c2badb735",
"type": "minio"
}
"storage": {
"access_key": "${MINIO_ACCESS_KEY}",
"address": "172.17.0.2:9000",
"input_buckets": [
"210.thumbnailer-0-input"
],
"instance_id": "69161516e1cd11774ba54925d1828a14b6e43ca389468897bc7fa3bedb339f54",
"mapped_port": 9011,
"output_buckets": [
"210.thumbnailer-0-output"
],
"secret_key": "${MINIO_SECRET_KEY}",
"type": "minio"
}
Tools
Gitleaks

37-37: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)


47-47: Detected a Generic API Key, potentially exposing access to various services and sensitive operations.

(generic-api-key)

}
27 changes: 27 additions & 0 deletions load_generator/readme.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# SeBS Load Generator

## Overview

This tool allows the user to specify parameters for entry into a docker container.

## New Prerequisites
- Artillery
- Pyyaml

## Setup

### 1. Install Dependencies

`pip install artillery pyyaml`

### 2. Build Config File

`python artillery_generator <users> <phase duration> <num cycles>`

### 3. Run the File

`export PAYLOAD_FILE=/payloads/<benchmark_number>_payload.json`

`artillery run load_test_config.yml`

### 4. Enjoy!