Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add interface for NoSQL storage #214

Draft
wants to merge 114 commits into
base: master
Choose a base branch
from
Draft
Changes from 1 commit
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
0363899
[system] Add basic interface for allocating NoSQL storage
mcopik Jul 26, 2024
dd3384c
[aws] Implement allocation of DynamoDB tables
mcopik Jul 27, 2024
847534f
[system] Allocate NoSQL tables for the selected benchmark
mcopik Jul 27, 2024
6a62633
[system] Remove debug printouts
mcopik Jul 27, 2024
1c30372
[aws] Add first version of DynamoDB wrapper
mcopik Jul 27, 2024
ed22ea3
[azure] First implementation of wrapper for CosmosDB
mcopik Jul 27, 2024
b21cd34
[aws] Update AWS wrapper with query interface
mcopik Jul 27, 2024
e85c348
[gcp] Add wrapper for google cloud datastore
mcopik Jul 27, 2024
f28b496
[system] Add packages needed to support storage API
mcopik Jul 27, 2024
a1cd286
[aws] Formatting AWS nosql wrapper
mcopik Jul 28, 2024
9b3a284
[system] Split system packages into benchmark modules
mcopik Jul 28, 2024
3f3b440
[aws] Support sorting keys in DynamoDB
mcopik Jul 28, 2024
8ca8e24
[system] Fix incorrect variable name
mcopik Jul 28, 2024
1fdf04f
[system] Adapt cache implementation of NoSQL to support different lay…
mcopik Jul 28, 2024
2cd1bc4
[azure] Implement CosmosDB management
mcopik Jul 28, 2024
f90971f
[azure] Add management of CosmosDB accoutns
mcopik Jul 28, 2024
020fcd6
[azure] Install locally CosmosDB library to allocate containers
mcopik Jul 28, 2024
6959cd6
[azure] Add missing handler file
mcopik Jul 28, 2024
f089017
[azure] Allocate clients to databases and containers when they alread…
mcopik Jul 28, 2024
d6e14de
[system] Improved logging
mcopik Jul 28, 2024
0c7906a
[azure] Export NoSQL database information
mcopik Jul 28, 2024
8314f41
[system] Initialize storage before deploying function
mcopik Jul 28, 2024
b975d78
[azure] Help mypy recognize that variable is never None
mcopik Jul 28, 2024
b9ae6ca
[azure] Linting nosql wrapper
mcopik Jul 28, 2024
53bb33e
[azure] Export the proper database name as env
mcopik Jul 28, 2024
8d211a2
[azure] Export tables as envs
mcopik Jul 29, 2024
43680a1
[azure] Fix error of allocating the same container many times
mcopik Jul 29, 2024
cf6fc22
[azure] Avoid creating duplicated HTTP triggers
mcopik Jul 29, 2024
f49714c
[aws] Remove Decimals from DynamoDB results
mcopik Jul 29, 2024
36dd13f
[aws] Append environment variables to send Table data
mcopik Jul 29, 2024
4312ebc
[aws] Linting
mcopik Jul 29, 2024
ec62b9c
[azure] Prevent overwriting existing variables
mcopik Jul 29, 2024
ab62ec9
[azure Add sleep to avoid problems with stale values of envs
mcopik Jul 29, 2024
e36cf80
[gcp] Add Google Cloud manage image with gcloud
mcopik Jul 29, 2024
69fe546
[system] [aws] Move table management to each implementation to adjust…
mcopik Jul 29, 2024
50f8ac6
[azure] Remove mapping of tables
mcopik Jul 29, 2024
b836597
[system] Fix bug in cache processing - now storage is updated before …
mcopik Jul 29, 2024
853d083
[gcp] Add gcloud CLI handling
mcopik Jul 29, 2024
89560ca
[system] Linting
mcopik Jul 29, 2024
e8288a0
[gcp] Extend Docker images
mcopik Jul 29, 2024
4a944a0
[gcp] Update wrapper with database init
mcopik Jul 29, 2024
e511885
[gcp] Update docs with permissions
mcopik Jul 29, 2024
ce9076b
[aws] Ensure we update envs
mcopik Jul 29, 2024
8204715
[gcp] Add management of datastore instances
mcopik Jul 29, 2024
8e66ca3
[gcp] Correctly mount read-only files in CLI
mcopik Jul 29, 2024
994f107
[gcp] Update function envs to support NoSQL storage
mcopik Jul 29, 2024
22333a5
[benchmarks] Add initial config for 130.crud-api benchmark
mcopik Jul 29, 2024
820c5b4
[benchmarks] Update module configuration
mcopik Jul 29, 2024
85de381
[azure] [gcp] Fix nosql import
mcopik Jul 29, 2024
7f23fe2
[azure] Move storage connection string from request to config
mcopik Jul 29, 2024
2af6082
[system] Update benchmark input config
mcopik Jul 29, 2024
92c7ce6
[gcp] Add missing dependency
mcopik Jul 29, 2024
ed00c14
[experiments] Reorder input processing and function generation
mcopik Jul 29, 2024
12b475d
[whisk] Skip nosql
mcopik Jul 29, 2024
06ece22
[gcp] Fix mypy issues
mcopik Jul 29, 2024
7118b58
[benchmarks] Add test implementation of the CRUD benchmark
mcopik Jul 30, 2024
6c78152
[benchmarks] Work in progress on the CRUD benchmark
mcopik Jul 31, 2024
3d58e42
[storage] Adapt interface to multiple tiers of storage
mcopik Jul 31, 2024
045f077
[local] Adapt local to the new storage config
mcopik Jul 31, 2024
1c48a10
[system] First attempt to break the circular dependency between Syste…
mcopik Jul 31, 2024
b8b04f1
[local] Adapt new resource style
mcopik Jul 31, 2024
c9e2635
[system] Expand storage configuration
mcopik Jul 31, 2024
e9c954b
[system] remove dbg output
mcopik Jul 31, 2024
f4bd76f
[minio] Support data volume and explicit version
mcopik Jul 31, 2024
afa5d7e
[minio] Fix bug in setting volume path
mcopik Jul 31, 2024
a0c88ad
[scylla] Add ScyllaDB as a local NoSQL backend
mcopik Jul 31, 2024
898328f
[scylla] Add stop for the container
mcopik Jul 31, 2024
18f4692
[aws] Restructurize storage allocation
mcopik Jul 31, 2024
67fa42b
[benchmarks] Updated version of 130.crud-api
mcopik Jul 31, 2024
da0e009
[system] Add initialization of NoSQL results
mcopik Jul 31, 2024
f953768
[aws] Implement initialization of DynamoDB
mcopik Jul 31, 2024
e5d72b8
[system] Fix cache issue - we now prepare benchmark data before code …
mcopik Jul 31, 2024
dc60969
[benchmarks] Update input API
mcopik Jul 31, 2024
ef54402
[system] Additional logging
mcopik Aug 1, 2024
ff9314c
[benchmarks] Update input API
mcopik Aug 1, 2024
c6463e2
[benchmarks] Fix incorrect IDs and division by zero
mcopik Aug 1, 2024
9916028
[bechmarks] Implement final sizes for 130.crud-api
mcopik Aug 1, 2024
9dc5d03
[aws] Implement the update method for DynamoDB
mcopik Aug 2, 2024
078bdbc
[aws] Fix waiting condition when creating DynamoDB table
mcopik Aug 2, 2024
3d7d1dc
[gcp] Implement new resource class
mcopik Aug 2, 2024
51fafd4
[gcp] Implement upload function for Datastore
mcopik Aug 2, 2024
0843b00
[gcp] Add wrapper for update function
mcopik Aug 2, 2024
1a7325f
[gcp] Fix incorrect update implementation
mcopik Aug 2, 2024
23ed66b
[gcp] Fix linting issues
mcopik Aug 2, 2024
d277923
[azure] Move resources to a different file
mcopik Aug 2, 2024
ff1a696
[azure] Linting
mcopik Aug 2, 2024
1077605
[azure] Implement the missing function for CosmosDB
mcopik Aug 2, 2024
6758863
[azure] Add the new abstraction of system resources
mcopik Aug 2, 2024
b9dfc92
[azure] Replace insert with upsert
mcopik Aug 2, 2024
383a62b
[azure] Add NoSQL update
mcopik Aug 2, 2024
ecc2816
[azure] Fix allocation of database client
mcopik Aug 3, 2024
0e1e1fc
[local] Add base methods for local storage config
mcopik Aug 7, 2024
945b0fd
[local] Make storage paths consistent
mcopik Aug 7, 2024
be85773
[local] Drop unused storage config in output
mcopik Aug 7, 2024
7bb8d87
[local] Ignore Docker volumes in git
mcopik Aug 7, 2024
a5977b7
[system] Allow for multiple input files with storage definition
mcopik Aug 7, 2024
c168c0f
[local] Better error description
mcopik Aug 7, 2024
2caa564
[system] Remove debug printout
mcopik Aug 7, 2024
2e6f75a
[local] Make storage implementation cacheable and more generic by off…
mcopik Aug 7, 2024
92d610b
[local] Final standardization of volume names
mcopik Aug 7, 2024
3ac12e7
[storage] Remove unnecessary storage classes
mcopik Aug 7, 2024
a76c7d9
[system] Fix type hint
mcopik Aug 7, 2024
0a678b5
[local] Support ScyllaDB serialization
mcopik Aug 7, 2024
cc9c103
[local] Linting
mcopik Aug 7, 2024
cf66c2c
[system] Support removing functions that are no longer available
mcopik Aug 7, 2024
8db684c
[aws] Linting
mcopik Aug 7, 2024
223f368
[storage] Implement allocation of ScyllaDB tables
mcopik Aug 7, 2024
124cf79
[storage] Fix correct deployment name in output
mcopik Aug 7, 2024
ecc5cbf
[storage] Linting
mcopik Aug 7, 2024
dae8ec1
[local] Export NoSQL settings
mcopik Aug 7, 2024
1796ca8
[storage] Add default implementation of envs for NoSQL
mcopik Aug 7, 2024
e504674
[local] Add ScyllaDB wrapper
mcopik Aug 7, 2024
0642eaa
[local] Implement function update for local container
mcopik Aug 7, 2024
b293f7a
[storage] Linting
mcopik Aug 7, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
[aws] Linting
  • Loading branch information
mcopik committed Aug 7, 2024
commit 8db684cf3f3d3cb373e70c3d127296b31230169a
5 changes: 3 additions & 2 deletions sebs/aws/dynamodb.py
Original file line number Diff line number Diff line change
@@ -85,6 +85,7 @@ def writer_func(
):

table_name = self._get_table_name(benchmark, table)
assert table_name is not None

for key in (primary_key, secondary_key):
if key is not None:
@@ -118,8 +119,8 @@ def create_table(
ret = self.client.create_table(
TableName=table_name,
BillingMode="PAY_PER_REQUEST",
AttributeDefinitions=definitions,
KeySchema=key_schema,
AttributeDefinitions=definitions, # type: ignore
KeySchema=key_schema, # type: ignore
)

if ret["TableDescription"]["TableStatus"] == "CREATING":