Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Register challenges in batches #358

Merged
merged 11 commits into from
Aug 28, 2023
Merged

Conversation

poszu
Copy link
Contributor

@poszu poszu commented Aug 28, 2023

Closes #357

The submitted challenges are collected in a batch that is written to disk (with sync):

  • when it reaches max batch size (default: 1000)
  • periodically (default: 100ms since the batch was created).

Comparison of TestLoadSubmits before and after on an NVME drive with default settings:

submitting 10000 challenges took 1m3.840035945s

vs

submitting 10000 challenges took 1.397493686s

@poszu poszu requested review from dshulyak and fasmat August 28, 2023 09:53
@codecov
Copy link

codecov bot commented Aug 28, 2023

Codecov Report

Merging #358 (e0648df) into develop (41d1da3) will increase coverage by 1.9%.
The diff coverage is 94.1%.

@@            Coverage Diff            @@
##           develop    #358     +/-   ##
=========================================
+ Coverage     71.7%   73.6%   +1.9%     
=========================================
  Files           20      20             
  Lines         1749    1859    +110     
=========================================
+ Hits          1255    1370    +115     
+ Misses         366     363      -3     
+ Partials       128     126      -2     
Files Changed Coverage Δ
service/service.go 76.7% <80.0%> (-0.2%) ⬇️
service/round.go 83.1% <98.8%> (+5.9%) ⬆️
config/config.go 74.0% <100.0%> (+0.6%) ⬆️
server/server.go 75.8% <100.0%> (+1.4%) ⬆️
service/pow_verifier.go 96.1% <100.0%> (-3.9%) ⬇️

... and 3 files with indirect coverage changes

service/round_test.go Outdated Show resolved Hide resolved
service/service.go Show resolved Hide resolved
service/round.go Show resolved Hide resolved
config/config.go Show resolved Hide resolved
config/config.go Show resolved Hide resolved
service/round.go Outdated
Subsystem: "round",
Name: "batch_write_latency_seconds",
Help: "Latency of batch write operations",
Buckets: prometheus.ExponentialBuckets(0.0001, 1.5, 20),
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i think it can start at 10ms (0.01s). otherwise there will be a lot of wasted metrics

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think 10ms is a little high as I observe latencies < 10ms. Changed min to 1ms.

server/server.go Outdated Show resolved Hide resolved
@poszu poszu added this pull request to the merge queue Aug 28, 2023
Merged via the queue into develop with commit 925a5b8 Aug 28, 2023
11 checks passed
@poszu poszu deleted the register-challenges-in-batches branch August 28, 2023 17:09
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Write submitted challenges to DB in batches
2 participants