Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Smash Processor #613

Merged
merged 179 commits into from
Sep 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
179 commits
Select commit Hold shift + click to select a range
901b908
Smash key-gen into independent crate
kayabaNerve Aug 16, 2024
722e86b
Smash key-gen out of processor
kayabaNerve Aug 16, 2024
914184c
Smash out the signer
kayabaNerve Aug 19, 2024
50ad450
Cleanup DB handling a bit in key-gen/attempt-manager
kayabaNerve Aug 19, 2024
99661aa
Move scanner.rs to scanner/lib.rs
kayabaNerve Aug 19, 2024
6bc607b
Extend serai-db with support for generic keys/values
kayabaNerve Aug 19, 2024
090c0c1
Definition and delineation of tasks within the scanner
kayabaNerve Aug 20, 2024
e19a126
Flesh out new scanner a bit more
kayabaNerve Aug 20, 2024
26a5475
Correct misc compilation errors
kayabaNerve Aug 20, 2024
a8cc368
Differentiate BlockHeader from Block
kayabaNerve Aug 20, 2024
0bc03c3
Handle dust
kayabaNerve Aug 20, 2024
2578ba6
Document expectations on Eventuality task and correct code determinin…
kayabaNerve Aug 20, 2024
cdcfe4a
Add bounds for the eventuality task
kayabaNerve Aug 21, 2024
ceb1921
Add Eventuality back to processor primitives
kayabaNerve Aug 22, 2024
9566fd3
Decide flow between scan/eventuality/report
kayabaNerve Aug 24, 2024
e8dd4de
Correct compilation errors
kayabaNerve Aug 24, 2024
e1e7432
Finish scan task
kayabaNerve Aug 24, 2024
2f5dc4b
Flesh out report task
kayabaNerve Aug 24, 2024
6873331
Have the scan flag blocks with change/branch/forwarded as notable
kayabaNerve Aug 24, 2024
2f08a12
Flesh out eventuality task
kayabaNerve Aug 25, 2024
ebf0990
Add a callback to accumulate outputs and return the new Eventualities
kayabaNerve Aug 27, 2024
a861637
Move ContinuallyRan into primitives
kayabaNerve Aug 27, 2024
2dccfa8
Add helper methods
kayabaNerve Aug 27, 2024
414060e
Make index a folder, not a file
kayabaNerve Aug 27, 2024
9810dbf
Make a dedicated IndexDb
kayabaNerve Aug 27, 2024
bd6f39d
Make Eventuality a folder, not a file
kayabaNerve Aug 27, 2024
960ecd8
Add dedicated Eventuality DB and stub missing fns
kayabaNerve Aug 27, 2024
65e4862
Don't expose IndexDb throughout the crate
kayabaNerve Aug 27, 2024
1b949d6
Add a DbChannel between scan and eventuality task
kayabaNerve Aug 27, 2024
14b9c62
Use a channel for the InInstructions
kayabaNerve Aug 27, 2024
a632630
Add sorts for safety even upon non-determinism
kayabaNerve Aug 27, 2024
e720fe7
Logs, documentation, misc
kayabaNerve Aug 27, 2024
eebb5c2
Implement key retiry
kayabaNerve Aug 28, 2024
ff02b40
Make scan.rs a folder, not a file
kayabaNerve Aug 28, 2024
a92330c
Add ScanDb
kayabaNerve Aug 28, 2024
bba553a
Make report.rs a folder, not a file
kayabaNerve Aug 28, 2024
9e13423
Add ReportDb
kayabaNerve Aug 28, 2024
e5cba72
Have Scanner::new spawn tasks
kayabaNerve Aug 29, 2024
85bab09
Fill in various DB functions
kayabaNerve Aug 29, 2024
3a23c2c
Cache the cost to aggregate
kayabaNerve Aug 29, 2024
84b0bec
Add API to publish Batches with
kayabaNerve Aug 29, 2024
d822346
Route burns through the scanner
kayabaNerve Aug 29, 2024
497d7fb
Clarify output-to-self to output-to-Serai
kayabaNerve Aug 29, 2024
ca52e6a
Add helper to intake Eventualities
kayabaNerve Aug 29, 2024
b610135
Add key management to the scheduler
kayabaNerve Aug 29, 2024
7db8bd9
Call flush_key
kayabaNerve Aug 29, 2024
06d0e49
Pass the lifetime information to the scheduler
kayabaNerve Aug 29, 2024
d1852f9
Add sanity checks we haven't prior reported an InInstruction for/accu…
kayabaNerve Aug 30, 2024
8c3a37c
Add note on why LifetimeStage is monotonic
kayabaNerve Aug 30, 2024
7dc567d
Have the Eventuality task drop outputs which aren't ours and aren't w…
kayabaNerve Aug 30, 2024
e888b97
Check a queued key has never been queued before
kayabaNerve Aug 30, 2024
d441d6f
Impl ScanData serialization in the DB
kayabaNerve Aug 30, 2024
7283a53
Remove unused ID -> number lookup
kayabaNerve Aug 30, 2024
d5434b0
Route the DB w.r.t. forwarded outputs' information
kayabaNerve Aug 30, 2024
41b5c9b
Have acknowledge_block take in the results of the InInstructions exec…
kayabaNerve Aug 30, 2024
5b1ccaf
Replace acknowledge_block with acknowledge_batch
kayabaNerve Aug 30, 2024
68c46a6
Don't have `acknowledge_batch` immediately run
kayabaNerve Aug 30, 2024
b6db456
Add crate for the transaction-chaining Scheduler
kayabaNerve Aug 30, 2024
3c43630
Add processor/scheduler/utxo/primitives
kayabaNerve Sep 1, 2024
1711efb
Expand primitives/scanner with niceties needed for the scheduler
kayabaNerve Sep 1, 2024
e1a965f
Outline of the transaction-chaining scheduler
kayabaNerve Sep 1, 2024
5e7e3e4
Add scheduler-primitives
kayabaNerve Sep 2, 2024
7a35231
Better document the forwarded output flow
kayabaNerve Sep 3, 2024
65ed943
Fix bug in the scanner regarding forwarded output amounts
kayabaNerve Sep 3, 2024
565fd59
Add input aggregation in the transaction-chaining scheduler
kayabaNerve Sep 3, 2024
408420e
Ensure the transaction-chaining scheduler doesn't accumulate the same…
kayabaNerve Sep 3, 2024
fe889f9
Work on the tree logic in the transaction-chaining scheduler
kayabaNerve Sep 3, 2024
ea519ad
Near-complete version of the tree algorithm in the transaction-chaini…
kayabaNerve Sep 3, 2024
db9df8a
Finish the tree logic in the transaction-chaining scheduler
kayabaNerve Sep 4, 2024
ba8477d
Finish routing output flushing
kayabaNerve Sep 4, 2024
4c25367
Add non-transaction-chaining scheduler
kayabaNerve Sep 4, 2024
07e7068
Remove vast swaths of legacy code in the processor
kayabaNerve Sep 4, 2024
451170e
Move additional_key.rs to serai-processor-view-keys
kayabaNerve Sep 4, 2024
91d04b6
Split processor into bitcoin-processor, ethereum-processor, monero-pr…
kayabaNerve Sep 5, 2024
d2fb327
Add empty serai-processor-signers library
kayabaNerve Sep 5, 2024
b61e5b0
Minor work on the transaction signing task
kayabaNerve Sep 5, 2024
db74a71
Further work on transaction signing
kayabaNerve Sep 6, 2024
45576eb
Finish transaction signing task with TX rebroadcast code
kayabaNerve Sep 6, 2024
ebd8675
Tidy transaction signing task
kayabaNerve Sep 6, 2024
279cb72
Work on the higher-level signers API
kayabaNerve Sep 7, 2024
442198c
Expand task management
kayabaNerve Sep 8, 2024
30c4c1a
Route the coordinator, fix race conditions in the signers library
kayabaNerve Sep 9, 2024
aa71206
Replace scanner's BatchPublisher with a pair of DB channels
kayabaNerve Sep 9, 2024
ac106e3
Add BatchSignerTask
kayabaNerve Sep 9, 2024
336ffe2
Have the coordinator task publish Batches
kayabaNerve Sep 9, 2024
26fb7c2
Tidy messages, publish all Batches to the coordinator
kayabaNerve Sep 9, 2024
4963735
Strongly type SlashReport, populate cosign/slash report tasks with work
kayabaNerve Sep 9, 2024
d4ff05d
SlashReport signing and signature publication
kayabaNerve Sep 9, 2024
9669a89
Add CosignerTask to signers, completing it
kayabaNerve Sep 9, 2024
1143b67
Remove old signer impls
kayabaNerve Sep 9, 2024
fe9ca4c
Bitcoin Output/Transaction definitions
kayabaNerve Sep 10, 2024
5265744
Add bitcoin Block trait impl
kayabaNerve Sep 10, 2024
3c331a3
Remove bound that WINDOW_LENGTH < CONFIRMATIONS
kayabaNerve Sep 10, 2024
eadf2bb
Bitcoin ScannerFeed
kayabaNerve Sep 10, 2024
d3c4e01
Satisfy Scheduler for Bitcoin
kayabaNerve Sep 11, 2024
87abac1
Bitcoin TransactionPublisher
kayabaNerve Sep 11, 2024
ad3b07c
Bitcoin processor lib.rs -> main.rs
kayabaNerve Sep 11, 2024
876327d
Implement presumed_origin
kayabaNerve Sep 11, 2024
b2f06cc
Add an anyone-can-pay output to every Bitcoin transaction
kayabaNerve Sep 11, 2024
196fc3b
Misc changes in response to prior two commits
kayabaNerve Sep 11, 2024
9427993
Adjust Bitcoin processor layout
kayabaNerve Sep 11, 2024
a0bdcf9
Bitcoin Key Gen
kayabaNerve Sep 11, 2024
b5c040f
Start on the new processor main loop
kayabaNerve Sep 11, 2024
a4e72c0
Add note to signers on reducing disk IO
kayabaNerve Sep 11, 2024
69bbf09
Note better message structure in messages
kayabaNerve Sep 11, 2024
507a37d
Continue filling out main loop
kayabaNerve Sep 11, 2024
bd962eb
Misc tidying of serai-db calls
kayabaNerve Sep 11, 2024
dadce1a
Add section documenting the safety of txindex upon reorganizations
kayabaNerve Sep 11, 2024
dff2ef5
Add binary search to find the block to start scanning from
kayabaNerve Sep 11, 2024
57dc197
serai-processor-bin
kayabaNerve Sep 11, 2024
2f5d141
Use a local DB channel for sending to the message-queue
kayabaNerve Sep 11, 2024
b9b591a
Monero processor primitives
kayabaNerve Sep 12, 2024
e3c1b12
Correct Multisig Rotation to use WINDOW_LENGTH where proper
kayabaNerve Sep 13, 2024
ea9cd48
Misc continuances on the Monero processor
kayabaNerve Sep 13, 2024
a969240
Remove async-trait from processor/
kayabaNerve Sep 13, 2024
3db4983
Define subaddress indexes to use
kayabaNerve Sep 13, 2024
68f46d1
cargo fmt signers/scanner
kayabaNerve Sep 13, 2024
558a6d8
Monero Processor scan, check_for_eventuality_resolutions
kayabaNerve Sep 13, 2024
81651cb
Monero time_for_block, dust
kayabaNerve Sep 13, 2024
4bfeb9d
Add a database of all Monero outs into the processor
kayabaNerve Sep 14, 2024
47e9333
Allow scheduler's creation of transactions to be async and error
kayabaNerve Sep 14, 2024
1cc7d81
Stub out Scheduler in the Monero processor
kayabaNerve Sep 14, 2024
ae9835b
Tighten documentation on Block::number
kayabaNerve Sep 14, 2024
3bd8383
Change dummy payment ID behavior on 2-output, no change
kayabaNerve Sep 14, 2024
b537c29
Monero Planner
kayabaNerve Sep 14, 2024
01679d3
Smart Contract Scheduler
kayabaNerve Sep 14, 2024
da3c758
Outline the Ethereum processor
kayabaNerve Sep 14, 2024
9752018
Don't use a different address for DAI in test
kayabaNerve Sep 14, 2024
7b63bff
Move ethereum-serai under the processor
kayabaNerve Sep 14, 2024
31659da
Add crate for the Ethereum contracts
kayabaNerve Sep 15, 2024
0c026f7
Add dedicated crate for building Solidity contracts
kayabaNerve Sep 15, 2024
a0702e6
Dedicated crate for the Schnorr contract
kayabaNerve Sep 15, 2024
c42137d
Remove publish = false
kayabaNerve Sep 15, 2024
11edc90
rust-toolchain 1.81
kayabaNerve Sep 15, 2024
5d2164f
OUT_DIR > artifacts
kayabaNerve Sep 15, 2024
cb5755f
Add tests for the premise of the Schnorr contract to the Schnorr crate
kayabaNerve Sep 15, 2024
b70c91e
Remove the Sandbox contract
kayabaNerve Sep 15, 2024
ce18294
Remove artifacts for serai-processor-ethereum-contracts
kayabaNerve Sep 15, 2024
002cf1b
Remove OutInstruction's data field
kayabaNerve Sep 15, 2024
ae49e8e
Break Ethereum Deployer into crate
kayabaNerve Sep 15, 2024
b162125
forge fmt
kayabaNerve Sep 17, 2024
8c79b78
Smash ERC20 into its own library
kayabaNerve Sep 17, 2024
3899db8
Smash out the router library
kayabaNerve Sep 17, 2024
d74e6e7
Hash the message before the challenge function in the Schnorr contract
kayabaNerve Sep 17, 2024
6057a9d
Don't have the Deployer store the deployment block
kayabaNerve Sep 17, 2024
6692679
Trim dead code
kayabaNerve Sep 17, 2024
b1b7b6d
Add calls to get the messages to sign for the router
kayabaNerve Sep 17, 2024
3f78af1
Don't have the router drop transactions which may have top-level tran…
kayabaNerve Sep 17, 2024
62da845
Ethereum SignableTransaction, Eventuality
kayabaNerve Sep 18, 2024
ffd291b
Remove ethereum-serai/serai-processor-ethereum-contracts
kayabaNerve Sep 18, 2024
4f5773c
Add note on the returned top-level transfers being unordered
kayabaNerve Sep 18, 2024
ece0b03
Move the Ethereum Action machine to its own file
kayabaNerve Sep 18, 2024
9175007
Have the TransactionPublisher build a TxLegacy from Transaction
kayabaNerve Sep 18, 2024
6c0f98a
cargo fmt, move ScannerFeed from String to the RPC error
kayabaNerve Sep 18, 2024
5f3da02
Explicitly add an unspendable script path to the processor's generate…
kayabaNerve Sep 19, 2024
4310101
Finish the Ethereum TX publishing code
kayabaNerve Sep 19, 2024
8c88b53
Pass the latest active key to the Block's scan function
kayabaNerve Sep 19, 2024
bb84b81
Call tidy_keys upon queue_key
kayabaNerve Sep 19, 2024
bf06384
Add hooks to the main loop
kayabaNerve Sep 19, 2024
40f5c35
Finish Ethereum ScannerFeed
kayabaNerve Sep 19, 2024
155f80e
Remove unused code in the Ethereum processor
kayabaNerve Sep 19, 2024
f837cc9
Read NetworkId from ScannerFeed trait, not env
kayabaNerve Sep 19, 2024
b85df85
Merge BlockWithAcknowledgedBatch and BatchWithoutAcknowledgeBatch
kayabaNerve Sep 19, 2024
3385c26
Mark files in TODO/ with "TODO" to ensure it pops up on search
kayabaNerve Sep 19, 2024
dd9f6c0
Update to the latest bitcoin-serai
kayabaNerve Sep 20, 2024
1cb82d7
Correct forge fmt config
kayabaNerve Sep 20, 2024
30e3a0c
Update the Router smart contract to pay fees to the caller
kayabaNerve Sep 20, 2024
d2d4158
Have the Ethereum scheduler create Batches as necessary
kayabaNerve Sep 20, 2024
05e007f
Remove accidentally included bitcoin feature from processor-bin
kayabaNerve Sep 20, 2024
c583a00
Set a fixed fee transferred to the caller for publication
kayabaNerve Sep 20, 2024
af9e73f
Add dummy fee values to the scheduler
kayabaNerve Sep 20, 2024
7fd239b
Report a Change Output with every Eventuality to ensure we don't fall…
kayabaNerve Sep 20, 2024
3aedbd1
Transfer ETH with CREATE, not prior to CREATE
kayabaNerve Sep 20, 2024
b6a408d
Don't have the ERC20 collapse the top-level transfer ID to the transa…
kayabaNerve Sep 20, 2024
5f625eb
Have the Router track its deployment block
kayabaNerve Sep 20, 2024
fe1342d
Don't track deployment block in the Router
kayabaNerve Sep 20, 2024
5a2a278
Misc comments
kayabaNerve Sep 20, 2024
862ab5c
Move old processor/src directory to processor/TODO
kayabaNerve Sep 20, 2024
d40a772
machete, drain > mem::swap for clarity reasons
kayabaNerve Sep 20, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/actions/bitcoin/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,4 +37,4 @@ runs:

- name: Bitcoin Regtest Daemon
shell: bash
run: PATH=$PATH:/usr/bin ./orchestration/dev/networks/bitcoin/run.sh -daemon
run: PATH=$PATH:/usr/bin ./orchestration/dev/networks/bitcoin/run.sh -txindex -daemon
4 changes: 2 additions & 2 deletions .github/actions/build-dependencies/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -42,8 +42,8 @@ runs:
shell: bash
run: |
cargo install svm-rs
svm install 0.8.25
svm use 0.8.25
svm install 0.8.26
svm use 0.8.26

# - name: Cache Rust
# uses: Swatinem/rust-cache@a95ba195448af2da9b00fb742d14ffaaf3c21f43
9 changes: 9 additions & 0 deletions .github/workflows/lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,6 +73,15 @@ jobs:
- name: Run rustfmt
run: cargo +${{ steps.nightly.outputs.version }} fmt -- --check

- name: Install foundry
uses: foundry-rs/foundry-toolchain@8f1998e9878d786675189ef566a2e4bf24869773
with:
version: nightly-41d4e5437107f6f42c7711123890147bc736a609
cache: false

- name: Run forge fmt
run: FOUNDRY_FMT_SORT_INPUTS=false FOUNDRY_FMT_LINE_LENGTH=100 FOUNDRY_FMT_TAB_WIDTH=2 FOUNDRY_FMT_BRACKET_SPACING=true FOUNDRY_FMT_INT_TYPES=preserve forge fmt --check $(find . -iname "*.sol")

machete:
runs-on: ubuntu-latest
steps:
Expand Down
3 changes: 2 additions & 1 deletion .github/workflows/networks-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,8 +30,9 @@ jobs:
run: |
GITHUB_CI=true RUST_BACKTRACE=1 cargo test --all-features \
-p bitcoin-serai \
-p build-solidity-contracts \
-p ethereum-schnorr-contract \
-p alloy-simple-request-transport \
-p ethereum-serai \
-p serai-ethereum-relayer \
-p monero-io \
-p monero-generators \
Expand Down
20 changes: 19 additions & 1 deletion .github/workflows/tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,25 @@ jobs:
GITHUB_CI=true RUST_BACKTRACE=1 cargo test --all-features \
-p serai-message-queue \
-p serai-processor-messages \
-p serai-processor \
-p serai-processor-key-gen \
-p serai-processor-view-keys \
-p serai-processor-frost-attempt-manager \
-p serai-processor-primitives \
-p serai-processor-scanner \
-p serai-processor-scheduler-primitives \
-p serai-processor-utxo-scheduler-primitives \
-p serai-processor-utxo-scheduler \
-p serai-processor-transaction-chaining-scheduler \
-p serai-processor-smart-contract-scheduler \
-p serai-processor-signers \
-p serai-processor-bin \
-p serai-bitcoin-processor \
-p serai-processor-ethereum-primitives \
-p serai-processor-ethereum-deployer \
-p serai-processor-ethereum-router \
-p serai-processor-ethereum-erc20 \
-p serai-ethereum-processor \
-p serai-monero-processor \
-p tendermint-machine \
-p tributary-chain \
-p serai-coordinator \
Expand Down
Loading
Loading