Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cadence Account Storage Map Migration #6761

Open
wants to merge 22 commits into
base: auto-update-onflow-cadence-v1.3.0
Choose a base branch
from

Conversation

turbolent
Copy link
Member

@turbolent turbolent commented Nov 26, 2024

Depends on #6779

Start work on the migration logic for onflow/cadence#3584

  • Add a new system contract AccountV2Migration, which is deployed to the service account during bootstrapping. Its Admin resource is able to trigger the migration for the next batch of accounts to be migrated, by calling the injected scheduleAccountV2Migration function
  • Expose a new function fun scheduleAccountV2Migration(addressStartIndex: UInt64, count: UInt64): Bool to the new contract AccountV2Migration in the service account. It generates the addresses in the given range, and schedules the migration to the new account storage format V2. It essentially just calls the internal Go function runtime.Storage.ScheduleV2Migration() provided by Cadence.

@turbolent turbolent requested review from fxamacker and a team November 26, 2024 00:29
@turbolent turbolent self-assigned this Nov 26, 2024
@fxamacker
Copy link
Member

@turbolent Thanks for working on this part of the integration!

Do we want to create a feature branch and target this PR to the feature branch?

@turbolent
Copy link
Member Author

@onflow/flow-cadence-execution is there a way to call accountV2Migration.DeclareScheduleAccountV2MigrationFunction during bootstrapping?

@codecov-commenter
Copy link

codecov-commenter commented Nov 27, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 41.50%. Comparing base (a164070) to head (0a393ef).
Report is 80 commits behind head on master.

Additional details and impacted files
@@             Coverage Diff             @@
##           master    #6761       +/-   ##
===========================================
+ Coverage   39.71%   41.50%    +1.78%     
===========================================
  Files        1332      111     -1221     
  Lines      123050    12400   -110650     
===========================================
- Hits        48875     5146    -43729     
+ Misses      69986     6868    -63118     
+ Partials     4189      386     -3803     
Flag Coverage Δ
unittests 41.50% <ø> (+1.78%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@janezpodhostnik
Copy link
Contributor

@turbolent during bootstrapping you have access to the function invokeMetaTransaction to that function you have to pass an fvm context.

When you want to run DeclareScheduleAccountV2MigrationFunction you can create a child context NewContextFromParent to which you add WithReusableCadenceRuntimePool and add the new cadence runtime pool.

Let me know if that helps.

@turbolent
Copy link
Member Author

@janezpodhostnik Thanks! It turned out that setting up the functions in the runtime won't work, because an environments chain ID might change (default config chain is MN, but then actual chain is e.g. Emulator). I refactored the code to be similar to the EVM setup, i.e. declaring the functions in the transaction invoker and in the script invoker.

@turbolent turbolent changed the base branch from master to auto-update-onflow-cadence-v1.3.0 December 4, 2024 00:16
@turbolent
Copy link
Member Author

@fxamacker I fixed a few things and added a test in 8b597cb which demonstrates that the migration of the service account works. Next step is probably to write an integration test which does this using the system chunk transaction, but I'm very new to this. Maybe someone from @onflow/flow-cadence-execution can take this on or at least point me in the right direction?

@fxamacker
Copy link
Member

@turbolent Thanks for adding the test and other updates!

Would you like me to start reviewing this PR or should I wait until integration test is added and DRAFT status is removed? No rush, just checking before I begin something needed next week.

@turbolent
Copy link
Member Author

@fxamacker Feel free to already start reviewing it. We'll probably want to have the integration tests before we merge, but feedback is already appreciated, thanks!

Copy link
Member

@fxamacker fxamacker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! 👍 I mostly reviewed from Go programming perspective, and left one minor comment.

fvm/accountV2Migration/AccountV2Migration.cdc Show resolved Hide resolved
@turbolent
Copy link
Member Author

As discussed in the last Execution Team meeting, I've added another Migration system account contract, and changed the system chunk transaction to call into it instead. The Migration contract then calls into the AccountV2Migration.

We'll have to deploy this PR in an HCU, given the system chunk transaction change, but with this change, we can repurpose the Migration logic in the future for other "on-the-fly" migrations, potentially saving an HCU.

@turbolent turbolent marked this pull request as ready for review December 12, 2024 23:27
@turbolent turbolent requested a review from a team December 12, 2024 23:27
Copy link
Contributor

@janezpodhostnik janezpodhostnik left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I imagined that scheduleAccountV2Migration would be on the Migration contract and would just be called scheduleAccountMigration which would then dispatch this to the V2 contract.

I also though we could avoid using the V2 contract directly anywhere in the FVM (excluding tests of course).

}
let evmHeartbeat = serviceAccount.storage
.borrow<&EVM.Heartbeat>(from: /storage/EVMHeartbeat)
evmHeartbeat?.heartbeat()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I spent way to long looking for a '. operator in the cadence language docs only to find out there is something wrong with the fonts:
image

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is this screenshot from? GitHub? VS Code? It's probably not a problem with the PR though?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is github on firefox. No not an issue in the PR. Just some sort of font issue somewhere.

Comment on lines +50 to +57
self.adminStoragePath = /storage/accountV2MigrationAdmin
self.nextAddressStartIndex = 1
self.batchSize = 0

self.account.storage.save(
<-create Admin(),
to: self.adminStoragePath
)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is really small, but the batching logic could probably live in the Migration contract instead of in the V2 one

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See #6761 (comment): The idea is that all logic related to the particular migration, the "account v2 migration", lives in AccountV2Migration, and Migration only delegates.

If we add the logic of the "account v2 migration" to the Migration contract, we won't be able to reuse it for other purposes, as the contract update rules e.g. don't allow removing enums, etc.

@turbolent
Copy link
Member Author

@janezpodhostnik

I imagined that scheduleAccountV2Migration would be on the Migration contract and would just be called scheduleAccountMigration which would then dispatch this to the V2 contract.

Sorry, I'm not quite following what's being suggested.

scheduleAccountV2Migration is the internal function that gets injected to allow migrating an account. Given that it is specific to the "account V2 format" migration, shouldn't it be only available in the contract for it, AccountV2Migration?

The idea for the Migration contract is that it is just just delegates/proxies calls from the system chunk transaction to whatever migration code (if any) is currently in the network. Once the "account V2 format" migration is complete, we can just update the contract to no longer call into AccountV2Migration through a normal contract upgrade.

If another migration needs to be added, we can reuse the Migration contract and trigger it.

Why should scheduleAccountV2Migration be available in Migration?

I also though we could avoid using the V2 contract directly anywhere in the FVM (excluding tests of course).

Agreed, we should not use it directly, only through the Migration contract. Maybe I missed a spot?

Happy to have another sync to discuss this further

@janezpodhostnik
Copy link
Contributor

Agreed, we should not use it directly, only through the Migration contract. Maybe I missed a spot?

The thing that threw me off was the reference to accountV2Migration.DeclareFunctions in the transaction invoker.

Would it be possible to have scheduleAccountMigration on the Migration contract that then gets forwarded to scheduleAccountV2Migration?

Yes maybe lets just quickly sync.

@turbolent
Copy link
Member Author

@janezpodhostnik re: moving bootstrapping of the AccountV2Migration out of the bootstrapping procedure: From what I can see, we would also have to move the Migration contract out, because it currently depends on the AccountV2Migration contract, and then the system chunk transaction would not work.

I guess we could deploy an "no-op" Migration contract during bootstrapping, and then update it to call AccountV2Migration, once ``AccountV2Migration` got deployed?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants