Schema Migrations

[jamesbayly]

Background

If we have an existing SubQuery Project hosted and we want to amend it in the following scenario

• Scenario A: Add a new event we want to index without any changes to the Schema
• Scenario B: Add a new entity we want to record to the database
• Scenario C: Add a new property to an existing entity

Scenario A is now supported via #1214 - you can reindex from a certain block height

Scenario B and C are not yet supported because it means a schema and db migration must occur.

Proposal

Allow users to provide a versioned list of GraphQl schemas (schema.graphql) files in the project manifest

schema:
  file: ./schema.graphql # links to the latest version
  startBlock: 500000 # block at which the latest migration occurs
  versions:
     - file: ./schema-v1.graphql
       startBlock: 0 
     - file: ./schema-v2.graphql
       startBlock: 12321332

Use Sequelize to migrate these schemas (https://sequelize.org/docs/v6/other-topics/migrations/)

• Potentially use the subql cli to allow a user to create migrations files, then reference these migrations file in the versions array above. Additionally this means users can fine tune the migrations files
• How will this affect historical indexing table?

Questions

Question 1: Is it possible to amend like in all scenarios an existing SubQuery Project hosted on SubQuery Network using code (no manual steps) and preserving the previous indexing/database, by indexing and adding to the existing database from the last block we updated the code, instead of re-indexing from scratch for all entities?

E.g. Lets assume we want to query about transfers of MOVR for account 0x123 on Moonriver
a) We can query the Moonriver dictionary
b) We can write our own SubQuery Project that indexes only the relevant event

Question 2: What are the advantages and disadvantages between A and B/C?

For example (some of these are obvious to us)
In b) we can add our own code and create our own entities and form the database as we want it
In a) if in the future we want to query staking of MOVR then this is readily available in the dictionary. But for b) we have to amend our existing project to index another event

Is it the case that the fees for a Dictionary query will be more expensive than an Individual SubQuery Project on the SubQuery Network? - Unsure at this stage, Dictionary will likely be subsidised somewhat
Is it the case that the query in our individual SubQuery Project will be significantly faster than the equivalent query on SubQuery Moonriver Dictionary? - Yes

[Entity54]

Hi,
Many thanks for the responses, some additional thoughts if we may
A> With regards to #1214
Please correct me if I am wrong but I believe the actual code of a SubQuery Project can be stored on IPFS. If this is true adding HTTP api to run these commands could be triggered via CID <=> IPFS pattern
e.g. the admin of the specific SubQuery Project that wants to amend it, would have access to the CID of the IPFS file and the privilege to amend it.

B> Assume for a moment a scenario where SubQuery is a parachain
i) There is a SubQuery Pallet that among other

1. Keeps track of the CID for each SubQuery Project that is indexed along with the admin account that is allowed certain privilleges
2. Exposes extrinsics to;
   a) amend/append events/ extrinsics to be indexed
   b) amend/append the entities recorded to the database
3. Includes a Batch Extrinsic to group all the above

ii) xTokens Pallet with tokens similar to Moonbeam xc20 tokens (i.e. tokens that have both a Substrate representation and ERC20 compatibility)
The actual genesis of these xc20 tokens can happen in Moonbeam network but SubQuery Parachain would have to register these in the Assets and establish the relevant HRMP channels

iii) Includes an Open Democracy Pallet very close to the current Substrate Democracy Pallet with the added functionality that each team that is admin of a SubQuery Project, if it wants, it can register its own Token (part of AssetsRegistry/xTokens) and use this in the Democracy Pallet.

Play Scenario
A> Team ABC has token xcABC and is admin of a SubQuery Project with code stored in IPFS with CID 123
B> Team ABC (or anyone really) wants to add to existing project, 1 new event to index and 2 new entities for the database
C> Team ABC registers xcABC to SubQuery Parachain AssetsRegistry pallet.
D> Team ABC (or anyone really) creates a proposal on Democracy Pallet with the following specs wrapped in a batch call
I) add 1 event
ii) add 2 entities (actual code syntax can mimic the UI of creating an XCM transfer)
iii) input CID of Project
iv) state token to be used for Voting xcABC
E> Users of the Team ABC application can use XCM transfers and the above pallets to vote on the amendments
F> Enactment period elapses and SubQuery Network mines the batched Extrinsic transaction