Nara off-chain migration script fixes (#315)

* Add is short field to video entity (#301)

* add isShort field to video entity

* regenerate db migrations

* remove @joystream/metadata-protobuf patch from assets/patches

* fix lint issue

* Disable both in Appp and eail notifications for video posted events (#299)

* bump package version and update CHANGELOG (#302)

* bump package version and update CHANGELOG

* change release version

* Simple public homefeed query and mutation (#304)

* update graphql schema

* add partial index on 'video.include_in_home_feed' field

* update video view definition to only include public videos

* regenerate migrations

* add dumbPublicFeedVideos custom query

* add setPublicFeedVideos mutation

* fix lint issue

* add arg to skip video IDs

* revert: update video view definition to only include public videos

* add feat. to unset public feed videos

* address requested change

* bump package version and update CHANGELOG

* Update `nara` from `master` (#300)

* Adds mappings for `ChannelAssetsDeletedByModerator` & `VideoAssetsDeletedByModerator` events (#199)

* mark 'VideoDeletedByModerator' & 'ChannelDeletedByModerator' events deprecated

* Implements mappings for 'Content.VideoAssetsDeletedByModerator and 'Content.ChannelAssetsDeletedByModerator' runtime events

* remove unused import

* Nara/crt update (#244)

* feat: build orion

* feat: start generating schema

* fix: extra entities

* fixup!

* fix: continue implementing design specs

* fix: review and fix foreign key relationships

* fix: formatting

* fix: generation errors

* fix: add comment

* fix: relations

* fix: final review

* fixup!

* fix: add ending blocks

* fix: generate type & set typegen to ipv4

* fix: add support for event backward compatibility

* feat: start adding mappings

* fix: continue with mappnigs

* feat: init sale

* feat: patronage decreased to & fixed build

* feat: claim patronage event

* feat: tokens bought on amm

* feat: tokens sold on amm

* fix: add relation between sales and vesting schedules

* feat: add Tokens sold on sale vente

* feat: update upcoming sale

* feat: revenue share issued

* feat: member joined whitelist

* feat: amm deactivated

* feat: burned token

* feat: transfer policy changed to permissionless

* feat: sale finalized

* feat: finish mappings

* fix: review

* fix: remove cascade deletions

* fix: renaming & formatting

* fixup!

* fixup!

* fix: patched protobuf packages with token proto

* feat: update metadata and add event handler scheleton

* feat: token metadata

* feat: sale metadata

* fix: review comments

* fix: formatting

* fix: revenue

* Revert "fix: revenue"

This reverts commit 0821abe.

* fix: token status after sale

* fix: fixmes

* fix: formatting

* fix: funds accounting during sale

* fix: amount accounting

* fix: linter

* fix: review

* fix: review 2

* fix: review

* fix: linter

* feat: migration for new db scheam

* fix: update event versions

* fix: patch types with crt_release types

* fix: patch types

* fix: generate all events versions since mainnet

* fix: temp fix after event version generation

* fix: event versioning

* fix: add migration

* fix: mignations

* fix: solve channel not being added

* fix: add id to TokenChannel

* fix: non-nullable deleted field set

* fix: format

* feat: creator token init sale re enabling

* feat: re enable sale init code

* fix: update types

* fix: amm id

* fix: id computation for revenue share

* fix: amm id computation for token

* fix: issuer transfer accounting

* fix: amm tx id

* fix: destination accounting

* feat: minor fix on holder transfer processing

* fix: re-enable metadata

* fix: metadata parsing

* fix: post reword cleanup

* fix: format

* fix: silence ci checks

* fix: event version

* fix: address PR changes

I edited all the entity that have a composite index like TokenAccount so that they have
a synthetic ID and an optionally unique @Index

* fix: add hidden entities conditions

* fix: add extra fields to token in order to keep track of ongoing status

* fix: build errors

* fix: adapt mapping to new token fields

* fix: format

* feat: add trailer video entity

this is required so we can simply make trailer video hidden if video is hidden

* fix: linter

* chore: prettier

* fix: from PR review

* fix: vesting schedule schema & mappings

I have replaced the vesting schedule back to the original schema with:
- VestingSchedule: holding vesting schedule information such being amount agnostic
- VestedAccount: contains information regarded to a vested account, the goal is to mimic the
runtime logic

* fix: burning from vesting

* patch: metadata-protobuf package

* patch: metadata-protobuf package

* fix: generate migrations

* fix: purchase token on sale

* Update schema/token.graphql

Co-authored-by: Leszek Wiesner <[email protected]>

* Update schema/token.graphql

Co-authored-by: Leszek Wiesner <[email protected]>

* fix: address PR

* fix: hidden entities

* fix: migration ok

* feat: add extra check for migrations

* fix: docker network

* fix: format

* fix: remove unrequired constraint

* fix: 🐛 post rebase fixes

* feat: 🎨 add metadata processing for issue token

* feat(crt-v1): ✨ chain metadata for v 2003

* fix(crt-v1): 🚑 comment out view element for orion playgroud

* fix(crt-v1): 🎨 add playground config variable to .env

* feat: ✅ add tests

* fix(crt-v1): 📦 packages and patches

* fix(crt-v1): ✅ update entity id used and other minor fixes

* fix(crt-v1): ✅ update entity id used and other minor fixes

* test(crt-v1): 🐛 misc fixes to have tests working

* test(crt-v1): 🐛 misc fixes to have tests working

* fix(crt-v1): 🐛 metadata and trailer video

* feat(crt-v1): 🎨 update types

* fix(crt-v1): ✨ Add correct Ratio denomination (Permill)

* update with master

* fix: 🐛 metadata not being set

* fix: 🐛 parameters order

* test: 🧪 fixing integration tests

* test(crt-v1): 🧪 fix integration tests

* feat(crt-v1): ✨ last price for token and recovered field for rev share part

* feat: ✨ add resolver for dividend amount

* feat(crt-v1): ✨ start adding channel fields for trackingtotal revenue

* feat(crt-v1): ✨ add utils for royalty computation

* feat(crt-v1): ✨ cumulative revenue on channel

* feat(crt-v1): ✨ add resolver for transferrable amount

* fix(crt-v1): ✨ add `acquiredAt` to pinpoint latest vesting schedule for account

* Token metadata processing update

* Prettier

* chore(crt-v1): ⚡ dbgen

* fix(crt-v1): 🧪 fix integration tests

* fix(crt-v1): 🐛 missing fields in token sale vesting source

* test(crt-v1): 🧪 test for transferrable balance amount

* fix(crt-v1): 🐛 transferrable amount

* test: 🧪 update tests after resolver fix

* fix: 🐛 error on vesting schedules array

* fix: 🎨 CI fixes

* docs: update gitignore

* fix: 🚨 prettier

* build: 📌 chai depnedencies

---------

Co-authored-by: Leszek Wiesner <[email protected]>
Co-authored-by: WRadoslaw <[email protected]>

* Clear benefits even if not passed (#282)

* 🤑 Fix revenue share dividend estimation (#297)

* Fix on revenue share dividend estimation

* Fix type on result

* 🛕 Historical revenue share participants (#286)

* New field for revenue share

* Set potential revenue share particitants at the time of start

* fix: .gitignore not working

* fix lint issues

* re-generate db migrations

* commit register.html.mst file

* fix: notifications integration test

---------

Co-authored-by: Ignazio Bovo <[email protected]>
Co-authored-by: Leszek Wiesner <[email protected]>
Co-authored-by: WRadoslaw <[email protected]>
Co-authored-by: WRadoslaw <[email protected]>

* Revert "Update `nara` from `master` (#300)" (#306)

This reverts commit 887427c.

* generate auth api docs and types

* add is short derived field to video entity (#310)

* add is shirt derived field to video entity

* add indices on is short fields

* fix: video language detection fix (#309)

* fix: video language detection fix

* address requested changes

* fix: predictVideoLanguage function

* fix: include max 1 video per channel in homepage videos (#313)

* fix: include max 1 video per channel in homepage videos

* update setOrionLanguage Migration script

* format updateVideoRelevanceValue SQL query

* fix: use UTC midnight epoch instead of current epoch to calculate video relevance score

* bump package version and update CHANGELOG

* fix: lint bug

* remove NextEntityIdManager migration script

* [offchainState] add v4.0.0 (CRT release) migrations

* [offchainState] remove ORDER BY clause from UPDATE statements

* add migration for NextEntityId

* bump package version and update CHANGELOG

---------

Co-authored-by: Ignazio Bovo <[email protected]>
Co-authored-by: Leszek Wiesner <[email protected]>
Co-authored-by: WRadoslaw <[email protected]>
Co-authored-by: WRadoslaw <[email protected]>

CHANGELOG.md

@@ -1,3 +1,7 @@
+# 4.0.0
+
+This is Creator Tokens (CRT) release. It introduces the CRT mappings, custom resolvers and mutations.
+
 # 3.7.0
 
 ## Schema changes

package.json

@@ -1,6 +1,6 @@
 {
   "name": "orion",
-  "version": "3.7.0",
+  "version": "4.0.0",
   "engines": {
     "node": ">=16"
   },

src/mappings/utils.ts

@@ -1,20 +1,18 @@
-import { metaToObject } from '@joystream/metadata-protobuf/utils'
 import { AnyMetadataClass, DecodedMetadataObject } from '@joystream/metadata-protobuf/types'
-import { Logger } from '../logger'
-import { SubstrateBlock } from '@subsquid/substrate-processor'
-import { Event, MetaprotocolTransactionResultFailed, NftActivity, NftHistoryEntry } from '../model'
-import { encodeAddress } from '@polkadot/util-crypto'
-import { EntityManagerOverlay } from '../utils/overlay'
-import { Bytes } from '@polkadot/types/primitive'
+import { metaToObject } from '@joystream/metadata-protobuf/utils'
 import { createType } from '@joystream/types'
+import { Bytes } from '@polkadot/types/primitive'
 import { u8aToHex } from '@polkadot/util'
+import { encodeAddress } from '@polkadot/util-crypto'
+import { SubstrateBlock } from '@subsquid/substrate-processor'
+import { Logger } from '../logger'
+import { Event, MetaprotocolTransactionResultFailed, NftActivity, NftHistoryEntry } from '../model'
 import { CommentCountersManager } from '../utils/CommentsCountersManager'
 import { VideoRelevanceManager } from '../utils/VideoRelevanceManager'
-import { NextEntityIdManager } from '../utils/NextEntityIdManager'
+import { EntityManagerOverlay } from '../utils/overlay'
 
 export const commentCountersManager = new CommentCountersManager()
 export const videoRelevanceManager = new VideoRelevanceManager()
-export const migrateCounters = new NextEntityIdManager()
 // eslint-disable-next-line no-void
 void videoRelevanceManager.init({
   fullUpdateLoopTime: 1000 * 60 * 60 * 12, // 12 hrs

src/processor.ts

@@ -113,7 +113,7 @@ import {
   processUpcomingTokenSaleUpdatedEvent,
   processUserParticipatedInSplitEvent,
 } from './mappings/token'
-import { commentCountersManager, migrateCounters, videoRelevanceManager } from './mappings/utils'
+import { commentCountersManager, videoRelevanceManager } from './mappings/utils'
 import { Event } from './types/support'
 import { EventHandler, EventInstance, EventNames, eventConstructors } from './utils/events'
 import { assertAssignable } from './utils/misc'
@@ -415,7 +415,6 @@ processor.run(new TypeormDatabase({ isolationLevel: 'READ COMMITTED' }), async (
       await commentCountersManager.updateVideoCommentsCounters(em, true)
       await commentCountersManager.updateParentRepliesCounters(em, true)
       await videoRelevanceManager.updateVideoRelevanceValue(em, true)
-      await migrateCounters.migrateCounters(overlay)
       ctx.log.info(`Offchain state successfully imported!`)
     }
   }

src/utils/notification/helpers.ts

@@ -19,7 +19,7 @@ import { EntityManagerOverlay } from '../overlay'
 export const RUNTIME_NOTIFICATION_ID_TAG = 'RuntimeNotification'
 export const OFFCHAIN_NOTIFICATION_ID_TAG = 'OffchainNotification'
 
-function notificationPrefAllTrue(): NotificationPreference {
+export function notificationPrefAllTrue(): NotificationPreference {
   return new NotificationPreference({ inAppEnabled: true, emailEnabled: true })
 }
 

src/utils/offchainState.ts

@@ -5,8 +5,15 @@ import fs from 'fs'
 import path from 'path'
 import { EntityManager } from 'typeorm'
 import * as model from '../model'
+import {
+  AccountNotificationPreferences,
+  fromJsonDeliveryStatus,
+  fromJsonNotificationType,
+  fromJsonReadOrUnread,
+  fromJsonRecipientType,
+} from '../model'
 import { uniqueId } from './crypto'
-import { defaultNotificationPreferences } from './notification/helpers'
+import { defaultNotificationPreferences, notificationPrefAllTrue } from './notification/helpers'
 
 const DEFAULT_EXPORT_PATH = path.resolve(__dirname, '../../db/export/export.json')
 
@@ -112,11 +119,51 @@ function migrateExportDataToV320(data: ExportedData): ExportedData {
   return data
 }
 
+export function setCrtNotificationPreferences(
+  notificationPreferencesObj: any
+): AccountNotificationPreferences {
+  notificationPreferencesObj.crtIssued = notificationPrefAllTrue()
+  notificationPreferencesObj.crtMarketStarted = notificationPrefAllTrue()
+  notificationPreferencesObj.crtMarketMint = notificationPrefAllTrue()
+  notificationPreferencesObj.crtMarketBurn = notificationPrefAllTrue()
+  notificationPreferencesObj.crtSaleStarted = notificationPrefAllTrue()
+  notificationPreferencesObj.crtSaleMint = notificationPrefAllTrue()
+  notificationPreferencesObj.crtRevenueShareStarted = notificationPrefAllTrue()
+  notificationPreferencesObj.crtRevenueSharePlanned = notificationPrefAllTrue()
+  notificationPreferencesObj.crtRevenueShareEnded = notificationPrefAllTrue()
+  const notificationPreferences = new AccountNotificationPreferences(
+    undefined,
+    notificationPreferencesObj
+  )
+  return notificationPreferences
+}
+
+function migrateExportDataToV400(data: ExportedData): ExportedData {
+  data.Account?.values.forEach((account) => {
+    // account will find himself with all CRT notification pref. enabled by default
+    account.notificationPreferences = setCrtNotificationPreferences(
+      account.notificationPreferences as AccountNotificationPreferences
+    )
+  })
+
+  data.Notification?.values.forEach((notification) => {
+    notification.notificationType = fromJsonNotificationType(notification.notificationType)
+    notification.status = fromJsonReadOrUnread(notification.status)
+    notification.recipient = fromJsonRecipientType(notification.recipient)
+  })
+
+  data.EmailDeliveryAttempt?.values.forEach((emailDeliveryAttempt) => {
+    emailDeliveryAttempt.status = fromJsonDeliveryStatus(emailDeliveryAttempt.status)
+  })
+  return data
+}
+
 export class OffchainState {
   private logger = createLogger('offchainState')
   private _isImported = false
 
   private migrations: Migrations = {
+    '4.0.0': migrateExportDataToV400,
     '3.2.0': migrateExportDataToV320,
     '3.0.0': migrateExportDataToV300,
   }
@@ -252,7 +299,6 @@ export class OffchainState {
                 })
                 .join(', ')}
             ) AS "data"
-            ORDER BY "id"
             WHERE "${meta.tableName}"."id" = "data"."id"`,
             fieldNames.map((fieldName) => batch.map((v) => v[fieldName]))
           )
@@ -269,7 +315,23 @@ export class OffchainState {
               values.length
             } entities left)...`
           )
-          await em.getRepository(entityName).insert(batch)
+
+          // UPSERT operation specifically for NextEntityId
+          if (entityName === 'NextEntityId') {
+            for (const entity of batch) {
+              await em.query(
+                `
+                INSERT INTO "next_entity_id" ("entity_name", "next_id")
+                VALUES ($1, $2)
+                ON CONFLICT (entity_name)
+                DO UPDATE SET next_id = EXCLUDED.next_id;
+              `,
+                [entity.entityName, entity.nextId]
+              )
+            }
+          } else {
+            await em.getRepository(entityName).insert(batch)
+          }
         }
       }
       this.logger.info(

src/utils/overlay.ts

@@ -361,11 +361,6 @@ export class EntityManagerOverlay {
     return this.em
   }
 
-  // reason: during migration the overlay would write to the database the old nextId
-  public invalidateRepository(entityName: string) {
-    this.repositories.delete(entityName)
-  }
-
   // Create an entity repository overlay or load already cached one
   public getRepository<E extends AnyEntity>(entityClass: Constructor<E>): RepositoryOverlay<E> {
     const loadedRepository = this.repositories.get(entityClass.name)