Orion v3 release: user accounts (#160)

* User accounts (#122)

* Initial commit

* Initial orion draft

* Hide private data

* Fix extendedChannels query

* docker-compose db config adjustments

* Fix ExtendedBasicChannelFields

* Add `video` view

* Storage, Membership and ChannelCreated mappings

* Channel events processing

* Video mappings + some generic fixes

* Mappings: Comments, reactions + some generic fixes

* Cleanup

* NFT mappings, some post-testing fixes

* Video views mutation

* Channel follows + video views improvements

* Channel/video reports

* Category filtering: Part 1

* State subscription

* Category filtering: Part 2 + some refactorization

* Category filtering: Part 3 - add missing views and operator auth

* mostViewedVideosConnection custom query + generic custom queries fixes

* Kill switch

* mostRecentChannels query

* channelNftCollectors query

* extendedVideoCategories query

* Fix AND-OR queries and "eq: null"

* Docker setup improvements

* Video hero

* Category featured videos

* Entity caching (overlay) refactorization

* Prettier

* Documentation, setup improvements, cleanup

* FIX: Save NextEntityIds

* Excludable content

* - Allow limiting the total number of cached entities
- Deployment manifest fixes
- Custom ARCHIVE_GATEWAY_URL instead of HOST/PORT
- Updated dependencies
- `processAssets` bug fix

* setVideoViewPerIpTimeLimit operator mutation

* Ephesus mappings WIP

* Fix conflicts with local Joystream docker setup

* Metadata-protobuf patch location fix

* Ephesus Atlas queries

* Update CHANGELOG

* Linting/formatting fixes, dead code removal

* Apps metaprotocol mappings

* Open auction topBid fix

* Fix metadata-protobuf patch

* Add signAppActionCommitment endpoint

* Tests: Basic data comparison script + bug fixes

* Remove channel-owned apps

* Remove Ephesus scope

* Apps as member-owned initial rework

* Fixes after compareState vs mainnet

* Remove lead-owned apps, fix typeorm-codegen patch

* AppAction support (Joystream/joystream#4631)

* Update CHANGELOG based on tests

* Allow controlling "is_censored" content exclusion via env

* buildExtendedChannelsQuery fix (activeVideosCount_gt: 0)

* Benchmarking script

* Benchmarking results

* Event queries optimalizations, fix gitignored v1 and v2 schemas

* Adjust autovacuum_analyze_scale_factor

* Fix entity cache cleanup

* Add index on event's `inExtrinsic` field

* Benchmarking round 2

* Fix req.ip reverse proxy issue, allow conditionally displaying excluded content in results

* Operator queries

* Update docs

* Prettier format

* Add CI checks

* Remove issue template, fix "workflows" location

* Disallow app deletion

* Update `generateAppActionCommitment` from `@joystream/js`

* Fix channel follows, video views and reports after operator queries

* Fix #82

* Comment notifications: Don't notify the author about their own comment

* Introduce a mechanism to preserve views, follows, reports and config data when updating the processor

* mostViewedVideosConnection missing params fix

* Fix assets encoding (AppAction)

* Assets url resolving

* Remove accidently commited file

* Fix Aquarium deployment

* Update CHANGELOG

* Improve caching speed and add more logs

* Latest apps-related updated (generateAppActionCommitment, mappings)

* Process member banning/unbanning

* Fix: Remove banned members when channel is removed

* Update migration

* State queries: Re-enable apps

* OpenAPI draft docs

* Auth API: Part 1 - Anonymous Auth

* Views: Add missing chanel/video related entities

* Excluding/restoring comments: Update:
- parentComment.repliesCount,
- parentComment.reactionsAndRepliesCount,
- video.commentsCount

* Operator queries: 401 Unauthorized, remove `x-display-hidden-entities` (operator just sees all hidden entities by default)

* Update changelog

* Simplify processor state subscription

* Prevent the same comment from being "excluded"/"restored" multiple times

* Small adjustments to docs and compareState script

* Orion v1 migration scripts

* New logic for comment exclusion

* Fix: Reference to `yarn` in package.json

* Increase interval of ProcessorStateRetriever from 100 ms to 1000 ms

* Auth API: Part 2 - Session extension and registration

* Auth API: Part 3 - Basic e-mail confirmation

* Auth API: Part 4 - Login and logout

* Auth API: Part 5 - Connect / disconnect Joystream account(s)

* Auth API: Part 6 - E-mail confirmation token re-send

* Auth API: Schema adjustments for signature sign-in

* Auth API: Encryption artifacts and related tests

* Auth API: Cookie authentication

* Auth API: Introduce rate limits and adjust proxy configuration

* Auth API: DROP ANY NOTION OF E-MAIL CONFIRMATION AND PASSWORD RESET

* Auth API: Max number of connected accounts and other data insertion limits

* Formatting

* Bring back e-mail confirmation features

* AuthAPI: OpenAPI playground, fixes, dev settings

* Auth API: Login - provide accountId in response

* Auth API: Fix GraphQL server cors config to allow testing

* Auth API: Update docs

* Auth API: Minor getSessionArtifacts response data fix

* Auth API: An attempt to fix CI tests

* Auth API: 2nd attempt to fix CI tests

* FIX: Set session cookie even if the session isn't new

* FIX: disconnect-account should not require signing

* Update developer guide + small package.json / Makefile fixes

* Fix OperatorOnly and add AccountOnly middleware

* Use `signatureVerify` instead of `sr25519Verify` to support external signer verification

* Add warning about `tests:auth-api` to the docs

* Fix link in README

* Auth API adjustmnets due to new schema, bug fixes, `accountData` query and docs updates

* Fix session artifacts unit test

* Fix docker build

* Fix generate-schema-file.sh

* Adjustments:
    - one-to-one rel between Gateway accs, Blockchain accs and Memberships,
    - bug fixes

* Operator guide + related developer guide adjustments

* Adjustments to facilitate Orion deployment, upgrading and maintenance

* Fix formatting, remove accidently commited files

* Connect follows, reports and views w/ users

* Use jsonl file as typegen input

* Remove usused patch

* Fix example env values in Operator guide

* Fix config values visibility issue

* 📝 Doc: update changelog for version 3.0.0 (#158)

* 📝 Doc: update changelog for version 3.0.0

* ✏️ Fix: add disable same site explainer

* 🩹 Improved changelog (#159)

* Update CHANGELOG.md

Remove docs directory structure

---------

Co-authored-by: Leszek Wiesner <[email protected]>

.dockerignore

@@ -2,5 +2,5 @@
 /node_modules
 /lib
 /*Versions.jsonl
-/db/export.json*
+/db/export
 /db/persisted

.env

@@ -1,35 +1,62 @@
-# LOCAL ENVIRONMENT
+# LOCAL DEV ENVIRONMENT
+
+ORION_ENV=development
+DEV_DISABLE_SAME_SITE=true
 
 # Db config
 DB_NAME=squid
 DB_PASS=squid
+DB_ADMIN_USER=admin
+DB_ADMIN_PASS=admin
 DB_PORT=23798
-# Processor service host
-PROCESSOR_HOST=localhost
+
 # Processor service prometheus port
 PROCESSOR_PROMETHEUS_PORT=3337
 # Graphql server port
 GQL_PORT=4350
+# Auth api port
+AUTH_API_PORT=4074
 
 # Archive gateway url
 ARCHIVE_GATEWAY_URL=${CUSTOM_ARCHIVE_GATEWAY_URL:-http://localhost:8888/graphql}
 
-# Default config values
+# ======================================================
+# ============== DEFAULT CONFIG VALUES =================
+# ======================================================
+APP_NAME=Gleev
 SUPPORT_NO_CATEGORY_VIDEOS=true
 SUPPORT_NEW_CATEGORIES=true
 KILL_SWITCH_ON=false
-VIDEO_VIEW_PER_IP_TIME_LIMIT=86400 # 86400 seconds = 24 hours
-VIDEO_RELEVANCE_VIEWS_TICK=50 # every 50 views video relevance score will be recalculated
-RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3]]" # [newness (negative number of days since created) weight, views weight, comments weight, rections weights, [joystream creation weight, YT creation weight]]
-
+# 10 seconds
+VIDEO_VIEW_PER_USER_TIME_LIMIT=10
 # Operator API secret
 OPERATOR_SECRET=this-is-not-so-secret-change-it
-# Processor configuration
+# every 50 views video relevance score will be recalculated
+VIDEO_RELEVANCE_VIEWS_TICK=50
+# [
+#   newness (negative number of days since created) weight,
+#   views weight,
+#   comments weight,
+#   rections weights,
+#   [joystream creation weight, YT creation weight]
+# ]
+RELEVANCE_WEIGHTS="[1, 0.03, 0.3, 0.5, [7,3]]"
 MAX_CACHED_ENTITIES=1000
 APP_PRIVATE_KEY=this-is-not-so-secret-change-it
+SESSION_EXPIRY_AFTER_INACTIVITY_MINUTES=60
+SESSION_MAX_DURATION_HOURS=720
+EMAIL_CONFIRMATION_ROUTE=http://localhost:4074/api/v1/confirm-email?token={token}
+EMAIL_CONFIRMATION_TOKEN_EXPIRY_TIME_HOURS=24
+EMAIL_CONFIRMATION_TOKEN_RATE_LIMIT=5
+ACCOUNT_OWNERSHIP_PROOF_EXPIRY_TIME_SECONDS=300 # 5 minutes
+COOKIE_SECRET=this-is-not-so-secret-change-it
+
+TRUST_PROXY=uniquelocal
 
-# Adjust accordingly with the number of trusted (used) reverse proxies!
-TRUSTED_REVERSE_PROXIES=1
+# Sendgrid API
+SENDGRID_API_KEY=
+SENDGRID_FROM_EMAIL=[email protected]
 
 # Debug settings
 SQD_DEBUG=api:*
+OPENAPI_PLAYGROUND=true

.eslintrc.js

@@ -28,6 +28,12 @@ module.exports = {
         ignoreVoid: true,
       },
     ],
+    '@typescript-eslint/no-unused-vars': [
+      'warn',
+      {
+        varsIgnorePattern: '^_',
+      },
+    ],
     '@typescript-eslint/no-misused-promises': 'error',
   },
   plugins: ['standard', '@typescript-eslint', 'prettier'],

.github/workflows/tests.yml

@@ -0,0 +1,22 @@
+name: Tests
+on: [push, pull_request]
+
+jobs:
+  auth-api:
+    name: Auth API tests
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        node-version: [16.x]
+      fail-fast: true
+    steps:
+      - uses: actions/checkout@v2
+      - name: Use Node.js ${{matrix.node-version}}
+        uses: actions/setup-node@v1
+        with:
+          node-version: ${{matrix.node-version}}
+      - name: Prepare workspace
+        run: make prepare
+      - name: Run tests
+        run: npm run tests:auth-api

.gitignore

@@ -10,4 +10,4 @@ src/model/generated
 /schema.graphql
 /db/persisted
 /scripts/orion-v1-migration/data
-/db/export.json*
+/db/export

.prettierignore

@@ -8,4 +8,4 @@ src/model/generated
 db/migrations/*.js
 schema.graphql
 /scripts/orion-v1-migration/data
-/db/export.json*
+/db/export

CHANGELOG.md

@@ -1,3 +1,156 @@
+# 3.0.0
+This is a major release that will contains several breaking changes due to the 
+introduction of the user account feature. Throught this release changelog the term
+"registered account","account", "gateway account" will be used interchangeably in order
+to denote a user that has registered its credential using the provided feature
+
+### Architecture
+The most prominent introduction is the new authentication api, which can be run as a docker service 
+
+#### Authentication Api
+A new authentication api in order to authenticate accounts has been introduced, based on the open-api specification
+The `docs/developer-guide/tutorials/authentication-api.md` has a detailed description about this, I will just list
+the routes provided by the api:
+
+Version 1.0.0 
+
+- Added new routes:
+  - `/anonymous-auth`: Authenticate as an anonymous user, either using an existing user identifier or creating a new one.
+  - `/login`: Login to a user's account by providing a message signed by the associated blockchain account.
+  - `/artifacts`: Get wallet seed encryption artifacts.
+  - `/session-artifacts`: Get and save wallet seed encryption artifacts for the current session.
+  - `/account`: Create a new Gateway account. Requires anonymous authentication to be performed first.
+  - `/confirm-email`: Confirm the account's email address provided during registration.
+  - `/request-email-confirmation-token`: Request a token to be sent to the account's email address for email confirmation.
+  - `/change-account`: Change the blockchain (Joystream) account associated with the Gateway account.
+  - `/logout`: Terminate the current session.
+
+- Implemented new methods:
+  - `POST /anonymous-auth`: Perform anonymous authentication.
+  - `POST /login`: Perform user login.
+  - `GET /artifacts`: Retrieve wallet seed encryption artifacts.
+  - `GET /session-artifacts`: Retrieve wallet seed encryption artifacts for the current session.
+  - `POST /session-artifacts`: Save wallet seed encryption artifacts for the current session on the server.
+  - `POST /account`: Create a new Gateway account.
+  - `POST /confirm-email`: Confirm the account's email address.
+  - `POST /request-email-confirmation-token`: Request a token for email confirmation.
+  - `POST /change-account`: Change the blockchain (Joystream) account associated with the Gateway account.
+  - `POST /logout`: Terminate the current session.
+
+- Deprecated routes/methods:
+  - None.
+
+- Added comprehensive documentation for easy integration and usage inside `src/auth-api/docs`
+- Added `src/auth-api/email` folder used for html template emails. Currently only registration email for a new gateway account is supported, 
+but more email type will be supported in future releases
+
+Note: For more detailed information about each route and method, please refer to the API documentation, inside `src/auth-api/docs`
+
+
+
+#### Config Variables changes
+- Orion archive `WS_SOURCE` default value has been changed to the public endpoint `wss://rpc.joystream.org:9944`
+(before was pointing to the local host deployment at port `9944`)
+- `ORION_ENV`: variables has been introduced to specify between `development` and `production` stages
+- `DEV_DISABLE_SAME_SITE`: disables the "SameSite" attribute for cookies is used to control how cookies are sent in cross-site requests, when `ORION_DEV` is set to `development`
+- `PROCESSOR_HOST` variable has been removed
+- `DB_ADMIN_USER` and `DB_ADMIN_PASS` in order to create the `admin` user that has access to the `admin` schema
+- `AUTH_API_PORT` for specifying port for the authentication api
+- `APP_NAME`: Gateway name, it will be used in the email sent to the account owners.
+- `VIDEO_VIEW_PER_IP_TIME_LIMIT` replaced by `VIDEO_VIEW_PER_USER_TIME_LIMIT`
+- `SESSION_EXPIRY_AFTER_INACTIVITY_MINUTES` how much a session should last for an inactive user
+- `SESSION_MAX_DURATION_HOURS` max session duration in hours
+- `EMAIL_CONFIRMATION_ROUTE` api route for email confirmation
+- `EMAIL_CONFIRMATION_TOKEN_EXPIRY_HOURS`: how many hours does a confirmation token lasts 
+- EMAIL_CONFIRMATION_TOKEN_RATE_LIMIT`: how many requests for a new e-mail confirmation token can be made within
+ `EMAIL_CONFIRMATION_TOKEN_EXPIRY_TIME_HOURS` for a given e-mail address
+- `ACCOUNT_OWNERSHIP_PROOF_EXPIRY_TIME_SECONDS`
+- `COOKIE_SECRET`: secret used to sign cookies, to make sure they come from Orion and have not been tampered with
+- `TRUSTED_REVERSE_PROXY` variable has been superseeded by `TRUST_PROXY`
+- `OPEN_API_PLAYGROUND` whether or not have a openapi playground on localhost
+- `SENDGRID_API_KEY` API key from sendgrid, used for sending email to the gateway account owners (for the purpose of
+email confirmation only at the moment)
+- `OPERATOR_SECRET`: string used as identifier for the root user. Despite not being a new addition to the release now it is 
+being stored in the database and it gives access to the hidden entities and `OperatorOnly` queries/mutations
+
+#### Makefile
+- A new rule for spinning up the auth api has been added as `make serve-auth-api`
+- A new `joystream.jsonl` has been added. This file contains the metadata necessary for generating correct events with respective 
+appropriate version numbers from the metadata via the `make typegen` command
+
+### Entities
+#### Changes
+- `VideoViewEvent.ip: String` replaced by `VideoViewEvent.user: User` 
+- `NftFeaturingRequest.ip: String` replaced by `NftFeaturingRequest.user: User` 
+- `ChannelFollow.ip: String` replaced by `ChannelFollow.user: User` 
+#### Additions
+The following entities have been introduced together with the new account management system, more information about
+them is provided in the developer guide
+- `User`: basic representation of a client App / Oriol user,  it can be either an anonymous user (have no related Account) or a gateway account owner.
+A User can be associated with activities such as viewing a video, or searching for specific content, 
+which can be later used to provide a personalized experience to the user once they create an account.
+- `EncryptionArtifacts`: SessionEncryptionArtifacts represents a set of encryption artifacts (cipherIv and cipherKey) 
+associated with a given session, allowing the Client app to more securely store
+Blockchain account's seed throughout the session 
+- `SessionEncryptionArtifacts`: represents a set of encryption artifacts (cipherIv and encryptedSeed) 
+which can be used by the Client app to decrypt the seed of a Blockchain account based on the account's
+login credentials (email and password)
+- `Session`: represents a specific activity period for a `User`
+- `Account`: represents a Gateway Account which can be accessed by the Gateway account owner by logging in
+- `Token`: represets a unique, securely random string generated by the Auth API for a given account, which allows
+executing a specific action on the account's behalf without authentication
+- `AccountData` short form version of `Account` displaying relevant account information
+- `FollowedChannel` entities containing information about a channel being followed by an account
+
+Furthermore:
+- `GatewayConfig` entity has been added in order to allow persisting configuration variables of different types in 
+the database. The logic of retrieving setting and updating configration variables is defined in `src/utils/config.ts`
+Each configuration variable specified in `src/utils/config.ts` has a corresponding environment variable which serves as a 
+default value in case the cnofig value is not set in the database. The information stored will be: `( config_variable_name, value, last_modified_at_timestamp)`
+
+### Middleware
+Following the introduction of user-accounts a new middleware authentication has been introduced, allowing the execution of resolvers
+only to user that have registered an account on orion. In particular a new `MiddleWareFn` has been introduced `AccountOnly` for this 
+purpose only. This means that some previously accessible queries / mutations now have been restricted to registered users only
+### Queries
+#### Additions
+- `accountData`: resolver for which a registered account can gather information such as id, mail, joystream address, joystream memberid, 
+and whether his email has been verified
+#### Changes
+- `getVideoPerIpLimit`: has been removed
+
+###  Mutations
+#### Changes
+Several changes are due to the `ContextWithIp` type being replaced by a `Context` that contains user's sesion and 
+possibly account information
+- `followChannel`:
+    - now executable only by registered account 
+    - argument `ctx: ContextWithIp` has been replaced by a `ctx: Context` containing account informations
+    - `ChannelFollowResult.cancelToken` field on the return type has been removed and `ChannelFollowResult.followId` has been added
+- `unfollowChannel`: 
+    - now executable only by registered account
+    - context argument `ctx: ContextWithIp` has been replaced by  `ctx: Context` already containing registered account informations
+    - `UnfollowChannelArgs.token` argument for the `unfollowChannel` resolver has been removed
+- `requestNftFeatured` `ctx: ContextWithIp` arg has been replaced by `ctx: Context` in order to access user information and `NftFeatureRequestInfo.reporterId` field has been dropped
+- `addVideoView` `ctx: ContextWithIp` argument has been replaced by a `ctx: Context` argument
+- `reportVideo`:
+    - `ctx: ContextWithIp` argument has been replaced by a `ctx: Context` argument
+    - `VideoReportInfo.reporterIp: string` field on the return type been dropped
+- `reportChannel`:
+    - `ctx: ContextWithIp` argument has been replaced by a `ctx: Context` argument
+    - `ChannelReportInfo.reporterIp: string` field on the return type been dropped
+- `setVideoViewPerIpLimit`: 
+    - has been renamed to `setVideoViewPerUserLimit`
+    - argument changed from `VideoViewPerIpTimeLimitInput` to `VideoViewPerUserTimeLimitInput`
+
+### Data migration
+- `VideoView`, `Report` and `NftFeaturingRequest` entities will be persisted. However, they will all be assigned to a mock
+"migration user", created during import (with `id: v2-migration-{random-id-string}`)
+- `ChannelFollows` will not be persisted. In v3, you need to have a registered account in order to follow a channel. 
+
+### Documentation
+The `/doc` folder has been improved by adding several pieces of documentation, containing explainers and tutorials for both
+developers and gateway operators
 # 2.3.0
 
 ### Features:
@@ -264,4 +417,4 @@ For detailed overview of the new architecture, see the [developer guide](docs/de
 - `setVideoViewPerIpTimeLimit` - allows specifying the time after which a video view triggered from the same ip address will be counted again (see: `addVideoView`)
 - `excludeContent` - allows excluding specified channels/videos/comments from all query results. Can be used as a gateway-level mechanism to censor some of the content. Comments are the only entities that don't get completely filtered-out from query results when excluded. Instead, their `text` becomes hidden and `isExcluded` property is set to `true`.
 - `restoreContent` - effectively the opposite of `excludeContent`, can be used to make content appear in the query results again (if previously excluded).
-- `setFeaturedNfts` - allows the operator to provide the list of nfts (ids) that are currently featured by the Gateway. This will affect the `isFeatured` propety of the `OwnedNft` entity.
+- `setFeaturedNfts` - allows the operator to provide the list of nfts (ids) that are currently featured by the Gateway. This will affect the `isFeatured` propety of the `OwnedNft` entity.

Dockerfile

@@ -12,6 +12,7 @@ RUN npm ci
 ADD tsconfig.json .
 ADD src src
 ADD schema schema
+ADD scripts scripts
 RUN npx squid-typeorm-codegen
 RUN npm run build
 
@@ -32,6 +33,7 @@ RUN echo -e "loglevel=silent\nupdate-notifier=false" > /squid/.npmrc
 ADD db db
 ADD assets assets
 ADD schema schema
+ADD scripts scripts
 # TODO: use shorter PROMETHEUS_PORT
 ENV PROCESSOR_PROMETHEUS_PORT 3000
 EXPOSE 3000

Makefile

@@ -2,6 +2,7 @@ process: migrate
 	@SQD_DEBUG=sqd:processor:mapping node -r dotenv-expand/config lib/processor.js
 
 install:
+	@rm -rf node_modules # clean up node_modules to avoid issues with patch-package
 	@npm install
 
 build:
@@ -13,6 +14,9 @@ build-docker:
 serve:
 	@npx squid-graphql-server --subscriptions
 
+serve-auth-api:
+	@npm run auth-server-start
+
 migrate:
 	@npx squid-typeorm-migration apply
 
@@ -40,8 +44,7 @@ up-archive:
 up: up-archive up-squid
 
 down-squid:
-	@docker-compose stop orion_processor
-	@npm run offchain-state:export && docker-compose down -v
+	@docker-compose down -v
 
 down-archive:
 	@docker-compose -f archive/docker-compose.yml down -v

README.md

@@ -1,7 +1,7 @@
-# Orion V2
+# Orion
 
-Orion V2 is a backed node powering [Atlas](https://github.com/Joystream/atlas).
+Orion is a backed node powering [Atlas](https://github.com/Joystream/atlas).
 The project is using [Subsquid framework](https://docs.subsquid.io/) and is based on [squid-substrate-template](https://github.com/subsquid/squid-substrate-template).
 
-## [Developer guide](docs/developer-guide.md)
+## [Developer guide](docs/developer-guide/index.md)
 ## [Operator guide](docs/operator-guide.md)

archive/.env

@@ -1,4 +1,4 @@
 DB_PORT=12345
 GATEWAY_PORT=8888
-WS_SOURCE=ws://joystream-node:9944
+WS_SOURCE=wss://rpc.joystream.org:9944
 EXPLORER_PORT=4444