Preparing for pilot: hardening core for stability of V1.0

[ghost]

Items for 1.0 hardening

Roll-up issue corresponding to task on the project board of the same name:
https://github.com/inblockio/DataAccounting/projects/2#card-72943597

Based on initial review of the DataAccounting codebase the following items should be considered and prioritized. This list is non-exhaustive and a work in progress. It should be broken down into actionable pieces once discussed internally.

Special considerations for pilot

• Smooth backup/restore process to increase user confidence
• MediaWiki "Just Works" (including media uploads with hashing)
• Add extension to the Chrome Web Store (with automatic or easy updates)
• Breaking changes causing users to have to manually update (or very easy container update via script)
• Changes in MW and/or DA extension causing issues
• Breaking changes caused by API changes

In Scope

• backups [REFACTOR] Scheduled backups #180
  • Add version number to backup script
  • Warn when restoring between enviornments
  • add scheduled backups See related issue in micro-PKC When PKC has a critical failure which leads to complete data-loss of the database there is a quick recovery aqua-PKC#88
  • fix naming conventsion See related issue in PKC for changing naming convention It's not possible to have more then one backup per day - improve naming convention for backups! aqua-PKC#89
  • [Backup] add domain ID as the first part of the backup-naming-convention to avoid name-collision aqua-PKC#38 (will be obsolete by It's not possible to have more then one backup per day - improve naming convention for backups! aqua-PKC#89)
  • Related: [Backup] Database locked after backup aqua-PKC#45
  • Related: Remove unnecessary $pkcServer in LocalSettings.php aqua-PKC#46
  • Related: [Restore / Recovery] Check if Localsettings.php is restored completely after backup and recovery aqua-PKC#49
  • Related: After './pkc restore' a non-default main-page will not be recovered aqua-PKC#86
  • Related: [Backup / Restore] Include Localsettings.php in .pkc backup / .pkc restore #139 opened 15 days ago by FantasticoFox
  • Localsettings.php / symlink in backup is overriding Localsettings in the container #158
• Secure hard-coded database authentication details in build manifest
• [SECURITY] database passwords are set to a hard-coded password, should be random and undisclosed aqua-PKC#4
• Own vendored code. WikiMedia back-up script should be forked internally or transferred.
  • Docker images under inblockio org or single user on Docker Hub aqua-PKC#76
    • xlp0/mariadb | https://github.com/xlp0/DockerizedMariaDB
    • fantasticofox/pkc | https://github.com/inblockio/mw-docker | https://github.com/inblockio/docker-aqua-mediawiki
    • pelith/node-eauth-server | https://github.com/inblockio/node-eauth-server
• Isolate environments. Create clear boundaries between development and production.
  • Remove .env file from repo aqua-PKC#65
  • Create development environment aqua-PKC#71
• Repeatable builds. Fully automated and documented in code.
  • Add shell format linter ci: 🎡 Add gitlint GH Actions aqua-PKC#75
  • Standardize scripts dirs CLOSED: Standardize on scripts and refactor as we go.
  • Integrate the docker build process into github actions aqua-docker-mediawiki#2
• Reduce dependencies. Eliminate moving parts and reduce size of codebase.
  • Upgrade to Docker Compose V2 aqua-PKC#63 nice to have
  • Remove parent container wrapping MariaDB (with an attempt at cron) from PKC and build on db images directly
• Add ability to Measure Test Coverage
  • At least partial integration completed as shown here: [BREAKING][BUGFIX] Properly use domain manifest verification hash for witness hash #173
  • Add CodeCov to DataAccounting
  • Review other repos and integrate CodeCov (and add where it brings most value only)
• Increase unit test coverage of: DB Access (schema), Function (dependencies), API surface
  • Review current CodeCov levels and set expectations and targets (as a team).
• Scan docker containers (automated) and upstream Node deps. Update deps and Node version in Eauth plugin. Surface vulnerable code in CI.
• Decouple applications. Each can run on its own with a test harness). Each should run their own CI tests independently.
• Automatically generate change logs
  • Adopt semantic versioning within Core
  • Choose initial version numbers (start with 0.1.0 or 1.0.0-alphaN)

Out of scope for 1.0

• Investigate ability to run a smoke test as part of CI
  • Review WebDriver and CLI wallets (https://pkc.inblock.io/index.php/PKC_Manual_Test_Protocol)
• Create a provisioning script and skel to automatically provision new Playground vps instances
• Add test to ensure ENV file does not contain spaces Re: Eauth login cannot be completed as expected aqua-PKC#81
• [NGINX] Hardening aqua-PKC#55
• Harden around MW login, Eauth, Nginx or other entry points
• Normalize build tooling between repos
  • Add Makefile to build and publish project in docker-aqua-mediawiki (model from mw-docker)
• Establish Test Matrix. Document support levels for interpreted code (e.g. PHP >= 8.x, Node >= 12.x).

Scrub repos for additional issues.

[FantasticoFox]

I propose to postpone inblockio/aqua-PKC#57 until 1.4

[ghost]

Added the following non-trivial item:

• Secure hard-coded database authentication details in build manifest

@rht @FantasticoFox fyi

[ghost]

Added a few more pertinent items to the hardening list above. The list is large enough now we should revisit it as a team to discuss resource allocation and schedule risk. /cc @FantasticoFox

[FantasticoFox]

Lets do so on 29th @0xc0000007b

[FantasticoFox]

Related but maybe let's put that for later. inblockio/aqua-PKC#90

[FantasticoFox]

This is continues work to harden our core. Not part of the prototype release schedule.

[FantasticoFox]

@vhs-dev please advice what we should do with this?

[vhscom]

Closed. Let's continue to iterate on tasks in priority order. Thanks for letting me hijack this issue for planning purpose prior to the PKC 1.0 Release