Docs/scripts used to recreate CAR archive

docs/README.md

@@ -0,0 +1,109 @@
+# Recreate the full CAR archive
+
+If you're ready for months of toiling away, going through 100s of terabytes of data over and over, spending some hunnids per month on hardware and bandwidth and going through all the error and trial required for this task please continue reading below.
+
+
+## Hardware & other requirements
+
+1. Servers
+2. Object storage
+3. Software
+
+### Servers
+
+You'll need _big_ machines:
+- 20Tb working storage for juggling multiple snapshots + working data should be comfortable
+- 10Gbit uplink can shave hours of downloading/uploading stuff for each epoch
+- Good CPU power speeds up car generation
+
+We've used this Ryzen 9 from Hetzner with great success
+
+- AX102 with 10Gbit uplink + 4x7.68 NVMe (disks are pooled using LVM)
+
+https://www.hetzner.com/dedicated-rootserver/ax102/configurator/#/
+
+Order multiple servers to parallelize the work and speed up the process.
+
+### Object storage:
+
+We've uploaded all CAR files + indexes to Backblaze B2 due to storage cost + bandwidth alliance + egress fees.
+
+If we had to start again we could've probably self-hosted as upload/download performance and reliability was lacking at times.
+
+### Software:
+
+- Ubuntu 20
+- python3
+  - gsutil
+  - b2 (if you use B2 to store CAR files)
+  - `python3 -m pip install --upgrade gsutil b2`
+- s5cmd
+- zstd
+- /usr/local/bin/faithful-cli [faithful-cli](https://github.com/rpcpool/yellowstone-faithful)
+- /usr/local/bin/filecoin-data-prep [go-fil-dataprep](https://github.com/rpcpool/go-fil-dataprep)
+- Node Exporter
+
+
+### Preparation
+
+Find and replace the variables below with your own values (or use it as an Ansible template):
+
+- {{ warehouse_processor_snapshot_folder }} - Where to store snapshots, i.e. `/snapshots`
+- {{ warehouse_processor_car_folder }} - Where to save CAR files, i.e. `/cars`
+- {{ warehouse_slack_token }} - Webhook token to send messages to your Slack Channel
+- {{ inventory_hostname }} - Hostname where the script is installed, i.e. `$(hostname)`
+
+## Running
+
+We've used Rundeck as a controller node to initiate and run jobs on the multiple processing nodes.
+
+On each, run the script and specify a range of _epochs_ to go through where the script will attempt to download all relevant snapshots from Solana Foundation GCP buckets in order to gather all relevant slots.
+
+
+`create-cars.sh 100-200`
+
+`create-cars.sh 300-500`
+
+`create-cars.sh 600-620`
+
+There's some functions that send notifications to Slack and create metric files, you can comment it out if not relevant for you.
+## Manual CAR generation
+
+create-cars.sh only tries to find snapshots in the EU region GCP bucket.
+
+When script fails to find the required snapshots, you may try to do it manually following the steps below:
+
+```
+export EPOCH=500
+
+# try to find snapshots in other regions
+download_ledgers_gcp.py $EPOCH us
+# or download_ledgers_gcp.py $EPOCH ap
+
+# check the given snapshots contain all required slots
+/usr/local/bin/radiance car create2 --db snapshot1/rocksdb $EPOCH --db snapshots2/rocksdb --out="$CARDIR/epoch-$EPOCH.car" --check
+
+# generate car
+/usr/local/bin/radiance car create2 --db snapshot1/rocksdb $EPOCH --db snapshots2/rocksdb --out="$CARDIR/epoch-$EPOCH.car"
+
+```
+
+Then you can create a copy of `create-cars.sh` that skips the downloading and cargen steps and proceeds with the remaining tasks.
+
+## Costs
+
+Using B2 object storage service and the Hetzner node recommended above your costs:
+
+- servers: €500/month per processor node
+- object storage: 230TB will cost min 1300$/month - data includes CAR + split file + index storage
+
+## Dashboards
+
+create-cars.sh is creating .prom files in `/var/lib/node_exporter/`, so by having node exporter running and being scraped (Prometheus) you can monitor the car generation progress on Grafana .
+
+Import dashboard.yml in this directory into Grafana, update the variable with your node names (or use a query instead).
+
+## Gotchas
+
+1. First 8 epochs are missing bounds.txt file so you may need to do it manually.
+2. Around epoch 32 there was a change in `shred revision`, depending on what snapshot you use you may need to adjust `SHRED_REVISION_CHANGE_EPOCH` (currently set at 32, but could be 24)

docs/create-cars.sh

@@ -0,0 +1,221 @@
+#!/bin/bash
+
+set -e
+shopt -s nullglob
+
+SNAPSHOT_FOLDER="{{ warehouse_processor_snapshot_folder }}"
+
+# find out the right number
+SHRED_REVISION_CHANGE_EPOCH="32"
+
+CARDIR="{{ warehouse_processor_car_folder }}"
+
+# add defaults?
+RANGE_START=$1
+
+RANGE_END=$2
+
+get_archive_file() {
+    find "$1" -type f \( -name "rocksdb.tar.zst" -o -name "rocksdb.tar.bz2" \)
+}
+
+cleanup_artifacts() {
+
+    cardirepoch="$1"
+    workdir="$2"
+    set -x
+    echo "cleaning up generated files"
+    rm -rf $cardirepoch/*index
+    rm -rf $cardirepoch/*gsfa.index.tar.bz2
+    rm -rf $cardirepoch/*.car
+    rm -rf $workdir/*
+    set +x
+}
+
+# remove rocksdb data
+cleanup_workfiles() {
+    workdir="$1"
+    prev_workdir="$2"
+
+    if [[ "${prev_workdir}" != "" ]]; then
+        rm -rf $prev_workdir/rocksdb*
+    fi
+    # only remove the archive, we might need prev snapshot on the next snapshot cargen
+    rm -rf $workdir/rocksdb.tar*
+}
+
+metric() {
+    EPOCH="$1"
+    STATUS="$2"
+    echo "cargen_status{epoch=\"$EPOCH\",status=\"$STATUS\"} 1" | sponge /var/lib/node_exporter/cargen_$EPOCH.prom
+}
+
+
+slack() {
+    msg="$1"
+    set -x
+    curl -XPOST "https://hooks.slack.com/services/{{ warehouse_slack_token }}" -H "Content-type: application/json" -d "{\"text\": \"$msg\"}"
+    set +x
+}
+
+find_ledgers_file_snapshot() {
+    epoch="$1"
+    ledgers_file="$SNAPSHOT_FOLDER/epoch-$epoch.ledgers"
+    set -x
+    if [ ! -f "$ledgers_file" ]; then
+        # echo "Ledgers file not found for epoch $epoch: $ledgers_file"
+        echo ""
+        return 0
+    fi
+
+    cat "$ledgers_file"
+    set +x
+}
+
+
+download_ledger() {
+    epoch="$1"
+    echo "pulling snapshot for $epoch..."
+    metric $EPOCH "downloading"
+    python3 download_ledgers_gcp.py $epoch eu
+}
+
+for EPOCH in $(seq $RANGE_START $RANGE_END); do
+
+    download_ledger $EPOCH
+    WORKDIR=`find_ledgers_file_snapshot $EPOCH | tail -n1`
+    SNAPSHOT=$(basename "$WORKDIR")
+
+    SHRED_REVISION_FLAG=""
+
+
+    # if test ! -f "$WORKDIR/epoch"; then
+    #     echo "no epoch file"
+    #     # find way to calculate epoch instead of skipping
+    #     # or manually create bounds file (only first 8 epochs)
+    #     # or manually run the cargen
+    #     continue
+    # fi
+    #
+    # EPOCH=`cat $WORKDIR/epoch | tr -d '\n'`
+
+    # epochs under SHRED_REVISION_CHANGE_EPOCH need different handling
+    if [[ "${EPOCH}" -lt "${SHRED_REVISION_CHANGE_EPOCH}" ]]; then
+        echo "setting shred-revision=1 for epoch under ${SHRED_REVISION_CHANGE_EPOCH}"
+        SHRED_REVISION_FLAG="--shred-revision=1"
+    fi
+
+    # SHRED_REVISION_CHANGE_EPOCH epoch needs special handling at a specific slot
+    if [[ "${EPOCH}" -eq "${SHRED_REVISION_CHANGE_EPOCH}" ]]; then
+        echo "need special handling for epoch ${SHRED_REVISION_CHANGE_EPOCH}"
+        SHRED_REVISION_FLAG="--shred-revision=1 --next-shred-revision-activation-slot=10367766"
+    fi
+
+    echo "Working on snapshot: $SNAPSHOT for epoch: $EPOCH"
+
+    # remove incomplete work
+    mkdir "$CARDIR/" || true
+    rm "$CARDIR/epoch-$EPOCH.car" || true
+
+    DBS=""
+
+    # The <epoch>.ledgers file is created by download_ledgers.py
+    # this will catch when previous snapshots are required from reading bounds.txt
+    echo "reading ledgers file for $EPOCH..."
+    while read -r line; do
+        DBS+=" --db=${line}/rocksdb";
+    done <<< "$(find_ledgers_file_snapshot $EPOCH)"
+
+    # use check mode to check if we need to add next snapshot
+    set -x
+    if ! /usr/local/bin/radiance car create2 "$EPOCH" $DBS --out="$CARDIR/epoch-$EPOCH.car" --check; then
+        # set +x
+
+        echo "pulling next snapshot..."
+        NEXT_SNAP=$((EPOCH + 1))
+        download_ledger "$NEXT_SNAP"
+
+        echo "reading ledgers file for $NEXT_SNAP..."
+        while read -r line; do
+            DBS+=" --db=${line}/rocksdb";
+        done <<< "$(find_ledgers_file_snapshot $NEXT_SNAP)"
+
+        # DBS="--db=$WORKDIR/rocksdb"
+    fi
+    # set +x
+
+    echo "Using databases: ${DBS}"
+
+    metric $EPOCH "cargen"
+
+    # create car file
+    set -x
+    /usr/local/bin/radiance car create2 "$EPOCH" $DBS --out="$CARDIR/epoch-$EPOCH.car" $SHRED_REVISION_FLAG
+    # set +x
+
+    # cleanup work files
+    echo "cleaning up work files"
+    set -x
+
+    # some epochs may need 2 snapshots
+    # so we need to keep at least 2 snapshots going back
+    PREV_WORKDIR_LEDGERS=$(find_ledgers_file_snapshot $((EPOCH-2)) | tail -n1)
+    if [ -n "$PREV_WORKDIR_LEDGERS" ]; then
+        cleanup_workfiles $WORKDIR $PREV_WORKDIR_CONTENT
+    else
+        cleanup_workfiles $WORKDIR
+    fi
+    # set +x
+
+    # check file exists and non empty
+    if ! test -s "$CARDIR/epoch-$EPOCH.car" ; then
+        echo "car file is empty/non existant, skipping next steps"
+        continue;
+    fi
+
+    if ! test -f "$CARDIR/$EPOCH.cid"; then
+        echo "no root cid, generation probably failed"
+        continue;
+    fi
+
+    # Create the cardir epoch dir
+    mkdir -p "$CARDIR/$EPOCH" || true
+    # move cid/recap/slots files
+    mv $CARDIR/$EPOCH.cid $CARDIR/$EPOCH/epoch-$EPOCH.cid # IMP RENAME THE CID
+    mv $CARDIR/$EPOCH.* $CARDIR/$EPOCH/
+    /usr/local/bin/radiance version >$CARDIR/$EPOCH/radiance.version.txt
+
+    # split into 30gb files
+    echo "splitting car file..."
+    metric $EPOCH "splitting"
+    /usr/local/bin/split-epoch.sh $EPOCH $CARDIR/epoch-$EPOCH.car $CARDIR
+
+    # creates the indexes and the appropriate car files
+    # since split-epoch moves the epoch file, we need to also use a different path here for the epoch file
+    echo "creating indexes..."
+    metric $EPOCH "indexing"
+    /usr/local/bin/index-epoch.sh all $EPOCH $CARDIR/$EPOCH/epoch-$EPOCH.car $CARDIR
+
+    # upload
+    echo "uploading car file..."
+    metric $EPOCH "uploading"
+    B2_ACCOUNT_INFO=/etc/b2/filecoin_account_info /usr/local/bin/upload-epoch.sh $EPOCH $CARDIR
+
+    # clean up
+    echo "cleaning up artifacts..."
+    cleanup_artifacts "$CARDIR/$EPOCH" "$WORKDIR"
+
+    slack "{{ inventory_hostname }} finished generating epoch $EPOCH car files"
+
+    metric $EPOCH "complete"
+
+    # done
+    touch "$WORKDIR/.cargen"
+
+    # reached end of range
+    if [[ "${EPOCH}" -eq "${RANGE_END}" ]]; then
+        echo "reached end of the given range, please restart job with new parameters to resume cargen"
+        exit 0
+    fi
+
+done