Skip to content

Commit

Permalink
docs: clarify badger caveats
Browse files Browse the repository at this point in the history
For non-English, badger is fine, so it is better to list it as the default.
  • Loading branch information
lidel authored May 22, 2021
1 parent 541336d commit b6cb7f3
Showing 1 changed file with 4 additions and 4 deletions.
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -114,15 +114,14 @@ It is advised to use separate IPFS node for this:

```console
$ export IPFS_PATH=/path/to/IPFS_PATH_WIKIPEDIA_MIRROR
$ ipfs init -p server,local-discovery,flatfs,randomports --empty-repo
$ ipfs init -p server,local-discovery,badgerds,randomports --empty-repo
```

#### Tune datastore for speed

Make sure repo uses `flatfs` datastore with `sync` set to `false` for improved performance.
Make sure repo is initialized with datastore backed by `badgerds` for improved performance, or if you choose to use slower `flatfs` at least use it with `sync` set to `false`.

**NOTE:** While badgerv1 datastore _is_ faster, we avoid using it with bigger builds like English because of [memory issues due to the number of files](https://github.com/ipfs/distributed-wikipedia-mirror/issues/85).
With `flatfs` having `sync` set to `false` and a fast SSD one should be able to import 300GB of English wikipedia under 5 hours, which is still acceptable, and way easier on CPU/memory.
**NOTE:** While badgerv1 datastore _is_ faster, one may choose to avoid using it with bigger builds like English because of [memory issues due to the number of files](https://github.com/ipfs/distributed-wikipedia-mirror/issues/85). Potential workaround is to use [`filestore`](https://github.com/ipfs/go-ipfs/blob/master/docs/experimental-features.md#ipfs-filestore) that avoids duplicating data and reuses unpacked files as-is.

#### Enable HAMT sharding

Expand All @@ -132,6 +131,7 @@ Configure your IPFS node to enable directory sharding
$ ipfs config --json 'Experimental.ShardingEnabled' true
```

This step won't be necessary when automatic sharding lands in go-ipfs (wip).

### Step 3: Download the latest snapshot from kiwix.org

Expand Down

0 comments on commit b6cb7f3

Please sign in to comment.