Skip to content

Commit

Permalink
update docs/reference.md (#5009)
Browse files Browse the repository at this point in the history
Co-authored-by: Changjian Gao <[email protected]>
Co-authored-by: Git'Fellow <[email protected]>
  • Loading branch information
3 people authored Jul 22, 2024
1 parent 61a0099 commit c7fe006
Show file tree
Hide file tree
Showing 50 changed files with 2,759 additions and 417 deletions.
16 changes: 7 additions & 9 deletions docs/en/administration/fault_diagnosis_and_analysis.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Different JuiceFS clients print logs in different ways, which are described as f

### Mount point

When a JuiceFS file system is mounted with the [`-d` option](../reference/command_reference.md#mount) (indicating running in the background), it will print logs to the system log file and local log file simultaneously. Depending on which user is running when mounting the file system, the paths of the local log files are slightly different. For root, the local log file locates at `/var/log/juicefs.log`, while it locates at `$HOME/.juicefs/juicefs.log` for non-root users. Please refer to [`--log` option](../reference/command_reference.md#mount) for details.
When a JuiceFS file system is mounted with the [`-d` option](../reference/command_reference.mdx#mount) (indicating running in the background), it will print logs to the system log file and local log file simultaneously. Depending on which user is running when mounting the file system, the paths of the local log files are slightly different. For root, the local log file locates at `/var/log/juicefs.log`, while it locates at `$HOME/.juicefs/juicefs.log` for non-root users. Please refer to [`--log` option](../reference/command_reference.mdx#mount) for details.

Depending on the operating system, there are different commands to retrieve system logs or read local log files directly.

Expand Down Expand Up @@ -118,7 +118,7 @@ kubectl -n kube-system exec juicefs-chaos-k8s-002-pvc-d4b8fb4f-2c0b-48e8-a2dc-53

### S3 Gateway

You need to add the [`--access-log` option](../reference/command_reference.md#gateway) when starting the S3 gateway to specify the path to output the access log. By default, the S3 gateway does not output the access log.
You need to add the [`--access-log` option](../reference/command_reference.mdx#gateway) when starting the S3 gateway to specify the path to output the access log. By default, the S3 gateway does not output the access log.

### Hadoop Java SDK

Expand All @@ -142,9 +142,7 @@ This command collects the following information:
6. Go pprof information
7. JuiceFS logs (defaulting to the last 5000 lines)

By default, a `debug` directory is created in the current directory, and the collected information is saved in that directory.

Here's an example:
By default, a `debug` directory is created in the current directory, and the collected information is saved in that directory. Here's an example:

```shell
$ juicefs debug /tmp/mountpoint
Expand Down Expand Up @@ -177,7 +175,7 @@ JuiceFS provides the `profile` and `stats` subcommands to visualize real-time pe

### `juicefs profile` {#profile}

[`juicefs profile`](../reference/command_reference.md#profile) will collect data from [file system access log](#access-log), run the `juicefs profile MOUNTPOINT` command, you can see the real-time statistics of each file system operation based on the latest access log:
[`juicefs profile`](../reference/command_reference.mdx#profile) will collect data from [file system access log](#access-log), run the `juicefs profile MOUNTPOINT` command, you can see the real-time statistics of each file system operation based on the latest access log:

![JuiceFS-profiling](../images/juicefs-profiling.gif)

Expand All @@ -201,7 +199,7 @@ juicefs profile /tmp/juicefs.accesslog --uid 12345

### `juicefs stats` {#stats}

The [`juicefs stats`](../reference/command_reference.md#stats) command reads JuiceFS Client internal metrics data, and output performance data in a format similar to `dstat`:
The [`juicefs stats`](../reference/command_reference.mdx#stats) command reads JuiceFS Client internal metrics data, and output performance data in a format similar to `dstat`:

![juicefs_stats_watcher](../images/juicefs_stats_watcher.png)

Expand All @@ -211,7 +209,7 @@ Metrics description:

- `cpu`: CPU usage of the process.
- `mem`: Physical memory used by the process.
- `buf`: Current [buffer size](../guide/cache.md#buffer-size), if this value is constantly close to (or even exceeds) the configured [`--buffer-size`](../reference/command_reference.md#mount), you should increase buffer size or decrease application workload.
- `buf`: Current [buffer size](../guide/cache.md#buffer-size), if this value is constantly close to (or even exceeds) the configured [`--buffer-size`](../reference/command_reference.mdx#mount-data-cache-options), you should increase buffer size or decrease application workload.
- `cache`: Internal metric, ignore this.

#### `fuse`
Expand Down Expand Up @@ -276,7 +274,7 @@ You can also use the `juicefs debug` command to automatically collect these runt
juicefs debug /mnt/jfs
```

For more information about the `juicefs debug` command, see [command reference](../reference/command_reference.md#debug).
For more information about the `juicefs debug` command, see [command reference](../reference/command_reference.mdx#debug).
:::

If you have the `go` command installed, you can analyze it directly with the `go tool pprof` command. For example to analyze CPU performance statistics:
Expand Down
4 changes: 2 additions & 2 deletions docs/en/administration/metadata/redis_best_practices.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,7 @@ used_memory_dataset: 13439673592
used_memory_dataset_perc: 70.12%
```

Among them, `used_memory_rss` is the total memory size actually used by Redis, which includes not only the size of data stored in Redis (that is, `used_memory_dataset` above) but also some Redis [system overhead](https://redis.io/commands/memory-stats) (that is, `used_memory_overhead` above). As mentioned earlier that the metadata of each file occupies about 300 bytes, this is actually calculated by `used_memory_dataset`. If you find that the metadata of a single file in your JuiceFS file system occupies much more than 300 bytes, you can try to run [`juicefs gc`](../../reference/command_reference.md#gc) command to clean up possible redundant data.
Among them, `used_memory_rss` is the total memory size actually used by Redis, which includes not only the size of data stored in Redis (that is, `used_memory_dataset` above) but also some Redis [system overhead](https://redis.io/commands/memory-stats) (that is, `used_memory_overhead` above). As mentioned earlier that the metadata of each file occupies about 300 bytes, this is actually calculated by `used_memory_dataset`. If you find that the metadata of a single file in your JuiceFS file system occupies much more than 300 bytes, you can try to run [`juicefs gc`](../../reference/command_reference.mdx#gc) command to clean up possible redundant data.

## High availability

Expand Down Expand Up @@ -124,7 +124,7 @@ After generating the AOF or RDB backup file, you can restore the data by copying

If both AOF and RDB persistence are enabled, Redis will use the AOF file first on starting to recover the data because AOF is guaranteed to be the most complete data.

After recovering Redis data, you can continue to use the JuiceFS file system via the new Redis address. It is recommended to run [`juicefs fsck`](../../reference/command_reference.md#fsck) command to check the integrity of the file system data.
After recovering Redis data, you can continue to use the JuiceFS file system via the new Redis address. It is recommended to run [`juicefs fsck`](../../reference/command_reference.mdx#fsck) command to check the integrity of the file system data.

## Recommended Managed Redis Service

Expand Down
6 changes: 3 additions & 3 deletions docs/en/administration/metadata_dump_load.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ slug: /metadata_dump_load
- JuiceFS v1.0.4 starts to support importing an encrypted backup.
:::

JuiceFS supports [multiple metadata engines](../reference/how_to_set_up_metadata_engine.md), and each engine stores and manages data in a different format internally. JuiceFS provides the [`dump`](../reference/command_reference.md#dump) command to export metadata in a uniform JSON format, also there's the [`load`](../reference/command_reference.md#load) command to restore or migrate backups to any metadata storage engine. This dump / load process can also be used to migrate a community edition file system to enterprise edition (read [enterprise docs](https://juicefs.com/docs/cloud/metadata_dump_load) for more), and vice versa.
JuiceFS supports [multiple metadata engines](../reference/how_to_set_up_metadata_engine.md), and each engine stores and manages data in a different format internally. JuiceFS provides the [`dump`](../reference/command_reference.mdx#dump) command to export metadata in a uniform JSON format, also there's the [`load`](../reference/command_reference.mdx#load) command to restore or migrate backups to any metadata storage engine. This dump / load process can also be used to migrate a community edition file system to enterprise edition (read [enterprise docs](https://juicefs.com/docs/cloud/metadata_dump_load) for more), and vice versa.

## Metadata backup {#backup}

Expand Down Expand Up @@ -81,7 +81,7 @@ JuiceFS periodically cleans up backups according to the following rules.

## Metadata recovery and migration {#recovery-and-migration}

Use the [`load`](../reference/command_reference.md#load) command to restore the metadata dump file into an empty database, for example:
Use the [`load`](../reference/command_reference.mdx#load) command to restore the metadata dump file into an empty database, for example:

```shell
juicefs load redis://192.168.1.6:6379 meta-dump.json
Expand Down Expand Up @@ -111,7 +111,7 @@ It is also possible to migrate directly through the system's pipe:
juicefs dump redis://192.168.1.6:6379 | juicefs load mysql://user:password@(192.168.1.6:3306)/juicefs
```

Note that since the API access key for object storage is excluded by default from the backup, when loading metadata, you need to use the [`juicefs config`](../reference/command_reference.md#config) command to reconfigure the object storage credentials. For example:
Note that since the API access key for object storage is excluded by default from the backup, when loading metadata, you need to use the [`juicefs config`](../reference/command_reference.mdx#config) command to reconfigure the object storage credentials. For example:

```shell
juicefs config --secret-key xxxxx mysql://user:password@(192.168.1.6:3306)/juicefs
Expand Down
2 changes: 1 addition & 1 deletion docs/en/administration/monitoring.md
Original file line number Diff line number Diff line change
Expand Up @@ -89,7 +89,7 @@ For different types of JuiceFS Client, metrics data is handled slightly differen

### Mount point {#mount-point}

When the JuiceFS file system is mounted via the [`juicefs mount`](../reference/command_reference.md#mount) command, you can collect monitoring metrics via the address `http://localhost:9567/metrics`, or you can customize it via the `--metrics` option. For example:
When the JuiceFS file system is mounted via the [`juicefs mount`](../reference/command_reference.mdx#mount) command, you can collect monitoring metrics via the address `http://localhost:9567/metrics`, or you can customize it via the `--metrics` option. For example:

```shell
juicefs mount --metrics localhost:9567 ...
Expand Down
6 changes: 3 additions & 3 deletions docs/en/administration/troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -98,8 +98,8 @@ If the problem is a network connection issue, or the object storage has service

The first issue with slow connection is upload / download timeouts (demonstrated in the above error logs), to tackle this problem:

* Reduce upload concurrency, e.g. [`--max-uploads=1`](../reference/command_reference.md#mount), to avoid upload timeouts.
* Reduce buffer size, e.g. [`--buffer-size=64`](../reference/command_reference.md#mount) or even lower. In a large bandwidth condition, increasing buffer size improves parallel performance. But in a low speed environment, this only makes `flush` operations slow and prone to timeouts.
* Reduce upload concurrency, e.g. [`--max-uploads=1`](../reference/command_reference.mdx#mount-data-storage-options), to avoid upload timeouts.
* Reduce buffer size, e.g. [`--buffer-size=64`](../reference/command_reference.mdx#mount-data-cache-options) or even lower. In a large bandwidth condition, increasing buffer size improves parallel performance. But in a low speed environment, this only makes `flush` operations slow and prone to timeouts.
* Default timeout for GET / PUT requests are 60 seconds, increasing `--get-timeout` and `--put-timeout` may help with read / write timeouts.

In addition, the ["Client Write Cache"](../guide/cache.md#client-write-cache) feature needs to be used with caution in low bandwidth environment. Let's briefly go over the JuiceFS Client background job design: every JuiceFS Client runs background jobs by default, one of which is data compaction, and if the client has poor internet speed, it'll drag down performance for the whole system. A worse case is when client write cache is also enabled, compaction results are uploaded too slowly, forcing other clients into a read hang when accessing the affected files:
Expand All @@ -111,7 +111,7 @@ In addition, the ["Client Write Cache"](../guide/cache.md#client-write-cache) fe
<WARNING>: fail to read sliceId 1771585458 (off:4194304, size:4194304, clen: 37746372): get chunks/0/0/1_0_4194304: oss: service returned error: StatusCode=404, ErrorCode=NoSuchKey, ErrorMessage="The specified key does not exist.", RequestId=62E8FB058C0B5C3134CB80B6
```

To avoid this type of issue, we recommend disabling background jobs on low-bandwidth clients, i.e. adding [`--no-bgjob`](../reference/command_reference.md#mount) option to the mount command.
To avoid this type of issue, we recommend disabling background jobs on low-bandwidth clients, i.e. adding [`--no-bgjob`](../reference/command_reference.mdx#mount-metadata-options) option to the mount command.

### WARNING log: block not found in object storage {#warning-log-block-not-found-in-object-storage}

Expand Down
4 changes: 2 additions & 2 deletions docs/en/benchmark/performance_evaluation_guide.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ JuiceFS v1.0+ has Trash enabled by default, which means the benchmark tools will

### `juicefs bench`

The [`juicefs bench`](../reference/command_reference.md#bench) command can help you do a quick performance test on a standalone machine. With the test results, it is easy to evaluate if your environment configuration and JuiceFS performance are normal. Assuming you have mounted JuiceFS to `/mnt/jfs` on your server, execute the following command for this test (the `-p` option is recommended to set to the number of CPU cores on the server). If you need help with initializing or mounting JuiceFS, please refer to [Create a File System](../getting-started/standalone.md#juicefs-format).
The [`juicefs bench`](../reference/command_reference.mdx#bench) command can help you do a quick performance test on a standalone machine. With the test results, it is easy to evaluate if your environment configuration and JuiceFS performance are normal. Assuming you have mounted JuiceFS to `/mnt/jfs` on your server, execute the following command for this test (the `-p` option is recommended to set to the number of CPU cores on the server). If you need help with initializing or mounting JuiceFS, please refer to [Create a File System](../getting-started/standalone.md#juicefs-format).

```bash
juicefs bench /mnt/jfs -p 4
Expand Down Expand Up @@ -79,7 +79,7 @@ The data above is from [AWS official documentation](https://docs.aws.amazon.com/

### `juicefs objbench`

The [`juicefs objbench`](../reference/command_reference.md#objbench) command can run some tests on object storage to evaluate how well it performs as a backend storage for JuiceFS. Take testing Amazon S3 as an example:
The [`juicefs objbench`](../reference/command_reference.mdx#objbench) command can run some tests on object storage to evaluate how well it performs as a backend storage for JuiceFS. Take testing Amazon S3 as an example:

```bash
juicefs objbench \
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -93,4 +93,4 @@ For details about the logrotate configuration, see this [link](https://linux.die

## Command line auto-completion

JuiceFS provides command line auto-completion scripts for Bash and Zsh to facilitate the use of `juicefs` commands. For details, see this [document](../reference/command_reference.md#Auto-completion) for details.
JuiceFS provides command line auto-completion scripts for Bash and Zsh to facilitate the use of `juicefs` commands. For details, see this [document](../reference/command_reference.mdx#Auto-completion) for details.
4 changes: 2 additions & 2 deletions docs/en/deployment/s3_gateway.md
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,7 @@ juicefs gateway redis://localhost:6379 localhost:9000

The first two commands of the above three are used to set environment variables. Note that the length of `MINIO_ROOT_USER` is at least 3 characters, and the length of `MINIO_ROOT_PASSWORD` is at least 8 characters. If you are a Windows user, replace `export` with `set` in the above commands to set the environment variable. i.e., `set MINIO_ROOT_USER=admin`.

The last command is used to enable the S3 gateway. The `gateway` subcommand requires at least two parameters. The first is the URL of the database where the metadata is stored, and the second is the address and port on which the S3 gateway is listening. You can add [other options](../reference/command_reference.md#gateway) to the `gateway` subcommand to optimize the S3 gateway as needed, for example, to set the default local cache to 20 GiB.
The last command is used to enable the S3 gateway. The `gateway` subcommand requires at least two parameters. The first is the URL of the database where the metadata is stored, and the second is the address and port on which the S3 gateway is listening. You can add [other options](../reference/command_reference.mdx#gateway) to the `gateway` subcommand to optimize the S3 gateway as needed, for example, to set the default local cache to 20 GiB.

```shell
juicefs gateway --cache-size 20480 redis://localhost:6379 localhost:9000
Expand Down Expand Up @@ -140,7 +140,7 @@ Then download the S3 gateway [deployment YAML](https://github.com/juicedata/juic
- The latest version of `juicedata/juicefs-csi-driver` image is used by default, which has already integrated the latest version of JuiceFS client. Please check [here](https://github.com/juicedata/juicefs-csi-driver/releases) for the specific integrated JuiceFS client version.
- The `initContainers` of `Deployment` will first try to format the JuiceFS file system, if you have already formatted it in advance, this step will not affect the existing JuiceFS file system.
- The default port number that the S3 gateway listens on is 9000
- The [startup options](../reference/command_reference.md#gateway) of S3 gateway will use default values if not specified.
- The [startup options](../reference/command_reference.mdx#gateway) of S3 gateway will use default values if not specified.
- The value of `MINIO_ROOT_USER` environment variable is `access-key` in Secret, and the value of `MINIO_ROOT_PASSWORD` environment variable is `secret-key` in Secret.

```shell
Expand Down
2 changes: 1 addition & 1 deletion docs/en/development/internals.md
Original file line number Diff line number Diff line change
Expand Up @@ -178,7 +178,7 @@ type Attr struct {

There are a few fields that need clarification.

- Atime/Atimensec: See [`--atime-mode`](../reference/command_reference.md#mount)
- Atime/Atimensec: See [`--atime-mode`](../reference/command_reference.mdx#mount-metadata-options)
- Nlink
- Directory file: initial value is 2 ('.' and '..'), add 1 for each subdirectory
- Other files: initial value is 1, add 1 for each hard link created
Expand Down
Loading

0 comments on commit c7fe006

Please sign in to comment.