diff --git a/docs/en/api/visualization.rst b/docs/en/api/visualization.rst index 4e1f4aebc0..bf17a4148e 100644 --- a/docs/en/api/visualization.rst +++ b/docs/en/api/visualization.rst @@ -35,3 +35,4 @@ visualization Backend TensorboardVisBackend WandbVisBackend ClearMLVisBackend + NeptuneVisBackend diff --git a/docs/en/common_usage/visualize_training_log.md b/docs/en/common_usage/visualize_training_log.md index 996da0e052..d85ba075a9 100644 --- a/docs/en/common_usage/visualize_training_log.md +++ b/docs/en/common_usage/visualize_training_log.md @@ -1,12 +1,12 @@ # Visualize Training Logs -MMEngine integrates experiment management tools such as [TensorBoard](https://www.tensorflow.org/tensorboard), [Weights & Biases (WandB)](https://docs.wandb.ai/), [MLflow](https://mlflow.org/docs/latest/index.html) and [ClearML](https://clear.ml/docs/latest/docs), making it easy to track and visualize metrics like loss and accuracy. +MMEngine integrates experiment management tools such as [TensorBoard](https://www.tensorflow.org/tensorboard), [Weights & Biases (WandB)](https://docs.wandb.ai/), [MLflow](https://mlflow.org/docs/latest/index.html), [ClearML](https://clear.ml/docs/latest/docs) and [Neptune](https://docs.neptune.ai/), making it easy to track and visualize metrics like loss and accuracy. Below, we'll show you how to configure an experiment management tool in just one line, based on the example from [15 minutes to get started with MMEngine](../get_started/15_minutes.md). ## TensorBoard -Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to `TensorboardVisBackend`. +Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to [TensorboardVisBackend](mmengine.visualization.TensorboardVisBackend). ```python runner = Runner( @@ -32,7 +32,7 @@ pip install wandb wandb login ``` -Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to `WandbVisBackend`. +Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to [WandbVisBackend](mmengine.visualization.WandbVisBackend). ```python runner = Runner( @@ -81,7 +81,7 @@ pip install clearml clearml-init ``` -Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to `ClearMLVisBackend`. +Configure the `visualizer` in the initialization parameters of the Runner, and set `vis_backends` to [ClearMLVisBackend](mmengine.visualization.ClearMLVisBackend). ```python runner = Runner( @@ -99,3 +99,53 @@ runner.train() ``` ![image](https://github.com/open-mmlab/mmengine/assets/58739961/d68e1dd2-9e82-40fb-ad81-00a647549adc) + +## Neptune + +Before using Neptune, you need to install `neptune` dependency library and refer to [Neptune.AI](https://docs.neptune.ai/) for configuration. + +```bash +pip install neptune +``` + +Configure the `Runner` in the initialization parameters of the Runner, and set `vis_backends` to [NeptuneVisBackend](mmengine.visualization.NeptuneVisBackend). + +```python +runner = Runner( + model=MMResNet50(), + work_dir='./work_dir', + train_dataloader=train_dataloader, + optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)), + train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1), + val_dataloader=val_dataloader, + val_cfg=dict(), + val_evaluator=dict(type=Accuracy), + visualizer=dict(type='Visualizer', vis_backends=[dict(type='NeptuneVisBackend')]), +) +runner.train() +``` + +![image](https://github.com/open-mmlab/mmengine/assets/58739961/9122e2ac-cc4f-43b2-bad3-ae33faa64043) + +Please note: If the `project` and `api_token` are not specified, neptune will be set to offline mode and the generated files will be saved to the local `.neptune` file. +It is recommended to specify the `project` and `api_token` during initialization as shown below. + +```python +runner = Runner( + ... + visualizer=dict( + type='Visualizer', + vis_backends=[ + dict( + type='NeptuneVisBackend', + init_kwargs=dict(project='workspace-name/project-name', + api_token='your api token') + ), + ], + ), + ... +) +runner.train() +``` + +More initialization configuration parameters are available at [neptune.init_run API](https://docs.neptune.ai/api/neptune/#init_run). diff --git a/docs/zh_cn/api/visualization.rst b/docs/zh_cn/api/visualization.rst index 4e1f4aebc0..bf17a4148e 100644 --- a/docs/zh_cn/api/visualization.rst +++ b/docs/zh_cn/api/visualization.rst @@ -35,3 +35,4 @@ visualization Backend TensorboardVisBackend WandbVisBackend ClearMLVisBackend + NeptuneVisBackend diff --git a/docs/zh_cn/common_usage/visualize_training_log.md b/docs/zh_cn/common_usage/visualize_training_log.md index 3c5487479e..36830df96d 100644 --- a/docs/zh_cn/common_usage/visualize_training_log.md +++ b/docs/zh_cn/common_usage/visualize_training_log.md @@ -1,12 +1,12 @@ # 可视化训练日志 -MMEngine 集成了 [TensorBoard](https://www.tensorflow.org/tensorboard?hl=zh-cn)、[Weights & Biases (WandB)](https://docs.wandb.ai/)、[MLflow](https://mlflow.org/docs/latest/index.html) 和 [ClearML](https://clear.ml/docs/latest/docs) 实验管理工具，你可以很方便地跟踪和可视化损失及准确率等指标。 +MMEngine 集成了 [TensorBoard](https://www.tensorflow.org/tensorboard?hl=zh-cn)、[Weights & Biases (WandB)](https://docs.wandb.ai/)、[MLflow](https://mlflow.org/docs/latest/index.html) 、[ClearML](https://clear.ml/docs/latest/docs) 和 [Neptune](https://docs.neptune.ai/) 实验管理工具，你可以很方便地跟踪和可视化损失及准确率等指标。 下面基于[15 分钟上手 MMENGINE](../get_started/15_minutes.md)中的例子介绍如何一行配置实验管理工具。 ## TensorBoard -设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 `TensorboardVisBackend`。 +设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [TensorboardVisBackend](mmengine.visualization.TensorboardVisBackend)。 ```python runner = Runner( @@ -32,7 +32,7 @@ pip install wandb wandb login ``` -设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 `WandbVisBackend`。 +设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [WandbVisBackend](mmengine.visualization.WandbVisBackend)。 ```python runner = Runner( @@ -81,7 +81,7 @@ pip install clearml clearml-init ``` -设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 `ClearMLVisBackend`。 +设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [ClearMLVisBackend](mmengine.visualization.ClearMLVisBackend)。 ```python runner = Runner( @@ -99,3 +99,53 @@ runner.train() ``` ![image](https://github.com/open-mmlab/mmengine/assets/58739961/d68e1dd2-9e82-40fb-ad81-00a647549adc) + +## Neptune + +使用 Neptune 前需先安装依赖库 `neptune` 并登录 [Neptune.AI](https://docs.neptune.ai/) 进行配置。 + +```bash +pip install neptune +``` + +设置 `Runner` 初始化参数中的 `visualizer`，并将 `vis_backends` 设置为 [NeptuneVisBackend](mmengine.visualization.NeptuneVisBackend)。 + +```python +runner = Runner( + model=MMResNet50(), + work_dir='./work_dir', + train_dataloader=train_dataloader, + optim_wrapper=dict(optimizer=dict(type=SGD, lr=0.001, momentum=0.9)), + train_cfg=dict(by_epoch=True, max_epochs=5, val_interval=1), + val_dataloader=val_dataloader, + val_cfg=dict(), + val_evaluator=dict(type=Accuracy), + visualizer=dict(type='Visualizer', vis_backends=[dict(type='NeptuneVisBackend')]), +) +runner.train() +``` + +![image](https://github.com/open-mmlab/mmengine/assets/58739961/9122e2ac-cc4f-43b2-bad3-ae33faa64043) + +请注意：若未提供 `project` 和 `api_token` ，neptune 将被设置成离线模式，产生的文件将保存到本地 `.neptune` 文件下。 +推荐在初始化时提供 `project` 和 `api_token` ，具体方法如下所示： + +```python +runner = Runner( + ... + visualizer=dict( + type='Visualizer', + vis_backends=[ + dict( + type='NeptuneVisBackend', + init_kwargs=dict(project='workspace-name/project-name', + api_token='your api token') + ), + ], + ), + ... +) +runner.train() +``` + +更多初始化配置参数可点击 [neptune.init_run API](https://docs.neptune.ai/api/neptune/#init_run) 查询。 diff --git a/mmengine/visualization/__init__.py b/mmengine/visualization/__init__.py index a0a518e675..9dcd772db4 100644 --- a/mmengine/visualization/__init__.py +++ b/mmengine/visualization/__init__.py @@ -1,10 +1,11 @@ # Copyright (c) OpenMMLab. All rights reserved. from .vis_backend import (BaseVisBackend, ClearMLVisBackend, LocalVisBackend, - MLflowVisBackend, TensorboardVisBackend, - WandbVisBackend) + MLflowVisBackend, NeptuneVisBackend, + TensorboardVisBackend, WandbVisBackend) from .visualizer import Visualizer __all__ = [ 'Visualizer', 'BaseVisBackend', 'LocalVisBackend', 'WandbVisBackend', - 'TensorboardVisBackend', 'MLflowVisBackend', 'ClearMLVisBackend' + 'TensorboardVisBackend', 'MLflowVisBackend', 'ClearMLVisBackend', + 'NeptuneVisBackend' ] diff --git a/mmengine/visualization/vis_backend.py b/mmengine/visualization/vis_backend.py index a1d896ea5b..48c6a90761 100644 --- a/mmengine/visualization/vis_backend.py +++ b/mmengine/visualization/vis_backend.py @@ -986,3 +986,147 @@ def close(self) -> None: for file_path in file_paths: self._task.upload_artifact(os.path.basename(file_path), file_path) self._task.close() + + +@VISBACKENDS.register_module() +class NeptuneVisBackend(BaseVisBackend): + """Neptune visualization backend class. + + Examples: + >>> from mmengine.visualization import NeptuneVisBackend + >>> from mmengine import Config + >>> import numpy as np + >>> init_kwargs = {'project': 'your_project_name'} + >>> neptune_vis_backend = NeptuneVisBackend(init_kwargs=init_kwargs) + >>> img = np.random.randint(0, 256, size=(10, 10, 3)) + >>> neptune_vis_backend.add_image('img', img) + >>> neptune_vis_backend.add_scalar('mAP', 0.6) + >>> neptune_vis_backend.add_scalars({'loss': 0.1, 'acc': 0.8}) + >>> cfg = Config(dict(a=1, b=dict(b1=[0, 1]))) + >>> neptune_vis_backend.add_config(cfg) + + Note: + `New in version 0.8.5.` + + Args: + save_dir (str, optional): The root directory to save the files + produced by the visualizer. NeptuneVisBackend does + not require this argument. Defaults to None. + init_kwargs (dict, optional): Neptune initialization parameters. + Defaults to None. + + - project (str): Name of a project in a form of + `namespace/project_name`. If `project` is not specified, + the value of `NEPTUNE_PROJECT` environment variable + will be taken. + - api_token (str): User's API token. If api_token is not api_token, + the value of `NEPTUNE_API_TOKEN` environment variable will + be taken. Note: It is strongly recommended to use + `NEPTUNE_API_TOKEN` environment variable rather than + placing your API token here. + + If 'project' and 'api_token are not specified in `init_kwargs`, + the 'mode' will be set to 'offline'. + See `neptune.init_run + `_ for + details. + """ + + def __init__(self, + save_dir: Optional[str] = None, + init_kwargs: Optional[dict] = None): + super().__init__(save_dir) # type:ignore + self._init_kwargs = init_kwargs + + def _init_env(self): + """Setup env for neptune.""" + try: + import neptune + except ImportError: + raise ImportError( + 'Please run "pip install -U neptune" to install neptune') + if self._init_kwargs is None: + self._init_kwargs = {'mode': 'offline'} + + self._neptune = neptune.init_run(**self._init_kwargs) + + @property # type: ignore + @force_init_env + def experiment(self): + """Return Neptune object.""" + return self._neptune + + @force_init_env + def add_config(self, config: Config, **kwargs) -> None: + """Record the config to neptune. + + Args: + config (Config): The Config object + """ + from neptune.types import File + self._neptune['config'].upload(File.from_content(config.pretty_text)) + + @force_init_env + def add_image(self, + name: str, + image: np.ndarray, + step: int = 0, + **kwargs) -> None: + """Record the image. + + Args: + name (str): The image identifier. + image (np.ndarray): The image to be saved. The format + should be RGB. Defaults to None. + step (int): Global step value to record. Defaults to 0. + """ + from neptune.types import File + + # values in the array need to be in the [0, 1] range + img = image.astype(np.float32) / 255.0 + self._neptune['images'].append( + File.as_image(img), name=name, step=step) + + @force_init_env + def add_scalar(self, + name: str, + value: Union[int, float], + step: int = 0, + **kwargs) -> None: + """Record the scalar. + + Args: + name (str): The scalar identifier. + value (int, float): Value to save. + step (int): Global step value to record. Defaults to 0. + """ + self._neptune[name].append(value, step=step) + + @force_init_env + def add_scalars(self, + scalar_dict: dict, + step: int = 0, + file_path: Optional[str] = None, + **kwargs) -> None: + """Record the scalars' data. + + Args: + scalar_dict (dict): Key-value pair storing the tag and + corresponding values. + step (int): Global step value to record. Defaults to 0. + file_path (str, optional): The scalar's data will be + saved to the `file_path` file at the same time + if the `file_path` parameter is specified. + Defaults to None. + """ + assert isinstance(scalar_dict, dict) + assert 'step' not in scalar_dict, 'Please set it directly ' \ + 'through the step parameter' + + for k, v in scalar_dict.items(): + self._neptune[k].append(v, step=step) + + def close(self) -> None: + """close an opened object.""" + if hasattr(self, '_neptune'): + self._neptune.stop() diff --git a/requirements/tests.txt b/requirements/tests.txt index f049bc1ecf..4084323ed8 100644 --- a/requirements/tests.txt +++ b/requirements/tests.txt @@ -4,5 +4,6 @@ dadaptation lion-pytorch lmdb mlflow +neptune parameterized pytest diff --git a/tests/test_visualizer/test_vis_backend.py b/tests/test_visualizer/test_vis_backend.py index 09abea370a..b53a67dc53 100644 --- a/tests/test_visualizer/test_vis_backend.py +++ b/tests/test_visualizer/test_vis_backend.py @@ -13,8 +13,8 @@ from mmengine.fileio import load from mmengine.registry import VISBACKENDS from mmengine.visualization import (ClearMLVisBackend, LocalVisBackend, - MLflowVisBackend, TensorboardVisBackend, - WandbVisBackend) + MLflowVisBackend, NeptuneVisBackend, + TensorboardVisBackend, WandbVisBackend) class TestLocalVisBackend: @@ -353,3 +353,41 @@ def test_close(self): clearml_vis_backend._init_env() clearml_vis_backend.add_config(cfg) clearml_vis_backend.close() + + +class TestNeptuneVisBackend: + + def test_init(self): + NeptuneVisBackend() + VISBACKENDS.build(dict(type='NeptuneVisBackend')) + + def test_experiment(self): + neptune_vis_backend = NeptuneVisBackend() + assert neptune_vis_backend.experiment == neptune_vis_backend._neptune + + def test_add_config(self): + cfg = Config(dict(a=1, b=dict(b1=[0, 1]))) + neptune_vis_backend = NeptuneVisBackend() + neptune_vis_backend.add_config(cfg) + + def test_add_image(self): + image = np.random.randint(0, 256, size=(10, 10, 3)).astype(np.uint8) + neptune_vis_backend = NeptuneVisBackend() + neptune_vis_backend.add_image('img', image) + neptune_vis_backend.add_image('img', image, step=1) + + def test_add_scalar(self): + neptune_vis_backend = NeptuneVisBackend() + neptune_vis_backend.add_scalar('map', 0.9) + neptune_vis_backend.add_scalar('map', 0.9, step=1) + neptune_vis_backend.add_scalar('map', 0.95, step=2) + + def test_add_scalars(self): + neptune_vis_backend = NeptuneVisBackend() + input_dict = {'map': 0.7, 'acc': 0.9} + neptune_vis_backend.add_scalars(input_dict) + + def test_close(self): + neptune_vis_backend = NeptuneVisBackend() + neptune_vis_backend._init_env() + neptune_vis_backend.close()