From d48cf85d3564087e3006adf142b0f61c2f22ee26 Mon Sep 17 00:00:00 2001 From: Thorsten Beier Date: Fri, 5 Jan 2024 11:00:12 +0100 Subject: [PATCH] Readme (#16) * added example environments * more options * readme --- README.md | 157 +++++++++++++++++------- environment-dev.yaml | 21 ++++ example_envs/env_with_extension.yaml | 2 +- example_envs/env_with_many_kernels.yaml | 2 +- example_envs/env_with_pip.yaml | 2 +- 5 files changed, 139 insertions(+), 45 deletions(-) create mode 100644 environment-dev.yaml diff --git a/README.md b/README.md index bc0659b..9b8c8a6 100644 --- a/README.md +++ b/README.md @@ -15,81 +15,154 @@ To install the extension, execute: pip install jupyterlite_xeus ``` -## Uninstall +## Usage -To remove the extension, execute: +### From environment.yaml -```bash -pip uninstall jupyterlite_xeus +#### xeus-python kernel +To create a `xeus-python` kernel with a custom environment, one creates an `environment.yaml` file with `xeus-python` and the desiered dependencies. Here is an example with `numpy` as a additional dependency: + +```yaml +name: xeus-lite-wasm +channels: + - https://repo.mamba.pm/emscripten-forge + - conda-forge +dependencies: + - xeus-python + - numpy ``` +To build the jupyterlite run the following command, where `environment.yaml` is the path to the file you just created -## Contributing -### Development install +```bash +jupyter lite build --XeusAddon.environment_file=some_path/to/environment.yaml +``` + +#### xeus-lua / xeus-sqlite / xeus-\ -Note: You will need NodeJS to build the extension package. +To create a `xeus-lua` or `xeus-sqlite` kernel you can +do the same as above, just with +```yaml +dependencies: + - xeus-lua +``` +or +```yaml +dependencies: + - xeus-sqlite +``` -The `jlpm` command is JupyterLab's pinned version of -[yarn](https://yarnpkg.com/) that is installed with JupyterLab. You may use -`yarn` or `npm` in lieu of `jlpm` below. +Note that xeus-sqlite and xeus-lua do not yet support additional dependencies. +To build the jupyterlite, run again: ```bash -# Clone the repo to your local environment -# Change directory to the jupyterlite_xeus directory -# Install package in development mode -pip install -e "." -# Link your development version of the extension with JupyterLab -jupyter labextension develop . --overwrite -# Rebuild extension Typescript source after making changes -jlpm build +jupyter lite build --XeusAddon.environment_file=environment.yaml ``` -You can watch the source directory and run JupyterLab at the same time in different terminals to watch for changes in the extension's source and automatically rebuild the extension. + +#### Multiple kernels + +To create a deployment with multiple kernels, you can just add them to the `environment.yaml` file: + +```yaml +name: xeus-lite-wasm +channels: + - https://repo.mamba.pm/emscripten-forge + - conda-forge +dependencies: + - xeus-python + - xeus-lua + - xeus-sqlite + - numpy +``` + +### From local environment / prefix + +When developing a xeus-kernel, it is very usefull to be able to test it in a jupyterlite without having to publish it to emscripten-forge. Therefore you can also use a local environment / prefix to build a jupyterlite with a custom kernel. + + +#### Create a local environment / prefix + +This workflow usually starts with creating a local conda environment / prefix for `emscripten-wasm32` with all the dependencies you need to build your kernel (here we install dependencies for `xeus-python` ) ```bash -# Watch the source directory in one terminal, automatically rebuilding when needed -jlpm watch -# Run JupyterLab in another terminal -jupyter lab +micromamba create -n xeus-python-dev \ + --platform=emscripten-wasm32 \ + -c https://repo.mamba.pm/emscripten-forge \ + -c https://repo.mamba.pm/conda-forge \ + --yes \ + "python>=3.11" pybind11 nlohmann_json pybind11_json numpy pytest \ + bzip2 sqlite zlib libffi xtl pyjs \ + xeus xeus-sqlite ``` +#### Build the kernel -With the watch command running, every saved change will immediately be built locally and available in your running JupyterLab. Refresh JupyterLab to load the change in your browser (you may need to wait several seconds for the extension to be rebuilt). +This dependes on your kernel but will look smth like this: -By default, the `jlpm build` command generates the source maps for this extension to make it easier to debug using the browser dev tools. To also generate source maps for the JupyterLab core extensions, you can run the following command: +```bash +# path to your emscripten emsdk +source $EMSDK_DIR/emsdk_env.sh + +WASM_ENV_NAME=xeus-python-dev +WASM_ENV_PREFIX=$MAMBA_ROOT_PREFIX/envs/$WASM_ENV_NAME + +# let cmake know where the env is +export PREFIX=$WASM_ENV_PREFIX +export CMAKE_PREFIX_PATH=$PREFIX +export CMAKE_SYSTEM_PREFIX_PATH=$PREFIX + +cd /path/to/your/kernel/src +mkdir build_wasm +cd build_wasm +emcmake cmake \ + -DCMAKE_BUILD_TYPE=Release \ + -DCMAKE_FIND_ROOT_PATH_MODE_PACKAGE=ON \ + -DCMAKE_INSTALL_PREFIX=$PREFIX + .. +emmake make -j8 install +``` +When running `jupyter lite build` we pass the `prefix` options and point it to the local environment / prefix we just created: ```bash -jupyter lab build --minimize=False +jupyter lite build --XeusAddon.prefix=$WASM_ENV_PREFIX ``` -### Development uninstall +### Mounting additional files + +To copy additional files and directories into the virtual filesytem of the xeus-lite kernels you can use the `--XeusAddon.mount` option. +Each mount is specified as a pair of paths separated by a colon `:`. The first path is the path to the file or directory on the host machine, the second path is the path to the file or directory in the virtual filesystem of the kernel. ```bash -pip uninstall jupyterlite_xeus +jupyter lite build \ + --XeusAddon.environment_file=environment.yaml \ + --XeusAddon.mounts=/some/path/on/host_machine:/some/path/in/virtual/filesystem ``` -In development mode, you will also need to remove the symlink created by `jupyter labextension develop` -command. To find its location, you can run `jupyter labextension list` to figure out where the `labextensions` -folder is located. Then you can remove the symlink named `@jupyterlite/xeus` within that folder. -### Testing the extension -#### Frontend tests -This extension is using [Jest](https://jestjs.io/) for JavaScript code testing. +## Contributing + +### Development install from a conda / mamba environment + +Create the conda environment with `conda`/`mamba`/`micromamba` (replace `micromamba` with `conda` or `mamba` according to your preference): -To execute them, execute: -```sh -jlpm -jlpm test +```bash +micromamba create -f environment-dev.yml -n xeus-lite-dev ``` -#### Integration tests +Activate the environment: + +```bash +micromamba activate xeus-lite-dev +``` -This extension uses [Playwright](https://playwright.dev/docs/intro) for the integration tests (aka user level tests). -More precisely, the JupyterLab helper [Galata](https://github.com/jupyterlab/jupyterlab/tree/master/galata) is used to handle testing the extension in JupyterLab. -More information are provided within the [ui-tests](./ui-tests/README.md) README. +```bash +python -m pip install -e . -v --no-build-isolation +``` + ### Packaging the extension diff --git a/environment-dev.yaml b/environment-dev.yaml new file mode 100644 index 0000000..5c819f5 --- /dev/null +++ b/environment-dev.yaml @@ -0,0 +1,21 @@ +name: xeus-lite-dev +channels: + - conda-forge +dependencies: + - black + - build + - empack >=3.3 + - hatch-jupyter-builder + - hatch-nodejs-version + - jupyterlab >=4.0 + - jupyterlite-core + - mamba + - nodejs + - pip + - pytest + - python + - requests + - ruff + - traitlets + - typer + - yarn \ No newline at end of file diff --git a/example_envs/env_with_extension.yaml b/example_envs/env_with_extension.yaml index 13be59c..feea13a 100644 --- a/example_envs/env_with_extension.yaml +++ b/example_envs/env_with_extension.yaml @@ -1,4 +1,4 @@ -name: xeus-lite +name: xeus-lite-wasm channels: - https://repo.mamba.pm/emscripten-forge - conda-forge diff --git a/example_envs/env_with_many_kernels.yaml b/example_envs/env_with_many_kernels.yaml index b627f10..c1c7b9a 100644 --- a/example_envs/env_with_many_kernels.yaml +++ b/example_envs/env_with_many_kernels.yaml @@ -1,4 +1,4 @@ -name: xeus-lite +name: xeus-lite-wasm channels: - https://repo.mamba.pm/emscripten-forge - conda-forge diff --git a/example_envs/env_with_pip.yaml b/example_envs/env_with_pip.yaml index dc01ea4..56faac7 100644 --- a/example_envs/env_with_pip.yaml +++ b/example_envs/env_with_pip.yaml @@ -1,4 +1,4 @@ -name: xeus-lite +name: xeus-lite-wasm channels: - https://repo.mamba.pm/emscripten-forge - conda-forge