Skip to content

Commit

Permalink
Address tharvik comments
Browse files Browse the repository at this point in the history
  • Loading branch information
@JulienVig
JulienVig committed Feb 8, 2024
1 parent 63640e6 commit 08ce89a
Show file tree
Hide file tree
Showing 6 changed files with 96 additions and 148 deletions.
95 changes: 44 additions & 51 deletions DEV.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -45,13 +45,13 @@ flowchart LR

The following instructions will install the required dependencies, build Disco.js and launch a DISCO server and a web client. If you run into any sort of trouble check our [FAQ](./docs/FAQ.md); otherwise please create a new issue or feel free to ask on [our slack](https://join.slack.com/t/disco-decentralized/shared_invite/zt-fpsb7c9h-1M9hnbaSonZ7lAgJRTyNsw).

1. We recommend using [nvm](https://github.com/nvm-sh/nvm) (Node Version Manager) to handle multiple Node.js versions. Start by installing `nvm` by following [their installation instructions](https://github.com/nvm-sh/nvm).
**1.** We recommend using [nvm](https://github.com/nvm-sh/nvm) (Node Version Manager) to handle multiple Node.js versions. Start by installing `nvm` by following [their installation instructions](https://github.com/nvm-sh/nvm).
After installation, you should be able to run
```
nvm -v
0.39.7 # my nvm version at the time
```
2. Install Node.js version 16
**2.** Install Node.js version 16
```
nvm install 16
```
Expand All @@ -66,64 +66,20 @@ npm --version
```
`nvm` manages your different Node.js versions while `npm` handles your different Node.js project packages within one version.

3. Clone the repository
**3.** Clone the repository
```
git clone [email protected]:epfml/disco.git
cd disco
```

4. Run the installation script
**4.** Run the installation script
```
sh install.sh
```
The [Installation Script](#installation-script) Section goes over what <code>install.sh</code> does. If you are planning to contribute to DISCO, it will certainly prove useful to understand what the script does.

**5.** Launch DISCO

<details>
<summary><b>What does <code>install.sh</code> do?</b></summary>
</br>
The installation script installs the dependencies required by the different parts of the project, which are described in the Structure section.
It first installs the Disco.js library dependencies, notably, `TensorFlow.js`, and anything else required for federated and decentralized learning logic.
The script then builds the library, a step necessary to compile TypeScript into JavaScript.

```
cd discojs
npm ci # stands for `clean install`, to ensure than only expected dependencies are being installed.
npm run build
```
The script then installs dependencies for the web client, which implements a browser UI.
By default, the project points to the [@epfml/disco-web](https://www.npmjs.com/package/@epfml/discojs) package published on the `npm` remote repository. In a development environment, we want to use the local web client in the `discojs/web-client` folder. To do so, we need to link the local folder as the actual dependency.

```
cd ../web-client
npm ci
npm link ../discojs/discojs-web
```
You can verify than the link is effective by checking that `npm ls` lists `@epfml/[email protected] -> ./../discojs/discojs-web`.

Similarly, we install the server dependencies, and then the `discojs-node` dependency to the local folder rather than the remote npm package [@epfml/disco-node](https://www.npmjs.com/package/@epfml/discojs-node):
```
cd ../server
npm ci
npm link ../discojs/discojs-node
```
Install the CLI dependencies:
```
cd ../cli
npm ci
```
Install `ts-node` globally. `ts-node` lets us compile and run TypeScript code in a single command.
```
cd ..
npm install -g ts-node
```
Download and extract the sample training datasets. These datasets are used in the automated tests.
```
sh get_training_data.sh
```

</details>

5. Launch DISCO

As you may have seen, there are many ways to use DISCO. Here we will run a server and a web client. From there, a user can use DISCO from their browser.
* First launch a `server` instance, which is used for federated and decentralized learning tasks, e.g. to list peers participating in a decentralized task.
```
Expand Down Expand Up @@ -170,3 +126,40 @@ As there are many guides in the project, here is a table of contents referencing
* [`server` README](./server/README.md)
* [`web-client` README](./web-client/README.md)
* [`cli` README](./cli/README.md)

## Installation Script

The installation script installs the dependencies required by the different parts of the project, which are described in the [Structure Section](#structure).
It first installs the Disco.js library dependencies, notably, `TensorFlow.js`, and anything else required for federated and decentralized learning logic.
The script then builds the library, a step necessary to compile TypeScript into JavaScript.

```
cd discojs
npm ci # stands for `clean install`, to ensure that only expected dependencies are being installed.
npm run build
```
The script then installs dependencies for the web client, which implements a browser UI.
By default, the project points to the [@epfml/disco-web](https://www.npmjs.com/package/@epfml/discojs) package published on the `npm` remote repository. In a development environment, we want to use the local web client in the `web-client` folder. To do so, we need to link the local folder as the actual dependency.

```
cd ../web-client
npm ci
npm link ../discojs/discojs-web
```
You can verify than the link is effective by checking that `npm ls` lists `@epfml/[email protected] -> ./../discojs/discojs-web`.

Similarly, we install the server dependencies, and then the `discojs-node` dependency to the local folder rather than the remote npm package [@epfml/disco-node](https://www.npmjs.com/package/@epfml/discojs-node):
```
cd ../server
npm ci
npm link ../discojs/discojs-node
```
Install the CLI dependencies:
```
cd ../cli
npm ci
```
Download and extract the sample training datasets. These datasets are used in the automated tests.
```
./get_training_data.sh
```
11 changes: 6 additions & 5 deletions discojs/test.sh
View file
Original file line number Diff line number Diff line change
@@ -1,20 +1,21 @@
#!/bin/sh
set -e # Exit script on error
DIR="$( cd "$( dirname "$0" )" ; pwd -P )" # Fetch current directory

echo '\n>>> Make sure a server instance is running and reachable at http://localhost:8080! <<<'
printf '\n>>> Make sure a server instance is running and reachable at http://localhost:8080! <<<'

echo '\n>>> Rebuilding discojs\n'
printf '\n>>> Rebuilding discojs\n'
cd $DIR
npm run build

echo '\n>>> Testing disco-core\n'
printf '\n>>> Testing disco-core\n'
cd $DIR/discojs-core
npm run test

echo '\n>>> Testing disco-node\n'
printf '\n>>> Testing disco-node\n'
cd $DIR/discojs-node
npm run test

echo '\n>>> Testing disco-web\n'
printf '\n>>> Testing disco-web\n'
cd $DIR/discojs-web
npm run test
43 changes: 33 additions & 10 deletions docs/ONBOARDING.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,8 @@ Disco has grown a lot since its early days, and like any sizeable code base, get
difficult and intimidating: there are a *lot* of files, it's not clear what's important at first, and even where to start
is a bit of a puzzle. This document aims at giving you an efficient process to get familiar with DISCO.

The two main technologies behind Disco are TypeScript and distributed machine learning. In the following sections I will assume that you are familiar
with both to some extent. If not, the following references might be useful:
The two main technologies behind DISCO are TypeScript and distributed machine learning. In the following sections I will assume that you are familiar
with both to a certain extent. If not, the following references might be useful:

- [JavaScript](https://eloquentjavascript.net)
- [TypeScript](https://www.typescriptlang.org/docs/handbook/intro.html)
Expand All @@ -21,16 +21,44 @@ a `server` and a `cli` (e.g., for benchmarking). Depending on what your goal is,

1. If you are going to work, contribute and improve the project, I first recommend you get a good understand of what DISCO does: play around with the [website](https://epfml.github.io/disco/#/), train a model from the pre-defined tasks, or even create your own custom task. Feedback is always appreciated, feel free to let us know on slack/in the github issues/in person if you noticed any issues or thought of an improvement.

2. Then, get a high-level understanding of the different parts of the projects in the [developer guide](../DEV.md), even if you're planning on working on a subset of the project. If you to know more about a specific part of the project, refer to the table of contents at the end of the guide.
2. Then, get a high-level understanding of the different parts of the projects in the [developer guide](../DEV.md), even if you're planning on working on a subset of the project. If you want to know more about a specific part of the project, refer to the table of contents at the end of the DEV guide.

3. Following the installation instructions from the [developer guide](../DEV.md) to launch a DISCO instance working in your browser.
3. Follow the installation instructions from the [developer guide](../DEV.md) to launch a DISCO instance working in your browser.

> [!TIP]
> The most common issues with running DISCO are usually due to using old Node.js versions and setting the appropriate environment on M1 Macs, see [our FAQ](./FAQ.md) for more troubleshooting. Note that DISCO has been not tested on Windows (only Linux and macOS).
As mentioned in the [developer guide](../DEV.md), there are many ways to use Disco.js: from a browser, a CLI, by importing `discojs-node` in your own Node.js scripts and applications, from your own UI implementation, etc. Note that whatever your setting, using Disco.js always requires running a `server` instance. As described in the [`server` README file](../server/REDME.md), the server is in charge of connecting peers to the ML tasks. In order to connect and partake in a distributed training session you first need to find the session and how to join it, the sever exposes an API to that end.

Below you will find instructions and documentation on how to run DISCO in different settings.
### Things to know

As a contributor, you will certainly end up having to run TypeScript scripts. A practical way to do so is to use on ts-node:
```
npm i -g ts-node # globally to run scripts from anywhere
ts-node your_script.ts
```

Because TypeScript needs to be transpiled to JavaScript, you need to rebuild the `discojs` folder every time you make any changes to it:
``` js
cd discojs
npm run build
```

When cloning the repo, the server and the web-client points to the [@epfml/disco-node](https://www.npmjs.com/package/@epfml/discojs-node) and the [@epfml/disco-web](https://www.npmjs.com/package/@epfml/discojs) packages respectively published on the `npm` remote repository. In a development environment, we want to use the local implementations in the `discojs/discojs-node` and `discojs/discojs-web` folders for changes to take effect immediately. To do so, we need to link the local folders as the actual dependencies:
```
cd server
npm link ../discojs/discojs-node
cd ../web-client
npm link ../discojs/discojs-web
```
You can verify than the links are effective by checking that running `npm ls` from the `server` folder lists `@epfml/[email protected] -> ./../discojs/discojs-node` in server and `@epfml/[email protected] -> ./../discojs/discojs-web` in the web-client.

> [!TIP]
> If you are using VSCode, know that you may not be able to open the editor from the repo root level without VSCode raising imports errors. If that is the case, you should start VSCode from inside the module you are working.
> In practice, that is any folder level that contains a `package.json` such as `server`, `web-client`, etc.
> For example, if you are working on the CLI, you should start VSCode with the command `code server` from the root level (or `cd server; code .`)
Next you will find instructions and documentation on how to run DISCO in different settings.

### Using DISCO from the `web-client`

Expand All @@ -44,11 +72,6 @@ The CLI is currently not working. Until then, you can find more information in t

A standalone example of disco can be found [in this folder](./node_example), with code and documentation.

> [!TIP]
> If you are using VSCode, know that you may not be able to open the editor from the repo root level without VSCode raising imports errors. If that is the case, you should start VSCode from inside the module you are working.
> In practice, that is any folder level that contains a `package.json` such as `server`, `web-client`, etc.
> For example, if you are working on the CLI, you should start VSCode with the command `code cli` from the root level (or `cd cli`, `code .`)
### Next steps

1. If you are planning to contribute to the project you should read the [contributing guide](./CONTRIBUTING.md)
Expand Down
78 changes: 4 additions & 74 deletions docs/TASK.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,9 @@ to use preprocessing functions to resize the data (we also describe how to add c
The `Task` class contains all the crucial information for training the model (batchSize, learningRate, ...) and also the
scheme of distributed learning (federated or decentralized), along with other meta data about the model and data.
> In the appendix (end of this document) you find all possible [`TrainingInformation`](../discojs/discojs-core/src/task/training_information.ts) parameters with a short description.
The [`TrainingInformation` object](../discojs/src/task/training_information.ts) of a task contains all the customizable parameters and their descriptions.
As an example, the task class for `simple-face` can be found [here](../discojs/discojs-core/src/default_tasks/simple_face.ts), suppose
As an example, the task class for `simple-face` can be found [here](../discojs/discojs-core/src/default_tasks/simple_face.ts). Suppose
our own task is a binary classification for age detection (similar to simple face), then we could write:
Expand Down Expand Up @@ -250,6 +250,7 @@ export const task: Task = {
}
```
> [!TIP]
> Note that you need to rebuild discojs every time you make changes to it (`cd discojs; rm -rf dist/; npm run build`).
## Summary
Expand All @@ -264,75 +265,4 @@ export const task: Task = {
Your task has been successfully uploaded.
**Or** just use the NPM `disco-server` package and add your own custom `TaskProvider` directly to the server.
## Appendix
The [`TrainingInformation`](../discojs/src/task/training_information.ts) of a task contains the following customizable parameters
```js
export interface TrainingInformation {
// modelID: unique ID for the model
modelID: string
// epochs: number of epochs to run training for
epochs: number
// roundDuration: number of batches between each weight sharing round, e.g. if 3 then after every
// 3 batches we share weights (in the distributed setting).
roundDuration: number
// validationSplit: fraction of data to keep for validation, note this only works for image data
validationSplit: number
// batchSize: batch size of training data
batchSize: number
// preprocessingFunctions: preprocessing functions such as resize and normalize
preprocessingFunctions?: Preprocessing[]
// modelCompileData: interface of additional training information (optimizer, loss and metrics)
modelCompileData: ModelCompileData
// dataType, e.g. image or tabular
dataType: string
// inputColumns: for tabular data, the columns to be chosen as input data for the model
inputColumns?: string[]
// outputColumns: for tabular data, the columns to be predicted by the model
outputColumns?: string[]
// IMAGE_H height of image (or RESIZED_IMAGE_H if ImagePreprocessing.Resize in preprocessingFunctions)
IMAGE_H?: number
// IMAGE_W width of image (or RESIZED_IMAGE_W if ImagePreprocessing.Resize in preprocessingFunctions)
IMAGE_W?: number
// Model URL to download the base task model from. Useful for pretrained or/and hosted models.
modelURL?: string
// LABEL_LIST of classes, e.g. if two class of images, one with dogs and one with cats, then we would
// define ['dogs', 'cats'].
LABEL_LIST?: string[]
// learningRate: learning rate for the optimizer
learningRate?: number
// scheme: Distributed training scheme, i.e. Federated and Decentralized
scheme: string
// noiseScale: Differential Privacy (DP): Affects the variance of the Gaussian noise added to the models / model updates.
// Number or undefined. If undefined, then no noise will be added.
noiseScale?: number
// clippingRadius: Privacy (DP and Secure Aggregation):
// Number or undefined. If undefined, then no model updates will be clipped.
// If number, then model updates will be scaled down if their norm exceeds clippingRadius.
clippingRadius?: number
// decentralizedSecure: Secure Aggregation on/off:
// Boolean. true for secure aggregation to be used, if the training scheme is decentralized, false otherwise
decentralizedSecure?: boolean
// byzantineRobustAggregator: Byzantine robust aggregator on/off:
// Boolean. true to use byzantine robust aggregation, if the training scheme is federated, false otherwise
byzantineRobustAggregator?: boolean
// tauPercentile: it indicates the percentile to take when choosing the tau for byzantine robust aggregator:
// Number (>0 && <1). It must be a number between 0 and 1 and it is used only if byzantineRobustAggregator is true.
tauPercentile?: number
// maxShareValue: Secure Aggregation: maximum absolute value of a number in a randomly generated share
// default is 100, must be a positive number, check the ~/disco/information/PRIVACY.md file for more information on significance of maxShareValue selection
// only relevant if secure aggregation is true (for either federated or decentralized learning)
maxShareValue?: number
// minimumReadyPeers: Decentralized Learning: minimum number of peers who must be ready to participate in aggregation before model updates are shared between clients
// default is 3, range is [3, totalNumberOfPeersParticipating]
minimumReadyPeers?: number
// aggregator: aggregator to be used by the server for federated learning, or by the peers for decentralized learning
// default is 'average', other options include for instance 'bandit'
aggregator?: AggregatorChoice
}
}
```
**Or** just use the NPM `disco-server` package and add your own custom `TaskProvider` directly to the server.
1 change: 1 addition & 0 deletions get_training_data → get_training_data.sh
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,4 @@
#!/bin/sh
DIR="$( cd "$( dirname "$0" )" ; pwd -P )"
ARCHIVE="example_training_data.tar.gz"

Expand Down
16 changes: 8 additions & 8 deletions install.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,29 +2,29 @@
set -e # Exit script on error
DIR="$( cd "$( dirname "$0" )" ; pwd -P )" # Fetch current directory

echo '\n>>> Installing disco.js dependencies and building the library\n'
printf '\n>>> Installing disco.js dependencies and building the library\n'
cd $DIR/discojs
npm ci
npm run build

echo '\n>>> Installing the web client dependencies\n'
printf '\n>>> Installing the web client dependencies\n'
cd $DIR/web-client
npm ci
npm link $DIR/discojs/discojs-web

echo '\n>>> Installing the server dependencies\n'
printf '\n>>> Installing the server dependencies\n'
cd $DIR/server
npm ci
npm link $DIR/discojs/discojs-node

echo '\n>>> Installing the CLI dependencies\n'
printf '\n>>> Installing the CLI dependencies\n'
cd $DIR/cli
npm ci

echo '\n>>> Installing ts-node\n'
printf '\n>>> Installing ts-node\n'
npm install -g ts-node

echo '\n>>> Downloading and extracting sample training data\n'
sh get_training_data
printf '\n>>> Downloading and extracting sample training data\n'
./get_training_data.sh

echo '\n>>> Installation sucessful!\n'
printf '\n>>> Installation sucessful!\n'

0 comments on commit 08ce89a

Please sign in to comment.