Dmkats/docs

ADBench/run-all.ps1

@@ -7,18 +7,18 @@ This is a PowerShell script to run all autodiff benchmarking tests.
 This script loops through each of a set of tools (defined using the Tool class) and runs a set of test functions on each of them.
 
 .EXAMPLE
-./run-all.ps1 -buildtype "Release" -minimum_measurable_time 0.5 -nruns_f 10 -nruns_J 10 -time_limit 180 -timeout 600 -tmpdir "C:/path/to/tmp/" -repeat $FALSE  -tools (echo Finite Manual Julia) -gmm_d_vals_param @(2,10)
+./run-all.ps1 -buildtype "Release" -minimum_measurable_time 0.5 -nruns_f 10 -nruns_J 10 -time_limit 180 -timeout 600 -tmpdir "C:/path/to/tmp/" -tools (echo Finite Manual Julia) -gmm_d_vals_param @(2,5,10,64)
 
 This will:
 - run only release builds
-- loop measured function while total calculation time less than 0.5 seconds
+- loop measured function while total calculation time is less than 0.5 seconds
 - aim to run 10 tests of each function, and 10 tests of the derivative of each function
 - stop (having completed a whole number of tests) at any point after 180 seconds
 - allow each program a maximum of 600 seconds to run all tests
 - output results to "C:/path/to/tmp/"
 - not repeat any tests for which there already exist a results file
 - run only Finite, Manual, and Julia
-- try GMM d values of 2,10
+- try GMM d values of 2, 5, 10, 64
 
 .NOTES
 See below for adding new tools or tests.
@@ -37,7 +37,8 @@ param(# Which build to test.
 
       # Estimated time of accurate result achievement. 
       # A runner cyclically reruns measured function until total time becomes more than that value. 
-      # Currently not supported by all runners. 
+      # Supported only by the benchmark runner-based tools
+      # (those with ToolType cpp, dotnet, julia, or python).
       [double]$minimum_measurable_time = 0.5,
 
       # Maximum number of times to run the function for timing                

CONTRIBUTING.md

@@ -1,42 +1,6 @@
 # Contributing
 
-Here, 'tool' refers to an Automatic Differentiation tool to be benchmarked, and 'test' (or 'objective') refers to a function being tested.
-
-## Adding new tools
-
-1. Create a folder in `/tools` for your tool
-2. Add the tool as a dependency. There are various way to do this (listed in descending order of preference):
-    - If using a Python module, add it to the list of `pip` (or `conda` if necessary) requirements in [README.md](/README.md)
-    - If available as a [Hunter package](https://docs.hunter.sh/en/latest/packages.html), add that in the relevant CMakeLists.txt file
-    - If available through `git`, add it as a submodule in `/submodules`
-    - If available through NuGet, build your tool using `dotnet` and add it as a dependency (see DiffSharp as an example of this)
-3. Add the tool, with objectives
-    - `{tool name}` should match the name of the folder you have created
-    - `{test name}` should include additional details such as `GMM-FULL` or `Hand-Eigen-simple`. See the existing tools as examples to ensure this is done correctly.
-    - For compiled languages:
-        - Create a main/program file as the entry-point for all tests
-        - If appropriate, create separate files with functions for each test
-        - Create a `CMakeLists.txt` file in your folder. If in a language which CMake builds, add it to the CMake build list here. If not, tell CMake to run any necessary build scripts using `execute_process`.
-        - Executables should be built (in `${CMAKE_CURRENT_BINARY_DIR}`) for each test (i.e. using different `#defines`). They should be named in the format `Tools-{tool name (as in the folder name)}-{test name (uppercase)}.{ext}`
-        - Add your folder as a subdirectory in `/tools/CMakeLists.txt`
-    - For interpreted languages:
-        - There should be a script for each test to run
-        - Scripts should be named in the format `'{tool name (as in the folder name)}_{test name (lowercase)}.{ext}'`
-        - For examples, see the Python and Matlab tools
-4. Add to `run-all.ps1`
-    - Add it to the `$tools` array using the `[Tool]` class
-5. Run tests on your tool and re-generate graphs
-
-## Adding new tests
-
-1. Add the test to all tools where possible
-2. Add the test to `run-all.ps1`
-    1. Create a new method on the `[Tool]` class (format should be `[Tool]::test{your test name}()`)
-    2. Add this to `[Tool]::runall()`
-    3. Modify the `$objectives` parameter for each of the `[Tool]` constructors
-        - Append a `1` for any tools for which the test is implemented
-        - Append a `0` where the test is not yet implemented
-3. Run your test and generate graphs for it
+If you want to benchmark a new Automatic Differentiation tool or add a new objective function as another benchmark, please, refer to our [development guide](docs/Development.md).
 
 # Code of Conduct
 

ISSUES.md

@@ -20,9 +20,9 @@ This is a list of known issues/TODOs for the project. Any fixes are welcomed.
 ## Tools
 
 - Test building and running on non-windows platforms
-    - Will need to produce alternatives to existing batch files (currently in ADOL-C, DiffSharp, MuPad, Theano)
+    - Will need to produce alternatives to existing batch files (currently in ADOL-C, MuPad, Theano)
 - Matlab tools need testing properly
-    - `run-all.ps1` has Matlab tools commented out due to an issue with the trial version - it should be possible to remove these comments and run them with the full version
+    - `run-all.ps1` has Matlab tools commented out by default.  If Matlab is available on your system, uncomment.
 - ADiMat
     - `ADiMat_ba.m` - test seems to crash during Jacobian loop
 - ADOLC
@@ -31,21 +31,8 @@ This is a list of known issues/TODOs for the project. Any fixes are welcomed.
         - Try to automate Windows SDK selection, and/or check properly for correct SDK and give clear error message
     - `main.cpp` - build fails with `DO_BA_BLOCK` and `DO_EIGEN`
 - Ceres
-    - Memory issue when building GMM at instance sizes above about d=20,K=50
-- DiffSharp
-    - GMM build fails (expected float[] but got D[])
-    - BA is very slow (may just be slow, but could be an error)
-- Manual
-    - `ba_d.cpp` - refactor functions under `DO_CPP` to work without Eigen
-    - Perhaps add Hand derivative without Eigen
+    - Memory issue when building GMM at instance sizes above about d=20, K=50
 - MuPad
     - Missing mex files for MuPad_Hand
-- PyTorch
-    - BA times out (as with DiffSharp, may just be slow or may be an error)
 - Theano
     - `run.bat` - remove hard path to Miniconda
-
-
-## Accuracy
-
-Use `ADBench/check_J.py` to compare the jacobian output files. There seem to be some problems with mismatches here.

README.md

@@ -2,152 +2,44 @@
 
 This project aims to provide a running-time comparison for different tools for automatic differentiation, 
 as described in https://arxiv.org/abs/1807.10129, (source in [Documentation/ms.tex](Documentation/ms.tex)).
-It outputs a set of relevant graphs (see [Example Graphs](#Example%20Graphs)).
+It outputs a set of relevant graphs (see [Graph Archive](#graph-archive)).
 
-For information about the layout and status of the project, see [Status](STATUS.md).
+For information about the layout of the project, see [Development](docs/Development.md#structure-of-the-repository).
 
-## Prerequisites
+For information about the current status of the project, see [Status](/STATUS.md).
 
-- [CMake](https://cmake.org/) (see [Installation](#installation) for details)
-- [.NET](https://www.microsoft.com/net) (should be in PATH)
-- [FSharp](https://fsharp.org/)
-- [Matlab](https://www.mathworks.com/products/matlab.html) (in PATH)
-	- **Note:** while the free trial version works, the obtrusive login dialog makes it impossible to automatically run all matlab tests without manually logging in each time
-- [Python](https://www.python.org/) (in PATH), with the following `pip` modules:
-	- numpy
-	- scipy
-	- matplotlib
-	- plotly
-	- autograd
-	- PyTorch (install from [here](https://pytorch.org/))
-- [Miniconda](https://conda.io/miniconda.html) (only required to run Theano)
-	- Once installed, open the Anaconda Prompt as an administrator and run:
-	- `conda install numpy scipy mkl-service libpython m2w64-toolchain`
-	- `conda install theano pygpu`
-- [Powershell](https://docs.microsoft.com/en-us/powershell/scripting/setup/installing-powershell
-) (default on Windows)
+## Methodology
 
-## Installation
+For explanations on how do we perform the benchmarking see [Benchmarking Methodology](docs/Methodology.md), [Jacobian Correctness Verification](docs/JacobianCheck.md).
 
-All tools should build (along with any external packages) and run very easily, provided you are able to use CMake. CMake should always work similarly, but the easiest methods may vary by operating system.
+## Build and Run
 
-The basic steps (more specific instructions for Windows are below):
+The easiest way to build and run the benchmarks is to [use Docker](docs/Docker.md). If that doesn't work for you, please, refer to our [build and run guide](docs/BuildAndTest.md).
 
-1) Clone the repository (make sure submodules are cloned properly)
-2) Optional: set HUNTER_ROOT environment variable to choose where HunterGate pacakges are stored.  If not set, will default to etc/HunterGate-Root.
-3) Run cmake
-	- mkdir my-build  # Directories with name *-build/ are ignored in .gitignore
-	- cd my-build
-	- cmake ..
-	- If you get an error of the form "The input line is too long.  The syntax of the command is incorrect.", your $ENV:PATH may be too long (!)
+## Plot Results
 
-3) Build
+Use `ADBench/plot_graphs.py` script to plot graphs of the resulting timings.
 
-### Windows
-
-The easiest way to build on windows is using [CMake tools for Visual Studio](https://blogs.msdn.microsoft.com/vcblog/2016/10/05/cmake-support-in-visual-studio/), which can be installed as a component in the [Visual Studio Installer](https://docs.microsoft.com/en-us/visualstudio/install/install-visual-studio).
-
-1) Open Visual Studio
-2) [Clone](http://www.malgreve.net/2014/06/17/cloning-getting-code-from-git-repository-to-visual-studio/) this repository
-3) Open the cloned folder in Visual Studio. CMake should run automatically, but it may need to be started manually.
-	- **NOTE:** CMake (specifically HunterGate) often seems to crash the first time it is run, so try re-running several times.
-	- **NOTE:** HunterGate requires that the path to the folder does not contain spaces. This can be disabled but is not currently.
-4) When CMake has finished, run CMake>Build All
-	- **NOTE:** this is sometimes not available as an option. Try restarting Visual Studio or waiting a while.
-
-All tools should now be built. See [Usage](#usage) below.
-
-#### Windows from the command line
-
-Instead of using Visual Studio you can execute the following command
-
-```
-cmake -G "Ninja" '-DCMAKE_TOOLCHAIN_FILE=<path-to-top-level>\toolchain.cmake' '-DCMAKE_BUILD_TYPE="RelWithDebInfo"' "<path-to-top-level>"
-ninja
+```bash
+python ADBench/plot_graphs.py --save
 ```
+This will save graphs as .png files to `tmp/graphs/static/`
 
-You have to somehow ensure that `cmake`, `cl` and `ninja` are on your
-path.
+Refer to [PlotCreating](docs/PlotCreating.md) for other possible command line arguments and the complete documentation.
 
-Alternatively, you can build with MSBuild
-
-```
-cd [directory containing this README]
-mkdir my-build
-cd my-build
-cmake .. -G 'Visual Studio 15 2017 Win64'
-MSBuild.exe .\ADBench.sln '/p:Configuration=Release'
-..\ADBench\run-all.ps1
-```
-
-### Windows on a Visual Studio VM
-
-Very brief instructions
-
-```
-[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
-Invoke-WebRequest -Uri "https://www.python.org/ftp/python/3.7.1/python-3.7.1-amd64.exe" -OutFile install-python.exe
-Invoke-WebRequest -Uri "https://github.com/ninja-build/ninja/releases/download/v1.8.2/ninja-win.zip" -OutFile ninja-win.zip
+## Graph Archive
 
-git clone https://github.com/awf/ADBench
+From time to time we run the benchmarks and publish the resulting plots here:
+https://adbenchwebviewer.azurewebsites.net/
 
-$env:path="$env:path;C:\Users\toelli\AppData\Local\Programs\Python\Python37"
-$env:path="$env:path;C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\Common7\IDE\CommonExtensions\Microsoft\CMake\CMake\bin"
-$env:build_sourcesdirectory="C:\Users\toelli\ADBench"
-$env:path="$env:path;C:\Users\toelli\ninja-win"
-
-git submodule init
-git submodule update
-```
-
-## Usage
-
-1) Run `powershell ADBench/run-all.ps1` to run all of the tools and write the timings to `/tmp/`.
-2) Run `python ADBench/plot_graphs.py` to plot graphs of the resulting timings and write them to `/Documents/New Figures/`
-
-### CLI reference: run-all
-
-`powershell ADBench/run-all.ps1 buildtype nruns_f nruns_J time_limit tmpdir repeat`
-- `buildtype`: The build configuration of the tests to be run (`Debug` or `Release`). By default, both will be run (pass an empty string `""` to replicate this).
-- `nruns_f`: Number of times to run the original functions (BA, GMM) for each tool (default = `10`)
-- `nruns_J`: Number of times to run the autodiff process for each tool (default = `10`)
-- `time_limit`: The maximum amount of time (in seconds) to spend benchmarking each tool (default = `60`)
-	- After each run, the program will check if it has exceeded the time limit
-	- `time_limit` will never cause a task to run less than once, and the program will always output a resultant time
-	- **Note**: The time limits are only implemented in C++ and Python currently
-- `timeout`: The maximum amount of time (in seconds) to allow each tool to run for
-	- If a program exceeds this time limit, PowerShell kills it
-	- This may result in no runs being completed, and no output being produced (PowerShell will output a file with `inf` timings in it to mark the test as failed)
-- `tmpdir`: The output directory to use (default = `/tmp/`)
-- `repeat`: Whether to repeat tasks for which an output file already exists (default = `false`)
-
-### CLI reference: plot_graphs
-
-`python plot_graphs.py [--save] [--plotly] [--show]`
-- `--save`: Save graphs as .png files to `Documents/New Figures/`
-- `--plotly`: Save graphs as [Plot.ly](https://plot.ly/) .html files to `Documents/New Figures/plotly/`
-- `--show`: Display graphs in new windows
-- If neither `--save` or `--plotly` are included, `--show` will be `True` by default - otherwise, it must be manually enabled
-
-### check_J
-
-`python check_J.py`
-- This script compares all available Jacobian output files, and outputs results to the console
-- It will flag missing output files, and indicate where there may be a mismatch between files
-- For comparison, all derivatives are rounded to a certain number of significant figures, as specified by a constant at the top of the script
+The cloud infrastructure that generates these plots is described [here](docs/AzureBatch.md).
 
 ## Contributing
 
 Contributions to fix bugs, test on new systems or add new tools are welcomed. See [Contributing](/CONTRIBUTING.md) for details on how to add new tools, and [Issues](/ISSUES.md) for known bugs and TODOs.
 
 ## Known Issues
 
-See [Issues](/ISSUES.md) for a complete list of known problems and TODOs.
-
-## Example Graphs
-
-Below are two examples of the graphs produced by ADBench/plot_graphs.py. The full range of graphs (over 40) can be found in both static (png) and [Plot.ly](https://plot.ly/) (html) formats in [Documents/New Figures](/Documents/New%20Figures/).
-
-![GMM 1k Jacobian Release Graph](/Documents/New%20Figures/static/Release/jacobian/GMM%20%281k%29%20[Jacobian]%20-%20Release%20Graph.png)
+See [Issues](/ISSUES.md) for a list of some of the known problems and TODOs.
 
-![Hand Simple Small Jacobian÷Objective Debug Graph](/Documents/New%20Figures/static/Debug/jacobian%20÷%20objective/HAND%20%28Simple,%20Small%29%20[Jacobian%20÷%20objective]%20-%20Debug%20Graph.png)
+There's [GitHub's issue](https://github.com/awf/ADBench/issues) page as well.

STATUS.md

@@ -1,23 +1,6 @@
 # Status
 
-This file contains details about the layout of the repository, and the current development status of different tools and objectives.
-
-
-## Folder structure
-
-| Folder    | Purpose
-| --------- | ------- |
-| ADBench   | Orchestration scripts, plotting etc
-| Backup	| Old files
-| Documents | Graphs, papers, presentations etc
-| bak		| Old files
-| data      | Data files for different examples 
-| etc		| [HunterGate](https://github.com/ruslo/hunter) files
-| submodules| Dependencies on other git repositories
-| tmp       | Output of benchmark files, figures etc
-| tools     | Implementations for each tested tool, one folder per tool, with *-common folders for shared files by language
-| usr       | Per-user scratch folder
-
+This file contains the current development status of different tools and objectives.
 
 ## Tools
 
@@ -40,13 +23,14 @@ Checked items are built (where relevant) by CMake and can be run by run-all.ps1
 - [x] Manual
 	- Manually-differentiated C++ functions
 	- No external dependencies
-- [ ] Tapenade
+- [x] Tapenade
 
 ### Python
-All Python tools use pip/conda modules. See list under [Prerequisites](#prerequisites).
+All Python tools use pip/conda modules. Dependencies for checked items are automatically fetched during CMake configure.
 - [x] [Autograd](https://github.com/HIPS/autograd)
-- [x] [Theano](https://github.com/Theano/Theano)
+- [ ] [Theano](https://github.com/Theano/Theano)
 - [x] [PyTorch](https://pytorch.org/)
+- [x] [Tensorflow](https://www.tensorflow.org/)
 
 ### Matlab
 Matlab tools are not currently run by `run-all.ps1` due to the limitations of the Matlab free trial. With the full version, they *should* run correctly.
@@ -56,10 +40,12 @@ Matlab tools are not currently run by `run-all.ps1` due to the limitations of th
 
 ### F#
 - [x] [DiffSharp](https://github.com/DiffSharp/DiffSharp)
-	- Built using `dotnet build` (in batch file), which restores NuGet packages
+	- Built using `dotnet build`, which restores NuGet packages
 
 ### Julia
+Dependencies for Julia-based tools are listed in `JuliaProject.toml` and automatically fetched during CMake configure.
 - [ ] ForwardDiff.jl
+- [x] Zygote
 
 
 ## Completeness
@@ -73,17 +59,20 @@ This is a table of all current tools, with their status in terms of running each
 | ADOLC     |   x   |   x   |   x   |   -   |
 | Autograd  |   x   |   x   |   *   |   -   |
 | Ceres     |   x   |   x   |   *   |   -   |
-| DiffSharp |   !   |   x   |   -   |   -   |
+| DiffSharp |   x!  |   x   |   x   |   x   |
 | Finite    |   x   |   x   |   x   |   x   |
-| Julia     |   -   |   -   |   -   |   -   |
-| Manual    |   x   |   x   |   x   |   ?   |
+| Julia     |   x   |   x   |   -   |   -   |
+| Manual    |   x   |   x   |   x   |   x   |
 | MuPad     |   x   |   !   |   ?   |   -   |
-| PyTorch   |   x   |   !   |   x   |   x   |
-| Tapenade  |   -   |   -   |   -   |   -   |
+| PyTorch   |   x   |   x   |   x   |   x   |
+| Tapenade  |   x   |   x   |   x   |   x   |
+| Tensorflow|   x   |   x   |   x   |   x   |
 | Theano    |   x   |   x   |   x   |   -   |
+| Zygote    |   x!  |   x   |   x   |   x   |
 
 ### Key
 - `x` = Runs successfully
+- `x!` = Generally runs successfully, but fails on some of the problem sizes
 - `!` = Has been attempted, with problems
 - `?` = Has been started, but is not complete
 - `-` = Not attempted yet