Merge pull request #200 from davidsd/release-2.7.0

Update docs for 2.7.0 + fix compilation on Imperial HPC

.circleci/config.yml

@@ -7,7 +7,7 @@
 # 3. 'deploy-master': On each 'master' branch update,
 #     push the fresh image to DockerHub as '${DOCKER_USERNAME}/sdpb:master', e.g. 'davidsd/sdpb:master'
 # 4. 'deploy-tag': If new tag is pushed to the repo,
-#     push the image '${DOCKER_USERNAME}/sdpb:${TAG}', e.g. 'davidsd/sdpb:2.6.1'
+#     push the image '${DOCKER_USERNAME}/sdpb:${TAG}', e.g. 'davidsd/sdpb:2.7.0'
 #
 # Deploy works only if you specify DOCKER_USERNAME and DOCKER_PASSWORD environment variables for the CircleCI project
 # See instructions here:
@@ -134,7 +134,7 @@ workflows:
       - deploy-tag:
           filters:
             tags:
-              only: /^\d+\.\d+\.\d+$/ # only release tags, e.g 2.6.1
+              only: /^\d+\.\d+\.\d+$/ # only release tags, e.g 2.7.0
             # See https://discuss.circleci.com/t/tag-not-triggered-by-circleci-not-filter-in-workflow/32036
             branches:
               ignore: /.*/ # Ignore all branches, otherwise the job is triggered by any git push

Changelog.md

@@ -1,3 +1,45 @@
+# Version 2.7.0
+
+### Breaking changes:
+
+- New executable `pmp2sdp` instead of separate `sdp2input` and `pvm2sdp`. Reads Polynomial Matrix Programs in JSON, Mathematica, or XML formats. We recommend using JSON since it is more compact, efficient and universal than Mathematica or XML.
+See [#181](https://github.com/davidsd/sdpb/pull/181), [#184](https://github.com/davidsd/sdpb/pull/184). 
+Example:
+```
+pmp2sdp --precision 768 --input in/pmp.json --output out/sdp --verbosity regular
+```
+By default, `pmp2sdp` writes sdp to plain directory (as in 2.4.0 and earlier). Run it with `--zip` flag to write to zip archive instead (as in 2.5.0 - 2.6.1).
+Note that writing to zip can be slow for large problems, see plots in [#176](https://github.com/davidsd/sdpb/pull/176).
+
+`sdp2input` and `pvm2sdp` are deprecated and will be removed in future versions.
+
+
+- `sdpb` option `--procsPerNode` is deprecated and will be removed in future versions. Now MPI configuration is determined automatically. See [#169](https://github.com/davidsd/sdpb/pull/169).
+- Mathematica script [SDPB.m](mathematica/SDPB.m) now writes PMP in JSON format instead of XML. Use `WritePmpJson` (or its alias `WriteBootstrapSDP`) function. `WritePmpXml` is left for backward compatibility.
+**NB:** you should also change PMP file extension from `.xml` to `.json` in your Mathematica scripts.
+Using JSON instead of XML results in ~x2 speedup when running [Bootstrap2dExample.m](mathematica/Bootstrap2dExample.m) script. See [#199](https://github.com/davidsd/sdpb/pull/199).
+
+### New features:
+- Added `sdpb` option `--writeSolution=z`. Restores the vector `z` (see [SDPB Manual](docs/SDPB_Manual/SDPB-Manual.pdf), eq. 3.1) from the vector `y` (eq. 2.2) and `sdp/normalization.json`, and writes it to `out/z.txt`.
+See [#191](https://github.com/davidsd/sdpb/pull/191).
+- Added new field `"block_path"` to `spectrum` JSON output. For each block, `"block_path"` contains its path from the PMP input. See [#190](https://github.com/davidsd/sdpb/pull/190).
+- Added optional fields to `pmp.json` format: `"samplePoints"`, `"sampleScalings"` and `"bilinearBasis"`, same as in the old XML format.
+See [#199](https://github.com/davidsd/sdpb/pull/199).
+
+### Other improvements:
+- Optimized IO and RAM usage when reading big SDP blocks, see [#167](https://github.com/davidsd/sdpb/pull/167).
+- Optimized reading and writing for `pmp2sdp`, got order-of-magnitude speedup.
+See [#176](https://github.com/davidsd/sdpb/pull/176) and [#181](https://github.com/davidsd/sdpb/pull/181).
+- Fix undefined behaviour (sometimes leading to SEGFAULT) when reading PMP with a prefactor containing duplicate poles. See [#195](https://github.com/davidsd/sdpb/pull/195).
+- Fix incorrect block mapping for cyclic MPI job distribution across nodes, see [#170](https://github.com/davidsd/sdpb/pull/170).
+- Print number of MPI processes and nodes, print SDP dimensions and number of blocks. See [#169](https://github.com/davidsd/sdpb/pull/169).
+- Print source code location and stacktrace for exceptions, see [#180](https://github.com/davidsd/sdpb/pull/180) and [#198](https://github.com/davidsd/sdpb/pull/198).
+- Rewrote Section 3 of [SDPB Manual](docs/SDPB_Manual/SDPB-Manual.pdf), see [#199](https://github.com/davidsd/sdpb/pull/199).
+- Internal improvements: new macros `RUNTIME_ERROR`, `ASSERT`, `ASSERT_EQUAL` and `DEBUG_STRING`, new JSON parser, better GMP<->MPFR conversion, fix compiler warnings, refactor integration tests.
+See [#180](https://github.com/davidsd/sdpb/pull/180), [#189](https://github.com/davidsd/sdpb/pull/189), [#196](https://github.com/davidsd/sdpb/pull/196), [#197](https://github.com/davidsd/sdpb/pull/197).
+
+See https://github.com/davidsd/sdpb/releases/tag/2.7.0 for the full changelog.
+
 # Version 2.6.1
 
 - Fixed major memory leak in binary SDP deserialization. See [#160](https://github.com/davidsd/sdpb/pull/160).

docs/Docker.md

@@ -20,9 +20,9 @@ can be downloaded from [Docker Hub](https://hub.docker.com/r/bootstrapcollaborat
 
     docker pull bootstrapcollaboration/sdpb:master
 
-You can also download image for a specific release, e.g., 2.6.1:
+You can also download image for a specific release, e.g., 2.7.0:
 
-    docker pull bootstrapcollaboration/sdpb:2.6.1
+    docker pull bootstrapcollaboration/sdpb:2.7.0
 
 The list of all available tags can be found in
 [bootstrapcollaboration/sdpb](https://hub.docker.com/r/bootstrapcollaboration/sdpb/tags) repo on Docker Hub.

docs/SDPB_Manual/SDPB-Manual.tex

@@ -38,7 +38,7 @@
 
 {\Large
 \begin{center}
-{\bf SDPB 2.6.1 \\\vspace{.1in}}
+{\bf SDPB 2.7.0 \\\vspace{.1in}}
 \end{center}
 }
 \begin{center}

docs/Singularity.md

@@ -54,21 +54,21 @@ and push it to [local registry](https://docs.docker.com/registry/):
 
 ## Run Singularity image
 Singularity should automatically mount your home directory.  If your
-Singularity image is named `sdpb-2.6.1.sif`, you can invoke the SDPB
+Singularity image is named `sdpb-2.7.0.sif`, you can invoke the SDPB
 programs by prepending the command with
 
-    singularity exec sdpb-2.6.1.sif
+    singularity exec sdpb-2.7.0.sif
 
 So to convert the JSON input file at `/home/user/input.json`, run the command
 
-    singularity exec sdpb-2.6.1.sif mpirun -n 4 pmp2sdp --precision 1024 -i /home/user/input.json -o /home/user/input
+    singularity exec sdpb-2.7.0.sif mpirun -n 4 pmp2sdp --precision 1024 -i /home/user/input.json -o /home/user/input
 
 This uses 4 cores when running pmp2sdp. You can change that number to
 match your own machine.
 
 To find a primal-dual solution, 
 
-    singularity exec sdpb-2.6.1.sif mpirun -n 4 sdpb --precision=1024 -s /home/user/input
+    singularity exec sdpb-2.7.0.sif mpirun -n 4 sdpb --precision=1024 -s /home/user/input
 
 In theory, Singularity can be used to run jobs across multiple nodes.
 We have not been able to make that work yet.  So for large, multi-node

docs/site_installs/Boston.md

@@ -32,12 +32,12 @@ You may list all available versions via
 
 Fo example, `sdpb-master` is built from the latest [master](https://github.com/davidsd/sdpb/tree/master) branch (
 run `sdpb --version` to see commit hash, e.g. `SDPB 2.5.1-130-g88b1c9ae`),
-and `sdpb-2.6.1` is a stable [2.6.1](https://github.com/davidsd/sdpb/releases/tag/2.6.1) release.
+and `sdpb-2.7.0` is a stable [2.7.0](https://github.com/davidsd/sdpb/releases/tag/2.7.0) release.
 
 Examples below are for `sdpb-master`.
-You may replace it with another version, e.g. `sdpb-2.6.1`.
+You may replace it with another version, e.g. `sdpb-2.7.0`.
 In that case, please refer
-to [2.6.1 documentation](https://github.com/davidsd/sdpb/blob/2.6.1/docs/site_installs/Boston.md).
+to [2.7.0 documentation](https://github.com/davidsd/sdpb/blob/2.7.0/docs/site_installs/Boston.md).
 
 ## Run SDPB
 

docs/site_installs/Caltech.md

@@ -13,7 +13,7 @@
 
 For compiling and/or running SDPB, you have to load modules first:
 
-    module load cmake/3.25.1 gcc/9.2.0 openmpi/4.1.5 boost/1_81_0_openmpi-4.1.1_gcc-9.2.0 eigen/eigen mpfr/4.0.2
+    module load python3/3.8.5 git/2.37.2 cmake/3.25.1 gcc/9.2.0 openmpi/4.1.5 boost/1_81_0_openmpi-4.1.1_gcc-9.2.0 eigen/eigen mpfr/4.0.2
 
 You may run `module -t list` to view loaded modules,
 and `module purge` to unload all modules.
@@ -33,12 +33,12 @@ You may list all available versions via
 
 Fo example, `sdpb-master` is built from the latest [master](https://github.com/davidsd/sdpb/tree/master) branch (
 run `sdpb --version` to see commit hash, e.g. `SDPB 2.5.1-130-g88b1c9ae`),
-and `sdpb-2.6.1` is a stable [2.6.1](https://github.com/davidsd/sdpb/releases/tag/2.6.1) release.
+and `sdpb-2.7.0` is a stable [2.7.0](https://github.com/davidsd/sdpb/releases/tag/2.7.0) release.
 
 Examples below are for `sdpb-master`.
-You may replace it with another version, e.g. `sdpb-2.6.1`.
+You may replace it with another version, e.g. `sdpb-2.7.0`.
 In that case, please refer
-to [2.6.1 documentation](https://github.com/davidsd/sdpb/blob/2.6.1/docs/site_installs/Caltech.md).
+to [2.7.0 documentation](https://github.com/davidsd/sdpb/blob/2.7.0/docs/site_installs/Caltech.md).
 
 ## Run SDPB
 
@@ -88,8 +88,8 @@ should compile on a login node.
 
 ## sdpb
 
-    ./waf configure --elemental-dir=$HOME/install --rapidjson-dir=$HOME/install --libarchive-dir=$HOME/install --mpfr-dir=/central/software/mpfr/4.0.2 --prefix=$HOME/install/sdpb-master
-    ./waf # -j 1
+    python3 ./waf configure --elemental-dir=$HOME/install --rapidjson-dir=$HOME/install --libarchive-dir=$HOME/install --mpfr-dir=/central/software/mpfr/4.0.2 --prefix=$HOME/install/sdpb-master
+    python3 ./waf # -j 1
     ./test/run_all_tests.sh
-    ./waf install
+    python3 ./waf install
     cd ..

docs/site_installs/Expanse.md

@@ -32,12 +32,12 @@ You may list all available versions via
 
 Fo example, `sdpb-master` is built from the latest [master](https://github.com/davidsd/sdpb/tree/master) branch (
 run `sdpb --version` to see commit hash, e.g. `SDPB 2.5.1-130-g88b1c9ae`),
-and `sdpb-2.6.1` is a stable [2.6.1](https://github.com/davidsd/sdpb/releases/tag/2.6.1) release.
+and `sdpb-2.7.0` is a stable [2.7.0](https://github.com/davidsd/sdpb/releases/tag/2.7.0) release.
 
 Examples below are for `sdpb-master`.
-You may replace it with another version, e.g. `sdpb-2.6.1`.
+You may replace it with another version, e.g. `sdpb-2.7.0`.
 In that case, please refer
-to [2.6.1 documentation](https://github.com/davidsd/sdpb/blob/2.6.1/docs/site_installs/Expanse.md).
+to [2.7.0 documentation](https://github.com/davidsd/sdpb/blob/2.7.0/docs/site_installs/Expanse.md).
 
 ## Run SDPB
 

docs/site_installs/Harvard.md

@@ -32,12 +32,12 @@ You may list all available versions via
 
 Fo example, `sdpb-master` is built from the latest [master](https://github.com/davidsd/sdpb/tree/master) branch (
 run `sdpb --version` to see commit hash, e.g. `SDPB 2.5.1-130-g88b1c9ae`),
-and `sdpb-2.6.1` is a stable [2.6.1](https://github.com/davidsd/sdpb/releases/tag/2.6.1) release.
+and `sdpb-2.7.0` is a stable [2.7.0](https://github.com/davidsd/sdpb/releases/tag/2.7.0) release.
 
 Examples below are for `sdpb-master`.
-You may replace it with another version, e.g. `sdpb-2.6.1`.
+You may replace it with another version, e.g. `sdpb-2.7.0`.
 In that case, please refer
-to [2.6.1 documentation](https://github.com/davidsd/sdpb/blob/2.6.1/docs/site_installs/Harvard.md).
+to [2.7.0 documentation](https://github.com/davidsd/sdpb/blob/2.7.0/docs/site_installs/Harvard.md).
 
 ## Run SDPB
 

docs/site_installs/Imperial.md

@@ -33,12 +33,12 @@ You may list all available versions via
 
 Fo example, `sdpb-master` is built from the latest [master](https://github.com/davidsd/sdpb/tree/master) branch (
 run `sdpb --version` to see commit hash, e.g. `SDPB 2.5.1-130-g88b1c9ae`),
-and `sdpb-2.6.1` is a stable [2.6.1](https://github.com/davidsd/sdpb/releases/tag/2.6.1) release.
+and `sdpb-2.7.0` is a stable [2.7.0](https://github.com/davidsd/sdpb/releases/tag/2.7.0) release.
 
 Examples below are for `sdpb-master`.
-You may replace it with another version, e.g. `sdpb-2.6.1`.
+You may replace it with another version, e.g. `sdpb-2.7.0`.
 In that case, please refer
-to [2.6.1 documentation](https://github.com/davidsd/sdpb/blob/2.6.1/docs/site_installs/Imperial.md).
+to [2.7.0 documentation](https://github.com/davidsd/sdpb/blob/2.7.0/docs/site_installs/Imperial.md).
 
 ## Run SDPB
 

test/src/unit_tests/cases/json.test.cxx

@@ -100,7 +100,7 @@ namespace
     auto on_parsed = [&output_values](auto &&parse_result) {
       output_values = std::forward<std::vector<El::BigFloat>>(parse_result);
     };
-    Json_Float_Vector_Parser<El::BigFloat> parser(skip, on_parsed);
+    Json_Float_Vector_Parser<El::BigFloat> parser(skip, on_parsed, []{});
 
     json_parse(json_string, parser);
     return output_values;