Add clone-dependencies.sh script

Currently, builds of our Python SDK clone several C++ dependencies directly from the `main` branch of their canonical git repositories and build them from source. Sometimes, though, `main` might be broken, causing our nightly builds to spuriously fail. Instead, we should clone from tagged releases of our dependencies, pinning the versions so we know that our builds will not fail due to development issues in our dependencies. This patch refactors the `bin/build-*.sh` scripts to enable us to do this more easily. First, it creates a new `bin/clone-dependencies.sh` script, which specifies the tagged versions of and clones the dependencies `bde-tools`, `bde`, `ntf-core`, and `blazingmq`. This consolidates the dependency cloning that is common between all the `bin/build-*.sh` scripts. This script is not intended to be run by users directly; instead, this patch secondarily removes the dependency cloning from each build script, and sources the `bin/clone-dependencies.sh` script. The existing dependency cloning logic allows for a user to manually clone these dependencies in the correct location before running the script, and those clones will be used instead. This patch preserves this functionality, which allows users to still build and test the Python SDK against the `main` branches of these dependencies, at the cost of doing the cloning and checking out themselves. To update the release tags that the SDK is built against, the environment variables at the top of `.bin/clone-dependencies.sh` should be modified. Testing ------- This patch was tested by building and testing the SDK with the `./bin/build-macos-universal.sh` script on an M2 Mac system, and manually ensuring that all dependencies were correctly cloned to the correct tag. Signed-off-by: Patrick M. Niedzielski <[email protected]>
bloomberg · Nov 30, 2023 · 59c2b1e · 59c2b1e
1 parent d3e3f43
commit 59c2b1e
Show file tree

Hide file tree

Showing 3 changed files with 55 additions and 24 deletions.
diff --git a/bin/build-macos-universal.sh b/bin/build-macos-universal.sh
@@ -44,18 +44,7 @@ fi
 
 # :: Clone dependencies :::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
-if [ ! -d "${DIR_THIRDPARTY}/bde-tools" ]; then
-    git clone https://github.com/bloomberg/bde-tools "${DIR_THIRDPARTY}/bde-tools"
-fi
-if [ ! -d "${DIR_THIRDPARTY}/bde" ]; then
-    git clone --depth 1 https://github.com/bloomberg/bde.git "${DIR_THIRDPARTY}/bde"
-fi
-if [ ! -d "${DIR_THIRDPARTY}/ntf-core" ]; then
-    git clone --depth 1 https://github.com/bloomberg/ntf-core.git "${DIR_THIRDPARTY}/ntf-core"
-fi
-if [ ! -d "${DIR_THIRDPARTY}/blazingmq" ]; then
-    git clone --depth 1 https://github.com/bloomberg/blazingmq.git "${DIR_THIRDPARTY}/blazingmq"
-fi
+source ./bin/clone-dependencies.sh
 
 # Build and install BDE
 # Refer to https://bloomberg.github.io/bde/library_information/build.html

diff --git a/bin/build-manylinux.sh b/bin/build-manylinux.sh
@@ -42,25 +42,16 @@ fi
 
 # :: Clone dependencies :::::::::::::::::::::::::::::::::::::::::::::::::::::::
 
-if [ ! -d "${DIR_THIRDPARTY}/bde-tools" ]; then
-    git clone https://github.com/bloomberg/bde-tools "${DIR_THIRDPARTY}/bde-tools"
-fi
-if [ ! -d "${DIR_THIRDPARTY}/bde" ]; then
-    git clone --depth 1 https://github.com/bloomberg/bde.git "${DIR_THIRDPARTY}/bde"
-fi
-if [ ! -d "${DIR_THIRDPARTY}/ntf-core" ]; then
-    git clone --depth 1 https://github.com/bloomberg/ntf-core.git "${DIR_THIRDPARTY}/ntf-core"
-fi
+source ./bin/clone-dependencies.sh
+
+# Download additional dependencies that are not packaged on manylinux:
 if [ ! -d "${DIR_THIRDPARTY}/google-benchmark" ]; then
     git clone --depth 1 https://github.com/google/benchmark.git "${DIR_THIRDPARTY}/google-benchmark"
 fi
 if [ ! -d "${DIR_THIRDPARTY}/bison" ]; then
     mkdir -p "${DIR_THIRDPARTY}/bison"
     curl https://ftp.gnu.org/gnu/bison/bison-3.8.2.tar.xz | tar -Jx -C "${DIR_THIRDPARTY}/bison" --strip-components 1
 fi
-if [ ! -d "${DIR_THIRDPARTY}/blazingmq" ]; then
-    git clone --depth 1 https://github.com/bloomberg/blazingmq.git "${DIR_THIRDPARTY}/blazingmq"
-fi
 
 if [ ! -e "${DIR_THIRDPARTY}/cmake/.complete" ]; then
     mkdir -p "${DIR_THIRDPARTY}/cmake"

diff --git a/bin/clone-dependencies.sh b/bin/clone-dependencies.sh
@@ -0,0 +1,51 @@
+#!/usr/bin/env bash
+
+# This script clones the dependencies of the BlazingMQ Python SDK that are not
+# in a package manager.  This script should not be run by users directly, but
+# is instead sourced by the `./build-*.sh` scripts in this directory.
+
+
+set -e
+set -u
+[ -z "$BASH" ] || shopt -s expand_aliases
+
+
+# These are the release tags for each of the dependencies we manually clone.
+# Update these to update the version of each dependency we build against.
+BDE_TOOLS_TAG=3.124.0.0
+BDE_TAG=3.124.0.0
+NTF_CORE_TAG=latest
+BLAZINGMQ_TAG=BMQBRKR_0.90.20
+
+
+if [ ! -d "${DIR_THIRDPARTY}/bde-tools" ]; then
+    git clone                                                                  \
+        --depth 1                                                              \
+        --branch ${BDE_TOOLS_TAG}                                              \
+        https://github.com/bloomberg/bde-tools                                 \
+        "${DIR_THIRDPARTY}/bde-tools"
+fi
+
+if [ ! -d "${DIR_THIRDPARTY}/bde" ]; then
+    git clone                                                                  \
+        --depth 1                                                              \
+        --branch ${BDE_TAG}                                                    \
+        https://github.com/bloomberg/bde.git                                   \
+        "${DIR_THIRDPARTY}/bde"
+fi
+
+if [ ! -d "${DIR_THIRDPARTY}/ntf-core" ]; then
+    git clone                                                                  \
+        --depth 1                                                              \
+        --branch ${NTF_CORE_TAG}                                               \
+        https://github.com/bloomberg/ntf-core.git                              \
+        "${DIR_THIRDPARTY}/ntf-core"
+fi
+
+if [ ! -d "${DIR_THIRDPARTY}/blazingmq" ]; then
+    git clone                                                                  \
+        --depth 1                                                              \
+        --branch ${BLAZINGMQ_TAG}                                              \
+        https://github.com/bloomberg/blazingmq.git                             \
+        "${DIR_THIRDPARTY}/blazingmq"
+fi