Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] OXT-1736 : do-build.sh zeus uprev and usability improvements #365

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

[WIP] OXT-1736 : do-build.sh zeus uprev and usability improvements #365

Show file tree
Hide file tree
Changes from 8 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
89e6381
do_build.sh: test for bash as system shell
Jan 24, 2020
365a728
do_build.sh: keep bitbake memory-resident
Jan 24, 2020
57712d6
do_build.sh: enable optional interactive build
Jan 24, 2020
55a5221
do_build.sh: remove obsolete steps: syncui, nilfvm
Jan 24, 2020
e9c80eb
do_build.sh: for PXE builds, symlink latest output
Jan 24, 2020
5a617ab
do_build.sh: regenerate local.conf for each build
Jan 24, 2020
853d7a4
example-config: uprev to OE zeus, bitbake 1.44
Jan 24, 2020
c53f8c0
local-conf: enable bitbake parallel builds
Jan 24, 2020
6436d4e
do_build.sh: initialize variables
Jan 24, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions build/conf/local.conf-dist
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,9 @@
CONF_VERSION = "1"

# Enable parallel builds on multi-core machines
BB_NUMBER_THREADS ?= "4"
PARALLEL_MAKE ?= "-j 4"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These were removed in c6b7942 since bitbake sets a default.


# 1) Basic config
#
## TODO: DISTRO should be in auto.conf
Expand Down
139 changes: 47 additions & 92 deletions do_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
#! /bin/bash -e
set -o pipefail

STEPS="setupoe,initramfs,stubinitramfs,dom0,uivm,ndvm,syncvm,installer,installer2,syncui,source,sdk,license,sourceinfo,ship"
STEPS="setupoe,initramfs,stubinitramfs,dom0,uivm,ndvm,syncvm,installer,installer2,source,sdk,license,sourceinfo,ship"

# Additional steps:

Expand All @@ -16,6 +16,17 @@ STEPS="setupoe,initramfs,stubinitramfs,dom0,uivm,ndvm,syncvm,installer,installer
# Packages tree can use hardlinks to save disk space, if $SYNC_CACHE_OE/oe-archives is populated
# Requires a valid NETBOOT_HTTP_URL in .config

# OpenXT build requires system shell to be bash
SYSTEM_SHELL=$(basename $(readlink -f /bin/sh))
[[ ${SYSTEM_SHELL} == "bash" ]] || \
{ echo "OpenXT build requires bash as system shell. Please symlink /bin/sh to bash"; exit 1; }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

build-scripts/oe/setup.sh suggests this which seems more debian friendly:

# Use bash instead of dash for /bin/sh
echo "dash dash/sh boolean false" > /tmp/preseed.txt
debconf-set-selections /tmp/preseed.txt
dpkg-reconfigure -f noninteractive dash


# Keep bitbake memory-resident between invocations
export BB_SERVER_TIMEOUT=30

# Optional interactive use: set to 1 to disable logs
INTERACTIVE=0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems weird - just run MACHINE=... ./bb <recipe> from openxt/build manually? I expected to see a cli flag to enable this option.


TOPDIR=`pwd`
OUTPUT_DIR="$TOPDIR/build-output"
CMD="$0"
Expand All @@ -30,6 +41,8 @@ BRANCH=master
BUILD_UID=`id -u`
export BUILD_UID

CONF_LOCAL="conf/local.conf"

# TODO: move some of the above definitions into common-config

if [ -f ${CMD_DIR}/common-config ]; then
Expand Down Expand Up @@ -89,16 +102,18 @@ do_oe_setup()
mkdir -p conf
fi

if [ ! -f "conf/local.conf" -o "conf/local.conf" -ot "conf/local.conf-dist" ]; then
cp conf/local.conf-dist conf/local.conf
# Force generation of local.conf with latest build version
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

whitespace

rm -f ${CONF_LOCAL}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Clobber local.conf every time will lose customization. If you want changing build info, then many do something like bordel generating and including a separate file:
https://github.com/apertussolutions/bordel/blob/fed91e907435072ceab1309bcc8ab46454b93041/cmds/build#L24
https://github.com/apertussolutions/bordel/blob/008cf2793b229ad9b138ea12f6e450436f29cc8a/templates/master/local.conf#L97

if [ ! -f "${CONF_LOCAL}" -o "${CONF_LOCAL}" -ot "${CONF_LOCAL}-dist" ]; then
cp "${CONF_LOCAL}-dist" "${CONF_LOCAL}"

if [ ! -z "${OE_TARBALL_MIRROR}" ] ; then
cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF
# Tarball mirror
PREMIRRORS = "(ftp|https?)$://.*/.*/ ${OE_TARBALL_MIRROR}"
EOF
fi
cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF

# Distribution feed
XENCLIENT_PACKAGE_FEED_URI="${NETBOOT_HTTP_URL}/${BRANCH}/${NAME}/packages/ipk"
Expand All @@ -120,12 +135,12 @@ OPENXT_TAG="$BRANCH"
EOF

if [ "x$ID" != "x" ]; then
echo "XENCLIENT_BUILD = \"$ID\"" >> conf/local.conf
echo "XENCLIENT_BUILD = \"$ID\"" >> ${CONF_LOCAL}
else
echo "XENCLIENT_BUILD = \"$NAME\"" >> conf/local.conf
echo "XENCLIENT_BUILD = \"$NAME\"" >> ${CONF_LOCAL}
fi

cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF
XENCLIENT_BUILD_DATE = "`date +'%T %D'`"
XENCLIENT_BUILD_BRANCH = "${BRANCH}"
XENCLIENT_VERSION = "$VERSION"
Expand All @@ -137,7 +152,7 @@ XCT_DEB_PKGS_DIR := "${OE_BUILD_CACHE}/xct_deb_packages"
EOF

if [ -f ${CMD_DIR}/common-config ]; then
cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF
# xen version and source
XEN_VERSION="${XEN_VERSION}"
XEN_SRC_URI="${XEN_SRC_URI}"
Expand All @@ -147,7 +162,7 @@ XEN_SRC_SHA256SUM="${XEN_SRC_SHA256SUM}"
EOF
fi

cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF
# Production and development repository-signing CA certificates
REPO_PROD_CACERT="$REPO_PROD_CACERT_PATH"
REPO_DEV_CACERT="$REPO_DEV_CACERT_PATH"
Expand All @@ -156,15 +171,15 @@ EOF

if [ $SOURCE -eq 1 ]
then
cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF

XENCLIENT_BUILD_SRC_PACKAGES = "1"
XENCLIENT_COLLECT_SRC_INFO = "1"
EOF
fi
if [ "x$FREEZE_URIS" = "xyes" ]
then
cat >> conf/local.conf <<EOF
cat >> ${CONF_LOCAL} <<EOF

INHERIT += "freezer"
EOF
Expand All @@ -174,7 +189,7 @@ EOF
if [ $VERBOSE -eq 1 ]
then
echo "Generated config is:"
cat conf/local.conf
cat ${CONF_LOCAL}
fi

if [ $VERBOSE -eq 1 ]
Expand Down Expand Up @@ -227,7 +242,11 @@ do_oe()
fi
echo "STARTING OE BUILD $image $machine, started at" `date -u +'%H:%M:%S UTC'`

< /dev/null ./bb $BBFLAGS "$image" | do_oe_log
if [ $INTERACTIVE -eq 1 ]; then
./bb $BBFLAGS "$image"
else
< /dev/null ./bb $BBFLAGS "$image" | do_oe_log
fi
popd
}

Expand Down Expand Up @@ -300,28 +319,6 @@ do_oe_ndvm()
do_oe_ndvm_copy $path
}

do_oe_nilfvm_copy()
{
local path="$1"
do_oe_copy "$path" "nilfvm" "xenclient-nilfvm" "xenclient-nilfvm"

local binaries="tmp-glibc/deploy/images"
pushd "$path"
cp "$binaries/service-nilfvm" "$OUTPUT_DIR/$NAME/raw/service-nilfvm"
popd
}

do_oe_nilfvm()
{
local path="$1"

echo This step is now useless, everything we need should be built as part of the tools.
return

do_oe "$path" "xenclient-nilfvm" "xenclient-nilfvm-image"
do_oe_nilfvm_copy $path
}

do_oe_syncvm_copy()
{
local path="$1"
Expand All @@ -335,23 +332,6 @@ do_oe_syncvm()
do_oe_syncvm_copy $path
}

do_oe_syncui_copy()
{
local path="$1"
pushd "$path"
mkdir -p "$OUTPUT_DIR/$NAME/raw"
cp tmp-glibc/deploy/tar/sync-wui-0+git*.tar.gz "$OUTPUT_DIR/$NAME/raw/sync-wui-${RELEASE}.tar.gz"
cp tmp-glibc/deploy/tar/sync-wui-sources-0+git*.tar.gz "$OUTPUT_DIR/$NAME/raw/sync-wui-sources-${RELEASE}.tar.gz"
popd
}

do_oe_syncui()
{
local path="$1"
do_oe "$path" "xenclient-syncui" "sync-wui"
do_oe_syncui_copy "$path"
}

do_oe_dom0_copy()
{
local path="$1"
Expand Down Expand Up @@ -1169,10 +1149,6 @@ do_xctools_debian_repo()
local dest_dir="${path}/tmp-glibc/deb-xctools-image/"
local d_output_dir="${OUTPUT_DIR}/${NAME}/xctools-debian-repo/debian"

echo "Building Debian Service VM tools"
do_oe "${path}" "xenclient-nilfvm" "linux-xenclient-nilfvm"
do_oe "${path}" "xenclient-nilfvm" "deb-servicevm-tools"

echo "Building XC Tools Debian/Ubuntu repository"
do_oe "${path}" "xenclient-dom0" "deb-xctools-image"

Expand Down Expand Up @@ -1230,26 +1206,6 @@ do_xctools() {
do_xctools_linux "$1"
}

do_syncui()
{
local name="sync-wui-${RELEASE}.tar.gz"
local file="$OUTPUT_DIR/$NAME/raw/$name"
local out="$OUTPUT_DIR/$NAME/sync"

if [ ! -r "${file}" ]; then
echo "syncui: Not built, skipping"
return 0
fi

echo "syncui:"
echo " - copy $name"

mkdir -p "$out"
cp "$file" "$out"

echo
}

do_info()
{
local out="$OUTPUT_DIR/$NAME"
Expand Down Expand Up @@ -1281,12 +1237,20 @@ do_ship()
do_updates
do_netboots
do_installer_isos
do_source_iso
do_source_info
do_licences
do_syncui
do_info
do_logs

if [ $INTERACTIVE -eq 0 ]; then
do_source_iso
do_source_info
do_licences
do_info
do_logs
fi

pushd $OUTPUT_DIR
SYMLINK_OUTPUT="_do_build_latest"
rm -f $SYMLINK_OUTPUT
ln -s $NAME $SYMLINK_OUTPUT
popd
}

do_copy()
Expand Down Expand Up @@ -1381,15 +1345,8 @@ do_build()
do_oe_uivm "$path" ;;
ndvm)
do_oe_ndvm "$path" ;;
nilfvm)
do_oe_nilfvm "$path" ;;
vpnvm)
# for retro-compatibility
do_oe_nilfvm "$path" ;;
syncvm)
do_oe_syncvm "$path" ;;
syncui)
do_oe_syncui "$path" ;;
uivmcp)
do_oe_uivm_copy "$path" ;;
ndvmcp)
Expand All @@ -1398,8 +1355,6 @@ do_build()
do_oe_vpnvm_copy "$path" ;;
syncvmcp)
do_oe_syncvm_copy "$path" ;;
syncuicp)
do_oe_syncui_copy "$path" ;;
xctools*)
do_xctools "$path/xctools" ;;
debian)
Expand Down
4 changes: 2 additions & 2 deletions example-config
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
BRANCH="master"

# OpenEmbedded branch to build.
OE_BRANCH="pyro"
OE_BRANCH="zeus"

# OpenXT git repositories. Optionally replace with a local mirror.
OPENXT_GIT_MIRROR="github.com/OpenXT"
Expand All @@ -12,7 +12,7 @@ OPENXT_GIT_PROTOCOL="git"
OE_GIT_MIRROR=

# Bitbake default version to checkout
BB_BRANCH="1.34"
BB_BRANCH="1.44"
# Bitbake tag to override checkout
#BITBAKE_TAG="1.34"

Expand Down