-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fixing buildlibs.sh script + cmark reference in config.nims + prebuilt arm linux libs #113
Conversation
f80fdc1
to
0fc4325
Compare
# if musl-gcc is already installed, use it | ||
existing_musl=$(which musl-gcc 2> /dev/null) | ||
if [[ -n "${existing_musl}" ]]; then |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
if musl-gcc
is already installed, we dont need to compile musl from scratch which requires more system deps like make
. for example avoids errors like:
#14 15.44 /tmp/nimble_9/githubcom_crashappseccon4mgit_#ms/libs/files/bin/buildlibs.sh: line 156: make: command not found
all 4 cases seem to be working as expected:
see below for logs: con4mcon4m
con4m
con4m in devmode
chalkchalk
chalk in devmode
|
What’s the exception it’s eating? Seems related to the one Theo was
getting?
…On Mon, Sep 18, 2023 at 9:56 AM Miroslav Shubernetskiy < ***@***.***> wrote:
all 4 cases seem to be working as expected:
- con4m local dev
- con4m nimble
- chalk local dev
- chalk nimble
see below for logs:
con4m
con4m nimble install (with existing musl):
***@***.***:/con4m# git log -1
commit 0fc4325 (HEAD -> ms/libs, origin/ms/libs)
Author: Miroslav Shubernetskiy ***@***.***>
Date: Fri Sep 15 15:56:12 2023 -0400
use existing musl-gcc if available
***@***.***:/con4m# nimble install
Warning: Package 'con4m' has an incorrect structure. It should contain a single directory hierarchy for source files, named 'con4mpkg', but file 'builtins.nim' is in a directory named 'con4m' instead. This will be an error in the future.
Hint: If 'con4m' contains source files for building 'con4m', rename it to 'con4mpkg'. Otherwise, prevent its installation by adding `skipDirs = @["con4m"]` to the .nimble file.
Verifying dependencies for ***@***.***
Info: Dependency on https://github.com/crashappsec/nimutils@#head already satisfied
Verifying dependencies for ***@***.***
Info: Dependency on ***@***.*** already satisfied
Verifying dependencies for ***@***.***
Installing ***@***.***
Building con4m/con4m using c backend
Linking existing musl-gcc: /usr/bin/musl-gcc -> /root/.local/c0/musl/bin/musl-gcc
All dependencies satisfied.
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/con4m/files/con4m/c42spec.nim(12, 22) Warning: imported and not used: 'run' [UnusedImport]
/con4m/files/con4m/doc.nim(2, 28) Warning: imported and not used: 'spec' [UnusedImport]
Success: con4m installed successfully.
con4m nimble install (without musl):
***@***.***:/con4m# git log -1
commit 0fc4325 (HEAD -> ms/libs, origin/ms/libs)
Author: Miroslav Shubernetskiy ***@***.***>
Date: Fri Sep 15 15:56:12 2023 -0400
use existing musl-gcc if available
***@***.***:/con4m# nimble install
Warning: Package 'con4m' has an incorrect structure. It should contain a single directory hierarchy for source files, named 'con4mpkg', but file 'builtins.nim' is in a directory named 'con4m' instead. This will be an error in the future.
Hint: If 'con4m' contains source files for building 'con4m', rename it to 'con4mpkg'. Otherwise, prevent its installation by adding `skipDirs = @["con4m"]` to the .nimble file.
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/nimutils@#head
Downloading https://github.com/crashappsec/nimutils using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Downloading https://github.com/nitely/nim-unicodedb using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: unicodedb installed successfully.
Installing ***@***.***
Success: nimutils installed successfully.
Installing ***@***.***
Building con4m/con4m using c backend
Downloading musl from: git://git.musl-libc.org/musl
Cloning into 'musl'...
remote: Counting objects: 37223, done.
remote: Compressing objects: 100% (16085/16085), done.
remote: Total 37223 (delta 27228), reused 28568 (delta 20733)
Receiving objects: 100% (37223/37223), 5.62 MiB | 6.70 MiB/s, done.
Resolving deltas: 100% (27228/27228), done.
Building musl
checking for C compiler... gcc
checking whether C compiler works... yes
<TRUNCATED>
creating config.mak... done
rm -rf obj lib
mkdir -p lib
<TRUNCATED>
sed -f ./tools/mkalltypes.sed ./arch/x86_64/bits/alltypes.h.in ./include/alltypes.h.in > obj/include/bits/alltypes.h
cp arch/x86_64/bits/syscall.h.in obj/include/bits/syscall.h
sed -n -e s/__NR_/SYS_/p < arch/x86_64/bits/syscall.h.in >> obj/include/bits/syscall.h
gcc -std=c99 -nostdinc -ffreestanding -fexcess-precision=standard -frounding-math -fno-strict-aliasing -Wa,--noexecstack -D_XOPEN_SOURCE=700 -I./arch/x86_64 -I./arch/generic -Iobj/src/internal -I./src/include -I./src/internal -Iobj/include -I./include -O2 -fno-align-jumps -fno-align-functions -fno-align-loops -fno-align-labels -fira-region=one -fira-hoist-pressure -freorder-blocks-algorithm=simple -fno-prefetch-loop-arrays -fno-tree-ch -pipe -fomit-frame-pointer -fno-unwind-tables -fno-asynchronous-unwind-tables -ffunction-sections -fdata-sections -Wno-pointer-to-int-cast -Werror=implicit-function-declaration -Werror=implicit-int -Werror=pointer-sign -Werror=pointer-arith -Werror=int-conversion -Werror=incompatible-pointer-types -Werror=discarded-qualifiers -Werror=discarded-array-qualifiers -Waddress -Warray-bounds -Wchar-subscripts -Wduplicate-decl-specifier -Winit-self -Wreturn-type -Wsequence-point -Wstrict-aliasing -Wunused-function -Wunused-label -Wunused-variable -fPIC -fno-stack-protector -DCRT -c -o obj/crt/Scrt1.o crt/Scrt1.c
cp obj/crt/Scrt1.o lib/Scrt1.o
<TRUNCATED>
ranlib lib/libc.a
rm -f lib/libm.a
ar rc lib/libm.a
rm -f lib/librt.a
ar rc lib/librt.a
rm -f lib/libpthread.a
ar rc lib/libpthread.a
rm -f lib/libcrypt.a
ar rc lib/libcrypt.a
rm -f lib/libutil.a
ar rc lib/libutil.a
rm -f lib/libxnet.a
ar rc lib/libxnet.a
rm -f lib/libresolv.a
ar rc lib/libresolv.a
rm -f lib/libdl.a
ar rc lib/libdl.a
sh tools/musl-gcc.specs.sh "/root/.local/c0/musl/include" "/root/.local/c0/musl/lib" "/lib/ld-musl-x86_64.so.1" > lib/musl-gcc.specs
printf '#!/bin/sh\nexec "${REALGCC:-gcc}" "$@" -specs "%s/musl-gcc.specs"\n' "/root/.local/c0/musl/lib" > obj/musl-gcc
chmod +x obj/musl-gcc
<TRUNCATED>
./tools/install.sh -D obj/musl-gcc /root/.local/c0/musl/bin/musl-gcc
Installed musl wrapper to: /root/.local/c0/musl/bin/musl-gcc
All dependencies satisfied.
Removing code (because CON4M_DEV is not set)
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/con4m/files/con4m/c42spec.nim(12, 22) Warning: imported and not used: 'run' [UnusedImport]
/con4m/files/con4m/doc.nim(2, 28) Warning: imported and not used: 'spec' [UnusedImport]
Success: con4m installed successfully.
con4m in devmode nimble build:
***@***.***:/con4m# CON4M_DEV=true nimble build
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/nimutils@#head
Downloading https://github.com/crashappsec/nimutils using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Downloading https://github.com/nitely/nim-unicodedb using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: unicodedb installed successfully.
Installing ***@***.***
Success: nimutils installed successfully.
Building con4m/con4m using c backend
Linking existing musl-gcc: /usr/bin/musl-gcc -> /root/.local/c0/musl/bin/musl-gcc
All dependencies satisfied.
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/con4m/files/con4m/c42spec.nim(12, 22) Warning: imported and not used: 'run' [UnusedImport]
/con4m/files/con4m/doc.nim(2, 28) Warning: imported and not used: 'spec' [UnusedImport]
chalk
chalk nimble install:
***@***.***:/chalk# git log -1
commit 41b90b2ec9b3171c766815f7eae10b189fd8063f (HEAD -> ms/static, origin/ms/static)
Author: Miroslav Shubernetskiy ***@***.***>
Date: Fri Sep 15 16:06:41 2023 -0400
removing staticlibs cache in favor of con4m vendored libs
as con4m now ships .a libs files we no longer need to install/cache
static libs separately which otherwise requires additional
system dependencies such as gcc, g++, etc
this also guarantees dockerfile is using the same version of
con4m everywhere for both ensuring static libs and compiling deps
***@***.***:/chalk# nimble install
Looking for changes to src/configs/chalk.c42spec
No change to chalk.c42spec
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/con4m.git@#ms/libs
Downloading https://github.com/crashappsec/con4m.git using git
Warning: Package 'con4m' has an incorrect structure. It should contain a single directory hierarchy for source files, named 'con4mpkg', but file 'builtins.nim' is in a directory named 'con4m' instead. This will be an error in the future.
Hint: If 'con4m' contains source files for building 'con4m', rename it to 'con4mpkg'. Otherwise, prevent its installation by adding `skipDirs = @["con4m"]` to the .nimble file.
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/nimutils@#head
Downloading https://github.com/crashappsec/nimutils using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Downloading https://github.com/nitely/nim-unicodedb using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: unicodedb installed successfully.
Installing ***@***.***
Success: nimutils installed successfully.
Installing ***@***.***
Building con4m/con4m using c backend
Linking existing musl-gcc: /usr/bin/musl-gcc -> /root/.local/c0/musl/bin/musl-gcc
All dependencies satisfied.
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/tmp/nimble_97/githubcom_crashappseccon4mgit_#ms/libs/files/con4m/c42spec.nim(12, 22) Warning: imported and not used: 'run' [UnusedImport]
/tmp/nimble_97/githubcom_crashappseccon4mgit_#ms/libs/files/con4m/doc.nim(2, 28) Warning: imported and not used: 'spec' [UnusedImport]
Success: con4m installed successfully.
Installing https://github.com/crashappsec/nimutils.git@#v0.1.3
Downloading https://github.com/crashappsec/nimutils.git using git
Verifying dependencies for ***@***.***
Info: Dependency on ***@***.*** already satisfied
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: nimutils installed successfully.
Installing ***@***.***
Downloading https://github.com/viega/zippy using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: zippy installed successfully.
Installing ***@***.***
Downloading https://github.com/aruZeta/QRgen using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: QRgen installed successfully.
Installing ***@***.***
Looking for changes to src/configs/chalk.c42spec
No change to chalk.c42spec
Building chalk_internal/chalk using c backend
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/chalk/src/plugin_load.nim(37, 12) template/generic instantiation of `loadPlugins` from here
/chalk/src/plugins/elf.nim(258, 43) Warning: conversion to enum with holes is unsafe: ProgramHeaderType(programHeader.headerType.value) [HoleEnumConv]
+ strip chalk
+ ./chalk --no-use-external-config --skip-command-report load default
info: Wrote bash auto-completion file to: /root/.local/share/bash_completion/completions/chalk.bash
info: Added sourcing of autocomplete to ~/.bash_completion file
info: Installing the default configuration file.
info: Configuration replaced in binary: /chalk/chalk
info: Updated configuration for /chalk/chalk
warn: When collecting run-time host info, plugin implementation metsys threw an exception it didn't handle.
info: Skipping the command report, because you said so.
Success: chalk_internal installed successfully.
chalk in devmode nimble build:
***@***.***:/chalk# CON4M_DEV=true nimble build
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/con4m.git@#ms/libs
Downloading https://github.com/crashappsec/con4m.git using git
Verifying dependencies for ***@***.***
Installing https://github.com/crashappsec/nimutils@#head
Downloading https://github.com/crashappsec/nimutils using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Downloading https://github.com/nitely/nim-unicodedb using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: unicodedb installed successfully.
Installing ***@***.***
Success: nimutils installed successfully.
Installing ***@***.***
Building con4m/con4m using c backend
Linking existing musl-gcc: /usr/bin/musl-gcc -> /root/.local/c0/musl/bin/musl-gcc
All dependencies satisfied.
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/tmp/nimble_55/githubcom_crashappseccon4mgit_#ms/libs/files/con4m/c42spec.nim(12, 22) Warning: imported and not used: 'run' [UnusedImport]
/tmp/nimble_55/githubcom_crashappseccon4mgit_#ms/libs/files/con4m/doc.nim(2, 28) Warning: imported and not used: 'spec' [UnusedImport]
Success: con4m installed successfully.
Installing https://github.com/crashappsec/nimutils.git@#v0.1.3
Downloading https://github.com/crashappsec/nimutils.git using git
Verifying dependencies for ***@***.***
Info: Dependency on ***@***.*** already satisfied
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: nimutils installed successfully.
Installing ***@***.***
Downloading https://github.com/viega/zippy using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: zippy installed successfully.
Installing ***@***.***
Downloading https://github.com/aruZeta/QRgen using git
Verifying dependencies for ***@***.***
Installing ***@***.***
Success: QRgen installed successfully.
con4m directory: /con4m
nimutils direct: /nimutils
nimble packages: /root/.nimble/pkgs2
con4m version: con4m-0.1.3-8e228b83d56f06fb3fab3194805d6e684a9db10b
nimutils vers: nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80
Copying new file: test.c
cp: cannot create regular file '/root/.nimble/pkgs2/con4m-0.1.3-8e228b83d56f06fb3fab3194805d6e684a9db10b/c/test.c': No such file or directory
Copying new file: con4m.h
cp: cannot create regular file '/root/.nimble/pkgs2/con4m-0.1.3-8e228b83d56f06fb3fab3194805d6e684a9db10b/c/con4m.h': No such file or directory
Looking for changes to src/configs/chalk.c42spec
No change to chalk.c42spec
Building chalk_internal/chalk using c backend
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/s3client.nim(63, 14) template/generic instantiation of `request` from here
/root/.nimble/pkgs2/nimutils-0.1.3-cb3670863a3505f8832726bc749a4a44782f3c80/nimutils/awsclient.nim(106, 27) template/generic instantiation of `request` from here
/src/nim-2.0.0/lib/pure/httpclient.nim(1122, 14) Warning: Deprecated since v1.5; use HttpMethod enum instead; string parameter httpMethod is deprecated [User]
/chalk/src/plugin_load.nim(37, 12) template/generic instantiation of `loadPlugins` from here
/chalk/src/plugins/elf.nim(258, 43) Warning: conversion to enum with holes is unsafe: ProgramHeaderType(programHeader.headerType.value) [HoleEnumConv]
+ strip chalk
+ ./chalk --no-use-external-config --skip-command-report load default
info: Wrote bash auto-completion file to: /root/.local/share/bash_completion/completions/chalk.bash
info: Added sourcing of autocomplete to ~/.bash_completion file
info: Installing the default configuration file.
info: Configuration replaced in binary: /chalk/chalk
info: Updated configuration for /chalk/chalk
warn: When collecting run-time host info, plugin implementation metsys threw an exception it didn't handle.
info: Skipping the command report, because you said so.
—
Reply to this email directly, view it on GitHub
<#113 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABELGQMAYM6A2YPX2SXIYVTX3BHKPANCNFSM6AAAAAA42CTMIA>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
which exception? |
It’s there in the output that there was one but it’s not being reported.
…On Mon, Sep 18, 2023 at 9:59 AM Miroslav Shubernetskiy < ***@***.***> wrote:
which exception?
—
Reply to this email directly, view it on GitHub
<#113 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ABELGQJHLXYTVXD4M6VDRHDX3BHVHANCNFSM6AAAAAA42CTMIA>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
running with
|
@viega this PR now includes CI job for generating staticlibs and adding them for |
Cool pretty sure I know what the exception is then, and unrelated to this, so thanks. |
@@ -139,6 +139,15 @@ function ensure_musl { | |||
if [[ ${OS} = "macosx" ]] ; then | |||
return | |||
fi | |||
if [[ ! -f ${MUSL_GCC} ]] ; then | |||
# if musl-gcc is already installed, use it |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't love this, they might not have permission to install the headers. Things can go wrong. They can delete the thing after they build it, and indeed the script does.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Wait this was merged???
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
vanilla apt-get install musl-tools
installs correct musl and I am able to use it in chalk docker builds
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
not sure what you mean if it was merged. PR is still open
what this fixes (initially documented in https://github.com/crashappsec/chalk-internal/pull/679#issuecomment-1720545706):
buildlibs.sh
no longer installs cmark lib, this removes it fromconfig.nims
. otherwise it would be attempting to link against something which is not thereCMakelists.txt
but cmake does not use that file and instead usesCMakeLists.txt
(noteL
vsl
). original file. without that it compiles all examples/etc which requires additional system libs without which you get errors like:musl-gcc
if it is already installed. otherwisebuildlibs.sh
attempts to install musl from source which requires additional deps likemake
which can leak to errors like:new features: