fluent-bit: link against Nix dependencies, fix Darwin builds, and add NixOS module

[commiterate]

Link against Nix dependencies, fix Darwin builds, and add NixOS module (upstreams commiterate/nix-fluent-bit).

fluent-bit vends its dependencies by keeping a copy of their sources under lib/ in their source tree and building them from source. All of the dependencies their CMake setup can source from the system are already Nix packages (e.g. libbacktrace, luajit).

Darwin builds have also been silently broken for awhile now.

Things done

• Built on platform(s)
  • x86_64-linux
  • aarch64-linux
  • x86_64-darwin
  • aarch64-darwin
• For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)
  • sandbox = relaxed
  • sandbox = true
• Tested, as applicable:
  • NixOS test(s) (look inside nixos/tests)
  • and/or package tests
  • or, for functions and "core" functionality, tests in lib/tests or pkgs/test
  • made sure NixOS tests are linked to the relevant packages
• Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
• Tested basic functionality of all binary files (usually in ./result/bin/)
• 25.05 Release Notes (or backporting 24.11 and 25.05 Release notes)
  • (Package updates) Added a release notes entry if the change is major or breaking
  • (Module updates) Added a release notes entry if the change is significant
  • (Module addition) Added a release notes entry if adding a new NixOS module
• Fits CONTRIBUTING.md.

---

Add a 👍 reaction to pull requests you find important.

[commiterate]

@samrose @fpletz This is tentative. Will adjust if either of you aren't interested in maintaining this alongside the fluent-bit package.

[fpletz]

Unfortunately, I'm not using fluent-bit anymore. Please remove me because I can't really maintain it. AFAIK @samrose wasn't really active in the last few years maintaining this package so if he doesn't respond it's probably best to remove him too.

You can add yourself and I'll be happy to review your changes though. 😄

[commiterate]

Removed you from the ci/OWNERS file and the package maintainers list. I'll keep @samrose in the meanwhile.

[fpletz]

This will only set TimeoutStopSec in the systemd service but not service.grace in the fluent-bit configuration file. While this is documented to a degree, the description does not mention this changes the timeout in the systemd service and thus kills the process. Calling the option the same as in the configuration makes this behavior pretty unintuitive.

IMHO this won't be a problem in most cases anyway since the default for TimeoutStopSec (or DefaultTimeoutStopSec) is 90s. If users change this, they have to deal with the consequences.

I would either remove this or explicitly mention in the documentation what it does and rename it accordingly.

[commiterate]

Hmm how about renaming it graceLimit and changing the type from int to null | int | str with the default set to null?

That way we default to DefaultTimeoutStopSec but still allow overrides (e.g. in case DefaultTimeoutStopSec is lowered or a user wants a more aggressive shutdown timeout).

If we omit it, then people need to set systemd.services.fluent-bit.serviceConfig.TimeoutStopSec like how I'm setting LogsDirectory in the NixOS module test. It's not the end of the world, but it feels a bit awkward needing to reach into the module yourself compared to having an exposed interface.

Setting it via configuration.service.grace is a non-option IMO since it looks confusing when both configuration and configurationFile are set and configuration.service.grace still has some effect.

[commiterate]

We actually don't need this anymore.

fluent/fluent-bit@36c46b7

Same with some of the fluent-bit CMake feature flags which are On by default.

The package also needs to be migrated to the refactored Darwin SDKs.

@fpletz Given the long turnaround for getting changes outside of version bumps merged into Nixpkgs right now, should I just expand the scope of this PR to also refactor the package definition?

[commiterate]

Might not do that actually since the fluent-bit build setup is...something to behold.

[commiterate]

Well, I tried it anyways.

pkgs/by-name/fl/fluent-bit/package.nix

{
  lib,
  bison,
  c-ares,
  cmake,
  curl,
  fetchFromGitHub,
  flex,
  git,
  jemalloc,
  libbacktrace,
  libbpf,
  libedit,
  libnghttp2,
  libyaml,
  luajit,
  nix-update-script,
  nixosTests,
  openssl,
  pkg-config,
  postgresql,
  rdkafka,
  stdenv,
  systemd,
  versionCheckHook,
  zlib,
  zstd,
}:

stdenv.mkDerivation rec {
  pname = "fluent-bit";
  version = "3.2.2";

  src = fetchFromGitHub {
    owner = "fluent";
    repo = "fluent-bit";
    rev = "refs/tags/v${version}";
    hash = "sha256-BnrULjcWVBAOHfxlmd1RTQ8gfwlfZcwrUyLU27/9Z3M=";
  };

  # `src/CMakeLists.txt` installs fluent-bit's systemd unit files at the path in the `SYSTEMD_UNITDIR` CMake variable.
  #
  # The initial value of `SYSTEMD_UNITDIR` is set in `cmake/FindJournald` which uses pkg-config to find the systemd
  # unit directory. `src/CMakeLists.txt` only sets `SYSTEMD_UNITDIR` to `/lib/systemd` if it's unset.
  #
  # Unfortunately, this resolves to systemd's Nix store path which is immutable. Consequently, CMake fails when trying
  # to install fluent-bit's systemd unit files to the systemd Nix store path.
  #
  # We fix this by replacing `${SYSTEMD_UNITDIR}` instances in `src/CMakeLists.txt`.
  postPatch = ''
    substituteInPlace src/CMakeLists.txt \
      --replace-fail \''${SYSTEMD_UNITDIR} $out/lib/systemd
  '';

  # The source build documentation covers some dependencies and CMake options.
  #
  # - Linux: https://docs.fluentbit.io/manual/installation/sources/build-and-install
  # - Darwin: https://docs.fluentbit.io/manual/installation/macos#compile-from-source
  #
  # Unfortunately, fluent-bit vends many dependencies (e.g. luajit) as source files and tries to compile them by
  # default, with none of their dependencies and CMake options documented.
  #
  # Fortunately, there's the undocumented `FLB_PREFER_SYSTEM_LIBS` CMake option to link against system libraries for
  # some dependencies.
  #
  # See https://github.com/fluent/fluent-bit/blob/v3.2.3/CMakeLists.txt#L211-L218.
  #
  # Like `FLB_PREFER_SYSTEM_LIBS`, several CMake options aren't documented.
  #
  # See https://github.com/fluent/fluent-bit/blob/v3.2.3/CMakeLists.txt#L111-L157.
  #
  # The CMake options may differ across target platforms. We'll stick to the minimum.
  #
  # See https://github.com/fluent/fluent-bit/tree/v3.2.3/packaging/distros.

  nativeBuildInputs = [
    bison
    cmake
    flex
    git
    libedit
    pkg-config
  ];

  buildInputs =
    [
      c-ares
      # Needed by rdkafka.
      curl
      jemalloc
      libbacktrace
      libnghttp2
      libyaml
      luajit
      openssl
      postgresql
      rdkafka
      # Needed by rdkafka.
      zlib
      # Needed by rdkafka.
      zstd
    ]
    ++ lib.optionals stdenv.hostPlatform.isLinux [
      # libbpf doesn't build for Darwin yet.
      libbpf
      systemd
    ];

  cmakeFlags = [
    "-DFLB_RELEASE=Yes"
    "-DFLB_PREFER_SYSTEM_LIBS=Yes"
  ];

  outputs = [
    "out"
    "dev"
  ];

  doInstallCheck = true;

  nativeInstallCheckInputs = [ versionCheckHook ];

  versionCheckProgram = "${builtins.placeholder "out"}/bin/${meta.mainProgram}";

  versionCheckProgramArg = "--version";

  passthru = {
    tests = lib.optionalAttrs stdenv.isLinux {
      inherit (nixosTests) fluent-bit;
    };

    updateScript = nix-update-script { };
  };

  meta = {
    description = "Fast and lightweight logs and metrics processor for Linux, BSD, OSX and Windows";
    homepage = "https://fluentbit.io";
    license = lib.licenses.asl20;
    mainProgram = "fluent-bit";
    maintainers = with lib.maintainers; [ samrose ];
  };
}

Unlike the current definition which builds all vendored dependencies from the copied source in the fluent-bit repository, this links against existing Nix packages (e.g. luajit).

Darwin build is still broken though due to libc pthread not being available for some reason.

nix-build -A fluent-bit on aarch64-darwin

Running phase: unpackPhase
unpacking source archive /nix/store/6ci9v5ihxcf1wdhs9cp23alkwfa20wp6-source
source root is source
Running phase: patchPhase
Running phase: updateAutotoolsGnuConfigScriptsPhase
Updating Autotools / GNU config script to a newer upstream version: ./lib/libbacktrace-8602fda/config.sub
Updating Autotools / GNU config script to a newer upstream version: ./lib/onigmo/config.sub
Updating Autotools / GNU config script to a newer upstream version: ./lib/jemalloc-5.3.0/build-aux/config.sub
Updating Autotools / GNU config script to a newer upstream version: ./lib/c-ares-1.33.1/config/config.sub
Updating Autotools / GNU config script to a newer upstream version: ./lib/libbacktrace-8602fda/config.guess
Updating Autotools / GNU config script to a newer upstream version: ./lib/onigmo/config.guess
Updating Autotools / GNU config script to a newer upstream version: ./lib/jemalloc-5.3.0/build-aux/config.guess
Updating Autotools / GNU config script to a newer upstream version: ./lib/c-ares-1.33.1/config/config.guess
Running phase: configurePhase
fixing cmake files...
cmake flags: -DCMAKE_FIND_USE_SYSTEM_PACKAGE_REGISTRY=OFF -DCMAKE_FIND_USE_PACKAGE_REGISTRY=OFF -DCMAKE_EXPORT_NO_PACKAGE_REGISTRY=ON -DCMAKE_BUILD_TYPE=Release -DBUILD_TESTING=OFF -DCMAKE_INSTALL_LOCALEDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/share/locale -DCMAKE_INSTALL_LIBEXECDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/libexec -DCMAKE_INSTALL_LIBDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/lib -DCMAKE_INSTALL_DOCDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/share/doc/fluent-bit -DCMAKE_INSTALL_INFODIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/share/info -DCMAKE_INSTALL_MANDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/share/man -DCMAKE_INSTALL_OLDINCLUDEDIR=/nix/store/2kpi2sfq8s3pzs5nzy719ndha9z8hngf-fluent-bit-3.2.2-dev/include -DCMAKE_INSTALL_INCLUDEDIR=/nix/store/2kpi2sfq8s3pzs5nzy719ndha9z8hngf-fluent-bit-3.2.2-dev/include -DCMAKE_INSTALL_SBINDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/sbin -DCMAKE_INSTALL_BINDIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/bin -DCMAKE_INSTALL_NAME_DIR=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2/lib -DCMAKE_POLICY_DEFAULT_CMP0025=NEW -DCMAKE_FIND_FRAMEWORK=LAST -DCMAKE_STRIP=/nix/store/l8n4jk7l60yb7jag3i17bz57s7rvf0rd-clang-wrapper-16.0.6/bin/strip -DCMAKE_RANLIB=/nix/store/l8n4jk7l60yb7jag3i17bz57s7rvf0rd-clang-wrapper-16.0.6/bin/ranlib -DCMAKE_AR=/nix/store/l8n4jk7l60yb7jag3i17bz57s7rvf0rd-clang-wrapper-16.0.6/bin/ar -DCMAKE_C_COMPILER=clang -DCMAKE_CXX_COMPILER=clang++ -DCMAKE_INSTALL_PREFIX=/nix/store/l6a4r8dl8qhk7gagd0c3w3kxi1b6wi75-fluent-bit-3.2.2 -DFLB_RELEASE=Yes -DFLB_PREFER_SYSTEM_LIBS=Yes
-- The C compiler identification is Clang 16.0.6
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working C compiler: /nix/store/l8n4jk7l60yb7jag3i17bz57s7rvf0rd-clang-wrapper-16.0.6/bin/clang - skipped
-- Detecting C compile features
-- Detecting C compile features - done
-- Found PkgConfig: /nix/store/3gakp4idgx224rzx23y6502pryd5y6qs-pkg-config-wrapper-0.29.2/bin/pkg-config (found version "0.29.2")
Package systemd was not found in the pkg-config search path.
Perhaps you should add the directory containing `systemd.pc'
to the PKG_CONFIG_PATH environment variable
No package 'systemd' found
-- Could NOT find Journald (missing: JOURNALD_LIBRARY JOURNALD_INCLUDE_DIR) 
-- Found Monkey: /tmp/nix-build-fluent-bit-3.2.2.drv-0/source/lib/monkey/include
-- Looking for sys/wait.h
-- Looking for sys/wait.h - found
-- Found LibEdit: /nix/store/bd767aa5cg331m3xh64c42xxf2jd2hh8-libedit-20240808-3.1-dev/include (found version ".")
-- Found Git: /nix/store/i7460fczrp4ac460v0gibqddj7n8w8iz-git-2.47.0/bin/git (found version "2.47.0")
-- Git hash: 
-- Found FLEX: /nix/store/0ag85xd3gylpgcwv161kdhrvbn8m927v-flex-2.6.4/bin/flex (found suitable version "2.6.4", minimum required is "2")
-- Found BISON: /nix/store/gynxhf1z1h7610ddq5fa5dpk5jnqs1xl-bison-3.8.2/bin/bison (found suitable version "3.8.2", minimum required is "3")
WarningIPO is not supported on this platform
-- Enabling FLB_REGEX since FLB_PARSER requires
-- Performing Test CFL_HAVE_TIMESPEC_GET
-- Performing Test CFL_HAVE_TIMESPEC_GET - Failed
-- Performing Test CFL_HAVE_GMTIME_R
-- Performing Test CFL_HAVE_GMTIME_R - Failed
-- Performing Test CFL_HAVE_GMTIME_S
-- Performing Test CFL_HAVE_GMTIME_S - Failed
-- Performing Test CFL_HAVE_CLOCK_GET_TIME
-- Performing Test CFL_HAVE_CLOCK_GET_TIME - Failed
CMake Deprecation Warning at lib/cfl/lib/xxhash/cmake_unofficial/CMakeLists.txt:8 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- xxHash build type: RelWithDebInfo
CMake Deprecation Warning at lib/fluent-otel-proto/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at lib/msgpack-c/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at lib/miniz/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_VISIBILITY - Failed
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY
-- Performing Test COMPILER_HAS_HIDDEN_INLINE_VISIBILITY - Failed
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR
-- Performing Test COMPILER_HAS_DEPRECATED_ATTR - Failed
-- Performing Test COMPILER_HAS_DEPRECATED
-- Performing Test COMPILER_HAS_DEPRECATED - Failed
CMake Deprecation Warning at lib/lwrb/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


Entering /tmp/nix-build-fluent-bit-3.2.2.drv-0/source/lib/lwrb/lwrb/CMakeLists.txt
Exiting /tmp/nix-build-fluent-bit-3.2.2.drv-0/source/lib/lwrb/lwrb/CMakeLists.txt
CMake Deprecation Warning at lib/tutf8e/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at lib/snappy-fef67ac/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


CMake Deprecation Warning at lib/cmetrics/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test CMT_HAVE_TIMESPEC_GET
-- Performing Test CMT_HAVE_TIMESPEC_GET - Failed
-- Performing Test CMT_HAVE_GMTIME_R
-- Performing Test CMT_HAVE_GMTIME_R - Failed
-- Performing Test CMT_HAVE_GMTIME_S
-- Performing Test CMT_HAVE_GMTIME_S - Failed
-- Performing Test CMT_HAVE_CLOCK_GET_TIME
-- Performing Test CMT_HAVE_CLOCK_GET_TIME - Failed
-- Performing Test CMT_HAVE_MSGPACK
-- Performing Test CMT_HAVE_MSGPACK - Failed
-- Performing Test CMT_HAVE_CFL
-- Performing Test CMT_HAVE_CFL - Failed
-- Performing Test CMT_HAVE_FLUENT_OTEL_PROTO
-- Performing Test CMT_HAVE_FLUENT_OTEL_PROTO - Failed
CMake Error at lib/cmetrics/CMakeLists.txt:240 (add_subdirectory):
  add_subdirectory given source "lib/cfl" which is not an existing directory.


CMake Error at lib/cmetrics/CMakeLists.txt:279 (add_subdirectory):
  add_subdirectory given source "lib/fluent-otel-proto" which is not an
  existing directory.


CMake Deprecation Warning at lib/ctraces/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test CTR_HAVE_TIMESPEC_GET
-- Performing Test CTR_HAVE_TIMESPEC_GET - Failed
-- Performing Test CTR_HAVE_GMTIME_R
-- Performing Test CTR_HAVE_GMTIME_R - Failed
-- Performing Test CTR_HAVE_GMTIME_S
-- Performing Test CTR_HAVE_GMTIME_S - Failed
-- Performing Test CTR_HAVE_CLOCK_GET_TIME
-- Performing Test CTR_HAVE_CLOCK_GET_TIME - Failed
-- Performing Test CTR_HAVE_GETRANDOM
-- Performing Test CTR_HAVE_GETRANDOM - Failed
-- Performing Test CTR_HAVE_MSGPACK
-- Performing Test CTR_HAVE_MSGPACK - Failed
-- Performing Test CTR_HAVE_CFL
-- Performing Test CTR_HAVE_CFL - Failed
-- Performing Test CTR_HAVE_FLUENT_OTEL_PROTO
-- Performing Test CTR_HAVE_FLUENT_OTEL_PROTO - Failed
CMake Error at lib/ctraces/CMakeLists.txt:199 (add_subdirectory):
  add_subdirectory given source "lib/cfl" which is not an existing directory.


CMake Error at lib/ctraces/CMakeLists.txt:208 (add_subdirectory):
  add_subdirectory given source "lib/fluent-otel-proto" which is not an
  existing directory.


CMake Deprecation Warning at lib/cprofiles/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test CPROF_HAVE_TIMESPEC_GET
-- Performing Test CPROF_HAVE_TIMESPEC_GET - Failed
-- Performing Test CPROF_HAVE_GMTIME_R
-- Performing Test CPROF_HAVE_GMTIME_R - Failed
-- Performing Test CPROF_HAVE_GMTIME_S
-- Performing Test CPROF_HAVE_GMTIME_S - Failed
-- Performing Test CPROF_HAVE_CLOCK_GET_TIME
-- Performing Test CPROF_HAVE_CLOCK_GET_TIME - Failed
-- Performing Test CPROF_HAVE_CFL
-- Performing Test CPROF_HAVE_CFL - Failed
-- Performing Test CPROF_HAVE_FLUENT_OTEL_PROTO
-- Performing Test CPROF_HAVE_FLUENT_OTEL_PROTO - Failed
CMake Error at lib/cprofiles/CMakeLists.txt:189 (add_subdirectory):
  add_subdirectory given source "lib/cfl" which is not an existing directory.


CMake Error at lib/cprofiles/CMakeLists.txt:228 (add_subdirectory):
  add_subdirectory given source "lib/fluent-otel-proto" which is not an
  existing directory.


-- Checking for module 'libnghttp2>=1.0.0'
--   Found libnghttp2, version 1.64.0
-- Checking for module 'libcares>=1.18.0'
--   Found libcares, version 1.27.0
CMake Deprecation Warning at lib/chunkio/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test CIO_HAVE_GETPAGESIZE
-- Performing Test CIO_HAVE_GETPAGESIZE - Failed
-- Performing Test CIO_HAVE_FALLOCATE
-- Performing Test CIO_HAVE_FALLOCATE - Failed
-- Performing Test CIO_HAVE_POSIX_FALLOCATE
-- Performing Test CIO_HAVE_POSIX_FALLOCATE - Failed
-- FLB Event loop backend > auto discovery (Monkey library)
CMake Deprecation Warning at lib/monkey/CMakeLists.txt:2 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Performing Test CMAKE_HAVE_LIBC_PTHREAD
-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - not found
-- Check if compiler accepts -pthread
-- Check if compiler accepts -pthread - no
-- Could NOT find Threads (missing: Threads_FOUND) 
-- Looking for accept4
-- Looking for accept4 - not found
-- Looking for execinfo.h
-- Looking for execinfo.h - not found
-- Performing Test HAVE_C_TLS
-- Performing Test HAVE_C_TLS - Failed
-- Performing Test MK_HAVE_VALGRIND
-- Performing Test MK_HAVE_VALGRIND - Failed
-- Performing Test HAVE_REGEX
-- Performing Test HAVE_REGEX - Failed
-- Looking for posix_memalign
-- Looking for posix_memalign - not found
-- Looking for posix_memalign
-- Looking for posix_memalign - not found
-- Performing Test HAVE_STAT_H
-- Performing Test HAVE_STAT_H - Failed
-- Performing Test HAVE_SYS_UIO_H
-- Performing Test HAVE_SYS_UIO_H - Failed
-- Performing Test HAVE_MEMMEM
-- Performing Test HAVE_MEMMEM - Failed
-- Performing Test HAVE_UNISTD_H
-- Performing Test HAVE_UNISTD_H - Failed
-- Performing Test HAVE_SELECT
-- Performing Test HAVE_SELECT - Failed
-- Performing Test HAVE_POLL
-- Performing Test HAVE_POLL - Failed
-- Performing Test HAVE_KQUEUE
-- Performing Test HAVE_KQUEUE - Failed
-- Performing Test HAVE_EPOLL
-- Performing Test HAVE_EPOLL - Failed
-- Performing Test HAVE_TIMERFD_CREATE
-- Performing Test HAVE_TIMERFD_CREATE - Failed
-- Performing Test HAVE_EVENTFD
-- Performing Test HAVE_EVENTFD - Failed
-- Performing Test MK_HAVE_MEMRCHR
-- Performing Test MK_HAVE_MEMRCHR - Failed
-- Plugin dirlisting enabled 
-- Plugin liana enabled [== static ==]
-- Plugin mandril enabled 
-- LINKING monkey-liana-static;
-- Found OpenSSL: /nix/store/1a4j0c5xn6xl53z77sifkncg3fing1qj-openssl-3.3.2/lib/libcrypto.dylib (found version "3.3.2")
-- The ASM compiler identification is Clang with GNU-like command-line
-- Found assembler: /nix/store/l8n4jk7l60yb7jag3i17bz57s7rvf0rd-clang-wrapper-16.0.6/bin/clang
-- Performing Test FLB_HAVE_VALGRIND
-- Performing Test FLB_HAVE_VALGRIND - Failed
-- Performing Test FLB_HAVE_FORK
-- Performing Test FLB_HAVE_FORK - Failed
-- Performing Test FLB_HAVE_TIMESPEC_GET
-- Performing Test FLB_HAVE_TIMESPEC_GET - Failed
-- Performing Test FLB_HAVE_GMTOFF
-- Performing Test FLB_HAVE_GMTOFF - Failed
-- Performing Test FLB_HAVE_CLOCK_GET_TIME
-- Performing Test FLB_HAVE_CLOCK_GET_TIME - Failed
-- Performing Test FLB_HAVE_UNIX_SOCKET
-- Performing Test FLB_HAVE_UNIX_SOCKET - Failed
-- Performing Test FLB_HAVE_ATTRIBUTE_ALLOC_SIZE
-- Performing Test FLB_HAVE_ATTRIBUTE_ALLOC_SIZE - Failed
-- libbacktrace found (/nix/store/qgla2qxwqdzh31vrzla3d4gdp283a13q-libbacktrace-0-unstable-2024-03-02/lib/libbacktrace.dylib)
-- Checking for module 'rdkafka>=2.3.0'
--   Found rdkafka, version 2.6.1
CMake Deprecation Warning at lib/onigmo/CMakeLists.txt:1 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.


-- Using python executable is: PYTHON_EXECUTABLE-NOTFOUND
-- Overriding setttings with macos-setup.cmake
-- Looking for sys/types.h
-- Looking for sys/types.h - not found
-- Looking for stdint.h
-- Looking for stdint.h - not found
-- Looking for stddef.h
-- Looking for stddef.h - not found
-- Check size of int
-- Check size of int - failed
-- Check size of short
-- Check size of short - failed
-- Check size of long
-- Check size of long - failed
-- Check size of void*
-- Check size of void* - failed
-- Check size of long long
-- Check size of long long - failed
-- Check size of size_t
-- Check size of size_t - failed
-- Performing Test STDC_HEADERS
-- Performing Test STDC_HEADERS - Failed
-- Looking for dlfcn.h
-- Looking for dlfcn.h - not found
-- Looking for inttypes.h
-- Looking for inttypes.h - not found
-- Looking for memory.h
-- Looking for memory.h - not found
-- Looking for stdlib.h
-- Looking for stdlib.h - not found
-- Looking for strings.h
-- Looking for strings.h - not found
-- Looking for string.h
-- Looking for string.h - not found
-- Looking for sys/stat.h
-- Looking for sys/stat.h - not found
-- Performing Test TIME_WITH_SYS_TIME
-- Performing Test TIME_WITH_SYS_TIME - Failed
-- Looking for sys/time.h
-- Looking for sys/time.h - not found
-- Looking for sys/times.h
-- Looking for sys/times.h - not found
-- Looking for alloca
-- Looking for alloca - not found
-- Looking for memcmp
-- Looking for memcmp - not found
-- Checking for module 'luajit>=2.1.0'
--   Found luajit, version 2.1.1713773202
-- Found PostgreSQL: /nix/store/c3a1ij8vyxv5fyg7qp6la2mn54jn84sn-postgresql-16.5-lib/lib/libpq.dylib (found version "16.5")
-- Performing Test FLB_HAVE_C_TLS
-- Performing Test FLB_HAVE_C_TLS - Failed
-- Performing Test FLB_HAVE_ACCEPT4
-- Performing Test FLB_HAVE_ACCEPT4 - Failed
-- Performing Test FLB_HAVE_INOTIFY
-- Performing Test FLB_HAVE_INOTIFY - Failed
-- Looking for getentropy
-- Looking for getentropy - not found
-- Looking for getentropy
-- Looking for getentropy - not found
-- Check size of unsigned __int128
-- Check size of unsigned __int128 - failed
-- Check size of unsigned int __attribute__((mode(TI)))
-- Check size of unsigned int __attribute__((mode(TI))) - failed
-- Build as target AARCH64
-- Build Configurations:
     Build as target AARCH64
     CMAKE_BUILD_TYPE RelWithDebInfo
     WAMR Interpreter enabled
     WAMR AOT enabled
     WAMR Fast JIT disabled
     WAMR LLVM ORC JIT disabled
     Libc builtin enabled
     Libc WASI enabled
     Fast interpreter enabled
     Multiple modules disabled
     Bulk memory feature enabled
     Shared memory enabled
     Thread manager enabled
     Lib pthread enabled
     Wakeup of blocking operations enabled
     Reference types disabled
     Module instance context enabled
-- Could NOT find Threads (missing: Threads_FOUND) 
-- Could NOT find Threads (missing: Threads_FOUND) 
-- Could NOT find Threads (missing: Threads_FOUND) 
-- Configuring incomplete, errors occurred!
error: builder for '/nix/store/wzf1j58vj7ili3i396q61nkf8xmakbnp-fluent-bit-3.2.2.drv' failed with exit code 1;
       last 25 log lines:
       > -- Check size of unsigned int __attribute__((mode(TI)))
       > -- Check size of unsigned int __attribute__((mode(TI))) - failed
       > -- Build as target AARCH64
       > -- Build Configurations:
       >      Build as target AARCH64
       >      CMAKE_BUILD_TYPE RelWithDebInfo
       >      WAMR Interpreter enabled
       >      WAMR AOT enabled
       >      WAMR Fast JIT disabled
       >      WAMR LLVM ORC JIT disabled
       >      Libc builtin enabled
       >      Libc WASI enabled
       >      Fast interpreter enabled
       >      Multiple modules disabled
       >      Bulk memory feature enabled
       >      Shared memory enabled
       >      Thread manager enabled
       >      Lib pthread enabled
       >      Wakeup of blocking operations enabled
       >      Reference types disabled
       >      Module instance context enabled
       > -- Could NOT find Threads (missing: Threads_FOUND)
       > -- Could NOT find Threads (missing: Threads_FOUND)
       > -- Could NOT find Threads (missing: Threads_FOUND)
       > -- Configuring incomplete, errors occurred!
       For full logs, run 'nix log /nix/store/wzf1j58vj7ili3i396q61nkf8xmakbnp-fluent-bit-3.2.2.drv'.

[commiterate]

@reckenrode found that it's the FLB_SECURITY CMake option which causes bad linker options for Clang to be set. Toggling it off for Clang builds fixes it.

Here's the working package definition:

pkgs/by-name/fl/fluent-bit/package.nix

{
  lib,
  bison,
  c-ares,
  cmake,
  curl,
  fetchFromGitHub,
  flex,
  jemalloc,
  libbacktrace,
  libbpf,
  libnghttp2,
  libyaml,
  luajit,
  nix-update-script,
  nixosTests,
  openssl,
  pkg-config,
  postgresql,
  rdkafka,
  stdenv,
  systemd,
  versionCheckHook,
  zlib,
  zstd,
}:

stdenv.mkDerivation rec {
  pname = "fluent-bit";
  version = "3.2.2";

  src = fetchFromGitHub {
    owner = "fluent";
    repo = "fluent-bit";
    rev = "refs/tags/v${version}";
    hash = "sha256-BnrULjcWVBAOHfxlmd1RTQ8gfwlfZcwrUyLU27/9Z3M=";
  };

  # `src/CMakeLists.txt` installs fluent-bit's systemd unit files at the path in the `SYSTEMD_UNITDIR` CMake variable.
  #
  # The initial value of `SYSTEMD_UNITDIR` is set in `cmake/FindJournald` which uses pkg-config to find the systemd
  # unit directory. `src/CMakeLists.txt` only sets `SYSTEMD_UNITDIR` to `/lib/systemd/system` if it's unset.
  #
  # Unfortunately, this resolves to systemd's Nix store path which is immutable. Consequently, CMake fails when trying
  # to install fluent-bit's systemd unit files to the systemd Nix store path.
  #
  # We fix this by replacing `${SYSTEMD_UNITDIR}` instances in `src/CMakeLists.txt`.
  postPatch = ''
    substituteInPlace src/CMakeLists.txt \
      --replace-fail \''${SYSTEMD_UNITDIR} $out/lib/systemd
  '';

  # The source build documentation covers some dependencies and CMake options.
  #
  # - Linux: https://docs.fluentbit.io/manual/installation/sources/build-and-install
  # - Darwin: https://docs.fluentbit.io/manual/installation/macos#compile-from-source
  #
  # Unfortunately, fluent-bit vends many dependencies (e.g. luajit) as source files and tries to compile them by
  # default, with none of their dependencies and CMake options documented.
  #
  # Fortunately, there's the undocumented `FLB_PREFER_SYSTEM_LIBS` CMake option to link against system libraries for
  # some dependencies.
  #
  # See https://github.com/fluent/fluent-bit/blob/v3.2.3/CMakeLists.txt#L211-L218.
  #
  # Like `FLB_PREFER_SYSTEM_LIBS`, several CMake options aren't documented.
  #
  # See https://github.com/fluent/fluent-bit/blob/v3.2.3/CMakeLists.txt#L111-L157.
  #
  # The CMake options may differ across target platforms. We'll stick to the minimum.
  #
  # See https://github.com/fluent/fluent-bit/tree/v3.2.3/packaging/distros.

  strictDeps = true;

  nativeBuildInputs = [
    bison
    cmake
    flex
    pkg-config
  ];

  buildInputs =
    [
      c-ares
      # Needed by rdkafka.
      curl
      jemalloc
      libbacktrace
      libnghttp2
      libyaml
      luajit
      openssl
      postgresql
      rdkafka
      # Needed by rdkafka.
      zlib
      # Needed by rdkafka.
      zstd
    ]
    ++ lib.optionals stdenv.hostPlatform.isLinux [
      # libbpf doesn't build for Darwin yet.
      libbpf
      systemd
    ];

  cmakeFlags =
    [
      "-DFLB_RELEASE=Yes"
      "-DFLB_PREFER_SYSTEM_LIBS=Yes"
    ]
    ++ lib.optionals stdenv.cc.isClang [
      # FLB_SECURITY causes bad linker options for Clang to be set.
      "-DFLB_SECURITY=Off"
    ];

  outputs = [
    "out"
    "dev"
  ];

  doInstallCheck = true;

  nativeInstallCheckInputs = [ versionCheckHook ];

  versionCheckProgram = "${builtins.placeholder "out"}/bin/${meta.mainProgram}";

  versionCheckProgramArg = "--version";

  passthru = {
    tests = lib.optionalAttrs stdenv.isLinux {
      inherit (nixosTests) fluent-bit;
    };

    updateScript = nix-update-script { };
  };

  meta = {
    description = "Fast and lightweight logs and metrics processor for Linux, BSD, OSX and Windows";
    homepage = "https://fluentbit.io";
    license = lib.licenses.asl20;
    mainProgram = "fluent-bit";
    maintainers = with lib.maintainers; [ samrose ];
  };
}

[commiterate]

Going to increase the scope of this PR since that package definition refactor fixes the current broken Darwin builds.