Skip to content

Commit

Permalink
Add pkg_sub_rpm rule for RPM subpackages (bazelbuild#824)
Browse files Browse the repository at this point in the history
* rpm: Add support for sub packages to make_rpm.py script

Before we can enable support for sub RPM building as part of a single
`pkg_rpm()` rule we must add the underlying support to make_pkg.py
which is the underlying driver for `pkg_rpm()`.

This covers three pieces:

 * specifying `buildsubdir` rpm variable
 * capturing multiple RPM files as outputs
 * injecting sub RPM definitions into specfile

* rpm: Factor out rpm_ctx helper

The various processing functions pass around a bunch of collections
everywhere which is a bit fragile.  This collects them together into a
struct to make it a bit less messy.

* rpm: Factor out _process_dep() helper function

_process_dep() handles processing an individual dep and is currently
called from the processing loop.  We'll need to re-use this logic for
processing individual sub RPMs as well so we want it in a helper.

* rpm: Capture generated output RPM files in rpm_ctx

Currently we only generate one RPM file, but once we generate sub RPM
files we'll need to be able to capture those outputs as well.  This
prepares us for that step.

* rpm: Add args for make_rpm to rpm_ctx

We'll need to add additional arguments to make_rpm for sub RPM
building.  It's easier to capture this in our context object than to
try to shuttle these bits around.

* rpm: Pass correct `--name` argument to make_rpm

If we don't pass the correct RPM name to `make_rpm.py` than we won't be
able to correctly determine the subrpm names.  Currently, the name is
only used by `make_rpm` to generate some progress output, so this
shouldn't break anything internally.

* rpm: Implementation of `pkg_sub_rpm` rule

This introduces a `pkg_sub_rpm` rule that allows us to generate and
capture RPM subpackages and generate them as part of a single RPM
invocation in lieu of cobbling this together from multiple RPM rules.
This has a few benefits:

- faster execution due to single rpmbuild invocation
- sharing configuration between RPMs in the same fashion as vanilla
  RPM building from a specfile
- will enable the proper construction of debuginfo RPMs in a later PR

The current implementation *only* works with non-specfile based rules
and currently allows for a subset of the general RPM configuration.

Internally, the process is for `pkg_sub_rpm` to generate a
PackageSubRPMInfo provider that will subsequently get consumed by the
`pkg_rpm` rule that generates the actual RPMs.  We re-use the internal
dependency processing logic that's used by the top-level RPM to
generate the content related to the sub-RPM.

* rpm: Update entry points for RPM rules to include pkg_sub_rpm

This change updates `rpm.bzl` in two ways to account for sub RPMs:

- it adds an entrypoint for `pkg_sub_rpm`

- it adds a check in `pkg_rpm` to assert incompatibility with
  `spec_file`

* examples: Add an example of how to use the subrpm rule

This provides a basic example using the `pkg_sub_rpm` rule to generate
multiple RPMs.

* Fix buildifier noise

* Fix make_rpm failures

* doc: Clean up sub-RPM docstring and add to doc_build

The initial docstring for pkg_sub_rpm is not great, so this change
fixes that while adding this to the list of rules to have
documentation generated for them.

* doc: Additional documentation for subrpms in pkg_rpm

This clarifies the `subrpms` attribute usage as well as indicating the
incompatibility with `spec_file` mode.

* Fix issue in subrpm passthrough

When adding the check to verify if we can use subrpms, this pass
through wasn't added.

* Add a basic test for pkg_sub_rpm

This introduces a basic test with a single sub RPM and main RPM each
containing a single source file.

* Tweaks to test

* Test fixes for CentOS 7

The `rpm` version on CentOS 7 doesn't work exactly the same way as
newer versions of rpm and requires a `-p` parameter to inspect
non-installed RPMs.  CentOS 7 also inserts a `Relocations` field that
we didn't see on other platforms so we'll filter that out to make our
test a bit more portable.

* Move PackageSubRPMInfoProvider into rpm_pfg.bzl

This is private to the RPM rules so should probably live there.
  • Loading branch information
kellyma2 authored Mar 15, 2024
1 parent 61132fe commit 2aa2b8e
Show file tree
Hide file tree
Showing 11 changed files with 659 additions and 190 deletions.
1 change: 1 addition & 0 deletions doc_build/BUILD
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,7 @@ ORDER = [
("common", None),
("pkg_deb", "//pkg/private/deb:deb.bzl"),
("pkg_deb_impl", "//pkg/private/deb:deb.bzl"),
("pkg_sub_rpm", "//pkg:rpm_pfg.bzl"),
("pkg_rpm", "//pkg:rpm_pfg.bzl"),
("pkg_tar", "//pkg/private/tar:tar.bzl"),
("pkg_tar_impl", "//pkg/private/tar:tar.bzl"),
Expand Down
79 changes: 79 additions & 0 deletions examples/rpm/subrpm/BUILD
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
# Copyright 2024 The Bazel Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# -*- coding: utf-8 -*-

load("@rules_pkg//pkg:mappings.bzl", "pkg_files")
load("@rules_pkg//pkg:rpm.bzl", "pkg_sub_rpm", "pkg_rpm")

pkg_files(
name = "subrpm_files",
srcs = [
"BUILD",
],
)

pkg_sub_rpm(
name = "subrpm",
package_name = "subrpm",
summary = "Test subrpm",
description = "Test subrpm description",
requires = [
"somerpm",
],
provides = [
"someprovision",
],
srcs = [
":subrpm_files",
],
)

pkg_files(
name = "rpm_files",
srcs = [
"MODULE.bazel",
"README.md",
],
)

pkg_rpm(
name = "test-rpm",
srcs = [
":rpm_files",
],
release = "0",
version = "1",
summary = "rules_pkg example RPM",
description = "This is a package description.",
license = "Apache License, v2.0",
architecture = "x86_64",
requires = [
"somerpm",
],
provides = [
"somefile",
],
subrpms = [
":subrpm",
],
)

# If you have rpmbuild, you probably have rpm2cpio too.
# Feature idea: Add rpm2cpio and cpio to the rpmbuild toolchain
genrule(
name = "inspect_content",
srcs = [":test-rpm"],
outs = ["content.txt"],
cmd = "rpm2cpio $(locations :test-rpm) | cpio -ivt >$@",
)
32 changes: 32 additions & 0 deletions examples/rpm/subrpm/MODULE.bazel
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
# Copyright 2024 The Bazel Authors. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

module(name = "rules_pkg_example_rpm_system_rpmbuild_bzlmod")

bazel_dep(name = "rules_pkg")

local_path_override(
module_name = "rules_pkg",
path = "../../..",
)

find_rpmbuild = use_extension(
"@rules_pkg//toolchains/rpm:rpmbuild_configure.bzl",
"find_system_rpmbuild_bzlmod",
)
use_repo(find_rpmbuild, "rules_pkg_rpmbuild")

register_toolchains(
"@rules_pkg_rpmbuild//:all",
)
14 changes: 14 additions & 0 deletions examples/rpm/subrpm/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,14 @@
# Using system rpmbuild with bzlmod

## Summary

This example builds an RPM using the system `rpmbuild` from a pure bazel
`pkg_rpm()` definition instead of using a separate specfile.

## To use

```
bazel build :*
rpm2cpio bazel-bin/test-rpm.rpm | cpio -ivt
cat bazel-bin/content.txt
```
64 changes: 50 additions & 14 deletions pkg/make_rpm.py
Original file line number Diff line number Diff line change
Expand Up @@ -81,10 +81,9 @@ def Cleanup():

def FindOutputFile(log):
"""Find the written file from the log information."""

m = WROTE_FILE_RE.search(log)
m = WROTE_FILE_RE.findall(log)
if m:
return m.group('rpm_path')
return m
return None

def SlurpFile(input_path):
Expand Down Expand Up @@ -187,7 +186,7 @@ def __init__(self, name, version, release, arch, rpmbuild_path,
self.arch = arch
self.files = []
self.rpmbuild_path = FindRpmbuild(rpmbuild_path)
self.rpm_path = None
self.rpm_paths = None
self.source_date_epoch = helpers.GetFlagValue(source_date_epoch)
self.debug = debug

Expand All @@ -204,6 +203,7 @@ def __init__(self, name, version, release, arch, rpmbuild_path,
self.post_scriptlet = None
self.preun_scriptlet = None
self.postun_scriptlet = None
self.subrpms = None

def AddFiles(self, paths, root=''):
"""Add a set of files to the current RPM.
Expand All @@ -227,6 +227,7 @@ def SetupWorkdir(self,
preamble_file=None,
description_file=None,
install_script_file=None,
subrpms_file=None,
pre_scriptlet_path=None,
post_scriptlet_path=None,
preun_scriptlet_path=None,
Expand Down Expand Up @@ -267,6 +268,8 @@ def SetupWorkdir(self,
SlurpFile(os.path.join(original_dir, postun_scriptlet_path)) if postun_scriptlet_path is not None else ''
self.posttrans_scriptlet = \
SlurpFile(os.path.join(original_dir, posttrans_scriptlet_path)) if posttrans_scriptlet_path is not None else ''
self.subrpms = \
SlurpFile(os.path.join(original_dir, subrpms_file)) if subrpms_file is not None else ''

# Then prepare for textual substitution. This is typically only the case for the
# experimental `pkg_rpm`.
Expand All @@ -276,6 +279,7 @@ def SetupWorkdir(self,
'PREUN_SCRIPTLET': ("%preun\n" + self.preun_scriptlet) if self.preun_scriptlet else "",
'POSTUN_SCRIPTLET': ("%postun\n" + self.postun_scriptlet) if self.postun_scriptlet else "",
'POSTTRANS_SCRIPTLET': ("%posttrans\n" + self.posttrans_scriptlet) if self.posttrans_scriptlet else "",
'SUBRPMS' : (self.subrpms if self.subrpms else ""),
'CHANGELOG': ""
}

Expand Down Expand Up @@ -362,6 +366,7 @@ def CallRpmBuild(self, dirname, rpmbuild_args):
'--define', '_topdir %s' % dirname,
'--define', '_tmppath %s/TMP' % dirname,
'--define', '_builddir %s/BUILD' % dirname,
'--define', 'buildsubdir .',
'--bb',
'--buildroot=%s' % buildroot,
] # yapf: disable
Expand Down Expand Up @@ -405,9 +410,9 @@ def CallRpmBuild(self, dirname, rpmbuild_args):

if p.returncode == 0:
# Find the created file.
self.rpm_path = FindOutputFile(output)
self.rpm_paths = FindOutputFile(output)

if p.returncode != 0 or not self.rpm_path:
if p.returncode != 0 or not self.rpm_paths:
print('Error calling rpmbuild:')
print(output)
elif self.debug:
Expand All @@ -416,20 +421,35 @@ def CallRpmBuild(self, dirname, rpmbuild_args):
# Return the status.
return p.returncode

def SaveResult(self, out_file):
def SaveResult(self, out_file, subrpm_out_files):
"""Save the result RPM out of the temporary working directory."""

if self.rpm_path:
shutil.copy(self.rpm_path, out_file)
if self.debug:
print('Saved RPM file to %s' % out_file)
if self.rpm_paths:
for p in self.rpm_paths:
is_subrpm = False

for subrpm_name, subrpm_out_file in subrpm_out_files:
subrpm_prefix = self.name + '-' + subrpm_name

if os.path.basename(p).startswith(subrpm_prefix):
shutil.copy(p, subrpm_out_file)
is_subrpm = True
if self.debug or True:
print('Saved %s sub RPM file to %s' % (
subrpm_name, subrpm_out_file))
break

if not is_subrpm:
shutil.copy(p, out_file)
if self.debug or True:
print('Saved RPM file to %s' % out_file)
else:
print('No RPM file created.')

def Build(self, spec_file, out_file,
def Build(self, spec_file, out_file, subrpm_out_files=None,
preamble_file=None,
description_file=None,
install_script_file=None,
subrpms_file=None,
pre_scriptlet_path=None,
post_scriptlet_path=None,
preun_scriptlet_path=None,
Expand All @@ -446,12 +466,21 @@ def Build(self, spec_file, out_file,
original_dir = os.getcwd()
spec_file = os.path.join(original_dir, spec_file)
out_file = os.path.join(original_dir, out_file)

if subrpm_out_files is not None:
subrpm_out_files = (s.split(':') for s in subrpm_out_files)
subrpm_out_files = [
(s[0], os.path.join(original_dir, s[1])) for s in subrpm_out_files]
else:
subrpm_out_files = []

with Tempdir() as dirname:
self.SetupWorkdir(spec_file,
original_dir,
preamble_file=preamble_file,
description_file=description_file,
install_script_file=install_script_file,
subrpms_file=subrpms_file,
file_list_path=file_list_path,
pre_scriptlet_path=pre_scriptlet_path,
post_scriptlet_path=post_scriptlet_path,
Expand All @@ -460,7 +489,7 @@ def Build(self, spec_file, out_file,
posttrans_scriptlet_path=posttrans_scriptlet_path,
changelog_file=changelog_file)
status = self.CallRpmBuild(dirname, rpmbuild_args or [])
self.SaveResult(out_file)
self.SaveResult(out_file, subrpm_out_files)

return status

Expand All @@ -483,6 +512,9 @@ def main(argv):
help='The file containing the RPM specification.')
parser.add_argument('--out_file', required=True,
help='The destination to save the resulting RPM file to.')
parser.add_argument('--subrpm_out_file', action='append',
help='List of destinations to save resulting ' +
'Sub RPMs to in the form of name:destination')
parser.add_argument('--rpmbuild', help='Path to rpmbuild executable.')
parser.add_argument('--source_date_epoch',
help='Value for the SOURCE_DATE_EPOCH rpmbuild '
Expand All @@ -499,6 +531,8 @@ def main(argv):
help='File containing the RPM Preamble')
parser.add_argument('--description',
help='File containing the RPM %description text')
parser.add_argument('--subrpms',
help='File containing the RPM subrpm details')
parser.add_argument('--pre_scriptlet',
help='File containing the RPM %pre scriptlet, if to be substituted')
parser.add_argument('--post_scriptlet',
Expand Down Expand Up @@ -526,9 +560,11 @@ def main(argv):
debug=options.debug)
builder.AddFiles(options.files)
return builder.Build(options.spec_file, options.out_file,
options.subrpm_out_file,
preamble_file=options.preamble,
description_file=options.description,
install_script_file=options.install_script,
subrpms_file=options.subrpms,
file_list_path=options.file_list,
pre_scriptlet_path=options.pre_scriptlet,
post_scriptlet_path=options.post_scriptlet,
Expand Down
10 changes: 8 additions & 2 deletions pkg/rpm.bzl
Original file line number Diff line number Diff line change
Expand Up @@ -29,10 +29,12 @@ The mechanism for choosing between the two is documented in the function itself.
"""

load("//pkg:rpm_pfg.bzl", pkg_rpm_pfg = "pkg_rpm")
load("//pkg:rpm_pfg.bzl", _pkg_sub_rpm = "pkg_sub_rpm", pkg_rpm_pfg = "pkg_rpm")
load("//pkg/legacy:rpm.bzl", pkg_rpm_legacy = "pkg_rpm")

def pkg_rpm(name, srcs = None, spec_file = None, **kwargs):
pkg_sub_rpm = _pkg_sub_rpm

def pkg_rpm(name, srcs = None, spec_file = None, subrpms = None, **kwargs):
"""pkg_rpm wrapper
This rule selects between the two implementations of pkg_rpm as described in
Expand All @@ -54,13 +56,17 @@ def pkg_rpm(name, srcs = None, spec_file = None, **kwargs):
if srcs and spec_file:
fail("Cannot determine which pkg_rpm rule to use. `srcs` and `spec_file` are mutually exclusive")

if subrpms and spec_file:
fail("Cannot build sub RPMs with a specfile. `subrpms` and `spec_file` are mutually exclusive")

if not srcs and not spec_file:
fail("Either `srcs` or `spec_file` must be provided.")

if srcs:
pkg_rpm_pfg(
name = name,
srcs = srcs,
subrpms = subrpms,
**kwargs
)
elif spec_file:
Expand Down
2 changes: 2 additions & 0 deletions pkg/rpm/template.spec.tpl
Original file line number Diff line number Diff line change
Expand Up @@ -21,4 +21,6 @@ ${POSTUN_SCRIPTLET}

${POSTTRANS_SCRIPTLET}

${SUBRPMS}

${CHANGELOG}
Loading

0 comments on commit 2aa2b8e

Please sign in to comment.