pkg_zip does not maintain directory structure

[hborawski]

pkg_zip does not maintain the directory structure of the files placed inside it, nor does it have an option to.

For the iOS package manager CocoaPods, when archiving code for distribution (rather than relying on git), the directory structure of the decompressed archive must match the globs in the package specification file (.podspec)

if i have a podspec with the following:

package.source_files = ["Sources/**/*.swift"]

and use the same in pkg_zip:

pkg_zip(
    name = "PodName",
    srcs = glob(["Sources/**/*.swift"])
  )

All of the .swift files are unpacked at the root level of the archive. This breaks the CocoaPod import, since the package expects files to be under Sources

While this specific usecase is fixable (with package_dir), CocoaPods supports "subspecs", that allows for optionally importing chunks of code:

package.subspec "Module1" do |sub|
  sub.source_files = ["Sources/package1/**/*.swift"]
end
package.subspec "Module2" do |sub|
  sub.source_files = ["Sources/package2/**/*.swift"]
end

When compressed, all of the code needs to be available in the archive, so adding package_dir as Sources would then leave everything in the Sources directory when decompressed, and the .podspec would not match any files.

In order for these pkg_ rules to work for CocoaPods, there needs to be a way to maintain the directory structure when archiving files.

[aiuto]

This should not be done within pkg_zip. You can do it with a glob in a pkg_files.
Have you tried that?
If the docs are not sufficient, maybe the examples here might give an idea.
https://github.com/bazelbuild/rules_pkg/tree/main/examples/rich_structure

[tony-scio]

I hit the same thing. pkg_files obviously don't produce a zip. So is the recommendation to do a pkg_files then a genrule w/ a manual zip on top of that in order to make a zip that preserves the directory structure?

[nacl]

The recommendation is to create a one or more pkg_files targets, and then pass them to pkg_zip in its srcs attribute. Note that the path manipulation functions in pkg_files supplant the ones provided by pkg_zip (and also pkg_tar).

For a more concrete example, take a peek at the one @aiuto mentioned in his comment, above.

[aiuto]

pkg_files should be the srcs to pkg_zip.

load("//:pkg.bzl", "pkg_zip")
load("//pkg:mappings.bzl", "pkg_files", "strip_prefix")

pkg_files(
    name = "srcs",
    srcs = glob(["Sources/**/*.swift"]),
    strip_prefix = strip_prefix.from_pkg(),
)

pkg_zip(
    name = "pod",
    srcs = ["srcs"],
)

$ bazel build :*
INFO: Analyzed 6 targets (1 packages loaded, 6 targets configured).
...
$ unzip -v ../bazel-bin/i434/pod.zip 
Archive:  ../bazel-bin/i434/pod.zip
 Length   Method    Size  Cmpr    Date    Time   CRC-32   Name
--------  ------  ------- ---- ---------- ----- --------  ----
      20  Defl:N       22 -10% 1980-01-01 00:00 ab34387b  Sources/a/foo.swift
      26  Defl:N       28  -8% 1980-01-01 00:00 0cd071f9  Sources/b/foo.swift
--------          -------  ---                            -------
      46               50  -9%                            2 files

The complaint some people might have that this takes two rules instead of one. That design choice is intentional. pkg_files and related rules like pkg_dirs are a suite of composable rules that allow you do define the structure you want in the final output, and then package that up with any of pkg_{tar, zip, deb, rpm}. This lets us put the mapping logic in one place, rather than reimplementing things like strip_prefix in every rule.

[hborawski]

Sorry to get back to this so late, went down some other related rabbit holes. Using pkg_files works, and I get why, still learning a lot about bazel, so some of the intentional verbosity in the BUILD files isn't what I expect.

Thank you for the help!

[tony-scio]

Thanks! pkg_files works like a charm. I had incorrectly assumed a filegroup would work in its place.

[aiuto]

Just for closure. I do actually agree that pkg_*(srcs=glob(["**/*.switft"])) would be least surprising if it maintained the directory structure. It's just that we have years of people using and expecting the older (and dumb) behavior of flattening the tree. As tempting as it is, I don't want to potentially break a lot of people.

[achew22]

@aiuto out of curiosity, WDYT about adding a flatten flag to pkg_* that defaults to false and doing a breaking change? It would let people keep the old (and I agree with you, bad) behavior until they get a chance without increasing the ecosystem wide tech debt over time?

[nacl]

Perhaps this discussion should be continued in #354, where we are considering this very problem?

I would personally argue that it isn't necessarily bad behavior -- if your default mode of operation has files going to odd locations that have nothing to do with your source paths, you're going to be stripping everything pretty much by default anyway.

Regardless, that behavior may be too surprising to be effective.

[aiuto]

Sounds good. One place for discussion is best.

…

On Thu, Dec 2, 2021 at 7:58 AM Andrew Psaltis ***@***.***> wrote: Perhaps this discussion should be continued in #354 <#354>, where we are considering this very problem? I would personally argue that it isn't necessarily bad behavior -- if your default mode of operation has files going to odd locations that have nothing to do with your source paths, you're going to be stripping everything pretty much by default anyway. Regardless, that behavior may be too surprising to be effective. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#434 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHBTSQXC2HX43AY42TLUO5UO7ANCNFSM5F3DXCBQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.