Nesting of pkg_filegroup is not possible

[criemen]

I am using pkg_filegroup on sources that are a pkg_filegroup again.
This fails with

in srcs attribute of pkg_filegroup rule //:codeql-cpp: '//odasa-buildutils:dist-generic' does not have mandatory providers: 'PackageFilesInfo' or 'PackageDirsInfo' or 'PackageSymlinkInfo'.

Is there any specific reason that this is not allowed?
Using pkg_files everywhere (except at the top-level) is legal starlark, but needs very careful strip_prefix handling to not have too many/the wrong prefixes (or I don't understand strip_prefix well enough).

[nacl]

Thanks for asking.

Is there any specific reason that this is not allowed?

I thought about doing this in the original pkg_filegroup design, but in discussions with @aiuto, we concluded that doing so would add too much complexity for a feature without immediate use. That aside, it should be doable.

Using pkg_files everywhere (except at the top-level) is legal starlark, but needs very careful strip_prefix handling to not have too many/the wrong prefixes (or I don't understand strip_prefix well enough).

Yeah, you do need to be careful with strip_prefix, and the behavior can be confusing or surprising at times. We're looking into making it more intuitive in #354.

I'm not opposed to implementing this in the long-term, and would be willing to look at patches to add such support.

[aiuto]

To add to Andrew's point about "too much complexity ... without immediate use". Please feel free to show a good example of what you want to accomplish with a syntax that is intuitive and not to bulky. We are driving a lot of design from actual use cases and a "why this feature is important" is super helpful.

…

On Thu, Aug 19, 2021 at 1:56 PM Andrew Psaltis ***@***.***> wrote: Thanks for asking. Is there any specific reason that this is not allowed? I thought about doing this in the original pkg_filegroup design, but in discussions with @aiuto <https://github.com/aiuto>, we concluded that doing so would add too much complexity for a feature without immediate use. That aside, it should be doable. Using pkg_files everywhere (except at the top-level) is legal starlark, but needs very careful strip_prefix handling to not have too many/the wrong prefixes (or I don't understand strip_prefix well enough). Yeah, you do need to be careful with strip_prefix, and the behavior can be confusing or surprising at times. We're looking into making it more intuitive in #354 <#354>. I'm not opposed to implementing this in the long-term, and would be willing to look at patches to add such support. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#415 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAXHHHDYLWHWGIYSBBFZQLLT5VAVFANCNFSM5COOFVKQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&utm_campaign=notification-email> .

[criemen]

Okay, let me explain better what I want to achieve, with hopefully not too many internal terms.

I am building a prototype how to package what we call a CodeQL distribution (a zip file containing our software).
This is a zip archive that contains:

• A base distribution (arch-specific, think compiled C binaries)
• A base distribution (generic, think config files+java binaries)
• 1..n language packs

Each language pack again can contain an arch-specific part and a generic part.

It makes sense for us to package up these language packs into zip files independent from the distribution, but also re-use the same pkg_filegroup to pack the distribution.
Thus, I want to map:

1. Each part of the CodeQL distribution to a pkg_filegroup. These get joined together by a pkg_filegroup of filegroups into the main distribution (no prefix-stripping or anything going on, just taking the union of the filegroups).
2. each languagepack can consist of a filegroup of arch-specifc and generic bits, as these usually need different prefix-munging applied to land in the correct paths

These filegroups can then serve as input to a bunch of pkg_zip targets, for different combinations.
Think: A full distribution/just a single language pack/a base distribution+a single language pack.

In the future, I'd also like to generate a pkg_tree instance for each of the zip targets, as we then can skip the zip/unzip part for the local development cycle.
For local development, we need the packaging step to move all the files to the correct paths, as our source paths and the paths in the released artifact are different, but we do not need to spend CPU cycles on actually packing/compressing anything.

Syntax-wise, I think you're there already, if I could use a pkg_filegroup label as srcs for another pkg_filegroup, and no extra path munging is applied. I.e. the top-level pkg_filegroup would just be a union over the pkg_filegroups specified as srcs.

[nacl]

Thanks for the description. The idea makes sense, it's just a matter of implementing it.

We actually found a internal use case for this functionality, and a colleague of mine has an implementation available. A PR for it should be available soon.