Bazel: add codeql specific packaging library #16432
Conversation
This encapsulate arch specific logic, local installation and separation of zip files into generic and arch-specific parts as required by the internal build.
misc/bazel/pkg.bzl
Outdated
| the requested `kind`. | ||
| """, | ||
| attrs = { | ||
| "base": attr.label( |
There was a problem hiding this comment.
Is there a need to expose this separately from zips?
Couldn't we just merge the zip files in zips? I don't see the benefit of a base argument, but happy to be convinced.
There was a problem hiding this comment.
ah, after trying it again I remembered why I had done this. The distinction whether to select a zip or not depends on its prefix containing {CODEQL_PLATFORM}. The base zip would be added with an empty "" prefix, which would disqualify it from the arch zipmerge. So the base_zip is always included, while zips are included conditionally.
There was a problem hiding this comment.
Ah, then a comment to that effect would be great, to explain that design choice to future us.
| @@ -20,6 +20,7 @@ cc_binary( | |||
| cc_test( | |||
| name = "test", | |||
| size = "small", | |||
| linkstatic = True, # required to build the test in the internal repo | |||
There was a problem hiding this comment.
this is only required on macOS, so I suspect something's wrong with our internal toolchain there. Will need to investigate
| ] + select({ | ||
| "@platforms//os:windows": [], | ||
| "//conditions:default": [ | ||
| ":dbscheme_files", | ||
| "//swift/downgrades", |
There was a problem hiding this comment.
I removed this distinction because the genric pack shouldn't really select on the platform, but this is what is causing the failure on the internal repo...
There was a problem hiding this comment.
turns out, windows requires --nobuild_python_zip for the //swift/extractor/trap:cppgen rule to work 🤷
misc/bazel/pkg.bzl
Outdated
| for zip_info in ctx.attr.srcs: | ||
| zip_info = zip_info[_ZipInfo] | ||
| manifest += ["%s:%s" % (p, z.short_path) for z, p in zip_info.zips_to_prefixes.items()] | ||
| files += list(zip_info.zips_to_prefixes) |
There was a problem hiding this comment.
I had to look up the definition of list on a dict, it acts as keys function
There was a problem hiding this comment.
the python-y way to write this would be l.extend(d), but I think that doesn't work in bazel (extend expects a list, not a generic iterable like in python). l += d.keys() works in bazel, but wouldn't work in python (+= expects a list, but in python dict.keys() is an iterable, not a list), so that could look surprising to a python dev. list(iterable) to be sure to get a list is pretty idiomatic thing to do in python, and works in bazel as well so I went with it.
There was a problem hiding this comment.
I should've added: (no action required)
There was a problem hiding this comment.
I stand corrected, extend works with a dict, I'll use that then 🙂
misc/bazel/pkg.bzl
Outdated
| ] | ||
| zips.append(zip) | ||
| else: | ||
| zips = zip_target.files.to_list() |
There was a problem hiding this comment.
I don't see how this works if the srcs are mixed, some with a ZipInfo provider, some not, as this overwrites zips hard, and the other code path for ZipInfo providers doesn't set transitive_zips at all
There was a problem hiding this comment.
ouch, the zips rewriting is indeed a bug, thanks for catching this!
There was a problem hiding this comment.
I also don't fully understand how this was working at all though 😕
This encapsulate arch specific logic, local installation and separation of zip files into generic and arch-specific parts as required by the internal build. The logic behind separating generic and arch-specific files lies in using
{CODEQL_PLATFORM}in the prefix, which then gets expanded to the appropriate value.Moreover, this PR the ability to import
.zipfiles into the pack in a highly optimized way:ripunzipwill be used to unzip the zip file contents to the destination prefixzipmerge(ported from the internal repository) merges the imported zips into the output, without decompressing.These changes are applied to
swiftas an example.