-
Notifications
You must be signed in to change notification settings - Fork 67
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Mount special filesystem in chroot runners #116
base: master
Are you sure you want to change the base?
Mount special filesystem in chroot runners #116
Conversation
8a86c36
to
ea9d269
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👎
To my reading the mount calls require 'CAP_SYS_ADMIN', there is no 'CAP_MOUNT', so this may have consequences for deployments. But then again, to use chroot
where the tools crash is also ill-advised.
pkg/runner/local_runner.go
Outdated
if requiresSpecialFilesystems { | ||
var dir filesystem.Directory | ||
dir = r.buildDirectory | ||
// TODO(nils): How to best open the input root `Directory` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be a misunderstanding on my part.
I want (to hold on to) a Directory
object for the input root.
It can be reopened, but do not want to call this opening code twice, I don't think.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you likely also don't want it, because the directory can be moved around to other places, right?
pkg/runner/local_runner.go
Outdated
var dir filesystem.Directory | ||
dir = r.buildDirectory | ||
// TODO(nils): How to best open the input root `Directory` | ||
// It must be a `localDirectory`, but outer directories can be `lazyDirectory`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This strings-split is a cop out for my lack of understanding of the builder and scope walker.
I want to replace this block.
👎
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries. I can also help out with this part, after we get the bb-storage part sorted out.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this, Nils!
pkg/runner/local_runner.go
Outdated
defer dircloser.Close() | ||
} | ||
|
||
dir = dircloser.(filesystem.Directory) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Doesn't DirectoryCloser implement Directory?
pkg/runner/local_runner.go
Outdated
if requiresSpecialFilesystems { | ||
var dir filesystem.Directory | ||
dir = r.buildDirectory | ||
// TODO(nils): How to best open the input root `Directory` |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah, you likely also don't want it, because the directory can be moved around to other places, right?
pkg/runner/local_runner.go
Outdated
var dir filesystem.Directory | ||
dir = r.buildDirectory | ||
// TODO(nils): How to best open the input root `Directory` | ||
// It must be a `localDirectory`, but outer directories can be `lazyDirectory`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
No worries. I can also help out with this part, after we get the bb-storage part sorted out.
ea9d269
to
0172979
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
General remark: Could this be implemented as a decorator for Runner?
type mountingRunner struct {
base Runner
}
0172979
to
d813b2f
Compare
// some tools require access to special filesystems | ||
// that are created when the operating system boots. | ||
// An input root with a full userland implementation may need these. | ||
// Typical choices are: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is this a good place for this? The chroot connection is smeared all over this work,
but the work can also stand on its own feet, maybe a central "how-to-chroot" document is better?
// {"proc", "/proc", "proc"}
// {"sys", "/sys", "sysfs"}
// that mounts `mount` before running a build action. | ||
// | ||
// This decorator can be used for chroot runners | ||
// that must mount special filesystems into the input root. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We could also compare it to the clean runner that has generalized pre- and post-command execution.
13e7f92
to
5dd227a
Compare
pkg/runner/mounting_runner.go
Outdated
defer root.Close() | ||
|
||
mountpoint := path.MustNewComponent(r.mount.Mountpoint) | ||
err = root.Mkdir(mountpoint, 0o555) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Should we be creating the directories ourselves? Maybe it makes more sense to only establish these mounts if they actually exist in the input root?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point, with this as an advanced feature, we can probably require very strict compliance from clients that wants to use it. I can require that in the proto specification.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just a heads-up: When implementing a reproduction case with the directories I realized that the inputRootCharacterDevices
inside /dev have the runner create /dev:
func (be *localBuildExecutor) createCharacterDevices(inputRootDirectory BuildDirectory) error { |
5dd227a
to
0945ccb
Compare
b1c0f7e
to
20533db
Compare
20533db
to
97cecc7
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for working on this!
pkg/runner/mounting_runner.go
Outdated
scopeWalker := path.NewRelativeScopeWalker(&pathResolver) | ||
defer pathResolver.closeAll() | ||
|
||
fullpath := filepath.Join(request.InputRootDirectory, r.mount.Mountpoint) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My suggestion would be to not use filepath.Join()
here. It assumes that the native path separator is used, even though request.InputRootDirectory
always uses forward slashes, even on Windows. There is also no need to do that if you just call path.Resolve()
multiple times:
if err := path.Resolve(request.InputRootDirectory, path.NewRelativeScopeWalker(&pathResolver)); err != nil {
...
}
if err := path.Resolve(r.mount.Mountpoint, path.NewRelativeScopeWalker(&pathResolver)); err != nil {
...
}
I would also strongly advise against using the same PathResolver for both processing the input root directory and the mount point. Notice how the path resolver contains a stack. That stack is there to process ".."
entries. Because the stack isn't cleared/truncated, a mount point containing ".."
components might leave the input root directory.
This is not a big issue right now, because the mount point is fully specified in config. But if someone ever comes along and wants the mount point path resolution to respect symlinks, it can become dangerous fairly quickly.
pkg/runner/mounting_runner.go
Outdated
stack: util.NewNonEmptyStack(filesystem.NopDirectoryCloser(r.buildDirectory)), | ||
} | ||
scopeWalker := path.NewRelativeScopeWalker(&pathResolver) | ||
defer pathResolver.closeAll() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This might keep more file descriptors open than strictly necessary. Would there be a way for us to close intermediate directories while Run()
is called?
b60b67b
to
9b52000
Compare
I think this is a flaky storage error in the check, but other storage error that occurred around the same time had more explicit error messages.
I cannot retrigger it. Update 10:45 I pushed a rebased PS, this requires Benjamin's updated LLVM toolchain PR |
9b52000
to
07ffd9d
Compare
This can be used to mount special filesystems like '/proc' and '/sys' in the input root of actions if 'chroot' is enabled. The filesystems are required for many tools to work. Solves: buildbarn#115
07ffd9d
to
2dfbdb8
Compare
This mounts '/proc' and '/sys' in the input root of runners if 'chroot' is enabled. It is done with "at"-syscall semantics in the 'Directory', to avoid side effects. A program-scope mutex is required for unmounting because there is no easy-to-use "unmountat".
#115