docs: how to manage remote systems

[IronCore864]

A new how-to guide on managing remote systems (use Pebble in containers).

As previously discussed, the current "Pebble in containers" reference doc is removed.

Note that this PR contains content from #545.

Preview: https://canonical-pebble--547.com.readthedocs.build/en/547/how-to/manage-remote-systems/

[dwilding]

If I'm being ultra-picky, I would leave out the whole paragraph starting "While it's possible to manually execute commands on remote servers", as I don't think it's essential to the purpose of the doc. However, I do appreciate that it's interesting background context.

How about combining all of this and compressing it slightly? Like this:

Suggested change

	While it's possible to manually execute commands on remote servers and transfer files from/to remote servers, it is not the most efficient (or secure) approach. Several other methods exist for remote command execution and system administration. Secure Shell (SSH) is one of them, and there are also dedicated configuration management tools like Ansible. However, to use these tools, an extra port for SSH needs to be opened, which not only increases the operational overhead of managing public/private keys but also the potential attack surface. Remember March 29, 2024 which seemed to be another normal Friday? A developer shocked the world by revealing an XZ Utils (data-compression utilities) backdoor which could potentially enable unauthorized access via SSH and remote code execution (read the full story [here](https://www.ssh.com/blog/a-recap-of-the-openssh-and-xz-liblzma-incident)).
	---
	## Pebble in remote systems
	Pebble provides various commands and APIs to run commands and manage files and directories in remote systems. With these capabilities, if we run Pebble in a remote system like in a separate container, we can use Pebble APIs to coordinate with the remote system without opening a new port.
	## Why use Pebble?
	Pebble provides commands and an HTTP API to run commands and manage files and directories in remote systems. If we run Pebble in a remote system, for example in a separate container, we can use Pebble to coordinate with the remote system without opening a new port.
	While it's possible to manually run commands on remote servers and transfer files from/to remote servers, this isn't the most efficient or secure approach. Even a configuration management tool such as Ansible requires an extra port to be opened for Secure Shell (SSH), which not only increases the operational overhead of managing public/private keys but also the potential attack surface.
	Remember March 29, 2024 which seemed to be another normal Friday? A developer shocked the world by revealing an XZ Utils (data-compression utilities) backdoor which could potentially enable unauthorized access via SSH and remote code execution (read the full story [here](https://www.ssh.com/blog/a-recap-of-the-openssh-and-xz-liblzma-incident)).

For the part about the XZ Utils incident, is there an alternative write-up that we could link to? Or perhaps an article that talks more generally (and neutrally) about why opening ports is an increased attack surface? The link to ssh.com looks like a marketing piece for that company.

[IronCore864]

For the XZ utils incident, how about using a Wikipedia link? https://en.wikipedia.org/wiki/XZ_Utils_backdoor.

[dwilding]

Thanks @IronCore864, I like the new how-to doc! I've added some suggestions about specific parts.

There's one general question I have - this is more a question about my understanding of Pebble, rather than the doc itself. When we say "remote system/server" we're still talking about the same physical machine, right? So Pebble could be used to manage a container from the host system, but Pebble couldn't be used if data needs to go over the network. (My understanding is, the only feature of Pebble that works over the network is the open-access API endpoints to allow Juju to check health). Do I have the right understanding?

Perhaps we should state somewhere in the doc what "remote" means in the context of Pebble. What do you think?

[IronCore864]

Thanks @IronCore864, I like the new how-to doc! I've added some suggestions about specific parts.

There's one general question I have - this is more a question about my understanding of Pebble, rather than the doc itself. When we say "remote system/server" we're still talking about the same physical machine, right? So Pebble could be used to manage a container from the host system, but Pebble couldn't be used if data needs to go over the network. (My understanding is, the only feature of Pebble that works over the network is the open-access API endpoints to allow Juju to check health). Do I have the right understanding?

Perhaps we should state somewhere in the doc what "remote" means in the context of Pebble. What do you think?

"Remote system" - Your understanding is correct because only very few APIs work over HTTP at the moment, most APIs only work with the UNIX socket, which is for inter-process communication within the same host where data sent through the socket is handled within the kernel.

Even in the context of charms, the charm code is in the same pod as Pebble, so it's not exactly "remote" in the sense that the client and Pebble are in two different machines.

However, if we look at this from the perspective of a Pod, the client (charms) and Pebble (built in the workload image) are in separate containers (of the same Pod), so it's kind of "remote" as long as the client can access the UNIX socket.

The current situation might change soon, especially since we are adding another type of identity for the metrics endpoint, so maybe there will be more identity types, allowing exposing all APIs over HTTP and making it truly remote.

I'm not sure how to address this in the doc, because if we add a note like "remote means the same host, or the same Pod", although accurate, it doesn't look nice...

[dwilding]

I realized that I didn't reply to your latest comment. Thanks for adding your detailed thoughts about what "remote system" means

Looking again at the doc, I think that mentioning "remote servers" is perhaps risking confusion. This seems to be the only place in the docs that we talk about servers in this way, and it gets me thinking about managing a web server using Pebble as a direct replacement for some other tool (which it sounds like Pebble isn't)

So I think might be best to limit the discussion to "remote systems" only