First cut at remote support (#8)

* checkpoint work on remote to a branch
* this is very incomplete but some of it works
* lots of work yet to remove wrinkles
* updating logger for better error output in debug mode

* fix work in progress. 'ls' now mostly works

* more wip fixes
things are messy right now

* checkpoint wip

* change "Rem" to "Location" in component struct
move ssh initialisation into connect
set 5 second timeout for ssh (maybe override later per remote)

* rename to have better meaning

* missed

* re-order

* updates to package downloading
* save downloads to packages/archives by default
* allow override of active_prod for update

* checkpoint WIP

* checkpoint WIP
* starting a remote command works, but is untidy

* refactor RemoteName() to Location()
split findInstanceProc into two
implement stop for remotes

* remove excessive debug
redo stop/start/restart output
tidy up

* add support for remote log cat and tail
not follow (yet) as watcher doesn't exist for sftp
some refactor required for above

* add remote support to tls command
* still need to distribute chain.pem

* update ls to include locations

* checkpoint WIP
support update on remotes
list certs regardless of root or signing cert existing
start updating README
add more info for failed startup

* tidy up a little

* add a 'sync' subcommand to copy chain.pem to remotes

* change defaults to ALL

* describe tls sync

* skip LOCAL for sync

Dockerfile

@@ -11,8 +11,8 @@ RUN go build
 
 FROM debian
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
-COPY --from=build /app/cmd/geneos/geneos /bin
 RUN apt update && apt install -y fontconfig
+COPY --from=build /app/cmd/geneos/geneos /bin
 RUN useradd -ms /bin/bash geneos
 USER geneos
 WORKDIR /home/geneos

cmd/geneos/README.md

@@ -3,9 +3,10 @@
 The `geneos` program will help you manage your Geneos environment, one server at a time. Some of it's features, existing and planned, include:
 
 * Set-up a new environment with a series of simple commands
+* Remote systems support (very much early testing)
 * Add new instances of common componments with sensible defaults
 * Check the status of components
-* Stop, start, restart components (without unnecessary delays)
+* Stop, start, restart components (without minimal delays)
 * Support fo existing gatewayctl/netprobectl/licdctl scripts and their file and configuration layout
 * Convert existing set-ups to JSON based configs with more options
 * Edit individual settings of instances
@@ -64,6 +65,49 @@ You still have to configure the Gateway to connect to the Netprobe, but all thre
 
 This program has been written in such a way that is *should* be safe to install SETUID root or run using `sudo` for almost all cases. The program will refuse to accidentally run an instance as root unless the `User` config parameter is explicitly set - for example when a Netprobe needs to run as root. As with many complex programs, care should be taken and privileged execution should be used when required.
 
+## Remote Management (NEW!)
+
+The `geneos` command can now transparently manage instances across multiple systems using SSH. Some things works well, some work with minor issues and some features do not work at all.
+
+This feature is still very much under development there will be changes coming.
+
+### What does this mean?
+
+See if these commands give you a hint:
+
+```bash
+$ geneos add remote server2 ssh://[email protected]/opt/geneos
+$ geneos add gateway newgateway@server2
+$ geneos start
+```
+
+Command like `ls` and `ps` will works transparently and merge all instances together, showing you where they are runnng (or not).
+
+A remote is a psuedo-instance and you add and manage it with the normal commands. At the moment the only supported transport is SSH and the URL is a slightly extended version of the RFC standard to include the Geneos home directory. The format, for the `add` command is:
+
+`ssh://[USER@]HOST[:PORT][/PATH]`
+
+If not set, USER defaults to the current username. Similarly PORT defaults to 22. PATH defaults to the local ITRSHome path. The most basic SSH URL of the form `ssh://hostname` results in a remote accessed as the current user on the default SSH port and rooted in the same directory as the local set-up. Is the remote directory is empty (dot files are ignored) then the standard file layout is created.
+
+### How does it work?
+
+There are a number of prerequisites for remote support:
+
+1. Linux on amd64 for all servers
+2. Passwordless SSH access, either via an `ssh-agent` or unprotected private keys
+3. At this time the only private keys supported are those in your `.ssh` directory beginning `id_` - later updates will allow you to set the name of the key to load, but using an agent is recommended.
+4. The remote user must be confiugured to use a `bash` shell or similar. See limitations below.
+
+If you can log in to a remote Linux server using `ssh user@server` and no be prompted for a password or passphrase then you are set to go. It's beyond the scope of this README to explain how to set-up `ssh-agent` or how to create an unprotected private key file, so please search online.
+
+### Limitations
+
+The remote connections over SSH mean there are limitations to the features available on remote servers:
+
+1. No following logs (i.e. the `-f` option). The program is written to use `fsnotify` and that only works on local filesystems and not over sftp. This may be added using a more primitive polling mecahnism later.
+2. Control over instance processes is done via shell commands and little error checking is done, so it is possible to cause damage and/or processes not to to start or stop as expected. Contributions of fixes are welcomed.
+3. All actions are taken as the user given in the SSH URL (which should NEVER be `root`!) and so instances that are meant to run as other users cannot be controlled. Files and directories may not be available if the user does not have suitable permissions.
+
 ## Usage
 
 Please note that the full list of commands and parameters is changing all the time. This list below is mostly, but not completely, up-to-date.
@@ -215,7 +259,7 @@ The `geneos tls` command provides a number of subcommands to create and manage c
 Once enabled then all new instances will also have certificates created and configuration set to use secure (encrypted) connections where possible.
 
 * `geneos tls init`
-  Initialised the TLS environment by creating a `tls` directory in ITRSHome and populkating it with a new root and intermediate (signing) certificate and keys as well as a `chain.pem` which includes both CA certificates. The keys are only readable by the user running the command.
+  Initialised the TLS environment by creating a `tls` directory in ITRSHome and populkating it with a new root and intermediate (signing) certificate and keys as well as a `chain.pem` which includes both CA certificates. The keys are only readable by the user running the command. Also does a `sync` if remotes are configured.
 
   Any existing instances have certificates created and their configurations updated to reference them. This means that any legacy `.rc` configurations will be migrated to `.json` files.
 
@@ -231,6 +275,9 @@ Once enabled then all new instances will also have certificates created and conf
 * `geneos tls ls [-c | -j | -i | -l] [TYPE] [NAME...]`
   List instance certificate information. Options are the same as for the main `ls` command but the data shown is specific to certificates.
 
+* `geneos tls sync`
+  Copies chain.pem to all remotes
+
 ## Configuration Files
 
 ### General Configuration

cmd/geneos/add.go

@@ -18,7 +18,7 @@ func init() {
 	commands["add"] = Command{
 		Function:    commandAdd,
 		ParseFlags:  defaultFlag,
-		ParseArgs:   parseArgs,
+		ParseArgs:   parseArgsNoWildcard,
 		CommandLine: "geneos add TYPE NAME",
 		Summary:     `Add a new instance`,
 		Description: `Add a new instance called NAME with the TYPE supplied. The details will depends on the
@@ -48,6 +48,12 @@ user in the instance configuration or the default user. Currently only one file
 time.`}
 }
 
+// Add a single instance
+//
+// XXX argument validation is minimal
+//
+// remote support would be of the form name@remotename
+//
 func commandAdd(ct ComponentType, args []string, params []string) (err error) {
 	if len(args) == 0 {
 		logError.Fatalln("not enough args")
@@ -72,7 +78,7 @@ func commandAdd(ct ComponentType, args []string, params []string) (err error) {
 	if err != nil {
 		return
 	}
-	log.Printf("new %s %q added, listening port %s\n", Type(c), Name(c), getIntAsString(c, Prefix(c)+"Port"))
+	log.Printf("new %s %q added, port %s\n", Type(c), Name(c), getIntAsString(c, Prefix(c)+"Port"))
 
 	return
 }
@@ -227,8 +233,8 @@ func uploadFile(c Instance, source string) (err error) {
 				logError.Fatalln("dest path must be relative to (and in) instance directory")
 			}
 			// if the destination exists is it a directory?
-			if st, err := os.Stat(filepath.Join(Home(c), destfile)); err == nil {
-				if st.IsDir() {
+			if s, err := statFile(Location(c), filepath.Join(Home(c), destfile)); err == nil {
+				if s.st.IsDir() {
 					destdir = filepath.Join(Home(c), destfile)
 					destfile = ""
 				}
@@ -274,7 +280,7 @@ func uploadFile(c Instance, source string) (err error) {
 
 	default:
 		// support globbing later
-		from, err = os.Open(source)
+		from, _, err = openStatFile(LOCAL, source)
 		if err != nil {
 			return err
 		}
@@ -286,49 +292,73 @@ func uploadFile(c Instance, source string) (err error) {
 
 	destfile = filepath.Join(destdir, destfile)
 
-	if _, err := os.Stat(filepath.Dir(destfile)); err != nil {
-		err = os.MkdirAll(filepath.Dir(destfile), 0775)
+	if _, err := statFile(Location(c), filepath.Dir(destfile)); err != nil {
+		err = mkdirAll(Location(c), filepath.Dir(destfile), 0775)
 		if err != nil && !errors.Is(err, fs.ErrExist) {
 			logError.Fatalln(err)
 		}
 		// if created, chown the last element
 		if err == nil {
-			if err = os.Chown(filepath.Dir(destfile), int(uid), int(gid)); err != nil {
+			if err = chown(Location(c), filepath.Dir(destfile), int(uid), int(gid)); err != nil {
 				return err
 			}
 		}
 	}
 
 	// xxx - wrong way around. create tmp first, move over later
-	if st, err := os.Stat(destfile); err == nil {
-		if !st.Mode().IsRegular() {
+	if s, err := statFile(Location(c), destfile); err == nil {
+		if !s.st.Mode().IsRegular() {
 			logError.Fatalln("dest exists and is not a plain file")
 		}
 		datetime := time.Now().UTC().Format("20060102150405")
 		backuppath = destfile + "." + datetime + ".old"
-		if err = os.Rename(destfile, backuppath); err != nil {
+		if err = renameFile(Location(c), destfile, backuppath); err != nil {
 			return err
 		}
 	}
-	out, err := os.Create(destfile)
-	if err != nil {
-		return err
-	}
-	defer out.Close()
 
-	if err = out.Chown(int(uid), int(gid)); err != nil {
-		os.Remove(out.Name())
-		if backuppath != "" {
-			if err = os.Rename(backuppath, destfile); err != nil {
+	var out io.Writer
+
+	switch Location(c) {
+	case LOCAL:
+		cf, err := os.Create(destfile)
+		if err != nil {
+			return err
+		}
+		out = cf
+		defer cf.Close()
+
+		if err = cf.Chown(int(uid), int(gid)); err != nil {
+			removeFile(Location(c), destfile)
+			if backuppath != "" {
+				if err = renameFile(Location(c), backuppath, destfile); err != nil {
+					return err
+				}
 				return err
 			}
+		}
+	default:
+		cf, err := createRemoteFile(Location(c), destfile)
+		if err != nil {
 			return err
 		}
+		out = cf
+		defer cf.Close()
+
+		if err = cf.Chown(int(uid), int(gid)); err != nil {
+			removeFile(Location(c), destfile)
+			if backuppath != "" {
+				if err = renameFile(Location(c), backuppath, destfile); err != nil {
+					return err
+				}
+				return err
+			}
+		}
 	}
 
 	if _, err = io.Copy(out, from); err != nil {
 		return err
 	}
-	log.Println("uploaded", source, "to", out.Name())
+	log.Println("uploaded", source, "to", destfile)
 	return nil
 }

cmd/geneos/command.go

@@ -4,7 +4,6 @@ import (
 	"bufio"
 	"bytes"
 	"fmt"
-	"os"
 	"os/exec"
 	"path/filepath"
 	"strings"
@@ -38,26 +37,30 @@ type Commands map[string]Command
 // return a single slice of all instances, ordered and grouped
 // configuration are not loaded, just the defaults ready for overlay
 func allInstances() (confs []Instance) {
-	for _, ct := range componentTypes() {
-		confs = append(confs, instances(ct)...)
+	for _, ct := range realComponentTypes() {
+		for _, remote := range allRemotes() {
+			confs = append(confs, instancesOfComponent(Name(remote), ct)...)
+		}
 	}
 	return
 }
 
-// return a slice of instance for a given ComponentType
-func instances(ct ComponentType) (confs []Instance) {
-	for _, name := range instanceDirs(ct) {
+// return a slice of instancesOfComponent for a given ComponentType
+func instancesOfComponent(remote string, ct ComponentType) (confs []Instance) {
+	for _, name := range instanceDirsForComponent(remote, ct) {
 		confs = append(confs, newComponent(ct, name)...)
 	}
 	return
 }
 
+// return a slice of component types that exist for this name
 func findInstances(name string) (cts []ComponentType) {
-	for _, t := range componentTypes() {
-		compdirs := instanceDirs(t)
-		for _, dir := range compdirs {
+	local, remote := splitInstanceName(name)
+	for _, t := range realComponentTypes() {
+		for _, dir := range instanceDirsForComponent(remote, t) {
 			// for case insensitive match change to EqualFold here
-			if filepath.Base(dir) == name {
+			ldir, _ := splitInstanceName(dir)
+			if filepath.Base(ldir) == local {
 				cts = append(cts, t)
 			}
 		}
@@ -72,7 +75,7 @@ func loadConfig(c Instance, update bool) (err error) {
 	baseconf := filepath.Join(Home(c), Type(c).String())
 	j := baseconf + ".json"
 
-	if err = readConfigFile(j, &c); err == nil {
+	if err = readConfigFile(Location(c), j, &c); err == nil {
 		// return if NO error, else drop through
 		return
 	}
@@ -84,7 +87,7 @@ func loadConfig(c Instance, update bool) (err error) {
 			logError.Println("failed to wrtite config file:", err)
 			return
 		}
-		if err = os.Rename(baseconf+".rc", baseconf+".rc.orig"); err != nil {
+		if err = renameFile(Location(c), baseconf+".rc", baseconf+".rc.orig"); err != nil {
 			logError.Println("failed to rename old config:", err)
 		}
 		logDebug.Println(Type(c), Name(c), "migrated to JSON config")
@@ -121,7 +124,7 @@ func buildCmd(c Instance) (cmd *exec.Cmd, env []string) {
 // save off extra env too
 // XXX - scan file line by line, protect memory
 func readRCConfig(c Instance) (err error) {
-	rcdata, err := os.ReadFile(filepath.Join(Home(c), Type(c).String()+".rc"))
+	rcdata, err := readFile(Location(c), filepath.Join(Home(c), Type(c).String()+".rc"))
 	if err != nil {
 		return
 	}