Swiftly proxies (swiftlang#155)

Add a design proposal for the new swiftly proxy system

Provide a mechanism to find the currently in-use toolchain
physical location

Clarify the boundaries of the swiftly toolchain abstraction
and elaborate on how to work around them

Change the nature of the swiftly symlinks so that they point
to the swiftly executable at install time. These do not change
when new toolchains are used. Toolchain selection happens each
time when the proxies are run. The proxies are created for a
well-known set of toolchain binaries that are constant for
a wide variety of toolchain versions and platforms.

Add support for .swift-version files for toolchain selection.
Update the use command so that it can point out which toolchain
is in use based on context, such as swift version files that are
located in the current working directory or above. The fallback
selection comes from the global default configuration's 'inUse'
setting. When querying for what's in use the global default
is shown with the "(default)" tag. If the in-use toolchain is
selected by a swift-version file the path to that file is displayed.

Provide a print location flag to the use subcommand that can print
the file path of the toolchain that is in use in the current
context.

When using a new toolchain, depending on whether a swift version
is selecting the current one, update the swift version file with
the selected toolchain version. If no swift version file can be
located, attempt to create a new one at the top of the git worktree.
If there is no git worktree, then fallback to updating the global
default in the configuration.

Provide a global default flag for the use subcommand so that only
the global default in-use toolchain is considered and not any of
the swift version files.

Provide a run command that allows arbitrary commands to be run
in the context of the selected toolchain, and also a one-off selection
mechanism with the special syntax.

Update the list command to decorate default, and in-use toolchains

Make the version argument optional in the install subcommand, which
causes it to use the toolchain selection through the .swift-version files
to decide what toolchain to install.

Guard automatic creation of .swift-version file from `swiftly use` around a
prompt overridable using an `--assume-yes`.

Fix all of the swift.org urls so that they use www.swift.org to avoid redirection

Fix symlink target selection for swiftly when it is system managed

Create proxies on toolchain installation, creating only the necessary ones,
giving a message about the shell path refresh.

DESIGN.md

@@ -4,21 +4,28 @@ This document contains the high level design of swiftly. Not all features have b
 
 ## Index
 
+- [Swiftly's purpose](#swiftlys-purpose)
+- [Installation of swiftly](#installation-of-switly)
 - [Linux](#linux)
-  - [Installation of swiftly](#installation-of-swiftly)
   - [Installation of a Swift toolchain](#installation-of-a-swift-toolchain)
 - [macOS](#macos)
-  - [Installation of swiftly](#installation-of-swiftly-1)
   - [Installation of a Swift toolchain](#installation-of-a-swift-toolchain-1)
 - [Interface](#interface)
   - [Toolchain names and versions](#toolchain-names-and-versions)
   - [Commands](#commands)
+  - [Toolchain selection](#toolchain-selection)
 - [Detailed design](#detailed-design)
   - [Implementation sketch - Core](#implementation-sketch---core)
   - [Implementation sketch - Ubuntu 20.04](#implementation-sketch---ubuntu-2004)
   - [Implementation sketch - macOS](#implementation-sketch---macos)
   - [`config.json` schema](#configjson-schema)
 
+## Swiftly's purpose
+
+Swiftly helps you to easily install different Swift toolchains locally on your account. It also provides a single path where you can run the tools in the currently selected toolchain. Toolchain selection is [configurable](#toolchain-selection) using different mechanisms.
+
+Note that swiftly is *not* a virtual toolchain in itself since there are cases where it cannot behave as a self-contained Swift toolchain. For example, there can be external dependencies on specific files, such as headers or libraries. There are far too many files that change between toolchain versions to be managed by swiftly. Also, for long-lived processes, there is no way to gracefully restart them without help from the client.
+
 ## Installation of swiftly
 
 The installation of swiftly is divided into two phases: delivery and initialization. Delivery of the swiftly binary can be accomplished using different methods:
@@ -60,7 +67,7 @@ A simple setup for managing the toolchains could look like this:
 
 The toolchains (i.e. the contents of a given Swift download tarball) would be contained in the toolchains directory, each named according to the major/minor/patch version. `config.json` would contain any required metadata (e.g. the latest Swift version, which toolchain is selected, etc.). If pulling in Foundation to use `JSONEncoder`/`JSONDecoder` (or some other JSON tool) would be a problem, we could also use something simpler.
 
-The `~/.local/bin` directory would include symlinks pointing to the `bin` directory of the "active" toolchain, if any.
+The `~/.local/share/swiftly/bin` directory would include symlinks pointing to swiftly itself. When the proxies binaries are executed swiftly proxies them to the requested toolchain, or the default.
 
 This is all very similar to how rustup does things, but I figure there's no need to reinvent the wheel here.
 
@@ -78,7 +85,7 @@ The contents of `~/Library/Application Support/swiftly` would look like this:
    – env
 ```
 
-Instead of downloading tarballs containing the toolchains and storing them directly in `~/.local/share/swiftly/toolchains`, we instead install Swift toolchains to `~/Library/Developer/Toolchains` via the `.pkg` files provided for download at swift.org. To select a toolchain for use, we update the symlinks at `~/Library/Application Support/swiftly/bin` to point to the desired toolchain in `~/Library/Developer/Toolchains`. In the env file, we’ll contain a line that looks like `export PATH="$HOME/Library/Application Support/swiftly:$PATH"`, so the version of swift being used will automatically always be from the active toolchain. `config.json` will contain version information about the selected toolchain as well as its actual location on disk.
+Instead of downloading tarballs containing the toolchains and storing them directly in `~/.local/share/swiftly/toolchains`, we instead install Swift toolchains to `~/Library/Developer/Toolchains` via the `.pkg` files provided for download at swift.org. In the env file, we’ll add a line that looks like `export PATH="$HOME/Library/Application Support/swiftly:$PATH"`, so that swiftly can proxy toolchain commands to the requested toolchain, or default. `config.json` will contain version information about the selected toolchain as well as its actual location on disk.
 
 This scheme works for ensuring the version of Swift used on the command line can be controlled, but it doesn’t affect the active toolchain used by Xcode, which uses its own mechanisms for that. Xcode, if it is installed, can find the toolchains installed by swiftly.
 
@@ -108,7 +115,7 @@ This will install the latest available stable release of Swift. If the latest ve
 
 ##### Installing a specific release version of Swift
 
-To install a specific version of Swift, the user can provide it. 
+To install a specific version of Swift, the user can provide it.
 
 If a patch version isn't specified, it’ll install the latest patch version that matches the minor version provided. If a version is already installed that has the same major and minor version, a message will be printed indicating so and directing the user to `swiftly update a.b` if they wish to check for updates. 
 
@@ -138,6 +145,14 @@ Installing a specific snapshot from a swift version development branch
 
 `swiftly install 5.5-snapshot-2022-1-28`
 
+##### Installing the version from the `.swift-version` file
+
+A package could have a ".swift-version" file that specifies the recommended toolchain version. A swiftly install with no version will search for a version file and install that version.
+
+`swiftly install`
+
+If no ".swift-version" file can be found then the installation fails indicating that it couldn't fine the file.
+
 #### uninstall
 
 Uninstalling versions of Swift should be in a similar form to install. Uninstalling a toolchain that is currently “in use” (see the “use” command section below) will cause swiftly to use the latest Swift release toolchain that is installed. If none are, the latest snapshot will be used. If no snapshots are installed either, then a message will be printed indicating that all Swift versions are uninstalled.
@@ -178,7 +193,7 @@ To list all the versions of swift installed on your system
 
 #### use
 
-“Using” a toolchain sets it as the active toolchain, meaning it will be the one found via $PATH and invoked via `swift` commands executed in the shell. Only a single toolchain can be used at a given time. Using a toolchain doesn’t uninstall anything; it only updates symlinks so that the requested toolchain can be found by the shell. 
+“Using” a toolchain sets it as the default toolchain, meaning it will be the default one that is used when running toolchain commands from the shell. Only a single toolchain can be the default at a given time and location. Using a toolchain doesn’t uninstall anything; it only updates the configuration.
 
 To use the toolchain associated with the most up-to-date Swift version, the “latest” version can be specified:
 
@@ -208,6 +223,10 @@ To use the latest installed main snapshot, leave off the date:
 
 `swiftly use main-snapshot`
 
+The use subcommand also supports `.swift-version` files. If a ".swift-version" file is present in the current working directory, or an ancestory directory, then swiftly will update that file with the new version to use. This can be a useful feature for a team to share and align on toolchain versions with git. As a special case, if swiftly could not find a version file, but it could find a Package.swift file it will create a new version file for you in the package and set that to the requested toolchain version.
+
+Note: The `.swift-version` file mechanisms can be overridden using the `--global-default` flag so that your swiftly installation's default toolchain can be set explicitly.
+
 #### update
 
 Update replaces a given toolchain with a later version of that toolchain. For a stable release, this means updating to a later patch version. For snapshots, this means updating to the most recently available snapshot. 
@@ -266,6 +285,60 @@ This command checks to see if there are new versions of `swiftly` itself and upg
 
 `swiftly self-update`
 
+### Toolchain selection
+
+Swiftly will create a set of symbolic links in its SWIFTLY_BIN_DIR during installation that point to the swiftly binary itself for each of the common toolchain commands, such as swift, swiftc, clang, etc. This mechanism will allows swiftly to proxy those command invocations to a selected toolchain at the time of invocation. A toolchain can be selected in these ways in order of precedence:
+
+* The presence of a .swift-version file in the current working directory, or ancestor directory, with the required toolchain version
+* The swiftly default (in-use) toolchain set in the swftly config.json by `swiftly install` or `swiftly use` commands
+
+If swiftly cannot find an installed toolchain that matches the selection then it fails with an error and instructions how to use `swiftly install` to satisfy the selection next time.
+
+#### Resolve selected toolchain
+
+For cases where the physical toolchain must be located, such as references specific header files, or shared libraries that are not proxied by swiftly there is a method to resolve the currently selected toolchain to its physical location using `swiftly use`.
+
+```
+swiftly use --print-location
+```
+
+This command will provide the full path to the directory where the selected toolchain is installed to standard output if such a toolchain exists. An external tool can directly navigate to the resources that it requires. For external tools that manage long-lived processes from the toolchain, such as the language server, and lldb, this command can be used in a poll to detect cases where the processes should be restarted.
+
+#### Run with a selected toolchain
+
+There are cases where you might want to run an arbitrary command using a selected toolchain. An example could be that you want to build something with CMake or Autoconf.
+
+```
+# CMake
+swiftly run cmake -G ninja -D CMAKE_C_COMPILER=clang -D CMAKE_CXX_COMPILER=clang++
+swiftly run ninja build
+
+# Autoconf
+CC=clang swiftly run ./configure
+CC=clang swiftly run make
+```
+
+Swiftly prefixes the PATH to the selected toolchain directory and runs the command so that the toolchain executables are available and have precedence.
+
+If you want to explicitly specify a toolchain for the command you can do that with a selector notation like this:
+
+```
+swiftly run swift build +5.10.1 # Runs swift build with the 5.10.1 toolchain
+```
+
+A few notes about the '+' prefix. First, if a literal '+' prefix should be sent directly to the tool as an argument then it is escaped by doubling it with '++'. An argument with only '++' is ignored entirely, and any additional arguments are sent directly to the command without any further inspection of their prefixes. This is analogous to the special '--' token that certain argument parsers accept so that they don't interpret anything following that token as command flags or options.
+
+If the selected toolchain is not installed then swiftly will exit with a message indicating that you need to run `swiftly install x.y.z` to install it.
+
+```
+# Use the latest main snapshot toolchain and run 'swift build' to build the package with it.
+swiftly run swift build +main-snapshot
+
+# Generate makefiles with the latest released Swift toolchain
+swiftly run +latest cmake -G "Unix Makefile" -D CMAKE_C_COMPILER=clang
+CC=clang swiftly run +latest make
+```
+
 ## Detailed Design
 
 Swiftly itself will be a SPM project consisting of several executable products, one per supported platform, and all of these will share the core module that handles argument parsing, printing help information, and dispatching commands. Each platform’s executable will be built to statically link the stdlib so that they can be run without having installed Swift first.
@@ -427,7 +500,7 @@ If the tag is a newer version than the installed one, a prompt indicating the ne
 $ dpkg --status libcurl4
 ```
 
-If the exit code of the previous command was 0, then we know the dependency exists and can return true. If it wasn't, then we call fall back to attempting to locate the library via `pkg-config`:
+If the exit code of the previous command was 0, then we know the dependency exists and can return true. If it wasn't, then we can fall back to attempting to locate the library via `pkg-config`:
 
 ```
 $ pkg-config --exists libcurl
@@ -457,15 +530,7 @@ https://download.swift.org/swift-5.5.1-release/ubuntu1604/swift-5.5.1-RELEASE/sw
 $ tar -xf <URL> --directory ~/.local/share/swiftly/toolchains
 ```
 
-It also updates `config.json` to include this toolchain as the latest for the provided version. If installing a new patch release toolchain, the now-outdated one can be deleted (e.g. `5.5.0` can be deleted when `5.5.1` is installed).
-
-Finally, the use implementation executes the following to update the link:
-
-```
-$ ln -s ~/.local/share/swiftly/toolchains/<toolchain>/usr/bin/swift ~/.local/bin/swift
-```
-
-It also updates `config.json` to include this version as the currently selected one.
+It also updates `config.json` to include this toolchain as the latest for the provided version. If installing a new patch release toolchain, the now-outdated one can be deleted (e.g. `5.5.0` can be deleted when `5.5.1` is installed). The `config.json` is updated to include this version as the currently selected (default) one.
 
 ### Implementation Sketch - macOS
 
@@ -481,18 +546,13 @@ https://download.swift.org/swift-<version>-RELEASE/xcode/swift-<version>-RELEASE
 
 `config.json` is then updated to include this toolchain as the latest for the provided version.
 
-Finally, the use implementation executes the following to update the link:
-
-```
-$ ln -s ~/Library/Developer/Toolchains/<toolchain name> ~/.swiftly/active-toolchain
-```
-
-It also updates `config.json` to include this version as the currently selected one.
+It also updates `config.json` to include this version as the currently selected (default) one.
 
 ### `config.json` Schema
 
 ```
 {
+  "version": "<version of swiftly that created/updated this config.json file>",
   "platform": {
     "namePretty": <OS name pretty printed>,
     "fullName": <OS name used in toolchain file name>,

Documentation/SwiftlyDocs.docc/SwiftlyDocs.md

@@ -13,6 +13,7 @@ Install and manage your Swift programming language toolchains.
 ### HOWTOS
 
 - <doc:install-toolchains>
+- <doc:use-toolchains>
 - <doc:uninstall-toolchains>
 - <doc:update-toolchain>
 - <doc:automated-install>

Documentation/SwiftlyDocs.docc/getting-started.md

@@ -27,9 +27,11 @@ $ swift --version
 
 Swift version 5.8.1 (swift-5.8.1-RELEASE)
 Target: x86_64-unknown-linux-gnu
+
+$ swift build    # Build with the latest (5.8.1) toolchain
 ```
 
-Or, you can install (and use) a swift release:
+You can install (and use) another release toolchain:
 
 ```
 $ swiftly install --use 5.7
@@ -38,12 +40,27 @@ $ swift --version
 
 Swift version 5.7.2 (swift-5.7.2-RELEASE)
 Target: x86_64-unknown-linux-gnu
+
+$ swift build    # Build with the 5.7.2 toolchain
 ```
 
-There's also an option to install the latest snapshot release and get access to the latest features:
+Quickly test your package with the latest nightly snapshot to prepare for the next release:
 
 ```
 $ swiftly install main-snapshot
+$ swiftly run swift test +main-snapshot   # Run "swift test" with the main-snapshot toolchain
+$ swift build                             # Continue to build with my usual toolchain
 ```
 
-> Note: This last example just installed the toolchain. You can run "swiftly use" to switch to it and other installed toolchahins when you're ready.
+Uninstall this toolchain after you're finished with it:
+
+```
+$ swiftly uninstall main-snapshot
+```
+
+# See Also:
+
+- [Install Toolchains](install-toolchains)
+- [Using Toolchains](use-toolchains)
+- [Uninstall Toolchains](uninstall-toolchains)
+- [Swiftly CLI Reference](swiftly-cli-reference)