doc: end host stack projects overview

[matzf]

There are multiple development projects that concern the SCION end host stack.
This document aims to give an overview on how these parts will fit together.

---

This change is 

[marcfrei]

Reviewed 4 of 5 files at r1, 1 of 1 files at r2, all commit messages.
Reviewable status: all files reviewed, 24 unresolved discussions (waiting on @matzf)

---

doc/dev/design/endhost-overview.rst line 42 at r2 (raw file):

    and dispatching them to the relevant application socket.
  - transmitting outgoing to the network (from UDP 30041).
  - Communicates to applications over bespoke API via Unix sockets.

communicating

Code quote:

Communicates

---

doc/dev/design/endhost-overview.rst line 43 at r2 (raw file):

  - transmitting outgoing to the network (from UDP 30041).
  - Communicates to applications over bespoke API via Unix sockets.
  - Maintains mapping of ports to application sockets.

maintaining a

Code quote:

Maintains

---

doc/dev/design/endhost-overview.rst line 62 at r2 (raw file):

- Dispatcher removal (:issue:`4280`).

  As part of this chage, BR will be updated to inspects the L4 header and to deliver packets directly to the applications underlay port for a locally configured range.

change

Code quote:

chage

---

doc/dev/design/endhost-overview.rst line 68 at r2 (raw file):

  On some platforms, there is no good path to install and run the default SCION stack with the daemon and the dispatcher as a dependency for running individual SCION-native applications.
  This was one of the observations that led the dispatcher removal work.

led to the

Code quote:

led the

---

doc/dev/design/endhost-overview.rst line 72 at r2 (raw file):

  The "shim" is intentionally designed to be optional; if it's not there, applications are only
  restricted to receive UDP/SCION packets, no SCMP or other L4 types.

only receive

Suggestion:

are only restricted to receive

---

doc/dev/design/endhost-overview.rst line 76 at r2 (raw file):

  the "shim" allows end applications to still receive traffic from those legacy BRs.

  The functionality of the daemon, however, is necessary. It might already possible to bundle the daemon into an

already be possible

Code quote:

already possible

---

doc/dev/design/endhost-overview.rst line 78 at r2 (raw file):

  The functionality of the daemon, however, is necessary. It might already possible to bundle the daemon into an
  application, but it is certainly tedious and not well supported.
  The (simplified) functionality of the daemon should be made available as plug-able library.

remove?

Code quote:

plug-able

---

doc/dev/design/endhost-overview.rst line 80 at r2 (raw file):

  The (simplified) functionality of the daemon should be made available as plug-able library.

- Automated end host configuration integrated into scion daemon.

Is this referring to the bootstrapper? Maybe reference the corresponding repo like for 'pan' below?

Code quote:

Automated end host configuration

---

doc/dev/design/endhost-overview.rst line 81 at r2 (raw file):

- Automated end host configuration integrated into scion daemon.
  Configuration reloading?

Say a little more about this?

Code quote:

Configuration reloading?

---

doc/dev/design/endhost-overview.rst line 84 at r2 (raw file):

- Extend or replace ``pkg/snet`` API with higher-level library like `netsec-ethz/scion-apps/pkg/pan <https://pkg.go.dev/github.com/netsec-ethz/scion-apps/pkg/pan>`_.

A compatiblity mechanism of the dispatcher removal project appears to imply that the underlay address cannot reliably be obtained for received SCION packets.

compatibility

Code quote:

compatiblity

---

doc/dev/design/endhost-overview.rst line 94 at r2 (raw file):

1. Interface ID to underlay address mapping:

   - ``snet`` keeps a table mapping Interface ID to underlay address

address.

Code quote:

address

---

doc/dev/design/endhost-overview.rst line 102 at r2 (raw file):

     path.
   - **Later**: this table is updated with an event subscription from the daemon (proto API with stream response).
     Updates are generally very infrequent and the connection to the daemon does not need to be maintained with high reliablity, e.g. for deamon restarts.

reliability

Code quote:

reliablity

---

doc/dev/design/endhost-overview.rst line 102 at r2 (raw file):

     path.
   - **Later**: this table is updated with an event subscription from the daemon (proto API with stream response).
     Updates are generally very infrequent and the connection to the daemon does not need to be maintained with high reliablity, e.g. for deamon restarts.

daemon

Code quote:

deamon

---

doc/dev/design/endhost-overview.rst line 105 at r2 (raw file):

2. The dispatcher is removed (see :issue:`4280`).
   Applications open an underlay UDP/IP socket directly and use the same port for the UDP/SCION.

remove?

Code quote:

the

---

doc/dev/design/endhost-overview.rst line 109 at r2 (raw file):

   A "shim" component takes care of locally dispatching incoming packets, a) from old routers that don't
   support the port dispatching, and b) with any L4 other than UDP/SCION.

remove?

Code quote:

with

---

doc/dev/design/endhost-overview.rst line 115 at r2 (raw file):

   :doc:`endhost-bootstrap` into the daemon and control service.

   For this, we add a proto rpc ``Interfaces()`` to the control service.

rpc call

Code quote:

proto rpc

---

doc/dev/design/endhost-overview.rst line 123 at r2 (raw file):

   The topology information is authenticated with the AS's CP-PKI signing key; either, by using this
   to establish a TLS channel, or by signing the information as a message, directly.

remove?

Code quote:

as a message, 

---

doc/dev/design/endhost-overview.rst line 131 at r2 (raw file):

   local AS's CP-PKI certificate.

   Finally, the daemon also obtain the endhost port range for the local AS, using also a rpc call ``PortRange()``.

obtains

Code quote:

obtain

---

doc/dev/design/endhost-overview.rst line 136 at r2 (raw file):

   In current ``snet``, the application is responsible for managing and selecting paths, and
   and passes a path for every individual ``Write`` call.

remove?

Suggestion:

and and passes a path

---

doc/dev/design/endhost-overview.rst line 138 at r2 (raw file):

   and passes a path for every individual ``Write`` call.
   In this model, paths often need to be piped from the application, through
   various layers (like e.g. an http library), down into the actual ``snet`` library Write; or the

into the snet library;

Suggestion:

into the actual ``snet`` library Write;

---

doc/dev/design/endhost-overview.rst line 141 at r2 (raw file):

   other way around, for reads.
   To make this possible, ``snet`` bundles a path into a ``net.Addr``.
   This, however, is both rather inconvient and conceptually unclean.

inconvenient

Code quote:

inconvient

---

doc/dev/design/endhost-overview.rst line 147 at r2 (raw file):

   granular control, but the default case becomes extremely simple for applications.
   In particular, this model does not require passing paths along with every individual Write call,
   obviating the need to bundle paths into addresses.

This is true for writes from a client. For writes from a server (i.e. writes typically representing responses) we still have the issue that the new snet/pan library would need to somehow know over which path it should reply to the client, right? For QUIC listeners, e.g., we currently have a map in PAN to lookup such reply paths, iirc. Wouldn't it be simpler at least in this case to still attach the path to the peer address when receiving a packet over a listener socket before passing the address together with the received data on to the QUIC library and eventually to the application specific server code?

---

doc/dev/design/endhost-overview.rst line 167 at r2 (raw file):

   At build time, the application can chose to initialize ``snet`` with either the default stub connecting to the daemon,
   or the in-process implementation of the same interface.
   It's also possible to choose dynamically; if no daemon is detected running at the expected

:

Code quote:

;

---

doc/dev/design/endhost-overview.rst line 177 at r2 (raw file):

   - no sharing of cached path information between applications.

   .. admonition:: Open question

At the moment, the daemon maintains DB files for various kinds of data. Would the new library have a mode to also operate purely in memory?

Code quote:

Open question

[marcfrei]

Reviewable status: all files reviewed, 24 unresolved discussions (waiting on @matzf)

[matzf]

The dispatcher removal has already been merged, #4344. This obsoletes a large part of the background description and the described topics.
The remaining "projects" are:

• Self-contained distribution of SCION-native applications, in particular making the functionality of the SCION daemon available as a library.
• Automated end host configuration (netsec-ethz/bootstrapper) integrated into scion daemon.
• Extend or replace pkg/snet API with higher-level library like netsec-ethz/scion-apps/pkg/pan.

There is not a great amount of details about any of these projects in this document. Where there are details, they mostly are my personal view, not necessarily a common vision. I expect that once work on any of these topics start, focused and detailed design proposals will be created and discussed. Instead of updating the document, I'm closing this PR now.

If this document ever seems interesting for historical reference, this is the link to the github-rendered content: https://github.com/scionproto/scion/blob/880bdc220e258f4dc57f45694052706e9b927a3d/doc/dev/design/endhost-overview.rst