Supporting Windows

[Connoropolous]

Based on attempts to bring applications like Snapmail and Acorn out into the wild bundled as Electron apps, we have run in to the requirement that Windows users install wsl2 which is a big technical hurdle for ordinary users. We have been wanting to make these apps available at lower friction. When @ddd-mtl @guillemcordoba and I discussed recently with @zippy it was brought up that this "Unix Domain Socket for Lair" thing might be the main blocker for native Windows support.

We are thinking about investigating it and seeing what we can do to move this forward. I started a bit of digging and immediately found something interesting, which appears to be that an implementation (if not 100% feature complete) of Unix sockets was brought into Windows 10, with a release in 2017 first to Windows Insider, then it became generally available in version 1809 (aka the October 2018 Update).
Microsoft open source seems to have published a library that may suit:
https://github.com/Azure/tokio-uds-windows
There's also the Tokio UDS on Windows thread to add:
tokio-rs/tokio#2201
The new Deno language (like nodejs) is also pulling for this clearly.

Were we collectively unaware of this? I am assuming so, given these lines

lair/crates/lair_keystore_api/src/internal/ipc.rs

Lines 11 to 19 in 3bd7105

	#[cfg(not(windows))]
	mod unix_ipc;
	#[cfg(not(windows))]
	use unix_ipc::*;
	#[cfg(windows)]
	mod win_ipc;
	#[cfg(windows)]
	use win_ipc::*;

and the comments in

lair/crates/lair_keystore_api/src/internal/ipc/win_ipc.rs

Line 1 in 3bd7105

//! windows version of ipc stream tools

The original post is here, from 2017
https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/
But there's more relevant comments here
microsoft/WSL#4240

I found looking at this commit which referred to that issue gave a bit of insight
MisterDA/ocaml@5855ce5

lair/.github/workflows/ops.test.yml

Lines 14 to 15 in 3bd7105

	# @todo windows support
	#windows-latest,

@thedavidmeister @neonphog

[ddd-mtl]

I was planning on looking into it this month. The problem is with tokio since UnixStreams dont compile on windows. The fix is to add support for TcpStreams which can we used on windows, regardless of version.

…

On Thu., Jul. 1, 2021, 20:51 Connor Turland, ***@***.***> wrote: Based on attempts to bring applications like Snapmail and Acorn out into the wild bundled as Electron apps, we have run in to the requirement that Windows users install wsl2 which is a big technical hurdle for ordinary users. We have been wanting to make these apps available at lower friction. When @ddd-mtl <https://github.com/ddd-mtl> @guillemcordoba <https://github.com/guillemcordoba> and I discussed recently with @zippy <https://github.com/zippy> it was brought up that this "Unix Domain Socket for Lair" thing might be the main blocker for native Windows support. We are thinking about investigating it and seeing what we can do to move this forward. I started a bit of digging and immediately found something interesting, which appears to be that an implementation (if not 100% feature complete) of Unix sockets was brought into Windows 10, with a release in 2017. Were we collectively unaware of this? I am assuming so, given these lines https://github.com/holochain/lair/blob/3bd7105108ab241d6719e200dd15905cd3e74da1/crates/lair_keystore_api/src/internal/ipc.rs#L11-L19 and the comments in https://github.com/holochain/lair/blob/3bd7105108ab241d6719e200dd15905cd3e74da1/crates/lair_keystore_api/src/internal/ipc/win_ipc.rs#L1 The original post is here, from 2017 https://devblogs.microsoft.com/commandline/af_unix-comes-to-windows/ But there's more relevant comments here microsoft/WSL#4240 <microsoft/WSL#4240> I found looking at this commit which referred to that issue gave a bit of insight ***@***.*** <MisterDA/ocaml@5855ce5> https://github.com/holochain/lair/blob/3bd7105108ab241d6719e200dd15905cd3e74da1/.github/workflows/ops.test.yml#L14-L15 @thedavidmeister <https://github.com/thedavidmeister> @neonphog <https://github.com/neonphog> — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEPDHPZZXVYIW32FJDQJELTVUERXANCNFSM47VXJV2A> .

[Connoropolous]

I see, so you think it's a better solution to add it (do you estimate it to be more work?) because it adds broader compatibility?

[ddd-mtl]

Also because Lair is supposed to work over TCP anyway. So it's a 2 birds one stone solution.

…

On Thu., Jul. 1, 2021, 21:23 Connor Turland, ***@***.***> wrote: I see, so you think it's a better solution to add it (do you estimate it to be more work?) because it adds broader compatibility? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEPDHPF5GKCQCOZTPZHX23TVUII3ANCNFSM47VXJV2A> .

[Connoropolous]

Cool cool. That's good and there might be one additional prospect here which is that Unix Sockets works on all of mac/linux/windows AND you could select via config to use TCP.
UnixStream for windows for tokio was implemented here by Microsoft devs themselves, so could be used:
https://github.com/Azure/tokio-uds-windows

3 birds with 2 stones?

[ddd-mtl]

Oh that's cool. Should be easier to use that to start with. A recent win10 version was already required for WSL anyway.

…

On Thu., Jul. 1, 2021, 21:33 Connor Turland, ***@***.***> wrote: Cool cool. That's good and there might be one additional prospect here which is that UDP works on both mac/linux/windows AND you can select for via config to use TCP. UnixStream for windows for tokio was implemented here by Microsoft devs themselves, so could be used: https://github.com/Azure/tokio-uds-windows 3 birds with 2 stones? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#34 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AIEPDHNDH3H3PAUHLRQZASDTVUJM7ANCNFSM47VXJV2A> .

[Connoropolous]

my thoughts exactly

[neonphog]

@Connoropolous @ddd-mtl - I hadn't been intending to introduce TCP to lair because of security concerns, although we could discuss it.

The original intention was to use tokio-named-pipes on windows.

[Connoropolous]

Oh, I see. Different stories going around here then.

Sounds like the named pipes solution would be the number 1 preference for you @neonphog ?

It seems to be folded right into tokio nowadays:
https://docs.rs/tokio/1.8.0/tokio/net/windows/named_pipe/index.html

[neonphog]

@Connoropolous

It seems to be folded right into tokio nowadays:
https://docs.rs/tokio/1.8.0/tokio/net/windows/named_pipe/index.html

Oh nice, I hadn't realized that, awesome!

Sounds like the named pipes solution would be the number 1 preference for you @neonphog ?

Yeah, if this is part of core tokio, let's do that to start, and see if it covers our use-cases. It eliminates a whole set of concerns if we can guarantee only local processes have access, and can apply os security.

[Connoropolous]

@neonphog
I'm glad it's a good surprise. Can you confirm a rough outline of the work @neonphog ?

My guess is that it involves writing in the win_rpc.rs file equivalent implementations of:
IpcRead, IpcWrite, ipc_connect, and IpcServer from unix_ipc.rs

Is UnixStream completely abstracted out?

[neonphog]

@Connoropolous

My guess is that it involves writing in the win_rpc.rs file equivalent implementations of:
IpcRead, IpcWrite, ipc_connect, and IpcServer from unix_ipc.rs

Is UnixStream completely abstracted out?

You got it. IpcRead must implement tokio::io::AsyncRead, IpcWrite tokio::io::AsyncWrite, and ipc_connect / IpcServer should have the same signatures. They're switched out at compile time with config directives here.

[ddd-mtl]

Actually TCP was my extrapolation for the Holo use-case and the lair-shim thing, but now I see I could be totally wrong so nevermind the TcpStream solution I mentionned.

[Connoropolous]

@ddd-mtl I have access to Windows on a laptop here dual booted from a Mac, so I can co-code, or also run and compile code you write, etc. Any preferences? :)

[thedavidmeister]

this looks like a good thing that is happening here, but i don't have much to add so i'm taking myself off notifications :)

[neonphog]

v0.0.1-alpha.13 keystore / api / client published with #37 included. Closing this -- feel free to re-open if there's anything remaining to be addressed.