ARM architecture support. #99

ProKn1fe · 2023-03-05T10:45:19Z

This allow only run exists tor files (set TorSharpSettings.ZippedToolsDirectory and TorSharpSettings.UseExistingTools), because fetch and run code must be entire rewritten.
Arm version can be downloaded here https://sourceforge.net/projects/tor-browser-ports/files/ but they have different folder structure and it require to repack .tar.gz archive to get it work.

For example tor-browser-linux-arm64-12.0.3_ALL.tar.xz:
Tor and Data folders are located in tor-browser\Browser\TorBrowser.
Repack example:
tor-linux-aarch64-12.0.3.tar.gz

Maybe create nuget package to provide repacked tar.gz archives? Or rewrite current code to support auto download (those archive size larger that x86 bundles because they contains entire browser) and different folders structure?

Tested on Radxa Rock-5B board.

var settings = new TorSharpSettings
{
    PrivoxySettings = { Disable = true }
};

using var httpClient = new HttpClient();
if (RuntimeInformation.ProcessArchitecture == Architecture.Arm64)
{
    settings.ZippedToolsDirectory = Path.Combine(AppContext.BaseDirectory, "Tor");
    settings.UseExistingTools = true;
}
else
{
    var fetcher = new TorSharpToolFetcher(settings, httpClient);
    await fetcher.FetchAsync();
}

TorProxy = new TorSharpProxy(settings);
await TorProxy.ConfigureAndStartAsync();

var handler = new HttpClientHandler
{
    Proxy = new WebProxy(new Uri("socks5://localhost:" + settings.TorSettings.SocksPort))
};
HttpClient = new HttpClient(handler)
{
    Timeout = TimeSpan.FromSeconds(300)
};
HttpClient.DefaultRequestHeaders.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/109.0.0.0 Safari/537.36");

Initial aarch64 support that allow this library to run tor on arm based systems. Signed-off-by: Zuev Aleksandr <[email protected]>

Exclude arm64 from test. Signed-off-by: Zuev Aleksandr <[email protected]>

joelverhagen · 2023-03-05T14:34:17Z

Interesting concept. I really like the idea of supporting more platforms (similar to the macOS gap tracked here: #41).

For ARM64, there are a couple of additional challenges:

No CI support for ARM64 (Support Linux ARM64 build images actions/runner-images#5631) so automated testing would be harder/impossible.
No official distribution of ARM64 Tor. The binaries linked are unofficial. I wonder if the trustworthiness is the same. Tor, perhaps more than other binary distributions, should be a trusted executable with well-understood provenance.

Regarding repacking the archive format: it looks like the (unofficial) ARM64 binary distribution is the "browser bundle" rather than the "expert bundle". TorSharp used to handle both of these formats (see this diff as an example 673a359#diff-a6e7e09d5cdd0cb97db6eed09e1d45e7df7475fed489629a708c9321a1a3d7b5) so that could be reintroduced for ARM64. In other words, repacking should not be necessary.

I don't have an ARM64 PC/board so I wouldn't be able to drive this implementation to completion.

If you'd like to bring this support into the main TorSharp version, I have a couple of ideas:

Add an option called "AllowUnofficialDistributions" or something like that which folks need to opt in to (manually set from the default false to true) to enable the ARM64 fetcher to download from this unofficial place.
Modify the Tor fetcher to bring back the browser bundle compatibility removed in 673a359, but just for ARM64.
Disable Privoxy on ARM64 since I don't see an ARM64 build for it (example: https://github.com/joelverhagen/TorSharp/blob/release/samples/NativeSocksProxy/Program.cs)

ProKn1fe · 2023-03-06T17:32:23Z

Yes i think i can provide full arm support.

I agree with AllowUnofficialDistributions. I think also for any linux system it is possible try to find if system already have installed or running tor?
For example i install tor on my board and it already running as service. The only difference is default port 9050 not 19050.

apt install tor
whereis tor
tor: /usr/bin/tor /usr/sbin/tor /etc/tor /usr/share/tor /usr/share/man/man1/tor.1.gz
whereis torrc
torrc: /usr/share/man/man5/torrc.5.gz
Also torrc in /etc/tor/torrc

If it already running it show config path and it possible to parse port from config:

ps aux | grep tor
debian-+   14662 10.1  0.2  39012 34660 ?        Ss   21:01   0:00 /usr/bin/tor --defaults-torrc /usr/share/tor/tor-service-defaults-torrc -f /etc/tor/torrc --RunAsDaemon 0

Oh i see there is easy way to parse sourceforge repository.
It's actually also exists in repository and it can be detected if system already have it.

apt install privoxy
whereis privoxy
privoxy: /usr/sbin/privoxy /etc/privoxy /usr/share/privoxy /usr/share/man/man8/privoxy.8.gz
ps aux | grep privoxy
privoxy    15049  0.0  0.0   5468  3528 ?        Ss   21:07   0:00 /usr/sbin/privoxy --pidfile /run/privoxy.pid --user privoxy /etc/privoxy/config

Something like (both false by default):

public class TorSharpSettings
{
    public bool DetectSystemHaveTor { get; set; }
    public bool DetectSystemHavePrivoxy { get; set; }
}

If tor in system detected but cannot run skip it and go by default way.
Also i see tor and privoxy configs are auto generated so it make system tor usage more easy.

joelverhagen · 2023-03-06T17:39:52Z

Using the system-wide Privoxy or Tor would be great. It's roughly supported in Privoxy, like this:
https://github.com/joelverhagen/TorSharp#privoxy-fetched-by-torsharp-fails-to-start-try-executablepathoverride

This could be extended to work with Tor just like you mentioned.

One tricky part is that the process and config should still be managed by TorSharp, it's just a matter of using the binary available on the system. So I guess you can try depending on tor in the path and then use tor -f <torrc> to pass the generated config file to tor instead of using the default system one (which I assume exists for distro-provided tor).

You could also opt for this required system-installed Tor/Privoxy on ARM64 if you don't want to do the unofficial tool fetcher. I think the tool fetcher is very convenient so it's up to you how far you want to go.

joelverhagen · 2023-03-06T17:40:52Z

Oh and whatever new options you add, feel free to set the defaults to something that is nice and helpful on the ARM64 system. We shouldn't mess with existing platform defaults but new platforms can have new defaults that are useful and make sense on that plat.

Automate binaries detection in linux. Signed-off-by: Zuev Aleksandr <[email protected]>

Armhf/Arm64 auto download feature. Signed-off-by: Zuev Aleksandr <[email protected]>

ARM archive correct extract. Signed-off-by: Zuev Aleksandr <[email protected]>

Fix unsecure Path.GetTempFileName(). Signed-off-by: Zuev Aleksandr <[email protected]>

process == null check goes after use methods inside variable. Signed-off-by: Zuev Aleksandr <[email protected]>

Remove useless null check. Simplify code. Make class static. Signed-off-by: Zuev Aleksandr <[email protected]>

joelverhagen · 2023-03-10T14:18:54Z

src/TorSharp/ToolUtility.cs

+                return false;
+
+            // In linux most binaries exists in any bin directory, e.g /bin/ /sbin/ /usr/bin/
+            var toolPath = WhereIsUtility.WhereIs(toolSettings.TryFindExecutableName);


I'm curious why whereis is used instead of which. Isn't the goal here just to find some system-provided executable? I think one benefit of using which is that it only returns the first option by default. Consider a case where, prior to running the tool, the user adds tor or privoxy into PATH manually which (generally) is a way to override which executable is used in subsequent steps of a script. It might be nice to use where in such a case since I believe is documented to respect PATH.

Additionally, isn't enforcing /bin/ being in the path an unnecessary restriction? It seems to me that as long as where returns a non-empty result for an executable name, we can just put that executable name in ExecutablePathOverride and forget about the absolute full path to it (much like what privoxy does today with PrivoxySettings.ExecutablePathOverride).

It's whereis simply because of linux command whereis.

Additionally, isn't enforcing /bin/ being in the path an unnecessary restriction

Actually search in bin directory can be removed because whereis by default search in limited places as PATH variable.
But if user know where he put executable is it much easy for him just set ExecutablePathOverride them self, no?

By default whereis tries to find files from hard-coded paths, which are defined with glob patterns. The command attempts to use the contents of $PATH and $MANPATH environment variables as default search path. The easiest way to know what paths are in use is to add the -l listing option. Effects of the -B, -M, and -S are displayed with -l.

https://manpages.ubuntu.com/manpages/bionic/man1/whereis.1.html

we can just put that executable name in ExecutablePathOverride

I think this is can be also target directory when binary will be searched.

But if user know where he put executable is it much easy for him just set ExecutablePathOverride them self, no?

Yes, I agree. But from my play with whereis it seems to provide a more complex set of output and behavior than the simple which tool which is also available on all distros I've used (perhaps maybe not arm64 distros and that's what I'm missing)? Maybe I was naive in thinking "which" is the canonical (no pun intended) way to find the resolve path of an executable.

Seems which is better solution here. I now remember why regex here, it's because whereis return more that just executable and regex needs to select from multiple matches. which return only executable.

joelverhagen · 2023-03-10T14:19:53Z

src/TorSharp/Tools/DirectoryUtility.cs

+        public static void CreateDirectoryIfNotExists(string path)
+        {
+            if (!Directory.Exists(path))
+                Directory.CreateDirectory(path);


Directory.CreateDirectory already gracefully handles the case when the directory already exists. I think this CreateDirectoryIfNotExists method is not necessary.

joelverhagen · 2023-03-10T14:21:25Z

src/TorSharp/Tools/SimpleToolRunner.cs

@@ -34,26 +34,18 @@ public Task StartAsync(Tool tool)
                startInfo.EnvironmentVariables[pair.Key] = pair.Value;
            }

-            Process process = Process.Start(startInfo);
+            Process process = Process.Start(startInfo) ?? 


Nice catch. I didn't know Process.Start could return null but I see that in the docs now.

src/TorSharp/Tools/Tool.cs

joelverhagen · 2023-03-10T14:22:19Z

src/TorSharp/Tools/Tool.cs

@@ -41,5 +41,10 @@ internal class Tool
        /// The full path to the tool's configuration file.
        /// </summary>
        public string ConfigurationPath { get; set; }
+
+        /// <summary>
+        /// Automate detected in system.


Suggested change

/// Automate detected in system.

/// The tool was automatically detected in the PATH.

(assuming we switch to which)

src/TorSharp/Tools/ToolSettings.cs

joelverhagen · 2023-03-10T14:23:32Z

src/TorSharp/Tools/Tor/TorFetcher.cs

+            if (_settings.Architecture.HasFlag(TorSharpArchitecture.Arm))
+            {
+                if (!_settings.AllowUnofficialSources)
+                    throw new TorSharpException($"AllowUnofficialDistribution == false, there is no official tor distribution for {_settings.Architecture} platform.");


This is great. Thanks!

src/TorSharp/Tools/Tor/TorFetcher.cs

src/TorSharp/TorSharp.csproj

src/TorSharp/TorSharpArchitecture.cs

src/TorSharp/TorSharpPrivoxySettings.cs

joelverhagen · 2023-03-10T14:32:39Z

src/TorSharp/TorSharpSettings.cs

 #else
            OSPlatform = TorSharpOSPlatform.Windows;
            Architecture = Environment.Is64BitProcess ? TorSharpArchitecture.X64 : TorSharpArchitecture.X86;
 #endif

            PrivoxySettings = new TorSharpPrivoxySettings();
            TorSettings = new TorSharpTorSettings();
+
+#if NETSTANDARD
+            if (OSPlatform == TorSharpOSPlatform.Linux)


This is a change in default behavior. Can we just scope this to the Arm32 and Arm64 architectures?

One drawback of defaulting to the system-provided on is that the Tor or Privoxy version on the system (i.e. provided by distro package maintainers) may not update to the latest version as quickly as what the tool fetcher in TorSharp can provide (which takes the latest build released by the upstream maintainers).

may not update to the latest version

Fair. But still bottom page of readme says why this is better for linux) Last version not always best and most dists e.g ubuntu or debian update security flaws in packages and yes it almost never latest version.

It's a good point. Using the system version is probably better on Linux.

If you want to change the defaults for non-ARM Linux, make an update to the README with an FAQ on how to revert to the previous behavior (2.14.0 and older, using some configuration setup) where the system-provided version is not used by default. That way people can choose to opt out of this behavior change if they want.

src/TorSharp/TorSharpTorSettings.cs

test/TorSharp.Tests/TorSharpFetcherTests.cs

joelverhagen · 2023-03-10T14:36:13Z

test/TorSharp.Tests/ARM/FetcherTests.cs

+            Assert.NotNull(result.Url);
+        }
+
+        //[Fact]


You can delete commented code

joelverhagen · 2023-03-10T14:37:13Z

test/TorSharp.Tests/ARM/FetcherTests.cs

+
+using Xunit;
+
+namespace Knapcode.TorSharp.Tests.ARM


We should have a test suite for the system-provided privoxy and tor. I wonder if we can do an apt-get install for privoxy and tor in the GitHub Actions YAML and then test the system-provided flow.

I believe you can run any command inside. Docs says you can run apt install.
https://docs.github.com/en/actions/using-github-hosted-runners/customizing-github-hosted-runners

Directory.Build.props

joelverhagen

This is looking great so far! I've left some comments about the system-provided behavior, testing on GitHub Actions, and some nitpicks.

I'd like to personally try this out on QEMU -- which I just learned is a thing -- and maybe my Raspberry pi (which I forgot I had 😂)

joelverhagen · 2023-03-10T14:43:14Z

src/TorSharp/Tools/LineByLineConfigurer.cs

@@ -26,7 +26,7 @@ public async Task ApplySettings(Tool tool, TorSharpSettings settings)
            try
            {
                // write first to a temporary file
-                temporaryPath = Path.GetTempFileName();
+                temporaryPath = Path.Combine(Path.GetTempPath(), Path.GetRandomFileName());


Interesting: https://rules.sonarsource.com/csharp/RSPEC-5445.
I never heard of this before. Thanks!

Change some variable names and comments. Signed-off-by: Zuev Aleksandr <[email protected]> Co-authored-by: Joel Verhagen <[email protected]>

Apply variable rename in code. Signed-off-by: Zuev Aleksandr <[email protected]>

Replace whereis with which. Add windows auto search.

src/TorSharp/Tools/WhichUtility.cs

ProKn1fe added 2 commits March 5, 2023 14:27

Initial arm64 support.

8ef264e

Initial aarch64 support that allow this library to run tor on arm based systems. Signed-off-by: Zuev Aleksandr <[email protected]>

Exclude arm64 from test.

fa4e52b

Exclude arm64 from test. Signed-off-by: Zuev Aleksandr <[email protected]>

ProKn1fe and others added 7 commits March 7, 2023 02:23

Automate binaries detection in linux

5053ab9

Automate binaries detection in linux. Signed-off-by: Zuev Aleksandr <[email protected]>

ARM auto download

b9efdab

Armhf/Arm64 auto download feature. Signed-off-by: Zuev Aleksandr <[email protected]>

ARM archive correct extract

9483fc3

ARM archive correct extract. Signed-off-by: Zuev Aleksandr <[email protected]>

Fix unsecure Path.GetTempFileName()

1b2c507

Fix unsecure Path.GetTempFileName(). Signed-off-by: Zuev Aleksandr <[email protected]>

Fix wrong null check.

585dd5e

process == null check goes after use methods inside variable. Signed-off-by: Zuev Aleksandr <[email protected]>

Minor code changes

54662fa

Remove useless null check. Simplify code. Make class static. Signed-off-by: Zuev Aleksandr <[email protected]>

Fix branch name to version mapping

594da7a