Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Rework the pages about dist cross-compiling #20

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
Open

Rework the pages about dist cross-compiling #20

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
5d7adde
Move paragraphs a level below
solsticedhiver Sep 14, 2019
3f1a662
Remove b tag for mardown syntax
solsticedhiver Sep 14, 2019
833f98e
More markdown syntax and add new lines
solsticedhiver Sep 14, 2019
b1d7f2b
Rework intro
solsticedhiver Sep 14, 2019
9a61bb2
AUR packages
solsticedhiver Sep 14, 2019
59fcf26
Things here and there
solsticedhiver Sep 14, 2019
5a4ccf9
distcc/distccd and no archlinux needed
solsticedhiver Sep 14, 2019
69eebc1
Correct spelling of crosstool-NG
solsticedhiver Sep 14, 2019
b6b93de
distcc and crosstool-NG
solsticedhiver Sep 14, 2019
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 32 additions & 16 deletions Distcc_Cross-Compiling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,37 @@
This is the **official** cross-compiling method used at **Arch Linux ARM**. If you plan on building a lot of packages and want to speed up the process, the following guide will turn an *x86* Linux computer into an ARM cross-compiler. It's also much easier than most cross-compile setups.

## Introduction
*Disclaimer: This guide will appear vague and incomplete if you aren't sure what you're doing. This is intentional. This is specifically not designed for users new to software compilation and toolchain components.*
This guide makes use of *distcc* and *crosstool-NG* in order to not have to build a full ARM development environment on x86.

This is the official cross-compiling method used at Arch Linux ARM. If you plan on building a lot of packages and want to speed up the process, the following guide will turn an x86 Linux computer into an ARM cross-compiler. It's also much easier than most cross-compile setups.
*distcc* allow us to use network clients to spread the load of the compilation to slaves from a master. For example, an ARM (distcc) master will send its compile jobs to *x86_64* (distccd) clients that will use *crosstool-NG* toolchain. The clients don't need to run *archlinux*.

This guide makes use of distcc in order to not have to build a full ARM development environment on x86. As the distcc project website states, "distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed." This is particularly advantageous to us since all that is needed is a working cross-compiler for ARM on a faster machine, while controlling the build from an ARM computer that has all of the current libraries and headers.
As the *distcc* project website states, "distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed." This is particularly advantageous to us since all that is needed is a working cross-compiler for ARM on a faster machine, while controlling the build from an ARM computer that has all of the current libraries and headers.

## Distcc
Before beginning, a distcc environment needs to be set up. Follow the [[Distributed Compiling]] guide to establish a master system. The x86 machines will be known as the clients.
Before beginning, a *distcc* environment needs to be set up. Follow the [[Distributed Compiling]] guide to establish a master system. The x86 machines will be known as the clients.

## Pre-built crosstool-ng toolchains
In lieu of building the toolchain as detailed below, if you are running a 64-bit Linux installation you can use these packaged toolchains that are employed in the official build system.
## Pre-built crosstool-NG toolchains
In lieu of building the toolchain as detailed below, if you are running a *64-bit* Linux installation you can use these packaged toolchains that are employed in the official build system.

It is *highly recommended* to use these tarballs as they have been thoroughly tested, and are maintained to be version- and source-matched to the current toolchain components for our ARM architectures. If you build the toolchain yourself, you must assemble patched source tarballs to build versions that match what is in use here.
It is **highly recommended** to use these tarballs as they have been thoroughly tested, and are maintained to be version- and source-matched to the current toolchain components for our ARM architectures. If you build the toolchain yourself, you must assemble patched source tarballs to build versions that match what is in use here.

* [ARMv5te soft-float](/builder/xtools/x-tools.tar.xz) (8b93708e0f7ef971b01b71dfedaffe40)
* [ARMv6l hard-float](/builder/xtools/x-tools6h.tar.xz) (8b8c261c91f11545f4b54b0bddaf1db1)
* [ARMv7l hard-float](/builder/xtools/x-tools7h.tar.xz) (856aa3e81afda734deff8fb2848d5760)
* [ARMv8](/builder/xtools/x-tools8.tar.xz) (56c435708efe81dcf0e668907219c3ed)

## Building a crosstool-ng toolchain
This process is very automated, courtesy of [crosstool-ng](http://crosstool-ng.org). As a normal user (<b>not root!</b>), clone revision ec384f2b of the git repository into a directory called "cross" in your home directory. Enter the source directory and configure with a prefix for the "cross" directory, make, and make install. If you are missing any pre-requisites, the configure script will let you know what they are.
### AUR packages
For easier setup, one can use the relevant packages from [AUR](https://aur.archlinux.org). They just wrap up the above tarball into a package with included systemd services:

* [ARMv5te soft-float](https://aur.archlinux.org/packages/distccd-alarm-armv5/)
* [ARMv6l hard-float](https://aur.archlinux.org/packages/distccd-alarm-armv6h/)
* [ARMv7l hard-float](https://aur.archlinux.org/packages/distccd-alarm-armv7h/)
* [ARMv8](https://aur.archlinux.org/packages/distccd-alarm-armv8/)

They provide an easier way to setup the systemd services, and the possibility to run all 4 distccd toolchains in parallel (if needed).

## Building a crosstool-NG toolchain
This process is very automated, courtesy of [crosstool-NG](http://crosstool-ng.org). As a normal user (**not root!**), clone revision *ec384f2b* of the git repository into a directory called `cross` in your home directory. Enter the source directory and configure with a prefix for the `cross` directory, make, and make install. If you are missing any pre-requisites, the configure script will let you know what they are.

```
mkdir -p cross/src
Expand All @@ -33,25 +45,29 @@ make
make install
```

At this point crosstool-ng is ready to be configured. The program "ct-ng" in the "bin" directory is where the magic happens. It also has a menu configuration like the Linux kernel.
At this point *crosstool-NG* is ready to be configured. The program `ct-ng` in the `bin` directory is where the magic happens. It also has a menu configuration like the Linux kernel.

## Downloading a CrossTool Configuration
Download the default .config file to place in "~/cross/bin" as shown below. <b>Once you do this, do not run "menuconfig" or values will be overwritten</b>. Choose either v5, v6, v7, or v8 depending on the target platform.
### Downloading a CrossTool Configuration
Download the default .config file to place in `~/cross/bin` as shown below. **Once you do this, do not run "menuconfig" or values will be overwritten**. Choose either v5, v6, v7, or v8 depending on the target platform.

```
cd /home/your_user/cross/bin
wget http://archlinuxarm.org/builder/xtools/xtools-dotconfig-[v5|v6|v7|v8] -O .config
```

## Build the cross-toolchain
### Build the cross-toolchain

```
cd /home/your_user/cross/bin
./ct-ng build
```

## Make nice with distcc
The toolchain will install under "~/x-tools" for armv5, "~/x-tools6h" for armv6 hard-float, "~/x-tools7h" for armv7 hard-float, and "~/x-tools8" for AArch64. You can move this somewhere else if you like, or leave it where it is. Before we can use the compiler binaries that were created, links need to be created to make their names more appropriate. When compile jobs are sent to distcc, the program specified in the CC environment variable on the build master is what gets executed on the build clients. All the binaries that have been produced by crosstool-ng are in the correct format specifying the target platform as a prefix, though not with the correct platform and lacking tuple-less variants. This script will fix our problem. Make "~/x-tools[6h|7h|8]/arm-unknown-linux-gnueabi[hf]/bin/" writable and in there create a file called "link" and paste the following into it. Uncomment for your target architecture and run it.
### Make nice with distcc
The toolchain will install under `~/x-tools` for *armv5*, `~/x-tools6h` for *armv6* hard-float, `~/x-tools7h` for *armv7* hard-float, and `~/x-tools8` for *AArch64*. You can move this somewhere else if you like, or leave it where it is.

Before we can use the compiler binaries that were created, links need to be created to make their names more appropriate. When compile jobs are sent to *distcc*, the program specified in the CC environment variable on the build master is what gets executed on the build clients.

All the binaries that have been produced by *crosstool-ng* are in the correct format specifying the target platform as a prefix, though not with the correct platform and lacking tuple-less variants. This script will fix our problem. Make "~/x-tools[6h|7h|8]/arm-unknown-linux-gnueabi[hf]/bin/" writable and in there create a file called "link" and paste the following into it. Uncomment for your target architecture and run it.

```bash
#!/bin/bash
Expand All @@ -73,7 +89,7 @@ for file in `ls`; do
done
```

Now the "bin" directory contains links with names that distcc will play nice with.
Now the `bin` directory contains links with names that distcc will play nice with.

## Compile!
Back on ARM master device, make sure that distcc has been enabled in makepkg.conf per the [[Distributed Compiling]] guide. Now all builds using makepkg will make use of the distcc and cross compiler setup.
Expand Down
57 changes: 45 additions & 12 deletions Distributed_Compiling.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
If you have spare ARM or x86 systems, using distcc is a very simple way to get all of them in on compiling packages. makepkg is already set up for using distcc, all that's needed is to enable the functionality.
If you have spare *ARM* or *x86* systems, using **distcc** is a very simple way to get all of them in on compiling packages. *makepkg* is already set up for using *distcc*, all that's needed is to enable the functionality.

For distributing to other ARM systems, you will need to have base-devel installed. Each system needs to have the same packages installed that are required for compiling the package you're about to build using this method. See the [Arch Wiki](https://wiki.archlinux.org) for makepkg, pacman, etc.
For distributing to other ARM systems, you will need to have **base-devel** installed. Each system needs to have the same packages installed that are required for compiling the package you're about to build using this method. See the [Arch Wiki](https://wiki.archlinux.org) for makepkg, pacman, etc.

For distributing to x86 systems, install the cross compilers for the needed targets from the [[Distcc Cross-Compiling]] page.
For distributing to x86 systems, install the cross compilers for the needed target architectures from the [[Distcc Cross-Compiling]] page.

Install distcc on all systems to be used:

Expand All @@ -11,7 +11,7 @@ pacman -S distcc
```

## Configuration: Introduction
For the purposes of this example, all of the hosts are in the local network of 10.3.0.0/24, and access is allowed from that entire subnet. If you want to restrict access further, you will have to specify individual IPs. Be sure to change the address from the examples to fit your local network configuration.
For the purposes of this example, all of the hosts are in the local network of `10.3.0.0/24`, and access is allowed from that entire subnet. If you want to restrict access further, you will have to specify individual IPs. Be sure to change the address from the examples to fit your local network configuration.

Additionally, this example assumes that you are starting compiles from one specific system (master), which is sending compiles out to the other system(s) (client). If you want to compile from any system, simply apply the client configuration to the master system, and the master system configuration to the client system(s).

Expand All @@ -20,25 +20,37 @@ If you want to use the master system to compile as well, apply the client config
## Configuration: Master System
These are the configurations for the ARM system you'll be running *makepkg* from.

In /etc/makepkg.conf, or in makepkg.conf in your chroot, change BUILDENV to un-negate the distcc option:
In `/etc/makepkg.conf`, or in `makepkg.conf` in your *chroot*, change BUILDENV to un-negate the distcc option:

```
BUILDENV=(distcc color !ccache check !sign)
```

Change DISTCC_HOSTS to a space-delimited list of the distcc clients on your network. If you want to use the master system, be sure to put it's IP address in there too. The hosts listed will be prioritized in the order listed. To keep configurations simple, use the LAN IP address for the local system, not "localhost" or "127.0.0.1".
Change `DISTCC_HOSTS` to a space-delimited list of the *distcc* clients on your network.

If you want to use the master system, be sure to put its IP address in there too. The hosts listed will be prioritized in the order listed. To keep configurations simple, use the LAN IP address for the local system, not "localhost" or "127.0.0.1".

```
DISTCC_HOSTS="10.3.0.6 10.3.0.2"
```

Additional options can be specified in DISTCC_HOSTS to control how distcc uses the hosts. For example, to randomize which host is used, limit the maximum number of jobs sent to the first host to 4 and the second host to 8, and to specify a different port that distccd is running on:
Additional options can be specified in `DISTCC_HOSTS` to control how distcc uses the hosts.

- You can specify the port on which the distant distcc runs by appending it after the IP like:

```
DISTCC_HOSTS="--randomize 10.3.0.6:3932/4 10.3.0.2:3932/8"
DISTCC_HOSTS="10.3.0.6:3634"
```

Change the MAKEFLAGS -j flag to reflect the total number of processors available on the master system. The common wisdom is to set this to the number of physical cores + 1. Keep in mind that only compiles are distributed; preprocessing and linking still takes place on the master system. Therefore, this number should reflect the capabilities of the master system and not the total number of distributed cores available.
- For example, to randomize which host is used, limit the maximum number of jobs sent to the first host to 4 and the second host to 8, and to specify a different port that distccd is running on:

```
DISTCC_HOSTS="--randomize 10.3.0.6/4 10.3.0.2/8"
```

Change the MAKEFLAGS -j flag to reflect the total number of processors available on the master system. The common wisdom is to set this to the number of physical cores + 1.
Keep in mind that only compiles are distributed; preprocessing and linking still takes place on the master system.
Therefore, this number should reflect the capabilities of the master system and not the total number of distributed cores available.

```
MAKEFLAGS="-j3"
Expand All @@ -47,13 +59,23 @@ MAKEFLAGS="-j3"
## Configuration: Client System(s)
These are the configurations for a second ARM system or x86 cross-compiling computer.

In /etc/conf.d/distccd, change DISTCC_ARGS to reflect the hosts or network you're allowing to connect and the number of distccd instances (jobs) to start. These arguments will be passed to distccd when it's started, so this is a good place to add any additional arguments you want.
In `/etc/conf.d/distccd`, change `DISTCC_ARGS` to reflect the hosts or network you're allowing to connect and the number of distccd instances (jobs) to start.

These arguments will be passed to distccd when it's started, so this is a good place to add any additional arguments you want.

```
DISTCC_ARGS="--allow 10.3.0.0/24 --jobs 20"
```

As of distcc 3.3, protections for [CVE-2004-2687](https://nvd.nist.gov/vuln/detail/CVE-2004-2687) have been implemented. Specifically, the symlinks to distcc in /usr/lib/distcc are a whitelist of what executables are allowed to be run through distcc. For cross-compiling systems, this means that symlinks of each ARM architecture's compiler must be created (c++, cc, cpp, g++, and gcc). For example:
If you have used the AUR package, just edit the relevant conf file like, for example, `/etc/conf.d/distccd-armv6h`.

### Symlinks

Note: This part is not needed for AUR package users.

As of distcc 3.3, protections for [CVE-2004-2687](https://nvd.nist.gov/vuln/detail/CVE-2004-2687) have been implemented. Specifically, the symlinks to distcc in `/usr/lib/distcc` are a whitelist of what executables are allowed to be run through distcc.

For cross-compiling systems, this means that symlinks of each ARM architecture's compiler must be created (c++, cc, cpp, g++, and gcc). For example:

```
ln -s /usr/bin/distcc /usr/lib/distcc/aarch64-unknown-linux-gnu-gcc
Expand All @@ -68,8 +90,17 @@ On ARM systems where the installed toolchain will be used, simply enable the shi
systemctl enable distccd
systemctl start distccd
```
If you use the AUR packages, *systemd* services have already been installed. Just start (and enable) the needed services. For example:

```
systemctl start distccd-armv6h
```

#### Manual method

On x86 systems where a cross toolchain will be used, create a new systemd service file for the toolchain. When setting up for cross-compiling multiple ARM architectures, each service file should start distcc with different ports from the default (3632) and specify a unique log file. The new port number must be set in DISTCC_HOSTS in makepkg.conf. This example shows a service file for the AArch64 toolchain in a multiple toolchain environment:
On x86 systems where a cross toolchain will be used, create a new systemd service file for the toolchain. When setting up for cross-compiling multiple ARM architectures, each service file should start distcc with different ports from the default (3632) and specify a unique log file.

The new port number must be set in DISTCC_HOSTS in makepkg.conf. This example shows a service file for the AArch64 toolchain in a multiple toolchain environment:

`/etc/systemd/system/distccd8.service`

Expand Down Expand Up @@ -100,4 +131,6 @@ systemctl enable distccd8
systemctl start distccd8
```

## Run

You can now run *makepkg* as you normally would and the hosts specified will start receiving work. If the hosts are unavailable, the package will simply compile locally.