diff --git a/Distcc_Cross-Compiling.md b/Distcc_Cross-Compiling.md index 674639e..59021f2 100644 --- a/Distcc_Cross-Compiling.md +++ b/Distcc_Cross-Compiling.md @@ -1,25 +1,37 @@ +This is the **official** cross-compiling method used at **Arch Linux ARM**. If you plan on building a lot of packages and want to speed up the process, the following guide will turn an *x86* Linux computer into an ARM cross-compiler. It's also much easier than most cross-compile setups. + ## Introduction -*Disclaimer: This guide will appear vague and incomplete if you aren't sure what you're doing. This is intentional. This is specifically not designed for users new to software compilation and toolchain components.* +This guide makes use of *distcc* and *crosstool-NG* in order to not have to build a full ARM development environment on x86. -This is the official cross-compiling method used at Arch Linux ARM. If you plan on building a lot of packages and want to speed up the process, the following guide will turn an x86 Linux computer into an ARM cross-compiler. It's also much easier than most cross-compile setups. +*distcc* allow us to use network clients to spread the load of the compilation to slaves from a master. For example, an ARM (distcc) master will send its compile jobs to *x86_64* (distccd) clients that will use *crosstool-NG* toolchain. The clients don't need to run *archlinux*. -This guide makes use of distcc in order to not have to build a full ARM development environment on x86. As the distcc project website states, "distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed." This is particularly advantageous to us since all that is needed is a working cross-compiler for ARM on a faster machine, while controlling the build from an ARM computer that has all of the current libraries and headers. +As the *distcc* project website states, "distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed." This is particularly advantageous to us since all that is needed is a working cross-compiler for ARM on a faster machine, while controlling the build from an ARM computer that has all of the current libraries and headers. ## Distcc -Before beginning, a distcc environment needs to be set up. Follow the [[Distributed Compiling]] guide to establish a master system. The x86 machines will be known as the clients. +Before beginning, a *distcc* environment needs to be set up. Follow the [[Distributed Compiling]] guide to establish a master system. The x86 machines will be known as the clients. -## Pre-built crosstool-ng toolchains -In lieu of building the toolchain as detailed below, if you are running a 64-bit Linux installation you can use these packaged toolchains that are employed in the official build system. +## Pre-built crosstool-NG toolchains +In lieu of building the toolchain as detailed below, if you are running a *64-bit* Linux installation you can use these packaged toolchains that are employed in the official build system. -It is *highly recommended* to use these tarballs as they have been thoroughly tested, and are maintained to be version- and source-matched to the current toolchain components for our ARM architectures. If you build the toolchain yourself, you must assemble patched source tarballs to build versions that match what is in use here. +It is **highly recommended** to use these tarballs as they have been thoroughly tested, and are maintained to be version- and source-matched to the current toolchain components for our ARM architectures. If you build the toolchain yourself, you must assemble patched source tarballs to build versions that match what is in use here. * [ARMv5te soft-float](/builder/xtools/x-tools.tar.xz) (8b93708e0f7ef971b01b71dfedaffe40) * [ARMv6l hard-float](/builder/xtools/x-tools6h.tar.xz) (8b8c261c91f11545f4b54b0bddaf1db1) * [ARMv7l hard-float](/builder/xtools/x-tools7h.tar.xz) (856aa3e81afda734deff8fb2848d5760) * [ARMv8](/builder/xtools/x-tools8.tar.xz) (56c435708efe81dcf0e668907219c3ed) -## Building a crosstool-ng toolchain -This process is very automated, courtesy of [crosstool-ng](http://crosstool-ng.org). As a normal user (not root!), clone revision ec384f2b of the git repository into a directory called "cross" in your home directory. Enter the source directory and configure with a prefix for the "cross" directory, make, and make install. If you are missing any pre-requisites, the configure script will let you know what they are. +### AUR packages +For easier setup, one can use the relevant packages from [AUR](https://aur.archlinux.org). They just wrap up the above tarball into a package with included systemd services: + +* [ARMv5te soft-float](https://aur.archlinux.org/packages/distccd-alarm-armv5/) +* [ARMv6l hard-float](https://aur.archlinux.org/packages/distccd-alarm-armv6h/) +* [ARMv7l hard-float](https://aur.archlinux.org/packages/distccd-alarm-armv7h/) +* [ARMv8](https://aur.archlinux.org/packages/distccd-alarm-armv8/) + +They provide an easier way to setup the systemd services, and the possibility to run all 4 distccd toolchains in parallel (if needed). + +## Building a crosstool-NG toolchain +This process is very automated, courtesy of [crosstool-NG](http://crosstool-ng.org). As a normal user (**not root!**), clone revision *ec384f2b* of the git repository into a directory called `cross` in your home directory. Enter the source directory and configure with a prefix for the `cross` directory, make, and make install. If you are missing any pre-requisites, the configure script will let you know what they are. ``` mkdir -p cross/src @@ -33,25 +45,29 @@ make make install ``` -At this point crosstool-ng is ready to be configured. The program "ct-ng" in the "bin" directory is where the magic happens. It also has a menu configuration like the Linux kernel. +At this point *crosstool-NG* is ready to be configured. The program `ct-ng` in the `bin` directory is where the magic happens. It also has a menu configuration like the Linux kernel. -## Downloading a CrossTool Configuration -Download the default .config file to place in "~/cross/bin" as shown below. Once you do this, do not run "menuconfig" or values will be overwritten. Choose either v5, v6, v7, or v8 depending on the target platform. +### Downloading a CrossTool Configuration +Download the default .config file to place in `~/cross/bin` as shown below. **Once you do this, do not run "menuconfig" or values will be overwritten**. Choose either v5, v6, v7, or v8 depending on the target platform. ``` cd /home/your_user/cross/bin wget http://archlinuxarm.org/builder/xtools/xtools-dotconfig-[v5|v6|v7|v8] -O .config ``` -## Build the cross-toolchain +### Build the cross-toolchain ``` cd /home/your_user/cross/bin ./ct-ng build ``` -## Make nice with distcc -The toolchain will install under "~/x-tools" for armv5, "~/x-tools6h" for armv6 hard-float, "~/x-tools7h" for armv7 hard-float, and "~/x-tools8" for AArch64. You can move this somewhere else if you like, or leave it where it is. Before we can use the compiler binaries that were created, links need to be created to make their names more appropriate. When compile jobs are sent to distcc, the program specified in the CC environment variable on the build master is what gets executed on the build clients. All the binaries that have been produced by crosstool-ng are in the correct format specifying the target platform as a prefix, though not with the correct platform and lacking tuple-less variants. This script will fix our problem. Make "~/x-tools[6h|7h|8]/arm-unknown-linux-gnueabi[hf]/bin/" writable and in there create a file called "link" and paste the following into it. Uncomment for your target architecture and run it. +### Make nice with distcc +The toolchain will install under `~/x-tools` for *armv5*, `~/x-tools6h` for *armv6* hard-float, `~/x-tools7h` for *armv7* hard-float, and `~/x-tools8` for *AArch64*. You can move this somewhere else if you like, or leave it where it is. + +Before we can use the compiler binaries that were created, links need to be created to make their names more appropriate. When compile jobs are sent to *distcc*, the program specified in the CC environment variable on the build master is what gets executed on the build clients. + +All the binaries that have been produced by *crosstool-ng* are in the correct format specifying the target platform as a prefix, though not with the correct platform and lacking tuple-less variants. This script will fix our problem. Make "~/x-tools[6h|7h|8]/arm-unknown-linux-gnueabi[hf]/bin/" writable and in there create a file called "link" and paste the following into it. Uncomment for your target architecture and run it. ```bash #!/bin/bash @@ -73,7 +89,7 @@ for file in `ls`; do done ``` -Now the "bin" directory contains links with names that distcc will play nice with. +Now the `bin` directory contains links with names that distcc will play nice with. ## Compile! Back on ARM master device, make sure that distcc has been enabled in makepkg.conf per the [[Distributed Compiling]] guide. Now all builds using makepkg will make use of the distcc and cross compiler setup. diff --git a/Distributed_Compiling.md b/Distributed_Compiling.md index a978228..6018f60 100644 --- a/Distributed_Compiling.md +++ b/Distributed_Compiling.md @@ -1,8 +1,8 @@ -If you have spare ARM or x86 systems, using distcc is a very simple way to get all of them in on compiling packages. makepkg is already set up for using distcc, all that's needed is to enable the functionality. +If you have spare *ARM* or *x86* systems, using **distcc** is a very simple way to get all of them in on compiling packages. *makepkg* is already set up for using *distcc*, all that's needed is to enable the functionality. -For distributing to other ARM systems, you will need to have base-devel installed. Each system needs to have the same packages installed that are required for compiling the package you're about to build using this method. See the [Arch Wiki](https://wiki.archlinux.org) for makepkg, pacman, etc. +For distributing to other ARM systems, you will need to have **base-devel** installed. Each system needs to have the same packages installed that are required for compiling the package you're about to build using this method. See the [Arch Wiki](https://wiki.archlinux.org) for makepkg, pacman, etc. -For distributing to x86 systems, install the cross compilers for the needed targets from the [[Distcc Cross-Compiling]] page. +For distributing to x86 systems, install the cross compilers for the needed target architectures from the [[Distcc Cross-Compiling]] page. Install distcc on all systems to be used: @@ -11,7 +11,7 @@ pacman -S distcc ``` ## Configuration: Introduction -For the purposes of this example, all of the hosts are in the local network of 10.3.0.0/24, and access is allowed from that entire subnet. If you want to restrict access further, you will have to specify individual IPs. Be sure to change the address from the examples to fit your local network configuration. +For the purposes of this example, all of the hosts are in the local network of `10.3.0.0/24`, and access is allowed from that entire subnet. If you want to restrict access further, you will have to specify individual IPs. Be sure to change the address from the examples to fit your local network configuration. Additionally, this example assumes that you are starting compiles from one specific system (master), which is sending compiles out to the other system(s) (client). If you want to compile from any system, simply apply the client configuration to the master system, and the master system configuration to the client system(s). @@ -20,25 +20,37 @@ If you want to use the master system to compile as well, apply the client config ## Configuration: Master System These are the configurations for the ARM system you'll be running *makepkg* from. -In /etc/makepkg.conf, or in makepkg.conf in your chroot, change BUILDENV to un-negate the distcc option: +In `/etc/makepkg.conf`, or in `makepkg.conf` in your *chroot*, change BUILDENV to un-negate the distcc option: ``` BUILDENV=(distcc color !ccache check !sign) ``` -Change DISTCC_HOSTS to a space-delimited list of the distcc clients on your network. If you want to use the master system, be sure to put it's IP address in there too. The hosts listed will be prioritized in the order listed. To keep configurations simple, use the LAN IP address for the local system, not "localhost" or "127.0.0.1". +Change `DISTCC_HOSTS` to a space-delimited list of the *distcc* clients on your network. + +If you want to use the master system, be sure to put its IP address in there too. The hosts listed will be prioritized in the order listed. To keep configurations simple, use the LAN IP address for the local system, not "localhost" or "127.0.0.1". ``` DISTCC_HOSTS="10.3.0.6 10.3.0.2" ``` -Additional options can be specified in DISTCC_HOSTS to control how distcc uses the hosts. For example, to randomize which host is used, limit the maximum number of jobs sent to the first host to 4 and the second host to 8, and to specify a different port that distccd is running on: +Additional options can be specified in `DISTCC_HOSTS` to control how distcc uses the hosts. + +- You can specify the port on which the distant distcc runs by appending it after the IP like: ``` -DISTCC_HOSTS="--randomize 10.3.0.6:3932/4 10.3.0.2:3932/8" +DISTCC_HOSTS="10.3.0.6:3634" ``` -Change the MAKEFLAGS -j flag to reflect the total number of processors available on the master system. The common wisdom is to set this to the number of physical cores + 1. Keep in mind that only compiles are distributed; preprocessing and linking still takes place on the master system. Therefore, this number should reflect the capabilities of the master system and not the total number of distributed cores available. +- For example, to randomize which host is used, limit the maximum number of jobs sent to the first host to 4 and the second host to 8, and to specify a different port that distccd is running on: + +``` +DISTCC_HOSTS="--randomize 10.3.0.6/4 10.3.0.2/8" +``` + +Change the MAKEFLAGS -j flag to reflect the total number of processors available on the master system. The common wisdom is to set this to the number of physical cores + 1. +Keep in mind that only compiles are distributed; preprocessing and linking still takes place on the master system. +Therefore, this number should reflect the capabilities of the master system and not the total number of distributed cores available. ``` MAKEFLAGS="-j3" @@ -47,13 +59,23 @@ MAKEFLAGS="-j3" ## Configuration: Client System(s) These are the configurations for a second ARM system or x86 cross-compiling computer. -In /etc/conf.d/distccd, change DISTCC_ARGS to reflect the hosts or network you're allowing to connect and the number of distccd instances (jobs) to start. These arguments will be passed to distccd when it's started, so this is a good place to add any additional arguments you want. +In `/etc/conf.d/distccd`, change `DISTCC_ARGS` to reflect the hosts or network you're allowing to connect and the number of distccd instances (jobs) to start. + +These arguments will be passed to distccd when it's started, so this is a good place to add any additional arguments you want. ``` DISTCC_ARGS="--allow 10.3.0.0/24 --jobs 20" ``` -As of distcc 3.3, protections for [CVE-2004-2687](https://nvd.nist.gov/vuln/detail/CVE-2004-2687) have been implemented. Specifically, the symlinks to distcc in /usr/lib/distcc are a whitelist of what executables are allowed to be run through distcc. For cross-compiling systems, this means that symlinks of each ARM architecture's compiler must be created (c++, cc, cpp, g++, and gcc). For example: +If you have used the AUR package, just edit the relevant conf file like, for example, `/etc/conf.d/distccd-armv6h`. + +### Symlinks + +Note: This part is not needed for AUR package users. + +As of distcc 3.3, protections for [CVE-2004-2687](https://nvd.nist.gov/vuln/detail/CVE-2004-2687) have been implemented. Specifically, the symlinks to distcc in `/usr/lib/distcc` are a whitelist of what executables are allowed to be run through distcc. + +For cross-compiling systems, this means that symlinks of each ARM architecture's compiler must be created (c++, cc, cpp, g++, and gcc). For example: ``` ln -s /usr/bin/distcc /usr/lib/distcc/aarch64-unknown-linux-gnu-gcc @@ -68,8 +90,17 @@ On ARM systems where the installed toolchain will be used, simply enable the shi systemctl enable distccd systemctl start distccd ``` +If you use the AUR packages, *systemd* services have already been installed. Just start (and enable) the needed services. For example: + +``` +systemctl start distccd-armv6h +``` + +#### Manual method -On x86 systems where a cross toolchain will be used, create a new systemd service file for the toolchain. When setting up for cross-compiling multiple ARM architectures, each service file should start distcc with different ports from the default (3632) and specify a unique log file. The new port number must be set in DISTCC_HOSTS in makepkg.conf. This example shows a service file for the AArch64 toolchain in a multiple toolchain environment: +On x86 systems where a cross toolchain will be used, create a new systemd service file for the toolchain. When setting up for cross-compiling multiple ARM architectures, each service file should start distcc with different ports from the default (3632) and specify a unique log file. + +The new port number must be set in DISTCC_HOSTS in makepkg.conf. This example shows a service file for the AArch64 toolchain in a multiple toolchain environment: `/etc/systemd/system/distccd8.service` @@ -100,4 +131,6 @@ systemctl enable distccd8 systemctl start distccd8 ``` +## Run + You can now run *makepkg* as you normally would and the hosts specified will start receiving work. If the hosts are unavailable, the package will simply compile locally.