Added new Windows win32x86 build instructions and scripts

README-Win32.md

@@ -1,116 +1,75 @@
-#Windows (32bits) Build Setup:
+# Windows (32bits) Build Setup:
 
-##Prerequisites
+## Prerequisites
 
-**NOTE: I'm updating the instructions, it will be more clear next days. [Here](README-Win32-fasttrack.md) is a fast track (the "happy path") for building VM on windows right now.**
+* git
+* Windows 7 SP1 or higher
+* powershell
 
-We provide a set of scripts which automates the build process on Windows platforms. The only thing you need to have in advance is an installation of Git (as of this writing it is nearly impossible to install Git from a script without compiling it).
-There are two options:
-
-- the official [Git client for Windows](http://git-scm.com/download/win)
-- [Git for Windows / msysGit](http://msysgit.github.io)
+### PowerShell
 
-Either choice is fine, just make sure that Git is on your `PATH`. You can test if it is by [opening a (new) cmd shell](http://www.google.com/search?q=windows+open+cmd) and typing `git`. If it's on the `PATH` you'll see the help page. If Git is not yet on your `PATH` you need to [add it](http://www.google.com/search?q=windows+add+PATH).
+PowerShell is automatically installed in all versions of Windows 7 SP1 or higher, the only thing you will need to change execution policy. This is one-time thing if you've never used powershell before.
 
-##Advanced environment setup
-*If you only want to quickly build the VM you can skip this section and continue with [Starting the build] (#starting-the-build)*
+* Find PowerShell in the start menu, right click it and choose Run As Administrator
+* Execute `Set-ExecutionPolicy RemoteSigned` and confirm `Y`
+    * this will enable you to launch local powershell scripts
+* Close the administrator PowerShell
 
-There are a couple more things we require to build the VM:
+## Git
 
-- [MinGW](...) + msys
-- [CMake](http://cmake.org) **IMPORTANT: You has to install [CMake 2.8.8](http://www.cmake.org/files/v2.8/cmake-2.8.8-win32-x86.exe), this is the only version we certify that works (and we know some newer versions are not generating proper scripts). Of course, we will change this in the future :)**
-
-The scripts will download these dependencies if they're not present but they will only be added to the temporary (process) `PATH`. If you intend to build the VM multiple times you'll achieve shorter build times by having these dependencies preinstalled and on your `PATH`.
+We provide a set of scripts which automates the build process on Windows platforms. The only thing you need to have in advance is an installation of Git (as of this writing it is nearly impossible to install Git from a script without compiling it).
+There are two options:
 
-###MinGW + msys installation
-1. Download the [MinGW setup installer] (http://sourceforge.net/projects/mingw/files/Automated%20MinGW%20Installer/mingw-get-inst/) and run it. Make sure to [add the MinGW binaries to your `PATH`](http://www.google.com/search?q=windows+add+PATH).
-2. Install the following additional MinGW packages by running the following command in a shell (cmd shell will work):
-  ```mingw-get install msys-unzip msys-wget msys-zip```
-3. Make sure that the msys paths appear on the `PATH` **before** the Windows default paths. Otherwise you will experience build failures because of name clashes (e.g. the build scripts use the GNU `find` program which clashes with the Windows `find` program)
+- the official [Git client for Windows](http://git-scm.com/download/win)
+- [Git for Windows / msysGit](http://msysgit.github.io)
 
-###CMake installation
-Install [CMake](http://www.cmake.org/): during installation, in install options, make sure that you choose to [add CMake to `PATH`](http://www.google.com/search?q=windows+add+PATH&btnI).
+Either choice is fine, just make sure that Git is on your `PATH`. You can test if it is by [opening a (new) PowerShell or cmd shell](http://www.google.com/search?q=windows+open+cmd) and typing `git`. If it's on the `PATH` you'll see the help page. If Git is not yet on your `PATH` you need to [add it](http://www.google.com/search?q=windows+add+PATH).
 
+### Long paths
 
-To check if everything is installed, open a shell (again, cmd will work) execute the different commands: `git`, `make` and `cmake`.
+Windows (by default) has path names limited to 260 or so characters, so you have to enable long paths in git.
 
+Execute the following (in PowerShell or cmd)
 
-###Notes:
-Should you need a copy of crtdll32.dll, it lives in C:\Windows\SysWOW64 on 64-bit system. The build process fails to find it on such machines.
+`git config --system core.longpaths true`
 
+**NOTE:** *My system requires privilege escalation for this (it's modifying system configuration), so I had to execute it from Admin PowerShell (see section #PowerShell)*
 
-If you need to find out where the msys paths are, you can add the following to your `.profile` and start an msys shell:
-```
-winpath() {
-    if [ -z "$1" ]; then
-        echo "$@"
-    else
-        if [ -f "$1" ]; then
-            local dir=$(dirname "$1")
-            local fn=$(basename "$1")
-            echo "$(cd "$dir"; echo "$(pwd -W)/$fn")" | sed 's|/|\\|g';
-        else
-            if [ -d "$1" ]; then
-                echo "$(cd "$1"; pwd -W)" | sed 's|/|\\|g';
-            else
-                echo "$1" | sed 's|^/\(.\)/|\1:\\|g; s|/|\\|g';
-            fi
-        fi
-    fi
-}
-```
 
+## Installing Cygwin
 
-##Starting the build
-**Note:**
-While the setup script is actually part of this repository, we recommend that you follow the steps below and don't clone the repository yourself (see [Documentation of script operations](#documentation-of-script-operations)).
+* Open a non-admin window of PowerShell and navigate to the _root_ of the cloned repo (`cd`/`ls` commands, just like in bash)
+* Execute `.\scripts\windows\install-cygwin.ps1`
+    * this will download cygwin and install/upgrade all necessary packages
 
-1. [Download the build setup script](scripts/windows/setup.cmd)
-2. Either double click on the script (the shell window will close immediately after the build) or launch the script from a shell. In either case output will be logged to `<currentDirectory>/vm-build/build-setup.log` and `<currentDirectory>/vm-build/build.log` (which you can `tail` if you want).
-3. Wait for the build to finish
-4. The built objects reside in `<currentDirectory>/vm-build/pharo-vm/results`
+## Starting The Build
 
+**NOTE:** *Execute the scripts from the repo's root folder.*
 
+* To perform a complete installation, execute `.\scripts\windows\builder.ps1 -All`
 
-Documentation of script operations
-====================================
+Or perform it step by step (typing `.\scripts\windows\builder.ps1 -<TAB>` will offer you available options)
 
-##General
-1. create a build directory
-2. download MinGW and msys, add them to `PATH` and install packages
-3. download CMake and add it to `PATH`
-4. clone pharo-vm repository (see below)
-5. fix build environment (see below)
-6. run the actual build script
+* `.\scripts\windows\builder.ps1 -BuildSources`
+* `.\scripts\windows\builder.ps1 -BuildVm`
+* `.\scripts\windows\builder.ps1 -PackVm`
 
-##Modify repository configuration before checkout
-1. clone the repository with the `--no-checkout` option
-2. set the `text` configuration option to `auto`: `git config --add core.text auto`
-3. `git checkout -f HEAD`
+See also `Manual Compilation` below.
 
-##Modify float.h
-For GCC 4.6.1+ float.h (`<mingw_directory>/lib/gcc/<version>/include/float.h`) add this to the end of the file:
-```
-#ifndef _MINGW_FLOAT_H_
-#include_next <float.h>
-#endif
-```
+The first run of the complete build will take approximately **40 minutes**, but third party libraries are cached, so it will be much much faster next time.
 
-## Add 32-bit dependencies
-For 64-bit machines (e.g. Windows 8.1 Pro) we need to add `libcrtdll.dll` (which is obsolete on those platforms).
+### Theoretical Speedup
 
-1. `cp <pharovmDirectory>/platforms/win32/extras/libcrtdll.a <mingwDirectory>/lib`
-2. The _mingw.h header has to be amended to deal with changes in some typedefs. Add the following to the end of _mingw.h:
+During compilation system's prefetcher and Windows Defender were quite busy (taking about 1GB RAM each and a whole CPU core). This could possibly slow down the build, although I have no metrics to back up this claim. In either case, you can disable Prefetcher in Windows Services, and add exception to Windows Defender the cloned repo or disable it during the build (at your own risk).
 
-```
-#define off64_t _off64_t
-#define off_t _off_t
-```
+## Manual Compilation
 
-## Configure Git for long paths
-In order to get rid of the GeniePlugin message about path being too long, make sure your version of Git is above 1.8.x (current version installed on 2015-09-29 is 2.5.3).
+You can also switch to cygwin's bash and run the compilation by hand as follows:
 
-```
-git config core.text auto
-git config --system core.longpaths true
-```
+* In PowerShell (in repository's root) execute `. .\scripts\windows\config.ps1`
+    * this will load necessary environment and modify PATH (for the shell only, not the system)
+* Execute `bash`
+* Inside bash execute `. scripts/windows/config.sh`
+    * same as for PowerShell, just make sure you are sourcing file with the correct extension (`ps1` vs `sh`)
+* Now you can e.g. go to `cd opensmalltalk-vm/build.win32x86/pharo.cog.spur`
+* And build VM with asserts `./mvm -a`

scripts/windows/build_vm.sh

@@ -0,0 +1,7 @@
+#!/usr/bin/env bash
+set -eu
+
+. scripts/windows/config.sh
+
+cd opensmalltalk-vm
+./.travis_build.sh

scripts/windows/builder.ps1

@@ -0,0 +1,30 @@
+param (
+    [switch]$All,
+    [switch]$BuildSources,
+    [switch]$BuildVm,
+    [switch]$PackVm
+)
+
+. $PSScriptRoot\config.ps1
+
+function Run-Bash($Command) {
+    Invoke-Expression "bash -c ""$Command"""
+}
+
+
+
+if ($All) {
+    $BuildSources = $BuildVm = $PackVm = $true
+}
+
+if ($BuildSources) {
+    Run-Bash -Command "cd scripts; ./build-sources.sh -a $SRC_ARCH"
+}
+
+if ($BuildVm) {
+    Run-Bash -Command "scripts/windows/build_vm.sh"
+}
+
+if ($PackVm) {
+    Run-Bash -Command "cd scripts; export ARCH=$ARCH; export SRC_ARCH=$SRC_ARCH; ./pack-vm.sh"
+}

scripts/windows/config.ps1

@@ -0,0 +1,13 @@
+# Pick a mirror that is close to you https://cygwin.com/mirrors.html
+# US mirror (used by appveyor)
+#$CYG_MIRROR = "http://cygwin.mirror.constant.com"
+$CYG_MIRROR = "http://ftp.inf.tu-dresden.de/software/windows/cygwin32/"
+
+$FLAVOR = "pharo.cog.spur"
+$CYG_ROOT = "C:\cygwin"
+$CYG_SETUP = "setup-x86.exe"
+$ARCH = "win32x86"
+$MINGW_ARCH = "i686"
+$SRC_ARCH = "i386"
+
+$env:Path = "$CYG_ROOT\bin;$env:Path"

scripts/windows/config.sh

@@ -0,0 +1,5 @@
+export FLAVOR=pharo.cog.spur
+export ARCH=win32x86
+export MINGW_ARCH=i686
+export APPVEYOR=1
+export THIRDPARTYCACHEDIR=$(readlink -f ./opensmalltalk-vm/.thirdparty-cache)

scripts/windows/install-cygwin.ps1

@@ -0,0 +1,37 @@
+. $PSScriptRoot\config.ps1
+
+function Start-FileDownload($Url, $FileName)
+{
+    Invoke-WebRequest -Uri $Url -OutFile $FileName
+}
+
+function Install-Packages($List)
+{
+    # https://cygwin.com/faq/faq.html#faq.setup.cli
+    # Command Line Options:
+    # -d --no-desktop                   Disable creation of desktop shortcut
+    # -g --upgrade-also                 also upgrade installed packages
+    # -n --no-shortcuts                 Disable creation of desktop and start menu shortcuts
+    # -q --quiet-mode                   Unattended setup mode
+    # -N --no-startmenu                 Disable creation of start menu shortcut
+    # -O --only-site                    Ignore all sites except for -s
+    # -B --no-admin                     Do not check for and enforce running as Administrator
+
+    # -R --root                         Root installation directory
+    # -s --site                         Download site
+    # -l --local-package-dir            Local package directory
+    # -P --packages                     Specify packages to install
+
+    Start-Process -Wait -FilePath .\$CYG_SETUP -ArgumentList "-dgnqNO -R $CYG_ROOT -s $CYG_MIRROR -l $CYG_ROOT\var\cache\setup -P $List"
+}
+
+if (!(Test-Path $CYG_SETUP)) {
+    Start-FileDownload "https://cygwin.com/setup-x86.exe" -FileName $CYG_SETUP
+}
+
+# Presumably pre-install packages for appveyor
+Install-Packages autoconf,automake,bison,gcc-core,gcc-g++,mingw-runtime,mingw-binutils,mingw-gcc-core,mingw-gcc-g++,mingw-pthreads,mingw-w32api,libtool,make,python,gettext-devel,gettext,intltool,libiconv,pkg-config,git,wget,curl
+# From pharo-vm's appveyor list
+Install-Packages mingw64-$MINGW_ARCH-gcc-core,mingw64-$MINGW_ARCH-gcc-g++,mingw64-$MINGW_ARCH-headers,mingw64-$MINGW_ARCH-runtime,zip,mingw64-$MINGW_ARCH-clang,libiconv-devel,libglib2.0-devel,perl,mingw64-$MINGW_ARCH-zlib,cmake,mingw64-$MINGW_ARCH-win-iconv
+# zip != unzip; we need both
+Install-Packages wget,unzip