Skip to content

Commit

Permalink
WIP Drop Premake (shader-slang#3703)
Browse files Browse the repository at this point in the history
* Remove premake lua

* Remove premake generated vs project

* remove deps file

* Remove premake driving bat files

* Full test matrix under CMake

* Remove premake based ci workflows

* Wiggle CI

* remove cmake from ci name

* find frameworks correctly on osx

* remove cmake from ci name

* Cope with sccache not being available

* cmake based falcor tests

* ci wobble

* only install ninja if necessary

* more appropriate cache name

* Remove premake from build instructions

* Add some docs on ci setup

* remove premake from regression tests

* remove premake from perf test

* Set SLANGC_PATH

* ci wobble

* bump slang-binaries

* ci wobble

* Bump spirv tools

* dont use timestamp in cache

* remote debug code

* cache key wobble

* Install sccache after building llvm

* Do not build llvm tools

* ci wobble

* ci wobble

* ci wobble

* ci wobble

* ci wobble

* Tests spirv via glsl in ci

* Define SLANG_ENABLE_XLIB=1

* osx builds on aarch64

* ci wobble

* ci wobble

* ci wobble

* ci wobble

* ci wobble

* ci wobble

* package documentation and metadata with cmake

* ci wobble

* Split hlsl double intrinsic tests

* ci wobble

* Correct type for double log10

Fixes shader-slang#4549

* remove working test from expected failures

* add broken test to expected failures

* smaller build for falcor tests

* ci wobble

* A few exclusions in ci

* wip, release script

* Enable examples in ci

* neaten release script

* Correct building docs

* Only use xlib vulkan when slang_enable_xlib is true

* bump slang-llvm version

* Remove toolchain file use

* Bump slang-llvm preset version

* slash direction

* Improve build directions

* Add msvc cross build documentation

* Disable old release files

* Smaller set of releases for test

* Allow not building llvm

* simplify release matrix

* Cross releases

* formatting

* formatting

* ci wiggle

* ci wiggle

* cleaner

* neaten

* ci wobble

* formatting

* Install cross tools on linux

* do not clean build dir

* neaten ci

* neaten ci

* neaten ci

* remove unused release workflow files

* Build llvm on some platforms

* neaten ci

* notarize on osx

* s/x64/x86_64

* ci wobble

* Embed stdlib for release build

* wobble ci

* wobble ci

* s/x64/x86_64

* ci wobble

* ci wobble

* ci wobble

* vk-gl-cts on cmake

* neaten ci

* neaten ci

* bump cache action version

* Cope with windows being weird about case

* old glibc version

* old glibc version

* Correct action file

* Keep cache hot on main branch

* separate small script for old glibc releases

* ci wobble

* ci wobble

* Run cmake outside of docker

* only sign on releases

* Revert "Run cmake outside of docker"

This reverts commit a58aaba.

* python3 on build image

* less parallel

* ci wobble

* ci wobble

* ci wobble

* newer git

* ci wobble

* ci wobble

* Use newer docker image

* Use newer docker image

* sccache wobble

* permissions issue

* neaten

* build llvm in ci

* build llvm in ci

* Remove linux clang build in ci

* Only install crossbuild tools on non-aarch64 systems

* neaten ci.yml

* Correct github matrix

* Simplify github matrix

* ci wobble

* Disable broken test

See shader-slang#4589

* ci wobble

* Neater slang-llvm archive filename

* Neater path for uploading artifacts

* Neater ci names

* Use Windows SDK 10.0.19041.0 in cmake builds
  • Loading branch information
expipiplus1 authored Jul 10, 2024
1 parent 9f1ff45 commit 0e56999
Show file tree
Hide file tree
Showing 119 changed files with 514 additions and 22,659 deletions.
83 changes: 54 additions & 29 deletions .github/actions/common-setup/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,86 +5,109 @@ description: Performs setup common to all our actions
inputs:
os:
required: true
runs-on:
required: true
compiler:
required: true
platform:
required: true
config:
required: true
build-llvm:
required: true
runs:
using: composite
steps:
- name: Set up MSVC dev tools on Windows
uses: ilammy/msvc-dev-cmd@v1
- name: Set up sccache
uses: hendrikmuhs/[email protected]
with:
key: ${{inputs.os}}-${{inputs.compiler}}-${{inputs.platform}}-${{inputs.config}}
variant: sccache
append-timestamp: false
sdk: "10.0.19041.0"

- shell: bash
run: |
# Set up system dependencies
# Install Ninja
case "${{inputs.os}}" in
linux*) sudo apt-get install -y ninja-build;;
windows*) choco install ninja;;
macos*) brew install ninja;;
esac
if ! command -v ninja; then
case "${{inputs.os}}" in
linux*) sudo apt-get install -y ninja-build;;
windows*) choco install ninja;;
macos*) brew install ninja;;
esac
fi
# Install cross tools on Ubuntu
if [[ "${{inputs.os}}" == linux* && "${{inputs.platform}}" == "aarch64" && "$(uname -m)" != "aarch64" ]]; then
sudo apt-get install -y crossbuild-essential-arm64
fi
# Set compiler
CC=${{inputs.compiler}}
CXX=${{inputs.compiler}}
# infer C++ compiler
CXX=${CXX/gcc/g++}
CXX=${CXX/clang/clang++}
# Correct version on older ubuntu
if [[ "${{inputs.runs-on}}" == ubuntu-20.04 ]]; then
CC=${CC/gcc/gcc-10}
CXX=${CXX/g++/g++-10}
# Correct gcc version on older ubuntu
if [[ "${{inputs.os}}" == linux* ]]; then
gcc_version=$(gcc -dumpversion | cut -d'.' -f1)
if [ "$gcc_version" -lt 10 ]; then
CC=${CC/gcc/gcc-10}
CXX=${CXX/g++/g++-10}
fi
fi
# Export
echo "CC=$CC" >> "$GITHUB_ENV"
echo "CXX=$CXX" >> "$GITHUB_ENV"
# Set CMake to use sccache
echo "CMAKE_CXX_COMPILER_LAUNCHER=sccache" >> "$GITHUB_ENV"
echo "CMAKE_C_COMPILER_LAUNCHER=sccache" >> "$GITHUB_ENV"
# Some useful variables
config=${{inputs.config}}
Config=$(echo "${{inputs.config}}" | sed 's/debug/Debug/;s/release/Release/')
bin_dir=$(pwd)/build/$Config/bin
lib_dir=$(pwd)/build/$Config/lib
echo "config=$config" >> "$GITHUB_ENV"
echo "Config=$Config" >> "$GITHUB_ENV"
echo "bin_dir=$bin_dir" >> "$GITHUB_ENV"
echo "lib_dir=$lib_dir" >> "$GITHUB_ENV"
# Try to restore a LLVM install, and build it otherwise
- uses: actions/cache/restore@v3
- uses: actions/cache/restore@v4
id: cache-llvm
if: inputs.build-llvm == 'true'
with:
path: ${{ github.workspace }}/build/llvm-project-install
# Use os*compiler*platform in lieu of an ABI key here, which is what we really want
key: ${{ inputs.os }}-${{ inputs.compiler }}-${{ inputs.platform }}-${{ hashFiles('external/build-llvm.sh') }}
key: llvm-${{ inputs.os }}-${{ inputs.compiler }}-${{ inputs.platform }}-${{ hashFiles('external/build-llvm.sh') }}
- name: Build LLVM
if: steps.cache-llvm.outputs.cache-hit != 'true'
if: inputs.build-llvm == 'true' && steps.cache-llvm.outputs.cache-hit != 'true'
shell: bash
run: ./external/build-llvm.sh --install-prefix "${{ github.workspace }}/build/llvm-project-install"
- uses: actions/cache/save@v3
if: steps.cache-llvm.outputs.cache-hit != 'true'
- uses: actions/cache/save@v4
if: inputs.build-llvm == 'true' && steps.cache-llvm.outputs.cache-hit != 'true'
with:
path: ${{ github.workspace }}/build/llvm-project-install
key: ${{ steps.cache-llvm.outputs.cache-primary-key }}

# Run this after building llvm, it's pointless to fill the caches with
# infrequent llvm build products
- name: Set up sccache
uses: hendrikmuhs/[email protected]
with:
key: ${{inputs.os}}-${{inputs.compiler}}-${{inputs.platform}}-${{inputs.config}}
variant: sccache
# Opportunistically use sccache, it's not available for example on self
# hosted runners or ARM
continue-on-error: true

- name: Set environment variable for CMake
shell: bash
run: |
echo "LLVM_DIR=${{ github.workspace }}/build/llvm-project-install" >> "$GITHUB_ENV"
echo "Clang_DIR=${{ github.workspace }}/build/llvm-project-install" >> "$GITHUB_ENV"
if [ "${{inputs.build-llvm}}" == "true" ]; then
echo "LLVM_DIR=${{ github.workspace }}/build/llvm-project-install" >> "$GITHUB_ENV"
echo "Clang_DIR=${{ github.workspace }}/build/llvm-project-install" >> "$GITHUB_ENV"
fi
# Set CMake to use sccache if it's available
if command -v sccache; then
echo "CMAKE_CXX_COMPILER_LAUNCHER=sccache" >> "$GITHUB_ENV"
echo "CMAKE_C_COMPILER_LAUNCHER=sccache" >> "$GITHUB_ENV"
fi
# Install swiftshader
- uses: robinraju/[email protected]
Expand All @@ -107,7 +130,9 @@ runs:
# Put spirv-tools in path
- shell: bash
run: |
win_platform="${{ inputs.platform }}"
win_platform="${win_platform//x86_64/x64}"
case "${{inputs.os}}" in
windows*) echo "${{github.workspace}}/external/slang-binaries/spirv-tools/windows-${{inputs.platform}}/bin" >> "$GITHUB_PATH";;
windows*) echo "${{github.workspace}}/external/slang-binaries/spirv-tools/windows-$win_platform/bin" >> "$GITHUB_PATH";;
linux*) echo "${{github.workspace}}/external/slang-binaries/spirv-tools/$(uname -m)-linux/bin" >> "$GITHUB_PATH";;
esac
105 changes: 49 additions & 56 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
name: CI (CMake)
name: CI

on:
push:
Expand All @@ -12,60 +12,54 @@ jobs:
build:
strategy:
matrix:
# Until we switch to CMake fully, just run something minimal
os: [linux, macos, windows]
config: [debug, release]
compiler: [gcc, clang, cl]
platform: [x86_64, aarch64]
exclude:
# Default to x64, but aarch64 on osx
- { os: linux, platform: aarch64 }
- { os: windows, platform: aarch64 }
- { os: macos, platform: x86_64 }
# Unused compiler configs
- { os: linux, compiler: clang }
- { os: linux, compiler: cl }
- { os: windows, compiler: gcc }
- { os: windows, compiler: clang }
- { os: macos, compiler: gcc }
- { os: macos, compiler: cl }
include:
- { os: linux, runs-on: ubuntu-20.04 }
- { os: macos, runs-on: macos-latest }
- { os: windows, runs-on: windows-latest }
# When to have warnings
- warnings-as-errors: false
- { compiler: cl, warnings-as-errors true }
# Set a test category depending on the config, smoke by default,
# quick or full conditionally otherwise
- test-category: smoke
- { os: windows, test-category: quick }
- { config: release, test-category: full }
# default not full gpu tests
- full-gpu-tests: false
# Self-hosted aarch64 build
- os: linux
compiler: gcc
platform: x64
config: debug
warnings-as-errors: true
platform: aarch64
config: release
warnings-as-errors: false
test-category: smoke
full-gpu-tests: false
runs-on: ubuntu-20.04
# os: [linux, macos, windows]
# compiler: ['gcc', 'clang', 'cl']
# config: ['debug', 'release']
# platform: ['x64']
# full-gpu-tests: [false]
# exclude:
# # Exclude invalid or undesired os/compiler
# - { os: linux, compiler: cl }
# - { os: macos, compiler: cl }
# - { os: macos, compiler: gcc }
# - { os: windows, compiler: gcc }
# - { os: windows, compiler: clang }
# # or os/config combinations
# - { os: macos, config: debug }
# include:
# - { os: linux, runs-on: ubuntu-20.04 }
# - { os: windows, runs-on: windows-latest }
# - { os: macos, runs-on: macos-latest }
# # Set a test category depending on the config, smoke by default,
# # quick or full conditionally otherwise
# - test-category: smoke
# - { os: windows, test-category: quick }
# - { config: release, test-category: full }
# # Only CL has werror
# - warnings-as-errors: false
# - { compiler: cl, warnings-as-errors: true }
# # Self-hosted aarch64 build
# - os: linux
# compiler: gcc
# platform: aarch64
# config: release
# warnings-as-errors: false
# test-category: smoke
# full-gpu-tests: false
# runs-on: [self-hosted, Linux, ARM64]
# # Self-hosted full gpu build
# - os: windows
# compiler: cl
# platform: x64
# config: release
# warnings-as-errors: false
# test-category: full
# full-gpu-tests: true
# runs-on: [Windows, self-hosted]
runs-on: [self-hosted, Linux, ARM64]
# Self-hosted full gpu build
- os: windows
compiler: cl
platform: x86_64
config: release
warnings-as-errors: false
test-category: full
full-gpu-tests: true
runs-on: [Windows, self-hosted]
fail-fast: false
runs-on: ${{ matrix.runs-on }}

Expand All @@ -82,10 +76,10 @@ jobs:
uses: ./.github/actions/common-setup
with:
os: ${{matrix.os}}
runs-on: ${{matrix.runs-on}}
compiler: ${{matrix.compiler}}
platform: ${{matrix.platform}}
config: ${{matrix.config}}
build-llvm: true
- name: Build Slang
run: |
if [[ "${{ matrix.os }}" =~ "windows" && "${{ matrix.config }}" != "release" ]]; then
Expand All @@ -96,22 +90,21 @@ jobs:
# Configure, pointing to our just-generated slang-llvm archive
cmake --preset default --fresh \
-DSLANG_SLANG_LLVM_FLAVOR=FETCH_BINARY \
-DSLANG_SLANG_LLVM_BINARY_URL=$(pwd)/build/dist-release/slang-slang-llvm.zip \
-DSLANG_ENABLE_EXAMPLES=OFF \
-DSLANG_SLANG_LLVM_BINARY_URL=$(pwd)/build/dist-release/slang-llvm.zip \
-DCMAKE_COMPILE_WARNING_AS_ERROR=${{matrix.warnings-as-errors}}
cmake --workflow --preset "${{matrix.config}}"
else
# Otherwise, use the system llvm we have just build or got from the
# cache in the setup phase
cmake --preset default --fresh \
-DSLANG_SLANG_LLVM_FLAVOR=USE_SYSTEM_LLVM \
-DSLANG_ENABLE_EXAMPLES=OFF \
-DCMAKE_COMPILE_WARNING_AS_ERROR=${{matrix.warnings-as-errors}}
cmake --workflow --preset "${{matrix.config}}"
fi
- name: Test Slang
run: |
export SLANG_RUN_SPIRV_VALIDATION=1
export SLANG_USE_SPV_SOURCE_LANGUAGE_UNKNOWN=1
failed=0
if [[ "${{matrix.full-gpu-tests}}" == "true" ]]; then
$bin_dir/slang-test \
Expand All @@ -123,7 +116,7 @@ jobs:
$bin_dir/slang-test \
-use-test-server \
-server-count 8 \
-emit-spirv-directly \
-emit-spirv-via-glsl \
-api vk \
-category ${{ matrix.test-category }} \
-expected-failure-list tests/expected-failure.txt ||
Expand All @@ -142,4 +135,4 @@ jobs:
with:
name: slang-build-${{matrix.os}}-${{matrix.platform}}-${{matrix.compiler}}-${{matrix.config}}
# The install directory used in the packaging step
path: build/dist-${{matrix.config}}/**/slang/*
path: build/dist-${{matrix.config}}/**/ZIP/slang/*
53 changes: 31 additions & 22 deletions .github/workflows/compile-regression-test.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
# This is a basic workflow to help you get started with Actions

name: Compile Regression-Test

on:
Expand All @@ -12,37 +10,48 @@ concurrency:
cancel-in-progress: true
jobs:
build:
runs-on: [self-hosted, Windows, regression-test]
timeout-minutes: 100
continue-on-error: true
strategy:
fail-fast: false
matrix:
configuration: ['Release']
platform: ['x64']
os: [windows]
config: [release]
compiler: [cl]
platform: [x86_64]
include:
- platform: x64
testPlatform: x64
- platform: x64
testCategory: full
# Self-hosted falcor tests
- warnings-as-errors: false
test-category: full
full-gpu-tests: false
runs-on: [Windows, self-hosted, regression-test]
runs-on: ${{ matrix.runs-on }}
defaults:
run:
shell: bash
steps:
- uses: actions/checkout@v3
with:
submodules: 'true'
fetch-depth: '0'
- name: setup-msbuild
uses: microsoft/setup-msbuild@v1
- name: build
- name: Setup
uses: ./.github/actions/common-setup
with:
os: ${{matrix.os}}
compiler: ${{matrix.compiler}}
platform: ${{matrix.platform}}
config: ${{matrix.config}}
build-llvm: true
- name: Build Slang
run: |
.\premake.bat vs2019 --arch=${{matrix.platform}} --deps=true --no-progress=true --enable-cuda=true --enable-examples=false
.\make-slang-tag-version.bat
MSBuild.exe slang.sln -v:m -m -property:Configuration=${{matrix.configuration}} -property:Platform=${{matrix.platform}} -property:WindowsTargetPlatformVersion=10.0.19041.0 -maxcpucount:12
cmake --preset default --fresh \
-DSLANG_SLANG_LLVM_FLAVOR=USE_SYSTEM_LLVM \
-DCMAKE_COMPILE_WARNING_AS_ERROR=${{matrix.warnings-as-errors}} \
-DSLANG_ENABLE_CUDA=1
cmake --workflow --preset "${{matrix.config}}"
- name: Run compile and validation test
run: |
$gitbash = 'C:\Program Files\git\bin\bash.exe'
cp -r 'C:\slang_compile_test_suite_a' .\
cd .\slang_compile_test_suite_a
& $gitbash compile_all_slang.sh
cp -r /c/slang_compile_test_suite_a .
cd slang_compile_test_suite_a
export SLANGC_PATH="$bin_dir/slangc.exe"
bash ./compile_all_slang.sh
Loading

0 comments on commit 0e56999

Please sign in to comment.