Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Deno 2 upgrades and project reorganization. #2171

Closed
wants to merge 34 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
9d18828
add further type checks to catching errors, and then immediately ts-i…
rwblair Oct 14, 2024
cbe29a7
bump github workflows to use deno v2
rwblair Oct 14, 2024
0c4088e
do we really need to specify --node-modules-dir=auto for these deno c…
rwblair Oct 14, 2024
ff37b3a
allow-env during build step.
rwblair Oct 14, 2024
15ebfce
why does installing std library require special permissions?
rwblair Oct 14, 2024
8877ba6
build needs to run git, set --allow-run
rwblair Oct 14, 2024
e6aea46
should of just -A'd it eh?
rwblair Oct 14, 2024
0af7d7e
Update .github/workflows/deno_tests.yml
effigies Oct 15, 2024
a174a76
specify default import from ajv in validators/json.ts imports
rwblair Oct 15, 2024
d941cf6
Merge branch 'fix/deno_2_test_and_build' of github.com:rwblair/bids-v…
rwblair Oct 15, 2024
b67e28f
revert ajv import bump version. Deno may be installing from package.j…
rwblair Oct 15, 2024
dde7007
bump ajv in package.json. Will this break legacy?
rwblair Oct 15, 2024
0b88900
Revert "bump ajv in package.json. Will this break legacy?"
rwblair Oct 15, 2024
fd86b31
move legacy validator to its own directory.
rwblair Oct 15, 2024
3ac00ad
working-dir -> working-directory
rwblair Oct 15, 2024
3331f80
more working-directory setting. fix sym link from legacy test to curr…
rwblair Oct 15, 2024
d3258d2
more working-directory setting. fix sym link from legacy test to curr…
rwblair Oct 15, 2024
c9ed50c
in progress README updates
rwblair Oct 15, 2024
064fea7
Merge branch 'master' of github.com:bids-standard/bids-validator into…
rwblair Oct 17, 2024
c87f70c
set working directory for circleci
rwblair Oct 17, 2024
e9e919c
Merge branch 'fix/deno_2_test_and_build' of github.com:rwblair/bids-v…
rwblair Oct 17, 2024
10044b7
deno loader back to version 11
rwblair Oct 17, 2024
efbbc77
amend circle cache paths, they may not respect working-directory
rwblair Oct 17, 2024
bc6afbb
working-directory -> working_directory
rwblair Oct 17, 2024
8f4fd8f
fix call to get local version
rwblair Oct 17, 2024
25c4bb3
remove circleci config
rwblair Oct 17, 2024
0c3f8a8
first pass at deno dockerfile. Update docker github workflow accordin…
rwblair Oct 21, 2024
5f75b1e
Revert "remove circleci config"
rwblair Oct 21, 2024
6923b6f
Move dockerfile to base. Deno cli now handles git not existing.
rwblair Oct 24, 2024
ddb0e25
empty circle jobs array
rwblair Oct 24, 2024
283366d
Merge branch 'master' of github.com:bids-standard/bids-validator into…
rwblair Oct 24, 2024
a77945a
empty out circle workflows entirely
rwblair Oct 24, 2024
40dbd62
dont think working directory directive needed for docker build action…
rwblair Oct 24, 2024
7e40d43
Update .circleci/config.yml
effigies Oct 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
29 changes: 7 additions & 22 deletions .circleci/config.yml
Original file line number Diff line number Diff line change
Expand Up @@ -4,11 +4,12 @@ jobs:
test:
docker:
- image: node:18-alpine
working_directory: ~/project/legacy
steps:
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
- npm-deps-{{ checksum "./package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
- run: apk --no-cache add git
- run: npm install -g npm@^7
- run: npm install
Expand Down Expand Up @@ -37,6 +38,7 @@ jobs:
# Ubuntu 20.04, Docker v20.10.11, Docker Compose v1.29.2
# see: https://circleci.com/docs/2.0/configuration-reference/#available-machine-images
image: ubuntu-2004:202111-02
working_directory: ~/project/legacy
steps:
- checkout
- run: docker build -t $IMAGE_NAME:latest .
Expand All @@ -55,21 +57,22 @@ jobs:
githubPagesTest:
docker:
- image: node:18-alpine
working_directory: ~/project/legacy
steps:
- run: npm install --global npm
- run: apk --no-cache add ca-certificates git openssh-client rsync
- checkout
- restore_cache:
keys:
- npm-deps-{{ checksum "package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
- npm-deps-{{ checksum "./package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
- run:
name: Install bids-validator and bids-validator-web dependencies
command: npm install
- run:
name: Build and export web-validator
command: npm run web-export && rsync -av ./bids-validator-web/out/ ~/web_version
- save_cache:
key: npm-deps-{{ checksum "package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
key: npm-deps-{{ checksum "./package.json" }}-{{ checksum "./bids-validator/package.json" }}-{{ checksum "./bids-validator-web/package.json" }}-v1
paths:
- ./node_modules
- ./.next/cache
Expand All @@ -78,22 +81,4 @@ jobs:
workflows:
effigies marked this conversation as resolved.
Show resolved Hide resolved
version: 2
build-deploy:
jobs:
- test:
filters:
branches:
only: /.*/
tags:
only: /.*/
- test_docker:
filters:
tags:
only: /.*/
- githubPagesTest:
requires:
- test
filters:
branches:
only: /.*/
tags:
only: /.*/
jobs: []
1 change: 0 additions & 1 deletion .dockerignore
Original file line number Diff line number Diff line change
@@ -1,3 +1,2 @@
tests
.git
node_modules
10 changes: 5 additions & 5 deletions .github/workflows/deno_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,10 @@ jobs:
- name: Save describe stamp
id: describe
run: echo version=$( git describe ) >> $GITHUB_OUTPUT
- uses: denoland/setup-deno@v1
- uses: denoland/setup-deno@v2
with:
deno-version: v1.x
- run: ./build.ts
deno-version: v2.x
- run: deno --node-modules-dir=auto -A ./build.ts
working-directory: ./bids-validator
- run: deno run -A ./bids-validator/dist/validator/bids-validator.js --version
- uses: actions/upload-artifact@v4
Expand All @@ -52,14 +52,14 @@ jobs:
submodules: true
- uses: denoland/setup-deno@v1
with:
deno-version: v1.x
deno-version: v2.x
- name: Set permissions with network access
run: echo 'PERMS=--allow-read --allow-write --allow-env --allow-run --allow-net' >> $GITHUB_ENV
if: ${{ matrix.allow-net }}
- name: Set permissions without network access
run: echo 'PERMS=--allow-read --allow-write --allow-env --allow-run --deny-net' >> $GITHUB_ENV
if: ${{ ! matrix.allow-net }}
- run: deno test $PERMS --coverage=cov/ src/
- run: deno test --node-modules-dir=auto $PERMS --coverage=cov/ src/
- name: Collect coverage
run: deno coverage cov/ --lcov --output=coverage.lcov
if: ${{ always() }}
Expand Down
61 changes: 5 additions & 56 deletions .github/workflows/docker-build-push.yml
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
name: Build & publish containers

on:
pull_request:
branches: [master]
push:
tags:
- 'v[0-9]+.[0-9]+.[0-9]+'
Expand Down Expand Up @@ -29,61 +31,8 @@ jobs:
uses: docker/build-push-action@v6
with:
context: .
push: true
push: false
target: base
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

push_singularity_to_registry:
runs-on: ubuntu-latest
needs:
- push_docker_to_registry
steps:
- name: Set up Go 1.13
uses: actions/setup-go@v5
with:
go-version: 1.13
id: go
- name: Install Dependencies
run: |
sudo apt-get update && sudo apt-get install -y \
build-essential \
libssl-dev \
uuid-dev \
libgpgme11-dev \
squashfs-tools \
libseccomp-dev \
pkg-config
- name: Install Singularity
env:
SINGULARITY_VERSION: 3.8.0
GOPATH: /tmp/go
run: |
mkdir -p $GOPATH
sudo mkdir -p /usr/local/var/singularity/mnt && \
mkdir -p $GOPATH/src/github.com/sylabs && \
cd $GOPATH/src/github.com/sylabs && \
wget -qO- https://github.com/sylabs/singularity/releases/download/v${SINGULARITY_VERSION}/singularity-ce-${SINGULARITY_VERSION}.tar.gz | \
tar xzv && \
cd singularity-ce-${SINGULARITY_VERSION} && \
./mconfig -p /usr/local && \
make -C builddir && \
sudo make -C builddir install
- name: Check out code for the container build
uses: actions/checkout@v4
- name: Extract metadata (tags, labels) for Docker
id: meta
uses: docker/metadata-action@v5
with:
images: bids/validator
- name: Build Container
run: |
tags=${{ steps.meta.outputs.tags }}
echo $tags
singularity pull container.sif docker://${tags%,*}
- name: Login and Deploy Container
if: (github.event_name != 'pull_request')
run: |
tags=${{ steps.meta.outputs.tags }}
echo ${{ secrets.GITHUB_TOKEN }} | singularity remote login -u ${{ github.actor }} --password-stdin oras://ghcr.io
for tag in $(echo $tags | sed "s/,/ /g" | sed "s/bids\/validator/bids-standard\/bids-validator/g");
do echo $tag; singularity push container.sif oras://ghcr.io/$tag; done
outputs: type=cacheonly
6 changes: 6 additions & 0 deletions .github/workflows/node_tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -35,20 +35,26 @@ jobs:
git config --global user.email "[email protected]"
git config --global user.name "bids-maintenance"
- name: Set up Node
working-directory: ./legacy
run: |
npm install -g npm@^7
npm install
- name: Eslint
working-directory: ./legacy
run: npm run lint
- name: Get bids-examples data
working-directory: ./legacy
run: git submodule update --init
- name: Jest tests
working-directory: ./legacy
run: npm run coverage -- --maxWorkers=2 --testTimeout=10000 --colors
- name: Upload to codecov
working-directory: ./legacy
uses: codecov/codecov-action@v4
with:
token: ${{ secrets.CODECOV_TOKEN }}
- name: Smoke tests
working-directory: ./legacy
run: |
bids-validator/bin/bids-validator bids-validator/tests/data/valid_headers/ --ignoreNiftiHeaders
bids-validator/bin/bids-validator bids-validator/tests/data/valid_headers/ --ignoreNiftiHeaders --json
4 changes: 2 additions & 2 deletions .github/workflows/schema_web_build.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,9 +17,9 @@ jobs:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: denoland/setup-deno@v1
- uses: denoland/setup-deno@v2
with:
deno-version: v1.x
deno-version: v2.x
- run: deno task build
working-directory: ./web
- uses: actions/upload-artifact@v4
Expand Down
8 changes: 5 additions & 3 deletions .github/workflows/schema_web_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -18,17 +18,19 @@ jobs:
- uses: actions/setup-node@v4
with:
node-version: 18
- uses: denoland/setup-deno@v1
- uses: denoland/setup-deno@v2
with:
deno-version: v1.x
deno-version: v2.x
- run: deno task build
working-directory: ./web
- name: Install NPM deps
run: npm install
working-directory: ./legacy
- name: Build legacy validator website
run: npm run web-export
working-directory: ./legacy
- name: Move legacy validator build into deno website
run: mv bids-validator-web/out web/dist/legacy
run: mv legacy/bids-validator-web/out web/dist/legacy
- name: Upload GitHub Pages artifact
uses: actions/upload-pages-artifact@v3
with:
Expand Down
7 changes: 6 additions & 1 deletion .github/workflows/test-bids-examples.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,16 +37,19 @@ jobs:
run: npm install --global npm

- name: Install bids-validator
working-directory: ./legacy
run: |
npm install
npm link ./bids-validator
echo "./node_modules/.bin" >> $GITHUB_PATH

- name: Get bids-examples data
working-directory: ./legacy
run: |
git clone --depth 1 https://github.com/bids-standard/bids-examples

- name: Display versions and environment information
working-directory: ./legacy
run: |
echo $TZ
export PATH="../node_modules/.bin:$PATH"
Expand All @@ -58,15 +61,17 @@ jobs:
ls -a $(which bids-validator)

- name: Ignore examples not supported by legacy validator
working-directory: ./legacy
run: |
for DS in bids-examples/mrs_*; do
touch $DS/.SKIP_VALIDATION
done

- name: Validate all BIDS datasets using bids-validator
working-directory: ./legacy
run: |
export PATH="../node_modules/.bin:$PATH"
cat ./run_tests.sh
bash ./run_tests.sh
shell: bash
working-directory: ./bids-examples
working-directory: /bids-examples
2 changes: 1 addition & 1 deletion .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ coverage/
__pycache__/
dist/
*.egg-info/
bids-validator-web/out/
legacy/bids-validator-web/out/
.next/

# Only use global yarn.lock
Expand Down
2 changes: 1 addition & 1 deletion .gitmodules
Original file line number Diff line number Diff line change
@@ -1,3 +1,3 @@
[submodule "bids-examples"]
path = bids-validator/tests/data/bids-examples
path = legacy/bids-validator/tests/data/bids-examples
url = https://github.com/bids-standard/bids-examples.git
36 changes: 25 additions & 11 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,16 +1,30 @@
FROM node:18-alpine as build
RUN npm install -g npm

COPY . /src
ARG BASE_IMAGE=denoland/deno:2.0.1
FROM ${BASE_IMAGE} AS build
WORKDIR /src

RUN npm install
RUN npm -w bids-validator run build
RUN npm -w bids-validator pack
RUN apt-get update && \
apt-get install -y git jq && \
apt-get clean && \
rm -rf /var/lib/apt/lists/*

ADD . .
RUN export VERSION=`git -C . -c safe.directory=* describe --tags --always` && \
jq -r ".version|=\"$VERSION\"" bids-validator/deno.json > ._deno.json

FROM node:18-alpine
WORKDIR /src/bids-validator
RUN deno cache ./bids-validator-deno
RUN ./build.ts

COPY --from=build /src/bids-validator-*.tgz /tmp
RUN npm install -g /tmp/bids-validator-*.tgz
FROM ${BASE_IMAGE} AS base
WORKDIR /src
COPY . .
COPY --from=build /src/._deno.json /src/bids-validator/deno.json
WORKDIR /src/bids-validator
RUN deno cache ./bids-validator-deno
ENTRYPOINT ["./bids-validator-deno"]

FROM ${BASE_IMAGE} AS min
WORKDIR /src
COPY --from=build /src/bids-validator/dist/validator/* .

ENTRYPOINT ["/usr/local/bin/bids-validator"]
ENTRYPOINT ["deno", "-A", "./bids-validator.js"]
1 change: 0 additions & 1 deletion README.md

This file was deleted.

Loading