[Test] Add transformers test #1175
+143
−1
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| import torch | ||
|
|
||
| DEVICE_NAME = 'xpu' | ||
|
|
||
| MANUAL_SEED_FN = torch.xpu.manual_seed | ||
| EMPTY_CACHE_FN = torch.xpu.empty_cache | ||
| DEVICE_COUNT_FN = torch.xpu.device_count |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,60 @@ | ||
| transformers_test=${1:-backbone} | ||
| let expected_fail_number=0 | ||
|
|
||
| grep "^FAILED" summary_short.txt | awk '{print $2}' > ./${transformers_test}_failed.log | ||
| grep "PASSED" summary_short.txt | awk '{print $2}' > ./${transformers_test}_passed.log | ||
| grep "SKIPPED" summary_short.txt | awk -F "] " '{print $2}' > ./${transformers_test}_skipped.log | ||
| num_failed=$(cat ./${transformers_test}_failed.log | wc -l) | ||
| num_passed=$(cat ./${transformers_test}_passed.log | wc -l) | ||
| num_skipped=$(cat ./${transformers_test}_skipped.log | wc -l) | ||
| num_errors=$(grep "errors" stats.txt | awk -F " " '{print $10}') | ||
|
|
||
| echo -e "=========================================================================" | ||
| echo -e "Show results in ${transformers_test}" | ||
| echo -e "=========================================================================" | ||
| echo -e "Pass: $num_passed" | ||
| echo -e "Fail: $num_failed" | ||
| echo -e "Skip: $num_skipped" | ||
| echo -e "Error: $num_errors" | ||
| echo -e "=========================================================================" | ||
| printf "%-10s %-4s %-4s %-4s\n" Testgroup Passed Failed Skipped | ||
| printf "%-10s %-4s %-4s %-4s\n" ${transformers_test} $num_passed $num_failed $num_skipped | ||
| echo -e "=========================================================================" | ||
| echo -e "=========================================================================" | ||
|
|
||
| case ${transformers_test} in | ||
| tests_py) | ||
| let expected_fail_number=8 | ||
| ;; | ||
| tests_benchmark) | ||
| let expected_fail_number=0 | ||
| ;; | ||
| tests_generation) | ||
| let expected_fail_number=18 | ||
| ;; | ||
| tests_models) | ||
| let expected_fail_number=407 | ||
| ;; | ||
| tests_pipelines) | ||
| let expected_fail_number=9 | ||
| ;; | ||
| tests_trainer) | ||
| let expected_fail_number=3 | ||
| ;; | ||
| tests_utils) | ||
| let expected_fail_number=1 | ||
| ;; | ||
| backbone) | ||
| let expected_fail_number=0 | ||
| ;; | ||
| tests_trainer_not_ray) | ||
| let expected_fail_number=3 | ||
| ;; | ||
| esac | ||
|
|
||
| if [[ "$num_failed" -gt "$expected_fail_number" ]] || [[ "$num_passed -le 0" ]] || [[ "$num_errors -ne 0" ]]; then | ||
| echo -e "[FAIL] ${transformers_test} test Fail" | ||
| exit 1 | ||
| else | ||
| echo -e "[PASS] ${transformers_test} test Pass" | ||
| fi | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,173 @@ | ||
| name: Linux Transformers Test | ||
|
|
||
| on: | ||
| workflow_call: | ||
|
There was a problem hiding this comment. Add |
||
| inputs: | ||
| pytorch: | ||
| required: false | ||
| type: string | ||
| default: 'nightly' | ||
| description: Pytorch branch/commit | ||
| python: | ||
| required: false | ||
| type: string | ||
| default: '3.10' | ||
| description: Python version | ||
| runner: | ||
| required: true | ||
| type: string | ||
| default: 'linux.idc.xpu' | ||
| description: Runner label | ||
| driver: | ||
| required: false | ||
| type: string | ||
| default: 'lts' | ||
| description: Driver lts/rolling | ||
| nightly_whl: | ||
| required: false | ||
| type: string | ||
| default: '' | ||
| description: Pytorch nightly wheel version | ||
| transformers: | ||
| required: false | ||
| type: string | ||
| default: 'v4.47.0' | ||
| description: Transformers version | ||
| transformers_test: | ||
|
There was a problem hiding this comment. Drop this. Let's focus on a single test - There was a problem hiding this comment. Please, drop |
||
| required: true | ||
| type: string | ||
| default: '' | ||
| description: Test scope. `tests_py,tests_benchmark,tests_generation,tests_models,tests_pipelines,tests_trainer,tests_utils,backbone,tests_trainer_not_ray` Delimiter is comma | ||
|
|
||
| permissions: read-all | ||
|
|
||
| jobs: | ||
| Torch-XPU-Transformers-Tests: | ||
| runs-on: ${{ inputs.runner }} | ||
| env: | ||
| NEOReadDebugKeys: ${{ inputs.driver == 'rolling' && '1' || '0' }} | ||
| DisableScratchPages: ${{ inputs.driver == 'rolling' && '1' || '0' }} | ||
| pytorch: ${{ github.event_name == 'schedule' && 'nightly' || inputs.pytorch }} | ||
|
There was a problem hiding this comment. missing |
||
| steps: | ||
| - name: Checkout torch-xpu-ops | ||
| uses: actions/checkout@v4 | ||
| - name: Prepare Conda ENV | ||
| run: | | ||
| which conda && conda clean -ay | ||
| conda remove --all -y -n huggingface_transformers_test || rm -rf $(dirname ${CONDA_EXE})/../envs/huggingface_transformers_test | ||
| conda create -n huggingface_transformers_test python=${{ env.python }} cmake ninja -y | ||
| source activate huggingface_transformers_test | ||
| sudo apt-get install -y espeak-ng | ||
|
There was a problem hiding this comment. Not a conda setup steps - move to separate step. |
||
| sudo apt-get install -y pkg-config libavformat-dev libavcodec-dev libavdevice-dev libavutil-dev libavfilter-dev libswscale-dev libswresample-dev | ||
| pip install pandas scipy tqdm | ||
| - name: Prepare Transformers | ||
| run: | | ||
| pwd | ||
| source activate huggingface_transformers_test | ||
| cd .. | ||
| git clone https://github.com/huggingface/transformers.git | ||
|
There was a problem hiding this comment. Use |
||
| cd transformers && git checkout $(echo ${{ inputs.transformers }}) | ||
| pip install -e . | ||
| pip install -e ".[dev-torch,testing,video]" | ||
| mkdir -p tests_log | ||
| cp ${{ github.workspace }}/.github/scripts/spec.py ./ | ||
| - name: Prepare Stock Pytorch | ||
| id: installed | ||
| run: | | ||
| pwd | ||
| source activate huggingface_transformers_test | ||
| if [ -z ${{ inputs.nightly_whl }} ]; then | ||
| pip install torch torchvision torchaudio --force --pre --index-url https://download.pytorch.org/whl/nightly/xpu | ||
|
There was a problem hiding this comment. Drop |
||
| else | ||
| pip install torch==$(echo ${{ inputs.nightly_whl }}) torchvision torchaudio --force --pre --index-url https://download.pytorch.org/whl/nightly/xpu | ||
| fi | ||
| echo "TORCH_BRANCH_ID=$(python -c 'import torch; print(torch.__version__)')" |tee -a "${GITHUB_OUTPUT}" >> "${GITHUB_ENV}" | ||
| echo "TORCH_COMMIT_ID=$(python -c 'import torch; print(torch.version.git_version)')" |tee -a "${GITHUB_OUTPUT}" >> "${GITHUB_ENV}" | ||
| pip install pytest-pspec accelerate==1.1.1 timm sentencepiece librosa soundfile | ||
| sudo apt install -y git-lfs | ||
|
There was a problem hiding this comment. Does not belong here - move to OS installation step. |
||
| git lfs install | ||
| pip list >> ${{ github.workspace }}/../transformers/tests_log/pip_list.txt | ||
| cat /sys/class/drm/render*/device/device >> ${{ github.workspace }}/../transformers/tests_log/device_IDs.txt | ||
| - name: Run XPU tests/*.py | ||
| if: contains(input.transformers_test, 'tests_py') || github.event_name == 'schedule' | ||
|
There was a problem hiding this comment. drop condition and all the cases but |
||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_py tests/*.py | tee tests_log/tests_py.log | ||
|
There was a problem hiding this comment. Add |
||
| - name: Run XPU tests/benchmark | ||
| if: contains(input.transformers_test, 'tests_benchmark') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_benchmark tests/benchmark | tee tests_log/tests_benchmark.log | ||
| - name: Run XPU tests/generation | ||
| if: contains(input.transformers_test, 'tests_generation') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_generation tests/generation | tee tests_log/tests_generation.log | ||
| - name: Run XPU tests/pipelines | ||
| if: contains(input.transformers_test, 'tests_pipelines') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_pipelines tests/pipelines | tee tests_log/tests_pipelines.log | ||
| - name: Run XPU tests/trainer | ||
| if: contains(input.transformers_test, 'tests_trainer') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_trainer tests/trainer | tee tests_log/tests_trainer.log | ||
| - name: Run XPU tests/trainer "not ray" | ||
| if: contains(input.transformers_test, 'tests_trainer_not_ray') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_trainer_not_ray -k "not ray" tests/trainer | tee tests_log/tests_trainer_not_ray.log | ||
| - name: Run XPU tests/utils | ||
| if: contains(input.transformers_test, 'tests_utils') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_utils tests/utils | tee tests_log/tests_utils.log | ||
| - name: Run XPU tests/models | ||
| if: contains(input.transformers_test, 'tests_models') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_models tests/models | tee tests_log/tests_models.log | ||
| - name: Run XPU backbone | ||
| if: contains(input.transformers_test, 'backbone') || github.event_name == 'schedule' | ||
| run: | | ||
| source activate huggingface_transformers_test | ||
| cd ../transformers | ||
| TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_utils -k backbone tests | tee tests_log/backbone.log | ||
| - name: Upload Test log | ||
| if: ${{ ! cancelled() }} | ||
| uses: actions/upload-artifact@v4 | ||
| with: | ||
| name: Torch-XPU-Windows-Log-${{ github.event.pull_request.number || github.sha }} | ||
|
There was a problem hiding this comment. Rename file (Windows is wrong :)) |
||
| path: | | ||
| ${{ github.workspace }}/../transformers/reports | ||
| ${{ github.workspace }}/../transformers/tests_log | ||
| - name: Results Check | ||
|
There was a problem hiding this comment. Remove this step for now. |
||
| shell: bash | ||
| run: | | ||
| function contains() { | ||
| contains_status="echo 'Start $2 ...'" | ||
| { | ||
| [[ $1 =~ (^|,)$2($|,) ]] | ||
| } || { | ||
| echo "[Warning] $2 is not suppotted type! Skipped!" | ||
| contains_status="continue" | ||
| } | ||
| } | ||
| set -xe | ||
| for transformers_test in $(echo ${{ input.transformers_test }} |sed 's/,/ /g') | ||
| do | ||
| contains "tests_py,tests_benchmark,tests_generation,tests_models,tests_pipelines,tests_trainer,tests_utils,backbone,tests_trainer_not_ray" $transformers_test | ||
| $contains_status | ||
| cd ${{ github.workspace }}/../transformers/reports/${transformers_test}/ | ||
| cp ${{ github.workspace }}/.github/scripts/transformers_result_check.sh ./ | ||
| bash transformers_result_check.sh ${transformers_test} | ||
| done | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Let's drop this script and associated step for now. Please, save script somewhere since we might need it in the future.