[Test] Add transformers test #1175

RUIJIEZHONG66166 · 2024-12-17T13:24:31Z

For validating the upstream of transformers, we plan to add regular test for tracking the status of transformers.

Signed-off-by: Dmitry Rogozhkin <[email protected]>

These logs will be available on the github actions page. Signed-off-by: Dmitry Rogozhkin <[email protected]>

-k "not ray" is a workoaround for the hang running tests/trainer. No need a separate testing. Signed-off-by: Dmitry Rogozhkin <[email protected]>

Signed-off-by: Dmitry Rogozhkin <[email protected]>

dvrogozh

@RUIJIEZHONG66166 : Please, see comments. However, I suggest you can just merge in #1177 with all the changes I have suggested. (1177 merges into your dev. branch, not main)

dvrogozh · 2024-12-17T23:29:21Z

.github/scripts/transformers_result_check.sh

@@ -0,0 +1,60 @@
+transformers_test=${1:-backbone}


Let's drop this script and associated step for now. Please, save script somewhere since we might need it in the future.

dvrogozh · 2024-12-17T23:29:52Z

.github/workflows/_linux_transformers.yml

+name: Linux Transformers Test
+
+on:
+  workflow_call:


Add on: pull_request. We need verify workflow before merging.

dvrogozh · 2024-12-17T23:30:09Z

.github/workflows/_linux_transformers.yml

+        type: string
+        default: 'v4.47.0'
+        description: Transformers version
+      transformers_test:


Drop this. Let's focus on a single test - backbone.

Please, drop transformers_test. We will most likely rework how to setup test scope.

dvrogozh · 2024-12-17T23:30:25Z

.github/workflows/_linux_transformers.yml

+    env:
+      NEOReadDebugKeys: ${{ inputs.driver == 'rolling' && '1' || '0' }}
+      DisableScratchPages: ${{ inputs.driver == 'rolling' && '1' || '0' }}
+      pytorch: ${{ github.event_name == 'schedule' && 'nightly' || inputs.pytorch }}


missing env.python which is used below

dvrogozh · 2024-12-17T23:30:48Z

.github/workflows/_linux_transformers.yml

+          conda remove --all -y -n huggingface_transformers_test || rm -rf $(dirname ${CONDA_EXE})/../envs/huggingface_transformers_test
+          conda create -n huggingface_transformers_test python=${{ env.python }} cmake ninja -y
+          source activate huggingface_transformers_test
+          sudo apt-get install -y espeak-ng


Not a conda setup steps - move to separate step.

dvrogozh · 2024-12-17T23:33:27Z

.github/workflows/_linux_transformers.yml

+        run: |
+          source activate huggingface_transformers_test
+          cd ../transformers
+          TRANSFORMERS_TEST_DEVICE_SPEC=spec.py python3 -m pytest -rsf --make-reports=tests_py tests/*.py | tee tests_log/tests_py.log


Add TRANSFORMERS_TEST_DEVICE_SPEC to env.
Drop tee.

dvrogozh · 2024-12-17T23:34:03Z

.github/workflows/_linux_transformers.yml

+        if: ${{ ! cancelled() }}
+        uses: actions/upload-artifact@v4
+        with:
+          name: Torch-XPU-Windows-Log-${{ github.event.pull_request.number || github.sha }}


Rename file (Windows is wrong :))

dvrogozh · 2024-12-17T23:34:11Z

.github/workflows/_linux_transformers.yml

+          path: | 
+            ${{ github.workspace }}/../transformers/reports
+            ${{ github.workspace }}/../transformers/tests_log
+      - name: Results Check


Remove this step for now.

dvrogozh · 2024-12-17T23:34:25Z

.github/workflows/nightly_ondemand_whl.yml

+        type: string
+        default: ''
+        description: Pytorch nightly wheel version
+      transformers_test:


remove this.

dvrogozh · 2024-12-17T23:34:33Z

.github/workflows/nightly_ondemand_whl.yml

+    if: github.event_name == 'schedule' && github.event.schedule == '0 14 * * 0-4' || ${{ inputs.transformers_test }}
+    uses: ./.github/workflows/_linux_transformers.yml
+    with:
+      transformers_test: ${{ github.event_name == 'schedule' && 'tests_py,tests_benchmark,tests_generation,tests_models,tests_pipelines,tests_trainer,tests_utils,backbone,tests_trainer_not_ray' || inputs.transformers_test }}


remove this.

Signed-off-by: Dmitry Rogozhkin <[email protected]>

This PR adds a number of fixes for transformers tests: - [x] Drop all tests but one (backbone) - [x] Drop reporting - [x] Add on PR trigger (limited to modifications to workflow file) - [x] Use actions/checkout to clone Transformers - [x] Combine steps to install environment - [x] Install XPU PyTorch first - [x] Few other adjustments

dvrogozh · 2024-12-19T03:31:41Z

.github/workflows/_linux_transformers.yml

+        type: string
+        default: 'v4.47.0'
+        description: Transformers version
+      transformers_test:


Please, drop transformers_test. We will most likely rework how to setup test scope.

dvrogozh · 2024-12-19T03:33:23Z

.github/workflows/_linux_transformers.yml

+      # On-demand launch
+      - name: OnDemand Test (${transformers_test})
+        if: github.event_name == 'workflow_dispatch'
+        run: |
+          source activate huggingface_transformers_test
+          cd transformers
+
+          # check param
+          function contains() {
+              contains_status="echo 'Start $2 ...'"
+              {
+                [[ $1 =~ (^|,)$2($|,) ]]
+              } || {
+                echo "[Warning] $2 is not suppotted type! Skipped!"
+                contains_status="continue"
+              }
+          }
+          set -xe
+          for transformers_test in $(echo ${{ input.transformers_test }} |sed 's/,/ /g')
+          do
+            contains "test_py,benchmark,generation,models,pipelines,trainer,utils,backbone,trainer_not_ray" $transformers_test
+            $contains_status
+            if [ "${transformers_test}" == "test_py" ];then
+              python3 -m pytest -rsf --make-reports=tests_py tests/*.py
+            elif [ "${transformers_test}" == "trainer_not_ray" ];then
+              python3 -m pytest -rsf --make-reports=tests_trainer_not_ray -k "not ray" tests/trainer
+            elif [ "${transformers_test}" == "backbone" ];then
+              python3 -m pytest -rsf --make-reports=tests_backbone -k backbone tests
+            else
+              python3 -m pytest -rsf --make-reports=tests_${transformers_test} tests/${transformers_test}
+            fi
+          done


Drop this. As we agreed - let's do any additional tests on a follow up commit.

Basically limit this change to just s/workflow_call/workflow_dispatch.

dvrogozh · 2024-12-19T03:35:09Z

.github/workflows/_linux_transformers.yml

@@ -39,6 +39,8 @@ on:
        type: string
        default: 'v4.47.0'
        description: Transformers version
+  workflow_dispatch:


This won't work. You will need to duplicate all inputs from workflow_call

Never mind. Just check. I have renamed workflow_call.

dvrogozh

Looks good. Let's merge!

RUIJIEZHONG66166 and others added 9 commits December 17, 2024 05:20

Add transformers est

3d59000

Add step to install OS environment

9c0142e

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Use actions/cheсkout to download transformers

1e4d7df

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Add env variable for python version

067cdd9

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Add dedicated versions reporting step

2d22174

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Add TRANSFORMERS_TEST_DEVICE_SPEC to workflow env vars

59db2bb

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Do not save logs from pytest to file

10bc574

These logs will be available on the github actions page. Signed-off-by: Dmitry Rogozhkin <[email protected]>

Drop tests_trainer_not_ray

91301cf

-k "not ray" is a workoaround for the hang running tests/trainer. No need a separate testing. Signed-off-by: Dmitry Rogozhkin <[email protected]>

Drop all transformer tests but one

dfcff7b

Signed-off-by: Dmitry Rogozhkin <[email protected]>

dvrogozh suggested changes Dec 17, 2024

View reviewed changes

dvrogozh and others added 9 commits December 18, 2024 18:01

Add PR trigger

87de5fa

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Install XPU PyTorch first

8863a94

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Add sanity checks of torch version

a7979df

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Rename log file

aef2f47

Signed-off-by: Dmitry Rogozhkin <[email protected]>

Drop not needed packages

6e01ae8

Signed-off-by: Dmitry Rogozhkin <[email protected]>

move ondemand test to transformer workflow

fdc7741

update workflow_dispatch set

2fb9b1d

solve input type issue

298b09c

dvrogozh suggested changes Dec 19, 2024

View reviewed changes

update workflow_call to worfkflow_dispatch

dec58de

dvrogozh approved these changes Dec 19, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Test] Add transformers test #1175

[Test] Add transformers test #1175

RUIJIEZHONG66166 commented Dec 17, 2024

dvrogozh left a comment

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 19, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 17, 2024

dvrogozh Dec 19, 2024

dvrogozh Dec 19, 2024

dvrogozh Dec 19, 2024

RUIJIEZHONG66166 Dec 19, 2024

dvrogozh left a comment

[Test] Add transformers test #1175

Are you sure you want to change the base?

[Test] Add transformers test #1175

Conversation

RUIJIEZHONG66166 commented Dec 17, 2024

dvrogozh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dvrogozh left a comment

Choose a reason for hiding this comment