Sync with develop and resolve conflicts

Signed-off-by: Abolfazl Shahbazi <[email protected]>

GetStarted.md

@@ -75,7 +75,7 @@ approaches.
       Clone the repo:
 
       ```
-      git clone https://github.com/IntelAI/transfer-learning.git
+      git clone https://github.com/Intel/transfer-learning.git
       cd transfer-learning
       ```
 

README.md

@@ -1,12 +1,6 @@
-![Style Checks](https://github.com/intel-innersource/frameworks.ai.transfer-learning/actions/workflows/style-test.yaml/badge.svg)
-![Doc Test](https://github.com/intel-innersource/frameworks.ai.transfer-learning/actions/workflows/docs-test.yaml/badge.svg)
-![Unit Tests](https://github.com/intel-innersource/frameworks.ai.transfer-learning/actions/workflows/unittest.yaml/badge.svg)
-![Integration Tests](https://github.com/intel-innersource/frameworks.ai.transfer-learning/actions/workflows/integration.yaml/badge.svg)
-![Notebook Test](https://github.com/intel-innersource/frameworks.ai.transfer-learning/actions/workflows/notebook-test.yaml/badge.svg)
-
 *Note: You may find it easier to read about Intel Transfer Learning tool, follow the Get
 Started guide, and browse the API material from our published documentation site
-https://intelai.github.io/transfer-learning.*
+https://intel.github.io/transfer-learning.*
 
 <!-- SkipBadges -->
 
@@ -109,12 +103,12 @@ command can be found using, for example, `tlt train --help`.
 
 ## Note on Evaluation and Bias
 
-Intel Transfer Learning Tool provides standard evaluation metrics such as accuracy and loss for validation/test/train sets. While important, it's essential to acknowledge that these metrics may not explicitly capture biases. Users should be cautious and consider potential biases by analyzing disparities in the data and model prediction. Techniques such as confusion matrices, PR curves, ROC curves, local attribution-based and `gradCAM` explanations, can all be good indicators for bias. Clear documentation of model behavior and performance is also crucial for iterative bias mitigation. [Intel® Explainable AI Tools](https://github.com/IntelAI/intel-xai-tools/tree/main) provides components that demonstrate the aformentioned techniques with [Explainer](https://github.com/IntelAI/intel-xai-tools/tree/main/explainer), a simple API providing post-hoc model distillation and visualization methods, as well as The [Model Card Generator](https://github.com/IntelAI/intel-xai-tools/tree/main/model_card_gen) which provides an interactive HTML report that containing these workflows and demonstrations of model behavior.
+Intel Transfer Learning Tool provides standard evaluation metrics such as accuracy and loss for validation/test/train sets. While important, it's essential to acknowledge that these metrics may not explicitly capture biases. Users should be cautious and consider potential biases by analyzing disparities in the data and model prediction. Techniques such as confusion matrices, PR curves, ROC curves, local attribution-based and `gradCAM` explanations, can all be good indicators for bias. Clear documentation of model behavior and performance is also crucial for iterative bias mitigation. [Intel® Explainable AI Tools](https://github.com/Intel/intel-xai-tools/tree/main) provides components that demonstrate the aformentioned techniques with [Explainer](https://github.com/Intel/intel-xai-tools/tree/main/explainer), a simple API providing post-hoc model distillation and visualization methods, as well as The [Model Card Generator](https://github.com/Intel/intel-xai-tools/tree/main/model_card_gen) which provides an interactive HTML report that containing these workflows and demonstrations of model behavior.
 
 ## Support
 
 The Intel Transfer Learning Tool team tracks bugs and enhancement requests using
-[GitHub issues](https://github.com/IntelAI/transfer-learning-tool/issues). Before submitting a
+[GitHub issues](https://github.com/Intel/transfer-learning-tool/issues). Before submitting a
 suggestion or bug report, search the existing GitHub issues to see if your issue has already been reported.
 
 See [Legal Information](Legal.md) for Disclaimers, Trademark, and Licensing information.

api.md

@@ -1,4 +1,4 @@
 # API Reference
 
 Low-code Python\* API documentation is automatically generated from the code and
-appears in the Transfer Learning Tool documentation website's [API](https://intelai.github.io/transfer-learning/main/api.html) page.
+appears in the Transfer Learning Tool documentation website's [API](https://intel.github.io/transfer-learning/main/api.html) page.

cli.md

@@ -1,4 +1,4 @@
 # CLI Reference
 
 No-code bash CLI documentation is automatically generated from the code and
-appears in the Transfer Learning Tool documentation website's [CLI](https://intelai.github.io/transfer-learning/main/cli.html) page.
+appears in the Transfer Learning Tool documentation website's [CLI](https://intel.github.io/transfer-learning/main/cli.html) page.

docker/hf_k8s/Dockerfile

@@ -35,30 +35,28 @@ SHELL ["/bin/bash", "-c"]
 
 RUN apt-get update -y && \
     apt-get install -y --no-install-recommends --fix-missing \
-    ca-certificates \
     google-perftools \
-    libgl1 \
     libjemalloc2 \
     libomp-dev \
     numactl
 
 RUN python -m pip install --no-cache-dir \
+    'SentencePiece==0.2.0' \
+    'accelerate==0.28.0' \
+    'datasets==2.19.0' \
+    'einops==0.7.0' \
+    'evaluate==0.4.1' \
     'mkl-include==2023.2.0' \
     'mkl==2023.2.0' \
+    'nltk==3.8.1' \
+    'onnxruntime-extensions==0.10.1' \
+    'onnxruntime==1.17.3' \
+    'peft==0.10.0' \
     'protobuf==4.24.4' \
-    SentencePiece \
-    accelerate==0.28.0 \
-    datasets \
-    einops \
-    evaluate \
-    nltk \
-    onnxruntime \
-    onnxruntime-extensions \
-    peft \
-    psutil \
-    py-cpuinfo \
-    rouge_score \
-    tokenizers
+    'psutil==5.9.5' \
+    'py-cpuinfo==9.0.0' \
+    'rouge_score==0.1.2' \
+    'tokenizers==0.19.1'
 
 # Install OpenSSH for MPI to communicate between containers
 RUN apt-get update && apt-get install -y --no-install-recommends --fix-missing \
@@ -81,11 +79,11 @@ COPY scripts /workspace/scripts/
 COPY generate_ssh_keys.sh /workspace/generate_ssh_keys.sh
 
 RUN cat /workspace/generate_ssh_keys.sh >> ${HOME}/.bash_profile && \
+    sed -i 's#source /inc/bin/activate##g' ${HOME}/.bashrc && \
     cat /workspace/generate_ssh_keys.sh >> ${HOME}/.bashrc
 
 ENV BASH_ENV=${HOME}/.bash_profile
 
 # Set CCL env vars
 ENV PATH="/usr/local/lib/${PYTHON}${PYTHON_VER}/dist-packages/oneccl_bindings_for_pytorch/bin:${PATH}"
 ENV LD_LIBRARY_PATH="/usr/local/lib/${PYTHON}${PYTHON_VER}/dist-packages/oneccl_bindings_for_pytorch/lib:${LD_LIBRARY_PATH}"
-

docker/hf_k8s/Dockerfile.gaudi

@@ -0,0 +1,36 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+ARG GAUDI_SW_VER=1.15.1
+ARG OS=ubuntu22.04
+ARG TORCH_VER=2.2.0
+
+FROM vault.habana.ai/gaudi-docker/${GAUDI_SW_VER}/${OS}/habanalabs/pytorch-installer-${TORCH_VER}:latest
+
+ARG GAUDI_SW_VER=1.15.1
+ARG OPTIMUM_HABANA_VER=1.11.1
+
+RUN pip install --no-cache-dir optimum-habana==${OPTIMUM_HABANA_VER} && \
+    git clone https://github.com/huggingface/optimum-habana.git --single-branch --branch v${OPTIMUM_HABANA_VER} && \
+    pip install --no-cache-dir -r optimum-habana/examples/language-modeling/requirements.txt && \
+    mkdir -p /workspace/optimum-habana && \
+    cp -r optimum-habana/examples /workspace/optimum-habana/examples && \
+    rm -rf optimum-habana && \
+    pip install --no-cache-dir git+https://github.com/HabanaAI/DeepSpeed.git@${GAUDI_SW_VER}
+
+COPY scripts /workspace/scripts
+
+WORKDIR /workspace

docker/hf_k8s/chart/gaudi_multicard_values.yaml

@@ -0,0 +1,108 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+
+metadata:
+  name: llama2-gaudi-multicard
+  namespace: kubeflow
+
+secret:
+  encodedToken:
+
+image:
+  name:  # Specify the image name that was pushed to docker hub or copied to the nodes
+  tag:  # Specify the image tag that was pushed to docker hub or copied to the nodes
+  pullPolicy: Always
+
+securityContext:
+  runAsUser:
+  runAsGroup:
+  fsGroup:
+  privileged: true
+  allowPrivilegeEscalation: false
+
+distributed:
+  script: /workspace/scripts/finetune.py
+  modelNameOrPath: meta-llama/Llama-2-7b-hf
+  logLevel: info
+
+  doTrain: True
+  doEval: True
+
+  train:
+    useMpi: false
+    useDeepSpeed: false
+    datasetName: medalpaca/medical_meadow_medical_flashcards  # Name of the Hugging Face dataset to use. Leave blank if using a data file
+    dataFile:
+    datasetConcatenation: True
+    promptWithInput: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+    promptWithoutInput: Below is an instruction that describes a task. Write a response that appropriately completes the request.
+    perDeviceBatchSize: 12
+    epochs: 3
+    maxSteps: -1
+    maxGradNorm: 0.3
+    gradientAccumulationSteps: 1
+    learningRate: 1e-4
+    lrSchedularType: "constant"
+    useFastTokenizer: False
+    outputDir: /tmp/pvc-mount/output/saved_model
+    loggingSteps: 1020
+    saveTotalLimit: 2
+    evaluationStrategy: "no"
+    saveStrategy: "no"
+    warmupRatio: 0.03
+    throughputWarmupSteps: 3
+    useLora: True
+    useLazyMode: True
+    loraRank: 8
+    loraAlpha: 16
+    loraDropout: 0.05
+    loraTargetModules: q_proj vproj
+    noCuda: True
+    overwriteOutputDir: True
+    adamEpsilon: 1e-08
+    bf16: True
+    useHabana: true
+  eval:
+    perDeviceBatchSize: 8
+    validationSplitPercentage: 0.20
+
+envVars:
+  ldPreload:
+  logLevel: INFO
+  transformersCache: /tmp/pvc-mount/transformers_cache
+  hfDatasetsCache: /tmp/pvc-mount/hf_dataset_cache
+  hfHome: /tmp/home
+  httpProxy:
+  httpsProxy:
+  noProxy:
+  ftpProxy:
+  socksProxy:
+
+# Resources allocated to each worker
+resources:
+  hpu: 8
+  memoryRequest: 409Gi
+  memoryLimit: 409Gi
+  nodeSelectorLabel:
+  nodeSelectorValue:
+  hugePages2Mi: 35202Mi
+
+# Persistent volume claim storage resources
+storage:
+  storageClassName: nfs-client
+  resources: 50Gi
+  pvcMountPath: /tmp/pvc-mount

docker/hf_k8s/chart/gaudi_values.yaml

@@ -0,0 +1,106 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+# SPDX-License-Identifier: Apache-2.0
+
+
+metadata:
+  name: llama2-gaudi-finetuning
+  namespace: kubeflow
+
+secret:
+  encodedToken:
+
+image:
+  name:  # Specify the image name that was pushed to docker hub or copied to the nodes
+  tag:  # Specify the image tag that was pushed to docker hub or copied to the nodes
+  pullPolicy: IfNotPresent
+
+securityContext:
+  runAsUser:
+  runAsGroup:
+  fsGroup:
+  privileged: true
+  allowPrivilegeEscalation: false
+
+distributed:
+  script: /workspace/scripts/finetune.py
+  modelNameOrPath: meta-llama/Llama-2-7b-chat-hf
+  logLevel: info
+
+  doTrain: True
+  doEval: True
+
+  train:
+    datasetName: # Name of the Hugging Face dataset to use. Leave blank if using a data file
+    dataFile: /tmp/pvc-mount/dataset/financial-alpaca.json
+    datasetConcatenation: True
+    promptWithInput: Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request.
+    promptWithoutInput: Below is an instruction that describes a task. Write a response that appropriately completes the request.
+    perDeviceBatchSize: 8
+    epochs: 3
+    maxSteps: -1
+    maxGradNorm: 0.3
+    gradientAccumulationSteps: 1
+    learningRate: 1e-4
+    lrSchedularType: "constant"
+    useFastTokenizer: False
+    outputDir: /tmp/pvc-mount/output/saved_model
+    loggingSteps: 10
+    saveTotalLimit: 2
+    evaluationStrategy: "no"
+    saveStrategy: "no"
+    warmupRatio: 0.03
+    throughputWarmupSteps: 3
+    useLora: True
+    useLazyMode: True
+    loraRank: 8
+    loraAlpha: 16
+    loraDropout: 0.05
+    loraTargetModules: q_proj vproj
+    noCuda: True
+    overwriteOutputDir: True
+    adamEpsilon: 1e-08
+    bf16: True
+    useHabana: true
+  eval:
+    perDeviceBatchSize: 8
+    validationSplitPercentage: 0.20
+
+envVars:
+  ldPreload: /usr/lib/x86_64-linux-gnu/libtcmalloc.so.4.5.9:/usr/local/lib/libiomp5.so
+  logLevel: INFO
+  transformersCache: /tmp/pvc-mount/transformers_cache
+  hfDatasetsCache: /tmp/pvc-mount/hf_dataset_cache
+  hfHome: /tmp/home
+  httpProxy:
+  httpsProxy:
+  noProxy:
+  ftpProxy:
+  socksProxy:
+
+# Resources allocated to each worker
+resources:
+  hpu: 1
+  memoryRequest: 409Gi
+  memoryLimit: 409Gi
+  nodeSelectorLabel:
+  nodeSelectorValue:
+  hugePages2Mi: 35202Mi
+
+# Persistent volume claim storage resources
+storage:
+  storageClassName: nfs-client
+  resources: 50Gi
+  pvcMountPath: /tmp/pvc-mount