Use universal downloader for bert-base (#780)

mlcommons · Jun 1, 2023 · 66b4306 · 66b4306
2 parents 6805473 + 61bad16
commit 66b4306
Show file tree

Hide file tree

Showing 36 changed files with 155 additions and 118 deletions.
diff --git a/cm-mlops/automation/list_of_scripts.md b/cm-mlops/automation/list_of_scripts.md
@@ -0,0 +1,37 @@
+[ [Back to index](README.md) ]
+
+<!--
+This file is generated automatically - don't edit!
+-->
+
+This is an automatically generated list of reusable CM scripts being developed
+by the [open taskforce on automation and reproducibility](https://github.com/mlcommons/ck/issues/536) 
+to make MLOps and DevOps tools more interoperable, portable, deterministic and reproducible.
+These scripts suppport the community effort to modularize ML Systems and automate their bechmarking, optimization,
+design space exploration and deployment across continuously changing software and hardware. 
+
+# List of CM scripts by categories
+
+<details>
+<summary>Click here to see the table of contents.</summary>
+
+* [Platform information](#platform-information)
+
+
+</details>
+
+### Platform information
+
+* [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
+
+
+# List of all sorted CM scripts 
+
+* [detect-os](https://github.com/mlcommons/ck/tree/master/cm-mlops/script/detect-os)
+
+
+
+
+# Maintainers
+
+* [Open MLCommons taskforce on automation and reproducibility](https://github.com/mlcommons/ck/blob/master/docs/taskforce.md)'
diff --git a/cm-mlops/automation/script/README-extra.md b/cm-mlops/automation/script/README-extra.md
@@ -708,12 +708,12 @@ as shown in the next example.
 Instead of adding this flag to all scripts, you can specify it 
 using `CM_SCRIPT_EXTRA_CMD` environment variable as follows:
 ```bash
-export CM_SCRIPT_EXTRA_CMD="--adr.python.name.mlperf"
+export CM_SCRIPT_EXTRA_CMD="--adr.python.name=mlperf"
 ```
 
 You can even specify min Python version required as follows:
 ```bash
-export CM_SCRIPT_EXTRA_CMD="--adr.python.name.mlperf --adr.python.version_min=3.9"
+export CM_SCRIPT_EXTRA_CMD="--adr.python.name=mlperf --adr.python.version_min=3.9"
 ```
 
 ### Assembling pipelines with other artifacts included

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-scc2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-scc2023/README.md
@@ -4,7 +4,7 @@ Under preparation: Reproduce and optimize MLPerf inference benchmarks during Stu
 
 See our [related challange from 2022](https://access.cknowledge.org/playground/?action=challenges&name=optimize-mlperf-inference-scc2023).
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cKnowledge](https://cKnowledge.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v2.1-2022/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v2.1-2022/README.md
@@ -3,7 +3,7 @@
 Prepare, optimize and reproduce MLPerf inference v2.1 benchmarks across diverse implementations, software and hardware
 using the [MLCommons CK framework](https://github.com/mlcommons/ck).
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.0-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.0-2023/README.md
@@ -9,7 +9,7 @@ using the [MLCommons CK framework](https://github.com/mlcommons/ck):
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/README.md
@@ -3,9 +3,10 @@
 Prepare, optimize and reproduce MLPerf inference v3.1 benchmarks across diverse implementations, models, software and hardware.
 
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
-how to use and enhance CK to run and optimize MLPerf inference benchmarks.
+how to use and enhance [CM scripts and workflows](https://github.com/mlcommons/ck/blob/master/docs/README.md) 
+to run and optimize MLPerf inference benchmarks on your software/hardware stack.
 
-## Organizers
+#### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)
@@ -22,6 +23,9 @@ For MLPerf inference 3.1 we have the following benchmark tasks
 6. Recommendation using DLRM model and Criteo dataset
 7. Large Language Model (Pending)
 
-All the six tasks are applicable to datacenter while all except Recommendation are applicable to edge category. Further, language processing and medical imaging models have a high accuracy variant where the achieved accuracy must be within `99.9%` (`99%` is the default accuracy requirement) of the fp32 reference model. Recommendation task is only having a high accuracy variant. Currently we are not supporting Recommendation task as we are not having a highend server which is a requirement.
+All the six tasks are applicable to datacenter while all except Recommendation are applicable to edge category. 
+Further, language processing and medical imaging models have a high accuracy variant where the achieved accuracy 
+must be within `99.9%` (`99%` is the default accuracy requirement) of the fp32 reference model. 
+Recommendation task is only having a high accuracy variant. Currently we are not supporting Recommendation task as we are not having a highend server which is a requirement.
 
 This challenge is integrated with [our platform](https://github.com/ctuning/mlcommons-ck/tree/master/platform)
diff --git a/...allenge/optimize-mlperf-inference-v3.1-2023/docs/generate-3d-unet-submission.md b/...allenge/optimize-mlperf-inference-v3.1-2023/docs/generate-3d-unet-submission.md
@@ -1,11 +1,19 @@
 ## Setup
-Please follow the MLCommons CK [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) to install CM.
-Download the ck repo to get the CM script for MLPerf submission
+
+Please follow this [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) 
+to install the MLCommons CM reproducibility and automation language in your native environment or Docker container.
+
+Then install the repository with CM automation scripts to run MLPerf benchmarks out-of-the-box
+across different software, hardware, models and data sets:
+
 
 ```
 cm pull repo mlcommons@ck
 ```
 
+Note that you can install Python virtual environment via CM to avoid contaminating 
+your local Python installation as described [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
 ## Run Commands
 
 3d-unet has two variants - `3d-unet-99` and `3d-unet-99.9` where the `99` and `99.9` specifies the required accuracy constraint with respect to the reference floating point model. Both models can be submitter under edge as well as datacenter category.

diff --git a/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/generate-bert-submission.md b/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/generate-bert-submission.md
@@ -1,15 +1,19 @@
 ## Setup
+
 Please follow this [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) 
-to install the [MLCommons CM scripting language](https://github.com/mlcommons/ck/tree/master/docs#collective-mind-language-cm)
-on your system with minimal dependencies.
+to install the MLCommons CM reproducibility and automation language in your native environment or Docker container.
+
+Then install the repository with CM automation scripts to run MLPerf benchmarks out-of-the-box
+across different software, hardware, models and data sets:
 
-Download a GitHub repository with [portable and reusable CM scripts](https://github.com/mlcommons/ck/tree/master/cm-mlops/script) 
-for unified MLPerf benchmarking and submission:
 
 ```
 cm pull repo mlcommons@ck
 ```
 
+Note that you can install Python virtual environment via CM to avoid contaminating 
+your local Python installation as described [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
 ## Run Commands
 
 Bert has two variants - `bert-99` and `bert-99.9` where the `99` and `99.9` specifies the required accuracy constraint with respect to the reference floating point model. `bert-99.9` model is applicable only on a datacenter system.

diff --git a/...llenge/optimize-mlperf-inference-v3.1-2023/docs/generate-resnet50-submission.md b/...llenge/optimize-mlperf-inference-v3.1-2023/docs/generate-resnet50-submission.md
@@ -1,11 +1,19 @@
 ## Setup
-Please follow the MLCommons CK [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) to install CM.
-Download the ck repo to get the CM script for MLPerf submission
+
+Please follow this [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) 
+to install the MLCommons CM reproducibility and automation language in your native environment or Docker container.
+
+Then install the repository with CM automation scripts to run MLPerf benchmarks out-of-the-box
+across different software, hardware, models and data sets:
+
 
 ```
 cm pull repo mlcommons@ck
 ```
 
+Note that you can install Python virtual environment via CM to avoid contaminating 
+your local Python installation as described [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
 ## Run Commands
 
 We need to get imagenet full dataset to make image-classification submissions for MLPerf inference. Since this dataset is not publicly available via a URL please follow the instructions given [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/script/get-dataset-imagenet-val/README-extra.md) to download the dataset and register in CM.

diff --git a/...lenge/optimize-mlperf-inference-v3.1-2023/docs/generate-retinanet-submission.md b/...lenge/optimize-mlperf-inference-v3.1-2023/docs/generate-retinanet-submission.md
@@ -1,11 +1,19 @@
 ## Setup
-Please follow the MLCommons CK [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) to install CM.
-Download the ck repo to get the CM script for MLPerf submission
+
+Please follow this [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) 
+to install the MLCommons CM reproducibility and automation language in your native environment or Docker container.
+
+Then install the repository with CM automation scripts to run MLPerf benchmarks out-of-the-box
+across different software, hardware, models and data sets:
+
 
 ```
 cm pull repo mlcommons@ck
 ```
 
+Note that you can install Python virtual environment via CM to avoid contaminating 
+your local Python installation as described [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
 ## Run Commands
 
 

diff --git a/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/generate-rnnt-submission.md b/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/generate-rnnt-submission.md
@@ -1,11 +1,19 @@
 ## Setup
-Please follow the MLCommons CK [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) to install CM.
-Download the ck repo to get the CM script for MLPerf submission
+
+Please follow this [installation guide](https://github.com/mlcommons/ck/blob/master/docs/installation.md) 
+to install the MLCommons CM reproducibility and automation language in your native environment or Docker container.
+
+Then install the repository with CM automation scripts to run MLPerf benchmarks out-of-the-box
+across different software, hardware, models and data sets:
+
 
 ```
 cm pull repo mlcommons@ck
 ```
 
+Note that you can install Python virtual environment via CM to avoid contaminating 
+your local Python installation as described [here](https://github.com/mlcommons/ck/blob/master/cm-mlops/automation/script/README-extra.md#using-python-virtual-environments).
+
 ## Run Commands
 
 ### TensorRT backend

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-aws-instance.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-aws-instance.md
@@ -1,3 +1,5 @@
+## Setup ASW instance for MLPerf
+
 The below instructions are for creating an AWS instance from the CLI. You can also create an instance via web and setup CM on it.
 
 ## Prerequisites

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-gcp-instance.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-gcp-instance.md
@@ -1,3 +1,5 @@
+## Setup GCP instance for MLPerf
+
 The below instructions are for creating a Google Cloud instance from the CLI. You can also create an instance via web and setup CM on it.
 
 ## Prerequisites

diff --git a/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-nvidia-jetson-orin.md b/.../challenge/optimize-mlperf-inference-v3.1-2023/docs/setup-nvidia-jetson-orin.md
@@ -1,4 +1,5 @@
 ## Setup
+
 We used Nvidia Jetson AGX Orin developer kit with 32GB RAM and 64GB eMMC. We also connected a 500GB SSD disk via USB and Wifi connection for internet connectivity.
 
 We used the out of the box developer kit image which was running Ubuntu 20.04 and JetPack 5.0.1 Developer Preview (L4T 34.1.1) with CUDA 11.4. We were also using the default 4k page size (Nvidia recommends 64k for MLPerf inference).

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-amazon-inferentia-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-amazon-inferentia-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for publicly-available Ama
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-amd-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-amd-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for AMD-based platforms.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-create-end-to-end-app/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-create-end-to-end-app/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for AMD-based platforms.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-google-tpu-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-google-tpu-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for publicly-available Goo
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/...ops/challenge/optimize-mlperf-inference-v3.1-hugging-face-models-2023/README.md b/...ops/challenge/optimize-mlperf-inference-v3.1-hugging-face-models-2023/README.md
@@ -11,7 +11,7 @@ MLPerf BERT model is available at Hugging Face [here](https://huggingface.co/ctu
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-intel-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-intel-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for Intel-based platforms.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-kilt-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-kilt-2023/README.md
@@ -6,7 +6,7 @@ using KILT. Compare usability with MLCommons MITL.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-mitl-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-mitl-2023/README.md
@@ -8,7 +8,7 @@ Compare usability and results with KILT.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-nvidia-gpu-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-nvidia-gpu-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for Nvidia GPUs.
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-qualcomm-ai100-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-qualcomm-ai100-2023/README.md
@@ -5,7 +5,7 @@ Prepare and optimize MLPerf inference v3.1 submission for Qualcomm AI100-based p
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-tvm-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-tvm-2023/README.md
@@ -9,7 +9,7 @@ using the [MLCommons CK framework](https://github.com/mlcommons/ck):
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to run and optimize MLPerf inference benchmarks.
 
-## Organizers
+### Organizers
 
 * [Deelvin](https://deelvin.com)
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)

diff --git a/cm-mlops/challenge/optimize-mlperf-inference-v3.1-windows-2023/README.md b/cm-mlops/challenge/optimize-mlperf-inference-v3.1-windows-2023/README.md
@@ -3,7 +3,7 @@
 Make it possible to run MLPerf inference v3.1 benchmarks on Windows
 using the [MLCommons CK framework](https://github.com/mlcommons/ck).
 
-## Organizers
+### Organizers
 
 * Stanley Mwangi (Microsoft)
 * Grigori Fursin (MLCommons, cTuning & cKnowledge)

diff --git a/cm-mlops/challenge/optimize-mlperf-training-v3.0-2023/README.md b/cm-mlops/challenge/optimize-mlperf-training-v3.0-2023/README.md
@@ -7,7 +7,7 @@ Join this public [Discord server](https://discord.gg/JjWNWXKxwT)
 to discuss with the community and organizers 
 how to use and enhance CK to run and optimize MLPerf inference benchmarks.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/optimize-tinymlperf-inference-v3.0-2023/README.md b/cm-mlops/challenge/optimize-tinymlperf-inference-v3.0-2023/README.md
@@ -6,7 +6,7 @@ Join this public [Discord server](https://discord.gg/JjWNWXKxwT)
 to discuss with the community and organizers 
 how to use and enhance CK to run and optimize MLPerf inference benchmarks.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/repro-mlperf-inf-v3.0-orin/README.md b/cm-mlops/challenge/repro-mlperf-inf-v3.0-orin/README.md
@@ -7,7 +7,7 @@ Reproduce MLPerf inference v3.0 benchmark results for Nvidia Jetson Orin
 Join this public [Discord server](https://discord.gg/JjWNWXKxwT) to discuss with the community and organizers
 how to use and enhance CK to benchmark and optimize ML Systems.
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/repro-mlperf-inference-retinanet-scc2022/README.md b/cm-mlops/challenge/repro-mlperf-inference-retinanet-scc2022/README.md
@@ -3,7 +3,7 @@
 Reproduce the MLPerf inference RetinaNet benchmark during Student Cluster Competition at SuperComputing'22
 using the following [CK2(CM) tutorial](https://github.com/mlcommons/ck/blob/master/docs/tutorials/sc22-scc-mlperf.md).
 
-## Organizers
+### Organizers
 
 * [MLCommons taskforce on automation and reproducibility](https://cKnowledge.org/mlcommons-taskforce)
 * [cTuning foundation](https://cTuning.org)

diff --git a/cm-mlops/challenge/reproduce-and-automate-ipol-paper/README.md b/cm-mlops/challenge/reproduce-and-automate-ipol-paper/README.md
@@ -2,7 +2,7 @@
 
 Reproduce and automate IPOL paper (proof-of-concept): 
 
-### Organizers
+#### Organizers
 
 * Jose Hernandez
 * Miguel Colom