From 54c2b298cc9645c29a434bf353168093744b9ca3 Mon Sep 17 00:00:00 2001
From: Xiang ZHU <xiang.zhu@outlook.com>
Date: Sat, 21 Oct 2023 13:05:42 +0200
Subject: [PATCH] add post / 2023-10-21-hashing-files

---
 _posts/2023/2023-10-21-hashing-files.md | 89 +++++++++++++++++++++++++
 1 file changed, 89 insertions(+)
 create mode 100644 _posts/2023/2023-10-21-hashing-files.md

diff --git a/_posts/2023/2023-10-21-hashing-files.md b/_posts/2023/2023-10-21-hashing-files.md
new file mode 100644
index 00000000..17bc1aa8
--- /dev/null
+++ b/_posts/2023/2023-10-21-hashing-files.md
@@ -0,0 +1,89 @@
+---
+last_modified_at:
+title: "Hashing files"
+excerpt: ""
+tags:
+  - cicd
+  - githubaction
+  - azure
+  - shell
+  - cache
+published: true
+# header:
+#   teaserlogo:
+#   teaser: ''
+#   image: ''
+#   caption:
+gallery:
+  - image_path: ''
+    url: ''
+    title: ''
+---
+
+During CI/CD processes, and particularly during CI, we frequently hash dependency files to create cache keys (referred to as `key` input in Github Action [actions/cache](https://github.com/actions/cache) and `key` parameter in Azure pipelines [Cache@2 task](https://learn.microsoft.com/en-us/azure/devops/pipelines/tasks/reference/cache-v2?view=azure-pipelines)). However, the default hash functions come with certain limitations like [this comment](https://github.com/orgs/community/discussions/25761#discussioncomment-6508758). To address this, we can use the following pure Bash shell command to manually generate the hash value.
+
+For Github Actions, we can use following snippet:
+
+{% raw %}
+
+```yaml
+# github actions example
+inputs:
+  req-files:
+    description: >
+      requirements files separated by comma or space, glob pattern is allowed.
+      e.g. "requirements/*.txt, requirements.txt"
+    required: true
+runs:
+  using: "composite"
+  steps:
+    - name: Compute hash key
+      shell: bash
+      env:
+        REQ_FILES: ${{ inputs.req-files }}
+      run: |
+        files=$(echo "$REQ_FILES" | tr "," " ")
+        files_sep_by_space=""
+        for file in $files; do
+            files_sep_by_space="$files_sep_by_space $(ls $file | tr '\n' ' ')"
+        done
+        files_sep_by_space=$(echo $files_sep_by_space | tr ' ' '\n' | sort | uniq | tr '\n' ' ')
+        files_hash=$(cat $files_sep_by_space | md5sum | awk '{print $1}')
+        echo "files_hash: $files_hash"
+```
+
+{% endraw %}
+
+For Azure pipelines, the process is nearly identical to the above Github Action example. The only difference is that we first need to convert the  `reqFiles` parameter from an object to a string. But if you set the parameter type to `string` (as in the Github Action), the process becomes identical.
+
+{% raw %}
+
+```yaml
+# azure pipelines example
+parameters:
+  - name: reqFiles
+    displayName: >
+      requirements files, glob pattern is allowed.
+      e.g.:
+      - requirements/*.txt
+      - requirements.txt
+    type: object
+  steps:
+    - script: |
+        files=$(echo "$REQ_FILES_JSON" | jq  '. | join(" ")' -r)
+        files_sep_by_space=""
+        for file in $files; do
+            files_sep_by_space="$files_sep_by_space $(ls $file | tr '\n' ' ')"
+        done
+        files_sep_by_space=$(echo $files_sep_by_space | tr ' ' '\n' | sort | uniq | tr '\n' ' ')
+        files_hash=$(cat $files_sep_by_space | md5sum | awk '{print $1}')
+        echo "files_hash: $files_hash"
+      displayName: Compute hash key
+      env:
+        REQ_FILES_JSON: "${{ convertToJson(parameters.reqFiles) }}"
+```
+
+{% endraw %}
+
+When creating the cache key, we also need to include os version, the one provided by Github action and Azure pipelines environment vars are not precise enough, they do not give patch version number. We can generate the **full os version** by the following command {%raw%}`cat /etc/os-release | grep -i "version=" | cut -c9- | tr -d '"' | tr ' ' '_'`{%endraw%}
+{: .notice--info}