new post 2023-12-16-migrating-my-blog-from-jekyll-minimal-mistakes-to…

…-mkdocs-material.md

docs/posts/2023/2023-12-16-migrating-my-blog-from-jekyll-minimal-mistakes-to-mkdocs-material.md

@@ -0,0 +1,133 @@
+---
+authors:
+- copdips
+categories:
+- web
+- mkdocs
+comments: true
+date:
+  created: 2023-12-16
+description: ''
+---
+
+# Migrating my blog from Jekyll Minimal Mistakes to Mkdocs Material
+
+After using [Jekyll Minimal Mistakes](https://mmistakes.github.io/minimal-mistakes/) for years, I decided to migrate [my blog](https://copdips.com) to [MkDocs Material](https://squidfunk.github.io/mkdocs-material/), as it's written in Python and I'm more familiar with it.
+
+I am very grateful to the [MkDocs Material](https://squidfunk.github.io/mkdocs-material/) maintainers for giving me [some valuable tips](https://github.com/squidfunk/mkdocs-material/discussions/6430#discussioncomment-7753446) for the migration kickoff.
+
+## Create blog folder
+
+I needed a pure blog without documentation, so I need the URL to be `https://copdips.com/`, instead of `https://copdips.com/blog/` to be compatible with Jekyll blog URL format. To do this, I followed [this documentation](https://squidfunk.github.io/mkdocs-material/setup/setting-up-a-blog/?h=authors+yml#blog-only).
+
+## Creating .authors.yml
+
+I created an authors file at `./docs/authors.yml`. See [this doc](https://squidfunk.github.io/mkdocs-material/setup/setting-up-a-blog/?h=authors+yml#adding-authors) for more details.
+
+## Copying all posts and converting the YAML metadata
+
+I used this [Python script](https://github.com/copdips/copdips.github.io/blob/main/scripts/jekyll_to_mkdocs/convert_yaml_metadata.py) to copy all the posts from `./_posts` to `./docs/posts/`, and converted the YAML metadata from the Jekyll format to the Mkdocs Material format.
+
+## Converting admonition
+
+The two blog engines use different admonition syntax. See [this doc](https://squidfunk.github.io/mkdocs-material/reference/admonitions/?h=admonition#syntax) for more details.
+
+I used this [Python script](https://github.com/copdips/copdips.github.io/blob/main/scripts/jekyll_to_mkdocs/convert_admonition.py) to convert the format.
+
+## Fixing headers and excerpt
+
+In the old blog, for some posts, I used `#` for all first-level headers, but in Mkdocs-Material, `#` is reserved for page title, so I needed to change all the headers to `##`, and also convert previous `##` to `###`, and so on. And for the comment symbol `#` in the code blocks, I need to skipped it.
+
+For excerpt, many of my Jekyll posts have already excerpt right after the YAML metadata and start by `>`, like `> excerpt text from here`. But in Mkdocs Material, I needed to use `<!-- more -->`. I referred to [this documentation](https://squidfunk.github.io/mkdocs-material/setup/setting-up-a-blog/?h=excerpt#adding-an-excerpt) for more details. And I chose not to set excerpt from the `excerpt` within the old posts YAML metadata, there was no special reason, but it might be useful for some other people.
+
+For example, I needed to convert following markdown:
+
+  ```markdown
+    # title
+
+    > excerpt text from here.
+
+    Another text.
+
+    # chapter 1
+
+    ```py
+    # this is a comment
+    ...
+    ```
+
+    ## section 1
+
+    # chapter 2
+
+    ## section 1
+  ```
+
+to:
+
+  ```markdown
+    # title
+
+    excerpt text from here.
+
+    <!-- more -->
+
+    Another text.
+
+    ## chapter 1
+
+    ```py
+    # this is a comment
+    ...
+    ```
+
+    ### section 1
+
+    ## chapter 2
+
+    ### section 1
+  ```
+
+I used this [Python script](https://github.com/copdips/copdips.github.io/blob/main/scripts/fix_headers.py) to fix the headers.
+
+Some posts in the old blog didn't have an excerpt, so I had to add `<!-- more -->` manually.
+
+## Fixing multiple blank lines
+
+After the above steps, there were sometimes multiple continuous blank lines in the markdown files, I used this [Python script](https://github.com/copdips/copdips.github.io/blob/main/scripts/fix_multiple_blank_lines.py) to ensure there's only one blank line each time.
+
+## Post URL
+
+### Hyphen `-` in title
+
+Mkdocs Material computes post url slug by keeping hyphen `-`, while Jekyll discards it. So given title `Github - Test`, Jekyll will generate `github-test`, while Mkdocs-Material will generate `github---test`. To keep the url the same after the migration, the workaround was to change the title to `Github: Test`.
+
+I used the VSCode find/replace feature with following regex:
+
+```yaml
+Source: `^(#[^#].*?) - (.*?)`
+Replace: `$1: $2`
+Files to include: `./docs/posts`
+```
+
+### Ending with `.html`
+
+Jekyll generates post URLs ending with `.html`, while Mkdocs-Material doesn't by default. To keep the url the same after the migration, I checked this [tip](https://github.com/squidfunk/mkdocs-material/discussions/6430#discussioncomment-7753446) by disabling the `use_directory_urls` option in mkdocs.yml.
+
+### Removing word blog from url
+
+By default, Mkdocs-Material adds the word `blog` in the URL path, but I didn't want it. To remove it, I checked this [tip](https://github.com/squidfunk/mkdocs-material/discussions/6430#discussioncomment-7753446).
+
+## Image path
+
+I used VSCode find/replace feature to replace all the image paths.
+
+## Code action view Source
+
+Code action view Source is bound to `master` branch by default, not `main` branch. To use `main` branch, I added `edit_uri: edit/main/docs/` to `mkdocs.yml`. See [this doc](https://squidfunk.github.io/mkdocs-material/setup/adding-a-git-repository/?h=content+action#code-actions)
+
+## Deploying to GitHub Pages
+
+Jekyll uses `gh-pages` branch to publish blog, but I used GitHub actions within Mkdocs, so I didn't need to use `gh-pages` branch. To use GitHub actions, I went to my repository at https://github.com/copdips/copdips.github.io/, entered `Settings` -> `Pages`, and set Github Actions as `Source`.
+
+My GitHub Actions for blog publishing can be found [here](https://github.com/copdips/copdips.github.io/blob/main/.github/workflows/build_and_deploy.yml).

mkdocs.yml

@@ -83,6 +83,7 @@ plugins:
         - linux
         - markdown
         - migration
+        - mkdocs
         - multithreading
         - network
         - package

scripts/fix_head.py → scripts/fix_headers.py

@@ -63,7 +63,7 @@ def adjust_headers(markdown_file):
 
 
 def main():
-    for root, dirs, files in os.walk("./new_posts"):
+    for root, dirs, files in os.walk("./docs/posts"):
         for filename in files:
             if filename.endswith(".md"):
                 markdown_file = os.path.join(root, filename)

scripts/jekyll_to_mkdcos.py/jeklly_to_mkdocs.py → scripts/jekyll_to_mkdocs/convert_yaml_metadata.py

@@ -7,7 +7,7 @@
 AUTHORS = ["copdips"]  # Global variable for authors
 
 
-def convert_jekyll_to_mkdocs(jekyll_file):
+def convert_metadata_from_jekyll_to_mkdocs(jekyll_file):
     with open(jekyll_file, "r", encoding="utf-8") as file:
         content = file.read()
 
@@ -66,11 +66,11 @@ def main():
         for filename in files:
             if filename.endswith(".md"):
                 jekyll_file = os.path.join(root, filename)
-                new_content = convert_jekyll_to_mkdocs(jekyll_file)
+                new_content = convert_metadata_from_jekyll_to_mkdocs(jekyll_file)
 
                 if new_content:
                     # Fixing the path for new_root
-                    new_root = root.replace("./_posts", "./new_posts")
+                    new_root = root.replace("./_posts", "./docs/posts")
                     os.makedirs(new_root, exist_ok=True)
                     with open(
                         os.path.join(new_root, filename), "w", encoding="utf-8"

tools/pymdownx_md_render.py

@@ -0,0 +1,107 @@
+"""Generate Markdown isolated from our current document options."""
+import markdown
+import yaml
+import re
+from collections import OrderedDict
+from csv2md.table import Table
+
+
+def yaml_load(stream, loader=yaml.Loader):
+    """
+    Custom YAML loader.
+
+    Load all strings as Unicode.
+    http://stackoverflow.com/a/2967461/3609487
+    """
+
+    def construct_yaml_str(self, node):
+        """Override the default string handling function to always return Unicode objects."""
+
+        return self.construct_scalar(node)
+
+    class Loader(loader):
+        """Custom Loader."""
+
+    Loader.add_constructor("tag:yaml.org,2002:str", construct_yaml_str)
+
+    return yaml.load(stream, Loader)
+
+
+def get_frontmatter(text):
+    """Get front matter from string."""
+
+    frontmatter = OrderedDict()
+
+    if text.startswith("---"):
+        m = re.search(
+            r"^(-{3}\r?\n(?!\r?\n)(.*?)(?<=\n)(?:-{3}|\.{3})\r?\n)", text, re.DOTALL
+        )
+        if m:
+            yaml_okay = True
+            try:
+                frontmatter = yaml_load(m.group(2))
+                if frontmatter is None:
+                    frontmatter = OrderedDict()
+                # If we didn't get a dictionary, we don't want this as it isn't front matter.
+                assert isinstance(frontmatter, (dict, OrderedDict)), TypeError
+            except Exception:
+                # We had a parsing error. This is not the YAML we are looking for.
+                yaml_okay = False
+                frontmatter = OrderedDict()
+
+            if yaml_okay:
+                text = text[m.end(1) :]
+
+    return frontmatter, text
+
+
+def md_sub_render(src="", language="", class_name=None, options=None, md="", **kwargs):
+    """Formatter wrapper."""
+    try:
+        import ipdb; ipdb.set_trace()
+        fm, text = get_frontmatter(src)
+        md = markdown.markdown(
+            text,
+            extensions=fm.get("extensions", []),
+            extension_configs=fm.get("extension_configs", {}),
+        )
+        return md
+    except Exception:
+        import traceback
+
+        print(traceback.format_exc())
+        raise
+
+def md_csv_render(src="", language="", class_name=None, options=None, md="", **kwargs):
+    """Formatter wrapper."""
+    try:
+
+        with open(src) as f:
+            table = Table.parse_csv(f)
+
+        text = table.markdown()
+
+        # Specify the file path for the markdown file
+        md_file_path = src.replace('.csv', '.md')
+
+        # Open the markdown file in write mode and write the content of 'new' to it
+        # with open(md_file_path, 'w') as md_file:
+        #     md_file.write(text)
+        with open("docs/cicd/databricks/onboarding_checklist.md", 'w') as md_file:
+            md_file.write(text)
+
+
+        fm = {}
+        md = markdown.markdown(
+            text,
+            extensions=fm.get("extensions", []),
+            extension_configs=fm.get("extension_configs", {}),
+        )
+        return md
+        return f'\n--8<-- "{md_file_path}"\n'
+        return text
+    except Exception:
+        import traceback
+
+        print(traceback.format_exc())
+        raise