Support configurable Hugo modules

[myrthos]

I think having the ability to show (Latex) formulas in my markdown files is a nice feature. It is not something that I use often, but it is a nice feature to have available.

I have currently added it to the template myself. I am using Katex, the other option would be mathjax. I am currently using the version where I load the javascript and CSS files in partials/assets/katex.html and in layouts/partials/head/head.html I conditionally load the Katex partial, depending on whether or not in the frontmatter math is set to true. This makes it not load on every page. Obviously, I also needed to modify server.toml to allow these external files to be loaded.

There is also a nodejs solution that I have not tried. Nor did I try the mathjax option. If you think it would be a nice addition to include in the template, I'll leave it to you to decide what the best solution is. If you don't like it, that is also fine, given that I already have a working solution :)

[markdumay]

Thanks for submitting your feature request. It triggers me to look into some kind of customizable "modules" approach. I prefer the current npm setup, where all files are bundled and served without the need of a CDN - and are updated by dependabot automatically. But as you mention already, you wouldn't want to include a lot of unnecessary files either.

Would you mind sharing your current code as an example? I'll look into the KaTeX documentation, it looks like a mix of Node.js and Browser is needed.

[myrthos]

Sure.

I added the file katex.html in layouts/partials/assets/ with the following contents:

<link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.css" integrity="sha384-GvrOXuhMATgEsSwCs4smul74iXGOixntILdUW9XmUC6+HX0sLNAK3q71HotJqlAn" crossorigin="anonymous">
<!-- The loading of KaTeX is deferred to speed up page rendering -->
<script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/katex.min.js" integrity="sha384-cpW21h6RZv/phavutF+AuVYrr+dA8xD9zs6FwLpaCct6O9ctzYFfFr4dgmgccOTx" crossorigin="anonymous"></script>
<!-- To automatically render math in text elements, include the auto-render extension: -->
<script defer src="https://cdn.jsdelivr.net/npm/[email protected]/dist/contrib/auto-render.min.js" integrity="sha384-+VBxd3r6XgURycqtZ117nYw44OOcIax56Z4dCRWbxyPt0Koah1uHoK0o4+/RRE05" crossorigin="anonymous"
    onload="renderMathInElement(document.body);"></script>

<script>document.addEventListener("DOMContentLoaded",function(){renderMathInElement(document.body,{delimiters:[{left:"$$",right:"$$",display:true},{left:"$",right:"$",display:false}]})});</script>

In the file layouts/partials/head/head.html I added {{ if .Params.math }}{{ partial "assets/katex.html" . }}{{ end }} to load the Katex partial.

in the file server.toml I have added https://cdn.jsdelivr.net to script_src, style_src, font_src and connect_src. And the hash 'sha256-zqrgdmVTCHeZA3joAgSlAjQWAQSAPSJwoKyRhD8nqnc=' has been added to script_src, which takes care of the last line of what is in katex.html.
This is my complete content-security-policy. It is quite different from yours as I removed everything from google and also added support for GOAT, which is default supported in Hugo already, and also support for utterances. I'm not sure if the hashes for GOAT also work in production, as I have not tried that yet. (I actually wanted to enable mermaid instead of GOAT, but that requires to add so many hashes in script-src, that it is impossible).

    Content-Security-Policy = """\
        default-src 'self'; \
        script-src 'self' \
            https://utteranc.es/client.js https://cdn.jsdelivr.net \
            'sha256-zqrgdmVTCHeZA3joAgSlAjQWAQSAPSJwoKyRhD8nqnc=' 'sha256-Yl2Mo78PNibh19ibhdKx03FFbKW9+8fljFAUUyUY1IQ=' \
            'sha256-avLhcj2GD+l0pju24dWnj9qz5CZkAxQ6WrkK1IaSva4='; \
        style-src 'self' 'unsafe-hashes' https://utteranc.es https://fonts.googleapis.com https://www.youtube.com https://cdn.jsdelivr.net \
            'sha256-kFAIUwypIt04FgLyVU63Lcmp2AQimPh/TdYjy04Flxs=' 'sha256-XzJlZKVo+ff9ozww9Sr2p/2TbJXITZuaWMZ9p53zN1U=' \
            'sha256-hqhQ1AAR6jgr9lel8hs9sNOeqSwsGx6HH+B7TkLcmyY=' ;\
        object-src 'none'; \
        base-uri 'self'; \
        connect-src 'self' \
            https://cdn.jsdelivr.net; \
        font-src 'self' https://fonts.gstatic.com https://cdn.jsdelivr.net; \
        frame-src 'self' https://utteranc.es https://www.youtube-nocookie.com https://www.youtube.com; \
        img-src 'self' https://i.vimeocdn.com https://i.ytimg.com  https://img.youtube.com; \
        manifest-src 'self'; \
        media-src 'self' \
        """

In the markdown file math: true has to be added to the frontmatter to enable katex.

This is what I used for testing in the markdown file:

This is an inline $-b \pm \sqrt{b^2 - 4ac} \over 2a$ formula

This is not an inline formula:

$$x = a_0 + \frac{1}{a_1 + \frac{1}{a_2 + \frac{1}{a_3 + a_4}}}$$  
$$\forall x \in X, \quad \exists y \leq \epsilon$$

That should be it.

Note that I am not an expert in Latex, so I use CodeCogs to help me build the equations.

[markdumay]

Thanks for sharing! I've added a new feature request #323 to track the "modules" approach too. I'll use your code and the KaTeX npm library as a case example. Feel free to share your thoughts in either issue.

The feature might require a significant code change, so I can't commit to any schedule yet. However, the summer break is coming up, which might give me some ample time to work on this...

[myrthos]

That would be a great way to add additional functionality.
I do hope that not all feature requests I might make, end up in being a big time sink :)

[markdumay]

After some experimentation I believe to have found a satisfactory approach to include configurable Hugo modules. The findings are described in #323.

The next steps:

• define an example module (using Leaflet for now)
• adjust the build pipelines for JS and SCSS files
• have an option to either include a module globally (site level) or individually (page by page using frontmatter params)
• adjust the CI/CD pipeline to automatically upgrade (npm) dependencies AND to publish new releases (for modules)
• adjust the CI/CD pipeline to automatically upgrade (hugo module) dependencies (core library and template)
• add documentation on how to use and configure modules
• update the guide on https://gethinode.com/guides/packages/
• add additional modules including KaTeX
• consider adapting the module approach for current npm modules (FlexSearch, Bootstrap, FontAwesome)

Feel free to comment!

[myrthos]

I've read through all that and it would be great if that will work. My experience with Hugo is limited to working through the book, 'Hugo in Action' and modifying the Hinode template, so I might not be the one adding very meaningful comments.
From a management and maintenance perspective, I think it is good if the page that is being loaded into the browser has a payload that is as small as possible. Obviously CSS and JS files are cached on the client side, but still, if the page does not need a CSS or JS file, why bother including them. The second thing is that there is no reason to have modules on a site that are not used.
What you wrote, addresses both, so I would be very happy if you get that to work.

That said, I do have a comment related to your last bullet point. Some modules are essential to have on a site. Being able to search is one of them. It should be a basic module. The only reason to make it possible for such a module to remove it, is if someone wants to replace it with another search module. Having a site that you cannot search, is bad in my opinion.
For Bootstrap and FontAwesome this is not true, but I personally would not remove them. The additions to Markdown, bootstrap gives are great. There are other modules that also would add great things to Markdown and could replace bootstrap, but why not install them next to bootstrap.
FontAwesome is really nice to have, besides that, it is also used everywhere in the template where an icon could or is used, so that would break a number of things, which need to be changed if it would be removed.

I think it is fine to have core components that the template cannot do without. And for these 3 specific components I'm not sure if anyone will use the option to remove or replace them. As such I think the balance between the effort you put into it and how often it would be used is off.
If you however cannot resist the technical challenge to also make it happen for these modules, I do think it would be good to add an example of not only adding a module, but also removing one and/or replacing a module (but, perhaps you meant that with the 'add documentation' point).

[markdumay]

Thanks with your elaborate feedback! It's of great help to have an outside-in perspective. I agree with your remark - the core modules are unlikely to be disabled and should be included by default. The main reason to put these core libraries into to modules is to standardize the approach and thus to minimize maintenance. For now, it has low priority.

[markdumay]

I released an alpha version of the new module system. The basic functionality has been implemented, although there are probably a few things to iron out.

I plan to work on the KaTeX module next, I'll keep you posted.

[myrthos]

Nice. I will try it out later this week.

[myrthos]

I checked it out and it looks quit nice.

It works offline (once the modules are downloaded), which is good.
I tried to remove a module, but was unable to get that accomplished. It is very likely that I am missing something, which is going to be in the future documentation. So, I'll wait on that.
I'm still not sure about the usefulness of a bootstrap module, given that a lot of the partials and shortcodes depend on bootstrap CSS. Not having bootstrap would not render the site properly. But perhaps there is a use case where having this makes sense.

Once it is out of the alpha stage, I would like to try to convert my additions to Hinode into a module (currently these are a gallery and a different way of displaying single images).

[markdumay]

Thanks for taking the time to do an initial review. Meanwhile, I have now included the KaTeX module and updated the docs. A live preview is available on https://modules--gethinode-docs.netlify.app (it's updated on every push to the modules branch, so it might be unstable).

I tried to remove a module, but was unable to get that accomplished. It is very likely that I am missing something, which is going to be in the future documentation. So, I'll wait on that.

Would you care to elaborate on that? The documentation is available on https://modules--gethinode-docs.netlify.app/docs/configuration/modules/ (subject to change).

I'm still not sure about the usefulness of a bootstrap module, given that a lot of the partials and shortcodes depend on bootstrap CSS. Not having bootstrap would not render the site properly. But perhaps there is a use case where having this makes sense.

I agree that Bootstrap is an essential component of Hinode. Actually, I included a check in app.scss to confirm the module is included as a core module. However, there are two main reasons why I decided to move the code to a separate module:

1. Shifting all npm packages to a (separate) module means that Hinode itself is no longer reliant on npm. It becomes an option instead of a prerequisite. This means Hinode becomes more of a 'regular Hugo theme' and works like any other theme out-of-the-box. For advanced users - especially those interested in applying CI/CD - npm is still an option.

2. Standardizing the module approach means the maintenance becomes easier. I have now automated all dependency upgrades and releases for the five modules. I know from experience that it becomes quite tedious to manually test and merge every single dependency. The base code has been added to a module template that I intend to keep up-to-date.

Once it is out of the alpha stage, I would like to try to convert my additions to Hinode into a module (currently these are a gallery and a different way of displaying single images).

I'd be very interested in your experience with developing custom modules. I've prepared a guide on https://modules--gethinode-docs.netlify.app/guides/modules/ to get you started. If you're willing, you might want to share your modules with the OSS community too?

[myrthos]

Before I elaborate on not being able to not install a module, I'll have a closer look at the documentation first, because the error is probably on my side :)

As to Bootstrap, after writing my post I noticed also the re-opened post about the Go modules and it makes sense in that context. For the record, I was not challenging your decision. It was just me failing to grasp what the benefits of such a move were.

My gallery module resembles Katex. Not from the context, but from the steps needed to get it working. So I will use the Katex module as a reference and I will read the documentation first, this time.

And I will put everything in Github; the site, modules and the projects. I work in a self-hosted Gitea environment and as I am working on three different computers in my home, I use Gitea to sync between those computers. As a result my commits are quite often breaking the software, which doesn't matter as I am the only one having access to the repository.
At the moment I am cleaning things up to move the site to Github somewhere in the near future.

[markdumay]

Thanks for your feedback, appreciate it! Let me know if you need any help with the module development. I used Gitea in the past myself (self-hosted on a NAS), so I'm not too unfamiliar with the challenge to keep it up and running. :-)

I'll update the Hinode template next and aim for a final release of v0.16 soon. There's one optimization left (the alphabetical sort of folders and files in layouts/partials/utilities/bundle.html requires some adjustment) - but I'll probably leave that for now.

[markdumay]

Done! The docs and template have been updated too. Let me know if you encounter any issues.

[myrthos]

I created a new issue, instead of replying here. My mistake, so here is the text again in the right location.

I have tried to create a module. I followed your guide, as it is my first time in creating a module. The gallery uses Lightbox 2, which I installed with NPM. Following the Katex module guide, the correct files are ending up in the /dist folder. So that went well. The documentation is quite clear in that respect.
I also added the mgallery shortcode and mgallery css files, which I created myself, manually.

I tried loading this module in my site, but could not get it to work. As it was my first time, I had to figure out how that should be done, but it is loading as it is entered into _vendor and I do get the mgallery shortcode in my site. However, I am not getting the two css files, nor the javascript file, or the images.
I checked your modules with mine, but can't detect what I am doing wrong.
(I checked with the web developer tool of the browser, and I indeed don't see any of the javascript code, nor any of the classes.)
I'm still trying to figure this one out, based on the available documentation.

As to the documentation, I think it would be good to mention somewhere that when you want to use Katex on a page, the following should be present in the frontmatter, to load the javascript file:

modules: ["katex] 

[markdumay]

Thanks for giving it try! I can imagine the guide requires more clarification. Before revising the guide, let's see if we can get your module up and running.

I created a fork of your module in my personal account. I pushed several changes to the develop branch for consideration. The example site is probably not rendered 100% correct yet, but it least I got it working. You can run the example with the following command:

npm run start

This is a screenshot of the local site. It renders a grid of example images, however, it appears the thumbnails are not scaled correctly yet. They do open when you click on them.

A few observations regarding your initial code:

• I would typically only put files that are being redistributed (e.g. copied from the npm package) in the dist folder. Your own maintained code (shortcode, custom styles, etc) should be kept in the usual Hugo folders (assets, layouts, etc). I have moved your code in the fork to these folders.
• You seem to mix the module names, I see both mgallery and lightbox in your codebase. Hinode expects all code (both scss and js) to be present in mgallery for the build pipelines to work correctly. See the documentation for more details. I have revised the fork accordingly.
• I have added an example site to do some unit testing. I'd suggest to get this example working correctly first, before doing the integration test with Hinode. The example site also functions as gatekeeper for any (automated) updates (either manually or in CI/CD).

[myrthos]

Great, with your help I figured it out and it works now as it should. Thanks a lot.
While going through the documentation again, I noticed that all the information is there and all the links are there as well. I should really work on my reading skills 😖
Again, thanks for the help.

I did deviate somewhat from your first bullet point. I understand the reasoning, but I should either rename the module or the mgallery shortcode/scss files to a different name than the module, to prevent confusion. So for now, I have the mgallery.scss file import the lightbox class. I'm thinking about replacing lightbox, so I will see how I fix this when I make that change.

As to the image you showed. It should look something like this

The shortcode and classes assume that they are on a Hinode page, this means they make use of the already existing bootstrap classes on a page in formatting the gallery. The shortcode is not designed to be used on a non-Hinode installation.

To simulate that on the example site, HTML should be enabled

[markup]
  [markup.goldmark]
    [markup.goldmark.renderer]
      unsafe = true

And the contents of the index page should look like:

<div class="d-flex flex-column flex-fill">
    <div class="container-xxl">
        <div class="row row-cols-1">
            <div class="col col-sm-12 col-md-8">

## Grid layout

{{< mgallery list="img/*" unique="true" cols=3 type="grid" gap="m" radius="true" zoom="true" >}}

## Masonry layout

{{< mgallery list="img/*" unique="true" cols=3 type="masonry" gap="m" radius="true" zoom="true" >}}

</div>
</div>
</div>
</div>

I might be missing a class somewhere, but this is good enough for testing purposes.
My file structure in the content folder is also different from yours, to resemble the structure I also use on my site, where images are located in their own folders, hence the "img/* blob.

[markdumay]

Ok, good to hear! Let me know if you have any suggestions to further improve the guide / docs.

[myrthos]

I have a different request, related to the search-input partial, which is now part of a module.
I prefer to emphasize the border of the search box a bit. So, I originally just changed the search box class form-control.is-search to change the border, but that doesn't work like that anymore, now that it is a module.
Would you consider the possibility for adding a class as a parameter to the search-input partial and add it to the input element?

[markdumay]

Sure! Would you mind submitting a feature request so I can track it?