How to rebuild the minimized JS code in docs/theme/assets/pymdownx-extras?

[tijuca]

Hi,

the upstream code currently includes some minimized JS code in docs/theme/assets/pymdownx-extras.
While working on packaging pymdown-extensions for Debian the QS tool Lintian is mentioning that some source code files are missing for these minimized JS files.

E: pymdown-extensions source: source-is-missing docs/theme/assets/pymdownx-extras/extra-loader-9e2c67b9.js
E: pymdown-extensions source: source-is-missing docs/theme/assets/pymdownx-extras/material-extra-3rdparty-69a692a6.js
E: pymdown-extensions source: source-is-missing docs/theme/assets/pymdownx-extras/material-extra-theme-7c147bb7.js

The reason for this is that the DFSG and the Debian packaging rules see such minimized files not as usual source code because it's not really readable by human. From a Debian POV it would be enough if the source for the files is available to, even better would be if we could rebuild the minimized files while we build the package.

Is it possible to add the minimized files or at least some kind of small read me where the source is coming from and how it was built? That would help me to get the package prepared for an upload to the Debian archive. Thanks!

[facelessuser]

I'm confused, the JS files are included in the tar ball, but are only needed for the documentation if it is built. Are you actually building and using the documentation? The JS files are not used in the actual library.

I feel like I've had this conversation with a package maintainer at one time...I can't remember if it was for this package or another, but ultimately, they didn't bother generating them as it is not useful for the package itself.

[facelessuser]

If you are adamant about building the JS from the source, you can regenerate, assuming Node is installed.

npm install
npm run build

Node dependencies are kind of a nightmare, so you may have to use npm install --legacy-peer-deps. It isn't always needed, but depending on what version of what libraries are being included, it can be required. It also seems to depend on what version of node is being used.

It's all shown in the build matrix for CI: https://github.com/facelessuser/pymdown-extensions/blob/main/.github/workflows/build.yml#L114.

Again, I think this is a complete waste of time for packaging as the assets are already provided and are not needed for any functional use of the documentation.

Users can rebuild the documentation without rebuilding the assets if they want to rebuild docs locally by just running python3 -m mkdocs build (assuming all the documentation requirements are installed).

[facelessuser]

Oh, I see we don't include the JS and CSS assets in the tar ball. But I don't see the value in including those. We are providing what is needed for documentation in the tar ball. I'm sure there are some projects that just include the built documentation only.

I'd be interested in why this is absolutely required, especially as the last conversation concluded wth no change.

[tijuca]

Apparently, we had this conversation on this very repository: #1536. Is there a new reason you are asking this again?

Ohh, I've forgotten that I did ask the very same in the past, I'm getting old.

Last year some other fellow Debian Developer was quicker than me and started a packaging of this library too. The technical packaging was done by him but until now the package did not made its way into the archive yet. Since then more and more other packages are depending on pymdownx-extensions and I'm trying to do at least some helping to get this tool packaged.

It's all shown in the build matrix for CI: https://github.com/facelessuser/pymdown-extensions/blob/main/.github/workflows/build.yml#L114.

Again, I think this is a complete waste of time for packaging as the assets are already provided and are not needed for any functional use of the documentation.

Thanks for the pointer to the CI pipeline. The npm world is definitely not my world due this dep hell. :-)

....
I'd be interested in why this is absolutely required, especially as the last conversation concluded with no change.

It's not a hard requirement for Debian packaging as long the maintainer can explain how the minimized files (no matter if this is JS, CSS or also Shell or Python real long one-liners e.g.) were created. This mostly can only be done together with the upstream maintainers so really thanks for your feedback! For the creation of binary packages we have a generic rule in Debian, we want be able to build all the output on our very own based on the shipped source code files and the tools from the package archive.

Minimized JS or CSS code isn't really source code from a POV of the Debian project because we or any other people can't easily modify such files. They are created to save bandwidth and to be read by machines, not humans, Debian Developers saying they are not the preferred form for modification.

I'm working on my day job within a closed environment and can't easily reach any website from the internet so my colleagues and myself are really happy if Debian is also providing the documentation packages for the various packages too we typically use. That's why I'm always trying to also package any HTML based documentation from an upstream project and so also for pymdown-extensions. For this all the files in the source tarball are relevant due the Debian packaging rules. And so the minimized JS files are relevant finally for me.

Oh, I see we don't include the JS and CSS assets in the tar ball.

Would it be possible to write some small explanation why this is done and also some sentences about the update process in case into docs/src? That would make it easier for me to explain my packaging work against the FTP masters of Debian which finally have to decide if the package can go into the archive. Doing this would also make it clear to other users how things work together and potentially prevent others asking similar questions in the future. I'd write some PR if I know what to write, but I'm sure it will be more effective if such things would be written by the upstream.

[facelessuser]

Would it be possible to write some small explanation why this is done and also some sentences about the update process in case into docs/src?

Are you asking that I commit a document inside docs/src that explains this?

I can give a general explanation:

• My goal is to provide a useful working package and documentation in case it is desired to read it locally. The tarball includes all of that, minus the required dependencies to build the docs, but the markdown source is available. The code provides the entire package source, but the doc source is irrelevant to the package, it is for stylistic and functional purposes of the docs. The minimized source, which is all the user needs to generate the docs is included in the distribution, all they need to do is install relevant python dependencies via requirements/docs.txt and build.
• To generate, they should simply run python3 -m mkdocs build.

As far as how I update the JS and CSS build environment and generate the minimized source.

• This requires a user to check out the git source as the JS, CSS, and related infrastructure is only available from the git repo. All of this unnecessary bloat is stripped out of the tarball.
• For maintainability, I use babel to allow the latest JS conventions. The code is then compiled into something accessible to most browsers.
• I use SCSS styles for the custom CSS in the project and compile them all into a single compatible CSS file.
• JS and CSS and minimized to reduce the payload size.
• All of this is a pain to maintain, so we use a script to automate this gulpfile.babel.js. Gulp is used to automate the process in node.js.
• Node projects allow you to pin specific versions and manage dependencies in package.json and package-lock.json files. When I wish to update dependencies, I use npm-check-updates. For some reason, a more sensible approach is not really built into npm, so using this package, we can call ncu -u. Then we call npm install. This updates the JSON lock files and such and installs the new dependencies. Then we regenerate the minimized files via npm run build. Usually, there should be no changes unless we change the source, but if there are changes, we will usually double-check any changes by generating the unminimized source and ensuring there is nothing suspicious.
• python3 -m mkdocs serve will run generate the files unminimized and run a simple server to allow viewing and validation of the documentation.

[sandrotosi]

Hello there, this is the Debian developer that's doing the packaging for pymdown-extensions. The reason we want to know the origin of a minified JS is that we want to be able to:

1. reproduce the build (if we want to, it's not done automatically)
2. inspect which version of which software is in the global Debian archive
3. security reasons (libX version Y is buggy, which package contains it?)
4. licensing reasons (is this lib at this version released under a DFSG license?)
5. probably more i cant think of right now

please also note that the source we're basing our packaging is the git tag tarball, so all the files in this repo are included at that tag.

the main concern that we have is the origin of these files:

docs/theme/assets/pymdownx-extras/extra-e384f43f0f.css
docs/theme/assets/pymdownx-extras/extra-e384f43f0f.css.map
docs/theme/assets/pymdownx-extras/extra-loader-5bb526e4.js
docs/theme/assets/pymdownx-extras/extra-loader-5bb526e4.js.map
docs/theme/assets/pymdownx-extras/material-extra-3rdparty-69a692a6.js
docs/theme/assets/pymdownx-extras/material-extra-3rdparty-69a692a6.js.map
docs/theme/assets/pymdownx-extras/material-extra-theme-7c147bb7.js
docs/theme/assets/pymdownx-extras/material-extra-theme-7c147bb7.js.map

please also note that they dont even have to be used at all at any point in the build and/or deploy process: we need to know their origin for the sole reason they are in the source tarball we use.

now, to conclude this long post with the info we need: @facelessuser are you saying this 2 pairs of files (.js & .js.map) are generated from docs/src/js/* and docs/src/scss/? if that's the case, that's probably all we need to know

thanks!

[facelessuser]

reproduce the build (if we want to, it's not done automatically)

I see no reason for this in the tarball, we ship the minimum needed so docs can be built locally if desired, but currently have no intention of providing all the JS framework bloat. I'm struggling to see the value in this.

inspect which version of which software is in the global Debian archive

I don't quite understand this because since we don't include any of the JS deps, we don't need to worry about them.

security reasons (libX version Y is buggy, which package contains it?)

Again, the JS build deps aren't being included, just our own custom loaders for things like MathJax, and Mermaid UML diagrams.
There is no security risks with the compressed CSS, and I'd be surprised if there are any security risks in our custom loaders, but if needed, they can be audited in the repo: https://github.com/facelessuser/pymdown-extensions/tree/main/docs/src/js.

licensing reasons (is this lib at this version released under a DFSG license?)

Again, what libs are we talking about? None are included.

The only external libs for documentation that are actively used in the tarball are defined in the included source: https://github.com/facelessuser/pymdown-extensions/blob/main/mkdocs.yml#L167

I've kind of already explained the process, and I'm happy to help anyone that is interested recreate the build if it is truly desired, but I see no value in including all of this in the tarball. No other package maintainers have ever asked for this, and it is included in many, various distributions.

[sandrotosi]

I've kind of already explained the process, and I'm happy to help anyone that is interested recreate the build if it is truly desired, but I see no value in including all of this in the tarball. No other package maintainers have ever asked for this, and it is included in many, various distributions.

i'm gonna re-read all your replies again, but please note we did not ask to provide any extra file in the release tarball. We're only looking for the location/explanation of the sources that produce those file, then the rest is taken care of from the Debian side.

thanks for your help and for maintaining pymdown-extensions!

[facelessuser]

I'm more than happy to answer specific questions. If any thing is unclear please direct the question to exactly what is unclear.