Add fix-byte-order-marker and pretty-format-json to pre-commit

[dekkers]

Changes

This pretty formats all JSON files. The reason is I saw the whitespace changes in #2556 which only causes unnecessary conflicts when there are other changes. It will also help the JSON files in #2545 to be better readable automatically and consistently formatted.

I added --no-ensure-ascii because as far as I know there is no reason to escape utf-8 characters in JSON files.

This also adds fix-byte-order-marker because keiko/glossaries/glossary-export.json had a byte order mark which is unnecessary and not recommended for UTF-8 but caused pretty-format-json to give an error.

---

Checklist for code reviewers:

Copy-paste the checklist from the docs/source/templates folder into your comment.

---

Checklist for QA:

Copy-paste the checklist from the docs/source/templates folder into your comment.

[underdarknl]

I agree with --no-ensure-ascii and the other utf8 related fixes.

[ammar92]

Looks good to me. I prefer to also use --no-sort-keys, because order of properties may also have semantic meaning and currently without that option it produces a lot of changes.

[dekkers]

Looks good to me. I prefer to also use --no-sort-keys, because order of properties may also have semantic meaning

I saw that we tried to order some files in a certain way, but JSON is unordered and that makes it harder to enforce a non-lexicographical order. For example do the environment_keys in the boefje.json need to be specified before or after the scan_level? One boefje does it differently than the other boefjes, are we going to document this currently undocumented ordering and put time into "fixing" this?

and currently without that option it produces a lot of changes.

It only produces a lot of changes once. After that there is a strict order and it is automatically fixed. There will also be no noise in PRs if things accidentally move around.

[ammar92]

Looks good to me. I prefer to also use --no-sort-keys, because order of properties may also have semantic meaning

I saw that we tried to order some files in a certain way, but JSON is unordered and that makes it harder to enforce a non-lexicographical order. For example do the environment_keys in the boefje.json need to be specified before or after the scan_level? One boefje does it differently than the other boefjes, are we going to document this currently undocumented ordering and put time into "fixing" this?

I understand that JSON properties are unordered, but what does it exactly "fix" by messing around with (primarily) human written manifests, test stubs and schemas?

And they actually more or less follow the order as defined in the respective Python model. And for me it makes more sense to have some expectation in ordering. Instead of fixating on enforcing a rigid lexicographical order, we should prioritize readability and logical organization within the JSON files. This means having some level of expectation in ordering, such as placing essential identifiers like ID or title at the top followed by logically grouped properties such as consumes and produces. It doesn't matter if the order isn't strict, who cares if scan_level or environment_keys is two or three positions "off"? I don't think these minor order variations are problematic and worthy of putting a significant time and effort into fixing this.

and currently without that option it produces a lot of changes.

It only produces a lot of changes once. After that there is a strict order and it is automatically fixed. There will also be no noise in PRs if things accidentally move around.

The noise is usually whitespace and line endings. I understand that there will be minor changes after this, but I'm still struggling to see how reordering the properties (at least for non-generated documents) benefits us in the first place.

[dekkers]

I have added --no-sort-keys and ran pre-commit again on the original files.