[ANNOUNCEMENT] MBINCompiler to start outputting MXML instead of EXML

[monkeyman192]

With the latest changes to NMS to improve mod support, the game can now load EXML files within a mod directory.
These EXML files however differ from the ones which are currently generated by MBINCompiler in two important ways:

1. The actual xml structure is actually expected to be the same as MXML - We found that when giving the game an MBINCompiler EXML file, it would crash, but if we reformatted it to look like MXML, it would work.
2. The EXML file doesn't need to be a complete file, but it can be just the fields which are modifying the vanilla file.

Because of these two points, this means that the EXML files produced by MBINCompiler are actually not compatible with the EXML files the game expects.

Why not continue having MBINCompiler produce EXML files but with the correct MXML formatting?

If we did this, then we'd not be able to know if an EXML file is a fragment or an entire file. If it were an entire file this would be fine, but if it were a fragment, then passing this to MBINCompiler would cause an error. We could get around this by potentially adding some extra meta to the fragment so that it's understood that it's a fragment, but in this case the EXML file would not be expected to be able to recompiled to an MBIN file anyway.

I am of the opinion that changing MBINCompiler to produce MXML files and then create some extra tooling which would allow EXML files to be generated (see attached quick script I whipped up to at least partially do this) is the best way to move forward with this, but I'd like community feedback before implementing any changes.

I have made the code changes in this PR for those who would like to download the artefact and try it out.

EXML generator demo

# Prototype EXML generator from two MXML files.
__author__ = "monkeyman192"
__version__ = "0.0.1"

import argparse
from collections import namedtuple
import os.path as op

from lxml import etree


Property = namedtuple("Property", ["name", "value"])


class Tree:
    def __init__(self, xpath = None):
        self.xpath = xpath
        self.property = None
        self.children: dict[str, Tree] = {}

    def add(self, seq: list[str], property: Property):
        top = seq[0]
        if top not in self.children:
            t = Tree(top)
            self.children[top] = t
            if (rest := seq[1:]):
                t.add(rest, property)
            else:
                t.property = property
        else:
            t = self.children[top]
            if (rest := seq[1:]):
                t.add(rest, property)

    def as_element(self):
        if self.xpath is None:
            # Root level. Bit of a weird case since we'll only have one child.
            root = list(self.children.values())
            if len(root) > 0:
                root_element = root[0].as_element()
                return root_element
            return None
        else:
            if self.property is None:
                element_ = tree.find(self.xpath)
                element = etree.Element("Property")
                for key, value in element_.items():
                    element.set(key, value)
            else:
                element = etree.Element("Property", name=self.property.name, value=self.property.value)
            for child in self.children.values():
                element.append(child.as_element())
            return element

    def __str__(self):
        if self.property is not None and len(self.children) == 0:
            return str(self.property)
        else:
            return str({x: str(y) for x, y in self.children.items()})


def property_from_element(element: etree.Element):
    return Property(element.get("name"), element.get("value"))


def parent_xpaths(element):
    xpaths = []
    if (parent := element.getparent()) is not None:
        xpaths.append(mod_tree.getelementpath(parent))
        xpaths.extend(parent_xpaths(parent))
    return xpaths


def parse_node(mod_node, vanilla_node):
    for i, child in enumerate(mod_node):
        name = child.get("name")
        vanilla_element = vanilla_node[i]
        mod_value = child.get("value")
        vanilla_value = vanilla_element.get("value")
        if mod_value != vanilla_value:
            xpath = mod_tree.getelementpath(child)
            p_xpaths = [xpath, *parent_xpaths(child)]
            p_xpaths.reverse()
            data_tree.add(p_xpaths, Property(name, mod_value))
            print(f"{name} is different: Modded: {mod_value}, Original: {vanilla_value}")
        parse_node(child, vanilla_element)


if __name__ == "__main__":
    parser = argparse.ArgumentParser(
        prog="EXML Patch generator",
        description="Simple prototype script to generate exml files from two mxml files")

    parser.add_argument("vanilla", help="The unmodified file path")
    parser.add_argument("modded", help="The modded file path")
    parser.add_argument("-o", "--output", help="The destination path")

    args = parser.parse_args()

    with open(args.vanilla) as f:
        tree = etree.parse(f)
        root = tree.getroot()
        template = root.get("template")

    with open(args.modded) as f:
        mod_tree = etree.parse(f)
        mod_root = mod_tree.getroot()
        mod_template = mod_root.get("template")

    assert template == mod_template
    title = f"Comaparing {template}"
    print(title)
    print("-" * len(title) + "\n")

    data_tree = Tree()

    parse_node(mod_root, root)

    data_element = data_tree.as_element()

    if data_element is not None:
        if args.output:
            dst = args.output
        else:
            basename = op.splitext(op.basename(args.vanilla))[0]
            dst = basename + ".EXML"
        with open(dst, "wb") as f:
            f.write(etree.tostring(data_element, pretty_print=True, xml_declaration=True, encoding="utf-8"))
    else:
        print("nothing to do...")

[cmkushnir]

Agree, mbinc should focus on converting: .mbin <=> .NET Object <=> .mxml (full).
Another tool should be responsible for creating the .exml diffs from two .mxml files (game mxml and mod mxml).
NMSMB will focus on creating full .mbin's; i assume amumss will (initially) focus on creating full .mxml files.
There are currently no plans for NMSMB to support generating .exml fragments, it will focus on creating full mbin's. If users want they can then take the .mbin's and convert to .mxml using mbincompiler, then use the 'other tool' to create exml files. It may provide a button users can toggle to choose whether to save .mbin's or .mxml's.