[Node] Measuring svg output children

[ShaMan123]

Hi,
I've been working this past week on a better way to interact with math in react-native (#342 ) react-native-math-view.
V3 has unlocked options that have been near to impossible to implement natively with proper responsiveness.
One of them is editing math.
I've been struggling with this for a few days. I went through the files again and again and am near to giving up 😓 (for the near future).
I need a way to measure the layout of children relative to the root parent svg output node (in node runtime environment). Then I could compute the layout, intercept touches and change the tex input accordingly (with a custom tex keyboard i.e.).

You can see what I did in MathjaxAdaptor.toSVGArray
My approach was to walk up the svg tree and concat all transformations for each deep child, then apply the layout onto it.
I seem to be missing something because if there are lower/upper chars, fractions etc. my calculation are wrong. See below.

Is there any better way of achieving this?

Any help would be much appreciated.

Thanks,

[ShaMan123]

OK, my bad.
I forgot to reverse the matrix.
Still, is there a better approach?
Can you guys point out touch handling in the source code, it might come in handy. Especially finding the node that was hit and it's latex expression (i.e. I touched an s is it part of cos or sin, is it a part of a fraction?)
I can walk up the tree, but there are too many cases to handle.

[dpvc]

I think there is another approach to this that doesn't require you to know anything about the bounding boxes of individual terms or their locations. You can put an event listener on the outermost SVG element itself, and that will get click events for any element that that it contains. You can use the event's target property to identify which element was clicked (you don't have to determine coordinates yourself).

The difficulty with this approach in SVG output is that, because the glyphs are rendered as filled paths rather than text, a click would only be registered only if it occurred on the "ink" of the letter (not the letter's background). This problem can be overcome by adding transparent "hit boxes" into the structure that are the size of the character bounding boxes.

Here is one example of that:

MathJax = {
  startup: {
    ready() {
      const SVGWrapper = MathJax._.output.svg.Wrapper.SVGWrapper;
      const createSVGnode = SVGWrapper.prototype.createSVGnode;
      SVGWrapper.prototype.createSVGnode = function (parent) {
        createSVGnode.call(this, parent);
        const {h, d, w} = this.getBBox();
        this.adaptor.append(this.element, this.svg('rect', {
          'data-hitbox': true, fill: 'none', stroke: 'none', 'pointer-events': 'all',
          width: this.fixed(w), height: this.fixed(h + d), y: this.fixed(-d)
        }));
        return this.element;
      }
      MathJax.startup.defaultReady();
    }
  },
  options: {
    renderActions: {
      events: [170, (doc) => {for (const math of doc.math) {addClickHandler(math)}}, addClickHandler]
    }
  }
};
function addClickHandler(math) {
  math.typesetRoot.firstChild.addEventListener('click', (event) => {
    let target = event.target;
    if (target) {
      while (target && !target.id && target.nodeName !== 'svg') {
        target = target.parentNode;
      }
      if (target.nodeName === 'svg') {
        target = target.lastChild.firstChild;
      }
      console.log(target.getAttribute('data-mml-node') + ' ' + target.id);
    }
  });
}
</script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"></script>

Here, we we use the startup's ready() function to tie into the SVG output's createSVGnode() method that is used by all elements to create their basic SVG <g> element. The new code calls the original method, and inserts a transparent hit box of the correct size into the group. This makes it possible to collect the clicks on the various elements that make up the SVG image.

Then a render action is added at priority 170 (so that it follows the typesetting at priority 150) that attaching the click handler to the typeset math.

The click handler itself uses the click target to move up the tree to the closest container that represents an item in the math tree that we care about. In this case, the ones we care about are marked with an id attribute. That is done in the TeX using the \cssId{} macro. So the following example shows how that might work:

<!DOCTYPE html>
<html>
<head>
<meta charset="UTF-8" />
<title>SVG events</title>
<script>
function addClickHandler(math) {
  math.typesetRoot.firstChild.addEventListener('click', (event) => {
    let target = event.target;
    if (target) {
      while (target && !target.id && target.nodeName !== 'svg') {
        target = target.parentNode;
      }
      if (target.nodeName === 'svg') {
        target = target.lastChild.firstChild;
      }
      console.log(target.getAttribute('data-mml-node') + ' ' + target.id);
    }
  });
}
MathJax = {
  startup: {
    ready() {
      const SVGWrapper = MathJax._.output.svg.Wrapper.SVGWrapper;
      const createSVGnode = SVGWrapper.prototype.createSVGnode;
      SVGWrapper.prototype.createSVGnode = function (parent) {
        createSVGnode.call(this, parent);
        const {h, d, w} = this.getBBox();
        this.adaptor.append(this.element, this.svg('rect', {
          'data-hitbox': true, fill: 'none', stroke: 'none', 'pointer-events': 'all',
          width: this.fixed(w), height: this.fixed(h + d), y: this.fixed(-d)
        }));
        return this.element;
      }
      MathJax.startup.defaultReady();
    }
  },
  options: {
    renderActions: {
      events: [170, (doc) => {for (const math of doc.math) {addClickHandler(math)}}, addClickHandler]
    }
  }
};
</script>
<script src="https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-svg.js"></script>
</head>
<body>

$$
\cssId{7}{\cssId{10}{\cssId{1}{x}^\cssId{9}{2}}\cssId{2}{+}\cssId{3}{1}\over \cssId{13}{\sqrt{\cssId{4}{x}}}
\cssId{5}{-}\cssId{6}{1}}\cssId{11}{+}\cssId{12}{z}$$

</body>
</html>

I'm assuming that your editor using some sort of expression tree as its internal format and that you generate the LaTeX from that as the expression is edited. (LaTeX itself would be a horrible internal format for this sort of thing.) In that case, you can insert the \cssId{} macros as you build the LaTeX from the expression tree, and the ids themselves could be used to identify the original locations in the expression tree. For example, you could maintain an array of pointers to the nodes in your expression tree, and the ids could be the indices into that array. There are many possible approaches. I discussed this idea some years ago on the MathJax Users Forum. You can see an example like the one above there, and some other discussion of how the css id can be used to encode your internal tree structure.

[ShaMan123]

Thanks for the thorough response!
It seems you had in mind that I'm using Mathjax in a web environment, but that's not the case.
Will click handlers work in node environment? on a LiteAdaptor?
I read your response from the forum and will look into it further.
Got what you said about the representation, a good and important advice.
If, as I suspect click handlers won't work on LiteAdaptor I'm thinking your logic is pointing towards creating another OutputJax for react-native based on the SVG OutputJax.... 😬

[jacobp100]

I'm doing a similar thing (for RN too). I iterated through the LiteAdaptor and looked at the transform property to get the origin for each element

See #190

[ShaMan123]

@jacobp100 let's join efforts, what say you?
I see you're more knowledgeable in Mathjax than I am.
Take a look at my solution.
It doesn't use react-native-svg because:
a. I didn't manage to get it working with SVG output
and
b. because of performance.
But I started looking into it once more a week back.

[dpvc]

Will click handlers work in node environment? on a LiteAdaptor?

I'm not sure quite what you mean by that. The LiteDOM is not a setting where there is a mouse or clicking. It's not a real DOM. But if you added an onclick attribute (with adaptor.setAttribute('onclick', 'code-string'), that would be including in the serialization of the node, if that's what you mean.

But I think you don't have to use the event handler on the MathJax element itself, but could use React Native support for touch events to do a similar thing. I don't have any experience with React Native, but some quick searching around suggests that TouchableOpacity might be the thing you need. For example, this post has some examples that might help. The TouchableOpacity onPress event handler should give you the event object whose event.target property is used in the click handler I gave in the code above. I suspect you should be able to get something like that to work.

I don't think you should need to make a new output jax for this.

[ShaMan123]

Indeed that is my approach. The only problem is understanding what symbol is being hit, which is the reason I opened this issue.

[jacobp100]

@ShaMan123 sorry, I meant to get back to you. My use-case specifically handles three layers, and I flatten these layers into a single svg path data string to make it easier on react-native-svg

You can see my hacked-together code here

Note this probably doesn't handle MML generally, it only needs to handle the MML output from my input editor

Also note it's gathering other data in addition to the path data

[ShaMan123]

@jacobp100 Thanks. I'll look into it