Sorting out tabular with luatex

[jbezos]

2023-03-14

This post will be updated (regularly, I hope).

This issue will discuss how tabular is adapted to bidi in luatex (focused on LTR columns with RTL text, which is sometimes a requirement). This is one of the cases where RTL is particularly problematic, the other being math and graphics, because they mix RTL and LTR stuff.

1. tabular is executed inside a fake math. However, when we enter in math mode, all dirs are reset to \mathdir, which is usually TLT. Omega provided a companion to \mathdir (\nextfakemath) for those cases where we did not want it to be applied, so that the writing direction of the main text was left unchanged. This primitive is not available in luatex (admittedly, it’s somewhat hackish, but no alternative has been provided, afaik, and after all there are lots of fake maths in the TeXbook).

2 \halign is particularly difficult to deal with in a general way if the direction of columns is different to the text one, because they both are governed by \textdir. Note an explicit direction at the beginning of \halign applies only to the first column, so we can’t just do something like textdir TLT\halign{\textdir TRT ...}.

Furthermore, no hooks at the TeX level (\every...) are executed by \halign, except \everycr, which is of little help here. Nothing like to \everycell exists. Besides some bizarre ideas like making & active (perhaps no so bizarre, but anyway such a change have to be done at the LaTeX level), there are apparently just one option: patching the way the \halign preamble is built for every more or less relevant package. Attempts to fix the direction somehow with lua where promising at first but always led to a dead end.

On the other hand, all-RTL and all-LTR \halign’s are more straighforward, although there are still some issues. For example, when array is loaded, some directions are wrong.

3. LaTeX treats vertical rules, which should be placed visually, as part of the cell text and not as separate graphic elements. This explains why rules are “reversed”.

Currently, tabular, array, colortbl and tabularx seem to work. On the other hand, tabulary still misplaces vertical rules.

Related issues (closed, but part of the whole problem) are:

• bidi=basic cause strange result with incorrect alignement in tabular cells with Arabic content #214
• Wrong equation direction inline cases and array environment (with bidi=basic option) #167
• babel with bidi=basic option produce reversed math in first column of array and tabular #161
• Wrong equation direction in first column in array and tabular environment (with bidi=basic option) #158
• diagbox with babel RTL tabular issue #43
• Reversed arabic inside c r l columns with xcolor package table option  #28
• Tabular layout cause issue with array package  #15

[josephwright]

Good place to collect things up. On the 'fake math' part, for other reasons there are likely to be efforts to remove this.

[lvjr]

There is also a related issue with tabularray package (lvjr/tabularray#351), although it doesn't use \haligh.

[car222222]

There is another current project from Team LaTeX that may have some synergy with this.

Our current plan is to instrument LaTeX array-like structures with some kind of hooks; at first we are looking only at "displayed math", starting with those from amsmath and a few other packages.

Therefore it would be useful to know more about the locations where such hooks need to be placed in order to at least help the support of bidi typesetting.

[car222222]

Please can we have more explanation of this:
"LaTeX treats vertical rules, which should be placed visually, . . ."

What exactly is meant by "placed visually"?

Can you please give some ideas, at least, on how these should better be coded. Thanks!

[jbezos]

@car222222 Let’s consider an l-column:

\hskip\tabcolsep
{\hskip 1sp\ignorespaces\@sharp\unskip\hfil}
\hskip\tabcolsep
\hskip -.5\arrayrulewidth
\vrule width\arrayrulewidth
\hskip -.5\arrayrulewidth

There are 4 logical element in a row:

• The column separation (lines 1, 3)
• The cell alignment (line 2)
• The cell text (line 2)
• The cell rule, which is usually the column rule (lines 5-7)

In a tabular with mixed directions (columns → LTR, cells → RTL), direction should be:

• text RTL, for sure
• alignment, I think RTL, too
• column separation, not sure, but it’s symmetrical
• rules, LTR

[jbezos]

@car222222 I’ve made a few patches for the standard tabular, as well as for array and colortbl. Before proposing a change, I’d like to see what other related packages do.

[Udi-Fogiel]

@jbezos The following might interest you, I think it is better than the fake math, although the fake math code is cleaver.

\documentclass{article}

\directlua{
local function process_alignment(head)
    for n in node.traverse(head) do
        if n.id == node.id("hlist") or n.id == node.id("vlist") then
            n.list = process_alignment(n.list)
        end
        if n.id == node.id("hlist") and (n.subtype == 4 or n.subtype == 5) then
            n.dir = "TRT"
        end
    end
    return head
end
luatexbase.add_to_callback("vpack_filter", process_alignment, "foo")
}

\begin{document}

\begin{tabular}{||lc|r||}
 \hline
Month & commodity & price \\
January & notebook & \$ 700 \\
February & skateboard & \$ 100 \\
July & yacht & k\$ 170 \\
\hline
\end{tabular}

\end{document}

[jbezos]

@Udi-Fogiel Interesting idea, which I'm going to explore.

[jbezos]

@Udi-Fogiel So far, all my tests work. I don't like to make changes hastily, so I'll continue exploring and testing for a few more weeks. I never liked \bbl@nextfake, which is very hackish, and your solution is more elegant and, very likely, more reliable.