Format refactor

[mmatera]

During the last weeks, I was working on refactoring the implementation of
format routines in Mathics. To give a little bit of context, let's remember that
the evaluation process has three steps:

• Parsing the input.
• Evaluate the obtained M-Expression.
• Format the M-Expression to produce an output.

The last step actually has several sub-steps:

• Evaluate Format rules to the expression.
• Evaluate$Post and $PrePrint hooks.
• Evaluate MakeBoxes to produce a "Boxed" Expression.
• Parse the Boxed expression to produce an output in a given format.

Here, the first two steps are just "regular" evaluations that produce M-Expressions following the standard
evaluation process.
On the other hand, the third step is a little different: MakeBoxes produces expressions that
usually are not "reevaluated", but something that would be equivalent to a "formatted" String.
Hence, we can think about these "boxed" expressions as the way in which WL stores formatted text.
Such a format is in principle agnostic about how it is going to be translated into a printed plane (ascii/unicode) text
or translated to a MathML/LaTeX/SVG/whatever file format, which is performed in the last step.

Before Mathics 4.1.0dev,

• there was not a distinction between Expression and BoxExpression objects. So, if we have for example an expression of the form

F[RowBox[{...}]]

the evaluation method was going to try to evaluate each leaf of the argument, which in general does not make any sense.

• The other problem with the organization of the formatting code in 4.0 and before was that the formatting code was spread in the core and in the builtin module.
  So, any attempt to change the behaviour of the formatting requires to introduce changes in several modules.

• Moreover, inside mathics.core, the current architecture has a lot of crossed references with the builtin modules, and among its different parts
  (strings require expressions, and boxes rules defined in mathics.builtin.inout, etc) requiring a lot of local imports.

• The interface of Elements was unnecesarily complex and misleading: in most of the cases, there is no need to ask to an Integer how to convert it into a LaTex form. Moreover, we shouldn't, because an Integer shouldn't be a part of a Box expression.

After the last changes in #311, I think that this situation was improved: Now, most of the format routines are localized in mathics.builtin.inout and mathics.builtin.box.inout. Still, there are some pieces of code that could be moved. In particular,

• Element.format shoudn't be a member method, but a function, and probably belongs to a different module.
• same applies for Element.do_format: probably it shouldn't be member functions, but functions in the same formatting module.
• Atom.atom_to_boxes are another method that could be removed from the interface of Atom, and moved to such a module.

Another aspect to improve is the class hierarchy: Differently from the other atoms, String is an element that can be a part (or the whole) of a Box Expression.
To make the new changes work, (without introducing several other changes) I had to "duplicate" the class introducing a _BoxedString as a daughter class of BoxExpression. However, I am not very satisfied with this solution. Worst, a BoxString shouldn't be a BoxExpression but a BoxElement.
Probably, a better hierarchy would be

                            BaseElement
			        |
	----------------------------------------------------------------
	|	     |                           |                      |
   Expression      Atom                     BoxElement              BuiltinExpression
		     |                           |                       |
		     |                 |-----------------|               |
   -----------------------------       |                 |               |
           |         |         |       |                 -----------------
           |         |         |       |                         |
	 Symbol    Integer   String    |                   BoxExpression
                               |       |
                               ----------
                                   |
                               BoxedString

[rocky]

Good explanation and write up.

As for the organization of modules I would appreciate it if we could follow the organization as how these are described in WMS. See

• https://reference.wolfram.com/language/guide/DefiningCustomNotation.html
• https://reference.wolfram.com/language/guide/LowLevelNotebookStructure.html

In particular, I'd make Low-Level Definitions be its own modul, pull out MakeBoxes, MakeExpression Format, ToBoxes and StripBoxes from inout (or the ones that are there). It looks like TagBox, TemplateBox, InterpretationBox, FormBox, ErrorBox, are part of a Low-Level Notebook Structure.

[rocky]

As for the class hierarchy we now have ListExpression as a subclass of Expression. Ideally there would be a base class for Expression that both ListExpression and Expression and BuiltinExpression would subclass.

[mmatera]

The past weekend I have finished merging the first part of this refactor. Thanks @rocky for the feedback!

Now, the class hierarchy is

                            BaseElement
			        |
	----------------------------------------------------------------
	|	     |                                                 |                
   Expression      Atom                BoxElementMixin          BuiltinExpression
		     |                       |                         |
		     |            ---------------------                |
   -----------------------------  |                    |               |
           |         |         |  |                    ----------------
           |         |         |  |                           |
	 Symbol    Integer   String                      BoxExpression
                                     
                              

Notice that now _BoxedString is not needed any more. Also, several of the the old boxes_to_ methods are delegated in BoxElementMixin into mathics.core.formatter.boxes_to_format.

Still, there are several things in the TODO list:

• Move all the other remaining boxes_to_ to the corresponding mathics.format module (done in boxes_to_ -> boxes_to_format. Improving support for encodings. #539)
• Implement the text conversion of special characters using the tables in mathics_scanner.
• Add a field in the routines included in mathics.format.text that control the codepage used in the conversion from Boxes to strings. Use these routines in ToString (this part was implemented in boxes_to_ -> boxes_to_format. Improving support for encodings. #539).
• Use the mathics.format modules in the Export routines (partially done in boxes_to_ -> boxes_to_format. Improving support for encodings. #539).
• Improving the compatibility of MakeBoxes with WL. (See for instance the commented doctests).
• Improve the pytests.
• Improve documentation on the related modules.
• See if we can improve the implementation of ToExpression.
• Improve the implementation of the backward compatibility use of boxes_to_ methods (done in Improving backward compatibility support for boxes_to_ methods in BoxExpression subclasses. #540).
• ...