Update grammar.py to work with the latest pyparsing.

[mikeurbach]

A few updates that allow this file to typecheck with mypy and the latest pyparsing:

• Don't type check enable_packrat, since it has incorrect type annotations, which disallow None as its first argument, even thought it is documented and has a specific behavior (unbounded cache size)
• Use <<= to "assign" to a Forward declaration.

[nick-knight]

I'm new to pyparsing, but I believe that using <<= changes the semantics, expanding the grammar. My intuition is that these changes actually match the original intent; perhaps this fixes some undiscovered bugs. On the other hand, since the current behavior has existed for at least three years, it is possible that such bugs are known and considered to be features. And it is also possible that this has been hiding more serious bugs elsewhere.

I don't know how comprehensive the testing is in this repository. This may be a case where we need to get it into the hands of downstream consumers for validation.

[richardxia]

I'm also new to pyparsing, and after quickly reviewing the docs, I'd agree with @nick-knight's assessment here. It also looks like there are several warnings that can be turned on that should explicitly catch the way we were using Forward declarations previously:

warn_name_set_on_empty_Forward - flag to enable warnings when a Forward is defined with a results name, but has no contents defined
warn_on_parse_using_empty_Forward - flag to enable warnings when a Forward is defined in a grammar but has never had an expression attached to it
warn_on_assignment_to_Forward - flag to enable warnings when a Forward is defined but is overwritten by assigning using '=' instead of '<<=' or '<<'

Since I'm really not sure what the behavior of the previous code was, I'd also agree with Nick that we should do some testing on real DTS files. The arith_expr change I'm less worried about, since I don't think the DTS we generate has any arithmetic expressions, but the property_values change is more interesting.

[nick-knight]

Suggest merging main into this branch and see if it resolves the CI failures. (Still might want further testing before merging this branch into main.)

[mikeurbach]

I can rebase master, but from local testing I know that this will allow mypy checks to pass, but there are hundreds of other lint errors from pylint.

[nick-knight]

I admit I don't understand how such a localized (and apparently syntactically valid) change could cause such widespread lint regression.

[richardxia]

I admit I don't understand how such a localized (and apparently syntactically valid) change could cause such widespread lint regression.

It's not this change that causes the lint regression; it's the fact that we don't pin the linting tool version, and in the years since this codebase was last tested, the linting tool has added many new lint rules.

[mikeurbach]

Yeah this change doesn't cause any regression I know of, I just mean that there are many other lint failures besides the mypy failures that this fixes.

[nick-knight]

Understood. And addressing these pylint issues risks changing the behavior of the code (just like the mypy failures).

I think, for now, we should just leave this open to community contributions.

For future reference, here is the pylint output I get. (Since versions don't seem to be pinned, let me add that this is with Python 3.11.3 and latest versions of packages on 21-July 2023.)

$ make test-lint
. venv/bin/activate && pylint pydevicetree
************* Module /home/nickk/pydevicetree/pylintrc
pylintrc:1:0: E0015: Unrecognized option found: no-space-check (unrecognized-option)
pylintrc:1:0: R0022: Useless option value for '--disable', 'print-statement' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'parameter-unpacking' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'unpacking-in-except' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'old-raise-syntax' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'backtick' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'import-star-module-level' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'apply-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'basestring-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'buffer-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'cmp-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'coerce-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'execfile-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'file-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'long-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'raw_input-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'reduce-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'standarderror-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'unicode-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'xrange-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'coerce-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'delslice-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'getslice-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'setslice-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'no-absolute-import' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'old-division' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'dict-iter-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'dict-view-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'next-method-called' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'metaclass-assignment' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'indexing-exception' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'raising-string' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'reload-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'oct-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'hex-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'nonzero-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'cmp-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'input-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'round-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'intern-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'unichr-builtin' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'map-builtin-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'zip-builtin-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'range-builtin-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'filter-builtin-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'using-cmp-argument' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'div-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'idiv-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'rdiv-method' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'exception-message-attribute' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'invalid-str-codec' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'sys-max-int' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'bad-python3-import' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-string-function' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-str-translate-call' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-itertools-function' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-types-field' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'next-method-defined' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'dict-items-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'dict-keys-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'dict-values-not-iterating' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-operator-function' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-urllib-function' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'xreadlines-attribute' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'deprecated-sys-function' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'exception-escape' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: R0022: Useless option value for '--disable', 'comprehension-escape' was removed from pylint, see https://github.com/PyCQA/pylint/pull/4942. (useless-option-value)
pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'long-suffix' (unknown-option-value)
pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'old-ne-operator' (unknown-option-value)
pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'old-octal-literal' (unknown-option-value)
pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'non-ascii-bytes-literal' (unknown-option-value)
pylintrc:1:0: W0012: Unknown option value for '--disable', expected a valid pylint message and got 'eq-without-hash' (unknown-option-value)
pylint: Command line or configuration file:1: UserWarning: Specifying exception names in the overgeneral-exceptions option without module name is deprecated and support for it will be removed in pylint 3.0. Use fully qualified name (maybe 'builtins.BaseException' ?) instead.
pylint: Command line or configuration file:1: UserWarning: Specifying exception names in the overgeneral-exceptions option without module name is deprecated and support for it will be removed in pylint 3.0. Use fully qualified name (maybe 'builtins.Exception' ?) instead.
************* Module pydevicetree.ast.directive
pydevicetree/ast/directive.py:32:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/directive.py:40:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/directive.py:44:42: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/directive.py:45:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/directive.py:46:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module pydevicetree.ast.helpers
pydevicetree/ast/helpers.py:20:27: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/helpers.py:23:31: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module pydevicetree.ast.node
pydevicetree/ast/node.py:92:19: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:93:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:123:31: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:125:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:127:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:129:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:186:69: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:218:12: W0719: Raising too general exception: Exception (broad-exception-raised)
pydevicetree/ast/node.py:218:28: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:352:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:358:12: W0719: Raising too general exception: Exception (broad-exception-raised)
pydevicetree/ast/node.py:358:28: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:423:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/node.py:452:4: W0237: Parameter 'source' has been renamed to 'dts' in overriding 'Devicetree.from_dts' method (arguments-renamed)
pydevicetree/ast/node.py:463:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
pydevicetree/ast/node.py:479:8: W0719: Raising too general exception: Exception (broad-exception-raised)
************* Module pydevicetree.ast.property
pydevicetree/ast/property.py:64:30: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/property.py:101:12: W0719: Raising too general exception: Exception (broad-exception-raised)
pydevicetree/ast/property.py:156:12: W0719: Raising too general exception: Exception (broad-exception-raised)
pydevicetree/ast/property.py:241:15: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/property.py:261:38: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/ast/property.py:262:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
************* Module pydevicetree.ast.reference
pydevicetree/ast/reference.py:76:0: R0022: Useless option value for 'disable', 'no-self-use' was moved to an optional extension, see https://pylint.readthedocs.io/en/latest/whatsnew/2/2.14/summary.html#removed-checkers. (useless-option-value)
************* Module pydevicetree.source.grammar
pydevicetree/source/grammar.py:46:0: C0325: Unnecessary parens after '=' keyword (superfluous-parens)
pydevicetree/source/grammar.py:19:18: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/grammar.py:20:0: E1121: Too many positional arguments for staticmethod call (too-many-function-args)
pydevicetree/source/grammar.py:27:8: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/grammar.py:30:13: E1121: Too many positional arguments for function call (too-many-function-args)
pydevicetree/source/grammar.py:31:12: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/grammar.py:32:8: E1121: Too many positional arguments for function call (too-many-function-args)
pydevicetree/source/grammar.py:32:8: E1123: Unexpected keyword argument 'delim' in function call (unexpected-keyword-arg)
pydevicetree/source/grammar.py:32:8: E1123: Unexpected keyword argument 'combine' in function call (unexpected-keyword-arg)
pydevicetree/source/grammar.py:42:11: E1121: Too many positional arguments for function call (too-many-function-args)
pydevicetree/source/grammar.py:46:13: E1123: Unexpected keyword argument 'content' in function call (unexpected-keyword-arg)
pydevicetree/source/grammar.py:55:18: E1121: Too many positional arguments for function call (too-many-function-args)
pydevicetree/source/grammar.py:57:55: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/grammar.py:76:8: E1121: Too many positional arguments for method call (too-many-function-args)
************* Module pydevicetree.source.parser
pydevicetree/source/parser.py:138:0: C0325: Unnecessary parens after '=' keyword (superfluous-parens)
pydevicetree/source/parser.py:140:0: C0325: Unnecessary parens after '=' keyword (superfluous-parens)
pydevicetree/source/parser.py:86:20: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:110:0: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/parser.py:113:0: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/parser.py:118:0: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/parser.py:128:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:130:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:133:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:136:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:143:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:145:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:148:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:150:34: C0209: Formatting a regular string which could be a f-string (consider-using-f-string)
pydevicetree/source/parser.py:168:21: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)
pydevicetree/source/parser.py:177:15: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/parser.py:177:15: E1123: Unexpected keyword argument 'parseAll' in method call (unexpected-keyword-arg)
pydevicetree/source/parser.py:189:11: E1136: Value 'grammar.node_definition.parseString(dts, parseAll=True)' is unsubscriptable (unsubscriptable-object)
pydevicetree/source/parser.py:189:11: E1121: Too many positional arguments for method call (too-many-function-args)
pydevicetree/source/parser.py:189:11: E1123: Unexpected keyword argument 'parseAll' in method call (unexpected-keyword-arg)
pydevicetree/source/parser.py:198:13: W1514: Using open without explicitly specifying an encoding (unspecified-encoding)

------------------------------------------------------------------
Your code has been rated at 7.92/10 (previous run: 7.92/10, +0.00)

make: *** [Makefile:37: test-lint] Error 30