Stack overflow at ~20,000 characters

[rscarson]

I have a StringLiteral token with the following expressions on it:

    #[regex(r#"(?:/(?:\\.|[^\\/])+/[a-zA-Z]*)"#)] // Regex literal
    #[regex(r#"(?:"(?:(?:[^"\\])|(?:\\.))*")"#)] // " string literal "
    #[regex(r#"(?:'(?:(?:[^'\\])|(?:\\.))*')"#)] // ' string literal '

And encountering a string literal above ~20,000 characters stack overflows within logos generated code
Is there an error in one of these expressions?

[maciejhirsz]

Groups are never capturing in logos so you don't need ?:, and you don't need groups around |, so string literal can be rewritten as #[regex(r#""([^"\\]|\\.)*""#)]. If that still overflows try #[regex(r#""([^"\\]+|\\.)*""#)]. I believe it loops internally by doing tail call recursion so the greedily looping all non-escaped non-quote characters inside the group could help (but I have not tested it).

This is one of those cases that really needs the codegen rewrite (#291) into a loop with enum branching working as faux-goto instead of function calls.

[rscarson]

I had already rewritten it as #[regex(r#""([^"\\]|\\.)*""#)] with no luck but #[regex(r#""([^"\\]+|\\.)*""#)] did the trick!
Thanks!

[rscarson]

it still overflows at some point though, at a larger number

Should I leave this issue open for tracking?

[maciejhirsz]

Should I leave this issue open for tracking?

Ye, let's do that.

[0x2a-42]

You can also avoid a stack overflow by compiling with optimizations. Apparently opt-level 1 is enough for tail call optimization to occur.

[profile.dev]
opt-level = 1

There currently seems to be no way to selectively optimize the generated lex function. Maye rust-lang/rust#54882 can be used, if it is stabilized before the logos code generator is rewritten.