Accept more character types for GenericToken.Identifier

[jbartlau]

I'm currently trying to write a lexer/parser for a custom script language. However, identifiers may contain alphanumeric characters (but not starting with a number) as well as "_", ".", ":" and a number of other characters.

Is there any way to extend the parsing logic for the GenericToken.Identifier without moving to the RegEx parsing logic (which I'm trying to avoid for obvious performance reasons)?

EDIT: probably I'd need to write a generic lexer extension then?

[b3b00]

Hello @jbartlau,

There are 2 ways to get it done.

• First as you said using generic lexer extension.
• Second use a custom identifier.

For you definition it would look like this :

    public enum Discussion468Lexer
    {
// starting with a letter or '_' , '.' or ':'
// next the same chars plus digits  
        [CustomId("_.:a-zA-Z","_.:0-9a-zA-Z")]
        INDENTIFIER
    }

here is a simple test case

 public class Discussion468Tests
    {



        private static void CheckId(ILexer<Discussion468Lexer> lexer,string source, bool isOk)
        {
            
            var lexingResult = lexer.Tokenize(source);
            if (isOk)
            {
                Check.That(lexingResult).IsOkLexing();
                var tokens = lexingResult.Tokens;
                Check.That(tokens).IsNotNull();
                Check.That(tokens).CountIs(2);
                var idToken = tokens[0];
                Check.That(idToken.TokenID).IsEqualTo(Discussion468Lexer.INDENTIFIER);
                Check.That(idToken.Value).IsEqualTo(source);
            }
            else
            {
                Check.That(lexingResult).Not.IsOkLexing();
            }
        }

        [Fact]
        public static void Test468()
        {
            var buildResult = LexerBuilder.BuildLexer<Discussion468Lexer>();
            Check.That(buildResult).IsOk();
            var lexer = buildResult.Result;
            Check.That(lexer).IsNotNull();
            CheckId(lexer,"_:word1234",true);
            CheckId(lexer,"word_1234.otherword",true);
            CheckId(lexer,"1_:word1234",false);            
        }
    }

you can find the lexer and test case on commit ab0e2c4

[jbartlau]

Hi @b3b00, thanks for the swift reply. I discovered the custom identifier as well and currently use it like so in my code

[Lexeme(GenericToken.Identifier, IdentifierType.Custom, "[a-zA-Z]", @"[0-9a-zA-Z \.\-_\$%&#/]")] IDENTIFIER = 4,

However I was wondering if the usage of such custom identifiers comes with a major performance hit due to the RegEx matching involved? Performance is important for my use case,

I didn't succeed with a lexer extension, my experiment looked like

        public static Action<ExpressionToken, LexemeAttribute, GenericLexer<ExpressionToken>> Extensions()
        {
            return (ExpressionToken token, LexemeAttribute lexem, GenericLexer<ExpressionToken> lexer) => 
            { 
                if (token == ExpressionToken.IDENTIFIER)
                {
                    var fsmBuilder = lexer.FSMBuilder;
                    fsmBuilder.GoTo(GenericLexer<ExpressionToken>.in_identifier).RangeTransition('0', '9');
                }
            
            };
        }

which worked fine for "Test1" but not for "Test12", as it only swallows one number where I'd need an arbitrary sequence of characters, numbers and special characters. I also tried using a TransitionPrecondition but didn't get it to work as required. It's probably me, I'm quite new to the topic of lexers, parsers and state machines 🙂. Thanks for your thoughts anyway - if the custom identifier would be a viable alternative without causing too much performance drain that looks like a good alternative.

[b3b00]

hello @jbartlau , there is no regex matching. In fact the lexer builder parse the pattern and build an FSM as you would with a lexer extension. So there is no particular perfomance concern.

Regarding your experiment with lexer extension, I agree that is not the easier part of generic lexer. You've just tried to reuse the existing FSM for your need but you have to write your own FSM from scratch to match your pattern.

I've commited a sample with test cases. see 71145ac

I would recommend that you use the customId instead of an extension that is not that easy to understand.

[jbartlau]

@b3b00 Thanks again - that's brilliant to hear. And thanks for your great work here! Appreciated.