0.11.0

New face, rehauled API

Logos has a new cute logo 🤓.

There is a number of breaking changes this release, aimed at reducing API surface and being more idiomatic, while adding some long awaited features, like the ability to put slices of the source or arbitrary values returned from callbacks directly into a token.

Most of the changes related to the #[derive] macro will trigger a compile error message with an explanation to aid migration from 0.10. Those will be removed in the future.

API changes

• 🔨 [breaking] LOGOS NO LONGER HANDLES WHITESPACE BY DEFAULT. The #[trivia] attribute has been removed. Whitespace handling is easily added by defining #[regex(r"[ \n\t\f]+", logos::skip)] on any token enum variant. Putting it along #[error] is recommended.
• 🔨 [breaking] Lexer no longer has the advance method, or publicly visible token field. Instead Lexer now implements the Iterator trait and has a next method that returns an Option of a token.
• 🔨 [breaking] #[end] attribute was removed since it became obsolete.
• 🔨 [breaking] #[regex = "..."] and #[token = "..."] definitions are no longer valid syntax and will error when used in this way. Those need to be transformed to #[regex("...")] or #[token("...")] respectively.
• 🔨 [breaking] Callbacks are now defined as a second parameter to either #[regex] or #[token]. Those can be either paths to functions defined elsewhere (#[token("...", my_callback)]), or inlined directly into the attribute using closure syntax (#[token("...", |lex| { ... })]).
• 🔨 [breaking] Extras trait has been removed. You can still associate a custom struct to the Lexer using #[logos(extras = MyExtras)] with any type that implements Default, and manipulate it using callbacks.
• 🔨 [breaking] #[callback] attribute was removed since it was just polluting attribute namespace while offering no advantages over attaching callbacks to #[regex] or #[token].
• 🔨 [breaking] Lexer::range has been renamed to span. span and slice methods continue to return information for the most recently returned token.
• ✨ [new] Lexer has a new spanned method takes the ownership of Lexer and returns an interator over (Token, Span) tuples.
• ✨ [new] Callbacks can return arbitrary values (or Options/Results of values) that can be put into token enum variants. Currently only a single value in a tuple-like variants is supported (Token::Variant(T)).
• ✨ [new] Logos will automatically populate Token::Variant(&str) with a matching str slice if no callback is provided.
• ✨ [new] Callback return values can be used to skip matches using the Skip type. It's also possible to dynamically skip matches using the Filter type.

Internals

• 🚀 Generated code is now more likely to produce optimized jump tables for complex branches.
• 🚀 Logos can now stack multiple boolean lookup tables into a single table, reducing the memory cost of using the tables by a factor of 8, and improving cache locality.
• 🚀 For narrow range boolean branches Logos can now produce a lookup table packed into a single u64.