Lexer

I keep pressing keys until I have a fully functional lexer. I wonder, how will it perform against a professional one like ANTLR4? 🤔

Using the black magic we developed for the regex engine we can easily create a state machine for any set of lexer rules.

On the journey to reinvent the wheel yet again while learning formal languages, it is time to build a lexer to generate old-school non-AI tokens