A modern lexer implementation with zero-copy UTF-8 processing, two-phase regex pattern parsing, and multi-path tokenization for handling ambiguous input. Part of the GrammarForge step-parser architecture.