Lexer
A lexer breaks the source-code character stream into a token sequence (keywords, identifiers, literals, operators) — the first step of compilation.
Purpose
Structure the character stream into tokens, providing input for parsing.
Mechanism
Matches token patterns with regexes or finite automata. Skips whitespace and comments. Handles keyword vs identifier ambiguity. Error recovery skips illegal characters and continues. Flex/ANTLR generate lexers.
Use cases
- Compiler implementation
- Language tooling
- Parsers