public override tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult
Tokenizes a single line of Rust source code, handling #[...] / #![...] attributes, /// / //! doc comments, and 'lifetime annotations without misclassifying them as unclosed character literals.
| Parameter | Description |
|---|---|
line | Source line text. |
initialState | Lexical state at the start of line. |
LineScanResult with spans and endState.
import klyn.code
import klyn.code.rust
t = RustTokenizer()
res = t.tokenizeLine("#[derive(Debug)]")
assert res.spans[0].kind == TokenKind.ANNOTATION