public tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult
Tokenizes a single source line starting from initialState and returns the resulting TokenSpan list together with the end-of-line LexicalState.
| Parameter | Description |
|---|---|
line | Single source line (without trailing newline). |
initialState | Lexical state inherited from the previous line, or null for LexicalState.NORMAL. |
LineScanResult containing the ordered TokenSpans and endState.
import klyn.code
import klyn.code.klyn
tokenizer = KlynTokenizer()
scan = tokenizer.tokenizeLine("x as Int = 42", LexicalState.NORMAL)
assert scan.spans.size >= 4