public tokenize(source as String) as ArrayList<Token>
Tokenizes a complete multi-line source document into materialized Token instances.
| Parameter | Description |
|---|---|
source | Full source code text. |
Ordered list of Token instances with line, column, and character offsets.
import klyn.code
import klyn.code.klyn
tokenizer = KlynTokenizer()
tokens = tokenizer.tokenize("def add(a as Int, b as Int) as Int:\n return a + b")
assert tokens.size > 0