public tokenize(source as String) as ArrayList<Token>
Tokenizes a complete multi-line source document into materialized Token instances.
| Parameter | Description |
|---|---|
source | Full source code text. |
Ordered list of Token objects.
import klyn.code
p = LanguageProfile("klyn", "Klyn").withKeywords(["def", "return"])
t = CodeTokenizer(p)
tokens = t.tokenize("def f():\n return 1")
assert tokens.size > 0