public static fromSource(tokenizer as Tokenizer, source as String, skipTrivia as Boolean = true) as SourceTokenCursor
Tokenizes source with tokenizer and returns a ready-to-use SourceTokenCursor.
| Parameter | Description |
|---|---|
tokenizer | Language tokenizer to apply. |
source | Full source code text. |
skipTrivia | true to skip whitespace and comments. |
New SourceTokenCursor instance.
import klyn.code import klyn.code.klyn cursor = SourceTokenCursor.fromSource(KlynTokenizer(), "class User:\n pass") assert cursor.next().text == "class" assert cursor.next().text == "User"