KlynTokenizer

classin packageklyn.code.klyn

public class KlynTokenizer extends CodeTokenizer

Object

└ CodeTokenizer

└ KlynTokenizer

Specialized CodeTokenizer for Klyn source files (.kn, .wp).

Extends CodeTokenizer with support for:

Contextual query: / query ( KQL blocks tracked across lines via LexicalState.queryIndent
Regular expression literals (/pattern/gimsuy) disambiguated from division operators -Formatted strings (f"...", fr"...", f"""...""") with nested {...} interpolation
Method/constructor declaration highlighting (public myMethod(...) as ReturnType:)

import klyn.code
import klyn.code.klyn

tokenizer = KlynTokenizer()
res = tokenizer.tokenizeLine("public compute(x as Int) as Int:")
assert res.spans.size >= 5

Inherited Properties

propertyInherited Properties from Object: type

Constructors

Modifier and Type	Member	Description
public	KlynTokenizer `KlynTokenizer()`	Constructs a `KlynTokenizer` initialized with `KlynLanguage.profile()`.

Methods

Modifier and Type	Member	Description
public override	classifyIdentifier `classifyIdentifier( token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean ) as TokenKind`	Classifies an identifier in Klyn source code, adding recognition for Klyn method declarations (`public compute(x as Int)`) even when `def` is omitted.
public override	nextLineState `nextLineState(line as String, initialState as LexicalState = null) as LexicalState`	Advances the lexical state across `line` without allocating `TokenSpan` objects, preserving Klyn `query:` block indentation tracking.
public	scanRegexLiteralEnd `scanRegexLiteralEnd( line as String, index as Long, previousKind as TokenKind, previousWord as String ) as Long`	Checks whether a `/` at `index` in `line` can start a `/pattern/flags` regular expression literal rather than a division operator, and returns the exclusive end index of the regex literal (or `-1` if it is not a regex literal).
public override	tokenizeLine `tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult`	Tokenizes a single line of Klyn code with contextual KQL `query:` state and regex literal support.

Inherited Methods

methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml