Tokenizer

interfacein packageklyn.code

public interface Tokenizer

Tokenizer

Common contract implemented by all source code and markup tokenizers in klyn.code.

A Tokenizer supports both: 1. **Incremental line-by-line scanning** (tokenizeLine and nextLineState) for real-time syntax highlighting in terminal and GUI code editors without allocating substrings or rescanning the entire file. 2. **Full-document tokenization** (tokenize) producing materialized Token objects with line/column coordinates for parsers, refactoring tools, and metrics scanners.

import klyn.code
import klyn.code.klyn

tokenizer = KlynTokenizer()
result = tokenizer.tokenizeLine("public class Demo:")
assert result.spans.size > 0

Methods

Modifier and Type	Member	Description
public	isCodePosition `isCodePosition(line as String, column as Long, initialState as LexicalState = null) as Boolean`	Checks whether the zero-based `column` in `line` is a code position (outside comments and string/character/regex literals).
public	nextLineState `nextLineState(line as String, initialState as LexicalState = null) as LexicalState`	Computes only the end-of-line `LexicalState` after scanning `line` from `initialState`, without allocating `TokenSpan` objects when possible.
public	profile `profile() as LanguageProfile`	Returns the declarative `LanguageProfile` associated with this tokenizer.
public	tokenize `tokenize(source as String) as ArrayList<Token>`	Tokenizes a complete multi-line source document into materialized `Token` instances.
public	tokenizeLine `tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult`	Tokenizes a single source line starting from `initialState` and returns the resulting `TokenSpan` list together with the end-of-line `LexicalState`.