Token - Klyn API

Token

classin packageklyn.code

public class Token extends Object

Object

└ Token

Represents a materialized lexical token with its text and source coordinates.

Unlike TokenSpan (which is optimized for zero-allocation line rendering), Token carries its sliced text, zero-based line and column numbers, and document character offsets for full-document analysis, refactoring, and parsing.

import klyn.code

tok = Token(TokenKind.KEYWORD, "class", 0, 0, 0, 5)
assert tok.text == "class"
assert tok.kind == TokenKind.KEYWORD

Properties

Modifier and Type	Member	Description
public readonly property	column `column as Long`	Zero-based column number where the token starts.
public readonly property	endOffset `endOffset as Long`	Zero-based exclusive end offset in the full source text.
public readonly property	kind `kind as TokenKind`	Lexical classification of the token.
public readonly property	line `line as Long`	Zero-based line number where the token starts.
public readonly property	startOffset `startOffset as Long`	Zero-based inclusive start offset in the full source text.
public readonly property	text `text as String`	Exact source substring of the token.

Inherited Properties

propertyInherited Properties from Object: type

Constructors

Modifier and Type	Member	Description
public	Token `Token( kind as TokenKind, text as String, line as Long = 0, column as Long = 0, startOffset as Long = 0, endOffset as Long = 0 )`	Constructs a materialized `Token`.

Methods

Modifier and Type	Member	Description
public	isComment `isComment() as Boolean`	Returns `true` when this token is a comment (`COMMENT` or `DOC_COMMENT`).
public	isLiteral `isLiteral() as Boolean`	Returns `true` when this token is a literal.
public	isTrivia `isTrivia() as Boolean`	Returns `true` when this token is whitespace or a comment.
public override	toString `toString() as String`	Formats this token for debugging.

Inherited Methods

methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml