CodeTokenizer

classin packageklyn.code

public class CodeTokenizer extends Object implements Tokenizer

Object

└ CodeTokenizer

All Implemented Interfaces: Tokenizer

High-performance, table-driven lexical scanner implementing Tokenizer for programming languages.

CodeTokenizer uses zero-allocation character classification and $O(1)$ HashSet vocabulary lookups from LanguageProfile. Language-specific tokenizers (such as KlynTokenizer, JavaTokenizer, PythonTokenizer, CppTokenizer, and SqlTokenizer) inherit from or configure CodeTokenizer.

import klyn.code

profile = LanguageProfile("demo", "Demo")
profile.withLineComments(["//"])
profile.withStringDelimiters(["\""])
profile.withKeywords(["if", "return", "class", "def"])
profile.withTypes(["Int", "String"])
tokenizer = CodeTokenizer(profile)
res = tokenizer.tokenizeLine("def compute(x as Int): // fast")
assert res.spans[0].kind == TokenKind.KEYWORD

Inherited Properties

propertyInherited Properties from Object: type

Constructors

Modifier and Type	Member	Description
public	CodeTokenizer `CodeTokenizer(languageProfile as LanguageProfile)`	Constructs a `CodeTokenizer` driven by the given `LanguageProfile`.

Methods

Modifier and Type	Member	Description
public static	appendLiteralSpans `appendLiteralSpans( spans as ArrayList<TokenSpan>, line as String, start as Long, end as Long, highlightEscapes as Boolean, tokenKind as TokenKind = TokenKind.STRING ) as Void`	Emits string content and escape spans (`TokenKind.STRING` and `TokenKind.STRING_ESCAPE`) for the literal segment `line[start:end]`.
public	classifyIdentifier `classifyIdentifier(token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean) as TokenKind`	Classifies an identifier token according to the active `LanguageProfile` and preceding word context.
public	collectIdentifiers `collectIdentifiers(source as String, minLength as Long = 2) as ArrayList<String>`	Collects all unique identifier tokens of length `>= minLength` occurring in code positions (outside comments and string/character/regex literals) in `source`.
public	identifierAt `identifierAt(source as String, targetLine as Long, targetColumn as Long) as String`	Extracts the complete identifier at or immediately before `(targetLine, targetColumn)`.
public	identifierPrefixAt `identifierPrefixAt(source as String, targetLine as Long, targetColumn as Long) as String`	Extracts the identifier prefix immediately preceding `(targetLine, targetColumn)`.
public static	isAsciiAlpha `isAsciiAlpha(ch as Char) as Boolean`	Fast ASCII alphabetic character test (`'a'..'z'` or `'A'..'Z'`).
public static	isAsciiDigit `isAsciiDigit(ch as Char) as Boolean`	Fast ASCII decimal digit test (`'0'..'9'`).
public	isCodePosition `isCodePosition(line as String, column as Long, initialState as LexicalState = null) as Boolean`	Checks whether `column` in `line` is a code position (outside comments and string/character/regex literals).
public	isCodePositionInSource `isCodePositionInSource(source as String, targetLine as Long, targetColumn as Long) as Boolean`	Checks whether `(targetLine, targetColumn)` in a multi-line `source` document is a code position (outside comments and string/character/regex literals).
public static	isEscapedAt `isEscapedAt(line as String, index as Long) as Boolean`	Checks whether the character at `index` in `line` is escaped by an odd number of preceding backslashes.
public static	isHexDigit `isHexDigit(ch as Char) as Boolean`	Fast ASCII hexadecimal digit test (`'0'..'9'`, `'a'..'f'`, `'A'..'F'`).
public	isIdentPart `isIdentPart(ch as Char) as Boolean`	Checks whether `ch` can continue an identifier in this language.
public	isIdentStart `isIdentStart(ch as Char) as Boolean`	Checks whether `ch` can start an identifier in this language.
public	isNumberStart `isNumberStart(line as String, index as Long) as Boolean`	Checks whether a numeric literal starts at `index` in `line`.
public static	isOperatorChar `isOperatorChar(ch as Char) as Boolean`	Checks whether `ch` is a standard operator character.
public	isPreprocessorLine `isPreprocessorLine(line as String, firstNonSpace as Long) as Boolean`	Checks whether the line starts with a preprocessor directive (such as `#include` or `#define`).
public static	isPunctuationChar `isPunctuationChar(ch as Char) as Boolean`	Checks whether `ch` is a punctuation/delimiter symbol (`(`, `)`, `[`, `]`, `{`, `}`, `,`, `;`, `.`).
public	isValidIdentifier `isValidIdentifier(text as String) as Boolean`	Checks whether `text` is a syntactically valid identifier according to this tokenizer's profile.
public static	isValidIdentifier `isValidIdentifier(text as String, extraChars as String = "") as Boolean`	Checks whether `text` is a valid identifier using standard identifier rules plus `extraChars`.
public	lineCommentAt `lineCommentAt(line as String, index as Long) as String`	Checks whether a single-line comment starts at `index` in `line`.
public	literalDelimiterAt `literalDelimiterAt(line as String, index as Long) as String`	Returns the character/secondary literal delimiter starting at `index` in `line`, or `""`.
public	nextLineState `nextLineState(line as String, initialState as LexicalState = null) as LexicalState`	Fast state-only line transition that computes the resulting `LexicalState` without allocating `TokenSpan` objects.
public	profile `profile() as LanguageProfile`	Returns the `LanguageProfile` driving this tokenizer.
public	renameIdentifier `renameIdentifier(source as String, oldName as String, newName as String) as String`	Renames every occurrence of code identifier `oldName` to `newName` in `source` while leaving comments, string/character/regex literals, and line endings untouched.
public static	scanEscapeEnd `scanEscapeEnd(line as String, index as Long, limit as Long) as Long`	Scans a backslash escape sequence starting at `index` (where `line[index] == '\\'`) up to `limit`.
public	scanNumberEnd `scanNumberEnd(line as String, start as Long) as Long`	Scans to the end of a numeric literal starting at `start` in `line`.
public static	startsWithAt `startsWithAt(text as String, token as String, start as Long) as Boolean`	Zero-allocation substring prefix check at `start` inside `text`.
public static	startsWithIgnoreCaseAt `startsWithIgnoreCaseAt(text as String, token as String, start as Long) as Boolean`	Zero-allocation ASCII case-insensitive prefix check at `start` inside `text`.
public	stringDelimiterAt `stringDelimiterAt(line as String, index as Long) as String`	Returns the string delimiter starting at `index` in `line`, or `""` if none matches.
public	stringPrefixAt `stringPrefixAt(line as String, index as Long) as String`	Checks whether a string prefix (`f`, `r`, `fr`, etc.) starts at `index` in `line` and is immediately followed by a string delimiter.
public	stripLineComments `stripLineComments(line as String, initialState as LexicalState, out endState as LexicalState) as String`	Strips line and block comments from a single `line` while preserving string literals and updating `endState` for multi-line block comments or strings.
public	tokenize `tokenize(source as String) as ArrayList<Token>`	Tokenizes a complete multi-line source document into materialized `Token` instances.
public	tokenizeLine `tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult`	Tokenizes a single line of source code starting from `initialState`.

Inherited Methods

methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml