Overview > klyn.code.java > JavaTokenizer
Version 0.2.1 Overview Index Tree Deprecated
JavaTokenizer
classin packageklyn.code.java
public class JavaTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ JavaTokenizer

Specialized CodeTokenizer for Java source files (.java).

Supports Java single-line (//), multi-line block, and Javadoc documentation comments, Java 15+ text blocks ("""..."""), @Annotation markers, $ identifiers, and method declaration classification.

import klyn.code
import klyn.code.java

tokenizer = JavaTokenizer()
res = tokenizer.tokenizeLine("@Override public String toString() { return \"ok\"; }")
assert res.spans[0].kind == TokenKind.ANNOTATION
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public JavaTokenizer
JavaTokenizer()
 Constructs a JavaTokenizer configured with JavaLanguage.profile().
Methods
Modifier and Type Member Description
public override classifyIdentifier
classifyIdentifier( token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean ) as TokenKind
 Classifies Java identifiers, promoting method names followed by ( after a type or modifier to TokenKind.DECLARATION_NAME or TokenKind.FUNCTION.
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml