Overview > klyn.code.python > PythonTokenizer
Version 0.2.1 Overview Index Tree Deprecated
PythonTokenizer
classin packageklyn.code.python
public class PythonTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ PythonTokenizer

Specialized CodeTokenizer for Python source files (.py, .pyw, .pyi).

Supports Python # comments, single/double and triple-quoted strings ("""...""", '''...'''), f/r/b/u string prefixes with {...} interpolation, and @decorator lines.

import klyn.code
import klyn.code.python

tokenizer = PythonTokenizer()
res = tokenizer.tokenizeLine("def greet(name: str) -> str: # hello")
assert res.spans[0].kind == TokenKind.KEYWORD
assert res.spans[2].kind == TokenKind.DECLARATION_NAME
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public PythonTokenizer
PythonTokenizer()
 Constructs a PythonTokenizer configured with PythonLanguage.profile().
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml