Overview > klyn.code.json > JsonTokenizer
Version 0.2.2 Overview Index Tree Deprecated
JsonTokenizer
classin packageklyn.code.json
public class JsonTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ JsonTokenizer

Specialized CodeTokenizer for JSON and JSONC documents.

Distinguishes JSON object property keys ("key": -> TokenKind.PROPERTY_NAME) from string values (TokenKind.STRING), numbers (TokenKind.NUMBER), and boolean/null literals (TokenKind.CONSTANT).

import klyn.code
import klyn.code.json

tokenizer = JsonTokenizer()
res = tokenizer.tokenizeLine("{\"name\": \"Klyn\", \"fast\": true}")
assert res.spans[1].kind == TokenKind.PROPERTY_NAME
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public JsonTokenizer
JsonTokenizer()
 Constructs a JsonTokenizer configured with JsonLanguage.profile().
Methods
Modifier and Type Member Description
public override tokenizeLine
tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult
 Tokenizes a single line of JSON/JSONC, promoting "key" tokens immediately followed by : to TokenKind.PROPERTY_NAME.
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml