Overview > klyn.code.markup > MarkupTokenizer
Version 0.2.1 Overview Index Tree Deprecated
MarkupTokenizer
classin packageklyn.code.markup
public class MarkupTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ MarkupTokenizer

Specialized CodeTokenizer for HTML, XML, and SVG markup files.

Tokenizes <!-- ... --> multi-line comments, <!DOCTYPE ...> / <?xml ...?> declarations, <tag / </tag> element names (TokenKind.TAG), attribute names (TokenKind.ATTRIBUTE), quoted attribute values (TokenKind.STRING), and &entity; references (TokenKind.ENTITY).

import klyn.code
import klyn.code.markup

tokenizer = MarkupTokenizer()
res = tokenizer.tokenizeLine("<div class=\"card\">Hello &amp; welcome</div>")
assert res.spans.size >= 5
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public MarkupTokenizer
MarkupTokenizer(xmlMode as Boolean = false)
 Constructs a MarkupTokenizer for HTML (xmlMode = false) or XML (xmlMode = true).
Methods
Modifier and Type Member Description
public override tokenizeLine
tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult
 Tokenizes a single line of HTML/XML markup.
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml