MarkdownTokenizer

classin packageklyn.code.markdown

public class MarkdownTokenizer extends CodeTokenizer

Object

└ CodeTokenizer

└ MarkdownTokenizer

Specialized CodeTokenizer for Markdown documents with embedded fenced-code delegation.

Highlights:

ATX headings (#, ##, ..., ######) as TokenKind.HEADING
Blockquotes (>) as TokenKind.QUOTE
Unordered and ordered list markers (-, *, +, 1.) as TokenKind.LIST_MARKER
Fenced code blocks ( lang or ~~~lang) as TokenKind.CODE_FENCE`, delegating inner lines to the matching language Tokenizer (Klyn, Java, Python, C++, SQL, JSON, etc.)
Inline code spans ( code ) as TokenKind.INLINE_CODE
Links ([label](url)) as TokenKind.LINK_TEXT and TokenKind.LINK_URL
Bold (**bold**) and italic (*italic*) spans as TokenKind.BOLD and TokenKind.ITALIC

import klyn.code
import klyn.code.markdown

tokenizer = MarkdownTokenizer()
h = tokenizer.tokenizeLine("## Architecture Overview")
assert h.spans[0].kind == TokenKind.HEADING

Inherited Properties

propertyInherited Properties from Object: type

Constructors

Modifier and Type	Member	Description
public	MarkdownTokenizer `MarkdownTokenizer()`	Constructs a `MarkdownTokenizer` configured with `MarkdownLanguage.profile()`.

Methods

Modifier and Type	Member	Description
public override	tokenizeLine `tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult`	Tokenizes a single line of Markdown, tracking multi-line fenced code blocks via `LexicalState`.

Inherited Methods

methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml