RustTokenizer

classin packageklyn.code.rust

public class RustTokenizer extends CodeTokenizer

Object

└ CodeTokenizer

└ RustTokenizer

Specialized CodeTokenizer for Rust source files (.rs).

Supports Rust attributes (#[derive(...)], #![allow(...)]), lifetime annotations ('a, 'static, '_) distinguished from character literals ('x', '\n', b'x'), macro invocations (println!, vec!), numeric type suffixes (42usize, 100_u128, 3.5f64), /// and //! documentation comments, and nested block comments.

Example

import klyn.code
import klyn.code.rust

tokenizer = RustTokenizer()
res = tokenizer.tokenizeLine("pub fn borrow<'a>(input: &'a str) -> &'a str { input }")
assert res.spans[0].kind == TokenKind.KEYWORD

Inherited Properties

propertyInherited Properties from Object: type

Constructors

Modifier and Type	Member	Description
public	RustTokenizer `RustTokenizer()`	Constructs a `RustTokenizer` configured with `RustLanguage.profile()`.

Methods

Modifier and Type	Member	Description
public override	classifyIdentifier `classifyIdentifier( token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean ) as TokenKind`	Classifies Rust identifiers, recognizing macro invocations (`ident!`), `type` / `mod` declarations, associated types, and `::` path segments.
public override	tokenizeLine `tokenizeLine(line as String, initialState as LexicalState = null) as LineScanResult`	Tokenizes a single line of Rust source code, handling `#[...]` / `#![...]` attributes, `///` / `//!` doc comments, and `'lifetime` annotations without misclassifying them as unclosed character literals.

Inherited Methods

methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml