Overview > klyn.code.cpp > CppTokenizer
Version 0.2.1 Overview Index Tree Deprecated
CppTokenizer
classin packageklyn.code.cpp
public class CppTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ CppTokenizer

Specialized CodeTokenizer for C and C++ source and header files.

Supports #include / #define / #pragma preprocessor directives, Doxygen documentation comments, C++14 digit separators (1'000'000), and function declaration classification.

import klyn.code
import klyn.code.cpp

tokenizer = CppTokenizer()
res = tokenizer.tokenizeLine("#include <vector> // header")
assert res.spans[0].kind == TokenKind.PREPROCESSOR
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public CppTokenizer
CppTokenizer()
 Constructs a CppTokenizer configured with CppLanguage.profile().
Methods
Modifier and Type Member Description
public override classifyIdentifier
classifyIdentifier( token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean ) as TokenKind
 Classifies C and C++ identifiers, recognizing POSIX/standard *_t typedefs, typename parameters, and std:: / template type references.
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml