Overview > klyn.code.go > GoTokenizer
Version 0.2.1 Overview Index Tree Deprecated
GoTokenizer
classin packageklyn.code.go
public class GoTokenizer extends CodeTokenizer
Object
└ CodeTokenizer
└ GoTokenizer

Specialized CodeTokenizer for Go source files (.go).

Supports Go raw backtick string literals ( ... ) where backslashes are not escape sequences, imaginary numeric suffixes (42i, 3.14i), //go: compiler directives, type Name struct/interface declarations, and receiver method declarations (func (s *Server) Handle(...)).

Example
import klyn.code
import klyn.code.go

tokenizer = GoTokenizer()
res = tokenizer.tokenizeLine("func (s *Server) Start(ctx context.Context) error {")
assert res.spans[0].kind == TokenKind.KEYWORD
Inherited Properties
propertyInherited Properties from Object: type
Constructors
Modifier and Type Member Description
public GoTokenizer
GoTokenizer()
 Constructs a GoTokenizer configured with GoLanguage.profile().
Methods
Modifier and Type Member Description
public override classifyIdentifier
classifyIdentifier( token as String, line as String, endIndex as Long, previousWord as String, inQueryBlock as Boolean ) as TokenKind
 Classifies Go identifiers, recognizing type <Name> declarations, receiver method names, package-qualified types (context.Context, *http.Request), and built-in calls.
Inherited Methods
methodInherited Methods from CodeTokenizer: appendLiteralSpans, classifyIdentifier, collectIdentifiers, identifierAt, identifierPrefixAt, isAsciiAlpha, isAsciiDigit, isCodePosition, isCodePositionInSource, isEscapedAt, isHexDigit, isIdentPart, isIdentStart, isNumberStart, isOperatorChar, isPreprocessorLine, isPunctuationChar, isValidIdentifier, isValidIdentifier, lineCommentAt, literalDelimiterAt, nextLineState, profile, renameIdentifier, scanEscapeEnd, scanNumberEnd, startsWithAt, startsWithIgnoreCaseAt, stringDelimiterAt, stringPrefixAt, stripLineComments, tokenize, tokenizeLine
methodInherited Methods from Object: deepCopy, fromJson, hash, toDict, toJson, toMap, toString, toXml