Lexer

View source
loreline.Lexer (Class)

The lexical analyzer for the Loreline language. Converts source code text into a sequence of tokens.

Static Members

hasNonSpecialChar_specialChars: String

Instance Members

reset(): Void

Resets the lexer to the beginning of the input.

tokenize(): Tokens

Converts the entire input into an array of tokens.

Returns Description
Tokens Array of tokens
nextToken(): Token

Reads and returns the next token from the input.

Returns Description
Token The next token
getErrors(): Array<LexerError>

Gets the array of lexing errors encountered.

Returns Description
Array<LexerError> Array of LexerError objects
new(input: String): Void

Creates a new lexer for the given input.

Name Type Description
input String The source code to lex

Private Members

KEYWORDS: haxe.ds.Map<String, Anonymous>

Mapping of keywords to their corresponding token types.

input: String

The input source code being lexed.

pos: Int

Current position in the input.

length: Int

Length of the input string.

line: Int

Current line number.

column: Int

Current column number.

startLine: Int

Starting line of the current token.

startColumn: Int

Starting column of the current token.

Previous token that was read.

tokenized: Tokens

Current token lists during tokenization

A stack to keep track of whether we are inside a beat or a state/character block. Depending on that, the rules for reading unquoted string tokens are different.

nextBlock: TokenStackType

The token type that will be added to the stack next time we find a LBrace token

strictExprs: Array<StrictExprType>

When last value is true, that means only strictly correct expressions are accepted, thus unquoted strings will be disabled. Mostly used to handle interpolated values in strings or object/array literals in strict expressions that loosen the rules

indentLevel: Int

Current indentation level (number of spaces/tabs)

indentStack: Array<Int>

Stack of indentation levels

indentTokens: Tokens

Queue of generated indentation tokens

indentSize: Int

The indentation size (e.g., 4 spaces or 1 tab)

allowTabs: Bool

Whether tabs are allowed for indentation

Errors collected during lexing, if any

compareAttachments(l: Anonymous, r: Anonymous): Int

Compares two string attachments by their starting position.

Name Type Description
l Anonymous First attachment
r Anonymous Second attachment
Returns Description
Int Negative if l comes before r, positive if r comes before l, 0 if equal
countIndentation(): Int
Returns
Int
parentBlockType(): Anonymous

Returns the token type of the parent block.

Returns Description
Anonymous The token type of the parent block or KwBeat if at top level
inBeat(): Bool

Checks if currently in a beat block.

Returns Description
Bool True if inside a beat block, false otherwise
inStateOrCharacter(): Bool

Checks if currently in a state or character block.

Returns Description
Bool True if inside a state or character block, false otherwise
readLineBreak(): Token

Reads a line break token (newline or carriage return + newline).

Returns Description
Token Line break token
isWhitespace(c: Int): Bool

Helper function to check if a character is whitespace.

Name Type Description
c Int The character code to check
Returns Description
Bool True if the character is whitespace, false otherwise
matchIdentifier(pos: Int): Null<String>

Helper function to match an identifier at the given position.

Name Type Description
pos Int The position to start matching from
Returns Description
Null<String> The matched identifier, or null if not matching
skipWhitespaceAndComments(pos: Int, ?stopNextLine: Bool = false): Int

Helper function to skip whitespace and comments

Name Type Default
pos Int
stopNextLine Bool false
Returns
Int
isIfStart(pos: Int): Bool

Returns whether the input at the given position is the start of an if condition.

Name Type Description
pos Int Position to check from
Returns Description
Bool True if an if condition starts at the position, false otherwise
isIdentifierExpressionStart(pos: Int, lowercaseIdentOnly: Bool): Bool

Returns whether the input at the given position is the start of an identifier expression.

Name Type Description
pos Int Position to check from
lowercaseIdentOnly Bool
Returns Description
Bool True if an identifier expression starts at the position, false otherwise
isTransitionStart(pos: Int): Bool

Returns whether the input at the given position is a valid transition start. A valid transition consists of "->" followed by an identifier, with optional whitespace and comments in between. Nothing but whitespace and comments can follow the identifier.

Name Type Description
pos Int Position to check from
Returns Description
Bool True if a valid transition starts at the position, false otherwise
isInsertionStart(pos: Int): Bool
Name Type
pos Int
Returns
Bool
isLabelStart(pos: Int): Bool

Returns whether the input at the given position begins with a label pattern (identifier:).

Name Type Description
pos Int Position to check from
Returns Description
Bool True if a label starts at the position, false otherwise
isCallStart(pos: Int): Bool

Returns whether the input at the given position is the start of a function or method call.

Name Type Description
pos Int Position to check from
Returns Description
Bool True if a function call starts at the position, false otherwise
isAssignStart(pos: Int, strict: Bool): Bool

Returns whether the input at the given position is the start of an assignment.

Name Type Description
pos Int Position to check from
strict Bool
Returns Description
Bool True if an assignment starts at the position, false otherwise
isColon(pos: Int, ?skipWhitespaces: Bool = true): Bool

Returns whether the next token is a colon.

Name Type Default Description
pos Int Position to check from
skipWhitespaces Bool true Whether to skip whitespace before checking
Returns Description
Bool True if next token is a colon, false otherwise
afterLabelIdentifierToken(?inSameLine: Bool = true): Int

If we are currently right after a label, return the related identifier token index (before colon), or -1 if not after a label.

Name Type Default
inSameLine Bool true
Returns
Int
isStartingMultilineDialogue(labelIdentifierTokenIndex: Int): Int

Checks if a dialogue statement continues over multiple lines.

Name Type
labelIdentifierTokenIndex Int
Returns Description
Int The indentation of the multiline dialogue or -1
isContinuingMultilineText(pos: Int, indent: Int): Bool
Name Type
pos Int
indent Int
Returns
Bool
isAfterComma(): Bool
Returns
Bool
isAfterLBracket(): Bool
Returns
Bool
isInsideBrackets(): Bool

Returns whether we are currently inside brackets [ ].

Returns Description
Bool True if inside brackets, false otherwise
followsAssignStart(): Bool
Returns
Bool
followsOnlyLabelOrCommentsInLine(): Bool

Returns whether the resolved tokens in the current line are only white spaces, a single label someLabel:, or comments.

Returns Description
Bool True if line only contains label or comments, false otherwise
followsOnlyWhitespacesOrCommentsInLine(): Bool

Returns whether the resolved tokens in the current line are only white spaces or comments.

Returns Description
Bool True if line only contains whitespace or comments, false otherwise
hasNonSpecialChar(str: String): Bool

Returns whether the given string has a non special character that could be considered as actual text.

Name Type Description
str String String to check
Returns Description
Bool True if string contains non-special characters, false otherwise
isNumber(value: String): Bool

Returns whether the given input is a valid number.

Name Type Description
value String String to check
Returns Description
Bool True if string represents a valid number, false otherwise
makeLooseOrStrictAfterBracket(): Void
makeStrictIfFollowingCallable(): Void
followsCallableOrIndexable(): Bool
Returns
Bool
isStrict(): Bool
Returns
Bool
readImportValue(c: Int, startPos: Position): Token
Name Type
c Int
startPos Position
Returns
Token
tryReadUnquotedString(): Null<Token>

Tries to read an unquoted string literal from the current position. Returns null if the current position cannot start an unquoted string.

Returns Description
Null<Token> Token if an unquoted string was read, null otherwise
readString(stringStart: Position): Token

Reads a string literal, handling escape sequences, interpolation, and tags.

Name Type Description
stringStart Position Starting position of the string
Returns Description
Token String literal token
readComplexInterpolation(interpStart: Position): Tokens

Reads a complex interpolation expression inside ${...}.

Name Type Description
interpStart Position Starting position of the interpolation
Returns Description
Tokens Array of tokens making up the interpolation expression
nextTokenWithPosition(startPos: Position): Token

Reads a token with proper position tracking in a nested context.

Name Type Description
startPos Position Starting position for position calculation
Returns Description
Token Token with adjusted position
readFieldAccessInterpolation(interpStart: Position): Tokens

Reads a simple field access interpolation (e.g. $foo.bar[baz].qux).

Name Type Description
interpStart Position Starting position of the interpolation
Returns Description
Tokens Array of tokens making up the field access
readIdentifierTokenInInterpolation(stringStart: Position): Token

Reads an identifier token in a string interpolation context.

Name Type Description
stringStart Position Starting position for position calculation
Returns Description
Token Identifier token
makePositionRelativeTo(stringStart: Position): Position

Makes position relative to the given string start position.

Name Type Description
stringStart Position Starting position for calculation
Returns Description
Position Position object relative to string start
readLineComment(): Token

Reads a single-line comment.

Returns Description
Token Comment token
readMultiLineComment(): Token

Reads a multi-line comment, handling nested comments.

Returns Description
Token Comment token
readNumber(): Token

Reads a numeric literal.

Returns Description
Token Number token
readIdentifier(): Token

Reads an identifier or keyword.

Returns Description
Token Identifier or keyword token
readFunction(start: Position): Token
Name Type
start Position
Returns
Token
skipQuotedString(): Void
makePosition(): Position

Creates a new position at the current location.

Returns Description
Position Position object
makeToken(type: Anonymous, ?position: Position): Token

Creates a new token with the given type and optional position.

Name Type Default Description
type Anonymous Token type
position Position (optional) Optional position (defaults to current position)
Returns Description
Token Created token
advance(?count: Int = 1): Void

Advances the lexer position by the given number of characters.

Name Type Default Description
count Int 1 Number of characters to advance (default 1)
peek(?offset: Int = 1): Int

Looks ahead in the input without advancing the position.

Name Type Default Description
offset Int 1 Number of characters to look ahead (default 1)
Returns Description
Int Character code at the offset position, or 0 if beyond input length
error(message: String, fatal: Bool): LexerError

Creates and throws a lexer error.

Name Type Description
message String Error message
fatal Bool
Returns
LexerError
skipWhitespace(): Void

Skips whitespace characters (space and tab).

isDigit(c: Int): Bool

Checks if a character is a digit (0-9). @*

Name Type Description
c Int Character code to check
Returns Description
Bool Whether the character is a digit
isIdentifierStart(c: Int): Bool

Checks if a character is valid as the start of an identifier. Valid identifier starts are letters and underscore.

Name Type Description
c Int Character code to check
Returns Description
Bool Whether the character can start an identifier
isLowerCase(c: Int): Bool
Name Type
c Int
Returns
Bool
isIdentifierPart(c: Int): Bool

Checks if a character is valid as part of an identifier. Valid identifier parts are letters, numbers, and underscore.

Name Type Description
c Int Character code to check
Returns Description
Bool Whether the character can be part of an identifier

Metadata

Name Parameters
:hxGen -