Lexer

The lexical analyzer for the Loreline language. Converts source code text into a sequence of tokens.

Static Members #

Instance Members #

Resets the lexer to the beginning of the input.

Converts the entire input into an array of tokens.

Reads and returns the next token from the input.

Gets the array of lexing errors encountered.

Creates a new lexer for the given input.

Private Members #

Mapping of keywords to their corresponding token types.

The input source code being lexed.

Current position in the input.

Length of the input string.

Current line number.

Current column number.

Starting line of the current token.

Starting column of the current token.

Previous token that was read.

Current token lists during tokenization

A stack to keep track of whether we are inside a beat or a state/character block. Depending on that, the rules for reading unquoted string tokens are different.

The token type that will be added to the stack next time we find a LBrace token

When last value is true, that means only strictly correct expressions are accepted, thus unquoted strings will be disabled. Mostly used to handle interpolated values in strings or object/array literals in strict expressions that loosen the rules

Current indentation level (number of spaces/tabs)

Stack of indentation levels

Queue of generated indentation tokens

The indentation size (e.g., 4 spaces or 1 tab)

Whether tabs are allowed for indentation

Errors collected during lexing, if any

Compares two string attachments by their starting position.

Returns the token type of the parent block.

Checks if currently in a beat block.

Checks if currently in a state or character block.

Reads a line break token (newline or carriage return + newline).

Helper function to check if a character is whitespace.

Helper function to match an identifier at the given position.

Helper function to skip whitespace and comments

Returns whether the input at the given position is the start of an if condition.

Returns whether the input at the given position is the start of an identifier expression.

Returns whether the input at the given position is a valid transition start. A valid transition consists of "->" followed by an identifier, with optional whitespace and comments in between. Nothing but whitespace and comments can follow the identifier.

Returns whether the input at the given position begins with a label pattern (identifier:).

Returns whether the input at the given position is the start of a function or method call.

Returns whether the input at the given position is the start of an assignment.

Returns whether the next token is a colon.

If we are currently right after a label, return the related identifier token index (before colon), or -1 if not after a label.

Checks if a dialogue statement continues over multiple lines.

Returns whether we are currently inside brackets [ ].

Returns whether the resolved tokens in the current line are only white spaces, a single label someLabel:, or comments.

Returns whether the resolved tokens in the current line are only white spaces or comments.

Returns whether the given string has a non special character that could be considered as actual text.

Returns whether the given input is a valid number.

Tries to read an unquoted string literal from the current position. Returns null if the current position cannot start an unquoted string.

Reads a string literal, handling escape sequences, interpolation, and tags.

Reads a complex interpolation expression inside ${...}.

Reads a token with proper position tracking in a nested context.

Reads a simple field access interpolation (e.g. $foo.bar[baz].qux).

Reads an identifier token in a string interpolation context.

Makes position relative to the given string start position.

Reads a single-line comment.

Reads a multi-line comment, handling nested comments.

Reads a numeric literal.

Reads an identifier or keyword.

Creates a new position at the current location.

Creates a new token with the given type and optional position.

Advances the lexer position by the given number of characters.

Looks ahead in the input without advancing the position.

Creates and throws a lexer error.

Skips whitespace characters (space and tab).

Checks if a character is a digit (0-9). @*

Checks if a character is valid as the start of an identifier. Valid identifier starts are letters and underscore.

Checks if a character is valid as part of an identifier. Valid identifier parts are letters, numbers, and underscore.