halotukozak.alpaca.internal.lexer
Members list
Type members
Classlikes
Attributes
- Source
- Column.scala
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
Column.type
A lazy character sequence that reads from a Reader on demand.
A lazy character sequence that reads from a Reader on demand.
This class is useful for tokenizing large files without loading them entirely into memory. It buffers characters as they are accessed and can efficiently skip over processed characters.
Like the Reader it wraps, it is consumed as it is read. Create it with LazyReader.from.
Attributes
- Companion
- object
- Source
- LazyReader.scala
- Supertypes
-
trait Closeabletrait AutoCloseabletrait CharSequenceclass Objecttrait Matchableclass AnyShow all
Factory methods for creating LazyReader instances.
Factory methods for creating LazyReader instances.
Attributes
- Companion
- class
- Source
- LazyReader.scala
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
LazyReader.type
A token matched in the input, as tokenize returns it: the token's name, its value, the matched text, and the lexer context's fields, read by name (lexeme.line when the context has a line field). In a signature, write a lexer's lexemes as MyLexer.Lexeme to keep those fields.
A token matched in the input, as tokenize returns it: the token's name, its value, the matched text, and the lexer context's fields, read by name (lexeme.line when the context has a line field). In a signature, write a lexer's lexemes as MyLexer.Lexeme to keep those fields.
The context fields are a snapshot taken after the rule body ran, except that a tracked field, like line or column, keeps its value from before the match (even if the rule body assigned it), so it describes where the token starts.
Type parameters
- Name
-
the token name type
- Value
-
the value type
Value parameters
- name
-
the token name
- text
-
the matched text
- value
-
the token's value: what the rule passed to
Token["NAME"](value), or()
Attributes
- Source
- Lexeme.scala
- Supertypes
-
trait Selectableclass Objecttrait Matchableclass Any
The result of compiling a lexer definition.
The result of compiling a lexer definition.
This abstract class represents a compiled lexer that can tokenize input. It is generated by the lexer macro and provides methods to access tokens and perform tokenization.
Type parameters
- Ctx
-
the global context type
Attributes
- Source
- Lexer.scala
- Supertypes
-
trait Selectableclass Objecttrait Matchableclass Any
Attributes
- Source
- Line.scala
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
Line.type
Base trait for all token types.
Base trait for all token types.
A token represents a lexical unit matched by the lexer. It contains information about the token's name, pattern, and how to manipulate the lexer context when matched.
Type parameters
- Ctx
-
the global context type
- Name
-
the token name type
- Value
-
the value type extracted from the matched text
Attributes
- Source
- Token.scala
- Supertypes
-
class Objecttrait Matchableclass Any
A per-token update for a single lexer-context field (a "fragment" such as Line or Column).
Tracking.materialize applies one Tracking instance to each case field of the context for which a given Tracking[FieldType] resolves at the lexer call, threading the result through a functional copy; fields with none are left untouched by the hook (they only change in rule bodies).
Lookup is ordinary given resolution at the lexer call: a given in lexical scope there wins, otherwise the one in the field type's companion is used. So a given Tracking[Int] in scope tracks every Int field of that lexer's context, and a local given Tracking[Column] replaces Column's own for that lexer only.
This is how tracking composes without inheritance: define a distinct field type (e.g. an opaque type) and a given Tracking[YourType] in its companion, then use it as a context field. A distinct type also keeps the update to the fields meant for it, where a given for a common type like Int would reach every field of that type.
Type parameters
- Field
-
the fragment field type
Attributes
- Companion
- object
- Source
- Tracking.scala
- Supertypes
-
class Objecttrait Matchableclass Any
Attributes
- Companion
- trait
- Source
- Tracking.scala
- Supertypes
-
class Objecttrait Matchableclass Any
- Self type
-
Tracking.type
Types
A context fragment that tracks the current 1-based column within the line, counted in Unicode code points (an emoji is one column), restarting after every \n.
A context fragment that tracks the current 1-based column within the line, counted in Unicode code points (an emoji is one column), restarting after every \n.
Use it as a field of a lexer context:
case class MyCtx(column: Column = Column.Start) extends LexerCtx
The lexer updates it after every match with the given Tracking[Column] below, so the field can stay an immutable val. Column <: Int, so ctx.column reads as a plain Int everywhere; a rule body can also assign it (ctx.column = Column(...)).
In a lexeme it is the column the token starts at, in a LexerError the column the unmatched input starts at.
Attributes
- Source
- Column.scala
A context fragment that tracks the current 1-based line number.
A context fragment that tracks the current 1-based line number.
Use it as a field of a lexer context:
case class MyCtx(line: Line = Line.Start) extends LexerCtx
The lexer updates it after every match with the given Tracking[Line] below, so the field can stay an immutable val. Line <: Int, so ctx.line reads as a plain Int everywhere; a rule body can also assign it (ctx.line = Line(...)).
In a lexeme it is the line the token starts on, in a LexerError the line the unmatched input starts on.
Attributes
- Source
- Line.scala