Lexer Context

Every Alpaca lexer carries a context object that evolves as the input is processed. Context lets you do stateful lexing: counting brackets, tracking indentation, recording whether you are inside a string literal, or anything else that depends on the tokens seen so far.

By default, the lexer uses LexerCtx.Default, which gives you line and column tracking with no extra setup.

Default Context

When you write a lexer: block without a type parameter, the lexer uses LexerCtx.Default. It is a case class with two tracking fields:

import halotukozak.alpaca.*

final case class Default(
  column: Column = Column.Start,
  line: Line = Line.Start,
) extends LexerCtx

• column -- 1-based column within the current line, counted in Unicode code points (an emoji is one column), advanced by the matched text and restarted after every \n in it
• line -- 1-based line number, advanced by every \n in the matched text

Newlines are counted wherever they appear in a match, so case "\\s+" => Token.Ignored matching " \n " advances line by one and leaves column at 3. A Windows line ending \r\n counts once, through its \n.

Column and Line are opaque subtypes of Int, so ctx.column and ctx.line read as plain Ints everywhere. Each carries a given Tracking that the lexer macro finds and applies after every match.

import halotukozak.alpaca.*

val BrainLexer = lexer:
  case "\\+" => Token["inc"]
  case "-" => Token["dec"]
  case "\\s+" => Token.Ignored

val lexed = BrainLexer.tokenize("+ - +")


val lexemes = lexed.getOrThrow
// lexed.ctx.column == 6
// lexed.ctx.line   == 1
//
// Each lexeme records where its token starts:
// inc: text="+", column=1, line=1
// dec: text="-", column=3, line=1
// inc: text="+", column=5, line=1

In the context, column and line are the position right after the last match. In a lexeme, they are the position where its token starts -- the place to point an error message at.

The LexerCtx Trait

LexerCtx is the base trait for all lexer contexts. Any custom context must satisfy two rules:

1. It must be a case class -- LexerCtx extends Product directly, and the auto-derivation machinery requires a Product instance.
2. All fields must have default values -- The lexer macro reads default parameter values from the companion to construct the initial context. If any parameter lacks a default, the macro fails at compile time.

Note: Context fields are read by name on lexemes and lexer errors, so the names of their public members are reserved, for example name, value, text, toString and selectDynamic (members of Lexeme), unexpected and message (members of LexerError), and getClass, eq or synchronized, which every object has. The lexer macro reports a field with one of these names as a compile error.

State fields are ordinary immutable val case-class parameters. Writing ctx.count += 1 in a rule body still type-checks and does the expected thing -- the lexer macro rewrites every such assignment into a functional copy before the rule is compiled. A field of a mutable collection type (e.g., scala.collection.mutable.Stack) works too: you mutate the collection in place and never reassign the field. Every lexeme keeps a snapshot of the context's fields, though, and a mutable collection is shared by all of them, so each snapshot shows its final contents; an immutable value keeps the contents at match time.

Custom Context

The BrainFuck lexer from Getting Started does not validate bracket matching -- it tokenizes ] even without a prior [. To fix that, we track bracket depth in a custom context:

import halotukozak.alpaca.*

case class BrainLexContext(
  brackets: Int = 0,
  squareBrackets: Int = 0,
) extends LexerCtx

val BrainLexer = lexer[BrainLexContext]:
  case ">" => Token["next"]
  case "<" => Token["prev"]
  case "\\+" => Token["inc"]
  case "-" => Token["dec"]
  case "\\." => Token["print"]
  case "," => Token["read"]
  case "\\[" =>
    ctx.squareBrackets += 1
    Token["jumpForward"]
  case "\\]" =>
    require(ctx.squareBrackets > 0, "Mismatched brackets")
    ctx.squareBrackets -= 1
    Token["jumpBack"]
  case name @ "[A-Za-z]+" => Token["functionName"](name)
  case "\\(" =>
    ctx.brackets += 1
    Token["functionOpen"]
  case "\\)" =>
    require(ctx.brackets > 0, "Mismatched brackets")
    ctx.brackets -= 1
    Token["functionClose"]
  case "!" => Token["functionCall"]
  case "." => Token.Ignored
  case "\n" => Token.Ignored

The type parameter lexer[BrainLexContext] tells the macro which context to use. The final context state is the ctx of the Result that tokenize() returns:

import halotukozak.alpaca.*

case class BrainLexContext(
  brackets: Int = 0,
  squareBrackets: Int = 0,
) extends LexerCtx

val BrainLexer = lexer[BrainLexContext]:
  case ">" => Token["next"]
  case "<" => Token["prev"]
  case "\\+" => Token["inc"]
  case "-" => Token["dec"]
  case "\\." => Token["print"]
  case "," => Token["read"]
  case "\\[" =>
    ctx.squareBrackets += 1
    Token["jumpForward"]
  case "\\]" =>
    require(ctx.squareBrackets > 0, "Mismatched brackets")
    ctx.squareBrackets -= 1
    Token["jumpBack"]
  case name @ "[A-Za-z]+" => Token["functionName"](name)
  case "\\(" =>
    ctx.brackets += 1
    Token["functionOpen"]
  case "\\)" =>
    require(ctx.brackets > 0, "Mismatched brackets")
    ctx.brackets -= 1
    Token["functionClose"]
  case "!" => Token["functionCall"]
  case "." => Token.Ignored
  case "\n" => Token.Ignored
val lexed = BrainLexer.tokenize("[>+<-]")
val lexemes = lexed.getOrThrow
// lexed.ctx.squareBrackets == 0  -- balanced

Accessing Context in Patterns

Inside a lexer[Ctx]: block, the name ctx is implicitly available and refers to the current context object. You can read any field and assign to it -- the assignment is rewritten to a copy. ctx and Token[...] compile only inside a lexer rule, like these:

import halotukozak.alpaca.*

case class BrainLexContext(
  brackets: Int = 0,
  squareBrackets: Int = 0,
) extends LexerCtx

val BrainLexer = lexer[BrainLexContext]:
  case ">" => Token["next"]
  case "<" => Token["prev"]
  case "\\+" => Token["inc"]
  case "-" => Token["dec"]
  case "\\." => Token["print"]
  case "," => Token["read"]
  case "\\[" =>
    ctx.squareBrackets += 1
    Token["jumpForward"]
  case "\\]" =>
    require(ctx.squareBrackets > 0, "Mismatched brackets")
    ctx.squareBrackets -= 1
    Token["jumpBack"]
  case name @ "[A-Za-z]+" => Token["functionName"](name)
  case "\\(" =>
    ctx.brackets += 1
    Token["functionOpen"]
  case "\\)" =>
    require(ctx.brackets > 0, "Mismatched brackets")
    ctx.brackets -= 1
    Token["functionClose"]
  case "!" => Token["functionCall"]
  case "." => Token.Ignored
  case "\n" => Token.Ignored
val ExampleLexer = lexer[BrainLexContext]:
  case "\\[" =>
    ctx.squareBrackets += 1       // write
    Token["jumpForward"]
  case "\\]" =>
    require(ctx.squareBrackets > 0, "Mismatched brackets")  // read + validate
    ctx.squareBrackets -= 1       // write
    Token["jumpBack"]

Lexer rules cannot have guards; check the context in the rule body instead, as above (see Guards Not Supported).

Context Snapshots in Lexemes

Each lexeme carries a snapshot of the context's fields as they were right after its rule body ran; The Lexeme Structure describes the snapshot and its one exception, tracked fields. With a custom context, all of its case class fields appear in the snapshot:

import halotukozak.alpaca.*

case class BrainLexContext(
  squareBrackets: Int = 0,
) extends LexerCtx

val BrainLexer = lexer[BrainLexContext]:
  case "\\[" =>
    ctx.squareBrackets += 1
    Token["jumpForward"]
  case "\\]" =>
    ctx.squareBrackets -= 1
    Token["jumpBack"]
  case "\\+" => Token["inc"]
  case "." => Token.Ignored

val lexemes = BrainLexer.tokenize("[+[+]]").getOrThrow
// lexemes(0).squareBrackets == 1  -- after first [
// lexemes(2).squareBrackets == 2  -- after second [
// lexemes(4).squareBrackets == 1  -- after first ]

Built-in Tracking Fragments

Alpaca ships two ready-made tracking fields, both re-exported from halotukozak.alpaca:

Column -- an opaque Int that advances by the number of code points matched after each token; when the match contains a \n, it becomes 1 plus the code points after the last one.

Line -- an opaque Int that advances by the number of \ns in each match.

Each is a plain case-class field with a given Tracking in its companion. Use either one, both, or neither. LexerCtx.Default uses both. Like any context field, they can be called anything, and lexemes and LexerErrors carry them under that name. To add them to a custom context:

import halotukozak.alpaca.*

case class BrainLexContext(
  squareBrackets: Int = 0,
  column: Column = Column.Start,
  line: Line = Line.Start,
) extends LexerCtx

With this context, every lexeme carries squareBrackets, column, and line. squareBrackets changes only where a rule body assigns it; column and line advance automatically after every match.

The Post-Match Update

After every successful token match, the lexer:

1. applies each tracked field's Tracking update (column, line, and any custom fragments),
2. applies the rule body's own context changes,
3. records the lexeme snapshot, taking every tracked field from before step 1.

So inside a rule body, ctx.line and ctx.column are already the position after the match, while the lexeme gets the position where the token starts.

Steps 1 and 3 are derived by the lexer macro from the context's case fields -- there is nothing to wire up by hand.

import halotukozak.alpaca.*

// Step 1: a distinct type for what you track
opaque type Indent <: Int = Int
object Indent:
  val Start: Indent = 0

  // Step 2: a given Tracking in its companion
  given Tracking[Indent] =
    case ("\t", n) => n + 1
    case ("\n", _) => 0
    case (_, n)    => n

// Step 3: use it as a context field
case class MyCtx(indent: Indent = Indent.Start) extends LexerCtx

// Step 4: the lexer macro applies the update automatically
val MyLexer = lexer[MyCtx]:
  case "\t" => Token.Ignored
  case "\n" => Token.Ignored
  case id @ "[a-z]+" => Token["ID"](id)

Which Tracking Is Used

The lexer macro looks up a given Tracking for each field's type where lexer is called, by ordinary given resolution: a given in scope there wins, otherwise the one in the type's companion is used.

So a given Tracking[Int] in scope tracks every Int field of the context. To track one field, give it its own type, an opaque type with the given in its companion, as Indent does above.

The flip side: a local given replaces a fragment's tracking for the lexers defined in its scope. Here, a tab counts as 4 columns:

import halotukozak.alpaca.*

object WideTabs:
  given Tracking[Column] = (matched, column) =>
    val lastLine = matched.substring(matched.lastIndexOf('\n') + 1)
    val width = lastLine.map(char => if char == '\t' then 4 else 1).sum
    if matched.contains('\n') then Column(1 + width) else Column(column + width)

  val MyLexer = lexer:
    case id @ "[a-z]+" => Token["ID"](id)
    case "\\s+" => Token.Ignored

val lexemes = WideTabs.MyLexer.tokenize("ab\tcd").getOrThrow
// lexemes(1).column == 7

Lexers defined outside WideTabs keep the default Column tracking.

LexerCtx.Empty

For cases where you need no tracking at all -- no column, no line counter, no custom fields -- use LexerCtx.Empty:

import halotukozak.alpaca.*

val MyLexer = lexer[LexerCtx.Empty]:
  case "\\+" => Token["inc"]
  case "." => Token.Ignored

val lexemes = MyLexer.tokenize("+ +").getOrThrow
// lexemes(0).text == "+"  -- no context fields: no column, no line

See Between Stages to learn how context snapshots in lexemes flow into the parser.