The parser

The package is github.com/go-embedded-r/r/syntax; the parser was released in v0.1.0 and deparse completes M1 in v0.2.0 (see Deparse).

f, err := syntax.ParseAll(src)                    // or ParseAllLimits(src, syntax.Limits{MaxBytes, MaxDepth, MaxNodes})
names, err := syntax.TopLevelNames(cell)          // names a cell assigns and reads
lines := syntax.Deparse(f.Exprs[0], nil)          // deparse(), one string per line

The tree

R’s own shape: operators, if/for/{/[ and the rest are calls to a symbol; Function carries the formals; |> is rewritten as R does; -> is <- with the sides swapped; a string callee is a symbol. Every node has a Span (line, column and byte range). File.Tokens are the terminals of getParseData() (SYMBOL_FUNCTION_CALL, SYMBOL_SUB, EQ_FORMALS, COMMENT, LINE_DIRECTIVE…).

Errors

*syntax.Error has Line, Col and Message, and its text is the message parse() signals, byte for byte, source context and caret included. Limits give a *syntax.LimitError (wraps ErrLimit). The statements before an error are kept in the returned File.

Bounds

Bytes, tree depth (also for chains such as a+a+a+..., built by a loop) and node count; R’s own 50-level context stack (contextstack overflow). Bomb tests (200 000-deep chains of every shape) assert a refusal under a memory limit.

TopLevelNames

• Assigned: <-, =, ->, "x" <-, f(x) <-, x$a <- (assign x), assign("lit", ...), for variables.
• Read: free variables and called names not bound earlier in the cell, under R scoping (function parameters and locals, local(); a function body may use anything the cell assigns anywhere).
• Dynamic (with Reasons) when names cannot be resolved: <<-, assign/get with a computed name or another environment, eval, parse, source, attach, library, require, sys.function, environment<-, load, list2env…

Numbers

A decimal literal is computed in double precision, digit by digit with an explicit fused multiply-add (math.FMA), then scaled by a power of ten: what R does on macOS arm64, where long double is double and the compiler fuses 10*ans + digit. 150 000 random literals are identical to that R’s, bit for bit, and the result is the same on every platform. R on x86-64 Linux accumulates in 80-bit extended precision and can differ by one unit in the last place on literals of more than 15 digits (the CI oracle lane saw it on 1e300), and by up to 6 ulp on literals near the subnormal range (measured on R’s own tests/: 290 literals differ, at most by 6 ulp). The parser is deterministic and matches the 64-bit reference; the platform-dependent literals are in testdata/numbers.R, compared within 16 ulp and kept out of the goldens. One literal differs by more: 1.7976931348623157e308 (the largest double) is Inf in 64-bit-double R and in the Go parser, and is kept by 80-bit R. Hexadecimal literals follow R’s rule (0x1.8p1).

Parity with R 4.6.2

Judged by the oracle, never by expectations written by hand: go run ./cmd/rparity ... parses the same text with R and with Go and compares

1. the terminal tokens and positions with getParseData();
2. the tree with a dump of the language objects of parse(keep.source = FALSE) (names, constants as IEEE bits, empty arguments, formals);
3. the error text with parse()’s;
4. the output of deparse() and print() (see Deparse).

R’s own sources are read where they are unpacked and never committed.

Speed

One core each, interleaved rounds, median of 11 after a warm-up round, 1-minute load 7.8 on an Apple M-series laptop: the 842 files of src/library that both parse (7.8 MB) take 0.214 s in Go (syntax.ParseAll, which also builds the source map and the token list) and 0.269 s in R’s parse(text =, keep.source = FALSE): Go/R = 0.80 (go run ./cmd/rparity bench DIR).

Known gaps

The effect of #line directives on the line numbers of later errors; parse(file=) encoding options; an invalid UTF-8 input is parsed as R’s lexer does, and for such text R cannot produce parse data, so tokens are not compared.