Oracle testing

Every implementation decision in Phasis is gated on one question: does V8 produce the same output?

JavaScript has subtle semantics — NaN !== NaN, -0 === 0, [1] + 1 === "11", the exact order of property enumeration, the timing of Promise microtasks. Re-deriving these by reading the spec is slow and error-prone. Phasis sidesteps that by treating Node.js (V8) as an oracle: write a JS scenario, run it through V8, capture the output as truth; run it through Phasis, capture the output again; diff.

1. SETUP    → a JavaScript source file or snippet
2. ORACLE   → Node.js executes it, output captured as truth
3. ACTUAL   → Phasis executes it, output captured
4. COMPARE  → oracle vs actual, diff measures the gap

Two levels of testing

Level 1 — Custom scenarios

Each scenario in scenarios/ is a small JS program with known output. Run a single scenario:

./bin/test-scenario operators/arithmetic

Run all of them, or just one category, or in parallel:

./bin/test-regression
./bin/test-regression --jobs 4
./bin/test-regression --category expressions

Refresh the oracle (e.g. after a Node.js version update):

./bin/oracle --refresh operators/arithmetic

Scenarios are organised by ECMAScript chapter: literals, operators, variables, control-flow, functions, objects, arrays, classes, builtins, errors, interop, edge cases.

Level 2 — test262

The official TC39 conformance suite is checked out as a git submodule under test262/. Each test is self-verifying: it passes if it doesn't throw, fails if it does (or the reverse for negative tests). The harness loads assert.js and sta.js before each test.

./bin/test262                                    # full suite
./bin/test262 --category built-ins/Array         # subset
./bin/test262 --jobs 4                           # parallel
./bin/test262 --failures                         # one line per failing test
./bin/compat-report                              # full COMPAT.md + compat.json

The full-suite snapshot lives in COMPAT.md, regenerated by bin/compat-report. In CI the compat-matrix.yml workflow runs it in shards and commits the result; that workflow is started by hand, not on every push.

Local pre-push check

./bin/verify-all runs the quality gate that CI enforces:

=== PHPStan ===                  level 6, zero errors
=== Code Standards ===           PHPCS, zero warnings
=== PHPUnit ===                  unit suite passes
=== Oracle Regression ===        scenarios under `scenarios/` pass

It is not sufficient on its own — test262 must also be sampled for any change touching the parser, interpreter, or built-ins. On every pull request CI runs PHPStan, PHPCS, the unit suite and the oracle regression; the WPT and popular-package suites run when a pull request changes src/ or their fixtures. The full test262 matrix is not part of pull-request CI, so sample the affected categories locally.

Why this works

Treating V8 as the source of truth removes nearly all spec-reading from the development loop. Disagreement with V8 always indicates a Phasis bug — never a spec ambiguity that can be argued. Once a fix lands, the corresponding test262 entries become permanent regression tests.

The 100 % pass rate is downstream of this discipline: every PR is judged against V8, CI rejects regressions in the oracle, WPT and popular-package suites, and the test262 matrix confirms the full-suite result when it is run.