The tests

This guide shows you how to run a workspace’s tests: from source or from the build, with coverage, in a browser, as Storybook stories, and against a web standard’s own tests.

Every package’s tests run from the root, in one run of Vitest. Each package’s vitest.config.ts is a project, next to the root project that runs the repo’s own tests in test/.

The repo names vitest and @vitest/coverage-v8 at its root, plus @vitest/browser-playwright and playwright if it has browser tests. Each is named by one exact version, the one @wtf/project-vitest takes as its peer, 5.0.5. That way the vitest bin, the tests’ imports of vitest and the coverage provider all come from the Vitest that runs. Vite comes from @wtf/project-vite, which @wtf/project-vitest depends on, and a repo names vite only where its own code imports it: see the Vite guide.

import { packageConfig } from '@wtf/project-vitest/config'

// A package's vitest.config.ts gives only what differs.
export default packageConfig({ include: ['test/**/*.test.{ts,tsx}'] })

packageConfig(test, config) takes test options, each over its own default, and optionally a Vite config, which it merges into its own the way Vite’s mergeConfig does: your aliases first, so they win, and your other lists after its own, as for server.fs.allow or test.server.deps.inline. Every config of the package (packageConfig, rootConfig, integrationConfig, browserConfig, browserRootConfig and storybookConfig) returns a promise, because it reads the workspace through the platform’s file system handles. Vitest awaits a config file’s default export, and a test of a config awaits it too.

import { packageConfig } from '@wtf/project-vitest/config'

export default packageConfig(
  { include: ['test/**/*.test.{ts,tsx}'], environment: 'jsdom' },
  { resolve: { dedupe: ['react'] } }
)

Run from the source, or from the build

pnpm test reads every package from its source, with nothing built first. Vite resolves each workspace package through an alias from each export to its @wtf/source. What Node loads itself, such as a Vitest config or a module oclif imports, goes through the resolve hook the test script gives Node:

NODE_OPTIONS="$NODE_OPTIONS --import=@wtf/project-resolve/register" vitest run --configLoader native

--configLoader native makes Node load each config, so its imports go through the hook. The hook adds @wtf/source for the workspace’s own packages only, and resolves a relative import written ./x.js or ./x to its .ts.

The workspace’s packages are those in packages/ that the packages globs of pnpm-workspace.yaml name, minus their ! exclusions, as pnpm reads them. A folder there that the globs don’t name, such as a git submodule of another repo, keeps using its build, for the hook and for Vite’s aliases alike. If the globs name none, every folder in packages/ is a package. Each test worker registers the hook too, through packageConfig’s setup file, in any environment: for a browser-like one such as jsdom, Vite serves the workspace and the hook’s package, wherever the hook is installed from.

pnpm run test:dist (TEST_FROM=dist) reads every package as it is published, through its export map to its dist, and a package’s own tests load its modules from dist too. Run it after a build. CI runs every suite this way, and you run the source; the testing guide shows the pair of scripts each suite needs.

Measure coverage

--coverage measures each source file in packages/*/src, counting a module no test loads as 0%. It leaves out stories (*.stories.*), since no unit test runs them as code of their own. When you run from the build, the source maps in dist map each result back to its source, using a v8 provider that counts no module twice. @vitest/coverage-v8 is an optional peer.

A run filtered by --project, such as vitest run --project @scope/lib --coverage, measures each project it runs from that project’s own folder, as Vitest does. The coverage patterns are then relative to the package’s folder (src/** instead of packages/*/src/**), so the report holds the sources of what ran. A filter that also runs the root’s project, the guards, measures from the root, the same as a run of every project.

Set the options of a root config

rootConfig, integrationConfig and browserRootConfig take the same options, all optional:

Option	Default	What it does
`folders`	`['packages']`	The folders that hold the packages; each package is a project
`projects`	`[]`	Extra projects, after the packages’
`coverage.include`	the `src` of each package in `folders`	What `--coverage` measures; a module no test loads counts as 0%
`coverage.exclude`	each package’s stories, `.stories.`	What it leaves out, after the source maps too, on top of the defaults
`skip`	none	Packages in `folders` the run leaves out, each by its folder, as `packages/site`: no project, no coverage
`reportsDirectory`	`'reports'`	Where the JUnit report and the coverage go
`junit`	the suite’s report in `reportsDirectory`, as `reports/test-results.xml`	The JUnit report’s path, for a repo that keeps it apart from the coverage

rootConfig takes one more option, testTimeout: how long a test in the root project (the guards) may take before it fails. The default is GUARD_TIMEOUT, 30 seconds, not Vitest’s 5, because a guard that reads the whole workspace takes seconds while every project runs at once. Each package’s tests use their own config’s timeout. The guards’ project extends nothing from the root config (extends: false): it resolves on its own, so the build plugin and each alias apply to it once.

import { rootConfig } from '@wtf/project-vitest/config'

export default rootConfig({
  skip: ['packages/site'],
  reportsDirectory: 'coverage/unit',
  junit: 'coverage/unit-junit.xml'
})

Start a process from a test

childEnv() of ./process returns the test’s environment without the resolve hook in NODE_OPTIONS. A process started in a folder of its own, where the hook is not installed, would fail to start with it. childEnv() is withoutHook(process.env) of @wtf/project-resolve/env; a tool that starts a process outside a test calls that with the environment it is given.

import { spawnSync } from 'node:child_process'

import { childEnv } from '@wtf/project-vitest/process'

spawnSync('pnpm', ['--version'], { env: childEnv() })

Run browser and integration tests

A browser test is named *.browser.test.{ts,tsx}. browserConfig() and browserRootConfig() of ./browser run them in Chromium with Playwright; both are optional peers. An integration test is named *.integration.test.ts; run them with integrationConfig() at the root and packageConfig({ ...integration }) in a package, one file at a time. A unit run picks up neither.

Browser tests also run from the build, as the unit suite does with test:dist. The root’s test:browser:dist is TEST_FROM=dist vitest run --configLoader native --config vitest.browser.config.ts. With --coverage, it measures what ran of dist, mapped back to the sources by the source maps beside it, using v8’s browser module, which counts no module twice:

{
  "scripts": {
    "test:browser": "NODE_OPTIONS=\"$NODE_OPTIONS --import=@wtf/project-resolve/register\" vitest run --configLoader native --config vitest.browser.config.ts",
    "test:browser:dist": "TEST_FROM=dist vitest run --configLoader native --config vitest.browser.config.ts"
  }
}

pnpm run test:browser:dist --coverage gives the browser tests’ coverage, after a build. The integration suite takes the same pair, test:integration and test:integration:dist, with --config vitest.integration.config.ts. CI runs the :dist script of each suite: see the testing guide.

Run stories as tests

A package of stories (a Storybook) runs each story as a test in headless Chromium, with storybookConfig() of ./storybook as its vitest.config.ts. It is browserConfig() with two differences: its project is named storybook, and its only tests are its stories. The @storybook/addon-vitest plugin, storybookTest(), supplies the files its Storybook picks up, and Vite adds them to an empty include, so no *.browser.test.{ts,tsx} file is among them.

The package passes in the plugin, built from its own .storybook folder, as storybook. The plugin applies what .storybook/main.ts gives Vite through viteFinal, so the stories compile with the same plugins and conditions as in Storybook, and nothing is given twice. Use plugins only for a plugin the tests need beyond those; Storybook’s plugin comes after them. Storybook applies .storybook/preview.ts itself, so you need no setup file for it:

import { resolve } from 'node:path'

import { storybookConfig } from '@wtf/project-vitest/storybook'
import type { Plugin } from 'vitest/config'

declare const storybookTest: (options: { configDir: string }) => Plugin // import { storybookTest } from '@storybook/addon-vitest/vitest-plugin'

export default storybookConfig({
  storybook: storybookTest({ configDir: resolve(import.meta.dirname, '.storybook') })
})

Nothing here depends on Storybook. The repo installs @storybook/addon-vitest at its Storybook’s version (one that supports Vitest 5), plus @vitest/browser-playwright and playwright, as for browser tests. Stories need a browser, so keep them out of pnpm test: the root’s rootConfig({ skip: ['packages/storybook'] }) leaves the package out, and a test:browser script runs it through a vitest.browser.config.ts of browserRootConfig({ projects: [...] }), in CI on the Playwright image.

Standard tests

Where you implement a web standard, the standard’s own tests, the Web Platform Tests (https://github.com/web-platform-tests/wpt), show whether you do. wptSuite of @wtf/project-wpt/suite runs them in the repo’s Vitest run, each subtest of each test file as a test. They run against the implementation the repo maps in, and never touch the process’s own globals:

import { nodeSystem } from '@wtf/project-wpt/node'
import { wptSuite } from '@wtf/project-wpt/suite'

declare const MyDecoderStream: unknown // import { MyDecoderStream } from '../src/decoder.ts'

wptSuite('encoding-streams', nodeSystem(), {
  commit: '4ad507b28db0a636f528def9161d109fe11b93a3',
  dirs: ['encoding/streams'],
  globals: () => ({ TextDecoderStream: MyDecoderStream }),
  expectations: new URL('./wpt/encoding-streams.json', import.meta.url)
})

nodeSystem() of @wtf/project-wpt/node gives the suite what it takes of Node: the process’s environment, timers and working folder, the disk, git, a node:vm context for each file, and the process’s unhandled rejections.

Pin the tests and choose what to fetch

The suite’s options pin the WPT to a commit and name its tests:

Option	What it is
`commit`	The WPT commit, as a full hash: moving it is a change to the repo
`repository`	The WPT’s git repository; the WPT’s own on GitHub by default
`files`	Test files, from the root of the WPT, each run as it is
`dirs`	WPT folders; each `*.any.js` file in them, and in their subfolders, that runs outside a window
`resources`	WPT files and folders the tests fetch, such as `interfaces/*.idl` for idlharness.js, which `file.fetch` serves
`expectations`	The suite’s known failures: a JSON file, by its path from `root` or its URL, or the object
`run`	The name of the run, of a suite held to more than one implementation
`root`	The project’s folder, whose `node_modules/.cache/wpt` keeps the checkouts: the working folder by default
`timeout`	How long each file may take, in milliseconds

What a test fetches is not read from its source, because a test builds its URLs as it runs: name it in resources. To share one commit among the suites of a repo, keep it in a constant of your own and pass it to each.

When Vitest collects a suite, it checks out whatever it lacks of the WPT with git, into node_modules/.cache/wpt/<commit>. That is a shallow, blobless fetch of the commit and a sparse checkout of just the suite’s folders and files, resources/testharness.js, and each script that a file’s // META: script= lines load. A script the WPT’s server serves at another path, such as /resources/WebIDLParser.js for idlharness.js, is checked out as the file it serves. A path asked for later is added to the checkout. Once everything is there, nothing runs git and the run is offline. The WPT is never committed to the repo: a CI job that runs offline caches the folder.

What a test file sees

Each test file runs in a realm of its own; in Node that is a node:vm context (nodeRealm of @wtf/project-wpt/node, which nodeSystem() passes as realm). Its globals are built from the layers below, each over the one before; a name the repo maps to undefined is removed:

the language’s built-ins, from the context’s own realm;
intrinsics() of ./builtins: the errors, buffers and Promise of the realm the implementation runs in, so assert_throws_js(TypeError, …) matches the TypeError it throws;
builtins(): the values and constructors of the WinterTC Minimum Common Web API that Node has (Web Streams, URL, TextEncoder, AbortController, DOMException, Event, structuredClone and the others), as they are; no service such as fetch, crypto or performance;
the harness’s host: self, the context’s global; location, the file’s address, such as http://web-platform.test:8000/encoding/api-basics.any.js; GLOBAL; the platform’s timers, which the run clears when the file ends; and addEventListener, through which the harness hears what no script catches;
whatever the repo’s globals(file) returns, made fresh for each file: the implementation under test, such as BroadcastChannel, navigator: { locks }, URLPattern or console. file.fetch is a fetch of the checkout’s files, for a test that loads its data, as url/url-constructor.any.js does; its resources are checked out for it.

testharness.js runs first, with its results hooked, then the file’s scripts and the file itself, as one evaluation, so the harness starts once all are there. A file has 10 seconds, or 60 with // META: timeout=long, or the suite’s timeout; subtests not done by then are a TIMEOUT. An uncaught error, or an unhandled rejection while the file runs, fires the context’s error or unhandledrejection event, as in a browser, and the harness reports ERROR, unless the file allows them with setup({ allow_uncaught_exception: true }), as web-locks/held.https.any.js does. Neither ever becomes a Vitest error: while a file runs, the process’s own rejection listeners are set aside. Each // META: variant= is a test of its own, with the path and the variant as its id. A window test (a *.window.js or *.html file) does not run, because a context has no document.

List the known failures

A suite’s known failures live in a JSON file, keyed by each test file’s id. For each, it says why the file doesn’t run (skip), why its harness doesn’t end OK (harness), or which subtests fail, by name, and why (fail). A variant’s id, such as encoding/api-invalid-label.any.js?1-1000, takes precedence over its path’s.

{
  "encoding/streams/decode-utf8.any.js": {
    "fail": {
      "decoding a transferred ArrayBuffer chunk should give no output": "no transfer of a buffer yet"
    }
  },
  "encoding/streams/backpressure.any.js": { "skip": "hangs" },
  "url/url-constructor.any.js": { "harness": "throws at its end" }
}

Each subtest is a test, by its name, inside a describe named by its file’s id, next to a test of its harness. A failing subtest passes its test when the expectations list it, and fails it when they don’t (SUBTEST_FAILED). A listed subtest that passes fails its test (UNEXPECTED_PASS), so a fix takes it off the list, and the list only shrinks. A skipped file is a test Vitest skips, and its // META: script= scripts are neither fetched nor looked for. One more test fails if the expectations name a file or a subtest the suite doesn’t have (STALE_EXPECTATION).

To hold two implementations to one suite, such as a test double and Node’s own, run it twice: one wptSuite per run, each with its own expectations. Each run’s tests are named after the run, and each fails on a listed entry that passes or that it doesn’t have.

import { nodeSystem } from '@wtf/project-wpt/node'
import { wptSuite } from '@wtf/project-wpt/suite'

const system = nodeSystem()
const locksSuite = { commit: '4ad507b28db0a636f528def9161d109fe11b93a3', dirs: ['web-locks'] }

declare const MemoryLocks: unknown // import { MemoryLocks } from '../src/memory.ts'
declare const locks: unknown // a LockManager of the platform

wptSuite('locks', system, {
  ...locksSuite,
  run: 'doubles',
  globals: () => ({ navigator: { locks: MemoryLocks } }),
  expectations: 'test/wpt/locks.doubles.json'
})
wptSuite('locks', system, {
  ...locksSuite,
  run: 'node',
  globals: () => ({ navigator: { locks } }),
  expectations: 'test/wpt/locks.json'
})

This is how Node’s core runs the WPT, with a status file per module, and how Deno does, with one expectation file: a known failure is listed, and its passing fails the run. Both run the tests in the runtime’s own global, since the runtime is the platform under test. wpt-runner runs them in jsdom, with its window as the global a test installs into. Your implementation is a module, not the platform, so here each file runs in a context of what the repo maps in, and the test process’s global stays untouched.