Skip to content
brix.v2.15.0+b120c49

Search documentation

Loading search…

    Documentation menu
    Docs / Testing and tooling / TestingRead Markdown
    On this page

    End-to-end tests and hydration benchmarks

    Use managed Playwright suites for complete application behavior and the Node-only hydration benchmark for controlled comparisons between equivalent production deployments.

    Prefer route and component tests for isolated behavior. Add E2E when the requirement crosses the real server/browser boundary, such as navigation, SSR-to-hydration behavior, or production service integration. Managed defaults support full-stack Preact applications and managed React applications; full-stack suites also need a provider-free test-run stage. Libraries need an explicit webServer.

    Test the complete application

    #

    Add e2e/products.spec.ts and import the managed fixtures through Brix:

    import { expect, test } from "brix/test/e2e";
    
    test("opens a product", async ({ page }) => {
      await page.goto("/products/sku-123");
      await expect(page.getByRole("heading", { name: "Everyday backpack" })).toBeVisible();
    });
    

    No extra dependency, config, package script, or install command is required for an application. Brix starts and stops brix dev for the project, prepares Chromium only when tests execute, and stores failure traces and screenshots under .brix/e2e/results. Tests in e2e/ run fully parallel in headless Chromium with the list reporter.

    The managed server listens on a free 127.0.0.1 port that Brix selects once per run and shares with Playwright workers, so parallel checkouts and other local servers do not collide. Set BRIX_E2E_PORT to a TCP port from 1 to 65535 to pin it. The server never reuses an existing process: a port that is already in use fails the run instead of testing an unrelated server. A full-stack server is ready when brix dev has loaded the application and answers its framework-owned /__brix/dev-ready probe, so the application need not serve /; a managed React server is ready when / responds. Server startup may take up to two minutes before the run fails.

    When CI is set, managed defaults retry each failing test twice and reject committed .only calls; locally tests run once. Workspace-root brix test sets CI=true for member runs, so it uses the CI policy too. Set retries or forbidOnly in brix.e2e.ts to choose another policy.

    For a full-stack application, the server inherits an already resolved environment; otherwise managed E2E loads the test stage once in the Playwright runner, including local and personal overrides. Workers reuse the running server and do not load it again. That load can resolve providers even without --env. Keep env/.env.test-run provider-free for offline suites and prevent local files from selecting production services. Use brix test --env --no-local --project e2e for an explicit isolated stage, or select another stage deliberately for integration coverage. Managed React applications do not use this server environment loader. A custom webServer owns its own loading.

    Managed traces capture DOM, accessibility (ARIA), and screen snapshots around actions and retain the trace only when a test fails. Successful tests still incur capture overhead, but their traces are discarded. Use Playwright's trace viewer to compare the action, rendered page, and accessibility tree. Traces can contain page text, screenshots, network data, and credentials; use disposable test data and review artifacts before sharing them. Brix does not redact traces.

    brix test e2e/products.spec.ts -t "opens a product"
    brix test --project e2e
    brix test --project e2e --list
    brix test --project e2e --headed
    brix test e2e/products.spec.ts --debug
    

    Unqualified brix test and brix verify include discovered e2e/ suites. -t maps to Playwright's --grep after E2E selection. Keep a focused selection within one runner and workspace member.

    The default suite runs against development output, not a production bundle. Use a configured production service or deployed candidate when testing platform bindings, production asset URLs, cache policy, or behavior that differs after bundling. A passing development suite is not a deployment smoke check.

    Customize the E2E service and browser

    #

    Add brix.e2e.ts only when the managed defaults are insufficient:

    import { defineBrixE2EConfig } from "brix/test/e2e";
    
    export default defineBrixE2EConfig({
      webServer: [],
      use: { baseURL: "https://staging.example.test" },
    });
    

    The helper accepts native Playwright configuration. Top-level options such as testDir, outputDir, retries, reporter, and projects replace the managed values. webServer replaces the managed server, its port selection, and its environment loading, so also set use.baseURL; otherwise it defaults to http://127.0.0.1:4173 (or BRIX_E2E_PORT). use merges with managed browser, trace, and screenshot defaults. You can also configure browser projects, retries, reporting, or a custom test directory. Select --project e2e explicitly when focusing a file outside conventional e2e/ discovery.

    An explicit use.trace replaces the entire trace policy. Set it to "off" to disable traces, or use { mode: "on-first-retry", snapshots: { dom: true, aria: true, screen: true } } to capture only a first retry (configure retries when running locally). Screenshots have a separate policy; set use.screenshot: "off" too when artifacts must not capture page contents.

    Existing playwright.config.* files remain supported, but their authors own server settings and tests must still import from brix/test/e2e for automatic browser preparation. An existing test:e2e package script overrides automatic full-suite dispatch and must not call unqualified brix test recursively. Remote browsers, custom executables, and branded browsers remain caller-provisioned.

    Use E2E tests for navigation, real browser/server integration, and behavior before hydration. Use the route harness for fast server-route coverage and component tests for isolated browser interactions.

    Playwright behavior to keep in mind

    #
    • Device descriptors forward screen, which affects window.screen and media queries. Brix's plain Chromium default is not a desktop device descriptor; review responsive assertions and screenshots when you add one.
    • JSX in E2E files compiles with the jsx, jsxFactory, jsxFragmentFactory, and jsxImportSource settings of the selected tsconfig, defaulting to React's automatic runtime when none are set. Fix Preact test compilation in the owning project's configuration instead of adding React.
    • --update-snapshots=missing passes tests that only create missing snapshots, while the normal default mode writes them and fails. Keep CI's acceptance policy explicit and review new baselines rather than treating a green creation run as comparison evidence.
    • Elements inside hidden iframes are hidden for actions and visibility assertions. Reveal the intended frame before interacting; do not force actions past a real visibility requirement.

    Optional upstream capabilities, such as video frame-rate options, WebP screenshots, and experimental WebMCP, are not Brix defaults. Browser-version changes in a Brix upgrade can alter snapshots; review them with application behavior and disposable fixture data.

    Compare hydration architectures

    #

    runHydrationBenchmark from brix/test/benchmark compares selective islands with a full-page island using caller-supplied production URLs. It does not modify source, build, start, deploy, instrument, or rewrite applications.

    Prepare two checkouts with the same Brix pin, lockfile, data, locale, CSS, environment, and application code. Keep selective islands in one. In the other, add "use island"; to the actual page component only if its browser graph contains no server-only code. Build and serve both production applications at separate URLs.

    Create an application script:

    import { expect, runHydrationBenchmark } from "brix/test/benchmark";
    
    const islandsURL = process.env.ISLANDS_URL;
    const fullPageURL = process.env.FULL_PAGE_URL;
    if (!islandsURL || !fullPageURL) throw new Error("Set benchmark URLs");
    
    await runHydrationBenchmark({
      variants: [
        { name: "islands", baseURL: islandsURL },
        { name: "full-page", baseURL: fullPageURL },
      ],
      runs: 10,
      scenarios: [
        {
          name: "product",
          path: "/products/example",
          controls: { quantity: '[data-testid="quantity"]' },
          verify: async (page) => {
            const quantity = page.getByTestId("quantity");
            await quantity.fill("2");
            await expect(quantity).toHaveValue("2");
            return { quantity: await quantity.inputValue() };
          },
        },
      ],
    });
    

    Run it with brix run scripts/benchmark-hydration.ts. The promise resolves to the report also written to results.json.

    Required options:

    • variants: at least two, each with a unique nonempty name and an HTTP(S) baseURL without embedded credentials.
    • scenarios: at least one, each with a unique name, a path that stays on every variant's origin, a nonempty controls map of name to selector, and a verify callback.

    Optional settings and defaults:

    • variants[].context: Playwright browser-context options for that variant. Keep them identical unless the difference is deliberate.
    • variants[].setup: optional async (context) => void for cookies or authentication before navigation. Do not install request routing.
    • runs: 10 repetitions per condition; must be a positive integer. Use 1 only for smoke verification.
    • cache: ["cold", "warm"]; any nonempty subset without duplicates. Cold samples clear the browser cache first; warm samples prime by navigating and activating the scenario first.
    • profiles: hydrationBenchmarkProfiles, which are desktop (no throttling), 4× cpu, and constrained (4× CPU, 80 ms latency, 200,000 B/s down, 93,750 B/s up).
    • timeoutMs: 30000 for browser waits, actions, and navigation; not a deadline for your callbacks.
    • outputDir: .brix/benchmarks/hydration, with a unique <timestamp>-<uuid> subdirectory per invocation.

    Select presets with profiles: { cpu: hydrationBenchmarkProfiles.cpu }, or define { cpu, network? } profiles: cpu is a throttling factor of at least 1, and network takes latencyMs (zero or more) plus positive downloadBytesPerSecond and uploadBytesPerSecond. Omit network for no throttling. Invalid options fail before the browser launches. Ten runs × two variants × three profiles × two cache modes is 120 measured navigations per scenario, run sequentially, plus warm priming.

    Control selectors must uniquely identify SSR elements inside Brix island boundaries. The probe associates them before mount and observes existing boundary events. It supports repeated and multi-root boundaries and independent slot children; a parent event does not make an unhydrated child ready. Client-only elements, portals, static controls, and elements absent from initial HTML are not valid readiness probes.

    verify runs after readiness measurement and must return the same stable JSON-compatible functional snapshot for every sample and variant. Browser errors, ambiguous controls, missing readiness, assertion failures, and snapshot differences fail the run. Verification interactions are excluded from startup timing, so use E2E tests—not the benchmark—to specify pre-hydration behavior or interaction latency.

    For lazy or idle controls, add a separate deferred object with named controls and an activate(page) callback. Deferred control names must differ from the scenario's controls. Its readiness includes caller-driven activation and is not automatic startup time. Avoid Playwright request routing because it disables HTTP caching and invalidates warm comparisons.

    Interpret benchmark output

    #

    summary.md reports median important-control readiness, script execution, JavaScript transfer, and per-control readiness. results.json retains raw samples, environment/profile metadata, boundary events, completion timing, TTFB, FCP, payload sizes, resource inventories, and long-task excess. A failed run still writes results.json with status: "failed" and the error, but no summary.md, then rejects.

    Boundary readiness means synchronous mount completion—not paint, effects, data readiness, INP, or responsiveness. Resource Timing zeros can indicate cache hits or unavailable cross-origin timing; configure Timing-Allow-Origin where appropriate and compare encoded, decoded, and transferred values together.

    One run is a smoke check only. Keep the machine quiet, use non-mutating or isolated test accounts, retain compared revisions, and review reports for errors or sensitive paths before sharing them.