Testing & CI
Test suites
| Command | Scope |
|---|---|
pnpm typecheck | tsc --noEmit over src/. |
pnpm test | Vitest unit/contract suites; six tests in two live Supabase files are skipped unless explicitly enabled. |
SUPABASE_RUN_LIVE_TESTS=true pnpm test | Enables the six additional live integration cases against an explicitly configured Supabase target. |
npx vite build | Production build (two chunks: main + observability). |
cargo test --workspace | Rust unit tests in both crates. |
pnpm research:conformance | Ten-seed, four-workload digest equality across direct, browser, Node worker, and replay adapters. |
pnpm research:test:service | Loopback service origin, token, schema, lifecycle, and path boundaries. |
pnpm research:test | Gymnasium checker, deterministic SDK behavior, and trajectory lifecycle. |
PLAYWRIGHT_OFFLINE_BYPASS=1 pnpm exec playwright test tests/e2e/research-runs.spec.ts | Research Runs UI without contacting hosted Supabase. |
TypeScript typecheck
pnpm typecheck runs tsc --noEmit with the strict settings from
tsconfig.json (strict, noImplicitAny, strictNullChecks, etc.). It does
not emit output: it only checks types. A clean clone must run
pnpm build:wasm first so the generated brain_sim module and declarations
exist; CI performs that step explicitly before typechecking.
Vitest
pnpm test runs Vitest (environment: 'node', includes
src/**/*.{test,spec}.ts).
Offline Research Alpha browser checks
Set PLAYWRIGHT_OFFLINE_BYPASS=1 for routine Research Alpha Playwright runs.
Global setup then skips .env loading and hosted Supabase user/session setup
entirely. This is stronger than merely omitting live-test flags and is the mode
used by the Research Alpha CI job.
The current suite includes the following core and chatbot-focused files. The table is a maintained highlight list rather than an exhaustive test-count ledger:
| Suite | Tests | Covering |
|---|---|---|
telemetry-contract.test.ts | 4 | TelemetrySnapshot shape contract |
cost.test.ts | 21 | Price book, regex priority, cost backfill, rollup |
auth-session.test.ts | 9 | Better Auth facade, legacy fallback, redirect normalization, and offline degradation |
evaluator-dal.test.ts | 7 | Evaluator + annotation config DAL offline paths |
session-compare.test.ts | 7 | Project state, session rollup, region activity |
typesense-client.test.ts | 7 | Typesense client offline degradation |
search-schema.test.ts | 8 | TRACE_SPANS_SCHEMA + DOCS_SCHEMA validation |
search-indexer.test.ts | 19 | Pure mapping helpers (rowToTraceDoc, markdownToDocs, etc.) |
search.test.ts | 15 | debounced multi_search, normalizeHits, buildSearchRequest |
search-box.test.ts | 4 | Search box offline degradation |
search-sync.test.ts | 10 | SyncQueue throttle, bulkUpsert REST client, JSONL formatter |
chat-client.test.ts | 7 | Backward-compatible text/image request shaping, catalog-version fallback, legacy SSE parsing, structured errors, and public-copy sanitization |
chat-models.test.ts | 8 | Live-catalog normalization, explicit image capability, friendly names/families/badges, hidden-id search, and technical-value masking |
prompt-playground-presentation.test.ts | 3 | Clean model labels, safe unknown-model names, and vendor-neutral failure copy |
research-chat.test.ts | 6 | Exact response version/shape, bounds, fallback, Journal slugs, and history migration |
research-prompt.test.ts | 1 | Scientific, privacy, telemetry, Journal, and exact-JSON prompt boundaries |
journal-grounding.test.ts | 2 | Deterministic relevance selection and the two-entry/6,000-character budget |
nine-router.test.ts | 12 | HTTPS/default validation, conservative provider/configured/combo image capability, multimodal bodies, redaction, resolution, cache, and stale fallback |
run-chat-request-validation.test.ts | 4 | Malformed body rejection, identifier/input bounds, image-only turns, and attachment-shape validation |
run-chat-image-attachments.test.ts | 4 | PNG/JPEG/WebP magic and dimension checks, count/size budgets, multimodal serialization, and metadata-only observability |
run-chat-context.test.ts | 4 | Text-segment accounting, unavailable image measurement, and provider prompt-usage truthfulness |
run-chat-errors.test.ts | 7 | Sanitized, retry-aware model/router/configuration/request/image diagnostics |
chat-saved-prompts-contract.test.ts | 3 | Saved-prompt SQL owner/non-anonymous predicates, authenticated action dispatch, and Edge Function JWT configuration |
chat-diagnostics.test.ts | 2 | Chat diagnostics redaction and payload bounding |
vercel-analytics.test.ts | 2 | Vercel Web Analytics and Speed Insights injector boot guards |
trace-to-brain.test.ts | 13 | Trace → brain mapping |
openinference-parser.test.ts | 7 | OpenInference attribute parsing |
live-brain.test.ts | 4 | Realtime live-brain bridge |
waterfall.test.ts | 4 | Waterfall node builder |
tracer.test.ts | 4 | SDK tracer flush + span builder |
error-reporting.test.ts | 3 | reportError routing |
dal-smoke.test.ts | 7 | DAL graceful offline degradation |
dal-live.test.ts | 4 | Live Supabase integration (env-gated, skipped by default) |
realtime-live.test.ts | 2 | Live Realtime end-to-end (env-gated, skipped by default) |
All DAL functions return Result<T>: the offline tests assert that every
function degrades to a typed supabase.not_configured error rather than
throwing.
Live integration tests (env-gated)
src/__tests__/dal-live.test.ts hits the real Supabase Cloud project to verify
seeded data shape. src/__tests__/realtime-live.test.ts verifies the Realtime
channel can deliver postgres_changes INSERT events end-to-end (writes a test
span, waits for delivery, then cleans up). Both suites are gated by
SUPABASE_RUN_LIVE_TESTS=true and skipped by default; in a normal pnpm test
run the 6 tests report as skipped (not failures), keeping the suite green.
Config prerequisite for Realtime test: the spans table must be in the
Supabase Realtime publication (supabase_realtime). This is already configured
via ALTER PUBLICATION supabase_realtime ADD TABLE spans;. If the test fails
with a FK/constraint error, verify publication membership with
SELECT * FROM pg_publication_tables WHERE pubname = 'supabase_realtime';.
Live tests use SUPABASE_SERVICE_ROLE_KEY (bypasses RLS) rather than the anon
key because the seeded demo project has owner_id = NULL and RLS policies
require is_project_owner(). This is appropriate for data-verification tests
and is never the client path.
Environment variables in tests
vitest.config.ts uses loadEnv(mode, process.cwd(), '') (from Vite) to
expose all .env variables into process.env.* via Vite's define transform.
This is required for live tests to access SUPABASE_SERVICE_ROLE_KEY, which is
not a VITE_-prefixed variable. The config reads .env, .env.local,
.env.test, and .env.test.local. Confirm secrets are appropriate before
running in CI.
loadEnv with empty prefix intentionally broadens the scope from the previous
config that passed no env vars. This is by design for the live test path: the
test harness only runs when explicitly opted in via the gate flag.
.env.test: a committed file that overrides VITE_TYPESENSE_* to empty
strings during Vitest runs. This ensures the offline degradation tests
(typesense-client, search-box, search) see "Typesense unconfigured"
regardless of the real .env credentials. vi.stubEnv cannot override
Vite's compile-time import.meta.env replacement, so .env.test is the
idiomatic Vite solution: Vite loads .env.test automatically in test mode,
applying the override before the define transform bakes in the values.
Rust tests
cargo test --workspace runs the unit tests in both crates:
rust/brain_sim: region classification, Bezier endpoint correctness, and signal-pool acquire/release.rust/brain_engine: the standalone scoring heuristics.
Continuous integration
.github/workflows/ci.yml runs on push and PR to main/master/revamp with
four jobs:
| Job | What it does |
|---|---|
typecheck | Runs pnpm typecheck. |
test | Runs Vitest. |
build | Installs Rust + the wasm32-unknown-unknown target + wasm-pack, runs pnpm build, and uploads dist. |
cargo | Runs cargo build and cargo test --workspace --locked. |
The build job mirrors a clean local build: it provisions the Rust toolchain
before building because pnpm build compiles the WASM module first.
Recommended pre-commit check
Before opening a PR, run all four locally:
pnpm typecheck
pnpm test
npx vite build
cargo test --workspace
When live credentials are set, also run the gated integration tests:
SUPABASE_RUN_LIVE_TESTS=true pnpm test
For a full end-to-end check that also exercises the WASM and Vite pipeline,
run pnpm build (compiles Rust → WASM, then runs Vite).