This is the full developer documentation for Autonoma # Introduction > Autonoma is an agentic end-to-end testing platform. Connect your repo and every pull request gets reviewed automatically on a live preview environment - no test scripts to write or maintain. ![An AI testing agent reviewing a web app - reading the code change, running the app, and checking the database](/img/introduction/hero.jpg) Autonoma is an agentic end-to-end testing platform. Connect your repository and every pull request gets reviewed automatically on a live preview environment - an AI agent exercises your app in a real browser, checks that it still works, and reports back on the PR. You don't write or maintain test scripts. There are no selectors to update when a button moves. Autonoma generates a suite of natural-language tests from your codebase, and an agent figures out how to run them - picking elements, making assertions, and healing itself when your UI changes. ## How it works Every pull request runs through the same loop, from opening the PR to a reviewed result: ![The Autonoma review loop: pull request, preview environment, seed data, run tests, review on PR](/img/introduction/test-lifecycle.jpg) 1. **Pull request** - you open a PR (or push a change to one). 2. **Preview environment** - [Previewkit](/previewkit/) builds an isolated, full-stack preview of your app for that PR and gives it a live URL. 3. **Seed data** - the [Environment Factory](/environment-factory/) creates fresh, isolated test data so every run starts from a known state. 4. **Run tests** - the agent runs your test suite against the preview, exercising the app in a real browser. 5. **Review on PR** - Autonoma reviews each run, confirms what passed and what broke, and comments the result back on the pull request. When the PR closes, the preview environment and its data are torn down automatically. ## What you set up Autonoma gets you value in two stages: **connect your repo** to go live with preview environments and automated PR reviews, then **deepen coverage** by generating a test suite and wiring up real test data. Three pieces, set up in order: 1\. Preview Environments Install the GitHub app and configure your stack. Every PR then gets a live, isolated preview - and an automated review. This alone gets you live. [Set up Preview Environments →](/previewkit/) 2\. The Planner CLI Reads your codebase and generates a complete, natural-language E2E test suite - pages, flows, and scenarios. One command. [Run the planner →](/test-planner/) 3\. The Environment Factory One endpoint in your backend that creates isolated test data before each run and tears it down after, so tests always start clean. [Set up the Environment Factory →](/environment-factory/) The Environment Factory has an SDK for every major stack - TypeScript, Python, Elixir, Java, Ruby, Rust, Go, and PHP. [Copy a working endpoint for yours.](/environment-factory/examples/) ## Using these docs with AI Every page is available as plain text for coding agents. Point Claude Code, Cursor, or Copilot at the file below and it can pull in exactly the pages it needs: ```plaintext https://docs.autonoma.app/llms.txt ``` A single [complete file](/llms-full.txt) with all pages concatenated is also available. ## Contributing Want to run Autonoma locally or work on the platform itself? Development setup Clone the repo, install dependencies, and get the platform running locally. [Get started →](/development/setup/) Architecture overview How the monorepo fits together and the key design decisions behind it. [Read the overview →](/development/architecture/) # Previewkit > Vercel-style preview environments for every pull request. Configure your stack in the Autonoma dashboard, open a PR, get a live URL. Previewkit gives every pull request its own live, isolated, full-stack preview of your app. It's the foundation Autonoma reviews run against - and the first thing you set up when you connect a repo. ![A pull request gets an isolated preview environment with its apps, database, and cache, and a live URL posted back to the PR](/img/previewkit/lifecycle.jpg) You describe your stack once - apps, the services they depend on, and their environment variables - and Previewkit handles the rest: building the containers, provisioning the supporting services, wiring environment variables, and posting the URL back to the PR. ## How it works Once the Previewkit GitHub App is installed on your repository, every `pull_request` event triggers the pipeline: 1. **Opened / synchronized / reopened** - Previewkit fetches the head commit, builds each app, provisions service recipes (Postgres, Redis, etc.), deploys to a dedicated Kubernetes namespace, and comments the preview URL on the PR. 2. **Closed** - Previewkit deletes the namespace and all resources tied to that PR, then updates the comment. Each preview gets a stable, unguessable URL - a short hash derived from the service name, PR number, and repo, so the same PR always resolves to the same address. One PR may expose several apps, each with its own hostname under `preview.autonoma.app`. A repository can also have a standing **main-branch environment**: a preview deployed from the repository’s main branch instead of a PR. Once it exists, every push to that branch redeploys it at the new head automatically, the same way a new commit updates a PR’s preview. ## What you configure You set up your stack in the Autonoma dashboard (the Previewkit onboarding flow), which walks through four steps - **Apps**, **Services**, **Env vars and secrets**, and **Hooks** - and saves the configuration for your repository. It declares: * **Apps** to build and deploy (each becomes a public HTTPS URL) - see [Apps and builds](/previewkit/apps/) * **Services** the apps depend on (databases, caches, etc.), picked from a curated catalog of recipes * **Environment variables and secrets** for each app and service, with templates that resolve service hostnames at deploy time and a per-row toggle to mark a value as a secret * **Hooks** that run after deploy (typical use: database migrations) ## How apps are built Each app builds one of two ways, chosen per app: * **Manual** - pick a runtime (Node, Python, Go, and more), then write a short bash build script and an entrypoint. No Dockerfile required. * **Dockerfile** - point Previewkit at an existing Dockerfile in your repo, built with [BuildKit](https://github.com/moby/buildkit). Either way, images are pushed to a private registry and pulled by the preview cluster - you never touch credentials. See [Apps and builds](/previewkit/apps/) for the full reference. ## Secrets Secrets such as API keys and third-party tokens are stored encrypted and kept out of your stack configuration. Flag any value as a secret with the per-row toggle in the onboarding **Env vars and secrets** step, or manage them out-of-band via the REST API (handy for CI and rotating values without editing the config). They can be owner-scoped (every PR sees them) or PR-scoped (just this PR, useful for testing prod credentials in isolation). Previewkit also injects a few [built-in environment variables](/previewkit/secrets/#built-in-environment-variables) (`AUTONOMA_PREVIEWKIT`, `AUTONOMA_PREVIEWKIT_PR`, `AUTONOMA_PREVIEWKIT_URL`) into every preview so your app can detect it’s running in a preview. See [Secrets](/previewkit/secrets/). ## What’s next * [Apps and builds](/previewkit/apps/) - build methods, runtimes, and per-app settings * [Multiple repositories](/previewkit/multirepo/) - pull apps from more than one repository * [Manage secrets](/previewkit/secrets/) - REST API reference # Apps and builds > How Previewkit turns each app in your repo into a running container - the build method, the runtime catalog, build context, and the per-app settings. An app is the unit Previewkit builds and deploys: a piece of your repo that becomes a container with its own public HTTPS URL in every preview. This page covers how each app is built and the settings on its card. Most projects have one app - your web server. A repo can declare several (a frontend, an API, a worker), and each is configured the same way: pick how it builds, then fill in a few fields about how it runs. ## Build method Every app builds one of two ways, chosen with the **Build method** toggle at the top of the app card: | Method | Use it when | What you provide | | -------------- | ------------------------------------------------------------------- | -------------------------------------------------------------------- | | **Manual** | You don’t have a Dockerfile, or you want a fast, transparent build. | A runtime (Node, Python, …), a bash build script, and an entrypoint. | | **Dockerfile** | Your repo already has a Dockerfile you trust. | The path to that Dockerfile. | Manual is the default because it needs nothing in your repo - you pick a language and describe the build in two boxes. ### Manual builds A manual build starts from a language image, installs your dependencies with a build script, and runs your app with an entrypoint. You pick a **runtime** from the catalog: | Runtime | Base image | Default version | | ------- | -------------------------------- | --------------- | | Node.js | `node:{version}-bookworm-slim` | 22 | | Python | `python:{version}-slim-bookworm` | 3.12 | | Go | `golang:{version}-bookworm` | 1.22 | | Rust | `rust:{version}-slim-bookworm` | 1.77 | | Java | `eclipse-temurin:{version}-jdk` | 21 | | Ruby | `ruby:{version}-slim-bookworm` | 3.3 | | PHP | `php:{version}-cli-bookworm` | 8.3 | | C / C++ | `gcc:{version}-bookworm` | 13 | | Debian | `debian:{version}-slim` | bookworm | Pick **Debian** when you want a bare base image and will install everything yourself. Any published tag works in the **Version** field - the default is only a starting point, so a repo pinned to an older toolchain is never forced onto ours. Two boxes describe the build: * **Build script** - bash that runs at image build time, from the repo root. Selecting a runtime prefills a sensible default (for Node, `npm install` then `npm run build`). It’s optional - leave it blank for an app that needs no build step. * **Entrypoint** - the command that starts the container (for Node, `npm start`). The **Build spec** panel on the right previews exactly what you’ll get: the runtime and version, the resolved image, the build context (**the repo root**), the working directory (`/workspace/`), and the entrypoint. Because a manual build copies the whole repo, you don’t set a Path or build context for it - the build script and entrypoint define everything. Every manual runtime also ships a common toolbelt so your scripts have what they need without an install step: `git`, `curl`, `wget`, `jq`, `rg`, `make`, `ssh`, `tmux`, `sqlite3`, `tar`, `zip`, and `unzip`, plus the language’s own tools (for Node, `npm`, `pnpm`, and `yarn`). ### Dockerfile builds If your repo already has a Dockerfile, pick **Dockerfile** and give its path. Previewkit builds it with [BuildKit](https://github.com/moby/buildkit), pushes the image to a private registry, and pulls it into the preview - you never handle registry credentials. The Dockerfile path is resolved **relative to the build context**, which is where the two location fields below come in. ## Dockerfile build settings These fields appear only for Dockerfile builds (a manual build always uses the repo root): * **Path** - the directory of the app inside the repo, e.g. `apps/web`. It sets the default build context and is checked to exist in your repo. In a single-app repo, leave it blank for the root. * **Root directory** - the Docker **build context**: the folder Docker builds from, and everything a `COPY` in your Dockerfile can read. Leave it blank to inherit **Path**; set it explicitly when a Dockerfile deeper in the repo needs to `COPY` files from higher up. * **Start command** - overrides the container’s default start command. For a monorepo web app whose Dockerfile lives at `apps/web/Dockerfile` but copies a shared `packages/` folder from the repo root, set Path to `apps/web`, Root directory to `.` (the repo root - not blank, since blank inherits Path), and the Dockerfile path to `apps/web/Dockerfile` (it resolves relative to the build context). ## Per-app settings A few fields apply to every app, whichever build method you pick: | Field | What it does | | ---------------- | --------------------------------------------------------------------- | | **Name** | Lowercase identifier used in resource names and the preview URL. | | **Port** | The port your app listens on inside the container. | | **Health check** | A path Previewkit requests to confirm the app is up (e.g. `/health`). | ### The frontend app When a project has more than one app, one is marked the **frontend** with a toggle. The frontend is the app Autonoma’s agents open in the browser to test, and its URL becomes the preview’s primary URL. Each project has exactly one frontend; the others still get their own URLs but aren’t the entry point. ### Depends on Once your project pulls in a [connected repository](/previewkit/multirepo/), each app gets a **Depends on** control for start ordering: the app waits for the apps and services it lists before it starts. Use it when, for example, your frontend shouldn’t boot until an API from another repository is reachable. ## Next steps * [Environment variables and secrets](/previewkit/secrets/) - wire config and credentials into each app * [Multiple repositories](/previewkit/multirepo/) - pull apps from more than one repository into the same preview # Multiple repositories > Deploy your frontend and the apps it depends on from more than one repository into a single preview, and control which branch of each connected repository gets built. Autonoma tests a pull request by opening one app in a browser - the frontend. The apps and services behind it can live in a single repository or several; when they span repositories, Previewkit pulls them all into the same preview. ## The frontend Every preview has exactly one **frontend**: the app Autonoma opens in the browser to run its tests, and whose address becomes the preview’s URL. It lives in the repository you open pull requests against. The frontend rarely stands alone - it calls an API, background workers, a database. Those can sit in the same repository, or in their own. Either way they deploy together, into the single preview environment for that pull request, and Previewkit wires them to each other. Think of the frontend as the root of a tree: everything else is there to support the one thing the browser opens. ![A tree with the frontend app at the root and the API, worker, and database it depends on branching below it, each tagged with the repository it comes from](/img/previewkit/multirepo-tree.jpg) ## Connected repositories When an app your frontend needs lives in a different repository, you add that repository as a **connected repository**. You do it while adding an app: pick which repository the app comes from, or connect a new one through the Previewkit GitHub App. Each connected repository carries two settings: * **Alias** - a short, lowercase name (e.g. `api`) that identifies the repository in your config and in generated resource names. It has to be unique across your repositories. * **Fallback branch** - the branch to deploy when branch matching finds no match (see below). Defaults to `main`. Every app from the same connected repository shares these two settings. ## Which branch gets deployed For the repository you open pull requests against, the answer is obvious: the pull request’s own branch. For a connected repository it isn’t - the pull request’s branch usually doesn’t exist there. **Branch matching** is the single rule that decides which branch of every connected repository Previewkit builds for a given pull request. If the branch it picks doesn’t exist in the connected repository, Previewkit always falls back to that repository’s **fallback branch**, so a preview never fails just because a connected repository has no matching branch. | Branch matching | For a PR on branch `feature/x`, a connected repository builds… | | ------------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **Same branch name** (default) | `feature/x` if that branch exists there, otherwise the fallback branch. Use this when you develop a feature across repositories on branches with the same name. | | **Fallback branch only** | Always the fallback branch (e.g. `main`). Use this when the connected repository is a stable service you don’t branch per feature. | | **Regex rewrite** | A branch name derived by rewriting `feature/x` with a regular expression (e.g. stripping a `feature/` prefix), falling back if the result doesn’t exist. Use this when your repositories follow different but predictable branch conventions. | Branch matching is set once and applies to every connected repository in the project; the fallback branch is per repository. > **Note:** > > Branch matching only affects **connected** repositories. The repository you open pull requests against always builds the pull request’s own branch. # Connect your coding agent (MCP) > Connect your coding agent to Autonoma's MCP server so it can see why a pull request's preview broke - deploy status, logs, diagnosis, and missing secrets - and fix it in your repo. Autonoma runs an MCP server so your coding agent can read a pull request's live debugging data - deploy status, build and runtime logs, a diagnosis, and which secrets are missing - and fix a broken preview from inside your repo, the same way it fixes anything else. When Autonoma flags a problem on your PR, the fix usually lives in your codebase. Instead of copying logs out of a dashboard, point your agent at the MCP server and let it pull exactly what it needs. ## Connection details Everything below is the same server, just configured per client. When a client asks for these values, use: | Setting | Value | | ------------------ | ------------------------------------------------------------------------------------------------------ | | **URL** | `https://api.autonoma.app/v1/mcp/debug` | | **Transport** | Streamable HTTP | | **Authentication** | OAuth - your client opens a browser to sign in to Autonoma and authorize; no API key or token to paste | > **Note:** > > You need an Autonoma account with your app connected, and a pull request that has a preview environment. The MCP is scoped to your organization - it only ever sees your own deploys, logs, and secret status. ## Connect your coding agent Add the server with the Claude Code CLI: ```bash claude mcp add --transport http autonoma https://api.autonoma.app/v1/mcp/debug ``` The first time your agent uses a tool, Claude Code opens a browser to sign in and authorize. Run `claude mcp list` to confirm it connected. Add the server to `~/.cursor/mcp.json` (global) or `.cursor/mcp.json` in your project: .cursor/mcp.json ```json { "mcpServers": { "autonoma": { "url": "https://api.autonoma.app/v1/mcp/debug" } } } ``` Reload Cursor, then complete the browser sign-in when prompted from **Settings → MCP**. For GitHub Copilot's agent mode, add the server to `.vscode/mcp.json`: .vscode/mcp.json ```json { "servers": { "autonoma": { "type": "http", "url": "https://api.autonoma.app/v1/mcp/debug" } } } ``` Start the server from the `mcp.json` editor lens, then authorize in the browser when prompted. Add the server to `~/.codeium/windsurf/mcp_config.json`: \~/.codeium/windsurf/mcp\_config.json ```json { "mcpServers": { "autonoma": { "serverUrl": "https://api.autonoma.app/v1/mcp/debug" } } } ``` Refresh MCP servers from Cascade's settings, then complete the browser sign-in. Codex reaches remote servers through the `mcp-remote` bridge, which also handles the OAuth browser flow. Add it to `~/.codex/config.toml`: \~/.codex/config.toml ```toml [mcp_servers.autonoma] command = "npx" args = ["-y", "mcp-remote", "https://api.autonoma.app/v1/mcp/debug"] ``` Any MCP client that speaks Streamable HTTP can connect with the [connection details](#connection-details) above. For a client that only supports STDIO servers, bridge to the remote server with [`mcp-remote`](https://github.com/geelen/mcp-remote): ```json { "mcpServers": { "autonoma": { "command": "npx", "args": ["-y", "mcp-remote", "https://api.autonoma.app/v1/mcp/debug"] } } } ``` `mcp-remote` opens the browser for the OAuth sign-in and proxies the connection over STDIO. > **Caution:** > > Client config formats change over time. If a snippet above no longer matches, keep the [connection details](#connection-details) and follow your client's current "add an MCP server" instructions. ## Point your agent at it The server gives your agent the tools; a short line in your agent's instructions tells it *when* to reach for them. The fastest way is to invoke the **`setup_autonoma`** prompt - your agent adds the section to `AGENTS.md` (or `CLAUDE.md`) for you. Or add it by hand: ```markdown After you push a PR, Autonoma reviews its preview deploy. If it flagged a problem, use the Autonoma MCP tools to find the cause (get_deploy_status, diagnose_deploy, get_build_logs, get_app_logs, get_secret_status), fix it (set_secret for a missing value, edit_previewkit_config for build/wiring), and confirm with wait_for_deploy - before merging. ``` Because that file is read every session, your agent pauses to check the preview without you having to ask. The server also ships two things any client can use without a setup file: a **`debug_broken_preview`** prompt (a guided fix flow for a given PR) and a readable **debugging guide** resource. And its connect-time instructions already tell your agent what Autonoma is and the recommended order to use the tools - so even an agent that has never heard of Autonoma knows where to start. ## What your agent can do Every tool takes your repo (`owner/repo`); the per-PR tools also take the PR number. Your organization is inferred from the repo (which you must belong to), so every call is automatically scoped to it. You don't need to hand your agent the repo name - it infers it from the repository's git remote, or calls `list_apps` to let you pick. You do not need GitHub access; the repo name is just how Autonoma identifies your app. ### Read the evidence | Tool | Input | Returns | | ------------------- | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `list_apps` | none | The repos you can debug across your organizations - use when the repo isn't obvious | | `get_deploy_status` | repo, PR | Per-service deploy status, endpoints, and the latest build outcome | | `diagnose_deploy` | repo, PR | The raw evidence in one call - status, service/addon states, latest build outcome, a rule-based failure classification, the config's env-key surface, and error-shaped logs - plus deterministic findings categorized as a missing env var, setup problem, or platform error. It is not an AI summary; you reason over the signals | | `get_build_logs` | repo, PR | Build-log lines, from the `tail` (newest) or `head` (start of the build), optionally for one service | | `get_app_logs` | repo, PR | Runtime (stdout/stderr) log lines, from the `tail` (a crash) or `head` (startup) | | `get_endpoints` | repo, PR | The preview URL, a suggested SDK URL, and one entry per service (internal services like a database report `url: null` with a reason) | | `get_secret_status` | repo | The full env-var surface per app: topology connections (with template values) and secret-backed vars (declared build secrets + runtime secrets), with masked length and a fingerprint only, plus which declared build secrets are missing | ### Fix and confirm | Tool | Input | Returns | | ------------------------ | -------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | `set_secret` | repo, PR, app, key, value? | Sets (or, without a value, removes) a secret env var's value and applies it - rebuilds if it's a build secret, restarts otherwise. Values are stored encrypted and never returned | | `edit_previewkit_config` | repo, PR, app, fields | Changes structural config for one service (path, Dockerfile, port, health check, build-secret keys, connections) and rebuilds it. Only the fields you pass change; never sets a secret value | | `wait_for_deploy` | repo, PR, app? | Blocks (up to \~45s, then re-callable) until the deploy settles (ready or failed) and returns the outcome plus the last few log lines, so you can see progress and keep debugging after a `set_secret` or `edit_previewkit_config` rebuild | The two write tools split cleanly by what they change, so your agent never has to guess which to use: * A **secret value** (an API key, token, or password) - `set_secret`. It stores the value and applies the minimal action itself: a rebuild if the key is a declared build secret (baked into the image at build), a restart otherwise. You do not tell it which; it reads your config. * **How the app is built or wired** (build path, Dockerfile, port, health check, which keys are injected at build, topology connections) - `edit_previewkit_config`. It saves a new config revision and rebuilds the service. Both apply asynchronously, so the loop is: **fix** with `set_secret` / `edit_previewkit_config`, **confirm** with `wait_for_deploy` (which streams a short log tail and tells you `settled: false` if the rebuild is still running so you can call it again), then re-read if it failed. > **Secrets are never exposed:** > > The MCP never returns a secret's value - not through `get_secret_status`, not through `set_secret`, not anywhere. To let your agent check whether a set value matches one it already holds, both return a non-reversible **fingerprint** (the first 12 hex chars of SHA-256 of the value) instead: your agent computes `sha256(value).hex.slice(0, 12)` on its candidate and compares. The value never leaves Autonoma; only whether it matches can be inferred. ## Troubleshooting **The tools do not show up.** Confirm the client connected (e.g. `claude mcp list`) and that you completed the browser sign-in. A client that only supports STDIO needs the `mcp-remote` bridge shown under **Other clients**. **A tool says no live preview environment was found.** Autonoma tears the preview down after testing, so the live-surface tools (`get_deploy_status`, `get_endpoints`, `wait_for_deploy`) return `unavailable` once it is gone. This does **not** mean there is nothing to inspect: `get_build_logs` and `get_app_logs` still work for a post-mortem (see below). Open the PR and let a new preview deploy if you need the live surface again. **Logs from a torn-down or old preview.** Build and app logs are retained for about 30 days and stay readable **after** the preview is torn down - so you can debug why a past deploy failed without redeploying. If the logs come back empty, the preview may never have deployed, or its logs have aged out; re-run the preview to get fresh ones. **Which pull requests can I use?** Any PR in a repo you have connected to Autonoma, in an organization you belong to. # Test Planner > The planner is a CLI that reads your codebase and generates a complete, natural-language E2E test suite - pages, flows, scenarios, and the test-data helpers to run them - in one command. The planner is a command-line tool that reads your codebase and generates a complete end-to-end test suite: the pages and flows to cover, the data each test needs, and the helpers that create and clean up that data. One command, and it runs on managed Autonoma credits - no LLM API key required. ![The planner reads your codebase and produces a knowledge base, scenarios, and a test suite](/img/test-planner/pipeline.jpg) ## Run it You launch the planner from the **Finish setup → Upload test artifacts** step in the dashboard, which hands you a ready-to-paste command with your token and generation id baked in: ```bash AUTONOMA_API_TOKEN=... AUTONOMA_GENERATION_ID=... npx @autonoma-ai/planner@latest ``` That `npx` line downloads and runs the planner, published on npm as `@autonoma-ai/planner` (the installed command is `autonoma-planner`). Run it from the root of the repo you want to test. It works against your **frontend** (to map pages and flows) and your **backend** (to map data models and wire up test data). If those live in separate repos, run it where it can reach both. Before the run starts, the planner asks a few quick questions to steer the output: * What is this project? * Why do you want E2E tests? * What are the most critical flows? Then it works through six steps, pausing for your review between each. ## What it does | # | Step | What it does | Output | | - | -------------------------- | ---------------------------------------------------------------------------------------------------------------------------------- | ----------------- | | 1 | **Find your pages** | Maps every page and route in your app. | - | | 2 | **Build a knowledge base** | Learns your features, flows, and UI patterns. | `AUTONOMA.md` | | 3 | **Map your data models** | Finds what your app stores and how each record is created. | `entity-audit.md` | | 4 | **Design test scenarios** | Decides the realistic data each test runs against. | `scenarios.md` | | 5 | **Set up test data** | Wires small helpers that create and clean up that data, validates the full up/down cycle against your app, and submits the recipe. | `recipe.json` | | 6 | **Generate the tests** | Writes the E2E tests as natural-language markdown, covering every page and feature. | `qa-tests/` | Step 5 is where the planner connects to the [Environment Factory](/environment-factory/) - it wires the create/teardown helpers and proves they work end to end before any tests are written. ## Review checkpoints The planner pauses after each step and shows you what it produced. Review it before moving on - these checkpoints determine the quality of the final suite. If a step got something wrong, you can retry it with a note steering the agent in the right direction. The steps that matter most to review: * **Find your pages / Knowledge base** - sets what gets tested and how coverage is prioritized. * **Map your data models** - decides which models run your real business logic during tests. * **Design test scenarios** - fixed values become assertions; variable values become tokens substituted at run time. * **Set up test data** - confirms the scenarios actually work against your real database. ## Commands and flags ```bash autonoma-planner # run the pipeline (default command) autonoma-planner status # show progress for the current project # Useful flags --project # target a repo other than the current directory --step # run or re-run a single step --resume # continue from where a previous run stopped --non-interactive # run without the review pauses (CI) --model # pick a different Autonoma-hosted model (still no key needed) ``` ## Output Artifacts are written to `~/.autonoma//` as they're produced. When you run the planner from the dashboard's Finish setup flow, the knowledge base, scenarios, recipe, and generated tests are uploaded to Autonoma automatically at the end - ready to run against your preview environments. > **Note:** > > The planner and the [Environment Factory](/environment-factory/) work together: the planner **designs** the scenarios and generates the tests; the Environment Factory **provisions** the real, isolated data those tests run against, on every run. # Environment Factory > The Environment Factory is one endpoint in your backend that creates isolated test data before each test run and deletes it after - by reusing the real functions your app already uses to create that data. # Setup Guide > Install the Autonoma SDK, create the Environment Factory endpoint, register factories, and wire up authentication - from an empty backend to a validated, production-ready endpoint. This guide takes you from nothing to a working Environment Factory endpoint at `/api/autonoma`. Plan for anywhere from 30 minutes to a couple of hours, depending on how many models your app has. You'll do eight things: 1. [Install the SDK](#1-install-the-sdk) in your existing backend 2. [Pick your scope field](#2-pick-your-scope-field) 3. [Generate two secrets](#3-generate-two-secrets) 4. [Create the endpoint](#4-create-the-endpoint) 5. [Register factories](#5-register-factories) 6. [Implement the auth callback](#6-implement-the-auth-callback) 7. [Validate](#7-validate) 8. [Go live](#8-go-live) ## 1. Install the SDK The endpoint lives **inside your existing backend**, next to your other routes. It is not a separate server, sidecar, or standalone process. ![Add the endpoint to your existing backend rather than running a separate sidecar](/img/environment-factory/integrated-endpoint.jpg) Pick the SDK in the **same language as your backend** - running a Python sidecar next to a Node app means your test data skips the real auth, hashing, and hooks that ship in production. Add the core SDK plus the adapter for your framework: ```bash # Next.js App Router, Bun, Deno (Web-standard Request/Response) pnpm add @autonoma-ai/sdk @autonoma-ai/server-web zod # Express / Fastify pnpm add @autonoma-ai/sdk @autonoma-ai/server-express zod # Hono pnpm add @autonoma-ai/sdk @autonoma-ai/server-hono zod # Node.js http pnpm add @autonoma-ai/sdk @autonoma-ai/server-node zod ``` | Framework | Adapter package | Handler export | | ------------------ | ----------------------------- | ---------------------- | | Next.js, Bun, Deno | `@autonoma-ai/server-web` | `createHandler` | | Express, Fastify | `@autonoma-ai/server-express` | `createExpressHandler` | | Hono | `@autonoma-ai/server-hono` | `createHonoHandler` | | Node.js `http` | `@autonoma-ai/server-node` | `createNodeHandler` | One package covers the core SDK and every adapter (`autonoma_fastapi`, `autonoma_flask`, `autonoma_django`): ```bash pip install autonoma-ai ``` | Framework | Handler export | | --------- | ------------------------ | | FastAPI | `create_fastapi_handler` | | Flask | `create_flask_handler` | | Django | `create_django_handler` | | Your backend | SDK package | | ------------ | ---------------------------------------- | | Go | `github.com/autonoma-ai/autonoma-sdk-go` | | Rust | `autonoma` crate | | Java | `ai.autonoma:autonoma-sdk` | | Ruby | `autonoma` gem | | PHP | `autonoma/sdk` | | Elixir | `autonoma` hex package | See [Examples](/environment-factory/examples/) for a complete, runnable endpoint in each of these. > **Note:** > > No SDK for your language? Open an issue - don't spin up a polyglot sidecar to reach one that exists. ## 2. Pick your scope field Choose the field most of your models use to reference the root tenant - usually `organizationId`, `orgId`, `tenantId`, or `workspaceId`. The SDK doesn't introspect foreign keys to find this. It just declares the field in the `discover` response so the dashboard knows how to scope test data. Your factories still own every write, including the tenant column. ## 3. Generate two secrets You need two **different** secrets. The SDK throws an error at startup if they match. ```bash openssl rand -hex 32 # AUTONOMA_SHARED_SECRET - shared with Autonoma openssl rand -hex 32 # AUTONOMA_SIGNING_SECRET - kept private, never shared ``` ```bash AUTONOMA_SHARED_SECRET=abc123... AUTONOMA_SIGNING_SECRET=def456... ``` The **shared secret** signs every request between Autonoma and your endpoint. The **signing secret** signs the teardown token and never leaves your server. See [Security](/environment-factory/security/) for what each one protects. ## 4. Create the endpoint Mount the handler at the conventional path `/api/autonoma`. app/api/autonoma/route.ts ```typescript import { createHandler } from '@autonoma-ai/server-web' export const POST = createHandler({ scopeField: 'organizationId', sharedSecret: process.env.AUTONOMA_SHARED_SECRET!, signingSecret: process.env.AUTONOMA_SIGNING_SECRET!, factories: { /* step 5 */ }, auth: async (user) => { /* step 6 */ }, }) ``` routes/autonoma.ts ```typescript import { createExpressHandler } from '@autonoma-ai/server-express' app.post('/api/autonoma', createExpressHandler({ scopeField: 'organizationId', sharedSecret: process.env.AUTONOMA_SHARED_SECRET!, signingSecret: process.env.AUTONOMA_SIGNING_SECRET!, factories: { /* step 5 */ }, auth: async (user) => { /* step 6 */ }, })) ``` src/routes/autonoma.ts ```typescript import { createHonoHandler } from '@autonoma-ai/server-hono' app.post('/api/autonoma', createHonoHandler({ scopeField: 'organizationId', sharedSecret: process.env.AUTONOMA_SHARED_SECRET!, signingSecret: process.env.AUTONOMA_SIGNING_SECRET!, factories: { /* step 5 */ }, auth: async (user) => { /* step 6 */ }, })) ``` autonoma\_handler.py ```python import os from autonoma.types import HandlerConfig from autonoma_fastapi import create_fastapi_handler config = HandlerConfig( scope_field='organization_id', shared_secret=os.environ['AUTONOMA_SHARED_SECRET'], signing_secret=os.environ['AUTONOMA_SIGNING_SECRET'], factories={ ... }, # step 5 auth=lambda user, ctx: { ... }, # step 6 ) router = create_fastapi_handler(config) app.include_router(router, prefix='/api/autonoma') ``` Using another framework? Find a complete, runnable endpoint for your stack in [Examples](/environment-factory/examples/). ## 5. Register factories Register one **factory** per model the dashboard can create. Each factory's input schema (Zod in TypeScript, Pydantic in Python) drives both the `discover` schema and the validation of incoming data. ```typescript import { z } from 'zod' import { defineFactory } from '@autonoma-ai/sdk' factories: { Organization: defineFactory({ inputSchema: z.object({ name: z.string(), slug: z.string() }), refSchema: z.object({ id: z.string(), name: z.string(), slug: z.string() }), // `data` is typed from inputSchema - no z.infer needed create: async (data) => organizationService.create(data), // `record` is typed from refSchema teardown: async (record) => organizationService.delete(record.id), }), User: defineFactory({ inputSchema: z.object({ email: z.string(), name: z.string() }), create: async (data) => userService.create({ ...data, password: 'test-password-123' }), // no teardown: this model is left alone on `down` }), } ``` Always call your real service or repository from `create` - the same function production uses. That's what keeps test data honest. This is the heart of the Environment Factory. Once your first two factories work, read [Factories & the create payload](/environment-factory/factories/) to learn how records link together with `_alias` / `_ref` and how to tear down dependent rows. ## 6. Implement the auth callback The `auth` callback receives the first `User` created during `up` and must return **real, working credentials** the test runner can log in with. ```typescript auth: async (user) => { // user can be null - not every scenario creates a User const session = await createSession(user!.id) return { cookies: [{ name: 'session', value: session.token, httpOnly: true, sameSite: 'lax', path: '/' }], } } ``` > **Caution:** > > If the auth callback returns a fake or expired token, **every test fails at the login step**. This is the single most common setup mistake. There are three patterns - session cookies, bearer tokens, and login credentials. See [Authentication](/environment-factory/authentication/) for all three. ## 7. Validate Before you write a single test, prove that `up` creates the right data and `down` removes all of it. **Smoke-test `discover` with curl:** ```bash SECRET="your-shared-secret" BODY='{"action":"discover"}' SIG=$(echo -n "$BODY" | openssl dgst -sha256 -hmac "$SECRET" | sed 's/.*= //') curl -s -X POST http://localhost:3000/api/autonoma \ -H "Content-Type: application/json" \ -H "x-signature: $SIG" \ -d "$BODY" | jq . ``` You should get back your schema - every registered model and its fields, plus `scopeField`. **Run the full lifecycle with `checkScenario`:** ```typescript import { checkScenario } from '@autonoma-ai/sdk' const result = await checkScenario( factories, { create: { Organization: [{ _alias: 'org', name: 'Test Org', slug: 'test-org' }], User: [{ name: 'Admin', email: 'admin@test.com', organizationId: { _ref: 'org' } }], }, }, { scopeField: 'organizationId' }, ) // result.valid - true if up + down both succeeded // result.phase - 'ok' | 'up' | 'down' (where it failed) // result.errors - [{ phase, message, fix? }] ``` Then confirm three things by hand: 1. **After `up`** - the expected records exist with the right values. 2. **After `down`** - every created record is gone, no orphans. 3. **Auth works** - the returned cookies or headers authenticate a real request. ## 8. Go live **Enable in production.** The endpoint returns `404` in production by default. When you're ready, opt in: ```typescript export const POST = createHandler({ // ... allowProduction: true, }) ``` **Connect to Autonoma.** Deploy your endpoint and set the same `AUTONOMA_SHARED_SECRET` in your deployment secrets. Autonoma then calls `discover` to learn your schema, generates scenario data, and sends `up` / `down` around every test run. > **Tip:** > > Prefer to iterate on a branch? Open a `feat: autonoma-sdk` pull request - Autonoma auto-detects it and lets you validate against a preview environment from the **Finish setup** tab, so you never push straight to `main`. # Factories & the Create Payload > How to register factories, link records with _alias and _ref, and tear down dependent rows. The factory is the only way the SDK writes data - there is no raw-SQL fallback. A **factory** is a small function that knows how to create - and optionally delete - one model. The SDK writes data *only* through the factories you register. There is no SQL introspection and no raw-SQL fallback. ## Anatomy of a factory ```typescript import { z } from 'zod' import { defineFactory } from '@autonoma-ai/sdk' Organization: defineFactory({ // 1. Drives the discover schema and validates incoming data inputSchema: z.object({ name: z.string(), slug: z.string() }), // 2. Optional: validates the record on teardown and types `record` refSchema: z.object({ id: z.string(), name: z.string(), slug: z.string() }), // 3. Called during `up`. `data` is typed from inputSchema. create: async (data) => organizationService.create(data), // 4. Called during `down`. `record` is typed from refSchema. teardown: async (record) => organizationService.delete(record.id), }) ``` The generics are inferred from the schemas, so you never write `z.infer<...>`: * `data` in `create` is the parsed `inputSchema`. * `record` in `teardown` is the parsed `refSchema` (or `Record & { id }` if you omit `refSchema`). * `create` must return an object with an **`id` field** (e.g. `{ id: "..." }`). This is what `down` uses to delete the record and what other factories reference. If your service returns a differently-named key, map it in the factory: `create: async (data) => { const u = await userService.create(data); return { id: u.userId } }`. Everything else you return is stored in refs and passed to later factories. A factory that returns no `id` fails with `FACTORY_MISSING_PK`. ## Always call your real code Register a factory for **every** model the dashboard can create. Point `create` at the function your app already uses: | What your code has | What `create` should do | | ---------------------------------------------------------------------- | --------------------------------------------------------- | | A `create` / `insert` / `register` function in a service or repository | Call that function | | That function also hashes passwords, generates slugs, syncs to Stripe… | Call it anyway - your factory inherits the logic for free | | Only inline ORM calls scattered across route handlers | Make the same ORM call directly in `create` | | A seed-only lookup table that’s never created at runtime | Omit it, or write a factory that re-creates the seed row | Even if `ProjectService.create()` today just wraps `prisma.project.create()`, wire it up. The day it gains a side effect, your tests keep working with zero rewiring. ## Linking records with `_alias` and `_ref` The `create` field in an `up` request is a **flat map keyed by model name**. Each value is the array of records to create. Records point at each other with two reserved keys: * `_alias` - a unique name you give a record so others can reference it. * `_ref` - `{ "_ref": "alias" }` resolves to the real `id` of the aliased record once it exists. ```json { "create": { "Organization": [{ "_alias": "acme", "name": "Acme Corp", "slug": "acme-corp" }], "Application": [{ "_alias": "webApp", "name": "Marketing Website", "organizationId": { "_ref": "acme" } }], "Test": [{ "name": "Homepage Test", "applicationId": { "_ref": "webApp" } }] } } ``` Set **every foreign key explicitly** on the record that owns it, using `_ref`. This includes the scope/tenant field - the SDK never injects it for you. ### How the SDK resolves the graph From that `_alias` / `_ref` graph, the SDK builds a dependency tree and sorts it so a referenced record is always created before the records that point at it: ![The SDK reads the reference graph and sorts records into a safe creation order](/img/environment-factory/ref-graph.jpg) 1. Walk every record, collecting each `_alias` and every `_ref`. 2. Topologically sort so parents come before children - regardless of key order. 3. Validate each record through its `inputSchema`, then call `create`. 4. Replace every `{ "_ref": "alias" }` with the real id before the factory runs - your factory never sees a placeholder. On `down`, factories with a `teardown` run in **reverse** order. Rules worth remembering: * `_alias` must be unique across the whole payload. * Every alias a `_ref` points at must be declared **in the same payload** - the SDK never looks it up in the database. * A `_ref` can appear anywhere in a record: a top-level FK, a nested JSON blob, an array element. The SDK finds it. * Models are created **only** when they appear as a top-level key. A record array nested inside another record’s field is passed to the factory as opaque data, not created separately. ### What to include and omit **Include:** required fields without defaults, every foreign key (via `_ref`), the scope field, and unique fields made unique per run (use `testRunId` in emails and slugs). **Omit:** `id`, fields with database defaults, auto-updated timestamps, and any row your factory mints transitively (see below). ## Dependents, cascades, and teardown A single `create` often mints more than one row - a `WorkspaceService.create` might insert a workspace plus a default channel and an onboarding record in one transaction. The SDK doesn’t know about those extra rows, so you have to tell it how to clean them up. Four options, best first: 1. **Schema cascade.** If the foreign keys from every dependent back to the root are `onDelete: Cascade`, deleting the root is enough. Nothing to configure. This is usually the intent when one transaction mints everything. 2. **Call your app’s delete function.** If you already have a `WorkspaceService.delete` that removes the whole subtree, call it from `teardown`: ```typescript Workspace: defineFactory({ inputSchema: WorkspaceInput, create: async (data) => workspaceService.create(data), teardown: async (record) => workspaceService.delete(record.id), }) ``` 3. **Forward the dependent IDs `create` already returns.** If the production `create` returns the child IDs, surface them into refs and delete them in reverse FK order: ```typescript Workspace: defineFactory({ inputSchema: WorkspaceInput, create: async (data) => { const { workspace, channel } = await workspaceService.create(data) return { id: workspace.id, channelId: channel.id } }, teardown: async (record) => { await db.channel.delete({ where: { id: record.channelId } }) await db.workspace.delete({ where: { id: record.id } }) }, }) ``` 4. **None of the above - stop.** Don’t modify a production `create` just to return more IDs for the test harness. Instead, add a cascade to the schema, add a delete function to the service, or accept orphans between runs (fine when the test database is reset periodically). > **Note:** > > Pure dependent models still get a factory (a thin repository call) **unless** they’re minted transitively by a parent. If they are, leave them out of the payload and let the parent’s `teardown` clean them up. ## Factory context Both `create` and `teardown` receive a context object. There is no SDK-managed database client - import the same client your app’s services already use. ```typescript interface FactoryContext { refs: Record[]> // everything created so far scenarioName: string testRunId: string } ``` > **Note:** > > This is the *factory* context. The [auth callback](/environment-factory/authentication/) receives a **different** context object (`scopeValue`, `refs`) - don’t reach for `testRunId` in `auth`, or `scopeValue` in a factory. # Authentication > The auth callback turns a created User into real, working credentials the test runner uses to log in. Covers session cookies, bearer tokens, and email/password credentials for mobile. The `auth` callback is what lets the test runner log in as the user your `up` request created. It receives that user and returns credentials the runner authenticates with. > **Caution:** > > This is the single most common place setups break. If `auth` returns a fake or expired token, **every test fails at the login step** - no matter how good your factories are. ## What the callback receives ```typescript auth: async (user, context) => { // user: the first User record from refs, or null if the scenario has no User. // Always handle null. Shape: { id, name, email, ... } // context: // scopeValue - the detected scope value (e.g. organization id), or the testRunId fallback // refs - all created records, keyed by model, for looking up related data } ``` Not every scenario creates a `User`, so `user` can be `null`. Guard for it. ## What the callback returns ```typescript interface AuthResult { cookies?: Array<{ name: string value: string httpOnly?: boolean sameSite?: 'strict' | 'lax' | 'none' path?: string domain?: string secure?: boolean maxAge?: number }> headers?: Record // custom headers, e.g. Authorization: Bearer ... credentials?: Record // key/value pairs for a manual login flow } ``` There is no top-level `token` field. Return a bearer token on `headers`; return login credentials on `credentials`. ## Pattern 1 - Session cookies The default for most server-rendered web apps. Create a real session and return its cookie. ```typescript auth: async (user) => { const session = await lucia.createSession(user!.id, {}) const cookie = lucia.createSessionCookie(session.id) return { cookies: [{ name: cookie.name, value: cookie.value, httpOnly: true, sameSite: 'lax', path: '/', }], } } ``` ## Pattern 2 - Bearer token For APIs and SPAs that authenticate with an `Authorization` header. ```typescript auth: async (user) => { const token = jwt.sign( { sub: user!.id, email: user!.email }, process.env.JWT_SECRET!, { expiresIn: '1h' }, ) return { headers: { Authorization: `Bearer ${token}` } } } ``` ## Pattern 3 - Email/password credentials When the agent needs to log in through your app’s actual login screen instead of receiving a cookie or token, return credentials. ```typescript auth: async (user) => ({ credentials: { email: user!.email, password: 'test-password-123', }, }) ``` For this to work, the `User` must be created with a **known password**. Hash it in the User factory during `create`: ```typescript User: defineFactory({ inputSchema: z.object({ email: z.string(), name: z.string() }), create: async (data) => userService.create({ ...data, password: 'test-password-123' }), }) ``` ## Common mistakes | Mistake | What happens | Fix | | ------------------------------------ | ---------------------------------- | --------------------------------------------- | | Returning a hardcoded `"test-token"` | Every test fails at login | Use your real session / JWT creation | | No password set on the User | Email/password login fails | Hash a known password in the User factory | | Token expires too quickly | Tests fail midway through | Set expiry to at least 1 hour | | Wrong cookie name | The browser never sends the cookie | Check your app’s real cookie name in DevTools | > **Note:** > > The one-hour expiry above is for the **login token you return here** (the session cookie or JWT). It has nothing to do with the Environment Factory’s internal teardown token, which the SDK signs and manages itself and which expires after 24 hours - see [Security](/environment-factory/security/). # Security & Troubleshooting > The Environment Factory's two secrets, three security layers, and hard safety guarantees - plus a full reference of error codes and common fixes. The endpoint creates and deletes data, so it’s protected by three independent layers and two separate secrets. This page also collects every error code and the fixes for the problems you’re most likely to hit. ## The two secrets Two secrets with different jobs. They **must be different values** - the SDK throws `SAME_SECRETS` at startup if they match. | Secret | Env variable | Who knows it | Purpose | | ------------------ | ------------------------- | -------------- | --------------------------------------------------------------------------------------------------------------- | | **Shared secret** | `AUTONOMA_SHARED_SECRET` | You + Autonoma | HMAC-signs every request. Autonoma signs; your SDK verifies. | | **Signing secret** | `AUTONOMA_SIGNING_SECRET` | Only you | Signs the teardown token during `up`, verifies it during `down`. Autonoma stores it opaquely and can’t read it. | ```bash openssl rand -hex 32 # AUTONOMA_SHARED_SECRET openssl rand -hex 32 # AUTONOMA_SIGNING_SECRET (must differ) ``` ## The three layers ![Three layers protect the endpoint: a production guard, request signing, and a signed teardown token](/img/environment-factory/security-layers.jpg) **Layer 1 - Production guard.** The endpoint returns `404` whenever the app runs in production, unless you set `allowProduction: true`. Even if someone finds the URL, it stays dark in production. Each SDK reads its ecosystem’s standard production signal - `NODE_ENV=production` for Node, `DJANGO_SETTINGS_MODULE` / `DEBUG=False` for Django, `MIX_ENV=prod` for Elixir, `APP_ENV`/`RAILS_ENV` for PHP and Rails, and so on. **Layer 2 - Request signing (HMAC-SHA256).** Every request carries an `x-signature` header: the HMAC-SHA256 of the raw body, keyed with the shared secret. The SDK verifies it automatically and rejects unsigned or tampered requests with `401`. **Layer 3 - Signed refs token.** When `up` creates data, the SDK signs the created record IDs into a `refsToken` using the signing secret. On `down`, it verifies that token before deleting anything - so `down` can only ever delete what `up` actually created. Autonoma just stores the opaque string and passes it back; it cannot forge or modify it. | Attack | Why it fails | | ------------------------------- | ------------------------------------- | | Fake refs with made-up IDs | No valid token → rejected | | A real token with altered refs | Refs don’t match the token → rejected | | A replayed token from last week | Token expired (24h) → rejected | ## What the SDK can and cannot do * **`up` can only create.** It invokes the factories you registered, which call your own services. It cannot update, delete, drop, truncate, or run raw SQL outside your factory bodies. * **`down` can only delete what `up` created**, verified by the signed token. It calls each factory’s `teardown` in reverse order. * **The SDK never runs SQL itself.** It calls your factories; they use whatever client your app already has. ## Error codes Every code the endpoint can return, with its fix: | Code | HTTP | Meaning | Fix | | -------------------- | ---- | ----------------------------------------------------- | -------------------------------------------------------------------------------- | | `INVALID_SIGNATURE` | 401 | HMAC signature missing or doesn’t match | Make `AUTONOMA_SHARED_SECRET` match the value Autonoma uses for your app | | `INVALID_BODY` | 400 | Body isn’t valid JSON, or a required field is missing | Match each record to its own top-level model key and supply every required field | | `UNKNOWN_ACTION` | 400 | `action` isn’t `discover`, `up`, or `down` | Check the request is one of the three actions | | `INVALID_REFS_TOKEN` | 403 | Refs token missing, malformed, or failed verification | Use the same `AUTONOMA_SIGNING_SECRET` between `up` and `down` | | `PRODUCTION_BLOCKED` | 404 | Endpoint disabled in production mode | Set `allowProduction: true`, or ensure the app isn’t in production mode | | `SAME_SECRETS` | 500 | `sharedSecret` and `signingSecret` are identical | Use two different `openssl rand -hex 32` values | | `FACTORY_MISSING_PK` | 500 | A factory’s `create` didn’t return an id | Return at least `{ id: "..." }` from every `create` | | `INTERNAL_ERROR` | 500 | Unexpected server error | Check your factory bodies and server logs | ## Other common problems These aren’t error codes - they surface as database or validation failures: | Problem | Cause | Fix | | ------------------------------ | ------------------------------------------------------------ | ---------------------------------------------------------------------------------------- | | FK violation on `up` | A required foreign key is missing | Set every FK (including the scope field) explicitly as a `{ "_ref": "alias" }` | | `Invalid input for ""` | Missing required field, or records under the wrong model key | Match each record to its own top-level model key and supply every required field | | `references unknown alias(es)` | A `_ref` points at an alias no record declares | Declare the alias with `_alias` in the same payload, or fix the typo | | FK violation on `down` | Circular FK between tables | The SDK handles cycles with deferred updates; if it still fails, check for untracked FKs | | Parallel tests collide | Same email/slug across runs | Put `testRunId` in every unique field | # Examples > Working Environment Factory endpoints across 8 languages and 11 framework combinations - copy one for your stack. Every example follows the same shape: install the SDK, configure the handler, register a factory for each model, and expose a single POST endpoint. Each factory carries an input schema (Pydantic in Python, Zod in TypeScript, and so on) so the SDK can describe the model to the dashboard and validate the create payload before invoking your code. There is no SQL introspection and no SQL fallback. > **New here?:** > > Read the [Environment Factory overview](/environment-factory/) for the concepts and the [Setup guide](/environment-factory/setup/) for the step-by-step. These examples are the finished code. ## Available examples All examples live in the [SDK repository](https://github.com/Autonoma-AI/sdk/tree/main/examples). Each one ships with a README covering prerequisites, quick start, project structure, and how it works. | Language | Framework | Schema lib | Source | | ------------------------------------------------------- | -------------------- | ----------------------- | ------------------------------------------------------------------------------------- | | [TypeScript](/environment-factory/examples/typescript/) | Express | Zod | [express](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/express) | | [TypeScript](/environment-factory/examples/typescript/) | Next.js (App Router) | Zod | [nextjs](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/nextjs) | | [TypeScript](/environment-factory/examples/typescript/) | Hono | Zod | [hono](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/hono) | | [Python](/environment-factory/examples/python/) | FastAPI | Pydantic | [fastapi](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/fastapi) | | [Python](/environment-factory/examples/python/) | Flask | Pydantic | [flask](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/flask) | | [Python](/environment-factory/examples/python/) | Django | Pydantic | [django](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/django) | | [Elixir](/environment-factory/examples/elixir/) | Phoenix | Ecto schemas | [phoenix](https://github.com/Autonoma-AI/sdk/tree/main/examples/elixir/phoenix) | | [Java](/environment-factory/examples/java/) | Spring Boot | Bean Validation | [spring-boot](https://github.com/Autonoma-AI/sdk/tree/main/examples/java/spring-boot) | | [Ruby](/environment-factory/examples/ruby/) | Rails | dry-validation | [rails](https://github.com/Autonoma-AI/sdk/tree/main/examples/ruby/rails) | | [Rust](/environment-factory/examples/rust/) | Axum | serde + validator | [axum](https://github.com/Autonoma-AI/sdk/tree/main/examples/rust/axum) | | [Go](/environment-factory/examples/go/) | Gin | go-playground/validator | [gin](https://github.com/Autonoma-AI/sdk/tree/main/examples/go/gin) | | [PHP](/environment-factory/examples/php/) | Laravel | Symfony Validator | [laravel](https://github.com/Autonoma-AI/sdk/tree/main/examples/php/laravel) | ## Configuration reference Every example configures the same handler fields: | Field | Description | | --------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `scopeField` | The column that scopes all models to a tenant (e.g. `organizationId`). Declared in `discover` so the dashboard knows how to isolate test data. | | `sharedSecret` | Shared between your server and Autonoma. Verifies incoming requests via HMAC-SHA256. Generate with `openssl rand -hex 32`. | | `signingSecret` | Private to your server. Signs the refs token so teardown can only delete what was created. Generate with `openssl rand -hex 32`, and make it different from `sharedSecret`. | | `factories` | One factory per model. Each declares an `inputSchema` / `input_model` plus a `create` that calls your real service, and an optional `teardown`. | | `auth` | Called during `up` with the created user. Returns credentials (cookies, headers, or credentials) so Autonoma can act as the test user. | For what each field does in depth, see [Factories & the create payload](/environment-factory/factories/), [Authentication](/environment-factory/authentication/), and [Security](/environment-factory/security/). # TypeScript > Autonoma Environment Factory examples with Express, Hono, and Next.js. The TypeScript SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s Zod `inputSchema`. There is no database introspection, no ORM executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. `zod` is a peer dependency: `npm install zod` (any v3.23+ or v4 release works). ## Express Uses `createExpressHandler` from `@autonoma-ai/server-express`. The factories use whatever Prisma / Drizzle / pg client your app already has - the SDK does not need a connection. src/index.ts ```typescript import express from 'express' import { z } from 'zod' import { defineFactory } from '@autonoma-ai/sdk' import { createExpressHandler } from '@autonoma-ai/server-express' import { PrismaClient } from '@prisma/client' import { OrganizationRepository } from './repositories/organization' import { UserRepository } from './repositories/user' const prisma = new PrismaClient() const organizationRepo = new OrganizationRepository(prisma) const userRepo = new UserRepository(prisma) const OrganizationInput = z.object({ name: z.string() }) const UserInput = z.object({ email: z.string(), name: z.string(), organizationId: z.string(), }) const app = express() app.use(express.json()) app.post( '/api/autonoma', createExpressHandler({ // The column that scopes all models to a tenant scopeField: 'organizationId', // Shared with Autonoma - verifies incoming requests via HMAC-SHA256 sharedSecret: process.env.AUTONOMA_SHARED_SECRET!, // Private to your server - signs the refs token so teardown only deletes what was created signingSecret: process.env.AUTONOMA_SIGNING_SECRET!, // Every model the dashboard can create needs a factory. // `defineFactory` infers `data`'s type from `inputSchema` - no z.infer<...> needed. factories: { Organization: defineFactory({ inputSchema: OrganizationInput, create: async (data) => organizationRepo.create({ name: data.name }), teardown: async (record) => organizationRepo.delete(record.id as string), }), User: defineFactory({ inputSchema: UserInput, create: async (data) => userRepo.create({ email: data.email, name: data.name, organizationId: data.organizationId, }), }), }, // Called after `up` - returns credentials so Autonoma can make authenticated requests auth: async (user) => ({ headers: { Authorization: 'Bearer test-token' } }), }), ) ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/express) *** ## Next.js (App Router) `createHandler` from `@autonoma-ai/server-web` works with any Web-standard runtime: Next.js App Router, Hono, Bun, Deno. src/app/api/autonoma/route.ts ```typescript import { z } from 'zod' import { defineFactory } from '@autonoma-ai/sdk' import { createHandler } from '@autonoma-ai/server-web' import { db } from '@/db' import { OrganizationRepository } from '@/repositories/organization' import { UserRepository } from '@/repositories/user' const organizationRepo = new OrganizationRepository(db) const userRepo = new UserRepository(db) const OrganizationInput = z.object({ name: z.string() }) const UserInput = z.object({ email: z.string(), name: z.string(), organizationId: z.string(), }) export const POST = createHandler({ scopeField: 'organizationId', sharedSecret: process.env.AUTONOMA_SHARED_SECRET!, signingSecret: process.env.AUTONOMA_SIGNING_SECRET!, factories: { Organization: defineFactory({ inputSchema: OrganizationInput, create: async (data) => organizationRepo.create({ name: data.name }), teardown: async (record) => organizationRepo.delete(record.id as string), }), User: defineFactory({ inputSchema: UserInput, create: async (data) => userRepo.create({ email: data.email, name: data.name, organizationId: data.organizationId, }), }), }, auth: async () => ({ headers: { Authorization: 'Bearer test-token' } }), }) ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/nextjs) *** ## Hono Same factories, the `createHonoHandler` adapter unwraps a Hono `Context` into the Web-standard request the SDK expects. ```typescript import { Hono } from 'hono' import { createHonoHandler } from '@autonoma-ai/server-hono' const app = new Hono() app.post('/api/autonoma', createHonoHandler({ /* same config as above */ })) ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/typescript/hono) *** ## What `inputSchema` does The Zod schema you pass as `inputSchema`: 1. **Drives discover** - the SDK walks the schema’s shape to describe the model to the dashboard (field names, types, required/optional, defaults). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK calls `inputSchema.safeParse(payload)` and passes the parsed value in. Validation failures bubble up as a 500 with the field path the dashboard can show inline. 3. **Drives types** - `defineFactory` is generic over the schemas you pass. `data` inside `create` is automatically typed as `z.infer` and `record` inside `teardown` is automatically typed as `z.infer` when you set one. No `z.infer<...>` annotations at the call site. 4. **Lets you accept extras** - recipes can carry display-only metadata (e.g. `_alias`) without failing validation; Zod ignores keys that aren’t part of your schema by default. ### Validated teardown with `refSchema` Adding a `refSchema` lets `teardown` work against a typed record (validated through Zod first). When `refSchema` is set, `create`’s return type is constrained to its input shape - the same record flows from `create` → `down` token → `teardown` with no manual casts. ```typescript const ProjectInput = z.object({ name: z.string(), organizationId: z.string() }) const ProjectRef = z.object({ id: z.string(), name: z.string() }) defineFactory({ inputSchema: ProjectInput, refSchema: ProjectRef, // `data` typed as { name: string; organizationId: string } create: async (data) => projectService.create(data), // `record` typed as { id: string; name: string } teardown: async (record) => projectService.delete(record.id), }) ``` Without `refSchema`, `create`’s return type widens to `Record & { id: string | number }` and `record` in `teardown` matches that shape - the existing factories above keep compiling. # Python > Autonoma Environment Factory examples with FastAPI, Flask, and Django. The Python SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s Pydantic `input_model`. There is no database introspection, no ORM executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## FastAPI + SQLAlchemy Uses `create_fastapi_handler` from `autonoma_fastapi`. The factories use whatever SQLAlchemy session your app already has - the SDK does not need a connection. app.py ```python import os from pydantic import BaseModel, ConfigDict from autonoma.types import HandlerConfig from autonoma.factory import define_factory from autonoma_fastapi import create_fastapi_handler from database import session from repositories.organization import OrganizationRepository from repositories.user import UserRepository organization_repo = OrganizationRepository(session) user_repo = UserRepository(session) class OrganizationInput(BaseModel): model_config = ConfigDict(extra="ignore") name: str class UserInput(BaseModel): model_config = ConfigDict(extra="ignore") email: str name: str organization_id: str config = HandlerConfig( # The column that scopes all models to a tenant - used to isolate test data scope_field="organization_id", # Shared with Autonoma - verifies incoming requests via HMAC-SHA256 shared_secret=os.environ["AUTONOMA_SHARED_SECRET"], # Private to your server - signs the refs token so teardown only deletes what was created signing_secret=os.environ["AUTONOMA_SIGNING_SECRET"], # Every model the dashboard can create needs a factory. # The factory's input_model drives both validation and discover. factories={ "Organization": define_factory( create=lambda data, ctx: organization_repo.create({"name": data.name}), teardown=lambda record, ctx: organization_repo.delete(record["id"]), input_model=OrganizationInput, ), "User": define_factory( create=lambda data, ctx: user_repo.create({ "email": data.email, "name": data.name, "organization_id": data.organization_id, }), input_model=UserInput, ), }, # Called after `up` - returns credentials so Autonoma can make authenticated requests auth=lambda user, context: {"headers": {"Authorization": "Bearer test-token"}}, ) router = create_fastapi_handler(config) app.include_router(router, prefix="/api/autonoma") ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/fastapi-sqlalchemy) *** ## Flask + SQLAlchemy Same `HandlerConfig`, different server adapter. `create_flask_handler` returns a Flask Blueprint. app.py ```python from autonoma_flask import create_flask_handler # Same HandlerConfig as FastAPI - scope_field, secrets, factories, auth. # The only difference is the server adapter. bp = create_flask_handler(config) app.register_blueprint(bp, url_prefix="/api/autonoma") ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/flask-sqlalchemy) *** ## Django `create_django_handler` returns a Django view function (already decorated with `@csrf_exempt` + `@require_POST`). core/autonoma\_config.py ```python import os from pydantic import BaseModel, ConfigDict from autonoma.types import HandlerConfig from autonoma.factory import define_factory from autonoma_django import create_django_handler from core.repositories.organization import OrganizationRepository from core.repositories.user import UserRepository organization_repo = OrganizationRepository() user_repo = UserRepository() class OrganizationInput(BaseModel): model_config = ConfigDict(extra="ignore") name: str class UserInput(BaseModel): model_config = ConfigDict(extra="ignore") email: str name: str organization_id: str config = HandlerConfig( scope_field="organization_id", shared_secret=os.environ["AUTONOMA_SHARED_SECRET"], signing_secret=os.environ["AUTONOMA_SIGNING_SECRET"], factories={ "Organization": define_factory( create=lambda data, ctx: organization_repo.create({"name": data.name}), teardown=lambda record, ctx: organization_repo.delete(record["id"]), input_model=OrganizationInput, ), "User": define_factory( create=lambda data, ctx: user_repo.create({ "email": data.email, "name": data.name, "organization_id": data.organization_id, }), input_model=UserInput, ), }, auth=lambda user, context: {"headers": {"Authorization": "Bearer test-token"}}, ) handler = create_django_handler(config) ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/python/django) *** ## What `input_model` does The Pydantic class you pass as `input_model`: 1. **Drives discover** - the SDK introspects `model_fields` to describe the model to the dashboard (field names, types, required/optional, defaults). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK calls `input_model.model_validate(payload)` and passes the typed instance in. Your factory body works on a real Python object, not a `dict`. 3. **Lets you accept extras with `extra="ignore"`** - recipes can carry display-only metadata (e.g. `_alias`) without failing validation. If you also want validated teardown, declare a `ref_model` (a Pydantic class describing the record returned by `create`) and the SDK will call `ref_model.model_validate(record)` before each `teardown` call. # Elixir > Autonoma Environment Factory example with Phoenix. The Elixir SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `input_fields`. There is no database introspection, no Ecto executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Phoenix Uses `Autonoma.Plug.Handler` as a Plug mounted via Phoenix’s `forward` macro. The factories use whatever Ecto Repo or service module your app already has - the SDK does not need a database connection. lib/autonoma\_example/router.ex ```elixir defmodule AutonomaExample.Router do use Phoenix.Router alias AutonomaExample.Repositories @autonoma_config %{ # The column that scopes all models to a tenant - used to isolate test data scope_field: "organization_id", # Shared with Autonoma - verifies incoming requests via HMAC-SHA256 shared_secret: System.get_env("AUTONOMA_SHARED_SECRET") || "", # Private to your server - signs the refs token so teardown only deletes what was created signing_secret: System.get_env("AUTONOMA_SIGNING_SECRET") || "", # Every model the dashboard can create needs a factory. # The factory's input_fields drives both validation and discover. factories: %{ "Organization" => Autonoma.Factory.define_factory(%{ input_fields: [ %{name: "name", type: "string", required: true} ], create: fn data, _ctx -> Repositories.Organization.create(data) end, teardown: fn record, _ctx -> Repositories.Organization.delete(record["id"]) end }), "User" => Autonoma.Factory.define_factory(%{ input_fields: [ %{name: "email", type: "string", required: true}, %{name: "name", type: "string", required: true}, %{name: "organization_id", type: "string", required: true} ], create: fn data, _ctx -> Repositories.User.create(data) end }) }, # Called after `up` - returns credentials so Autonoma can make authenticated requests auth: fn _user, _context -> %{"headers" => %{"Authorization" => "Bearer test-token"}} end } forward "/api/autonoma", Autonoma.Plug.Handler, @autonoma_config end ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/elixir/phoenix) *** ## What `input_fields` does The field list you pass as `input_fields`: 1. **Drives discover** - the SDK uses the field definitions to describe the model to the dashboard (field names, types, required/optional). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK checks that all required fields are present and strips unknown keys. Your factory body works on a clean map. 3. **Keeps it simple** - no external dependencies required. Use `"string"`, `"integer"`, `"number"`, `"boolean"`, `"timestamp"`, `"date"`, `"uuid"`, or `"json"` as the type. # Java > Autonoma Environment Factory example with Spring Boot. The Java SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `inputClass` (a Java class). There is no database introspection, no JDBC executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Spring Boot Uses `AutonomaController` from `ai.autonoma.spring`. Configured as a Spring `@Configuration` bean. The factories use whatever `JdbcTemplate`, JPA repository, or service layer your app already has - the SDK does not need a database connection. AutonomaConfig.java ```java @Configuration public class AutonomaConfig { public record OrganizationInput(String name) {} public record UserInput(String email, String name, String organizationId) {} @Bean public AutonomaController autonomaController() { OrganizationRepository organizationRepo = new OrganizationRepository(dataSource); UserRepository userRepo = new UserRepository(dataSource); HandlerConfig config = new HandlerConfig( // The column that scopes all models to a tenant - used to isolate test data "organization_id", // Shared with Autonoma - verifies incoming requests via HMAC-SHA256 System.getenv("AUTONOMA_SHARED_SECRET"), // Private to your server - signs the refs token so teardown only deletes what was created System.getenv("AUTONOMA_SIGNING_SECRET"), // Called after `up` - returns credentials so Autonoma can make authenticated requests (user, context) -> AuthResult.ofHeaders( Map.of("Authorization", "Bearer test-token") ) ); // Every model the dashboard can create needs a factory. // The factory's inputClass drives both validation and discover. config.setFactories(Map.of( "Organization", FactoryUtil.defineFactory( OrganizationInput.class, (data, ctx) -> organizationRepo.create(data), (record, ctx) -> organizationRepo.delete((String) record.get("id")) ), "User", FactoryUtil.defineFactory( UserInput.class, (data, ctx) -> userRepo.create(data) ) )); return new AutonomaController(config); } } ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/java/spring-boot) *** ## What `inputClass` does The Java class you pass as the first argument to `defineFactory`: 1. **Drives discover** - the SDK uses reflection to walk the class’s declared fields and map Java types to the dashboard’s type system. No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK uses Jackson’s `ObjectMapper.convertValue` to deserialize the incoming map into an instance of your class. Type mismatches fail validation. 3. **Uses standard Java conventions** - field names come from Jackson `@JsonProperty` annotations (or the field name itself); Java types map to SDK types automatically (`String`→“string”, `int/long`→“integer”, `double`→“number”, `boolean`→“boolean”, `Instant`→“timestamp”, `UUID`→“uuid”). # Ruby > Autonoma Environment Factory example with Rails. The Ruby SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `input_fields`. There is no database introspection, no ActiveRecord executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Rails Uses `AutonomaRails::Handler` mixin in a standard Rails controller. The factories use whatever ActiveRecord models, service objects, or repositories your app already has - the SDK does not need a database connection. app/controllers/autonoma\_controller.rb ```ruby require "autonoma" require "autonoma_rails" class AutonomaController < ApplicationController include AutonomaRails::Handler def handle autonoma_handle(autonoma_config) end private def autonoma_config @autonoma_config ||= Autonoma::Types::HandlerConfig.new( # The column that scopes all models to a tenant - used to isolate test data scope_field: "organization_id", # Shared with Autonoma - verifies incoming requests via HMAC-SHA256 shared_secret: ENV.fetch("AUTONOMA_SHARED_SECRET", ""), # Private to your server - signs the refs token so teardown only deletes what was created signing_secret: ENV.fetch("AUTONOMA_SIGNING_SECRET", ""), # Every model the dashboard can create needs a factory. # The factory's input_fields drives both validation and discover. factories: { "Organization" => Autonoma::Factory.define_factory( input_fields: [ { name: "name", type: "string", required: true } ], create: ->(data, _ctx) { OrganizationRepository.create(data) }, teardown: ->(record, _ctx) { OrganizationRepository.delete(record["id"]) } ), "User" => Autonoma::Factory.define_factory( input_fields: [ { name: "email", type: "string", required: true }, { name: "name", type: "string", required: true }, { name: "organization_id", type: "string", required: true } ], create: ->(data, _ctx) { UserRepository.create(data) } ), }, # Called after `up` - returns credentials so Autonoma can make authenticated requests auth: ->(_user, _context) { { "headers" => { "Authorization" => "Bearer test-token" } } } ) end end ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/ruby/rails) *** ## What `input_fields` does The field definitions you pass as `input_fields`: 1. **Drives discover** - the SDK uses the field definitions to describe the model to the dashboard (field names, types, required/optional). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK checks that all required fields are present and strips unknown keys. Your factory body works on a clean Hash. 3. **Keeps it simple** - no external gems required. Use `"string"`, `"integer"`, `"number"`, `"boolean"`, `"timestamp"`, `"date"`, `"uuid"`, or `"json"` as the type. # Rust > Autonoma Environment Factory example with Axum. The Rust SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `input_fields`. There is no database introspection, no SQLx executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Axum Uses `create_axum_handler` from `autonoma_sdk::axum`. Factories are registered in a `HashMap`. The factories use whatever SQLx pool, Diesel connection, or service layer your app already has - the SDK does not need a database connection. src/main.rs ```rust use autonoma_sdk::axum::create_axum_handler; use autonoma_sdk::factory::define_factory; use autonoma_sdk::types::{FactoryContext, FactoryRegistry, FieldDef, HandlerConfig}; use std::collections::HashMap; let mut factories: FactoryRegistry = HashMap::new(); factories.insert( "Organization".to_string(), define_factory( vec![FieldDef::required("name", "string")], |data, ctx| Box::pin(create_organization(data, ctx)), Some(|record, ctx| Box::pin(delete_organization(record, ctx))), ), ); factories.insert( "User".to_string(), define_factory( vec![ FieldDef::required("email", "string"), FieldDef::required("name", "string"), FieldDef::required("organization_id", "string"), ], |data, ctx| Box::pin(create_user(data, ctx)), None, ), ); let config = HandlerConfig { // The column that scopes all models to a tenant - used to isolate test data scope_field: "organization_id".to_string(), // Shared with Autonoma - verifies incoming requests via HMAC-SHA256 shared_secret, // Private to your server - signs the refs token so teardown only deletes what was created signing_secret, factories: Some(factories), // Called after `up` - returns credentials so Autonoma can make authenticated requests auth: Some(Box::new(|_user, _ctx| { Box::pin(async { Ok(serde_json::json!({"headers": {"Authorization": "Bearer test-token"}})) }) })), ..Default::default() }; let app = Router::new() .route("/api/autonoma", post(create_axum_handler(config))); ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/rust/axum) *** ## What `input_fields` does The `Vec` you pass as the first argument to `define_factory`: 1. **Drives discover** - the SDK uses the field definitions to describe the model to the dashboard (field names, types, required/optional). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK checks that all required fields are present in the `serde_json::Map`. Your factory body works on a validated map. 3. **Keeps it simple** - no external dependencies required beyond `serde_json`. Use `"string"`, `"integer"`, `"number"`, `"boolean"`, `"timestamp"`, `"date"`, `"uuid"`, or `"json"` as the type. # Go > Autonoma Environment Factory example with Gin. The Go SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `InputStruct` (a Go struct type). There is no database introspection, no SQL executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Gin Uses `autonoma.GinHandler` with factories registered in an `autonoma.FactoryRegistry` map. The factories use whatever `*sql.DB`, GORM, or service layer your app already has - the SDK does not need a database connection. main.go ```go import ( "os" "github.com/autonoma-ai/sdk-go/autonoma" "github.com/gin-gonic/gin" ) type OrganizationInput struct { Name string `json:"name"` } type UserInput struct { Email string `json:"email"` Name string `json:"name"` OrganizationID string `json:"organization_id"` } config := &autonoma.HandlerConfig{ // The column that scopes all models to a tenant - used to isolate test data ScopeField: "organization_id", // Shared with Autonoma - verifies incoming requests via HMAC-SHA256 SharedSecret: os.Getenv("AUTONOMA_SHARED_SECRET"), // Private to your server - signs the refs token so teardown only deletes what was created SigningSecret: os.Getenv("AUTONOMA_SIGNING_SECRET"), // Every model the dashboard can create needs a factory. // The factory's InputStruct drives both validation and discover. Factories: autonoma.FactoryRegistry{ "Organization": autonoma.DefineFactory(autonoma.FactoryOpts{ InputStruct: reflect.TypeOf(OrganizationInput{}), Create: func(data any, ctx autonoma.FactoryContext) (map[string]any, error) { input := data.(*OrganizationInput) return createOrganization(db, input) }, Teardown: func(record map[string]any, ctx autonoma.FactoryContext) error { return deleteOrganization(db, record["id"].(string)) }, }), "User": autonoma.DefineFactory(autonoma.FactoryOpts{ InputStruct: reflect.TypeOf(UserInput{}), Create: func(data any, ctx autonoma.FactoryContext) (map[string]any, error) { input := data.(*UserInput) return createUser(db, input) }, }), }, // Called after `up` - returns credentials so Autonoma can make authenticated requests Auth: func(user map[string]any, ctx autonoma.AuthContext) (*autonoma.AuthResult, error) { return &autonoma.AuthResult{ Extra: map[string]any{"headers": map[string]any{"Authorization": "Bearer test-token"}}, }, nil }, } r := gin.Default() r.POST("/api/autonoma", autonoma.GinHandler(config)) ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/go/gin) *** ## What `InputStruct` does The Go struct type you pass as `InputStruct`: 1. **Drives discover** - the SDK uses `reflect` to walk the struct’s fields and `json` tags to describe the model to the dashboard (field names, types, required/optional). No database introspection runs. 2. **Validates the create payload** - before invoking your `Create` function, the SDK uses `json.Unmarshal` into a new instance of the struct. Type mismatches and missing required fields fail validation. Your factory body receives a typed pointer to the struct. 3. **Uses standard Go conventions** - field names come from `json` struct tags; Go types map to SDK types automatically (`string`→“string”, `int`→“integer”, `float64`→“number”, `bool`→“boolean”, `time.Time`→“timestamp”, `uuid.UUID`→“uuid”). # PHP > Autonoma Environment Factory example with Laravel. The PHP SDK is **factory-driven**: you register one factory per model and the SDK derives the discover schema from each factory’s `inputFields`. There is no database introspection, no Eloquent executor, and no SQL fallback - your factories own creation, the SDK owns the protocol. ## Laravel Uses the auto-discovered service provider from `autonoma/sdk`. The entire setup is configuration-driven via `config/autonoma.php`. The factories use whatever Eloquent models, repositories, or service classes your app already has - the SDK does not need a database connection. config/autonoma.php ```php 'organization_id', // Shared with Autonoma - verifies incoming requests via HMAC-SHA256 'shared_secret' => env('AUTONOMA_SHARED_SECRET', ''), // Private to your server - signs the refs token so teardown only deletes what was created 'signing_secret' => env('AUTONOMA_SIGNING_SECRET', ''), 'path' => 'api/autonoma', // Every model the dashboard can create needs a factory. // The factory's inputFields drives both validation and discover. 'factories' => [ 'Organization' => Factory::define( inputFields: [ new FieldInfo('name', 'string', true), ], create: function (array $data, FactoryContext $ctx) { return (new OrganizationRepository())->create(['name' => $data['name']]); }, teardown: function (array $record, FactoryContext $ctx) { (new OrganizationRepository())->delete($record['id']); } ), 'User' => Factory::define( inputFields: [ new FieldInfo('email', 'string', true), new FieldInfo('name', 'string', true), new FieldInfo('organization_id', 'string', true), ], create: function (array $data, FactoryContext $ctx) { return (new UserRepository())->create([ 'email' => $data['email'], 'name' => $data['name'], 'organization_id' => $data['organization_id'], ]); } ), ], // Called after `up` - returns credentials so Autonoma can make authenticated requests 'auth' => function (?array $user, array $context): array { return ['headers' => ['Authorization' => 'Bearer test-token']]; }, ]; ``` [Full source code on GitHub](https://github.com/Autonoma-AI/sdk/tree/main/examples/php/laravel) *** ## What `inputFields` does The `FieldInfo` array you pass as `inputFields`: 1. **Drives discover** - the SDK uses the field definitions to describe the model to the dashboard (field names, types, required/optional). No database introspection runs. 2. **Validates the create payload** - before invoking your `create` function, the SDK checks that all required fields are present and strips unknown keys. Your factory body works on a clean associative array. 3. **Keeps it simple** - no external dependencies required. Use `'string'`, `'integer'`, `'number'`, `'boolean'`, `'timestamp'`, `'date'`, `'uuid'`, or `'json'` as the type. # Secrets > How to manage credentials, API keys, and other sensitive values for your Previewkit environments. Anything you wouldn’t commit to your repo - API keys, database URLs, signed tokens - should not live in your stack configuration. Manage it through the autonoma API instead. Every key you upload is mounted into your running app as an environment variable; your code just reads `process.env.STRIPE_API_KEY` and gets the value. ## Managing secrets ```plaintext GET /v1/previewkit/secrets/:applicationId/:app # list keys (no values) PUT /v1/previewkit/secrets/:applicationId/:app # batch upsert; body: {"items":[{"key","value"},...]} PUT /v1/previewkit/secrets/:applicationId/:app/:key # single upsert; body: {"value":"..."} DELETE /v1/previewkit/secrets/:applicationId/:app/:key # delete one key ``` `applicationId` is your autonoma Application row id. Look it up once via the dashboard and hardcode it in your CI. `app` matches an app’s `name` in your stack configuration. For a single-app repo it’s just that one name; for a monorepo each app has its own bundle. ### Authentication Every call needs an `Authorization: Bearer ` header. Create an API key from the autonoma dashboard (Settings → API keys); keys are scoped to your organization, so they can only see and modify your own applications’ secrets. Treat them like a password. ```bash export AUTONOMA_API_KEY="ak_live_..." # Batch upsert curl -X PUT "https://api.autonoma.app/v1/previewkit/secrets/app_abc123/web" \ -H "Authorization: Bearer $AUTONOMA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"items":[{"key":"STRIPE_API_KEY","value":"sk_live_..."},{"key":"SENTRY_DSN","value":"https://..."}]}' # Single key upsert curl -X PUT "https://api.autonoma.app/v1/previewkit/secrets/app_abc123/web/STRIPE_API_KEY" \ -H "Authorization: Bearer $AUTONOMA_API_KEY" \ -H "Content-Type: application/json" \ -d '{"value":"sk_live_..."}' # List keys (names only, never values) curl "https://api.autonoma.app/v1/previewkit/secrets/app_abc123/web" \ -H "Authorization: Bearer $AUTONOMA_API_KEY" # Delete curl -X DELETE "https://api.autonoma.app/v1/previewkit/secrets/app_abc123/web/STRIPE_API_KEY" \ -H "Authorization: Bearer $AUTONOMA_API_KEY" ``` Calls without a valid Bearer token get a 401. Calls referencing an `applicationId` your key doesn’t have access to are indistinguishable from “no secrets yet” - the API never reveals whether a foreign application exists. Updates take effect on the next preview deploy for that app. ## Build-time secrets (`build_secrets`) `NEXT_PUBLIC_*` values for Next.js, `VITE_*` values for Vite, anything else baked into a client bundle at compile time - these need to be present during `next build` / `vite build`, not just at runtime. List them in an app’s `build_secrets` and Previewkit will pass them to your builder: ```yaml apps: - name: web port: 3000 build_secrets: - NEXT_PUBLIC_FIREBASE_API_KEY - NEXT_PUBLIC_FIREBASE_PROJECT_ID - NEXT_PUBLIC_STRIPE_PUBLISHABLE_KEY ``` Each name must already be a key you’ve uploaded via the API. The build fails fast with a clear error if a listed key isn’t there. Server-only secrets (those your running pod reads via `process.env`) do NOT need to be in `build_secrets` - the runtime mount already covers them. Listing them anyway is harmless but verbose. ## Config-level overrides If you also define a key in an app’s `env` map in your stack configuration, the value there wins over the uploaded one. Use this for behaviour switches you want pinned alongside the rest of the config: ```yaml apps: - name: api port: 4000 env: # Pin a preview to safe defaults so it can't talk to live services. PLAID_ENV: "sandbox" SEND_EMAILS_LOCALLY: "false" ``` Template substitutions (`{{api.host}}`, `{{pr}}`, etc.) inside `env` resolve the same way. ## Built-in environment variables Previewkit injects a few variables into every preview app automatically. You don’t upload them, and you can’t override them - the names are reserved, so the API rejects any secret you try to set with one of these keys. | Variable | Value | Notes | | ------------------------- | ------------------------------------- | --------------------------------------------------------------------------------------------------- | | `AUTONOMA_PREVIEWKIT` | `true` | Always set inside a preview. Use it to detect the environment. | | `AUTONOMA_PREVIEWKIT_PR` | `123` | The pull request number this preview was built from. | | `AUTONOMA_PREVIEWKIT_URL` | `https://.preview.autonoma.app` | The public HTTPS URL of this app in the preview. In a multi-app preview, each app gets its own URL. | A common use is tagging your error reporter so preview errors are grouped per PR: ```ts import * as Sentry from "@sentry/node"; Sentry.init({ dsn: process.env.SENTRY_DSN, // "pr-123" in a preview, "production" everywhere else. environment: process.env.AUTONOMA_PREVIEWKIT_PR != null ? `pr-${process.env.AUTONOMA_PREVIEWKIT_PR}` : "production", }); ``` ## What goes where | Value type | Where it lives | | --------------------------------------------------------------------------------------------- | ------------------------------------------------------------------- | | Third-party API keys, database URLs, signed tokens | Previewkit API | | `NEXT_PUBLIC_*` / `VITE_*` baked into a client bundle | Previewkit API, also listed in `build_secrets` | | In-cluster service URLs (`{{db.host}}`, `{{api.host}}`) | Stack config `env` - resolved automatically, no upload needed | | PR / owner / namespace metadata (`{{pr}}`, `{{owner}}`, `{{namespace}}`) | Stack config `env` - resolved automatically, no upload needed | | Behaviour switches (`PLAID_ENV=sandbox`, `SEND_EMAILS_LOCALLY=false`) | Stack config `env` - pinned alongside the rest of the configuration | | Anything non-sensitive that varies between environments | Stack config `env` | | Preview metadata (`AUTONOMA_PREVIEWKIT`, `AUTONOMA_PREVIEWKIT_PR`, `AUTONOMA_PREVIEWKIT_URL`) | Injected automatically - reserved, no upload needed | If you’re unsure, default to the Previewkit API. You only need to think about `build_secrets` when a value must be present *during* the build (the client-bundle case above). # Scenario Recipe Schema > Canonical JSON contract for the scenario recipes file uploaded to Autonoma at POST /v1/setup/setups/:id/scenario-recipe-versions. This page documents the **canonical upload contract** for scenario recipes. It is language-agnostic: the schema is described as JSON with per-field expectations. The source of truth lives in `packages/types/src/schemas/scenarios.ts` (`ScenarioRecipesFileSchema`). The file is posted as the JSON body of: ```plaintext POST /v1/setup/setups/:setupId/scenario-recipe-versions ``` ## Top-level shape ```json { "version": 1, "source": { "discoverPath": "string", "scenariosPath": "string" }, "validationMode": "sdk-check" | "endpoint-lifecycle", "recipes": [ /* at least one ScenarioRecipe */ ] } ``` | Field | Type | Required | Notes | | ---------------------- | --------------------------------------- | -------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `version` | integer, must equal `1` | yes | Contract version. Currently only `1` is accepted. Not a string. | | `source` | object | yes | Provenance pointers. Additional keys are preserved. | | `source.discoverPath` | string | yes | Path (relative to the application repo) to the discovery output, e.g. `autonoma/discover.json`. **Required** - omitting it causes Zod to fail with `expected string, received undefined`. | | `source.scenariosPath` | string | yes | Path to the human-readable scenarios document, e.g. `autonoma/scenarios.md`. | | `validationMode` | `"sdk-check"` \| `"endpoint-lifecycle"` | yes | How Autonoma validated the recipes before upload. `sdk-check` = `checkScenario`/`checkAllScenarios`. `endpoint-lifecycle` = real HTTP `up`/`down`. | | `recipes` | array, minimum length `1` | yes | One entry per scenario. See below. | ## `ScenarioRecipe` (one entry in `recipes[]`) ```json { "name": "string", "description": "string", "create": { /* arbitrary model graph, see below */ }, "variables": { /* optional, see below */ }, "validation": { "status": "validated", "method": "checkScenario" | "checkAllScenarios" | "endpoint-up-down", "phase": "ok", "up_ms": 0, "down_ms": 0 } } ``` | Field | Type | Required | Notes | | -------------------- | --------------------------------------------------------------------- | -------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `name` | string | yes | Stable identifier. Must match the scenario name used in the LLM-facing docs. | | `description` | string | yes | Human-readable summary of the scenario state. | | `create` | object | yes | The model graph passed to the SDK’s `createScenario` / `up` flow. A flat map: keys are model names, values are arrays of seeded rows. Rows link with `_alias` / `_ref` (no nesting). Extra keys are preserved. | | `variables` | object (map of name → definition) | no | Per-recipe dynamic values. See **Variable definitions** below. | | `validation` | object | yes | Proof that the recipe was validated. All fields must be present. | | `validation.status` | literal string `"validated"` | yes | | | `validation.method` | one of `"checkScenario"`, `"checkAllScenarios"`, `"endpoint-up-down"` | yes | Which validator produced this result. | | `validation.phase` | literal string `"ok"` | yes | | | `validation.up_ms` | non-negative integer | no | Milliseconds the `up` phase took. | | `validation.down_ms` | non-negative integer | no | Milliseconds the `down` phase took. | ## Variable definitions `variables` is a map from variable name to a **tagged union** discriminated by the `strategy` field. Exactly one of the three shapes below is valid per entry. Unknown `strategy` values are rejected. ### `literal` Emits a fixed scalar on every run. ```json { "strategy": "literal", "value": "admin@example.com" } ``` | Field | Type | Required | Notes | | ---------- | ----------------------------------- | -------- | -------------------------------------------------------- | | `strategy` | literal `"literal"` | yes | | | `value` | string \| number \| boolean \| null | yes | Any JSON scalar. Objects and arrays are **not** allowed. | ### `derived` Derives a deterministic value from the test run ID (so every invocation of the same test gets the same value, but different runs get different values). ```json { "strategy": "derived", "source": "testRunId", "format": "user-{shortId}@example.com" } ``` | Field | Type | Required | Notes | | ---------- | --------------------- | -------- | ---------------------------------------------------------------------------- | | `strategy` | literal `"derived"` | yes | | | `source` | literal `"testRunId"` | yes | Only `testRunId` is supported today. | | `format` | string | yes | Template. The token `{shortId}` is replaced with a short hash of the run ID. | ### `faker` Generates a fresh random value per run using Faker. ```json { "strategy": "faker", "generator": "internet.email" } ``` | Field | Type | Required | Notes | | ----------- | ------------------------ | -------- | ------------------------------------------------------------------ | | `strategy` | literal `"faker"` | yes | | | `generator` | dotted Faker method path | yes | e.g. `internet.email`, `person.firstName`, `commerce.productName`. | ## Full example ```json { "version": 1, "source": { "discoverPath": "autonoma/discover.json", "scenariosPath": "autonoma/scenarios.md" }, "validationMode": "sdk-check", "recipes": [ { "name": "adminWithTwoProjects", "description": "Organization with an admin user and two projects.", "create": { "Organization": [{ "_alias": "org-1", "name": "Acme" }], "User": [ { "email": "{adminEmail}", "role": "admin", "organizationId": { "_ref": "org-1" } } ], "Project": [ { "name": "Alpha", "organizationId": { "_ref": "org-1" } }, { "name": "Beta", "organizationId": { "_ref": "org-1" } } ] }, "variables": { "adminEmail": { "strategy": "derived", "source": "testRunId", "format": "admin-{shortId}@acme.test" } }, "validation": { "status": "validated", "method": "checkScenario", "phase": "ok", "up_ms": 142, "down_ms": 61 } } ] } ``` ## Common rejection reasons * **`expected string, received undefined` under `source.discoverPath`** - the `source` object is missing `discoverPath`. Both `discoverPath` and `scenariosPath` are required. * **Discriminated union error under `recipes[n].variables.`** - an unknown or missing `strategy` key. Use exactly one of `"literal"`, `"derived"`, `"faker"`. * **`version` must be literal `1`** - don’t send `"1"` or `"1.0"`. Integer `1`. * **`recipes` must contain at least 1 element** - empty arrays are rejected. * **`validation.status` / `validation.phase` mismatch** - both are fixed literals (`"validated"` / `"ok"`). Any other value fails. ## Related * [Test Planner](/test-planner/) - how scenarios are designed and recipes are validated before upload. * [Environment Factory](/environment-factory/) - the `up` / `down` / `discover` SDK that consumes these recipes at runtime. # Development Setup > How to get Autonoma AI running locally - from prerequisites through a working dev environment. ## Prerequisites You need three things installed before starting: | Tool | Version | How to get it | | --------------------------------- | ------- | --------------------------------------------------------------- | | [Node.js](https://nodejs.org/) | >= 24 | Use [nvm](https://github.com/nvm-sh/nvm) or download directly | | [pnpm](https://pnpm.io/) | 10.x | Run `corepack enable` - the version is pinned in `package.json` | | [Docker](https://www.docker.com/) | Latest | Docker Desktop or Docker Engine | **Optional tools** (only needed if you’re working on specific engines): * [Playwright](https://playwright.dev/) - for `engine-web` development * [Appium](https://appium.io/) - for `engine-mobile` development ## Clone and install ```bash git clone https://github.com/autonoma-ai/autonoma.git cd agent pnpm install ``` `pnpm install` handles the entire monorepo - all apps and packages get their dependencies in one pass. ## Start infrastructure PostgreSQL and Redis run via Docker Compose: ```bash docker compose up -d ``` This starts: * **PostgreSQL 18** on `localhost:5432` (user: `postgres`, password: `postgres`) * **Redis** on `localhost:6379` Verify they’re running: ```bash docker compose ps ``` Both containers should show `running` status. ## Environment variables Copy the example file and fill in the required values: ```bash cp .env.example .env ``` ### Minimum required variables | Variable | Description | Where to get it | | ---------------------- | ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | PostgreSQL connection string | Use `postgresql://postgres:postgres@localhost:5432/autonoma` for the Docker Compose setup | | `REDIS_URL` | Redis connection string | Use `redis://localhost:6379` for the Docker Compose setup | | `BETTER_AUTH_SECRET` | Session signing secret | Generate any random string: `openssl rand -hex 32` | | `GOOGLE_CLIENT_ID` | Google OAuth client ID | Create OAuth credentials in the [Google Cloud Console](https://console.cloud.google.com/apis/credentials). Set the authorized redirect URI to `http://localhost:4000/api/auth/callback/google` | | `GOOGLE_CLIENT_SECRET` | Google OAuth client secret | Same Google Cloud Console OAuth credentials page | | `GEMINI_API_KEY` | Google Gemini API key | Get one from [Google AI Studio](https://aistudio.google.com/apikey) | ### How environment variables work in the codebase The project uses `createEnv` from `@t3-oss/env-core` for environment variable validation. Each app has an `env.ts` file that defines its required variables with Zod schemas. Variables are validated at startup - if something is missing, you get a clear error message telling you exactly what to add. You should never read `process.env` directly in application code. Instead, import from the app’s `env.ts` file. See `.env.example` for the full list of variables grouped by service. Most optional variables have sensible defaults or are only needed for specific features (S3 storage, Sentry, PostHog, etc.). ## Database setup Generate the Prisma client and run migrations: ```bash pnpm db:generate pnpm db:migrate ``` `db:generate` creates the TypeScript client from the Prisma schema. `db:migrate` applies all migrations to create the database tables. You need to re-run `db:generate` whenever the Prisma schema changes (after pulling new changes or editing the schema yourself). ## Start development servers ```bash pnpm dev ``` This starts both servers concurrently: * **UI** at `http://localhost:3000` (Vite + React) * **API** at `http://localhost:4000` (Hono + tRPC) To run them individually: ```bash pnpm api # API only (port 4000) pnpm ui # UI only (port 3000) ``` ## Verify everything works 1. Open `http://localhost:3000` in your browser 2. You should see the login page 3. Sign in with Google OAuth 4. If you see the dashboard, everything is working Run the full check suite to make sure nothing is broken: ```bash pnpm typecheck # TypeScript type checking pnpm lint # ESLint pnpm test # Vitest pnpm build # Full build ``` ## Other useful commands | Command | Description | | ------------------ | ------------------------------------------------ | | `pnpm dev` | Start API + UI in development mode | | `pnpm build` | Build all packages and apps | | `pnpm typecheck` | Run TypeScript type checking across all packages | | `pnpm lint` | Lint all packages | | `pnpm test` | Run tests across all packages | | `pnpm format` | Format code with Biome | | `pnpm check` | Lint and format with Biome | | `pnpm db:generate` | Generate Prisma client from schema | | `pnpm db:migrate` | Run database migrations | | `pnpm docs` | Start the documentation site (port 4321) | ## Troubleshooting ### `pnpm install` fails Make sure you’re using pnpm 10.x. Run `corepack enable` to let Node manage the pnpm version, then try again. ### Database connection refused Check that Docker Compose is running: `docker compose ps`. If PostgreSQL isn’t up, check logs with `docker compose logs postgres`. ### Prisma generate fails This usually means dependencies aren’t installed. Run `pnpm install` first, then `pnpm db:generate`. ### Port already in use Another process is using port 3000 or 4000. Find and kill it: ```bash lsof -i :3000 # or :4000 kill ``` ### Google OAuth redirect error Make sure your Google Cloud OAuth credentials have `http://localhost:4000/api/auth/callback/google` as an authorized redirect URI. ### ”Missing environment variable” error on startup The app validates all required environment variables at startup using `createEnv`. Check the error message for which variable is missing, then add it to your `.env` file. ### TypeScript errors after pulling changes Run `pnpm db:generate` first (the Prisma client may have changed), then `pnpm build` to rebuild all packages. TypeScript errors in the UI or API often come from stale package builds. # Architecture Overview > High-level architecture of Autonoma AI - how the monorepo is organized, how data flows, and why each technology was chosen. ## How Autonoma works Autonoma is an agentic E2E testing platform. Users describe tests in natural language, and an AI agent executes them on real browsers and devices. The core loop is: 1. User writes a test instruction (“Log in, go to settings, verify the avatar is visible”) 2. The execution agent takes a screenshot of the current screen 3. An LLM decides which action to perform (click, type, scroll, assert) 4. Platform drivers execute the action (Playwright for web, Appium for mobile) 5. The agent records the step and repeats until the test is done Everything else - the API, the UI, the jobs - exists to support this loop. ## Monorepo structure The codebase is split into **apps** (deployable services) and **packages** (shared libraries). Each package has exactly one concern. ```plaintext apps/ api/ Hono + tRPC API server ui/ Vite + React 19 SPA engine-web/ Playwright web test execution engine-mobile/ Appium mobile test execution docs/ Astro Starlight documentation site jobs/ Background jobs (multiple sub-services) packages/ ai/ AI primitives - models, vision, point detection analytics/ PostHog server-side event tracking billing/ Subscription and billing logic blacklight/ Shared UI component library db/ Prisma schema + generated client diffs/ Test diff computation emulator/ Mobile emulator management engine/ Platform-agnostic execution agent core errors/ Custom error hierarchy image/ Image processing utilities integration-test/ Test harness with Testcontainers k8s/ Kubernetes helpers logger/ Sentry-based structured logging review/ Post-execution AI review scenario/ Environment Factory scenario logic storage/ S3 file storage test-updates/ Test suite update logic types/ Shared Zod schemas and TypeScript types utils/ Shared utilities workflow/ Temporal workflow definitions ``` ### Why apps vs packages? **Apps** are independently deployable. Each one becomes its own Docker image and runs as its own process. The API, UI, and each engine are separate images - they never share a runtime. **Packages** are shared code. They’re consumed by apps at build time via pnpm workspaces. A package like `@autonoma/ai` is used by both `engine-web` and `engine-mobile`, but it never runs on its own. ## How the apps connect ```plaintext Browser | | HTTP (port 3000) v UI (Vite + React SPA) | | tRPC (port 4000) v API (Hono + tRPC) | |--- Prisma ---> PostgreSQL |--- Redis ----> Device locks, caching | | (dispatches jobs) v Engine Web / Engine Mobile | | Execution Agent (packages/engine) |--- Playwright (web) or Appium (mobile) |--- AI models (packages/ai) v Test results, recordings, artifacts ``` **UI to API**: The React SPA communicates with the API exclusively through tRPC. Types flow end-to-end - the frontend never manually defines API response types. Zod schemas in `packages/types` are the single source of truth for both sides. **API to Database**: The API uses Prisma as its ORM. The schema lives in `packages/db` and is shared across all backend services. **API to Engines**: When a test run starts, the API dispatches it to the appropriate engine (web or mobile). Engines execute tests independently and report results back. **Engines to AI**: During execution, engines call into `packages/ai` for element detection, visual assertions, and agent decision-making. AI calls go to external providers (Google Gemini, Groq, OpenRouter). ## Tech stack | Layer | Technology | Why | | -------------- | ------------------------------------------ | ---------------------------------------------------------------------------------------------------- | | Runtime | Node.js 24, ESM-only | Latest LTS with native ESM. No CommonJS compatibility issues | | Monorepo | pnpm workspaces + Turborepo | pnpm for fast, disk-efficient installs. Turborepo for cached, parallel builds | | Language | TypeScript (strictest) | Full type safety with `noUncheckedIndexedAccess`, `exactOptionalPropertyTypes`, and all strict flags | | API | Hono + tRPC | Hono is fast and lightweight. tRPC gives end-to-end type safety without code generation | | Frontend | React 19 + Vite + TanStack Router | Vite for fast dev builds. TanStack Router for type-safe routing with built-in data loading | | Database | PostgreSQL + Prisma | PostgreSQL for reliability. Prisma for type-safe queries and migration management | | Cache/Locking | Redis | Distributed device locking and caching across engine instances | | AI | Gemini, Groq, OpenRouter via Vercel AI SDK | Multiple providers for different tasks. Vercel AI SDK unifies the interface | | Web testing | Playwright | Most reliable browser automation library. Supports all major browsers | | Mobile testing | Appium | Industry standard for iOS and Android automation on real devices | | UI components | Radix UI + Tailwind CSS v4 + CVA | Accessible primitives (Radix), utility-first styling (Tailwind), type-safe variants (CVA) | | Observability | Sentry | Error tracking, performance monitoring, and structured logging in one tool | | Analytics | PostHog | Product analytics with server-side event tracking | | Deployment | Kubernetes + Temporal | K8s for orchestration. Temporal for workflow-based test execution pipelines | ## The execution flow This is the most important flow in the system - how a test goes from natural language to executed results. ### 1. Test creation The user writes a test as a natural language instruction, optionally with a URL and configuration. The API stores it in PostgreSQL. ### 2. Test dispatch When a test run starts, the API dispatches it to the appropriate engine based on the application type (web or mobile). For mobile, Redis-based device locking ensures exclusive access to physical devices. ### 3. Execution agent loop The execution agent (`packages/engine`) runs a loop powered by the Vercel AI SDK: ```plaintext Screenshot -> LLM decides action -> Execute command -> Record step -> Repeat ``` The agent has access to these commands: | Command | What it does | | ---------- | --------------------------------------------------------------------------------------- | | **click** | Uses vision AI to locate an element from a natural language description, then clicks it | | **type** | Locates an element, clicks it, then types text | | **scroll** | Scrolls up or down | | **assert** | Checks visual conditions against the current screenshot | | **wait** | Pauses for a specified duration (for loading states) | The LLM (currently Gemini) sees the screenshot, the test instruction, and the steps taken so far, then decides which command to call next. When it determines the test is complete (or has failed), it calls `execution-finished`. ### 4. AI-powered element detection Instead of CSS selectors or XPaths, the agent uses vision models to find UI elements. The `PointDetector` takes a screenshot and a natural language description (“the blue Submit button”) and returns pixel coordinates. This is what makes tests resilient to UI changes - the AI adapts to visual changes automatically. ### 5. Results and artifacts Every test run produces: * Step-by-step execution log with before/after screenshots * Video recording of the entire session * AI conversation log (what the model “thought” at each step) * Success/failure status with reasoning These artifacts are stored in S3 and accessible through the UI. ## Key design decisions ### ESM-only Every `package.json` has `"type": "module"`. No CommonJS anywhere. This eliminates an entire class of import/export bugs and aligns with the direction of the Node.js ecosystem. ### Strictest TypeScript All strict flags enabled, including `noUncheckedIndexedAccess` (array/object access returns `T | undefined`) and `exactOptionalPropertyTypes`. This catches real bugs at compile time. It’s more work upfront, but prevents entire categories of runtime errors. ### Constructor injection All dependencies are passed through constructors. No DI framework, no decorators, no magic. You can read any class and immediately see what it depends on. ### Separate Docker images Each engine (web, mobile) and each job type gets its own Docker image. This keeps images small and deployment independent. A change to the web engine doesn’t require redeploying the mobile engine. ### Platform-agnostic agent core All execution logic lives in `packages/engine`. Platform-specific apps (`engine-web`, `engine-mobile`) only implement driver interfaces (`ScreenDriver`, `MouseDriver`, etc.). The same agent loop, command system, and AI integration works for both Playwright and Appium. ## Deployment model The platform runs on Kubernetes: * **API** and **UI** are standard deployments with horizontal scaling * **Engines** run on device-hosting machines (physical or virtual). Web engines need browsers, mobile engines need connected devices or emulators * **Jobs** run as Temporal workflows - triggered on demand via Temporal workers * **Redis** handles distributed device locking across engine instances * **PostgreSQL** is the single source of truth for all state # Package Guide > What each package and app does, what it exports, and when you would modify it. ## Packages Every package in `packages/` is a shared library consumed by one or more apps. Each has exactly one concern. ### ai AI primitives used by the execution agent. Contains the model registry (manages LLM instances and providers), visual AI (screenshot analysis, assertion checking, element selection), point detection (locating UI elements from natural language descriptions), object detection (bounding box generation), and structured output generation. **Key exports:** `ModelRegistry`, `PointDetector`, `ObjectDetector`, `VisualConditionChecker`, `AssertChecker`, `ObjectGenerator`, `AssertionSplitter` **When to modify:** Adding a new AI model or provider, changing how elements are detected, adjusting assertion logic, or adding a new visual AI capability. ### analytics PostHog server-side event tracking. Wraps `posthog-node` with Sentry trace linking. No-ops when not initialized, so it’s safe to import in dev and test environments. **Key exports:** `analytics` (singleton) **When to modify:** Adding new server-side analytics events, changing event properties, or adjusting the PostHog integration. ### billing Subscription and billing logic. Handles plan management, usage tracking, and payment integration. **Key exports:** Billing service classes and plan definitions **When to modify:** Changing pricing plans, adding billing features, or integrating new payment providers. ### blacklight Shared UI component library built on Radix UI + Tailwind CSS v4 + CVA. This is where all reusable frontend components live - buttons, cards, inputs, dialogs, tables, and more. Follows shadcn/ui patterns. **Key exports:** `Button`, `Card`, `Input`, `Dialog`, `Table`, `Select`, `cn()`, and many more components **When to modify:** Adding new UI components, updating component styles, or changing the design system. The path alias `@/*` maps to `packages/blacklight/src/*` inside the package. ### db Prisma schema and generated client for PostgreSQL. This is the single source of truth for the database structure. **Key exports:** `PrismaClient`, generated types for all models **When to modify:** Adding or changing database tables, columns, relations, or indexes. After editing the schema, run `pnpm db:generate` and `pnpm db:migrate`. ### diffs Test diff computation. Computes differences between test suite versions for change tracking and review. **Key exports:** Diff computation functions **When to modify:** Changing how test diffs are calculated or displayed. ### emulator Mobile emulator management. Handles lifecycle management of iOS simulators and Android emulators. **Key exports:** Emulator management classes **When to modify:** Adding support for new device types, changing emulator configuration, or adjusting lifecycle management. ### engine The core of test execution. This is a platform-agnostic AI agent that web and mobile engines extend. Contains the execution agent loop, command system (click, type, scroll, assert), driver interfaces, runner orchestration, and artifact management. Everything is parameterized with generics (`TSpec` for command specs, `TContext` for driver context), so the same agent core works for both Playwright and Appium. **Key exports:** `ExecutionAgent`, `ExecutionAgentRunner`, `AgentCommand`, `CommandRegistry`, driver interfaces (`ScreenDriver`, `MouseDriver`, `KeyboardDriver`, `NavigationDriver`, `ApplicationDriver`) **When to modify:** Adding new commands to the agent, changing the execution loop, adjusting the system prompt, or modifying how steps are recorded. ### errors Custom error hierarchy for the project. All errors extend `AutonomaError` with specific subclasses for different failure types. **Key exports:** `AutonomaError`, `TestError`, `DriverError`, `PreconditionError`, `VerificationError`, `ThirdPartyError` **When to modify:** Adding new error types or changing how errors are categorized. ### image Image processing utilities. Handles screenshot manipulation, resizing, and format conversion used throughout the execution pipeline. **Key exports:** Image processing functions **When to modify:** Changing how screenshots are processed, adding new image operations, or adjusting compression settings. ### integration-test Test harness using Testcontainers. Provides `IntegrationHarness` and `integrationTestSuite` for writing integration tests that use real PostgreSQL and Redis containers. **Key exports:** `IntegrationHarness`, `integrationTestSuite` **When to modify:** Changing the test harness setup, adding new test utilities, or supporting new infrastructure in tests. ### k8s Kubernetes helpers. Utilities for interacting with the K8s API, managing pods, and reading cluster state. **Key exports:** Kubernetes client wrappers and helpers **When to modify:** Changing how the platform interacts with Kubernetes, or adding new K8s operations. ### logger Sentry-based structured logging. Provides a logger that integrates with Sentry for error tracking, performance monitoring, and structured context. **Key exports:** `logger` (root logger), `Logger` type **When to modify:** Changing the logging format, adjusting Sentry integration, or adding new logging capabilities. ### review Post-execution AI review. Analyzes test execution recordings and results to validate whether tests passed correctly. **Key exports:** Review service classes **When to modify:** Changing how test results are reviewed, adjusting AI review prompts, or adding new review criteria. ### scenario Environment Factory scenario logic. Handles test scenario definitions, data seeding, and teardown for isolated test environments. **Key exports:** Scenario classes and types **When to modify:** Adding new test scenarios, changing how test data is seeded, or adjusting the Environment Factory protocol. ### storage S3 file storage. Handles uploading and downloading artifacts (screenshots, videos, test results) to S3-compatible storage. **Key exports:** Storage service classes **When to modify:** Changing storage providers, adjusting upload/download logic, or adding new artifact types. ### test-updates Test suite update logic. Handles applying changes to test suites - adding, removing, and modifying test cases. **Key exports:** Test update service classes **When to modify:** Changing how test suites are modified, or adding new update operations. ### types Shared Zod schemas and TypeScript types. This is the contract layer between the API and frontend. Schemas defined here are used for tRPC input validation and frontend type inference. **Key exports:** Zod schemas for all API inputs/outputs, TypeScript types, constants **When to modify:** Adding new API endpoints, changing request/response shapes, or adding shared constants. ### utils Shared utilities that don’t fit into a more specific package. **Key exports:** Various utility functions **When to modify:** Adding general-purpose utilities used across multiple packages. ### workflow Temporal workflow definitions and client. Orchestrates test execution pipelines using Temporal workflows and activities. **Key exports:** Workflow builder classes **When to modify:** Changing how test execution is orchestrated, adjusting workflow templates, or adding new pipeline steps. ## Apps ### api The backend server. Built with Hono (HTTP framework) and tRPC (type-safe API layer). Routers are thin - they wire tRPC procedures to controller files in `controllers//`. One file per procedure. **When to modify:** Adding new API endpoints, changing business logic, or adjusting authentication. ### ui The frontend SPA. Built with React 19, Vite, and TanStack Router. Compiled to static files - no SSR. Uses `@autonoma/blacklight` for all UI components. **When to modify:** Adding new pages, changing the UI, or adjusting frontend behavior. ### engine-web Playwright-based web test execution. Implements the driver interfaces from `packages/engine` using Playwright’s API. Handles browser lifecycle, screenshot capture, network idle detection, and video recording. **When to modify:** Changing web-specific test execution behavior, adjusting Playwright configuration, or fixing browser-related issues. ### engine-mobile Appium-based mobile test execution for iOS and Android. Implements the same driver interfaces using Appium/WebDriver. Uses `@autonoma/device-lock` for Redis-based device allocation. **When to modify:** Changing mobile-specific test execution behavior, adjusting Appium configuration, or adding support for new device types. ### docs This documentation site. Built with Astro Starlight and deployed to S3 + CloudFront. **When to modify:** Adding or updating documentation pages. ### jobs Background job services, each deployed as a separate Docker image: | Job | Purpose | | ------------------------------- | ------------------------------------------------- | | **run-completion-notification** | Slack/email notifications when test runs complete | | **scenario** | Environment Factory scenario execution | | **diffs** | Computes test suite diffs | ## Dependency graph The general dependency flow (simplified): ```plaintext apps (api, ui, engines, jobs) | +-- packages/types (shared schemas - used by almost everything) +-- packages/db (database - used by api, jobs) +-- packages/engine (execution core - used by engines) +-- packages/ai (AI primitives - used by engine, jobs) +-- packages/try (error handling - used by everything) +-- packages/logger (logging - used by everything) +-- packages/errors (error types - used by engine, api) +-- packages/storage (S3 - used by api, engines, jobs) +-- packages/blacklight (UI components - used by ui only) +-- packages/analytics (PostHog - used by api) +-- packages/workflow (Temporal workflows - used by api, workers) ``` Key relationships: * `packages/engine` depends on `packages/ai` for all AI operations * `packages/ai` is self-contained - it only depends on `try`, `logger`, and `image` * `packages/types` is a leaf dependency - it depends on nothing else in the monorepo * `packages/try` is a leaf dependency - used everywhere, depends on nothing * Both `engine-web` and `engine-mobile` depend on `packages/engine` but never on each other # Code Conventions > The rules of the Autonoma AI codebase - TypeScript patterns, error handling, logging, testing, and style guidelines. ## ESM-only Every `package.json` has `"type": "module"`. No CommonJS anywhere in the codebase. **Never use `.js` extensions in imports.** TypeScript and the bundler resolve modules automatically. ```ts // Good import { foo } from "./foo"; import { bar } from "@autonoma/types"; // Bad import { foo } from "./foo.js"; ``` ## TypeScript strictness All strict flags are enabled. Every package extends `tsconfig.base.json`, which includes: * `strict: true` (enables all strict checks) * `noUncheckedIndexedAccess` - array and object index access returns `T | undefined` * `exactOptionalPropertyTypes` - optional properties can’t be assigned `undefined` explicitly unless typed that way * `verbatimModuleSyntax` - enforces explicit `type` imports In practice, this means: * You must check array access results before using them * You must narrow types before passing them to functions that expect non-nullable values * You must use `import type { ... }` for type-only imports ## Classes vs functions **Needs state or dependencies?** Use a class with constructor injection. **Pure logic with no state?** Use a function file. In practice, almost everything is a class because most logic needs a logger, a database client, or some other dependency. ## Dependency injection Plain constructor injection. No DI framework, no decorators. ```ts class StepExecutor { private readonly logger: Logger; constructor( private readonly engine: Engine, private readonly db: PrismaClient, ) { this.logger = logger.child({ name: this.constructor.name }); } } ``` You can read any class constructor and immediately see all its dependencies. No magic, no hidden state. ## One export per file A file exports exactly one thing - a class, a function, or a type. The exported item tells the story top-to-bottom. Private helpers follow in call order. This keeps files focused and makes imports predictable. ### Custom error hierarchy ```plaintext AutonomaError (base) TestError - test execution failures DriverError - Appium/Playwright driver failures PreconditionError - setup/precondition failures VerificationError - assertion failures ThirdPartyError - external service failures ``` ## Prefer undefined over null Always use `undefined` as the absence-of-value sentinel. Use optional properties (`?`) instead of `| null` types. Never initialize to `null`. ```ts // Good private timeout?: number // Bad private timeout: number | null = null ``` This applies everywhere: class properties, function parameters, return types, object shapes. ## Nullish checks Always `??`, never `||`. Always `!= null` / `== null`, never truthy/falsy checks. ```ts // Good const timeout = config.timeout ?? 3000; if (element != null) { /* ... */ } // Bad - truthy/falsy has unexpected behavior with 0, "", false const timeout = config.timeout || 3000; // 0 becomes 3000! if (element) { /* ... */ } ``` The `!= null` check covers both `null` and `undefined`, which is exactly what you want. ## Early returns Always prefer early returns to reduce nesting. If a function has deeply nested `if` blocks, extract the inner logic into a separate function with guard clauses. ```ts // Good function processOrder(order: Order): Result { if (order.status === "cancelled") throw new OrderCancelledError(); if (order.items.length === 0) throw new EmptyOrderError(); return calculateTotal(order); } // Bad - deeply nested function processOrder(order: Order): Result { if (order.status !== "cancelled") { if (order.items.length > 0) { return calculateTotal(order); } } // ... } ``` ## No complex destructuring or spread If constructing an object requires multiple `...` spreads or ternary-based spreads, build the object explicitly instead. ```ts // Good const permissions = isAdmin ? allPermissions : readOnly; return { name: baseConfig.name, timeout: baseConfig.timeout, permissions, retries: overrides.retries ?? baseConfig.retries, }; // Bad return { ...baseConfig, ...((isAdmin) ? { permissions: allPermissions } : { permissions: readOnly }), ...overrides, }; ``` ## Extract complex conditions If a condition isn’t immediately obvious, extract it into a descriptively named variable. ```ts // Good const isTrialExpired = subscription.status === "trial" && subscription.endsAt < now; const hasNoPaymentMethod = user.paymentMethods.length === 0; if (isTrialExpired && hasNoPaymentMethod) { /* ... */ } // Bad - what does this check? if (subscription.status === "trial" && subscription.endsAt < now && user.paymentMethods.length === 0) { /* ... */ } ``` ## Avoid let + conditional assignment Instead of using `let` and assigning in `if/else` blocks, extract a function with early returns. ## Logging with Sentry Every class and every function file must have logging. When in doubt, add a log. Overlogging is always better than underlogging. ### What to log * Service startup and configuration * Incoming requests and their resolution (success/failure) * External API calls (start, success, failure) * State transitions (agent steps, job status changes) * Resource acquisition/release (device locks, browser sessions) * Every public method entry with relevant parameters * Every method exit with relevant results Use structured context (Sentry breadcrumbs, tags, extra data) so logs are searchable. Never log sensitive data (credentials, tokens). ### Class logger pattern Every class gets a `private readonly logger` instance, created in the constructor as a child of the root logger with the class name and identifying context. ```ts import { type Logger, logger } from "@autonoma/logger"; export class TestSuiteUpdater { private readonly logger: Logger; constructor(private readonly snapshotId: string) { this.logger = logger.child({ name: this.constructor.name, snapshotId }); } public async apply(change: TestSuiteChange) { this.logger.info("Applying test suite change", { type: change.constructor.name }); // ... do work ... this.logger.info("Finished applying change"); } } ``` ### Function logger pattern - called from classes If a reusable function is called from a class method, accept a `Logger` parameter to preserve the logging context chain. ```ts import type { Logger } from "@autonoma/logger"; export function computeChanges(branchId: string, logger: Logger) { logger.info("Computing changes", { branchId }); // ... do work ... logger.info("Changes computed", { count: changes.length }); return changes; } ``` ### Function logger pattern - standalone files If a file exports independently useful functions (not called from a single class), import the root logger and create a child per function. ```ts import { logger as rootLogger } from "@autonoma/logger"; export function syncDevices(deviceIds: string[]) { const logger = rootLogger.child({ name: "syncDevices" }); logger.info("Syncing devices", { count: deviceIds.length }); // ... do work ... logger.info("Devices synced"); } ``` ## Testing ### Philosophy * **Vitest** for all tests * **Prefer integration tests** over unit tests. Test the real thing, not mocks * **Never mock the database.** Use Testcontainers with a real PostgreSQL container * Only test what makes sense - don’t test trivial getters ### Setup Test files go in `test/` directories that mirror the `src/` structure. File naming: `*.test.ts`. For integration tests that need a database, use the `@autonoma/integration-test` package: ```ts import { integrationTestSuite } from "@autonoma/integration-test"; integrationTestSuite("MyService", (harness) => { it("should create a record", async () => { const db = harness.db; // ... test with a real database }); }); ``` The harness spins up a real PostgreSQL container via Testcontainers, runs migrations, and gives you a fresh database for each test suite. ### Running tests ```bash pnpm test # run all tests pnpm test --filter=ai # run tests in a specific package ``` ## Database transactions Wrap sequential database queries in a Prisma `$transaction` when they must be consistent. If a service method reads then writes (or writes to multiple tables), use `$transaction`: ```ts async createGeneration(userId: string, orgId: string, appId: string) { return await this.db.$transaction(async (tx) => { const app = await tx.application.findFirst({ where: { id: appId, organizationId: orgId }, }); if (app == null) throw new Error("Application not found"); const generation = await tx.applicationGeneration.create({ data: { /* ... */ }, }); await tx.onboardingState.upsert({ where: { applicationId: appId }, /* ... */ }); return { id: generation.id }; }); } ``` Pass `tx` to all queries inside the transaction - not the original `db` client. ## Adding dependencies **Always check `pnpm-workspace.yaml` first.** The catalog section defines pinned versions for shared dependencies. When adding a dependency: 1. Check if it already exists in the `catalog:` section 2. If it does, use `"catalog:"` as the version in `package.json` 3. If it doesn’t, consider whether it should be added to the catalog (used by multiple packages) or pinned locally ```jsonc // Good - uses catalog version "dependencies": { "zod": "catalog:" } // Bad - hardcodes a version when a catalog entry exists "dependencies": { "zod": "^3.23.0" } ``` ## Environment variables Never read `process.env` directly. Define all environment variables in a dedicated `env.ts` file using `createEnv` from `@t3-oss/env-core` with Zod schemas: ```ts import { createEnv } from "@t3-oss/env-core"; import { z } from "zod"; export const env = createEnv({ server: { DATABASE_URL: z.string().url(), REDIS_URL: z.string().url(), BETTER_AUTH_SECRET: z.string().min(1), }, runtimeEnv: process.env, }); ``` This gives you type safety, runtime validation, and a single source of truth for all required variables. Pass validated env values as function parameters rather than reading `process.env` in library code. # Common Workflows > Step-by-step guides for common development tasks - adding routes, pages, commands, models, tests, and more. This page covers the most common development tasks you will perform in the Autonoma monorepo. Each workflow is a step-by-step guide with file paths and code patterns. ## Adding a New tRPC Route Types flow through tRPC from API to frontend. Never manually define API response types on the frontend. **1. Define Zod schemas** in `packages/types/src/schemas/`: packages/types/src/schemas/my-feature.ts ```ts import z from "zod"; export const myFeatureInput = z.object({ name: z.string(), organizationId: z.string(), }); export const myFeatureOutput = z.object({ id: z.string(), createdAt: z.date(), }); ``` **2. Create a controller** in `apps/api/src/controllers//.ts`. Controllers hold all business logic: apps/api/src/controllers/myFeature/create.ts ```ts import type { PrismaClient } from "@autonoma/db"; import type { z } from "zod"; import type { myFeatureInput } from "@autonoma/types"; export async function createMyFeature( db: PrismaClient, input: z.infer, ) { return db.myFeature.create({ data: { name: input.name, organizationId: input.organizationId }, }); } ``` **3. Create or update the router** in `apps/api/src/routers/`. Routers are thin wiring - they delegate to controllers: apps/api/src/routers/my-feature.ts ```ts import { router, protectedProcedure } from "../trpc"; import { myFeatureInput } from "@autonoma/types"; import { createMyFeature } from "../controllers/myFeature/create"; export const myFeatureRouter = router({ create: protectedProcedure .input(myFeatureInput) .mutation(async ({ ctx, input }) => { return createMyFeature(ctx.db, input); }), }); ``` **4. Add to `appRouter`** in `apps/api/src/router.ts` (if this is a new router): ```ts export const appRouter = router({ // ...existing routers myFeature: myFeatureRouter, }); ``` **5. Use on the frontend.** For queries, use `useSuspenseQuery` with `queryOptions`: ```ts const { data } = useSuspenseQuery( trpc.myFeature.list.queryOptions({ organizationId }), ); ``` For mutations, use `useAPIMutation` with `mutationOptions`: ```ts const createMutation = useAPIMutation( trpc.myFeature.create.mutationOptions(), ); ``` ## Adding a New Page TanStack Router with file-based routing makes this straightforward. **1. Create a route file** in `apps/ui/src/routes/`: apps/ui/src/routes/my-feature.tsx ```ts import { createFileRoute } from "@tanstack/react-router"; export const Route = createFileRoute("/my-feature")({ component: MyFeaturePage, }); function MyFeaturePage() { return
My Feature
; } ``` **2. That’s it.** The TanStack Router plugin auto-generates the route tree. The page is immediately accessible at `/my-feature`. For pages that need data, add a `loader`: ```ts export const Route = createFileRoute("/my-feature")({ loader: ({ context }) => { context.queryClient.ensureQueryData( trpc.myFeature.list.queryOptions(), ); }, component: MyFeaturePage, }); ``` ## Database Schema Changes **1. Edit the schema** at `packages/db/prisma/schema.prisma`. **2. Create a migration:** ```bash pnpm db:migrate ``` This generates a migration file and applies it to your local database. **3. Regenerate the Prisma client:** ```bash pnpm db:generate ``` **4. Run typecheck** to catch any type errors from the schema change: ```bash pnpm typecheck ``` If multiple queries in a service method need to be consistent (read-then-write, or writes to multiple tables), wrap them in a Prisma `$transaction`: ```ts return await this.db.$transaction(async (tx) => { const existing = await tx.myTable.findFirst({ where: { id } }); if (existing == null) throw new Error("Not found"); return tx.myTable.update({ where: { id }, data: { ... } }); }); ``` ## Adding a New Command to the Execution Agent See the [Execution Agent](/architecture/execution-agent/#adding-a-new-command) page for a detailed walkthrough. The short version: **1. Define the spec** with a `CommandSpec` interface and Zod schema in `packages/engine/src/commands/commands//.def.ts`. **2. Implement the command** by extending `Command` in `packages/engine/src/commands/commands//.command.ts`. **3. Create the tool wrapper** by extending `CommandTool` in `packages/engine/src/execution-agent/agent/tools/commands/.tool.ts`. **4. Add the spec** to the union type in `packages/engine/src/commands/command-defs.ts`. **5. Register the tool** in the `ExecutionAgentFactory` subclass for the relevant platform(s). **6. Write tests** in `packages/engine/src/commands/commands//.test.ts`. Use the test utilities in `packages/engine/src/commands/test-utils/` for fake drivers and model registries. ## Adding a New AI Model See the [AI Package](/architecture/ai-package/#adding-a-new-model) page for full details. The short version: **1. Add the model entry** to `MODEL_ENTRIES` in `packages/ai/src/registry/model-entries.ts`: ```ts MY_MODEL: { createModel: () => googleProvider.getModel("my-model-id"), pricing: simpleCostFunction({ inputCostPerM: 0.5, outputCostPerM: 1.5, }), }, ``` **2. Add a provider** in `packages/ai/src/registry/providers.ts` if the model uses a new provider. Add the API key to `packages/ai/src/env.ts` using `createEnv`. **3. Use it** via `registry.getModel({ model: "MY_MODEL", tag: "my-use-case" })`. ## Running and Writing Tests Vitest is used everywhere. Every package has it installed. ### Running Tests ```bash # Run all tests across the monorepo pnpm test # Run tests for a specific package pnpm --filter @autonoma/engine test # Run a specific test file pnpm --filter @autonoma/ai test -- src/visual/assert-checker.test.ts # Run in watch mode pnpm --filter @autonoma/engine test -- --watch ``` ### Writing Tests **Prefer integration tests over unit tests.** Only test what provides value - don’t test trivial getters. Test files go in `test/` directories or alongside source files as `*.test.ts`. **Never mock the database.** For tests that need a database, use Testcontainers with a real PostgreSQL container via the `@autonoma/integration-test` package: ```ts import { integrationTestSuite } from "@autonoma/integration-test"; integrationTestSuite("MyService", ({ getDb }) => { it("creates a record", async () => { const db = getDb(); const result = await myService.create(db, { name: "test" }); expect(result.name).toBe("test"); }); }); ``` For command tests, use the fake drivers in `packages/engine/src/commands/test-utils/`: ```ts import { FakeScreenDriver } from "../test-utils/fake-screen.driver"; import { FakeMouseDriver } from "../test-utils/fake-mouse.driver"; ``` ## Working with the UI Component Library All frontend components come from `@autonoma/blacklight`, built on Radix UI + Tailwind CSS v4 + CVA. ### Using Components ```tsx import { Button, Card, Input, cn } from "@autonoma/blacklight"; function MyComponent() { return ( ); } ``` ### Icons Use Lucide React for all icons: ```tsx import { Plus, Settings } from "lucide-react"; ``` ### Custom Variants Use CVA (class-variance-authority) for component variants: ```tsx import { cva } from "class-variance-authority"; const badgeVariants = cva("rounded-full px-2 py-0.5 text-xs font-medium", { variants: { status: { active: "bg-green-100 text-green-800", inactive: "bg-gray-100 text-gray-800", }, }, }); ``` ## Adding Environment Variables Never read `process.env` directly. Always use `createEnv` from `@t3-oss/env-core`. **1. Define the variable** in a dedicated `env.ts` file for the package or app: packages/my-package/src/env.ts ```ts import { createEnv } from "@t3-oss/env-core"; import z from "zod"; export const env = createEnv({ server: { MY_API_KEY: z.string().min(1), MY_TIMEOUT: z.coerce.number().default(5000), }, runtimeEnv: process.env, }); ``` **2. Use the validated env** in your code: ```ts import { env } from "./env"; const client = new MyClient({ apiKey: env.MY_API_KEY }); ``` **3. For library code**, prefer passing values as function parameters rather than reading env directly. This keeps the library testable and reusable: ```ts // Good - library accepts config export class MyService { constructor(private readonly apiKey: string) {} } // App wires it up with env const service = new MyService(env.MY_API_KEY); ``` **4. Check the catalog** in `pnpm-workspace.yaml` before adding `@t3-oss/env-core` as a dependency. If it is already in the catalog, use `"@t3-oss/env-core": "catalog:"` in your `package.json`. ## Adding Dependencies Before adding any dependency, check `pnpm-workspace.yaml` for the catalog: ```bash # Check if the package exists in the catalog grep "my-package" pnpm-workspace.yaml ``` If the package is in the catalog, use `catalog:` as the version: ```json { "dependencies": { "zod": "catalog:" } } ``` If it is not in the catalog but will be shared across multiple packages, consider adding it there first. Then install: ```bash pnpm install ``` ## Building and Type Checking ```bash # Build everything (Turborepo handles dependency order) pnpm build # Type check all packages pnpm typecheck # Lint all packages pnpm lint # Run dev servers (web on 3000, API on 4000) pnpm dev ``` All packages are ESM-only. Never use `.js` extensions in imports - TypeScript resolves modules automatically. # Environment Variables > Complete reference for every environment variable used across the Autonoma AI monorepo - API server, frontend, AI services, database, storage, logging, billing, and infrastructure. ## Quick Start - Minimum for Local Development To get the API and UI running locally, you need a surprisingly small set of variables. Copy `.env.example` to `.env` at the repo root and fill in these essentials: ```bash # Database DATABASE_URL=postgresql://postgres:postgres@localhost:5432/autonoma # Redis REDIS_URL=redis://localhost:6379 # API server API_PORT=4000 SCENARIO_ENCRYPTION_KEY=any-string-at-least-1-char # Google OAuth (create credentials at console.cloud.google.com) GOOGLE_CLIENT_ID=your-google-client-id GOOGLE_CLIENT_SECRET=your-google-client-secret # AI model keys (needed for test execution) GEMINI_API_KEY=your-gemini-key GROQ_KEY=your-groq-key OPENROUTER_API_KEY=your-openrouter-key # S3-compatible storage (can use MinIO locally) S3_BUCKET=autonoma-local S3_REGION=us-east-1 S3_ACCESS_KEY_ID=minioadmin S3_SECRET_ACCESS_KEY=minioadmin ``` Everything else has sensible defaults or is optional for local development. The sections below cover every variable in detail. ## How Environment Variables Work in This Project Every app and package defines its environment variables in a dedicated `env.ts` file using [`createEnv` from `@t3-oss/env-core`](https://env.t3.gg/). This gives you: * **Zod validation at startup** - the process crashes immediately if a required variable is missing or malformed, rather than failing mysteriously at runtime. * **Type safety** - `env.DATABASE_URL` is typed as `string`, not `string | undefined`. No more `process.env.DATABASE_URL!` casts. * **Composability** - packages export their `env` object, and apps extend them. For example, the API server’s `env.ts` extends the database, storage, logger, and billing envs, inheriting all their variables. You should **never read `process.env` directly** in application code. Always import from the nearest `env.ts`: ```ts // Good import { env } from "./env"; const port = env.API_PORT; // Bad - bypasses validation const port = process.env.API_PORT; ``` The `emptyStringAsUndefined: true` option is enabled everywhere, so setting a variable to an empty string is treated the same as not setting it at all. For boolean variables, the codebase uses `z.stringbool()` which accepts `"true"`, `"false"`, `"1"`, `"0"`, `"yes"`, and `"no"`. *** ## Core API Server **Source:** `apps/api/src/env.ts` The API server extends the database, storage, logger, and billing environments, so all variables from those sections apply here too. | Variable | Required | Default | Description | | -------------------------- | -------- | ------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------- | | `API_PORT` | Yes | - | Port the API server listens on. Typically `4000`. | | `INTERNAL_DOMAIN` | No | `autonoma.app` | Internal domain used for routing and service discovery. | | `ALLOWED_ORIGINS` | No | `http://localhost:3000` | Comma-separated list of CORS origins. Must include the frontend URL. | | `SCENARIO_ENCRYPTION_KEY` | Yes | - | Key used to encrypt scenario data. Any non-empty string works for local dev. | | `GOOGLE_CLIENT_ID` | Yes | - | OAuth 2.0 client ID from Google Cloud Console. Required for user authentication. | | `GOOGLE_CLIENT_SECRET` | Yes | - | OAuth 2.0 client secret from Google Cloud Console. | | `AGENT_VERSION` | No | `latest` | Version tag for the execution agent. Used when dispatching engine jobs. | | `POSTHOG_KEY` | No | - | PostHog project API key for server-side analytics. Omit to disable analytics. | | `POSTHOG_HOST` | No | `https://us.i.posthog.com` | PostHog ingestion endpoint. Override for self-hosted PostHog instances. | | `GEMINI_API_KEY` | Yes | - | Google Gemini API key. Used by the API for AI features like test generation. | | `OPENROUTER_API_KEY` | No | - | Server-side OpenRouter key the managed LLM proxy (`/v1/llm-proxy`, planner CLI) forwards requests with. The proxy returns `503` without it. | | `LLM_PROXY_ENABLED` | No | `false` | Master switch for the managed LLM proxy. The route mounts only when this and `STRIPE_ENABLED` are both `true`, so usage is always metered. | | `LLM_PROXY_ALLOWED_MODELS` | No | `google/gemini-3-flash-preview` | Comma-separated allowlist of OpenRouter model ids the proxy may route. Empty falls back to the default. | | `REDIS_URL` | Yes | - | Redis connection string (e.g., `redis://localhost:6379`). Used for device locking, caching, and pub/sub. | | `TESTING` | No | `false` | Set to `true` in test environments. Prevents importing certain modules. Not for general use. | | `ENGINE_BILLING_SECRET` | No | - | Shared secret for authenticating billing calls from the engine. | *** ## Frontend (UI) **Source:** `apps/ui/src/env.ts` The frontend uses Vite’s `import.meta.env` and requires the `VITE_` prefix for all variables. | Variable | Required | Default | Description | | ---------------------- | -------- | ----------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `VITE_API_URL` | No | `http://localhost:4000` | URL of the API server. The frontend makes all tRPC calls to this address. | | `VITE_INTERNAL_DOMAIN` | No | `autonoma.app` | Internal domain, used for UI routing logic. | | `VITE_TEMPORAL_URL` | No | - | URL of the Temporal UI. When set, enables links to workflow runs in the dashboard. | | `VITE_SENTRY_DSN` | No | - | Sentry DSN for frontend error tracking. Omit to disable Sentry in the browser. | | `VITE_SENTRY_URL` | No | - | Sentry organization URL. Used for linking to Sentry issues from the UI. | | `VITE_POSTHOG_KEY` | No | - | PostHog project API key for frontend analytics. Omit to disable analytics. PostHog events are proxied through the API server at `/ingest` to bypass ad blockers. | *** ## Database **Source:** `packages/db/src/env.ts` | Variable | Required | Default | Description | | -------------- | -------- | ------- | ---------------------------------------------------------------------------------------------------------------------------------- | | `DATABASE_URL` | Yes | - | PostgreSQL connection string. Format: `postgresql://user:password@host:port/database`. Used by Prisma for all database operations. | > **Note:** > > For local development, a typical value is `postgresql://postgres:postgres@localhost:5432/autonoma`. Make sure PostgreSQL is running and the database exists before starting the API. *** ## AI Services **Source:** `packages/ai/src/env.ts` These keys are required by the execution engines (web and mobile) and any service that runs AI inference. The API server only needs `GEMINI_API_KEY` directly - the other keys are consumed by the engine apps. | Variable | Required | Default | Description | | -------------------- | -------- | ------- | ----------------------------------------------------------------------------------------------------------------------------------------- | | `GEMINI_API_KEY` | Yes | - | Google Gemini API key. Used for the primary model (Gemini 3 Flash/Pro), point detection, object detection, and visual condition checking. | | `GROQ_KEY` | Yes | - | Groq API key. Used for fast inference with open-source models (e.g., GPT-OSS-120B). | | `OPENROUTER_API_KEY` | Yes | - | OpenRouter API key. Provides access to Ministral-8B and serves as a fallback provider for open-source models. | > **Note:** > > Validation is skipped when running in Vitest (`VITEST` env var is set), so you do not need these keys to run unit tests. *** ## Storage (S3) **Source:** `packages/storage/src/env.ts` Used for storing screenshots, video recordings, test artifacts, and other binary assets. | Variable | Required | Default | Description | | ---------------------- | -------- | ------- | ------------------------------------------------------------------ | | `S3_BUCKET` | Yes | - | S3 bucket name for storing artifacts. | | `S3_REGION` | Yes | - | AWS region of the S3 bucket (e.g., `us-east-1`). | | `S3_ACCESS_KEY_ID` | Yes | - | AWS access key ID (or MinIO equivalent) for S3 authentication. | | `S3_SECRET_ACCESS_KEY` | Yes | - | AWS secret access key (or MinIO equivalent) for S3 authentication. | > **Local development with MinIO:** > > You can run [MinIO](https://min.io/) locally as an S3-compatible object store. The default credentials are `minioadmin`/`minioadmin`. Point `S3_REGION` to any valid region string (e.g., `us-east-1`) and create a bucket matching your `S3_BUCKET` value. *** ## Logging and Observability **Source:** `packages/logger/src/env.ts` | Variable | Required | Default | Description | | ---------------- | -------- | ------------- | ---------------------------------------------------------------------------------------------------------- | | `NODE_ENV` | No | `development` | Node environment. Accepts `development`, `production`, or `test`. Affects log formatting and behavior. | | `SENTRY_DSN` | No | - | Sentry DSN for backend error tracking and performance monitoring. Omit to disable Sentry. | | `SENTRY_ENV` | No | `production` | Sentry environment tag (e.g., `staging`, `production`). | | `SENTRY_RELEASE` | No | `unknown` | Sentry release identifier. Typically set to the git SHA or version tag in CI. | | `DEBUG` | No | - | Debug filter string. When set, enables verbose debug logging for matching namespaces (e.g., `autonoma:*`). | *** ## Billing (Stripe) **Source:** `packages/billing/src/env.ts` Billing is entirely optional. When `STRIPE_ENABLED` is `false` (the default), all billing features are disabled and no other Stripe variables are needed. | Variable | Required | Default | Description | | ------------------------------ | -------- | ----------------------- | -------------------------------------------------------------------------------------------------------------- | | `STRIPE_ENABLED` | No | `false` | Master switch for billing. Set to `true` to enable Stripe integration. | | `STRIPE_SECRET_KEY` | No | - | Stripe secret API key. Required when `STRIPE_ENABLED` is `true`. | | `STRIPE_WEBHOOK_SECRET` | No | - | Stripe webhook signing secret for verifying incoming webhook events. Required when `STRIPE_ENABLED` is `true`. | | `STRIPE_SUBSCRIPTION_PRICE_ID` | No | - | Stripe Price ID for the subscription plan. Required when `STRIPE_ENABLED` is `true`. | | `STRIPE_TOPUP_PRICE_ID` | No | - | Stripe Price ID for credit top-up purchases. Required when `STRIPE_ENABLED` is `true`. | | `BILLING_GRACE_PERIOD_DAYS` | No | `3` | Number of days after a subscription lapses before access is revoked. | | `APP_URL` | No | `http://localhost:3000` | Frontend application URL. Used in Stripe checkout redirect URLs and billing emails. | *** ## Kubernetes and Workflows **Source:** `packages/k8s/src/env.ts` and `packages/workflow/src/env.ts` These variables are only needed in production or when running engine jobs on Kubernetes. Not required for local development. | Variable | Required | Default | Description | | ----------- | ------------ | ------- | ---------------------------------------------------------------------- | | `NAMESPACE` | Yes (in K8s) | - | Kubernetes namespace where jobs are deployed. Used by `@autonoma/k8s`. | The workflow package also reads: | Variable | Required | Default | Description | | -------------- | -------- | ------- | ----------------------------------------------------------------------------------------------- | | `DATABASE_URL` | Yes | - | PostgreSQL connection string. The workflow package needs direct DB access for job coordination. | | `SENTRY_ENV` | No | - | Sentry environment tag for workflow jobs. | *** ## Engine - Web (Playwright) **Source:** `apps/engine-web/src/platform/env.ts` and `apps/engine-web/src/execution-agent/env.ts` The web engine extends the AI, database, logger, and storage environments. All variables from those sections apply. | Variable | Required | Default | Description | | -------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `REMOTE_BROWSER_URL` | No | - | WebSocket URL of a remote browser instance (e.g., Browserless or Playwright remote). When omitted, launches a local Chromium browser. | | `HEADLESS` | No | - | Set to any value to run Playwright in headless mode. When omitted, the browser window is visible (useful for local debugging). | *** ## Engine - Mobile (Appium) **Source:** `apps/engine-mobile/src/platform/env.ts` The mobile engine extends the AI, database, logger, and storage environments. All variables from those sections apply. | Variable | Required | Default | Description | | -------------------------- | -------- | ------- | --------------------------------------------------------------------------------------------------------------- | | `APPIUM_HOST` | No | - | Hostname of the Appium server. | | `APPIUM_PORT` | No | - | Port of the Appium server. | | `APPIUM_MJPEG_PORT` | No | - | Port for the Appium MJPEG video stream. Used for live frame capture during test execution. | | `APPIUM_SYSTEM_PORT` | No | - | System port used by Appium’s UiAutomator2 (Android) or WebDriverAgent (iOS). | | `APPIUM_SKIP_INSTALLATION` | No | `true` | When `true`, skips reinstalling the app before each test. Speeds up repeated runs on the same device. | | `DEVICE_NAME` | No | - | Name of the target device or emulator (e.g., `iPhone 15 Pro`, `Pixel 7`). | | `IOS_PLATFORM_VERSION` | No | - | iOS version to target (e.g., `17.2`). Required for iOS testing. | | `ANDROID_DAEMON_HOSTS` | No | - | Comma-separated list of Android daemon host addresses for distributed device access. | | `IOS_DAEMON_HOSTS` | No | - | Comma-separated list of iOS daemon host addresses for distributed device access. | | `SKIP_DEVICE_DATE_UPDATE` | No | `false` | When `true`, skips updating the device date/time before tests. Useful when the device clock is already correct. | *** ## Jobs ### Execution Agent Runner **Source:** `packages/engine/src/execution-agent/runner/env.ts` | Variable | Required | Default | Description | | -------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------ | | `ARTIFACT_DIR` | No | - | Local directory for saving test artifacts (screenshots, videos, step logs). Used by the local runner during development. | ### Run Completion Notification **Source:** `apps/jobs/run-completion-notification/src/env.ts` | Variable | Required | Default | Description | | ----------------------- | -------- | ------- | ------------------------------------------------------- | | `DATABASE_URL` | Yes | - | PostgreSQL connection string. | | `API_URL` | No | - | API server URL for callbacks. | | `ENGINE_BILLING_SECRET` | No | - | Shared secret for authenticating billing-related calls. | | `STRIPE_ENABLED` | No | `false` | Whether to process billing events on run completion. | ### Worker - Diffs **Source:** `apps/workers/diffs/src/env.ts` Diffs analysis and resolution run as Temporal activities in the `@autonoma/worker-diffs` worker. AI model keys come from the AI Services section; this worker adds the GitHub App credentials it needs to clone repositories and read PRs. | Variable | Required | Default | Description | | --------------------------- | -------- | ------- | --------------------------------------------------------------------- | | `GITHUB_APP_ID` | Yes | - | GitHub App ID for repository access. | | `GITHUB_APP_PRIVATE_KEY` | Yes | - | GitHub App private key, base64-encoded PEM (`cat key.pem \| base64`). | | `GITHUB_APP_WEBHOOK_SECRET` | Yes | - | GitHub App webhook secret for verifying events. | | `GITHUB_APP_SLUG` | Yes | - | GitHub App slug (URL-friendly name). | | `SENTRY_DSN_WORKER_DIFFS` | No | - | Sentry DSN for the diffs worker. | ### Review Jobs (Generation Reviewer) **Source:** `packages/diffs/src/env.ts` The generation reviewer job re-exports from `@autonoma/diffs/env`, which extends the AI, logger, and storage environments. No additional variables beyond those from the AI, logger, and storage sections. *** ## GitHub App These variables appear in `.env.example` and are used by the API server and the diffs worker for GitHub integration features (repository connections, PR-triggered test runs). | Variable | Required | Default | Description | | --------------------------- | -------- | ------- | -------------------------------------------------------------------------------------- | | `GITHUB_APP_ID` | No | - | GitHub App ID. Required for GitHub integration features. | | `GITHUB_APP_PRIVATE_KEY` | No | - | GitHub App private key, base64-encoded PEM (`cat key.pem \| base64`). Decoded at boot. | | `GITHUB_APP_WEBHOOK_SECRET` | No | - | Secret for verifying GitHub webhook payloads. | | `GITHUB_APP_SLUG` | No | - | GitHub App slug (URL-friendly name). Used for generating installation links. | *** ## Authentication These variables are referenced in `.env.example` for the Better Auth integration used by the API server. | Variable | Required | Default | Description | | -------------------- | -------- | ------- | -------------------------------------------------------------------------------------------------- | | `BETTER_AUTH_SECRET` | Yes | - | Secret key for Better Auth session signing. Generate with `openssl rand -hex 32`. | | `BETTER_AUTH_URL` | Yes | - | Base URL of the API server (e.g., `http://localhost:4000`). Used by Better Auth for callback URLs. | *** ## Tips for Local Development **What you can skip entirely:** * **Billing** - Leave `STRIPE_ENABLED=false` (the default). No Stripe keys needed. * **Analytics** - Omit `POSTHOG_KEY` and `VITE_POSTHOG_KEY`. Analytics calls become no-ops. * **Sentry** - Omit `SENTRY_DSN` and `VITE_SENTRY_DSN`. Error tracking is disabled gracefully. * **Kubernetes** - Omit `NAMESPACE`. Only needed when deploying to K8s. * **GitHub App** - Omit all `GITHUB_APP_*` variables unless you are working on GitHub integration. * **Temporal** - Omit `VITE_TEMPORAL_URL`. The UI hides workflow links when this is unset. **What uses defaults that just work:** * `ALLOWED_ORIGINS` defaults to `http://localhost:3000` - correct for local dev. * `VITE_API_URL` defaults to `http://localhost:4000` - correct for local dev. * `APP_URL` defaults to `http://localhost:3000` - correct for local dev. * `NODE_ENV` defaults to `development`. * `AGENT_VERSION` defaults to `latest`. **What you must provide:** * `DATABASE_URL` - there is no default. You need a running PostgreSQL instance. * `REDIS_URL` - there is no default. You need a running Redis instance. * `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET` - required for authentication. Create OAuth credentials in the [Google Cloud Console](https://console.cloud.google.com/apis/credentials). * `SCENARIO_ENCRYPTION_KEY` - any non-empty string works locally. * `BETTER_AUTH_SECRET` - generate one with `openssl rand -hex 32`. * `BETTER_AUTH_URL` - set to `http://localhost:4000`. * AI keys (`GEMINI_API_KEY`, `GROQ_KEY`, `OPENROUTER_API_KEY`) - required if you are running test execution. Not needed if you are only working on the UI or API without triggering test runs. * S3 credentials - required for artifact storage. Use MinIO locally. # Execution Agent > Deep dive into the core test execution engine - a platform-agnostic AI agent that powers web and mobile test execution through natural language. The execution agent is the brain of Autonoma’s test execution. It is a **generic, platform-agnostic AI agent** that takes a natural language test instruction, interacts with a live application through screenshots and commands, and produces a structured test result with recorded steps. Web (`engine-web`) and mobile (`engine-mobile`) engines both extend this shared core. Everything is parameterized with `TSpec` (command spec) and `TContext` (driver context), so the same agent logic works across Playwright and Appium without code duplication. ## The Agent Loop Every test execution follows the same cycle: ```plaintext ┌─────────────────────────────────────────────────────┐ │ 1. Screenshot - capture current screen state │ │ 2. Inject context - screenshot + instruction + │ │ steps-so-far + memory into a user message │ │ 3. LLM decides - model picks a tool/command │ │ (or calls execution-finished) │ │ 4. Command executes - the chosen command runs │ │ against platform drivers │ │ 5. Record step - save before/after metadata, │ │ execution output, and screenshots │ │ 6. Loop or stop - continue until execution-finished │ │ is called or maxSteps is reached │ └─────────────────────────────────────────────────────┘ ``` The agent wraps the Vercel AI SDK’s `ToolLoopAgent`. Before each step, it captures a screenshot and injects it alongside the test instruction, all previous steps, and any stored memory variables. The LLM then decides which command to call next. **Loop detection:** If the model’s reasoning mentions “loop”, “stuck”, “no progress”, or “repeating” in a `success: false` finish, the result is flagged as a loop. **Success validation:** Even if the model calls `execution-finished` with `success: true`, the agent verifies that at least one command step was executed and at least one `assert` step exists. If either check fails, the result is overridden to `success: false`. ## Directory Structure ```plaintext packages/engine/src/ ├── commands/ # Command abstraction system │ ├── command-spec.ts # CommandSpec type definition │ ├── command.ts # Abstract Command base class │ ├── command-defs.ts # Union of all command specs │ ├── step.ts # StepData type │ └── commands/ # Built-in command implementations │ ├── click/ # AI-powered element clicking │ ├── type/ # Find element + type text │ ├── scroll/ # Scroll with condition checking │ ├── assert/ # Visual assertion checking │ ├── hover/ # Hover over elements (web only) │ ├── drag/ # Drag from one element to another │ ├── read/ # Extract text from screen into memory │ ├── navigate/ # Navigate to URL (web only, last resort) │ ├── refresh/ # Refresh the current page │ └── save-clipboard/ # Save clipboard content to memory ├── execution-agent/ # Core AI agent loop │ ├── agent/ │ │ ├── execution-agent.ts # Main agent class │ │ ├── execution-agent-factory.ts # Abstract factory for building agents │ │ ├── execution-result.ts # Result types │ │ ├── test-case.ts # TestCase interface │ │ ├── system-prompt.ts # Agent system prompt │ │ ├── memory/ # Variable memory store │ │ └── tools/ # LLM tools │ │ ├── command-tool.ts # Wraps Command as an AI SDK tool │ │ ├── execution-finished-tool.ts │ │ ├── ask-user-tool.ts │ │ └── wait-tool.ts │ ├── runner/ │ │ ├── execution-agent-runner.ts # Main runner - ties installer + factory + recording │ │ ├── artifacts.ts # Writes screenshots, steps, video to disk │ │ └── events.ts # Event hooks (beforeStep, attempt, frame) │ └── local-dev/ │ ├── local-runner.ts # Local dev runner (loads markdown test files) │ └── load-test-case.ts # Parses markdown frontmatter into test cases └── platform/ # Platform driver interfaces ├── context/ │ ├── base-context.ts # BaseCommandContext (screen + application drivers) │ ├── installer.ts # Abstract Installer │ ├── image-stream.ts # Live frame streaming interface │ └── video-recorder.ts # Abstract VideoRecorder with state machine └── drivers/ ├── screen.driver.ts # screenshot(), getResolution() ├── mouse.driver.ts # click(), hover(), drag(), scroll() ├── keyboard.driver.ts # type(), press(), selectAll(), clear() ├── application.driver.ts # waitUntilStable() ├── navigation.driver.ts # navigate(), getCurrentUrl(), refresh() └── clipboard.driver.ts # read() ``` ## CommandSpec - The Command Type System Every command is defined by a `CommandSpec`: ```ts interface CommandSpec { interaction: string; // command name (e.g., "click") params: object; // what gets stored on the step record output: BaseOutput; // what the command returns (always includes `outcome: string`) } ``` The `Command` abstract base class is what all commands extend: ```ts abstract class Command { abstract readonly interaction: TSpec["interaction"]; abstract readonly paramsSchema: z.ZodSchema>; abstract execute(params: CommandParams, context: TContext): Promise>; } ``` The `CommandTool` class wraps a `Command` to make it compatible with the AI SDK. It adds: * An `inputSchema()` that defines what the LLM provides (may differ from `paramsSchema`) * A `description()` shown to the AI model * An `extractParams()` method that converts LLM input into command parameters This separation means the LLM can provide a natural language description (“the blue submit button”) while the stored params contain the resolved coordinates and structured data recorded on the step. ## Built-in Commands | Command | Exposed to LLM | Params | What it does | | ------------------ | -------------- | ----------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | **click** | Yes | `{ description, options }` | Takes a natural-language element description, uses `PointDetector` AI to locate pixel coordinates, calls `mouse.click(x, y)` | | **type** | Yes | `{ description, text, overwrite }` | Uses `PointDetector` to find the input element, clicks it, then types the text. Supports overwrite mode to replace existing content | | **assert** | Yes | `{ instruction }` | Takes an instruction (can contain multiple assertions). Uses `AssertionSplitter` to decompose, takes one screenshot, runs `AssertChecker` on all assertions in parallel | | **scroll** | Yes | `{ elementDescription?, direction, condition, maxScrolls }` | Scrolls up or down on a specific element or the page, checking a visual condition after each scroll | | **hover** | Yes | `{ description }` | Hovers over an element identified by natural language description (web only) | | **drag** | Yes | `{ startDescription, endDescription }` | Drags from one element to another, both identified by natural language | | **read** | Yes | `{ description, variableName }` | Extracts text from the screen and stores it in the agent’s memory under `variableName` for use in later steps via `{{variableName}}` syntax | | **navigate** | Yes | `{ url }` | Navigates directly to a URL. Accepts full URLs, URLs without protocol (adds `https://`), or relative paths (resolved against the current page origin). **Last resort only** - prefer UI interaction (clicking links, buttons) to find bugs. Use only when you can’t reach something through the UI, or you’ve already tested the UI navigation in the same test (web only) | | **refresh** | Yes | (none) | Refreshes the current page | | **save-clipboard** | Yes | `{ variableName }` | Reads clipboard content and stores it in memory under `variableName` | ## LLM Tools (Non-Command) These tools are available to the model but are not recorded as test steps: | Tool | Purpose | | ---------------------- | ------------------------------------------------------------------------------------------------------------------------ | | **wait** | Sleeps for N seconds. Useful for loading screens or animations | | **ask-user** | Sends questions to a human via WebSocket. Pauses execution until answered. Only available in frontend-connected sessions | | **execution-finished** | Called by the model to end the test. Takes `{ success, reasoning }` | ## Driver Interfaces Platform-specific apps (`engine-web`, `engine-mobile`) implement these interfaces: ### ScreenDriver ```ts interface ScreenDriver { getResolution(): Promise; screenshot(): Promise; } ``` ### MouseDriver ```ts interface MouseDriver> { click(x: number, y: number, options?: TClickOptions): Promise; hover?(x: number, y: number): Promise; drag(startX: number, startY: number, endX: number, endY: number): Promise; scroll(args: ScrollArgs): Promise; } ``` ### KeyboardDriver ```ts interface KeyboardDriver { selectAll(): Promise; clear(): Promise; type(text: string, options?: TypeOptions): Promise; press(key: string): Promise; } ``` ### ApplicationDriver ```ts interface ApplicationDriver { waitUntilStable(): Promise; } ``` ### NavigationDriver ```ts interface NavigationDriver { navigate(url: string): Promise; getCurrentUrl(): Promise; refresh(): Promise; } ``` ### ClipboardDriver ```ts interface ClipboardDriver { read(): Promise; } ``` The `BaseCommandContext` requires only `screen` and `application` drivers. Each platform extends this with additional drivers as needed. ## Memory System The agent maintains a `MemoryStore` - a key-value store that persists across steps within a single execution. Commands like `read` and `save-clipboard` write values into memory, and any subsequent command can reference stored values using `{{variableName}}` template syntax. When a command executes, the agent resolves `{{variableName}}` templates in the parameters before passing them to the command. The unresolved params are stored on the step record (keeping the template references), while the resolved values are used for actual execution. ## Adding a New Command 1. **Define the spec.** Create a `CommandSpec` type for the command’s interaction, params, and output: packages/engine/src/commands/commands/my-command/my-command.def.ts ```ts import z from "zod"; export interface MyCommandSpec { interaction: "my-command"; params: { target: string; value: number }; output: { outcome: string; success: boolean }; } export const myCommandParamsSchema = z.object({ target: z.string().describe("Description for the LLM"), value: z.number().describe("A numeric value"), }); ``` 2. **Implement the command.** Create a class extending `Command`: packages/engine/src/commands/commands/my-command/my-command.command.ts ```ts import { Command } from "../../command"; import { type MyCommandSpec, myCommandParamsSchema } from "./my-command.def"; export class MyCommand extends Command { readonly interaction = "my-command" as const; readonly paramsSchema = myCommandParamsSchema; async execute(params, context) { // Use context drivers to perform the action return { outcome: "Did the thing", success: true }; } } ``` 3. **Create the tool wrapper.** Create a `CommandTool` subclass that defines how the LLM interacts with the command: packages/engine/src/execution-agent/agent/tools/commands/my-command.tool.ts ```ts import { CommandTool } from "../command-tool"; import type { MyCommandSpec } from "../../../../commands/commands/my-command/my-command.def"; export class MyCommandTool extends CommandTool { protected inputSchema() { return myCommandParamsSchema; } description() { return "Description shown to the AI model"; } protected async extractParams(input, context) { return input; } } ``` 4. **Register it.** Add the tool to the command tools array in your `ExecutionAgentFactory` subclass. 5. **Add the spec to the union type** in `packages/engine/src/commands/command-defs.ts` so TypeScript knows about it. ## Extending for a New Platform 1. **Implement all driver interfaces** using your platform’s SDK. At minimum you need `ScreenDriver` and `ApplicationDriver` (the `BaseCommandContext`). Add `MouseDriver`, `KeyboardDriver`, `NavigationDriver`, and `ClipboardDriver` as needed. 2. **Create an `Installer` subclass** that builds the context. The installer receives application data (URL, device config, etc.) and returns the context with all drivers, plus an `ImageStream` and `VideoRecorder`: ```ts class MyPlatformInstaller extends Installer { async install(appData: MyAppData) { // Launch browser/device, create driver instances return { context, imageStream, videoRecorder }; } } ``` 3. **Create an `ExecutionAgentFactory` subclass** that builds the agent with platform-specific command tools: ```ts class MyPlatformAgentFactory extends ExecutionAgentFactory { async buildAgent(params) { return new ExecutionAgent({ model: this.model, systemPrompt: this.systemPrompt, maxSteps: 50, commandTools: [new ClickTool(...), new TypeTool(...), ...], // ...rest of config ...params, }); } } ``` 4. **Create a runner entry point** that wires the installer, factory, and event handlers together using `ExecutionAgentRunner`. ## The Runner and Artifacts `ExecutionAgentRunner` orchestrates a full test run: 1. Calls `Installer.install()` to build the platform context (browser/device + drivers) 2. Registers a frame handler for live streaming 3. Builds the `ExecutionAgent` via the factory 4. Wraps `agent.generate()` in `VideoRecorder.withRecording()` 5. Returns `{ result, videoPath }` `LocalRunner` extends this for local development - it loads test cases from markdown files and saves artifacts to disk: ```plaintext artifacts/{timestamp}-{testName}/ ├── screenshots/step-0-before.jpeg, step-0-after.jpeg, ... ├── steps.json # Array of step execution outputs ├── conversation.json # Sanitized AI turn log ├── instruction.txt # The test prompt └── video.{ext} # Recording ``` ## Attempt Timeline The agent keeps a single in-memory timeline of **every** command attempt, modelled as a discriminated union on `status`: **`GeneratedStep`** (`status: "success"`) - a command whose parameter extraction and `execute()` both completed: * `executionOutput` - the command’s step data (interaction + params) and result * `beforeMetadata` / `afterMetadata` - screenshots and other metadata from before/after the step **`FailedStep`** (`status: "failed"`) - a command whose parameter extraction or `execute()` threw (e.g. a failed assertion, a point-detection miss, a driver error): * `interaction` / `input` / `params` - the command attempted, the raw tool input, and the extracted params (absent if extraction itself threw) * `error` / `errorName` - the thrown error’s message and class name (attribution signal) * `beforeMetadata` - the screenshot the model saw when it chose the command * `afterMetadata` - a best-effort after-screenshot (absent if even that capture failed) **`StepAttempt`** - the union `GeneratedStep | FailedStep`. The successful steps are derived by filtering `status === "success"`; failed attempts never trigger memory writes. Each attempt fires the runner’s `attempt` event as it happens, so the generation persister can live-persist it: successes write a `StepAttempt(success)` plus the `StepInput` / `StepOutput` committed step rows; failures write only a `StepAttempt(failed)`. ## Result Types **`ExecutionResult`** - the full test result: * `generatedSteps` - the successful steps only (the success subset of the attempt timeline) * `memory` - final state of extracted variables * `success` - whether the test passed (computed over successful steps only - a failed assertion never counts as a passing assertion) * `finishReason` - `"success"`, `"max_steps"`, or `"error"` * `reasoning` - the model’s explanation for finishing * `conversation` - the full AI message history **`LeanExecutionResult`** - a network-safe version that strips large image buffers from step metadata. ## Test Cases as Markdown Test files use [gray-matter](https://github.com/jonschlinkert/gray-matter) frontmatter for parameters, with the body containing the natural language prompt: ```markdown --- url: https://example.com --- Navigate to the login page, enter "user@test.com" and "password123", click Sign In, and assert the dashboard is visible. ``` The `loadTestCase` function parses the frontmatter against a Zod schema and extracts the prompt from the body. # AI Package > Deep dive into the AI primitives that power test execution - model registry, visual checkers, point detection, object detection, and structured output generation. The `@autonoma/ai` package provides every AI primitive used by the execution agent. It handles model management, visual analysis, element location, structured output generation, and evaluation benchmarking. No AI logic should be duplicated in platform apps - everything lives here. ## Directory Structure ```plaintext packages/ai/src/ ├── index.ts # Package re-exports ├── env.ts # Environment variables (API keys) ├── registry/ # Model registry and configuration │ ├── model-registry.ts # Core ModelRegistry class │ ├── model-entries.ts # Model definitions and pricing │ ├── providers.ts # LLM provider singletons │ ├── options.ts # ModelOptions, reasoning effort levels │ ├── costs.ts # Cost calculation functions │ ├── cost-collector.ts # Aggregated cost tracking │ ├── usage.ts # Token usage tracking │ └── monitoring.ts # Logging middleware and telemetry ├── visual/ # Visual AI primitives │ ├── visual-condition-checker.ts # Check if a condition is met on a screenshot │ ├── assert-checker.ts # Validate test assertions │ ├── visual-chooser.ts # Pick which UI element matches an instruction │ └── text-extractor.ts # Extract text from screenshots ├── text/ │ └── assertion-splitter.ts # Split compound assertions into atomic ones ├── object/ # Structured output generation │ ├── object-generator.ts # Core structured JSON generator │ ├── retry.ts # Retry with exponential backoff │ ├── user-messages.ts # Build multimodal messages (text + images + video) │ └── video/ │ ├── video-processor.ts # Upload videos to Google GenAI Files API │ └── video-input.ts # Video input types and model support └── freestyle/ # Point and object detection ├── resolution-fallback.ts # Coordinate resolution management ├── point/ │ ├── point-detector.ts # Abstract PointDetector base │ ├── gemini-computer-use-point-detector.ts │ └── object-point-detector.ts # Adapter: ObjectDetector -> PointDetector └── object/ ├── object-detector.ts # Abstract ObjectDetector base └── gemini-object-detector.ts # Gemini-based bounding box detection ``` ## Model Registry `ModelRegistry` manages all LLM instances with middleware for cost calculation and monitoring. It wraps the Vercel AI SDK’s language models with provider-specific configuration. ### How It Works The registry is constructed with a map of model entries. Each entry knows how to create its model instance and how to calculate costs: ```ts const registry = new ModelRegistry({ models: MODEL_ENTRIES, defaultSettings: { temperature: 0 }, monitoring: { onGenerate: (result) => { /* log it */ } }, }); ``` The registry is a stateless, construct-once singleton - it holds no mutable per-run state. When you request a model, it wraps it with middleware for monitoring, cost calculation, and default settings: ```ts const model = registry.getModel({ model: "GEMINI_3_FLASH_PREVIEW", tag: "assert-checker", reasoning: "low", }); ``` The `tag` field identifies the use case (e.g., “assert-checker”, “click-detector”) for monitoring and cost attribution. The `reasoning` field sets the thinking effort level. ### Current Models | Key | Model ID | Provider | | ------------------------ | ----------------------------- | ---------- | | `GEMINI_3_FLASH_PREVIEW` | `gemini-3-flash-preview` | Google | | `MINISTRAL_8B` | `mistralai/ministral-8b-2512` | OpenRouter | | `GPT_OSS_120B` | `openai/gpt-oss-120b` | Groq | An alternative `OPENROUTER_MODEL_ENTRIES` set routes all models through OpenRouter, including a Gemini variant (`google/gemini-3-flash-preview`) and a Llama variant (`meta-llama/llama-4-maverick`) in place of Ministral. ### Providers Three LLM provider singletons are available, each lazily initialized with their respective API key: | Provider | SDK | Env Variable | | -------------------- | ----------------------------- | -------------------- | | `googleProvider` | `@ai-sdk/google` | `GEMINI_API_KEY` | | `groqProvider` | `@ai-sdk/groq` | `GROQ_KEY` | | `openRouterProvider` | `@openrouter/ai-sdk-provider` | `OPENROUTER_API_KEY` | The `LLMProvider` class wraps each provider as a singleton - the underlying SDK instance is created on first use. ### Reasoning Effort The `ModelReasoningEffort` type supports four levels: | Level | Groq | Google | | ---------- | --------------------------- | ------------------------- | | `"none"` | `reasoningEffort: "none"` | Thinking disabled | | `"low"` | `reasoningEffort: "low"` | `thinkingLevel: "low"` | | `"medium"` | `reasoningEffort: "medium"` | `thinkingLevel: "medium"` | | `"high"` | `reasoningEffort: "high"` | `thinkingLevel: "high"` | Reasoning effort is translated to provider-specific options in `buildSettings()`, so callers never need to think about which provider they are targeting. ### Cost Tracking Per-run cost and usage tracking flows through a `CostCollector`. Construct one per run and pass it to `getModel`; every call issued by that model is metered into the collector: ```ts const costCollector = new CostCollector(); const model = registry.getModel({ model: "GEMINI_3_FLASH_PREVIEW", tag: "assert-checker" }, costCollector); // After execution, aggregate the per-call records: const records = costCollector.getRecords(); // Each record carries { model, tag, inputTokens, outputTokens, reasoningTokens, cacheReadTokens, costMicrodollars }. ``` Keeping this state on a per-run collector (rather than the registry) lets a single shared registry attribute cost to many concurrent runs without mutable per-instance state. Group records by `tag` or `model` to trace costs back to specific use cases. ## Visual AI Primitives ### VisualConditionChecker The base class for checking whether a condition is met on a screenshot. It extends `ObjectGenerator` with a predefined schema: ```ts const checker = new VisualConditionChecker({ model }); const result = await checker.checkCondition( "The login form is visible with email and password fields", screenshot, ); // result: { metCondition: true, reason: "The form is visible with both fields" } ``` Returns `{ metCondition: boolean, reason: string }`. ### AssertChecker Extends `VisualConditionChecker` with a specialized system prompt for test assertions. It handles both positive assertions (“validate there’s a title that says Hello”) and negative assertions (“assert there’s no download button”): ```ts const checker = new AssertChecker(model); const result = await checker.checkCondition( "The submit button is disabled", screenshot, ); ``` Used by the `assert` command to validate each individual assertion against a screenshot. ### VisualChooser Picks which UI element from a set of options matches a user instruction. It draws numbered bounding boxes on the screenshot and asks the model to choose: ```ts const chooser = new VisualChooser({ model }); const result = await chooser.chooseOption({ options: [ { boundingBox: { x: 10, y: 20, width: 100, height: 30 }, description: "Submit" }, { boundingBox: { x: 10, y: 60, width: 100, height: 30 }, description: "Cancel" }, ], instruction: "Click the submit button", screenshot, }); // result: { reasoning: "Option 1 is the submit button", option: { ... } } ``` Throws `NoValidOptionFoundError` if no option matches, or `InvalidIndexError` if the model returns an out-of-bounds index. ### AssertionSplitter Splits a compound assertion instruction into individual atomic assertions that can be checked independently: ```ts const splitter = new AssertionSplitter(model); const result = await splitter.splitAssertions( "validate that the title is visible, the subtitle as well but the button is not", ); // result.assertions: [ // "validate that the title is visible", // "validate that the subtitle is visible", // "validate that the button is not visible" // ] ``` Importantly, the splitter ensures each split assertion contains enough context to stand alone. It repairs incomplete fragments (e.g., “the subtitle as well” becomes “validate that the subtitle is visible”). ## Point Detection Point detectors locate where to interact on screen, given a natural language description. They are used by the `click`, `type`, `hover`, and `drag` commands. ### Abstract Base All point detectors extend `PointDetector`: ```ts abstract class PointDetector { protected abstract detectPointForResolution( screenshot: Screenshot, prompt: string, resolution: ScreenResolution, ): Promise; async detectPoint( screenshot: Screenshot, prompt: string, targetResolution?: ScreenResolution, ): Promise; } ``` The public `detectPoint` method handles resolution fallback automatically - if no target resolution is provided, it defaults to the device resolution (if configured) or the image resolution. ### GeminiComputerUsePointDetector Uses Google’s Gemini computer-use API with a `click_at` tool. The model returns coordinates in a normalized 0-1000 space, which are then scaled to actual pixel coordinates based on the target resolution. ### ObjectPointDetector An adapter that converts an `ObjectDetector` into a `PointDetector`. It detects the bounding box of an element and returns the center point. Useful when you have an object detector but need point-level precision. ## Object Detection ### ObjectDetector (Abstract Base) Detects objects in an image and returns bounding boxes: ```ts abstract class ObjectDetector { async detectObjects( screenshot: Screenshot, prompt: string, targetResolution?: ScreenResolution, ): Promise; } ``` Each `DetectedObject` contains a `boundingBox` and an optional `label`. ### GeminiObjectDetector Uses Gemini’s structured output to return bounding boxes as normalized 0-1000 coordinates. Useful for detecting multiple UI elements at once. ## ObjectGenerator The core structured output engine used by almost every AI primitive in the package. It wraps the AI SDK’s `generateText` with: * **Zod schema validation** for structured JSON output * **Automatic retry** with exponential backoff (default: 5 retries, 100ms initial delay, 2x backoff factor) * **Multimodal input** via `ObjectGenerationParams` - supports text, images, and video * **Null byte stripping** from responses for PostgreSQL compatibility * **Tool support** for agentic generation workflows (stops after 5 tool steps) ```ts const generator = new ObjectGenerator({ model, systemPrompt: "You are a UI analysis expert.", schema: z.object({ elements: z.array(z.object({ label: z.string(), visible: z.boolean(), })), }), }); const result = await generator.generate({ userPrompt: "List all visible buttons", images: [screenshot], }); ``` Video input is supported for models that handle it (checked via `modelSupportsVideo`). Videos are uploaded through the Google GenAI Files API via `VideoProcessor`. If generation fails after all retries, an `ObjectGenerationFailedError` is thrown wrapping the original error. ## Adding a New Model 1. **Add the model entry** to `packages/ai/src/registry/model-entries.ts`: ```ts export const MODEL_ENTRIES = { // ...existing entries MY_NEW_MODEL: { createModel: () => googleProvider.getModel("my-new-model-id"), pricing: simpleCostFunction({ inputCostPerM: 0.5, outputCostPerM: 1.5, }), }, } as const; ``` 2. **Choose the right cost function.** Use `simpleCostFunction` for models without cache pricing, or `inputCacheCostFunction` for models that support input caching (adds a `cachedInputCostPerM` field). 3. **Add a provider** if needed. If the model uses a provider not yet configured, add a new `LLMProvider` singleton in `providers.ts` and add the corresponding API key to `env.ts`. 4. **Use the model** by referencing its key when calling `registry.getModel()`: ```ts const model = registry.getModel({ model: "MY_NEW_MODEL", tag: "my-use-case", reasoning: "medium", }); ``` ## Adding a New Visual AI Primitive Most visual primitives follow the same pattern: extend `ObjectGenerator` with a specialized schema and system prompt. 1. **Define the output schema** with Zod: ```ts const myPrimitiveSchema = z.object({ elements: z.array(z.object({ name: z.string(), confidence: z.number(), })), }); type MyPrimitiveResult = z.infer; ``` 2. **Create the class** extending `ObjectGenerator`: ```ts export class MyPrimitive extends ObjectGenerator { constructor(model: LanguageModel) { super({ model, systemPrompt: "Your specialized system prompt here.", schema: myPrimitiveSchema, }); } async analyze(screenshot: Screenshot, instruction: string): Promise { return this.generate({ images: [screenshot], userPrompt: instruction }); } } ``` 3. **Export it** from the package index. For point or object detection, extend `PointDetector` or `ObjectDetector` instead and implement the `detectPointForResolution` or `detectObjectsForResolution` method. ## Evaluation Framework The `evals/` directory contains a Vitest-integrated framework for benchmarking AI accuracy: * **`Evaluation`** - base class that defines test cases and runs them against models * **`ModelEvaluation`** - tracks token usage and cost per model across an evaluation run * **Three eval types:** * `assert-condition/` - measures assertion checking accuracy * `freestyle-click/` - measures point detection accuracy * `wait-for-instruction/` - measures wait condition generation accuracy Results are saved as JSON with pass rates and per-case breakdowns, making it easy to compare models and track accuracy over time.