Scenario Validator: iterative fix loop + reality reconciliation

The Environment Factory endpoint exists (Step 4 wrote autonoma/.endpoint-implemented). Your job is to prove it actually works and keep iterating until it does. The E2E test generator (Step 6) is gated on your sentinel — if you do not write autonoma/.endpoint-validated, no tests get generated.

Database safety (absolute)

• ALL writes go through the SDK endpoint only. Never INSERT/UPDATE/DELETE/DROP/TRUNCATE via psql or raw SQL.
• You MAY run SELECT via psql / ORM read queries to verify data.
• The SDK’s down action deletes only what up created (signed refs token).

Inputs

• autonoma/entity-audit.md
• autonoma/scenarios.md (may contain mistakes you will correct)
• The handler file created in Step 4
• A running dev server
• AUTONOMA_SDK_ENDPOINT and AUTONOMA_SHARED_SECRET

Outputs

• autonoma/scenario-recipes.json
• autonoma/.scenario-validation.json
• autonoma/.endpoint-validated

The loop

Repeat until all three actions succeed for every scenario OR you exhaust 5 iterations:

1. Fetch protocol docs (first iteration only):

   curl -sSfL "$(cat autonoma/.docs-url)/llms/protocol.txt"
   curl -sSfL "$(cat autonoma/.docs-url)/llms/scenarios.txt"
   curl -sSfL "$(cat autonoma/.docs-url)/llms/test-planner/step-5-validate.txt"

2. Export working secrets:

   export AUTONOMA_SHARED_SECRET=${AUTONOMA_SHARED_SECRET:-$(openssl rand -hex 32)}
   export AUTONOMA_SIGNING_SECRET=${AUTONOMA_SIGNING_SECRET:-$(openssl rand -hex 32)}

3. Run discover via curl with proper HMAC.

   • Response MUST contain schema.models, schema.edges, schema.relations, schema.scopeField.
   • Coverage check: every model in entity-audit.md MUST appear in schema.models.
   • Factory coverage check: every model with independently_created: true MUST be registered on the handler.
   • Factory-body integrity check (deterministic, MANDATORY): grep the handler for raw DB/ORM writes. Any inline ORM/raw-SQL create inside a factory body for a model marked independently_created: true is a FAIL — fix the handler to import and call the audited function and restart.

4. For each scenario in scenarios.md:

   1. Build {action:"up", create:..., testRunId:"<scenario>-<iteration>"} from the scenario.
   2. HMAC-sign and POST.
   3. On failure, pick one of three paths:
      • Handler bug → fix the handler and restart.
      • Scenario bug (field does not exist, FK target wrong) → edit scenarios.md to match reality and restart. Log the change.
      • Unfeasible scenario → REMOVE it from scenarios.md with justification. Restart.
   4. On 200, parse auth, refs, refsToken.
      • Auth check: auth MUST be non-null and contain at least one of { cookies, headers, token, user }.
      • Refs check: every top-level model in the create tree MUST appear in refs.
   5. Verify DB state with a read-only SELECT for at least one refs id.
   6. POST {action:"down", refsToken}. Expect {ok:true}.
   7. Verify the refs rows are gone.

5. After every scenario passes cleanly, emit the scenario recipes.

   Write autonoma/scenario-recipes.json:

   {
     "version": 1,
     "source": {
       "discoverPath": "autonoma/discover.json",
       "scenariosPath": "autonoma/scenarios.md"
     },
     "validationMode": "endpoint-lifecycle",
     "recipes": [
       {
         "name": "standard",
         "description": "Realistic dataset for core flows",
         "create": {
           "Organization": [{ "_alias": "org1", "name": "Acme Corp" }]
         },
         "variables": {
           "testRunId": { "strategy": "derived", "source": "testRunId", "format": "{testRunId}" }
         },
         "validation": { "status": "validated", "method": "endpoint-up-down", "phase": "ok", "up_ms": 12, "down_ms": 8 }
       }
     ]
   }

   Rules:

   • Top-level keys MUST be exactly version, source, validationMode, recipes
   • version must be integer 1
   • source MUST be an object with BOTH discoverPath (path to autonoma/discover.json) and scenariosPath (path to autonoma/scenarios.md) as non-empty strings. The dashboard /v1/setup/setups/:id/scenario-recipe-versions endpoint will reject the upload if either is missing.
   • validationMode must be sdk-check or endpoint-lifecycle
   • recipes MUST include standard, empty, and large
   • Every recipe MUST contain name, description, create, and validation
   • create MUST use a nested tree rooted at the scope entity. Do NOT use flat top-level model keys connected only by _ref.
   • If create contains {{token}} placeholders, include a variables object. Every {{token}} in create must match a key in variables; every key in variables must be used in create.

6. Run preflight on the emitted recipes:

   python3 "$(cat /tmp/autonoma-plugin-root)/hooks/preflight_scenario_recipes.py" \
     autonoma/scenario-recipes.json

   This resolves tokenized payloads and re-runs signed up/down against the live endpoint. If preflight exits non-zero, fix the failing recipe and re-run.

7. Write autonoma/.scenario-validation.json:

   {
     "status": "ok",
     "preflightPassed": true,
     "smokeTestPassed": true,
     "validatedScenarios": ["standard", "empty", "large"],
     "failedScenarios": [],
     "blockingIssues": [],
     "recipePath": "autonoma/scenario-recipes.json",
     "validationMode": "endpoint-lifecycle",
     "endpointUrl": "http://localhost:3000/api/autonoma"
   }

8. Write the sentinel autonoma/.endpoint-validated via the Write tool (NOT touch) with a short plain-text report.

Iteration discipline

• One handler fix per iteration, then re-run everything.
• If the same scenario fails twice in a row with the same error, the scenario itself is probably wrong — prefer editing scenarios.md.
• If you have edited scenarios.md, re-read it from disk after every edit.

When you hit the 5-iteration cap

STOP and write a clear failure report. Do NOT write .endpoint-validated. Include the last failing curl body + response, which scenario(s) failed, and which handler file + line range is most likely at fault. The orchestrator surfaces this to the user.

scenarios.md reconciliation rules

Preserve the frontmatter shape (the validator hook checks it). Allowed:

• Drop a scenario entirely (decrement scenario_count, update the scenarios summary).
• Remove/rename fields on a model to match what discover reports.
• Adjust FK aliases so they reference models that actually exist.
• Flatten cross-branch references that the handler cannot resolve.

Disallowed: silently changing a scenario’s intent (e.g. renaming “admin with one project” to “user with one project” without reflecting that in the description).