Autonomous reliability verification

Prove it
before you ship.

The AI reliability engineer that breaks your integration, repairs it, and proves the repair in a real sandbox.

Preset: Webhook Reliability — Duplicate Fulfillment · synthetic events only
Scenario
Duplicate fulfillment
webhook-idempotency preset
Frozen replay cases
V1–V6
hash-pinned, not model-editable
Patch budget
1 + 2
initial candidate + repairs
Execution
CreateOS
isolated sandbox VM

Run configuration. Every count, exit code and hash on a run page comes from a real sandbox execution.

Run pipelineevery step is a server event
  1. 01Agent writes testsagent
    adversarial replay module
  2. 02CreateOS sandbox runs baselinecreateos
    isolated VM · real exit code
  3. 03Failure reproducedcreateos
    observed, never assumed
  4. 04Agent patchesagent
    handler.mjs only · up to 2 repairs
  5. 05Frozen verifierhost
    V1–V6 · hash-pinned
  6. 06Proof receipthost
    diff · counts · sha256
Scenario 01 · preset

Webhook Reliability — Duplicate Fulfillment

One paid order → one fulfillment, even under duplicate or concurrent delivery.

handler.mjsNode.jssynthetic eventsno live Stripe
V1  same event ×20, overlapping
V2  two event ids, one order
V3  unknown event type
V4  two different orders
V5  fail once, then retry
V6  module interface

Frozen verifier

Host-owned · the patching model can't edit it
{
  "id": "evt_demo_001",
  "type": "payment_intent.succeeded",
  "data": { "object": {
    "id": "pi_demo_001",
    "metadata": {
      "order_id": "order_demo_001"
    }
  } }
}

Real sandbox

Model code runs only inside CreateOS
Evidence, not vibes

What every run proves

Real VM exit codes

Every pass or fail is a CreateOS process exit code

Frozen verifier hash

sha256 pinned on the host, re-measured inside the VM

Fresh-VM verdict

Final check runs in a new VM: patched handler + verifier only

Egress blocked

A request to a non-allowlisted host must fail from inside

Downloadable receipt

Report .md, patch .diff and receipt.json per run

What a verdict means

RUNVA never asks the model whether it fixed the code. A run is verified only when the unmodified, hash-pinned verifier passes in a real CreateOS execution.

  • Baseline failure must be reproduced by a real sandbox exit code
  • The patch is written by the agent from real stdout, stderr and failing assertions
  • The frozen verifier's sha256 is pinned before any model call
  • Pass/fail is parsed on the host from structured verifier output, never from the model
VERIFIED
frozen verifier passed V1–V6 on the patched handler
FAILED
no candidate passed within the attempt budget
INCONCLUSIVE
failure not reproduced, so no repair can be proven
INFRA_ERROR
the environment failed; no verdict is shown