Website Verifiers

​

Access

​

Check Types

​

State Checks

​

Components

​

Example Use Cases

• “events table has entry with title ‘Meeting’” (Calendr)
• “users table has email updated to ‘new@email.com’” (Cloudfile)
• “cart_items table is empty after checkout” (Shopora)
• “orders table has new record with status ‘completed’” (Shopora)
• “messages table has email with subject ‘Re: Project Update’” (Pandora’s Inbox)

​

Best Practices

• Target specific tables and fields
• Use precise conditions to isolate records
• Include both existence and value assertions
• Consider order-independent matching for arrays

​

Log Checks

​

Components

​

Example Use Cases

• “User clicked the ‘Submit’ button”
• “User navigated to /settings”
• “User entered ‘John’ in name field”
• “User selected ‘California’ from dropdown”
• “User opened email thread with subject ‘Q4 Budget Allocation’” (Pandora’s Inbox)

​

What Gets Logged

• Button clicks with element IDs
• Form inputs with values
• Page navigations
• Dropdown selections
• Checkbox/radio changes

​

Rubric Checks

​

Components

​

Behavior

1. Agent completes task and produces response
2. Rubric check returns “PENDING” status
3. External LLM evaluates response against criteria
4. Final pass/fail determined

​

Example Use Cases

• “Response accurately summarizes the calendar events” (Calendr)
• “Agent provided helpful and accurate information”
• “Output follows the requested format”
• “Response correctly identifies the date of the calendar invitation in the email” (Pandora’s Inbox)

​

Response Structure

{
  "passed": true,
  "checks": [
    { "type": "state", "passed": true, "message": "Event created" },
    { "type": "log", "passed": true, "message": "Submit clicked" }
  ],
  "message": "All checks passed"
}

​

Debugging Failed Checks

​

Next Steps