> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gym.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Desktop Verifiers

> File comparison and rules-based evaluation for desktop environments

Desktop environments use evaluator functions that compare VM state against expected outcomes (gold standards).

<Info>
  Explore live evaluator examples at [gym.scale.com](https://gym.scale.com).
</Info>

## Access

| Method | Endpoint               |
| ------ | ---------------------- |
| API    | `POST /run_evaluator`  |
| Web UI | "Run Evaluator" button |

## Evaluator Structure

The evaluator is defined in the `task_config.evaluator` object:

```json theme={null}
{
  "evaluator": {
    "func": "compare_table",
    "expected": { ... },
    "result": { ... },
    "postconfig": [ ... ],
    "options": { ... }
  }
}
```

***

## Core Fields

| Field        | Type   | Description                        |
| ------------ | ------ | ---------------------------------- |
| `func`       | string | Evaluation function to use         |
| `expected`   | object | Gold standard file configuration   |
| `result`     | object | Result file from the VM to compare |
| `postconfig` | array  | Actions to run before evaluation   |
| `options`    | object | Comparison rules and settings      |

***

## File Types

### Cloud File (Gold Standard)

Files hosted at a URL, typically your expected output:

```json theme={null}
{
  "expected": {
    "type": "cloud_file",
    "path": "https://example.s3.amazonaws.com/gold.xlsx",
    "dest": "gold.xlsx"
  }
}
```

### VM File (Result)

Files on the VM filesystem, the agent's output:

```json theme={null}
{
  "result": {
    "type": "vm_file",
    "path": "/Users/user/Desktop/output.xlsx",
    "dest": "output.xlsx"
  }
}
```

***

## Postconfig Actions

Actions executed before evaluation to prepare VM state (e.g., save files).

### Execute

Run a shell command:

```json theme={null}
{
  "type": "execute",
  "parameters": {
    "command": ["osascript", "-e", "tell application \"Microsoft Excel\" to activate"]
  }
}
```

### Sleep

Wait for specified seconds:

```json theme={null}
{
  "type": "sleep",
  "parameters": { "seconds": 1 }
}
```

### Common Postconfig Pattern

Save file before evaluation (macOS Excel example):

```json theme={null}
{
  "postconfig": [
    {
      "type": "execute",
      "parameters": {
        "command": ["osascript", "-e", "tell application \"Microsoft Excel\" to activate"]
      }
    },
    { "type": "sleep", "parameters": { "seconds": 1 } },
    {
      "type": "execute",
      "parameters": {
        "command": ["python", "-c", "import pyautogui; pyautogui.hotkey('command', 's')"]
      }
    },
    { "type": "sleep", "parameters": { "seconds": 0.5 } }
  ]
}
```

***

## Comparison Rules

Rules define how files are compared. Specified in `options.rules`:

```json theme={null}
{
  "options": {
    "rules": [
      { "type": "pivot_table", ... },
      { "type": "freeze", ... }
    ]
  }
}
```

### Available Rule Types

| Rule Type     | Description                   | Use Case               |
| ------------- | ----------------------------- | ---------------------- |
| `pivot_table` | Compare pivot table structure | Excel pivot tasks      |
| `freeze`      | Compare freeze pane settings  | Spreadsheet formatting |
| `exact_match` | Byte-for-byte comparison      | Exact output tasks     |
| `structural`  | Compare document structure    | Document editing       |

### Pivot Table Rule

```json theme={null}
{
  "type": "pivot_table",
  "sheet_idx0": "RNPivotData",
  "sheet_idx1": "ENPivotData",
  "pivot_props": ["col_fields", "row_fields", "data_fields"]
}
```

### Freeze Rule

```json theme={null}
{
  "type": "freeze",
  "sheet_idx0": "RNPivotData",
  "sheet_idx1": "ENPivotData"
}
```

***

## Evaluator Functions

The `func` field specifies which evaluator to use. Over 100 functions are available, organized by category:

### Spreadsheets & Tables

| Function        | Description                                                                |
| --------------- | -------------------------------------------------------------------------- |
| `compare_table` | Excel/spreadsheet comparison with rules (pivot tables, freeze panes, etc.) |
| `compare_csv`   | CSV file comparison                                                        |

### Documents (Word/DOCX)

| Function                 | Description              |
| ------------------------ | ------------------------ |
| `compare_docx_files`     | Full document comparison |
| `compare_docx_lines`     | Line-by-line comparison  |
| `compare_line_spacing`   | Spacing verification     |
| `compare_font_names`     | Font verification        |
| `is_first_line_centered` | Alignment check          |

### Presentations (PowerPoint/PPTX)

| Function                           | Description                  |
| ---------------------------------- | ---------------------------- |
| `compare_pptx_files`               | Full presentation comparison |
| `check_slide_orientation_Portrait` | Orientation check            |
| `check_image_stretch_and_center`   | Image positioning            |

### Images

| Function              | Description                 |
| --------------------- | --------------------------- |
| `compare_images`      | Image similarity comparison |
| `compare_image_list`  | Multiple image comparison   |
| `check_structure_sim` | Structural similarity       |

### General

| Function            | Description              |
| ------------------- | ------------------------ |
| `exact_match`       | Byte-for-byte comparison |
| `fuzzy_match`       | Fuzzy string matching    |
| `compare_text_file` | Text file comparison     |
| `compare_pdfs`      | PDF comparison           |
| `check_json`        | JSON validation          |
| `check_file_exists` | File existence check     |
| `infeasible`        | Mark task as impossible  |

### VS Code

| Function                          | Description           |
| --------------------------------- | --------------------- |
| `is_extension_installed`          | Extension check       |
| `check_json_settings`             | Settings verification |
| `check_python_file_by_test_suite` | Run Python tests      |

### Browser

| Function                           | Description           |
| ---------------------------------- | --------------------- |
| `is_expected_bookmarks`            | Bookmark verification |
| `is_expected_tabs`                 | Tab verification      |
| `is_expected_installed_extensions` | Extension check       |

<Accordion title="View all 100+ functions">
  **Documents**: `compare_docx_tables`, `compare_highlighted_text`, `compare_insert_equation`, `compare_references`, `has_page_numbers_in_footers`, `check_highlighted_words`, `check_tabstops`, `contains_page_break`, `evaluate_alignment`, `evaluate_spacing`

  **Slides**: `check_strikethrough`, `check_transition`, `check_slide_numbers_color`, `check_presenter_console_disable`, `evaluate_presentation_fill_to_rgb_distance`

  **Images**: `check_image_size`, `check_image_mirror`, `check_brightness_decrease_and_structure_sim`, `check_contrast_increase_and_structure_sim`, `check_saturation_increase_and_structure_sim`

  **Browser**: `compare_htmls`, `compare_archive`, `is_cookie_deleted`, `check_history_deleted`, `is_expected_url_pattern_match`, `compare_pdf_images`

  **General**: `check_csv`, `check_list`, `check_include_exclude`, `run_sqlite3`, `file_contains`, `diff_text_file`, `literal_match`, `is_in_list`

  **VS Code**: `compare_config`, `compare_zip_files`, `check_json_keybindings`, `check_python_file_by_gold_file`

  **Media**: `compare_audios`, `compare_videos`, `is_vlc_playing`, `is_vlc_fullscreen`

  **Other**: `compare_epub`, `check_mp3_meta`, `check_pdf_pages`, `check_thunderbird_prefs`
</Accordion>

***

## Response Structure

```json theme={null}
{
  "status": "success",
  "message": "Verifier passed",
  "vm_id": "vm-abc123",
  "task_id": "task-001",
  "evaluation_score": 1
}
```

### Score Interpretation

| Score | Meaning                             |
| ----- | ----------------------------------- |
| `1.0` | Task completed successfully         |
| `0.0` | Task not completed or failed checks |

### Failed Example

```json theme={null}
{
  "status": "success",
  "message": "Rule 1 (pivot_table): Pivot tables differ between RNPivotData and ENPivotData",
  "vm_id": "vm-abc123",
  "task_id": "task-001",
  "evaluation_score": 0
}
```

***

## Complete Example

Excel pivot table verification:

```json theme={null}
{
  "evaluator": {
    "func": "compare_table",
    "expected": {
      "type": "cloud_file",
      "path": "https://temp-cua.s3.us-west-2.amazonaws.com/gold.xlsx",
      "dest": "historical_unemployment-gold.xlsx"
    },
    "result": {
      "type": "vm_file",
      "path": "/Users/user/Desktop/historical_unemployment.xlsx",
      "dest": "historical_unemployment.xlsx"
    },
    "postconfig": [
      {
        "type": "execute",
        "parameters": {
          "command": ["osascript", "-e", "tell application \"Microsoft Excel\" to activate"]
        }
      },
      { "type": "sleep", "parameters": { "seconds": 1 } },
      {
        "type": "execute",
        "parameters": {
          "command": ["python", "-c", "import pyautogui; pyautogui.hotkey('command', 's')"]
        }
      }
    ],
    "options": {
      "rules": [
        {
          "type": "pivot_table",
          "sheet_idx0": "RNPivotData",
          "sheet_idx1": "ENPivotData",
          "pivot_props": ["col_fields", "row_fields", "data_fields"]
        },
        {
          "type": "freeze",
          "sheet_idx0": "RNPivotData",
          "sheet_idx1": "ENPivotData"
        }
      ]
    }
  }
}
```

***

## Best Practices

<Accordion title="Always save files before evaluation">
  Many applications don't auto-save. Use `postconfig` to trigger save before evaluation.
</Accordion>

<Accordion title="Add delays for UI stability">
  Use `sleep` actions between commands to allow the UI to settle (typically 0.5-1 second).
</Accordion>

<Accordion title="Use specific rules for complex comparisons">
  For spreadsheets, define exactly which aspects to compare rather than using exact match.
</Accordion>

<Accordion title="Test gold standards manually first">
  Verify your expected output is correct before using it for automated evaluation.
</Accordion>

***

## Next Steps

<CardGroup cols={2}>
  <Card title="Run Evaluator API" icon="code" href="/api-reference/desktop/control-plane/run-evaluator">
    Full API endpoint documentation
  </Card>

  <Card title="MCP Verifiers" icon="wrench" href="/deep-dives/verifiers/mcp">
    LLM Judge for tool-use tasks
  </Card>
</CardGroup>