> ## Documentation Index
> Fetch the complete documentation index at: https://docs.gym.scale.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Run Evaluator

> Evaluate task completion and get a score

Runs the evaluator to check if a task was completed successfully. Compares the current VM state against the expected outcome defined in the task configuration and produces a score from 0.0 to 1.0.

<Info>
  Explore live examples of evaluator configurations at [gym.scale.com](https://gym.scale.com).
</Info>

## Request Body

<ParamField body="vm_id" type="string" required>
  VM identifier
</ParamField>

<ParamField body="task_config" type="object" required>
  Task configuration with evaluator definition
</ParamField>

## Evaluator Configuration

The `evaluator` object within `task_config` defines how to verify task completion.

### Core Fields

<ParamField body="evaluator.func" type="string" required>
  Evaluation function to use (e.g., `compare_table`, `file_check`, `exact_match`)
</ParamField>

<ParamField body="evaluator.expected" type="object" required>
  Expected/gold standard file configuration
</ParamField>

<ParamField body="evaluator.result" type="object" required>
  Result file from the VM to compare
</ParamField>

<ParamField body="evaluator.postconfig" type="array">
  Actions to run before evaluation (e.g., save file, activate app)
</ParamField>

<ParamField body="evaluator.options" type="object">
  Evaluation options and rules
</ParamField>

### Expected/Result File Types

| Type         | Description                              |
| ------------ | ---------------------------------------- |
| `cloud_file` | File hosted at a URL (for gold standard) |
| `vm_file`    | File on the VM filesystem (for result)   |

### Postconfig Actions

Actions to prepare the VM state before evaluation:

| Type      | Description                |
| --------- | -------------------------- |
| `execute` | Run a shell command        |
| `sleep`   | Wait for specified seconds |

### Evaluation Rules

Rules define how files are compared:

| Rule Type     | Description                   |
| ------------- | ----------------------------- |
| `pivot_table` | Compare pivot table structure |
| `freeze`      | Compare freeze pane settings  |
| `exact_match` | Byte-for-byte comparison      |
| `structural`  | Compare document structure    |

## Response

<ResponseField name="status" type="string">
  Result status (`success`)
</ResponseField>

<ResponseField name="message" type="string">
  Evaluation result message (describes pass/fail reason)
</ResponseField>

<ResponseField name="vm_id" type="string">
  VM identifier
</ResponseField>

<ResponseField name="task_id" type="string">
  Task identifier
</ResponseField>

<ResponseField name="evaluation_score" type="number">
  Score from 0.0 to 1.0 (1.0 = fully completed)
</ResponseField>

<Note>
  Response structure may be simplified to `{"score": number, "message": string}` in future versions.
</Note>

## Example: Excel Pivot Table Verification

This example verifies that a pivot table was correctly created in Excel:

<RequestExample>
  ```bash cURL theme={null}
  curl -X POST "http://CONTROL_PLANE_IP:PORT/run_evaluator" \
    -H "Content-Type: application/json" \
    -d '{
      "vm_id": "vm-abc123",
      "task_config": {
        "id": "task-001",
        "instruction": "Create a pivot table from the unemployment data",
        "evaluator": {
          "func": "compare_table",
          "expected": {
            "type": "cloud_file",
            "path": "https://temp-cua.s3.us-west-2.amazonaws.com/gold.xlsx",
            "dest": "historical_unemployment-gold.xlsx"
          },
          "result": {
            "type": "vm_file",
            "path": "/Users/user/Desktop/historical_unemployment.xlsx",
            "dest": "historical_unemployment.xlsx"
          },
          "postconfig": [
            {
              "type": "execute",
              "parameters": {
                "command": ["osascript", "-e", "tell application \"Microsoft Excel\" to activate"]
              }
            },
            {
              "type": "sleep",
              "parameters": { "seconds": 1 }
            },
            {
              "type": "execute",
              "parameters": {
                "command": ["python", "-c", "import pyautogui; pyautogui.hotkey('command', 's')"]
              }
            },
            {
              "type": "sleep",
              "parameters": { "seconds": 0.5 }
            }
          ],
          "options": {
            "rules": [
              {
                "type": "pivot_table",
                "sheet_idx0": "RNPivotData",
                "sheet_idx1": "ENPivotData",
                "pivot_props": ["col_fields", "row_fields", "data_fields"]
              },
              {
                "type": "freeze",
                "sheet_idx0": "RNPivotData",
                "sheet_idx1": "ENPivotData"
              }
            ]
          }
        }
      }
    }'
  ```
</RequestExample>

<ResponseExample>
  ```json Passed theme={null}
  {
    "status": "success",
    "message": "Verifier passed",
    "vm_id": "vm-abc123",
    "task_id": "task-001",
    "evaluation_score": 1
  }
  ```

  ```json Failed - Pivot Table Mismatch theme={null}
  {
    "status": "success",
    "message": "Rule 1 (pivot_table): Pivot tables differ between RNPivotData and ENPivotData",
    "vm_id": "vm-abc123",
    "task_id": "task-001",
    "evaluation_score": 0
  }
  ```

  ```json Error - VM Not Found theme={null}
  {
    "error": "Desktop environment not found for VM: vm-abc123. Call /create_desktop first."
  }
  ```
</ResponseExample>

## Evaluator Functions

Over 100 functions available. Common ones:

| Function                 | Use Case                     |
| ------------------------ | ---------------------------- |
| `compare_table`          | Excel/spreadsheet with rules |
| `compare_docx_files`     | Word documents               |
| `compare_pptx_files`     | PowerPoint presentations     |
| `compare_pdfs`           | PDF files                    |
| `compare_images`         | Image similarity             |
| `compare_text_file`      | Text files                   |
| `exact_match`            | Byte-for-byte comparison     |
| `fuzzy_match`            | Fuzzy string matching        |
| `check_json`             | JSON validation              |
| `check_file_exists`      | File existence               |
| `is_extension_installed` | VS Code extensions           |
| `infeasible`             | Task cannot be completed     |

See [Desktop Verifiers](/deep-dives/verifiers/desktop) for the complete list.

## Best Practices

<Accordion title="Use postconfig to save files">
  Many applications don't auto-save. Use `postconfig` to trigger save before evaluation:

  ```json theme={null}
  {
    "postconfig": [
      {
        "type": "execute",
        "parameters": {
          "command": ["python", "-c", "import pyautogui; pyautogui.hotkey('command', 's')"]
        }
      }
    ]
  }
  ```
</Accordion>

<Accordion title="Add delays for UI stability">
  Use `sleep` actions between commands to allow the UI to settle:

  ```json theme={null}
  {
    "type": "sleep",
    "parameters": { "seconds": 1 }
  }
  ```
</Accordion>

<Accordion title="Use specific rules for complex comparisons">
  For spreadsheets, define which aspects to compare:

  ```json theme={null}
  {
    "options": {
      "rules": [
        { "type": "pivot_table", "pivot_props": ["col_fields", "row_fields"] },
        { "type": "freeze" }
      ]
    }
  }
  ```
</Accordion>