ct run sabotage-eval
Run sabotage evaluation: honest eval + attack eval + monitor + graphs.
Usage
ct run sabotage-eval [OPTIONS]
Options
| Option | Description |
|---|---|
-c, --config PATH | YAML config file for multi-attack mode. |
--attack-policy TEXT | Attack policy name. |
--attack-policy-arg TEXT | Attack policy args in key=value format (repeatable). |
--honest-policy TEXT | Honest policy name. |
--honest-policy-arg TEXT | Honest policy args in key=value format (repeatable). |
--protocol TEXT | Blue protocol to run both evals under (e.g. defer-to-resample, trusted-editing). Attacker/honest become the protocol's untrusted policy. |
--protocol-arg TEXT | Protocol args in key=value format (repeatable). |
--audit-budgets TEXT | Audit budget(s), comma-separated (e.g. 0.0001,0.001,0.01). Each produces graphs. |
--thresholds TEXT | Fixed threshold(s), comma-separated (e.g. 3.0,4.0,5.0). Skips honest eval. |
--model-role TEXT | Bind an eval-level model role as role=model (inspect's --model-role). Only trusted — the model the monitors run on — is bindable here; the untrusted models come from the policy args. |
--task-file TEXT | Task set name or path. |
--main-tasks-with-no-scorers [error|include|exclude] | Main tasks lacking a scorer: refuse the run (error), keep them recorded as unscored (include), or drop them (exclude). |
--log-dir TEXT | Where eval logs go. |
-o, --output-dir TEXT | Where graphs and summary go. |
--ec2 | Run evals on EC2. |
--no-graph | Skip graph generation. |
--no-upload | Disable uploading results. |
--dry-run | Validate the local execution plan and print it without executing. |
--no-cache | Re-run every attack even when a prior results.json recorded an identical one. |
--preflight | Probe the run's external contracts (monitor selections, model endpoints, pricing, credentials) and exit; runs nothing and spends nothing. |
--tags TEXT | Tags for the run (repeatable). Added to all eval runs alongside built-in sabotage-eval tags. |
--help | Show this message and exit. |