temporal — CLI Agent Evaluation
Evaluated against the CLI Agent Spec — a specification defining 71 failure modes for CLI tools used under AI agent orchestration.
CLI version: temporal version 1.7.2 (Server 1.31.1, UI 2.49.1) Evaluated: 2026-07-07 Scope: critical (22 of 71 failure modes)
Scores
| Metric | Result |
|---|---|
| Failure mode score | 1.1/3 — 2 passing · 15 partial · 4 failing · 1 indeterminate |
| Readiness score | 3/15 [F] |
| Observed bugs | 4 confirmed during live evaluation |
| Worst gaps | §1, §13, §25, §74 |
Key Findings
--output jsonworks for success paths but not for validation, timeout, or connection failures.- Missing args, not-found, network, and timeout-like failures collapse to exit 1.
- Destructive commands have partial prompt protection but no dry-run/effect contract.
- No machine-readable schema, command manifest, required scopes, or check-permissions command was found.
- API keys can be passed through argv, which is risky for agent traces and process listings.
Files
| File | What it is |
|---|---|
| report-index.md | Full scorecard — all failure modes, readiness breakdown, links to all reports |
| report-issues.md | Concrete bugs and gaps agents will hit when using this CLI as-is |
| report-runtime.md | Compact operational brief — what to set, what to avoid, what to watch for |
| report-agent-dev.md | Integration guide — invocation invariants and per-gap workarounds for agent developers |
| report-dev.md | Fix list for CLI authors — what to implement, mapped to spec requirements |
| findings.md | Raw scorecard — one row per evaluated failure mode |
| issues.md | Observed bugs recorded during live evaluation |
| trace.md | Audit trail — exact check commands, exit codes, stdout/stderr per §N |
| environment.md | CLI environment profile — binary path, version, flags, timeout method |
| readiness.md | Proactive readiness scores across 5 dimensions |
Generated by cli-agent-audit · CLI Agent Spec