cube — CLI Agent Evaluation
Evaluated against the CLI Agent Spec — a specification defining 74 failure modes for CLI tools used under AI agent orchestration
CLI version: 1.7.16 Evaluated: 2026-08-06 Scope: Critical severity (22 of 74 failure modes)
Scores
| Metric | Result |
|---|---|
| Failure mode score | 0.8/3 — 2 passing · 11 partial · 9 failing |
| Readiness score | 7/15 [C] |
| Observed bugs | 17 confirmed during live evaluation |
| Worst gaps | §10, §11, §12, §23, §25, §43, §60, §64, §74 |
Key Findings
- The official installer is non-interactive and idempotent, but agent-specific setup guidance is not packaged in
AGENTS.md - OAuth login launches a browser and blocks in headless execution; use
CUBE_API_URLandCUBE_API_KEYinstead - JSON success output lacks a stable envelope, while failures remain prose on stderr with generic exit 1
- Large responses are emitted in full only after completion, with no size bound, truncation metadata, or heartbeat
- Mutating and destructive operations lack idempotency keys, dry-run previews, and structured partial-failure recovery
Files
| File | What it is |
|---|---|
| report-index.md | Full scorecard — all failure modes, readiness breakdown, links to all reports |
| report-issues.md | Concrete bugs and gaps agents will hit when using this CLI as-is |
| report-runtime.md | Compact operational brief — what to set, what to avoid, what to watch for |
| report-agent-dev.md | Integration guide — invocation invariants and per-gap workarounds for agent developers |
| report-dev.md | Fix list for CLI authors — what to implement, mapped to spec requirements |
| findings.md | Raw scorecard — one row per evaluated failure mode |
| issues.md | Observed bugs recorded during live evaluation |
| trace.md | Audit trail — exact check commands, exit codes, stdout/stderr per §N |
| environment.md | CLI environment profile — binary path, version, flags, timeout method |
| readiness.md | Proactive readiness scores across 5 dimensions |
Generated by cli-agent-audit · CLI Agent Spec