neon - CLI Agent Evaluation
Evaluated against the CLI Agent Spec, a specification defining 71 failure modes for CLI tools used under AI agent orchestration.
CLI version: 2.30.1 Evaluated: 2026-07-05 Scope: Critical (22 of 71 failure modes)
Scores
| Metric | Result |
|---|---|
| Failure mode score | 0.4/3 - 0 passing · 7 partial · 12 failing · 3 indeterminate |
| Readiness score | 8/15 [C] |
| Observed bugs | 6 confirmed during live evaluation |
| Worst gaps | §45 Headless Authentication, §10 Interactivity, §1 Exit Codes |
Key Findings
- Neon CLI tries browser OAuth in headless/no-credential command paths instead of returning a structured auth-required response.
- The documented
link --no-checksoffline path still triggered login in the audit probe. --output jsonis available, but error paths emit prose stderr rather than a stable JSON envelope.- Destructive and mutating workflows lack dry-run, idempotency, and machine-readable effect contracts.
- The CLI has useful docs and agent-mode islands, but no CLI-wide schema/manifest for commands, scopes, exit codes, or interactivity.
Files
| File | What it is |
|---|---|
| report-index.md | Full scorecard, readiness breakdown, and links to all reports |
| report-issues.md | Concrete bugs and gaps agents will hit when using this CLI as-is |
| report-runtime.md | Compact operational brief for agents invoking Neon CLI |
| report-agent-dev.md | Integration guide and invocation invariants for agent developers |
| report-dev.md | Fix list for CLI authors |
| findings.md | Raw scorecard, one row per evaluated failure mode |
| issues.md | Observed bugs recorded during live evaluation |
| trace.md | Audit trail with check commands, exit codes, stdout, and stderr |
| environment.md | CLI environment profile |
| readiness.md | Proactive readiness scores across 5 dimensions |
Generated by cli-agent-audit · CLI Agent Spec