Schema: ConformanceProfile

File: conformance-profile.json

Used by: conformance/run.py · conformance/profiles/

Purpose

A profile tells the conformance kit how to invoke a CLI and which invocations are safe to probe. The kit cannot guess a tool's subcommands or which of them delete data, so the author of the profile states it once and every run is reproducible.

Key decisions:

Probes declare intent. kind says what a correct CLI does with the call: finish without side effects, exit 2, refuse until confirmed, or stream JSONL lines to a terminal line
Confirmation flags never appear in a probe. Destructive probes run without confirmation and with their declared dry_run_flag; the kit never deletes on purpose
Paths are relative to the profile. A committed profile works from any working directory

Values

Field	Type	Required	Description
`schema_version`	`"1.0"`	yes	Profile format version
`tool`	string	yes	Display name
`command`	string[]	yes	Invocation prefix; a first element containing `/` resolves against the profile directory, a bare name through `PATH`
`timeout_seconds`	number `(0, 120]`	yes	Per-run limit; exceeding it counts as a hang
`manifest`	string[]	no	Arguments that print the manifest; omit to skip `manifest_valid`
`argument_order`	`ArgumentOrder`	no	A read command and a global option to move around it; omit to skip `argument_order`
`probes`	`Probe[]`	yes	Invocations to run

Probe

Field	Type	Required	Description
`name`	string	yes	Unique label shown in evidence
`argv`	string[]	yes	Arguments after the prefix
`kind`	`"read"` \| `"destructive"` \| `"invalid"` \| `"stream"`	yes	Expected behavior class
`dry_run_flag`	string	when destructive	Flag that turns the probe into a preview; not allowed on a `stream` probe
`deadline_seconds`	number `(0, 120]`	no	`stream` only: limit for the whole stream; defaults to `timeout_seconds`
`signal`	`"INT"`	no	`stream` only: signal sent after `after_lines` lines; requires `after_lines`
`after_lines`	integer `>= 1`	no	`stream` only: stdout lines read before `signal` is sent; requires `signal`

A stream probe runs once and skips the single-envelope checks. The kit reads its stdout line by line until the process exits, and kills the process group at the deadline (see conformance/README.md).

ArgumentOrder

The kit runs command_path with local_args, placing global_flag before the command path, between the path and the local option, and after the local option. Every placement must exit 0 with the same result; alternate_value must change stdout identically in every placement; the flag given twice with value and then alternate_value must exit 2. With positional, the kit also runs local_args after and before the positional: both must give the same data, and that data must differ from the positional alone, which proves the option was parsed rather than read as a second positional.

Field	Type	Required	Description
`command_path`	string[]	yes	Path of a side-effect-free command
`local_args`	string[]	yes	A command-local option and its value
`global_flag`	string	yes	Long name of a global option, usually `--format`
`value`	string	yes	Value whose stdout is a `ResponseEnvelope`, usually `json`
`alternate_value`	string	yes	Another accepted value with different stdout, such as `plain`; must differ from `value`
`positional`	string	no	A positional `command_path` accepts; `local_args` must change the result when given after it

Examples

Valid

{
  "schema_version": "1.0",
  "tool": "democli",
  "command": ["../../benchmark/harness/cli/good/democli"],
  "timeout_seconds": 5,
  "manifest": ["manifest"],
  "argument_order": { "command_path": ["deployments", "list"], "local_args": ["--cursor", "cGFnZTI="], "global_flag": "--format", "value": "json", "alternate_value": "plain", "positional": "staging" },
  "probes": [
    { "name": "list deployments", "argv": ["deployments", "list"], "kind": "read" },
    { "name": "unknown flag", "argv": ["deployments", "list", "--no-such-flag"], "kind": "invalid" },
    { "name": "delete staging", "argv": ["deployments", "delete", "--filter", "env=staging"], "kind": "destructive", "dry_run_flag": "--dry-run" },
    { "name": "stream deployments", "argv": ["deployments", "list", "--stream"], "kind": "stream" },
    { "name": "interrupt the deployment stream", "argv": ["deployments", "list", "--stream"], "kind": "stream", "signal": "INT", "after_lines": 2 }
  ]
}

Invalid — destructive probe without a preview flag

{
  "schema_version": "1.0",
  "tool": "democli",
  "command": ["democli"],
  "timeout_seconds": 5,
  "probes": [
    { "name": "delete staging", "argv": ["deployments", "delete", "--filter", "env=staging"], "kind": "destructive" }
  ]
}

Violation: dry_run_flag is required when kind is destructive.

Invalid — signal without a line count

{ "name": "interrupt", "argv": ["events", "watch"], "kind": "stream", "signal": "INT" }

Violation: signal and after_lines require each other, and both apply only to stream probes.

Common mistakes

Putting --yes or --force in a destructive probe. The kit then executes the deletion for real; confirmation flags never belong in a profile
Marking a mutating command as read. Read probes run several times (stdin closed, stdin open, NO_COLOR); anything that writes will write repeatedly
Pointing a profile at production credentials. Probes call the real tool; use a sandbox account or a mock
Adding --format json to probe argv. The envelope check exists to prove JSON activates in a non-TTY without flags (REQ-F-003); argument_order is the one place a profile names the format flag
Choosing local_args that do not change the result. With positional, the kit proves the option was parsed by comparing against the positional alone; an option with no visible effect fails that comparison
Choosing an alternate_value that renders like the default. The overwrite check compares stdout; an alternate that prints the same bytes as value reads as an ignored option
Setting after_lines beyond what the stream writes. A stream that ends first never receives SIGINT, and stream_sigint fails; pick a count well inside a normal run
Pointing a stream probe at a stream that never ends without a signal. The kit waits for the terminal line and fails at the deadline; interrupt follow-mode streams with signal

Agent interpretation

Generate a profile from tool manifest: danger_level: "safe" commands become read probes, destructive commands become destructive probes with the declared dry-run flag, and safe commands with streaming_default: true or a --stream flag become stream probes
Never add a probe for a command whose danger_level is mutating unless a sandbox is confirmed
Keep timeout_seconds short (5 or less); the hang checks rely on it

Coding agent notes

Validate the profile against this schema before running anything; conformance/run.py exits 2 with INVALID_PROFILE otherwise
Commit profiles next to the CLI they describe and run the kit in that CLI's CI
Tests: a profile with a destructive probe and no dry_run_flag is rejected; duplicate probe names are rejected; signal without after_lines, or either on a non-stream probe, is rejected

Implementation notes

Probe kinds are deliberately few. A mutating kind was left out because the kit cannot undo side effects, and a conformance check that damages state is worse than no check. Commands that mutate are covered by the destructive-refusal and dry-run checks when they declare a preview flag. A stream probe is side-effect free like read; REQ-O-004 refuses destructive streams, so the kit never interrupts one that deletes.