CLI Reference

skilled provides commands for validating Skills and evaluation suites, running cases, generating and comparing reports, importing formats, and serving MCP.

skilled setup

skilled setup --host codex
skilled setup --host claude-code --project /absolute/path/to/project
skilled setup --host codex --remove

Available from version 0.3.0. Installs the bundled skilled-improve workflow into .agents/skills/skilled-improve for Codex or .claude/skills/skilled-improve for Claude Code. --host is required; --project defaults to the current directory and must name an existing project. Start a new host session after setup.

Repeat setup after upgrading the CLI. It updates only an unchanged managed copy; local edits and unrelated host settings are preserved. --remove removes only an unchanged managed copy. Setup does not log in to an engine or approve host tools. See getting started for the evaluation workflow.

skilled mcp

skilled mcp --workspace /absolute/path/to/project
skilled mcp --workspace /absolute/path/to/project --allow-execution

Starts a stdio MCP server with validation, comparison and run tools, bundled workflow resources and the skilled-improve prompt. The required --workspace sets the root for tool paths. Execution is disabled unless --allow-execution is set; dry-run and read-only tools work without it. See MCP setup.

skilled compare

skilled compare <before.json> <after.json> [--format text|json] [--fail-on-regression]

Compares existing evaluation reports by case ID and configuration. Defaults to text output. The optional regression flag returns exit code 1 for matched-case regressions; added/removed coverage is reported separately. No model is called. See comparison semantics and scoring.

skilled run

Run evaluation cases and produce reports.

bash

skilled run [path] [flags]

Argument

Argument	Description
`path`	Path to `eval.yaml`. Defaults to `evals/eval.yaml` in the current dir

Flags

Flag	Default	Description
`--auto`	`false`	Auto-detect the `evals/` directory; can directly consume an Anthropic `evals.json`
`--include-case-name`	—	Run only matching cases (glob; can be repeated)
`--exclude-case-name`	—	Exclude matching cases (glob; can be repeated)
`--format`	—	Extra report formats: `junit` / `html` (repeatable). `result.json` is always written; `--format junit` produces `report.xml`, `--format html` produces `report.html`; `--format json` is a no-op
`--output-dir`	`<skill-name>-workspace/` next to the skill dir	Output directory for reports and artifacts
`--workspace`	—	Reuse an existing host directory as the agent workspace. Supported only with `environment.type: none`, `cases.parallelism: 1`, and benchmark disabled. The agent can modify this directory, but skilled never deletes it.
`--iteration`	`0` (auto)	Repeat selected cases for stability/flakiness sampling. `0` auto-appends one run after the latest `iteration-N/` without summarizing history; positive `N` runs N samples and writes `iteration-1/` … `iteration-N/`; when `N > 1`, the terminal summary covers only samples from the current command
`--engine`	From config	Override engine name
`--provider`	From config	Override `engine.model.provider`. When set, the complete `--model` value is preserved as an opaque model ID.
`--model`	From config	Override model. Without `--provider`, the legacy `provider/name` form is split when the prefix is configured; otherwise the complete value is preserved.
`--parallelism`	From config	Override `cases.parallelism`. Allowed range: 1–256
`--baseline`	From config	Enable with/without-Skill comparison. Requires the verified isolated `docker` runtime; native `none` is rejected because separate folders do not restrict reads.
`--api-key`	—	Pass an API key (higher precedence than env vars)
`--no-delete`	`false`	Preserve workspaces or containers created by skilled after evaluation for debugging.
`--event-log`	—	Write an ordered v1 JSONL evaluation event stream to a local file. The parent directory must already exist
`--event-attribute`	—	Attach a namespaced `key=value` string attribute to every event (repeatable; requires `--event-log`)
`-v, --verbose`	`0`	Increase log verbosity. Default `info`; `-v` / `--verbose` / `--verbose=true` → `debug`; `-vv` / `--verbose=2` → `trace`; `--verbose=false` disables extra detail

Validation scope. run validates the eval-level config plus only the cases selected after --include-case-name / --exclude-case-name filters. An invalid case that is filtered out does not block the run — so a shared eval can hold both a quick smoke subset and heavier cases without the latter blocking a filtered smoke run. Use skilled validate to validate the whole suite (every case) regardless of filters.

Existing workspace state. --workspace deliberately reuses the same directory for every selected case, retry, and iteration. Runs are serialized, but configured setup/fixtures/skill installation and agent changes carry forward. The report --output-dir and --event-log must be outside this directory. Select one case and one iteration, with retries disabled, when each result must start from exactly the same state.

Examples

bash

# Run all cases
skilled run ./evals/eval.yaml

# Run a subset
skilled run ./evals/eval.yaml --include-case-name "basic-*"

# Evaluate an agent in an already prepared local workspace
skilled run ./evals/eval.yaml --workspace /path/to/project --parallelism 1 --no-delete

# Exclude cases
skilled run ./evals/eval.yaml --exclude-case-name "*-old" --exclude-case-name "*-deprecated"

# Override engine and model
skilled run ./evals/eval.yaml --engine codex --model openai/gpt-4

# Preserve an opaque upstream model ID when its prefix is not a configured provider
skilled run ./evals/eval.yaml --engine claude_code --model anthropic_modelscope/deepseek-v4-pro

# Preserve a slashed model ID unconditionally with a configured provider
export GATEWAY_API_KEY=your-key
export GATEWAY_BASE_URL=https://gateway.example.com/openai/v1
skilled run ./evals/eval.yaml --engine codex --provider gateway --model team/model-v2

# Temporarily override case parallelism
skilled run ./evals/eval.yaml --parallelism 4

# Run with baseline comparison after configuring environment.image and container credentials
skilled run ./evals/eval.yaml --runtime docker --baseline

# Multiple report formats
skilled run ./evals/eval.yaml --format json --format html --format junit

# Three samples in this command, one folder per run, plus a simple flaky summary
skilled run ./evals/eval.yaml --iteration 3

# Produce a live event stream that a CI process can tail while the run is active
skilled run ./evals/eval.yaml \
  --event-log ./events.jsonl \
  --event-attribute com.example.ci.build_id=build-123

# Auto-detect mode (consumes Anthropic evals.json directly)
skilled run --auto
skilled run --auto --engine codex
skilled run ./my-skill/ --auto

--event-log creates or truncates one file for the invocation. It cannot be used with --dry-run, and the file must not be placed inside a scheduled iteration-N/ output directory because that directory is recreated during the run. Consumers should read only newline-terminated records; the final record of a gracefully completed stream carries last_event: true.

Exit codes

Exit code	Meaning
`0`	All cases passed
`1`	At least one case failed/errored

Use the exit code in CI to determine whether the evaluation succeeded.

OTLP trace export

Telemetry is disabled by default. After standard OpenTelemetry env vars are set, skilled run exports run traces over OTLP and decorates verbose slog logs with trace_id / span_id:

bash

export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
export OTEL_EXPORTER_OTLP_PROTOCOL=grpc
export OTEL_METRICS_EXPORTER=otlp
export OTEL_RESOURCE_ATTRIBUTES=deployment.environment=local,service.namespace=skilled
skilled run ./evals/eval.yaml -v

You can also use trace-specific overrides like OTEL_EXPORTER_OTLP_TRACES_ENDPOINT and OTEL_EXPORTER_OTLP_TRACES_PROTOCOL. Currently grpc and http/protobuf are supported.

OTEL_METRICS_EXPORTER=otlp enables low-cardinality metrics (counters and durations for run/case/runtime exec). OTEL_METRICS_EXPORTER=console is also available for local debugging. OTEL_RESOURCE_ATTRIBUTES is forwarded as resource attributes. Set OTEL_METRICS_EXPORTER=none to explicitly disable metrics.

skilled validate

Validate the eval config files. Run this before run to catch issues early. Unlike run (which validates only the selected cases), validate always checks the whole suite — the eval config and every referenced case file — ignoring any case-name filters.

bash

skilled validate [path to eval.yaml]

Examples

bash

skilled validate
skilled validate ./evals/eval.yaml
skilled validate --skill ./my-skill

On success:

text

✓ eval.yaml is valid (loaded 3 case(s))

The validator checks that:

eval.yaml and every referenced case file exist and parse correctly
All required fields are present
Field values are within the allowed range

--skill <directory> selects source validation instead of eval validation; do not also supply an eval path. It requires a regular SKILL.md no larger than 4 MiB, nonempty name and description metadata, and valid declared local file references. See source and installed-file integrity.

skilled list-cases

List every case referenced by an eval config — handy for quickly inspecting your suite.

bash

skilled list-cases [path to eval.yaml]

Examples

bash

skilled list-cases
skilled list-cases ./evals/eval.yaml

Sample output:

text

ID                    Title                           Tag              Prompt
basic-success         Agent should find null bug      functional_test  Review the current diff and report ...
edge-case-empty       Handle empty input gracefully   functional_test  Review an empty repository with no ...
regression-001        Fix: no longer misreports       functional_test  Review the payment processor code ...

skilled report

Regenerate reports from an existing result file without re-running the evaluation.

bash

skilled report <path to result.json> [flags]

Flags

Flag	Default	Description
`--format`	`json`	Report format: `json` / `junit` / `html` / `markdown` (repeatable)
`--output-dir`	Same dir as `result.json`	Output directory; created if missing

Examples

bash

# Generate an HTML report from existing results
skilled report result.json --format html

# Multiple formats at once
skilled report result.json --format json --format junit --format html --format markdown

# Generate a Markdown summary for CI or a PR comment
skilled report result.json --format markdown

# Pin the output directory
skilled report result.json --format html --output-dir ./reports

Report formats

Format	File	Use case
`json`	`report.json`	Machine-readable structured data; consumable by downstream tools
`junit`	`report.xml`	JUnit XML; parseable by CI systems (Jenkins, GitHub Actions, …)
`html`	`report.html`	Human-readable visualization; open in a browser
`markdown`	`report.md`	Concise Markdown summary for CI logs and GitHub PR comments

markdown is intentionally limited to skilled report as an offline conversion format for an existing result.json. It is not currently accepted by skilled run --format, skilled debug report --format, skilled debug judge --report, or eval.yaml report.formats.

skilled init

Write a skilled user-config file. See User Configuration for the discovery chain and schema.

bash

skilled init [flags]

Flags

Flag	Default	Description
`--local`	`false`	Write target is `$PWD/.skilled.yaml` instead of the XDG path
`--config`	—	Source file to read (validated, copied verbatim with comments). Without this, a commented template is written.
`--print`	`false`	Print to stdout instead of writing to disk
`--force`	`false`	Overwrite an existing target

Note: for init, --config names the source to read. For every other subcommand (run, validate, …) it is the load-path override sitting at the top of the discovery chain.

Examples

bash

# Write a commented template
skilled init                                  # -> ~/.config/skilled/config.yaml
skilled init --local                          # -> ./.skilled.yaml
skilled init --print                          # -> stdout

# Seed from an existing config (validates, preserves comments)
skilled init --config ./team-config.yaml          # -> ~/.config/skilled/config.yaml
skilled init --config ./team-config.yaml --local  # -> ./.skilled.yaml
skilled init --config ./team-config.yaml --print  # validate & dump to stdout

# Overwrite
skilled init --local --force

skilled import

One-shot conversion of an Anthropic evals.json into skilled's native YAML format.

bash

skilled import <path to evals.json> [flags]

Flags

Flag	Default	Description
`--output`	Same dir as `evals.json`	Output directory

Examples

bash

# Convert in place
skilled import ./evals/evals.json

# Custom output directory
skilled import ./evals/evals.json --output ./new-evals

The import produces:

eval.yaml — entrypoint config (with sensible defaults; review before running)
cases/*.yaml — one case file per evals.json entry

import vs --auto: import is a one-time format conversion — afterwards you maintain YAML files. run --auto consumes evals.json at runtime without producing intermediate files. See Migrating from Anthropic.

Output layout

After a run, the output directory looks like:

text

<skill-name>-workspace/
  iteration-1/                    # First iteration
    benchmark.json                # Aggregated stats
    <case-id>/
      with_skill/                 # Run with the Skill installed
        outputs/                  # Files generated by the Agent
          workspace/              # collect_artifacts matches (relative paths preserved)
        grading.json              # Grading result
      without_skill/              # Baseline (only when benchmark.enabled=true)
        outputs/
          workspace/
        grading.json

The outputs/workspace/ subtree appears only when collect_artifacts is configured; see Writing evals → Collecting workspace artifacts.

result.json configuration identity

The iteration-level result.json keeps engine_name and the requested-value model_name semantics for compatibility and also records credential-free objects:

requested_configuration: the engine, provider namespace, model, and version selected after YAML/CLI/credential precedence;
applied_configuration: the adapter protocol and model skilled forwarded to the CLI. An empty applied model means model selection was delegated to local/default settings. Its provider is also empty when the adapter did not actively select one;
observed_configuration: present only when every runner session explicitly reported the same model. If absent, the runtime model is unknown. Per-case observed_model retains an individual agent-reported value when available.

Applied configuration is not proof of the CLI's final choice: local settings may override command-line or environment values. skilled does not make an extra model request merely to inspect local login state. When a Codex custom provider cannot be applied, its provider-scoped endpoint and credential are omitted from the fallback invocation rather than being sent to Codex's local/default provider.

Capability warnings and requested/applied values are also attached to each agent SessionResult; its legacy model field is reserved for an explicitly agent-reported observation. API keys and other credential values are never written to these fields.

grading.json

Per-case grading result (Anthropic-compatible):

json

{
  "expectations": [
    {
      "text": "Output contains the keyword `null`",
      "passed": true,
      "evidence": "final_message contains 'null pointer at line 42'"
    }
  ],
  "summary": {
    "passed": 1,
    "failed": 0,
    "total": 1,
    "pass_rate": 1.0
  }
}

Note: the grading.json under the workspace uses the Anthropic-compatible shape — only expectations and summary at the top level. The full evaluation status (status, turns_executed, turns_total, assertion_results) lives under case_results[].grading inside result.json.

The grading object inside result.json carries the full status:

PASS — all assertions passed
FAIL — at least one assertion failed
ERROR — execution exception (timeout, engine crash, etc.)

benchmark.json

Aggregated statistics across all cases:

json

{
  "run_summary": {
    "with_skill": {
      "pass_rate": { "mean": 0.83 },
      "time_seconds": { "mean": 45.0 },
      "tokens": { "mean": 3800 }
    },
    "without_skill": null,
    "delta": null
  }
}

CLI Reference ​

skilled setup ​

skilled mcp ​

skilled compare ​

skilled run ​

Argument ​

Flags ​

Examples ​

Exit codes ​

OTLP trace export ​

skilled validate ​

Examples ​

skilled list-cases ​

Examples ​

skilled report ​

Flags ​

Examples ​

Report formats ​

skilled init ​

Flags ​

Examples ​

skilled import ​

Flags ​

Examples ​

Output layout ​

result.json configuration identity ​

grading.json ​

benchmark.json ​

CLI Reference

skilled setup

skilled mcp

skilled compare

skilled run

Argument

Flags

Examples

Exit codes

OTLP trace export

skilled validate

Examples

skilled list-cases

Examples

skilled report

Flags

Examples

Report formats

skilled init

Flags

Examples

skilled import

Flags

Examples

Output layout

result.json configuration identity

grading.json

benchmark.json