Migrating from Anthropic

If you used Anthropic's skill-creator to build your Skill, you already have an evals/evals.json file. skilled consumes it directly — no rewrite required.

---

Two onboarding paths

Path 1 — consume evals.json directly (fastest)

--auto mode auto-detects evals/evals.json and runs it inline, without producing any intermediate files:

# Inside your Skill directory
cd my-skill/
skilled run --auto

# Or with an explicit directory
skilled run ./my-skill/ --auto

# Run the same suite against a different engine
skilled run --auto --engine codex

When to use it:

• Quickly wire your Skill into CI for regression testing
• Validate the same suite against Codex or other engines
• Stay in sync with the Anthropic workflow — updates to evals.json are picked up automatically on the next run

Path 2 — convert to native YAML (deep customization)

The import command transforms evals.json into skilled's YAML format:

skilled import ./evals/evals.json

After conversion you get:

evals/
  eval.yaml                # Entrypoint config (review and adjust)
  cases/
    case-1.yaml            # One file per evals.json entry
    case-2.yaml
    case-3.yaml

Once in native form you can:

• Add expect gating checks (deterministic verification)
• Replace pure LLM grading with rule_based
• Configure MCP tool-call assertions

When to use it:

• You need skilled–only capabilities (structured assertions, MCP tool-call assertions, …)
• You want to fine-tune grading logic
• You no longer need to stay in lock-step with evals.json

---

Comparison

	--auto mode	import conversion
Operation	Zero config; consumed at runtime	One-time conversion; YAML thereafter
Sync	Updates to evals.json apply automatically	Independent maintenance after conversion
Customization	Limited to what evals.json already expresses	Full freedom
Default judge	agent_judge (because expectations are NL)	Switch to rule_based or script freely
Typical user	Fast onboarding, CI regression	Long-term maintenance, deep grading

The two paths are complementary. Start with --auto to validate quickly, then import the cases that need deep customization.

---

evals.json format

Anthropic's evals.json looks like:

{
  "skill_name": "my-skill",
  "evals": [
    {
      "id": 1,
      "prompt": "Help me create a release plan",
      "expected_output": "Should call the create_plan tool",
      "files": [],
      "expectations": [
        "Calls create_plan correctly",
        "The arguments include the plan name"
      ]
    }
  ]
}

Conversion mapping:

evals.json field	skilled equivalent
prompt	input.prompt
expectations	judge.criteria (agent_judge rubric)
expected_output	The case description
files	context.files

---

Recommended migration path

1. Get CI green with --auto
   skilled run --auto

2. Once that is stable, import the cases that need customization
   skilled import ./evals/evals.json --output ./evals-v2

3. Edit the converted YAML — add expect gating and rule_based rules

4. Run the native config
   skilled run ./evals-v2/eval.yaml

The recommended user journey: build and iterate the Skill with Anthropic skill-creator → onboard CI with skilled run --auto → use skilled import to switch to native YAML when deeper customization is needed.

Configuration and compatibility

After importing your evaluation cases, use skilled to validate and run them. The bundled skilled-improve workflow helps your agent inspect failures and revise a Skill. Project configuration belongs in .skilled.yaml; see user configuration for configuration and credential locations.

Import converts supported case fields; it does not establish that a configured engine is installed or authenticated. Validate the imported suite, inspect the selected cases with a dry run, and perform a real evaluation before using the result as a baseline. See verification and release scope for tested behavior and remaining limits.

Source and releases are available in DeadHead04/skillEd, with public documentation. The installation guide covers release installers and source builds. An npm registry package is not published; provider-backed evaluation requires your configured engine and account.