Commands

Complete CLI reference for semtest.

`semtest run`

Run semantic tests and generate reports.

bash

semtest run [files...] [options]

Arguments

Argument	Description
`[files...]`	Optional test files or directories. Runs all discovered tests if omitted.

You can pass individual spec files, directories, or a mix:

bash

# Run all tests
semtest run

# Run a single file
semtest run semtests/auth.spec.md

# Run a directory
semtest run semtests/api/

# Run multiple targets
semtest run semtests/auth.spec.md semtests/api/

Options

Flag	Description	Default
`--tag <tags>`	Comma-separated tag filter	off
`-t, --testNamePattern <pattern>`	Regex filter on test name or file path	off
`--repeat <n>`	Run each test N times	`1`
`--bail`	Stop after first failure	off
`--maxfail <n>`	Stop after N failures	off
`--verbose`	Show detailed per-test output	off
`--timestamp`	Generate a timestamped copy of the Markdown report	off
`--include-passing`	Include passing tests in the Markdown report	off
`--strict`	Exit code 2 if validation issues are found	off
`--skip-validation`	Skip post-run validation checks	off
`--debug`	Log raw LLM output to a debug directory	off
`--timeout <ms>`	Per-test timeout in milliseconds	`0` (none)
`--junit`	Generate a JUnit XML report	off
`--skip-permissions-if-possible`	Skip tool permission prompts where supported	off

Empty LLM responses are automatically retried up to 3 times per test before being marked as an error. Only empty responses trigger retries — parse failures and non-zero exit codes do not.

CLI flags override config file values. See Configuration for the full list of config properties.

Filtering examples

Filter by tags defined in spec frontmatter:

bash

# Run only tests tagged "ci"
semtest run --tag ci

# Run tests tagged "security" or "auth"
semtest run --tag security,auth

Filter by test name or file path using a regex:

bash

# Run tests with "auth" in the name or path
semtest run -t auth

# Run tests matching a pattern
semtest run -t "api.*validation"

Exit codes

Code	Meaning
`0`	All tests passed
`1`	One or more tests failed
`2`	Error: LLM failure, invalid config, bad CLI arguments, or validation issues with `--strict`

`semtest list`

List all available model keys.

bash

semtest list [options]

Options

Flag	Description
`--json`	Output as JSON

By default, model keys are printed grouped by tool with the underlying model name. Use --json for machine-readable output.

bash

# Human-readable grouped output
semtest list

# JSON output
semtest list --json

`semtest init`

Scaffold a new semtest setup in the current directory.

bash

semtest init

Creates:

File	Description
`semtest.config.ts`	Configuration file with sensible defaults

This command is idempotent. If the config file already exists, it's skipped.

`semtest uninstall`

Print uninstall instructions for your installation method.

bash

semtest uninstall

Output:

text

To uninstall semtest, run the command for your installation method:

npm:      npm uninstall -g @westopp/semtest
Homebrew: brew uninstall semtest

Previous← llms.txt NextConfiguration →