Commands
semtest run
Run semantic tests and generate reports.
semtest run [files...] [options]
Arguments
| Argument | Description |
|---|---|
[files...] |
Optional test files or directories. Runs all discovered tests if omitted. |
You can pass individual spec files, directories, or a mix:
# Run all tests
semtest run
# Run a single file
semtest run semtests/auth.spec.md
# Run a directory
semtest run semtests/api/
# Run multiple targets
semtest run semtests/auth.spec.md semtests/api/
Options
| Flag | Description | Default |
|---|---|---|
--tag <tags> |
Comma-separated tag filter | off |
-t, --testNamePattern <pattern> |
Regex filter on test name or file path | off |
--repeat <n> |
Run each test N times | 1 |
--bail |
Stop after first failure | off |
--maxfail <n> |
Stop after N failures | off |
--verbose |
Show detailed per-test output | off |
--timestamp |
Generate a timestamped copy of the Markdown report | off |
--include-passing |
Include passing tests in the Markdown report | off |
--strict |
Exit code 2 if validation issues are found | off |
--skip-validation |
Skip post-run validation checks | off |
--debug |
Log raw LLM output to a debug directory | off |
--timeout <ms> |
Per-test timeout in milliseconds | 0 (none) |
--junit |
Generate a JUnit XML report | off |
--skip-permissions-if-possible |
Skip tool permission prompts where supported | off |
Empty LLM responses are automatically retried up to 3 times per test before being marked as an error. Only empty responses trigger retries — parse failures and non-zero exit codes do not.
CLI flags override config file values. See Configuration for the full list of config properties.
Filtering examples
Filter by tags defined in spec frontmatter:
# Run only tests tagged "ci"
semtest run --tag ci
# Run tests tagged "security" or "auth"
semtest run --tag security,auth
Filter by test name or file path using a regex:
# Run tests with "auth" in the name or path
semtest run -t auth
# Run tests matching a pattern
semtest run -t "api.*validation"
Exit codes
| Code | Meaning |
|---|---|
0 |
All tests passed |
1 |
One or more tests failed |
2 |
Error: LLM failure, invalid config, bad CLI arguments, or validation issues with --strict |
semtest list
List all available model keys.
semtest list [options]
Options
| Flag | Description |
|---|---|
--json |
Output as JSON |
By default, model keys are printed grouped by tool with the underlying model name. Use --json for machine-readable output.
# Human-readable grouped output
semtest list
# JSON output
semtest list --json
semtest init
Scaffold a new semtest setup in the current directory.
semtest init
Creates:
| File | Description |
|---|---|
semtest.config.ts |
Configuration file with sensible defaults |
This command is idempotent. If the config file already exists, it's skipped.
semtest uninstall
Print uninstall instructions for your installation method.
semtest uninstall
Output:
To uninstall semtest, run the command for your installation method:
npm: npm uninstall -g @westopp/semtest
Homebrew: brew uninstall semtest