Semtest

Exit codes

Semtest uses exit codes to signal results to CI systems:

Code	Meaning
`0`	All tests passed
`1`	One or more tests failed
`2`	Error — LLM subprocess failure, invalid config, or validation issues in strict mode

Use these in pipeline conditionals to gate deployments or flag PRs.

Output formats for CI

Every semtest run produces two files by default, with a third available on opt-in:

JSON (`ci-results.json`)

Always generated. Machine-readable summary designed for scripts and custom tooling.

{
  "status": "fail",
  "summary": {
    "total": 5,
    "passed": 3,
    "failed": 1,
    "errored": 1
  },
  "tests": [
    {
      "id": "session-expiry",
      "sourceFile": "auth.spec.md",
      "status": "fail",
      "group": "auth",
      "location": "src/auth/session.ts"
    },
    {
      "id": "rate-limiting",
      "sourceFile": "api-routes.spec.md",
      "status": "pass",
      "group": "api"
    },
    {
      "id": "broken-test",
      "sourceFile": "broken.spec.md",
      "status": "error",
      "error": "LLM subprocess timed out after 120s"
    }
  ]
}

Fields like errored, invalid, and skipped in summary are omitted when zero. The location field appears only on failures, and error only on errors. The group field appears when a test is in a subdirectory.

JUnit XML (`junit-results.xml`)

Opt-in via --junit. Compatible with most CI test reporters — GitHub Actions, GitLab, Jenkins, CircleCI.

semtest run --junit

Markdown (`latest.md`)

Always generated. Human-readable report. Can be posted as a PR comment or uploaded as a build artifact.

Example: GitHub Actions

name: Semantic Tests
on: [push, pull_request]

jobs:
  semtest:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: 20

      - name: Install semtest
        run: npm install -g @westopp/semtest

      - name: Run semantic tests
        run: semtest run --junit --strict --skip-permissions-if-possible

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: semtest-results
          path: semtest-results/

Example: GitLab CI

semtest:
  image: node:20
  script:
    - npm install -g @westopp/semtest
    - semtest run --junit --strict --skip-permissions-if-possible
  artifacts:
    when: always
    paths:
      - semtest-results/
    reports:
      junit: semtest-results/junit-results.xml

Skipping permission prompts

LLM CLIs often prompt for user confirmation before executing actions. In CI, these prompts block execution indefinitely. Use --skip-permissions-if-possible to auto-append each tool's bypass flag:

semtest run --skip-permissions-if-possible --strict --junit

This is strongly recommended for all CI runs. See Permission bypass for details on which tools support it.

Timeouts

In CI environments, set a timeout to prevent indefinite hangs if the LLM is slow or unresponsive:

semtest run --timeout 120000

This sets a 2-minute timeout per test. Timed-out tests receive error status and the run continues with remaining tests.

Strict mode

Use --strict in CI to catch validation issues — duplicate test IDs across files or scenarios the LLM marks as invalid:

semtest run --strict

Without --strict, validation issues appear in the report but don't affect the exit code. With --strict, they cause exit code 2, failing the pipeline.

CI integration