semtest/Guides/CI integration
Guides

CI integration

Running semtest in continuous integration pipelines.

Exit codes

Semtest uses exit codes to signal results to CI systems:

CodeMeaning
0All tests passed
1One or more tests failed
2Error — LLM subprocess failure, invalid config, or validation issues in strict mode

Use these in pipeline conditionals to gate deployments or flag PRs.

Output formats for CI

Every semtest run produces two files by default, with a third available on opt-in:

JSON (ci-results.json)

Always generated. Machine-readable summary designed for scripts and custom tooling.

JSON
{
"status": "fail",
"summary": {
  "total": 5,
  "passed": 3,
  "failed": 1,
  "errored": 1
},
"tests": [
  {
    "id": "session-expiry",
    "sourceFile": "auth.spec.md",
    "status": "fail",
    "group": "auth",
    "location": "src/auth/session.ts"
  },
  {
    "id": "rate-limiting",
    "sourceFile": "api-routes.spec.md",
    "status": "pass",
    "group": "api"
  },
  {
    "id": "broken-test",
    "sourceFile": "broken.spec.md",
    "status": "error",
    "error": "LLM subprocess timed out after 120s"
  }
]
}

Fields like errored, invalid, and skipped in summary are omitted when zero. The location field appears only on failures, and error only on errors. The group field appears when a test is in a subdirectory.

JUnit XML (junit-results.xml)

Opt-in via --junit. Compatible with most CI test reporters — GitHub Actions, GitLab, Jenkins, CircleCI.

bash
semtest run --junit

Markdown (latest.md)

Always generated. Human-readable report. Can be posted as a PR comment or uploaded as a build artifact.

Example: GitHub Actions

YAML.github/workflows/semtest.yml
name: Semantic Tests
on: [push, pull_request]

jobs:
semtest:
  runs-on: ubuntu-latest
  steps:
    - uses: actions/checkout@v4

    - uses: actions/setup-node@v4
      with:
        node-version: 20

    - name: Install semtest
      run: npm install -g @westopp/semtest

    - name: Run semantic tests
      run: semtest run --junit --strict --skip-permissions-if-possible

    - name: Upload test results
      if: always()
      uses: actions/upload-artifact@v4
      with:
        name: semtest-results
        path: semtest-results/

Example: GitLab CI

YAML.gitlab-ci.yml
semtest:
image: node:20
script:
  - npm install -g @westopp/semtest
  - semtest run --junit --strict --skip-permissions-if-possible
artifacts:
  when: always
  paths:
    - semtest-results/
  reports:
    junit: semtest-results/junit-results.xml

Skipping permission prompts

LLM CLIs often prompt for user confirmation before executing actions. In CI, these prompts block execution indefinitely. Use --skip-permissions-if-possible to auto-append each tool's bypass flag:

bash
semtest run --skip-permissions-if-possible --strict --junit

This is strongly recommended for all CI runs. See Permission bypass for details on which tools support it.

Timeouts

In CI environments, set a timeout to prevent indefinite hangs if the LLM is slow or unresponsive:

bash
semtest run --timeout 120000

This sets a 2-minute timeout per test. Timed-out tests receive error status and the run continues with remaining tests.

Strict mode

Use --strict in CI to catch validation issues — duplicate test IDs across files or scenarios the LLM marks as invalid:

bash
semtest run --strict

Without --strict, validation issues appear in the report but don't affect the exit code. With --strict, they cause exit code 2, failing the pipeline.

Previous← Under the hoodNextllms.txt →