Writing tests

File naming

By default, semtest discovers files ending in .spec.md or .test.md:

  • auth.spec.md
  • checkout.test.md
  • api/routes.spec.md

Files that don't match these patterns are ignored:

  • auth-test.md — wrong suffix
  • README.md — not a spec file
  • .hidden.spec.md — dotfiles are excluded
  • api.spec.yml — not .md (requires custom testMatch, see Controlling test discovery)

Anatomy of a spec file

A spec file describes one or more expectations about your codebase in plain language. Markdown is the recommended format because headings give clear boundaries between scenarios.

# Authentication

## Session Handling

User sessions should expire after 30 minutes of inactivity.
The session store should be configured in src/config/.

## Password Hashing

Passwords must be hashed using bcrypt before storage.
Plain text passwords should never be written to the database.

The LLM reads this file, identifies two scenarios ("Session Handling" and "Password Hashing"), then examines your codebase to evaluate each one independently.

Single vs multi-scenario files

A spec file can contain one scenario or many. The LLM identifies distinct scenarios from structural markers:

  • Markdown headings (##, ###)
  • Numbered lists
  • Frontmatter fields
  • Any clear structural boundary

Single scenario:

# License File

The repository root should contain a LICENSE or LICENSE.md file.

Multiple scenarios:

# API Design

## RESTful Routes

All API endpoints should follow RESTful naming conventions.

## Error Responses

API errors should return JSON with a "message" field and an appropriate HTTP status code.

## Rate Limiting

Public endpoints should have rate limiting configured.

Each scenario gets its own pass/fail verdict in the report.

Directory structure and grouping

Subdirectories under the test root become group labels in reports. Use them to organize specs by domain:

semtests/
├── auth/
│   ├── login.spec.md
│   └── session.spec.md
├── api/
│   ├── routes.spec.md
│   └── validation.spec.md
└── infra/
    └── config.spec.md

In the Markdown report, tests are grouped under their directory name (auth, api, infra).

Tips for effective specs

Be specific about what, not how. Describe the expected behavior or property, not the implementation details.

# Good
API error responses should include a "message" field with a human-readable description.

# Too vague
Error handling should be good.

# Too prescriptive
The catch block on line 45 of src/api/handler.ts should call formatError().

Reference concrete paths when helpful. If you're testing something about a specific module, name it:

The database connection pool in src/db/pool.ts should have a maximum of 20 connections.

One concern per scenario. Don't bundle unrelated expectations. Split them so each gets a clear pass/fail verdict.

Know what the LLM can and can't verify. The LLM reads your code — it doesn't execute it. Specs about static properties (structure, naming, configuration, presence of logic) work well. Specs that require runtime behavior ("the API returns a 200 on valid input") cannot be verified this way.

Frontmatter overrides

Spec files support YAML frontmatter to override global config on a per-test basis. Supported fields:

  • tags — comma-separated or array of tags for filtering
  • timeout — per-test timeout in milliseconds
  • llm — model key to use for this test (run semtest list to see options)
  • skipPermissionsIfPossible — override the global permission-bypass setting for this test
---
skipPermissionsIfPossible: true
tags: ci
---

# CI Pipeline Check

The CI configuration should run all test suites.

Controlling test discovery

Test discovery is controlled by two config properties:

  • testMatch — Glob patterns for files to include (default: ["**/*.spec.md", "**/*.test.md"])
  • testPathIgnorePatterns — Paths to exclude (default: ["node_modules", "dist", ".git", "vendor"])

When you pass a specific file that doesn't follow the .spec.md / .test.md convention (e.g. semtest run notes.txt), semtest will run it but print a warning:

Warning: "notes.txt" does not follow the .spec.md or .test.md naming convention

This is informational only — the file is still executed.

See Configuration for details.