CI/CD Integration

Run ShipTested automatically in your CI/CD pipeline to generate and validate tests on every push or pull request. This guide covers GitHub Actions setup, CI mode flags, and best practices.

GitHub Actions#

The fastest way to integrate ShipTested into your pipeline is with the official GitHub Action.

Basic setup#

name: ShipTested
on:
  pull_request:
    branches: [main]

jobs:
  generate-tests:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - uses: shiptested/action@v1
        with:
          api-key: ${{ secrets.SHIPTESTED_API_KEY }}

This runs ShipTested on every pull request targeting main. The action handles installation, authentication, and test generation automatically.

CI Mode Flags#

When running ShipTested in CI, use these flags to control behavior:

Manual CLI usage in CI#

If you prefer not to use the GitHub Action, you can call the CLI directly:

npx shiptested generate --ci --changed-only --threshold 80

Example Workflows#

PR checks with changed files only#

Generate tests only for files modified in the pull request. This keeps CI fast by skipping unchanged files.

name: Test Generation (PR)
on:
  pull_request:
    branches: [main, develop]

jobs:
  shiptested:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Generate tests for changed files
        run: npx shiptested generate --ci --changed-only --diff-base origin/main --threshold 80
        env:
          SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

Set fetch-depth: 0 in the checkout step so git history is available for diff comparison.

Scheduled full project run#

Run ShipTested on a schedule to keep your entire test suite up to date.

name: Test Generation (Scheduled)
on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6am UTC

jobs:
  shiptested:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Generate tests for all files
        run: npx shiptested generate --ci --all
        env:
          SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

PR check with JSON output#

Output results as JSON for further processing, such as posting a summary comment on the pull request.

- name: Generate tests
  id: shiptested
  run: |
    npx shiptested generate --ci --changed-only --json > results.json
  env:
    SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

- name: Process results
  run: cat results.json | jq '.summary'

Exit Codes and Threshold Behavior#

The generate command uses exit codes to signal success or failure in CI:

The --threshold flag controls what "success" means. For example:

• --threshold 100 (default) - every generated test must pass
• --threshold 80 - at least 80% of generated tests must pass
• --threshold 0 - always exit 0, even if tests fail (useful for advisory runs)

Set a lower threshold during initial adoption to avoid blocking PRs, then increase it as your test suite matures.

Using API Keys in CI#

In CI environments, OAuth is not available. Use an API key for authentication instead.

Creating an API key#

1. Open the Billing page in the dashboard
2. Scroll to the API Keys section
3. Click Create API Key and give it a name (e.g. "GitHub Actions")
4. Copy the key immediately. It will not be shown again.

Adding the key to GitHub#

1. Go to your repository on GitHub
2. Navigate to Settings > Secrets and variables > Actions
3. Click New repository secret
4. Name it SHIPTESTED_API_KEY and paste your key

The key is then available in workflows as ${{ secrets.SHIPTESTED_API_KEY }}. You can pass it as an environment variable:

env:
  SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

Or use it directly with the --api-key flag (not recommended, as it exposes the key in logs):

npx shiptested generate --ci --api-key $SHIPTESTED_API_KEY

Always use environment variables or secrets for API keys. Never hardcode them in workflow files.

MCP Server#

ShipTested also provides an MCP (Model Context Protocol) server for integration with AI-powered development tools. This allows compatible editors and assistants to access ShipTested capabilities directly.

npx shiptested-mcp

The MCP server exposes ShipTested's analysis and generation features through the standard MCP protocol. Add it to your editor's MCP configuration to enable AI-assisted test generation from within your development environment.

Next Steps#

• CLI Reference - full list of commands and flags
• Configuration - customize test generation settings
• Dashboard Guide - explore your results in the web dashboard