CI/CD Integration

Run ShipTested automatically in your CI/CD pipeline to generate and validate tests on every push or pull request. This guide covers how to use the CLI in CI environments, configure API keys, and set up common workflows.

GitHub Action (Coming Soon)#

A dedicated GitHub Action for ShipTested is on the roadmap and will simplify setup for GitHub-based workflows. In the meantime, you can run the CLI directly in any CI environment - see the examples below.

Check the Roadmap for the latest timeline on the GitHub Action release.

Running the CLI in CI#

The recommended approach for CI/CD integration is to call the ShipTested CLI directly in your pipeline scripts. This works with any CI provider, including GitHub Actions, GitLab CI, CircleCI, and others.

npx shiptested@latest generate --ci --changed-only --threshold 80

CI Mode Flags#

When running ShipTested in CI, use these flags to control behavior:

Example Workflows#

GitHub Actions - PR checks with changed files only#

Generate tests only for files modified in the pull request. This keeps CI fast by skipping unchanged files.

name: Test Generation (PR)
on:
  pull_request:
    branches: [main, develop]

jobs:
  shiptested:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Generate tests for changed files
        run: npx shiptested@latest generate --ci --changed-only --diff-base origin/main --threshold 80
        env:
          SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

Set fetch-depth: 0 in the checkout step so git history is available for diff comparison.

Scheduled full project run#

Run ShipTested on a schedule to keep your entire test suite up to date.

name: Test Generation (Scheduled)
on:
  schedule:
    - cron: "0 6 * * 1"  # Every Monday at 6am UTC

jobs:
  shiptested:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: "20"

      - name: Install dependencies
        run: npm ci

      - name: Generate tests for all files
        run: npx shiptested@latest generate --ci --all
        env:
          SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

JSON output for further processing#

Output results as JSON for further processing, such as posting a summary comment on the pull request.

- name: Generate tests
  id: shiptested
  run: |
    npx shiptested@latest generate --ci --changed-only --json > results.json
  env:
    SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

- name: Process results
  run: cat results.json | jq '.summary'

Exit Codes and Threshold Behavior#

The generate command uses exit codes to signal success or failure in CI:

The --threshold flag controls what "success" means. For example:

• --threshold 100 (default) - every generated test must pass
• --threshold 80 - at least 80% of generated tests must pass
• --threshold 0 - always exit 0, even if tests fail (useful for advisory runs)

Set a lower threshold during initial adoption to avoid blocking PRs, then increase it as your test suite matures.

Using API Keys in CI#

In CI environments, OAuth is not available. Use an API key for authentication instead.

Creating an API key#

1. Open the Billing page in the dashboard
2. Scroll to the API Keys section
3. Click Create API Key and give it a name (e.g. "GitHub Actions")
4. Copy the key immediately. It will not be shown again.

Adding the key to your CI provider#

Store your API key as a secret or environment variable in your CI provider. For GitHub Actions:

1. Go to your repository on GitHub
2. Navigate to Settings > Secrets and variables > Actions
3. Click New repository secret
4. Name it SHIPTESTED_API_KEY and paste your key

The key is then available in workflows as ${{ secrets.SHIPTESTED_API_KEY }}. Pass it as an environment variable:

env:
  SHIPTESTED_API_KEY: ${{ secrets.SHIPTESTED_API_KEY }}

Or use it directly with the --api-key flag (not recommended, as it exposes the key in logs):

npx shiptested@latest generate --ci --api-key $SHIPTESTED_API_KEY

Always use environment variables or secrets for API keys. Never hardcode them in workflow files.

Next Steps#

• CLI Reference - full list of commands and flags
• Configuration - customize test generation settings
• Dashboard Guide - explore your results in the web dashboard
• Roadmap - see what integrations are coming next