Troubleshooting & FAQ

This page covers common issues, debugging tips, and answers to frequently asked questions about ShipTested.

Common Issues#

Authentication failures#

If you see an authentication error when running CLI commands, try the following:

  1. Check that your credentials file exists at ~/.shiptested/credentials
  2. Verify the file is valid JSON and contains a token field
  3. Your token may have expired (tokens last 7 days). Run shiptested login to re-authenticate.
  4. If using an API key, confirm SHIPTESTED_API_KEY is set correctly in your environment
  5. Run shiptested whoami to verify your current authentication status
# Re-authenticate
shiptested login

# Check who you are logged in as
shiptested whoami

# Check if your API key is set
echo $SHIPTESTED_API_KEY

Test timeouts#

ShipTested applies a 60-second timeout per test execution. If your tests are timing out:

  • Check for infinite loops or unresolved promises in your source code
  • Ensure external API calls in your code are properly mocked (the generated tests should mock them, but complex setups may need manual adjustment)
  • Large test suites with many tests may take longer. Consider generating tests for individual files instead of the entire project at once.

The coverage command has a separate 5-minute timeout for running your full test suite with coverage reporters.

Framework detection issues#

If ShipTested detects the wrong framework or test runner:

  • Run shiptested init --force to regenerate your configuration file with fresh detection
  • Override the test framework in your config file:
// shiptested.config.ts
export default {
  testFramework: "vitest", // or "jest"
}
  • You can also override via CLI flag: shiptested generate --framework vitest
  • Or via environment variable: SHIPTESTED_TEST_FRAMEWORK=vitest

Config file not found#

ShipTested looks for configuration in the following order:

  1. shiptested.config.ts
  2. shiptested.config.js
  3. .shiptested.json
  4. "shiptested" key in package.json

Make sure you are running commands from the root of your project (where package.json lives). If no config is found, run shiptested init to create one.

Tests fail after generation#

If the generated tests do not pass after all iterations:

  • Check the iteration logs in the terminal output or dashboard. They show exactly what failed and why at each step.
  • Try increasing the maximum iterations: shiptested generate --max-iterations 5
  • Some files are inherently difficult to test automatically (heavy side effects, complex external dependencies). Consider adding custom instructions in your project settings to guide the AI.
  • If a file consistently fails, you can exclude it via your config:
// shiptested.config.ts
export default {
  exclude: ["src/lib/problematic-file.ts"],
}

Node.js version issues#

ShipTested requires Node.js 18 or later. Check your version:

node --version

If you are using an older version, upgrade via your preferred version manager (nvm, fnm, volta, etc.):

nvm install 18

Frequently Asked Questions#

Can AI really understand my codebase well enough to write useful tests?#

Yes. The CLI collects your source code, dependencies, types, and project config locally, then sends that context to our API where Claude generates tests tailored to your logic. The key difference: tests run on your machine, in your real environment. If they fail, we read the errors, fix the tests, and re-run. The final output is verified, passing test code.

How is this different from asking ChatGPT to write tests?#

ChatGPT generates tests that look right but do not compile or pass. You end up debugging AI-generated test code manually, which often takes longer than writing tests from scratch. ShipTested generates, runs, debugs, and iterates automatically until every test passes. You get working tests, not homework.

Does my code leave my machine?#

Tests run entirely locally. Only file context (source code and dependencies) is sent to our API for AI generation, and it is never stored. Your private npm packages, monorepo setup, and environment stay on your machine. All connections are encrypted. See the Security page for full details.

Is $15/month worth it for a testing tool?#

A single afternoon debugging broken tests costs more than a year of ShipTested. At average developer rates, 2 hours of manual test writing is $100+ in salary. ShipTested does the same work in under 3 minutes for a fraction of the cost. Most users recover their investment in the first session.

How do I know the generated tests are actually good?#

Every test has been executed against your real code in your real environment and passed. We measure statement, branch, and function coverage. Results sync to your dashboard with full iteration logs so you can see exactly what happened. If a test is in your output, it ran and it passed.

What if the AI cannot make the tests pass?#

After max iterations, we deliver the best attempt with a clear log of what failed and why. Most files reach 100% pass rate within 2 to 3 iterations. If we cannot get your tests green, you do not get charged for that file.

What test runners do you support?#

Currently Vitest and Jest for TypeScript/JavaScript projects. Python (pytest), Vue, and Svelte support is coming soon.

How can I use ShipTested?#

Four ways: the CLI (npx shiptested generate), a GitHub Action that runs on pull requests and posts results as PR comments, an MCP server for Cursor and Claude Desktop, and a web dashboard that tracks all your test history and coverage trends.

What about monorepos and private packages?#

Because tests run locally on your machine, monorepos, private npm packages, custom path aliases, and your exact Node or Bun version all just work. No Docker containers trying to approximate your setup.

Debug Tips#

Check your credentials#

# Verify authentication
shiptested whoami

# Check credentials file exists
ls -la ~/.shiptested/credentials

# Check file permissions (should be -rw-------)
stat ~/.shiptested/credentials

Verify Node.js version#

# Must be 18+
node --version

# Check which node binary is being used
which node

Check config loading#

# Run analyze to see what ShipTested detects
shiptested analyze

# Use --json for detailed output
shiptested analyze --json

The analyze command shows detected language, framework, test runner, and the list of testable files. This is useful for verifying that your configuration is being loaded correctly.

Check usage limits#

# See current usage
shiptested usage

# Machine-readable output
shiptested usage --json

Getting Help#

If you are experiencing an issue not covered here:

  • Check the CLI Reference for correct command syntax and available flags
  • Review the Configuration page to ensure your setup is correct
  • Reach out on X (@shiptested) for support
  • Team plan users have access to priority support

Was this page helpful?

Previous
Security and Privacy