Skip to content
DashboardWebsite

Testing

This page explains how to run OpenTremor’s Python and end-to-end test suites, how core’s API integration tests work, and how to replay webhooks against a local stack.

From the workspace root, make test runs core’s suite with all packs installed and a throwaway MongoDB. See Development Setup for other component test commands.

To run core’s suite directly, start MongoDB and point TEST_MONGODB_URI at it. The tests also accept MONGODB_URI, then default to mongodb://localhost:27017.

Terminal window
docker network create ot-test
docker run -d --rm --name ot-test-mongo --network ot-test mongo:7.0
docker run --rm --network ot-test -v "$PWD":/ws:ro -w /ws/OpenTremor-core \
-e TEST_MONGODB_URI=mongodb://ot-test-mongo:27017 -e UV_PROJECT_ENVIRONMENT=/tmp/venv \
ghcr.io/astral-sh/uv:0.12-python3.14-trixie-slim \
sh -c 'uv sync -q --frozen --group internal && uv run --no-sync pytest -q -p no:cacheprovider'
docker rm -f ot-test-mongo && docker network rm ot-test

Pass a test file or -k expression after pytest to narrow the run. For coverage, add --with pytest-cov to uv run and --cov=opentremor_core to pytest.

Core tests live in src/tests/. These files are useful starting points:

src/tests/
├── conftest.py Shared fixtures and database cleanup
├── mongo.py Test MongoDB connection and storage helper
├── basic-plan.txt Terraform plan fixture
├── fixtures/ Webhook payloads and other test data
├── test_api_integration.py API integration tests using ASGI transport
├── test_storage.py Storage tests against MongoDB
├── test_config.py Configuration tests
├── test_formatter.py Report formatting tests
├── test_hash.py Hash normalization tests
├── test_diff.py Diff reader tests
├── test_llm.py LLM provider tests
├── test_analysis_runner.py Analysis runner tests
├── test_github_client.py GitHub client tests
├── test_gitlab_integration.py GitLab integration tests
├── test_repositories_api.py Repository API tests
├── test_metrics.py Metrics tests
└── test_integration_roles.py Integration role tests

Core has 90 test files. Platform authentication tests live in OpenTremor-platform; each pack’s parser tests live in that pack’s repository.

Run API integration tests without a server

Section titled “Run API integration tests without a server”

test_api_integration.py uses httpx with ASGI transport. Its FastAPI app fixture has module scope, so the tests do not open a port or start a server process.

@pytest.fixture(scope="module")
def app():
return create_app(config_file=_CONFIG)
@pytest.fixture
async def client(app):
async with httpx.AsyncClient(
transport=httpx.ASGITransport(app=app), base_url="http://test/v1"
) as c:
yield c

The tests still need MongoDB. They start the job worker in the client fixture because ASGI transport does not run the app lifespan.

GitHub and GitLab send webhooks from outside your network. You can test the receivers without exposing your stack by using scripts/webhook-replay in OpenTremor-core. It posts a stored payload and signs or tokens it as the code host would.

Terminal window
WEBHOOK_SECRET=<the App's webhook secret> scripts/webhook-replay \
github pull_request src/tests/fixtures/webhooks/github/pull_request.opened.json

The script runs in a container with host networking. Its --base-url defaults to http://localhost:8001. Sanitized payloads for supported events are in src/tests/fixtures/webhooks/. To test network access rules, send the request through HAProxy and set the caller address with --from, for example:

Terminal window
scripts/webhook-replay --base-url http://localhost:8080 --from 198.51.100.9 \
github pull_request src/tests/fixtures/webhooks/github/pull_request.opened.json

OpenTremor-e2e runs journeys through the API, the MCP service, the dashboard and Terraform against the release images on a full stack: MongoDB, core with every pack, the dashboard, the docs, the MCP service, the task scheduler and HAProxy. Fakes stand in for the LLM, GitHub, GitLab, Jira, an OIDC identity provider and S3. A sink receives notifications and the audit log over HTTPS and syslog with TLS. Mailpit receives the email. Chromium drives the dashboard through HAProxy, and the Terraform provider is built from the workspace.

The suite covers every feature that runs on the stack alone. Features that need a real third party are out of scope: the Slack install callback, social sign-in with Google or Microsoft, SAML sign-in, and SSO domain verification by DNS.

OpenTremor-e2e/TESTS.md lists every test: what it checks, the endpoints it calls and the pages it opens. Each run writes the same catalogue to artifacts/tests.md and writes artifacts/coverage.md with the endpoints no test called.

Terminal window
cd OpenTremor-e2e
make e2e # build release images, then run the suite
make e2e PYTEST_ADDOPTS="-k dashboard"

The stack runs in containers without host ports. Results are saved in OpenTremor-e2e/artifacts/. The release workflow runs the suite against the images before pushing them; a failed run stops the release. To run it from CI, open Actions → e2e → Run workflow in OpenTremor-e2e.

LLM providers — Tests use fake clients that return canned analysis and token usage. They patch get_llm_client where the code imports it: opentremor_core.controllers.services.llm.connections.get_llm_client.

GitHub API — test_github_client.py injects httpx.MockTransport into GitHubClient. Webhook tests in test_api_integration.py patch the GitHubClient class because they test router and job behavior, not HTTP calls.

MongoDB — Tests use MongoDB rather than mocking storage behavior. Core uses the opentremor_core_tests database; platform uses opentremor_platform_tests. Core’s conftest.py sets up indexes once and empties the test database after each test. The pytest configuration uses one event loop for the session because the async MongoDB client stays bound to the loop where it first runs.

async def test_create_and_read(self):
s = mongo_storage() # tests/mongo.py — a MongoBackend on the test database
await s.create_resource(ORG, "h1", _resource("h1"))
doc = await s.read_resource(ORG, "h1")
assert doc["hash"] == "h1"
assert doc["analysis"] is None

test_plan_parser.py in OpenTremor-pack-terraform-plan

Section titled “test_plan_parser.py in OpenTremor-pack-terraform-plan”
def test_tricky_non_header_line_not_matched():
# Lines ending with "will be destroyed" but without the " # " prefix
# must NOT be treated as resource block headers.
units = parse_plan(TRICKY_LINE)
assert len(units) == 1

Core’s pyproject.toml sets these pytest options:

pyproject.toml
[tool.pytest.ini_options]
asyncio_mode = "auto"
asyncio_default_fixture_loop_scope = "session"
asyncio_default_test_loop_scope = "session"
testpaths = ["src"]
  • Place core tests in src/tests/test_<module>.py.
  • Use async def for tests that access storage or make HTTP calls.
  • Use mongo_storage() from tests/mongo.py for storage tests; do not mock storage behavior.
  • To create state the storage API cannot produce, write directly to the collection, such as storage._jobs.update_one(...), instead of adding a test-only method.
  • Give each test a fresh uuid4() changeset or use the shared changeset fixture to avoid cross-test state.
  • If a test needs a fresh analysis in the module-scoped app, tag its input content with a unique value as described above.