Testing
This page explains how to run OpenTremor’s Python and end-to-end test suites, how core’s API integration tests work, and how to replay webhooks against a local stack.
Run the Python test suite
Section titled “Run the Python test suite”From the workspace root, make test runs core’s suite with all packs installed and a throwaway MongoDB. See Development Setup for other component test commands.
To run core’s suite directly, start MongoDB and point TEST_MONGODB_URI at it. The tests also accept MONGODB_URI, then default to mongodb://localhost:27017.
docker network create ot-testdocker run -d --rm --name ot-test-mongo --network ot-test mongo:7.0
docker run --rm --network ot-test -v "$PWD":/ws:ro -w /ws/OpenTremor-core \ -e TEST_MONGODB_URI=mongodb://ot-test-mongo:27017 -e UV_PROJECT_ENVIRONMENT=/tmp/venv \ ghcr.io/astral-sh/uv:0.12-python3.14-trixie-slim \ sh -c 'uv sync -q --frozen --group internal && uv run --no-sync pytest -q -p no:cacheprovider'
docker rm -f ot-test-mongo && docker network rm ot-testPass a test file or -k expression after pytest to narrow the run. For coverage, add --with pytest-cov to uv run and --cov=opentremor_core to pytest.
Find tests in core
Section titled “Find tests in core”Core tests live in src/tests/. These files are useful starting points:
src/tests/├── conftest.py Shared fixtures and database cleanup├── mongo.py Test MongoDB connection and storage helper├── basic-plan.txt Terraform plan fixture├── fixtures/ Webhook payloads and other test data├── test_api_integration.py API integration tests using ASGI transport├── test_storage.py Storage tests against MongoDB├── test_config.py Configuration tests├── test_formatter.py Report formatting tests├── test_hash.py Hash normalization tests├── test_diff.py Diff reader tests├── test_llm.py LLM provider tests├── test_analysis_runner.py Analysis runner tests├── test_github_client.py GitHub client tests├── test_gitlab_integration.py GitLab integration tests├── test_repositories_api.py Repository API tests├── test_metrics.py Metrics tests└── test_integration_roles.py Integration role testsCore has 90 test files. Platform authentication tests live in OpenTremor-platform; each pack’s parser tests live in that pack’s repository.
Run API integration tests without a server
Section titled “Run API integration tests without a server”test_api_integration.py uses httpx with ASGI transport. Its FastAPI app fixture has module scope, so the tests do not open a port or start a server process.
@pytest.fixture(scope="module")def app(): return create_app(config_file=_CONFIG)
@pytest.fixtureasync def client(app): async with httpx.AsyncClient( transport=httpx.ASGITransport(app=app), base_url="http://test/v1" ) as c: yield cThe tests still need MongoDB. They start the job worker in the client fixture because ASGI transport does not run the app lifespan.
Replay webhooks against a local stack
Section titled “Replay webhooks against a local stack”GitHub and GitLab send webhooks from outside your network. You can test the receivers without exposing your stack by using scripts/webhook-replay in OpenTremor-core. It posts a stored payload and signs or tokens it as the code host would.
WEBHOOK_SECRET=<the App's webhook secret> scripts/webhook-replay \ github pull_request src/tests/fixtures/webhooks/github/pull_request.opened.jsonThe script runs in a container with host networking. Its --base-url defaults to http://localhost:8001. Sanitized payloads for supported events are in src/tests/fixtures/webhooks/. To test network access rules, send the request through HAProxy and set the caller address with --from, for example:
scripts/webhook-replay --base-url http://localhost:8080 --from 198.51.100.9 \ github pull_request src/tests/fixtures/webhooks/github/pull_request.opened.jsonRun the end-to-end suite
Section titled “Run the end-to-end suite”OpenTremor-e2e runs journeys through the API, the MCP service, the dashboard and Terraform
against the release images on a full stack: MongoDB, core with every pack, the dashboard, the
docs, the MCP service, the task scheduler and HAProxy. Fakes stand in for the LLM, GitHub,
GitLab, Jira, an OIDC identity provider and S3. A sink receives notifications and the audit log
over HTTPS and syslog with TLS. Mailpit receives the email. Chromium drives the dashboard through
HAProxy, and the Terraform provider is built from the workspace.
The suite covers every feature that runs on the stack alone. Features that need a real third party are out of scope: the Slack install callback, social sign-in with Google or Microsoft, SAML sign-in, and SSO domain verification by DNS.
OpenTremor-e2e/TESTS.md lists every test: what it checks, the endpoints it calls and the pages
it opens. Each run writes the same catalogue to artifacts/tests.md and writes
artifacts/coverage.md with the endpoints no test called.
cd OpenTremor-e2emake e2e # build release images, then run the suitemake e2e PYTEST_ADDOPTS="-k dashboard"The stack runs in containers without host ports. Results are saved in OpenTremor-e2e/artifacts/. The release workflow runs the suite against the images before pushing them; a failed run stops the release. To run it from CI, open Actions → e2e → Run workflow in OpenTremor-e2e.
Mock external services
Section titled “Mock external services”LLM providers — Tests use fake clients that return canned analysis and token usage. They patch get_llm_client where the code imports it: opentremor_core.controllers.services.llm.connections.get_llm_client.
GitHub API — test_github_client.py injects httpx.MockTransport into GitHubClient. Webhook tests in test_api_integration.py patch the GitHubClient class because they test router and job behavior, not HTTP calls.
MongoDB — Tests use MongoDB rather than mocking storage behavior. Core uses the opentremor_core_tests database; platform uses opentremor_platform_tests. Core’s conftest.py sets up indexes once and empties the test database after each test. The pytest configuration uses one event loop for the session because the async MongoDB client stays bound to the loop where it first runs.
Example unit tests
Section titled “Example unit tests”test_storage.py
Section titled “test_storage.py”async def test_create_and_read(self): s = mongo_storage() # tests/mongo.py — a MongoBackend on the test database await s.create_resource(ORG, "h1", _resource("h1")) doc = await s.read_resource(ORG, "h1") assert doc["hash"] == "h1" assert doc["analysis"] is Nonetest_plan_parser.py in OpenTremor-pack-terraform-plan
Section titled “test_plan_parser.py in OpenTremor-pack-terraform-plan”def test_tricky_non_header_line_not_matched(): # Lines ending with "will be destroyed" but without the " # " prefix # must NOT be treated as resource block headers. units = parse_plan(TRICKY_LINE) assert len(units) == 1Pytest configuration
Section titled “Pytest configuration”Core’s pyproject.toml sets these pytest options:
[tool.pytest.ini_options]asyncio_mode = "auto"asyncio_default_fixture_loop_scope = "session"asyncio_default_test_loop_scope = "session"testpaths = ["src"]Test conventions
Section titled “Test conventions”- Place core tests in
src/tests/test_<module>.py. - Use
async deffor tests that access storage or make HTTP calls. - Use
mongo_storage()fromtests/mongo.pyfor storage tests; do not mock storage behavior. - To create state the storage API cannot produce, write directly to the collection, such as
storage._jobs.update_one(...), instead of adding a test-only method. - Give each test a fresh
uuid4()changeset or use the sharedchangesetfixture to avoid cross-test state. - If a test needs a fresh analysis in the module-scoped app, tag its input content with a unique value as described above.