Mocking HTTP

When a pipeline test calls a real API, you have silently introduced five dependencies into your test suite: network availability, API uptime, your rate limit quota, credential validity, and the API's current data. Any of those can cause a test to fail for reasons entirely unrelated to your code. The tests also run ten to a hundred times slower than they need to.

The solution is to intercept HTTP at the transport layer and return a pre-programmed response — fast, free, deterministic, and with no side effects.

Why not just mock the function?

You could replace requests.get with a MagicMock. That works, but it gives you no guarantees about the URL, headers, or request body. The responses library intercepts at a lower level: it registers URL patterns and returns fake responses only when those exact URLs are called, raising ConnectionError for any URL you forgot to register. This means tests fail loudly when your code calls an unexpected endpoint.

The three approaches

1. responses library (unit tests)

responses monkeypatches the urllib3 transport adapter for the duration of the test. Activate it with the @responses.activate decorator or the responses.RequestsMock context manager:

import responses, requests

@responses.activate
def test_fetch_report():
    responses.add(
        responses.GET,
        "https://api.example.com/report",
        json={"rows": 3, "status": "ok"},
        status=200,
    )
    result = fetch_report("https://api.example.com/report")
    assert result["rows"] == 3
    assert len(responses.calls) == 1  # exactly one request was made

If fetch_report calls any URL not registered with responses.add, the test raises ConnectionError. This prevents accidental live calls from slipping through silently.

2. VCR / cassettes (integration smoke tests)

VCR (Video Cassette Recorder) libraries — vcrpy is the most popular — record real HTTP interactions to YAML "cassette" files on first run, then replay them on subsequent runs. This lets you write tests against the real API once and run them offline thereafter.

The tradeoff: cassettes go stale when the API changes, and the YAML files can be large and noisy in version control. VCR is most useful for integration tests that you run occasionally against a staging environment, not for unit tests that run on every commit.

3. Local mock server (end-to-end tests)

For full end-to-end tests, spin up a local HTTP server — httpretty, pytest-httpserver, or a minimal http.server subprocess — that implements a minimal version of the upstream API. This is the highest-fidelity approach and the most expensive to maintain.

What responses cannot do

responses only intercepts calls made through the requests library. If your code uses urllib.request, httpx, or aiohttp, you need a different interceptor (pytest-httpx for httpx, aioresponses for aiohttp). Check which HTTP library your pipeline uses before choosing a mocking strategy.

Where to go next

Next: mocking HTTP in practice — a runnable example using responses to mock a successful GET, assert the URL was called, and test the error-handling path with a simulated 500.