Prefect concepts
Prefect wraps your Python functions with flow and task decorators, adding retries, observability, parametrisation, and scheduling without rewriting your pipeline logic.
- Explain what Prefect adds over cron and Make — retries, observability, parametrisation, a UI
- Identify the flow/task model and how Prefect determines execution order
- Understand when Prefect is the right tool and when simpler alternatives suffice
Make resolves dependency graphs and runs steps incrementally. Cron schedules scripts on a timer. Both are useful, and both show their limits as pipelines grow in complexity: no visibility into run history, no per-step retry policies, no way to pass parameters to a run, and no alerting beyond a log file on disk.
Prefect addresses all of these without requiring you to rewrite your pipeline in a new language or framework. It is a decorator-based layer over ordinary Python.
The flow/task model
Prefect has two core primitives:
@task decorates a function that does a discrete unit of work — fetching data,
transforming a dataframe, writing a file. Tasks can be retried independently,
cached, and run in parallel.
@flow decorates the orchestrator function that calls tasks. A flow run is the
unit of observability: it has a name, a state (Running / Completed / Failed), a
start time, a duration, and a log of every task it ran.
from prefect import flow, task
@task(retries=3, retry_delay_seconds=10)
def fetch_data(url: str) -> list:
response = requests.get(url)
response.raise_for_status()
return response.json()
@task
def transform(records: list) -> list:
return [r for r in records if r["active"]]
@flow(name="daily-report")
def pipeline(url: str = "https://api.example.com/data"):
raw = fetch_data(url)
cleaned = transform(raw)
return cleanedCalling pipeline() runs the flow locally and prints task states to the terminal.
No infrastructure required for local development.
What Prefect adds over cron + Make
| Feature | cron | Make | Prefect |
|---|---|---|---|
| Scheduling | Yes | No | Yes |
| Dependency graph | No | Yes | Yes |
| Per-task retries | No | No | Yes |
| Parametrised runs | Fragile | Fragile | First-class |
| Run history / UI | No | No | Yes |
| Alerting on failure | Manual | Manual | Built-in |
| Parallel tasks | No | With -j | Yes |
The UI (Prefect Cloud or self-hosted) shows every run, its duration, which tasks failed, the logs from each task, and the input parameters. This is the observability gap that cron and Make cannot fill.
Prefect vs Airflow
Airflow is the older, more widely deployed alternative. Its DAGs are defined in
Python but in a more verbose, graph-construction style. Prefect's task graph is
inferred from data flow between tasks — if transform receives the output of
fetch_data, Prefect knows fetch_data must run first, without an explicit
set_downstream call.
Airflow requires a database, a scheduler process, and worker infrastructure even for local development. Prefect runs locally with no infrastructure until you need the UI or cloud scheduling.
For Python-native teams starting fresh, Prefect is the pragmatic choice. Teams with existing Airflow infrastructure and years of DAGs should not migrate lightly.
Prefect 2 (the current major version) is significantly simpler than Prefect 1.
If you encounter documentation, tutorials, or Stack Overflow answers that mention
prefect.engine, DaskExecutor, or LocalExecutor by name, they are for
Prefect 1 and do not apply.
Where to go next
Next: Prefect in practice — a runnable example defining two tasks and a flow, running locally, and observing task states.
Makefiles for pipelines
Make is the oldest widely-used DAG executor — its target/dependency syntax maps directly onto data pipeline DAGs, with automatic incremental rebuilds and parallel execution built in.
Prefect in practice
Define a two-task Prefect flow, run it locally, and observe task states — all in ordinary Python without any external infrastructure.