Python data pipeline development Patterns for building production-quality data processing pipelines with Python. Targeted at Python 3.11+ for and exception groups; Python 3.12+ for the lighter syntax. Pin a 3.13+ runtime if you want the JIT or experimental free-threading; the patterns here don't depend on either. Choosing a DataFrame engine: pandas vs polars vs DuckDB For a long time pandas was the default for any tabular work in Python. As of 2026 the default has shifted: polars is the right pick for multi-GB pipelines on a single machine, DuckDB is the right pick when SQL or larger-than-RAM…