The Open-source Framework for Dataset Validation

Data validation for scientists, engineers, and analysts seeking correctness.

Pandera is a Union.ai open source project that provides a flexible and expressive API for performing data validation on dataframe-like objects. The goal of Pandera is to make data processing pipelines more readable and robust with statistically typed dataframes.

Dataframes contain information that pandera explicitly validates at runtime. This is useful in production-critical data pipelines or reproducible research settings. With pandera, you can:

1. Define a schema once and use it to validate different dataframe types including pandas, polars, dask, modin, ibis, and pyspark. Use the Narwhals-powered backend to keep validation fully lazy when possible.

2. Check the types and properties of columns in a pd.DataFrame or values in a pd.Series.

3. Perform more complex statistical validation like hypothesis testing.

4. Parse data to standardize the preprocessing steps needed to produce valid data.

5. Seamlessly integrate with existing data analysis/processing pipelines via function decorators.

6. Define dataframe models with the class-based API with pydantic-style syntax and validate dataframes using the typing syntax.

7. Synthesize data from schema objects for property-based testing with pandas data structures.

8. Lazily Validate dataframes so that all validation rules are executed before raising an error.

9. Integrate with a rich ecosystem of python tools like pydantic, fastapi and mypy.

Install

Pandera supports multiple dataframe libraries, including pandas, polars, pyspark, and ibis.

Most of the documentation will use the pandas DataFrames, install Pandera with the pandas extra:

With pip:

pip install 'pandera[pandas]'

With uv:

uv pip install 'pandera[pandas]'

With conda:

conda install -c conda-forge pandera-pandas

Extras

Installing additional functionality:

pip

pip install 'pandera[hypotheses]'   # hypothesis checks
pip install 'pandera[io]'           # yaml/script schema io utilities
pip install 'pandera[frictionless]' # frictionless schema serialization/deserialization
pip install 'pandera[strategies]'   # data synthesis strategies
pip install 'pandera[mypy]'         # enable static type-linting of pandas
pip install 'pandera[fastapi]'      # fastapi integration
pip install 'pandera[dask]'         # validate dask dataframes
pip install 'pandera[pyspark]'      # validate pyspark dataframes
pip install 'pandera[modin]'        # validate modin dataframes
pip install 'pandera[modin-ray]'    # validate modin dataframes with ray
pip install 'pandera[modin-dask]'   # validate modin dataframes with dask
pip install 'pandera[geopandas]'    # validate geopandas geodataframes
pip install 'pandera[polars]'       # validate polars dataframes
pip install 'pandera[ibis]'         # validate ibis tables
pip install 'pandera[xarray]'       # validate xarray data structures
pip install 'pandera[narwhals]'     # use the Narwhals-powered backend

conda

conda install -c conda-forge pandera-hypotheses   # hypothesis checks
conda install -c conda-forge pandera-io           # yaml/script schema io utilities
conda install -c conda-forge pandera-frictionless # frictionless schema serialization/deserialization
conda install -c conda-forge pandera-strategies   # data synthesis strategies
conda install -c conda-forge pandera-mypy         # enable static type-linting of pandas
conda install -c conda-forge pandera-fastapi      # fastapi integration
conda install -c conda-forge pandera-dask         # validate dask dataframes
conda install -c conda-forge pandera-pyspark      # validate pyspark dataframes
conda install -c conda-forge pandera-modin        # validate modin dataframes
conda install -c conda-forge pandera-modin-ray    # validate modin dataframes with ray
conda install -c conda-forge pandera-modin-dask   # validate modin dataframes with dask
conda install -c conda-forge pandera-geopandas    # validate geopandas geodataframes
conda install -c conda-forge pandera-polars       # validate polars dataframes
conda install -c conda-forge pandera-ibis         # validate ibis tables
conda install -c conda-forge pandera-narwhals     # use the Narwhals-powered backend

Quick Start

import pandas as pd
import pandera.pandas as pa

# data to validate
df = pd.DataFrame({
    "column1": [1, 2, 3],
    "column2": [1.1, 1.2, 1.3],
    "column3": ["a", "b", "c"],
})

schema = pa.DataFrameSchema({
    "column1": pa.Column(int, pa.Check.ge(0)),
    "column2": pa.Column(float, pa.Check.lt(10)),
    "column3": pa.Column(
        str,
        [
            pa.Check.isin([*"abc"]),
            pa.Check(lambda series: series.str.len() == 1),
        ]
    ),
}
)

validated_df = schema.validate(df)
print(validated_df)

   column1  column2 column3
0        1      1.1       a
1        2      1.2       b
2        3      1.3       c

Dataframe Model

pandera also provides a class-based API for writing schemas inspired by dataclasses and pydantic. The equivalent DataFrameModel for the above DataFrameSchema would be:

# define a schema
class Schema(pa.DataFrameModel):
    column1: int = pa.Field(ge=0)
    column2: float = pa.Field(lt=10)
    column3: str = pa.Field(isin=[*"abc"])

    @pa.check("column3")
    def custom_check(cls, series: pd.Series) -> pd.Series:
        return series.str.len() == 1

Schema.validate(df)

	column1	column2	column3
0	1	1.1	a
1	2	1.2	b
2	3	1.3	c

Warning

Pandera v0.24.0 introduces the pandera.pandas module, which is now the (highly) recommended way of defining DataFrameSchemas and DataFrameModels for pandas data structures like DataFrames. Defining a dataframe schema from the top-level pandera module will produce a FutureWarning:

import pandera as pa

schema = pa.DataFrameSchema({"col": pa.Column(str)})

Update your import to:

import pandera.pandas as pa

And all of the rest of your pandera code should work. Using the top-level pandera module to access DataFrameSchema and the other pandera classes or functions will be deprecated in version 0.29.0

Informative Errors

If the dataframe does not pass validation checks, pandera provides useful error messages. An error argument can also be supplied to Check for custom error messages.

In the case that a validation Check is violated:

simple_schema = pa.DataFrameSchema({
    "column1": pa.Column(
        int,
        pa.Check(
            lambda x: 0 <= x <= 10,
            element_wise=True,
            error="range checker [0, 10]"
        )
    )
})

# validation rule violated
fail_check_df = pd.DataFrame({
    "column1": [-20, 5, 10, 30],
})

try:
    simple_schema(fail_check_df)
except pa.errors.SchemaError as exc:
    print(exc)

Column 'column1' failed element-wise validator number 0: <Check <lambda>: range checker [0, 10]> failure cases: -20, 30

And in the case of a mis-specified column name:

# column name mis-specified
wrong_column_df = pd.DataFrame({
    "foo": ["bar"] * 10,
    "baz": [1] * 10
})


try:
    simple_schema(wrong_column_df)
except pa.errors.SchemaError as exc:
    print(exc)

column 'column1' not in dataframe. Columns in dataframe: ['foo', 'baz']

Error Reports

If the dataframe is validated lazily with lazy=True, errors will be aggregated into an error report. The error report groups DATA and SCHEMA errors to to give an overview of error sources within a dataframe. Take the following schema and dataframe:

schema = pa.DataFrameSchema(
    {"id": pa.Column(int, pa.Check.lt(10))},
    name="MySchema",
    strict=True,
)

df = pd.DataFrame({"id": [1, None, 30], "extra_column": [1, 2, 3]})

try:
    schema.validate(df, lazy=True)
except pa.errors.SchemaErrors as exc:
    print(exc)

{
    "SCHEMA": {
        "COLUMN_NOT_IN_SCHEMA": [
            {
                "schema": "MySchema",
                "column": "MySchema",
                "check": "column_in_schema",
                "error": "column 'extra_column' not in DataFrameSchema {'id': <Schema Column(name=id, type=DataType(int64))>}"
            }
        ],
        "SERIES_CONTAINS_NULLS": [
            {
                "schema": "MySchema",
                "column": "id",
                "check": "not_nullable",
                "error": "non-nullable series 'id' contains null values:1   NaNName: id, dtype: float64"
            }
        ],
        "WRONG_DATATYPE": [
            {
                "schema": "MySchema",
                "column": "id",
                "check": "dtype('int64')",
                "error": "expected series 'id' to have type int64, got float64"
            }
        ]
    },
    "DATA": {
        "DATAFRAME_CHECK": [
            {
                "schema": "MySchema",
                "column": "id",
                "check": "less_than(10)",
                "error": "Column 'id' failed element-wise validator number 0: less_than(10) failure cases: 30.0"
            }
        ]
    }
}

Validating the above dataframe will result in data level errors, namely the id column having a value which fails a check, as well as schema level errors, such as the extra column and the None value.

This error report can be useful for debugging, with each item in the various lists corresponding to a SchemaError

Supported Features by DataFrame Backend

Currently, pandera provides four validation backends: pandas, pyspark, polars, and ibis. The table below shows which of pandera’s features are available for the supported dataframe libraries:

feature	pandas	pyspark	polars	ibis
DataFrameSchema validation	✅	✅	✅	✅
DataFrameModel validation	✅	✅	✅	✅
SeriesSchema validation	✅	🚫	❌	❌
Index/MultiIndex validation	✅	🚫	🚫	🚫
Built-in and custom Checks	✅	✅	✅	✅
Groupby checks	✅	❌	❌	❌
Custom check registration	✅	✅	❌	❌
Hypothesis testing	✅	❌	❌	❌
Built-in and custom DataTypes	✅	✅	✅	✅
Preprocessing with Parsers	✅	❌	❌	❌
Data synthesis strategies	✅	❌	❌	❌
Validation decorators	✅	✅	✅	✅
Lazy validation	✅	✅	✅	✅
Dropping invalid rows	✅	❌	✅	❌
Pandera configuration	✅	✅	✅	✅
Schema Inference	✅	❌	❌	❌
Schema persistence	✅	❌	❌	❌
Data Format Conversion	✅	❌	❌	❌
Pydantic type support	✅	❌	❌	❌
FastAPI support	✅	❌	❌	❌

Legend

• ✅: Supported

• ❌: Not supported

• 🚫: Not applicable

Note

The dask, modin, geopandas, and pyspark.pandas support in pandera all leverage the pandas validation backend.

Important

new in 0.32.0; Pandera ships an optional Narwhals-powered backend that keeps validation fully lazy when possible. Install the extra for each backend you use and enable it with either of the following:

pip install 'pandera[narwhals,polars]'    # Polars
pip install 'pandera[narwhals,ibis]'      # Ibis
pip install 'pandera[narwhals,pyspark]'   # PySpark SQL

export PANDERA_USE_NARWHALS_BACKEND=True

import pandera

pandera.set_config(use_narwhals_backend=True)

You can call set_config() before or after importing a pandera backend module. See Backend registration for details on lazy registration and runtime re-registration.

Contributing

All contributions, bug reports, bug fixes, documentation improvements, enhancements and ideas are welcome.

A detailed overview on how to contribute can be found in the contributing guide on GitHub.

Issues

Submit issues, feature requests or bugfixes on github.

Need Help?

There are many ways of getting help with your questions. You can ask a question on Github Discussions page or reach out to the maintainers and pandera community on Slack

How to Cite

If you use pandera in the context of academic or industry research, please consider citing the paper and/or software package.

Paper

@InProceedings{ niels_bantilan-proc-scipy-2020,
  author    = { {N}iels {B}antilan },
  title     = { pandera: {S}tatistical {D}ata {V}alidation of {P}andas {D}ataframes },
  booktitle = { {P}roceedings of the 19th {P}ython in {S}cience {C}onference },
  pages     = { 116 - 124 },
  year      = { 2020 },
  editor    = { {M}eghann {A}garwal and {C}hris {C}alloway and {D}illon {N}iederhut and {D}avid {S}hupe },
  doi       = { 10.25080/Majora-342d178e-010 }
}

Software Package

License and Credits

pandera is licensed under the MIT license and is written and maintained by Niels Bantilan (niels@union.ai)

Indices and tables

• Index