`pippen.data.schemas`¶

Validates the shape of every table at the point of ingest, before a bad file reaches any calculation.

A column that silently changes dtype between two seasons will not stop the solver from printing an answer. It stops the answer from being right, and nothing else in the pipeline would say so.

pippen.data.schemas ¶

pandera schemas for every table this project ingests.

A downstream regression cannot tell a malformed download from a bad season. If a column silently changes dtype between two seasons' Parquet files, or a required identifier goes missing for a handful of rows, the RAPM solver or the reliability measurement will still run and will still print an answer. The answer will be wrong and nothing will say so. These schemas exist to fail loudly at the point of ingest, before a bad file reaches any calculation.

Known limitation, load-bearing for the rest of the pipeline The play_by_play table validated here is ESPN-sourced, via hoopR. It records event participants only (athlete_id_1/2/3) and carries no on-court lineup column. RAPM needs the full five-man lineup for both teams on every possession, so it cannot be computed from this table, no matter how clean a copy of it passes validation. That requires pbpstats running against NBA API data instead. This module checks the shape of the data; it makes no claim about what can be computed from it.

Schema source and confidence Every schema below has now been checked against a real hoopR file. Doing so corrected two of them. player_box has no percentage column at all, only made and attempted, and schedules names its teams home_id and away_id rather than following the play-by-play naming. Both had been guessed from convention and both guesses were wrong, which is the argument for checking a schema against a sample rather than against documentation.

Each schema declares a core rather than every column. The real files carry
57 to 78 columns; what is declared here is what this project depends on.
``strict=False`` lets the rest through untouched.

Strictness Every schema below sets strict=False. hoopR adds columns to its published tables over time, and a hard failure the day the publisher adds one would break every scheduled refresh for a reason that has nothing to do with data quality. strict=False lets unrecognised columns pass through untouched rather than raising or silently dropping them. Dropping (strict="filter") was considered and rejected: a discarded column leaves no trace, and a future stage that starts to depend on it would fail somewhere downstream instead of here.

Coercion Every schema also sets coerce=True. One season's file read as int32 and another's as int64 for the same column is a real, observed source of silent divergence: a join or a concatenation between the two can produce upcasting, duplication, or a dtype-driven mismatch that never raises. Coercing every declared column to its schema dtype means every season leaves validation in the same dtype, or validation fails and says which column would not coerce.

SchemaValidationError ¶

Bases: ValueError

A table failed schema validation.

Raised by :func:validate with every failing row and check already collected, rather than only the first one, so a human fixes the file once instead of re-running validation after every single fix.

Source code in src/pippen/data/schemas.py

class SchemaValidationError(ValueError):
    """A table failed schema validation.

    Raised by :func:`validate` with every failing row and check already
    collected, rather than only the first one, so a human fixes the file once
    instead of re-running validation after every single fix.
    """

get_schema ¶

get_schema(dataset)

Return the pandera schema for one dataset, by name.

Parameters:

Name	Type	Description	Default
`dataset`	`str`	Dataset name, e.g. `"play_by_play"`. Must be a key of :data:`SCHEMAS`.	required

Returns:

Type	Description
`DataFrameSchema`	The schema registered for `dataset`.

Raises:

Type	Description
`ValueError`	If `dataset` is not a known dataset name.

Source code in src/pippen/data/schemas.py

def get_schema(dataset: str) -> DataFrameSchema:
    """Return the pandera schema for one dataset, by name.

    Args:
        dataset: Dataset name, e.g. ``"play_by_play"``. Must be a key of
            :data:`SCHEMAS`.

    Returns:
        The schema registered for ``dataset``.

    Raises:
        ValueError: If ``dataset`` is not a known dataset name.
    """
    try:
        return SCHEMAS[dataset]
    except KeyError:
        valid = ", ".join(sorted(SCHEMAS))
        raise ValueError(f"unknown dataset {dataset!r}; expected one of: {valid}") from None

validate ¶

validate(frame, dataset)

Validate a table against its schema and return the validated copy.

Validation is lazy: every failing column and check is collected before raising, instead of stopping at the first one. Coercion happens as part of validation, so the returned frame's dtypes match the schema even when the input's did not.

Parameters:

Name	Type	Description	Default
`frame`	`DataFrame`	The table to validate.	required
`dataset`	`str`	Dataset name identifying which schema to check against, e.g. `"play_by_play"`. Must be a key of :data:`SCHEMAS`.	required

Returns:

Type	Description
`DataFrame`	`frame` with schema-declared columns coerced to their schema dtype.
`DataFrame`	Columns not declared in the schema are passed through unchanged.

Raises:

Type	Description
`ValueError`	If `dataset` is not a known dataset name.
`SchemaValidationError`	If `frame` fails one or more checks. The message lists every failure: which column, which check, which value, and which row.

Source code in src/pippen/data/schemas.py

def validate(frame: pd.DataFrame, dataset: str) -> pd.DataFrame:
    """Validate a table against its schema and return the validated copy.

    Validation is lazy: every failing column and check is collected before
    raising, instead of stopping at the first one. Coercion happens as part
    of validation, so the returned frame's dtypes match the schema even when
    the input's did not.

    Args:
        frame: The table to validate.
        dataset: Dataset name identifying which schema to check against, e.g.
            ``"play_by_play"``. Must be a key of :data:`SCHEMAS`.

    Returns:
        ``frame`` with schema-declared columns coerced to their schema dtype.
        Columns not declared in the schema are passed through unchanged.

    Raises:
        ValueError: If ``dataset`` is not a known dataset name.
        SchemaValidationError: If ``frame`` fails one or more checks. The
            message lists every failure: which column, which check, which
            value, and which row.
    """
    schema = get_schema(dataset)
    try:
        validated = schema.validate(frame, lazy=True)
    except pa.errors.SchemaErrors as exc:
        raise SchemaValidationError(_describe_failures(dataset, exc)) from exc
    return validated

pippen.data.schemas¶