Quickstart#

Installation#

Requirements: Python 3.10 or later.

The recommended installation uses uv:

git clone https://github.com/parulgupta1004/fair-seldonian.git
cd fair-seldonian
uv sync

To include optional dependencies for visualization (matplotlib):

uv sync --extra plots

Alternatively, with pip:

pip install -e .
pip install -e ".[plots]"

Dependencies#

Package	Purpose	Required
NumPy	Array operations and linear algebra	Yes
pandas	Tabular data handling	Yes
PyTorch	Tensor operations and automatic differentiation	Yes
scikit-learn	Baseline logistic regression model	Yes
SciPy	Statistical functions and numerical optimization	Yes
matplotlib	Visualization of experiment results	Optional

Running Experiments#

Experiments are executed via the command line. The seldonian_type argument selects the algorithm variant (see Algorithm Variants for details):

uv run python scripts/run_paper_experiments.py --out exp/paper --only <mode>

where <mode> is one of base, mod, bound, const, opt or affine. Omit --only to run every variant.

Each variant writes summary.csv and its four result panels to exp/paper/<mode>/; the run, the aggregation and the plots all happen in that one command. Use --trials to set the repetitions per dataset size (default 40) and --jobs to run trials in parallel.

The generated plots show three metrics as a function of training set size:

Log loss — primary objective performance.
Probability of solution — fraction of trials where a solution was found.
Probability of constraint violation — fraction of trials where $g(\theta) > 0$ on test data (should remain below $\delta$ ).

Library Usage#

The framework can also be used programmatically:

"""Minimal end-to-end example: train a fairness-certified classifier.

Runs the Quasi-Seldonian Algorithm (QSA) on synthetic data and reads back the
high-confidence fairness guarantee. This is the script version of the first part
of ``examples/quickstart.ipynb``.

Run it with::

    uv run python examples/quickstart.py
    # or, once the package is installed:  python examples/quickstart.py
"""

from __future__ import annotations

import numpy as np
import torch

from fair_seldonian.algorithms import QSA
from fair_seldonian.data import data_split, get_data
from fair_seldonian.models import eval_ghat, predict


def accuracy(
    theta: torch.Tensor, theta1: torch.Tensor, X: np.ndarray, Y: np.ndarray
) -> float:
    """Fraction of correct predictions at a 0.5 decision threshold."""
    probs = predict(theta, theta1, X).detach().numpy()
    return float(((probs >= 0.5).astype(int) == Y).mean())


def main() -> None:
    # 1. Generate synthetic data. Each row has feature columns (the last of which
    #    is the sensitive attribute T), a binary label Y, and the group label T.
    #    Here both groups have the same base rate (tp0 == tp1), so the data is fair.
    data = get_data(
        N=20000,
        features=5,
        t_ratio=0.5,
        tp0_ratio=0.5,
        tp1_ratio=0.5,
        random_seed=7,
    )

    # 2. Split into test/train arrays: (X_test, Y_test, T_test, X_train, ...).
    X_te, Y_te, T_te, X_tr, Y_tr, T_tr = data_split(
        frac=0.6, all_data=data, random_state=1, m_test=0.3
    )
    print(f"train examples: {X_tr.shape[0]}, test examples: {X_te.shape[0]}")

    # 3. Train with QSA. The result carries the model parameters, a boolean that
    #    is True only for a model certified to satisfy the fairness constraint
    #    with probability >= 1 - delta, and diagnostics explaining the outcome.
    result = QSA(X_tr, Y_tr, T_tr, "opt", None, None)
    theta, theta1 = result.theta, result.theta1

    if not result.passed_safety:
        print("\nNo Solution Found - QSA could not certify a fair model on this data.")
        # failure_mode distinguishes the two reasons, which call for different
        # remedies: "candidate_infeasible" means no feasible model was found at
        # all, "safety_test_rejected" means one was found but the safety data
        # could not certify it.
        print(f"  reason: {result.diagnostics.failure_mode}")
        print("Try more data, a smaller fairness gap, or a larger delta.")
        return

    # 4. The guarantee: eval_ghat returns a high-confidence upper bound on the
    #    constraint function g. A value <= 0 means the constraint is satisfied.
    upper_bound = float(eval_ghat(theta, theta1, X_te, Y_te, T_te, "opt"))
    print("\nModel certified fair.")
    print(f"  fairness upper bound g(theta) <= {upper_bound:.4f}  (<= 0 is satisfied)")
    print(f"  test accuracy: {accuracy(theta, theta1, X_te, Y_te):.3f}")

Configuration#

The algorithm is configured via the SeldonianConfig dataclass. All parameters have sensible defaults, so no configuration is required for basic usage.

Parameter	Default	Description
`delta`	0.05	Significance level $\delta$ ; the constraint holds with probability $\geq 1 - \delta$
`inequality`	Hoeffding	Concentration inequality used for bound computation (`Inequality`)
`constraint`	See below	Fairness constraint in reverse Polish notation
`candidate_ratio`	0.40	Fraction of training data allocated to the candidate set

The default constraint string TP(1) TP(0) - abs 0.25 TP(1) * - encodes a relaxed equalized opportunity condition (see Introduction for details).

Example: custom configuration

    strict = SeldonianConfig(
        delta=0.01,
        inequality=Inequality.T_TEST,
        candidate_ratio=0.5,
        constraint="TP(1) TP(0) - abs 0.1 -",
    )

Extending the Framework#

To use a custom model, replace the following functions in fair_seldonian.models.logistic_regression:

predict() — returns $P(Y=1 \mid X, \theta)$ as a tensor.
simple_logistic() — trains the base model and returns initial parameter values.
f_hat() — computes the primary objective function.

The constraint expression (constraint field on SeldonianConfig) can be set to any fairness condition expressible over the base variables, of which there are three kinds:

Cells — TP(g), FP(g), TN(g), FN(g): joint probabilities within group g, so TP(g) is $P(\hat{Y}=1, Y=1 \mid G=g)$ and the four sum to 1.
Label-conditioned rates — TPR(g), FPR(g), TNR(g), FNR(g): additionally conditioned on the true label, so TPR(g) is $P(\hat{Y}=1 \mid Y=1, G=g)$ , averaged over only that group’s positive rows.
Predicted rates — PR(g), NR(g): the label is irrelevant, so PR(g) is $P(\hat{Y}=1 \mid G=g)$ , equal to TP(g) + FP(g).

The distinction matters: a constraint over cells and one over rates express different fairness definitions and are certified from different row counts. See Fairness constraints for the built-in builders.