fair_seldonian.experiments package#

Submodules#

fair_seldonian.experiments.plots module#

fair_seldonian.experiments.plots.plot_all(rows, out_dir)[source]#

Write the four result panels for one experiment variant.

Parameters:
Return type:

None

fair_seldonian.experiments.results module#

fair_seldonian.experiments.results.clopper_pearson(successes, n, alpha=0.05)[source]#

Exact binomial confidence interval.

Probabilities here are estimated from a few dozen trials, where the normal approximation is poor and - at the boundary, where a rate of exactly 0 or 1 is common - gives a zero-width interval. Reporting “violation rate 0.00” from 40 trials as though it established anything about a 0.05 threshold is the specific error this avoids: 0/40 is consistent with a true rate up to about 0.088.

Parameters:
Return type:

tuple[float, float]

fair_seldonian.experiments.results.save_summary(rows, filename)[source]#
Parameters:
Return type:

None

fair_seldonian.experiments.results.summarise(records)[source]#

Collapse per-trial records into one row per dataset size.

Two aggregation choices here are deliberate and easy to get wrong.

Loss and constraint value are averaged only over trials that returned a solution, and the number of such trials is reported alongside. They are conditional means, and the conditioning set changes along the x-axis - at one end almost every trial contributes, at the other almost none. Comparing two variants’ loss curves without knowing how many trials sit behind each point is not meaningful, so n_solutions travels with the number.

The violation rate counts trials where a solution was returned and its true constraint value is positive. A run that returns “no solution found” has not violated anything - it has declined to answer - so it is excluded from the numerator but kept in the denominator, which is the event the guarantee actually bounds.

Parameters:

records (dict[str, ndarray])

Return type:

list[dict[str, float]]

fair_seldonian.experiments.runner module#

class fair_seldonian.experiments.runner.StudySpec(
seldonian_type='base',
ms=(2000,
5000,
10000,
20000,
40000,
80000,
160000),
num_trials=40,
force_start=False,
eval_size=200000,
config=SeldonianConfig(delta=0.05,
inequality=<Inequality.HOEFFDING_INEQUALITY: 2>,
constraint='TP(1) TP(0) - abs 0.25 TP(1) * -',
candidate_ratio=0.4,
optimizer='Powell',
max_iter=10000,
penalty=100.0),
t_ratio=0.5,
tp0_ratio=0.4,
tp1_ratio=0.6,
features=5,
seed=0,
)[source]#

Bases: object

One learning-curve experiment.

Parameters:

  • seldonian_type (str) – bound-propagation variant (base, mod, bound, const, opt).

  • ms (tuple[int, ...]) – dataset sizes to sweep.

  • num_trials (int) – independent repetitions at each size. Each repetition draws a fresh dataset. Generating one dataset and having every “trial” subsample it is cheaper but measures subsampling noise rather than sampling noise, and at the largest size every trial would see identical data.

  • force_start (bool) – warm-start candidate selection from the previous size’s solution. This is sound only because each run has its own independent data. Under a shared-dataset design the warm start would have been fitted on data overlapping the next run’s safety set, breaking the independence the safety test relies on.

  • eval_size (int) – size of the held-out set used to measure the true constraint value and log loss.

  • config (SeldonianConfig)

  • t_ratio (float)

  • tp0_ratio (float)

  • tp1_ratio (float)

  • features (int)

  • seed (int)

seldonian_type: str = 'base'#
ms: tuple[int, ...] = (2000, 5000, 10000, 20000, 40000, 80000, 160000)#
num_trials: int = 40#
force_start: bool = False#
eval_size: int = 200000#
config: SeldonianConfig = SeldonianConfig(delta=0.05, inequality=<Inequality.HOEFFDING_INEQUALITY: 2>, constraint='TP(1) TP(0) - abs 0.25 TP(1) * -', candidate_ratio=0.4, optimizer='Powell', max_iter=10000, penalty=100.0)#
t_ratio: float = 0.5#
tp0_ratio: float = 0.4#
tp1_ratio: float = 0.6#
features: int = 5#
seed: int = 0#
fair_seldonian.experiments.runner.run_study(spec, output_file=None, n_jobs=1)[source]#

Run every trial of spec and return the raw per-trial records.

Nothing is averaged here. “No solution found” is recorded as NaN for loss and constraint value rather than 0, so that it cannot be silently averaged in. Storing 0 instead produces a dip-and-return shape in the constraint panel that reads as though found solutions sit comfortably below the threshold, when it is really an artifact of mixing refusals into the mean.

Parameters:
Return type:

dict[str, ndarray]

fair_seldonian.experiments.runner.true_g(theta, theta1, X, Y, T, config)[source]#

Plug-in value of the constraint on a held-out set - no confidence interval.

The Seldonian guarantee is a statement about g(theta) <= 0, so testing it means evaluating g itself. Reaching instead for eval_ghat - the high-confidence upper bound - and recording 1 if eval_ghat(...) > 0 tests a strictly more conservative event, which understates the violation rate and leaves the safety claim untested. It also makes the metric saturate: it reads zero for every roughly-correct variant and so distinguishes none of them.

Parameters:
Return type:

float

Module contents#