{ "cells": [ { "cell_type": "markdown", "id": "b909e406", "metadata": {}, "source": [ "# Certifying demographic parity, step by step\n", "\n", "Five steps, start to finish:\n", "\n", "1. **Define** the fairness criterion — demographic parity.\n", "2. **Specify $\\varepsilon$** — how equal is equal enough.\n", "3. **Specify $\\delta$** — how sure you need to be.\n", "4. **Train.**\n", "5. **Read the verdict** — a certified model, or *No Solution Found*.\n", "\n", "Steps 2 and 3 are the two numbers you choose, and they do different jobs.\n", "$\\varepsilon$ is a property of the *definition*: shrink it and you demand a\n", "smaller disparity, until no model in the class can deliver one.\n", "$\\delta$ is a property of the *evidence*: shrink it and you demand more\n", "confidence from the same rows, until the data can no longer support the claim.\n", "\n", "They fail in different ways, so the last two sections vary them one at a time.\n" ] }, { "cell_type": "code", "execution_count": 1, "id": "6462f938", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:35.105288Z", "iopub.status.busy": "2026-08-24T18:03:35.105153Z", "iopub.status.idle": "2026-08-24T18:03:38.210408Z", "shell.execute_reply": "2026-08-24T18:03:38.209917Z" } }, "outputs": [], "source": [ "import warnings\n", "\n", "import matplotlib.pyplot as plt\n", "import numpy as np\n", "\n", "from fair_seldonian import demographic_parity\n", "from fair_seldonian.algorithms import QSA\n", "from fair_seldonian.config import SeldonianConfig\n", "from fair_seldonian.data import data_split, get_data\n", "from fair_seldonian.models import predict\n", "\n", "warnings.filterwarnings(\"ignore\") # keep the demo output tidy\n", "\n", "# Two groups with clearly different base rates: 35% of group 0 are positive\n", "# against 65% of group 1. A disparity has to exist before it is worth certifying.\n", "data = get_data(\n", " N=8000, features=5, t_ratio=0.5, tp0_ratio=0.35, tp1_ratio=0.65, random_seed=7\n", ")\n", "X_te, Y_te, T_te, X_tr, Y_tr, T_tr = data_split(\n", " frac=1.0, all_data=data, random_state=1, m_test=0.3\n", ")" ] }, { "cell_type": "markdown", "id": "7fb27b941602401d91542211134fc71a", "metadata": {}, "source": [ "## 0. Coming from scikit-learn\n", "\n", "If you already have `X`, `y` and a sensitive column as arrays, you are one call\n", "away — but it is worth being precise about what this library does and does not\n", "do.\n", "\n", "**It does not wrap your estimator.** There is no way to hand `QSA` a fitted\n", "`RandomForestClassifier`, a `Pipeline`, or a gradient-boosted model and get a\n", "fair version back. `QSA` fits *its own* logistic regression,\n", "$\\sigma(X\\theta + \\theta_1)$, subject to the constraint. scikit-learn appears\n", "inside the library only to supply a warm start.\n", "\n", "**What it does** is answer this: given the arrays you would hand to\n", "`LogisticRegression`, produce a logistic model that *provably* satisfies a\n", "fairness constraint — or refuse. So the realistic comparison is your\n", "unconstrained `LogisticRegression` against a certified-fair one on the same\n", "data, which is what the rest of this notebook builds.\n", "\n", "Start with the model you would have written anyway.\n" ] }, { "cell_type": "code", "execution_count": 2, "id": "acae54e37e7d407bbb7b55eff062a284", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.211888Z", "iopub.status.busy": "2026-08-24T18:03:38.211777Z", "iopub.status.idle": "2026-08-24T18:03:38.218343Z", "shell.execute_reply": "2026-08-24T18:03:38.218013Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "sklearn LogisticRegression\n", " accuracy : 0.848\n", " positive rate, group 1 : 0.662\n", " positive rate, group 0 : 0.348\n", " demographic-parity gap : 0.315\n" ] } ], "source": [ "from sklearn.linear_model import LogisticRegression\n", "\n", "# The model you would have written: no fairness constraint anywhere.\n", "baseline = LogisticRegression(solver=\"lbfgs\", max_iter=1000).fit(X_tr, Y_tr)\n", "pred_sk = baseline.predict(X_te)\n", "\n", "T_test = np.asarray(T_te)\n", "rate_1 = float(pred_sk[T_test == 1].mean())\n", "rate_0 = float(pred_sk[T_test == 0].mean())\n", "acc_sk = float((pred_sk == np.asarray(Y_te)).mean())\n", "\n", "print(\"sklearn LogisticRegression\")\n", "print(f\" accuracy : {acc_sk:.3f}\")\n", "print(f\" positive rate, group 1 : {rate_1:.3f}\")\n", "print(f\" positive rate, group 0 : {rate_0:.3f}\")\n", "print(f\" demographic-parity gap : {abs(rate_1 - rate_0):.3f}\")" ] }, { "cell_type": "markdown", "id": "9a63283cbaf04dbcab1f6479b197f3a8", "metadata": {}, "source": [ "## 1. Define demographic parity\n", "\n", "Demographic parity asks that both groups receive positive predictions at the\n", "same rate. Within a group $P(\\hat{Y}=1)$ is $\\mathrm{TP} + \\mathrm{FP}$, so the\n", "criterion is\n", "\n", "$$ g \\;=\\; \\bigl|\\,(\\mathrm{TP}(1) + \\mathrm{FP}(1)) - (\\mathrm{TP}(0) + \\mathrm{FP}(0))\\,\\bigr| \\;-\\; \\varepsilon \\;\\le\\; 0 $$\n", "\n", "`demographic_parity(epsilon)` writes that in the postfix notation the library\n", "expects, so there is no string to hand-assemble. The gap measured above is the\n", "disparity this constraint will have to close.\n" ] }, { "cell_type": "code", "execution_count": 3, "id": "8dd0d8092fe74a7c96281538738b07e2", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.219522Z", "iopub.status.busy": "2026-08-24T18:03:38.219438Z", "iopub.status.idle": "2026-08-24T18:03:38.221289Z", "shell.execute_reply": "2026-08-24T18:03:38.220960Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "the constraint string, for epsilon = 0.1:\n", " PR(1) PR(0) - abs 0.1 -\n" ] } ], "source": [ "eps_demo = 0.10\n", "print(f\"the constraint string, for epsilon = {eps_demo}:\")\n", "print(\" \", demographic_parity(eps_demo))" ] }, { "cell_type": "markdown", "id": "090b499e", "metadata": {}, "source": [ "## 2 & 3. Specify $\\varepsilon$ and $\\delta$\n", "\n", "$\\varepsilon$ goes into the constraint; $\\delta$ goes into the config. Together\n", "they say: *the predicted-positive rates differ by at most $\\varepsilon$, and I\n", "want that to hold with probability at least $1 - \\delta$.*\n" ] }, { "cell_type": "code", "execution_count": 4, "id": "a79ab48e", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.222334Z", "iopub.status.busy": "2026-08-24T18:03:38.222262Z", "iopub.status.idle": "2026-08-24T18:03:38.224310Z", "shell.execute_reply": "2026-08-24T18:03:38.223994Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "constraint : PR(1) PR(0) - abs 0.1 -\n", "delta : 0.05 -> holds w.p. >= 0.95\n" ] } ], "source": [ "EPSILON = 0.10 # tolerated gap in predicted-positive rate\n", "DELTA = 0.05 # guarantee holds with probability >= 1 - DELTA\n", "\n", "config = SeldonianConfig(constraint=demographic_parity(EPSILON), delta=DELTA)\n", "print(f\"constraint : {config.constraint}\")\n", "print(f\"delta : {config.delta} -> holds w.p. >= {1 - config.delta:.2f}\")" ] }, { "cell_type": "markdown", "id": "5eee6d67", "metadata": {}, "source": [ "## 4. Train\n", "\n", "`QSA` splits the training data into a **candidate** set, used to search for a\n", "model, and a **safety** set, used to certify it. Nothing the candidate search\n", "sees is reused as evidence.\n" ] }, { "cell_type": "code", "execution_count": 5, "id": "de56be7b", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.225485Z", "iopub.status.busy": "2026-08-24T18:03:38.225411Z", "iopub.status.idle": "2026-08-24T18:03:38.381720Z", "shell.execute_reply": "2026-08-24T18:03:38.381286Z" } }, "outputs": [], "source": [ "result = QSA(X_tr, Y_tr, T_tr, \"opt\", None, None, config)" ] }, { "cell_type": "markdown", "id": "00cf17d8", "metadata": {}, "source": [ "## 5. Read the verdict\n", "\n", "When the run certifies, the number to quote is `diagnostics.safety_upper_bound`\n", "— the upper bound computed on the safety set, which is the evidence the\n", "guarantee actually rests on. Re-evaluating the bound on some other split gives a\n", "different number that carries no such claim.\n", "\n", "When it does not certify, `diagnostics.failure_mode` says which of the two\n", "refusals happened, and they call for opposite remedies:\n", "\n", "- `candidate_infeasible` — no feasible model was found at all. More data will\n", " not help on its own; $\\varepsilon$ is too tight for this model class.\n", "- `safety_test_rejected` — a feasible model *was* found, but the safety set\n", " could not certify it. Here more data, or a tighter bound, is what helps.\n" ] }, { "cell_type": "code", "execution_count": 6, "id": "4578457a", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.383266Z", "iopub.status.busy": "2026-08-24T18:03:38.383167Z", "iopub.status.idle": "2026-08-24T18:03:38.386361Z", "shell.execute_reply": "2026-08-24T18:03:38.386018Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "CERTIFIED\n", " safety bound on g : -0.0115 (<= 0)\n", "\n", " sklearn QSA\n", " parity gap 0.315 0.040\n", " accuracy 0.848 0.738\n" ] } ], "source": [ "if result.passed_safety:\n", " p = predict(result.theta, result.theta1, X_te).detach().numpy()\n", " pred = (p >= 0.5).astype(int)\n", " g1 = float(pred[T_test == 1].mean())\n", " g0 = float(pred[T_test == 0].mean())\n", " acc = float((pred == np.asarray(Y_te)).mean())\n", " print(\"CERTIFIED\")\n", " print(f\" safety bound on g : {result.diagnostics.safety_upper_bound:+.4f} (<= 0)\")\n", " print()\n", " print(f\"{'':<22}{'sklearn':>10}{'QSA':>10}\")\n", " print(f\" {'parity gap':<20}{abs(rate_1 - rate_0):>10.3f}{abs(g1 - g0):>10.3f}\")\n", " print(f\" {'accuracy':<20}{acc_sk:>10.3f}{acc:>10.3f}\")\n", "else:\n", " print(\"NO SOLUTION FOUND\")\n", " print(f\" reason : {result.diagnostics.failure_mode}\")" ] }, { "cell_type": "markdown", "id": "1e59dfe2", "metadata": {}, "source": [ "## Varying $\\varepsilon$: how equal is equal enough\n", "\n", "$\\delta$ fixed at 0.05. Tightening $\\varepsilon$ eventually asks for a model\n", "that does not exist, and the refusal is `candidate_infeasible`.\n" ] }, { "cell_type": "code", "execution_count": 7, "id": "1b6d14ab", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:38.387830Z", "iopub.status.busy": "2026-08-24T18:03:38.387746Z", "iopub.status.idle": "2026-08-24T18:03:39.174510Z", "shell.execute_reply": "2026-08-24T18:03:39.174047Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " epsilon = 0.02 no solution (candidate_infeasible)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " epsilon = 0.05 no solution (candidate_infeasible)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " epsilon = 0.08 no solution (candidate_infeasible)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " epsilon = 0.1 certified\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " epsilon = 0.15 certified\n" ] } ], "source": [ "EPSILONS = [0.02, 0.05, 0.08, 0.10, 0.15]\n", "eps_rows = []\n", "for eps in EPSILONS:\n", " cfg = SeldonianConfig(constraint=demographic_parity(eps), delta=0.05)\n", " r = QSA(X_tr, Y_tr, T_tr, \"opt\", None, None, cfg)\n", " eps_rows.append((eps, r.passed_safety, r.diagnostics))\n", " verdict = (\n", " \"certified\"\n", " if r.passed_safety\n", " else f\"no solution ({r.diagnostics.failure_mode})\"\n", " )\n", " print(f\" epsilon = {eps:<5} {verdict}\")" ] }, { "cell_type": "markdown", "id": "89aa7a88", "metadata": {}, "source": [ "## Varying $\\delta$: how sure you need to be\n", "\n", "$\\varepsilon$ fixed at 0.10. Here the disparity is *achievable* — what runs out\n", "is **evidence**. Demanding more confidence widens every interval until the\n", "bound can no longer clear zero, and the run is refused on data that was ample\n", "at a looser $\\delta$.\n", "\n", "Worth noting what those refusals report. They come back `candidate_infeasible`,\n", "the same mode as tightening $\\varepsilon$ — because the wider intervals make\n", "even the candidate search infeasible before the safety test gets a say. So\n", "`failure_mode` tells you *where* the run stopped, not which knob caused it. The\n", "two knobs differ in the remedy they call for, not always in the mode they\n", "surface.\n", "\n", "Note too that the certified bounds are not one model measured more cautiously:\n", "candidate selection optimises against a $\\delta$-dependent prediction of the\n", "safety test, so each row is a slightly different model.\n" ] }, { "cell_type": "code", "execution_count": 8, "id": "09b52b5d", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:39.175987Z", "iopub.status.busy": "2026-08-24T18:03:39.175880Z", "iopub.status.idle": "2026-08-24T18:03:39.945363Z", "shell.execute_reply": "2026-08-24T18:03:39.944977Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ " delta = 0.25 certified (safety bound -0.0263)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " delta = 0.1 certified (safety bound -0.0175)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " delta = 0.05 certified (safety bound -0.0115)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " delta = 0.01 no solution (candidate_infeasible)\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ " delta = 0.001 no solution (candidate_infeasible)\n" ] } ], "source": [ "DELTAS = [0.25, 0.10, 0.05, 0.01, 0.001]\n", "delta_rows = []\n", "for dlt in DELTAS:\n", " cfg = SeldonianConfig(constraint=demographic_parity(0.10), delta=dlt)\n", " r = QSA(X_tr, Y_tr, T_tr, \"opt\", None, None, cfg)\n", " delta_rows.append((dlt, r.passed_safety, r.diagnostics))\n", " if r.passed_safety:\n", " verdict = f\"certified (safety bound {r.diagnostics.safety_upper_bound:+.4f})\"\n", " else:\n", " verdict = f\"no solution ({r.diagnostics.failure_mode})\"\n", " print(f\" delta = {dlt:<6} {verdict}\")" ] }, { "cell_type": "code", "execution_count": 9, "id": "b5dcd206", "metadata": { "execution": { "iopub.execute_input": "2026-08-24T18:03:39.946741Z", "iopub.status.busy": "2026-08-24T18:03:39.946655Z", "iopub.status.idle": "2026-08-24T18:03:40.116866Z", "shell.execute_reply": "2026-08-24T18:03:40.116491Z" } }, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAA90AAAF5CAYAAACC3ElXAAAAOnRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjEwLjksIGh0dHBzOi8vbWF0cGxvdGxpYi5vcmcvJkbTWQAAAAlwSFlzAAAPYQAAD2EBqD+naQAARgZJREFUeJzt3QeYE9XawPF3F3apy9J7770LFopXEEEEVGyACIIIqNeC9drwihURVMSGBbFxEQVREUEFC4JIFb3Se0d6W9rO97zHO/mSJdkkuzm72ez/9zxhyWQyOfNmMmfeOWfOxDmO4wgAAAAAAIi4+MgvEgAAAAAAkHQDAAAAAGARLd0AAAAAAFhC0g0AAAAAgCUk3QAAAAAAWELSDQAAAACAJSTdAAAAAABYQtINAAAAAIAlJN0AAAAAAFhC0g0AMeSxxx6TuLg485gwYUK2lWPu3LmecvTv3z/byoHY5W5fVatWlZzq5MmT8sgjj0iNGjUkISHBrM/ll18u0U5j7sYfABBc3hDmAYBcTQ8wN23aFNK8c+bMkQsvvNB6mRDeiQhVtGhRufPOOwldFnjhhRfkwIEDPvHH2UaPHi1PPPEEoQGAGEfSDQCIaf/+97/N3ypVqpB0Z2HS7Z6oIukO7IsvvvD8/5VXXpFGjRpJiRIlsuAbAgBkJZJuAAhiypQpkpKS4nl+9dVXy86dO83/X3rpJWnWrJnnNT1oBpC7HD16VAoVKhT2+7Zv3+75/5AhQ+iuDQAximu6ASCIli1bSps2bTyPfPny+STZ7vQ77rjDdGFOTEyU48ePm9fXrl3rufbRu8Xv2muv9Uz/888/PdN1/htvvFEqVapklqOtXpdeeql8++23Gf6eTp06JV26dPF83s033yyO4/hc//3OO++Y1smaNWua9WvSpIl89913Zy0ro+XTZbVq1Ury588v1apVM5/lLTU1VZ588klp2LChFChQwMxXuXJl6dq1q7z11lsZWm93/Vza8up9HfDBgwclT5485rl+f64333zTM59em+4qW7asmaZ/066blrNkyZImJhobvY59zZo1Qcs4duxYz2fp57q0PGmvV/7qq688895///1m2rZt22TAgAHm+9LP1+uCixcvLhdddJFMmzbN8962bdt63rt+/XqfMlxxxRWe1xYvXmymLV++XHr06CGlS5c2y9TvuWnTpiYx3Lx5c8D10XEEdDnel2O4y/b+LnT7e+ONN+Tcc8+VpKQk833XrVtXHnzwQfO9hGvjxo2mvLosjcOtt95qEuG0PvvsM+nYsaMUK1bMbOd16tQxPSHc32sw3tvPihUr5OKLL5bChQub79+1YcMGGTRokOlZoZ+hMdTfu/fv3I2TzuuKj4/37CfSGxPB37Xs4fx+jhw5Yj7DnbdIkSLmkhjdvtI6duyY3H777VKqVCmznt27dzexBgCEyQEAhKVKlSqO7j71MWfOHM/0O+64wzP9xx9/NNMmTpzomdapUyfPvJUqVTLTSpUq5Zn2yy+/OElJSZ75vR9xcXHOK6+8ErRsw4cP97znnXfecVJTU50+ffp4pvXt29c5c+bMWfNWr179rM/Usuzbty9D5dO4uK/Vq1fPSUhIOOs9Tz/9tGf+xx9/3O9y9XHBBRdkaAv1Xr+0D/0OVZMmTczz/PnzOydPnjTTBgwY4JnvqaeeMtPWrVvnmXbVVVd5PmPcuHFm3f19hsZq4cKF6ZZx6dKlnvkHDhxopmk5tDzu9O3bt5vpjzzyiGfaF198YabNnz8/4Drq49133zXz6bbgThsxYoTn848fP+4UKlTITK9du7aZ9tdff5ntMtAyZ8+eHXB9vD/H30PpNnndddcFnKdu3bo+210g7vzFixd3KlaseNZyOnfu7DO/d/zSPtq2beucOHEi5M9MTk52SpQo4Xnevn178/rixYudokWL+v2MwoULm99QsDjpduv9++nXr5/fMrjbcDi/nwMHDjiNGjUKOK9uz966du161jwaa42593cKAEgfe0sAiFDS/cknn3imjxw50kwbOnSoz4G6Jrxbt271TLviiis8iUj9+vV9Ersvv/zSJArx8fFmWmJiorN58+awku4777zT8/zqq692Tp8+7Xdefdx///3O9OnTPYmoPl5++eUMlc87adBH7969zfx33XWXZ1q+fPmcPXv2mPlbtGhhpmnC8v777zvffPONOWExZMgQnyQ3HJs2bTInP9zPK1u2rHmuj19//dXM889//tPzupsg60kCd9pll11mpr333nueaS+++KKZpuuq66zTNAYPP/ywWUeNszuvxkxjF4huD7pduPMqTcy8YzdlyhQzvUOHDp7P2r9/v5m2YcMG55lnnjHbnsZM466Jtps016pVy8x35MgRzwkTTWpdmrx7J3vq888/90zr1auXSbKnTZvmjBo1yiSX3333XcD12bVrl4mvxtpdhhtz90TUpEmTPK8VK1bMeeONN5ypU6c6jRs39kzX7z0Y7xjp9qNlHDt2rFOwYEHPdN2elX637rRy5co5b731ljNz5kyfpFLjGM5nli5d2pT966+/NtuHfs8NGzb0vH733Xc7s2bNcp599lknT548PttDenHS7TbcpDvU38+tt97qef+ll15qtledzy2H929Y4+POW6BAAeeFF14wMW7ZsqVPHAAAwbG3BIAIJd27d+8+K5lu2rSped6gQQPzd8WKFc7kyZM9840ePdrMt2TJEp/k0G11VT179vS8NmbMmHTL5p1Iex8cd+/e3WeZaeft0aOHZ7p3UqRJe0bK5500VK5c2SfZ15Y39zU94FfnnnuueV6hQgXTenv06NGIbZf+khSX93ehybQms9pyXa1aNZNolCxZ0sx3yy23eObTWCj97txpGgOXxsY7mdLW7PRo8qPz6efq52ty473NaPKmybmbNGty6m3ChAmmpVYTLn+t7gcPHjTzDRo0yDNNW2TV4MGDPdNWrlx5VrJ13333mSQsvRMHwX4jaem26L6mSbJLfxveyXiwz/RexzVr1nimP/TQQ57p2mshbS+UBx980JPgep9g0IQ5GO/P1IQ6UK8F/d17n2w477zzPK8tWrQoaJzCTbpD+f3oNqRxdZNrTczd8nlv33pyJe0Jw3vvvdeznNWrV5N0A0CYuKYbACJEr3vU61LV/PnzzbWTet2nXlt55ZVXmuk///yzeXhfa6tWr17tmda8eXNzHa1Lr4V2ec8XzKJFi8xfvfbz448/9llmWu3bt/f833v0ZPe2T5kpn14Tr9dO+5vfvb544MCBnmuUzzvvPHP9qF5fPnjw4LDWOVzt2rXz/F+/swULFpjrjfV70XL/9ddf5vPd70yvf9Xrp9Oua+vWrT3/19h4D64XrPxuGfRzf/nlF1MO5d7eTD9bt6PDhw/7bDNqzJgx5prfH3/80XxXf+dkvtzv0I2x+uCDD8y87ujZWl69vtldfq1atcz/R44cabbf5ORkc93v+PHjzfXDmREobnqNccGCBc3/9+/fL3v27AlpeXoNu24r6W1f3p/51FNPmXXUR7du3TzTV65cGfI66DXTej13oPVatmyZ5zP04X6nyvva7kgJ5fej27LG1b0/uF7b7pZPR05PWz7va//POeccz/9129Br4gEAoSPpBoAIchMoHd188uTJcubMGXMQrA+lB9/uAbgeGHsnZ4F4D0AVDjfR1YGPXnzxxXTn9T6Izpv3/29s4S+Jy2z5/M1/0003mYGc+vbta5IvHZBs3bp1ZrAtPSHgJo6RVqZMGaldu/ZZ3433dzZ79myT9Krzzz/fDHgVyZikTfz1od/dddddZwbhWrJkic+Abt5Jtw7E5rrvvvvMgHaagHuPou8myZrgNmjQwPz/o48+kl9//dUkaap3796e+TXxnTdvnjz++ONmQDYdOE4T/u+//94MwqeJeDTL6O/l9OnTcuLEiZDm1e8lo/wN8JbeOug+xKWJsz+R/P2EWz4AQHAk3QAQQWlbIZUmbzpKszsa9tKlSz0JnJsYu4mf0tc1AXBp66fLe75gdCRoHclZ6WjXmmhlVGbKpyNie7eOes9fvXp1T3LfuXNnmThxoklwtZeA29KrJzC8eweEy00QArXQut+Zjrj96aefnpV0a2LrJj7eCbL3ui5cuNBntHj3O047nz/aoq6jSCv9fB0dXJNmPSmjZdBE8NVXX/XM710GN2nW3gnPPvusSZL1RI47PVCL6I4dO2TYsGGe+Ojo2i79LrTXxiOPPGKSeJ1XWz21PG4Zg/E+MZE27oHi9vvvv5vRst2TQFqGUOzbt8+Mqp/e9uX9mTpS//8ur/N5aLLpfWeCcJNO78/QRDfQZ2jrczDas8Dl3p5QzZw50+/8ofx+dF/gnlzT71JPpKQtn27nGh/v2Hn3mlEaa405ACB03KcbACLIOyHSJEJp4qQHu9p917sLq3eCrrdjqlevnunaqUlOnz59TLdhTSCmTp1q5tHWq549e4ZcFj1o1ttGdejQwSRuujxttfzHP/4R9nplpnyazPbr18+0pmoSp62oShMcTRTUVVddZW73pDGpWLGiSeq9D/TdFkhttddbjrmJjXcLcCAae00S9J7I2q1ab+WkLdxuF2r9ztzbKul3pgmJtha6twZbtWqV3+9My6wnMzTJ1kR0+PDh5uTKu+++a2Kk6tev7+mOHoh2R9f3zZkzx9Oi7ib8+ldvc+WWoUaNGlKuXDnPe3Vd9NZke/fulWeeeUYaN25sejUESoq0JfSBBx4w3Yvd70FvT6a3OXNpgqa3idLvUmOkydpvv/3mSYhDaQ3WmLu3w9KTFi1atDCJpJ5M0O1g+vTp5rVHH33UbAf6GXrrrrS31AuVLvPhhx+WrVu3+tyOTm8j5r7u9va46667THw0VtoCrC3Cs2bNMrF8++23JaP0e9btRrch7RVwww03yNVXX22+X91u9QSD/lbcLt7p0W1cT1zoCQu9JZ2eQNPfh37H/oTy+9Hl9erVy3Ql16S8U6dO5nvW2GvctNy6HWsM9FICvT2Ye7Ln5ZdfNsvVGOmtyQAAYQr3InAAyO0CDaTm0oHD3Ne9b0V14403+gxANHfuXJ/32bhlmNLRjN1pOlL2b7/9FnDe9AZxyugtw/R2ZO4I596PJ554wjO/OzK3v0eZMmXMrY7c0brT3qYpGO+B3vyt1/r1631eu+iiizyv6YBq3qOtp6Sk+Cw7s7cMcz366KM+73UHmPv+++99pvfv39/nfc8999xZn6uDv9WpU8fzXGPmTUez9p4/7XblPeK7v4f3rd4C0cHf0r7P/b50gLRrr702YrcM023a3y3OLr74Yp/B2NK7ZZi/AcvCHZQv2C3D/A2Ylt6AczpyfNr3eo+s712GUH8/OlBfercMS7tP69Kly1mva6zdEff9lRsAcDa6lwOAxdZubeFzBx1zWy/dVmHvQaTcAaC0K7a2CleoUMFcW60thtoarC1xQ4cOzVB5tFVaWxTVwYMHpUuXLrJly5awl5PR8mnrm7ZsardnbdXU1rLnn39eHnroIc88t9xyi2nd1JZcbWnWZetnaNl/+uknT3db767KoXYF1la6a665JmB3ZW1V1FY8l/f35P1/Xf+0n6nl1mu+NaY6oJeWu3z58qaVU2PlPQBVqNuM9+dq13Pva+y9W9rdVtsnnnjCxFSvxdYWSm0ZdVvp/fEeUE2Xra2xabtJawu+tr5rjwCdR78TXZdx48aZ14LRVn+9/ltjkbbFWp9/+OGH8tprr5mYFipUyMRVP1db4XUwu3AG6ipatKi5jl23Q12Wfg9Dhgwxrbben63XqOvAcTqfdsfX36VuY9rSry3I3i3tGaWDDOogavr52tNEf+daPm0B12na0yNU2kNAvxtdJ93+dZv64Ycf/M4b6u9Hy6JjBowYMcK0zOtlDbrdaI8GbS3XS1D0e3fpAIy33nqriZfOd8kll5gy6HIAAKGL08w7jPkBAMg2mki5Xdg10U87gjSC067HmshpF3M9WTBjxgzCBgCARVzTDQDIMfRaWaWJNwl3eDTJ1uuyJ0yYYP6vtPUUAADYRUs3ACDH0AHddOAwHdBN7x+N0D322GM+Xah1YDwdIM27+zoAAIg8aloAQI6h18sic/SaX702XK91J+EGAMA+WroBAAAAALCE0csBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AYsuvPBCufPOO4kxAABRjPoagE0k3YBFn376qYwYMSJHxHjnzp3Su3dvKVu2rCQmJkr58uVl1KhREf+ccePGSdWqVSV//vzSunVrWbhwYabf89hjj0lcXJzPo27duhEvOwAgNmVlfZ1V9W1W19c//PCDdOvWzayP1sPTpk07a54zZ87II488ItWqVZMCBQpIjRo1TNwdx7G4JkD2y5vdBQByopMnT5qKMpjixYtLTjF48GA5deqUfPPNN1KsWDHZtWuXHDhwIKKf8Z///EeGDRsmr732mqnAX3jhBbnkkktk1apVUrp06Uy9p0GDBqbsrrx52b0BQG4XjfV1VtS32VFfHz16VJo0aSIDBgyQK6+80u88zz77rLz66qvy7rvvmnp70aJFcuONN0pycrLcfvvtltcKyD60dCPmvfHGG+asa2pqqs/0Hj16mIpBzZw5U9q0aSNFixaVEiVKyGWXXSbr1q3z6XZ22223ma7iJUuWNBXPxIkTzbwnTpzwWe7ll18uffv29du9XJ9rpXLfffeZCl7PcmsrrbfDhw9Lnz59pFChQlKuXDkZM2ZMSN3e3nzzTWncuLE5c6yV10UXXRRWnHQ9NmzYIPPnzzcHKc2bNw97GcGMHj1aBg0aZCrY+vXrm8q8YMGC8vbbb2f6PZpkazzdh35PAICcI7fU15GobzNb59uor7t06SJPPPGEXHHFFQHn+fnnn8332bVrV9OKftVVV0mnTp1C6vUG5GQk3Yh5V199tezdu1fmzJnjmbZv3z5TcWtl6Z6d1TO6esb122+/lfj4eFNpeFf8elZWz5bPmzfPVD66XO0mNX36dM88u3fvli+//NJzcOCPLkcr6F9++UVGjhwpjz/+uMyePdvzupZDP0OXq9N//PFHWbJkSdBucXpgoF229Cy0Vmp33313yDE6ffq0dO7cWSZNmiQXX3yx6VLWvXt3OXLkiN/5n3rqKSlcuHC6j82bN/u8Rw8sFi9eLB07dvRM0zjrcz3w8Cec96xZs8YcrFWvXt18r2k/HwAQ3XJDfR1ufZvZOj+r6utQnX/++eZ7W716tXm+fPly+emnn0zCDsQ0B8gFevTo4QwYMMDz/PXXX3fKly/vnDlzxu/8e/bs0YuLnBUrVpjn7du3d5o1a3bWfEOHDnW6dOnief7888871atXd1JTUz3vu+OOOzyv6/M2bdr4LOOcc85x7r//fvP/Q4cOOQkJCc7HH3/sef3AgQNOwYIFfZaT1lNPPeU0bdrU2b9/v5MRt9xyizNlyhSfaVWqVHFGjhzpd/69e/c6a9asSfdx6tQpn/ds27bNxPTnn3/2mX7vvfc6rVq18vs5ob5nxowZzuTJk53ly5c7M2fOdM477zyncuXKJp4AgJwj1uvrcOvbzNb5WVVfp6Xvnzp16lnT9XvUGMbFxTl58+Y1f3V9gFhHSzdyBT1D/sknn3i6ln3wwQdy3XXXmTO3bitpr169TCtpkSJFTJcn5X32t0WLFmctV7tezZo1S7Zt22aeT5gwQfr3728GEAlEu4N50y5pesZdrV+/3lzn1apVK8/r2m2sTp066a6flkPrOO0Cp2ettdtaqJYtWybvv/++OdPuTT93x44dft+jn1OzZs10H1l5TbWeIdeWDI2tdiWcMWOGuT5u8uTJWVYGAEDmxXJ9nZH6NrN1frTV11ov63f64Ycfml4B2ptAB5HTv0AsI+lGrqCjaWoFpV3JtmzZYrqAuV3V3Ne1C9v48eNNNzJ9uF2sXNrFLK1mzZqZQUP0ejHtivXHH3+YSjw9CQkJPs+1wk97/Vo4tNLXAxLtsvXrr7+aSt09CAmFHtzUrl3bp1zafU+7fukgJ5HqrqbX1uXJk8cMGONNn+u1cv5k5D1Kr/XTdVq7dm1IMQAARIdYrq8zUt9mts7Pqvo6VPfee6888MADZh0aNWpkrqm/66675Omnn87UcoFox/C+yBX0dhc6kqaeXdVETM9E68AlSq8f02uitAJv27atmabXF4XqpptuMqN66tlzvd6pUqVKGS6nnrnXylgr0sqVK5tpBw8eNBVyu3bt/L5n6tSpZp28R+4Ox/79+02ln3YwGxVo9NEhQ4bINddck+5y9fpqb3p9nbY+6LVcOniN0oMXfa6D3viTkfcovTZOB9ZxB8gBAOQMsVxfZ6S+zWydn1X1daiOHTvm6bXg0gQ/MyczgJyApBu5hp4p11FO9ez29ddf75mut+vQUU214tOuY3rGV8/ChkrvtXnPPfeYgwA9g54ZSUlJ0q9fP3MmWLuE6W05hg8fbiqoQF3g9Oy+dkt77733zEGIJpw6sMvAgQND6jKmtwLRgVx01FWNjw5Y869//UteeeUVExt/tGwZub2KDjqj69eyZUvTJU8PfvQAREdHdb388svmoEIr91Dfo/HX1o8qVarI9u3bTcy0EtcuiACAnCVW6+uM1LeZrfOzsr7Wsnj3MNNu79oSr5/vnpjQuvrJJ580z7V1f+nSpWak9PQGtANiQnZfVA5kFR28o1y5cmZwj3Xr1vm8Nnv2bKdevXpOvnz5nMaNGztz5871GQQk7QArafXt29cpXry4k5KS4jPd38AsaZejg8b069fP81wHZ+ndu7cZjKVs2bLO6NGjzcAlDzzwgN/P1gFQhg0b5lSsWNEM6lKmTBnzftc777xj1iUQHUTmiSeecKpWreokJSU5559/vvPll186towdO9YMcpaYmGjWa8GCBT6vDx8+3AwqE857rr32WvPd6usVKlQwz9euXWttHQAA9sRqfR1KfRuszg5W52dnfT1nzhxT9rSPtDHTuOpy8+fPbwaze+ihh5wTJ05YWQcgWsTpP9md+AM5XYcOHcwZ25deeiniy9YzyxUqVJDnn3/enMkOl555//7772Xu3LkRLxsAADlJNNfXijobiE10LwcyQa/P0mRWH9o9LBK0q9XKlStNdy69PkzvC6p69OiRoeV99dVXpgsYAAC5VU6orxV1NhCbSLqBTNDRULUif/bZZ4Pe1iscevsMHSzGHcxER2/V0UQzYuHChRErFwAAOVFOqK8VdTYQm+heDgAAAACAJdynGwAAAAAAS0i6AQAAAACwhKQbAAAAAIDsHEgtNTVVtm/fLklJSRIXF2erLAAA4O+b9Mrhw4elfPnyEh8fH9b7Dh06ZP5PfQ0AQHTU1yEl3ZpwV6pUKZLlAwAAQWzZskUqVqwYcpy04i9atChxBQAgiurrkJJubeF2F1akSJHIlQ4AAJxFW6v1ZLdb/4aL+hoAgOipr0NKut0uappwk3QDAJA1MtpFnPoaAIDoqa8ZSA0AAAAAAEtIugEAAAAAsISkGwAAAAAAS0i6AQAAAACwhKQbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsCSvZLN5PXpmdxGi2gWffRKR5RDnrIkzsc66WHebcFdElhOrPu8/JiLLIc5ZF+tox7YQHL+7rEGcc1acFceh6eN4P+cd74eLlm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMCSXJ10xyUkyAWffWIeqkjDBub/DZ/4t9/5y3buZF6vefttIS2/XLeuUum6ayRPoYKSmxFn4gwAAMAxaOzieD99eYO8nqsc37JFVo0aLacOHIzI8sp3u0zylyktu7+bI2eOHovIMmMBcSbOAAAAHIPGLo73c1DSXbh2LalyfW8pVL2axCckyIFlv8m2qdOk5q1DJV/pUmae49t3yJaPJsm+hYvMc7fVevOHk6TMJRdLXJ68svn9D2TX7G/N9NIX/UMq9+kl8YkJsm3adJ/PK1CpktS5Z5gcXPG7ecTnyyc1bhksxVudIyk7d8nhVat95i9x3rlS5YbrJbFEcXHOnJFjmzfLxncmyuGVq6TFG6+ahFu1HP+a+TuvR0/JV6qUVO1/gxRpWF/i8ybI4VWrZMPbE+T41m2SXYgzcY6l7RkAAOQMHIMS69xyHBq13cs1qW74+HAp0qC+7Jw5Sza8+bak7NwpqSdOmpbj9ePfli2Tp0hCkSJS++67zurCXahGddnx+ZeSWDRZqg0aKPGJiVKgUkWpedtQyZM/v2z+aLIUrlkz3TJUuuYqKX1hezm6br3s/OprKX5OC5/XTx89aqZvGP+WbJ82XQpq0n7f3ea19ePfklMH/24xX//Gm6YFXeLjpd7D/5JiLZvL7m/nyPbpn0vhWrWk/qMPSVze7Dn/QZyJcyxtzwAAIGfgGJRY56bj0Ogr0f8Ua95c8hQoILvnzJXN73/omZ5Ut46UurC9FKxcSeLi//+cQYEKFeTI6jWe52tfflVOHzpkrqvOV6KEJBYvLkUbN5K4PHnkr5/myc4ZX8ne+fOl5PnnBixD0aZNzN9N731gWrnzJhWWqjdc73ldk/eyl3aWAuXKeqblLVRIEpKTZf+vi+RMyglJSBbZ9+siObF7jxSoWEEKVa1i5qvY8wrPexKKJJkN6+iGDZLViDNxjqXtGQAA5AwcgxLrqjGWV+XIpDuQqjfeYAK89ZOpcmD5b6b7eVLtWqYl25sm3Mo5fcb81WQ70moMvdkk89qN4ejGTVLztlskf+lSEp/PLYvj930pu/fI2pdf8TyPi4uTlN27JZoQZ+IcS9szAADIGTgGJdZHYyyviuqke/+SJXLm+HEp2baNnNjzl6Ts2i0FK1X0vJ63cGEpXLOGFKpWNeRlapKu1wiUbHOBHN20WZIbNUy/DEuXmc+o0reP7PnhJyl3aWe/8+VNSjKt6LpheDt9+IhImTLmOvIja9aa5elGpCcNSpzb2kzLV6a0lGrfTpYMuVWyA3EmzrG0PdtWunAx6VCzlew+sk++XfurmdawbA15uvNtsmLnWnlw5jgzbeA5PaRDzXMkKV8h+WHDEtl6cLf0btpZPlw2Uz5a9nVYn1kxubS8esW/ZNeRfXLTlBGSWxBrsC3wm4tF7Nv+H8egWYdYZ7+ovaZbuw388dgIOfTfP02yW33QAMlfrqxsfPtdObZ1m5S6sJ0UrlFdDv62IuRl6kX12u38TEqKVLq6pxlVLz1bP/5Eds/93gzkVrbLJXJg2XKf19e9+rqc2LNHyl92qeQpVEiOrF/v+/5Pp8nJffulcq9rpeqA/iKpqfLnE0+bBL7Eea2l+pBBUqpdWzm4/DfJLsSZOMfS9mxTfFy8lC5c3CTPmni7thzYJSO/nyiT/pdMJ+cvLJc3uFDyxueVMT9+KNP/+4PM27jczKN/QazB7y7asH8jztmBY1BinZuOQ+Mcx/HfVu/l0KFDkpycLAcPHpQiRYpEtAA68hwCc0djJ845I86KbTprYt1twl0hz1unVBW5vtmlUqNEBUmIzyvLdqyWp757R3o2ukgurtVaShRMNi3Jn/7+nacF+/P+Y8zfSctnSada58q2Q7ulUVnfwRe/XbtQvlm70NPS/cJPH8pbVz3qM4+2bivvlu6kfAWlX4vLpEWFelIwMb9s2Ldd3lk0XVbt2WTmvbpRR+lWv52kpp6RWWt+kV5NLwm7pdstf1bGWRHryMhovWuzvmZbiM7fHb854hzt9Yji2Ch9HO/nvOP9cOvdqG3pBoBIdeV7vNMQ0w38q5U/y/iFU2XHob/kiob/MInv5gM7TSJ8KOWo3NmmtzQrX8fn/Q3KVJf3lnwpHy6d6WnN1vdoy/WMlfN85j2YclReX/Dp//5/JGDr9rC2faRjzdYyf9Nv8smKb6VkoaLyWMebTSu5fv4NLbrK6dTT8tHyWabcOQWxBtsCv7lY3L+xbwMQs9d0A0AkmNbkhPzy3dpf5b2lMzzTR3W90/w9t3Ij83C1rFhflm5f5Xk++ocP5K9jB/5+EidynVxiEuofNyw1k7wPGk+cPikLt/4ug+VKSTl90jPPBfL3nRBUvryJ0rxCXdOdU1shvNUrXU3ql65m/q8J/der58vG/ds9ZY12xBpsC/zmYnH/xr4NQMwm3XrvPr35uQ6gtvjmoUHn1779pS+6UBKSkmTPjz/J6lGR6xLjzb05+6JBQ8zz9MpYpGEDafTk43Jwxe/y+8PDJVoRa+Ica9t0OF5b8IkZ6Mx14Phhn9c9CbcKejFO6DRBf/K7tyXV6wqfLQd3eQ5KYxGxBtsCv7lYxL4t4zgGzRrEOftFbdJ96uAhc+NzvSdbMAnJRaRCj25mtPPVL4yVlO3brZVLb86eJ38+Uz793FhArIlzrG3T3hZv+1OOnUqRdtWby56j+801bZWSy8jPm34z10J2rNlKZqyaJwUS8pmujz9sWCqbDuzwu6zDJ46Zv+WTSsqF1VvI2r3pD8YYKNlesm2laVHX68m1fMULJEvbak3lie/eMq3s2vX90roXmM9rX7255BTEGmwL/OZicf/Gvs0ejkGzBnHOflGbdOvBf517hpkWty1JSVLrjtvMLYqcM6lSpH5dSdm+Q1Y9N1qc1DOmZU7lKVBAat/5T9n80X/k8Oo1UuHKy6VMxw6SWKK4nNi9W7Z9+pns/m6Ombfm7bdKsRbNzU3XTx85IvuXLJMN49+UM8dTzG3Cqg8eJAUrV9KbvcmJXbtl/Rtvmta96oMG+rQKqrj4OKk2sL+UubijnDxwQDaMf1v2L15y9krFxaVbpuxCrIlzrG3T3nYf2S+PzX5D+jTrIl3rtpG88XnMQGrvLPpce4tLx1qtZXDrK01ivm7vVtPdMRBNxr9fv1haV2ood7e7Xt5d/IWs3LMx7DKN/vEDuaF5V2lRsZ60rtxQ9h8/LP/dtV6OnDhuDkonLv5SutdvJ9c07ig/blh21gBu0YpYg22B31ws7t/Yt9nDMWjWIM7ZL2qTbn/0vtqbP5wkknpGirc6Rypec5Wsf328SYir33yTnDp40LREH9u0WSpc3l2q3nC97F2wUHbN/kaKtWxhEveT+/fLgaXL5NjGzXJ41WqJi4+XIvXqSpkO/5CT+/bJ5vc/lIpXXyVJtWvJhrcnyJljx0zyHZc3cKjylSol8YmJsuU/H0vlPr2kzr3DZPGQ286aL1iZogmxJs6xtE3/uXuDPPz1K2dN/+T378wjnNGDR/3wfrrz6sFZ2vfqQG3e9+fWFp5x8z8OWN6PV3xjHq4Jiz+XnIJYg22B31ws7t/Yt2UdjkGJc3wMHYPmyKRb75O97ZOpcqRJY5N0FyhXVlJPnJB9vy4ySbd2Rf/rx79HE675z1vM3xLntjIPl7Zu63Lylysjpf9xoWkddxWu/ve1lMe3bhWRVlL8nJZyZO06OfTflXIgnXu+nT5yVNa9Nl7EcaRI/XqmbNoaf+qQ77WhJc4/N3CZomzjINbEOda2aQAAEP04BiXO62LwGDRvTrseQTlnzpi/cXnyBH3PutfflOPbtv3/Mg4ckKJNGku5S7vI8R07ZOOYlySxRAmpMfgmc1ZFbXrvA9OVPKlObUmqV1cqXNFDtn32uWx8e0JE1sNfmaINsSbOsbZNAwCA6McxKHGOxWPQmL1P996fF5i/2m1cu39rF/Hy3btJof+1Zqv4hERJSE6Wkhec5/PeStdeLYVr1ZSU3bvl6Ia/r9fMV6pkwM/KW7iQ1BgySMpf3l2KNmtqBnQ79MefGSpTTkSsiXOsbdMAACD6cQxKnMvkkGPQHNXSHY5t06abQZ7KdLxIqt880FybfWTdBjm2cZNJpHfOnCWl2reVilf3lF1fz5Lkhg087009fdoMIJWvRHHzf2313vzBRwE/68SePZJ68pRUvPJyOfHXX7LhzXfM9eUFKlUMuUw5GbEmzrG2TQMAgOjHMShxLpNDjkHjHMfrJrEBHDp0SJKTk+XgwYNSpEhkbyk0r0fPiC4v1lzw2ScRWQ5xzpo4E+usi3Wggc7wt8/7jyHOOSzWkah3bdbX/OaC43eXNYhzzoqz4jg0fRzv57zj/XDr3ZjtXg4AAAAAQHYj6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAAEi6AQAAAADIWWjpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwJK8oczkOI75e+jQoYgX4OipUxFfZiyJVMyJc9bEmVhnXaxPHT8RkeXEKuKc82Ltb5lu/ZvR90cSv7msizuxJs7RgGOjnBdrjvejt76Oc0Ko0bdu3SqVKlWKXOkAAEBQW7ZskYoVK4ZV+ScnJxNZAACiqL4OKelOTU2V7du3S1JSksTFxUms0oMVPbmgQStSpEh2FydmEWdiHWvYpol1pGnVfPjwYSlfvrzEx8eH9T73rDv1NSKB/VvWIM5Zh1gT5+yor0PqXq4LCOdMe06nCTdJN3GOJWzTxDnW5IZtOiMt1ppo56aW7tywHUQLYk2cYw3bNHGOlFDqXQZSAwAAAADAEpJuAAAAAAAsIen2ki9fPhk+fLj5C3uIc9Yh1sQ51rBNg+2A31wsYt9GrGMN23QGBlIDAAAAAADho6UbAAAAAABLSLoBAAAAALCEpBsAAAAAAEtIugEAAAAAsCSmk+5x48ZJ1apVJX/+/NK6dWtZuHBhuvN//PHHUrduXTN/o0aNZMaMGZ7XTp06Jffff7+ZXqhQISlfvrzccMMNsn379ixYk9wVa9W/f3+Ji4vzeXTu3NnyWuS+OB85ckRuu+02qVixohQoUEDq168vr732muW1iP3Y//HHH9KzZ08zv267L7zwQpaWNTfH9bHHHjtr36G/AcTWtjB+/Hhp27atFCtWzDw6dux41vzUI3Ziz/7NTlwVx0bREedPP/1UOnXqJCVKlDB1yLJlyyQ3inRcHceRRx99VMqVK2eOOXW/vWbNGp95nnzySTn//POlYMGCUrRoUYkpToyaNGmSk5iY6Lz99tvOH3/84QwaNMgpWrSos2vXLr/zz5s3z8mTJ48zcuRI57///a/z8MMPOwkJCc6KFSvM6wcOHHA6duzo/Oc//3FWrlzpzJ8/32nVqpXTokULJ7eLdKxVv379nM6dOzs7duzwPPbt2+fkZjbirMuoUaOGM2fOHGfDhg3O66+/bt7z2WefZeGaxV7sFy5c6Nxzzz3ORx995JQtW9YZM2ZMlpc5t8Z1+PDhToMGDXz2HXv27MmCtUFWbgu9e/d2xo0b5yxdutT5888/nf79+zvJycnO1q1bPfNQj9iJPfs3O3Hl2Ch64jxx4kTn3//+tzN+/Hi9w5PZz+Q2NuL6zDPPmP30tGnTnOXLlzvdu3d3qlWr5hw/ftwzz6OPPuqMHj3aGTZsmJk3lsRs0q0J8a233up5fubMGad8+fLO008/7Xf+a665xunatavPtNatWzuDBw8O+Bla8eiPcdOmTU5uZiPWerDUo0cPi6XOeWzEWZOTxx9/3Gee5s2bOw899FDEy5+bYu+tSpUqJN1ZGFdNups0aRL0/YidbUGdPn3aSUpKct59913PNOoR+7Fn/xa5uHJsFH3H+9oYkVuT7kjHNTU11Zwsf+655zyva4Nmvnz5zIn0tN55552YS7pjsnv5yZMnZfHixabbgis+Pt48nz9/vt/36HTv+dUll1wScH518OBB0+0k5ro/REms586dK6VLl5Y6derI0KFDZe/evZJb2YqzduGZPn26bNu2zXT7mTNnjqxevdp0q0LGYw8723SotLuaXgJUvXp16dOnj2zevJmvJMa3hWPHjpnLwIoXL+4znXrEfuwRmbhybBS9x/u5jY24btiwQXbu3OkzT3Jysum2nltiH5NJ919//SVnzpyRMmXK+EzX5/qF+6PTw5k/JSXFXOPdq1cvKVKkiORWtmKt129PnDhRvv32W3n22Wfl+++/ly5dupjPyo1sxXns2LHmOm69pjsxMdHEXa/hadeunaU1yR2xR/bFVSvwCRMmyMyZM+XVV181Fb1e+3v48GG+lhjeFrQ+1hMt3gd01CNZE3tEJq4cG0Xn8X5uZCOuO//3NzfHPm92FyAn0rPp11xzjWkZ1IM6RN51113n+b8OxtC4cWOpUaOGabXo0KEDIY8QTboXLFhgWrurVKkiP/zwg9x6661nHbwCOYWenHPpfkOTcN22J0+eLAMHDszWssGOZ555RiZNmmTqBx3Ax0U9gljDNg3kXDHZ0l2yZEnJkyeP7Nq1y2e6Pi9btqzf9+j0UOZ3E+5NmzbJ7Nmzc3Urt+1Ye9NuovpZa9euldzIRpyPHz8uDz74oIwePVq6detmEhQdyfzaa6+VUaNGWVyb2I89oieuevlP7dq1c+2+I9a3Bd1XadI9a9Yssw9LT26vR/xh/xY9ceXYKHrjnNvYiGvZ//3NzbGPyaRbu8m2aNHCdE12paammufnnXee3/fodO/5lSbV3vO7CbdeL/jNN9+YWwnkdrZindbWrVvNNd16m4HcyEacdXvWh16n4013tLpsZDz2sLNNZ4TeFm/dunW5dt8Ry9vCyJEjZcSIEeZSgpYtWwb9nNxej/jD/i164sqxUfTGObexEddq1aqZ5Np7nkOHDskvv/ySe2LvxPBQ9zoi3oQJE8zQ9TfffLMZ6n7nzp3m9b59+zoPPPCAz1D3efPmdUaNGmVuP6Ij4HoPdX/y5EkztH3FihWdZcuW+dyO5sSJE05uFulYHz582NwWSG/LpiNHfvPNN2ZE7Vq1ajkpKSlObhXpOKv27dubEcz1lmHr1683o0Xmz5/feeWVV7JlHWMl9rpP0NFO9VGuXDmzPev/16xZk41rkTvievfddztz5841+w79DeitHkuWLOns3r07W9YRdrYFvfWM3s5mypQpPvWx1h+KeiR07N/s4Ngo5x4b7d2719QtX375pRm9XD9Dn+s+JrewEddnnnnGLENvS/vbb7+ZuxSlvWWY3hFKY623bCtcuLCnznf37TlZzCbdauzYsU7lypVNxaxD3y9YsMAn2dDbiXibPHmyU7t2bTO/JiL6Y0t72wB/D01YcrtIxvrYsWNOp06dnFKlSpkfrN6SRO8P6P7Qc7NIxllpBaL3t9XbQGiyXadOHef55583t3ZAxmMfaH+h88FuXK+99lqTkOvyKlSoYJ6vXbuWsMfYtqD1gr9tQQ/0FPWIvdizf7MTV8WxUXQcG2kDRHr7l9wi0nFNTU11HnnkEadMmTImoe/QoYOzatUqn3l0mbGaa8XpP9nd2g4AAAAAQCyKyWu6AQAAAACIBiTdAAAAAABYQtINAAAAAIAlJN0AAAAAAFhC0g0AAAAAgCUk3QAAAAAAWELSDQAAAACAJSTdAAAAAABYQtINAAAAAIAlJN1AGPbu3SulS5eWjRs3ZjpuF154odx55505Kv7RVObrrrtOnn/++ewuBgBErD5xHEduvvlmKV68uMTFxcmyZcvC3i9H0346kiK1XsGWEyyekSpHZuqwUI9Fwi1rrG472bmOWfUdcEwU/Ui6gTA8+eST0qNHD6latWqOqaSivXwZ9fDDD5vv4+DBg9ldFACQnTt3Su/evaVs2bKSmJgo5cuXl1GjRoVVn8ycOVMmTJggX3zxhezYsUMaNmwYNLKffvqpjBgxgm8gQoLFM+3rGa1jM1OHpd12ApXBxrZh+5giksuPhuOfrPoOOCaKfiTdQIiOHTsmb731lgwcODAqYnby5EnJzfRgtEaNGvL+++9nd1EAQAYPHiwHDhyQb775RjZs2GAS5+bNm4dVn6xbt07KlSsn559/vkne8+bNGzSy2iqelJSUY+uXaKvLgsUzUvHOaB0WzrGIjW0jVr93W+uXVd8Bx0TRj6Qbud6bb74pjRs3lgIFCkhycrJcdNFFfmMyY8YMyZcvn5x77rmeaf3795fvv/9eXnzxRdMVUB/a3evEiRNy++23m+5f+fPnlzZt2sivv/6abqxTU1Pl6aeflmrVqpmyNGnSRKZMmeJzZvO2224zZzdLliwpl1xyiadlRJdftGhRKVGihFx22WXmwC298oXyeUePHpUbbrhBChcubA4CQ+0Gd/jwYenTp48UKlTIvG/MmDE+Z2XTK2/addWHfie6vo888ojpeumtW7duMmnSpFy/DQPIfrrf12R7/vz55oBbE+5w65N//vOfsnnzZrOvdlsxg+0zg7Xm6XJeeOEFn2lNmzaVxx57LN36JVgd4U8o++5AdVko9ebp06fTXXYo9Uuw5YTT/dxfHfv444+bz9b18Xb55ZdL3759M12Hpd120qvn065LsPpZ6fd+3333mWRRT/x4byeZOaYI9L17S2/54R5XBStroHUMZV38CbR+3vENJf6Z+Q44JopuJN3I1bTbj+7YtMJdtWqV/Pzzz3L33Xf7nffHH3+UFi1a+EzTnd55550ngwYNMl0B9VGpUiWzzE8++UTeffddWbJkidSsWdPsgPft2xewLLqDnzhxorz22mvyxx9/yF133SXXX3+92bm6dHnabXHevHlmPjc5HjZsmCxatEi+/fZbiY+PlyuuuMLstAOVL5TPu/fee83/P/vsM5k1a5bMnTvXrEswWhYt3/Tp02X27Nkmbt7vS6+83nRdtZVn4cKFZj1Gjx5tTpB4a9WqlXk97cENAGQlTeI6d+5sEqiLL75Yxo0bJ927d5cjR46EVZ9owlaxYkWzr3YTilD3mZmVtn4JpU4KtJxg+25/dVko9WawZUeyfgmFvzpWjyHOnDlj6kDX7t275csvv5QBAwZkug5Lu+2kV8+HWz+7sdGk8JdffpGRI0eabVLnDfZZGT2GCRZPd/nhHlelt6z01jHUdfEn2PqFEv/MfAccE0U5B8jFnnrqKadp06bO/v37g87bo0cPZ8CAAWdNb9++vXPHHXd4nh85csRJSEhwPvjgA8+0kydPOuXLl3dGjhzp930pKSlOwYIFnZ9//tln2QMHDnR69erlmb9Zs2ZBy7lnzx49Xe+sWLHCb/lC+bzDhw87iYmJzuTJkz2v7d271ylQoMBZy/J26NAhs+4ff/yxZ9qBAwfMZwV6X9ryumWuV6+ek5qa6pl2//33m2neli9fbt67cePGoHEBAFtuueUWZ8qUKT7TqlSp4rPPD6U+GTNmjHlfZvbxaZ/r8nS53po0aeIMHz7c5z3e9UsodZI/oey7/dVlodSbodYL6cUqlOUEi2ew52ro0KFOly5dPM+ff/55p3r16j6fmdE6zN+2468MaaeHUj/r/G3atPFZxjnnnGPik95nRfIYxt/yQz2uCmVZwdYxM9u+v/VzyxDq8VFGvwPFMVF0o6UbuZqeKdQuZdqFR7tRa9fAQI4fP266NAWjXdlOnTolF1xwgWdaQkKCOQP5559/+n3P2rVrzXVa2kKi5XAfeqbVu2tc2pYRtWbNGunVq5dUr15dihQp4umSqF0UAwn2efrQ7pGtW7f2vEdjVKdOnXTXff369WbddV1d2n3P+32hlle7zmm3KZee2dX3aguCS7t9KV0XAMgOOsK4XperLdvedN+nrVCZqU8yuo/PCO/6JdQ6yZ9Q9t1p67JQ681gy45k/ZLZYwvtIbZt2zbzXAfH027B3p+Z0TosnG0n3PpZ6eV23rQbtLbUpyczxzChyMhxVXrSW8fMbPvprV+o8Q9WvvRwTBTdgo/QAcQo3fnpLRZ0wBodlER3ft6jyKal1+js37/fSlncLoja/axChQo+r+m1Wy7tbpSWXsNTpUoVGT9+vBktV7vR6YAa6Q1SEuzz0usGn1kZKW8gbjlLlSploaQAEJx2ea1du7ZJAry7Oa9evdpcg5rZ+iSz+0ztYp12PAyt/9Lyrl9CrZMyyl9dFm31S2Y0a9bMXAesiVqnTp1MF2WNZSTqMJvHIsp7O1Z6oiDYpQyZOYbJDumtY2a2/UitX0a+A8UxUXQj6UauNXXqVHNGU0eaDbUS9TfKqF6/4312XEcjda/p0crfPcDR6/MCDc5Sv359szPXs/Ht27cP616dei26HmC0bdvWTPvpp5/SLV8on1esWDGz09friSpXrmymaSWvB5HplU9bF/R9uq7u+/R2KPq+du3ahVRel362twULFkitWrUkT548nmm///67uf5RD0IAIDvovlGTbG9vvPGG+XvllVeGVZ+kFc4+MxBN6Lxb3A8dOpRur67M1Emh7rvTCrXeTG/Zka5fQuWvjlU33XSTGcBOW7s7duzo9zrrjNRh/radQGUIp34OVUaOKcLhb/kZOa4KtKxgIrkuNuKf3npxTBTdSLqRa+mZbz0Qee+990wFrWc3dYeut+Hwd5sWHbDjX//6lznA0qTUpa3jWoHr6JHaBUm7YQ8dOtQMRKb/152rDoSh3ZUC3eJDbydxzz33mME69GymjsqpO2Mtj3aR69evn9/3aTl0lFQ9wNPuR1pJPPDAAz7z+CtfKJ+nZdV10OXraKEPPfSQaTFJjy5X3+uuu75v+PDh5n16pjaU8rr0NR10RG/DowONjB079qwR1HUQEm1FAIDsopfh6MBpOhKxjpatI2hrXfHKK6/41BWh1CdphbPPDERHUNfuzdoKrKN6P/roo0GTy4zWSaHuu/21EIZSb6a37EjXL6HyV8dqnaf3bNcY6kkAbfH2JyN1mL9tJ1AZwqmfM7O+mdleQll+qNtHKMsKJpLrYiP+gdZLl8MxUXQj6UaupV3Lly5dKg8++KDs2rXL7LQ6dOhgKmF/GjVqZG4BM3nyZJ95dOesO1I9O6rXWmkLwjPPPGN21np7EL1FRMuWLeXrr79O9+BqxIgRpkVCR83Ua3/04Eg/T8sXiO5kdbRc7cKoXej02qCXXnrJ3IIivfLpDjvY5z333HPmRIQeqGlloSOyasUTjI4CO2TIEHPwqRWUjji6ZcsWcw1aKOV16e3KtLx6/ZMeIN5xxx1y8803e15PSUmRadOmmQNcAMguOqqxJnG6L9ODaK0r9M4Yl156acD3BKpP0gpnnxmIJmi639d9sl5Gpfv+YC3dGa2TQtl3BxJKvZnesiNZv4QjUB2rse7Zs6fppqy3C0sro3WYv20nUBnCqZ8zu74Z3V5CXX5Gjqv8LSsUkVoXG/EPtF56azGOiaJbnI6mlt2FAHIKrTz1LKV24QnW6ou/abdLvS5KWxHSOyPtTQ+S9D6yae8t6+3VV181lwjoYDUAkNPEYn0Syr47N9ET+Q0aNDDJfyTrsEhtOxmpnxE5kYw/x0TRj5ZuIAxdu3Y1I5zqNVqB7oOZ22nvgZUrV5oWBG0Z1/tLqh49ekT0c/TaKO0SCAA5EfVJ7NKu33PnzjUPvcwg0nVYRredrKqfkfXx55go+pF0A2FKb9AO/G3UqFFmQBsd7ENvoaHXGUV6sDMdpAYAcjLqk9ikg51p4v3ss88GvNVmZuuwjG47WVE/I+vjzzFR9KN7OQAAAAAAlsTGRUQAAAAAAEQhkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAADAEpJuAAAAAAAsIekGAAAAAMASkm4AAAAAACwh6QYAAAAAwBKSbgAAAAAALCHpBgAAAABA7Pg/HPv8bbTgPAgAAAAASUVORK5CYII=", "text/plain": [ "
" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "fig, (ax_e, ax_d) = plt.subplots(1, 2, figsize=(10, 3.8))\n", "\n", "for ax, rows, xs, xlabel, title in (\n", " (\n", " ax_e,\n", " eps_rows,\n", " EPSILONS,\n", " r\"$\\epsilon$ (tolerated gap)\",\n", " r\"varying $\\epsilon$, $\\delta=0.05$\",\n", " ),\n", " (\n", " ax_d,\n", " delta_rows,\n", " DELTAS,\n", " r\"$\\delta$ (failure probability)\",\n", " r\"varying $\\delta$, $\\epsilon=0.18$\",\n", " ),\n", "):\n", " colors = [\"#4C9F70\" if ok else \"#C44E52\" for _, ok, _ in rows]\n", " ax.bar(range(len(xs)), [1] * len(xs), color=colors)\n", " ax.set_xticks(range(len(xs)))\n", " ax.set_xticklabels([str(x) for x in xs])\n", " ax.set_yticks([])\n", " ax.set_xlabel(xlabel)\n", " ax.set_title(title, fontsize=10)\n", " for i, (_, ok, diag) in enumerate(rows):\n", " ax.text(\n", " i,\n", " 0.5,\n", " \"certified\" if ok else diag.failure_mode.replace(\"_\", \"\\n\"),\n", " ha=\"center\",\n", " va=\"center\",\n", " color=\"white\",\n", " fontsize=8,\n", " fontweight=\"bold\",\n", " )\n", "\n", "ax_d.set_xlabel(ax_d.get_xlabel() + \" (tighter to the right)\")\n", "fig.suptitle(\"Two knobs, two ways to be refused\", fontweight=\"bold\")\n", "fig.tight_layout(rect=[0, 0, 1, 0.94])" ] }, { "cell_type": "markdown", "id": "b7efab6e", "metadata": {}, "source": [ "## What to take away\n", "\n", "Refusal is the design, not a failure of it. A Seldonian algorithm returns a\n", "model only when it can demonstrate the constraint holds; the alternative is\n", "returning one whose fairness is merely hoped for.\n", "\n", "When it refuses, `failure_mode` tells you which lever to reach for — relax the\n", "definition, or gather more evidence. Those are different problems, and a bare\n", "\"No Solution Found\" hides which one you have.\n" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.7" } }, "nbformat": 4, "nbformat_minor": 5 }