Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Using Project Source Code

This chapter demonstrates how to import library code from src/ without sys.path hacks. The project is installed in editable mode when you run uv sync, so notebooks can import packages directly.

The example pipeline:

  1. Generate synthetic data with a small simulation

  2. Bundle a Random Forest and an SVM (with StandardScaler) into a single MultiRegressor composite estimator

  3. Hand that composite to compare_models, which fits each base, computes bootstrap confidence intervals, MAPIE split conformal prediction intervals, and bootstrap-CI’d regression metrics

  4. Visualize the comparison with faceted Altair charts

1Imports

No path bootstrapping is required — the packages are installed by uv sync.

from sklearn.model_selection import train_test_split

from analysis import compare_models
from core import ModelKind, Settings, TrainingData, build_split_dataset
from prediction import (
    MultiRegressor,
    random_forest_regressor,
    regression_pipeline,
    svm_regressor,
)
from simulation import generate_dataset
from visualization import (
    plot_dataset,
    plot_interval_metrics,
    plot_intervals,
    plot_regression_metrics,
)

2Generate synthetic data

settings = Settings(n_samples=5000, seed=0, svm_gamma=0.025)
data = generate_dataset(settings)
data.head()
Loading...
plot_dataset(data)
Loading...

3Split into train, calibration, and test sets

train, remainder = train_test_split(data, test_size=0.3, random_state=0)
calib, test = train_test_split(remainder, test_size=0.5, random_state=0)
split_data = build_split_dataset(
    TrainingData.validate(train),
    TrainingData.validate(calib),
    TrainingData.validate(test),
)
len(train), len(calib), len(test)
(3500, 750, 750)

4Build the composite estimator

Each model is wired through regression_pipeline(...) to inherit the polynomial + Fourier feature expansion. The SVM factory additionally wraps SVR behind a StandardScaler (an inner pipeline), so scaling happens after feature expansion and only for the model that needs it.

MultiRegressor is a sklearn-native composite (BaseEstimator + RegressorMixin + TransformerMixin). .transform(X) returns per-base predictions as a (n_samples, n_estimators) matrix, and the unfitted estimator specs stay introspectable via .estimators — which is what bootstrap_confidence_intervals and fit_conformal need to clone() per base.

random_forest_pipeline = regression_pipeline(random_forest_regressor(settings), settings)
random_forest_pipeline
Loading...
svm_pipeline = regression_pipeline(svm_regressor(settings), settings)
svm_pipeline
Loading...
regressors = MultiRegressor(
    estimators=[
        (ModelKind.RANDOM_FOREST.value, random_forest_pipeline),
        (ModelKind.SVM.value, svm_pipeline),
    ],
)
regressors
Loading...

5Fit, calibrate, score, and bootstrap in one call

compare_models walks each (name, pipeline) pair in the composite once and returns a ModelComparisonReport dataclass with six tagged DataFrames:

  • predictions — per-model point predictions with ground truth

  • confidence — bootstrap confidence intervals (refit-on-resample) per model

  • prediction — MAPIE split conformal prediction intervals per model

  • regression_metrics — RMSE/MAE/R² with bootstrap CIs, per model

  • confidence_metrics and prediction_metrics — interval width and MWI scores, per model

All concatenation and model-column tagging happens inside src/; the notebook stays declarative.

report = compare_models(split_data, regressors, settings)
report.predictions.head()
Bootstrap:   0%|          | 0/200 [00:00<?, ?it/s]
Bootstrap:   0%|          | 1/200 [00:01<03:19,  1.00s/it]
Bootstrap:   1%|          | 2/200 [00:02<03:19,  1.01s/it]
Bootstrap:   2%|▏         | 3/200 [00:03<03:17,  1.00s/it]
Bootstrap:   2%|▏         | 4/200 [00:04<03:17,  1.01s/it]
Bootstrap:   2%|▎         | 5/200 [00:05<03:18,  1.02s/it]
Bootstrap:   3%|▎         | 6/200 [00:06<03:16,  1.01s/it]
Bootstrap:   4%|▎         | 7/200 [00:07<03:14,  1.01s/it]
Bootstrap:   4%|▍         | 8/200 [00:08<03:13,  1.01s/it]
Bootstrap:   4%|▍         | 9/200 [00:09<03:11,  1.00s/it]
Bootstrap:   5%|▌         | 10/200 [00:10<03:10,  1.00s/it]
Bootstrap:   6%|▌         | 11/200 [00:11<03:09,  1.00s/it]
Bootstrap:   6%|▌         | 12/200 [00:12<03:08,  1.00s/it]
Bootstrap:   6%|▋         | 13/200 [00:13<03:07,  1.00s/it]
Bootstrap:   7%|▋         | 14/200 [00:14<03:05,  1.00it/s]
Bootstrap:   8%|▊         | 15/200 [00:15<03:04,  1.00it/s]
Bootstrap:   8%|▊         | 16/200 [00:16<03:03,  1.00it/s]
Bootstrap:   8%|▊         | 17/200 [00:17<03:02,  1.00it/s]
Bootstrap:   9%|▉         | 18/200 [00:18<03:02,  1.00s/it]
Bootstrap:  10%|▉         | 19/200 [00:19<03:00,  1.00it/s]
Bootstrap:  10%|█         | 20/200 [00:20<03:00,  1.00s/it]
Bootstrap:  10%|█         | 21/200 [00:21<02:59,  1.00s/it]
Bootstrap:  11%|█         | 22/200 [00:22<02:58,  1.00s/it]
Bootstrap:  12%|█▏        | 23/200 [00:23<02:57,  1.00s/it]
Bootstrap:  12%|█▏        | 24/200 [00:24<02:56,  1.00s/it]
Bootstrap:  12%|█▎        | 25/200 [00:25<02:54,  1.00it/s]
Bootstrap:  13%|█▎        | 26/200 [00:26<02:53,  1.00it/s]
Bootstrap:  14%|█▎        | 27/200 [00:27<02:52,  1.00it/s]
Bootstrap:  14%|█▍        | 28/200 [00:28<02:51,  1.00it/s]
Bootstrap:  14%|█▍        | 29/200 [00:29<02:51,  1.00s/it]
Bootstrap:  15%|█▌        | 30/200 [00:30<02:49,  1.00it/s]
Bootstrap:  16%|█▌        | 31/200 [00:31<02:48,  1.00it/s]
Bootstrap:  16%|█▌        | 32/200 [00:32<02:46,  1.01it/s]
Bootstrap:  16%|█▋        | 33/200 [00:32<02:45,  1.01it/s]
Bootstrap:  17%|█▋        | 34/200 [00:33<02:44,  1.01it/s]
Bootstrap:  18%|█▊        | 35/200 [00:34<02:43,  1.01it/s]
Bootstrap:  18%|█▊        | 36/200 [00:35<02:41,  1.01it/s]
Bootstrap:  18%|█▊        | 37/200 [00:36<02:40,  1.01it/s]
Bootstrap:  19%|█▉        | 38/200 [00:37<02:40,  1.01it/s]
Bootstrap:  20%|█▉        | 39/200 [00:38<02:38,  1.01it/s]
Bootstrap:  20%|██        | 40/200 [00:39<02:38,  1.01it/s]
Bootstrap:  20%|██        | 41/200 [00:40<02:37,  1.01it/s]
Bootstrap:  21%|██        | 42/200 [00:41<02:36,  1.01it/s]
Bootstrap:  22%|██▏       | 43/200 [00:42<02:35,  1.01it/s]
Bootstrap:  22%|██▏       | 44/200 [00:43<02:35,  1.00it/s]
Bootstrap:  22%|██▎       | 45/200 [00:44<02:35,  1.00s/it]
Bootstrap:  23%|██▎       | 46/200 [00:45<02:34,  1.00s/it]
Bootstrap:  24%|██▎       | 47/200 [00:46<02:33,  1.01s/it]
Bootstrap:  24%|██▍       | 48/200 [00:47<02:34,  1.01s/it]
Bootstrap:  24%|██▍       | 49/200 [00:48<02:32,  1.01s/it]
Bootstrap:  25%|██▌       | 50/200 [00:49<02:31,  1.01s/it]
Bootstrap:  26%|██▌       | 51/200 [00:50<02:29,  1.01s/it]
Bootstrap:  26%|██▌       | 52/200 [00:51<02:28,  1.00s/it]
Bootstrap:  26%|██▋       | 53/200 [00:52<02:27,  1.00s/it]
Bootstrap:  27%|██▋       | 54/200 [00:53<02:26,  1.00s/it]
Bootstrap:  28%|██▊       | 55/200 [00:54<02:25,  1.00s/it]
Bootstrap:  28%|██▊       | 56/200 [00:55<02:24,  1.00s/it]
Bootstrap:  28%|██▊       | 57/200 [00:56<02:23,  1.00s/it]
Bootstrap:  29%|██▉       | 58/200 [00:57<02:22,  1.00s/it]
Bootstrap:  30%|██▉       | 59/200 [00:58<02:20,  1.00it/s]
Bootstrap:  30%|███       | 60/200 [00:59<02:19,  1.00it/s]
Bootstrap:  30%|███       | 61/200 [01:00<02:18,  1.00it/s]
Bootstrap:  31%|███       | 62/200 [01:01<02:18,  1.00s/it]
Bootstrap:  32%|███▏      | 63/200 [01:02<02:17,  1.00s/it]
Bootstrap:  32%|███▏      | 64/200 [01:03<02:16,  1.00s/it]
Bootstrap:  32%|███▎      | 65/200 [01:04<02:14,  1.00it/s]
Bootstrap:  33%|███▎      | 66/200 [01:05<02:14,  1.00s/it]
Bootstrap:  34%|███▎      | 67/200 [01:06<02:12,  1.00it/s]
Bootstrap:  34%|███▍      | 68/200 [01:07<02:11,  1.00it/s]
Bootstrap:  34%|███▍      | 69/200 [01:08<02:10,  1.00it/s]
Bootstrap:  35%|███▌      | 70/200 [01:09<02:09,  1.00it/s]
Bootstrap:  36%|███▌      | 71/200 [01:10<02:08,  1.00it/s]
Bootstrap:  36%|███▌      | 72/200 [01:11<02:08,  1.00s/it]
Bootstrap:  36%|███▋      | 73/200 [01:12<02:06,  1.00it/s]
Bootstrap:  37%|███▋      | 74/200 [01:13<02:05,  1.00it/s]
Bootstrap:  38%|███▊      | 75/200 [01:14<02:04,  1.00it/s]
Bootstrap:  38%|███▊      | 76/200 [01:15<02:03,  1.00it/s]
Bootstrap:  38%|███▊      | 77/200 [01:16<02:01,  1.01it/s]
Bootstrap:  39%|███▉      | 78/200 [01:17<02:00,  1.01it/s]
Bootstrap:  40%|███▉      | 79/200 [01:18<01:59,  1.02it/s]
Bootstrap:  40%|████      | 80/200 [01:19<01:58,  1.02it/s]
Bootstrap:  40%|████      | 81/200 [01:20<01:56,  1.02it/s]
Bootstrap:  41%|████      | 82/200 [01:21<01:55,  1.02it/s]
Bootstrap:  42%|████▏     | 83/200 [01:22<01:55,  1.02it/s]
Bootstrap:  42%|████▏     | 84/200 [01:23<01:54,  1.02it/s]
Bootstrap:  42%|████▎     | 85/200 [01:24<01:52,  1.02it/s]
Bootstrap:  43%|████▎     | 86/200 [01:25<01:51,  1.02it/s]
Bootstrap:  44%|████▎     | 87/200 [01:26<01:51,  1.02it/s]
Bootstrap:  44%|████▍     | 88/200 [01:27<01:50,  1.01it/s]
Bootstrap:  44%|████▍     | 89/200 [01:28<01:49,  1.01it/s]
Bootstrap:  45%|████▌     | 90/200 [01:29<01:49,  1.01it/s]
Bootstrap:  46%|████▌     | 91/200 [01:30<01:48,  1.01it/s]
Bootstrap:  46%|████▌     | 92/200 [01:31<01:47,  1.00it/s]
Bootstrap:  46%|████▋     | 93/200 [01:32<01:46,  1.00it/s]
Bootstrap:  47%|████▋     | 94/200 [01:33<01:45,  1.00it/s]
Bootstrap:  48%|████▊     | 95/200 [01:34<01:44,  1.00it/s]
Bootstrap:  48%|████▊     | 96/200 [01:35<01:43,  1.00it/s]
Bootstrap:  48%|████▊     | 97/200 [01:36<01:42,  1.00it/s]
Bootstrap:  49%|████▉     | 98/200 [01:37<01:41,  1.00it/s]
Bootstrap:  50%|████▉     | 99/200 [01:38<01:40,  1.00it/s]
Bootstrap:  50%|█████     | 100/200 [01:39<01:39,  1.00it/s]
Bootstrap:  50%|█████     | 101/200 [01:40<01:38,  1.00it/s]
Bootstrap:  51%|█████     | 102/200 [01:41<01:37,  1.00it/s]
Bootstrap:  52%|█████▏    | 103/200 [01:42<01:36,  1.00it/s]
Bootstrap:  52%|█████▏    | 104/200 [01:43<01:35,  1.00it/s]
Bootstrap:  52%|█████▎    | 105/200 [01:44<01:34,  1.00it/s]
Bootstrap:  53%|█████▎    | 106/200 [01:45<01:33,  1.00it/s]
Bootstrap:  54%|█████▎    | 107/200 [01:46<01:32,  1.00it/s]
Bootstrap:  54%|█████▍    | 108/200 [01:47<01:31,  1.00it/s]
Bootstrap:  55%|█████▍    | 109/200 [01:48<01:30,  1.00it/s]
Bootstrap:  55%|█████▌    | 110/200 [01:49<01:29,  1.00it/s]
Bootstrap:  56%|█████▌    | 111/200 [01:50<01:28,  1.01it/s]
Bootstrap:  56%|█████▌    | 112/200 [01:51<01:27,  1.00it/s]
Bootstrap:  56%|█████▋    | 113/200 [01:52<01:26,  1.00it/s]
Bootstrap:  57%|█████▋    | 114/200 [01:53<01:25,  1.00it/s]
Bootstrap:  57%|█████▊    | 115/200 [01:54<01:24,  1.00it/s]
Bootstrap:  58%|█████▊    | 116/200 [01:55<01:23,  1.00it/s]
Bootstrap:  58%|█████▊    | 117/200 [01:56<01:22,  1.01it/s]
Bootstrap:  59%|█████▉    | 118/200 [01:57<01:21,  1.01it/s]
Bootstrap:  60%|█████▉    | 119/200 [01:58<01:20,  1.01it/s]
Bootstrap:  60%|██████    | 120/200 [01:59<01:19,  1.01it/s]
Bootstrap:  60%|██████    | 121/200 [02:00<01:18,  1.01it/s]
Bootstrap:  61%|██████    | 122/200 [02:01<01:17,  1.01it/s]
Bootstrap:  62%|██████▏   | 123/200 [02:02<01:16,  1.01it/s]
Bootstrap:  62%|██████▏   | 124/200 [02:03<01:15,  1.01it/s]
Bootstrap:  62%|██████▎   | 125/200 [02:04<01:14,  1.01it/s]
Bootstrap:  63%|██████▎   | 126/200 [02:05<01:13,  1.01it/s]
Bootstrap:  64%|██████▎   | 127/200 [02:06<01:12,  1.01it/s]
Bootstrap:  64%|██████▍   | 128/200 [02:07<01:11,  1.01it/s]
Bootstrap:  64%|██████▍   | 129/200 [02:08<01:10,  1.01it/s]
Bootstrap:  65%|██████▌   | 130/200 [02:09<01:09,  1.01it/s]
Bootstrap:  66%|██████▌   | 131/200 [02:10<01:08,  1.01it/s]
Bootstrap:  66%|██████▌   | 132/200 [02:11<01:07,  1.01it/s]
Bootstrap:  66%|██████▋   | 133/200 [02:12<01:06,  1.01it/s]
Bootstrap:  67%|██████▋   | 134/200 [02:13<01:05,  1.01it/s]
Bootstrap:  68%|██████▊   | 135/200 [02:14<01:04,  1.01it/s]
Bootstrap:  68%|██████▊   | 136/200 [02:15<01:03,  1.01it/s]
Bootstrap:  68%|██████▊   | 137/200 [02:16<01:02,  1.01it/s]
Bootstrap:  69%|██████▉   | 138/200 [02:17<01:01,  1.01it/s]
Bootstrap:  70%|██████▉   | 139/200 [02:18<01:00,  1.01it/s]
Bootstrap:  70%|███████   | 140/200 [02:19<00:59,  1.01it/s]
Bootstrap:  70%|███████   | 141/200 [02:20<00:58,  1.01it/s]
Bootstrap:  71%|███████   | 142/200 [02:21<00:57,  1.01it/s]
Bootstrap:  72%|███████▏  | 143/200 [02:22<00:56,  1.01it/s]
Bootstrap:  72%|███████▏  | 144/200 [02:23<00:55,  1.01it/s]
Bootstrap:  72%|███████▎  | 145/200 [02:24<00:54,  1.00it/s]
Bootstrap:  73%|███████▎  | 146/200 [02:25<00:53,  1.00it/s]
Bootstrap:  74%|███████▎  | 147/200 [02:26<00:52,  1.00it/s]
Bootstrap:  74%|███████▍  | 148/200 [02:27<00:51,  1.00it/s]
Bootstrap:  74%|███████▍  | 149/200 [02:28<00:50,  1.00it/s]
Bootstrap:  75%|███████▌  | 150/200 [02:29<00:49,  1.00it/s]
Bootstrap:  76%|███████▌  | 151/200 [02:30<00:48,  1.00it/s]
Bootstrap:  76%|███████▌  | 152/200 [02:31<00:47,  1.00it/s]
Bootstrap:  76%|███████▋  | 153/200 [02:32<00:46,  1.00it/s]
Bootstrap:  77%|███████▋  | 154/200 [02:33<00:45,  1.00it/s]
Bootstrap:  78%|███████▊  | 155/200 [02:34<00:45,  1.00s/it]
Bootstrap:  78%|███████▊  | 156/200 [02:35<00:43,  1.00it/s]
Bootstrap:  78%|███████▊  | 157/200 [02:36<00:42,  1.00it/s]
Bootstrap:  79%|███████▉  | 158/200 [02:37<00:41,  1.00it/s]
Bootstrap:  80%|███████▉  | 159/200 [02:38<00:40,  1.01it/s]
Bootstrap:  80%|████████  | 160/200 [02:39<00:39,  1.00it/s]
Bootstrap:  80%|████████  | 161/200 [02:40<00:38,  1.00it/s]
Bootstrap:  81%|████████  | 162/200 [02:41<00:37,  1.00it/s]
Bootstrap:  82%|████████▏ | 163/200 [02:42<00:36,  1.00it/s]
Bootstrap:  82%|████████▏ | 164/200 [02:43<00:35,  1.00it/s]
Bootstrap:  82%|████████▎ | 165/200 [02:44<00:34,  1.00it/s]
Bootstrap:  83%|████████▎ | 166/200 [02:45<00:33,  1.00it/s]
Bootstrap:  84%|████████▎ | 167/200 [02:46<00:32,  1.00it/s]
Bootstrap:  84%|████████▍ | 168/200 [02:47<00:32,  1.00s/it]
Bootstrap:  84%|████████▍ | 169/200 [02:48<00:30,  1.00it/s]
Bootstrap:  85%|████████▌ | 170/200 [02:49<00:29,  1.00it/s]
Bootstrap:  86%|████████▌ | 171/200 [02:50<00:28,  1.00it/s]
Bootstrap:  86%|████████▌ | 172/200 [02:51<00:27,  1.00it/s]
Bootstrap:  86%|████████▋ | 173/200 [02:52<00:26,  1.00it/s]
Bootstrap:  87%|████████▋ | 174/200 [02:53<00:25,  1.01it/s]
Bootstrap:  88%|████████▊ | 175/200 [02:54<00:24,  1.01it/s]
Bootstrap:  88%|████████▊ | 176/200 [02:55<00:23,  1.00it/s]
Bootstrap:  88%|████████▊ | 177/200 [02:56<00:22,  1.00it/s]
Bootstrap:  89%|████████▉ | 178/200 [02:57<00:21,  1.00it/s]
Bootstrap:  90%|████████▉ | 179/200 [02:58<00:20,  1.01it/s]
Bootstrap:  90%|█████████ | 180/200 [02:59<00:19,  1.00it/s]
Bootstrap:  90%|█████████ | 181/200 [03:00<00:18,  1.00it/s]
Bootstrap:  91%|█████████ | 182/200 [03:01<00:17,  1.00it/s]
Bootstrap:  92%|█████████▏| 183/200 [03:02<00:16,  1.00it/s]
Bootstrap:  92%|█████████▏| 184/200 [03:03<00:15,  1.01it/s]
Bootstrap:  92%|█████████▎| 185/200 [03:04<00:14,  1.00it/s]
Bootstrap:  93%|█████████▎| 186/200 [03:05<00:13,  1.01it/s]
Bootstrap:  94%|█████████▎| 187/200 [03:06<00:12,  1.01it/s]
Bootstrap:  94%|█████████▍| 188/200 [03:07<00:11,  1.01it/s]
Bootstrap:  94%|█████████▍| 189/200 [03:08<00:10,  1.01it/s]
Bootstrap:  95%|█████████▌| 190/200 [03:09<00:09,  1.01it/s]
Bootstrap:  96%|█████████▌| 191/200 [03:10<00:08,  1.00it/s]
Bootstrap:  96%|█████████▌| 192/200 [03:11<00:07,  1.00it/s]
Bootstrap:  96%|█████████▋| 193/200 [03:12<00:06,  1.00it/s]
Bootstrap:  97%|█████████▋| 194/200 [03:13<00:05,  1.00it/s]
Bootstrap:  98%|█████████▊| 195/200 [03:14<00:04,  1.00it/s]
Bootstrap:  98%|█████████▊| 196/200 [03:15<00:03,  1.01it/s]
Bootstrap:  98%|█████████▊| 197/200 [03:16<00:02,  1.00it/s]
Bootstrap:  99%|█████████▉| 198/200 [03:17<00:01,  1.00it/s]
Bootstrap: 100%|█████████▉| 199/200 [03:18<00:00,  1.00it/s]
Bootstrap: 100%|██████████| 200/200 [03:19<00:00,  1.00it/s]
Bootstrap: 100%|██████████| 200/200 [03:19<00:00,  1.00it/s]


Bootstrap:   0%|          | 0/200 [00:00<?, ?it/s]
Bootstrap:   0%|          | 1/200 [00:00<01:22,  2.41it/s]
Bootstrap:   1%|          | 2/200 [00:00<01:22,  2.39it/s]
Bootstrap:   2%|▏         | 3/200 [00:01<01:22,  2.40it/s]
Bootstrap:   2%|▏         | 4/200 [00:01<01:21,  2.40it/s]
Bootstrap:   2%|▎         | 5/200 [00:02<01:21,  2.39it/s]
Bootstrap:   3%|▎         | 6/200 [00:02<01:20,  2.40it/s]
Bootstrap:   4%|▎         | 7/200 [00:02<01:20,  2.40it/s]
Bootstrap:   4%|▍         | 8/200 [00:03<01:20,  2.40it/s]
Bootstrap:   4%|▍         | 9/200 [00:03<01:19,  2.40it/s]
Bootstrap:   5%|▌         | 10/200 [00:04<01:19,  2.40it/s]
Bootstrap:   6%|▌         | 11/200 [00:04<01:18,  2.40it/s]
Bootstrap:   6%|▌         | 12/200 [00:04<01:18,  2.40it/s]
Bootstrap:   6%|▋         | 13/200 [00:05<01:17,  2.40it/s]
Bootstrap:   7%|▋         | 14/200 [00:05<01:17,  2.40it/s]
Bootstrap:   8%|▊         | 15/200 [00:06<01:16,  2.41it/s]
Bootstrap:   8%|▊         | 16/200 [00:06<01:16,  2.41it/s]
Bootstrap:   8%|▊         | 17/200 [00:07<01:16,  2.41it/s]
Bootstrap:   9%|▉         | 18/200 [00:07<01:15,  2.41it/s]
Bootstrap:  10%|▉         | 19/200 [00:07<01:15,  2.40it/s]
Bootstrap:  10%|█         | 20/200 [00:08<01:14,  2.40it/s]
Bootstrap:  10%|█         | 21/200 [00:08<01:14,  2.40it/s]
Bootstrap:  11%|█         | 22/200 [00:09<01:14,  2.41it/s]
Bootstrap:  12%|█▏        | 23/200 [00:09<01:13,  2.40it/s]
Bootstrap:  12%|█▏        | 24/200 [00:09<01:13,  2.40it/s]
Bootstrap:  12%|█▎        | 25/200 [00:10<01:12,  2.40it/s]
Bootstrap:  13%|█▎        | 26/200 [00:10<01:12,  2.39it/s]
Bootstrap:  14%|█▎        | 27/200 [00:11<01:12,  2.38it/s]
Bootstrap:  14%|█▍        | 28/200 [00:11<01:11,  2.39it/s]
Bootstrap:  14%|█▍        | 29/200 [00:12<01:11,  2.40it/s]
Bootstrap:  15%|█▌        | 30/200 [00:12<01:10,  2.39it/s]
Bootstrap:  16%|█▌        | 31/200 [00:12<01:10,  2.40it/s]
Bootstrap:  16%|█▌        | 32/200 [00:13<01:10,  2.40it/s]
Bootstrap:  16%|█▋        | 33/200 [00:13<01:09,  2.40it/s]
Bootstrap:  17%|█▋        | 34/200 [00:14<01:09,  2.41it/s]
Bootstrap:  18%|█▊        | 35/200 [00:14<01:08,  2.41it/s]
Bootstrap:  18%|█▊        | 36/200 [00:14<01:08,  2.41it/s]
Bootstrap:  18%|█▊        | 37/200 [00:15<01:07,  2.41it/s]
Bootstrap:  19%|█▉        | 38/200 [00:15<01:07,  2.40it/s]
Bootstrap:  20%|█▉        | 39/200 [00:16<01:06,  2.40it/s]
Bootstrap:  20%|██        | 40/200 [00:16<01:06,  2.41it/s]
Bootstrap:  20%|██        | 41/200 [00:17<01:06,  2.40it/s]
Bootstrap:  21%|██        | 42/200 [00:17<01:05,  2.40it/s]
Bootstrap:  22%|██▏       | 43/200 [00:17<01:05,  2.40it/s]
Bootstrap:  22%|██▏       | 44/200 [00:18<01:04,  2.40it/s]
Bootstrap:  22%|██▎       | 45/200 [00:18<01:04,  2.40it/s]
Bootstrap:  23%|██▎       | 46/200 [00:19<01:04,  2.40it/s]
Bootstrap:  24%|██▎       | 47/200 [00:19<01:03,  2.40it/s]
Bootstrap:  24%|██▍       | 48/200 [00:19<01:03,  2.40it/s]
Bootstrap:  24%|██▍       | 49/200 [00:20<01:02,  2.40it/s]
Bootstrap:  25%|██▌       | 50/200 [00:20<01:02,  2.40it/s]
Bootstrap:  26%|██▌       | 51/200 [00:21<01:01,  2.41it/s]
Bootstrap:  26%|██▌       | 52/200 [00:21<01:01,  2.41it/s]
Bootstrap:  26%|██▋       | 53/200 [00:22<01:01,  2.41it/s]
Bootstrap:  27%|██▋       | 54/200 [00:22<01:00,  2.41it/s]
Bootstrap:  28%|██▊       | 55/200 [00:22<01:00,  2.41it/s]
Bootstrap:  28%|██▊       | 56/200 [00:23<00:59,  2.41it/s]
Bootstrap:  28%|██▊       | 57/200 [00:23<00:59,  2.41it/s]
Bootstrap:  29%|██▉       | 58/200 [00:24<00:58,  2.41it/s]
Bootstrap:  30%|██▉       | 59/200 [00:24<00:58,  2.40it/s]
Bootstrap:  30%|███       | 60/200 [00:24<00:58,  2.41it/s]
Bootstrap:  30%|███       | 61/200 [00:25<00:58,  2.39it/s]
Bootstrap:  31%|███       | 62/200 [00:25<00:57,  2.40it/s]
Bootstrap:  32%|███▏      | 63/200 [00:26<00:57,  2.40it/s]
Bootstrap:  32%|███▏      | 64/200 [00:26<00:56,  2.40it/s]
Bootstrap:  32%|███▎      | 65/200 [00:27<00:56,  2.41it/s]
Bootstrap:  33%|███▎      | 66/200 [00:27<00:55,  2.41it/s]
Bootstrap:  34%|███▎      | 67/200 [00:27<00:55,  2.41it/s]
Bootstrap:  34%|███▍      | 68/200 [00:28<00:54,  2.40it/s]
Bootstrap:  34%|███▍      | 69/200 [00:28<00:54,  2.41it/s]
Bootstrap:  35%|███▌      | 70/200 [00:29<00:53,  2.41it/s]
Bootstrap:  36%|███▌      | 71/200 [00:29<00:53,  2.41it/s]
Bootstrap:  36%|███▌      | 72/200 [00:29<00:53,  2.41it/s]
Bootstrap:  36%|███▋      | 73/200 [00:30<00:52,  2.41it/s]
Bootstrap:  37%|███▋      | 74/200 [00:30<00:52,  2.41it/s]
Bootstrap:  38%|███▊      | 75/200 [00:31<00:51,  2.41it/s]
Bootstrap:  38%|███▊      | 76/200 [00:31<00:51,  2.41it/s]
Bootstrap:  38%|███▊      | 77/200 [00:32<00:51,  2.40it/s]
Bootstrap:  39%|███▉      | 78/200 [00:32<00:50,  2.40it/s]
Bootstrap:  40%|███▉      | 79/200 [00:32<00:50,  2.41it/s]
Bootstrap:  40%|████      | 80/200 [00:33<00:49,  2.41it/s]
Bootstrap:  40%|████      | 81/200 [00:33<00:49,  2.41it/s]
Bootstrap:  41%|████      | 82/200 [00:34<00:48,  2.42it/s]
Bootstrap:  42%|████▏     | 83/200 [00:34<00:48,  2.42it/s]
Bootstrap:  42%|████▏     | 84/200 [00:34<00:48,  2.41it/s]
Bootstrap:  42%|████▎     | 85/200 [00:35<00:47,  2.41it/s]
Bootstrap:  43%|████▎     | 86/200 [00:35<00:47,  2.40it/s]
Bootstrap:  44%|████▎     | 87/200 [00:36<00:46,  2.41it/s]
Bootstrap:  44%|████▍     | 88/200 [00:36<00:46,  2.41it/s]
Bootstrap:  44%|████▍     | 89/200 [00:37<00:46,  2.41it/s]
Bootstrap:  45%|████▌     | 90/200 [00:37<00:45,  2.41it/s]
Bootstrap:  46%|████▌     | 91/200 [00:37<00:45,  2.40it/s]
Bootstrap:  46%|████▌     | 92/200 [00:38<00:45,  2.40it/s]
Bootstrap:  46%|████▋     | 93/200 [00:38<00:44,  2.40it/s]
Bootstrap:  47%|████▋     | 94/200 [00:39<00:44,  2.41it/s]
Bootstrap:  48%|████▊     | 95/200 [00:39<00:43,  2.41it/s]
Bootstrap:  48%|████▊     | 96/200 [00:39<00:43,  2.41it/s]
Bootstrap:  48%|████▊     | 97/200 [00:40<00:42,  2.41it/s]
Bootstrap:  49%|████▉     | 98/200 [00:40<00:42,  2.41it/s]
Bootstrap:  50%|████▉     | 99/200 [00:41<00:41,  2.41it/s]
Bootstrap:  50%|█████     | 100/200 [00:41<00:41,  2.41it/s]
Bootstrap:  50%|█████     | 101/200 [00:42<00:41,  2.40it/s]
Bootstrap:  51%|█████     | 102/200 [00:42<00:41,  2.36it/s]
Bootstrap:  52%|█████▏    | 103/200 [00:42<00:40,  2.38it/s]
Bootstrap:  52%|█████▏    | 104/200 [00:43<00:40,  2.39it/s]
Bootstrap:  52%|█████▎    | 105/200 [00:43<00:39,  2.39it/s]
Bootstrap:  53%|█████▎    | 106/200 [00:44<00:39,  2.39it/s]
Bootstrap:  54%|█████▎    | 107/200 [00:44<00:38,  2.40it/s]
Bootstrap:  54%|█████▍    | 108/200 [00:44<00:38,  2.40it/s]
Bootstrap:  55%|█████▍    | 109/200 [00:45<00:37,  2.40it/s]
Bootstrap:  55%|█████▌    | 110/200 [00:45<00:37,  2.40it/s]
Bootstrap:  56%|█████▌    | 111/200 [00:46<00:37,  2.40it/s]
Bootstrap:  56%|█████▌    | 112/200 [00:46<00:36,  2.40it/s]
Bootstrap:  56%|█████▋    | 113/200 [00:47<00:36,  2.41it/s]
Bootstrap:  57%|█████▋    | 114/200 [00:47<00:35,  2.41it/s]
Bootstrap:  57%|█████▊    | 115/200 [00:47<00:35,  2.41it/s]
Bootstrap:  58%|█████▊    | 116/200 [00:48<00:34,  2.40it/s]
Bootstrap:  58%|█████▊    | 117/200 [00:48<00:34,  2.40it/s]
Bootstrap:  59%|█████▉    | 118/200 [00:49<00:33,  2.41it/s]
Bootstrap:  60%|█████▉    | 119/200 [00:49<00:33,  2.41it/s]
Bootstrap:  60%|██████    | 120/200 [00:49<00:33,  2.41it/s]
Bootstrap:  60%|██████    | 121/200 [00:50<00:33,  2.39it/s]
Bootstrap:  61%|██████    | 122/200 [00:50<00:32,  2.37it/s]
Bootstrap:  62%|██████▏   | 123/200 [00:51<00:32,  2.39it/s]
Bootstrap:  62%|██████▏   | 124/200 [00:51<00:31,  2.40it/s]
Bootstrap:  62%|██████▎   | 125/200 [00:52<00:31,  2.40it/s]
Bootstrap:  63%|██████▎   | 126/200 [00:52<00:31,  2.38it/s]
Bootstrap:  64%|██████▎   | 127/200 [00:52<00:30,  2.37it/s]
Bootstrap:  64%|██████▍   | 128/200 [00:53<00:30,  2.37it/s]
Bootstrap:  64%|██████▍   | 129/200 [00:53<00:30,  2.36it/s]
Bootstrap:  65%|██████▌   | 130/200 [00:54<00:29,  2.36it/s]
Bootstrap:  66%|██████▌   | 131/200 [00:54<00:29,  2.37it/s]
Bootstrap:  66%|██████▌   | 132/200 [00:54<00:28,  2.37it/s]
Bootstrap:  66%|██████▋   | 133/200 [00:55<00:28,  2.35it/s]
Bootstrap:  67%|██████▋   | 134/200 [00:55<00:28,  2.34it/s]
Bootstrap:  68%|██████▊   | 135/200 [00:56<00:27,  2.34it/s]
Bootstrap:  68%|██████▊   | 136/200 [00:56<00:27,  2.35it/s]
Bootstrap:  68%|██████▊   | 137/200 [00:57<00:26,  2.34it/s]
Bootstrap:  69%|██████▉   | 138/200 [00:57<00:26,  2.35it/s]
Bootstrap:  70%|██████▉   | 139/200 [00:57<00:25,  2.35it/s]
Bootstrap:  70%|███████   | 140/200 [00:58<00:25,  2.35it/s]
Bootstrap:  70%|███████   | 141/200 [00:58<00:25,  2.35it/s]
Bootstrap:  71%|███████   | 142/200 [00:59<00:24,  2.36it/s]
Bootstrap:  72%|███████▏  | 143/200 [00:59<00:24,  2.36it/s]
Bootstrap:  72%|███████▏  | 144/200 [01:00<00:23,  2.35it/s]
Bootstrap:  72%|███████▎  | 145/200 [01:00<00:23,  2.35it/s]
Bootstrap:  73%|███████▎  | 146/200 [01:00<00:22,  2.35it/s]
Bootstrap:  74%|███████▎  | 147/200 [01:01<00:22,  2.35it/s]
Bootstrap:  74%|███████▍  | 148/200 [01:01<00:22,  2.35it/s]
Bootstrap:  74%|███████▍  | 149/200 [01:02<00:21,  2.35it/s]
Bootstrap:  75%|███████▌  | 150/200 [01:02<00:21,  2.37it/s]
Bootstrap:  76%|███████▌  | 151/200 [01:03<00:20,  2.38it/s]
Bootstrap:  76%|███████▌  | 152/200 [01:03<00:20,  2.39it/s]
Bootstrap:  76%|███████▋  | 153/200 [01:03<00:19,  2.38it/s]
Bootstrap:  77%|███████▋  | 154/200 [01:04<00:19,  2.38it/s]
Bootstrap:  78%|███████▊  | 155/200 [01:04<00:18,  2.39it/s]
Bootstrap:  78%|███████▊  | 156/200 [01:05<00:18,  2.40it/s]
Bootstrap:  78%|███████▊  | 157/200 [01:05<00:17,  2.40it/s]
Bootstrap:  79%|███████▉  | 158/200 [01:05<00:17,  2.41it/s]
Bootstrap:  80%|███████▉  | 159/200 [01:06<00:17,  2.41it/s]
Bootstrap:  80%|████████  | 160/200 [01:06<00:16,  2.41it/s]
Bootstrap:  80%|████████  | 161/200 [01:07<00:16,  2.42it/s]
Bootstrap:  81%|████████  | 162/200 [01:07<00:15,  2.42it/s]
Bootstrap:  82%|████████▏ | 163/200 [01:08<00:15,  2.41it/s]
Bootstrap:  82%|████████▏ | 164/200 [01:08<00:14,  2.41it/s]
Bootstrap:  82%|████████▎ | 165/200 [01:08<00:14,  2.42it/s]
Bootstrap:  83%|████████▎ | 166/200 [01:09<00:14,  2.41it/s]
Bootstrap:  84%|████████▎ | 167/200 [01:09<00:13,  2.41it/s]
Bootstrap:  84%|████████▍ | 168/200 [01:10<00:13,  2.41it/s]
Bootstrap:  84%|████████▍ | 169/200 [01:10<00:12,  2.41it/s]
Bootstrap:  85%|████████▌ | 170/200 [01:10<00:12,  2.41it/s]
Bootstrap:  86%|████████▌ | 171/200 [01:11<00:12,  2.41it/s]
Bootstrap:  86%|████████▌ | 172/200 [01:11<00:11,  2.40it/s]
Bootstrap:  86%|████████▋ | 173/200 [01:12<00:11,  2.40it/s]
Bootstrap:  87%|████████▋ | 174/200 [01:12<00:10,  2.40it/s]
Bootstrap:  88%|████████▊ | 175/200 [01:13<00:10,  2.41it/s]
Bootstrap:  88%|████████▊ | 176/200 [01:13<00:09,  2.40it/s]
Bootstrap:  88%|████████▊ | 177/200 [01:13<00:09,  2.41it/s]
Bootstrap:  89%|████████▉ | 178/200 [01:14<00:09,  2.41it/s]
Bootstrap:  90%|████████▉ | 179/200 [01:14<00:08,  2.40it/s]
Bootstrap:  90%|█████████ | 180/200 [01:15<00:08,  2.41it/s]
Bootstrap:  90%|█████████ | 181/200 [01:15<00:07,  2.41it/s]
Bootstrap:  91%|█████████ | 182/200 [01:15<00:07,  2.41it/s]
Bootstrap:  92%|█████████▏| 183/200 [01:16<00:07,  2.41it/s]
Bootstrap:  92%|█████████▏| 184/200 [01:16<00:06,  2.41it/s]
Bootstrap:  92%|█████████▎| 185/200 [01:17<00:06,  2.41it/s]
Bootstrap:  93%|█████████▎| 186/200 [01:17<00:05,  2.42it/s]
Bootstrap:  94%|█████████▎| 187/200 [01:18<00:05,  2.42it/s]
Bootstrap:  94%|█████████▍| 188/200 [01:18<00:04,  2.41it/s]
Bootstrap:  94%|█████████▍| 189/200 [01:18<00:04,  2.41it/s]
Bootstrap:  95%|█████████▌| 190/200 [01:19<00:04,  2.41it/s]
Bootstrap:  96%|█████████▌| 191/200 [01:19<00:03,  2.40it/s]
Bootstrap:  96%|█████████▌| 192/200 [01:20<00:03,  2.40it/s]
Bootstrap:  96%|█████████▋| 193/200 [01:20<00:02,  2.40it/s]
Bootstrap:  97%|█████████▋| 194/200 [01:20<00:02,  2.41it/s]
Bootstrap:  98%|█████████▊| 195/200 [01:21<00:02,  2.41it/s]
Bootstrap:  98%|█████████▊| 196/200 [01:21<00:01,  2.41it/s]
Bootstrap:  98%|█████████▊| 197/200 [01:22<00:01,  2.41it/s]
Bootstrap:  99%|█████████▉| 198/200 [01:22<00:00,  2.41it/s]
Bootstrap: 100%|█████████▉| 199/200 [01:22<00:00,  2.41it/s]
Bootstrap: 100%|██████████| 200/200 [01:23<00:00,  2.41it/s]
Bootstrap: 100%|██████████| 200/200 [01:23<00:00,  2.40it/s]
Loading...

6Visualize the comparison

plot_intervals renders one small-multiples panel per model: the data scatter, the bootstrap confidence band, the conformal prediction band, and the regression line.

plot_intervals(data, report)
Loading...

7Pipeline evaluation

plot_regression_metrics facets by metric so each metric (RMSE, MAE, R²) gets its own y-axis — comparing models across metrics on a shared scale would be misleading because the metrics live in different units. Models sit on the x-axis within each facet, and each error bar shows the bootstrap CI.

plot_interval_metrics facets by interval kind (confidence vs. prediction). Within each facet, the metrics (width, MWI) are grouped on the x-axis and the models are color-coded — so the chart compares confidence-against-confidence and prediction-against-prediction, not CI-against-PI.

plot_regression_metrics(report)
Loading...
plot_interval_metrics(report)
Loading...
report.regression_metrics
Loading...
report.confidence_metrics
Loading...
report.prediction_metrics
Loading...

8Tests

Each module under src/ has a matching test module under tests/. Run the full suite with:

uv run poe test

CI runs tests before building the book (uv run poe ci).

Markdown Cheat Sheet
Jupyter Based Content