{ "cells": [ { "cell_type": "code", "execution_count": 1, "id": "4b1e5503", "metadata": { "execution": { "iopub.execute_input": "2026-05-11T22:14:06.145265Z", "iopub.status.busy": "2026-05-11T22:14:06.145198Z", "iopub.status.idle": "2026-05-11T22:14:06.162629Z", "shell.execute_reply": "2026-05-11T22:14:06.162253Z" }, "tags": [ "remove-cell" ] }, "outputs": [], "source": [ "%load_ext autoreload\n", "%autoreload 2\n", "\n", "import warnings\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "warnings.simplefilter(\"ignore\")" ] }, { "cell_type": "markdown", "id": "5ef6be29", "metadata": {}, "source": [ "# Monitoring training\n", "\n", "Real datasets can span millions of points across thousands of features. Fitting Glass Box UMAP at that scale takes real time, and in such cases you'll want to observe how training is proceeding. This guide shows you how to monitor training progress and keep a record of each fit on disk.\n", "\n", "## Automated logging with Tensorboard\n", "\n", "Under the hood, Glass Box UMAP trains its encoder with [PyTorch Lightning](https://lightning.ai/), and every fit is automatically logged with [TensorBoard](https://www.tensorflow.org/tensorboard) to a temporary directory. To persist these logs, just pass an explicit `checkpoint_dir` to {class}`GlassBoxUMAP `:" ] }, { "cell_type": "code", "execution_count": 2, "id": "df5745b0", "metadata": { "execution": { "iopub.execute_input": "2026-05-11T22:14:06.164093Z", "iopub.status.busy": "2026-05-11T22:14:06.164025Z", "iopub.status.idle": "2026-05-11T22:14:54.116599Z", "shell.execute_reply": "2026-05-11T22:14:54.116095Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "\u001b[1;36mruns/\u001b[0m\r\n", "├── \u001b[1;36mcheckpoints\u001b[0m\r\n", "│   └── best.ckpt\r\n", "└── \u001b[1;36mlogs\u001b[0m\r\n", " ├── events.out.tfevents.1778537655.evans-Apple-MacBook-Pro.71219.0\r\n", " └── hparams.yaml\r\n", "\r\n", "3 directories, 3 files\r\n" ] } ], "source": [ "from pathlib import Path\n", "import shutil\n", "\n", "from sklearn.datasets import load_digits\n", "from sklearn.preprocessing import StandardScaler\n", "from glass_box_umap import GlassBoxUMAP\n", "\n", "# Store logs to ./runs/\n", "checkpoint_dir = Path.cwd() / \"runs\"\n", "\n", "# The directory can exist, but needn't. We remove it for a fresh start.\n", "shutil.rmtree(checkpoint_dir)\n", "\n", "embedder = GlassBoxUMAP(\n", " random_state=0,\n", " checkpoint_dir=checkpoint_dir,\n", " quiet=True,\n", ")\n", "\n", "X, y = load_digits(return_X_y=True)\n", "X = StandardScaler().fit_transform(X)\n", "embedder.fit(X)\n", "\n", "!tree runs/" ] }, { "cell_type": "markdown", "id": "76727438", "metadata": {}, "source": [ "- `checkpoints/best.ckpt` is the same checkpoint that `restore_best_weights` (default `True`) reloads at the end of training, so you don't normally need to touch it. It stays on disk in case you want to inspect or reload a specific run later.\n", "- `logs/events.out.tfevents.…` is the TensorBoard event file." ] }, { "cell_type": "markdown", "id": "c9aac56a", "metadata": {}, "source": [ "The TensorBoard events file isn't human-readable, but can be viewed using a tensoboard server:\n", "\n", "```bash\n", "tensorboard --logdir runs/\n", "```\n", "\n", ":::{note}\n", "`tensorboard` ships as a dependency of `glass-box-umap`, so nothing extra needs to be installed. From the project root:\n", ":::\n", "\n", "That starts a server (default `http://localhost:6006`) which auto-discovers every event file under `runs/`. Leave it running while you train and it will poll the directory and refresh logged data as new events are written, allowing you to watch a fit in progress." ] }, { "cell_type": "markdown", "id": "5a7b151b", "metadata": {}, "source": [ "## Visualizing embedding evolution during training\n", "\n", "As an alternative diagnostic, Glass Box UMAP exposes a {class}`LiveEmbeddingCallback ` that streams the embedding itself to a Bokeh server in your browser. After each training epoch, it runs `transform` on a slice of `X` and pushes the new 2D coordinates to the page, where a slider lets you scrub back through every epoch, a play button replays the trajectory, and a save button writes a self-contained HTML snapshot of the run.\n", "\n", ":::{warning}\n", "Running `transform` after every epoch is not free. On large datasets the extra forward pass per epoch will noticeably slow training, so pass a representative subsample to the callback (as above) rather than the full `X`. When all you need is the loss curve, prefer TensorBoard.\n", ":::\n", "\n", "This live embedding offers diagnostic insight into how the learned manifold is forming. You can watch the geometry settle (or fail to) and catch a misbehaving run within a few epochs instead of waiting for the training to complete.\n", "\n", "It plugs in through `extra_callbacks`:\n", "\n", ":::{admonition} Install the plotting extras\n", ":class: tip dropdown\n", "\n", "`glass_box_umap.plotting` is an optional dependency that's required for this feature. It can be installed like so:\n", "\n", "```bash\n", "pip install \"glass-box-umap[plotting]\"\n", "# or\n", "uv pip install \"glass-box-umap[plotting]\"\n", "```\n", ":::\n", "\n", ":::{admonition} LiveEmbeddingCallback API\n", ":class: api, dropdown\n", "\n", "From the {class}`API docs `:\n", "\n", "```{eval-rst}\n", ".. automethod:: glass_box_umap.plotting.LiveEmbeddingCallback\n", " :noindex:\n", "```\n", ":::" ] }, { "cell_type": "code", "execution_count": 3, "id": "d31f8253", "metadata": { "execution": { "iopub.execute_input": "2026-05-11T22:14:54.118210Z", "iopub.status.busy": "2026-05-11T22:14:54.118101Z", "iopub.status.idle": "2026-05-11T22:15:45.895213Z", "shell.execute_reply": "2026-05-11T22:15:45.894766Z" } }, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Live embedding serving at http://localhost:65267/\n" ] }, { "name": "stdout", "output_type": "stream", "text": [ "Training done. Server still serving at http://localhost:65267/.\n" ] } ], "source": [ "from glass_box_umap.plotting import LiveEmbeddingCallback\n", "\n", "# Create the embedder\n", "embedder = GlassBoxUMAP(\n", " random_state=0,\n", " quiet=True\n", ")\n", "\n", "# Now create the callback, passing embedder.transform\n", "labels = [str(idx) for idx in y]\n", "callback = LiveEmbeddingCallback(\n", " transform_fn=embedder.transform,\n", " X=X[:500],\n", " labels=labels[:500],\n", " block_after_fit=False,\n", ")\n", "\n", "# Append the callback to `extra_callbacks`\n", "embedder.extra_callbacks.append(callback)\n", "\n", "_ = embedder.fit(X)" ] }, { "cell_type": "markdown", "id": "15a81b8b", "metadata": {}, "source": [ ":::{note}\n", "The above code will open an interface in your browser, updating after each epoch. For your convenience, we replicate the interface below.\n", ":::" ] }, { "cell_type": "code", "execution_count": 4, "id": "e7ab5fce", "metadata": { "execution": { "iopub.execute_input": "2026-05-11T22:15:45.896413Z", "iopub.status.busy": "2026-05-11T22:15:45.896325Z", "iopub.status.idle": "2026-05-11T22:15:45.942338Z", "shell.execute_reply": "2026-05-11T22:15:45.941947Z" }, "tags": [ "remove-input" ] }, "outputs": [ { "data": { "text/html": [ "" ], "text/plain": [ "" ] }, "execution_count": 4, "metadata": {}, "output_type": "execute_result" } ], "source": [ "from html import escape\n", "from pathlib import Path\n", "\n", "from IPython.display import HTML\n", "\n", "artifact = Path(\"../_static/monitoring_training_example.html\")\n", "\n", "embedded = HTML(\n", " f''\n", ")\n", "embedded" ] }, { "cell_type": "markdown", "id": "5e66cf3a", "metadata": {}, "source": [ "Hit \"Play\" to observe the embedding evolve throughout the training." ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.13.1" } }, "nbformat": 4, "nbformat_minor": 5 }