hypertower/scripts/main/pipeline.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Hypertower Repro Pipeline\n",
    "\n",
    "This notebook documents the full run sequence used to reproduce current results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 0) Environment + Paths\n",
    "\n",
    "- Activate `fundus_imaging` environment\n",
    "- Run from repo root\n",
    "- Confirm data paths:\n",
    "  - `Papila/FundusImages`\n",
    "  - `Papila/ClinicalData`\n",
    "  - `Papila/ExpertsSegmentations`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pathlib import Path\n",
    "\n",
    "required = [\n",
    "    Path(\"Papila/FundusImages\"),\n",
    "    Path(\"Papila/ClinicalData\"),\n",
    "    Path(\"Papila/ExpertsSegmentations\"),\n",
    "    Path(\"REFUGE\"),\n",
    "]\n",
    "for p in required:\n",
    "    print(f\"{p}:\", \"OK\" if p.exists() else \"MISSING\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1) Build UNet Manifest"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 scripts/main/refuge/build_manifest.py --output manifest.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2) Train UNet Segmenter (per-image normalization)\n",
    "\n",
    "Current tuned baseline:\n",
    "- `--device cuda`\n",
    "- `--batch-size 8`\n",
    "- `--loader-workers 14`\n",
    "- `--in-memory-cache`\n",
    "- `--cache-workers 4`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 scripts/run_unet_segmenter.py \\\n",
    "  --manifest manifest.csv \\\n",
    "  --train --evaluate \\\n",
    "  --normalize per_image \\\n",
    "  --train-datasets refuge --val-datasets refuge --holdout-datasets refuge \\\n",
    "  --epochs 40 --batch-size 8 \\\n",
    "  --device cuda --loader-workers 14 \\\n",
    "  --in-memory-cache --cache-workers 4 \\\n",
    "  --checkpoint-dir models/v2/refuge/segmentation/per_image \\\n",
    "  --eval-output analysis_data/segmenter_eval/v2_refuge_per_image \\\n",
    "  --eval-metrics-path analysis_data/segmenter_eval/v2_refuge_per_image/metrics.csv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3) Run V2 Hypertower Modes (cropped with UNet)\n",
    "\n",
    "Runs binary + multiclass across:\n",
    "- `single`\n",
    "- `ensemble`\n",
    "- `bilateral`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!python3 scripts/basic_analysis/compare_hypertower_modes.py \\\n",
    "  --eval-modes binary multiclass \\\n",
    "  --tower-modes single ensemble bilateral \\\n",
    "  --epochs 40 \\\n",
    "  --n-splits 5 \\\n",
    "  --batch-size 8 \\\n",
    "  --backbone refugelike \\\n",
    "  --img-crop-manifest manifest.csv \\\n",
    "  --img-crop-weights models/v2/refuge/segmentation/per_image/best.pt \\\n",
    "  --img-crop-normalize per_image \\\n",
    "  --img-crop-cache analysis_data/v2_crops_unet_refuge \\\n",
    "  --run-name v2_modes_full_40ep_5fold_unet_perimage"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4) Quick Result Snapshot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "from pathlib import Path\n",
    "\n",
    "root = Path(\"analysis_data/v2_modes_full_40ep_5fold_unet_perimage\")\n",
    "summary = root / \"summary.json\"\n",
    "if summary.exists():\n",
    "    data = json.loads(summary.read_text())\n",
    "    print(\"run_name:\", data.get(\"run_name\"))\n",
    "    print(\"timestamp:\", data.get(\"timestamp\"))\n",
    "    print(\"keys:\", list(data.get(\"summaries\", {}).keys()))\n",
    "else:\n",
    "    print(\"Summary not found:\", summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5) Notes / Decisions\n",
    "\n",
    "- Mixed-label patient handling used:\n",
    "- Warmup settings used:\n",
    "- Backbone / batch / workers used:\n",
    "- Any deviations from default run:"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "name": "python",
   "version": "3.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}