patient_leakage_detection/plan.md

# Implementation Plan — Potter et al. Data Leakage Reproduction

## Manuscript pipeline (from docx)

1. **Feature extraction** — 5 CNNs (VGG16, DenseNet121, EfficientNetB1, MobileNetV2, ResNet50), frozen, include_top=False
2. **PCA + t-SNE** — visualize feature space (Figure 5 / S2)
3. **Patient clustering** — K-means on 64×64 grayscale thumbnails, per-class (15/40/55)
4. **Random Forest** — feature importance ranking (2000 trees → we use 500, equivalent)
5. **SVM with GridSearchCV** — grid over γ per nfeatures, C=10 fixed, 5-fold CV
6. **Image-level vs patient-level** — compare test accuracy with both split types
7. **20-seed repetition** — boxplots showing distribution (Figure 6 / S4)

## Current status

### ✅ Implemented & Working

| Component | Location | Notes |
|---|---|---|
| Feature extraction (5 models, PyTorch) | `classes/features.py` | ResNet50 GAP stripped (100K-d). EfficientNetB1 dim error documented |
| Patient clustering (K-means) | `classes/patient_identifier.py` | Thumbnail + feature-based modes |
| Classification pipeline | `classes/classifier.py` | `PatientLeakageClassifier.run()` — RF→GridSearchCV→SVM |
| t-SNE visualization | `scripts/visualizations/simple_patient_tsne.py` | Per-class patient coloring |
| Figure 6 (20-seed boxplots) | `scripts/visualizations/figure6.py` | Image vs patient distributions |
| Figure S4 (CV accuracy curves) | `scripts/visualizations/figure_s4.py` | Per-model, log₁₀ x-scale, patient-level |
| C×γ grid search | archived | Confirmed C=10 is adequate |
| SVM class-weight analysis | archived | Confirmed no benefit |
| Task06 NIfTI dataset loader | `classes/nifti_dataset.py` | HU windowing, orientation control |
| Clustering validation (Task06) | `scripts/validate_clustering_task06.py` | Thumbnail + feature K-means vs ground truth |
| Clustering comparison (IQ-OTH) | `scripts/analysis/compare_clustering_methods.py` | Thumbnail vs feature vs v8 reference |
| Siamese patient matching | `classes/siamese.py` | `SiamesePatientMatcher` class |
| Siamese training | `scripts/train_siamese.py` | Task06 train/test split, hard negatives |

### 🚧 In Progress

| Component | Status |
|---|---|
| Siamese-based patient manifest for IQ-OTH | `scripts/siamese_identify.py` written, needs trained model |

### ❌ Still Needed

| Manuscript Figure/Table | What we need | Priority |
|---|---|---|
| **Figure 5** (PCA + t-SNE) | Per-model t-SNE plots, matching manuscript style | Medium |
| **Figure S1** (image-level CV curves) | 5 panels, image-level split, log₁₀ x-scale. Nearly identical to S4 but with image split | Low |
| **Figure S2** (PCA/t-SNE per model) | Like Figure 5 but for all 5 models | Medium |
| **Figure S3** (example cluster images) | 5 example slices from a single K-means cluster per class | Low |
| **Figure S5** (confusion matrix) | VGG16 patient-level confusion matrix | Low |
| **Table 2 comparison** | Run classification with thumbnail K-means manifest to match manuscript numbers | High |
| **Siamese results integration** | Once trained: run siamese_identify, compare manifest against K-means manifests | High |
| **Final patient manifest** | Choose best method, produce canonical patient assignments | High |

## Proposed final structure

```
patient_leakage_detector/
├── classes/
│   ├── features.py              # FeatureExtractor (5 CNNs)
│   ├── patient_identifier.py    # PatientIdentifier (K-means clustering)
│   ├── classifier.py            # PatientLeakageClassifier (RF→SVM pipeline)
│   ├── nifti_dataset.py         # NiftiSliceDataset (Task06 loader)
│   └── siamese.py               # SiamesePatientMatcher (learned matching)
│
├── scripts/
│   ├── classification.py        # Core: single-seed classification
│   ├── train_siamese.py         # Core: siamese training on Task06
│   ├── siamese_identify.py      # Core: IQ-OTH patient manifest via siamese
│   ├── validate_clustering_task06.py  # Core: Task06 validation
│   │
│   ├── visualizations/
│   │   ├── figure5.py           # [TODO] PCA + t-SNE plots
│   │   ├── figure6.py           # ✅ 20-seed boxplots
│   │   ├── figure_s1.py         # [TODO] Image-level CV curves
│   │   ├── figure_s2.py         # [TODO] PCA/t-SNE per model
│   │   ├── figure_s3.py         # [TODO] Example cluster images
│   │   ├── figure_s4.py         # ✅ Patient-level CV curves
│   │   ├── figure_s5.py         # [TODO] Confusion matrix
│   │   └── simple_patient_tsne.py  # ✅ t-SNE with patient coloring
│   │
│   └── analysis/
│       ├── compare_clustering_methods.py  # ✅ Thumbnail vs feature vs v8
│       └── verify_feature_dims.py        # ✅ TF dimension verification
│
├── features/
│   ├── VGG16_features.npz       # Extracted features (all 5 models)
│   └── task06_lung/             # Cached Task06 features
│
├── models/
│   └── siamese_resnet18.pt      # Trained siamese model
│
├── results/
│   ├── simple_patient_manifest.csv     # Feature-based K-means manifest
│   ├── siamese_manifest.csv            # [TODO] Siamese-based manifest
│   ├── classification_results.json     # Single-seed results (all 5 models)
│   ├── figure6_data.json              # 20-seed data
│   ├── task06_clustering_validation.json  # Task06 validation results
│   └── ...
│
├── plots/
│   ├── figure6.png
│   ├── figure_s4.png
│   └── ...
│
├── plan.md                     # This file
├── notes.md                    # Review notes for supervisor
├── .gitignore
└── .archive/                   # Superseded scripts and results
```

## Immediate next steps

1. **Finish siamese training** → run `python scripts/siamese_identify.py` → produce `siamese_manifest.csv`
2. **Run classification with thumbnail K-means manifest** to reproduce manuscript Table 2 numbers
3. **Compare manifests**: K-means thumbnail vs K-means feature vs siamese — which gives the best patient-level split?
4. **Choose canonical manifest** and produce final classification results
5. **Generate remaining figures** (Figure 5, S1, S2, S3, S5)
6. **Draft findings for supervisor discussion**