{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "f75d5967",
   "metadata": {},
   "source": [
    "# Part 1: Exploratory Data Analysis\n",
    "\n",
    "In this notebook, we explore the preprocessed datasets generated from my autoimmunity profiling study on Exfoliative Glaucoma (XFG).\n",
    "\n",
    "I made three \"slices\" of the data so I can try each one and see which makes the nicest product for this assignment. These are 1) the proteins we found to be significantly different between XFG and controls, 2) group 1 plus 33 random for a total of 40 features, and 3) all 260 antigens included in the study.\n",
    "I probably will only include group 1, unless these are too easily separable.\n",
    "\n",
    "These slices are already preprocessed similar to in the study, including background correction, box-cox transformation, and z-score normalization. If you want the code used for this, I can provide it, but I won't include it here since it's not the focus of this assignment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4ffff549",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Slice 1 (Significant 7) shape: (60, 10)\n",
      "Slice 2 (Sig 7 + 33 random) shape: (60, 43)\n",
      "Slice 3 (All proteins) shape: (60, 263)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Internal LIMS ID</th>\n",
       "      <th>Original ID</th>\n",
       "      <th>group</th>\n",
       "      <th>HPRA000767</th>\n",
       "      <th>HPRA034083</th>\n",
       "      <th>HPRA006876</th>\n",
       "      <th>HPRA019035</th>\n",
       "      <th>HPRA003490</th>\n",
       "      <th>HPRA022019</th>\n",
       "      <th>HPRA017192</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>GLA_02-0001</td>\n",
       "      <td>HD1</td>\n",
       "      <td>0</td>\n",
       "      <td>0.352329</td>\n",
       "      <td>-0.074663</td>\n",
       "      <td>0.442451</td>\n",
       "      <td>0.210802</td>\n",
       "      <td>-0.025678</td>\n",
       "      <td>-0.923763</td>\n",
       "      <td>0.028253</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>GLA_02-0002</td>\n",
       "      <td>GC1</td>\n",
       "      <td>1</td>\n",
       "      <td>2.184865</td>\n",
       "      <td>0.467075</td>\n",
       "      <td>1.272995</td>\n",
       "      <td>1.857599</td>\n",
       "      <td>2.709290</td>\n",
       "      <td>1.367955</td>\n",
       "      <td>0.220041</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>GLA_02-0003</td>\n",
       "      <td>HD2</td>\n",
       "      <td>0</td>\n",
       "      <td>0.519969</td>\n",
       "      <td>-0.358698</td>\n",
       "      <td>0.124177</td>\n",
       "      <td>1.383003</td>\n",
       "      <td>1.227493</td>\n",
       "      <td>-0.128889</td>\n",
       "      <td>-0.343792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>GLA_02-0004</td>\n",
       "      <td>GC2</td>\n",
       "      <td>1</td>\n",
       "      <td>1.155189</td>\n",
       "      <td>0.467075</td>\n",
       "      <td>0.630671</td>\n",
       "      <td>0.080847</td>\n",
       "      <td>-0.025678</td>\n",
       "      <td>-0.416995</td>\n",
       "      <td>0.753164</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>GLA_02-0005</td>\n",
       "      <td>HD3</td>\n",
       "      <td>0</td>\n",
       "      <td>-0.590507</td>\n",
       "      <td>-0.074663</td>\n",
       "      <td>-1.423605</td>\n",
       "      <td>-1.591214</td>\n",
       "      <td>-0.375025</td>\n",
       "      <td>-0.165849</td>\n",
       "      <td>-1.292029</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Internal LIMS ID Original ID  group  HPRA000767  HPRA034083  HPRA006876  \\\n",
       "0      GLA_02-0001         HD1      0    0.352329   -0.074663    0.442451   \n",
       "1      GLA_02-0002         GC1      1    2.184865    0.467075    1.272995   \n",
       "2      GLA_02-0003         HD2      0    0.519969   -0.358698    0.124177   \n",
       "3      GLA_02-0004         GC2      1    1.155189    0.467075    0.630671   \n",
       "4      GLA_02-0005         HD3      0   -0.590507   -0.074663   -1.423605   \n",
       "\n",
       "   HPRA019035  HPRA003490  HPRA022019  HPRA017192  \n",
       "0    0.210802   -0.025678   -0.923763    0.028253  \n",
       "1    1.857599    2.709290    1.367955    0.220041  \n",
       "2    1.383003    1.227493   -0.128889   -0.343792  \n",
       "3    0.080847   -0.025678   -0.416995    0.753164  \n",
       "4   -1.591214   -0.375025   -0.165849   -1.292029  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Load the slices\n",
    "slice_1 = pd.read_csv('slice_1_significant_7.csv')\n",
    "slice_2 = pd.read_csv('slice_2_sig7_plus_33_random.csv')\n",
    "slice_3 = pd.read_csv('slice_3_all_proteins.csv')\n",
    "\n",
    "print(\"Slice 1 (Significant 7) shape:\", slice_1.shape)\n",
    "print(\"Slice 2 (Sig 7 + 33 random) shape:\", slice_2.shape)\n",
    "print(\"Slice 3 (All proteins) shape:\", slice_3.shape)\n",
    "\n",
    "slice_1.head()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "768a0c3e",
   "metadata": {},
   "source": [
    "### Class Distribution\n",
    "\n",
    "We already know this, but let's check the distribution of our target variable `group`, where `1` represents Exfoliative Glaucoma (XFG) and `0` represents Healthy Controls.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "e2d1802b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 19492 characters]",
      "text/plain": [
       "<Figure size 600x400 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "group\n",
      "0    30\n",
      "1    30\n",
      "Name: count, dtype: int64\n"
     ]
    }
   ],
   "source": [
    "plt.figure(figsize=(6, 4))\n",
    "sns.countplot(x='group', data=slice_1, hue='group', palette='Set2', legend=False)\n",
    "plt.title('Distribution of Target Variable (Group)')\n",
    "plt.xlabel('Group (0 = Healthy, 1 = XFG)')\n",
    "plt.ylabel('Count')\n",
    "plt.show()\n",
    "\n",
    "print(slice_1['group'].value_counts())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fd3851",
   "metadata": {},
   "source": [
    "### Feature Distributions\n",
    "\n",
    "We will first examine the distributions of the 7 significant protein fragments across the two groups. This helps us visualize how well individual features might separate the classes on an individual basis.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "13a170fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 237312 characters]",
      "text/plain": [
       "<Figure size 1800x1000 with 7 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Extract the feature columns (excluding identifiers and target)\n",
    "significant_cols = [c for c in slice_1.columns if c not in ['Internal LIMS ID', 'Original ID', 'group']]\n",
    "\n",
    "fig, axes = plt.subplots(nrows=2, ncols=4, figsize=(18, 10))\n",
    "axes = axes.flatten()\n",
    "\n",
    "for i, col in enumerate(significant_cols):\n",
    "    sns.histplot(data=slice_1, x=col, hue='group', kde=True, ax=axes[i], palette='Set2')\n",
    "    axes[i].set_title(col)\n",
    "\n",
    "# Remove the empty subplot\n",
    "fig.delaxes(axes[7])\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d885a4f",
   "metadata": {},
   "source": [
    "### Correlation Analysis\n",
    "\n",
    "Let's look at the correlation matrix to see if these 7 significant protein fragments are highly correlated with each other. If they are highly correlated, they might carry redundant information for our Decision Tree.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "dddc7b3c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 92568 characters]",
      "text/plain": [
       "<Figure size 800x600 with 2 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure(figsize=(8, 6))\n",
    "sns.heatmap(slice_1[significant_cols].corr(), annot=True, cmap='coolwarm', fmt=\".2f\", vmin=-1, vmax=1)\n",
    "plt.title('Correlation Matrix of Significant Features')\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "13f6e649",
   "metadata": {},
   "source": [
    "We can make a scatter matrix to visualize the pairwise relationships between these features and see if there are any clear separations between the two groups in the feature space."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "499aa028",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "[encoded data, 367472 characters]",
      "text/plain": [
       "<Figure size 1500x1500 with 49 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "scatter_matrix = pd.plotting.scatter_matrix(slice_1[significant_cols], figsize=(15, 15), c=slice_1['group'], cmap='seismic', diagonal='kde')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38ccf061",
   "metadata": {},
   "source": [
    "Motivation for using this data for classification:\n",
    "\n",
    "- The 7 significant protein fragments were identified as being associated with XFG in the original study, so they are likely to contain relevant information for distinguishing between XFG and healthy controls.\n",
    "- There is no clear separation between groups across any one or two features, which is good because it will let us demonstrate the capabilities of the decision tree and mlp.\n",
    "- The features are not highly correlated with each other, which means they may provide complementary information for classification.\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "intro_ds",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.14.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
