{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\\rightarrow$Run All).\n",
    "\n",
    "Make sure you fill in any place that says `YOUR CODE HERE` or \"YOUR ANSWER HERE\", as well as your name below.\n",
    "\n",
    "Rename this problem sheet as follows:\n",
    "\n",
    "    ps{number of lab}_{your user name}_problem{number of problem sheet in this lab}\n",
    "    \n",
    "for example\n",
    "    \n",
    "    ps2_blja_problem1\n",
    "\n",
    "Submit your homework within one week until next Monday, 9 a.m."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"\"\n",
    "EMAIL = \"\"\n",
    "USERNAME = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Data Science\n",
    "## Lab 13: Prinicipal Component Regression and Partial Least Squares\n",
    "\n",
    "## Part A - Data preparation\n",
    "\n",
    "We start this problem sheet with probably the most important step in data science: data preparation.\n",
    "This time, we are going to investigate a data set that contains baseball data from the (North American) Major League during 1986 and 1987."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Import the Hitters data set, available on the class web page and drop all rows containing missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "87e7d826c32cf3f52d06d9e121cc689a",
     "grade": false,
     "grade_id": "cell-1fc6cb98984aeba0",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ecc8f9db69921f6b0a6ff6423bfa17b3",
     "grade": true,
     "grade_id": "cell-f1e9a2e61a15035d",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert df.shape == (263, 20)\n",
    "assert abs(df['Runs'].mean() - 54.745247148288975) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Identify the three variables containing categorical variables and store their labels in a list `dummy_vars`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "09f8b69ebee4cf490c3f4e0589e00b27",
     "grade": false,
     "grade_id": "cell-75d63dfd30c60974",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5c478c34ffa9f74d0967908c29ced1bb",
     "grade": true,
     "grade_id": "cell-a74df9e7d7db59c4",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert 'League' in dummy_vars\n",
    "assert len(dummy_vars) == 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The pandas function `get_dummies` converts categorical variables into 0-1-dummy variables. This has been discussed in Chapter 3 (slide 112).\n",
    "\n",
    "**Task (1 point)**: Convert the three categorical variables in the dataset into dummy variables and store them in a new `DataFrame` called `df_dummy`.\n",
    "Take a look at the new `DataFrame` using the method `head`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "c8734ef457ecb69e05e1f2cf87079dae",
     "grade": false,
     "grade_id": "cell-492794b27b6094fb",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ad5be52bc1491c521cdccffafca5a8e8",
     "grade": true,
     "grade_id": "cell-af6600237d5b1f2a",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(df_dummy['League_A'].mean() - 0.5285171102661597) < 1e-8\n",
    "assert df_dummy.shape == (263, 6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Once you have done this, you should see that there are only two categories in each of the dummy variables.\n",
    "Thus we should only include one of each into our final data frame.\n",
    "\n",
    "**Task (no points)**: If you did everything right so far, the following code should execute without errors."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# real_vars = ['AtBat', 'Hits', 'HmRun', 'Runs',\n",
    "#              'RBI', 'Walks', 'Years', 'CAtBat',\n",
    "#              'CHits', 'CHmRun', 'CRuns', 'CRBI',\n",
    "#              'CWalks','PutOuts', 'Assists', 'Errors']\n",
    "\n",
    "int_vars = df.columns[df.dtypes=='int64']\n",
    "\n",
    "dfX = pd.concat([df.loc[:,int_vars], df_dummy.loc[:,['League_A', 'Division_E', 'NewLeague_A']]], axis=1)\n",
    "dfy = df.Salary"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part B - Applying PCR"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have studied intensively PCA during the lab.\n",
    "We will now combine this knowledge to perform a PCR.\n",
    "\n",
    "According to [Wikipedia](https://en.wikipedia.org/wiki/Principal_component_regression), the PCR method may be broadly divided into three major steps:\n",
    "\n",
    "1. Perform PCA on the observed data matrix for the explanatory variables to obtain the principal components, and then (usually) select a subset, based on some appropriate criteria, of the principal components so obtained for further use.\n",
    "2. Now regress the observed vector of outcomes on the selected principal components as covariates, using ordinary least squares regression (linear regression) to get a vector of estimated regression coefficients (with dimension equal to the number of selected principal components).\n",
    "3. Now transform this vector back to the scale of the actual covariates, using the selected PCA loadings (the eigenvectors corresponding to the selected principal components) to get the final PCR estimator (with dimension equal to the total number of covariates) for estimating the regression coefficients characterizing the original model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Scale the data using `StandardScaler` from `sklearn.preprocessing` and perform a (full) principal component analysis.\n",
    "Store the learned model as `pca`, and the principal components as `pc`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1d79a8e08e5826cbfab752b750d07e72",
     "grade": false,
     "grade_id": "cell-01179cd66837573d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.decomposition import PCA\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e83ccc67b8617f009252a2d31c299029",
     "grade": true,
     "grade_id": "cell-676e21559bcebd37",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(pca.components_.mean() - 0.007661708572604116) < 1e-8\n",
    "assert abs(pc[0].mean() - 0.2501177953642573) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should implement a loop over the number of principal components in your model.\n",
    "We want to measure the quality by a cross-validated mean squared error using 10 folds.\n",
    "\n",
    "**Task (2 points)**:\n",
    "Implement a loop over the number of principal components in your model.\n",
    "Use a `LinearRegression()` model as an estimator in `cross_val_score`.\n",
    "As data, you should choose the first $j$ principal components.\n",
    "Store the means of the mean squared errors in a list called `mse`.\n",
    "You can use an appropriate `scoring` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "d9a2bb28f66b745b41e7562dcc19753d",
     "grade": false,
     "grade_id": "cell-d4d921de8b105fc4",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.linear_model import LinearRegression\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "N = pca.n_components_\n",
    "\n",
    "mse = []\n",
    "for i in range(N):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "83e92e5cf42d89e52a10d89ad2248ec9",
     "grade": true,
     "grade_id": "cell-a54236518eea368d",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(np.mean(mse) - 120661.07475660776) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Determine the number of components for which the MSE is smallest and store it in the variable `n_min`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4807d7dd7e14f425548e0ca4a0110aa0",
     "grade": false,
     "grade_id": "cell-266b5cc2796781e1",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "eba31c1fe2f6cea37ef5b4b55d95ed91",
     "grade": true,
     "grade_id": "cell-cb85a1b092614b36",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert 'n_min' in locals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should observe that the MSE is minimized by taking all but one principal components into consideration.\n",
    "This corresponds to no dimensionality reduction at all, and simply performs a linear regression using all of the variables.\n",
    "But we also observe that the values do not change very much, even using only one predictor yields a good fit."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Plot the MSE against the number of components in your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "403aa147e326aab63d21e4b77669d403",
     "grade": true,
     "grade_id": "cell-4d743236ea8c344c",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Plot the percentage of variance explained by the first $j$ principal components against the number of principal components $j$.\n",
    "You should use the attribute `explained_variance_ratio_` from your `PCA()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1d843f904d91d60c676f22570ea6d474",
     "grade": true,
     "grade_id": "cell-206b6673874808df",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Part C - Partial Least Squares"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to apply partial least squares regression to this data set.\n",
    "The function `PLSRegresssion` is provided by sklearn in the module `cross_decomposition`.\n",
    "\n",
    "**Task (1 point)**: Similar to the PCR loop from above, implement a loop over the number of components in your PLS regression model.\n",
    "Use 10-fold cross-validation and store the means of the mean squared errors in a list called `mse_pls`.\n",
    "You can use an appropriate `scoring` option."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "4099ffe6a4d8b3df3d3162e673c18553",
     "grade": false,
     "grade_id": "cell-1ab9b3590d5bdbc9",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.cross_decomposition import PLSRegression\n",
    "\n",
    "mse_pls = []\n",
    "\n",
    "for i in range(N):\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8febbfebda7d1170066a1d2ce97926a1",
     "grade": true,
     "grade_id": "cell-cd1889cda3fe5b7e",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(np.mean(mse_pls) - 117660.18445105157) < 1e-8\n",
    "assert len(mse_pls) == 19"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Determine the number of components, for which the MSE is minimized in PLS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7ce240689b6178685b0152a99ef5e0e5",
     "grade": false,
     "grade_id": "cell-e84a521b03d1086f",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f2f1a88412a79fbcc8151066c05f81c2",
     "grade": true,
     "grade_id": "cell-f84707308702d98f",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert 'n_min_pls' in locals()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Plot the MSE against the number of components in your model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "35ef882cd95b16e9129cb2040a32937c",
     "grade": true,
     "grade_id": "cell-e039c26ee69b1ab1",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Again, we might draw a similar conclusion as for PCA.\n",
    "Altough the MSE is minimized for 14 components, it is fairly low for other values as well.\n",
    "\n",
    "**Task (no points)**: Finally, we want to take a look at the declared variance in the response in terms of the number of compontens used in the PLS regression.\n",
    "What do you observe?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "r2 = []\n",
    "for i in range(N):\n",
    "    pls = PLSRegression(n_components=i+1)\n",
    "    cvs = cross_val_score(pls, X, y, cv=10, scoring='r2')\n",
    "    r2.append(cvs.mean())\n",
    "    print(cvs.mean())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Observation**: Most of the variance in the response variable is already declared by taking only one component of partial least squares. Probably, a train-test split show a different picture."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}