{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before you turn this problem in, make sure everything runs as expected. First, **restart the kernel** (in the menubar, select Kernel$\\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\\rightarrow$Run All).\n",
    "\n",
    "Make sure you fill in any place that says `YOUR CODE HERE` or \"YOUR ANSWER HERE\", as well as your name below.\n",
    "\n",
    "Rename this problem sheet as follows:\n",
    "\n",
    "    ps{number of lab}_{your user name}_problem{number of problem sheet in this lab}\n",
    "    \n",
    "for example\n",
    "    \n",
    "    ps2_blja_problem1\n",
    "\n",
    "Submit your homework within one week until next Monday, 9 a.m."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "NAME = \"\"\n",
    "EMAIL = \"\"\n",
    "USERNAME = \"\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Introduction to Data Science"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Lab 12 - Is scaling always important?\n",
    "In this problem, we consider a new data set.\n",
    "The data set consists of 1499 samples of a particular red wine from Minho, Portugal, called *Vinho verde*.\n",
    "The first 11 columns in the csv file contain different measurements, the last column contains an expert rating of the quality.\n",
    "This set became popular in a kaggle competition, but is also publicly available [here](http://www3.dsi.uminho.pt/pcortez/wine/).\n",
    "The data set resides also on our [webpage](https://www.tu-chemnitz.de/mathematik/numa/lehre/ds-2018/).\n",
    "\n",
    "**Task (1 point)**: Download the new csv-files `wine-train.csv` and `wine-test.csv` from the lecture's webpage.\n",
    "\n",
    "Import the data from the csv-file `wine-train.csv` and store it in the `pandas.DataFrame` df."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b6f5872a652b2d376ccae3856354c21b",
     "grade": false,
     "grade_id": "cell-fb4c5835492e49bb",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "b72a2b81d985c050996965066bcaf430",
     "grade": true,
     "grade_id": "cell-6a2d0979bc1401ef",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert df.shape == (1278,12)\n",
    "assert df.columns[5] == 'free sulfur dioxide'\n",
    "assert abs(df['pH'].mean() - 3.2998122065727697) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Our target variable is stored in the column labeled `'quality'`.\n",
    "Extract the values of this column into a `numpy.array` named `y` and the remaining data into another array `X`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "91510185b24f283df00cd4442f987112",
     "grade": false,
     "grade_id": "cell-622819a03b5d9db9",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ff130944fbd7307a3adcffa43840eaeb",
     "grade": true,
     "grade_id": "cell-65256d7bd888c065",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert X.shape == (1278,11)\n",
    "assert y.shape == (1278,)\n",
    "assert abs(y.mean() - 5.663536776212832) < 1e-8\n",
    "assert X.dtype == 'float64'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we want to look at the coefficient selection for both the scaled and unscaled case.\n",
    "In this example, *scaled* means that we shift the mean of the features to *zero* and scale the standard deviation to *one*.\n",
    "This can be easily done by using a `StandardScaler` provided within `sklearn.preprocessing`.\n",
    "\n",
    "**Task (1 point)**: Set up a `StandardScaler` named `stdScaler` and normalize your predictor matrix `X` using the method `fit_transform()`.\n",
    "Store the scaled matrix as `Xscaled`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1845c9288f3cbd1358c808bbee45ca5b",
     "grade": false,
     "grade_id": "cell-2ecbfafadccc407c",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "806b8d69cb133fb297c46f2ced0e35e7",
     "grade": true,
     "grade_id": "cell-3c32d630d3554005",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert max(abs(Xscaled.mean(axis=0))) < 1e-10\n",
    "assert max(abs(Xscaled.std(axis=0)-1)) < 1e-10"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We want to compare the coefficients obtained for the scaled and unscaled predictors.\n",
    "\n",
    "**Task (2 points)**: Fill in the two gaps in the following code cell.\n",
    "\n",
    "When done correctly, it computes the *Lasso* estimates for `m` different values of the regularization parameter $\\alpha$ and stores the coefficients as well as the cross-validation score for each $\\alpha$ for both the **scaled and unscaled data**.\n",
    "\n",
    "Finally, it plots the coefficients in the upper part of the figure, and the corresponding cv-scores in the lower part."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "e947e2339e4683889a92d6f543b09448",
     "grade": true,
     "grade_id": "cell-373bc148519c4ae9",
     "locked": false,
     "points": 1,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "from sklearn.linear_model import Lasso\n",
    "from sklearn.preprocessing import scale\n",
    "from sklearn.model_selection import cross_val_score\n",
    "\n",
    "# Get dimensions of X\n",
    "n,p = X.shape\n",
    "m = 50\n",
    "Alpha = np.logspace(-4,1,m)\n",
    "\n",
    "cscaled = np.zeros((m,p+1))\n",
    "corig = np.zeros((m,p+1))\n",
    "cvscaled = np.zeros((m,))\n",
    "cvorig = np.zeros((m,))\n",
    "\n",
    "for (i,a) in enumerate(Alpha):\n",
    "    \n",
    "    # Task: Perform a Lasso regression with tol = 1e-8\n",
    "    #       using the scaled input and current alpha\n",
    "    #       Store the learned coefficients in the array cscaled with\n",
    "    #       the intercept being stored in the first column of cscaled\n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "    # Additionaly, we perform a cross-validation for the current model\n",
    "    # with the scaled data\n",
    "    cvscaled[i] = cross_val_score(lm, Xscaled, y, cv=10).mean()\n",
    "    \n",
    "    # Task: Perform a Lasso regression with tol = 1e-8\n",
    "    #       using the original input and current alpha\n",
    "    #       Store the learned coefficients in the array corig with\n",
    "    #       the intercept being stored in the first column of corig    \n",
    "    # YOUR CODE HERE\n",
    "    raise NotImplementedError()\n",
    "\n",
    "    # Additionaly, we perform a cross-validation for the current model\n",
    "    # with the unscaled data\n",
    "    cvorig[i] = cross_val_score(lm, X, y, cv=10).mean()\n",
    "    \n",
    "# Plot the output\n",
    "import matplotlib.pyplot as plt\n",
    "plt.rcParams['figure.figsize']=(15,10)\n",
    "fig, ax = plt.subplots(2,2)\n",
    "ax[0][0].semilogx(Alpha, cscaled[:,1:])\n",
    "ax[0][0].set_title('Scaled predictors')\n",
    "ax[0][0].set_xlabel('Alpha')\n",
    "ax[0][0].set_ylabel('Coefficients');\n",
    "\n",
    "ax[0][1].semilogx(Alpha, corig[:,1:])\n",
    "ax[0][1].set_title('Unscaled predictors')\n",
    "ax[0][1].set_xlabel('Alpha')\n",
    "ax[0][1].set_ylabel('Coefficients');\n",
    "\n",
    "ax[1][0].semilogx(Alpha, cvscaled)\n",
    "ax[1][0].set_xlabel('Alpha')\n",
    "ax[1][0].set_ylabel('cv-score');\n",
    "ax[1][1].semilogx(Alpha, cvorig);\n",
    "ax[1][1].set_xlabel('Alpha')\n",
    "ax[1][1].set_ylabel('cv-score');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "5bd29d2b94e45f5a9490c9a06f529c4b",
     "grade": true,
     "grade_id": "cell-708c73e9bd15cae0",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(corig.mean() -0.29114416793438974) < 1e-6\n",
    "assert abs(cscaled.mean() - 0.4755654903995176) < 1e-6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You should make the following observations:\n",
    "- the ranges of the coefficients differ by a magnitude of 10\n",
    "- in the unscaled model, some predictors don't enter the model even for the smallest regularization parameter\n",
    "- the order of variables entering/leaving the model differ"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**:\n",
    "Compute the values of $\\alpha$, for which the cv-scores are maximized in variables `alphascaled` and `alphaorig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "3f84fd257087fdf95506f739126cf195",
     "grade": false,
     "grade_id": "cell-12866775eaadd1db",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "7bd95926615f80a8920360144743ae49",
     "grade": true,
     "grade_id": "cell-e490e3d8b7c70d5c",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(alphascaled - 0.01757510624854793) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Store the indices of the predictors selected by the best models (in this case given by highest cv score) using both the scaled and unscaled data in variables `idxscaled` and `idxorig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "dae8963e18d5960ef536a87604425dbb",
     "grade": false,
     "grade_id": "cell-0709bb1a917c881d",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "8dcc908ef773baa7a94900faa1e61317",
     "grade": true,
     "grade_id": "cell-4da27c45084d33ea",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert len(idxscaled) == 7\n",
    "assert len(idxorig) == 11"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (2 points)**: Now we want to compare the training mean squared errors for both regressions using the value of $\\alpha$ which maximizes the cv-score.\n",
    "Store the MSE's in variables `msescaled` and `mseorig`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f193310a89188b884053d32ce63e2344",
     "grade": false,
     "grade_id": "cell-31849bcef8b1719c",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "1e7a5f52fe2a0945762029b9790c5cc5",
     "grade": true,
     "grade_id": "cell-438c0ad7b30b61e5",
     "locked": true,
     "points": 2,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(mseorig - 0.4161264558967955) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this case, the scaling does not improve the mean squared error on the training data.\n",
    "Actually, you should observe that the MSE using the unscaled data is only slighty superior to the scaled data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Import now the csv-file `wine-test.csv` and store the predictors as a numpy array `Xtest` and the target variables as `ytest`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a48ec9422645e3e1221e5f02ebf11832",
     "grade": false,
     "grade_id": "cell-2c8a753767acedbe",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "a40388c1d6713d4178ddf7b3be4d1f7a",
     "grade": true,
     "grade_id": "cell-65bf74dfc06a2e08",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(Xtest.mean() - 8.069555417728688) < 1e-8\n",
    "assert abs(ytest.mean() - 5.526479750778816) < 1e-8\n",
    "Xtest.shape == (321, 11)\n",
    "ytest.shape == (321,)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (1 point)**: Compute the test MSE's for the unscaled and scaled data and store their values in variables `testmseorig` and `testmsescaled`.\n",
    "\n",
    "Don't forget to scale the test data using the previously trained `StandardScaler`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "9add5aae8ca01fb792eb3bb15420167b",
     "grade": false,
     "grade_id": "cell-24667bbe32ea2e23",
     "locked": false,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()\n",
    "\n",
    "print('Test error on orig set:\\n\\t {}'.format(testmseorig))\n",
    "print('Test error on scaled set:\\n\\t {}'.format(testmsescaled))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "editable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "f6d54a2e24bbed5511fadd6cdc91c59f",
     "grade": true,
     "grade_id": "cell-18caf9252554a1e5",
     "locked": true,
     "points": 1,
     "schema_version": 3,
     "solution": false,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "assert abs(testmseorig - 0.4355191924296641) < 1e-8\n",
    "assert abs(testmsescaled - 0.439533544980072) < 1e-8"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Task (2 points)**: Interpret the results. What could be the reason, why the unscaled model behaves (slightly) better than the scaled model?\n",
    "In absolute terms, are your predictions good or bad?\n",
    "Which model do you prefer? Why?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "code",
     "checksum": "ac587dcab37998dcf7debb9fb2bc0b67",
     "grade": true,
     "grade_id": "cell-c9abefc46c95a211",
     "locked": false,
     "points": 0,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "outputs": [],
   "source": [
    "# You should use this cell to study both models further\n",
    "\n",
    "# YOUR CODE HERE\n",
    "raise NotImplementedError()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "nbgrader": {
     "cell_type": "markdown",
     "checksum": "ac7f3a295539b67c8d288653508bc7ba",
     "grade": true,
     "grade_id": "cell-89529fff61428fe4",
     "locked": false,
     "points": 2,
     "schema_version": 3,
     "solution": true,
     "task": false
    }
   },
   "source": [
    "YOUR ANSWER HERE"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}