{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Problem sheet 12 - Support Vector Machines" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "In the lecture about [support vector machines](https://www.tu-chemnitz.de/mathematik/numa/lehre/ds-2018/Slides/ds-intro-chapter9.pdf), we introduced three classifiers all based on the same principle: they try to determine a ($(d-1)$-dimensional) hyperplane to separate the data.\n", "\n", "The first one discussed was the **maximal margin classifier**. This classifier only works if the data is completely separable, otherwise, the optimization problem\n", "\n", "$$ \\max_{\\beta_0, \\beta_1, \\ldots, \\beta_p, M} M\\\\\n", "\\begin{aligned}\n", "\\text{such that }& & \\sum_{j=0}^p \\beta_j^2 &= 1 \\\\\n", "\\text{and }& & y_i \\, (\\beta^\\top x_i) &\\ge M, \\quad \\text{for } i=1,\\ldots,N\n", "\\end{aligned}$$\n", "\n", "has no admissable solution.\n", "Furthemore, it is quite unstable with respect to adding new observations.\n", "\n", "Therefore, we quickly derived a second algorithm, known as the **support vector classifier** or **soft-margin classifier**.\n", "This classifier allows some observations to be on the wrong side of margin or even on the wrong side of the hyperplane.\n", "The associated optimization problems reads\n", "\n", "$$ \\max_{\\beta_0, \\beta_1, \\ldots, \\beta_p, M, \\varepsilon_1, \\ldots, \\varepsilon_N} M\\\\\n", "\\begin{aligned}\n", "\\text{such that }& & \\sum_{j=0}^p \\beta_j^2 &= 1 \\\\\n", "\\text{and }& & y_i \\, (\\beta^\\top x_i) &\\ge M \\, (1 - \\varepsilon_i), \\quad \\text{for } i=1,\\ldots,N,\\\\\n", "\\text{and }& & \\varepsilon_i \\ge 0, \\sum_{i=1}^N \\varepsilon_i &\\le C \\quad \\text{for } i=1, \\ldots, N,\n", "\\end{aligned}$$\n", "\n", "where $C \\ge 0$ is a tuning parameter.\n", "Furthermore, we see that for $C = 0$ the **soft-margin classifier** simplifies to the **maximal margin classifier**.\n", "This is an important observation, and helps us with our first problem." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 12.1 Maximal margin classification\n", "\n", "In this problem, we examine maximal margin classification. We can generate a test problem by exploiting the `samples_generator` functionality from `sklearn`, i.e., \n", "\n", " from sklearn.datasets.samples_generator import make_blobs\n", " X, y = make_blobs(n_samples=60, centers=2,\n", " random_state=0, cluster_std=0.60)\n", " \n", "**Task**: Complete the following code cell. Generate a classification sample data set with $60$ samples, two classes, random state $0$ and a standard devitiation within the classes of $0.6$.\n", "You can use `plt.scatter` with the option `c=y` to colorize the blobs in accordance with their class." ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "## TODO: Generate data and make scatter plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Solution**:" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3Xl4XGXd//H3d/bsaZu2FOjCjoDVllCg8GMHC4hVEBAeFRUpyCI+gD6Az8OiIiKLyKJYFpFVFtk3ZV8t0JallAoUKGuXdEuaZDKTmbl/f5xpmy4hk3RmzkzyeV1XrmYmZ8755DT55sx97sWcc4iISPkI+B1ARER6R4VbRKTMqHCLiJQZFW4RkTKjwi0iUmZUuEVEyowKt4hImVHhFhEpMyrcIiJlJlSInTY0NLgxY8YUYtciIv3SjBkzFjvnhuaybUEK95gxY5g+fXohdi0i0i+Z2Ue5bqumEhGRMqPCLSJSZlS4RUTKjAq3iEiZUeEWESkzKtwiImVGhbsX3p3xPsePO4OjRh7P3867E60eJCJ+KEg/7v6oraWdX+z3K9qa2wG4+5IHaNhkEAcft7/PyURkoNEVd44+e2/+GlfYHe0JXnvyLR8TichApcKdo+Gjh5JOZVY9jlZE2Gr8Zj4mEpGBSoU7R3UNtZxz1+nUDK4mGA6y27d25tunHeJ3LBEZgNTG3QsTDhzHPYv/6ncMERngdMUtIlJmVLhFRMqMCreISJlR4RYRKTO6OSklJ5no5PO5Cxg0vI66hlq/4/jq0/fm8+ydL1FZU8GkY/ehoirmdyQpASrcUlKaPl3CTyeeTdvydtKpNKf+eQoHHLOX37F88fF/PuPkCWeSiCcJhYM8ct0T/Gn6RYQjYb+jic/UVCIl5ZrTbmTp/OXEWztIdnRy+QlTScQTfsfyxaPXP0m8rYNMOkOyo5OFHzXxn5fn+h1LSoAKt5SUxZ8vI5POrPFce0vcpzT+ilZECAaDqx67jCMS09W2qHBLiTnw2H2IVkYACEdCjNlhJPXD6nxO5Y/JJx9I3dBaKmoqiFXHGL//WLZu3MLvWFIC1MYtJWXSD/chHA3z7J0vsfGWG/G9cw7HzPyO5YtBw+q4Yc7lvPHMbCprKhi753YD9lzImiyXOaXNbB6wAkgDKedc4xdt39jY6KZPn56XgCLl7vP3F3DVT29g2YJlHPjj/TjkhANUgGUdZjajp9q6Um+uuPd2zi3uYyaRASkRT3Dqbv9L8+IWXMYx9ec3U1kTY7/v7ul3NCljauMWKaDP3ltAMp7EZbx3ton2BC/e96rPqaTc5Vq4HfAvM5thZlMKGUikPxk8op5UKr3qcTgWZuQ2G/uYSPqDXAv3bs658cCBwElmtsfaG5jZFDObbmbTm5qa8hpSpFzVD63j9OtOIBwNEwwF2XbCVhz9y8P8jiVlLqebk2u8wOw8oNU5d0l32+jmpMia0qk0iXiSypoKv6NIierNzcker7jNrMrMalZ+DhwAaLFFkV4IhoIq2pI3ufQqGQ7cm+2+FAJuc849VtBUIiLSrR4Lt3PuA+ArRcgiIiI50MjJErD486XcfuG9tC1v45CffI3tJ27jdyQRKWEq3D6Lt8Y5aaczaV7UTDqd4YV7XuaSp89j2wlb+R1NREqUBuD4bM7Lc+lo6yCdnREvEU/yzJ0v+ZxKREpZyRTuWc/P4chNjuOgiqP49ZGXkUx0+h2pKGqHVJPuOkAjGmbQAJ0NT0RyUxKFO97WwS+//luWzl9OZyLFtIdmcOtv/uF3rKLY8qubceCx+xKJhYlVRRm57cZ848Sv+R1LREpYSbRxL/lsqTeoPisZT/LOqwNnpY+T/vgjDj31YNpXxBmz/UiCoWDPLxKRAaskCvewUQ2EIqujRCujfHXvHXxMVHwjNh/udwQRKRMl0VQSiUW45Knz2OKrYxg0vI6Dj9uXw884xO9YUiDJjiRNny4hnU73vLGIrKMkrrgBNh87mmtmXux3jLK2dMEyFny4iFFf2pTq+iq/4wDgnOOBPz3G4zc/x7BRDew2eScuP2EqLuOoH1bHZc/9imEjG9Z5XSaTIdWZJhLVGosia+v1JFO50CRTxffCvS/zu+9eQTDstY9f/OS5bL2j/+sTPnzt4/z5v/9Goj2BBQzn3Kr7GYFggJ0OHMdvHjhzjdc8e/e/+f0xV5FKdDJ+/69w/r0/JxKL5HS8ZKKTZDxZMn+4RHKV10mmpDxc/MOrScSTtLfEaW+J84fj/+J3JACev3saifYE4K1S3vUmdCadYeG8RaxY1spjf32aJ255joUfN/H7Y64iGU+SyTjefHY2f//dvTkd69Hrn+Sbdd/n8OHH8vN9zyfZkSzEtyTiu5JpKpG+c87R0ZZY47nWpW0+pVnTxluO4M1n36YzmQLAzAiGg6SSKaKVUXb5+o78eIfTaGtuxwwGDa8nGFp9PZHs6GTe7E97PE7Tp0u46pTrVx3n7X+/w52XPMB3//fbhfnGRHykwt0PmBl7HTmRF+99hUQ8SawyysHH7+93LAB+dMFRzJn2LvNmf0IgYPz0z8fx9kvv8vGcT9n1GzsRjgRpXdZKssMbcLVsUTNm3vfknPOK+yE79nicxZ8tJRQJrdpPsqOTz96dX9DvTcQvKtz9xC9uPJn7dnyEua/NY/z+Y9nvu+ssUpQ3yUQn77wyl0gszNaNW3zhiuXV9VX8afpFtCxZQUVNBZFomK8ds/eqr997xSNrbG/Ajy44mhfueZlli5o56Mf7sv/3el5Yd8z2mxKOhLx29IwjWhlh90N37vP3KFLKdHNSeqV9RZxTdjmLpk+X4DKOxq99lXPuOv0Li/cXaV7cwpSvnLGqqWT46KFc/erviFZEe72vz+bO508/u5HlTc1MPmkSB3x/r169/qX7X2XOtHfZctxm7HH4rn3+nkT6ojc3J3XFLb3y6HVPsuDDRauaJKb/6w1mPT+HsXts16f91TXUcu2sS3npvlcJhoP8v8N26VPRBthkyxFc8NBZfXrtHRffzy3n30VHe4JYVZS5b8zj2AuO7tO+RApNvUqkV9pa2kllbwACBMyIr4hv0D5rB9cw6Uf7sP/39iRW2beivaH+cdmDdGR7v3S0Jbj/ykd9ySGSCxVu6ZW9v7Mb0cooGATDQSprK/hyH6+2S0mkYs1+4uGYBv5I6VJTifTKyG024cppv+Wxvz5NrDLK5JMn9YtFcE/903Gc/+1LCYUDpDrT/PTq4/yOJNIt3ZwUyVr4URMfvPkRo7fblI232MjvODLA6OakSB8MHz2U4aOHdvt15xxP3PIcz989jY232ojvn3tEv3i3IeVHhXsDLf5sCXOmvcewUQ1ss9OWfscpOw/8+Z/c+uu7CUVCnHj5D9ntmxP8jtSth699gmtO8+ZdCUdDzH7xHa546YKCdBtcumAZT9z8HGbGAT/Yi7qG2rwfQ8qXCvcGeG/mB5y+17lYwEinMhx+xiEcc96RfscqGzOfnMXUn9+8ai6TC//rj1w9/SJGf2lTn5Ot32PXP7Uqa2cixdzXPmTF0lZqh9Tk9TjLm5q9vu3L28CMu//wENe9dRk1g6rzehwpX+pVsgGu/Z9biLd20N4SJ9Ge4PYL7yXe1uF3rIJpa2nn8Zue5fGbn6V9A7sAArz76lw6u6wtGggFeP/1eRu830Jp2GQQgeDqXxkzI1Ydy/txXrjnFTpaO0h1pkklU7S3tDPtwRl5P46UL11xb4BkfN3Z5zKp8lgcYMn8ZbQub6N+aC1m1uNVY1tLOyeM+znLFzUDcPP5d3HNaxdvUBvvZmNHE4mFV02QlUllGLP9yD7vr9BOvPyH/OeVubS3xEmn0vz3tScUZL7wcDQEXZtfzIioe6J0ocK9AY46+1B+ffildCY6iVRE2PngHamqK/15oG+78B5u+dXdZNJp0qkMoXCQ3Q/dmbNuPZVAYP1vwl6452WWL2peVWSXLVzOi/e9ktM8It3Z+aDxHHXWt/j7RfcRDAU54bJj2Hzs6D7vr9CGjRrKTe9fzfz3FzBoeH3em0hW2vOIidx16YMs+rgJHGy6zcbsOnmnghxLypMK9wbY+aDxXPzUecx8/E2GjW5gn6N39ztSjxZ93MStv757jSaKVGeaaQ/N4Lm7/s1eR+5GW3Mb//rbs3QmOtn7qN0ZuukQ1ttrNA89SY8++zCOPvuwDd9RkUSiYUZvV9h3BbHKKH969Xe89uQsLBBg3L47EI7oiltWy7lwm1kQmA585pz7euEilZcv7bwVX9p5K79j5Kx58QqC4RB0dK7xfKozRdMnS4i3dXDC+F+wdP4yMukMt194L1PfvJT/d9jO3PyrO1dtXz+sjt2+Vbo9QMpdJOa9gxNZn95ccZ8KzAHUL6mMjd5+JLWDq0nEE2RSmVXPB0NBxu8/lteemEXz4pZVk0h1tHXw1G0vcOQvJjP19Ut4/p5XMIPdD91ZfZhFfJJT4TazTYGDgQuA0wqaSAoqEg1z5csXctO5d/DJO5/T3hKnqr6S//rlYWzxlTEsnb9sjSYQCwS8m2VAVV0Vk364dzd7FpFiyfWK+3LgF0C3d2PMbAowBWDUqFEbnkwKZtCwOk7985T1fm38fmPZYtwY3n9tHpg37erXfrBXUfOJyBfrsXCb2deBRc65GWa2V3fbOeemAlPBm6skbwmlqIKhIJc8eR4zn3iTZEcn4/cfS0VV/vsqi0jf5XLFvRvwDTM7CIgBtWZ2i3Puu4WNJn4JhoLsNGmc3zFEpBs9jpx0zp3lnNvUOTcG+A7wlIq2DARLFyzjlF3P5qCKozlh/M+9ftUiJUBD3kW68duj/8h7M96nM9HJh29+xDnf+r3fkUSAXg7Acc49AzxTkCQiOZo3+xOeuPlZKmsr+MaJk6iuL8xo1Q9nfUQ622Uyk3F8MufzghxHpLc0clLKyrzZn3DKLmfR0Z4gFA7x2A1Pc+2sS/u8wPAX2XbnrZj5xCxSyRTBUJAtx2+W92OI9IWaSqSsPH7TM958KQ5SyRTNTS3MmfZeQY515s0/Zfx+Y6lrqGHsnttx3j/OKMhxRHpLV9xSVipqKghFQqtWms9kMlQUYGpVgJpB1Vzw0FkF2XexPHPHi1xz+t/IpDN879zDOeSEr/kdSfJAV9xSViafNImGTQZTWVNBrCrKzgfvyNaNW/gdqyR9OOsjLjn2Tyz5fBnLFjbzlzNu5o1nZ/sdS/JAV9xSVmoGVXPdW5cx+6V3qaiOse2ELQuydFh/MPe1eWtM05vuTPHuq+/zlT239zGV5IMKt5SdaEWU8ft+2e8YJW+zL48ik149kVgoEmKLcbrB2h+oqUTWsXTBMm777T+45dd30fTpEr/jSB9tOW4zTrn6x9QOqaaqrpLvn3+E/uD1E+bWO0P+hmlsbHTTp0/P+36l8FqWrODY7X9G67I2nHNU1lZy3VuXMXijQX5HE+nXzGyGc64xl211xS1rePnhmXS0JUh1esuaJeJJXrz3Fb9jiUgXKtyyhmhlZI2bfYGAEa3M/+AWEek7FW5Zw67faGTMDiOpqI5RURNjk61GsNeRE/2OJSJdqFdJGUnEE0x7aCY4x4SDxxdknuxwJMxlz/6KWc/PwWUcX97jS1qoVqTEqHCXiY72BCdPOJOFHy/GgEEb1XPNzN9TUZ3/dR9D4RDj9lHvA1lTOp3mnssfZvZL7/Dl3b/EN396IMFg0O9YA5IKd5l46f5XWfhRkzdPB5D5fBnP3vlvJv1oH5+TyUBx5cnX8cTNz5NoTzD9n28w/4OFnHzlsX7HGpDUxl0m0qn0Ws+4NQZXiBTa07e/SKLdu3BItCd48rbnfU40cKlwl4mJk3eidkgN0YoIkYow1fVV7H7Yzn7HkgGkZnD1mo8HVXezpRSamkrKRFVtJX95/RKe/vuLuIxjr+9MpHZwjd+xZAA586ZTOPug367qLvo/N53ic6KBSyMnRSRnbc1tzP9wESM2H05VbaXfcfqV3oyc1BV3P7W8qZk//uRa5s3+hMYDvsKUi7+nbn2ywarqqtjyq5qoym8q3P3U2Qf9lg/e/Ih0Z5qmjxeTyWQ45cof+x1LRPJAhbsfSqfTzJ35AStbwRLxJK8++rq/oUQkb9SrpB8KBoPUD6tb/TgUYOQ2G/uYSETySYW7n/r1g2dRN7SWQMDYdOuNOe26n/gdSUTyRE0l/ciKZa3Men4OtUNq2H7iNty14Do6E51EYhG/o4lIHqlw9xOLPm7ixMYz6Ux2kkln2POIiZxx/Ykq2iL9kJpK+onbf3cvK5a10t4Sp6MtwdN/f5HP5s73O5aIFIAKdz+RaE+uMXdJIGAk40kfE4lIofRYuM0sZmavmNkbZjbbzM4vRjDpncknTSJaGcUCRrQiwpjtRzJqu039jiUiBZBLG3cC2Mc512pmYeAFM3vUOTetwNmkF7bZaUuueOkCXrzvFWqH1HDgsftormSRfqrHwu28yUxasw/D2Y/8T3AiG2zzsaPZfOxov2OISIHl1MZtZkEzex1YBDzunHt5PdtMMbPpZja9qakp3zlFRCQrp8LtnEs7574KbApMMLMd1rPNVOdco3OucejQofnOKSIiWb3qVeKcWw48A0wqSBoREelRLr1KhppZffbzCmA/4D+FDiYiIuuXS6+SEcDfzCyIV+jvdM49VNhYIiLSnVx6lbwJjCtCFhERyYFGToqIlBkVbhGRMqPCLSJSZlS4RUTKjAq3iEiZUeEWESkzKtwiImVGhVtEpMyocIuIlBkVbhGRMqPCLSJSZlS4RUTKjAq3iEiZUeEWESkzKtwiImVGhVtEpMyocIuIlBkVbhGRMqPCLSJSZlS4RUTKjAq3iEiZUeEWESkzKtwiImVGhVtEpMyocIuIlJmQ3wFEisl1vo1ruwEsiFWdgIU28zuSSK/1eMVtZiPN7Gkzm2Nms83s1GIEE8k3l5qHW3oUdDwA8ftwSw7DpRfl/PpM+/1kFu5CZuEEMm03Fi6oFFQm/k8yy88gs+JKnOvwO06f5HLFnQJOd87NNLMaYIaZPe6ce7vA2UTyK/EMuHT2gQOXgeS/oWJyjy91nXOg5f+A7C/6ij/gQltj0YmFSisFkGm/B1rOw/t/jOKSr2BDbvY5Ve/1eMXtnJvvnJuZ/XwFMAfYpNDBRPIuMAgIrn5sQGBwbq9NvQPW9delE1Jz8hguv1xqLi45o2yvKAum/TZW/fElAZ0zcJlmPxP1Sa/auM1sDDAOeLkQYUQKKnYwxO+D5AzvcXRviOye22tD23pX6CtZGELb5T9jHmRaLoL2W8GCYHXQcA+W6x+o/i4wGO96deX/pYFV+Biob3Iu3GZWDfwD+JlzrmU9X58CTAEYNWpU3gKK5ItZCAbdAOkPgSAER2Fmub02vC2u9tfQepHX3FJ9IhbdtbCB+8ClPob2W4AEOMAlcK3XYLVn+x2tJFjtL3FL3gA6wXVC7TmYRfyO1Ws5FW4zC+MV7Vudc/esbxvn3FRgKkBjY6PLW0KRPDIzCG3ep9cGKidDZc/t4b5yrd6V9qrfwBRklvuZqKRYaDQMfQrS70NgGBbcyO9IfZJLrxIDrgfmOOcuK3wkEemz0FYQGAGEs0/EsMoj/ExUcixQhYXHlm3RhtyuuHcDvgfMMrPXs8+d7Zx7pHCxRKQvzMIw5E6vr3pmKVYxGYuM9zuW5FmPhds59wLe/XcRKQMWqMFqNNyiP9PISRGRDeCcw8Xvho5HvBveNadhgbqCHlOFW0RkA7j2O2DFhUAceAXX+RoMuT/nHkt9oUmmREQ2RMe9eEUbvIFZH0BmSUEPqcItIgA4l8J1PIWLP4zLrDNUQ7oTHMEaI3IxCFQX9JBqKhERnEvjlv4AUm95T1gVDHkACw7xNVc5sJqzcZ1vQnoJkIG6CzGLFfSYKtwiJcBlWsC1QWCjgraNdqtzple0XXs2UBLXfgdWc2Lxs5QZCw6Dhn9B+nMIDMYKfLUNKtwivsu03QgrLgYC3gCawTdjgarihnCdrNnr1+FNDCq5MAtBqHhTfaiNW8RHLj0fVlwKdAIJSL2La7u2+EEijRDcFKgAImBVWMWhxc8hOdEVt4ifMk3eTIMukX0iCenPih7DLAJD7oT4A+DiEJtU1kPC+zsVbhnwXOpjXPx+74ZS5REFHzyxhtBW3rSirh1vqtEYFjuoeMfvwqwCKo/05djSOyrcMqC51Ce4Jd8E144j6M1j3fBw0dqYzSpgyN24FRd7fX8rjsJiexfl2FK+VLil33LpxbjWSyA9H2KHetOyrr1Nx0Ne0wAZ78M1Q/JliO1TtJwWHIHVF3fiTecyuNY/QvxeCAzB6n6Lhb9U1AzSdyrc0i8514lbeqRXtElB8jUyFiBQcQguvRAyCyG4BWYVOEJAl7UoC9wHtxS49tug/Ubvj1ZmAW7p92Doc1igsm/7c3Hcissh9S5E98Qqv49Z/+774FzSW7MUB5GJRV2QQYVb+qf0J5BZzOoubR0Qf5CMS3qLxVoYCMOg6yG4UXZbB+FxENnFt9hFk3wx+05jpbR3zgLb9Gl3btlPskvCJSA5E+daseqT8xK1FDmXxC05AtLzAIPgJjDk7oIPvFmpf/9JlIErMKjLiu4AYa9At5yLt6xXq9cs0voHrOFBrP6P2KCp2KDrS+pK0TmHSzflf9Hf8PZAtMsTmezQ7d5zrhOS04CVPWPiEL9/AwOWuI5/eUXbtXsDp1KfQMejRTt86fyEiuSRBQZB7TlABIhBcDRUHc/qJhGADGSWYBbDontgkQmlVbQzLbgl38Q17Y1buBOZ+MN527dVTfEWSyYIVo/VX40Favu4t9BaC+4GIDA8DylLWRJc1xUaXZcunYWnphLptwKVR+BiB0KmGYIjMAuSiUyA5Ey8q8MKqDza75jdcisuhdRcvME5QPOZuOieeRlSbRbBBl3xxcd3SSDgjQr8wn0Z1P8Rt+wUMAOrxOp+s8EZS1p0HwhcAplsU5xVQuyAoh1ehVv6NQvUQKBm9eNBU3Ft10HqQ4juS6DiQB/T9SD9MauKNgAByCwt+MxzzjlcyzkQvwsI4Kp/RqB6yhe+xqJ7wLCXvHsFwRFluXJ6b1igHoY8CB33AQ5ik7HA4KIdX4VbBhSzKFZ9kt8xchM7KPvuIA4EITgEghsX/rgdj3gjKFd2kWy9GhediIV3+MKXWaAKij3Hio8sOASqjvXl2CrcIiXKKr6Ncylvov7gxljN2T02W+RF+mNW32gELOD1OOmhcEvxqHCLlCgzw6qOgqqjinvgyAS8m7ore7JkIDy2uBnkC5XOLXQRKQkW2RHqfg+hbSE81usiGdzE71jSha64pSRl2u+Etr9CoBqrPbfH9lXJr0DFJKiY5HcM6YYKt5Qc1/EUtFwAxCFNdjj2k0W9a98fudSH3sCRQA1UHFq0UX6SfyrcUnJc4gVWr5oNEIDOORDdza9IZc91vodbenh2kEgY2u/MDtFWCShHauOWkmPhrYAuV4MuCcGRvuXpD1z77dk5v9NAhzdcu/Mtn1NJX6lwS+mpOAJi++H9eEag9jysiOv59UsWA4JdnnDQzwfJ9Gc9Fm4zu8HMFpmZ/jxLUZgFCdRfhg2fhQ1/k0DlYX5HKntW9QMI1INVecOzI3tASPNvl6tcGrhuBK4CbipsFJE1mYX9jtBvWHAYNDzmLRJhNRDZxZtjRMpSj4XbOfecmY0pfBQRKSQL1BV1IiQA1/kfXPuNQBirOk5NXnmiW8oiJcplWr2Z9kpoqtnecKl5uKXfyd4UDeA6HoWGx7Bgg9/Ryl7efiLMbIqZTTez6U1NTfnarYivnEuTaT6bzIIdyCycgOt4uvDHTDeRaToQt2gn7yM5o+DHLIjE0+BWzm6YAZfKLvUlGypvhds5N9U51+icaxw6dGi+diviK9d+K8Qfwps4fzlu+am49OLCHrPlHEh/hDf6aAVu2fG4NSbtLxOBOujaT9zwbpDKBivP92AixdL5FqsnW8IrROmPC3vM1IesXisTb5k1ire6St7EDoHQl7Or48S8niyR3f1O1S/02MZtZrcDewENZvYpcK5z7vpCBxMpCZFdoOOfrB7J6SC0eWGPGd0H2j/H+4MRgtBWZTk83SwMg2/yVvGxIAQ3V0+WPMmlV0mR55QUKR1W8S1c5jNo/ztYLVZ3obf6SSGPWXMajgAkn/GKXd15BT1eIZkFILx1Ttu6jidwnW9h4e2x2P4FTlberBBtZ42NjW769Ol536+I9E+Z1j9D6zV472wqoHoKgW5WKnKZdtzyn0LyRQgMxuqv8KaiLXNmNsM515jLtmrjFhH/tf2V1c1R8ezj9XOtl3oDiUhDpgm37LjswsYDhwq3iPhv7XlTLNr9tp2zWeNmrUt5iygPICrcIuI7qz0PiIFVe//WnNf9xpGJrJ490rxV7wMDa1CPRk6KDEAuvQAyiyG0ZUn0WLHYftDwEKTe9XrRhEZ3v231iTjX7C0KEdwIq7towM0rPrC+WxEh03YjrLjU65Nu1TDkLiy4kd+xvHlMcpjLxCyE1f4f1P5fEVKVJjWViAwgLtMMKy4GEuDaILMYt+JSv2NJL6lwiwwkrpU1F1RIe00mUlZUuEVy5E1Regcu+arfUfouMCI78nNlL44YVBzpZyLpA7Vxi+TAdTyJW/7f3gMzXNVJBKqn+BuqD8wCMPgWXNtfIP05Fvs6Ftvb71jSSyrcIjlwK37PqsmmHNB6Ba7quLKce8MC1VjN6Tlv7zIt4OIQGFaW329/pKYSkZysPTVEGU6z2geZ1r/gFu2Ka9oXt/S7OFeGsxT2QyrcIrmoPh1v0EcUby6NE/Ny9elchkzrVWSaDiaz9Hivf3WJcKkPoPUqoBNIQuebuLab/Y4lqKlEJCeBiq/hQptA8jUIbYFFJ+Zlv67tOmi9FohD+gPc0qOh4YnSWK4svQgsDKuushOQ/szXSOJR4RbJkYV3gPAO+d1p4glWT66UhnST1z0vOCy/x+mL8PZAGO+NeQaIYUVebFjWrwT+rIsMYKEt8YpjlgVLZnkvC9RgQ+6C6P4Q3gWrvxyL7up3LEFX3CK+spozcZ3vQOptsChWfzW29kx5PrLQKGzQlX7HkLWocIv4yAK1WMM/cJl2sFhptG1LyVPhFikChxi8AAAHLklEQVQBFqj0O4KUEf15F5H1ci7tDfNPfeh3FFmLrrhFZB3OdeCW/Bek3gcyuIqvY7UXaORkidAVt4isq/1Ob1ED2oEOiD8Mna/5nUqydMUtIutwmeVAlwV4LQCZZt/yrOQyrdA5E6wOwmMH7DsAFW4RWYfFDsC1XY83OCgIRCGyo6+ZXHoBbsmh4DqANMQOwep+42smv6ipRETWYeFtsSG3QMW3oOI7WMN9WKDW10yudSpklnmLQbg4xO/Hpeb5mskvuuIWkfWy8Jexuov8jtFFG5Be/dCC4Np9S+NcBrfiYui4HwKDvUWLw9sX5di64haRsmAVRwMVgAFRCI6B0Na+5XHtN0H7bd7cMql3cUu/j8u0FeXYORVuM5tkZu+Y2VwzO7PQoURE1maRr2BDboeq47Ga07Aht2PmY6NB8mVWTxAGkIH0p0U5dI/ftZkFgauB/YFPgVfN7AHn3NuFDici0pWFt8PC2/kdwxMeC4kXWbUyEhkIblyUQ+dyxT0BmOuc+8A5lwT+DkwubCwRKTWu810ySw4n07QvmdapODcwVgHqjlX9GGL7AxEIDMXq/4IFaopy7FzeZ2wCfNLl8afAzoWJIyKlyGXavEUeXIv3ROvVuMBgrPLb/gbzkVkYq78UuLTox87lint9PdzX+VNrZlPMbLqZTW9qatrwZCJSOtLz8BZTWCkOied8CiO5FO5PgZFdHm8KfL72Rs65qc65Rudc49ChQ/OVT0RKQXAEuFSXJ2IQ3ta3OANdLoX7VWArM9vMvBnevwM8UNhYIlJKLDAY6i4DqwICEN3da+MVX/TYxu2cS5nZycA/8ca+3uCcm13wZCJSUgIV++FiM4EMXmcz8UtOnSCdc48AjxQ4i4iUOG9SJxVtv2nkpIhImVHhFhEpMyrcIiJlRoVbRKTMqHCLiJQZFW4RkTJjhZgoxsyagI/yuMsGYHEe95dvyrfhSj2j8m24Us/od77Rzrmchp0XpHDnm5lNd841+p2jO8q34Uo9o/JtuFLPWOr5ulJTiYhImVHhFhEpM+VSuKf6HaAHyrfhSj2j8m24Us9Y6vlWKYs2bhERWa1crrhFRCSrJAu3mQ02s8fN7L3sv4O62S5tZq9nPwo+R3hPq92bWdTM7sh+/WUzG1PoTL3M9wMza+pyzoo6obKZ3WBmi8zsrW6+bmZ2RTb/m2Y2vsTy7WVmzV3O3zlFzjfSzJ42szlmNtvMTl3PNr6dwxzz+X0OY2b2ipm9kc14/nq28fX3OCfOuZL7AH4PnJn9/Ezgom62ay1ipiDwPrA5EAHeALZba5sTgWuyn38HuKPE8v0AuMrH/9c9gPHAW918/SDgUbzl8nYBXi6xfHsBD/l4/kYA47Of1wDvruf/2LdzmGM+v8+hAdXZz8PAy8Aua23j2+9xrh8lecWNt4r837Kf/w34po9ZVspltfuuue8G9jVvAuNSyecr59xzwNIv2GQycJPzTAPqzWxEcdLllM9Xzrn5zrmZ2c9XAHPwFvPuyrdzmGM+X2XPS2v2YTj7sfaNPj9/j3NSqoV7uHNuPng/DMCwbraLZRconmZmhS7u61vtfu0fylXbOOdSQDMwpMC51jl21vryARyWfQt9t5mNXM/X/ZTr9+CnXbNvsx81s+39CpF9+z4O74qxq5I4h1+QD3w+h2YWNLPXgUXA4865bs+hD7/HOclpBZxCMLMngI3W86Vf9mI3o5xzn5vZ5sBTZjbLOfd+fhKuI5fV7nPZplByOfaDwO3OuYSZnYB3VbFPwZPlzs/zl4uZeMOSW83sIOA+YKtihzCzauAfwM+ccy1rf3k9LynqOewhn+/n0DmXBr5qZvXAvWa2g3Ou630N389hT3y74nbO7eec22E9H/cDC1e+vcv+u6ibfXye/fcD4Bm8v/CFkstq96u2MbMQUEfx3nr3mM85t8Q5l8g+vBbYsUjZcpXLOfaNc65l5dts5y3nFzazhmJmMLMwXlG81Tl3z3o28fUc9pSvFM5hlyzL8erGpLW+5OfvcU5KtankAeCY7OfHAPevvYGZDTKzaPbzBmA34O0CZspltfuuub8NPOWydziKoMd8a7V1fgOvDbKUPAB8P9szYhegeWWTWSkws41WtnWa2QS8358lRTy+AdcDc5xzl3WzmW/nMJd8JXAOh2avtDGzCmA/4D9rbebn73Fu/L47ur4PvPakJ4H3sv8Ozj7fCFyX/XwiMAuv98Qs4Ngi5DoI7075+8Avs8/9CvhG9vMYcBcwF3gF2LzI562nfBcCs7Pn7Glg2yLnux2YD3TiXdUcC5wAnJD9ugFXZ/PPAhpLLN/JXc7fNGBikfPtjveW/U3g9ezHQaVyDnPM5/c5HAu8ls34FnBO9vmS+T3O5UMjJ0VEykypNpWIiEg3VLhFRMqMCreISJlR4RYRKTMq3CIiZUaFW0SkzKhwi4iUGRVuEZEy8/8BeoeRtnaL3x0AAAAASUVORK5CYII=\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "import numpy as np\n", "import matplotlib.pyplot as plt\n", "%matplotlib inline\n", "\n", "from sklearn.datasets.samples_generator import make_blobs\n", "X, y = make_blobs(n_samples=60, centers=2,\n", " random_state=0, cluster_std=0.60)\n", "plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The following function takes a model as input and plots the decision function." ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "def plotDecisionFunction(model, ax=None):\n", " \"\"\"Plot the decision function of a SVC model in 2 dimensions\"\"\"\n", " \n", " if ax is None:\n", " ax = plt.gca()\n", " xlim = ax.get_xlim()\n", " ylim = ax.get_ylim()\n", " \n", " x = np.linspace(xlim[0], xlim[1], 30)\n", " y = np.linspace(ylim[0], ylim[1], 30)\n", " X,Y = np.meshgrid(x, y)\n", " xy = np.vstack([X.ravel(), Y.ravel()]).T\n", " P = model.decision_function(xy).reshape(X.shape)\n", " \n", " # Plot decision boundary and margins\n", " ax.contour(X, Y, P, colors='k',\n", " levels=[-1, 0, 1], alpha=0.5,\n", " linestyles=['--', '-', '--'])\n", " \n", " # Plot support vectors\n", " ax.scatter(model.support_vectors_[:, 0],\n", " model.support_vectors_[:, 1],\n", " s=30, linewidth=1, facecolors='none', edgecolor='red');\n", " ax.set_xlim(xlim)\n", " ax.set_ylim(ylim)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Task**: Implement a maximal margin classifier for the test problem from above.\n", "You can use the function `SVC` from `sklearn.svm` with `kernel='linear'` and a parameter `C=1e10`. We will see in a minute, that the `sklearn` parameter `C` behaves like the inverse of the tuning parameter $C^{-1}$ from the lecture." ] }, { "cell_type": "code", "execution_count": 7, "metadata": {}, "outputs": [], "source": [ "## TODO: Train your model\n", "\n", "#plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis')\n", "#plotDecisionFunction(model);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Solution**:" ] }, { "cell_type": "code", "execution_count": 8, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XecVNXZwPHfmT6zvcHSuwgoIL0YCwoiiMQYscQSG5aYGDXmtbwxmuZrTWLUJPao0VhBYyyxgg2UEqkRkCKdXbbv9Jnz/jE7w87W2WVn78zu8/189jPtzr3PXNhnz5x7znmU1hohhBDpw2R0AEIIIdpGErcQQqQZSdxCCJFmJHELIUSakcQthBBpRhK3EEKkGUncQgiRZiRxCyFEmpHELYQQacaSjJ0WFhbqgQMHJmPXQgjRJa1cubJUa12UyLZJSdwDBw5kxYoVydi1EEJ0SUqpHYluK10lQgiRZiRxCyFEmpHELYQQaUYStxBCpBlJ3EIIkWYkcQshRJqRxN0Gm1Z+wxXH/Ixz+13B325/EakeJIQwQlLGcXdFtVVufn7yr6itdAPw8r2vU9gnj7mXzzQ4MiFEdyMt7gTt3rw3roXtdftY/f46AyMSQnRXkrgT1HNAEaFgOPbY7rQxbNwgAyMSQnRXkrgTlFOYzW0v3UBWfiZmq5npZ0zm+9fPMzosIUQ3JH3cbTDp1GN4tfRJo8MQQnRz0uIWQog0I4lbCCHSjCRuIYRIM5K4hRAizcjFSZFy/L4Ae7bsI69nDjmF2UaHY6hdm/ey5MXPcGU5mX3pDJwZDqNDEilAErdIKSW7DvKTabdQW+EmFAxx7Z8XMuuiE4wOyxDf/nc310y6CZ/Hj8Vq5s3H3uPhFXdhtVmNDk0YTLpKREr5y/VPUba3Ak+NF783wB+ufASfx2d0WIZ46/H38dR6CYfC+L0B9u8o4b/LtxgdlkgBkrhFSindU044FI57zl3lMSgaY9mdNsxmc+yxDmtsDmltC0ncIsWceukM7C4bAFabhYFH9SO3R47BURlj/jWnklOUjTPLiSPTwbiZozliwhCjwxIpQPq4RUqZffEMrHYrS178jN5Di7ngtrNQShkdliHyeuTwxMY/8NVH63FlORl9/Mhuey5EPJXImtJKqe1ANRACglrrCS1tP2HCBL1ixYoOCVCIdLfnm308+JMnKN9XzqmXncy8K2dJAhaNKKVWtpZbo9rS4j5Ra13azpiE6JZ8Hh/XTv9fKkur0GHNIzc+gyvLwcnnH290aCKNSR+3EEm0e/M+/B4/Ohz5Zutz+/h08ZcGRyXSXaKJWwP/VkqtVEotTGZAQnQl+b1yCQZDscdWh5V+w3sbGJHoChJN3NO11uOAU4EfKaWOa7iBUmqhUmqFUmpFSUlJhwYpRLrKLcrhhseuxGq3YraYOXLSMM679UyjwxJpLqGLk3FvUOp2oEZrfW9z28jFSSHihYIhfB4/riyn0aGIFNWWi5OttriVUhlKqazofWAWIMUWhWgDs8UsSVt0mERGlfQEFtUNX7IAz2mt305qVEIIIZrVauLWWm8FxnRCLEIIIRIgMydTQOmeMp6/cxG1FbXMu+oURk0bbnRIQogUJonbYJ4aDz+aeBOVByoJhcJ88upy7v3wdo6cNMzo0IQQKUom4Bhs4/IteGu9hOpWxPN5/Hz04mcGRyWESGUpk7jXfryRs/tczhznufz67Pvx+wJGh9QpsgsyCdWfoGG3ktdNV8MTQiQmJRK3p9bLraf9jrK9FQR8QZa9sZK//+YVo8PqFEPHDuLUS0/C5rDiyLDT78jenH71KUaHJYRIYSnRx31wd1lkUn0dv8fP1192n0ofP/rjJXzv2rm4qz0MHNUPs8Xc+puEEN1WSiTuHv0LsdgOhWJ32Rl74lEGRtT5eg3uaXQIQog0kRJdJTaHjXs/uJ0hYweS1zOHuZefxFk/m2d0WCJJ/F4/JbsOEgqFWt9YCNFISrS4AQaPHsBfVt1jdBhprWxfOfu2HaD/iL5k5mYYHQ4AWmtef/ht3n1mKT36FzJ9/kT+cOUj6LAmt0cO9y/9FT36FTZ6XzgcJhgIYbNLjUUhGmrzIlOJkEWmOt8ni5bzf+c/gNka6R+/5/1fcsR44+sT/uvRd/nzdX/D5/ahTAqtdex6hslsYuKpx/Cb12+Ke8+Slz/n7oseJOgLMG7mGO5YdCM2hy2h4/l9Afwef8r84RIiUR26yJRID/dc/BA+jx93lQd3lYffX/FXo0MC4OOXl+Fz+4BIlfL6F6HDoTD7tx+guryGt5/8kPeeXcr+b0u4+6IH8Xv8hMOaNUvW84//W5TQsd56/H2+m3MhZ/W8lBtPugO/15+MjySE4VKmq0S0n9Yab60v7rmaslqDoonXe2gv1izZQMAfBEAphdlqJugPYnfZmXLaeC476npqK90oBXk9czFbDrUn/N4A29fvavU4JbsO8uCPH48dZ8PnX/Piva9z/v9+PzkfTAgDSeLuApRSnHD2ND5d9AU+jx+Hy87cK2YaHRYAl/z2XDYu28T29TsxmRQ/+fPlbPhsE99u3MXU0yditZmpKa/B741MuCo/UIlSkc+ktY4k93njWz1O6e4yLDZLbD9+b4Ddm/Ym9bMJYRRJ3F3Ez5+6hsXj32TL6u2Mmzmak89vVKSow/h9Ab7+Ygs2h5UjJgxpsWJ5Zm4GD6+4i6qD1TiznNjsVk656MTY64seeDNuewVc8tvz+OTV5ZQfqGTOZScx84LWC+sOHNUXq80S6UcPa+wuG8d+b3K7P6MQqUwuToo2cVd7+PGUmynZdRAd1kw4ZSy3vXRDi8m7JZWlVSwc87NYV0nPAUU89OX/YXfa27yv3Vv28vBPn6KipJL5P5rNrAtPaNP7P3vtSzYu28TQYwZx3FlT2/2ZhGiPtlyclBa3aJO3HnuffdsOxLokVvz7K9Z+vJHRx41s1/5yCrN5dO19fLb4S8xWM985c0q7kjZAn6G9+O0bN7frvS/c8xrP3vESXrcPR4adLV9t59LfnteufQmRbDKqRLRJbZWbYN0FQACTUniqPYe1z+z8LGZfMoOZFxyPw9W+pH24Xrn/n3jrRr94a3289qe3DIlDiERI4hZtcuI507G77KDAbDXjynZydDtb26nE5owfJ251yMQfkbqkq0S0Sb/hffjTst/x9pMf4nDZmX/N7C5RBPfahy/nju/fh8VqIhgI8ZOHLjc6JCGaJRcnhaizf0cJW9fsYMDIvvQeUmx0OKKbkYuTQrRDzwFF9BxQ1OzrWmvee3YpH7+8jN7Dirnwlwu6xLcNkX4kcR+m0t0H2bhsMz36FzJ84lCjw0k7r//5Hf7+65ex2Cxc/YeLmf7dSUaH1Kx/Pfoef7k+su6K1W5h/adf88Bnv03KsMGyfeW898xSlFLM+uEJ5BRmd/gxRPqSxH0YNq/ayg0n/BJlUoSCYc762Twuuv1so8NKG6veX8sjNz4TW8vkzh/8kYdW3MWAEX0Njqxpbz/+QSzWgC/IltXbqC6rIbsgq0OPU1FSGRnbXlELSvHy79/gsXX3k5WX2aHHEelLRpUchkf/51k8NV7cVR58bh/P37kIT63X6LCSprbKzbtPL+HdZ5bgPswhgACbvtxCoF5tUZPFxDf/2X7Y+02Wwj55mMyHfmWUUjgyHR1+nE9e/QJvjZdgIETQH8Rd5WbZP1d2+HFE+pIW92HwexqvPhcOpkdxgIN7y6mpqCW3KBulVKutxtoqN1cecyMVByoBeOaOl/jL6nsOq4930OgB2BzW2AJZ4WCYgaP6tXt/yXb1Hy7mv19swV3lIRQMcd2jVyZlvXCr3QL1u1+UwibDE0U9krgPw7m3fI9fn3UfAV8Am9PG5LnjychJ/XWgn7vzVZ791cuEQyFCwTAWq5ljvzeZm/9+LSZT01/CPnl1ORUHKmNJtnx/BZ8u/iKhdUSaM3nOOM69+Qz+cddizBYzV95/EYNHD2j3/pKtR/8inv7mIfZ+s4+8nrkd3kUSdfyCabx03z858G0JaOg7vDdT509MyrFEepLEfRgmzxnHPR/czqp319BjQCEzzjvW6JBadeDbEv7+65fjuiiCgRDL3ljJ0pc+54Szp1NbWcu//7aEgC/AieceS1HfApocNdoBI0nPu+VMzrvlzMPfUSex2a0MGJncbwUOl52Hv/w/Vr+/FmUyccxJR2G1SYtbHJJw4lZKmYEVwG6t9WnJCym9jJg8jBGThxkdRsIqS6sxWy3gDcQ9HwwEKdl5EE+tlyvH/ZyyveWEQ2Gev3MRj6y5j++cOZlnfvVibPvcHjlMPyN1R4CkO5sj8g1OiKa0pcV9LbARkHFJaWzAqH5k52fi8/gIB8Ox580WM+Nmjmb1e2upLK2KLSLlrfXywXOfcPbP5/PIf+7l41e/QCk49nuTZQyzEAZJKHErpfoCc4HfAtcnNSKRVDa7lT8tv5Onf/kCO7/eg7vKQ0auix/ceiZDxgykbG95XBeIMpkiF8uAjJwMZl98YjN7FkJ0lkRb3H8Afg40ezVGKbUQWAjQv3//w49MJE1ejxyu/fPCJl8bd/JohhwzkG9WbwcVWXb1lB+e0KnxCSFa1mriVkqdBhzQWq9USp3Q3HZa60eARyCyVkmHRSg6ldli5t73b2fVe2vwewOMmzkaZ0bHj1UWQrRfIi3u6cDpSqk5gAPIVko9q7U+P7mhCaOYLWYmzj7G6DCEEM1odeak1vpmrXVfrfVA4BzgA0naojso21fOj6fewhzneVw57sbIuGohUoBMeReiGb87749sXvkNAV+AbWt2cNsZdxsdkhBAGyfgaK0/Aj5KSiRCJGj7+p2898wSXNlOTr96Npm5yZmtum3tDkJ1QybDYc3OjXuSchwh2kpmToq0sn39Tn485Wa8bh8Wq4W3n/iQR9fe1+4Cwy05cvIwVr23lqA/iNliZui4QR1+DCHaQ7pKRFp59+mPIuulaAj6g1SWVLFx2eakHOumZ37CuJNHk1OYxejjR3L7Kz9LynGEaCtpcYu04sxyYrFZYpXmw+EwziQsrQqQlZfJb9+4OSn77iwfvfApf7nhb4RDYS745VnMu/IUo0MSHUBa3CKtzP/RbAr75OPKcuLIsDN57niOmDDE6LBS0ra1O7j30oc5uKec8v2V/PVnz/DVkvVGhyU6gLS4RVrJysvksXX3s/6zTTgzHRw5aWhSSod1BVtWb49bpjcUCLLpy28Yc/woA6MSHUESt0g7dqedcScdbXQYKW/Q0f0Jhw4tJGaxWRhyjFxg7Qqkq0Q0UravnOd+9wrP/volSnYdNDoc0U5DjxnEjx+6jOyCTDJyXFx4xwL5g9dFKN3kCvmHZ8KECXrFihUdvl+RfFUHq7l01E+pKa9Fa40r28Vj6+4nvzjP6NCE6DK01gQCAdxuNx6PB4/Hw5AhQ1ZqrSck8n7pKhFxlv9rFd5aH8FApHamz+Pn00VfMO8qGY0gRFPC4TAejwez2YzD4cDv97N+/Xo8Hk8sMbvdbsaOHcvw4cM5cOAAjzzyCMFgsN3HlMQt4thdtriLfSaTwu7q+MktQqSyvXv3Nkq8xcXFDB8+nEAgwFNPPRV73uv1AnDccccxY8YMAoEAr732GgAmkwmXy4XT6cTj8QCQmZnJ5MmTY89Hb++4446E45PELeJMPX0CA4/qx/Z1O0FB7yHFnHD2NKPDEqJNoq3g+onXbrczaFDk4uw777xDRUVF3DZDhw5l/vz5ADzxxBMEAvHl/SZOnMjw4cOxWCw4nU7y8/PjEm/fvn0BcLlcXHvttbhcLmw2W6NRTy6Xi5kzZx7W55PEnUZ8Hh/L3lgFWjNp7rikrJNttVm5f8mvWPvxRnRYc/RxI6RQrYgIh+Hdd2HdOhg9Gk46CUzJHd9Qvy84GAxSWFgIwIYNGygpKYlLvFlZWZx++ukA/PWvf2X//v1x+xo4cGAsce/atQuv14vL5SI/Px+Xy0W/foeKQC9YsACbzRaXmM1mMwBKKc4/v/kFUpVS5OUl95qQJO404XX7uGbSTez/thQF5BXn8pdVd+PM7Pi6jxarhWNmyOgDUY/Hg54zh4pNO/ivqw8j3A+RM2Iw6o03wJFYAyIcDsfGlZeWllJaWhqXeEOhEKecErmW8uabb7Jx40bcbjehUOR6S25uLj/96U8BWLlyJd988w12uz2WXLOyDhXomjp1Kn6/Py7xZmZmxl6/9NJLW4x12LDULgAuiTtNfPbal+zfURJZpwMI7ylnyYufM/uSGQZHJrqFhx9m67YyfloxFe9eP3ZHH363aQ1Fd9+N+wc/iCXg8ePHYzab+eqrr1i7dm1cYvb5fPziF7/AZDKxbNky6o88M5lMZGVlMWvWLJRSFBYWMmzYsGYT74IFC7BYLLFWcENjx45N+ikxkiTuNBEKhho8o+MmVwjRHg37gnv27Indbmfv3r2sX7/+UB/xk0/y6h4/1YEarMrGfs9+rt/n5thnnol0odQ58sgjyc7Oxufz4Xa747oinE5nrNU9bdo0xo0bF3u+YV/wpEmTWozbbu/eF8wlcaeJafMn8tQv/oEOazSazNwMjj1zstFhiRQR7Qs2mUxYLBZqa2vZunVrbIxwNAFPnTqVXr16sXnzZl555ZXYiIioSy65hP79+1NSUsLnn39+qMWbmUlfeyUb667XOcnkeFceZ40agPPii2MJOCMjsjb6pEmTWky++fn5STsX3YEk7jSRke3ir/+5lw//8Sk6rDnhnGlk52e1/kaRlkKhEGVlZY1GRgwcOJA+ffpQXl7O4sWLG/URn3HGGYwZM4aysjJeeeWV2P7sdjsul4sxY8YAkJOTw5gxY+K6IlwuF0VFRQAcddRRHH300YdawWPGcP78M7gjVMs6SzZTQ+XcZNmN9aY/w4ABnX5+ujuZOSlEkmit8fv9cck1IyOD4uJigsEg7777btzMObfbzYQJE5g+fTrV1dXcd999jfZ58sknc+yxx1JVVcUrr7wSS7rRxDts2DB69OhBIBCgoqICl8uFw+Foti+4TRYvJnzLLZg2biQ8ciSmO++EulEc4vAppRKeOSmJu4uqKKnkj1c9yvb1O5kwawwL77lAhvUdhlAohMfjIRwOk52dDcDatWupqqqKS8y9e/fmO9/5DgB33nknPp8vbj/jx49n3rx5aK25++67cTgcccl3xIgRjBw5klAoxIYNGxpN0mhqXHCnC4eTPgywO2pL4pauki7qljm/Y+uaHYQCIUq+LSUcDvPjP11mdFgppaSkhKqqqriuCIfDwZQpUwB4+eWX2b17N263O5aABw8ezIUXXgjABx98QHl5OWazOZZcCwoKYvufPn06ZrM5Lvnm5OQAkbG+//M//9NsbGazmaOPTtEhmZK0DSeJuwsKhUJsWbWV6Jcpn8fPl2/9x9igkiDaCq7f1RAIBGIJb/ny5Wzbti2uRexwOLjmmmsAeOutt9i6dWvcPvv27RtL3NnZ2ZhMprgWb/2Lapdccgk2m63ZVvBxxx2XrI8uujlJ3F2Q2Wwmt0cO5fsrI48tJvoN721wVM2L9gVHk2txcTEmk4kdO3Y0Srwej4fLLrsMpRRvvvkmK1eujNuXxWKJJe7y8nLKyspwuVwUFhbicrli3RwQ6S8OBoNxibl+4YFZs2a1GHf9CR9CdCZJ3F2J1rBzJ2Rn8+t/3sytc39H9cFq+h7Rm+sfu6pTQgiFQiilMJlMVFVVsWfPnriuCI/Hw4wZM8jMzGTlypV8+OGHsRERUT/72c/IzMxk69atLFmyBIfDEZdcg8EgVquVUaNG0bNnz0YjI7TWKKWYPXt2i7H27p26f8yEaIkk7q7is88IXXwJoZJSzH4fR3z/TF7a9jABkwWbw9auXYZCoUYX39xuN0cccQR5eXns3LmTJUuW4Ha742bHXXrppfTr14+tW7eyePHi2P6i/b1Tp04lMzOT3Nxchg8f3ijxRidXfOc73+H444+PawXXN3jwYAYPHtyuzyZEOpPE3RVUVBA+fT73BUbzKROx4+fXS9Yw/Oc/x/bgg7G+4GhyzcnJITc3l5qaGj7//PNGiXnGjBmMGDGCXbt28eSTTzY6XFZWFnl5eWit8Xg8sa6IhmtGDBs2jCuuuCL2vNVqjesLHjJkCEOGNF/o12KR/55CNEV+M9JItC9Ya43D4YgNGXO/8AKbbXn8ozxAMLyFDLL4dXggDz/5JA/17o3P74/bz8yZM5k+fTp+v59ly5bFtXYLCwtx1C0aVFhYyPz58xsNSXM6Iwtb9e/fn8svv7zZeDMyMmIz6YQQHUcSt8FKS0upra2Nu/iWm5vLqFGRStxPP/001dXVsddCoRATJkzgtNNOQykVmR23ejX7vW6qw+WYMGPHid9kxen1c8yYMTgzM+OSbnR2XF5eHv/7v//b7LjgjIwMjjnmmE47F0KIxLSauJVSDmApYK/b/mWt9S+THVi6aDgiwu12YzKZYn2vS5cupaSkJO713r17s2DBAgCeeeYZKisr4/Y5YsSIWOK22WyxERHRVm/0oprJZOKaa67BefAgtomTucY5jm1+BzabhUtztmOZNY/Zc+c2G7vhEzmEEO2SSIvbB8zQWtcopazAJ0qpt7TWy5IcW6eL9gX7fL7YRIotW7awb9++uH5gq9XKmWeeCUQSb8OxwD179uSqqyKjOL799lvKyspiC/AUFhbSq1ev2Lbz5s1rNFbYaj00w/Gcc85pMebCwkIoLIQ//ZGHf/wT9vUZTFZtBZmF2agHH+yQ8yKESC2tJm4dmRNfU/fQWvfT8fPkO1B0Gr9SioqKCkpLS+O6IjweD7Nnz0YpxZIlS1i9enUsYUOklXvLLbcAxNYVjpYrcrlccdUtxo4dy9ChQ5tdN7ilShkAQ4cO7ZgPfdFFmOfNo8/HH0NBAUybJjPchDBYw2ruzd1G61EmKqE+bqWUGVgJDAUe0lovb2KbhcBCiFy06igNR0REb0eOHInD4WDTpk2sWrWq0es33HADGRkZrFq1iqVLl9aPE4fDwUknnYTNZiM7O5v+/fs3ugAXHQs8d+5c5s2b12hERNTo0aM77LMetvx8qKuZJ4ToWNG1y1tKwE3d1p+j0FB01cboBf9EtWmRKaVULrAI+LHWel1z2zW1yJTWGp/PF/sw0UKbpaWljSpluN1u5s+fT3FxMatXr45VTK7vyiuvpLi4mK+++orPPvus0VjgadOm4XQ6KS8vp6amJva8w+FodlywEKLra9gKTjQBN1y7vL761dwTva1fxxKSuMiU1rpCKfURMBtoNnFXVFSwY8cOBgwYwK5du3j++edjK6tFnX322YwYMYKKigqWLl0aNzsuMzMz1rrt168fc+fObfSho90RY8aMia0x3JS8vLykF+4UQhijPa1gj8dDMBhsdp/161g2Vc29qdvOXrUxkVElRUCgLmk7gZOBu1p6j8/no6qqCoDMzExGjBjR6C9Nnz59gMjst2gduqYUFhbGKjsLIbqmRPuCG962pRVcv4Raw3XMm6rmnspa7SpRSo0G/gaYARPwotb6Vy29R9bjFqL7aljHsqWWb3v6ghO9TYm1y9ugQ7tKtNZrAJmFIUQ3k4xWcHTt8pZawa31BQuZOSlEtxAKhfB6vUltBSeShNOtFZyqJHELkUaaqmOZyG3DEmr11a/gE70YJ63g1CaJWwiDJNoX3PC2pVZw/dFZDVdtbO5inLSC048kbiEOU3tawYn2BUeTa0FBAX379m2xJdxh1dxFypPELUQ99etYtmWCRltawQUFBdIXLA6LJG7RJTW1amMiybgtfcHSChZGkcQtUl5TreBEuiIOpy+4qYtx0goWqUISt+g0yRoR0bCCT0vJt6lq7kKkG0ncol1SoRXcVB1LIboDSdzdXGt9wc3dtrUVnMgUZWkFC5EYSdxdSHNrl7d2Ya7+qo0NORyOWGKNVvCRVrAQxpLEnYI6qxWcyOw4aQULkXokcSdZoq3ghrdtbQW3loSlFSxE1yGJO0H1K/hIK1gIYaRumbilFSyESAX1K/i0RVon7oZ1LBO9MNdSK7h+NXen00lRUVGzi/NIK1gIAZFcpLXGZDIRDAb59ttvcbvdcXln+PDhDBo0iNLSUp577rlW61i2JGUSd7JbwZmZmRQVFUkrWAjRqtra2kb5Jj8/nwEDBhAMBnnllVcavX7sscdy4okn4vV6efrpp+P2Z7fbyc/PZ9CgQTgcDvr06dMo/9xxxx0Jx5eUxK21pry8vE3rRLSnFdzaGhHSChai+2pYwcdkMtGzZ08APvvsMyorK+Py0cCBA5k5cyYAv//97xsVFB43bhwDBgzAbDZz8ODBRmuXDxgwAACXy8XFF1/cbB3LzMxMzjzzzMP6bK3WnGyP3r1764ULFzY+mFJxs+MSuZVWsBAiHA7j8/lwOp0A7Nixg7KysrjGX0ZGBjNmzADgqaeeYufOnXEzdQcPHsyFF14IwAMPPEBtbW1crhk8eDDTp08HYPXq1Y0ajBkZGdhstqR9xg6tOdke2dnZfPe735VWsBCiSbW1tVRVVTUqkzZ16lQAPvroIzZv3hx7zev1kpeXx7XXXgvAkiVL2Lp1K3ComnufPn1i+x86dCh9+vSJyz85OTmx16+55poWc9Exx6R2md2kJO7MzEzGjh2bjF0LIVJEUxV8hg0bhtls5uuvv2bTpk1xr3m9Xq677jqUUnzwwQesXLkybn8Wi4UpU6bEvl037IrIzs6ObTtv3jwg0i3R1KqNxx57bIuxp3sDMmUuTgohjFG/LzgzMxOLxUJpaSlbt25tdC3q9NNPJzs7m88//5x33nmn0b5uuOEGsrKy2L9/P19//XWjQsKhUAiLxcL48eMZNmxYo27RaAI+4YQTWow5Ly8vGacibUjiFqIL8vl87N+/v9GAgLFjx1JYWMi2bdt466234ropAC699FL69evH7t27efPNN4H4au6BQACAvn37csIJJzR5bQrguOOO47jjjms2vt69e9O7d+8kn4WuSxK3ECmoubXLi4uLKSoqorKykvfee6/RaK05c+YwevRo9u/fzxNPPBG3T7PZTL9+/SgsLMRut1NQUNBojkK0JTtixAiGDBnSbDX3fv360a+dp1J8AAAY+klEQVRfv045F6IxSdyie9m9GxYvBosFzjgDevRI+iFDoRBerxe3243VaiU3N5dQKMSyZcsaJebRo0czfvx4qquruf/++xvta+bMmRQVFaG1ZteuXbGkG61jmZ+fD0CPHj244IILmq1j2bt3b84+++xmY7bZbEkdQSEOjyRu0X289BL6isvg1H4Q1HDLTajnX4BZsxJ6u9Y61kcLsG3bNmpqauKSb48ePRg/fjwADz30EFVVVXFzFCZMmMBpp52GUor33nsvNiKiYR9vRkYGM2fObNQVEb1Al5ubGxth0RSHw8GQIUPadZpE6mt1HLdSqh/wNFAMhIFHtNZ/bOk9EyZM0CtWrOiwIIU4bLW16P590S8VwUgAhfvDIO4bavF+8hnuum4Jq9XKUUcdBcBbb73Fvn37DrWIq1YxoMdyLjirADKv5oFHK6ioqIgdwuFwMHLkSE4//XQA/vWvfzVKzEVFRRQXFwPg9/tljoIBwp53wPcumAegMi9HKYfRIQEdP447CNygtV6llMoCViql3tVabzisKIXoAA3XLvf7/QwcOBCAtWvXsmvXrshrK1ZQm2vDts7NRSNdgOalkgDbfbWoe+6Buhl1PXr0iCVur9eLUipSPMJ2EEdgGUUFIdAVUP17zj7jN1hcU5qt4DN37twWY5euiM4Xdr8KVbcDXsCO9n+BKnjG4KjartXErbXeC+ytu1+tlNoI9AEkcYsOFV2vJiMjA6UU+/btY8+ePY0uwC1YsAClFG+//TZffvll3Ow4q9XKrbfeCsDmzZvZtGlTpKvBZsNV6SEnxxnbdvoYJxNC5bjOPx/noEGxBBx1xhlnxO5rz2J0VTZod90zAYoLD6AyipJ7UtpJB7dAuBKso1KmRZkS3M8RSdoAPgisRIcrUaaclt6VctrUx62UGggcAyxPRjCia/F6vZSVlTUaCzxp0iRcLhdr167l888/b1TN/cYbbyQjI4MNGzawdOlSIH7t8mAwiNVqpX///o2mJbtcLrTWKKU444wz4roh9JKP4JMD6EkKgpphf1Zw8imoumnOLbIcCbregmbKCpaRHXm6Oky46i5w/x2UGVQOFL6KMuUbHVZqMOUDJiK9vgAKlLOFN6SmhBO3UioTeAX4qda6qonXFwILAfr3799hAQpjNazm7na76du3L1lZWezbt48vvviiUWI+99xz6dOnDxs3buS1115rtM8jjzwSl8uFxWJpsphw9OLf5MmTGTduXLN1LEeOHMnIkc0nz4bbq1deRV93HWrci2A2ww/OR913X0LnQVmPRGf/GmruAh2CzKtR9qkJvbcz6eC34H4W8IEGtA9d8xdU9i1Gh5YSVPat6INfAQHQAci+DaXSr8sqoUWmlFJW4A3gHa114zFKDcjFydRTvy84OqHC7Xazbt26Rol36tSpDBkyhB07dvDkk0822tfZZ5/NiBEj2Lp1K4sWLWrU4p06dSqFhYVUVlayd+9eXC5Xaq1dHv0/3wUvCurABnTZefW6dADHdzHl3m1cUClGh2sh9A2YeqDMxUaHE9OhFydVpNnyOLAxkaQtki/aFRAOh9m5c2ejxDtgwACOOOIIamtreeqpp2It5mhf8KxZs5g2bRoejyc2O67+qo3R5Szz8/M58cQTGyXm6FjhwYMHc8MNNzQbZ05OTtzCPimjCybsGMswMPWC0LdAAHCgXAuMjiqlKFMGmEYbHcZhSaSrZDpwAbBWKfWfuudu0Vq/mbywug+fz9co8WZmZjJo0CAAFi1a1Gis8JgxY5gzZw5a60YtYrPZjNVq5YgjjsBmszVZQq1v375AZL2HG2+8sdlWcFZWFscff3zyT4LoMEpZoeBFdO0TEC5DOeejbOOMDkt0sERGlXwCdOEmSseoX8FHax1bsH3VqlUcPHgwLvEWFRVx2mmnAYcmadR35JFHxhJ3aWkpQFwdy+iC7WazmYsuuiiu0k/9vmCr1dri7DiTyURGRkbHnghhOGXKQmU1PzlHpD+ZOdlAdKW06BjbPXv2UFpaGpd4zWYzp5xyCgAvvfQSW7ZsiZsdV1xczJVXXglEEne0n7f+uuRRM2bMQGsd93r9ZHr55Ze3GG80wQshjKG1RnteBu+bYO6Pyro+6cMLu3zi9nq9VFVVNRoLPG3aNJRSLF++nPXr18e9brVaufnmm4FIiaN169bF9hddIziqX79+ZGVlNTktGeCHP/whZrO52dlxsm65EOlNu1+A6jsBD/AFOrAaCl5L6ozYtEjcTVVz79+/Pzabje3bt7Nhw4ZG/cRXXHEFTqeTTz/9lI8//rjRPidOnIjNZiMUCmEymRrVsYxeADzppJNiy1c2VcFnypQpLcYeHdomhOiivIuIJG2AAAS3QvggmAuTdshOzyqhUAi3243D4cBqtVJZWck333zTKPGefPLJFBYWsmbNGhYvXtyomvtVV11Fz549KSkpYe3atbGkm5mZSY8ePWLbjxo1ip49ezZZzR1g2rRpTJs2rdl4u/uC7aL70DoIvqWgPWD/DsqU3fqbBJh7QWANEJ3Bq8CUmdRDJiVxV1dXs3fvXnr16sXevXt5/fXXY0nZ7/cDcO655zJ8+HAOHDjA66+/Hgmm3iy4aJ9xUVER06dPbzQkLZpQJ06cyMSJE5uNpbi4OLaojxCiaVqH0GU/hGBdt6DKgILXUeYCQ+NKByrrFnRgDYQOAmHIuTPpywwkPXHbbLZYK7h+8u1Rtw7ygAEDuO6665qt5t6rVy969eqVjDCFSBk6XAW6FkzFxqwWGFgVSdrRiTvaj3a/gMq6uvNjSTPK3AMK/w2hPWDKRyW5tQ1JSty9e/dm3LjI2NGCggJ+8IMfNLutLNguurtw7VNQfQ9gikygyX8mMkmkM+kA8aN+NZGFQUUilLKApfOW+kjvUsdCpDkd2gvV9xGZ5eiD4CZ07aOdH4htApj7Ak7ABioD5fxe58chEiJDHoQwUrgkstKgjs4D8ENod6eHoZQNCl4Ez+uRi5OO2Sm1joeIJ4lbdHs6+C3a81rkgpJrQeeuzWwZFllWVLuJLDXqQDnmdN7x61HKCa7mZ9qK1CGJW3RrOrgTffC7oN1ozJF1rAv/1Wl9zEo5oeBldPU9kbG/znNRjhM75dgifUniFl2WDpWia+6F0F5wfA+Ta37jbbxvRLoGCEd+dCX4l4NjRqfFqcy9ULmdu/Cm1mF0zR/BswhMBaic36GsIzo1BtF+krhFl6R1AF12diRpEwT/asLKhMk5Dx3aD+H9YB6CUk40Fg5NntDQDUp9afdz4H4q8kcrvA9ddgEULUWZXO3bn/agq/8AwU1gPx7luhCluvbYB6394P8c0GCb1qkFGSRxi64ptBPCpRwa0uYFzz8Ja3+kWKyyAlbIexzMxXXbarAeA7aWlzHoEvyf1n3TiApFzplpeLt2p8uvAv9KwAf+VWhdg8q8pkNCTUVa+9EHF0BoO6DA3AcKXu60+p5d+0+i6L5MeZESYzHWSIKu+iWRsl41kW6Rmt+jCv+Jyv0jKu8RVN7jKdVS1FqjQyVo7W1947awjgLs9Z4IR6Zut4PWAfAvA6IjYzzgaVyyrkvx/juStLU7MnEquBO8b3Xa4VPnf6gQHUiZ8iD7NsAGOMA8ADKu4FCXCEAYwgdRyoGyH4eyTUqtpB2uQh/8LrrkRPT+iYQ9/+qwfauMhWA/ETCDykXlPnQYa5NYGhTcNYGpZwdEmcr8h0rgAaDrDelMPukqEV2WybUA7TgVwpVg7oVSZsK2SeBfRaR16ATXeUaH2SxdfR8EtxCZnANU3oS2H98hU6qVsqHyHmj5+NoPmCKzAlvcl4LcP6LLfxwpC6dcqJzfHHaMKc0+A0z3QriuK065wDGr0w4viVt0acqUBaasQ4/zHkHXPgbBbWA/CZPzVAOja0WsbmSUCcJlSV95TmuNrroNPC8BJnTmTzFlLmzxPcp+HPT4LHKtwNwrLSunt4Uy5ULBP8G7GNDgmI8y5bf6vo4iiVt0K0rZUZk/MjqMxDjm1H078ABmMBeAuXfyj+t9MzKDMjpEsuYhtH0aynpUi2+LFOHtPqXwlLkAMi415NiSuIVIUcr5/cga2d5FYO6Nyrql1W6LDhH6lkMXGgFliow4aSVxi84jiVuIFKWUQmWcCxnndu6BbZOIXNSNjmQJg3V058YgWpQ6l9CFEClB2cZDzt1gORKsoyNDJM19jA5L1CMtbpGSwu4XofZJMGWisn/Zav+q6Fgm52xwzjY6DNEMSdwi5WjvB1D1W8ADIeqmY7/fqVftuyId3BaZOGLKAuf3Om2Wn+h4krhFytG+TzhUNRvABIGNYJ9uVEhpTwc2o8vOqpskYgX3i3VTtCUFpCPp4xYpR1mHAfVag9oP5n6GxdMVaPfzdWt+hwBvZLp2YJ3BUYn2ksQtUo9zAThOJvLf0wbZt6M6sZ5fl6QcgLneExq6+CSZrqzVxK2UekIpdUApJX+eRadQyowp935Uz7Wonmswuc40OqS0pzJ+CKZcUBmR6dm248Ai62+nq0Q6uJ4CHgSeTm4oQsRTymp0CF2GMveAwrcjRSJUFtimRNYYEWmp1cSttV6qlBqY/FCEEMmkTDmduhASgA78F+1+CrCiMi6XLq8OIpeUhUhROlwTWWkvhZaabQsd3I4uO6fuoqgJ7X0LCt9GmQuNDi3tddj/CKXUQqXUCqXUipKSko7arRCG0jpEuPIWwvuOIrx/Etr7YfKPGSohXHIq+sDEyI9/ZdKPmRS+D0FHVzcMgw7WlfoSh6vDErfW+hGt9QSt9YSioqKO2q0QhtLuv4PnDSIL51egK65Fh0qTe8yq2yC0g8jso2p0+RXouEX704QpB+qPE1dELpCKw5ae38GE6CyBdRxabIlIIgp9m9xjBrdxqFYmkTJrdF51lQ7jmAeWo+uq4zgiI1lsxxodVZfQah+3Uup54ASgUCm1C/il1vrxZAcmREqwTQHvOxyayanBMji5x7TPAPceIn8wLGAZlpbT05WyQv7TkSo+ygzmwTKSpYMkMqqkk9eUFCJ1KOcZ6PBucP8DVDYq585I9ZNkHjPrejQm8H8USXY5tyf1eMmklAmsRyS0rfa+hw6sQ1lHoRwzkxxZelPJ6DubMGGCXrFiRYfvVwjRNYVr/gw1fyHyzcYJmQsxNVOpSIfd6IqfgP9TMOWjch+ILEWb5pRSK7XWExLZVvq4hRDGq32SQ91RnrrHTdM190UmEhGCcAm6/PK6wsbdhyRuIYTxGq6bouzNbxtYT9zFWh2MFFHuRiRxCyEMp7JvBxygMiO3Wbc3v7FtGodWj1SRqvem7jWpR2ZOCtEN6dA+CJeCZWhKjFhRjpOh8A0IboqMorEMaH7bzKvRujJSFMJcjMq5q9utK969Pq0QgnDtU1B9X2RMusqEgpdQ5mKjw4qsY5LAWiZKWVDZv4DsX3RCVKlJukqE6EZ0uBKq7wF8oGshXIquvs/osEQbSeIWojvRNcQXVAhFukxEWpHELUSCIkuUvoD2f2l0KO1n6lU38zM6isMBzrONjEi0g/RxC5EA7X0fXXFd5IFS6IwfYcpcaGxQ7aCUCfKfRdf+FUJ7UI7TUI4TjQ5LtJEkbiESoKvvJrbYlAZqHkBnXJ6Wa28oUyYq64aEt9fhKtAeMPVIy8/bFUlXiRAJabg0RBous9oO4Zq/og9MRZechC47H63TcJXCLkgStxCJyLyByKQPO5G1NK7ukNan1mHCNQ8SLplLuOyKyPjqFKGDW6HmQSAA+CGwBl37jNFhCaSrRIiEmJynoC19wL8aLENQ9mkdsl9d+xjUPAp4ILQVXXYeFL6XGuXKQgdAWSHWyvZBaLehIYkISdxCJEhZjwLrUR27U997HFpcKQShksjwPHOPjj1Oe1hHAVYiX8zDgAPVycWGRdNS4M+6EN2YZSiR5FhHmVOmvJcyZaEKXgL7TLBOQeX+AWWfanRYAmlxC2EolXUTOvA1BDeAsqNyH0I1XCnPQMrSH5X3J6PDEA1I4hbCQMqUjSp8BR12g3KkRt+2SHmSuIVIAcrkMjoEkUbkz7sQoklahyLT/IPbjA5FNCAtbiFEI1p70Qd/AMFvgDDaeRoq+7cyczJFSItbCNGY+8VIUQPcgBc8/4LAaqOjEnWkxS2EaESHK4B6BXiVCcKVhsUTpcM1EFgFKgeso7vtNwBJ3EKIRpRjFrr2cSKTg8yAHWzjDY1Jh/ahD34PtBcIgWMeKuc3hsZkFOkqEUI0oqxHogqeBecZ4DwHVbgYZco2NCZd8wiEyyPFILQHPK+hg9sNjcko0uIWQjRJWY9G5dxldBj11AKhQw+VGbTbsGi0DqOr7wHva2DKjxQtto7qlGNLi1sIkRaU8zzACSjADuaBYDnCsHi0+2lwPxdZWya4CV12ITpc2ynHTihxK6VmK6W+VkptUUrdlOyghBCiIWUbgyp4HjKuQGVdjyp4HqUM7DTwL+fQAmEAYQjt6pRDt/qplVJm4CFgJrAL+FIp9brWekOygxNCiPqUdSTKOtLoMCKso8H3KbHKSITB3LtTDp1Ii3sSsEVrvVVr7Qf+AcxPblhCiFSjA5sIHzyLcMlJhGseQevuUQWoOSrjMnDMBGxgKkLl/hVlyuqUYyfyPaMPsLPe413A5OSEI4RIRTpcGynyoKsiT9Q8hDblo1zfNzYwAyllReXeB9zX6cdOpMXd1Aj3Rn9qlVILlVIrlFIrSkpKDj8yIUTqCG0nUkwhygO+pQYFIxJJ3LuAfvUe9wX2NNxIa/2I1nqC1npCUVFRR8UnhEgF5l6gg/WecID1SMPC6e4SSdxfAsOUUoNUZIX3c4DXkxuWECKVKFM+5NwPKgMwgf3YSB+vMESrfdxa66BS6hrgHSJzX5/QWq9PemRCiJRicp6MdqwCwkQGmwmjJDQIUmv9JvBmkmMRQqS4yKJOkrSNJjMnhRAizUjiFkKINCOJWwgh0owkbiGESDOSuIUQIs1I4hZCiDSjkrFQjFKqBNjRgbssBEo7cH8dTeI7fKkeo8R3+FI9RqPjG6C1TmjaeVISd0dTSq3QWk8wOo7mSHyHL9VjlPgOX6rHmOrx1SddJUIIkWYkcQshRJpJl8T9iNEBtELiO3ypHqPEd/hSPcZUjy8mLfq4hRBCHJIuLW4hhBB1UjJxK6XylVLvKqU2193mNbNdSCn1n7qfpK8R3lq1e6WUXSn1Qt3ry5VSA5MdUxvj+6FSqqTeOevUBZWVUk8opQ4opdY187pSSj1QF/8apdS4FIvvBKVUZb3zd1snx9dPKfWhUmqjUmq9UuraJrYx7BwmGJ/R59ChlPpCKfVVXYx3NLGNob/HCdFap9wPcDdwU939m4C7mtmuphNjMgPfAIMBG/AVMLLBNlcDf6m7fw7wQorF90PgQQP/XY8DxgHrmnl9DvAWkXJ5U4DlKRbfCcAbBp6/XsC4uvtZwKYm/o0NO4cJxmf0OVRAZt19K7AcmNJgG8N+jxP9SckWN5Eq8n+ru/834LsGxhKVSLX7+nG/DJykIgsYp0p8htJaLwXKWthkPvC0jlgG5CqlenVOdAnFZyit9V6t9aq6+9XARiLFvOsz7BwmGJ+h6s5LTd1Da91Pwwt9Rv4eJyRVE3dPrfVeiPxnAHo0s52jrkDxMqVUspN7U9XuG/6njG2jtQ4ClUBBkuNqdOw6TcUHcGbdV+iXlVL9mnjdSIl+BiNNrfua/ZZSapRRQdR9fT+GSIuxvpQ4hy3EBwafQ6WUWSn1H+AA8K7WutlzaMDvcUISqoCTDEqp94DiJl66tQ276a+13qOUGgx8oJRaq7X+pmMibCSRaveJbJMsiRz7n8DzWmufUupKIq2KGUmPLHFGnr9ErCIyLblGKTUHWAwM6+wglFKZwCvAT7XWVQ1fbuItnXoOW4nP8HOotQ4BY5VSucAipdRRWuv61zUMP4etMazFrbU+WWt9VBM/rwH7o1/v6m4PNLOPPXW3W4GPiPyFT5ZEqt3HtlFKWYAcOu+rd6vxaa0Paq19dQ8fBcZ3UmyJSuQcG0ZrXRX9mq0j5fysSqnCzoxBKWUlkhT/rrV+tYlNDD2HrcWXCuewXiwVRPLG7AYvGfl7nJBU7Sp5Hbio7v5FwGsNN1BK5Sml7HX3C4HpwIYkxpRItfv6cX8f+EDXXeHoBK3G16Cv83QifZCp5HXgwrqREVOAymiXWSpQShVH+zqVUpOI/P4c7MTjK+BxYKPW+v5mNjPsHCYSXwqcw6K6ljZKKSdwMvDfBpsZ+XucGKOvjjb1Q6Q/6X1gc91tft3zE4DH6u5PA9YSGT2xFri0E+KaQ+RK+TfArXXP/Qo4ve6+A3gJ2AJ8AQzu5PPWWnx3AuvrztmHwJGdHN/zwF4gQKRVcylwJXBl3esKeKgu/rXAhBSL75p6528ZMK2T4zuWyFf2NcB/6n7mpMo5TDA+o8/haGB1XYzrgNvqnk+Z3+NEfmTmpBBCpJlU7SoRQgjRDEncQgiRZiRxCyFEmpHELYQQaUYStxBCpBlJ3EIIkWYkcQshRJqRxC2EEGnm/wFxEnfNVR4nWgAAAABJRU5ErkJggg==\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "from sklearn.svm import SVC # \"Support vector classifier\"\n", "model = SVC(kernel='linear', C=1e10)\n", "model.fit(X, y)\n", "plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis')\n", "plotDecisionFunction(model);" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 12.2 - Soft-margin classification\n", "\n", "As explained in the lecture, the tuning parameter $C$ determines how many misclassifications are allowed.\n", "Check the following two claims from lecture slide 427:\n", "- C large: margin wide, many observations violate margin, many support vectors, many observations involved determining hyperplane, classifier has low variance, potentially high bias\n", "- C small: fewer support vectors, lower bias, higher variance" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Task**: \n", "Use the following interactive plot to see how the margin as well as the decision boundary changes with varying values of $C$.\n", "What do you observe?" ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "dc48a6ae7cee404ab1b30ac91ffc5a98", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(FloatSlider(value=0.745, description='myC', max=1.5, min=0.005, step=0.02), Output(layou…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "from ipywidgets import interactive\n", "def mySVC(myC):\n", " model = SVC(kernel='linear', C=myC)\n", " model.fit(X, y)\n", " plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis')\n", " plotDecisionFunction(model);\n", "\n", "interactive_plot = interactive(mySVC, myC=(0.005,1.5,0.02))\n", "output = interactive_plot.children[-1]\n", "output.layout.height = '300px'\n", "interactive_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Observation**: In contrast to the lecture, the margin is small for large values of $C$ and wide otherwise.\n", "This is due to the fact that the tuning parameter $C$ used in the lecture is not the same as the tuning parameter $C$ in the function `SVC`, see the [scikit learn documentation](https://scikit-learn.org/stable/modules/svm.html#svc).\n", "\n", "You will learn more in the **Optimization in Machine Learning** class about this reformulation." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now, we generate a second data set." ] }, { "cell_type": "code", "execution_count": 12, "metadata": {}, "outputs": [ { "data": { "image/png": "iVBORw0KGgoAAAANSUhEUgAAAW4AAAD8CAYAAABXe05zAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4wLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvqOYd8AAAIABJREFUeJzt3XeYVOX9/vH3Z+oWWHqRjiURxEJEVDAaW6JGRY2xJBpbNBp7jC3RJGqMxm7U/BRLjCVGv9iNRo0a7MTFggVRrIBIW8ouu1PP8/tjVnZBWWbZmT17Zu7XdXHpjGfOuXdc7j175jnPY845REQkOEJ+BxARkfZRcYuIBIyKW0QkYFTcIiIBo+IWEQkYFbeISMCouEVEAkbFLSISMCpuEZGAiRRjp3379nUjRowoxq5FRErS9OnTFzvn+uWzbVGKe8SIEdTW1hZj1yIiJcnMPst3W10qEREJGBW3iEjAqLhFRAJGxS0iEjAqbhGRgFFxi4gETMkVt0t/gLdwF7wvR+EtOQTnLc3jNbPwVlyG13AjzqvvhJQiIuuv9Ip76THgzQWykJ6BW35e29unZ+KWHASNt0DD9bglB+Jcqt3H9RofwVt8AF7dkbj0h+uZXkRk3UqquJ1Lg7ew1TMZyLzf9msa7weamh+lcq9Pv92+4yanworzIPMOpF7B1R2C85a3ax8iIvkqqeI2i0J4I1q+rDjEtm37RaFqVruB1HlgVe06rks8ByS+epT7R3pmu/YhIpKvkipuAOt9G8S2gdAAqNgTq/ld29tXHwGh/mDVYJVQ8X2IbNq+g0ZGAPGWxy4N4UHtzi4iko+izFXiJwsPxHrfmf/2od7Q7wlIvQGhbhDZHDNr3zGrfopLvQLJ54EQdD8LiwxrZ3IRkfyUXHGvD7NKiE/owOujWK+bcF4DWAyzWAHTSVu8huug4SYgDN3PIFT9M78jiRRdyV0q8ZOFuvlS2s65Tj9mV+CSU2HlLUAKaIL6K3DpGX7HEik6FXeAeY0P4y3YCrdgDN7yC8uvwDOzwWVaHlsIMh/5l0ekk6i4A8plPoUV54NrBNKQuB8Sj/gdq3NFxwLhlsfOg+jmvsUR6Swq7qDKfALWehhjEy49y788PrDYd6DmIggNgfAIrOc1WGRjv2OJFJ0+nAyq6Cgg2+qJSiy2jV9pfBOqmgRVk/yOIdKpSuKM26U/wDU9UlZnnBYeiPW6GcIbQ2gQdD8Tq9jZ71gi0gkCf8btNf0Llp8LFgaXxdVclDsLKwMWG4/1ezzv7Z1zuMbboekRCA/Cas7HwgOLF7CIXOpNXOJJLDwAqg7FLL7uF4mUiMAXN/V/AhKr7jSn/pIu96uz8+ohOwfCg7FQD/9yNP4D6q8BmiDzPm7Je9DvacyC9W3gktNwS48FEjjikHgCet+DWUn8AimyTsH/TnfZNZ7IfONmfnGpN3CLdsTVHYZbtBMuOc2/MIknaZlQKwuuDrLz/cuznlzj7bTMDZPMzQuTneNjIpHOFfzi7nYCUEnul4dKqD7B50Crc8vPArcSXAO4RtzyM/0LExkJtLpByDkI9fYtznqz7qz+revl5pkRKRPB+h35G4Sqj8BFNoH0uxAdjcUn+h1pdd6y1R87/xZqsO5n5O4szHwAhKDHlVio2rc868u6nZK7a5J07jeuqsOwcH+/Y4l0msAXN4DFJ3RorpGiqtwPGu8jd4miAir29i2KhWqgzwPgVoBV5abBDSCLDIF+T0N6BoT6YNFRfkcS6VQlUdxdmXU/FxceCqnXILoVVn1ku/eRu5Xdwyy8zm3XmccMzL8PSAvFQjUQ38HvGCK+yKu4zex04Ofkxm68DRzlnEu0/SoBMAth1T+D9Zy1ziVfxi07GVwDLrY91vOvWKh9Cz2ISGlZ54eTZjYYOAUY55wbQ25yiEOKHUzAuSbcsl82Xxd3kKrFNVztdywR8Vm+o0oiQKXlBvxWAV8UL5Kskl1MywB1gBSU0d2hIvLN1lnczrl5wBXA58B8YLlz7qliBxMgPDC3pNqq/02VEN/Rz0Qi0gXkc6mkFzAJGAkMAqrN7LBv2O44M6s1s9pFixYVPmkZMotive+B6NYQHgrVR2LVR/sdS0R8ls+Hk7sBnzjnFgGY2QPABOCu1hs55yYDkwHGjRtXZjP6F49FhmN97vY7hoh0Iflc4/4c2M7Mqiy3iu6uwMzixhIRkbXJ5xr3NGAK8Dq5oYAhms+sS53zluLV/Rxv4fZ4dcfi1rwLsky45It4i/fFW7wPLvGs33FEyp4VY53CcePGudra2oLvt7N5dYdDajq5iasiEBtPqPftPqfqXC4zG7f4AFomdarA+vwTi472M5ZIyTGz6c65cflsG/xJpoop/Q4tsw1mcrdYl5vUG4C1eiI3nlxE/KPibkt4I1reohCU43qGkRFrPBGCyHA/kohIMxV3G6zXdRD5NhCFyKZYz7/4HanTWWwb6PZzcgOQwlD1U4hpLLmInzTJVBssvAHW92G/Y/gu1O1kXPUvARe41XJESpH+FkpeCjEzoYgUhi6VFJBzHi77Bc5r9DuKiJQwFXeBOK8Ot3hP3KI9cAu3xWvKf/V1EZH2UHEXiFtxafOCtQkgCcvP1pm3iBSFirtQsnNZfYV5yy0RJiJSYCruQqnYl9xq8wARCA+CkBawFZHC06iSArGqg3Fm0PQIRIZi3c/CrPN+LjqvEdd4B2QXYJV7Y7GtO+3YItK5VNwFYmZY1cFQdXCnH9u5LK7uJ5CZDaRwTfdDr5uw+PadnkVEik+XSkpB9lPIfAKkmp9I4Bo1h7dIqVJxlwKrBrKtnghDqIdfaTqNS8/ANT2Gy8zxO4pIp9KlkhJg4YG46qNg5d/BImCVWLeT/Y5VVF7DTdDwV3LnHh70viU3r4pIGVBxl4hQ9zNwlZPAq4PIZlio2u9IReNcBhqupfXwS1f/Z6zPFP9CiXQiFXcJsbKZdtZr/tOKS33jliKlSNe4JXDMYlB5ELlx82GgAut2os+pRDqPzrglkKzmAohvD5nPITYei431O5JIp1FxSyCZGVTs6XcMEV/oUomISMCouEVEAkbFLSISMCpuEZGAUXGLiASMiltEJGBU3CIiAaPiFhEJGBW3iEjAqLhFRAJGxS0iEjAqbhGRgFFxi4gEjIpbRCRgVNwiIgGTV3GbWU8zm2Jm75vZTDPbvtjBRETkm+W7kMK1wL+dcweaWQyoKmImERFpwzqL28xqgB2BIwGccylAK7OKiPgkn0slGwKLgL+Z2RtmdouZVa+5kZkdZ2a1Zla7aNGiggcVEZGcfIo7AnwH+H/OubHASuCcNTdyzk12zo1zzo3r169fgWOKiMhX8inuucBc59y05sdTyBW5iIj4YJ3F7Zz7EphjZt9ufmpX4L2iphIRkbXKd1TJycDdzSNKPgaOKl4kERFpS17F7Zx7ExhX5CwiIpIH3TkpIhIwKm4RkYBRcYuIBIyKW0QkYFTcIiIBo+IWEQkYFbeISMCouEVEAkbFLSISMPne8i4iJaJ+aQMvP/wa0ViEifuPJ14Z9ztSwaSSae6/6lE+nzmP7fcdx44HluZiXSpukTKyfPEKjtvy1zSuaATgn39+iOunXUKsIuZzssL4wwGX89Z/3yXVlOKFB6ZRv7SBHx67u9+xCk6XSkTKyDN3vUB9XQOJlUkSK5N8+clCap96y+9YBZFoTDL9qbdINeUW6Eo2Jnnkhid9TlUcKm6RsuPafNjVrFy+ki8++pJsJtvmdtFYhHCkpdLMjJq+3YsdzxcqbpEysstPd6Bbz27Eq+JUVMfpP7wf436wpd+x1urpO6dy0AbHctyWv+bo0aexdOHytW4bjoQ57aZfEKuIUlVTSbde1Zx03TGdmLbzmHOF/3E7btw4V1tbW/D9ikjHrVhSz4sP/o9oLMJ3D9yOiqqu+eFkY30TB/Y/hnQyDUA4GmaXn+zAWX87qc3XLZ63hIVzljB89BCqa6o6I2pBmNl051xe02frw0mRMlPTpzt7/XxXv2OsU8PSBkIhW/U4m86y8LPF63xd38F96Du4TzGj+U6XSkSkS+o7pA8DRvQjEg0DEK+Ks9vhO/qcqmvQGbeIdEmhUIirn7+IW39zNws+XcTOh+7AD47c2e9YXYKKW0S6rJo+3Tn9puP9jtHl6FKJFIRzjlQi5XcMkbKg4pYOm/70W0zq+TP26XYYp+1wHiub78oTkeJQcUuHJJuS/OGAy2mqT+B5jg+mf8Qt59ztdyyRkqbilg5Zvrge57XcC5BOZvjs3Tk+JhIpfSpu6ZA+g3pR07c7oXDuWyleFWfbvbf2OVV5euGBafxk+PEcNOhYHrru8bVu55zj+Smv8M8/P8TMaR92YkIpFI0qkQ4Jh8Nc/fxFXHP8TSz4bDHfO3gCPz5jH79jlZ3PZs7lz4f/hWTzBEu3/uYfDBs1hO/stsXXtv3Libfwnzunkk5miETD/OYfpzFh0jZr3fe9lz3ElKseJRqPccoNP2c7/WD2nYpbOmzA8H5c8sR5fscoax+/9RmhVhMspRJpZr320deKO51K8/jN/8HLegBkM1nuuOC+tRb3y4+8xl0XTSGxMgnAHw+5iptnXMUGGw4o0lci+dClEpEuLtGY5K4/TuGKo29g2uOvf+M2I8YMxct4qx7HKqJsuOXwr20XCoUws9Wei8bWfv4267WPVpU25CZy+uTtz9eZOZVMM2/2fJpWJta5rbSfilukC3POcfb3L+KePz3Ak7f/l4sOupKp//fK17YbOWYYp954HDV9ulFVU8mh5+7Ptnt952vbhSNhDv/9j4lXxansXkFFVZxjLzt8rcffeKsRVFS3TEKVzWQZNnpIm5k/fXcOhw75BcePPZODBx3LjOffa8dXLPnQ7IAiXdiS+Us5fKMTSSfSq57bYsfRXPnfCzq035nTPmT+xwsYte0mbV72cM5xxx/u44Fr/0U0HuXk645hp4MmtLnvE8adxezXP1n1uNeAntw3/+YO5S0Hmh1QpERUdqtYbbhlKByiR7+aDu931LabMGrbTda5nZlxxAUHc8QFB+e977r5y1Z7vKKuvt35pG26VCLShVV1r+TYPx+2anGAmt7dOPayw9q9n8Vf1LF43pIiJPy6nQ+dSLx5ju9YRVSjUIpAl0pEAuDLTxdSN38pIzcfRmW3yrxf55zjzz+7juenvAoGE/bdht/841RCoeKds2WzWf7vikd545kZbLL1Rvzs9z8u+mLEnucx+cw7+Pdtz1Hdo4oz/3YiW+08pqjHLLT2XCpRcYuUsFcfm87Fh169amRIRbcKzv77Seyw/7Y+Jyusx256ihvPuINkY/PXWR3nzo9voGe/Hj4ny197iluXSqRdnHPMmTWPj2d8Rjbb9uKt4r+6+UtpfXLmsh5Lv1zWxiuC6d1XPlhV2pAbPfPF7C99TFRceX84aWZhoBaY55zbu3iRpKtyznHp4dfx0oPTsJCx4RbDufzZPxCLR/2OJmsxdrfNV7ssYiHjO7t//W7KoNts+2/xwpRXV5V3NpNl8CYb+JyqeNpzxn0qMLNYQYJs5rQPOWXCbzluqzN47p8v+R2naGZMfY+XH/4fyaYUiZVJPnrrM566/b9+x5I2bDByAFdNvZCJ+49nwqRtuOK5Cxi8cekV2l7H7sbev9iN6p5V9B/el4seOYcefTs++qaryuuM28yGAD8ELgZ+VdREAbN43hLO3v1Cmhpyd4hd+fO/0ntgT7b83mY+Jyu8FXUNWKuzt3QyTX1dg4+JJB8bjx3JH+4/0+8YRRUKhTj+yiM5/soj/Y7SKfI9474GOAvw1raBmR1nZrVmVrto0aKChAuCD2o/xlqtRJ1sTPHGs2/7mKh4ttxpNNFYeNXK27GKKBP2W/vkRCJSHOssbjPbG1jonJve1nbOucnOuXHOuXH9+vUrWMCuboMN+5PNtHxIV1EdL8lfRSG3/t9fay9jr+N25/tHfI+rn7+I4aPavv1ZpGllgvP2uYS9Kg/lpyNOYNZrs/2OFHjrHA5oZpcAhwMZoAKoAR5wzq31LoByGw74wLWPcfNZd+F5jt0O25Ezbj2hqONkRYrl3ZdnUfvUmwwY1o/dj9iJcDjc4X1ef/KtPHHrM6Sab9vv1rOaKQtvJRzp+L5LSUFveXfOnQuc27zj7wG/bqu0y9EBp+7NpJP2JJvxNMJCAmvav6Zz0UFXkUykiFfGefVf0wtybXxW7UerShsglUixfPEKeg/s1eF9lyudFhZIOBxWaUug3XPpQ7mFGBwkG5NMe2w69Us7/uHzljuNJl6Zu3PSzKjuWV2Q+VbKWbsmmXLO/Rf4b1GS+Oitqe/yzgvvM2zUYHY4YNuvzVcsUg4qqla/Ld0BkTbm6s7XERceTH1dAy8/Ukv/YX05586TC3IJppyV/eyAT985lWtPmEw6kSZWGeOdl97nhKuO9DuWSKc79rLDmbnj78Agm/E4+KxJVFZXdHi/0ViU0ycfz+mTCxBSgC48V0kqkeKtqe8RiYbZYsfRRfsg48hvncy8VrfGhiNhHk/8Qx8uSlmq+3Ip70+bTf9hfdl47Ei/45SVwM/H3dTQxEnbnsuiuUvA5W4guPyZ3xelvKMVq1+XDke+vrRTR2XSGWY8n7vpdIsdRxGJdsm3XYTeA3u1uXCwdA1dpkGcc7z9wkwalq3ki4++ZMGni1atWD37jU947d9vFmVe3xOuPorfTfoz4UiIbCbLcZcdXtDiTiVSnPbd85k76wsAho0ewtXPX0g0pg8yRWT9dInids7xp59cw6uPTScUCuE5t9pNLcCqEi+07+y6OTfPuJIPaj9iyLcHsdGWIwq6/+envMqc9+etmlbzs3fn8NKD/+N7B08s6HFExD+P3/Ifnrj1Wfps0JMTrj6KAcOLexNilyjuT9/5nFcenb5qZq9QyAhHw4TCIULhENU9qhj3gy2LdvwNNhzQ5rp7HZFsTK4+raaDRGNxfgiJSOd79p4X+Otpt5NsTBIKh5g5bTZ3fnR9UReP6BLFnUqkV81/AeB5jp0P3J6hmw4mEo2wx9E7U11T5WPC9Tdhv/Hc9tt7yKQygFFRHWf7fXKXfJ6+cyr/vPRB4pUxfnnNUYzZYZS/YUWk3Z5vNZ2sl/Voamhi3ofzGbn58KIds0sU90ZbjWDQxgOZM+sLMsncsLyDzpzEhlsU7wvvLL369+Cmt67g37c9i5mxx9G70KNvDW88+zbXnjCZZPPZ97l7Xsyt711D/6F9fU4sIu0xeOOBRONR0snc3aHZtEevgT2LeswuUdyRaIRrXvwj/771WRqWr+S7P9qupCYv6juoN4edd+Bqz7353LurShtyq3fPfv0TFbdIwBx2/oHMeH4ms9/4BDPj1BuPLfqSaV2iuAEqquLsd/KefsfoNMM2HUxFVZzEVyt2pLNssFFxrrOLSPFUdqvkLy9fTH1dAxXV8aIvjAyaq8Q3Ox86kV1+ugOhcIhILMJxlx/OyDHD/I4lIuvBzKjp071TShu68J2T5SKbyWIh052aJSCbyXLtCTcz9f9epkffGs656xRGb/ctv2NJQGiV9wAJR8Iq7RJx72UP8+w9L9C4oon5Hy/g3D3+SNPKhN+xpASpMQJqxZJ6Xnrof7zz0vsU47cmab93X35/tQ+cnXMs/Hyxj4mkVHWZDyfb47OZc/n8vblsuOXwgi8T5pyj9sk3WbpgOWN33Zx+Q/oUdP+FMP+TBZw0/hwy6Sxe1mPnQybyq5tP8DtW2RszcVPe+m/LaCEzo/8wjRKSwgtccT/zjxe4+tgbCUfDZNNZzrv3VwWbw8Q5x6WHX8fLD/8PyK0c/ZdXLmb46KEF2X+h3HPJgzQsXYnn5c60n7n7BQ4994Ci3f0p+TnozEl8+ekipt7Xco27ENOiiqwpcJdKbjjlNpJNKRpXNJFsSnH9ybcWbN8LP1/Miw+8SmJlksTKJE0NTdx98QMF23+hJBuTq0obcmPAUwndRu+3cCTM6Tf9goeW/p2/f3gdo7bdxO9IUqICV9y5W8dbpJLptWzZfl7Wg1YzAzoH3hqTXXUF+5+yF/GqOKGQEa+KsdGWIxi66eD12tfShct58cFpzJz2YYFTikixBO5SyY9+tQ9TrnyERGOSeGWcQ8/Zv2D7HjiyP2N33Zy3nnuHbCZLOBLm4LP3K9j+C2XT8Ztw3SsX89LDr9Gjbw0/OGrn9RqZ8tnMuZw64bc45/CyHj88bneOv/KIIiQWkUIK3Dhu5xwvP/waH8/4jE3Hb8w2e4wt6P6z2SwvTHmVpQuWs82eYxmySWE//OxKLvzxlbz4wLRVo1IisQj3zLmx6LfrisjXBX4FnLaYGRP3G8/E/cYXZf/hcLhs5spOrDHlbChkpJOZNl4hIl1B4K5xS+Ec9Ot9iVfFsJARr4qzxY6j6Tu4t9+xRGQdAnfG/RXnHF9+upBoPErfQSqb9bHVzmO4auqFTHv8dfps0JvvH7FTwdfbFJHCC2RxZ9IZztv7Et5+8X2c5/jBUTtzyg0/V+msh29tvRHf2nojv2OISDsE8lLJ03dM5Z2XZpFqSpFOpvnPnVN596X3/Y61Xpxz3HPpgxyz2Wmc/YOLmP/xAr8jiUgXF8jiXrpg+arVJiB3h+PSBct9TLT+Hr7h3/zjj/fz+cx5vPnM25z23fNJpwo3Nl1ESk8gi3u7vbcmGs9d5TEzLGSM+W4w12t89dHaVYspeJ6jcUWjJiYSkTYFsrg33GI4lz55PhMmbcNOB0/gulf/RK/+wRx7PHyzoUTj0VWPnXP0GlDc9epEJNgC+eEk5GZiGzNxU79jdNiRFx7M7Dc+4Z0X3ycWj3LevadT1b3S71gi0oUFtrhLRWW3Sq587gJSiRTReFQjY0RknVTcXURnrVUnIsGn4i5TC+cs5u+/u5eGZSvZ95c/YOvdt/Q7kojkScVdhhrrmzhp/DksX1yPl/WY/vRbXPLEeWwe0JE5IuUmkKNKpGM+fP1jkk2p3PzjQLIxxdT7XvI5lRSb8xpwiedwqde0TmnArfOM28yGAncAAwEPmOycu7bYwaR4evbvQTbdskBEtCJKn8Fdb21NKRzn1eEW7weuHnAQmwg9r9eH4QGVzxl3BjjDOTcK2A440cxGFzdWeVr4+SLO3/dSjv/OmUy5+tGinRUNHzWEH/1qb6LxKPGqGCPHDGP/U/YqyrGka3Ar7wJvMbiV4Boh+SKkZ/gdS9bTOs+4nXPzgfnN/15vZjOBwcB7Rc5WVjLpDKfucB51XyzF8xzzPpxPRVWcvX/x/aIc76iLDmXSiXvQWJ9g0EYD1msFHQkQ1wS0WobPQkDSrzTSQe3622pmI4CxwLRihClnCz5btNrK7YmVSV588H9FPWbvgb0YsskGKu0yYFU/Aqsk91c+DqEBEN3K71iynvIeVWJm3YD7gdOccyu+4b8fBxwHMGzYsIIFLBc9+/dYbeX2aDzC0G8P8jGRlBKLbAx9HsQ1PYaFukHljzHTvQNBldeplplFyZX23c65B75pG+fcZOfcOOfcuH79+hUyY1morqni3LtOIVYZIxwJs9FWIznyokP8jiUlxCIjCXU/Gas+KlfeElj5jCox4FZgpnPuquJHKl877L8tjyy/g0RjkuqaKr/jiEgXlc8Z90TgcGAXM3uz+Y+GIBRJOBJWaYtIm/IZVfIioMGeIiJdhIYTiIgEjIpbRCRgVNwiIgGj4hYRCRgVt4hIwKi4RaTdnFeH85b5HaNsqbhFJG/OeXjLTsct/C5u4US8FRdpbm8fqLhFJH+Jf0PyWSCd+9M0BVKac66zqbhFJH/el+AyrZ6w3HPSqVTcImXKuRTOZde9YWuxHYDwGs+NL1gmyY+KW6TM5K5Tn41bsAVuwRZ4K+/I+7UW/RbW+1aITYDYTljvu7Fw/tMPO5fBeSvXJ3ZBudQbeMvPw6u/HOct9TtOu2mVd5Fyk3god60aL/en/gpcbDss+q28Xm6xbbDet7f7sF7TY7D8HCCLi22D9ZqMWUW799NRLvUWru4IIAFEcIknoO8TmMU7Pcv60hm3SJlxmU+BppYnLALZucU9ZnYJLD8XSAFZSL2Ba7ixqMdca5amB8iVNkAGsnW4JQfhfTkab9FuuPSHvuRqDxW3SJmx2PZA6zNdD6KbFfeg3oLcD4hVkpD5qLjHXJtQTyDa6okEZD4kV+Kf45Ye40+udlBxi5QZi28PNRdD5NsQ3Qrr9XcsPKC4B42MBOK0VE4lxHcr7jHXwqqPhvBgsGqgovmfrUbKeAtwLu1LtnzpGrdIGQpV7QNV+3Ta8cwqoc+9uBUXgLcYKg/CKvfttOOvliXUA/o+BumZEOqOW3kLND1KbtX7EIRHkFutsetScYtIp7DIcKz3bX7HAMgtlBzbMveg5nycS0HqZQiPxHpe4W+4PKi4RaSsmVUGoqxb0zVukTLlsl/iklNxmc/9jiLtpDNukTLkki/hlv4SLAwug+txKaHK4K0B7lwKsgsgPCB3+aNMqLhFypBb8XugCb6a2G/F76ELFLdzaUg+g8t8AVaNRTfFvroWvea26XdwdUeSm/AqCr3/jhV7WGMXoeIWKUcuscbjpD85WkdwGVzdYbnRHs03yDjiuG4nEOr2S1zmE0i/mRv1ERuLW3Y6uBXNr27CLfsV1u/JwmRJPIervwTIQvUphKomFWS/haLiFilH1UdB/V/I3UFZCVWH+J0IUtMhM4uWuxoBktBwPV50c1h6IlgInMN1+wV4S1Z/vbe4IDFcZjZu2aktOVacj4sMw2JjC7L/QlBxi5ShUPUxuPAwXOp1LDoaKvb2OxK5uVPsG553UH85kGi5tNNwPcR/CMknc89TUbgbetIzmn9AfPVEFtJvgIpbRPxmFbtjFbv7HaNFbGsID4fMbHJzmgDEoXJfSL+1xsYOelwAKwflztRjW2PdTixMjvBG4LyWxxaByMaF2XeBqLhFpEswi0GfeyHxePNET2EsOgoq9sx9YLnsV0A2V6SVhxAKVUH30wufI7YlrvsZUH8l4EHVMVh8x4IfpyOsGOvFjRs3ztXW1hZ8vyJSvlx6BqReg/AIiO+C2TddVgkuM5vunBuXz7Y64xaRQLDoFhDdwu8YXYLunBQRCRgVt4hIwKi4RUQCRsUR9l7cAAAEtklEQVQtIl2ecym8FZfiLd4Xb9k5OK/e70i+0oeTItLluRV/hKaHyC0z9jEuOw/rc6ffsXyjM24R6fqSz9FyK3wK0q9RjKHMQZFXcZvZHmY2y8xmm9k5xQ4lIsHlMnPwlp2Nt/QkXKpA93OEh7JaXYX6ldw47vZYZ3GbWRi4AdgTGA0camajix1MRILHectwS34EiYch+RSu7mhc+p0O79d6XAahDYAoWA+s1187HjbA8rnGPR6Y7Zz7GMDM/glMAt4rZjARCaDUG+RWTP9qro8kLvE0Fh3Tod1aZAj0ezY3jat1x6y8r/Lm89UPBua0ejy3+TkRkdWF+4DLtnoijoX6FWTXZoaFepR9aUN+xb2WeRbX2MjsODOrNbPaRYsWdTyZiASORbeAqoOAGFAB0c2aH6/OuTRe/bV4S36GV391bgkyyVs+l0rmAkNbPR4CfLHmRs65ycBkyE0yVZB0IhI4oZrf4qqPya2yEx7+jR8iuhXnQ9PjQALSb+Ky87Gel3V+2IDKp7hfAzYxs5HAPOAQ4CdFTSUigWbhgW1vkHialuF9CUj+p9iRSso6i9s5lzGzk4AngTBwm3Pu3aInE5HSFeoL2VZ3P4b6+JclgPK6c9I59zjweJGziEiZsJ5X5lZod2mwKNbzKr8jBYpueReRTmfRMdD/JcgugPAAzOJ+RwoUFbeI+MIsDpFhfscIJA2IFBEJGBW3iEjAqLhFRAJGxS0iEjAqbhGRgFFxi4gEjBVjFQkzWwR8VvAdr1tfYLEPx+2K9F7k6H1oofcip6u+D8Odc3lNpViU4vaLmdU658b5naMr0HuRo/ehhd6LnFJ4H3SpREQkYFTcIiIBU2rFPdnvAF2I3oscvQ8t9F7kBP59KKlr3CIi5aDUzrhFREpeyRW3mV1uZu+b2Qwze9DMevqdyQ9m9mMze9fMPDML9Cfo68vM9jCzWWY228zO8TuPX8zsNjNbaGbv+J3FT2Y21MyeM7OZzX83TvU70/oqueIGngbGOOe2AD4AzvU5j1/eAQ4Anvc7iB/MLAzcAOwJjAYONbPR/qbyze3AHn6H6AIywBnOuVHAdsCJQf2eKLnids495ZzLND98ldzixmXHOTfTOTfL7xw+Gg/Mds597HJLiP8TmORzJl84554H6vzO4Tfn3Hzn3OvN/14PzAQG+5tq/ZRcca/haOAJv0OILwYDc1o9nktA/5JK4ZnZCGAsMM3fJOsnkCvgmNl/gG9aRvq3zrmHm7f5Lblfje7uzGydKZ/3oYzZNzynIVSCmXUD7gdOc86t8DvP+ghkcTvndmvrv5vZEcDewK6uhMc7rut9KHNzgaGtHg8BvvApi3QRZhYlV9p3O+ce8DvP+iq5SyVmtgdwNrCvc67R7zzim9eATcxspJnFgEOAR3zOJD4yMwNuBWY65wK9rHzJFTdwPdAdeNrM3jSzG/0O5Acz29/M5gLbA/8ysyf9ztSZmj+gPgl4ktyHUPc55971N5U/zOwe4BXg22Y218yO8TuTTyYChwO7NHfDm2a2l9+h1ofunBQRCZhSPOMWESlpKm4RkYBRcYuIBIyKW0QkYFTcIiIBo+IWEQkYFbeISMCouEVEAub/A4xLjbDEUbY6AAAAAElFTkSuQmCC\n", "text/plain": [ "
" ] }, "metadata": { "needs_background": "light" }, "output_type": "display_data" } ], "source": [ "# Generate data with three centers\n", "X, y = make_blobs(n_samples=60, centers=4,\n", " random_state=0, cluster_std=0.60)\n", "# Change class 2 also into class 0, and class 3 into class 1\n", "y[y==2] = 0\n", "y[y==3] = 1\n", "plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis');" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Task**: Copy the code from above and generate an interactive chart that visualizes the **soft-margin classifier** for different penalty parameters $C$." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Solution**:" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "82e4fe533db24619bb53f268ce5b8333", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(FloatSlider(value=0.745, description='myC', max=1.5, min=0.005, step=0.02), Output(layou…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def mySVC(myC):\n", " model = SVC(kernel='linear', C=myC)\n", " model.fit(X, y)\n", " plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis')\n", " plotDecisionFunction(model);\n", "\n", "interactive_plot = interactive(mySVC, myC=(0.005,1.5,0.02))\n", "output = interactive_plot.children[-1]\n", "output.layout.height = '300px'\n", "interactive_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "As you can see, the classes cannot be separated by an affine linear function. At first glance it appears as if the function `plotDecisionFunction()` may have highlighted too many support vectors, especially the blue ones close to the right boundary, but they are indeed all support vectors; all samples in the lower left corner (below the separating line) are assigned to the blue group." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Problem 12.3 - Support vector machine\n", "\n", "We now consider the support vector machine, which generalizes the soft-margin classifier through the introduction of basis functions (the soft-margin classifier corresponds in the choice of a linear basis function)." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Task**: \n", "Experiment with different basis functions, e.g., `'poly', 'rbf'` and `'sigmoid'`, to obtain a better separation between the two classes.\n", "See the [documentation](https://scikit-learn.org/stable/modules/svm.html#kernel-functions) for further information." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Solution**:" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [ { "data": { "application/vnd.jupyter.widget-view+json": { "model_id": "07c3961b457d4c85b4c4e4c6b2f1be51", "version_major": 2, "version_minor": 0 }, "text/plain": [ "interactive(children=(FloatSlider(value=0.745, description='myC', max=1.5, min=0.005, step=0.02), Output(layou…" ] }, "metadata": {}, "output_type": "display_data" } ], "source": [ "def mySVC(myC):\n", " #model = SVC(kernel='poly', gamma='auto', degree=3, C=myC)\n", " #model = SVC(kernel='poly', gamma='auto', degree=2, C=myC)\n", " #model = SVC(kernel='sigmoid', gamma='auto', C=myC)\n", " model = SVC(kernel='rbf', gamma='auto', degree=3, C=myC)\n", " model.fit(X, y)\n", " plt.scatter(X[:, 0], X[:, 1], c=y, s=15, cmap='viridis')\n", " plotDecisionFunction(model);\n", "\n", "interactive_plot = interactive(mySVC, myC=(0.005,1.5,0.02))\n", "output = interactive_plot.children[-1]\n", "output.layout.height = '350px'\n", "interactive_plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.6.6" } }, "nbformat": 4, "nbformat_minor": 2 }