{ "cells": [ { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "OPZb4UFhwzvV" }, "source": [ "# Python Course 4 - Data Analysis\n", "\n", "Application data (from the *real world*) does usually not come in any form that we would like. It is not nicely organized inside of vectors, matrices and tensors. An problem is also posed by missing values which appears quite often.\n", "\n", "The goal of this exercise is to look into a few basics of data analysis using Python." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "BufZwibVx6ut" }, "source": [ "We start by loading the relevant libraries." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "NLPQ3TInx93o" }, "outputs": [], "source": [ "import pandas as pd\n", "import numpy as np\n", "import seaborn as sns \n", "import matplotlib.pyplot as plt \n", "\n", "%matplotlib inline \n", "sns.set(color_codes=True)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "dTDiSfD1yCt_" }, "source": [ "Data is for example given in a CSV format. Here, we use a data set about cars. Loading the data into the pandas data frame is certainly one of the most important steps in EDA, as we can see that the value from the data set is comma-separated. So all we have to do is to just read the CSV into a data frame and pandas data frame does the job for us." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 397 }, "colab_type": "code", "executionInfo": { "elapsed": 541, "status": "ok", "timestamp": 1568721927699, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "wN17uIFJzm7b", "outputId": "63459d6a-2642-4e74-cb4f-a4d919cc368c" }, "outputs": [], "source": [ "df = pd.read_csv(\"data.csv\") \n", "df.head(5) # display top 5 rows, \n", "#df.tail(5) # display bottom 5 rows " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "hxbuLX2f0vxG" }, "source": [ "Datatypes may also pose a problem since columns might have the *wrong* assigned to them. It is for example possible that values are stored as strings but we need integers or floats to work with.\n", "\n", "Here we check for the datatypes because sometimes the MSRP or the price of the car would be stored as a string, if in that case, we have to convert that string to the integer data only then we can plot the data via a graph. Here, in this case, the data is already in integer format so nothing to worry.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 329 }, "colab_type": "code", "executionInfo": { "elapsed": 512, "status": "ok", "timestamp": 1568721932574, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "a8x2xso01YeF", "outputId": "09855347-dd3e-43df-b9a3-9d66b1f8c93c" }, "outputs": [], "source": [ "df.dtypes" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "spL4Ed4a0ldo" }, "source": [ "The next step is to drop irrelevant columns. This is needed most of the time since we are not going to use a lot of the data. In this case, the columns such as Engine Fuel Type, Market Category, Vehicle style, Popularity, Number of doors, Vehicle Size don't interest us, so we just drop them for this instance.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "colab_type": "code", "executionInfo": { "elapsed": 512, "status": "ok", "timestamp": 1568721935034, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "1qlgeCSk2WTS", "outputId": "7b5b603c-e581-46c0-b515-0a2409fdcc98" }, "outputs": [], "source": [ "df = df.drop(['Engine Fuel Type', 'Market Category', 'Vehicle Style', 'Popularity', 'Number of Doors', 'Vehicle Size'], axis=1)\n", "df.head(5)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "u_-T2Z1-2qQC" }, "source": [ "Now, we want to rename the columns to shorten the names and removes spaces." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "colab_type": "code", "executionInfo": { "elapsed": 524, "status": "ok", "timestamp": 1568721937236, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "Qp1K3Mqi22_p", "outputId": "1df420f4-0683-4170-e4de-36ed72be5465" }, "outputs": [], "source": [ "df = df.rename(columns={\"Engine HP\": \"HP\", \"Engine Cylinders\": \"Cylinders\", \"Transmission Type\": \"Transmission\", \"Driven_Wheels\": \"Drive Mode\",\"highway MPG\": \"MPG-H\", \"city mpg\": \"MPG-C\", \"MSRP\": \"Price\" })\n", "df.head(5)" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "LXYayMyH242-" }, "source": [ "In big data sets it might also happen that we have duplicate rows. The next step is to remove those." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "executionInfo": { "elapsed": 520, "status": "ok", "timestamp": 1568721940547, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "QzjiXExxB4FY", "outputId": "aa5a1c62-bbcd-4247-ba06-d645c54f2036" }, "outputs": [], "source": [ "df.shape" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "executionInfo": { "elapsed": 509, "status": "ok", "timestamp": 1568721943782, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "5A56miC2B5oM", "outputId": "aaa9abac-0a43-4ed4-b41b-f708fdb4c7ec" }, "outputs": [], "source": [ "duplicate_rows_df = df[df.duplicated()]\n", "print(\"number of duplicate rows: \", duplicate_rows_df.shape)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 219 }, "colab_type": "code", "executionInfo": { "elapsed": 530, "status": "ok", "timestamp": 1568721945999, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "kd6ZbnYnCA2l", "outputId": "fab60585-6ea0-427c-dc3c-badb6e3fdb48" }, "outputs": [], "source": [ "df.count() " ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "z1Q-kZGUCDxc" }, "source": [ "So seen above there are 11914 rows and we are removing 989 rows of duplicate data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 206 }, "colab_type": "code", "executionInfo": { "elapsed": 508, "status": "ok", "timestamp": 1568721948257, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "ePs5vtIvCFDh", "outputId": "e91a757c-8ec8-48ef-f7fa-fdadfd620bef" }, "outputs": [], "source": [ "df = df.drop_duplicates()\n", "df.head(5)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 219 }, "colab_type": "code", "executionInfo": { "elapsed": 526, "status": "ok", "timestamp": 1568721949902, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "v2lpgJMNCI2z", "outputId": "28b63910-263e-400b-ae9e-fadf5ddf9b3e" }, "outputs": [], "source": [ "df.count()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "1wzZc_asCSSI" }, "source": [ "We return to the problem of missing (or NULL) values. There are different ways to deal with this. For our data set it makes sense to just drop rows with missing values.\n", "\n", "Depending on the problem, it is also possible to replace the missing values with the average of the column." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 219 }, "colab_type": "code", "executionInfo": { "elapsed": 516, "status": "ok", "timestamp": 1568721951785, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "qVj8rtbuC9GV", "outputId": "ba9da270-f625-449b-fb01-498a9aae3f85" }, "outputs": [], "source": [ "print( df.isnull().sum() )" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 219 }, "colab_type": "code", "executionInfo": { "elapsed": 518, "status": "ok", "timestamp": 1568721953535, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "52YaRS3jDD3v", "outputId": "d6eab4ad-9d21-4e07-a4a4-e38e52a27c9b" }, "outputs": [], "source": [ "df = df.dropna() # dropping the missing values.\n", "df.count()" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 219 }, "colab_type": "code", "executionInfo": { "elapsed": 549, "status": "ok", "timestamp": 1568721955878, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "9XGlYR5rDOol", "outputId": "3a11857d-3f3a-4da1-ade2-1b56c5fa09d6" }, "outputs": [], "source": [ "print(df.isnull().sum()) # after dropping the values" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "IOP8TuWRDRKS" }, "source": [ "Before we visualize the data, one problem remains - outliers. Outliers are in general points that don't really fit with the rest of the data. The meaning and importance of outliers depends of course on the kind of problem you are considering.\n", "\n", "The outlier detection and removing that we are going to perform is called IQR score technique. Often outliers can be seen with visualizations using a box plot. Shown below are the box plot of MSRP, Cylinders, Horsepower and EngineSize. Some points are outside the box which we consider outliers. " ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 309 }, "colab_type": "code", "executionInfo": { "elapsed": 737, "status": "ok", "timestamp": 1568721960235, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "qX6odvEjEUkh", "outputId": "fe4b12ac-caa8-4249-c65b-f15ed000fa28" }, "outputs": [], "source": [ "sns.boxplot(x=df['Price'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 309 }, "colab_type": "code", "executionInfo": { "elapsed": 732, "status": "ok", "timestamp": 1568721963258, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "wl2JnXMkEZFy", "outputId": "4779d7da-10c2-446c-fb0a-b7f4024c6283" }, "outputs": [], "source": [ "sns.boxplot(x=df['HP'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 309 }, "colab_type": "code", "executionInfo": { "elapsed": 544, "status": "ok", "timestamp": 1568721965317, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "0MZwjuIxEZlj", "outputId": "01bb96c8-f4c3-466d-b15e-8adf09aff35a" }, "outputs": [], "source": [ "sns.boxplot(x=df['Cylinders'])" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 146 }, "colab_type": "code", "executionInfo": { "elapsed": 522, "status": "ok", "timestamp": 1568721968037, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "A5z0Lk3nEcGa", "outputId": "90ea9c33-f3e9-4658-f5c3-b70e9a948977" }, "outputs": [], "source": [ "Q1 = df.quantile(0.25)\n", "Q3 = df.quantile(0.75)\n", "IQR = Q3 - Q1\n", "print(IQR)" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 35 }, "colab_type": "code", "executionInfo": { "elapsed": 531, "status": "ok", "timestamp": 1568721970023, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "X4uPBFrhEobW", "outputId": "d68358a7-67b8-4915-d904-ad50b354c160" }, "outputs": [], "source": [ "df = df[~((df < (Q1 - 1.5 * IQR)) |(df > (Q3 + 1.5 * IQR))).any(axis=1)]\n", "df.shape" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "p69TphinEqsU" }, "source": [ "This did not completely remove the outliers, but we dealt with most of them. Please read up on the IQR score technique if you are interested in the idea behind it." ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "quwx2nv9E6m2" }, "source": [ "# Visualization\n", "\n", "Visualization is an important part of data analysis. It helps you to get a better understanding of your data and it surely is necessary if you want to present your results. \n", "\n", "**Histogram**\n", "\n", "Histogram refers to the frequency of occurrence of variables in an interval. In this case, there are mainly ten different types of car manufacturing companies, but it is often important to know who has the most number of cars. To do this histogram is one of the trivial solutions which lets us know the total number of car manufactured by a different company." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 448 }, "colab_type": "code", "executionInfo": { "elapsed": 1150, "status": "ok", "timestamp": 1568721973150, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "suWoPgW3Foip", "outputId": "eb20cb51-9df8-4e5d-bfb6-205bf8454847" }, "outputs": [], "source": [ "df.Make.value_counts().nlargest(40).plot(kind='bar', figsize=(10,5))\n", "plt.title(\"Number of cars by make\")\n", "plt.ylabel('Number of cars')\n", "plt.xlabel('Make')" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "TqEGZYSgFr9d" }, "source": [ "**Heat Maps**\n", "\n", "Heat Maps is a type of plot which is necessary when we need to find the dependent variables. One of the best way to find a relationship between the features can be done using heat maps. In the below heat map we get the idea that the price feature depends on the Engine Size, Horsepower, and Cylinders.\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 549 }, "colab_type": "code", "executionInfo": { "elapsed": 1137, "status": "ok", "timestamp": 1568721979031, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "Pjo-2y2EGgn7", "outputId": "08ca16f8-0c41-4bbf-f5f9-8492f13717a8" }, "outputs": [], "source": [ "plt.figure(figsize=(10,5))\n", "c = df.corr()\n", "sns.heatmap(c,cmap=\"BrBG\",annot=True)\n", "c" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "JEBtlr2MGgwk" }, "source": [ "**Scatterplot**\n", "\n", "We generally use scatter plots to find the correlation between two variables. Here the scatter plots are plotted between Horsepower and Price and we can see the plot below. \n" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": { "base_uri": "https://localhost:8080/", "height": 400 }, "colab_type": "code", "executionInfo": { "elapsed": 1033, "status": "ok", "timestamp": 1568721982415, "user": { "displayName": "Michael Schmischke", "photoUrl": "https://lh3.googleusercontent.com/a-/AAuE7mBk37xIhy2lxwcd2GDv2vS9Qa507gTMNY6NfYkK=s64", "userId": "08419039828870973803" }, "user_tz": -120 }, "id": "p5fTWss1GuOR", "outputId": "b99163ef-68da-43ee-92c3-5a130e4f0cba" }, "outputs": [], "source": [ "fig, ax = plt.subplots(figsize=(10,6))\n", "ax.scatter(df['HP'], df['Price'])\n", "ax.set_xlabel('HP')\n", "ax.set_ylabel('Price')\n", "plt.show()" ] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "Eq8xpJQRH_eg" }, "source": [ "# Do it yourself!\n", "\n", "It is time to try a data set for yourself! The code below loads data about the AirBnb usage in New York City." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "lw8l6-ftKVuz" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "KS5JpXHDKb6p" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "0aJm5WgEKcC8" }, "outputs": [], "source": [] }, { "cell_type": "code", "execution_count": null, "metadata": { "colab": {}, "colab_type": "code", "id": "1P6d9dzpKcQD" }, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "colab_type": "text", "id": "-5xentPcKWGp" }, "source": [ "Sheet adapted from content in https://github.com/Tanu-N-Prabhu/Python/." ] } ], "metadata": { "colab": { "collapsed_sections": [], "name": "sheet_02_solution.ipynb", "provenance": [], "version": "0.3.2" }, "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.7.3" } }, "nbformat": 4, "nbformat_minor": 1 }