{"cells":[{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Diagnose data for cleaning** :\n    Start with Load data using pandas, then visually inspect to get some information about data."},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\ndata=pd.read_csv(\"../input/lower-back-pain-symptoms-dataset/Dataset_spine.csv\") \ndata.head()\ndata.shape\ndata.info()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Exploratory data analysis**\n    1. *Data preparations*\n        : label columns and convert categorical features to numerical value : (Abnormal = 1 , Normal = 0) \n        \n    2. *Frequency count*\n        : Observe number of normal and abnormal labels \n        \n    3. *Summary statistics*\n        3.1 Statistical analysis \n            :use describe method which return statictical numeric data such as Count, mean, SD, min, 25%, 50%, 75% and max values in the dataset. \n            \n        3.2 Detecting outlier using boxplot \n            : Outliers are datapoints that line outside of the distribution of the dataset. It could cause significant impact on statiscal analysis.\n            : IQR(interquatile range) determined by Q3-Q1, IQR can be used to measure how spread-out of values from the central value. We assume that values suppose to clusterd around some central value, so if some other values are \"too far\"from the central values will considered as \"Outlier\"\n            \n        3.3 Correcting outlier based on IQR scores\n        The data points are correcting by using this criteria: \n            If any points which falls more than 1.5*IQR range above Q3 or below Q1 will considered as outlier and replace using mean value of the dataset.\n            \n    4. *Data Visualization*\n        4.1 Observe correlation between features using heatmap\n        4.2 Observe distribution of each features using pairplot\n        \n"},{"metadata":{"trusted":true},"cell_type":"code","source":"import pandas as pd\nfrom sklearn.preprocessing import StandardScaler \ndata=pd.read_csv(\"../input/lower-back-pain-symptoms-dataset/Dataset_spine.csv\") \ndata.drop('Unnamed: 13', axis=1, inplace=True) #drop unwated colums\ndata=pd.concat([data,pd.get_dummies(data['Class_att'])],axis=1)# convet categorical feature using dummies\ndata.drop(['Class_att','Normal'], axis=1, inplace=True)\ndata.columns = ['Pelvic Incidence','Pelvic Tilt','Lumbar Lordosis Angle','Sacral Slope','Pelvic Radius', \n                'Spondylolisthesis Degree', 'Pelvic Slope', 'Direct Tilt', 'Thoracic Slope', \n                'Cervical Tilt','Sacrum Angle', 'Scoliosis Slope','Labels']# Rename columns\n\nnum_ab=data.Labels.sum()  # number of abnormal label\nnum_nor= data.Labels.count()-num_ab  # number of normal label\n\ndata.describe() # statiscal analysis\n\nimport matplotlib.pyplot as plt # Visualize outlier using boxplot\nplt.subplots(figsize=(15,6))\n\ndata.boxplot(patch_artist=True, sym=\"k.\")\nplt.xticks(rotation=90)\n","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# correcting outlier using IQR score\nminimum = 0\nmaximum = 0\n\ndef detect_outlier(feature):\n    q1, q3= np.percentile(X[feature],[25,75])\n    iqr = q3 - q1\n    lower= q1 -(1.5 * iqr) \n    upper= q3 +(1.5 * iqr) \n    X.loc[X[feature] < lower, feature] = X[feature].median() \n    X.loc[X[feature] > upper, feature] = X[feature].median()\nX = data.iloc[:, :-1]\nfor i in range(len(X.columns)): \n        detect_outlier(X.columns[i])\nplt.subplots(figsize=(15,6))\nX.boxplot(patch_artist=True, sym=\"k.\")\nplt.xticks(rotation=90)\n# observe again with boxplot","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Visualization** : heatmap"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\nplt.subplots(figsize=(12,8))\nhm = sns.heatmap(data.corr(), cmap='YlGnBu',annot=True)\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Visualization** : Distribution of each feature using pairplot"},{"metadata":{"trusted":true},"cell_type":"code","source":"import seaborn as sns\nimport matplotlib.pyplot as plt\nsns.pairplot(data, hue=\"Labels\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Data Preprocessing**:\n 1. Normalizing (stadardization)\n 2. Dimension reduction using PCA\n     In order to be more efficient in computation and storage, some features are determined as less informative (noise) have to removed.\n"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.preprocessing import StandardScaler \nfrom sklearn.decomposition import PCA\nfrom sklearn.pipeline import make_pipeline\nimport matplotlib.pyplot as plt\nScaler = StandardScaler()\npca=PCA()\npipeline=make_pipeline(Scaler,pca)\npipeline.fit(data)\nfeatures=range(pca.n_components_)\nnon_standard=plt.figure(figsize=(10,8))\nplt.bar(features,pca.explained_variance_)\nplt.xticks(features)\nplt.title('Observe intrinsic dimension')\nplt.ylabel('Variance')\nplt.xlabel('PCA feature')\n\n\n","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}