{"cells":[{"metadata":{},"cell_type":"markdown","source":"![](https://media2.giphy.com/media/81ecbZk1fk2yc/giphy.gif?cid=790b76114d4afd87e8f2d35cf717561a747cd51e20b095ea&rid=giphy.gif)"},{"metadata":{},"cell_type":"markdown","source":"This \"smth\" Notebook has codes from @drcapa Rico Hoffman https://www.kaggle.com/drcapa/categorical-feature-engineering-2-xgb/output\n* He was inspired by https://www.kaggle.com/shahules/an-overview-of-encoding-techniques"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list all files under the input directory\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n for filename in filenames:\n print(os.path.join(dirname, filename))\n\n# Any results you write to the current directory are saved as output.","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.impute import SimpleImputer\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.preprocessing import LabelEncoder\nfrom sklearn.metrics import roc_auc_score\nfrom xgboost import XGBClassifier\nimport scipy.special\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"path_in = '../input/cat-in-the-dat-ii/'\nprint(os.listdir(path_in))","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"train_data = pd.read_csv(path_in+'train.csv', index_col=0)\ntest_data = pd.read_csv(path_in+'test.csv', index_col=0)\nsamp_subm = pd.read_csv(path_in+'sample_submission.csv', index_col=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test_data.head()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar(data, name):\n data_label = data[name].value_counts()\n dict_train = dict(zip(data_label.keys(), ((data_label.sort_index())).tolist()))\n names = list(dict_train.keys())\n values = list(dict_train.values())\n plt.bar(names, values)\n plt.grid()\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"def plot_bar_compare(train, test, name, rot=False):\n \"\"\" Compare the distribution between train and test data \"\"\"\n fig, axs = plt.subplots(1, 2, figsize=(9, 3), sharey=True)\n \n train_label = train[name].value_counts().sort_index()\n dict_train = dict(zip(train_label.keys(), ((100*(train_label)/len(train.index)).tolist())))\n train_names = list(dict_train.keys())\n train_values = list(dict_train.values())\n \n test_label = test[name].value_counts().sort_index()\n dict_test = dict(zip(test_label.keys(), ((100*(test_label)/len(test.index)).tolist())))\n test_names = list(dict_test.keys())\n test_values = list(dict_test.values())\n \n axs[0].bar(train_names, train_values, color='yellowgreen')\n axs[1].bar(test_names, test_values, color = 'sandybrown')\n axs[0].grid()\n axs[1].grid()\n axs[0].set_title('Train data')\n axs[1].set_title('Test data')\n axs[0].set_ylabel('%')\n if(rot==True):\n axs[0].set_xticklabels(train_names, rotation=45)\n axs[1].set_xticklabels(test_names, rotation=45)\n plt.show()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](http://media0.giphy.com/media/3o6vY7kb1ySy3G2i08/giphy.webp?cid=790b76115682cde32a72e3324d29f4955a551672c1623cd5&rid=giphy.webp)"},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"print('# samples train:', len(train_data))\nprint('# samples test:', len(test_data))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"cols_with_missing_train_data = [col for col in train_data.columns if train_data[col].isnull().any()]\ncols_with_missing_test_data = [col for col in test_data.columns if test_data[col].isnull().any()]\nprint('train cols with missing data:', cols_with_missing_train_data)\nprint('test cols with missing data:', cols_with_missing_test_data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true,"_kg_hide-output":true},"cell_type":"code","source":"train_data.columns","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"train_data.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"![](http://media3.giphy.com/media/VbnUQpnihPSIgIXuZv/giphy.webp?cid=790b7611ac8775e44aefd3b9e20bd6e109b07a75e7b7445b&rid=giphy.webp)"},{"metadata":{},"cell_type":"markdown","source":"Since 7:00 am I'm \"writing\" (=coping) this Notebook. Now it's 04:03 pm. My eyes are burning and I couldn't make a single submission. "},{"metadata":{"trusted":true},"cell_type":"code","source":"plot_bar(train_data, 'target')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Time to go to the toilet. It takes almost 4 minutes running."},{"metadata":{},"cell_type":"markdown","source":"![](http://media2.giphy.com/media/26BoDGYuuLMb5q5X2/giphy.webp?cid=790b7611b0b39371b9dcecf24c48d1dc7b7ab1e5c3aaef86&rid=giphy.webp)"},{"metadata":{},"cell_type":"markdown","source":"I'm tired to wait for the imputation, XGboost or whatever. I need a Tiger not a cat."},{"metadata":{},"cell_type":"markdown","source":"![](http://)"},{"metadata":{},"cell_type":"markdown","source":""},{"metadata":{},"cell_type":"markdown","source":"Cats - Memory - Original Broadway Cast\n\nMidnight not a sound from the pavement/\nHas the moon lost her memory?/\nShe is smiling alone/\nIn the lamplight, the withered leaves collect at my feet/\nAnd the wind begins to moan/\nMemory, all alone in the moonlight/\nI can dream of the old days/\nI was beautiful then/\nI remember the time I knew what happiness was/\nLet the memory live again/\nEvery streetlamp seems to beat/\nA fatalistic warning/\nSomeone mutters and the street lamp gutters/\nAnd soon/\nIt will be morning/\nDaylight/\nI must wait for the sunrise/\nI must think of a new life/\nAnd I mustn''t give in./\nWhen the dawn comes/\nTonight will be a memory too/\nAnd a new day will begin/\nBurnt out ends of/\nBurnt out ends of smoky days/\nThe stale cold smell of morning/\nA streetlamp dies; another night is over/\nAnother day is dawning\nTouch me!/\nIt''s so easy to leave me/\nAll alone with the memory/\nOf my days in the sun/\nIf you touch me, you''ll understand what happiness is/\nLook, a new day has begun./\n\n*Compositores: Andrew Lloyd Webber / Trevor Nunn / T.S. Eliot / Zdenek Hruby\nLetra de Memory © Universal Music Publishing Group, Songtrust Ave, The Bicycle Music Company"},{"metadata":{},"cell_type":"markdown","source":" Sorry Rico Hoffman, I couldn't even copy your codes without receiving errors. I gave up after more than 4 hours trying to make a submission.\n* Kaggle Notebook Runner: Marília Prata @ mpwolke"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}