{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Expedia Unsupervised Learning\n-[Rishit Dagli](rishitdagli.ml)\n\n## About Me\n\n[Twitter](https://twitter.com/rishit_dagli)\n[GitHub](https://github.com/Rishit-dagli)\n[Medium](https://medium.com/@rishit.dagli)"},{"metadata":{},"cell_type":"markdown","source":"If you find the notebook useful and/or learn something from it please upvote it, the complet repository for 10 Days of ML is available here - \n\nhttps://github.com/Rishit-dagli/10-Days-of-ML\n\nPlease Star to show your support"},{"metadata":{},"cell_type":"markdown","source":"This Notebook is also a part of the 10 Days Of ML Challenge by TFUG Mumbai"},{"metadata":{},"cell_type":"markdown","source":"## Knowing about the dataset"},{"metadata":{},"cell_type":"markdown","source":"“Hotel” refers to hotels, apartments, B&Bs, hostels and other properties appearing on Expedia’s websites.  Room types are not distinguished and the data can be assumed to apply to the least expensive room type.\n\nMost of the data are for searches that resulted in a purchase, but a small proportion are for searches not leading to a purchase.\n\nMore info can be found [here](https://www.kaggle.com/c/expedia-personalized-sort/data)"},{"metadata":{},"cell_type":"markdown","source":"## Some imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd \nimport seaborn as sns\n\nimport matplotlib.pyplot as plt\n%matplotlib inline","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true},"cell_type":"code","source":"import tensorflow as tf\nimport sklearn\nfrom zipfile import ZipFile","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Unzip files"},{"metadata":{},"cell_type":"markdown","source":"There are many libraries you can use for the same but we would prefer the simple command line utility `unzip`. Please note that I am not unzipping the train and test files together as Kaggle runs out of memory during the process. You are requested to not try that on Kaggle."},{"metadata":{"trusted":true},"cell_type":"code","source":"!unzip /kaggle/input/expedia-personalized-sort/data.zip","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Load the data"},{"metadata":{},"cell_type":"markdown","source":"Thsi step can take quite some time due to huge size of the data"},{"metadata":{"trusted":true},"cell_type":"code","source":"train = pd.read_csv('train.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"View first 10 rows of the data to get an idea about it"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.head(10)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Data preprocessing"},{"metadata":{},"cell_type":"markdown","source":"The amount of null values in dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"train.isnull()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Taking notr of the above output\n\n```\nsrch_id                              0\ndate_time                            0\nsite_id                              0\nvisitor_location_country_id          0\nvisitor_hist_starrating        9412233\nvisitor_hist_adr_usd           9409918\nprop_country_id                      0\nprop_id                              0\nprop_starrating                      0\nprop_review_score                14630\nprop_brand_bool                      0\nprop_location_score1                 0\nprop_location_score2           2178380\nprop_log_historical_price            0\nposition                             0\nprice_usd                            0\npromotion_flag                       0\nsrch_destination_id                  0\nsrch_length_of_stay                  0\nsrch_booking_window                  0\nsrch_adults_count                    0\nsrch_children_count                  0\nsrch_room_count                      0\nsrch_saturday_night_bool             0\nsrch_query_affinity_score      9281966\norig_destination_distance      3216461\nrandom_bool                          0\ncomp1_rate                     9681724\ncomp1_inv                      9663097\ncomp1_rate_percent_diff        9732623\ncomp2_rate                     5876897\ncomp2_inv                      5665992\ncomp2_rate_percent_diff        8807683\ncomp3_rate                     6858257\ncomp3_inv                      6625309\ncomp3_rate_percent_diff        8973523\ncomp4_rate                     9297431\ncomp4_inv                      9225059\ncomp4_rate_percent_diff        9653317\ncomp5_rate                     5473236\ncomp5_inv                      5196697\ncomp5_rate_percent_diff        8236524\ncomp6_rate                     9435043\ncomp6_inv                      9393385\ncomp6_rate_percent_diff        9724218\ncomp7_rate                     9286453\ncomp7_inv                      9204355\ncomp7_rate_percent_diff        9639692\ncomp8_rate                     6098487\ncomp8_inv                      5957142\ncomp8_rate_percent_diff        8691823\nclick_bool                           0\ngross_bookings_usd             9640938\nbooking_bool                         0\ndtype: int64\n```\n\nWe see that majority of values in some columns are null, to prevent us from making some very bad models it is safe to remove them. Also note that imputation i not preferred here and is an wasted effort as we have very few values"},{"metadata":{},"cell_type":"markdown","source":"You can use this code to create a sample visualization however I have commented it out as Kaggle exhausts its resources in doing so. You can try this out in your local machine or cloud"},{"metadata":{"trusted":true},"cell_type":"code","source":"'''\ndf.plot(x='date_time', y = 'price_usd', figsize = (20,5))\nplt.xlabel('Date time')\nplt.ylabel('Price in USD')\nplt.title('Time Series of room price by date time of search')\n'''","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I have performed PCA and got that these 3 columns have the maximum weightage on the data and the other data sources have very low or no effect on the data.\n\nTo create a good model, lets take these data columns"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = train.loc[train['prop_id'] == 104517]\n\ndf = df.loc[df['visitor_location_country_id'] == 219]\n\ndf = df.loc[df['srch_room_count'] == 1]\n\ndf = df[['date_time', 'price_usd', 'srch_booking_window', 'srch_saturday_night_bool']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Few statistics of the dataset"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We see majority of values in `price_usd` are `< 5584` categorical benchmark so what you can now do is see your stats for data below this"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.loc[df['price_usd'] < 5584]\ndf['price_usd'].describe()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Earlier this might have seemed as a time series problem to you, but given that you need to perform unsupervised learning, we will convert timestamps to numerical entities to help us and thus provide a good inference"},{"metadata":{},"cell_type":"markdown","source":"This is the range of time we nned to use"},{"metadata":{"trusted":true},"cell_type":"code","source":"print(df['date_time'].min())\nprint(df['date_time'].max())","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As, we will now be working on the `date_time` column letss see a few stats which might enable us to choose best conversion types"},{"metadata":{"trusted":true},"cell_type":"code","source":"df['date_time'].describe()\n\ndf['date_time'] = pd.to_datetime(df['date_time'])\n\ndf.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"As in time series and sequences, we create a plot of this. How this helps?\n\nWell you can atleast  eye out the trend and/or seasonality in your dataset and these are the indispensable tools on which Time Series stands. So, this is very useful"},{"metadata":{"trusted":true},"cell_type":"code","source":"df.plot(x = 'date_time', \n        y = 'price_usd', \n        figsize = (16, 8))\n\nplt.xlabel('dates')\nplt.ylabel('USD')\nplt.title('Time series of room price by date of search');","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is some pretty crazy amount of noise. But can you see one thing?\n\nNot yet, spend some time looking at it"},{"metadata":{},"cell_type":"markdown","source":"I can now eye out that my dataset probably has some autocorelation features, I see same trends, repeating at a different slope at the start of the plot, it goes on to become bad at the end, but now I have some hope that I can figure out things"},{"metadata":{},"cell_type":"markdown","source":"We just created a plot for time vs USD what next?\n\nLet's create a plot for 'srch_saturday_night_bool' and the prices, This could maybe help us. We will create a bar graph"},{"metadata":{"trusted":true},"cell_type":"code","source":"a = df.loc[df['srch_saturday_night_bool'] == 0, 'price_usd']\nb = df.loc[df['srch_saturday_night_bool'] == 1, 'price_usd']\n\nplt.figure(figsize = (16, 8))\n\nplt.hist(a, bins = 80, \n         alpha = 0.3, \n         label = 'search w/o Sat night stay')\n\nplt.hist(b, bins = 80, \n         alpha = 0.3, \n         label = 'search w/ Sat night stay')\n\nplt.xlabel('Price')\nplt.ylabel('Freq')\nplt.legend()\nplt.title('Sat night search')\nplt.plot();\n\n","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"There seems to be some strong relation between Price and Saturay Night Search that's good, I knew that from PCAA, but now I can see that out clearly by the peak"},{"metadata":{},"cell_type":"markdown","source":"And now similarly for booking_window"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.distplot(df['price_usd'], \n                 hist = False, label = 'USD')\n\nsns.distplot(df['srch_booking_window'], \n                  hist = False, label = 'booking window')\n\nplt.xlabel('dist')\nsns.despine()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.pairplot(df)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Converting the times"},{"metadata":{},"cell_type":"markdown","source":"My model understands number and I have a date this is not going to work so I need to create a new column called date_time integer version or for short date_time_int, here we will be having numeric conversion of the date_time column"},{"metadata":{"trusted":true},"cell_type":"code","source":"df = df.sort_values('date_time')\ndf['date_time_int'] = df.date_time.astype(np.int64)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"markdown","source":"A Kernel density estimation would be helpful"},{"metadata":{"trusted":true},"cell_type":"code","source":"sns.kdeplot(df[[\"price_usd\", \"srch_booking_window\", \"srch_saturday_night_bool\"]])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Models and Algorithms"},{"metadata":{},"cell_type":"markdown","source":"Now we come to the part of applying models and algorithms, I don't know which model to use. So, what will we do?\n\n### **Experiment!!!**"},{"metadata":{},"cell_type":"markdown","source":"### K-Means"},{"metadata":{},"cell_type":"markdown","source":"Let's start with the simplest and most common one. We will be using sklearn to simplify things. \n\nSpoiler alert: I shifted K-means up because at the end I saw K-Means was promising 😀 😀 "},{"metadata":{},"cell_type":"markdown","source":"Let's try experimenting with hyperparameters"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.cluster import KMeans\ndata = df[['price_usd', 'srch_booking_window', 'srch_saturday_night_bool']]\nn_cluster = range(1, 20)\n\nkmeans = [KMeans(n_clusters = i).fit(data) for i in n_cluster]\nscores = [kmeans[i].score(data) for i in range(len(kmeans))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I trained it for 1 to 19 clusters, let see the scores we receive and choose right amount of clusters, Lets see a neat looking plot of the same"},{"metadata":{"trusted":true},"cell_type":"code","source":"fig, ax = plt.subplots(figsize = (16, 8))\nax.plot(n_cluster, scores, color = 'orange')\n\nplt.xlabel('clusters num')\nplt.ylabel('score')\nplt.title('Elbow curve for K-Means')\nplt.show();","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"We come to a saturation at almost 16-17 clusters with a **wonderful score of -0.2**\n\nPlease do not get intimidated by negative sign, it shows us the distance which means you consider it as $|-0.2 | = 0.2$"},{"metadata":{},"cell_type":"markdown","source":"This is a huge dataset lets free up some resources which we do not need any longer, also includes huge variables"},{"metadata":{"trusted":true},"cell_type":"code","source":"del train","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"This is an importnat step as we are now going to load the test set which requires a good amount of resources"},{"metadata":{"trusted":true},"cell_type":"code","source":"import os\nos.remove('train.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"test = pd.read_csv('test.csv')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"del kmeans","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Lets fit the mmodel with the decided number of clusters"},{"metadata":{"trusted":true},"cell_type":"code","source":"km = KMeans(n_clusters = 17).fit(data)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"**Thinngs are getting exciting!!** We will now"},{"metadata":{"trusted":true},"cell_type":"code","source":"X = df[['price_usd', 'srch_booking_window', 'srch_saturday_night_bool']]\nX = X.reset_index(drop = True)\n\nkm.predict(X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"I didn't know about `mpl_toolkits` first, I googled it out. By, this I would like to tell you its not necessary to know everything trying to learn is mandatory. Let's create a wonderful 3D plot for predictions"},{"metadata":{"trusted":true},"cell_type":"code","source":"from mpl_toolkits.mplot3d import Axes3D\nfig = plt.figure(1, figsize = (7, 7))\n\nax = Axes3D(fig, rect = [0, 0, 0.95, 1], \n            elev = 48, azim = 134)\n\nax.scatter(X.iloc[:, 0], \n           X.iloc[:, 1], \n           X.iloc[:, 2],\n           c = km.labels_.astype(np.float), edgecolor = 'm')\n\nax.set_xlabel('USD')\nax.set_ylabel('srch_booking_window')\nax.set_zlabel('srch_saturday_night_bool')\n\nplt.title('K Means', fontsize = 10);","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Restricted Boltzman Machines"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.neural_network import BernoulliRBM\nmodel = BernoulliRBM(n_components=2)\nmodel.fit(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"model.score_samples(X)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## DB Scan"},{"metadata":{"trusted":true},"cell_type":"code","source":"from sklearn.cluster import DBSCAN\ndb = DBSCAN(eps=0.3, min_samples=10).fit(data)\ncore_samples_mask = np.zeros_like(db.labels_, dtype=bool)\ncore_samples_mask[db.core_sample_indices_] = True","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Conclusion"},{"metadata":{},"cell_type":"markdown","source":"K-Means worked extraordinarily well, similarly we had restricted boltzman machines which did a slightly better job than K-Means, but the difference is too minuscle. I also feel that Boltzman Machines are complex, making our model simple should be key so I traded some accuracy for simplicity and ideally one should do that. So, I have plotted graphs of K-Means. I would advise you to not use Boltzman Machines"}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}