{"cells":[{"cell_type":"markdown","metadata":{},"source":"#Facebook Challenge 5 - Predicting Check-in\n##Notebook #1: Data Exploration\n###Introduction:\nThis Facebook Challenge is concerned with predicting the business that is associated with each check-in events. \nIn particular, as described by the competition host, the businesses are distributed within an area defined by\na square of 10km by 10km. The participants are given a set of training data, which include: x, y (the x,y coordinates), \ntime (time at which the check-in event occurs), accuracy (definition unknown), and place_id (the business which we need to predict). \nIn this first notebook, the nature of the training data will be investigated using statistical and visualisation tools.\n"},{"cell_type":"markdown","metadata":{},"source":"###Overview:\nTo investigate the training data, the first step is obviously to load the data. Pandas will be used as the \nprimary tool for this exercise."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"import numpy as np \nimport pandas as pd \n\n\ntraining_data = pd.read_csv(\"../input/train.csv\")\ntest_data = pd.read_csv(\"../input/test.csv\")\n\ntraining_data.head()\n"},{"cell_type":"markdown","metadata":{},"source":"It can be seen from the first few rows of data that as expected, the x and y values are \nfloating point numbers in between 0 and 10 (This will be checked later) corresponding to the check-in coordinates\nwithin the 10km by 10km area in which the businesses are located. It can also be seen that the accuracy is also a \nfloating point number, and from first glance it seems that the accuracy might be quoted as percentages (i.e. floating\npoints of magnitude between 0 and 100). Both time and place_id seems to be very large integers. In the next sections, each of these \nfeatures will be investigated in details. "},{"cell_type":"markdown","metadata":{},"source":"###Time:\n\nTime is the first feature to be investigated, as it is the most important feature after the x,y coordinates. That is because each\nbusiness would have different opening hours, and each business will experience different busy and quiet periods, depending on \nthe nature of the business. For example, a night club would probably have significantly higher customer influx (hence more check-in\nevents) on Friday Night and Saturday Night, whereas a bakery or cafe would probably have higher customer influx on weekday mornings. \nThis can potentially be further extended to months of the year: a ski-shop is more likely to have more customers just before winter, \nwhile a surf shop would have more customers just before or during summer. Therefore, the time of check-in is going to be the key to distingushing \nbetween businesses that are located close together. \n\nHowever, unlike the x,y coordinates, the unit time provided in the data is ambiguious. Therefore, compared to the x,y coordinate, much more \ninvestigation will be needed to make use of the time feature. This would be the goal of this section. \n\nThe first investigation will be a comparison between the time record for the training data and test data:\n"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.time.describe()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"test_data.time.describe()"},{"cell_type":"markdown","metadata":{},"source":"The first thing that can be noted from these two summaries is the fact that the test data time follows the training data time. This means that all events in\nthe test data set happens after the training data set. This is not surprising, as it has been vaguely mentioned in the competition details that the test data and \nthe training data are divided in time. This essentially means that the time (or a scaled version of it) cannot be used directly for fiting a predictive model, as \nthe time range in the test and training data sets are different. The time need to be converted into a form that is common to both data sets: a cyclical expression \nof the time, i.e. time of the day, time of the week, and/or time of the year. This is consistent with the need of a cyclical measure of time to represent \nthe dependence of business on time as described above."},{"cell_type":"markdown","metadata":{},"source":"From the extremely large values of the time record, it can be concluded that it is impossible for the time to be recorded in months, days or hours. The only possibilities\nare minutes, seconds and milliseconds (nanosecond is too short for human activities). In order to find out, the histogram of a selected data range \ncan be plotted by first assuming that time is recorded in unit of minutes"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"#Choose an arbitary range for dividing the data into days\ndata_range = [400000, 500000]\nno_of_days_train = (500000 - 400000)/(60*24)\ntraining_data.time.hist(bins = int(no_of_days_train), range = data_range)"},{"cell_type":"markdown","metadata":{},"source":"There is not a clear pattern for when the training data is divided this way. But how about the test data?"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"#Choose an arbitary range for dividing the data into days\ndata_range = [800000, 900000]\nno_of_days_test = (900000 - 800000)/(60*24)\ntest_data.time.hist(bins = int(no_of_days_test), range = data_range)"},{"cell_type":"markdown","metadata":{},"source":"There is a very clear pattern here, that there is a periodicity with a spike occuring every 7 bins\nSince each bin represents a day, this corresponds well with the occurance of weekends. Hence, it can be concluded that \nthe time data is represented in minutes. Note that this is consistent with what can be found on your facebook \npage: if you go to security settings -> where you logged in, you will find that your log-ins are recorded\nin date and time to the nearest minute. So time is recorded in minutes"},{"cell_type":"markdown","metadata":{},"source":"###place_id\nplace_id records the business that is present in the area. Let's first look at the basic statistics."},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.place_id.describe()"},{"cell_type":"markdown","metadata":{},"source":"It can be seen that place_id are big numbers. Furthermore, since there are about 30 million entries, but the \nnumbers go from about 100 million to 1 billion, it can be conclude that the place_id are discrete. In fact, there are:"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.place_id.nunique()"},{"cell_type":"markdown","metadata":{},"source":"So there are just over 100k unique ids. This immediately points to the need of partitioning the data, as predicting\n100k classes with one single model would be difficult. Finally, the code below gives the most popular business in the \narea"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.place_id.value_counts()[:10]"},{"cell_type":"markdown","metadata":{},"source":"It seems that there is not a a dominant business that overwhelm other businesses. Again, this is what is expected."},{"cell_type":"markdown","metadata":{},"source":"### x, y\nAgain, starting with basic statistics, we have:"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.x.describe()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.y.describe()"},{"cell_type":"markdown","metadata":{},"source":"This confirms that x and y are floates between 0 and 10. To investigate in more details the relationship between \nx, y and place_id, Let's select a particular place_id and analysis that data"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"X1 = training_data.query('place_id == 8772469670')\nX1.x.describe()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"X1.y.describe()"},{"cell_type":"markdown","metadata":{},"source":"It can be seen from above that the spread in y is significantly smaller than x. This means that y will be a \nbetter predictor for predicting the business compared to x. This information can be visualised as follows:"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"from matplotlib import pyplot as plt\nfrom matplotlib import patches as patches\nmean_x = np.mean(X1.x)\nmean_y = np.mean(X1.y)\nstd_x = np.std(X1.x)\nstd_y = np.std(X1.y)\nfig = plt.figure()\nax1 = fig.add_subplot(111)\nax1.axis([0,10,8.15,8.45])\nax1.set_xlabel('x')\nax1.set_ylabel('y')\nax1.set_title('distribution of instances corresponding to place_id = 8772469670')\nax1.add_patch(patches.Rectangle((mean_x - 2*std_x, mean_y - 2*std_y), 4*std_x, 4*std_y, fill = False))\nax1.scatter(X1.x, X1.y, s = 40, marker = 'x')"},{"cell_type":"markdown","metadata":{},"source":"The graph above shows the distribution of instances corresponding to place_id = 877249670. It can be seen clearly \nthat the y-values are much tighter (plotted within a 0.3 range) compared to the x-value which is spread across\nthe whole 0 - 10 axis. The little square in the plot shows an area which is +/- 2 standard deviations away from\nthe mean. But now if we look at data points within the plotted area that is not of this place_id"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"X2 = training_data.query('x < 4 and x > 2 and y > 8.15 and y < 8.45 and place_id != 8772469670')\nX2.place_id.nunique()"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"len(X2)"},{"cell_type":"markdown","metadata":{},"source":"It can be seen that there are 70k data points of 3000+ different place_id within this small area. To make it \nmore dramatic, here is the plot of all this data points, with the rectangle representing the area that is 2 \nstandard deviation from the mean of place_id = 877249670 still drawn:"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"fig = plt.figure()\nax1 = fig.add_subplot(111)\nax1.axis([2,4,8.25,8.35])\nax1.set_xlabel('x')\nax1.set_ylabel('y')\nax1.set_title('distribution of instances of other place_ids')\nax1.add_patch(patches.Rectangle((mean_x - std_x, mean_y - std_y), 2*std_x, 2*std_y, fill = False))\nax1.scatter(X2.x, X2.y, s = 40, marker = 'x')"},{"cell_type":"markdown","metadata":{},"source":"Of course, it is expected that when the time data should be able to more finely divide these data points. However, \nit shows how closely spaced these data points are in x and y, and how difficult it will be to model these data points. "},{"cell_type":"markdown","metadata":{},"source":"### Accuracy\nAccuracy is the last feature that needs to be explored. Starting with basic statistics:\n"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"training_data.accuracy.describe()"},{"cell_type":"markdown","metadata":{},"source":"It can be seen that the accuracy seems to be mostly under 100, but the max in not - is it possible that it is \nindeed a percentage measure, and anything above 100 is error in the data? If it is indeed a percentage score, then \nwe expect that data points with high accuracy should cluster together within the same place_id. This can be investigated\nby plotting a scatter plot, again using place_id = 877249670 as a sample"},{"cell_type":"code","execution_count":null,"metadata":{"collapsed":false},"outputs":[],"source":"import matplotlib.colors as colors\n\n# define the colormap\ncmap = plt.cm.jet\n## extract all colors from the .jet map\ncmaplist = [cmap(i) for i in range(cmap.N)]\n## create the new map\ncmap = cmap.from_list('Custom cmap', cmaplist, cmap.N)\nbounds = np.array([0, 20, 40, 60, 80, 100])\nnorm = colors.BoundaryNorm(boundaries=bounds, ncolors=cmap.N)\nX1_sample = X1.sample(500)\nfig = plt.figure()\nax1 = fig.add_subplot(111)\nax1.axis([2,4,8.25,8.35])\nax1.set_xlabel('x')\nax1.set_ylabel('y')\nax1.set_title('distribution instances of place_id = 8772469670, colored by accuracy')\nax1.add_patch(patches.Rectangle((mean_x - std_x, mean_y - std_y), 2*std_x, 2*std_y, fill = False))\nax1.scatter(X1_sample.x, X1_sample.y, s = 40, marker = 'x', c = X1_sample.accuracy, cmap = cmap, norm = norm)"},{"cell_type":"markdown","metadata":{},"source":"It can be seen that there is no clear clustering of colours. The data is everywhere. In particular, high accuracy\nscores (red and orange) can be found both within and out of the 2-standard deviation area. Simiarly for low accuracy\nSo at this point, it is unclear what the accuracy is. However, a quick search online reveal that in usual cases for \ndetecting IP locations, accuracy is usually quoted as the confidence that the data is within a certain km. So a accuracy\nof 10 would mean that there is a 99% confidence that the actual location is within 10km (or m) of the quoted coordinates\nIf that is true, then in actual fact, lower accuracy means that the data x,y coordinates are more accurate. "},{"cell_type":"markdown","metadata":{},"source":"Unfortunately, due to time constrains, there is no time to investigate the accuracy value further. \nMore information would probably reveal itself when we perform the actual modelling. "}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}},"nbformat":4,"nbformat_minor":0}