{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Timestamp analysis\nI also want to take a look at the timestamp. I especially want to focus on finding a good offset to organize the timestamp in days/weeks since there are already a few scripts analyzing the general structure of timestamps."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# This Python 3 environment comes with many helpful analytics libraries installed\n# It is defined by the kaggle/python docker image: https://github.com/kaggle/docker-python\n# For example, here's several helpful packages to load in \n\nimport numpy as np\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\n%matplotlib inline\n\n# Input data files are available in the \"../input/\" directory.\n# For example, running this (by clicking run or pressing Shift+Enter) will list the files in the input directory\n\nfb_train = pd.read_csv('../input/train.csv')\nfb_test = pd.read_csv('../input/test.csv')\n\n# Any results you write to the current directory are saved as output.\n"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "fb_train.describe()"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "common_places = fb_train.place_id.value_counts()[0:30].index\ntrain_common = fb_train[fb_train.place_id.isin(common_places)]\ntrain_common.shape"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "train_common.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "These are the offset values I empirically chose. I first examined the offset for each day and afterwards the weekly offset. I will explain later why I think that these offsets are reasonable."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "day_offset = 9*60\nweek_offset = 60*24*3\ncomplete_offset = day_offset + week_offset\n\nnumber_of_places_to_display = 12"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Let's plot a daily aggregate of all check-ins for the most common places and see if my chosen offset is reasonable:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,21))\nplt.title('Daily check-ins')\nfor i in range(0,number_of_places_to_display):\n    ax = plt.subplot(number_of_places_to_display/3,3,i+1)\n    ax.set_xlim([0, 60*24])\n    sns.distplot((train_common[train_common.place_id==common_places[i]].time + complete_offset)%(60*24), bins=100)\n    plt.xlabel(\"Time\")\n    plt.ylabel(\"Fraction\")\n    plt.xticks(np.arange(0, 60*24, 60))\n    labels = [i for i in range(0,25)]\n    ax = plt.gca()\n    ax.set_xticklabels(labels)\n    plt.title(\"pid: \" + str(common_places[i]))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now do the same with a weekly aggregate for each place..."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,21))\nplt.title('Weekly check-ins')\nfor i in range(0,number_of_places_to_display):\n    ax = plt.subplot(number_of_places_to_display/3,3,i+1)\n    ax.set_xlim([0, 60*24*7])\n    sns.distplot((train_common[train_common.place_id==common_places[i]].time + complete_offset)%(60*24*7), bins=100)\n    plt.xlabel(\"Time\")\n    plt.ylabel(\"Fraction\")\n    plt.gca().get_xaxis().set_ticks([])\n    plt.title(\"pid: \" + str(common_places[i]))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now let's have a look at a daily and weekly aggregate of all common places and let's see if my chosen offset is reasonable:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,9))\n\nplt.subplot(211)\nsns.distplot((train_common.time + complete_offset)%(60*24), bins=100)\nplt.xticks(np.arange(0, 60*24, 60))\nlabels = [i for i in range(0,25)]\nax = plt.gca()\nax.set_xticklabels(labels)\n\nplt.xlabel(\"Time in h\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Daily check-ins of all common places\")\n\nplt.subplot(212)\nsns.distplot((train_common.time + complete_offset)%(60*24*7), bins=100)\nplt.xticks(np.arange(0, 60*24*7, int(60*24/4)))\nlabels = [i*6%24 for i in range(0,int(7*24/4+1))]\nax = plt.gca()\nax.set_xticklabels(labels)\n\nplt.xlabel(\"Time in h\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Weekly check-ins of all common places\")"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Daily aggregate\nThere is a small peak at 8:00 for the daily aggregate which could indicate breakfast check-ins. There is also a small peak at 12:00-13:00 which could indicate lunch check-ins. The same appears to be true for dinner between 16:00 and 19:00. There are less check-ins at night between 21:00 and 7:00 because most people probably are asleep/at home then.\n\n## Weekly aggregate\nThe plot for the daily aggregate starts at what I think is monday. The first four days are pretty similar. They have a small peak at breakfast and dinner hour each and they have less check-ins than the other days. On thursday is a high peak at dinner hour which I can't really explain. The same is the case on friday. I think the last two days are saturday and sunday because the check-ins are more uniformly distributed over the whole day and there are also more check-ins during these days. This could be the case because people don't have to work on a weekend. "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now let's have a look at a historgram with bins for each hour/day:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,9))\n\nplt.subplot(211)\nsns.distplot((train_common.time + complete_offset)%(60*24), bins=24)\nplt.xlabel(\"Time in h\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Daily check-ins of all common places\")\nplt.xticks(np.arange(0, 60*24, 60))\nlabels = [i for i in range(0,25)]\nax = plt.gca()\nax.set_xticklabels(labels)\n\nplt.subplot(212)\nsns.distplot((train_common.time + complete_offset)%(60*24*7), bins=7)\nplt.xticks(np.arange(0, 60*24*7, 60*24))\nlabels = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\nax = plt.gca()\nax.set_xticklabels(labels)\nplt.xlabel(\"Weekday\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Weekly check-ins of all common places\")"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The daily plot still shows peaks at breakfast, lunch and dinner. The weekly plot shows more check-ins at the weekend than during weekdays. "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Finally, let's add new columns for the day of week and the hour of the day in our dataset:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "fb_train['day'] = (fb_train.time + complete_offset)  % (60*24*7) / (60*24)\nfb_train['day'] = fb_train['day'].apply(np.floor)\n\nfb_train['hour'] = (fb_train.time + complete_offset)  % (60*24) / 60\nfb_train['hour'] = fb_train['hour'].apply(np.floor)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "fb_train.describe()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Now let's plot our new classification for day/hour of each entry:"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,4))\nax = plt.subplot()\nax.set_xlim([-1, 7])\nsns.distplot(fb_train.day, kde=False)\nplt.xticks(np.arange(0, 7, 1))\nlabels = ['Monday', 'Tuesday', 'Wednesday', 'Thursday', 'Friday', 'Saturday', 'Sunday']\nax = plt.gca()\nax.set_xticklabels(labels)\nplt.xlabel(\"Weekday\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Weekly check-ins of all places\")"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The daily check-ins seem to be uniformly distributed with slightly more check-ins on Saturday and Sunday. "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "plt.figure(figsize=(18,4))\nax = plt.subplot()\nax.set_xlim([-1, 24])\nsns.distplot(fb_train.hour, kde=False)\nplt.xlabel(\"Time in h\")\nplt.ylabel(\"Fraction\")\nplt.title(\"Daily check-ins of all places\")\nplt.xticks(np.arange(0, 25, 1))\nlabels = [i for i in range(0,25)]\nax = plt.gca()\nax.set_xticklabels(labels)"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The hourly check-ins of all places seem to be almost uniformly distributed."
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Conclusion\nI think I found a good offset for the timestamp and even if it's not 100% correct it shouldn't have a big impact on classification accuracy.\n\nI hope that I could help you with my script. It's my first script on kaggle so please don't be too hard on me :-)"
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}