{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Density Plots of Hours, Days, Months\n\nStep 1: Load data and calculate hours (1 to 24), weekdays (1 to 7), months (1 to 12) and years (1 to 2)"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport datetime\nimport scipy.signal\nimport seaborn as sns\n\nprint('Reading train and test data')\ntrain = pd.read_csv('../input/train.csv')\ntest = pd.read_csv('../input/test.csv')\n\nprint('Calculate hour, weekday, month and year for train and test')\ntrain['hour'] = (train['time']//60)%24+1 # 1 to 24\ntrain['weekday'] = (train['time']//1440)%7+1\ntrain['month'] = (train['time']//43200)%12+1 # rough estimate, month = 30 days\ntrain['year'] = (train['time']//525600)+1 \n\ntest['hour'] = (test['time']//60)%24+1 # 1 to 24\ntest['weekday'] = (test['time']//1440)%7+1\ntest['month'] = (test['time']//43200)%12+1 # rough estimate, month = 30 days\ntest['year'] = (test['time']//525600)+1"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Kdes for weekday Vs hour for 6 place ids with the highest counts. \nThe plots show a preference for certain hours and weekdays for each place id. Maybe weekends and weekdays and holidays can be separated if more place ids are analyzed."
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('group by place_id and get count')\nplaces = train[['place_id', 'time']].groupby('place_id').count()\nplaces.rename(columns={'time': 'count'}, inplace=True)\n\nprint('plot weekday Vs hour for 6 place_ids with highest counts')\nplt.figure(1, figsize=(14,10))\nplaceindex = places['count'].sort_values(ascending=False)[:6]\nfor (i, placeid) in enumerate(placeindex.index):\n    ax = plt.subplot(2,3,i+1)\n    df_place = train.query('place_id == @placeid')\n    # df_place = train.query('place_id == @placeid and year==1') # to separate by year      \n    sns.kdeplot(df_place.weekday, df_place.hour, shade=True, ax = ax)\n    plt.title(\"place_id \" + str(placeid)) \n    ax.set(xlim=(0, 8))\n    ax.set(ylim=(0, 25))"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Next, let's see the trends with weekday Vs month for the 6 place ids with highest counts"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('plot weekday Vs month for 6 place_ids with highest counts')\nplt.figure(2, figsize=(14,10))\nplaceindex = places['count'].sort_values(ascending=False)[:6]\nfor (i, placeid) in enumerate(placeindex.index):\n    df_place = train.query('place_id == @placeid and year==1')\n    ax = plt.subplot(2,3,i+1)\n    sns.kdeplot(df_place.weekday, df_place.month, shade=True, ax=ax)\n    plt.title(\"place_id \" + str(placeid)) "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "We can see the month to month changes in the above plots for each place id. Some businesses seem to be highly seasonal\n\nNext, let's plot a very small subset of train and test data and compare the kdes. This is just a confirmation that the test data is lining up well with the train data"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('plot a small XY subset of train and test data (overlaid)' )      \nxmin, xmax = 2,2.3\nymin, ymax = 2,2.3\n\nprint('train data is subset by month >=7 for first year to match test data timeperiod for second year' )      \ntrain_subset = train.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax) and (year == 1) and (month >=7)')\ntest_subset = test.query('( @xmin<= x <= @xmax) and ( @ymin<=y<= @ymax)')\nfig = plt.figure()\nax1 = fig.add_subplot(111)\nax1.scatter(train_subset['x'], train_subset['y'], s=1, c='r', marker=\"s\", label='first', edgecolors='none')\nax1.scatter(test_subset['x'], test_subset['y'], s=1, c='b', marker=\"s\", label='first', edgecolors='none')\nax1.set(xlim=(xmin, xmax))\nax1.set(ylim=(ymin, ymax))\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "Above: a scatterplot of test and train data for a very small XY window "
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('plot weekday Vs hour kdes for train and test subsets' )  \nsns.kdeplot(train_subset.weekday, train_subset.hour, shade=True)\nplt.title(\"Train data, X: \"+ str(xmin) + ' to ' + str(xmax) + ' , ' + \"Y: \" + str(ymin) + ' to ' + str(ymax) )\nplt.show()\nsns.kdeplot(test_subset.weekday, test_subset.hour, shade=True)\nplt.title(\"Test data, X: \"+ str(xmin) + ' to ' + str(xmax) + ' , ' + \"Y: \" + str(ymin) + ' to ' + str(ymax) )\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The plots are similar to each other, so the calculations seem to be correct"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "print('plot weekday Vs month kdes for train and test subsets' ) \nsns.kdeplot(train_subset.weekday, train_subset.month, shade=True)\nplt.title(\"Train data, X: \"+ str(xmin) + ' to ' + str(xmax) + ' , ' + \"Y: \" + str(ymin) + ' to ' + str(ymax) )\nplt.show()\nsns.kdeplot(test_subset.weekday, test_subset.month, shade=True)\nplt.title(\"Test data, X: \"+ str(xmin) + ' to ' + str(xmax) + ' , ' + \"Y: \" + str(ymin) + ' to ' + str(ymax) )\nplt.show()  "
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "The weekday vs month plots seem to have some differences. Not sure why. For the test dataset, the data does not cover the entire 12th month, maybe that's why the 12th month has very little events"
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}