{"cells":[
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "# Is there any structure to the timestamp data?"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "import pandas as pd\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport datetime\nimport scipy.signal\n\n%matplotlib inline\nplt.rcParams['figure.figsize'] = (15.0, 15.0)\n\nprint('Reading train data')\ndf_train = pd.read_csv('../input/train.csv')\n\nprint('\\nSize of training data: ' + str(df_train.shape))\nprint('Columns:' + str(df_train.columns.values))\nprint('Number of places: ' + str(len(list(set(df_train['place_id'].values.tolist())))))\nprint('\\n')"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Estimate the autocorrelation of all of the timestamps to see if there are any repeating patterns"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# Pull out timestamps\ntimes = np.squeeze(df_train.as_matrix(columns=['time']))\n\nn_events = times.size\nn_samples = n_events\nhist_range = (-100000.0, 100000.0)\nn_bins = 100000 # One bin per 2.0 time units\nn_loops = 100\n\n# Estimate autocorrelations\n# Randomly pull timestamps and subtract from each other. Get histogram of these values toestimate autocorrelation\nprint('Estimating autocorrelation')\nall_autocorrs_global = np.zeros((n_bins, n_loops))\nfor loop_n in range(n_loops):\n  hist_vals, bin_edges = np.histogram(np.random.choice(times, size=n_samples, replace=True) - \\\n                                      np.random.choice(times, size=n_samples, replace=True), bins=n_bins, range=hist_range)\n  all_autocorrs_global[:, loop_n] = hist_vals\n\n# Plot the autocorrelation and fft of autocorrelation\nfig, axs = plt.subplots(2,1)\n\naxs[0].plot(bin_edges[:-1], all_autocorrs_global)\naxs[0].set_xlim([-20000.0, 20000.0])\naxs[0].set_title('Autocorrelation of all timestamps')\naxs[0].set_xlabel('time units')\n\n# Plot the fft of the autocorrelation\nf, psd = scipy.signal.welch(all_autocorrs_global, nperseg=25000, noverlap=20000, return_onesided=True, axis=0)\n\n# Adjust the X axis to be in time points instead of 1/F\nf /= 2.0  # Remember that there is one bin per 4.0 time units\nf = 1.0/f # Go back to time points\n\n# Plot the fft of the autocorrelation (The PSD of the timestamps)\naxs[1].plot(f, np.log(psd))\naxs[1].set_title('Log FFT of global autocorrelations')\naxs[1].set_xlabel('time units')\naxs[1].set_xlim([0.0, 20000.0])\naxs[1].set_xticks(np.arange(0.0, 20000.0, 1000))\naxs[1].grid(True)\n\nfig.tight_layout()\n\nplt.show()"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## Each of the lines in these plots represent an iteration of random samples.\n\n## The top plot is the autocorrelation. Any peaks or waves in the autocorrelation would indicate some kind of repeating pattern in the data.\n\n## The bottom plot is created from the FFT of the autocorrelation, which gives the power spectral density (PSD). Note that the x axis is the inverse of frequency, so it corresponds to raw time units. There are no clear peaks in the PSD.\n\n## There does not appear to be any global structure to the timestamps. Maybe there is for individual places?"
 },
 {
  "cell_type": "code",
  "execution_count": null,
  "metadata": {
   "collapsed": false
  },
  "outputs": [],
  "source": "# Get places with most check-ins\nplaces_by_frequency = df_train.groupby('place_id')['place_id'].agg('count').sort_values(ascending=False).index.tolist()\nn_places_to_analyze = 100\nplaces_by_frequency = places_by_frequency[:n_places_to_analyze]\n\n\nhist_range = (-100000.0, 100000.0)\nn_bins = 50000 # One bin per 4.0 time units\n\n# Get the autocorrelation between timestamps for each place\nall_autocorrs = np.zeros((n_bins, n_places_to_analyze))\nplace_n = 0\nfor place_id in places_by_frequency:\n  times = np.squeeze(df_train[df_train['place_id']==place_id].as_matrix(columns=['time']))\n  n_events = times.size\n  n_samples = n_events*n_events # We are still randomly choosing timestamps, but this should give good coverage\n  hist_vals, bin_edges = np.histogram(np.random.choice(times, size=n_samples, replace=True) - \\\n                                    np.random.choice(times, size=n_samples, replace=True), bins=n_bins,\n                                    range=hist_range)\n  all_autocorrs[:, place_n] = hist_vals\n  place_n += 1\n\n\n# Plot the autocorrelation and fft of autocorrelation\nfig, axs = plt.subplots(3,1)\n\naxs[0].plot(bin_edges[:-1], all_autocorrs)\naxs[0].set_title('Autocorrelations for each place')\naxs[0].set_xlabel('time units')\naxs[0].set_xlim([-20000, 20000.0])\naxs[0].set_ylim([0.0, 200.0])\n\n\nf, psd = scipy.signal.welch(all_autocorrs, nperseg=25000, noverlap=20000, return_onesided=True, axis=0)\n\n# Adjust the X axis to be in time points instead of 1/F\nf /= 4.0 # Remember that there is one bin per 4.0 time units\nf = 1.0/f # Go back to time points\n\n# Plot the fft of the autocorrelation (The PSD of the timestamps)\naxs[1].plot(f, np.log(psd))\naxs[1].set_title('Log FFT of autocorrelations for each place')\naxs[1].set_xlabel('time units')\naxs[1].set_xlim([0.0, 40000.0])\naxs[1].set_xticks(np.arange(0.0, 40000.0, 2000))\naxs[1].grid(True)\n\n# Plot the fft of the autocorrelation (zoomed)\naxs[2].plot(f, np.log(psd))\naxs[2].set_title('Log FFT of autocorrelations for each place')\naxs[2].set_xlabel('time units')\naxs[2].set_xlim([0.0, 2500.0])\naxs[2].set_xticks(np.arange(0.0, 2500.0, 100))\naxs[2].grid(True)\n\nfig.tight_layout()\nplt.show()\n"
 },
 {
  "cell_type": "markdown",
  "metadata": {},
  "source": "## That's more like it!\n\n## There is clearly a repeating pattern to this data. A wavelike pattern is seen in the autocorrelations and there are clear peaks in the PSD.\n\n## The PSDs give some insight into the different time cycles occurring at each place.\n\n## This result confirms that the time units are in minutes. The largest peak is at around 1440, which is the number of minutes in a day. There is another peak at around 10000, which is near the number of minutes in a week. Some other peaks can be seen too, at 5000, 1650, etc. Not sure what the significance of those are.\n\n## The most interesting result is that each place kind of lives in its own little timezone. You'd think that the people living in this world would all behave according to a particular cycle, but that doesn't pan out in the global timestamp data."
 }
],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"}}, "nbformat": 4, "nbformat_minor": 0}