{
  "id": 114422,
  "title": "Computation issue",
  "url": "/competitions/ashrae-energy-prediction/discussion/114422",
  "author_name": "",
  "post_date": "2019-10-26T05:07:32.333134800Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>It looks like converting timestamp to a meaningful value take a lot of computation resources. I am actually trying to run the model on Google cloud with a VM (16 vCPU, 16GB) parallelized, but it is taking too long to process the column and return. \nWhat do you suggest to reduce computation complexity?\n Thanks :)</p>",
  "messages": [
    {
      "id": "658515",
      "postDate": "10/26/2019 05:07:32",
      "content": "<p>It looks like converting timestamp to a meaningful value take a lot of computation resources. I am actually trying to run the model on Google cloud with a VM (16 vCPU, 16GB) parallelized, but it is taking too long to process the column and return. \nWhat do you suggest to reduce computation complexity?\n Thanks :)</p>",
      "rawMarkdown": "It looks like converting timestamp to a meaningful value take a lot of computation resources. I am actually trying to run the model on Google cloud with a VM (16 vCPU, 16GB) parallelized, but it is taking too long to process the column and return. \nWhat do you suggest to reduce computation complexity?\n Thanks :)",
      "votes": null
    },
    {
      "id": "658516",
      "postDate": "10/26/2019 05:11:21",
      "content": "<p>```\nfor df in [df_train, df_test]:</p>\n\n<pre><code># timestamp and timedelta (hours)\ndf['timestamp'] = pd.to_datetime(df['timestamp'], infer_datetime_format=True)\ndf['timedelta'] = ((pd.to_timedelta(df['timestamp'] - df_train['timestamp'].min()).dt.total_seconds().astype('int64')) / 3600)\n\n# Categorical date and  time features\ndf['HourOfDay'] = df['timestamp'].dt.hour.values.astype(np.uint8)\ndf['DayOfWeek'] = df['timestamp'].dt.dayofweek.values.astype(np.uint8)\ndf['DayOfMonth'] = df['timestamp'].dt.day.values.astype(np.uint8)  \ndf['DayOfYear'] = df['timestamp'].dt.dayofyear.values.astype(np.uint16)\ndf['WeekOfYear'] = df['timestamp'].dt.weekofyear.values.astype(np.uint8)\ndf['MonthOfYear'] = df['timestamp'].dt.month.values.astype(np.uint8)\n\n# Continuous date and time features\ndf['Hour'] = df['timedelta'].astype(np.uint16)\ndf['Day'] = (df['Hour'] / 24).astype(np.uint16)\ndf['Week'] = (df['Day'] / 7).astype(np.uint8)\n</code></pre>\n\n<p>```</p>\n\n<p>This takes 1 minute for me to do it on Kaggle kernels.</p>",
      "rawMarkdown": "```\nfor df in [df_train, df_test]:\n    \n    # timestamp and timedelta (hours)\n    df['timestamp'] = pd.to_datetime(df['timestamp'], infer_datetime_format=True)\n    df['timedelta'] = ((pd.to_timedelta(df['timestamp'] - df_train['timestamp'].min()).dt.total_seconds().astype('int64')) / 3600)\n    \n    # Categorical date and  time features\n    df['HourOfDay'] = df['timestamp'].dt.hour.values.astype(np.uint8)\n    df['DayOfWeek'] = df['timestamp'].dt.dayofweek.values.astype(np.uint8)\n    df['DayOfMonth'] = df['timestamp'].dt.day.values.astype(np.uint8)  \n    df['DayOfYear'] = df['timestamp'].dt.dayofyear.values.astype(np.uint16)\n    df['WeekOfYear'] = df['timestamp'].dt.weekofyear.values.astype(np.uint8)\n    df['MonthOfYear'] = df['timestamp'].dt.month.values.astype(np.uint8)\n    \n    # Continuous date and time features\n    df['Hour'] = df['timedelta'].astype(np.uint16)\n    df['Day'] = (df['Hour'] / 24).astype(np.uint16)\n    df['Week'] = (df['Day'] / 7).astype(np.uint8)\n```\n\nThis takes 1 minute for me to do it on Kaggle kernels.",
      "votes": null
    },
    {
      "id": "658529",
      "postDate": "10/26/2019 05:46:01",
      "content": "<p>That looks more optimized, but I do not see you using sin/cos for cyclic variables. How you will say to the algorithm that 23h00 and 00h00 are close </p>",
      "rawMarkdown": "That looks more optimized, but I do not see you using sin/cos for cyclic variables. How you will say to the algorithm that 23h00 and 00h00 are close",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 658516,
      "author_name": "gunesevitan",
      "author_url": "",
      "post_date": "10/26/2019 05:11:21",
      "content": "<p>```\nfor df in [df_train, df_test]:</p>\n\n<pre><code># timestamp and timedelta (hours)\ndf['timestamp'] = pd.to_datetime(df['timestamp'], infer_datetime_format=True)\ndf['timedelta'] = ((pd.to_timedelta(df['timestamp'] - df_train['timestamp'].min()).dt.total_seconds().astype('int64')) / 3600)\n\n# Categorical date and  time features\ndf['HourOfDay'] = df['timestamp'].dt.hour.values.astype(np.uint8)\ndf['DayOfWeek'] = df['timestamp'].dt.dayofweek.values.astype(np.uint8)\ndf['DayOfMonth'] = df['timestamp'].dt.day.values.astype(np.uint8)  \ndf['DayOfYear'] = df['timestamp'].dt.dayofyear.values.astype(np.uint16)\ndf['WeekOfYear'] = df['timestamp'].dt.weekofyear.values.astype(np.uint8)\ndf['MonthOfYear'] = df['timestamp'].dt.month.values.astype(np.uint8)\n\n# Continuous date and time features\ndf['Hour'] = df['timedelta'].astype(np.uint16)\ndf['Day'] = (df['Hour'] / 24).astype(np.uint16)\ndf['Week'] = (df['Day'] / 7).astype(np.uint8)\n</code></pre>\n\n<p>```</p>\n\n<p>This takes 1 minute for me to do it on Kaggle kernels.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 658529,
      "author_name": "mohameddjemai",
      "author_url": "",
      "post_date": "10/26/2019 05:46:01",
      "content": "<p>That looks more optimized, but I do not see you using sin/cos for cyclic variables. How you will say to the algorithm that 23h00 and 00h00 are close </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "658515": "It looks like converting timestamp to a meaningful value take a lot of computation resources. I am actually trying to run the model on Google cloud with a VM (16 vCPU, 16GB) parallelized, but it is taking too long to process the column and return. \nWhat do you suggest to reduce computation complexity?\n Thanks :)",
    "658516": "```\nfor df in [df_train, df_test]:\n    \n    # timestamp and timedelta (hours)\n    df['timestamp'] = pd.to_datetime(df['timestamp'], infer_datetime_format=True)\n    df['timedelta'] = ((pd.to_timedelta(df['timestamp'] - df_train['timestamp'].min()).dt.total_seconds().astype('int64')) / 3600)\n    \n    # Categorical date and  time features\n    df['HourOfDay'] = df['timestamp'].dt.hour.values.astype(np.uint8)\n    df['DayOfWeek'] = df['timestamp'].dt.dayofweek.values.astype(np.uint8)\n    df['DayOfMonth'] = df['timestamp'].dt.day.values.astype(np.uint8)  \n    df['DayOfYear'] = df['timestamp'].dt.dayofyear.values.astype(np.uint16)\n    df['WeekOfYear'] = df['timestamp'].dt.weekofyear.values.astype(np.uint8)\n    df['MonthOfYear'] = df['timestamp'].dt.month.values.astype(np.uint8)\n    \n    # Continuous date and time features\n    df['Hour'] = df['timedelta'].astype(np.uint16)\n    df['Day'] = (df['Hour'] / 24).astype(np.uint16)\n    df['Week'] = (df['Day'] / 7).astype(np.uint8)\n```\n\nThis takes 1 minute for me to do it on Kaggle kernels.",
    "658529": "That looks more optimized, but I do not see you using sin/cos for cyclic variables. How you will say to the algorithm that 23h00 and 00h00 are close"
  },
  "source": "meta"
}