{
  "id": 113784,
  "title": "Some tips for convert 'timestamp' feature",
  "url": "/competitions/ashrae-energy-prediction/discussion/113784",
  "author_name": "Lyalikov Artyom",
  "post_date": "2019-10-22T08:09:05.320000",
  "votes": 44,
  "comment_count": 13,
  "views": 0,
  "content": "<p>Recently I came across an interesting article that describes how to encode a time variable:</p>\n\n<p><a href=\"https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/\">https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/</a></p>\n\n<p>After reading, I wrote a class for convenient transformation of time variables into cyclic ones. In this class you can set any time period for the conversion.</p>\n\n<p>```\nfrom sklearn import skbase\nimport numpy as np</p>\n\n<p>class DatetimeConvertCyclical(skbase.BaseEstimator, skbase.TransformerMixin):\n    def <strong>init</strong>(self):\n        self.time_periods = {'second': 24 * 60 * 60,\n                             'minute': 24 * 60,\n                             'hour': 24,\n                             'day': 30,\n                             'dayofweek': 7,\n                             'month': 12}</p>\n\n<pre><code>def fit(self, X, y=None):\n    return self\n\ndef transform(self, X):\n    for period, value in self.time_periods.items():\n        X[period] = getattr(X['timestamp'].dt, period)\n\n        X['sin_' + period] = np.sin(2 * np.pi * X[period] / value)\n        X['cos_' + period] = np.cos(2 * np.pi * X[period] / value)\n\n        X.drop(str(period), axis=1, inplace=True)\n\n    return X\n</code></pre>\n\n<p>```</p>\n\n<p>Happy kaggling!</p>",
  "messages": [
    {
      "id": 654722,
      "postDate": "2019-10-22T08:09:05.320Z",
      "content": "<p>Recently I came across an interesting article that describes how to encode a time variable:</p>\n\n<p><a href=\"https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/\">https://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/</a></p>\n\n<p>After reading, I wrote a class for convenient transformation of time variables into cyclic ones. In this class you can set any time period for the conversion.</p>\n\n<p>```\nfrom sklearn import skbase\nimport numpy as np</p>\n\n<p>class DatetimeConvertCyclical(skbase.BaseEstimator, skbase.TransformerMixin):\n    def <strong>init</strong>(self):\n        self.time_periods = {'second': 24 * 60 * 60,\n                             'minute': 24 * 60,\n                             'hour': 24,\n                             'day': 30,\n                             'dayofweek': 7,\n                             'month': 12}</p>\n\n<pre><code>def fit(self, X, y=None):\n    return self\n\ndef transform(self, X):\n    for period, value in self.time_periods.items():\n        X[period] = getattr(X['timestamp'].dt, period)\n\n        X['sin_' + period] = np.sin(2 * np.pi * X[period] / value)\n        X['cos_' + period] = np.cos(2 * np.pi * X[period] / value)\n\n        X.drop(str(period), axis=1, inplace=True)\n\n    return X\n</code></pre>\n\n<p>```</p>\n\n<p>Happy kaggling!</p>",
      "rawMarkdown": "Recently I came across an interesting article that describes how to encode a time variable:\n\nhttps://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/\n\nAfter reading, I wrote a class for convenient transformation of time variables into cyclic ones. In this class you can set any time period for the conversion.\n\n```\nfrom sklearn import skbase\nimport numpy as np\n\n\nclass DatetimeConvertCyclical(skbase.BaseEstimator, skbase.TransformerMixin):\n    def __init__(self):\n        self.time_periods = {'second': 24 * 60 * 60,\n                             'minute': 24 * 60,\n                             'hour': 24,\n                             'day': 30,\n                             'dayofweek': 7,\n                             'month': 12}\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        for period, value in self.time_periods.items():\n            X[period] = getattr(X['timestamp'].dt, period)\n\n            X['sin_' + period] = np.sin(2 * np.pi * X[period] / value)\n            X['cos_' + period] = np.cos(2 * np.pi * X[period] / value)\n\n            X.drop(str(period), axis=1, inplace=True)\n\n        return X\n```\n\nHappy kaggling!",
      "votes": 44
    },
    {
      "id": 655057,
      "postDate": "2019-10-22T16:40:17.680Z",
      "content": "<p>Other machine learning algorithms can be a lot of strong towards raw cyclic features, notably tree-based approaches. However, deep neural networks stand to learn from the sine and cos transformation of such features, notably in terms of aiding the convergence speed of the network.</p>",
      "rawMarkdown": "Other machine learning algorithms can be a lot of strong towards raw cyclic features, notably tree-based approaches. However, deep neural networks stand to learn from the sine and cos transformation of such features, notably in terms of aiding the convergence speed of the network.",
      "votes": 8
    },
    {
      "id": 654737,
      "postDate": "2019-10-22T08:38:11.840Z",
      "content": "<p>Thanks, but I think <code>Series.dt</code> properties are easier for cyclical datetime features</p>\n\n<p><a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties\">https://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties</a></p>",
      "rawMarkdown": "Thanks, but I think `Series.dt` properties are easier for cyclical datetime features\n\nhttps://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties",
      "votes": 2,
      "replies": [
        {
          "id": 654748,
          "postDate": "2019-10-22T08:55:00.653Z",
          "content": "<p>Maybe :)\nBut this approach allows you to implement features with a more cunning choice of the period, in contrast to just seconds, minutes, etc.</p>",
          "rawMarkdown": "Maybe :)\nBut this approach allows you to implement features with a more cunning choice of the period, in contrast to just seconds, minutes, etc.",
          "votes": 2
        }
      ]
    },
    {
      "id": 689194,
      "postDate": "2019-12-06T16:02:11.980Z",
      "content": "<p>That's ingenious! \nbut...</p>\n\n<p>X['sin_' + period] = np.sin(2 * np.pi * value/X[period] )    ?\nThanks for sharing.</p>",
      "rawMarkdown": "That's ingenious! \nbut...\n\nX['sin_' + period] = np.sin(2 * np.pi * value/X[period] )    ?\nThanks for sharing.",
      "votes": 1
    },
    {
      "id": 656376,
      "postDate": "2019-10-24T07:17:01.753Z",
      "content": "<p>From my understanding, this sin/cos transformation can be assumed as a part of fourie transformation, which has following advantages:</p>\n\n<ul>\n<li>can adjust trade-off between smoothness and number of variables (when apply one-hot encoding) by considering the number of sin/cos pairs</li>\n<li>can handle multimodal periodicities</li>\n</ul>\n\n<p>Please refer following for more detail:\n<a href=\"https://otexts.com/fpp2/useful-predictors.html\">https://otexts.com/fpp2/useful-predictors.html</a>\n<a href=\"https://otexts.com/fpp2/complexseasonality.html\">https://otexts.com/fpp2/complexseasonality.html</a></p>\n\n<p>I'm not sure this variable transformation is effective for tree-based ensemble modeling, but believe that effective for linear regression.</p>",
      "rawMarkdown": "From my understanding, this sin/cos transformation can be assumed as a part of fourie transformation, which has following advantages:\n\n* can adjust trade-off between smoothness and number of variables (when apply one-hot encoding) by considering the number of sin/cos pairs\n* can handle multimodal periodicities\n\nPlease refer following for more detail:\nhttps://otexts.com/fpp2/useful-predictors.html\nhttps://otexts.com/fpp2/complexseasonality.html\n\nI'm not sure this variable transformation is effective for tree-based ensemble modeling, but believe that effective for linear regression.\n",
      "votes": 1
    },
    {
      "id": 655974,
      "postDate": "2019-10-23T18:30:53.640Z",
      "content": "<p>It's <code>from sklearn import base</code> in kaggle kernel. Thanks for sharing.</p>",
      "rawMarkdown": "It's `from sklearn import base` in kaggle kernel. Thanks for sharing.",
      "votes": 1
    },
    {
      "id": 654814,
      "postDate": "2019-10-22T10:53:21.567Z",
      "content": "<p>Ah, great. Thanks! I already encountered this idea in a different context and totally forgot about the concept. 🙈 </p>\n\n<p>Have you already tried this out and if so, how did it affect your submissions?</p>",
      "rawMarkdown": "Ah, great. Thanks! I already encountered this idea in a different context and totally forgot about the concept. 🙈 \n\n\nHave you already tried this out and if so, how did it affect your submissions?",
      "votes": 1,
      "replies": [
        {
          "id": 654895,
          "postDate": "2019-10-22T12:55:08.417Z",
          "content": "<p>While I didn’t check, I think I would check with a bundle of features</p>",
          "rawMarkdown": "While I didn’t check, I think I would check with a bundle of features",
          "votes": 1
        }
      ]
    },
    {
      "id": 656226,
      "postDate": "2019-10-24T02:57:47.097Z",
      "content": "<p>Simplest way to get year, month, day, hour, minute and seconds\npd.to_datetime(data['timestamp']).dt.year\npd.to_datetime(data['timestamp']).dt.month\npd.to_datetime(data['timestamp']).dt.day\npd.to_datetime(data['timestamp']).dt.hour\npd.to_datetime(data['timestamp']).dt.minute\npd.to_datetime(data['timestamp']).dt.seconds</p>",
      "rawMarkdown": "Simplest way to get year, month, day, hour, minute and seconds\npd.to_datetime(data['timestamp']).dt.year\npd.to_datetime(data['timestamp']).dt.month\npd.to_datetime(data['timestamp']).dt.day\npd.to_datetime(data['timestamp']).dt.hour\npd.to_datetime(data['timestamp']).dt.minute\npd.to_datetime(data['timestamp']).dt.seconds",
      "votes": 2
    },
    {
      "id": 654948,
      "postDate": "2019-10-22T14:22:00.603Z",
      "content": "<p>Great. This can be very useful. Thanks for sharing!</p>",
      "rawMarkdown": "Great. This can be very useful. Thanks for sharing!",
      "votes": 2
    },
    {
      "id": 689393,
      "postDate": "2019-12-06T22:01:38.383Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 655713,
      "postDate": "2019-10-23T12:18:21.777Z",
      "content": "<p>wow very fast approach thanks <a href=\"/timon88\">@timon88</a> </p>",
      "rawMarkdown": "wow very fast approach thanks @timon88 ",
      "votes": 1
    },
    {
      "id": 689535,
      "postDate": "2019-12-07T02:35:25.460Z",
      "content": "<p>Helpful for me, thank you!</p>",
      "rawMarkdown": "Helpful for me, thank you!"
    }
  ],
  "comments": [
    {
      "id": 655057,
      "author_name": "itsshavar",
      "author_url": "",
      "post_date": "2019-10-22T16:40:17.680000",
      "content": "<p>Other machine learning algorithms can be a lot of strong towards raw cyclic features, notably tree-based approaches. However, deep neural networks stand to learn from the sine and cos transformation of such features, notably in terms of aiding the convergence speed of the network.</p>",
      "votes": 8,
      "replies": []
    },
    {
      "id": 654737,
      "author_name": "Gunes Evitan",
      "author_url": "",
      "post_date": "2019-10-22T08:38:11.840000",
      "content": "<p>Thanks, but I think <code>Series.dt</code> properties are easier for cyclical datetime features</p>\n\n<p><a href=\"https://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties\">https://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties</a></p>",
      "votes": 2,
      "replies": [
        {
          "id": 654748,
          "author_name": "Lyalikov Artyom",
          "author_url": "",
          "post_date": "2019-10-22T08:55:00.653000",
          "content": "<p>Maybe :)\nBut this approach allows you to implement features with a more cunning choice of the period, in contrast to just seconds, minutes, etc.</p>",
          "votes": 2,
          "replies": []
        }
      ]
    },
    {
      "id": 689194,
      "author_name": "seabiscuit2019",
      "author_url": "",
      "post_date": "2019-12-06T16:02:11.980000",
      "content": "<p>That's ingenious! \nbut...</p>\n\n<p>X['sin_' + period] = np.sin(2 * np.pi * value/X[period] )    ?\nThanks for sharing.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 656376,
      "author_name": "statefb",
      "author_url": "",
      "post_date": "2019-10-24T07:17:01.753000",
      "content": "<p>From my understanding, this sin/cos transformation can be assumed as a part of fourie transformation, which has following advantages:</p>\n\n<ul>\n<li>can adjust trade-off between smoothness and number of variables (when apply one-hot encoding) by considering the number of sin/cos pairs</li>\n<li>can handle multimodal periodicities</li>\n</ul>\n\n<p>Please refer following for more detail:\n<a href=\"https://otexts.com/fpp2/useful-predictors.html\">https://otexts.com/fpp2/useful-predictors.html</a>\n<a href=\"https://otexts.com/fpp2/complexseasonality.html\">https://otexts.com/fpp2/complexseasonality.html</a></p>\n\n<p>I'm not sure this variable transformation is effective for tree-based ensemble modeling, but believe that effective for linear regression.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 655974,
      "author_name": "Tim Yee",
      "author_url": "",
      "post_date": "2019-10-23T18:30:53.640000",
      "content": "<p>It's <code>from sklearn import base</code> in kaggle kernel. Thanks for sharing.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 654814,
      "author_name": "chmaxx",
      "author_url": "",
      "post_date": "2019-10-22T10:53:21.567000",
      "content": "<p>Ah, great. Thanks! I already encountered this idea in a different context and totally forgot about the concept. 🙈 </p>\n\n<p>Have you already tried this out and if so, how did it affect your submissions?</p>",
      "votes": 1,
      "replies": [
        {
          "id": 654895,
          "author_name": "Lyalikov Artyom",
          "author_url": "",
          "post_date": "2019-10-22T12:55:08.417000",
          "content": "<p>While I didn’t check, I think I would check with a bundle of features</p>",
          "votes": 1,
          "replies": []
        }
      ]
    },
    {
      "id": 656226,
      "author_name": "Prashant Kolkur",
      "author_url": "",
      "post_date": "2019-10-24T02:57:47.097000",
      "content": "<p>Simplest way to get year, month, day, hour, minute and seconds\npd.to_datetime(data['timestamp']).dt.year\npd.to_datetime(data['timestamp']).dt.month\npd.to_datetime(data['timestamp']).dt.day\npd.to_datetime(data['timestamp']).dt.hour\npd.to_datetime(data['timestamp']).dt.minute\npd.to_datetime(data['timestamp']).dt.seconds</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 654948,
      "author_name": "Adrian Zinovei",
      "author_url": "",
      "post_date": "2019-10-22T14:22:00.603000",
      "content": "<p>Great. This can be very useful. Thanks for sharing!</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 689393,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-12-06T22:01:38.383000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 655713,
      "author_name": "",
      "author_url": "",
      "post_date": "2019-10-23T12:18:21.777000",
      "content": "<p>wow very fast approach thanks <a href=\"/timon88\">@timon88</a> </p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 689535,
      "author_name": "lonelycat",
      "author_url": "",
      "post_date": "2019-12-07T02:35:25.460000",
      "content": "<p>Helpful for me, thank you!</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "654722": "Recently I came across an interesting article that describes how to encode a time variable:\n\nhttps://ianlondon.github.io/blog/encoding-cyclical-features-24hour-time/\n\nAfter reading, I wrote a class for convenient transformation of time variables into cyclic ones. In this class you can set any time period for the conversion.\n\n```\nfrom sklearn import skbase\nimport numpy as np\n\n\nclass DatetimeConvertCyclical(skbase.BaseEstimator, skbase.TransformerMixin):\n    def __init__(self):\n        self.time_periods = {'second': 24 * 60 * 60,\n                             'minute': 24 * 60,\n                             'hour': 24,\n                             'day': 30,\n                             'dayofweek': 7,\n                             'month': 12}\n\n    def fit(self, X, y=None):\n        return self\n\n    def transform(self, X):\n        for period, value in self.time_periods.items():\n            X[period] = getattr(X['timestamp'].dt, period)\n\n            X['sin_' + period] = np.sin(2 * np.pi * X[period] / value)\n            X['cos_' + period] = np.cos(2 * np.pi * X[period] / value)\n\n            X.drop(str(period), axis=1, inplace=True)\n\n        return X\n```\n\nHappy kaggling!",
    "655057": "Other machine learning algorithms can be a lot of strong towards raw cyclic features, notably tree-based approaches. However, deep neural networks stand to learn from the sine and cos transformation of such features, notably in terms of aiding the convergence speed of the network.",
    "654737": "Thanks, but I think `Series.dt` properties are easier for cyclical datetime features\n\nhttps://pandas.pydata.org/pandas-docs/stable/reference/series.html#datetime-properties",
    "689194": "That's ingenious! \nbut...\n\nX['sin_' + period] = np.sin(2 * np.pi * value/X[period] )    ?\nThanks for sharing.",
    "656376": "From my understanding, this sin/cos transformation can be assumed as a part of fourie transformation, which has following advantages:\n\n* can adjust trade-off between smoothness and number of variables (when apply one-hot encoding) by considering the number of sin/cos pairs\n* can handle multimodal periodicities\n\nPlease refer following for more detail:\nhttps://otexts.com/fpp2/useful-predictors.html\nhttps://otexts.com/fpp2/complexseasonality.html\n\nI'm not sure this variable transformation is effective for tree-based ensemble modeling, but believe that effective for linear regression.\n",
    "655974": "It's `from sklearn import base` in kaggle kernel. Thanks for sharing.",
    "654814": "Ah, great. Thanks! I already encountered this idea in a different context and totally forgot about the concept. 🙈 \n\n\nHave you already tried this out and if so, how did it affect your submissions?",
    "656226": "Simplest way to get year, month, day, hour, minute and seconds\npd.to_datetime(data['timestamp']).dt.year\npd.to_datetime(data['timestamp']).dt.month\npd.to_datetime(data['timestamp']).dt.day\npd.to_datetime(data['timestamp']).dt.hour\npd.to_datetime(data['timestamp']).dt.minute\npd.to_datetime(data['timestamp']).dt.seconds",
    "654948": "Great. This can be very useful. Thanks for sharing!",
    "689393": "",
    "655713": "wow very fast approach thanks @timon88 ",
    "689535": "Helpful for me, thank you!"
  }
}