{
  "id": 67982,
  "title": "How to handle -inf and NaN values in predictions",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/67982",
  "author_name": "",
  "post_date": "2018-10-08T04:07:06.373635300Z",
  "votes": null,
  "comment_count": 2,
  "views": 0,
  "content": "<p>I searched in discussion but couldn't find the answer to my question.</p>\n\n<p>Here's what I did:</p>\n\n<pre><code>result = df_test.groupby('fullVisitorId', as_index=False)['PredictedLogRevenue'].sum()\nresult.PredictedLogRevenue = result.PredictedLogRevenue.apply(np.log1p)\nresult.PredictedLogRevenue = result.PredictedLogRevenue.replace([np.inf, -np.inf], np.nan)\nresult.PredictedLogRevenue = result.PredictedLogRevenue.fillna(0)\n</code></pre>\n\n<p>In short, after adding the transactionRevenues predictions to my test dataset, I grouped and summed rows per fullVisitorId, applied log+1, then replaced all occurences of -infinity and NaN with 0.</p>\n\n<p>The resulting submission performed poorly with a score of 3.82 on the leaderboard.\nMight I have been incorrect in coverting -inf and NaN to 0 ? Are we even supposed to get this kind of values ?</p>\n\n<p>Thank you for your support</p>",
  "messages": [
    {
      "id": "400297",
      "postDate": "10/08/2018 04:07:06",
      "content": "<p>I searched in discussion but couldn't find the answer to my question.</p>\n\n<p>Here's what I did:</p>\n\n<pre><code>result = df_test.groupby('fullVisitorId', as_index=False)['PredictedLogRevenue'].sum()\nresult.PredictedLogRevenue = result.PredictedLogRevenue.apply(np.log1p)\nresult.PredictedLogRevenue = result.PredictedLogRevenue.replace([np.inf, -np.inf], np.nan)\nresult.PredictedLogRevenue = result.PredictedLogRevenue.fillna(0)\n</code></pre>\n\n<p>In short, after adding the transactionRevenues predictions to my test dataset, I grouped and summed rows per fullVisitorId, applied log+1, then replaced all occurences of -infinity and NaN with 0.</p>\n\n<p>The resulting submission performed poorly with a score of 3.82 on the leaderboard.\nMight I have been incorrect in coverting -inf and NaN to 0 ? Are we even supposed to get this kind of values ?</p>\n\n<p>Thank you for your support</p>",
      "rawMarkdown": "I searched in discussion but couldn't find the answer to my question.\n\nHere's what I did:\n\n    result = df_test.groupby('fullVisitorId', as_index=False)['PredictedLogRevenue'].sum()\n    result.PredictedLogRevenue = result.PredictedLogRevenue.apply(np.log1p)\n    result.PredictedLogRevenue = result.PredictedLogRevenue.replace([np.inf, -np.inf], np.nan)\n    result.PredictedLogRevenue = result.PredictedLogRevenue.fillna(0)\n\nIn short, after adding the transactionRevenues predictions to my test dataset, I grouped and summed rows per fullVisitorId, applied log+1, then replaced all occurences of -infinity and NaN with 0.\n\nThe resulting submission performed poorly with a score of 3.82 on the leaderboard.\nMight I have been incorrect in coverting -inf and NaN to 0 ? Are we even supposed to get this kind of values ?\n\nThank you for your support",
      "votes": null
    },
    {
      "id": "400893",
      "postDate": "10/09/2018 05:04:41",
      "content": "<p>As the revenues cannot be negative log1p should be at least 0. -Inf is an erratic outcome.</p>\n\n<p>And the reason for NaN outcome might be because your test set have some extra variables than train sets, so predict functions not able to find relativities for such extra variables.</p>",
      "rawMarkdown": "As the revenues cannot be negative log1p should be at least 0. -Inf is an erratic outcome.\n\nAnd the reason for NaN outcome might be because your test set have some extra variables than train sets, so predict functions not able to find relativities for such extra variables.",
      "votes": null
    },
    {
      "id": "400998",
      "postDate": "10/09/2018 09:00:55",
      "content": "<p>@Hemant Rupani thank you for your reply ! I realized that the data type for my column was incorrect and it got mistakenly categorized during my pre-processing. Forcing a dtype of int32 fixed the issue :)</p>",
      "rawMarkdown": "Hemant Rupani thank you for your reply ! I realized that the data type for my column was incorrect and it got mistakenly categorized during my pre-processing. Forcing a dtype of int32 fixed the issue :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 400893,
      "author_name": "hemantrupani",
      "author_url": "",
      "post_date": "10/09/2018 05:04:41",
      "content": "<p>As the revenues cannot be negative log1p should be at least 0. -Inf is an erratic outcome.</p>\n\n<p>And the reason for NaN outcome might be because your test set have some extra variables than train sets, so predict functions not able to find relativities for such extra variables.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 400998,
      "author_name": "fandekasp",
      "author_url": "",
      "post_date": "10/09/2018 09:00:55",
      "content": "<p>@Hemant Rupani thank you for your reply ! I realized that the data type for my column was incorrect and it got mistakenly categorized during my pre-processing. Forcing a dtype of int32 fixed the issue :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "400297": "I searched in discussion but couldn't find the answer to my question.\n\nHere's what I did:\n\n    result = df_test.groupby('fullVisitorId', as_index=False)['PredictedLogRevenue'].sum()\n    result.PredictedLogRevenue = result.PredictedLogRevenue.apply(np.log1p)\n    result.PredictedLogRevenue = result.PredictedLogRevenue.replace([np.inf, -np.inf], np.nan)\n    result.PredictedLogRevenue = result.PredictedLogRevenue.fillna(0)\n\nIn short, after adding the transactionRevenues predictions to my test dataset, I grouped and summed rows per fullVisitorId, applied log+1, then replaced all occurences of -infinity and NaN with 0.\n\nThe resulting submission performed poorly with a score of 3.82 on the leaderboard.\nMight I have been incorrect in coverting -inf and NaN to 0 ? Are we even supposed to get this kind of values ?\n\nThank you for your support",
    "400893": "As the revenues cannot be negative log1p should be at least 0. -Inf is an erratic outcome.\n\nAnd the reason for NaN outcome might be because your test set have some extra variables than train sets, so predict functions not able to find relativities for such extra variables.",
    "400998": "Hemant Rupani thank you for your reply ! I realized that the data type for my column was incorrect and it got mistakenly categorized during my pre-processing. Forcing a dtype of int32 fixed the issue :)"
  },
  "source": "meta"
}