{
  "id": 303286,
  "title": "Notebook timeout - how to process features engineering?",
  "url": "/competitions/g-research-crypto-forecasting/discussion/303286",
  "author_name": "",
  "post_date": "2022-01-26T20:19:06.568036300Z",
  "votes": null,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi all,</p>\n<p>I still haven't managed to submit my predictions as I run into Notebook timeout.<br>\nAs far as I understood this happens because my notebook takes more than 0.25 seconds to make predictions: that is definitely true, it takes a few minutes, as it performs a features engineering step.<br>\nMy code looks something like this:</p>\n<pre><code>submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    submission = submission.append(sample_prediction_df, ignore_index=True)\nsubmission.to_csv(\"submission.csv\", index=False)\n</code></pre>\n<p>How do you guys perform features engineering in less than 0.25 seconds? I am personally looking at past values (rolling averages for example), therefore I need to look at the training set as well, which extends the features engineering duration.</p>",
  "messages": [
    {
      "id": "1665424",
      "postDate": "01/26/2022 20:19:06",
      "content": "<p>Hi all,</p>\n<p>I still haven't managed to submit my predictions as I run into Notebook timeout.<br>\nAs far as I understood this happens because my notebook takes more than 0.25 seconds to make predictions: that is definitely true, it takes a few minutes, as it performs a features engineering step.<br>\nMy code looks something like this:</p>\n<pre><code>submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    submission = submission.append(sample_prediction_df, ignore_index=True)\nsubmission.to_csv(\"submission.csv\", index=False)\n</code></pre>\n<p>How do you guys perform features engineering in less than 0.25 seconds? I am personally looking at past values (rolling averages for example), therefore I need to look at the training set as well, which extends the features engineering duration.</p>",
      "rawMarkdown": "Hi all,\n\nI still haven't managed to submit my predictions as I run into Notebook timeout.\nAs far as I understood this happens because my notebook takes more than 0.25 seconds to make predictions: that is definitely true, it takes a few minutes, as it performs a features engineering step.\nMy code looks something like this:\n\n```\nsubmission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    submission = submission.append(sample_prediction_df, ignore_index=True)\nsubmission.to_csv(\"submission.csv\", index=False)\n```\n\nHow do you guys perform features engineering in less than 0.25 seconds? I am personally looking at past values (rolling averages for example), therefore I need to look at the training set as well, which extends the features engineering duration.",
      "votes": null
    },
    {
      "id": "1665460",
      "postDate": "01/26/2022 21:26:01",
      "content": "<pre><code>#submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    #submission = submission.append(sample_prediction_df, ignore_index=True)\n#submission.to_csv(\"submission.csv\", index=False)\n</code></pre>\n<p>The commented out codes should be removed.</p>",
      "rawMarkdown": "```\n#submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    #submission = submission.append(sample_prediction_df, ignore_index=True)\n#submission.to_csv(\"submission.csv\", index=False)\n```\nThe commented out codes should be removed.",
      "votes": null
    },
    {
      "id": "1665466",
      "postDate": "01/26/2022 21:34:06",
      "content": "<p>I thought so, but I don’t manage to submit the prediction without a submission.csv file. How to work around that?</p>",
      "rawMarkdown": "I thought so, but I don’t manage to submit the prediction without a submission.csv file. How to work around that?",
      "votes": null
    },
    {
      "id": "1665693",
      "postDate": "01/27/2022 04:22:15",
      "content": "<blockquote>\n  <p>but I don’t manage to submit the prediction without a submission.csv file.</p>\n</blockquote>\n<p>Does it mean that you once tried removing the commented out codes and found that no submission.csv was created?</p>",
      "rawMarkdown": "> but I don’t manage to submit the prediction without a submission.csv file.\n\nDoes it mean that you once tried removing the commented out codes and found that no submission.csv was created?",
      "votes": null
    },
    {
      "id": "1665710",
      "postDate": "01/27/2022 04:49:02",
      "content": "<p>I believe most participants can not complete this process well …</p>",
      "rawMarkdown": "I believe most participants can not complete this process well ...",
      "votes": null
    },
    {
      "id": "1666051",
      "postDate": "01/27/2022 11:06:43",
      "content": "<p>Yes, that is why I included the submission dataframe myself. Is it supposed to show up? If yes, something is wrong in my procedure. I will double check, thank you.</p>",
      "rawMarkdown": "Yes, that is why I included the submission dataframe myself. Is it supposed to show up? If yes, something is wrong in my procedure. I will double check, thank you.",
      "votes": null
    },
    {
      "id": "1666257",
      "postDate": "01/27/2022 15:56:30",
      "content": "<p>Yes. Just follow the code structure, no extra to_csv() function is needed.</p>\n<pre><code>import gresearch_crypto\nenv = gresearch_crypto.make_env()   # initialize the environment\niter_test = env.iter_test()    # an iterator which loops over the test set and sample submission\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = 0  # make your predictions here\n    env.predict(sample_prediction_df)   # register your predictions\n</code></pre>",
      "rawMarkdown": "Yes. Just follow the code structure, no extra to_csv() function is needed.\n\n```\nimport gresearch_crypto\nenv = gresearch_crypto.make_env()   # initialize the environment\niter_test = env.iter_test()    # an iterator which loops over the test set and sample submission\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = 0  # make your predictions here\n    env.predict(sample_prediction_df)   # register your predictions\n```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1665460,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "01/26/2022 21:26:01",
      "content": "<pre><code>#submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    #submission = submission.append(sample_prediction_df, ignore_index=True)\n#submission.to_csv(\"submission.csv\", index=False)\n</code></pre>\n<p>The commented out codes should be removed.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1665466,
          "author_name": "andrearicciardi",
          "author_url": "",
          "post_date": "01/26/2022 21:34:06",
          "content": "<p>I thought so, but I don’t manage to submit the prediction without a submission.csv file. How to work around that?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1665693,
          "author_name": "yus002",
          "author_url": "",
          "post_date": "01/27/2022 04:22:15",
          "content": "<blockquote>\n  <p>but I don’t manage to submit the prediction without a submission.csv file.</p>\n</blockquote>\n<p>Does it mean that you once tried removing the commented out codes and found that no submission.csv was created?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1666051,
          "author_name": "andrearicciardi",
          "author_url": "",
          "post_date": "01/27/2022 11:06:43",
          "content": "<p>Yes, that is why I included the submission dataframe myself. Is it supposed to show up? If yes, something is wrong in my procedure. I will double check, thank you.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1666257,
          "author_name": "yus002",
          "author_url": "",
          "post_date": "01/27/2022 15:56:30",
          "content": "<p>Yes. Just follow the code structure, no extra to_csv() function is needed.</p>\n<pre><code>import gresearch_crypto\nenv = gresearch_crypto.make_env()   # initialize the environment\niter_test = env.iter_test()    # an iterator which loops over the test set and sample submission\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = 0  # make your predictions here\n    env.predict(sample_prediction_df)   # register your predictions\n</code></pre>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1665710,
      "author_name": "yuanzhezhou",
      "author_url": "",
      "post_date": "01/27/2022 04:49:02",
      "content": "<p>I believe most participants can not complete this process well …</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1665424": "Hi all,\n\nI still haven't managed to submit my predictions as I run into Notebook timeout.\nAs far as I understood this happens because my notebook takes more than 0.25 seconds to make predictions: that is definitely true, it takes a few minutes, as it performs a features engineering step.\nMy code looks something like this:\n\n```\nsubmission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    submission = submission.append(sample_prediction_df, ignore_index=True)\nsubmission.to_csv(\"submission.csv\", index=False)\n```\n\nHow do you guys perform features engineering in less than 0.25 seconds? I am personally looking at past values (rolling averages for example), therefore I need to look at the training set as well, which extends the features engineering duration.",
    "1665460": "```\n#submission = pd.DataFrame()\nfor (test_df, sample_prediction_df) in iter_test:\n    X_test = features_engineering(test_df)\n    pred = my_model.predict(X_test)\n    sample_prediction_df[\"Target\"] = pred\n    env.predict(sample_prediction_df)\n    #submission = submission.append(sample_prediction_df, ignore_index=True)\n#submission.to_csv(\"submission.csv\", index=False)\n```\nThe commented out codes should be removed.",
    "1665466": "I thought so, but I don’t manage to submit the prediction without a submission.csv file. How to work around that?",
    "1665693": "> but I don’t manage to submit the prediction without a submission.csv file.\n\nDoes it mean that you once tried removing the commented out codes and found that no submission.csv was created?",
    "1665710": "I believe most participants can not complete this process well ...",
    "1666051": "Yes, that is why I included the submission dataframe myself. Is it supposed to show up? If yes, something is wrong in my procedure. I will double check, thank you.",
    "1666257": "Yes. Just follow the code structure, no extra to_csv() function is needed.\n\n```\nimport gresearch_crypto\nenv = gresearch_crypto.make_env()   # initialize the environment\niter_test = env.iter_test()    # an iterator which loops over the test set and sample submission\nfor (test_df, sample_prediction_df) in iter_test:\n    sample_prediction_df['Target'] = 0  # make your predictions here\n    env.predict(sample_prediction_df)   # register your predictions\n```"
  },
  "source": "meta"
}