{
  "id": 308489,
  "title": "The best thing is that we have time ahead",
  "url": "/competitions/g-research-crypto-forecasting/discussion/308489",
  "author_name": "",
  "post_date": "2022-02-18T22:34:08.500410300Z",
  "votes": 12,
  "comment_count": 4,
  "views": 0,
  "content": "<p>Hi <a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a>,</p>\n<p>I am embarrassed also!</p>\n<p>It's entirely possible to get a negative score, of course, but when it happens to more than a few kagglers it might be worth digging a little deeper…</p>\n<p>The best thing is that we have time ahead (with subsequent re-runs) to find any problems and correct if necessary after finding any errors.</p>\n<p>Something that plays in favor of everything being ok is that probably most of us did not evaluate your solution for over 2 weeks (I did not) and this does not make sense with the fact that now we are surprised to see negative scores in this first re-run.</p>\n<p>None of us can know if the problem is with the data February 1, 2022 - Final submission deadline, since we have not seen it.</p>\n<p>From my 2 submissions I can't know if the problem is with the added data &gt; 2021-01-24 00:00:00 since both solutions use this data and I can't check any difference between using it and not using it, BUT sure there will be more than one competitor that will be able to do it and this could shed some light.</p>\n<p>If everything stays the same, I wouldn't understand my code corrupting the solution at this final stage of inference if it always worked ok.</p>\n<p>I didn't use target lags but those who did should expect better results on near time predictions so if the score has already blown them up in these first 2 weeks, the final score may be anything but great.</p>\n<p>Although there are many nuances behind the solutions and validation scores of each one of us who are competing here, my experience in this competition can be summarized as follows,</p>\n<p>-I worked with 6 different folds from a period of 3 months ahead.<br>\n-I worked with 1 different model for each asset, that is, each pipeline would have 14 different models.<br>\n-I worked with 5 different pipelines.<br>\n-I evaluated a total of 420 out of fold scores and ONLY more than 30 times I found a negative result for some asset and in a fairly scattered occurrence. I mean that they were not focused on a fold, on a pipeline or on a specific asset.<br>\n-The worst of the pipelines scores &gt; 0.03.</p>\n<p>Really, I have such a hard time understanding the failure of my two negative scores -0.0008 (train and inference 14x2 models) and -0.0036 (inference 14x3 models)!</p>\n<p>We'll see,</p>",
  "messages": [
    {
      "id": "1696495",
      "postDate": "02/18/2022 22:34:08",
      "content": "<p>Hi <a href=\"https://www.kaggle.com/sohier\" target=\"_blank\">@sohier</a>,</p>\n<p>I am embarrassed also!</p>\n<p>It's entirely possible to get a negative score, of course, but when it happens to more than a few kagglers it might be worth digging a little deeper…</p>\n<p>The best thing is that we have time ahead (with subsequent re-runs) to find any problems and correct if necessary after finding any errors.</p>\n<p>Something that plays in favor of everything being ok is that probably most of us did not evaluate your solution for over 2 weeks (I did not) and this does not make sense with the fact that now we are surprised to see negative scores in this first re-run.</p>\n<p>None of us can know if the problem is with the data February 1, 2022 - Final submission deadline, since we have not seen it.</p>\n<p>From my 2 submissions I can't know if the problem is with the added data &gt; 2021-01-24 00:00:00 since both solutions use this data and I can't check any difference between using it and not using it, BUT sure there will be more than one competitor that will be able to do it and this could shed some light.</p>\n<p>If everything stays the same, I wouldn't understand my code corrupting the solution at this final stage of inference if it always worked ok.</p>\n<p>I didn't use target lags but those who did should expect better results on near time predictions so if the score has already blown them up in these first 2 weeks, the final score may be anything but great.</p>\n<p>Although there are many nuances behind the solutions and validation scores of each one of us who are competing here, my experience in this competition can be summarized as follows,</p>\n<p>-I worked with 6 different folds from a period of 3 months ahead.<br>\n-I worked with 1 different model for each asset, that is, each pipeline would have 14 different models.<br>\n-I worked with 5 different pipelines.<br>\n-I evaluated a total of 420 out of fold scores and ONLY more than 30 times I found a negative result for some asset and in a fairly scattered occurrence. I mean that they were not focused on a fold, on a pipeline or on a specific asset.<br>\n-The worst of the pipelines scores &gt; 0.03.</p>\n<p>Really, I have such a hard time understanding the failure of my two negative scores -0.0008 (train and inference 14x2 models) and -0.0036 (inference 14x3 models)!</p>\n<p>We'll see,</p>",
      "rawMarkdown": "Hi @sohier,\n\nI am embarrassed also!\n\nIt's entirely possible to get a negative score, of course, but when it happens to more than a few kagglers it might be worth digging a little deeper...\n\nThe best thing is that we have time ahead (with subsequent re-runs) to find any problems and correct if necessary after finding any errors.\n\nSomething that plays in favor of everything being ok is that probably most of us did not evaluate your solution for over 2 weeks (I did not) and this does not make sense with the fact that now we are surprised to see negative scores in this first re-run.\n\nNone of us can know if the problem is with the data February 1, 2022 - Final submission deadline, since we have not seen it.\n\nFrom my 2 submissions I can't know if the problem is with the added data > 2021-01-24 00:00:00 since both solutions use this data and I can't check any difference between using it and not using it, BUT sure there will be more than one competitor that will be able to do it and this could shed some light.\n\nIf everything stays the same, I wouldn't understand my code corrupting the solution at this final stage of inference if it always worked ok.\n\nI didn't use target lags but those who did should expect better results on near time predictions so if the score has already blown them up in these first 2 weeks, the final score may be anything but great.\n\nAlthough there are many nuances behind the solutions and validation scores of each one of us who are competing here, my experience in this competition can be summarized as follows,\n\n-I worked with 6 different folds from a period of 3 months ahead.\n-I worked with 1 different model for each asset, that is, each pipeline would have 14 different models.\n-I worked with 5 different pipelines.\n-I evaluated a total of 420 out of fold scores and ONLY more than 30 times I found a negative result for some asset and in a fairly scattered occurrence. I mean that they were not focused on a fold, on a pipeline or on a specific asset.\n-The worst of the pipelines scores > 0.03.\n\nReally, I have such a hard time understanding the failure of my two negative scores -0.0008 (train and inference 14x2 models) and -0.0036 (inference 14x3 models)!\n\nWe'll see,",
      "votes": null
    },
    {
      "id": "1696512",
      "postDate": "02/18/2022 22:50:51",
      "content": "<p>The competition hosts are pretty much ignoring us, they don’t seem to care, we tagged them so many times on our problems are they never even reply to acknowledge any issues..</p>\n<p>They got their Kaggle listing fee so no longer give a toss it seems. 🤷🏻‍♂️</p>",
      "rawMarkdown": "The competition hosts are pretty much ignoring us, they don’t seem to care, we tagged them so many times on our problems are they never even reply to acknowledge any issues..\n\nThey got their Kaggle listing fee so no longer give a toss it seems. 🤷🏻‍♂️",
      "votes": null
    },
    {
      "id": "1696521",
      "postDate": "02/18/2022 22:52:35",
      "content": "<p>It's not like you say… you'll see how we'll have news!</p>",
      "rawMarkdown": "It's not like you say... you'll see how we'll have news!",
      "votes": null
    },
    {
      "id": "1696966",
      "postDate": "02/19/2022 09:13:59",
      "content": "<p><a href=\"https://www.kaggle.com/coreacasa\" target=\"_blank\">@coreacasa</a>, why are you so sure they are going to give us an answer?</p>",
      "rawMarkdown": "coreacasa, why are you so sure they are going to give us an answer?",
      "votes": null
    },
    {
      "id": "1697091",
      "postDate": "02/19/2022 11:15:30",
      "content": "<p><a href=\"https://www.kaggle.com/agenlu\" target=\"_blank\">@agenlu</a>, I am sure that host and kaggle-staff review all the threads. If there are no bugs maybe it doesn't make sense to give us news but what I have no doubt about is that they will work hard to get there and rectify if necessary.</p>",
      "rawMarkdown": "agenlu, I am sure that host and kaggle-staff review all the threads. If there are no bugs maybe it doesn't make sense to give us news but what I have no doubt about is that they will work hard to get there and rectify if necessary.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1696512,
      "author_name": "julianmukaj",
      "author_url": "",
      "post_date": "02/18/2022 22:50:51",
      "content": "<p>The competition hosts are pretty much ignoring us, they don’t seem to care, we tagged them so many times on our problems are they never even reply to acknowledge any issues..</p>\n<p>They got their Kaggle listing fee so no longer give a toss it seems. 🤷🏻‍♂️</p>",
      "votes": null,
      "replies": [
        {
          "id": 1696521,
          "author_name": "coreacasa",
          "author_url": "",
          "post_date": "02/18/2022 22:52:35",
          "content": "<p>It's not like you say… you'll see how we'll have news!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1696966,
          "author_name": "agenlu",
          "author_url": "",
          "post_date": "02/19/2022 09:13:59",
          "content": "<p><a href=\"https://www.kaggle.com/coreacasa\" target=\"_blank\">@coreacasa</a>, why are you so sure they are going to give us an answer?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1697091,
          "author_name": "coreacasa",
          "author_url": "",
          "post_date": "02/19/2022 11:15:30",
          "content": "<p><a href=\"https://www.kaggle.com/agenlu\" target=\"_blank\">@agenlu</a>, I am sure that host and kaggle-staff review all the threads. If there are no bugs maybe it doesn't make sense to give us news but what I have no doubt about is that they will work hard to get there and rectify if necessary.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1696495": "Hi @sohier,\n\nI am embarrassed also!\n\nIt's entirely possible to get a negative score, of course, but when it happens to more than a few kagglers it might be worth digging a little deeper...\n\nThe best thing is that we have time ahead (with subsequent re-runs) to find any problems and correct if necessary after finding any errors.\n\nSomething that plays in favor of everything being ok is that probably most of us did not evaluate your solution for over 2 weeks (I did not) and this does not make sense with the fact that now we are surprised to see negative scores in this first re-run.\n\nNone of us can know if the problem is with the data February 1, 2022 - Final submission deadline, since we have not seen it.\n\nFrom my 2 submissions I can't know if the problem is with the added data > 2021-01-24 00:00:00 since both solutions use this data and I can't check any difference between using it and not using it, BUT sure there will be more than one competitor that will be able to do it and this could shed some light.\n\nIf everything stays the same, I wouldn't understand my code corrupting the solution at this final stage of inference if it always worked ok.\n\nI didn't use target lags but those who did should expect better results on near time predictions so if the score has already blown them up in these first 2 weeks, the final score may be anything but great.\n\nAlthough there are many nuances behind the solutions and validation scores of each one of us who are competing here, my experience in this competition can be summarized as follows,\n\n-I worked with 6 different folds from a period of 3 months ahead.\n-I worked with 1 different model for each asset, that is, each pipeline would have 14 different models.\n-I worked with 5 different pipelines.\n-I evaluated a total of 420 out of fold scores and ONLY more than 30 times I found a negative result for some asset and in a fairly scattered occurrence. I mean that they were not focused on a fold, on a pipeline or on a specific asset.\n-The worst of the pipelines scores > 0.03.\n\nReally, I have such a hard time understanding the failure of my two negative scores -0.0008 (train and inference 14x2 models) and -0.0036 (inference 14x3 models)!\n\nWe'll see,",
    "1696512": "The competition hosts are pretty much ignoring us, they don’t seem to care, we tagged them so many times on our problems are they never even reply to acknowledge any issues..\n\nThey got their Kaggle listing fee so no longer give a toss it seems. 🤷🏻‍♂️",
    "1696521": "It's not like you say... you'll see how we'll have news!",
    "1696966": "coreacasa, why are you so sure they are going to give us an answer?",
    "1697091": "agenlu, I am sure that host and kaggle-staff review all the threads. If there are no bugs maybe it doesn't make sense to give us news but what I have no doubt about is that they will work hard to get there and rectify if necessary."
  },
  "source": "meta"
}