{ "id": 285289, "title": "Meaningful submission scores / sharing the lower boundary of public test data", "url": "/competitions/g-research-crypto-forecasting/discussion/285289", "author_name": "", "post_date": "2021-11-04T04:12:04.844466800Z", "votes": 28, "comment_count": 7, "views": 0, "content": "

Hello,

\n

I hope what I'm saying makes any sense:

\n

I would like to remove the time range of the public LB from the train data in order to get some meaning from the submissions scores. I know that they won't reflect the private score, since the lapse will be larger, but I would like to use it anyway.

\n

The test.csv placeholder has the 4 iterations, going from '2021-06-13 00:00:00' to '2021-06-13 03:00:00'

\n

I am keeping the data before '2021-06-13 00:00:00', but I realized this boundary might be different in the actual public test.csv.

\n

So my question is: could it be possible to make public the lower boundary of the public LB test data, so we can remove all the data after it, and have meaningful submission scores?

\n

EDIT

\n

[S] 🪙💲 Proposal for a meaningful LB

\n

👆👆👆 A proposal for an agreement to get comparable, non-leaky models

\n

Long story short, the idea is to drop the data after 2021-06-13 for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag [Strict] or [S].

\n
def read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "messages": [ { "id": "1570329", "postDate": "11/04/2021 04:12:04", "content": "

Hello,

\n

I hope what I'm saying makes any sense:

\n

I would like to remove the time range of the public LB from the train data in order to get some meaning from the submissions scores. I know that they won't reflect the private score, since the lapse will be larger, but I would like to use it anyway.

\n

The test.csv placeholder has the 4 iterations, going from '2021-06-13 00:00:00' to '2021-06-13 03:00:00'

\n

I am keeping the data before '2021-06-13 00:00:00', but I realized this boundary might be different in the actual public test.csv.

\n

So my question is: could it be possible to make public the lower boundary of the public LB test data, so we can remove all the data after it, and have meaningful submission scores?

\n

EDIT

\n

[S] 🪙💲 Proposal for a meaningful LB

\n

👆👆👆 A proposal for an agreement to get comparable, non-leaky models

\n

Long story short, the idea is to drop the data after 2021-06-13 for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag [Strict] or [S].

\n
def read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n    df = pd.read_csv(file_name)\n    df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n    df = df[df['datetime'] < '2021-06-13 00:00:00']\n    return df\n
", "rawMarkdown": "Hello,\n\nI hope what I'm saying makes any sense:\n\nI would like to remove the time range of the public LB from the train data in order to get some meaning from the submissions scores. I _know_ that they won't reflect the private score, since the lapse will be larger, but I would like to use it anyway.\n\nThe `test.csv` placeholder has the 4 iterations, going from `'2021-06-13 00:00:00' ` to ` '2021-06-13 03:00:00'`\n\nI am keeping the data before `'2021-06-13 00:00:00'`, but I realized this boundary might be different in the actual public `test.csv`. \n\nSo my question is: could it be possible to make public the lower boundary of the public LB test data, so we can remove all the data after it, and have meaningful submission scores?\n\n\n\n__EDIT__\n\n# [[S] 🪙💲 Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n# 👆👆👆 A proposal for an agreement to get comparable, non-leaky models\n\nLong story short, the idea is to drop the data after `2021-06-13` for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag `[Strict]` or `[S]`.\n```\ndef read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "votes": null }, { "id": "1572469", "postDate": "11/05/2021 17:38:05", "content": "

It seems really silly by including the data in test.. Maybe because data is so limited, to have as much data to train on, there is no test data available.

\n

It feels demotivating to have LB that is not much use..

", "rawMarkdown": "It seems really silly by including the data in test.. Maybe because data is so limited, to have as much data to train on, there is no test data available. \n\nIt feels demotivating to have LB that is not much use..", "votes": null }, { "id": "1572692", "postDate": "11/05/2021 21:28:00", "content": "

Yes, it is demotivating, but have in mind what the organizers say, that is true: the target is publicly known information.

\n

Even removing it from the train set, there is no way the leaderboard is not corrupted (It could have been harder to get though).

\n

Keep it up, it's a fun competition!

\n

I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.

", "rawMarkdown": "Yes, it is demotivating, but have in mind what the organizers say, that is true: _the target is publicly known information_. \n\nEven removing it from the train set, there is no way the leaderboard is not corrupted (It could have been harder to get though).\n\nKeep it up, it's a fun competition!\n\nI'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.", "votes": null }, { "id": "1572701", "postDate": "11/05/2021 21:51:22", "content": "

yes agreed

\n

\"I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.\" great look forward to it :)

", "rawMarkdown": "yes agreed\n\n\"I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.\" great look forward to it :)", "votes": null }, { "id": "1572715", "postDate": "11/05/2021 22:03:05", "content": "

It's out there already, I was working on it actually. Take a look! I hope it helps get public work fun as it generally is:

\n

[S] 🪙💲 Proposal for submission common ground

", "rawMarkdown": "It's out there already, I was working on it actually. Take a look! I hope it helps get public work fun as it generally is:\n\n [[S] 🪙💲 Proposal for submission common ground](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)", "votes": null }, { "id": "1573037", "postDate": "11/06/2021 08:03:45", "content": "

great - read, liked up, and replied ;)

", "rawMarkdown": "great - read, liked up, and replied ;)", "votes": null }, { "id": "1573884", "postDate": "11/07/2021 02:43:25", "content": "

Pretty great!

", "rawMarkdown": "Pretty great!", "votes": null }, { "id": "1673810", "postDate": "02/03/2022 02:14:19", "content": "

Great notebook. Still reading through it!

", "rawMarkdown": "Great notebook. Still reading through it!", "votes": null } ], "comments": [ { "id": 1572469, "author_name": "regipublic", "author_url": "", "post_date": "11/05/2021 17:38:05", "content": "

It seems really silly by including the data in test.. Maybe because data is so limited, to have as much data to train on, there is no test data available.

\n

It feels demotivating to have LB that is not much use..

", "votes": null, "replies": [ { "id": 1572692, "author_name": "julian3833", "author_url": "", "post_date": "11/05/2021 21:28:00", "content": "

Yes, it is demotivating, but have in mind what the organizers say, that is true: the target is publicly known information.

\n

Even removing it from the train set, there is no way the leaderboard is not corrupted (It could have been harder to get though).

\n

Keep it up, it's a fun competition!

\n

I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.

", "votes": null, "replies": [] }, { "id": 1572701, "author_name": "regipublic", "author_url": "", "post_date": "11/05/2021 21:51:22", "content": "

yes agreed

\n

\"I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.\" great look forward to it :)

", "votes": null, "replies": [] }, { "id": 1572715, "author_name": "julian3833", "author_url": "", "post_date": "11/05/2021 22:03:05", "content": "

It's out there already, I was working on it actually. Take a look! I hope it helps get public work fun as it generally is:

\n

[S] 🪙💲 Proposal for submission common ground

", "votes": null, "replies": [] }, { "id": 1573037, "author_name": "regipublic", "author_url": "", "post_date": "11/06/2021 08:03:45", "content": "

great - read, liked up, and replied ;)

", "votes": null, "replies": [] }, { "id": 1673810, "author_name": "crained", "author_url": "", "post_date": "02/03/2022 02:14:19", "content": "

Great notebook. Still reading through it!

", "votes": null, "replies": [] } ] }, { "id": 1573884, "author_name": "guojiangzhao", "author_url": "", "post_date": "11/07/2021 02:43:25", "content": "

Pretty great!

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "1570329": "Hello,\n\nI hope what I'm saying makes any sense:\n\nI would like to remove the time range of the public LB from the train data in order to get some meaning from the submissions scores. I _know_ that they won't reflect the private score, since the lapse will be larger, but I would like to use it anyway.\n\nThe `test.csv` placeholder has the 4 iterations, going from `'2021-06-13 00:00:00' ` to ` '2021-06-13 03:00:00'`\n\nI am keeping the data before `'2021-06-13 00:00:00'`, but I realized this boundary might be different in the actual public `test.csv`. \n\nSo my question is: could it be possible to make public the lower boundary of the public LB test data, so we can remove all the data after it, and have meaningful submission scores?\n\n\n\n__EDIT__\n\n# [[S] 🪙💲 Proposal for a meaningful LB](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)\n# 👆👆👆 A proposal for an agreement to get comparable, non-leaky models\n\nLong story short, the idea is to drop the data after `2021-06-13` for submission to the public LB. We can call that \"Strict Public LB\" and flag public notebooks with the tag `[Strict]` or `[S]`.\n```\ndef read_csv_strict(file_name='/kaggle/input/g-research-crypto-forecasting/train.csv'):\n df = pd.read_csv(file_name)\n df['datetime'] = pd.to_datetime(df['timestamp'], unit='s')\n df = df[df['datetime'] < '2021-06-13 00:00:00']\n return df\n```", "1572469": "It seems really silly by including the data in test.. Maybe because data is so limited, to have as much data to train on, there is no test data available. \n\nIt feels demotivating to have LB that is not much use..", "1572692": "Yes, it is demotivating, but have in mind what the organizers say, that is true: _the target is publicly known information_. \n\nEven removing it from the train set, there is no way the leaderboard is not corrupted (It could have been harder to get though).\n\nKeep it up, it's a fun competition!\n\nI'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.", "1572701": "yes agreed\n\n\"I'm thinking of an idea to make public scores \"comparable\" and \"interesting\". I'll share it soon.\" great look forward to it :)", "1572715": "It's out there already, I was working on it actually. Take a look! I hope it helps get public work fun as it generally is:\n\n [[S] 🪙💲 Proposal for submission common ground](https://www.kaggle.com/julian3833/s-proposal-for-submission-common-ground)", "1573037": "great - read, liked up, and replied ;)", "1573884": "Pretty great!", "1673810": "Great notebook. Still reading through it!" }, "source": "meta" }