![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F238237%2F2c918668a222540f3c2b27f50bf932ad%2F2016-01____0_027601.jpg?generation=1572274013417060&alt=media\")
Seems like the private dataset is just the last 22% of the data time ordered (test set and row_id column are not strictly time ordered).
Sample submission score to 4.69 submitting only zeros. I just replaced values from August 2018 to December 2018 (20.96% of the test set) by exp(25) and got the same score (4.69).
\n\nFeel free to try yourself ;)
\n\n```python\nimport numpy as np\nimport pandas as pd
\n\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('submission/sample_submission.csv')
\n\nmask = test.query('timestamp >= \"2018-08-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)
\n\nsubmission.to_csv('submission/probing.csv', index=False)\n```
\n\nEdit: Thx to @berserker408, @robinvaysse and @adilztn submissions which demonstrated that the whole 2018 year is part of the private set and 31st December 2017 is part of the public. We can then safely (it's up to you) consider that public is 2017 and private is 2018 ✔️
", "messages": [ { "id": 654487, "postDate": "2019-10-21T23:10:56.750Z", "content": "Seems like the private dataset is just the last 22% of the data time ordered (test set and row_id column are not strictly time ordered).
Sample submission score to 4.69 submitting only zeros. I just replaced values from August 2018 to December 2018 (20.96% of the test set) by exp(25) and got the same score (4.69).
\n\nFeel free to try yourself ;)
\n\n```python\nimport numpy as np\nimport pandas as pd
\n\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('submission/sample_submission.csv')
\n\nmask = test.query('timestamp >= \"2018-08-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)
\n\nsubmission.to_csv('submission/probing.csv', index=False)\n```
\n\nEdit: Thx to @berserker408, @robinvaysse and @adilztn submissions which demonstrated that the whole 2018 year is part of the private set and 31st December 2017 is part of the public. We can then safely (it's up to you) consider that public is 2017 and private is 2018 ✔️
", "rawMarkdown": "Seems like the private dataset is just the last 22% of the data **time ordered** (test set and `row_id` column are not strictly time ordered). \n\nSample submission score to 4.69 submitting only zeros. I just replaced values from August 2018 to December 2018 (20.96% of the test set) by exp(25) and got the same score (4.69).\n\nFeel free to try yourself ;)\n\n```python\nimport numpy as np\nimport pandas as pd\n\n# load data\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('submission/sample_submission.csv')\n\n# replace last 5 months predictions to exp(25) \nmask = test.query('timestamp >= \"2018-08-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)\n\n# save submission\nsubmission.to_csv('submission/probing.csv', index=False)\n```\n\nEdit: Thx to @berserker408, @robinvaysse and @adilztn submissions which demonstrated that the whole 2018 year is part of the private set and 31st December 2017 is part of the public. We can then safely (it's up to you) consider that public is 2017 and private is 2018 ✔️ ", "votes": 98 }, { "id": 659813, "postDate": "2019-10-28T09:44:02.243Z", "content": "As describe in data description: \n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n\nSo there might be more than the last 22% of data which are in private set.
\n\nI tried to set exp(25) for all values from 2018 and it also score 4.69. To check if it was a problem due to exp(25) being too big, I tried to set this value for all timestamp of the test set and it scored 20.6 (so there is no issue with exp(25)).
\n\nFor now WE NEED YOU to test the real dates of the private leaderboard (we have just 2 submission a day per person). To do so, here is some code to create a submission with 0 for all timestamps exept for dates greater than 2018-01-01.
\n\n```\nimport numpy as np\nimport pandas as pd
\n\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('data/sample_submission.csv')
\n\nmask = test.query('timestamp >= \"2018-01-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)
\n\nsubmission.to_csv('submission/probing_all.csv.zip', index=False, compression='zip')\n```
\n\nPlease try some random dates like 2017-12-01 and share your findings in order to find the truth 🙏
", "rawMarkdown": "As describe in data description: \n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\n\nSo there might be more than the last 22% of data which are in private set.\n\nI tried to set exp(25) for all values from 2018 and it also score 4.69. To check if it was a problem due to exp(25) being too big, I tried to set this value for **all** timestamp of the test set and it scored 20.6 (so there is no issue with exp(25)).\n\nFor now **WE NEED YOU** to test the real dates of the private leaderboard (we have just 2 submission a day per person). To do so, here is some code to create a submission with 0 for all timestamps exept for dates greater than 2018-01-01.\n\n```\nimport numpy as np\nimport pandas as pd\n\n\n# load data\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('data/sample_submission.csv')\n\n# change last year predictions to exp(25)\nmask = test.query('timestamp >= \"2018-01-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)\n\n# save submission\nsubmission.to_csv('submission/probing_all.csv.zip', index=False, compression='zip')\n```\n\nPlease try some random dates like 2017-12-01 and share your findings in order to find the truth 🙏 ", "votes": 3, "replies": [ { "id": 660064, "postDate": "2019-10-28T16:05:47.147Z", "content": "I added the 31st of December 2017 in addition to the whole 2018 year and replace exp(25) by exp(50). It scored 5.27.\nmask = test.query('timestamp >= \"2017-12-31\"').row_id.tolist()
Thx guys! Now we can be pretty confident saying that public and private sets are respectively 2017 and 2018. If the private part really represents 22% of test data as organizers said, it means it has 4-ish more gaps than the public...
", "rawMarkdown": "Thx guys! Now we can be pretty confident saying that public and private sets are respectively 2017 and 2018. If the private part really represents 22% of test data as organizers said, it means it has 4-ish more gaps than the public...", "votes": 2 }, { "id": 660194, "postDate": "2019-10-28T21:10:52.607Z", "content": "Great! Thanks for your investigations!\nFrom Adil's number, 5.27 with exp(50) and all 0 score of 4.69 we can write
\n\n\n1/N * sum_inDec31 (y_true - 50)**2 - y_true**2 = 5.27**2 - 4.69**2,\n
where y = log(1 + meter_reading).
\n\nApproximating all y_true with 4 and assuming N days of equal amount of data,
\n\n\nN ~ (46**2 - 4**2)/(5.27**2 - 4.69**2) = 363.52.\n
This is consistent with 365 days (if there are \"gaps,\" Dec 31 has the same ratio of gaps as the year 2017 average.) It's most natural to think that gaps are minor in year 2017, as they are in the training set. I rather doubt the official statement that the public is 78%, I think public:private is 50:50.
\n\n[I first forgot about the square in the metric, I erased my stupid comment and rewrote]
", "rawMarkdown": "Great! Thanks for your investigations!\nFrom Adil's number, 5.27 with exp(50) and all 0 score of 4.69 we can write\n\n```\n1/N * sum_inDec31 (y_true - 50)**2 - y_true**2 = 5.27**2 - 4.69**2,\n```\n\nwhere y = log(1 + meter_reading). \n\nApproximating all y_true with 4 and assuming N days of equal amount of data,\n\n```\nN ~ (46**2 - 4**2)/(5.27**2 - 4.69**2) = 363.52.\n```\n\nThis is consistent with 365 days (if there are \"gaps,\" Dec 31 has the same ratio of gaps as the year 2017 average.) It's most natural to think that gaps are minor in year 2017, as they are in the training set. I rather doubt the official statement that the public is 78%, I think public:private is 50:50.\n\n[I first forgot about the square in the metric, I erased my stupid comment and rewrote]", "votes": 3 } ] }, { "id": 659081, "postDate": "2019-10-27T03:46:53.050Z", "content": "Based on your code, I probed more data with the code below:
\n\nmask = test[(test.timestamp>'2018-07-20 00:00:00') & (test.timestamp<'2019-01-01 00:00:00')].row_id.tolist()
\n\nAnd still the same score (4.69).
\n\nps.
\n\n\n", "rawMarkdown": "Based on your code, I probed more data with the code below:\n\nmask = test[(test.timestamp>'2018-07-20 00:00:00') & (test.timestamp<'2019-01-01 00:00:00')].row_id.tolist()\n\nAnd still the same score (4.69).\n\nps.\n>'2018-07-20 00:00:00' means 22.5%.", "votes": 3, "replies": [ { "id": 659225, "postDate": "2019-10-27T09:51:54.350Z", "content": "'2018-07-20 00:00:00' means 22.5%.
\n
Probed a bit more before seeing your message adding last 14 days of July with the same result (4.69 for 22.88%):\n\nmask = test.query('timestamp >= \"2018-07-18\"').row_id.tolist()\n\nNo more submission for me today…
Probed from April and got the same result (4.69 for 37.67%):
\n\n\nmask = test.query('timestamp >= \"2018-04-01\"').row_id.tolist()\n
As organizers said in the data section:
\n\n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n\nPrivate part containing way more gaps than the public could explain such results, meaning also that test distribution (78% public, 22% private) has been computed excluding those gaps
", "rawMarkdown": "Probed from April and got the same result (4.69 for 37.67%):\n\n```\nmask = test.query('timestamp >= \"2018-04-01\"').row_id.tolist()\n```\n\nAs organizers said in the data section: \n\n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\n\nPrivate part containing way more gaps than the public could explain such results, meaning also that test distribution (78% public, 22% private) has been computed excluding those gaps", "votes": 4 }, { "id": 659995, "postDate": "2019-10-28T14:48:51.240Z", "content": "Following are the fraction of rows belonging to a particular month-year (across train and test). It's very clear that the number of rows across different months in 2016 are varying w.r.t each other, but for each month across 2017 & 2018, the number of rows are matching exactly (up to sixth decimal). For months with 31 days, the fraction value is 0.028600 for 2017/2018. For months with 30 days, the fraction value is 0.027677. These two numbers are not matching with any of the months from 2016.
\n\nThis probably indicates the gaps have been systematically filled across the entire test data (2017/2018), but that is NOT the case for train data (2016).
\n\n
Oh, awesome! Thank you for sharing this! It's nice to confirm own thoughts like this :) Though it is quite logical that they have decided to reserve recent data for the private :P Best of luck in the competition
", "rawMarkdown": "Oh, awesome! Thank you for sharing this! It's nice to confirm own thoughts like this :) Though it is quite logical that they have decided to reserve recent data for the private :P Best of luck in the competition", "votes": 1 }, { "id": 656872, "postDate": "2019-10-24T17:57:29.590Z", "content": "Interesting! Thank you for sharing this. Seems legitimate to reserve the most recent data for the private leaderboard score.
", "rawMarkdown": "Interesting! Thank you for sharing this. Seems legitimate to reserve the most recent data for the private leaderboard score.", "votes": 1 }, { "id": 655688, "postDate": "2019-10-23T11:12:49.083Z", "content": "\n\n\n(test set and row_id column are not strictly time ordered).
\n
Blame on me. I didn't find it before you said. THANK YOU! \nSeems you are right. \nI did test with 22% of data but it was just last 22% of rows (and that test didn't show time split but I was wrong).
", "rawMarkdown": "> (test set and row_id column are not strictly time ordered).\n\nBlame on me. I didn't find it before you said. THANK YOU! \nSeems you are right. \nI did test with 22% of data but it was just last 22% of rows (and that test didn't show time split but I was wrong).", "votes": 2, "replies": [ { "id": 655863, "postDate": "2019-10-23T15:34:47.300Z", "content": "Not easy to spot at first glance. Glad to have been of help @kyakovlev!
", "rawMarkdown": "Not easy to spot at first glance. Glad to have been of help @kyakovlev!", "votes": 1 } ] }, { "id": 660670, "postDate": "2019-10-29T12:59:58.507Z", "content": "An entire year being in the private set makes more sense than August-December, because we don't skip the important summer period.
\n\nHowever, how do we reconcile that with the 22% figure, even with the gaps? It would mean there is 3.5 (78/22) more data in 2017 than 2018!
\n\n(Edit: OP already mentioned that same issue in another post)
", "rawMarkdown": "An entire year being in the private set makes more sense than August-December, because we don't skip the important summer period.\n\nHowever, how do we reconcile that with the 22% figure, even with the gaps? It would mean there is 3.5 (78/22) more data in 2017 than 2018!\n\n(Edit: OP already mentioned that same issue in another post)" }, { "id": 656840, "postDate": "2019-10-24T17:11:08.920Z", "content": "Really nice! 👍
", "rawMarkdown": "Really nice! 👍 " }, { "id": 655717, "postDate": "2019-10-23T12:27:12.583Z", "content": "wow huge!!!
", "rawMarkdown": "wow huge!!!" }, { "id": 655590, "postDate": "2019-10-23T08:16:16.790Z", "content": "\"Seems like the private dataset is just the last 22% of the data time ordered\"\nWhich it's what makes sense to do.\nThanks for confirming!
", "rawMarkdown": "\"Seems like the private dataset is just the last 22% of the data time ordered\"\nWhich it's what makes sense to do.\nThanks for confirming!" }, { "id": 655393, "postDate": "2019-10-23T02:17:18.087Z", "content": "your probing says that last 5 months is not in public test, but not saying 'private dataset is just the last 22% of the data time ordered'... maybe i misunderstood
", "rawMarkdown": "your probing says that last 5 months is not in public test, but not saying 'private dataset is just the last 22% of the data time ordered'... maybe i misunderstood", "replies": [ { "id": 655559, "postDate": "2019-10-23T07:22:38.200Z", "content": "There is a sentence above the leaderboard which indicates the proportion of data in the public and private datasets:\n> This leaderboard is calculated with approximately 78% of the test data.\nThe final results will be based on the other 22%, so the final standings may be different.
", "rawMarkdown": "There is a sentence above the leaderboard which indicates the proportion of data in the public and private datasets:\n> This leaderboard is calculated with approximately 78% of the test data.\nThe final results will be based on the other 22%, so the final standings may be different.\n", "votes": 3 }, { "id": 655607, "postDate": "2019-10-23T08:55:56.247Z", "content": "Oh thanks! Now it makes sense to me)))
", "rawMarkdown": "Oh thanks! Now it makes sense to me)))", "votes": 1 } ] }, { "id": 655121, "postDate": "2019-10-22T17:41:40.723Z", "rawMarkdown": "", "votes": -2, "isDeleted": true }, { "id": 661393, "postDate": "2019-10-30T08:32:48.820Z", "content": "Thanks for sharing!
", "rawMarkdown": "Thanks for sharing!" } ], "comments": [ { "id": 659813, "author_name": "Robin Vaysse", "author_url": "", "post_date": "2019-10-28T09:44:02.243000", "content": "As describe in data description: \n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n\nSo there might be more than the last 22% of data which are in private set.
\n\nI tried to set exp(25) for all values from 2018 and it also score 4.69. To check if it was a problem due to exp(25) being too big, I tried to set this value for all timestamp of the test set and it scored 20.6 (so there is no issue with exp(25)).
\n\nFor now WE NEED YOU to test the real dates of the private leaderboard (we have just 2 submission a day per person). To do so, here is some code to create a submission with 0 for all timestamps exept for dates greater than 2018-01-01.
\n\n```\nimport numpy as np\nimport pandas as pd
\n\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('data/sample_submission.csv')
\n\nmask = test.query('timestamp >= \"2018-01-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)
\n\nsubmission.to_csv('submission/probing_all.csv.zip', index=False, compression='zip')\n```
\n\nPlease try some random dates like 2017-12-01 and share your findings in order to find the truth 🙏
", "votes": 3, "replies": [ { "id": 660064, "author_name": "Adil Zouitine", "author_url": "", "post_date": "2019-10-28T16:05:47.147000", "content": "I added the 31st of December 2017 in addition to the whole 2018 year and replace exp(25) by exp(50). It scored 5.27.\nmask = test.query('timestamp >= \"2017-12-31\"').row_id.tolist()
Thx guys! Now we can be pretty confident saying that public and private sets are respectively 2017 and 2018. If the private part really represents 22% of test data as organizers said, it means it has 4-ish more gaps than the public...
", "votes": 2, "replies": [] }, { "id": 660194, "author_name": "🐢 Jun Koda", "author_url": "", "post_date": "2019-10-28T21:10:52.607000", "content": "Great! Thanks for your investigations!\nFrom Adil's number, 5.27 with exp(50) and all 0 score of 4.69 we can write
\n\n\n1/N * sum_inDec31 (y_true - 50)**2 - y_true**2 = 5.27**2 - 4.69**2,\n
where y = log(1 + meter_reading).
\n\nApproximating all y_true with 4 and assuming N days of equal amount of data,
\n\n\nN ~ (46**2 - 4**2)/(5.27**2 - 4.69**2) = 363.52.\n
This is consistent with 365 days (if there are \"gaps,\" Dec 31 has the same ratio of gaps as the year 2017 average.) It's most natural to think that gaps are minor in year 2017, as they are in the training set. I rather doubt the official statement that the public is 78%, I think public:private is 50:50.
\n\n[I first forgot about the square in the metric, I erased my stupid comment and rewrote]
", "votes": 3, "replies": [] } ] }, { "id": 659081, "author_name": "LIGHT", "author_url": "", "post_date": "2019-10-27T03:46:53.050000", "content": "Based on your code, I probed more data with the code below:
\n\nmask = test[(test.timestamp>'2018-07-20 00:00:00') & (test.timestamp<'2019-01-01 00:00:00')].row_id.tolist()
\n\nAnd still the same score (4.69).
\n\nps.
\n\n\n", "votes": 3, "replies": [ { "id": 659225, "author_name": "Geoffrey Bolmier", "author_url": "", "post_date": "2019-10-27T09:51:54.350000", "content": "'2018-07-20 00:00:00' means 22.5%.
\n
Probed a bit more before seeing your message adding last 14 days of July with the same result (4.69 for 22.88%):\n\nmask = test.query('timestamp >= \"2018-07-18\"').row_id.tolist()\n\nNo more submission for me today…
Probed from April and got the same result (4.69 for 37.67%):
\n\n\nmask = test.query('timestamp >= \"2018-04-01\"').row_id.tolist()\n
As organizers said in the data section:
\n\n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.
\n\nPrivate part containing way more gaps than the public could explain such results, meaning also that test distribution (78% public, 22% private) has been computed excluding those gaps
", "votes": 4, "replies": [] }, { "id": 659995, "author_name": "arnab", "author_url": "", "post_date": "2019-10-28T14:48:51.240000", "content": "Following are the fraction of rows belonging to a particular month-year (across train and test). It's very clear that the number of rows across different months in 2016 are varying w.r.t each other, but for each month across 2017 & 2018, the number of rows are matching exactly (up to sixth decimal). For months with 31 days, the fraction value is 0.028600 for 2017/2018. For months with 30 days, the fraction value is 0.027677. These two numbers are not matching with any of the months from 2016.
\n\nThis probably indicates the gaps have been systematically filled across the entire test data (2017/2018), but that is NOT the case for train data (2016).
\n\n
Oh, awesome! Thank you for sharing this! It's nice to confirm own thoughts like this :) Though it is quite logical that they have decided to reserve recent data for the private :P Best of luck in the competition
", "votes": 1, "replies": [] }, { "id": 656872, "author_name": "Carlo", "author_url": "", "post_date": "2019-10-24T17:57:29.590000", "content": "Interesting! Thank you for sharing this. Seems legitimate to reserve the most recent data for the private leaderboard score.
", "votes": 1, "replies": [] }, { "id": 655688, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-10-23T11:12:49.083000", "content": "\n\n\n(test set and row_id column are not strictly time ordered).
\n
Blame on me. I didn't find it before you said. THANK YOU! \nSeems you are right. \nI did test with 22% of data but it was just last 22% of rows (and that test didn't show time split but I was wrong).
", "votes": 2, "replies": [ { "id": 655863, "author_name": "Geoffrey Bolmier", "author_url": "", "post_date": "2019-10-23T15:34:47.300000", "content": "Not easy to spot at first glance. Glad to have been of help @kyakovlev!
", "votes": 1, "replies": [] } ] }, { "id": 660670, "author_name": "S D", "author_url": "", "post_date": "2019-10-29T12:59:58.507000", "content": "An entire year being in the private set makes more sense than August-December, because we don't skip the important summer period.
\n\nHowever, how do we reconcile that with the 22% figure, even with the gaps? It would mean there is 3.5 (78/22) more data in 2017 than 2018!
\n\n(Edit: OP already mentioned that same issue in another post)
", "votes": 0, "replies": [] }, { "id": 656840, "author_name": "AlvaroSanchez", "author_url": "", "post_date": "2019-10-24T17:11:08.920000", "content": "Really nice! 👍
", "votes": 0, "replies": [] }, { "id": 655717, "author_name": "", "author_url": "", "post_date": "2019-10-23T12:27:12.583000", "content": "wow huge!!!
", "votes": 0, "replies": [] }, { "id": 655590, "author_name": "bluetrain", "author_url": "", "post_date": "2019-10-23T08:16:16.790000", "content": "\"Seems like the private dataset is just the last 22% of the data time ordered\"\nWhich it's what makes sense to do.\nThanks for confirming!
", "votes": 0, "replies": [] }, { "id": 655393, "author_name": "LeiZhou", "author_url": "", "post_date": "2019-10-23T02:17:18.087000", "content": "your probing says that last 5 months is not in public test, but not saying 'private dataset is just the last 22% of the data time ordered'... maybe i misunderstood
", "votes": 0, "replies": [ { "id": 655559, "author_name": "Robin Vaysse", "author_url": "", "post_date": "2019-10-23T07:22:38.200000", "content": "There is a sentence above the leaderboard which indicates the proportion of data in the public and private datasets:\n> This leaderboard is calculated with approximately 78% of the test data.\nThe final results will be based on the other 22%, so the final standings may be different.
", "votes": 3, "replies": [] }, { "id": 655607, "author_name": "LeiZhou", "author_url": "", "post_date": "2019-10-23T08:55:56.247000", "content": "Oh thanks! Now it makes sense to me)))
", "votes": 1, "replies": [] } ] }, { "id": 655121, "author_name": "", "author_url": "", "post_date": "2019-10-22T17:41:40.723000", "content": "", "votes": -2, "replies": [] }, { "id": 661393, "author_name": "Wonho Song", "author_url": "", "post_date": "2019-10-30T08:32:48.820000", "content": "Thanks for sharing!
", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "654487": "Seems like the private dataset is just the last 22% of the data **time ordered** (test set and `row_id` column are not strictly time ordered). \n\nSample submission score to 4.69 submitting only zeros. I just replaced values from August 2018 to December 2018 (20.96% of the test set) by exp(25) and got the same score (4.69).\n\nFeel free to try yourself ;)\n\n```python\nimport numpy as np\nimport pandas as pd\n\n# load data\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('submission/sample_submission.csv')\n\n# replace last 5 months predictions to exp(25) \nmask = test.query('timestamp >= \"2018-08-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)\n\n# save submission\nsubmission.to_csv('submission/probing.csv', index=False)\n```\n\nEdit: Thx to @berserker408, @robinvaysse and @adilztn submissions which demonstrated that the whole 2018 year is part of the private set and 31st December 2017 is part of the public. We can then safely (it's up to you) consider that public is 2017 and private is 2018 ✔️ ", "659813": "As describe in data description: \n> There are gaps in some of the meter readings for both the train and test sets. Gaps in the test set are not revealed or scored.\n\nSo there might be more than the last 22% of data which are in private set.\n\nI tried to set exp(25) for all values from 2018 and it also score 4.69. To check if it was a problem due to exp(25) being too big, I tried to set this value for **all** timestamp of the test set and it scored 20.6 (so there is no issue with exp(25)).\n\nFor now **WE NEED YOU** to test the real dates of the private leaderboard (we have just 2 submission a day per person). To do so, here is some code to create a submission with 0 for all timestamps exept for dates greater than 2018-01-01.\n\n```\nimport numpy as np\nimport pandas as pd\n\n\n# load data\ntest = pd.read_csv('data/test.csv', parse_dates=['timestamp'])\nsubmission = pd.read_csv('data/sample_submission.csv')\n\n# change last year predictions to exp(25)\nmask = test.query('timestamp >= \"2018-01-01\"').row_id.tolist()\nsubmission.loc[submission['row_id'].isin(mask), 'meter_reading'] = np.exp(25)\n\n# save submission\nsubmission.to_csv('submission/probing_all.csv.zip', index=False, compression='zip')\n```\n\nPlease try some random dates like 2017-12-01 and share your findings in order to find the truth 🙏 ", "659081": "Based on your code, I probed more data with the code below:\n\nmask = test[(test.timestamp>'2018-07-20 00:00:00') & (test.timestamp<'2019-01-01 00:00:00')].row_id.tolist()\n\nAnd still the same score (4.69).\n\nps.\n>'2018-07-20 00:00:00' means 22.5%.", "693712": "Oh, awesome! Thank you for sharing this! It's nice to confirm own thoughts like this :) Though it is quite logical that they have decided to reserve recent data for the private :P Best of luck in the competition", "656872": "Interesting! Thank you for sharing this. Seems legitimate to reserve the most recent data for the private leaderboard score.", "655688": "> (test set and row_id column are not strictly time ordered).\n\nBlame on me. I didn't find it before you said. THANK YOU! \nSeems you are right. \nI did test with 22% of data but it was just last 22% of rows (and that test didn't show time split but I was wrong).", "660670": "An entire year being in the private set makes more sense than August-December, because we don't skip the important summer period.\n\nHowever, how do we reconcile that with the 22% figure, even with the gaps? It would mean there is 3.5 (78/22) more data in 2017 than 2018!\n\n(Edit: OP already mentioned that same issue in another post)", "656840": "Really nice! 👍 ", "655717": "wow huge!!!", "655590": "\"Seems like the private dataset is just the last 22% of the data time ordered\"\nWhich it's what makes sense to do.\nThanks for confirming!", "655393": "your probing says that last 5 months is not in public test, but not saying 'private dataset is just the last 22% of the data time ordered'... maybe i misunderstood", "655121": "", "661393": "Thanks for sharing!" } }