Hello everyone and congratulations on participating in this unusual competition. As the LB and the shake-up show, It was a hard one, not because of the task in itself but because of the conditions.
\nAn observation that confused me quite a bit was the following (taken from the data section):
\n\n\"A percentage of the public test set publications are drawn from the training set\"
\n
In this post I will show you how we can compute, quite accurately, the size of the hidden public train set and how we could use this information to reduce public LB overfitting. I'm sure most of the top teams were aware of that, but it might be that some of you missed it.
\nThe idea is to detect the training documents just by comparing them to the test documents at inference time:
\nMake the first submission as follows:
\nIterate as follows
\n: operator) the found training documents until when we found the smallest limit value large enough to reproduce the same score as before. Using dichotomy search, we can quickly (in about 5 submissions) find it.I found that the size of the public hidden train set is between 780 and 820 (I could get more precise, but that was enough for my case).
\nA bit of maths:
\n\n\nNote that the hidden test set has roughly ~8000 publications
\n
270, that's the size of the public LB test publications.
\nNote also that the same information could be used in other ways to reduce public LB overfitting. For instance, I found it helpful to fix the hidden public train set (either with the predictions or the empty string \"\") and evaluate the models only on the 270 hidden test publications.
To be honest, I didn't like too much the fact that we had to deal with training data in the test set. I hope future challenges will avoid this (Kaggle might need to give clearer guidelines to competitions host). I'm interested to know your opinion about that and how you used this information for reducing overfitting.
\nThanks for reading, and good luck to anyone for the next challenges!
", "messages": [ { "id": 1362063, "postDate": "2021-06-23T07:46:07.113Z", "content": "Hello everyone and congratulations on participating in this unusual competition. As the LB and the shake-up show, It was a hard one, not because of the task in itself but because of the conditions.
\nAn observation that confused me quite a bit was the following (taken from the data section):
\n\n\"A percentage of the public test set publications are drawn from the training set\"
\n
In this post I will show you how we can compute, quite accurately, the size of the hidden public train set and how we could use this information to reduce public LB overfitting. I'm sure most of the top teams were aware of that, but it might be that some of you missed it.
\nThe idea is to detect the training documents just by comparing them to the test documents at inference time:
\nMake the first submission as follows:
\nIterate as follows
\n: operator) the found training documents until when we found the smallest limit value large enough to reproduce the same score as before. Using dichotomy search, we can quickly (in about 5 submissions) find it.I found that the size of the public hidden train set is between 780 and 820 (I could get more precise, but that was enough for my case).
\nA bit of maths:
\n\n\nNote that the hidden test set has roughly ~8000 publications
\n
270, that's the size of the public LB test publications.
\nNote also that the same information could be used in other ways to reduce public LB overfitting. For instance, I found it helpful to fix the hidden public train set (either with the predictions or the empty string \"\") and evaluate the models only on the 270 hidden test publications.
To be honest, I didn't like too much the fact that we had to deal with training data in the test set. I hope future challenges will avoid this (Kaggle might need to give clearer guidelines to competitions host). I'm interested to know your opinion about that and how you used this information for reducing overfitting.
\nThanks for reading, and good luck to anyone for the next challenges!
", "rawMarkdown": "Hello everyone and congratulations on participating in this unusual competition. As the LB and the shake-up show, It was a hard one, not because of the task in itself but because of the conditions.\n\nAn observation that confused me quite a bit was the following (taken from the `data` section): \n\n> \"A percentage of the public test set publications are drawn from the training set\"\n\n**In this post I will show you how we can compute, quite accurately, the size of the hidden public train set and how we could use this information to reduce public LB overfitting.** I'm sure most of the top teams were aware of that, but it might be that some of you missed it. \n\nThe idea is to detect the training documents just by comparing them to the test documents at inference time:\n\n 1. Make the first submission as follows:\n \n - Find all hidden training documents and make a submission using as predictions only the provided \"cleaned_labels\" from the training document. The score is 0.532.\n\n 2. Iterate as follows\n - Limit (using the slice `:` operator) the found training documents until when we found the smallest limit value large enough to reproduce the same score as before. Using dichotomy search, we can quickly (in about 5 submissions) find it.\n\nI found that the size of the public hidden train set is between **780** and **820** (I could get more precise, but that was enough for my case).\n\nA bit of maths:\n\n> Note that the hidden test set has roughly ~8000 publications\n\n 1. This means that the public test set has roughly 8000 / 88 * 12 =~ 1090 publications\n 2. 1090 - 820 (worst case) =~ **270 pubblications**\n\n**270**, that's the size of the public LB test publications.\n\nNote also that the same information could be used in other ways to reduce public LB overfitting. For instance, I found it helpful to fix the hidden public train set (either with the predictions or the empty string `\"\"`) and evaluate the models only on the 270 hidden test publications.\n\nTo be honest, I didn't like too much the fact that we had to deal with training data in the test set. I hope future challenges will avoid this (Kaggle might need to give clearer guidelines to competitions host). I'm interested to know your opinion about that and how you used this information for reducing overfitting.\n\nThanks for reading, and good luck to anyone for the next challenges!", "votes": 2 } ], "comments": [], "raw_markdown_by_id": { "1362063": "Hello everyone and congratulations on participating in this unusual competition. As the LB and the shake-up show, It was a hard one, not because of the task in itself but because of the conditions.\n\nAn observation that confused me quite a bit was the following (taken from the `data` section): \n\n> \"A percentage of the public test set publications are drawn from the training set\"\n\n**In this post I will show you how we can compute, quite accurately, the size of the hidden public train set and how we could use this information to reduce public LB overfitting.** I'm sure most of the top teams were aware of that, but it might be that some of you missed it. \n\nThe idea is to detect the training documents just by comparing them to the test documents at inference time:\n\n 1. Make the first submission as follows:\n \n - Find all hidden training documents and make a submission using as predictions only the provided \"cleaned_labels\" from the training document. The score is 0.532.\n\n 2. Iterate as follows\n - Limit (using the slice `:` operator) the found training documents until when we found the smallest limit value large enough to reproduce the same score as before. Using dichotomy search, we can quickly (in about 5 submissions) find it.\n\nI found that the size of the public hidden train set is between **780** and **820** (I could get more precise, but that was enough for my case).\n\nA bit of maths:\n\n> Note that the hidden test set has roughly ~8000 publications\n\n 1. This means that the public test set has roughly 8000 / 88 * 12 =~ 1090 publications\n 2. 1090 - 820 (worst case) =~ **270 pubblications**\n\n**270**, that's the size of the public LB test publications.\n\nNote also that the same information could be used in other ways to reduce public LB overfitting. For instance, I found it helpful to fix the hidden public train set (either with the predictions or the empty string `\"\"`) and evaluate the models only on the 270 hidden test publications.\n\nTo be honest, I didn't like too much the fact that we had to deal with training data in the test set. I hope future challenges will avoid this (Kaggle might need to give clearer guidelines to competitions host). I'm interested to know your opinion about that and how you used this information for reducing overfitting.\n\nThanks for reading, and good luck to anyone for the next challenges!" } }