A quick clarification:
\nI initially planned to use F1 score for evaluation.
\nUnfortunately, Kaggle’s evaluation backend threw errors when I tried both Micro-F1 and Macro-F1 with this multi-class setup.
\nTo keep the leaderboard functional, I switched to accuracy.
\nBecause over 90% of the data points are class N, predicting all N gives a baseline score of about 0.93840. Any score above this baseline reflects a real improvement. Please view the current leaderboard as a guideline only.
\nIn addition, I also maintain a non-public dataset that I plan to use for the final evaluation after the competition ends.
\nI’d like your input on how we should handle this final step:
\nShould we run final evaluation on the non-public dataset only, after the competition ends (similar to Kaggle’s private leaderboard)?
\nShould I resubmit another dataset with private values to Kaggle and use their standard public/private split?
\nOr would you prefer another process (e.g., collecting submissions and running them offline on my dataset)?
\nSince Kaggle normally handles a public leaderboard (based on a subset) and a private leaderboard (final scoring) automatically, I want to align with how participants expect things to work.
\nLooking forward to your suggestions on how to make the final evaluation fair and transparent.
", "messages": [ { "id": 3295893, "postDate": "2025-09-29T18:13:50.650Z", "content": "A quick clarification:
\nI initially planned to use F1 score for evaluation.
\nUnfortunately, Kaggle’s evaluation backend threw errors when I tried both Micro-F1 and Macro-F1 with this multi-class setup.
\nTo keep the leaderboard functional, I switched to accuracy.
\nBecause over 90% of the data points are class N, predicting all N gives a baseline score of about 0.93840. Any score above this baseline reflects a real improvement. Please view the current leaderboard as a guideline only.
\nIn addition, I also maintain a non-public dataset that I plan to use for the final evaluation after the competition ends.
\nI’d like your input on how we should handle this final step:
\nShould we run final evaluation on the non-public dataset only, after the competition ends (similar to Kaggle’s private leaderboard)?
\nShould I resubmit another dataset with private values to Kaggle and use their standard public/private split?
\nOr would you prefer another process (e.g., collecting submissions and running them offline on my dataset)?
\nSince Kaggle normally handles a public leaderboard (based on a subset) and a private leaderboard (final scoring) automatically, I want to align with how participants expect things to work.
\nLooking forward to your suggestions on how to make the final evaluation fair and transparent.
", "rawMarkdown": "A quick clarification:\n\nI initially planned to use F1 score for evaluation.\n\nUnfortunately, Kaggle’s evaluation backend threw errors when I tried both Micro-F1 and Macro-F1 with this multi-class setup.\n\nTo keep the leaderboard functional, I switched to accuracy.\n\nBecause over 90% of the data points are class N, predicting all N gives a baseline score of about 0.93840. Any score above this baseline reflects a real improvement. Please view the current leaderboard as a guideline only.\n\nIn addition, I also maintain a non-public dataset that I plan to use for the final evaluation after the competition ends.\n\nI’d like your input on how we should handle this final step:\n\nShould we run final evaluation on the non-public dataset only, after the competition ends (similar to Kaggle’s private leaderboard)?\n\nShould I resubmit another dataset with private values to Kaggle and use their standard public/private split?\n\nOr would you prefer another process (e.g., collecting submissions and running them offline on my dataset)?\n\nSince Kaggle normally handles a public leaderboard (based on a subset) and a private leaderboard (final scoring) automatically, I want to align with how participants expect things to work.\n\nLooking forward to your suggestions on how to make the final evaluation fair and transparent.", "votes": 2 }, { "id": 3306310, "postDate": "2025-10-24T09:37:38.857Z", "content": "something else
\npredict (H,L, None) as mention flat model score 0.008 while replace the None with N scored 0.77 !!!
the None were mentioned in contest page
the final accurate labels are what, kindly confirm ?
\n@ozzymet
Do we have to predict only H/L? not 'None'?
\nare all the ids in test are those where you identified 'H' or 'L', or we have 'None' also in test set?
Test set contains None, bro. There's a public notebook named baseline, it predicted all the sample of test set is None, and got lb 0.93
", "rawMarkdown": "Test set contains None, bro. There's a public notebook named baseline, it predicted all the sample of test set is None, and got lb 0.93" } ] }, { "id": 3295999, "postDate": "2025-09-30T00:57:01.187Z", "content": "I hope you host can provide a clearer explanation of the features in the dataset, specifically how the four main features—ratio, momentum, sm_ratio, and sm_momentum—are generated, as well as the difference between happens_within and occurs_within.
", "rawMarkdown": "I hope you host can provide a clearer explanation of the features in the dataset, specifically how the four main features—ratio, momentum, sm_ratio, and sm_momentum—are generated, as well as the difference between happens_within and occurs_within." }, { "id": 3295902, "postDate": "2025-09-29T18:28:19.713Z", "content": "Thanks for the clarification, @OzzyMet 🙌
\nI think the fairest and most transparent approach would be to follow a process similar to Kaggle’s public/private leaderboard setup:
\nUse the current leaderboard for public scores,
\nThen evaluate final results on your non-public dataset (as a private leaderboard) after the competition ends.
\nThis way, participants can still iterate and compare progress during the competition, while the final ranking reflects true generalization performance.
\nReally appreciate your effort in keeping things fair and transparent! 💪📊
", "rawMarkdown": "Thanks for the clarification, @OzzyMet 🙌\n\nI think the fairest and most transparent approach would be to follow a process similar to Kaggle’s public/private leaderboard setup:\n\nUse the current leaderboard for public scores,\n\nThen evaluate final results on your non-public dataset (as a private leaderboard) after the competition ends.\n\nThis way, participants can still iterate and compare progress during the competition, while the final ranking reflects true generalization performance.\n\nReally appreciate your effort in keeping things fair and transparent! 💪📊", "replies": [ { "id": 3296149, "postDate": "2025-09-30T09:20:18.930Z", "content": "I aggre with you
", "rawMarkdown": "I aggre with you", "isDeleted": true } ] } ], "comments": [ { "id": 3306310, "author_name": "Mostafa Beik", "author_url": "", "post_date": "2025-10-24T09:37:38.857000", "content": "something else
\npredict (H,L, None) as mention flat model score 0.008 while replace the None with N scored 0.77 !!!
the None were mentioned in contest page
the final accurate labels are what, kindly confirm ?
\n@ozzymet
Do we have to predict only H/L? not 'None'?
\nare all the ids in test are those where you identified 'H' or 'L', or we have 'None' also in test set?
Test set contains None, bro. There's a public notebook named baseline, it predicted all the sample of test set is None, and got lb 0.93
", "votes": 0, "replies": [] } ] }, { "id": 3295999, "author_name": "Sam Bui", "author_url": "", "post_date": "2025-09-30T00:57:01.187000", "content": "I hope you host can provide a clearer explanation of the features in the dataset, specifically how the four main features—ratio, momentum, sm_ratio, and sm_momentum—are generated, as well as the difference between happens_within and occurs_within.
", "votes": 0, "replies": [] }, { "id": 3295902, "author_name": "Prince Rajak", "author_url": "", "post_date": "2025-09-29T18:28:19.713000", "content": "Thanks for the clarification, @OzzyMet 🙌
\nI think the fairest and most transparent approach would be to follow a process similar to Kaggle’s public/private leaderboard setup:
\nUse the current leaderboard for public scores,
\nThen evaluate final results on your non-public dataset (as a private leaderboard) after the competition ends.
\nThis way, participants can still iterate and compare progress during the competition, while the final ranking reflects true generalization performance.
\nReally appreciate your effort in keeping things fair and transparent! 💪📊
", "votes": 0, "replies": [ { "id": 3296149, "author_name": "", "author_url": "", "post_date": "2025-09-30T09:20:18.930000", "content": "I aggre with you
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "3295893": "A quick clarification:\n\nI initially planned to use F1 score for evaluation.\n\nUnfortunately, Kaggle’s evaluation backend threw errors when I tried both Micro-F1 and Macro-F1 with this multi-class setup.\n\nTo keep the leaderboard functional, I switched to accuracy.\n\nBecause over 90% of the data points are class N, predicting all N gives a baseline score of about 0.93840. Any score above this baseline reflects a real improvement. Please view the current leaderboard as a guideline only.\n\nIn addition, I also maintain a non-public dataset that I plan to use for the final evaluation after the competition ends.\n\nI’d like your input on how we should handle this final step:\n\nShould we run final evaluation on the non-public dataset only, after the competition ends (similar to Kaggle’s private leaderboard)?\n\nShould I resubmit another dataset with private values to Kaggle and use their standard public/private split?\n\nOr would you prefer another process (e.g., collecting submissions and running them offline on my dataset)?\n\nSince Kaggle normally handles a public leaderboard (based on a subset) and a private leaderboard (final scoring) automatically, I want to align with how participants expect things to work.\n\nLooking forward to your suggestions on how to make the final evaluation fair and transparent.", "3306310": "something else\npredict (`H`,`L`, `None`) as mention flat model score 0.008 while replace the `None` with `N` scored 0.77 !!! \n\nthe `None` were mentioned in contest page\n\nthe final accurate labels are what, kindly confirm ?\n@ozzymet ", "3301132": "Do we have to predict only H/L? not 'None'?\nare all the ids in test are those where you identified 'H' or 'L', or we have 'None' also in test set?", "3295999": "I hope you host can provide a clearer explanation of the features in the dataset, specifically how the four main features—ratio, momentum, sm_ratio, and sm_momentum—are generated, as well as the difference between happens_within and occurs_within.", "3295902": "Thanks for the clarification, @OzzyMet 🙌\n\nI think the fairest and most transparent approach would be to follow a process similar to Kaggle’s public/private leaderboard setup:\n\nUse the current leaderboard for public scores,\n\nThen evaluate final results on your non-public dataset (as a private leaderboard) after the competition ends.\n\nThis way, participants can still iterate and compare progress during the competition, while the final ranking reflects true generalization performance.\n\nReally appreciate your effort in keeping things fair and transparent! 💪📊" } }