If I use a generative model like Donut, it could generate anything as an answer. For example it could insert an \"o\" instead of a 0. If the data series is numerical, and the scoring function does something like this: list(map(float, data_series.split(\";\"))), then it will get a value error when doing float(\"1o\"). Will this cause a submission error or will the score for the row just be 0?
I am facing scoring errors now, and I have a feeling it is because my model is generating text when it should be generating numbers.
\nI can come up with a way to filter out bad characters, but I am anticipating that my approach will not be 100% correct.
\n", "messages": [ { "id": 2203079, "postDate": "2023-03-30T13:51:12.873Z", "content": "If I use a generative model like Donut, it could generate anything as an answer. For example it could insert an \"o\" instead of a 0. If the data series is numerical, and the scoring function does something like this: list(map(float, data_series.split(\";\"))), then it will get a value error when doing float(\"1o\"). Will this cause a submission error or will the score for the row just be 0?
I am facing scoring errors now, and I have a feeling it is because my model is generating text when it should be generating numbers.
\nI can come up with a way to filter out bad characters, but I am anticipating that my approach will not be 100% correct.
\n", "rawMarkdown": "If I use a generative model like Donut, it could generate anything as an answer. For example it could insert an \"o\" instead of a 0. If the data series is numerical, and the scoring function does something like this: `list(map(float, data_series.split(\";\")))`, then it will get a value error when doing `float(\"1o\")`. Will this cause a submission error or will the score for the row just be 0? \n\nI am facing scoring errors now, and I have a feeling it is because my model is generating text when it should be generating numbers. \n\nI can come up with a way to filter out bad characters, but I am anticipating that my approach will not be 100% correct.\n\n@ryanholbrook", "votes": 8 }, { "id": 2203085, "postDate": "2023-03-30T13:55:54.130Z", "content": "Hey @nbroad, I actually just deployed a change to the metric that will score a 0.0 for an incorrect data type instead of failing. (You beat me to the announcement. 😅) I'll run a rescore on submissions so far, so you won't need to resubmit.
\nEDIT: Okay, rescore is finished. Let me know if it's not what you're expecting.
", "rawMarkdown": "Hey @nbroad, I actually just deployed a change to the metric that will score a 0.0 for an incorrect data type instead of failing. (You beat me to the announcement. 😅) I'll run a rescore on submissions so far, so you won't need to resubmit.\n\nEDIT: Okay, rescore is finished. Let me know if it's not what you're expecting.", "votes": 4, "replies": [ { "id": 2203089, "postDate": "2023-03-30T14:00:42.197Z", "content": "Lol it jumped from 0 to 0.42 😂
\nThank you!
", "rawMarkdown": "Lol it jumped from 0 to 0.42 😂\n\nThank you! ", "votes": 3, "replies": [ { "id": 2211023, "postDate": "2023-04-05T18:52:04.587Z", "content": "\nIs my understanding correct?
\nwhen GT is \"categorical\"
\nEven if the predicted value is of numeric type, it is converted to string and the score is calculated. In this case, the conversion never fails.
When GT is \"numerical\"
\nThe predicted value is converted to numeric type by float() and the score is calculated. If the conversion fails, the score is 0.
Yes, that's correct!
", "rawMarkdown": "Yes, that's correct!", "votes": 3 } ] } ] }, { "id": 2263807, "postDate": "2023-05-18T00:11:30.850Z", "content": "Here are a few lines I added to the end of the benetech_score function to set up local cv.
\n (gt_series, gt_type), (pred_series, pred_type) pairs:\n \n (gt_series[], ):\n pred_series = [(x) x pred_series]\n \n (gt_series[], ) == (pred_series[], ) == :\n scores.append()\n \n\n gt_type != pred_type: \n scores.append()\n : \n scores.append(score_series(gt_series, pred_series))\nHow did you solve if lists are empty
", "rawMarkdown": "How did you solve if lists are empty", "replies": [ { "id": 2275105, "postDate": "2023-05-26T14:04:45.787Z", "content": "You could add something like this as the first condition.
\nif len(pred_series) == 0:\n scores.append(0)\nthanks for your reply.you try to use pix2struct to get cv score?
", "rawMarkdown": "thanks for your reply.you try to use pix2struct to get cv score?", "votes": 1 }, { "id": 2275116, "postDate": "2023-05-26T14:18:19.137Z", "content": "Yes. I tested out the code with pix2struct and it gets 0.47-0.49 CV when trained on the generated data, and validated on the extracted data.
", "rawMarkdown": "Yes. I tested out the code with pix2struct and it gets 0.47-0.49 CV when trained on the generated data, and validated on the extracted data." }, { "id": 2276162, "postDate": "2023-05-26T14:51:21.320Z", "content": "Your results look pretty good. I modify benetech_score and still get 0
", "rawMarkdown": "Your results look pretty good. I modify benetech_score and still get 0" }, { "id": 2276177, "postDate": "2023-05-26T14:57:51.920Z", "content": "I found that writing all the model predictions to a TXT file and then scoring the results manually was useful for debugging.
\nIn my case, one prediction was somehow causing the normalized_rmse() function from the Benetech Mixed-Match notebook to return NAN. This cause my score to be 0 as well.
Not sure if this is the same as your error, but hope it helps :)
", "rawMarkdown": "I found that writing all the model predictions to a TXT file and then scoring the results manually was useful for debugging. \n\nIn my case, one prediction was somehow causing the `normalized_rmse()` function from the [Benetech Mixed-Match](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match?scriptVersionId=123629556&cellId=3) notebook to return NAN. This cause my score to be 0 as well.\n\nNot sure if this is the same as your error, but hope it helps :)" }, { "id": 2276187, "postDate": "2023-05-26T15:08:19.560Z", "content": "you solve this normalized_rmse problem?
", "rawMarkdown": "you solve this normalized_rmse problem?" }, { "id": 2276197, "postDate": "2023-05-26T15:15:27.487Z", "content": "Not sure if it is the right solution, but I solved by clipping the R2_score between 0-1.
\n def normalized_rmse(self, y_true, y_pred):\n # The argument to the sigmoid transform is equal to \n # rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true))\n return self.sigmoid((1 - np.clip(r2_score(y_true, y_pred), 0, 1) ** 0.5))\nI modify my code,but still give me 0 cv score
", "rawMarkdown": "I modify my code,but still give me 0 cv score" } ] } ] }, { "id": 2210441, "postDate": "2023-04-05T11:49:04.297Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 2203085, "author_name": "Ryan Holbrook", "author_url": "", "post_date": "2023-03-30T13:55:54.130000", "content": "Hey @nbroad, I actually just deployed a change to the metric that will score a 0.0 for an incorrect data type instead of failing. (You beat me to the announcement. 😅) I'll run a rescore on submissions so far, so you won't need to resubmit.
\nEDIT: Okay, rescore is finished. Let me know if it's not what you're expecting.
", "votes": 4, "replies": [ { "id": 2203089, "author_name": "Nicholas Broad", "author_url": "", "post_date": "2023-03-30T14:00:42.197000", "content": "Lol it jumped from 0 to 0.42 😂
\nThank you!
", "votes": 3, "replies": [ { "id": 2211023, "author_name": "penguin46", "author_url": "", "post_date": "2023-04-05T18:52:04.587000", "content": "\nIs my understanding correct?
\nwhen GT is \"categorical\"
\nEven if the predicted value is of numeric type, it is converted to string and the score is calculated. In this case, the conversion never fails.
When GT is \"numerical\"
\nThe predicted value is converted to numeric type by float() and the score is calculated. If the conversion fails, the score is 0.
Yes, that's correct!
", "votes": 3, "replies": [] } ] } ] }, { "id": 2263807, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-18T00:11:30.850000", "content": "Here are a few lines I added to the end of the benetech_score function to set up local cv.
\n (gt_series, gt_type), (pred_series, pred_type) pairs:\n \n (gt_series[], ):\n pred_series = [(x) x pred_series]\n \n (gt_series[], ) == (pred_series[], ) == :\n scores.append()\n \n\n gt_type != pred_type: \n scores.append()\n : \n scores.append(score_series(gt_series, pred_series))\nHow did you solve if lists are empty
", "votes": 0, "replies": [ { "id": 2275105, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-26T14:04:45.787000", "content": "You could add something like this as the first condition.
\nif len(pred_series) == 0:\n scores.append(0)\nthanks for your reply.you try to use pix2struct to get cv score?
", "votes": 1, "replies": [] }, { "id": 2275116, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-26T14:18:19.137000", "content": "Yes. I tested out the code with pix2struct and it gets 0.47-0.49 CV when trained on the generated data, and validated on the extracted data.
", "votes": 0, "replies": [] }, { "id": 2276162, "author_name": "Catch_Bat", "author_url": "", "post_date": "2023-05-26T14:51:21.320000", "content": "Your results look pretty good. I modify benetech_score and still get 0
", "votes": 0, "replies": [] }, { "id": 2276177, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-26T14:57:51.920000", "content": "I found that writing all the model predictions to a TXT file and then scoring the results manually was useful for debugging.
\nIn my case, one prediction was somehow causing the normalized_rmse() function from the Benetech Mixed-Match notebook to return NAN. This cause my score to be 0 as well.
Not sure if this is the same as your error, but hope it helps :)
", "votes": 0, "replies": [] }, { "id": 2276187, "author_name": "Catch_Bat", "author_url": "", "post_date": "2023-05-26T15:08:19.560000", "content": "you solve this normalized_rmse problem?
", "votes": 0, "replies": [] }, { "id": 2276197, "author_name": "Bartley", "author_url": "", "post_date": "2023-05-26T15:15:27.487000", "content": "Not sure if it is the right solution, but I solved by clipping the R2_score between 0-1.
\n def normalized_rmse(self, y_true, y_pred):\n # The argument to the sigmoid transform is equal to \n # rmse(y_true, y_pred) / rmse(y_true, np.mean(y_true))\n return self.sigmoid((1 - np.clip(r2_score(y_true, y_pred), 0, 1) ** 0.5))\nI modify my code,but still give me 0 cv score
", "votes": 0, "replies": [] } ] } ] }, { "id": 2210441, "author_name": "", "author_url": "", "post_date": "2023-04-05T11:49:04.297000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "2203079": "If I use a generative model like Donut, it could generate anything as an answer. For example it could insert an \"o\" instead of a 0. If the data series is numerical, and the scoring function does something like this: `list(map(float, data_series.split(\";\")))`, then it will get a value error when doing `float(\"1o\")`. Will this cause a submission error or will the score for the row just be 0? \n\nI am facing scoring errors now, and I have a feeling it is because my model is generating text when it should be generating numbers. \n\nI can come up with a way to filter out bad characters, but I am anticipating that my approach will not be 100% correct.\n\n@ryanholbrook", "2203085": "Hey @nbroad, I actually just deployed a change to the metric that will score a 0.0 for an incorrect data type instead of failing. (You beat me to the announcement. 😅) I'll run a rescore on submissions so far, so you won't need to resubmit.\n\nEDIT: Okay, rescore is finished. Let me know if it's not what you're expecting.", "2263807": "Here are a few lines I added to the end of the benetech_score function to set up local cv.\n\n```python\n for (gt_series, gt_type), (pred_series, pred_type) in pairs:\n # Convert ground predicted series to strings\n if isinstance(gt_series[0], str):\n pred_series = [str(x) for x in pred_series]\n # Return if pred_series is str and ground truth is not\n if isinstance(gt_series[0], str) == False and isinstance(pred_series[0], str) == True:\n scores.append(0)\n continue\n \n if gt_type != pred_type: # Check chart_type condition\n scores.append(0.0)\n else: # Score with RMSE or Levenshtein as appropriate\n scores.append(score_series(gt_series, pred_series))\n```", "2210441": "" } }