I have two questions:
\n- Should I convert all values of x and y lists to string type in the submission file?
\nI am asking this question because when I submit my final submission file, if the error occurs, that will prevent me from getting the score.
\nThank you for your help
", "messages": [ { "id": "2305574", "postDate": "06/16/2023 19:18:53", "content": "I have two questions:
\n- Should I convert all values of x and y lists to string type in the submission file?
\nI am asking this question because when I submit my final submission file, if the error occurs, that will prevent me from getting the score.
\nThank you for your help
", "rawMarkdown": "I have two questions: \n\n**- Should I convert all values of x and y lists to string type in the submission file?**\n\n- I have seen that if the list of x values is integer list and the corresponding ground truth list is string, that throws an error? For example, imagine you have a vertical bar chat, the ground truth list of the x-axis should be string type but your classifier incorrectly predicts this as horizontal bar chart and your algorithm, based on this incorrect prediction, returns integer x-list. In this case, if we use the code of the evaluation metric is used in Kaggle, that throws an error. How can I suppress this error or fix this problem?\n\nI am asking this question because when I submit my final submission file, if the error occurs, that will prevent me from getting the score.\n\nThank you for your help", "votes": null }, { "id": "2305645", "postDate": "06/16/2023 20:14:29", "content": "The data has to be a string because it is delimited by ;
Good:
\n1;2;3
\n1.0;2.0;3.0
Bad:
\n\"1\";\"2\";\"3\"
\n[1, 2, 3]
Thank you, Nicholas. So all data should be in a string format in the submission.csv file.
", "rawMarkdown": "Thank you, Nicholas. So all data should be in a string format in the submission.csv file.", "votes": null }, { "id": "2305683", "postDate": "06/16/2023 20:57:53", "content": "@hamzahabduljalil
\nRegarding your second question, which score calculation code are you using? The official published notebooks will not be able to handle some corner cases. If the chart_type and value_type do not match on submission, I think you will only get a score of 0 for that sample, but you will get an overall score.
Thank you for your reply. I should have given a clearer example than in the question. Imagine that you have a vertical bar and your algorithm correctly classifies it. The y values are 1.5;2.5;3.5;4.5;5.5;6.5 as ground truth, but let assume that your algorithm (OCR) cannot detect the y-ticks values correctly, and you end up with i, 2.5, 3, 4, 5. The OCR mistakenly recognizes the 1 as i and, therefore, your predicted y values will be categorical rather than numerical. Now, if you evaluate your predicted y-values against the ground truth values by the publicly published notebook in Kaggle, that will throw an error because it cannot evaluate numerical values against categorical. I have my own validation data and I have few examples where it throws an error and because of the bad recognition of the OCR. That's why I am asking how we can solve this problem.
", "rawMarkdown": "Thank you for your reply. I should have given a clearer example than in the question. Imagine that you have a vertical bar and your algorithm correctly classifies it. The y values are 1.5;2.5;3.5;4.5;5.5;6.5 as ground truth, but let assume that your algorithm (OCR) cannot detect the y-ticks values correctly, and you end up with i, 2.5, 3, 4, 5. The OCR mistakenly recognizes the 1 as i and, therefore, your predicted y values will be categorical rather than numerical. Now, if you evaluate your predicted y-values against the ground truth values by the publicly published notebook in Kaggle, that will throw an error because it cannot evaluate numerical values against categorical. I have my own validation data and I have few examples where it throws an error and because of the bad recognition of the OCR. That's why I am asking how we can solve this problem.", "votes": null }, { "id": "2305830", "postDate": "06/16/2023 23:49:02", "content": "Could you please read my above comment because I try to explain the problem clearer.
", "rawMarkdown": "Could you please read my above comment because I try to explain the problem clearer.", "votes": null }, { "id": "2305859", "postDate": "06/17/2023 00:40:57", "content": "https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528
", "rawMarkdown": "https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528", "votes": null }, { "id": "2306021", "postDate": "06/17/2023 04:06:27", "content": "In that case the score will be 0, as Nicholas is referring to the thread. If the numerical type is expected to be correct, the error can be avoided by replacing the prediction with 0 when it cannot be converted to a float type, for example. (ex. i, 2.5, 3, 4, 5 -> 0, 2.5, 3, 4, 5)
", "rawMarkdown": "In that case the score will be 0, as Nicholas is referring to the thread. If the numerical type is expected to be correct, the error can be avoided by replacing the prediction with 0 when it cannot be converted to a float type, for example. (ex. i, 2.5, 3, 4, 5 -> 0, 2.5, 3, 4, 5)", "votes": null }, { "id": "2306344", "postDate": "06/17/2023 08:17:56", "content": "Thank you for both of you. I checked this notebook but there is no check if the y_pred is the same type of y_true before calling:
\n normalized_levenshtein_score(y_true, y_pred)\n :\n normalized_rmse(y_true, y_pred)\nIs the evaluation metric used in the submission file on the website the same as this notebook?
", "rawMarkdown": "Thank you for both of you. I checked [this notebook](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match) but there is no check if the **y_pred** is the same type of y_true before calling: \n```python\n return normalized_levenshtein_score(y_true, y_pred)\n else:\n return normalized_rmse(y_true, y_pred)\n```\nIs the evaluation metric used in the submission file on the website the same as this notebook?", "votes": null }, { "id": "2306399", "postDate": "06/17/2023 09:11:49", "content": "I think not.
", "rawMarkdown": "I think not.", "votes": null }, { "id": "2306417", "postDate": "06/17/2023 09:32:32", "content": "Then, the the error will occur in the case of mismatch between the types of GT values and predicted y-values on the website over the test dataset?
\nWhat do you think @ryanholbrook?
In the case of data type mismatch, that particular prediction will get 0 score. You can follow the discussion here: https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528#2203085
", "rawMarkdown": "In the case of data type mismatch, that particular prediction will get 0 score. You can follow the discussion here: https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528#2203085", "votes": null }, { "id": "2306443", "postDate": "06/17/2023 09:49:00", "content": "Thank you @conjuring92 . There is an update on the notebook which evaluates the test dataset on the website. But this is notebook is not updated, right?
", "rawMarkdown": "Thank you @conjuring92 . There is an update on the notebook which evaluates the test dataset on the website. But this is [notebook](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match) is not updated, right?", "votes": null }, { "id": "2306451", "postDate": "06/17/2023 10:04:09", "content": "yes, I think actual eval code is slightly different from the notebook
", "rawMarkdown": "yes, I think actual eval code is slightly different from the notebook", "votes": null }, { "id": "2308119", "postDate": "06/18/2023 16:56:40", "content": "@nbroad hi Nicholas, I was expecting I will have a submission score error. Could you please look at this question here and tell me if I have something wrong? I highly appriciate your time.
", "rawMarkdown": "nbroad hi Nicholas, I was expecting I will have a submission score error. Could you please look at this question [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/418077) and tell me if I have something wrong? I highly appriciate your time.", "votes": null }, { "id": "2308121", "postDate": "06/18/2023 16:57:40", "content": "Hi @conjuring92 I was expecting I will have a submission score error. Could you please look at this question hereand tell me if I have something wrong? I highly appriciate your time.
", "rawMarkdown": "Hi @conjuring92 I was expecting I will have a submission score error. Could you please look at this question [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/418077)and tell me if I have something wrong? I highly appriciate your time.", "votes": null }, { "id": "2308145", "postDate": "06/18/2023 17:16:51", "content": "I'm not quite sure why you're getting the scoring error. Maybe instead of 'chart_type': ' ' use 'chart_type': 'line' (or any other possible types) inside the exception block…
@conjuring92 , thank you, could you please also tell me what I should fill the x_values and y_values in case of exception to avoid this nightmare error?
", "rawMarkdown": "conjuring92 , thank you, could you please also tell me what I should fill the x_values and y_values in case of exception to avoid this nightmare error?", "votes": null }, { "id": "2308172", "postDate": "06/18/2023 17:48:08", "content": "I think your current exception handling for x_values and y_values are okay
", "rawMarkdown": "I think your current exception handling for x_values and y_values are okay", "votes": null }, { "id": "2308180", "postDate": "06/18/2023 17:54:57", "content": "@conjuring92 you so much. I hope you the best :)
", "rawMarkdown": "conjuring92 you so much. I hope you the best :)", "votes": null } ], "comments": [ { "id": 2305645, "author_name": "nbroad", "author_url": "", "post_date": "06/16/2023 20:14:29", "content": "The data has to be a string because it is delimited by ;
Good:
\n1;2;3
\n1.0;2.0;3.0
Bad:
\n\"1\";\"2\";\"3\"
\n[1, 2, 3]
Thank you, Nicholas. So all data should be in a string format in the submission.csv file.
", "votes": null, "replies": [] }, { "id": 2305830, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/16/2023 23:49:02", "content": "Could you please read my above comment because I try to explain the problem clearer.
", "votes": null, "replies": [ { "id": 2308119, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/18/2023 16:56:40", "content": "@nbroad hi Nicholas, I was expecting I will have a submission score error. Could you please look at this question here and tell me if I have something wrong? I highly appriciate your time.
", "votes": null, "replies": [] } ] } ] }, { "id": 2305683, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "06/16/2023 20:57:53", "content": "@hamzahabduljalil
\nRegarding your second question, which score calculation code are you using? The official published notebooks will not be able to handle some corner cases. If the chart_type and value_type do not match on submission, I think you will only get a score of 0 for that sample, but you will get an overall score.
Thank you for your reply. I should have given a clearer example than in the question. Imagine that you have a vertical bar and your algorithm correctly classifies it. The y values are 1.5;2.5;3.5;4.5;5.5;6.5 as ground truth, but let assume that your algorithm (OCR) cannot detect the y-ticks values correctly, and you end up with i, 2.5, 3, 4, 5. The OCR mistakenly recognizes the 1 as i and, therefore, your predicted y values will be categorical rather than numerical. Now, if you evaluate your predicted y-values against the ground truth values by the publicly published notebook in Kaggle, that will throw an error because it cannot evaluate numerical values against categorical. I have my own validation data and I have few examples where it throws an error and because of the bad recognition of the OCR. That's why I am asking how we can solve this problem.
", "votes": null, "replies": [ { "id": 2305859, "author_name": "nbroad", "author_url": "", "post_date": "06/17/2023 00:40:57", "content": "https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528
", "votes": null, "replies": [ { "id": 2306021, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "06/17/2023 04:06:27", "content": "In that case the score will be 0, as Nicholas is referring to the thread. If the numerical type is expected to be correct, the error can be avoided by replacing the prediction with 0 when it cannot be converted to a float type, for example. (ex. i, 2.5, 3, 4, 5 -> 0, 2.5, 3, 4, 5)
", "votes": null, "replies": [ { "id": 2306344, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/17/2023 08:17:56", "content": "Thank you for both of you. I checked this notebook but there is no check if the y_pred is the same type of y_true before calling:
\n normalized_levenshtein_score(y_true, y_pred)\n :\n normalized_rmse(y_true, y_pred)\nIs the evaluation metric used in the submission file on the website the same as this notebook?
", "votes": null, "replies": [ { "id": 2306399, "author_name": "ryotayoshinobu", "author_url": "", "post_date": "06/17/2023 09:11:49", "content": "I think not.
", "votes": null, "replies": [ { "id": 2306417, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/17/2023 09:32:32", "content": "Then, the the error will occur in the case of mismatch between the types of GT values and predicted y-values on the website over the test dataset?
\nWhat do you think @ryanholbrook?
In the case of data type mismatch, that particular prediction will get 0 score. You can follow the discussion here: https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528#2203085
", "votes": null, "replies": [ { "id": 2306443, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/17/2023 09:49:00", "content": "Thank you @conjuring92 . There is an update on the notebook which evaluates the test dataset on the website. But this is notebook is not updated, right?
", "votes": null, "replies": [ { "id": 2306451, "author_name": "conjuring92", "author_url": "", "post_date": "06/17/2023 10:04:09", "content": "yes, I think actual eval code is slightly different from the notebook
", "votes": null, "replies": [ { "id": 2308121, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/18/2023 16:57:40", "content": "Hi @conjuring92 I was expecting I will have a submission score error. Could you please look at this question hereand tell me if I have something wrong? I highly appriciate your time.
", "votes": null, "replies": [ { "id": 2308145, "author_name": "conjuring92", "author_url": "", "post_date": "06/18/2023 17:16:51", "content": "I'm not quite sure why you're getting the scoring error. Maybe instead of 'chart_type': ' ' use 'chart_type': 'line' (or any other possible types) inside the exception block…
@conjuring92 , thank you, could you please also tell me what I should fill the x_values and y_values in case of exception to avoid this nightmare error?
", "votes": null, "replies": [ { "id": 2308172, "author_name": "conjuring92", "author_url": "", "post_date": "06/18/2023 17:48:08", "content": "I think your current exception handling for x_values and y_values are okay
", "votes": null, "replies": [ { "id": 2308180, "author_name": "hamzahabduljalil", "author_url": "", "post_date": "06/18/2023 17:54:57", "content": "@conjuring92 you so much. I hope you the best :)
", "votes": null, "replies": [] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ] } ], "raw_markdown_by_id": { "2305574": "I have two questions: \n\n**- Should I convert all values of x and y lists to string type in the submission file?**\n\n- I have seen that if the list of x values is integer list and the corresponding ground truth list is string, that throws an error? For example, imagine you have a vertical bar chat, the ground truth list of the x-axis should be string type but your classifier incorrectly predicts this as horizontal bar chart and your algorithm, based on this incorrect prediction, returns integer x-list. In this case, if we use the code of the evaluation metric is used in Kaggle, that throws an error. How can I suppress this error or fix this problem?\n\nI am asking this question because when I submit my final submission file, if the error occurs, that will prevent me from getting the score.\n\nThank you for your help", "2305645": "The data has to be a string because it is delimited by `;`\n\nGood: \n1;2;3 \n1.0;2.0;3.0 \n\nBad: \n\"1\";\"2\";\"3\" \n[1, 2, 3]", "2305671": "Thank you, Nicholas. So all data should be in a string format in the submission.csv file.", "2305683": "hamzahabduljalil \nRegarding your second question, which score calculation code are you using? The official published notebooks will not be able to handle some corner cases. If the chart_type and value_type do not match on submission, I think you will only get a score of 0 for that sample, but you will get an overall score.", "2305828": "Thank you for your reply. I should have given a clearer example than in the question. Imagine that you have a vertical bar and your algorithm correctly classifies it. The y values are 1.5;2.5;3.5;4.5;5.5;6.5 as ground truth, but let assume that your algorithm (OCR) cannot detect the y-ticks values correctly, and you end up with i, 2.5, 3, 4, 5. The OCR mistakenly recognizes the 1 as i and, therefore, your predicted y values will be categorical rather than numerical. Now, if you evaluate your predicted y-values against the ground truth values by the publicly published notebook in Kaggle, that will throw an error because it cannot evaluate numerical values against categorical. I have my own validation data and I have few examples where it throws an error and because of the bad recognition of the OCR. That's why I am asking how we can solve this problem.", "2305830": "Could you please read my above comment because I try to explain the problem clearer.", "2305859": "https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528", "2306021": "In that case the score will be 0, as Nicholas is referring to the thread. If the numerical type is expected to be correct, the error can be avoided by replacing the prediction with 0 when it cannot be converted to a float type, for example. (ex. i, 2.5, 3, 4, 5 -> 0, 2.5, 3, 4, 5)", "2306344": "Thank you for both of you. I checked [this notebook](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match) but there is no check if the **y_pred** is the same type of y_true before calling: \n```python\n return normalized_levenshtein_score(y_true, y_pred)\n else:\n return normalized_rmse(y_true, y_pred)\n```\nIs the evaluation metric used in the submission file on the website the same as this notebook?", "2306399": "I think not.", "2306417": "Then, the the error will occur in the case of mismatch between the types of GT values and predicted y-values on the website over the test dataset? \nWhat do you think @ryanholbrook?", "2306434": "In the case of data type mismatch, that particular prediction will get 0 score. You can follow the discussion here: https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/398528#2203085", "2306443": "Thank you @conjuring92 . There is an update on the notebook which evaluates the test dataset on the website. But this is [notebook](https://www.kaggle.com/code/ryanholbrook/competition-metric-benetech-mixed-match) is not updated, right?", "2306451": "yes, I think actual eval code is slightly different from the notebook", "2308119": "nbroad hi Nicholas, I was expecting I will have a submission score error. Could you please look at this question [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/418077) and tell me if I have something wrong? I highly appriciate your time.", "2308121": "Hi @conjuring92 I was expecting I will have a submission score error. Could you please look at this question [here](https://www.kaggle.com/competitions/benetech-making-graphs-accessible/discussion/418077)and tell me if I have something wrong? I highly appriciate your time.", "2308145": "I'm not quite sure why you're getting the scoring error. Maybe instead of `'chart_type': ' '` use `'chart_type': 'line'` (or any other possible types) inside the exception block...", "2308154": "conjuring92 , thank you, could you please also tell me what I should fill the x_values and y_values in case of exception to avoid this nightmare error?", "2308172": "I think your current exception handling for x_values and y_values are okay", "2308180": "conjuring92 you so much. I hope you the best :)" }, "source": "meta" }