{ "id": 232540, "title": "Submission Scoring Error.", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/232540", "author_name": "Alexey Savin", "post_date": "2021-04-14T08:39:11.631000", "votes": 1, "comment_count": 11, "views": 0, "content": "

Hi everyone!

\n

Could anyone please explain what's wrong?

\n

Notebook running without any errors. Submission file created like in previous versions.

\n

But after 30-40 min of submitting process status changing to \"Submission Scoring Error\"
\nNo additional details provided for this error.

\n

or

\n

What possible reasons of Submission Scoring Error.
\nIf notebook running without any errors and submission file created in correct format?

", "messages": [ { "id": 1276672, "postDate": "2021-04-17T20:33:58.140Z", "content": "

Ok, so just fixed my submission scoring error.
\nIt was due to not having all the test files in the Id column of submission.csv
\nSo, always initialize your submission dataframe with all the test files (and prediction strings with empty string). Then fill in the predictionstring column as you go on!
\nAnother thing to keep in mind is, values in Id column does not contain .json extension.

\n

Hope that saves you some submission attempts!

\n

In python:

\n

submission = pd.Dataframe(data= {'Id' : ['test_file1, 'test_file2'…..], 'PredictionString' : ['', '', …]})

\n

TODO: fill in the PredictionString as you find it

\n

submission.to_csv('submission.csv', index=False)

", "rawMarkdown": "Ok, so just fixed my submission scoring error. \nIt was due to not having all the test files in the **Id** column of submission.csv\nSo, always initialize your submission dataframe with all the test files (and prediction strings with empty string). Then fill in the predictionstring column as you go on! \nAnother thing to keep in mind is, values in **Id** column does not contain .json extension.\n\nHope that saves you some submission attempts!\n\nIn python:\n\nsubmission = pd.Dataframe(data= {'Id' : ['test_file1, 'test_file2'.....], 'PredictionString' : ['', '', ...]})\n# TODO: fill in the PredictionString as you find it\nsubmission.to_csv('submission.csv', index=False)\n", "votes": 1 }, { "id": 1273314, "postDate": "2021-04-14T08:39:11.630Z", "content": "

Hi everyone!

\n

Could anyone please explain what's wrong?

\n

Notebook running without any errors. Submission file created like in previous versions.

\n

But after 30-40 min of submitting process status changing to \"Submission Scoring Error\"
\nNo additional details provided for this error.

\n

or

\n

What possible reasons of Submission Scoring Error.
\nIf notebook running without any errors and submission file created in correct format?

", "rawMarkdown": "Hi everyone!\n\nCould anyone please explain what's wrong?\n\nNotebook running without any errors. Submission file created like in previous versions.\n\nBut after 30-40 min of submitting process status changing to \"Submission Scoring Error\"\nNo additional details provided for this error.\n\nor\n\nWhat possible reasons of Submission Scoring Error.\nIf notebook running without any errors and submission file created in correct format?\n\n", "votes": 1 }, { "id": 1275195, "postDate": "2021-04-16T04:19:35.337Z", "content": "

Hey guys. @whywhyc , @alexeisavin , @poluminous . I ran out of tries again for the day.

\n

I just noticed that when I read the submission csv, the prediction column is numbers instead of strings. Could that be the problem?

\n

submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv', index_col=0)
\nprint(submission_df.dtypes)
\nsubmission_df['PredictionString'] = submission_df['PredictionString'].astype('string')
\nprint(submission_df.dtypes)

\n

The above lines return:

\n

PredictionString float64
\ndtype: object
\nPredictionString string
\ndtype: object

\n

Can you guys try and see if changing to string makes this thing work?. I really would like to start seeing the results of what I've been doing tomorrow.

", "rawMarkdown": "Hey guys. @whywhyc , @alexeisavin , @poluminous . I ran out of tries again for the day.\n\nI just noticed that when I read the submission csv, the prediction column is numbers instead of strings. Could that be the problem?\n\n`submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv', index_col=0)`\n`print(submission_df.dtypes)`\n`submission_df['PredictionString'] = submission_df['PredictionString'].astype('string')`\n`print(submission_df.dtypes)`\n\nThe above lines return:\n\nPredictionString **float64**\ndtype: object\nPredictionString **string**\ndtype: object\n\nCan you guys try and see if changing to string makes this thing work?. I really would like to start seeing the results of what I've been doing tomorrow.\n\n", "replies": [ { "id": 1275379, "postDate": "2021-04-16T08:29:19.663Z", "content": "

I've checked it, its really float64, but I'm not sure that's make any impact to scoring. When you will write your prediction_text(labels) to this column, it will have String type.
\nIn my case, when I processed exception which i described below, I started getting the score instead of submission scoring error.

", "rawMarkdown": "I've checked it, its really float64, but I'm not sure that's make any impact to scoring. When you will write your prediction_text(labels) to this column, it will have String type.\nIn my case, when I processed exception which i described below, I started getting the score instead of submission scoring error." } ] }, { "id": 1274748, "postDate": "2021-04-15T15:09:24.823Z", "content": "

I found reason, maybe :)
\ntesting atm

\n

The reason is .predict can't receive samples with zero amount of sample to predict.

\n

If you have sequences of .prediction , check, maybe one of them decide that's no data labels and then next one .predict receive zero samples to predict.

", "rawMarkdown": "I found reason, maybe :)\ntesting atm\n\nThe reason is .predict can't receive samples with zero amount of sample to predict.\n\nIf you have sequences of .prediction , check, maybe one of them decide that's no data labels and then next one .predict receive zero samples to predict.", "replies": [ { "id": 1275186, "postDate": "2021-04-16T03:48:23.053Z", "content": "

What is .predict? Do you mean the 'PredictionString' column?

", "rawMarkdown": "What is .predict? Do you mean the 'PredictionString' column?" }, { "id": 1275374, "postDate": "2021-04-16T08:20:13.877Z", "content": "

I'm talking about prediction method, not PredicitionString
\nmodel.fit(X_train,Y_train)
\nmodel.predict(X_test) <--- this one need something to predict.

\n

But in my case it was like
\nfor example:
\nprediction = model_1.predict(X_1)
\nmodel_2.predict(prediction). <--- this one failing, because first model decided, that's no datasets labels in X_1, and model_2.predict receiving nothing to predict and rising exception. But you can't see it, because it on hidden test data.

", "rawMarkdown": "I'm talking about prediction method, not PredicitionString\nmodel.fit(X_train,Y_train)\nmodel.predict(X_test) <--- this one need something to predict.\n\nBut in my case it was like\nfor example:\nprediction = model_1.predict(X_1) \nmodel_2.predict(prediction). <--- this one failing, because first model decided, that's no datasets labels in X_1, and model_2.predict receiving nothing to predict and rising exception. But you can't see it, because it on hidden test data." } ] }, { "id": 1273536, "postDate": "2021-04-14T12:56:27.627Z", "content": "

Hi - I share your pain … and I have no answer. My next 'fix' (have to wait for tomorrow) will be to limit the length of the prediction strings (my code doesn't separate them properly, but there's no point improving it until I know I can submit). My fear is that the testing data contains a feature that my code dislikes. Without an error statement I stand little chance of fixing it.

", "rawMarkdown": "Hi - I share your pain ... and I have no answer. My next 'fix' (have to wait for tomorrow) will be to limit the length of the prediction strings (my code doesn't separate them properly, but there's no point improving it until I know I can submit). My fear is that the testing data contains a feature that my code dislikes. Without an error statement I stand little chance of fixing it.\n\n", "replies": [ { "id": 1273951, "postDate": "2021-04-14T19:52:09.353Z", "content": "

I've been in the same boat for a few days. Yesterday I tried limiting the length to 200 characters and that didn't work. I also removed repeated solutions. Not too sure what to try tonight.

", "rawMarkdown": "I've been in the same boat for a few days. Yesterday I tried limiting the length to 200 characters and that didn't work. I also removed repeated solutions. Not too sure what to try tonight." }, { "id": 1275289, "postDate": "2021-04-16T06:51:06.253Z", "content": "

Hi. I've made some progress (I think). The 'Submission Scoring Error' appears to have been due to my code not finding an entity in a line where I identified a dataset usage - so the entry in the prediction string was '||'. Now I'm getting a different error because I use too much memory (but that's sortable) - I think there must be 8000+ papers in the test data.
\nSo things are moving - just not sure of the direction.
\nPeter.

", "rawMarkdown": "Hi. I've made some progress (I think). The 'Submission Scoring Error' appears to have been due to my code not finding an entity in a line where I identified a dataset usage - so the entry in the prediction string was '||'. Now I'm getting a different error because I use too much memory (but that's sortable) - I think there must be 8000+ papers in the test data.\nSo things are moving - just not sure of the direction.\nPeter.\n " }, { "id": 1276193, "postDate": "2021-04-17T08:19:08.767Z", "content": "

Hello. If it helps … the 'Submission Scoring Error' means that the csv file is not in the correct format. I found the errors (eventually) by running against 10,000 of the training papers and searching through the rows. The data in the 'test' folder is not enough to successfully validate the output.
\nI also got an out-of-memory error that was sorted by processing chunks of files at a time.
\nHopefully, this may help someone.

", "rawMarkdown": "Hello. If it helps ... the 'Submission Scoring Error' means that the csv file is not in the correct format. I found the errors (eventually) by running against 10,000 of the training papers and searching through the rows. The data in the 'test' folder is not enough to successfully validate the output. \nI also got an out-of-memory error that was sorted by processing chunks of files at a time. \nHopefully, this may help someone.\n " } ] }, { "id": 1274044, "postDate": "2021-04-15T00:00:20.797Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ], "comments": [ { "id": 1276672, "author_name": "MananG", "author_url": "", "post_date": "2021-04-17T20:33:58.140000", "content": "

Ok, so just fixed my submission scoring error.
\nIt was due to not having all the test files in the Id column of submission.csv
\nSo, always initialize your submission dataframe with all the test files (and prediction strings with empty string). Then fill in the predictionstring column as you go on!
\nAnother thing to keep in mind is, values in Id column does not contain .json extension.

\n

Hope that saves you some submission attempts!

\n

In python:

\n

submission = pd.Dataframe(data= {'Id' : ['test_file1, 'test_file2'…..], 'PredictionString' : ['', '', …]})

\n

TODO: fill in the PredictionString as you find it

\n

submission.to_csv('submission.csv', index=False)

", "votes": 1, "replies": [] }, { "id": 1275195, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-16T04:19:35.337000", "content": "

Hey guys. @whywhyc , @alexeisavin , @poluminous . I ran out of tries again for the day.

\n

I just noticed that when I read the submission csv, the prediction column is numbers instead of strings. Could that be the problem?

\n

submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv', index_col=0)
\nprint(submission_df.dtypes)
\nsubmission_df['PredictionString'] = submission_df['PredictionString'].astype('string')
\nprint(submission_df.dtypes)

\n

The above lines return:

\n

PredictionString float64
\ndtype: object
\nPredictionString string
\ndtype: object

\n

Can you guys try and see if changing to string makes this thing work?. I really would like to start seeing the results of what I've been doing tomorrow.

", "votes": 0, "replies": [ { "id": 1275379, "author_name": "Alexey Savin", "author_url": "", "post_date": "2021-04-16T08:29:19.663000", "content": "

I've checked it, its really float64, but I'm not sure that's make any impact to scoring. When you will write your prediction_text(labels) to this column, it will have String type.
\nIn my case, when I processed exception which i described below, I started getting the score instead of submission scoring error.

", "votes": 0, "replies": [] } ] }, { "id": 1274748, "author_name": "Alexey Savin", "author_url": "", "post_date": "2021-04-15T15:09:24.823000", "content": "

I found reason, maybe :)
\ntesting atm

\n

The reason is .predict can't receive samples with zero amount of sample to predict.

\n

If you have sequences of .prediction , check, maybe one of them decide that's no data labels and then next one .predict receive zero samples to predict.

", "votes": 0, "replies": [ { "id": 1275186, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-16T03:48:23.053000", "content": "

What is .predict? Do you mean the 'PredictionString' column?

", "votes": 0, "replies": [] }, { "id": 1275374, "author_name": "Alexey Savin", "author_url": "", "post_date": "2021-04-16T08:20:13.877000", "content": "

I'm talking about prediction method, not PredicitionString
\nmodel.fit(X_train,Y_train)
\nmodel.predict(X_test) <--- this one need something to predict.

\n

But in my case it was like
\nfor example:
\nprediction = model_1.predict(X_1)
\nmodel_2.predict(prediction). <--- this one failing, because first model decided, that's no datasets labels in X_1, and model_2.predict receiving nothing to predict and rising exception. But you can't see it, because it on hidden test data.

", "votes": 0, "replies": [] } ] }, { "id": 1273536, "author_name": "Peter Hughes", "author_url": "", "post_date": "2021-04-14T12:56:27.627000", "content": "

Hi - I share your pain … and I have no answer. My next 'fix' (have to wait for tomorrow) will be to limit the length of the prediction strings (my code doesn't separate them properly, but there's no point improving it until I know I can submit). My fear is that the testing data contains a feature that my code dislikes. Without an error statement I stand little chance of fixing it.

", "votes": 0, "replies": [ { "id": 1273951, "author_name": "Wilmer E. Henao", "author_url": "", "post_date": "2021-04-14T19:52:09.353000", "content": "

I've been in the same boat for a few days. Yesterday I tried limiting the length to 200 characters and that didn't work. I also removed repeated solutions. Not too sure what to try tonight.

", "votes": 0, "replies": [] }, { "id": 1275289, "author_name": "Peter Hughes", "author_url": "", "post_date": "2021-04-16T06:51:06.253000", "content": "

Hi. I've made some progress (I think). The 'Submission Scoring Error' appears to have been due to my code not finding an entity in a line where I identified a dataset usage - so the entry in the prediction string was '||'. Now I'm getting a different error because I use too much memory (but that's sortable) - I think there must be 8000+ papers in the test data.
\nSo things are moving - just not sure of the direction.
\nPeter.

", "votes": 0, "replies": [] }, { "id": 1276193, "author_name": "Peter Hughes", "author_url": "", "post_date": "2021-04-17T08:19:08.767000", "content": "

Hello. If it helps … the 'Submission Scoring Error' means that the csv file is not in the correct format. I found the errors (eventually) by running against 10,000 of the training papers and searching through the rows. The data in the 'test' folder is not enough to successfully validate the output.
\nI also got an out-of-memory error that was sorted by processing chunks of files at a time.
\nHopefully, this may help someone.

", "votes": 0, "replies": [] } ] }, { "id": 1274044, "author_name": "", "author_url": "", "post_date": "2021-04-15T00:00:20.797000", "content": "", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "1276672": "Ok, so just fixed my submission scoring error. \nIt was due to not having all the test files in the **Id** column of submission.csv\nSo, always initialize your submission dataframe with all the test files (and prediction strings with empty string). Then fill in the predictionstring column as you go on! \nAnother thing to keep in mind is, values in **Id** column does not contain .json extension.\n\nHope that saves you some submission attempts!\n\nIn python:\n\nsubmission = pd.Dataframe(data= {'Id' : ['test_file1, 'test_file2'.....], 'PredictionString' : ['', '', ...]})\n# TODO: fill in the PredictionString as you find it\nsubmission.to_csv('submission.csv', index=False)\n", "1273314": "Hi everyone!\n\nCould anyone please explain what's wrong?\n\nNotebook running without any errors. Submission file created like in previous versions.\n\nBut after 30-40 min of submitting process status changing to \"Submission Scoring Error\"\nNo additional details provided for this error.\n\nor\n\nWhat possible reasons of Submission Scoring Error.\nIf notebook running without any errors and submission file created in correct format?\n\n", "1275195": "Hey guys. @whywhyc , @alexeisavin , @poluminous . I ran out of tries again for the day.\n\nI just noticed that when I read the submission csv, the prediction column is numbers instead of strings. Could that be the problem?\n\n`submission_df = pd.read_csv('../input/coleridgeinitiative-show-us-the-data/sample_submission.csv', index_col=0)`\n`print(submission_df.dtypes)`\n`submission_df['PredictionString'] = submission_df['PredictionString'].astype('string')`\n`print(submission_df.dtypes)`\n\nThe above lines return:\n\nPredictionString **float64**\ndtype: object\nPredictionString **string**\ndtype: object\n\nCan you guys try and see if changing to string makes this thing work?. I really would like to start seeing the results of what I've been doing tomorrow.\n\n", "1274748": "I found reason, maybe :)\ntesting atm\n\nThe reason is .predict can't receive samples with zero amount of sample to predict.\n\nIf you have sequences of .prediction , check, maybe one of them decide that's no data labels and then next one .predict receive zero samples to predict.", "1273536": "Hi - I share your pain ... and I have no answer. My next 'fix' (have to wait for tomorrow) will be to limit the length of the prediction strings (my code doesn't separate them properly, but there's no point improving it until I know I can submit). My fear is that the testing data contains a feature that my code dislikes. Without an error statement I stand little chance of fixing it.\n\n", "1274044": "" } }