Hello, This is my first time participating in a kaggle competition and I'm having trouble submitting my result. I made my model using gbm and made a prediction table like below.
\n\npred_oob = predict(object = model_gbm, newdata = te_df, n.trees = opt_ntree_oob, type = 'response')\npred_oob = ifelse(pred_oob<0, 0, pred_oob)\n\ntest$Predicted = pred_oob \nmysub = test %>%\n select(fullVisitorId, Predicted)\n\nmysub = mysub %>%\n group_by(fullVisitorId) %>%\n summarise(PredictedLogRevenue = round(sum(Predicted), 5)) \n\nwrite.csv(mysub, file = 'mysub_gbm.csv', row.names = F)\nWhenever I tried to submit the file, I got the 'fail' message saying ...
\n\nERROR: Unable to find 614143 required key values in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000000259678714014' in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000049363351866189' in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000053049821714864' in the 'fullVisitorId' column\nSo I inspected my submission data again with that values, but...
\n\nerror = c(614143, 259678714014, 49363351866189, 53049821714864, 59488412965267, 8584037063378, 91131414287111, 117255350596610, 118334805178127, 130646294093000)\n mysub[mysub$fullVisitorId %in% error, ]\nBut there was no empty row...I have no clue from what point I made mistake....Please help!
", "messages": [ { "id": 394967, "postDate": "2018-09-27T18:23:39.267Z", "content": "Make sure when you initially read the CSV data to import it as character. I had this issue too, and if you import it without specifying the fullVisitorId field is character, R will assume numeric and remove the leading zeros from the ID. Something like this would work when you initially read the data:
\n\ntrain <- read.csv('train.csv', colClasses=c(fullVisitorId = 'character'))\nOkay. Now I understood what was wrong...but when I save it as csv file using write.csv(), then the fullVisitorId column becomes numeric again. So the same problem happens. What should I do?
", "rawMarkdown": "Okay. Now I understood what was wrong...but when I save it as csv file using write.csv(), then the fullVisitorId column becomes numeric again. So the same problem happens. What should I do? " }, { "id": 395005, "postDate": "2018-09-27T20:14:37.817Z", "content": "How are you confirming it's numeric after you save it? If you're using software like Excel to view the file after you save it, Excel usually does some undesired auto-formatting. I recommend trying to submit after you write.csv and see if the error persists.
So you mean.. I have to run the code on kaggle and submit, not from my local Rstudio and just submit the file?
", "rawMarkdown": "So you mean.. I have to run the code on kaggle and submit, not from my local Rstudio and just submit the file? " }, { "id": 395048, "postDate": "2018-09-27T21:40:30.093Z", "content": "No, you can run local and submit the CSV. I'm just trying to understand why you think write.csv is causing fullVisitorId to become numeric.
It's because I got the same error message. I read the test set and then changed fullVisitorId into character.
Have you instead tried train <- read.csv('train.csv', colClasses=list(character='fullVisitorId'))
test <- read.csv('test.csv', colClasses=list(character='fullVisitorId'))
Or, you can do it how I like to do it, which makes it so I never have to specify it because it automatically detects it as a string.
\n\nlibrary(data.table)\ntrain <- fread('train.csv')\ntest <- fread('test.csv')\nOkay, now I got it!! The problem was that I was reading my preprocessed file whose fullVisitorId had already been numeric, so losing all leading zeros.
\nBoth fread() and read.csv('test.csv', colClasses = ) work well! Thank you soooo much everyone :-)
Hello, This is my first time participating in a kaggle competition and I'm having trouble submitting my result. I made my model using gbm and made a prediction table like below.
\n\npred_oob = predict(object = model_gbm, newdata = te_df, n.trees = opt_ntree_oob, type = 'response')\npred_oob = ifelse(pred_oob<0, 0, pred_oob)\n\ntest$Predicted = pred_oob \nmysub = test %>%\n select(fullVisitorId, Predicted)\n\nmysub = mysub %>%\n group_by(fullVisitorId) %>%\n summarise(PredictedLogRevenue = round(sum(Predicted), 5)) \n\nwrite.csv(mysub, file = 'mysub_gbm.csv', row.names = F)\nWhenever I tried to submit the file, I got the 'fail' message saying ...
\n\nERROR: Unable to find 614143 required key values in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000000259678714014' in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000049363351866189' in the 'fullVisitorId' column\nERROR: Unable to find the required key value '0000053049821714864' in the 'fullVisitorId' column\nSo I inspected my submission data again with that values, but...
\n\nerror = c(614143, 259678714014, 49363351866189, 53049821714864, 59488412965267, 8584037063378, 91131414287111, 117255350596610, 118334805178127, 130646294093000)\n mysub[mysub$fullVisitorId %in% error, ]\nBut there was no empty row...I have no clue from what point I made mistake....Please help!
", "rawMarkdown": "Hello, This is my first time participating in a kaggle competition and I'm having trouble submitting my result. I made my model using gbm and made a prediction table like below.\n\n pred_oob = predict(object = model_gbm, newdata = te_df, n.trees = opt_ntree_oob, type = 'response')\n pred_oob = ifelse(pred_oob<0, 0, pred_oob)\n \n test$Predicted = pred_oob \n mysub = test %>%\n select(fullVisitorId, Predicted)\n \n mysub = mysub %>%\n group_by(fullVisitorId) %>%\n summarise(PredictedLogRevenue = round(sum(Predicted), 5)) \n \n write.csv(mysub, file = 'mysub_gbm.csv', row.names = F)\n\nWhenever I tried to submit the file, I got the 'fail' message saying ...\n\n ERROR: Unable to find 614143 required key values in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000000259678714014' in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000049363351866189' in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000053049821714864' in the 'fullVisitorId' column\n\nSo I inspected my submission data again with that values, but...\n\n error = c(614143, 259678714014, 49363351866189, 53049821714864, 59488412965267, 8584037063378, 91131414287111, 117255350596610, 118334805178127, 130646294093000)\n mysub[mysub$fullVisitorId %in% error, ]\n\nBut there was no empty row...I have no clue from what point I made mistake....Please help!\n", "votes": 2 }, { "id": 395132, "postDate": "2018-09-28T02:34:09.767Z", "content": "I believe your fullVisitorId is numeric instead of character, which drops the leading zeros. Try the following:
\n\nsubmission_sample <- read.csv('../input/sample_submission.csv', stringsAsFactors=F, colClasses=\"character\")\n\nsubmission <- data.frame(fullVisitorId=submission_sample$fullVisitorId, PredictedLogRevenue=mysub$PredictedLogRevenue)\n\nwrite.csv(submission, file = paste0('../output/your_submission.csv'), row.names=F, quote=F)\nThanks yilisg. You're right. It was because my fullVistorId was numeric. Now it perfectly works.
yes, but when predicting, doesn't the visitor id needs to be converted to float; the leading zeroes get deleted from some of the Id and i am unable to submit. Can someone help pls?
", "rawMarkdown": "yes, but when predicting, doesn't the visitor id needs to be converted to float; the leading zeroes get deleted from some of the Id and i am unable to submit. Can someone help pls?\n" }, { "id": 409249, "postDate": "2018-10-24T02:04:24.820Z", "content": "You don't need to input Id column when you predict. You just make predictions and bind the values to the test set.
", "rawMarkdown": "You don't need to input Id column when you predict. You just make predictions and bind the values to the test set. " } ] }, { "id": 395125, "postDate": "2018-09-28T02:20:12.350Z", "content": "I am actually also having the same issue submitting, the fullVisitorId is red as character using the method colClasses=c(fullVisitorId = 'character'). So getting 714167 rows instead of 614143 predictions.
", "rawMarkdown": "I am actually also having the same issue submitting, the fullVisitorId is red as character using the method colClasses=c(fullVisitorId = 'character'). So getting 714167 rows instead of 614143 predictions.", "replies": [ { "id": 395137, "postDate": "2018-09-28T02:40:16.440Z", "content": "Hi, submission_sample should be 617242 x 2 if you run my code above. Not sure how you got 714167, but if you are reading test csv it should contain 804684 rows (of transactions or visits), where a single fullVisitorId could have multiple visits so you'll need to aggregate the sum of the revenues before submitting. You can verify this easily that there are 617242 unique fullVisitorId in test as well.
", "rawMarkdown": "Hi, submission_sample should be 617242 x 2 if you run my code above. Not sure how you got 714167, but if you are reading test csv it should contain 804684 rows (of transactions or visits), where a single fullVisitorId could have multiple visits so you'll need to aggregate the sum of the revenues before submitting. You can verify this easily that there are 617242 unique fullVisitorId in test as well.\n\n" } ] }, { "id": 395104, "postDate": "2018-09-28T01:21:43.560Z", "content": "Jean, you might try using write_csv from the readr package, or fwrite from the data.table package. Their parsers might work better. Plus, both are faster than base R's write.csv (especially fwrite).
", "rawMarkdown": "Jean, you might try using write_csv from the readr package, or fwrite from the data.table package. Their parsers might work better. Plus, both are faster than base R's write.csv (especially fwrite).", "replies": [ { "id": 395124, "postDate": "2018-09-28T02:17:50.607Z", "content": "Thanks John. write_csv() works much faster like you said but sadly.. no difference. \nI think I'm missing something very basic.... have no idea.
", "rawMarkdown": "Thanks John. write_csv() works much faster like you said but sadly.. no difference. \nI think I'm missing something very basic.... have no idea. " } ] }, { "id": 395090, "postDate": "2018-09-28T00:34:21.187Z", "content": "have you tried check fullVisitorId 0000000259678714014 in your csv files??
Yes, I checked every number that appears from the error message like below, and there were no empty values in mysub.
mysub[mysub$fullVisitorId == 0000000259678714014, ]
", "rawMarkdown": "Yes, I checked every number that appears from the error message like below, and there were no empty values in `mysub`. \n\nmysub[mysub$fullVisitorId == 0000000259678714014, ]" } ] }, { "id": 394945, "postDate": "2018-09-27T17:33:25.940Z", "content": "Leading zeros may be an issue. Is the fullVisitorId field in your data a character column?
", "rawMarkdown": "Leading zeros may be an issue. Is the fullVisitorId field in your data a character column?", "replies": [ { "id": 394963, "postDate": "2018-09-27T18:20:06.567Z", "content": "No, it's numeric. Do you mean the predicted values containing zeros are the problem?
", "rawMarkdown": "No, it's numeric. Do you mean the predicted values containing zeros are the problem?" } ] }, { "id": 394962, "postDate": "2018-09-27T18:19:19.367Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 394967, "author_name": "Nick", "author_url": "", "post_date": "2018-09-27T18:23:39.267000", "content": "Make sure when you initially read the CSV data to import it as character. I had this issue too, and if you import it without specifying the fullVisitorId field is character, R will assume numeric and remove the leading zeros from the ID. Something like this would work when you initially read the data:
\n\ntrain <- read.csv('train.csv', colClasses=c(fullVisitorId = 'character'))\nOkay. Now I understood what was wrong...but when I save it as csv file using write.csv(), then the fullVisitorId column becomes numeric again. So the same problem happens. What should I do?
", "votes": 0, "replies": [] }, { "id": 395005, "author_name": "Nick", "author_url": "", "post_date": "2018-09-27T20:14:37.817000", "content": "How are you confirming it's numeric after you save it? If you're using software like Excel to view the file after you save it, Excel usually does some undesired auto-formatting. I recommend trying to submit after you write.csv and see if the error persists.
So you mean.. I have to run the code on kaggle and submit, not from my local Rstudio and just submit the file?
", "votes": 0, "replies": [] }, { "id": 395048, "author_name": "Nick", "author_url": "", "post_date": "2018-09-27T21:40:30.093000", "content": "No, you can run local and submit the CSV. I'm just trying to understand why you think write.csv is causing fullVisitorId to become numeric.
It's because I got the same error message. I read the test set and then changed fullVisitorId into character.
Have you instead tried train <- read.csv('train.csv', colClasses=list(character='fullVisitorId'))
test <- read.csv('test.csv', colClasses=list(character='fullVisitorId'))
Or, you can do it how I like to do it, which makes it so I never have to specify it because it automatically detects it as a string.
\n\nlibrary(data.table)\ntrain <- fread('train.csv')\ntest <- fread('test.csv')\nOkay, now I got it!! The problem was that I was reading my preprocessed file whose fullVisitorId had already been numeric, so losing all leading zeros.
\nBoth fread() and read.csv('test.csv', colClasses = ) work well! Thank you soooo much everyone :-)
I believe your fullVisitorId is numeric instead of character, which drops the leading zeros. Try the following:
\n\nsubmission_sample <- read.csv('../input/sample_submission.csv', stringsAsFactors=F, colClasses=\"character\")\n\nsubmission <- data.frame(fullVisitorId=submission_sample$fullVisitorId, PredictedLogRevenue=mysub$PredictedLogRevenue)\n\nwrite.csv(submission, file = paste0('../output/your_submission.csv'), row.names=F, quote=F)\nThanks yilisg. You're right. It was because my fullVistorId was numeric. Now it perfectly works.
yes, but when predicting, doesn't the visitor id needs to be converted to float; the leading zeroes get deleted from some of the Id and i am unable to submit. Can someone help pls?
", "votes": 0, "replies": [] }, { "id": 409249, "author_name": "Jean", "author_url": "", "post_date": "2018-10-24T02:04:24.820000", "content": "You don't need to input Id column when you predict. You just make predictions and bind the values to the test set.
", "votes": 0, "replies": [] } ] }, { "id": 395125, "author_name": "Alphonse Okossi", "author_url": "", "post_date": "2018-09-28T02:20:12.350000", "content": "I am actually also having the same issue submitting, the fullVisitorId is red as character using the method colClasses=c(fullVisitorId = 'character'). So getting 714167 rows instead of 614143 predictions.
", "votes": 0, "replies": [ { "id": 395137, "author_name": "yilisg", "author_url": "", "post_date": "2018-09-28T02:40:16.440000", "content": "Hi, submission_sample should be 617242 x 2 if you run my code above. Not sure how you got 714167, but if you are reading test csv it should contain 804684 rows (of transactions or visits), where a single fullVisitorId could have multiple visits so you'll need to aggregate the sum of the revenues before submitting. You can verify this easily that there are 617242 unique fullVisitorId in test as well.
", "votes": 0, "replies": [] } ] }, { "id": 395104, "author_name": "JohnM", "author_url": "", "post_date": "2018-09-28T01:21:43.560000", "content": "Jean, you might try using write_csv from the readr package, or fwrite from the data.table package. Their parsers might work better. Plus, both are faster than base R's write.csv (especially fwrite).
", "votes": 0, "replies": [ { "id": 395124, "author_name": "Jean", "author_url": "", "post_date": "2018-09-28T02:17:50.607000", "content": "Thanks John. write_csv() works much faster like you said but sadly.. no difference. \nI think I'm missing something very basic.... have no idea.
", "votes": 0, "replies": [] } ] }, { "id": 395090, "author_name": "choco", "author_url": "", "post_date": "2018-09-28T00:34:21.187000", "content": "have you tried check fullVisitorId 0000000259678714014 in your csv files??
Yes, I checked every number that appears from the error message like below, and there were no empty values in mysub.
mysub[mysub$fullVisitorId == 0000000259678714014, ]
", "votes": 0, "replies": [] } ] }, { "id": 394945, "author_name": "Dmitriy Guller", "author_url": "", "post_date": "2018-09-27T17:33:25.940000", "content": "Leading zeros may be an issue. Is the fullVisitorId field in your data a character column?
", "votes": 0, "replies": [ { "id": 394963, "author_name": "Jean", "author_url": "", "post_date": "2018-09-27T18:20:06.567000", "content": "No, it's numeric. Do you mean the predicted values containing zeros are the problem?
", "votes": 0, "replies": [] } ] }, { "id": 394962, "author_name": "", "author_url": "", "post_date": "2018-09-27T18:19:19.367000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "394967": "Make sure when you initially read the CSV data to import it as character. I had this issue too, and if you import it without specifying the fullVisitorId field is character, R will assume numeric and remove the leading zeros from the ID. Something like this would work when you initially read the data:\n\n train <- read.csv('train.csv', colClasses=c(fullVisitorId = 'character'))", "394937": "Hello, This is my first time participating in a kaggle competition and I'm having trouble submitting my result. I made my model using gbm and made a prediction table like below.\n\n pred_oob = predict(object = model_gbm, newdata = te_df, n.trees = opt_ntree_oob, type = 'response')\n pred_oob = ifelse(pred_oob<0, 0, pred_oob)\n \n test$Predicted = pred_oob \n mysub = test %>%\n select(fullVisitorId, Predicted)\n \n mysub = mysub %>%\n group_by(fullVisitorId) %>%\n summarise(PredictedLogRevenue = round(sum(Predicted), 5)) \n \n write.csv(mysub, file = 'mysub_gbm.csv', row.names = F)\n\nWhenever I tried to submit the file, I got the 'fail' message saying ...\n\n ERROR: Unable to find 614143 required key values in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000000259678714014' in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000049363351866189' in the 'fullVisitorId' column\n ERROR: Unable to find the required key value '0000053049821714864' in the 'fullVisitorId' column\n\nSo I inspected my submission data again with that values, but...\n\n error = c(614143, 259678714014, 49363351866189, 53049821714864, 59488412965267, 8584037063378, 91131414287111, 117255350596610, 118334805178127, 130646294093000)\n mysub[mysub$fullVisitorId %in% error, ]\n\nBut there was no empty row...I have no clue from what point I made mistake....Please help!\n", "395132": "I believe your fullVisitorId is numeric instead of character, which drops the leading zeros. Try the following:\n\n submission_sample <- read.csv('../input/sample_submission.csv', stringsAsFactors=F, colClasses=\"character\")\n \n submission <- data.frame(fullVisitorId=submission_sample$fullVisitorId, PredictedLogRevenue=mysub$PredictedLogRevenue)\n \n write.csv(submission, file = paste0('../output/your_submission.csv'), row.names=F, quote=F)", "395125": "I am actually also having the same issue submitting, the fullVisitorId is red as character using the method colClasses=c(fullVisitorId = 'character'). So getting 714167 rows instead of 614143 predictions.", "395104": "Jean, you might try using write_csv from the readr package, or fwrite from the data.table package. Their parsers might work better. Plus, both are faster than base R's write.csv (especially fwrite).", "395090": "have you tried check fullVisitorId `0000000259678714014` in your csv files??", "394945": "Leading zeros may be an issue. Is the fullVisitorId field in your data a character column?", "394962": "" } }