{
  "id": 23571,
  "title": "Error in prediction: New factor levels not present in the training data",
  "url": "/competitions/bosch-production-line-performance/discussion/23571",
  "author_name": "ZohairZahid",
  "post_date": "2016-09-11T13:51:23.327000",
  "votes": 0,
  "comment_count": 7,
  "views": 1190,
  "content": "<p>I am using randomforest in R. the train and test data consists of categorical and numerical data. I have tried alot of techniques to solve this error but was unsuccessful. Perhaps someone who has has handled this error in this challenge can help. </p>\n\n<p>Following is the script that I tried to use to handle the new factor levels . But it did not work.</p>\n\n<pre><code>myfunction &lt;- function(x){\n  #levels(xtest[,x]) %in% levels(xtrain[,x])\n  levels(xtest[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n  #xtest[,x] &lt;- factor(xtest[,x],levels(xtrain[,x]))\n}\n\nlapply(x,&quot;myfunction&quot;)\n\nError in predict.randomForest(fit, xtest) : \n  New factor levels not present in the training data\n</code></pre>",
  "messages": [
    {
      "id": 135175,
      "postDate": "2016-09-11T20:19:03.303Z",
      "content": "<p>John this also did not work, I tried it.\n[quote=JohnM;135166]</p>\n\n<p>Zohair - would something like this work? </p>\n\n<pre><code>for (f in names(train)) {\n   if (class(train[[f]])==&quot;character&quot;) {\n      levels &lt;- unique(c(train[[f]], test[[f]]))\n      train[[f]] &lt;- as.integer(factor(train[[f]], levels=levels))\n      test[[f]]  &lt;- as.integer(factor(test[[f]],  levels=levels))\n   }\n }\n</code></pre>\n\n<p>I use it to convert data table features and I believe it also works on regular data frames.</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "John this also did not work, I tried it.\r\n[quote=JohnM;135166]\r\n\r\nZohair - would something like this work? \r\n\r\n    for (f in names(train)) {\r\n       if (class(train[[f]])==\"character\") {\r\n          levels <- unique(c(train[[f]], test[[f]]))\r\n          train[[f]] <- as.integer(factor(train[[f]], levels=levels))\r\n          test[[f]]  <- as.integer(factor(test[[f]],  levels=levels))\r\n       }\r\n     }\r\n\r\nI use it to convert data table features and I believe it also works on regular data frames.\r\n\r\n[/quote]\r\n"
    },
    {
      "id": 135174,
      "postDate": "2016-09-11T20:10:01.267Z",
      "content": "<p>I tried this: </p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>as well but did not work. I am using these parameters:</p>\n\n<pre><code>fit &lt;- randomForest(as.factor(ytrain)~.,\n                    data=xtrain, \n                    proximity=TRUE, \n                    ntree=100)\n</code></pre>\n\n<p>Are these correct\n[quote=MathandBikes;135154]</p>\n\n<p>The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.</p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>[/quote]</p>",
      "rawMarkdown": "I tried this: \r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\n as well but did not work. I am using these parameters:\r\n\r\n    fit <- randomForest(as.factor(ytrain)~.,\r\n                        data=xtrain, \r\n                        proximity=TRUE, \r\n                        ntree=100)\r\n\r\nAre these correct\r\n[quote=MathandBikes;135154]\r\n\r\nThe problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.\r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\n[/quote]\r\n\r\n"
    },
    {
      "id": 135166,
      "postDate": "2016-09-11T19:12:49.107Z",
      "content": "<p>Zohair - would something like this work? </p>\n\n<pre><code>for (f in names(train)) {\n   if (class(train[[f]])==&quot;character&quot;) {\n      levels &lt;- unique(c(train[[f]], test[[f]]))\n      train[[f]] &lt;- as.integer(factor(train[[f]], levels=levels))\n      test[[f]]  &lt;- as.integer(factor(test[[f]],  levels=levels))\n   }\n }\n</code></pre>\n\n<p>I use it to convert data table features and I believe it also works on regular data frames.</p>",
      "rawMarkdown": "Zohair - would something like this work? \r\n\r\n    for (f in names(train)) {\r\n       if (class(train[[f]])==\"character\") {\r\n          levels <- unique(c(train[[f]], test[[f]]))\r\n          train[[f]] <- as.integer(factor(train[[f]], levels=levels))\r\n          test[[f]]  <- as.integer(factor(test[[f]],  levels=levels))\r\n       }\r\n     }\r\n\r\nI use it to convert data table features and I believe it also works on regular data frames."
    },
    {
      "id": 135154,
      "postDate": "2016-09-11T18:24:45.227Z",
      "content": "<p>The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.</p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>",
      "rawMarkdown": "The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.\r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))"
    },
    {
      "id": 135149,
      "postDate": "2016-09-11T17:12:18.353Z",
      "content": "<p>If I try this statement: </p>\n\n<pre><code>levels(xtest[,x]) &lt;- intersect(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>Error:</p>\n\n<pre><code>in `levels&lt;-.factor`(`*tmp*`, value = NULL) : \n  number of levels differs\n</code></pre>",
      "rawMarkdown": "If I try this statement: \r\n\r\n    levels(xtest[,x]) <- intersect(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\nError:\r\n\r\n    in `levels<-.factor`(`*tmp*`, value = NULL) : \r\n      number of levels differs\r\n\r\n"
    },
    {
      "id": 135138,
      "postDate": "2016-09-11T14:39:08.457Z",
      "content": "<p>Try to use intersect instead of union</p>",
      "rawMarkdown": "Try to use intersect instead of union"
    },
    {
      "id": 135137,
      "postDate": "2016-09-11T13:51:23.327Z",
      "content": "<p>I am using randomforest in R. the train and test data consists of categorical and numerical data. I have tried alot of techniques to solve this error but was unsuccessful. Perhaps someone who has has handled this error in this challenge can help. </p>\n\n<p>Following is the script that I tried to use to handle the new factor levels . But it did not work.</p>\n\n<pre><code>myfunction &lt;- function(x){\n  #levels(xtest[,x]) %in% levels(xtrain[,x])\n  levels(xtest[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n  #xtest[,x] &lt;- factor(xtest[,x],levels(xtrain[,x]))\n}\n\nlapply(x,&quot;myfunction&quot;)\n\nError in predict.randomForest(fit, xtest) : \n  New factor levels not present in the training data\n</code></pre>",
      "rawMarkdown": "I am using randomforest in R. the train and test data consists of categorical and numerical data. I have tried alot of techniques to solve this error but was unsuccessful. Perhaps someone who has has handled this error in this challenge can help. \r\n\r\nFollowing is the script that I tried to use to handle the new factor levels . But it did not work.\r\n\r\n\r\n    myfunction <- function(x){\r\n      #levels(xtest[,x]) %in% levels(xtrain[,x])\r\n      levels(xtest[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n      #xtest[,x] <- factor(xtest[,x],levels(xtrain[,x]))\r\n    }\r\n\r\n    lapply(x,\"myfunction\")\r\n\r\n    Error in predict.randomForest(fit, xtest) : \r\n      New factor levels not present in the training data\r\n\r\n"
    },
    {
      "id": 135179,
      "postDate": "2016-09-11T20:43:43.347Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 135175,
      "author_name": "ZohairZahid",
      "author_url": "",
      "post_date": "2016-09-11T20:19:03.303000",
      "content": "<p>John this also did not work, I tried it.\n[quote=JohnM;135166]</p>\n\n<p>Zohair - would something like this work? </p>\n\n<pre><code>for (f in names(train)) {\n   if (class(train[[f]])==&quot;character&quot;) {\n      levels &lt;- unique(c(train[[f]], test[[f]]))\n      train[[f]] &lt;- as.integer(factor(train[[f]], levels=levels))\n      test[[f]]  &lt;- as.integer(factor(test[[f]],  levels=levels))\n   }\n }\n</code></pre>\n\n<p>I use it to convert data table features and I believe it also works on regular data frames.</p>\n\n<p>[/quote]</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135174,
      "author_name": "ZohairZahid",
      "author_url": "",
      "post_date": "2016-09-11T20:10:01.267000",
      "content": "<p>I tried this: </p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>as well but did not work. I am using these parameters:</p>\n\n<pre><code>fit &lt;- randomForest(as.factor(ytrain)~.,\n                    data=xtrain, \n                    proximity=TRUE, \n                    ntree=100)\n</code></pre>\n\n<p>Are these correct\n[quote=MathandBikes;135154]</p>\n\n<p>The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.</p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>[/quote]</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135166,
      "author_name": "JohnM",
      "author_url": "",
      "post_date": "2016-09-11T19:12:49.107000",
      "content": "<p>Zohair - would something like this work? </p>\n\n<pre><code>for (f in names(train)) {\n   if (class(train[[f]])==&quot;character&quot;) {\n      levels &lt;- unique(c(train[[f]], test[[f]]))\n      train[[f]] &lt;- as.integer(factor(train[[f]], levels=levels))\n      test[[f]]  &lt;- as.integer(factor(test[[f]],  levels=levels))\n   }\n }\n</code></pre>\n\n<p>I use it to convert data table features and I believe it also works on regular data frames.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135154,
      "author_name": "MathandBikes",
      "author_url": "",
      "post_date": "2016-09-11T18:24:45.227000",
      "content": "<p>The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.</p>\n\n<pre><code>levels(xtrain[,x]) &lt;- union(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135149,
      "author_name": "ZohairZahid",
      "author_url": "",
      "post_date": "2016-09-11T17:12:18.353000",
      "content": "<p>If I try this statement: </p>\n\n<pre><code>levels(xtest[,x]) &lt;- intersect(levels(xtest[,x]),levels(xtrain[,x]))\n</code></pre>\n\n<p>Error:</p>\n\n<pre><code>in `levels&lt;-.factor`(`*tmp*`, value = NULL) : \n  number of levels differs\n</code></pre>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135138,
      "author_name": "Massimiliano",
      "author_url": "",
      "post_date": "2016-09-11T14:39:08.457000",
      "content": "<p>Try to use intersect instead of union</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 135179,
      "author_name": "",
      "author_url": "",
      "post_date": "2016-09-11T20:43:43.347000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "135175": "John this also did not work, I tried it.\r\n[quote=JohnM;135166]\r\n\r\nZohair - would something like this work? \r\n\r\n    for (f in names(train)) {\r\n       if (class(train[[f]])==\"character\") {\r\n          levels <- unique(c(train[[f]], test[[f]]))\r\n          train[[f]] <- as.integer(factor(train[[f]], levels=levels))\r\n          test[[f]]  <- as.integer(factor(test[[f]],  levels=levels))\r\n       }\r\n     }\r\n\r\nI use it to convert data table features and I believe it also works on regular data frames.\r\n\r\n[/quote]\r\n",
    "135174": "I tried this: \r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\n as well but did not work. I am using these parameters:\r\n\r\n    fit <- randomForest(as.factor(ytrain)~.,\r\n                        data=xtrain, \r\n                        proximity=TRUE, \r\n                        ntree=100)\r\n\r\nAre these correct\r\n[quote=MathandBikes;135154]\r\n\r\nThe problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.\r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\n[/quote]\r\n\r\n",
    "135166": "Zohair - would something like this work? \r\n\r\n    for (f in names(train)) {\r\n       if (class(train[[f]])==\"character\") {\r\n          levels <- unique(c(train[[f]], test[[f]]))\r\n          train[[f]] <- as.integer(factor(train[[f]], levels=levels))\r\n          test[[f]]  <- as.integer(factor(test[[f]],  levels=levels))\r\n       }\r\n     }\r\n\r\nI use it to convert data table features and I believe it also works on regular data frames.",
    "135154": "The problem is that your test data has levels of the factor that your training data doesn't have.  I'm pretty sure you just need to put the union of the levels in your training set, not your test set.  Hope this works for you.\r\n\r\n    levels(xtrain[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))",
    "135149": "If I try this statement: \r\n\r\n    levels(xtest[,x]) <- intersect(levels(xtest[,x]),levels(xtrain[,x]))\r\n\r\nError:\r\n\r\n    in `levels<-.factor`(`*tmp*`, value = NULL) : \r\n      number of levels differs\r\n\r\n",
    "135138": "Try to use intersect instead of union",
    "135137": "I am using randomforest in R. the train and test data consists of categorical and numerical data. I have tried alot of techniques to solve this error but was unsuccessful. Perhaps someone who has has handled this error in this challenge can help. \r\n\r\nFollowing is the script that I tried to use to handle the new factor levels . But it did not work.\r\n\r\n\r\n    myfunction <- function(x){\r\n      #levels(xtest[,x]) %in% levels(xtrain[,x])\r\n      levels(xtest[,x]) <- union(levels(xtest[,x]),levels(xtrain[,x]))\r\n      #xtest[,x] <- factor(xtest[,x],levels(xtrain[,x]))\r\n    }\r\n\r\n    lapply(x,\"myfunction\")\r\n\r\n    Error in predict.randomForest(fit, xtest) : \r\n      New factor levels not present in the training data\r\n\r\n",
    "135179": ""
  }
}