{
  "id": 10144,
  "title": "Problems when using sklearn lib",
  "url": "/competitions/avito-prohibited-content/discussion/10144",
  "author_name": "",
  "post_date": "2014-08-29T06:44:02.460Z",
  "votes": null,
  "comment_count": 9,
  "views": 2948,
  "content": "<p>I am using sklearn lib as my tool to create models to generate models for this competition. But I found some problems.</p>\n<ol>\n<li><span style=\"line-height: 1.4\">predicting the result is quite slow. I use 50w data to training the model which is quite fast but when I use the whole dataset, it is quite slow. And the function `predict_proba` is also quite slow... Is there some method to make it faster.</span></li>\n<li><span style=\"line-height: 1.4\">I use bag of word so there is a lot of features. The matrix is too big so it has to be a sparse matrix structure. But random forest and grandient boosting do not support sparse matrix. I tried some features selection method to make the matrix smaller but There is still `segmentation fault` error without any explaination.</span></li>\n</ol>\n<p><span style=\"line-height: 1.4\">By the way, I use a computer with about 25GB RAM to generate&nbsp;my models.</span></p>",
  "messages": [
    {
      "id": "52617",
      "postDate": "08/29/2014 06:44:02",
      "content": "<p>I am using sklearn lib as my tool to create models to generate models for this competition. But I found some problems.</p>\n<ol>\n<li><span style=\"line-height: 1.4\">predicting the result is quite slow. I use 50w data to training the model which is quite fast but when I use the whole dataset, it is quite slow. And the function `predict_proba` is also quite slow... Is there some method to make it faster.</span></li>\n<li><span style=\"line-height: 1.4\">I use bag of word so there is a lot of features. The matrix is too big so it has to be a sparse matrix structure. But random forest and grandient boosting do not support sparse matrix. I tried some features selection method to make the matrix smaller but There is still `segmentation fault` error without any explaination.</span></li>\n</ol>\n<p><span style=\"line-height: 1.4\">By the way, I use a computer with about 25GB RAM to generate&nbsp;my models.</span></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52662",
      "postDate": "08/29/2014 20:53:19",
      "content": "<p>[quote=aisensiy;52617]</p>\n<p><span style=\"line-height: 1.4\">predicting the result is quite slow. I use 50w data to training the model which is quite fast but when I use the whole dataset, it is quite slow. And the function `predict_proba` is also quite slow... Is there some method to make it faster.</span>[/quote]&nbsp;</p>\n<p>What is 50w? sklearn consist of a lot of methods - which one do you use?&nbsp;</p>\n<p>[quote=aisensiy;52617]</p>\n<p>I use bag of word so there is a lot of features. The matrix is too big so it has to be a sparse matrix structure. But random forest and grandient boosting do not support sparse matrix. I tried some features selection method to make the matrix smaller but There is still `segmentation fault` error without any explaination.[/quote]</p>\n<p>It's not a good idea to try use RF or GBRT on sparse matrix. Try to learn about linear methods.</p>\n<p>Or try to use RF(or GB) on SVD of original sparse matrix.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52677",
      "postDate": "08/30/2014 09:17:10",
      "content": "<p>By 50w, he means 500k (500 thousand). He made the same &quot;mistake&quot; as I did by using Chinglish :)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52704",
      "postDate": "08/30/2014 15:41:08",
      "content": "<p>Yeah, I really didn't realize the environment... 500k... not 50w...Thanks a lot.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52709",
      "postDate": "08/30/2014 17:33:06",
      "content": "<p>I tried a simple logistic regression model in scikit-learn, the cross-validation AUC score on my local machine is ~0.97 but I only get ~0.05 on leaderboard. Does anyone know what the problem is?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52747",
      "postDate": "08/31/2014 12:41:49",
      "content": "<p>Score 0.05 is equal to all-zeros benchmark.</p>\n<p>It means that you prabably have error in solution's format.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52756",
      "postDate": "08/31/2014 15:36:18",
      "content": "<p>50w means 500k...And it costs about 6 hours to run a full dataset solution. Is there some way to make it faster? Could you please share some ideas.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52757",
      "postDate": "08/31/2014 15:38:17",
      "content": "<p>[quote=Mikhail Trofimov;52662]</p>\n<p>sklearn consist of a lot of methods - which one do you use?</p>\n<p>[/quote]</p>\n\n<p>I use TfidfVectorizer&nbsp;SelectPercentile and LogisticRegression</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52760",
      "postDate": "08/31/2014 16:10:34",
      "content": "<p>[quote=Mikhail Trofimov;52747]</p>\n<p>Score 0.05 is equal to all-zeros benchmark.</p>\n<p>It means that you prabably have error in solution's format.</p>\n<p>[/quote]</p>\n<p>My bad, it should be ~0.5. I checked the format and made sure IDs were sorted by predicted probability in&nbsp;descending order.</p>\n<p>I never expected such a huge discrepancy between AUC and AP@k.</p>\n\n<p>--------</p>\n<p>Then I used Vowpal Wabbit instead, using the same feature set. The performance of vw is way much better.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "52761",
      "postDate": "08/31/2014 16:15:42",
      "content": "<p>Which of them are slow?</p>\n<p>TfidfVectorizer has linear complexity by samples. It means that if you spend 2 minutes on vectorize 500k - you need 20 minutes to vectorize 5kk samples.</p>\n<p>I suppose that LogisticRegression is slow. Some of possible solutions are:</p>\n<p>1) Use another method. For example,&nbsp;&nbsp;Naive Bayes or&nbsp;SGDClassifier.</p>\n<p>2) Set bigger tolerance parameter in LogisticRegression</p>\n<p>3) Use just a part of dataset to train model</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 52662,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "08/29/2014 20:53:19",
      "content": "<p>[quote=aisensiy;52617]</p>\n<p><span style=\"line-height: 1.4\">predicting the result is quite slow. I use 50w data to training the model which is quite fast but when I use the whole dataset, it is quite slow. And the function `predict_proba` is also quite slow... Is there some method to make it faster.</span>[/quote]&nbsp;</p>\n<p>What is 50w? sklearn consist of a lot of methods - which one do you use?&nbsp;</p>\n<p>[quote=aisensiy;52617]</p>\n<p>I use bag of word so there is a lot of features. The matrix is too big so it has to be a sparse matrix structure. But random forest and grandient boosting do not support sparse matrix. I tried some features selection method to make the matrix smaller but There is still `segmentation fault` error without any explaination.[/quote]</p>\n<p>It's not a good idea to try use RF or GBRT on sparse matrix. Try to learn about linear methods.</p>\n<p>Or try to use RF(or GB) on SVD of original sparse matrix.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52677,
      "author_name": "chenglongchen",
      "author_url": "",
      "post_date": "08/30/2014 09:17:10",
      "content": "<p>By 50w, he means 500k (500 thousand). He made the same &quot;mistake&quot; as I did by using Chinglish :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52704,
      "author_name": "aisensiy",
      "author_url": "",
      "post_date": "08/30/2014 15:41:08",
      "content": "<p>Yeah, I really didn't realize the environment... 500k... not 50w...Thanks a lot.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52709,
      "author_name": "apophenia",
      "author_url": "",
      "post_date": "08/30/2014 17:33:06",
      "content": "<p>I tried a simple logistic regression model in scikit-learn, the cross-validation AUC score on my local machine is ~0.97 but I only get ~0.05 on leaderboard. Does anyone know what the problem is?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52747,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "08/31/2014 12:41:49",
      "content": "<p>Score 0.05 is equal to all-zeros benchmark.</p>\n<p>It means that you prabably have error in solution's format.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52756,
      "author_name": "aisensiy",
      "author_url": "",
      "post_date": "08/31/2014 15:36:18",
      "content": "<p>50w means 500k...And it costs about 6 hours to run a full dataset solution. Is there some way to make it faster? Could you please share some ideas.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52757,
      "author_name": "aisensiy",
      "author_url": "",
      "post_date": "08/31/2014 15:38:17",
      "content": "<p>[quote=Mikhail Trofimov;52662]</p>\n<p>sklearn consist of a lot of methods - which one do you use?</p>\n<p>[/quote]</p>\n\n<p>I use TfidfVectorizer&nbsp;SelectPercentile and LogisticRegression</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52760,
      "author_name": "apophenia",
      "author_url": "",
      "post_date": "08/31/2014 16:10:34",
      "content": "<p>[quote=Mikhail Trofimov;52747]</p>\n<p>Score 0.05 is equal to all-zeros benchmark.</p>\n<p>It means that you prabably have error in solution's format.</p>\n<p>[/quote]</p>\n<p>My bad, it should be ~0.5. I checked the format and made sure IDs were sorted by predicted probability in&nbsp;descending order.</p>\n<p>I never expected such a huge discrepancy between AUC and AP@k.</p>\n\n<p>--------</p>\n<p>Then I used Vowpal Wabbit instead, using the same feature set. The performance of vw is way much better.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 52761,
      "author_name": "mikhailtrofimov",
      "author_url": "",
      "post_date": "08/31/2014 16:15:42",
      "content": "<p>Which of them are slow?</p>\n<p>TfidfVectorizer has linear complexity by samples. It means that if you spend 2 minutes on vectorize 500k - you need 20 minutes to vectorize 5kk samples.</p>\n<p>I suppose that LogisticRegression is slow. Some of possible solutions are:</p>\n<p>1) Use another method. For example,&nbsp;&nbsp;Naive Bayes or&nbsp;SGDClassifier.</p>\n<p>2) Set bigger tolerance parameter in LogisticRegression</p>\n<p>3) Use just a part of dataset to train model</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "52617": "",
    "52662": "",
    "52677": "",
    "52704": "",
    "52709": "",
    "52747": "",
    "52756": "",
    "52757": "",
    "52760": "",
    "52761": ""
  },
  "source": "meta"
}