{
  "id": 12314,
  "title": "scikit-learn",
  "url": "/competitions/avazu-ctr-prediction/discussion/12314",
  "author_name": "",
  "post_date": "2015-01-23T08:20:49.767Z",
  "votes": 3,
  "comment_count": 23,
  "views": 9959,
  "content": "<p>Did someone use scikit learn?</p>\n<p>I'm using scikit-learn and I'm stuck at 0.397...</p>",
  "messages": [
    {
      "id": "63047",
      "postDate": "01/23/2015 08:20:49",
      "content": "<p>Did someone use scikit learn?</p>\n<p>I'm using scikit-learn and I'm stuck at 0.397...</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63059",
      "postDate": "01/23/2015 15:09:02",
      "content": "<p>I am using scikit-learn!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63061",
      "postDate": "01/23/2015 18:35:36",
      "content": "<p>Which function do you guys use? sklearn.linear_model.LogisticRegression? I have tried this, but I don't know how to implement a mini-batch efficiently&nbsp;on&nbsp;it. I just divided the original data into many parts, and then train each of them using this function. It converges, but the result was disappointing</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63076",
      "postDate": "01/23/2015 21:38:10",
      "content": "<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63079",
      "postDate": "01/23/2015 22:57:14",
      "content": "<p>[quote=Jiming Ye;63061]</p>\n<p>Which function do you guys use? sklearn.linear_model.LogisticRegression? I have tried this, but I don't know how to implement a mini-batch efficiently on it. I just divided the original data into many parts, and then train each of them using this function. It converges, but the result was disappointing</p>\n<p>[/quote]</p>\n<p>I was thinking about trying this approach to run sklearn's LogisticRegression (which uses liblinear I believe). May I know how you divided the original data? Subsampling the data (this is what I'm doing now) or something else?</p>\n<p>Also, how disappointing was the result? What LB score (or validation score on your part) were you able to achieve with your try?</p>\n\n<p>[quote=Elena Cuoco;63076]</p>\n<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>\n<p>[/quote]</p>\n<p>I want to try sklearn's SGDClassifier too. Do we need to call <strong>fit</strong> once before calling subsequent <strong>partial_fit</strong>'s (I think I've heard this before; I'm not sure, though)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63090",
      "postDate": "01/24/2015 06:52:06",
      "content": "<p>No, you don't need to call fit before partial_fit. You need only to pass the classes in the function partial_fit.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63091",
      "postDate": "01/24/2015 08:06:36",
      "content": "<p>[quote=Elena Cuoco;63090]</p>\n<p>No, you don't need to call fit before partial_fit. You need only to pass the classes in the function partial_fit.</p>\n<p>[/quote]</p>\n<p>Thanks.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63188",
      "postDate": "01/26/2015 12:24:38",
      "content": "<p>[quote=Elena Cuoco;63076]</p>\n<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>\n<p>[/quote]</p>\n<p>Oh, thanks so much. I didn't know SGDClassifier has a partial fit function. It should be much better than LogisticRegression.&nbsp;</p>\n<p>Another question, how do you use SGDClassifier to predict the probilities? The predict function in it seems only give the class labels&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63190",
      "postDate": "01/26/2015 15:16:38",
      "content": "<p>[quote=Elena Cuoco;63047]</p>\n<p>Did someone use scikit learn?</p>\n<p>I'm using scikit-learn and I'm stuck at 0.397...</p>\n<p>[/quote]</p>\n\n<p>I'm using scikit-learn and just make 2 submission with score 0.44. Could you please share something about using scikit-learn in kaggle big data competition.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63200",
      "postDate": "01/26/2015 19:06:23",
      "content": "<p>Take a look at my post here:</p>\n\n<p>https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11246/beat-the-benchmark-with-scikit-learn-low-memory-403</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63203",
      "postDate": "01/26/2015 19:53:24",
      "content": "<p>@Jiming. SGDClassifier, if you select loss=log as&nbsp; function, has the method&nbsp; predict_proba that you can use to predict probability.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63238",
      "postDate": "01/27/2015 15:40:17",
      "content": "<p>[quote=MikeK;63200]</p>\n<p>Take a look at my post here:</p>\n<p>https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11246/beat-the-benchmark-with-scikit-learn-low-memory-403</p>\n<p>[/quote]</p>\n\n<p>Thanks for sharing!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63343",
      "postDate": "01/30/2015 07:11:04",
      "content": "<p>Stuck with 0.414 for now. Using one hot encoding and&nbsp;SGDClassifier. Also converted timestamp into hour and week day. Trying to make FeatureHasher work, but it gives weird results so far, all predictions are zero. Elena, any hints on how you climbed to 0.39? ;)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63345",
      "postDate": "01/30/2015 07:53:40",
      "content": "<p>Hi Artem.</p>\n<p>With scikit I reached 0.397 and then stuck there. I used FeaturesHasher for all the features and SGDclassifier. Take care of organize the data in the right format FeaturesHasher needs.</p>\n<p>Then I decided to try Vowpal Wabbit as I read in the forum, and I reached 0.394.</p>\n<p>In both case I made also some features engineering before the hashing.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63346",
      "postDate": "01/30/2015 08:11:35",
      "content": "<p>Thanks, Elena. For FeatureHasher I organized the data in a dictionary format</p>\n<p>raw_X_train = [dict(row[1]) for row in X_train.iterrows()]<br>fh = FeatureHasher(n_features = 2 ** 20)<br>X_enc_train = fh.transform(raw_X_train)</p>\n\n<p>Where X_train is a pandas DataFrame. Did you do something similar?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63347",
      "postDate": "01/30/2015 08:30:57",
      "content": "<p>I decided to pass strings instead of dict. I started with a pandas dataframe and then set them in a numpy array of strings</p>\n<p>X=np.asarray(clean.astype(str))</p>\n<p>where clean was my dataframe.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63410",
      "postDate": "02/01/2015 16:48:45",
      "content": "<p>Thanks, Elena, that worked!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63449",
      "postDate": "02/02/2015 23:09:28",
      "content": "<p>Hi,</p>\n<p>I was wondering if you would be willing to share any further info on reaching 0.397 with scikit learn.</p>\n<p>I've reached .399 but can't get any further, at this point it would be easier and faster to go with Vowpal Wabbit&nbsp; or FTRLProximal but I want to know what I'm missing :/ .</p>\n<p>I've tried adding all polynomial features at once but then my score goes out the window.</p>\n<p>I've also tried to split the training sets between mobile and browser but then my local score goes way low and I can't cross validate because it doesn't seem to have a linear relationship with the lb score.</p>\n<p>For info, I've been using SGD with hashed 1 500 000 rows mini batches, I tried converting the timestamp to hour and day but it heightened my score so I stopped.</p>\n<p>I tried learning rates from 0.1 to 0.000001 and number of features from 2**20 to 2**24.</p>\n<p>I tried thinking of logical relationships between features but with half of them unnamed and my little knowledge of advertising I'm struggling to create meaningful interactions. At this point I'm thinking of trying all polynomial features one by one but it would take about 55 hours on my PC so I'm reticent to invest the time in this if it's going to lead nowhere.</p>\n<p>Any advice?</p>\n<p>Cheers</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63450",
      "postDate": "02/02/2015 23:27:39",
      "content": "<p>Hm, you got .399 just with SGD + feature hasher and without preprocessing the timestamp? I tried to convert timestamp to hour and hour and day (2 separate columns) but the best I got was .41 so far. &nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63457",
      "postDate": "02/03/2015 02:19:35",
      "content": "<p>or to convert pandas dataframe to numpy array, following also works :&nbsp;&nbsp; df.astype(str).as_matrix()</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63470",
      "postDate": "02/03/2015 08:09:06",
      "content": "<p>[quote=Etienne Desticourt;63449]</p>\n<p>Hi,</p>\n<p>I was wondering if you would be willing to share any further info on reaching 0.397 with scikit learn.</p>\n<p>I've reached .399 but can't get any further, at this point it would be easier and faster to go with Vowpal Wabbit&nbsp; or FTRLProximal but I want to know what I'm missing :/ .</p>\n<p>I've tried adding all polynomial features at once but then my score goes out the window.</p>\n<p>I've also tried to split the training sets between mobile and browser but then my local score goes way low and I can't cross validate because it doesn't seem to have a linear relationship with the lb score.</p>\n<p>For info, I've been using SGD with hashed 1 500 000 rows mini batches, I tried converting the timestamp to hour and day but it heightened my score so I stopped.</p>\n<p>I tried learning rates from 0.1 to 0.000001 and number of features from 2**20 to 2**24.</p>\n<p>I tried thinking of logical relationships between features but with half of them unnamed and my little knowledge of advertising I'm struggling to create meaningful interactions. At this point I'm thinking of trying all polynomial features one by one but it would take about 55 hours on my PC so I'm reticent to invest the time in this if it's going to lead nowhere.</p>\n<p>Any advice?</p>\n<p>Cheers</p>\n<p>[/quote]</p>\n<p>You can go a long way with feature engineering. You can give a try for your last idea, but it's better to add interactions on your own.</p>\n<p>hint: If you have looked at the data really&nbsp;carefully, you should have found that 'device_ip' might&nbsp;have a big influence on&nbsp;the result. But it's a little bit tricky to use it correctly</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63473",
      "postDate": "02/03/2015 09:17:39",
      "content": "<p>Hello again,</p>\n<p>[quote=Artem Yankov;63450]</p>\n<p>Hm, you got .399 just with SGD + feature hasher and without preprocessing the timestamp? I tried to convert timestamp to hour and hour and day (2 separate columns) but the best I got was .41 so far. &nbsp;</p>\n<p>[/quote]</p>\n<p>I did get .399 with SGD + feature hasher, I'm unsure if I was using timestamp at the time because I hadn't started taking notes :/. I got .403 yesterday without timestamp preprocessing and with a single crafted feature.</p>\n<p>[quote=Jiming Ye;63470]</p>\n<p>You can go a long way with feature engineering. You can give a try for your last idea, but it's better to add interactions on your own.</p>\n<p>hint: If you have looked at the data really carefully, you should have found that 'device_ip' might have a big influence on the result. But it's a little bit tricky to use it correctly</p>\n<p>[/quote]</p>\n<p>Alright, I'll give it a try.</p>\n<p>I did see the importance of device_ip, I had noticed it scaled with the number of rows so I had tried to remove it to lower my memory usage but the results were hard on my score.</p>\n<p>I'll look deeper into it creating new features from this one, thanks for this precious advice!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "63474",
      "postDate": "02/03/2015 09:55:22",
      "content": "<p>Hi All,</p>\n<p>I did not use the polynomial features, since it is almost impossible to work with memory limits.</p>\n<p>I performed some features engineering, as also Jiming suggested, before the featureshasher.</p>\n<p>Maybe you can use less rows and more memory for the hashing&nbsp; and play a bit more with higher values for the learning parameters.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "65352",
      "postDate": "03/03/2015 17:58:46",
      "content": "<p>If you are interested, I put my 'scikit-learn' version on github</p>\n<p>https://github.com/elenacuoco/Avazu</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 63059,
      "author_name": "aparai",
      "author_url": "",
      "post_date": "01/23/2015 15:09:02",
      "content": "<p>I am using scikit-learn!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63061,
      "author_name": "yejiming",
      "author_url": "",
      "post_date": "01/23/2015 18:35:36",
      "content": "<p>Which function do you guys use? sklearn.linear_model.LogisticRegression? I have tried this, but I don't know how to implement a mini-batch efficiently&nbsp;on&nbsp;it. I just divided the original data into many parts, and then train each of them using this function. It converges, but the result was disappointing</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63076,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/23/2015 21:38:10",
      "content": "<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63079,
      "author_name": "kuo000",
      "author_url": "",
      "post_date": "01/23/2015 22:57:14",
      "content": "<p>[quote=Jiming Ye;63061]</p>\n<p>Which function do you guys use? sklearn.linear_model.LogisticRegression? I have tried this, but I don't know how to implement a mini-batch efficiently on it. I just divided the original data into many parts, and then train each of them using this function. It converges, but the result was disappointing</p>\n<p>[/quote]</p>\n<p>I was thinking about trying this approach to run sklearn's LogisticRegression (which uses liblinear I believe). May I know how you divided the original data? Subsampling the data (this is what I'm doing now) or something else?</p>\n<p>Also, how disappointing was the result? What LB score (or validation score on your part) were you able to achieve with your try?</p>\n\n<p>[quote=Elena Cuoco;63076]</p>\n<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>\n<p>[/quote]</p>\n<p>I want to try sklearn's SGDClassifier too. Do we need to call <strong>fit</strong> once before calling subsequent <strong>partial_fit</strong>'s (I think I've heard this before; I'm not sure, though)?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63090,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/24/2015 06:52:06",
      "content": "<p>No, you don't need to call fit before partial_fit. You need only to pass the classes in the function partial_fit.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63091,
      "author_name": "kuo000",
      "author_url": "",
      "post_date": "01/24/2015 08:06:36",
      "content": "<p>[quote=Elena Cuoco;63090]</p>\n<p>No, you don't need to call fit before partial_fit. You need only to pass the classes in the function partial_fit.</p>\n<p>[/quote]</p>\n<p>Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63188,
      "author_name": "yejiming",
      "author_url": "",
      "post_date": "01/26/2015 12:24:38",
      "content": "<p>[quote=Elena Cuoco;63076]</p>\n<p>scikit has some classifier which implements partial fit. I'm using SGDClassifier. Have a look at this link http://scikit-learn.org/stable/modules/scaling_strategies.html.</p>\n<p>[/quote]</p>\n<p>Oh, thanks so much. I didn't know SGDClassifier has a partial fit function. It should be much better than LogisticRegression.&nbsp;</p>\n<p>Another question, how do you use SGDClassifier to predict the probilities? The predict function in it seems only give the class labels&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63190,
      "author_name": "aisensiy",
      "author_url": "",
      "post_date": "01/26/2015 15:16:38",
      "content": "<p>[quote=Elena Cuoco;63047]</p>\n<p>Did someone use scikit learn?</p>\n<p>I'm using scikit-learn and I'm stuck at 0.397...</p>\n<p>[/quote]</p>\n\n<p>I'm using scikit-learn and just make 2 submission with score 0.44. Could you please share something about using scikit-learn in kaggle big data competition.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63200,
      "author_name": "michaelkneier",
      "author_url": "",
      "post_date": "01/26/2015 19:06:23",
      "content": "<p>Take a look at my post here:</p>\n\n<p>https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11246/beat-the-benchmark-with-scikit-learn-low-memory-403</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63203,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/26/2015 19:53:24",
      "content": "<p>@Jiming. SGDClassifier, if you select loss=log as&nbsp; function, has the method&nbsp; predict_proba that you can use to predict probability.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63238,
      "author_name": "aisensiy",
      "author_url": "",
      "post_date": "01/27/2015 15:40:17",
      "content": "<p>[quote=MikeK;63200]</p>\n<p>Take a look at my post here:</p>\n<p>https://www.kaggle.com/c/avazu-ctr-prediction/forums/t/11246/beat-the-benchmark-with-scikit-learn-low-memory-403</p>\n<p>[/quote]</p>\n\n<p>Thanks for sharing!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63343,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/30/2015 07:11:04",
      "content": "<p>Stuck with 0.414 for now. Using one hot encoding and&nbsp;SGDClassifier. Also converted timestamp into hour and week day. Trying to make FeatureHasher work, but it gives weird results so far, all predictions are zero. Elena, any hints on how you climbed to 0.39? ;)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63345,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/30/2015 07:53:40",
      "content": "<p>Hi Artem.</p>\n<p>With scikit I reached 0.397 and then stuck there. I used FeaturesHasher for all the features and SGDclassifier. Take care of organize the data in the right format FeaturesHasher needs.</p>\n<p>Then I decided to try Vowpal Wabbit as I read in the forum, and I reached 0.394.</p>\n<p>In both case I made also some features engineering before the hashing.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63346,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "01/30/2015 08:11:35",
      "content": "<p>Thanks, Elena. For FeatureHasher I organized the data in a dictionary format</p>\n<p>raw_X_train = [dict(row[1]) for row in X_train.iterrows()]<br>fh = FeatureHasher(n_features = 2 ** 20)<br>X_enc_train = fh.transform(raw_X_train)</p>\n\n<p>Where X_train is a pandas DataFrame. Did you do something similar?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63347,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "01/30/2015 08:30:57",
      "content": "<p>I decided to pass strings instead of dict. I started with a pandas dataframe and then set them in a numpy array of strings</p>\n<p>X=np.asarray(clean.astype(str))</p>\n<p>where clean was my dataframe.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63410,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "02/01/2015 16:48:45",
      "content": "<p>Thanks, Elena, that worked!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63449,
      "author_name": "etiennedesticourt",
      "author_url": "",
      "post_date": "02/02/2015 23:09:28",
      "content": "<p>Hi,</p>\n<p>I was wondering if you would be willing to share any further info on reaching 0.397 with scikit learn.</p>\n<p>I've reached .399 but can't get any further, at this point it would be easier and faster to go with Vowpal Wabbit&nbsp; or FTRLProximal but I want to know what I'm missing :/ .</p>\n<p>I've tried adding all polynomial features at once but then my score goes out the window.</p>\n<p>I've also tried to split the training sets between mobile and browser but then my local score goes way low and I can't cross validate because it doesn't seem to have a linear relationship with the lb score.</p>\n<p>For info, I've been using SGD with hashed 1 500 000 rows mini batches, I tried converting the timestamp to hour and day but it heightened my score so I stopped.</p>\n<p>I tried learning rates from 0.1 to 0.000001 and number of features from 2**20 to 2**24.</p>\n<p>I tried thinking of logical relationships between features but with half of them unnamed and my little knowledge of advertising I'm struggling to create meaningful interactions. At this point I'm thinking of trying all polynomial features one by one but it would take about 55 hours on my PC so I'm reticent to invest the time in this if it's going to lead nowhere.</p>\n<p>Any advice?</p>\n<p>Cheers</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63450,
      "author_name": "yankov",
      "author_url": "",
      "post_date": "02/02/2015 23:27:39",
      "content": "<p>Hm, you got .399 just with SGD + feature hasher and without preprocessing the timestamp? I tried to convert timestamp to hour and hour and day (2 separate columns) but the best I got was .41 so far. &nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63457,
      "author_name": "sravanb",
      "author_url": "",
      "post_date": "02/03/2015 02:19:35",
      "content": "<p>or to convert pandas dataframe to numpy array, following also works :&nbsp;&nbsp; df.astype(str).as_matrix()</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63470,
      "author_name": "yejiming",
      "author_url": "",
      "post_date": "02/03/2015 08:09:06",
      "content": "<p>[quote=Etienne Desticourt;63449]</p>\n<p>Hi,</p>\n<p>I was wondering if you would be willing to share any further info on reaching 0.397 with scikit learn.</p>\n<p>I've reached .399 but can't get any further, at this point it would be easier and faster to go with Vowpal Wabbit&nbsp; or FTRLProximal but I want to know what I'm missing :/ .</p>\n<p>I've tried adding all polynomial features at once but then my score goes out the window.</p>\n<p>I've also tried to split the training sets between mobile and browser but then my local score goes way low and I can't cross validate because it doesn't seem to have a linear relationship with the lb score.</p>\n<p>For info, I've been using SGD with hashed 1 500 000 rows mini batches, I tried converting the timestamp to hour and day but it heightened my score so I stopped.</p>\n<p>I tried learning rates from 0.1 to 0.000001 and number of features from 2**20 to 2**24.</p>\n<p>I tried thinking of logical relationships between features but with half of them unnamed and my little knowledge of advertising I'm struggling to create meaningful interactions. At this point I'm thinking of trying all polynomial features one by one but it would take about 55 hours on my PC so I'm reticent to invest the time in this if it's going to lead nowhere.</p>\n<p>Any advice?</p>\n<p>Cheers</p>\n<p>[/quote]</p>\n<p>You can go a long way with feature engineering. You can give a try for your last idea, but it's better to add interactions on your own.</p>\n<p>hint: If you have looked at the data really&nbsp;carefully, you should have found that 'device_ip' might&nbsp;have a big influence on&nbsp;the result. But it's a little bit tricky to use it correctly</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63473,
      "author_name": "etiennedesticourt",
      "author_url": "",
      "post_date": "02/03/2015 09:17:39",
      "content": "<p>Hello again,</p>\n<p>[quote=Artem Yankov;63450]</p>\n<p>Hm, you got .399 just with SGD + feature hasher and without preprocessing the timestamp? I tried to convert timestamp to hour and hour and day (2 separate columns) but the best I got was .41 so far. &nbsp;</p>\n<p>[/quote]</p>\n<p>I did get .399 with SGD + feature hasher, I'm unsure if I was using timestamp at the time because I hadn't started taking notes :/. I got .403 yesterday without timestamp preprocessing and with a single crafted feature.</p>\n<p>[quote=Jiming Ye;63470]</p>\n<p>You can go a long way with feature engineering. You can give a try for your last idea, but it's better to add interactions on your own.</p>\n<p>hint: If you have looked at the data really carefully, you should have found that 'device_ip' might have a big influence on the result. But it's a little bit tricky to use it correctly</p>\n<p>[/quote]</p>\n<p>Alright, I'll give it a try.</p>\n<p>I did see the importance of device_ip, I had noticed it scaled with the number of rows so I had tried to remove it to lower my memory usage but the results were hard on my score.</p>\n<p>I'll look deeper into it creating new features from this one, thanks for this precious advice!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 63474,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "02/03/2015 09:55:22",
      "content": "<p>Hi All,</p>\n<p>I did not use the polynomial features, since it is almost impossible to work with memory limits.</p>\n<p>I performed some features engineering, as also Jiming suggested, before the featureshasher.</p>\n<p>Maybe you can use less rows and more memory for the hashing&nbsp; and play a bit more with higher values for the learning parameters.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 65352,
      "author_name": "elenacuoco",
      "author_url": "",
      "post_date": "03/03/2015 17:58:46",
      "content": "<p>If you are interested, I put my 'scikit-learn' version on github</p>\n<p>https://github.com/elenacuoco/Avazu</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "63047": "",
    "63059": "",
    "63061": "",
    "63076": "",
    "63079": "",
    "63090": "",
    "63091": "",
    "63188": "",
    "63190": "",
    "63200": "",
    "63203": "",
    "63238": "",
    "63343": "",
    "63345": "",
    "63346": "",
    "63347": "",
    "63410": "",
    "63449": "",
    "63450": "",
    "63457": "",
    "63470": "",
    "63473": "",
    "63474": "",
    "65352": ""
  },
  "source": "meta"
}