{
  "id": 5258,
  "title": "How I modified Miroslaw's code",
  "content": "<p>I thank again&nbsp;Miroslaw for posting his great code.</p>\n<p>Not only it was a very performing code but it gave us all a fantastic occasion to learn and test how to work with Python. Thank you again.</p>\n<p>I slightly modified Miroslaw 's code making it a bit more performing and I would like to share it with other Kagglers, so it could be an occasion to improve furthermore our abilities and knowledge when working with Python on similar problems.</p>\n<p>My results were entirely based on such a code.</p>\n<p>Here are the changes I applied:</p>\n<p><br>1) An option to start from a set of predictors<br>2) An option to immediately compute the final solution without further feature selection<br>3) Multiprocessor, automatically choses the best number of jobs for maximum computation speed<br>4) Introduced a small_change variable fixing the minimum change in a model to be acceptable in order to avoid overfitting<br>5) Features with less than 3 cases are clustered together in a rare group<br>6) After inserting a new variable it checks if the already present variables are still meaningful to be enclosed in the model (pruning)<br>7) As for as cross validation, it fixes test_size=.15 and it uses median not mean to average the cross validation results<br>8) It prints out only significative model changes, history of the model, best C value<br>9) Randomized start, final CV score saved with the filename</p>",
  "comments": [
    {
      "id": 28037,
      "author_name": "Leustagos",
      "author_url": "",
      "post_date": "2013-08-02T03:38:27.640000",
      "content": "<ul>\n<li>Added options:</li>\n<ol>\n<li>train file path</li>\n<li>test file path</li>\n<li>submiossion file path</li>\n<li>seed</li>\n<li>minimum number of occurences of an id to not consider it rare (if it is rare, group into a the same value)</li>\n<li>option to specify good features. if this is specified, it will bypass the feature selection option</li>\n</ol>\n<li>Added all degrees of combination</li>\n<li>used secretary theorem to speed up each feature selection loop</li>\n<li>changed the regularization param to a random value</li>\n<li>changed the k-fold to use all instances</li>\n<li>removed a feature if it makes the score worst (to speed up the procces)</li>\n</ul>\n<p>The model in wich we used it scored&nbsp;0.91568 in public leaderboard and 0.91360 in the private leaderboard. This model was a simple mean of many runs. Each run take about 20 minutes to run, even with all interaction degrees included.</p>\n<p>&nbsp;</p>",
      "votes": 15,
      "replies": []
    },
    {
      "id": 28026,
      "author_name": "Ben S",
      "author_url": "",
      "post_date": "2013-08-01T21:45:44.043000",
      "content": "<p>Before I merged with Paul a modified version of Miroslaw's code was 2/3 of my best submission. &nbsp;I just changed a couple of small things:</p>\n<ul>\n<li>Grouped all the 'rare' sparse features together for each type of feature. This one change raised my score about 0.008.</li>\n<li>Decreased the CV data from 20% to 10%, this was to avoid creating more 'rare features', i.e. in the CV loops. &nbsp;</li>\n<li>Got rid of a couple feature combinations that were redundant, partly just to speed things up and cut down on complexity. &nbsp;For instance, role_title was just a subset of role_family so no reason to look at a higher order feature that combines the two.</li>\n<li>Added some lines to switch between logistic and naive bayes at the command line. &nbsp;It would be easy to add more models, pretty much any sklearn classifier that can handle sparseness could be included, but none of those I tried really improved on logistic by itself.</li>\n</ul>\n<p>I also ran it with a bunch of random seeds and used CV and the leaderboard to choose the best one. &nbsp;The variation wasn't so small, CVs went from 0.9004 to 0.9140 just by changing the seed.&nbsp;</p>",
      "votes": 16,
      "replies": []
    },
    {
      "id": 27999,
      "author_name": "Giba",
      "author_url": "",
      "post_date": "2013-08-01T12:46:41.153000",
      "content": "<ul>\n<li>First I preprocessed the datasets using R to generate up to level 4 of features combination and grouped the rare features into only one group and exported this new dataset to .csv</li>\n<li>In Miroslaw code I just load the preprocessed csv dataset.</li>\n<li>Then I basically modified the cv_loop function to make sure that all instances are used to train (see attached).</li>\n<li>Then I execute the greed algorithm using 6 fold cv</li>\n<li>Find the best C values (hyperparameter)</li>\n<li>Then using the best C I did a 200 fold cross validation on the train and test set (see attached) and save both train and test results. Saving train predictions helps using ensembling algorithms later.</li>\n</ul>\n<p>That single model&nbsp;takes ~6 hours to run in a single core cpu and gives me public: 0.91798 &nbsp;private: 091322.</p>",
      "votes": 16,
      "replies": []
    },
    {
      "id": 28034,
      "author_name": "Alessandro Mariani",
      "author_url": "",
      "post_date": "2013-08-02T02:22:23.610000",
      "content": "<p>I've attached the code I've to set my optimal minimum frequencies values for each feature. This basically implements Nick's idea he shared on the forum.</p>\n<p>Basically, after you've come up with your &quot;good features&quot; set, it will loop on all of features increasing the minimum frequency one by one. My model before increasing the minimum frequencies had CV: 0.889 | Public: 0.90715 | Private <strong>0.90002</strong>. After the adhoc increases, the same set of features scored CV: 0.902 | Public: 0.91786 | Private: <strong>0.91535</strong></p>\n<p>After that I've re-run the greedy features selection including 6 new features, readjusting the frequency and I've got a new model with CV: 903 | Public: 0.91817 | Private: <strong>0.91355</strong></p>\n<p>Funny that the second model had a drop on the private, while it had an increasing of either CV &amp; public! And even funnier that I've chosen the second, discarding the first for my ensembles which didn't lead to any improvements at all!!!!</p>\n<p>&nbsp;</p>\n<p>whoever want to give it a go, the first set was from the original Miroslaw code:<br>[0, 1, 7, 8, 9, 10, 34, 36, 37, 38, 40, 41, 42, 43, 47, 49, 53, 55, 60, 61, 63, 64, 66, 67, 69, 71, 75, 81, 82, 85, 90]</p>",
      "votes": 10,
      "replies": []
    },
    {
      "id": 28053,
      "author_name": "Abhishek Thakur",
      "author_url": "",
      "post_date": "2013-08-02T14:16:40.960000",
      "content": "<p>I used some different types of models combined with Miroslaw's code.&nbsp;</p>\n<ul>\n<li>Grouped original data into 4 -&gt; greedy selection -&gt; logistic regression</li>\n<li>Manually added three new features -&gt; group into 4 -&gt; Greedy -&gt; infrequent feature removal -&gt; GBM (previously LR)</li>\n<li>Divide each feature into sub features -&gt; greedy -&gt; combine with the above features -&gt; Log Res</li>\n<li>Original data -&gt; remove/replace infrequent(1st, 2nd and 3rd degree) [I included ACTION also, and this enhanced my score] -&gt; Log Regression</li>\n</ul>\n<p>I found out on the last day that blending with GBM improved the AUC on the leaderboard. I didnt have much time to play with it though.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 28093,
      "author_name": "CodingNeo",
      "author_url": "",
      "post_date": "2013-08-02T22:16:28.030000",
      "content": "<p>[quote=BS Man;28026]</p>\n<p>Before I merged with Paul a modified version of Miroslaw's code was 2/3 of my best submission. &nbsp;I just changed a couple of small things:</p>\n<ul>\n<li>Grouped all the 'rare' sparse features together for each type of feature. This one change raised my score about 0.008.</li>\n<li>Decreased the CV data from 20% to 10%, this was to avoid creating more 'rare features', i.e. in the CV loops. &nbsp;</li>\n<li>Got rid of a couple feature combinations that were redundant, partly just to speed things up and cut down on complexity. &nbsp;For instance, role_title was just a subset of role_family so no reason to look at a higher order feature that combines the two.</li>\n<li>Added some lines to switch between logistic and naive bayes at the command line. &nbsp;It would be easy to add more models, pretty much any sklearn classifier that can handle sparseness could be included, but none of those I tried really improved on logistic by itself.</li>\n</ul>\n<p>I also ran it with a bunch of random seeds and used CV and the leaderboard to choose the best one. &nbsp;The variation wasn't so small, CVs went from 0.9004 to 0.9140 just by changing the seed.&nbsp;</p>\n<p>[/quote]</p>\n<p>One question about what's your range to choose seed?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 28067,
      "author_name": "Adam Daum",
      "author_url": "",
      "post_date": "2013-08-02T15:24:45.397000",
      "content": "<p>Thanks everybody for sharing your code.&nbsp; I learned a lot from this experience.</p>\n<p>Adam</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 28058,
      "author_name": "CodingNeo",
      "author_url": "",
      "post_date": "2013-08-02T14:38:16.557000",
      "content": "<p>[quote=Abhishek;28053]</p>\n<p>I used some different types of models combined with Miroslaw's code.&nbsp;</p>\n<ul>\n<li>Grouped original data into 4 -&gt; greedy selection -&gt; logistic regression</li>\n<li>Manually added three new features -&gt; group into 4 -&gt; Greedy -&gt; infrequent feature removal -&gt; GBM (previously LR)</li>\n<li>Divide each feature into sub features -&gt; greedy -&gt; combine with the above features -&gt; Log Res</li>\n<li>Original data -&gt; remove/replace infrequent(1st, 2nd and 3rd degree) [I included ACTION also, and this enhanced my score] -&gt; Log Regression</li>\n</ul>\n<p>I found out on the last day that blending with GBM improved the AUC on the leaderboard. I didnt have much time to play with it though.</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>What you describe is not very clear for me.&nbsp;</p>\n<p>* Grouped original data into 4. 4 you mean group data until 4th degree or what else?</p>\n<p>* Manually added three new features (what are these three features)</p>\n<p>* Divide each feature into sub features? what do you mean?</p>\n<p>* You included ACTION, but how to encode test data where ACTION is not available?</p>\n<p>Sorry for so many question, but to benefit the community, it is better for the information to be clear.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 27986,
      "author_name": "Vikas Agrawal",
      "author_url": "",
      "post_date": "2013-08-01T09:19:31.107000",
      "content": "<p>Thanks Luca for sharing the code....</p>",
      "votes": 0,
      "replies": []
    }
  ],
  "comment_count": 9,
  "messages": [
    {
      "id": 27984,
      "postDate": "2013-08-01T09:00:51.820Z",
      "content": "<p>I thank again&nbsp;Miroslaw for posting his great code.</p>\n<p>Not only it was a very performing code but it gave us all a fantastic occasion to learn and test how to work with Python. Thank you again.</p>\n<p>I slightly modified Miroslaw 's code making it a bit more performing and I would like to share it with other Kagglers, so it could be an occasion to improve furthermore our abilities and knowledge when working with Python on similar problems.</p>\n<p>My results were entirely based on such a code.</p>\n<p>Here are the changes I applied:</p>\n<p><br>1) An option to start from a set of predictors<br>2) An option to immediately compute the final solution without further feature selection<br>3) Multiprocessor, automatically choses the best number of jobs for maximum computation speed<br>4) Introduced a small_change variable fixing the minimum change in a model to be acceptable in order to avoid overfitting<br>5) Features with less than 3 cases are clustered together in a rare group<br>6) After inserting a new variable it checks if the already present variables are still meaningful to be enclosed in the model (pruning)<br>7) As for as cross validation, it fixes test_size=.15 and it uses median not mean to average the cross validation results<br>8) It prints out only significative model changes, history of the model, best C value<br>9) Randomized start, final CV score saved with the filename</p>",
      "votes": 22
    },
    {
      "id": 28037,
      "postDate": "2013-08-02T03:38:27.640Z",
      "content": "<ul>\n<li>Added options:</li>\n<ol>\n<li>train file path</li>\n<li>test file path</li>\n<li>submiossion file path</li>\n<li>seed</li>\n<li>minimum number of occurences of an id to not consider it rare (if it is rare, group into a the same value)</li>\n<li>option to specify good features. if this is specified, it will bypass the feature selection option</li>\n</ol>\n<li>Added all degrees of combination</li>\n<li>used secretary theorem to speed up each feature selection loop</li>\n<li>changed the regularization param to a random value</li>\n<li>changed the k-fold to use all instances</li>\n<li>removed a feature if it makes the score worst (to speed up the procces)</li>\n</ul>\n<p>The model in wich we used it scored&nbsp;0.91568 in public leaderboard and 0.91360 in the private leaderboard. This model was a simple mean of many runs. Each run take about 20 minutes to run, even with all interaction degrees included.</p>\n<p>&nbsp;</p>",
      "votes": 15
    },
    {
      "id": 28026,
      "postDate": "2013-08-01T21:45:44.043Z",
      "content": "<p>Before I merged with Paul a modified version of Miroslaw's code was 2/3 of my best submission. &nbsp;I just changed a couple of small things:</p>\n<ul>\n<li>Grouped all the 'rare' sparse features together for each type of feature. This one change raised my score about 0.008.</li>\n<li>Decreased the CV data from 20% to 10%, this was to avoid creating more 'rare features', i.e. in the CV loops. &nbsp;</li>\n<li>Got rid of a couple feature combinations that were redundant, partly just to speed things up and cut down on complexity. &nbsp;For instance, role_title was just a subset of role_family so no reason to look at a higher order feature that combines the two.</li>\n<li>Added some lines to switch between logistic and naive bayes at the command line. &nbsp;It would be easy to add more models, pretty much any sklearn classifier that can handle sparseness could be included, but none of those I tried really improved on logistic by itself.</li>\n</ul>\n<p>I also ran it with a bunch of random seeds and used CV and the leaderboard to choose the best one. &nbsp;The variation wasn't so small, CVs went from 0.9004 to 0.9140 just by changing the seed.&nbsp;</p>",
      "votes": 16
    },
    {
      "id": 27999,
      "postDate": "2013-08-01T12:46:41.153Z",
      "content": "<ul>\n<li>First I preprocessed the datasets using R to generate up to level 4 of features combination and grouped the rare features into only one group and exported this new dataset to .csv</li>\n<li>In Miroslaw code I just load the preprocessed csv dataset.</li>\n<li>Then I basically modified the cv_loop function to make sure that all instances are used to train (see attached).</li>\n<li>Then I execute the greed algorithm using 6 fold cv</li>\n<li>Find the best C values (hyperparameter)</li>\n<li>Then using the best C I did a 200 fold cross validation on the train and test set (see attached) and save both train and test results. Saving train predictions helps using ensembling algorithms later.</li>\n</ul>\n<p>That single model&nbsp;takes ~6 hours to run in a single core cpu and gives me public: 0.91798 &nbsp;private: 091322.</p>",
      "votes": 16
    },
    {
      "id": 28034,
      "postDate": "2013-08-02T02:22:23.610Z",
      "content": "<p>I've attached the code I've to set my optimal minimum frequencies values for each feature. This basically implements Nick's idea he shared on the forum.</p>\n<p>Basically, after you've come up with your &quot;good features&quot; set, it will loop on all of features increasing the minimum frequency one by one. My model before increasing the minimum frequencies had CV: 0.889 | Public: 0.90715 | Private <strong>0.90002</strong>. After the adhoc increases, the same set of features scored CV: 0.902 | Public: 0.91786 | Private: <strong>0.91535</strong></p>\n<p>After that I've re-run the greedy features selection including 6 new features, readjusting the frequency and I've got a new model with CV: 903 | Public: 0.91817 | Private: <strong>0.91355</strong></p>\n<p>Funny that the second model had a drop on the private, while it had an increasing of either CV &amp; public! And even funnier that I've chosen the second, discarding the first for my ensembles which didn't lead to any improvements at all!!!!</p>\n<p>&nbsp;</p>\n<p>whoever want to give it a go, the first set was from the original Miroslaw code:<br>[0, 1, 7, 8, 9, 10, 34, 36, 37, 38, 40, 41, 42, 43, 47, 49, 53, 55, 60, 61, 63, 64, 66, 67, 69, 71, 75, 81, 82, 85, 90]</p>",
      "votes": 10
    },
    {
      "id": 28053,
      "postDate": "2013-08-02T14:16:40.960Z",
      "content": "<p>I used some different types of models combined with Miroslaw's code.&nbsp;</p>\n<ul>\n<li>Grouped original data into 4 -&gt; greedy selection -&gt; logistic regression</li>\n<li>Manually added three new features -&gt; group into 4 -&gt; Greedy -&gt; infrequent feature removal -&gt; GBM (previously LR)</li>\n<li>Divide each feature into sub features -&gt; greedy -&gt; combine with the above features -&gt; Log Res</li>\n<li>Original data -&gt; remove/replace infrequent(1st, 2nd and 3rd degree) [I included ACTION also, and this enhanced my score] -&gt; Log Regression</li>\n</ul>\n<p>I found out on the last day that blending with GBM improved the AUC on the leaderboard. I didnt have much time to play with it though.</p>",
      "votes": 1
    },
    {
      "id": 28093,
      "postDate": "2013-08-02T22:16:28.030Z",
      "content": "<p>[quote=BS Man;28026]</p>\n<p>Before I merged with Paul a modified version of Miroslaw's code was 2/3 of my best submission. &nbsp;I just changed a couple of small things:</p>\n<ul>\n<li>Grouped all the 'rare' sparse features together for each type of feature. This one change raised my score about 0.008.</li>\n<li>Decreased the CV data from 20% to 10%, this was to avoid creating more 'rare features', i.e. in the CV loops. &nbsp;</li>\n<li>Got rid of a couple feature combinations that were redundant, partly just to speed things up and cut down on complexity. &nbsp;For instance, role_title was just a subset of role_family so no reason to look at a higher order feature that combines the two.</li>\n<li>Added some lines to switch between logistic and naive bayes at the command line. &nbsp;It would be easy to add more models, pretty much any sklearn classifier that can handle sparseness could be included, but none of those I tried really improved on logistic by itself.</li>\n</ul>\n<p>I also ran it with a bunch of random seeds and used CV and the leaderboard to choose the best one. &nbsp;The variation wasn't so small, CVs went from 0.9004 to 0.9140 just by changing the seed.&nbsp;</p>\n<p>[/quote]</p>\n<p>One question about what's your range to choose seed?</p>"
    },
    {
      "id": 28067,
      "postDate": "2013-08-02T15:24:45.397Z",
      "content": "<p>Thanks everybody for sharing your code.&nbsp; I learned a lot from this experience.</p>\n<p>Adam</p>"
    },
    {
      "id": 28058,
      "postDate": "2013-08-02T14:38:16.557Z",
      "content": "<p>[quote=Abhishek;28053]</p>\n<p>I used some different types of models combined with Miroslaw's code.&nbsp;</p>\n<ul>\n<li>Grouped original data into 4 -&gt; greedy selection -&gt; logistic regression</li>\n<li>Manually added three new features -&gt; group into 4 -&gt; Greedy -&gt; infrequent feature removal -&gt; GBM (previously LR)</li>\n<li>Divide each feature into sub features -&gt; greedy -&gt; combine with the above features -&gt; Log Res</li>\n<li>Original data -&gt; remove/replace infrequent(1st, 2nd and 3rd degree) [I included ACTION also, and this enhanced my score] -&gt; Log Regression</li>\n</ul>\n<p>I found out on the last day that blending with GBM improved the AUC on the leaderboard. I didnt have much time to play with it though.</p>\n<p>[/quote]</p>\n<p>&nbsp;</p>\n<p>What you describe is not very clear for me.&nbsp;</p>\n<p>* Grouped original data into 4. 4 you mean group data until 4th degree or what else?</p>\n<p>* Manually added three new features (what are these three features)</p>\n<p>* Divide each feature into sub features? what do you mean?</p>\n<p>* You included ACTION, but how to encode test data where ACTION is not available?</p>\n<p>Sorry for so many question, but to benefit the community, it is better for the information to be clear.</p>"
    },
    {
      "id": 27986,
      "postDate": "2013-08-01T09:19:31.107Z",
      "content": "<p>Thanks Luca for sharing the code....</p>"
    }
  ],
  "complete": true,
  "source": "live",
  "fetched_at": "2026-10-19T19:06:51Z",
  "competition_id": 3338,
  "competition_slug": "amazon-employee-access-challenge",
  "team_id": 52512,
  "solution_url": "/competitions/amazon-employee-access-challenge/writeups/lm-how-i-modified-miroslaw-s-code"
}