I'm really curious to hear from the rest of the top 10, but I'll go first:
\n\nOnce I see the public vs private LB for all my subs, I'll have more details on what helped or what hurt.
", "comments": [ { "id": 149860, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:57:45.977000", "content": "Also, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
", "votes": 9, "replies": [ { "id": 149866, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T03:01:43.500000", "content": "[quote=Zach;149860]
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
\n\n[/quote]
\n\nLink to the repository? :)
", "votes": 1, "replies": [ { "id": 149874, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T03:15:03.113000", "content": "[quote=Vladimir Iglovikov;149866]\nLink to the repository? :)\n[/quote]
\n\nI gotta put it all up on github (was using a local repo).
", "votes": 2, "replies": [] } ] }, { "id": 149869, "author_name": "Tilii", "author_url": "", "post_date": "2016-12-13T03:08:57.097000", "content": "[quote=Zach;149860]
\n\nAlso, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
\n\n[/quote]
\n\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...
\n\nThat's why I never could publish a nice post-mortem for this competition like Vladimir did. Props for that, by the way.
", "votes": 1, "replies": [ { "id": 149870, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T03:11:46.440000", "content": "[quote=Tilii;149869]\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\n[/quote]
\n\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/
", "votes": 1, "replies": [] }, { "id": 149877, "author_name": "Tilii", "author_url": "", "post_date": "2016-12-13T03:20:53.577000", "content": "[quote=Zach;149870]
\n\n[quote=Tilii;149869]\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\n[/quote]
\n\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/
\n\n[/quote]
\n\nYou just made $20 for Tim Harford. Let's hope I read it before the holidays are over, or else I'll start the New Year with my guilt undiminished over the state of my garage, basement, office ...
", "votes": 2, "replies": [] } ] } ] }, { "id": 149863, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:58:55.620000", "content": "Also, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.
", "votes": 3, "replies": [] }, { "id": 149846, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T02:03:05.567000", "content": "[quote=Vladimir Iglovikov;149846]
\n\n[/quote]
\n\n/edit: changed 90% to 95%. I relied pretty heavily on XGB + keras, but I did tweak the keras scripts (e.g. adding highway layers to some).
", "votes": 3, "replies": [] } ] }, { "id": 149845, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:02:03.763000", "content": "One follow up: I started with a simple stacking model: BFGS to find the weights that minimize MAE. I then slowly made that stack more complicated, first by adding the PC1 component, then by adding L1 regularization, and finally splitting into thirds based on the first PC component.
", "votes": 3, "replies": [] }, { "id": 150037, "author_name": "Ben Gorman", "author_url": "", "post_date": "2016-12-13T16:51:05.027000", "content": "Congrats on making it to Grand Master, Zach!
\n\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.
", "votes": 1, "replies": [ { "id": 150044, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T17:16:05.377000", "content": "[quote=Ben Gorman;150037]
\n\nCongrats on making it to Grand Master, Zach!
\n\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.
\n\n[/quote]
\n\nThanks! I did all my data prep and ran all my ensembles on my 2 year old macbook pro, with a core i7 processor and 16 GB of RAM. Macs have really good memory management, and I've been really happy with this laptop for kaggle (unfortunately it has an AMD, rather than NVIDIA GPU).
\n\nI also ran a lot of XGBoost models on my laptop.
\n\nFor the keras models, I ran them on a workstation with an NVIDIA GPU, which I also used to run a bunch of variants of XGBoost towards the end of the competition.
", "votes": 2, "replies": [] } ] }, { "id": 149997, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T13:42:35.467000", "content": "Also, one more follow up! Trust your local CV! I make 2 subs at the end, my best local CV, and my best public LB. The best public LB did a lot worse on the private LB, while my local CV still did pretty well.
\n\nIn hindsight, when my local CV started to diverge from the public LB, I should have stopped looking at the LB.
", "votes": 1, "replies": [] }, { "id": 149852, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:30:58.897000", "content": "Also, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.
", "votes": 1, "replies": [ { "id": 149889, "author_name": "SkyLibrary", "author_url": "", "post_date": "2016-12-13T04:21:23.527000", "content": "Thanks for sharing, never thought about using highway layers myself...\nCongrats on becoming grandmaster!
\n\n[quote=Zach;149852]
\n\nAlso, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.
\n\n[/quote]
", "votes": 2, "replies": [ { "id": 149998, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T13:43:07.920000", "content": "[quote=SkyLibrary;149889]\nCongrats on becoming grandmaster!\n[/quote]
\n\nThanks!
", "votes": 3, "replies": [] } ] } ] }, { "id": 149854, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:37:20.203000", "content": "Also, I think what got me to #4 was \"lots of (diverse) models\" + \"BFGS on the stacked predictions.\" Everything else is probably noise.
", "votes": 2, "replies": [] }, { "id": 150124, "author_name": "Yang Yang", "author_url": "", "post_date": "2016-12-13T23:37:51.233000", "content": "Sorry for the bad question and thank you again!
\n\nNever thought using PCA in this way , learned a lot!
", "votes": 0, "replies": [] }, { "id": 150067, "author_name": "Yang Yang", "author_url": "", "post_date": "2016-12-13T18:13:45.530000", "content": "Congratz! I really learned a lot from you.
\n\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?
\n\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)
\n\nThank you
", "votes": 0, "replies": [ { "id": 150072, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T18:46:08.207000", "content": "[quote=Yang Yang;150067]
\n\nCongratz! I really learned a lot from you.
\n\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?
\n\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)
\n\nThank you
\n\n[/quote]
\n\nI really only used PCA to add diversity to my level 1 models. (I wouldn't call it heavy usage). So most of my models were on the raw data. Some of my models were on the PCA transformed data. Some of my models were on the raw data + PCA variables, etc.
\n\nPCA is a different way of representing the data, so you get a different model. Even if you lose information, the new model can contribute to your ensemble.
\n\nUsing it in ensembling is a red herring. I don't think adding the PC1 component to my ensemble made much of a difference. The optimizer mostly used the model's stacked predicitons
", "votes": 1, "replies": [] } ] }, { "id": 150048, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T17:21:08.467000", "content": "If you buy a workstation, I'd say you want these things, in this order:
\n\nHi Zach,
\n\nCongratulations on an impressive result. At a lot of your steps, I found myself asking 'why?' Then that lead me to the question 'how much of this could be automated?'
\n\nPs. My brother graduated from Dartmouth in Econ.
", "votes": 0, "replies": [ { "id": 150030, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T16:13:03.050000", "content": "[quote=RickyTall;150028]\n'how much of this could be automated?'\n[/quote]
\n\nI work for a company that tries to automate most of these steps :-D
", "votes": 3, "replies": [] } ] }, { "id": 149850, "author_name": "Yizhi", "author_url": "", "post_date": "2016-12-13T02:28:38.400000", "content": "What does step6 mean? And how many stacked predictions you got for ensemble? Thanks.
", "votes": 0, "replies": [ { "id": 149853, "author_name": "Zach", "author_url": "", "post_date": "2016-12-13T02:32:49.190000", "content": "[quote=Yizhi Tao;149850]
\n\nWhat does step6 mean? And how many stacked predictions you got for ensemble? Thanks.
\n\n[/quote]
\n\nI did PCA on the stacked predicitons, and then added the first principle component to the stacked predictions before optimizing. I had 59 models in my final blend + the first PC component of those models.
", "votes": 1, "replies": [] } ] }, { "id": 149847, "author_name": "modkzs", "author_url": "", "post_date": "2016-12-13T02:04:51.620000", "content": "Could you explain what's PC1?
", "votes": 0, "replies": [ { "id": 149848, "author_name": "Vladimir Iglovikov", "author_url": "", "post_date": "2016-12-13T02:05:44.247000", "content": "[quote=modkzs;149847]
\n\nCould you explain what's PC1?
\n\n[/quote]\nFirst principal component?
", "votes": 1, "replies": [] } ] } ], "comment_count": 27, "messages": [ { "id": 149844, "postDate": "2016-12-13T01:57:04.980Z", "content": "I'm really curious to hear from the rest of the top 10, but I'll go first:
\n\nOnce I see the public vs private LB for all my subs, I'll have more details on what helped or what hurt.
", "rawMarkdown": "I'm really curious to hear from the rest of the top 10, but I'll go first:\r\n\r\n1. First, I started with the best public script, in particular the XGBoost scripts and keras scripts that were floating around. Picking the best tuned single model didn't seem to matter a whole lot, e.g. I used one of the the PB LB 1108 XG scripts and never got around to running the 1106 one (maybe that was my mistake!). I edited most of the keras scripts to run 10 CV folds and 10 bags, and most of the XGBoost scripts to use 10 folds and no bags.\r\n2. I modified each public script to generate stacked predictions, if it didn't already. All the keras script generated stacked predictions, but I had to modify some of the XGBoost scripts to generate stacked predictions. This was key.\r\n3. Next, I generated a bunch of different datasets to run through those scripts. This was the part where I got a bit creative. I can go into more detail here later, but I did things like PCA on all the numeric variables, PCA on all the categorical variables, PCA on all the numeric + categorical variables, all the numeric variables differences and ratios, all the categorical variable interactions, etc.\r\n4. Steps #2 and #3 were my core loop. Generate new datasets, and then run them through my XGBoost and Keras scripts to generate stacked predictions and test-set predictions. New datasets usually gave me more lift than tuning my models.\r\n5. After a few iterations, I'd end up with a bunch of models with stacked predictions and test-set predictions. Before any ensembling, I ran these models through wilcox.test in R, to make sure the stacked predictions were similar to the test set predictions. Anything with a p.value of less than .10 I dropped from my ensemble. Sometimes this step indicated models where I'd done something silly, like forgetting to un-transform the test-set predictions.\r\n6. Then I took the set of remaining stacked predictions, and ran PCA on them. I added the first PCA component to the stacked predictions and the test set predictions.\r\n7. Now for the key step. I ran the stacked predictions through the optim() function in R, to find linear weights that minimized MAE. I used the BFGS solver for optim.\r\n8. Later in the competition, I split the stacked predictions into thirds, based on the PC1 component, and used optim on each third. I then applied these 3 sets of weights to the test set, based on the PC1 component for the test set. I'm not sure if this helped or hurt me at the end of the competition; it may have been unnecessary complexity.\r\n9. I also added some L1 regularization to my call to the optim function. I simple added mean(abs(w))*(a penalty factor) to the objective function for optimizing the weights. I'm pretty sure this regularization helped at the end of the competition.\r\n10. I made 2 subs at the end: my best leaderboard score, and my best local CV score. I think my best local CV score ended up being the best on the private LB. Based on the small shakeup at the end of the competition, I'm happy I made one sub with my best local CV. Trust your cross-validation!\r\n \r\nOnce I see the public vs private LB for all my subs, I'll have more details on what helped or what hurt.", "votes": 50 }, { "id": 149860, "postDate": "2016-12-13T02:57:45.977Z", "content": "Also, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
", "rawMarkdown": "Also, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.\r\n\r\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.\r\n\r\n(and use git!)", "votes": 9, "replies": [ { "id": 149866, "postDate": "2016-12-13T03:01:43.500Z", "content": "[quote=Zach;149860]
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
\n\n[/quote]
\n\nLink to the repository? :)
", "rawMarkdown": "[quote=Zach;149860]\r\n\r\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.\r\n\r\n(and use git!)\r\n\r\n[/quote]\r\n\r\nLink to the repository? :)", "votes": 1, "replies": [ { "id": 149874, "postDate": "2016-12-13T03:15:03.113Z", "content": "[quote=Vladimir Iglovikov;149866]\nLink to the repository? :)\n[/quote]
\n\nI gotta put it all up on github (was using a local repo).
", "rawMarkdown": "[quote=Vladimir Iglovikov;149866]\r\nLink to the repository? :)\r\n[/quote]\r\n\r\nI gotta put it all up on github (was using a local repo). \r\n", "votes": 2 } ] }, { "id": 149869, "postDate": "2016-12-13T03:08:57.097Z", "content": "[quote=Zach;149860]
\n\nAlso, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.
\n\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.
\n\n(and use git!)
\n\n[/quote]
\n\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...
\n\nThat's why I never could publish a nice post-mortem for this competition like Vladimir did. Props for that, by the way.
", "rawMarkdown": "[quote=Zach;149860]\r\n\r\nAlso, another piece of advice: don't be afraid to be messy. My final folder for this competition has something like 100 r and python scripts in it. I basically made a new script for everything I tried. This made it a lot easier to backtrack when I needed to.\r\n\r\nTrying to put every single model you run into one script isn't going to work. Do one model per script, and do your stacking in another script.\r\n\r\n(and use git!)\r\n\r\n[/quote]\r\n\r\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\r\n\r\nThat's why I never could publish a nice post-mortem for this competition like Vladimir did. Props for that, by the way.\r\n", "votes": 1, "replies": [ { "id": 149870, "postDate": "2016-12-13T03:11:46.440Z", "content": "[quote=Tilii;149869]\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\n[/quote]
\n\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/
", "rawMarkdown": "[quote=Tilii;149869]\r\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\r\n[/quote]\r\n\r\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/", "votes": 1 }, { "id": 149877, "postDate": "2016-12-13T03:20:53.577Z", "content": "[quote=Zach;149870]
\n\n[quote=Tilii;149869]\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\n[/quote]
\n\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/
\n\n[/quote]
\n\nYou just made $20 for Tim Harford. Let's hope I read it before the holidays are over, or else I'll start the New Year with my guilt undiminished over the state of my garage, basement, office ...
", "rawMarkdown": "[quote=Zach;149870]\r\n\r\n[quote=Tilii;149869]\r\nMy general approach is the same, except that your \"messy\" is probably a Russian marching line compared to my \"messy.\" Wait, I shouldn't be bragging about this ...\r\n[/quote]\r\n\r\nThis is the first book on my holiday reading list, lol: http://timharford.com/books/messy/\r\n\r\n[/quote]\r\n\r\nYou just made $20 for Tim Harford. Let's hope I read it before the holidays are over, or else I'll start the New Year with my guilt undiminished over the state of my garage, basement, office ...\r\n", "votes": 2 } ] } ] }, { "id": 149863, "postDate": "2016-12-13T02:58:55.620Z", "content": "Also, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.
", "rawMarkdown": "Also, I let my weights go negative. I think since all my models were so highly correlated, allowing negative weights allowed the optimizer to find a better solution.", "votes": 3 }, { "id": 149846, "postDate": "2016-12-13T02:03:05.567Z", "content": "[quote=Vladimir Iglovikov;149846]
\n\n[/quote]
\n\n/edit: changed 90% to 95%. I relied pretty heavily on XGB + keras, but I did tweak the keras scripts (e.g. adding highway layers to some).
", "rawMarkdown": "[quote=Vladimir Iglovikov;149846]\r\n\r\n 1. Just to verify, only models that you used are xgb and NN? \r\n 2. Do you have statistics on how different version of the dataset affected first level model performance?\r\n 3. What is your best final CV score?\r\n 4. Did you come to this idea of using PCA during stacking in this competition or it is based on the previous experience with some other problem that I can read about? \r\n\r\n[/quote]\r\n\r\n1. I also ran some glmnet models on some of my datasets, but I think 95% of my score was XGB and keras scripts.\r\n2. I don't. Many of my datasets gave worst first level models but still helped the ensemble.\r\n3. 1115.2134 +/- 8.9\r\n4. I thought that doing PCA, and then optimizing on the PCA components would help the BFGS solver. It didn't. But then adding the first PCA component to the mix did, so I kept that.\r\n\r\n/edit: changed 90% to 95%. I relied pretty heavily on XGB + keras, but I did tweak the keras scripts (e.g. adding highway layers to some).\r\n", "votes": 3 } ] }, { "id": 149845, "postDate": "2016-12-13T02:02:03.763Z", "content": "One follow up: I started with a simple stacking model: BFGS to find the weights that minimize MAE. I then slowly made that stack more complicated, first by adding the PC1 component, then by adding L1 regularization, and finally splitting into thirds based on the first PC component.
", "rawMarkdown": "One follow up: I started with a simple stacking model: BFGS to find the weights that minimize MAE. I then slowly made that stack more complicated, first by adding the PC1 component, then by adding L1 regularization, and finally splitting into thirds based on the first PC component.", "votes": 3 }, { "id": 150037, "postDate": "2016-12-13T16:51:05.027Z", "content": "Congrats on making it to Grand Master, Zach!
\n\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.
", "rawMarkdown": "Congrats on making it to Grand Master, Zach!\r\n\r\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.", "votes": 1, "replies": [ { "id": 150044, "postDate": "2016-12-13T17:16:05.377Z", "content": "[quote=Ben Gorman;150037]
\n\nCongrats on making it to Grand Master, Zach!
\n\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.
\n\n[/quote]
\n\nThanks! I did all my data prep and ran all my ensembles on my 2 year old macbook pro, with a core i7 processor and 16 GB of RAM. Macs have really good memory management, and I've been really happy with this laptop for kaggle (unfortunately it has an AMD, rather than NVIDIA GPU).
\n\nI also ran a lot of XGBoost models on my laptop.
\n\nFor the keras models, I ran them on a workstation with an NVIDIA GPU, which I also used to run a bunch of variants of XGBoost towards the end of the competition.
", "rawMarkdown": "[quote=Ben Gorman;150037]\r\n\r\nCongrats on making it to Grand Master, Zach!\r\n\r\nWhat kind of hardware setup did you use? I entered the competition late with the expectation of running everything on my laptop - bad idea. Trying to decide if I should learn AWS, another cloud service, or just buy a second machine for running models that take hours to train.\r\n\r\n[/quote]\r\n\r\nThanks! I did all my data prep and ran all my ensembles on my 2 year old macbook pro, with a core i7 processor and 16 GB of RAM. [Macs have really good memory management][1], and I've been really happy with this laptop for kaggle (unfortunately it has an AMD, rather than NVIDIA GPU).\r\n\r\nI also ran a lot of XGBoost models on my laptop.\r\n\r\nFor the keras models, I ran them on a workstation with an NVIDIA GPU, which I also used to run a bunch of variants of XGBoost towards the end of the competition.\r\n\r\n\r\n\r\n\r\n [1]: http://arstechnica.com/apple/2013/10/os-x-10-9/17/", "votes": 2 } ] }, { "id": 149997, "postDate": "2016-12-13T13:42:35.467Z", "content": "Also, one more follow up! Trust your local CV! I make 2 subs at the end, my best local CV, and my best public LB. The best public LB did a lot worse on the private LB, while my local CV still did pretty well.
\n\nIn hindsight, when my local CV started to diverge from the public LB, I should have stopped looking at the LB.
", "rawMarkdown": "Also, one more follow up! Trust your local CV! I make 2 subs at the end, my best local CV, and my best public LB. The best public LB did a lot worse on the private LB, while my local CV still did pretty well.\r\n\r\nIn hindsight, when my local CV started to diverge from the public LB, I should have stopped looking at the LB.", "votes": 1 }, { "id": 149852, "postDate": "2016-12-13T02:30:58.897Z", "content": "Also, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.
", "rawMarkdown": "Also, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.", "votes": 1, "replies": [ { "id": 149889, "postDate": "2016-12-13T04:21:23.527Z", "content": "Thanks for sharing, never thought about using highway layers myself...\nCongrats on becoming grandmaster!
\n\n[quote=Zach;149852]
\n\nAlso, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.
\n\n[/quote]
", "rawMarkdown": "Thanks for sharing, never thought about using highway layers myself...\r\nCongrats on becoming grandmaster!\r\n\r\n[quote=Zach;149852]\r\n\r\nAlso, I played around with some of the keras scripts to generate new models. One thing that I think helped a little was adding ~20 highway layers after the 3 layer public script. Again, this model itself was worse on the LB, but contributed to my blend.\r\n\r\n[/quote]\r\n\r\n", "votes": 2, "replies": [ { "id": 149998, "postDate": "2016-12-13T13:43:07.920Z", "content": "[quote=SkyLibrary;149889]\nCongrats on becoming grandmaster!\n[/quote]
\n\nThanks!
", "rawMarkdown": "[quote=SkyLibrary;149889]\r\nCongrats on becoming grandmaster!\r\n[/quote]\r\n\r\nThanks!", "votes": 3 } ] } ] }, { "id": 149854, "postDate": "2016-12-13T02:37:20.203Z", "content": "Also, I think what got me to #4 was \"lots of (diverse) models\" + \"BFGS on the stacked predictions.\" Everything else is probably noise.
", "rawMarkdown": "Also, I think what got me to #4 was \"lots of (diverse) models\" + \"BFGS on the stacked predictions.\" Everything else is probably noise.", "votes": 2 }, { "id": 150124, "postDate": "2016-12-13T23:37:51.233Z", "content": "Sorry for the bad question and thank you again!
\n\nNever thought using PCA in this way , learned a lot!
", "rawMarkdown": "Sorry for the bad question and thank you again!\r\n\r\nNever thought using PCA in this way , learned a lot!" }, { "id": 150067, "postDate": "2016-12-13T18:13:45.530Z", "content": "Congratz! I really learned a lot from you.
\n\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?
\n\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)
\n\nThank you
", "rawMarkdown": "Congratz! I really learned a lot from you.\r\n\r\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?\r\n\r\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)\r\n\r\nThank you", "replies": [ { "id": 150072, "postDate": "2016-12-13T18:46:08.207Z", "content": "[quote=Yang Yang;150067]
\n\nCongratz! I really learned a lot from you.
\n\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?
\n\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)
\n\nThank you
\n\n[/quote]
\n\nI really only used PCA to add diversity to my level 1 models. (I wouldn't call it heavy usage). So most of my models were on the raw data. Some of my models were on the PCA transformed data. Some of my models were on the raw data + PCA variables, etc.
\n\nPCA is a different way of representing the data, so you get a different model. Even if you lose information, the new model can contribute to your ensemble.
\n\nUsing it in ensembling is a red herring. I don't think adding the PC1 component to my ensemble made much of a difference. The optimizer mostly used the model's stacked predicitons
", "rawMarkdown": "[quote=Yang Yang;150067]\r\n\r\nCongratz! I really learned a lot from you.\r\n\r\nWill you explain a little bit of the rationale of why you heavily used PCA in this competition (generate new date set & 1st PC used in ensembling etc ) rather than try other different models in the first level?\r\n\r\nI thought using dimension reduction like PCA will lose information and it's better not to use it if not for the limit of computational capability.( i am new with it :)\r\n\r\nThank you\r\n\r\n[/quote]\r\n\r\nI really only used PCA to add diversity to my level 1 models. (I wouldn't call it heavy usage). So most of my models were on the raw data. Some of my models were on the PCA transformed data. Some of my models were on the raw data + PCA variables, etc.\r\n\r\nPCA is a different way of representing the data, so you get a different model. Even if you lose information, the new model can contribute to your ensemble.\r\n\r\nUsing it in ensembling is a red herring. I don't think adding the PC1 component to my ensemble made much of a difference. The optimizer mostly used the model's stacked predicitons", "votes": 1 } ] }, { "id": 150048, "postDate": "2016-12-13T17:21:08.467Z", "content": "If you buy a workstation, I'd say you want these things, in this order:
\n\nHi Zach,
\n\nCongratulations on an impressive result. At a lot of your steps, I found myself asking 'why?' Then that lead me to the question 'how much of this could be automated?'
\n\nPs. My brother graduated from Dartmouth in Econ.
", "rawMarkdown": "Hi Zach, \r\n\r\nCongratulations on an impressive result. At a lot of your steps, I found myself asking 'why?' Then that lead me to the question 'how much of this could be automated?'\r\n\r\nPs. My brother graduated from Dartmouth in Econ. ", "replies": [ { "id": 150030, "postDate": "2016-12-13T16:13:03.050Z", "content": "[quote=RickyTall;150028]\n'how much of this could be automated?'\n[/quote]
\n\nI work for a company that tries to automate most of these steps :-D
", "rawMarkdown": "[quote=RickyTall;150028]\r\n'how much of this could be automated?'\r\n[/quote]\r\n\r\nI work for a company that tries to automate most of these steps :-D", "votes": 3 } ] }, { "id": 149850, "postDate": "2016-12-13T02:28:38.400Z", "content": "What does step6 mean? And how many stacked predictions you got for ensemble? Thanks.
", "rawMarkdown": "What does step6 mean? And how many stacked predictions you got for ensemble? Thanks.", "replies": [ { "id": 149853, "postDate": "2016-12-13T02:32:49.190Z", "content": "[quote=Yizhi Tao;149850]
\n\nWhat does step6 mean? And how many stacked predictions you got for ensemble? Thanks.
\n\n[/quote]
\n\nI did PCA on the stacked predicitons, and then added the first principle component to the stacked predictions before optimizing. I had 59 models in my final blend + the first PC component of those models.
", "rawMarkdown": "[quote=Yizhi Tao;149850]\r\n\r\nWhat does step6 mean? And how many stacked predictions you got for ensemble? Thanks.\r\n\r\n[/quote]\r\n\r\nI did PCA on the stacked predicitons, and then added the first principle component to the stacked predictions before optimizing. I had 59 models in my final blend + the first PC component of those models.", "votes": 1 } ] }, { "id": 149847, "postDate": "2016-12-13T02:04:51.620Z", "content": "Could you explain what's PC1?
", "rawMarkdown": "Could you explain what's PC1?", "replies": [ { "id": 149848, "postDate": "2016-12-13T02:05:44.247Z", "content": "[quote=modkzs;149847]
\n\nCould you explain what's PC1?
\n\n[/quote]\nFirst principal component?
", "rawMarkdown": "[quote=modkzs;149847]\r\n\r\nCould you explain what's PC1?\r\n\r\n[/quote]\r\nFirst principal component?", "votes": 1 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T17:15:26Z", "competition_id": 5325, "competition_slug": "allstate-claims-severity", "team_id": 384857, "solution_url": "/competitions/allstate-claims-severity/writeups/zach-my-approach" }