I only trained on the rows from week 9. I used Apache Spark to create 12 features for those rows and XGBoost to train the model
\n\nPredicted demand for this week by linear regression FTRL model using the following features:
\n\nFor every row in week 9, I calculated the historic average demand (before week 9) that that client\nhad sold. Then I made tuples of the type (This_product_id,previously_sold_product_id) and hashed them. \nThe feature value was the average demand of the previously sold product. I also hashed the ID's in the row and gave them the feature value 1.0\nThe output value was the demand for the given row.
Predicted demand for this week by linear regression FTRL model using the following features:
\n\nFor all rows before week 9, I hashed all IDs and gave them the value 1.0.\nI also used poly4 interaction of the IDs and gave them the value 1.0\nThe output value was the demand for the given row.
I only trained on the rows from week 9. I used Apache Spark to create 12 features for those rows and XGBoost to train the model
\n\nPredicted demand for this week by linear regression FTRL model using the following features:
\n\nFor every row in week 9, I calculated the historic average demand (before week 9) that that client\nhad sold. Then I made tuples of the type (This_product_id,previously_sold_product_id) and hashed them. \nThe feature value was the average demand of the previously sold product. I also hashed the ID's in the row and gave them the feature value 1.0\nThe output value was the demand for the given row.
Predicted demand for this week by linear regression FTRL model using the following features:
\n\nFor all rows before week 9, I hashed all IDs and gave them the value 1.0.\nI also used poly4 interaction of the IDs and gave them the value 1.0\nThe output value was the demand for the given row.
Congratulations @ML-Bure! Almost the same as mine (24th)! But I didn't use your 9th, 10th, 11th features. Instead I used these more features: all IDs, weight, PCA. I also uses FTRL similar to your 12th feature (Vowpal Wabbit).
\n\nAnother difference: my model uses pairs of 2 weeks (ensemble), while yours uses only 1 week (week 9). Surprising that one week model performs so good, now I regret not thinking of that ;)
", "rawMarkdown": "Congratulations @ML-Bure! Almost the same as mine (24th)! But I didn't use your 9th, 10th, 11th features. Instead I used these more features: all IDs, weight, PCA. I also uses FTRL similar to your 12th feature (Vowpal Wabbit). \r\n\r\nAnother difference: my model uses pairs of 2 weeks (ensemble), while yours uses only 1 week (week 9). Surprising that one week model performs so good, now I regret not thinking of that ;)", "votes": null }, { "id": "133101", "postDate": "08/31/2016 08:50:17", "content": "Thank you, I learn so much.
", "rawMarkdown": "Thank you, I learn so much.", "votes": null }, { "id": "133113", "postDate": "08/31/2016 10:49:49", "content": "Hi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
", "rawMarkdown": "Hi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?", "votes": null }, { "id": "133122", "postDate": "08/31/2016 12:41:09", "content": "How'd you get the 1-week lag for week 10?
", "rawMarkdown": "How'd you get the 1-week lag for week 10?", "votes": null }, { "id": "133125", "postDate": "08/31/2016 13:12:01", "content": "[quote=Zach;133122]
\n\nHow'd you get the 1-week lag for week 10?
\n\n[/quote]
\n\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10
", "rawMarkdown": "[quote=Zach;133122]\r\n\r\nHow'd you get the 1-week lag for week 10?\r\n\r\n[/quote]\r\n\r\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10", "votes": null }, { "id": "133141", "postDate": "08/31/2016 15:03:54", "content": "Darn, such a simple solution for that I never thought of! Good job :)
\n\n(My best feature, accd to xgboost, was State ID's as a categorical value, which is covered by your per-city means so it probably wouldn't help your model too much)
", "rawMarkdown": "Darn, such a simple solution for that I never thought of! Good job :)\r\n\r\n(My best feature, accd to xgboost, was State ID's as a categorical value, which is covered by your per-city means so it probably wouldn't help your model too much)", "votes": null }, { "id": "133142", "postDate": "08/31/2016 15:05:07", "content": "[quote=ML-Bure;133125]
\n\n[quote=Zach;133122]
\n\nHow'd you get the 1-week lag for week 10?
\n\n[/quote]
\n\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10
\n\n[/quote]
\n\nThat could be problematic. I think you could do better than this, if you are choosing the right the validation strategy.
", "rawMarkdown": "[quote=ML-Bure;133125]\r\n\r\n[quote=Zach;133122]\r\n\r\nHow'd you get the 1-week lag for week 10?\r\n\r\n[/quote]\r\n\r\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10\r\n\r\n[/quote]\r\n\r\nThat could be problematic. I think you could do better than this, if you are choosing the right the validation strategy.", "votes": null }, { "id": "133165", "postDate": "08/31/2016 18:35:45", "content": "@ML-Bure, Congratulations.
\n\nCan you share your features importance scores. it would be very helpful to understand how important the features we missed.
", "rawMarkdown": "ML-Bure, Congratulations. \r\n\r\nCan you share your features importance scores. it would be very helpful to understand how important the features we missed.", "votes": null }, { "id": "133180", "postDate": "08/31/2016 20:27:07", "content": "[quote=leexa;133113]
\n\nHi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
\n\n[/quote]
\n\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use Incremental PCA with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM).
", "rawMarkdown": "[quote=leexa;133113]\r\n\r\nHi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?\r\n\r\n\r\n[/quote]\r\n\r\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use [Incremental PCA][1] with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM). \r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html", "votes": null }, { "id": "133233", "postDate": "09/01/2016 02:22:07", "content": "[quote=aldente;133180]
\n\n[quote=leexa;133113]
\n\nHi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
\n\n[/quote]
\n\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use Incremental PCA with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM).
\n\n[/quote]
\n\nThats interesting, and even though it seems weird, that it did work :), possibly because value the different categories themselves carried some information. I notice that demand was correlated with customer ID etc, with correlation plots.
\n\nI could try such an approache next time, instead of a tree based approach to get around the high cardinality ( RF and xgb).
", "rawMarkdown": "[quote=aldente;133180]\r\n\r\n[quote=leexa;133113]\r\n\r\nHi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?\r\n\r\n\r\n[/quote]\r\n\r\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use [Incremental PCA][1] with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM). \r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html\r\n\r\n[/quote]\r\n\r\nThats interesting, and even though it seems weird, that it did work :), possibly because value the different categories themselves carried some information. I notice that demand was correlated with customer ID etc, with correlation plots. \r\n\r\nI could try such an approache next time, instead of a tree based approach to get around the high cardinality ( RF and xgb).", "votes": null }, { "id": "133243", "postDate": "09/01/2016 03:14:09", "content": "@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit.
\n\n"..possibly because value the different categories themselves carried some information" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.
", "rawMarkdown": "leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit. \r\n\r\n\"*..possibly because value the different categories themselves carried some information*\" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.", "votes": null }, { "id": "133253", "postDate": "09/01/2016 04:02:56", "content": "[quote=aldente;133243]
\n\n@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit.
\n\n"..possibly because value the different categories themselves carried some information" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.
\n\n[/quote]\nThanks !,
\n\nI think this method will be good to read up more about.
", "rawMarkdown": "[quote=aldente;133243]\r\n\r\n@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit. \r\n\r\n\"*..possibly because value the different categories themselves carried some information*\" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables. \r\n\r\n[/quote]\r\nThanks !,\r\n\r\nI think this method will be good to read up more about.", "votes": null } ], "comments": [ { "id": 133092, "author_name": "aldente", "author_url": "", "post_date": "08/31/2016 07:48:58", "content": "Congratulations @ML-Bure! Almost the same as mine (24th)! But I didn't use your 9th, 10th, 11th features. Instead I used these more features: all IDs, weight, PCA. I also uses FTRL similar to your 12th feature (Vowpal Wabbit).
\n\nAnother difference: my model uses pairs of 2 weeks (ensemble), while yours uses only 1 week (week 9). Surprising that one week model performs so good, now I regret not thinking of that ;)
", "votes": null, "replies": [] }, { "id": 133101, "author_name": "soulcoder", "author_url": "", "post_date": "08/31/2016 08:50:17", "content": "Thank you, I learn so much.
", "votes": null, "replies": [] }, { "id": 133113, "author_name": "leexiongan", "author_url": "", "post_date": "08/31/2016 10:49:49", "content": "Hi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
", "votes": null, "replies": [] }, { "id": 133122, "author_name": "zachmayer", "author_url": "", "post_date": "08/31/2016 12:41:09", "content": "How'd you get the 1-week lag for week 10?
", "votes": null, "replies": [] }, { "id": 133125, "author_name": "mlbure", "author_url": "", "post_date": "08/31/2016 13:12:01", "content": "[quote=Zach;133122]
\n\nHow'd you get the 1-week lag for week 10?
\n\n[/quote]
\n\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10
", "votes": null, "replies": [] }, { "id": 133141, "author_name": "happycube", "author_url": "", "post_date": "08/31/2016 15:03:54", "content": "Darn, such a simple solution for that I never thought of! Good job :)
\n\n(My best feature, accd to xgboost, was State ID's as a categorical value, which is covered by your per-city means so it probably wouldn't help your model too much)
", "votes": null, "replies": [] }, { "id": 133142, "author_name": "amazingbob", "author_url": "", "post_date": "08/31/2016 15:05:07", "content": "[quote=ML-Bure;133125]
\n\n[quote=Zach;133122]
\n\nHow'd you get the 1-week lag for week 10?
\n\n[/quote]
\n\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10
\n\n[/quote]
\n\nThat could be problematic. I think you could do better than this, if you are choosing the right the validation strategy.
", "votes": null, "replies": [] }, { "id": 133165, "author_name": "kumaran5555", "author_url": "", "post_date": "08/31/2016 18:35:45", "content": "@ML-Bure, Congratulations.
\n\nCan you share your features importance scores. it would be very helpful to understand how important the features we missed.
", "votes": null, "replies": [] }, { "id": 133180, "author_name": "aldente", "author_url": "", "post_date": "08/31/2016 20:27:07", "content": "[quote=leexa;133113]
\n\nHi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
\n\n[/quote]
\n\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use Incremental PCA with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM).
", "votes": null, "replies": [] }, { "id": 133233, "author_name": "leexiongan", "author_url": "", "post_date": "09/01/2016 02:22:07", "content": "[quote=aldente;133180]
\n\n[quote=leexa;133113]
\n\nHi aldente,
\n\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?
\n\n[/quote]
\n\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use Incremental PCA with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM).
\n\n[/quote]
\n\nThats interesting, and even though it seems weird, that it did work :), possibly because value the different categories themselves carried some information. I notice that demand was correlated with customer ID etc, with correlation plots.
\n\nI could try such an approache next time, instead of a tree based approach to get around the high cardinality ( RF and xgb).
", "votes": null, "replies": [] }, { "id": 133243, "author_name": "aldente", "author_url": "", "post_date": "09/01/2016 03:14:09", "content": "@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit.
\n\n"..possibly because value the different categories themselves carried some information" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.
", "votes": null, "replies": [] }, { "id": 133253, "author_name": "leexiongan", "author_url": "", "post_date": "09/01/2016 04:02:56", "content": "[quote=aldente;133243]
\n\n@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit.
\n\n"..possibly because value the different categories themselves carried some information" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.
\n\n[/quote]\nThanks !,
\n\nI think this method will be good to read up more about.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133084": "I only trained on the rows from week 9. I used Apache Spark to create 12 features for those rows and XGBoost to train the model\r\n\r\n 1. Avg demand for this client\r\n 2. Avg demand for thhis product\r\n 3. Weighted average demand for this product, weighted by total demand for each client\r\n 4. Avg demand for this product in this city\r\n 5. Avg demand for this product at this client\r\n 6. Avg demand one weeks ago for this product at this client\r\n 7. Avg demand two weeks ago for this product at this client\r\n 8. Avg demand three weeks ago for this product at this client\r\n 9. Avg demand four weeks ago for this product at this client\r\n 10. Avg demand five weeks ago for this product at this client\r\n 11. Predicted demand for this week by linear regression FTRL model using the following features:\r\n\r\n\tFor every row in week 9, I calculated the historic average demand (before week 9) that that client\r\n\thad sold. Then I made tuples of the type (This_product_id,previously_sold_product_id) and hashed them. \r\n\tThe feature value was the average demand of the previously sold product. I also hashed the ID's in the row and gave them the feature value 1.0\r\n\tThe output value was the demand for the given row.\r\n 12. Predicted demand for this week by linear regression FTRL model using the following features:\r\n\r\n\tFor all rows before week 9, I hashed all IDs and gave them the value 1.0.\r\n\tI also used poly4 interaction of the IDs and gave them the value 1.0\r\n\tThe output value was the demand for the given row.", "133092": "Congratulations @ML-Bure! Almost the same as mine (24th)! But I didn't use your 9th, 10th, 11th features. Instead I used these more features: all IDs, weight, PCA. I also uses FTRL similar to your 12th feature (Vowpal Wabbit). \r\n\r\nAnother difference: my model uses pairs of 2 weeks (ensemble), while yours uses only 1 week (week 9). Surprising that one week model performs so good, now I regret not thinking of that ;)", "133101": "Thank you, I learn so much.", "133113": "Hi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?", "133122": "How'd you get the 1-week lag for week 10?", "133125": "[quote=Zach;133122]\r\n\r\nHow'd you get the 1-week lag for week 10?\r\n\r\n[/quote]\r\n\r\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10", "133141": "Darn, such a simple solution for that I never thought of! Good job :)\r\n\r\n(My best feature, accd to xgboost, was State ID's as a categorical value, which is covered by your per-city means so it probably wouldn't help your model too much)", "133142": "[quote=ML-Bure;133125]\r\n\r\n[quote=Zach;133122]\r\n\r\nHow'd you get the 1-week lag for week 10?\r\n\r\n[/quote]\r\n\r\nI used the same lagged features for week 10 and 11. I treated the entire test set as if it was week 10\r\n\r\n[/quote]\r\n\r\nThat could be problematic. I think you could do better than this, if you are choosing the right the validation strategy.", "133165": "ML-Bure, Congratulations. \r\n\r\nCan you share your features importance scores. it would be very helpful to understand how important the features we missed.", "133180": "[quote=leexa;133113]\r\n\r\nHi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?\r\n\r\n\r\n[/quote]\r\n\r\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use [Incremental PCA][1] with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM). \r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html", "133233": "[quote=aldente;133180]\r\n\r\n[quote=leexa;133113]\r\n\r\nHi aldente, \r\n\r\nI was wondering how u did pca (principle components analysis) on your data, especially for catogorical variables?\r\n\r\n\r\n[/quote]\r\n\r\n@leexa, I use data as they are, treating them as float numpy array (Semana, Producto_ID, ... ), as expected by sklearn. I use [Incremental PCA][1] with n_components=2. Only second component is important, first component is removed. I know this is not good approach, better approach maybe to do one hot encoding (but I only have low RAM). \r\n\r\n\r\n [1]: http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.IncrementalPCA.html\r\n\r\n[/quote]\r\n\r\nThats interesting, and even though it seems weird, that it did work :), possibly because value the different categories themselves carried some information. I notice that demand was correlated with customer ID etc, with correlation plots. \r\n\r\nI could try such an approache next time, instead of a tree based approach to get around the high cardinality ( RF and xgb).", "133243": "leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit. \r\n\r\n\"*..possibly because value the different categories themselves carried some information*\" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables.", "133253": "[quote=aldente;133243]\r\n\r\n@leexa, I've always used PCA (and/or SVD and/or tSNE) in previous competitions, one or two of the components were always good feature(s). Although it's not the strongest feature, but when I removed it the CV score getting worse a bit. \r\n\r\n\"*..possibly because value the different categories themselves carried some information*\" I guess so, e.g. see at the Scirpus' FTRL script how feature hashing performs good with the categorical variables. \r\n\r\n[/quote]\r\nThanks !,\r\n\r\nI think this method will be good to read up more about." }, "source": "meta" }