In our team, we decided to figure out our own approaches. I developed the CV set and that was rock solid. It was taken as the last 2 weeks (like test) and I analyzed how many routes, products, clients and agencies appeared in test that did not appear in train. Then I just took the last two weeks and stripped data away from left-over training data (there was plenty anyway), to effectively remove products, clients and routes that had similar distributions. My CV was +0.02 vs. the LB and against private it was +0.004. The difference may have been that the number of unseen clients was quite a bit higher.
\n\nIn my solution, I ended up using online regression learning from vowpal wabbit, so my training target was preconverted to log(1+x), then unconverted for the predictions, so I could use straightforward rmse. I used some basic features and made an assumption which week was halfway through the month, the first week of the month and the last week of the month and gave them binary indicators.
\n\nMy feature pipeline could be built pretty fast. I had a bunch of python scripts that only calculate 2-3 features for a specific purpose, saved them as a new set (against CV+subset train, train+test, so 4 times), then with an app in C++ I could combine all features into a single file for training or validation in the VW format.
\n\nUsing another python app, I converted the vw format to libsvm and ran a 150-round XGB model on that. In our best results, XGB and VW scored 0.512-0.517, but only had 0.87 correlation. A ridge regression over that with 2nd polynomials gave us a 0.481 CV score, which lead to 0.46047 on public LB and 0.474 on private. I expected that, because I noticed that in every week, new products and clients pop up and those are the difficult bit to predict for (unseen data).
\n\nMy approach revolved around a nearest neighbor for clients, where I used a "type" based on the name (XOXO, walmart, etc) as a binary id, a log of number of product types purchased, state id and some others. I removed "NO IDENTIFICADO" from the train set for NN, because they were too widely spread.
\n\nFor products, I used "product similarity" based on the name, a simple dice distance. So "bag of chips" would get close to "bag of crisps". The idea was that little variations in the # grams would allow that product to get close to similar products. Maybe I should have done that on weight / pieces instead.
\n\nIn the end I noticed that NN would still produce unreliable results, so I also included a k-means cluster, which did look at specific client types. It looks at the number of products purchased per known client and produce 20 clusters.
\n\nThe reasoning behind that is there are some clients that typically buy a lot of products, but some end up to be quite durable perhaps? So in a 7 week period, maybe some products only get bought after 6+ weeks or so. That makes the data pretty sparse for those product types and I was looking for a way to derive what the demand should be by looking in the same cluster.
\n\nI found it really difficult to improve on the end result because of the large number of clients and products and that's where I put my focus. I expect that people at the top would have create separate models for clients and products and used them as features in the main model to improve on results. The only thing I used were avg/mean for a range of different combo's (agency/product, etc.).
", "messages": [ { "id": "133067", "postDate": "08/31/2016 06:01:27", "content": "In our team, we decided to figure out our own approaches. I developed the CV set and that was rock solid. It was taken as the last 2 weeks (like test) and I analyzed how many routes, products, clients and agencies appeared in test that did not appear in train. Then I just took the last two weeks and stripped data away from left-over training data (there was plenty anyway), to effectively remove products, clients and routes that had similar distributions. My CV was +0.02 vs. the LB and against private it was +0.004. The difference may have been that the number of unseen clients was quite a bit higher.
\n\nIn my solution, I ended up using online regression learning from vowpal wabbit, so my training target was preconverted to log(1+x), then unconverted for the predictions, so I could use straightforward rmse. I used some basic features and made an assumption which week was halfway through the month, the first week of the month and the last week of the month and gave them binary indicators.
\n\nMy feature pipeline could be built pretty fast. I had a bunch of python scripts that only calculate 2-3 features for a specific purpose, saved them as a new set (against CV+subset train, train+test, so 4 times), then with an app in C++ I could combine all features into a single file for training or validation in the VW format.
\n\nUsing another python app, I converted the vw format to libsvm and ran a 150-round XGB model on that. In our best results, XGB and VW scored 0.512-0.517, but only had 0.87 correlation. A ridge regression over that with 2nd polynomials gave us a 0.481 CV score, which lead to 0.46047 on public LB and 0.474 on private. I expected that, because I noticed that in every week, new products and clients pop up and those are the difficult bit to predict for (unseen data).
\n\nMy approach revolved around a nearest neighbor for clients, where I used a "type" based on the name (XOXO, walmart, etc) as a binary id, a log of number of product types purchased, state id and some others. I removed "NO IDENTIFICADO" from the train set for NN, because they were too widely spread.
\n\nFor products, I used "product similarity" based on the name, a simple dice distance. So "bag of chips" would get close to "bag of crisps". The idea was that little variations in the # grams would allow that product to get close to similar products. Maybe I should have done that on weight / pieces instead.
\n\nIn the end I noticed that NN would still produce unreliable results, so I also included a k-means cluster, which did look at specific client types. It looks at the number of products purchased per known client and produce 20 clusters.
\n\nThe reasoning behind that is there are some clients that typically buy a lot of products, but some end up to be quite durable perhaps? So in a 7 week period, maybe some products only get bought after 6+ weeks or so. That makes the data pretty sparse for those product types and I was looking for a way to derive what the demand should be by looking in the same cluster.
\n\nI found it really difficult to improve on the end result because of the large number of clients and products and that's where I put my focus. I expect that people at the top would have create separate models for clients and products and used them as features in the main model to improve on results. The only thing I used were avg/mean for a range of different combo's (agency/product, etc.).
", "rawMarkdown": "In our team, we decided to figure out our own approaches. I developed the CV set and that was rock solid. It was taken as the last 2 weeks (like test) and I analyzed how many routes, products, clients and agencies appeared in test that did not appear in train. Then I just took the last two weeks and stripped data away from left-over training data (there was plenty anyway), to effectively remove products, clients and routes that had similar distributions. My CV was +0.02 vs. the LB and against private it was +0.004. The difference may have been that the number of unseen clients was quite a bit higher.\r\n\r\nIn my solution, I ended up using online regression learning from vowpal wabbit, so my training target was preconverted to log(1+x), then unconverted for the predictions, so I could use straightforward rmse. I used some basic features and made an assumption which week was halfway through the month, the first week of the month and the last week of the month and gave them binary indicators. \r\n\r\nMy feature pipeline could be built pretty fast. I had a bunch of python scripts that only calculate 2-3 features for a specific purpose, saved them as a new set (against CV+subset train, train+test, so 4 times), then with an app in C++ I could combine all features into a single file for training or validation in the VW format.\r\n\r\nUsing another python app, I converted the vw format to libsvm and ran a 150-round XGB model on that. In our best results, XGB and VW scored 0.512-0.517, but only had 0.87 correlation. A ridge regression over that with 2nd polynomials gave us a 0.481 CV score, which lead to 0.46047 on public LB and 0.474 on private. I expected that, because I noticed that in every week, new products and clients pop up and those are the difficult bit to predict for (unseen data). \r\n\r\nMy approach revolved around a nearest neighbor for clients, where I used a \"type\" based on the name (XOXO, walmart, etc) as a binary id, a log of number of product types purchased, state id and some others. I removed \"NO IDENTIFICADO\" from the train set for NN, because they were too widely spread. \r\n\r\nFor products, I used \"product similarity\" based on the name, a simple dice distance. So \"bag of chips\" would get close to \"bag of crisps\". The idea was that little variations in the # grams would allow that product to get close to similar products. Maybe I should have done that on weight / pieces instead. \r\n\r\nIn the end I noticed that NN would still produce unreliable results, so I also included a k-means cluster, which did look at specific client types. It looks at the number of products purchased per known client and produce 20 clusters. \r\n\r\nThe reasoning behind that is there are some clients that typically buy a lot of products, but some end up to be quite durable perhaps? So in a 7 week period, maybe some products only get bought after 6+ weeks or so. That makes the data pretty sparse for those product types and I was looking for a way to derive what the demand should be by looking in the same cluster.\r\n\r\nI found it really difficult to improve on the end result because of the large number of clients and products and that's where I put my focus. I expect that people at the top would have create separate models for clients and products and used them as features in the main model to improve on results. The only thing I used were avg/mean for a range of different combo's (agency/product, etc.).", "votes": null }, { "id": "133088", "postDate": "08/31/2016 07:19:35", "content": "Thanks for sharing your approach. Could you show more detail about how to make CV set , and how to handle temporary files as they are too large to read and merge in momery ? Thanks in advanced.
", "rawMarkdown": "Thanks for sharing your approach. Could you show more detail about how to make CV set , and how to handle temporary files as they are too large to read and merge in momery ? Thanks in advanced.", "votes": null }, { "id": "133179", "postDate": "08/31/2016 20:24:57", "content": "CV took a week to compose. You can follow the entire strategy here:
\n\nhttps://drive.google.com/file/d/0Bz5_5lIVJmVUUmNVcXFjYlBTVGs/view?usp=sharing
\n\nBasically, because you take holdout from train, the frequencies of client id's that do not exist in train are different. So I investigated the intersections of client ID's, route ID's, product ID's to see how many are missing in test. For all of them, I create a list of ID's that I want to 'carve' out of the training set. The last couple of lines in the doc remove those ID's from the training set.
\n\nFor some datasets that have a large number of rows, pandas really slows down. When the number of columns is reasonably small, just process files line by line instead and parse the meaning through indices. You'd do that in a library file/function, not in every script file (obviously sometimes things change). Pass through a function from your script through the library that gets called for each line/record that it finds and you perform your calculations in there.
\n\nBecause all of this is line-by-line, you can execute this on a pretty simple machine, but probably 8GB is minimum. Sometimes you need to pass through the data twice: once to get average/statistics info, the second time to apply that to each training sample. The output record goes into the format you determine, whether that is libsvm, csv or vw. You can call that traina.txt, holdouta.txt, testa.txt, etc.
\n\nFrom C++, write a "feature builder" that takes in file streams for as many features spaces as you find, then just concatenate it all together into a single training file. Examples:
\n\nFeature A:\n |a a1_1 a2_1 a3_1\nfeature B:\n |b b1:0.2 b2:1.5
\n\nAfter builder:\n |a a1_1 a2_1 a3_1 |b b1:0.2 b2:1.5 ( --> |p p1:0.2 p2:0.5 etc.... )
\n\nAt the end shuffle using "shuf" command on Linux.
\n\nThe highest memory needed is to fit the numbers for average calculation for product/client etc., which is probably around 5 GB or so.
\n\nVW works with this ok, but I ran XGB on a 64 GB machine, because the trees need more memory. Observation there: I didn't use pandas to read in csv file first in python to then make a copy to "XGB.Matrix"... I used the cmd line version of xgb to read in libsvm files directly and then process them. The behavior of the command line tools are very different, so still learning things there.
", "rawMarkdown": "CV took a week to compose. You can follow the entire strategy here:\r\n\r\nhttps://drive.google.com/file/d/0Bz5_5lIVJmVUUmNVcXFjYlBTVGs/view?usp=sharing\r\n\r\nBasically, because you take holdout from train, the frequencies of client id's that do not exist in train are different. So I investigated the intersections of client ID's, route ID's, product ID's to see how many are missing in test. For all of them, I create a list of ID's that I want to 'carve' out of the training set. The last couple of lines in the doc remove those ID's from the training set.\r\n\r\n\r\nFor some datasets that have a large number of rows, pandas really slows down. When the number of columns is reasonably small, just process files line by line instead and parse the meaning through indices. You'd do that in a library file/function, not in every script file (obviously sometimes things change). Pass through a function from your script through the library that gets called for each line/record that it finds and you perform your calculations in there. \r\n\r\nBecause all of this is line-by-line, you can execute this on a pretty simple machine, but probably 8GB is minimum. Sometimes you need to pass through the data twice: once to get average/statistics info, the second time to apply that to each training sample. The output record goes into the format you determine, whether that is libsvm, csv or vw. You can call that traina.txt, holdouta.txt, testa.txt, etc. \r\n\r\nFrom C++, write a \"feature builder\" that takes in file streams for as many features spaces as you find, then just concatenate it all together into a single training file. Examples:\r\n\r\nFeature A:\r\n |a a1_1 a2_1 a3_1\r\nfeature B:\r\n |b b1:0.2 b2:1.5\r\n\r\nAfter builder:\r\n |a a1_1 a2_1 a3_1 |b b1:0.2 b2:1.5 ( --> |p p1:0.2 p2:0.5 etc.... )\r\n\r\nAt the end shuffle using \"shuf\" command on Linux. \r\n\r\nThe highest memory needed is to fit the numbers for average calculation for product/client etc., which is probably around 5 GB or so.\r\n\r\nVW works with this ok, but I ran XGB on a 64 GB machine, because the trees need more memory. Observation there: I didn't use pandas to read in csv file first in python to then make a copy to \"XGB.Matrix\"... I used the cmd line version of xgb to read in libsvm files directly and then process them. The behavior of the command line tools are very different, so still learning things there.", "votes": null }, { "id": "133188", "postDate": "08/31/2016 20:58:39", "content": "[quote=Gerard Toonstra;133179]
\n\nAt the end shuffle using "shuf" command on Linux.
\n\n[/quote]
\n\nThanks @Gerard for sharing. Regarding the VW, I also tried to shuffle data, but getting worse score for VW. My best is to sort data in 4 orders:
\n\n-- original order
\n\n-- sort based on: \norder(train$Semana, train$Producto_ID, train$Cliente_ID, \n train$Agencia_ID, train$Ruta_SAK, train$Canal_ID)
\n\n-- sort based on: \norder(train$Producto_ID, train$Cliente_ID, train$Agencia_ID, \n train$Semana, train$Ruta_SAK, train$Canal_ID)
\n\n-- sort based on: \norder(train$Semana, train$Cliente_ID, train$Producto_ID, \n train$Ruta_SAK, train$Agencia_ID, train$Canal_ID)
\n\nAny other orders I tried only yield worse score. As beginner in VW, am I missing something about shuffling? Thank you.
\n\nAlso for VW FTRL I use log2 for demanda, instead of natural log for better score.
", "rawMarkdown": "[quote=Gerard Toonstra;133179]\r\n\r\nAt the end shuffle using \"shuf\" command on Linux. \r\n\r\n[/quote]\r\n\r\nThanks @Gerard for sharing. Regarding the VW, I also tried to shuffle data, but getting worse score for VW. My best is to sort data in 4 orders: \r\n\r\n-- original order\r\n\r\n-- sort based on: \r\norder(train$Semana, train$Producto_ID, train$Cliente_ID, \r\n train$Agencia_ID, train$Ruta_SAK, train$Canal_ID)\r\n\r\n-- sort based on: \r\norder(train$Producto_ID, train$Cliente_ID, train$Agencia_ID, \r\n train$Semana, train$Ruta_SAK, train$Canal_ID)\r\n\r\n-- sort based on: \r\norder(train$Semana, train$Cliente_ID, train$Producto_ID, \r\n train$Ruta_SAK, train$Agencia_ID, train$Canal_ID)\r\n\r\nAny other orders I tried only yield worse score. As beginner in VW, am I missing something about shuffling? Thank you. \r\n\r\nAlso for VW FTRL I use log2 for demanda, instead of natural log for better score.", "votes": null }, { "id": "133249", "postDate": "09/01/2016 03:46:16", "content": "@Gerard Toonstra , Thanks for replying so much in detail. I will take some time to absorb and practice.
", "rawMarkdown": "Gerard Toonstra , Thanks for replying so much in detail. I will take some time to absorb and practice.", "votes": null } ], "comments": [ { "id": 133088, "author_name": "soulcoder", "author_url": "", "post_date": "08/31/2016 07:19:35", "content": "Thanks for sharing your approach. Could you show more detail about how to make CV set , and how to handle temporary files as they are too large to read and merge in momery ? Thanks in advanced.
", "votes": null, "replies": [] }, { "id": 133179, "author_name": "remap1", "author_url": "", "post_date": "08/31/2016 20:24:57", "content": "CV took a week to compose. You can follow the entire strategy here:
\n\nhttps://drive.google.com/file/d/0Bz5_5lIVJmVUUmNVcXFjYlBTVGs/view?usp=sharing
\n\nBasically, because you take holdout from train, the frequencies of client id's that do not exist in train are different. So I investigated the intersections of client ID's, route ID's, product ID's to see how many are missing in test. For all of them, I create a list of ID's that I want to 'carve' out of the training set. The last couple of lines in the doc remove those ID's from the training set.
\n\nFor some datasets that have a large number of rows, pandas really slows down. When the number of columns is reasonably small, just process files line by line instead and parse the meaning through indices. You'd do that in a library file/function, not in every script file (obviously sometimes things change). Pass through a function from your script through the library that gets called for each line/record that it finds and you perform your calculations in there.
\n\nBecause all of this is line-by-line, you can execute this on a pretty simple machine, but probably 8GB is minimum. Sometimes you need to pass through the data twice: once to get average/statistics info, the second time to apply that to each training sample. The output record goes into the format you determine, whether that is libsvm, csv or vw. You can call that traina.txt, holdouta.txt, testa.txt, etc.
\n\nFrom C++, write a "feature builder" that takes in file streams for as many features spaces as you find, then just concatenate it all together into a single training file. Examples:
\n\nFeature A:\n |a a1_1 a2_1 a3_1\nfeature B:\n |b b1:0.2 b2:1.5
\n\nAfter builder:\n |a a1_1 a2_1 a3_1 |b b1:0.2 b2:1.5 ( --> |p p1:0.2 p2:0.5 etc.... )
\n\nAt the end shuffle using "shuf" command on Linux.
\n\nThe highest memory needed is to fit the numbers for average calculation for product/client etc., which is probably around 5 GB or so.
\n\nVW works with this ok, but I ran XGB on a 64 GB machine, because the trees need more memory. Observation there: I didn't use pandas to read in csv file first in python to then make a copy to "XGB.Matrix"... I used the cmd line version of xgb to read in libsvm files directly and then process them. The behavior of the command line tools are very different, so still learning things there.
", "votes": null, "replies": [] }, { "id": 133188, "author_name": "aldente", "author_url": "", "post_date": "08/31/2016 20:58:39", "content": "[quote=Gerard Toonstra;133179]
\n\nAt the end shuffle using "shuf" command on Linux.
\n\n[/quote]
\n\nThanks @Gerard for sharing. Regarding the VW, I also tried to shuffle data, but getting worse score for VW. My best is to sort data in 4 orders:
\n\n-- original order
\n\n-- sort based on: \norder(train$Semana, train$Producto_ID, train$Cliente_ID, \n train$Agencia_ID, train$Ruta_SAK, train$Canal_ID)
\n\n-- sort based on: \norder(train$Producto_ID, train$Cliente_ID, train$Agencia_ID, \n train$Semana, train$Ruta_SAK, train$Canal_ID)
\n\n-- sort based on: \norder(train$Semana, train$Cliente_ID, train$Producto_ID, \n train$Ruta_SAK, train$Agencia_ID, train$Canal_ID)
\n\nAny other orders I tried only yield worse score. As beginner in VW, am I missing something about shuffling? Thank you.
\n\nAlso for VW FTRL I use log2 for demanda, instead of natural log for better score.
", "votes": null, "replies": [] }, { "id": 133249, "author_name": "soulcoder", "author_url": "", "post_date": "09/01/2016 03:46:16", "content": "@Gerard Toonstra , Thanks for replying so much in detail. I will take some time to absorb and practice.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "133067": "In our team, we decided to figure out our own approaches. I developed the CV set and that was rock solid. It was taken as the last 2 weeks (like test) and I analyzed how many routes, products, clients and agencies appeared in test that did not appear in train. Then I just took the last two weeks and stripped data away from left-over training data (there was plenty anyway), to effectively remove products, clients and routes that had similar distributions. My CV was +0.02 vs. the LB and against private it was +0.004. The difference may have been that the number of unseen clients was quite a bit higher.\r\n\r\nIn my solution, I ended up using online regression learning from vowpal wabbit, so my training target was preconverted to log(1+x), then unconverted for the predictions, so I could use straightforward rmse. I used some basic features and made an assumption which week was halfway through the month, the first week of the month and the last week of the month and gave them binary indicators. \r\n\r\nMy feature pipeline could be built pretty fast. I had a bunch of python scripts that only calculate 2-3 features for a specific purpose, saved them as a new set (against CV+subset train, train+test, so 4 times), then with an app in C++ I could combine all features into a single file for training or validation in the VW format.\r\n\r\nUsing another python app, I converted the vw format to libsvm and ran a 150-round XGB model on that. In our best results, XGB and VW scored 0.512-0.517, but only had 0.87 correlation. A ridge regression over that with 2nd polynomials gave us a 0.481 CV score, which lead to 0.46047 on public LB and 0.474 on private. I expected that, because I noticed that in every week, new products and clients pop up and those are the difficult bit to predict for (unseen data). \r\n\r\nMy approach revolved around a nearest neighbor for clients, where I used a \"type\" based on the name (XOXO, walmart, etc) as a binary id, a log of number of product types purchased, state id and some others. I removed \"NO IDENTIFICADO\" from the train set for NN, because they were too widely spread. \r\n\r\nFor products, I used \"product similarity\" based on the name, a simple dice distance. So \"bag of chips\" would get close to \"bag of crisps\". The idea was that little variations in the # grams would allow that product to get close to similar products. Maybe I should have done that on weight / pieces instead. \r\n\r\nIn the end I noticed that NN would still produce unreliable results, so I also included a k-means cluster, which did look at specific client types. It looks at the number of products purchased per known client and produce 20 clusters. \r\n\r\nThe reasoning behind that is there are some clients that typically buy a lot of products, but some end up to be quite durable perhaps? So in a 7 week period, maybe some products only get bought after 6+ weeks or so. That makes the data pretty sparse for those product types and I was looking for a way to derive what the demand should be by looking in the same cluster.\r\n\r\nI found it really difficult to improve on the end result because of the large number of clients and products and that's where I put my focus. I expect that people at the top would have create separate models for clients and products and used them as features in the main model to improve on results. The only thing I used were avg/mean for a range of different combo's (agency/product, etc.).", "133088": "Thanks for sharing your approach. Could you show more detail about how to make CV set , and how to handle temporary files as they are too large to read and merge in momery ? Thanks in advanced.", "133179": "CV took a week to compose. You can follow the entire strategy here:\r\n\r\nhttps://drive.google.com/file/d/0Bz5_5lIVJmVUUmNVcXFjYlBTVGs/view?usp=sharing\r\n\r\nBasically, because you take holdout from train, the frequencies of client id's that do not exist in train are different. So I investigated the intersections of client ID's, route ID's, product ID's to see how many are missing in test. For all of them, I create a list of ID's that I want to 'carve' out of the training set. The last couple of lines in the doc remove those ID's from the training set.\r\n\r\n\r\nFor some datasets that have a large number of rows, pandas really slows down. When the number of columns is reasonably small, just process files line by line instead and parse the meaning through indices. You'd do that in a library file/function, not in every script file (obviously sometimes things change). Pass through a function from your script through the library that gets called for each line/record that it finds and you perform your calculations in there. \r\n\r\nBecause all of this is line-by-line, you can execute this on a pretty simple machine, but probably 8GB is minimum. Sometimes you need to pass through the data twice: once to get average/statistics info, the second time to apply that to each training sample. The output record goes into the format you determine, whether that is libsvm, csv or vw. You can call that traina.txt, holdouta.txt, testa.txt, etc. \r\n\r\nFrom C++, write a \"feature builder\" that takes in file streams for as many features spaces as you find, then just concatenate it all together into a single training file. Examples:\r\n\r\nFeature A:\r\n |a a1_1 a2_1 a3_1\r\nfeature B:\r\n |b b1:0.2 b2:1.5\r\n\r\nAfter builder:\r\n |a a1_1 a2_1 a3_1 |b b1:0.2 b2:1.5 ( --> |p p1:0.2 p2:0.5 etc.... )\r\n\r\nAt the end shuffle using \"shuf\" command on Linux. \r\n\r\nThe highest memory needed is to fit the numbers for average calculation for product/client etc., which is probably around 5 GB or so.\r\n\r\nVW works with this ok, but I ran XGB on a 64 GB machine, because the trees need more memory. Observation there: I didn't use pandas to read in csv file first in python to then make a copy to \"XGB.Matrix\"... I used the cmd line version of xgb to read in libsvm files directly and then process them. The behavior of the command line tools are very different, so still learning things there.", "133188": "[quote=Gerard Toonstra;133179]\r\n\r\nAt the end shuffle using \"shuf\" command on Linux. \r\n\r\n[/quote]\r\n\r\nThanks @Gerard for sharing. Regarding the VW, I also tried to shuffle data, but getting worse score for VW. My best is to sort data in 4 orders: \r\n\r\n-- original order\r\n\r\n-- sort based on: \r\norder(train$Semana, train$Producto_ID, train$Cliente_ID, \r\n train$Agencia_ID, train$Ruta_SAK, train$Canal_ID)\r\n\r\n-- sort based on: \r\norder(train$Producto_ID, train$Cliente_ID, train$Agencia_ID, \r\n train$Semana, train$Ruta_SAK, train$Canal_ID)\r\n\r\n-- sort based on: \r\norder(train$Semana, train$Cliente_ID, train$Producto_ID, \r\n train$Ruta_SAK, train$Agencia_ID, train$Canal_ID)\r\n\r\nAny other orders I tried only yield worse score. As beginner in VW, am I missing something about shuffling? Thank you. \r\n\r\nAlso for VW FTRL I use log2 for demanda, instead of natural log for better score.", "133249": "Gerard Toonstra , Thanks for replying so much in detail. I will take some time to absorb and practice." }, "source": "meta" }