I investigated the distribution Num_Diffs of consecutive demands Diff_Semana = t2.Semana - t1.Semana (t1.Semana < t2.Semana) with Demanda_uni_equil > 0 for (Cliente_ID, Producto_ID)-tuples,\ndepending on demand classes Class_Demanda_uni_equil (0 < 0.2 < 0.5 < 1.0 < 2 < 5 < 10 < 100 < 1000) of demand averages Avg_Demanda_uni_equil = sum(Demanda_uni_equil)/7.
\n\nSee the SQL for Hive on the bottom.
\n\nTable of distribution Num_Diffs:
\n\nDiff_Semana 0,2 0,5 1 2 5 10 100 Total\n\n0 4,3% 6,3% 3,6% 1,2% 0,4% 0,2% 0,0% 15,96%\n\n1 0,0% 2,5% 9,6% 15,5% 23,1% 12,2% 10,8% 73,75%\n\n2 0,0% 0,0% 0,1% 0,2% 0,2% 0,1% 0,1% 0,81%\n\n3 0,0% 0,5% 1,7% 1,9% 1,7% 0,5% 0,3% 6,51%\n\n4 0,0% 0,2% 0,5% 0,4% 0,3% 0,1% 0,0% 1,60%\n\n5 0,0% 0,2% 0,4% 0,3% 0,2% 0,0% 0,0% 1,02%\n\n6 0,0% 0,1% 0,1% 0,1% 0,0% 0,0% 0,0% 0,34%\nNotes on Diff_Semana:
\n\nThe distribution Num_Diffs peaks on consecutive demands Diff_Semana = 1.\nWhat are the possible explanations?
\n\nselect\n SKUV.Class_Demanda_uni_equil,\n SKUS.Diff_Semana,\n count(*) as Num_Diffs\nfrom\n (\n select\n t1.Cliente_ID,\n t1.Producto_ID,\n t1.Semana,\n min(t2.Semana-t1.Semana) as Diff_Semana\n from\n BimboTrain t1\n join\n BimboTrain t2 on\n t2.Cliente_ID = t1.Cliente_ID and\n t2.Producto_ID = t1.Producto_ID\n where\n t2.Semana > t1.Semana and\n t1.Demanda_uni_equil > 0 and\n t2.Demanda_uni_equil > 0\n group by\n t1.Cliente_ID,\n t1.Producto_ID,\n t1.Semana\n union\n select\n t1.Cliente_ID,\n t1.Producto_ID,\n min(t1.Semana) as Semana,\n 0 as Diff_Semana\n from\n BimboTrain t1\n where\n t1.Demanda_uni_equil > 0\n group by\n t1.Cliente_ID,\n t1.Producto_ID\n having\n count(*) = 1\n ) SKUS\njoin\n (\n select\n Cliente_ID,\n Producto_ID,\n case\n when Avg_Demanda_uni_equil <= 0.0 then 0.0\n when Avg_Demanda_uni_equil > 0.0 and Avg_Demanda_uni_equil <= 0.2 then 0.2\n when Avg_Demanda_uni_equil > 0.2 and Avg_Demanda_uni_equil <= 0.5 then 0.5\n when Avg_Demanda_uni_equil > 0.5 and Avg_Demanda_uni_equil <= 1.0 then 1.0\n when Avg_Demanda_uni_equil > 1.0 and Avg_Demanda_uni_equil <= 2.0 then 2.0\n when Avg_Demanda_uni_equil > 2.0 and Avg_Demanda_uni_equil <= 5.0 then 5.0\n when Avg_Demanda_uni_equil > 5.0 and Avg_Demanda_uni_equil <= 10.0 then 10.0\n when Avg_Demanda_uni_equil > 10.0 and Avg_Demanda_uni_equil <= 100.0 then 100.0\n when Avg_Demanda_uni_equil > 100.0 then 1000.0\n end as Class_Demanda_uni_equil\n from\n (\n select\n Cliente_ID,\n Producto_ID,\n sum(Demanda_uni_equil)*1.0/7 as Avg_Demanda_uni_equil\n from\n BimboTrain\n group by\n Cliente_ID,\n Producto_ID\n ) AVUH\n ) SKUV on\n SKUV.Cliente_ID = SKUS.Cliente_ID and\n SKUV.Producto_ID = SKUS.Producto_ID\ngroup by\n SKUV.Class_Demanda_uni_equil,\n SKUS.Diff_Semana\nI investigated the distribution Num_Diffs of consecutive demands Diff_Semana = t2.Semana - t1.Semana (t1.Semana < t2.Semana) with Demanda_uni_equil > 0 for (Cliente_ID, Producto_ID)-tuples,\ndepending on demand classes Class_Demanda_uni_equil (0 < 0.2 < 0.5 < 1.0 < 2 < 5 < 10 < 100 < 1000) of demand averages Avg_Demanda_uni_equil = sum(Demanda_uni_equil)/7.
\n\nSee the SQL for Hive on the bottom.
\n\nTable of distribution Num_Diffs:
\n\nDiff_Semana 0,2 0,5 1 2 5 10 100 Total\n\n0 4,3% 6,3% 3,6% 1,2% 0,4% 0,2% 0,0% 15,96%\n\n1 0,0% 2,5% 9,6% 15,5% 23,1% 12,2% 10,8% 73,75%\n\n2 0,0% 0,0% 0,1% 0,2% 0,2% 0,1% 0,1% 0,81%\n\n3 0,0% 0,5% 1,7% 1,9% 1,7% 0,5% 0,3% 6,51%\n\n4 0,0% 0,2% 0,5% 0,4% 0,3% 0,1% 0,0% 1,60%\n\n5 0,0% 0,2% 0,4% 0,3% 0,2% 0,0% 0,0% 1,02%\n\n6 0,0% 0,1% 0,1% 0,1% 0,0% 0,0% 0,0% 0,34%\nNotes on Diff_Semana:
\n\nThe distribution Num_Diffs peaks on consecutive demands Diff_Semana = 1.\nWhat are the possible explanations?
\n\nselect\n SKUV.Class_Demanda_uni_equil,\n SKUS.Diff_Semana,\n count(*) as Num_Diffs\nfrom\n (\n select\n t1.Cliente_ID,\n t1.Producto_ID,\n t1.Semana,\n min(t2.Semana-t1.Semana) as Diff_Semana\n from\n BimboTrain t1\n join\n BimboTrain t2 on\n t2.Cliente_ID = t1.Cliente_ID and\n t2.Producto_ID = t1.Producto_ID\n where\n t2.Semana > t1.Semana and\n t1.Demanda_uni_equil > 0 and\n t2.Demanda_uni_equil > 0\n group by\n t1.Cliente_ID,\n t1.Producto_ID,\n t1.Semana\n union\n select\n t1.Cliente_ID,\n t1.Producto_ID,\n min(t1.Semana) as Semana,\n 0 as Diff_Semana\n from\n BimboTrain t1\n where\n t1.Demanda_uni_equil > 0\n group by\n t1.Cliente_ID,\n t1.Producto_ID\n having\n count(*) = 1\n ) SKUS\njoin\n (\n select\n Cliente_ID,\n Producto_ID,\n case\n when Avg_Demanda_uni_equil <= 0.0 then 0.0\n when Avg_Demanda_uni_equil > 0.0 and Avg_Demanda_uni_equil <= 0.2 then 0.2\n when Avg_Demanda_uni_equil > 0.2 and Avg_Demanda_uni_equil <= 0.5 then 0.5\n when Avg_Demanda_uni_equil > 0.5 and Avg_Demanda_uni_equil <= 1.0 then 1.0\n when Avg_Demanda_uni_equil > 1.0 and Avg_Demanda_uni_equil <= 2.0 then 2.0\n when Avg_Demanda_uni_equil > 2.0 and Avg_Demanda_uni_equil <= 5.0 then 5.0\n when Avg_Demanda_uni_equil > 5.0 and Avg_Demanda_uni_equil <= 10.0 then 10.0\n when Avg_Demanda_uni_equil > 10.0 and Avg_Demanda_uni_equil <= 100.0 then 100.0\n when Avg_Demanda_uni_equil > 100.0 then 1000.0\n end as Class_Demanda_uni_equil\n from\n (\n select\n Cliente_ID,\n Producto_ID,\n sum(Demanda_uni_equil)*1.0/7 as Avg_Demanda_uni_equil\n from\n BimboTrain\n group by\n Cliente_ID,\n Producto_ID\n ) AVUH\n ) SKUV on\n SKUV.Cliente_ID = SKUS.Cliente_ID and\n SKUV.Producto_ID = SKUS.Producto_ID\ngroup by\n SKUV.Class_Demanda_uni_equil,\n SKUS.Diff_Semana\nNobody replied, so i try to clarify my post.
\n\nLook at following consecutive demands for (Cliente_ID, Producto_ID)-tuples:
\n\n(Producto_ID, Cliente_ID) [Semana] [Demanda_uni_equil] [Demanda_uni_equil in Semana 3-9]\n(73, 150707) [3, 8, 9] [2, 2, 1] [2, 0, 0, 0, 0, 2, 1]\n(73, 166863) [5, 6, 7] [1, 1, 0] [0, 0, 1, 1, 0, 0, 0]\n(73, 120096) [3, 4, 5] [2, 2, 3] [2, 2, 3, 0, 0, 0, 0]\nIt seems to me that many consecutive demands > 0 are `glued´ together, intermittent with consecutive demands = 0.
\n\nMy gut feeling tells me, that these demand patterns do not obey `typical´ demand distributions, for example Poisson-Gamma.
", "rawMarkdown": "Nobody replied, so i try to clarify my post.\r\n\r\nLook at following consecutive demands for (Cliente_ID, Producto_ID)-tuples:\r\n\r\n (Producto_ID, Cliente_ID) [Semana] [Demanda_uni_equil] [Demanda_uni_equil in Semana 3-9]\r\n (73, 150707) [3, 8, 9] [2, 2, 1] [2, 0, 0, 0, 0, 2, 1]\r\n (73, 166863) [5, 6, 7] [1, 1, 0] [0, 0, 1, 1, 0, 0, 0]\r\n (73, 120096) [3, 4, 5] [2, 2, 3] [2, 2, 3, 0, 0, 0, 0]\r\n\r\nIt seems to me that many consecutive demands > 0 are `glued´ together, intermittent with consecutive demands = 0.\r\n\r\nMy gut feeling tells me, that these demand patterns do not obey `typical´ demand distributions, for example Poisson-Gamma.", "votes": null }, { "id": "132884", "postDate": "08/30/2016 07:01:41", "content": "Quite interesting... I wasn't able to do anything with the 0's myself, but I bet at least some of the top X on the private LB tomorrow will have!
", "rawMarkdown": "Quite interesting... I wasn't able to do anything with the 0's myself, but I bet at least some of the top X on the private LB tomorrow will have!", "votes": null }, { "id": "133024", "postDate": "08/30/2016 23:30:54", "content": "I noticed similar things when looking at the data and what interesting is in the test data such consecutive appearance is no more than 5 percent of the data. Not sure if the sampling of the training data and test data are done in a similar way.
", "rawMarkdown": "I noticed similar things when looking at the data and what interesting is in the test data such consecutive appearance is no more than 5 percent of the data. Not sure if the sampling of the training data and test data are done in a similar way.", "votes": null }, { "id": "133053", "postDate": "08/31/2016 03:43:32", "content": "I feel the 'Demand_uni_equil' is not the actual demand. It is sales minus returns, while the returns could be those sold a few weeks back (unfortunately we do not have the shelf time info). One possibility I think of may be for some long shelf time product, retail stores could buy some extra quantity (save delivery cost?) that can last for several weeks.
", "rawMarkdown": "I feel the 'Demand_uni_equil' is not the actual demand. It is sales minus returns, while the returns could be those sold a few weeks back (unfortunately we do not have the shelf time info). One possibility I think of may be for some long shelf time product, retail stores could buy some extra quantity (save delivery cost?) that can last for several weeks.", "votes": null } ], "comments": [ { "id": 132791, "author_name": "hormiga", "author_url": "", "post_date": "08/29/2016 10:29:08", "content": "Nobody replied, so i try to clarify my post.
\n\nLook at following consecutive demands for (Cliente_ID, Producto_ID)-tuples:
\n\n(Producto_ID, Cliente_ID) [Semana] [Demanda_uni_equil] [Demanda_uni_equil in Semana 3-9]\n(73, 150707) [3, 8, 9] [2, 2, 1] [2, 0, 0, 0, 0, 2, 1]\n(73, 166863) [5, 6, 7] [1, 1, 0] [0, 0, 1, 1, 0, 0, 0]\n(73, 120096) [3, 4, 5] [2, 2, 3] [2, 2, 3, 0, 0, 0, 0]\nIt seems to me that many consecutive demands > 0 are `glued´ together, intermittent with consecutive demands = 0.
\n\nMy gut feeling tells me, that these demand patterns do not obey `typical´ demand distributions, for example Poisson-Gamma.
", "votes": null, "replies": [] }, { "id": 132884, "author_name": "happycube", "author_url": "", "post_date": "08/30/2016 07:01:41", "content": "Quite interesting... I wasn't able to do anything with the 0's myself, but I bet at least some of the top X on the private LB tomorrow will have!
", "votes": null, "replies": [] }, { "id": 133024, "author_name": "tonyzhangrt", "author_url": "", "post_date": "08/30/2016 23:30:54", "content": "I noticed similar things when looking at the data and what interesting is in the test data such consecutive appearance is no more than 5 percent of the data. Not sure if the sampling of the training data and test data are done in a similar way.
", "votes": null, "replies": [] }, { "id": 133053, "author_name": "tangweipku", "author_url": "", "post_date": "08/31/2016 03:43:32", "content": "I feel the 'Demand_uni_equil' is not the actual demand. It is sales minus returns, while the returns could be those sold a few weeks back (unfortunately we do not have the shelf time info). One possibility I think of may be for some long shelf time product, retail stores could buy some extra quantity (save delivery cost?) that can last for several weeks.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "130901": "I investigated the distribution Num_Diffs of consecutive demands Diff_Semana = t2.Semana - t1.Semana (t1.Semana < t2.Semana) with Demanda_uni_equil > 0 for (Cliente_ID, Producto_ID)-tuples,\r\ndepending on demand classes Class_Demanda_uni_equil (0 < 0.2 < 0.5 < 1.0 < 2 < 5 < 10 < 100 < 1000) of demand averages Avg_Demanda_uni_equil = sum(Demanda_uni_equil)/7.\r\n\r\nSee the SQL for Hive on the bottom.\r\n\r\nTable of distribution Num_Diffs:\r\n\r\n Diff_Semana\t0,2\t0,5\t1\t2\t5\t10\t100\tTotal\r\n\r\n 0\t4,3%\t6,3%\t3,6%\t1,2%\t0,4%\t0,2%\t0,0%\t15,96%\r\n \r\n 1\t0,0%\t2,5%\t9,6%\t15,5%\t23,1%\t12,2%\t10,8%\t73,75%\r\n \r\n 2\t0,0%\t0,0%\t0,1%\t0,2%\t0,2%\t0,1%\t0,1%\t0,81%\r\n \r\n 3\t0,0%\t0,5%\t1,7%\t1,9%\t1,7%\t0,5%\t0,3%\t6,51%\r\n \r\n 4\t0,0%\t0,2%\t0,5%\t0,4%\t0,3%\t0,1%\t0,0%\t1,60%\r\n \r\n 5\t0,0%\t0,2%\t0,4%\t0,3%\t0,2%\t0,0%\t0,0%\t1,02%\r\n \r\n 6\t0,0%\t0,1%\t0,1%\t0,1%\t0,0%\t0,0%\t0,0%\t0,34%\r\n\r\nNotes on Diff_Semana:\r\n\r\n - Diff_Semana = 0 represents that there is exatcly one demand for a\r\n (Cliente_ID, Producto_ID)-tuple.\r\n - Diff_Semana = 1 represents two\r\n consecutive demands t1.Semana+1 = t2.Semana for a (Cliente_ID,\r\n Producto_ID)-tuple.\r\n - Diff_Semana >= 2 represents two consecutive demands t1.Semana+Diff_Semana = t2.Semana beetween a gap of Diff_Semana-1\r\n weeks with no demand for a (Cliente_ID, Producto_ID)-tuple.\r\n\r\n**The distribution Num_Diffs peaks on consecutive demands Diff_Semana = 1.\r\nWhat are the possible explanations?**\r\n\r\n----------\r\n\r\n \tselect\r\n\t\tSKUV.Class_Demanda_uni_equil,\r\n\t\tSKUS.Diff_Semana,\r\n\t\tcount(*) as Num_Diffs\r\n\tfrom\r\n\t\t(\r\n\t\tselect\r\n\t\t\tt1.Cliente_ID,\r\n\t\t\tt1.Producto_ID,\r\n\t\t\tt1.Semana,\r\n\t\t\tmin(t2.Semana-t1.Semana) as Diff_Semana\r\n\t\tfrom\r\n\t\t\tBimboTrain t1\r\n\t\tjoin\r\n\t\t\tBimboTrain t2 on\r\n\t\t\tt2.Cliente_ID = t1.Cliente_ID and\r\n\t\t\tt2.Producto_ID = t1.Producto_ID\r\n\t\twhere\r\n\t\t\tt2.Semana > t1.Semana and\r\n\t\t\tt1.Demanda_uni_equil > 0 and\r\n\t\t\tt2.Demanda_uni_equil > 0\r\n\t\tgroup by\r\n\t\t\tt1.Cliente_ID,\r\n\t\t\tt1.Producto_ID,\r\n\t\t\tt1.Semana\r\n\t\tunion\r\n\t\tselect\r\n\t\t\tt1.Cliente_ID,\r\n\t\t\tt1.Producto_ID,\r\n\t\t\tmin(t1.Semana) as Semana,\r\n\t\t\t0 as Diff_Semana\r\n\t\tfrom\r\n\t\t\tBimboTrain t1\r\n\t\twhere\r\n\t\t\tt1.Demanda_uni_equil > 0\r\n\t\tgroup by\r\n\t\t\tt1.Cliente_ID,\r\n\t\t\tt1.Producto_ID\r\n\t\thaving\r\n\t\t\tcount(*) = 1\r\n\t\t) SKUS\r\n\tjoin\r\n\t\t(\r\n\t\tselect\r\n\t\t\tCliente_ID,\r\n\t\t\tProducto_ID,\r\n\t\t\tcase\r\n\t\t\t\twhen Avg_Demanda_uni_equil <= 0.0 then 0.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 0.0 and Avg_Demanda_uni_equil <= 0.2 then 0.2\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 0.2 and Avg_Demanda_uni_equil <= 0.5 then 0.5\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 0.5 and Avg_Demanda_uni_equil <= 1.0 then 1.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 1.0 and Avg_Demanda_uni_equil <= 2.0 then 2.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 2.0 and Avg_Demanda_uni_equil <= 5.0 then 5.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 5.0 and Avg_Demanda_uni_equil <= 10.0 then 10.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 10.0 and Avg_Demanda_uni_equil <= 100.0 then 100.0\r\n\t\t\t\twhen Avg_Demanda_uni_equil > 100.0 then 1000.0\r\n\t\t\tend as Class_Demanda_uni_equil\r\n\t\tfrom\r\n\t\t\t(\r\n\t\t\tselect\r\n\t\t\t\tCliente_ID,\r\n\t\t\t\tProducto_ID,\r\n\t\t\t\tsum(Demanda_uni_equil)*1.0/7 as Avg_Demanda_uni_equil\r\n\t\t\tfrom\r\n\t\t\t\tBimboTrain\r\n\t\t\tgroup by\r\n\t\t\t\tCliente_ID,\r\n\t\t\t\tProducto_ID\r\n\t\t\t) AVUH\r\n\t\t) SKUV on\r\n\t\t\tSKUV.Cliente_ID = SKUS.Cliente_ID and\r\n\t\t\tSKUV.Producto_ID = SKUS.Producto_ID\r\n\tgroup by\r\n\t\tSKUV.Class_Demanda_uni_equil,\r\n\t\tSKUS.Diff_Semana", "132791": "Nobody replied, so i try to clarify my post.\r\n\r\nLook at following consecutive demands for (Cliente_ID, Producto_ID)-tuples:\r\n\r\n (Producto_ID, Cliente_ID) [Semana] [Demanda_uni_equil] [Demanda_uni_equil in Semana 3-9]\r\n (73, 150707) [3, 8, 9] [2, 2, 1] [2, 0, 0, 0, 0, 2, 1]\r\n (73, 166863) [5, 6, 7] [1, 1, 0] [0, 0, 1, 1, 0, 0, 0]\r\n (73, 120096) [3, 4, 5] [2, 2, 3] [2, 2, 3, 0, 0, 0, 0]\r\n\r\nIt seems to me that many consecutive demands > 0 are `glued´ together, intermittent with consecutive demands = 0.\r\n\r\nMy gut feeling tells me, that these demand patterns do not obey `typical´ demand distributions, for example Poisson-Gamma.", "132884": "Quite interesting... I wasn't able to do anything with the 0's myself, but I bet at least some of the top X on the private LB tomorrow will have!", "133024": "I noticed similar things when looking at the data and what interesting is in the test data such consecutive appearance is no more than 5 percent of the data. Not sure if the sampling of the training data and test data are done in a similar way.", "133053": "I feel the 'Demand_uni_equil' is not the actual demand. It is sales minus returns, while the returns could be those sold a few weeks back (unfortunately we do not have the shelf time info). One possibility I think of may be for some long shelf time product, retail stores could buy some extra quantity (save delivery cost?) that can last for several weeks." }, "source": "meta" }