For me, this is the hardest competition I have ever entered - I have never experienced anything like it - almost everything I tried get scores between 2.68 (Genetic Programming and LGB) and 3.88 (FFM) - those blinking outliers are killing me!
\n\nI have decided to spend a few days just looking at the data and get away from the weeds of modelling - there must be a way to connect all these card_ids and merchant_ids!
\n\nHas anyone had a modicum of success using FFM or NMF - I tried but failed sadly. I hope I don't have to use networkx graphs!
\n\nI am even starting to think we are trying to emulate their logistic regression model and the values are the raw values of a sigmoid function
\n\n1/(1+exp(-p)) => p is our targets! Yes I am that desperate/clueless! ;)
", "messages": [ { "id": 467294, "postDate": "2019-02-06T19:56:40.050Z", "content": "For me, this is the hardest competition I have ever entered - I have never experienced anything like it - almost everything I tried get scores between 2.68 (Genetic Programming and LGB) and 3.88 (FFM) - those blinking outliers are killing me!
\n\nI have decided to spend a few days just looking at the data and get away from the weeds of modelling - there must be a way to connect all these card_ids and merchant_ids!
\n\nHas anyone had a modicum of success using FFM or NMF - I tried but failed sadly. I hope I don't have to use networkx graphs!
\n\nI am even starting to think we are trying to emulate their logistic regression model and the values are the raw values of a sigmoid function
\n\n1/(1+exp(-p)) => p is our targets! Yes I am that desperate/clueless! ;)
", "rawMarkdown": "For me, this is the hardest competition I have ever entered - I have never experienced anything like it - almost everything I tried get scores between 2.68 (Genetic Programming and LGB) and 3.88 (FFM) - those blinking outliers are killing me! \n\nI have decided to spend a few days just looking at the data and get away from the weeds of modelling - there must be a way to connect all these card_ids and merchant_ids!\n\nHas anyone had a modicum of success using FFM or NMF - I tried but failed sadly. I hope I don't have to use networkx graphs!\n\nI am even starting to think we are trying to emulate their logistic regression model and the values are the raw values of a sigmoid function\n\n1/(1+exp(-p)) => p is our targets! Yes I am that desperate/clueless! ;)\n", "votes": 31 }, { "id": 467851, "postDate": "2019-02-07T20:55:53.233Z", "content": "@Scirpus, have you just became a Discussion GRANDMASTER? Nice! Congratulations!
", "rawMarkdown": "@Scirpus, have you just became a Discussion GRANDMASTER? Nice! Congratulations!", "votes": 9, "replies": [ { "id": 467869, "postDate": "2019-02-07T21:36:01.973Z", "content": "Thanks - only took 5 years LOL! ;)
", "rawMarkdown": "Thanks - only took 5 years LOL! ;)", "votes": 13 }, { "id": 467870, "postDate": "2019-02-07T21:37:24.363Z", "content": "Wow! congrats Scirpus!
", "rawMarkdown": "Wow! congrats Scirpus!", "votes": 4 }, { "id": 467890, "postDate": "2019-02-07T22:40:44.100Z", "content": "Congrats @Scirpus on your discussion GM status.
\n\n\n", "rawMarkdown": "Congrats @Scirpus on your discussion GM status.\n\n> Thanks - only took 5 years LOL! ;)\n", "votes": 3 }, { "id": 467936, "postDate": "2019-02-08T01:55:12.950Z", "content": "Thanks - only took 5 years LOL! ;)
\n
Congratulations Scripus!
\nYou definitely gave a whole new perspective to me(us), with genetic programming.
\nHope you will continue contributing :)
Congrats
", "rawMarkdown": "Congrats" } ] }, { "id": 467311, "postDate": "2019-02-06T21:14:38.230Z", "content": "\"cardids and merchantids\"
\n\nI clearly see \"time series\" data and think there could be the key to success.\nThe host of this competition said that they removed historical data for the last two months because of \"data leakage\".\nProbably the key is there. If we can predict sales for that removed moths we can get lower RMSE.
\n\nP.S. @scirpus thank you for all your public kernels, I don't have words to appreciate your work. I've learned so much from you. Thank you.
", "rawMarkdown": "\"cardids and merchantids\"\n\nI clearly see \"time series\" data and think there could be the key to success.\nThe host of this competition said that they removed historical data for the last two months because of \"data leakage\".\nProbably the key is there. If we can predict sales for that removed moths we can get lower RMSE.\n\nP.S. @scirpus thank you for all your public kernels, I don't have words to appreciate your work. I've learned so much from you. Thank you. ", "votes": 5 }, { "id": 468432, "postDate": "2019-02-08T21:17:35.667Z", "content": "Hey @scirpus, congratulations for this well deserved GM (Genetically Modified) grade ! Thanks for all the things you share ;-)
", "rawMarkdown": "Hey @scirpus, congratulations for this well deserved GM (Genetically Modified) grade ! Thanks for all the things you share ;-)", "votes": 4 }, { "id": 471114, "postDate": "2019-02-14T03:38:01.477Z", "content": "Genetic Programming is charming but i don't know much
", "rawMarkdown": "Genetic Programming is charming but i don't know much", "votes": 1 }, { "id": 469775, "postDate": "2019-02-11T19:53:12.423Z", "content": "Well done!
", "rawMarkdown": "Well done!", "votes": 1 }, { "id": 469142, "postDate": "2019-02-10T15:23:29.413Z", "content": "Good job
", "rawMarkdown": "Good job", "votes": 1 }, { "id": 469073, "postDate": "2019-02-10T12:42:30.803Z", "content": "congrats
", "rawMarkdown": "congrats", "votes": 1 }, { "id": 467812, "postDate": "2019-02-07T19:00:15.323Z", "content": "After my local CV improved from 3.641 to 6.635 (e.g. -0.006), my LB went up +0.014\nThen I changed seeds and here is what I got:\nbefore: CV 3.63552 std 0.10288\nafter: CV: 3.63921 std 0.03521
\n\nI already expected an increase in the score, but std?!?! Local CV grew and std dropped A LOT, by like 2/3. If I overfitted to the old seeds, I would have expected the score AND maybe std to increase but NO.
\n\nAnd I am already doing a repeated kfold.
\n\nI am not giving up yet, but this competition is really weird.
", "rawMarkdown": "After my local CV improved from 3.641 to 6.635 (e.g. -0.006), my LB went up +0.014\nThen I changed seeds and here is what I got:\nbefore: CV 3.63552 std 0.10288\nafter: CV: 3.63921 std 0.03521\n\nI already expected an increase in the score, but std?!?! Local CV grew and std dropped A LOT, by like 2/3. If I overfitted to the old seeds, I would have expected the score AND maybe std to increase but NO.\n\nAnd I am already doing a repeated kfold.\n\nI am not giving up yet, but this competition is really weird.", "votes": 1, "replies": [ { "id": 467860, "postDate": "2019-02-07T21:09:53.263Z", "content": "For sure, this competition is all kinds of inconsistent. Take its title for instance.
\n\nElo Merchant Category Recommendation\nHelp understand customer loyalty
\n\nAre we trying to determine the most relevant merchant category to recommend from a given context? This is implied in the overview: we must \"develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty.\" But this cannot be the case as the target is continuous.
\n\nWhat are the independent variables here? It is unlikely to be any of the historical or new information which leaves the three mysterious features. Are these survey responses? Different types of promotions being tested? Both?
\n\nI guess my point is I have no idea whats going on.
", "rawMarkdown": "For sure, this competition is all kinds of inconsistent. Take its title for instance. \n\nElo Merchant Category Recommendation\nHelp understand customer loyalty\n\nAre we trying to determine the most relevant merchant category to recommend from a given context? This is implied in the overview: we must \"develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty.\" But this cannot be the case as the target is continuous. \n\nWhat are the independent variables here? It is unlikely to be any of the historical or new information which leaves the three mysterious features. Are these survey responses? Different types of promotions being tested? Both?\n\nI guess my point is I have no idea whats going on.", "votes": 2 } ] }, { "id": 467603, "postDate": "2019-02-07T12:04:08.353Z", "content": "@Scirpus thanks for all your sharing. On this competition, I am coming to a similar conclusion as you have. I took a few days away, came back to it today and so far not much success.
", "rawMarkdown": "@Scirpus thanks for all your sharing. On this competition, I am coming to a similar conclusion as you have. I took a few days away, came back to it today and so far not much success.", "votes": 1, "replies": [ { "id": 467681, "postDate": "2019-02-07T15:01:38.840Z", "content": "Definitely - treading water for days is not pleasant - good luck
", "rawMarkdown": "Definitely - treading water for days is not pleasant - good luck", "votes": 1 }, { "id": 471056, "postDate": "2019-02-14T00:49:20.583Z", "content": "@Scirpus , it is still hard but made some progress. Following the discussion threads, I got inspired especially reading about @raddar's finds, I am able to improve my single model to 0.687 LB. I have also built a few weak models, so it is time to start working on stacking.
", "rawMarkdown": "@Scirpus , it is still hard but made some progress. Following the discussion threads, I got inspired especially reading about @raddar's finds, I am able to improve my single model to 0.687 LB. I have also built a few weak models, so it is time to start working on stacking." } ] }, { "id": 467317, "postDate": "2019-02-06T21:36:47.953Z", "content": "I’m right there with you. I have not yet been able to make a single model of whatever type that gets below a 3.89 on lb. It’s been quite frustrating but informative.
\n\nHaving no idea what the target is seems to prevent most rational strategies. The monster aggs people are using do appear to capture some relevant information but by throwing everything at the wall and seeing what sticks. And one can only go so far with regression, I’m certain all the teams at the top are much better than me at identifying outliers.
", "rawMarkdown": "I’m right there with you. I have not yet been able to make a single model of whatever type that gets below a 3.89 on lb. It’s been quite frustrating but informative. \n\nHaving no idea what the target is seems to prevent most rational strategies. The monster aggs people are using do appear to capture some relevant information but by throwing everything at the wall and seeing what sticks. And one can only go so far with regression, I’m certain all the teams at the top are much better than me at identifying outliers. ", "votes": 1, "replies": [ { "id": 467322, "postDate": "2019-02-06T21:51:35.377Z", "content": "Or they have used 200 submissions to probe the lb ;)
", "rawMarkdown": "Or they have used 200 submissions to probe the lb ;)", "votes": 2 }, { "id": 467332, "postDate": "2019-02-06T22:16:12.523Z", "content": "For real. Though in this case that's probably a good strategy. Knowing some of the outliers in test can kind of let you work backwards and inform the classification models, though most of mine are still terrible. I'm not quite ready to give up on regression yet, but after making about 7k features and not seeing much improvement I'm getting sick of it. Oh well, can't be helped I suppose.
", "rawMarkdown": "For real. Though in this case that's probably a good strategy. Knowing some of the outliers in test can kind of let you work backwards and inform the classification models, though most of mine are still terrible. I'm not quite ready to give up on regression yet, but after making about 7k features and not seeing much improvement I'm getting sick of it. Oh well, can't be helped I suppose.", "votes": 1 }, { "id": 467719, "postDate": "2019-02-07T16:06:49.993Z", "content": "as for me, i cannot find success in model ensembling. everything i tried to combine my models increases the rmse
", "rawMarkdown": "as for me, i cannot find success in model ensembling. everything i tried to combine my models increases the rmse" }, { "id": 467747, "postDate": "2019-02-07T16:46:21.050Z", "content": "Keep in mind that LB score is only based on 30% of the data. No improvement on the 30% LB data doesn't necessarily mean no improvement on the held out 70%.
", "rawMarkdown": "Keep in mind that LB score is only based on 30% of the data. No improvement on the 30% LB data doesn't necessarily mean no improvement on the held out 70%.", "votes": 1 } ] }, { "id": 468401, "postDate": "2019-02-08T20:17:06.033Z", "content": "Congratulations
", "rawMarkdown": "Congratulations", "votes": 2 }, { "id": 467784, "postDate": "2019-02-07T18:08:54.947Z", "content": "If by modicum you mean \"0.001\" , then yes i had improvement with FFM.
", "rawMarkdown": "If by modicum you mean \"0.001\" , then yes i had improvement with FFM.", "votes": 2, "replies": [ { "id": 468243, "postDate": "2019-02-08T14:56:42.923Z", "content": "What is FFM?
", "rawMarkdown": "What is FFM?\n" }, { "id": 468339, "postDate": "2019-02-08T17:40:12.613Z", "content": "To my knowledge, it stands for Field Aware Factorization Machines. \nHere's another discussion thread that talks about it too.\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76480
", "rawMarkdown": "To my knowledge, it stands for Field Aware Factorization Machines. \nHere's another discussion thread that talks about it too.\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76480", "votes": 1 }, { "id": 468347, "postDate": "2019-02-08T17:57:23.857Z", "content": "Field aware factorization Machine. You can say upper version of factorization Machine. https://www.youtube.com/watch?v=1cRGpDXTJC8 This might be useful
", "rawMarkdown": "Field aware factorization Machine. You can say upper version of factorization Machine. https://www.youtube.com/watch?v=1cRGpDXTJC8 This might be useful", "votes": 1 } ] }, { "id": 467690, "postDate": "2019-02-07T15:18:29.773Z", "content": "I think that the key issue is that we have little information about the data and the business logic behind the loyalty score calculation. I spent the first 1.5 months, believing that the target is to calculate the next 2 months after the new merchants file, loyalty scores. Now, after reading several discussion topics i believe that i have the correct understanding :)
\n\nMoreover, the host has not indicated how the are simulated. Is it just noise and they want to see modelling approaches? is it a statistic model? is it a formula? is it a combination of all ?
\n\nAs the google store competition, we are trying to figure out by ourselves what is the target.....
\n\na simple explanation by the host could have helped us a lot
", "rawMarkdown": "I think that the key issue is that we have little information about the data and the business logic behind the loyalty score calculation. I spent the first 1.5 months, believing that the target is to calculate the next 2 months after the new merchants file, loyalty scores. Now, after reading several discussion topics i believe that i have the correct understanding :)\n\nMoreover, the host has not indicated how the are simulated. Is it just noise and they want to see modelling approaches? is it a statistic model? is it a formula? is it a combination of all ? \n\nAs the google store competition, we are trying to figure out by ourselves what is the target.....\n\na simple explanation by the host could have helped us a lot", "votes": 2 }, { "id": 467347, "postDate": "2019-02-06T22:55:28.617Z", "content": "As it is the first competition that i decided to give it all i have, it's a big disappointment. It is probably the worst competition to begin on kaggle and learn. I am stuck at 3.87 and no matter what i do it doesn't improve. Ι tried feature engineering, different models, feature selection, different tuning, stacking, averaging, ffm, post-processing. Things that work elsewhere like averaging, don't help. I think i lost a lot of time here..
", "rawMarkdown": "As it is the first competition that i decided to give it all i have, it's a big disappointment. It is probably the worst competition to begin on kaggle and learn. I am stuck at 3.87 and no matter what i do it doesn't improve. Ι tried feature engineering, different models, feature selection, different tuning, stacking, averaging, ffm, post-processing. Things that work elsewhere like averaging, don't help. I think i lost a lot of time here.. ", "votes": 2, "replies": [ { "id": 467523, "postDate": "2019-02-07T09:18:09.057Z", "content": "Munge Data using Bogan's Mitsura Script and then use LGB - this should get you below 3.69
", "rawMarkdown": "Munge Data using Bogan's Mitsura Script and then use LGB - this should get you below 3.69" }, { "id": 467584, "postDate": "2019-02-07T11:32:28.627Z", "content": "@Dimitrios Effrosynidis\ndon't give up!!\nalso - listen to your local CV, even across folds, the differences in score varies, and public LB could be not representing of what's to come.
", "rawMarkdown": "@Dimitrios Effrosynidis\ndon't give up!!\nalso - listen to your local CV, even across folds, the differences in score varies, and public LB could be not representing of what's to come." }, { "id": 467741, "postDate": "2019-02-07T16:40:25.477Z", "content": "I agree with @AmirH and others said, trust your CV, a big shake up is coming up.
", "rawMarkdown": "I agree with @AmirH and others said, trust your CV, a big shake up is coming up. ", "votes": 1 }, { "id": 467811, "postDate": "2019-02-07T18:59:57.380Z", "content": "Well! I find it hard to believe in my CVs:\nmodel1, CV=3.64819, LB=3.690\nmodel2, CV=3.58815, LB=3.690\n0.5*model1+0.5*model2, CV=3.60061, LB=3.680
", "rawMarkdown": "Well! I find it hard to believe in my CVs:\nmodel1, CV=3.64819, LB=3.690\nmodel2, CV=3.58815, LB=3.690\n0.5*model1+0.5*model2, CV=3.60061, LB=3.680", "votes": 3 }, { "id": 467816, "postDate": "2019-02-07T19:12:41.880Z", "content": "I have the same scenario, Two models have 3.697 and 3.698 on lb, when i blend them, got 3.685 on lb
", "rawMarkdown": "I have the same scenario, Two models have 3.697 and 3.698 on lb, when i blend them, got 3.685 on lb" }, { "id": 467826, "postDate": "2019-02-07T19:48:15.143Z", "content": "Same thing for me, but only if the models either used different features or different algorithms, which tells us something though I'm not sure what yet.
", "rawMarkdown": "Same thing for me, but only if the models either used different features or different algorithms, which tells us something though I'm not sure what yet. " }, { "id": 467917, "postDate": "2019-02-08T00:20:26.493Z", "content": "I used Konrad's Lightgbm parameters since the beginning of the competition, and was able to achieve 3.674. \nAs soon as I use regularized model it does improve locally but way worse score on LB.\nI was getting mad for severals days, not understanding why. But aparently I'm not the only one.
\n\nFind bellow a scatter plot of two of my submissions.\nhttps://drive.google.com/file/d/1O-aIsIR66rnpoMRbZou2xPCpW95wdyK_/view?usp=sharing\nThe \"bad\" sub is a model more regularized than the good one.
", "rawMarkdown": "I used Konrad's Lightgbm parameters since the beginning of the competition, and was able to achieve 3.674. \nAs soon as I use regularized model it does improve locally but way worse score on LB.\nI was getting mad for severals days, not understanding why. But aparently I'm not the only one.\n\nFind bellow a scatter plot of two of my submissions.\nhttps://drive.google.com/file/d/1O-aIsIR66rnpoMRbZou2xPCpW95wdyK_/view?usp=sharing\nThe \"bad\" sub is a model more regularized than the good one.", "votes": 2 }, { "id": 467929, "postDate": "2019-02-08T01:09:01.040Z", "content": "@jacquespeeters, you are definitely not the only one as I felt like I am missing something fundamental here. I do not have one yet but feel the need to come up with a plan B fast :-)
", "rawMarkdown": "@jacquespeeters, you are definitely not the only one as I felt like I am missing something fundamental here. I do not have one yet but feel the need to come up with a plan B fast :-)" }, { "id": 468111, "postDate": "2019-02-08T10:13:10.320Z", "content": "Well I think I got good clues on how to improve. I might be underfitting, reducing learning rate should help.
\n\nWanna team-up? :)
", "rawMarkdown": "Well I think I got good clues on how to improve. I might be underfitting, reducing learning rate should help. \n\nWanna team-up? :) " }, { "id": 468132, "postDate": "2019-02-08T10:45:33.740Z", "content": "@jacquespeeters, is that a serious team-up suggestion? :-)
", "rawMarkdown": "@jacquespeeters, is that a serious team-up suggestion? :-)" }, { "id": 479523, "postDate": "2019-02-27T08:08:31.163Z", "content": "Congrats @jacquespeeters, I see me you stayed away from the temptation known as \"outliers treatment'. Could we have got gold combining our features? Perhaps, one can dream :-)
", "rawMarkdown": "Congrats @jacquespeeters, I see me you stayed away from the temptation known as \"outliers treatment'. Could we have got gold combining our features? Perhaps, one can dream :-)" } ] }, { "id": 467560, "postDate": "2019-02-07T10:34:28.413Z", "content": "Same situation here from me. I did not deal with the outliers (did not remove or replace them) I got 1.55X CV but, 3.84~3.86LB
", "rawMarkdown": "Same situation here from me. I did not deal with the outliers (did not remove or replace them) I got 1.55X CV but, 3.84~3.86LB", "replies": [ { "id": 467593, "postDate": "2019-02-07T11:46:21.480Z", "content": "did you get 1.55 rmse on CV with outliers on the training dataset??
", "rawMarkdown": "did you get 1.55 rmse on CV with outliers on the training dataset??" }, { "id": 467597, "postDate": "2019-02-07T11:54:28.343Z", "content": "Yes, I did not deal with outliers. I know this is strange. I used 360 features. The only thing I can imagine is overfitting.
", "rawMarkdown": "Yes, I did not deal with outliers. I know this is strange. I used 360 features. The only thing I can imagine is overfitting." }, { "id": 468300, "postDate": "2019-02-08T16:39:21.920Z", "content": "did you have the outlier column as part of CV? i did that by mistake and had similar results
", "rawMarkdown": "did you have the outlier column as part of CV? i did that by mistake and had similar results" } ] }, { "id": 468677, "postDate": "2019-02-09T12:45:06.263Z", "content": "i'm interested in your work(Genetic Programming ).I don’t know if you can share it after the competition.
", "rawMarkdown": "i'm interested in your work(Genetic Programming ).I don’t know if you can share it after the competition." }, { "id": 467838, "postDate": "2019-02-07T20:26:45.297Z", "content": "What are some of the craziest things you all have tried? In the absence of information about the task I have tried all sorts of things that don't make sense. For example, I tried digitizing the targets and training to predict the target bin, then added the predicted test bins back to test as a feature, it did not work well, shocking.
", "rawMarkdown": "What are some of the craziest things you all have tried? In the absence of information about the task I have tried all sorts of things that don't make sense. For example, I tried digitizing the targets and training to predict the target bin, then added the predicted test bins back to test as a feature, it did not work well, shocking. " }, { "id": 473778, "postDate": "2019-02-18T13:49:14.887Z", "rawMarkdown": "", "votes": 2, "isDeleted": true, "replies": [ { "id": 477349, "postDate": "2019-02-24T11:23:58.807Z", "content": "Or they found some proper CV scheme. There was an idea from the Top1 to use uniform distribution for validation but tbh I not sure why it should be better than simple stratification by outliers. But smth in this direction might have helped a lot.
\n\nAnother possible direction of thought is actually NOT to ensemble in some tricky way. Because ensembling averages out all our magic.
", "rawMarkdown": "Or they found some proper CV scheme. There was an idea from the Top1 to use uniform distribution for validation but tbh I not sure why it should be better than simple stratification by outliers. But smth in this direction might have helped a lot.\n\nAnother possible direction of thought is actually NOT to ensemble in some tricky way. Because ensembling averages out all our magic. ", "votes": 1 }, { "id": 477354, "postDate": "2019-02-24T11:41:09.223Z", "content": "@snovik1975, ensembling also reduces the errors as per correct prediction :-)
", "rawMarkdown": "@snovik1975, ensembling also reduces the errors as per correct prediction :-)" }, { "id": 477413, "postDate": "2019-02-24T13:58:14.407Z", "content": "aren't we hunting for errors? :)
", "rawMarkdown": "aren't we hunting for errors? :)" } ] }, { "id": 468724, "postDate": "2019-02-09T14:55:58.003Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 467851, "author_name": "Oleg Yaroshevskiy", "author_url": "", "post_date": "2019-02-07T20:55:53.233000", "content": "@Scirpus, have you just became a Discussion GRANDMASTER? Nice! Congratulations!
", "votes": 9, "replies": [ { "id": 467869, "author_name": "Scirpus", "author_url": "", "post_date": "2019-02-07T21:36:01.973000", "content": "Thanks - only took 5 years LOL! ;)
", "votes": 13, "replies": [] }, { "id": 467870, "author_name": "raddar", "author_url": "", "post_date": "2019-02-07T21:37:24.363000", "content": "Wow! congrats Scirpus!
", "votes": 4, "replies": [] }, { "id": 467890, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-07T22:40:44.100000", "content": "Congrats @Scirpus on your discussion GM status.
\n\n\n", "votes": 3, "replies": [] }, { "id": 467936, "author_name": "pocket", "author_url": "", "post_date": "2019-02-08T01:55:12.950000", "content": "Thanks - only took 5 years LOL! ;)
\n
Congratulations Scripus!
\nYou definitely gave a whole new perspective to me(us), with genetic programming.
\nHope you will continue contributing :)
Congrats
", "votes": 0, "replies": [] } ] }, { "id": 467311, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2019-02-06T21:14:38.230000", "content": "\"cardids and merchantids\"
\n\nI clearly see \"time series\" data and think there could be the key to success.\nThe host of this competition said that they removed historical data for the last two months because of \"data leakage\".\nProbably the key is there. If we can predict sales for that removed moths we can get lower RMSE.
\n\nP.S. @scirpus thank you for all your public kernels, I don't have words to appreciate your work. I've learned so much from you. Thank you.
", "votes": 5, "replies": [] }, { "id": 468432, "author_name": "olivier", "author_url": "", "post_date": "2019-02-08T21:17:35.667000", "content": "Hey @scirpus, congratulations for this well deserved GM (Genetically Modified) grade ! Thanks for all the things you share ;-)
", "votes": 4, "replies": [] }, { "id": 471114, "author_name": "Neo Zhao", "author_url": "", "post_date": "2019-02-14T03:38:01.477000", "content": "Genetic Programming is charming but i don't know much
", "votes": 1, "replies": [] }, { "id": 469775, "author_name": "mtodisco10", "author_url": "", "post_date": "2019-02-11T19:53:12.423000", "content": "Well done!
", "votes": 1, "replies": [] }, { "id": 469142, "author_name": "Soslan Tabuev", "author_url": "", "post_date": "2019-02-10T15:23:29.413000", "content": "Good job
", "votes": 1, "replies": [] }, { "id": 469073, "author_name": "Mayank Gandotra", "author_url": "", "post_date": "2019-02-10T12:42:30.803000", "content": "congrats
", "votes": 1, "replies": [] }, { "id": 467812, "author_name": "snovik", "author_url": "", "post_date": "2019-02-07T19:00:15.323000", "content": "After my local CV improved from 3.641 to 6.635 (e.g. -0.006), my LB went up +0.014\nThen I changed seeds and here is what I got:\nbefore: CV 3.63552 std 0.10288\nafter: CV: 3.63921 std 0.03521
\n\nI already expected an increase in the score, but std?!?! Local CV grew and std dropped A LOT, by like 2/3. If I overfitted to the old seeds, I would have expected the score AND maybe std to increase but NO.
\n\nAnd I am already doing a repeated kfold.
\n\nI am not giving up yet, but this competition is really weird.
", "votes": 1, "replies": [ { "id": 467860, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-07T21:09:53.263000", "content": "For sure, this competition is all kinds of inconsistent. Take its title for instance.
\n\nElo Merchant Category Recommendation\nHelp understand customer loyalty
\n\nAre we trying to determine the most relevant merchant category to recommend from a given context? This is implied in the overview: we must \"develop algorithms to identify and serve the most relevant opportunities to individuals, by uncovering signal in customer loyalty.\" But this cannot be the case as the target is continuous.
\n\nWhat are the independent variables here? It is unlikely to be any of the historical or new information which leaves the three mysterious features. Are these survey responses? Different types of promotions being tested? Both?
\n\nI guess my point is I have no idea whats going on.
", "votes": 2, "replies": [] } ] }, { "id": 467603, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-07T12:04:08.353000", "content": "@Scirpus thanks for all your sharing. On this competition, I am coming to a similar conclusion as you have. I took a few days away, came back to it today and so far not much success.
", "votes": 1, "replies": [ { "id": 467681, "author_name": "Scirpus", "author_url": "", "post_date": "2019-02-07T15:01:38.840000", "content": "Definitely - treading water for days is not pleasant - good luck
", "votes": 1, "replies": [] }, { "id": 471056, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-14T00:49:20.583000", "content": "@Scirpus , it is still hard but made some progress. Following the discussion threads, I got inspired especially reading about @raddar's finds, I am able to improve my single model to 0.687 LB. I have also built a few weak models, so it is time to start working on stacking.
", "votes": 0, "replies": [] } ] }, { "id": 467317, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-06T21:36:47.953000", "content": "I’m right there with you. I have not yet been able to make a single model of whatever type that gets below a 3.89 on lb. It’s been quite frustrating but informative.
\n\nHaving no idea what the target is seems to prevent most rational strategies. The monster aggs people are using do appear to capture some relevant information but by throwing everything at the wall and seeing what sticks. And one can only go so far with regression, I’m certain all the teams at the top are much better than me at identifying outliers.
", "votes": 1, "replies": [ { "id": 467322, "author_name": "Scirpus", "author_url": "", "post_date": "2019-02-06T21:51:35.377000", "content": "Or they have used 200 submissions to probe the lb ;)
", "votes": 2, "replies": [] }, { "id": 467332, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-06T22:16:12.523000", "content": "For real. Though in this case that's probably a good strategy. Knowing some of the outliers in test can kind of let you work backwards and inform the classification models, though most of mine are still terrible. I'm not quite ready to give up on regression yet, but after making about 7k features and not seeing much improvement I'm getting sick of it. Oh well, can't be helped I suppose.
", "votes": 1, "replies": [] }, { "id": 467719, "author_name": "dylonLL", "author_url": "", "post_date": "2019-02-07T16:06:49.993000", "content": "as for me, i cannot find success in model ensembling. everything i tried to combine my models increases the rmse
", "votes": 0, "replies": [] }, { "id": 467747, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2019-02-07T16:46:21.050000", "content": "Keep in mind that LB score is only based on 30% of the data. No improvement on the 30% LB data doesn't necessarily mean no improvement on the held out 70%.
", "votes": 1, "replies": [] } ] }, { "id": 468401, "author_name": "Kenil Shah", "author_url": "", "post_date": "2019-02-08T20:17:06.033000", "content": "Congratulations
", "votes": 2, "replies": [] }, { "id": 467784, "author_name": "HarshitMehta", "author_url": "", "post_date": "2019-02-07T18:08:54.947000", "content": "If by modicum you mean \"0.001\" , then yes i had improvement with FFM.
", "votes": 2, "replies": [ { "id": 468243, "author_name": "MansoorScorpio", "author_url": "", "post_date": "2019-02-08T14:56:42.923000", "content": "What is FFM?
", "votes": 0, "replies": [] }, { "id": 468339, "author_name": "Alistair Murray", "author_url": "", "post_date": "2019-02-08T17:40:12.613000", "content": "To my knowledge, it stands for Field Aware Factorization Machines. \nHere's another discussion thread that talks about it too.\nhttps://www.kaggle.com/c/elo-merchant-category-recommendation/discussion/76480
", "votes": 1, "replies": [] }, { "id": 468347, "author_name": "HarshitMehta", "author_url": "", "post_date": "2019-02-08T17:57:23.857000", "content": "Field aware factorization Machine. You can say upper version of factorization Machine. https://www.youtube.com/watch?v=1cRGpDXTJC8 This might be useful
", "votes": 1, "replies": [] } ] }, { "id": 467690, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2019-02-07T15:18:29.773000", "content": "I think that the key issue is that we have little information about the data and the business logic behind the loyalty score calculation. I spent the first 1.5 months, believing that the target is to calculate the next 2 months after the new merchants file, loyalty scores. Now, after reading several discussion topics i believe that i have the correct understanding :)
\n\nMoreover, the host has not indicated how the are simulated. Is it just noise and they want to see modelling approaches? is it a statistic model? is it a formula? is it a combination of all ?
\n\nAs the google store competition, we are trying to figure out by ourselves what is the target.....
\n\na simple explanation by the host could have helped us a lot
", "votes": 2, "replies": [] }, { "id": 467347, "author_name": "Dimitrios Effrosynidis", "author_url": "", "post_date": "2019-02-06T22:55:28.617000", "content": "As it is the first competition that i decided to give it all i have, it's a big disappointment. It is probably the worst competition to begin on kaggle and learn. I am stuck at 3.87 and no matter what i do it doesn't improve. Ι tried feature engineering, different models, feature selection, different tuning, stacking, averaging, ffm, post-processing. Things that work elsewhere like averaging, don't help. I think i lost a lot of time here..
", "votes": 2, "replies": [ { "id": 467523, "author_name": "Scirpus", "author_url": "", "post_date": "2019-02-07T09:18:09.057000", "content": "Munge Data using Bogan's Mitsura Script and then use LGB - this should get you below 3.69
", "votes": 0, "replies": [] }, { "id": 467584, "author_name": "AmirH", "author_url": "", "post_date": "2019-02-07T11:32:28.627000", "content": "@Dimitrios Effrosynidis\ndon't give up!!\nalso - listen to your local CV, even across folds, the differences in score varies, and public LB could be not representing of what's to come.
", "votes": 0, "replies": [] }, { "id": 467741, "author_name": "Mendrika Ramarlina", "author_url": "", "post_date": "2019-02-07T16:40:25.477000", "content": "I agree with @AmirH and others said, trust your CV, a big shake up is coming up.
", "votes": 1, "replies": [] }, { "id": 467811, "author_name": "Philippe Lonjoux", "author_url": "", "post_date": "2019-02-07T18:59:57.380000", "content": "Well! I find it hard to believe in my CVs:\nmodel1, CV=3.64819, LB=3.690\nmodel2, CV=3.58815, LB=3.690\n0.5*model1+0.5*model2, CV=3.60061, LB=3.680
", "votes": 3, "replies": [] }, { "id": 467816, "author_name": "HarshitMehta", "author_url": "", "post_date": "2019-02-07T19:12:41.880000", "content": "I have the same scenario, Two models have 3.697 and 3.698 on lb, when i blend them, got 3.685 on lb
", "votes": 0, "replies": [] }, { "id": 467826, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-07T19:48:15.143000", "content": "Same thing for me, but only if the models either used different features or different algorithms, which tells us something though I'm not sure what yet.
", "votes": 0, "replies": [] }, { "id": 467917, "author_name": "Jacques Peeters", "author_url": "", "post_date": "2019-02-08T00:20:26.493000", "content": "I used Konrad's Lightgbm parameters since the beginning of the competition, and was able to achieve 3.674. \nAs soon as I use regularized model it does improve locally but way worse score on LB.\nI was getting mad for severals days, not understanding why. But aparently I'm not the only one.
\n\nFind bellow a scatter plot of two of my submissions.\nhttps://drive.google.com/file/d/1O-aIsIR66rnpoMRbZou2xPCpW95wdyK_/view?usp=sharing\nThe \"bad\" sub is a model more regularized than the good one.
", "votes": 2, "replies": [] }, { "id": 467929, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-08T01:09:01.040000", "content": "@jacquespeeters, you are definitely not the only one as I felt like I am missing something fundamental here. I do not have one yet but feel the need to come up with a plan B fast :-)
", "votes": 0, "replies": [] }, { "id": 468111, "author_name": "Jacques Peeters", "author_url": "", "post_date": "2019-02-08T10:13:10.320000", "content": "Well I think I got good clues on how to improve. I might be underfitting, reducing learning rate should help.
\n\nWanna team-up? :)
", "votes": 0, "replies": [] }, { "id": 468132, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-08T10:45:33.740000", "content": "@jacquespeeters, is that a serious team-up suggestion? :-)
", "votes": 0, "replies": [] }, { "id": 479523, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-27T08:08:31.163000", "content": "Congrats @jacquespeeters, I see me you stayed away from the temptation known as \"outliers treatment'. Could we have got gold combining our features? Perhaps, one can dream :-)
", "votes": 0, "replies": [] } ] }, { "id": 467560, "author_name": "Jie Wu", "author_url": "", "post_date": "2019-02-07T10:34:28.413000", "content": "Same situation here from me. I did not deal with the outliers (did not remove or replace them) I got 1.55X CV but, 3.84~3.86LB
", "votes": 0, "replies": [ { "id": 467593, "author_name": "Dimosthenis Karaflos", "author_url": "", "post_date": "2019-02-07T11:46:21.480000", "content": "did you get 1.55 rmse on CV with outliers on the training dataset??
", "votes": 0, "replies": [] }, { "id": 467597, "author_name": "Jie Wu", "author_url": "", "post_date": "2019-02-07T11:54:28.343000", "content": "Yes, I did not deal with outliers. I know this is strange. I used 360 features. The only thing I can imagine is overfitting.
", "votes": 0, "replies": [] }, { "id": 468300, "author_name": "RVPrakash", "author_url": "", "post_date": "2019-02-08T16:39:21.920000", "content": "did you have the outlier column as part of CV? i did that by mistake and had similar results
", "votes": 0, "replies": [] } ] }, { "id": 468677, "author_name": "process531", "author_url": "", "post_date": "2019-02-09T12:45:06.263000", "content": "i'm interested in your work(Genetic Programming ).I don’t know if you can share it after the competition.
", "votes": 0, "replies": [] }, { "id": 467838, "author_name": "interneuron", "author_url": "", "post_date": "2019-02-07T20:26:45.297000", "content": "What are some of the craziest things you all have tried? In the absence of information about the task I have tried all sorts of things that don't make sense. For example, I tried digitizing the targets and training to predict the target bin, then added the predicted test bins back to test as a feature, it did not work well, shocking.
", "votes": 0, "replies": [] }, { "id": 473778, "author_name": "", "author_url": "", "post_date": "2019-02-18T13:49:14.887000", "content": "", "votes": 2, "replies": [ { "id": 477349, "author_name": "snovik", "author_url": "", "post_date": "2019-02-24T11:23:58.807000", "content": "Or they found some proper CV scheme. There was an idea from the Top1 to use uniform distribution for validation but tbh I not sure why it should be better than simple stratification by outliers. But smth in this direction might have helped a lot.
\n\nAnother possible direction of thought is actually NOT to ensemble in some tricky way. Because ensembling averages out all our magic.
", "votes": 1, "replies": [] }, { "id": 477354, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-24T11:41:09.223000", "content": "@snovik1975, ensembling also reduces the errors as per correct prediction :-)
", "votes": 0, "replies": [] }, { "id": 477413, "author_name": "snovik", "author_url": "", "post_date": "2019-02-24T13:58:14.407000", "content": "aren't we hunting for errors? :)
", "votes": 0, "replies": [] } ] }, { "id": 468724, "author_name": "", "author_url": "", "post_date": "2019-02-09T14:55:58.003000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "467294": "For me, this is the hardest competition I have ever entered - I have never experienced anything like it - almost everything I tried get scores between 2.68 (Genetic Programming and LGB) and 3.88 (FFM) - those blinking outliers are killing me! \n\nI have decided to spend a few days just looking at the data and get away from the weeds of modelling - there must be a way to connect all these card_ids and merchant_ids!\n\nHas anyone had a modicum of success using FFM or NMF - I tried but failed sadly. I hope I don't have to use networkx graphs!\n\nI am even starting to think we are trying to emulate their logistic regression model and the values are the raw values of a sigmoid function\n\n1/(1+exp(-p)) => p is our targets! Yes I am that desperate/clueless! ;)\n", "467851": "@Scirpus, have you just became a Discussion GRANDMASTER? Nice! Congratulations!", "467311": "\"cardids and merchantids\"\n\nI clearly see \"time series\" data and think there could be the key to success.\nThe host of this competition said that they removed historical data for the last two months because of \"data leakage\".\nProbably the key is there. If we can predict sales for that removed moths we can get lower RMSE.\n\nP.S. @scirpus thank you for all your public kernels, I don't have words to appreciate your work. I've learned so much from you. Thank you. ", "468432": "Hey @scirpus, congratulations for this well deserved GM (Genetically Modified) grade ! Thanks for all the things you share ;-)", "471114": "Genetic Programming is charming but i don't know much", "469775": "Well done!", "469142": "Good job", "469073": "congrats", "467812": "After my local CV improved from 3.641 to 6.635 (e.g. -0.006), my LB went up +0.014\nThen I changed seeds and here is what I got:\nbefore: CV 3.63552 std 0.10288\nafter: CV: 3.63921 std 0.03521\n\nI already expected an increase in the score, but std?!?! Local CV grew and std dropped A LOT, by like 2/3. If I overfitted to the old seeds, I would have expected the score AND maybe std to increase but NO.\n\nAnd I am already doing a repeated kfold.\n\nI am not giving up yet, but this competition is really weird.", "467603": "@Scirpus thanks for all your sharing. On this competition, I am coming to a similar conclusion as you have. I took a few days away, came back to it today and so far not much success.", "467317": "I’m right there with you. I have not yet been able to make a single model of whatever type that gets below a 3.89 on lb. It’s been quite frustrating but informative. \n\nHaving no idea what the target is seems to prevent most rational strategies. The monster aggs people are using do appear to capture some relevant information but by throwing everything at the wall and seeing what sticks. And one can only go so far with regression, I’m certain all the teams at the top are much better than me at identifying outliers. ", "468401": "Congratulations", "467784": "If by modicum you mean \"0.001\" , then yes i had improvement with FFM.", "467690": "I think that the key issue is that we have little information about the data and the business logic behind the loyalty score calculation. I spent the first 1.5 months, believing that the target is to calculate the next 2 months after the new merchants file, loyalty scores. Now, after reading several discussion topics i believe that i have the correct understanding :)\n\nMoreover, the host has not indicated how the are simulated. Is it just noise and they want to see modelling approaches? is it a statistic model? is it a formula? is it a combination of all ? \n\nAs the google store competition, we are trying to figure out by ourselves what is the target.....\n\na simple explanation by the host could have helped us a lot", "467347": "As it is the first competition that i decided to give it all i have, it's a big disappointment. It is probably the worst competition to begin on kaggle and learn. I am stuck at 3.87 and no matter what i do it doesn't improve. Ι tried feature engineering, different models, feature selection, different tuning, stacking, averaging, ffm, post-processing. Things that work elsewhere like averaging, don't help. I think i lost a lot of time here.. ", "467560": "Same situation here from me. I did not deal with the outliers (did not remove or replace them) I got 1.55X CV but, 3.84~3.86LB", "468677": "i'm interested in your work(Genetic Programming ).I don’t know if you can share it after the competition.", "467838": "What are some of the craziest things you all have tried? In the absence of information about the task I have tried all sorts of things that don't make sense. For example, I tried digitizing the targets and training to predict the target bin, then added the predicted test bins back to test as a feature, it did not work well, shocking. ", "473778": "", "468724": "" } }