oh i understand sorry ^^
", "rawMarkdown": "oh i understand sorry ^^", "votes": null }, { "id": "138227", "postDate": "10/07/2016 17:24:20", "content": "[quote=PrakashSubedi;138219]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
\n\n[/quote]
\n\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from here. Then:
\n\ncsv2libsvm.py data_numeric.csv data_numeric.libsvm 969\nThis will convert the .csv file into LibSVM format. The LibSVM file can be loaded as such into a sparse csr matrix.
\n\nfrom sklearn.datasets import load_svmlight_file\n# This will load the data into a sparse csr matrix and the target into an np array\nprint("\\n Reading train data ...")\ntrain_data, train_target = load_svmlight_file("./train_numeric.libsvm")\nprint(" Done reading train data (%d x %d)" % (train_data.shape[0], train_data.shape[1]))\nprint(" Memory consumption of the sparse matrix is %d bytes" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.
\n\nEdit: I should give credit to SVMLight as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.
", "rawMarkdown": "[quote=PrakashSubedi;138219]\r\n\r\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.\r\n\r\n[/quote]\r\n\r\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from [__here__](https://github.com/zygmuntz/phraug2). Then:\r\n\r\n\r\n csv2libsvm.py data_numeric.csv data_numeric.libsvm 969\r\n\r\n\r\nThis will convert the .csv file into [LibSVM](http://www.csie.ntu.edu.tw/~cjlin/libsvm/) format. The LibSVM file can be loaded as such into a sparse csr matrix.\r\n\r\n from sklearn.datasets import load_svmlight_file\r\n # This will load the data into a sparse csr matrix and the target into an np array\r\n print(\"\\n Reading train data ...\")\r\n train_data, train_target = load_svmlight_file(\"./train_numeric.libsvm\")\r\n print(\" Done reading train data (%d x %d)\" % (train_data.shape[0], train_data.shape[1]))\r\n print(\" Memory consumption of the sparse matrix is %d bytes\" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\r\n\r\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.\r\n\r\nEdit: I should give credit to [SVMLight](http://svmlight.joachims.org/) as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.", "votes": null }, { "id": "138228", "postDate": "10/07/2016 17:40:45", "content": "[quote=Tilii;138227]
\n\n[quote=PrakashSubedi;138219]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
\n\n[/quote]
\n\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from here. Then:
\n\ncsv2libsvm.py data_numeric.csv data_numeric.libsvm 969\nThis will convert the .csv file into LibSVM format. The LibSVM file can be loaded as such into a sparse csr matrix.
\n\nfrom sklearn.datasets import load_svmlight_file\n# This will load the data into a sparse csr matrix and the target into an np array\nprint("\\n Reading train data ...")\ntrain_data, train_target = load_svmlight_file("./train_numeric.libsvm")\nprint(" Done reading train data (%d x %d)" % (train_data.shape[0], train_data.shape[1]))\nprint(" Memory consumption of the sparse matrix is %d bytes" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.
\n\nEdit: I should give credit to SVMLight as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.
\n\n[/quote]
\n\nI was doing the same until I found out lots of columns in date table is exact duplicates.. Removing them eased the burden,and SVMLIGHT was not needed.
", "rawMarkdown": "[quote=Tilii;138227]\r\n\r\n[quote=PrakashSubedi;138219]\r\n\r\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.\r\n\r\n[/quote]\r\n\r\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from [__here__](https://github.com/zygmuntz/phraug). Then:\r\n\r\n\r\n csv2libsvm.py data_numeric.csv data_numeric.libsvm 969\r\n\r\n\r\nThis will convert the .csv file into [LibSVM](http://www.csie.ntu.edu.tw/~cjlin/libsvm/) format. The LibSVM file can be loaded as such into a sparse csr matrix.\r\n\r\n from sklearn.datasets import load_svmlight_file\r\n # This will load the data into a sparse csr matrix and the target into an np array\r\n print(\"\\n Reading train data ...\")\r\n train_data, train_target = load_svmlight_file(\"./train_numeric.libsvm\")\r\n print(\" Done reading train data (%d x %d)\" % (train_data.shape[0], train_data.shape[1]))\r\n print(\" Memory consumption of the sparse matrix is %d bytes\" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\r\n\r\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.\r\n\r\nEdit: I should give credit to [SVMLight](http://svmlight.joachims.org/) as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.\r\n\r\n\r\n[/quote]\r\n\r\nI was doing the same until I found out lots of columns in date table is exact duplicates.. Removing them eased the burden,and SVMLIGHT was not needed.", "votes": null }, { "id": "138232", "postDate": "10/07/2016 17:51:56", "content": "@anokas , but surely you got engineered your features from given tables ... ?
", "rawMarkdown": "anokas , but surely you got engineered your features from given tables ... ?", "votes": null }, { "id": "138233", "postDate": "10/07/2016 17:53:32", "content": "[quote=anokas;138225]
\n\n[quote=<<>>;138224]
\n\n@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
\n\n[/quote]
\n\n0.49 without any of the original tables I should say.
\n\n[/quote]\nthank you for clarifying, now i am more sure than before that i spotted the right feature.
", "rawMarkdown": "[quote=anokas;138225]\r\n\r\n[quote=<<point taken
", "rawMarkdown": "point taken", "votes": null }, { "id": "138235", "postDate": "10/07/2016 17:56:14", "content": "@Alam Check your 'golden' Inbox ....
", "rawMarkdown": "Alam Check your 'golden' Inbox ....", "votes": null }, { "id": "138236", "postDate": "10/07/2016 17:58:00", "content": "Maybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks.
", "rawMarkdown": "Maybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks.", "votes": null }, { "id": "138237", "postDate": "10/07/2016 17:59:06", "content": "As entertaining as all of this is we can probably do without the smacktalk. General Bickering is really not what this place is about or more correctly, what I had hoped this would be about. While it can be frustrating if you are not in the know, most of the time (and the current leaders have been in this category) at the end of the competition, they are more than happy to share insights. The post-competition discussion is probably as much a chance of learning as the pre-deadline discussion.
\n\nAlso I need to learn to unsubscribe to threads as you people blew up my mailbox over lunch.
\n\n@Tilli--ty for pointing out the SVM method. I did my the old-school way to convert for some vw fun. I should have known there was a potentially easier way.
\n\nAnd while you guys were blowing up my mailbox, I did realize a faster way to check for duplicate variables. Sometimes brute force through massive hardware is not the answer. It's nice to have, but unless you really understand what you are doing, it what is popularly called "shotgunning" the problem--fire a bunch a bullets and hope you hit something good.
", "rawMarkdown": "As entertaining as all of this is we can probably do without the smacktalk. General Bickering is really not what this place is about or more correctly, what I had hoped this would be about. While it can be frustrating if you are not in the know, most of the time (and the current leaders have been in this category) at the end of the competition, they are more than happy to share insights. The post-competition discussion is probably as much a chance of learning as the pre-deadline discussion.\r\n\r\nAlso I need to learn to unsubscribe to threads as you people blew up my mailbox over lunch.\r\n\r\n@Tilli--ty for pointing out the SVM method. I did my the old-school way to convert for some vw fun. I should have known there was a potentially easier way.\r\n\r\nAnd while you guys were blowing up my mailbox, I did realize a faster way to check for duplicate variables. Sometimes brute force through massive hardware is not the answer. It's nice to have, but unless you really understand what you are doing, it what is popularly called \"shotgunning\" the problem--fire a bunch a bullets and hope you hit something good.", "votes": null }, { "id": "138238", "postDate": "10/07/2016 18:06:15", "content": "[quote=Devin;138236]
\n\nMaybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks.
\n\n[/quote]
\n\nThis could be a Kaggle text-mining competition. Build a model that automatically sifts through all the forums and identifies only the posts that reveal the magic features. The question is, "Would that competition itself have any magic features?"
", "rawMarkdown": "[quote=Devin;138236]\r\n\r\nMaybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks. \r\n\r\n[/quote]\r\n\r\nThis could be a Kaggle text-mining competition. Build a model that automatically sifts through all the forums and identifies only the posts that reveal the magic features. The question is, \"Would that competition itself have any magic features?\"", "votes": null }, { "id": "138239", "postDate": "10/07/2016 18:06:16", "content": "The following works pretty well for nuking duplicates - it significantly reduces columns\n\nremove = []\ncols = rawdata.columns\nfor i in range(len(cols)-1):\n v = rawdata[cols[i]].values\n for j in range(i+1,len(cols)):\n if np.array_equal(v,rawdata[cols[j]].values):\n remove.append(cols[j])\n\nrawdata.drop(remove, axis=1, inplace=True)\nremove = []\nfor col in rawdata.columns[1:]:\n print(col, rawdata[col].std())\n if rawdata[col].std() == 0.0:\n remove.append(col)\n\nrawdata.drop(remove, axis=1, inplace=True)\n@BreakfastPirate, I can think of quite a few useful features. But to see some real magic I believe we would, for instance, need to ban leaderboard info and then get sloppy.
", "rawMarkdown": "BreakfastPirate, I can think of quite a few useful features. But to see some real magic I believe we would, for instance, need to ban leaderboard info and then get sloppy.", "votes": null }, { "id": "138249", "postDate": "10/07/2016 18:57:20", "content": "[quote=Mohd Shadab Alam;138233]
\n\n[quote=anokas;138225]
\n\n[quote=<<>>;138224]
\n\n@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
\n\n[/quote]
\n\n0.49 without any of the original tables I should say.
\n\n[/quote]\nthank you for clarifying, now i am more sure than before that i spotted the right feature.
\n\n[/quote]
\n\nIf you spotted the right features, then you should be able to run just on those features and get 0.4+ - If you can't then it's not the same features as us ;D
", "rawMarkdown": "[quote=Mohd Shadab Alam;138233]\r\n\r\n[quote=anokas;138225]\r\n\r\n[quote=<<@BreakfastPirate
\n\n"one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a model that predicts the remaining 70% of test data (i.e. predictive modeling)."
\n\nYour assumption that an overfit to the public leaderboard will result in a drop on the private leaderboard is not necessarily true. If this were true predictive modeling, where models are built on a relatively sparse amount of data, then this might be true. But with these large data sets, overfitting to the public leaderboard could, and does it seems, translate to an overfitting on the private leaderboard. And yes, the goal of this modeling is similar to the goal of predictive modeling, but my point is that the implementation of the modeling on a competition like this is actually not predictive modeling.
", "rawMarkdown": "BreakfastPirate\r\n\r\n\"one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a model that predicts the remaining 70% of test data (i.e. predictive modeling).\"\r\n\r\nYour assumption that an overfit to the public leaderboard will result in a drop on the private leaderboard is not necessarily true. If this were true predictive modeling, where models are built on a relatively sparse amount of data, then this might be true. But with these large data sets, overfitting to the public leaderboard could, and does it seems, translate to an overfitting on the private leaderboard. And yes, the goal of this modeling is similar to the goal of predictive modeling, but my point is that the implementation of the modeling on a competition like this is actually not predictive modeling.", "votes": null }, { "id": "138253", "postDate": "10/07/2016 19:02:52", "content": "@BreakfastPirate
\n\n"I am using xgboost.Booster.predict() so it is predictive modeling. :D"
\n\nI was hoping you were joking about that.
\n\n"But seriously do not worry too much about the number of submissions. In this case having more submissions is not really competitive advantage. If you think it is, you can submit 5 per day so there are still 165 submissions left for you."
\n\nI'm not worried about the number of submissions, I'm simply pointing out that the way this competition is set up, it's not really predictive modeling.
\n\n"Bigger problem could be that the train set contains more information about the test set than it was probably intended. Leakages/ magic features are usually appear in these competitions."
\n\nThat could be a problem as well. But that is a problem that we can't say for sure will occur with every competition. But what we can say about a competition like this is, by giving information about results from a test set, that test set is now a training set. That's my point.
", "rawMarkdown": "BreakfastPirate\r\n\r\n\"I am using xgboost.Booster.predict() so it is predictive modeling. :D\"\r\n\r\nI was hoping you were joking about that.\r\n\r\n\r\n\"But seriously do not worry too much about the number of submissions. In this case having more submissions is not really competitive advantage. If you think it is, you can submit 5 per day so there are still 165 submissions left for you.\"\r\n\r\nI'm not worried about the number of submissions, I'm simply pointing out that the way this competition is set up, it's not really predictive modeling.\r\n\r\n\r\n\"Bigger problem could be that the train set contains more information about the test set than it was probably intended. Leakages/ magic features are usually appear in these competitions.\"\r\n\r\nThat could be a problem as well. But that is a problem that we can't say for sure will occur with every competition. But what we can say about a competition like this is, by giving information about results from a test set, that test set is now a training set. That's my point.", "votes": null }, { "id": "138254", "postDate": "10/07/2016 19:09:58", "content": "@happycube
\n\n"Also note that CV and LB scores are very highly correlated, so raddar didn't need to use the public LB to know he's making progress."
\n\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher. But somehow they knew to continue in a certain direction and get up to the 0.50 point that they are now.
\n\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.
", "rawMarkdown": "happycube\r\n\r\n\"Also note that CV and LB scores are very highly correlated, so raddar didn't need to use the public LB to know he's making progress.\"\r\n\r\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher. But somehow they knew to continue in a certain direction and get up to the 0.50 point that they are now.\r\n\r\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.", "votes": null }, { "id": "138255", "postDate": "10/07/2016 19:12:34", "content": "@BreakfastPirate
\n\n"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nYou just reiterated my point right there. That 30% holdout set that was labeled as a "test" set. You just said that you're using that "test" set for training. Therefore, what was put up as a test set is actually a training set. Now you see, and we are in agreement.
", "rawMarkdown": "BreakfastPirate\r\n\r\n\"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?\"\r\n\r\nYou just reiterated my point right there. That 30% holdout set that was labeled as a \"test\" set. You just said that you're using that \"test\" set for training. Therefore, what was put up as a test set is actually a training set. Now you see, and we are in agreement.", "votes": null }, { "id": "138256", "postDate": "10/07/2016 19:14:48", "content": "[quote=GWillie;138254]
\n\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.
\n\n[/quote]
\n\nYou are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. This is your test set!
\n\nEDIT:\n"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nThat was me.
", "rawMarkdown": "[quote=GWillie;138254]\r\n\r\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.\r\n\r\n[/quote]\r\n\r\nYou are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. **This is** your test set!\r\n\r\n\r\n\r\nEDIT:\r\n\"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?\"\r\n\r\nThat was me.", "votes": null }, { "id": "138257", "postDate": "10/07/2016 19:15:03", "content": "GWillie, not to be pedantic, but... nobody likes pedantic people.
\n\nWe get that you have issues with the absolute definition of "predictive modeling". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?
\n\nCan we please move on?
", "rawMarkdown": "GWillie, not to be pedantic, but... nobody likes pedantic people.\r\n\r\nWe get that you have issues with the absolute definition of \"predictive modeling\". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?\r\n\r\nCan we please move on?", "votes": null }, { "id": "138259", "postDate": "10/07/2016 19:22:17", "content": "[quote=Taffey Lewis;138257]
\n\nGWillie, not to be pedantic, but... nobody likes pedantic people.
\n\n[/quote]
\n\nI like pedantic people, and I am not saying that just to be a contrarian.
\n\nI also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens.
", "rawMarkdown": "[quote=Taffey Lewis;138257]\r\n\r\nGWillie, not to be pedantic, but... nobody likes pedantic people.\r\n\r\n[/quote]\r\n\r\nI like pedantic people, and I am not saying that just to be a contrarian.\r\n\r\nI also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens.", "votes": null }, { "id": "138260", "postDate": "10/07/2016 19:26:12", "content": "Well, Now there is no link whatsoever between the thread title and the discussion over last two pages !!!
", "rawMarkdown": "Well, Now there is no link whatsoever between the thread title and the discussion over last two pages !!!", "votes": null }, { "id": "138261", "postDate": "10/07/2016 19:32:49", "content": "Let us debate the magical Feature issues. I'm really at the end of my rope. I can`t find pattern from data.
", "rawMarkdown": "Let us debate the magical Feature issues. I'm really at the end of my rope. I can`t find pattern from data.", "votes": null }, { "id": "138266", "postDate": "10/07/2016 19:42:11", "content": "Couldn't resist.
\n\n![\"nice\"](\"http://i.imgur.com/jLZz4Na.gif\")
@MaxPower
\n\nRegarding the Grupo Bimbo competition, were you given any feedback (i.e., scores) up until Week 9? If you were given no information about the test sets up until the final/private leaderboard was revealed, then sure, that would be predictive modeling. But that's not what's happening with this competition, nor with the others I've seen during my short time on here.
\n\nAnd yes, I am basically criticizing the train/test splits. That and the fact that 30% of the "test" set is actually a training set, which adds to the train/test split. I would suggest that the process could definitely be improved.
", "rawMarkdown": "MaxPower\r\n\r\nRegarding the Grupo Bimbo competition, were you given any feedback (i.e., scores) up until Week 9? If you were given no information about the test sets up until the final/private leaderboard was revealed, then sure, that would be predictive modeling. But that's not what's happening with this competition, nor with the others I've seen during my short time on here.\r\n\r\nAnd yes, I am basically criticizing the train/test splits. That and the fact that 30% of the \"test\" set is actually a training set, which adds to the train/test split. I would suggest that the process could definitely be improved.", "votes": null }, { "id": "138287", "postDate": "10/07/2016 22:49:39", "content": "@Tilii
\n\n"You said "not predictive modeling" 5 times in the last few messages. Based on your level of conviction, one would think that you must be absolutely right. And you are not."
\n\nIn my first post, I said it only a couple times and none of the responses addressed that directly. So I decided to repeat it more often for effect, to see if it would elicit responses that could lead to a more meaningful discussion. Apparently it worked, and thanks for at least addressing that point directly.
\n\n"To expand on BreakfastPirate's fine argument: we get feedback only on portion of the data, and our models are ultimately tested on data that have not been part of that feedback. That is different from "regular" predictive modeling only in a sense that our only info there is our local cross-validation score, while here we can compare it with leaderboard. Those who know how to do cross-validation properly would do just fine without that piece of information."
\n\nSo if those who climb the leaderboard could truly do that without that piece of information and associated feedback (information and feedback that is disqualifying this from being predictive modeling up to the private leaderboard part), why have that piece of information and the associated feedback? So we're just supposed to assume that they would have doubled their score in 2 weeks (prior to private leaderboard time) without that "piece of information?" With this much money at stake, I'm not necessarily inclined to give someone that benefit of the doubt.
\n\nHere's the real problem with all this. I get the whole 30/70 split on the "test" set. But let's consider the Red Hat competition, which I believe was run the same way. The training/"test" set split on that was 82%/18%. Now take 30% of that 18% test set which has actually become a training set because of the information/feedback during the public leaderboard portion, and you're up to a 87%/13% training/test set split. And considering that you're probably getting more useful information than just the 30% from that "test" set, the split is probably even higher - mabye even getting over 90%. One could argue that this kind of split is true predictive modeling, but I would say that as you approach this split level, you're really just constructing a model to fit the ~90% of observed data. The 10% (or probably less) of "test" data is simply serving as a means to assess the results and determine a winner. Maybe the 10% is different enough that you see significant drops in the private leaderboard, that's possible. But my point is, up until that point - during the public leaderboard time - the training/test set notion is misleading. What is labeled as test data during that time is actually training data because of the feedback.
\n\nSo yes, as far as this competition and others like it, I stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data. And the problem with this is, it is presented as though it is predictive modeling (training set/test set). And to someone like myself who is fairly new here, I came expecting to see predictive modeling. But after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.
", "rawMarkdown": "Tilii\r\n\r\n\"You said \"not predictive modeling\" 5 times in the last few messages. Based on your level of conviction, one would think that you must be absolutely right. And you are not.\"\r\n\r\nIn my first post, I said it only a couple times and none of the responses addressed that directly. So I decided to repeat it more often for effect, to see if it would elicit responses that could lead to a more meaningful discussion. Apparently it worked, and thanks for at least addressing that point directly.\r\n\r\n\r\n\"To expand on BreakfastPirate's fine argument: we get feedback only on portion of the data, and our models are ultimately tested on data that have not been part of that feedback. That is different from \"regular\" predictive modeling only in a sense that our only info there is our local cross-validation score, while here we can compare it with leaderboard. Those who know how to do cross-validation properly would do just fine without that piece of information.\"\r\n\r\nSo if those who climb the leaderboard could truly do that without that piece of information and associated feedback (information and feedback that is disqualifying this from being predictive modeling up to the private leaderboard part), why have that piece of information and the associated feedback? So we're just supposed to assume that they would have doubled their score in 2 weeks (prior to private leaderboard time) without that \"piece of information?\" With this much money at stake, I'm not necessarily inclined to give someone that benefit of the doubt.\r\n\r\nHere's the real problem with all this. I get the whole 30/70 split on the \"test\" set. But let's consider the Red Hat competition, which I believe was run the same way. The training/\"test\" set split on that was 82%/18%. Now take 30% of that 18% test set which has actually become a training set because of the information/feedback during the public leaderboard portion, and you're up to a 87%/13% training/test set split. And considering that you're probably getting more useful information than just the 30% from that \"test\" set, the split is probably even higher - mabye even getting over 90%. One could argue that this kind of split is true predictive modeling, but I would say that as you approach this split level, you're really just constructing a model to fit the ~90% of observed data. The 10% (or probably less) of \"test\" data is simply serving as a means to assess the results and determine a winner. Maybe the 10% is different enough that you see significant drops in the private leaderboard, that's possible. But my point is, up until that point - during the public leaderboard time - the training/test set notion is misleading. What is labeled as test data during that time is actually training data because of the feedback.\r\n\r\nSo yes, as far as this competition and others like it, I stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data. And the problem with this is, it is presented as though it is predictive modeling (training set/test set). And to someone like myself who is fairly new here, I came expecting to see predictive modeling. But after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.", "votes": null }, { "id": "138288", "postDate": "10/07/2016 22:58:52", "content": "@Charly
\n\n"You are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. This is your test set!"
\n\nYes, I get this. But again, consider that in the Red Hat competition, that 70% of the "test" set represented only 13% of the overall data. So that was basically just constructing a model to fit 87% (or more likely over 90%) of the data. There is not much prediction done there.
\n\nEDIT: "Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nAgain, the difference could be bumping up the training/test split from about 80/20 to more like 90/10.
\n\nBut all that is really irrelevant to what I said from the start. If you consider the public leaderboard part of these competitions, it's not predictive modeling that's going on. That was my point. It's purported to be, given the "training set" and "test set" labels. But by giving feedback on that "test set" and allowing people to refine their models based on that feedback, that is no longer a test set and that is no longer predictive modeling. So for competitions that are run like that, there is no predictive modeling going on during a majority of the competition. And, by making the training set such a large % of the total data, there really isn't much prediction going on, even at the end.
", "rawMarkdown": "Charly\r\n\r\n\"You are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. This is your test set!\"\r\n\r\nYes, I get this. But again, consider that in the Red Hat competition, that 70% of the \"test\" set represented only 13% of the overall data. So that was basically just constructing a model to fit 87% (or more likely over 90%) of the data. There is not much prediction done there.\r\n\r\n\r\nEDIT: \"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?\"\r\n\r\nAgain, the difference could be bumping up the training/test split from about 80/20 to more like 90/10.\r\n\r\nBut all that is really irrelevant to what I said from the start. If you consider the public leaderboard part of these competitions, it's not predictive modeling that's going on. That was my point. It's purported to be, given the \"training set\" and \"test set\" labels. But by giving feedback on that \"test set\" and allowing people to refine their models based on that feedback, that is no longer a test set and that is no longer predictive modeling. So for competitions that are run like that, there is no predictive modeling going on during a majority of the competition. And, by making the training set such a large % of the total data, there really isn't much prediction going on, even at the end.", "votes": null }, { "id": "138289", "postDate": "10/07/2016 23:03:16", "content": "@Tilii
\n\n"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens."
\n\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.
", "rawMarkdown": "Tilii\r\n\r\n\"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens.\"\r\n\r\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.", "votes": null }, { "id": "138290", "postDate": "10/07/2016 23:06:50", "content": "Enough.... please.... (or if you must please start another thread!)
", "rawMarkdown": "Enough.... please.... (or if you must please start another thread!)", "votes": null }, { "id": "138291", "postDate": "10/07/2016 23:09:44", "content": "@Taffey Lewis
\n\nThat's the whole reason I brought this up. I wonder if a company like Bosch really does understand EXACTLY what they're getting into with Kaggle. I agree that this exercise in constructing a model to fit a large set of data is useful and not an easy task. I agree that they will see value in this. But what is happening over and over in a wide variety of industries is that there is a certain level of disappointment when trying to apply these models in the true sense of predictive modeling. That is, when you have much less than 90% of the data available as observations upon which to build a model. When overfitting becomes your worst enemy instead of your ally. That is true predictive modeling, and it is not being addressed in these recent competitions I've seen. Given that it remains a huge problem, I would argue that we should all care.
\n\nWe get that you have issues with the absolute definition of "predictive modeling". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?
\n\nCan we please move on?
", "rawMarkdown": "Taffey Lewis\r\n\r\nThat's the whole reason I brought this up. I wonder if a company like Bosch really does understand EXACTLY what they're getting into with Kaggle. I agree that this exercise in constructing a model to fit a large set of data is useful and not an easy task. I agree that they will see value in this. But what is happening over and over in a wide variety of industries is that there is a certain level of disappointment when trying to apply these models in the true sense of predictive modeling. That is, when you have much less than 90% of the data available as observations upon which to build a model. When overfitting becomes your worst enemy instead of your ally. That is true predictive modeling, and it is not being addressed in these recent competitions I've seen. Given that it remains a huge problem, I would argue that we should all care.\r\n\r\n\r\nWe get that you have issues with the absolute definition of \"predictive modeling\". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?\r\n\r\nCan we please move on?", "votes": null }, { "id": "138293", "postDate": "10/07/2016 23:11:02", "content": "[quote=GWillie;138289]
\n\n@Tilii
\n\n"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens."
\n\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.
\n\n[/quote]
\n\nYou are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man. From wikipedia:
\n\n\n\n\nPredictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place.
\n
You build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling. Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction. You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects.
", "rawMarkdown": "[quote=GWillie;138289]\r\n\r\n@Tilii\r\n\r\n\"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens.\"\r\n\r\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.\r\n\r\n[/quote]\r\n\r\nYou are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man. From wikipedia:\r\n> Predictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place.\r\n\r\nYou build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling. Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction. You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects.", "votes": null }, { "id": "138294", "postDate": "10/07/2016 23:13:56", "content": "@Chippy
\n\nIt is relevant to this competition. That's why I made the comment here. I'm happy to take the general discussion to another thread, although I would think that people would actually want to give this some thought rather than just attack the messenger. People are on here to try to get information about how to become or improve their skills as a data scientist. Misleading them is doing a huge disservice to them and the field. But let's just sweep it under the rug and move on. That should adequately address the problems we're seeing in the field.
", "rawMarkdown": "Chippy\r\n\r\nIt is relevant to this competition. That's why I made the comment here. I'm happy to take the general discussion to another thread, although I would think that people would actually want to give this some thought rather than just attack the messenger. People are on here to try to get information about how to become or improve their skills as a data scientist. Misleading them is doing a huge disservice to them and the field. But let's just sweep it under the rug and move on. That should adequately address the problems we're seeing in the field.", "votes": null }, { "id": "138296", "postDate": "10/07/2016 23:24:14", "content": "I also agree with Tilii, but to my point of view your arguments are just not right. The train/ test split here is 0.65/ 0.35 with the special condition that 15 % of the set is only available by checking your model on the public leaderboard. 35 % are not seen until the end. That fits my definition of predictive modeling.
\n\nLet's agree that we disagree and go on chase some wild goose.
", "rawMarkdown": "I also agree with Tilii, but to my point of view your arguments are just not right. The train/ test split here is 0.65/ 0.35 with the special condition that 15 % of the set is only available by checking your model on the public leaderboard. 35 % are not seen until the end. That fits my definition of predictive modeling.\r\n\r\nLet's agree that we disagree and go on chase some wild goose.", "votes": null }, { "id": "138297", "postDate": "10/07/2016 23:29:38", "content": "[quote=GWillie;138287]
\n\nI stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data....\nBut after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.
\n\n[/quote]
\n\nYou say "the majority of the competition" is focused on the public LB, BUT in reality the ENTIRE duration of every competition is focused on getting the highest rank on the private LB. The public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...
\n\nSo your whole argument is based on your misunderstanding of how Kaggle competitions really work.
", "rawMarkdown": "[quote=GWillie;138287]\r\n\r\nI stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data....\r\nBut after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.\r\n\r\n[/quote]\r\n\r\nYou say \"the majority of the competition\" is focused on the public LB, BUT in reality the ENTIRE duration of every competition is focused on getting the highest rank on the *private* LB. The public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...\r\n\r\nSo your whole argument is based on your misunderstanding of how Kaggle competitions really work.", "votes": null }, { "id": "138298", "postDate": "10/07/2016 23:31:47", "content": "@Jared Turkewiz
\n\nThank you for clarifying my position. Everything you posted points to me being correct.
\n\n"You are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man."
\n\nAnd if the private leaderboard scores are based on only about 10% of available data, you're not really doing predictive modeling. So that argument is not a straw-man.
\n\n"From wikipedia:
\n\nPredictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event,"
\n\nLet's stop it there for a minute. So by giving information about the events which you are trying to "predict", those events cease to be "uknown" events. Thus I am right about the public leaderboard portion not being predictive modeling. Thank you for providing even more evidence to support me.
\n\n"regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place."
\n\nRight, and this is not an issue of timing, it is an issue of information that is made available.
\n\n"You build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling."
\n\nRight, and by giving people the "answer" as to how many of their modeled data is correct, it fails to be predictive modeling, according to the textbook definition that you provided.
\n\n"Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction."
\n\nI never said anything about requiring the prediction to be in the future. Again, it is about information. During the leaderboard portion of a competition, you are getting information about the answer. Therefore you do know something about the answer, therefore, according to you, you are not doing predictive modeling.
\n\n"You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects."
\n\nOf course you are allowed to know properties of the event you are trying to predict the outcome of, that's what predictive models are based on. But if you know, or have information about, the actual outcome (not just properties of it, but "it"), then you're not doing predictive modeling. And by getting a total number of correct outcomes from your model (as you do from the public leaderboard scores), that is information about the actual outcomes. And once you have that information, you are just fitting a model to that data, you're not predicting that data. And this is exactly the kind of information that people are using to move up the public leaderboard and double their score in 2 weeks time. If you fail to acknowledge this, then don't be surprised if you continue to see the same people game this and move up to the top - competition after competition. I guess just keep attributing that to luck?
", "rawMarkdown": "Jared Turkewiz\r\n\r\nThank you for clarifying my position. Everything you posted points to me being correct.\r\n\r\n\r\n\"You are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man.\"\r\n\r\nAnd if the private leaderboard scores are based on only about 10% of available data, you're not really doing predictive modeling. So that argument is not a straw-man.\r\n\r\n\r\n\"From wikipedia:\r\n\r\nPredictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event,\"\r\n\r\nLet's stop it there for a minute. So by giving information about the events which you are trying to \"predict\", those events cease to be \"uknown\" events. Thus I am right about the public leaderboard portion not being predictive modeling. Thank you for providing even more evidence to support me.\r\n\r\n\r\n\"regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place.\"\r\n\r\nRight, and this is not an issue of timing, it is an issue of information that is made available.\r\n\r\n\r\n\"You build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling.\"\r\n\r\nRight, and by giving people the \"answer\" as to how many of their modeled data is correct, it fails to be predictive modeling, according to the textbook definition that you provided.\r\n\r\n\r\n\"Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction.\"\r\n\r\nI never said anything about requiring the prediction to be in the future. Again, it is about information. During the leaderboard portion of a competition, you are getting information about the answer. Therefore you do know something about the answer, therefore, according to you, you are not doing predictive modeling.\r\n\r\n\r\n\"You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects.\"\r\n\r\nOf course you are allowed to know properties of the event you are trying to predict the outcome of, that's what predictive models are based on. But if you know, or have information about, the actual outcome (not just properties of it, but \"it\"), then you're not doing predictive modeling. And by getting a total number of correct outcomes from your model (as you do from the public leaderboard scores), that is information about the actual outcomes. And once you have that information, you are just fitting a model to that data, you're not predicting that data. And this is exactly the kind of information that people are using to move up the public leaderboard and double their score in 2 weeks time. If you fail to acknowledge this, then don't be surprised if you continue to see the same people game this and move up to the top - competition after competition. I guess just keep attributing that to luck?", "votes": null }, { "id": "138300", "postDate": "10/07/2016 23:42:29", "content": "@BreakfastPirate
\n\nI am new here and don't totally understand the private LB. Are you saying that you get feedback during the competition about the test data that will be used for the private LB? If so, that only strengthens my position and does not weaken it. If you're getting feedback on the private LB, then you're getting information on 100% of the true test set, and thus there really is no test set, and thus there really is no predictive modeling.
\n\nIf you don't get feedback on the private LB until the end, then my argument remains the same and I am not wrong. Yes, the final rankings are based on the private LB. But during the public LB portion, you are getting feedback about your submissions (this I know). So that portion is not predictive modeling - it is building a model to fit the training data + 30% of the "test" data. And in some cases, like Red Hat, that data can be upwards of 90% of the total data. This is basically fitting a model to data. You say that public LB means nothing in the end, but it actually means everything in the end for a competition like that. It is the time that people spend constructing and refining their models. And when that final "prediction" is on only 10% of all the data, then it is likely that the ones who did best on the public LB will do best on the private LB (not guaranteed, but likely). Therefore, being highly ranked on the public LB in those competitions is pretty much what it's all about. Granted, in true predictive modeling, where training set data is much less than 90% of all available data, then ranking on public LB would not be as important an indicator of success on the private LB. But that isn't the case in most of these competitions that I've seen.
\n\nSo, my whole argument is based on my understanding of how the public LB works in Kaggle competitions. I may have some misunderstanding of how the private LB works, but clearing that up could only strengthen my argument.
", "rawMarkdown": "BreakfastPirate\r\n\r\nI am new here and don't totally understand the private LB. Are you saying that you get feedback during the competition about the test data that will be used for the private LB? If so, that only strengthens my position and does not weaken it. If you're getting feedback on the private LB, then you're getting information on 100% of the true test set, and thus there really is no test set, and thus there really is no predictive modeling.\r\n\r\nIf you don't get feedback on the private LB until the end, then my argument remains the same and I am not wrong. Yes, the final rankings are based on the private LB. But during the public LB portion, you are getting feedback about your submissions (this I know). So that portion is not predictive modeling - it is building a model to fit the training data + 30% of the \"test\" data. And in some cases, like Red Hat, that data can be upwards of 90% of the total data. This is basically fitting a model to data. You say that public LB means nothing in the end, but it actually means everything in the end for a competition like that. It is the time that people spend constructing and refining their models. And when that final \"prediction\" is on only 10% of all the data, then it is likely that the ones who did best on the public LB will do best on the private LB (not guaranteed, but likely). Therefore, being highly ranked on the public LB in those competitions is pretty much what it's all about. Granted, in true predictive modeling, where training set data is much less than 90% of all available data, then ranking on public LB would not be as important an indicator of success on the private LB. But that isn't the case in most of these competitions that I've seen.\r\n\r\nSo, my whole argument is based on my understanding of how the public LB works in Kaggle competitions. I may have some misunderstanding of how the private LB works, but clearing that up could only strengthen my argument.", "votes": null }, { "id": "138301", "postDate": "10/07/2016 23:44:04", "content": "Guys, let's just stop answering. To my point of view you just not have the right mindset for kaggle. You know what my biggest incentive for participating is? I learned how to deal with in memory computing of data with billions of entries. ..
", "rawMarkdown": "Guys, let's just stop answering. To my point of view you just not have the right mindset for kaggle. You know what my biggest incentive for participating is? I learned how to deal with in memory computing of data with billions of entries. ..", "votes": null }, { "id": "138302", "postDate": "10/07/2016 23:45:28", "content": "@GWillie, either your definition of "predictive modelling" is out of line with the general usage or there is a misunderstanding about how the public leaderboard is used by Kagglers.
\n\nFor the former, see @Jared Turkewitz's response. As a hypothetical example, let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did "predictive modelling" because he did not use any information on the leaderboard.
\n\nNow, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do "predictive modelling"?
\n\nFor the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing "predictive modelling".
\n\nThe other group spends dozens or hundreds of hours creating many different kinds of models. They are doing "predictive modelling", irrespective of how many submissions were made. I'm speaking in generalities here. There have been competitions that were broken due to leakage or when the distribution of the test set is really different than the train set. I'm hoping this isn't the case with Bosch.
", "rawMarkdown": "GWillie, either your definition of \"predictive modelling\" is out of line with the general usage or there is a misunderstanding about how the public leaderboard is used by Kagglers.\r\n\r\nFor the former, see @Jared Turkewitz's response. As a hypothetical example, let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did \"predictive modelling\" because he did not use any information on the leaderboard.\r\n\r\nNow, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do \"predictive modelling\"?\r\n\r\nFor the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing \"predictive modelling\".\r\n\r\nThe other group spends dozens or hundreds of hours creating many different kinds of models. They are doing \"predictive modelling\", irrespective of how many submissions were made. I'm speaking in generalities here. There have been competitions that were broken due to leakage or when the distribution of the test set is really different than the train set. I'm hoping this isn't the case with Bosch.", "votes": null }, { "id": "138304", "postDate": "10/07/2016 23:50:17", "content": "[quote=BreakfastPirate;138297]
\n\nThe public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...
\n\n[/quote]
\n\nIt means nothing to people who understand Kaggle. But I've seen so many resumes in which people claim they've gotten in the top 10% in a competition. But a quick check shows that they only participated in 1 competition and fell thousands of places in the private leaderboard.
", "rawMarkdown": "[quote=BreakfastPirate;138297]\r\n\r\nThe public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...\r\n\r\n[/quote]\r\n\r\nIt means nothing to people who understand Kaggle. But I've seen so many resumes in which people claim they've gotten in the top 10% in a competition. But a quick check shows that they only participated in 1 competition and fell thousands of places in the private leaderboard.", "votes": null }, { "id": "138305", "postDate": "10/07/2016 23:50:24", "content": "@Charly
\n\nI think you're getting two different arguments confused here. One is my original one, the other is one that came up subsequently.
\n\nRegarding the train/test split, I know that the split on the Red Hat competition was 0.82/0.18 to start, and that became at least 0.87/0.13, more like 0.9/0.1 after considering that 30% of the "test" set was actually training data. So we can squabble over the splits and whether that constitutes true predictive modeling or not, but I will stand by my statement that a 0.9/0.1 is certainly approaching the point where you're not doing true predictive modeling.
\n\nBut all of that is irrelevant to my initial point. That is, during the public LB portion, where there is supposedly "predictive modeling" going on, there is actually not. This is due to the fact that you are getting information about the outcomes of your "test" set. By definition, that then fails to be a test set and becomes a training set, and you cease to do predictive modeling. This last argument is not a matter of opinion - like the train/test split could be. This is a fact according to the definition of predictive modeling. You can disagree with a more subjective matter like the train/test splits, but I find it difficult to see how one can knowingly disagree with facts.
", "rawMarkdown": "Charly\r\n\r\nI think you're getting two different arguments confused here. One is my original one, the other is one that came up subsequently.\r\n\r\nRegarding the train/test split, I know that the split on the Red Hat competition was 0.82/0.18 to start, and that became at least 0.87/0.13, more like 0.9/0.1 after considering that 30% of the \"test\" set was actually training data. So we can squabble over the splits and whether that constitutes true predictive modeling or not, but I will stand by my statement that a 0.9/0.1 is certainly approaching the point where you're not doing true predictive modeling.\r\n\r\nBut all of that is irrelevant to my initial point. That is, during the public LB portion, where there is supposedly \"predictive modeling\" going on, there is actually not. This is due to the fact that you are getting information about the outcomes of your \"test\" set. By definition, that then fails to be a test set and becomes a training set, and you cease to do predictive modeling. This last argument is not a matter of opinion - like the train/test split could be. This is a fact according to the definition of predictive modeling. You can disagree with a more subjective matter like the train/test splits, but I find it difficult to see how one can knowingly disagree with facts.", "votes": null }, { "id": "138308", "postDate": "10/08/2016 00:07:19", "content": "@GWillie,\nTake a closer look at how kaggle splits data or read one more time on what others said. Private LB is only revealed in the end of competition. \nSay you have 100K data points, now you split these 100K into 3 parts, the first one is training, second one is public LB, and third one is private LB.\nSo you train on training and get feedback from public LB, and then select best submissions based on everything you have and then hope for the best in private LB.
", "rawMarkdown": "GWillie,\r\nTake a closer look at how kaggle splits data or read one more time on what others said. Private LB is only revealed in the end of competition. \r\nSay you have 100K data points, now you split these 100K into 3 parts, the first one is training, second one is public LB, and third one is private LB.\r\nSo you train on training and get feedback from public LB, and then select best submissions based on everything you have and then hope for the best in private LB.", "votes": null }, { "id": "138314", "postDate": "10/08/2016 00:45:39", "content": "@GWillie. The main purpose of public and private LB split is, "to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set." (https://www.kaggle.com/wiki/Leaderboard)
\n\nIn real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction.
\n\nKaggle uses "hidden private test set & leaderboard" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction.
\n\nThe only way to have good private LB score is to generalize the model, it's happened when your local CV score is consistent with public LB score. This is similar to the real world when you do cross validation.
\n\nSo, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case).
\n\nPublic test set (30%) and its LB table is an helper.
\n\n\n\n
And now, please someone kindly share hint of the golden feature ;))
", "rawMarkdown": "GWillie. The main purpose of public and private LB split is, \"*to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set*.\" (https://www.kaggle.com/wiki/Leaderboard)\r\n\r\nIn real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction. \r\n\r\nKaggle uses \"hidden private test set & leaderboard\" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction. \r\n\r\nThe only way to have good private LB score is to generalize the model, it's happened when your local CV score is consistent with public LB score. This is similar to the real world when you do cross validation. \r\n\r\nSo, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case). \r\n\r\nPublic test set (30%) and its LB table is an helper. \r\n\r\n-------------------\r\n\r\nAnd now, please someone kindly share hint of the golden feature ;))", "votes": null }, { "id": "138315", "postDate": "10/08/2016 01:18:31", "content": "@Wilie Liao
\n\nThanks for your thoughtful response. Finally we have something we can use as a basis for real discussion.
\n\nI do think my definition of "predictive modeling" (which is based on the true definition) is out of line with the general usage on here. That's kinda my whole point; i.e., that term is being used inappropriately on here. And I am still confused about people's interpretation of how the public LB is being used by Kagglers. I'm hoping we can clear some of that up.
\n\nI saw Jared's response and used his statement of the true definition of "predictive modeling" to further support my position. Everything he said supported my claims, he just didn't realize it before posting it.
\n\nSo yes, let's consider the hypothetical examples:
\n\n"Let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did "predictive modelling" because he did not use any information on the leaderboard."
\n\nI would agree - if there were not an immediate assessment of his prediction in the form of a leaderboard score. As soon as that score pops up, it represents information on that "test" set. That test set ceases to be completely "unknown" in the sense that we now know how many right we got from our prediction. Therefore, the test set becomes a training set at that point (by definition) whether he uses it or not.
\n\n"Now, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do "predictive modelling"?
\n\nAgain, same response as above. It would be predictive modeling if not for that information given on the leaderboard, whether he uses it or not.
\n\n"For the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing "predictive modelling"."
\n\nExactly. And the point is, there is no guarantee that this latter case is not being used. We have no way of assuming that your more innocent first hypothetical is indeed the case.
\n\nHere, let me give you a hypothetical to try to clarify my point.
\n\nLet's say I ask you and 100 other people to play the game "I'm thinking of a number between 1 and 100, try to guess (predict) it. Whoever comes closest wins $50,000. And I'll give you 3 months to study it (me)." You're a smart guy, you're going to do whatever you can to increase your odds of winning. So you do your homework on me (age, address, zip code, birthday, birthday of kids, etc.). You put a lot of work into it, as you should, there's $50,000 on the line. Now, instead of just saying, at the end, I want all of you to guess and the closest wins, I say, you can make 2 guesses a day for 3 months. AND, there will be a guy on the side who is pointing thumbs up (higher) or thumbs down (lower) after you make each guess. But remember, you can only make 2 guesses per day (and each guess is assumed to be a true "prediction"). A few of the people get smart after they have each made a few guesses and team up, share information about their guesses, and work together from that point forward. And then we notice that there are some (especially that group that pooled their knowledge) who come much closer and closer to the true number as they make more and more guesses (and the guy on the side gives the thumbs up or thumbs down after each guess). Now, according to the participants, all of that is irrelevant - all that matters if their final guess. But some are saying, wait a minute, it seems fishy that there's this guy on the side giving feedback after each guess AND their guesses are getting better and better. Nooooo, you say, people aren't gonna actually use thumb guy information. Each guess is totally independent and in no way are they using the feedback from thumb guy on the side. I mean, that's just not right, and not in the spirit of the competition for them to be looking at thumb guy. To that I would say, seriously? There's $50,000 on the line and they have this source of information, and their guesses are getting better, yet they're not looking over at thumb guy after each guess? Seriously? So then, at the end, this group gets the number right on the nose. Dang, they're good. They just know how to do their homework and make really good educated guesses. What? That guy on the side, how could that have possibly played into it? To that I would again reiterate - seriously?
", "rawMarkdown": "Wilie Liao\r\n\r\nThanks for your thoughtful response. Finally we have something we can use as a basis for real discussion.\r\n\r\nI do think my definition of \"predictive modeling\" (which is based on the true definition) is out of line with the general usage on here. That's kinda my whole point; i.e., that term is being used inappropriately on here. And I am still confused about people's interpretation of how the public LB is being used by Kagglers. I'm hoping we can clear some of that up.\r\n\r\nI saw Jared's response and used his statement of the true definition of \"predictive modeling\" to further support my position. Everything he said supported my claims, he just didn't realize it before posting it.\r\n\r\nSo yes, let's consider the hypothetical examples:\r\n\r\n\r\n\"Let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did \"predictive modelling\" because he did not use any information on the leaderboard.\"\r\n\r\nI would agree - if there were not an immediate assessment of his prediction in the form of a leaderboard score. As soon as that score pops up, it represents information on that \"test\" set. That test set ceases to be completely \"unknown\" in the sense that we now know how many right we got from our prediction. Therefore, the test set becomes a training set at that point (by definition) whether he uses it or not.\r\n\r\n\r\n\"Now, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do \"predictive modelling\"?\r\n\r\nAgain, same response as above. It would be predictive modeling if not for that information given on the leaderboard, whether he uses it or not.\r\n\r\n\r\n\"For the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing \"predictive modelling\".\"\r\n\r\nExactly. And the point is, there is no guarantee that this latter case is not being used. We have no way of assuming that your more innocent first hypothetical is indeed the case.\r\n\r\nHere, let me give you a hypothetical to try to clarify my point.\r\n\r\nLet's say I ask you and 100 other people to play the game \"I'm thinking of a number between 1 and 100, try to guess (predict) it. Whoever comes closest wins $50,000. And I'll give you 3 months to study it (me).\" You're a smart guy, you're going to do whatever you can to increase your odds of winning. So you do your homework on me (age, address, zip code, birthday, birthday of kids, etc.). You put a lot of work into it, as you should, there's $50,000 on the line. Now, instead of just saying, at the end, I want all of you to guess and the closest wins, I say, you can make 2 guesses a day for 3 months. AND, there will be a guy on the side who is pointing thumbs up (higher) or thumbs down (lower) after you make each guess. But remember, you can only make 2 guesses per day (and each guess is assumed to be a true \"prediction\"). A few of the people get smart after they have each made a few guesses and team up, share information about their guesses, and work together from that point forward. And then we notice that there are some (especially that group that pooled their knowledge) who come much closer and closer to the true number as they make more and more guesses (and the guy on the side gives the thumbs up or thumbs down after each guess). Now, according to the participants, all of that is irrelevant - all that matters if their final guess. But some are saying, wait a minute, it seems fishy that there's this guy on the side giving feedback after each guess AND their guesses are getting better and better. Nooooo, you say, people aren't gonna actually use thumb guy information. Each guess is totally independent and in no way are they using the feedback from thumb guy on the side. I mean, that's just not right, and not in the spirit of the competition for them to be looking at thumb guy. To that I would say, seriously? There's $50,000 on the line and they have this source of information, and their guesses are getting better, yet they're not looking over at thumb guy after each guess? Seriously? So then, at the end, this group gets the number right on the nose. Dang, they're good. They just know how to do their homework and make really good educated guesses. What? That guy on the side, how could that have possibly played into it? To that I would again reiterate - seriously?", "votes": null }, { "id": "138316", "postDate": "10/08/2016 01:23:12", "content": "@Little Boat
\n\nYou must have missed my post about the Red Hat competition. I have taken a good look at how kaggle splits data. In that competition, the training set represented 82% of the data (not 67%), public LB was 5% of the data (but more considering how much information it's giving), and private LB was 13% (but less considering that there is correlation between the 30% and 70% split in the "test" set). So in that one, you were training on about 90% of the data. That's approaching the limit of true predictive modeling.
", "rawMarkdown": "Little Boat\r\n\r\nYou must have missed my post about the Red Hat competition. I have taken a good look at how kaggle splits data. In that competition, the training set represented 82% of the data (not 67%), public LB was 5% of the data (but more considering how much information it's giving), and private LB was 13% (but less considering that there is correlation between the 30% and 70% split in the \"test\" set). So in that one, you were training on about 90% of the data. That's approaching the limit of true predictive modeling.", "votes": null }, { "id": "138318", "postDate": "10/08/2016 01:34:11", "content": "@aldente
\n\n"The main purpose of public and private LB split is, "to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set." (https://www.kaggle.com/wiki/Leaderboard)"
\n\nThis is exactly what you're not getting. When you are training on 90% of the data, there will not be a big drop between public and private LB. So the contest becomes one of EXACTLY overfitting to the public leaderboard. And when people attempt to apply these overfitting methods to true predictive modeling problems - where you are training on much less than 90% of the data - the methods do not translate very well to making good predictions because the thing that benefitted you most on Kaggle (overfitting) now hurts you the most. This is what we see over and over in the real world. And yet people keep trudging on as though that's not a real issue. Doesn't make sense to me.
\n\n"In real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction.
\n\nKaggle uses "hidden private test set & leaderboard" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction."
\n\nRight, but by giving feedback on the public LB about next year's numbers, you are getting information about the thing that you are supposedly "predicting." This, by definition, disqualifies it as being a "prediction." I'm not sure why people are having so much trouble seeing this. Seems pretty straightforward to me. I guess people just don't want to admit that what they're doing is not really what they thought they were doing.
\n\n"So, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case)."
\n\nRight, and 10% of the total amount of data in the case of Red Hat. "Predicting" only 10% of the data is approaching the limit of being able to really call it "prediction."
\n\n"Public test set (30%) and its LB table is an helper."
\n\n90% of the data is simply a "helper?" That's a whole lot of "help."
", "rawMarkdown": "aldente\r\n\r\n\"The main purpose of public and private LB split is, \"to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set.\" (https://www.kaggle.com/wiki/Leaderboard)\"\r\n\r\nThis is exactly what you're not getting. When you are training on 90% of the data, there will not be a big drop between public and private LB. So the contest becomes one of EXACTLY overfitting to the public leaderboard. And when people attempt to apply these overfitting methods to true predictive modeling problems - where you are training on much less than 90% of the data - the methods do not translate very well to making good predictions because the thing that benefitted you most on Kaggle (overfitting) now hurts you the most. This is what we see over and over in the real world. And yet people keep trudging on as though that's not a real issue. Doesn't make sense to me.\r\n\r\n\r\n\"In real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction.\r\n\r\nKaggle uses \"hidden private test set & leaderboard\" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction.\"\r\n\r\nRight, but by giving feedback on the public LB about next year's numbers, you are getting information about the thing that you are supposedly \"predicting.\" This, by definition, disqualifies it as being a \"prediction.\" I'm not sure why people are having so much trouble seeing this. Seems pretty straightforward to me. I guess people just don't want to admit that what they're doing is not really what they thought they were doing.\r\n\r\n\r\n\"So, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case).\"\r\n\r\nRight, and 10% of the total amount of data in the case of Red Hat. \"Predicting\" only 10% of the data is approaching the limit of being able to really call it \"prediction.\"\r\n\r\n\r\n\"Public test set (30%) and its LB table is an helper.\"\r\n\r\n90% of the data is simply a \"helper?\" That's a whole lot of \"help.\"", "votes": null }, { "id": "138319", "postDate": "10/08/2016 01:46:19", "content": "@GWillie Your analogy is not particularly applicable. Extending your analogy, it is more like there are people who might be similar to you. You know their answers to a survey and the numbers they pick at the end. That's your training set. Then you have to build a model to guess the numbers a second group of other people pick knowing their answers to a survey. That is the public test set. This helps guide you to ensure your model is generalizable but that is not what you are predicting in the end. You get feedback which helps you to know if your generalization error is low. So in a sense, it helps train your model a bit but has no impact on your score because the true goal is to guess numbers for a final group of people. You get no feedback at all on that group and that is what you are predicting. Of course, you can use all of that info in the survey to help and find whatever correlations might exist to model the system, but this is true of all predictive modeling. You always start with your initial data (the survey) and make inferences using whatever clever methods you can find. But the point is, your model must generalize well to a set that you have no feedback on except you can train on a similar set and check if you are generalizing well on the public LB. if you overfit, you WILL NOT generalize well in your PREDICTIONS.
", "rawMarkdown": "GWillie Your analogy is not particularly applicable. Extending your analogy, it is more like there are people who might be similar to you. You know their answers to a survey and the numbers they pick at the end. That's your training set. Then you have to build a model to guess the numbers a second group of other people pick knowing their answers to a survey. That is the public test set. This helps guide you to ensure your model is generalizable but that is not what you are predicting in the end. You get feedback which helps you to know if your generalization error is low. So in a sense, it helps train your model a bit but has no impact on your score because the true goal is to guess numbers for a final group of people. You get no feedback at all on that group and that is what you are predicting. Of course, you can use all of that info in the survey to help and find whatever correlations might exist to model the system, but this is true of all predictive modeling. You always start with your initial data (the survey) and make inferences using whatever clever methods you can find. But the point is, your model must generalize well to a set that you have no feedback on except you can train on a similar set and check if you are generalizing well on the public LB. if you overfit, you WILL NOT generalize well in your PREDICTIONS.", "votes": null }, { "id": "138322", "postDate": "10/08/2016 02:34:13", "content": "import gc;\ngc.collect();
", "rawMarkdown": "import gc;\r\ngc.collect();", "votes": null }, { "id": "138324", "postDate": "10/08/2016 02:37:11", "content": "Is it possible to overfit the model with features not involving the outcome ??? Overfit can happen if we dont stop the learning when evaluation metric starts degrading. But, is there possibility of overfit purely bucause of a feature that is not at all derived from the actual outcome !!!
", "rawMarkdown": "Is it possible to overfit the model with features not involving the outcome ??? Overfit can happen if we dont stop the learning when evaluation metric starts degrading. But, is there possibility of overfit purely bucause of a feature that is not at all derived from the actual outcome !!!", "votes": null }, { "id": "138325", "postDate": "10/08/2016 02:47:31", "content": "I totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)
\n\nBut still, I haven't seen the magiccal rabbit in this thread =^.^!=
", "rawMarkdown": "I totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)\r\n\r\nBut still, I haven't seen the magiccal rabbit in this thread =^.^!=", "votes": null }, { "id": "138328", "postDate": "10/08/2016 03:03:02", "content": "[quote=Banana Kitty;138325]
\n\nI totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)
\n\nBut still, I haven't seen the magiccal rabbit in this thread :D
\n\n[/quote]\nThe rabbit seems to hiding well deep in its hole... Its so strenuous to catch it, the one catching it wants to eat it all by themselves. Well, we caught the rabbit finally it seems, though lot of effort still to turn it into delicious meal.
\n\nAs many already mentioned, can't say for sure if its leak or not. But, I am in the opinion of whatever pattern we found being a leak caused by carelessness in data preparation (I may be wrong). Jumped Local CV score from 0.27 to 0.34 with a single feature.. I am sure about not overfitting the model, as the pattern is clearly visible from the data table, and would know it in a short while after submission.
\n\nThanks to 'raddar' for the quick 3 points. The first point did worked well. There was also a hint about it on the other thread; someone did mentioned about this particular data property although in a completely different context. I believe the guys who already cooked the rabbit could give a lot more info about it.
", "rawMarkdown": "[quote=Banana Kitty;138325]\r\n\r\nI totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)\r\n\r\nBut still, I haven't seen the magiccal rabbit in this thread :D\r\n\r\n[/quote]\r\nThe rabbit seems to hiding well deep in its hole... Its so strenuous to catch it, the one catching it wants to eat it all by themselves. Well, we caught the rabbit finally it seems, though lot of effort still to turn it into delicious meal.\r\n\r\nAs many already mentioned, can't say for sure if its leak or not. But, I am in the opinion of whatever pattern we found being a leak caused by carelessness in data preparation (I may be wrong). Jumped Local CV score from 0.27 to 0.34 with a single feature.. I am sure about not overfitting the model, as the pattern is clearly visible from the data table, and would know it in a short while after submission.\r\n\r\nThanks to 'raddar' for the quick 3 points. The first point did worked well. There was also a hint about it on the other thread; someone did mentioned about this particular data property although in a completely different context. I believe the guys who already cooked the rabbit could give a lot more info about it.", "votes": null }, { "id": "138329", "postDate": "10/08/2016 03:07:11", "content": "Smart people know things.
\n\nSmart people know when other people know things.
\n\nSmart people know about being right or wrong based on quality and quantity of the feedback received.
\n\nSmart people know when to quit.
", "rawMarkdown": "Smart people know things.\r\n\r\nSmart people know when other people know things.\r\n\r\nSmart people know about being right or wrong based on quality and quantity of the feedback received.\r\n\r\nSmart people know when to quit.", "votes": null }, { "id": "138330", "postDate": "10/08/2016 03:12:51", "content": "I'm pretty sure that applies to both threads of this conversation.
", "rawMarkdown": "I'm pretty sure that applies to both threads of this conversation.", "votes": null }, { "id": "138333", "postDate": "10/08/2016 04:01:18", "content": "[quote=GWillie;138254]
\n\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher.
\n\n[/quote]
\n\n@GWillie, I'm going to assume you're not a troll and that you really don't know. The answer is cross validation. Again, my remarks are in the context of a normal competition, not a broken one like Redhat. I almost said "usual competition" but had to stop myself. :(
", "rawMarkdown": "[quote=GWillie;138254]\r\n\r\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher.\r\n\r\n[/quote]\r\n\r\n@GWillie, I'm going to assume you're not a troll and that you really don't know. The answer is cross validation. Again, my remarks are in the context of a normal competition, not a broken one like Redhat. I almost said \"usual competition\" but had to stop myself. :(", "votes": null }, { "id": "138340", "postDate": "10/08/2016 04:57:11", "content": "@Tony S
\n\nMy analogy is applicable. You made the statement that it is not, but then failed to back that statement up (that's becoming a pattern here with those who respond to me).
\n\nAnd, you are also failing to see my point. In fact, you just provided support for my point without knowing it. My point is, and has been, that the public "test" set is not really a test set. It can be considered a hold-out set to be used in a cross-validation study, but it is not a test set in the sense that it is the ultimate set that we are predicting on. You even said, and I quote, "That is the public test set ... So in a sense, it helps train your model." Exactly, so we are in agreement, that public "test" set is actually a training set. Was that really so difficult?
", "rawMarkdown": "Tony S\r\n\r\nMy analogy is applicable. You made the statement that it is not, but then failed to back that statement up (that's becoming a pattern here with those who respond to me).\r\n\r\nAnd, you are also failing to see my point. In fact, you just provided support for my point without knowing it. My point is, and has been, that the public \"test\" set is not really a test set. It can be considered a hold-out set to be used in a cross-validation study, but it is not a test set in the sense that it is the ultimate set that we are predicting on. You even said, and I quote, \"That is the public test set ... So in a sense, it helps train your model.\" Exactly, so we are in agreement, that public \"test\" set is actually a training set. Was that really so difficult?", "votes": null }, { "id": "138341", "postDate": "10/08/2016 05:11:37", "content": "@Willie Liao
\n\nCross validation is a procedure where you hold out various portions of your data (not the data that you will ultimately be trying to predict) to test how well your model might do at making that final prediction. But the data being used in cross validation is data that you know the answer to. Therefore, it is not truly a test set, but is merely a "test" set for the purpose of cross validation. It is, therefore, being used to train your model for its ultimate task of predicting outcomes that are completely unknown to you. It is, therefore, by definition, a training set.
\n\nSo your argument is basically that the 30% of the "test" set used for the public leaderboard can be viewed as a "test" set because it can be viewed as a "test" set for cross validation. But in that cross validation, you could also hold out portions of the training set to be used as "test" sets. So, by your logic then, all of the training set would actually be a test set, and there would not really be a training set, which of course we know is not true. Thus, there is a contradiction to your argument.
\n\nThink of it this way - to reduce variability in a cross-validation study, multiple rounds of cross-validation are performed using different partitions (including data from the training set), and the validation results are averaged over the rounds. So these things you're calling "test" sets in a cross-validation study can be and actually are derived from the training set. How can they be true test sets if they come from the training set?
", "rawMarkdown": "Willie Liao\r\n\r\nCross validation is a procedure where you hold out various portions of your data (not the data that you will ultimately be trying to predict) to test how well your model might do at making that final prediction. But the data being used in cross validation is data that you know the answer to. Therefore, it is not truly a test set, but is merely a \"test\" set for the purpose of cross validation. It is, therefore, being used to train your model for its ultimate task of predicting outcomes that are completely unknown to you. It is, therefore, by definition, a training set.\r\n\r\nSo your argument is basically that the 30% of the \"test\" set used for the public leaderboard can be viewed as a \"test\" set because it can be viewed as a \"test\" set for cross validation. But in that cross validation, you could also hold out portions of the training set to be used as \"test\" sets. So, by your logic then, all of the training set would actually be a test set, and there would not really be a training set, which of course we know is not true. Thus, there is a contradiction to your argument.\r\n\r\nThink of it this way - to reduce variability in a cross-validation study, multiple rounds of cross-validation are performed using different partitions (including data from the training set), and the validation results are averaged over the rounds. So these things you're calling \"test\" sets in a cross-validation study can be and actually are derived from the training set. How can they be true test sets if they come from the training set?", "votes": null }, { "id": "138343", "postDate": "10/08/2016 05:16:08", "content": "@Tilii
\n\nCouldn't agree more ...
\n\nSmart people know about being right or wrong based on quality and quantity of the feedback received.
\n\nIn terms of the feedback received, I see a lot of quantity, but am still waiting for something of quality. Providing feedback that is irrelevant to the argument is not providing quality feedback. Smart people know that.
", "rawMarkdown": "Tilii\r\n\r\nCouldn't agree more ...\r\n\r\nSmart people know about being right or wrong based on quality and quantity of the feedback received.\r\n\r\nIn terms of the feedback received, I see a lot of quantity, but am still waiting for something of quality. Providing feedback that is irrelevant to the argument is not providing quality feedback. Smart people know that.", "votes": null }, { "id": "138346", "postDate": "10/08/2016 05:59:19", "content": "@gWillie I look forward to also seeing you crush the competition! It sounds like you have much to teach us and its probably going to be a cakewalk for someone of your abilities amongst mere mortals. Good luck!
", "rawMarkdown": "gWillie I look forward to also seeing you crush the competition! It sounds like you have much to teach us and its probably going to be a cakewalk for someone of your abilities amongst mere mortals. Good luck!", "votes": null }, { "id": "138348", "postDate": "10/08/2016 06:10:10", "content": "After 7 pages the cat and rabbit are still being hidden
", "rawMarkdown": "After 7 pages the cat and rabbit are still being hidden", "votes": null }, { "id": "138350", "postDate": "10/08/2016 06:30:06", "content": "@Tony S
\n\nThanks for the vote of confidence there big fella! If you were really doing predictive modeling here, I'd probably do pretty well. I have less interest in doing large-scale data fitting and data mining. But I have to say, given the money involved and the level of "expertise" here (basing that on the "brilliant" responses I've received), I should probably look into it.
", "rawMarkdown": "Tony S\r\n\r\nThanks for the vote of confidence there big fella! If you were really doing predictive modeling here, I'd probably do pretty well. I have less interest in doing large-scale data fitting and data mining. But I have to say, given the money involved and the level of \"expertise\" here (basing that on the \"brilliant\" responses I've received), I should probably look into it.", "votes": null }, { "id": "138352", "postDate": "10/08/2016 06:32:37", "content": "More like, after hundreds of competitions, the cat and rabbit are still being hidden. But whatever we do, let's not try to find them.
", "rawMarkdown": "More like, after hundreds of competitions, the cat and rabbit are still being hidden. But whatever we do, let's not try to find them.", "votes": null }, { "id": "138353", "postDate": "10/08/2016 06:45:59", "content": "Does anybody know how to unsubscribe from a single thread in the forum? I am interested in most others, but the discussion on purity of predictive modelling is really getting annoying (because it's spamming my mailbox). 7 pages? Come on... Go have a flame war via PM or sth.
", "rawMarkdown": "Does anybody know how to unsubscribe from a single thread in the forum? I am interested in most others, but the discussion on purity of predictive modelling is really getting annoying (because it's spamming my mailbox). 7 pages? Come on... Go have a flame war via PM or sth.", "votes": null }, { "id": "138354", "postDate": "10/08/2016 06:53:02", "content": "@KonradBanachewicz Clicking here should do the trick?
\n\n![\"enter](\"https://www.dropbox.com/s/s0qjyp8cw520u8d/unsubscribe.JPG?raw=1\")
FACEPALM: i was only checking the top of the page, didn't occur to me there can be buttons at the bottom for the same functionality. Thanks a lot @gingerman :-)
", "rawMarkdown": "FACEPALM: i was only checking the top of the page, didn't occur to me there can be buttons at the bottom for the same functionality. Thanks a lot @gingerman :-)", "votes": null }, { "id": "138380", "postDate": "10/08/2016 12:50:42", "content": "Just one simple question:
\n\nDoes this magic feature, is in form of some probability ?
", "rawMarkdown": "Just one simple question: \r\n\r\nDoes this magic feature, is in form of some probability ?", "votes": null }, { "id": "138394", "postDate": "10/08/2016 15:04:46", "content": "Keep searching for that magical feature, I made new thread concerning predictive modeling here
", "rawMarkdown": "Keep searching for that magical feature, I made new thread concerning predictive modeling [here][1]\r\n\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/24200/is-this-true-predictive-modeling", "votes": null }, { "id": "138424", "postDate": "10/08/2016 20:18:28", "content": "follow the white rabbit .. if you want to enter the 0.4+ club.
", "rawMarkdown": "[follow the white rabbit][1] .. if you want to enter the 0.4+ club.\r\n\r\n\r\n [1]: https://www.kaggle.com/mmueller/bosch-production-line-performance/road-2-0-4/log", "votes": null }, { "id": "138426", "postDate": "10/08/2016 20:39:35", "content": "We have read this thread carefully since the first people hit the 0.4 threshold and irrespective of what people were saying or even swearing that this is not a leak, our mind was fixated in the fact that it was most likely a leak.
\n\nWe spent quite some time doing more traditional analysis/machine learning examination on this problem before realizing we cannot get there with just traditional approaches but we would have to look the data in a way that does not make sense for the client in order to get a better score and reach the top guys.
\n\nAfter doing so the leak was found.
\n\nThere are 2 type of competitors right now in my opinion:
\n\n- \n
- Those that are searching for leak as all the work they’ll do would\nbe futile unless they find it (in order to just build a useless\nmodel). \n
- Those that have found it and a) Build other features as well as the leak b) And try to exploit the leak even further (making more useless models, but probably getting a higher chance to win) \n
At least #2 implies some more ml hence it makes sense to release it - besides it is debatable whether the top scores that use the leak can add value to the real problem.
\n\nIn the past I had said that if we discover a leak and it is still fairly early in the competition, we will release it…so as to be a competition.
\n\nAs the current #2 team has indicated namely: Another Haunted and Leaked One, this dataset actually suffers from multiple leaks and is indeed very haunted…
\n\nHowever, most of the leakage (err …“data property”) so far can be extracted from the combination of sorting based on date time and then creating features based on the original sorting of the data (or just the Id).
\n\nFaron who spotted the leak shares it in this kernel.
\n\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.
\n\nI have to stress that in my opinion many of the comments in this thread have been misleading. It’s better not to say anything if it is to mislead...
", "rawMarkdown": "We have read this thread carefully since the first people hit the 0.4 threshold and irrespective of what people were saying or even swearing that this is not a leak, our mind was fixated in the fact that it was most likely a leak. \r\n\r\nWe spent quite some time doing more traditional analysis/machine learning examination on this problem before realizing we cannot get there with just traditional approaches but **we would have to look the data in a way that does not make sense for the client** in order to get a better score and reach the top guys. \r\n\r\nAfter doing so the leak was found. \r\n\r\nThere are 2 type of competitors right now in my opinion:\r\n\r\n 1. Those that are searching for leak as all the work they’ll do would\r\n be futile unless they find it (in order to just build a useless\r\n model).\r\n 2. Those that have found it and a) Build other features as well as the leak b) And try to exploit the leak even further (making more useless models, but probably getting a higher chance to win)\r\n\r\nAt least #2 implies some more ml hence it makes sense to release it - besides it is debatable whether the top scores that use the leak can add value to the real problem. \r\n \r\nIn the past I had said that [if we discover a leak and it is still fairly early in the competition, we will release it][1]…so as to be a competition.\r\n\r\nAs the current #2 team has indicated namely: *Another Haunted and Leaked One*, this dataset actually suffers from multiple leaks and is indeed very haunted…\r\n\r\nHowever, most of the leakage (err …“data property”) so far can be extracted from the combination of sorting based on date time and then creating features based on the original sorting of the data (or just the Id). \r\n\r\n[Faron who spotted the leak shares it in this kernel][2].\r\n\r\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.\r\n\r\nI have to stress that in my opinion many of the comments in this thread have been misleading. It’s better not to say anything if it is to mislead...\r\n\r\n\r\n [1]: https://www.kaggle.com/c/talkingdata-mobile-user-demographics/forums/t/23445/5th-place-solution-kazanova-faron-and-some-thoughts\r\n [2]: https://www.kaggle.com/mmueller/bosch-production-line-performance/road-2-0-4/log", "votes": null }, { "id": "138428", "postDate": "10/08/2016 20:41:52", "content": "Yep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
", "rawMarkdown": "Yep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.\r\n\r\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!", "votes": null }, { "id": "138429", "postDate": "10/08/2016 20:44:37", "content": "[quote=PrakashSubedi;138428]\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition
\n\n[/quote]
\n\nAs we all...
\n\nTo be honest , you can probably get somehow there with subtracting the times too, not just the ids.
", "rawMarkdown": "[quote=PrakashSubedi;138428]\r\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition\r\n\r\n[/quote]\r\n\r\nAs we all... \r\n\r\nTo be honest , you can probably get somehow there with subtracting the times too, not just the ids.", "votes": null }, { "id": "138431", "postDate": "10/08/2016 20:53:18", "content": "In the first page of this thread, 'amaia' mentioned about failure occuring in pairs. No bigger hint could be to find this leak - RABBIT was there already !!!
", "rawMarkdown": "In the first page of this thread, 'amaia' mentioned about failure occuring in pairs. No bigger hint could be to find this leak - RABBIT was there already !!!", "votes": null }, { "id": "138432", "postDate": "10/08/2016 20:56:45", "content": "thanks Faron for releasing the leak. i probably will never find it myself. I wish the magic feature was something else though.
", "rawMarkdown": "thanks Faron for releasing the leak. i probably will never find it myself. I wish the magic feature was something else though.", "votes": null }, { "id": "138434", "postDate": "10/08/2016 21:12:26", "content": "Adding these features to our model has not helped much, which leads me to believe that we modelled it using different features. Tomorrow I'll regenerate all our features with randomised IDs and row order, to see if our score holds.
", "rawMarkdown": "Adding these features to our model has not helped much, which leads me to believe that we modelled it using different features. Tomorrow I'll regenerate all our features with randomised IDs and row order, to see if our score holds.", "votes": null }, { "id": "138435", "postDate": "10/08/2016 21:15:12", "content": "[quote=PrakashSubedi;138428]
\n\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
\n\n[/quote]
\n\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)
\n\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.
", "rawMarkdown": "[quote=PrakashSubedi;138428]\r\n\r\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.\r\n\r\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!\r\n\r\n[/quote]\r\n\r\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)\r\n\r\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.", "votes": null }, { "id": "138438", "postDate": "10/08/2016 21:33:08", "content": "I should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.
\n\nkicking self
\n\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.
", "rawMarkdown": "I should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.\r\n\r\n*kicking self*\r\n\r\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.", "votes": null }, { "id": "138439", "postDate": "10/08/2016 21:35:53", "content": "[quote=anokas;138435]
\n\n[quote=PrakashSubedi;138428]
\n\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
\n\n[/quote]
\n\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)
\n\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.
\n\n[/quote]\nUnderstandable !!! There could be other ways of exploiting this leak without looking at ID, and you were smart enough to spot that so early in this competition. But, now since we all know that features based on ID giving a heavy boost to the score, safe to say that - "yet another discouraging LEAKED Competition" !!!
", "rawMarkdown": "[quote=anokas;138435]\r\n\r\n[quote=PrakashSubedi;138428]\r\n\r\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.\r\n\r\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!\r\n\r\n[/quote]\r\n\r\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)\r\n\r\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.\r\n\r\n[/quote]\r\nUnderstandable !!! There could be other ways of exploiting this leak without looking at ID, and you were smart enough to spot that so early in this competition. But, now since we all know that features based on ID giving a heavy boost to the score, safe to say that - \"yet another discouraging LEAKED Competition\" !!!", "votes": null }, { "id": "138441", "postDate": "10/08/2016 21:37:27", "content": "[quote=Bob, Lord of Cats;138438]
\n\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.
\n\n[/quote]
\n\nThank you Bob, but it was Faron/Mathias who spotted that .
", "rawMarkdown": "[quote=Bob, Lord of Cats;138438]\r\n\r\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.\r\n\r\n[/quote]\r\n\r\nThank you Bob, but it was Faron/Mathias who spotted that .", "votes": null }, { "id": "138442", "postDate": "10/08/2016 21:39:08", "content": "anokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain.
", "rawMarkdown": "anokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain.", "votes": null }, { "id": "138444", "postDate": "10/08/2016 21:46:39", "content": "I take anokas at his word. The problem would have been solved by a rules clarification. If you are angry make sure it is in the right place. Thanks to Faron - all aboard the ghost train l
", "rawMarkdown": "I take anokas at his word. The problem would have been solved by a rules clarification. If you are angry make sure it is in the right place. Thanks to Faron - all aboard the ghost train l", "votes": null }, { "id": "138445", "postDate": "10/08/2016 21:49:59", "content": "[quote=Bob, Lord of Cats;138438]
\n\nI should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.
\n\n[/quote]
\n\nDo not feel too bad about it, 'cause this can also happen with a feature made of random values (in cases where xgb has trouble to make proper splits on the real features).
\n\nA way to reveal the Id information here, is to calculate the AUC of binned id likelihoods, which yield ~0.55 .. clearly too high to be by chance.
", "rawMarkdown": "[quote=Bob, Lord of Cats;138438]\r\n\r\nI should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.\r\n\r\n[/quote]\r\n\r\nDo not feel too bad about it, 'cause this can also happen with a feature made of random values (in cases where xgb has trouble to make proper splits on the *real* features).\r\n\r\nA way to reveal the Id information here, is to calculate the AUC of binned id likelihoods, which yield ~0.55 .. clearly too high to be by chance.", "votes": null }, { "id": "138446", "postDate": "10/08/2016 21:50:38", "content": "[quote=Sameh Faidi;138442]
\n\nanokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain.
\n\n[/quote]
\n\nHope it meant to be a humor. It is possible to capture this leaked information unknowingly by some other means. In which case, just merely adding ID based features on top may not give a ~0.1 boost. But, if anokas gets another ~0.1 improvement, I am gonna go Mad .
", "rawMarkdown": "[quote=Sameh Faidi;138442]\r\n\r\nanokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain. \r\n\r\n[/quote]\r\n\r\nHope it meant to be a humor. It is possible to capture this leaked information unknowingly by some other means. In which case, just merely adding ID based features on top may not give a ~0.1 boost. But, if anokas gets another ~0.1 improvement, I am gonna go Mad .", "votes": null }, { "id": "138448", "postDate": "10/08/2016 21:58:05", "content": "Im quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation) and now far0n has his version. all i can say, all 3 ways are surprisingly very different.
\n\nSo there are at least 3 ways how people see this data property/leak, and people downvoting him should feel silly or put some thought about themselves why they haven't seen something, when 3 different guys found 3 different solutions to it.
\n\nNonetheless, Good luck every1. Seems the race just have started.
", "rawMarkdown": "Im quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation) and now far0n has his version. all i can say, all 3 ways are surprisingly very different. \r\n\r\nSo there are at least 3 ways how people see this data property/leak, and people downvoting him should feel silly or put some thought about themselves why they haven't seen something, when 3 different guys found 3 different solutions to it.\r\n\r\nNonetheless, Good luck every1. Seems the race just have started.", "votes": null }, { "id": "138450", "postDate": "10/08/2016 22:01:13", "content": "[quote=raddar;138448]
\n\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation)
\n\n[/quote]
\n\nNow You are provoking me!
\n\nI had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to "there is no leak we swear" .
", "rawMarkdown": "[quote=raddar;138448]\r\n\r\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation) \r\n\r\n[/quote]\r\n\r\nNow You are provoking me!\r\n\r\n I had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to **\"there is no leak we swear\" .**", "votes": null }, { "id": "138451", "postDate": "10/08/2016 22:02:12", "content": "@Scirpus @Prakash Thanks for not getting out the pitchforks just yet. Although don't hold your breath for a 0.6 in the next few hours.
\n\nPersonally, it feels really shitty to have several thousand lines of feature engineering, and then having an equivalent 20 line script pop up. But such is the way of Kaggle I guess.
\n\nNot going to sleep happy today.
", "rawMarkdown": "Scirpus @Prakash Thanks for not getting out the pitchforks just yet. Although don't hold your breath for a 0.6 in the next few hours.\r\n\r\nPersonally, it feels really shitty to have several thousand lines of feature engineering, and then having an equivalent 20 line script pop up. But such is the way of Kaggle I guess.\r\n\r\nNot going to sleep happy today.", "votes": null }, { "id": "138452", "postDate": "10/08/2016 22:08:34", "content": "[quote=Μαριος Μιχαηλιδης KazAnova;138450]
\n\n[quote=raddar;138448]
\n\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation)
\n\n[/quote]
\n\nNow You are provoking me!
\n\nI had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to "there is no leak we swear" .
\n\n[/quote]
\n\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.
\n\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.
", "rawMarkdown": "[quote=Μαριος Μιχαηλιδης KazAnova;138450]\r\n\r\n[quote=raddar;138448]\r\n\r\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation) \r\n\r\n[/quote]\r\n\r\nNow You are provoking me!\r\n\r\n I had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to **\"there is no leak we swear\" .** \r\n\r\n\r\n\r\n\r\n[/quote]\r\n\r\n\r\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.\r\n\r\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.", "votes": null }, { "id": "138453", "postDate": "10/08/2016 22:11:22", "content": "[quote=raddar;138452]
\n\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.
\n\n[/quote]
\n\nOk, lets save it for the end-of-the-competition thread.
", "rawMarkdown": "[quote=raddar;138452]\r\n\r\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.\r\n\r\n[/quote]\r\n\r\nOk, lets save it for the end-of-the-competition thread.", "votes": null }, { "id": "138455", "postDate": "10/08/2016 22:18:18", "content": "So, what's next? The last 2 competitions I participated have some leaks, and people spent time in finding them. Reading this forum it seems that there were other competitions, where participants found leaks in the data. My question is: should we stop trying to build up a good model, or learn something from the raw data and just spend time in finding any leak in the construction of the train and test data? \nI like participating to these competitions since I love learning new things from other kagglers, study new solutions etc... \nI started this competition with the idea of using it to learn Spark and Spark ML. Spent time in writing a Scala pipeline ( I usually work in python) to link spark and xgboost for these competition to be able to work on a single pc with full data set, but my score (without any engineering of the features) was always below 0.25...\nNow, I will try to add these leak features, thanks to Faron and KazAnova, to the data as the others and check my score...But I'm wondering why do not use new rules for the competitions, in order to let the participants concentrated only on the models, the techniques, the data analysis and not on leaks hunting?
", "rawMarkdown": "So, what's next? The last 2 competitions I participated have some leaks, and people spent time in finding them. Reading this forum it seems that there were other competitions, where participants found leaks in the data. My question is: should we stop trying to build up a good model, or learn something from the raw data and just spend time in finding any leak in the construction of the train and test data? \r\nI like participating to these competitions since I love learning new things from other kagglers, study new solutions etc... \r\nI started this competition with the idea of using it to learn Spark and Spark ML. Spent time in writing a Scala pipeline ( I usually work in python) to link spark and xgboost for these competition to be able to work on a single pc with full data set, but my score (without any engineering of the features) was always below 0.25...\r\nNow, I will try to add these leak features, thanks to Faron and KazAnova, to the data as the others and check my score...But I'm wondering why do not use new rules for the competitions, in order to let the participants concentrated only on the models, the techniques, the data analysis and not on leaks hunting?", "votes": null }, { "id": "138456", "postDate": "10/08/2016 22:18:57", "content": "quick tl;dr
\n\nClearly in a tl;dr: there are four things that should hold true for Bosch:
\n\n- \n
- The date features are intentionally here \n
- The data was recorded in a very special way (there is a structure, look at the ID alone) \n
- The date features are (not all) leak and are useful for Bosch (if they intend to train/run models in near-real-time or in the past), as long as they don't overshoot the future using the past as structural changes may happen... very quickly \n
- If the problem for Bosch is to analyze past data, they are going to be very pleased (in a good way if they didn't change how the data was recorded) \n
Longer post
\n\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]
\n\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.
\n\n[/quote]
\n\nAssuming what I quoted from KazAnova applies right for Bosch... (it is right for most production environments, but I'm not working at Bosch so I can't know myself) - I'm not talking about the ID itself but more about the date features.
\n\nP.S: I wanted to make this post earlier to explain the reason why the date feature was included, but did not do it as "some stuff were visibly going out of control". I had also some discussions with my team members about why the date features was included and why it was normal it was there (and explaining why it was there).
\n\n\n\n
Gold standard stuff (the great ends bad)
\n\nI think from Bosch this "leak" is not intentional, more a cause of what they thought was good (if I assume they follow the gold standard for production quality management...).
\n\nRationale: if you follow the gold standards in production quality assessment, you must do the following:
\n\n- \n
- Pick production samples to assess for quality during a specific timeframe \n
- Assess the quality of these production samples \n
- Assume that if the defect quality quota is not reached, it generalizes to all samples (otherwise, you either try again with new samples - if it is still reached, then you must work on quality to improve your production "rank" in the official production quality tables if you do comparisons with them) \n
- And then you take a break (to minimize costs), you do it again, then a break, then again... you do that over and over... (thinking naively it generalizes - if you had a bad production batch then you know you will not pass the quality tests, big emphasis: ____and you probably know why____ (if you don't, then ___you should try to know___ at least...)...) \n
You can also do blind testing: two production samples are picked instead of one, with the same timeframe. This is what happens there (or what is intended to be replicated - in fact, it may be just a single production sample split in two as seeing the ID alone seems to depict).
\n\nNot so gold...
\n\nHence, what happens is you pick off samples from a timeframe if you want to generalize to a whole population (whether the date is before or after). This is the production "gold standard" (not so "gold" as we can see here) to assess the quality of your production (in a global way). However, that gold standard.. was from centuries ago (if not a millennium already!) where you had no real powerful machine learning available along with all the data. This should be also the reason we were provided the time data (some people wondered why we were given the data).
\n\nStructural vs timeframes
\n\nNow that ML and data are existing to be used, it's possible to catch how the structural production works (but then - you know only how the production works and how it fails at specific points, not how it will generalize). In the past, no one would have ever challenged a quality manager by doing that (because they never had the data for).
\n\nAnother usage than predicting the future in exact real-time
\n\nThere is also a small trend in the date features (alone), the higher the lag the worse the quality becomes. This is more a production quality issue, and I think by the time the product reaches the end, higher odds it should have been removed at least once by a worker for extra inspection during the flow... Such feature cannot hold generalization if the target is to have a fully automated production (therefore, it is up to what Bosch is going to do with the models/scripts, and what is the problem they are trying to solve).
\n\nHowever, it could be a good one for detecting bad stations before they are tagged "bad" when inspected manually (there are some cycles there you can find on the dataset too): if there is a structural change (specifically: dates), then something must have changed in at least one station (or on the pathing). The closer to real-time they train the models and run them, the faster they can detect defective stations (some stations are dishing out tons of defects at specific times in this dataset, clearly normal from a production environment).
\n\nBack to leakage...
\n\nTherefore, if assuming what I quoted is true for Bosch, some "leaks" are useful (and may become deprecated). But as I do not know all the "leaks" in this dataset, it is fairly possible some are entirely useless for a real case. Overall, it depends on what Bosch expects to do with the model (and they probably will need a "production specialist" of their own production before even trying to put such model in their production - to clear up all real leakage and keep the "good leakage").
\n\nThis quote should be added on what is leakage and what to do with it, as it sums up what to do to assess a leak in a model in a real environment...:
\n\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]
\n\nit is debatable whether the top scores that use the leak can add value to the real problem.
\n\n[/quote]
\n\n... because it always depends on what leak is used and what problem is targeted (so far we don't know the real problem Bosch is trying to solve [not "lowering costs", that's the objective] - or I'm blind). After all, it is all about what "leak" for us in this competition. For the "problem" part, Bosch should know better than us what they are trying to do.
", "rawMarkdown": "# quick tl;dr\r\n\r\nClearly in a tl;dr: there are four things that should hold true for Bosch:\r\n\r\n* The date features are intentionally here\r\n* The data was recorded in a very special way (there is a structure, look at the ID alone)\r\n* The date features are (not all) leak and are useful for Bosch (if they intend to train/run models in near-real-time or in the past), as long as they don't overshoot the future using the past as structural changes may happen... very quickly\r\n* If the problem for Bosch is to analyze past data, they are going to be very pleased (in a good way if they didn't change how the data was recorded)\r\n\r\n# Longer post\r\n\r\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]\r\n\r\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.\r\n\r\n[/quote]\r\n\r\nAssuming what I quoted from KazAnova applies right for Bosch... (it is right for most production environments, but I'm not working at Bosch so I can't know myself) - **I'm not talking about the ID itself but more about the date features.**\r\n\r\nP.S: I wanted to make this post earlier to explain the reason why the date feature was included, but did not do it as \"some stuff were visibly going out of control\". I had also some discussions with my team members about why the date features was included and why it was normal it was there (and explaining why it was there).\r\n\r\n\r\n----------\r\n\r\n\r\n## Gold standard stuff (the great ends bad)\r\n\r\nI think from Bosch this \"leak\" is not intentional, more a cause of what they thought was good (if I assume they follow the gold standard for production quality management...).\r\n\r\nRationale: if you follow the gold standards in production quality assessment, you must do the following:\r\n\r\n* Pick production samples to assess for quality during a **specific timeframe**\r\n* Assess the quality of these production samples\r\n* Assume that if the defect quality quota is not reached, it **generalizes** to all samples (otherwise, you either try again with new samples - if it is still reached, then you must work on quality to improve your production \"rank\" in the official production quality tables if you do comparisons with them)\r\n* And then you take a break (to minimize costs), you do it again, then a break, then again... you do that over and over... (**thinking naively it generalizes** - if you had a bad production batch then you know you will not pass the quality tests, **big emphasis: ____and you probably know why____ (if you don't, then ___you should try to know___ at least...)**...)\r\n\r\nYou can also do blind testing: two production samples are picked instead of one, with the same timeframe. This is what happens there (or what is intended to be replicated - in fact, it may be just a single production sample split in two as seeing the ID alone seems to depict).\r\n\r\n## Not so gold...\r\n\r\nHence, what happens is you pick off **samples from a timeframe if you want to generalize to a whole population** (whether the date is before or after). This is the production \"gold standard\" (not so \"gold\" as we can see here) to assess the quality of your production (in a global way). However, that gold standard.. was from centuries ago (if not a millennium already!) where you had no real powerful machine learning available along with all the data. This should be also the reason we were provided the time data (some people wondered why we were given the data).\r\n\r\n## Structural vs timeframes\r\n\r\nNow that ML and data are existing to be used, it's possible to catch how the __structural__ production works (but then - you know only how the production works and how it fails at specific points, not how it will generalize). In the past, no one would have ever challenged a quality manager by doing that (because they never had the data for).\r\n\r\n## Another usage than predicting the future in exact real-time\r\n\r\nThere is also a small trend in the date features (alone), the higher the lag the worse the quality becomes. This is more a production quality issue, and I think by the time the product reaches the end, higher odds it should have been removed at least once by a worker for extra inspection during the flow... Such feature cannot hold generalization if the target is to have a fully automated production (therefore, it is up to **what Bosch is going to do with the models/scripts, and what is the problem they are trying to solve**).\r\n\r\nHowever, it could be a good one for detecting bad stations before they are tagged \"bad\" when inspected manually (there are some cycles there you can find on the dataset too): if there is a **structural change** (specifically: dates), then something must have **changed in at least one station** (or on the pathing). The closer to real-time they train the models and run them, the faster they can detect **defective stations (some stations are dishing out tons of defects at specific times in this dataset, clearly normal from a production environment)**.\r\n\r\n## Back to leakage...\r\n\r\nTherefore, if assuming what I quoted is true for Bosch, some \"leaks\" are **useful** (and may become deprecated). But as I do not know all the \"leaks\" in this dataset, it is fairly possible some are entirely useless for a real case. Overall, it depends on what Bosch expects to do with the model (and they probably will need a \"production specialist\" of their own production **before even trying to put such model in their production** - to clear up all real leakage and keep the \"good leakage\").\r\n\r\nThis quote should be added on what is leakage and what to do with it, as it sums up what to do to assess a leak in a model in a real environment...:\r\n\r\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]\r\n\r\nit is debatable whether the top scores that use the leak can **add value to the real problem**.\r\n\r\n[/quote]\r\n\r\n... because it always depends on what leak is used and what problem is targeted (so far we don't know **the real problem Bosch is trying to solve** [not \"lowering costs\", that's the objective] - or I'm blind). After all, it is all about what \"leak\" for us in this competition. For the \"problem\" part, Bosch should know better than us what they are trying to do.", "votes": null }, { "id": "138457", "postDate": "10/08/2016 22:20:46", "content": "[quote=raddar;138452]
\n\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.
\n\n[/quote]
\n\nThe data page says "Each part has a unique Id" and if this is true, it makes me really wonder how the magic feature(s) come about. That is what made me wonder whether it was really a leak or not.
", "rawMarkdown": "[quote=raddar;138452]\r\n\r\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.\r\n\r\n[/quote]\r\n\r\nThe data page says \"Each part has a unique Id\" and if this is true, it makes me really wonder how the magic feature(s) come about. That is what made me wonder whether it was really a leak or not.", "votes": null }, { "id": "138458", "postDate": "10/08/2016 22:25:05", "content": "Please let me know, how to express my gratitude to those sharing their hardwork for benefit of others?
", "rawMarkdown": "Please let me know, how to express my gratitude to those sharing their hardwork for benefit of others?", "votes": null }, { "id": "138459", "postDate": "10/08/2016 22:32:03", "content": "I see it this way: As soon as anything else than the provided features yield predictive power, I label it leakage. If the id-structure is intended information they should call it Order Feature (or because we like to make things anonym Beqre Srngher) and give the rows real random ids.
", "rawMarkdown": "I see it this way: As soon as anything else than the provided features yield predictive power, I label it leakage. If the *id*-structure is intended information they should call it *Order Feature* (or because we like to make things anonym *Beqre Srngher*) and give the rows *real random* ids.", "votes": null }, { "id": "138460", "postDate": "10/08/2016 22:42:44", "content": "Imagine a situation, that Bosch truly believe that id's are random... does this insight alone bring value to their production/data recording pipeline? Put some thought about it:) Yes, we might call it a leak as we are used to call everything like that as unwanted data property. But does this help solve their problem to detect defects? if such hiccups in a system are recorded and it is a reason of higher defect rate, this might have golden value itself. And still ML has to help figuring out which hiccups are normal and not, hence the relevance of ML is important.
\n\nSo other question is, how much do You believe that above scenario is likely; then its just a personal opinion if it is a leak or a data property.
", "rawMarkdown": "Imagine a situation, that Bosch truly believe that id's are random... does this insight alone bring value to their production/data recording pipeline? Put some thought about it:) Yes, we might call it a leak as we are used to call everything like that as unwanted data property. But does this help solve their problem to detect defects? if such hiccups in a system are recorded and it is a reason of higher defect rate, this might have golden value itself. And still ML has to help figuring out which hiccups are normal and not, hence the relevance of ML is important.\r\n\r\nSo other question is, how much do You believe that above scenario is likely; then its just a personal opinion if it is a leak or a data property.", "votes": null }, { "id": "138462", "postDate": "10/08/2016 23:00:20", "content": "[quote=raddar;138460]
\n\nBut does this help solve their problem to detect defects?
\n\n[/quote]
\n\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. "Response" is leaked in adjacent rows because when they found a failed part, they do something in response immediately.
\n\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure.
\n\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P
", "rawMarkdown": "[quote=raddar;138460]\r\n\r\nBut does this help solve their problem to detect defects? \r\n\r\n[/quote]\r\n\r\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. \"Response\" is leaked in adjacent rows because when they found a failed part, they do something in response immediately. \r\n\r\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure. \r\n\r\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P", "votes": null }, { "id": "138463", "postDate": "10/08/2016 23:06:41", "content": "To be honest, I actually stumbled upon to very very similar thing in one of projects (which was so silly and so many analysis have been done on that), so nothing can shock me when such kind of things appear :)
\n\nanyway, rabbit is out & have fun.
", "rawMarkdown": "To be honest, I actually stumbled upon to very very similar thing in one of projects (which was so silly and so many analysis have been done on that), so nothing can shock me when such kind of things appear :)\r\n\r\nanyway, rabbit is out & have fun.", "votes": null }, { "id": "138464", "postDate": "10/08/2016 23:09:13", "content": "But I also want to say we should not criticize raddar and his teammates. They do have a misleading team name but so do we :P (out computer actually has 1000 GB memory not 1TB)
\n\nActually they have given a lot of hints and valuable advice along the way. When I read them, I immediately realize they are talking about leak in my definition and "special data property" in their words. So i want to thank them for sharing.
", "rawMarkdown": "But I also want to say we should not criticize raddar and his teammates. They do have a misleading team name but so do we :P (out computer actually has 1000 GB memory not 1TB) \r\n\r\nActually they have given a lot of hints and valuable advice along the way. When I read them, I immediately realize they are talking about leak in my definition and \"special data property\" in their words. So i want to thank them for sharing.", "votes": null }, { "id": "138465", "postDate": "10/08/2016 23:17:52", "content": "I put some thought into it this afternoon and I'm extremely hesitant call this a leak. At facilities I've worked in the past the ID is not random, but generated based on timing and style of the product. It's almost certain there is some order information encoded in the ID but more importantly the class of product should be present as well. If there wasn't important information here why include Id in the dataset at all?
\n\nThe first two features in Faron's kernel were not significant features when I added them to my model, however the second two, based on the differences in consecutive ID's when sorted by start time, became my top features.
\n\nFor lack of a better hypothesis, I think these two features might be capturing transitions between product styles.
", "rawMarkdown": "I put some thought into it this afternoon and I'm extremely hesitant call this a leak. At facilities I've worked in the past the ID is not random, but generated based on timing and style of the product. It's almost certain there is some order information encoded in the ID but more importantly the class of product should be present as well. If there wasn't important information here why include Id in the dataset at all?\r\n\r\nThe first two features in Faron's kernel were not significant features when I added them to my model, however the second two, based on the differences in consecutive ID's when sorted by start time, became my top features. \r\n\r\nFor lack of a better hypothesis, I think these two features might be capturing transitions between product styles.", "votes": null }, { "id": "138467", "postDate": "10/08/2016 23:45:32", "content": "[quote=rcarson;138462]
\n\n[quote=raddar;138460]
\n\nBut does this help solve their problem to detect defects?
\n\n[/quote]
\n\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. "Response" is leaked in adjacent rows because when they found a failed part, they do something in response immediately.
\n\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure.
\n\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P
\n\n[/quote]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
", "rawMarkdown": "[quote=rcarson;138462]\r\n\r\n[quote=raddar;138460]\r\n\r\nBut does this help solve their problem to detect defects? \r\n\r\n[/quote]\r\n\r\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. \"Response\" is leaked in adjacent rows because when they found a failed part, they do something in response immediately. \r\n\r\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure. \r\n\r\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P\r\n\r\n\r\n[/quote]\r\n\r\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.", "votes": null }, { "id": "138468", "postDate": "10/09/2016 00:14:40", "content": "[quote=ClockWorkWorld;138467]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
\n\n[/quote]
\n\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.
", "rawMarkdown": "[quote=ClockWorkWorld;138467]\r\n\r\n\r\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.\r\n\r\n\r\n[/quote]\r\n\r\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.", "votes": null }, { "id": "138469", "postDate": "10/09/2016 00:50:26", "content": "[quote=rcarson;138468]
\n\n[quote=ClockWorkWorld;138467]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
\n\n[/quote]
\n\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.
\n\n[/quote]
\n\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.
", "rawMarkdown": "[quote=rcarson;138468]\r\n\r\n[quote=ClockWorkWorld;138467]\r\n\r\n\r\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.\r\n\r\n\r\n[/quote]\r\n\r\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.\r\n\r\n\r\n[/quote]\r\n\r\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.", "votes": null }, { "id": "138472", "postDate": "10/09/2016 01:44:58", "content": "My model isn't finished yet (and will probably need a few passes anyhow) but I think .4 is achievable by just using features 3 and 4 from Faron's script (mindate<->id, I think 1+2 are linenum<->id) Since there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.
\n\nEDIT: Finished my very first run and it got .408 public LB. Aside from the mindate<->id features it got in the .27 range.
\n\nEDIT2: Local testing indicates that processing the feature with train+test not combined is much less effective - probably around .32. I'm not going to burn a submission figuring out what it gets on the LB tho. ;)
\n\nIf Id can be treated as a legitimate feature, and the feature still works by doing the diffs with train+test separated, it isn't a leak. However if there's a lift from row #<->id that can't be replicated elsewise, it's a leak - likewise if it requires a combined train+test set it might run afoul of the 'time machine rule(sic)'
\n\nThanks for sharing Faron!
", "rawMarkdown": "My model isn't finished yet (and will probably need a few passes anyhow) but I think .4 is achievable by just using features 3 and 4 from Faron's script (mindate<->id, I think 1+2 are linenum<->id) Since there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.\r\n\r\nEDIT: Finished my very first run and it got .408 public LB. Aside from the mindate<->id features it got in the .27 range.\r\n\r\nEDIT2: Local testing indicates that processing the feature with train+test not combined is much less effective - probably around .32. I'm not going to burn a submission figuring out what it gets on the LB tho. ;)\r\n\r\nIf Id can be treated as a legitimate feature, and the feature still works by doing the diffs with train+test separated, it isn't a leak. However if there's a lift from row #<->id that can't be replicated elsewise, it's a leak - likewise if it requires a combined train+test set it might run afoul of the 'time machine rule(sic)'\r\n\r\nThanks for sharing Faron!", "votes": null }, { "id": "138474", "postDate": "10/09/2016 02:08:28", "content": "[quote=happycube;138472]
\n\nSince there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.
\n\n[/quote]
\n\nOne reason why it works is because there are often rows x, y with abs(id(x) - id(y) ) = 1 for a given "StartTime". Besides, they are often duplicates of each other.
", "rawMarkdown": "[quote=happycube;138472]\r\n\r\nSince there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.\r\n\r\n[/quote]\r\n\r\nOne reason why it works is because there are often rows *x, y with abs(id(x) - id(y) ) = 1* for a given \"StartTime\". Besides, they are often duplicates of each other.", "votes": null }, { "id": "138485", "postDate": "10/09/2016 04:50:15", "content": "[quote=ClockWorkWorld;138469]
\n\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.
\n\n[/quote]
\n\nI agree with this assessment, but I don't think this represents a leak. When a part is 'created' Bosch does not know if it will be good or bad. We don't know for sure until it reaches the end of the production line. It's course through the line (station sequence and timestamp) has a strong influence on the final quality.
\n\nBesides, we don't know about their manufacturing process. Bosch may start batches of similar parts at the same time. This is a common manufacturing procedure which can reduce the amount of production time lost when changing a station setup from one part to another. Bosch may start 20 items of part A which has a high failure rate, and then 20 items of part B which has a normal failure rate.
", "rawMarkdown": "[quote=ClockWorkWorld;138469]\r\n\r\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.\r\n\r\n[/quote]\r\n\r\nI agree with this assessment, but I don't think this represents a leak. When a part is 'created' Bosch does not know if it will be good or bad. We don't know for sure until it reaches the end of the production line. It's course through the line (station sequence and timestamp) has a strong influence on the final quality. \r\n\r\nBesides, we don't know about their manufacturing process. Bosch may start batches of similar parts at the same time. This is a common manufacturing procedure which can reduce the amount of production time lost when changing a station setup from one part to another. Bosch may start 20 items of part A which has a high failure rate, and then 20 items of part B which has a normal failure rate.", "votes": null }, { "id": "138487", "postDate": "10/09/2016 05:34:33", "content": "It looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
", "rawMarkdown": "It looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)\r\n\r\nI really hope that is intentional, somehow...", "votes": null }, { "id": "138489", "postDate": "10/09/2016 05:51:23", "content": "Would the leak have been avoided if the data was split by time instead of Id?
", "rawMarkdown": "Would the leak have been avoided if the data was split by time instead of Id?", "votes": null }, { "id": "138490", "postDate": "10/09/2016 05:52:30", "content": "I don't know, but randomizing Id would have made it harder to find...
", "rawMarkdown": "I don't know, but randomizing Id would have made it harder to find...", "votes": null }, { "id": "138494", "postDate": "10/09/2016 07:25:54", "content": "IDs are not features. You can argue all you want. To me, any feature using IDs is a leak.
\n\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.
\n\nI hope the forums don't become a place where people pull heists, and get upvotes all around for it.
", "rawMarkdown": "IDs are not features. You can argue all you want. To me, any feature using IDs is a leak.\r\n\r\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.\r\n\r\nI hope the forums don't become a place where people pull heists, *and* get upvotes all around for it.", "votes": null }, { "id": "138503", "postDate": "10/09/2016 09:07:01", "content": "[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nThe best part is, that thanks to Chippy it is known for weeks actually: Sequential duplicates in test files? This thread really diserved more attention. :))
", "rawMarkdown": "[quote=happycube;138487]\r\n\r\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)\r\n\r\nI really hope that is intentional, somehow...\r\n\r\n\r\n[/quote]\r\n\r\nThe best part is, that thanks to [Chippy][1] it is known for weeks actually: [Sequential duplicates in test files?][2] This thread really diserved more attention. :))\r\n\r\n\r\n [1]: https://www.kaggle.com/nigelcarpenter\r\n [2]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083", "votes": null }, { "id": "138507", "postDate": "10/09/2016 09:20:01", "content": "[quote=Sameh Faidi;138489]
\n\nWould the leak have been avoided if the data was split by time instead of Id?
\n\n[/quote]
\n\nI think that if Bosch was intent on predicting the future then they would have split the data based on time. It's more likely that, like others have said, that they want to inspect say a random 10% of products in a production batch by hand, and then by building a model on that they are able to detect possible faults in the other 90% across the same time period, while at the moment they might have to inspect all 100% of the products that come off the production line.
\n\nI thik about it like this - If Bosch really wanted a model that could predict the future, then literally just running XGBoost on the provided data would make a useless model, since the date data has been provided in absolute time format, and that would mean that future data would fall into the wrong branches when running the model on a set with different date info.
\n\nI would have expected them to split the data based on time if they wanted to predict the future, or given us features that are time invariant such as 'time taken at station 2'
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
", "rawMarkdown": "[quote=Sameh Faidi;138489]\r\n\r\nWould the leak have been avoided if the data was split by time instead of Id? \r\n\r\n[/quote]\r\n\r\nI think that if Bosch was intent on predicting the future then they would have split the data based on time. It's more likely that, like others have said, that they want to inspect say a random 10% of products in a production batch by hand, and then by building a model on that they are able to detect possible faults in the other 90% across the same time period, while at the moment they might have to inspect all 100% of the products that come off the production line.\r\n\r\nI thik about it like this - If Bosch really wanted a model that could predict the future, then literally just running XGBoost on the provided data would make a useless model, since the date data has been provided in absolute time format, and that would mean that future data would fall into the wrong branches when running the model on a set with different date info.\r\n\r\nI would have expected them to split the data based on time if they wanted to predict the future, or given us features that are time invariant such as 'time taken at station 2'\r\n\r\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.", "votes": null }, { "id": "138508", "postDate": "10/09/2016 09:22:24", "content": "[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
", "rawMarkdown": "[quote=anokas;138507]\r\n\r\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.\r\n\r\n[/quote]\r\n\r\nI 2nd that. :))", "votes": null }, { "id": "138509", "postDate": "10/09/2016 09:26:41", "content": "I thought about the nature of why this feature works for about a week now and my explanation is the following. If they see a bad part, they will also check the next product, which somehow determines the order in the ids. \nThis also means, that a model based on this feature only helps to identify the broken products, which bosch was already able to find themselfes. If they wont find it themselfes this feature doesnt work. Based on that i dont think this feature has any worth for them and is just a leak.
", "rawMarkdown": "I thought about the nature of why this feature works for about a week now and my explanation is the following. If they see a bad part, they will also check the next product, which somehow determines the order in the ids. \r\nThis also means, that a model based on this feature only helps to identify the broken products, which bosch was already able to find themselfes. If they wont find it themselfes this feature doesnt work. Based on that i dont think this feature has any worth for them and is just a leak.", "votes": null }, { "id": "138511", "postDate": "10/09/2016 09:30:14", "content": "[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
", "rawMarkdown": "[quote=Faron;138508]\r\n\r\n[quote=anokas;138507]\r\n\r\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.\r\n\r\n[/quote]\r\n\r\nI 2nd that. :))\r\n\r\n[/quote]\r\n\r\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.", "votes": null }, { "id": "138512", "postDate": "10/09/2016 09:43:01", "content": "[quote=anokas;138511]
\n\n[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
\n\n[/quote]\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.
", "rawMarkdown": "[quote=anokas;138511]\r\n\r\n[quote=Faron;138508]\r\n\r\n[quote=anokas;138507]\r\n\r\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.\r\n\r\n[/quote]\r\n\r\nI 2nd that. :))\r\n\r\n[/quote]\r\n\r\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.\r\n\r\n[/quote]\r\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.", "votes": null }, { "id": "138517", "postDate": "10/09/2016 09:49:40", "content": "[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nI was suspicious of the data when I found too many sequential duplicates a month ago (link to post). They shouldn't have existed if the Id's were random and the row order had been shuffled.
\n\nThis annoyed me because it is often a sign of bad data preparation and precursor to leak identification and exploitation. Roll forward a month and the "leak" is out and it utilises Id's and roworders.
\n\nSo proof of another bricked competition?
\n\nWell I'm not so sure. The nature of this problem is a little different. The posts of those with domain expertise (JeffH and others) give insight into how use of sequence information may be valid and useful to Bosch.
\n\nHowever I am sure that in the current competition format leaks or the the suspicion of leaks are damaging. Damaging for
\n\n- \n
- us as competitors (I want to establish and demonstrate my predictive analytics, not plumbing skills) \n
- competition hosts (they don't get the predictive model they hoped for) \n
- Kaggle (leaks alienate competitors and hosts and don't deliver predictive analytics) \n
That's a real shame because the principle upon which Kaggle was created is sound and beneficial to society at large.
\n\nNow I believe that leaks will always exist as long as fallible humans are involved in data setup. So as much as we don't like leaks they are not going away. We need to find a way of turning leaks into something that benefits the Kaggle format.
\n\nSo something needs to be done
\n\nIf you agree +1 this post to demonstrate to Kaggle admins our collective strength of feeling.
\n\nAlso contribute your ideas and thoughts on how we can turn leak identification into something beneficial to our community.
\n\nI'll create a new thread to collate ideas and post a link to it here
\n\n- \n
- edited to add link to ideas to improve thread \n
[quote=Malte Nalenz;138512]
\n\n[quote=anokas;138511]
\n\n[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
\n\n[/quote]\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.
\n\n[/quote]
\n\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.
", "rawMarkdown": "[quote=Malte Nalenz;138512]\r\n\r\n[quote=anokas;138511]\r\n\r\n[quote=Faron;138508]\r\n\r\n[quote=anokas;138507]\r\n\r\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.\r\n\r\n[/quote]\r\n\r\nI 2nd that. :))\r\n\r\n[/quote]\r\n\r\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.\r\n\r\n[/quote]\r\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.\r\n\r\n[/quote]\r\n\r\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.", "votes": null }, { "id": "138521", "postDate": "10/09/2016 09:55:46", "content": "[quote=Faron;138503]
\n\n[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nThe best part is, that thanks to Chippy it is known for weeks actually: Sequential duplicates in test files? This thread really diserved more attention. :))
\n\n[/quote]
\n\nI've detected all of them sometimes ago, rows with exactly the same values between train-test (except Id) = 1772 rows having Response '1', and 75504 rows having Response '0'.
\n\nSame rows with Response both '1' and '0' are omitted.
\n\nI've made submission based on only those 1772 '1's, if they're all correct then score should be 0.5+ with 1772 true positives, but in fact I only got around 0.18 public LB score.
\n\nI guess, (1) the 1772 same rows are NOT distributed proportionally between public/private (30/70%), or (2) they have false positives.
\n\nSo, I decided to not use them in my submission based on Santander experience.
\n\nIF it's indeed another leakage, then I'm just too dumb because I don't know how to use it ;)) Any insights? Thank you.
", "rawMarkdown": "[quote=Faron;138503]\r\n\r\n[quote=happycube;138487]\r\n\r\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)\r\n\r\nI really hope that is intentional, somehow...\r\n\r\n\r\n[/quote]\r\n\r\nThe best part is, that thanks to [Chippy][1] it is known for weeks actually: [Sequential duplicates in test files?][2] This thread really diserved more attention. :))\r\n\r\n\r\n [1]: https://www.kaggle.com/nigelcarpenter\r\n [2]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files/135083\r\n\r\n[/quote]\r\n\r\nI've detected all of them sometimes ago, rows with exactly the same values between train-test (except Id) = 1772 rows having Response '1', and 75504 rows having Response '0'. \r\n\r\nSame rows with Response both '1' and '0' are omitted. \r\n\r\nI've made submission based on only those 1772 '1's, if they're all correct then score should be 0.5+ with 1772 true positives, but in fact I only got around 0.18 public LB score. \r\n\r\nI guess, (1) the 1772 same rows are NOT distributed proportionally between public/private (30/70%), or (2) they have false positives. \r\n\r\nSo, I decided to not use them in my submission based on Santander experience. \r\n\r\nIF it's indeed another leakage, then I'm just too dumb because I don't know how to use it ;)) Any insights? Thank you.", "votes": null }, { "id": "138548", "postDate": "10/09/2016 14:02:28", "content": "There are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.
\n\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those "duplicates". Why it occurs? Kaggle and Bosch can tell us.
\n\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...
\n\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training.
", "rawMarkdown": "There are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.\r\n\r\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those \"duplicates\". Why it occurs? Kaggle and Bosch can tell us.\r\n \r\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...\r\n\r\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training.", "votes": null }, { "id": "138549", "postDate": "10/09/2016 14:15:53", "content": "Gilberto great post - I set up a thread on the forums called leak soup - I would love the best of the best to give Kaggle tips to reduce leaks - after all you Grand Masters are so good at finding them!
", "rawMarkdown": "Gilberto great post - I set up a thread on the forums called leak soup - I would love the best of the best to give Kaggle tips to reduce leaks - after all you Grand Masters are so good at finding them!", "votes": null }, { "id": "138551", "postDate": "10/09/2016 14:27:23", "content": "[quote=anokas;138520]
\n\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.
\n\n[/quote]
\n\nyes, i meant the roworder of the observations. thank you for clarifying!
", "rawMarkdown": "[quote=anokas;138520]\r\n\r\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.\r\n\r\n[/quote]\r\n\r\nyes, i meant the roworder of the observations. thank you for clarifying!", "votes": null }, { "id": "138557", "postDate": "10/09/2016 15:08:28", "content": "Gilberto: I counted only the exact matches and it is up to 40% of all 1 responses are in these pairs of identical rows. To my surprise also a good amount had different response (sequential rows with 100 % identical values but one is faulty and the other is fine).
", "rawMarkdown": "Gilberto: I counted only the exact matches and it is up to 40% of all 1 responses are in these pairs of identical rows. To my surprise also a good amount had different response (sequential rows with 100 % identical values but one is faulty and the other is fine).", "votes": null }, { "id": "138574", "postDate": "10/09/2016 16:13:08", "content": "So let me see if I have this straight. I said that it appeared as though very little, if any, true predictive modeling was going on in this competition (and others like it), and that was doing a disservice to both the companies who thought they were getting predictive models and the competitors who thought they were doing predictive modeling (because that what Kaggle bills it as). MANY people jumped down my throat, offering different reasons as to why I was wrong and that I didn't know what I was talking about. They offered reasons like "cross validation" to try to explain what was happening here. Then, in the end, it turns out I was right and they were wrong.
\n\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!
", "rawMarkdown": "So let me see if I have this straight. I said that it appeared as though very little, if any, true predictive modeling was going on in this competition (and others like it), and that was doing a disservice to both the companies who thought they were getting predictive models and the competitors who thought they were doing predictive modeling (because that what Kaggle bills it as). MANY people jumped down my throat, offering different reasons as to why I was wrong and that I didn't know what I was talking about. They offered reasons like \"cross validation\" to try to explain what was happening here. Then, in the end, it turns out I was right and they were wrong.\r\n\r\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!", "votes": null }, { "id": "138577", "postDate": "10/09/2016 16:17:15", "content": "[quote=Gilberto Titericz Junior;138548]
\n\nThere are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.
\n\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those "duplicates". Why it occurs? Kaggle and Bosch can tell us.
\n\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...
\n\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training.
\n\n[/quote]
\n\nI was able to achieve 0.297 before Faron shared his kernel using a sloppy xgboost (without any ensembling, model tuning, or cross validation) with the addition of a very simple set of physically relevant hand generated features. The best modelling will not beat the best feature engineering here.
\n\nThe data here is incredibly feature rich and it's frustrating to see so many experienced participants jumping to the conclusion there must be a leak and that the results will be useless for Bosch. I hate to be that guy but All Models Are Wrong.
\n\nWe need more information on how the Id's are generated to know if they are leaking information from the test set.
\n\nThere are two scenarios I can think of:
\n\n- \n
The Id's have too much information related to the train/test split. In this case, why is Id left in the dataset, wouldn't the automatic row numbering be good enough?
\nThe Id's are machine generated when the product enters the assembly line (perhaps bosch may have subtracted the minimum original value it from the rest so we start at zero). If this is the case then the Id will probably contain information about the type of part (summarizing the equipment path) and the order. The "duplicate" parts you are seeing are actually not duplicates at all but different parts produced under nominally identical conditions because they were processed one after the other. I think the Id feature effectively captures some nonlinear information about transitions to and from certain styles of part.
\n
A final point to discuss is the choice to split the train and test data from within the same time period. Obviously this makes "prediction" a somewhat abstract activity. Consider that absolute prediction may not be the goal. If I was at Bosch I would use these results to upgrade my quality control sampling system to minimize the cost of testing (materials, inventory, customer lead time) as well as to identify areas of process improvement. Ultimately an engineer may not be as concerned with the ability to predict failures part to part and are probably more interested in driving the overall rate of failures down.
\n\nAt the risk of giving up any advantage that come with domain experience, I'll be happy to weigh in on any leak hypothesis.
", "rawMarkdown": "[quote=Gilberto Titericz Junior;138548]\r\n\r\nThere are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.\r\n\r\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those \"duplicates\". Why it occurs? Kaggle and Bosch can tell us.\r\n \r\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...\r\n\r\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training. \r\n\r\n[/quote]\r\n\r\nI was able to achieve 0.297 before Faron shared his kernel using a sloppy xgboost (without any ensembling, model tuning, or cross validation) with the addition of a very simple set of physically relevant hand generated features. The best modelling will not beat the best feature engineering here.\r\n\r\nThe data here is incredibly feature rich and it's frustrating to see so many experienced participants jumping to the conclusion there must be a leak and that the results will be useless for Bosch. I hate to be that guy but [All Models Are Wrong][1].\r\n\r\nWe need more information on how the Id's are generated to know if they are leaking information from the test set. \r\n\r\nThere are two scenarios I can think of:\r\n\r\n 1. The Id's have too much information related to the train/test split. In this case, why is Id left in the dataset, wouldn't the automatic row numbering be good enough?\r\n\r\n\r\n 2. The Id's are machine generated when the product enters the assembly line (perhaps bosch may have subtracted the minimum original value it from the rest so we start at zero). If this is the case then the Id will probably contain information about the type of part (summarizing the equipment path) and the order. The \"duplicate\" parts you are seeing are actually not duplicates at all but different parts produced under nominally identical conditions because they were processed one after the other. I think the Id feature effectively captures some nonlinear information about transitions to and from certain styles of part.\r\n\r\nA final point to discuss is the choice to split the train and test data from within the same time period. Obviously this makes \"prediction\" a somewhat abstract activity. Consider that absolute prediction may not be the goal. If I was at Bosch I would use these results to upgrade my quality control sampling system to minimize the cost of testing (materials, inventory, customer lead time) as well as to identify areas of process improvement. Ultimately an engineer may not be as concerned with the ability to predict failures part to part and are probably more interested in driving the overall rate of failures down. \r\n\r\nAt the risk of giving up any advantage that come with domain experience, I'll be happy to weigh in on any leak hypothesis.\r\n\r\n [1]: https://en.wikipedia.org/wiki/All_models_are_wrong", "votes": null }, { "id": "138581", "postDate": "10/09/2016 16:32:13", "content": "[quote=GWillie;138574]
\n\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!
\n\n[/quote]
\n\n@GWillie, I am not very happy that this comp is leaky and generally 3 out 3 competitions I participated as of late had a leak which is frustrating .
\n\nHowever I have to admit that I have picked up many of my predictive modelling skills in this platform (kaggle) - and they have been very useful (and very tested) in corporate environments as well. So I am keen to disagree on this "I will most likely not be here" . If you joined lately, I can see why you might feel this way, but many innovative solutions and good models have been produced here.
\n\nKaggle is still for me the best place to learn predictive modelling and build up your resume in data science.
\n\nKaggle experience does count on your resume for a data science position- at least where I work for!
", "rawMarkdown": "[quote=GWillie;138574]\r\n\r\n\r\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!\r\n\r\n[/quote]\r\n\r\n@GWillie, I am not very happy that this comp is leaky and generally 3 out 3 competitions I participated as of late had a leak which is frustrating .\r\n\r\nHowever I have to admit that I have picked up many of my predictive modelling skills in this platform (kaggle) - and they have been very useful (and very tested) in corporate environments as well. So I am keen to disagree on this \"I will most likely not be here\" . If you joined lately, I can see why you might feel this way, but many innovative solutions and good models have been produced here.\r\n\r\nKaggle is still for me the best place to learn predictive modelling and build up your resume in data science. \r\n\r\nKaggle experience does count on your resume for a data science position- at least where I work for!", "votes": null }, { "id": "138582", "postDate": "10/09/2016 16:43:10", "content": "But that's just it, exactly my point. People like you think they are picking up predictive modeling skills, but with competitions like this, you are not. You are picking up data analysis, data mining, large-scale computation/memory/storage skills, and thus "data science" skills (particularly "big data"), but not necessarily predictive modeling skills. And therein lies the real danger. The field of predictive modeling is being held back by things like this where everyone thinks they're doing true predictive modeling and thus we focus our efforts here instead of other pursuits where we might actually be doing true predictive modeling. This is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.
\n\nI do agree there is value to companies in what is being done here, but we need to be very careful in stating exactly what that is and what it is not. From what I see, it is certainly not predictive modeling value.
", "rawMarkdown": "But that's just it, exactly my point. People like you think they are picking up predictive modeling skills, but with competitions like this, you are not. You are picking up data analysis, data mining, large-scale computation/memory/storage skills, and thus \"data science\" skills (particularly \"big data\"), but not necessarily predictive modeling skills. And therein lies the real danger. The field of predictive modeling is being held back by things like this where everyone thinks they're doing true predictive modeling and thus we focus our efforts here instead of other pursuits where we might actually be doing true predictive modeling. This is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.\r\n\r\nI do agree there is value to companies in what is being done here, but we need to be very careful in stating exactly what that is and what it is not. From what I see, it is certainly not predictive modeling value.", "votes": null }, { "id": "138586", "postDate": "10/09/2016 17:01:51", "content": "Come on please Don't Do it Again... There is separate thread to discuss the meaning and relevance of Predictive Modelling. Please stick to the discussion here on Magical feature or leak or data property whatever it is called.
", "rawMarkdown": "Come on please Don't Do it Again... There is separate thread to discuss the meaning and relevance of Predictive Modelling. Please stick to the discussion here on Magical feature or leak or data property whatever it is called.", "votes": null }, { "id": "138588", "postDate": "10/09/2016 17:05:18", "content": "[quote=barisumog;138494]
\n\nIDs are not features. You can argue all you want. To me, any feature using IDs is a leak.
\n\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.
\n\nI hope the forums don't become a place where people pull heists, and get upvotes all around for it.
\n\n[/quote]
\n\nVery correct sir. I may not be an expert at statistics or machine learning, but I am an expert in manufacturing having spent several years with the likes of Intel, Texas Instruments and other large companies. The ID is nothing more than the id. At best it is a sequential number/combination of when the part was started. If you cannot measure in your process to predict a bad part, but your part start sequence matters--you don't need machine learning, you need basic process improvement.
\n\nFor example, one of my processes had a time window. Basically from the start of the previous step to the start of my step, we had X number of hours or we knew the part would be bad.
\n\nOr we had burn-ins--the first part through we be bad, so we knew we had to "Season" the machine with a test part.
\n\nBut the point is, we could measure those with an in-line process. Aka the resistance after step 62 was too high, or the thickness after step 71 was too low.
\n\nIn 10 years of manufacturing, ID did not matter. Ever. And I've seen some weird 4-level failures/interactions. When I did my experimental splits, we didn't care about the part number, we cared about what split (experiment portion) was on the part.
\n\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a "Tag" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people.
\n\nSo unless you assign "Steve" to overly aggressive drivers, the tag should not matter.
", "rawMarkdown": "[quote=barisumog;138494]\r\n\r\nIDs are not features. You can argue all you want. To me, any feature using IDs is a leak.\r\n\r\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.\r\n\r\nI hope the forums don't become a place where people pull heists, *and* get upvotes all around for it.\r\n\r\n\r\n[/quote]\r\n\r\nVery correct sir. I may not be an expert at statistics or machine learning, but I am an expert in manufacturing having spent several years with the likes of Intel, Texas Instruments and other large companies. The ID is nothing more than the id. At best it is a sequential number/combination of when the part was started. If you cannot measure in your process to predict a bad part, but your part start sequence matters--you don't need machine learning, you need basic process improvement.\r\n\r\nFor example, one of my processes had a time window. Basically from the start of the previous step to the start of my step, we had X number of hours or we knew the part would be bad. \r\n\r\nOr we had burn-ins--the first part through we be bad, so we knew we had to \"Season\" the machine with a test part.\r\n\r\nBut the point is, we could measure those with an in-line process. Aka the resistance after step 62 was too high, or the thickness after step 71 was too low.\r\n\r\nIn 10 years of manufacturing, ID did not matter. Ever. And I've seen some weird 4-level failures/interactions. When I did my experimental splits, we didn't care about the part number, we cared about what split (experiment portion) was on the part.\r\n\r\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a \"Tag\" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people. \r\n\r\nSo unless you assign \"Steve\" to overly aggressive drivers, the tag should not matter.", "votes": null }, { "id": "138589", "postDate": "10/09/2016 17:08:19", "content": "You still need to understand the data, cross-validate, test against a metric, perform feature engineering and feature selection, build multiple models, do ensembling etc . The problem is the data has information it should not have .
\n\nFrom the competitors' point of view, there is value (IMHO).
\n\nFrom the companies' point of view not much as a solution , but they may still leverage the benefits of knowing what is new in the market, what the top approaches were , which features performed best (possibly outside the leakage) and they can understand the theoretical best you could get from this data set
\n\nI think the competitors lose if kaggle starts receiving the reputation that they don't solve the client's problems efficiently (which is my primary concern with the leaks) and the fact that you need less predictive modelling skills and more data exploration ones .
\n\n[quote=GWillie;138582]
\n\nThis is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.
\n\n[/quote]
\n\nI have not seen lack of progression in predictive modelling (the contrary) - what do you mean? There seems to be huge focus in the field right now. I can see that as PhD student, Professional, kaggler , one_that_uses_google etc!
\n\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?
", "rawMarkdown": "You still need to understand the data, cross-validate, test against a metric, perform feature engineering and feature selection, build multiple models, do ensembling etc . The problem is the data has information it should not have .\r\n \r\nFrom the competitors' point of view, there is value (IMHO). \r\n\r\nFrom the companies' point of view not much as a solution , but they may still leverage the benefits of knowing what is new in the market, what the top approaches were , which features performed best (possibly outside the leakage) and they can understand the theoretical best you could get from this data set\r\n\r\nI think the competitors lose if kaggle starts receiving the reputation that they don't solve the client's problems efficiently (which is my primary concern with the leaks) and the fact that you need less predictive modelling skills and more data exploration ones . \r\n\r\n\r\n[quote=GWillie;138582]\r\n\r\n This is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.\r\n\r\n[/quote]\r\n\r\nI have not seen lack of progression in predictive modelling (the contrary) - what do you mean? There seems to be huge focus in the field right now. I can see that as PhD student, Professional, kaggler , one_that_uses_google etc!\r\n\r\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?", "votes": null }, { "id": "138591", "postDate": "10/09/2016 17:18:31", "content": "[quote=Μαριος Μιχαηλιδης KazAnova;138589]
\n\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?
\n\n[/quote]\n A checklist for avoiding leak?
", "rawMarkdown": "[quote=Μαριος Μιχαηλιδης KazAnova;138589]\r\n\r\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?\r\n\r\n[/quote]\r\n A checklist for avoiding leak?", "votes": null }, { "id": "138592", "postDate": "10/09/2016 17:22:15", "content": "EDIT:Deleted
", "rawMarkdown": "EDIT:Deleted", "votes": null }, { "id": "138702", "postDate": "10/10/2016 13:57:05", "content": "[quote=Bob, Lord of Cats;138588]
\n\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a "Tag" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people.
\n\nSo unless you assign "Steve" to overly aggressive drivers, the tag should not matter.
\n\n[/quote]
\n\nOne problem with this statement is that in the case you describe name is an Independent Identically Distributed Random Variable, whereas ID is neither independent, identically distributed, nor truly random.
\n\nConsider the following hypothetical scheme for internal IDs:
\n\nEach Id is made up of 4 numbers xx,yy, tt, and zz: XXYYTTZZ
\n\n- \n
- XX represents the customer, \n
- YY represents the family of product \n
- TT represents the batch number \n
- ZZ represents the part # within the particular batch. \n
In this situation differencing the ID in adjecent rows can give you significant amounts of information. Just because we weren't clever enough to spot it on our own doesn't mean it's not there.
\n\nFor our dataset, the bottom line is knowing whether the ID feature captures information about how the stratified splitting was performed (leak), or if the ID's are machine generated (potentially not a leak).
", "rawMarkdown": "[quote=Bob, Lord of Cats;138588]\r\n\r\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a \"Tag\" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people. \r\n\r\nSo unless you assign \"Steve\" to overly aggressive drivers, the tag should not matter.\r\n\r\n[/quote]\r\n\r\n\r\nOne problem with this statement is that in the case you describe name is an [Independent Identically Distributed Random Variable][1], whereas ID is neither independent, identically distributed, nor truly random. \r\n\r\nConsider the following hypothetical scheme for internal IDs:\r\n\r\nEach Id is made up of 4 numbers xx,yy, tt, and zz: XXYYTTZZ\r\n\r\n - XX represents the customer, \r\n - YY represents the family of product\r\n - TT represents the batch number \r\n - ZZ represents the part # within the particular batch.\r\n\r\nIn this situation differencing the ID in adjecent rows can give you significant amounts of information. Just because we weren't clever enough to spot it on our own doesn't mean it's not there.\r\n\r\nFor our dataset, the bottom line is knowing whether the ID feature captures information about how the stratified splitting was performed (leak), or if the ID's are machine generated (potentially not a leak). \r\n\r\n\r\n [1]: https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables", "votes": null }, { "id": "138704", "postDate": "10/10/2016 14:22:35", "content": "Except of course in my experience that is NOT how part numbering worked. Parts were sequential from when they started. But even at the first manufacturing step, they were already out of order. By the end of the line it was an extreme jumble. Product ID, and in particular customer details were not stored as part of the part ID. It's like a key in a 3NF database. If the lookup key is part of your model, we'll you have a problem.
\n\nThe additional important point is that the true measurement of performance difference was contained in data elsewhere.
\n\nAnd let's have a reality check. The id's here are integers starting at 1, not some fancy labeling scheme, and with specific time data included. If you could build your data on date alone--that's fine. But because you need date and ID, it's a leak.
", "rawMarkdown": "Except of course in my experience that is NOT how part numbering worked. Parts were sequential from when they started. But even at the first manufacturing step, they were already out of order. By the end of the line it was an extreme jumble. Product ID, and in particular customer details were not stored as part of the part ID. It's like a key in a 3NF database. If the lookup key is part of your model, we'll you have a problem.\r\n\r\nThe additional important point is that the true measurement of performance difference was contained in data elsewhere. \r\n\r\nAnd let's have a reality check. The id's here are integers starting at 1, not some fancy labeling scheme, and with specific time data included. If you could build your data on date alone--that's fine. But because you need date and ID, it's a leak.", "votes": null }, { "id": "138711", "postDate": "10/10/2016 15:10:35", "content": "[quote=Bob, Lord of Cats;138704]
\n\nAnd let's have a reality check. The id's here are integers starting at 1, not some fancy labeling scheme, and with specific time data included. If you could build your data on date alone--that's fine. But because you need date and ID, it's a leak.
\n\n[/quote]
\n\nI suppose here Bosch follows ISO 9003 (which they probably do - or the more recent ISO 9001 which is the old 9001-9003 put together and refined).
\n\nISO 9001 is about quality management systems and is a top-down approach. According to ISO 9001, the most probable position we are in the production workflow is the following, with the sampling exception rule used:
\n\n![\"enter](\"http://img.ctrlv.in/img/16/10/10/57fbab4310668.png\")
When you compare two samples to assess the quality of your production:
\n\n- \n
- They must have the same time frame from the same population \n
- ID may provide information depending on how the data was sampled (hence, it's a feature alone already) \n
And ID can start by whatever number you want, for instance here they decided it should start by 1 when they provided the data sets for us. To perform a validation on a machine learning model for this specific quality management setup, you need to turn this into a blind experiment which follows the same exact conditions, and to pass at least once "the minimal quality production" (the confusion matrix % passes, or any other quantitative metric) from your specifications.
\n\nAs part of ISO 9003 (now deprecated and included as part of ISO 9001), you need to perform calibration of your model to perform a validation. As long as it works in one sample, ISO says it generalizes to all your samples. The only thing you need to, is re-calibration when you change samples, more exactly re-training your model when you change population (= change time frame). If you are not calibrating, you are not conforming to ISO 9001/9003. The "regular basis" in your specifications define the frequency of calibration (part 4.13 in ISO 9003, called the "Exception 4.12" when you must sample to work appropriately).
\n\nThis is how it works when you are performing quality assessment from the high level. At the bottom level (where we are), we create the script which will train a model and validate on another sample, where train+test are from the same population. Assuming "our winner scripts" pass the quality test at Bosch, this script theoretically generalizes to new samples (obviously, you re-train your model when you change the population).
\n\nLeak or no leak, it doesn't matter what people call it because your train and test are from the same population (and that population is defined by a time frame). As the observations seem to have been sampled in a very special way (look at the date batches and you will see what I mean), it is normal for ID to carry information. You will never use such model to predict the future. You predict only the past using the same past information.
\n\nThis refers also to the following, as the way you apply a tag may provide information especially if a sequence proves useful (and correct, it is not a measurement, it is issued from a sequential sampling plan in product quality controlling - how Bosch sampled, remains a mystery as only them know as it did generalize for 2 years - 750 days = t=[0, 1800] if 0.01=6 min ):
\n\n[quote=Bob, Lord of Cats;138704]
\n\nID is not a measurement, it is a tag.
\n\n[/quote]
\n\nIf the "script" is going to be used for Bosch's internal audits, then the same exact rules apply and it will work. But in any case, no one is going to (be able to) ever use such model in a population which is not the same of the training data.
\n\nPost-edit add-on:
\n\n- \n
- ID is the by-product of sampling (directly) \n
- Sampling depends on time (indirectly here) \n
- Time is dependent on the outcome (indirectly here) \n
- If I understand Faron's script, the features depend on both on ID and time, hence the reason it works very well - but there must be other ways to engineer clear features for machine learning models without using the by-product ID \n
[quote=Laurae;138711]\n As the observations seem to have been sampled in a very special way (look at the date batches and you will see what I mean), it is normal for ID to carry information. You will never use such model to predict the future. You predict only the past using the same past information.\n[/quote]
\n\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).
\n\nAnd there is no guarantee of a "ID-Free" feature. It may or may not exist--time will tell. But there is no evidence "demanding" such a feature exist (sorry put on my physics/math hat for your last point)
", "rawMarkdown": "[quote=Laurae;138711]\r\n As the observations seem to have been sampled in a very special way (look at the date batches and you will see what I mean), it is normal for ID to carry information. You will never use such model to predict the future. You predict only the past using the same past information.\r\n[/quote]\r\n\r\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).\r\n\r\nAnd there is no guarantee of a \"ID-Free\" feature. It may or may not exist--time will tell. But there is no evidence \"demanding\" such a feature exist (sorry put on my physics/math hat for your last point)", "votes": null }, { "id": "138738", "postDate": "10/10/2016 17:46:15", "content": "[quote=Bob, Lord of Cats;138730]
\n\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).
\n\n[/quote]
\n\nI believe it is unintentional also. It is 100% sure it would be useless if their intent is to predict right in the production line, unless you train a model in near real-time. There could be value behind this, as you could then use automated feature importance analyses to find which station is causing a batch of defects with the small lag of training a model - incurring a small batch loss / path freezing with a human supervision obviously.
\n\n[quote=Bob, Lord of Cats;138730]
\n\nAnd there is no guarantee of a "ID-Free" feature. It may or may not exist--time will tell. But there is no evidence "demanding" such a feature exist (sorry put on my physics/math hat for your last point)
\n\n[/quote]
\n\nAlso, putting your words in reverse, there is no demand for ID existing, as the information came from the sampling elsewhere (you are right there is no demand for it to exist right from the beginning). It's still a feature (and there are "ID-free" features), but when Bosch sampling methods for assigning ID will change there will be no guarantee anymore for the model to work properly.
\n\nIf I were to generalize the previous paragraph, I would say this: if Bosch receives a model which is too much tailored for this data set, then it will overfit severely and not fit enough the others. I think Bosch is prepared to work on the model script provided (if they did invest money already, they should not stop looking at potential valuation of the models they are going to receive).
\n\n\n\n
Anyway, for all those who still think it is a leak, free to read this:
\n\n![\"enter](\"http://i.imgur.com/0esNjne.png\")
Going for a witch hunt like in TalkingData or here is not how Data Scientists are supposed to behave. A good community should not fight each other, but team up for the greater good and knowledge improvement of everyone.
", "rawMarkdown": "[quote=Bob, Lord of Cats;138730]\r\n\r\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).\r\n\r\n[/quote]\r\n\r\nI believe it is unintentional also. It is 100% sure it would be useless if their intent is to predict right in the production line, unless you train a model in near real-time. There could be value behind this, as you could then use automated feature importance analyses to find which station is causing a batch of defects with the small lag of training a model - incurring a small batch loss / path freezing with a human supervision obviously.\r\n\r\n[quote=Bob, Lord of Cats;138730]\r\n\r\nAnd there is no guarantee of a \"ID-Free\" feature. It may or may not exist--time will tell. But there is no evidence \"demanding\" such a feature exist (sorry put on my physics/math hat for your last point)\r\n\r\n[/quote]\r\n\r\nAlso, putting your words in reverse, there is no demand for ID existing, as the information came from the sampling elsewhere (you are right there is no demand for it to exist right from the beginning). It's still a feature (and there are \"ID-free\" features), but when Bosch sampling methods for assigning ID will change there will be no guarantee anymore for the model to work properly.\r\n\r\nIf I were to generalize the previous paragraph, I would say this: if Bosch receives a model which is too much tailored for this data set, then it will overfit severely and not fit enough the others. I think Bosch is prepared to work on the model script provided (if they did invest money already, they should not stop looking at potential valuation of the models they are going to receive).\r\n\r\n\r\n----------\r\n\r\n\r\nAnyway, for all those who still think it is a leak, free to read [this][1]:\r\n\r\n![enter image description here][2]\r\n\r\nGoing for a witch hunt like in [TalkingData][3] or here is not how Data Scientists are supposed to behave. A good community should not fight each other, but team up for the greater good and knowledge improvement of everyone.\r\n\r\n [1]: https://www.kaggle.com/forums/f/15/kaggle-forum/t/24216/leak-soup/138697#post138697\r\n [2]: http://i.imgur.com/0esNjne.png\r\n [3]: https://www.kaggle.com/c/talkingdata-mobile-user-demographics/forums/t/23286/you-were-only-supposed-to-blow-the-doors-off", "votes": null }, { "id": "138744", "postDate": "10/10/2016 18:17:08", "content": "I really hope William was talking about this competition as if it was useful insight. I really hate how the word leakage has such a negative meaning in a community, that people start to avoid using it when itself might be useful in the end. As I said before, too many people jump to conclusions too fast.
", "rawMarkdown": "I really hope William was talking about this competition as if it was useful insight. I really hate how the word leakage has such a negative meaning in a community, that people start to avoid using it when itself might be useful in the end. As I said before, too many people jump to conclusions too fast.", "votes": null }, { "id": "138750", "postDate": "10/10/2016 19:00:30", "content": "I'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
", "rawMarkdown": "I'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild. \r\n\r\nI'll stop posting--I thought actual domain experience might be of use.", "votes": null }, { "id": "138752", "postDate": "10/10/2016 19:05:01", "content": "[quote=Bob, Lord of Cats;138750]
\n\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
\n\n[/quote]
\n\nThat part was not aimed at you at all. But there are people in this thread who are doing the witch hunting on "leakage" without any argumentation (look at all the downvotes in this thread - excluding the True/False predictive modeling issue).
\n\nIn your case, you add information which is useful to read. There is nothing wrong with it.
", "rawMarkdown": "[quote=Bob, Lord of Cats;138750]\r\n\r\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild. \r\n\r\nI'll stop posting--I thought actual domain experience might be of use.\r\n\r\n[/quote]\r\n\r\nThat part was not aimed at you at all. But there are people in this thread who are doing the witch hunting on \"leakage\" without any argumentation (look at all the downvotes in this thread - excluding the True/False predictive modeling issue).\r\n\r\nIn your case, you add information which is useful to read. There is nothing wrong with it.", "votes": null }, { "id": "138755", "postDate": "10/10/2016 19:49:43", "content": "[quote=Bob, Lord of Cats;138750]
\n\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
\n\n[/quote]
\n\nConsider there are others here with domain experiences "in the wild" different than your own. Language like never and always and blanket statements like "that's not how it works" can be just as damaging to the discussion as witch hunting.
", "rawMarkdown": "[quote=Bob, Lord of Cats;138750]\r\n\r\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild. \r\n\r\nI'll stop posting--I thought actual domain experience might be of use.\r\n\r\n\r\n[/quote]\r\n\r\nConsider there are others here with domain experiences \"in the wild\" different than your own. Language like never and always and blanket statements like \"that's not how it works\" can be just as damaging to the discussion as witch hunting.", "votes": null }, { "id": "138877", "postDate": "10/11/2016 12:24:19", "content": "![\"an](\"http://screenshot.net/rrjxkcn.jpg\")
Thanks Marios I thought my email was broken ;)
", "rawMarkdown": "Thanks Marios I thought my email was broken ;)", "votes": null }, { "id": "141002", "postDate": "10/24/2016 16:38:11", "content": "I think Kaggle should start a new competition on "How to read the relevant/useful comments" from this thread :)
", "rawMarkdown": "I think Kaggle should start a new competition on \"How to read the relevant/useful comments\" from this thread :)", "votes": null }, { "id": "518078", "postDate": "04/16/2019 20:26:48", "content": ".
", "rawMarkdown": ".", "votes": null } ], "comments": [ { "id": 137694, "author_name": "nthanhtam", "author_url": "", "post_date": "10/04/2016 06:03:48", "content": "No, it is not a single feature. I have to generate some features to bridge the gap.
", "votes": null, "replies": [] }, { "id": 137695, "author_name": "samehif", "author_url": "", "post_date": "10/04/2016 06:05:21", "content": "i have been digging for this "magical feature/trick" for days now. from what i see it is not in the categorical features. it must be with the time line.
", "votes": null, "replies": [] }, { "id": 137696, "author_name": "raddar", "author_url": "", "post_date": "10/04/2016 06:17:11", "content": "[quote=Tam T. Nguyen;137694]
\n\nNo, it is not a single feature. I have to generate some features to bridge the gap.
\n\n[/quote]
\n\nI disagree. with single feature i got 0.45xx
", "votes": null, "replies": [] }, { "id": 137697, "author_name": "samehif", "author_url": "", "post_date": "10/04/2016 06:24:04", "content": "@raddar: i guess you mean single engineered feature on top of existing given features ?
", "votes": null, "replies": [] }, { "id": 137698, "author_name": "raddar", "author_url": "", "post_date": "10/04/2016 06:30:41", "content": "Yes.
", "votes": null, "replies": [] }, { "id": 137715, "author_name": "adityakumarsinha", "author_url": "", "post_date": "10/04/2016 11:00:01", "content": "@raddar other than the single feature are you using all the feature or only some selected features. I am stuck with 8GB RAM laptop and confused whether start thinking for new features (engineered) or first looking for method for further dimension reduction. Currently I am working on featues as suggested by basic XGB model.
", "votes": null, "replies": [] }, { "id": 137716, "author_name": "raddar", "author_url": "", "post_date": "10/04/2016 11:15:16", "content": "my suggestion is to do some pre-processing on every file before joining all files together (this requires you to know how to load each file in a RAM-friendly way, many topics on that in a forum):
\n\na) drop duplicate categorical features (in your case, dont use them at all, they are not that significant)
\n\nb) drop duplicate date features within stations - all dates in every station are duplicated (except S24 & S25); For S24 & S25 you can calculate min/max dates (or just apply min/max for every station - will get same result)
\n\nc) use all numeric data - this is where the most important information is anyway
\n\nAfter these steps you will work with less than 1000 feature matrix, which can be handled in 8GB RAM, I think.
", "votes": null, "replies": [] }, { "id": 137717, "author_name": "ascending", "author_url": "", "post_date": "10/04/2016 12:17:37", "content": "@raddar\na) How do you deal with null values in numeric data? \nb) In categorical data, how do you deal with categories? Do you keep them as is or one hot encode? Any tips?
", "votes": null, "replies": [] }, { "id": 137718, "author_name": "laurae2", "author_url": "", "post_date": "10/04/2016 12:21:41", "content": "[quote=raddar;137716]
\n\nAfter these steps you will work with less than 1000 feature matrix, which can be handled in 8GB RAM, I think.
\n\n[/quote]
\n\nConfirming it fits in 8GB RAM. As sparse, it even fits under 3GB - so there's no real reason to not be able to compete with a 8GB machine (the only brick wall is loading the data to do that).
\n\nData fully filtered (removed perfect correlation and duplicates) results as this in R:
\n\n- \n
- numeric dense = 604 features, 5473.1 MB \n
- numeric sparse = 604 features, 1715.5 MB \n
- categorical dense = 227 features, 1034.2 MB \n
- categorical sparse = 597 features (hybrid design matrix), 801.7 MB (= binary as binary, uniques as deleted, levels>2 as one-hot encoded - missing counting as a level obviously) \n
- date dense = 126 features, 1147 MB \n
- date sparse = 126 features, 231.6 MB \n
So:
\n\n- \n
- Total dense = 5473.1 + 1034.2 + 1147 = approx 7654.3 MB (peak during creation should be ~12GB or under that if smart and you got time) \n
- Total sparse = 1715.5 + 801.7 + 231.6 = approx 2748.8 MB (peak during creation should be ~6GB or under that if smart and you got time) \n
If you shave this even more (using appropriate selection of features), you can more than halve it (75%-90% reduction).
\n\nIn R, if you manage to make such matrix, I recommend you to save it as RDS so you can load and work on it back nearly instantly once you run a new R session.
\n\nWith a 4GB laptop, you are left to try getting the sparse matrix or use precooked feature sets from the kernels of your choice of this competition. Under 1GB guaranteed sparse matrix, xgboost model under 4GB and >0.26 largely achievable.
", "votes": null, "replies": [] }, { "id": 137743, "author_name": "gullu39", "author_url": "", "post_date": "10/04/2016 15:37:13", "content": "@Laurae Well explained !!! I ran XGB separately on numeric, categorical and date and selected importance features from each of the set, combined them and ran xgb again. I guess a 8GB RAM should be able to handle it that way.
", "votes": null, "replies": [] }, { "id": 137761, "author_name": "salkaa", "author_url": "", "post_date": "10/04/2016 21:26:53", "content": "@raddar when you say one feature, you mean add one column at the end ? or you add one column for each feature given ?
", "votes": null, "replies": [] }, { "id": 137848, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/05/2016 14:42:06", "content": "So I did find a single feature that took my single-model XGBoost score from about 0.27 to about 0.44. In the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, "I suggest taking a step back, and just looking at the raw data." I know ... it's vague, but it's true.
", "votes": null, "replies": [] }, { "id": 137853, "author_name": "jiweiliu", "author_url": "", "post_date": "10/05/2016 15:48:09", "content": "[quote=BreakfastPirate;137848]\nIn the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, "I suggest taking a step back, and just looking at the raw data." I know ... it's vague, but it's true.\n[/quote]
\n\nAfter I saw it, it is so obvious. I could even imagine why it is there in the first place.
", "votes": null, "replies": [] }, { "id": 137854, "author_name": "barisumog", "author_url": "", "post_date": "10/05/2016 15:53:19", "content": "[quote=BreakfastPirate;137848]
\n\nSo I did find a single feature that took my single-model XGBoost score from about 0.27 to about 0.44. In the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, "I suggest taking a step back, and just looking at the raw data." I know ... it's vague, but it's true.
\n\n[/quote]
\n\nLink for the lazy, an 8 page discussion about the magic feature in Telstra. The mentioned post is on top of the 2nd page.
\n\n@BreakfastPirate, do you think the feature you found makes sense within the context we have available, or is it more like a magic / leak feature that doesn't really make sense?
\n\nI'd fully understand if you wouldn't want to share such info at this stage. I'm just curious about how this competition will evolve in the remaining month.
", "votes": null, "replies": [] }, { "id": 137856, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/05/2016 16:16:55", "content": "[quote=barisumog;137854]
\n\n@BreakfastPirate, do you think the feature you found makes sense within the context we have available, or is it more like a magic / leak feature that doesn't really make sense?
\n\n[/quote]
\n\nI have to say I honestly don't know. There's so much we don't know about the meaning of the fields and how the data was gathered that it is difficult to be sure the feature does or doesn't make sense. It will be interesting to hear discussion on that topic after the contest is over. Even after Telstra was over, there was a lot of debate on whether the feature in that contest was a leak or not, and I think this is a similar situation. Sorry I can't be more helpful.
", "votes": null, "replies": [] }, { "id": 137866, "author_name": "samehif", "author_url": "", "post_date": "10/05/2016 18:47:06", "content": "Ok. So what we know already about the magical feature?
\n\n- \n
- it is one single engineered feature. Or atleast one. \n
- it is not related to the categorical features \n
- It is not certainly clear if the feature is a leak or not as people who found it seems to disagree about this \n
What else?
", "votes": null, "replies": [] }, { "id": 137872, "author_name": "barisumog", "author_url": "", "post_date": "10/05/2016 19:32:50", "content": "[quote=Sameh Faidi;137866]\nIt is not certainly clear if the feature is a leak or not as people who found it seems to disagree about this\n[/quote]
\n\nIn my humble opinion, if there was no leak, there wouldn't be disagreement.
\n\nIf it all boils down to what the definition of leak is, it probably is a leak.
", "votes": null, "replies": [] }, { "id": 137873, "author_name": "samehif", "author_url": "", "post_date": "10/05/2016 19:47:52", "content": "We have seen so many problems with the data in previous competitions. Why can't some of the people who found the magical feature confirm (in private) with the admins or sponsors if this is a valid feature. Now is a good time as we are still in early phase of the competition. Just an opinion!
", "votes": null, "replies": [] }, { "id": 137874, "author_name": "tilii7", "author_url": "", "post_date": "10/05/2016 19:53:39", "content": "\n\n\nIf it all boils down to what the definition of leak is, it probably is a leak.
\n
I think it has to do with a simple distinction - at least in my mind - between leaks and hard-to-find data properties. Any data transformation that was introduced after data collection could be the cause of leaks. That includes all kinds of sorting by date, by time, by stations, giving non-random IDs to data, and whatever else one can think of. If, however, our hard-working Kagglers have discovered an underlying relationship between data points that helps them make better predictions, that would be squarely in the category of good prediction skills.
\n\nI have no problem believing that at this point it is difficult to know which of the two possibilities is a correct one. By that I mean there could be a natural time series that improves prediction, or that ordered series of events was introduced by post-processing the data. Hard to know the truth without knowing what was done during data preparation.
", "votes": null, "replies": [] }, { "id": 137876, "author_name": "barisumog", "author_url": "", "post_date": "10/05/2016 20:18:28", "content": "@Tilii, in the context of Kaggle competitions, I'd say leaks are hard-to-find data properties. I respect the work of those who find them.
\n\nI was just thinking out loud, out of experience - I haven't looked at the data set thoroughly yet.
\n\nIf the magic feature was something like "count of stations" or "mean of numeric", then there would be no disagreement about whether it's a leak or not. So I'm just guessing it's something more exotic.
", "votes": null, "replies": [] }, { "id": 137877, "author_name": "interesting", "author_url": "", "post_date": "10/05/2016 20:19:49", "content": "I think that some of us are easy on the leak claim.
\n\nWe all have to remember that given that results are usable for the competition host - is all that matters (prizes and rankings aside)
\n\nThe high goal of data science is to derive business value of the data at hand, and if that is achieved - its a win-win situation. That is the great value proposition behind Kaggle for competition hosts (at least the way I see it...)
\n\nIn any other case of problematic "game-setting" - both the host of the competition and Kaggle have to learn and improve,in order to serve that initial value proposition mentioned above again in future competitions.
\n\nI think that the constant questions about leaked/non-leaked so called "magical" feature is a waste of time that if spent in more prolific ways could get us all to better results.
", "votes": null, "replies": [] }, { "id": 137881, "author_name": "tilii7", "author_url": "", "post_date": "10/05/2016 20:43:41", "content": "\n\n\n@Tilii, in the context of Kaggle competitions, I'd say leaks are hard-to-find data properties.
\n
To me, the leak is something that was introduced during data manipulation, either by data owners or by Kaggle. In essence, the leak is an unintentional tipping of the final outcome by those who know the outcome. We've had plenty of leaks that satisfy this definition. In fact, we've had too many of them.
\n\nFinding non obvious relationships within data is not the same as finding a leak. In Santander competition it was discovered that customers younger than 24 do things a certain way. To me that is not finding a leak - it is finding a true and non-obvious property within that dataset.
\n\nAnyway, I believe those who claim that at this point they can't say whether this is a leak or a true property of data.
", "votes": null, "replies": [] }, { "id": 137885, "author_name": "tilii7", "author_url": "", "post_date": "10/05/2016 20:53:56", "content": "\n\n\nI think that the constant questions about leaked/non-leaked so called "magical" feature is a waste of time that if spent in more prolific ways could get us all to better results.
\n
Couldn't agree more.
\n\nI wish Kaggle was displaying the last submitted score rather than best score. That way those who wish to hide their true score could just resubmit one of their lower-score submissions - assuming they are willing to waste one attempt in order to conceal the score. We'd have way less traffic about magic features and leaks.
", "votes": null, "replies": [] }, { "id": 137886, "author_name": "barisumog", "author_url": "", "post_date": "10/05/2016 20:56:17", "content": "[quote=Tilii;137881]\nTo me, the leak is something that was introduced during data manipulation, either by data owners or by Kaggle. In essence, the leak is an unintentional tipping of the final outcome by those who know the outcome. We've had plenty of leaks that satisfy this definition. In fact, we've had too many of them.
\n\nFinding non obvious relationships within data is not the same as finding a leak. In Santander competition it was discovered that customers younger than 24 do things a certain way. To me that is not finding a leak - it is finding a true and non-obvious property within that dataset.
\n\nAnyway, I believe those who claim that at this point they can't say whether this is a leak or a true property of data.\n[/quote]
\n\nI agree.
\n\n"In theory, theory and practice are the same. In practice, they are not."
\n\nEdited to add the quote I replied to
", "votes": null, "replies": [] }, { "id": 137890, "author_name": "raddar", "author_url": "", "post_date": "10/05/2016 21:16:32", "content": "[quote=Tilii;137885]
\n\nI wish Kaggle was displaying the last submitted score rather than best score. That way those who wish to hide their true score could just resubmit one of their lower-score submissions - assuming they are willing to waste one attempt in order to conceal the score. We'd have way less traffic about magic features and leaks.
\n\n[/quote]
\n\nThat would just be silly. We would have bots who would track leaderboard position every second and have non-official leaderboard:)
", "votes": null, "replies": [] }, { "id": 137891, "author_name": "happycube", "author_url": "", "post_date": "10/05/2016 21:16:47", "content": "Hmm, if I'm right about what this feature is based on, it could've been prevented by splitting train and test timewise, probably in the middle of the "lull" - there's little excuse for not doing that, unlike RedHat - aside from probably making things a lot harder still.
", "votes": null, "replies": [] }, { "id": 137899, "author_name": "tilii7", "author_url": "", "post_date": "10/05/2016 21:46:38", "content": "[quote=raddar;137890]
\n\nThat would just be silly. We would have bots who would track leaderboard position every second and have non-official leaderboard:)
\n\n[/quote]
\n\nHa, good one. I guess I underestimated that some people "just want to know" and would probably be doing exactly what you suggested.
\n\nMy larger point was that focusing on magic features, leaks, and score differences at 3rd and 4th decimal places is not necessarily good for us. Sure it sparks competition, but in many cases it will lead to over-fitting and to wild-goose chases after magic features.
\n\nJust to be clear: I am not against those who look for leaks, those who find them, or those who divulge them.
", "votes": null, "replies": [] }, { "id": 137994, "author_name": "satyasheel", "author_url": "", "post_date": "10/06/2016 14:16:52", "content": "@raddar did you use whole dataset (I mean did you dropped any column/s)?
", "votes": null, "replies": [] }, { "id": 137995, "author_name": "raddar", "author_url": "", "post_date": "10/06/2016 14:19:46", "content": "dropped only duplicate ones.
", "votes": null, "replies": [] }, { "id": 138050, "author_name": "rdslater", "author_url": "", "post_date": "10/06/2016 20:46:14", "content": "But what about the empty ones =)
", "votes": null, "replies": [] }, { "id": 138062, "author_name": "anokas", "author_url": "", "post_date": "10/06/2016 22:08:12", "content": "[quote=Bob, Lord of Cats;138050]
\n\nBut what about the empty ones =)
\n\n[/quote]
\n\nThat too =)
", "votes": null, "replies": [] }, { "id": 138063, "author_name": "aamaia", "author_url": "", "post_date": "10/06/2016 22:09:38", "content": "I noticed many failures happen in sequence -- id and id+1\nhttps://www.kaggle.com/aamaia/bosch-production-line-performance/failures-in-pairs
", "votes": null, "replies": [] }, { "id": 138065, "author_name": "happycube", "author_url": "", "post_date": "10/06/2016 22:32:03", "content": "facepalm ... another appearance of the Talkingdata randomizer? Clumping by date is one thing, but supposedly random ID #'s...???
", "votes": null, "replies": [] }, { "id": 138066, "author_name": "nigelcarpenter", "author_url": "", "post_date": "10/06/2016 23:02:42", "content": "I have to admit that this competition (or my inability to find the magic feature) is becoming very frustrating... And the more that others give hints that lead me nowhere the more frustrated I become!
\n\nAnyhow I thought I noticed something strange with id's and row duplicates link here but couldn't make much of it. I've also tried a train and test pairs submission that wasn't much better than a random submission and I think @anokas has said as much here.
\n\nAnyone willing to confirm we should we be looking elsewhere for an invite to the 0.4+ club?
", "votes": null, "replies": [] }, { "id": 138085, "author_name": "samehif", "author_url": "", "post_date": "10/07/2016 03:37:10", "content": "@Chippy: I share your frustration. I have been looking everywhere for the magical 0.1 boost but with no luck. One thing I promise is to make it public when and if I find it. To be positive, while looking for it I discovered many interesting characteristics of the data.
", "votes": null, "replies": [] }, { "id": 138089, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 05:08:54", "content": "How is any of this "predictive modeling?" Whenever there is any information given regarding test sets, they then become training sets. And by giving people an aggregate score on their model with respect to the "test" sets, they are getting information about those "test" sets. Therefore, those "test" sets are now training sets, and no true "prediction" is being done. All that's being done is fitting a model to a large set of data. There is value in that, and it is of use to companies, but it is not predictive modeling. This site should be clear about that, and they should communicate that to the companies who think they are getting predictive models and predictive modelers from this venture.
\n\nFor example, back on 9/22, radar had submitted 9 entries and his high score was 0.26727. Now all of a sudden, he's part of a team that has 123 submissions and a score of 0.50175. Did they just randomly double their score in 2 weeks, using no additional information other than the training sets? Doesn't seem too likely, does it? How did they even know they were moving in the right direction with each additional submission if they were not getting some sort of additional information with each submission? They were getting information with each submission, in the form of a score for that submission.
\n\nHow do we know how much information and submissions were shared among those team members? Why bother with cracking down on multiple accounts when people are submitting individually, then as part of a team, in the same competition? And now there is all this talk of a magical boost. How could they have known about this boost and refined their model if they weren't getting feedback from each submission?
\n\nBottom line, this site needs to call this what it is. It's certainly not predictive modeling.
", "votes": null, "replies": [] }, { "id": 138093, "author_name": "jiweiliu", "author_url": "", "post_date": "10/07/2016 06:00:22", "content": "[quote=GWillie;138089]
\n\nFor example, back on 9/22, radar had submitted 9 entries and his high score was 0.26727. Now all of a sudden, he's part of a team that has 123 submissions and a score of 0.50175.
\n\n[/quote]
\n\n1) raddar was 0.48+ before he merged, not all of sudden.
\n\n2) i personally didn't call this predictive modeling. I called it kaggling.
\n\n3) The very first thing of predictive modeling is proper data preparation to remove any unwanted leak. Now we are doing their job since bosch didn't get it done in the first place.
\n\n4) Exploiting leak is only part of the story. Non-leak features has tremendous value.
\n\n5) I believe the final solution is of pretty good value to bosch. They know what's wrong in data and how misleading it can be. They get a good benchmark. and they also get a pretty decent non-leak model out of the final solution.
\n\n6) last, as a kaggler, i believe whatever is given is part of the puzzle and is legit to use. In this case, exploiting the leak requires data cleaning, spotting pattern and proper encoding. It is also a good practice to me.
", "votes": null, "replies": [] }, { "id": 138095, "author_name": "raddar", "author_url": "", "post_date": "10/07/2016 06:08:07", "content": "@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.
\n\nAnd the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1.
\n\nAnd no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models.
", "votes": null, "replies": [] }, { "id": 138100, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 07:48:44", "content": "[quote=raddar;138095]
\n\n@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.
\n\n\n\n
I agree your opinion, I also spend lot of time to do feature engineering. \nBut I just kill time. Could you let me know how to start data exploration?
", "votes": null, "replies": [] }, { "id": 138102, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 07:51:22", "content": "Same curiosity as LEE. How to improve feature engineering skills ?? How to be better at understanding the data ? Not just specific to this competition, but in general.
", "votes": null, "replies": [] }, { "id": 138106, "author_name": "raddar", "author_url": "", "post_date": "10/07/2016 08:14:57", "content": "I picked 3 that help me the most:
\n\n1) One of the the first tricks I do is try to load the data in excel (well the portion that fits in excel row limit). Then do many kinds of conditional formatting. Colouring raw data helps to see some patterns (it depends if you have an eye for it and see something instantly, or it takes some time with filtering and etc.)
\n\n2) ploting cumulative distribution of Response=0 and Response=1 on same plot (and iterate through all variables). It often reveals interesting patterns within variables as well.
\n\n3) use some information criteria for each variable (IV, Gini, entropy, etc.), pick the most important ones and try scatterplotting every combination. Usually useful when making feature transformations for non-tree methods
", "votes": null, "replies": [] }, { "id": 138108, "author_name": "interesting", "author_url": "", "post_date": "10/07/2016 08:21:50", "content": "[quote=raddar;138095]
\n\n@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.
\n\nAnd the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1.
\n\nAnd no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models.
\n\n[/quote]
\n\nraddar, there will always be some who will discard the hard work done by others, I don't think you have to justify any of your work. your achievements and contribution to the Kaggle community are great.
\n\nIf you decide to open a course on data exploration and feature engineering i'd be the first to sign in :-)
\n\nwith great respect,
\n\nNati
", "votes": null, "replies": [] }, { "id": 138110, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 08:28:42", "content": "@raddar... Those are quite informative and sounds well structured and methodological. Thanks for the tips...
", "votes": null, "replies": [] }, { "id": 138127, "author_name": "gingerman", "author_url": "", "post_date": "10/07/2016 10:04:47", "content": "I side with @breakfastpirate that one cannot assess if the spillover effects are really exploits / leaks or if they may actually be of value to Bosch. Ordering (think Ids) is natural in manufacturing environment with production jobs, batches, lot sizing. So any information originating from sequences, ordering could be relevant to for failure detection. Similarly, the overlap between test and training data is also realistic because in the real setting Bosch would want to figure out the minimal amount of ground truth necessary to assess their production line performance. Ground truth is sourced from expensive testing and they could limit that if the predictions are more predictive (as measured by MCC).
", "votes": null, "replies": [] }, { "id": 138129, "author_name": "fibrato", "author_url": "", "post_date": "10/07/2016 10:13:14", "content": "[quote=GWillie;138089]
\n\nHow is any of this "predictive modeling?" Whenever there is any information given regarding test sets, they then become training sets. And by giving people an aggregate score on their model with respect to the "test" sets, they are getting information about those "test" sets. Therefore, those "test" sets are now training sets, and no true "prediction" is being done.....
\n\nBottom line, this site needs to call this what it is. It's certainly not predictive modeling.
\n\n[/quote]
\n\n@GWillie Although, I don't agree with your approach and conjectures regarding certain folks gaming the system, I think you brought up an interesting point about 'those "test" sets become training sets' that you should possibly reconsider.
\n\nIf your sole purpose is to chase previous submission scores, without regard to improving your methodology, you could get an unexpected surprise when the models are scored on the held out test data. Look at the movement in the Santander competition to see why final scoring is what matters most.
\n\nTo those who took the time to clarify GWillie's comments, kudos to you. It's better to get the facts out there, then let the fiction lie.
\n\nCheers
", "votes": null, "replies": [] }, { "id": 138160, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 13:20:12", "content": "If its not too much of a revelation: can someone using the golden feature tell me whether one can benefit from it by only using 60-70 numeric columns +(the golden feature). I have a feeling that i have discovered this feature but am unable to utilize it to the fullest due to limited memory.
", "votes": null, "replies": [] }, { "id": 138163, "author_name": "kailicht", "author_url": "", "post_date": "10/07/2016 13:24:44", "content": "[quote=Mohd Shadab Alam;138160]
\n\nIf its not too much of a revelation: can someone using the golden feature tell me whether one can benefit from it by only using 60-70 numeric columns +(the golden feature). I have a feeling that i have discovered this feature but am unable to utilize it to the fullest due to limited memory.
\n\n[/quote]
\n\nWhy aren't you just using 60-70 numeric columns + the awesome golden feature and check if it performs better?
", "votes": null, "replies": [] }, { "id": 138166, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 13:40:16", "content": "Yeah it does but the improvement is nowhere close to 0.1 types
", "votes": null, "replies": [] }, { "id": 138182, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 15:19:40", "content": "I appreciate all the comments on my post, but I couldn't help but notice (and even expected) that few, if any, actually addressed my point. First, I never said there wasn't hard work going into this. I acknowledge and respect that there is. And I said that there was value to this. But my point is, it's not predictive modeling - it's large scale data mining and fitting a model to a large set of data. And yes, a big part of that is data exploration. I never said it wasn't. But by getting information about your test sets in the form of leaderboard feedback, then by definition, those sets are no longer test sets, they are training sets, and you are no longer doing predictive modeling, you are doing data mining and model fitting. And the ability to refine those models in the form of hundreds of submissions is evidence that the test sets are actually training sets.
\n\nWhy is this important to point out? Well, I happen to know from experience that there is a vast demand and need for true predictive modeling. And this site generates revenue for these competitions by promoting itself to companies as being able to provide predictive models and predictive modelers. But when competitions are run like this, then those companies are not getting true predictive models and predictive modelers. This is not doing anything to promote the field of predictive modeling/analytics. In fact, I would argue that it is hindering the advancement of the field.
\n\nI'm not coming down on any of the participants nor trying to minimize any of the work they're doing. It is not easy, and it does have value. But we all - most importantly the companies buying into it - need to be clear about what it is and what it isn't. The site was founded and initially funded as a platform for predictive modeling and analytics, and it bills itself to competitors and potential funding companies as being just that. And there's a good bit of money involved in that pursuit. But the way it's currently run, a competition like this is data mining and model fitting, not predictive modeling/analytics. I just think everyone not only has a right to know this but really should know this.
", "votes": null, "replies": [] }, { "id": 138183, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 15:28:46", "content": "@rcarson
\n\n1) raddar was 0.48+ before he merged, not all of sudden.
\n\nOk. So then raddar went from 0.27 to 0.48 before he merged. If this were true predictive modeling, he would have had no feedback from his test set submissions, so the only way to make this kind of increase would basically be random. If it were true predictive modeling, his final score would be somewhere below 0.27.
\n\n2) i personally didn't call this predictive modeling. I called it kaggling.
\n\nThank you for agreeing with me. That's all I'm saying. But the danger is, kaggle IS calling it predictive modeling.
\n\n3) The very first thing of predictive modeling is proper data preparation to remove any unwanted leak. Now we are doing their job since bosch didn't get it done in the first place.
\n\nAgreed. But the analogy to predictive modeling, in this case, stops there.
\n\n4) Exploiting leak is only part of the story. Non-leak features has tremendous value.
\n\nAgreed, but in this case, it's not predictive modeling.
\n\n5) I believe the final solution is of pretty good value to bosch. They know what's wrong in data and how misleading it can be. They get a good benchmark. and they also get a pretty decent non-leak model out of the final solution.
\n\nAgreed, but in this case, it's not predictive modeling. And it's my opinion that Bosch deserves to know that.
\n\n6) last, as a kaggler, i believe whatever is given is part of the puzzle and is legit to use. In this case, exploiting the leak requires data cleaning, spotting pattern and proper encoding. It is also a good practice to me.
\n\nAgreed. I have no problem with competitors using whatever they can to advance their score. I'm just saying that we need to call it what it is. The way the rules are set up for competitions, it's not predictive modeling.
", "votes": null, "replies": [] }, { "id": 138184, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 15:33:09", "content": "@raddar
\n\n"@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that."
\n\nYes, and these are all important parts of large-scale data mining and fitting a model to a large set of data, I don't disagree with that and never took exception to that. I have done that type of work, as I have also done predictive modeling work. Having done the latter, I know that this is not true predictive modeling.
\n\n"And the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1."
\n\nYes, and this "serious modeling" that you have done is data mining and fitting a model to data. It is not true predictive modeling, given how the rules of the competition are set up. That's my point.
\n\n"And no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models."
\n\nYou don't need to be getting any other feedback than that in order for this to be disqualified as true predictive modeling. That's my point.
", "votes": null, "replies": [] }, { "id": 138187, "author_name": "thawatt", "author_url": "", "post_date": "10/07/2016 15:35:11", "content": "And now back to the actual topic...anyone have anything new to add to the 'magical feature' discussion?
", "votes": null, "replies": [] }, { "id": 138188, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 15:40:37", "content": "@wordsmith
\n\n"@GWillie Although, I don't agree with your approach and conjectures regarding certain folks gaming the system, I think you brought up an interesting point about 'those "test" sets become training sets' that you should possibly reconsider.
\n\nIf your sole purpose is to chase previous submission scores, without regard to improving your methodology, you could get an unexpected surprise when the models are scored on the held out test data. Look at the movement in the Santander competition to see why final scoring is what matters most.
\n\nTo those who took the time to clarify GWillie's comments, kudos to you. It's better to get the facts out there, then let the fiction lie."
\n\nYou seem to not fully understand that which you are disagreeing with. I never stated that "the sole purpose is to chase previous submission scores, without regard to improving your methodology." In fact, I am saying just the opposite. People are improving methodology based on previous submission scores. You are using information from those "test" sets to improve methodology and score. Therefore, those "test" sets are now, by definition, training sets. You have used those sets to improve your methodology/score, so you have trained on those sets.
\n\nVery little was done to "clarify" my comments. In fact, my main point was not really addressed at all. So yes, it is better to keep trying to get the facts out rather than let the fiction lie, and I will continue to do so.
", "votes": null, "replies": [] }, { "id": 138190, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/07/2016 15:46:41", "content": "@GWillie, one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a model that predicts the remaining 70% of test data (i.e. predictive modeling).
", "votes": null, "replies": [] }, { "id": 138194, "author_name": "gaborfodor", "author_url": "", "post_date": "10/07/2016 16:04:06", "content": "I am using xgboost.Booster.predict() so it is predictive modeling. :D
\n\nBut seriously do not worry too much about the number of submissions. In this case having more submissions is not really competitive advantage. If you think it is, you can submit 5 per day so there are still 165 submissions left for you.
\n\nBigger problem could be that the train set contains more information about the test set than it was probably intended. Leakages/ magic features are usually appear in these competitions.
", "votes": null, "replies": [] }, { "id": 138196, "author_name": "happycube", "author_url": "", "post_date": "10/07/2016 16:07:26", "content": "Also note that CV and LB scores are very highly correlated, so raddar didn't need to use the public LB to know he's making progress.
\n\nI think there are some good clues now to The Feature(tm), and IMO it's a matter of time until it's public, and there'll probably be a .4 script within the next two weeks I figure, if it doesn't need too much time/RAM to compute. So happy hunting and stay patient :)
", "votes": null, "replies": [] }, { "id": 138199, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 16:11:31", "content": "[quote=raddar;138106]
\n\nI picked 3 that help me the most:
\n\n[/quote]
\n\nThank you for your kind advice. I try as hard as I can.
", "votes": null, "replies": [] }, { "id": 138201, "author_name": "kailicht", "author_url": "", "post_date": "10/07/2016 16:18:43", "content": "[quote=BreakfastPirate;138190]
\n\n@GWillie, one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a model that predicts the remaining 70% of test data (i.e. predictive modeling).
\n\n[/quote]
\n\n@GWillie
\n\nTotally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?
", "votes": null, "replies": [] }, { "id": 138203, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 16:34:15", "content": "I believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.
\n\n[quote=happycube;138196]
\n\nAlso note that CV and LB scores are very highly correlated, so raddar didn't need to use the public LB to know he's making progress.
\n\nI think there are some good clues now to The Feature(tm), and IMO it's a matter of time until it's public, and there'll probably be a .4 script within the next two weeks I figure, if it doesn't need too much time/RAM to compute. So happy hunting and stay patient :)
\n\n[/quote]
", "votes": null, "replies": [] }, { "id": 138205, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 16:38:06", "content": "Give me RAM i'll give you golden feature :P...\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\nMail me at: golden.reflection@gmail.com (its a real id)
", "votes": null, "replies": [] }, { "id": 138206, "author_name": "maxpowerwastaken", "author_url": "", "post_date": "10/07/2016 16:42:15", "content": "@GWillie, regarding your point that Kaggle competitions are data mining but not predictive modeling...
\n\nIn the last Kaggle competition (Grupo Bimbo), we were given a training dataset through "Week 9", public leaderboard scores were based on unlabeled "Week 10" predictions (from 'test set'), and final/private leaderboard winners selected based on unlabeled "Week 11" predictions (from 'test set'). Would you call this predictive modeling? Winning entries were selected based on predictions for a later time period than both the training data and the public leaderboard scores.
\n\nIf you consider that predictive modeling, I'd interpret your posts as criticizing how the train/test split is in this competition, as opposed to a problem with every Kaggle competition. In the forums, there's always a healthy discussion of how these splits are done in each competition, and how optimal/useful or not they are. My understanding is that Kaggle works with the data owner to derive the most useful way to do train/test data split. Maybe the process could be improved, apparently it is non-trivial to get right in every case.
", "votes": null, "replies": [] }, { "id": 138207, "author_name": "tilii7", "author_url": "", "post_date": "10/07/2016 16:43:28", "content": "\n\n\nBut the way it's currently run, a competition like this is data mining and model fitting, not predictive modeling/analytics. I just think everyone not only has a right to know this but really should know this.
\n
You said "not predictive modeling" 5 times in the last few messages. Based on your level of conviction, one would think that you must be absolutely right. And you are not.
\n\nTo expand on BreakfastPirate's fine argument: we get feedback only on portion of the data, and our models are ultimately tested on data that have not been part of that feedback. That is different from "regular" predictive modeling only in a sense that our only info there is our local cross-validation score, while here we can compare it with leaderboard. Those who know how to do cross-validation properly would do just fine without that piece of information.
\n\nIt is not uncommon at all for large train datasets that 1/5 (or even 1/4) of the data is set aside for "fixed" validation. After modeling the remaining train data based on cross-validation and adjusting the model to the validation set, eventually we test its performance on the test dataset. If you think about it, the leaderboard score is nothing but a validation score on our hold-out data, which still gets tested on the rest of the data (private leaderboard).
", "votes": null, "replies": [] }, { "id": 138208, "author_name": "maxpowerwastaken", "author_url": "", "post_date": "10/07/2016 16:45:15", "content": "@Mohd\nhttps://aws.amazon.com/ec2/instance-types/
", "votes": null, "replies": [] }, { "id": 138209, "author_name": "tilii7", "author_url": "", "post_date": "10/07/2016 16:45:33", "content": "[quote=Mohd Shadab Alam;138205]
\n\nGive me RAM i'll give you golden feature :P...\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\nMail me at: golden.reflection@gmail.com (its a real id)
\n\n[/quote]
\n\nWould you like fries with your supersized RAM? :D
", "votes": null, "replies": [] }, { "id": 138210, "author_name": "", "author_url": "", "post_date": "10/07/2016 16:47:25", "content": "Where is the rabbit?
", "votes": null, "replies": [] }, { "id": 138211, "author_name": "anokas", "author_url": "", "post_date": "10/07/2016 16:51:16", "content": "[quote=Mohd Shadab Alam;138203]
\n\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.
\n\n[/quote]
\n\nThis is simply not true.
", "votes": null, "replies": [] }, { "id": 138213, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 16:54:44", "content": "dunno about rabbit,i can spot a kitten though ;)
", "votes": null, "replies": [] }, { "id": 138216, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 16:57:23", "content": "[quote=anokas;138211]
\n\n[quote=Mohd Shadab Alam;138203]
\n\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.
\n\n[/quote]
\n\nThis is simply not true.
\n\n[/quote]\nDude you were using huge ram right from the begining....when did you get time to try it with 60-70 features....
", "votes": null, "replies": [] }, { "id": 138218, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 17:00:01", "content": "[quote=anokas;138211]
\n\n[quote=Mohd Shadab Alam;138203]
\n\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.
\n\n[/quote]
\n\nThis is simply not true.
\n\n[/quote]
\n\nLol. Please don't break someone who is convinced of spotting the golden duck.
", "votes": null, "replies": [] }, { "id": 138219, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 17:01:41", "content": "[quote=Mohd Shadab Alam;138205]
\n\nGive me RAM i'll give you golden feature :P...\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\nMail me at: golden.reflection@gmail.com (its a real id)
\n\n[/quote]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
", "votes": null, "replies": [] }, { "id": 138220, "author_name": "anokas", "author_url": "", "post_date": "10/07/2016 17:03:34", "content": "[quote=Mohd Shadab Alam;138216]
\n\nDude you were using huge ram right from the begining....when did you get time to try it with 60-70 features....\n[quote=anokas;138211]
\n\n[quote=Mohd Shadab Alam;138203]
\n\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.
\n\n[/quote]
\n\nThis is simply not true.
\n\n[/quote]
\n\n[/quote]
\n\nWe can get 0.49 using exactly 0 numeric features in our model, and that can run in 8GB ram no problem.
\n\nJust because we have access to a lot of RAM, doesn't mean we don't make our code memory efficient. If you need 256GB+ RAM to run your model, then you need to optimise your memory usage. Our full LB model can still run no problem start-to-finish on a 64GB machine.
\n\nAt the end of the competition, if you want I'll build a 0.4 model on a Raspberry Pi.
", "votes": null, "replies": [] }, { "id": 138221, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 17:07:28", "content": "@anokas 'Everyone Stop Improving Please' -> I have stopped improving for a long time now.
", "votes": null, "replies": [] }, { "id": 138222, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 17:08:36", "content": "[quote=PrakashSubedi;138219]
\n\n[quote=Mohd Shadab Alam;138205]
\n\nGive me RAM i'll give you golden feature :P...\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\nMail me at: golden.reflection@gmail.com (its a real id)
\n\n[/quote]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
\n\n[/quote]\nIf you think your model can handle this duck feature then lets team up and crack it big. BTW i am using python with dtype option to load this data but am unable to do so.
", "votes": null, "replies": [] }, { "id": 138223, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 17:13:29", "content": "@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
", "votes": null, "replies": [] }, { "id": 138224, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 17:13:35", "content": "", "votes": null, "replies": [] }, { "id": 138225, "author_name": "anokas", "author_url": "", "post_date": "10/07/2016 17:16:03", "content": "[quote=<<>>;138224]
\n\n@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
\n\n[/quote]
\n\n0.49 without any of the original tables I should say.
", "votes": null, "replies": [] }, { "id": 138226, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 17:21:11", "content": "oh i understand sorry ^^
", "votes": null, "replies": [] }, { "id": 138227, "author_name": "tilii7", "author_url": "", "post_date": "10/07/2016 17:24:20", "content": "[quote=PrakashSubedi;138219]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
\n\n[/quote]
\n\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from here. Then:
\n\ncsv2libsvm.py data_numeric.csv data_numeric.libsvm 969\nThis will convert the .csv file into LibSVM format. The LibSVM file can be loaded as such into a sparse csr matrix.
\n\nfrom sklearn.datasets import load_svmlight_file\n# This will load the data into a sparse csr matrix and the target into an np array\nprint("\\n Reading train data ...")\ntrain_data, train_target = load_svmlight_file("./train_numeric.libsvm")\nprint(" Done reading train data (%d x %d)" % (train_data.shape[0], train_data.shape[1]))\nprint(" Memory consumption of the sparse matrix is %d bytes" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.
\n\nEdit: I should give credit to SVMLight as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.
", "votes": null, "replies": [] }, { "id": 138228, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 17:40:45", "content": "[quote=Tilii;138227]
\n\n[quote=PrakashSubedi;138219]
\n\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.
\n\n[/quote]
\n\nFor those who don't have 16Gb of RAM, here is an alternative solution. Get csv2libsvm.py from here. Then:
\n\ncsv2libsvm.py data_numeric.csv data_numeric.libsvm 969\nThis will convert the .csv file into LibSVM format. The LibSVM file can be loaded as such into a sparse csr matrix.
\n\nfrom sklearn.datasets import load_svmlight_file\n# This will load the data into a sparse csr matrix and the target into an np array\nprint("\\n Reading train data ...")\ntrain_data, train_target = load_svmlight_file("./train_numeric.libsvm")\nprint(" Done reading train data (%d x %d)" % (train_data.shape[0], train_data.shape[1]))\nprint(" Memory consumption of the sparse matrix is %d bytes" % (train_data.data.nbytes + train_data.indptr.nbytes + train_data.indices.nbytes))\nIt takes ~2.4Gb, so there is room to do the same with test data and still be well under 8Gb.
\n\nEdit: I should give credit to SVMLight as well since sklearn named its routine after this program rather than LibSVM. Both use the same file format.
\n\n[/quote]
\n\nI was doing the same until I found out lots of columns in date table is exact duplicates.. Removing them eased the burden,and SVMLIGHT was not needed.
", "votes": null, "replies": [] }, { "id": 138232, "author_name": "salkaa", "author_url": "", "post_date": "10/07/2016 17:51:56", "content": "@anokas , but surely you got engineered your features from given tables ... ?
", "votes": null, "replies": [] }, { "id": 138233, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 17:53:32", "content": "[quote=anokas;138225]
\n\n[quote=<<>>;138224]
\n\n@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
\n\n[/quote]
\n\n0.49 without any of the original tables I should say.
\n\n[/quote]\nthank you for clarifying, now i am more sure than before that i spotted the right feature.
", "votes": null, "replies": [] }, { "id": 138234, "author_name": "outliar", "author_url": "", "post_date": "10/07/2016 17:55:33", "content": "point taken
", "votes": null, "replies": [] }, { "id": 138235, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 17:56:14", "content": "@Alam Check your 'golden' Inbox ....
", "votes": null, "replies": [] }, { "id": 138236, "author_name": "devinanzelmo", "author_url": "", "post_date": "10/07/2016 17:58:00", "content": "Maybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks.
", "votes": null, "replies": [] }, { "id": 138237, "author_name": "rdslater", "author_url": "", "post_date": "10/07/2016 17:59:06", "content": "As entertaining as all of this is we can probably do without the smacktalk. General Bickering is really not what this place is about or more correctly, what I had hoped this would be about. While it can be frustrating if you are not in the know, most of the time (and the current leaders have been in this category) at the end of the competition, they are more than happy to share insights. The post-competition discussion is probably as much a chance of learning as the pre-deadline discussion.
\n\nAlso I need to learn to unsubscribe to threads as you people blew up my mailbox over lunch.
\n\n@Tilli--ty for pointing out the SVM method. I did my the old-school way to convert for some vw fun. I should have known there was a potentially easier way.
\n\nAnd while you guys were blowing up my mailbox, I did realize a faster way to check for duplicate variables. Sometimes brute force through massive hardware is not the answer. It's nice to have, but unless you really understand what you are doing, it what is popularly called "shotgunning" the problem--fire a bunch a bullets and hope you hit something good.
", "votes": null, "replies": [] }, { "id": 138238, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/07/2016 18:06:15", "content": "[quote=Devin;138236]
\n\nMaybe there should be spoiler tags in this forum. That way I don't have to read the whole thread and can just jump to reading all the leaks.
\n\n[/quote]
\n\nThis could be a Kaggle text-mining competition. Build a model that automatically sifts through all the forums and identifies only the posts that reveal the magic features. The question is, "Would that competition itself have any magic features?"
", "votes": null, "replies": [] }, { "id": 138239, "author_name": "scirpus", "author_url": "", "post_date": "10/07/2016 18:06:16", "content": "The following works pretty well for nuking duplicates - it significantly reduces columns\n\nremove = []\ncols = rawdata.columns\nfor i in range(len(cols)-1):\n v = rawdata[cols[i]].values\n for j in range(i+1,len(cols)):\n if np.array_equal(v,rawdata[cols[j]].values):\n remove.append(cols[j])\n\nrawdata.drop(remove, axis=1, inplace=True)\nremove = []\nfor col in rawdata.columns[1:]:\n print(col, rawdata[col].std())\n if rawdata[col].std() == 0.0:\n remove.append(col)\n\nrawdata.drop(remove, axis=1, inplace=True)\n@BreakfastPirate, I can think of quite a few useful features. But to see some real magic I believe we would, for instance, need to ban leaderboard info and then get sloppy.
", "votes": null, "replies": [] }, { "id": 138249, "author_name": "anokas", "author_url": "", "post_date": "10/07/2016 18:57:20", "content": "[quote=Mohd Shadab Alam;138233]
\n\n[quote=anokas;138225]
\n\n[quote=<<>>;138224]
\n\n@anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?
\n\n[/quote]
\n\n0.49 without any of the original tables I should say.
\n\n[/quote]\nthank you for clarifying, now i am more sure than before that i spotted the right feature.
\n\n[/quote]
\n\nIf you spotted the right features, then you should be able to run just on those features and get 0.4+ - If you can't then it's not the same features as us ;D
", "votes": null, "replies": [] }, { "id": 138250, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 18:59:11", "content": "@BreakfastPirate
\n\n"one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a model that predicts the remaining 70% of test data (i.e. predictive modeling)."
\n\nYour assumption that an overfit to the public leaderboard will result in a drop on the private leaderboard is not necessarily true. If this were true predictive modeling, where models are built on a relatively sparse amount of data, then this might be true. But with these large data sets, overfitting to the public leaderboard could, and does it seems, translate to an overfitting on the private leaderboard. And yes, the goal of this modeling is similar to the goal of predictive modeling, but my point is that the implementation of the modeling on a competition like this is actually not predictive modeling.
", "votes": null, "replies": [] }, { "id": 138253, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 19:02:52", "content": "@BreakfastPirate
\n\n"I am using xgboost.Booster.predict() so it is predictive modeling. :D"
\n\nI was hoping you were joking about that.
\n\n"But seriously do not worry too much about the number of submissions. In this case having more submissions is not really competitive advantage. If you think it is, you can submit 5 per day so there are still 165 submissions left for you."
\n\nI'm not worried about the number of submissions, I'm simply pointing out that the way this competition is set up, it's not really predictive modeling.
\n\n"Bigger problem could be that the train set contains more information about the test set than it was probably intended. Leakages/ magic features are usually appear in these competitions."
\n\nThat could be a problem as well. But that is a problem that we can't say for sure will occur with every competition. But what we can say about a competition like this is, by giving information about results from a test set, that test set is now a training set. That's my point.
", "votes": null, "replies": [] }, { "id": 138254, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 19:09:58", "content": "@happycube
\n\n"Also note that CV and LB scores are very highly correlated, so raddar didn't need to use the public LB to know he's making progress."
\n\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher. But somehow they knew to continue in a certain direction and get up to the 0.50 point that they are now.
\n\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.
", "votes": null, "replies": [] }, { "id": 138255, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 19:12:34", "content": "@BreakfastPirate
\n\n"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nYou just reiterated my point right there. That 30% holdout set that was labeled as a "test" set. You just said that you're using that "test" set for training. Therefore, what was put up as a test set is actually a training set. Now you see, and we are in agreement.
", "votes": null, "replies": [] }, { "id": 138256, "author_name": "kailicht", "author_url": "", "post_date": "10/07/2016 19:14:48", "content": "[quote=GWillie;138254]
\n\nBasically, you would use feedback from your training set to improve your model and improve the accuracy of that model in fitting the data in the training set. But in true predictive modeling, that model would be applied to the test set, and the results from that would be your score. But by getting the additional feedback of how your model did on the test set, that test set becomes a training set. So then you're actually fitting to both the initial training set and test set. This is not predictive modeling.
\n\n[/quote]
\n\nYou are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. This is your test set!
\n\nEDIT:\n"Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nThat was me.
", "votes": null, "replies": [] }, { "id": 138257, "author_name": "taffeylewis", "author_url": "", "post_date": "10/07/2016 19:15:03", "content": "GWillie, not to be pedantic, but... nobody likes pedantic people.
\n\nWe get that you have issues with the absolute definition of "predictive modeling". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?
\n\nCan we please move on?
", "votes": null, "replies": [] }, { "id": 138259, "author_name": "tilii7", "author_url": "", "post_date": "10/07/2016 19:22:17", "content": "[quote=Taffey Lewis;138257]
\n\nGWillie, not to be pedantic, but... nobody likes pedantic people.
\n\n[/quote]
\n\nI like pedantic people, and I am not saying that just to be a contrarian.
\n\nI also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens.
", "votes": null, "replies": [] }, { "id": 138260, "author_name": "gullu39", "author_url": "", "post_date": "10/07/2016 19:26:12", "content": "Well, Now there is no link whatsoever between the thread title and the discussion over last two pages !!!
", "votes": null, "replies": [] }, { "id": 138261, "author_name": "niks2000", "author_url": "", "post_date": "10/07/2016 19:32:49", "content": "Let us debate the magical Feature issues. I'm really at the end of my rope. I can`t find pattern from data.
", "votes": null, "replies": [] }, { "id": 138266, "author_name": "diegofrata", "author_url": "", "post_date": "10/07/2016 19:42:11", "content": "Couldn't resist.
\n\n
@MaxPower
\n\nRegarding the Grupo Bimbo competition, were you given any feedback (i.e., scores) up until Week 9? If you were given no information about the test sets up until the final/private leaderboard was revealed, then sure, that would be predictive modeling. But that's not what's happening with this competition, nor with the others I've seen during my short time on here.
\n\nAnd yes, I am basically criticizing the train/test splits. That and the fact that 30% of the "test" set is actually a training set, which adds to the train/test split. I would suggest that the process could definitely be improved.
", "votes": null, "replies": [] }, { "id": 138287, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 22:49:39", "content": "@Tilii
\n\n"You said "not predictive modeling" 5 times in the last few messages. Based on your level of conviction, one would think that you must be absolutely right. And you are not."
\n\nIn my first post, I said it only a couple times and none of the responses addressed that directly. So I decided to repeat it more often for effect, to see if it would elicit responses that could lead to a more meaningful discussion. Apparently it worked, and thanks for at least addressing that point directly.
\n\n"To expand on BreakfastPirate's fine argument: we get feedback only on portion of the data, and our models are ultimately tested on data that have not been part of that feedback. That is different from "regular" predictive modeling only in a sense that our only info there is our local cross-validation score, while here we can compare it with leaderboard. Those who know how to do cross-validation properly would do just fine without that piece of information."
\n\nSo if those who climb the leaderboard could truly do that without that piece of information and associated feedback (information and feedback that is disqualifying this from being predictive modeling up to the private leaderboard part), why have that piece of information and the associated feedback? So we're just supposed to assume that they would have doubled their score in 2 weeks (prior to private leaderboard time) without that "piece of information?" With this much money at stake, I'm not necessarily inclined to give someone that benefit of the doubt.
\n\nHere's the real problem with all this. I get the whole 30/70 split on the "test" set. But let's consider the Red Hat competition, which I believe was run the same way. The training/"test" set split on that was 82%/18%. Now take 30% of that 18% test set which has actually become a training set because of the information/feedback during the public leaderboard portion, and you're up to a 87%/13% training/test set split. And considering that you're probably getting more useful information than just the 30% from that "test" set, the split is probably even higher - mabye even getting over 90%. One could argue that this kind of split is true predictive modeling, but I would say that as you approach this split level, you're really just constructing a model to fit the ~90% of observed data. The 10% (or probably less) of "test" data is simply serving as a means to assess the results and determine a winner. Maybe the 10% is different enough that you see significant drops in the private leaderboard, that's possible. But my point is, up until that point - during the public leaderboard time - the training/test set notion is misleading. What is labeled as test data during that time is actually training data because of the feedback.
\n\nSo yes, as far as this competition and others like it, I stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data. And the problem with this is, it is presented as though it is predictive modeling (training set/test set). And to someone like myself who is fairly new here, I came expecting to see predictive modeling. But after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.
", "votes": null, "replies": [] }, { "id": 138288, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 22:58:52", "content": "@Charly
\n\n"You are obviously not understanding how the evaluation is done: 70% of the data is seen by no one until the end of the competition. This is your test set!"
\n\nYes, I get this. But again, consider that in the Red Hat competition, that 70% of the "test" set represented only 13% of the overall data. So that was basically just constructing a model to fit 87% (or more likely over 90%) of the data. There is not much prediction done there.
\n\nEDIT: "Totally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?"
\n\nAgain, the difference could be bumping up the training/test split from about 80/20 to more like 90/10.
\n\nBut all that is really irrelevant to what I said from the start. If you consider the public leaderboard part of these competitions, it's not predictive modeling that's going on. That was my point. It's purported to be, given the "training set" and "test set" labels. But by giving feedback on that "test set" and allowing people to refine their models based on that feedback, that is no longer a test set and that is no longer predictive modeling. So for competitions that are run like that, there is no predictive modeling going on during a majority of the competition. And, by making the training set such a large % of the total data, there really isn't much prediction going on, even at the end.
", "votes": null, "replies": [] }, { "id": 138289, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:03:16", "content": "@Tilii
\n\n"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens."
\n\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.
", "votes": null, "replies": [] }, { "id": 138290, "author_name": "nigelcarpenter", "author_url": "", "post_date": "10/07/2016 23:06:50", "content": "Enough.... please.... (or if you must please start another thread!)
", "votes": null, "replies": [] }, { "id": 138291, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:09:44", "content": "@Taffey Lewis
\n\nThat's the whole reason I brought this up. I wonder if a company like Bosch really does understand EXACTLY what they're getting into with Kaggle. I agree that this exercise in constructing a model to fit a large set of data is useful and not an easy task. I agree that they will see value in this. But what is happening over and over in a wide variety of industries is that there is a certain level of disappointment when trying to apply these models in the true sense of predictive modeling. That is, when you have much less than 90% of the data available as observations upon which to build a model. When overfitting becomes your worst enemy instead of your ally. That is true predictive modeling, and it is not being addressed in these recent competitions I've seen. Given that it remains a huge problem, I would argue that we should all care.
\n\nWe get that you have issues with the absolute definition of "predictive modeling". Others beg to differ. All of us are approaching this for our own reasons, and for a company like Bosch, I'm sure they understand EXACTLY what they're getting into with Kaggle, so from their perspective, who cares?
\n\nCan we please move on?
", "votes": null, "replies": [] }, { "id": 138293, "author_name": "jturkewitz", "author_url": "", "post_date": "10/07/2016 23:11:02", "content": "[quote=GWillie;138289]
\n\n@Tilii
\n\n"I also like people with strong opinions who can actually back them up with arguments, or modify the opinion when a better argument is presented. Let's see if that happens."
\n\nI couldn't agree more. I have presented my argument as to why the public leaderboard portion of competitions run like this are not predictive modeling, even though they are made out to be. If you are disagreeing with that, then I look forward to your argument in that regard.
\n\n[/quote]
\n\nYou are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man. From wikipedia:
\n\n\n\n\nPredictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event, regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place.
\n
You build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling. Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction. You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects.
", "votes": null, "replies": [] }, { "id": 138294, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:13:56", "content": "@Chippy
\n\nIt is relevant to this competition. That's why I made the comment here. I'm happy to take the general discussion to another thread, although I would think that people would actually want to give this some thought rather than just attack the messenger. People are on here to try to get information about how to become or improve their skills as a data scientist. Misleading them is doing a huge disservice to them and the field. But let's just sweep it under the rug and move on. That should adequately address the problems we're seeing in the field.
", "votes": null, "replies": [] }, { "id": 138296, "author_name": "kailicht", "author_url": "", "post_date": "10/07/2016 23:24:14", "content": "I also agree with Tilii, but to my point of view your arguments are just not right. The train/ test split here is 0.65/ 0.35 with the special condition that 15 % of the set is only available by checking your model on the public leaderboard. 35 % are not seen until the end. That fits my definition of predictive modeling.
\n\nLet's agree that we disagree and go on chase some wild goose.
", "votes": null, "replies": [] }, { "id": 138297, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/07/2016 23:29:38", "content": "[quote=GWillie;138287]
\n\nI stand by my statement that during the majority of the competition, it is not predictive modeling but rather fitting a model to a very large set of data....\nBut after watching people climb up the public leaderboard over several months and hundreds of submissions, I realized that for a majority of these competition, we're not seeing predictive modeling as it is purported to be.
\n\n[/quote]
\n\nYou say "the majority of the competition" is focused on the public LB, BUT in reality the ENTIRE duration of every competition is focused on getting the highest rank on the private LB. The public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...
\n\nSo your whole argument is based on your misunderstanding of how Kaggle competitions really work.
", "votes": null, "replies": [] }, { "id": 138298, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:31:47", "content": "@Jared Turkewiz
\n\nThank you for clarifying my position. Everything you posted points to me being correct.
\n\n"You are judged on the private leaderboard, not the public leaderboard so your argument is a straw-man."
\n\nAnd if the private leaderboard scores are based on only about 10% of available data, you're not really doing predictive modeling. So that argument is not a straw-man.
\n\n"From wikipedia:
\n\nPredictive modeling uses statistics to predict outcomes.[1] Most often the event one wants to predict is in the future, but predictive modelling can be applied to any type of unknown event,"
\n\nLet's stop it there for a minute. So by giving information about the events which you are trying to "predict", those events cease to be "uknown" events. Thus I am right about the public leaderboard portion not being predictive modeling. Thank you for providing even more evidence to support me.
\n\n"regardless of when it occurred. For example, predictive models are often used to detect crimes and identify suspects, after the crime has taken place."
\n\nRight, and this is not an issue of timing, it is an issue of information that is made available.
\n\n"You build a model, then you use this model to predict outcomes for data that you don't know the answer to. This is pretty much text book predictive modeling."
\n\nRight, and by giving people the "answer" as to how many of their modeled data is correct, it fails to be predictive modeling, according to the textbook definition that you provided.
\n\n"Note that it doesn't require the prediction to be in the future, you just have to not know the answer when you make your prediction."
\n\nI never said anything about requiring the prediction to be in the future. Again, it is about information. During the leaderboard portion of a competition, you are getting information about the answer. Therefore you do know something about the answer, therefore, according to you, you are not doing predictive modeling.
\n\n"You are allowed to know properties of the event you are trying to predict the outcome of, as would be the case if you are using predictive models to detect crimes and identify suspects."
\n\nOf course you are allowed to know properties of the event you are trying to predict the outcome of, that's what predictive models are based on. But if you know, or have information about, the actual outcome (not just properties of it, but "it"), then you're not doing predictive modeling. And by getting a total number of correct outcomes from your model (as you do from the public leaderboard scores), that is information about the actual outcomes. And once you have that information, you are just fitting a model to that data, you're not predicting that data. And this is exactly the kind of information that people are using to move up the public leaderboard and double their score in 2 weeks time. If you fail to acknowledge this, then don't be surprised if you continue to see the same people game this and move up to the top - competition after competition. I guess just keep attributing that to luck?
", "votes": null, "replies": [] }, { "id": 138300, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:42:29", "content": "@BreakfastPirate
\n\nI am new here and don't totally understand the private LB. Are you saying that you get feedback during the competition about the test data that will be used for the private LB? If so, that only strengthens my position and does not weaken it. If you're getting feedback on the private LB, then you're getting information on 100% of the true test set, and thus there really is no test set, and thus there really is no predictive modeling.
\n\nIf you don't get feedback on the private LB until the end, then my argument remains the same and I am not wrong. Yes, the final rankings are based on the private LB. But during the public LB portion, you are getting feedback about your submissions (this I know). So that portion is not predictive modeling - it is building a model to fit the training data + 30% of the "test" data. And in some cases, like Red Hat, that data can be upwards of 90% of the total data. This is basically fitting a model to data. You say that public LB means nothing in the end, but it actually means everything in the end for a competition like that. It is the time that people spend constructing and refining their models. And when that final "prediction" is on only 10% of all the data, then it is likely that the ones who did best on the public LB will do best on the private LB (not guaranteed, but likely). Therefore, being highly ranked on the public LB in those competitions is pretty much what it's all about. Granted, in true predictive modeling, where training set data is much less than 90% of all available data, then ranking on public LB would not be as important an indicator of success on the private LB. But that isn't the case in most of these competitions that I've seen.
\n\nSo, my whole argument is based on my understanding of how the public LB works in Kaggle competitions. I may have some misunderstanding of how the private LB works, but clearing that up could only strengthen my argument.
", "votes": null, "replies": [] }, { "id": 138301, "author_name": "kailicht", "author_url": "", "post_date": "10/07/2016 23:44:04", "content": "Guys, let's just stop answering. To my point of view you just not have the right mindset for kaggle. You know what my biggest incentive for participating is? I learned how to deal with in memory computing of data with billions of entries. ..
", "votes": null, "replies": [] }, { "id": 138302, "author_name": "willieliao", "author_url": "", "post_date": "10/07/2016 23:45:28", "content": "@GWillie, either your definition of "predictive modelling" is out of line with the general usage or there is a misunderstanding about how the public leaderboard is used by Kagglers.
\n\nFor the former, see @Jared Turkewitz's response. As a hypothetical example, let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did "predictive modelling" because he did not use any information on the leaderboard.
\n\nNow, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do "predictive modelling"?
\n\nFor the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing "predictive modelling".
\n\nThe other group spends dozens or hundreds of hours creating many different kinds of models. They are doing "predictive modelling", irrespective of how many submissions were made. I'm speaking in generalities here. There have been competitions that were broken due to leakage or when the distribution of the test set is really different than the train set. I'm hoping this isn't the case with Bosch.
", "votes": null, "replies": [] }, { "id": 138304, "author_name": "willieliao", "author_url": "", "post_date": "10/07/2016 23:50:17", "content": "[quote=BreakfastPirate;138297]
\n\nThe public LB means nothing in the end. You get no points for being highly ranked on the public LB. You get no recognition for being highly ranked on the public LB. Nothing...
\n\n[/quote]
\n\nIt means nothing to people who understand Kaggle. But I've seen so many resumes in which people claim they've gotten in the top 10% in a competition. But a quick check shows that they only participated in 1 competition and fell thousands of places in the private leaderboard.
", "votes": null, "replies": [] }, { "id": 138305, "author_name": "gwillie", "author_url": "", "post_date": "10/07/2016 23:50:24", "content": "@Charly
\n\nI think you're getting two different arguments confused here. One is my original one, the other is one that came up subsequently.
\n\nRegarding the train/test split, I know that the split on the Red Hat competition was 0.82/0.18 to start, and that became at least 0.87/0.13, more like 0.9/0.1 after considering that 30% of the "test" set was actually training data. So we can squabble over the splits and whether that constitutes true predictive modeling or not, but I will stand by my statement that a 0.9/0.1 is certainly approaching the point where you're not doing true predictive modeling.
\n\nBut all of that is irrelevant to my initial point. That is, during the public LB portion, where there is supposedly "predictive modeling" going on, there is actually not. This is due to the fact that you are getting information about the outcomes of your "test" set. By definition, that then fails to be a test set and becomes a training set, and you cease to do predictive modeling. This last argument is not a matter of opinion - like the train/test split could be. This is a fact according to the definition of predictive modeling. You can disagree with a more subjective matter like the train/test splits, but I find it difficult to see how one can knowingly disagree with facts.
", "votes": null, "replies": [] }, { "id": 138308, "author_name": "xiaozhouwang", "author_url": "", "post_date": "10/08/2016 00:07:19", "content": "@GWillie,\nTake a closer look at how kaggle splits data or read one more time on what others said. Private LB is only revealed in the end of competition. \nSay you have 100K data points, now you split these 100K into 3 parts, the first one is training, second one is public LB, and third one is private LB.\nSo you train on training and get feedback from public LB, and then select best submissions based on everything you have and then hope for the best in private LB.
", "votes": null, "replies": [] }, { "id": 138314, "author_name": "aldente", "author_url": "", "post_date": "10/08/2016 00:45:39", "content": "@GWillie. The main purpose of public and private LB split is, "to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set." (https://www.kaggle.com/wiki/Leaderboard)
\n\nIn real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction.
\n\nKaggle uses "hidden private test set & leaderboard" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction.
\n\nThe only way to have good private LB score is to generalize the model, it's happened when your local CV score is consistent with public LB score. This is similar to the real world when you do cross validation.
\n\nSo, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case).
\n\nPublic test set (30%) and its LB table is an helper.
\n\n\n\n
And now, please someone kindly share hint of the golden feature ;))
", "votes": null, "replies": [] }, { "id": 138315, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 01:18:31", "content": "@Wilie Liao
\n\nThanks for your thoughtful response. Finally we have something we can use as a basis for real discussion.
\n\nI do think my definition of "predictive modeling" (which is based on the true definition) is out of line with the general usage on here. That's kinda my whole point; i.e., that term is being used inappropriately on here. And I am still confused about people's interpretation of how the public LB is being used by Kagglers. I'm hoping we can clear some of that up.
\n\nI saw Jared's response and used his statement of the true definition of "predictive modeling" to further support my position. Everything he said supported my claims, he just didn't realize it before posting it.
\n\nSo yes, let's consider the hypothetical examples:
\n\n"Let's say someone makes 2 submissions per day throughout the competition but always picks the last submission as his final model, regardless of leaderboard score. Based on your comments, I think you'd agree he did "predictive modelling" because he did not use any information on the leaderboard."
\n\nI would agree - if there were not an immediate assessment of his prediction in the form of a leaderboard score. As soon as that score pops up, it represents information on that "test" set. That test set ceases to be completely "unknown" in the sense that we now know how many right we got from our prediction. Therefore, the test set becomes a training set at that point (by definition) whether he uses it or not.
\n\n"Now, what happens if I tell you that he originally intended to always take the best 2 submissions but it just happens that he improved his models on a daily basis? Do you now invalidate everything he did and say he did not do "predictive modelling"?
\n\nAgain, same response as above. It would be predictive modeling if not for that information given on the leaderboard, whether he uses it or not.
\n\n"For the latter, the leaderboard is used differently by different groups of people. There's one kind that forks Kernels, makes small changes in the parameters or random seeds to see if they can climb the leaderboard. Most would agree with you that they are not doing "predictive modelling"."
\n\nExactly. And the point is, there is no guarantee that this latter case is not being used. We have no way of assuming that your more innocent first hypothetical is indeed the case.
\n\nHere, let me give you a hypothetical to try to clarify my point.
\n\nLet's say I ask you and 100 other people to play the game "I'm thinking of a number between 1 and 100, try to guess (predict) it. Whoever comes closest wins $50,000. And I'll give you 3 months to study it (me)." You're a smart guy, you're going to do whatever you can to increase your odds of winning. So you do your homework on me (age, address, zip code, birthday, birthday of kids, etc.). You put a lot of work into it, as you should, there's $50,000 on the line. Now, instead of just saying, at the end, I want all of you to guess and the closest wins, I say, you can make 2 guesses a day for 3 months. AND, there will be a guy on the side who is pointing thumbs up (higher) or thumbs down (lower) after you make each guess. But remember, you can only make 2 guesses per day (and each guess is assumed to be a true "prediction"). A few of the people get smart after they have each made a few guesses and team up, share information about their guesses, and work together from that point forward. And then we notice that there are some (especially that group that pooled their knowledge) who come much closer and closer to the true number as they make more and more guesses (and the guy on the side gives the thumbs up or thumbs down after each guess). Now, according to the participants, all of that is irrelevant - all that matters if their final guess. But some are saying, wait a minute, it seems fishy that there's this guy on the side giving feedback after each guess AND their guesses are getting better and better. Nooooo, you say, people aren't gonna actually use thumb guy information. Each guess is totally independent and in no way are they using the feedback from thumb guy on the side. I mean, that's just not right, and not in the spirit of the competition for them to be looking at thumb guy. To that I would say, seriously? There's $50,000 on the line and they have this source of information, and their guesses are getting better, yet they're not looking over at thumb guy after each guess? Seriously? So then, at the end, this group gets the number right on the nose. Dang, they're good. They just know how to do their homework and make really good educated guesses. What? That guy on the side, how could that have possibly played into it? To that I would again reiterate - seriously?
", "votes": null, "replies": [] }, { "id": 138316, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 01:23:12", "content": "@Little Boat
\n\nYou must have missed my post about the Red Hat competition. I have taken a good look at how kaggle splits data. In that competition, the training set represented 82% of the data (not 67%), public LB was 5% of the data (but more considering how much information it's giving), and private LB was 13% (but less considering that there is correlation between the 30% and 70% split in the "test" set). So in that one, you were training on about 90% of the data. That's approaching the limit of true predictive modeling.
", "votes": null, "replies": [] }, { "id": 138318, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 01:34:11", "content": "@aldente
\n\n"The main purpose of public and private LB split is, "to prevent people from winning by overfitting to the public leaderboard. Participants then are motivated to make sure their models will generalize well to the private leaderboard test set." (https://www.kaggle.com/wiki/Leaderboard)"
\n\nThis is exactly what you're not getting. When you are training on 90% of the data, there will not be a big drop between public and private LB. So the contest becomes one of EXACTLY overfitting to the public leaderboard. And when people attempt to apply these overfitting methods to true predictive modeling problems - where you are training on much less than 90% of the data - the methods do not translate very well to making good predictions because the thing that benefitted you most on Kaggle (overfitting) now hurts you the most. This is what we see over and over in the real world. And yet people keep trudging on as though that's not a real issue. Doesn't make sense to me.
\n\n"In real world of predictive modeling, as an example you predict amount of sales of next year. Because next year is not yet happened, you don't have the true value of the sales prediction.
\n\nKaggle uses "hidden private test set & leaderboard" to mimic that real world condition. We don't know the private LB score until the end, the same as in real world of ML we don't know the score of next year's sales prediction."
\n\nRight, but by giving feedback on the public LB about next year's numbers, you are getting information about the thing that you are supposedly "predicting." This, by definition, disqualifies it as being a "prediction." I'm not sure why people are having so much trouble seeing this. Seems pretty straightforward to me. I guess people just don't want to admit that what they're doing is not really what they thought they were doing.
\n\n"So, basically you do predictive modeling to the (hidden) private test set (70% of test set in this case)."
\n\nRight, and 10% of the total amount of data in the case of Red Hat. "Predicting" only 10% of the data is approaching the limit of being able to really call it "prediction."
\n\n"Public test set (30%) and its LB table is an helper."
\n\n90% of the data is simply a "helper?" That's a whole lot of "help."
", "votes": null, "replies": [] }, { "id": 138319, "author_name": "fredtony", "author_url": "", "post_date": "10/08/2016 01:46:19", "content": "@GWillie Your analogy is not particularly applicable. Extending your analogy, it is more like there are people who might be similar to you. You know their answers to a survey and the numbers they pick at the end. That's your training set. Then you have to build a model to guess the numbers a second group of other people pick knowing their answers to a survey. That is the public test set. This helps guide you to ensure your model is generalizable but that is not what you are predicting in the end. You get feedback which helps you to know if your generalization error is low. So in a sense, it helps train your model a bit but has no impact on your score because the true goal is to guess numbers for a final group of people. You get no feedback at all on that group and that is what you are predicting. Of course, you can use all of that info in the survey to help and find whatever correlations might exist to model the system, but this is true of all predictive modeling. You always start with your initial data (the survey) and make inferences using whatever clever methods you can find. But the point is, your model must generalize well to a set that you have no feedback on except you can train on a similar set and check if you are generalizing well on the public LB. if you overfit, you WILL NOT generalize well in your PREDICTIONS.
", "votes": null, "replies": [] }, { "id": 138322, "author_name": "mikeskim", "author_url": "", "post_date": "10/08/2016 02:34:13", "content": "import gc;\ngc.collect();
", "votes": null, "replies": [] }, { "id": 138324, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 02:37:11", "content": "Is it possible to overfit the model with features not involving the outcome ??? Overfit can happen if we dont stop the learning when evaluation metric starts degrading. But, is there possibility of overfit purely bucause of a feature that is not at all derived from the actual outcome !!!
", "votes": null, "replies": [] }, { "id": 138325, "author_name": "", "author_url": "", "post_date": "10/08/2016 02:47:31", "content": "I totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)
\n\nBut still, I haven't seen the magiccal rabbit in this thread =^.^!=
", "votes": null, "replies": [] }, { "id": 138328, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 03:03:02", "content": "[quote=Banana Kitty;138325]
\n\nI totally agree with @Mike Kim. This is a good exercise of memory management of gc() and to show that there are still old school techniques that can outperform modern software/hardware :)
\n\nBut still, I haven't seen the magiccal rabbit in this thread :D
\n\n[/quote]\nThe rabbit seems to hiding well deep in its hole... Its so strenuous to catch it, the one catching it wants to eat it all by themselves. Well, we caught the rabbit finally it seems, though lot of effort still to turn it into delicious meal.
\n\nAs many already mentioned, can't say for sure if its leak or not. But, I am in the opinion of whatever pattern we found being a leak caused by carelessness in data preparation (I may be wrong). Jumped Local CV score from 0.27 to 0.34 with a single feature.. I am sure about not overfitting the model, as the pattern is clearly visible from the data table, and would know it in a short while after submission.
\n\nThanks to 'raddar' for the quick 3 points. The first point did worked well. There was also a hint about it on the other thread; someone did mentioned about this particular data property although in a completely different context. I believe the guys who already cooked the rabbit could give a lot more info about it.
", "votes": null, "replies": [] }, { "id": 138329, "author_name": "tilii7", "author_url": "", "post_date": "10/08/2016 03:07:11", "content": "Smart people know things.
\n\nSmart people know when other people know things.
\n\nSmart people know about being right or wrong based on quality and quantity of the feedback received.
\n\nSmart people know when to quit.
", "votes": null, "replies": [] }, { "id": 138330, "author_name": "fredtony", "author_url": "", "post_date": "10/08/2016 03:12:51", "content": "I'm pretty sure that applies to both threads of this conversation.
", "votes": null, "replies": [] }, { "id": 138333, "author_name": "willieliao", "author_url": "", "post_date": "10/08/2016 04:01:18", "content": "[quote=GWillie;138254]
\n\nSo that also goes to my point. If he (they) weren't using public LB feedback to know they were making progress, how did they know to continue in the direction in which they were making progress. If they weren't using public LB feedback, then all submissions subsequent to his scores in the 0.27 range would essentially be a random shotgun approach, where they hope that one scores higher.
\n\n[/quote]
\n\n@GWillie, I'm going to assume you're not a troll and that you really don't know. The answer is cross validation. Again, my remarks are in the context of a normal competition, not a broken one like Redhat. I almost said "usual competition" but had to stop myself. :(
", "votes": null, "replies": [] }, { "id": 138340, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 04:57:11", "content": "@Tony S
\n\nMy analogy is applicable. You made the statement that it is not, but then failed to back that statement up (that's becoming a pattern here with those who respond to me).
\n\nAnd, you are also failing to see my point. In fact, you just provided support for my point without knowing it. My point is, and has been, that the public "test" set is not really a test set. It can be considered a hold-out set to be used in a cross-validation study, but it is not a test set in the sense that it is the ultimate set that we are predicting on. You even said, and I quote, "That is the public test set ... So in a sense, it helps train your model." Exactly, so we are in agreement, that public "test" set is actually a training set. Was that really so difficult?
", "votes": null, "replies": [] }, { "id": 138341, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 05:11:37", "content": "@Willie Liao
\n\nCross validation is a procedure where you hold out various portions of your data (not the data that you will ultimately be trying to predict) to test how well your model might do at making that final prediction. But the data being used in cross validation is data that you know the answer to. Therefore, it is not truly a test set, but is merely a "test" set for the purpose of cross validation. It is, therefore, being used to train your model for its ultimate task of predicting outcomes that are completely unknown to you. It is, therefore, by definition, a training set.
\n\nSo your argument is basically that the 30% of the "test" set used for the public leaderboard can be viewed as a "test" set because it can be viewed as a "test" set for cross validation. But in that cross validation, you could also hold out portions of the training set to be used as "test" sets. So, by your logic then, all of the training set would actually be a test set, and there would not really be a training set, which of course we know is not true. Thus, there is a contradiction to your argument.
\n\nThink of it this way - to reduce variability in a cross-validation study, multiple rounds of cross-validation are performed using different partitions (including data from the training set), and the validation results are averaged over the rounds. So these things you're calling "test" sets in a cross-validation study can be and actually are derived from the training set. How can they be true test sets if they come from the training set?
", "votes": null, "replies": [] }, { "id": 138343, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 05:16:08", "content": "@Tilii
\n\nCouldn't agree more ...
\n\nSmart people know about being right or wrong based on quality and quantity of the feedback received.
\n\nIn terms of the feedback received, I see a lot of quantity, but am still waiting for something of quality. Providing feedback that is irrelevant to the argument is not providing quality feedback. Smart people know that.
", "votes": null, "replies": [] }, { "id": 138346, "author_name": "fredtony", "author_url": "", "post_date": "10/08/2016 05:59:19", "content": "@gWillie I look forward to also seeing you crush the competition! It sounds like you have much to teach us and its probably going to be a cakewalk for someone of your abilities amongst mere mortals. Good luck!
", "votes": null, "replies": [] }, { "id": 138348, "author_name": "", "author_url": "", "post_date": "10/08/2016 06:10:10", "content": "After 7 pages the cat and rabbit are still being hidden
", "votes": null, "replies": [] }, { "id": 138350, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 06:30:06", "content": "@Tony S
\n\nThanks for the vote of confidence there big fella! If you were really doing predictive modeling here, I'd probably do pretty well. I have less interest in doing large-scale data fitting and data mining. But I have to say, given the money involved and the level of "expertise" here (basing that on the "brilliant" responses I've received), I should probably look into it.
", "votes": null, "replies": [] }, { "id": 138352, "author_name": "gwillie", "author_url": "", "post_date": "10/08/2016 06:32:37", "content": "More like, after hundreds of competitions, the cat and rabbit are still being hidden. But whatever we do, let's not try to find them.
", "votes": null, "replies": [] }, { "id": 138353, "author_name": "konradb", "author_url": "", "post_date": "10/08/2016 06:45:59", "content": "Does anybody know how to unsubscribe from a single thread in the forum? I am interested in most others, but the discussion on purity of predictive modelling is really getting annoying (because it's spamming my mailbox). 7 pages? Come on... Go have a flame war via PM or sth.
", "votes": null, "replies": [] }, { "id": 138354, "author_name": "gingerman", "author_url": "", "post_date": "10/08/2016 06:53:02", "content": "@KonradBanachewicz Clicking here should do the trick?
\n\n
FACEPALM: i was only checking the top of the page, didn't occur to me there can be buttons at the bottom for the same functionality. Thanks a lot @gingerman :-)
", "votes": null, "replies": [] }, { "id": 138380, "author_name": "salkaa", "author_url": "", "post_date": "10/08/2016 12:50:42", "content": "Just one simple question:
\n\nDoes this magic feature, is in form of some probability ?
", "votes": null, "replies": [] }, { "id": 138394, "author_name": "livingprogram", "author_url": "", "post_date": "10/08/2016 15:04:46", "content": "Keep searching for that magical feature, I made new thread concerning predictive modeling here
", "votes": null, "replies": [] }, { "id": 138424, "author_name": "mmueller", "author_url": "", "post_date": "10/08/2016 20:18:28", "content": "follow the white rabbit .. if you want to enter the 0.4+ club.
", "votes": null, "replies": [] }, { "id": 138426, "author_name": "kazanova", "author_url": "", "post_date": "10/08/2016 20:39:35", "content": "We have read this thread carefully since the first people hit the 0.4 threshold and irrespective of what people were saying or even swearing that this is not a leak, our mind was fixated in the fact that it was most likely a leak.
\n\nWe spent quite some time doing more traditional analysis/machine learning examination on this problem before realizing we cannot get there with just traditional approaches but we would have to look the data in a way that does not make sense for the client in order to get a better score and reach the top guys.
\n\nAfter doing so the leak was found.
\n\nThere are 2 type of competitors right now in my opinion:
\n\n- \n
- Those that are searching for leak as all the work they’ll do would\nbe futile unless they find it (in order to just build a useless\nmodel). \n
- Those that have found it and a) Build other features as well as the leak b) And try to exploit the leak even further (making more useless models, but probably getting a higher chance to win) \n
At least #2 implies some more ml hence it makes sense to release it - besides it is debatable whether the top scores that use the leak can add value to the real problem.
\n\nIn the past I had said that if we discover a leak and it is still fairly early in the competition, we will release it…so as to be a competition.
\n\nAs the current #2 team has indicated namely: Another Haunted and Leaked One, this dataset actually suffers from multiple leaks and is indeed very haunted…
\n\nHowever, most of the leakage (err …“data property”) so far can be extracted from the combination of sorting based on date time and then creating features based on the original sorting of the data (or just the Id).
\n\nFaron who spotted the leak shares it in this kernel.
\n\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.
\n\nI have to stress that in my opinion many of the comments in this thread have been misleading. It’s better not to say anything if it is to mislead...
", "votes": null, "replies": [] }, { "id": 138428, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 20:41:52", "content": "Yep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
", "votes": null, "replies": [] }, { "id": 138429, "author_name": "kazanova", "author_url": "", "post_date": "10/08/2016 20:44:37", "content": "[quote=PrakashSubedi;138428]\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition
\n\n[/quote]
\n\nAs we all...
\n\nTo be honest , you can probably get somehow there with subtracting the times too, not just the ids.
", "votes": null, "replies": [] }, { "id": 138431, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 20:53:18", "content": "In the first page of this thread, 'amaia' mentioned about failure occuring in pairs. No bigger hint could be to find this leak - RABBIT was there already !!!
", "votes": null, "replies": [] }, { "id": 138432, "author_name": "lishubin", "author_url": "", "post_date": "10/08/2016 20:56:45", "content": "thanks Faron for releasing the leak. i probably will never find it myself. I wish the magic feature was something else though.
", "votes": null, "replies": [] }, { "id": 138434, "author_name": "anokas", "author_url": "", "post_date": "10/08/2016 21:12:26", "content": "Adding these features to our model has not helped much, which leads me to believe that we modelled it using different features. Tomorrow I'll regenerate all our features with randomised IDs and row order, to see if our score holds.
", "votes": null, "replies": [] }, { "id": 138435, "author_name": "anokas", "author_url": "", "post_date": "10/08/2016 21:15:12", "content": "[quote=PrakashSubedi;138428]
\n\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
\n\n[/quote]
\n\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)
\n\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.
", "votes": null, "replies": [] }, { "id": 138438, "author_name": "rdslater", "author_url": "", "post_date": "10/08/2016 21:33:08", "content": "I should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.
\n\nkicking self
\n\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.
", "votes": null, "replies": [] }, { "id": 138439, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 21:35:53", "content": "[quote=anokas;138435]
\n\n[quote=PrakashSubedi;138428]
\n\nYep. I found the last two features (after sorting by date and Id) just yesterday, got LB of 0.38867.
\n\nI wonder how anokas earlier mentioned that 'ID' is not helpful in this competition. Well, its not the case. One More LEAKED Competition - back to back TalkingMobile (screwed in last week), RedHat (screwed in first week), and here we go BOSCH screwed as well !!!
\n\n[/quote]
\n\nI said that 'ID' wasn't helpful, because it didn't help me with the things I tried - however this is one thing that I didn't try. I guess I just wasn't smart enough to figure it out :)
\n\nIt didn't occur to me at the time, but now that I think about it this may be a side effect of the way that Bosch conducted stratified splitting on the dataset.
\n\n[/quote]\nUnderstandable !!! There could be other ways of exploiting this leak without looking at ID, and you were smart enough to spot that so early in this competition. But, now since we all know that features based on ID giving a heavy boost to the score, safe to say that - "yet another discouraging LEAKED Competition" !!!
", "votes": null, "replies": [] }, { "id": 138441, "author_name": "kazanova", "author_url": "", "post_date": "10/08/2016 21:37:27", "content": "[quote=Bob, Lord of Cats;138438]
\n\n@Μαριος -- You are a class act--I remember your team's write up from the BNP competition. We need a few more of you around to collaborate with.
\n\n[/quote]
\n\nThank you Bob, but it was Faron/Mathias who spotted that .
", "votes": null, "replies": [] }, { "id": 138442, "author_name": "samehif", "author_url": "", "post_date": "10/08/2016 21:39:08", "content": "anokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain.
", "votes": null, "replies": [] }, { "id": 138444, "author_name": "scirpus", "author_url": "", "post_date": "10/08/2016 21:46:39", "content": "I take anokas at his word. The problem would have been solved by a rules clarification. If you are angry make sure it is in the right place. Thanks to Faron - all aboard the ghost train l
", "votes": null, "replies": [] }, { "id": 138445, "author_name": "mmueller", "author_url": "", "post_date": "10/08/2016 21:49:59", "content": "[quote=Bob, Lord of Cats;138438]
\n\nI should have realize something was up when i accidentally left ID in a feature selection model and it turned out to be a very high relevance. Sometimes our mistakes hint and we don't listen.
\n\n[/quote]
\n\nDo not feel too bad about it, 'cause this can also happen with a feature made of random values (in cases where xgb has trouble to make proper splits on the real features).
\n\nA way to reveal the Id information here, is to calculate the AUC of binned id likelihoods, which yield ~0.55 .. clearly too high to be by chance.
", "votes": null, "replies": [] }, { "id": 138446, "author_name": "gullu39", "author_url": "", "post_date": "10/08/2016 21:50:38", "content": "[quote=Sameh Faidi;138442]
\n\nanokas: show the world your credibility and submit ~ 0.1 boost to your current score using the ID leak that your current model does not contain.
\n\n[/quote]
\n\nHope it meant to be a humor. It is possible to capture this leaked information unknowingly by some other means. In which case, just merely adding ID based features on top may not give a ~0.1 boost. But, if anokas gets another ~0.1 improvement, I am gonna go Mad .
", "votes": null, "replies": [] }, { "id": 138448, "author_name": "raddar", "author_url": "", "post_date": "10/08/2016 21:58:05", "content": "Im quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation) and now far0n has his version. all i can say, all 3 ways are surprisingly very different.
\n\nSo there are at least 3 ways how people see this data property/leak, and people downvoting him should feel silly or put some thought about themselves why they haven't seen something, when 3 different guys found 3 different solutions to it.
\n\nNonetheless, Good luck every1. Seems the race just have started.
", "votes": null, "replies": [] }, { "id": 138450, "author_name": "kazanova", "author_url": "", "post_date": "10/08/2016 22:01:13", "content": "[quote=raddar;138448]
\n\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation)
\n\n[/quote]
\n\nNow You are provoking me!
\n\nI had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to "there is no leak we swear" .
", "votes": null, "replies": [] }, { "id": 138451, "author_name": "anokas", "author_url": "", "post_date": "10/08/2016 22:02:12", "content": "@Scirpus @Prakash Thanks for not getting out the pitchforks just yet. Although don't hold your breath for a 0.6 in the next few hours.
\n\nPersonally, it feels really shitty to have several thousand lines of feature engineering, and then having an equivalent 20 line script pop up. But such is the way of Kaggle I guess.
\n\nNot going to sleep happy today.
", "votes": null, "replies": [] }, { "id": 138452, "author_name": "raddar", "author_url": "", "post_date": "10/08/2016 22:08:34", "content": "[quote=Μαριος Μιχαηλιδης KazAnova;138450]
\n\n[quote=raddar;138448]
\n\nIm quite surprised people downvoting @anokas. I do know they did not use ID's, although I myself used it (thats why our solutions had only 60% correlation)
\n\n[/quote]
\n\nNow You are provoking me!
\n\nI had said I will share this and I will stop . But here you admit you used the ID, but you did change your team name to "there is no leak we swear" .
\n\n[/quote]
\n\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.
\n\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.
", "votes": null, "replies": [] }, { "id": 138453, "author_name": "kazanova", "author_url": "", "post_date": "10/08/2016 22:11:22", "content": "[quote=raddar;138452]
\n\nI am not going to get involved into explaining this, as people are jumping to conclusions too soon here.
\n\n[/quote]
\n\nOk, lets save it for the end-of-the-competition thread.
", "votes": null, "replies": [] }, { "id": 138455, "author_name": "elenacuoco", "author_url": "", "post_date": "10/08/2016 22:18:18", "content": "So, what's next? The last 2 competitions I participated have some leaks, and people spent time in finding them. Reading this forum it seems that there were other competitions, where participants found leaks in the data. My question is: should we stop trying to build up a good model, or learn something from the raw data and just spend time in finding any leak in the construction of the train and test data? \nI like participating to these competitions since I love learning new things from other kagglers, study new solutions etc... \nI started this competition with the idea of using it to learn Spark and Spark ML. Spent time in writing a Scala pipeline ( I usually work in python) to link spark and xgboost for these competition to be able to work on a single pc with full data set, but my score (without any engineering of the features) was always below 0.25...\nNow, I will try to add these leak features, thanks to Faron and KazAnova, to the data as the others and check my score...But I'm wondering why do not use new rules for the competitions, in order to let the participants concentrated only on the models, the techniques, the data analysis and not on leaks hunting?
", "votes": null, "replies": [] }, { "id": 138456, "author_name": "laurae2", "author_url": "", "post_date": "10/08/2016 22:18:57", "content": "quick tl;dr
\n\nClearly in a tl;dr: there are four things that should hold true for Bosch:
\n\n- \n
- The date features are intentionally here \n
- The data was recorded in a very special way (there is a structure, look at the ID alone) \n
- The date features are (not all) leak and are useful for Bosch (if they intend to train/run models in near-real-time or in the past), as long as they don't overshoot the future using the past as structural changes may happen... very quickly \n
- If the problem for Bosch is to analyze past data, they are going to be very pleased (in a good way if they didn't change how the data was recorded) \n
Longer post
\n\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]
\n\nWhat we think is happening is that there are some “stops-and-restarts” (or lags) in production at times that are very predictive.
\n\n[/quote]
\n\nAssuming what I quoted from KazAnova applies right for Bosch... (it is right for most production environments, but I'm not working at Bosch so I can't know myself) - I'm not talking about the ID itself but more about the date features.
\n\nP.S: I wanted to make this post earlier to explain the reason why the date feature was included, but did not do it as "some stuff were visibly going out of control". I had also some discussions with my team members about why the date features was included and why it was normal it was there (and explaining why it was there).
\n\n\n\n
Gold standard stuff (the great ends bad)
\n\nI think from Bosch this "leak" is not intentional, more a cause of what they thought was good (if I assume they follow the gold standard for production quality management...).
\n\nRationale: if you follow the gold standards in production quality assessment, you must do the following:
\n\n- \n
- Pick production samples to assess for quality during a specific timeframe \n
- Assess the quality of these production samples \n
- Assume that if the defect quality quota is not reached, it generalizes to all samples (otherwise, you either try again with new samples - if it is still reached, then you must work on quality to improve your production "rank" in the official production quality tables if you do comparisons with them) \n
- And then you take a break (to minimize costs), you do it again, then a break, then again... you do that over and over... (thinking naively it generalizes - if you had a bad production batch then you know you will not pass the quality tests, big emphasis: ____and you probably know why____ (if you don't, then ___you should try to know___ at least...)...) \n
You can also do blind testing: two production samples are picked instead of one, with the same timeframe. This is what happens there (or what is intended to be replicated - in fact, it may be just a single production sample split in two as seeing the ID alone seems to depict).
\n\nNot so gold...
\n\nHence, what happens is you pick off samples from a timeframe if you want to generalize to a whole population (whether the date is before or after). This is the production "gold standard" (not so "gold" as we can see here) to assess the quality of your production (in a global way). However, that gold standard.. was from centuries ago (if not a millennium already!) where you had no real powerful machine learning available along with all the data. This should be also the reason we were provided the time data (some people wondered why we were given the data).
\n\nStructural vs timeframes
\n\nNow that ML and data are existing to be used, it's possible to catch how the structural production works (but then - you know only how the production works and how it fails at specific points, not how it will generalize). In the past, no one would have ever challenged a quality manager by doing that (because they never had the data for).
\n\nAnother usage than predicting the future in exact real-time
\n\nThere is also a small trend in the date features (alone), the higher the lag the worse the quality becomes. This is more a production quality issue, and I think by the time the product reaches the end, higher odds it should have been removed at least once by a worker for extra inspection during the flow... Such feature cannot hold generalization if the target is to have a fully automated production (therefore, it is up to what Bosch is going to do with the models/scripts, and what is the problem they are trying to solve).
\n\nHowever, it could be a good one for detecting bad stations before they are tagged "bad" when inspected manually (there are some cycles there you can find on the dataset too): if there is a structural change (specifically: dates), then something must have changed in at least one station (or on the pathing). The closer to real-time they train the models and run them, the faster they can detect defective stations (some stations are dishing out tons of defects at specific times in this dataset, clearly normal from a production environment).
\n\nBack to leakage...
\n\nTherefore, if assuming what I quoted is true for Bosch, some "leaks" are useful (and may become deprecated). But as I do not know all the "leaks" in this dataset, it is fairly possible some are entirely useless for a real case. Overall, it depends on what Bosch expects to do with the model (and they probably will need a "production specialist" of their own production before even trying to put such model in their production - to clear up all real leakage and keep the "good leakage").
\n\nThis quote should be added on what is leakage and what to do with it, as it sums up what to do to assess a leak in a model in a real environment...:
\n\n[quote=Μαριος Μιχαηλιδης KazAnova;138426]
\n\nit is debatable whether the top scores that use the leak can add value to the real problem.
\n\n[/quote]
\n\n... because it always depends on what leak is used and what problem is targeted (so far we don't know the real problem Bosch is trying to solve [not "lowering costs", that's the objective] - or I'm blind). After all, it is all about what "leak" for us in this competition. For the "problem" part, Bosch should know better than us what they are trying to do.
", "votes": null, "replies": [] }, { "id": 138457, "author_name": "breakfastpirate", "author_url": "", "post_date": "10/08/2016 22:20:46", "content": "[quote=raddar;138452]
\n\nThat is why i still think its not a leak, but how the data is recorded, and the problem itself might be very relevant to Bosch. Having more insight into it, I am quite certain that these findings are very useful for Bosch, and still ML part is even more relevant.
\n\n[/quote]
\n\nThe data page says "Each part has a unique Id" and if this is true, it makes me really wonder how the magic feature(s) come about. That is what made me wonder whether it was really a leak or not.
", "votes": null, "replies": [] }, { "id": 138458, "author_name": "aldente", "author_url": "", "post_date": "10/08/2016 22:25:05", "content": "Please let me know, how to express my gratitude to those sharing their hardwork for benefit of others?
", "votes": null, "replies": [] }, { "id": 138459, "author_name": "mmueller", "author_url": "", "post_date": "10/08/2016 22:32:03", "content": "I see it this way: As soon as anything else than the provided features yield predictive power, I label it leakage. If the id-structure is intended information they should call it Order Feature (or because we like to make things anonym Beqre Srngher) and give the rows real random ids.
", "votes": null, "replies": [] }, { "id": 138460, "author_name": "raddar", "author_url": "", "post_date": "10/08/2016 22:42:44", "content": "Imagine a situation, that Bosch truly believe that id's are random... does this insight alone bring value to their production/data recording pipeline? Put some thought about it:) Yes, we might call it a leak as we are used to call everything like that as unwanted data property. But does this help solve their problem to detect defects? if such hiccups in a system are recorded and it is a reason of higher defect rate, this might have golden value itself. And still ML has to help figuring out which hiccups are normal and not, hence the relevance of ML is important.
\n\nSo other question is, how much do You believe that above scenario is likely; then its just a personal opinion if it is a leak or a data property.
", "votes": null, "replies": [] }, { "id": 138462, "author_name": "jiweiliu", "author_url": "", "post_date": "10/08/2016 23:00:20", "content": "[quote=raddar;138460]
\n\nBut does this help solve their problem to detect defects?
\n\n[/quote]
\n\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. "Response" is leaked in adjacent rows because when they found a failed part, they do something in response immediately.
\n\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure.
\n\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P
", "votes": null, "replies": [] }, { "id": 138463, "author_name": "raddar", "author_url": "", "post_date": "10/08/2016 23:06:41", "content": "To be honest, I actually stumbled upon to very very similar thing in one of projects (which was so silly and so many analysis have been done on that), so nothing can shock me when such kind of things appear :)
\n\nanyway, rabbit is out & have fun.
", "votes": null, "replies": [] }, { "id": 138464, "author_name": "jiweiliu", "author_url": "", "post_date": "10/08/2016 23:09:13", "content": "But I also want to say we should not criticize raddar and his teammates. They do have a misleading team name but so do we :P (out computer actually has 1000 GB memory not 1TB)
\n\nActually they have given a lot of hints and valuable advice along the way. When I read them, I immediately realize they are talking about leak in my definition and "special data property" in their words. So i want to thank them for sharing.
", "votes": null, "replies": [] }, { "id": 138465, "author_name": "clockworkworld", "author_url": "", "post_date": "10/08/2016 23:17:52", "content": "I put some thought into it this afternoon and I'm extremely hesitant call this a leak. At facilities I've worked in the past the ID is not random, but generated based on timing and style of the product. It's almost certain there is some order information encoded in the ID but more importantly the class of product should be present as well. If there wasn't important information here why include Id in the dataset at all?
\n\nThe first two features in Faron's kernel were not significant features when I added them to my model, however the second two, based on the differences in consecutive ID's when sorted by start time, became my top features.
\n\nFor lack of a better hypothesis, I think these two features might be capturing transitions between product styles.
", "votes": null, "replies": [] }, { "id": 138467, "author_name": "clockworkworld", "author_url": "", "post_date": "10/08/2016 23:45:32", "content": "[quote=rcarson;138462]
\n\n[quote=raddar;138460]
\n\nBut does this help solve their problem to detect defects?
\n\n[/quote]
\n\nAs a shameless leak exploiter, i have to say no. The leak doesn't help detecting defects. Bosch carelessly keep the original sample orders. "Response" is leaked in adjacent rows because when they found a failed part, they do something in response immediately.
\n\nAs i said previously, this helps Bosch realize the problem they have in data preparation but not detect failure.
\n\nDespite that, the leak helps nothing but our scores. Come on, do you really believe magic feature in real life? I doubt very much. :P
\n\n[/quote]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
", "votes": null, "replies": [] }, { "id": 138468, "author_name": "jiweiliu", "author_url": "", "post_date": "10/09/2016 00:14:40", "content": "[quote=ClockWorkWorld;138467]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
\n\n[/quote]
\n\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.
", "votes": null, "replies": [] }, { "id": 138469, "author_name": "clockworkworld", "author_url": "", "post_date": "10/09/2016 00:50:26", "content": "[quote=rcarson;138468]
\n\n[quote=ClockWorkWorld;138467]
\n\nThere's no way Bosch is identifying and responding to failed parts at a timescale comparable to the time between adjacent rows... And there are certainly situations which would require the shutdown of entire machines to correct.
\n\n[/quote]
\n\nI'm making these claims only based on what i see in the data. If you have more insight and domain knowledge why this is what it is, please let us know.
\n\n[/quote]
\n\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.
", "votes": null, "replies": [] }, { "id": 138472, "author_name": "happycube", "author_url": "", "post_date": "10/09/2016 01:44:58", "content": "My model isn't finished yet (and will probably need a few passes anyhow) but I think .4 is achievable by just using features 3 and 4 from Faron's script (mindate<->id, I think 1+2 are linenum<->id) Since there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.
\n\nEDIT: Finished my very first run and it got .408 public LB. Aside from the mindate<->id features it got in the .27 range.
\n\nEDIT2: Local testing indicates that processing the feature with train+test not combined is much less effective - probably around .32. I'm not going to burn a submission figuring out what it gets on the LB tho. ;)
\n\nIf Id can be treated as a legitimate feature, and the feature still works by doing the diffs with train+test separated, it isn't a leak. However if there's a lift from row #<->id that can't be replicated elsewise, it's a leak - likewise if it requires a combined train+test set it might run afoul of the 'time machine rule(sic)'
\n\nThanks for sharing Faron!
", "votes": null, "replies": [] }, { "id": 138474, "author_name": "mmueller", "author_url": "", "post_date": "10/09/2016 02:08:28", "content": "[quote=happycube;138472]
\n\nSince there are many different Id's sharing a minimum date, I find the values generated a bit odd on those, but xgboost seems to like the features a lot.
\n\n[/quote]
\n\nOne reason why it works is because there are often rows x, y with abs(id(x) - id(y) ) = 1 for a given "StartTime". Besides, they are often duplicates of each other.
", "votes": null, "replies": [] }, { "id": 138485, "author_name": "jeffhebert", "author_url": "", "post_date": "10/09/2016 04:50:15", "content": "[quote=ClockWorkWorld;138469]
\n\nI'm still not convinced it isn't a leak, however I don't think it's related to the reason you suggest. It appears that the final pass/fail judgement is not made until after the part has exited the assembly line. For example, if a change at station 0 was the source of the failures, there will be a significant lag between when the change was made and when it is finally detected.
\n\n[/quote]
\n\nI agree with this assessment, but I don't think this represents a leak. When a part is 'created' Bosch does not know if it will be good or bad. We don't know for sure until it reaches the end of the production line. It's course through the line (station sequence and timestamp) has a strong influence on the final quality.
\n\nBesides, we don't know about their manufacturing process. Bosch may start batches of similar parts at the same time. This is a common manufacturing procedure which can reduce the amount of production time lost when changing a station setup from one part to another. Bosch may start 20 items of part A which has a high failure rate, and then 20 items of part B which has a normal failure rate.
", "votes": null, "replies": [] }, { "id": 138487, "author_name": "happycube", "author_url": "", "post_date": "10/09/2016 05:34:33", "content": "It looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
", "votes": null, "replies": [] }, { "id": 138489, "author_name": "samehif", "author_url": "", "post_date": "10/09/2016 05:51:23", "content": "Would the leak have been avoided if the data was split by time instead of Id?
", "votes": null, "replies": [] }, { "id": 138490, "author_name": "happycube", "author_url": "", "post_date": "10/09/2016 05:52:30", "content": "I don't know, but randomizing Id would have made it harder to find...
", "votes": null, "replies": [] }, { "id": 138494, "author_name": "barisumog", "author_url": "", "post_date": "10/09/2016 07:25:54", "content": "IDs are not features. You can argue all you want. To me, any feature using IDs is a leak.
\n\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.
\n\nI hope the forums don't become a place where people pull heists, and get upvotes all around for it.
", "votes": null, "replies": [] }, { "id": 138503, "author_name": "mmueller", "author_url": "", "post_date": "10/09/2016 09:07:01", "content": "[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nThe best part is, that thanks to Chippy it is known for weeks actually: Sequential duplicates in test files? This thread really diserved more attention. :))
", "votes": null, "replies": [] }, { "id": 138507, "author_name": "anokas", "author_url": "", "post_date": "10/09/2016 09:20:01", "content": "[quote=Sameh Faidi;138489]
\n\nWould the leak have been avoided if the data was split by time instead of Id?
\n\n[/quote]
\n\nI think that if Bosch was intent on predicting the future then they would have split the data based on time. It's more likely that, like others have said, that they want to inspect say a random 10% of products in a production batch by hand, and then by building a model on that they are able to detect possible faults in the other 90% across the same time period, while at the moment they might have to inspect all 100% of the products that come off the production line.
\n\nI thik about it like this - If Bosch really wanted a model that could predict the future, then literally just running XGBoost on the provided data would make a useless model, since the date data has been provided in absolute time format, and that would mean that future data would fall into the wrong branches when running the model on a set with different date info.
\n\nI would have expected them to split the data based on time if they wanted to predict the future, or given us features that are time invariant such as 'time taken at station 2'
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
", "votes": null, "replies": [] }, { "id": 138508, "author_name": "mmueller", "author_url": "", "post_date": "10/09/2016 09:22:24", "content": "[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
", "votes": null, "replies": [] }, { "id": 138509, "author_name": "malten", "author_url": "", "post_date": "10/09/2016 09:26:41", "content": "I thought about the nature of why this feature works for about a week now and my explanation is the following. If they see a bad part, they will also check the next product, which somehow determines the order in the ids. \nThis also means, that a model based on this feature only helps to identify the broken products, which bosch was already able to find themselfes. If they wont find it themselfes this feature doesnt work. Based on that i dont think this feature has any worth for them and is just a leak.
", "votes": null, "replies": [] }, { "id": 138511, "author_name": "anokas", "author_url": "", "post_date": "10/09/2016 09:30:14", "content": "[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
", "votes": null, "replies": [] }, { "id": 138512, "author_name": "malten", "author_url": "", "post_date": "10/09/2016 09:43:01", "content": "[quote=anokas;138511]
\n\n[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
\n\n[/quote]\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.
", "votes": null, "replies": [] }, { "id": 138517, "author_name": "nigelcarpenter", "author_url": "", "post_date": "10/09/2016 09:49:40", "content": "[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nI was suspicious of the data when I found too many sequential duplicates a month ago (link to post). They shouldn't have existed if the Id's were random and the row order had been shuffled.
\n\nThis annoyed me because it is often a sign of bad data preparation and precursor to leak identification and exploitation. Roll forward a month and the "leak" is out and it utilises Id's and roworders.
\n\nSo proof of another bricked competition?
\n\nWell I'm not so sure. The nature of this problem is a little different. The posts of those with domain expertise (JeffH and others) give insight into how use of sequence information may be valid and useful to Bosch.
\n\nHowever I am sure that in the current competition format leaks or the the suspicion of leaks are damaging. Damaging for
\n\n- \n
- us as competitors (I want to establish and demonstrate my predictive analytics, not plumbing skills) \n
- competition hosts (they don't get the predictive model they hoped for) \n
- Kaggle (leaks alienate competitors and hosts and don't deliver predictive analytics) \n
That's a real shame because the principle upon which Kaggle was created is sound and beneficial to society at large.
\n\nNow I believe that leaks will always exist as long as fallible humans are involved in data setup. So as much as we don't like leaks they are not going away. We need to find a way of turning leaks into something that benefits the Kaggle format.
\n\nSo something needs to be done
\n\nIf you agree +1 this post to demonstrate to Kaggle admins our collective strength of feeling.
\n\nAlso contribute your ideas and thoughts on how we can turn leak identification into something beneficial to our community.
\n\nI'll create a new thread to collate ideas and post a link to it here
\n\n- \n
- edited to add link to ideas to improve thread \n
[quote=Malte Nalenz;138512]
\n\n[quote=anokas;138511]
\n\n[quote=Faron;138508]
\n\n[quote=anokas;138507]
\n\nAlso, I'm now almost certain that the data properties of the leak is exploitable through some other features that don't use ID, but I'll wait until the end of the competition before arguing about that.
\n\n[/quote]
\n\nI 2nd that. :))
\n\n[/quote]
\n\nBefore we added any ID/row order features, we were at 0.485, so it's definitely possible to do well without using that leak :) - They added 0.01 to our model.
\n\n[/quote]\nBut did you use the order of the observations? this is basically the same. I also found it by looking at the starting times.
\n\n[/quote]
\n\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.
", "votes": null, "replies": [] }, { "id": 138521, "author_name": "aldente", "author_url": "", "post_date": "10/09/2016 09:55:46", "content": "[quote=Faron;138503]
\n\n[quote=happycube;138487]
\n\nIt looks like there are duplicates even between train+test with only Id different by one. This probably explains why the top groups caught the rabbit a few different ways ;)
\n\nI really hope that is intentional, somehow...
\n\n[/quote]
\n\nThe best part is, that thanks to Chippy it is known for weeks actually: Sequential duplicates in test files? This thread really diserved more attention. :))
\n\n[/quote]
\n\nI've detected all of them sometimes ago, rows with exactly the same values between train-test (except Id) = 1772 rows having Response '1', and 75504 rows having Response '0'.
\n\nSame rows with Response both '1' and '0' are omitted.
\n\nI've made submission based on only those 1772 '1's, if they're all correct then score should be 0.5+ with 1772 true positives, but in fact I only got around 0.18 public LB score.
\n\nI guess, (1) the 1772 same rows are NOT distributed proportionally between public/private (30/70%), or (2) they have false positives.
\n\nSo, I decided to not use them in my submission based on Santander experience.
\n\nIF it's indeed another leakage, then I'm just too dumb because I don't know how to use it ;)) Any insights? Thank you.
", "votes": null, "replies": [] }, { "id": 138548, "author_name": "titericz", "author_url": "", "post_date": "10/09/2016 14:02:28", "content": "There are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.
\n\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those "duplicates". Why it occurs? Kaggle and Bosch can tell us.
\n\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...
\n\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training.
", "votes": null, "replies": [] }, { "id": 138549, "author_name": "scirpus", "author_url": "", "post_date": "10/09/2016 14:15:53", "content": "Gilberto great post - I set up a thread on the forums called leak soup - I would love the best of the best to give Kaggle tips to reduce leaks - after all you Grand Masters are so good at finding them!
", "votes": null, "replies": [] }, { "id": 138551, "author_name": "malten", "author_url": "", "post_date": "10/09/2016 14:27:23", "content": "[quote=anokas;138520]
\n\nYou mean the date features? We did use that in other features for the 0.485, but not in our superfeatures. And row order of observations was not used either in any features.
\n\n[/quote]
\n\nyes, i meant the roworder of the observations. thank you for clarifying!
", "votes": null, "replies": [] }, { "id": 138557, "author_name": "samehif", "author_url": "", "post_date": "10/09/2016 15:08:28", "content": "Gilberto: I counted only the exact matches and it is up to 40% of all 1 responses are in these pairs of identical rows. To my surprise also a good amount had different response (sequential rows with 100 % identical values but one is faulty and the other is fine).
", "votes": null, "replies": [] }, { "id": 138574, "author_name": "gwillie", "author_url": "", "post_date": "10/09/2016 16:13:08", "content": "So let me see if I have this straight. I said that it appeared as though very little, if any, true predictive modeling was going on in this competition (and others like it), and that was doing a disservice to both the companies who thought they were getting predictive models and the competitors who thought they were doing predictive modeling (because that what Kaggle bills it as). MANY people jumped down my throat, offering different reasons as to why I was wrong and that I didn't know what I was talking about. They offered reasons like "cross validation" to try to explain what was happening here. Then, in the end, it turns out I was right and they were wrong.
\n\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!
", "votes": null, "replies": [] }, { "id": 138577, "author_name": "clockworkworld", "author_url": "", "post_date": "10/09/2016 16:17:15", "content": "[quote=Gilberto Titericz Junior;138548]
\n\nThere are many ways to explore the leak on this dataset. Using Id or time sort is only one way to explore it. But you can do many feature engineering that explores the leak: lead/lag, histogram, etc even without notice what the leak is.
\n\nBasically some rows have a duplicated (or almost duplicated) pair right next it. And almost all positives targets of the dataset are located on those "duplicates". Why it occurs? Kaggle and Bosch can tell us.
\n\nSo Kaggle must do Id randomization and sort in the future, because there are 3 competitions in a row that solutions are destroyed by that simple approach that can be avoided very easy. Also Bosch (or Kaggle) could have normalized timestamps on dataset so all times will start on 0. Also even doing that there is another source of this leak...
\n\nWithout using the leak (inter duplicated rows information), taking into account row as i.i.d. and using only numerical/categorical and normalized timestamps by row you hardly can beat 0.30, even ensembling many models (and working hard) up to the level 3 of training.
\n\n[/quote]
\n\nI was able to achieve 0.297 before Faron shared his kernel using a sloppy xgboost (without any ensembling, model tuning, or cross validation) with the addition of a very simple set of physically relevant hand generated features. The best modelling will not beat the best feature engineering here.
\n\nThe data here is incredibly feature rich and it's frustrating to see so many experienced participants jumping to the conclusion there must be a leak and that the results will be useless for Bosch. I hate to be that guy but All Models Are Wrong.
\n\nWe need more information on how the Id's are generated to know if they are leaking information from the test set.
\n\nThere are two scenarios I can think of:
\n\n- \n
The Id's have too much information related to the train/test split. In this case, why is Id left in the dataset, wouldn't the automatic row numbering be good enough?
\nThe Id's are machine generated when the product enters the assembly line (perhaps bosch may have subtracted the minimum original value it from the rest so we start at zero). If this is the case then the Id will probably contain information about the type of part (summarizing the equipment path) and the order. The "duplicate" parts you are seeing are actually not duplicates at all but different parts produced under nominally identical conditions because they were processed one after the other. I think the Id feature effectively captures some nonlinear information about transitions to and from certain styles of part.
\n
A final point to discuss is the choice to split the train and test data from within the same time period. Obviously this makes "prediction" a somewhat abstract activity. Consider that absolute prediction may not be the goal. If I was at Bosch I would use these results to upgrade my quality control sampling system to minimize the cost of testing (materials, inventory, customer lead time) as well as to identify areas of process improvement. Ultimately an engineer may not be as concerned with the ability to predict failures part to part and are probably more interested in driving the overall rate of failures down.
\n\nAt the risk of giving up any advantage that come with domain experience, I'll be happy to weigh in on any leak hypothesis.
", "votes": null, "replies": [] }, { "id": 138581, "author_name": "kazanova", "author_url": "", "post_date": "10/09/2016 16:32:13", "content": "[quote=GWillie;138574]
\n\nMaybe in the future, Kagglers will listen to people like me who are trying to help them in their pursuit of becoming predictive modelers. I will most likely not be here, since I will be off doing real predictive modeling, and this is obviously not the platform in which to hone those skills. And keep in mind, on a resume, Kaggler != Predictive Modeler. But good luck Kagglers - Kaggle on!
\n\n[/quote]
\n\n@GWillie, I am not very happy that this comp is leaky and generally 3 out 3 competitions I participated as of late had a leak which is frustrating .
\n\nHowever I have to admit that I have picked up many of my predictive modelling skills in this platform (kaggle) - and they have been very useful (and very tested) in corporate environments as well. So I am keen to disagree on this "I will most likely not be here" . If you joined lately, I can see why you might feel this way, but many innovative solutions and good models have been produced here.
\n\nKaggle is still for me the best place to learn predictive modelling and build up your resume in data science.
\n\nKaggle experience does count on your resume for a data science position- at least where I work for!
", "votes": null, "replies": [] }, { "id": 138582, "author_name": "gwillie", "author_url": "", "post_date": "10/09/2016 16:43:10", "content": "But that's just it, exactly my point. People like you think they are picking up predictive modeling skills, but with competitions like this, you are not. You are picking up data analysis, data mining, large-scale computation/memory/storage skills, and thus "data science" skills (particularly "big data"), but not necessarily predictive modeling skills. And therein lies the real danger. The field of predictive modeling is being held back by things like this where everyone thinks they're doing true predictive modeling and thus we focus our efforts here instead of other pursuits where we might actually be doing true predictive modeling. This is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.
\n\nI do agree there is value to companies in what is being done here, but we need to be very careful in stating exactly what that is and what it is not. From what I see, it is certainly not predictive modeling value.
", "votes": null, "replies": [] }, { "id": 138586, "author_name": "gullu39", "author_url": "", "post_date": "10/09/2016 17:01:51", "content": "Come on please Don't Do it Again... There is separate thread to discuss the meaning and relevance of Predictive Modelling. Please stick to the discussion here on Magical feature or leak or data property whatever it is called.
", "votes": null, "replies": [] }, { "id": 138588, "author_name": "rdslater", "author_url": "", "post_date": "10/09/2016 17:05:18", "content": "[quote=barisumog;138494]
\n\nIDs are not features. You can argue all you want. To me, any feature using IDs is a leak.
\n\nYes, it has happened many time before. But the people who wanted to protect their secrets at least had the decency to shut up, instead of posting misleading info.
\n\nI hope the forums don't become a place where people pull heists, and get upvotes all around for it.
\n\n[/quote]
\n\nVery correct sir. I may not be an expert at statistics or machine learning, but I am an expert in manufacturing having spent several years with the likes of Intel, Texas Instruments and other large companies. The ID is nothing more than the id. At best it is a sequential number/combination of when the part was started. If you cannot measure in your process to predict a bad part, but your part start sequence matters--you don't need machine learning, you need basic process improvement.
\n\nFor example, one of my processes had a time window. Basically from the start of the previous step to the start of my step, we had X number of hours or we knew the part would be bad.
\n\nOr we had burn-ins--the first part through we be bad, so we knew we had to "Season" the machine with a test part.
\n\nBut the point is, we could measure those with an in-line process. Aka the resistance after step 62 was too high, or the thickness after step 71 was too low.
\n\nIn 10 years of manufacturing, ID did not matter. Ever. And I've seen some weird 4-level failures/interactions. When I did my experimental splits, we didn't care about the part number, we cared about what split (experiment portion) was on the part.
\n\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a "Tag" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people.
\n\nSo unless you assign "Steve" to overly aggressive drivers, the tag should not matter.
", "votes": null, "replies": [] }, { "id": 138589, "author_name": "kazanova", "author_url": "", "post_date": "10/09/2016 17:08:19", "content": "You still need to understand the data, cross-validate, test against a metric, perform feature engineering and feature selection, build multiple models, do ensembling etc . The problem is the data has information it should not have .
\n\nFrom the competitors' point of view, there is value (IMHO).
\n\nFrom the companies' point of view not much as a solution , but they may still leverage the benefits of knowing what is new in the market, what the top approaches were , which features performed best (possibly outside the leakage) and they can understand the theoretical best you could get from this data set
\n\nI think the competitors lose if kaggle starts receiving the reputation that they don't solve the client's problems efficiently (which is my primary concern with the leaks) and the fact that you need less predictive modelling skills and more data exploration ones .
\n\n[quote=GWillie;138582]
\n\nThis is evident in the fact that frustration still exists over the lack of general progress in the field of predictive modeling. We should be smart enough to identify the reasons behind the lack of progress.
\n\n[/quote]
\n\nI have not seen lack of progression in predictive modelling (the contrary) - what do you mean? There seems to be huge focus in the field right now. I can see that as PhD student, Professional, kaggler , one_that_uses_google etc!
\n\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?
", "votes": null, "replies": [] }, { "id": 138591, "author_name": "ascending", "author_url": "", "post_date": "10/09/2016 17:18:31", "content": "[quote=Μαριος Μιχαηλιδης KazAnova;138589]
\n\nIn any case how do you propose to tackle the problem? I am sure kaggle did not want leakage in the first place! How should we get predictive modelling back on track (according to your suggestion) ?
\n\n[/quote]\n A checklist for avoiding leak?
", "votes": null, "replies": [] }, { "id": 138592, "author_name": "abriosi", "author_url": "", "post_date": "10/09/2016 17:22:15", "content": "EDIT:Deleted
", "votes": null, "replies": [] }, { "id": 138702, "author_name": "clockworkworld", "author_url": "", "post_date": "10/10/2016 13:57:05", "content": "[quote=Bob, Lord of Cats;138588]
\n\nFrom manufacturing experience, it's a leak. ID is not a measurement, it is a tag. If the assignment of a "Tag" gives you information, that is in no way predictive modeling, because your information comes from a meaningless tag. It would be like predicting who will have a car accident based on names you assign people.
\n\nSo unless you assign "Steve" to overly aggressive drivers, the tag should not matter.
\n\n[/quote]
\n\nOne problem with this statement is that in the case you describe name is an Independent Identically Distributed Random Variable, whereas ID is neither independent, identically distributed, nor truly random.
\n\nConsider the following hypothetical scheme for internal IDs:
\n\nEach Id is made up of 4 numbers xx,yy, tt, and zz: XXYYTTZZ
\n\n- \n
- XX represents the customer, \n
- YY represents the family of product \n
- TT represents the batch number \n
- ZZ represents the part # within the particular batch. \n
In this situation differencing the ID in adjecent rows can give you significant amounts of information. Just because we weren't clever enough to spot it on our own doesn't mean it's not there.
\n\nFor our dataset, the bottom line is knowing whether the ID feature captures information about how the stratified splitting was performed (leak), or if the ID's are machine generated (potentially not a leak).
", "votes": null, "replies": [] }, { "id": 138704, "author_name": "rdslater", "author_url": "", "post_date": "10/10/2016 14:22:35", "content": "Except of course in my experience that is NOT how part numbering worked. Parts were sequential from when they started. But even at the first manufacturing step, they were already out of order. By the end of the line it was an extreme jumble. Product ID, and in particular customer details were not stored as part of the part ID. It's like a key in a 3NF database. If the lookup key is part of your model, we'll you have a problem.
\n\nThe additional important point is that the true measurement of performance difference was contained in data elsewhere.
\n\nAnd let's have a reality check. The id's here are integers starting at 1, not some fancy labeling scheme, and with specific time data included. If you could build your data on date alone--that's fine. But because you need date and ID, it's a leak.
", "votes": null, "replies": [] }, { "id": 138711, "author_name": "laurae2", "author_url": "", "post_date": "10/10/2016 15:10:35", "content": "[quote=Bob, Lord of Cats;138704]
\n\nAnd let's have a reality check. The id's here are integers starting at 1, not some fancy labeling scheme, and with specific time data included. If you could build your data on date alone--that's fine. But because you need date and ID, it's a leak.
\n\n[/quote]
\n\nI suppose here Bosch follows ISO 9003 (which they probably do - or the more recent ISO 9001 which is the old 9001-9003 put together and refined).
\n\nISO 9001 is about quality management systems and is a top-down approach. According to ISO 9001, the most probable position we are in the production workflow is the following, with the sampling exception rule used:
\n\n
When you compare two samples to assess the quality of your production:
\n\n- \n
- They must have the same time frame from the same population \n
- ID may provide information depending on how the data was sampled (hence, it's a feature alone already) \n
And ID can start by whatever number you want, for instance here they decided it should start by 1 when they provided the data sets for us. To perform a validation on a machine learning model for this specific quality management setup, you need to turn this into a blind experiment which follows the same exact conditions, and to pass at least once "the minimal quality production" (the confusion matrix % passes, or any other quantitative metric) from your specifications.
\n\nAs part of ISO 9003 (now deprecated and included as part of ISO 9001), you need to perform calibration of your model to perform a validation. As long as it works in one sample, ISO says it generalizes to all your samples. The only thing you need to, is re-calibration when you change samples, more exactly re-training your model when you change population (= change time frame). If you are not calibrating, you are not conforming to ISO 9001/9003. The "regular basis" in your specifications define the frequency of calibration (part 4.13 in ISO 9003, called the "Exception 4.12" when you must sample to work appropriately).
\n\nThis is how it works when you are performing quality assessment from the high level. At the bottom level (where we are), we create the script which will train a model and validate on another sample, where train+test are from the same population. Assuming "our winner scripts" pass the quality test at Bosch, this script theoretically generalizes to new samples (obviously, you re-train your model when you change the population).
\n\nLeak or no leak, it doesn't matter what people call it because your train and test are from the same population (and that population is defined by a time frame). As the observations seem to have been sampled in a very special way (look at the date batches and you will see what I mean), it is normal for ID to carry information. You will never use such model to predict the future. You predict only the past using the same past information.
\n\nThis refers also to the following, as the way you apply a tag may provide information especially if a sequence proves useful (and correct, it is not a measurement, it is issued from a sequential sampling plan in product quality controlling - how Bosch sampled, remains a mystery as only them know as it did generalize for 2 years - 750 days = t=[0, 1800] if 0.01=6 min ):
\n\n[quote=Bob, Lord of Cats;138704]
\n\nID is not a measurement, it is a tag.
\n\n[/quote]
\n\nIf the "script" is going to be used for Bosch's internal audits, then the same exact rules apply and it will work. But in any case, no one is going to (be able to) ever use such model in a population which is not the same of the training data.
\n\nPost-edit add-on:
\n\n- \n
- ID is the by-product of sampling (directly) \n
- Sampling depends on time (indirectly here) \n
- Time is dependent on the outcome (indirectly here) \n
- If I understand Faron's script, the features depend on both on ID and time, hence the reason it works very well - but there must be other ways to engineer clear features for machine learning models without using the by-product ID \n
[quote=Laurae;138711]\n As the observations seem to have been sampled in a very special way (look at the date batches and you will see what I mean), it is normal for ID to carry information. You will never use such model to predict the future. You predict only the past using the same past information.\n[/quote]
\n\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).
\n\nAnd there is no guarantee of a "ID-Free" feature. It may or may not exist--time will tell. But there is no evidence "demanding" such a feature exist (sorry put on my physics/math hat for your last point)
", "votes": null, "replies": [] }, { "id": 138738, "author_name": "laurae2", "author_url": "", "post_date": "10/10/2016 17:46:15", "content": "[quote=Bob, Lord of Cats;138730]
\n\nThat doesn't help Bosch a lot if it is the case--that's why I feel it was unintentional. Otherwise if they could measure things in line to predict a failure, it saves them money (even if you measure at the end, before part leaves plant--keeping defects from the field saves money).
\n\n[/quote]
\n\nI believe it is unintentional also. It is 100% sure it would be useless if their intent is to predict right in the production line, unless you train a model in near real-time. There could be value behind this, as you could then use automated feature importance analyses to find which station is causing a batch of defects with the small lag of training a model - incurring a small batch loss / path freezing with a human supervision obviously.
\n\n[quote=Bob, Lord of Cats;138730]
\n\nAnd there is no guarantee of a "ID-Free" feature. It may or may not exist--time will tell. But there is no evidence "demanding" such a feature exist (sorry put on my physics/math hat for your last point)
\n\n[/quote]
\n\nAlso, putting your words in reverse, there is no demand for ID existing, as the information came from the sampling elsewhere (you are right there is no demand for it to exist right from the beginning). It's still a feature (and there are "ID-free" features), but when Bosch sampling methods for assigning ID will change there will be no guarantee anymore for the model to work properly.
\n\nIf I were to generalize the previous paragraph, I would say this: if Bosch receives a model which is too much tailored for this data set, then it will overfit severely and not fit enough the others. I think Bosch is prepared to work on the model script provided (if they did invest money already, they should not stop looking at potential valuation of the models they are going to receive).
\n\n\n\n
Anyway, for all those who still think it is a leak, free to read this:
\n\n
Going for a witch hunt like in TalkingData or here is not how Data Scientists are supposed to behave. A good community should not fight each other, but team up for the greater good and knowledge improvement of everyone.
", "votes": null, "replies": [] }, { "id": 138744, "author_name": "raddar", "author_url": "", "post_date": "10/10/2016 18:17:08", "content": "I really hope William was talking about this competition as if it was useful insight. I really hate how the word leakage has such a negative meaning in a community, that people start to avoid using it when itself might be useful in the end. As I said before, too many people jump to conclusions too fast.
", "votes": null, "replies": [] }, { "id": 138750, "author_name": "rdslater", "author_url": "", "post_date": "10/10/2016 19:00:30", "content": "I'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
", "votes": null, "replies": [] }, { "id": 138752, "author_name": "laurae2", "author_url": "", "post_date": "10/10/2016 19:05:01", "content": "[quote=Bob, Lord of Cats;138750]
\n\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
\n\n[/quote]
\n\nThat part was not aimed at you at all. But there are people in this thread who are doing the witch hunting on "leakage" without any argumentation (look at all the downvotes in this thread - excluding the True/False predictive modeling issue).
\n\nIn your case, you add information which is useful to read. There is nothing wrong with it.
", "votes": null, "replies": [] }, { "id": 138755, "author_name": "clockworkworld", "author_url": "", "post_date": "10/10/2016 19:49:43", "content": "[quote=Bob, Lord of Cats;138750]
\n\nI'm not on a witch hunt myself, but apparently someone didn't like my contribution. Just putting out how manufacturing works in the wild.
\n\nI'll stop posting--I thought actual domain experience might be of use.
\n\n[/quote]
\n\nConsider there are others here with domain experiences "in the wild" different than your own. Language like never and always and blanket statements like "that's not how it works" can be just as damaging to the discussion as witch hunting.
", "votes": null, "replies": [] }, { "id": 138877, "author_name": "kazanova", "author_url": "", "post_date": "10/11/2016 12:24:19", "content": "
Thanks Marios I thought my email was broken ;)
", "votes": null, "replies": [] }, { "id": 141002, "author_name": "lalthan", "author_url": "", "post_date": "10/24/2016 16:38:11", "content": "I think Kaggle should start a new competition on "How to read the relevant/useful comments" from this thread :)
", "votes": null, "replies": [] }, { "id": 518078, "author_name": "avoronoff", "author_url": "", "post_date": "04/16/2019 20:26:48", "content": ".
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "137689": "There seems to be lot of participants stuck in 0.26-0.30 score (including myself) and top contenders fighting at 0.4+, leaving the score range of 0.3-0.4 fairly empty. Wondering if it is a single feature causing that big a leap !!!", "137694": "No, it is not a single feature. I have to generate some features to bridge the gap.", "137695": "i have been digging for this \"magical feature/trick\" for days now. from what i see it is not in the categorical features. it must be with the time line.", "137696": "[quote=Tam T. Nguyen;137694]\r\n\r\nNo, it is not a single feature. I have to generate some features to bridge the gap. \r\n\r\n[/quote]\r\n\r\nI disagree. with single feature i got 0.45xx", "137697": "raddar: i guess you mean single engineered feature on top of existing given features ?", "137698": "Yes.", "137715": "raddar other than the single feature are you using all the feature or only some selected features. I am stuck with 8GB RAM laptop and confused whether start thinking for new features (engineered) or first looking for method for further dimension reduction. Currently I am working on featues as suggested by basic XGB model.", "137716": "my suggestion is to do some pre-processing on every file before joining all files together (this requires you to know how to load each file in a RAM-friendly way, many topics on that in a forum): \r\n\r\na) drop duplicate categorical features (in your case, dont use them at all, they are not that significant)\r\n\r\nb) drop duplicate date features within stations - all dates in every station are duplicated (except S24 & S25); For S24 & S25 you can calculate min/max dates (or just apply min/max for every station - will get same result)\r\n\r\nc) use all numeric data - this is where the most important information is anyway\r\n\r\nAfter these steps you will work with less than 1000 feature matrix, which can be handled in 8GB RAM, I think.", "137717": "raddar\r\na) How do you deal with null values in numeric data? \r\nb) In categorical data, how do you deal with categories? Do you keep them as is or one hot encode? Any tips?", "137718": "[quote=raddar;137716]\r\n\r\nAfter these steps you will work with less than 1000 feature matrix, which can be handled in 8GB RAM, I think.\r\n\r\n[/quote]\r\n\r\nConfirming it fits in 8GB RAM. As sparse, it even fits under 3GB - so there's no real reason to not be able to compete with a 8GB machine (the only brick wall is loading the data to do that).\r\n\r\nData fully filtered (removed perfect correlation and duplicates) results as this in R:\r\n\r\n* numeric dense = 604 features, 5473.1 MB\r\n* numeric sparse = 604 features, 1715.5 MB\r\n* categorical dense = 227 features, 1034.2 MB\r\n* categorical sparse = 597 features (hybrid design matrix), 801.7 MB (= binary as binary, uniques as deleted, levels>2 as one-hot encoded - missing counting as a level obviously)\r\n* date dense = 126 features, 1147 MB\r\n* date sparse = 126 features, 231.6 MB\r\n\r\nSo:\r\n\r\n* Total dense = 5473.1 + 1034.2 + 1147 = approx 7654.3 MB (peak during creation should be ~12GB or under that if smart and you got time)\r\n* Total sparse = 1715.5 + 801.7 + 231.6 = approx 2748.8 MB (peak during creation should be ~6GB or under that if smart and you got time)\r\n\r\nIf you shave this even more (using appropriate selection of features), you can more than halve it (75%-90% reduction).\r\n\r\nIn R, if you manage to make such matrix, I recommend you to save it as RDS so you can load and work on it back nearly instantly once you run a new R session.\r\n\r\nWith a 4GB laptop, you are left to try getting the sparse matrix or use precooked feature sets from the kernels of your choice of this competition. Under 1GB guaranteed sparse matrix, xgboost model under 4GB and >0.26 largely achievable.", "137743": "Laurae Well explained !!! I ran XGB separately on numeric, categorical and date and selected importance features from each of the set, combined them and ran xgb again. I guess a 8GB RAM should be able to handle it that way.", "137761": "raddar when you say one feature, you mean add one column at the end ? or you add one column for each feature given ?", "137848": "So I did find a single feature that took my single-model XGBoost score from about 0.27 to about 0.44. In the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, \"I suggest taking a step back, and just looking at the raw data.\" I know ... it's vague, but it's true.", "137853": "[quote=BreakfastPirate;137848]\r\nIn the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, \"I suggest taking a step back, and just looking at the raw data.\" I know ... it's vague, but it's true.\r\n[/quote]\r\n\r\nAfter I saw it, it is so obvious. I could even imagine why it is there in the first place.", "137854": "[quote=BreakfastPirate;137848]\r\n\r\nSo I did find a single feature that took my single-model XGBoost score from about 0.27 to about 0.44. In the Telstra contest, ( ͡° ͜ʖ ͡°) gave the best advice which was, \"I suggest taking a step back, and just looking at the raw data.\" I know ... it's vague, but it's true.\r\n\r\n[/quote]\r\n\r\n[Link for the lazy][1], an 8 page discussion about the magic feature in Telstra. The mentioned post is on top of the 2nd page.\r\n\r\n@BreakfastPirate, do you think the feature you found makes sense within the context we have available, or is it more like a magic / leak feature that doesn't really make sense?\r\n\r\nI'd fully understand if you wouldn't want to share such info at this stage. I'm just curious about how this competition will evolve in the remaining month.\r\n\r\n\r\n\r\n\r\n [1]: https://www.kaggle.com/c/telstra-recruiting-network/forums/t/19092/is-there-a-magic-feature-in-the-data", "137856": "[quote=barisumog;137854]\r\n\r\n@BreakfastPirate, do you think the feature you found makes sense within the context we have available, or is it more like a magic / leak feature that doesn't really make sense?\r\n\r\n[/quote]\r\n\r\nI have to say I honestly don't know. There's so much we don't know about the meaning of the fields and how the data was gathered that it is difficult to be sure the feature does or doesn't make sense. It will be interesting to hear discussion on that topic after the contest is over. Even after Telstra was over, there was a lot of debate on whether the feature in that contest was a leak or not, and I think this is a similar situation. Sorry I can't be more helpful.", "137866": "Ok. So what we know already about the magical feature?\r\n\r\n 1. it is one single engineered feature. Or atleast one.\r\n 2. it is not related to the categorical features\r\n 3. It is not certainly clear if the feature is a leak or not as people who found it seems to disagree about this\r\n \r\nWhat else?", "137872": "[quote=Sameh Faidi;137866]\r\nIt is not certainly clear if the feature is a leak or not as people who found it seems to disagree about this\r\n[/quote]\r\n\r\nIn my humble opinion, if there was no leak, there wouldn't be disagreement.\r\n\r\nIf it all boils down to what the definition of leak is, it probably is a leak.", "137873": "We have seen so many problems with the data in previous competitions. Why can't some of the people who found the magical feature confirm (in private) with the admins or sponsors if this is a valid feature. Now is a good time as we are still in early phase of the competition. Just an opinion!", "137874": "> If it all boils down to what the definition of leak is, it probably is a leak.\r\n\r\nI think it has to do with a simple distinction - at least in my mind - between leaks and hard-to-find data properties. Any data transformation that was introduced after data collection could be the cause of leaks. That includes all kinds of sorting by date, by time, by stations, giving non-random IDs to data, and whatever else one can think of. If, however, our hard-working Kagglers have discovered an underlying relationship between data points that helps them make better predictions, that would be squarely in the category of good prediction skills.\r\n\r\nI have no problem believing that at this point it is difficult to know which of the two possibilities is a correct one. By that I mean there could be a natural time series that improves prediction, or that ordered series of events was introduced by post-processing the data. Hard to know the truth without knowing what was done during data preparation.", "137876": "Tilii, in the context of Kaggle competitions, I'd say leaks **are** hard-to-find data properties. I respect the work of those who find them.\r\n\r\nI was just thinking out loud, out of experience - I haven't looked at the data set thoroughly yet.\r\n\r\nIf the magic feature was something like \"count of stations\" or \"mean of numeric\", then there would be no disagreement about whether it's a leak or not. So I'm just guessing it's something more exotic.", "137877": "I think that some of us are easy on the leak claim.\r\n\r\nWe all have to remember that given that results are usable for the competition host - is all that matters (prizes and rankings aside) \r\n\r\nThe high goal of data science is to derive business value of the data at hand, and if that is achieved - its a win-win situation. That is the great value proposition behind Kaggle for competition hosts (at least the way I see it...) \r\n\r\nIn any other case of problematic \"game-setting\" - both the host of the competition and Kaggle have to learn and improve,in order to serve that initial value proposition mentioned above again in future competitions.\r\n\r\nI think that the constant questions about leaked/non-leaked so called \"magical\" feature is a waste of time that if spent in more prolific ways could get us all to better results.", "137881": "> @Tilii, in the context of Kaggle competitions, I'd say leaks are hard-to-find data properties.\r\n\r\nTo me, the leak is something that was introduced during data manipulation, either by data owners or by Kaggle. In essence, the leak is an unintentional tipping of the final outcome by those who know the outcome. We've had plenty of leaks that satisfy this definition. In fact, we've had __too many__ of them.\r\n\r\nFinding non obvious relationships within data is not the same as finding a leak. In Santander competition it was discovered that customers younger than 24 do things a certain way. To me that is not finding a leak - it is finding a true and non-obvious property within that dataset.\r\n\r\nAnyway, I believe those who claim that at this point they can't say whether this is a leak or a true property of data.", "137885": "> I think that the constant questions about leaked/non-leaked so called \"magical\" feature is a waste of time that if spent in more prolific ways could get us all to better results.\r\n\r\nCouldn't agree more. \r\n\r\nI wish Kaggle was displaying the last submitted score rather than best score. That way those who wish to hide their true score could just resubmit one of their lower-score submissions - assuming they are willing to waste one attempt in order to conceal the score. We'd have way less traffic about magic features and leaks.", "137886": "[quote=Tilii;137881]\r\nTo me, the leak is something that was introduced during data manipulation, either by data owners or by Kaggle. In essence, the leak is an unintentional tipping of the final outcome by those who know the outcome. We've had plenty of leaks that satisfy this definition. In fact, we've had __too many__ of them.\r\n\r\nFinding non obvious relationships within data is not the same as finding a leak. In Santander competition it was discovered that customers younger than 24 do things a certain way. To me that is not finding a leak - it is finding a true and non-obvious property within that dataset.\r\n\r\nAnyway, I believe those who claim that at this point they can't say whether this is a leak or a true property of data.\r\n[/quote]\r\n\r\nI agree.\r\n\r\n\"In theory, theory and practice are the same. In practice, they are not.\"\r\n\r\n*Edited to add the quote I replied to*", "137890": "[quote=Tilii;137885]\r\n\r\nI wish Kaggle was displaying the last submitted score rather than best score. That way those who wish to hide their true score could just resubmit one of their lower-score submissions - assuming they are willing to waste one attempt in order to conceal the score. We'd have way less traffic about magic features and leaks.\r\n\r\n[/quote]\r\n\r\nThat would just be silly. We would have bots who would track leaderboard position every second and have non-official leaderboard:)", "137891": "Hmm, if I'm right about what this feature is based on, it could've been prevented by splitting train and test timewise, probably in the middle of the \"lull\" - there's little excuse for not doing that, unlike RedHat - aside from probably making things a lot harder still.", "137899": "[quote=raddar;137890]\r\n\r\nThat would just be silly. We would have bots who would track leaderboard position every second and have non-official leaderboard:)\r\n\r\n[/quote]\r\n\r\nHa, good one. I guess I underestimated that some people \"just want to know\" and would probably be doing exactly what you suggested.\r\n\r\nMy larger point was that focusing on magic features, leaks, and score differences at 3rd and 4th decimal places is not necessarily good for us. Sure it sparks competition, but in many cases it will lead to over-fitting and to wild-goose chases after magic features.\r\n\r\nJust to be clear: I am not against those who look for leaks, those who find them, or those who divulge them.", "137994": "raddar did you use whole dataset (I mean did you dropped any column/s)?", "137995": "dropped only duplicate ones.", "138050": "But what about the empty ones =)", "138062": "[quote=Bob, Lord of Cats;138050]\r\n\r\nBut what about the empty ones =)\r\n\r\n[/quote]\r\n\r\nThat too =)", "138063": "I noticed many failures happen in sequence -- id and id+1\r\nhttps://www.kaggle.com/aamaia/bosch-production-line-performance/failures-in-pairs", "138065": "*facepalm* ... another appearance of the Talkingdata randomizer? Clumping by date is one thing, but supposedly random ID #'s...???", "138066": "I have to admit that this competition (or my inability to find the magic feature) is becoming very frustrating... And the more that others give hints that lead me nowhere the more frustrated I become! \r\n\r\nAnyhow I thought I noticed something strange with id's and row duplicates [link here][1] but couldn't make much of it. I've also tried a train and test pairs submission that wasn't much better than a random submission and I think @anokas has said as much [here][2].\r\n\r\nAnyone willing to confirm we should we be looking elsewhere for an invite to the 0.4+ club?\r\n\r\n [1]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23553/sequential-duplicates-in-test-files\r\n [2]: https://www.kaggle.com/c/bosch-production-line-performance/forums/t/23971/rule-clarification/137220#post137220", "138085": "Chippy: I share your frustration. I have been looking everywhere for the magical 0.1 boost but with no luck. One thing I promise is to make it public when and if I find it. To be positive, while looking for it I discovered many interesting characteristics of the data.", "138089": "How is any of this \"predictive modeling?\" Whenever there is any information given regarding test sets, they then become training sets. And by giving people an aggregate score on their model with respect to the \"test\" sets, they are getting information about those \"test\" sets. Therefore, those \"test\" sets are now training sets, and no true \"prediction\" is being done. All that's being done is fitting a model to a large set of data. There is value in that, and it is of use to companies, but it is not predictive modeling. This site should be clear about that, and they should communicate that to the companies who think they are getting predictive models and predictive modelers from this venture.\r\n\r\nFor example, back on 9/22, radar had submitted 9 entries and his high score was 0.26727. Now all of a sudden, he's part of a team that has 123 submissions and a score of 0.50175. Did they just randomly double their score in 2 weeks, using no additional information other than the training sets? Doesn't seem too likely, does it? How did they even know they were moving in the right direction with each additional submission if they were not getting some sort of additional information with each submission? They were getting information with each submission, in the form of a score for that submission.\r\n\r\nHow do we know how much information and submissions were shared among those team members? Why bother with cracking down on multiple accounts when people are submitting individually, then as part of a team, in the same competition? And now there is all this talk of a magical boost. How could they have known about this boost and refined their model if they weren't getting feedback from each submission?\r\n\r\nBottom line, this site needs to call this what it is. It's certainly not predictive modeling.", "138093": "[quote=GWillie;138089]\r\n\r\nFor example, back on 9/22, radar had submitted 9 entries and his high score was 0.26727. Now all of a sudden, he's part of a team that has 123 submissions and a score of 0.50175. \r\n\r\n[/quote]\r\n\r\n1) raddar was 0.48+ before he merged, not all of sudden.\r\n\r\n2) i personally didn't call this predictive modeling. I called it kaggling.\r\n\r\n3) The very first thing of predictive modeling is proper data preparation to remove any unwanted leak. Now we are doing their job since bosch didn't get it done in the first place. \r\n\r\n4) Exploiting leak is only part of the story. Non-leak features has tremendous value. \r\n\r\n5) I believe the final solution is of pretty good value to bosch. They know what's wrong in data and how misleading it can be. They get a good benchmark. and they also get a pretty decent non-leak model out of the final solution. \r\n\r\n6) last, as a kaggler, i believe whatever is given is part of the puzzle and is legit to use. In this case, exploiting the leak requires data cleaning, spotting pattern and proper encoding. It is also a good practice to me.", "138095": "GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.\r\n\r\nAnd the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1. \r\n\r\nAnd no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models.", "138100": "[quote=raddar;138095]\r\n\r\n@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.\r\n\r\n\r\n----------\r\n\r\nI agree your opinion, I also spend lot of time to do feature engineering. \r\nBut I just kill time. Could you let me know how to start data exploration?", "138102": "Same curiosity as LEE. How to improve feature engineering skills ?? How to be better at understanding the data ? Not just specific to this competition, but in general.", "138106": "I picked 3 that help me the most:\r\n\r\n1) One of the the first tricks I do is try to load the data in excel (well the portion that fits in excel row limit). Then do many kinds of conditional formatting. Colouring raw data helps to see some patterns (it depends if you have an eye for it and see something instantly, or it takes some time with filtering and etc.)\r\n\r\n2) ploting cumulative distribution of Response=0 and Response=1 on same plot (and iterate through all variables). It often reveals interesting patterns within variables as well. \r\n\r\n3) use some information criteria for each variable (IV, Gini, entropy, etc.), pick the most important ones and try scatterplotting every combination. Usually useful when making feature transformations for non-tree methods", "138108": "[quote=raddar;138095]\r\n\r\n@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.\r\n\r\nAnd the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1. \r\n\r\nAnd no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models.\r\n\r\n[/quote]\r\n\r\nraddar, there will always be some who will discard the hard work done by others, I don't think you have to justify any of your work. your achievements and contribution to the Kaggle community are great.\r\n\r\nIf you decide to open a course on data exploration and feature engineering i'd be the first to sign in :-)\r\n\r\nwith great respect,\r\n\r\nNati", "138110": "raddar... Those are quite informative and sounds well structured and methodological. Thanks for the tips...", "138127": "I side with @breakfastpirate that one cannot assess if the spillover effects are really exploits / leaks or if they may actually be of value to Bosch. Ordering (think Ids) is natural in manufacturing environment with production jobs, batches, lot sizing. So any information originating from sequences, ordering could be relevant to for failure detection. Similarly, the overlap between test and training data is also realistic because in the real setting Bosch would want to figure out the minimal amount of ground truth necessary to assess their production line performance. Ground truth is sourced from expensive testing and they could limit that if the predictions are more predictive (as measured by MCC).", "138129": "[quote=GWillie;138089]\r\n\r\nHow is any of this \"predictive modeling?\" Whenever there is any information given regarding test sets, they then become training sets. And by giving people an aggregate score on their model with respect to the \"test\" sets, they are getting information about those \"test\" sets. Therefore, those \"test\" sets are now training sets, and no true \"prediction\" is being done.....\r\n\r\nBottom line, this site needs to call this what it is. It's certainly not predictive modeling.\r\n\r\n[/quote]\r\n\r\n\r\n@GWillie Although, I don't agree with your approach and conjectures regarding certain folks gaming the system, I think you brought up an interesting point about 'those \"test\" sets become training sets' that you should possibly reconsider.\r\n\r\nIf your sole purpose is to chase previous submission scores, without regard to improving your methodology, you could get an unexpected surprise when the models are scored on the held out test data. Look at the movement in the Santander competition to see why final scoring is what matters most. \r\n\r\nTo those who took the time to clarify GWillie's comments, kudos to you. It's better to get the facts out there, then let the fiction lie.\r\n\r\nCheers", "138160": "If its not too much of a revelation: can someone using the golden feature tell me whether one can benefit from it by only using 60-70 numeric columns +(the golden feature). I have a feeling that i have discovered this feature but am unable to utilize it to the fullest due to limited memory.", "138163": "[quote=Mohd Shadab Alam;138160]\r\n\r\nIf its not too much of a revelation: can someone using the golden feature tell me whether one can benefit from it by only using 60-70 numeric columns +(the golden feature). I have a feeling that i have discovered this feature but am unable to utilize it to the fullest due to limited memory. \r\n\r\n[/quote]\r\n\r\nWhy aren't you just using 60-70 numeric columns + the awesome golden feature and check if it performs better?", "138166": "Yeah it does but the improvement is nowhere close to 0.1 types", "138182": "I appreciate all the comments on my post, but I couldn't help but notice (and even expected) that few, if any, actually addressed my point. First, I never said there wasn't hard work going into this. I acknowledge and respect that there is. And I said that there was value to this. But my point is, it's not predictive modeling - it's large scale data mining and fitting a model to a large set of data. And yes, a big part of that is data exploration. I never said it wasn't. But by getting information about your test sets in the form of leaderboard feedback, then by definition, those sets are no longer test sets, they are training sets, and you are no longer doing predictive modeling, you are doing data mining and model fitting. And the ability to refine those models in the form of hundreds of submissions is evidence that the test sets are actually training sets.\r\n\r\nWhy is this important to point out? Well, I happen to know from experience that there is a vast demand and need for true predictive modeling. And this site generates revenue for these competitions by promoting itself to companies as being able to provide predictive models and predictive modelers. But when competitions are run like this, then those companies are not getting true predictive models and predictive modelers. This is not doing anything to promote the field of predictive modeling/analytics. In fact, I would argue that it is hindering the advancement of the field.\r\n\r\nI'm not coming down on any of the participants nor trying to minimize any of the work they're doing. It is not easy, and it does have value. But we all - most importantly the companies buying into it - need to be clear about what it is and what it isn't. The site was founded and initially funded as a platform for predictive modeling and analytics, and it bills itself to competitors and potential funding companies as being just that. And there's a good bit of money involved in that pursuit. But the way it's currently run, a competition like this is data mining and model fitting, not predictive modeling/analytics. I just think everyone not only has a right to know this but really should know this.", "138183": "rcarson\r\n\r\n1) raddar was 0.48+ before he merged, not all of sudden.\r\n\r\nOk. So then raddar went from 0.27 to 0.48 before he merged. If this were true predictive modeling, he would have had no feedback from his test set submissions, so the only way to make this kind of increase would basically be random. If it were true predictive modeling, his final score would be somewhere below 0.27.\r\n\r\n\r\n2) i personally didn't call this predictive modeling. I called it kaggling.\r\n\r\nThank you for agreeing with me. That's all I'm saying. But the danger is, kaggle IS calling it predictive modeling.\r\n\r\n\r\n3) The very first thing of predictive modeling is proper data preparation to remove any unwanted leak. Now we are doing their job since bosch didn't get it done in the first place. \r\n\r\nAgreed. But the analogy to predictive modeling, in this case, stops there.\r\n\r\n\r\n4) Exploiting leak is only part of the story. Non-leak features has tremendous value.\r\n\r\nAgreed, but in this case, it's not predictive modeling.\r\n\r\n\r\n5) I believe the final solution is of pretty good value to bosch. They know what's wrong in data and how misleading it can be. They get a good benchmark. and they also get a pretty decent non-leak model out of the final solution. \r\n\r\nAgreed, but in this case, it's not predictive modeling. And it's my opinion that Bosch deserves to know that.\r\n\r\n\r\n6) last, as a kaggler, i believe whatever is given is part of the puzzle and is legit to use. In this case, exploiting the leak requires data cleaning, spotting pattern and proper encoding. It is also a good practice to me. \r\n\r\nAgreed. I have no problem with competitors using whatever they can to advance their score. I'm just saying that we need to call it what it is. The way the rules are set up for competitions, it's not predictive modeling.", "138184": "raddar\r\n\r\n\"@GWillie, Thank you for bringing my current path into attention. I spent pure 35 hours just looking at the data and not touching any modelling aspect of it. Yes, I was lucky to find this golden feature(s), but so can anyone, if they dedicate that much time into it and know how to handle the data. You conclude that it is impossible to double the score after lots of dedicated time in feature engineering - well, I think you just havent done any of that.\"\r\n\r\nYes, and these are all important parts of large-scale data mining and fitting a model to a large set of data, I don't disagree with that and never took exception to that. I have done that type of work, as I have also done predictive modeling work. Having done the latter, I know that this is not true predictive modeling.\r\n\r\n\"And the serious modelling was not something I started doing in last week - I just got stuck with my models, offered few teams to team up, and they were willing to do so. When joining with anokas & ZFTurbo I had no knowledge on what path they took (which probably happens 99% of the time when team merges). Luckily, they have went completely in different direction, and our solutions had only 60-70% of correlation. Enough to simply average and take #1.\"\r\n\r\nYes, and this \"serious modeling\" that you have done is data mining and fitting a model to data. It is not true predictive modeling, given how the rules of the competition are set up. That's my point.\r\n\r\n\"And no, I am not getting any other feedback from submissions rather than just my current progress of my feature engineering part and how it performed on simple benchmark models.\"\r\n\r\nYou don't need to be getting any other feedback than that in order for this to be disqualified as true predictive modeling. That's my point.", "138187": "And now back to the actual topic...anyone have anything new to add to the 'magical feature' discussion?", "138188": "wordsmith\r\n\r\n\"@GWillie Although, I don't agree with your approach and conjectures regarding certain folks gaming the system, I think you brought up an interesting point about 'those \"test\" sets become training sets' that you should possibly reconsider.\r\n\r\nIf your sole purpose is to chase previous submission scores, without regard to improving your methodology, you could get an unexpected surprise when the models are scored on the held out test data. Look at the movement in the Santander competition to see why final scoring is what matters most. \r\n\r\nTo those who took the time to clarify GWillie's comments, kudos to you. It's better to get the facts out there, then let the fiction lie.\"\r\n\r\nYou seem to not fully understand that which you are disagreeing with. I never stated that \"the sole purpose is to chase previous submission scores, without regard to improving your methodology.\" In fact, I am saying just the opposite. People are improving methodology based on previous submission scores. You are using information from those \"test\" sets to improve methodology and score. Therefore, those \"test\" sets are now, by definition, training sets. You have used those sets to improve your methodology/score, so you have trained on those sets.\r\n\r\nVery little was done to \"clarify\" my comments. In fact, my main point was not really addressed at all. So yes, it is better to keep trying to get the facts out rather than let the fiction lie, and I will continue to do so.", "138190": "GWillie, one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a *model* that *predicts* the remaining 70% of test data (i.e. predictive modeling).", "138194": "I am using *xgboost.Booster.predict()* so it is predictive modeling. :D\r\n\r\nBut seriously do not worry too much about the number of submissions. In this case having more submissions is not really competitive advantage. If you think it is, you can submit 5 per day so there are still 165 submissions left for you. \r\n\r\nBigger problem could be that the train set contains more information about the test set than it was probably intended. Leakages/ magic features are usually appear in these competitions.", "138196": "Also note that CV and LB scores are *very* highly correlated, so raddar didn't need to use the public LB to know he's making progress.\r\n\r\nI think there are some good clues now to The Feature(tm), and IMO it's a matter of time until it's public, and there'll probably be a .4 script within the next two weeks I figure, if it doesn't need too much time/RAM to compute. So happy hunting and stay patient :)", "138199": "[quote=raddar;138106]\r\n\r\nI picked 3 that help me the most:\r\n\r\n[/quote]\r\n\r\n\r\nThank you for your kind advice. I try as hard as I can.", "138201": "[quote=BreakfastPirate;138190]\r\n\r\n@GWillie, one of your arguments is that people are getting feedback from the leaderboard. This is true, BUT the public leaderboard score (the one we see now) is based on only 30% of the test set. The final standings are based on the remaining 70% of the test data (the private leaderboard). So if a person overfits to the public leaderboard, they will drop when the private leaderboard is revealed. So the real goal of a competition is not to get a good score on the public LB, it is to build a *model* that *predicts* the remaining 70% of test data (i.e. predictive modeling).\r\n\r\n[/quote]\r\n\r\n@GWillie\r\n\r\nTotally agree with that. What's the difference between having the train data + the 30 % of the holdout set for training vs. a split you would use in predictive modeling?", "138203": "I believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.\r\n\r\n[quote=happycube;138196]\r\n\r\nAlso note that CV and LB scores are *very* highly correlated, so raddar didn't need to use the public LB to know he's making progress.\r\n\r\nI think there are some good clues now to The Feature(tm), and IMO it's a matter of time until it's public, and there'll probably be a .4 script within the next two weeks I figure, if it doesn't need too much time/RAM to compute. So happy hunting and stay patient :)\r\n\r\n[/quote]", "138205": "Give me RAM i'll give you golden feature :P...\r\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\r\nMail me at: golden.reflection@gmail.com (its a real id)", "138206": "GWillie, regarding your point that Kaggle competitions are data mining but not predictive modeling...\r\n\r\nIn the last Kaggle competition (Grupo Bimbo), we were given a training dataset through \"Week 9\", public leaderboard scores were based on unlabeled \"Week 10\" predictions (from 'test set'), and final/private leaderboard winners selected based on unlabeled \"Week 11\" predictions (from 'test set'). Would you call this predictive modeling? Winning entries were selected based on predictions for a later time period than both the training data and the public leaderboard scores.\r\n\r\nIf you consider that predictive modeling, I'd interpret your posts as criticizing how the train/test split is in this competition, as opposed to a problem with every Kaggle competition. In the forums, there's always a healthy discussion of how these splits are done in each competition, and how optimal/useful or not they are. My understanding is that Kaggle works with the data owner to derive the most useful way to do train/test data split. Maybe the process could be improved, apparently it is non-trivial to get right in every case.", "138207": "> But the way it's currently run, a competition like this is data mining and model fitting, not predictive modeling/analytics. I just think everyone not only has a right to know this but really should know this.\r\n\r\nYou said \"not predictive modeling\" 5 times in the last few messages. Based on your level of conviction, one would think that you must be absolutely right. And you are not.\r\n\r\nTo expand on BreakfastPirate's fine argument: we get feedback only on portion of the data, and our models are ultimately tested on data that have not been part of that feedback. That is different from \"regular\" predictive modeling only in a sense that our only info there is our local cross-validation score, while here we can compare it with leaderboard. Those who know how to do cross-validation properly would do just fine without that piece of information.\r\n\r\nIt is not uncommon at all for large train datasets that 1/5 (or even 1/4) of the data is set aside for \"fixed\" validation. After modeling the remaining train data based on cross-validation and adjusting the model to the validation set, eventually we test its performance on the test dataset. If you think about it, the leaderboard score is nothing but a validation score on our hold-out data, which still gets tested on the rest of the data (private leaderboard).", "138208": "Mohd\r\nhttps://aws.amazon.com/ec2/instance-types/", "138209": "[quote=Mohd Shadab Alam;138205]\r\n\r\nGive me RAM i'll give you golden feature :P...\r\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\r\nMail me at: golden.reflection@gmail.com (its a real id)\r\n\r\n[/quote]\r\n\r\nWould you like fries with your supersized RAM? :D", "138210": "Where is the rabbit?", "138211": "[quote=Mohd Shadab Alam;138203]\r\n\r\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. **But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.**\r\n\r\n[/quote]\r\n\r\nThis is simply not true.", "138213": "dunno about rabbit,i can spot a kitten though ;)", "138216": "[quote=anokas;138211]\r\n\r\n[quote=Mohd Shadab Alam;138203]\r\n\r\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. **But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.**\r\n\r\n[/quote]\r\n\r\nThis is simply not true.\r\n\r\n[/quote]\r\nDude you were using huge ram right from the begining....when did you get time to try it with 60-70 features....", "138218": "[quote=anokas;138211]\r\n\r\n[quote=Mohd Shadab Alam;138203]\r\n\r\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. **But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.**\r\n\r\n[/quote]\r\n\r\nThis is simply not true.\r\n\r\n[/quote]\r\n\r\nLol. Please don't break someone who is convinced of spotting the golden duck.", "138219": "[quote=Mohd Shadab Alam;138205]\r\n\r\nGive me RAM i'll give you golden feature :P...\r\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\r\nMail me at: golden.reflection@gmail.com (its a real id)\r\n\r\n[/quote]\r\n\r\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.", "138220": "[quote=Mohd Shadab Alam;138216]\r\n\r\nDude you were using huge ram right from the begining....when did you get time to try it with 60-70 features....\r\n[quote=anokas;138211]\r\n\r\n[quote=Mohd Shadab Alam;138203]\r\n\r\nI believe that by now 20-25% participants using <=8GB RAM would have tried this feature in their model. **But its real value is not apparent as whole training numeric is needed to realize 0.1+ LB gain.**\r\n\r\n[/quote]\r\n\r\nThis is simply not true.\r\n\r\n[/quote]\r\n\r\n\r\n[/quote]\r\n\r\nWe can get 0.49 using exactly 0 numeric features in our model, and that can run in 8GB ram no problem.\r\n\r\nJust because we have access to a lot of RAM, doesn't mean we don't make our code memory efficient. If you need 256GB+ RAM to run your model, then you need to optimise your memory usage. Our full LB model can still run no problem start-to-finish on a 64GB machine.\r\n\r\nAt the end of the competition, if you want I'll build a 0.4 model on a Raspberry Pi.", "138221": "anokas 'Everyone Stop Improving Please' -> I have stopped improving for a long time now.", "138222": "[quote=PrakashSubedi;138219]\r\n\r\n[quote=Mohd Shadab Alam;138205]\r\n\r\nGive me RAM i'll give you golden feature :P...\r\nLooking for a team mate: Sole criteria, member should have access to >255GB RAM\r\nMail me at: golden.reflection@gmail.com (its a real id)\r\n\r\n[/quote]\r\n\r\n255GB RAM is not needed at all. I am using 16 GB RAM and it could easily run all the numeric, date and couple of engineered features.\r\n\r\n[/quote]\r\nIf you think your model can handle this duck feature then lets team up and crack it big. BTW i am using python with dtype option to load this data but am unable to do so.", "138223": "anoksa i agree your opnion , by the way 0 numeric feature? it means date features is more important?", "138224": "", "138225": "[quote=<<