{
  "id": 22909,
  "title": "Expeditive exploration+models on data",
  "url": "/competitions/bosch-production-line-performance/discussion/22909",
  "author_name": "Laurae",
  "post_date": "2016-08-17T14:44:17.660000",
  "votes": 72,
  "comment_count": 25,
  "views": 6975,
  "content": "<p>P.S: don't ask me why all images are breaking for no reason on Kaggle, I never get to manage to have them viewable for hours on a post embedded directly</p>\n\n<h2>Expeditive file analysis part</h2>\n\n<h3>A quick analysis on the label:</h3>\n\n<ul>\n<li>We are playing with an imbalance ratio of 1:172 (172 negatives for 1 positive): 99.41% accuracy if you predict the negative case only</li>\n<li>1183747 rows</li>\n<li>1176868 negative labels</li>\n<li>6879 positive labels</li>\n</ul>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4680/Bosch_SizeREsp.JPG\" alt=\"enter image description here\" title></p>\n\n<p>The evolution of the metric is the following (True Positive vs True Negative - FP/FN are straightforward):</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4688/Bosch_MCC.JPG\" alt=\"enter image description here\" title></p>\n\n<p>Interactive MCC: <a href=\"https://public.tableau.com/views/KaggleMCCMetric/Binned0_01?:embed=n&:display_count=no&:toolbar=no\">here at Tableau</a></p>\n\n<h3>A quick analysis on train_numeric.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 969] (969 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random. See <a href=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4690/Missing_Train_Numeric.htm\">this webpage</a> for more details</li>\n<li>Sparse</li>\n<li>Contains the label we need to predict</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4684/Bosch_TrainPic1.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>A quick analysis on train_categorical.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 2141] (2141 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random. See <a href=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4691/Missing_Train_Categorical.htm\">this webpage</a> for more details (edit: seems there are import errors when loading the dataset, need to check but these missing value counts are not reliable at all).</li>\n<li>Many columns have 1183747 missing values or near that (which means you can remove a lot of them already!).</li>\n<li>Extremely sparse</li>\n<li>Lot of 0 only</li>\n<li>L1_S24_F7XX to L1_S24_F9XX (replace XX by numbers) seems linked (same zeroes)</li>\n<li>Loads of features with NOTHING inside (just sparse, see below)</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4692/Bosch_TrainPic2.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>A quick analysis on train_data.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 1157] (1157 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random.</li>\n<li>Sparse</li>\n<li>Dates are numeric already, seems &quot;normalized&quot; overall by days?</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4685/Bosch_TrainPic3.JPG\" alt=\"enter image description here\" title></p>\n\n<h2>Expeditive modeling part</h2>\n\n<h3>Elastic Net on raw numeric data, clearly bad:</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4681/Bosch_GLM.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>Random Forest on raw numeric data, &quot;not bad&quot; but can do much better (10 trees, 7 minutes):</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4682/Bosch_RFAUC.JPG\" alt=\"enter image description here\" title>\n<img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4679/Bosch_RFImp.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>Single decision tree with 5-fold validation on the first 50000 observations, with all features:</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4687/Bosch_DT.jpg\" alt=\"enter image description here\" title></p>",
  "messages": [
    {
      "id": 131475,
      "postDate": "2016-08-17T14:44:17.660Z",
      "content": "<p>P.S: don't ask me why all images are breaking for no reason on Kaggle, I never get to manage to have them viewable for hours on a post embedded directly</p>\n\n<h2>Expeditive file analysis part</h2>\n\n<h3>A quick analysis on the label:</h3>\n\n<ul>\n<li>We are playing with an imbalance ratio of 1:172 (172 negatives for 1 positive): 99.41% accuracy if you predict the negative case only</li>\n<li>1183747 rows</li>\n<li>1176868 negative labels</li>\n<li>6879 positive labels</li>\n</ul>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4680/Bosch_SizeREsp.JPG\" alt=\"enter image description here\" title></p>\n\n<p>The evolution of the metric is the following (True Positive vs True Negative - FP/FN are straightforward):</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4688/Bosch_MCC.JPG\" alt=\"enter image description here\" title></p>\n\n<p>Interactive MCC: <a href=\"https://public.tableau.com/views/KaggleMCCMetric/Binned0_01?:embed=n&:display_count=no&:toolbar=no\">here at Tableau</a></p>\n\n<h3>A quick analysis on train_numeric.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 969] (969 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random. See <a href=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4690/Missing_Train_Numeric.htm\">this webpage</a> for more details</li>\n<li>Sparse</li>\n<li>Contains the label we need to predict</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4684/Bosch_TrainPic1.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>A quick analysis on train_categorical.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 2141] (2141 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random. See <a href=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4691/Missing_Train_Categorical.htm\">this webpage</a> for more details (edit: seems there are import errors when loading the dataset, need to check but these missing value counts are not reliable at all).</li>\n<li>Many columns have 1183747 missing values or near that (which means you can remove a lot of them already!).</li>\n<li>Extremely sparse</li>\n<li>Lot of 0 only</li>\n<li>L1_S24_F7XX to L1_S24_F9XX (replace XX by numbers) seems linked (same zeroes)</li>\n<li>Loads of features with NOTHING inside (just sparse, see below)</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4692/Bosch_TrainPic2.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>A quick analysis on train_data.csv:</h3>\n\n<ul>\n<li>Matrix size: [1183747 x 1157] (1157 features, 1183747 observations)</li>\n<li>Missing values do not seem to be missing at random.</li>\n<li>Sparse</li>\n<li>Dates are numeric already, seems &quot;normalized&quot; overall by days?</li>\n</ul>\n\n<p>Sneak peak:</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4685/Bosch_TrainPic3.JPG\" alt=\"enter image description here\" title></p>\n\n<h2>Expeditive modeling part</h2>\n\n<h3>Elastic Net on raw numeric data, clearly bad:</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4681/Bosch_GLM.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>Random Forest on raw numeric data, &quot;not bad&quot; but can do much better (10 trees, 7 minutes):</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4682/Bosch_RFAUC.JPG\" alt=\"enter image description here\" title>\n<img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4679/Bosch_RFImp.JPG\" alt=\"enter image description here\" title></p>\n\n<h3>Single decision tree with 5-fold validation on the first 50000 observations, with all features:</h3>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4687/Bosch_DT.jpg\" alt=\"enter image description here\" title></p>",
      "rawMarkdown": "P.S: don't ask me why all images are breaking for no reason on Kaggle, I never get to manage to have them viewable for hours on a post embedded directly\r\n\r\n##Expeditive file analysis part\r\n\r\n###A quick analysis on the label:\r\n\r\n* We are playing with an imbalance ratio of 1:172 (172 negatives for 1 positive): 99.41% accuracy if you predict the negative case only\r\n* 1183747 rows\r\n* 1176868 negative labels\r\n* 6879 positive labels\r\n\r\n![enter image description here][1]\r\n\r\nThe evolution of the metric is the following (True Positive vs True Negative - FP/FN are straightforward):\r\n\r\n![enter image description here][2]\r\n\r\nInteractive MCC: [here at Tableau][3]\r\n\r\n###A quick analysis on train_numeric.csv:\r\n\r\n* Matrix size: [1183747 x 969] (969 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random. See [this webpage][4] for more details\r\n* Sparse\r\n* Contains the label we need to predict\r\n\r\nSneak peak:\r\n\r\n![enter image description here][5]\r\n\r\n###A quick analysis on train_categorical.csv:\r\n\r\n* Matrix size: [1183747 x 2141] (2141 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random. See [this webpage][6] for more details (edit: seems there are import errors when loading the dataset, need to check but these missing value counts are not reliable at all).\r\n* Many columns have 1183747 missing values or near that (which means you can remove a lot of them already!).\r\n* Extremely sparse\r\n* Lot of 0 only\r\n* L1_S24_F7XX to L1_S24_F9XX (replace XX by numbers) seems linked (same zeroes)\r\n* Loads of features with NOTHING inside (just sparse, see below)\r\n\r\nSneak peak:\r\n\r\n![enter image description here][7]\r\n\r\n###A quick analysis on train_data.csv:\r\n\r\n* Matrix size: [1183747 x 1157] (1157 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random.\r\n* Sparse\r\n* Dates are numeric already, seems \"normalized\" overall by days?\r\n\r\nSneak peak:\r\n\r\n![enter image description here][8]\r\n\r\n## Expeditive modeling part\r\n\r\n###Elastic Net on raw numeric data, clearly bad:\r\n\r\n![enter image description here][9]\r\n\r\n###Random Forest on raw numeric data, \"not bad\" but can do much better (10 trees, 7 minutes):\r\n\r\n![enter image description here][10]\r\n![enter image description here][11]\r\n\r\n###Single decision tree with 5-fold validation on the first 50000 observations, with all features:\r\n\r\n![enter image description here][12]\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4680/Bosch_SizeREsp.JPG\r\n  [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4688/Bosch_MCC.JPG\r\n  [3]: https://public.tableau.com/views/KaggleMCCMetric/Binned0_01?:embed=n&:display_count=no&:toolbar=no\r\n  [4]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4690/Missing_Train_Numeric.htm\r\n  [5]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4684/Bosch_TrainPic1.JPG\r\n  [6]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4691/Missing_Train_Categorical.htm\r\n  [7]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4692/Bosch_TrainPic2.JPG\r\n  [8]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4685/Bosch_TrainPic3.JPG\r\n  [9]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4681/Bosch_GLM.JPG\r\n  [10]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4682/Bosch_RFAUC.JPG\r\n  [11]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4679/Bosch_RFImp.JPG\r\n  [12]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4687/Bosch_DT.jpg",
      "votes": 72
    },
    {
      "id": 131516,
      "postDate": "2016-08-17T20:11:06.400Z",
      "content": "<p>Anomaly detection is based on calculating probability distribution of training (mostly non-anomalous) data based on each 'important' feature. Then a new example is tested using this distribution. If the probability is below a threshold then the example is anomalous. Usually a Gaussian distribution is assumed for each feature. If the distribution of the feature is non-Gaussian one can transform using log, ^0.5, etc.. Then for each feature estimate the mean and variance based on the training data. The overall probability distribution is the product of the probability distribution of all features.\nSee this video from Andrew Ng\n<a href=\"https://www.youtube.com/watch?v=h5iVXB9mczo\">https://www.youtube.com/watch?v=h5iVXB9mczo</a></p>",
      "rawMarkdown": "Anomaly detection is based on calculating probability distribution of training (mostly non-anomalous) data based on each 'important' feature. Then a new example is tested using this distribution. If the probability is below a threshold then the example is anomalous. Usually a Gaussian distribution is assumed for each feature. If the distribution of the feature is non-Gaussian one can transform using log, ^0.5, etc.. Then for each feature estimate the mean and variance based on the training data. The overall probability distribution is the product of the probability distribution of all features.\r\nSee this video from Andrew Ng\r\nhttps://www.youtube.com/watch?v=h5iVXB9mczo",
      "votes": 9
    },
    {
      "id": 131489,
      "postDate": "2016-08-17T16:02:56.960Z",
      "content": "<p>[quote=raddar;131480]</p>\n\n<p>Nice quick post:) </p>\n\n<p>What kind of software are you using there?</p>\n\n<p>[/quote]</p>\n\n<ul>\n<li>H2O Flow for most (pure GUI based and can run on external memory - can script too and make workbooks).</li>\n<li>Tableau for the MCC table</li>\n<li>JMP (SAS) for the decision tree</li>\n</ul>\n\n<p>I'm still adding things to the main post as there are many things that can be quick to run to look at :p</p>\n\n<p>On the 50,000 subsampling I find it very interesting to get 0.91 AUC with a simple decision tree (5-fold).</p>\n\n<p>Currently running a missing value analysis while doing other things, and it is taking forever (15,000 cases analyzed over 1 hour &gt;_&gt;).</p>\n\n<p>All the images got suddenly broken when embedded, any reason? (fixed)</p>",
      "rawMarkdown": "[quote=raddar;131480]\r\n\r\nNice quick post:) \r\n\r\nWhat kind of software are you using there?\r\n\r\n[/quote]\r\n\r\n* H2O Flow for most (pure GUI based and can run on external memory - can script too and make workbooks).\r\n* Tableau for the MCC table\r\n* JMP (SAS) for the decision tree\r\n\r\nI'm still adding things to the main post as there are many things that can be quick to run to look at :p\r\n\r\nOn the 50,000 subsampling I find it very interesting to get 0.91 AUC with a simple decision tree (5-fold).\r\n\r\nCurrently running a missing value analysis while doing other things, and it is taking forever (15,000 cases analyzed over 1 hour >_>).\r\n\r\nAll the images got suddenly broken when embedded, any reason? (fixed)",
      "votes": 7
    },
    {
      "id": 132183,
      "postDate": "2016-08-23T10:19:25Z",
      "content": "<p>Train numeric with 17 features can get you 0.10 MCC with a <strong>simple decision tree</strong> (not even an ensemble, just a decision tree) on 3 fold (yes only 3) cross-validation (see picture for other metrics). Not even tuned for maximum MCC, just went maximizing R^2 which is a proxy to Phi Coefficient (which is MCC in a confusion matrix).</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4716/Bosch_17f.jpg\" alt=\"enter image description here\" title></p>\n\n<p>With Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an &quot;easy to copy&quot; format:</p>\n\n<pre><code>&quot;L3_S32_F3850&quot;, &quot;L1_S24_F1846&quot;, &quot;L1_S24_F1723&quot;, &quot;L3_S29_F3348&quot;, &quot;L3_S29_F3336&quot;, &quot;L3_S30_F3519&quot;, &quot;L1_S24_F1844&quot;, &quot;L3_S29_F3367&quot;, &quot;L3_S30_F3539&quot;, &quot;L3_S30_F3809&quot;, &quot;L3_S30_F3554&quot;, &quot;L3_S29_F3433&quot;, &quot;L1_S24_F1604&quot;, &quot;L1_S24_F1695&quot;, &quot;L1_S24_F1758&quot;, &quot;L1_S24_F1667&quot;, &quot;L1_S24_F1632&quot;\n</code></pre>\n\n<p>If you need a line-separated vector:</p>\n\n<pre><code>L3_S32_F3850\nL1_S24_F1846\nL1_S24_F1723\nL3_S29_F3348\nL3_S29_F3336\nL3_S30_F3519\nL1_S24_F1844\nL3_S29_F3367\nL3_S30_F3539\nL3_S30_F3809\nL3_S30_F3554\nL3_S29_F3433\nL1_S24_F1604\nL1_S24_F1695\nL1_S24_F1758\nL1_S24_F1667\nL1_S24_F1632\n</code></pre>",
      "rawMarkdown": "Train numeric with 17 features can get you 0.10 MCC with a **simple decision tree** (not even an ensemble, just a decision tree) on 3 fold (yes only 3) cross-validation (see picture for other metrics). Not even tuned for maximum MCC, just went maximizing R^2 which is a proxy to Phi Coefficient (which is MCC in a confusion matrix).\r\n\r\n![enter image description here][1]\r\n\r\nWith Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an \"easy to copy\" format:\r\n\r\n    \"L3_S32_F3850\", \"L1_S24_F1846\", \"L1_S24_F1723\", \"L3_S29_F3348\", \"L3_S29_F3336\", \"L3_S30_F3519\", \"L1_S24_F1844\", \"L3_S29_F3367\", \"L3_S30_F3539\", \"L3_S30_F3809\", \"L3_S30_F3554\", \"L3_S29_F3433\", \"L1_S24_F1604\", \"L1_S24_F1695\", \"L1_S24_F1758\", \"L1_S24_F1667\", \"L1_S24_F1632\"\r\n\r\nIf you need a line-separated vector:\r\n\r\n    L3_S32_F3850\r\n    L1_S24_F1846\r\n    L1_S24_F1723\r\n    L3_S29_F3348\r\n    L3_S29_F3336\r\n    L3_S30_F3519\r\n    L1_S24_F1844\r\n    L3_S29_F3367\r\n    L3_S30_F3539\r\n    L3_S30_F3809\r\n    L3_S30_F3554\r\n    L3_S29_F3433\r\n    L1_S24_F1604\r\n    L1_S24_F1695\r\n    L1_S24_F1758\r\n    L1_S24_F1667\r\n    L1_S24_F1632\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4716/Bosch_17f.jpg",
      "votes": 8
    },
    {
      "id": 133709,
      "postDate": "2016-09-03T13:43:03.313Z",
      "content": "<p>[quote=LMNU;133703]</p>\n\n<p>Hi Laurae, thanks for showing all this! How did you pick these 17 features?</p>\n\n<p>[/quote]</p>\n\n<p>2000 xgboost runs on different subsets of features of the train_numeric set I think if I didn't mess up with other feature selections I have locally.</p>\n\n<p>Detailed log per feature in the attached CSV, column significations below:</p>\n\n<ul>\n<li>Feature name</li>\n<li>Number of occurrences to be randomly picked up for assessing feature in xgboost =&gt; appearance</li>\n<li>How many times the feature was selected (picked up for constructing a tree) once during a xgboost run =&gt; selection</li>\n<li>How many times the feature went over the threshold I manually set (for both domination on other features + MCC threshold) =&gt; hit</li>\n<li>Average hit</li>\n<li>Average gain</li>\n<li>Average gain per hit</li>\n</ul>\n\n<p>See this interactive dashboard: <a href=\"https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&:display_count=no\">https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&amp;:display_count=no</a></p>",
      "rawMarkdown": "[quote=LMNU;133703]\r\n\r\nHi Laurae, thanks for showing all this! How did you pick these 17 features?\r\n\r\n[/quote]\r\n\r\n2000 xgboost runs on different subsets of features of the train_numeric set I think if I didn't mess up with other feature selections I have locally.\r\n\r\nDetailed log per feature in the attached CSV, column significations below:\r\n\r\n* Feature name\r\n* Number of occurrences to be randomly picked up for assessing feature in xgboost => appearance\r\n* How many times the feature was selected (picked up for constructing a tree) once during a xgboost run => selection\r\n* How many times the feature went over the threshold I manually set (for both domination on other features + MCC threshold) => hit\r\n* Average hit\r\n* Average gain\r\n* Average gain per hit\r\n\r\nSee this interactive dashboard: https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&:display_count=no",
      "votes": 3
    },
    {
      "id": 131761,
      "postDate": "2016-08-19T13:18:45.530Z",
      "content": "<p>puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the &quot;list&quot; view in the Environment window, and then you can click on the blue arrow to expand.</p>\n\n<p>With or without RStudio, you can use a series of functions like these:</p>\n\n<pre><code>    class(dates2)          \n    dim(dates2)\n    names(dates2)\n    sapply(dates2, class)  \n    head(dates2)           \n    tail(dates2)    \n    summary(dates2)\n    sapply(dates2, function(y) sum(is.na(y)))\n</code></pre>",
      "rawMarkdown": "puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the \"list\" view in the Environment window, and then you can click on the blue arrow to expand.\r\n\r\nWith or without RStudio, you can use a series of functions like these:\r\n\r\n        class(dates2)          \r\n        dim(dates2)\r\n        names(dates2)\r\n        sapply(dates2, class)  \r\n        head(dates2)           \r\n        tail(dates2)    \r\n        summary(dates2)\r\n        sapply(dates2, function(y) sum(is.na(y)))\r\n",
      "votes": 3
    },
    {
      "id": 132079,
      "postDate": "2016-08-22T12:21:59.800Z",
      "content": "<p>Hi David,\nThe data is split by id\nThe response is in the train_numeric features (last feature)\nHope that helps...</p>",
      "rawMarkdown": "Hi David,\r\nThe data is split by id\r\nThe response is in the train_numeric features (last feature)\r\nHope that helps...",
      "votes": 1
    },
    {
      "id": 131735,
      "postDate": "2016-08-19T09:34:37.067Z",
      "content": "<p>Is there any  package in R  which  can  give us summary of this kind?</p>",
      "rawMarkdown": "Is there any  package in R  which  can  give us summary of this kind?",
      "votes": 1
    },
    {
      "id": 131502,
      "postDate": "2016-08-17T18:23:42.817Z",
      "content": "<p>Nice work. The imbalance in the response calls for an anomaly detection algorithm. Figuring out the important features to keep and handling the large data will be challenging.</p>",
      "rawMarkdown": "Nice work. The imbalance in the response calls for an anomaly detection algorithm. Figuring out the important features to keep and handling the large data will be challenging.",
      "votes": 1
    },
    {
      "id": 131480,
      "postDate": "2016-08-17T15:18:31.093Z",
      "content": "<p>Nice quick post:) </p>\n\n<p>What kind of software are you using there?</p>",
      "rawMarkdown": "Nice quick post:) \r\n\r\nWhat kind of software are you using there?",
      "votes": 1
    },
    {
      "id": 131643,
      "postDate": "2016-08-18T20:36:11.147Z",
      "content": "<p>Is the data set split by time? So are the train observations followed by the test observation?</p>",
      "rawMarkdown": "Is the data set split by time? So are the train observations followed by the test observation?",
      "votes": 2
    },
    {
      "id": 134240,
      "postDate": "2016-09-05T23:40:30.093Z",
      "content": "<p>Could you please detail a bit the evolution of the metric ? How do you get the FP / FN ?</p>",
      "rawMarkdown": "Could you please detail a bit the evolution of the metric ? How do you get the FP / FN ?"
    },
    {
      "id": 133933,
      "postDate": "2016-09-04T14:11:30.893Z",
      "content": "<p>[quote=Jesse Burstr&#246;m;133931]</p>\n\n<p>Did you use stratified subsampling?</p>\n\n<p>[/quote]</p>\n\n<p>I took the first 50000 rows, as outlined in the first post.</p>",
      "rawMarkdown": "[quote=Jesse Burström;133931]\r\n\r\nDid you use stratified subsampling?\r\n\r\n[/quote]\r\n\r\nI took the first 50000 rows, as outlined in the first post."
    },
    {
      "id": 133715,
      "postDate": "2016-09-03T14:09:21.127Z",
      "content": "<p>Thanks! I really appreciate your involvement and sharing on this forum.</p>",
      "rawMarkdown": "Thanks! I really appreciate your involvement and sharing on this forum."
    },
    {
      "id": 133703,
      "postDate": "2016-09-03T12:46:26.867Z",
      "content": "<p>Hi Laurae, thanks for showing all this! How did you pick these 17 features?</p>\n\n<p>[quote=Laurae;132183]</p>\n\n<p>With Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an &quot;easy to copy&quot; format:</p>\n\n<pre><code>&quot;L3_S32_F3850&quot;, &quot;L1_S24_F1846&quot;, &quot;L1_S24_F1723&quot;, &quot;L3_S29_F3348&quot;, &quot;L3_S29_F3336&quot;, &quot;L3_S30_F3519&quot;, &quot;L1_S24_F1844&quot;, &quot;L3_S29_F3367&quot;, &quot;L3_S30_F3539&quot;, &quot;L3_S30_F3809&quot;, &quot;L3_S30_F3554&quot;, &quot;L3_S29_F3433&quot;, &quot;L1_S24_F1604&quot;, &quot;L1_S24_F1695&quot;, &quot;L1_S24_F1758&quot;, &quot;L1_S24_F1667&quot;, &quot;L1_S24_F1632&quot;\n</code></pre>\n\n<p>[/quote]</p>",
      "rawMarkdown": "Hi Laurae, thanks for showing all this! How did you pick these 17 features?\r\n\r\n[quote=Laurae;132183]\r\n\r\nWith Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an \"easy to copy\" format:\r\n\r\n    \"L3_S32_F3850\", \"L1_S24_F1846\", \"L1_S24_F1723\", \"L3_S29_F3348\", \"L3_S29_F3336\", \"L3_S30_F3519\", \"L1_S24_F1844\", \"L3_S29_F3367\", \"L3_S30_F3539\", \"L3_S30_F3809\", \"L3_S30_F3554\", \"L3_S29_F3433\", \"L1_S24_F1604\", \"L1_S24_F1695\", \"L1_S24_F1758\", \"L1_S24_F1667\", \"L1_S24_F1632\"\r\n\r\n[/quote]"
    },
    {
      "id": 132716,
      "postDate": "2016-08-28T17:01:54.783Z",
      "content": "<p>Summary of features, you can try \nmlr package has handy \nsummarizeColumns() </p>",
      "rawMarkdown": "Summary of features, you can try \r\nmlr package has handy \r\nsummarizeColumns() \r\n\r\n\r\n\r\n"
    },
    {
      "id": 132284,
      "postDate": "2016-08-24T08:47:30.230Z",
      "content": "<p>Hi  Laurae</p>\n\n<p>I have tried to insert image by embedding directory.\nCould you how to manage upload image from a  local PC not a website directory?</p>\n\n<p>Best. \nKaige</p>",
      "rawMarkdown": "Hi  Laurae\r\n\r\nI have tried to insert image by embedding directory.\r\nCould you how to manage upload image from a  local PC not a website directory?\r\n\r\nBest. \r\nKaige"
    },
    {
      "id": 132170,
      "postDate": "2016-08-23T08:33:01.717Z",
      "content": "<p>It did, thanks!</p>\n\n<p>[quote=Nathaniel Shimoni;132079]</p>\n\n<p>Hi David,\nThe data is split by id\nThe response is in the train_numeric features (last feature)\nHope that helps...</p>\n\n<p>[/quote]</p>",
      "rawMarkdown": "It did, thanks!\r\n\r\n[quote=Nathaniel Shimoni;132079]\r\n\r\nHi David,\r\nThe data is split by id\r\nThe response is in the train_numeric features (last feature)\r\nHope that helps...\r\n\r\n[/quote]\r\n"
    },
    {
      "id": 132078,
      "postDate": "2016-08-22T12:18:44.927Z",
      "content": "<p>Hi,\nThank for all the information, very useful post. \nI am looking at the data and I am still struggling understanding it: in the training data, which is the column that gives the positive/negative label? \nI cannot find any of the columns that corresponds to the failure response label.\nCan anyone help?\nthanks,\nDavide</p>",
      "rawMarkdown": "Hi,\r\nThank for all the information, very useful post. \r\nI am looking at the data and I am still struggling understanding it: in the training data, which is the column that gives the positive/negative label? \r\nI cannot find any of the columns that corresponds to the failure response label.\r\nCan anyone help?\r\nthanks,\r\nDavide"
    },
    {
      "id": 131931,
      "postDate": "2016-08-20T18:21:34.567Z",
      "content": "<p>[quote=JohnM;131761]</p>\n\n<p>puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the &quot;list&quot; view in the Environment window, and then you can click on the blue arrow to expand.</p>\n\n<p>With or without RStudio, you can use a series of functions like these:</p>\n\n<pre><code>    class(dates2)          \n    dim(dates2)\n    names(dates2)\n    sapply(dates2, class)  \n    head(dates2)           \n    tail(dates2)    \n    summary(dates2)\n    sapply(dates2, function(y) sum(is.na(y)))\n</code></pre>\n\n<p>[/quote]</p>\n\n<p>Thanks a lot John.</p>",
      "rawMarkdown": "[quote=JohnM;131761]\r\n\r\npuneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the \"list\" view in the Environment window, and then you can click on the blue arrow to expand.\r\n\r\nWith or without RStudio, you can use a series of functions like these:\r\n\r\n        class(dates2)          \r\n        dim(dates2)\r\n        names(dates2)\r\n        sapply(dates2, class)  \r\n        head(dates2)           \r\n        tail(dates2)    \r\n        summary(dates2)\r\n        sapply(dates2, function(y) sum(is.na(y)))\r\n\r\n\r\n[/quote]\r\n\r\nThanks a lot John."
    },
    {
      "id": 131508,
      "postDate": "2016-08-17T19:17:35.807Z",
      "content": "<p>Can you elaborate on how to implement anomaly detection algorithm? Which algorithm to use? Will the approach be different from using models like xgboost, Ann etc? </p>",
      "rawMarkdown": "Can you elaborate on how to implement anomaly detection algorithm? Which algorithm to use? Will the approach be different from using models like xgboost, Ann etc? "
    },
    {
      "id": 131498,
      "postDate": "2016-08-17T18:00:11.077Z",
      "content": "<p>Very nice... Thanks for sharing this....</p>",
      "rawMarkdown": "Very nice... Thanks for sharing this....\r\n"
    },
    {
      "id": 131482,
      "postDate": "2016-08-17T15:34:33.777Z",
      "content": "<p>It would need to be the enterprise miner though, since base SAS does not have random forests.</p>",
      "rawMarkdown": "It would need to be the enterprise miner though, since base SAS does not have random forests."
    },
    {
      "id": 131481,
      "postDate": "2016-08-17T15:34:08.947Z",
      "content": "<p>Looks like SAS.</p>",
      "rawMarkdown": "Looks like SAS."
    },
    {
      "id": 133946,
      "postDate": "2016-09-04T14:53:13.287Z",
      "rawMarkdown": "",
      "isDeleted": true
    },
    {
      "id": 133931,
      "postDate": "2016-09-04T14:09:09.280Z",
      "rawMarkdown": "",
      "isDeleted": true
    }
  ],
  "comments": [
    {
      "id": 131516,
      "author_name": "Agir",
      "author_url": "",
      "post_date": "2016-08-17T20:11:06.400000",
      "content": "<p>Anomaly detection is based on calculating probability distribution of training (mostly non-anomalous) data based on each 'important' feature. Then a new example is tested using this distribution. If the probability is below a threshold then the example is anomalous. Usually a Gaussian distribution is assumed for each feature. If the distribution of the feature is non-Gaussian one can transform using log, ^0.5, etc.. Then for each feature estimate the mean and variance based on the training data. The overall probability distribution is the product of the probability distribution of all features.\nSee this video from Andrew Ng\n<a href=\"https://www.youtube.com/watch?v=h5iVXB9mczo\">https://www.youtube.com/watch?v=h5iVXB9mczo</a></p>",
      "votes": 9,
      "replies": []
    },
    {
      "id": 131489,
      "author_name": "Laurae",
      "author_url": "",
      "post_date": "2016-08-17T16:02:56.960000",
      "content": "<p>[quote=raddar;131480]</p>\n\n<p>Nice quick post:) </p>\n\n<p>What kind of software are you using there?</p>\n\n<p>[/quote]</p>\n\n<ul>\n<li>H2O Flow for most (pure GUI based and can run on external memory - can script too and make workbooks).</li>\n<li>Tableau for the MCC table</li>\n<li>JMP (SAS) for the decision tree</li>\n</ul>\n\n<p>I'm still adding things to the main post as there are many things that can be quick to run to look at :p</p>\n\n<p>On the 50,000 subsampling I find it very interesting to get 0.91 AUC with a simple decision tree (5-fold).</p>\n\n<p>Currently running a missing value analysis while doing other things, and it is taking forever (15,000 cases analyzed over 1 hour &gt;_&gt;).</p>\n\n<p>All the images got suddenly broken when embedded, any reason? (fixed)</p>",
      "votes": 7,
      "replies": []
    },
    {
      "id": 132183,
      "author_name": "Laurae",
      "author_url": "",
      "post_date": "2016-08-23T10:19:25",
      "content": "<p>Train numeric with 17 features can get you 0.10 MCC with a <strong>simple decision tree</strong> (not even an ensemble, just a decision tree) on 3 fold (yes only 3) cross-validation (see picture for other metrics). Not even tuned for maximum MCC, just went maximizing R^2 which is a proxy to Phi Coefficient (which is MCC in a confusion matrix).</p>\n\n<p><img src=\"https://www.kaggle.com/blobs/download/forum-message-attachment-files/4716/Bosch_17f.jpg\" alt=\"enter image description here\" title></p>\n\n<p>With Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an &quot;easy to copy&quot; format:</p>\n\n<pre><code>&quot;L3_S32_F3850&quot;, &quot;L1_S24_F1846&quot;, &quot;L1_S24_F1723&quot;, &quot;L3_S29_F3348&quot;, &quot;L3_S29_F3336&quot;, &quot;L3_S30_F3519&quot;, &quot;L1_S24_F1844&quot;, &quot;L3_S29_F3367&quot;, &quot;L3_S30_F3539&quot;, &quot;L3_S30_F3809&quot;, &quot;L3_S30_F3554&quot;, &quot;L3_S29_F3433&quot;, &quot;L1_S24_F1604&quot;, &quot;L1_S24_F1695&quot;, &quot;L1_S24_F1758&quot;, &quot;L1_S24_F1667&quot;, &quot;L1_S24_F1632&quot;\n</code></pre>\n\n<p>If you need a line-separated vector:</p>\n\n<pre><code>L3_S32_F3850\nL1_S24_F1846\nL1_S24_F1723\nL3_S29_F3348\nL3_S29_F3336\nL3_S30_F3519\nL1_S24_F1844\nL3_S29_F3367\nL3_S30_F3539\nL3_S30_F3809\nL3_S30_F3554\nL3_S29_F3433\nL1_S24_F1604\nL1_S24_F1695\nL1_S24_F1758\nL1_S24_F1667\nL1_S24_F1632\n</code></pre>",
      "votes": 8,
      "replies": []
    },
    {
      "id": 133709,
      "author_name": "Laurae",
      "author_url": "",
      "post_date": "2016-09-03T13:43:03.313000",
      "content": "<p>[quote=LMNU;133703]</p>\n\n<p>Hi Laurae, thanks for showing all this! How did you pick these 17 features?</p>\n\n<p>[/quote]</p>\n\n<p>2000 xgboost runs on different subsets of features of the train_numeric set I think if I didn't mess up with other feature selections I have locally.</p>\n\n<p>Detailed log per feature in the attached CSV, column significations below:</p>\n\n<ul>\n<li>Feature name</li>\n<li>Number of occurrences to be randomly picked up for assessing feature in xgboost =&gt; appearance</li>\n<li>How many times the feature was selected (picked up for constructing a tree) once during a xgboost run =&gt; selection</li>\n<li>How many times the feature went over the threshold I manually set (for both domination on other features + MCC threshold) =&gt; hit</li>\n<li>Average hit</li>\n<li>Average gain</li>\n<li>Average gain per hit</li>\n</ul>\n\n<p>See this interactive dashboard: <a href=\"https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&:display_count=no\">https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&amp;:display_count=no</a></p>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 131761,
      "author_name": "JohnM",
      "author_url": "",
      "post_date": "2016-08-19T13:18:45.530000",
      "content": "<p>puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the &quot;list&quot; view in the Environment window, and then you can click on the blue arrow to expand.</p>\n\n<p>With or without RStudio, you can use a series of functions like these:</p>\n\n<pre><code>    class(dates2)          \n    dim(dates2)\n    names(dates2)\n    sapply(dates2, class)  \n    head(dates2)           \n    tail(dates2)    \n    summary(dates2)\n    sapply(dates2, function(y) sum(is.na(y)))\n</code></pre>",
      "votes": 3,
      "replies": []
    },
    {
      "id": 132079,
      "author_name": "Nathaniel Shimoni",
      "author_url": "",
      "post_date": "2016-08-22T12:21:59.800000",
      "content": "<p>Hi David,\nThe data is split by id\nThe response is in the train_numeric features (last feature)\nHope that helps...</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 131735,
      "author_name": "puneeth019",
      "author_url": "",
      "post_date": "2016-08-19T09:34:37.067000",
      "content": "<p>Is there any  package in R  which  can  give us summary of this kind?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 131502,
      "author_name": "Agir",
      "author_url": "",
      "post_date": "2016-08-17T18:23:42.817000",
      "content": "<p>Nice work. The imbalance in the response calls for an anomaly detection algorithm. Figuring out the important features to keep and handling the large data will be challenging.</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 131480,
      "author_name": "raddar",
      "author_url": "",
      "post_date": "2016-08-17T15:18:31.093000",
      "content": "<p>Nice quick post:) </p>\n\n<p>What kind of software are you using there?</p>",
      "votes": 1,
      "replies": []
    },
    {
      "id": 131643,
      "author_name": "AiO",
      "author_url": "",
      "post_date": "2016-08-18T20:36:11.147000",
      "content": "<p>Is the data set split by time? So are the train observations followed by the test observation?</p>",
      "votes": 2,
      "replies": []
    },
    {
      "id": 134240,
      "author_name": "Alchemist",
      "author_url": "",
      "post_date": "2016-09-05T23:40:30.093000",
      "content": "<p>Could you please detail a bit the evolution of the metric ? How do you get the FP / FN ?</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133933,
      "author_name": "Laurae",
      "author_url": "",
      "post_date": "2016-09-04T14:11:30.893000",
      "content": "<p>[quote=Jesse Burstr&#246;m;133931]</p>\n\n<p>Did you use stratified subsampling?</p>\n\n<p>[/quote]</p>\n\n<p>I took the first 50000 rows, as outlined in the first post.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133715,
      "author_name": "LMNU",
      "author_url": "",
      "post_date": "2016-09-03T14:09:21.127000",
      "content": "<p>Thanks! I really appreciate your involvement and sharing on this forum.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133703,
      "author_name": "LMNU",
      "author_url": "",
      "post_date": "2016-09-03T12:46:26.867000",
      "content": "<p>Hi Laurae, thanks for showing all this! How did you pick these 17 features?</p>\n\n<p>[quote=Laurae;132183]</p>\n\n<p>With Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an &quot;easy to copy&quot; format:</p>\n\n<pre><code>&quot;L3_S32_F3850&quot;, &quot;L1_S24_F1846&quot;, &quot;L1_S24_F1723&quot;, &quot;L3_S29_F3348&quot;, &quot;L3_S29_F3336&quot;, &quot;L3_S30_F3519&quot;, &quot;L1_S24_F1844&quot;, &quot;L3_S29_F3367&quot;, &quot;L3_S30_F3539&quot;, &quot;L3_S30_F3809&quot;, &quot;L3_S30_F3554&quot;, &quot;L3_S29_F3433&quot;, &quot;L1_S24_F1604&quot;, &quot;L1_S24_F1695&quot;, &quot;L1_S24_F1758&quot;, &quot;L1_S24_F1667&quot;, &quot;L1_S24_F1632&quot;\n</code></pre>\n\n<p>[/quote]</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 132716,
      "author_name": "Narasimharao ",
      "author_url": "",
      "post_date": "2016-08-28T17:01:54.783000",
      "content": "<p>Summary of features, you can try \nmlr package has handy \nsummarizeColumns() </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 132284,
      "author_name": "Kaige Yang",
      "author_url": "",
      "post_date": "2016-08-24T08:47:30.230000",
      "content": "<p>Hi  Laurae</p>\n\n<p>I have tried to insert image by embedding directory.\nCould you how to manage upload image from a  local PC not a website directory?</p>\n\n<p>Best. \nKaige</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 132170,
      "author_name": "David Frazzetto",
      "author_url": "",
      "post_date": "2016-08-23T08:33:01.717000",
      "content": "<p>It did, thanks!</p>\n\n<p>[quote=Nathaniel Shimoni;132079]</p>\n\n<p>Hi David,\nThe data is split by id\nThe response is in the train_numeric features (last feature)\nHope that helps...</p>\n\n<p>[/quote]</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 132078,
      "author_name": "David Frazzetto",
      "author_url": "",
      "post_date": "2016-08-22T12:18:44.927000",
      "content": "<p>Hi,\nThank for all the information, very useful post. \nI am looking at the data and I am still struggling understanding it: in the training data, which is the column that gives the positive/negative label? \nI cannot find any of the columns that corresponds to the failure response label.\nCan anyone help?\nthanks,\nDavide</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 131931,
      "author_name": "puneeth019",
      "author_url": "",
      "post_date": "2016-08-20T18:21:34.567000",
      "content": "<p>[quote=JohnM;131761]</p>\n\n<p>puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the &quot;list&quot; view in the Environment window, and then you can click on the blue arrow to expand.</p>\n\n<p>With or without RStudio, you can use a series of functions like these:</p>\n\n<pre><code>    class(dates2)          \n    dim(dates2)\n    names(dates2)\n    sapply(dates2, class)  \n    head(dates2)           \n    tail(dates2)    \n    summary(dates2)\n    sapply(dates2, function(y) sum(is.na(y)))\n</code></pre>\n\n<p>[/quote]</p>\n\n<p>Thanks a lot John.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 131508,
      "author_name": "aimlnerd",
      "author_url": "",
      "post_date": "2016-08-17T19:17:35.807000",
      "content": "<p>Can you elaborate on how to implement anomaly detection algorithm? Which algorithm to use? Will the approach be different from using models like xgboost, Ann etc? </p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 131498,
      "author_name": "Sachin Adnaik",
      "author_url": "",
      "post_date": "2016-08-17T18:00:11.077000",
      "content": "<p>Very nice... Thanks for sharing this....</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 131482,
      "author_name": "MichaelLanier",
      "author_url": "",
      "post_date": "2016-08-17T15:34:33.777000",
      "content": "<p>It would need to be the enterprise miner though, since base SAS does not have random forests.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 131481,
      "author_name": "MichaelLanier",
      "author_url": "",
      "post_date": "2016-08-17T15:34:08.947000",
      "content": "<p>Looks like SAS.</p>",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133946,
      "author_name": "",
      "author_url": "",
      "post_date": "2016-09-04T14:53:13.287000",
      "content": "",
      "votes": 0,
      "replies": []
    },
    {
      "id": 133931,
      "author_name": "",
      "author_url": "",
      "post_date": "2016-09-04T14:09:09.280000",
      "content": "",
      "votes": 0,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "131475": "P.S: don't ask me why all images are breaking for no reason on Kaggle, I never get to manage to have them viewable for hours on a post embedded directly\r\n\r\n##Expeditive file analysis part\r\n\r\n###A quick analysis on the label:\r\n\r\n* We are playing with an imbalance ratio of 1:172 (172 negatives for 1 positive): 99.41% accuracy if you predict the negative case only\r\n* 1183747 rows\r\n* 1176868 negative labels\r\n* 6879 positive labels\r\n\r\n![enter image description here][1]\r\n\r\nThe evolution of the metric is the following (True Positive vs True Negative - FP/FN are straightforward):\r\n\r\n![enter image description here][2]\r\n\r\nInteractive MCC: [here at Tableau][3]\r\n\r\n###A quick analysis on train_numeric.csv:\r\n\r\n* Matrix size: [1183747 x 969] (969 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random. See [this webpage][4] for more details\r\n* Sparse\r\n* Contains the label we need to predict\r\n\r\nSneak peak:\r\n\r\n![enter image description here][5]\r\n\r\n###A quick analysis on train_categorical.csv:\r\n\r\n* Matrix size: [1183747 x 2141] (2141 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random. See [this webpage][6] for more details (edit: seems there are import errors when loading the dataset, need to check but these missing value counts are not reliable at all).\r\n* Many columns have 1183747 missing values or near that (which means you can remove a lot of them already!).\r\n* Extremely sparse\r\n* Lot of 0 only\r\n* L1_S24_F7XX to L1_S24_F9XX (replace XX by numbers) seems linked (same zeroes)\r\n* Loads of features with NOTHING inside (just sparse, see below)\r\n\r\nSneak peak:\r\n\r\n![enter image description here][7]\r\n\r\n###A quick analysis on train_data.csv:\r\n\r\n* Matrix size: [1183747 x 1157] (1157 features, 1183747 observations)\r\n* Missing values do not seem to be missing at random.\r\n* Sparse\r\n* Dates are numeric already, seems \"normalized\" overall by days?\r\n\r\nSneak peak:\r\n\r\n![enter image description here][8]\r\n\r\n## Expeditive modeling part\r\n\r\n###Elastic Net on raw numeric data, clearly bad:\r\n\r\n![enter image description here][9]\r\n\r\n###Random Forest on raw numeric data, \"not bad\" but can do much better (10 trees, 7 minutes):\r\n\r\n![enter image description here][10]\r\n![enter image description here][11]\r\n\r\n###Single decision tree with 5-fold validation on the first 50000 observations, with all features:\r\n\r\n![enter image description here][12]\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4680/Bosch_SizeREsp.JPG\r\n  [2]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4688/Bosch_MCC.JPG\r\n  [3]: https://public.tableau.com/views/KaggleMCCMetric/Binned0_01?:embed=n&:display_count=no&:toolbar=no\r\n  [4]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4690/Missing_Train_Numeric.htm\r\n  [5]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4684/Bosch_TrainPic1.JPG\r\n  [6]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4691/Missing_Train_Categorical.htm\r\n  [7]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4692/Bosch_TrainPic2.JPG\r\n  [8]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4685/Bosch_TrainPic3.JPG\r\n  [9]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4681/Bosch_GLM.JPG\r\n  [10]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4682/Bosch_RFAUC.JPG\r\n  [11]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4679/Bosch_RFImp.JPG\r\n  [12]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4687/Bosch_DT.jpg",
    "131516": "Anomaly detection is based on calculating probability distribution of training (mostly non-anomalous) data based on each 'important' feature. Then a new example is tested using this distribution. If the probability is below a threshold then the example is anomalous. Usually a Gaussian distribution is assumed for each feature. If the distribution of the feature is non-Gaussian one can transform using log, ^0.5, etc.. Then for each feature estimate the mean and variance based on the training data. The overall probability distribution is the product of the probability distribution of all features.\r\nSee this video from Andrew Ng\r\nhttps://www.youtube.com/watch?v=h5iVXB9mczo",
    "131489": "[quote=raddar;131480]\r\n\r\nNice quick post:) \r\n\r\nWhat kind of software are you using there?\r\n\r\n[/quote]\r\n\r\n* H2O Flow for most (pure GUI based and can run on external memory - can script too and make workbooks).\r\n* Tableau for the MCC table\r\n* JMP (SAS) for the decision tree\r\n\r\nI'm still adding things to the main post as there are many things that can be quick to run to look at :p\r\n\r\nOn the 50,000 subsampling I find it very interesting to get 0.91 AUC with a simple decision tree (5-fold).\r\n\r\nCurrently running a missing value analysis while doing other things, and it is taking forever (15,000 cases analyzed over 1 hour >_>).\r\n\r\nAll the images got suddenly broken when embedded, any reason? (fixed)",
    "132183": "Train numeric with 17 features can get you 0.10 MCC with a **simple decision tree** (not even an ensemble, just a decision tree) on 3 fold (yes only 3) cross-validation (see picture for other metrics). Not even tuned for maximum MCC, just went maximizing R^2 which is a proxy to Phi Coefficient (which is MCC in a confusion matrix).\r\n\r\n![enter image description here][1]\r\n\r\nWith Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an \"easy to copy\" format:\r\n\r\n    \"L3_S32_F3850\", \"L1_S24_F1846\", \"L1_S24_F1723\", \"L3_S29_F3348\", \"L3_S29_F3336\", \"L3_S30_F3519\", \"L1_S24_F1844\", \"L3_S29_F3367\", \"L3_S30_F3539\", \"L3_S30_F3809\", \"L3_S30_F3554\", \"L3_S29_F3433\", \"L1_S24_F1604\", \"L1_S24_F1695\", \"L1_S24_F1758\", \"L1_S24_F1667\", \"L1_S24_F1632\"\r\n\r\nIf you need a line-separated vector:\r\n\r\n    L3_S32_F3850\r\n    L1_S24_F1846\r\n    L1_S24_F1723\r\n    L3_S29_F3348\r\n    L3_S29_F3336\r\n    L3_S30_F3519\r\n    L1_S24_F1844\r\n    L3_S29_F3367\r\n    L3_S30_F3539\r\n    L3_S30_F3809\r\n    L3_S30_F3554\r\n    L3_S29_F3433\r\n    L1_S24_F1604\r\n    L1_S24_F1695\r\n    L1_S24_F1758\r\n    L1_S24_F1667\r\n    L1_S24_F1632\r\n\r\n\r\n  [1]: https://www.kaggle.com/blobs/download/forum-message-attachment-files/4716/Bosch_17f.jpg",
    "133709": "[quote=LMNU;133703]\r\n\r\nHi Laurae, thanks for showing all this! How did you pick these 17 features?\r\n\r\n[/quote]\r\n\r\n2000 xgboost runs on different subsets of features of the train_numeric set I think if I didn't mess up with other feature selections I have locally.\r\n\r\nDetailed log per feature in the attached CSV, column significations below:\r\n\r\n* Feature name\r\n* Number of occurrences to be randomly picked up for assessing feature in xgboost => appearance\r\n* How many times the feature was selected (picked up for constructing a tree) once during a xgboost run => selection\r\n* How many times the feature went over the threshold I manually set (for both domination on other features + MCC threshold) => hit\r\n* Average hit\r\n* Average gain\r\n* Average gain per hit\r\n\r\nSee this interactive dashboard: https://public.tableau.com/views/BoschFeatureSelectionxgboost/xgboostFeatureSelection?:embed=n&:display_count=no",
    "131761": "puneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the \"list\" view in the Environment window, and then you can click on the blue arrow to expand.\r\n\r\nWith or without RStudio, you can use a series of functions like these:\r\n\r\n        class(dates2)          \r\n        dim(dates2)\r\n        names(dates2)\r\n        sapply(dates2, class)  \r\n        head(dates2)           \r\n        tail(dates2)    \r\n        summary(dates2)\r\n        sapply(dates2, function(y) sum(is.na(y)))\r\n",
    "132079": "Hi David,\r\nThe data is split by id\r\nThe response is in the train_numeric features (last feature)\r\nHope that helps...",
    "131735": "Is there any  package in R  which  can  give us summary of this kind?",
    "131502": "Nice work. The imbalance in the response calls for an anomaly detection algorithm. Figuring out the important features to keep and handling the large data will be challenging.",
    "131480": "Nice quick post:) \r\n\r\nWhat kind of software are you using there?",
    "131643": "Is the data set split by time? So are the train observations followed by the test observation?",
    "134240": "Could you please detail a bit the evolution of the metric ? How do you get the FP / FN ?",
    "133933": "[quote=Jesse Burström;133931]\r\n\r\nDid you use stratified subsampling?\r\n\r\n[/quote]\r\n\r\nI took the first 50000 rows, as outlined in the first post.",
    "133715": "Thanks! I really appreciate your involvement and sharing on this forum.",
    "133703": "Hi Laurae, thanks for showing all this! How did you pick these 17 features?\r\n\r\n[quote=Laurae;132183]\r\n\r\nWith Random Forest if I'm not mistaken, I was getting 0.15+ MCC peak only with these 17 features. Names of features are provided below in an \"easy to copy\" format:\r\n\r\n    \"L3_S32_F3850\", \"L1_S24_F1846\", \"L1_S24_F1723\", \"L3_S29_F3348\", \"L3_S29_F3336\", \"L3_S30_F3519\", \"L1_S24_F1844\", \"L3_S29_F3367\", \"L3_S30_F3539\", \"L3_S30_F3809\", \"L3_S30_F3554\", \"L3_S29_F3433\", \"L1_S24_F1604\", \"L1_S24_F1695\", \"L1_S24_F1758\", \"L1_S24_F1667\", \"L1_S24_F1632\"\r\n\r\n[/quote]",
    "132716": "Summary of features, you can try \r\nmlr package has handy \r\nsummarizeColumns() \r\n\r\n\r\n\r\n",
    "132284": "Hi  Laurae\r\n\r\nI have tried to insert image by embedding directory.\r\nCould you how to manage upload image from a  local PC not a website directory?\r\n\r\nBest. \r\nKaige",
    "132170": "It did, thanks!\r\n\r\n[quote=Nathaniel Shimoni;132079]\r\n\r\nHi David,\r\nThe data is split by id\r\nThe response is in the train_numeric features (last feature)\r\nHope that helps...\r\n\r\n[/quote]\r\n",
    "132078": "Hi,\r\nThank for all the information, very useful post. \r\nI am looking at the data and I am still struggling understanding it: in the training data, which is the column that gives the positive/negative label? \r\nI cannot find any of the columns that corresponds to the failure response label.\r\nCan anyone help?\r\nthanks,\r\nDavide",
    "131931": "[quote=JohnM;131761]\r\n\r\npuneeth019 - I don't know of a specific package, but if you use RStudio, then it will give a good idea of what's in the data frame: dimensions, types, values, etc. I like the \"list\" view in the Environment window, and then you can click on the blue arrow to expand.\r\n\r\nWith or without RStudio, you can use a series of functions like these:\r\n\r\n        class(dates2)          \r\n        dim(dates2)\r\n        names(dates2)\r\n        sapply(dates2, class)  \r\n        head(dates2)           \r\n        tail(dates2)    \r\n        summary(dates2)\r\n        sapply(dates2, function(y) sum(is.na(y)))\r\n\r\n\r\n[/quote]\r\n\r\nThanks a lot John.",
    "131508": "Can you elaborate on how to implement anomaly detection algorithm? Which algorithm to use? Will the approach be different from using models like xgboost, Ann etc? ",
    "131498": "Very nice... Thanks for sharing this....\r\n",
    "131482": "It would need to be the enterprise miner though, since base SAS does not have random forests.",
    "131481": "Looks like SAS.",
    "133946": "",
    "133931": ""
  }
}