{
  "id": 23022,
  "title": "Out of the doldrums...",
  "url": "/competitions/bosch-production-line-performance/discussion/23022",
  "author_name": "",
  "post_date": "2016-08-21T21:33:29.160Z",
  "votes": 3,
  "comment_count": 2,
  "views": 908,
  "content": "<p>Hi,\nLike many of you I downloaded the data and immediately uploaded the submission file to get my name and the big 'goose egg' onto the leader board.  Sometime early this morning I submitted a file that gave me a small score of 0.03 and got me out of the doldrums.</p>\n\n<p>This is an interesting data set because my system and my skills sets are both taxed to the max. I wanted to build a data pipeline from training-&gt;testing-&gt;submission that would allow me to move up on the leader board. It was also after midnight my time. I wanted to sleep but I also wanted to try something.</p>\n\n<p>I'm sharing my approach in case it is helpful to others. I'm including no code because frankly its all very basic and easy to find online.</p>\n\n<p>NOTE: For fun turn on your system monitoring (e.g. Task Manager) and watch the Memory and CPU during the process.</p>\n\n<ol>\n<li><p>Wrote simple python script to iterate through &quot;train_numerical.csv&quot;. I looked for each Response value of &quot;1&quot;. Saved the entire row to another file. There were 6,879 parts that failed out of 1,183,748 (0.58% failure rate). I then added another 6,879 rows that did not fail. I did not randomize, but chose consecutive rows for non-failures. This is not an ideal training data set but works for this purpose.</p></li>\n<li><p>Loaded the newly created &quot;test_numerical_short.csv&quot; into R.</p></li>\n<li><p>Ran linear regression on data, regressing on all features.</p></li>\n<li><p>Displayed summary of 968 columns from model. Noted statistically significant values.</p></li>\n<li><p>Copied summary text into Excel.  Used formulas to extract feature names if item had *.  NOTE: There are some shortcuts and tricks in R to do this but I didn't want to visit SO.</p></li>\n<li><p>Used VBA to create the R formula string &quot;response ~ ....&quot;. Replaced  last regressors with new list of regressors using as.formula( ) function.</p></li>\n<li><p>Repeated steps 3 through 6 until I whittled the features down to less than 50 features. </p></li>\n<li><p>Cleared everything from R but model. Shut down all apps (including Netflix).</p></li>\n<li><p>Loaded test_numerical_data.csv.</p></li>\n<li><p>Ran the R prediction function against test data. NOTE: I processed only 100,000 rows from the test data against the model at a time. With each prediction I wrote the submission data to a CSV file. Task monitor showed Red for Memory during most of this processing.</p></li>\n<li><p>Submitted file and scored non-zero.</p></li>\n</ol>\n\n<p>Regards,\nChuck</p>",
  "messages": [
    {
      "id": "132031",
      "postDate": "08/21/2016 21:33:29",
      "content": "<p>Hi,\nLike many of you I downloaded the data and immediately uploaded the submission file to get my name and the big 'goose egg' onto the leader board.  Sometime early this morning I submitted a file that gave me a small score of 0.03 and got me out of the doldrums.</p>\n\n<p>This is an interesting data set because my system and my skills sets are both taxed to the max. I wanted to build a data pipeline from training-&gt;testing-&gt;submission that would allow me to move up on the leader board. It was also after midnight my time. I wanted to sleep but I also wanted to try something.</p>\n\n<p>I'm sharing my approach in case it is helpful to others. I'm including no code because frankly its all very basic and easy to find online.</p>\n\n<p>NOTE: For fun turn on your system monitoring (e.g. Task Manager) and watch the Memory and CPU during the process.</p>\n\n<ol>\n<li><p>Wrote simple python script to iterate through &quot;train_numerical.csv&quot;. I looked for each Response value of &quot;1&quot;. Saved the entire row to another file. There were 6,879 parts that failed out of 1,183,748 (0.58% failure rate). I then added another 6,879 rows that did not fail. I did not randomize, but chose consecutive rows for non-failures. This is not an ideal training data set but works for this purpose.</p></li>\n<li><p>Loaded the newly created &quot;test_numerical_short.csv&quot; into R.</p></li>\n<li><p>Ran linear regression on data, regressing on all features.</p></li>\n<li><p>Displayed summary of 968 columns from model. Noted statistically significant values.</p></li>\n<li><p>Copied summary text into Excel.  Used formulas to extract feature names if item had *.  NOTE: There are some shortcuts and tricks in R to do this but I didn't want to visit SO.</p></li>\n<li><p>Used VBA to create the R formula string &quot;response ~ ....&quot;. Replaced  last regressors with new list of regressors using as.formula( ) function.</p></li>\n<li><p>Repeated steps 3 through 6 until I whittled the features down to less than 50 features. </p></li>\n<li><p>Cleared everything from R but model. Shut down all apps (including Netflix).</p></li>\n<li><p>Loaded test_numerical_data.csv.</p></li>\n<li><p>Ran the R prediction function against test data. NOTE: I processed only 100,000 rows from the test data against the model at a time. With each prediction I wrote the submission data to a CSV file. Task monitor showed Red for Memory during most of this processing.</p></li>\n<li><p>Submitted file and scored non-zero.</p></li>\n</ol>\n\n<p>Regards,\nChuck</p>",
      "rawMarkdown": "Hi,\r\nLike many of you I downloaded the data and immediately uploaded the submission file to get my name and the big 'goose egg' onto the leader board.  Sometime early this morning I submitted a file that gave me a small score of 0.03 and got me out of the doldrums.\r\n\r\nThis is an interesting data set because my system and my skills sets are both taxed to the max. I wanted to build a data pipeline from training->testing->submission that would allow me to move up on the leader board. It was also after midnight my time. I wanted to sleep but I also wanted to try something.\r\n\r\nI'm sharing my approach in case it is helpful to others. I'm including no code because frankly its all very basic and easy to find online.\r\n\r\nNOTE: For fun turn on your system monitoring (e.g. Task Manager) and watch the Memory and CPU during the process.\r\n\r\n1. Wrote simple python script to iterate through \"train_numerical.csv\". I looked for each Response value of \"1\". Saved the entire row to another file. There were 6,879 parts that failed out of 1,183,748 (0.58% failure rate). I then added another 6,879 rows that did not fail. I did not randomize, but chose consecutive rows for non-failures. This is not an ideal training data set but works for this purpose.\r\n\r\n2. Loaded the newly created \"test_numerical_short.csv\" into R.\r\n\r\n3. Ran linear regression on data, regressing on all features.\r\n\r\n4. Displayed summary of 968 columns from model. Noted statistically significant values.\r\n\r\n5. Copied summary text into Excel.  Used formulas to extract feature names if item had *.  NOTE: There are some shortcuts and tricks in R to do this but I didn't want to visit SO.\r\n\r\n6. Used VBA to create the R formula string \"response ~ ....\". Replaced  last regressors with new list of regressors using as.formula( ) function.\r\n\r\n7. Repeated steps 3 through 6 until I whittled the features down to less than 50 features. \r\n\r\n8. Cleared everything from R but model. Shut down all apps (including Netflix).\r\n\r\n9. Loaded test_numerical_data.csv.\r\n\r\n10. Ran the R prediction function against test data. NOTE: I processed only 100,000 rows from the test data against the model at a time. With each prediction I wrote the submission data to a CSV file. Task monitor showed Red for Memory during most of this processing.\r\n\r\n11. Submitted file and scored non-zero.\r\n\r\nRegards,\r\nChuck",
      "votes": null
    },
    {
      "id": "132034",
      "postDate": "08/21/2016 22:05:11",
      "content": "<p>Noice!</p>",
      "rawMarkdown": "Noice!",
      "votes": null
    },
    {
      "id": "132179",
      "postDate": "08/23/2016 09:32:51",
      "content": "<p>A very important data science skill is extracting value from your data given real-world constraints. I love the creativity of this approach.</p>",
      "rawMarkdown": "A very important data science skill is extracting value from your data given real-world constraints. I love the creativity of this approach.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132034,
      "author_name": "brentbrewington",
      "author_url": "",
      "post_date": "08/21/2016 22:05:11",
      "content": "<p>Noice!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132179,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "08/23/2016 09:32:51",
      "content": "<p>A very important data science skill is extracting value from your data given real-world constraints. I love the creativity of this approach.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132031": "Hi,\r\nLike many of you I downloaded the data and immediately uploaded the submission file to get my name and the big 'goose egg' onto the leader board.  Sometime early this morning I submitted a file that gave me a small score of 0.03 and got me out of the doldrums.\r\n\r\nThis is an interesting data set because my system and my skills sets are both taxed to the max. I wanted to build a data pipeline from training->testing->submission that would allow me to move up on the leader board. It was also after midnight my time. I wanted to sleep but I also wanted to try something.\r\n\r\nI'm sharing my approach in case it is helpful to others. I'm including no code because frankly its all very basic and easy to find online.\r\n\r\nNOTE: For fun turn on your system monitoring (e.g. Task Manager) and watch the Memory and CPU during the process.\r\n\r\n1. Wrote simple python script to iterate through \"train_numerical.csv\". I looked for each Response value of \"1\". Saved the entire row to another file. There were 6,879 parts that failed out of 1,183,748 (0.58% failure rate). I then added another 6,879 rows that did not fail. I did not randomize, but chose consecutive rows for non-failures. This is not an ideal training data set but works for this purpose.\r\n\r\n2. Loaded the newly created \"test_numerical_short.csv\" into R.\r\n\r\n3. Ran linear regression on data, regressing on all features.\r\n\r\n4. Displayed summary of 968 columns from model. Noted statistically significant values.\r\n\r\n5. Copied summary text into Excel.  Used formulas to extract feature names if item had *.  NOTE: There are some shortcuts and tricks in R to do this but I didn't want to visit SO.\r\n\r\n6. Used VBA to create the R formula string \"response ~ ....\". Replaced  last regressors with new list of regressors using as.formula( ) function.\r\n\r\n7. Repeated steps 3 through 6 until I whittled the features down to less than 50 features. \r\n\r\n8. Cleared everything from R but model. Shut down all apps (including Netflix).\r\n\r\n9. Loaded test_numerical_data.csv.\r\n\r\n10. Ran the R prediction function against test data. NOTE: I processed only 100,000 rows from the test data against the model at a time. With each prediction I wrote the submission data to a CSV file. Task monitor showed Red for Memory during most of this processing.\r\n\r\n11. Submitted file and scored non-zero.\r\n\r\nRegards,\r\nChuck",
    "132034": "Noice!",
    "132179": "A very important data science skill is extracting value from your data given real-world constraints. I love the creativity of this approach."
  },
  "source": "meta"
}