{
  "id": 23137,
  "title": "Useful Linux csv pre-processing commands ",
  "url": "/competitions/bosch-production-line-performance/discussion/23137",
  "author_name": "",
  "post_date": "2016-08-28T13:46:49.837Z",
  "votes": 25,
  "comment_count": 4,
  "views": 1155,
  "content": "<p>Here are some Linux commands which I found useful and efficient for csv pre-processing. Hope they are also useful for you.</p>\n\n<p><strong>extract id and response from train_numeric (remove columns 2 to 969)</strong></p>\n\n<pre><code>cut --complement -f 2-969 -d, train_numeric.csv &gt; train_id_response.csv\n</code></pre>\n\n<p><strong>extract response from train_numeric (remove columns 1 to 969)</strong></p>\n\n<pre><code>cut --complement -f 1-969 -d, train_numeric.csv &gt; train_response.csv\n</code></pre>\n\n<p><strong>add response to the end of train_categorical</strong></p>\n\n<pre><code>paste -d, train_categorical.csv train_response.csv &gt; train_categorical_response.csv\n</code></pre>\n\n<p><strong>add response to the end of train_date</strong></p>\n\n<pre><code>paste -d, train_date.csv train_response.csv &gt; train_date_response.csv\n</code></pre>",
  "messages": [
    {
      "id": "132689",
      "postDate": "08/28/2016 13:46:49",
      "content": "<p>Here are some Linux commands which I found useful and efficient for csv pre-processing. Hope they are also useful for you.</p>\n\n<p><strong>extract id and response from train_numeric (remove columns 2 to 969)</strong></p>\n\n<pre><code>cut --complement -f 2-969 -d, train_numeric.csv &gt; train_id_response.csv\n</code></pre>\n\n<p><strong>extract response from train_numeric (remove columns 1 to 969)</strong></p>\n\n<pre><code>cut --complement -f 1-969 -d, train_numeric.csv &gt; train_response.csv\n</code></pre>\n\n<p><strong>add response to the end of train_categorical</strong></p>\n\n<pre><code>paste -d, train_categorical.csv train_response.csv &gt; train_categorical_response.csv\n</code></pre>\n\n<p><strong>add response to the end of train_date</strong></p>\n\n<pre><code>paste -d, train_date.csv train_response.csv &gt; train_date_response.csv\n</code></pre>",
      "rawMarkdown": "Here are some Linux commands which I found useful and efficient for csv pre-processing. Hope they are also useful for you.\r\n\r\n**extract id and response from train_numeric (remove columns 2 to 969)**\r\n\r\n    cut --complement -f 2-969 -d, train_numeric.csv > train_id_response.csv\r\n\r\n**extract response from train_numeric (remove columns 1 to 969)**\r\n\r\n    cut --complement -f 1-969 -d, train_numeric.csv > train_response.csv\r\n\r\n**add response to the end of train_categorical**\r\n\r\n    paste -d, train_categorical.csv train_response.csv > train_categorical_response.csv\r\n\r\n**add response to the end of train_date**\r\n\r\n    paste -d, train_date.csv train_response.csv > train_date_response.csv",
      "votes": null
    },
    {
      "id": "133114",
      "postDate": "08/31/2016 10:58:25",
      "content": "<p>@Leochu,</p>\n\n<p>thanks a lot.  On Mac OSX the <code>--complement</code> option isn't available apparently, but we can still get the same with a slight variation:</p>\n\n<p><strong>extract id and response from train_numeric (keep columns 1 and 970)</strong></p>\n\n<pre><code>cut -f 1,970 -d , train_numeric.csv &gt; train_id_response.csv\n</code></pre>\n\n<p><strong>extract response from train_numeric (keep column 970)</strong></p>\n\n<pre><code>cut -f 970 -d , train_numeric.csv &gt; train_response.csv\n</code></pre>\n\n<p><code>paste</code> works the same way as Linux.</p>",
      "rawMarkdown": "Leochu,\r\n\r\nthanks a lot.  On Mac OSX the `--complement` option isn't available apparently, but we can still get the same with a slight variation:\r\n\r\n**extract id and response from train_numeric (keep columns 1 and 970)**\r\n\r\n    cut -f 1,970 -d , train_numeric.csv > train_id_response.csv\r\n\r\n**extract response from train_numeric (keep column 970)**\r\n\r\n    cut -f 970 -d , train_numeric.csv > train_response.csv\r\n\r\n`paste` works the same way as Linux.",
      "votes": null
    },
    {
      "id": "133319",
      "postDate": "09/01/2016 13:41:16",
      "content": "<p>Great, that's more readable.</p>\n\n<p>I wrote a Python script to generate cut command for selecting required features from training/testing data.</p>\n\n<p>You just have to run the following script,</p>\n\n<pre><code>python3 gen_cut_cmd.py test_numeric.csv\n</code></pre>\n\n<p>And then input your required features, </p>\n\n<pre><code>L3_S30_F3659\nL3_S30_F3664\nL3_S30_F3669\nL3_S30_F3674\n</code></pre>\n\n<p>The script will generate cut command for you.</p>\n\n<pre><code>cut -d, -f811,812,813,814 test_numeric.csv\n</code></pre>",
      "rawMarkdown": "Great, that's more readable.\r\n\r\nI wrote a Python script to generate cut command for selecting required features from training/testing data.\r\n\r\nYou just have to run the following script,\r\n\r\n    python3 gen_cut_cmd.py test_numeric.csv\r\n\r\nAnd then input your required features, \r\n\r\n    L3_S30_F3659\r\n    L3_S30_F3664\r\n    L3_S30_F3669\r\n    L3_S30_F3674\r\n\r\nThe script will generate cut command for you.\r\n\r\n    cut -d, -f811,812,813,814 test_numeric.csv",
      "votes": null
    },
    {
      "id": "137724",
      "postDate": "10/04/2016 13:13:13",
      "content": "<p>it would be nice to learn these codes.</p>",
      "rawMarkdown": "it would be nice to learn these codes.",
      "votes": null
    },
    {
      "id": "137740",
      "postDate": "10/04/2016 15:09:17",
      "content": "<p>On Windows 10, you can run those scripts from the OP by enabling Bash (<a href=\"https://msdn.microsoft.com/en-au/commandline/wsl/install_guide\">https://msdn.microsoft.com/en-au/commandline/wsl/install_guide</a>) It works as well :)</p>",
      "rawMarkdown": "On Windows 10, you can run those scripts from the OP by enabling Bash (https://msdn.microsoft.com/en-au/commandline/wsl/install_guide) It works as well :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 133114,
      "author_name": "cpmpml",
      "author_url": "",
      "post_date": "08/31/2016 10:58:25",
      "content": "<p>@Leochu,</p>\n\n<p>thanks a lot.  On Mac OSX the <code>--complement</code> option isn't available apparently, but we can still get the same with a slight variation:</p>\n\n<p><strong>extract id and response from train_numeric (keep columns 1 and 970)</strong></p>\n\n<pre><code>cut -f 1,970 -d , train_numeric.csv &gt; train_id_response.csv\n</code></pre>\n\n<p><strong>extract response from train_numeric (keep column 970)</strong></p>\n\n<pre><code>cut -f 970 -d , train_numeric.csv &gt; train_response.csv\n</code></pre>\n\n<p><code>paste</code> works the same way as Linux.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 133319,
      "author_name": "leochu",
      "author_url": "",
      "post_date": "09/01/2016 13:41:16",
      "content": "<p>Great, that's more readable.</p>\n\n<p>I wrote a Python script to generate cut command for selecting required features from training/testing data.</p>\n\n<p>You just have to run the following script,</p>\n\n<pre><code>python3 gen_cut_cmd.py test_numeric.csv\n</code></pre>\n\n<p>And then input your required features, </p>\n\n<pre><code>L3_S30_F3659\nL3_S30_F3664\nL3_S30_F3669\nL3_S30_F3674\n</code></pre>\n\n<p>The script will generate cut command for you.</p>\n\n<pre><code>cut -d, -f811,812,813,814 test_numeric.csv\n</code></pre>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137724,
      "author_name": "guobinliu",
      "author_url": "",
      "post_date": "10/04/2016 13:13:13",
      "content": "<p>it would be nice to learn these codes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 137740,
      "author_name": "",
      "author_url": "",
      "post_date": "10/04/2016 15:09:17",
      "content": "<p>On Windows 10, you can run those scripts from the OP by enabling Bash (<a href=\"https://msdn.microsoft.com/en-au/commandline/wsl/install_guide\">https://msdn.microsoft.com/en-au/commandline/wsl/install_guide</a>) It works as well :)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132689": "Here are some Linux commands which I found useful and efficient for csv pre-processing. Hope they are also useful for you.\r\n\r\n**extract id and response from train_numeric (remove columns 2 to 969)**\r\n\r\n    cut --complement -f 2-969 -d, train_numeric.csv > train_id_response.csv\r\n\r\n**extract response from train_numeric (remove columns 1 to 969)**\r\n\r\n    cut --complement -f 1-969 -d, train_numeric.csv > train_response.csv\r\n\r\n**add response to the end of train_categorical**\r\n\r\n    paste -d, train_categorical.csv train_response.csv > train_categorical_response.csv\r\n\r\n**add response to the end of train_date**\r\n\r\n    paste -d, train_date.csv train_response.csv > train_date_response.csv",
    "133114": "Leochu,\r\n\r\nthanks a lot.  On Mac OSX the `--complement` option isn't available apparently, but we can still get the same with a slight variation:\r\n\r\n**extract id and response from train_numeric (keep columns 1 and 970)**\r\n\r\n    cut -f 1,970 -d , train_numeric.csv > train_id_response.csv\r\n\r\n**extract response from train_numeric (keep column 970)**\r\n\r\n    cut -f 970 -d , train_numeric.csv > train_response.csv\r\n\r\n`paste` works the same way as Linux.",
    "133319": "Great, that's more readable.\r\n\r\nI wrote a Python script to generate cut command for selecting required features from training/testing data.\r\n\r\nYou just have to run the following script,\r\n\r\n    python3 gen_cut_cmd.py test_numeric.csv\r\n\r\nAnd then input your required features, \r\n\r\n    L3_S30_F3659\r\n    L3_S30_F3664\r\n    L3_S30_F3669\r\n    L3_S30_F3674\r\n\r\nThe script will generate cut command for you.\r\n\r\n    cut -d, -f811,812,813,814 test_numeric.csv",
    "137724": "it would be nice to learn these codes.",
    "137740": "On Windows 10, you can run those scripts from the OP by enabling Bash (https://msdn.microsoft.com/en-au/commandline/wsl/install_guide) It works as well :)"
  },
  "source": "meta"
}