{
  "id": 23029,
  "title": "Question about loading data in Python",
  "url": "/competitions/bosch-production-line-performance/discussion/23029",
  "author_name": "",
  "post_date": "2016-08-22T10:29:03.307Z",
  "votes": null,
  "comment_count": 2,
  "views": 1140,
  "content": "<p>So I've wanted to use what has been described here for the Bosch dataset\n<a href=\"https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\">https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion</a></p>\n\n<p>As the train datasets have ~1000 columns I want to avoid typing it by hand :)</p>\n\n<p>I want to create an array like this where types consists of these L* columns (plus Response and Id variable:</p>\n\n<pre><code>types = {'L0_S0_F0':np.float16,...}\n</code></pre>\n\n<p>So far i was able to create the desired strings as printed statements (and copied the output manually into a new code-cell):</p>\n\n<pre><code>path_to_csv = &quot;../data/train_numeric.csv&quot;\ndf = pd.read_csv(path_to_csv,nrows=20, na_filter=False)\n#first and last column are Id and Response, different format needed\ntst = np.delete(df.columns, (0,np.size(df.columns)-1))\nfor ind in range(0,np.size(tst)-1):\n    print &quot;'&quot;+tst[ind]+&quot;':np.float16,&quot;\nprint &quot;'&quot;+df.columns[-1]+&quot;':np.uint8,&quot;\nprint &quot;'&quot;+df.columns[0]+&quot;':np.uint32&quot;\n</code></pre>\n\n<p>There must be an easy way to directly create an array like this, no?</p>\n\n<p>Why do I want to do this?\nThen I could easily loop through sets of 20 columns, read all observations (!) and make some univariate analyses to reduce dimensions</p>\n\n<p>EDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it</p>",
  "messages": [
    {
      "id": "132064",
      "postDate": "08/22/2016 10:29:03",
      "content": "<p>So I've wanted to use what has been described here for the Bosch dataset\n<a href=\"https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\">https://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion</a></p>\n\n<p>As the train datasets have ~1000 columns I want to avoid typing it by hand :)</p>\n\n<p>I want to create an array like this where types consists of these L* columns (plus Response and Id variable:</p>\n\n<pre><code>types = {'L0_S0_F0':np.float16,...}\n</code></pre>\n\n<p>So far i was able to create the desired strings as printed statements (and copied the output manually into a new code-cell):</p>\n\n<pre><code>path_to_csv = &quot;../data/train_numeric.csv&quot;\ndf = pd.read_csv(path_to_csv,nrows=20, na_filter=False)\n#first and last column are Id and Response, different format needed\ntst = np.delete(df.columns, (0,np.size(df.columns)-1))\nfor ind in range(0,np.size(tst)-1):\n    print &quot;'&quot;+tst[ind]+&quot;':np.float16,&quot;\nprint &quot;'&quot;+df.columns[-1]+&quot;':np.uint8,&quot;\nprint &quot;'&quot;+df.columns[0]+&quot;':np.uint32&quot;\n</code></pre>\n\n<p>There must be an easy way to directly create an array like this, no?</p>\n\n<p>Why do I want to do this?\nThen I could easily loop through sets of 20 columns, read all observations (!) and make some univariate analyses to reduce dimensions</p>\n\n<p>EDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it</p>",
      "rawMarkdown": "So I've wanted to use what has been described here for the Bosch dataset\r\nhttps://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\r\n\r\nAs the train datasets have ~1000 columns I want to avoid typing it by hand :)\r\n\r\nI want to create an array like this where types consists of these L* columns (plus Response and Id variable:\r\n\r\n    types = {'L0_S0_F0':np.float16,...}\r\n\r\nSo far i was able to create the desired strings as printed statements (and copied the output manually into a new code-cell):\r\n\r\n    path_to_csv = \"../data/train_numeric.csv\"\r\n    df = pd.read_csv(path_to_csv,nrows=20, na_filter=False)\r\n    #first and last column are Id and Response, different format needed\r\n    tst = np.delete(df.columns, (0,np.size(df.columns)-1))\r\n    for ind in range(0,np.size(tst)-1):\r\n        print \"'\"+tst[ind]+\"':np.float16,\"\r\n    print \"'\"+df.columns[-1]+\"':np.uint8,\"\r\n    print \"'\"+df.columns[0]+\"':np.uint32\"\r\n\r\n\r\nThere must be an easy way to directly create an array like this, no?\r\n\r\nWhy do I want to do this?\r\nThen I could easily loop through sets of 20 columns, read all observations (!) and make some univariate analyses to reduce dimensions\r\n\r\n\r\nEDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it",
      "votes": null
    },
    {
      "id": "132178",
      "postDate": "08/23/2016 09:20:02",
      "content": "<p>[quote=Simon Preis;132064]</p>\n\n<p>EDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it</p>\n\n<p>[/quote]</p>\n\n<p>Right. Use a dictionary comprehension for each type, updating the master dictionary.</p>\n\n<pre><code>dtype_map = {}\ndtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\ndtype_map.update({feature: np.float16 for feature in float16_feature_list})\n</code></pre>\n\n<p>Etc.</p>",
      "rawMarkdown": "[quote=Simon Preis;132064]\r\n\r\nEDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it\r\n\r\n[/quote]\r\n\r\nRight. Use a dictionary comprehension for each type, updating the master dictionary.\r\n\r\n    dtype_map = {}\r\n    dtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\r\n    dtype_map.update({feature: np.float16 for feature in float16_feature_list})\r\n\r\nEtc.",
      "votes": null
    },
    {
      "id": "132180",
      "postDate": "08/23/2016 09:39:47",
      "content": "<p>[quote=inversion;132178]</p>\n\n<pre><code>dtype_map = {}\ndtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\ndtype_map.update({feature: np.float16 for feature in float16_feature_list})\n</code></pre>\n\n<p>[/quote]</p>\n\n<p>Nice. This code looks way smoother than &quot;my version&quot;. \nI had to create a second list containing all datatypes</p>\n\n<pre><code>names = Variables\nkeys = [np.float16] * 104\nnames_and_keys = zip(names, keys)\ncol_dict = {}\nfor name, keys in names_and_keys:\n    col_dict[name] = keys\ncol_dict['Response'] =np.uint8\ncol_dict['Id'] =np.uint32\n</code></pre>",
      "rawMarkdown": "[quote=inversion;132178]\r\n\r\n\r\n    dtype_map = {}\r\n    dtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\r\n    dtype_map.update({feature: np.float16 for feature in float16_feature_list})\r\n\r\n[/quote]\r\n\r\nNice. This code looks way smoother than \"my version\". \r\nI had to create a second list containing all datatypes\r\n\r\n    names = Variables\r\n    keys = [np.float16] * 104\r\n    names_and_keys = zip(names, keys)\r\n    col_dict = {}\r\n    for name, keys in names_and_keys:\r\n        col_dict[name] = keys\r\n    col_dict['Response'] =np.uint8\r\n    col_dict['Id'] =np.uint32",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 132178,
      "author_name": "inversion",
      "author_url": "",
      "post_date": "08/23/2016 09:20:02",
      "content": "<p>[quote=Simon Preis;132064]</p>\n\n<p>EDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it</p>\n\n<p>[/quote]</p>\n\n<p>Right. Use a dictionary comprehension for each type, updating the master dictionary.</p>\n\n<pre><code>dtype_map = {}\ndtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\ndtype_map.update({feature: np.float16 for feature in float16_feature_list})\n</code></pre>\n\n<p>Etc.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 132180,
      "author_name": "simonp",
      "author_url": "",
      "post_date": "08/23/2016 09:39:47",
      "content": "<p>[quote=inversion;132178]</p>\n\n<pre><code>dtype_map = {}\ndtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\ndtype_map.update({feature: np.float16 for feature in float16_feature_list})\n</code></pre>\n\n<p>[/quote]</p>\n\n<p>Nice. This code looks way smoother than &quot;my version&quot;. \nI had to create a second list containing all datatypes</p>\n\n<pre><code>names = Variables\nkeys = [np.float16] * 104\nnames_and_keys = zip(names, keys)\ncol_dict = {}\nfor name, keys in names_and_keys:\n    col_dict[name] = keys\ncol_dict['Response'] =np.uint8\ncol_dict['Id'] =np.uint32\n</code></pre>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "132064": "So I've wanted to use what has been described here for the Bosch dataset\r\nhttps://www.kaggle.com/ericcouto/grupo-bimbo-inventory-demand/using-82-less-memory/discussion\r\n\r\nAs the train datasets have ~1000 columns I want to avoid typing it by hand :)\r\n\r\nI want to create an array like this where types consists of these L* columns (plus Response and Id variable:\r\n\r\n    types = {'L0_S0_F0':np.float16,...}\r\n\r\nSo far i was able to create the desired strings as printed statements (and copied the output manually into a new code-cell):\r\n\r\n    path_to_csv = \"../data/train_numeric.csv\"\r\n    df = pd.read_csv(path_to_csv,nrows=20, na_filter=False)\r\n    #first and last column are Id and Response, different format needed\r\n    tst = np.delete(df.columns, (0,np.size(df.columns)-1))\r\n    for ind in range(0,np.size(tst)-1):\r\n        print \"'\"+tst[ind]+\"':np.float16,\"\r\n    print \"'\"+df.columns[-1]+\"':np.uint8,\"\r\n    print \"'\"+df.columns[0]+\"':np.uint32\"\r\n\r\n\r\nThere must be an easy way to directly create an array like this, no?\r\n\r\nWhy do I want to do this?\r\nThen I could easily loop through sets of 20 columns, read all observations (!) and make some univariate analyses to reduce dimensions\r\n\r\n\r\nEDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it",
    "132178": "[quote=Simon Preis;132064]\r\n\r\nEDIT: Nevermind. One simply has to create a dictionary. If there is any interest I will post my script on how I solved it\r\n\r\n[/quote]\r\n\r\nRight. Use a dictionary comprehension for each type, updating the master dictionary.\r\n\r\n    dtype_map = {}\r\n    dtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\r\n    dtype_map.update({feature: np.float16 for feature in float16_feature_list})\r\n\r\nEtc.",
    "132180": "[quote=inversion;132178]\r\n\r\n\r\n    dtype_map = {}\r\n    dtype_map.update({feature: np.uint8 for feature in uint8_feature_list})\r\n    dtype_map.update({feature: np.float16 for feature in float16_feature_list})\r\n\r\n[/quote]\r\n\r\nNice. This code looks way smoother than \"my version\". \r\nI had to create a second list containing all datatypes\r\n\r\n    names = Variables\r\n    keys = [np.float16] * 104\r\n    names_and_keys = zip(names, keys)\r\n    col_dict = {}\r\n    for name, keys in names_and_keys:\r\n        col_dict[name] = keys\r\n    col_dict['Response'] =np.uint8\r\n    col_dict['Id'] =np.uint32"
  },
  "source": "meta"
}