{
  "id": 347785,
  "title": "Black magic in the wifi's numpy file?",
  "url": "/competitions/ai-village-ctf/discussion/347785",
  "author_name": "",
  "post_date": "2022-08-25T11:10:05.610918Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>I'd like to understand what kind of black magic is happening here. </p>\n<p>This code</p>\n<pre><code>wifi = np.load(\"data/wifi/Embedded_characters.npz\")\nwifi['tokens']\n</code></pre>\n<p>produces this output</p>\n<pre><code>array('!!\"\"##$$%%&amp;&amp;\\'\\'(())**++,,--..//00112233445566778899::;;&lt;&lt;==&gt;&gt;??@@AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTTUUVVWWXXYYZZ[[\\\\\\\\]]^^__``aabbccddeeffgghhiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~',\n      dtype='&lt;U188')\n</code></pre>\n<p>What baffles me beyond comprehension is that this is an empty array. <code>wifi['tokens'].shape == ()</code> and <code>wifi['tokens'][0]</code> throws an <code>IndexError</code>. </p>\n<p>So, if the array is empty, where do all those characters come from? What on Earth is <code>dtype='&lt;U188'</code> (Unicode-string of length 188?) And why can I access the characters of an empty array with <code>np.array_str(wifi['tokens'])</code>?</p>\n<p>My best guess: We are using numpy versions which are subtly incompatible.</p>",
  "messages": [
    {
      "id": "1913555",
      "postDate": "08/25/2022 11:10:05",
      "content": "<p>I'd like to understand what kind of black magic is happening here. </p>\n<p>This code</p>\n<pre><code>wifi = np.load(\"data/wifi/Embedded_characters.npz\")\nwifi['tokens']\n</code></pre>\n<p>produces this output</p>\n<pre><code>array('!!\"\"##$$%%&amp;&amp;\\'\\'(())**++,,--..//00112233445566778899::;;&lt;&lt;==&gt;&gt;??@@AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTTUUVVWWXXYYZZ[[\\\\\\\\]]^^__``aabbccddeeffgghhiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~',\n      dtype='&lt;U188')\n</code></pre>\n<p>What baffles me beyond comprehension is that this is an empty array. <code>wifi['tokens'].shape == ()</code> and <code>wifi['tokens'][0]</code> throws an <code>IndexError</code>. </p>\n<p>So, if the array is empty, where do all those characters come from? What on Earth is <code>dtype='&lt;U188'</code> (Unicode-string of length 188?) And why can I access the characters of an empty array with <code>np.array_str(wifi['tokens'])</code>?</p>\n<p>My best guess: We are using numpy versions which are subtly incompatible.</p>",
      "rawMarkdown": "I'd like to understand what kind of black magic is happening here. \n\nThis code\n\n```\nwifi = np.load(\"data/wifi/Embedded_characters.npz\")\nwifi['tokens']\n```\nproduces this output\n\r```\narray('!!\"\"##$$%%&&\\'\\'(())**++,,--..//00112233445566778899::;;<<==>>??@@AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTTUUVVWWXXYYZZ[[\\\\\\\\]]^^__``aabbccddeeffgghhiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~',\n      dtype='<U188')\n```\n\nWhat baffles me beyond comprehension is that this is an empty array. `wifi['tokens'].shape == ()` and `wifi['tokens'][0]` throws an `IndexError`. \n\nSo, if the array is empty, where do all those characters come from? What on Earth is `dtype='<U188'` (Unicode-string of length 188?) And why can I access the characters of an empty array with `np.array_str(wifi['tokens'])`?\n\nMy best guess: We are using numpy versions which are subtly incompatible.",
      "votes": null
    },
    {
      "id": "1913601",
      "postDate": "08/25/2022 11:51:19",
      "content": "<p>No, it's a scalar string value in numpy, mostly because it was loaded from an npz file. Your version of numpy is probably fine. And you can use the .item() method on the scalar to get its value</p>\n<p><a href=\"https://numpy.org/doc/stable/reference/arrays.scalars.html\" target=\"_blank\">https://numpy.org/doc/stable/reference/arrays.scalars.html</a></p>\n<p><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html\" target=\"_blank\">https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html</a></p>",
      "rawMarkdown": "No, it's a scalar string value in numpy, mostly because it was loaded from an npz file. Your version of numpy is probably fine. And you can use the .item() method on the scalar to get its value\n\nhttps://numpy.org/doc/stable/reference/arrays.scalars.html\n\nhttps://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html",
      "votes": null
    },
    {
      "id": "1913614",
      "postDate": "08/25/2022 12:03:38",
      "content": "<p>It's a zero-dimensional numpy array.  Think of it as a scalar (so you can't index it and it doesn't have a shape).  In this case, it's fine to cast it to a string <code>str(wifi['tokens'])</code> is probably what you want.</p>",
      "rawMarkdown": "It's a zero-dimensional numpy array.  Think of it as a scalar (so you can't index it and it doesn't have a shape).  In this case, it's fine to cast it to a string `str(wifi['tokens'])` is probably what you want.",
      "votes": null
    },
    {
      "id": "1913725",
      "postDate": "08/25/2022 13:19:29",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/andrewrrose\" target=\"_blank\">@andrewrrose</a> and <a href=\"https://www.kaggle.com/vwoodnz\" target=\"_blank\">@vwoodnz</a> ! What I still don't undestand: How do you insert a string into an array so that the array stays empty? In other words, how can a 0-dimensional array contain <em>anything</em>?</p>",
      "rawMarkdown": "Thank you @andrewrrose and @vwoodnz ! What I still don't undestand: How do you insert a string into an array so that the array stays empty? In other words, how can a 0-dimensional array contain _anything_?",
      "votes": null
    },
    {
      "id": "1913854",
      "postDate": "08/25/2022 14:42:12",
      "content": "<p>I was in the same situation a couple of days back until I looked into the numpy documentation about scalar arrays. I think we can probably just ignore/forget about the type being shown as array..<br>\nJust do <code>.tolist()</code> and you will get the string that you can play around (I am guessing that there is probably some numpy way of accessing the chars, idk)</p>",
      "rawMarkdown": "I was in the same situation a couple of days back until I looked into the numpy documentation about scalar arrays. I think we can probably just ignore/forget about the type being shown as array..\nJust do `.tolist()` and you will get the string that you can play around (I am guessing that there is probably some numpy way of accessing the chars, idk)",
      "votes": null
    },
    {
      "id": "1914002",
      "postDate": "08/25/2022 16:26:28",
      "content": "<p>Another way to extract the string from it would be this strange sintax.<br>\n<code>tokens=wifi[\"tokens\"][()]</code></p>",
      "rawMarkdown": "Another way to extract the string from it would be this strange sintax.\n`tokens=wifi[\"tokens\"][()]`",
      "votes": null
    },
    {
      "id": "1914020",
      "postDate": "08/25/2022 16:49:19",
      "content": "<p>to extract the items just call .item():<br>\n<code>npfile= np.load('../input/ai-village-ctf/wifi/Embedded_characters.npz')</code><br>\n<code>tokens = npfile['tokens'].item()</code></p>",
      "rawMarkdown": "to extract the items just call .item():\n```npfile= np.load('../input/ai-village-ctf/wifi/Embedded_characters.npz')```\n```tokens = npfile['tokens'].item()```",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1913601,
      "author_name": "vwoodnz",
      "author_url": "",
      "post_date": "08/25/2022 11:51:19",
      "content": "<p>No, it's a scalar string value in numpy, mostly because it was loaded from an npz file. Your version of numpy is probably fine. And you can use the .item() method on the scalar to get its value</p>\n<p><a href=\"https://numpy.org/doc/stable/reference/arrays.scalars.html\" target=\"_blank\">https://numpy.org/doc/stable/reference/arrays.scalars.html</a></p>\n<p><a href=\"https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html\" target=\"_blank\">https://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1913614,
      "author_name": "andrewrrose",
      "author_url": "",
      "post_date": "08/25/2022 12:03:38",
      "content": "<p>It's a zero-dimensional numpy array.  Think of it as a scalar (so you can't index it and it doesn't have a shape).  In this case, it's fine to cast it to a string <code>str(wifi['tokens'])</code> is probably what you want.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1913725,
      "author_name": "lesamu",
      "author_url": "",
      "post_date": "08/25/2022 13:19:29",
      "content": "<p>Thank you <a href=\"https://www.kaggle.com/andrewrrose\" target=\"_blank\">@andrewrrose</a> and <a href=\"https://www.kaggle.com/vwoodnz\" target=\"_blank\">@vwoodnz</a> ! What I still don't undestand: How do you insert a string into an array so that the array stays empty? In other words, how can a 0-dimensional array contain <em>anything</em>?</p>",
      "votes": null,
      "replies": [
        {
          "id": 1913854,
          "author_name": "vangap",
          "author_url": "",
          "post_date": "08/25/2022 14:42:12",
          "content": "<p>I was in the same situation a couple of days back until I looked into the numpy documentation about scalar arrays. I think we can probably just ignore/forget about the type being shown as array..<br>\nJust do <code>.tolist()</code> and you will get the string that you can play around (I am guessing that there is probably some numpy way of accessing the chars, idk)</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1914002,
      "author_name": "pietromaldini1",
      "author_url": "",
      "post_date": "08/25/2022 16:26:28",
      "content": "<p>Another way to extract the string from it would be this strange sintax.<br>\n<code>tokens=wifi[\"tokens\"][()]</code></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1914020,
      "author_name": "titericz",
      "author_url": "",
      "post_date": "08/25/2022 16:49:19",
      "content": "<p>to extract the items just call .item():<br>\n<code>npfile= np.load('../input/ai-village-ctf/wifi/Embedded_characters.npz')</code><br>\n<code>tokens = npfile['tokens'].item()</code></p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1913555": "I'd like to understand what kind of black magic is happening here. \n\nThis code\n\n```\nwifi = np.load(\"data/wifi/Embedded_characters.npz\")\nwifi['tokens']\n```\nproduces this output\n\r```\narray('!!\"\"##$$%%&&\\'\\'(())**++,,--..//00112233445566778899::;;<<==>>??@@AABBCCDDEEFFGGHHIIJJKKLLMMNNOOPPQQRRSSTTUUVVWWXXYYZZ[[\\\\\\\\]]^^__``aabbccddeeffgghhiijjkkllmmnnooppqqrrssttuuvvwwxxyyzz{{||}}~~',\n      dtype='<U188')\n```\n\nWhat baffles me beyond comprehension is that this is an empty array. `wifi['tokens'].shape == ()` and `wifi['tokens'][0]` throws an `IndexError`. \n\nSo, if the array is empty, where do all those characters come from? What on Earth is `dtype='<U188'` (Unicode-string of length 188?) And why can I access the characters of an empty array with `np.array_str(wifi['tokens'])`?\n\nMy best guess: We are using numpy versions which are subtly incompatible.",
    "1913601": "No, it's a scalar string value in numpy, mostly because it was loaded from an npz file. Your version of numpy is probably fine. And you can use the .item() method on the scalar to get its value\n\nhttps://numpy.org/doc/stable/reference/arrays.scalars.html\n\nhttps://numpy.org/doc/stable/reference/generated/numpy.ndarray.item.html",
    "1913614": "It's a zero-dimensional numpy array.  Think of it as a scalar (so you can't index it and it doesn't have a shape).  In this case, it's fine to cast it to a string `str(wifi['tokens'])` is probably what you want.",
    "1913725": "Thank you @andrewrrose and @vwoodnz ! What I still don't undestand: How do you insert a string into an array so that the array stays empty? In other words, how can a 0-dimensional array contain _anything_?",
    "1913854": "I was in the same situation a couple of days back until I looked into the numpy documentation about scalar arrays. I think we can probably just ignore/forget about the type being shown as array..\nJust do `.tolist()` and you will get the string that you can play around (I am guessing that there is probably some numpy way of accessing the chars, idk)",
    "1914002": "Another way to extract the string from it would be this strange sintax.\n`tokens=wifi[\"tokens\"][()]`",
    "1914020": "to extract the items just call .item():\n```npfile= np.load('../input/ai-village-ctf/wifi/Embedded_characters.npz')```\n```tokens = npfile['tokens'].item()```"
  },
  "source": "meta"
}