{
  "id": 67093,
  "title": "How I dealt with JSON columns [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/67093",
  "author_name": "Diogo Menezes Borges",
  "post_date": "2018-09-28T16:57:24.628000",
  "votes": 2,
  "comment_count": 0,
  "views": 0,
  "content": "<p>In both datasets there are four columns which seem to have JSON data. There might be some interesting data on these columns which we might have difficulties to access to if you don't treat \nthem.</p>\n\n<p>Here's my approach. Any input is more than welcome! Hope it helps!</p>\n\n<pre><code>def string_to_dict(dict_string):\n    # Convert to proper json format, from a string. json.load is for dictionary\n    return json.loads(dict_string)\n\njson_cols = [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]\n\n\ndef create_new_dataset(df):\n    new = df.copy()\n    for col in json_cols:\n        new[col] = new[col].apply(string_to_dict)\n        #the following line will convert the JSON values into series and concatenat to the train dataframe\n        new = pd.concat([new,(pd.io.json.json_normalize(new[col]))], axis=1)\n\n    new = new.drop([\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"], axis = 1)\n\n    return new\n\n\nnew_train = create_new_dataset(train)\nnew_train.to_csv(\"data/new_train.csv\", sep='\\t')\nnew_test = create_new_dataset(test)\nnew_test.to_csv(\"data/new_test.csv\", sep='\\t')\n</code></pre>",
  "messages": [
    {
      "id": 395483,
      "postDate": "2018-09-28T16:57:24.630Z",
      "content": "<p>In both datasets there are four columns which seem to have JSON data. There might be some interesting data on these columns which we might have difficulties to access to if you don't treat \nthem.</p>\n\n<p>Here's my approach. Any input is more than welcome! Hope it helps!</p>\n\n<pre><code>def string_to_dict(dict_string):\n    # Convert to proper json format, from a string. json.load is for dictionary\n    return json.loads(dict_string)\n\njson_cols = [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]\n\n\ndef create_new_dataset(df):\n    new = df.copy()\n    for col in json_cols:\n        new[col] = new[col].apply(string_to_dict)\n        #the following line will convert the JSON values into series and concatenat to the train dataframe\n        new = pd.concat([new,(pd.io.json.json_normalize(new[col]))], axis=1)\n\n    new = new.drop([\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"], axis = 1)\n\n    return new\n\n\nnew_train = create_new_dataset(train)\nnew_train.to_csv(\"data/new_train.csv\", sep='\\t')\nnew_test = create_new_dataset(test)\nnew_test.to_csv(\"data/new_test.csv\", sep='\\t')\n</code></pre>",
      "rawMarkdown": "In both datasets there are four columns which seem to have JSON data. There might be some interesting data on these columns which we might have difficulties to access to if you don't treat \nthem.\n\nHere's my approach. Any input is more than welcome! Hope it helps!\n\n    def string_to_dict(dict_string):\n        # Convert to proper json format, from a string. json.load is for dictionary\n        return json.loads(dict_string)\n    \n    json_cols = [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]\n    \n    \n    def create_new_dataset(df):\n        new = df.copy()\n        for col in json_cols:\n            new[col] = new[col].apply(string_to_dict)\n            #the following line will convert the JSON values into series and concatenat to the train dataframe\n            new = pd.concat([new,(pd.io.json.json_normalize(new[col]))], axis=1)\n    \n        new = new.drop([\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"], axis = 1)\n        \n        return new\n\n\n    new_train = create_new_dataset(train)\n    new_train.to_csv(\"data/new_train.csv\", sep='\\t')\n    new_test = create_new_dataset(test)\n    new_test.to_csv(\"data/new_test.csv\", sep='\\t')\n\n\n\n\n  \n",
      "votes": 2
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "395483": "In both datasets there are four columns which seem to have JSON data. There might be some interesting data on these columns which we might have difficulties to access to if you don't treat \nthem.\n\nHere's my approach. Any input is more than welcome! Hope it helps!\n\n    def string_to_dict(dict_string):\n        # Convert to proper json format, from a string. json.load is for dictionary\n        return json.loads(dict_string)\n    \n    json_cols = [\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"]\n    \n    \n    def create_new_dataset(df):\n        new = df.copy()\n        for col in json_cols:\n            new[col] = new[col].apply(string_to_dict)\n            #the following line will convert the JSON values into series and concatenat to the train dataframe\n            new = pd.concat([new,(pd.io.json.json_normalize(new[col]))], axis=1)\n    \n        new = new.drop([\"device\", \"geoNetwork\",\"totals\",\"trafficSource\"], axis = 1)\n        \n        return new\n\n\n    new_train = create_new_dataset(train)\n    new_train.to_csv(\"data/new_train.csv\", sep='\\t')\n    new_test = create_new_dataset(test)\n    new_test.to_csv(\"data/new_test.csv\", sep='\\t')\n\n\n\n\n  \n"
  }
}