{
  "id": 11179,
  "title": "Variable Occurrence Plots and Intepretation ",
  "url": "/competitions/avazu-ctr-prediction/discussion/11179",
  "author_name": "",
  "post_date": "2014-12-08T23:32:23.803Z",
  "votes": 8,
  "comment_count": 3,
  "views": 1262,
  "content": "<p>Hi Fellow Kagglers,</p>\n<p>I did a variable occurrence count for the training data, in the hope of gaining insights on feature distribution and feature engineering. For every column, I counted the occurrence of each unique value, sorted in descending order, and generated x - log (y) plots.</p>\n<p>Overall I found three types of curves:</p>\n<p>(1). Linear (on a x-log(y) plot)</p>\n<p>app_category (plot attached), site_category, C1, C21 (plot attached)</p>\n<p>(2). Power low</p>\n<p>app_domain (plot attached), app_id, device_id, device_ip (plot attached), device_model</p>\n<p>(3). &quot;S&quot; shape.</p>\n<p>C14, C17 (plot attached), C19, C20, hour (plot attached)</p>\n<p>I attached two example plot for each type. Given my limited knowledge on data exploration, I was hoping the forum would shed some light on the information regarding the property of the variables when seeing these three types of distributions.</p>\n<p>Update: axes are labeled for easy read.</p>",
  "messages": [
    {
      "id": "59843",
      "postDate": "12/08/2014 23:32:23",
      "content": "<p>Hi Fellow Kagglers,</p>\n<p>I did a variable occurrence count for the training data, in the hope of gaining insights on feature distribution and feature engineering. For every column, I counted the occurrence of each unique value, sorted in descending order, and generated x - log (y) plots.</p>\n<p>Overall I found three types of curves:</p>\n<p>(1). Linear (on a x-log(y) plot)</p>\n<p>app_category (plot attached), site_category, C1, C21 (plot attached)</p>\n<p>(2). Power low</p>\n<p>app_domain (plot attached), app_id, device_id, device_ip (plot attached), device_model</p>\n<p>(3). &quot;S&quot; shape.</p>\n<p>C14, C17 (plot attached), C19, C20, hour (plot attached)</p>\n<p>I attached two example plot for each type. Given my limited knowledge on data exploration, I was hoping the forum would shed some light on the information regarding the property of the variables when seeing these three types of distributions.</p>\n<p>Update: axes are labeled for easy read.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59847",
      "postDate": "12/09/2014 00:03:54",
      "content": "<p>I don't quite get your plots. What is x and what is log(y)?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59860",
      "postDate": "12/09/2014 03:24:22",
      "content": "<p>[quote=Little Boat;59847]</p>\n<p>I don't quite get your plots. What is x and what is log(y)?</p>\n<p>[/quote]</p>\n<p>What he has plotted is -&nbsp;</p>\n<p>x-axis shows the unique values of the variable sorted in descending order on their occurrences and y-axis shows the count of that unique value&nbsp;</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "59870",
      "postDate": "12/09/2014 07:28:01",
      "content": "<p>@Little Boat - I just added axes for easier read. &nbsp;</p>\n<p>@Abhishek - thanks! Yes, that was what I meant by these plots.&nbsp;</p>\n<p>For some reason I couldn't&nbsp;find a way to delete my six obsolete plots - would appreciate if someone could show me how...</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 59847,
      "author_name": "xiaozhouwang",
      "author_url": "",
      "post_date": "12/09/2014 00:03:54",
      "content": "<p>I don't quite get your plots. What is x and what is log(y)?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59860,
      "author_name": "abhishekpriyam",
      "author_url": "",
      "post_date": "12/09/2014 03:24:22",
      "content": "<p>[quote=Little Boat;59847]</p>\n<p>I don't quite get your plots. What is x and what is log(y)?</p>\n<p>[/quote]</p>\n<p>What he has plotted is -&nbsp;</p>\n<p>x-axis shows the unique values of the variable sorted in descending order on their occurrences and y-axis shows the count of that unique value&nbsp;</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 59870,
      "author_name": "derekzh",
      "author_url": "",
      "post_date": "12/09/2014 07:28:01",
      "content": "<p>@Little Boat - I just added axes for easier read. &nbsp;</p>\n<p>@Abhishek - thanks! Yes, that was what I meant by these plots.&nbsp;</p>\n<p>For some reason I couldn't&nbsp;find a way to delete my six obsolete plots - would appreciate if someone could show me how...</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "59843": "",
    "59847": "",
    "59860": "",
    "59870": ""
  },
  "source": "meta"
}