{
  "id": 10852,
  "title": "NULL values in datasets??",
  "url": "/competitions/avazu-ctr-prediction/discussion/10852",
  "author_name": "",
  "post_date": "2014-11-06T12:17:28.737Z",
  "votes": 1,
  "comment_count": 9,
  "views": 3270,
  "content": "<p>Hi All....Did anyone find NULL/NA values in the training and test datasets. It has been informed by contest organizers that NULL values in device_id field have been hashed and appears as &quot;d41d8cd9&quot;. I found -1 in C23 field. Is that supposed to be interpreted as NULL?&nbsp;</p>\n\n<p>Just wondering how NULL values can be detected in different&nbsp;fields.</p>",
  "messages": [
    {
      "id": "57398",
      "postDate": "11/06/2014 12:17:28",
      "content": "<p>Hi All....Did anyone find NULL/NA values in the training and test datasets. It has been informed by contest organizers that NULL values in device_id field have been hashed and appears as &quot;d41d8cd9&quot;. I found -1 in C23 field. Is that supposed to be interpreted as NULL?&nbsp;</p>\n\n<p>Just wondering how NULL values can be detected in different&nbsp;fields.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57436",
      "postDate": "11/06/2014 21:04:47",
      "content": "<p>After filter out the 'd41d8cd9' as NULL values, I found the following:</p>\n<p>'7e5068fc' in `site_category` field&nbsp;has over 70% of the frequency. (38% if NULL isn't filtered)</p>\n<p>'7e5068fc' in `app_category` field has over 60% of the frequency. (22% if NULL isn't filtered)</p>\n<p>'85262c2b' in `app_domain` field has over 60% of the frequency. (24% if NULL isn't filtered)</p>\n<p>All of these take a significant portion of the data, and I start to wonder if these have some special meanings too.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57461",
      "postDate": "11/07/2014 10:29:40",
      "content": "",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57462",
      "postDate": "11/07/2014 10:32:10",
      "content": "<p>yes, all NULL values hashed to d41d8cd9</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "57464",
      "postDate": "11/07/2014 13:23:42",
      "content": "<p>There is no NA or NULL in the dataset...The null values are coded as 'd41d8cd9'. See below % of NULL in different fields:</p>\n<p>1) site_id &nbsp;34.48%<br>2) site_domain 36.40%<br>3) site_category 40.46%<br>4) app_id 65.52%<br>5) app_domain 69.77%<br>6) app_category 66.54%<br>7) device_id 79.86%<br>8) device_model 1.05%</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62045",
      "postDate": "12/29/2014 21:00:46",
      "content": "<p>[quote=Steve Wang;57462]</p>\n<p>yes, all NULL values hashed to d41d8cd9</p>\n<p>[/quote]</p>\n<p>Hi Admin,</p>\n\n<p>I am new to this competition and I would just like to confirm if the only expected value for NULL is d41d8cd9?</p>\n<p>There are several references in this competition that mention different ones per feature...</p>\n\n<p>Thanks in advance</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62076",
      "postDate": "12/30/2014 02:49:41",
      "content": "<p>Hi there,</p>\n<p>Confused with hashed values for NULL. In the train dataset there is no such value 'd41d8cd9' considered as null, I searched for this value in site_id, site_domain and others. The only hashed values, considered as null, I found are the ones suggested by laserwolf in &nbsp;<a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10819/unique-missing-train-and-test-values\">http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10819/unique-missing-train-and-test-values</a>.</p>\n<p>Which are the hashed values for null?</p>\n<p>Thanks in advance</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62092",
      "postDate": "12/30/2014 09:44:40",
      "content": "<p>Thanks, Ofd</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62653",
      "postDate": "01/12/2015 15:46:15",
      "content": "<p>I don't see C23 field, but value -1 occurs in column labeled C20.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "62657",
      "postDate": "01/12/2015 16:24:07",
      "content": "<p>This post is based post revision 1&nbsp;data which has been replaced with rev2 data right now so you may not find C23 field.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 57436,
      "author_name": "",
      "author_url": "",
      "post_date": "11/06/2014 21:04:47",
      "content": "<p>After filter out the 'd41d8cd9' as NULL values, I found the following:</p>\n<p>'7e5068fc' in `site_category` field&nbsp;has over 70% of the frequency. (38% if NULL isn't filtered)</p>\n<p>'7e5068fc' in `app_category` field has over 60% of the frequency. (22% if NULL isn't filtered)</p>\n<p>'85262c2b' in `app_domain` field has over 60% of the frequency. (24% if NULL isn't filtered)</p>\n<p>All of these take a significant portion of the data, and I start to wonder if these have some special meanings too.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57461,
      "author_name": "ronald4",
      "author_url": "",
      "post_date": "11/07/2014 10:29:40",
      "content": "",
      "votes": null,
      "replies": []
    },
    {
      "id": 57462,
      "author_name": "stevewang0",
      "author_url": "",
      "post_date": "11/07/2014 10:32:10",
      "content": "<p>yes, all NULL values hashed to d41d8cd9</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 57464,
      "author_name": "prakash5801",
      "author_url": "",
      "post_date": "11/07/2014 13:23:42",
      "content": "<p>There is no NA or NULL in the dataset...The null values are coded as 'd41d8cd9'. See below % of NULL in different fields:</p>\n<p>1) site_id &nbsp;34.48%<br>2) site_domain 36.40%<br>3) site_category 40.46%<br>4) app_id 65.52%<br>5) app_domain 69.77%<br>6) app_category 66.54%<br>7) device_id 79.86%<br>8) device_model 1.05%</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62045,
      "author_name": "evaristoc",
      "author_url": "",
      "post_date": "12/29/2014 21:00:46",
      "content": "<p>[quote=Steve Wang;57462]</p>\n<p>yes, all NULL values hashed to d41d8cd9</p>\n<p>[/quote]</p>\n<p>Hi Admin,</p>\n\n<p>I am new to this competition and I would just like to confirm if the only expected value for NULL is d41d8cd9?</p>\n<p>There are several references in this competition that mention different ones per feature...</p>\n\n<p>Thanks in advance</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62076,
      "author_name": "ofdomejean",
      "author_url": "",
      "post_date": "12/30/2014 02:49:41",
      "content": "<p>Hi there,</p>\n<p>Confused with hashed values for NULL. In the train dataset there is no such value 'd41d8cd9' considered as null, I searched for this value in site_id, site_domain and others. The only hashed values, considered as null, I found are the ones suggested by laserwolf in &nbsp;<a href=\"http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10819/unique-missing-train-and-test-values\">http://www.kaggle.com/c/avazu-ctr-prediction/forums/t/10819/unique-missing-train-and-test-values</a>.</p>\n<p>Which are the hashed values for null?</p>\n<p>Thanks in advance</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62092,
      "author_name": "evaristoc",
      "author_url": "",
      "post_date": "12/30/2014 09:44:40",
      "content": "<p>Thanks, Ofd</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62653,
      "author_name": "oneoff",
      "author_url": "",
      "post_date": "01/12/2015 15:46:15",
      "content": "<p>I don't see C23 field, but value -1 occurs in column labeled C20.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 62657,
      "author_name": "prakash5801",
      "author_url": "",
      "post_date": "01/12/2015 16:24:07",
      "content": "<p>This post is based post revision 1&nbsp;data which has been replaced with rev2 data right now so you may not find C23 field.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "57398": "",
    "57436": "",
    "57461": "",
    "57462": "",
    "57464": "",
    "62045": "",
    "62076": "",
    "62092": "",
    "62653": "",
    "62657": ""
  },
  "source": "meta"
}