{
  "id": 5605,
  "title": "Extended summary stats for the data provided",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5605",
  "author_name": "",
  "post_date": "2013-08-31T21:27:42.277Z",
  "votes": 33,
  "comment_count": 13,
  "views": 7305,
  "content": "<p><span style=\"line-height: 1.4\">You may use this as a reference to make sure that you got the right no. of rows etc... All of this done inside PostgreSQL (see:&nbsp;http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5606/sql-script-to-load-data-into-postgresql-db)&nbsp;</span></p>\n<p><strong><span style=\"line-height: 1.4\">Train.csv:</span></strong></p>\n<p>No. of rows:&nbsp;6,034,195</p>\n<p>Min. value for Id: 1</p>\n<p>Max. value for Id:&nbsp;6,034,195</p>\n<p>No. of unique tags: 42,048</p>\n<p>No. of occurrences of tags:&nbsp;17,409,994</p>\n<p>Max. no. of tags/question: 5</p>\n<p>Avg. no. of tags/question:&nbsp;2.89</p>\n<p>% of questions with specified no. of tags:</p>\n<p>1 : 13.76<br>2 : 26.65<br>3 : 28.65<br>4 : 19.17<br>5 : 11.77</p>\n<p>Top 10 tags(tag||count||% of all occurrences of tags):</p>\n<p>c#||463526||2.66<br>java||412189||2.37<br>php||392451||2.25<br>javascript||365623||2.10<br>android||320622||1.84<br>jquery||305614||1.76<br>c++||199280||1.14<br>python||184928||1.06<br>iphone||183573||1.05<br>asp.net||177334||1.02</p>\n<p><strong>Test.csv:</strong></p>\n<p>No. of rows:&nbsp;2,013,337</p>\n<p>Min. value for Id:&nbsp;6,034,196</p>\n<p>Max. value for Id:&nbsp;8,047,532</p>",
  "messages": [
    {
      "id": "29867",
      "postDate": "08/31/2013 21:27:42",
      "content": "<p><span style=\"line-height: 1.4\">You may use this as a reference to make sure that you got the right no. of rows etc... All of this done inside PostgreSQL (see:&nbsp;http://www.kaggle.com/c/facebook-recruiting-iii-keyword-extraction/forums/t/5606/sql-script-to-load-data-into-postgresql-db)&nbsp;</span></p>\n<p><strong><span style=\"line-height: 1.4\">Train.csv:</span></strong></p>\n<p>No. of rows:&nbsp;6,034,195</p>\n<p>Min. value for Id: 1</p>\n<p>Max. value for Id:&nbsp;6,034,195</p>\n<p>No. of unique tags: 42,048</p>\n<p>No. of occurrences of tags:&nbsp;17,409,994</p>\n<p>Max. no. of tags/question: 5</p>\n<p>Avg. no. of tags/question:&nbsp;2.89</p>\n<p>% of questions with specified no. of tags:</p>\n<p>1 : 13.76<br>2 : 26.65<br>3 : 28.65<br>4 : 19.17<br>5 : 11.77</p>\n<p>Top 10 tags(tag||count||% of all occurrences of tags):</p>\n<p>c#||463526||2.66<br>java||412189||2.37<br>php||392451||2.25<br>javascript||365623||2.10<br>android||320622||1.84<br>jquery||305614||1.76<br>c++||199280||1.14<br>python||184928||1.06<br>iphone||183573||1.05<br>asp.net||177334||1.02</p>\n<p><strong>Test.csv:</strong></p>\n<p>No. of rows:&nbsp;2,013,337</p>\n<p>Min. value for Id:&nbsp;6,034,196</p>\n<p>Max. value for Id:&nbsp;8,047,532</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "29907",
      "postDate": "09/01/2013 08:53:34",
      "content": "<p>Cool, I got about 42k unique tags as well after parsing the training set with awk and sed.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31303",
      "postDate": "09/21/2013 13:41:13",
      "content": "<p>Few more stats:</p>\n<ol>\n<li>There are 12861 (30.58%) tags with occurrence less than or equal 10</li>\n<li>There are 31910 (75.89%) tags with occurrence less than or equal&nbsp;100</li>\n</ol>\n<p>Complete statistics on every tag can be found <a href=\"https://docs.google.com/spreadsheet/ccc?key=0ArwwCLtKpvwNdGdMTG5xaFRhZE42VS1UcFd3WDZyYmc&usp=sharing\" target=\"_blank\">over here</a>.</p>\n<p>P.S. These computations have been done using Python + Unix</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31344",
      "postDate": "09/22/2013 17:56:18",
      "content": "<p>Shash, dimitrim, Mahendra,</p>\n<p>Thanks for the numbers to cross check!</p>\n<p>From my notebook, the first thing I did a while ago was to just see::</p>\n<p>File Size = 7,253,917,400 ( 7 gigs )</p>\n<p>Lines = 145,447,257&nbsp;&nbsp; (lines are c-strings with a c line terminator&nbsp; ( \\n )</p>\n<p>Longest line = 34,079 chars</p>\n<p>Average line len = 50 chars</p>\n<p>time to just chunk these stats in a 5 line (not optimized, fgets() )&nbsp; c-program: 302 secs (5mins)</p>\n<p>(BTW Excel surrenders rather quickly and silently when faced with this file)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31479",
      "postDate": "09/24/2013 12:17:28",
      "content": "<p>I recently implemented hyperloglog (http://highscalability.com/blog/2012/4/5/big-data-counting-how-to-count-a-billion-distinct-objects-us.html)&nbsp;in go &nbsp;and ran it on Train.csv. Turns out the cardinality is ~41 million.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31539",
      "postDate": "09/25/2013 15:05:21",
      "content": "<p>what were you counting that you got 41 million ?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "31542",
      "postDate": "09/25/2013 15:23:59",
      "content": "<p>unique strings split by strings.Fields(), which i think splits by white space. not a very useful statistic</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "33753",
      "postDate": "11/15/2013 15:30:17",
      "content": "<p>Could someone get the max size of a single element?</p>\n<p>I just started and I would like to understand which is the best approach to process data.</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "34988",
      "postDate": "11/21/2013 09:26:25",
      "content": "<p>I'm trying to get some other statistics from the file.</p>\n<p>This is what I'm getting (processed 2 million ids out of 6 millions):</p>\n<p>average number of words per id: 168 (strangely this values does not change as the processing goes ahead)</p>\n<p>average number of unique words per id: 2</p>\n<p>&nbsp;</p>\n<p>If these numbers are confirmed until the end of the processing, this means that there are nearly 12 million words, that are often repeated in the posts, for a total number of nearly 1 billion words in total.</p>\n<p>For words I mean deleting special characters but keeping the dot and parenthesis (for method names)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35823",
      "postDate": "12/04/2013 13:46:54",
      "content": "<p>I am just starting now. Thanks for these numbers. 41 million unique words makes an exciting contest with its constraints!!</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35865",
      "postDate": "12/05/2013 14:56:48",
      "content": "<p>Here's a plot of the number of times a tag occurs (x-axis) vs the number of tags that this applies to (y-axis):</p>\n<p><img src=\"http://i.imgur.com/KnJPWEq.png\" alt=\"Number of tag occurances\" width=\"410\" height=\"271\"></p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35867",
      "postDate": "12/05/2013 15:15:10",
      "content": "<p>Interesting graph, Tobias. But there is something that is not clear to me: if the number of total unique tags is 42,048 (see the OP and it reflects also my evaluations), the integral of your curve along the whole range of outcomes should be 42k. But I don't understand how it can be if the curve start near 35k.</p>\n<p>Your kind of graph makes me think about a question on the result we need to provide.</p>\n<p>If there are thousands of tags with an occurrence lower than 1000, this means that the most part of tags are present in less than 1 per 1 thousand posts (assuming that the total number of posts is 6 millions). Are we expected to predict also this very uncommon tags?</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35868",
      "postDate": "12/05/2013 15:22:46",
      "content": "<p>oh sorry, I should have been more clear:</p>\n<p>the x-axis is not the number of tags with the exact number of occurrences, but rather the number of tags that have <strong>x occurrences or more</strong>. This way it won't add up to exactly 42k. (but more)</p>",
      "rawMarkdown": "",
      "votes": null
    },
    {
      "id": "35880",
      "postDate": "12/05/2013 20:11:09",
      "content": "<p>Nothing compels you to predict the very rare tags, and you can get an OK score without doing so. But your competition is certainly predicting them. It's part of what makes the problem interesting.</p>",
      "rawMarkdown": "",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 29907,
      "author_name": "dmitrim",
      "author_url": "",
      "post_date": "09/01/2013 08:53:34",
      "content": "<p>Cool, I got about 42k unique tags as well after parsing the training set with awk and sed.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31303,
      "author_name": "mahendrakariya",
      "author_url": "",
      "post_date": "09/21/2013 13:41:13",
      "content": "<p>Few more stats:</p>\n<ol>\n<li>There are 12861 (30.58%) tags with occurrence less than or equal 10</li>\n<li>There are 31910 (75.89%) tags with occurrence less than or equal&nbsp;100</li>\n</ol>\n<p>Complete statistics on every tag can be found <a href=\"https://docs.google.com/spreadsheet/ccc?key=0ArwwCLtKpvwNdGdMTG5xaFRhZE42VS1UcFd3WDZyYmc&usp=sharing\" target=\"_blank\">over here</a>.</p>\n<p>P.S. These computations have been done using Python + Unix</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31344,
      "author_name": "karle1",
      "author_url": "",
      "post_date": "09/22/2013 17:56:18",
      "content": "<p>Shash, dimitrim, Mahendra,</p>\n<p>Thanks for the numbers to cross check!</p>\n<p>From my notebook, the first thing I did a while ago was to just see::</p>\n<p>File Size = 7,253,917,400 ( 7 gigs )</p>\n<p>Lines = 145,447,257&nbsp;&nbsp; (lines are c-strings with a c line terminator&nbsp; ( \\n )</p>\n<p>Longest line = 34,079 chars</p>\n<p>Average line len = 50 chars</p>\n<p>time to just chunk these stats in a 5 line (not optimized, fgets() )&nbsp; c-program: 302 secs (5mins)</p>\n<p>(BTW Excel surrenders rather quickly and silently when faced with this file)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31479,
      "author_name": "foobarbazquux",
      "author_url": "",
      "post_date": "09/24/2013 12:17:28",
      "content": "<p>I recently implemented hyperloglog (http://highscalability.com/blog/2012/4/5/big-data-counting-how-to-count-a-billion-distinct-objects-us.html)&nbsp;in go &nbsp;and ran it on Train.csv. Turns out the cardinality is ~41 million.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31539,
      "author_name": "monsieurcactus",
      "author_url": "",
      "post_date": "09/25/2013 15:05:21",
      "content": "<p>what were you counting that you got 41 million ?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 31542,
      "author_name": "foobarbazquux",
      "author_url": "",
      "post_date": "09/25/2013 15:23:59",
      "content": "<p>unique strings split by strings.Fields(), which i think splits by white space. not a very useful statistic</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 33753,
      "author_name": "sergiotorassa",
      "author_url": "",
      "post_date": "11/15/2013 15:30:17",
      "content": "<p>Could someone get the max size of a single element?</p>\n<p>I just started and I would like to understand which is the best approach to process data.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 34988,
      "author_name": "sergiotorassa",
      "author_url": "",
      "post_date": "11/21/2013 09:26:25",
      "content": "<p>I'm trying to get some other statistics from the file.</p>\n<p>This is what I'm getting (processed 2 million ids out of 6 millions):</p>\n<p>average number of words per id: 168 (strangely this values does not change as the processing goes ahead)</p>\n<p>average number of unique words per id: 2</p>\n<p>&nbsp;</p>\n<p>If these numbers are confirmed until the end of the processing, this means that there are nearly 12 million words, that are often repeated in the posts, for a total number of nearly 1 billion words in total.</p>\n<p>For words I mean deleting special characters but keeping the dot and parenthesis (for method names)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35823,
      "author_name": "kosmos",
      "author_url": "",
      "post_date": "12/04/2013 13:46:54",
      "content": "<p>I am just starting now. Thanks for these numbers. 41 million unique words makes an exciting contest with its constraints!!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35865,
      "author_name": "tdomhan",
      "author_url": "",
      "post_date": "12/05/2013 14:56:48",
      "content": "<p>Here's a plot of the number of times a tag occurs (x-axis) vs the number of tags that this applies to (y-axis):</p>\n<p><img src=\"http://i.imgur.com/KnJPWEq.png\" alt=\"Number of tag occurances\" width=\"410\" height=\"271\"></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35867,
      "author_name": "sergiotorassa",
      "author_url": "",
      "post_date": "12/05/2013 15:15:10",
      "content": "<p>Interesting graph, Tobias. But there is something that is not clear to me: if the number of total unique tags is 42,048 (see the OP and it reflects also my evaluations), the integral of your curve along the whole range of outcomes should be 42k. But I don't understand how it can be if the curve start near 35k.</p>\n<p>Your kind of graph makes me think about a question on the result we need to provide.</p>\n<p>If there are thousands of tags with an occurrence lower than 1000, this means that the most part of tags are present in less than 1 per 1 thousand posts (assuming that the total number of posts is 6 millions). Are we expected to predict also this very uncommon tags?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35868,
      "author_name": "tdomhan",
      "author_url": "",
      "post_date": "12/05/2013 15:22:46",
      "content": "<p>oh sorry, I should have been more clear:</p>\n<p>the x-axis is not the number of tags with the exact number of occurrences, but rather the number of tags that have <strong>x occurrences or more</strong>. This way it won't add up to exactly 42k. (but more)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 35880,
      "author_name": "jfkingiii",
      "author_url": "",
      "post_date": "12/05/2013 20:11:09",
      "content": "<p>Nothing compels you to predict the very rare tags, and you can get an OK score without doing so. But your competition is certainly predicting them. It's part of what makes the problem interesting.</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "29867": "",
    "29907": "",
    "31303": "",
    "31344": "",
    "31479": "",
    "31539": "",
    "31542": "",
    "33753": "",
    "34988": "",
    "35823": "",
    "35865": "",
    "35867": "",
    "35868": "",
    "35880": ""
  },
  "source": "meta"
}