{
  "id": 6148,
  "title": "Using tm package for R against question document",
  "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/6148",
  "author_name": "David Cho",
  "post_date": "2013-10-29T07:31:27.217000",
  "votes": 0,
  "comment_count": 0,
  "views": 910,
  "content": "<p>Wondering if someone had already gone down this road in clustering 2 or 3 unigrams/words across all the question documents using the CRAN tm package. Is it effective? &nbsp;I've tried the openNLP maxent models and it's not so effective and it doesn't run particularly fast either. &nbsp;The code snippets in the questions are causing much consternation because straight unigram association to tags produces 'noisy' unigrams which are difficult to distinguish from word stems that come from coding such as 'at' which is a popular Linux command and a tag word but also a popular preposition. &nbsp;One way around this is to find the unigram clusters around 'at', for example, which would allow me to drop 'noisy' &nbsp;'at' and so train my model better. I have way tooooooooo many question unigrams and one puny laptop, so filtering out the noise is to also reduce the number of conditional probabilities I have on tags based on a given set of unigrams and/or clusters. Any insights would be very much appreciated! Thanks!</p>",
  "messages": [
    {
      "id": 32817,
      "postDate": "2013-10-29T07:31:27.217Z",
      "content": "<p>Wondering if someone had already gone down this road in clustering 2 or 3 unigrams/words across all the question documents using the CRAN tm package. Is it effective? &nbsp;I've tried the openNLP maxent models and it's not so effective and it doesn't run particularly fast either. &nbsp;The code snippets in the questions are causing much consternation because straight unigram association to tags produces 'noisy' unigrams which are difficult to distinguish from word stems that come from coding such as 'at' which is a popular Linux command and a tag word but also a popular preposition. &nbsp;One way around this is to find the unigram clusters around 'at', for example, which would allow me to drop 'noisy' &nbsp;'at' and so train my model better. I have way tooooooooo many question unigrams and one puny laptop, so filtering out the noise is to also reduce the number of conditional probabilities I have on tags based on a given set of unigrams and/or clusters. Any insights would be very much appreciated! Thanks!</p>"
    }
  ],
  "comments": [],
  "raw_markdown_by_id": {
    "32817": ""
  }
}