{
  "id": 294512,
  "title": "NLTK and the Baseline",
  "url": "/competitions/cs5785-fall-2021-final/discussion/294512",
  "author_name": "",
  "post_date": "2021-12-11T03:45:38.498774Z",
  "votes": 2,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Just as a heads-up, I was inspired from assignment 2 to use NLTK to pre-process the descriptions before running them through the baseline code. </p>\n<p>I stemmed the words and it significantly decreased performance. I was puzzled by this until I realized that word2vec is pre-trained on most words in the English language - and stemming the descriptions turns many words into something that is not in the dictionary! (i.e. studies -&gt; studi), which would render word2vec useless. </p>",
  "messages": [
    {
      "id": "1614403",
      "postDate": "12/11/2021 03:45:38",
      "content": "<p>Just as a heads-up, I was inspired from assignment 2 to use NLTK to pre-process the descriptions before running them through the baseline code. </p>\n<p>I stemmed the words and it significantly decreased performance. I was puzzled by this until I realized that word2vec is pre-trained on most words in the English language - and stemming the descriptions turns many words into something that is not in the dictionary! (i.e. studies -&gt; studi), which would render word2vec useless. </p>",
      "rawMarkdown": "Just as a heads-up, I was inspired from assignment 2 to use NLTK to pre-process the descriptions before running them through the baseline code. \n\nI stemmed the words and it significantly decreased performance. I was puzzled by this until I realized that word2vec is pre-trained on most words in the English language - and stemming the descriptions turns many words into something that is not in the dictionary! (i.e. studies -> studi), which would render word2vec useless.",
      "votes": null
    },
    {
      "id": "1614576",
      "postDate": "12/11/2021 08:50:36",
      "content": "<p>Lemmas can be a reliable approach if you are looking out for the most accurate root words in English or any other language. But given it takes time to process and give the output, with the pros, follows the cons. </p>",
      "rawMarkdown": "Lemmas can be a reliable approach if you are looking out for the most accurate root words in English or any other language. But given it takes time to process and give the output, with the pros, follows the cons.",
      "votes": null
    },
    {
      "id": "1614874",
      "postDate": "12/11/2021 14:30:58",
      "content": "<p>thanks for sharing! We didn't realize that stem would decrease the performance.</p>",
      "rawMarkdown": "thanks for sharing! We didn't realize that stem would decrease the performance.",
      "votes": null
    },
    {
      "id": "1616839",
      "postDate": "12/13/2021 16:55:48",
      "content": "<p>I just get rid of word2vec, and purely use bag of words. This helps the performance a lot. </p>",
      "rawMarkdown": "I just get rid of word2vec, and purely use bag of words. This helps the performance a lot.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1614576,
      "author_name": "saiakhil032",
      "author_url": "",
      "post_date": "12/11/2021 08:50:36",
      "content": "<p>Lemmas can be a reliable approach if you are looking out for the most accurate root words in English or any other language. But given it takes time to process and give the output, with the pros, follows the cons. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1614874,
      "author_name": "siyichen12138",
      "author_url": "",
      "post_date": "12/11/2021 14:30:58",
      "content": "<p>thanks for sharing! We didn't realize that stem would decrease the performance.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1616839,
      "author_name": "candicehe",
      "author_url": "",
      "post_date": "12/13/2021 16:55:48",
      "content": "<p>I just get rid of word2vec, and purely use bag of words. This helps the performance a lot. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1614403": "Just as a heads-up, I was inspired from assignment 2 to use NLTK to pre-process the descriptions before running them through the baseline code. \n\nI stemmed the words and it significantly decreased performance. I was puzzled by this until I realized that word2vec is pre-trained on most words in the English language - and stemming the descriptions turns many words into something that is not in the dictionary! (i.e. studies -> studi), which would render word2vec useless.",
    "1614576": "Lemmas can be a reliable approach if you are looking out for the most accurate root words in English or any other language. But given it takes time to process and give the output, with the pros, follows the cons.",
    "1614874": "thanks for sharing! We didn't realize that stem would decrease the performance.",
    "1616839": "I just get rid of word2vec, and purely use bag of words. This helps the performance a lot."
  },
  "source": "meta"
}