{
  "id": 244988,
  "title": "How does stemming & stop words presence/absence impact LB score?",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/244988",
  "author_name": "",
  "post_date": "2021-06-09T07:25:16.998970300Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Would like to know, peoples experience on LB score, for predictions with stop words(the, of, and etc) or without stop words?</p>\n<p>Does stemming impact F1 score against the actual labales??</p>\n<p>Your answers will help me plan Feature Engineering &amp; tokenization(1-gram or bi-gram) to train BERT NER.</p>\n<p>Thanks</p>",
  "messages": [
    {
      "id": "1342106",
      "postDate": "06/09/2021 07:25:17",
      "content": "<p>Would like to know, peoples experience on LB score, for predictions with stop words(the, of, and etc) or without stop words?</p>\n<p>Does stemming impact F1 score against the actual labales??</p>\n<p>Your answers will help me plan Feature Engineering &amp; tokenization(1-gram or bi-gram) to train BERT NER.</p>\n<p>Thanks</p>",
      "rawMarkdown": "Would like to know, peoples experience on LB score, for predictions with stop words(the, of, and etc) or without stop words?\n\nDoes stemming impact F1 score against the actual labales??\n\nYour answers will help me plan Feature Engineering & tokenization(1-gram or bi-gram) to train BERT NER.\n\nThanks",
      "votes": null
    },
    {
      "id": "1345618",
      "postDate": "06/11/2021 17:53:24",
      "content": "<ol>\n<li>Stopwords:<br>\nlabels contains stop words-- stop words removal impact the your score(it give low score). <br>\nExample: your label_1:  is :    'noaa sea lake and overland surges from hurricanes'  here stopword are [\"and\",\"from\"]<br>\nstop word removed label is : 'noaa sea lake overland surges hurricanes' <br>\nlabel_2: 'noaa sea lake is overland surges for hurricanes'  here stopword are [\"is\",\"for\"]<br>\nstop word removed label is : 'noaa sea lake overland surges hurricanes' </li>\n</ol>\n<p>actually without stopword removal the label_1 and label_2 different, once you removed stopwords both are same.</p>\n<p>2.Stemming:<br>\nApplying stemming will improve the F1 score. it brings the words into root form so its better to apply.</p>",
      "rawMarkdown": "1. Stopwords:\nlabels contains stop words-- stop words removal impact the your score(it give low score). \nExample: your label_1:  is :    'noaa sea lake and overland surges from hurricanes'  here stopword are [\"and\",\"from\"]\nstop word removed label is : 'noaa sea lake overland surges hurricanes' \nlabel_2: 'noaa sea lake is overland surges for hurricanes'  here stopword are [\"is\",\"for\"]\nstop word removed label is : 'noaa sea lake overland surges hurricanes' \n\nactually without stopword removal the label_1 and label_2 different, once you removed stopwords both are same.\n\n2.Stemming:\nApplying stemming will improve the F1 score. it brings the words into root form so its better to apply.",
      "votes": null
    },
    {
      "id": "1351605",
      "postDate": "06/16/2021 13:04:10",
      "content": "<p>Adding on to <a href=\"https://www.kaggle.com/mrparamatma\" target=\"_blank\">@mrparamatma</a> . I actually did submit a Notebook after Lemmatization (Just playing around and forgot to use the original text without lemmatization version to submit), LB Score only tanked by <strong>0.001</strong>. Then again I did this with a Bi-LSTM model without string matching and considering the \"Hu~~~uuuge\" size of the visible test dataset, this really doesn't mean much.</p>\n<p>All the best for your model training</p>",
      "rawMarkdown": "Adding on to @mrparamatma . I actually did submit a Notebook after Lemmatization (Just playing around and forgot to use the original text without lemmatization version to submit), LB Score only tanked by **0.001**. Then again I did this with a Bi-LSTM model without string matching and considering the \"Hu~~~uuuge\" size of the visible test dataset, this really doesn't mean much.\n\nAll the best for your model training",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1345618,
      "author_name": "mrparamatma",
      "author_url": "",
      "post_date": "06/11/2021 17:53:24",
      "content": "<ol>\n<li>Stopwords:<br>\nlabels contains stop words-- stop words removal impact the your score(it give low score). <br>\nExample: your label_1:  is :    'noaa sea lake and overland surges from hurricanes'  here stopword are [\"and\",\"from\"]<br>\nstop word removed label is : 'noaa sea lake overland surges hurricanes' <br>\nlabel_2: 'noaa sea lake is overland surges for hurricanes'  here stopword are [\"is\",\"for\"]<br>\nstop word removed label is : 'noaa sea lake overland surges hurricanes' </li>\n</ol>\n<p>actually without stopword removal the label_1 and label_2 different, once you removed stopwords both are same.</p>\n<p>2.Stemming:<br>\nApplying stemming will improve the F1 score. it brings the words into root form so its better to apply.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1351605,
          "author_name": "toxicmaze",
          "author_url": "",
          "post_date": "06/16/2021 13:04:10",
          "content": "<p>Adding on to <a href=\"https://www.kaggle.com/mrparamatma\" target=\"_blank\">@mrparamatma</a> . I actually did submit a Notebook after Lemmatization (Just playing around and forgot to use the original text without lemmatization version to submit), LB Score only tanked by <strong>0.001</strong>. Then again I did this with a Bi-LSTM model without string matching and considering the \"Hu~~~uuuge\" size of the visible test dataset, this really doesn't mean much.</p>\n<p>All the best for your model training</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1342106": "Would like to know, peoples experience on LB score, for predictions with stop words(the, of, and etc) or without stop words?\n\nDoes stemming impact F1 score against the actual labales??\n\nYour answers will help me plan Feature Engineering & tokenization(1-gram or bi-gram) to train BERT NER.\n\nThanks",
    "1345618": "1. Stopwords:\nlabels contains stop words-- stop words removal impact the your score(it give low score). \nExample: your label_1:  is :    'noaa sea lake and overland surges from hurricanes'  here stopword are [\"and\",\"from\"]\nstop word removed label is : 'noaa sea lake overland surges hurricanes' \nlabel_2: 'noaa sea lake is overland surges for hurricanes'  here stopword are [\"is\",\"for\"]\nstop word removed label is : 'noaa sea lake overland surges hurricanes' \n\nactually without stopword removal the label_1 and label_2 different, once you removed stopwords both are same.\n\n2.Stemming:\nApplying stemming will improve the F1 score. it brings the words into root form so its better to apply.",
    "1351605": "Adding on to @mrparamatma . I actually did submit a Notebook after Lemmatization (Just playing around and forgot to use the original text without lemmatization version to submit), LB Score only tanked by **0.001**. Then again I did this with a Bi-LSTM model without string matching and considering the \"Hu~~~uuuge\" size of the visible test dataset, this really doesn't mean much.\n\nAll the best for your model training"
  },
  "source": "meta"
}