{
  "id": 241281,
  "title": "CRF + Tensorflow",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/241281",
  "author_name": "",
  "post_date": "2021-05-23T23:13:12.353920900Z",
  "votes": 3,
  "comment_count": 9,
  "views": 0,
  "content": "<p>Hello guys, </p>\n<p>(NER approach here)<br>\nI'm posting this to create a discussion for those who use <strong>Conditional Random Fields in Tensorflow</strong> (since online there is not much documentation on CRF on TF)</p>\n<p>I've discovered that using a CRF layer + CRF loss, instead of just a fully connected with Softmax, helped a lot with the predictions and boosted my score. However, we have the 9 hour limit, and as it seems the CRF layer on tensorflow addons is slowing down inference (x3 times slower) - from 50ms per sentence to 150ms per sentence</p>\n<p>So due to the time limit, I can only use distilbert-base and not bert-base - and generally I'm very limited to the overall architecture of the model, since I'm on the edge of the limit.</p>\n<p>Is anyone else experiencing this issue and has some insight to share? Online there is not much, and I'm guessing that maybe here there are many of us, who are dealing with the same problems.<br>\nCheers!</p>",
  "messages": [
    {
      "id": "1320244",
      "postDate": "05/23/2021 23:13:12",
      "content": "<p>Hello guys, </p>\n<p>(NER approach here)<br>\nI'm posting this to create a discussion for those who use <strong>Conditional Random Fields in Tensorflow</strong> (since online there is not much documentation on CRF on TF)</p>\n<p>I've discovered that using a CRF layer + CRF loss, instead of just a fully connected with Softmax, helped a lot with the predictions and boosted my score. However, we have the 9 hour limit, and as it seems the CRF layer on tensorflow addons is slowing down inference (x3 times slower) - from 50ms per sentence to 150ms per sentence</p>\n<p>So due to the time limit, I can only use distilbert-base and not bert-base - and generally I'm very limited to the overall architecture of the model, since I'm on the edge of the limit.</p>\n<p>Is anyone else experiencing this issue and has some insight to share? Online there is not much, and I'm guessing that maybe here there are many of us, who are dealing with the same problems.<br>\nCheers!</p>",
      "rawMarkdown": "Hello guys, \n\n(NER approach here)\nI'm posting this to create a discussion for those who use **Conditional Random Fields in Tensorflow** (since online there is not much documentation on CRF on TF)\n\nI've discovered that using a CRF layer + CRF loss, instead of just a fully connected with Softmax, helped a lot with the predictions and boosted my score. However, we have the 9 hour limit, and as it seems the CRF layer on tensorflow addons is slowing down inference (x3 times slower) - from 50ms per sentence to 150ms per sentence\n\nSo due to the time limit, I can only use distilbert-base and not bert-base - and generally I'm very limited to the overall architecture of the model, since I'm on the edge of the limit.\n\nIs anyone else experiencing this issue and has some insight to share? Online there is not much, and I'm guessing that maybe here there are many of us, who are dealing with the same problems.\nCheers!",
      "votes": null
    },
    {
      "id": "1320297",
      "postDate": "05/24/2021 01:34:06",
      "content": "<p>great!! its great that you managed to get some results with kaggle limited resources. I tried but couldnt get the results with in the limited time frames. Could you pls number your results?</p>",
      "rawMarkdown": "great!! its great that you managed to get some results with kaggle limited resources. I tried but couldnt get the results with in the limited time frames. Could you pls number your results?",
      "votes": null
    },
    {
      "id": "1320610",
      "postDate": "05/24/2021 07:36:19",
      "content": "<p>try this:<a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716\" target=\"_blank\">https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716</a>    ,maybe it works .and glad to see that still people are using ner with crf,good luck for you .</p>",
      "rawMarkdown": "try this:https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716    ,maybe it works .and glad to see that still people are using ner with crf,good luck for you .",
      "votes": null
    },
    {
      "id": "1321455",
      "postDate": "05/24/2021 17:34:14",
      "content": "<p>You could try managing your memory tighter so that you can use a larger batch size. </p>",
      "rawMarkdown": "You could try managing your memory tighter so that you can use a larger batch size.",
      "votes": null
    },
    {
      "id": "1321831",
      "postDate": "05/25/2021 03:00:20",
      "content": "<p>Thanks for sharing!</p>\n<p>I also used a HuggingFace transformers model (model size: distilBERT &lt; a transformer model I used &lt; BERT) + some other layers + CRF.<br>\nMy inference kernel finished in about an hour with batch size 128.</p>",
      "rawMarkdown": "Thanks for sharing!\n\nI also used a HuggingFace transformers model (model size: distilBERT < a transformer model I used < BERT) + some other layers + CRF.\nMy inference kernel finished in about an hour with batch size 128.",
      "votes": null
    },
    {
      "id": "1322363",
      "postDate": "05/25/2021 12:11:40",
      "content": "<p>thanks for sharing. i will try those methods</p>",
      "rawMarkdown": "thanks for sharing. i will try those methods",
      "votes": null
    },
    {
      "id": "1323901",
      "postDate": "05/26/2021 13:48:01",
      "content": "<p>Good to see people still solving this as ner problem. Given that we only have 180 tagged datasets in the training with limited words combination and the testing data contains many unseen words and phrases, maybe it is worthy trying to focus on the context instead of trying to recognize the dataset names…</p>",
      "rawMarkdown": "Good to see people still solving this as ner problem. Given that we only have 180 tagged datasets in the training with limited words combination and the testing data contains many unseen words and phrases, maybe it is worthy trying to focus on the context instead of trying to recognize the dataset names...",
      "votes": null
    },
    {
      "id": "1327091",
      "postDate": "05/28/2021 22:17:36",
      "content": "<p>Any tips on dealing with class imbalance and using CRFs? I tried weighting the emission scores with class weights but that didn't work at all (and just doesn't feel right). I've been using this <a href=\"https://github.com/kmkurn/pytorch-crf/tree/8f3203a1f1d7984c87718bfe31853242670258db\" target=\"_blank\">implementation</a>. Thanks in advance!</p>",
      "rawMarkdown": "Any tips on dealing with class imbalance and using CRFs? I tried weighting the emission scores with class weights but that didn't work at all (and just doesn't feel right). I've been using this [implementation](https://github.com/kmkurn/pytorch-crf/tree/8f3203a1f1d7984c87718bfe31853242670258db). Thanks in advance!",
      "votes": null
    },
    {
      "id": "1327165",
      "postDate": "05/29/2021 02:33:14",
      "content": "<p>In my opinion, class weights are not useful for model using the CRF layer because this layer calculate the dependencies between classes(e.g. The sequence \"B-I-O\" is more probable than the sequence \"I-B-O\".).</p>\n<p>Instead, you can use sample weights between sequence included the named entities or not or negative down sampling.</p>",
      "rawMarkdown": "In my opinion, class weights are not useful for model using the CRF layer because this layer calculate the dependencies between classes(e.g. The sequence \"B-I-O\" is more probable than the sequence \"I-B-O\".).\n\nInstead, you can use sample weights between sequence included the named entities or not or negative down sampling.",
      "votes": null
    },
    {
      "id": "1328022",
      "postDate": "05/29/2021 21:14:16",
      "content": "<p>Thank you!</p>",
      "rawMarkdown": "Thank you!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1320297,
      "author_name": "udaygurugubelli",
      "author_url": "",
      "post_date": "05/24/2021 01:34:06",
      "content": "<p>great!! its great that you managed to get some results with kaggle limited resources. I tried but couldnt get the results with in the limited time frames. Could you pls number your results?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1320610,
      "author_name": "coffeebear98",
      "author_url": "",
      "post_date": "05/24/2021 07:36:19",
      "content": "<p>try this:<a href=\"https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716\" target=\"_blank\">https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716</a>    ,maybe it works .and glad to see that still people are using ner with crf,good luck for you .</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1321455,
      "author_name": "nbroad",
      "author_url": "",
      "post_date": "05/24/2021 17:34:14",
      "content": "<p>You could try managing your memory tighter so that you can use a larger batch size. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1321831,
      "author_name": "spshota",
      "author_url": "",
      "post_date": "05/25/2021 03:00:20",
      "content": "<p>Thanks for sharing!</p>\n<p>I also used a HuggingFace transformers model (model size: distilBERT &lt; a transformer model I used &lt; BERT) + some other layers + CRF.<br>\nMy inference kernel finished in about an hour with batch size 128.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1322363,
      "author_name": "udaygurugubelli",
      "author_url": "",
      "post_date": "05/25/2021 12:11:40",
      "content": "<p>thanks for sharing. i will try those methods</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1323901,
      "author_name": "xiaonanji",
      "author_url": "",
      "post_date": "05/26/2021 13:48:01",
      "content": "<p>Good to see people still solving this as ner problem. Given that we only have 180 tagged datasets in the training with limited words combination and the testing data contains many unseen words and phrases, maybe it is worthy trying to focus on the context instead of trying to recognize the dataset names…</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1327091,
      "author_name": "lichena",
      "author_url": "",
      "post_date": "05/28/2021 22:17:36",
      "content": "<p>Any tips on dealing with class imbalance and using CRFs? I tried weighting the emission scores with class weights but that didn't work at all (and just doesn't feel right). I've been using this <a href=\"https://github.com/kmkurn/pytorch-crf/tree/8f3203a1f1d7984c87718bfe31853242670258db\" target=\"_blank\">implementation</a>. Thanks in advance!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1327165,
          "author_name": "spshota",
          "author_url": "",
          "post_date": "05/29/2021 02:33:14",
          "content": "<p>In my opinion, class weights are not useful for model using the CRF layer because this layer calculate the dependencies between classes(e.g. The sequence \"B-I-O\" is more probable than the sequence \"I-B-O\".).</p>\n<p>Instead, you can use sample weights between sequence included the named entities or not or negative down sampling.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 1328022,
          "author_name": "lichena",
          "author_url": "",
          "post_date": "05/29/2021 21:14:16",
          "content": "<p>Thank you!</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1320244": "Hello guys, \n\n(NER approach here)\nI'm posting this to create a discussion for those who use **Conditional Random Fields in Tensorflow** (since online there is not much documentation on CRF on TF)\n\nI've discovered that using a CRF layer + CRF loss, instead of just a fully connected with Softmax, helped a lot with the predictions and boosted my score. However, we have the 9 hour limit, and as it seems the CRF layer on tensorflow addons is slowing down inference (x3 times slower) - from 50ms per sentence to 150ms per sentence\n\nSo due to the time limit, I can only use distilbert-base and not bert-base - and generally I'm very limited to the overall architecture of the model, since I'm on the edge of the limit.\n\nIs anyone else experiencing this issue and has some insight to share? Online there is not much, and I'm guessing that maybe here there are many of us, who are dealing with the same problems.\nCheers!",
    "1320297": "great!! its great that you managed to get some results with kaggle limited resources. I tried but couldnt get the results with in the limited time frames. Could you pls number your results?",
    "1320610": "try this:https://www.kaggle.com/c/riiid-test-answer-prediction/discussion/198414#1085716    ,maybe it works .and glad to see that still people are using ner with crf,good luck for you .",
    "1321455": "You could try managing your memory tighter so that you can use a larger batch size.",
    "1321831": "Thanks for sharing!\n\nI also used a HuggingFace transformers model (model size: distilBERT < a transformer model I used < BERT) + some other layers + CRF.\nMy inference kernel finished in about an hour with batch size 128.",
    "1322363": "thanks for sharing. i will try those methods",
    "1323901": "Good to see people still solving this as ner problem. Given that we only have 180 tagged datasets in the training with limited words combination and the testing data contains many unseen words and phrases, maybe it is worthy trying to focus on the context instead of trying to recognize the dataset names...",
    "1327091": "Any tips on dealing with class imbalance and using CRFs? I tried weighting the emission scores with class weights but that didn't work at all (and just doesn't feel right). I've been using this [implementation](https://github.com/kmkurn/pytorch-crf/tree/8f3203a1f1d7984c87718bfe31853242670258db). Thanks in advance!",
    "1327165": "In my opinion, class weights are not useful for model using the CRF layer because this layer calculate the dependencies between classes(e.g. The sequence \"B-I-O\" is more probable than the sequence \"I-B-O\".).\n\nInstead, you can use sample weights between sequence included the named entities or not or negative down sampling.",
    "1328022": "Thank you!"
  },
  "source": "meta"
}