{
  "id": 230341,
  "title": "Data preparation for NER",
  "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/230341",
  "author_name": "",
  "post_date": "2021-04-03T05:44:36.406801800Z",
  "votes": 8,
  "comment_count": 1,
  "views": 0,
  "content": "<p>Hi, I have prepared a dataset for training NER for this problem taking this sample ner dataset as <a href=\"https://www.kaggle.com/namanj27/ner-dataset\" target=\"_blank\">reference</a>.</p>\n<p>The data format is</p>\n<pre><code>The --&gt; o\ndata --&gt;o\nis --&gt;o\ntaken --&gt;o\nfrom --&gt;o\nIndian --&gt;o-dataset\nmysteries --&gt;o-dataset\ndataset --&gt;o-dataset\n</code></pre>\n<p>The code is used to do it</p>\n<pre><code>def get_text(filename, test=False):\n    if test:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n    else:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n    text = \" \".join(list(df['text']))\n    return text\n\n\ndef convert_tokens(x,m,train=True):\n    df = pd.DataFrame()\n    text = x['text'].replace('\\uf0b7','').split()\n    entity = x['dataset_label']\n\n    ## main\n    tokens=[]\n    k=0\n    for i,x in enumerate(text):\n\n        if k==0:\n            if x==entity.split()[0]:\n                entity_len = len(entity.split())\n                if entity == ' '.join(text[i:i+entity_len]):\n                    tokens.extend(['o-dataset']*len(entity.split()))\n                    k = entity_len\n                else:\n                    tokens.append('o')\n            else:\n                tokens.append('o')\n\n\n        k = max(0,k-1)\n\n    k=0\n    sentence_hash=[]\n    for i in range(0,len(text),290):\n        sentence_hash.extend([f'sentence#{k}']* len(text[i:i+290]))\n        k+=1\n\n    df['token']=list(map(str,text))\n    df['sentence#'] = sentence_hash\n    df['sentence'] = f'sentence{m}'\n    df['entity'] = tokens\n\n    return df\n\n## read data\ndf_train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ndf_train['text'] = df_train['Id'].apply(get_text)\n\n\n## process\nfinal = pd.DataFrame()\nfor i,row in tqdm(df_train.iterrows()):\n    df = convert_tokens(row,i)\n    final = final.append(df,ignore_index=True)\n</code></pre>\n<p>The data is available <a href=\"https://www.kaggle.com/shahules/ner-coleridge-initiative\" target=\"_blank\">here</a>.</p>",
  "messages": [
    {
      "id": "1261495",
      "postDate": "04/03/2021 05:44:36",
      "content": "<p>Hi, I have prepared a dataset for training NER for this problem taking this sample ner dataset as <a href=\"https://www.kaggle.com/namanj27/ner-dataset\" target=\"_blank\">reference</a>.</p>\n<p>The data format is</p>\n<pre><code>The --&gt; o\ndata --&gt;o\nis --&gt;o\ntaken --&gt;o\nfrom --&gt;o\nIndian --&gt;o-dataset\nmysteries --&gt;o-dataset\ndataset --&gt;o-dataset\n</code></pre>\n<p>The code is used to do it</p>\n<pre><code>def get_text(filename, test=False):\n    if test:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n    else:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n    text = \" \".join(list(df['text']))\n    return text\n\n\ndef convert_tokens(x,m,train=True):\n    df = pd.DataFrame()\n    text = x['text'].replace('\\uf0b7','').split()\n    entity = x['dataset_label']\n\n    ## main\n    tokens=[]\n    k=0\n    for i,x in enumerate(text):\n\n        if k==0:\n            if x==entity.split()[0]:\n                entity_len = len(entity.split())\n                if entity == ' '.join(text[i:i+entity_len]):\n                    tokens.extend(['o-dataset']*len(entity.split()))\n                    k = entity_len\n                else:\n                    tokens.append('o')\n            else:\n                tokens.append('o')\n\n\n        k = max(0,k-1)\n\n    k=0\n    sentence_hash=[]\n    for i in range(0,len(text),290):\n        sentence_hash.extend([f'sentence#{k}']* len(text[i:i+290]))\n        k+=1\n\n    df['token']=list(map(str,text))\n    df['sentence#'] = sentence_hash\n    df['sentence'] = f'sentence{m}'\n    df['entity'] = tokens\n\n    return df\n\n## read data\ndf_train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ndf_train['text'] = df_train['Id'].apply(get_text)\n\n\n## process\nfinal = pd.DataFrame()\nfor i,row in tqdm(df_train.iterrows()):\n    df = convert_tokens(row,i)\n    final = final.append(df,ignore_index=True)\n</code></pre>\n<p>The data is available <a href=\"https://www.kaggle.com/shahules/ner-coleridge-initiative\" target=\"_blank\">here</a>.</p>",
      "rawMarkdown": "Hi, I have prepared a dataset for training NER for this problem taking this sample ner dataset as [reference](https://www.kaggle.com/namanj27/ner-dataset).\n\nThe data format is\n\n```\nThe --> o\ndata -->o\nis -->o\ntaken -->o\nfrom -->o\nIndian -->o-dataset\nmysteries -->o-dataset\ndataset -->o-dataset\n```\n\nThe code is used to do it\n\n```\ndef get_text(filename, test=False):\n    if test:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n    else:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n    text = \" \".join(list(df['text']))\n    return text\n\n\ndef convert_tokens(x,m,train=True):\n    df = pd.DataFrame()\n    text = x['text'].replace('\\uf0b7','').split()\n    entity = x['dataset_label']\n    \n    ## main\n    tokens=[]\n    k=0\n    for i,x in enumerate(text):\n\n        if k==0:\n            if x==entity.split()[0]:\n                entity_len = len(entity.split())\n                if entity == ' '.join(text[i:i+entity_len]):\n                    tokens.extend(['o-dataset']*len(entity.split()))\n                    k = entity_len\n                else:\n                    tokens.append('o')\n            else:\n                tokens.append('o')\n\n\n        k = max(0,k-1)\n            \n    k=0\n    sentence_hash=[]\n    for i in range(0,len(text),290):\n        sentence_hash.extend([f'sentence#{k}']* len(text[i:i+290]))\n        k+=1\n      \n    df['token']=list(map(str,text))\n    df['sentence#'] = sentence_hash\n    df['sentence'] = f'sentence{m}'\n    df['entity'] = tokens\n     \n    return df\n\n## read data\ndf_train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ndf_train['text'] = df_train['Id'].apply(get_text)\n\n\n## process\nfinal = pd.DataFrame()\nfor i,row in tqdm(df_train.iterrows()):\n    df = convert_tokens(row,i)\n    final = final.append(df,ignore_index=True)\n```\n\n\nThe data is available [here](https://www.kaggle.com/shahules/ner-coleridge-initiative).",
      "votes": null
    },
    {
      "id": "1261645",
      "postDate": "04/03/2021 09:52:49",
      "content": "<p>Thanks for sharing👍</p>",
      "rawMarkdown": "Thanks for sharing👍",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1261645,
      "author_name": "majunhua",
      "author_url": "",
      "post_date": "04/03/2021 09:52:49",
      "content": "<p>Thanks for sharing👍</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1261495": "Hi, I have prepared a dataset for training NER for this problem taking this sample ner dataset as [reference](https://www.kaggle.com/namanj27/ner-dataset).\n\nThe data format is\n\n```\nThe --> o\ndata -->o\nis -->o\ntaken -->o\nfrom -->o\nIndian -->o-dataset\nmysteries -->o-dataset\ndataset -->o-dataset\n```\n\nThe code is used to do it\n\n```\ndef get_text(filename, test=False):\n    if test:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/test/{}.json'.format(filename))\n    else:\n        df = pd.read_json('../input/coleridgeinitiative-show-us-the-data/train/{}.json'.format(filename))\n    text = \" \".join(list(df['text']))\n    return text\n\n\ndef convert_tokens(x,m,train=True):\n    df = pd.DataFrame()\n    text = x['text'].replace('\\uf0b7','').split()\n    entity = x['dataset_label']\n    \n    ## main\n    tokens=[]\n    k=0\n    for i,x in enumerate(text):\n\n        if k==0:\n            if x==entity.split()[0]:\n                entity_len = len(entity.split())\n                if entity == ' '.join(text[i:i+entity_len]):\n                    tokens.extend(['o-dataset']*len(entity.split()))\n                    k = entity_len\n                else:\n                    tokens.append('o')\n            else:\n                tokens.append('o')\n\n\n        k = max(0,k-1)\n            \n    k=0\n    sentence_hash=[]\n    for i in range(0,len(text),290):\n        sentence_hash.extend([f'sentence#{k}']* len(text[i:i+290]))\n        k+=1\n      \n    df['token']=list(map(str,text))\n    df['sentence#'] = sentence_hash\n    df['sentence'] = f'sentence{m}'\n    df['entity'] = tokens\n     \n    return df\n\n## read data\ndf_train = pd.read_csv(\"../input/coleridgeinitiative-show-us-the-data/train.csv\")\ndf_train['text'] = df_train['Id'].apply(get_text)\n\n\n## process\nfinal = pd.DataFrame()\nfor i,row in tqdm(df_train.iterrows()):\n    df = convert_tokens(row,i)\n    final = final.append(df,ignore_index=True)\n```\n\n\nThe data is available [here](https://www.kaggle.com/shahules/ner-coleridge-initiative).",
    "1261645": "Thanks for sharing👍"
  },
  "source": "meta"
}