{ "id": 239418, "title": "Mismatch in dataset_label column contents in train.csv", "url": "/competitions/coleridgeinitiative-show-us-the-data/discussion/239418", "author_name": "Girish Nadiger", "post_date": "2021-05-16T09:08:59.041000", "votes": 1, "comment_count": 0, "views": 0, "content": "

Hello Everyone, I was just wondering about few mismatch in dataset_label column in train.csv, and found for multiple files, the content in dataset_label column is not present in the file. Is this expected or am I missing something?

\n

Also if this is expected, then it will perform poorly on the hidden dataset.

\n

Ex Code to check for file_id cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b

\n

input_path = '../input/coleridgeinitiative-show-us-the-data'
\ndf = pd.read_csv(os.path.join(input_path, 'train.csv'))
\nfile_id = 'cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b'
\ndata_title = df[df.Id == file_id].dataset_title.tolist()
\ndata_label = df[df.Id == file_id].dataset_label.tolist()
\nprint(data_title)
\nprint(data_label)

\n

with open(os.path.join(input_path, 'train',file_id + '.json'),'r') as f:
\n local_data = json.load(f)

\n

present = False
\nfor item in local_data:
\nif data_label[-1] in item['text']:
\nprint('present')
\npresent = True
\nif present == False:
\nprint('not present')

\n

In the above example, the dataset_label is ['ADNI', \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"], but in the file \"Alzheimer's Disease Neuroimaging Initiative\" is present.

\n

The count where I was unable to find contents of dataset_label in the file is 1040, which is around 7.26% of the data

", "messages": [ { "id": 1309759, "postDate": "2021-05-16T09:08:59.040Z", "content": "

Hello Everyone, I was just wondering about few mismatch in dataset_label column in train.csv, and found for multiple files, the content in dataset_label column is not present in the file. Is this expected or am I missing something?

\n

Also if this is expected, then it will perform poorly on the hidden dataset.

\n

Ex Code to check for file_id cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b

\n

input_path = '../input/coleridgeinitiative-show-us-the-data'
\ndf = pd.read_csv(os.path.join(input_path, 'train.csv'))
\nfile_id = 'cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b'
\ndata_title = df[df.Id == file_id].dataset_title.tolist()
\ndata_label = df[df.Id == file_id].dataset_label.tolist()
\nprint(data_title)
\nprint(data_label)

\n

with open(os.path.join(input_path, 'train',file_id + '.json'),'r') as f:
\n local_data = json.load(f)

\n

present = False
\nfor item in local_data:
\nif data_label[-1] in item['text']:
\nprint('present')
\npresent = True
\nif present == False:
\nprint('not present')

\n

In the above example, the dataset_label is ['ADNI', \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"], but in the file \"Alzheimer's Disease Neuroimaging Initiative\" is present.

\n

The count where I was unable to find contents of dataset_label in the file is 1040, which is around 7.26% of the data

", "rawMarkdown": "Hello Everyone, I was just wondering about few mismatch in dataset_label column in train.csv, and found for multiple files, the content in dataset_label column is not present in the file. Is this expected or am I missing something?\n\nAlso if this is expected, then it will perform poorly on the hidden dataset.\n\nEx Code to check for file_id cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b\n\n`input_path = '../input/coleridgeinitiative-show-us-the-data'`\n`df = pd.read_csv(os.path.join(input_path, 'train.csv'))`\n`file_id = 'cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b'`\n`data_title = df[df.Id == file_id].dataset_title.tolist()`\n`data_label = df[df.Id == file_id].dataset_label.tolist()`\n`print(data_title)`\n`print(data_label)`\n\n`with open(os.path.join(input_path, 'train',file_id + '.json'),'r') as f:`\n ` local_data = json.load(f)`\n\n`present = False`\n`for item in local_data:`\n` if data_label[-1] in item['text']:`\n` print('present')`\n` present = True`\n`if present == False:`\n` print('not present')`\n\nIn the above example, the dataset_label is ['ADNI', \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"], but in the file \"Alzheimer's Disease Neuroimaging Initiative\" is present.\n\nThe count where I was unable to find contents of dataset_label in the file is 1040, which is around 7.26% of the data", "votes": 1 } ], "comments": [], "raw_markdown_by_id": { "1309759": "Hello Everyone, I was just wondering about few mismatch in dataset_label column in train.csv, and found for multiple files, the content in dataset_label column is not present in the file. Is this expected or am I missing something?\n\nAlso if this is expected, then it will perform poorly on the hidden dataset.\n\nEx Code to check for file_id cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b\n\n`input_path = '../input/coleridgeinitiative-show-us-the-data'`\n`df = pd.read_csv(os.path.join(input_path, 'train.csv'))`\n`file_id = 'cfc03cbc-c65c-490c-a1fa-0e5bed83ef1b'`\n`data_title = df[df.Id == file_id].dataset_title.tolist()`\n`data_label = df[df.Id == file_id].dataset_label.tolist()`\n`print(data_title)`\n`print(data_label)`\n\n`with open(os.path.join(input_path, 'train',file_id + '.json'),'r') as f:`\n ` local_data = json.load(f)`\n\n`present = False`\n`for item in local_data:`\n` if data_label[-1] in item['text']:`\n` print('present')`\n` present = True`\n`if present == False:`\n` print('not present')`\n\nIn the above example, the dataset_label is ['ADNI', \"Alzheimer's Disease Neuroimaging Initiative (ADNI)\"], but in the file \"Alzheimer's Disease Neuroimaging Initiative\" is present.\n\nThe count where I was unable to find contents of dataset_label in the file is 1040, which is around 7.26% of the data" } }