[EDIT/UPDATE]
\nThere are a few important updates.
\npd.Dataframe as a .csv, the following command should be used to avoid improper interpretation of newline character(s). train_df.to_csv(\n \"train.csv\", index=False, \n encoding='utf-8', \n quoting=csv.QUOTE_NONNUMERIC # <== THIS IS REQUIRED\n)\n.csv as a pd.Dataframe, the following command must be used to avoid misinterpretation of NaN like strings (null, nan, …) as pd.NaN values.train_df = pd.read_csv(\n \"/kaggle/input/ai4code-train-dataframe/train.csv\", \n keep_default_na=False # <== THIS IS REQUIRED\n)\n[ORIGINAL POST]
\nAs the title says. I generated a csv/pd.Dataframe from the raw json files. This will save you 20-30 minutes of processing if you do it yourself.
\nHere is the dataset link containing the train.csv file.
Here is the code used to generate the pd.DataFrame which was then saved as a csv.
\n[UPDATED CODE TO REMOVE JOBLIB]
\ndef load_json_to_df(all_fpaths):\n def __fpath_to_df(fpath):\n tmp_df = pd.read_json(fpath, dtype={'cell_type': 'category', 'source': 'str'}).reset_index().rename({\"index\":\"cell_id\"}, axis=1)\n tmp_df[\"id\"] = fpath.rsplit(\".\", 1)[0].rsplit(\"/\", 1)[-1]\n return tmp_df\n all_example_dfs = [__fpath_to_df(fpath) for fpath in tqdm(all_fpaths, total=len(all_fpaths))]\n return pd.concat(all_example_dfs).reset_index(drop=True)\nYou can also see this code within my in-progress EDA Notebook
\n[EDIT/UPDATE]
\nThere are a few important updates.
\npd.Dataframe as a .csv, the following command should be used to avoid improper interpretation of newline character(s). train_df.to_csv(\n \"train.csv\", index=False, \n encoding='utf-8', \n quoting=csv.QUOTE_NONNUMERIC # <== THIS IS REQUIRED\n)\n.csv as a pd.Dataframe, the following command must be used to avoid misinterpretation of NaN like strings (null, nan, …) as pd.NaN values.train_df = pd.read_csv(\n \"/kaggle/input/ai4code-train-dataframe/train.csv\", \n keep_default_na=False # <== THIS IS REQUIRED\n)\n[ORIGINAL POST]
\nAs the title says. I generated a csv/pd.Dataframe from the raw json files. This will save you 20-30 minutes of processing if you do it yourself.
\nHere is the dataset link containing the train.csv file.
Here is the code used to generate the pd.DataFrame which was then saved as a csv.
\n[UPDATED CODE TO REMOVE JOBLIB]
\ndef load_json_to_df(all_fpaths):\n def __fpath_to_df(fpath):\n tmp_df = pd.read_json(fpath, dtype={'cell_type': 'category', 'source': 'str'}).reset_index().rename({\"index\":\"cell_id\"}, axis=1)\n tmp_df[\"id\"] = fpath.rsplit(\".\", 1)[0].rsplit(\"/\", 1)[-1]\n return tmp_df\n all_example_dfs = [__fpath_to_df(fpath) for fpath in tqdm(all_fpaths, total=len(all_fpaths))]\n return pd.concat(all_example_dfs).reset_index(drop=True)\nYou can also see this code within my in-progress EDA Notebook
\n@dschettler8845
\nThere are 12 NA values in the train dataframe.
\nIndexes of missing values - 154891, 154891, 1305172, 1305172, 1373663, 1373663, 2076839, 2915102, 3416953, 4260449, 6154217, 6154217.
Let me check that and get back to you. Thanks for pointing it out.
", "rawMarkdown": "Let me check that and get back to you. Thanks for pointing it out.", "votes": null }, { "id": "1785295", "postDate": "05/12/2022 02:12:20", "content": "@dschettler8845 thanks for that!
", "rawMarkdown": "dschettler8845 thanks for that!", "votes": null }, { "id": "1785297", "postDate": "05/12/2022 02:13:50", "content": "Ok, so it looks like I messed up during multiprocessing and it added an empty string id that has a few values. I'm not sure if this is an extra addition or if valuable information was overwritten. I will check this and get back.
The iloc values exhibiting this are the following
The other numbers you mentioned look to be related to the source information being missing. I will also investigate this and get back to you.
", "rawMarkdown": "Ok, so it looks like I messed up during multiprocessing and it added an empty string **`id`** that has a few values. I'm not sure if this is an extra addition or if valuable information was overwritten. I will check this and get back. \n\nThe **`iloc`** values exhibiting this are the following\n* 154891, 1305172, 1373663, 6154217\n\nThe other numbers you mentioned look to be related to the source information being missing. I will also investigate this and get back to you.", "votes": null }, { "id": "1785942", "postDate": "05/12/2022 14:02:19", "content": "Great dataset!
", "rawMarkdown": "Great dataset!", "votes": null }, { "id": "1786073", "postDate": "05/12/2022 15:31:58", "content": "Ok, I have rerun without the parallelization and all the nan values are gone. I am updating the dataset as we speak to be correct. Thanks again for catching this!
\n[EDIT]
\nIt appears that while the NaN values do not exist after parsing… after saving and then loading the csv directly they appear. I am troubleshooting this now and will update it shortly with the corrected version (ETA May 12 ~2PM EST).
\n[EDIT 2]
\nIt appears that there were two problems:
\nThe newline character in some code cells is interpreted during saving and results in the creation of a new row that doesn't exist (this usually can be seen as a row with > and all nan values). I have recreated the dataset and used a different newline interpreter to avoid this.
The interpretation during loading of certain strings as NaN instead of the string representation. i.e. the code block contains only the text \"null\" but pandas interpret this as pd.NaN. This can be avoided by specifying keep_default_na=False in the pd.read_csv() command (i.e. no default NaN parsing). I will update the main body of the post with information about specifying this command when loading.
I hope this clears everything up!
", "rawMarkdown": "Ok, I have rerun without the parallelization and all the nan values are gone. I am updating the dataset as we speak to be correct. Thanks again for catching this!\n\n[EDIT] \n\nIt appears that while the NaN values do not exist after parsing... after saving and then loading the csv directly they appear. I am troubleshooting this now and will update it shortly with the corrected version (ETA May 12 ~2PM EST).\n\n[EDIT 2] \n\nIt appears that there were two problems:\n\n1. The newline character in some code cells is interpreted during saving and results in the creation of a new row that doesn't exist (this usually can be seen as a row with > and all nan values). **I have recreated the dataset and used a different newline interpreter to avoid this.**\n\n2. The interpretation during loading of certain strings as NaN instead of the string representation. i.e. the code block contains only the text \"null\" but pandas interpret this as pd.NaN. This can be avoided by specifying **`keep_default_na=False`** in the **`pd.read_csv()`** command (i.e. no default NaN parsing). I will update the main body of the post with information about specifying this command when loading.\n\n---\n\nI hope this clears everything up!", "votes": null }, { "id": "1786136", "postDate": "05/12/2022 16:19:36", "content": "@dschettler8845
\nwhile extracting id there is dot(.) comming with id Ex (8a2564b730a575.)
\nQuickFix :
tmp_df[\"id\"] = fpath[:-4].rsplit(\"/\", 1)[-1].strip('.')\nThanks for the nice code :)
", "rawMarkdown": "dschettler8845 \nwhile extracting `id` there is `dot(.)` comming with `id` Ex (`8a2564b730a575.`)\nQuickFix : \n````python\ntmp_df[\"id\"] = fpath[:-4].rsplit(\"/\", 1)[-1].strip('.')\n````\nThanks for the nice code :)", "votes": null }, { "id": "1786198", "postDate": "05/12/2022 17:06:11", "content": "That should be fixed now. My apologies!
", "rawMarkdown": "That should be fixed now. My apologies!", "votes": null }, { "id": "1786214", "postDate": "05/12/2022 17:30:20", "content": "Thanks for the update @dschettler8845
\nI am dropping null values in my notebook. This will most probably have no effects after any updates in the dataset versions.
Hey! Nice work. Looks like we had the same idea. I ran the process on my twitch stream tonight and created a parquet version too. If people find it helpful it's here: https://www.kaggle.com/datasets/robikscube/ai4code-parquet-tabular
", "rawMarkdown": "Hey! Nice work. Looks like we had the same idea. I ran the process on my twitch stream tonight and created a parquet version too. If people find it helpful it's here: https://www.kaggle.com/datasets/robikscube/ai4code-parquet-tabular", "votes": null }, { "id": "1789514", "postDate": "05/14/2022 01:09:37", "content": "thank you for your work, I was gonna do it myself but I am glad I don't have to lol.
", "rawMarkdown": "thank you for your work, I was gonna do it myself but I am glad I don't have to lol.", "votes": null }, { "id": "1789950", "postDate": "05/14/2022 11:26:34", "content": "Nice work:)
", "rawMarkdown": "Nice work:)", "votes": null }, { "id": "1790323", "postDate": "05/14/2022 19:14:00", "content": "Nice! I love your streams!
", "rawMarkdown": "Nice! I love your streams!", "votes": null }, { "id": "1790324", "postDate": "05/14/2022 19:14:19", "content": "You're welcome! glad I could help :)
", "rawMarkdown": "You're welcome! glad I could help :)", "votes": null }, { "id": "1790325", "postDate": "05/14/2022 19:14:26", "content": "Thank you ! Glad to help!
", "rawMarkdown": "Thank you ! Glad to help!", "votes": null }, { "id": "1790326", "postDate": "05/14/2022 19:14:38", "content": "Thank you, glad I could help!
", "rawMarkdown": "Thank you, glad I could help!", "votes": null }, { "id": "1790327", "postDate": "05/14/2022 19:14:46", "content": "Hahaha yes! That's the idea. :)
", "rawMarkdown": "Hahaha yes! That's the idea. :)", "votes": null } ], "comments": [ { "id": 1785288, "author_name": "odins0n", "author_url": "", "post_date": "05/12/2022 01:56:04", "content": "@dschettler8845
\nThere are 12 NA values in the train dataframe.
\nIndexes of missing values - 154891, 154891, 1305172, 1305172, 1373663, 1373663, 2076839, 2915102, 3416953, 4260449, 6154217, 6154217.
Let me check that and get back to you. Thanks for pointing it out.
", "votes": null, "replies": [] }, { "id": 1785297, "author_name": "dschettler8845", "author_url": "", "post_date": "05/12/2022 02:13:50", "content": "Ok, so it looks like I messed up during multiprocessing and it added an empty string id that has a few values. I'm not sure if this is an extra addition or if valuable information was overwritten. I will check this and get back.
The iloc values exhibiting this are the following
The other numbers you mentioned look to be related to the source information being missing. I will also investigate this and get back to you.
", "votes": null, "replies": [] }, { "id": 1786073, "author_name": "dschettler8845", "author_url": "", "post_date": "05/12/2022 15:31:58", "content": "Ok, I have rerun without the parallelization and all the nan values are gone. I am updating the dataset as we speak to be correct. Thanks again for catching this!
\n[EDIT]
\nIt appears that while the NaN values do not exist after parsing… after saving and then loading the csv directly they appear. I am troubleshooting this now and will update it shortly with the corrected version (ETA May 12 ~2PM EST).
\n[EDIT 2]
\nIt appears that there were two problems:
\nThe newline character in some code cells is interpreted during saving and results in the creation of a new row that doesn't exist (this usually can be seen as a row with > and all nan values). I have recreated the dataset and used a different newline interpreter to avoid this.
The interpretation during loading of certain strings as NaN instead of the string representation. i.e. the code block contains only the text \"null\" but pandas interpret this as pd.NaN. This can be avoided by specifying keep_default_na=False in the pd.read_csv() command (i.e. no default NaN parsing). I will update the main body of the post with information about specifying this command when loading.
I hope this clears everything up!
", "votes": null, "replies": [] }, { "id": 1786214, "author_name": "odins0n", "author_url": "", "post_date": "05/12/2022 17:30:20", "content": "Thanks for the update @dschettler8845
\nI am dropping null values in my notebook. This will most probably have no effects after any updates in the dataset versions.
@dschettler8845 thanks for that!
", "votes": null, "replies": [ { "id": 1790324, "author_name": "dschettler8845", "author_url": "", "post_date": "05/14/2022 19:14:19", "content": "You're welcome! glad I could help :)
", "votes": null, "replies": [] } ] }, { "id": 1785942, "author_name": "", "author_url": "", "post_date": "05/12/2022 14:02:19", "content": "Great dataset!
", "votes": null, "replies": [ { "id": 1790325, "author_name": "dschettler8845", "author_url": "", "post_date": "05/14/2022 19:14:26", "content": "Thank you ! Glad to help!
", "votes": null, "replies": [] } ] }, { "id": 1786136, "author_name": "coolcoder22", "author_url": "", "post_date": "05/12/2022 16:19:36", "content": "@dschettler8845
\nwhile extracting id there is dot(.) comming with id Ex (8a2564b730a575.)
\nQuickFix :
tmp_df[\"id\"] = fpath[:-4].rsplit(\"/\", 1)[-1].strip('.')\nThanks for the nice code :)
", "votes": null, "replies": [ { "id": 1786198, "author_name": "dschettler8845", "author_url": "", "post_date": "05/12/2022 17:06:11", "content": "That should be fixed now. My apologies!
", "votes": null, "replies": [] } ] }, { "id": 1786562, "author_name": "robikscube", "author_url": "", "post_date": "05/13/2022 02:42:37", "content": "Hey! Nice work. Looks like we had the same idea. I ran the process on my twitch stream tonight and created a parquet version too. If people find it helpful it's here: https://www.kaggle.com/datasets/robikscube/ai4code-parquet-tabular
", "votes": null, "replies": [ { "id": 1790323, "author_name": "dschettler8845", "author_url": "", "post_date": "05/14/2022 19:14:00", "content": "Nice! I love your streams!
", "votes": null, "replies": [] } ] }, { "id": 1789514, "author_name": "bluetriad", "author_url": "", "post_date": "05/14/2022 01:09:37", "content": "thank you for your work, I was gonna do it myself but I am glad I don't have to lol.
", "votes": null, "replies": [ { "id": 1790327, "author_name": "dschettler8845", "author_url": "", "post_date": "05/14/2022 19:14:46", "content": "Hahaha yes! That's the idea. :)
", "votes": null, "replies": [] } ] }, { "id": 1789950, "author_name": "happylive", "author_url": "", "post_date": "05/14/2022 11:26:34", "content": "Nice work:)
", "votes": null, "replies": [ { "id": 1790326, "author_name": "dschettler8845", "author_url": "", "post_date": "05/14/2022 19:14:38", "content": "Thank you, glad I could help!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1785274": "**[EDIT/UPDATE]**\n\n---\n\nThere are a few important updates. \n\n1. When **SAVING** the **`pd.Dataframe`** as a **`.csv`**, the following command should be used to avoid improper interpretation of newline character(s). \n\n```python\ntrain_df.to_csv(\n \"train.csv\", index=False, \n encoding='utf-8', \n quoting=csv.QUOTE_NONNUMERIC # <== THIS IS REQUIRED\n)\n```\n\n\n2. When **LOADING** the **`.csv`** as a **`pd.Dataframe`**, the following command must be used to avoid misinterpretation of **`NaN`** like strings (null, nan, ...) as **`pd.NaN`** values.\n\n```python\ntrain_df = pd.read_csv(\n \"/kaggle/input/ai4code-train-dataframe/train.csv\", \n keep_default_na=False # <== THIS IS REQUIRED\n)\n```\n\n---\n\n