\n Dealing with notebooks containing duplicate cells is tricky. Misordering a notebook affects the order of all other cells. Moreover, depending on how the selected model works, the model can get confused on predictions since 2 exact inputs have different positions. Do these cells come right after each other in the notebooks? what about removing one and putting it either always before/after the duplicate?\n
","metadata":{}},{"cell_type":"code","source":"tmp_df=nb_with_duplicate_cells_df.join(language_mapping)\ntmp_df=tmp_df[tmp_df['has_duplicates']==1]\n\nassert len(tmp_df)==np.sum(tmp_df['has_duplicates']==1), 'There is something wrong'\n\nenglish_mean=np.mean(tmp_df['language']=='en')\n\ndata=[english_mean,1-english_mean]\nkeys=['english','other']\nstartangle=90\nshadow=True\nexplode=[0.1,0]\nradius=1.9\n\n# define Seaborn color palette to use\npalette_color = sns.color_palette('bright')\n \n# plotting data on chart\nplt.pie(data, labels=keys, colors=palette_color,radius=radius, autopct='%.1f%%',shadow=shadow,explode=explode,startangle=startangle)\n\nplt.title('Proportion of english notebooks knowing the notebook have duplicates',pad=80)\n\n# displaying chart\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:39.072522Z","iopub.execute_input":"2022-06-22T15:35:39.073007Z","iopub.status.idle":"2022-06-22T15:35:39.282595Z","shell.execute_reply.started":"2022-06-22T15:35:39.072962Z","shell.execute_reply":"2022-06-22T15:35:39.281286Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n The proprotion of english markdown cells is equal in both, the whole dataset and the dataset having duplicate code cells.\n
\n\n\n Next I need to check the if duplicate cells are alongside each other or distributed through out the notebook.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(25,5))\nsns.countplot(x='language',data=tmp_df)","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:39.284394Z","iopub.execute_input":"2022-06-22T15:35:39.285023Z","iopub.status.idle":"2022-06-22T15:35:40.124509Z","shell.execute_reply.started":"2022-06-22T15:35:39.284971Z","shell.execute_reply":"2022-06-22T15:35:40.123437Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Get the markdown cells of notebooks with duplicate cells\nmarkdown_df=train_df[(train_df['cell_type']=='markdown')].copy()\nmarkdown_df=markdown_df[markdown_df['has_duplicates']==True]\nprint('length of markdown cells',len(markdown_df))\n\nid_to_dup_order={}\n\ni=0\nfor index, group_df in markdown_df.reset_index().groupby(['id']):\n \n# print(index)\n# print(group_df)\n# print('\\n\\n',group_df['source'].nunique())\n# print(len(group_df))\n duplicated_cells=group_df[group_df['source'].duplicated(keep=False)]\n if i<10: \n print(duplicated_cells[['cell_id','source']])\n print('-----------------------')\n \n dup_order=[]\n for cell_id in duplicated_cells['cell_id']:\n dup_order.append(order_dict[index].index(cell_id))\n \n #check if notebook has more than 1 cell that has a duplicate.\n id_to_dup_order[index]=(duplicated_cells['source'].nunique()==1,dup_order)\n i+=1\n \ndel markdown_df","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:40.125719Z","iopub.execute_input":"2022-06-22T15:35:40.126463Z","iopub.status.idle":"2022-06-22T15:35:47.994365Z","shell.execute_reply.started":"2022-06-22T15:35:40.126423Z","shell.execute_reply":"2022-06-22T15:35:47.993518Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"iterations=10\n\ni=0\nfor key, value in id_to_dup_order.items():\n print(f'{key}: {value}')\n \n if i > iterations: break\n i+=1","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:47.995815Z","iopub.execute_input":"2022-06-22T15:35:47.996277Z","iopub.status.idle":"2022-06-22T15:35:48.003213Z","shell.execute_reply.started":"2022-06-22T15:35:47.996235Z","shell.execute_reply":"2022-06-22T15:35:48.001742Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"
\n It looks like the duplicates are lazy comments or some generic line that they add to many to experiments. For example, there is a markdown text that just says export, and there are 3 of them! Maybe when exporting many files, the owner of the document just writes export. Another example is the one that starts with \"Green markers denote crimes commited within\". It is similar to adding footnotes to graphs in reports. You add the same text to all graphs to make it clear.\n
\n\n\n How to deal with this values? If we switch any of these cells, the meaning does not change and it is not wrong. Thus, could potentially become malicious examples that may confuse the model. An idea is to consider the 3 of them the same cell (same 'cell_id').\n
","metadata":{}},{"cell_type":"code","source":"def get_token_statistics(df,column='cell_type',value='markdown',spliter=' ',lower_percentile=10,upper_percentile=95,title=''):\n df_copy=df.copy()\n \n df_copy=df_copy[df_copy[column]==value]\n \n #Get the number of tokens\n df_copy['cell_len']=df_copy['source'].apply(lambda source: len(str(source).split(spliter)))\n \n print(f'The max number of tokens: {np.max(df_copy[\"cell_len\"])}')\n print(f'The min number of tokens: {np.min(df_copy[\"cell_len\"])}')\n print(f'The mean number of tokens: {np.mean(df_copy[\"cell_len\"])}')\n print(f'The median number of tokens: {np.median(df_copy[\"cell_len\"])}')\n print(f'The std number of tokens: {np.std(df_copy[\"cell_len\"])}')\n print(f'The {upper_percentile} percentile of the number of tokens: {np.percentile(df_copy[\"cell_len\"],upper_percentile)}')\n print(f'The {lower_percentile} percentile of the number of tokens: {np.percentile(df_copy[\"cell_len\"],lower_percentile)}')\n \n sns.displot(data=df_copy.iloc[:50000].reset_index(),x='cell_len',kind='kde',hue='cell_type')\n \n ax=plt.gca()\n ax.set_title(title)","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:48.004796Z","iopub.execute_input":"2022-06-22T15:35:48.00528Z","iopub.status.idle":"2022-06-22T15:35:48.01857Z","shell.execute_reply.started":"2022-06-22T15:35:48.005237Z","shell.execute_reply":"2022-06-22T15:35:48.017514Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"title='Number words in markdown cells'\nget_token_statistics(train_df,title=title)","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:48.020119Z","iopub.execute_input":"2022-06-22T15:35:48.020832Z","iopub.status.idle":"2022-06-22T15:35:57.647148Z","shell.execute_reply.started":"2022-06-22T15:35:48.020797Z","shell.execute_reply":"2022-06-22T15:35:57.645954Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n The 95th percentile is 115. Thus, maybe a 128 max length for markdown is enough.\n
","metadata":{}},{"cell_type":"code","source":"title='Number of lines in code cells'\nget_token_statistics(train_df,column='cell_type',value='code',spliter='\\n',lower_percentile=10,upper_percentile=95,title=title)","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:35:57.648371Z","iopub.execute_input":"2022-06-22T15:35:57.648678Z","iopub.status.idle":"2022-06-22T15:36:08.727465Z","shell.execute_reply.started":"2022-06-22T15:35:57.648651Z","shell.execute_reply":"2022-06-22T15:36:08.726271Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n The 95th percentile is 26. Thus, most cells have less than 26 lines of code.\n
","metadata":{}},{"cell_type":"code","source":"max_markdown_cells=np.max(train_df[train_df['cell_type']=='markdown'].groupby('id').size())\nprint(f'max markdown cells: {max_markdown_cells}')\nmax_code_cells=np.max(train_df[train_df['cell_type']=='code'].groupby('id').size())\nprint(f'max code cells: {max_code_cells}')","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:36:08.728955Z","iopub.execute_input":"2022-06-22T15:36:08.729312Z","iopub.status.idle":"2022-06-22T15:36:12.223118Z","shell.execute_reply.started":"2022-06-22T15:36:08.729281Z","shell.execute_reply":"2022-06-22T15:36:12.221938Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"markdown_cells_90=np.percentile(train_df[train_df['cell_type']=='markdown'].groupby('id').size(),90)\nprint(f'90th percentile markdown cells: {markdown_cells_90}')\ncode_cells_90=np.percentile(train_df[train_df['cell_type']=='code'].groupby('id').size(),90)\nprint(f'90th percentile code cells: {code_cells_90}')","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:36:12.224614Z","iopub.execute_input":"2022-06-22T15:36:12.224977Z","iopub.status.idle":"2022-06-22T15:36:15.713159Z","shell.execute_reply.started":"2022-06-22T15:36:12.224945Z","shell.execute_reply":"2022-06-22T15:36:15.711957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"\n It seems there are a lot of cells in some notebooks. I do not expect deep learning to work well on these examples. If we use a sequential model, vanishing gradient will make it difficult for the model to learn. Sequence model do not do well for large sequences and the large sequences are just 10% of the whole dataset. Perhaps, using different techniques depending on the number of cells is best? Or maybe finding a method to concatenate cells? \n
\nI will check some of these notebooks:","metadata":{}},{"cell_type":"code","source":"examples_df=pd.DataFrame()\n\nfor index,group in train_df.groupby('id'):\n if len(group)>800:\n examples_df=pd.concat((examples_df,group))\n \nexamples_df.to_csv('high_cardinal_examples.csv')","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:36:15.714258Z","iopub.execute_input":"2022-06-22T15:36:15.714866Z","iopub.status.idle":"2022-06-22T15:44:01.467441Z","shell.execute_reply.started":"2022-06-22T15:36:15.71483Z","shell.execute_reply":"2022-06-22T15:44:01.465919Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"examples_df","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:44:01.468624Z","iopub.status.idle":"2022-06-22T15:44:01.469087Z","shell.execute_reply.started":"2022-06-22T15:44:01.46886Z","shell.execute_reply":"2022-06-22T15:44:01.46888Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I preffer to download the CSV file and open it in excel. In excel I can add a table and filter by \"id\". I think it is easier than in the notebook. Although, I have added a code snippet to display the cells of a notebook below: ","metadata":{}},{"cell_type":"code","source":"demostration_df=pd.DataFrame()\nmax_number=examples_df.reset_index()['id'].nunique()\nprint(f'There are {max_number} number of cells in the notebook')\n\nexample_number=4\n\ni=1\nfor index, group in examples_df.groupby('id'):\n if i==example_number:\n demostration_df=group.copy()\n break\n i+=1\n\ncell_number=(i for i in range(0,len(demostration_df)))\ndemostration_df['source'].apply(lambda source: print(f'[cell {next(cell_number)}]',source))","metadata":{"execution":{"iopub.status.busy":"2022-06-22T15:44:01.470716Z","iopub.status.idle":"2022-06-22T15:44:01.471143Z","shell.execute_reply.started":"2022-06-22T15:44:01.470945Z","shell.execute_reply":"2022-06-22T15:44:01.470965Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]}]}