{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"**Created by Sanskar Hasija**\n\n**AI4Code Detailed EDA📊**\n\n**12 May 2021**\n","metadata":{}},{"cell_type":"markdown","source":"  # <center> <span style=\"color:#00BFC4;\"> AI4CODE DETAILED EDA📊</span> </center>\n## <center><span style=\"color:#00BFC4;\">If you find this notebook useful, support with an upvote👍</span></center>","metadata":{}},{"cell_type":"markdown","source":"# <center><span style=\"color:#e76f51;\">Table of Contents</span>\n<a id=\"toc\"></a>\n- [1. Introduction](#1)\n- [2. Imports](#2)\n- [3. EDA](#3)\n    - [3.1 Train Data](#3.1)\n        - [3.1.1 Train Data Distribution ](#3.1.1)\n    - [3.2 Code Cell Analysis](#3.2)\n        - [3.2.1 Code Cells Length Distribution ](#3.2.1)\n        - [3.2.2 Code Cells WordCloud ](#3.2.2)\n    - [3.3 Markdown Cell Analysis](#3.2)\n        - [3.3.1 Markdown Cells Length Distribution ](#3.3.1)\n        - [3.3.2 Markdown Cells WordCloud ](#3.3.2)\n    - [3.4 Notebooks Analysis](#3.4)\n        - [3.4.1 Code Cell Count Analysis ](#3.4.1)\n        - [3.4.2 Markdown Cell Count Analysis ](#3.4.2)\n        - [3.4.3 Minimum Cell Count Analysis ](#3.4.3)\n    ","metadata":{}},{"cell_type":"markdown","source":"<a id=\"1\"></a>\n# **<center><span style=\"color:#00BFC4;\">Introduction </span></center>**","metadata":{}},{"cell_type":"markdown","source":"![](https://raw.githubusercontent.com/sanskar-hasija/kaggle/main/images/ai4code_image.png)","metadata":{}},{"cell_type":"markdown","source":"<b>The goal of this competition is to understand the relationship between code and comments in Python notebooks. You are challenged to reconstruct the order of markdown cells in a given notebook based on the order of the code cells, demonstrating comprehension of which natural language references which code.</b><br>\n\n<b>Predictions are evaluated by the Kendall tau correlation between predicted cell orders and ground truth cell orders accumulated across the entire collection of test set notebooks.</b><br>\n\n<b>Check more about Kendall tau correlation - https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient</b>","metadata":{}},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"2\"></a>\n# **<center><span style=\"color:#00BFC4;\">IMPORTS </span></center>**","metadata":{}},{"cell_type":"code","source":"import os\nimport json\nimport wordcloud\nimport numpy as np\nimport pandas as pd\nimport plotly.express as px\nimport matplotlib.pyplot as plt\nimport plotly.graph_objects as go\nfrom tqdm.notebook import tqdm, trange","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-05-13T08:15:40.681244Z","iopub.execute_input":"2022-05-13T08:15:40.681633Z","iopub.status.idle":"2022-05-13T08:15:42.369167Z","shell.execute_reply.started":"2022-05-13T08:15:40.681542Z","shell.execute_reply":"2022-05-13T08:15:42.368147Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## <span style=\"color:#e76f51;\"> Loading Data : </span>\n","metadata":{}},{"cell_type":"markdown","source":"Pandas Dataframe used is created by [Darien Schettler](https://www.kaggle.com/dschettler8845). Link to dataset - https://www.kaggle.com/datasets/dschettler8845/ai4code-train-dataframe","metadata":{}},{"cell_type":"code","source":"## train dataframes\ndf = pd.read_csv(\"../input/ai4code-train-dataframe/train.csv\", index_col= [0,1])\ndf.dropna(inplace = True)\n\ndf_ancestors = pd.read_csv('../input/AI4Code/train_ancestors.csv', index_col='id')\ndf_orders = pd.read_csv(\"../input/AI4Code/train_orders.csv\",index_col='id',squeeze=True,).str.split()","metadata":{"execution":{"iopub.status.busy":"2022-05-13T08:15:42.370911Z","iopub.execute_input":"2022-05-13T08:15:42.371153Z","iopub.status.idle":"2022-05-13T08:17:21.784666Z","shell.execute_reply.started":"2022-05-13T08:15:42.371123Z","shell.execute_reply":"2022-05-13T08:17:21.783751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"3\"></a>\n# **<center><span style=\"color:#00BFC4;\">EDA</span></center>**","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp;<b><u>Observations :</u></b><br>\n\n* <i> A total of <b><u>139256</u></b> notebooks are provided <b><u>train</u></b> set.</i><br>\n* <i> A total of <b><u>4</u></b> notebooks are provided <b><u>test</u></b> set. This will be replaced with a <b><u> hidden test</u></b> set for scoring</i><br>\n* <i> There are total of <b><u>146300</u></b> cells in <b><u>train </u></b>the train dataframe constructed which include two types of cell_type.</i><br>\n* <i> Two types of cell_type - <b><u>code</u></b> and <b><u>markdown</u></b>.</i>\n* <i> Almost <b><u>2/3rd</u></b> of the training data consist of Code Cells and remaining <b><u>1/3rd</u></b> consist of Markdown Cells.</i>\n</div>","metadata":{}},{"cell_type":"code","source":"print(f\"\\033[94mNumber of notebooks present in train set  = \",len(os.listdir(\"../input/AI4Code/train\")))\nprint(f\"\\033[94mNumber of notebooks present in test set  = \",len(os.listdir(\"../input/AI4Code/test\")))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:17:21.786043Z","iopub.execute_input":"2022-05-13T08:17:21.786305Z","iopub.status.idle":"2022-05-13T08:17:23.973114Z","shell.execute_reply.started":"2022-05-13T08:17:21.786275Z","shell.execute_reply":"2022-05-13T08:17:23.972432Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.1\"></a>\n### <span style=\"color:#e76f51;\"> Quick View of train data  : </span>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp;<b>`df` is a Multi-level indexing Padnas Dataframe with 2 index - id and cell_id. </b><br>\nLearn more about Multi-level indexing Padnas Dataframe - https://pandas.pydata.org/docs/user_guide/advanced.html\n</div>","metadata":{}},{"cell_type":"code","source":"df.head()","metadata":{"execution":{"iopub.status.busy":"2022-05-13T08:17:23.975043Z","iopub.execute_input":"2022-05-13T08:17:23.975597Z","iopub.status.idle":"2022-05-13T08:17:24.001099Z","shell.execute_reply.started":"2022-05-13T08:17:23.975559Z","shell.execute_reply":"2022-05-13T08:17:24.000274Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.1.1\"></a>\n### <span style=\"color:#e76f51;\"> Train data distribution  : </span>","metadata":{}},{"cell_type":"code","source":"code_df = df[df[\"cell_type\"] == \"code\"]\nmkd_df = df[df[\"cell_type\"] == \"markdown\"]\n\n\nprint(f'\\033[94mNumber of Code Cells: {len(code_df)}')\nprint(f'\\033[94mNumber of Markdown Cells: {len(mkd_df)}')\n\nlabels=['Code Cells', 'Markdown Cells']\nvalues= [len(code_df), len(mkd_df)]\ncolors = ['#DE3163', '#58D68D']\n\nfig = go.Figure(data=[go.Pie(\n    labels=labels, \n    values=values, \n    pull=[0.1, 0 ],\n    marker=dict(colors=colors, \n                line=dict(color='#000000', \n                          width=2))\n)])\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:17:24.002502Z","iopub.execute_input":"2022-05-13T08:17:24.003098Z","iopub.status.idle":"2022-05-13T08:17:26.420096Z","shell.execute_reply.started":"2022-05-13T08:17:24.003048Z","shell.execute_reply":"2022-05-13T08:17:26.419283Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"3.2\"></a>\n## <span style=\"color:#e76f51;\"> Code cells analysis  : </span>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp;<b><u>Observations:</u></b><br>\n\n* <i> Mean Length for Code Cells is <b><u>25 words</u></b></i><br>\n* <i> Max words in a Code Cell is <b><u>74589 words</u></b></i><br>\n* <i> There are many outliers in Code Cells</i><br>\n</div>","metadata":{}},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Sample Code Cell: </span>","metadata":{}},{"cell_type":"code","source":"print(f'\\033[94m')\nprint(code_df.iloc[0][\"source\"])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:17:26.421285Z","iopub.execute_input":"2022-05-13T08:17:26.421545Z","iopub.status.idle":"2022-05-13T08:17:26.427815Z","shell.execute_reply.started":"2022-05-13T08:17:26.421513Z","shell.execute_reply":"2022-05-13T08:17:26.427092Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.2.1\"></a>\n### <span style=\"color:#e76f51;\"> Code cells Length Distribution  : </span>","metadata":{}},{"cell_type":"code","source":"code_lengths = np.array([len(code_df[\"source\"][i].split()) for i in range(len(code_df))])\nprint(f'\\033[94m Min Code Cells Length = ', min(code_lengths))\nprint(f'\\033[94m Mean Code cells Length = ', round(np.mean(code_lengths),2))\nprint(f'\\033[94m Max Code Cells Length = ', max(code_lengths))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:17:26.429286Z","iopub.execute_input":"2022-05-13T08:17:26.429614Z","iopub.status.idle":"2022-05-13T08:18:08.917541Z","shell.execute_reply.started":"2022-05-13T08:17:26.429580Z","shell.execute_reply":"2022-05-13T08:18:08.916740Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig,ax= plt.subplots(figsize= (18,6))\nplt.boxplot(code_lengths, vert = False)\nplt.xlabel(\"Lenght of Code Cells\");","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:08.918828Z","iopub.execute_input":"2022-05-13T08:18:08.919043Z","iopub.status.idle":"2022-05-13T08:18:09.951152Z","shell.execute_reply.started":"2022-05-13T08:18:08.919016Z","shell.execute_reply":"2022-05-13T08:18:09.950566Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.2.2\"></a>\n### <span style=\"color:#e76f51;\"> Code cells WordCloud  : </span>","metadata":{}},{"cell_type":"code","source":"wordcloud_notes = wordcloud.WordCloud(stopwords=wordcloud.STOPWORDS, max_font_size=120, max_words=5000,\n                      width = 600, height = 400,\n                      background_color='white').generate(\"\".join(code_df[\"source\"][:1000]))\nfig, ax = plt.subplots(figsize=(14,10))\nax.imshow(wordcloud_notes, interpolation='bilinear')\nax.set_axis_off()\nplt.imshow(wordcloud_notes);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:09.952230Z","iopub.execute_input":"2022-05-13T08:18:09.952563Z","iopub.status.idle":"2022-05-13T08:18:11.844326Z","shell.execute_reply.started":"2022-05-13T08:18:09.952534Z","shell.execute_reply":"2022-05-13T08:18:11.843456Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"## <span style=\"color:#e76f51;\"> Markdown cells analysis  : </span>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp;<b><u>Observations:</u></b><br>\n\n* <i> Mean Length for Markdown Cells is <b><u>29 words</u></b></i><br>\n* <i> Max words in a Markdown Cell is <b><u>38939 words</u></b></i><br>\n* <i> There are many outliers in Markdown Cells as well.</i><br>\n</div>","metadata":{}},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Sample Markdown Cell: </span>","metadata":{}},{"cell_type":"code","source":"print(f'\\033[94m')\nprint(mkd_df.iloc[59][\"source\"])","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:11.846648Z","iopub.execute_input":"2022-05-13T08:18:11.846910Z","iopub.status.idle":"2022-05-13T08:18:11.852968Z","shell.execute_reply.started":"2022-05-13T08:18:11.846879Z","shell.execute_reply":"2022-05-13T08:18:11.852199Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.3.1\"></a>\n### <span style=\"color:#e76f51;\"> Markdown Cells Length Distribution  : </span>","metadata":{"execution":{"iopub.status.busy":"2022-05-11T23:30:22.719828Z","iopub.execute_input":"2022-05-11T23:30:22.720332Z","iopub.status.idle":"2022-05-11T23:30:22.764389Z","shell.execute_reply.started":"2022-05-11T23:30:22.720276Z","shell.execute_reply":"2022-05-11T23:30:22.763352Z"}}},{"cell_type":"code","source":"mkd_lengths = np.array([len(mkd_df [\"source\"][i].split()) for i in range(len(mkd_df))])\nprint(f'\\033[94m Min Markdown Cells Length = ', min(mkd_lengths))\nprint(f'\\033[94m Mean Markdown cells Length = ', round(np.mean(mkd_lengths),2))\nprint(f'\\033[94m Max Markdown Cells Length = ', max(mkd_lengths))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:11.854156Z","iopub.execute_input":"2022-05-13T08:18:11.854742Z","iopub.status.idle":"2022-05-13T08:18:34.306932Z","shell.execute_reply.started":"2022-05-13T08:18:11.854695Z","shell.execute_reply":"2022-05-13T08:18:34.306099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"fig,ax= plt.subplots(figsize= (18,6))\nplt.boxplot(mkd_lengths, vert = False)\nplt.xlabel(\"Lenght of Markdown Cells\");","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:34.308346Z","iopub.execute_input":"2022-05-13T08:18:34.308669Z","iopub.status.idle":"2022-05-13T08:18:34.938936Z","shell.execute_reply.started":"2022-05-13T08:18:34.308628Z","shell.execute_reply":"2022-05-13T08:18:34.938407Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.3.3\"></a>\n### <span style=\"color:#e76f51;\"> Markdown Cells WordCloud  : </span>","metadata":{}},{"cell_type":"code","source":"wordcloud_notes = wordcloud.WordCloud(stopwords=wordcloud.STOPWORDS, max_font_size=120, max_words=5000,\n                      width = 600, height = 400,\n                      background_color='white').generate(\"\".join(mkd_df[\"source\"][:1000]))\nfig, ax = plt.subplots(figsize=(14,10))\nax.imshow(wordcloud_notes, interpolation='bilinear')\nax.set_axis_off()\nplt.imshow(wordcloud_notes);","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:34.939796Z","iopub.execute_input":"2022-05-13T08:18:34.940105Z","iopub.status.idle":"2022-05-13T08:18:37.047184Z","shell.execute_reply.started":"2022-05-13T08:18:34.940078Z","shell.execute_reply":"2022-05-13T08:18:37.046591Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"3.4\"></a>\n## <span style=\"color:#e76f51;\"> Notebooks Analysis: </span>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    📌 &nbsp;<b><u>Observations:</u></b><br>\n\n* <i> Minimum count for both Code Cells and Markdown cells across all notebooks is <b><u>1</u></b>.</i><br>\n* <i> Mean <b><u>Code cells</u></b> count across all notebooks is  <b><u>30 cells</u></b></i><br>\n* <i> Mean <b><u>Markdown cells</u></b> count across all notebooks is  <b><u>15 cells</u></b></i><br>\n* <i> Max count of <b><u>Code cells</u></b> and  <b><u>Markdown cells</u></b> across all notebooks is <b><u>809 cells</u></b> and <b><u>537 cells</u></b> respectively.</i><br>\n  \n</div>","metadata":{}},{"cell_type":"code","source":"## loading code_cell counts from notebooks \nnotebook_ids = [notebook[:-5] for notebook in os.listdir(\"../input/AI4Code/train\")]\ncode_counts= []\nmarkdown_counts= [] \nfor i in trange(len(notebook_ids)):\n    temp_df = df.loc[(notebook_ids[i])]\n    code_counts.append((temp_df[\"cell_type\"] == \"code\").sum())\n    markdown_counts.append((temp_df[\"cell_type\"] == \"markdown\").sum())\n\ncounts_df = pd.DataFrame(data = np.array([notebook_ids, code_counts, markdown_counts, ]).T, columns = [\"notebook_id\", \"code_count\", \"markdown_count\"])\ncounts_df[\"markdown_count\"] = counts_df[\"markdown_count\"].astype(str).astype(int)\ncounts_df[\"code_count\"] = counts_df[\"code_count\"].astype(str).astype(int)\ncounts_df[\"total_count\"] = counts_df[\"code_count\"] + counts_df[\"markdown_count\"]\nprint(f'\\033[94m Minimum Cell count in any notebook', counts_df[\"total_count\"].min())\nprint(f'\\033[94m Maximum Cell count in any notebook', counts_df[\"total_count\"].max())\nprint(f'\\033[94m Mean of Cell counts across all notebooks', round(counts_df[\"total_count\"].mean(), 2 ))\ncounts_df.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:18:37.048112Z","iopub.execute_input":"2022-05-13T08:18:37.048437Z","iopub.status.idle":"2022-05-13T08:39:57.807464Z","shell.execute_reply.started":"2022-05-13T08:18:37.048410Z","shell.execute_reply":"2022-05-13T08:39:57.806425Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Outlier Notebooks Analysis: </span>","metadata":{}},{"cell_type":"code","source":"k = 100\ntop_k = counts_df.sort_values(by = [\"total_count\"], ascending=False)[:k]\nfig = px.bar(data_frame=top_k, \n             x = \"notebook_id\" ,\n             y = [ \"code_count\", \"markdown_count\"], \n             color_discrete_sequence=['#DE3163', '#58D68D']\n         \n            )\nfig.update_layout(\n    title={\n        'text': \"Cell Type Count analysis for top 100 cell count notebooks(OUTLIERS)\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Notebook ID\",\n    yaxis_title=\"Count\",\n    template=\"plotly_white\"\n    \n)\nfig.update_traces(marker_line_color='black',\n                  marker_line_width=0.9,opacity = 0.9)\nfig.show()","metadata":{"execution":{"iopub.status.busy":"2022-05-13T09:06:18.787158Z","iopub.execute_input":"2022-05-13T09:06:18.788143Z","iopub.status.idle":"2022-05-13T09:06:18.931347Z","shell.execute_reply.started":"2022-05-13T09:06:18.788100Z","shell.execute_reply":"2022-05-13T09:06:18.930450Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id=\"3.4.1\"></a>\n### <span style=\"color:#e76f51;\"> Code Cell Count Analysis: </span>","metadata":{}},{"cell_type":"code","source":"print(f'\\033[94m Minimum Code Cell count in any notebook', counts_df[\"code_count\"].min())\nprint(f'\\033[94m Maximum Code Cell count in any notebook', counts_df[\"code_count\"].max())\nprint(f'\\033[94m Mean of Code Cell counts across all notebooks', round(counts_df[\"code_count\"].mean(), 2 ))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T08:39:57.809035Z","iopub.execute_input":"2022-05-13T08:39:57.809663Z","iopub.status.idle":"2022-05-13T08:39:57.819614Z","shell.execute_reply.started":"2022-05-13T08:39:57.809623Z","shell.execute_reply":"2022-05-13T08:39:57.818738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Code Cell Count Distribution across all notebooks: </span>","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(data_frame=counts_df, \n                   x= \"code_count\",\n                   color_discrete_sequence=[\"#DE3163\"],\n                   marginal=\"violin\")\nfig.update_layout(\n    title={\n        'text': \"Code Cell Count Distribution\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Code Cells\",\n    yaxis_title=\"Count\",\n    showlegend=False,\n    template=\"plotly_white\"\n)\nfig.show()\nfig = px.histogram(data_frame=counts_df[counts_df[\"code_count\"]<100], \n                   x= \"code_count\",\n                   color_discrete_sequence=[\"#58D68D\"],\n                   marginal=\"violin\")\nfig.update_layout(\n    title={\n        'text': \"Code Cell Count Distribution (COUNTS < 100 )\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Code Cells\",\n    yaxis_title=\"Count\",\n    showlegend=False,\n    template=\"plotly_white\"\n)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"3.4.2\"></a>\n### <span style=\"color:#e76f51;\"> Markdown Cell Count Analysis: </span>","metadata":{}},{"cell_type":"code","source":"print(f'\\033[94m Minimum Markdown Cell count in any notebook', counts_df[\"markdown_count\"].min())\nprint(f'\\033[94m Maximum Markdown Cell count in any notebook', counts_df[\"markdown_count\"].max())\nprint(f'\\033[94m Mean of Markdown Cell counts across all notebooks', round(counts_df[\"markdown_count\"].mean(), 2 ))","metadata":{"execution":{"iopub.status.busy":"2022-05-12T15:54:02.941955Z","iopub.execute_input":"2022-05-12T15:54:02.942267Z","iopub.status.idle":"2022-05-12T15:54:02.952449Z","shell.execute_reply.started":"2022-05-12T15:54:02.942235Z","shell.execute_reply":"2022-05-12T15:54:02.951422Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Markdown Cell Count Distribution across all notebooks: </span>","metadata":{}},{"cell_type":"code","source":"fig = px.histogram(data_frame=counts_df, \n                   x= \"markdown_count\",\n                   color_discrete_sequence=[\"#DE3163\"],\n                   marginal=\"violin\")\nfig.update_layout(\n    title={\n        'text': \"Markdown Cell Count Distribution\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Markdown Cells\",\n    yaxis_title=\"Count\",\n    showlegend=False,\n    template=\"plotly_white\"\n)\n\nfig.show()\nfig = px.histogram(data_frame=counts_df[counts_df[\"markdown_count\"]<100], \n                   x= \"markdown_count\",\n                   color_discrete_sequence=[\"#58D68D\"],\n                   marginal=\"violin\")\nfig.update_layout(\n    title={\n        'text': \"Markdown Cell Count Distribution (COUNTS < 100 )\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Markdown Cells\",\n    yaxis_title=\"Count\",\n    showlegend=False,\n    template=\"plotly_white\"\n)\nfig.show()","metadata":{"_kg_hide-input":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<a id=\"3.4.3\"></a>\n## <span style=\"color:#e76f51;\"> Minimum Cell Count Analysis: </span>","metadata":{}},{"cell_type":"code","source":"either_one = counts_df[(counts_df[\"markdown_count\"] == 1) | (counts_df[\"code_count\"] == 1 )] \nboth_one = counts_df[(counts_df[\"markdown_count\"] == 1) & (counts_df[\"code_count\"] == 1 )] \ncode_count_one = counts_df[counts_df[\"code_count\"] == 1 ]\nmarkdown_count_one  = counts_df[counts_df[\"markdown_count\"] == 1 ]\n\nprint(f\"\\033[94mTotal notebook with either 1 code cell or 1 markdown cell = \", len(either_one))\nprint(f\"\\033[94mTotal notebook with both 1 code cell and 1 markdown cell = \", len(both_one ))\nprint(f\"\\033[94mNotebook counts with only 1 code cell  = \" ,len(code_count_one))\nprint(f\"\\033[94mNotebook counts with only 1 markdown cell  = \" ,len(markdown_count_one))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-12T16:33:02.142244Z","iopub.execute_input":"2022-05-12T16:33:02.142548Z","iopub.status.idle":"2022-05-12T16:33:02.168784Z","shell.execute_reply.started":"2022-05-12T16:33:02.142518Z","shell.execute_reply":"2022-05-12T16:33:02.16746Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### <span style=\"color:#e76f51;\"> Code cells count vs Markdown cells count: </span>","metadata":{}},{"cell_type":"code","source":"fig = px.scatter(data_frame=counts_df, \n                 x = \"code_count\", \n                 y = \"markdown_count\", \n                 size = \"code_count\",\n                 color_discrete_sequence=[\"#DE3163\"])\nfig.add_shape(type='line',\n                x0=0,\n                y0=0,\n                x1=800,\n                y1=800,\n                line=dict(color='Black'),\n                xref='x',\n                yref='y',name = \"X=Y line\"\n             )\nfig.update_layout(\n    title={\n        'text': \"Code Cell Counts vs Markdown Cell Counts\",\n        'y':0.95,\n        'x':0.5,\n        'xanchor': 'center',\n        'yanchor': 'top'},\n    xaxis_title=\"Code Cell Counts\",\n    yaxis_title=\"Markdown Cell Counts\",\n    showlegend=False,\n    template=\"plotly_white\"\n)\nfig.show()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-05-13T09:08:46.292780Z","iopub.execute_input":"2022-05-13T09:08:46.293065Z","iopub.status.idle":"2022-05-13T09:08:47.379123Z","shell.execute_reply.started":"2022-05-13T09:08:46.293036Z","shell.execute_reply":"2022-05-13T09:08:47.378162Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a href=\"#toc\" role=\"button\" aria-pressed=\"true\" >⬆️Back to Table of Contents ⬆️</a>","metadata":{}},{"cell_type":"markdown","source":"<div class=\"alert alert-block alert-info\" style=\"font-size:14px; font-family:verdana; line-height: 1.7em;\">\n    \n    \n### <center>Thank you for reading🙂</center>\n### <center>If you have any feedback or find anything wrong, please let me know!</center>\n","metadata":{}}]}