{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# 7 Coolest Python Packages Top Kagglers Are Using Without Telling You\n## Let me expose the secrets...\n\n\n \n Photo by \n Miesha Maiden\n on \n Pexels. All images are by author unless specified otherwise.\n \n","metadata":{}},{"cell_type":"markdown","source":"## Setup","metadata":{"_kg_hide-input":true}},{"cell_type":"code","source":"import warnings\n\nimport matplotlib.pyplot as plt\nimport numpy as np\nimport optuna\nimport pandas as pd\nimport seaborn as sns\n\nwarnings.filterwarnings(\"ignore\")\noptuna.logging.set_verbosity(optuna.logging.WARNING)","metadata":{"execution":{"iopub.status.busy":"2021-08-06T10:07:33.134762Z","iopub.execute_input":"2021-08-06T10:07:33.135224Z","iopub.status.idle":"2021-08-06T10:07:33.142201Z","shell.execute_reply.started":"2021-08-06T10:07:33.135192Z","shell.execute_reply":"2021-08-06T10:07:33.140919Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Kaggle is a hot spot for what is trending in data science and machine learning.\n\nDue to its competitiveness, the top players are constantly looking for new tools, technologies, and frameworks that give them an edge over others. If a new package or an algorithm delivers actionable value, there is a good chance it receives immediate adoption and becomes popular.\n\nThis post is about 7 of such trendies packages that are direct replacements for many tools and technologies that are either outdated or need urgent upgrades.","metadata":{}},{"cell_type":"markdown","source":"# 1. UMAP","metadata":{}},{"cell_type":"markdown","source":"\n\n \n Link to the code of the plot.\n \n","metadata":{}},{"cell_type":"markdown","source":"Above is a 100k row dataset with 75 features projected to 2D using a package called UMAP. Each dot represents a single sample in a classification problem and is color-encoded based on their class.\n\nMassive datasets like these can make you miserable during EDA, mainly because of the computation and time expenses they come with. So, it is important that each plot you create is spot-on and reveals something significant about the data.\n\nI think that's one of the reasons why UMAP (Uniform Manifold Approximation and Projection) is so well-received on Kaggle. It is efficient, low-code, and lets you take a real \"look\" at the data from a high-dimensional perspective:\n\n\n ![](\"https://umap-learn.readthedocs.io/en/latest/_images/plotting_21_2.png\")
\n ![](\"https://umap-learn.readthedocs.io/en/latest/_images/plotting_32_2.png\")
\n ![](\"https://umap-learn.readthedocs.io/en/latest/_images/plotting_34_2.png\")
\n=6.0)\n\nIf you want to try out the library without installation limitations, Kaggle kernels are a great option. Here is a [notebook](https://www.kaggle.com/rohanrao/tutorial-on-reading-large-datasets?scriptVersionId=49328159&cellId=14) to get you started.\n\n### 🛠 GitHub and documentation\n- https://docs.rapids.ai/api/cudf/stable/\n- https://github.com/rapidsai/cudf\n\n### 💻 Demo\nHere is a snippet from the documentation that shows a simple GroupBy operation on the tips dataset:","metadata":{}},{"cell_type":"markdown","source":"```python\nimport cudf, io, requests\nfrom io import StringIO\n\nurl = \"https://github.com/plotly/datasets/raw/master/tips.csv\"\ncontent = requests.get(url).content.decode('utf-8')\n\ntips_df = cudf.read_csv(StringIO(content))\ntips_df['tip_percentage'] = tips_df['tip'] / tips_df['total_bill'] * 100\n\n# display average tip by dining party size\nprint(tips_df.groupby('size').tip_percentage.mean())\n```","metadata":{}},{"cell_type":"markdown","source":"# 7. Automatic EDA libraries","metadata":{}},{"cell_type":"markdown","source":"Normally, I am against any library or tool that takes a programmer away from writing actual code. But, since auto-EDA libraries are all the rage now on Kaggle, I had to include this section for completeness.\n\nInitially, this section was supposed to be only about AutoViz, which uses XGBoost under the hood to display the most important information of the dataset (that's why I chose it). Later, I decided to include a few others as well.\n\nHere is a list of the best auto EDA libraries I have found:\n- **DataPrep** - the most comprehensive auto EDA \\[[GitHub](https://github.com/sfu-db/dataprep), [Documentation](https://docs.dataprep.ai/index.html)\\]\n- **AutoViz** - the fastest auto EDA \\[[GitHub](https://github.com/AutoViML/AutoViz)]\n- **PandasProfiling** - the earliest and one of the best auto EDA tools \\[[GitHub](https://github.com/pandas-profiling/pandas-profiling), [Documentation](https://pandas-profiling.github.io/pandas-profiling/docs/master/rtd/)\\]\n- **Lux** - the most user-friendly and luxurious EDA \\[[GitHub](https://github.com/lux-org/lux), [Documentation](https://lux-api.readthedocs.io/en/latest/)\\]\n\n### 💻 Demo\nIf you want to see how each package performs EDA, check out this great [notebook](https://www.kaggle.com/andreshg/automatic-eda-libraries-comparisson/notebook#6.-%F0%9F%93%8A-D-Tale-%F0%9F%93%9A) on Kaggle.","metadata":{}},{"cell_type":"markdown","source":"## Summary","metadata":{}},{"cell_type":"markdown","source":"The available tools and packages to execute data science tasks are endless. Everyone has the right to be overwhelmed by this.\n\nBut I sincerely hope the tools outlined in this posted helped narrow down your focus to what is trending and delivering results in the hands of members of the largest ML community.","metadata":{}},{"cell_type":"markdown","source":"## Here are my latest viral hits...\n- [20 Burning XGBoost FAQs Answered to Use the Library Like a Pro](https://towardsdatascience.com/20-burning-xgboost-faqs-answered-to-use-the-library-like-a-pro-f8013b8df3e4?source=your_stories_page-------------------------------------)\n- [25 Pandas Functions You Didn’t Know Existed | P(Guarantee) = 0.8](https://towardsdatascience.com/25-pandas-functions-you-didnt-know-existed-p-guarantee-0-8-1a05dcaad5d0?source=your_stories_page-------------------------------------)\n- [Matplotlib vs. Plotly: Let’s Decide Once and for All\n](https://towardsdatascience.com/matplotlib-vs-plotly-lets-decide-once-and-for-all-ad25a5e43322?source=your_stories_page-------------------------------------)","metadata":{}}]}