{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# **Introduction**\n\nThe goal of this competition is to analyze horse racing tactics, drafting strategies, and path efficiency. A model is to be developed with the use of never-before-released coordinate data along with basic race information.\n\nThis notebook might help racing horse owners, trainers, and veterinarians better understand how equine performance and welfare fit together. With better data analysis, equine welfare could significantly improve.\n\nOne of the challenge is to develop a efficient explanatory data analysis report that will make the process of understanding dataset in more convenient processes. As this notebook was being worked on, a ilbrary named ` pandas_profiling` was discovered.\n\n## [Pandas_Profiling](https://pandas-profiling.ydata.ai/docs/master/pages/getting_started/overview.html)\npandas-profiling generates profile reports from a pandas DataFrame. The pandas df.describe() function is handy yet a little basic for exploratory data analysis. pandas-profiling extends pandas DataFrame with df.profile_report(), which automatically generates a standardized univariate and multivariate report for data understanding.\n\nFor each column, the following information (whenever relevant for the column type) is presented in an interactive HTML report:\n\n  `Type inference`: detect the types of columns in a DataFrame\n  `Essentials`: type, unique values, indication of missing values\n  `Quantile statistics`: minimum value, Q1, median, Q3, maximum, range, interquartile range\n  `Descriptive statistics`: mean, mode, standard deviation, sum, median absolute deviation, coefficient of variation, kurtosis, skewness\n    Most frequent and extreme values\n    `Histograms`: categorical and numerical\n    `Correlations`: high correlation warnings, based on different correlation metrics (Spearman, Pearson, Kendall, Cramér’s V, Phik)\n    `Missing values`: through counts, matrix, heatmap and dendrograms\n    `Duplicate rows`: list of the most common duplicated rows\n    `Text analysis`: most common categories (uppercase, lowercase, separator), scripts (Latin, Cyrillic) and blocks (ASCII, Cyrilic)\n    `File and Image analysis`: file sizes, creation dates, dimensions, indication of truncated images and existance of EXIF metadata\n","metadata":{}},{"cell_type":"markdown","source":"# **Setting up the environment**","metadata":{}},{"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nfrom pandas_profiling import ProfileReport\n\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n        \n        \n#Importing the dataset        \ncomplete_data=pd.read_csv(\"../input/big-data-derby-2022/nyra_2019_complete.csv\",low_memory=False)\ncomplete_data.head()","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-08-20T03:34:06.418905Z","iopub.execute_input":"2022-08-20T03:34:06.419471Z","iopub.status.idle":"2022-08-20T03:34:25.445641Z","shell.execute_reply.started":"2022-08-20T03:34:06.419351Z","shell.execute_reply":"2022-08-20T03:34:25.444315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As Pandas Profiling requires huge amount of computing resources to work with big data, there are different ways to deal with large dataset.","metadata":{}},{"cell_type":"markdown","source":"# **Different ways to deal with huge datasets**","metadata":{}},{"cell_type":"markdown","source":"As the dataset is huge, there are different processes of setting up the report. They are:","metadata":{}},{"cell_type":"markdown","source":"1. **Minimal Mode** <br>\n`pandas-profiling` includes a minimal configuration file where the most expensive computations are turned off by default.","metadata":{}},{"cell_type":"code","source":"minimalMode_Profile = ProfileReport(complete_data, title=\"Pandas Profiling Report\",minimal=True)\nminimalMode_Profile.to_notebook_iframe()","metadata":{"execution":{"iopub.status.busy":"2022-08-20T03:45:51.066389Z","iopub.execute_input":"2022-08-20T03:45:51.070046Z","iopub.status.idle":"2022-08-20T03:46:27.639116Z","shell.execute_reply.started":"2022-08-20T03:45:51.069990Z","shell.execute_reply":"2022-08-20T03:46:27.638214Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"2. **Sample the dataset**","metadata":{}},{"cell_type":"code","source":"# Sample 100,000 rows\nsample = complete_data.sample(100000)\n\nsample_profile = ProfileReport(sample)\nsample_profile.to_notebook_iframe()","metadata":{"execution":{"iopub.status.busy":"2022-08-20T03:47:31.078210Z","iopub.execute_input":"2022-08-20T03:47:31.078798Z","iopub.status.idle":"2022-08-20T03:48:26.395219Z","shell.execute_reply.started":"2022-08-20T03:47:31.078757Z","shell.execute_reply":"2022-08-20T03:48:26.393696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Thank you visiting this notebook, Please upvote it if you liked this notebook**","metadata":{}}]}