{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"![Big Data Derby Logo][1]\n\n[1]: https://storage.googleapis.com/kaggle-competitions/kaggle/37644/logos/header.png?t=2022-08-05-15-26-16\n\n\n# Big Data Derby #\n\n***","metadata":{}},{"cell_type":"markdown","source":"Analyze horse racing data to improve the health of the horse and strategy of competition on [Kaggle](https://www.kaggle.com/competitions/big-data-derby-2022/overview)\n","metadata":{}},{"cell_type":"markdown","source":"# Table Of Contents #\n\n***\n1. [Introduction](#intro)\n\n    * [Dataset Description](#description)\n   \n      * [Column ID's](#colid)  \n       \n    * [Objective](#objective)\n   \n    * [Importing Libraries](#import)  \n   \n2. [Reading in the Data](#readdata)\n\n    * [Viewing the Dataset](#viewdata)\n    \n    * [Cleaning the Dataset](#cleandata)\n    \n3. [Exploratory Data Analysis](#eda)\n\n    * [Plotting the Data](#plotdata)\n    \n4. [Equine Safety Analysis](#eqa)\n","metadata":{}},{"cell_type":"markdown","source":"<a id='intro'></a>\n# Introduction #\n***","metadata":{}},{"cell_type":"markdown","source":"<a id='description'></a>\n## Dataset description ##\n\nThe dataset's provided by NYTHA/NYRA which inludes, but not limited to: tracks races took place, race date, weight carried, jockey, odds, course type, course condition, etc.\n\nAnother dataset with all the dataset's combined is also provided.","metadata":{}},{"cell_type":"markdown","source":"<a id='colid'></a>\n### Column ID's ###\n\n* track_id - 3 character id for the track the race took place at. AQU -Aqueduct, BEL - Belmont, SAR - Saratoga.\n* race_date - date the race took place. YYYY-MM-DD.\n* race_number - Number of the race. Passed as 3 characters but can be cast or converted to int for this data set.\n* program_number - Program number of the horse in the race passed as 3 characters. Should remain 3 characters as it isn't limited to just numbers. Is essentially the unique identifier of the horse in the race.\n* trakus_index - The common collection of point of the lat / long of the horse in the race passed as an integer. From what we can tell, it's collected every 0.25 seconds.\n* latitude - The latitude of the horse in the race passed as a float.\n* longitude - The longitude of the horse in the race passed as a float.\n* distance_id - Distance of the race in furlongs passed as an integer. Example - 600 would be 6 furlongs.\n* course_type - The course the race was run over passed as one character. M - Hurdle, D - Dirt, O - Outer turf, I - Inner turf, T - turf.\n* track_condition - The condition of the course the race was run on passed as three characters. YL - Yielding, FM - Firm, SY - Sloppy, GD - Good, FT - Fast, MY - Muddy, SF - Soft.\n* run_up_distance - Distance in feet of the gate to the start of the race passed as an integer.\n* race_type - The classification of the race passed as as five characters. STK - Stakes, WCL - Waiver Claiming, WMC - Waiver Maiden Claiming, SST - Starter Stakes, SHP - Starter Handicap, CLM - Claiming, STR - Starter Allowance, AOC - Allowance Optionl Claimer, SOC - Starter Optional Claimer, MCL - Maiden Claiming, ALW - Allowance, MSW - Maiden Special Weight.\n* purse - Purse in US dollars of the race passed as an money with two decimal places.\n* post_time - Time of day the race began passed as 5 character. Example - 01220 would be 12:20.\n* weight_carried - An integer of the weight carried by the horse in the race.\n* jockey - Name of the jockey on the horse in the race. 50 character max.\n* odds - Odds to win the race passed as an integer. Divide by 100 to derive the odds to 1. Example - 1280 would be 12.8-1.","metadata":{}},{"cell_type":"markdown","source":"<a id='objective'></a>\n## Objective ###\n\nThe objective of the dataset is to analyze horse racing tactics, drafting strategies, and path efficiency to find solutions to equine safety \n\nWe will do this by cleaning the dataset, and performig EDA through building statistical models and visualizations.\n\n","metadata":{}},{"cell_type":"markdown","source":"<a id='import'></a>\n\n\n## Importing Libraries ##\n\nFist we will import the libraries needed to perform these functions.\n\nWe will be importing Pandas for processing the dataset,Numpy and Stastmodels for mathematical and statistical computations, and Matplotlib and Seaborn for data visualization. \n\nAfter importing the libraries we will move onto reading in the dataset.","metadata":{}},{"cell_type":"code","source":"# Importing libraries\n\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nimport seaborn as sns\nimport statsmodels as sm\nimport time\n%matplotlib inline\nimport os\nfor dirname, _, filenames in os.walk('/kaggle/input'):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))\n\n!pip install utm\nimport utm\n\nimport re\nimport warnings\n\nfrom shapely.geometry import Point\nfrom shapely.geometry.polygon import Polygon\nfrom tqdm.auto import tqdm\ntqdm.pandas()\n\nimport plotly.express as px\nfrom plotly.subplots import make_subplots\nimport plotly.graph_objects as go\n\nwarnings.simplefilter('ignore')\nsns.set_style(\"darkgrid\")","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-09-02T17:35:56.954449Z","iopub.execute_input":"2022-09-02T17:35:56.956019Z","iopub.status.idle":"2022-09-02T17:36:08.757044Z","shell.execute_reply.started":"2022-09-02T17:35:56.955955Z","shell.execute_reply":"2022-09-02T17:36:08.755354Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='readdata'></a>\n# Reading in the Data #\n\n***\n\nNow I will read in the dataset using Pandas read_csv() function.\n\nI will only be using the complete.csv for this analysis, for it has all the dataset's combined.","metadata":{}},{"cell_type":"code","source":"# Reading in the dataset using read_csv()\n\ndf = pd.read_csv('../input/big-data-derby-2022/nyra_2019_complete.csv', low_memory=False)\n\n# Printing the first 5 rows of the dataset.\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:08.760075Z","iopub.execute_input":"2022-09-02T17:36:08.760550Z","iopub.status.idle":"2022-09-02T17:36:22.638669Z","shell.execute_reply.started":"2022-09-02T17:36:08.760500Z","shell.execute_reply":"2022-09-02T17:36:22.637287Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='viewdata'></a>\n## Viewing the Dataset ##\n\nNow I will be viewing parts of the dataset such as how many rows and columns there is, as wel las some basic statistics about the dataset.\n","metadata":{}},{"cell_type":"code","source":"# I will now use the shape function to view how many rows, and columns there are, respectively.\n\ndf.shape","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:22.640387Z","iopub.execute_input":"2022-09-02T17:36:22.641389Z","iopub.status.idle":"2022-09-02T17:36:22.649297Z","shell.execute_reply.started":"2022-09-02T17:36:22.641348Z","shell.execute_reply":"2022-09-02T17:36:22.648161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see there are 5,228,430 rows of data and 17 columns.","metadata":{}},{"cell_type":"markdown","source":"Next, I will look at some statistics about the datset. Such as, the mean for each column, and unique items ine each column. I will do this by using the describe function.\n\n","metadata":{}},{"cell_type":"code","source":"# Using the describe function for some basic statistics.\n\ndf.describe(include='all')","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:22.652367Z","iopub.execute_input":"2022-09-02T17:36:22.653399Z","iopub.status.idle":"2022-09-02T17:36:27.192696Z","shell.execute_reply.started":"2022-09-02T17:36:22.653347Z","shell.execute_reply":"2022-09-02T17:36:27.191228Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"I also want to see if there are any columns with missing or duplicated data, and decide on which columns I will be using for my analysis. I will acheive this by using the info function.","metadata":{}},{"cell_type":"code","source":"# Here the info function is uded to print out the columns, and specific information such as; \n# column name, the data type for each column, and whether or not there are any null values\n\ndf.info(show_counts=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:27.194478Z","iopub.execute_input":"2022-09-02T17:36:27.195016Z","iopub.status.idle":"2022-09-02T17:36:28.846498Z","shell.execute_reply.started":"2022-09-02T17:36:27.194968Z","shell.execute_reply":"2022-09-02T17:36:28.845239Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like ther are no non-null values in the dataset. So, I will not need to alter any of the values in the dataset.\n\nNext, I wll check for any duplicate data in the dataset.\n\nI will do this by using the duplicated() function.","metadata":{}},{"cell_type":"code","source":"## Here I check for duplicates\n\nsum(df.duplicated())","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:28.847940Z","iopub.execute_input":"2022-09-02T17:36:28.848591Z","iopub.status.idle":"2022-09-02T17:36:39.810168Z","shell.execute_reply.started":"2022-09-02T17:36:28.848555Z","shell.execute_reply":"2022-09-02T17:36:39.809003Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Look like there are no duplicated values in the dataset.\n\nOnto cleaning the dataset.","metadata":{}},{"cell_type":"markdown","source":"<a id='cleandata'></a>\n## Cleaning the Dataset ##\n","metadata":{}},{"cell_type":"markdown","source":"Although I do have a clean dataset there is one thing I do want to clean up.\n\nIt seems as thoug all the numerical values are in scientific notation.\n\nI will be removing the scientific notation from these values to make it easier for the audience to read.","metadata":{}},{"cell_type":"code","source":"## Here is how I changed the scientific notation to float with 2 decimal points using the options function.\n\npd.options.display.float_format = '{:.2f}'.format\n\ndf.head()","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:39.811687Z","iopub.execute_input":"2022-09-02T17:36:39.812100Z","iopub.status.idle":"2022-09-02T17:36:39.832923Z","shell.execute_reply.started":"2022-09-02T17:36:39.812062Z","shell.execute_reply":"2022-09-02T17:36:39.831787Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df.describe(include='all')","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:39.834467Z","iopub.execute_input":"2022-09-02T17:36:39.834887Z","iopub.status.idle":"2022-09-02T17:36:44.382862Z","shell.execute_reply.started":"2022-09-02T17:36:39.834852Z","shell.execute_reply":"2022-09-02T17:36:44.381527Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now I have completed cleaning the dataset to my specifications.\n\nNext up, EDA. Exploratory Data Analysis.\n","metadata":{}},{"cell_type":"markdown","source":"<a id='eda'></a>\n# Exploratory Data Analysis #\n***","metadata":{"execution":{"iopub.status.busy":"2022-08-21T18:10:07.018348Z","iopub.execute_input":"2022-08-21T18:10:07.019018Z","iopub.status.idle":"2022-08-21T18:10:07.025736Z","shell.execute_reply.started":"2022-08-21T18:10:07.018980Z","shell.execute_reply":"2022-08-21T18:10:07.024440Z"}}},{"cell_type":"markdown","source":"In this section I will explore the data set and observe any trends in the data.","metadata":{}},{"cell_type":"markdown","source":"<a id='plotdata'></a>\n## Plotting the Data","metadata":{}},{"cell_type":"code","source":"## Here I used a basic histogram function to plot all the numerical variables in the dataset.\n\ndf.hist(alpha=.5, rwidth=1, ec='black', figsize=(20,15));\nplt.suptitle('Big Data Derby 2022', x=0.5, y=1, fontsize='xx-large');\nplt.ticklabel_format(style='plain', useOffset=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:44.387051Z","iopub.execute_input":"2022-09-02T17:36:44.387497Z","iopub.status.idle":"2022-09-02T17:36:47.933128Z","shell.execute_reply.started":"2022-09-02T17:36:44.387457Z","shell.execute_reply":"2022-09-02T17:36:47.932073Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plotting histogram for average jockey weight\n\ndf.hist(column='weight_carried', alpha=.5, rwidth=1, ec='black', figsize=(10,8));\nplt.suptitle('Average Jockey Weight', x=0.5, y=1, fontsize='xx-large');\nplt.ticklabel_format(style='plain')\nplt.xlabel('Weight');\nplt.ylabel('Count');\nplt.title(\"\");","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:47.936370Z","iopub.execute_input":"2022-09-02T17:36:47.937114Z","iopub.status.idle":"2022-09-02T17:36:48.382067Z","shell.execute_reply.started":"2022-09-02T17:36:47.937069Z","shell.execute_reply":"2022-09-02T17:36:48.380696Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Looks like there may be some outliers in this data.\n\nLets remove those so our result scan be stastically more significant.","metadata":{}},{"cell_type":"code","source":"# Dropping outliers \n\ndf.drop(df[df.weight_carried >= 140].index, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:48.383773Z","iopub.execute_input":"2022-09-02T17:36:48.384333Z","iopub.status.idle":"2022-09-02T17:36:49.397570Z","shell.execute_reply.started":"2022-09-02T17:36:48.384291Z","shell.execute_reply":"2022-09-02T17:36:49.396201Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"# Plottin histogram after removing outliers\n\ndf.hist(column='weight_carried', alpha=.5, rwidth=1, ec='black', figsize=(10,8));\nplt.suptitle('Average Jockey Weight', x=0.5, y=1, fontsize='xx-large');\nplt.ticklabel_format(style='plain')\nplt.xlabel('Weight');\nplt.ylabel('Count');\nplt.title(\"\");","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:36:49.399391Z","iopub.execute_input":"2022-09-02T17:36:49.400208Z","iopub.status.idle":"2022-09-02T17:36:49.838644Z","shell.execute_reply.started":"2022-09-02T17:36:49.400153Z","shell.execute_reply":"2022-09-02T17:36:49.837278Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After removing the outliers we see the histogram now is a little more normally distributed.\n","metadata":{}},{"cell_type":"code","source":"color_dict = {'D' : 'blue',\n             'I' : 'yellow',\n             'O' : 'red',\n             'T' : 'orange',\n             'M' : 'grey'}\n\n\n\nfor track in ['AQU', 'BEL', 'SAR']:\n    plt.figure(figsize = (16, 8))\n\n    sns.scatterplot(data = df[(df.track_id == track)], x = \"longitude\", y = \"latitude\", \n                    hue = \"course_type\", \n                    marker='.',\n                   palette=color_dict, )\n\n    plt.title(track)\n    plt.legend()\n\n    plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-09-02T17:41:49.157032Z","iopub.execute_input":"2022-09-02T17:41:49.158205Z","iopub.status.idle":"2022-09-02T17:43:43.541445Z","shell.execute_reply.started":"2022-09-02T17:41:49.158146Z","shell.execute_reply":"2022-09-02T17:43:43.539949Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"<a id='eqa'></a>\n\n# Equine Safety Analysis\n***","metadata":{}},{"cell_type":"markdown","source":"In this section we will analyze the data to see if we can find any information that will help owners keep their horses safer.\n\nThis will be diffult as the dataset, although very big; is very limited in the information it has.\n\nIt would of been better if we had data for variables such as heart rate, EKG's, movement, etc.\n\nThis data does include variables such as track distance, jockey weight, race type, course condition and track condition which may lead to some other insights though.\n\nLet's see what we can find.","metadata":{}},{"cell_type":"code","source":"","metadata":{},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# WORK IN PROGRESS... #","metadata":{}}]}