{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Kaggle Competition - 🏇 Big Data Derby 2022","metadata":{}},{"cell_type":"code","source":"!pip install pymap3d","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:11.615106Z","iopub.execute_input":"2022-08-29T10:12:11.615550Z","iopub.status.idle":"2022-08-29T10:12:22.970277Z","shell.execute_reply.started":"2022-08-29T10:12:11.615515Z","shell.execute_reply":"2022-08-29T10:12:22.968617Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport matplotlib.animation as plt_anim\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\nimport os\nimport random\nimport ipywidgets\nimport pymap3d as pm\n\nfrom tqdm.notebook import tqdm\n\n%matplotlib inline\nsns.set_theme(style='darkgrid')\npd.options.mode.chained_assignment = None\n\npath = \"../input/big-data-derby-2022\"\n\nfor dirname, _, filenames in os.walk(path):\n    for filename in filenames:\n        print(os.path.join(dirname, filename))","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:22.974201Z","iopub.execute_input":"2022-08-29T10:12:22.974766Z","iopub.status.idle":"2022-08-29T10:12:22.993499Z","shell.execute_reply.started":"2022-08-29T10:12:22.974720Z","shell.execute_reply":"2022-08-29T10:12:22.992368Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***\n## Introduction\n***\n\n### Context 🏇\n\nInjury prevention is a critical component in modern athletics. Sports that involve animals, such as horse racing, are no different than human sport. Typically, efficiency in movement correlates to both improvements in performance and injury prevention.\n\nA wealth of data is now collected, including measures for heart rate, EKG, longitudinal movement, dorsal/ventral movement, medial/lateral deviation, total power and total landing vibration. Your data science skills and analysis are needed to decipher what makes the most positive impact.\n\nIn this competition, you will create a model to interpret one aspect of this new data. You’ll be among the first to access X/Y coordinate mapping of horses during races. Using the data, you might analyze jockey decision making, compare race surfaces, or measure the relative importance of drafting. With considerable data, contestants can flex their creativity problem solving skills.\n\nThe New York Racing Association (NYRA) and the New York Thoroughbred Horsemen's Association (NYTHA) conduct world class thoroughbred racing at Aqueduct Racetrack, Belmont Park and Saratoga Race Course.\n\nWith your help, NYRA and NYTHA will better understand their vast data set, which could lead to new ways of racing and training in a highly traditional industry. With improved use of horse tracking data, you could help improve equine welfare, performance and rider decision making.\n\n### Goal of the Competition 🏇\n\nThe goal of this competition is to analyze horse racing tactics, drafting strategies, and path efficiency. You will develop a model using never-before-released coordinate data along with basic race information.\n\nYour work will help racing horse owners, trainers, and veterinarians better understand how equine performance and welfare fit together. With better data analysis, equine welfare could significantly improve.\n\n### Columns in dataset 🏇\n\n- `track_id`: 3 character id for the track the race took place at. AQU -Aqueduct, BEL - Belmont, SAR - Saratoga. \n- `race_date`\n- `race_number` \n- `program_number`\n- `trakus_index`:  Trakus is a speed-distance curve measurement system for thoroughbred horseracing that measures the speed of each horse in the race with respect to its distance travelled in the race and from that measurement, the distance between horses at the time of POC (point of call) can be measured.\n- `latitude` \n- `longitude` \n- `distance_id` \n- `course_type`: The course the race was run over passed as one character. M - Hurdle, D - Dirt, O - Outer turf, I - Inner turf, T - turf.\n- `track_condition` \n- `run_up_distance`: “Run-up” is the distance from where the gate is placed and the timing of the race begins – that is, the point at which the horses reach the published distance of the race.\n- `race_type`: The classification of the race passed as as five characters. STK - Stakes, WCL - Waiver Claiming, WMC - Waiver Maiden Claiming, SST - Starter Stakes, SHP - Starter Handicap, CLM - Claiming, STR - Starter Allowance, AOC - Allowance Optionl Claimer, SOC - Starter Optional Claimer, MCL - Maiden Claiming, ALW - Allowance, MSW - Maiden Special Weight. \n- `purse`: In horse racing, the term purse refers to the total amount of money paid out to the owners of horses racing at a particular track over a given period of time, or to the percentages of a race's total purse that are awarded to each of the highest finishers. \n- `post_time`: Post time is the designated starting time of a horse race.\n- `weight_carried` \n- `jockey`: Name of jockey.\n- `odds`","metadata":{}},{"cell_type":"markdown","source":"***\n## Exploratory Data Analysis\n***","metadata":{}},{"cell_type":"code","source":"#loading the whole dataset\nnyra_tracking = pd.read_csv(f\"{path}/nyra_tracking_table.csv\")\nnyra_start = pd.read_csv(f\"{path}/nyra_start_table.csv\")\nnyra_race = pd.read_csv(f\"{path}/nyra_race_table.csv\")\nnyra_2019 = pd.read_csv(f\"{path}/nyra_2019_complete.csv\")","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:22.994966Z","iopub.execute_input":"2022-08-29T10:12:22.995461Z","iopub.status.idle":"2022-08-29T10:12:43.582372Z","shell.execute_reply.started":"2022-08-29T10:12:22.995423Z","shell.execute_reply":"2022-08-29T10:12:43.579955Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_tracking.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:43.587512Z","iopub.execute_input":"2022-08-29T10:12:43.588217Z","iopub.status.idle":"2022-08-29T10:12:43.629059Z","shell.execute_reply.started":"2022-08-29T10:12:43.588156Z","shell.execute_reply":"2022-08-29T10:12:43.627827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#print tracks distribution\nnyra_tracking['track_id'].value_counts()/len(nyra_tracking['track_id'])*100","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:43.630878Z","iopub.execute_input":"2022-08-29T10:12:43.631979Z","iopub.status.idle":"2022-08-29T10:12:43.967657Z","shell.execute_reply.started":"2022-08-29T10:12:43.631929Z","shell.execute_reply":"2022-08-29T10:12:43.966145Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#print number of different races for each tracks\nnyra_tracking.groupby(by=\"track_id\")['race_number'].nunique()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:43.969779Z","iopub.execute_input":"2022-08-29T10:12:43.970553Z","iopub.status.idle":"2022-08-29T10:12:44.698541Z","shell.execute_reply.started":"2022-08-29T10:12:43.970514Z","shell.execute_reply":"2022-08-29T10:12:44.696942Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#print total number of race\nprint(\"Total number of race :\",len(nyra_race))","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:44.700150Z","iopub.execute_input":"2022-08-29T10:12:44.700516Z","iopub.status.idle":"2022-08-29T10:12:44.707025Z","shell.execute_reply.started":"2022-08-29T10:12:44.700483Z","shell.execute_reply":"2022-08-29T10:12:44.705707Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_race.groupby(by=['course_type','track_condition','race_type']).size().sort_values(ascending=False)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:44.708945Z","iopub.execute_input":"2022-08-29T10:12:44.709435Z","iopub.status.idle":"2022-08-29T10:12:44.734183Z","shell.execute_reply.started":"2022-08-29T10:12:44.709388Z","shell.execute_reply":"2022-08-29T10:12:44.733161Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Tracks Visualisation","metadata":{}},{"cell_type":"code","source":"def plot_track(track, race, allrace):\n    data = nyra_2019\n    plt.figure(figsize = (16,8))\n    if allrace:\n        sns.scatterplot(data = data[(data[\"track_id\"] == track)], x = \"longitude\", y = \"latitude\", hue = \"course_type\", palette = \"dark\")\n        plt.legend()\n        plt.title(f\"Track {track}\")\n    else:\n        sns.scatterplot(data = data[(data[\"track_id\"] == track) & (data[\"race_number\"] == race)], x = \"longitude\", y = \"latitude\", hue = \"course_type\", palette = \"dark\")\n        plt.title(f\"Track {track} - Race n°{race}\")","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:44.735462Z","iopub.execute_input":"2022-08-29T10:12:44.736448Z","iopub.status.idle":"2022-08-29T10:12:44.744457Z","shell.execute_reply.started":"2022-08-29T10:12:44.736411Z","shell.execute_reply":"2022-08-29T10:12:44.742986Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"track_wid = ipywidgets.ToggleButtons(\n    options=nyra_tracking['track_id'].unique(),\n    description='Track (ID):',\n    disabled=False,\n    button_style=''\n)\n\nrace_wid = ipywidgets.IntSlider(\n    value=1,\n    min=1,\n    max=nyra_tracking[nyra_tracking['track_id'] == 'AQU'][\"race_number\"].max(),\n    step=1,\n    description='Race number:',\n    disabled=True\n)\n\nallrace_wid = ipywidgets.Checkbox(\n    value=True,\n    description='All',\n    disabled=False\n)\n\n\ndef update_race_range(*args):\n    race_wid.max = nyra_tracking[nyra_tracking['track_id'] == track_wid.value][\"race_number\"].max()\n\ndef toggle_disable(*args):\n    race_wid.disabled = allrace_wid.value\n\ntrack_wid.observe(update_race_range, 'value')\nallrace_wid.observe(toggle_disable, 'value')","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:44.749480Z","iopub.execute_input":"2022-08-29T10:12:44.749950Z","iopub.status.idle":"2022-08-29T10:12:45.629984Z","shell.execute_reply.started":"2022-08-29T10:12:44.749910Z","shell.execute_reply":"2022-08-29T10:12:45.628507Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"track_visualiser = ipywidgets.interact_manual(plot_track ,track=track_wid, race=race_wid, allrace=allrace_wid)\ntrack_visualiser.widget.children[3].description = 'Display'\ntrack_visualiser","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:45.631703Z","iopub.execute_input":"2022-08-29T10:12:45.632339Z","iopub.status.idle":"2022-08-29T10:12:45.673273Z","shell.execute_reply.started":"2022-08-29T10:12:45.632298Z","shell.execute_reply":"2022-08-29T10:12:45.672028Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***\n## Data Augmentation\n***\n\nBefore continuing the EDA, let's make some data augmentation.\n\nIn fact, this dataset does not contain data such as : \n\n- `time` \n- `speed (at a t moment)` \n- `acceleration (at a t moment)`\n- `position (in the race)`\n\nHowever, we can easily infer those values that could be really usefull later.","metadata":{}},{"cell_type":"code","source":"nyra_2019.program_number.unique()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:45.675329Z","iopub.execute_input":"2022-08-29T10:12:45.675670Z","iopub.status.idle":"2022-08-29T10:12:45.836427Z","shell.execute_reply.started":"2022-08-29T10:12:45.675640Z","shell.execute_reply":"2022-08-29T10:12:45.835081Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We can notice that program_number column has some types issues and we need to fix this.","metadata":{}},{"cell_type":"code","source":"def program_number_fixer(x):\n    if type(x) == int:\n        return str(x) #Converts none string to string\n    elif type(x) == str:\n        return x.replace(' ','') #Remove useless space in the string\n\nnyra_2019['program_number'] = nyra_2019['program_number'].apply(lambda x: program_number_fixer(x))\nnyra_tracking['program_number'] = nyra_tracking['program_number'].apply(lambda x: program_number_fixer(x))\nnyra_2019.program_number.unique()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:45.837977Z","iopub.execute_input":"2022-08-29T10:12:45.839406Z","iopub.status.idle":"2022-08-29T10:12:53.906417Z","shell.execute_reply.started":"2022-08-29T10:12:45.839362Z","shell.execute_reply":"2022-08-29T10:12:53.904869Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Let's first sort our dataset to make it more readable\nsorting_order = [\"race_date\",\"track_id\",\"race_number\",\"program_number\",\"trakus_index\"]\n\nnyra_2019.sort_values(by=sorting_order, inplace=True)\nnyra_tracking.sort_values(by=sorting_order, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:12:53.908712Z","iopub.execute_input":"2022-08-29T10:12:53.909091Z","iopub.status.idle":"2022-08-29T10:13:01.898135Z","shell.execute_reply.started":"2022-08-29T10:12:53.909060Z","shell.execute_reply":"2022-08-29T10:13:01.896738Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019.head()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:01.899615Z","iopub.execute_input":"2022-08-29T10:13:01.899986Z","iopub.status.idle":"2022-08-29T10:13:01.919469Z","shell.execute_reply.started":"2022-08-29T10:13:01.899955Z","shell.execute_reply":"2022-08-29T10:13:01.918479Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"It seems better now !","metadata":{}},{"cell_type":"markdown","source":"#### Time","metadata":{}},{"cell_type":"code","source":"#Create a time column based on the trakus_index\nmeasure_timing = 0.25 #About 0.25 second according to the competition description.\nnyra_2019['timer'] = nyra_2019['trakus_index'] * measure_timing\nnyra_tracking['timer'] = nyra_tracking['trakus_index'] * measure_timing","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:01.920805Z","iopub.execute_input":"2022-08-29T10:13:01.921701Z","iopub.status.idle":"2022-08-29T10:13:01.967390Z","shell.execute_reply.started":"2022-08-29T10:13:01.921658Z","shell.execute_reply":"2022-08-29T10:13:01.966128Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Speed","metadata":{}},{"cell_type":"code","source":"#Altitudes of all tracks obtained from google earth\nALT_AQU = 3\nALT_BEL = 20\nALT_SAR = 93\n\ndef getAltitude(x):\n    if x == 'AQU':\n        return ALT_AQU\n    if x == 'BEL':\n        return ALT_BEL\n    if x == 'SAR':\n        return ALT_SAR\n\n#Create an altitude column\nnyra_2019['altitude'] = nyra_2019['track_id'].apply(lambda x: getAltitude(x))\nnyra_tracking['altitude'] = nyra_tracking['track_id'].apply(lambda x: getAltitude(x))\n\n#compute speed (in kph) function\ndef compute_speed(lat, lon, alt):\n    ecef = np.array(pm.geodetic2ecef(lat, lon, alt)).T\n    speed = np.sqrt(np.sum(np.diff(ecef,axis=0) ** 2, axis = 1))*3.6/measure_timing\n    return np.insert(speed, 0, 0)\n\n#Create a speed column based on horse position at a t moment.\nnyra_2019['speed_kph'] = compute_speed(nyra_2019['latitude'], nyra_2019['longitude'], nyra_2019['altitude'])\nnyra_2019['speed_kph'].fillna(0.0, inplace=True)\nnyra_2019.loc[nyra_2019['trakus_index'] == 1, 'speed_kph'] = 0.0 #Rectifying values that might be wrong because of the shift -> all participant start at 0.0 km/h\n\nnyra_tracking['speed_kph'] = compute_speed(nyra_tracking['latitude'], nyra_tracking['longitude'], nyra_tracking['altitude'])\nnyra_tracking['speed_kph'].fillna(0.0, inplace=True)\nnyra_tracking.loc[nyra_tracking['trakus_index'] == 1, 'speed_kph'] = 0.0 #Rectifying values that might be wrong because of the shift -> all participant start at 0.0 km/h","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:01.968908Z","iopub.execute_input":"2022-08-29T10:13:01.969869Z","iopub.status.idle":"2022-08-29T10:13:09.326251Z","shell.execute_reply.started":"2022-08-29T10:13:01.969832Z","shell.execute_reply":"2022-08-29T10:13:09.324928Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019.reset_index(drop=True, inplace=True)\nnyra_tracking.reset_index(drop=True, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:09.327856Z","iopub.execute_input":"2022-08-29T10:13:09.328265Z","iopub.status.idle":"2022-08-29T10:13:09.333946Z","shell.execute_reply.started":"2022-08-29T10:13:09.328228Z","shell.execute_reply":"2022-08-29T10:13:09.332655Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"sns.boxplot(nyra_2019['speed_kph'])","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:09.335475Z","iopub.execute_input":"2022-08-29T10:13:09.336094Z","iopub.status.idle":"2022-08-29T10:13:10.959536Z","shell.execute_reply.started":"2022-08-29T10:13:09.336027Z","shell.execute_reply":"2022-08-29T10:13:10.958083Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Here, we can see that the dataset is polluted by some outliers. We assume that all values exceeding 90 km/h can be considered as errors.","metadata":{}},{"cell_type":"code","source":"print('Percentage of speed measures error: ',nyra_2019[nyra_2019['speed_kph'] > 90]['speed_kph'].count()/len(nyra_2019)*100,'%')","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:10.961054Z","iopub.execute_input":"2022-08-29T10:13:10.961451Z","iopub.status.idle":"2022-08-29T10:13:11.480828Z","shell.execute_reply.started":"2022-08-29T10:13:10.961416Z","shell.execute_reply":"2022-08-29T10:13:11.479430Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is a very low error rate, however we can still fix some errors.","metadata":{}},{"cell_type":"code","source":"def get_race_data(data, track, date, race, program_number=None):\n    if program_number:\n        return data[(data['track_id'] == track) & (data['race_date'] == date) & (data['race_number'] == race) & (data['program_number'] == program_number)]\n    else:\n        return data[(data['track_id'] == track) & (data['race_date'] == date) & (data['race_number'] == race)]","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:11.482554Z","iopub.execute_input":"2022-08-29T10:13:11.482970Z","iopub.status.idle":"2022-08-29T10:13:11.490200Z","shell.execute_reply.started":"2022-08-29T10:13:11.482935Z","shell.execute_reply":"2022-08-29T10:13:11.488962Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Speed measures fixing function\ndef speed_measure_fixing(data):\n    ERROR_THRESHOLD = 90\n    for i in tqdm(range(len(data['speed_kph'])), desc='Speed measures fixing'):\n        if data.loc[i, 'speed_kph'] > ERROR_THRESHOLD:\n            j = i+1\n            while data.loc[j, 'speed_kph'] > 90: #Would mean that the next value is also an error\n                j+=1\n            fixed_speed = data.loc[j, 'speed_kph']\n            fixed_long = data.loc[j, 'longitude']\n            fixed_lat = data.loc[j, 'latitude']\n            while j > i:\n                fixed_speed = (fixed_speed+data.loc[i-1, 'speed_kph'])/2\n                fixed_long = (fixed_long+data.loc[i-1, 'longitude'])/2\n                fixed_lat = (fixed_lat+data.loc[i-1, 'latitude'])/2\n                j-=1\n            data.loc[i, 'speed_kph'] = fixed_speed\n            data.loc[i, 'longitude'] = fixed_long\n            data.loc[i, 'latitude'] = fixed_lat\n    return data\n\nnyra_2019 = speed_measure_fixing(nyra_2019)\nnyra_tracking = speed_measure_fixing(nyra_tracking)\n\nprint('Percentage of speed measures error: ',nyra_2019[nyra_2019['speed_kph'] > 90]['speed_kph'].count()/len(nyra_2019)*100,'%')","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:13:11.491800Z","iopub.execute_input":"2022-08-29T10:13:11.492249Z","iopub.status.idle":"2022-08-29T10:21:42.386287Z","shell.execute_reply.started":"2022-08-29T10:13:11.492212Z","shell.execute_reply":"2022-08-29T10:21:42.384685Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Apparently, all the measurement errors has been fixed !","metadata":{}},{"cell_type":"code","source":"sns.boxplot(nyra_2019['speed_kph'])","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:21:42.388253Z","iopub.execute_input":"2022-08-29T10:21:42.388830Z","iopub.status.idle":"2022-08-29T10:21:43.918616Z","shell.execute_reply.started":"2022-08-29T10:21:42.388782Z","shell.execute_reply":"2022-08-29T10:21:43.917252Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"MPH_TO_KPH_FACTOR = 0.62137119 #Conversion factor miles to kilometers.\n\nnyra_2019['speed_mph'] = nyra_2019['speed_kph']*MPH_TO_KPH_FACTOR #Convert kph to mph\nnyra_tracking['speed_mph'] = nyra_tracking['speed_kph']*MPH_TO_KPH_FACTOR #Convert kph to mph","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:21:43.920191Z","iopub.execute_input":"2022-08-29T10:21:43.920603Z","iopub.status.idle":"2022-08-29T10:21:43.974022Z","shell.execute_reply.started":"2022-08-29T10:21:43.920570Z","shell.execute_reply":"2022-08-29T10:21:43.972569Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Acceleration","metadata":{}},{"cell_type":"code","source":"#Create an acceleration column\nnyra_2019['acc_mps'] = np.append(np.diff((nyra_2019['speed_kph']/3.6)/measure_timing, axis=0),0)\nnyra_tracking['acc_mps'] = np.append(np.diff((nyra_tracking['speed_kph']/3.6)/measure_timing, axis=0),0)\n\n#Fixing acc for last trakus_index\ndef fixing_acc(data):\n    for i in tqdm(range(len(data)-1)):\n        if data.loc[i+1,'trakus_index'] < data.loc[i,'trakus_index']:\n            data.loc[i,'acc_mps'] = 0.0\n    return data\n\nnyra_2019 = fixing_acc(nyra_2019)\nnyra_tracking = fixing_acc(nyra_tracking)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:21:43.975327Z","iopub.execute_input":"2022-08-29T10:21:43.975698Z","iopub.status.idle":"2022-08-29T10:50:50.708610Z","shell.execute_reply.started":"2022-08-29T10:21:43.975665Z","shell.execute_reply":"2022-08-29T10:50:50.707007Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Position\n\nThe given dataset only provides data on location of horses during the race. However, it might be possible to infer position values from longitude and latitude data by computing the distance between participants and the finish point.\n\nNevertheless, since the shape of the track is ellipsoidal, this step will be a bit trickier...\n\nA solution would be to define reference checkpoints on the track to compute distance properly.\n\nLet's proceed like that:\n- `Step 1` : Computing checkpoints every n position records (trakus_index) as the average of participant positions at a t moment.\n- `Step 2` : Assigning a checkpoint to each records.\n- `Step 3` : Computing positions for each tracking index based on the distance between participants and checkpoints.","metadata":{}},{"cell_type":"code","source":"def assign_checkpoint(chp_pos, last_chp, trakus_index):\n    CHP_DIST = 30\n    CHP_OFFSET = 10\n    NB_TRAKUS = len(chp_pos)\n\n    next_chp = (trakus_index//CHP_DIST+1)*CHP_DIST+CHP_OFFSET\n    if next_chp > NB_TRAKUS:\n        return [last_chp[0], last_chp[1]]\n    else:\n        return [chp_pos.loc[next_chp, 'longitude'], chp_pos.loc[next_chp, 'latitude']]\n\n\ndef last_checkpoint(data, chp_pos):\n    CHP_DIST = 30\n    last_trakus = data['trakus_index'].max()\n    last_points = data.loc[data['trakus_index'] == last_trakus]\n    ref_long = chp_pos['longitude'].tolist()[-CHP_DIST]\n    ref_lat = chp_pos['latitude'].tolist()[-CHP_DIST]\n    last_points['distance'] = np.sqrt(np.square(last_points['longitude']-ref_long)+np.square(last_points['latitude']-ref_lat))\n    last_points = last_points.sort_values(['distance'], ascending=False).reset_index()\n    return (last_points['longitude'][0], last_points['latitude'][0])\n","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:50:50.710579Z","iopub.execute_input":"2022-08-29T10:50:50.711012Z","iopub.status.idle":"2022-08-29T10:50:50.723647Z","shell.execute_reply.started":"2022-08-29T10:50:50.710973Z","shell.execute_reply":"2022-08-29T10:50:50.722523Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_grouped = nyra_2019.groupby(by=['track_id','race_date','race_number'])\n\nchp_long = []\nchp_lat = []\n\nfor index, group in tqdm(race_grouped, total=len(race_grouped), desc='Checkpoint generation'):\n    chp_pos = group[['trakus_index','longitude','latitude']].groupby(by='trakus_index').mean()\n    last_chp = last_checkpoint(group, chp_pos)\n    chp_df = group.apply(lambda x: assign_checkpoint(chp_pos, last_chp, x['trakus_index']), axis=1, result_type='expand')\n    chp_long += chp_df[0].tolist()\n    chp_lat += chp_df[1].tolist()\n\nnyra_2019['checkpoint_long'] = chp_long\nnyra_2019['checkpoint_lat'] = chp_lat","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:50:50.725752Z","iopub.execute_input":"2022-08-29T10:50:50.726301Z","iopub.status.idle":"2022-08-29T10:57:17.784056Z","shell.execute_reply.started":"2022-08-29T10:50:50.726257Z","shell.execute_reply":"2022-08-29T10:57:17.782827Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_2019['checkpoint_distance'] = np.sqrt(np.square(nyra_2019['checkpoint_long']-nyra_2019['longitude'])+ np.square(nyra_2019['checkpoint_lat']-nyra_2019['latitude']))\nnyra_2019['race_position'] = nyra_2019.groupby(by=['track_id','race_date','race_number','trakus_index'])['checkpoint_distance'].rank(ascending=True).astype(int)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:57:17.789481Z","iopub.execute_input":"2022-08-29T10:57:17.789856Z","iopub.status.idle":"2022-08-29T10:57:21.249781Z","shell.execute_reply.started":"2022-08-29T10:57:17.789824Z","shell.execute_reply":"2022-08-29T10:57:21.248722Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def plot_race(track, date, race):\n    data = get_race_data(nyra_2019, track, date, race)\n\n    #Plot participant positions during the race\n    plt.figure(figsize=(18,data['race_position'].nunique()))\n    sns.lineplot(data=data, x=\"timer\", y=\"race_position\", hue=\"jockey\", palette='tab10', linewidth=3)\n    plt.xlabel(\"Timer (in second)\")\n    plt.ylabel(\"Position\")\n    plt.yticks(data['race_position'].unique())\n    plt.gca().invert_yaxis()\n    plt.legend(title='Jockey\\n', bbox_to_anchor=(1, 1))\n    plt.title(f\"Track: {track}, Date: {date}, Race n°{race}\")\n    plt.show()\n\n    #Race visualisation animtion\n    long_offset = (data['longitude'].max()-data['longitude'].min())*0.15\n    lat_offset = (data['latitude'].max()-data['latitude'].min())*0.15\n\n    plt.figure(figsize=(16,8))\n    fig = px.scatter(data, animation_frame='timer', x=\"longitude\", y=\"latitude\", hover_data = [\"jockey\"],\n                    range_x=(data['longitude'].min()-long_offset, data['longitude'].max()+long_offset),\n                    range_y=(data['latitude'].min()-lat_offset, data['latitude'].max()+lat_offset),\n                    title='Race Visualisation',\n                    color = \"jockey\",\n                    width=1200,\n                    height=600,\n                    template='seaborn',\n    )\n\n    fig.update_traces(marker = dict(symbol = \"circle\", size = 16))\n    fig.layout.updatemenus[0].buttons[0].args[1]['frame']['duration'] = 30\n    fig.layout.updatemenus[0].buttons[0].args[1]['transition']['duration'] = 5\n\n    fig.show()","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:57:21.252686Z","iopub.execute_input":"2022-08-29T10:57:21.254479Z","iopub.status.idle":"2022-08-29T10:57:21.267600Z","shell.execute_reply.started":"2022-08-29T10:57:21.254437Z","shell.execute_reply":"2022-08-29T10:57:21.266123Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"track_wid = ipywidgets.ToggleButtons(\n    options=nyra_tracking['track_id'].unique(),\n    description='Track (ID):',\n    disabled=False,\n    button_style=''\n)\n\nrace_wid = ipywidgets.IntSlider(\n    value=1,\n    min=1,\n    max=nyra_tracking[nyra_tracking['track_id'] == 'AQU'][\"race_number\"].max(),\n    step=1,\n    description='Race number:',\n    disabled=False\n)\n\ndate_wid = ipywidgets.Text(\n    description='Date:',\n    placeholder='YYYY-MM-DD',\n    disabled=False\n)\n\ndef update_race_range(*args):\n    race_wid.max = nyra_tracking[nyra_tracking['track_id'] == track_wid.value][\"race_number\"].max()\n\ntrack_wid.observe(update_race_range, 'value')","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:57:21.269542Z","iopub.execute_input":"2022-08-29T10:57:21.269903Z","iopub.status.idle":"2022-08-29T10:57:22.884165Z","shell.execute_reply.started":"2022-08-29T10:57:21.269871Z","shell.execute_reply":"2022-08-29T10:57:22.882700Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"all_races = nyra_2019.groupby(by=['track_id','race_date','race_number']).groups.keys()\nrace_visualiser = ipywidgets.interact_manual(plot_race ,track=track_wid, date=date_wid, race=race_wid)\nrace_visualiser","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:57:22.885687Z","iopub.execute_input":"2022-08-29T10:57:22.886022Z","iopub.status.idle":"2022-08-29T10:57:34.594881Z","shell.execute_reply.started":"2022-08-29T10:57:22.885994Z","shell.execute_reply":"2022-08-29T10:57:34.593721Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"***\n## Data Interpretation\n***","metadata":{}},{"cell_type":"code","source":"data = get_race_data(nyra_2019, 'AQU', '2019-01-01', 1)\ndata = data[data['timer']%1 == 0].pivot('jockey', 'timer','acc_mps')\n\nsns.heatmap(data)","metadata":{"execution":{"iopub.status.busy":"2022-08-29T10:57:34.596273Z","iopub.execute_input":"2022-08-29T10:57:34.596617Z","iopub.status.idle":"2022-08-29T10:57:36.750000Z","shell.execute_reply.started":"2022-08-29T10:57:34.596586Z","shell.execute_reply":"2022-08-29T10:57:36.749032Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"In progress...","metadata":{}}]}