{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"## Big Data Derby 2022\nTeam: Joanna Kajka, Laura Hoang, Piotr Bielecki, Jakub Grunas","metadata":{}},{"cell_type":"markdown","source":"## Introduction & Summary\n\nThe notebook contains the analysis of horse tracking data. The idea of the project was to exctract several features from horses' coordinates after every 0.25 second in order to then fit a model to the data and finally generate feature importance in order to potentially identify patterns in strategies of winning horses.\n\nThe notebook contains only the code necessary to generate results (with some basic descriptions and conclusions), without multiple secondary analyses.\nApart from feature engineering and modelling, we decided to cluster together similar races - the algorithm turned out to be succesful and is described later in the notebook.","metadata":{}},{"cell_type":"markdown","source":"## Importing packages and data","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport numpy as np\nimport seaborn as sns\nfrom matplotlib import pyplot as plt\nfrom matplotlib import image as mpimg\nimport datetime\nimport plotly.express as px\nimport statistics","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","execution":{"iopub.status.busy":"2022-11-10T21:16:40.290410Z","iopub.execute_input":"2022-11-10T21:16:40.291066Z","iopub.status.idle":"2022-11-10T21:16:40.299937Z","shell.execute_reply.started":"2022-11-10T21:16:40.291021Z","shell.execute_reply":"2022-11-10T21:16:40.298177Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"nyra_start_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_start_table.csv', \n                               names=['track_id', 'race_date', 'race_number', 'program_number', 'weight_carried', 'jockey', 'odds', 'position_at_finish'])\nnyra_race_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_race_table.csv')\nnyra_tracking_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_tracking_table.csv')\ndf = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv',\n                 names=['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index', 'latitude', 'longitude', 'distance_id', 'course_type', \n                        'track_condition', 'run_up_distance', 'race_type', 'purse', 'post_time', 'weight_carried', 'jockey', 'odds', 'position_at_finish'])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:40.308141Z","iopub.execute_input":"2022-11-10T21:16:40.309903Z","iopub.status.idle":"2022-11-10T21:16:58.879570Z","shell.execute_reply.started":"2022-11-10T21:16:40.309848Z","shell.execute_reply":"2022-11-10T21:16:58.877751Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Data Preprocessing","metadata":{}},{"cell_type":"markdown","source":"### Column formatting: `race_date` and `program_number`","metadata":{}},{"cell_type":"code","source":"df['race_date'] = pd.to_datetime(df['race_date'], format='%Y-%m-%d')\nnyra_tracking_table['race_date'] = pd.to_datetime(nyra_tracking_table['race_date'], format='%Y-%m-%d')\n\ndf['program_number'] = df['program_number'].map(lambda x : 91 if x == '1A ' else (92 if x == '1X ' else (93 if x == '2B ' else (94 if x == '3X ' else int(x)))))\nnyra_tracking_table['program_number'] = nyra_tracking_table['program_number'].map(lambda x : 91 if x == '1A ' else (92 if x == '1X ' else (93 if x == '2B ' else (94 if x == '3X ' else int(x)))))\nnyra_start_table['program_number'] = nyra_start_table['program_number'].map(lambda x : 91 if x == '1A ' else (92 if x == '1X ' else (93 if x == '2B ' else (94 if x == '3X ' else int(x)))))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.882062Z","iopub.execute_input":"2022-11-10T21:16:58.882873Z","iopub.status.idle":"2022-11-10T21:16:58.890894Z","shell.execute_reply.started":"2022-11-10T21:16:58.882810Z","shell.execute_reply":"2022-11-10T21:16:58.888644Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `Angle` column\nAngle calculated between horse's second to last position, last position and current position.","metadata":{}},{"cell_type":"code","source":"df.sort_values(['race_date', 'track_id', 'race_number', 'course_type', 'program_number', 'trakus_index'], inplace=True)\n\ndf['prev_lat'] = df.groupby(['track_id', 'race_date', 'race_number', 'course_type', 'program_number'])['latitude'].shift(1)\ndf['prev_lon'] = df.groupby(['track_id', 'race_date', 'race_number', 'course_type', 'program_number'])['longitude'].shift(1)\n\ndf['prev_prev_lat'] = df.groupby(['track_id', 'race_date', 'race_number', 'course_type', 'program_number'])['latitude'].shift(2)\ndf['prev_prev_lon'] = df.groupby(['track_id', 'race_date', 'race_number', 'course_type', 'program_number'])['longitude'].shift(2)\ndf[['track_id', 'race_date', 'race_number', 'program_number', 'trakus_index', 'latitude', 'longitude', 'prev_lat', 'prev_lon', 'prev_prev_lat', 'prev_prev_lon']].head()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.892897Z","iopub.execute_input":"2022-11-10T21:16:58.893435Z","iopub.status.idle":"2022-11-10T21:16:58.912061Z","shell.execute_reply.started":"2022-11-10T21:16:58.893395Z","shell.execute_reply":"2022-11-10T21:16:58.909860Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"from cmath import pi\n\ndef unit_vector(v):\n    \"\"\" Returns a unit vector of v. \"\"\"\n    return v / np.linalg.norm(v)\n\ndef angle(x1, y1, x2, y2, x3, y3):\n    \"\"\" Returns an angle (in degrees) between points P1, P2 and P3, where Pi=(xi,yi). \"\"\"\n    v1 = (x2-x1, y2-y1)\n    v2 = (x3-x2, y3-y2)\n    \n    u1 = unit_vector(v1)\n    u2 = unit_vector(v2)\n    rad = np.arccos(np.clip( np.dot(u1,u2), -1.0, 1.0 ))  # angle in radians\n    return rad*180/pi\n\ndf['angle'] = df.apply(\n    lambda df: angle(df['prev_prev_lon'], df['prev_prev_lat'], df['prev_lon'], df['prev_lat'], df['longitude'], df['latitude']), axis=1)\ndf['angle'] = df['angle'].fillna(0)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.916428Z","iopub.execute_input":"2022-11-10T21:16:58.917037Z","iopub.status.idle":"2022-11-10T21:16:58.926928Z","shell.execute_reply.started":"2022-11-10T21:16:58.916982Z","shell.execute_reply":"2022-11-10T21:16:58.925253Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Polar coordinates: `phi` column\nAngle phi calculated between two vectors: horse's position - track's center and vector parallel to the equator that crosses the center of the track. This feature gives us position of a horse on a track regardless of race's starting line.","metadata":{}},{"cell_type":"code","source":"AQU = df[df.track_id=='AQU']\nBEL = df[df.track_id=='BEL']\nSAR = df[df.track_id=='SAR']\npoints_aqu = AQU[(AQU.race_date=='2019-11-09')&(AQU.race_number==8)][['latitude','longitude']]\npoints_bel = BEL[(BEL.race_date=='2019-09-19')&(BEL.race_number==1)][['latitude','longitude']]\npoints_sar = SAR[(SAR.race_date=='2019-07-27')&(SAR.race_number==5)][['latitude','longitude']]\npts_aqu = np.column_stack((points_aqu.latitude.to_list(), points_aqu.longitude.to_list()))\npts_bel = np.column_stack((points_bel.latitude.to_list(), points_bel.longitude.to_list()))\npts_sar = np.column_stack((points_sar.latitude.to_list(), points_sar.longitude.to_list()))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.929003Z","iopub.execute_input":"2022-11-10T21:16:58.929591Z","iopub.status.idle":"2022-11-10T21:16:58.944304Z","shell.execute_reply.started":"2022-11-10T21:16:58.929536Z","shell.execute_reply":"2022-11-10T21:16:58.943227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Finding track centers","metadata":{}},{"cell_type":"code","source":"from warnings import warn\n\nfrom numpy import pi\nfrom numpy import sqrt\nfrom numpy import arccos\nfrom numpy import dot, outer\nfrom numpy import diag, transpose\nfrom numpy import append\nfrom numpy import asarray\nfrom numpy import ones\nfrom numpy import argmax\n\nfrom numpy.linalg import inv\nfrom numpy.linalg import norm\nfrom numpy.linalg import eig\n\n\ndef mvee( data, tolerance=1e-4, maxcnt=1000 ):\n    \"\"\"\n    param data: list of xy data points\n    param tolerance: termination condition for iterative approximation\n    param maxcnt: maximum number of iterations\n    type data: iterable of float\n    type tolerance: float\n    return: (offset, semiaxis, angle)\n    return type: ( (float, float), (float, float), float )\n    \"\"\"\n    locdata = asarray( data )\n    N = len( locdata )\n    if not locdata.shape == ( N, 2):\n        raise ValueError ( \" data must be of shape( n, 2 )\" )\n    if tolerance >= 1 or tolerance <= 0:\n        raise ValueError (\" 0 < tolerance < 1 required\")\n    if not isinstance( maxcnt, int ):\n        raise TypeError\n    if not maxcnt > 0:\n        raise ValueError\n    count = 1\n    err = 1\n    d = 2\n    d1 = d + 1\n    u = ones( N ) / N\n    P = transpose( locdata )\n    Q = append( P, ones( N ) ).reshape( 3, -1 )\n    while ( err > tolerance):\n        X = dot( Q, dot( diag( u ), transpose( Q ) ) )\n        M = diag( \n            dot( \n                transpose( Q ),\n                dot(\n                    inv( X ),\n                    Q\n                )\n            )\n        )\n        maximum = max( M )\n        j = argmax( M )\n        step_size = ( maximum - d1 ) / ( d1 * ( maximum - 1 ) )\n        new_u = ( 1 - step_size ) * u\n        new_u[ j ] += step_size\n        err = norm( new_u - u )\n        count = count + 1\n        u = new_u\n        if count > maxcnt:\n            warn(\n                \"Process did not converge in {} steps\".format(\n                    count - 1\n                ),\n                UserWarning\n            )\n            break\n    c = dot( P,  u )\n    return c","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.945807Z","iopub.execute_input":"2022-11-10T21:16:58.947516Z","iopub.status.idle":"2022-11-10T21:16:58.961355Z","shell.execute_reply.started":"2022-11-10T21:16:58.947430Z","shell.execute_reply":"2022-11-10T21:16:58.959844Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"center_aqu = mvee(pts_aqu)\ncenter_bel = mvee(pts_bel)\ncenter_sar = mvee(pts_sar)\n\ndef lat(row):\n    if row['track_id'] == 'AQU':\n        val = row['latitude'] - center_aqu[0]\n    elif row['track_id'] == 'BEL':\n        val = row['latitude'] - center_bel[0]\n    else:\n        val = row['latitude'] - center_sar[0]\n    return val\ndef long(row):\n    if row['track_id'] == 'AQU':\n        val = row['longitude'] - center_aqu[1]\n    elif row['track_id'] == 'BEL':\n        val = row['longitude'] - center_bel[1]\n    else:\n        val = row['longitude'] - center_sar[1]\n    return val\n\ndf['long'] = df.apply(long, axis=1)\ndf['lat'] = df.apply(lat, axis=1)\n\ndf = df.assign(phi = lambda x: np.arctan2(x.lat, x.long))\n\ndf.drop(columns=['lat', 'long'], inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.963585Z","iopub.execute_input":"2022-11-10T21:16:58.964692Z","iopub.status.idle":"2022-11-10T21:16:58.982688Z","shell.execute_reply.started":"2022-11-10T21:16:58.964620Z","shell.execute_reply":"2022-11-10T21:16:58.980971Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `Distance` and `speed` columns","metadata":{}},{"cell_type":"code","source":"import geopy.distance\n\ndef distance_covered(coords1, coords2):\n    if math.isnan(coords1[0]): return 0\n    return geopy.distance.geodesic((coords1), (coords2)).m\ndf['dst_covered'] = df.apply(lambda x: distance_covered((x['prev_lat'], x['prev_lon']), (x['latitude'], x['longitude'])), axis=1)\n\ndf['avg_speed'] = df['dst_covered'] / 0.25\ndf['max_race_speed'] = df.groupby(['track_id','race_date','race_number','course_type'])['avg_speed'].transform('max')\ndf['avg_rel_speed'] = df['avg_speed']/df['max_race_speed']\n\ndf['total_race_dist'] = df['run_up_distance']*0.3048 + df['distance_id']*2.01168\ndf['dst_covered_perc'] = df['dst_covered']/df['total_race_dist']\ndf['dst_covered_perc_cum'] = df.groupby(['track_id','race_date','race_number','course_type','program_number'])['dst_covered_perc'].cumsum()\ndf['dst_covered_bucket'] = pd.cut(df['dst_covered_perc_cum'],\n                                  [-0.1,0.05,0.1,0.15,0.2,0.25,0.3,0.35,0.4,0.45,0.5,0.55,0.6,0.65,0.7,0.75,0.8,0.85,0.9,0.95,10],\n                                  labels=range(20))","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:58.985358Z","iopub.execute_input":"2022-11-10T21:16:58.987333Z","iopub.status.idle":"2022-11-10T21:16:59.003081Z","shell.execute_reply.started":"2022-11-10T21:16:58.987260Z","shell.execute_reply":"2022-11-10T21:16:59.001529Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `Pack_speed` column","metadata":{}},{"cell_type":"code","source":"tmp = pd.DataFrame({'pack_speed' : df.groupby( [ \"track_id\", \"race_date\", \"race_number\", \"trakus_index\"]).mean()[\"avg_speed\"]}).reset_index()\ndf = df.merge(tmp, \"left\")\ndf[\"relative_speed\"] = df[\"avg_speed\"]-df[\"pack_speed\"]\ndf.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.005082Z","iopub.execute_input":"2022-11-10T21:16:59.005984Z","iopub.status.idle":"2022-11-10T21:16:59.019865Z","shell.execute_reply.started":"2022-11-10T21:16:59.005930Z","shell.execute_reply":"2022-11-10T21:16:59.017676Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"This is how the resulting table looks like:","metadata":{}},{"cell_type":"code","source":"big_data_derby_2022_11_07 = pd.read_csv('../input/datasets-heads/big_data_derby_2022_11_07_head.csv', index_col=0)\nbig_data_derby_2022_11_07","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.027287Z","iopub.execute_input":"2022-11-10T21:16:59.027752Z","iopub.status.idle":"2022-11-10T21:16:59.130217Z","shell.execute_reply.started":"2022-11-10T21:16:59.027719Z","shell.execute_reply":"2022-11-10T21:16:59.128825Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### `Section` column\n`Section` column was calculated based on phi angle for each moment and grouped together observations from 6 parts of the track (beginning/middle/end of curve/straight).","metadata":{}},{"cell_type":"markdown","source":"### Removing ties in `position_at_finish`","metadata":{}},{"cell_type":"code","source":"tmp = df.groupby([\"track_id\", \"race_date\", \"race_number\", \"position_at_finish\"]).jockey.nunique().sort_values(ascending = False)\ntmp = tmp.to_frame()\ntmp = tmp[tmp[\"jockey\"]>1]\n\nfor i in tmp.index:\n    equal = df[(df[\"track_id\"] == i[0]) &\n            (df[\"race_date\"] == i[1]) &\n            (df[\"race_number\"] == i[2]) &\n            (df[\"position_at_finish\"] == i[3])]\n    jockeys = pd.unique(equal[\"jockey\"])\n    jockey = jockeys[0]\n    df.loc[(df[\"track_id\"] == i[0]) &\n        (df[\"race_date\"] == i[1]) &\n        (df[\"race_number\"] == i[2]) &\n        (df[\"jockey\"] == jockey), \"position_at_finish\"] = df.loc[(df[\"track_id\"] == i[0]) &\n                                                                (df[\"race_date\"] == i[1]) &\n                                                                (df[\"race_number\"] == i[2]) &\n                                                                (df[\"jockey\"] == jockey), \"position_at_finish\"]+1","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.131730Z","iopub.execute_input":"2022-11-10T21:16:59.132135Z","iopub.status.idle":"2022-11-10T21:16:59.139023Z","shell.execute_reply.started":"2022-11-10T21:16:59.132100Z","shell.execute_reply":"2022-11-10T21:16:59.137396Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of horses ahead: `horses_in_front` column","metadata":{}},{"cell_type":"code","source":"tmp = df.drop(columns = [i for i in df.columns if not i in ['track_id', 'race_date', 'race_number', 'jockey',\n       'trakus_index', 'latitude', 'longitude', 'position_at_finish'] ] )\ntmp2 = tmp.pivot(index = [\"track_id\", \"race_date\", \"race_number\",\"trakus_index\"], columns = [\"position_at_finish\"], values = [\"latitude\",\"longitude\"])\ntmp2.columns = tmp2.columns.get_level_values(0) +\"_\" + [str(i) for i in tmp2.columns.get_level_values(1).to_list()]\ntmp2\ndf = df.merge(tmp2, on = [\"track_id\", \"race_date\", \"race_number\", \"trakus_index\"])\n\nfrom cmath import pi\ndef angle_to_horse(prev_x1, prev_y1, x1, y1, x2, y2):\n    \"\"\"returns angle in degrees to another horse\"\"\"\n    vel = (x1-prev_x1, y1-prev_y1)\n    radial = (x2-x1, y2-y1)\n    vel = vel / np.linalg.norm(vel, axis = 0)\n    radial = radial / np.linalg.norm(radial, axis = 0)\n    rad = np.arccos(np.array(vel[0])*np.array(radial[0])+np.array(vel[1])*np.array(radial[1]))\n    angle = rad*180/pi\n    return angle","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.140935Z","iopub.execute_input":"2022-11-10T21:16:59.141401Z","iopub.status.idle":"2022-11-10T21:16:59.154397Z","shell.execute_reply.started":"2022-11-10T21:16:59.141351Z","shell.execute_reply":"2022-11-10T21:16:59.152945Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We calculate the number of horses ahead as the number of horses whose distance is lower than 5m, which have a position vector at an angle <10 degrees to the velocity vector of our horse. This calculation is then used to get an approximation for energy saved as a percentage of all energy otherwise spent by a horse in last 1/4 second.","metadata":{}},{"cell_type":"code","source":"horses_in_front = np.zeros((len(df.loc[:,[\"trakus_index\"]]),1))\nfor i in range(1,14):\n    distance = \"distance_\" + str(i)\n    angle = \"angle_\"+str(i)\n    distance = np.array(df.loc[:,[distance]])\n    angle = np.array(df.loc[:,[angle]])\n    distance = distance < 5\n    angle = angle < 10\n    horse_in_front = distance * angle\n    horses_in_front = horses_in_front + horse_in_front\n\ndf[\"horses_in_front\"] = np.int8(horses_in_front)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.155907Z","iopub.execute_input":"2022-11-10T21:16:59.156346Z","iopub.status.idle":"2022-11-10T21:16:59.175380Z","shell.execute_reply.started":"2022-11-10T21:16:59.156310Z","shell.execute_reply":"2022-11-10T21:16:59.173687Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Energy saving\n\nAccording to: https://www.horsejournals.com/riding-training/english/other/slipstreaming-reduces-drag-horse-racing-66-percent#:~:text=Also%20known%20as%20drafting%2C%20slipstreaming,dash%20to%20the%20finish%20line.)\n\nWhen two horses are in front of one horse, the drag of the trailing horse is reduced by 66 percent;\nWhen two horses are running with one closely behind the other, the drag of the lead horse is reduced by 6.5 percent while the drag of the trailing horse is reduced by 38.5 percent.\nIn a situation where five horses are side by side, the drag of the centre horse actually increases by 25 percent.\n\nAccording to: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3391435/\n\nAt 18 m s−1 (cf. figure 1a), this results in a drag force of 209 N,\nrequiring a mechanical power of 3.8 kW. The total mechanical\npower of horse galloping has been estimated as 22.5 kW [10],\nand therefore a rough estimate based on these figures is that overcoming drag\naccounts for 17 per cent of total mechanical power.\n","metadata":{}},{"cell_type":"code","source":"def energy_saved(ahead):\n    \"\"\"return energy saved as % of all energy otherwise spent on the race\"\"\"\n    draft_percent = 0.385*(ahead == 1) + 0.66*(ahead == 2) + 0.75*(ahead == 3) + 0.8*(ahead == 4)\n    return(17/100 * draft_percent)\n\ndf[\"energy_saved\"] = df.apply(lambda x: energy_saved(x.horses_in_front), axis = 1)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.177174Z","iopub.execute_input":"2022-11-10T21:16:59.178198Z","iopub.status.idle":"2022-11-10T21:16:59.191234Z","shell.execute_reply.started":"2022-11-10T21:16:59.178135Z","shell.execute_reply":"2022-11-10T21:16:59.189823Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Number of competitors: `competitors_in_race` column","metadata":{}},{"cell_type":"code","source":"tmp = nyra_start_table.groupby([\"track_id\", \"race_date\", \"race_number\"]).jockey.nunique()\ntmp = tmp.to_frame()\ntmp.rename(columns = {\"jockey\":\"competitors_in_race\"}, inplace = True)\ndf = df.merge(tmp, \"inner\", on=[\"track_id\",\"race_date\",\"race_number\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.193382Z","iopub.execute_input":"2022-11-10T21:16:59.194248Z","iopub.status.idle":"2022-11-10T21:16:59.206513Z","shell.execute_reply.started":"2022-11-10T21:16:59.194197Z","shell.execute_reply":"2022-11-10T21:16:59.205298Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Calculating `position_in_race`","metadata":{}},{"cell_type":"markdown","source":"#### Number of competitors in the race","metadata":{}},{"cell_type":"code","source":"tmp = nyra_start_table.groupby([\"track_id\", \"race_date\", \"race_number\"]).jockey.nunique()\ntmp = tmp.to_frame()\ntmp.rename(columns = {\"jockey\":\"competitors_in_race\"}, inplace = True)\ndf = df.merge(tmp, \"inner\", on=[\"track_id\",\"race_date\",\"race_number\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.208207Z","iopub.execute_input":"2022-11-10T21:16:59.208698Z","iopub.status.idle":"2022-11-10T21:16:59.229964Z","shell.execute_reply.started":"2022-11-10T21:16:59.208643Z","shell.execute_reply":"2022-11-10T21:16:59.228102Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Number of horses behind","metadata":{}},{"cell_type":"code","source":"horses_behind = np.zeros((len(df2.loc[:,[\"trakus_index\"]]),1))\n\nfor i in range(1,14):\n    distance = \"distance_\" + str(i)\n    angle = \"angle_\"+str(i)\n    distance = np.array(df2.loc[:,[distance]])\n    angle = np.array(df2.loc[:,[angle]])\n    horse_behind = angle > 90\n    horses_behind = horses_behind + horse_behind\n\ndf2[\"horses_behind\"] = np.int8(horses_behind)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.232235Z","iopub.execute_input":"2022-11-10T21:16:59.232759Z","iopub.status.idle":"2022-11-10T21:16:59.245931Z","shell.execute_reply.started":"2022-11-10T21:16:59.232721Z","shell.execute_reply":"2022-11-10T21:16:59.244560Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Finally, we get the `position_in_race` column.","metadata":{}},{"cell_type":"code","source":"df2[\"position_in_race\"] = df2.competitors_in_race - df2.horses_behind","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.248535Z","iopub.execute_input":"2022-11-10T21:16:59.249015Z","iopub.status.idle":"2022-11-10T21:16:59.262375Z","shell.execute_reply.started":"2022-11-10T21:16:59.248977Z","shell.execute_reply":"2022-11-10T21:16:59.260579Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"`avg_speed_to_pack_speed`","metadata":{}},{"cell_type":"code","source":"result['avg_speed_to_pack_speed'] = np.where(result.pack_speed==0, 0, result.avg_speed / result.pack_speed)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.263856Z","iopub.execute_input":"2022-11-10T21:16:59.264358Z","iopub.status.idle":"2022-11-10T21:16:59.277019Z","shell.execute_reply.started":"2022-11-10T21:16:59.264318Z","shell.execute_reply":"2022-11-10T21:16:59.275575Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Right before pivoting, our dataframe looks like this:","metadata":{}},{"cell_type":"code","source":"final_preprocessing_head = pd.read_csv('../input/datasets-heads/final_preprocessing_head.csv', index_col=0)\nfinal_preprocessing_head","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.278688Z","iopub.execute_input":"2022-11-10T21:16:59.279507Z","iopub.status.idle":"2022-11-10T21:16:59.330047Z","shell.execute_reply.started":"2022-11-10T21:16:59.279418Z","shell.execute_reply":"2022-11-10T21:16:59.328779Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After caring out an additional analysis, we decided to change the range of `dst_covered_bucket` values.","metadata":{}},{"cell_type":"code","source":"data['dst_covered_bucket'] = data['dst_covered_bucket'].map({0:0,1:0,2:1,3:1,4:2,5:2,6:3,7:3,8:4,9:4,10:5,11:5,12:6,13:6,14:7,15:7,16:8,17:8,18:9,19:9})","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.331117Z","iopub.execute_input":"2022-11-10T21:16:59.331526Z","iopub.status.idle":"2022-11-10T21:16:59.339935Z","shell.execute_reply.started":"2022-11-10T21:16:59.331449Z","shell.execute_reply":"2022-11-10T21:16:59.338729Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Aggregating the data\nAggregation algorithm was performed in order to get features for each decile for each horse for each race. After this process each row in the dataframe corresponds to a horse in a race and the columns represent generated earlier features for each decile of a race.","metadata":{}},{"cell_type":"code","source":"df = data.groupby(\n                ['track_id', 'race_date', 'race_number', 'program_number', 'dst_covered_bucket'], as_index=False).aggregate({\n                'distance_id' : 'max',\n                'position_at_finish' : 'max',\n                'avg_rel_speed': 'mean',\n                'avg_speed_to_pack_speed' : 'mean',\n                'section': ['mean', lambda x: str(list(x.unique())), 'min', 'max', 'nunique'], # 'section': 'mean',\n                'position_in_race': 'mean',\n                'angle': 'mean',\n                'dst_covered_perc' : 'sum',\n                'dst_covered_perc_cum' : 'max',\n                'energy_saved' : 'mean'\n                })\n\ndf.columns = ['track_id', 'race_date', 'race_number', 'program_number', 'dst_covered_bucket',\n                'distance_id',\n                'position_at_finish',\n                'avg_rel_speed_mean',\n                'avg_speed_to_pack_speed_mean',\n                'section_mean', 'section_unique', 'section_min', 'section_max', 'section_nunique',\n                'position_in_race_mean',\n                'angle_mean',\n                'dst_covered_perc_sum',\n                'dst_covered_perc_cum_max',\n                'energy_saved_mean']\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.341102Z","iopub.execute_input":"2022-11-10T21:16:59.341505Z","iopub.status.idle":"2022-11-10T21:16:59.356514Z","shell.execute_reply.started":"2022-11-10T21:16:59.341435Z","shell.execute_reply":"2022-11-10T21:16:59.354950Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df = df[df.dst_covered_bucket != 9]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.358999Z","iopub.execute_input":"2022-11-10T21:16:59.359524Z","iopub.status.idle":"2022-11-10T21:16:59.379905Z","shell.execute_reply.started":"2022-11-10T21:16:59.359484Z","shell.execute_reply":"2022-11-10T21:16:59.377958Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df['section'] = np.where((df.section_nunique==1) | ((df.section_nunique==2) & ((df.section_min!=1) | (df.section_max!=6))) , df.section_mean, None)\ndf['section'] = np.where((df.section_nunique==2) & (df.section_min==1) & (df.section_max==6), np.where(df.section_mean<3.5, 1, 6), df.section)\ndf['section'] = np.where(df.section_nunique==3, df.section_unique.str[4].astype(float), df.section)\ndf['section'] = np.where((df.section_nunique==4) & (df.section_unique != \"[5, 6, 1, 2]\"), \n                         (df.section_unique.str[4].astype(float) + df.section_unique.str[7].astype(float))/2, df.section)\ndf['section'] = np.where((df.section_unique==\"[5, 6, 1, 2]\"), np.where(df.section_mean<3.5, 1, 6), df.section)\ndf['section'] = np.where((df.section_nunique==5), df.section_unique.str[7].astype(float), df.section)\ndf['section'] = np.where((df.section_nunique==6) & (df.section_unique != \"[4, 5, 6, 1, 2, 3]\"), \n                         (df.section_unique.str[7].astype(float) + df.section_unique.str[10].astype(float))/2, df.section)\ndf['section'] = np.where((df.section_unique == \"[4, 5, 6, 1, 2, 3]\"), np.where(df.section_mean<3.5, 1, 6), df.section)\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.381218Z","iopub.execute_input":"2022-11-10T21:16:59.381658Z","iopub.status.idle":"2022-11-10T21:16:59.396284Z","shell.execute_reply.started":"2022-11-10T21:16:59.381622Z","shell.execute_reply":"2022-11-10T21:16:59.394582Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Columns to delete","metadata":{}},{"cell_type":"code","source":"columns_to_del = [col for col in df.columns if \"section_\" in col]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.399284Z","iopub.execute_input":"2022-11-10T21:16:59.400418Z","iopub.status.idle":"2022-11-10T21:16:59.418881Z","shell.execute_reply.started":"2022-11-10T21:16:59.400327Z","shell.execute_reply":"2022-11-10T21:16:59.416894Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"id_cols = ['track_id','race_date','race_number','program_number']\nother = ['dst_covered_bucket', 'distance_id', 'position_at_finish']\nfeatures = [col for col in df.columns if col not in id_cols+other]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.420851Z","iopub.execute_input":"2022-11-10T21:16:59.421261Z","iopub.status.idle":"2022-11-10T21:16:59.434071Z","shell.execute_reply.started":"2022-11-10T21:16:59.421229Z","shell.execute_reply":"2022-11-10T21:16:59.432736Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"df_final = df[df['dst_covered_bucket']==0].reset_index(drop=True)\ndf_final.columns = id_cols + other + [feature + '_0' for feature in features]\n\nfor i in range(1,9):\n    tmp = df[df['dst_covered_bucket']==i].reset_index(drop=True)\n    tmp.columns = id_cols + other + [feature + '_' + str(i) for feature in features]\n    tmp.drop(other, axis=1, inplace=True)\n    df_final = df_final.merge(tmp, on=id_cols)\n\ndf_final = df_final.drop('dst_covered_bucket', axis=1)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.435824Z","iopub.execute_input":"2022-11-10T21:16:59.436331Z","iopub.status.idle":"2022-11-10T21:16:59.447719Z","shell.execute_reply.started":"2022-11-10T21:16:59.436293Z","shell.execute_reply":"2022-11-10T21:16:59.446210Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Modeling\n\n### Clustering similar races","metadata":{}},{"cell_type":"code","source":"df2 = df_final[id_cols + [col for col in df_final.columns if 'section' in col]]\ndf2 = df2.groupby(['track_id','race_date','race_number']).first().reset_index()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.457615Z","iopub.execute_input":"2022-11-10T21:16:59.458054Z","iopub.status.idle":"2022-11-10T21:16:59.465899Z","shell.execute_reply.started":"2022-11-10T21:16:59.458020Z","shell.execute_reply":"2022-11-10T21:16:59.464031Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_wcss_scores(X, k_max):\n    #  WCSS = within-cluster sum of squares\n    scores = []\n    for k in range(1, k_max+1):\n        kmeans = KMeans(n_clusters=k, random_state=0)\n        kmeans.fit(X)\n        wcss = kmeans.score(X) * -1 # score returns -WCSS\n        scores.append(wcss)\n    return scores\n\n\nX = df2[[col for col in df_final.columns if 'section' in col]]\nwcss_vec = count_wcss_scores(X, 10)\nx_ticks = list(range(1, len(wcss_vec) + 1))\nplt.plot(x_ticks, wcss_vec, 'bx-')\nplt.xlabel('k')\nplt.ylabel('Within-cluster sum of squares')\nplt.title('The Elbow Method showing the optimal k')\nplt.show()\n\n# plt.figure(figsize=(8, 8))\n# barplot0 = mpimg.imread('../input/BigDataDerby2/elbow.png')\n# plt.imshow(barplot0)\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.468791Z","iopub.execute_input":"2022-11-10T21:16:59.469727Z","iopub.status.idle":"2022-11-10T21:16:59.757514Z","shell.execute_reply.started":"2022-11-10T21:16:59.469679Z","shell.execute_reply":"2022-11-10T21:16:59.755930Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def count_clustering_scores(X, cluster_num, model, score_fun):\n    if isinstance(cluster_num, int):\n        cluster_num_iter = [cluster_num]\n    else:\n        cluster_num_iter = cluster_num\n        \n    scores = []    \n    for k in cluster_num_iter:\n        model_instance = model(n_clusters=k)\n        labels = model_instance.fit_predict(X)\n        wcss = score_fun(X, labels)\n        scores.append(wcss)\n    \n    if isinstance(cluster_num, int):\n        return scores[0]\n    else:\n        return scores\n\n\nfrom sklearn.metrics import silhouette_score\n\ncluster_num_seq = range(2, 11)\nsilhouette_vec = count_clustering_scores(X, cluster_num_seq, KMeans, silhouette_score)\nplt.plot(cluster_num_seq, silhouette_vec, 'bx-')\nplt.xlabel('k')\nplt.ylabel('Silhouette score')\nplt.show()\n\n# plt.figure(figsize=(8, 8))\n# barplot0 = mpimg.imread('../input/BigDataDerby2/silhouette.png')\n# plt.imshow(barplot0)\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:16:59.759347Z","iopub.execute_input":"2022-11-10T21:16:59.759776Z","iopub.status.idle":"2022-11-10T21:17:00.060122Z","shell.execute_reply.started":"2022-11-10T21:16:59.759738Z","shell.execute_reply":"2022-11-10T21:17:00.058750Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"KMeans for k=4","metadata":{}},{"cell_type":"code","source":"kmeans_4 = KMeans(n_clusters=4, random_state=0)\nkmeans_4.fit(X)\ny_kmeans_4 = kmeans_4.predict(X)\n\ndf_km_4 = X.copy()\ndf_km_4['pred'] = y_kmeans_4\ndf_km_4.groupby('pred').mean()\n\n# plt.figure(figsize=(15, 15))\n# barplot0 = mpimg.imread('../input/datasets-heads/kmeans4.png')\n# plt.imshow(barplot0)\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.061949Z","iopub.execute_input":"2022-11-10T21:17:00.062469Z","iopub.status.idle":"2022-11-10T21:17:00.365517Z","shell.execute_reply.started":"2022-11-10T21:17:00.062414Z","shell.execute_reply":"2022-11-10T21:17:00.363993Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"KMeans for k=8","metadata":{}},{"cell_type":"code","source":"kmeans_8 = KMeans(n_clusters=8, random_state=0)\nkmeans_8.fit(X)\ny_kmeans_8 = kmeans_8.predict(X)\n\ndf_km_8 = X.copy()\ndf_km_8['group'] = y_kmeans_8\ndf_km_8.groupby('group').mean()\n\n# plt.figure(figsize=(15, 15))\n# barplot0 = mpimg.imread('../input/datasets-heads/kmeans8.png')\n# plt.imshow(barplot0)\n# plt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.368023Z","iopub.execute_input":"2022-11-10T21:17:00.369001Z","iopub.status.idle":"2022-11-10T21:17:00.740969Z","shell.execute_reply.started":"2022-11-10T21:17:00.368922Z","shell.execute_reply":"2022-11-10T21:17:00.739683Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Based on graphical analysis, we considered two different numbers of clusters: k=4 and k=8. Analysis of aggregated section means for each bucket proved that the second solution is more accurate.","metadata":{}},{"cell_type":"code","source":"df2['group'] = y_kmeans_8\ndf_final = df_final.merge(df2[['track_id', 'race_date', 'race_number', 'group']], on = ['track_id', 'race_date', 'race_number'], how='left')\n\ndf_final.drop([col for col in df_final.columns if 'section_' in col], axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.742549Z","iopub.execute_input":"2022-11-10T21:17:00.744353Z","iopub.status.idle":"2022-11-10T21:17:00.751048Z","shell.execute_reply.started":"2022-11-10T21:17:00.744264Z","shell.execute_reply":"2022-11-10T21:17:00.749027Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Scaling the final position to range (0,1)","metadata":{}},{"cell_type":"code","source":"df_final['position_at_finish_max'] = df_final.groupby(['track_id', 'race_date', 'race_number']).position_at_finish.transform('max')\ndf_final['position_at_finish_rel'] = df_final.position_at_finish / df_final.position_at_finish_max\ndf_final.drop('position_at_finish_max', axis=1, inplace=True)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.753212Z","iopub.execute_input":"2022-11-10T21:17:00.754720Z","iopub.status.idle":"2022-11-10T21:17:00.765297Z","shell.execute_reply.started":"2022-11-10T21:17:00.754655Z","shell.execute_reply":"2022-11-10T21:17:00.763764Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"And finally, this is our final data:","metadata":{}},{"cell_type":"code","source":"big_data_derby_final_data_head = pd.read_csv('../input/datasets-heads/big_data_derby_final_data_head.csv', index_col=0)\nbig_data_derby_final_data_head","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.766959Z","iopub.execute_input":"2022-11-10T21:17:00.767919Z","iopub.status.idle":"2022-11-10T21:17:00.825439Z","shell.execute_reply.started":"2022-11-10T21:17:00.767878Z","shell.execute_reply":"2022-11-10T21:17:00.822158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Dividing the table into 8 (similar races)","metadata":{}},{"cell_type":"code","source":"df_final_0 = df_final[df_final.group == 0]\ndf_final_1 = df_final[df_final.group == 1]\ndf_final_2 = df_final[df_final.group == 2]\ndf_final_3 = df_final[df_final.group == 3]\ndf_final_4 = df_final[df_final.group == 4]\ndf_final_5 = df_final[df_final.group == 5]\ndf_final_6 = df_final[df_final.group == 6]\ndf_final_7 = df_final[df_final.group == 7]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.827073Z","iopub.execute_input":"2022-11-10T21:17:00.827943Z","iopub.status.idle":"2022-11-10T21:17:00.834916Z","shell.execute_reply.started":"2022-11-10T21:17:00.827898Z","shell.execute_reply":"2022-11-10T21:17:00.832997Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"columns_to_del_2 = ['Unnamed: 0','track_id', 'race_date', 'race_number', 'program_number', 'distance_id', 'position_at_finish', 'group', 'position_at_finish_rel']\ncolumns_to_del_3 = [col for col in df_final.columns if 'position_in_race_mean_' in col]\ncolumns_to_del_4 = [col for col in df_final.columns if '_5' in col or '_6' in col or '_7' in col or '_8' in col]\ncolumns_to_del_5 = [col for col in df_final.columns if 'avg_rel_speed_max_' in col or 'dst_covered_perc_sum' in col or 'angle_max' in col]\ncolumns_to_del_6 = [col for col in df_final.columns if 'avg_rel_speed_mean_' in col or 'dst_covered_perc_sum' in col or 'angle_max' in col]","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.839408Z","iopub.execute_input":"2022-11-10T21:17:00.840870Z","iopub.status.idle":"2022-11-10T21:17:00.852551Z","shell.execute_reply.started":"2022-11-10T21:17:00.840802Z","shell.execute_reply":"2022-11-10T21:17:00.851189Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"### Plots showing races in each clusters per `track_id`\nThe color indicates races' stage - the darker the more advanced the race.","metadata":{}},{"cell_type":"code","source":"plt.figure(figsize=(30, 30))\nbarplot0 = mpimg.imread('../input/bigdataderby3/3x8.png')\nplt.imshow(barplot0)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:00.854553Z","iopub.execute_input":"2022-11-10T21:17:00.855059Z","iopub.status.idle":"2022-11-10T21:17:02.077063Z","shell.execute_reply.started":"2022-11-10T21:17:00.855008Z","shell.execute_reply":"2022-11-10T21:17:02.075227Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 0  \nStart at the end of a curve and racing throug straight, curve and straight.","metadata":{}},{"cell_type":"code","source":"y = df_final_0['position_at_finish_rel']\nX = df_final_0.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.35)\nmodel_full_0 = DecisionTreeRegressor()\nmodel_full_0.fit(X_train, y_train)\npred_full_0 = model_full_0.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_0),'\\n',\n    mean_squared_error(y_train, model_full_0.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_0),'\\n',\n    max_error(np.array(y_train), model_full_0.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_0),'\\n',\n    median_absolute_error(np.array(y_train), model_full_0.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:30:26.080551Z","iopub.execute_input":"2022-11-10T21:30:26.081010Z","iopub.status.idle":"2022-11-10T21:30:26.088772Z","shell.execute_reply.started":"2022-11-10T21:30:26.080977Z","shell.execute_reply":"2022-11-10T21:30:26.087452Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap0.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot0.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap0.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:02.091928Z","iopub.execute_input":"2022-11-10T21:17:02.092886Z","iopub.status.idle":"2022-11-10T21:17:03.107213Z","shell.execute_reply.started":"2022-11-10T21:17:02.092808Z","shell.execute_reply":"2022-11-10T21:17:03.105564Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 1  \nStart at the beginning of a curve and racing for approximately 1 lap.","metadata":{}},{"cell_type":"code","source":"y = df_final_1['position_at_finish_rel']\nX = df_final_1.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_1 = DecisionTreeRegressor()\nmodel_full_1.fit(X_train, y_train)\npred_full_1 = model_full_1.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_1),'\\n',\n    mean_squared_error(y_train, model_full_1.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_1),'\\n',\n    max_error(np.array(y_train), model_full_1.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_1),'\\n',\n    median_absolute_error(np.array(y_train), model_full_1.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:03.109482Z","iopub.execute_input":"2022-11-10T21:17:03.110068Z","iopub.status.idle":"2022-11-10T21:17:03.118668Z","shell.execute_reply.started":"2022-11-10T21:17:03.110020Z","shell.execute_reply":"2022-11-10T21:17:03.117156Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap1.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot1.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap1.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:03.120190Z","iopub.execute_input":"2022-11-10T21:17:03.120642Z","iopub.status.idle":"2022-11-10T21:17:04.147553Z","shell.execute_reply.started":"2022-11-10T21:17:03.120605Z","shell.execute_reply":"2022-11-10T21:17:04.145756Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 2  \nStart at the beginning of a straight and racing throug straight, curve and straight.","metadata":{}},{"cell_type":"code","source":"y = df_final_2['position_at_finish_rel']\nX = df_final_2.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_2 = DecisionTreeRegressor()\nmodel_full_2.fit(X_train, y_train)\npred_full_2 = model_full_2.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_2),'\\n',\n    mean_squared_error(y_train, model_full_2.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_2),'\\n',\n    max_error(np.array(y_train), model_full_2.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_2),'\\n',\n    median_absolute_error(np.array(y_train), model_full_2.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:04.149642Z","iopub.execute_input":"2022-11-10T21:17:04.150204Z","iopub.status.idle":"2022-11-10T21:17:04.159900Z","shell.execute_reply.started":"2022-11-10T21:17:04.150150Z","shell.execute_reply":"2022-11-10T21:17:04.157895Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap2.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot2.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap2.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:04.161596Z","iopub.execute_input":"2022-11-10T21:17:04.162085Z","iopub.status.idle":"2022-11-10T21:17:05.231248Z","shell.execute_reply.started":"2022-11-10T21:17:04.162026Z","shell.execute_reply":"2022-11-10T21:17:05.228913Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 3  \nStart in the middle of a curve (or at the beginning of an extended straight) and racing through straight, curve and straight","metadata":{}},{"cell_type":"code","source":"y = df_final_3['position_at_finish_rel']\nX = df_final_3.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_3 = DecisionTreeRegressor()\nmodel_full_3.fit(X_train, y_train)\npred_full_3 = model_full_3.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_3),'\\n',\n    mean_squared_error(y_train, model_full_3.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_3),'\\n',\n    max_error(np.array(y_train), model_full_3.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_3),'\\n',\n    median_absolute_error(np.array(y_train), model_full_3.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:05.233388Z","iopub.execute_input":"2022-11-10T21:17:05.234189Z","iopub.status.idle":"2022-11-10T21:17:05.242937Z","shell.execute_reply.started":"2022-11-10T21:17:05.234135Z","shell.execute_reply":"2022-11-10T21:17:05.240957Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap3.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot3.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap3.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:05.244621Z","iopub.execute_input":"2022-11-10T21:17:05.245021Z","iopub.status.idle":"2022-11-10T21:17:06.291239Z","shell.execute_reply.started":"2022-11-10T21:17:05.244988Z","shell.execute_reply":"2022-11-10T21:17:06.289800Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 4  \nStart at the beginning of a straight and racing for 1 lap + 1 straight.","metadata":{}},{"cell_type":"code","source":"y = df_final_4['position_at_finish_rel']\nX = df_final_4.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_4 = DecisionTreeRegressor()\nmodel_full_4.fit(X_train, y_train)\npred_full_4 = model_full_4.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_4),'\\n',\n    mean_squared_error(y_train, model_full_4.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_4),'\\n',\n    max_error(np.array(y_train), model_full_4.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_4),'\\n',\n    median_absolute_error(np.array(y_train), model_full_4.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:06.292841Z","iopub.execute_input":"2022-11-10T21:17:06.293742Z","iopub.status.idle":"2022-11-10T21:17:06.300649Z","shell.execute_reply.started":"2022-11-10T21:17:06.293700Z","shell.execute_reply":"2022-11-10T21:17:06.299570Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap4.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot4.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap4.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:06.301987Z","iopub.execute_input":"2022-11-10T21:17:06.302439Z","iopub.status.idle":"2022-11-10T21:17:07.386932Z","shell.execute_reply.started":"2022-11-10T21:17:06.302402Z","shell.execute_reply":"2022-11-10T21:17:07.385423Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Cluster 5  \nStart in the middle of a satraight and racing through the remainder of this straight, curve and straight.","metadata":{}},{"cell_type":"code","source":"y = df_final_5['position_at_finish_rel']\nX = df_final_5.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_5 = DecisionTreeRegressor()\nmodel_full_5.fit(X_train, y_train)\npred_full_5 = model_full_5.predict(X_test)\n\n\nprint(\n    mean_squared_error(y_test, pred_full_5),'\\n',\n    mean_squared_error(y_train, model_full_5.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_5),'\\n',\n    max_error(np.array(y_train), model_full_5.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_5),'\\n',\n    median_absolute_error(np.array(y_train), model_full_5.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:07.388566Z","iopub.execute_input":"2022-11-10T21:17:07.389624Z","iopub.status.idle":"2022-11-10T21:17:07.398950Z","shell.execute_reply.started":"2022-11-10T21:17:07.389560Z","shell.execute_reply":"2022-11-10T21:17:07.396665Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap5.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot5.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap5.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:07.401422Z","iopub.execute_input":"2022-11-10T21:17:07.402049Z","iopub.status.idle":"2022-11-10T21:17:08.497695Z","shell.execute_reply.started":"2022-11-10T21:17:07.402000Z","shell.execute_reply":"2022-11-10T21:17:08.496678Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"#### Clusters 6 and 7  \nLow number of observations in these clusters might cause misleading conclusions.","metadata":{}},{"cell_type":"code","source":"y = df_final_6['position_at_finish_rel']\nX = df_final_6.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_6 = DecisionTreeRegressor()\nmodel_full_6.fit(X_train, y_train)\npred_full_6 = model_full_6.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_6),'\\n',\n    mean_squared_error(y_train, model_full_6.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_6),'\\n',\n    max_error(np.array(y_train), model_full_6.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_6),'\\n',\n    median_absolute_error(np.array(y_train), model_full_6.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:08.499574Z","iopub.execute_input":"2022-11-10T21:17:08.500320Z","iopub.status.idle":"2022-11-10T21:17:08.509071Z","shell.execute_reply.started":"2022-11-10T21:17:08.500277Z","shell.execute_reply":"2022-11-10T21:17:08.506816Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap6.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot6.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap6.png')\nplt.imshow(barplot0)\nplt.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:08.511646Z","iopub.execute_input":"2022-11-10T21:17:08.512335Z","iopub.status.idle":"2022-11-10T21:17:09.947502Z","shell.execute_reply.started":"2022-11-10T21:17:08.512281Z","shell.execute_reply":"2022-11-10T21:17:09.945695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"y = df_final_7['position_at_finish_rel']\nX = df_final_7.drop(columns_to_del_2 + columns_to_del_3 + columns_to_del_4 +  columns_to_del_6, axis = 1)\n\n\nX_train, X_test, y_train, y_test = train_test_split(X, y, random_state=420, test_size=0.3)\nmodel_full_7 = DecisionTreeRegressor()\nmodel_full_7.fit(X_train, y_train)\npred_full_7 = model_full_7.predict(X_test)\n\nprint(\n    mean_squared_error(y_test, pred_full_7),'\\n',\n    mean_squared_error(y_train, model_full_7.predict(X_train)),'\\n',\n    max_error(np.array(y_test), pred_full_7),'\\n',\n    max_error(np.array(y_train), model_full_7.predict(X_train)),'\\n',\n    median_absolute_error(np.array(y_test), pred_full_7),'\\n',\n    median_absolute_error(np.array(y_train), model_full_7.predict(X_train))\n)","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:09.950040Z","iopub.execute_input":"2022-11-10T21:17:09.950503Z","iopub.status.idle":"2022-11-10T21:17:09.959324Z","shell.execute_reply.started":"2022-11-10T21:17:09.950440Z","shell.execute_reply":"2022-11-10T21:17:09.957574Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.figure(figsize=(15, 10))\nbarplot0 = mpimg.imread('../input/BigDataDerby/heatmap7.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(10, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/barplot7.png')\nplt.imshow(barplot0)\nplt.show()\n\nplt.figure(figsize=(8, 8))\nbarplot0 = mpimg.imread('../input/BigDataDerby/shap7.png')\nplt.imshow(barplot0)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-10T21:17:09.962267Z","iopub.execute_input":"2022-11-10T21:17:09.963426Z","iopub.status.idle":"2022-11-10T21:17:11.051578Z","shell.execute_reply.started":"2022-11-10T21:17:09.963359Z","shell.execute_reply":"2022-11-10T21:17:11.050158Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Clustering algorithm included dividing the tracks into 6 parts:  \n1 - beginning of curve  \n2 - middle of curve  \n3 - end of curve  \n4 - beginning of straight  \n5 - middle of straight  \n6 - end of straight  \nand calculating average position (ranging from 1 to 6) of horses during subsequent deciles of a race.  \n\nThis approach allowed us to successfully (excluding several odd races) group races based on their length and placement of the starting line on the track.  \n\nTo cluster the processed data, consisting of 9 variables (1 for each decile, excluding the last one due to several anomalies), we used the K-means algorithm.  \n\nWe believe that this approach could prove very useful in further analyses of racing data and, especially with more data available, become a more accurate and powerful tool to examine racing strategies based on a particular race's cluster.  ","metadata":{}},{"cell_type":"markdown","source":"## Explaining the models","metadata":{}},{"cell_type":"markdown","source":"Unfortunately, the models did not produce as many insights as possible, but we managed to deduce a few things:\n\n* If a race starts on a curve, then its prefered to straighten out as soon as possible, in order to gain speed faster.\n\n* For races starting with a straight (e.g. cluster (2)) , its helpful to stay behind another horse in the beginning of a race, to make full use of drafting.\n\n* Drafting, being a useful tool in jockey's arsenal, is not always the best strategy. Drafting right before a curve tends to produce greater results, compared to anywhere else on the track.\n\n* In general, unsurprisingly, the most important features tend to be speed features (the fastest horse wins)\n\n","metadata":{}}]}