{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Intensity Throughout the Race\n\n**Are there any Latent Similarities Unique to Top Performers?** \n\nAll racers' ranking at any given snapshot is determined by their cumulative advantage (in distance) relative to that of their competitor's. Coordinates between two adjacent trakus indices tell us the *marginal distance* travelled from the time elapsed, providing insights into the marginal advantage (in meter) of a racer at any given snapshot relative to that of their competitors'.\n\nFor a small distance travelled across trakus indices (abt 0 to 5 meters, which can be around less than 10 horse steps), I am assuming distance to be equal to displacement -- that is, any curvatures to have negligible impacts on the horse's travel distance. Therefore I used the quickest way, that is EquiRectangular to compute the distance between every 0.25 second.\n\nTrack length differs across races. Therefore, the trakus index can be converted into *the percentage to race completion* using the formula: *PercentToRaceCompletion = trakus_index / maximum_trakus_index_in_the_race x 100* \nThe idea is to find, across all the races, is there a systematic progression in which Top performing horses deploy their marginal advantage over Bottom performing horses across time snapshots.\n\nDependencies include: The marginal performance a horse at any given snapshot could be dependent on their own cumulative performance (lag effects/ escalated commitment, arguably intractable), and also dependent on both the marginal and cumulative performance of their most important rival (one directly trailing behind). Yet, if the most important competitor has a particularly weak performance for several continuous snapshots, the horse next in line behind could emerge crucial. For any two horses in a race, at any snapshot, as their cumulative ranking gap increases, their relative importance in potentially changing each other's ranking decreases. \n\nTo find: in every race, for every horse, at a given percentage to race completion (10%, 20%...) :\n- What is the marginal distance travelled from one snapshot to the next\n- What is their marginal advantage (in meter) compared to the horse trailing directly behind them (-1) between each race completion percentage\n- What is their marginal advantage (in meter) compared to the horse trailing two places behind them (-2), three places behind them (-3) and so on..\n- Is there a systematic way in which Top performing horses speed up or slow down relative to their competitors at different percentage of completion of the race?\n\n**The Performance Metric** \n\nThe (derived) metric used to judge the performance of a horse is: \\\n*AdjustedFinalRankScore  = (position_at_finish / total_racers_in_the_race) x-100*  \\\nThis is a numerical score between -100 and 0 (negative because ranking is reversed in that less means better. Ie. first is better than fifth). Top performers are horses whose scores are at the Top 33 percentiles. Middle being 33-66 percentiles and Bottom being anything over 66 percentiles.\n\nThis normalised performance metric is used in order to allow for the trends observed across different races to be able to infer meaningful generalisation. An absolute ranking would infer meaningful implications only if the number of competitors is the same across the different races, which is not the case from this dataset. To illustrate, being placed third in a race of three could mean being placed last, whilst being placed third in a race of 12 could mean a good race. \n","metadata":{}},{"cell_type":"code","source":"import numpy as np \nimport pandas as pd\nfrom math import *\nimport matplotlib.pyplot as plt\nimport plotly.express as px\nimport plotly.graph_objects as go\nimport seaborn as sns\nsns.set_theme(style=\"darkgrid\")\nimport time\nimport warnings\nwarnings.filterwarnings(\"ignore\")\npd.set_option(\"display.max_columns\", None)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:45:04.222147Z","iopub.execute_input":"2022-11-09T15:45:04.222879Z","iopub.status.idle":"2022-11-09T15:45:06.133996Z","shell.execute_reply.started":"2022-11-09T15:45:04.222734Z","shell.execute_reply":"2022-11-09T15:45:06.132695Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"race_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_race_table.csv')\nstart_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_start_table.csv', header=None)\ntracking_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_tracking_table.csv')\ncomplete_table = pd.read_csv('/kaggle/input/big-data-derby-2022/nyra_2019_complete.csv')","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:45:06.136005Z","iopub.execute_input":"2022-11-09T15:45:06.136559Z","iopub.status.idle":"2022-11-09T15:45:39.237258Z","shell.execute_reply.started":"2022-11-09T15:45:06.136524Z","shell.execute_reply":"2022-11-09T15:45:39.236077Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def MarginalDistance(row):\n    return geodesic(row[\"Coordinates\"], row[\"NextCoordinates\"]).meters\n\ndef EquiRectangular(row):\n    lat1, lon1 = row[\"Coordinates\"][0], row[\"Coordinates\"][1]\n    lat2, lon2 = row[\"NextCoordinates\"][0], row[\"NextCoordinates\"][1]\n    \n    R = 6371 \n    x = (radians(lon2) - radians(lon1)) * cos(0.5 * (radians(lat2) + radians(lat1)))\n    y = radians(lat2) - radians(lat1)\n    d = R * sqrt(x*x + y*y)*1000\n    return d\n\ndef Haversine(row):\n    lat1, lon1 = row[int(start)][0], row[int(start)][1]\n    lat2, lon2 = row[int(end)][0], row[int(end)][1]\n    lon1, lat1, lon2, lat2 = map(radians, [lon1, lat1, lon2, lat2])\n    dlon = lon2 - lon1 \n    dlat = lat2 - lat1 \n    a = sin(dlat/2)**2 + cos(lat1) * cos(lat2) * sin(dlon/2)**2\n    c = 2 * asin(sqrt(a)) \n    m = 6371* c*1000\n    return m","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:45:39.238757Z","iopub.execute_input":"2022-11-09T15:45:39.239140Z","iopub.status.idle":"2022-11-09T15:45:39.250591Z","shell.execute_reply.started":"2022-11-09T15:45:39.239102Z","shell.execute_reply":"2022-11-09T15:45:39.249516Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"%%time\n\n#---------------------------\n# Race table\ndf1 = race_table\nfor x in [\"track_id\", \"race_date\", \"race_number\", \"course_type\", \"track_condition\", \"race_type\"]:\n    df1[x] = df1[x].astype(str)\ndf1 = df1.applymap(lambda x: x.strip() if isinstance(x, str) else x)\ndf1[\"UniqueRace\"] = df1.track_id + \"_\" + df1.race_date + \"_\" + df1.race_number\ndf1[\"post_time\"] = df1[\"post_time\"].astype(str).str.zfill(4)\ndf1[\"HalfPastIndicator\"] = np.where( df1[\"post_time\"].str[-2:].astype(int)>30, 1, 0)\ndf1[\"StartHourRounded\"] = np.where(df1[\"HalfPastIndicator\"]==0, df1[\"post_time\"].str[:2].astype(int), df1[\"post_time\"].str[:2].astype(int)+1)\ndf1 = df1.drop([\"track_id\", \"race_date\", \"race_number\", \"post_time\", \"HalfPastIndicator\"], axis=1)\n\n#---------------------------\n# Start table\ndf2 = start_table\nfor x in [1,2,3,5]:\n    df2[x] = df2[x].astype(str)\ndf2 = df2.applymap(lambda x: x.strip() if isinstance(x, str) else x)\ndf2[\"UniqueRace\"] = df2[0] + \"_\" + df2[1] + \"_\" + df2[2]\ndf2[\"UniqueHorseInRace\"] = df2[\"UniqueRace\"] + \"_\" + df2[3]\ndf2 = df2.rename(columns={0: \"track_id\", 4:\"weight_carried\", 5:\"jockey\", 6:\"odds\", 7:\"position_at_finish\"})\ndf2[\"PlayersCountInRace\"] = df2.groupby([\"UniqueRace\"])[\"position_at_finish\"].transform(\"max\")\ndf2[\"AdjustedFinalRankScore\"] = df2[\"position_at_finish\"]/df2[\"PlayersCountInRace\"]*-100\ndf2[\"JockeyScore\"] = df2.groupby([\"jockey\"])[\"AdjustedFinalRankScore\"].transform(\"mean\")\ndf2 = df2.drop([1,2,3, \"UniqueRace\", \"jockey\", \"PlayersCountInRace\"], axis=1)\n\n#---------------------------\n#Tracking table\ndf = tracking_table\nfor x in [\"track_id\", \"race_date\", \"race_number\", \"program_number\"]:\n    df[x] = df[x].astype(str)\ndf = df.applymap(lambda x: x.strip() if isinstance(x, str) else x)\ndf[\"UniqueRace\"] = df.track_id + \"_\" + df.race_date + \"_\"+ df.race_number\ndf[\"UniqueHorseInRace\"] = df.UniqueRace + \"_\" + df.program_number\ndf[\"Coordinates\"] =  list(zip(df.latitude, df.longitude))\ndf[\"NextCoordinates\"] = df.sort_values(by=[\"trakus_index\"], ascending=True).groupby([\"UniqueRace\", \"UniqueHorseInRace\"])[\"Coordinates\"].shift(-1)\ndf = df[pd.notnull(df[\"NextCoordinates\"])]\ndf[\"MarginalDistance\"] = df.apply(lambda row: EquiRectangular(row), axis=1)\ndf = df.sort_values(['trakus_index']).reset_index(drop=True)\ndf[\"CumulativeDistance\"]=df.groupby(['UniqueHorseInRace'])['MarginalDistance'].cumsum(axis=0)\ndf[\"MarginalRanking\"] = df.groupby([\"UniqueRace\", \"trakus_index\"])[\"MarginalDistance\"].rank(\"dense\", ascending=False)\ndf[\"CumulativeRanking\"] = df.groupby([\"UniqueRace\", \"trakus_index\"])[\"CumulativeDistance\"].rank(\"dense\", ascending=False)\ndf[\"TotalTrakus\"] = df.groupby([\"UniqueHorseInRace\"])[\"trakus_index\"].transform(\"max\")\ndf[\"ProgressInRace\"] = df.trakus_index/ df.TotalTrakus*100\nconditions, x = [], 10\nwhile True:\n    if x >=101:\n        break\n    else:\n        condition = f\"(df['ProgressInRace'] <= {x})\"\n        conditions.append(condition)\n        x +=10        \nprogress = list(range(10,101, 10))\ndf[\"PercentToRaceCompletion\"] = np.select([eval(x) for x in conditions], progress)\n\ntrails = [x for x in range(1,6)]\nfor no in trails:\n    df[\"TrailingHorse_\"+ str(no) +\"_MarginalDistance\"] =  \\\n            df.sort_values(by=[\"UniqueRace\",\"trakus_index\",\"CumulativeRanking\"], ascending=True)  \\\n                .groupby([\"UniqueRace\", \"trakus_index\"])[\"MarginalDistance\"].shift(-int(no))\n    df[\"MarginalAdvantageOverTrail_\" + str(no)] =   \\\n            np.where(pd.isnull(df[\"TrailingHorse_\" + str(no) +\"_MarginalDistance\"]), np.nan, df[\"MarginalDistance\"]-df[\"TrailingHorse_\" + str(no) +\"_MarginalDistance\"])","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:45:39.254071Z","iopub.execute_input":"2022-11-09T15:45:39.254486Z","iopub.status.idle":"2022-11-09T15:50:43.305039Z","shell.execute_reply.started":"2022-11-09T15:45:39.254446Z","shell.execute_reply":"2022-11-09T15:50:43.303636Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"#Tracking table\n#Extensive transformations have been done on this df, the column names are explanatory of the transformation performed.\ndf.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:50:43.306856Z","iopub.execute_input":"2022-11-09T15:50:43.308387Z","iopub.status.idle":"2022-11-09T15:50:43.350421Z","shell.execute_reply.started":"2022-11-09T15:50:43.308330Z","shell.execute_reply":"2022-11-09T15:50:43.349304Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The tracking table seen above currently expresses the marginal advantage of a horse relative to their rivals for **each elapsed trackus index.** Since total trakus indices is dependent on the competition track length, this makes comparison across races unmeaningful. To rectify, as the below block, dfconsol expresses each horse's marginal advantage as a cumulative (sum) of marginal advantage for every trakus indices belonging to a pre-defined range (in multiples of ten) of PercentToRaceCompletion.\nWe are able to see the **marginal** advantage of a horse relative to their rivals at the 10%, 20%... up to 100% to race completion. ","metadata":{}},{"cell_type":"code","source":"%%time\n# Marginal Advantage (in meter) of horses over their rivals by PercentToRaceCompletion\ndfconsol = df.groupby([\"UniqueRace\", 'UniqueHorseInRace', \"PercentToRaceCompletion\"]) \\\n        .agg({'MarginalDistance': 'sum', \\\n            'MarginalAdvantageOverTrail_1': 'sum', \\\n            'MarginalAdvantageOverTrail_2': 'sum', \\\n            'MarginalAdvantageOverTrail_3': 'sum', \\\n            'MarginalAdvantageOverTrail_4': 'sum', \\\n            'MarginalAdvantageOverTrail_5': 'sum'})  \\\n           .reset_index()\ndfconsol = dfconsol.sort_values(['PercentToRaceCompletion']).reset_index(drop=True)\ndfconsol[\"CumulativeDistance\"]= dfconsol.groupby(['UniqueHorseInRace'])['MarginalDistance'].cumsum(axis=0)\ndfconsol[\"CumulativeRanking\"]= dfconsol.groupby([\"UniqueRace\", \"PercentToRaceCompletion\"])[\"CumulativeDistance\"].rank(\"dense\", ascending=False)\ndfconsol[\"PlayersCountInRace\"] = dfconsol.groupby([\"UniqueRace\"])[\"CumulativeRanking\"].transform(\"max\")\ndfconsol[\"AdjustedCumulativeRankScore\"] = dfconsol[\"CumulativeRanking\"]/dfconsol[\"PlayersCountInRace\"]*-100\n\ndfconsol.head(5)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:50:43.351914Z","iopub.execute_input":"2022-11-09T15:50:43.352294Z","iopub.status.idle":"2022-11-09T15:50:50.308473Z","shell.execute_reply.started":"2022-11-09T15:50:43.352246Z","shell.execute_reply":"2022-11-09T15:50:50.307386Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As a further explanation, in the below table, when **PercentToRaceCompletion is 10, variable is MarginalAdvantageOverTrailOne** and **value is 1.5**, this means that from trakus index 1, until the trakus index indicative of 10% into the race, all the marginal advantage (in meter) of the horse relative to the horse directly behind it is summed up, and the sum is 1.5. For **PercentToRaceCompletion**==20, this means all the marginal advantage from the trakus index when PercentToRaceCompletion==10 up until PercentToRaceCompletion==20 is summed up as **value**. ","metadata":{}},{"cell_type":"code","source":"#Visualise the outcomes\ndf_vis =  pd.melt(dfconsol, id_vars=[\"UniqueRace\", \"UniqueHorseInRace\", \"PercentToRaceCompletion\"], value_vars=[\"MarginalAdvantageOverTrail_1\", \"MarginalAdvantageOverTrail_2\", \"MarginalAdvantageOverTrail_3\", \"MarginalAdvantageOverTrail_4\", \"MarginalAdvantageOverTrail_5\"])\ndf_vis = df_vis.merge(df2,on=\"UniqueHorseInRace\", how=\"left\") \\\n                                .merge(df1, on=\"UniqueRace\", how=\"left\") \n\n#Final Rank\nconditions = [\n    (df_vis['AdjustedFinalRankScore'] < -66),\n    (df_vis['AdjustedFinalRankScore'] >=-66 ) & (df_vis['AdjustedFinalRankScore'] <=-33),\n    (df_vis['AdjustedFinalRankScore'] > -33)]\nvalues = ['Bottom33%', 'Middle%', 'Top33%']\ndf_vis['FinalRankingCategory'] = np.select(conditions, values)\n\ndf_vis[[\"UniqueRace\", \"UniqueHorseInRace\", \"PercentToRaceCompletion\", \"variable\", \"value\", \"FinalRankingCategory\"]].head(5)\ndf_vis = df_vis.rename(columns={'value': 'Marginal Advantage (in meter)', \"PercentToRaceCompletion\": \"PercentToRaceCompletion(%)\"})\n\n#Current Rank\ndf_vis[\"CurrentRankAtPercentageToCompletion\"] = df_vis.groupby([\"variable\", \"UniqueRace\",\"PercentToRaceCompletion(%)\"])[\"Marginal Advantage (in meter)\"].rank(\"dense\", ascending=False)\ndf_vis[\"PlayersCountInRace\"] = df_vis.groupby([\"variable\",\"UniqueRace\"])[\"CurrentRankAtPercentageToCompletion\"].transform(\"max\")\ndf_vis[\"CurrentScoreAtPercentageToCompletion\"] = df_vis[\"CurrentRankAtPercentageToCompletion\"]/df_vis[\"PlayersCountInRace\"]*-100\n\nconditions = [\n    (df_vis['CurrentScoreAtPercentageToCompletion'] < -66),\n    (df_vis['CurrentScoreAtPercentageToCompletion'] >=-66 ) & (df_vis['AdjustedFinalRankScore'] <=-33),\n    (df_vis['CurrentScoreAtPercentageToCompletion'] > -33)]\nvalues = ['Bottom33%', 'Middle%', 'Top33%']\ndf_vis['CurrentRankingCategory'] = np.select(conditions, values)\n\ndf_vis.head(3)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:50:50.309880Z","iopub.execute_input":"2022-11-09T15:50:50.310207Z","iopub.status.idle":"2022-11-09T15:50:52.931195Z","shell.execute_reply.started":"2022-11-09T15:50:50.310175Z","shell.execute_reply":"2022-11-09T15:50:52.929934Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The aim is to able to visualise, for every horse grouped by their final ranking category (Top 33%, Middle 33%, Bottom 33% finisher), the manners in which a category exhibits marginal advantage in distance to the horse trailing behind them across timespan (by PercentToRaceCompletion) \n\nDo note that the horse whose ranking is last would not have any marginal advantage over any horse trailing behind (because they are last!). This makes marginal advantage expressed as average not indicative of true average for the Bottom 33 category. This inevitably makes any comparison with the Bottom 33% category inaccurate.\n\nIt will be useful to look at the first 2 graphs in the diagram. (they respectively are for the horse directly trailing behind, and the horse railing at -2). As a reminder, as ranking gap increases, two horses would have less potential to influence each others' final ranking.","metadata":{}},{"cell_type":"code","source":"#Average marginal advantage (meter)\nT = df_vis.groupby([\"variable\", \"FinalRankingCategory\", \"PercentToRaceCompletion(%)\"])['Marginal Advantage (in meter)'].mean().reset_index()\nfig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2, figsize=(22, 15))\nT1 = T[T[\"variable\"]==\"MarginalAdvantageOverTrail_1\"]\nT2 = T[T[\"variable\"]==\"MarginalAdvantageOverTrail_2\"]\nT3 = T[T[\"variable\"]==\"MarginalAdvantageOverTrail_3\"]\nT4 = T[T[\"variable\"]==\"MarginalAdvantageOverTrail_4\"]\n\nsns.lineplot(data=T1, x=\"PercentToRaceCompletion(%)\", y=\"Marginal Advantage (in meter)\", hue=\"FinalRankingCategory\", palette=\"icefire\", ax=ax1)\nsns.lineplot(data=T2, x=\"PercentToRaceCompletion(%)\", y=\"Marginal Advantage (in meter)\", hue=\"FinalRankingCategory\", palette=\"icefire\", ax=ax2)\nsns.lineplot(data=T3, x=\"PercentToRaceCompletion(%)\", y=\"Marginal Advantage (in meter)\", hue=\"FinalRankingCategory\", palette=\"icefire\", ax=ax3)\nsns.lineplot(data=T4, x=\"PercentToRaceCompletion(%)\", y=\"Marginal Advantage (in meter)\", hue=\"FinalRankingCategory\", palette=\"icefire\", ax=ax4)\n\nax1.set_title('Average Marginal Advantage (meter) Relative to The Horse Directly Trailing Behind', fontsize=13)\nax2.set_title('Average Marginal Advantage (meter) Relative to Their -2 Trailing Horse', fontsize=13)\nax3.set_title('Average Marginal Advantage (meter) Relative to Their -3 Trailing Horse', fontsize=13)\nax4.set_title('Average Marginal Advantage (meter) Relative to Their -4 Trailing Horse', fontsize=13)\nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:50:52.932869Z","iopub.execute_input":"2022-11-09T15:50:52.933368Z","iopub.status.idle":"2022-11-09T15:50:54.212557Z","shell.execute_reply.started":"2022-11-09T15:50:52.933329Z","shell.execute_reply":"2022-11-09T15:50:54.211207Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The trakus index is incremental by one at every 0.25 second. This consistent span means we can use marginal distance travelled as a proxy for speed/performance(Speed = Distance/Time). Please make exception, that the 100% completion point at x-axis (and arguably the 90% completion too) may not a complete picture of speed, because some Top performing horses would have completed the race. This explains the dip in average marginal distance travelled for Top Performers. There could be players-attrition mid-race. I didn't yet account for that.\n\nWe will be most interested in the top left graph, for the horse directly trailing behind. The other three graphs (-2, -3, -4 behind) are available for reference.","metadata":{}},{"cell_type":"markdown","source":"# Hot Take 1: The Timing of Inflection Point can be Crucial\n\nLooking at top left graph, the point where dy/dx=0 is at about 30% to race completion for top performers; about 40% for middle performers and about 70% for bottom performers.","metadata":{}},{"cell_type":"markdown","source":"# Hot Take 2: U-shaped Intensity is a Shared Strategy, but Implementation Determines Results\n\nThe same U-shaped intensity trajectory applies to all competitors. This could be indicative that it is well-known to racers, they generally should start strong but pick-up even stronger toward the end. Nuances in ability, implementation and timing could arguably be determining of the actual performance:-\n- Everyone seems to be doing the same thing by keeping close to their trailing enemy from 10% to 30% into the race. However, the exact timing (30% to 40%) during which this pace is intensely picked up again differs with Top and Middle performers (top performers generally start earlier, with difference visible at around 40% into the race, whilst middle to bottom performers take some time to pick up again after slowing down)\n- At the beginning of the race, there seems to be inherent difference in ability when all are assumedly attempting to attain this marginal advantage over the -1 trailing horse. \n- The rate of change of picking up pace, perhaps reflective of competence more than strategy, drives the largest wedge between performance category.  At around 60% to the race onwards, high performers are able to very distinctly keep a good marginal advantage (in distance) between themselves and the horse directly trailing behind.","metadata":{}},{"cell_type":"markdown","source":"Next, it could be useful to know:-\n- How does the race start for Top and Bottom performers? The above, we have looked at horses' average marginal advantage for snapshots in multiples of 10; can we look into the starting, on a more granular level -- that is, 0% up to 2% of the race progression.\n- How much variability exists in the dataset, which can be visualised using parallel coordinates.","metadata":{}},{"cell_type":"code","source":"# Look at how the race starts 0% -> 2% of race\ndf_Starting = df\ndf_Starting= df_Starting.merge(df2,on=\"UniqueHorseInRace\", how=\"left\") \\\n                                .merge(df1, on=\"UniqueRace\", how=\"left\") \n\n#Final Rank\nconditions = [\n    (df_Starting['AdjustedFinalRankScore'] < -66),\n    (df_Starting['AdjustedFinalRankScore'] >=-66 ) & (df_Starting['AdjustedFinalRankScore'] <=-33),\n    (df_Starting['AdjustedFinalRankScore'] > -33)]\nvalues = ['Bottom33%', 'Middle%', 'Top33%']\ndf_Starting['FinalRankingCategory'] = np.select(conditions, values)\ndf_Starting = df_Starting[df_Starting[\"ProgressInRace\"]<=2]\ntop= df_Starting[df_Starting[\"FinalRankingCategory\"]==\"Top33%\"]\nmiddle= df_Starting[df_Starting[\"FinalRankingCategory\"]==\"Middle%\"]\nbottom= df_Starting[df_Starting[\"FinalRankingCategory\"]==\"Bottom33%\"]\n\nfig, (ax1, ax2, ax3) = plt.subplots(1,3, figsize=(20, 10))\nsns.lineplot(data=top, x=\"ProgressInRace\", y=\"MarginalAdvantageOverTrail_1\", color=\"red\", ax=ax1)\nsns.lineplot(data=middle, x=\"ProgressInRace\", y=\"MarginalAdvantageOverTrail_1\", color=\"black\", ax=ax2)\nsns.lineplot(data=bottom, x=\"ProgressInRace\", y=\"MarginalAdvantageOverTrail_1\", color=\"blue\", ax=ax3)\nax1.set_title('Top Performers', fontsize=13)\nax1.set_ylim([-4, 6])\nax2.set_title('Middle Performers', fontsize=13)\nax2.set_ylim([-4, 6])\nax3.set_title('Bottom Performers', fontsize=13)\nax3.set_ylim([-4, 6])\nfig.suptitle('How the Race Started: from 0% to 2% of Completion') \nplt.show()","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:51:54.510095Z","iopub.execute_input":"2022-11-09T15:51:54.510621Z","iopub.status.idle":"2022-11-09T15:54:04.705772Z","shell.execute_reply.started":"2022-11-09T15:51:54.510579Z","shell.execute_reply":"2022-11-09T15:54:04.704442Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The data have told us all players follow a stylised-U progression, and high performers deploy their marginal maximum capability after mid-point. We are interested to know how are Top performing players beginning the race. Interestingly, from the above diagram, at the very start of the race (0% - 0.25%), Top performers (red) do not attain a large marginal advantage relative to the horse trailing behind. The opposite is true for Middle and Bottom performers, in which we see a spike during that range. ","metadata":{}},{"cell_type":"markdown","source":"Next, we are interested in how much variability exists in the data and whether that could undermine the aggregated findings thus far. Using a parallel coordinates, every row entry on the data is visualised as a line, as below.","metadata":{}},{"cell_type":"code","source":"%%time\n\nMarginalAdvantageOverTrailOne= pd.pivot(dfconsol, index=[\"UniqueRace\", \"UniqueHorseInRace\"], columns =\"PercentToRaceCompletion\",values = \"MarginalAdvantageOverTrail_1\").reset_index()\nfor col in [x for x in MarginalAdvantageOverTrailOne.columns if x not in [\"UniqueRace\", \"UniqueHorseInRace\"]]:\n    MarginalAdvantageOverTrailOne = MarginalAdvantageOverTrailOne.rename(columns={col:str(col)+\"th\"})\n\n# Consolidation\nfor x in [\"UniqueRace\", \"UniqueHorseInRace\"]:\n    for thedf in [\"MarginalAdvantageOverTrailOne\", \"df2\"]:\n        if x in eval(thedf).columns:\n            globals()[f\"{thedf}\"][x] = eval(thedf)[x].astype(str)\n\ndff = MarginalAdvantageOverTrailOne.merge(df2,on=\"UniqueHorseInRace\", how=\"left\") \n\ncols = dff.columns[:12].to_list()\ncols.append(\"AdjustedFinalRankScore\")\nconditions = [\n    (dff['AdjustedFinalRankScore'] < -66),\n    (dff['AdjustedFinalRankScore'] >=-66 ) & (dff['AdjustedFinalRankScore'] <=-33),\n    (dff['AdjustedFinalRankScore'] > -33)]\nvalues = [3, 2, 1]\ndff['FinalRankingCategory'] = np.select(conditions, values)\n# cols.extend([\"weight_carried\", \"odds\", \"AdjustedFinalRankScore\", \"JockeyScore\"])\ncols.append(\"FinalRankingCategory\")\n\ndf_vis =dff\nTop = df_vis[df_vis[\"FinalRankingCategory\"]==1]\nMiddle = df_vis[df_vis[\"FinalRankingCategory\"]==2]\nBottom = df_vis[df_vis[\"FinalRankingCategory\"]==3]\n\nfor x in [\"Top\", \"Middle\", \"Bottom\"]:\n    dimensions = list([ dict(range=(-50,50), label='10%IntoRace',values=eval(x)['10th']),\n                        dict(range=(-50,50), label='20%IntoRace',values=eval(x)['20th']),\n                       dict(range=(-50,50), label='30%IntoRace',values=eval(x)['30th']),\n                       dict(range=(-50,50), label='40%IntoRace',values=eval(x)['40th']),\n                       dict(range=(-50,50), label='50%IntoRace',values=eval(x)['50th']),\n                       dict(range=(-50,50), label='60%IntoRace',values=eval(x)['60th']),\n                       dict(range=(-50,50), label='70%IntoRace',values=eval(x)['70th']),\n                       dict(range=(-50,50), label='80%IntoRace',values=eval(x)['80th']),\n                       dict(range=(-50,50), label='90%IntoRace',values=eval(x)['90th']),\n                       dict(range=(-50,50), label='100%IntoRace',values=eval(x)['100th']),\n                      ])\n    if x ==\"Top\":\n        c = \"red\"\n    elif x==\"Middle\":\n        c = \"black\"\n    else:\n        c = \"blue\"\n\n    fig= go.Figure(data= go.Parcoords(line = dict(color = c), dimensions = dimensions))\n    fig.update_layout(\n        width=900,\n        height=300)\n    fig.update_layout(title_text=f'Parallel Coordinates for {x} Performers')\n    fig.show()\n","metadata":{"execution":{"iopub.status.busy":"2022-11-09T15:54:04.708009Z","iopub.execute_input":"2022-11-09T15:54:04.708831Z","iopub.status.idle":"2022-11-09T15:54:05.106778Z","shell.execute_reply.started":"2022-11-09T15:54:04.708767Z","shell.execute_reply":"2022-11-09T15:54:05.105458Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hot Take 3: Top Performers are Stable, in direct Contrast with Volatile Bottom Performers\nx-axis is PercentageofRaceCompletion; y axis is marginal advantage over trailing (-1) horse. Every row is represented by one line.\n\nInterestingly, top performers do not deviate too much from their trailing horse in marginal advantage. *Incredibly interestingly, no amount of volatility (70% IntotheRace onwards, even in a positive direction) salvages bottom performers from being bottom.* This could mean that during mid-race, if at a losing position, if the horse would be competing more than once on the day, more consideration could be given to winning the next race, in the interests of injury prevention.","metadata":{}},{"cell_type":"markdown","source":"_____________________________________________________________________________________________________________________________________________","metadata":{}},{"cell_type":"markdown","source":"# The Features Predictive of Performance","metadata":{}},{"cell_type":"markdown","source":"A quick and dirty implementation of gradient boosting to find the features predictive of AdjustedFinalRankScore. (mainly to confirm, if or the extent to which having marginal advantage at which few range of PercentToCompletion actually contributes to performance)\n\nVariables for which all players in the same race share (weather, track condition) could be used to creatively engineer feature interactions and may yield interesting insights. In the interest of time and capacity constraint, this has not been done, yet.","metadata":{}},{"cell_type":"code","source":"import xgboost as xgb\nfrom xgboost import plot_importance\nfrom scipy.stats import uniform, randint\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import mean_absolute_error\nfrom sklearn.model_selection import TimeSeriesSplit, cross_val_score, GridSearchCV, RandomizedSearchCV\nfrom sklearn.metrics import mean_squared_error\nimport shap\nfrom shap import KernelExplainer, summary_plot","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:31:02.038131Z","iopub.execute_input":"2022-11-09T17:31:02.038580Z","iopub.status.idle":"2022-11-09T17:31:02.045712Z","shell.execute_reply.started":"2022-11-09T17:31:02.038541Z","shell.execute_reply":"2022-11-09T17:31:02.044342Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"dff = MarginalAdvantageOverTrailOne.merge(df2,on=\"UniqueHorseInRace\", how=\"left\").merge(df1, on=\"UniqueRace\", how=\"left\") \nfor x in dff.columns[2:12].to_list():\n    dff = dff.rename(columns={x: \"MargAdvOverTrailOne\"+str(x)+\"toCompletion\"})\n\ndff.head(2)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:38:22.169818Z","iopub.execute_input":"2022-11-09T17:38:22.170947Z","iopub.status.idle":"2022-11-09T17:38:22.274363Z","shell.execute_reply.started":"2022-11-09T17:38:22.170889Z","shell.execute_reply":"2022-11-09T17:38:22.273402Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"identifier = [\"UniqueRace\", \"UniqueHorseInRace\"]\ncontinuous = dff.columns[2:12].to_list()\ncontinuous.extend([\"JockeyScore\", \"weight_carried\", \"distance_id\", \"odds\", \"run_up_distance\", \"purse\", \"StartHourRounded\"])\ncategorical = [\"track_id\", \"course_type\", \"track_condition\", \"race_type\"]\npredict = [\"AdjustedFinalRankScore\"]\n\nonehot_df = dff[categorical]\nonehot_df = pd.get_dummies(onehot_df, columns = categorical)\nscore_onehot_drop = dff.drop(categorical, axis = 1)\nfeatures = pd.concat([score_onehot_drop, onehot_df], axis = 1)\nfeatureslist = [x for x in features.columns if x not in [\"position_at_finish\", \"AdjustedFinalRankScore\", \"UniqueRace\", \"UniqueHorseInRace\"]]\n\nX = features[featureslist]\nY = dff[predict]\n\nX_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42)\n\nparams = {\"colsample_bytree\": uniform(0.7, 0.3),\n          \"gamma\": uniform(0, 0.5),\n          \"learning_rate\": uniform(0.003, 0.3), # default 0.1 \n          \"max_depth\": randint(2, 6), # default 3\n          \"n_estimators\": randint(100, 250), # default 100\n          \"subsample\": uniform(0.6, 0.4)}\n\nxgb_model = xgb.XGBRegressor(objective=\"reg:linear\", random_state=42)\ntime_split = TimeSeriesSplit(n_splits = 8)\nxgb_search = RandomizedSearchCV(xgb_model, param_distributions=params, random_state=42, n_iter=4, cv=time_split, verbose=1, n_jobs=1, return_train_score=True)\n\nxgb_search.fit(X_train, y_train)\nmodel = xgb_search.best_estimator_\ny_pred = model.predict(X_test)\nrms = sqrt(mean_squared_error(y_test, y_pred))\nprint ('RMSE:', rms)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:38:33.381362Z","iopub.execute_input":"2022-11-09T17:38:33.381887Z","iopub.status.idle":"2022-11-09T17:39:27.650833Z","shell.execute_reply.started":"2022-11-09T17:38:33.381836Z","shell.execute_reply":"2022-11-09T17:39:27.649370Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"The label range is -100 to 0. Having a RMSE of 19 is tolerable, and translates to about 80% accuracy","metadata":{}},{"cell_type":"code","source":"plt.rcParams[\"figure.figsize\"] = (14, 14)\nxgb.plot_importance(model)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:39:27.653266Z","iopub.execute_input":"2022-11-09T17:39:27.655411Z","iopub.status.idle":"2022-11-09T17:39:28.334892Z","shell.execute_reply.started":"2022-11-09T17:39:27.655236Z","shell.execute_reply":"2022-11-09T17:39:28.333562Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"X_sampled = X_train.sample(1000, random_state=10)\nexplainer = shap.Explainer(model.predict, X_test)\nshap_values = explainer(X_test)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:26:51.875172Z","iopub.execute_input":"2022-11-09T17:26:51.876850Z","iopub.status.idle":"2022-11-09T17:31:02.034369Z","shell.execute_reply.started":"2022-11-09T17:26:51.876744Z","shell.execute_reply":"2022-11-09T17:31:02.032974Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"plt.rcParams[\"figure.figsize\"] = (14, 14)\nshap.plots.bar(shap_values)","metadata":{"execution":{"iopub.status.busy":"2022-11-09T17:40:52.157610Z","iopub.execute_input":"2022-11-09T17:40:52.158167Z","iopub.status.idle":"2022-11-09T17:40:52.418796Z","shell.execute_reply.started":"2022-11-09T17:40:52.158122Z","shell.execute_reply":"2022-11-09T17:40:52.417538Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Hot Take 4: A More Complete Exploration is Needed, but..\nIt looks like there are potentials for future explorations on how marginal advantage (the strength exerted over a dynamic direct competitor) deployed throughout the course of the race could impact performance.","metadata":{}}]}