{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Information Gain From New Features\n\nIn this notebook I add new features to the datasets provided and check whether they add any value to the statistical model in predicting the finish position of horse.\n\nThe new features added are:\n\n1. **Speculated Finish Position Of Horse(SpeculatedPosition):** This feature is derived by using column      \"odds\" from nyra_start_table.The horse with lowest value in \"odds\" column for a    unique combination of track_id,race_number,race_date gets best rank.For            instance, if a particular race has three horses A,B and C with odds 100,400,200    respectively, then the Speculated Finish Position of the three horses is 1,3,2.\n\n2. **Difference in Speculated Finisih Position and Actual Finish Position of            horse(SpecActDiff):** This feature is calculated by subtracting the actual        finish position of horse from the Speculated Finish Position. For instance,if      a horse B finished 2nd and its Speculated Finish Position was 3rd, the the        SpecActDiff is (3-2)=1.\n\n3. **Measure of horse performance (Performance):** This feature indicates whether      the horse performed as per expectation or overperformed or underperformed its      expectation. This feature is derived by using \"SpecActDiff\". If \"SpecActDiff\"      is greater than 0(Zero), it indicates that the horse performed better than        expected. If \"SpecActDiff\" is less than 0(Zero), it indiactes that the horse      finished in a position that was less than what was expected and if                \"SpecActDiff\" is 0(Zero), that means the horse performed as expected.\n\n4. **Measure of last \"position at finish\" of horse(PastPosition):** This feature      is nothing but the \"position_at_finish\" for the horse in its most recent          race.For instance if a horse B has finished, 4,5,8,3,4,2 in the past 6 races,      then its PastPsoition for the current race would be 2.\n\n5. **Measure of past performance(PastPerformance):** This feature indicates the        horse performed as expected or performed better than expected in the last          race.This feature is derived by taking the value of Performance column for        the horse in most recent race it played.\n\n6. **Measure of past difference in Speculated & Actual position at finish(PastSpecActDiff):** This feature is derived by taking the value of                                    SpecActDiff column for the horse in most recent race it played.\n\nWe first read the data provided in the competition by the organizers. We also use two external data sets , one containing the horse_id for each race in the competition and another one containing the horse name for each horse_id. \n\nWe join these datasets and derive the aforementioned new features. We then do some basic visualizations on how the features play out on different track_ids, race_types, course_types etc.\n\nIn the penultimate step we use mutual_info_regression() function from sklearn to see how much each feature contributes in predicting the finish_position of horses. More on this later.\n\nIn the final step, we suggest what other features can be added to data set and explored.","metadata":{}},{"cell_type":"markdown","source":"**Read Necessary Files**","metadata":{}},{"cell_type":"code","source":"import pandas as pd\nimport warnings\nwarnings.filterwarnings('ignore')\nstart_table_col_names = ['track_id', 'race_date', 'race_number', 'program_number', 'weight_carried', 'jockey', 'odds', 'position_at_finish']\nnyra_start = pd.read_csv('../input/big-data-derby-2022/nyra_start_table.csv', names=start_table_col_names)\nnyra_track = pd.read_csv('../input/big-data-derby-2022/nyra_tracking_table.csv')\nnyra_race = pd.read_csv('../input/big-data-derby-2022/nyra_race_table.csv')\nnyra_complete = pd.read_csv('../input/big-data-derby-2022/nyra_2019_complete.csv')\nhorse_ids=pd.read_csv(\"../input/big-data-derby-2022-global-horse-ids-and-places/horse_ids.csv\")\nhorse_names=pd.read_csv(\"../input/big-data-derby-2022-global-horse-ids-and-places/horse_names.csv\")","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:37:55.209285Z","iopub.execute_input":"2022-11-10T12:37:55.209707Z","iopub.status.idle":"2022-11-10T12:38:04.403916Z","shell.execute_reply.started":"2022-11-10T12:37:55.209676Z","shell.execute_reply":"2022-11-10T12:38:04.402315Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Join the datasets provided by organizers with the horse_ids and horse_names.**","metadata":{}},{"cell_type":"code","source":"horse_ids = horse_ids.merge(horse_names, on='horse_id', how='inner')\nhorse_ids['program_number'] = horse_ids['program_number'].apply(lambda x: x.ljust(3, ' '))\nnyra_start = nyra_start.merge(horse_ids, \n                                left_on=['track_id', 'race_date', 'race_number', 'program_number'],\n                                right_on=['track_id', 'race_date', 'race', 'program_number'],\n                                how='inner'\n                               )","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:04.405807Z","iopub.execute_input":"2022-11-10T12:38:04.406148Z","iopub.status.idle":"2022-11-10T12:38:04.627641Z","shell.execute_reply.started":"2022-11-10T12:38:04.406116Z","shell.execute_reply":"2022-11-10T12:38:04.626572Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Define a function \"get_speculated_position()\" which gets the speculated finish position for the horse for the particular race based on \"odds\".**","metadata":{}},{"cell_type":"code","source":"def get_speculated_position(track_id,race_date,race_number):\n    return list(nyra_start[(nyra_start['track_id']==track_id)&(nyra_start['race_date']==race_date)&(nyra_start['race_number']==race_number)]['odds'].rank())","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:04.628966Z","iopub.execute_input":"2022-11-10T12:38:04.629300Z","iopub.status.idle":"2022-11-10T12:38:04.634610Z","shell.execute_reply.started":"2022-11-10T12:38:04.629270Z","shell.execute_reply":"2022-11-10T12:38:04.633716Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Calculate the speculated finish position of all horses for all races.**","metadata":{}},{"cell_type":"code","source":"unique_race=nyra_start.groupby(['track_id','race_date','race_number'])['horse_id'].count()\nunique_race=unique_race.reset_index()\ndel unique_race['horse_id']\nlst_speculated_position=[]\nfor i in unique_race.itertuples():\n    lst_speculated_position.extend(get_speculated_position(i[1],i[2],i[3]))\nnyra_start['SpeculatedPosition']=lst_speculated_position\ndel nyra_start['Unnamed: 0_x'],nyra_start['Unnamed: 0_y']\n","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:04.637080Z","iopub.execute_input":"2022-11-10T12:38:04.637760Z","iopub.status.idle":"2022-11-10T12:38:10.079623Z","shell.execute_reply.started":"2022-11-10T12:38:04.637721Z","shell.execute_reply":"2022-11-10T12:38:10.078072Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Calculate the difference in actual and speculated finish position of all horses for all races.**","metadata":{}},{"cell_type":"code","source":"nyra_start['SpecActDiff']=nyra_start['SpeculatedPosition']-nyra_start['position_at_finish']","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.081033Z","iopub.execute_input":"2022-11-10T12:38:10.081356Z","iopub.status.idle":"2022-11-10T12:38:10.090790Z","shell.execute_reply.started":"2022-11-10T12:38:10.081327Z","shell.execute_reply":"2022-11-10T12:38:10.089099Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Define a function \"performance()\" to get whether a horse overperformed or underperformed.**","metadata":{}},{"cell_type":"code","source":"def performance(x):\n    if x>0:\n        return 1\n    elif x<0:\n        return -1\n    else:\n        return 0\nnyra_start['Performance']=nyra_start['SpecActDiff'].apply(lambda x:performance(x))","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.092608Z","iopub.execute_input":"2022-11-10T12:38:10.092929Z","iopub.status.idle":"2022-11-10T12:38:10.115150Z","shell.execute_reply.started":"2022-11-10T12:38:10.092900Z","shell.execute_reply":"2022-11-10T12:38:10.112932Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"TotalRacesByHorse=nyra_start['horse_id'].value_counts()\nTotalRacesByHorse=TotalRacesByHorse.reset_index()#nyra_start.groupby(['horse_id','horse_name']).count()\nTotalRacesByHorse.columns=['horse_id','TotalRaces']\nTotalRacesByHorse=TotalRacesByHorse.merge(horse_names, on='horse_id', how='inner')\ndel TotalRacesByHorse['Unnamed: 0']\nTotalRacesByHorse=TotalRacesByHorse[['horse_id','horse_name','TotalRaces']]\n#TotalRacesByHorse.head()","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.116947Z","iopub.execute_input":"2022-11-10T12:38:10.117303Z","iopub.status.idle":"2022-11-10T12:38:10.137279Z","shell.execute_reply.started":"2022-11-10T12:38:10.117273Z","shell.execute_reply":"2022-11-10T12:38:10.135186Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Top 10 horses that have underperformed its expectation**\n\nThe column \"UnderPerformanceCount\" shows number of races in which the horse underpeformed its expectation and \"TotalRaces\" indicates total races it participated in.","metadata":{}},{"cell_type":"code","source":"##Top 10 horses that have underperformed its expectation\nunder_performer=nyra_start[nyra_start['Performance']==-1]['horse_id'].value_counts()\nunder_performer=under_performer.reset_index()\nunder_performer.columns=['horse_id','UnderPerformanceCount']\nunder_performer=under_performer.merge(TotalRacesByHorse,on='horse_id',how='outer')\nunder_performer['UnderPerformanceCount'].fillna(0,inplace=True)\nunder_performer['PercentageUnderPerformance']=under_performer['UnderPerformanceCount']/under_performer['TotalRaces']*100\nunder_performer.sort_values('PercentageUnderPerformance',ascending=False).head(10)","metadata":{"_kg_hide-input":true,"_kg_hide-output":false,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.139433Z","iopub.execute_input":"2022-11-10T12:38:10.139910Z","iopub.status.idle":"2022-11-10T12:38:10.169364Z","shell.execute_reply.started":"2022-11-10T12:38:10.139862Z","shell.execute_reply":"2022-11-10T12:38:10.167732Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Top 10 horses that have outperformed its expectation**\n\nThe column \"OutPerformanceCount\" shows number of races in which the horse outpeformed its expectation and \"TotalRaces\" indicates total races it participated in.","metadata":{}},{"cell_type":"code","source":"##Top 10 horses that have outperformed its expectation\nout_performer=nyra_start[nyra_start['Performance']==1]['horse_id'].value_counts()\nout_performer=out_performer.reset_index()\nout_performer.columns=['horse_id','OutPerformanceCount']\nout_performer=out_performer.merge(TotalRacesByHorse,on='horse_id',how='outer')\nout_performer['OutPerformanceCount'].fillna(0,inplace=True)\nout_performer['PercentageOutPerformance']=out_performer['OutPerformanceCount']/out_performer['TotalRaces']*100\nout_performer.sort_values('PercentageOutPerformance',ascending=False).head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.172934Z","iopub.execute_input":"2022-11-10T12:38:10.173359Z","iopub.status.idle":"2022-11-10T12:38:10.204516Z","shell.execute_reply.started":"2022-11-10T12:38:10.173316Z","shell.execute_reply":"2022-11-10T12:38:10.203129Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**Top 10 horses that have performed as per its expectation**\n\nThe column \"PerformanceCount\" shows number of races in which the horse peformed its expectation and \"TotalRaces\" indicates total races it participated in.","metadata":{}},{"cell_type":"code","source":"##Top 10 horses that have performed as per expectation\nperformer=nyra_start[nyra_start['Performance']==0]['horse_id'].value_counts()\nperformer=performer.reset_index()\nperformer.columns=['horse_id','PerformanceCount']\nperformer=performer.merge(TotalRacesByHorse,on='horse_id',how='outer')\nperformer['PerformanceCount'].fillna(0,inplace=True)\nperformer['PercentagetPerformance']=performer['PerformanceCount']/performer['TotalRaces']*100\nperformer.sort_values('PercentagetPerformance',ascending=False).head(10)","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.207880Z","iopub.execute_input":"2022-11-10T12:38:10.208236Z","iopub.status.idle":"2022-11-10T12:38:10.236496Z","shell.execute_reply.started":"2022-11-10T12:38:10.208204Z","shell.execute_reply":"2022-11-10T12:38:10.235378Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now, we group the performance on various other features and see how they look. To begin with, we see what percentage of horses outperformed their expectation on different track_ids. As we can see from the barplot below, almost 40% of horses have outperformed on each track_id.","metadata":{}},{"cell_type":"code","source":"kdf=nyra_start[nyra_start['Performance']==1]['track_id'].value_counts()/nyra_start['track_id'].value_counts()*100\n_=kdf.plot.bar(title='%Outperformance for Each Track Id')","metadata":{"_kg_hide-input":true,"execution":{"iopub.status.busy":"2022-11-10T12:38:10.237457Z","iopub.execute_input":"2022-11-10T12:38:10.237757Z","iopub.status.idle":"2022-11-10T12:38:10.400921Z","shell.execute_reply.started":"2022-11-10T12:38:10.237730Z","shell.execute_reply":"2022-11-10T12:38:10.399231Z"},"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the barplot below, almost 40% of horses have underperformed on each track_id.","metadata":{}},{"cell_type":"code","source":"kdf=nyra_start[nyra_start['Performance']==-1]['track_id'].value_counts()/nyra_start['track_id'].value_counts()*100\n_=kdf.plot.bar(title='%Underperformance for Each Track Id')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:10.402324Z","iopub.execute_input":"2022-11-10T12:38:10.402629Z","iopub.status.idle":"2022-11-10T12:38:10.546952Z","shell.execute_reply.started":"2022-11-10T12:38:10.402602Z","shell.execute_reply":"2022-11-10T12:38:10.546220Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from the barplot below, almost 20% of horses have performed as expected on each track_id","metadata":{}},{"cell_type":"code","source":"kdf=nyra_start[nyra_start['Performance']==0]['track_id'].value_counts()/nyra_start['track_id'].value_counts()*100\n_=kdf.plot.bar(title='%Performance for Each Track Id')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:10.548036Z","iopub.execute_input":"2022-11-10T12:38:10.548448Z","iopub.status.idle":"2022-11-10T12:38:10.676765Z","shell.execute_reply.started":"2022-11-10T12:38:10.548420Z","shell.execute_reply":"2022-11-10T12:38:10.675983Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we see similar stats for each course type.","metadata":{}},{"cell_type":"code","source":"## Next We Will Check The Under Performance/ Out Performance of horse with Respect to Course_type\ndf_complete=nyra_start.merge(nyra_race,on=['track_id','race_date','race_number'])\n##% Underperformance by course type\nkdf=df_complete[df_complete['Performance']==-1]['course_type'].value_counts()/df_complete['course_type'].value_counts()*100\n_=kdf.plot.bar(title='%Underperformance for Each Course Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:10.677875Z","iopub.execute_input":"2022-11-10T12:38:10.678327Z","iopub.status.idle":"2022-11-10T12:38:10.855828Z","shell.execute_reply.started":"2022-11-10T12:38:10.678297Z","shell.execute_reply":"2022-11-10T12:38:10.854401Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##% Outperformance by course type\nkdf=df_complete[df_complete['Performance']==1]['course_type'].value_counts()/df_complete['course_type'].value_counts()*100\n_=kdf.plot.bar(title='%Outperformance for Each Course Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:10.857145Z","iopub.execute_input":"2022-11-10T12:38:10.857447Z","iopub.status.idle":"2022-11-10T12:38:11.009390Z","shell.execute_reply.started":"2022-11-10T12:38:10.857419Z","shell.execute_reply":"2022-11-10T12:38:11.008650Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##% Performance by course type\nkdf=df_complete[df_complete['Performance']==0]['course_type'].value_counts()/df_complete['course_type'].value_counts()*100\n_=kdf.plot.bar(title='%Performance for Each Course Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.010558Z","iopub.execute_input":"2022-11-10T12:38:11.011194Z","iopub.status.idle":"2022-11-10T12:38:11.150194Z","shell.execute_reply.started":"2022-11-10T12:38:11.011165Z","shell.execute_reply":"2022-11-10T12:38:11.149364Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we see similar stats for each track condition.","metadata":{}},{"cell_type":"code","source":"##% Underperformance by track_condition\nkdf=df_complete[df_complete['Performance']==-1]['track_condition'].value_counts()/df_complete['track_condition'].value_counts()*100\n_=kdf.plot.bar(title='%UnderPerformance for Each Track Condition')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.151459Z","iopub.execute_input":"2022-11-10T12:38:11.152140Z","iopub.status.idle":"2022-11-10T12:38:11.311588Z","shell.execute_reply.started":"2022-11-10T12:38:11.152103Z","shell.execute_reply":"2022-11-10T12:38:11.310657Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##% Outperformance by track_condition\nkdf=df_complete[df_complete['Performance']==1]['track_condition'].value_counts()/df_complete['track_condition'].value_counts()*100\n_=kdf.plot.bar(title='%OutPerformance for Each Track Condition')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.313832Z","iopub.execute_input":"2022-11-10T12:38:11.314479Z","iopub.status.idle":"2022-11-10T12:38:11.478538Z","shell.execute_reply.started":"2022-11-10T12:38:11.314444Z","shell.execute_reply":"2022-11-10T12:38:11.477337Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##% Performance by track_condition\nkdf=df_complete[df_complete['Performance']==0]['track_condition'].value_counts()/df_complete['track_condition'].value_counts()*100\n_=kdf.plot.bar(title='%Performance for Each Track Condition')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.480039Z","iopub.execute_input":"2022-11-10T12:38:11.480319Z","iopub.status.idle":"2022-11-10T12:38:11.644254Z","shell.execute_reply.started":"2022-11-10T12:38:11.480294Z","shell.execute_reply":"2022-11-10T12:38:11.642883Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Now we see similar stats for each race type. Almost 66% of the time horses underperform when race_type is Waiver Maiden Claiming(WMC)","metadata":{}},{"cell_type":"code","source":"kdf=df_complete[df_complete['Performance']==-1]['race_type'].value_counts()/df_complete['race_type'].value_counts()*100\n_=kdf.plot.bar(title='%UnderPerformance for Each Race Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.646731Z","iopub.execute_input":"2022-11-10T12:38:11.647195Z","iopub.status.idle":"2022-11-10T12:38:11.844526Z","shell.execute_reply.started":"2022-11-10T12:38:11.647156Z","shell.execute_reply":"2022-11-10T12:38:11.843838Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"kdf=df_complete[df_complete['Performance']==1]['race_type'].value_counts()/df_complete['race_type'].value_counts()*100\n_=kdf.plot.bar(title='%OutPerformance for Each Race Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:11.845638Z","iopub.execute_input":"2022-11-10T12:38:11.846074Z","iopub.status.idle":"2022-11-10T12:38:12.039700Z","shell.execute_reply.started":"2022-11-10T12:38:11.846039Z","shell.execute_reply":"2022-11-10T12:38:12.038933Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"##% Performance by track_condition\nkdf=df_complete[df_complete['Performance']==0]['race_type'].value_counts()/df_complete['race_type'].value_counts()*100\n_=kdf.plot.bar(title='%Performance for Each Race Type')","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:12.040912Z","iopub.execute_input":"2022-11-10T12:38:12.041223Z","iopub.status.idle":"2022-11-10T12:38:12.242381Z","shell.execute_reply.started":"2022-11-10T12:38:12.041194Z","shell.execute_reply":"2022-11-10T12:38:12.240910Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"Mutual information between two features is a non-negative value, which measures the dependency between the features and target. It is equal to zero if and only if two feature and target variables are independent, and higher values mean higher dependency.\n\nIn short, higher the mutual information value, higher is the possibility that the feature is good at predicting the target.\n\nIn our case, we will use this to see which of the new features and the features already provided by the organizers are good at predicting the finish position of horses.","metadata":{}},{"cell_type":"code","source":"from sklearn.feature_selection import mutual_info_classif\n\ndef make_mi_scores(X, y):\n    mi_scores = mutual_info_classif(X, y,random_state=42)\n    mi_scores = pd.Series(mi_scores, name=\"MI Scores\", index=X.columns)\n    mi_scores = mi_scores.sort_values(ascending=False)\n    return mi_scores","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:12.244051Z","iopub.execute_input":"2022-11-10T12:38:12.244430Z","iopub.status.idle":"2022-11-10T12:38:12.250181Z","shell.execute_reply.started":"2022-11-10T12:38:12.244397Z","shell.execute_reply":"2022-11-10T12:38:12.249076Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"We first run the mutual_info_classifier from sklearn, that helps us estimate these values on the original set of features provided by organizers.","metadata":{}},{"cell_type":"code","source":"from sklearn.preprocessing import LabelEncoder\ncols = ['track_id','race_date', 'jockey', 'program_number','horse_name', 'course_type', 'track_condition', 'race_type']\n\nfor col in cols:\n    le = LabelEncoder()\n    df_complete[col] = le.fit_transform(df_complete[col].astype('str'))\n    \ndf_complete\nX_orignal=df_complete[['track_id', 'race_date', 'race_number',\n       'weight_carried', 'jockey', 'odds', 'race', 'distance_id', 'course_type',\n       'track_condition', 'run_up_distance', 'race_type', 'purse',\n       'post_time']]\ny=df_complete['position_at_finish']\nmi_scores = make_mi_scores(X_orignal, y)\nmi_scores\n","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:12.251704Z","iopub.execute_input":"2022-11-10T12:38:12.251997Z","iopub.status.idle":"2022-11-10T12:38:13.097633Z","shell.execute_reply.started":"2022-11-10T12:38:12.251970Z","shell.execute_reply":"2022-11-10T12:38:13.096708Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"**\"odds\",\"jockey\",\"race_number\",\"track_condition\",\"race\",\"course_type\",\"track_id\" and \"purse\"** contribute positively in detemining the finish position of horses.","metadata":{}},{"cell_type":"markdown","source":"We now run the mutual_info_classifier from sklearn, that helps us estimate these values on the original set of features provided by organizers and the features that we have added.","metadata":{}},{"cell_type":"code","source":"horse_list=list(df_complete['horse_id'].unique())\nsdf=pd.DataFrame()\nfor i in horse_list:\n    tdf=df_complete[df_complete['horse_id']==i]\n    tdf['PastPosition']=tdf['position_at_finish'].shift(1)\n    tdf['PastPerformance']=tdf['Performance'].shift(1)\n    tdf['PastSpecActDiff']=tdf['SpecActDiff'].shift(1)\n    sdf=sdf.append(tdf)\nX=sdf[['track_id', 'race_date', 'race_number',\n       'weight_carried', 'jockey', 'odds', 'race','SpeculatedPosition',\n       'horse_id', 'distance_id', 'course_type','PastPosition','PastPerformance','PastSpecActDiff',\n       'track_condition', 'run_up_distance', 'race_type', 'purse',\n       'post_time']]\ny=sdf['position_at_finish']\nmi_scores = make_mi_scores(X.fillna(0), y)\nmi_scores","metadata":{"execution":{"iopub.status.busy":"2022-11-10T12:38:13.098984Z","iopub.execute_input":"2022-11-10T12:38:13.099287Z","iopub.status.idle":"2022-11-10T12:38:26.812628Z","shell.execute_reply.started":"2022-11-10T12:38:13.099261Z","shell.execute_reply":"2022-11-10T12:38:26.811174Z"},"_kg_hide-input":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"As we can see from above scores, **\"SpeculatedPosition\",\n\"odds\",\n\"jockey\",\n\"PastPosition\",\n\"race\",\n\"weight_carried\",\n\"track_condition\",\n\"course_type\",\n\"race_number\",\n\"PastSpecActDiff\",\n\"PastPerformance\",\n\"track_id\",\n\"horse_id\",\n\"race_type\",\n\"purse\",\n\"distance_id\",**\n add value in predicting the finish position of horses.\n","metadata":{}},{"cell_type":"markdown","source":"> **So, we safely conclude that the features we have added contribute positively in predicting the finish position of horses.**","metadata":{}},{"cell_type":"markdown","source":"# Scope for further improvement","metadata":{}},{"cell_type":"markdown","source":"1. We can add features pertaining to weather condition of the place on the            particular day and see how the horse performance varies using the same            procedure detailed above.\n\n2. Even though we have seen that adding new features contributes positively in        predicting the final standing of horses, we have not built any model around the    same. Experimenting with different models will futher help in validating our      findings.","metadata":{}},{"cell_type":"markdown","source":"Note: The horse_id and horse_name data was taken fro following link:\nhttps://www.kaggle.com/datasets/themarkgreen/big-data-derby-2022-global-horse-ids-and-places , provided by Mark Green.","metadata":{}}]}