{"cells":[{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","collapsed":true,"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":false},"cell_type":"markdown","source":"# About\nThis notebook's about making predictions. Exploratory Data Analysis is nice and important, but 1) that horse's been beaten a few times and 2) in my opinion, developing models is a lot harder, more interesting, and in the real world typically more lucrative than just analyzing data. \n\nIn this notebook I hope to\n\n  1) Illustrate a robust data cleaning *pipeline* and its value  \n  2) Develop a simple validation strategy with scores that generalize to unseen data  \n  3) Train a neural network, gradient boosting model, and logistic regression model, and show how ensembling them provides extra lift"},{"metadata":{},"cell_type":"markdown","source":"### Why the name Professor?\nI like giving my projects names that give them life, so I tend to name things after people, pets, and places local to me. Professor is the name of my dog :)\n\n[@professor.walt](https://www.instagram.com/professor.walt/)\n![professor](https://i.imgur.com/Dx0mr1Cm.png)"},{"metadata":{},"cell_type":"markdown","source":"# Imports"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true},"cell_type":"code","source":"import numpy as np\nimport pandas as pd\n\nfrom tensorflow.keras.models import Sequential\nfrom tensorflow.keras.layers import Dense, Dropout, BatchNormalization, Activation\nfrom tensorflow.keras.callbacks import EarlyStopping\n\nfrom sklearn.metrics import log_loss\nfrom sklearn.linear_model import LogisticRegression\n\nimport lightgbm as lgb\n\nimport matplotlib.pyplot as plt","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Helper Functions\n---\n\nHelper functions are key to a good data wrangling pipeline. Typically I'd store these into a *helpers_generic.py* file and a *helpers_project.py* file where:\n\n- *helpers_generic.py* stores **really** generic functions that you'd use across different projects (e.g. *log_loss()*)\n- *helpers_project.py* stores functions that you'd only use for the current project (e.g. *rank_teams()*)\n\nBelow I present three generic helper functions which are critial to my data wrangling pipeline. I've tried to explain them in the docstrings, but the gist is\n\n1. *join_insertion(into_df, from_df, ...)*  \n    this is useful if you want to insert values from a dataframe into another dataframe via some matching columns  \n    \n2. *dataframe_cj(df1, df2)*  \n    performs a cartesian join between two dataframes. So if df1 has n rows and df2 has k rows, the result has n*k rows\n    \n3. *array_cj(arrays, out=None)*  \n    performs a cartesian join between a sequence of numpy 1d arrays"},{"metadata":{"trusted":true},"cell_type":"code","source":"def join_insertion(into_df, from_df, cols, on, by=None, direction=None, mult='error'):\n    \"\"\"\n    Suppose A and B are dataframes. A has columns {foo, bar, baz} and B has columns {foo, baz, buz}\n    This function allows you to do an operation like:\n    \"where A and B match via the column foo, insert the values of baz and buz from B into A\"\n    Note that this'll update A's values for baz and it'll insert buz as a new column.\n    This is a lot like DataFrame.update(), but that method annoyingly ignores NaN values in B!\n\n    Optionally, direction can be given as 'backward', 'forward', or nearest to implement a rolling join\n    insertion. forward means 'roll into_df values forward to match from_df values', etc. Additionally,\n    when doing a rolling join, 'on' should be the roll column and 'by' should be the exact-match columns.\n    See pandas.merge_asof() for details.\n\n    Note that 'mult' gets ignored when doing a rolling join. In the case of a rolling join, the first\n    appearing record is kept, even if two records match a key from the same distance. Perhaps this\n    can be improved...\n\n    :param into_df: dataframe you want to modify\n    :param from_df: dataframe with the values you want to insert\n    :param cols: list of column names (values to insert)\n    :param on: list of column names (values to join on), or a dict of {into:from} column name pairs\n    :param by: same format as on; when doing a rolling join insertion, what columns to exact-match on\n    :param direction: 'forward', 'backward', or 'nearest'. forward means roll into_df values to match from_df\n    :param mult: if a key of into_df matches multiple rows of from_df, how should this be handled?\n    an error can be raised, or the first matching value can be inserted, or the last matching value\n    can be inserted\n    :return: a modified copy of into_df, with updated values using from_df\n    \"\"\"\n\n    # Infer left_on, right_on\n    if (isinstance(on, dict)):\n        left_on = list(on.keys())\n        right_on = list(on.values())\n    elif(isinstance(on, list)):\n        left_on = on\n        right_on = on\n    elif(isinstance(on, str)):\n        left_on = [on]\n        right_on = [on]\n    else:\n        raise Exception(\"on should be a list or dictionary\")\n\n    # Infer left_by, right_by\n    if(by is not None):\n        if (isinstance(by, dict)):\n            left_by = list(by.keys())\n            right_by = list(by.values())\n        elif (isinstance(by, list)):\n            left_by = by\n            right_by = by\n        elif (isinstance(by, str)):\n            left_by = [by]\n            right_by = [by]\n        else:\n            raise Exception(\"by should be a list or dictionary\")\n    else:\n        left_by = None\n        right_by = None\n\n    # Make cols a list if it isn't already\n    if(isinstance(cols, str)):\n        cols = [cols]\n\n    # Setup\n    A = into_df.copy()\n    B = from_df[right_on + cols + ([] if right_by is None else right_by)].copy()\n\n    # Insert row ids\n    A['_A_RowId_'] = np.arange(A.shape[0])\n    B['_B_RowId_'] = np.arange(B.shape[0])\n\n    # Merge\n    if(direction is None):\n        A = pd.merge(\n            left=A,\n            right=B,\n            how='left',\n            left_on=left_on,\n            right_on=right_on,\n            suffixes=(None, '_y'),\n            indicator=True\n        ).sort_values(['_A_RowId_', '_B_RowId_'])\n\n        # Check for rows of A which got duplicated by the merge, and then handle appropriately\n        if (mult == 'error'):\n            if (A.groupby('_A_RowId_').size().max() > 1):\n                raise Exception(\"At least one key of into_df matched multiple rows of from_df.\")\n        elif (mult == 'first'):\n            A = A.groupby('_A_RowId_').first().reset_index()\n        elif (mult == 'last'):\n            A = A.groupby('_A_RowId_').last().reset_index()\n\n    else:\n        A.sort_values(left_on, inplace=True)\n        B.sort_values(right_on, inplace=True)\n        A = pd.merge_asof(\n            left=A,\n            right=B,\n            direction=direction,\n            left_on=left_on,\n            right_on=right_on,\n            left_by=left_by,\n            right_by=right_by,\n            suffixes=(None, '_y')\n        ).sort_values(['_A_RowId_', '_B_RowId_'])\n\n    # Insert values from new column(s) into pre-existing column(s)\n    mask = A._merge == 'both' if direction is None else np.repeat(True, A.shape[0])\n    cols_in_both = list(set(into_df.columns.to_list()).intersection(set(cols)))\n    for col in cols_in_both:\n        A.loc[mask, col] = A.loc[mask, col + '_y']\n\n    # Drop unwanted columns\n    A.drop(columns=list(set(A.columns).difference(set(into_df.columns.to_list() + cols))), inplace=True)\n\n    return A\n\ndef dataframe_cj(df1, df2):\n    \"\"\"\n    given two dataframes, compute their cartesian product\n\n    :param df1: dataframe\n    :param df2: dataframe\n    :return: cartesian product of input dataframes\n    \"\"\"\n\n    df1copy = df1.copy()\n    df2copy = df2.copy()\n    df1copy['_Temp_'] = 1\n    df2copy['_Temp_'] = 1\n    result = pd.merge(df1copy, df2copy, on='_Temp_').drop(columns = '_Temp_')\n\n    return result\n\ndef array_cj(arrays, out=None):\n    \"\"\"\n    Generate a cartesian product of input arrays.\n\n    Parameters\n    ----------\n    arrays : list of array-like\n        1-D arrays to form the cartesian product of.\n    out : ndarray\n        Array to place the cartesian product in.\n\n    Returns\n    -------\n    out : ndarray\n        2-D array of shape (M, len(arrays)) containing cartesian products\n        formed of input arrays.\n\n    Examples\n    --------\n    cj(([1, 2, 3], [4, 5], [6, 7]))\n    array([[1, 4, 6],\n           [1, 4, 7],\n           [1, 5, 6],\n           [1, 5, 7],\n           [2, 4, 6],\n           [2, 4, 7],\n           [2, 5, 6],\n           [2, 5, 7],\n           [3, 4, 6],\n           [3, 4, 7],\n           [3, 5, 6],\n           [3, 5, 7]])\n\n    \"\"\"\n\n    arrays = [np.asarray(x) for x in arrays]\n    dtype = arrays[0].dtype\n\n    n = np.prod([x.size for x in arrays])\n    if out is None:\n        out = np.zeros([n, len(arrays)], dtype=dtype)\n\n    m = int(n / arrays[0].size)\n    out[:,0] = np.repeat(arrays[0], m)\n    if arrays[1:]:\n        array_cj(arrays[1:], out=out[0:m,1:])\n        for j in range(1, arrays[0].size):\n            out[j*m:(j+1)*m,1:] = out[0:m,1:]\n    return out","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Load Data\nHere we load the stage-1 input files. Nothing special."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Load the data\nrsgames = pd.read_csv(\"../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MRegularSeasonCompactResults.csv\")\nseasons = pd.read_csv(\"../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MSeasons.csv\")\nteams = pd.read_csv(\"../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MTeams.csv\")\ntrnygames = pd.read_csv(\"../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneyCompactResults.csv\")\ntrnyseeds = pd.read_csv(\"../input/google-cloud-ncaa-march-madness-2020-division-1-mens-tournament/MDataFiles_Stage1/MNCAATourneySeeds.csv\")","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"***Pro Tip*******  \nLoad data objects in alphabetical order so its easier to see if something's missing or being loaded twice."},{"metadata":{},"cell_type":"markdown","source":"# Wrangle\n---\n\nNOT SHOWN HERE is me inspecting the input data, understanding its format, and validating certain assumptions about it.\n\nTrying to build a model directly from those input files is impractical. Our lives will be a lot easier if we reorganize the data into a more useful format. In practice, I whip out a pen and a notebook and draw the ideal datasets I want to work with, and then I make it happen.\n\nMy argument is that the following reoganized datasets would be a lot easier to work with:\n\n1. **games**  \nOne row per game. Includes columns: {Season, DayNum, Round, Team1ID, Team2ID, Team1Score, Team2Score, Team1Won} where Team1ID < Team2ID and Round = -1 for regular season, 0 for play-in games, 1 for 1st round of the tournament, ... 6 for championship game\n\n2. **gameteams**  \nOne row per (game, team). So, one game will correspond to two (game, team)s. Includes columns: {Season, DayNum, Round, TeamID, OppID, Score, OppScore, Won}\n\n3. **seasonteams**  \nOne row per (season, team). Includes columns: {Season, DayNum, Round, TeamID, Games, Wins, PPG (Points Per Game)} where stats like Games and Wins are based on regular-season play\n\nAdditionally, we'll improve some of the input datasets like **trnyseeds** and **trnygames**.\n"},{"metadata":{},"cell_type":"markdown","source":"### rd_pairs"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Determine all possible (Region1, Region2, SeedNum1, SeedNum2, Round)s excluding play-in games\n# The purpose of this is to develop a key such that, for any tournament game, if we know the Region and Seed\n# of the teams playing, we can lookup the Round (-1 = regular season, 0 = play-in, 1 = 1st round, 2 = 2nd round, ... 6 = championship)\n\nrd1_seeds = [np.array([i]) for i in [1, 16, 8, 9, 5, 12, 4, 13, 6, 11, 3, 14, 7, 10, 2, 15]]\nrd2_seeds = [np.concatenate((rd1_seeds[i], rd1_seeds[i + 1])) for i in range(0, len(rd1_seeds), 2)]\nrd3_seeds = [np.concatenate((rd2_seeds[i], rd2_seeds[i + 1])) for i in range(0, len(rd2_seeds), 2)]\nrd4_seeds = [np.concatenate((rd3_seeds[i], rd3_seeds[i + 1])) for i in range(0, len(rd3_seeds), 2)]\nrd1_matchups = [array_cj((rd1_seeds[i], rd1_seeds[i+1])) for i in range(0, len(rd1_seeds), 2)]\nrd2_matchups = [array_cj((rd2_seeds[i], rd2_seeds[i+1])) for i in range(0, len(rd2_seeds), 2)]\nrd3_matchups = [array_cj((rd3_seeds[i], rd3_seeds[i+1])) for i in range(0, len(rd3_seeds), 2)]\nrd4_matchups = [array_cj((rd4_seeds[i], rd4_seeds[i+1])) for i in range(0, len(rd4_seeds), 2)]\nrd1_pairs = pd.DataFrame(np.concatenate(rd1_matchups), columns = ['SeedNum1', 'SeedNum2'])\nrd2_pairs = pd.DataFrame(np.concatenate(rd2_matchups), columns = ['SeedNum1', 'SeedNum2'])\nrd3_pairs = pd.DataFrame(np.concatenate(rd3_matchups), columns = ['SeedNum1', 'SeedNum2'])\nrd4_pairs = pd.DataFrame(np.concatenate(rd4_matchups), columns = ['SeedNum1', 'SeedNum2'])\nrd1_pairs['Round'] = 1\nrd2_pairs['Round'] = 2\nrd3_pairs['Round'] = 3\nrd4_pairs['Round'] = 4\nrd1234_pairs_forward = pd.concat((rd1_pairs, rd2_pairs, rd3_pairs, rd4_pairs), sort=True)\nrd1234_pairs_reverse = rd1234_pairs_forward.rename(columns={'SeedNum2':'SeedNum1', 'SeedNum1':'SeedNum2'})\nrd1234_pairs = pd.concat((rd1234_pairs_forward, rd1234_pairs_reverse), sort=True)\nrd1234_pairs = dataframe_cj(rd1234_pairs, pd.DataFrame({'Region1':['W','X','Y','Z'], 'Region2':['W','X','Y','Z']}))\nrd1234_pairs = rd1234_pairs[['Region1', 'Region2', 'SeedNum1', 'SeedNum2', 'Round']]\nseed_combos = dataframe_cj(pd.DataFrame({'SeedNum1':np.arange(16) + 1}), pd.DataFrame({'SeedNum2':np.arange(16) + 1}))\nrd5_region_combos = pd.DataFrame({'Region1':['W','X','Y','Z'], 'Region2':['X','W','Z','Y']})\nrd5_pairs = dataframe_cj(seed_combos, rd5_region_combos)[['Region1', 'Region2', 'SeedNum1', 'SeedNum2']]\nrd5_pairs['Round'] = 5\nrd6_region_combos = pd.DataFrame({'Region1':['W','W','X','X','Y','Y','Z','Z'], 'Region2':['Y','Z','Y','Z','W','X','W','X']})\nrd6_pairs = dataframe_cj(seed_combos, rd6_region_combos)[['Region1', 'Region2', 'SeedNum1', 'SeedNum2']]\nrd6_pairs['Round'] = 6\nrd_pairs = pd.concat((rd1234_pairs, rd5_pairs, rd6_pairs), sort=True)\n\n# Print snippet\nrd_pairs.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### trnyseeds\n\n***Pro Tip***  \nUse regular expressions to extract information from strings. It's way more reliable than extracting substrings by character *position*. For a good tutorial on regular expressions, check out [RegexOne](https://regexone.com/)."},{"metadata":{"trusted":true},"cell_type":"code","source":"trnyseeds['SeedNum'] = trnyseeds.Seed.str.extract(r'(\\d+)').astype('int64')\ntrnyseeds['Region'] = trnyseeds.Seed.str.extract(r'(^[A-Z])')\ntrnyseeds['PlayIn'] = trnyseeds.Seed.str.extract(r'([a-z]$)')\n\n# Print snippet\ntrnyseeds.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### games"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Determine Team1, Team2\nrsgames['Team1ID'] = np.where(rsgames.WTeamID.to_numpy() < rsgames.LTeamID.to_numpy(), rsgames.WTeamID.to_numpy(), rsgames.LTeamID.to_numpy())\nrsgames['Team2ID'] = np.where(rsgames.WTeamID.to_numpy() > rsgames.LTeamID.to_numpy(), rsgames.WTeamID.to_numpy(), rsgames.LTeamID.to_numpy())\ntrnygames['Team1ID'] = np.where(trnygames.WTeamID.to_numpy() < trnygames.LTeamID.to_numpy(), trnygames.WTeamID.to_numpy(), trnygames.LTeamID.to_numpy())\ntrnygames['Team2ID'] = np.where(trnygames.WTeamID.to_numpy() > trnygames.LTeamID.to_numpy(), trnygames.WTeamID.to_numpy(), trnygames.LTeamID.to_numpy())\n\n# Insert Seed into trnygames\ntrnygames = join_insertion(into_df=trnygames, from_df=trnyseeds, on={'Season':'Season', 'Team1ID':'TeamID'}, cols=['SeedNum', 'Region', 'PlayIn'])\ntrnygames.rename(columns={'SeedNum':'SeedNum1', 'Region':'Region1', 'PlayIn':'PlayIn1'}, inplace=True)\ntrnygames = join_insertion(into_df=trnygames, from_df=trnyseeds, on={'Season':'Season', 'Team2ID':'TeamID'}, cols=['SeedNum', 'Region', 'PlayIn'])\ntrnygames.rename(columns={'SeedNum':'SeedNum2', 'Region':'Region2', 'PlayIn':'PlayIn2'}, inplace=True)\n\n# Insert Round\ntrnygames = join_insertion(\n    into_df=trnygames,\n    from_df=rd_pairs,\n    on=['Region1', 'Region2', 'SeedNum1', 'SeedNum2'],\n    cols='Round'\n)\ntrnygames.loc[(trnygames.Region1 == trnygames.Region2) & (trnygames.SeedNum1 == trnygames.SeedNum2), 'Round'] = 0\nrsgames['Round'] = -1\n\n# Make games\ntrnygames['Team1Won'] = np.where(trnygames.Team1ID == trnygames.WTeamID, True, False)\nrsgames['Team1Won'] = np.where(rsgames.Team1ID == rsgames.WTeamID, True, False)\ntrnygames['Team1Score'] = np.where(trnygames.Team1ID == trnygames.WTeamID, trnygames.WScore, trnygames.LScore)\nrsgames['Team1Score'] = np.where(rsgames.Team1ID == rsgames.WTeamID, rsgames.WScore, rsgames.LScore)\ntrnygames['Team2Score'] = np.where(trnygames.Team2ID == trnygames.WTeamID, trnygames.WScore, trnygames.LScore)\nrsgames['Team2Score'] = np.where(rsgames.Team2ID == rsgames.WTeamID, rsgames.WScore, rsgames.LScore)\ngames = pd.concat((rsgames, trnygames[rsgames.columns]), axis=0)\ngames['Team1MOV'] = games.Team1Score - games.Team2Score\n\n# Print snippet\ngames.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### gameteams"},{"metadata":{"trusted":true},"cell_type":"code","source":"gameteams1 = games.rename(columns={\n    'Team1ID':'TeamID',\n    'Team1Score':'Score',\n    'Team2ID':'OppID',\n    'Team2Score':'OppScore'\n}).drop(columns=['Team1Won', 'WTeamID', 'WScore', 'LTeamID', 'LScore', 'WLoc'])\ngameteams2 = games.rename(columns={\n    'Team2ID':'TeamID',\n    'Team2Score':'Score',\n    'Team1ID':'OppID',\n    'Team1Score':'OppScore'\n}).drop(columns=['Team1Won', 'WTeamID', 'WScore', 'LTeamID', 'LScore', 'WLoc'])\n\n# Combine gameteams1 and gameteams2\ngameteams = pd.concat((gameteams1, gameteams2), sort=True)\n\n# Insert fields, Won, MOV (Margin Of Victory)\ngameteams['Won'] = gameteams.Score > gameteams.OppScore\ngameteams['MOV'] = gameteams.Score - gameteams.OppScore\n\n# Print snippet\ngameteams.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### seasonteams"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Aggregate regular seaon gameteams by (Season, TeamID) and calculate basic stats\nseasonteams = gameteams.loc[gameteams.Round == -1].groupby(['Season', 'TeamID']).agg(\n    Games=pd.NamedAgg(column='TeamID', aggfunc=np.size),\n    Wins=pd.NamedAgg(column='Won', aggfunc='sum'),\n    Points=pd.NamedAgg(column='Score', aggfunc='sum'),\n    OppPoints=pd.NamedAgg(column='OppScore', aggfunc='sum'),\n    AvgMOV=pd.NamedAgg(column='MOV', aggfunc='mean')\n).reset_index()\nseasonteams['WinPct'] = seasonteams.Wins/seasonteams.Games\nseasonteams['PPG'] = seasonteams.Points/seasonteams.Games\nseasonteams['OppPPG'] = seasonteams.OppPoints/seasonteams.Games\n\n# For each (Season, Team), identify the team as making the tournament or not\ntrnyteams = gameteams.loc[gameteams.Round >= 0, ['Season', 'TeamID']].drop_duplicates()\ntrnyteams['InTourney'] = True\nseasonteams['InTourney'] = False\nseasonteams = join_insertion(into_df=seasonteams, from_df=trnyteams, cols='InTourney', on=['Season', 'TeamID'])\n\n# Print snippet\nseasonteams.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Professor\n----\n\nBefore we start building models, let's construct one more dataset - **modeldata**. This dataset will have every conceivable tournament game; (Season, Team1, Team2) tuples including tournament games that were never played but *could* have been played. For example, the 2019 tournament started with 68 teams. Pick any two teams from that tournament. As long as they keep winning, they'll eventually match up against each other (or perhaps they matched up at the very start). This means that tournaments that start with 68 teams will generate 68\\*67/2 conceivable matchups - each represented as a row in our **modeldata** dataset.\n\nThe purpose of this dataset is to have training, validation, and test records to draw from for developing our model."},{"metadata":{},"cell_type":"markdown","source":"## Model Data"},{"metadata":{"trusted":true},"cell_type":"code","source":"# Build every possible (Team1ID vs Team2ID) tournament game. Most of these will have never been played\nmodeldata = trnyteams[['Season', 'TeamID']]\nmodeldata = modeldata.groupby('Season').apply(lambda x: dataframe_cj(x[['TeamID']], x[['TeamID']]))\nmodeldata = modeldata.reset_index().drop(columns=['level_1']).rename(columns={'TeamID_x':'Team1ID', 'TeamID_y':'Team2ID'})\nmodeldata = modeldata.loc[modeldata.Team1ID < modeldata.Team2ID]\n\n# Insert team seeds and regions\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=trnyseeds,\n    cols=['SeedNum', 'Region', 'PlayIn'],\n    on={'Season':'Season', 'Team1ID':'TeamID'}\n).rename(columns={'SeedNum':'SeedNum1', 'Region':'Region1', 'PlayIn':'PlayIn1'})\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=trnyseeds,\n    cols=['SeedNum', 'Region', 'PlayIn'],\n    on={'Season':'Season', 'Team2ID':'TeamID'}\n).rename(columns={'SeedNum':'SeedNum2', 'Region':'Region2', 'PlayIn':'PlayIn2'})\n\n# Insert Round\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=rd_pairs,\n    on=['Region1', 'Region2', 'SeedNum1', 'SeedNum2'],\n    cols='Round'\n)\nmodeldata.loc[(modeldata.Region1 == modeldata.Region2) & (modeldata.SeedNum1 == modeldata.SeedNum2), 'Round'] = 0\n\n# Insert season-team stats\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=seasonteams,\n    cols=['WinPct', 'AvgMOV', 'PPG', 'OppPPG'],\n    on={'Season':'Season', 'Team1ID':'TeamID'}\n)\nmodeldata.rename(columns={'WinPct':'WinPct1', 'AvgMOV':'AvgMOV1', 'PPG':'PPG1', 'OppPPG':'OppPPG1'}, inplace=True)\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=seasonteams,\n    cols=['WinPct', 'AvgMOV', 'PPG', 'OppPPG'],\n    on={'Season':'Season', 'Team2ID':'TeamID'}\n)\nmodeldata.rename(columns={'WinPct':'WinPct2', 'AvgMOV':'AvgMOV2', 'PPG':'PPG2', 'OppPPG':'OppPPG2'}, inplace=True)\n\n# Insert result from most recent team1 vs team2 matchup in the same season\nrsgames['Team1MOV'] = rsgames.Team1Score - rsgames.Team2Score\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=rsgames,\n    cols='Team1MOV',\n    on=['Season', 'Team1ID', 'Team2ID'],\n    mult='last'\n)\nmodeldata.rename(columns={'Team1MOV':'Team1RecentMOV'}, inplace=True)\nmodeldata.fillna({'Team1RecentMOV':0}, inplace=True)\n\n# Insert game stats for games which were actually played\nmodeldata = join_insertion(\n    into_df=modeldata,\n    from_df=trnygames,\n    cols=['Team1Score', 'Team2Score', 'Team1Won'],\n    on=['Season', 'Team1ID', 'Team2ID']\n)\n\n# Print snippet\nmodeldata.head()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Train/Eval/Test Split\n\nHere we split modeldata into three datasets:\n\n1. **train** - modeldata from seasons 1985 - 2014. This is the input we'll feed into our learners (nnet, lightgbm, logistic regression).\n2. **eval** - model data from seasons 2015 - 2019. As our nnet and lightgbm models go throught the training procedure, we'll evaluate their performance on this dataset to know when to cut-off the training process and prevent overfitting\n3. **test** - model data from seasons 2015 - 2019. This is the dataset we want to make predicitons for\n\n**We're cheating a little bit**. Our models will have the benefit of using the actual test data in their training process. But this type of leakage isn't that bad since we're not exposing our models to the individual targets of the test dataset. The benefit of this approach is simplicity and not cutting our training data into a tiny dataset."},{"metadata":{"trusted":true},"cell_type":"code","source":"# train on seasons 1985 - 2014 (30 seasons)\n# use seasons 2015 - 2019 to evaluate results and perform early stopping\ntrain = modeldata.loc[(modeldata.Season < 2015) & (modeldata.Team1Won.notna())].copy()\neval = modeldata.loc[(modeldata.Season >= 2015) & (modeldata.Team1Won.notna())].copy()\n\ntrain['Team1Won'] = train.Team1Won.astype('bool')\neval['Team1Won'] = eval.Team1Won.astype('bool')\n\n# Build test dataset of (Season, Team1ID, Team2ID) for every possible Team1ID vs Team2ID within each tournament\ntest = modeldata.loc[modeldata.Season >= 2015].copy()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Keras NNet Model\n\nHere we build a densly connected neural network model with an input layer, two hidden layers, and an output layer, each with sigmoid activation, as well as a couple droupout layers to prevent over-fitting. Note the imports from tensorflow.keras up above ^^. \n\n**Why this particular neural network architecture?**  \nNo particular reason, other than I tinkered with a few structures and this one seemed to work well."},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- setup --------------------------------------\n\n# Build xtrain, xeval, xtest, ytrain, yeval\nfeats = [\n    'Round', 'Team1RecentMOV',\n    'SeedNum1', 'SeedNum2', 'WinPct1', 'WinPct2', 'AvgMOV1', 'AvgMOV2', 'PPG1', 'PPG2', 'OppPPG1', 'OppPPG2'\n]\n\nxtrain = train[feats].to_numpy()\nxeval = eval[feats].to_numpy()\nxtest = test[feats].to_numpy()\n\nytrain = train.Team1Won.to_numpy().astype('int64')\nyeval = eval.Team1Won.to_numpy().astype('int64')\n\n# Scale features to 0-1 (not quite true, but close enough)\nxtest = xtest/np.max(xtrain, axis=0)\nxeval = xeval/np.max(xtrain, axis=0)\nxtrain = xtrain/np.max(xtrain, axis=0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- model --------------------------------------\n\nnnet = Sequential()\n\nnnet.add(Dense(64, activation='sigmoid'))\nnnet.add(Dropout(0.5))\n\nnnet.add(Dense(16, activation='sigmoid'))\nnnet.add(Dropout(0.2))\n\nnnet.add(Dense(1, activation='sigmoid'))\n\nnnet.compile(optimizer='adam', loss='binary_crossentropy', metrics=['accuracy'])\n\nearly = EarlyStopping(patience=50, restore_best_weights=True)\nnnet.fit(x = xtrain, y = ytrain, batch_size=64, epochs=999, validation_data=(xeval, yeval), callbacks=[early])","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- evaluate --------------------------------------\n\n# Insert preds\neval['Prob_NNet'] = nnet.predict_proba(xeval).astype('float64')\ntest['Prob_NNet'] = nnet.predict_proba(xtest).astype('float64')\n\n# Plot validation score over time\npd.DataFrame(nnet.history.history).plot(figsize=(12, 8))\n\n# log loss on all seasons combined\nlog_loss(y_true=eval.Team1Won, y_pred=eval.Prob_NNet)  # 0.5499\n\n# log loss by season\neval.groupby('Season')[['Team1Won','Prob_NNet']].apply(lambda x: log_loss(x.Team1Won, x.Prob_NNet))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## LightGBM Model\n\nNext, we build a gradient boosting model via LightGBM. We'll use the same exact train & eval data as before for convenience. Unlike NNets, we don't need to our input data to be scaled to 0-1 for lightgbm, but doing so shouldn't degrade performance.\n\nI didn't spend a lot of time tuning hyper-parameters. But the params below seemed to work well."},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- model --------------------------------------\n\n# Create dataset for lightgbm\nlgb_train = lgb.Dataset(xtrain, ytrain)\nlgb_eval = lgb.Dataset(xeval, yeval, reference=lgb_train)\n\n# specify your configurations as a dict\nparams = {\n    'boosting_type': 'gbdt',\n    'objective': 'binary',\n    'metric': ['binary_logloss'],\n    'num_leaves': 21,\n    'learning_rate': 0.01,\n    'feature_fraction': 0.7,\n    'bagging_fraction': 0.5,\n    'bagging_freq': 5,\n    'verbose': 0\n}\n\n# train\ngbm = lgb.train(\n    params=params,\n    train_set=lgb_train,\n    num_boost_round=999,\n    valid_sets=lgb_eval,\n    early_stopping_rounds=10\n)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- evaluate --------------------------------------\n\n# Insert preds\neval['Prob_LGB'] = gbm.predict(xeval, num_iteration=gbm.best_iteration)\ntest['Prob_LGB'] = gbm.predict(xtest, num_iteration=gbm.best_iteration)\n\n# feature importance\nprint(pd.DataFrame({'Feat':feats, 'Gain':gbm.feature_importance(importance_type='gain')}).sort_values('Gain', ascending=False))\n\n# log loss on all seasons combined\nlog_loss(y_true=eval.Team1Won, y_pred=eval.Prob_LGB)  # 0.5482\n\n# log loss by season\neval.groupby('Season')[['Team1Won','Prob_LGB']].apply(lambda x: log_loss(x.Team1Won, x.Prob_LGB))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Logistic Regression Model\n\nHere it turns out that our scaled data does worse than our unscaled data, so we rebuild xtrain and xeval without scaling the features to 0-1."},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- setup --------------------------------------\n\n# Build xtrain, xeval, xtest, ytrain, yeval\nxtrain = train[feats].to_numpy()\nxeval = eval[feats].to_numpy()\nxtest = test[feats].to_numpy()\nytrain = train.Team1Won.to_numpy().astype('int64')\nyeval = eval.Team1Won.to_numpy().astype('int64')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- model --------------------------------------\n\n# Fit model\nlr = LogisticRegression(random_state=0, multi_class='ovr', solver='lbfgs', max_iter=1000).fit(xtrain, ytrain)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"#--- evaluate --------------------------------------\n\n# Insert preds\neval['Prob_LR'] = lr.predict_proba(xeval)[:, 1]\ntest['Prob_LR'] = lr.predict_proba(xtest)[:, 1]\n\n# log loss on all seasons combined\nlog_loss(y_true=eval.Team1Won, y_pred=eval.Prob_LR)  # 0.5535\n\n# log loss by season\neval.groupby('Season')[['Team1Won','Prob_LR']].apply(lambda x: log_loss(x.Team1Won, x.Prob_LR))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Ensemble\n\nIt's often the case that ensembling diverse models together can do better than your best single model in isolation. We test that here using a simple averaging of the predicted probabilities from each model."},{"metadata":{"trusted":true},"cell_type":"code","source":"# Combine predictions\neval['ProbAvg'] = (eval.Prob_NNet + eval.Prob_LGB + eval.Prob_LR)/3\ntest['ProbAvg'] = (test.Prob_NNet + test.Prob_LGB + test.Prob_LR)/3\n\n# Compare\nprint('NNet: {0}'.format(log_loss(eval.Team1Won, eval.Prob_NNet)))   # 0.5499\nprint('LGB: {0}'.format(log_loss(eval.Team1Won, eval.Prob_LGB)))     # 0.5482\nprint('LR: {0}'.format(log_loss(eval.Team1Won, eval.Prob_LR)))       # 0.5535\nprint('Ensemble: {0}'.format(log_loss(eval.Team1Won, eval.ProbAvg))) # 0.5475","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Indeed, tis the case here!"},{"metadata":{},"cell_type":"markdown","source":"# Prepare Submission\n---\n\nAlas, we prepare our submission and hold our breath."},{"metadata":{"trusted":true},"cell_type":"code","source":"test['ID'] = test.Season.astype('str') + '_' + test.Team1ID.astype('str') + '_' + test.Team2ID.astype('str')\nsubm = test[['ID', 'ProbAvg']].rename(columns = {'ProbAvg':'Pred'})\nsubm.to_csv('professor.csv', index=False)","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}