{"cells":[{"metadata":{},"cell_type":"markdown","source":"# 0. Introduction\n\n## 0.1 Context\nThe dataset contained reviews from the gaming platform Steam, including userID/gameID pairs of specific users who reviewed specific games. There were two tasks for this Kaggle competition:\n1. **Task #1 - Play Prediction**: Predict whether a certain user has played a certain game\n2. **Task #2 - Time Played Prediction**: Predict how many hours a certain user has played a certain game\n\n## 0.2 Methods\nThe following methods were attempted (code provided in the rest of the notebook):\n1. **Collaborative Filtering Heuristics**: Jaccard score, cosine similarity, and Pearson correlation were calculated among users and items.\n2. **Latent Factor Models**: Variations of latent factor models were used to make predictions based on similarity, including a simple model with a bias only term and Bayesian Personalized Recommendation for classification in Task #1. \n3. **Feature Generation**: Since the test set only provided userID/gameID pairs with no additional information, aggregate summary statistics for games and users were generated independently, including average number of reviews, average number of hours played, etc. Interactions between variables (e.g. \"game_num_reviews * user_max_cosine_similarity\") were also generated. See Section 0.3 for the full list.\n4. **Dimensionality Reduction**: PCA and feature selection using chi-squared statistics to reduce the number of features. \n5. **Model Selection**: GradientBoosting performed consistently better for both tasks, so I just used that throughout. Other models tried: XGBoost, Linear/Logistic, AdaBoost, RandomForest, Voting.\n6. **Hyperparameter tuning**: Grid search was used frequently. For latent factor models, I tuned: K, lambda, learning rate (Tensorflow). For features, I tuned: # of k-best features, # of PCA components. For GradientBoosting models, I tuned: learning_rate, n_estimators, max_sample_split, max_depth\n\n## 0.3 Final Submission\nThe following are the approaches used for the final Kaggle submission on the test set:\n\n### Task #1\n1. Of all the features generated, determine most influential feature in each category (ie. user/game_jaccard, user/game_reviews, user/game_date, etc.) by training single feature model and determining accuracy score on validation set. See the final list of features in \"Summary of Features - Task #1.\" \n2. Taking the list of best features in each category, use hyperparameter tuning to determine best \"k-best\" and \"pca\" parameters on the validation set (result: k-best = 16, pca_components = 8)\n3. Plug PCA components into GradientBoostingClassifier, using hyperparameter tuning to determine best parameters.\n\n### Task #2\n1. Use hyperparameter tuning to determine best value of K and lambda for latent factor model (result: K = 0, lamdba = 0.00005)\n2. Plug in resulting predictions into a single feature GradientBoostingRegressor, using hyperparameter tuning to determine best parameters.\n\n\n## 0.4 Summary of Features\n### Task #1 - Best Features\n['user_max_jaccard',\n 'game_avg_jaccard',\n 'user_max_cosine_similarity',\n 'game_avg_cosine_similarity',\n 'user_median_pearson_correlation',\n 'game_avg_pearson_correlation',\n 'user_num_reviews',\n 'game_num_reviews',\n 'user_avg_hours',\n 'game_avg_hours',\n 'user_avg_date',\n 'game_avg_date',\n 'avg_date_difference_scaled',\n 'user_num_early_access',\n 'game_num_early_access',\n 'user_num_found_funny',\n 'game_num_found_funny',\n 'lf_score'\n ]\n \n ### Task #2 - Best Features\n ['lf_complete_predictions']\n\n\n## 0.5 Future Directions\n- Extensions to latent factor models: e.g. incorporate user/item features, change over time, etc.\n- More hyperparameter tuning\n- Text analysis: top 100 dictionary\n- Neural net\n"},{"metadata":{},"cell_type":"markdown","source":"# 1. Setup"},{"metadata":{},"cell_type":"markdown","source":"## 1.0 Imports"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Imports\nimport pandas as pd\nimport numpy as np\nimport random\nimport gzip\nimport scipy\nimport math\nimport pickle\nfrom datetime import datetime\n\nimport xgboost\nimport tensorflow as tf\nimport sklearn.metrics\nimport sklearn.linear_model\nimport sklearn.ensemble\nimport sklearn.svm\nimport sklearn.decomposition\nimport sklearn.feature_selection\n\nfrom collections import defaultdict\nimport re","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.1 Helper Functions"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Linearly scales a series\ndef linear_scale(series):\n    min_val = series.min()\n    max_val = series.max()\n    scale = (max_val - min_val) / 2.0\n    return series.apply(lambda x:((x - min_val) / scale) - 1.0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Scales a series by log normalizing\ndef log_normalize(series):\n    return series.apply(lambda x:math.log(x+1.0))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Takes two sets and returns the Jaccard similarity between them\ndef jaccard_score(s1, s2):\n    numer = len(s1.intersection(s2))\n    denom = len(s1.union(s2))\n    return numer / denom","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Determine cosine similarity between two sets\ndef cosine_similarity(set1, set2):\n    union = list(set1 or set2) # Find the union of the two sets\n    list1 = [1 if x in set1 else 0 for x in union] # Determine which items in the union are in set1\n    list2 = [1 if x in set2 else 0 for x in union] # Determine which items in the union are in set2\n    \n    # Calculate the numerator, which is the sum of the dot product of list1 & list2\n    numerator = sum([list1[x] * list2[x] for x in range(0, len(union))])\n\n    # Calculate denominator, which is the sqrt of the product of the sums\n    denominator = math.sqrt(sum(list1) * sum(list2))\n\n    if denominator: # If the denominator is not 0, return cosine similarity\n        return numerator / denominator\n    else: # Else return 0\n        return 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# A class for quick gridsearchCV testing on training and validation sets\nclass gridsearchcv_model:\n#   model: saved model\n#   name: name for model\n#   train, val: object with {name, predictions, mse OR accuracy} \n\n    def __init__(self, model, X_train, Y_train, X_val, Y_val, parameter_matrix={}, is_classification=False, cv=5):\n        self.is_classification = is_classification\n        self.train_model(model, X_train, Y_train, X_val, Y_val, parameter_matrix, cv)\n        \n    # Trains model using a training set and predicts a validation set\n    def train_model(self, model, X_train, Y_train, X_val, Y_val, parameter_matrix={}, cv=5):\n        if self.is_classification:\n            ml_model = sklearn.model_selection.GridSearchCV(model, parameter_matrix, cv=cv, scoring='accuracy')\n        else:\n            ml_model = sklearn.model_selection.GridSearchCV(model, parameter_matrix, cv=cv, scoring='neg_mean_squared_error')\n        \n        ml_model.fit(X_train, Y_train)\n        \n        self.model = ml_model.best_estimator_\n        self.name = re.compile(\"(.*?)\\s*\\(\").match(str(self.model)).group(1)\n        \n        self.train = {'name': 'train'}\n        self.val = {'name': 'val'}\n        \n        self.calculate_error(self.train, X_train, Y_train, self.train['name'])\n        self.calculate_error(self.val, X_val, Y_val, self.val['name'])\n        \n        return ml_model\n    \n    def calculate_error(self, var, X_set, Y_set, name):\n        var['name'] = name\n        var['predictions'] = self.model.predict(X_set)\n        \n        if self.is_classification:\n            var['accuracy'] = sklearn.metrics.accuracy_score(Y_set, var['predictions'])\n        else:\n            var['mse'] = sklearn.metrics.mean_squared_error(Y_set, var['predictions'])\n        \n        self.print_error(var)\n        \n    # Prints error metrics\n    def print_error(self, var):\n        print(self.name + ' ('+ var['name'] + ')')\n        \n        if self.is_classification:\n            print(\"Accuracy: %0.4f\" % var['accuracy'])\n        else:\n            print(\"MSE: %0.4f\" % var['mse'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.2 Read Dataset"},{"metadata":{"trusted":false},"cell_type":"code","source":"def readJSON(path):\n    for l in gzip.open(path, 'rt'):\n        d = eval(l)\n        yield d","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Read in dataset\ndataset = []\nfor d in readJSON(\"train.json.gz\"):\n    dataset.append(d)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Convert dataset into DataFrame\ndf = pd.DataFrame(dataset)\ndf = df.reindex(np.random.permutation(df.index)).reset_index(drop=True)\ndf['has_played'] = 1\n\ndf","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Create train and validation sets\ndf_train = df[:165000]\ndf_val = df[165000:]\ndf_val = df_val.reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.3 Explore Dataset"},{"metadata":{"trusted":false},"cell_type":"code","source":"df.info()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df.isnull().sum()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"sum(df['early_access'])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.4 Useful Data Structures"},{"metadata":{"trusted":false},"cell_type":"code","source":"gameIDs = defaultdict(int)\nuserIDs = defaultdict(int)\n\nusersPerGame = defaultdict(set)\ngamesPerUser = defaultdict(set)\n\nhoursTransformedPerGame = defaultdict(dict)\nhoursTransformedPerUser = defaultdict(dict)\n\nhoursPerGame = defaultdict(list)\nhoursPerUser = defaultdict(list)\n\ndatePerGame = defaultdict(list)\ndatePerUser = defaultdict(list)\n\nearlyAccessPerGame = defaultdict(list)\nearlyAccessPerUser = defaultdict(list)\n\ntextPerGame = defaultdict(list)\ntextPerUser = defaultdict(list)\n\nfoundFunnyPerGame = defaultdict(list)\nfoundFunnyPerUser = defaultdict(list)\n\nfor i in range(len(df)):\n    userID = df['userID'].iloc[i]\n    gameID = df['gameID'].iloc[i]\n    hoursTransformed = df['hours_transformed'].iloc[i]\n    \n    if not userID in userIDs: userIDs[userID] = len(userIDs)\n    if not gameID in gameIDs: gameIDs[gameID] = len(gameIDs)\n    \n    usersPerGame[gameID].add(userID)\n    gamesPerUser[userID].add(gameID)\n    \n    hoursTransformedPerGame[gameID][userID] = hoursTransformed\n    hoursTransformedPerUser[userID][gameID] = hoursTransformed\n    \n    hoursPerGame[gameID].append(df['hours'].iloc[i])\n    hoursPerUser[userID].append(df['hours'].iloc[i])\n    \n    datePerGame[gameID].append(df['date'].iloc[i])\n    datePerUser[userID].append(df['date'].iloc[i])\n\n    earlyAccessPerGame[gameID].append(df['early_access'].iloc[i])\n    earlyAccessPerUser[userID].append(df['early_access'].iloc[i])\n\n    textPerGame[gameID].append(df['text'].iloc[i])\n    textPerUser[userID].append(df['text'].iloc[i])\n\n    foundFunnyPerGame[gameID].append(df['found_funny'].iloc[i])\n    foundFunnyPerUser[userID].append(df['found_funny'].iloc[i])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"gameIDs_train = defaultdict(int)\nuserIDs_train = defaultdict(int)\n\nusersPerGame_train = defaultdict(set)\ngamesPerUser_train = defaultdict(set)\n\nhoursTransformedPerGame_train = defaultdict(dict)\nhoursTransformedPerUser_train = defaultdict(dict)\n\nhoursPerGame_train = defaultdict(list)\nhoursPerUser_train = defaultdict(list)\n\ndatePerGame_train = defaultdict(list)\ndatePerUser_train = defaultdict(list)\n\nearlyAccessPerGame_train = defaultdict(list)\nearlyAccessPerUser_train = defaultdict(list)\n\ntextPerGame_train = defaultdict(list)\ntextPerUser_train = defaultdict(list)\n\nfoundFunnyPerGame_train = defaultdict(list)\nfoundFunnyPerUser_train = defaultdict(list)\n\nfor i in range(len(df_train)):\n    userID = df_train['userID'].iloc[i]\n    gameID = df_train['gameID'].iloc[i]\n    hoursTransformed = df['hours_transformed'].iloc[i]\n    \n    if not userID in userIDs_train: userIDs_train[userID] = len(userIDs_train)\n    if not gameID in gameIDs_train: gameIDs_train[gameID] = len(gameIDs_train)\n    \n    usersPerGame_train[gameID].add(userID)\n    gamesPerUser_train[userID].add(gameID)\n    \n    hoursTransformedPerGame_train[gameID][userID] = hoursTransformed\n    hoursTransformedPerUser_train[userID][gameID] = hoursTransformed\n    \n    hoursPerGame_train[gameID].append(df_train['hours'].iloc[i])\n    hoursPerUser_train[userID].append(df_train['hours'].iloc[i])\n    \n    datePerGame_train[gameID].append(df_train['date'].iloc[i])\n    datePerUser_train[userID].append(df_train['date'].iloc[i])\n\n    earlyAccessPerGame_train[gameID].append(df_train['early_access'].iloc[i])\n    earlyAccessPerUser_train[userID].append(df_train['early_access'].iloc[i])\n\n    textPerGame_train[gameID].append(df_train['text'].iloc[i])\n    textPerUser_train[userID].append(df_train['text'].iloc[i])\n\n    foundFunnyPerGame_train[gameID].append(df_train['found_funny'].iloc[i])\n    foundFunnyPerUser_train[userID].append(df_train['found_funny'].iloc[i])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 1.5 Create negative datasets for play prediction"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Games NOT played by each user using full dataset\ngamesNotPlayedPerUser = defaultdict(set) \n\nfor userID in userIDs.keys():\n    gamesNotPlayedPerUser[userID] = np.setdiff1d(list(gameIDs), list(gamesPerUser[userID]))","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# Add negative entries to training set\ndf_toAdd_train = df_train[['userID']] \n\ndf_toAdd_train['has_played'] = 0\ndf_toAdd_train['gameID'] = [np.random.choice(gamesNotPlayedPerUser[userID]) for userID in df_train['userID']]\n\n# Concat entries to add to validation set\ndf_train_negative = pd.concat([df_train, df_toAdd_train], axis=0)\ndf_train_negative.index = range(0, len(df_train_negative))\ndf_train_negative = df_train_negative.reindex(np.random.permutation(df_train_negative.index)).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# Add negative entries to validation set\ndf_toAdd_val = df_val[['userID']] \n\ndf_toAdd_val['has_played'] = 0\ndf_toAdd_val['gameID'] = [np.random.choice(gamesNotPlayedPerUser[userID]) for userID in df_val['userID']]\n\n# Concat entries to add to validation set\ndf_val_negative = pd.concat([df_val, df_toAdd_val], axis=0)\ndf_val_negative.index = range(0, len(df_val_negative))\ndf_val_negative = df_val_negative.reindex(np.random.permutation(df_val_negative.index)).reset_index(drop=True)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 2. Feature Construction"},{"metadata":{},"cell_type":"markdown","source":"## 2.1 Feature Generation"},{"metadata":{},"cell_type":"markdown","source":"### Collaborative Filtering"},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_user_jaccards(data, data_usersPerGame, data_gamesPerUser):\n    data['user_max_jaccard'] = 0\n    data['user_avg_jaccard'] = 0\n    data['user_median_jaccard'] = 0\n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n\n        similarities = [jaccard_score(data_usersPerGame[g], data_usersPerGame[game]) for g in data_gamesPerUser[user] if g != game]\n\n        if similarities: \n            data['user_max_jaccard'].iloc[i] = max(similarities) \n            data['user_avg_jaccard'].iloc[i] = np.mean(similarities) \n            data['user_median_jaccard'].iloc[i] = np.median(similarities)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_game_jaccards(data, data_usersPerGame, data_gamesPerUser):\n    data['game_max_jaccard'] = 0 \n    data['game_avg_jaccard'] = 0\n    data['game_median_jaccard'] = 0\n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n\n        similarities = [jaccard_score(data_gamesPerUser[u], data_gamesPerUser[user]) for u in data_usersPerGame[game] if u != user]\n\n        if similarities: \n            data['game_max_jaccard'].iloc[i] = max(similarities) \n            data['game_avg_jaccard'].iloc[i] = np.mean(similarities) \n            data['game_median_jaccard'].iloc[i] = np.median(similarities)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_user_cosine_similarities(data, data_usersPerGame, data_gamesPerUser):\n    data['user_max_cosine_similarity'] = 0 \n    data['user_avg_cosine_similarity'] = 0 \n    data['user_median_cosine_similarity'] = 0 \n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n\n        similarities = [cosine_similarity(data_usersPerGame[g], data_usersPerGame[game]) for g in data_gamesPerUser[user] if g != game]\n\n        if similarities: \n            data['user_max_cosine_similarity'].iloc[i] = max(similarities) \n            data['user_avg_cosine_similarity'].iloc[i] = np.mean(similarities) \n            data['user_median_cosine_similarity'].iloc[i] = np.median(similarities) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_game_cosine_similarities(data, data_usersPerGame, data_gamesPerUser):\n    data['game_max_cosine_similarity'] = 0 \n    data['game_avg_cosine_similarity'] = 0 \n    data['game_median_cosine_similarity'] = 0 \n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n\n        similarities = [cosine_similarity(data_gamesPerUser[u], data_gamesPerUser[user]) for u in data_usersPerGame[game] if u != user]\n\n        if similarities: \n            data['game_max_cosine_similarity'].iloc[i] = max(similarities) \n            data['game_avg_cosine_similarity'].iloc[i] = np.mean(similarities) \n            data['game_median_cosine_similarity'].iloc[i] = np.median(similarities) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_user_pearson_correlations(data, data_usersPerGame, data_gamesPerUser, data_hoursTransformedPerGame):\n    data['user_max_pearson_correlation'] = 0 \n    data['user_avg_pearson_correlation'] = 0 \n    data['user_median_pearson_correlation'] = 0 \n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n        similarities = []\n        \n        for g in data_gamesPerUser[user]:\n            if g == game:\n                continue\n            \n            set1 = data_usersPerGame[g]\n            set2 = data_usersPerGame[game]\n            union = list(set1 or set2)\n            \n            if len(union) > 1:\n                list1 = [data_hoursTransformedPerGame[g][user] if user in set1 else 0 for user in union] \n                list2 = [data_hoursTransformedPerGame[game][user] if user in set2 else 0 for user in union]\n                x = scipy.stats.pearsonr(list1, list2)[0]\n                if not math.isnan(x):\n                    similarities.append(x)\n                        \n        if similarities: \n            data['user_max_pearson_correlation'].iloc[i] = max(similarities) \n            data['user_avg_pearson_correlation'].iloc[i] = np.mean(similarities) \n            data['user_median_pearson_correlation'].iloc[i] = np.median(similarities) ","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_game_pearson_correlations(data, data_usersPerGame, data_gamesPerUser, data_hoursTransformedPerUser):\n    data['game_max_pearson_correlation'] = 0 \n    data['game_avg_pearson_correlation'] = 0 \n    data['game_median_pearson_correlation'] = 0 \n\n    for i in range(len(data)):\n        user = data['userID'][i]\n        game = data['gameID'][i]\n        similarities = []\n        \n        for u in data_usersPerGame[game]:\n            if u == user:\n                continue\n            \n            set1 = data_gamesPerUser[u]\n            set2 = data_gamesPerUser[user]\n            union = list(set1 or set2)\n            \n            if len(union) > 1:\n                list1 = [data_hoursTransformedPerUser[u][game] if game in set1 else 0 for game in union] \n                list2 = [data_hoursTransformedPerUser[user][game] if game in set2 else 0 for game in union]\n                x = scipy.stats.pearsonr(list1, list2)[0]\n                if not math.isnan(x):\n                    similarities.append(x)\n                        \n        if similarities: \n            data['game_max_pearson_correlation'].iloc[i] = max(similarities) \n            data['game_avg_pearson_correlation'].iloc[i] = np.mean(similarities) \n            data['game_median_pearson_correlation'].iloc[i] = np.median(similarities) ","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Dataset Features"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Creates a feature column of count of reviews\ndef feature_num_reviews(data, data_usersPerGame, data_gamesPerUser):\n    data['user_num_reviews'] = [len(data_gamesPerUser[userID]) for userID in data['userID']]\n    data['game_num_reviews'] = [len(data_usersPerGame[gameID]) for gameID in data['gameID']]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Creates a feature column of count of user's average hours played\ndef feature_hours(data, data_hoursPerGame, data_hoursPerUser):\n    data['user_avg_hours'] = [np.mean(data_hoursPerUser[userID]) for userID in data['userID']]\n    data['user_avg_hours'] = data['user_avg_hours'].fillna(0)\n    \n    data['game_avg_hours'] = [np.mean(data_hoursPerGame[gameID]) for gameID in data['gameID']]\n    data['game_avg_hours'] = data['game_avg_hours'].fillna(0)\n    \n    data['user_median_hours'] = [np.median(data_hoursPerUser[userID]) for userID in data['userID']]\n    data['user_median_hours'] = data['user_median_hours'].fillna(0)\n    \n    data['game_median_hours'] = [np.median(data_hoursPerGame[gameID]) for gameID in data['gameID']]\n    data['game_median_hours'] = data['game_median_hours'].fillna(0)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_date(data, data_datePerUser, data_datePerGame):\n    data['user_avg_date'] = [np.mean(pd.to_datetime(data_datePerUser[userID]).astype(int)) for userID in data['userID']]\n    data['game_avg_date'] = [np.mean(pd.to_datetime(data_datePerGame[gameID]).astype(int)) for gameID in data['gameID']]\n    data['avg_date_difference'] = data['user_avg_date'] - data['game_avg_date']\n    \n    data['user_median_date'] = [np.median(pd.to_datetime(data_datePerUser[userID]).astype(int)) for userID in data['userID']]\n    data['game_median_date'] = [np.median(pd.to_datetime(data_datePerGame[gameID]).astype(int)) for gameID in data['gameID']]\n    data['median_date_difference'] = data['user_median_date'] - data['game_median_date']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_other(data, data_earlyAccessPerUser, data_earlyAccessPerGame, data_foundFunnyPerUser, data_foundFunnyPerGame):\n    data['user_num_early_access'] = [sum(data_earlyAccessPerUser[userID]) if not math.isnan(sum(data_earlyAccessPerUser[userID])) else 0 for userID in data['userID']]\n    data['game_num_early_access'] = [sum(data_earlyAccessPerGame[gameID]) if not math.isnan(sum(data_earlyAccessPerGame[gameID])) else 0 for gameID in data['gameID']]\n    \n    data['user_num_found_funny'] = [sum(data_foundFunnyPerUser[userID]) if not math.isnan(sum(data_foundFunnyPerUser[userID])) else 0 for userID in data['userID']]\n    data['game_num_found_funny'] = [sum(data_foundFunnyPerGame[gameID]) if not math.isnan(sum(data_foundFunnyPerGame[gameID])) else 0 for gameID in data['gameID']]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"### Transforming Variables"},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_scaled(data):\n    data['user_num_reviews_scaled'] = log_normalize(data['user_num_reviews'])\n    data['game_num_reviews_scaled'] = log_normalize(data['game_num_reviews'])\n    \n    data['user_avg_hours_scaled'] = log_normalize(data['user_avg_hours'])\n    data['game_avg_hours_scaled'] = log_normalize(data['game_avg_hours'])\n    \n    data['user_median_hours_scaled'] = log_normalize(data['user_median_hours'])\n    data['game_median_hours_scaled'] = log_normalize(data['game_median_hours'])\n    \n    data['user_avg_date_scaled'] = linear_scale(data['user_avg_date'])\n    data['game_avg_date_scaled'] = linear_scale(data['game_avg_date'])\n    data['avg_date_difference_scaled'] = linear_scale(data['avg_date_difference'])\n    \n    data['user_median_date_scaled'] = linear_scale(data['user_median_date'])\n    data['game_median_date_scaled'] = linear_scale(data['game_median_date'])\n    data['median_date_difference_scaled'] = linear_scale(data['median_date_difference'])\n\n    data['user_num_early_access_scaled'] = log_normalize(data['user_num_early_access'])\n    data['game_num_early_access_scaled'] = log_normalize(data['game_num_early_access'])\n\n    data['user_num_found_funny_scaled'] = log_normalize(data['user_num_found_funny'])\n    data['game_num_found_funny_scaled'] = log_normalize(data['game_num_found_funny'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_interactions(data, feat_list):\n    for feat in feat_list:\n        new_feat = 'game_num_reviews_scaled_' + feat\n        data[new_feat] = data['game_num_reviews_scaled'] * data[feat]\n        interactions_negative.append(new_feat)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.2 Latent Factor Model - TensorFlow"},{"metadata":{"trusted":false},"cell_type":"code","source":"alpha = np.mean(df_train['hours_transformed'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Gradient descent optimizer, experiment with learning rate\nlf_optimizer = tf.keras.optimizers.Adam(0.001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class LatentFactorModel(tf.keras.Model):\n    def __init__(self, mu, K, lamb):\n        super(LatentFactorModel, self).__init__()\n        \n        # Initialize to average\n        self.alpha = tf.Variable(mu,dtype=tf.float32)\n        \n        # Initialize to small random values\n        self.betaU = tf.Variable(tf.random.normal([len(userIDs)],stddev=0.001),dtype=tf.float32)\n        self.betaI = tf.Variable(tf.random.normal([len(gameIDs)],stddev=0.001),dtype=tf.float32)\n        self.gammaU = tf.Variable(tf.random.normal([len(userIDs),K],stddev=0.001), dtype=tf.float32)\n        self.gammaI = tf.Variable(tf.random.normal([len(gameIDs),K],stddev=0.001),dtype=tf.float32)\n        self.lamb = lamb\n\n    # Prediction for a single instance (useful for evaluation)\n    def predict(self, u, i):\n        p = self.alpha + self.betaU[u] + self.betaI[i] + tf.tensordot(self.gammaU[u], self.gammaI[i], 1)\n        return p\n\n    # Regularizer\n    def reg(self):\n        return self.lamb * (tf.reduce_sum(self.betaU**2) + tf.reduce_sum(self.betaI**2)) #+ tf.reduce_sum(self.gammaU**2) + tf.reduce_sum(self.gammaI**2))\n    \n    # Prediction for a sample of instances\n    def predictSample(self, sampleU, sampleI):\n        u = tf.convert_to_tensor(sampleU, dtype=tf.int32)\n        i = tf.convert_to_tensor(sampleI, dtype=tf.int32)\n        beta_u = tf.nn.embedding_lookup(self.betaU, u)\n        beta_i = tf.nn.embedding_lookup(self.betaI, i)\n        gamma_u = tf.nn.embedding_lookup(self.gammaU, u)\n        gamma_i = tf.nn.embedding_lookup(self.gammaI, i)\n        pred = self.alpha + beta_u + beta_i + tf.reduce_sum(tf.multiply(gamma_u, gamma_i), 1)\n        return pred\n    \n    # Loss\n    def call(self, sampleU, sampleI, sampleR):\n        pred = self.predictSample(sampleU, sampleI)\n        r = tf.convert_to_tensor(sampleR, dtype=tf.float32)\n        return tf.nn.l2_loss(pred - r) / len(sampleR)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Experiment with number of factors and regularization rate\nlf_model = LatentFactorModel(alpha, 5, 0.001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def trainingStep(data, n_samples):\n    Nsamples = n_samples\n    with tf.GradientTape() as tape:\n        sampleU, sampleI, sampleR = [], [], []\n        for _ in range(Nsamples):\n            random_index = np.random.randint(len(data))\n            u,i,r = data['userID'].iloc[random_index],data['gameID'].iloc[random_index],data['hours_transformed'].iloc[random_index]\n            sampleU.append(userIDs[u])\n            sampleI.append(gameIDs[i])\n            sampleR.append(r)\n\n        loss = lf_model(sampleU,sampleI,sampleR)\n        loss += lf_model.reg()\n    gradients = tape.gradient(loss, lf_model.trainable_variables)\n    lf_optimizer.apply_gradients((grad, var) for\n                              (grad, var) in zip(gradients, lf_model.trainable_variables)\n                              if grad is not None)\n    return loss.numpy()","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# 10 iterations of gradient descent\nfor i in range(20):\n    obj = trainingStep(df_train, 50000)\n    print(\"iteration \" + str(i) + \", objective = \" + str(obj))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_lf_predictions(data):\n    data['lf_predictions'] = [lf_model.predict(userIDs[data['userID'].iloc[i]], gameIDs[data['gameID'].iloc[i]]).numpy() for i in range(len(data))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Latent Factor Model - Simple"},{"metadata":{"trusted":false},"cell_type":"code","source":"userBiases = defaultdict(float)\nitemBiases = defaultdict(float)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Prediction model\ndef makePrediction(user, item):\n    return alpha + userBiases[user] + itemBiases[item]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Unpacks 1D theta (used for package) into offset and bias\ndef unpack(theta):\n    # Use global values instead of scoped values\n    global alpha\n    global userBiases\n    global itemBiases\n    \n    alpha = theta[0] # alpha is first element\n    userBiases = dict(zip(userIDs_train.keys(), theta[1:len(userIDs_train)+1])) # Next nUsers elements are userBiases\n    itemBiases = dict(zip(gameIDs_train.keys(), theta[1+len(userIDs_train):])) # Last nItems elements are itemBiases","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Define cost function for package\ndef cost(theta, labels, lamb):\n    # Unpack theta\n    unpack(theta)\n    \n    # Make predictions based on simple latent factor model\n    predictions = [makePrediction(df_train['userID'].iloc[i], df_train['gameID'].iloc[i]) for i in range(len(df_train))]\n    \n    # Calculate MSE\n    cost = sklearn.metrics.mean_squared_error(labels, predictions)\n    print(\"MSE = \" + str(cost))\n    \n    # Calculate cost\n    for u in userBiases:\n        cost += lamb*userBiases[u]**2\n    for i in itemBiases:\n        cost += lamb*itemBiases[i]**2\n    \n    return cost","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Calculate derivative for package; could be avoided using Tensorflow\ndef derivative(theta, labels, lamb):\n    unpack(theta)\n    N = len(df_train)\n    dalpha = 0\n    dUserBiases = defaultdict(float)\n    dItemBiases = defaultdict(float)\n    \n    for x in range(len(df_train)):\n        u,i = df_train['userID'].iloc[x], df_train['gameID'].iloc[x]\n        pred = makePrediction(u, i)\n        diff = pred - df_train['hours_transformed'].iloc[x]\n        dalpha += 2/N*diff\n        dUserBiases[u] += 2/N*diff\n        dItemBiases[i] += 2/N*diff\n    \n    for u in userBiases:\n        dUserBiases[u] += 2*lamb*userBiases[u]\n    \n    for i in itemBiases:\n        dItemBiases[i] += 2*lamb*itemBiases[i]\n    \n    dtheta = [dalpha] + [dUserBiases[u] for u in userIDs_train.keys()] + [dItemBiases[i] for i in gameIDs_train.keys()]\n    return np.array(dtheta)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"params, _, _ = scipy.optimize.fmin_l_bfgs_b(cost, [alpha] + [0.0]*(len(userIDs_train)+len(gameIDs_train)), derivative, args = (df_train['hours_transformed'], 0.001))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Unpack params to update alpha and biases\nunpack(params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_lf_simple_predictions(data):\n    data['lf_simple_predictions'] = [makePrediction(data['userID'].iloc[i], data['gameID'].iloc[i]) for i in range(len(data))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.3 Latent Factor Model - Complete"},{"metadata":{"trusted":false},"cell_type":"code","source":"userGamma = {}\nitemGamma = {}\n\nK = 0","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for u in userIDs_train.keys():\n    userGamma[u] = [random.random() * 0.1 - 0.05 for k in range(K)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"for i in gameIDs_train.keys():\n    itemGamma[i] = [random.random() * 0.1 - 0.05 for k in range(K)]","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def unpack(theta):\n    global alpha\n    global userBiases\n    global itemBiases\n    global userGamma\n    global itemGamma\n    \n    index = 0\n    alpha = theta[index]\n    index += 1\n    userBiases = dict(zip(userIDs_train.keys(), theta[index:index+len(userIDs_train)]))\n    index += len(userIDs_train)\n    itemBiases = dict(zip(gameIDs_train.keys(), theta[index:index+len(gameIDs_train)]))\n    index += len(gameIDs_train)\n    \n    for u in userIDs_train.keys():\n        userGamma[u] = theta[index:index+K]\n        index += K\n    for i in gameIDs_train.keys():\n        itemGamma[i] = theta[index:index+K]\n        index += K","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def inner(x, y):\n    return sum([a*b for a,b in zip(x,y)])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def makePrediction(user, item):\n    return alpha + userBiases[user] + itemBiases[item] + inner(userGamma[user], itemGamma[item])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def cost(theta, labels, lamb):\n    unpack(theta)\n    predictions = [makePrediction(df_train['userID'].iloc[i], df_train['gameID'].iloc[i]) for i in range(len(df_train))]\n    cost = sklearn.metrics.mean_squared_error(predictions, labels)\n    print(\"MSE = \" + str(cost))\n    \n    for u in userIDs_train.keys():\n        cost += lamb*userBiases[u]**2\n        for k in range(K):\n            cost += lamb*userGamma[u][k]**2\n    for i in gameIDs_train.keys():\n        cost += lamb*itemBiases[i]**2\n        for k in range(K):\n            cost += lamb*itemGamma[i][k]**2\n    return cost","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def derivative(theta, labels, lamb):\n    unpack(theta)\n    N = len(df_train)\n    dalpha = 0\n    dUserBiases = defaultdict(float)\n    dItemBiases = defaultdict(float)\n    dUserGamma = {}\n    dItemGamma = {}\n    \n    for u in userIDs_train.keys():\n        dUserGamma[u] = [0.0 for k in range(K)]\n    for i in gameIDs_train.keys():\n        dItemGamma[i] = [0.0 for k in range(K)]\n    \n    for x in range(len(df_train)):\n        u,i = df_train['userID'].iloc[x], df_train['gameID'].iloc[x]\n        pred = makePrediction(u, i)\n        diff = pred - df_train['hours_transformed'].iloc[x]\n        dalpha += 2/N*diff\n        dUserBiases[u] += 2/N*diff\n        dItemBiases[i] += 2/N*diff\n        for k in range(K):\n            dUserGamma[u][k] += 2/N*itemGamma[i][k]*diff\n            dItemGamma[i][k] += 2/N*userGamma[u][k]*diff\n    \n    for u in userBiases:\n        dUserBiases[u] += 2*lamb*userBiases[u]\n        for k in range(K):\n            dUserGamma[u][k] += 2*lamb*userGamma[u][k]\n    \n    for i in itemBiases:\n        dItemBiases[i] += 2*lamb*itemBiases[i]\n        for k in range(K):\n            dItemGamma[i][k] += 2*lamb*itemGamma[i][k]\n    \n    dtheta = [dalpha] + [dUserBiases[u] for u in userIDs_train.keys()] + [dItemBiases[i] for i in gameIDs_train.keys()]\n    \n    for u in userIDs_train.keys():\n        dtheta += dUserGamma[u]\n    for i in gameIDs_train.keys():\n        dtheta += dItemGamma[i]\n    return np.array(dtheta)","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"params, _, _ = scipy.optimize.fmin_l_bfgs_b(cost, [alpha] + [0.0]*(len(userIDs_train)+len(gameIDs_train)) +  [random.random() * 0.1 - 0.05 for k in range(K*(len(userIDs_train)+len(gameIDs_train)))], derivative, args = (df_train['hours_transformed'], 0.00001))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Unpack params to update alpha and biases\nunpack(params)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_lf_complete_predictions(data):\n    data['lf_complete_predictions'] = [makePrediction(data['userID'].iloc[i], data['gameID'].iloc[i]) for i in range(len(data))]","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# Hyperparameter tuning - test many different values\nlam = [0.0001, 0.001, 0.01, 0.1, 1, 5, 10]\nkk = [1, 5, 10, 20]\nscore = [0 for i in range(len(lam))]\n\nfor i in range(len(lam)):\n    score[i] = [0 for j in range(len(kk))]\n    \n    for j in range(len(kk)):\n        if i == 0 & j == 0:\n            continue\n            \n        K = kk[j]\n        params, _, _ = scipy.optimize.fmin_l_bfgs_b(cost, [alpha] + [0.0]*(len(userIDs_train)+len(gameIDs_train)) +  [random.random() * 0.1 - 0.05 for k in range(K*(len(userIDs_train)+len(gameIDs_train)))], derivative, args = (df_train['hours_transformed'], lam[i]))\n        \n        unpack(params)\n        \n        feature_construction(df_train, df_train, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\n        feature_construction(df_val, df_train, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\n\n        feat = ['lf_complete_predictions']\n\n        X_train = df_train[feat]\n        Y_train = df_train['hours_transformed']\n\n        X_val = df_val[feat]\n        Y_val = df_val['hours_transformed']\n        \n        gbc_model = gridsearchcv_model(sklearn.ensemble.GradientBoostingRegressor(), X_train, Y_train, X_val, Y_val, parameter_matrix={}, is_classification=False)\n        pickle.dump(gbc_model.model, open('./saved_models/gbc_model{}.pkl'.format(str(datetime.now())), 'wb'))\n        \n        score[i][j] = gbc_model.val['mse']\n        print(str(i), str(j), score[i][j])","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.4 One-Class Latent Factor Model"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Experiment with learning rate\nLF_optimizer_negative = tf.keras.optimizers.Adam(0.001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"class BPRbatch(tf.keras.Model):\n    def __init__(self, K, lamb):\n        super(BPRbatch, self).__init__()\n        # Initialize variables\n        self.betaI = tf.Variable(tf.random.normal([len(gameIDs_train)],stddev=0.001))\n        self.gammaU = tf.Variable(tf.random.normal([len(userIDs_train),K],stddev=0.001))\n        self.gammaI = tf.Variable(tf.random.normal([len(gameIDs_train),K],stddev=0.001))\n        # Regularization coefficient\n        self.lamb = lamb\n\n    # Prediction for a single instance\n    def predict(self, u, i):\n        p = self.betaI[i] + tf.tensordot(self.gammaU[u], self.gammaI[i], 1)\n        return p\n\n    # Regularizer\n    def reg(self):\n        return self.lamb * tf.nn.l2_loss(self.betaI) + tf.nn.l2_loss(self.gammaU) + tf.nn.l2_loss(self.gammaI)\n    \n    def score(self, sampleU, sampleI):\n        u = tf.convert_to_tensor(sampleU, dtype=tf.int32)\n        i = tf.convert_to_tensor(sampleI, dtype=tf.int32)\n        beta_i = tf.nn.embedding_lookup(self.betaI, i)\n        gamma_u = tf.nn.embedding_lookup(self.gammaU, u)\n        gamma_i = tf.nn.embedding_lookup(self.gammaI, i)\n        x_ui = beta_i + tf.reduce_sum(tf.multiply(gamma_u, gamma_i), 1)\n        return x_ui\n\n    def call(self, sampleU, sampleI, sampleJ):\n        x_ui = self.score(sampleU, sampleI)\n        x_uj = self.score(sampleU, sampleJ)\n        return -tf.reduce_mean(tf.math.log(tf.math.sigmoid(x_ui - x_uj)))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"LF_model_negative = BPRbatch(5, 0.00001)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def trainingStep(data, n_samples):\n    Nsamples = n_samples\n    with tf.GradientTape() as tape:\n        sampleU, sampleI, sampleJ = [], [], []\n        for _ in range(Nsamples):\n            random_index = np.random.randint(len(data))\n            u,i = data['userID'].iloc[random_index],data['gameID'].iloc[random_index]  # positive sample\n            j = np.random.choice(gameIDs_train) # negative sample\n            while j in gamesPerUser_train[u]:\n                j = np.random.choice(gameIDs_train)\n            sampleU.append(userIDs[u])\n            sampleI.append(gameIDs[i])\n            sampleJ.append(gameIDs[j])\n\n        loss = LF_model_negative(sampleU,sampleI,sampleJ)\n        loss += LF_model_negative.reg()\n    gradients = tape.gradient(loss, LF_model_negative.trainable_variables)\n    LF_optimizer_negative.apply_gradients((grad, var) for\n                              (grad, var) in zip(gradients, LF_model_negative.trainable_variables)\n                              if grad is not None)\n    return loss.numpy()","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Run training iterations (probably want more than 10...)\nfor i in range(10):\n    obj = trainingStep(df_train_negative, 50000)\n    print(\"iteration \" + str(i) + \", objective = \" + str(obj))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_lf_score(data):\n    data['lf_score'] = [LF_model_negative.predict(userIDs[data['userID'].iloc[i]], gameIDs[data['gameID'].iloc[i]]).numpy() for i in range(len(data))]","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 2.5 Construct Features"},{"metadata":{"trusted":false},"cell_type":"code","source":"def feature_construction(data, data_train, data_usersPerGame, data_gamesPerUser, data_hoursTransformedPerUser, data_hoursTransformedPerGame, data_hoursPerUser, data_hoursPerGame, data_datePerUser, data_datePerGame, data_earlyAccessPerUser, data_earlyAccessPerGame, data_textPerUser, data_textPerGame, data_foundFunnyPerUser, data_foundFunnyPerGame):\n    feature_user_jaccards(data, data_usersPerGame, data_gamesPerUser)\n    feature_game_jaccards(data, data_usersPerGame, data_gamesPerUser)\n    feature_user_cosine_similarities(data, data_usersPerGame, data_gamesPerUser)\n    feature_game_cosine_similarities(data, data_usersPerGame, data_gamesPerUser)\n    feature_user_pearson_correlations(data, data_usersPerGame, data_gamesPerUser, data_hoursTransformedPerGame)\n    feature_game_pearson_correlations(data, data_usersPerGame, data_gamesPerUser, data_hoursTransformedPerUser)\n    \n    feature_num_reviews(data, data_usersPerGame, data_gamesPerUser)\n    feature_hours(data, data_hoursPerGame, data_hoursPerUser)\n    feature_date(data, data_datePerUser, data_datePerGame)\n    feature_other(data, data_earlyAccessPerUser, data_earlyAccessPerGame, data_foundFunnyPerUser, data_foundFunnyPerGame) \n    \n    feature_scaled(data)\n    feature_interactions(data)\n    \n    feature_lf_score(data)\n    feature_lf_predictions(data)\n    feature_lf_simple_predictions(data)\n    feature_lf_complete_predictions(data)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Task 1\ndf_train = pd.read_pickle('df_train.pkl')\ndf_val = pd.read_pickle('df_val.pkl')\ndf_test = pd.read_pickle('df_test.pkl')\n\n# Task 2\ndf_train_negative = pd.read_pickle('df_train_negative.pkl')\ndf_val_negative = pd.read_pickle('df_val_negative.pkl')\ndf_test_negative = pd.read_pickle('df_test_negative.pkl')","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Task 1\nfeature_construction(df_train, df_train, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\nfeature_construction(df_val, df_train, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\n\ndf = pd.concat([df_train, df_val], axis=0)\ndf.index = range(len(df))\n\nfeature_construction(df_test, df, usersPerGame, gamesPerUser, hoursTransformedPerUser, hoursTransformedPerGame, hoursPerUser, hoursPerGame, datePerUser, datePerGame, earlyAccessPerUser, earlyAccessPerGame, textPerUser, textPerGame, foundFunnyPerUser, foundFunnyPerGame)\n\n# Task 2\nfeature_construction(df_train_negative, df_train_negative, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\nfeature_construction(df_val_negative, df_train_negative, usersPerGame_train, gamesPerUser_train, hoursTransformedPerUser_train, hoursTransformedPerGame_train, hoursPerUser_train, hoursPerGame_train, datePerUser_train, datePerGame_train, earlyAccessPerUser_train, earlyAccessPerGame_train, textPerUser_train, textPerGame_train, foundFunnyPerUser_train, foundFunnyPerGame_train)\n\ndf_negative = pd.concat([df_train_negative, df_val_negative], axis=0)\ndf_negative.index = range(len(df_negative))\n\nfeature_construction(df_test_negative, df_negative, usersPerGame, gamesPerUser, hoursTransformedPerUser, hoursTransformedPerGame, hoursPerUser, hoursPerGame, datePerUser, datePerGame, earlyAccessPerUser, earlyAccessPerGame, textPerUser, textPerGame, foundFunnyPerUser, foundFunnyPerGame)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Task 1\ndf_train.to_pickle('./saved_pkls/df_train.pkl')\ndf_val.to_pickle('./saved_pkls/df_val.pkl')\ndf_test.to_pickle('./saved_pkls/df_test.pkl')\n\n# Task 2\ndf_train_negative.to_pickle('./saved_pkls/df_train_negative.pkl')\ndf_val_negative.to_pickle('./saved_pkls/df_val_negative.pkl')\ndf_test_negative.to_pickle('./saved_pkls/df_test_negative.pkl')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 4. Task #2 - Time Played Prediction"},{"metadata":{},"cell_type":"markdown","source":"## 4.0 Feature Pruning"},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# Hyperparameter tuning - Determine most influential features\nselect_scores = [0 for i in range(len(all_features))]\nbest_select_scores_index = [0 for i in range(len(all_features))]\ntest_model = sklearn.model_selection.GridSearchCV(sklearn.ensemble.GradientBoostingRegressor(), {}, scoring='neg_mean_squared_error')\n\nfor i in range(len(all_features)):\n    select_scores[i] = [0 for j in range(len(all_features[i]))]\n    \n    for j in range(len(all_features[i])):\n        test_model.fit(df_train[all_features[i][j]].values.reshape(-1,1), df_train['hours_transformed'])\n        preds = test_model.predict(df_val[all_features[i][j]].values.reshape(-1,1))\n        select_scores[i][j] = sklearn.metrics.mean_squared_error(preds, df_val['hours_transformed'])\n        \n    best_select_scores_index[i] = np.argmin(scores[i])\n    \nbest_features = [all_features[i][j] for i,j in zip(range(len(best_select_scores_index)), best_select_scores_index)]\n\nbest_features","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Hyperparameter tuning - Determine best values for 'k-best' and 'pca_components'\nprune_scores = defaultdict()\ntest_model = sklearn.model_selection.GridSearchCV(sklearn.ensemble.GradientBoostingRegressor(), {}, scoring='neg_mean_squared_error')\n\nY_train = df_train['hours_transformed']\n\nfor i in range(1, len(best_features)):\n    for j in range(1, i+1):\n        X_train = df_train[best_features]\n        X_val = df_val[best_features]\n        \n        k_best = sklearn.feature_selection.SelectKBest(sklearn.feature_selection.f_regression, k=i)\n        X_train = k_best.fit_transform(X_train, Y_train)\n        X_val = k_best.transform(X_val)\n        \n        pca = sklearn.decomposition.PCA(n_components=j)\n        X_train = pca.fit_transform(X_train, Y_train)\n        X_val = pca.transform(X_val)\n        \n        test_model.fit(X_train, Y_train)\n        preds = test_model.predict(X_val)\n        prune_scores[str(i) + ',' + str(j)] = sklearn.metrics.mean_squared_error(preds, df_val['hours_transformed'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"min(prune_scores, key=scores.get)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.1 Feature Selection"},{"metadata":{"trusted":false},"cell_type":"code","source":"feat = best_features\n\nfeat = ['lf_complete_predictions']\n\nX_train = df_train[feat]\nY_train = df_train['hours_transformed']\n\nX_val = df_val[feat]\nY_val = df_val['hours_transformed']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Select K best\nk_best = sklearn.feature_selection.SelectKBest(sklearn.feature_selection.f_regression, k=5)\nX_train = k_best.fit_transform(X_train, Y_train)\nX_val = k_best.transform(X_val)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Run PCA on features\npca = sklearn.decomposition.PCA(n_components=3)\npca.fit(X_train)\n\nX_train = pca.transform(X_train)\nX_val = pca.transform(X_val)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.2 Model Selection"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Linear Regression model\nlinear_model = gridsearchcv_model(sklearn.linear_model.LinearRegression(), X_train, Y_train, X_val, Y_val, parameter_matrix={}, is_classification=False)\npickle.dump(linear_model.model, open('./saved_models/linear_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# XGBoost Regression model\n\nparams = {\"learning_rate\"    : [0.05, 0.20, 0.30 ] ,\n \"max_depth\"        : [ 3, 6, 12 ],\n \"gamma\"            : [ 0.1, 0.3 ]}\n\nxgboost_model = gridsearchcv_model(xgboost.XGBRegressor(), X_train, Y_train, X_val, Y_val, parameter_matrix=params, is_classification=False)\npickle.dump(xgboost_model.model, open('./saved_models/xgboost_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"params = {'n_estimators': [10**4],\n               'max_features': [None],\n               'max_depth': [5, 10, 30],\n               'min_samples_split': [2, 5, 10],\n               'min_samples_leaf': [2, 5, 10]}\n\nrandomForest_model = gridsearchcv_model(sklearn.ensemble.RandomForestRegressor(n_jobs=-1), X_train, Y_train, X_val, Y_val, parameter_matrix=params, is_classification=False)\npickle.dump(randomForest_model.model, open('./saved_models/randomForest_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"params = {'n_estimators': [10**4],\n               'learning_rate': [1, 0.5, 0.25, 0.1, 0.05, 0.01],\n               'max_features': [None],\n               'max_depth': [5, 10, 30],\n               'min_samples_split': [2, 5, 10, 50],\n               'min_samples_leaf': [2, 5, 10]}\n\n\ngbc_model = gridsearchcv_model(sklearn.ensemble.GradientBoostingRegressor(), X_train, Y_train, X_val, Y_val, parameter_matrix=params, is_classification=False)\npickle.dump(gbc_model.model, open('./saved_models/gbc_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"params = {'n_estimators': [10**4],\n        'learning_rate': [1, 0.5, 0.25, 0.1, 0.05, 0.01]}\n\nabc_model = gridsearchcv_model(sklearn.ensemble.AdaBoostRegressor(), X_train, Y_train, X_val, Y_val, parameter_matrix=params, is_classification=False)\npickle.dump(abc_model.model, open('./saved_models/abc_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"params = {'n_estimators': [10**4],\n        'learning_rate': [1, 0.5, 0.25, 0.1, 0.05, 0.01]}\n\nvoting_model = gridsearchcv_model(sklearn.ensemble.VotingRegressor([('xgb', xgboost_model.model), ('gbc', gbc_model.model), ('linear', linear_model.model)]), X_train, Y_train, X_val, Y_val, parameter_matrix=params, is_classification=False)\npickle.dump(voting_model.model, open('./saved_models/voting_model{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 4.3 Kaggle Submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"X_test = df_test[best_features]\nX_test = k_best.transform(X_test)\nX_test = pca.transform(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df_test['predictions'] = gbc_model.model.predict(X_test)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictionsFile = open(\"predictions_Hours.txt\", 'w')\npredictionsFile.write('userID-gameID,prediction\\n')\nfor i in range(len(df_test)):\n    predictionsFile.write(df_test['userID'].iloc[i] + '-' + df_test['gameID'].iloc[i] + \",\" + str(df_test['predictions'].iloc[i]) + \"\\n\")\n\npredictionsFile.close()","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# 5. Play Prediction"},{"metadata":{},"cell_type":"markdown","source":"## 5.0 Feature Pruning"},{"metadata":{"trusted":false},"cell_type":"code","source":"select_scores_negative = [0 for i in range(len(all_features))]\nbest_select_scores_negative_index = [0 for i in range(len(all_features))]\ntest_model = sklearn.model_selection.GridSearchCV(sklearn.linear_model.LogisticRegression(), {}, scoring='accuracy')\n\nfor i in range(len(all_features)):\n    select_scores_negative[i] = [0 for j in range(len(all_features[i]))]\n    \n    for j in range(len(all_features[i])):\n        test_model.fit(df_train_negative[all_features[i][j]].values.reshape(-1,1), df_train_negative['has_played'])\n        preds = test_model.predict(df_val_negative[all_features[i][j]].values.reshape(-1,1))\n        select_scores_negative[i][j] = sklearn.metrics.mean_squared_error(preds, df_val_negative['has_played'])\n        \n    best_select_scores_negative_index[i] = np.argmax(scores[i])\n    \nbest_features_negative = [all_features[i][j] for i,j in zip(range(len(best_select_scores_negative_index)), best_select_scores_negative_index)]\n\nbest_features_negative","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"prune_scores_negative = defaultdict()\ntest_model = sklearn.model_selection.GridSearchCV(sklearn.linear_model.LogisticRegression(), {}, scoring='accuracy')\n\nY_train_negative = df_train_negative['has_played']\n\nfor i in range(1, len(best_features_negative)):\n    for j in range(1, i+1):\n        X_train_negative = df_train_negative[best_features_negative]\n        X_val_negative = df_val_negative[best_features_negative]\n        \n        k_best_negative = sklearn.feature_selection.SelectKBest(sklearn.feature_selection.f_classif, k=i)\n        X_train_negative = k_best_negative.fit_transform(X_train_negative, Y_train_negative)\n        X_val_negative = k_best_negative.transform(X_val_negative)\n        \n        pca_negative = sklearn.decomposition.PCA(n_components=j)\n        X_train_negative = pca_negative.fit_transform(X_train_negative, Y_train_negative)\n        X_val_negative = pca_negative.transform(X_val_negative)\n        \n        test_model.fit(X_train_negative, Y_train_negative)\n        preds = test_model.predict(X_val_negative)\n        prune_scores_negative[str(i) + ',' + str(j)] = sklearn.metrics.accuracy_score(preds, df_val_negative['has_played'])","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"max(prune_scores_negative, key=scores.get)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.1 Feature Selection"},{"metadata":{"trusted":false},"cell_type":"code","source":"feat_negative = best_features_negative\n\nX_train_negative = df_train_negative[feat_negative]\nY_train_negative = df_train_negative['has_played']\n\nX_val_negative = df_val_negative[feat_negative]\nY_val_negative = df_val_negative['has_played']","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Select K best\nk_best_negative = sklearn.feature_selection.SelectKBest(sklearn.feature_selection.f_classif, k=7)\nX_train_negative = k_best_negative.fit_transform(X_train_negative, Y_train_negative)\nX_val_negative = k_best_negative.transform(X_val_negative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Run PCA on features\npca_negative = sklearn.decomposition.PCA(n_components=2)\npca_negative.fit(X_train_negative)\n\nX_train_negative = pca_negative.transform(X_train_negative)\nX_val_negative = pca_negative.transform(X_val_negative)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.2 Model Selection"},{"metadata":{"trusted":false},"cell_type":"code","source":"# Logistic Regression model\nlogistic_model_negative = gridsearchcv_model(sklearn.linear_model.LogisticRegression(), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix={}, is_classification=True)\npickle.dump(logistic_model_negative.model, open('./saved_models/logistic_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# XGBoost model\n\nparams = {\"learning_rate\"    : [0.05, 0.20, 0.30 ] ,\n \"max_depth\"        : [ 3, 6, 12 ],\n \"gamma\"            : [ 0.1, 0.3 ]}\n\nxgboost_model_negative = gridsearchcv_model(xgboost.XGBClassifier(), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix=params, is_classification=True)\npickle.dump(xgboost_model_negative.model, open('./saved_models/xgboost_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Random Forest model\nrandomForest_model_negative = gridsearchcv_model(sklearn.ensemble.RandomForestClassifier(), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix={}, is_classification=True)\npickle.dump(randomForest_model_negative.model, open('./saved_models/randomForest_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Gradient Boosting model\ngbc_model_negative = gridsearchcv_model(sklearn.ensemble.GradientBoostingClassifier(), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix={}, is_classification=True)\npickle.dump(gbc_model_negative.model, open('./saved_models/gbc_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"collapsed":true,"jupyter":{"outputs_hidden":true},"trusted":false},"cell_type":"code","source":"# AdaBoost model\nada_model_negative = gridsearchcv_model(sklearn.ensemble.AdaBoostClassifier(), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix={}, is_classification=True)\npickle.dump(ada_model_negative.model, open('./saved_models/ada_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"# Voting model\nvoting_model_negative = gridsearchcv_model(sklearn.ensemble.VotingClassifier([('logistic', logistic_model_negative.model), ('xgb', xgboost_model_negative.model), ('gbc', gbc_model_negative.model), ('ada', ada_model_negative.model)]), X_train_negative, Y_train_negative, X_val_negative, Y_val_negative, parameter_matrix={}, is_classification=True)\npickle.dump(voting_model_negative.model, open('./saved_models/voting_model_negative{}.pkl'.format(str(datetime.now())), 'wb'))","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## 5.3 Kaggle Submission"},{"metadata":{"trusted":false},"cell_type":"code","source":"X_test_negative = df_test_negative[best_features_negative]\nX_test_negative = k_best_negative.transform(X_test_negative)\nX_test_negative = pca_negative.transform(X_test_negative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"df_test_negative['predictions'] = xgboost_model_negative.model.predict(X_test_negative)","execution_count":null,"outputs":[]},{"metadata":{"trusted":false},"cell_type":"code","source":"predictionsFile = open(\"predictions_Played.txt\", 'w')\npredictionsFile.write('userID-gameID,prediction\\n')\nfor i in range(len(df_test_negative)):\n    predictionsFile.write(df_test_negative['userID'].iloc[i] + '-' + df_test_negative['gameID'].iloc[i] + \",\" + str(df_test_negative['predictions'].iloc[i]) + \"\\n\")\n\npredictionsFile.close()","execution_count":null,"outputs":[]}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":4}