{"cells":[{"metadata":{},"cell_type":"markdown","source":"## Stacking the Best Models\n<pre><b>This Kernel shows how the scores can be improved using Stacking Method.\nCredit Goes to the following kernels\nref:\n1. https://www.kaggle.com/filemide/distance-criskiev-hyparam-cont-1-662\n2. https://www.kaggle.com/criskiev/distance-is-all-you-need-lb-1-481\n3. https://www.kaggle.com/marcelotamashiro/lgb-public-kernels-plus-more-features\n4. https://www.kaggle.com/scaomath/no-memory-reduction-workflow-for-each-type-lb-1-28\n5. https://www.kaggle.com/fnands/1-mpnn/output?scriptVersionId=18233432\n6. https://www.kaggle.com/harshit92/fork-from-kernel-1-481\n7. https://www.kaggle.com/xwxw2929/keras-neural-net-and-distance-features\n8. https://www.kaggle.com/marcogorelli/criskiev-s-distances-more-estimators-groupkfold?scriptVersionId=18843561\n9. https://www.kaggle.com/toshik/schnet-starter-kit\n10.https://www.kaggle.com/abazdyrev/nn-w-o-skew\n\n</b></pre>"},{"metadata":{},"cell_type":"markdown","source":"## Stat Stack"},{"metadata":{"_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","trusted":true},"cell_type":"code","source":"import os\nimport numpy as np \nimport pandas as pd \nimport matplotlib.pyplot as plt\nfrom scipy.stats.mstats import gmean\nimport seaborn as sns\n%matplotlib inline\nfrom subprocess import check_output\nprint(check_output([\"ls\", \"../input\"]).decode(\"utf8\"))","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"sub_path = \"../input/chemistry-models\"\nall_files = os.listdir(sub_path)\nall_files","execution_count":null,"outputs":[]},{"metadata":{"_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","trusted":true},"cell_type":"code","source":"import warnings\nwarnings.filterwarnings(\"ignore\")\nouts = [pd.read_csv(os.path.join(sub_path, f), index_col=0) for f in all_files]\nconcat_sub = pd.concat(outs, axis=1)\ncols = list(map(lambda x: \"mol\" + str(x), range(len(concat_sub.columns))))\nconcat_sub.columns = cols\nconcat_sub.reset_index(inplace=True)\nconcat_sub.head()\nncol = concat_sub.shape[1]","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# check correlation\nconcat_sub.iloc[:,1:].corr()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"corr = concat_sub.iloc[:,1:].corr()\nmask = np.zeros_like(corr, dtype=np.bool)\nmask[np.triu_indices_from(mask)] = True\n\n# Set up the matplotlib figure\nf, ax = plt.subplots(figsize=(11, 9))\n\n# Draw the heatmap with the mask and correct aspect ratio\nsns.heatmap(corr, mask=mask, cmap='prism', vmin=0.96, center=0, square=True, linewidths=1, annot=True, fmt='.4f')","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"# get the data fields ready for stacking\nconcat_sub['m_max'] = concat_sub.iloc[:, 1:].max(axis=1)\nconcat_sub['m_min'] = concat_sub.iloc[:, 1:].min(axis=1)\nconcat_sub['m_median'] = concat_sub.iloc[:, 1:].median(axis=1)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub.describe()","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"cutoff_lo = 0.8\ncutoff_hi = 0.2","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Mean Stacking"},{"metadata":{"trusted":true},"cell_type":"code","source":"rank = np.tril(concat_sub.iloc[:,1:ncol].corr().values,-1)\nm_gmean = 0\nn = 8\nwhile rank.max()>0:\n    mx = np.unravel_index(rank.argmax(), rank.shape)\n    m_gmean += n*(np.log(concat_sub.iloc[:, mx[0]+1]) + np.log(concat_sub.iloc[:, mx[1]+1]))/2\n    rank[mx] = 0\n    n += 1","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['m_mean'] = np.exp(m_gmean/(n-1)**2)","execution_count":null,"outputs":[]},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = concat_sub['m_mean']\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_mean.csv', \n                                        index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Median Stacking"},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = concat_sub['m_median']\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_median.csv', \n                                        index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Pushout + Median Stacking\n>* Pushout strategy is bit aggresive"},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = np.where(np.all(concat_sub.iloc[:,1:7] > cutoff_lo, axis=1), 1, \n                                    np.where(np.all(concat_sub.iloc[:,1:7] < cutoff_hi, axis=1),\n                                             0, concat_sub['m_median']))\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_pushout_median.csv', \n                                        index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MinMax + Mean Stacking\n>* MinMax seems more gentle and it outperforms the previous one"},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = np.where(np.all(concat_sub.iloc[:,1:7] > cutoff_lo, axis=1), \n                                    concat_sub['m_max'], \n                                    np.where(np.all(concat_sub.iloc[:,1:7] < cutoff_hi, axis=1),\n                                             concat_sub['m_min'], \n                                             concat_sub['m_mean']))\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_minmax_mean.csv', \n                                        index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# MinMax + Median Stacking"},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = np.where(np.all(concat_sub.iloc[:,1:7] > cutoff_lo, axis=1), \n                                    concat_sub['m_max'], \n                                    np.where(np.all(concat_sub.iloc[:,1:7] < cutoff_hi, axis=1),\n                                             concat_sub['m_min'], \n                                             concat_sub['m_median']))\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_minmax_median.csv', \n                                        index=False, float_format='%.6f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"## Averaging Ranks"},{"metadata":{"trusted":true},"cell_type":"code","source":"concat_sub['scalar_coupling_constant'] = concat_sub['mol0'].rank(method ='min') + concat_sub['mol1'].rank(method ='min') + concat_sub['mol2'].rank(method ='min') \nconcat_sub['scalar_coupling_constant'] = (concat_sub['scalar_coupling_constant']-concat_sub['scalar_coupling_constant'].min())/(concat_sub['scalar_coupling_constant'].max() - concat_sub['scalar_coupling_constant'].min())\nconcat_sub.describe()\nconcat_sub[['id', 'scalar_coupling_constant']].to_csv('stack_rank.csv', index=False, float_format='%.8f')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"Best Results : Stack Median"}],"metadata":{"kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":3},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython3","version":"3.6.6"}},"nbformat":4,"nbformat_minor":1}