{"cells":[{"metadata":{},"cell_type":"markdown","source":"# Which model performed the best ?\n\n\nIn this competition tree baseed solutions like LGBM have been the first superstars, mainly because their performances allowed to process the amount of examples which random forrest could not for example. Then they seem to have been superseded in performances by graph based solutions.\n\nWhile top ranking teams disclose their solution I'll keep this notebook updated with the type of model that worked to have a general view of their respective performances.\n"},{"metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"import numpy as np # linear algebra\nimport pandas as pd # data processing, CSV file I/O (e.g. pd.read_csv)\nimport matplotlib.pyplot as plt\nimport seaborn as sns\nsns.set()","execution_count":null,"outputs":[]},{"metadata":{"_uuid":"d629ff2d2480ee46fbb7e2d37f6b5fab8052498a","_cell_guid":"79c7e3d0-c299-4dcb-8224-4455121ee9b0","trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"lb = pd.read_csv('../input/champslb20190828/publicleaderboarddata (1)/champs-scalar-coupling-publicleaderboard.csv')","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"# Histogram view of public LB\n"},{"metadata":{"trusted":true,"_kg_hide-input":true},"cell_type":"code","source":"fig, ax = plt.subplots(1,1, figsize=(16,4))\nscores = lb[['TeamId','Score']].groupby('TeamId').min().sort_values(by='Score').Score\n#sns.distplot(scores, bins = np.linspace(-3.5, 2, 91), kde=False, ax=ax)\nax.hist(scores, bins = np.linspace(-3.3, 2, 111))\nxl = plt.xlim([-3.3, 2],)\nax.set_ylim(0,125)\nax.set_xticks(np.linspace(-3.3, 2, 26))\nax.set_xticklabels([ f\"{v:.1f}\" for v in np.linspace(-3.3, 2, 26)])\n\npublic_kernels = [\n    (\"@artgor EDA\", 0.289),\n    (\"@artgor Brute Force\", -0.595),\n    (\"@todnewman NN\", -1.073),\n    #(\"@giba\", -1.17),\n    (\"@fnand MPNN\", -1.28),\n    (\"@xwxw2929 NN\", -1.672),\n    (\"@iooohoooi Stack\", -1.829),\n    (\"Public leak\", -2.16),\n]\n\nbbox  = dict(boxstyle=\"round\", fc=\"0.8\")\narrow = dict(arrowstyle=\"->\",connectionstyle = \"angle,angleA=45,angleB=90,rad=10\",\n    edgecolor=\"darkgray\",linewidth=2)\n\nfor who, score in public_kernels:\n    ax.annotate(who, xy=(score, 0.1), xytext=(score, 130), rotation=45, bbox=bbox,arrowprops=arrow)","execution_count":null,"outputs":[]},{"metadata":{},"cell_type":"markdown","source":"A few public kernels have beel labelled so we can see on the LB big densities corresponding to the \"clickers\". The biggest one correspond has count=300 and correspond to the best public kernel (stacking). The scale has been clipped to show the distibution more clearly.\n\nApart from the top rankers, there is an interresting bump around -2.5, is it MPNN from Gilmer paper?\n\nFinally a spurious bump appear at the expected location of the public leaked -2.16 kernels, this might lead to animated discussions ;-)"},{"metadata":{},"cell_type":"markdown","source":"# Team models\n\n"},{"metadata":{},"cell_type":"markdown","source":"| Rank  | Score  | Team | Model\n|---|---|---|---|---|\n| 2   | -3.22349 | 🤖 Quantum Uncertainty 🤖 | [Transformer architecture based](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106468) -- [Attention Is All You Need](https://arxiv.org/abs/1706.03762)\n| 6   | -3.04456 |Robin N          |  MPNN based with [Attention layer](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106407)\n| 7   | -3.03280 | Pinkman Chemistry Lab | NN based with [LGB post processing](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106421)\n| 8   | -3.00189 | 4 GM and the brain  | Custom SchNet based using [densely connected GNN](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106347)\n| 10  | -2.95233 | Kha Zidmie Josh Kyle Akira | SchNEt based with [2 models pipeline](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106293)  |\n| 12  | -2.93335 | Team Bird Brain | [Covariant Molecular NN](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106275) -- [source article](https://arxiv.org/abs/1906.04015)\n| 13  | -2.92387 | KM | [SchNet based](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106377)\n| 14  | -2.90753 | Squid  | [Handicrafted GCN](https://www.kaggle.com/c/champs-scalar-coupling/discussion/106298) \n\n-----------------\n\nThe picture start to appear more clearly\n* The two best shared solutions uses both Transformer Encoder\n* The pinkman's NN based solution is competitive at 7th place (uses complex architecture and postprocessing)\n* Several flavors of GNN have competitive results among whic 3 uses SchNet\n"},{"metadata":{},"cell_type":"markdown","source":""}],"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat":4,"nbformat_minor":1}