{
  "id": 80673,
  "title": "How do you record your model performance?",
  "url": "/competitions/elo-merchant-category-recommendation/discussion/80673",
  "author_name": "",
  "post_date": "2019-02-15T10:48:32.036616900Z",
  "votes": 1,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hey everyone, i mean that we usually use different features and different hyparameters , and it outputs lots of scores. Currently i use Excel to record my features丶model used丶hyparameters丶cv_scores丶lb_scores ..., but i think it is not the best way . Could you share how you organize these progress?  </p>",
  "messages": [
    {
      "id": "472099",
      "postDate": "02/15/2019 10:48:32",
      "content": "<p>Hey everyone, i mean that we usually use different features and different hyparameters , and it outputs lots of scores. Currently i use Excel to record my features丶model used丶hyparameters丶cv_scores丶lb_scores ..., but i think it is not the best way . Could you share how you organize these progress?  </p>",
      "rawMarkdown": "Hey everyone, i mean that we usually use different features and different hyparameters , and it outputs lots of scores. Currently i use Excel to record my features丶model used丶hyparameters丶cv_scores丶lb_scores ..., but i think it is not the best way . Could you share how you organize these progress?",
      "votes": null
    },
    {
      "id": "472160",
      "postDate": "02/15/2019 12:35:01",
      "content": "<p>Hi Rui,</p>\n\n<p>I was scratching my head for the same question long back. I have seen people using multiple approaches for this:</p>\n\n<ul>\n<li><p><em>Using Excel</em></p></li>\n<li><p><em>Writing Python utilities</em></p></li>\n<li><p><em>Plotting the metrics</em></p></li>\n</ul>\n\n<p>I personally like plotting all metrics for different models. It's very easy to compare!</p>\n\n<p>I strongly recommend below Kernel by Mr. Pavan Raj. This kernel is awesome, where Pavan is implementing close to dozen models and comparing all using different visualizations!</p>\n\n<p><a href=\"https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction\">https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction</a></p>\n\n<p><img src=\"https://i.screenshot.net/66yyxcg\" alt=\"Comparison\"></p>\n\n<p>HTH</p>",
      "rawMarkdown": "Hi Rui,\n\nI was scratching my head for the same question long back. I have seen people using multiple approaches for this:\n\n* *Using Excel*\n\n* *Writing Python utilities*\n\n* *Plotting the metrics*\n\nI personally like plotting all metrics for different models. It's very easy to compare!\n\nI strongly recommend below Kernel by Mr. Pavan Raj. This kernel is awesome, where Pavan is implementing close to dozen models and comparing all using different visualizations!\n\nhttps://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction\n\n![Comparison](https://i.screenshot.net/66yyxcg)\n\nHTH",
      "votes": null
    },
    {
      "id": "472685",
      "postDate": "02/16/2019 13:20:15",
      "content": "<p>Using Excel.</p>",
      "rawMarkdown": "Using Excel.",
      "votes": null
    },
    {
      "id": "472773",
      "postDate": "02/16/2019 16:37:18",
      "content": "<p>I also use Excel, with some automation. I use code to append a csv file with timestamp, model name, hyperparams, cv score. I work with the file in Excel and add lb scores and notes manually. One of these days I plan to use the Kaggle API and include lb scores and submission notes.</p>",
      "rawMarkdown": "I also use Excel, with some automation. I use code to append a csv file with timestamp, model name, hyperparams, cv score. I work with the file in Excel and add lb scores and notes manually. One of these days I plan to use the Kaggle API and include lb scores and submission notes.",
      "votes": null
    },
    {
      "id": "472775",
      "postDate": "02/16/2019 16:38:46",
      "content": "<p>Inside each fold training\n```\n                    df_metric_tmp = pd.DataFrame({'fold': [fold_],\n                                                  'training': [model_gbm.best_score['training'][param['metric']]],\n                                                  'valid': [model_gbm.best_score['valid_1'][param['metric']]],\n                                                  'best_iter': [model_gbm.best_iteration]})</p>\n\n<pre><code>                df_metric = pd.concat([df_metric, df_metric_tmp], sort=True)\n</code></pre>\n\n<p><code>\nThen\n</code>\nutils.save_metric(df_metric, path=fname_df_metric)\n<code>\nWith\n</code>\ndef save_metric(df_metric, path='output/df_metric.csv'):\n    df_metric['timestamp'] = pd.Timestamp.today().round(\"S\")\n    try:\n        df_metric_old = pd.read_csv(path)\n        df_metric = pd.concat([df_metric_old, df_metric], sort=True)\n    except Exception as e:\n        print(e)</p>\n\n<pre><code>df_metric = df_metric.drop_duplicates()\ndf_metric.to_csv(path, index=False)\n\nprint('df_metric saved.')\nreturn None\n</code></pre>\n\n<p>```</p>\n\n<p>And also while submitting</p>\n\n<p>```\nmessage = \"local_score: {}\".format(local_score, 3)\nprint(message)\ndf_test[['card_id', 'target']].to_csv('output/submission.csv', index=False)</p>\n\n<p>cmd_submit_kaggle = 'kaggle competitions submit -c elo-merchant-category-recommendation -f output/submission.csv -m \"{} col_submit : {}\"  '.format(message, col_submit)\nprint(cmd_submit_kaggle)\n```</p>\n\n<p>Obviously it makes sense if and only if folders stay the same. </p>\n\n<p>Working with git is also a must. </p>",
      "rawMarkdown": "Inside each fold training\n```\n                    df_metric_tmp = pd.DataFrame({'fold': [fold_],\n                                                  'training': [model_gbm.best_score['training'][param['metric']]],\n                                                  'valid': [model_gbm.best_score['valid_1'][param['metric']]],\n                                                  'best_iter': [model_gbm.best_iteration]})\n\n                    df_metric = pd.concat([df_metric, df_metric_tmp], sort=True)\n```\nThen\n```\nutils.save_metric(df_metric, path=fname_df_metric)\n```\nWith\n```\ndef save_metric(df_metric, path='output/df_metric.csv'):\n    df_metric['timestamp'] = pd.Timestamp.today().round(\"S\")\n    try:\n        df_metric_old = pd.read_csv(path)\n        df_metric = pd.concat([df_metric_old, df_metric], sort=True)\n    except Exception as e:\n        print(e)\n\n    df_metric = df_metric.drop_duplicates()\n    df_metric.to_csv(path, index=False)\n\n    print('df_metric saved.')\n    return None\n```\n\nAnd also while submitting\n\n```\nmessage = \"local_score: {}\".format(local_score, 3)\nprint(message)\ndf_test[['card_id', 'target']].to_csv('output/submission.csv', index=False)\n\ncmd_submit_kaggle = 'kaggle competitions submit -c elo-merchant-category-recommendation -f output/submission.csv -m \"{} col_submit : {}\"  '.format(message, col_submit)\nprint(cmd_submit_kaggle)\n```\n\nObviously it makes sense if and only if folders stay the same. \n\nWorking with git is also a must.",
      "votes": null
    },
    {
      "id": "473024",
      "postDate": "02/17/2019 05:36:33",
      "content": "<p>It helps a lot ,thank you! \nBut it seems that your method does not include features. \nWhen you come up a new feature , you use it to predict test, would you write down this new feature in your excel file?</p>",
      "rawMarkdown": "It helps a lot ,thank you! \nBut it seems that your method does not include features. \nWhen you come up a new feature , you use it to predict test, would you write down this new feature in your excel file?",
      "votes": null
    },
    {
      "id": "473027",
      "postDate": "02/17/2019 05:40:08",
      "content": "<p>Thank you for your patience.I will try that later.</p>",
      "rawMarkdown": "Thank you for your patience.I will try that later.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 472160,
      "author_name": "vjcalling",
      "author_url": "",
      "post_date": "02/15/2019 12:35:01",
      "content": "<p>Hi Rui,</p>\n\n<p>I was scratching my head for the same question long back. I have seen people using multiple approaches for this:</p>\n\n<ul>\n<li><p><em>Using Excel</em></p></li>\n<li><p><em>Writing Python utilities</em></p></li>\n<li><p><em>Plotting the metrics</em></p></li>\n</ul>\n\n<p>I personally like plotting all metrics for different models. It's very easy to compare!</p>\n\n<p>I strongly recommend below Kernel by Mr. Pavan Raj. This kernel is awesome, where Pavan is implementing close to dozen models and comparing all using different visualizations!</p>\n\n<p><a href=\"https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction\">https://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction</a></p>\n\n<p><img src=\"https://i.screenshot.net/66yyxcg\" alt=\"Comparison\"></p>\n\n<p>HTH</p>",
      "votes": null,
      "replies": [
        {
          "id": 473027,
          "author_name": "destiny3027",
          "author_url": "",
          "post_date": "02/17/2019 05:40:08",
          "content": "<p>Thank you for your patience.I will try that later.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 472685,
      "author_name": "longyin2",
      "author_url": "",
      "post_date": "02/16/2019 13:20:15",
      "content": "<p>Using Excel.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 472773,
      "author_name": "jpmiller",
      "author_url": "",
      "post_date": "02/16/2019 16:37:18",
      "content": "<p>I also use Excel, with some automation. I use code to append a csv file with timestamp, model name, hyperparams, cv score. I work with the file in Excel and add lb scores and notes manually. One of these days I plan to use the Kaggle API and include lb scores and submission notes.</p>",
      "votes": null,
      "replies": [
        {
          "id": 473024,
          "author_name": "destiny3027",
          "author_url": "",
          "post_date": "02/17/2019 05:36:33",
          "content": "<p>It helps a lot ,thank you! \nBut it seems that your method does not include features. \nWhen you come up a new feature , you use it to predict test, would you write down this new feature in your excel file?</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 472775,
      "author_name": "jacquespeeters",
      "author_url": "",
      "post_date": "02/16/2019 16:38:46",
      "content": "<p>Inside each fold training\n```\n                    df_metric_tmp = pd.DataFrame({'fold': [fold_],\n                                                  'training': [model_gbm.best_score['training'][param['metric']]],\n                                                  'valid': [model_gbm.best_score['valid_1'][param['metric']]],\n                                                  'best_iter': [model_gbm.best_iteration]})</p>\n\n<pre><code>                df_metric = pd.concat([df_metric, df_metric_tmp], sort=True)\n</code></pre>\n\n<p><code>\nThen\n</code>\nutils.save_metric(df_metric, path=fname_df_metric)\n<code>\nWith\n</code>\ndef save_metric(df_metric, path='output/df_metric.csv'):\n    df_metric['timestamp'] = pd.Timestamp.today().round(\"S\")\n    try:\n        df_metric_old = pd.read_csv(path)\n        df_metric = pd.concat([df_metric_old, df_metric], sort=True)\n    except Exception as e:\n        print(e)</p>\n\n<pre><code>df_metric = df_metric.drop_duplicates()\ndf_metric.to_csv(path, index=False)\n\nprint('df_metric saved.')\nreturn None\n</code></pre>\n\n<p>```</p>\n\n<p>And also while submitting</p>\n\n<p>```\nmessage = \"local_score: {}\".format(local_score, 3)\nprint(message)\ndf_test[['card_id', 'target']].to_csv('output/submission.csv', index=False)</p>\n\n<p>cmd_submit_kaggle = 'kaggle competitions submit -c elo-merchant-category-recommendation -f output/submission.csv -m \"{} col_submit : {}\"  '.format(message, col_submit)\nprint(cmd_submit_kaggle)\n```</p>\n\n<p>Obviously it makes sense if and only if folders stay the same. </p>\n\n<p>Working with git is also a must. </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "472099": "Hey everyone, i mean that we usually use different features and different hyparameters , and it outputs lots of scores. Currently i use Excel to record my features丶model used丶hyparameters丶cv_scores丶lb_scores ..., but i think it is not the best way . Could you share how you organize these progress?",
    "472160": "Hi Rui,\n\nI was scratching my head for the same question long back. I have seen people using multiple approaches for this:\n\n* *Using Excel*\n\n* *Writing Python utilities*\n\n* *Plotting the metrics*\n\nI personally like plotting all metrics for different models. It's very easy to compare!\n\nI strongly recommend below Kernel by Mr. Pavan Raj. This kernel is awesome, where Pavan is implementing close to dozen models and comparing all using different visualizations!\n\nhttps://www.kaggle.com/pavanraj159/telecom-customer-churn-prediction\n\n![Comparison](https://i.screenshot.net/66yyxcg)\n\nHTH",
    "472685": "Using Excel.",
    "472773": "I also use Excel, with some automation. I use code to append a csv file with timestamp, model name, hyperparams, cv score. I work with the file in Excel and add lb scores and notes manually. One of these days I plan to use the Kaggle API and include lb scores and submission notes.",
    "472775": "Inside each fold training\n```\n                    df_metric_tmp = pd.DataFrame({'fold': [fold_],\n                                                  'training': [model_gbm.best_score['training'][param['metric']]],\n                                                  'valid': [model_gbm.best_score['valid_1'][param['metric']]],\n                                                  'best_iter': [model_gbm.best_iteration]})\n\n                    df_metric = pd.concat([df_metric, df_metric_tmp], sort=True)\n```\nThen\n```\nutils.save_metric(df_metric, path=fname_df_metric)\n```\nWith\n```\ndef save_metric(df_metric, path='output/df_metric.csv'):\n    df_metric['timestamp'] = pd.Timestamp.today().round(\"S\")\n    try:\n        df_metric_old = pd.read_csv(path)\n        df_metric = pd.concat([df_metric_old, df_metric], sort=True)\n    except Exception as e:\n        print(e)\n\n    df_metric = df_metric.drop_duplicates()\n    df_metric.to_csv(path, index=False)\n\n    print('df_metric saved.')\n    return None\n```\n\nAnd also while submitting\n\n```\nmessage = \"local_score: {}\".format(local_score, 3)\nprint(message)\ndf_test[['card_id', 'target']].to_csv('output/submission.csv', index=False)\n\ncmd_submit_kaggle = 'kaggle competitions submit -c elo-merchant-category-recommendation -f output/submission.csv -m \"{} col_submit : {}\"  '.format(message, col_submit)\nprint(cmd_submit_kaggle)\n```\n\nObviously it makes sense if and only if folders stay the same. \n\nWorking with git is also a must.",
    "473024": "It helps a lot ,thank you! \nBut it seems that your method does not include features. \nWhen you come up a new feature , you use it to predict test, would you write down this new feature in your excel file?",
    "473027": "Thank you for your patience.I will try that later."
  },
  "source": "meta"
}