{
  "id": 287088,
  "title": "Has Anyone Successfully Obtained Scores by Using More than One Model",
  "url": "/competitions/g-research-crypto-forecasting/discussion/287088",
  "author_name": "",
  "post_date": "2021-11-12T05:31:52.813503Z",
  "votes": 12,
  "comment_count": 6,
  "views": 0,
  "content": "<p>Hi,</p>\n<p>If we use k fold to split data, and k = 2, then we will train 2 models, and get the final result by taking the average of 2 predictions. The problem I am facing is that if I use more than 1 model to make predictions. I will get timeout error. </p>\n<p>For example, if I use</p>\n<pre><code>model = load_model('somewhere/model')\npred = model.predict(test_file)\n</code></pre>\n<p>Then I will get the score in the end after about 6 hours, but if I use </p>\n<pre><code>model_fold_1 = load_model('somewhere/model_fold_1')\nmodel_fold_2 = load_model('somewhere/model_fold_2')\npred = (model_fold_1.predict(test_file) + model_fold_2.predict(test_file)) / 2\n</code></pre>\n<p>I will get the timeout error in the end after 9 hour running.</p>\n<p>Has anyone managed to make the prediction without getting timeout error by using more than one model? How do you do that?</p>\n<p>Thanks!</p>\n<hr>\n<p>Models are loaded ahead of time and are saved in a list instead of loading each time in each loop. In each loop of prediction all I do is preprocessing data and making predictions. </p>",
  "messages": [
    {
      "id": "1579661",
      "postDate": "11/12/2021 05:31:52",
      "content": "<p>Hi,</p>\n<p>If we use k fold to split data, and k = 2, then we will train 2 models, and get the final result by taking the average of 2 predictions. The problem I am facing is that if I use more than 1 model to make predictions. I will get timeout error. </p>\n<p>For example, if I use</p>\n<pre><code>model = load_model('somewhere/model')\npred = model.predict(test_file)\n</code></pre>\n<p>Then I will get the score in the end after about 6 hours, but if I use </p>\n<pre><code>model_fold_1 = load_model('somewhere/model_fold_1')\nmodel_fold_2 = load_model('somewhere/model_fold_2')\npred = (model_fold_1.predict(test_file) + model_fold_2.predict(test_file)) / 2\n</code></pre>\n<p>I will get the timeout error in the end after 9 hour running.</p>\n<p>Has anyone managed to make the prediction without getting timeout error by using more than one model? How do you do that?</p>\n<p>Thanks!</p>\n<hr>\n<p>Models are loaded ahead of time and are saved in a list instead of loading each time in each loop. In each loop of prediction all I do is preprocessing data and making predictions. </p>",
      "rawMarkdown": "Hi,\n\nIf we use k fold to split data, and k = 2, then we will train 2 models, and get the final result by taking the average of 2 predictions. The problem I am facing is that if I use more than 1 model to make predictions. I will get timeout error. \n\nFor example, if I use\n```\nmodel = load_model('somewhere/model')\npred = model.predict(test_file)\n\n```\nThen I will get the score in the end after about 6 hours, but if I use \n```\nmodel_fold_1 = load_model('somewhere/model_fold_1')\nmodel_fold_2 = load_model('somewhere/model_fold_2')\npred = (model_fold_1.predict(test_file) + model_fold_2.predict(test_file)) / 2\n\n```\nI will get the timeout error in the end after 9 hour running.\n\nHas anyone managed to make the prediction without getting timeout error by using more than one model? How do you do that?\n\nThanks!\n\n\n--------------\nModels are loaded ahead of time and are saved in a list instead of loading each time in each loop. In each loop of prediction all I do is preprocessing data and making predictions.",
      "votes": null
    },
    {
      "id": "1579794",
      "postDate": "11/12/2021 07:56:13",
      "content": "<p>I remember this being a big topic for JS. In the end people managed to use ensemble so I would believe this to be possible here too. It probably means you have to:</p>\n<ul>\n<li>make sure the preprocessing run fast (numpy instead of pandas)</li>\n<li>benchmark your model for individual inference time</li>\n<li>go back trough JS discussion to see some techniques (I remember about Tflite for tf / keras and at least one of the 'in the money' solution reimplementing whole NN in numpy).</li>\n</ul>",
      "rawMarkdown": "I remember this being a big topic for JS. In the end people managed to use ensemble so I would believe this to be possible here too. It probably means you have to:\n- make sure the preprocessing run fast (numpy instead of pandas)\n- benchmark your model for individual inference time\n- go back trough JS discussion to see some techniques (I remember about Tflite for tf / keras and at least one of the 'in the money' solution reimplementing whole NN in numpy).",
      "votes": null
    },
    {
      "id": "1580632",
      "postDate": "11/13/2021 00:51:05",
      "content": "<p>Hi, I will start thinking about how to preprocess data by using numpy array. Thanks!</p>",
      "rawMarkdown": "Hi, I will start thinking about how to preprocess data by using numpy array. Thanks!",
      "votes": null
    },
    {
      "id": "1580679",
      "postDate": "11/13/2021 02:49:27",
      "content": "<p>The scoring takes more than 2/3 of the time taken , So in most of cases the scoring background job is the bottle neck</p>",
      "rawMarkdown": "The scoring takes more than 2/3 of the time taken , So in most of cases the scoring background job is the bottle neck",
      "votes": null
    },
    {
      "id": "1590311",
      "postDate": "11/21/2021 05:02:57",
      "content": "<p>my single model ran for 8 hours and got a score</p>",
      "rawMarkdown": "my single model ran for 8 hours and got a score",
      "votes": null
    },
    {
      "id": "1651623",
      "postDate": "01/15/2022 22:43:22",
      "content": "<p>Are you talking about multiple models as in one per Asset type? Or are you using one model which has learnt across multiple asset classes - I wonder if this learning helps? What has the experience been across the board, I'm quite curious about this.</p>",
      "rawMarkdown": "Are you talking about multiple models as in one per Asset type? Or are you using one model which has learnt across multiple asset classes - I wonder if this learning helps? What has the experience been across the board, I'm quite curious about this.",
      "votes": null
    },
    {
      "id": "1651636",
      "postDate": "01/15/2022 23:06:13",
      "content": "<p>Hi, I mean multiple models, each trained with all the data.  I know that, of course, the public notebook trains 13 models on 13 asset data, but the fact is that each model is not trained by using the whole data, but one piece of the data. I wonder whether someone could train multiple models which are all trained with the whole data to do the inference.</p>",
      "rawMarkdown": "Hi, I mean multiple models, each trained with all the data.  I know that, of course, the public notebook trains 13 models on 13 asset data, but the fact is that each model is not trained by using the whole data, but one piece of the data. I wonder whether someone could train multiple models which are all trained with the whole data to do the inference.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1579794,
      "author_name": "lucasmorin",
      "author_url": "",
      "post_date": "11/12/2021 07:56:13",
      "content": "<p>I remember this being a big topic for JS. In the end people managed to use ensemble so I would believe this to be possible here too. It probably means you have to:</p>\n<ul>\n<li>make sure the preprocessing run fast (numpy instead of pandas)</li>\n<li>benchmark your model for individual inference time</li>\n<li>go back trough JS discussion to see some techniques (I remember about Tflite for tf / keras and at least one of the 'in the money' solution reimplementing whole NN in numpy).</li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 1580632,
          "author_name": "yus002",
          "author_url": "",
          "post_date": "11/13/2021 00:51:05",
          "content": "<p>Hi, I will start thinking about how to preprocess data by using numpy array. Thanks!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 1580679,
      "author_name": "saumilagrawal10",
      "author_url": "",
      "post_date": "11/13/2021 02:49:27",
      "content": "<p>The scoring takes more than 2/3 of the time taken , So in most of cases the scoring background job is the bottle neck</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1590311,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "11/21/2021 05:02:57",
      "content": "<p>my single model ran for 8 hours and got a score</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1651623,
      "author_name": "neomatrix369",
      "author_url": "",
      "post_date": "01/15/2022 22:43:22",
      "content": "<p>Are you talking about multiple models as in one per Asset type? Or are you using one model which has learnt across multiple asset classes - I wonder if this learning helps? What has the experience been across the board, I'm quite curious about this.</p>",
      "votes": null,
      "replies": [
        {
          "id": 1651636,
          "author_name": "yus002",
          "author_url": "",
          "post_date": "01/15/2022 23:06:13",
          "content": "<p>Hi, I mean multiple models, each trained with all the data.  I know that, of course, the public notebook trains 13 models on 13 asset data, but the fact is that each model is not trained by using the whole data, but one piece of the data. I wonder whether someone could train multiple models which are all trained with the whole data to do the inference.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1579661": "Hi,\n\nIf we use k fold to split data, and k = 2, then we will train 2 models, and get the final result by taking the average of 2 predictions. The problem I am facing is that if I use more than 1 model to make predictions. I will get timeout error. \n\nFor example, if I use\n```\nmodel = load_model('somewhere/model')\npred = model.predict(test_file)\n\n```\nThen I will get the score in the end after about 6 hours, but if I use \n```\nmodel_fold_1 = load_model('somewhere/model_fold_1')\nmodel_fold_2 = load_model('somewhere/model_fold_2')\npred = (model_fold_1.predict(test_file) + model_fold_2.predict(test_file)) / 2\n\n```\nI will get the timeout error in the end after 9 hour running.\n\nHas anyone managed to make the prediction without getting timeout error by using more than one model? How do you do that?\n\nThanks!\n\n\n--------------\nModels are loaded ahead of time and are saved in a list instead of loading each time in each loop. In each loop of prediction all I do is preprocessing data and making predictions.",
    "1579794": "I remember this being a big topic for JS. In the end people managed to use ensemble so I would believe this to be possible here too. It probably means you have to:\n- make sure the preprocessing run fast (numpy instead of pandas)\n- benchmark your model for individual inference time\n- go back trough JS discussion to see some techniques (I remember about Tflite for tf / keras and at least one of the 'in the money' solution reimplementing whole NN in numpy).",
    "1580632": "Hi, I will start thinking about how to preprocess data by using numpy array. Thanks!",
    "1580679": "The scoring takes more than 2/3 of the time taken , So in most of cases the scoring background job is the bottle neck",
    "1590311": "my single model ran for 8 hours and got a score",
    "1651623": "Are you talking about multiple models as in one per Asset type? Or are you using one model which has learnt across multiple asset classes - I wonder if this learning helps? What has the experience been across the board, I'm quite curious about this.",
    "1651636": "Hi, I mean multiple models, each trained with all the data.  I know that, of course, the public notebook trains 13 models on 13 asset data, but the fact is that each model is not trained by using the whole data, but one piece of the data. I wonder whether someone could train multiple models which are all trained with the whole data to do the inference."
  },
  "source": "meta"
}