{
  "id": 437919,
  "title": "Please, some tips on how to train a model on such large dataset",
  "url": "/competitions/child-mind-institute-detect-sleep-states/discussion/437919",
  "author_name": "",
  "post_date": "2023-09-08T17:18:21.481994Z",
  "votes": 4,
  "comment_count": 3,
  "views": 0,
  "content": "<p>Hello everyone,</p>\n<p>In a past competition, I tried to train a model on a dataset with 8 million rows (which is way less than ours) and with very simple features (even simpler and fewer than ours). Just to train and validate a very basic LGBM, it took me almost 24 hours.</p>\n<p>For EDA and other tasks, it's feasible to just work with a sample of the data. However, when it comes to training, I believe that we should try to use all the available data. Using only a sample for this task will likely result in a weaker model.</p>\n<p>Can someone provide advice or share links/materials where I can find best practices for training models on such large datasets?</p>\n<p><strong>I'm looking for insights somewhere between these answers:</strong></p>\n<p><strong>a)</strong> \"Yeah man… it's tough. Sadly, technology hasn't caught up yet, and we all doomed with slow training times. Most gold medal models probably take forever to train. You should train your model in segments, saving your progress as you go.\" </p>\n<p>or</p>\n<p><strong>b)</strong> \"For such large datasets, you should consider using neural networks and activate the Kaggle GPU, as it's the fastest option available.\"</p>\n<p>or</p>\n<p><strong>c)</strong> \"There are other methods. Check out this link where this individual explains…\"</p>\n<p>Thanks for the help!!</p>",
  "messages": [
    {
      "id": "2429615",
      "postDate": "09/08/2023 17:18:21",
      "content": "<p>Hello everyone,</p>\n<p>In a past competition, I tried to train a model on a dataset with 8 million rows (which is way less than ours) and with very simple features (even simpler and fewer than ours). Just to train and validate a very basic LGBM, it took me almost 24 hours.</p>\n<p>For EDA and other tasks, it's feasible to just work with a sample of the data. However, when it comes to training, I believe that we should try to use all the available data. Using only a sample for this task will likely result in a weaker model.</p>\n<p>Can someone provide advice or share links/materials where I can find best practices for training models on such large datasets?</p>\n<p><strong>I'm looking for insights somewhere between these answers:</strong></p>\n<p><strong>a)</strong> \"Yeah man… it's tough. Sadly, technology hasn't caught up yet, and we all doomed with slow training times. Most gold medal models probably take forever to train. You should train your model in segments, saving your progress as you go.\" </p>\n<p>or</p>\n<p><strong>b)</strong> \"For such large datasets, you should consider using neural networks and activate the Kaggle GPU, as it's the fastest option available.\"</p>\n<p>or</p>\n<p><strong>c)</strong> \"There are other methods. Check out this link where this individual explains…\"</p>\n<p>Thanks for the help!!</p>",
      "rawMarkdown": "Hello everyone,\n\nIn a past competition, I tried to train a model on a dataset with 8 million rows (which is way less than ours) and with very simple features (even simpler and fewer than ours). Just to train and validate a very basic LGBM, it took me almost 24 hours.\n\nFor EDA and other tasks, it's feasible to just work with a sample of the data. However, when it comes to training, I believe that we should try to use all the available data. Using only a sample for this task will likely result in a weaker model.\n\nCan someone provide advice or share links/materials where I can find best practices for training models on such large datasets?\n\n\n**I'm looking for insights somewhere between these answers:**\n\n**a)** \"Yeah man... it's tough. Sadly, technology hasn't caught up yet, and we all doomed with slow training times. Most gold medal models probably take forever to train. You should train your model in segments, saving your progress as you go.\" \n\nor\n\n**b)** \"For such large datasets, you should consider using neural networks and activate the Kaggle GPU, as it's the fastest option available.\"\n\nor\n\n**c)** \"There are other methods. Check out this link where this individual explains...\"\n\n\nThanks for the help!!",
      "votes": null
    },
    {
      "id": "2430523",
      "postDate": "09/09/2023 10:36:24",
      "content": "<p>I'm surprised LGBM on 8 mil row rows is taking you 24 hours as it's designed to be efficient. Unless you have a crazy amount of CV-folds and/or hyperparameters slowing you down?</p>\n<p>To work with large datasets such as this one, you really need to start using GPU acceleration for data processing and model training. </p>\n<p>Check out NVIDIAs RAPIDS library!</p>",
      "rawMarkdown": "I'm surprised LGBM on 8 mil row rows is taking you 24 hours as it's designed to be efficient. Unless you have a crazy amount of CV-folds and/or hyperparameters slowing you down?\n\nTo work with large datasets such as this one, you really need to start using GPU acceleration for data processing and model training. \n\nCheck out NVIDIAs RAPIDS library!",
      "votes": null
    },
    {
      "id": "2430556",
      "postDate": "09/09/2023 11:13:27",
      "content": "<p>I'm looking for answers too so I'm leaving this comment here to get notifications :)</p>",
      "rawMarkdown": "I'm looking for answers too so I'm leaving this comment here to get notifications :)",
      "votes": null
    },
    {
      "id": "2440115",
      "postDate": "09/15/2023 09:21:28",
      "content": "<p>I might be biased as i'm working mainly with neural networks but here is a list of trick that i use when working on large datasets:</p>\n<ol>\n<li>start small : i always start with a subset of the dataset and with a small model. Doing so i can check that my pipeline is working without waiting ages for it to fail. This is especially true for this dataset that requires many transformations to be exploited (i.e reshaping the data into a list of timeseries )</li>\n<li>spend time on your evaluation procedure: make a good validation set, simulate locally your submission. I use my firsts submission to check the validity of my evaluation script rather than aiming for score. (Here the question is: will you split on series_id, step, or timestamp ?)</li>\n<li>iterate quickly at first: use a small model that trains fast, early phase of your development must be to find \"gamechangers\" that gives you a speedup or a bump in performance. Log everything (i use weight&amp;biases personally) comparing training curves is the best way to figure out which is your best run.</li>\n<li>scale your model at the very end: scaling you model too early make the tuning of your hyper parameters more difficult, so make sure you start with a good guess when trying a larger model.</li>\n</ol>\n<p>I don't know if this will work but at least  i can <strong><em>have fun</em></strong> learning/experimenting stuff without waiting age for a run to complete. ;-)</p>",
      "rawMarkdown": "I might be biased as i'm working mainly with neural networks but here is a list of trick that i use when working on large datasets:\n\n1. start small : i always start with a subset of the dataset and with a small model. Doing so i can check that my pipeline is working without waiting ages for it to fail. This is especially true for this dataset that requires many transformations to be exploited (i.e reshaping the data into a list of timeseries )\n2. spend time on your evaluation procedure: make a good validation set, simulate locally your submission. I use my firsts submission to check the validity of my evaluation script rather than aiming for score. (Here the question is: will you split on series_id, step, or timestamp ?)\n3. iterate quickly at first: use a small model that trains fast, early phase of your development must be to find \"gamechangers\" that gives you a speedup or a bump in performance. Log everything (i use weight&biases personally) comparing training curves is the best way to figure out which is your best run.\n4. scale your model at the very end: scaling you model too early make the tuning of your hyper parameters more difficult, so make sure you start with a good guess when trying a larger model.\n\nI don't know if this will work but at least  i can ***have fun*** learning/experimenting stuff without waiting age for a run to complete. ;-)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 2430523,
      "author_name": "danpietrow",
      "author_url": "",
      "post_date": "09/09/2023 10:36:24",
      "content": "<p>I'm surprised LGBM on 8 mil row rows is taking you 24 hours as it's designed to be efficient. Unless you have a crazy amount of CV-folds and/or hyperparameters slowing you down?</p>\n<p>To work with large datasets such as this one, you really need to start using GPU acceleration for data processing and model training. </p>\n<p>Check out NVIDIAs RAPIDS library!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2430556,
      "author_name": "newtoaii",
      "author_url": "",
      "post_date": "09/09/2023 11:13:27",
      "content": "<p>I'm looking for answers too so I'm leaving this comment here to get notifications :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 2440115,
      "author_name": "tboissin",
      "author_url": "",
      "post_date": "09/15/2023 09:21:28",
      "content": "<p>I might be biased as i'm working mainly with neural networks but here is a list of trick that i use when working on large datasets:</p>\n<ol>\n<li>start small : i always start with a subset of the dataset and with a small model. Doing so i can check that my pipeline is working without waiting ages for it to fail. This is especially true for this dataset that requires many transformations to be exploited (i.e reshaping the data into a list of timeseries )</li>\n<li>spend time on your evaluation procedure: make a good validation set, simulate locally your submission. I use my firsts submission to check the validity of my evaluation script rather than aiming for score. (Here the question is: will you split on series_id, step, or timestamp ?)</li>\n<li>iterate quickly at first: use a small model that trains fast, early phase of your development must be to find \"gamechangers\" that gives you a speedup or a bump in performance. Log everything (i use weight&amp;biases personally) comparing training curves is the best way to figure out which is your best run.</li>\n<li>scale your model at the very end: scaling you model too early make the tuning of your hyper parameters more difficult, so make sure you start with a good guess when trying a larger model.</li>\n</ol>\n<p>I don't know if this will work but at least  i can <strong><em>have fun</em></strong> learning/experimenting stuff without waiting age for a run to complete. ;-)</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "2429615": "Hello everyone,\n\nIn a past competition, I tried to train a model on a dataset with 8 million rows (which is way less than ours) and with very simple features (even simpler and fewer than ours). Just to train and validate a very basic LGBM, it took me almost 24 hours.\n\nFor EDA and other tasks, it's feasible to just work with a sample of the data. However, when it comes to training, I believe that we should try to use all the available data. Using only a sample for this task will likely result in a weaker model.\n\nCan someone provide advice or share links/materials where I can find best practices for training models on such large datasets?\n\n\n**I'm looking for insights somewhere between these answers:**\n\n**a)** \"Yeah man... it's tough. Sadly, technology hasn't caught up yet, and we all doomed with slow training times. Most gold medal models probably take forever to train. You should train your model in segments, saving your progress as you go.\" \n\nor\n\n**b)** \"For such large datasets, you should consider using neural networks and activate the Kaggle GPU, as it's the fastest option available.\"\n\nor\n\n**c)** \"There are other methods. Check out this link where this individual explains...\"\n\n\nThanks for the help!!",
    "2430523": "I'm surprised LGBM on 8 mil row rows is taking you 24 hours as it's designed to be efficient. Unless you have a crazy amount of CV-folds and/or hyperparameters slowing you down?\n\nTo work with large datasets such as this one, you really need to start using GPU acceleration for data processing and model training. \n\nCheck out NVIDIAs RAPIDS library!",
    "2430556": "I'm looking for answers too so I'm leaving this comment here to get notifications :)",
    "2440115": "I might be biased as i'm working mainly with neural networks but here is a list of trick that i use when working on large datasets:\n\n1. start small : i always start with a subset of the dataset and with a small model. Doing so i can check that my pipeline is working without waiting ages for it to fail. This is especially true for this dataset that requires many transformations to be exploited (i.e reshaping the data into a list of timeseries )\n2. spend time on your evaluation procedure: make a good validation set, simulate locally your submission. I use my firsts submission to check the validity of my evaluation script rather than aiming for score. (Here the question is: will you split on series_id, step, or timestamp ?)\n3. iterate quickly at first: use a small model that trains fast, early phase of your development must be to find \"gamechangers\" that gives you a speedup or a bump in performance. Log everything (i use weight&biases personally) comparing training curves is the best way to figure out which is your best run.\n4. scale your model at the very end: scaling you model too early make the tuning of your hyper parameters more difficult, so make sure you start with a good guess when trying a larger model.\n\nI don't know if this will work but at least  i can ***have fun*** learning/experimenting stuff without waiting age for a run to complete. ;-)"
  },
  "source": "meta"
}