{
  "id": 330642,
  "title": "Trying to Allocate More Memory.... !!",
  "url": "/competitions/be-honest-what-do-you-think-of-this-image/discussion/330642",
  "author_name": "",
  "post_date": "2022-06-13T14:02:31.225438500Z",
  "votes": 1,
  "comment_count": 2,
  "views": 0,
  "content": "<p>Need to load 18k+ images but can't train a model kernal crashes…..</p>\n<p>This is my notebook,<br>\n<a href=\"https://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow\" target=\"_blank\">https://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow</a></p>",
  "messages": [
    {
      "id": "1819179",
      "postDate": "06/13/2022 14:02:31",
      "content": "<p>Need to load 18k+ images but can't train a model kernal crashes…..</p>\n<p>This is my notebook,<br>\n<a href=\"https://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow\" target=\"_blank\">https://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow</a></p>",
      "rawMarkdown": "Need to load 18k+ images but can't train a model kernal crashes.....\n\nThis is my notebook,\nhttps://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow",
      "votes": null
    },
    {
      "id": "1819747",
      "postDate": "06/14/2022 05:34:40",
      "content": "<p>This issue is most likely due to a higher memory overhead by your model. Here's a couple of tips to try:</p>\n<ul>\n<li><p>Revert to the resnet18 model from TensorFlow Hub - currently you are using resnet50 and if the images are relatively small (less than 256px), then resnet18 will be sufficient. If you choose to keep resnet50 you will need to lower the image size. You can also experiment with using EfficientNet-Lite0, 1, 2 or 3 - as a base model, since they are designed for this application.</p></li>\n<li><p>Lower your batch size to 16</p></li>\n<li><p>You should stop using the Iterator.from_structure method since this is not designed for custom data pipelines. Instead, define the _parse_image method (as you did in the cell <strong>after</strong> the <strong># Custom Data Generator</strong> heading), then create the dataset from tf.data.Dataset.from_tensor_slices() method, where you pass the filenames and labels lists, for example:</p></li>\n</ul>\n<p>train_filenames = np.array(['filename1.jpg', 'filename2.jpg', 'filename3.jpg'])</p>\n<p>train_labels = np.array([0, 1, 2])</p>\n<p>train_ds = tf.data.Dataset.from_tensor_slices((train_filenames, train_labels))</p>\n<p>The rest of the training pipeline should be set up in the same way you did, using the <code>train_ds</code> as the <code>train_files</code> and use the <code>_parse_image</code> method as the <code>image_function</code>. Note, you will need to save the output of the <code>_parse_image</code> method as a tuple, since you are returning an image and a label:</p>\n<p>train_ds = train_ds.map(_parse_image, num_parallel_calls=AUTOTUNE)</p>\n<p>return image, label</p>\n<ul>\n<li><p>Don't use TensorFlow 1.x and use 2.0 instead. TF 1.x uses a more expensive method for building the graph, which leads to higher memory usage.</p></li>\n<li><p>If your labels are in string format (e.g. ['cat', 'dog', 'horse']), you can convert them to integer representation.</p></li>\n<li><p>Remove the 'batch_size' and 'shuffle_buffer_size' arguments from the _parse_image method and move them to the data pipeline as described above, this will also reduce the memory usage.</p></li>\n<li><p>You can use tf.contrib.data.parallel_interleave to shuffle and prefetch the images.</p></li>\n<li><p>Use the new method for checkpointing: tf.keras.callbacks.ModelCheckpoint</p></li>\n<li><p>Lower the <code>steps_per_epoch</code> argument and use the <code>validation_steps</code> argument instead.</p></li>\n<li><p>In the end, there are some limits that you can try to increase, but this may not solve the problem.</p></li>\n</ul>\n<p>In TensorFlow 2.0, you can also use model.fit() instead of the training pipeline - this is also somewhat easier.</p>",
      "rawMarkdown": "This issue is most likely due to a higher memory overhead by your model. Here's a couple of tips to try:\n\n- Revert to the resnet18 model from TensorFlow Hub - currently you are using resnet50 and if the images are relatively small (less than 256px), then resnet18 will be sufficient. If you choose to keep resnet50 you will need to lower the image size. You can also experiment with using EfficientNet-Lite0, 1, 2 or 3 - as a base model, since they are designed for this application.\n\n- Lower your batch size to 16\n\n- You should stop using the Iterator.from_structure method since this is not designed for custom data pipelines. Instead, define the _parse_image method (as you did in the cell **after** the **# Custom Data Generator** heading), then create the dataset from tf.data.Dataset.from_tensor_slices() method, where you pass the filenames and labels lists, for example:\n\ntrain_filenames = np.array(['filename1.jpg', 'filename2.jpg', 'filename3.jpg'])\n\ntrain_labels = np.array([0, 1, 2])\n\ntrain_ds = tf.data.Dataset.from_tensor_slices((train_filenames, train_labels))\n\nThe rest of the training pipeline should be set up in the same way you did, using the `train_ds` as the `train_files` and use the `_parse_image` method as the `image_function`. Note, you will need to save the output of the `_parse_image` method as a tuple, since you are returning an image and a label:\n\ntrain_ds = train_ds.map(_parse_image, num_parallel_calls=AUTOTUNE)\n\nreturn image, label\n\n- Don't use TensorFlow 1.x and use 2.0 instead. TF 1.x uses a more expensive method for building the graph, which leads to higher memory usage.\n\n- If your labels are in string format (e.g. ['cat', 'dog', 'horse']), you can convert them to integer representation.\n\n- Remove the 'batch_size' and 'shuffle_buffer_size' arguments from the _parse_image method and move them to the data pipeline as described above, this will also reduce the memory usage.\n\n- You can use tf.contrib.data.parallel_interleave to shuffle and prefetch the images.\n\n- Use the new method for checkpointing: tf.keras.callbacks.ModelCheckpoint\n\n- Lower the `steps_per_epoch` argument and use the `validation_steps` argument instead.\n\n- In the end, there are some limits that you can try to increase, but this may not solve the problem.\n\nIn TensorFlow 2.0, you can also use model.fit() instead of the training pipeline - this is also somewhat easier.",
      "votes": null
    },
    {
      "id": "1820267",
      "postDate": "06/14/2022 13:29:03",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/muki2003\" target=\"_blank\">@muki2003</a> , I think that <a href=\"https://www.kaggle.com/thedevastator\" target=\"_blank\">@thedevastator</a> had really good points of the origin of your problem and potential solution. On my side I don't have the hands on the infra used in a competition so I think it's really some optimisation on your training pipeline :) </p>",
      "rawMarkdown": "Hello @muki2003 , I think that @thedevastator had really good points of the origin of your problem and potential solution. On my side I don't have the hands on the infra used in a competition so I think it's really some optimisation on your training pipeline :)",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1819747,
      "author_name": "thedevastator",
      "author_url": "",
      "post_date": "06/14/2022 05:34:40",
      "content": "<p>This issue is most likely due to a higher memory overhead by your model. Here's a couple of tips to try:</p>\n<ul>\n<li><p>Revert to the resnet18 model from TensorFlow Hub - currently you are using resnet50 and if the images are relatively small (less than 256px), then resnet18 will be sufficient. If you choose to keep resnet50 you will need to lower the image size. You can also experiment with using EfficientNet-Lite0, 1, 2 or 3 - as a base model, since they are designed for this application.</p></li>\n<li><p>Lower your batch size to 16</p></li>\n<li><p>You should stop using the Iterator.from_structure method since this is not designed for custom data pipelines. Instead, define the _parse_image method (as you did in the cell <strong>after</strong> the <strong># Custom Data Generator</strong> heading), then create the dataset from tf.data.Dataset.from_tensor_slices() method, where you pass the filenames and labels lists, for example:</p></li>\n</ul>\n<p>train_filenames = np.array(['filename1.jpg', 'filename2.jpg', 'filename3.jpg'])</p>\n<p>train_labels = np.array([0, 1, 2])</p>\n<p>train_ds = tf.data.Dataset.from_tensor_slices((train_filenames, train_labels))</p>\n<p>The rest of the training pipeline should be set up in the same way you did, using the <code>train_ds</code> as the <code>train_files</code> and use the <code>_parse_image</code> method as the <code>image_function</code>. Note, you will need to save the output of the <code>_parse_image</code> method as a tuple, since you are returning an image and a label:</p>\n<p>train_ds = train_ds.map(_parse_image, num_parallel_calls=AUTOTUNE)</p>\n<p>return image, label</p>\n<ul>\n<li><p>Don't use TensorFlow 1.x and use 2.0 instead. TF 1.x uses a more expensive method for building the graph, which leads to higher memory usage.</p></li>\n<li><p>If your labels are in string format (e.g. ['cat', 'dog', 'horse']), you can convert them to integer representation.</p></li>\n<li><p>Remove the 'batch_size' and 'shuffle_buffer_size' arguments from the _parse_image method and move them to the data pipeline as described above, this will also reduce the memory usage.</p></li>\n<li><p>You can use tf.contrib.data.parallel_interleave to shuffle and prefetch the images.</p></li>\n<li><p>Use the new method for checkpointing: tf.keras.callbacks.ModelCheckpoint</p></li>\n<li><p>Lower the <code>steps_per_epoch</code> argument and use the <code>validation_steps</code> argument instead.</p></li>\n<li><p>In the end, there are some limits that you can try to increase, but this may not solve the problem.</p></li>\n</ul>\n<p>In TensorFlow 2.0, you can also use model.fit() instead of the training pipeline - this is also somewhat easier.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1820267,
      "author_name": "jeanmidev",
      "author_url": "",
      "post_date": "06/14/2022 13:29:03",
      "content": "<p>Hello <a href=\"https://www.kaggle.com/muki2003\" target=\"_blank\">@muki2003</a> , I think that <a href=\"https://www.kaggle.com/thedevastator\" target=\"_blank\">@thedevastator</a> had really good points of the origin of your problem and potential solution. On my side I don't have the hands on the infra used in a competition so I think it's really some optimisation on your training pipeline :) </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "1819179": "Need to load 18k+ images but can't train a model kernal crashes.....\n\nThis is my notebook,\nhttps://www.kaggle.com/muki2003/image-interest-evaluator-tensorflow",
    "1819747": "This issue is most likely due to a higher memory overhead by your model. Here's a couple of tips to try:\n\n- Revert to the resnet18 model from TensorFlow Hub - currently you are using resnet50 and if the images are relatively small (less than 256px), then resnet18 will be sufficient. If you choose to keep resnet50 you will need to lower the image size. You can also experiment with using EfficientNet-Lite0, 1, 2 or 3 - as a base model, since they are designed for this application.\n\n- Lower your batch size to 16\n\n- You should stop using the Iterator.from_structure method since this is not designed for custom data pipelines. Instead, define the _parse_image method (as you did in the cell **after** the **# Custom Data Generator** heading), then create the dataset from tf.data.Dataset.from_tensor_slices() method, where you pass the filenames and labels lists, for example:\n\ntrain_filenames = np.array(['filename1.jpg', 'filename2.jpg', 'filename3.jpg'])\n\ntrain_labels = np.array([0, 1, 2])\n\ntrain_ds = tf.data.Dataset.from_tensor_slices((train_filenames, train_labels))\n\nThe rest of the training pipeline should be set up in the same way you did, using the `train_ds` as the `train_files` and use the `_parse_image` method as the `image_function`. Note, you will need to save the output of the `_parse_image` method as a tuple, since you are returning an image and a label:\n\ntrain_ds = train_ds.map(_parse_image, num_parallel_calls=AUTOTUNE)\n\nreturn image, label\n\n- Don't use TensorFlow 1.x and use 2.0 instead. TF 1.x uses a more expensive method for building the graph, which leads to higher memory usage.\n\n- If your labels are in string format (e.g. ['cat', 'dog', 'horse']), you can convert them to integer representation.\n\n- Remove the 'batch_size' and 'shuffle_buffer_size' arguments from the _parse_image method and move them to the data pipeline as described above, this will also reduce the memory usage.\n\n- You can use tf.contrib.data.parallel_interleave to shuffle and prefetch the images.\n\n- Use the new method for checkpointing: tf.keras.callbacks.ModelCheckpoint\n\n- Lower the `steps_per_epoch` argument and use the `validation_steps` argument instead.\n\n- In the end, there are some limits that you can try to increase, but this may not solve the problem.\n\nIn TensorFlow 2.0, you can also use model.fit() instead of the training pipeline - this is also somewhat easier.",
    "1820267": "Hello @muki2003 , I think that @thedevastator had really good points of the origin of your problem and potential solution. On my side I don't have the hands on the infra used in a competition so I think it's really some optimisation on your training pipeline :)"
  },
  "source": "meta"
}