When i attempt to fit my tensorflow model with the following code
\nmodel.fit(gen(list_images_train, captions_train, batch_size= 32),
\n use_multiprocessing=True, epochs=10000, verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])
I get a cant pickle a generator error. When I google the error I see people saying when you use a generator in the fitting process you have to turn use_multiprocessing to false.
\nBut when I do that and try to fit the kernel always dies.
\nDoes anyone have any idea what I can try to do from here?
", "messages": [ { "id": 1613506, "postDate": "2021-12-10T03:44:06.693Z", "content": "@danishvirani or anyone else in a similar spot - I recommend taking a look at this simple model that's available out of the box from scikit learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor
\nIt definitely isn't as built out or robust as torch/tf, but I think you may find it easier to get started with if you're just looking to do something simple. They call out the limitations and expected uses here: https://scikit-learn.org/stable/modules/neural_networks_supervised.html#neural-networks-supervised
", "rawMarkdown": "@danishvirani or anyone else in a similar spot - I recommend taking a look at this simple model that's available out of the box from scikit learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor\n\nIt definitely isn't as built out or robust as torch/tf, but I think you may find it easier to get started with if you're just looking to do something simple. They call out the limitations and expected uses here: https://scikit-learn.org/stable/modules/neural_networks_supervised.html#neural-networks-supervised", "votes": 1 }, { "id": 1613619, "postDate": "2021-12-10T06:39:08.593Z", "content": "Is this tripletNN? If it is, I also found the same source, and my model did start running with our data.
\nHowever, there is a memory leak to this model, I don't know if it is caused by the structure of the model or it's an issue with Keras, but I did try all the methods available to prevent memory leak on Keras, none of them worked). It will stop training after 12 epochs, and the result is not very well. The model trained with 12 epochs is able to find the similarities among images pretty well, however, it doesn't build a strong connection between text and images.
\nI think the leak is caused by the gen function, but I have no idea on how to fix it. When you make it work, I would recommend you keep track of the memory usage after every epoch, if it keeps increasing, then it indicates the memory is leaking.
", "rawMarkdown": "Is this tripletNN? If it is, I also found the same source, and my model did start running with our data.\n\nHowever, there is a memory leak to this model, I don't know if it is caused by the structure of the model or it's an issue with Keras, but I did try all the methods available to prevent memory leak on Keras, none of them worked). It will stop training after 12 epochs, and the result is not very well. The model trained with 12 epochs is able to find the similarities among images pretty well, however, it doesn't build a strong connection between text and images.\n\nI think the leak is caused by the gen function, but I have no idea on how to fix it. When you make it work, I would recommend you keep track of the memory usage after every epoch, if it keeps increasing, then it indicates the memory is leaking.", "replies": [ { "id": 1614070, "postDate": "2021-12-10T15:00:56.853Z", "content": "Good tip about the memory leakage.
\nI did not encounter memory leakage in the generator function.
\nI am not using a generator function anymore.
\nIf you want i'd be happy to share the generator function I was using.
Thanks James.
\nI did manage to fix the GPU issue, i think.
\nI changed the environment settings a bit and i can run my tutorial models and the hand written numbers kaggle challenge.
Now my issue is:
\nValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(1, 224, 224, 3) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(1, 70) dtype=int32>, <tf.Tensor 'IteratorGetNext:2' shape=(1, 70) dtype=int32>, <tf.Tensor 'IteratorGetNext:3' shape=(1, 225) dtype=float32>]\nI believe i am formatting the inputs into the model fit incorrectly.
\nTo answer your question:
\nMy captions get embedded in the pipeline for the modeling. I'd be happy to show you.
\nAt this point i couldn't care less about the exam I just want to know what I did wrong/get it to work.
I did manage to fix the batch size issue.
\nI'm pretty sure my input itself is what i expect it to be
\nThe form it's in is:
\n[ [X1, X2, X3], Y]
\nThe model is defined as
\nx = Concatenate(axis=-1)([x1, x2, x3])
\nmodel = Model([input_1, input_2, input_3], x)
But the model was made for the generator which was in the form of a tuple. So maybe the model should look like below. Or maybe my input should look like ( [X1, X2, X3], Y ) *a tuple instead of a list.
\nmodel = Model( [[input_1, input_2, input_3], x] )
I feel this shouldn't be causing the issues but I'll try them.
", "rawMarkdown": "I did manage to fix the batch size issue. \n\nI'm pretty sure my input itself is what i expect it to be \n\nThe form it's in is:\n\n[ [X1, X2, X3], Y]\n\n\nThe model is defined as\n\nx = Concatenate(axis=-1)([x1, x2, x3])\nmodel = Model([input_1, input_2, input_3], x)\n\nBut the model was made for the generator which was in the form of a tuple. So maybe the model should look like below. Or maybe my input should look like ( [X1, X2, X3], Y ) *a tuple instead of a list.\nmodel = Model( [[input_1, input_2, input_3], x] )\n\nI feel this shouldn't be causing the issues but I'll try them. \n" }, { "id": 1613165, "postDate": "2021-12-09T17:28:54.443Z", "rawMarkdown": "", "isDeleted": true }, { "id": 1613202, "postDate": "2021-12-09T17:57:45.893Z", "content": "I am attempting to use keras funcitonal api.
\nThe code looks like below:
\ninput_1 = Input(shape=(None, None, 3))
\ninput_2 = Input(shape=(None,))
\ninput_3 = Input(shape=(None,))
\nmodel stuff
\nx = Concatenate(axis=-1)([x1, x2, x3])
\nmodel = Model( [input_1, input_2, input_3], x)
\nmodel.compile(loss=LossFunction, optimizer=Adam(lr))
Current error (changed 70 to 50):
\nValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [, , , ]
\nYes i'm aware they gave us resnet output. I just wanted to try playing around with it myself and trying out some other image encoders and filters.
\nModel: \"model\"
\ninput_2 (InputLayer) [(None, None)] 0 []
\ninput_3 (InputLayer) [(None, None)] 0 []
\nembed (Embedding) (None, None, 50) 123600 ['input_2[0][0]',
\n 'input_3[0][0]']
input_1 (InputLayer) [(None, None, None, 0 []
\n 3)]
spatial_dropout1d (SpatialDrop (None, None, 50) 0 ['embed[0][0]']
\n out1D)
spatial_dropout1d_1 (SpatialDr (None, None, 50) 0 ['embed[1][0]']
\n opout1D)
resnet50 (Functional) (None, None, None, 23587712 ['input_1[0][0]']
\n 2048)
gru_1 (Bidirectional) (None, None, 512) 473088 ['spatial_dropout1d[0][0]',
\n 'spatial_dropout1d_1[0][0]']
global_max_pooling2d (GlobalMa (None, 2048) 0 ['resnet50[0][0]']
\n xPooling2D)
global_max_pooling1d (GlobalMa (None, 512) 0 ['gru_1[0][0]']
\n xPooling1D)
global_max_pooling1d_1 (Global (None, 512) 0 ['gru_1[1][0]']
\n MaxPooling1D)
dense_image_1 (Dense) (None, 50) 102450 ['global_max_pooling2d[0][0]']
\ndense_text_1 (Dense) (None, 50) 25650 ['global_max_pooling1d[0][0]',
\n 'global_max_pooling1d_1[0][0]']
lambda (Lambda) (None, 50) 0 ['dense_image_1[0][0]',
\n 'dense_text_1[0][0]',
\n 'dense_text_1[1][0]']
concatenate (Concatenate) (None, 150) 0 ['lambda[0][0]',
\n 'lambda[1][0]',
\n 'lambda[2][0]']
I do have a text and a image model that are separate and working. I think the issue comes from just the input. The image/text model i have seperate and all the other tutorials i could find to follow have inputs of (X1,Y1) but I have inputs of ([X1,X2,X3],Y1) and I think that's where the error is coming from.
", "rawMarkdown": "I am attempting to use keras funcitonal api. \n\nThe code looks like below:\n\ninput_1 = Input(shape=(None, None, 3))\ninput_2 = Input(shape=(None,))\ninput_3 = Input(shape=(None,))\n*model stuff*\nx = Concatenate(axis=-1)([x1, x2, x3])\nmodel = Model( [input_1, input_2, input_3], x)\nmodel.compile(loss=LossFunction, optimizer=Adam(lr))\n\n\n\nCurrent error (changed 70 to 50):\n\nValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [When I change the generator to a list and use the following code:
\nmodel.fit(test,use_multiprocessing=True, batch_size = 1,
\n #validation_data=gen(list_images_val, captions_val, batch_size=BATCH_SIZE),
\n epochs=10000,
\n verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])
\nmodel.save_weights(file_path)
InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.
\n'Dst tensor is not initialized.' error means there was not enough memory
\nBecause it ssays GPU:0 i'm assuming it's because it's not detecting my GPU.
I did try to install cuda to get the gpu working and I do have an RTX 3080 which is cuda compatible. Any help with this would be greatly appreciated.
", "rawMarkdown": "When I change the generator to a list and use the following code:\n\nmodel.fit(test,use_multiprocessing=True, batch_size = 1,\n #validation_data=gen(list_images_val, captions_val, batch_size=BATCH_SIZE), \n epochs=10000,\n verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])\nmodel.save_weights(file_path)\n\nInternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.\n\n'Dst tensor is not initialized.' error means there was not enough memory \nBecause it ssays GPU:0 i'm assuming it's because it's not detecting my GPU.\n\nI did try to install cuda to get the gpu working and I do have an RTX 3080 which is cuda compatible. Any help with this would be greatly appreciated. " }, { "id": 1613113, "postDate": "2021-12-09T16:09:24.160Z", "content": "When i attempt to fit my tensorflow model with the following code
\nmodel.fit(gen(list_images_train, captions_train, batch_size= 32),
\n use_multiprocessing=True, epochs=10000, verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])
I get a cant pickle a generator error. When I google the error I see people saying when you use a generator in the fitting process you have to turn use_multiprocessing to false.
\nBut when I do that and try to fit the kernel always dies.
\nDoes anyone have any idea what I can try to do from here?
", "rawMarkdown": "When i attempt to fit my tensorflow model with the following code\n\nmodel.fit(gen(list_images_train, captions_train, batch_size= 32),\n use_multiprocessing=True, epochs=10000, verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])\n\nI get a cant pickle a generator error. When I google the error I see people saying when you use a generator in the fitting process you have to turn use_multiprocessing to false. \n\nBut when I do that and try to fit the kernel always dies.\n\nDoes anyone have any idea what I can try to do from here? " }, { "id": 1613124, "postDate": "2021-12-09T16:23:37.857Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 1613137, "postDate": "2021-12-09T16:44:34.603Z", "content": "My captions get embedded in the pipeline for the modeling. I'd be happy to show you.
\nAt this point i couldn't care less about the exam I just want to know what I did wrong/get it to work.
I believe i fixed the GPU issue.
\nThanks for the Cudatoolkit tip. The install guide for tensor flow GPU is seriously not simple.
", "rawMarkdown": "My captions get embedded in the pipeline for the modeling. I'd be happy to show you.\nAt this point i couldn't care less about the exam I just want to know what I did wrong/get it to work.\n\nI believe i fixed the GPU issue. \n\nThanks for the Cudatoolkit tip. The install guide for tensor flow GPU is seriously not simple. " }, { "id": 1613141, "postDate": "2021-12-09T16:48:30.167Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] } ], "comments": [ { "id": 1613506, "author_name": "Anna462", "author_url": "", "post_date": "2021-12-10T03:44:06.693000", "content": "@danishvirani or anyone else in a similar spot - I recommend taking a look at this simple model that's available out of the box from scikit learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor
\nIt definitely isn't as built out or robust as torch/tf, but I think you may find it easier to get started with if you're just looking to do something simple. They call out the limitations and expected uses here: https://scikit-learn.org/stable/modules/neural_networks_supervised.html#neural-networks-supervised
", "votes": 1, "replies": [] }, { "id": 1613619, "author_name": "Caroline Ming", "author_url": "", "post_date": "2021-12-10T06:39:08.593000", "content": "Is this tripletNN? If it is, I also found the same source, and my model did start running with our data.
\nHowever, there is a memory leak to this model, I don't know if it is caused by the structure of the model or it's an issue with Keras, but I did try all the methods available to prevent memory leak on Keras, none of them worked). It will stop training after 12 epochs, and the result is not very well. The model trained with 12 epochs is able to find the similarities among images pretty well, however, it doesn't build a strong connection between text and images.
\nI think the leak is caused by the gen function, but I have no idea on how to fix it. When you make it work, I would recommend you keep track of the memory usage after every epoch, if it keeps increasing, then it indicates the memory is leaking.
", "votes": 0, "replies": [ { "id": 1614070, "author_name": "Danish Virani", "author_url": "", "post_date": "2021-12-10T15:00:56.853000", "content": "Good tip about the memory leakage.
\nI did not encounter memory leakage in the generator function.
\nI am not using a generator function anymore.
\nIf you want i'd be happy to share the generator function I was using.
Thanks James.
\nI did manage to fix the GPU issue, i think.
\nI changed the environment settings a bit and i can run my tutorial models and the hand written numbers kaggle challenge.
Now my issue is:
\nValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [<tf.Tensor 'IteratorGetNext:0' shape=(1, 224, 224, 3) dtype=float32>, <tf.Tensor 'IteratorGetNext:1' shape=(1, 70) dtype=int32>, <tf.Tensor 'IteratorGetNext:2' shape=(1, 70) dtype=int32>, <tf.Tensor 'IteratorGetNext:3' shape=(1, 225) dtype=float32>]\nI believe i am formatting the inputs into the model fit incorrectly.
\nTo answer your question:
\nMy captions get embedded in the pipeline for the modeling. I'd be happy to show you.
\nAt this point i couldn't care less about the exam I just want to know what I did wrong/get it to work.
I did manage to fix the batch size issue.
\nI'm pretty sure my input itself is what i expect it to be
\nThe form it's in is:
\n[ [X1, X2, X3], Y]
\nThe model is defined as
\nx = Concatenate(axis=-1)([x1, x2, x3])
\nmodel = Model([input_1, input_2, input_3], x)
But the model was made for the generator which was in the form of a tuple. So maybe the model should look like below. Or maybe my input should look like ( [X1, X2, X3], Y ) *a tuple instead of a list.
\nmodel = Model( [[input_1, input_2, input_3], x] )
I feel this shouldn't be causing the issues but I'll try them.
", "votes": 0, "replies": [] }, { "id": 1613165, "author_name": "", "author_url": "", "post_date": "2021-12-09T17:28:54.443000", "content": "", "votes": 0, "replies": [] }, { "id": 1613202, "author_name": "Danish Virani", "author_url": "", "post_date": "2021-12-09T17:57:45.893000", "content": "I am attempting to use keras funcitonal api.
\nThe code looks like below:
\ninput_1 = Input(shape=(None, None, 3))
\ninput_2 = Input(shape=(None,))
\ninput_3 = Input(shape=(None,))
\nmodel stuff
\nx = Concatenate(axis=-1)([x1, x2, x3])
\nmodel = Model( [input_1, input_2, input_3], x)
\nmodel.compile(loss=LossFunction, optimizer=Adam(lr))
Current error (changed 70 to 50):
\nValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [, , , ]
\nYes i'm aware they gave us resnet output. I just wanted to try playing around with it myself and trying out some other image encoders and filters.
\nModel: \"model\"
\ninput_2 (InputLayer) [(None, None)] 0 []
\ninput_3 (InputLayer) [(None, None)] 0 []
\nembed (Embedding) (None, None, 50) 123600 ['input_2[0][0]',
\n 'input_3[0][0]']
input_1 (InputLayer) [(None, None, None, 0 []
\n 3)]
spatial_dropout1d (SpatialDrop (None, None, 50) 0 ['embed[0][0]']
\n out1D)
spatial_dropout1d_1 (SpatialDr (None, None, 50) 0 ['embed[1][0]']
\n opout1D)
resnet50 (Functional) (None, None, None, 23587712 ['input_1[0][0]']
\n 2048)
gru_1 (Bidirectional) (None, None, 512) 473088 ['spatial_dropout1d[0][0]',
\n 'spatial_dropout1d_1[0][0]']
global_max_pooling2d (GlobalMa (None, 2048) 0 ['resnet50[0][0]']
\n xPooling2D)
global_max_pooling1d (GlobalMa (None, 512) 0 ['gru_1[0][0]']
\n xPooling1D)
global_max_pooling1d_1 (Global (None, 512) 0 ['gru_1[1][0]']
\n MaxPooling1D)
dense_image_1 (Dense) (None, 50) 102450 ['global_max_pooling2d[0][0]']
\ndense_text_1 (Dense) (None, 50) 25650 ['global_max_pooling1d[0][0]',
\n 'global_max_pooling1d_1[0][0]']
lambda (Lambda) (None, 50) 0 ['dense_image_1[0][0]',
\n 'dense_text_1[0][0]',
\n 'dense_text_1[1][0]']
concatenate (Concatenate) (None, 150) 0 ['lambda[0][0]',
\n 'lambda[1][0]',
\n 'lambda[2][0]']
I do have a text and a image model that are separate and working. I think the issue comes from just the input. The image/text model i have seperate and all the other tutorials i could find to follow have inputs of (X1,Y1) but I have inputs of ([X1,X2,X3],Y1) and I think that's where the error is coming from.
", "votes": 0, "replies": [] }, { "id": 1613270, "author_name": "", "author_url": "", "post_date": "2021-12-09T19:43:41.970000", "content": "", "votes": 0, "replies": [] } ] }, { "id": 1613121, "author_name": "Danish Virani", "author_url": "", "post_date": "2021-12-09T16:21:17.527000", "content": "When I change the generator to a list and use the following code:
\nmodel.fit(test,use_multiprocessing=True, batch_size = 1,
\n #validation_data=gen(list_images_val, captions_val, batch_size=BATCH_SIZE),
\n epochs=10000,
\n verbose=2, workers=4, steps_per_epoch=300, callbacks=[checkpoint, reduce])
\nmodel.save_weights(file_path)
InternalError: Failed copying input tensor from /job:localhost/replica:0/task:0/device:CPU:0 to /job:localhost/replica:0/task:0/device:GPU:0 in order to run _EagerConst: Dst tensor is not initialized.
\n'Dst tensor is not initialized.' error means there was not enough memory
\nBecause it ssays GPU:0 i'm assuming it's because it's not detecting my GPU.
I did try to install cuda to get the gpu working and I do have an RTX 3080 which is cuda compatible. Any help with this would be greatly appreciated.
", "votes": 0, "replies": [] }, { "id": 1613124, "author_name": "", "author_url": "", "post_date": "2021-12-09T16:23:37.857000", "content": "", "votes": 1, "replies": [ { "id": 1613137, "author_name": "Danish Virani", "author_url": "", "post_date": "2021-12-09T16:44:34.603000", "content": "My captions get embedded in the pipeline for the modeling. I'd be happy to show you.
\nAt this point i couldn't care less about the exam I just want to know what I did wrong/get it to work.
I believe i fixed the GPU issue.
\nThanks for the Cudatoolkit tip. The install guide for tensor flow GPU is seriously not simple.
", "votes": 0, "replies": [] }, { "id": 1613141, "author_name": "", "author_url": "", "post_date": "2021-12-09T16:48:30.167000", "content": "", "votes": 1, "replies": [] } ] } ], "raw_markdown_by_id": { "1613506": "@danishvirani or anyone else in a similar spot - I recommend taking a look at this simple model that's available out of the box from scikit learn: https://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor\n\nIt definitely isn't as built out or robust as torch/tf, but I think you may find it easier to get started with if you're just looking to do something simple. They call out the limitations and expected uses here: https://scikit-learn.org/stable/modules/neural_networks_supervised.html#neural-networks-supervised", "1613619": "Is this tripletNN? If it is, I also found the same source, and my model did start running with our data.\n\nHowever, there is a memory leak to this model, I don't know if it is caused by the structure of the model or it's an issue with Keras, but I did try all the methods available to prevent memory leak on Keras, none of them worked). It will stop training after 12 epochs, and the result is not very well. The model trained with 12 epochs is able to find the similarities among images pretty well, however, it doesn't build a strong connection between text and images.\n\nI think the leak is caused by the gen function, but I have no idea on how to fix it. When you make it work, I would recommend you keep track of the memory usage after every epoch, if it keeps increasing, then it indicates the memory is leaking.", "1613134": "Thanks James.\n\nI did manage to fix the GPU issue, i think. \nI changed the environment settings a bit and i can run my tutorial models and the hand written numbers kaggle challenge. \n\nNow my issue is:\n\n ValueError: Layer \"model\" expects 3 input(s), but it received 4 input tensors. Inputs received: [