Dear All,
\nI have made a pre-trained BERT Multi-labeling model and upload it to my private dataset. However, I face the \"Notebook Exceeded Allowed Compute\" error. After some testing, I believe it resulted from the code showing below.
\n\ny_sub = tf.constant(text_df_test)\ny_sub = classifier_model(y_sub)\ny_sub = tf.sigmoid(y_sub)\ny_sub = y_sub.numpy()\n
I understand there are only 16GB of RAM, so that, I only load the first part for each test JSON file and tried to reduce the memory consumption with the following code.
\n`
\ndef read_test_json(test_file_list, path): #Get Training Text
\n file_list = []
\n text_data = []
\n for i, rec_id in tqdm(enumerate(test_file_list), total = len(test_file_list)):
\n location = f'{path}{rec_id}.json'
\n file_list.append(rec_id)
\n add_text = str(\"\")
\n with open(location, 'r') as f:
\n text_data.append(json.load(f))
\n for j in range(len(text_data)):
\n temp_dict = []
\n add_text = str(\"\")
\n temp_dict = text_data[j]
\n for k in range(1):
\n temp_text_iter = dict(temp_dict[k])
\n add_text= add_text + temp_text_iter['text']
\n text_data[j] = add_text
\n text_data[j] = clean_text(text_data[j])
text_df = pd.DataFrame(list(zip(file_list, text_data)),\n columns =['ID', 'text'])\n\nreturn text_df\n`
\nHowever, the above code didn't solve my problem at all. And in my point of view, it is nonsense that we build cutting edge models in data science on Kaggle, where can't running cutting-edge model due to insufficient RAM yet.
\nSincerely,
\nGavin
Dear All,
\nI have made a pre-trained BERT Multi-labeling model and upload it to my private dataset. However, I face the \"Notebook Exceeded Allowed Compute\" error. After some testing, I believe it resulted from the code showing below.
\n\ny_sub = tf.constant(text_df_test)\ny_sub = classifier_model(y_sub)\ny_sub = tf.sigmoid(y_sub)\ny_sub = y_sub.numpy()\n
I understand there are only 16GB of RAM, so that, I only load the first part for each test JSON file and tried to reduce the memory consumption with the following code.
\n`
\ndef read_test_json(test_file_list, path): #Get Training Text
\n file_list = []
\n text_data = []
\n for i, rec_id in tqdm(enumerate(test_file_list), total = len(test_file_list)):
\n location = f'{path}{rec_id}.json'
\n file_list.append(rec_id)
\n add_text = str(\"\")
\n with open(location, 'r') as f:
\n text_data.append(json.load(f))
\n for j in range(len(text_data)):
\n temp_dict = []
\n add_text = str(\"\")
\n temp_dict = text_data[j]
\n for k in range(1):
\n temp_text_iter = dict(temp_dict[k])
\n add_text= add_text + temp_text_iter['text']
\n text_data[j] = add_text
\n text_data[j] = clean_text(text_data[j])
text_df = pd.DataFrame(list(zip(file_list, text_data)),\n columns =['ID', 'text'])\n\nreturn text_df\n`
\nHowever, the above code didn't solve my problem at all. And in my point of view, it is nonsense that we build cutting edge models in data science on Kaggle, where can't running cutting-edge model due to insufficient RAM yet.
\nSincerely,
\nGavin
Dear All,
\nI didn't paste syntax correctly in the first post. Here is the code
\nThe following code is for making the prediction
\nhttps://drive.google.com/file/d/1ZQz9zIfAGvOQaAsxi2IUFIxp5ZCW3KKb/view?usp=sharing
The following code is for getting test text from JSON files
\nhttps://drive.google.com/file/d/1Msw4mRV_1iEtPOpXTXY8Zjd58qB25g9f/view?usp=sharing
Sincerely,
\nGavin
Dear All,
\nI found out the solution, it would be a huge tensor object if I cram all test text into tf.constant. So, I make the prediction one by one and solved this problem.
\nSincerely,
\nGavin
Previous related reply of mine：
\nNotebook Exceeded Allowed Compute means that the memory requirements are too high for the system to meet. The largest memory allocation in the GPU environment is 13GB. In the past few weeks, I was also affected by memory allocation problems and kept submitting failed. There are a few ideas that may help you:
\nReduce the length of the input text. Choosing an appropriate text length can reduce the number of input samples to be constructed.
\nOptimize the preprocessing process and reduce the creation of variables containing a large number of data.
\nReduce the construction of intermediate variables. Consider creating tensor variables such as Input id when running the GPU, which can reduce memory usage by 80%.
good luck! ：）
", "rawMarkdown": "gavinjang\n\nPrevious related reply of mine：\n\nNotebook Exceeded Allowed Compute means that the memory requirements are too high for the system to meet. The largest memory allocation in the GPU environment is 13GB. In the past few weeks, I was also affected by memory allocation problems and kept submitting failed. There are a few ideas that may help you:\n\nReduce the length of the input text. Choosing an appropriate text length can reduce the number of input samples to be constructed.\nOptimize the preprocessing process and reduce the creation of variables containing a large number of data.\nReduce the construction of intermediate variables. Consider creating tensor variables such as Input id when running the GPU, which can reduce memory usage by 80%.\n\ngood luck! ：）", "votes": null }, { "id": "1343871", "postDate": "06/10/2021 13:43:13", "content": "Yes, I have solved the problem this afternoon. I cram all test text into Tensorflow object, and that way more than 16GB. Anyway, thank you very much for your sharing.
", "rawMarkdown": "Yes, I have solved the problem this afternoon. I cram all test text into Tensorflow object, and that way more than 16GB. Anyway, thank you very much for your sharing.", "votes": null } ], "comments": [ { "id": 1343223, "author_name": "gavinjang", "author_url": "", "post_date": "06/10/2021 05:47:06", "content": "Dear All,
\nI didn't paste syntax correctly in the first post. Here is the code
\nThe following code is for making the prediction
\nhttps://drive.google.com/file/d/1ZQz9zIfAGvOQaAsxi2IUFIxp5ZCW3KKb/view?usp=sharing
The following code is for getting test text from JSON files
\nhttps://drive.google.com/file/d/1Msw4mRV_1iEtPOpXTXY8Zjd58qB25g9f/view?usp=sharing
Sincerely,
\nGavin
Dear All,
\nI found out the solution, it would be a huge tensor object if I cram all test text into tf.constant. So, I make the prediction one by one and solved this problem.
\nSincerely,
\nGavin
Previous related reply of mine：
\nNotebook Exceeded Allowed Compute means that the memory requirements are too high for the system to meet. The largest memory allocation in the GPU environment is 13GB. In the past few weeks, I was also affected by memory allocation problems and kept submitting failed. There are a few ideas that may help you:
\nReduce the length of the input text. Choosing an appropriate text length can reduce the number of input samples to be constructed.
\nOptimize the preprocessing process and reduce the creation of variables containing a large number of data.
\nReduce the construction of intermediate variables. Consider creating tensor variables such as Input id when running the GPU, which can reduce memory usage by 80%.
good luck! ：）
", "votes": null, "replies": [ { "id": 1343871, "author_name": "gavinjang", "author_url": "", "post_date": "06/10/2021 13:43:13", "content": "Yes, I have solved the problem this afternoon. I cram all test text into Tensorflow object, and that way more than 16GB. Anyway, thank you very much for your sharing.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1343179": "Dear All,\n\nI have made a pre-trained BERT Multi-labeling model and upload it to my private dataset. However, I face the \"Notebook Exceeded Allowed Compute\" error. After some testing, I believe it resulted from the code showing below.\n\n`\ny_sub = tf.constant(text_df_test)\ny_sub = classifier_model(y_sub)\ny_sub = tf.sigmoid(y_sub)\ny_sub = y_sub.numpy()\n`\n\nI understand there are only 16GB of RAM, so that, I only load the first part for each test JSON file and tried to reduce the memory consumption with the following code.\n\n`\ndef read_test_json(test_file_list, path): #Get Training Text\n file_list = []\n text_data = []\n for i, rec_id in tqdm(enumerate(test_file_list), total = len(test_file_list)):\n location = f'{path}{rec_id}.json'\n file_list.append(rec_id)\n add_text = str(\"\")\n with open(location, 'r') as f:\n text_data.append(json.load(f))\n for j in range(len(text_data)):\n temp_dict = []\n add_text = str(\"\")\n temp_dict = text_data[j]\n for k in range(1):\n temp_text_iter = dict(temp_dict[k])\n add_text= add_text + temp_text_iter['text'] \n text_data[j] = add_text\n text_data[j] = clean_text(text_data[j])\n \n text_df = pd.DataFrame(list(zip(file_list, text_data)),\n columns =['ID', 'text'])\n \n return text_df\n`\n\nHowever, the above code didn't solve my problem at all. And in my point of view, it is nonsense that we build cutting edge models in data science on Kaggle, where can't running cutting-edge model due to insufficient RAM yet.\n\nSincerely,\nGavin", "1343223": "Dear All,\n\nI didn't paste syntax correctly in the first post. Here is the code\n\nThe following code is for making the prediction\n[https://drive.google.com/file/d/1ZQz9zIfAGvOQaAsxi2IUFIxp5ZCW3KKb/view?usp=sharing](url)\n\n\n\nThe following code is for getting test text from JSON files\nhttps://drive.google.com/file/d/1Msw4mRV_1iEtPOpXTXY8Zjd58qB25g9f/view?usp=sharing\n\n\nSincerely,\nGavin", "1343467": "Dear All,\n\nI found out the solution, it would be a huge tensor object if I cram all test text into tf.constant. So, I make the prediction one by one and solved this problem.\n\nSincerely,\nGavin", "1343664": "gavinjang\n\nPrevious related reply of mine：\n\nNotebook Exceeded Allowed Compute means that the memory requirements are too high for the system to meet. The largest memory allocation in the GPU environment is 13GB. In the past few weeks, I was also affected by memory allocation problems and kept submitting failed. There are a few ideas that may help you:\n\nReduce the length of the input text. Choosing an appropriate text length can reduce the number of input samples to be constructed.\nOptimize the preprocessing process and reduce the creation of variables containing a large number of data.\nReduce the construction of intermediate variables. Consider creating tensor variables such as Input id when running the GPU, which can reduce memory usage by 80%.\n\ngood luck! ：）", "1343871": "Yes, I have solved the problem this afternoon. I cram all test text into Tensorflow object, and that way more than 16GB. Anyway, thank you very much for your sharing." }, "source": "meta" }