I have tried my best over the last week to debug this error but I would struggle with submission such as not being able to create a sumission.parquet. I used Claude to jump over this issue but now I am stuck with a \"Notebook Threw Exception\" error even though the model runs fine on the train and dev sets i have created. The model loads fine and all and works when I run it on the first 2 data points in the evaluation server. I just really want help since I've sunk so much time into this. Here is my inference code, or the second half since I don't want to leak my model and preproccesing (which I'm certain are not causing these errors as the model is a simply pytorch and the feature engineering is similar to what others have done and I have checked these):::::: print(\"Loading pre-trained CNN model and assets…\")
\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")
\nprint(f\"Using device: {device}\")
try:
\n # Load model
\n print(f\"Loading model from: {MODEL_PATH}\")
\n checkpoint = torch.load(MODEL_PATH, map_location=device)
\n print(f\"Model config: {checkpoint['model_config']}\")
final_model = OneD_CNN(\n in_channels=[][],\n num_classes=[][]\n)\nfinal_model.load_state_dict([])\nfinal_model.(device)\nfinal_model.eval()\nprint(\"Model loaded successfully.\")\n\n# label encoder\nprint(f\"Loading label encoder from: {LE_PATH}\")\nle = pickle.((LE_PATH, ))\nprint(f\"Label encoder classes: {le.classes_}\")\n\n# IMU names\nprint(f\"Loading IMU columns from: {IMU_COLS_PATH}\")\nimu_columns = pickle.((IMU_COLS_PATH, ))\nprint(f\"IMU columns count: {len(imu_columns)}\")\n\nprint(\"All assets loaded successfully.\")\nexcept Exception as e:
\n print(f\"Error loading assets: {e}\")
\n import traceback
\n traceback.print_exc()
\n raise
def predict(sequence: pl.DataFrame, demographics: pl.DataFrame = None) -> str:
\n \"\"\"
\n Receives a single test sequence and predicts the gesture.
\n \"\"\"
\n try:
\n # Convert to pandas for easier processing
\n seq_df = sequence.to_pandas()
\n =\n missing_cols:\n (f)\n\n \n =\n = seq_df[available_imu_cols]\n\n \n = seq_data.replace([np.nan, np.inf, -np.inf], )\n\n \n = torch.tensor(seq_data_clean.to_numpy(), dtype=torch.float32)\n\n \n \n = seq_tensor.unsqueeze().permute(, , ).to(device)\n\n \n torch.no_grad():\n = final_model(seq_tensor)\n = outputs.argmax(dim=).cpu().numpy()[]\n\n \n = le.inverse_transform([predicted_class])[]\n\n predicted_gesture\n\n Exception as e:\n (f)\n traceback\n .print_exc()\n \n 'Text phone'\nif IN_COMPETITION:
\n # Competition environment - use inference server
\n print(\"Starting inference server…\")
\n inference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)
\n inference_server.serve()
\nelse:
\n # Manual submission generation
\n print(\"Generating manual submission…\")
\ntest_path = \ndemographics_path = \n\n os.path.exists(test_path):\n (f)\n ()\n root, dirs, files os.walk():\n file files:\n (os.path.join(root, file))\n exit(1)\n\n\n()\ntest_df = pl.read_csv(test_path)\n\ntry:\n demographics_df = pl.read_csv(demographics_path)\nexcept:\n demographics_df = None\n\n(f)\n\n\nsequence_ids = test_df[].unique().sort()\n(f)\n\n\npredictions = []\n\n i, seq_id enumerate(sequence_ids):\n i % 100 == 0:\n (f)\n\n # sequence data\n sequence_data = test_df.filter(pl.col() == seq_id)\n\n\n\n # Make prediction\n pred = predict(sequence_data)\n predictions.append({\n : seq_id,\n : pred\n })\n\n\nsubmission_df = pl.DataFrame(predictions)\n(f)\n()\n(submission_df.head())\n\n\noutput_path = \nsubmission_df.write_parquet(output_path)\n(f)\n\n\n os.path.exists(output_path):\n ()\n file_size = os.path.getsize(output_path)\n (f)\n:\n ()\nI have tried my best over the last week to debug this error but I would struggle with submission such as not being able to create a sumission.parquet. I used Claude to jump over this issue but now I am stuck with a \"Notebook Threw Exception\" error even though the model runs fine on the train and dev sets i have created. The model loads fine and all and works when I run it on the first 2 data points in the evaluation server. I just really want help since I've sunk so much time into this. Here is my inference code, or the second half since I don't want to leak my model and preproccesing (which I'm certain are not causing these errors as the model is a simply pytorch and the feature engineering is similar to what others have done and I have checked these):::::: print(\"Loading pre-trained CNN model and assets…\")
\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")
\nprint(f\"Using device: {device}\")
try:
\n # Load model
\n print(f\"Loading model from: {MODEL_PATH}\")
\n checkpoint = torch.load(MODEL_PATH, map_location=device)
\n print(f\"Model config: {checkpoint['model_config']}\")
final_model = OneD_CNN(\n in_channels=[][],\n num_classes=[][]\n)\nfinal_model.load_state_dict([])\nfinal_model.(device)\nfinal_model.eval()\nprint(\"Model loaded successfully.\")\n\n# label encoder\nprint(f\"Loading label encoder from: {LE_PATH}\")\nle = pickle.((LE_PATH, ))\nprint(f\"Label encoder classes: {le.classes_}\")\n\n# IMU names\nprint(f\"Loading IMU columns from: {IMU_COLS_PATH}\")\nimu_columns = pickle.((IMU_COLS_PATH, ))\nprint(f\"IMU columns count: {len(imu_columns)}\")\n\nprint(\"All assets loaded successfully.\")\nexcept Exception as e:
\n print(f\"Error loading assets: {e}\")
\n import traceback
\n traceback.print_exc()
\n raise
def predict(sequence: pl.DataFrame, demographics: pl.DataFrame = None) -> str:
\n \"\"\"
\n Receives a single test sequence and predicts the gesture.
\n \"\"\"
\n try:
\n # Convert to pandas for easier processing
\n seq_df = sequence.to_pandas()
\n =\n missing_cols:\n (f)\n\n \n =\n = seq_df[available_imu_cols]\n\n \n = seq_data.replace([np.nan, np.inf, -np.inf], )\n\n \n = torch.tensor(seq_data_clean.to_numpy(), dtype=torch.float32)\n\n \n \n = seq_tensor.unsqueeze().permute(, , ).to(device)\n\n \n torch.no_grad():\n = final_model(seq_tensor)\n = outputs.argmax(dim=).cpu().numpy()[]\n\n \n = le.inverse_transform([predicted_class])[]\n\n predicted_gesture\n\n Exception as e:\n (f)\n traceback\n .print_exc()\n \n 'Text phone'\nif IN_COMPETITION:
\n # Competition environment - use inference server
\n print(\"Starting inference server…\")
\n inference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)
\n inference_server.serve()
\nelse:
\n # Manual submission generation
\n print(\"Generating manual submission…\")
\ntest_path = \ndemographics_path = \n\n os.path.exists(test_path):\n (f)\n ()\n root, dirs, files os.walk():\n file files:\n (os.path.join(root, file))\n exit(1)\n\n\n()\ntest_df = pl.read_csv(test_path)\n\ntry:\n demographics_df = pl.read_csv(demographics_path)\nexcept:\n demographics_df = None\n\n(f)\n\n\nsequence_ids = test_df[].unique().sort()\n(f)\n\n\npredictions = []\n\n i, seq_id enumerate(sequence_ids):\n i % 100 == 0:\n (f)\n\n # sequence data\n sequence_data = test_df.filter(pl.col() == seq_id)\n\n\n\n # Make prediction\n pred = predict(sequence_data)\n predictions.append({\n : seq_id,\n : pred\n })\n\n\nsubmission_df = pl.DataFrame(predictions)\n(f)\n()\n(submission_df.head())\n\n\noutput_path = \nsubmission_df.write_parquet(output_path)\n(f)\n\n\n os.path.exists(output_path):\n ()\n file_size = os.path.getsize(output_path)\n (f)\n:\n ()\nI strongly advice you to use the submission synax used in the sample submission (data path….) where the server is used for the run all option as well. our test set is rather small and not challenging p, but can use samples of the training data (with different features missing) to see where your code needs improvement.
", "rawMarkdown": "I strongly advice you to use the submission synax used in the sample submission (data path....) where the server is used for the run all option as well. our test set is rather small and not challenging p, but can use samples of the training data (with different features missing) to see where your code needs improvement.", "votes": null }, { "id": "3230760", "postDate": "06/23/2025 12:20:41", "content": "Yes thank you, this is a workaround sol as when I use that submission sort of formatting I can’t submit my file since “This Competition requires a submission file named submission.parquet and the selected Notebook Version does not output this file. Please select a Notebook Version that produces the expected output.”
", "rawMarkdown": "Yes thank you, this is a workaround sol as when I use that submission sort of formatting I can’t submit my file since “This Competition requires a submission file named submission.parquet and the selected Notebook Version does not output this file. Please select a Notebook Version that produces the expected output.”", "votes": null }, { "id": "3230846", "postDate": "06/23/2025 14:24:25", "content": "Hmm, I believe that EVERY submission has to run through the gateway. The reason is that they want to understand the time needed for interference, and just submitting a parquet file does not give you that information. Your workaround is therefore doomed to fail in the end.
\nI noticed that you are loading files for interference. The submission runs in a different environment, and only the input files are guaranteed to be accessible, everything else is not. You also will not see any print from the submission.
\nBut if you are running the gateway outside a submission, then you can use every dataset and any print/debug statement you like
\nSummary: Try the sample code with a simple predict function and the build up to your solution
", "rawMarkdown": "Hmm, I believe that EVERY submission has to run through the gateway. The reason is that they want to understand the time needed for interference, and just submitting a parquet file does not give you that information. Your workaround is therefore doomed to fail in the end. \n\nI noticed that you are loading files for interference. The submission runs in a different environment, and only the input files are guaranteed to be accessible, everything else is not. You also will not see any print from the submission. \n\nBut if you are running the gateway outside a submission, then you can use every dataset and any print/debug statement you like\n\nSummary: Try the sample code with a simple predict function and the build up to your solution", "votes": null } ], "comments": [ { "id": 3230742, "author_name": "fredericnicholson", "author_url": "", "post_date": "06/23/2025 12:03:02", "content": "I strongly advice you to use the submission synax used in the sample submission (data path….) where the server is used for the run all option as well. our test set is rather small and not challenging p, but can use samples of the training data (with different features missing) to see where your code needs improvement.
", "votes": null, "replies": [ { "id": 3230760, "author_name": "noahstefancik", "author_url": "", "post_date": "06/23/2025 12:20:41", "content": "Yes thank you, this is a workaround sol as when I use that submission sort of formatting I can’t submit my file since “This Competition requires a submission file named submission.parquet and the selected Notebook Version does not output this file. Please select a Notebook Version that produces the expected output.”
", "votes": null, "replies": [ { "id": 3230846, "author_name": "fredericnicholson", "author_url": "", "post_date": "06/23/2025 14:24:25", "content": "Hmm, I believe that EVERY submission has to run through the gateway. The reason is that they want to understand the time needed for interference, and just submitting a parquet file does not give you that information. Your workaround is therefore doomed to fail in the end.
\nI noticed that you are loading files for interference. The submission runs in a different environment, and only the input files are guaranteed to be accessible, everything else is not. You also will not see any print from the submission.
\nBut if you are running the gateway outside a submission, then you can use every dataset and any print/debug statement you like
\nSummary: Try the sample code with a simple predict function and the build up to your solution
", "votes": null, "replies": [] } ] } ] } ], "raw_markdown_by_id": { "3230382": "I have tried my best over the last week to debug this error but I would struggle with submission such as not being able to create a sumission.parquet. I used Claude to jump over this issue but now I am stuck with a \"Notebook Threw Exception\" error even though the model runs fine on the train and dev sets i have created. The model loads fine and all and works when I run it on the first 2 data points in the evaluation server. I just really want help since I've sunk so much time into this. Here is my inference code, or the second half since I don't want to leak my model and preproccesing (which I'm certain are not causing these errors as the model is a simply pytorch and the feature engineering is similar to what others have done and I have checked these):::::: print(\"Loading pre-trained CNN model and assets...\")\ndevice = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\nprint(f\"Using device: {device}\")\n\ntry:\n # Load model\n print(f\"Loading model from: {MODEL_PATH}\")\n checkpoint = torch.load(MODEL_PATH, map_location=device)\n print(f\"Model config: {checkpoint['model_config']}\")\n \n final_model = OneD_CNN(\n in_channels=checkpoint['model_config']['in_channels'],\n num_classes=checkpoint['model_config']['num_classes']\n )\n final_model.load_state_dict(checkpoint['model_state_dict'])\n final_model.to(device)\n final_model.eval()\n print(\"Model loaded successfully.\")\n\n # Load label encoder\n print(f\"Loading label encoder from: {LE_PATH}\")\n le = pickle.load(open(LE_PATH, 'rb'))\n print(f\"Label encoder classes: {le.classes_}\")\n\n # Load IMU column names\n print(f\"Loading IMU columns from: {IMU_COLS_PATH}\")\n imu_columns = pickle.load(open(IMU_COLS_PATH, 'rb'))\n print(f\"IMU columns count: {len(imu_columns)}\")\n\n print(\"All assets loaded successfully.\")\n \nexcept Exception as e:\n print(f\"Error loading assets: {e}\")\n import traceback\n traceback.print_exc()\n raise\n\n# --- 2. Define Predict Function ---\ndef predict(sequence: pl.DataFrame, demographics: pl.DataFrame = None) -> str:\n \"\"\"\n Receives a single test sequence and predicts the gesture.\n \"\"\"\n try:\n # Convert to pandas for easier processing\n seq_df = sequence.to_pandas()\n \n # Check if IMU columns exist\n missing_cols = [col for col in imu_columns if col not in seq_df.columns]\n if missing_cols:\n print(f\"Warning: Missing columns: {missing_cols}\")\n \n # Extract IMU features\n available_imu_cols = [col for col in imu_columns if col in seq_df.columns]\n seq_data = seq_df[available_imu_cols]\n \n # Clean data - replace NaN/Inf with 0\n seq_data_clean = seq_data.replace([np.nan, np.inf, -np.inf], 0)\n \n # Convert to tensor\n seq_tensor = torch.tensor(seq_data_clean.to_numpy(), dtype=torch.float32)\n \n # Add batch dimension and permute for CNN input\n # Shape: [1, seq_len, features] -> [1, features, seq_len]\n seq_tensor = seq_tensor.unsqueeze(0).permute(0, 2, 1).to(device)\n \n # Make prediction\n with torch.no_grad():\n outputs = final_model(seq_tensor)\n predicted_class = outputs.argmax(dim=1).cpu().numpy()[0]\n \n # Convert back to gesture name\n predicted_gesture = le.inverse_transform([predicted_class])[0]\n \n return predicted_gesture\n \n except Exception as e:\n print(f\"Error in prediction: {e}\")\n import traceback\n traceback.print_exc()\n # Return a default prediction if something goes wrong\n return 'Text on phone'\n\n# --- 3. Generate Submission ---\nif IN_COMPETITION:\n # Competition environment - use inference server\n print(\"Starting inference server...\")\n inference_server = kaggle_evaluation.cmi_inference_server.CMIInferenceServer(predict)\n inference_server.serve()\nelse:\n # Manual submission generation\n print(\"Generating manual submission...\")\n \n # Load test data\n test_path = '/kaggle/input/cmi-detect-behavior-with-sensor-data/test.csv'\n demographics_path = '/kaggle/input/cmi-detect-behavior-with-sensor-data/test_demographics.csv'\n \n if not os.path.exists(test_path):\n print(f\"Test file not found at {test_path}\")\n print(\"Available files in input directory:\")\n for root, dirs, files in os.walk('/kaggle/input/'):\n for file in files:\n print(os.path.join(root, file))\n exit(1)\n \n # Load data\n print(\"Loading test data...\")\n test_df = pl.read_csv(test_path)\n \n try:\n demographics_df = pl.read_csv(demographics_path)\n except:\n demographics_df = None\n \n print(f\"Test data shape: {test_df.shape}\")\n \n # Get unique sequence IDs\n sequence_ids = test_df['sequence_id'].unique().sort()\n print(f\"Number of sequences: {len(sequence_ids)}\")\n \n # Generate predictions\n predictions = []\n \n for i, seq_id in enumerate(sequence_ids):\n if i % 100 == 0:\n print(f\"Processing sequence {i+1}/{len(sequence_ids)}\")\n \n # Get sequence data\n sequence_data = test_df.filter(pl.col('sequence_id') == seq_id)\n \n \n \n # Make prediction\n pred = predict(sequence_data)\n predictions.append({\n 'sequence_id': seq_id,\n 'gesture': pred\n })\n \n # Create submission dataframe\n submission_df = pl.DataFrame(predictions)\n print(f\"Submission shape: {submission_df.shape}\")\n print(\"Sample predictions:\")\n print(submission_df.head())\n \n # Save submission\n output_path = '/kaggle/working/submission.parquet'\n submission_df.write_parquet(output_path)\n print(f\"Submission saved to {output_path}\")\n \n # Verify file was created\n if os.path.exists(output_path):\n print(\"✅ submission.parquet successfully created!\")\n file_size = os.path.getsize(output_path)\n print(f\"File size: {file_size} bytes\")\n else:\n print(\"❌ Failed to create submission.parquet\")", "3230742": "I strongly advice you to use the submission synax used in the sample submission (data path....) where the server is used for the run all option as well. our test set is rather small and not challenging p, but can use samples of the training data (with different features missing) to see where your code needs improvement.", "3230760": "Yes thank you, this is a workaround sol as when I use that submission sort of formatting I can’t submit my file since “This Competition requires a submission file named submission.parquet and the selected Notebook Version does not output this file. Please select a Notebook Version that produces the expected output.”", "3230846": "Hmm, I believe that EVERY submission has to run through the gateway. The reason is that they want to understand the time needed for interference, and just submitting a parquet file does not give you that information. Your workaround is therefore doomed to fail in the end. \n\nI noticed that you are loading files for interference. The submission runs in a different environment, and only the input files are guaranteed to be accessible, everything else is not. You also will not see any print from the submission. \n\nBut if you are running the gateway outside a submission, then you can use every dataset and any print/debug statement you like\n\nSummary: Try the sample code with a simple predict function and the build up to your solution" }, "source": "meta" }