I am very pleased to share that I've achieved a bronze medal in my second Featured Competition on Kaggle!
\nInitially, I developed a simple Bi-GRU model of my own which i got ideas from amazing Public Notebooks. To my surprise, it ranked in the bronze medal range. (Public score 0.709, Private score 0.758). However, following the release of @tubotubo's excellent notebook, I decided to utilize it as well. My tuned version focused primarily on Feature Engineering and Ensemble (Public 0.750, Private 0.775).
\nBelow, I briefly describe the details of my two solutions:
\n (nn.Module):\n ():\n (Bi_GRU_BASE_MODEL, self).__init__()\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.output_size = output_size\n self.num_layers = num_layers\n\n self.fc_in = nn.Linear(input_size, hidden_size)\n\n \n self.ln = nn.LayerNorm(hidden_size)\n\n \n self.gru = nn.GRU(input_size=hidden_size,\n hidden_size=hidden_size//,\n num_layers=num_layers,\n batch_first=, \n bidirectional=bidir, \n dropout=dropout_rate)\n\n self.fc_out = nn.Linear(hidden_size, output_size) \n\n ():\n x = self.fc_in(x)\n x = self.ln(x)\n x = torch.relu(x)\n\n x, _ = self.gru(x) \n x = self.fc_out(x)\n x\nThank you for reading. Kudos to everyone who participated in the competition. I'm immensely grateful to everyone who provided well-crafted Public Notebooks!!!!
", "messages": [ { "id": 2550700, "postDate": "2023-12-06T08:18:51.820Z", "content": "I am very pleased to share that I've achieved a bronze medal in my second Featured Competition on Kaggle!
\nInitially, I developed a simple Bi-GRU model of my own which i got ideas from amazing Public Notebooks. To my surprise, it ranked in the bronze medal range. (Public score 0.709, Private score 0.758). However, following the release of @tubotubo's excellent notebook, I decided to utilize it as well. My tuned version focused primarily on Feature Engineering and Ensemble (Public 0.750, Private 0.775).
\nBelow, I briefly describe the details of my two solutions:
\n (nn.Module):\n ():\n (Bi_GRU_BASE_MODEL, self).__init__()\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.output_size = output_size\n self.num_layers = num_layers\n\n self.fc_in = nn.Linear(input_size, hidden_size)\n\n \n self.ln = nn.LayerNorm(hidden_size)\n\n \n self.gru = nn.GRU(input_size=hidden_size,\n hidden_size=hidden_size//,\n num_layers=num_layers,\n batch_first=, \n bidirectional=bidir, \n dropout=dropout_rate)\n\n self.fc_out = nn.Linear(hidden_size, output_size) \n\n ():\n x = self.fc_in(x)\n x = self.ln(x)\n x = torch.relu(x)\n\n x, _ = self.gru(x) \n x = self.fc_out(x)\n x\nThank you for reading. Kudos to everyone who participated in the competition. I'm immensely grateful to everyone who provided well-crafted Public Notebooks!!!!
", "rawMarkdown": "I am very pleased to share that I've achieved a bronze medal in my second Featured Competition on Kaggle!\n\nInitially, I developed a simple Bi-GRU model of my own which i got ideas from amazing Public Notebooks. To my surprise, it ranked in the bronze medal range. (Public score 0.709, Private score 0.758). However, following the release of @tubotubo's excellent notebook, I decided to utilize it as well. My tuned version focused primarily on Feature Engineering and Ensemble (Public 0.750, Private 0.775).\n\nBelow, I briefly describe the details of my two solutions:\n\n# 1. KimSeungHee's Model (Public 0.709, Private 0.758)\n\n```python\nclass Bi_GRU_BASE_MODEL(nn.Module):\n def __init__(self, input_size, hidden_size, num_layers, output_size, dropout_rate, bidir=True):\n super(Bi_GRU_BASE_MODEL, self).__init__()\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.output_size = output_size\n self.num_layers = num_layers\n \n self.fc_in = nn.Linear(input_size, hidden_size)\n \n # Layer Normalization\n self.ln = nn.LayerNorm(hidden_size)\n \n # GRU Layer\n self.gru = nn.GRU(input_size=hidden_size,\n hidden_size=hidden_size//2,\n num_layers=num_layers,\n batch_first=True, \n bidirectional=bidir, \n dropout=dropout_rate)\n\n self.fc_out = nn.Linear(hidden_size, output_size) \n\n def forward(self, x):\n x = self.fc_in(x)\n x = self.ln(x)\n x = torch.relu(x)\n\n x, _ = self.gru(x) \n x = self.fc_out(x)\n return x\n```\n\n- Model: A very simple GRU model\n- No Downsampling, Using All Data\n- input Sequence length : 12x60x24x7=120,960\n- Removed series_id with no target\n- Features: anglez, enmo, enmo_diff, anglez_diff, rolling mean, std for enmo, anglez (ROLLING_WINDOW=[5, 30, 60], NaN values set to 0)\n- Clipped enmo outliers to 2\n- Applied Gaussian filter to target (Gaussian showed the best results among other filters)\n- Training: No use of LR Scheduler, Weight decay, or Gradient clipping\n- Reduced Gaussian filter's sigma value with each training epoch\n\n# 2. Tuned TUBO's Model (Public 0.750, Private 0.775)\n- Focused on Feature Engineering and Ensemble. Common features included [anglez, enmo, hour_sin, hour_cos].\nPreprocess 1: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs]\nPreprocess 2: [anglez_diff, anglez_diff_abs, anglez_diff.diff]\nPreprocess 3: [anglez_diff, anglez_diff_abs, enmo_diff_abs]\nPreprocess 4: [anglez_diff, anglez_diff_abs, anglez_diff.diff, anglez_diff_square]\nPreprocess 5: [anglez_diff, anglez_diff_abs, enmo_diff_abs, anglez_square]\nPreprocess 6: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs, anglez_diff_square, anglez_square]\nPreprocess 7: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs, anglez_diff_square, anglez_diff.diff, anglez_square]\nPreprocess 8: [anglez_diff, anglez_diff_abs, anglez_diff_square]\n- Ensemble of 25 models [P1: 7 models, P2: 3 models, P3: 3 models, P4: 3 models, P5: 1 model, P6: 2 models, P7: 3 models, P8: 3 models]. \n\nThank you for reading. Kudos to everyone who participated in the competition. I'm immensely grateful to everyone who provided well-crafted Public Notebooks!!!!", "votes": 10 } ], "comments": [], "raw_markdown_by_id": { "2550700": "I am very pleased to share that I've achieved a bronze medal in my second Featured Competition on Kaggle!\n\nInitially, I developed a simple Bi-GRU model of my own which i got ideas from amazing Public Notebooks. To my surprise, it ranked in the bronze medal range. (Public score 0.709, Private score 0.758). However, following the release of @tubotubo's excellent notebook, I decided to utilize it as well. My tuned version focused primarily on Feature Engineering and Ensemble (Public 0.750, Private 0.775).\n\nBelow, I briefly describe the details of my two solutions:\n\n# 1. KimSeungHee's Model (Public 0.709, Private 0.758)\n\n```python\nclass Bi_GRU_BASE_MODEL(nn.Module):\n def __init__(self, input_size, hidden_size, num_layers, output_size, dropout_rate, bidir=True):\n super(Bi_GRU_BASE_MODEL, self).__init__()\n self.input_size = input_size\n self.hidden_size = hidden_size\n self.output_size = output_size\n self.num_layers = num_layers\n \n self.fc_in = nn.Linear(input_size, hidden_size)\n \n # Layer Normalization\n self.ln = nn.LayerNorm(hidden_size)\n \n # GRU Layer\n self.gru = nn.GRU(input_size=hidden_size,\n hidden_size=hidden_size//2,\n num_layers=num_layers,\n batch_first=True, \n bidirectional=bidir, \n dropout=dropout_rate)\n\n self.fc_out = nn.Linear(hidden_size, output_size) \n\n def forward(self, x):\n x = self.fc_in(x)\n x = self.ln(x)\n x = torch.relu(x)\n\n x, _ = self.gru(x) \n x = self.fc_out(x)\n return x\n```\n\n- Model: A very simple GRU model\n- No Downsampling, Using All Data\n- input Sequence length : 12x60x24x7=120,960\n- Removed series_id with no target\n- Features: anglez, enmo, enmo_diff, anglez_diff, rolling mean, std for enmo, anglez (ROLLING_WINDOW=[5, 30, 60], NaN values set to 0)\n- Clipped enmo outliers to 2\n- Applied Gaussian filter to target (Gaussian showed the best results among other filters)\n- Training: No use of LR Scheduler, Weight decay, or Gradient clipping\n- Reduced Gaussian filter's sigma value with each training epoch\n\n# 2. Tuned TUBO's Model (Public 0.750, Private 0.775)\n- Focused on Feature Engineering and Ensemble. Common features included [anglez, enmo, hour_sin, hour_cos].\nPreprocess 1: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs]\nPreprocess 2: [anglez_diff, anglez_diff_abs, anglez_diff.diff]\nPreprocess 3: [anglez_diff, anglez_diff_abs, enmo_diff_abs]\nPreprocess 4: [anglez_diff, anglez_diff_abs, anglez_diff.diff, anglez_diff_square]\nPreprocess 5: [anglez_diff, anglez_diff_abs, enmo_diff_abs, anglez_square]\nPreprocess 6: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs, anglez_diff_square, anglez_square]\nPreprocess 7: [anglez_diff, enmo_diff, anglez_diff_abs, enmo_diff_abs, anglez_diff_square, anglez_diff.diff, anglez_square]\nPreprocess 8: [anglez_diff, anglez_diff_abs, anglez_diff_square]\n- Ensemble of 25 models [P1: 7 models, P2: 3 models, P3: 3 models, P4: 3 models, P5: 1 model, P6: 2 models, P7: 3 models, P8: 3 models]. \n\nThank you for reading. Kudos to everyone who participated in the competition. I'm immensely grateful to everyone who provided well-crafted Public Notebooks!!!!" } }