![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8060745%2Fff94c23e5aafd6df986cf467c6cf9254%2F_2025-06-03_223607_687.png?generation=1748961385687832&alt=media\")
Thank you to the organizers for this wonderful competition. I want to share my solution.
\nI used the original data and the provided interpolation script to extract more training data. Finally, I obtained 314,360 samples.
\nBut due to memory constraints, Kaggle couldn't train on all samples. So I used as many samples as possible within a memory threshold of about 210,000. That is, I randomly selected 210,000 samples for each training session.
\nFor feature engineering, I first manually added some simple features, but they didn't improve performance. So I abandoned traditional feature engineering and let the neural network extract features instead.
\nFor the \"geology_id\" column, I tried many methods like Word2Vec and KNN, but they reduced the model's performance. So, I dropped this column.
\nI used StandardScaler on the data. Since the input and target variables are essentially the same, I applied StandardScaler to both X and y, rather than just scaling the inputs as commonly done.
I use Kfold(5).\nAs mentioned here, the metric is actually equivalent to weighted MSE. So we can directly use it as the optimization metric instead of the default MSE.
class CustomLoss(Loss):\n def __init__(self, k_matrix):\n super().__init__()\n self.k_matrix = k_matrix\n\n def call(self, y_true, y_pred):\n diff = y_true - y_pred\n loss = diff * tf.convert_to_tensor(self.k_matrix,dtype=tf.float32) * diff\n return keras.backend.mean(loss)\nFor the model, it's based on my competition experience in the 'Forecasting Sticker Sales' competition, where I referenced these two notebooks and notebooks.
\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x)\n mha = MultiHeadAttention(num_heads=4, key_dim=64)(x, x)\n mha = LayerNormalization()(mha + x) \n mha = GlobalAveragePooling1D()(mha)\n ffn = Dense(512, activation=\"relu\")(mha)\n ffn = Dense(64, activation=\"relu\")(ffn)\n ffn = Dense(output_dim, activation=\"linear\")(ffn)\n model = keras.Model(inputs, ffn) \n return model\n
Position encoding is crucial in time series and transformers. Similarly, it's vital in geographical sequences, especially when order matters. I tried both sinusoidal positional encoding and learnable positional encoding, but found that directly adding positional encoding worked best.
\nAs mentioned earlier, I randomly selected 210,000 samples for each training session.
\nIf we use two-thirds of the data for training each time, seven iterations ensure that 99.9% of the samples are trained at least once.\n \\(1-(1/3)^6\\approx 0.9986\\).\n\\(1-(1/3)^7\\approx 0.9995\\).
\nSo:
\nfinal = (version 42+version 43+…+version 48)/7
\nWritten in a hurry, so there might be omissions. Feedback and corrections are welcome.
\nI have some questions that I would like to ask the organizers. @alin256
\nQuestion 1: I have the same issue here. \nWould it be acceptable if the entire model code, training logic, and inference pipeline are fully contained within the submitted Kaggle notebook? Must we upload our code to GitHub or FAIR?
\nQuestion 2: The overview mentioned fixing the random seed for reproducibility, but the rules didn't. Is it essential to fix the random seed or have the submitted code's output exactly match the submission file after running, or are minor differences acceptable as long as the results are all good?
\nI encountered an irreproducibility issue where adding a convolutional layer prevented identical outputs, and I couldn't resolve it before the competition ended. So even with all my code and data shared, the outputs can't match my submission. However, they consistently rank first. Does this situation exempt me from the identical output requirement?
\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x) #Adding this line prevents me from getting the same result even with a fixed random seed.\nFor example: \n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8060745%2F74897b0c6332d93f5030fdb275b229e4%2Fwechat_2025-06-02_230452_298.png?generation=1748876721572992&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8060745%2Fe1fa87ffaf257fef966f006358c40004%2Fwechat_2025-06-02_230602_797.png?generation=1748876795881749&alt=media\")
![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F8060745%2Fb498720f2e82cfdf2f1f8deda2967550%2Fwechat_2025-06-02_230614_485.png?generation=1748876808097833&alt=media\")
These are essentially the same code, yet none set a random seed. Still, all of them can secure the first place. This proves that the random seed wasn't the key to my work.
\nThanks for your time. I look forward to your response.
", "comments": [ { "id": 3217716, "author_name": "SergeyAlyaev", "author_url": "", "post_date": "2025-06-05T10:51:16.097000", "content": "I looked through your solution, and I am wondering if you trull tried to output 10 different solutions or just used the best average solution for all 10 answers? I understood it is the latter…
\nIf so, which strategy would you suggest to get diversified predictions?
", "votes": 0, "replies": [ { "id": 3218314, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-06T04:40:51.363000", "content": "First, to avoid any misunderstanding, I believe it's necessary to clarify the meaning of \"diversified.\" In my understanding, this means the prediction is not a traditional point prediction. Instead, the output should aim to cover all possible scenarios, or the output is a distribution. In other words, it is a prediction that incorporates uncertainty.
\nJust as the evaluation metrics implicitly suggest a Gaussian distribution, we can similarly assume that each position \\(x_t\\) follows a Gaussian distribution. The advantage of this is that the Gaussian distribution has very few parameters, and once we have obtained them, we can sample from it efficiently.
\n1. The simplest method is to first predict the mean for each position, and then predict the standard deviation for each position based on the residuals.
\nOne objection to this is that it might ignore the relationships between positions. This view is inaccurate. Our neural network uses a shared architecture, so it is capable of learning the relationships between positions.
\nAdditionally, assuming a Gaussian distribution is a relatively strong assumption.
\nWhen \\(x_{t-1}\\sim N(\\mu_{t-1},\\sigma_{t-1}^2)\\)，if \\(x_{t}\\sim N(\\mu_{t},\\sigma_{t}^2)\\)，then \\(x_t=A*x_{t-1}+b+\\epsilon\\)，and \\(\\epsilon\\) follows a Gaussian distribution. In other words, \\(x_t\\) can be seen as a linear transformation of \\(x_{t-1}\\) plus Gaussian noise. We can also apply the reparameterization trick to express this as Gaussian noise of the form \\(\\mu+\\sigma N(0,1)\\) . This does not involve a very complex transformation, so we may not need to use Markov Chains or more advanced tools. It can be learned using just the final linear layer of a neural network.
\n2. Another approach is to optimize the mean and standard deviation simultaneously, which is also what is done in CatBoost。
\nLet's explain this in detail，we assume that mean is\\(a_{i,0}\\)，var is \\(e^{2a_{i,1}}\\)，so \\(N(t_i|a_{i,0},e^{2a_{i,1}})=\\frac{1}{\\sqrt{2\\pi e^{2a_{i,1}}}}exp(-\\frac{(t_i-a_{i,0})^2}{2e^{2a_{i,1}}})\\)
\nNLL is \\(\\frac{1}{2}log2\\pi +a_{i,1}+\\frac{1}{2}e^{-2a_{i,1}}(t_i-a_{i,0})^2\\)，it only differs from CatBoost's version by a weight.
\nLet's see how to optimize it. Simply put:
\n$$L_0(a_0,a_1)=a_1+\\frac{1}{2}e^{-2a_1}(t-a_0)^2$$
\n$$\\frac{\\partial L}{\\partial a_0}=e^{-2a_1}(t-a_0)(-1)=-prec*diff$$
\n$$\\frac{\\partial L}{\\partial a_1}=1+\\frac{1}{2}(t-a_0)^2e^{-2a_1}(-2)=1-prec*diff^2$$
\nNote that CatBoost uses Newton's method, so the code in CatBoost is
\n(*der)[] = weight * diff;\n\n(*der)[] = weight * ((diff) * prec - );\n3. However, both of the above methods consider each position independently. We have an even more powerful tool that can also take the positions into account: the Gaussian Process.
\nWe are considering the relationships between each position, which means we need the covariance.
\nIn linear regression:
\n$$f(x)=\\Phi(x)^T w$$
\nFor simplicity, we consider the case where \\(w\\sim N(0,\\alpha^{-1}I)\\).
\n$$E[f(x)]=E[\\Phi(x)^T w]=\\Phi(x)^TE[ w]=0$$
\nFor \\(x,x'\\):
\n$$cov[f(x),f(x')]=E[(f(x)-E[f(x)])(f(x')-E[f(x')])]=E[f(x)f(x')]$$
$$=E[\\Phi(x)^T w\\Phi(x')^T w]$$
\n$$=\\Phi(x)^T E[ww^T]\\Phi(x')$$
\n$$=\\Phi(x)^T cov(w) \\Phi(x')$$
\n$$=K$$
\nTherefore, we can use a kernel function here.
\nA classic method that combines neural networks and Gaussian regression is Deep Kernel Learning.
\nThe above are just a few of my humble opinions. For more complex situations, such as those with more intricate relationships between positions or distributions that are not necessarily Gaussian, I do not have further insights at this time.
\nRegarding uncertainty, Kaggle also has a serious of competitions, and I think we could probably learn more methods from them.
\nPrediction interval competition II: House price
Prediction interval competition I: Birth weight
", "votes": 2, "replies": [ { "id": 3219686, "author_name": "SergeyAlyaev", "author_url": "", "post_date": "2025-06-08T07:24:31.630000", "content": "To be honest, you overwhelmed with your answer.
\nI hope, you can join the upcoming publication and we can have an in-depth discussion.
", "votes": 0, "replies": [] }, { "id": 3220798, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-10T01:54:13.967000", "content": "@alin256 Thank you so much for your kind words. I would be happy to discuss this further with you if the opportunity arises.
", "votes": 0, "replies": [] } ] } ] }, { "id": 3216147, "author_name": "Nikita Shevyrev", "author_url": "", "post_date": "2025-06-03T07:10:30.103000", "content": "@act18l I’ve made my final notebook public and created a discussion post linking to it, but I still don’t see the notebook icon appearing on the private leaderboard. Were there any additional steps you had to take to get the icon to appear on your row? Just want to make sure I’m not missing something. Thanks in advance!
", "votes": 0, "replies": [ { "id": 3216151, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-03T07:15:49.050000", "content": "You can set it up in the \"Share your solution\" section on this page.
", "votes": 1, "replies": [ { "id": 3216172, "author_name": "Nikita Shevyrev", "author_url": "", "post_date": "2025-06-03T07:38:44.817000", "content": "Ah, perfect — that did the trick. Thanks a lot for the quick help!
", "votes": 1, "replies": [] } ] } ] }, { "id": 3215921, "author_name": "SergeyAlyaev", "author_url": "", "post_date": "2025-06-02T21:17:38.970000", "content": "@act18l, congratulations on winning!
\nWe are asking to upload a solution to GitHub or Fair-aligned permanent repository because they offer a transparent way to add the Open Access license: BSD 2-Clause was selected. https://opensource.org/license/bsd-2-clause
\nUploading your existing notebook to GitHub can be a simple solution. I did not find the link to your latest full solution, but I am not a very experienced kaggler.
As for the fixed seed - it's preferable to have, but we would not be too strict about it as long as the solution retains the top position.
\nI am looking forward to studying your model in more detail. Amazing result!
", "votes": 0, "replies": [ { "id": 3215999, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-03T01:37:28.903000", "content": "Thanks for your reply! Kaggle notebooks can indeed only be shared under the Apache 2.0 License. I'll upload it to GitHub soon.
\nI didn't tidy up the code and didn't make the kaggle notebook public before.
\nFor now, I've made it public. Here's the code.
\nCode:
The dataset \"geo_submission\" contains the results from versions of the above notebooks.
\n\nIn the competition, I chose to \"Single Model(Version 25)\" and \"Ensemble(Version 9)\" as my submission.
\nYou can use \"Compare Versions\" to check the code changes.
\nI apologize for its current messy codes. I'll clean it up and add comments in the future.
", "votes": 1, "replies": [] }, { "id": 3216453, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-03T15:55:32.203000", "content": "I’ve organized my code, added comments, and add more details about model.
\nHere’s my GitHub link. The Kaggle notebook has also been updated.
\nPlease let me know if there are any further updates needed on my part.
@act18l, can you please send me a direct message, so that I have your email address for the prize fulfillment. Note, that I will need to share your contact details with ROGII Inc. who are sponsoring the prize pot.
", "votes": 0, "replies": [] }, { "id": 3217093, "author_name": "Spiritmilk", "author_url": "", "post_date": "2025-06-04T14:19:29.877000", "content": "@alin256 Sure! My email address is oldcity962@gmail.com
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 12, "messages": [ { "id": 3215750, "postDate": "2025-06-02T15:51:55.867Z", "content": "Thank you to the organizers for this wonderful competition. I want to share my solution.
\nI used the original data and the provided interpolation script to extract more training data. Finally, I obtained 314,360 samples.
\nBut due to memory constraints, Kaggle couldn't train on all samples. So I used as many samples as possible within a memory threshold of about 210,000. That is, I randomly selected 210,000 samples for each training session.
\nFor feature engineering, I first manually added some simple features, but they didn't improve performance. So I abandoned traditional feature engineering and let the neural network extract features instead.
\nFor the \"geology_id\" column, I tried many methods like Word2Vec and KNN, but they reduced the model's performance. So, I dropped this column.
\nI used StandardScaler on the data. Since the input and target variables are essentially the same, I applied StandardScaler to both X and y, rather than just scaling the inputs as commonly done.
I use Kfold(5).\nAs mentioned here, the metric is actually equivalent to weighted MSE. So we can directly use it as the optimization metric instead of the default MSE.
class CustomLoss(Loss):\n def __init__(self, k_matrix):\n super().__init__()\n self.k_matrix = k_matrix\n\n def call(self, y_true, y_pred):\n diff = y_true - y_pred\n loss = diff * tf.convert_to_tensor(self.k_matrix,dtype=tf.float32) * diff\n return keras.backend.mean(loss)\nFor the model, it's based on my competition experience in the 'Forecasting Sticker Sales' competition, where I referenced these two notebooks and notebooks.
\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x)\n mha = MultiHeadAttention(num_heads=4, key_dim=64)(x, x)\n mha = LayerNormalization()(mha + x) \n mha = GlobalAveragePooling1D()(mha)\n ffn = Dense(512, activation=\"relu\")(mha)\n ffn = Dense(64, activation=\"relu\")(ffn)\n ffn = Dense(output_dim, activation=\"linear\")(ffn)\n model = keras.Model(inputs, ffn) \n return model\n
Position encoding is crucial in time series and transformers. Similarly, it's vital in geographical sequences, especially when order matters. I tried both sinusoidal positional encoding and learnable positional encoding, but found that directly adding positional encoding worked best.
\nAs mentioned earlier, I randomly selected 210,000 samples for each training session.
\nIf we use two-thirds of the data for training each time, seven iterations ensure that 99.9% of the samples are trained at least once.\n \\(1-(1/3)^6\\approx 0.9986\\).\n\\(1-(1/3)^7\\approx 0.9995\\).
\nSo:
\nfinal = (version 42+version 43+…+version 48)/7
\nWritten in a hurry, so there might be omissions. Feedback and corrections are welcome.
\nI have some questions that I would like to ask the organizers. @alin256
\nQuestion 1: I have the same issue here. \nWould it be acceptable if the entire model code, training logic, and inference pipeline are fully contained within the submitted Kaggle notebook? Must we upload our code to GitHub or FAIR?
\nQuestion 2: The overview mentioned fixing the random seed for reproducibility, but the rules didn't. Is it essential to fix the random seed or have the submitted code's output exactly match the submission file after running, or are minor differences acceptable as long as the results are all good?
\nI encountered an irreproducibility issue where adding a convolutional layer prevented identical outputs, and I couldn't resolve it before the competition ended. So even with all my code and data shared, the outputs can't match my submission. However, they consistently rank first. Does this situation exempt me from the identical output requirement?
\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x) #Adding this line prevents me from getting the same result even with a fixed random seed.\nFor example: \n
These are essentially the same code, yet none set a random seed. Still, all of them can secure the first place. This proves that the random seed wasn't the key to my work.
\nThanks for your time. I look forward to your response.
", "rawMarkdown": "Thank you to the organizers for this wonderful competition. I want to share my solution.\n\n## Data Section\nI used the original data and the provided interpolation script to extract more training data. Finally, I obtained 314,360 samples.\n\nBut due to memory constraints, Kaggle couldn't train on all samples. So I used as many samples as possible within a memory threshold of about 210,000. That is, I randomly selected 210,000 samples for each training session.\n\nFor feature engineering, I first manually added some simple features, but they didn't improve performance. So I abandoned traditional feature engineering and let the neural network extract features instead. \n\nFor the \"geology_id\" column, I tried many methods like Word2Vec and KNN, but they reduced the model's performance. So, I dropped this column.\n\nI used `StandardScaler` on the data. Since the input and target variables are essentially the same, I applied `StandardScaler` to both X and y, rather than just scaling the inputs as commonly done.\n\n\n## Training and Model\nI use `Kfold(5)`.\nAs mentioned [here](https://www.kaggle.com/competitions/geology-forecast-challenge-open/discussion/569884), the metric is actually equivalent to weighted MSE. So we can directly use it as the optimization metric instead of the default MSE.\n```python\nclass CustomLoss(Loss):\n def __init__(self, k_matrix):\n super().__init__()\n self.k_matrix = k_matrix\n\n def call(self, y_true, y_pred):\n diff = y_true - y_pred\n loss = diff * tf.convert_to_tensor(self.k_matrix,dtype=tf.float32) * diff\n return keras.backend.mean(loss)\n```\nFor the model, it's based on my competition experience in the '[Forecasting Sticker Sales](https://www.kaggle.com/competitions/playground-series-s5e1)' competition, where I referenced these two [notebooks](https://www.kaggle.com/code/act18l/convnet-starter-lb-0-052) and [notebooks](https://www.kaggle.com/code/cdeotte/transformer-starter-lb-0-052).\n\n```python\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x)\n mha = MultiHeadAttention(num_heads=4, key_dim=64)(x, x)\n mha = LayerNormalization()(mha + x) \n mha = GlobalAveragePooling1D()(mha)\n ffn = Dense(512, activation=\"relu\")(mha)\n ffn = Dense(64, activation=\"relu\")(ffn)\n ffn = Dense(output_dim, activation=\"linear\")(ffn)\n model = keras.Model(inputs, ffn) \n return model\n```\n\n\nPosition encoding is crucial in time series and transformers. Similarly, it's vital in geographical sequences, especially when order matters. I tried both sinusoidal positional encoding and learnable positional encoding, but found that directly adding positional encoding worked best.\n\n\n## Ensemble\nAs mentioned earlier, I randomly selected 210,000 samples for each training session.\n\nIf we use two-thirds of the data for training each time, seven iterations ensure that 99.9% of the samples are trained at least once.\n \\\\(1-(1/3)^6\\approx 0.9986\\\\).\n\\\\(1-(1/3)^7\\approx 0.9995\\\\).\n\n\nSo:\n\nfinal = (version 42+version 43+...+version 48)/7\n\n---\n\n\n\n**Written in a hurry, so there might be omissions. Feedback and corrections are welcome.**\n\n---\n\nI have some questions that I would like to ask the organizers. @alin256\n\n**Question 1:** I have the same [issue](https://www.kaggle.com/competitions/geology-forecast-challenge-open/discussion/582452) here. \nWould it be acceptable if the entire model code, training logic, and inference pipeline are fully contained within the submitted Kaggle notebook? Must we upload our code to GitHub or FAIR?\n\n\n**Question 2:** The overview mentioned fixing the random seed for reproducibility, but the rules didn't. Is it essential to fix the random seed or have the submitted code's output exactly match the submission file after running, or are minor differences acceptable as long as the results are all good?\n\nI encountered an irreproducibility issue where adding a convolutional layer prevented identical outputs, and I couldn't resolve it before the competition ended. So even with all my code and data shared, the outputs can't match my submission. However, they consistently rank first. Does this situation exempt me from the identical output requirement?\n\n```python\ndef build_model(input_dim, output_dim):\n inputs = Input(shape=(input_dim,))\n col_indices = tf.range(1, 301, dtype=tf.float32) \n inputs = col_indices + inputs \n x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)\n x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x) #Adding this line prevents me from getting the same result even with a fixed random seed.\n```\n\nFor example: \n\n\n\n\n\n\nThese are essentially the same code, yet none set a random seed. Still, all of them can secure the first place. This proves that the random seed wasn't the key to my work.\n\nThanks for your time. I look forward to your response.", "votes": 7 }, { "id": 3217716, "postDate": "2025-06-05T10:51:16.097Z", "content": "I looked through your solution, and I am wondering if you trull tried to output 10 different solutions or just used the best average solution for all 10 answers? I understood it is the latter…
\nIf so, which strategy would you suggest to get diversified predictions?
", "rawMarkdown": "I looked through your solution, and I am wondering if you trull tried to output 10 different solutions or just used the best average solution for all 10 answers? I understood it is the latter...\n\nIf so, which strategy would you suggest to get diversified predictions?", "replies": [ { "id": 3218314, "postDate": "2025-06-06T04:40:51.363Z", "content": "First, to avoid any misunderstanding, I believe it's necessary to clarify the meaning of \"diversified.\" In my understanding, this means the prediction is not a traditional point prediction. Instead, the output should aim to cover all possible scenarios, or the output is a distribution. In other words, it is a prediction that incorporates uncertainty.
\nJust as the evaluation metrics implicitly suggest a Gaussian distribution, we can similarly assume that each position \\(x_t\\) follows a Gaussian distribution. The advantage of this is that the Gaussian distribution has very few parameters, and once we have obtained them, we can sample from it efficiently.
\n1. The simplest method is to first predict the mean for each position, and then predict the standard deviation for each position based on the residuals.
\nOne objection to this is that it might ignore the relationships between positions. This view is inaccurate. Our neural network uses a shared architecture, so it is capable of learning the relationships between positions.
\nAdditionally, assuming a Gaussian distribution is a relatively strong assumption.
\nWhen \\(x_{t-1}\\sim N(\\mu_{t-1},\\sigma_{t-1}^2)\\)，if \\(x_{t}\\sim N(\\mu_{t},\\sigma_{t}^2)\\)，then \\(x_t=A*x_{t-1}+b+\\epsilon\\)，and \\(\\epsilon\\) follows a Gaussian distribution. In other words, \\(x_t\\) can be seen as a linear transformation of \\(x_{t-1}\\) plus Gaussian noise. We can also apply the reparameterization trick to express this as Gaussian noise of the form \\(\\mu+\\sigma N(0,1)\\) . This does not involve a very complex transformation, so we may not need to use Markov Chains or more advanced tools. It can be learned using just the final linear layer of a neural network.
\n2. Another approach is to optimize the mean and standard deviation simultaneously, which is also what is done in CatBoost。
\nLet's explain this in detail，we assume that mean is\\(a_{i,0}\\)，var is \\(e^{2a_{i,1}}\\)，so \\(N(t_i|a_{i,0},e^{2a_{i,1}})=\\frac{1}{\\sqrt{2\\pi e^{2a_{i,1}}}}exp(-\\frac{(t_i-a_{i,0})^2}{2e^{2a_{i,1}}})\\)
\nNLL is \\(\\frac{1}{2}log2\\pi +a_{i,1}+\\frac{1}{2}e^{-2a_{i,1}}(t_i-a_{i,0})^2\\)，it only differs from CatBoost's version by a weight.
\nLet's see how to optimize it. Simply put:
\n$$L_0(a_0,a_1)=a_1+\\frac{1}{2}e^{-2a_1}(t-a_0)^2$$
\n$$\\frac{\\partial L}{\\partial a_0}=e^{-2a_1}(t-a_0)(-1)=-prec*diff$$
\n$$\\frac{\\partial L}{\\partial a_1}=1+\\frac{1}{2}(t-a_0)^2e^{-2a_1}(-2)=1-prec*diff^2$$
\nNote that CatBoost uses Newton's method, so the code in CatBoost is
\n(*der)[] = weight * diff;\n\n(*der)[] = weight * ((diff) * prec - );\n3. However, both of the above methods consider each position independently. We have an even more powerful tool that can also take the positions into account: the Gaussian Process.
\nWe are considering the relationships between each position, which means we need the covariance.
\nIn linear regression:
\n$$f(x)=\\Phi(x)^T w$$
\nFor simplicity, we consider the case where \\(w\\sim N(0,\\alpha^{-1}I)\\).
\n$$E[f(x)]=E[\\Phi(x)^T w]=\\Phi(x)^TE[ w]=0$$
\nFor \\(x,x'\\):
\n$$cov[f(x),f(x')]=E[(f(x)-E[f(x)])(f(x')-E[f(x')])]=E[f(x)f(x')]$$
$$=E[\\Phi(x)^T w\\Phi(x')^T w]$$
\n$$=\\Phi(x)^T E[ww^T]\\Phi(x')$$
\n$$=\\Phi(x)^T cov(w) \\Phi(x')$$
\n$$=K$$
\nTherefore, we can use a kernel function here.
\nA classic method that combines neural networks and Gaussian regression is Deep Kernel Learning.
\nThe above are just a few of my humble opinions. For more complex situations, such as those with more intricate relationships between positions or distributions that are not necessarily Gaussian, I do not have further insights at this time.
\nRegarding uncertainty, Kaggle also has a serious of competitions, and I think we could probably learn more methods from them.
\nPrediction interval competition II: House price
Prediction interval competition I: Birth weight
", "rawMarkdown": "First, to avoid any misunderstanding, I believe it's necessary to clarify the meaning of \"diversified.\" In my understanding, this means the prediction is not a traditional point prediction. Instead, the output should aim to cover all possible scenarios, or the output is a distribution. In other words, it is a prediction that incorporates uncertainty.\n\nJust as the evaluation metrics implicitly suggest a Gaussian distribution, we can similarly assume that each position \\\\(x_t\\\\) follows a Gaussian distribution. The advantage of this is that the Gaussian distribution has very few parameters, and once we have obtained them, we can sample from it efficiently.\n\n**1.** The simplest method is to first predict the mean for each position, and then predict the standard deviation for each position based on the residuals.\n\nOne objection to this is that it might ignore the relationships between positions. This view is inaccurate. Our neural network uses a shared architecture, so it is capable of learning the relationships between positions.\n\nAdditionally, assuming a Gaussian distribution is a relatively strong assumption.\n\nWhen \\\\(x_{t-1}\\sim N(\\mu_{t-1},\\sigma_{t-1}^2)\\\\)，if \\\\(x_{t}\\sim N(\\mu_{t},\\sigma_{t}^2)\\\\)，then \\\\(x_t=A*x_{t-1}+b+\\epsilon\\\\)，and \\\\(\\epsilon\\\\) follows a Gaussian distribution. In other words, \\\\(x_t\\\\) can be seen as a linear transformation of \\\\(x_{t-1}\\\\) plus Gaussian noise. We can also apply the reparameterization trick to express this as Gaussian noise of the form \\\\(\\mu+\\sigma N(0,1)\\\\) . This does not involve a very complex transformation, so we may not need to use Markov Chains or more advanced tools. It can be learned using just the final linear layer of a neural network.\n\n\n**2.** Another approach is to optimize the mean and standard deviation simultaneously, which is also [what is done in CatBoost](https://catboost.ai/docs/en/concepts/loss-functions-regression#RMSEWithUncertainty)。\n\nLet's explain this in detail，we assume that mean is\\\\(a_{i,0}\\\\)，var is \\\\(e^{2a_{i,1}}\\\\)，so \\\\(N(t_i|a_{i,0},e^{2a_{i,1}})=\\frac{1}{\\sqrt{2\\pi e^{2a_{i,1}}}}exp(-\\frac{(t_i-a_{i,0})^2}{2e^{2a_{i,1}}})\\\\)\n\nNLL is \\\\(\\frac{1}{2}log2\\pi +a_{i,1}+\\frac{1}{2}e^{-2a_{i,1}}(t_i-a_{i,0})^2\\\\)，it only differs from CatBoost's version by a weight.\n\n\n\nLet's see how to optimize it. Simply put:\n\n$$L_0(a_0,a_1)=a_1+\\frac{1}{2}e^{-2a_1}(t-a_0)^2$$\n\n$$\\frac{\\partial L}{\\partial a_0}=e^{-2a_1}(t-a_0)(-1)=-prec*diff$$\n\n$$\\frac{\\partial L}{\\partial a_1}=1+\\frac{1}{2}(t-a_0)^2e^{-2a_1}(-2)=1-prec*diff^2$$\n\nNote that CatBoost uses Newton's method, so the [code](https://github.com/catboost/catboost/blob/c128d0256cb702e844a0a3c0423b105465752e74/catboost/private/libs/algo_helpers/error_functions.h#L280) in CatBoost is\n\n```C++\n(*der)[0] = weight * diff;\n\n(*der)[1] = weight * (Sqr(diff) * prec - 1);\n```\n\n**3.** However, both of the above methods consider each position independently. We have an even more powerful tool that can also take the positions into account: the Gaussian Process.\n\nWe are considering the relationships between each position, which means we need the covariance.\n\nIn linear regression:\n\n$$f(x)=\\Phi(x)^T w$$\n\nFor simplicity, we consider the case where \\\\(w\\sim N(0,\\alpha^{-1}I)\\\\). \n\n$$E[f(x)]=E[\\Phi(x)^T w]=\\Phi(x)^TE[ w]=0$$\n\nFor \\\\(x,x'\\\\):\n$$cov[f(x),f(x')]=E[(f(x)-E[f(x)])(f(x')-E[f(x')])]=E[f(x)f(x')]$$\n\n$$=E[\\Phi(x)^T w\\Phi(x')^T w]$$\n\n$$=\\Phi(x)^T E[ww^T]\\Phi(x')$$\n\n\n$$=\\Phi(x)^T cov(w) \\Phi(x')$$\n\n$$=K$$\n\nTherefore, we can use a kernel function here.\n\nA classic method that combines neural networks and Gaussian regression is [Deep Kernel Learning](https://arxiv.org/abs/1511.02222).\n\n---\n\nThe above are just a few of my humble opinions. For more complex situations, such as those with more intricate relationships between positions or distributions that are not necessarily Gaussian, I do not have further insights at this time.\n\nRegarding uncertainty, Kaggle also has a serious of competitions, and I think we could probably learn more methods from them.\n[Prediction interval competition II: House price](https://www.kaggle.com/competitions/prediction-interval-competition-ii-house-price)\n\n[Prediction interval competition I: Birth weight](https://www.kaggle.com/competitions/prediction-interval-competition-i-birth-weight)", "votes": 2, "replies": [ { "id": 3219686, "postDate": "2025-06-08T07:24:31.630Z", "content": "To be honest, you overwhelmed with your answer.
\nI hope, you can join the upcoming publication and we can have an in-depth discussion.
", "rawMarkdown": "To be honest, you overwhelmed with your answer.\n\nI hope, you can join the upcoming publication and we can have an in-depth discussion." }, { "id": 3220798, "postDate": "2025-06-10T01:54:13.967Z", "content": "@alin256 Thank you so much for your kind words. I would be happy to discuss this further with you if the opportunity arises.
", "rawMarkdown": "@alin256 Thank you so much for your kind words. I would be happy to discuss this further with you if the opportunity arises." } ] } ] }, { "id": 3216147, "postDate": "2025-06-03T07:10:30.103Z", "content": "@act18l I’ve made my final notebook public and created a discussion post linking to it, but I still don’t see the notebook icon appearing on the private leaderboard. Were there any additional steps you had to take to get the icon to appear on your row? Just want to make sure I’m not missing something. Thanks in advance!
", "rawMarkdown": "@act18l I’ve made my final notebook public and created a discussion post linking to it, but I still don’t see the notebook icon appearing on the private leaderboard. Were there any additional steps you had to take to get the icon to appear on your row? Just want to make sure I’m not missing something. Thanks in advance!", "replies": [ { "id": 3216151, "postDate": "2025-06-03T07:15:49.050Z", "content": "You can set it up in the \"Share your solution\" section on this page.
", "rawMarkdown": "You can set it up in the \"Share your solution\" section on [this page](https://www.kaggle.com/competitions/geology-forecast-challenge-open/team).", "votes": 1, "replies": [ { "id": 3216172, "postDate": "2025-06-03T07:38:44.817Z", "content": "Ah, perfect — that did the trick. Thanks a lot for the quick help!
", "rawMarkdown": "Ah, perfect — that did the trick. Thanks a lot for the quick help!", "votes": 1 } ] } ] }, { "id": 3215921, "postDate": "2025-06-02T21:17:38.970Z", "content": "@act18l, congratulations on winning!
\nWe are asking to upload a solution to GitHub or Fair-aligned permanent repository because they offer a transparent way to add the Open Access license: BSD 2-Clause was selected. https://opensource.org/license/bsd-2-clause
\nUploading your existing notebook to GitHub can be a simple solution. I did not find the link to your latest full solution, but I am not a very experienced kaggler.
As for the fixed seed - it's preferable to have, but we would not be too strict about it as long as the solution retains the top position.
\nI am looking forward to studying your model in more detail. Amazing result!
", "rawMarkdown": "@act18l, congratulations on winning!\n\n\nWe are asking to upload a solution to GitHub or Fair-aligned permanent repository because they offer a transparent way to add the Open Access license: BSD 2-Clause was selected. https://opensource.org/license/bsd-2-clause\nUploading your existing notebook to GitHub can be a simple solution. I did not find the link to your latest full solution, but I am not a very experienced kaggler. \n\nAs for the fixed seed - it's preferable to have, but we would not be too strict about it as long as the solution retains the top position.\n\nI am looking forward to studying your model in more detail. Amazing result!", "replies": [ { "id": 3215999, "postDate": "2025-06-03T01:37:28.903Z", "content": "Thanks for your reply! Kaggle notebooks can indeed only be shared under the Apache 2.0 License. I'll upload it to GitHub soon.
\nI didn't tidy up the code and didn't make the kaggle notebook public before.
\nFor now, I've made it public. Here's the code.
\nCode:
The dataset \"geo_submission\" contains the results from versions of the above notebooks.
\n\nIn the competition, I chose to \"Single Model(Version 25)\" and \"Ensemble(Version 9)\" as my submission.
\nYou can use \"Compare Versions\" to check the code changes.
\nI apologize for its current messy codes. I'll clean it up and add comments in the future.
", "rawMarkdown": "Thanks for your reply! Kaggle notebooks can indeed only be shared under the Apache 2.0 License. I'll upload it to GitHub soon.\n\nI didn't tidy up the code and didn't make the kaggle notebook public before.\nFor now, I've made it public. Here's the code.\n**Code:**\n\n[\"Generate dataset\"](https://www.kaggle.com/code/act18l/generate-dataset)\n\n[\"Single Model\"](https://www.kaggle.com/code/act18l/single-model-for-geology-forecast-challenge) \n\nThe dataset [\"geo_submission\"](https://www.kaggle.com/datasets/act18l/geo-submission/data) contains the results from versions of the above notebooks.\n\n\n[\"Emsemble\"](https://www.kaggle.com/code/act18l/ensemble-for-geology-forecast-challenge)\n\nIn the competition, I chose to **\"Single Model(Version 25)\"** and **\"Ensemble(Version 9)\"** as my submission.\n\nYou can use \"Compare Versions\" to check the code changes.\n\n---\n\nI apologize for its current messy codes. I'll clean it up and add comments in the future.\n", "votes": 1 }, { "id": 3216453, "postDate": "2025-06-03T15:55:32.203Z", "content": "I’ve organized my code, added comments, and add more details about model.
\nHere’s my GitHub link. The Kaggle notebook has also been updated.
\nPlease let me know if there are any further updates needed on my part.
@act18l, can you please send me a direct message, so that I have your email address for the prize fulfillment. Note, that I will need to share your contact details with ROGII Inc. who are sponsoring the prize pot.
", "rawMarkdown": "@act18l, can you please send me a direct message, so that I have your email address for the prize fulfillment. Note, that I will need to share your contact details with ROGII Inc. who are sponsoring the prize pot." }, { "id": 3217093, "postDate": "2025-06-04T14:19:29.877Z", "content": "@alin256 Sure! My email address is oldcity962@gmail.com
", "rawMarkdown": "@alin256 Sure! My email address is oldcity962@gmail.com" } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T12:42:19Z", "competition_id": 95697, "competition_slug": "geology-forecast-challenge-open", "team_id": 13518048, "solution_url": "/competitions/geology-forecast-challenge-open/writeups/spiritmilk-public-6th-private-1st-solution" }