# public 6th, private 1st solution

Thank you to the organizers for this wonderful competition. I want to  share my solution.
Data Section
I used the original data and the provided interpolation script to extract more training data. Finally, I obtained 314,360 samples.
But due to memory constraints, Kaggle couldn't train on all samples. So I used as many samples as possible within a memory threshold of about 210,000. That is, I randomly selected 210,000 samples for each training session.
For feature engineering, I first manually added some simple features, but they didn't improve performance. So I abandoned traditional feature engineering and let the neural network extract features instead.
For the "geology_id" column, I tried many methods like Word2Vec and KNN, but they reduced the model's performance. So, I dropped this column.
I used
StandardScaler
on the data. Since the input and target variables are essentially the same, I applied
StandardScaler
to both X and y, rather than just scaling the inputs as commonly done.
Training and Model
I use
Kfold(5)
.
As mentioned
[here](https://www.kaggle.com/competitions/geology-forecast-challenge-open/discussion/569884)
, the  metric is actually equivalent to weighted MSE. So we can directly use it as the optimization metric instead of the default MSE.
class CustomLoss(Loss):
    def __init__(self, k_matrix):
        super().__init__()
        self.k_matrix = k_matrix

    def call(self, y_true, y_pred):
        diff = y_true - y_pred
        loss = diff * tf.convert_to_tensor(self.k_matrix,dtype=tf.float32) * diff
        return keras.backend.mean(loss)
For the model, it's based on my competition experience in the '
[Forecasting Sticker Sales](https://www.kaggle.com/competitions/playground-series-s5e1)
' competition, where I referenced these two
[notebooks](https://www.kaggle.com/code/act18l/convnet-starter-lb-0-052)
and
[notebooks](https://www.kaggle.com/code/cdeotte/transformer-starter-lb-0-052)
.
def build_model(input_dim, output_dim):
    inputs = Input(shape=(input_dim,))
    col_indices = tf.range(1, 301, dtype=tf.float32)  
    inputs =  col_indices + inputs  
    x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)
    x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x)
    mha = MultiHeadAttention(num_heads=4, key_dim=64)(x, x)
    mha = LayerNormalization()(mha + x)  
    mha = GlobalAveragePooling1D()(mha)
    ffn = Dense(512, activation="relu")(mha)
    ffn = Dense(64, activation="relu")(ffn)
    ffn = Dense(output_dim, activation="linear")(ffn)
    model = keras.Model(inputs, ffn)  
    return model
![](assets/0a9266dbf53b83f4a77b1b8f.png)
Position encoding is crucial in time series and transformers. Similarly, it's vital in geographical sequences, especially when order matters. I tried both sinusoidal positional encoding and learnable positional encoding, but found that directly adding positional encoding worked best.
Ensemble
As mentioned earlier, I randomly selected 210,000 samples for each training session.
If we use two-thirds of the data for training each time, seven iterations ensure that 99.9% of the samples are trained at least once.
 \(1-(1/3)^6\approx 0.9986\).
\(1-(1/3)^7\approx 0.9995\).
So:
final = (version 42+version 43+…+version 48)/7
Written in a hurry, so there might be omissions. Feedback and corrections are welcome.
I have some questions that I would like to ask the organizers.
[@alin256](https://www.kaggle.com/alin256)
Question 1:
I have the same
[issue](https://www.kaggle.com/competitions/geology-forecast-challenge-open/discussion/582452)
here. 
Would it be acceptable if the entire model code, training logic, and inference pipeline are fully contained within the submitted Kaggle notebook? Must we upload our code to GitHub or FAIR?
Question 2:
The overview mentioned fixing the random seed for reproducibility, but the rules didn't. Is it essential to fix the random seed or have the submitted code's output exactly match the submission file after running, or are minor differences acceptable as long as the results are all good?
I encountered an irreproducibility issue where adding a convolutional layer prevented identical outputs, and I couldn't resolve it before the competition ended. So even with all my code and data shared, the outputs can't match my submission. However, they consistently rank first. Does this situation exempt me from the identical output requirement?
def build_model(input_dim, output_dim):
    inputs = Input(shape=(input_dim,))
    col_indices = tf.range(1, 301, dtype=tf.float32)  
    inputs =  col_indices + inputs  
    x = Lambda(lambda t: tf.expand_dims(t, axis=1))(inputs)
    x = Conv1D(filters=64, kernel_size=7, padding='same', activation='relu')(x) #Adding this line prevents me from getting the same result even with a fixed random seed.
For example:
![](assets/a455d426b56f4e01cf034a96.png)
![](assets/7e7662489aa21ee0af152d11.png)
![](assets/b58322e84c97af9d59e7d270.png)
These are essentially the same code, yet none set a random seed. Still, all of them can secure the first place. This proves that the random seed wasn't the key to my work.
Thanks for your time. I look forward to your response.