# 1st solution using count-based features (Single model PB 0.54942)

First, I want to express my sincere gratitude to the organizers for providing such a large-scale, real-world dataset. It offered a rare opportunity to dive deep into practical challenges and learn from them.
I also want to thank the authors of several outstanding notebooks that guided my journey:
I began with
[XGBoost Ranker with Polars](https://www.kaggle.com/code/ka1242/xgboost-ranker-with-polars)
a clean and efficient starting point that helped me grasp the essentials of ranking models. Thank you
[@ka1242](https://www.kaggle.com/ka1242)
!
The most creative idea, in my opinion, came from
[reranking approach](https://www.kaggle.com/code/mango789/xgboost-ranker-rule-based-rerank)
. The rule-based reranking worked like a charm, and without that notebook, I wouldn’t have realized how crucial
flight_hash
is to the final performance. Huge thanks to
[@mango789](https://www.kaggle.com/mango789)
!
Approach overview and note on test data usage
My main single-model solution initially leveraged a leak-like feature: I included the test data from the very beginning of the competition. Count-based categorical encoding built on the full dataset noticeably improved both local validation and the public leaderboard.
I recall a discussion where the host mentioned that using the test set was acceptable because it contains no labels. While some recommendation competitions forbid using the test data, I didn’t find such a restriction here. Still, in real scenarios we cannot rely on future data, and we usually predict day by day using only past labels. A streaming setup would better reflect production.
Below I share results for the same single model trained without the test data (i.e., I removed the test set from the count-based encodings and training input).
Results without using test data
model
valid
LB
PB
best single model
0.504745
0.54062
0.54156
best single model without testdata
0.478704
0.51978
0.52692
This is an ongoing write-up; I may refine it or add more data stats later.
Key components
Validation strategy
Temporal split:
This is a recommendation task, so datetime order matters. The dataset spans
requestDate
from 2024-05-17 03:03:08 to 2024-12-31 18:54:00, which I index as days 1–229.
Fold design:
I used a single temporal fold: train on days 1–103, validate on days 104–166, and test on days 167–229.
Rationale:
Validation and test windows have similar lengths, and local validation generally matched the LB, especially for larger improvements. When there was a mismatch, I trusted local validation. One exception was the final Optuna weight tuning for ensembles, which can overfit to the validation set.
Count-based categorical encoding
XGBoost / LightGBM:
I applied count-based encoding to all categorical features, then treated all columns as numerical — no categorical indices were passed into training.
CatBoost:
I also experimented with exactly the same count-based encoding here, converting everything to numerical features. However, in my tests, all variations yielded very similar results, with no meaningful gain over the baseline.
Time features
Importance:
Time-derived features gave the largest lift compared to using only the provided CSV fields.
Cyclical encoding:
I added sine/cosine transforms so the model can capture wrap-around (e.g., hours 23 and 0 are adjacent).
Duration and distance features
Accurate duration:
I converted local times to UTC to compute true elapsed durations.
Distance-based signals:
I computed airport distances from lat/lon and derived related features such as
direct_price_per_km
.
Simple rank-based features on important base features
Motivation:
Ranking proxies (per group) for price, duration, and itinerary complexity helped the ranker focus on relative ordering within a user/query.
Ranking features code (Polars)
polars  pl


rank_order = {
    : ,
    : ,
    : ,
    : ,
    : ,
    : ,
    : ,
    : ,
    : ,
}

 () :
    exprs = []
     col, order  rank_order.items():
         col  df.columns:
            exprs.append(
                pl.col(col)
                  .rank(method=, descending=(order == ))
                  .over(group_col)
                  .alias()
            )
     df.with_columns(exprs)
There are more experiments to be done, for example adding  'miniRules0_statusInfos': 'desc',  'miniRules1_statusInfos': 'desc' will further improve local valid and online PB, unfortunately LB drop:)
model
valid
LB
PB
best single model
0.504745
0.54062
0.54156
adding more rank feats like  miniRules*_statusInfos
0.506949
0.53557
0.54284
Additonal ranking features based on (ranker_id, flight_hash)
As we fould rule based on flight_hash could improve local and online, we could add them to our feats so tree models could learn more.
So in additon to add_ranking_feats(df, 'ranker_id')  I also added add_ranking_feats(df, (ranker_id, flight_hash))
COLS_TO_COMPARE = [
    ,
    ,
    ,
    ,
    ,
    
]

df = df.with_columns(
    pl.concat_str(
        [pl.col(c).cast().fill_null()  c  COLS_TO_COMPARE]
    ).alias()
)

df = add_ranking_feats(df, )
df = add_ranking_feats(df, [, ])
Couting of important base features groupby profileId and compayID
():
  exprs = []
   col  rank_order.keys():
     col  df.columns:
      exprs.extend([
        pl.col(col).mean().over(group_col).alias(),
        pl.col(col).().over(group_col).alias(),
        pl.col(col).().over(group_col).alias(),
        pl.col(col).std().over(group_col).alias(),
        pl.col(col).median().over(group_col).alias(),
      ])
  df = df.with_columns(exprs)
  exprs = []
   col  rank_order.keys():
     col  df.columns:
      exprs.extend([
        ((pl.col(col) - pl.col()) / (pl.col() + )).alias(),
        ((pl.col(col) - pl.col()) / (pl.col() - pl.col() + )).alias(),
        (pl.col(col) / (pl.col() + )).alias(),
      ])

  df = df.with_columns(exprs)
   df

df = add_stats_feats(df, )
df = add_stats_feats(df, )
Ensembel
During the competition, I trained 41 models and tuned their weights with Optuna, but none beat a simple ensemble of my top five single models. For simplicity, I report that 5‑model ensemble result (late submission) here: all models use the same features but differ by learner and objective. XGBoost consistently performed best, with LightGBM adding useful diversity; CatBoost underperformed. The ensemble includes five models—XGBoost with rank:ndcg, rank:map, and rank:pairwise objectives, an XGBoost binary classifier, and LightGBM with LambdaRank (LightGBM’s xendcg performed much worse). The best setup uses equal weights across the five models.
model
valid
LB
PB
5 top single models
0.5101
0.54053
0.54725
Further improments
I intentionally avoided using any stat features derived from labels or selected outcomes, as including them could lead the model to "see" future labels during training, resulting in overfitting and poor generalization on unseen data.
However, one promising workaround is to construct
windowed features
—that is, using only historical statistics available prior to each instance's timestamp. This technique is effectively demonstrated in
[Mikhail Golubchik’s XGBoost notebook](https://www.kaggle.com/code/mikhailgolubchik/sm-xgboost-single)
. Thanks to
[@mikhailgolubchik](https://www.kaggle.com/mikhailgolubchik)
for the inspiration!
I haven’t implemented this yet, mainly because it’s time-consuming and I was concerned about a potential mismatch between training and test data: for training, you can use stats up to the instance time, but for test data, you're limited to stats from earlier days only. Still, this approach is definitely worth exploring further.
I merged the code from
[@mikhailgolubchik](https://www.kaggle.com/mikhailgolubchik)
's notebook and changed the feats to be used(source_cols) a bit , notice you could investigate more feats.
FLAGS.history_avg:
  test = util.get_test(df)

  df = util.get_nontest(df)
  train = util.get_train(df)

    FLAGS.online:
    valid = util.get_valid(df)
    test = pl.concat([valid, test], =)

  source_cols = [
      ,
      ,
      ,
      ,
      ,
      ,
      ,
      ,
      ,
      ,
      ,
      ,
  ]

  train, df_stats_pr = make_history_avg(train,
                                      =source_cols,
                                      =,
                                      =)
  train, df_stats_co = make_history_avg(train,
                                      =source_cols,
                                      =,
                                      =)

  test = test.join(df_stats_pr, =, =)
  test = test.join(df_stats_co, =, =)

  df = gz.align_and_concat([train, test])
model
valid
LB
PB
add selected/pos history avg feats
0.5120
0.54889
0.54942
4 history avg feats xgb ensemble
0.55201
0.55291
Well such great improvment, PB up about 0.008, so personally I think we might have over 0.6 score on PB but > 0.7 might be a bit too hard but not mission impossible:) For this dataset about 50% profileId in test exists in the train dataset, I did not investiage for each profileId how many ranker_ids exits, if there are many then may be we can also use more user history book info.  Another finding is with model perform better we do not need rerank anymore.
Opensource the solution
[https://www.kaggle.com/code/goldenlock/aeroclub-recsys-2025-1st-solution](https://www.kaggle.com/code/goldenlock/aeroclub-recsys-2025-1st-solution)
Notice aeroclub-recsys-2025-model2 has the 4 xgb models which ensemble with PB 55291 and the submission.parquet.
Please refer to the notebook and its README.