# 2nd Place Solution Writeup

2nd Place Solution: Pairwise Features + LLM Signals + LightGBM
Hi everyone,
here is a short overview of my approach. The core idea was to treat the task as a
tabular classification problem on pairwise features
, enhanced with
guidance from a large language model
. A final
LightGBM
model then learned how to combine all signals.
GitHub repository:
[https://github.com/emilpartow/kaggle-esa-impostor-hunt](https://github.com/emilpartow/kaggle-esa-impostor-hunt)
Key Takeaway
A key insight for me is that in a task like this - where the manipulated output was itself generated by a large model - zero-shot judgments from strong LLMs are not just helpful but naturally well-suited to the problem. LLMs implicitly encode priors about fluency, coherence shifts, and subtle “model fingerprints” that are extremely hard to capture through classical features alone.
While I still believe these signals should be distilled or grounded through a supervised classifier (as I did with LightGBM), I think that future detection or “impostor” problems involving LLM-generated text will benefit from leveraging zero-shot reasoning from strong LLMs.
Overall Pipeline
Pairwise raw texts
     ⬇
Classical single-text + pairwise features
     ⬇
(converted to numeric      ⬇
LightGBM classifier ( vs )
     ⬇
real_text_id ∈ {, }
Classical Feature Engineering
For each text (A and B), I extracted a set of lightweight, explainable features such as:
Word / character / sentence counts
Flesch reading ease
Average word length
Number of digits or uppercase words
Longest sentence length
Repeated token heuristics
Punctuation endings
Then I created pairwise interaction features:
Differences between A and B (e.g.
flesch_A - flesch_B
)
TF-IDF cosine similarity between the two texts
Token-level Jaccard similarity
These features already captured relevant stylistic differences between “real” and “impostor” texts.
LLM Judgments (Used as Features)
I used Grok to obtain zero-shot pairwise judgments, as it was freely accessible at the time of the competition. In principle, any high-capacity LLM (e.g., within the top-performing model range) should provide similar prior signals. For each text pair, I sent both texts to the model and asked which one was more likely to be the manipulated / “fake” one. The API returned a structured JSON response containing the predicted fake side ("A" or "B") along with continuous scores, e.g.:
{
  "fake": "A" or "B",
  "scores": {"A": float, "B": float}
}
To reduce randomness I performed
An initial pass
A second re-evaluation pass (“double precision”)
If results disagreed, a third tie-break pass
These outputs (e.g.
fake_label_1
,
A_score_1
,
fake_label_2
, …) were treated as
numeric/tabular features only
.
The classifier then learned how to trust (or ignore) these signals in combination with the classical features.
Model Training (LightGBM)
I converted the task into:
Target = 1 if B is fake, else 0
Then I trained a LightGBM classifier using both classical features and (when available) the external scores:
model = lgb.LGBMClassifier(
    n_estimators=,
    learning_rate=,
    num_leaves=,
    subsample=,
    colsample_bytree=,
    reg_alpha=,
    reg_lambda=,
    random_state=
)
model.fit(X_train, y_train, eval_set=[(X_val, y_val)], eval_metric=)
The validation performance was consistent with the leaderboard, which confirmed that the model was not overly dependent on noise or specific LLM outputs.
Inference & Submission
One empirical observation during experimentation was that using only a single LLM pass in combination with LightGBM produced decent performance, but the “double precision” strategy (with an additional tie-break when needed) consistently improved my private score from around 0.9 to roughly 0.94. This confirmed that even though the LLM signals were noisy, there was valuable signal to be extracted when treated as features rather than as a final decision.
What Worked Well
Pairwise classical features (length gaps, sentence structure, readability metrics, similarity scores, …) captured subtle stylistic shifts between real vs. impostor model outputs
Zero-shot judgments from Grok (queried via API) provided high-level prior intuition, especially because the manipulated texts themselves originated from a large language model
LightGBM was effective at learning how much to trust (or ignore) the external scores and blending them with classical features in a noise-tolerant way
Treating the task as a tabular classification problem made iteration fast, explainable and robust, rather than relying solely on end-to-end fine-tuning
What Didn’t Help Much
LLM only solutions produced reasonable baselines but did not capture sufficiently fine-grained impostor cues to compete for top ranks
Heavier architectures significantly increased training and inference time without delivering consistent improvements over a well-engineered feature-based + boosted tree model
Big thanks to the organizers for a creative challenge and to the community for all the ideas throughout the competition.
Congrats to the winners and everyone who took part!