First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure.
\n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this Kernel but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my Kernel. ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * ulmfit_pred_probs + 0.4 logit_pred_probs, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements.
So this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)
", "comments": [ { "id": 524257, "author_name": "Pavel Tsai", "author_url": "", "post_date": "2019-04-28T10:33:34.023000", "content": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)
", "votes": 1, "replies": [ { "id": 524267, "author_name": "Yury Kashnitsky", "author_url": "", "post_date": "2019-04-28T11:02:59.760000", "content": "fair enough for such a competition format :)
", "votes": 1, "replies": [] } ] }, { "id": 524265, "author_name": "Akash Palrecha", "author_url": "", "post_date": "2019-04-28T10:57:17.857000", "content": "This is great!
", "votes": 0, "replies": [] } ], "comment_count": 3, "messages": [ { "id": 524231, "postDate": "2019-04-28T08:55:01.630Z", "content": "First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure.
\n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this Kernel but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my Kernel. ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * ulmfit_pred_probs + 0.4 logit_pred_probs, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements.
So this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)
", "rawMarkdown": "First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure. \n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this [Kernel](https://www.kaggle.com/kashnitsky/ulmfit-starter) but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my [Kernel](https://www.kaggle.com/kashnitsky/eda-logit-tf-idf-starter). ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * `ulmfit_pred_probs` + 0.4 `logit_pred_probs`, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements. \n\nSo this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)\n\n\n ", "votes": 5 }, { "id": 524257, "postDate": "2019-04-28T10:33:34.023Z", "content": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)
", "rawMarkdown": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)", "votes": 1, "replies": [ { "id": 524267, "postDate": "2019-04-28T11:02:59.760Z", "content": "fair enough for such a competition format :)
", "rawMarkdown": "fair enough for such a competition format :)", "votes": 1 } ] }, { "id": 524265, "postDate": "2019-04-28T10:57:17.857Z", "content": "This is great!
", "rawMarkdown": "This is great!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T11:00:04Z", "competition_id": 13691, "competition_slug": "dlinnlp-spring-2019-clf", "team_id": 3074382, "solution_url": "/competitions/dlinnlp-spring-2019-clf/writeups/yorko-2nd-place-solution-brief-overview" }