First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure.
\n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this Kernel but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my Kernel. ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * ulmfit_pred_probs + 0.4 logit_pred_probs, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements.
So this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)
", "messages": [ { "id": "524231", "postDate": "04/28/2019 08:55:01", "content": "First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure.
\n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this Kernel but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my Kernel. ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * ulmfit_pred_probs + 0.4 logit_pred_probs, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements.
So this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)
", "rawMarkdown": "First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure. \n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this [Kernel](https://www.kaggle.com/kashnitsky/ulmfit-starter) but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my [Kernel](https://www.kaggle.com/kashnitsky/eda-logit-tf-idf-starter). ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * `ulmfit_pred_probs` + 0.4 `logit_pred_probs`, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements. \n\nSo this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)", "votes": null }, { "id": "524257", "postDate": "04/28/2019 10:33:34", "content": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)
", "rawMarkdown": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)", "votes": null }, { "id": "524265", "postDate": "04/28/2019 10:57:17", "content": "This is great!
", "rawMarkdown": "This is great!", "votes": null }, { "id": "524267", "postDate": "04/28/2019 11:02:59", "content": "fair enough for such a competition format :)
", "rawMarkdown": "fair enough for such a competition format :)", "votes": null } ], "comments": [ { "id": 524257, "author_name": "avens8", "author_url": "", "post_date": "04/28/2019 10:33:34", "content": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)
", "votes": null, "replies": [ { "id": 524267, "author_name": "kashnitsky", "author_url": "", "post_date": "04/28/2019 11:02:59", "content": "fair enough for such a competition format :)
", "votes": null, "replies": [] } ] }, { "id": 524265, "author_name": "akashpalrecha", "author_url": "", "post_date": "04/28/2019 10:57:17", "content": "This is great!
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "524231": "First of all, 1 week is definitely not enough for such a competition. The task is generally the same as at my work, so I would've been much more motivated to try a lot of approaches if the competition were held at least for 6 weeks. Including those approaches that I know working for sure. \n\nBut in this format I spent overall ~ 5 hours and my final solution was the following:\n - ULMFiT as in this [Kernel](https://www.kaggle.com/kashnitsky/ulmfit-starter) but with encoder trained with full available text data. I concatenated titles and texts from all files (train + valid + test + unlabeled) to get a 114k-length set to train a LM. A classifier finetuned on top of this LM produces 0.81 OOF F1 score (5-fold CV over train.csv + valid.csv), 0.85537 Public LB score and 0.85745 Private LB score. To build OOFs, I run 5 identical Kaggle Kernels (only fold ids differed) which is sort of having 5 GPUs - very convenient even when I have a powerful Quadro P6000 at work :)\n - Simple pipeline (Tf-Idf for titles + Tf-Idf for texts + logistic regression) was not tuned at all, taken basically again from my [Kernel](https://www.kaggle.com/kashnitsky/eda-logit-tf-idf-starter). ~ 0.77 OOF F1 score, around 0.825 LB score (didn't make submissions with it's latest version)\n - Probabilities for each class output by ULMFiT and logistic regression were then blended with weights tuned on OOF. 0.6 * `ulmfit_pred_probs` + 0.4 `logit_pred_probs`, after applying with argmax, yield 0.86452 Public and 0.86852 Private LB score. Actually, an optimal weight for ULMFiT was 0.9, according to OOF, but 0.6 gave a better LB score, and I decided not to spend time improving the validation scheme. Typically, in such situation, you'd probably switch to multiple 10-fold CVs with different seeds, and would make many more submissions to make sure that CV improvements correlate with LB improvements. \n\nSo this was a pretty simple solution, still I'm open to share ideas due to ~ the same task at work. But with Dutch language, logit + Tf-Idf is the best thing so far :)", "524257": "Thanks for sharing! For my 3rd place solution I mostly used your kernel. I also, like you, used unlabeled.csv to train encoder. Then, I ran classification stage of your kernel for 4 times(in 3 of them I also used 'title' column) and made average of predictions :)", "524265": "This is great!", "524267": "fair enough for such a competition format :)" }, "source": "meta" }