{ "id": 483099, "title": "1st Place Solution", "content": "

Hi everyone,
\nThanks for KAUST Academy for this fun competition. I will try to show my work here.

\n

Explore the data:

\n

I started by exploring the data first and got many surprises :) Firstly, there are some duplicated samples in training data. Deleting them reduce training size from 25k to 20k. But what is really surprising is that, there are MANY duplicates between train and test. What does that mean? Pretty simple, just look into the titles/authors in the training data, and take their targets and put them into the testing data same titles :D If a poem got the same title as a one you know, or you found the same author in train and test, is there any possibilities that this author would live in 2 different categories? Answer is NO.
\n
\nTaking the values of these duplicates and just map them into the test set without any model can already get accuracy 80%! In other words, 80% of the samples in the test data are somehow already in the training data :) So you can say that 80% of test data is already leaked in the train.
\nWhat your model has been doing is actually this exact thing. But if you discovered the pattern and did that manually you would reach > 80% with pretty simple models.
\nThe real challenge started from here. It is to predict categories for authors who don’t exist in train.

\n

Construct your Cross-validation (CV):

\n

Given that duplicated authors can all covered using the above mapping, then what we should focus on in our models are the non-duplicates. So, the authors in train data, should be different from validation data. Also, we should try to have all the categories in both training and validation folds. Therefore, I used StratifiedGroupKFold, stratifying on the target to preserve the classes ratios, and grouping on the authors to assure train and validation have different authors.
\nUsing this cv I found the accuracy in the validation set to be only 35% lol. So if there weren’t any leakage, we wouldn’t have such high scores in the leaderboard. Submitting without using the duplicates targets give accuracy 35% in lb, adding the duplicates gives 81%.
\nI think the host splitted the data using StratifiedKFold with 6 folds (because the target ratios in the train are same), then took the last fold, dropped its duplicates, shuffled it and considered this as a test set. He might didn’t take into account that there are duplicates between train and test and that’s why we are getting pretty high scores in lb.

\n

Modeling:

\n

Using ML algorithms gave me cv around 35%. I added a lot of feature engineering until I reached cv 45%. Submitting also gave 45%. Adding the duplicates gave 81%. I gave it up at the end and went into deep learning.
\nI finetuned a deberta variant that is trained on classic Arabic text. (Link: https://huggingface.co/CAMeL-Lab/bert-base-arabic-camelbert-ca ).
\nThis model gave cv around 80%! And also leaderboard 82% without adding duplicates! I tried several tricks like reinitializing the last layer, freezing some layers, custom pooling layer, custom losses,…etc. Most of them were helpful. I even took those test samples that are duplicates between train and test, added their labels using the what I found earlier, and added them to my training data so I get more data for my model. But they weren’t helpful for my cv actually so I dropped them.
\nI concatenated Title, author, and poem into one text.

\n

train_df['Poem'] = \"العنوان \" + train_df['Title'] + \" والكاتب هو \" + train_df['Author'] + \" :والنص هو \" + train_df['Poem']

\n

I used only the first 256 token of this text for training. It was the best for my cv.
\nModel takes 2 hours to finish 5 folds on Nvidia A4000 16GB. I use Pytorch and Pytorch Lightning for my training and inference.
\nAs for my submission choices, actually I got more than 15 submission with score 0.8432. Each one of them got pretty different cv score. So I had to choose between them for the private. I chose one submission that got cv 0.79 but was pretty stable between the folds (all the folds got score similar to 0.79). Then for my 2nd submission I chose an average between 2 subs that got cvs 0.84 and 0.79. The 0.84 got high score in one fold and much worse on the others. So for diversity, I averaged these two, and added the duplicates then submitted. Both got same score in private 0.8278.

\n

Further Exploring:

\n

I tried to keep it simple, but yeah if you explored the data a bit more you will find some more leakage :) In the first part, we assume if the title exist in train and test, then definitely the test target is the same as train. This is obvious right? Yea, the same goes for the authors. It is impossible for an author to live in two ages right? Well, no. There are 17 authors that got TWO different categories in the train data. I excluded them at the beginning and decided to use only the authors who got one category only. But when I returned to them at the end, I found this:
\n
\nDo you notice something? Yes, if the Title got diacritics (تشكيل) then this title is from a category different than that with no diacritics :D. Actaually for the last 17 authors that got more than one category, you can classify them almost perfectly without a model using the diacritics :). Resulting on classifying 3800 samples from the test perfectly, and only leave the rest 1000 to my Deberta which worked well. I found this to give me even better private lb 0.8282 but didn't choose it at the end.
\nAnyway, I think this data is a bit flawed but happy I managed to refresh my NLP skills using it. I think our models didn't really learned to differentiate between the categories by understanding the text, but instead by discovering such flaws that ease classifying. Didn't you notice that when you remove the diacritics (تشكيل) from data you get worse score? I hope you understand now why :D
\nThanks for reading!

\n

My Training Code:
\nhttps://www.kaggle.com/code/mohammad2012191/poem-training-notebook

\n

My Inference Code:
\nhttps://www.kaggle.com/code/mohammad2012191/poem-inference-notebook/

", "comments": [ { "id": 2693032, "author_name": "Salha Nasser", "author_url": "", "post_date": "2024-03-12T07:09:55.317000", "content": "

I really love your thinking and how you find the duplication between training and testing. I also , noticed the diacritics problem when I remove it I got about 53% a and when I use it I got 70%. I enjoy reading your post thank you

", "votes": 1, "replies": [] }, { "id": 2693017, "author_name": "Salha Nasser", "author_url": "", "post_date": "2024-03-12T06:51:50.497000", "content": "

about the author , I actually try it since it more distinct but it seems it causes data leakage when I use it with the poems as input I get 99% but when removing it the accurecy drops into 70%

", "votes": 0, "replies": [ { "id": 2693343, "author_name": "Mohamed Eltayeb", "author_url": "", "post_date": "2024-03-12T11:17:20.220000", "content": "

One should focus on building a reliable validation set at the beginning. I guess the same would happen to me if I used random split, but given I used grouping split, I guess that's why my validation was reliable and giving real scores. Knowing the correct split needs EDA at the beginning before going into modeling.

", "votes": 0, "replies": [] } ] } ], "comment_count": 3, "messages": [ { "id": 2690968, "postDate": "2024-03-10T22:41:56.193Z", "content": "

Hi everyone,
\nThanks for KAUST Academy for this fun competition. I will try to show my work here.

\n

Explore the data:

\n

I started by exploring the data first and got many surprises :) Firstly, there are some duplicated samples in training data. Deleting them reduce training size from 25k to 20k. But what is really surprising is that, there are MANY duplicates between train and test. What does that mean? Pretty simple, just look into the titles/authors in the training data, and take their targets and put them into the testing data same titles :D If a poem got the same title as a one you know, or you found the same author in train and test, is there any possibilities that this author would live in 2 different categories? Answer is NO.
\n
\nTaking the values of these duplicates and just map them into the test set without any model can already get accuracy 80%! In other words, 80% of the samples in the test data are somehow already in the training data :) So you can say that 80% of test data is already leaked in the train.
\nWhat your model has been doing is actually this exact thing. But if you discovered the pattern and did that manually you would reach > 80% with pretty simple models.
\nThe real challenge started from here. It is to predict categories for authors who don’t exist in train.

\n

Construct your Cross-validation (CV):

\n

Given that duplicated authors can all covered using the above mapping, then what we should focus on in our models are the non-duplicates. So, the authors in train data, should be different from validation data. Also, we should try to have all the categories in both training and validation folds. Therefore, I used StratifiedGroupKFold, stratifying on the target to preserve the classes ratios, and grouping on the authors to assure train and validation have different authors.
\nUsing this cv I found the accuracy in the validation set to be only 35% lol. So if there weren’t any leakage, we wouldn’t have such high scores in the leaderboard. Submitting without using the duplicates targets give accuracy 35% in lb, adding the duplicates gives 81%.
\nI think the host splitted the data using StratifiedKFold with 6 folds (because the target ratios in the train are same), then took the last fold, dropped its duplicates, shuffled it and considered this as a test set. He might didn’t take into account that there are duplicates between train and test and that’s why we are getting pretty high scores in lb.

\n

Modeling:

\n

Using ML algorithms gave me cv around 35%. I added a lot of feature engineering until I reached cv 45%. Submitting also gave 45%. Adding the duplicates gave 81%. I gave it up at the end and went into deep learning.
\nI finetuned a deberta variant that is trained on classic Arabic text. (Link: https://huggingface.co/CAMeL-Lab/bert-base-arabic-camelbert-ca ).
\nThis model gave cv around 80%! And also leaderboard 82% without adding duplicates! I tried several tricks like reinitializing the last layer, freezing some layers, custom pooling layer, custom losses,…etc. Most of them were helpful. I even took those test samples that are duplicates between train and test, added their labels using the what I found earlier, and added them to my training data so I get more data for my model. But they weren’t helpful for my cv actually so I dropped them.
\nI concatenated Title, author, and poem into one text.

\n

train_df['Poem'] = \"العنوان \" + train_df['Title'] + \" والكاتب هو \" + train_df['Author'] + \" :والنص هو \" + train_df['Poem']

\n

I used only the first 256 token of this text for training. It was the best for my cv.
\nModel takes 2 hours to finish 5 folds on Nvidia A4000 16GB. I use Pytorch and Pytorch Lightning for my training and inference.
\nAs for my submission choices, actually I got more than 15 submission with score 0.8432. Each one of them got pretty different cv score. So I had to choose between them for the private. I chose one submission that got cv 0.79 but was pretty stable between the folds (all the folds got score similar to 0.79). Then for my 2nd submission I chose an average between 2 subs that got cvs 0.84 and 0.79. The 0.84 got high score in one fold and much worse on the others. So for diversity, I averaged these two, and added the duplicates then submitted. Both got same score in private 0.8278.

\n

Further Exploring:

\n

I tried to keep it simple, but yeah if you explored the data a bit more you will find some more leakage :) In the first part, we assume if the title exist in train and test, then definitely the test target is the same as train. This is obvious right? Yea, the same goes for the authors. It is impossible for an author to live in two ages right? Well, no. There are 17 authors that got TWO different categories in the train data. I excluded them at the beginning and decided to use only the authors who got one category only. But when I returned to them at the end, I found this:
\n
\nDo you notice something? Yes, if the Title got diacritics (تشكيل) then this title is from a category different than that with no diacritics :D. Actaually for the last 17 authors that got more than one category, you can classify them almost perfectly without a model using the diacritics :). Resulting on classifying 3800 samples from the test perfectly, and only leave the rest 1000 to my Deberta which worked well. I found this to give me even better private lb 0.8282 but didn't choose it at the end.
\nAnyway, I think this data is a bit flawed but happy I managed to refresh my NLP skills using it. I think our models didn't really learned to differentiate between the categories by understanding the text, but instead by discovering such flaws that ease classifying. Didn't you notice that when you remove the diacritics (تشكيل) from data you get worse score? I hope you understand now why :D
\nThanks for reading!

\n

My Training Code:
\nhttps://www.kaggle.com/code/mohammad2012191/poem-training-notebook

\n

My Inference Code:
\nhttps://www.kaggle.com/code/mohammad2012191/poem-inference-notebook/

", "rawMarkdown": "Hi everyone,\nThanks for KAUST Academy for this fun competition. I will try to show my work here.\n# Explore the data:\nI started by exploring the data first and got many surprises :) Firstly, there are some duplicated samples in training data. Deleting them reduce training size from 25k to 20k. But what is really surprising is that, there are MANY duplicates between train and test. What does that mean? Pretty simple, just look into the titles/authors in the training data, and take their targets and put them into the testing data same titles :D If a poem got the same title as a one you know, or you found the same author in train and test, is there any possibilities that this author would live in 2 different categories? Answer is NO.\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7311816%2F3463e90023039e592dc9009ec2b92847%2FUntitl22ed.png?generation=1710107208380680&alt=media)\nTaking the values of these duplicates and just map them into the test set without any model can already get accuracy 80%! In other words, 80% of the samples in the test data are somehow already in the training data :) So you can say that 80% of test data is already leaked in the train. \nWhat your model has been doing is actually this exact thing. But if you discovered the pattern and did that manually you would reach > 80% with pretty simple models.\nThe real challenge started from here. It is to predict categories for authors who don’t exist in train.\n# Construct your Cross-validation (CV):\nGiven that duplicated authors can all covered using the above mapping, then what we should focus on in our models are the non-duplicates. So, the authors in train data, should be different from validation data. Also, we should try to have all the categories in both training and validation folds. Therefore, I used StratifiedGroupKFold, stratifying on the target to preserve the classes ratios, and grouping on the authors to assure train and validation have different authors.\nUsing this cv I found the accuracy in the validation set to be only 35% lol. So if there weren’t any leakage, we wouldn’t have such high scores in the leaderboard. Submitting without using the duplicates targets give accuracy 35% in lb, adding the duplicates gives 81%. \nI think the host splitted the data using StratifiedKFold with 6 folds (because the target ratios in the train are same), then took the last fold, dropped its duplicates, shuffled it and considered this as a test set. He might didn’t take into account that there are duplicates between train and test and that’s why we are getting pretty high scores in lb.\n# Modeling:\nUsing ML algorithms gave me cv around 35%. I added a lot of feature engineering until I reached cv 45%. Submitting also gave 45%. Adding the duplicates gave 81%. I gave it up at the end and went into deep learning.\nI finetuned a deberta variant that is trained on classic Arabic text. (Link: https://huggingface.co/CAMeL-Lab/bert-base-arabic-camelbert-ca ).\nThis model gave cv around 80%! And also leaderboard 82% without adding duplicates! I tried several tricks like reinitializing the last layer, freezing some layers, custom pooling layer, custom losses,…etc. Most of them were helpful. I even took those test samples that are duplicates between train and test, added their labels using the what I found earlier, and added them to my training data so I get more data for my model. But they weren’t helpful for my cv actually so I dropped them.\nI concatenated Title, author, and poem into one text. \n\n`train_df['Poem'] = \"العنوان \" + train_df['Title'] + \" والكاتب هو \" + train_df['Author'] + \" :والنص هو \" + train_df['Poem']`\n\nI used only the first 256 token of this text for training. It was the best for my cv. \nModel takes 2 hours to finish 5 folds on Nvidia A4000 16GB. I use Pytorch and Pytorch Lightning for my training and inference. \nAs for my submission choices, actually I got more than 15 submission with score 0.8432. Each one of them got pretty different cv score. So I had to choose between them for the private. I chose one submission that got cv 0.79 but was pretty stable between the folds (all the folds got score similar to 0.79). Then for my 2nd submission I chose an average between 2 subs that got cvs 0.84 and 0.79. The 0.84 got high score in one fold and much worse on the others. So for diversity, I averaged these two, and added the duplicates then submitted. Both got same score in private 0.8278.\n# Further Exploring:\nI tried to keep it simple, but yeah if you explored the data a bit more you will find some more leakage :) In the first part, we assume if the title exist in train and test, then definitely the test target is the same as train. This is obvious right? Yea, the same goes for the authors. It is impossible for an author to live in two ages right? Well, no. There are 17 authors that got TWO different categories in the train data. I excluded them at the beginning and decided to use only the authors who got one category only. But when I returned to them at the end, I found this:\n![](https://www.googleapis.com/download/storage/v1/b/kaggle-forum-message-attachments/o/inbox%2F7311816%2Fa4607e9473391953d0d4788b6da4b04f%2FUntitl22ed.png?generation=1710106771605817&alt=media)\nDo you notice something? Yes, if the Title got diacritics (تشكيل) then this title is from a category different than that with no diacritics :D. Actaually for the last 17 authors that got more than one category, you can classify them almost perfectly without a model using the diacritics :). Resulting on classifying 3800 samples from the test perfectly, and only leave the rest 1000 to my Deberta which worked well. I found this to give me even better private lb 0.8282 but didn't choose it at the end. \nAnyway, I think this data is a bit flawed but happy I managed to refresh my NLP skills using it. I think our models didn't really learned to differentiate between the categories by understanding the text, but instead by discovering such flaws that ease classifying. Didn't you notice that when you remove the diacritics (تشكيل) from data you get worse score? I hope you understand now why :D\nThanks for reading!\n\nMy Training Code:\nhttps://www.kaggle.com/code/mohammad2012191/poem-training-notebook\n\nMy Inference Code:\nhttps://www.kaggle.com/code/mohammad2012191/poem-inference-notebook/", "votes": 6 }, { "id": 2693032, "postDate": "2024-03-12T07:09:55.317Z", "content": "

I really love your thinking and how you find the duplication between training and testing. I also , noticed the diacritics problem when I remove it I got about 53% a and when I use it I got 70%. I enjoy reading your post thank you

", "rawMarkdown": "I really love your thinking and how you find the duplication between training and testing. I also , noticed the diacritics problem when I remove it I got about 53% a and when I use it I got 70%. I enjoy reading your post thank you ", "votes": 1 }, { "id": 2693017, "postDate": "2024-03-12T06:51:50.497Z", "content": "

about the author , I actually try it since it more distinct but it seems it causes data leakage when I use it with the poems as input I get 99% but when removing it the accurecy drops into 70%

", "rawMarkdown": "about the author , I actually try it since it more distinct but it seems it causes data leakage when I use it with the poems as input I get 99% but when removing it the accurecy drops into 70%", "replies": [ { "id": 2693343, "postDate": "2024-03-12T11:17:20.220Z", "content": "

One should focus on building a reliable validation set at the beginning. I guess the same would happen to me if I used random split, but given I used grouping split, I guess that's why my validation was reliable and giving real scores. Knowing the correct split needs EDA at the beginning before going into modeling.

", "rawMarkdown": "One should focus on building a reliable validation set at the beginning. I guess the same would happen to me if I used random split, but given I used grouping split, I guess that's why my validation was reliable and giving real scores. Knowing the correct split needs EDA at the beginning before going into modeling." } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T15:11:29Z", "competition_id": 71335, "competition_slug": "arabic-poem-classification", "team_id": 11684621, "solution_url": "/competitions/arabic-poem-classification/writeups/mohamed-eltayeb-1st-place-solution" }