First of all, I deeply appreciate the host to give us such great opportunity to enjoy exploring data science world particularly in terms of feature variable engineerings.
\nFrankly speaking, it's currently become rare to see such raw data like….mixed numbers and strings, haha.
\nHowever, as well as improving data-science skills, it should be a good practice to improve Python coding skills that make business operations more effective.
Unfortunately I was not able to reach the top score, but I enjoyed some approaches on the competition.
\nWith my gratitude, I would like to share about my approaches as bellow.
- 1st step : select best model\n- 2nd step : generate various feature variables\n- 3rd step : make ensemble the above various feature variable engineerings with the best model\nAs you know, basically we should apply \"various different models\" on ensemble, but….
\nSince the train-data only has around 6,000 records, it is seldom to work well with some different models on ensemble(it's just through my experiences). I decided to apply \"one best model\" to ensemble instead.
- not remove its missing values\n- \"name\" : pick up its first word(manufacturer's name, such as Toyota)\n- \"mileage\" : convert the str into numeric, such as 21.5kmpl into 21.5\n- \"max_power\" : convert the str into numeric, such as 108.5bhp into 108.5\n- \"tongue\" : convert the str into numeric, 260Nm@1500-2750rpm into 260\n- apply Label-encoding(or One-Hot) to objective variables, such as fuel etc\nIn my opinion, there are a lot of Kagglers who remove missing values immediately, but the missing values also contain important information often. Hence it could be better to leave the missing values as much as possible.
\nThrough my submissions, if the above basic feature engineerings were not carried out, it would be difficult to make the RMSE reach under 130,000.
\nBesides, to my big surprise….I conducted to \"convert and organize units\", such as [km/kg] into [kmpl] on mileage and [kgm] into [Nm] on torque, but they didn't work with my CV surprisingly. Hence I didn't convert the units on purpose…
\nAs for \"year\", I assumed that it could be better to care about \"TimeSeries validation\" at first, but because of its data-distribution, I felt it would not be necessary.
\n- focused on GBDT and NN\n- judged with own CV(KFold-Cross validation)\nSince the train-data is based on around 6,000 records and the test-data is based on around 2,000 records, I focused on GBDT models and NN.
\nThrough my experiences, if the train-data is over 10,000 records, we should focus on GBDT models and NN, if it is less than 1,000 or 2,000 records, we should focus on classical models like Ridge model etc. Honestly speaking I considered whether or not I should try classical models as well, but I didn't have enough time to do that(only 4 days and 20 submissions altogether). I assume that ExtraTrees could also possibly penetrate the data-set, but it is my homework as being late-submission, haha.
I confirmed the difference of data distributions(train data and test data) with adversarial-validation check, and I grasped it was not necessary to care. Also, because of the data amount balance of its train-data its test-data, I realized that I would be able to judge better model based on CV(Kfold-cross-validation).
\nThrough the above approaches, I found that XGBoost was the best. As I had anticipated, Neural Network(my favorite) was totally not effective, it was probably because there were few data-amounts let alone few variables. If there were some more effective categorical variables, CatBoost would be better…
\nAfter that, I made hyper-parameter tuning of XGBoost with Optuna, and then I slightly adjusted it by manually, cuz the data-amount is not big.(I avoided the model would be over-fitting to the train-data so much.)
\nAlso, I checked the best fold-number with my CV, and grasped that 15 folds was the best for me.
- make multi-categorical variables(connect some ones), such as \"seller_type\" and \"owner\" etc [not effective]\n- make some numeric variables into categorical with Binning [not effective]\n- make new numeric variables such as \"1/torque\" etc [effective]\n- target-encoding with some categorical variables [ not effective]\n- groupby with \"transform('mean') [ effective ]\nYou know, this process is the most important to improve the model, thus I spent much time here. Also, in order to get more effective ensemble, I concentrated to generate wide-variety variables so that my model can recognize the train-data deeply.
\nAs a matter of fact, the above concept should pretty important, because…
\n - The test data seems to have \"outlier records\".
\n - The unit-range of the target is relatively huge.
\n - I decided not to use different algorithms for ensemble.
If we were able to make use of different models such as RandomForest, LightGBM, NN etc, we would not need to generate wide-variety variables, but… I decided to use one model only on the competition.
\nIn order to make the one model recognize \"diversity\" of the train-data, I tried the above generate variables process again and again.
- simple average ensemble(without caring about its weight average)\n- stacking ensemble with Lasso\n- stacking ensemble with Ridge\nOf course, in the case that we use ensemble methods, we need to prepare some of predict datas whose metric values are roughly same, otherwise the ensemble result would become worse, haha.
\nIn my case, 3 predicts whose RMSE are around 124,000 to 125,000 became less than 120,000 in the end.
\nAlso, [simple average ensemble] and [stacking ensemble with Ridge] were roughly same, but the one with Lasso didn't work so much, as I had anticipated.
In my opinion, I should have looked into the reason why the unit-convert([km/kg] into [kmpl] and [kgm] into [Nm]) didn't work well, because it must have been very effective logically.
\nHowever, I am not good at physics…. no choice… haha.
Also, I plan to check other models, such as ExtraTreesRegressor, TabNet etc, when I have time as being my homework.
\nHope that the above approaches of mine could be useful, and I will have an opportunity to participate in your another competition again in the future.
\nCheers,
\nKenta from Japan
Thanks! I can see that I also did some similar approaches. It was useful to see your insights.
", "votes": 1, "replies": [ { "id": 2101411, "author_name": "KentaK0928", "author_url": "", "post_date": "2023-01-15T22:19:44.153000", "content": "@martiusbleu Thanks for the compliment😄🤝 I'm pretty happy to hear that my post was useful to you too😁
\nHope that we will have an opportunity to enjoy a same competition in the future!
Thanks for the article
", "votes": 1, "replies": [ { "id": 2100398, "author_name": "KentaK0928", "author_url": "", "post_date": "2023-01-15T06:12:27.007000", "content": "@zeeagle thanks for the message😄🤝
\nYou're welcome!
\nI'm also happy if my post is useful to you as well😁
Thank you for your detail explanation and congrats your high score.
\nI couldn't overtake you again(T-T)
\nWhen I checked you last night, my score was higher than that of you, so I thought I would win.
I was busy with my job this week, so I was not able to work hard here.
\nIf I had noticed that you also took part here more early, I would have tried more seriously(T-T)
By the way, is it effective to use ensemble with only one model?!
", "votes": 1, "replies": [ { "id": 2100085, "author_name": "KentaK0928", "author_url": "", "post_date": "2023-01-14T23:33:03.420000", "content": "@hiroh03 thanks for the compliment😄
\nI usually check the competitions which you join, because you always participate in an interesting competition, haha.
\nI conducted to make ensemble on the last day, if the ensemble was not effective, I would not pass you. I mean, it was just lucky.
As for your question(ensemble with only one model work?!), actually yes, but it really depends on the data.
\nAs I introduced in the above post, the target unit range is huge, and if we prepare \"different feature variables\", even same model will recognize \"different features\".
\nIt could be better that you will also try make ensemble with several of your high score's predicts.
Hope that we will be able to enjoy data again on another competition.
", "votes": 2, "replies": [] } ] } ], "comment_count": 6, "messages": [ { "id": 2098974, "postDate": "2023-01-14T02:01:28.507Z", "content": "First of all, I deeply appreciate the host to give us such great opportunity to enjoy exploring data science world particularly in terms of feature variable engineerings.
\nFrankly speaking, it's currently become rare to see such raw data like….mixed numbers and strings, haha.
\nHowever, as well as improving data-science skills, it should be a good practice to improve Python coding skills that make business operations more effective.
Unfortunately I was not able to reach the top score, but I enjoyed some approaches on the competition.
\nWith my gratitude, I would like to share about my approaches as bellow.
- 1st step : select best model\n- 2nd step : generate various feature variables\n- 3rd step : make ensemble the above various feature variable engineerings with the best model\nAs you know, basically we should apply \"various different models\" on ensemble, but….
\nSince the train-data only has around 6,000 records, it is seldom to work well with some different models on ensemble(it's just through my experiences). I decided to apply \"one best model\" to ensemble instead.
- not remove its missing values\n- \"name\" : pick up its first word(manufacturer's name, such as Toyota)\n- \"mileage\" : convert the str into numeric, such as 21.5kmpl into 21.5\n- \"max_power\" : convert the str into numeric, such as 108.5bhp into 108.5\n- \"tongue\" : convert the str into numeric, 260Nm@1500-2750rpm into 260\n- apply Label-encoding(or One-Hot) to objective variables, such as fuel etc\nIn my opinion, there are a lot of Kagglers who remove missing values immediately, but the missing values also contain important information often. Hence it could be better to leave the missing values as much as possible.
\nThrough my submissions, if the above basic feature engineerings were not carried out, it would be difficult to make the RMSE reach under 130,000.
\nBesides, to my big surprise….I conducted to \"convert and organize units\", such as [km/kg] into [kmpl] on mileage and [kgm] into [Nm] on torque, but they didn't work with my CV surprisingly. Hence I didn't convert the units on purpose…
\nAs for \"year\", I assumed that it could be better to care about \"TimeSeries validation\" at first, but because of its data-distribution, I felt it would not be necessary.
\n- focused on GBDT and NN\n- judged with own CV(KFold-Cross validation)\nSince the train-data is based on around 6,000 records and the test-data is based on around 2,000 records, I focused on GBDT models and NN.
\nThrough my experiences, if the train-data is over 10,000 records, we should focus on GBDT models and NN, if it is less than 1,000 or 2,000 records, we should focus on classical models like Ridge model etc. Honestly speaking I considered whether or not I should try classical models as well, but I didn't have enough time to do that(only 4 days and 20 submissions altogether). I assume that ExtraTrees could also possibly penetrate the data-set, but it is my homework as being late-submission, haha.
I confirmed the difference of data distributions(train data and test data) with adversarial-validation check, and I grasped it was not necessary to care. Also, because of the data amount balance of its train-data its test-data, I realized that I would be able to judge better model based on CV(Kfold-cross-validation).
\nThrough the above approaches, I found that XGBoost was the best. As I had anticipated, Neural Network(my favorite) was totally not effective, it was probably because there were few data-amounts let alone few variables. If there were some more effective categorical variables, CatBoost would be better…
\nAfter that, I made hyper-parameter tuning of XGBoost with Optuna, and then I slightly adjusted it by manually, cuz the data-amount is not big.(I avoided the model would be over-fitting to the train-data so much.)
\nAlso, I checked the best fold-number with my CV, and grasped that 15 folds was the best for me.
- make multi-categorical variables(connect some ones), such as \"seller_type\" and \"owner\" etc [not effective]\n- make some numeric variables into categorical with Binning [not effective]\n- make new numeric variables such as \"1/torque\" etc [effective]\n- target-encoding with some categorical variables [ not effective]\n- groupby with \"transform('mean') [ effective ]\nYou know, this process is the most important to improve the model, thus I spent much time here. Also, in order to get more effective ensemble, I concentrated to generate wide-variety variables so that my model can recognize the train-data deeply.
\nAs a matter of fact, the above concept should pretty important, because…
\n - The test data seems to have \"outlier records\".
\n - The unit-range of the target is relatively huge.
\n - I decided not to use different algorithms for ensemble.
If we were able to make use of different models such as RandomForest, LightGBM, NN etc, we would not need to generate wide-variety variables, but… I decided to use one model only on the competition.
\nIn order to make the one model recognize \"diversity\" of the train-data, I tried the above generate variables process again and again.
- simple average ensemble(without caring about its weight average)\n- stacking ensemble with Lasso\n- stacking ensemble with Ridge\nOf course, in the case that we use ensemble methods, we need to prepare some of predict datas whose metric values are roughly same, otherwise the ensemble result would become worse, haha.
\nIn my case, 3 predicts whose RMSE are around 124,000 to 125,000 became less than 120,000 in the end.
\nAlso, [simple average ensemble] and [stacking ensemble with Ridge] were roughly same, but the one with Lasso didn't work so much, as I had anticipated.
In my opinion, I should have looked into the reason why the unit-convert([km/kg] into [kmpl] and [kgm] into [Nm]) didn't work well, because it must have been very effective logically.
\nHowever, I am not good at physics…. no choice… haha.
Also, I plan to check other models, such as ExtraTreesRegressor, TabNet etc, when I have time as being my homework.
\nHope that the above approaches of mine could be useful, and I will have an opportunity to participate in your another competition again in the future.
\nCheers,
\nKenta from Japan
Thanks! I can see that I also did some similar approaches. It was useful to see your insights.
", "rawMarkdown": "Thanks! I can see that I also did some similar approaches. It was useful to see your insights. ", "votes": 1, "replies": [ { "id": 2101411, "postDate": "2023-01-15T22:19:44.153Z", "content": "@martiusbleu Thanks for the compliment😄🤝 I'm pretty happy to hear that my post was useful to you too😁
\nHope that we will have an opportunity to enjoy a same competition in the future!
Thanks for the article
", "rawMarkdown": "Thanks for the article", "votes": 1, "replies": [ { "id": 2100398, "postDate": "2023-01-15T06:12:27.007Z", "content": "@zeeagle thanks for the message😄🤝
\nYou're welcome!
\nI'm also happy if my post is useful to you as well😁
Thank you for your detail explanation and congrats your high score.
\nI couldn't overtake you again(T-T)
\nWhen I checked you last night, my score was higher than that of you, so I thought I would win.
I was busy with my job this week, so I was not able to work hard here.
\nIf I had noticed that you also took part here more early, I would have tried more seriously(T-T)
By the way, is it effective to use ensemble with only one model?!
", "rawMarkdown": "Thank you for your detail explanation and congrats your high score.\nI couldn't overtake you again(T-T)\nWhen I checked you last night, my score was higher than that of you, so I thought I would win. \n\nI was busy with my job this week, so I was not able to work hard here.\nIf I had noticed that you also took part here more early, I would have tried more seriously(T-T)\n\n\nBy the way, is it effective to use ensemble with only one model?!", "votes": 1, "replies": [ { "id": 2100085, "postDate": "2023-01-14T23:33:03.420Z", "content": "@hiroh03 thanks for the compliment😄
\nI usually check the competitions which you join, because you always participate in an interesting competition, haha.
\nI conducted to make ensemble on the last day, if the ensemble was not effective, I would not pass you. I mean, it was just lucky.
As for your question(ensemble with only one model work?!), actually yes, but it really depends on the data.
\nAs I introduced in the above post, the target unit range is huge, and if we prepare \"different feature variables\", even same model will recognize \"different features\".
\nIt could be better that you will also try make ensemble with several of your high score's predicts.
Hope that we will be able to enjoy data again on another competition.
", "rawMarkdown": "@hiroh03 thanks for the compliment😄\n\nI usually check the competitions which you join, because you always participate in an interesting competition, haha.\nI conducted to make ensemble on the last day, if the ensemble was not effective, I would not pass you. I mean, it was just lucky.\n\nAs for your question(ensemble with only one model work?!), actually yes, but it really depends on the data.\nAs I introduced in the above post, the target unit range is huge, and if we prepare \"different feature variables\", even same model will recognize \"different features\".\nIt could be better that you will also try make ensemble with several of your high score's predicts.\n\nHope that we will be able to enjoy data again on another competition.", "votes": 2 } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-20T03:12:57Z", "competition_id": 40603, "competition_slug": "act61-mfe-prediction-competition", "team_id": 9709803, "solution_url": "/competitions/act61-mfe-prediction-competition/writeups/kentak0928-appreciate-the-host-and-sharing-my-appr" }