# Appreciate the host, and sharing my approaches on the competition

First of all, I deeply appreciate the host to give us such great opportunity to enjoy exploring data science world particularly in terms of feature variable engineerings.
Frankly speaking, it's currently become rare to see such raw data like….mixed numbers and strings, haha.
However, as well as improving data-science skills, it should be a good practice to improve Python coding skills that make business operations more effective.
Unfortunately I was not able to reach the top score, but I enjoyed some approaches on the competition.
With my gratitude, I would like to share about my approaches as bellow.
My main strategies
- 1st step : select best model
- 2nd step : generate various feature variables
- 3rd step : make ensemble the above various feature variable engineerings with the best model
As you know, basically we should apply "various different models" on ensemble, but….
Since the train-data only has around 6,000 records, it is seldom to work well with some different models on ensemble(it's just through my experiences). I decided to apply "one best model" to ensemble instead.
1. Basic feature engineerings before selecting best model
- not remove its missing values
- "name" : pick up its first word(manufacturer's name, such as Toyota)
- "mileage" : convert the str into numeric, such as 21.5kmpl into 21.5
- "max_power" : convert the str into numeric, such as 108.5bhp into 108.5
- "tongue" : convert the str into numeric, 260Nm@1500-2750rpm into 260
- apply Label-encoding(or One-Hot) to objective variables, such as fuel etc
In my opinion, there are a lot of Kagglers who remove missing values immediately, but the missing values also contain important information often. Hence it could be better to leave the missing values as much as possible.
Through my submissions, if the above basic feature engineerings were not carried out, it would be difficult to make the RMSE reach under 130,000.
Besides, to my big surprise….I conducted to "convert and organize units", such as [km/kg] into [kmpl] on mileage and [kgm] into [Nm] on torque, but they didn't work with my CV surprisingly. Hence I didn't convert the units on purpose…
As for "year", I assumed that it could be better to care about "TimeSeries validation" at first, but because of its data-distribution, I felt it would not be necessary.
2. Select best model
- focused on GBDT and NN
- judged with own CV(KFold-Cross validation)
Since the train-data is based on around 6,000 records and the test-data is based on around 2,000 records, I focused on GBDT models and NN.
Through my experiences, if the train-data is over 10,000 records, we should focus on GBDT models and NN, if it is less than 1,000 or 2,000 records, we should focus on classical models like Ridge model etc. Honestly speaking I considered whether or not I should try classical models as well, but I didn't have enough time to do that(only 4 days and 20 submissions altogether). I assume that ExtraTrees could also possibly penetrate the data-set, but it is my homework as being late-submission, haha.
I confirmed the difference of data distributions(train data and test data) with adversarial-validation check, and I grasped it was not necessary to care. Also, because of the data amount balance of its train-data its test-data, I realized that I would be able to judge better model based on CV(Kfold-cross-validation).
Through the above approaches, I found that XGBoost was the best. As I had anticipated, Neural Network(my favorite) was totally not effective, it was probably because there were few data-amounts let alone few variables. If there were some more effective categorical variables, CatBoost would be better…
After that, I made hyper-parameter tuning of XGBoost with Optuna, and then I slightly adjusted it by manually, cuz the data-amount is not big.(I avoided the model would be over-fitting to the train-data so much.)
Also, I checked the best fold-number with my CV, and grasped that 15 folds was the best for me.
3. Generate various feature variables
- make multi-categorical variables(connect some ones), such as "seller_type" and "owner" etc [not effective]
- make some numeric variables into categorical with Binning [not effective]
- make new numeric variables such as "1/torque" etc [effective]
- target-encoding with some categorical variables [ not effective]
- groupby with "transform('mean') [ effective ]
You know, this process is the most important to improve the model, thus I spent much time here. Also, in order to get more effective ensemble, I concentrated to generate wide-variety variables so that my model can recognize the train-data deeply.
As a matter of fact, the above concept should pretty important, because…
- The test data seems to have "outlier records".
- The unit-range of the target is relatively huge.
- I decided not to use different algorithms for ensemble.
If we were able to make use of different models such as RandomForest, LightGBM, NN etc, we would not need to generate wide-variety variables, but… I decided to use one model only on the competition.
In order to make the one model recognize "diversity" of the train-data, I tried the above generate variables process again and again.
4. Ensemble
- simple average ensemble(without caring about its weight average)
- stacking ensemble with Lasso
- stacking ensemble with Ridge
Of course, in the case that we use ensemble methods, we need to prepare some of predict datas whose metric values are roughly same, otherwise the ensemble result would become worse, haha.
In my case, 3 predicts whose RMSE are around 124,000 to 125,000 became less than 120,000 in the end.
Also, [simple average ensemble] and [stacking ensemble with Ridge] were roughly same, but the one with Lasso didn't work so much, as I had anticipated.
In my opinion, I should have looked into the reason why the unit-convert([km/kg] into [kmpl] and [kgm] into [Nm]) didn't work well, because it must have been very effective logically.
However, I am not good at physics…. no choice… haha.
Also, I plan to check other models, such as ExtraTreesRegressor, TabNet etc, when I have time as being my homework.
Hope that the above approaches of mine could be useful, and I will have an opportunity to participate in your another competition again in the future.
Cheers,
Kenta from Japan