# PUBLIC LB 24th SOLUTION by team Electrium Z

As we are spending these festivities glancing and glancing again the LB waiting for the definitive results, we decided to already share our approach in this competition in order to let some time pass sharing some knowledge and ideas and also because we may be too much shocked later (it could be either in positive or negative ¯_(ツ)_/¯ ) for writing it down later.
Before starting, a big mention to all the team that made this possible and that fought until the very end, in spite of a lot of disappointing events (cheating and private sharing) about which you already know: Pietro Marinelli, Stefano Morelli, Serigne Cisse, Amine Marzouki :-)
Here we start:
[Feature Engineering and Feature Selection]
We first processed the weather data, by integrating the missing timestamps by site, then we estimated the missing values for "air_temperature", "cloud_coverage", "dew_temperature", "precip_depth_1_hr", "sea_level_pressure", "wind_direction", "wind_speed", using interpolation.
We then extracted temporal data, both based on UTC time and on the local time we presumed for each site. Apart from month, week, day, hour, notably we extracted also the day of the week, and the day of the year, correcting for the fact that 2016 was a bisestile year (and the following two were not). Then we fused all training, weather and building metadata together.
As for as feature creation:
•   We created the delta min-max air temperature by hour at site
•   We computed the sin and cos transformation of time data, such as hour, day of the week, day of the year (it actually didn’t prove so much useful)
•   We created group stats (min, max, mean, median, variance) for weather measures by grouping by site, month, day and hour.
•   We found useful to compute the delta wind in respect of 1, 2 and 3 hours before.
•   We computed some holidays by site and modelled them using the radial basis function in order to catch their effect not simply on the holiday day.
•   We used a 7 day rolling mean for air temperature that we used in order to compare with the actual temperature
•   We computed the expected average floor size of the building
[Validation Strategy]
We adopted two learning strategies: the first requiring downsampling the meter 0 and meter 1 cases (by 50% and 35% respectively) in order for a model to learn all the meter together, the second building a single model for each meter. In the end we also tried varying the strategy, such as building a model for working days and another one for weekend and festivities or such as a model for each meter by site combination, but the first two strategies proved the most efficient. Stefano also tried some pseudo-labelling.
In all these cases we learned a model for each fold iteration in cross-validation. Initially we opted for a 5 fold stratified CV, but alter we found out that a 3 fold strategy brought better LB results. We also noticed that by increasing the number of folds we obtained better oof scores, which didn’t translate in better LB performances, anyway.
[Outlier Removal]
We observed for outlier removal, both in train and in prediction phases. In the training phase we decided to remove the building 1099 and all the readings from building 104 for meter 0 before 2016-05-20. In the prediction phase we monitored for outlying predictions, but we noticed that by ensembling anomalous values were naturally flatten out.
[Leak]
After the leak was exposed for sites 0,1,2,15 and it become public to everyone, we started using the leaked data for ensembling, that is for finding the best combination of our models. In our submissions to the LB, after finding the best combination, we also replaced our estimations for sites 0,1,2,15 with the leaked labels since we expected everyone did the same O_o and because it helped us to understand if we were improving also on the other sites, which we were sure they would have been used for the final test. We didn't use the leaked data in any other way.
[Single Models]
We built a large number of models: lightgbm (also dart), catboost, xgboost, dnn, random forest. In the end we had over 30 models, though we decided to use only about half of them, because the most effective accordingly to our ensembling strategy.
[Ensembling]
We ensembled using the leaked data as a test data. In fact, we expected our model to predict well on all sites if it was performing fine both on the leaked data and on the public leaderboard. In order to obtain a fine ensembling for this purpose, we developed a Bayesian optimization that used a linear combination of the different meter predictions. Since we noticed that certain models worked best on specific meters, we decided to give a separated weight for each meter in a prediction. 
We also put a constrain on the optimization, thus all weights were positive or at most zero. Through successive iterations we found out that there was a sweet point, that is optimizing more for the leaked data at a certain point the LB score decreased. This information helped us to focus on certain models to be blended and exclude others.
[No one expects the Spanish Inquisition, aka the “shake up”]
Well, but we did. Since long it has been clear for all of the team that optimizing for the leaked sites could have proved a good strategy for scoring high on the public LB but it may also have turned into a huge overfitting penalizing predictions for other sites (in spite of our hopes for generalization). 
Consequently, we decided to prepare a B-plan, what we called with the codename “the Big Mean”. In this alternative plan, we simply took all our best LB submissions and their oof and we averaged them, slightly overweighting the submissions whose oof scored higher on sites that were not part of the leak. Based on our experience of many other forecasting challenges we took, especially when you do have to forecast far in the future (as in the case of our competition), sometimes simple stuff like a plain mean does more the kind of magic you need.
Will all our work lead to a good final position? We don’t know, but in the end we found that we would have liked to explore more about holidays at the different sites and doing more data cleaning of possible outliers in training data, instead of relying on a blend that could smooth out wrong estimations. Anyway, as a team we tried really a lot,with everyone bringing ideas and testing solution. In any case we all learned a lot and enjoyed exploring solutions for this competition. Let’s see what happens with the new year :-) Meanwhile have great festivities!