Thanks to organisers for hosting the competition. It was clearly difficult to manage this one due to the leak, and the scope of the problem has changed drastically after the relaunch, but I think you did your best to keep it running and make it as useful as possible. Respect.
\n\nOn my side, i was not able to spend much time on the comp after the relaunch. On Stage 1 i was also very low on the LB, but at least the validation and model building were clear.
\n\nOn stage 2 I spent most of the time figuring out how to run json parsing using dask. Using a virtual instance with a lot of memory or manually running reading in chunks in pandas was an alternative option, but I knew that dask is something that i want to learn and this was the perfect opportunity. I first tried using the pandas-internal json_normalise in dask. I was not able to make it work after several days of painful debugging (there is an issue already on dask github asking to add support of it). Then I switched to a more manual approach that was in Olivier's public kernel. This worked, but I spent more time than I planed. At the end this gave a scalable (but relatively slow) code that was parsing the dataset even in a kernel: see here. I persive this as my main achievement and main outcome for personal development :)
Next step, I started to think how to approach model building and how to establish a validation scheme. I did not have any experience with such time-series problems, there were no useful discussions on the forum (at east i did not find one) and I did not have time to dive into literature. So the best idea that I was able to come up with was the following:
\n\nLet's see in 2 month, if this makes any sense at all...
", "messages": [ { "id": "431027", "postDate": "12/01/2018 13:22:07", "content": "Thanks to organisers for hosting the competition. It was clearly difficult to manage this one due to the leak, and the scope of the problem has changed drastically after the relaunch, but I think you did your best to keep it running and make it as useful as possible. Respect.
\n\nOn my side, i was not able to spend much time on the comp after the relaunch. On Stage 1 i was also very low on the LB, but at least the validation and model building were clear.
\n\nOn stage 2 I spent most of the time figuring out how to run json parsing using dask. Using a virtual instance with a lot of memory or manually running reading in chunks in pandas was an alternative option, but I knew that dask is something that i want to learn and this was the perfect opportunity. I first tried using the pandas-internal json_normalise in dask. I was not able to make it work after several days of painful debugging (there is an issue already on dask github asking to add support of it). Then I switched to a more manual approach that was in Olivier's public kernel. This worked, but I spent more time than I planed. At the end this gave a scalable (but relatively slow) code that was parsing the dataset even in a kernel: see here. I persive this as my main achievement and main outcome for personal development :)
Next step, I started to think how to approach model building and how to establish a validation scheme. I did not have any experience with such time-series problems, there were no useful discussions on the forum (at east i did not find one) and I did not have time to dive into literature. So the best idea that I was able to come up with was the following:
\n\nLet's see in 2 month, if this makes any sense at all...
", "rawMarkdown": "Thanks to organisers for hosting the competition. It was clearly difficult to manage this one due to the leak, and the scope of the problem has changed drastically after the relaunch, but I think you did your best to keep it running and make it as useful as possible. Respect.\n\nOn my side, i was not able to spend much time on the comp after the relaunch. On Stage 1 i was also very low on the LB, but at least the validation and model building were clear.\n\nOn stage 2 I spent most of the time figuring out how to run json parsing using `dask`. Using a virtual instance with a lot of memory or manually running reading in chunks in pandas was an alternative option, but I knew that dask is something that i want to learn and this was the perfect opportunity. I first tried using the pandas-internal `json_normalise` in dask. I was not able to make it work after several days of painful debugging (there is [an issue already on dask github](https://github.com/dask/dask/issues/4170) asking to add support of it). Then I switched to a more manual approach that was in [Olivier's public kernel]( https://www.kaggle.com/ogrellier/create-extracted-json-fields-dataset). This worked, but I spent more time than I planed. At the end this gave a scalable (but relatively slow) code that was parsing the dataset even in a kernel: [see here](https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd). I persive this as my main achievement and main outcome for personal development :)\n\nNext step, I started to think how to approach model building and how to establish a validation scheme. I did not have any experience with such time-series problems, there were no useful discussions on the forum (at east i did not find one) and I did not have time to dive into literature. So the best idea that I was able to come up with was the following:\n\n- In short: Try to converge to a standard ML problem and tackle it with LGBM;\n- Chunk the train+test into periods of 6 month + 1.5 month gap + 2 month. Get features from the first period, ignore the middle gap period (as i did not have time to play with the demo account and due to large uncertainties it did not look like a good time investment), get the target from the last period (calculating log1p(sum) per user and setting target to 0 for not-returning users);\n- Build user-level feature set for the users from the 6-month period:\n + All info about the last visit (in particular, most relevant would be, was it a bounce? how much was spent? How many times did the user visit up until this last visit (=visit number)? Time since this last visit (wrt the end date of the 6-month period)? What's the country of the user?)\n + Some aggregates (sum/max/mean) over 2,3,6,12 month periods (this includes the sliced 6-month period and any prior visits) for this user: how much did he/she spent in those periods, how many bounces were there, how long did he/she spent on the site, how many purchases were made? Also ratios of those aggregates.\n- I created 3 such 6+1.5+2 slices shifted by 2 month one-wrt-another. And for each i build the feature-set as describes above plus the target\n- My idea was to use them independently (although they overlap, so I assume i might overtrain, see below). For each slice i build a regular model using a regular KFold split and evaluate it OOF. Since entries are users they do not overlap and are independent and I can do folds on them.\n- This way I got ~1e-3 (with large statistical uncertainty estimated from different folds and from independent slice) improvement wrt all-0 prediction. Not great, but it is what it is.\n- To make predictions for the 2 submissions, i used:\n + predictions from the model trained on the most close in time period (i.e. target in training coming from 15 Aug- 15 Oct, feature-set from 1 Jan to 30 Jun) and predicted on the featureset extracted from what was given to us as \"test_v2\".\n + concept the same as above, but use the same period as in the comp, but from 2017-2018 instead of 2018-2019\n\nLet's see in 2 month, if this makes any sense at all...", "votes": null }, { "id": "431051", "postDate": "12/01/2018 14:30:38", "content": "This is a carefully considered approach. For me I initially tried validation on the same period last year but then looking at macro patterns of revenue on the Google Analytics Demo account I focused validation on the latest period available mid Aug to Mid Oct. I also found in the Web Traffic competition last year that the latest data was the most important (although in that competition the gap was only a day or two after the final data was released).
\n\nGood luck and will be interesting to see where we end up in 2 months.
", "rawMarkdown": "This is a carefully considered approach. For me I initially tried validation on the same period last year but then looking at macro patterns of revenue on the Google Analytics Demo account I focused validation on the latest period available mid Aug to Mid Oct. I also found in the Web Traffic competition last year that the latest data was the most important (although in that competition the gap was only a day or two after the final data was released).\n\nGood luck and will be interesting to see where we end up in 2 months.", "votes": null }, { "id": "431104", "postDate": "12/01/2018 16:14:33", "content": "It is not carefully considered enough to consider change in macro patters, as you did :D Anyway, let's see what random in 2 months will say.
", "rawMarkdown": "It is not carefully considered enough to consider change in macro patters, as you did :D Anyway, let's see what random in 2 months will say.", "votes": null }, { "id": "431209", "postDate": "12/01/2018 20:58:20", "content": "Thank you so much for sharing this interesting approach, Misha!
\n\nTo make the predictions, did you divide the problem into classification+prediction or just predicted directly?
", "rawMarkdown": "Thank you so much for sharing this interesting approach, Misha! \n\nTo make the predictions, did you divide the problem into classification+prediction or just predicted directly?", "votes": null }, { "id": "431495", "postDate": "12/02/2018 11:37:24", "content": "I had a classification set up to use predicted probabilities as extra features in regression (which I assume you referred to as prediction). I tried it out and I saw feature importance (both splits and gain) for those features to be not outstanding and improvement in terms of RMSE gain was also tiny. So I dropped it i favour of a simpler model.
\n\nWhat I was doing, and can recommend to others was to train several classifiers with different weights for the positive label (for example, i was using 1,10,100). This is motivated by the fact that data are very unbalanced. And such different classifiers can provide slightly different separation between positive (purchase) and negative (no purchase) training examples. This was helpful in Stage 1, but as said i did not see a similar improvement in stage 2
", "rawMarkdown": "I had a classification set up to use predicted probabilities as extra features in regression (which I assume you referred to as prediction). I tried it out and I saw feature importance (both splits and gain) for those features to be not outstanding and improvement in terms of RMSE gain was also tiny. So I dropped it i favour of a simpler model.\n\nWhat I was doing, and can recommend to others was to train several classifiers with different weights for the positive label (for example, i was using 1,10,100). This is motivated by the fact that data are very unbalanced. And such different classifiers can provide slightly different separation between positive (purchase) and negative (no purchase) training examples. This was helpful in Stage 1, but as said i did not see a similar improvement in stage 2", "votes": null }, { "id": "431705", "postDate": "12/02/2018 19:14:03", "content": "Thank you for your answer. Yes, I meant regression.
\n\nVery interesting ideas.
", "rawMarkdown": "Thank you for your answer. Yes, I meant regression. \n\nVery interesting ideas.", "votes": null }, { "id": "431706", "postDate": "12/02/2018 19:14:50", "content": "(y)
", "rawMarkdown": "(y)", "votes": null }, { "id": "476258", "postDate": "02/21/2019 21:48:53", "content": "At the end the solution landed in top-50. Mostly dues to luck, rather than particular features :)
", "rawMarkdown": "At the end the solution landed in top-50. Mostly dues to luck, rather than particular features :)", "votes": null } ], "comments": [ { "id": 431051, "author_name": "datahobbit", "author_url": "", "post_date": "12/01/2018 14:30:38", "content": "This is a carefully considered approach. For me I initially tried validation on the same period last year but then looking at macro patterns of revenue on the Google Analytics Demo account I focused validation on the latest period available mid Aug to Mid Oct. I also found in the Web Traffic competition last year that the latest data was the most important (although in that competition the gap was only a day or two after the final data was released).
\n\nGood luck and will be interesting to see where we end up in 2 months.
", "votes": null, "replies": [ { "id": 431104, "author_name": "mlisovyi", "author_url": "", "post_date": "12/01/2018 16:14:33", "content": "It is not carefully considered enough to consider change in macro patters, as you did :D Anyway, let's see what random in 2 months will say.
", "votes": null, "replies": [] } ] }, { "id": 431209, "author_name": "sgb1710", "author_url": "", "post_date": "12/01/2018 20:58:20", "content": "Thank you so much for sharing this interesting approach, Misha!
\n\nTo make the predictions, did you divide the problem into classification+prediction or just predicted directly?
", "votes": null, "replies": [ { "id": 431495, "author_name": "mlisovyi", "author_url": "", "post_date": "12/02/2018 11:37:24", "content": "I had a classification set up to use predicted probabilities as extra features in regression (which I assume you referred to as prediction). I tried it out and I saw feature importance (both splits and gain) for those features to be not outstanding and improvement in terms of RMSE gain was also tiny. So I dropped it i favour of a simpler model.
\n\nWhat I was doing, and can recommend to others was to train several classifiers with different weights for the positive label (for example, i was using 1,10,100). This is motivated by the fact that data are very unbalanced. And such different classifiers can provide slightly different separation between positive (purchase) and negative (no purchase) training examples. This was helpful in Stage 1, but as said i did not see a similar improvement in stage 2
", "votes": null, "replies": [] }, { "id": 431705, "author_name": "sgb1710", "author_url": "", "post_date": "12/02/2018 19:14:03", "content": "Thank you for your answer. Yes, I meant regression.
\n\nVery interesting ideas.
", "votes": null, "replies": [] } ] }, { "id": 431706, "author_name": "gatti2602", "author_url": "", "post_date": "12/02/2018 19:14:50", "content": "(y)
", "votes": null, "replies": [] }, { "id": 476258, "author_name": "mlisovyi", "author_url": "", "post_date": "02/21/2019 21:48:53", "content": "At the end the solution landed in top-50. Mostly dues to luck, rather than particular features :)
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "431027": "Thanks to organisers for hosting the competition. It was clearly difficult to manage this one due to the leak, and the scope of the problem has changed drastically after the relaunch, but I think you did your best to keep it running and make it as useful as possible. Respect.\n\nOn my side, i was not able to spend much time on the comp after the relaunch. On Stage 1 i was also very low on the LB, but at least the validation and model building were clear.\n\nOn stage 2 I spent most of the time figuring out how to run json parsing using `dask`. Using a virtual instance with a lot of memory or manually running reading in chunks in pandas was an alternative option, but I knew that dask is something that i want to learn and this was the perfect opportunity. I first tried using the pandas-internal `json_normalise` in dask. I was not able to make it work after several days of painful debugging (there is [an issue already on dask github](https://github.com/dask/dask/issues/4170) asking to add support of it). Then I switched to a more manual approach that was in [Olivier's public kernel]( https://www.kaggle.com/ogrellier/create-extracted-json-fields-dataset). This worked, but I spent more time than I planed. At the end this gave a scalable (but relatively slow) code that was parsing the dataset even in a kernel: [see here](https://www.kaggle.com/mlisovyi/bigdata-dask-pandas-flat-json-trim-data-upd). I persive this as my main achievement and main outcome for personal development :)\n\nNext step, I started to think how to approach model building and how to establish a validation scheme. I did not have any experience with such time-series problems, there were no useful discussions on the forum (at east i did not find one) and I did not have time to dive into literature. So the best idea that I was able to come up with was the following:\n\n- In short: Try to converge to a standard ML problem and tackle it with LGBM;\n- Chunk the train+test into periods of 6 month + 1.5 month gap + 2 month. Get features from the first period, ignore the middle gap period (as i did not have time to play with the demo account and due to large uncertainties it did not look like a good time investment), get the target from the last period (calculating log1p(sum) per user and setting target to 0 for not-returning users);\n- Build user-level feature set for the users from the 6-month period:\n + All info about the last visit (in particular, most relevant would be, was it a bounce? how much was spent? How many times did the user visit up until this last visit (=visit number)? Time since this last visit (wrt the end date of the 6-month period)? What's the country of the user?)\n + Some aggregates (sum/max/mean) over 2,3,6,12 month periods (this includes the sliced 6-month period and any prior visits) for this user: how much did he/she spent in those periods, how many bounces were there, how long did he/she spent on the site, how many purchases were made? Also ratios of those aggregates.\n- I created 3 such 6+1.5+2 slices shifted by 2 month one-wrt-another. And for each i build the feature-set as describes above plus the target\n- My idea was to use them independently (although they overlap, so I assume i might overtrain, see below). For each slice i build a regular model using a regular KFold split and evaluate it OOF. Since entries are users they do not overlap and are independent and I can do folds on them.\n- This way I got ~1e-3 (with large statistical uncertainty estimated from different folds and from independent slice) improvement wrt all-0 prediction. Not great, but it is what it is.\n- To make predictions for the 2 submissions, i used:\n + predictions from the model trained on the most close in time period (i.e. target in training coming from 15 Aug- 15 Oct, feature-set from 1 Jan to 30 Jun) and predicted on the featureset extracted from what was given to us as \"test_v2\".\n + concept the same as above, but use the same period as in the comp, but from 2017-2018 instead of 2018-2019\n\nLet's see in 2 month, if this makes any sense at all...", "431051": "This is a carefully considered approach. For me I initially tried validation on the same period last year but then looking at macro patterns of revenue on the Google Analytics Demo account I focused validation on the latest period available mid Aug to Mid Oct. I also found in the Web Traffic competition last year that the latest data was the most important (although in that competition the gap was only a day or two after the final data was released).\n\nGood luck and will be interesting to see where we end up in 2 months.", "431104": "It is not carefully considered enough to consider change in macro patters, as you did :D Anyway, let's see what random in 2 months will say.", "431209": "Thank you so much for sharing this interesting approach, Misha! \n\nTo make the predictions, did you divide the problem into classification+prediction or just predicted directly?", "431495": "I had a classification set up to use predicted probabilities as extra features in regression (which I assume you referred to as prediction). I tried it out and I saw feature importance (both splits and gain) for those features to be not outstanding and improvement in terms of RMSE gain was also tiny. So I dropped it i favour of a simpler model.\n\nWhat I was doing, and can recommend to others was to train several classifiers with different weights for the positive label (for example, i was using 1,10,100). This is motivated by the fact that data are very unbalanced. And such different classifiers can provide slightly different separation between positive (purchase) and negative (no purchase) training examples. This was helpful in Stage 1, but as said i did not see a similar improvement in stage 2", "431705": "Thank you for your answer. Yes, I meant regression. \n\nVery interesting ideas.", "431706": "(y)", "476258": "At the end the solution landed in top-50. Mostly dues to luck, rather than particular features :)" }, "source": "meta" }