{ "id": 80276, "title": "How to do quick iterations", "url": "/competitions/elo-merchant-category-recommendation/discussion/80276", "author_name": "Andre Barbosa", "post_date": "2019-02-12T09:33:26.943000", "votes": 0, "comment_count": 13, "views": 0, "content": "

Hey there,

\n\n

I'm facing a problem in this competition related to computation effort. Every time I want to test new features or ideas I need to wait around ~1hour (+ 1 hour to compile the kernel ) due to multiple joins in historical/new merchants data. What do you guys usually do to make quick iterations and ideas?

", "messages": [ { "id": 470780, "postDate": "2019-02-13T15:01:15.743Z", "content": "

You should join Microsoft Malware competition. After that you will no longer complain the ELO iterations take too log...

", "rawMarkdown": "You should join Microsoft Malware competition. After that you will no longer complain the ELO iterations take too log...", "votes": 5, "replies": [ { "id": 470856, "postDate": "2019-02-13T17:30:08.063Z", "content": "

Indeed, it is much much faster on Elo dataset. It is possible to run ~100 rounds of HPO w/ 13-CV for me in one day.

", "rawMarkdown": "Indeed, it is much much faster on Elo dataset. It is possible to run ~100 rounds of HPO w/ 13-CV for me in one day." }, { "id": 471321, "postDate": "2019-02-14T09:55:18.667Z", "content": "

@CPMP, ditto on the Microsoft Malware competition experience. It is far more excitingly frustrating :-)

", "rawMarkdown": "@CPMP, ditto on the Microsoft Malware competition experience. It is far more excitingly frustrating :-)" }, { "id": 472290, "postDate": "2019-02-15T16:34:25.197Z", "content": "

Best competition ever. No kernels, and you start with 0 features with huge data.

", "rawMarkdown": "Best competition ever. No kernels, and you start with 0 features with huge data." } ] }, { "id": 470069, "postDate": "2019-02-12T10:30:41.270Z", "content": "

Save your \"train.pkl\" and \"test.pkl\" in kernel output and you can load kernel output in new kernel when you are trying new features and can pd.merge on \"card_id\" the additional features\n.This is what I am doing

", "rawMarkdown": "Save your \"train.pkl\" and \"test.pkl\" in kernel output and you can load kernel output in new kernel when you are trying new features and can pd.merge on \"card_id\" the additional features\n.This is what I am doing", "votes": 3 }, { "id": 470078, "postDate": "2019-02-12T10:41:05.687Z", "content": "

Try to save the file in parquet format. Convert data type to a lower grade (eg from float64 to float32 or float16).

", "rawMarkdown": "Try to save the file in parquet format. Convert data type to a lower grade (eg from float64 to float32 or float16). ", "votes": 1, "replies": [ { "id": 470861, "postDate": "2019-02-13T17:32:58.570Z", "content": "

Parquet format is quite useful to me, I tried to use HDF5 files it is nice to have dataframes stored by key-indexed, but the metadata limitation is a bit trouble some.

", "rawMarkdown": "Parquet format is quite useful to me, I tried to use HDF5 files it is nice to have dataframes stored by key-indexed, but the metadata limitation is a bit trouble some.", "votes": 1 }, { "id": 470893, "postDate": "2019-02-13T18:20:28.143Z", "content": "

Thanks for the insights ! I've never considered using HDF5 files

", "rawMarkdown": "Thanks for the insights ! I've never considered using HDF5 files " } ] }, { "id": 471050, "postDate": "2019-02-14T00:26:28.480Z", "content": "

Feature engineer many features at once. Then you can add them in intervals to test them separately. Then, If you are using lightgbm, you can train the model in less than 5 minutes per interval.

", "rawMarkdown": "Feature engineer many features at once. Then you can add them in intervals to test them separately. Then, If you are using lightgbm, you can train the model in less than 5 minutes per interval." }, { "id": 470886, "postDate": "2019-02-13T18:04:57.423Z", "content": "

Thanks for the awesome response ppl. This is being really helpful :)

\n\n

I did not considerer using parquet format yet. Maybe I'll consider that in another comp... I'm using pickle by now and it has been good so far

\n\n

However, the main problem that I'm facing is more related to huge joins (on historical_transactions) that I'm doing instead of the final table that I'm applying to train my model. Do you have any tips for that?

\n\n

Since the best relevant features are coming from aggregations, I was thinking about ways that I could make experiments quicker.

\n\n

For now I'm using a subset of data to make aggs but I would like to know if there are \"best\" ways for doing this.

", "rawMarkdown": "Thanks for the awesome response ppl. This is being really helpful :)\n\nI did not considerer using parquet format yet. Maybe I'll consider that in another comp... I'm using pickle by now and it has been good so far \n\nHowever, the main problem that I'm facing is more related to huge joins (on historical_transactions) that I'm doing instead of the final table that I'm applying to train my model. Do you have any tips for that?\n\nSince the best relevant features are coming from aggregations, I was thinking about ways that I could make experiments quicker.\n\nFor now I'm using a subset of data to make aggs but I would like to know if there are \"best\" ways for doing this.\n\n" }, { "id": 470863, "postDate": "2019-02-13T17:35:04.150Z", "content": "

Would you consider to 1) save those processed data set into parquet format and uploaded to kernel, and 2) run XGBoost w/ GPU kaggle kernel for experiments?

", "rawMarkdown": "Would you consider to 1) save those processed data set into parquet format and uploaded to kernel, and 2) run XGBoost w/ GPU kaggle kernel for experiments?" }, { "id": 470037, "postDate": "2019-02-12T09:33:26.943Z", "content": "

Hey there,

\n\n

I'm facing a problem in this competition related to computation effort. Every time I want to test new features or ideas I need to wait around ~1hour (+ 1 hour to compile the kernel ) due to multiple joins in historical/new merchants data. What do you guys usually do to make quick iterations and ideas?

", "rawMarkdown": "Hey there,\n\nI'm facing a problem in this competition related to computation effort. Every time I want to test new features or ideas I need to wait around ~1hour (+ 1 hour to compile the kernel ) due to multiple joins in historical/new merchants data. What do you guys usually do to make quick iterations and ideas?" }, { "id": 470303, "postDate": "2019-02-12T18:25:12.083Z", "rawMarkdown": "", "votes": 1, "isDeleted": true }, { "id": 470821, "postDate": "2019-02-13T16:09:47.890Z", "content": "

Thanks for sharing Andre!

", "rawMarkdown": "Thanks for sharing Andre!", "votes": 1 } ], "comments": [ { "id": 470780, "author_name": "CPMP", "author_url": "", "post_date": "2019-02-13T15:01:15.743000", "content": "

You should join Microsoft Malware competition. After that you will no longer complain the ELO iterations take too log...

", "votes": 5, "replies": [ { "id": 470856, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2019-02-13T17:30:08.063000", "content": "

Indeed, it is much much faster on Elo dataset. It is possible to run ~100 rounds of HPO w/ 13-CV for me in one day.

", "votes": 0, "replies": [] }, { "id": 471321, "author_name": "YaGana Sheriff-Hussaini", "author_url": "", "post_date": "2019-02-14T09:55:18.667000", "content": "

@CPMP, ditto on the Microsoft Malware competition experience. It is far more excitingly frustrating :-)

", "votes": 0, "replies": [] }, { "id": 472290, "author_name": "NxGTR", "author_url": "", "post_date": "2019-02-15T16:34:25.197000", "content": "

Best competition ever. No kernels, and you start with 0 features with huge data.

", "votes": 0, "replies": [] } ] }, { "id": 470069, "author_name": "Manoj", "author_url": "", "post_date": "2019-02-12T10:30:41.270000", "content": "

Save your \"train.pkl\" and \"test.pkl\" in kernel output and you can load kernel output in new kernel when you are trying new features and can pd.merge on \"card_id\" the additional features\n.This is what I am doing

", "votes": 3, "replies": [] }, { "id": 470078, "author_name": "FP", "author_url": "", "post_date": "2019-02-12T10:41:05.687000", "content": "

Try to save the file in parquet format. Convert data type to a lower grade (eg from float64 to float32 or float16).

", "votes": 1, "replies": [ { "id": 470861, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2019-02-13T17:32:58.570000", "content": "

Parquet format is quite useful to me, I tried to use HDF5 files it is nice to have dataframes stored by key-indexed, but the metadata limitation is a bit trouble some.

", "votes": 1, "replies": [] }, { "id": 470893, "author_name": "Andre Barbosa", "author_url": "", "post_date": "2019-02-13T18:20:28.143000", "content": "

Thanks for the insights ! I've never considered using HDF5 files

", "votes": 0, "replies": [] } ] }, { "id": 471050, "author_name": "Kyle Peters", "author_url": "", "post_date": "2019-02-14T00:26:28.480000", "content": "

Feature engineer many features at once. Then you can add them in intervals to test them separately. Then, If you are using lightgbm, you can train the model in less than 5 minutes per interval.

", "votes": 0, "replies": [] }, { "id": 470886, "author_name": "Andre Barbosa", "author_url": "", "post_date": "2019-02-13T18:04:57.423000", "content": "

Thanks for the awesome response ppl. This is being really helpful :)

\n\n

I did not considerer using parquet format yet. Maybe I'll consider that in another comp... I'm using pickle by now and it has been good so far

\n\n

However, the main problem that I'm facing is more related to huge joins (on historical_transactions) that I'm doing instead of the final table that I'm applying to train my model. Do you have any tips for that?

\n\n

Since the best relevant features are coming from aggregations, I was thinking about ways that I could make experiments quicker.

\n\n

For now I'm using a subset of data to make aggs but I would like to know if there are \"best\" ways for doing this.

", "votes": 0, "replies": [] }, { "id": 470863, "author_name": "Chia-Ta Tsai", "author_url": "", "post_date": "2019-02-13T17:35:04.150000", "content": "

Would you consider to 1) save those processed data set into parquet format and uploaded to kernel, and 2) run XGBoost w/ GPU kaggle kernel for experiments?

", "votes": 0, "replies": [] }, { "id": 470303, "author_name": "", "author_url": "", "post_date": "2019-02-12T18:25:12.083000", "content": "", "votes": 1, "replies": [] }, { "id": 470821, "author_name": "mtodisco10", "author_url": "", "post_date": "2019-02-13T16:09:47.890000", "content": "

Thanks for sharing Andre!

", "votes": 1, "replies": [] } ], "raw_markdown_by_id": { "470780": "You should join Microsoft Malware competition. After that you will no longer complain the ELO iterations take too log...", "470069": "Save your \"train.pkl\" and \"test.pkl\" in kernel output and you can load kernel output in new kernel when you are trying new features and can pd.merge on \"card_id\" the additional features\n.This is what I am doing", "470078": "Try to save the file in parquet format. Convert data type to a lower grade (eg from float64 to float32 or float16). ", "471050": "Feature engineer many features at once. Then you can add them in intervals to test them separately. Then, If you are using lightgbm, you can train the model in less than 5 minutes per interval.", "470886": "Thanks for the awesome response ppl. This is being really helpful :)\n\nI did not considerer using parquet format yet. Maybe I'll consider that in another comp... I'm using pickle by now and it has been good so far \n\nHowever, the main problem that I'm facing is more related to huge joins (on historical_transactions) that I'm doing instead of the final table that I'm applying to train my model. Do you have any tips for that?\n\nSince the best relevant features are coming from aggregations, I was thinking about ways that I could make experiments quicker.\n\nFor now I'm using a subset of data to make aggs but I would like to know if there are \"best\" ways for doing this.\n\n", "470863": "Would you consider to 1) save those processed data set into parquet format and uploaded to kernel, and 2) run XGBoost w/ GPU kaggle kernel for experiments?", "470037": "Hey there,\n\nI'm facing a problem in this competition related to computation effort. Every time I want to test new features or ideas I need to wait around ~1hour (+ 1 hour to compile the kernel ) due to multiple joins in historical/new merchants data. What do you guys usually do to make quick iterations and ideas?", "470303": "", "470821": "Thanks for sharing Andre!" } }