Memory can be an issue as soon as the number of features become large. Here are few tricks, some being discussed a while ago on this forum.
\n\nCreate one model per type instead of one global model. Only add features to the relevant part of train and test for each of the 8 models. For instance:
\n\nfor t in sorted(train.type.unique()):\n train_t = train[train.type == t]\n test_t = test[test.type == t]\n train_t, test_t, features = add_features(train_t, test_t)\nIf you use NNs, you can add features batch per batch in a generator
Downcast to float32 or int32 if not smaller. Some public kernels have a very elaborate function to downcast values and save memory, but it is applied after data is created/read once. Apply the same logic as soon as you add new columns, column per column.\nFor instance, if you just added some columns to your dataframes via, say, a merge operation, then downcast them immediately by:
\n\nfor c in new_columns:\n data[c] = data[c].astype('float32')\nDowncasting to float32 works in all cases I can foresee here. Downcast to integer if you are sure the values are integer and contain no NaN.
\n\nSet copy=False in pandas.DataFrame.merge()
Use numpy instead of pandas as much as possible.
Memory can be an issue as soon as the number of features become large. Here are few tricks, some being discussed a while ago on this forum.
\n\nCreate one model per type instead of one global model. Only add features to the relevant part of train and test for each of the 8 models. For instance:
\n\nfor t in sorted(train.type.unique()):\n train_t = train[train.type == t]\n test_t = test[test.type == t]\n train_t, test_t, features = add_features(train_t, test_t)\nIf you use NNs, you can add features batch per batch in a generator
Downcast to float32 or int32 if not smaller. Some public kernels have a very elaborate function to downcast values and save memory, but it is applied after data is created/read once. Apply the same logic as soon as you add new columns, column per column.\nFor instance, if you just added some columns to your dataframes via, say, a merge operation, then downcast them immediately by:
\n\nfor c in new_columns:\n data[c] = data[c].astype('float32')\nDowncasting to float32 works in all cases I can foresee here. Downcast to integer if you are sure the values are integer and contain no NaN.
\n\nSet copy=False in pandas.DataFrame.merge()
Use numpy instead of pandas as much as possible.
Thanks!\nanother possible way is to use dask.dataframe to replace pandas.DataFrame\n```\nimport dask.dataframe as dd\ndf= dd.from_pandas(df,npartitions=4)\ndf= df.groupby(['molecule_name','atom_index']).agg(AGGS1).compute()
``Dask is a python out-of-core parallelization framework that offers various parallelized container types, one of which is the dataframe. It let's you perform most commonpandas.DataFrame` operations in parallel and/or distributed with data that is too large to fit in memory. The core of dask is a set of schedulers and an API for building computation graphs, hence we have to call .compute() at the end in order for any computation to actually take place.
Hello! Nice to see you in this competition and in the top of LB with 1 submission!\nAnd good to know that LGBM can give -2. :) Thanks!
", "rawMarkdown": "Hello! Nice to see you in this competition and in the top of LB with 1 submission!\nAnd good to know that LGBM can give -2. :) Thanks!", "votes": 1 }, { "id": 581955, "postDate": "2019-07-22T15:30:13.513Z", "content": "Oncle CPMP,\nThanks for the tricks and quite good first submission !!
", "rawMarkdown": "Oncle CPMP,\nThanks for the tricks and quite good first submission !!", "votes": 1 }, { "id": 581408, "postDate": "2019-07-21T22:47:41.133Z", "content": "What do you think of?
\n\n``\ntrain = pd.read_csv('../input/train.csv')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))\nfor col in('type','molecule_name','atom_index_0','atom_index_1'):\n train[col]=train[col].astype('category')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))
```
\n\n```
\n\n 0 1\nIndex NaN 80\natom_index_0 int64 37265176\natom_index_1 int64 37265176\nid int64 37265176\nmolecule_name object 340044731\nscalar_coupling_constant float64 37265176\ntype object 284146967\n 0 1\nIndex NaN 80\natom_index_0 category 4659659\natom_index_1 category 4659659\nid int64 37265176\nmolecule_name category 27459247\nscalar_coupling_constant float64 37265176\ntype category 4658955\n```
\n\nHowever, categorical values may not be understood by ML software
", "rawMarkdown": "What do you think of?\n\n```\ntrain = pd.read_csv('../input/train.csv')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))\nfor col in('type','molecule_name','atom_index_0','atom_index_1'):\n train[col]=train[col].astype('category')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))`\n\n```\n\n```\n\n 0 1\nIndex NaN 80\natom_index_0 int64 37265176\natom_index_1 int64 37265176\nid int64 37265176\nmolecule_name object 340044731\nscalar_coupling_constant float64 37265176\ntype object 284146967\n 0 1\nIndex NaN 80\natom_index_0 category 4659659\natom_index_1 category 4659659\nid int64 37265176\nmolecule_name category 27459247\nscalar_coupling_constant float64 37265176\ntype category 4658955\n```\n\nHowever, categorical values may not be understood by ML software", "votes": 1, "replies": [ { "id": 581564, "postDate": "2019-07-22T05:36:15.850Z", "content": "Categorical are indeed a good way to compress. I am still using factorize, which is lower level, but with it you can drop the mapping from indices to strings.
\n\nI prefer to use factorize for that reason.
\n\ntrain = pd.read_csv('../input/train.csv')\nfor col in('type','molecule_name'):\n train[col] = pd.factorize(train[col])[0].astype('uint32')\nfor col in('atom_index_0','atom_index_1'):\n train[col] = train[col].astype('uint8')\nprint(pd.concat((train.dtypes, train.memory_usage(deep=True)), axis=1))\n\nIndex NaN 80\natom_index_0 uint8 4658147\natom_index_1 uint8 4658147\nid int64 37265176\nmolecule_name uint32 18632588\nscalar_coupling_constant float64 37265176\ntype uint32 18632588\ntype column can be just \"int8\".
\n\nI used a bit different approach to have lesser memory usage:\n\ntrain_dtypes = {\n 'molecule_name': 'category',\n 'atom_index_0': 'int8',\n 'atom_index_1': 'int8',\n 'type': 'category',\n 'scalar_coupling_constant': 'float32'\n}\ntrain_csv = pd.read_csv(f'{DATA_PATH}/train.csv', index_col='id', dtype=train_dtypes)\ntrain_csv['molecule_index'] = train_csv.molecule_name.str.replace('dsgdb9nsd_', '').astype('int32')\ntrain_csv = train_csv[['molecule_index', 'atom_index_0', 'atom_index_1', 'type', 'scalar_coupling_constant']]\n\nIn this way Pandas loads everything with lesser amount of memory.
But usually, after different merge and multiplication operations, everything is upcasted to float64, so explicit downcasting to float32 is required.
If it's still an issue, you can split into subbranches with number unique atoms (max five), if you find that useful
", "rawMarkdown": "If it's still an issue, you can split into subbranches with number unique atoms (max five), if you find that useful", "votes": 1 }, { "id": 589900, "postDate": "2019-08-01T14:35:40.683Z", "content": "@cpmpml Pretty awesome tricks. Could you pls elaborate or give some more tricks if I am working with images in a NN.\nLet me give you a clearer view. I had a dataset with 20000 train and test images which were 500*500 in size. Now I want some tricks or methods to make a model for this dataset.
", "rawMarkdown": "@cpmpml Pretty awesome tricks. Could you pls elaborate or give some more tricks if I am working with images in a NN.\nLet me give you a clearer view. I had a dataset with 20000 train and test images which were 500*500 in size. Now I want some tricks or methods to make a model for this dataset.", "replies": [ { "id": 590023, "postDate": "2019-08-01T17:49:56.507Z", "content": "Read the forum of any of the image competitions on Kaggle and you'll get plenty of useful information for your use case.
", "rawMarkdown": "Read the forum of any of the image competitions on Kaggle and you'll get plenty of useful information for your use case.", "votes": 1 }, { "id": 590034, "postDate": "2019-08-01T18:11:37.340Z", "content": "Thank you .😃
", "rawMarkdown": "Thank you .😃 " } ] }, { "id": 593291, "postDate": "2019-08-06T12:23:24.637Z", "rawMarkdown": "", "isDeleted": true, "replies": [ { "id": 593326, "postDate": "2019-08-06T13:05:27.050Z", "content": "lightgbm, graph neural networks.
", "rawMarkdown": "lightgbm, graph neural networks." }, { "id": 593340, "postDate": "2019-08-06T13:21:42.163Z", "rawMarkdown": "", "isDeleted": true } ] } ], "comments": [ { "id": 586979, "author_name": "Teng Lei", "author_url": "", "post_date": "2019-07-30T01:45:31.817000", "content": "Thanks!\nanother possible way is to use dask.dataframe to replace pandas.DataFrame\n```\nimport dask.dataframe as dd\ndf= dd.from_pandas(df,npartitions=4)\ndf= df.groupby(['molecule_name','atom_index']).agg(AGGS1).compute()
``Dask is a python out-of-core parallelization framework that offers various parallelized container types, one of which is the dataframe. It let's you perform most commonpandas.DataFrame` operations in parallel and/or distributed with data that is too large to fit in memory. The core of dask is a set of schedulers and an API for building computation graphs, hence we have to call .compute() at the end in order for any computation to actually take place.
Hello! Nice to see you in this competition and in the top of LB with 1 submission!\nAnd good to know that LGBM can give -2. :) Thanks!
", "votes": 1, "replies": [] }, { "id": 581955, "author_name": "eagle4", "author_url": "", "post_date": "2019-07-22T15:30:13.513000", "content": "Oncle CPMP,\nThanks for the tricks and quite good first submission !!
", "votes": 1, "replies": [] }, { "id": 581408, "author_name": "jmtest", "author_url": "", "post_date": "2019-07-21T22:47:41.133000", "content": "What do you think of?
\n\n``\ntrain = pd.read_csv('../input/train.csv')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))\nfor col in('type','molecule_name','atom_index_0','atom_index_1'):\n train[col]=train[col].astype('category')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))
```
\n\n```
\n\n 0 1\nIndex NaN 80\natom_index_0 int64 37265176\natom_index_1 int64 37265176\nid int64 37265176\nmolecule_name object 340044731\nscalar_coupling_constant float64 37265176\ntype object 284146967\n 0 1\nIndex NaN 80\natom_index_0 category 4659659\natom_index_1 category 4659659\nid int64 37265176\nmolecule_name category 27459247\nscalar_coupling_constant float64 37265176\ntype category 4658955\n```
\n\nHowever, categorical values may not be understood by ML software
", "votes": 1, "replies": [ { "id": 581564, "author_name": "CPMP", "author_url": "", "post_date": "2019-07-22T05:36:15.850000", "content": "Categorical are indeed a good way to compress. I am still using factorize, which is lower level, but with it you can drop the mapping from indices to strings.
\n\nI prefer to use factorize for that reason.
\n\ntrain = pd.read_csv('../input/train.csv')\nfor col in('type','molecule_name'):\n train[col] = pd.factorize(train[col])[0].astype('uint32')\nfor col in('atom_index_0','atom_index_1'):\n train[col] = train[col].astype('uint8')\nprint(pd.concat((train.dtypes, train.memory_usage(deep=True)), axis=1))\n\nIndex NaN 80\natom_index_0 uint8 4658147\natom_index_1 uint8 4658147\nid int64 37265176\nmolecule_name uint32 18632588\nscalar_coupling_constant float64 37265176\ntype uint32 18632588\ntype column can be just \"int8\".
\n\nI used a bit different approach to have lesser memory usage:\n\ntrain_dtypes = {\n 'molecule_name': 'category',\n 'atom_index_0': 'int8',\n 'atom_index_1': 'int8',\n 'type': 'category',\n 'scalar_coupling_constant': 'float32'\n}\ntrain_csv = pd.read_csv(f'{DATA_PATH}/train.csv', index_col='id', dtype=train_dtypes)\ntrain_csv['molecule_index'] = train_csv.molecule_name.str.replace('dsgdb9nsd_', '').astype('int32')\ntrain_csv = train_csv[['molecule_index', 'atom_index_0', 'atom_index_1', 'type', 'scalar_coupling_constant']]\n\nIn this way Pandas loads everything with lesser amount of memory.
But usually, after different merge and multiplication operations, everything is upcasted to float64, so explicit downcasting to float32 is required.
If it's still an issue, you can split into subbranches with number unique atoms (max five), if you find that useful
", "votes": 1, "replies": [] }, { "id": 589900, "author_name": "Aditya Khandelwal", "author_url": "", "post_date": "2019-08-01T14:35:40.683000", "content": "@cpmpml Pretty awesome tricks. Could you pls elaborate or give some more tricks if I am working with images in a NN.\nLet me give you a clearer view. I had a dataset with 20000 train and test images which were 500*500 in size. Now I want some tricks or methods to make a model for this dataset.
", "votes": 0, "replies": [ { "id": 590023, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-01T17:49:56.507000", "content": "Read the forum of any of the image competitions on Kaggle and you'll get plenty of useful information for your use case.
", "votes": 1, "replies": [] }, { "id": 590034, "author_name": "Aditya Khandelwal", "author_url": "", "post_date": "2019-08-01T18:11:37.340000", "content": "Thank you .😃
", "votes": 0, "replies": [] } ] }, { "id": 593291, "author_name": "", "author_url": "", "post_date": "2019-08-06T12:23:24.637000", "content": "", "votes": 0, "replies": [ { "id": 593326, "author_name": "CPMP", "author_url": "", "post_date": "2019-08-06T13:05:27.050000", "content": "lightgbm, graph neural networks.
", "votes": 0, "replies": [] }, { "id": 593340, "author_name": "", "author_url": "", "post_date": "2019-08-06T13:21:42.163000", "content": "", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "581151": "Memory can be an issue as soon as the number of features become large. Here are few tricks, some being discussed a while ago on this forum.\n\n1. Create one model per type instead of one global model. Only add features to the relevant part of train and test for each of the 8 models. For instance:\n\n for t in sorted(train.type.unique()):\n train_t = train[train.type == t]\n test_t = test[test.type == t]\n train_t, test_t, features = add_features(train_t, test_t)\n\n2. If you use NNs, you can add features batch per batch in a generator\n\n3. Downcast to float32 or int32 if not smaller. Some public kernels have a very elaborate function to downcast values and save memory, but it is applied after data is created/read once. Apply the same logic as soon as you add new columns, column per column.\nFor instance, if you just added some columns to your dataframes via, say, a merge operation, then downcast them immediately by:\n\n for c in new_columns:\n data[c] = data[c].astype('float32')\n\nDowncasting to float32 works in all cases I can foresee here. Downcast to integer if you are sure the values are integer and contain no NaN.\n\n4. Set copy=False in pandas.DataFrame.merge()\n\n5. Use numpy instead of pandas as much as possible.\n", "586979": "Thanks!\nanother possible way is to use `dask.dataframe ` to replace `pandas.DataFrame`\n```\nimport dask.dataframe as dd\ndf= dd.from_pandas(df,npartitions=4)\ndf= df.groupby(['molecule_name','atom_index']).agg(AGGS1).compute()\n\n```Dask is a python out-of-core parallelization framework that offers various parallelized container types, one of which is the dataframe. It let's you perform most common `pandas.DataFrame` operations in parallel and/or distributed with data that is too large to fit in memory. The core of dask is a set of schedulers and an API for building computation graphs, hence we have to call .compute() at the end in order for any computation to actually take place.", "582033": "Hello! Nice to see you in this competition and in the top of LB with 1 submission!\nAnd good to know that LGBM can give -2. :) Thanks!", "581955": "Oncle CPMP,\nThanks for the tricks and quite good first submission !!", "581408": "What do you think of?\n\n```\ntrain = pd.read_csv('../input/train.csv')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))\nfor col in('type','molecule_name','atom_index_0','atom_index_1'):\n train[col]=train[col].astype('category')\nprint(pd.concat((train.dtypes,train.memory_usage(deep=True)),axis=1))`\n\n```\n\n```\n\n 0 1\nIndex NaN 80\natom_index_0 int64 37265176\natom_index_1 int64 37265176\nid int64 37265176\nmolecule_name object 340044731\nscalar_coupling_constant float64 37265176\ntype object 284146967\n 0 1\nIndex NaN 80\natom_index_0 category 4659659\natom_index_1 category 4659659\nid int64 37265176\nmolecule_name category 27459247\nscalar_coupling_constant float64 37265176\ntype category 4658955\n```\n\nHowever, categorical values may not be understood by ML software", "581172": "If it's still an issue, you can split into subbranches with number unique atoms (max five), if you find that useful", "589900": "@cpmpml Pretty awesome tricks. Could you pls elaborate or give some more tricks if I am working with images in a NN.\nLet me give you a clearer view. I had a dataset with 20000 train and test images which were 500*500 in size. Now I want some tricks or methods to make a model for this dataset.", "593291": "" } }