I think price feature like upper shadow, lower shadow should be normalized.
\nFor example,
\nmean_map = df.groupby('Asset_ID')['Close'].mean()
\ndf['asset_mean'] = df['Asset_ID'].map(mean_map)
\ndf['Upper_Shadow'] = (df['High'] - np.maximum(df['Open'], df['Close'])) / df['asset_mean']
What do you think?
", "messages": [ { "id": "1598948", "postDate": "11/29/2021 02:58:29", "content": "I think price feature like upper shadow, lower shadow should be normalized.
\nFor example,
\nmean_map = df.groupby('Asset_ID')['Close'].mean()
\ndf['asset_mean'] = df['Asset_ID'].map(mean_map)
\ndf['Upper_Shadow'] = (df['High'] - np.maximum(df['Open'], df['Close'])) / df['asset_mean']
What do you think?
", "rawMarkdown": "I think price feature like upper shadow, lower shadow should be normalized.\n\nFor example, \n`mean_map = df.groupby('Asset_ID')['Close'].mean()`\n`df['asset_mean'] = df['Asset_ID'].map(mean_map)`\n`df['Upper_Shadow'] = (df['High'] - np.maximum(df['Open'], df['Close'])) / df['asset_mean']`\n\nWhat do you think?", "votes": null }, { "id": "1599351", "postDate": "11/29/2021 11:14:40", "content": "depends most regression decision tree models dont need normalized data , unless you are working with some model with backprop gradient calculation , where you can have exploding gradients like in neural networks for example . You can see this https://datascience.stackexchange.com/questions/5277/do-you-have-to-normalize-data-when-building-decision-trees-using-r if you want more intel basically because decision trees are just if else statements , found out the splitting by decision tree node by node
", "rawMarkdown": "depends most regression decision tree models dont need normalized data , unless you are working with some model with backprop gradient calculation , where you can have exploding gradients like in neural networks for example . You can see this https://datascience.stackexchange.com/questions/5277/do-you-have-to-normalize-data-when-building-decision-trees-using-r if you want more intel basically because decision trees are just if else statements , found out the splitting by decision tree node by node", "votes": null }, { "id": "1599388", "postDate": "11/29/2021 11:53:29", "content": "Thank you for your reply.
\nI know that need for normalization varies depending on the model.
But I mean here necessity of normalization due to price range each asset have differently.
\nFor example, Close price mean is
\nBitCoin → 15611.116274
\nDogecoin → 0.035383
\nThe same upper_shadow, lower_shadow value will have different impacts for different coins.
\nSo, I think these feature should be normalized like above.
@ryotak12 Good thought , I have been doing a lot of EDA and have already observed that , however most people are training different models on different asset ids one model for bitcoin suitable for a particular range one for dogecoin .
", "rawMarkdown": "ryotak12 Good thought , I have been doing a lot of EDA and have already observed that , however most people are training different models on different asset ids one model for bitcoin suitable for a particular range one for dogecoin .", "votes": null }, { "id": "1599550", "postDate": "11/29/2021 14:23:36", "content": "I just recently joined this competition and didn't know that people are training multi models for each asset. Thank you for comment. I will look Notebook a little more !
", "rawMarkdown": "I just recently joined this competition and didn't know that people are training multi models for each asset. Thank you for comment. I will look Notebook a little more !", "votes": null }, { "id": "1599767", "postDate": "11/29/2021 19:44:17", "content": "You are right. I prefer normalizing everything over one price (close). I don't think the features shared by the host are really usefull (linear combination + min / max). I've shared a significant attempt at feature engineering here: https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors and associated feature importance analysis here: https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance
", "rawMarkdown": "You are right. I prefer normalizing everything over one price (close). I don't think the features shared by the host are really usefull (linear combination + min / max). I've shared a significant attempt at feature engineering here: https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors and associated feature importance analysis here: https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance", "votes": null }, { "id": "1599939", "postDate": "11/30/2021 00:55:33", "content": "Hey, but can you provide some reason for it like why you want to normalize it? It depends on the model I guess you can see my comment below if you want to know more about it
", "rawMarkdown": "Hey, but can you provide some reason for it like why you want to normalize it? It depends on the model I guess you can see my comment below if you want to know more about it", "votes": null }, { "id": "1600443", "postDate": "11/30/2021 12:30:54", "content": "I want to normalize it so that O,L,H,C are comparable across assets. It doesn't matter if you build a model per asset. But if you want to build a general model I think you want to make features comparable accross assets. Not just standardizing multiplicatively over some range.
", "rawMarkdown": "I want to normalize it so that O,L,H,C are comparable across assets. It doesn't matter if you build a model per asset. But if you want to build a general model I think you want to make features comparable accross assets. Not just standardizing multiplicatively over some range.", "votes": null }, { "id": "1600583", "postDate": "11/30/2021 14:49:41", "content": "makes sense if we indeed want to conserve data from another asset_id , cuz the price of any crypto can increase in crisis like Corona Virus or drought no matter what asset id it is
", "rawMarkdown": "makes sense if we indeed want to conserve data from another asset_id , cuz the price of any crypto can increase in crisis like Corona Virus or drought no matter what asset id it is", "votes": null } ], "comments": [ { "id": 1599351, "author_name": "swaralipibose", "author_url": "", "post_date": "11/29/2021 11:14:40", "content": "depends most regression decision tree models dont need normalized data , unless you are working with some model with backprop gradient calculation , where you can have exploding gradients like in neural networks for example . You can see this https://datascience.stackexchange.com/questions/5277/do-you-have-to-normalize-data-when-building-decision-trees-using-r if you want more intel basically because decision trees are just if else statements , found out the splitting by decision tree node by node
", "votes": null, "replies": [ { "id": 1599388, "author_name": "ryotak12", "author_url": "", "post_date": "11/29/2021 11:53:29", "content": "Thank you for your reply.
\nI know that need for normalization varies depending on the model.
But I mean here necessity of normalization due to price range each asset have differently.
\nFor example, Close price mean is
\nBitCoin → 15611.116274
\nDogecoin → 0.035383
\nThe same upper_shadow, lower_shadow value will have different impacts for different coins.
\nSo, I think these feature should be normalized like above.
@ryotak12 Good thought , I have been doing a lot of EDA and have already observed that , however most people are training different models on different asset ids one model for bitcoin suitable for a particular range one for dogecoin .
", "votes": null, "replies": [] }, { "id": 1599550, "author_name": "ryotak12", "author_url": "", "post_date": "11/29/2021 14:23:36", "content": "I just recently joined this competition and didn't know that people are training multi models for each asset. Thank you for comment. I will look Notebook a little more !
", "votes": null, "replies": [] } ] }, { "id": 1599767, "author_name": "lucasmorin", "author_url": "", "post_date": "11/29/2021 19:44:17", "content": "You are right. I prefer normalizing everything over one price (close). I don't think the features shared by the host are really usefull (linear combination + min / max). I've shared a significant attempt at feature engineering here: https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors and associated feature importance analysis here: https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance
", "votes": null, "replies": [ { "id": 1599939, "author_name": "swaralipibose", "author_url": "", "post_date": "11/30/2021 00:55:33", "content": "Hey, but can you provide some reason for it like why you want to normalize it? It depends on the model I guess you can see my comment below if you want to know more about it
", "votes": null, "replies": [] }, { "id": 1600443, "author_name": "lucasmorin", "author_url": "", "post_date": "11/30/2021 12:30:54", "content": "I want to normalize it so that O,L,H,C are comparable across assets. It doesn't matter if you build a model per asset. But if you want to build a general model I think you want to make features comparable accross assets. Not just standardizing multiplicatively over some range.
", "votes": null, "replies": [] }, { "id": 1600583, "author_name": "swaralipibose", "author_url": "", "post_date": "11/30/2021 14:49:41", "content": "makes sense if we indeed want to conserve data from another asset_id , cuz the price of any crypto can increase in crisis like Corona Virus or drought no matter what asset id it is
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1598948": "I think price feature like upper shadow, lower shadow should be normalized.\n\nFor example, \n`mean_map = df.groupby('Asset_ID')['Close'].mean()`\n`df['asset_mean'] = df['Asset_ID'].map(mean_map)`\n`df['Upper_Shadow'] = (df['High'] - np.maximum(df['Open'], df['Close'])) / df['asset_mean']`\n\nWhat do you think?", "1599351": "depends most regression decision tree models dont need normalized data , unless you are working with some model with backprop gradient calculation , where you can have exploding gradients like in neural networks for example . You can see this https://datascience.stackexchange.com/questions/5277/do-you-have-to-normalize-data-when-building-decision-trees-using-r if you want more intel basically because decision trees are just if else statements , found out the splitting by decision tree node by node", "1599388": "Thank you for your reply.\nI know that need for normalization varies depending on the model.\n\nBut I mean here necessity of normalization due to price range each asset have differently.\nFor example, Close price mean is\nBitCoin → 15611.116274\nDogecoin → 0.035383\nThe same upper_shadow, lower_shadow value will have different impacts for different coins.\nSo, I think these feature should be normalized like above.", "1599513": "ryotak12 Good thought , I have been doing a lot of EDA and have already observed that , however most people are training different models on different asset ids one model for bitcoin suitable for a particular range one for dogecoin .", "1599550": "I just recently joined this competition and didn't know that people are training multi models for each asset. Thank you for comment. I will look Notebook a little more !", "1599767": "You are right. I prefer normalizing everything over one price (close). I don't think the features shared by the host are really usefull (linear combination + min / max). I've shared a significant attempt at feature engineering here: https://www.kaggle.com/lucasmorin/crypto-forecasting-common-factors and associated feature importance analysis here: https://www.kaggle.com/lucasmorin/crypto-forecasting-lgbm-feval-feature-importance", "1599939": "Hey, but can you provide some reason for it like why you want to normalize it? It depends on the model I guess you can see my comment below if you want to know more about it", "1600443": "I want to normalize it so that O,L,H,C are comparable across assets. It doesn't matter if you build a model per asset. But if you want to build a general model I think you want to make features comparable accross assets. Not just standardizing multiplicatively over some range.", "1600583": "makes sense if we indeed want to conserve data from another asset_id , cuz the price of any crypto can increase in crisis like Corona Virus or drought no matter what asset id it is" }, "source": "meta" }