Can we make a public dataset with daily frequency updations for the recent few months that are not in the dataset and treat it as data for evaluating our model's performance?
", "messages": [ { "id": "1572865", "postDate": "11/06/2021 03:30:52", "content": "Can we make a public dataset with daily frequency updations for the recent few months that are not in the dataset and treat it as data for evaluating our model's performance?
", "rawMarkdown": "Can we make a public dataset with daily frequency updations for the recent few months that are not in the dataset and treat it as data for evaluating our model's performance?", "votes": null }, { "id": "1572927", "postDate": "11/06/2021 05:13:59", "content": "I don know you have seen these topics before or not but here are Topics, check them out,
But in short, Yes,
\nFreely & publicly available external data is allowed, including pre-trained models.
\nand that has to be offline data because the submissions don't allow online NB
Thanks for the info. I had not seen the second topic.
", "rawMarkdown": "Thanks for the info. I had not seen the second topic.", "votes": null }, { "id": "1572957", "postDate": "11/06/2021 05:49:38", "content": "No problem, Check this out to add more data
\nExtra Data - Auto updating dataset & Baseline with CV 💎🤲
Hi @soumya9977 This notebook was good but have could been be better if a single dataset had all the assets included together.
", "rawMarkdown": "Hi @soumya9977 This notebook was good but have could been be better if a single dataset had all the assets included together.", "votes": null }, { "id": "1573055", "postDate": "11/06/2021 08:33:36", "content": "Yeah, I was thinking the same you know, I was trying to implement something but it requires only a train.csv and a test.csv, no train test split or something. I also saw some notebooks(I'd say most of the NBs) where they give inputs as a form of a combination of asset.csv and train.csv. for example,
def get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n\n # TODO: Try different features here!\n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.replace([np.inf, -np.inf], np.nan).dropna(how=\"any\")\n\n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n\n # TODO: Try different models here!\n model = LGBMRegressor(n_estimators=10)\n model.fit(X, y)\n return X, y, model\nCheck this code out. it's mainly training the model for each id, and the get_feature function gives some columns like 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP' with some new feature engineered columns. If I train on only the train.csv then what would be the problem? I don't get it. But I have seen some Nbs where someone noticed the author did not include the assets.csv and pointed out as a but (check out here) and the public score is also zero.
I have commented under some NBs and topics, If they answer, I will definitely update this thread.
\nI will try to run the code for the whole train.csv and see the score I get.
can you help @odins0n ?
", "rawMarkdown": "Yeah, I was thinking the same you know, I was trying to implement something but it requires only a `train.csv` and a `test.csv`, no train test split or something. I also saw some notebooks(I'd say most of the NBs) where they give inputs as a form of a combination of `asset.csv and train.csv`. for example, \n```python\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n \n # TODO: Try different features here!\n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.replace([np.inf, -np.inf], np.nan).dropna(how=\"any\")\n \n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n\n # TODO: Try different models here!\n model = LGBMRegressor(n_estimators=10)\n model.fit(X, y)\n return X, y, model\n```\n[Borrowed code source](https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017)\n\n\nCheck this code out. it's mainly training the model for each id, and the get_feature function gives some columns like 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP' with some new feature engineered columns. If I train on only the train.csv then what would be the problem? I don't get it. But I have seen some Nbs where someone noticed the author did not include the `assets.csv` and pointed out as a but ([check out here](https://www.kaggle.com/metathesis/torch-lstm-baseline-training-updated/comments#1571111)) and the public score is also zero. \n\nI have commented under some NBs and topics, If they answer, I will definitely update this thread. \nI will try to run the code for the whole `train.csv` and see the score I get.\n\ncan you help @odins0n ?", "votes": null }, { "id": "1573058", "postDate": "11/06/2021 08:35:52", "content": "People may update on this too,
\n", "rawMarkdown": "People may update on this too,\n- [Meaning of weights ?](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285183)", "votes": null }, { "id": "1573078", "postDate": "11/06/2021 08:50:14", "content": "Yes! See my post here
", "rawMarkdown": "Yes! See my post [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726)", "votes": null }, { "id": "1573215", "postDate": "11/06/2021 11:57:20", "content": "@soumya9977 Every asset(cryptocurrency) has a different price, the volume traded and starting time from which records are given in the train.csv. In this notebook that u mentioned the author has trained the same model for every asset separately and later predicted on test.csv for each asset from their saved model respectively. I think training each asset separately might get good results due to the above factors but at the same time, we will be unable to get the benefit of the influence of price change on one asset due to the price change in another.
\nI think we can try and train the train.csv file directly and check what results we get.
", "rawMarkdown": "soumya9977 Every asset(cryptocurrency) has a different price, the volume traded and starting time from which records are given in the train.csv. In [this](https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017) notebook that u mentioned the author has trained the same model for every asset separately and later predicted on test.csv for each asset from their saved model respectively. I think training each asset separately might get good results due to the above factors but at the same time, we will be unable to get the benefit of the influence of price change on one asset due to the price change in another.\n\nI think we can try and train the train.csv file directly and check what results we get.", "votes": null }, { "id": "1573290", "postDate": "11/06/2021 13:20:31", "content": "@odinson Yes a dataset with all assets included together will be useful. Because I think the data is till September only. Can you make a dataset public so that it can be appended in the train.csv ?
", "rawMarkdown": "odinson Yes a dataset with all assets included together will be useful. Because I think the data is till September only. Can you make a dataset public so that it can be appended in the train.csv ?", "votes": null }, { "id": "1573330", "postDate": "11/06/2021 14:28:12", "content": "I totally get your point. Both of the ways have their own pros and cons. And as you said trying out both of them, sounds like is a good idea. Thank you for answering 😇.
", "rawMarkdown": "I totally get your point. Both of the ways have their own pros and cons. And as you said trying out both of them, sounds like is a good idea. Thank you for answering 😇.", "votes": null }, { "id": "1573767", "postDate": "11/06/2021 22:21:16", "content": "Sure @kagglerrr1 !!! I will try and publish a dataset in the same format as train.csv
\nI will update this discussion once it's done.
I too have started to do this. See post here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913 and relevant notebooks / datasets:
\nGetting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:
\nraw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance
\nTransforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data
\npreprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance
I still have some configuration / automation parts to figure.
", "rawMarkdown": "I too have started to do this. See post here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913 and relevant notebooks / datasets:\n\nGetting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:\nraw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\nTransforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\npreprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance\n\nI still have some configuration / automation parts to figure.", "votes": null }, { "id": "1575026", "postDate": "11/08/2021 05:30:35", "content": "This will be so handy @lucasmorin. Thanks !!!
", "rawMarkdown": "This will be so handy @lucasmorin. Thanks !!!", "votes": null } ], "comments": [ { "id": 1572927, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 05:13:59", "content": "I don know you have seen these topics before or not but here are Topics, check them out,
But in short, Yes,
\nFreely & publicly available external data is allowed, including pre-trained models.
\nand that has to be offline data because the submissions don't allow online NB
Thanks for the info. I had not seen the second topic.
", "votes": null, "replies": [] }, { "id": 1572957, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 05:49:38", "content": "No problem, Check this out to add more data
\nExtra Data - Auto updating dataset & Baseline with CV 💎🤲
Hi @soumya9977 This notebook was good but have could been be better if a single dataset had all the assets included together.
", "votes": null, "replies": [] }, { "id": 1573055, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 08:33:36", "content": "Yeah, I was thinking the same you know, I was trying to implement something but it requires only a train.csv and a test.csv, no train test split or something. I also saw some notebooks(I'd say most of the NBs) where they give inputs as a form of a combination of asset.csv and train.csv. for example,
def get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n\n # TODO: Try different features here!\n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.replace([np.inf, -np.inf], np.nan).dropna(how=\"any\")\n\n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n\n # TODO: Try different models here!\n model = LGBMRegressor(n_estimators=10)\n model.fit(X, y)\n return X, y, model\nCheck this code out. it's mainly training the model for each id, and the get_feature function gives some columns like 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP' with some new feature engineered columns. If I train on only the train.csv then what would be the problem? I don't get it. But I have seen some Nbs where someone noticed the author did not include the assets.csv and pointed out as a but (check out here) and the public score is also zero.
I have commented under some NBs and topics, If they answer, I will definitely update this thread.
\nI will try to run the code for the whole train.csv and see the score I get.
can you help @odins0n ?
", "votes": null, "replies": [] }, { "id": 1573058, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 08:35:52", "content": "People may update on this too,
\n", "votes": null, "replies": [] }, { "id": 1573215, "author_name": "odins0n", "author_url": "", "post_date": "11/06/2021 11:57:20", "content": "@soumya9977 Every asset(cryptocurrency) has a different price, the volume traded and starting time from which records are given in the train.csv. In this notebook that u mentioned the author has trained the same model for every asset separately and later predicted on test.csv for each asset from their saved model respectively. I think training each asset separately might get good results due to the above factors but at the same time, we will be unable to get the benefit of the influence of price change on one asset due to the price change in another.
\nI think we can try and train the train.csv file directly and check what results we get.
", "votes": null, "replies": [] }, { "id": 1573330, "author_name": "soumya9977", "author_url": "", "post_date": "11/06/2021 14:28:12", "content": "I totally get your point. Both of the ways have their own pros and cons. And as you said trying out both of them, sounds like is a good idea. Thank you for answering 😇.
", "votes": null, "replies": [] } ] }, { "id": 1573078, "author_name": "", "author_url": "", "post_date": "11/06/2021 08:50:14", "content": "Yes! See my post here
", "votes": null, "replies": [] }, { "id": 1573290, "author_name": "kagglerrr1", "author_url": "", "post_date": "11/06/2021 13:20:31", "content": "@odinson Yes a dataset with all assets included together will be useful. Because I think the data is till September only. Can you make a dataset public so that it can be appended in the train.csv ?
", "votes": null, "replies": [ { "id": 1573767, "author_name": "odins0n", "author_url": "", "post_date": "11/06/2021 22:21:16", "content": "Sure @kagglerrr1 !!! I will try and publish a dataset in the same format as train.csv
\nI will update this discussion once it's done.
I too have started to do this. See post here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913 and relevant notebooks / datasets:
\nGetting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:
\nraw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance
\nTransforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data
\npreprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance
I still have some configuration / automation parts to figure.
", "votes": null, "replies": [ { "id": 1575026, "author_name": "odins0n", "author_url": "", "post_date": "11/08/2021 05:30:35", "content": "This will be so handy @lucasmorin. Thanks !!!
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "1572865": "Can we make a public dataset with daily frequency updations for the recent few months that are not in the dataset and treat it as data for evaluating our model's performance?", "1572927": "I don know you have seen these topics before or not but here are `Topics`, check them out, \n1. [External \"Elon Musk's Tweets\" dataset allowed? 😂](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284913)\n2. [Are we allowed to collect market data during these three months to extend the train set?](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/284942) \n\nBut in short, Yes, \n`Freely & publicly available external data is allowed, including pre-trained models`. \nand that has to be offline data because the submissions don't allow online NB", "1572936": "Thanks for the info. I had not seen the second topic.", "1572957": "No problem, Check this out to add more data\n[Extra Data - Auto updating dataset & Baseline with CV 💎🤲](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726)", "1573027": "Hi @soumya9977 This notebook was good but have could been be better if a single dataset had all the assets included together.", "1573055": "Yeah, I was thinking the same you know, I was trying to implement something but it requires only a `train.csv` and a `test.csv`, no train test split or something. I also saw some notebooks(I'd say most of the NBs) where they give inputs as a form of a combination of `asset.csv and train.csv`. for example, \n```python\ndef get_Xy_and_model_for_asset(df_train, asset_id):\n df = df_train[df_train[\"Asset_ID\"] == asset_id]\n \n # TODO: Try different features here!\n df_proc = get_features(df)\n df_proc['y'] = df['Target']\n df_proc = df_proc.replace([np.inf, -np.inf], np.nan).dropna(how=\"any\")\n \n X = df_proc.drop(\"y\", axis=1)\n y = df_proc[\"y\"]\n\n # TODO: Try different models here!\n model = LGBMRegressor(n_estimators=10)\n model.fit(X, y)\n return X, y, model\n```\n[Borrowed code source](https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017)\n\n\nCheck this code out. it's mainly training the model for each id, and the get_feature function gives some columns like 'Count', 'Open', 'High', 'Low', 'Close', 'Volume', 'VWAP' with some new feature engineered columns. If I train on only the train.csv then what would be the problem? I don't get it. But I have seen some Nbs where someone noticed the author did not include the `assets.csv` and pointed out as a but ([check out here](https://www.kaggle.com/metathesis/torch-lstm-baseline-training-updated/comments#1571111)) and the public score is also zero. \n\nI have commented under some NBs and topics, If they answer, I will definitely update this thread. \nI will try to run the code for the whole `train.csv` and see the score I get.\n\ncan you help @odins0n ?", "1573058": "People may update on this too,\n- [Meaning of weights ?](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285183)", "1573078": "Yes! See my post [here](https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285726)", "1573215": "soumya9977 Every asset(cryptocurrency) has a different price, the volume traded and starting time from which records are given in the train.csv. In [this](https://www.kaggle.com/julian3833/s-g-research-strict-lgbm-example-lb-0-017) notebook that u mentioned the author has trained the same model for every asset separately and later predicted on test.csv for each asset from their saved model respectively. I think training each asset separately might get good results due to the above factors but at the same time, we will be unable to get the benefit of the influence of price change on one asset due to the price change in another.\n\nI think we can try and train the train.csv file directly and check what results we get.", "1573290": "odinson Yes a dataset with all assets included together will be useful. Because I think the data is till September only. Can you make a dataset public so that it can be appended in the train.csv ?", "1573330": "I totally get your point. Both of the ways have their own pros and cons. And as you said trying out both of them, sounds like is a good idea. Thank you for answering 😇.", "1573767": "Sure @kagglerrr1 !!! I will try and publish a dataset in the same format as train.csv\nI will update this discussion once it's done.", "1574802": "I too have started to do this. See post here: https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/285913 and relevant notebooks / datasets:\n\nGetting all 1m data from Binance: https://www.kaggle.com/lucasmorin/getting-all-1m-data-from-binance:\nraw output is available in this dataset: https://www.kaggle.com/lucasmorin/raw-crypto-1m-data-from-binance\nTransforming 1m data From Binance: https://www.kaggle.com/lucasmorin/transforming-1m-data-from-binance/data\npreprocessed output is available in this dataset: https://www.kaggle.com/lucasmorin/transformed-1m-data-from-binance\n\nI still have some configuration / automation parts to figure.", "1575026": "This will be so handy @lucasmorin. Thanks !!!" }, "source": "meta" }