{
  "id": 285499,
  "title": "Loading gresearch_crypto from Jupyter on Windows",
  "url": "/competitions/g-research-crypto-forecasting/discussion/285499",
  "author_name": "",
  "post_date": "2021-11-04T23:19:28.751269400Z",
  "votes": 8,
  "comment_count": 5,
  "views": 0,
  "content": "<p>Hello,</p>\n<p>I'm trying to load the gresearch_crypto on a Jupyter Notebook running Python 3.7 on a Windows machine. However, I get:<br>\nModuleNotFoundError: No module named 'gresearch_crypto.competition'</p>\n<p>Probably because file competition.cpython-37m-x86_64-linux-gnu isn't loadable on Windows.<br>\nIs there a workaround or can the organizers provide a workaround ?</p>\n<p>Thanks,</p>\n<p>Jose</p>",
  "messages": [
    {
      "id": "1571496",
      "postDate": "11/04/2021 23:19:28",
      "content": "<p>Hello,</p>\n<p>I'm trying to load the gresearch_crypto on a Jupyter Notebook running Python 3.7 on a Windows machine. However, I get:<br>\nModuleNotFoundError: No module named 'gresearch_crypto.competition'</p>\n<p>Probably because file competition.cpython-37m-x86_64-linux-gnu isn't loadable on Windows.<br>\nIs there a workaround or can the organizers provide a workaround ?</p>\n<p>Thanks,</p>\n<p>Jose</p>",
      "rawMarkdown": "Hello,\n\nI'm trying to load the gresearch_crypto on a Jupyter Notebook running Python 3.7 on a Windows machine. However, I get:\nModuleNotFoundError: No module named 'gresearch_crypto.competition'\n\nProbably because file competition.cpython-37m-x86_64-linux-gnu isn't loadable on Windows.\nIs there a workaround or can the organizers provide a workaround ?\n\nThanks,\n\nJose",
      "votes": null
    },
    {
      "id": "1571511",
      "postDate": "11/04/2021 23:56:43",
      "content": "<p>Enter the Code section of the competition. Press \"New Notebook\", and use this notebook to run codes.</p>",
      "rawMarkdown": "Enter the Code section of the competition. Press \"New Notebook\", and use this notebook to run codes.",
      "votes": null
    },
    {
      "id": "1572644",
      "postDate": "11/05/2021 20:31:27",
      "content": "<p>I like to work offline as well - would be great if you could make that available for other OSs as well.</p>",
      "rawMarkdown": "I like to work offline as well - would be great if you could make that available for other OSs as well.",
      "votes": null
    },
    {
      "id": "1598597",
      "postDate": "11/28/2021 16:07:00",
      "content": "<p>You can try this <a href=\"https://www.kaggle.com/jcsantos\" target=\"_blank\">@jcsantos</a> <a href=\"https://www.kaggle.com/mgferreira\" target=\"_blank\">@mgferreira</a> <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299\" target=\"_blank\">https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299</a></p>",
      "rawMarkdown": "You can try this @jcsantos @mgferreira https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299",
      "votes": null
    },
    {
      "id": "1598782",
      "postDate": "11/28/2021 20:48:41",
      "content": "<p>Hello,<br>\nI also like to work offline to test ideas, and I'm also using Windows as a working environment…<br>\nFor this competition, I implemented some code that replicates the behavior of the \"gresearch_crypto\" module. Basically, it uses the data in \"supplemental_train.csv\" to populate the dataframes that are returned by the \"iter_test\" iterator.<br>\nI'm using this code as follows:</p>\n<pre><code>try:\n    import gresearch_crypto\nexcept:\n    FILL_MISSING_TIMESTAMPS = False\n\n    class MY_env:\n        def __init__(self, df_supplemental_train):\n            self.df_supplemental_train = df_supplemental_train\n            self.all_timestamps = self.df_supplemental_train.timestamp.unique()\n            self.all_timestamps.sort()\n            if FILL_MISSING_TIMESTAMPS:\n                # Update list to include timestamps with no data (3 instances in supplemental_train data...)\n                self.all_timestamps = np.arange(self.all_timestamps[0], self.all_timestamps[-1]+60, 60)\n            self.waiting_for_prediction = False\n\n        class TestIterator:\n            def __init__(self, env):\n                self.env = env\n                self.group_num = 0  # Incremented after each call to 'iter_test()'\n                self.last_row_id = 0\n\n            def __iter__(self):\n                return self\n\n            def __next__(self):\n                \"\"\"\n                ATTENTION : THIS METHOD SHOULD RETURN DATAFRAMES WITH POSSIBLE MISSING ENTRIES (to reproduce actual behavior!)\n                THEREFORE: filling missing values should NOT be performed here\n                \"\"\"\n                if self.group_num &lt; len(self.env.all_timestamps):\n                    if self.env.waiting_for_prediction:\n                        raise Exception(\"You must call `predict()` successfully before you can continue with `iter_test()`\");\n                    timestamp = self.env.all_timestamps[self.group_num]\n                    t_df = self.env.df_supplemental_train[self.env.df_supplemental_train.timestamp == timestamp].copy()\n                    t_df['group_num'] = self.group_num\n                    row_ids = np.arange(len(t_df)) + self.last_row_id\n                    t_df['row_id'] = row_ids\n\n                    pred_df = t_df[['group_num', 'row_id']].copy()\n                    pred_df['Target'] = 0\n\n                    self.env.waiting_for_prediction = True\n\n                    self.group_num += 1\n                    self.last_row_id += len(t_df)\n                    return t_df, pred_df\n                raise StopIteration\n\n        def iter_test(self):\n            return self.TestIterator(self)\n\n        def predict(self, prediction):\n            self.waiting_for_prediction = False\n\n\n    class MY_crypto_proxy:\n        def __init__(self):\n            print('Loading supplemental_train data to be used for test phase')\n            self.df_supplemental_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv', dtype=all_dtypes)\n            self.df_supplemental_train = self.df_supplemental_train.drop(columns=['Target'])\n            self.df_supplemental_train = reduce_mem_usage(self.df_supplemental_train)\n\n        def make_env(self):\n            return MY_env(self.df_supplemental_train)\n\n    print('No module \"gresearch_crypto\"')\n    if os.environ.get('OS', None) == 'Windows_NT':\n        print('Running in local Windows environment ==&gt; Use \"supplemental_train.csv\" data as test data. Train (including validation) data is composed only of train data that is not included in supplemental_train')\n    else:\n        raise Exception('Custom environment MUST only be used when running in local Windows')\n    gresearch_crypto = MY_crypto_proxy()\n</code></pre>\n<p>It is initialized as with the provided module:</p>\n<pre><code>env = gresearch_crypto.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()\n</code></pre>\n<p>The only assumption that this code makes is that all records in the dataframes that are returned by \"iter_test\" have the same timestamp, which seems reasonable to me.<br>\nHope this help!</p>",
      "rawMarkdown": "Hello,\nI also like to work offline to test ideas, and I'm also using Windows as a working environment...\nFor this competition, I implemented some code that replicates the behavior of the \"gresearch_crypto\" module. Basically, it uses the data in \"supplemental_train.csv\" to populate the dataframes that are returned by the \"iter_test\" iterator.\nI'm using this code as follows:\n\n```\ntry:\n    import gresearch_crypto\nexcept:\n    FILL_MISSING_TIMESTAMPS = False\n    \n    class MY_env:\n        def __init__(self, df_supplemental_train):\n            self.df_supplemental_train = df_supplemental_train\n            self.all_timestamps = self.df_supplemental_train.timestamp.unique()\n            self.all_timestamps.sort()\n            if FILL_MISSING_TIMESTAMPS:\n                # Update list to include timestamps with no data (3 instances in supplemental_train data...)\n                self.all_timestamps = np.arange(self.all_timestamps[0], self.all_timestamps[-1]+60, 60)\n            self.waiting_for_prediction = False\n\n        class TestIterator:\n            def __init__(self, env):\n                self.env = env\n                self.group_num = 0  # Incremented after each call to 'iter_test()'\n                self.last_row_id = 0\n\n            def __iter__(self):\n                return self\n\n            def __next__(self):\n                \"\"\"\n                ATTENTION : THIS METHOD SHOULD RETURN DATAFRAMES WITH POSSIBLE MISSING ENTRIES (to reproduce actual behavior!)\n                THEREFORE: filling missing values should NOT be performed here\n                \"\"\"\n                if self.group_num < len(self.env.all_timestamps):\n                    if self.env.waiting_for_prediction:\n                        raise Exception(\"You must call `predict()` successfully before you can continue with `iter_test()`\");\n                    timestamp = self.env.all_timestamps[self.group_num]\n                    t_df = self.env.df_supplemental_train[self.env.df_supplemental_train.timestamp == timestamp].copy()\n                    t_df['group_num'] = self.group_num\n                    row_ids = np.arange(len(t_df)) + self.last_row_id\n                    t_df['row_id'] = row_ids\n\n                    pred_df = t_df[['group_num', 'row_id']].copy()\n                    pred_df['Target'] = 0\n\n                    self.env.waiting_for_prediction = True\n\n                    self.group_num += 1\n                    self.last_row_id += len(t_df)\n                    return t_df, pred_df\n                raise StopIteration\n\n        def iter_test(self):\n            return self.TestIterator(self)\n        \n        def predict(self, prediction):\n            self.waiting_for_prediction = False\n            \n\n    class MY_crypto_proxy:\n        def __init__(self):\n            print('Loading supplemental_train data to be used for test phase')\n            self.df_supplemental_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv', dtype=all_dtypes)\n            self.df_supplemental_train = self.df_supplemental_train.drop(columns=['Target'])\n            self.df_supplemental_train = reduce_mem_usage(self.df_supplemental_train)\n            \n        def make_env(self):\n            return MY_env(self.df_supplemental_train)\n\n    print('No module \"gresearch_crypto\"')\n    if os.environ.get('OS', None) == 'Windows_NT':\n        print('Running in local Windows environment ==> Use \"supplemental_train.csv\" data as test data. Train (including validation) data is composed only of train data that is not included in supplemental_train')\n    else:\n        raise Exception('Custom environment MUST only be used when running in local Windows')\n    gresearch_crypto = MY_crypto_proxy()\n```\nIt is initialized as with the provided module:\n```\nenv = gresearch_crypto.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()\n```\nThe only assumption that this code makes is that all records in the dataframes that are returned by \"iter_test\" have the same timestamp, which seems reasonable to me.\nHope this help!",
      "votes": null
    },
    {
      "id": "1622450",
      "postDate": "12/18/2021 18:40:59",
      "content": "<p>You know - people like you make the world a better place.<br>\nThanks this really helped me a lot.</p>",
      "rawMarkdown": "You know - people like you make the world a better place.\nThanks this really helped me a lot.",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 1571511,
      "author_name": "yus002",
      "author_url": "",
      "post_date": "11/04/2021 23:56:43",
      "content": "<p>Enter the Code section of the competition. Press \"New Notebook\", and use this notebook to run codes.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1572644,
      "author_name": "mgferreira",
      "author_url": "",
      "post_date": "11/05/2021 20:31:27",
      "content": "<p>I like to work offline as well - would be great if you could make that available for other OSs as well.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1598597,
      "author_name": "isaac91",
      "author_url": "",
      "post_date": "11/28/2021 16:07:00",
      "content": "<p>You can try this <a href=\"https://www.kaggle.com/jcsantos\" target=\"_blank\">@jcsantos</a> <a href=\"https://www.kaggle.com/mgferreira\" target=\"_blank\">@mgferreira</a> <a href=\"https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299\" target=\"_blank\">https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 1598782,
      "author_name": "seuguh",
      "author_url": "",
      "post_date": "11/28/2021 20:48:41",
      "content": "<p>Hello,<br>\nI also like to work offline to test ideas, and I'm also using Windows as a working environment…<br>\nFor this competition, I implemented some code that replicates the behavior of the \"gresearch_crypto\" module. Basically, it uses the data in \"supplemental_train.csv\" to populate the dataframes that are returned by the \"iter_test\" iterator.<br>\nI'm using this code as follows:</p>\n<pre><code>try:\n    import gresearch_crypto\nexcept:\n    FILL_MISSING_TIMESTAMPS = False\n\n    class MY_env:\n        def __init__(self, df_supplemental_train):\n            self.df_supplemental_train = df_supplemental_train\n            self.all_timestamps = self.df_supplemental_train.timestamp.unique()\n            self.all_timestamps.sort()\n            if FILL_MISSING_TIMESTAMPS:\n                # Update list to include timestamps with no data (3 instances in supplemental_train data...)\n                self.all_timestamps = np.arange(self.all_timestamps[0], self.all_timestamps[-1]+60, 60)\n            self.waiting_for_prediction = False\n\n        class TestIterator:\n            def __init__(self, env):\n                self.env = env\n                self.group_num = 0  # Incremented after each call to 'iter_test()'\n                self.last_row_id = 0\n\n            def __iter__(self):\n                return self\n\n            def __next__(self):\n                \"\"\"\n                ATTENTION : THIS METHOD SHOULD RETURN DATAFRAMES WITH POSSIBLE MISSING ENTRIES (to reproduce actual behavior!)\n                THEREFORE: filling missing values should NOT be performed here\n                \"\"\"\n                if self.group_num &lt; len(self.env.all_timestamps):\n                    if self.env.waiting_for_prediction:\n                        raise Exception(\"You must call `predict()` successfully before you can continue with `iter_test()`\");\n                    timestamp = self.env.all_timestamps[self.group_num]\n                    t_df = self.env.df_supplemental_train[self.env.df_supplemental_train.timestamp == timestamp].copy()\n                    t_df['group_num'] = self.group_num\n                    row_ids = np.arange(len(t_df)) + self.last_row_id\n                    t_df['row_id'] = row_ids\n\n                    pred_df = t_df[['group_num', 'row_id']].copy()\n                    pred_df['Target'] = 0\n\n                    self.env.waiting_for_prediction = True\n\n                    self.group_num += 1\n                    self.last_row_id += len(t_df)\n                    return t_df, pred_df\n                raise StopIteration\n\n        def iter_test(self):\n            return self.TestIterator(self)\n\n        def predict(self, prediction):\n            self.waiting_for_prediction = False\n\n\n    class MY_crypto_proxy:\n        def __init__(self):\n            print('Loading supplemental_train data to be used for test phase')\n            self.df_supplemental_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv', dtype=all_dtypes)\n            self.df_supplemental_train = self.df_supplemental_train.drop(columns=['Target'])\n            self.df_supplemental_train = reduce_mem_usage(self.df_supplemental_train)\n\n        def make_env(self):\n            return MY_env(self.df_supplemental_train)\n\n    print('No module \"gresearch_crypto\"')\n    if os.environ.get('OS', None) == 'Windows_NT':\n        print('Running in local Windows environment ==&gt; Use \"supplemental_train.csv\" data as test data. Train (including validation) data is composed only of train data that is not included in supplemental_train')\n    else:\n        raise Exception('Custom environment MUST only be used when running in local Windows')\n    gresearch_crypto = MY_crypto_proxy()\n</code></pre>\n<p>It is initialized as with the provided module:</p>\n<pre><code>env = gresearch_crypto.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()\n</code></pre>\n<p>The only assumption that this code makes is that all records in the dataframes that are returned by \"iter_test\" have the same timestamp, which seems reasonable to me.<br>\nHope this help!</p>",
      "votes": null,
      "replies": [
        {
          "id": 1622450,
          "author_name": "mgferreira",
          "author_url": "",
          "post_date": "12/18/2021 18:40:59",
          "content": "<p>You know - people like you make the world a better place.<br>\nThanks this really helped me a lot.</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "1571496": "Hello,\n\nI'm trying to load the gresearch_crypto on a Jupyter Notebook running Python 3.7 on a Windows machine. However, I get:\nModuleNotFoundError: No module named 'gresearch_crypto.competition'\n\nProbably because file competition.cpython-37m-x86_64-linux-gnu isn't loadable on Windows.\nIs there a workaround or can the organizers provide a workaround ?\n\nThanks,\n\nJose",
    "1571511": "Enter the Code section of the competition. Press \"New Notebook\", and use this notebook to run codes.",
    "1572644": "I like to work offline as well - would be great if you could make that available for other OSs as well.",
    "1598597": "You can try this @jcsantos @mgferreira https://www.kaggle.com/c/g-research-crypto-forecasting/discussion/291299",
    "1598782": "Hello,\nI also like to work offline to test ideas, and I'm also using Windows as a working environment...\nFor this competition, I implemented some code that replicates the behavior of the \"gresearch_crypto\" module. Basically, it uses the data in \"supplemental_train.csv\" to populate the dataframes that are returned by the \"iter_test\" iterator.\nI'm using this code as follows:\n\n```\ntry:\n    import gresearch_crypto\nexcept:\n    FILL_MISSING_TIMESTAMPS = False\n    \n    class MY_env:\n        def __init__(self, df_supplemental_train):\n            self.df_supplemental_train = df_supplemental_train\n            self.all_timestamps = self.df_supplemental_train.timestamp.unique()\n            self.all_timestamps.sort()\n            if FILL_MISSING_TIMESTAMPS:\n                # Update list to include timestamps with no data (3 instances in supplemental_train data...)\n                self.all_timestamps = np.arange(self.all_timestamps[0], self.all_timestamps[-1]+60, 60)\n            self.waiting_for_prediction = False\n\n        class TestIterator:\n            def __init__(self, env):\n                self.env = env\n                self.group_num = 0  # Incremented after each call to 'iter_test()'\n                self.last_row_id = 0\n\n            def __iter__(self):\n                return self\n\n            def __next__(self):\n                \"\"\"\n                ATTENTION : THIS METHOD SHOULD RETURN DATAFRAMES WITH POSSIBLE MISSING ENTRIES (to reproduce actual behavior!)\n                THEREFORE: filling missing values should NOT be performed here\n                \"\"\"\n                if self.group_num < len(self.env.all_timestamps):\n                    if self.env.waiting_for_prediction:\n                        raise Exception(\"You must call `predict()` successfully before you can continue with `iter_test()`\");\n                    timestamp = self.env.all_timestamps[self.group_num]\n                    t_df = self.env.df_supplemental_train[self.env.df_supplemental_train.timestamp == timestamp].copy()\n                    t_df['group_num'] = self.group_num\n                    row_ids = np.arange(len(t_df)) + self.last_row_id\n                    t_df['row_id'] = row_ids\n\n                    pred_df = t_df[['group_num', 'row_id']].copy()\n                    pred_df['Target'] = 0\n\n                    self.env.waiting_for_prediction = True\n\n                    self.group_num += 1\n                    self.last_row_id += len(t_df)\n                    return t_df, pred_df\n                raise StopIteration\n\n        def iter_test(self):\n            return self.TestIterator(self)\n        \n        def predict(self, prediction):\n            self.waiting_for_prediction = False\n            \n\n    class MY_crypto_proxy:\n        def __init__(self):\n            print('Loading supplemental_train data to be used for test phase')\n            self.df_supplemental_train = pd.read_csv('../input/g-research-crypto-forecasting/supplemental_train.csv', dtype=all_dtypes)\n            self.df_supplemental_train = self.df_supplemental_train.drop(columns=['Target'])\n            self.df_supplemental_train = reduce_mem_usage(self.df_supplemental_train)\n            \n        def make_env(self):\n            return MY_env(self.df_supplemental_train)\n\n    print('No module \"gresearch_crypto\"')\n    if os.environ.get('OS', None) == 'Windows_NT':\n        print('Running in local Windows environment ==> Use \"supplemental_train.csv\" data as test data. Train (including validation) data is composed only of train data that is not included in supplemental_train')\n    else:\n        raise Exception('Custom environment MUST only be used when running in local Windows')\n    gresearch_crypto = MY_crypto_proxy()\n```\nIt is initialized as with the provided module:\n```\nenv = gresearch_crypto.make_env()\n\n# You can only iterate through a result from `env.iter_test()` once\n# so be careful not to lose it once you start iterating.\niter_test = env.iter_test()\n```\nThe only assumption that this code makes is that all records in the dataframes that are returned by \"iter_test\" have the same timestamp, which seems reasonable to me.\nHope this help!",
    "1622450": "You know - people like you make the world a better place.\nThanks this really helped me a lot."
  },
  "source": "meta"
}