{
  "id": 68162,
  "title": "How do you store the features?",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/68162",
  "author_name": "",
  "post_date": "2018-10-10T01:32:34.320324700Z",
  "votes": 3,
  "comment_count": 6,
  "views": 0,
  "content": "<p>During this competition, I've stored the features in one csv file when I updated new features.</p>\n\n<p>So, for now, I have 10 CSV files, but there are many duplicates in those different CSV files.</p>\n\n<p>I think that there will be an efficient way to store all features.</p>\n\n<p>I think below two strategies to store features, avoiding duplicates.\n(1) one feature - one csv file\n(2) group features - one csv file</p>\n\n<p>Please give me a wisdom.  I want to know how the kagglers do.  Thanks!</p>",
  "messages": [
    {
      "id": "401383",
      "postDate": "10/10/2018 01:32:34",
      "content": "<p>During this competition, I've stored the features in one csv file when I updated new features.</p>\n\n<p>So, for now, I have 10 CSV files, but there are many duplicates in those different CSV files.</p>\n\n<p>I think that there will be an efficient way to store all features.</p>\n\n<p>I think below two strategies to store features, avoiding duplicates.\n(1) one feature - one csv file\n(2) group features - one csv file</p>\n\n<p>Please give me a wisdom.  I want to know how the kagglers do.  Thanks!</p>",
      "rawMarkdown": "During this competition, I've stored the features in one csv file when I updated new features.\n\nSo, for now, I have 10 CSV files, but there are many duplicates in those different CSV files.\n\nI think that there will be an efficient way to store all features.\n\nI think below two strategies to store features, avoiding duplicates.\n(1) one feature - one csv file\n(2) group features - one csv file\n\nPlease give me a wisdom.  I want to know how the kagglers do.  Thanks!",
      "votes": null
    },
    {
      "id": "401390",
      "postDate": "10/10/2018 02:07:30",
      "content": "<p>I try to split feature generation across multiple scripts with self-contained themes. For example, one script might create visitor features while another might create everything derived directly from date/time information. The new feature outputs get saved to disk in the same order as the original data. So I end up with multiple files of non-overlapping features that share a unified order. This accomplishes 2 things for me:</p>\n\n<ol>\n<li>It's easy to selectively choose and quickly combine different feature sets when I build models</li>\n<li>Minimizes unnecessary compute time. When I add new features or modify existing feature sets, I create/modify and run a relatively small code block instead of re-running my entire feature pipeline. This is much faster than repeatedly looping through some expensive aggregation functions, etc.  </li>\n</ol>\n\n<p>I've seen people mention using 1 file per feature, but personally I find that a bit too specific. </p>\n\n<p>Since this dataset isn't too large csv format should work ok, but in general I'd recommend using a serialized file format like pickle or feather. Those are much faster for read/write and also take up less disk space.</p>",
      "rawMarkdown": "I try to split feature generation across multiple scripts with self-contained themes. For example, one script might create visitor features while another might create everything derived directly from date/time information. The new feature outputs get saved to disk in the same order as the original data. So I end up with multiple files of non-overlapping features that share a unified order. This accomplishes 2 things for me:\n\n 1. It's easy to selectively choose and quickly combine different feature sets when I build models\n 2. Minimizes unnecessary compute time. When I add new features or modify existing feature sets, I create/modify and run a relatively small code block instead of re-running my entire feature pipeline. This is much faster than repeatedly looping through some expensive aggregation functions, etc.  \n\nI've seen people mention using 1 file per feature, but personally I find that a bit too specific. \n\nSince this dataset isn't too large csv format should work ok, but in general I'd recommend using a serialized file format like pickle or feather. Those are much faster for read/write and also take up less disk space.",
      "votes": null
    },
    {
      "id": "401437",
      "postDate": "10/10/2018 04:42:32",
      "content": "<p>2 is very important. I finally realized this when I saw your time lag feature scripts. (did not think about this, partly because the current dataset is relatively small.  </p>\n\n<p>for easy exploration of dataset, I also store the features in sqlite and explore them use SQL. maybe I should use sqlite as primary storage? or some column oriented database.</p>",
      "rawMarkdown": "2 is very important. I finally realized this when I saw your time lag feature scripts. (did not think about this, partly because the current dataset is relatively small.  \n\nfor easy exploration of dataset, I also store the features in sqlite and explore them use SQL. maybe I should use sqlite as primary storage? or some column oriented database.",
      "votes": null
    },
    {
      "id": "401441",
      "postDate": "10/10/2018 04:50:06",
      "content": "<ul>\n<li>store a type of features per file;</li>\n<li>only store features that are time-consuming when creating them, and store generation script for those non time-consuming features</li>\n</ul>",
      "rawMarkdown": "*  store a type of features per file;\n * only store features that are time-consuming when creating them, and store generation script for those non time-consuming features",
      "votes": null
    },
    {
      "id": "401497",
      "postDate": "10/10/2018 07:10:31",
      "content": "<p>Thanks for the very kind explanation and advice!</p>\n\n<p>You pointed very important thing, 2. Actually, I've concerned about efficiency and computational cost.</p>\n\n<p>After merging your advice and my thought, I make script below. How about that?</p>\n\n<pre><code>feature_name = ['feat1', 'feat2']\ndef generate_features(data, name):\n    existing_feature_list = os.listdir('../features/')\n    for temp_feat in feature_name:\n        new_feat_name = 'operation_{}'.format(temp_feat)\n        csv_file = '{}.csv'.format(new_feat_name)\n\n        if csv_file in existing_feature_list:\n            data[new_feat_name] = pd.read_csv('../features/{}'.format(csv_file))\n        else:\n# generate new features (mean encoding, product, substract, etc)\n            new_feat = operation()        \n            data[new_feat_name] = new_feat\n            new_feat.to_csv('../features/{}'.format(csv_file))\n    return data \n</code></pre>\n\n<p>Csv can be replaced with other format.</p>",
      "rawMarkdown": "Thanks for the very kind explanation and advice!\n\nYou pointed very important thing, 2. Actually, I've concerned about efficiency and computational cost.\n\nAfter merging your advice and my thought, I make script below. How about that?\n\n    feature_name = ['feat1', 'feat2']\n    def generate_features(data, name):\n        existing_feature_list = os.listdir('../features/')\n        for temp_feat in feature_name:\n            new_feat_name = 'operation_{}'.format(temp_feat)\n            csv_file = '{}.csv'.format(new_feat_name)\n            \n            if csv_file in existing_feature_list:\n                data[new_feat_name] = pd.read_csv('../features/{}'.format(csv_file))\n            else:\n    # generate new features (mean encoding, product, substract, etc)\n                new_feat = operation()        \n                data[new_feat_name] = new_feat\n                new_feat.to_csv('../features/{}'.format(csv_file))\n        return data \n\nCsv can be replaced with other format.",
      "votes": null
    },
    {
      "id": "401501",
      "postDate": "10/10/2018 07:15:52",
      "content": "<p>You mean that make a meta-file including the information of features?</p>",
      "rawMarkdown": "You mean that make a meta-file including the information of features?",
      "votes": null
    },
    {
      "id": "401550",
      "postDate": "10/10/2018 09:01:10",
      "content": "<p>Yes, something like that. For example,  since creating frequency and mean encoding may cost some time, you can create them once, and store in a file. But creating simple  statistic like time transformation, ratio or simple groupby operations( mean, max, min, .etc) will not cost too much time, so just running generation scripts on meta-data to create them when modeling ( if there are too many features, IO duration may longer than running generation scripts ).</p>",
      "rawMarkdown": "Yes, something like that. For example,  since creating frequency and mean encoding may cost some time, you can create them once, and store in a file. But creating simple  statistic like time transformation, ratio or simple groupby operations( mean, max, min, .etc) will not cost too much time, so just running generation scripts on meta-data to create them when modeling ( if there are too many features, IO duration may longer than running generation scripts ).",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 401390,
      "author_name": "aquatic",
      "author_url": "",
      "post_date": "10/10/2018 02:07:30",
      "content": "<p>I try to split feature generation across multiple scripts with self-contained themes. For example, one script might create visitor features while another might create everything derived directly from date/time information. The new feature outputs get saved to disk in the same order as the original data. So I end up with multiple files of non-overlapping features that share a unified order. This accomplishes 2 things for me:</p>\n\n<ol>\n<li>It's easy to selectively choose and quickly combine different feature sets when I build models</li>\n<li>Minimizes unnecessary compute time. When I add new features or modify existing feature sets, I create/modify and run a relatively small code block instead of re-running my entire feature pipeline. This is much faster than repeatedly looping through some expensive aggregation functions, etc.  </li>\n</ol>\n\n<p>I've seen people mention using 1 file per feature, but personally I find that a bit too specific. </p>\n\n<p>Since this dataset isn't too large csv format should work ok, but in general I'd recommend using a serialized file format like pickle or feather. Those are much faster for read/write and also take up less disk space.</p>",
      "votes": null,
      "replies": [
        {
          "id": 401437,
          "author_name": "echowuhao",
          "author_url": "",
          "post_date": "10/10/2018 04:42:32",
          "content": "<p>2 is very important. I finally realized this when I saw your time lag feature scripts. (did not think about this, partly because the current dataset is relatively small.  </p>\n\n<p>for easy exploration of dataset, I also store the features in sqlite and explore them use SQL. maybe I should use sqlite as primary storage? or some column oriented database.</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 401497,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/10/2018 07:10:31",
          "content": "<p>Thanks for the very kind explanation and advice!</p>\n\n<p>You pointed very important thing, 2. Actually, I've concerned about efficiency and computational cost.</p>\n\n<p>After merging your advice and my thought, I make script below. How about that?</p>\n\n<pre><code>feature_name = ['feat1', 'feat2']\ndef generate_features(data, name):\n    existing_feature_list = os.listdir('../features/')\n    for temp_feat in feature_name:\n        new_feat_name = 'operation_{}'.format(temp_feat)\n        csv_file = '{}.csv'.format(new_feat_name)\n\n        if csv_file in existing_feature_list:\n            data[new_feat_name] = pd.read_csv('../features/{}'.format(csv_file))\n        else:\n# generate new features (mean encoding, product, substract, etc)\n            new_feat = operation()        \n            data[new_feat_name] = new_feat\n            new_feat.to_csv('../features/{}'.format(csv_file))\n    return data \n</code></pre>\n\n<p>Csv can be replaced with other format.</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 401441,
      "author_name": "chenshaomeng",
      "author_url": "",
      "post_date": "10/10/2018 04:50:06",
      "content": "<ul>\n<li>store a type of features per file;</li>\n<li>only store features that are time-consuming when creating them, and store generation script for those non time-consuming features</li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 401501,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/10/2018 07:15:52",
          "content": "<p>You mean that make a meta-file including the information of features?</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 401550,
          "author_name": "chenshaomeng",
          "author_url": "",
          "post_date": "10/10/2018 09:01:10",
          "content": "<p>Yes, something like that. For example,  since creating frequency and mean encoding may cost some time, you can create them once, and store in a file. But creating simple  statistic like time transformation, ratio or simple groupby operations( mean, max, min, .etc) will not cost too much time, so just running generation scripts on meta-data to create them when modeling ( if there are too many features, IO duration may longer than running generation scripts ).</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "401383": "During this competition, I've stored the features in one csv file when I updated new features.\n\nSo, for now, I have 10 CSV files, but there are many duplicates in those different CSV files.\n\nI think that there will be an efficient way to store all features.\n\nI think below two strategies to store features, avoiding duplicates.\n(1) one feature - one csv file\n(2) group features - one csv file\n\nPlease give me a wisdom.  I want to know how the kagglers do.  Thanks!",
    "401390": "I try to split feature generation across multiple scripts with self-contained themes. For example, one script might create visitor features while another might create everything derived directly from date/time information. The new feature outputs get saved to disk in the same order as the original data. So I end up with multiple files of non-overlapping features that share a unified order. This accomplishes 2 things for me:\n\n 1. It's easy to selectively choose and quickly combine different feature sets when I build models\n 2. Minimizes unnecessary compute time. When I add new features or modify existing feature sets, I create/modify and run a relatively small code block instead of re-running my entire feature pipeline. This is much faster than repeatedly looping through some expensive aggregation functions, etc.  \n\nI've seen people mention using 1 file per feature, but personally I find that a bit too specific. \n\nSince this dataset isn't too large csv format should work ok, but in general I'd recommend using a serialized file format like pickle or feather. Those are much faster for read/write and also take up less disk space.",
    "401437": "2 is very important. I finally realized this when I saw your time lag feature scripts. (did not think about this, partly because the current dataset is relatively small.  \n\nfor easy exploration of dataset, I also store the features in sqlite and explore them use SQL. maybe I should use sqlite as primary storage? or some column oriented database.",
    "401441": "*  store a type of features per file;\n * only store features that are time-consuming when creating them, and store generation script for those non time-consuming features",
    "401497": "Thanks for the very kind explanation and advice!\n\nYou pointed very important thing, 2. Actually, I've concerned about efficiency and computational cost.\n\nAfter merging your advice and my thought, I make script below. How about that?\n\n    feature_name = ['feat1', 'feat2']\n    def generate_features(data, name):\n        existing_feature_list = os.listdir('../features/')\n        for temp_feat in feature_name:\n            new_feat_name = 'operation_{}'.format(temp_feat)\n            csv_file = '{}.csv'.format(new_feat_name)\n            \n            if csv_file in existing_feature_list:\n                data[new_feat_name] = pd.read_csv('../features/{}'.format(csv_file))\n            else:\n    # generate new features (mean encoding, product, substract, etc)\n                new_feat = operation()        \n                data[new_feat_name] = new_feat\n                new_feat.to_csv('../features/{}'.format(csv_file))\n        return data \n\nCsv can be replaced with other format.",
    "401501": "You mean that make a meta-file including the information of features?",
    "401550": "Yes, something like that. For example,  since creating frequency and mean encoding may cost some time, you can create them once, and store in a file. But creating simple  statistic like time transformation, ratio or simple groupby operations( mean, max, min, .etc) will not cost too much time, so just running generation scripts on meta-data to create them when modeling ( if there are too many features, IO duration may longer than running generation scripts )."
  },
  "source": "meta"
}