{
  "id": 67850,
  "title": "Method to see the difference between train and test",
  "url": "/competitions/ga-customer-revenue-prediction/discussion/67850",
  "author_name": "",
  "post_date": "2018-10-06T07:21:40.217428600Z",
  "votes": 4,
  "comment_count": 5,
  "views": 0,
  "content": "<p>I've read many topics, and many kaggler said that train and test are different.</p>\n\n<p>So, I'm going to see the difference between train and test.</p>\n\n<p>I've searched and found that Kolmogorov–Smirnov test is a good candidate for the purpose.</p>\n\n<p>I want to know it is right! Could you, kagglers advise me on that? Thanks!</p>",
  "messages": [
    {
      "id": "399594",
      "postDate": "10/06/2018 07:21:40",
      "content": "<p>I've read many topics, and many kaggler said that train and test are different.</p>\n\n<p>So, I'm going to see the difference between train and test.</p>\n\n<p>I've searched and found that Kolmogorov–Smirnov test is a good candidate for the purpose.</p>\n\n<p>I want to know it is right! Could you, kagglers advise me on that? Thanks!</p>",
      "rawMarkdown": "I've read many topics, and many kaggler said that train and test are different.\n\nSo, I'm going to see the difference between train and test.\n\nI've searched and found that Kolmogorov–Smirnov test is a good candidate for the purpose.\n\nI want to know it is right! Could you, kagglers advise me on that? Thanks!",
      "votes": null
    },
    {
      "id": "399724",
      "postDate": "10/06/2018 15:29:25",
      "content": "<p>There are multiple ways to do this, you can:</p>\n\n<ul>\n<li>perform some sort of a dimensionality reduction (like T-SNE) on the whole data, so you could plot the train and test, An example is given <a href=\"https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44799\">here</a>. But one challenge with something like T-SNE in this problem is that you have to convert the categorical features into numerical features first, without increasing the required computational power (probably you need a powerful computer with enough RAM).</li>\n<li>calculate certain quantities for test and train in comparable time windows like number of page views  in different weeks of year for train and test to see how different test and train are. </li>\n<li>estimate and compare parameters like kurtosis, skewness, mean, median, ... of features in test and train.</li>\n<li>train a classifier on test and predict the train records using this classifier to see the difference in probabilities assigned to each sample in train (like in adversarial validation); if train and test are very different then train and test are highly distinguishable using a good classifier.</li>\n<li>...</li>\n</ul>",
      "rawMarkdown": "There are multiple ways to do this, you can:\n\n + perform some sort of a dimensionality reduction (like T-SNE) on the whole data, so you could plot the train and test, An example is given [here][1]. But one challenge with something like T-SNE in this problem is that you have to convert the categorical features into numerical features first, without increasing the required computational power (probably you need a powerful computer with enough RAM).\n + calculate certain quantities for test and train in comparable time windows like number of page views  in different weeks of year for train and test to see how different test and train are. \n + estimate and compare parameters like kurtosis, skewness, mean, median, ... of features in test and train.\n + train a classifier on test and predict the train records using this classifier to see the difference in probabilities assigned to each sample in train (like in adversarial validation); if train and test are very different then train and test are highly distinguishable using a good classifier.\n + ...\n\n \n\n \n\n\n  [1]: https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44799",
      "votes": null
    },
    {
      "id": "400039",
      "postDate": "10/07/2018 12:41:34",
      "content": "<p>Yes you can run a  Kolmogorov–Smirnov test on each variable separately and draw conclusions whether that variable is equally distributed in train and test sets with certain degree of confidence.</p>",
      "rawMarkdown": "Yes you can run a  Kolmogorov–Smirnov test on each variable separately and draw conclusions whether that variable is equally distributed in train and test sets with certain degree of confidence.",
      "votes": null
    },
    {
      "id": "400106",
      "postDate": "10/07/2018 15:54:42",
      "content": "<p>Very thanks for sharing your knowledge! I've learn many good things from your comments!</p>",
      "rawMarkdown": "Very thanks for sharing your knowledge! I've learn many good things from your comments!",
      "votes": null
    },
    {
      "id": "400108",
      "postDate": "10/07/2018 16:00:15",
      "content": "<p>Thanks. I have a question. You mentioned the certain degree of confidence. What threshold do people commonly use? If I calculated Kolmogorov–Smirnov test using function in scipy, I can get both statistic and p-value. Could you teach me to choose common threshold for statistic?</p>",
      "rawMarkdown": "Thanks. I have a question. You mentioned the certain degree of confidence. What threshold do people commonly use? If I calculated Kolmogorov–Smirnov test using function in scipy, I can get both statistic and p-value. Could you teach me to choose common threshold for statistic?",
      "votes": null
    },
    {
      "id": "400111",
      "postDate": "10/07/2018 16:04:08",
      "content": "<p>Happy to be helpful :-); I remember when I started competing at Kaggle, I know much more about ML  compared to back then... all because of shared discussions and kernels!</p>",
      "rawMarkdown": "Happy to be helpful :-); I remember when I started competing at Kaggle, I know much more about ML  compared to back then... all because of shared discussions and kernels!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 399724,
      "author_name": "kainsama",
      "author_url": "",
      "post_date": "10/06/2018 15:29:25",
      "content": "<p>There are multiple ways to do this, you can:</p>\n\n<ul>\n<li>perform some sort of a dimensionality reduction (like T-SNE) on the whole data, so you could plot the train and test, An example is given <a href=\"https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44799\">here</a>. But one challenge with something like T-SNE in this problem is that you have to convert the categorical features into numerical features first, without increasing the required computational power (probably you need a powerful computer with enough RAM).</li>\n<li>calculate certain quantities for test and train in comparable time windows like number of page views  in different weeks of year for train and test to see how different test and train are. </li>\n<li>estimate and compare parameters like kurtosis, skewness, mean, median, ... of features in test and train.</li>\n<li>train a classifier on test and predict the train records using this classifier to see the difference in probabilities assigned to each sample in train (like in adversarial validation); if train and test are very different then train and test are highly distinguishable using a good classifier.</li>\n<li>...</li>\n</ul>",
      "votes": null,
      "replies": [
        {
          "id": 400106,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/07/2018 15:54:42",
          "content": "<p>Very thanks for sharing your knowledge! I've learn many good things from your comments!</p>",
          "votes": null,
          "replies": []
        },
        {
          "id": 400111,
          "author_name": "kainsama",
          "author_url": "",
          "post_date": "10/07/2018 16:04:08",
          "content": "<p>Happy to be helpful :-); I remember when I started competing at Kaggle, I know much more about ML  compared to back then... all because of shared discussions and kernels!</p>",
          "votes": null,
          "replies": []
        }
      ]
    },
    {
      "id": 400039,
      "author_name": "fingoldo",
      "author_url": "",
      "post_date": "10/07/2018 12:41:34",
      "content": "<p>Yes you can run a  Kolmogorov–Smirnov test on each variable separately and draw conclusions whether that variable is equally distributed in train and test sets with certain degree of confidence.</p>",
      "votes": null,
      "replies": [
        {
          "id": 400108,
          "author_name": "youhanlee",
          "author_url": "",
          "post_date": "10/07/2018 16:00:15",
          "content": "<p>Thanks. I have a question. You mentioned the certain degree of confidence. What threshold do people commonly use? If I calculated Kolmogorov–Smirnov test using function in scipy, I can get both statistic and p-value. Could you teach me to choose common threshold for statistic?</p>",
          "votes": null,
          "replies": []
        }
      ]
    }
  ],
  "raw_markdown_by_id": {
    "399594": "I've read many topics, and many kaggler said that train and test are different.\n\nSo, I'm going to see the difference between train and test.\n\nI've searched and found that Kolmogorov–Smirnov test is a good candidate for the purpose.\n\nI want to know it is right! Could you, kagglers advise me on that? Thanks!",
    "399724": "There are multiple ways to do this, you can:\n\n + perform some sort of a dimensionality reduction (like T-SNE) on the whole data, so you could plot the train and test, An example is given [here][1]. But one challenge with something like T-SNE in this problem is that you have to convert the categorical features into numerical features first, without increasing the required computational power (probably you need a powerful computer with enough RAM).\n + calculate certain quantities for test and train in comparable time windows like number of page views  in different weeks of year for train and test to see how different test and train are. \n + estimate and compare parameters like kurtosis, skewness, mean, median, ... of features in test and train.\n + train a classifier on test and predict the train records using this classifier to see the difference in probabilities assigned to each sample in train (like in adversarial validation); if train and test are very different then train and test are highly distinguishable using a good classifier.\n + ...\n\n \n\n \n\n\n  [1]: https://www.kaggle.com/c/porto-seguro-safe-driver-prediction/discussion/44799",
    "400039": "Yes you can run a  Kolmogorov–Smirnov test on each variable separately and draw conclusions whether that variable is equally distributed in train and test sets with certain degree of confidence.",
    "400106": "Very thanks for sharing your knowledge! I've learn many good things from your comments!",
    "400108": "Thanks. I have a question. You mentioned the certain degree of confidence. What threshold do people commonly use? If I calculated Kolmogorov–Smirnov test using function in scipy, I can get both statistic and p-value. Could you teach me to choose common threshold for statistic?",
    "400111": "Happy to be helpful :-); I remember when I started competing at Kaggle, I know much more about ML  compared to back then... all because of shared discussions and kernels!"
  },
  "source": "meta"
}