{
  "id": 15432,
  "title": "How to run these tests before submission",
  "url": "/competitions/flavours-of-physics/discussion/15432",
  "author_name": "",
  "post_date": "2015-07-22T06:32:54.907Z",
  "votes": 1,
  "comment_count": 10,
  "views": 1597,
  "content": "<p>I am not sure how to run &quot;agreement test and the correlation test&quot; before submitting results. Also not sure how to use check_agreement.csv, check_correlation.cv files for this purpose.</p>",
  "messages": [
    {
      "id": "86423",
      "postDate": "07/22/2015 06:32:54",
      "content": "<p>I am not sure how to run &quot;agreement test and the correlation test&quot; before submitting results. Also not sure how to use check_agreement.csv, check_correlation.cv files for this purpose.</p>",
      "rawMarkdown": "I am not sure how to run \"agreement test and the correlation test\" before submitting results. Also not sure how to use check_agreement.csv, check_correlation.cv files for this purpose.",
      "votes": null
    },
    {
      "id": "86434",
      "postDate": "07/22/2015 07:55:42",
      "content": "<p>Check out the starter kit and see how they are used.\nCheers :)</p>",
      "rawMarkdown": "Check out the starter kit and see how they are used.\r\nCheers :)",
      "votes": null
    },
    {
      "id": "86738",
      "postDate": "07/24/2015 02:44:25",
      "content": "<p>Not sure if I'm the only one, but the notebook gave me a &quot;Bad Request&quot; error.\nI followed the Readme.md instructions, I have all the data available but somehow it wasn't working.</p>\n\n<p>Luckily Github displays notebooks properly, so if someone is in the same situation, you can make sense of how to run these tests looking directly at:\n<a href=\"https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\">https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb</a></p>",
      "rawMarkdown": "Not sure if I'm the only one, but the notebook gave me a \"Bad Request\" error.\r\nI followed the Readme.md instructions, I have all the data available but somehow it wasn't working.\r\n\r\nLuckily Github displays notebooks properly, so if someone is in the same situation, you can make sense of how to run these tests looking directly at:\r\nhttps://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb",
      "votes": null
    },
    {
      "id": "86829",
      "postDate": "07/24/2015 15:08:00",
      "content": "<p>Is running those  script necessary to  submit?</p>",
      "rawMarkdown": "Is running those  script necessary to  submit?",
      "votes": null
    },
    {
      "id": "86832",
      "postDate": "07/24/2015 15:22:12",
      "content": "<p>[quote=DaiaAlexandru;86829]</p>\n\n<p>Is running those  script necessary to  submit?</p>\n\n<p>[/quote]</p>\n\n<p>No, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.</p>\n\n<p>I keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  </p>\n\n<p>I'm not a big fan of competitions where random luck is the primary factor.</p>",
      "rawMarkdown": "[quote=DaiaAlexandru;86829]\r\n\r\nIs running those  script necessary to  submit?\r\n\r\n[/quote]\r\n\r\nNo, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.\r\n\r\nI keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n\r\nI'm not a big fan of competitions where random luck is the primary factor.",
      "votes": null
    },
    {
      "id": "86835",
      "postDate": "07/24/2015 15:43:01",
      "content": "<p>[quote=fchollet;86832]</p>\n\n<p>[quote=DaiaAlexandru;86829]</p>\n\n<p>Is running those  script necessary to  submit?</p>\n\n<p>[/quote]</p>\n\n<p>No, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.</p>\n\n<p>I keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  </p>\n\n<p>I'm not a big fan of competitions where random luck is the primary factor.</p>\n\n<p>[/quote]\nWhen I ensemble multiple models, this agreement test becomes tricky: because of error propagation, ensembling multiple models can increase the risk. Practically, these RF/XGboost methods are kind of stable for agreement test, by removing SDPHits, it is fine for this moment, but MLP method (Keras) is a little bit dangerous. The reason is beyond my knowledge for now :-(</p>",
      "rawMarkdown": "[quote=fchollet;86832]\r\n\r\n[quote=DaiaAlexandru;86829]\r\n\r\nIs running those  script necessary to  submit?\r\n\r\n[/quote]\r\n\r\nNo, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.\r\n\r\nI keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n\r\nI'm not a big fan of competitions where random luck is the primary factor.\r\n\r\n[/quote]\r\nWhen I ensemble multiple models, this agreement test becomes tricky: because of error propagation, ensembling multiple models can increase the risk. Practically, these RF/XGboost methods are kind of stable for agreement test, by removing SDPHits, it is fine for this moment, but MLP method (Keras) is a little bit dangerous. The reason is beyond my knowledge for now :-(",
      "votes": null
    },
    {
      "id": "86844",
      "postDate": "07/24/2015 16:03:18",
      "content": "<p>I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)</p>",
      "rawMarkdown": "I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)",
      "votes": null
    },
    {
      "id": "86865",
      "postDate": "07/24/2015 19:25:10",
      "content": "<p>[quote=fchollet;86832]\nincredibly frustrating [...] In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test. <br>\n[/quote]</p>\n\n<p>Getting 0.0901745866599304 on the agreement test sure feels like a big 'what-if'? </p>\n\n<p>I think it is possible to ensemble the agreement and correlation predictions before submitting though, but there remains some variance between local and LB on these tests. (Though I've had a ~0.087 local pass on LB).</p>",
      "rawMarkdown": "[quote=fchollet;86832]\r\nincredibly frustrating [...] In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n[/quote]\r\n\r\nGetting 0.0901745866599304 on the agreement test sure feels like a big 'what-if'? \r\n\r\nI think it is possible to ensemble the agreement and correlation predictions before submitting though, but there remains some variance between local and LB on these tests. (Though I've had a ~0.087 local pass on LB).",
      "votes": null
    },
    {
      "id": "86877",
      "postDate": "07/24/2015 21:24:58",
      "content": "<p>Hi I have modified Ben Hammer's script so that you can check all these scores before submission. Have a look. It might help you.\nlink : <a href=\"https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc\">https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc</a></p>",
      "rawMarkdown": "Hi I have modified Ben Hammer's script so that you can check all these scores before submission. Have a look. It might help you.\r\nlink : https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc",
      "votes": null
    },
    {
      "id": "86949",
      "postDate": "07/25/2015 14:45:06",
      "content": "<p>[quote=DaiaAlexandru;86844]</p>\n\n<p>I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)</p>\n\n<p>[/quote]</p>\n\n<p>When you submit an array of predictions, Kaggle computes the &quot;KS&quot; score of your predictions (Kolmogorov-Smirnov distance between predictions on &quot;real&quot; data and simulated data). This score must remain lower than 0.09.</p>\n\n<p>The space of all possible submissions is a ball in a vector space (ball of radius 1 in a N-dimensional space where N = number of data points in test.csv). The &quot;ks space&quot; I am talking about is the subspace of points x of this ball such that KS(x) &lt; 0.09. That is the space of all valid submissions (ignoring the correlation test here...).</p>\n\n<p>This space is not convex, nor in fact continuous. Which makes ensembling very tricky: mixing up various valid submissions can get you an invalid submission. </p>\n\n<p>This competition is going to be nothing but hair-pulling and frustration, because since the labels are apparently leaking from simulated data, any sufficiently performant ML method is going to fail the agreement test. All my valid submissions so far have been from models that were severely underfitting; basically the reason they were passing the test was that they were just plain bad from a ML perspective. Training the same models until convergence caused my submissions to fail the test.</p>\n\n<p>I do not believe it possible to &quot;clean up&quot; the data (transform it in a way that makes real data and simulated indistinguishable). So I think this competition is just fundamentally flawed, as it sometimes happens. Too bad, because the Higgs Boson competition last year was a great one. </p>",
      "rawMarkdown": "[quote=DaiaAlexandru;86844]\r\n\r\nI am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)\r\n\r\n[/quote]\r\n\r\nWhen you submit an array of predictions, Kaggle computes the \"KS\" score of your predictions (Kolmogorov-Smirnov distance between predictions on \"real\" data and simulated data). This score must remain lower than 0.09.\r\n\r\nThe space of all possible submissions is a ball in a vector space (ball of radius 1 in a N-dimensional space where N = number of data points in test.csv). The \"ks space\" I am talking about is the subspace of points x of this ball such that KS(x) < 0.09. That is the space of all valid submissions (ignoring the correlation test here...).\r\n\r\nThis space is not convex, nor in fact continuous. Which makes ensembling very tricky: mixing up various valid submissions can get you an invalid submission. \r\n\r\nThis competition is going to be nothing but hair-pulling and frustration, because since the labels are apparently leaking from simulated data, any sufficiently performant ML method is going to fail the agreement test. All my valid submissions so far have been from models that were severely underfitting; basically the reason they were passing the test was that they were just plain bad from a ML perspective. Training the same models until convergence caused my submissions to fail the test.\r\n\r\nI do not believe it possible to \"clean up\" the data (transform it in a way that makes real data and simulated indistinguishable). So I think this competition is just fundamentally flawed, as it sometimes happens. Too bad, because the Higgs Boson competition last year was a great one.",
      "votes": null
    },
    {
      "id": "86950",
      "postDate": "07/25/2015 14:58:30",
      "content": "<p>Ok, tnks!</p>",
      "rawMarkdown": "Ok, tnks!",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 86434,
      "author_name": "deepcnn",
      "author_url": "",
      "post_date": "07/22/2015 07:55:42",
      "content": "<p>Check out the starter kit and see how they are used.\nCheers :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86738,
      "author_name": "pierreph",
      "author_url": "",
      "post_date": "07/24/2015 02:44:25",
      "content": "<p>Not sure if I'm the only one, but the notebook gave me a &quot;Bad Request&quot; error.\nI followed the Readme.md instructions, I have all the data available but somehow it wasn't working.</p>\n\n<p>Luckily Github displays notebooks properly, so if someone is in the same situation, you can make sense of how to run these tests looking directly at:\n<a href=\"https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb\">https://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86829,
      "author_name": "",
      "author_url": "",
      "post_date": "07/24/2015 15:08:00",
      "content": "<p>Is running those  script necessary to  submit?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86832,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "07/24/2015 15:22:12",
      "content": "<p>[quote=DaiaAlexandru;86829]</p>\n\n<p>Is running those  script necessary to  submit?</p>\n\n<p>[/quote]</p>\n\n<p>No, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.</p>\n\n<p>I keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  </p>\n\n<p>I'm not a big fan of competitions where random luck is the primary factor.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86835,
      "author_name": "phunter",
      "author_url": "",
      "post_date": "07/24/2015 15:43:01",
      "content": "<p>[quote=fchollet;86832]</p>\n\n<p>[quote=DaiaAlexandru;86829]</p>\n\n<p>Is running those  script necessary to  submit?</p>\n\n<p>[/quote]</p>\n\n<p>No, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.</p>\n\n<p>I keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  </p>\n\n<p>I'm not a big fan of competitions where random luck is the primary factor.</p>\n\n<p>[/quote]\nWhen I ensemble multiple models, this agreement test becomes tricky: because of error propagation, ensembling multiple models can increase the risk. Practically, these RF/XGboost methods are kind of stable for agreement test, by removing SDPHits, it is fine for this moment, but MLP method (Keras) is a little bit dangerous. The reason is beyond my knowledge for now :-(</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86844,
      "author_name": "",
      "author_url": "",
      "post_date": "07/24/2015 16:03:18",
      "content": "<p>I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86865,
      "author_name": "triskelion",
      "author_url": "",
      "post_date": "07/24/2015 19:25:10",
      "content": "<p>[quote=fchollet;86832]\nincredibly frustrating [...] In particular, ensembling is near impossible because the ks&lt;0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test. <br>\n[/quote]</p>\n\n<p>Getting 0.0901745866599304 on the agreement test sure feels like a big 'what-if'? </p>\n\n<p>I think it is possible to ensemble the agreement and correlation predictions before submitting though, but there remains some variance between local and LB on these tests. (Though I've had a ~0.087 local pass on LB).</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86877,
      "author_name": "harshaneel",
      "author_url": "",
      "post_date": "07/24/2015 21:24:58",
      "content": "<p>Hi I have modified Ben Hammer's script so that you can check all these scores before submission. Have a look. It might help you.\nlink : <a href=\"https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc\">https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc</a></p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86949,
      "author_name": "fchollet",
      "author_url": "",
      "post_date": "07/25/2015 14:45:06",
      "content": "<p>[quote=DaiaAlexandru;86844]</p>\n\n<p>I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)</p>\n\n<p>[/quote]</p>\n\n<p>When you submit an array of predictions, Kaggle computes the &quot;KS&quot; score of your predictions (Kolmogorov-Smirnov distance between predictions on &quot;real&quot; data and simulated data). This score must remain lower than 0.09.</p>\n\n<p>The space of all possible submissions is a ball in a vector space (ball of radius 1 in a N-dimensional space where N = number of data points in test.csv). The &quot;ks space&quot; I am talking about is the subspace of points x of this ball such that KS(x) &lt; 0.09. That is the space of all valid submissions (ignoring the correlation test here...).</p>\n\n<p>This space is not convex, nor in fact continuous. Which makes ensembling very tricky: mixing up various valid submissions can get you an invalid submission. </p>\n\n<p>This competition is going to be nothing but hair-pulling and frustration, because since the labels are apparently leaking from simulated data, any sufficiently performant ML method is going to fail the agreement test. All my valid submissions so far have been from models that were severely underfitting; basically the reason they were passing the test was that they were just plain bad from a ML perspective. Training the same models until convergence caused my submissions to fail the test.</p>\n\n<p>I do not believe it possible to &quot;clean up&quot; the data (transform it in a way that makes real data and simulated indistinguishable). So I think this competition is just fundamentally flawed, as it sometimes happens. Too bad, because the Higgs Boson competition last year was a great one. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 86950,
      "author_name": "",
      "author_url": "",
      "post_date": "07/25/2015 14:58:30",
      "content": "<p>Ok, tnks!</p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "86423": "I am not sure how to run \"agreement test and the correlation test\" before submitting results. Also not sure how to use check_agreement.csv, check_correlation.cv files for this purpose.",
    "86434": "Check out the starter kit and see how they are used.\r\nCheers :)",
    "86738": "Not sure if I'm the only one, but the notebook gave me a \"Bad Request\" error.\r\nI followed the Readme.md instructions, I have all the data available but somehow it wasn't working.\r\n\r\nLuckily Github displays notebooks properly, so if someone is in the same situation, you can make sense of how to run these tests looking directly at:\r\nhttps://github.com/yandexdataschool/flavours-of-physics-start/blob/master/baseline.ipynb",
    "86829": "Is running those  script necessary to  submit?",
    "86832": "[quote=DaiaAlexandru;86829]\r\n\r\nIs running those  script necessary to  submit?\r\n\r\n[/quote]\r\n\r\nNo, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.\r\n\r\nI keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n\r\nI'm not a big fan of competitions where random luck is the primary factor.",
    "86835": "[quote=fchollet;86832]\r\n\r\n[quote=DaiaAlexandru;86829]\r\n\r\nIs running those  script necessary to  submit?\r\n\r\n[/quote]\r\n\r\nNo, but it gives you a vague idea of whether your model has a chance at generating a submission that will pass the real tests. However it's really just a vague idea, because you can well pass the tests locally and still fail them at submission time (since the ks score variance appears to be fairly high). It's basically a lottery.\r\n\r\nI keep passing the agreement test locally with some significant margin (0.07-0.085) and failing it at submission time (generally end up around 0.09-0.095). I'm going to keep at it for maybe a couple days then I will quit, as this is turning out to be an incredibly frustrating competition. In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n\r\nI'm not a big fan of competitions where random luck is the primary factor.\r\n\r\n[/quote]\r\nWhen I ensemble multiple models, this agreement test becomes tricky: because of error propagation, ensembling multiple models can increase the risk. Practically, these RF/XGboost methods are kind of stable for agreement test, by removing SDPHits, it is fine for this moment, but MLP method (Keras) is a little bit dangerous. The reason is beyond my knowledge for now :-(",
    "86844": "I am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)",
    "86865": "[quote=fchollet;86832]\r\nincredibly frustrating [...] In particular, ensembling is near impossible because the ks<0.09 space is not convex, i.e. a weighted average of two submissions that do pass the test can end up not passing the test.  \r\n[/quote]\r\n\r\nGetting 0.0901745866599304 on the agreement test sure feels like a big 'what-if'? \r\n\r\nI think it is possible to ensemble the agreement and correlation predictions before submitting though, but there remains some variance between local and LB on these tests. (Though I've had a ~0.087 local pass on LB).",
    "86877": "Hi I have modified Ben Hammer's script so that you can check all these scores before submission. Have a look. It might help you.\r\nlink : https://www.kaggle.com/harshaneel/flavours-of-physics/check-you-agreement-correlation-and-roc",
    "86949": "[quote=DaiaAlexandru;86844]\r\n\r\nI am not  very fammiliar  with  the terminology here  , could  I  ask what is  that  ks   space that is  not convex?In fact  , I am asking  because  I am  not   sure  if I wan't to try make  a submission here..:)\r\n\r\n[/quote]\r\n\r\nWhen you submit an array of predictions, Kaggle computes the \"KS\" score of your predictions (Kolmogorov-Smirnov distance between predictions on \"real\" data and simulated data). This score must remain lower than 0.09.\r\n\r\nThe space of all possible submissions is a ball in a vector space (ball of radius 1 in a N-dimensional space where N = number of data points in test.csv). The \"ks space\" I am talking about is the subspace of points x of this ball such that KS(x) < 0.09. That is the space of all valid submissions (ignoring the correlation test here...).\r\n\r\nThis space is not convex, nor in fact continuous. Which makes ensembling very tricky: mixing up various valid submissions can get you an invalid submission. \r\n\r\nThis competition is going to be nothing but hair-pulling and frustration, because since the labels are apparently leaking from simulated data, any sufficiently performant ML method is going to fail the agreement test. All my valid submissions so far have been from models that were severely underfitting; basically the reason they were passing the test was that they were just plain bad from a ML perspective. Training the same models until convergence caused my submissions to fail the test.\r\n\r\nI do not believe it possible to \"clean up\" the data (transform it in a way that makes real data and simulated indistinguishable). So I think this competition is just fundamentally flawed, as it sometimes happens. Too bad, because the Higgs Boson competition last year was a great one.",
    "86950": "Ok, tnks!"
  },
  "source": "meta"
}