{
  "id": 25120,
  "title": "F# script: LB 0.43 using < 4GB RAM with F# XGBoost",
  "url": "/competitions/bosch-production-line-performance/discussion/25120",
  "author_name": "",
  "post_date": "2016-11-05T15:40:48.630Z",
  "votes": 14,
  "comment_count": 52,
  "views": 4025,
  "content": "<p>Hi everyone,</p>\n\n<p>F# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:</p>\n\n<p><a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>You can run it in Visual Studio or VS Code (Windows only for now).</p>\n\n<p>My approach:</p>\n\n<ol>\n<li>Create &quot;magic&quot; features.</li>\n</ol>\n\n<p>The first script will merge train and test data by ID and minimum date from date dataset (&quot;StartDate&quot;). We can then identify a &quot;series&quot; as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various &quot;magic&quot; features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\nYou should be able to easily modify the script and add even more features.</p>\n\n<ol start=\"2\">\n<li>All categorical, numeric and &quot;magic&quot; data is imported into a binary format. </li>\n</ol>\n\n<p>This format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.</p>\n\n<ol start=\"4\">\n<li>Modelling uses my F# implementation of XGBoost.</li>\n</ol>\n\n<p>I separately model &quot;series&quot; data and &quot;not series&quot; data. The API is similar to original xgboost. </p>\n\n<p>The whole process will take a few hours with most of the time spent on creating magic features and importing the data.</p>\n\n<p>Have fun!</p>\n\n<p>Any feedback welcome:)</p>\n\n<p>Adam </p>",
  "messages": [
    {
      "id": "142931",
      "postDate": "11/05/2016 15:40:48",
      "content": "<p>Hi everyone,</p>\n\n<p>F# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:</p>\n\n<p><a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>You can run it in Visual Studio or VS Code (Windows only for now).</p>\n\n<p>My approach:</p>\n\n<ol>\n<li>Create &quot;magic&quot; features.</li>\n</ol>\n\n<p>The first script will merge train and test data by ID and minimum date from date dataset (&quot;StartDate&quot;). We can then identify a &quot;series&quot; as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various &quot;magic&quot; features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\nYou should be able to easily modify the script and add even more features.</p>\n\n<ol start=\"2\">\n<li>All categorical, numeric and &quot;magic&quot; data is imported into a binary format. </li>\n</ol>\n\n<p>This format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.</p>\n\n<ol start=\"4\">\n<li>Modelling uses my F# implementation of XGBoost.</li>\n</ol>\n\n<p>I separately model &quot;series&quot; data and &quot;not series&quot; data. The API is similar to original xgboost. </p>\n\n<p>The whole process will take a few hours with most of the time spent on creating magic features and importing the data.</p>\n\n<p>Have fun!</p>\n\n<p>Any feedback welcome:)</p>\n\n<p>Adam </p>",
      "rawMarkdown": "Hi everyone,\r\n\r\nF# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:\r\n\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nYou can run it in Visual Studio or VS Code (Windows only for now).\r\n\r\nMy approach:\r\n\r\n1. Create \"magic\" features.\r\n\r\nThe first script will merge train and test data by ID and minimum date from date dataset (\"StartDate\"). We can then identify a \"series\" as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various \"magic\" features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\r\nYou should be able to easily modify the script and add even more features.\r\n\r\n2. All categorical, numeric and \"magic\" data is imported into a binary format. \r\n\r\nThis format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n\r\n4. Modelling uses my F# implementation of XGBoost.\r\n\r\nI separately model \"series\" data and \"not series\" data. The API is similar to original xgboost. \r\n\r\nThe whole process will take a few hours with most of the time spent on creating magic features and importing the data.\r\n\r\nHave fun!\r\n\r\nAny feedback welcome:)\r\n\r\nAdam",
      "votes": null
    },
    {
      "id": "143043",
      "postDate": "11/06/2016 12:10:36",
      "content": "<p>This is really impressive. Nice work :)</p>",
      "rawMarkdown": "This is really impressive. Nice work :)",
      "votes": null
    },
    {
      "id": "143044",
      "postDate": "11/06/2016 12:19:02",
      "content": "<p>Thanks:)</p>",
      "rawMarkdown": "Thanks:)",
      "votes": null
    },
    {
      "id": "143273",
      "postDate": "11/07/2016 21:17:10",
      "content": "<p>Thanks xi, nice work.\nTried to reproduce your code, create_magic_features.fsx works but model.fsx throws MemoryException Error on a Win7 PC  with 16GB RAM.</p>\n\n<p>model.fsx(13,17): error FS3033: The type provider 'Xi.XiTypeProvider' reported an error: Exception of type 'System.OutOfMemoryException' was thrown.\nAlso tried FsiAnyCPU.exe 64bit version of Fsi but no luck.\nAny ideas?</p>",
      "rawMarkdown": "Thanks xi, nice work.\r\nTried to reproduce your code, create_magic_features.fsx works but model.fsx throws MemoryException Error on a Win7 PC  with 16GB RAM.\r\n\r\nmodel.fsx(13,17): error FS3033: The type provider 'Xi.XiTypeProvider' reported an error: Exception of type 'System.OutOfMemoryException' was thrown.\r\nAlso tried FsiAnyCPU.exe 64bit version of Fsi but no luck.\r\nAny ideas?",
      "votes": null
    },
    {
      "id": "143276",
      "postDate": "11/07/2016 21:20:32",
      "content": "<p>Hi,\nPlease run in fsi:\nSystem.Environment.Is64BitProcess\nto verify it is 64 bit version</p>\n\n<p>Adam</p>",
      "rawMarkdown": "Hi,\r\nPlease run in fsi:\r\nSystem.Environment.Is64BitProcess\r\nto verify it is 64 bit version\r\n\r\nAdam",
      "votes": null
    },
    {
      "id": "143279",
      "postDate": "11/07/2016 21:46:35",
      "content": "<p>System.Environment.Is64BitProcess = True</p>",
      "rawMarkdown": "System.Environment.Is64BitProcess = True",
      "votes": null
    },
    {
      "id": "143281",
      "postDate": "11/07/2016 21:51:04",
      "content": "<p>Are you running it in VS2015?\nWhen you run the code line after line, which one throws the exception?</p>",
      "rawMarkdown": "Are you running it in VS2015?\r\nWhen you run the code line after line, which one throws the exception?",
      "votes": null
    },
    {
      "id": "143287",
      "postDate": "11/07/2016 22:22:24",
      "content": "<p>Tried both in VS Code and command line fsi same error\nCode line <br>\ntype NumTrain = XiDataFrame&lt; &quot;train_numeric.csv.zip&quot; &gt;\nthrows the exception.  Same with line\ntype NumTest = XiDataFrame&lt; &quot;test_numeric.csv.zip&quot; &gt;\nSeems to trace to XiDataFrame thus referenced to Xi.dll, correct? So maybe some problem with the dll?</p>",
      "rawMarkdown": "Tried both in VS Code and command line fsi same error\r\nCode line  \r\ntype NumTrain = XiDataFrame< \"train_numeric.csv.zip\" >\r\nthrows the exception.  Same with line\r\ntype NumTest = XiDataFrame< \"test_numeric.csv.zip\" >\r\nSeems to trace to XiDataFrame thus referenced to Xi.dll, correct? So maybe some problem with the dll?",
      "votes": null
    },
    {
      "id": "143288",
      "postDate": "11/07/2016 22:28:17",
      "content": "<p>Did you import the numeric csv data using Xi or just zipped it?</p>\n\n<p>The line in model.fsx is:</p>\n\n<p>type NumTest = XiDataFrame&lt; &quot;test_numeric.zip&quot; &gt;</p>\n\n<p>It should be imported with:</p>\n\n<p>DataImport.importCsvAsync testNumPath [|','|] [||] Int32.MaxValue 10000 0.1 true String.Empty CompressionLevel.NoCompression</p>",
      "rawMarkdown": "Did you import the numeric csv data using Xi or just zipped it?\r\n\r\nThe line in model.fsx is:\r\n\r\ntype NumTest = XiDataFrame< \"test_numeric.zip\" >\r\n\r\nIt should be imported with:\r\n\r\nDataImport.importCsvAsync testNumPath [|','|] [||] Int32.MaxValue 10000 0.1 true String.Empty CompressionLevel.NoCompression",
      "votes": null
    },
    {
      "id": "143292",
      "postDate": "11/07/2016 22:47:38",
      "content": "<p>Just zipped it, I'll give it a try tomorrow and let you know, thanks again!</p>",
      "rawMarkdown": "Just zipped it, I'll give it a try tomorrow and let you know, thanks again!",
      "votes": null
    },
    {
      "id": "143319",
      "postDate": "11/08/2016 03:28:12",
      "content": "<p>Thank you very much.</p>\n\n<p>I got an Error</p>\n\n<p>The type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.</p>\n\n<p>VS 2015</p>",
      "rawMarkdown": "Thank you very much.\r\n\r\nI got an Error\r\n\r\nThe type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.\r\n\r\nVS 2015",
      "votes": null
    },
    {
      "id": "143351",
      "postDate": "11/08/2016 08:00:49",
      "content": "<p>Did you just zip the csv data or import it with DataImport.importCsvAsync?</p>",
      "rawMarkdown": "Did you just zip the csv data or import it with DataImport.importCsvAsync?",
      "votes": null
    },
    {
      "id": "143357",
      "postDate": "11/08/2016 09:02:37",
      "content": "<p>I imported it. With DataImport as the script. it is complaining about XiDataFrame and says </p>\n\n<p>The type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.</p>",
      "rawMarkdown": "I imported it. With DataImport as the script. it is complaining about XiDataFrame and says \r\n\r\nThe type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.",
      "votes": null
    },
    {
      "id": "143358",
      "postDate": "11/08/2016 09:07:28",
      "content": "<p>When you run it line by line, which line throws the error?\nAre you running it in VS2015 F# Interactive?</p>",
      "rawMarkdown": "When you run it line by line, which line throws the error?\r\nAre you running it in VS2015 F# Interactive?",
      "votes": null
    },
    {
      "id": "143359",
      "postDate": "11/08/2016 09:11:42",
      "content": "<p>This line</p>\n\n<p>type NumTrain = XiDataFrame&lt; &quot;train_numeric.zip&quot; &gt;</p>\n\n<p>I ran it from VS2015 F# Interactive and I also tired command line fsi.exe</p>",
      "rawMarkdown": "This line\r\n\r\ntype NumTrain = XiDataFrame< \"train_numeric.zip\" >\r\n\r\nI ran it from VS2015 F# Interactive and I also tired command line fsi.exe",
      "votes": null
    },
    {
      "id": "143361",
      "postDate": "11/08/2016 09:17:24",
      "content": "<p>do you see that error when you just open the script in F# interactive or only when you run it?</p>",
      "rawMarkdown": "do you see that error when you just open the script in F# interactive or only when you run it?",
      "votes": null
    },
    {
      "id": "143363",
      "postDate": "11/08/2016 09:25:08",
      "content": "<p>In Both Cases.</p>",
      "rawMarkdown": "In Both Cases.",
      "votes": null
    },
    {
      "id": "143364",
      "postDate": "11/08/2016 09:29:10",
      "content": "<p>How about the other zip files? Do they also have a compile error?</p>\n\n<p>Did you make any changes to the create_magic_features script?</p>",
      "rawMarkdown": "How about the other zip files? Do they also have a compile error?\r\n\r\nDid you make any changes to the create_magic_features script?",
      "votes": null
    },
    {
      "id": "143367",
      "postDate": "11/08/2016 10:02:33",
      "content": "<p>Thanks. I found the error as I changed the directory.</p>",
      "rawMarkdown": "Thanks. I found the error as I changed the directory.",
      "votes": null
    },
    {
      "id": "143375",
      "postDate": "11/08/2016 12:30:38",
      "content": "<p>I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.</p>",
      "rawMarkdown": "I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.",
      "votes": null
    },
    {
      "id": "143376",
      "postDate": "11/08/2016 12:32:25",
      "content": "<p>Great, many thanks:)</p>",
      "rawMarkdown": "Great, many thanks:)",
      "votes": null
    },
    {
      "id": "143390",
      "postDate": "11/08/2016 14:03:17",
      "content": "<p>You are welcome. Any plans on giving more documentation on Xi library? That would be great.</p>",
      "rawMarkdown": "You are welcome. Any plans on giving more documentation on Xi library? That would be great.",
      "votes": null
    },
    {
      "id": "143400",
      "postDate": "11/08/2016 15:42:10",
      "content": "<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>",
      "rawMarkdown": "[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?",
      "votes": null
    },
    {
      "id": "143401",
      "postDate": "11/08/2016 15:46:36",
      "content": "<p>This is still very much work in progress so not much documentation:(</p>\n\n<p>For people interested in using F# and Xi in data science research or Kaggle competitions I could create a Slack channel or something similar.</p>\n\n<p>F# Kernels would be really useful:)</p>",
      "rawMarkdown": "This is still very much work in progress so not much documentation:(\r\n\r\nFor people interested in using F# and Xi in data science research or Kaggle competitions I could create a Slack channel or something similar.\r\n\r\nF# Kernels would be really useful:)",
      "votes": null
    },
    {
      "id": "143402",
      "postDate": "11/08/2016 15:48:16",
      "content": "<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>",
      "rawMarkdown": "[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.",
      "votes": null
    },
    {
      "id": "143410",
      "postDate": "11/08/2016 16:28:25",
      "content": "<p>[quote=Ayman Khalafallah;143402]</p>\n\n<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>\n\n<p>[/quote]</p>\n\n<p>OK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? </p>",
      "rawMarkdown": "[quote=Ayman Khalafallah;143402]\r\n\r\n[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.\r\n\r\n[/quote]\r\n\r\nOK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct?",
      "votes": null
    },
    {
      "id": "143411",
      "postDate": "11/08/2016 16:29:59",
      "content": "<p>[quote=VictorGarcia;143410]</p>\n\n<p>[quote=Ayman Khalafallah;143402]</p>\n\n<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>\n\n<p>[/quote]</p>\n\n<p>OK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? </p>\n\n<p>[/quote]\nYes and also make sure that you run in 64bit mode.</p>",
      "rawMarkdown": "[quote=VictorGarcia;143410]\r\n\r\n[quote=Ayman Khalafallah;143402]\r\n\r\n[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.\r\n\r\n[/quote]\r\n\r\nOK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? \r\n\r\n[/quote]\r\nYes and also make sure that you run in 64bit mode.",
      "votes": null
    },
    {
      "id": "143620",
      "postDate": "11/09/2016 21:29:41",
      "content": "<p>error FS0193: internal error: No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)</p>\n\n<p>at the line</p>\n\n<p>type MagicTest = XiDataFrame&lt; &quot;test_magic.zip&quot; &gt;\nlet magicTest = new MagicTest()</p>\n\n<p>Previous zips load fine but last one throws an error\nAny suggestions?</p>",
      "rawMarkdown": "error FS0193: internal error: No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)\r\n\r\nat the line\r\n\r\ntype MagicTest = XiDataFrame< \"test_magic.zip\" >\r\nlet magicTest = new MagicTest()\r\n\r\nPrevious zips load fine but last one throws an error\r\nAny suggestions?",
      "votes": null
    },
    {
      "id": "143621",
      "postDate": "11/09/2016 21:36:44",
      "content": "<p>This usually happens when you import a data column with a lot of unique numeric values as a categorical factor.</p>\n\n<p>Make sure you import the data with exactly the same import settings as in the original script.</p>\n\n<p>Adam</p>",
      "rawMarkdown": "This usually happens when you import a data column with a lot of unique numeric values as a categorical factor.\r\n\r\nMake sure you import the data with exactly the same import settings as in the original script.\r\n\r\nAdam",
      "votes": null
    },
    {
      "id": "143623",
      "postDate": "11/09/2016 21:40:47",
      "content": "<p>I imported it using</p>\n\n<p>do! DataImport.importCsvAsync testMagicPath [|','|] [||] Int32.MaxValue 10000 0.0 true String.Empty \nCompressionLevel.NoCompression</p>",
      "rawMarkdown": "I imported it using\r\n\r\ndo! DataImport.importCsvAsync testMagicPath [|','|] [||] Int32.MaxValue 10000 0.0 true String.Empty \r\nCompressionLevel.NoCompression",
      "votes": null
    },
    {
      "id": "143625",
      "postDate": "11/09/2016 21:46:11",
      "content": "<p>Did you modify the script in any way?</p>\n\n<p>Is the zip file 353 MB?</p>",
      "rawMarkdown": "Did you modify the script in any way?\r\n\r\nIs the zip file 353 MB?",
      "votes": null
    },
    {
      "id": "143627",
      "postDate": "11/09/2016 21:54:32",
      "content": "<p>No, used original script\nJust run async imports separately because create_magic_features.fsx ended without creating the zips</p>\n\n<p>Filesize \n345.152 Kb </p>\n\n<p>MD5 Checksum</p>\n\n<p>9e114c98a0e2a4dfdb76230304efc1be</p>",
      "rawMarkdown": "No, used original script\r\nJust run async imports separately because create_magic_features.fsx ended without creating the zips\r\n\r\nFilesize \r\n345.152 Kb \r\n\r\nMD5 Checksum\r\n\r\n9e114c98a0e2a4dfdb76230304efc1be",
      "votes": null
    },
    {
      "id": "143629",
      "postDate": "11/09/2016 22:00:11",
      "content": "<p>When you unzip the file, you should see &quot;test_magic&quot; file inside. How big is it?</p>\n\n<p>Are you running this in 64bit fsi?</p>",
      "rawMarkdown": "When you unzip the file, you should see \"test_magic\" file inside. How big is it?\r\n\r\nAre you running this in 64bit fsi?",
      "votes": null
    },
    {
      "id": "143631",
      "postDate": "11/09/2016 22:10:38",
      "content": "<p>Just noticed that test_magic.csv contain some weird MinFaultFree values instead of nans in covariates . More precisely it contains NaN but in my local language. Maybe some unicode problem creating the file?</p>",
      "rawMarkdown": "Just noticed that test_magic.csv contain some weird MinFaultFree values instead of nans in covariates . More precisely it contains NaN but in my local language. Maybe some unicode problem creating the file?",
      "votes": null
    },
    {
      "id": "143633",
      "postDate": "11/09/2016 22:15:35",
      "content": "<p>So what should these MinFaultFree values be instead &quot;NA&quot;, &quot;NaN&quot; or something else?</p>",
      "rawMarkdown": "So what should these MinFaultFree values be instead \"NA\", \"NaN\" or something else?",
      "votes": null
    },
    {
      "id": "143634",
      "postDate": "11/09/2016 22:16:52",
      "content": "<p>That might explain. Try to make them empty.</p>",
      "rawMarkdown": "That might explain. Try to make them empty.",
      "votes": null
    },
    {
      "id": "143639",
      "postDate": "11/09/2016 22:26:46",
      "content": "<p>Thanks for your help, its my first experience with F# and your help has been very valuable.  I'll try and let you know. </p>",
      "rawMarkdown": "Thanks for your help, its my first experience with F# and your help has been very valuable.  I'll try and let you know.",
      "votes": null
    },
    {
      "id": "143640",
      "postDate": "11/09/2016 22:30:29",
      "content": "<p>You're welcome:)</p>",
      "rawMarkdown": "You're welcome:)",
      "votes": null
    },
    {
      "id": "143670",
      "postDate": "11/10/2016 02:13:11",
      "content": "<p>Impressive work :) And thumb-up for #3 :D</p>\n\n<p>[quote=xi;142931]</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.\n[/quote]</p>",
      "rawMarkdown": "Impressive work :) And thumb-up for #3 :D\r\n\r\n[quote=xi;142931]\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n[/quote]",
      "votes": null
    },
    {
      "id": "143675",
      "postDate": "11/10/2016 04:09:00",
      "content": "<p>Hi Xi:</p>\n\n<p>I was also impressed with your solution and decided to give a try and to learn more on Functional Programming.</p>\n\n<p>First script (creating magic features) was completed successfully and 6 zip files were generated.\nWhen I tried running the second (model), it failed with </p>\n\n<p>Out of Memory exception</p>\n\n<ol>\n<li><p>I run script from VS2015 and from command line, and it fails when it comes to reading the fourth file</p></li>\n<li><p>I tried to work only with numeric and magic files, but it loaded 3 files and then failed (Out Of Memory)</p></li>\n<li><p>My system is 64-bit with 8GB RAM</p></li>\n</ol>\n\n<p>Can you please advise how to improve efficiency by releasing vars, when they are not needed. How to better manage memory in F#. Is it necessary to load all files at the beginning?</p>",
      "rawMarkdown": "Hi Xi:\r\n\r\nI was also impressed with your solution and decided to give a try and to learn more on Functional Programming.\r\n\r\nFirst script (creating magic features) was completed successfully and 6 zip files were generated.\r\nWhen I tried running the second (model), it failed with \r\n\r\nOut of Memory exception\r\n\r\n1. I run script from VS2015 and from command line, and it fails when it comes to reading the fourth file\r\n\r\n2. I tried to work only with numeric and magic files, but it loaded 3 files and then failed (Out Of Memory)\r\n\r\n3. My system is 64-bit with 8GB RAM\r\n\r\nCan you please advise how to improve efficiency by releasing vars, when they are not needed. How to better manage memory in F#. Is it necessary to load all files at the beginning?",
      "votes": null
    },
    {
      "id": "143703",
      "postDate": "11/10/2016 07:59:25",
      "content": "<p>How big are your zip files? All the data should fit in under 4GB RAM. The zip files are not really compressed so what is on disk gets loaded into memory 1:1.</p>\n\n<p>You can run it line by line and monitor memory usage in Task Manager. Does it get above 4GB?</p>\n\n<p>Did you make any changes to the scripts?</p>",
      "rawMarkdown": "How big are your zip files? All the data should fit in under 4GB RAM. The zip files are not really compressed so what is on disk gets loaded into memory 1:1.\r\n\r\nYou can run it line by line and monitor memory usage in Task Manager. Does it get above 4GB?\r\n\r\nDid you make any changes to the scripts?",
      "votes": null
    },
    {
      "id": "143736",
      "postDate": "11/10/2016 12:29:55",
      "content": "<p>thank you </p>",
      "rawMarkdown": "thank you",
      "votes": null
    },
    {
      "id": "143772",
      "postDate": "11/10/2016 15:52:27",
      "content": "<p>[quote=Ayman Khalafallah;143375]</p>\n\n<p>I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.</p>\n\n<p>[/quote]</p>\n\n<p>May I ask how many variables did you choose after Fisher exact Test? Thanks.</p>",
      "rawMarkdown": "[quote=Ayman Khalafallah;143375]\r\n\r\nI can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.\r\n\r\n[/quote]\r\n\r\nMay I ask how many variables did you choose after Fisher exact Test? Thanks.",
      "votes": null
    },
    {
      "id": "143774",
      "postDate": "11/10/2016 16:03:26",
      "content": "<p>I think around 6000 </p>",
      "rawMarkdown": "I think around 6000",
      "votes": null
    },
    {
      "id": "143776",
      "postDate": "11/10/2016 16:05:42",
      "content": "<p>[quote=xi;143774]</p>\n\n<p>I think around 6000 </p>\n\n<p>[/quote]\nGot it. Thanks~</p>",
      "rawMarkdown": "[quote=xi;143774]\r\n\r\nI think around 6000 \r\n\r\n[/quote]\r\nGot it. Thanks~",
      "votes": null
    },
    {
      "id": "143810",
      "postDate": "11/10/2016 19:47:39",
      "content": "<p>[quote=xi;142931]</p>\n\n<p>Hi everyone,</p>\n\n<p>F# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:</p>\n\n<p><a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>You can run it in Visual Studio or VS Code (Windows only for now).</p>\n\n<p>My approach:</p>\n\n<ol>\n<li>Create &quot;magic&quot; features.</li>\n</ol>\n\n<p>The first script will merge train and test data by ID and minimum date from date dataset (&quot;StartDate&quot;). We can then identify a &quot;series&quot; as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various &quot;magic&quot; features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\nYou should be able to easily modify the script and add even more features.</p>\n\n<ol start=\"2\">\n<li>All categorical, numeric and &quot;magic&quot; data is imported into a binary format. </li>\n</ol>\n\n<p>This format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.</p>\n\n<ol start=\"4\">\n<li>Modelling uses my F# implementation of XGBoost.</li>\n</ol>\n\n<p>I separately model &quot;series&quot; data and &quot;not series&quot; data. The API is similar to original xgboost. </p>\n\n<p>The whole process will take a few hours with most of the time spent on creating magic features and importing the data.</p>\n\n<p>Have fun!</p>\n\n<p>Any feedback welcome:)</p>\n\n<p>Adam </p>\n\n<p>[/quote]\nHi xi, I am trying to repeat the 3rd part in your post: Fisher exact Test to all hot encoder data. However, Rstudio give a wrong reminder that: FEXACT error 40.\nOut of workspace.\nIt seems impossible to use all the train rows in each column to perform the Fisher exact Test. Did you do  some sampling operations in this step? Thanks! </p>",
      "rawMarkdown": "[quote=xi;142931]\r\n\r\nHi everyone,\r\n\r\nF# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:\r\n\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nYou can run it in Visual Studio or VS Code (Windows only for now).\r\n\r\nMy approach:\r\n\r\n1. Create \"magic\" features.\r\n\r\nThe first script will merge train and test data by ID and minimum date from date dataset (\"StartDate\"). We can then identify a \"series\" as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various \"magic\" features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\r\nYou should be able to easily modify the script and add even more features.\r\n\r\n2. All categorical, numeric and \"magic\" data is imported into a binary format. \r\n\r\nThis format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n\r\n4. Modelling uses my F# implementation of XGBoost.\r\n\r\nI separately model \"series\" data and \"not series\" data. The API is similar to original xgboost. \r\n\r\nThe whole process will take a few hours with most of the time spent on creating magic features and importing the data.\r\n\r\nHave fun!\r\n\r\nAny feedback welcome:)\r\n\r\nAdam \r\n\r\n\r\n[/quote]\r\nHi xi, I am trying to repeat the 3rd part in your post: Fisher exact Test to all hot encoder data. However, Rstudio give a wrong reminder that: FEXACT error 40.\r\nOut of workspace.\r\nIt seems impossible to use all the train rows in each column to perform the Fisher exact Test. Did you do  some sampling operations in this step? Thanks!",
      "votes": null
    },
    {
      "id": "143816",
      "postDate": "11/10/2016 20:47:15",
      "content": "<p>No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. </p>",
      "rawMarkdown": "No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop.",
      "votes": null
    },
    {
      "id": "143827",
      "postDate": "11/10/2016 22:33:09",
      "content": "<p>I give up, same error again,. Anyway thanks xi for the insights, and your interesting approach but with lack of proper documentation and a black box xi.dll not much can be done. Waiting for your update on further development of the library!</p>",
      "rawMarkdown": "I give up, same error again,. Anyway thanks xi for the insights, and your interesting approach but with lack of proper documentation and a black box xi.dll not much can be done. Waiting for your update on further development of the library!",
      "votes": null
    },
    {
      "id": "143828",
      "postDate": "11/10/2016 22:36:19",
      "content": "<p>[quote=xi;143816]</p>\n\n<p>No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. </p>\n\n<p>[/quote]\nI think the issue is caused by function fisher.test in R. Now I am trying try write a fisher test func by myself. Thanks. </p>",
      "rawMarkdown": "[quote=xi;143816]\r\n\r\nNo sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. \r\n\r\n[/quote]\r\nI think the issue is caused by function fisher.test in R. Now I am trying try write a fisher test func by myself. Thanks.",
      "votes": null
    },
    {
      "id": "143830",
      "postDate": "11/10/2016 22:43:14",
      "content": "<p>Gavrand, I can create a Webex online meeting and remotely fix your problem if you want.</p>\n\n<p>Or watch this space, I will create F# starter kits for other competitions. Best place to test data science software:)</p>\n\n<p>&#958; |&gt; &#955; :D</p>",
      "rawMarkdown": "Gavrand, I can create a Webex online meeting and remotely fix your problem if you want.\r\n\r\nOr watch this space, I will create F# starter kits for other competitions. Best place to test data science software:)\r\n\r\nξ |> λ :D",
      "votes": null
    },
    {
      "id": "143837",
      "postDate": "11/11/2016 00:09:38",
      "content": "<p>I managed to execute it. It helped me to pass that 0.43 threshold. What is more important, I become somewhat familiar with F# and Funct. Programming. Thanks Xi!</p>",
      "rawMarkdown": "I managed to execute it. It helped me to pass that 0.43 threshold. What is more important, I become somewhat familiar with F# and Funct. Programming. Thanks Xi!",
      "votes": null
    },
    {
      "id": "143838",
      "postDate": "11/11/2016 00:11:15",
      "content": "<p>Great:)</p>",
      "rawMarkdown": "Great:)",
      "votes": null
    },
    {
      "id": "143901",
      "postDate": "11/11/2016 09:49:13",
      "content": "<p>I have noticed some of you were probably confused by async data import (DataImport.importCsvAsync).</p>\n\n<p>With F# Async.Start the import starts on a background thread so you might think that nothing is happening. I have changed it now to Async.RunSynchronously</p>\n\n<p>Also, &quot;faultFreeAll&quot;  was using Single.NaN.ToString() which depends on your system local language and might cause issues when the values are parsed later on. So I changed it now to String.Empty.</p>\n\n<p>&#958; </p>",
      "rawMarkdown": "I have noticed some of you were probably confused by async data import (DataImport.importCsvAsync).\r\n\r\nWith F# Async.Start the import starts on a background thread so you might think that nothing is happening. I have changed it now to Async.RunSynchronously\r\n\r\nAlso, \"faultFreeAll\"  was using Single.NaN.ToString() which depends on your system local language and might cause issues when the values are parsed later on. So I changed it now to String.Empty.\r\n\r\nξ",
      "votes": null
    }
  ],
  "comments": [
    {
      "id": 143043,
      "author_name": "anokas",
      "author_url": "",
      "post_date": "11/06/2016 12:10:36",
      "content": "<p>This is really impressive. Nice work :)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143044,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/06/2016 12:19:02",
      "content": "<p>Thanks:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143273,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/07/2016 21:17:10",
      "content": "<p>Thanks xi, nice work.\nTried to reproduce your code, create_magic_features.fsx works but model.fsx throws MemoryException Error on a Win7 PC  with 16GB RAM.</p>\n\n<p>model.fsx(13,17): error FS3033: The type provider 'Xi.XiTypeProvider' reported an error: Exception of type 'System.OutOfMemoryException' was thrown.\nAlso tried FsiAnyCPU.exe 64bit version of Fsi but no luck.\nAny ideas?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143276,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/07/2016 21:20:32",
      "content": "<p>Hi,\nPlease run in fsi:\nSystem.Environment.Is64BitProcess\nto verify it is 64 bit version</p>\n\n<p>Adam</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143279,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/07/2016 21:46:35",
      "content": "<p>System.Environment.Is64BitProcess = True</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143281,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/07/2016 21:51:04",
      "content": "<p>Are you running it in VS2015?\nWhen you run the code line after line, which one throws the exception?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143287,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/07/2016 22:22:24",
      "content": "<p>Tried both in VS Code and command line fsi same error\nCode line <br>\ntype NumTrain = XiDataFrame&lt; &quot;train_numeric.csv.zip&quot; &gt;\nthrows the exception.  Same with line\ntype NumTest = XiDataFrame&lt; &quot;test_numeric.csv.zip&quot; &gt;\nSeems to trace to XiDataFrame thus referenced to Xi.dll, correct? So maybe some problem with the dll?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143288,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/07/2016 22:28:17",
      "content": "<p>Did you import the numeric csv data using Xi or just zipped it?</p>\n\n<p>The line in model.fsx is:</p>\n\n<p>type NumTest = XiDataFrame&lt; &quot;test_numeric.zip&quot; &gt;</p>\n\n<p>It should be imported with:</p>\n\n<p>DataImport.importCsvAsync testNumPath [|','|] [||] Int32.MaxValue 10000 0.1 true String.Empty CompressionLevel.NoCompression</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143292,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/07/2016 22:47:38",
      "content": "<p>Just zipped it, I'll give it a try tomorrow and let you know, thanks again!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143319,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 03:28:12",
      "content": "<p>Thank you very much.</p>\n\n<p>I got an Error</p>\n\n<p>The type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.</p>\n\n<p>VS 2015</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143351,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 08:00:49",
      "content": "<p>Did you just zip the csv data or import it with DataImport.importCsvAsync?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143357,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 09:02:37",
      "content": "<p>I imported it. With DataImport as the script. it is complaining about XiDataFrame and says </p>\n\n<p>The type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143358,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 09:07:28",
      "content": "<p>When you run it line by line, which line throws the error?\nAre you running it in VS2015 F# Interactive?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143359,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 09:11:42",
      "content": "<p>This line</p>\n\n<p>type NumTrain = XiDataFrame&lt; &quot;train_numeric.zip&quot; &gt;</p>\n\n<p>I ran it from VS2015 F# Interactive and I also tired command line fsi.exe</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143361,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 09:17:24",
      "content": "<p>do you see that error when you just open the script in F# interactive or only when you run it?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143363,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 09:25:08",
      "content": "<p>In Both Cases.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143364,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 09:29:10",
      "content": "<p>How about the other zip files? Do they also have a compile error?</p>\n\n<p>Did you make any changes to the create_magic_features script?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143367,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 10:02:33",
      "content": "<p>Thanks. I found the error as I changed the directory.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143375,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 12:30:38",
      "content": "<p>I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143376,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 12:32:25",
      "content": "<p>Great, many thanks:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143390,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 14:03:17",
      "content": "<p>You are welcome. Any plans on giving more documentation on Xi library? That would be great.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143400,
      "author_name": "garcia274",
      "author_url": "",
      "post_date": "11/08/2016 15:42:10",
      "content": "<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143401,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/08/2016 15:46:36",
      "content": "<p>This is still very much work in progress so not much documentation:(</p>\n\n<p>For people interested in using F# and Xi in data science research or Kaggle competitions I could create a Slack channel or something similar.</p>\n\n<p>F# Kernels would be really useful:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143402,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 15:48:16",
      "content": "<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143410,
      "author_name": "garcia274",
      "author_url": "",
      "post_date": "11/08/2016 16:28:25",
      "content": "<p>[quote=Ayman Khalafallah;143402]</p>\n\n<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>\n\n<p>[/quote]</p>\n\n<p>OK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143411,
      "author_name": "aymankh",
      "author_url": "",
      "post_date": "11/08/2016 16:29:59",
      "content": "<p>[quote=VictorGarcia;143410]</p>\n\n<p>[quote=Ayman Khalafallah;143402]</p>\n\n<p>[quote=VictorGarcia;143400]</p>\n\n<p>[quote=Ayman Khalafallah;143367]</p>\n\n<p>Thanks. I found the error as I changed the directory.</p>\n\n<p>[/quote]\nI was getting the same &quot;implementation error not permitted&quot; too. What error did you find/what did you fix?</p>\n\n<p>[/quote]\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.</p>\n\n<p>[/quote]</p>\n\n<p>OK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? </p>\n\n<p>[/quote]\nYes and also make sure that you run in 64bit mode.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143620,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 21:29:41",
      "content": "<p>error FS0193: internal error: No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)</p>\n\n<p>at the line</p>\n\n<p>type MagicTest = XiDataFrame&lt; &quot;test_magic.zip&quot; &gt;\nlet magicTest = new MagicTest()</p>\n\n<p>Previous zips load fine but last one throws an error\nAny suggestions?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143621,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/09/2016 21:36:44",
      "content": "<p>This usually happens when you import a data column with a lot of unique numeric values as a categorical factor.</p>\n\n<p>Make sure you import the data with exactly the same import settings as in the original script.</p>\n\n<p>Adam</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143623,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 21:40:47",
      "content": "<p>I imported it using</p>\n\n<p>do! DataImport.importCsvAsync testMagicPath [|','|] [||] Int32.MaxValue 10000 0.0 true String.Empty \nCompressionLevel.NoCompression</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143625,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/09/2016 21:46:11",
      "content": "<p>Did you modify the script in any way?</p>\n\n<p>Is the zip file 353 MB?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143627,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 21:54:32",
      "content": "<p>No, used original script\nJust run async imports separately because create_magic_features.fsx ended without creating the zips</p>\n\n<p>Filesize \n345.152 Kb </p>\n\n<p>MD5 Checksum</p>\n\n<p>9e114c98a0e2a4dfdb76230304efc1be</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143629,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/09/2016 22:00:11",
      "content": "<p>When you unzip the file, you should see &quot;test_magic&quot; file inside. How big is it?</p>\n\n<p>Are you running this in 64bit fsi?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143631,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 22:10:38",
      "content": "<p>Just noticed that test_magic.csv contain some weird MinFaultFree values instead of nans in covariates . More precisely it contains NaN but in my local language. Maybe some unicode problem creating the file?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143633,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 22:15:35",
      "content": "<p>So what should these MinFaultFree values be instead &quot;NA&quot;, &quot;NaN&quot; or something else?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143634,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/09/2016 22:16:52",
      "content": "<p>That might explain. Try to make them empty.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143639,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/09/2016 22:26:46",
      "content": "<p>Thanks for your help, its my first experience with F# and your help has been very valuable.  I'll try and let you know. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143640,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/09/2016 22:30:29",
      "content": "<p>You're welcome:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143670,
      "author_name": "",
      "author_url": "",
      "post_date": "11/10/2016 02:13:11",
      "content": "<p>Impressive work :) And thumb-up for #3 :D</p>\n\n<p>[quote=xi;142931]</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.\n[/quote]</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143675,
      "author_name": "iris2007",
      "author_url": "",
      "post_date": "11/10/2016 04:09:00",
      "content": "<p>Hi Xi:</p>\n\n<p>I was also impressed with your solution and decided to give a try and to learn more on Functional Programming.</p>\n\n<p>First script (creating magic features) was completed successfully and 6 zip files were generated.\nWhen I tried running the second (model), it failed with </p>\n\n<p>Out of Memory exception</p>\n\n<ol>\n<li><p>I run script from VS2015 and from command line, and it fails when it comes to reading the fourth file</p></li>\n<li><p>I tried to work only with numeric and magic files, but it loaded 3 files and then failed (Out Of Memory)</p></li>\n<li><p>My system is 64-bit with 8GB RAM</p></li>\n</ol>\n\n<p>Can you please advise how to improve efficiency by releasing vars, when they are not needed. How to better manage memory in F#. Is it necessary to load all files at the beginning?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143703,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/10/2016 07:59:25",
      "content": "<p>How big are your zip files? All the data should fit in under 4GB RAM. The zip files are not really compressed so what is on disk gets loaded into memory 1:1.</p>\n\n<p>You can run it line by line and monitor memory usage in Task Manager. Does it get above 4GB?</p>\n\n<p>Did you make any changes to the scripts?</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143736,
      "author_name": "jiandong",
      "author_url": "",
      "post_date": "11/10/2016 12:29:55",
      "content": "<p>thank you </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143772,
      "author_name": "jianshefeng",
      "author_url": "",
      "post_date": "11/10/2016 15:52:27",
      "content": "<p>[quote=Ayman Khalafallah;143375]</p>\n\n<p>I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.</p>\n\n<p>[/quote]</p>\n\n<p>May I ask how many variables did you choose after Fisher exact Test? Thanks.</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143774,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/10/2016 16:03:26",
      "content": "<p>I think around 6000 </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143776,
      "author_name": "jianshefeng",
      "author_url": "",
      "post_date": "11/10/2016 16:05:42",
      "content": "<p>[quote=xi;143774]</p>\n\n<p>I think around 6000 </p>\n\n<p>[/quote]\nGot it. Thanks~</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143810,
      "author_name": "jianshefeng",
      "author_url": "",
      "post_date": "11/10/2016 19:47:39",
      "content": "<p>[quote=xi;142931]</p>\n\n<p>Hi everyone,</p>\n\n<p>F# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:</p>\n\n<p><a href=\"https://github.com/Statfactory/Kaggle_Bosch\">https://github.com/Statfactory/Kaggle_Bosch</a></p>\n\n<p>You can run it in Visual Studio or VS Code (Windows only for now).</p>\n\n<p>My approach:</p>\n\n<ol>\n<li>Create &quot;magic&quot; features.</li>\n</ol>\n\n<p>The first script will merge train and test data by ID and minimum date from date dataset (&quot;StartDate&quot;). We can then identify a &quot;series&quot; as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various &quot;magic&quot; features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\nYou should be able to easily modify the script and add even more features.</p>\n\n<ol start=\"2\">\n<li>All categorical, numeric and &quot;magic&quot; data is imported into a binary format. </li>\n</ol>\n\n<p>This format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.</p>\n\n<ol start=\"3\">\n<li>One hot encode all data.</li>\n</ol>\n\n<p>All categoricals are &quot;one hot encoded&quot; and numerics  &quot;cut&quot; into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).</p>\n\n<p>Then I run Fisher exact test for each &quot;one hot&quot; against response and leave a subset of significant ones for modelling.</p>\n\n<ol start=\"4\">\n<li>Modelling uses my F# implementation of XGBoost.</li>\n</ol>\n\n<p>I separately model &quot;series&quot; data and &quot;not series&quot; data. The API is similar to original xgboost. </p>\n\n<p>The whole process will take a few hours with most of the time spent on creating magic features and importing the data.</p>\n\n<p>Have fun!</p>\n\n<p>Any feedback welcome:)</p>\n\n<p>Adam </p>\n\n<p>[/quote]\nHi xi, I am trying to repeat the 3rd part in your post: Fisher exact Test to all hot encoder data. However, Rstudio give a wrong reminder that: FEXACT error 40.\nOut of workspace.\nIt seems impossible to use all the train rows in each column to perform the Fisher exact Test. Did you do  some sampling operations in this step? Thanks! </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143816,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/10/2016 20:47:15",
      "content": "<p>No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143827,
      "author_name": "agavranis",
      "author_url": "",
      "post_date": "11/10/2016 22:33:09",
      "content": "<p>I give up, same error again,. Anyway thanks xi for the insights, and your interesting approach but with lack of proper documentation and a black box xi.dll not much can be done. Waiting for your update on further development of the library!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143828,
      "author_name": "jianshefeng",
      "author_url": "",
      "post_date": "11/10/2016 22:36:19",
      "content": "<p>[quote=xi;143816]</p>\n\n<p>No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. </p>\n\n<p>[/quote]\nI think the issue is caused by function fisher.test in R. Now I am trying try write a fisher test func by myself. Thanks. </p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143830,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/10/2016 22:43:14",
      "content": "<p>Gavrand, I can create a Webex online meeting and remotely fix your problem if you want.</p>\n\n<p>Or watch this space, I will create F# starter kits for other competitions. Best place to test data science software:)</p>\n\n<p>&#958; |&gt; &#955; :D</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143837,
      "author_name": "iris2007",
      "author_url": "",
      "post_date": "11/11/2016 00:09:38",
      "content": "<p>I managed to execute it. It helped me to pass that 0.43 threshold. What is more important, I become somewhat familiar with F# and Funct. Programming. Thanks Xi!</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143838,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/11/2016 00:11:15",
      "content": "<p>Great:)</p>",
      "votes": null,
      "replies": []
    },
    {
      "id": 143901,
      "author_name": "neostat",
      "author_url": "",
      "post_date": "11/11/2016 09:49:13",
      "content": "<p>I have noticed some of you were probably confused by async data import (DataImport.importCsvAsync).</p>\n\n<p>With F# Async.Start the import starts on a background thread so you might think that nothing is happening. I have changed it now to Async.RunSynchronously</p>\n\n<p>Also, &quot;faultFreeAll&quot;  was using Single.NaN.ToString() which depends on your system local language and might cause issues when the values are parsed later on. So I changed it now to String.Empty.</p>\n\n<p>&#958; </p>",
      "votes": null,
      "replies": []
    }
  ],
  "raw_markdown_by_id": {
    "142931": "Hi everyone,\r\n\r\nF# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:\r\n\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nYou can run it in Visual Studio or VS Code (Windows only for now).\r\n\r\nMy approach:\r\n\r\n1. Create \"magic\" features.\r\n\r\nThe first script will merge train and test data by ID and minimum date from date dataset (\"StartDate\"). We can then identify a \"series\" as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various \"magic\" features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\r\nYou should be able to easily modify the script and add even more features.\r\n\r\n2. All categorical, numeric and \"magic\" data is imported into a binary format. \r\n\r\nThis format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n\r\n4. Modelling uses my F# implementation of XGBoost.\r\n\r\nI separately model \"series\" data and \"not series\" data. The API is similar to original xgboost. \r\n\r\nThe whole process will take a few hours with most of the time spent on creating magic features and importing the data.\r\n\r\nHave fun!\r\n\r\nAny feedback welcome:)\r\n\r\nAdam",
    "143043": "This is really impressive. Nice work :)",
    "143044": "Thanks:)",
    "143273": "Thanks xi, nice work.\r\nTried to reproduce your code, create_magic_features.fsx works but model.fsx throws MemoryException Error on a Win7 PC  with 16GB RAM.\r\n\r\nmodel.fsx(13,17): error FS3033: The type provider 'Xi.XiTypeProvider' reported an error: Exception of type 'System.OutOfMemoryException' was thrown.\r\nAlso tried FsiAnyCPU.exe 64bit version of Fsi but no luck.\r\nAny ideas?",
    "143276": "Hi,\r\nPlease run in fsi:\r\nSystem.Environment.Is64BitProcess\r\nto verify it is 64 bit version\r\n\r\nAdam",
    "143279": "System.Environment.Is64BitProcess = True",
    "143281": "Are you running it in VS2015?\r\nWhen you run the code line after line, which one throws the exception?",
    "143287": "Tried both in VS Code and command line fsi same error\r\nCode line  \r\ntype NumTrain = XiDataFrame< \"train_numeric.csv.zip\" >\r\nthrows the exception.  Same with line\r\ntype NumTest = XiDataFrame< \"test_numeric.csv.zip\" >\r\nSeems to trace to XiDataFrame thus referenced to Xi.dll, correct? So maybe some problem with the dll?",
    "143288": "Did you import the numeric csv data using Xi or just zipped it?\r\n\r\nThe line in model.fsx is:\r\n\r\ntype NumTest = XiDataFrame< \"test_numeric.zip\" >\r\n\r\nIt should be imported with:\r\n\r\nDataImport.importCsvAsync testNumPath [|','|] [||] Int32.MaxValue 10000 0.1 true String.Empty CompressionLevel.NoCompression",
    "143292": "Just zipped it, I'll give it a try tomorrow and let you know, thanks again!",
    "143319": "Thank you very much.\r\n\r\nI got an Error\r\n\r\nThe type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.\r\n\r\nVS 2015",
    "143351": "Did you just zip the csv data or import it with DataImport.importCsvAsync?",
    "143357": "I imported it. With DataImport as the script. it is complaining about XiDataFrame and says \r\n\r\nThe type provider 'Xi.XiTypeProvider' reported an error: The method or operation is not implemented.",
    "143358": "When you run it line by line, which line throws the error?\r\nAre you running it in VS2015 F# Interactive?",
    "143359": "This line\r\n\r\ntype NumTrain = XiDataFrame< \"train_numeric.zip\" >\r\n\r\nI ran it from VS2015 F# Interactive and I also tired command line fsi.exe",
    "143361": "do you see that error when you just open the script in F# interactive or only when you run it?",
    "143363": "In Both Cases.",
    "143364": "How about the other zip files? Do they also have a compile error?\r\n\r\nDid you make any changes to the create_magic_features script?",
    "143367": "Thanks. I found the error as I changed the directory.",
    "143375": "I can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.",
    "143376": "Great, many thanks:)",
    "143390": "You are welcome. Any plans on giving more documentation on Xi library? That would be great.",
    "143400": "[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?",
    "143401": "This is still very much work in progress so not much documentation:(\r\n\r\nFor people interested in using F# and Xi in data science research or Kaggle competitions I could create a Slack channel or something similar.\r\n\r\nF# Kernels would be really useful:)",
    "143402": "[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.",
    "143410": "[quote=Ayman Khalafallah;143402]\r\n\r\n[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.\r\n\r\n[/quote]\r\n\r\nOK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct?",
    "143411": "[quote=VictorGarcia;143410]\r\n\r\n[quote=Ayman Khalafallah;143402]\r\n\r\n[quote=VictorGarcia;143400]\r\n\r\n[quote=Ayman Khalafallah;143367]\r\n\r\nThanks. I found the error as I changed the directory.\r\n\r\n[/quote]\r\nI was getting the same \"implementation error not permitted\" too. What error did you find/what did you fix?\r\n\r\n[/quote]\r\nI was getting The method or operation is not implemented because I was putting the zip files in different directory than the script.\r\n\r\n[/quote]\r\n\r\nOK. So all you did was place the .zip files like train_numeric.zip in the same directory as the model.fsx file correct? \r\n\r\n[/quote]\r\nYes and also make sure that you run in 64bit mode.",
    "143620": "error FS0193: internal error: No logical space left to create more user strings. (Exception from HRESULT: 0x80131198)\r\n\r\nat the line\r\n\r\ntype MagicTest = XiDataFrame< \"test_magic.zip\" >\r\nlet magicTest = new MagicTest()\r\n\r\nPrevious zips load fine but last one throws an error\r\nAny suggestions?",
    "143621": "This usually happens when you import a data column with a lot of unique numeric values as a categorical factor.\r\n\r\nMake sure you import the data with exactly the same import settings as in the original script.\r\n\r\nAdam",
    "143623": "I imported it using\r\n\r\ndo! DataImport.importCsvAsync testMagicPath [|','|] [||] Int32.MaxValue 10000 0.0 true String.Empty \r\nCompressionLevel.NoCompression",
    "143625": "Did you modify the script in any way?\r\n\r\nIs the zip file 353 MB?",
    "143627": "No, used original script\r\nJust run async imports separately because create_magic_features.fsx ended without creating the zips\r\n\r\nFilesize \r\n345.152 Kb \r\n\r\nMD5 Checksum\r\n\r\n9e114c98a0e2a4dfdb76230304efc1be",
    "143629": "When you unzip the file, you should see \"test_magic\" file inside. How big is it?\r\n\r\nAre you running this in 64bit fsi?",
    "143631": "Just noticed that test_magic.csv contain some weird MinFaultFree values instead of nans in covariates . More precisely it contains NaN but in my local language. Maybe some unicode problem creating the file?",
    "143633": "So what should these MinFaultFree values be instead \"NA\", \"NaN\" or something else?",
    "143634": "That might explain. Try to make them empty.",
    "143639": "Thanks for your help, its my first experience with F# and your help has been very valuable.  I'll try and let you know.",
    "143640": "You're welcome:)",
    "143670": "Impressive work :) And thumb-up for #3 :D\r\n\r\n[quote=xi;142931]\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n[/quote]",
    "143675": "Hi Xi:\r\n\r\nI was also impressed with your solution and decided to give a try and to learn more on Functional Programming.\r\n\r\nFirst script (creating magic features) was completed successfully and 6 zip files were generated.\r\nWhen I tried running the second (model), it failed with \r\n\r\nOut of Memory exception\r\n\r\n1. I run script from VS2015 and from command line, and it fails when it comes to reading the fourth file\r\n\r\n2. I tried to work only with numeric and magic files, but it loaded 3 files and then failed (Out Of Memory)\r\n\r\n3. My system is 64-bit with 8GB RAM\r\n\r\nCan you please advise how to improve efficiency by releasing vars, when they are not needed. How to better manage memory in F#. Is it necessary to load all files at the beginning?",
    "143703": "How big are your zip files? All the data should fit in under 4GB RAM. The zip files are not really compressed so what is on disk gets loaded into memory 1:1.\r\n\r\nYou can run it line by line and monitor memory usage in Task Manager. Does it get above 4GB?\r\n\r\nDid you make any changes to the scripts?",
    "143736": "thank you",
    "143772": "[quote=Ayman Khalafallah;143375]\r\n\r\nI can confirm that it runs in a machine with 4GB (ran it inside a virtual machine with 4GB of memory) and also gives the score mentioned. Thanks.\r\n\r\n[/quote]\r\n\r\nMay I ask how many variables did you choose after Fisher exact Test? Thanks.",
    "143774": "I think around 6000",
    "143776": "[quote=xi;143774]\r\n\r\nI think around 6000 \r\n\r\n[/quote]\r\nGot it. Thanks~",
    "143810": "[quote=xi;142931]\r\n\r\nHi everyone,\r\n\r\nF# Kernels do not seem to be supported by Kaggle so I have published my scripts on GitHub:\r\n\r\nhttps://github.com/Statfactory/Kaggle_Bosch\r\n\r\nYou can run it in Visual Studio or VS Code (Windows only for now).\r\n\r\nMy approach:\r\n\r\n1. Create \"magic\" features.\r\n\r\nThe first script will merge train and test data by ID and minimum date from date dataset (\"StartDate\"). We can then identify a \"series\" as a group of train or test rows with consecutive IDs and the same StartDate. I think of it as parts from the same device. Various \"magic\" features are created based on that grouping, e.g. length of the series, MinEndDate = MaxEndDate etc. In total almost 200 features are created.\r\nYou should be able to easily modify the script and add even more features.\r\n\r\n2. All categorical, numeric and \"magic\" data is imported into a binary format. \r\n\r\nThis format allows me to load all that sparse data into 3GB RAM and use F# type provider for strong column typing.\r\n\r\n3. One hot encode all data.\r\n\r\nAll categoricals are \"one hot encoded\" and numerics  \"cut\" into interval indicators at each unique value. So I end up with lots 2 levels categorical features. This process is virtual, they are not instantiated in memory (ca 700 000...).\r\n\r\nThen I run Fisher exact test for each \"one hot\" against response and leave a subset of significant ones for modelling.\r\n\r\n4. Modelling uses my F# implementation of XGBoost.\r\n\r\nI separately model \"series\" data and \"not series\" data. The API is similar to original xgboost. \r\n\r\nThe whole process will take a few hours with most of the time spent on creating magic features and importing the data.\r\n\r\nHave fun!\r\n\r\nAny feedback welcome:)\r\n\r\nAdam \r\n\r\n\r\n[/quote]\r\nHi xi, I am trying to repeat the 3rd part in your post: Fisher exact Test to all hot encoder data. However, Rstudio give a wrong reminder that: FEXACT error 40.\r\nOut of workspace.\r\nIt seems impossible to use all the train rows in each column to perform the Fisher exact Test. Did you do  some sampling operations in this step? Thanks!",
    "143816": "No sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop.",
    "143827": "I give up, same error again,. Anyway thanks xi for the insights, and your interesting approach but with lack of proper documentation and a black box xi.dll not much can be done. Waiting for your update on further development of the library!",
    "143828": "[quote=xi;143816]\r\n\r\nNo sampling, all rows and one hot encoded columns have been used, so ca 700 000 Fisher tests are performed. It takes about 1h on my laptop. \r\n\r\n[/quote]\r\nI think the issue is caused by function fisher.test in R. Now I am trying try write a fisher test func by myself. Thanks.",
    "143830": "Gavrand, I can create a Webex online meeting and remotely fix your problem if you want.\r\n\r\nOr watch this space, I will create F# starter kits for other competitions. Best place to test data science software:)\r\n\r\nξ |> λ :D",
    "143837": "I managed to execute it. It helped me to pass that 0.43 threshold. What is more important, I become somewhat familiar with F# and Funct. Programming. Thanks Xi!",
    "143838": "Great:)",
    "143901": "I have noticed some of you were probably confused by async data import (DataImport.importCsvAsync).\r\n\r\nWith F# Async.Start the import starts on a background thread so you might think that nothing is happening. I have changed it now to Async.RunSynchronously\r\n\r\nAlso, \"faultFreeAll\"  was using Single.NaN.ToString() which depends on your system local language and might cause issues when the values are parsed later on. So I changed it now to String.Empty.\r\n\r\nξ"
  },
  "source": "meta"
}