{ "id": 23135, "title": "Questions about categorical.csv", "url": "/competitions/bosch-production-line-performance/discussion/23135", "author_name": "Haom", "post_date": "2016-08-28T06:38:21.287000", "votes": 1, "comment_count": 21, "views": 4323, "content": "

What kind of data you got from categorical.csv? I can only see t1 or t, and nearly most of them are missing data.

\n\n

Another thing, what's the interpretation of the value in the cells of numeric.csv? Some values look like 0.030 or -0.197?

", "messages": [ { "id": 135618, "postDate": "2016-09-15T09:24:36.147Z", "content": "

I am not completely sure, but I think numbers of the categorical data (after removing “T”) are PLC, RTU or SCADA error codes. Probably, they are binary or hex error codes (look at the R code bellow)...

\n\n

Probably, the most frequent value, T1, is "temporal transmission timeout error" when there are communication collisions. This is typical in industrial networks with many involved PLCs or RTUs.

\n\n

The challenge could be to detect PLC, RTU or SCADA models (S7 Siemens, OMROM, ???) in order to understand binary error codes....

\n\n
   # Read columns\nlibrary(data.table)\nNUM.ROWS <- 100000\ntrain_data_cat <- fread(input = "../input/train_categorical.csv",\n                              nrows = NUM.ROWS,\n                              select=2:2140,\n                              header = TRUE,\n                              sep = ",",\n                              stringsAsFactors = FALSE,\n                              na.strings="",\n                              colClasses = "character",\n                              data.table=FALSE)\n\ncategories <- train_data_cat[!is.na(train_data_cat)]\nunique(categories)\n
\n\n

[1] "T1" "T9" "T32" "T48" "T96" "T64" "T97" "T98" "T8" "T4"
\n[11] "T48576" "T512" "T16777248" "T16777232" "T16777557" "T128" "T16" "T6553" "T33554944" "T2"
\n[21] "T1372" "T3" "T5" "T-2147483648" "T83888" "T24" "T262144" "T16777216" "T331648" "T6"
\n[31] "T7" "T786432" "T524288" "T8912896" "T145" "T143" "T256" "T-2147482816" "T1310" "T-21474819"
\n[41] "T32896" "T-2147482432" "T-2147482688" "T-21474872" "T-2147481664" "T1152" "T65536"

\n\n
numeric.cat <- as.numeric(gsub("T","",unique(categories)))\nnumeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\nprint(as.hexmode(numeric.cat))\n\n# as.hexmode() cannot convert numbers > 7FFFFFFF\nnumeric.cat.low <- numeric.cat[numeric.cat<=2147483647]\nprint(as.hexmode(numeric.cat.low))\n
\n\n

[1] "0000001" "0000009" "0000020" "0000030" "0000060" "0000040" "0000061" "0000062" "0000008" "0000004" "000bdc0" "0000200" "1000020" "1000010" "1000155"\n[16] "0000080" "0000010" "0001999" "2000200" "0000002" "000055c" "0000003" "0000005" "00147b0" "0000018" "0040000" "1000000" "0050f80" "0000006" "0000007"\n[31] "00c0000" "0080000" "0880000" "0000091" "000008f" "0000100" "000051e" "0008080" "0000480" "0010000"

\n\n
# Remove more significat bit in big numbers to look them\nnumeric.cat.high <- numeric.cat[numeric.cat>2147483647]\nnumeric.cat.high <- numeric.cat.high - 2^31\nprint(as.hexmode(numeric.cat.high))\n
\n\n

[1] "00000000" "00000340" "7eb851fd" "000004c0" "000003c0" "7eb851c8" "000007c0"

", "rawMarkdown": "I am not completely sure, but I think numbers of the categorical data (after removing “T”) are PLC, RTU or SCADA error codes. Probably, they are binary or hex error codes (look at the R code bellow)... \r\n\r\nProbably, the most frequent value, T1, is \"temporal transmission timeout error\" when there are communication collisions. This is typical in industrial networks with many involved PLCs or RTUs.\r\n\r\nThe challenge could be to detect PLC, RTU or SCADA models (S7 Siemens, OMROM, ???) in order to understand binary error codes....\r\n\r\n \r\n\r\n # Read columns\r\n library(data.table)\r\n NUM.ROWS <- 100000\r\n train_data_cat <- fread(input = \"../input/train_categorical.csv\",\r\n nrows = NUM.ROWS,\r\n select=2:2140,\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n na.strings=\"\",\r\n colClasses = \"character\",\r\n data.table=FALSE)\r\n \r\n categories <- train_data_cat[!is.na(train_data_cat)]\r\n unique(categories)\r\n [1] \"T1\" \"T9\" \"T32\" \"T48\" \"T96\" \"T64\" \"T97\" \"T98\" \"T8\" \"T4\" \r\n[11] \"T48576\" \"T512\" \"T16777248\" \"T16777232\" \"T16777557\" \"T128\" \"T16\" \"T6553\" \"T33554944\" \"T2\" \r\n[21] \"T1372\" \"T3\" \"T5\" \"T-2147483648\" \"T83888\" \"T24\" \"T262144\" \"T16777216\" \"T331648\" \"T6\" \r\n[31] \"T7\" \"T786432\" \"T524288\" \"T8912896\" \"T145\" \"T143\" \"T256\" \"T-2147482816\" \"T1310\" \"T-21474819\" \r\n[41] \"T32896\" \"T-2147482432\" \"T-2147482688\" \"T-21474872\" \"T-2147481664\" \"T1152\" \"T65536\" \r\n\r\n numeric.cat <- as.numeric(gsub(\"T\",\"\",unique(categories)))\r\n numeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\r\n print(as.hexmode(numeric.cat))\r\n \r\n # as.hexmode() cannot convert numbers > 7FFFFFFF\r\n numeric.cat.low <- numeric.cat[numeric.cat<=2147483647]\r\n print(as.hexmode(numeric.cat.low))\r\n \r\n **[1] \"0000001\" \"0000009\" \"0000020\" \"0000030\" \"0000060\" \"0000040\" \"0000061\" \"0000062\" \"0000008\" \"0000004\" \"000bdc0\" \"0000200\" \"1000020\" \"1000010\" \"1000155\"\r\n[16] \"0000080\" \"0000010\" \"0001999\" \"2000200\" \"0000002\" \"000055c\" \"0000003\" \"0000005\" \"00147b0\" \"0000018\" \"0040000\" \"1000000\" \"0050f80\" \"0000006\" \"0000007\"\r\n[31] \"00c0000\" \"0080000\" \"0880000\" \"0000091\" \"000008f\" \"0000100\" \"000051e\" \"0008080\" \"0000480\" \"0010000\"**\r\n\r\n # Remove more significat bit in big numbers to look them\r\n numeric.cat.high <- numeric.cat[numeric.cat>2147483647]\r\n numeric.cat.high <- numeric.cat.high - 2^31\r\n print(as.hexmode(numeric.cat.high))\r\n\r\n**[1] \"00000000\" \"00000340\" \"7eb851fd\" \"000004c0\" \"000003c0\" \"7eb851c8\" \"000007c0\"**", "votes": 12 }, { "id": 137364, "postDate": "2016-09-30T20:07:43.147Z", "content": "

In the attached script you can see that you can consider only 226 categorical features without loosing any info (just dropping 163 constant and 1751 duplicated features). If you play with the frequency of missing values you can reduce the file even more (but now loosing something).

\n\n

I hope it helps...

", "rawMarkdown": "In the attached script you can see that you can consider only 226 categorical features without loosing any info (just dropping 163 constant and 1751 duplicated features). If you play with the frequency of missing values you can reduce the file even more (but now loosing something).\r\n\r\nI hope it helps...", "votes": 5 }, { "id": 132849, "postDate": "2016-08-29T18:45:46.650Z", "content": "

[quote=Jos Theelen;132845]

\n\n

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?

\n\n

[/quote]

\n\n

[quote=Jos Theelen;132845]

\n\n

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?\n[/quote]

\n\n

My guess is they are defect categories.

\n\n

What is not clear to me is whether there is some sort of hierarchical relationship between the codes. For example,

\n\n

T1 = Surface Defect

\n\n

T16 = Surface Defect : Scratch

\n\n

T162 = Surface Defect : Scratch : Deep

\n\n

I don't have any evidence that this is the case from the data, but, it should be considered as a possibility.

", "rawMarkdown": "[quote=Jos Theelen;132845]\r\n\r\nI see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n\r\n\r\n[/quote]\r\n\r\n[quote=Jos Theelen;132845]\r\n\r\nI see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n[/quote]\r\n\r\nMy guess is they are defect categories. \r\n\r\nWhat is not clear to me is whether there is some sort of hierarchical relationship between the codes. For example, \r\n\r\nT1 = Surface Defect\r\n\r\nT16 = Surface Defect : Scratch\r\n\r\nT162 = Surface Defect : Scratch : Deep\r\n\r\nI don't have any evidence that this is the case from the data, but, it should be considered as a possibility.\r\n\r\n", "votes": 6 }, { "id": 132855, "postDate": "2016-08-29T20:36:50.027Z", "content": "

There are 93 categorical values, examples are T4, T8912896, T-2147483646. They fit with this regular expression: ^T-?\\d{1,10}$

", "rawMarkdown": "There are 93 categorical values, examples are T4, T8912896, T-2147483646. They fit with this regular expression: ^T-?\\d{1,10}$\r\n", "votes": 3 }, { "id": 133481, "postDate": "2016-09-02T13:47:41.477Z", "content": "

I wonder what the T would mean. Tech, Test, Type, Transport, Temperature, Technical Error, Teil (german for part), etc. Some numbers seem to be almost binary, like 65536 and 786432.

", "rawMarkdown": "I wonder what the T would mean. Tech, Test, Type, Transport, Temperature, Technical Error, Teil (german for part), etc. Some numbers seem to be almost binary, like 65536 and 786432.", "votes": 1 }, { "id": 132679, "postDate": "2016-08-28T10:50:58.397Z", "content": "

Please find attached the picture showing the distribution of the categorical featues. There are values other than T1 in the categorical.csv. Actually what we need to do is to add a Response column from the numeircal.csv to see its relationship.

\n\n

Kim

", "rawMarkdown": "Please find attached the picture showing the distribution of the categorical featues. There are values other than T1 in the categorical.csv. Actually what we need to do is to add a Response column from the numeircal.csv to see its relationship.\r\n\r\nKim", "votes": 1 }, { "id": 132668, "postDate": "2016-08-28T06:38:21.287Z", "content": "

What kind of data you got from categorical.csv? I can only see t1 or t, and nearly most of them are missing data.

\n\n

Another thing, what's the interpretation of the value in the cells of numeric.csv? Some values look like 0.030 or -0.197?

", "rawMarkdown": "What kind of data you got from categorical.csv? I can only see t1 or t, and nearly most of them are missing data. \r\n\r\nAnother thing, what's the interpretation of the value in the cells of numeric.csv? Some values look like 0.030 or -0.197? ", "votes": 1 }, { "id": 137210, "postDate": "2016-09-29T09:19:00.237Z", "content": "

Hi r4sn5

\n\n

I am not sure but I think numbers are in 32-bit two's complement (see: https://en.wikipedia.org/wiki/Signed_number_representations#Two.27s_complement)

\n\n

For example, in Eight-bit two’s complement -128 is in binary: 1000 0000 (0x80hex)

\n\n

Probably, this is not the best way but to remove the most significant bit (2^7) I did it

\n\n
100000000 (2^8) + (negative_number) – (most_significant_bit)\n2^8 + (-128) -2^7  =    256 -128-128 =   ~000 0000\n
\n\n

other example,

\n\n
-126 is in binary 10000010 \n256 -126 -128 =  2  ( ~000 0010)\n
\n\n

Now, with ‘T-2147483648’ with 32-bit two’s complement:

\n\n
-2147483648 is 1000 0000 0000 0000 0000 0000 0000 0000 (-2^31)\n
\n\n

Then:

\n\n
(2^32) + (-2147483648) – (2^31) = ~000 0000 0000 0000 0000 0000 0000 0000  (0x80000000hex)\n
\n\n

And with other category: "T-2147482432"

\n\n
> as.hexmode((2^32)-2147482432-(2^31))\n\n[1] "4c0"\n~000 000 0000 0000 0000 1000 1100 0000    (0x000004C0hex)\n
\n\n

The question is: Could we consider that each bit has a different meaning?

", "rawMarkdown": "Hi r4sn5\r\n\r\nI am not sure but I think numbers are in 32-bit two's complement (see: https://en.wikipedia.org/wiki/Signed_number_representations#Two.27s_complement)\r\n\r\nFor example, in Eight-bit two’s complement -128 is in binary: 1000 0000 (0x80hex)\r\n\r\nProbably, this is not the best way but to remove the most significant bit (2^7) I did it\r\n\r\n 100000000 (2^8) + (negative_number) – (most_significant_bit)\r\n 2^8 + (-128) -2^7 = 256 -128-128 = ~000 0000\r\n\r\nother example, \r\n\r\n -126 is in binary 10000010 \r\n 256 -126 -128 = 2 ( ~000 0010)\r\n\r\nNow, with ‘T-2147483648’ with 32-bit two’s complement:\r\n\r\n -2147483648 is 1000 0000 0000 0000 0000 0000 0000 0000 (-2^31)\r\n\r\nThen: \r\n\r\n (2^32) + (-2147483648) – (2^31) = ~000 0000 0000 0000 0000 0000 0000 0000 (0x80000000hex)\r\n\r\nAnd with other category: \"T-2147482432\"\r\n\r\n > as.hexmode((2^32)-2147482432-(2^31))\r\n \r\n [1] \"4c0\"\r\n ~000 000 0000 0000 0000 1000 1100 0000 (0x000004C0hex)\r\n\r\n\r\nThe question is: Could we consider that each bit has a different meaning?\r\n\r\n\r\n\r\n\r\n" }, { "id": 137181, "postDate": "2016-09-29T03:18:32.227Z", "content": "

Not 100% sure. It looks like he is trying to remove the negative numbers as this cannot be converted to hexa. He is doing that by creating a big number, maximum as.hexmode() can handle, which is 2^32 and adding it to the negative numbers.\n[quote=r4sn4;137169]

\n\n

Hi \nF.J.Martinez-de-Pison

\n\n

Could you please tell the purpose of doing this -

\n\n

numeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]

\n\n

Thanks

\n\n

[/quote]

", "rawMarkdown": "Not 100% sure. It looks like he is trying to remove the negative numbers as this cannot be converted to hexa. He is doing that by creating a big number, maximum as.hexmode() can handle, which is 2^32 and adding it to the negative numbers.\r\n[quote=r4sn4;137169]\r\n\r\nHi \r\nF.J.Martinez-de-Pison\r\n\r\nCould you please tell the purpose of doing this - \r\n \r\nnumeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\r\n\r\nThanks\r\n\r\n\r\n[/quote]\r\n" }, { "id": 134863, "postDate": "2016-09-08T23:09:18.963Z", "content": "

It makes it somewhat likely that they are some kind of error codes. Which doesn't always mean things are wrong. Some machines also give codes to indicate that they are doing something special.

", "rawMarkdown": "It makes it somewhat likely that they are some kind of error codes. Which doesn't always mean things are wrong. Some machines also give codes to indicate that they are doing something special.\r\n" }, { "id": 134787, "postDate": "2016-09-08T13:04:47.523Z", "content": "

Some interesting facts from http://numbermatics.com/n/786432/

\n\n

Prime factorization: (2^18) * 3\nBinary: 11000000000000000000 2 <- that 2 is a subscript\nHexadecimal: 0xc0000

", "rawMarkdown": " Some interesting facts from http://numbermatics.com/n/786432/\r\n\r\nPrime factorization: (2^18) * 3\r\nBinary: 11000000000000000000 2 <- that 2 is a subscript\r\nHexadecimal: 0xc0000" }, { "id": 134760, "postDate": "2016-09-08T10:11:59.240Z", "content": "

[quote=Jos Theelen;133481]

\n\n

Some numbers seem to be almost binary, like 65536 and 786432.

\n\n

[/quote]

\n\n

Yes! Also the minimum number and maximum number after the T are -2,147,483,648 (or -10000000000000000000000000000000 in binary, also the smallest integer that can be represented in a 32bit binary integer) and 268,435,456 (10000000000000000000000000000 in binary). Would that mean something?

", "rawMarkdown": "[quote=Jos Theelen;133481]\r\n\r\nSome numbers seem to be almost binary, like 65536 and 786432.\r\n\r\n[/quote]\r\n\r\nYes! Also the minimum number and maximum number after the T are -2,147,483,648 (or -10000000000000000000000000000000 in binary, also the smallest integer that can be represented in a 32bit binary integer) and 268,435,456 (10000000000000000000000000000 in binary). Would that mean something?" }, { "id": 134216, "postDate": "2016-09-05T20:52:32.913Z", "content": "

There are 44 categorical columns which are empty in train and test as well.

", "rawMarkdown": "There are 44 categorical columns which are empty in train and test as well." }, { "id": 133681, "postDate": "2016-09-03T09:14:13.177Z", "content": "

@Christian Hirsh, I can confirm that 163 for training data. Haven't check the test data though for the respective columns.

", "rawMarkdown": "@Christian Hirsh, I can confirm that 163 for training data. Haven't check the test data though for the respective columns." }, { "id": 133622, "postDate": "2016-09-02T23:23:08.850Z", "content": "

Yes, there are lots of columns that they have no training data.

", "rawMarkdown": "Yes, there are lots of columns that they have no training data." }, { "id": 133526, "postDate": "2016-09-02T18:12:36.273Z", "content": "

Can anybody confirm that there are 163 categorical columns for which we don't have any valid training data? L0_S3_F69 is the first of them.

", "rawMarkdown": "Can anybody confirm that there are 163 categorical columns for which we don't have any valid training data? *L0_S3_F69* is the first of them." }, { "id": 132845, "postDate": "2016-08-29T18:05:34.400Z", "content": "

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?

", "rawMarkdown": "I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n" }, { "id": 132742, "postDate": "2016-08-28T23:15:24.973Z", "content": "

Hello Kim, \nThank you for sharing the picture, it seems that it was made by ggplot2 in R? Would you mind to share the R code with me? I still cannot find any features except some "NA"... Maybe something was wrong for some reasons, I tried to open the csv with data.table.\nThanks again.

", "rawMarkdown": "Hello Kim, \r\nThank you for sharing the picture, it seems that it was made by ggplot2 in R? Would you mind to share the R code with me? I still cannot find any features except some \"NA\"... Maybe something was wrong for some reasons, I tried to open the csv with data.table.\r\nThanks again." }, { "id": 132671, "postDate": "2016-08-28T07:47:32.980Z", "content": "

Thank you so much!\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?

", "rawMarkdown": "Thank you so much!\r\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?\r\n\r\n" }, { "id": 132670, "postDate": "2016-08-28T07:34:53.343Z", "content": "

Thank you so much!\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?

", "rawMarkdown": "Thank you so much!\r\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?\r\n\r\n" }, { "id": 132669, "postDate": "2016-08-28T07:20:22.637Z", "content": "

1) Even if you don't build "smart" features, there's a lot of features in there in raw format (as is). Keep looking.

\n\n

2) None of us have idea (at least, to be sure about it), it's anonymous data.

", "rawMarkdown": "1) Even if you don't build \"smart\" features, there's a lot of features in there in raw format (as is). Keep looking.\r\n\r\n2) None of us have idea (at least, to be sure about it), it's anonymous data." }, { "id": 137169, "postDate": "2016-09-29T00:41:51.960Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 135618, "author_name": "F.J.Martinez-de-Pison", "author_url": "", "post_date": "2016-09-15T09:24:36.147000", "content": "

I am not completely sure, but I think numbers of the categorical data (after removing “T”) are PLC, RTU or SCADA error codes. Probably, they are binary or hex error codes (look at the R code bellow)...

\n\n

Probably, the most frequent value, T1, is "temporal transmission timeout error" when there are communication collisions. This is typical in industrial networks with many involved PLCs or RTUs.

\n\n

The challenge could be to detect PLC, RTU or SCADA models (S7 Siemens, OMROM, ???) in order to understand binary error codes....

\n\n
   # Read columns\nlibrary(data.table)\nNUM.ROWS <- 100000\ntrain_data_cat <- fread(input = "../input/train_categorical.csv",\n                              nrows = NUM.ROWS,\n                              select=2:2140,\n                              header = TRUE,\n                              sep = ",",\n                              stringsAsFactors = FALSE,\n                              na.strings="",\n                              colClasses = "character",\n                              data.table=FALSE)\n\ncategories <- train_data_cat[!is.na(train_data_cat)]\nunique(categories)\n
\n\n

[1] "T1" "T9" "T32" "T48" "T96" "T64" "T97" "T98" "T8" "T4"
\n[11] "T48576" "T512" "T16777248" "T16777232" "T16777557" "T128" "T16" "T6553" "T33554944" "T2"
\n[21] "T1372" "T3" "T5" "T-2147483648" "T83888" "T24" "T262144" "T16777216" "T331648" "T6"
\n[31] "T7" "T786432" "T524288" "T8912896" "T145" "T143" "T256" "T-2147482816" "T1310" "T-21474819"
\n[41] "T32896" "T-2147482432" "T-2147482688" "T-21474872" "T-2147481664" "T1152" "T65536"

\n\n
numeric.cat <- as.numeric(gsub("T","",unique(categories)))\nnumeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\nprint(as.hexmode(numeric.cat))\n\n# as.hexmode() cannot convert numbers > 7FFFFFFF\nnumeric.cat.low <- numeric.cat[numeric.cat<=2147483647]\nprint(as.hexmode(numeric.cat.low))\n
\n\n

[1] "0000001" "0000009" "0000020" "0000030" "0000060" "0000040" "0000061" "0000062" "0000008" "0000004" "000bdc0" "0000200" "1000020" "1000010" "1000155"\n[16] "0000080" "0000010" "0001999" "2000200" "0000002" "000055c" "0000003" "0000005" "00147b0" "0000018" "0040000" "1000000" "0050f80" "0000006" "0000007"\n[31] "00c0000" "0080000" "0880000" "0000091" "000008f" "0000100" "000051e" "0008080" "0000480" "0010000"

\n\n
# Remove more significat bit in big numbers to look them\nnumeric.cat.high <- numeric.cat[numeric.cat>2147483647]\nnumeric.cat.high <- numeric.cat.high - 2^31\nprint(as.hexmode(numeric.cat.high))\n
\n\n

[1] "00000000" "00000340" "7eb851fd" "000004c0" "000003c0" "7eb851c8" "000007c0"

", "votes": 12, "replies": [] }, { "id": 137364, "author_name": "Manel", "author_url": "", "post_date": "2016-09-30T20:07:43.147000", "content": "

In the attached script you can see that you can consider only 226 categorical features without loosing any info (just dropping 163 constant and 1751 duplicated features). If you play with the frequency of missing values you can reduce the file even more (but now loosing something).

\n\n

I hope it helps...

", "votes": 5, "replies": [] }, { "id": 132849, "author_name": "inversion", "author_url": "", "post_date": "2016-08-29T18:45:46.650000", "content": "

[quote=Jos Theelen;132845]

\n\n

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?

\n\n

[/quote]

\n\n

[quote=Jos Theelen;132845]

\n\n

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?\n[/quote]

\n\n

My guess is they are defect categories.

\n\n

What is not clear to me is whether there is some sort of hierarchical relationship between the codes. For example,

\n\n

T1 = Surface Defect

\n\n

T16 = Surface Defect : Scratch

\n\n

T162 = Surface Defect : Scratch : Deep

\n\n

I don't have any evidence that this is the case from the data, but, it should be considered as a possibility.

", "votes": 6, "replies": [] }, { "id": 132855, "author_name": "Attila Babo", "author_url": "", "post_date": "2016-08-29T20:36:50.027000", "content": "

There are 93 categorical values, examples are T4, T8912896, T-2147483646. They fit with this regular expression: ^T-?\\d{1,10}$

", "votes": 3, "replies": [] }, { "id": 133481, "author_name": "Jos Theelen", "author_url": "", "post_date": "2016-09-02T13:47:41.477000", "content": "

I wonder what the T would mean. Tech, Test, Type, Transport, Temperature, Technical Error, Teil (german for part), etc. Some numbers seem to be almost binary, like 65536 and 786432.

", "votes": 1, "replies": [] }, { "id": 132679, "author_name": "Kim Tang", "author_url": "", "post_date": "2016-08-28T10:50:58.397000", "content": "

Please find attached the picture showing the distribution of the categorical featues. There are values other than T1 in the categorical.csv. Actually what we need to do is to add a Response column from the numeircal.csv to see its relationship.

\n\n

Kim

", "votes": 1, "replies": [] }, { "id": 137210, "author_name": "F.J.Martinez-de-Pison", "author_url": "", "post_date": "2016-09-29T09:19:00.237000", "content": "

Hi r4sn5

\n\n

I am not sure but I think numbers are in 32-bit two's complement (see: https://en.wikipedia.org/wiki/Signed_number_representations#Two.27s_complement)

\n\n

For example, in Eight-bit two’s complement -128 is in binary: 1000 0000 (0x80hex)

\n\n

Probably, this is not the best way but to remove the most significant bit (2^7) I did it

\n\n
100000000 (2^8) + (negative_number) – (most_significant_bit)\n2^8 + (-128) -2^7  =    256 -128-128 =   ~000 0000\n
\n\n

other example,

\n\n
-126 is in binary 10000010 \n256 -126 -128 =  2  ( ~000 0010)\n
\n\n

Now, with ‘T-2147483648’ with 32-bit two’s complement:

\n\n
-2147483648 is 1000 0000 0000 0000 0000 0000 0000 0000 (-2^31)\n
\n\n

Then:

\n\n
(2^32) + (-2147483648) – (2^31) = ~000 0000 0000 0000 0000 0000 0000 0000  (0x80000000hex)\n
\n\n

And with other category: "T-2147482432"

\n\n
> as.hexmode((2^32)-2147482432-(2^31))\n\n[1] "4c0"\n~000 000 0000 0000 0000 1000 1100 0000    (0x000004C0hex)\n
\n\n

The question is: Could we consider that each bit has a different meaning?

", "votes": 0, "replies": [] }, { "id": 137181, "author_name": "Anon", "author_url": "", "post_date": "2016-09-29T03:18:32.227000", "content": "

Not 100% sure. It looks like he is trying to remove the negative numbers as this cannot be converted to hexa. He is doing that by creating a big number, maximum as.hexmode() can handle, which is 2^32 and adding it to the negative numbers.\n[quote=r4sn4;137169]

\n\n

Hi \nF.J.Martinez-de-Pison

\n\n

Could you please tell the purpose of doing this -

\n\n

numeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]

\n\n

Thanks

\n\n

[/quote]

", "votes": 0, "replies": [] }, { "id": 134863, "author_name": "Jos Theelen", "author_url": "", "post_date": "2016-09-08T23:09:18.963000", "content": "

It makes it somewhat likely that they are some kind of error codes. Which doesn't always mean things are wrong. Some machines also give codes to indicate that they are doing something special.

", "votes": 0, "replies": [] }, { "id": 134787, "author_name": "Brent Brewington", "author_url": "", "post_date": "2016-09-08T13:04:47.523000", "content": "

Some interesting facts from http://numbermatics.com/n/786432/

\n\n

Prime factorization: (2^18) * 3\nBinary: 11000000000000000000 2 <- that 2 is a subscript\nHexadecimal: 0xc0000

", "votes": 0, "replies": [] }, { "id": 134760, "author_name": "LMNU", "author_url": "", "post_date": "2016-09-08T10:11:59.240000", "content": "

[quote=Jos Theelen;133481]

\n\n

Some numbers seem to be almost binary, like 65536 and 786432.

\n\n

[/quote]

\n\n

Yes! Also the minimum number and maximum number after the T are -2,147,483,648 (or -10000000000000000000000000000000 in binary, also the smallest integer that can be represented in a 32bit binary integer) and 268,435,456 (10000000000000000000000000000 in binary). Would that mean something?

", "votes": 0, "replies": [] }, { "id": 134216, "author_name": "Attila Babo", "author_url": "", "post_date": "2016-09-05T20:52:32.913000", "content": "

There are 44 categorical columns which are empty in train and test as well.

", "votes": 0, "replies": [] }, { "id": 133681, "author_name": "Attila Babo", "author_url": "", "post_date": "2016-09-03T09:14:13.177000", "content": "

@Christian Hirsh, I can confirm that 163 for training data. Haven't check the test data though for the respective columns.

", "votes": 0, "replies": [] }, { "id": 133622, "author_name": "Mojtaba", "author_url": "", "post_date": "2016-09-02T23:23:08.850000", "content": "

Yes, there are lots of columns that they have no training data.

", "votes": 0, "replies": [] }, { "id": 133526, "author_name": "Christian Hirsch", "author_url": "", "post_date": "2016-09-02T18:12:36.273000", "content": "

Can anybody confirm that there are 163 categorical columns for which we don't have any valid training data? L0_S3_F69 is the first of them.

", "votes": 0, "replies": [] }, { "id": 132845, "author_name": "Jos Theelen", "author_url": "", "post_date": "2016-08-29T18:05:34.400000", "content": "

I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\nHas somebody another idea?

", "votes": 0, "replies": [] }, { "id": 132742, "author_name": "Haom", "author_url": "", "post_date": "2016-08-28T23:15:24.973000", "content": "

Hello Kim, \nThank you for sharing the picture, it seems that it was made by ggplot2 in R? Would you mind to share the R code with me? I still cannot find any features except some "NA"... Maybe something was wrong for some reasons, I tried to open the csv with data.table.\nThanks again.

", "votes": 0, "replies": [] }, { "id": 132671, "author_name": "Haom", "author_url": "", "post_date": "2016-08-28T07:47:32.980000", "content": "

Thank you so much!\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?

", "votes": 0, "replies": [] }, { "id": 132670, "author_name": "Haom", "author_url": "", "post_date": "2016-08-28T07:34:53.343000", "content": "

Thank you so much!\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?

", "votes": 0, "replies": [] }, { "id": 132669, "author_name": "NxGTR", "author_url": "", "post_date": "2016-08-28T07:20:22.637000", "content": "

1) Even if you don't build "smart" features, there's a lot of features in there in raw format (as is). Keep looking.

\n\n

2) None of us have idea (at least, to be sure about it), it's anonymous data.

", "votes": 0, "replies": [] }, { "id": 137169, "author_name": "", "author_url": "", "post_date": "2016-09-29T00:41:51.960000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "135618": "I am not completely sure, but I think numbers of the categorical data (after removing “T”) are PLC, RTU or SCADA error codes. Probably, they are binary or hex error codes (look at the R code bellow)... \r\n\r\nProbably, the most frequent value, T1, is \"temporal transmission timeout error\" when there are communication collisions. This is typical in industrial networks with many involved PLCs or RTUs.\r\n\r\nThe challenge could be to detect PLC, RTU or SCADA models (S7 Siemens, OMROM, ???) in order to understand binary error codes....\r\n\r\n \r\n\r\n # Read columns\r\n library(data.table)\r\n NUM.ROWS <- 100000\r\n train_data_cat <- fread(input = \"../input/train_categorical.csv\",\r\n nrows = NUM.ROWS,\r\n select=2:2140,\r\n header = TRUE,\r\n sep = \",\",\r\n stringsAsFactors = FALSE,\r\n na.strings=\"\",\r\n colClasses = \"character\",\r\n data.table=FALSE)\r\n \r\n categories <- train_data_cat[!is.na(train_data_cat)]\r\n unique(categories)\r\n [1] \"T1\" \"T9\" \"T32\" \"T48\" \"T96\" \"T64\" \"T97\" \"T98\" \"T8\" \"T4\" \r\n[11] \"T48576\" \"T512\" \"T16777248\" \"T16777232\" \"T16777557\" \"T128\" \"T16\" \"T6553\" \"T33554944\" \"T2\" \r\n[21] \"T1372\" \"T3\" \"T5\" \"T-2147483648\" \"T83888\" \"T24\" \"T262144\" \"T16777216\" \"T331648\" \"T6\" \r\n[31] \"T7\" \"T786432\" \"T524288\" \"T8912896\" \"T145\" \"T143\" \"T256\" \"T-2147482816\" \"T1310\" \"T-21474819\" \r\n[41] \"T32896\" \"T-2147482432\" \"T-2147482688\" \"T-21474872\" \"T-2147481664\" \"T1152\" \"T65536\" \r\n\r\n numeric.cat <- as.numeric(gsub(\"T\",\"\",unique(categories)))\r\n numeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\r\n print(as.hexmode(numeric.cat))\r\n \r\n # as.hexmode() cannot convert numbers > 7FFFFFFF\r\n numeric.cat.low <- numeric.cat[numeric.cat<=2147483647]\r\n print(as.hexmode(numeric.cat.low))\r\n \r\n **[1] \"0000001\" \"0000009\" \"0000020\" \"0000030\" \"0000060\" \"0000040\" \"0000061\" \"0000062\" \"0000008\" \"0000004\" \"000bdc0\" \"0000200\" \"1000020\" \"1000010\" \"1000155\"\r\n[16] \"0000080\" \"0000010\" \"0001999\" \"2000200\" \"0000002\" \"000055c\" \"0000003\" \"0000005\" \"00147b0\" \"0000018\" \"0040000\" \"1000000\" \"0050f80\" \"0000006\" \"0000007\"\r\n[31] \"00c0000\" \"0080000\" \"0880000\" \"0000091\" \"000008f\" \"0000100\" \"000051e\" \"0008080\" \"0000480\" \"0010000\"**\r\n\r\n # Remove more significat bit in big numbers to look them\r\n numeric.cat.high <- numeric.cat[numeric.cat>2147483647]\r\n numeric.cat.high <- numeric.cat.high - 2^31\r\n print(as.hexmode(numeric.cat.high))\r\n\r\n**[1] \"00000000\" \"00000340\" \"7eb851fd\" \"000004c0\" \"000003c0\" \"7eb851c8\" \"000007c0\"**", "137364": "In the attached script you can see that you can consider only 226 categorical features without loosing any info (just dropping 163 constant and 1751 duplicated features). If you play with the frequency of missing values you can reduce the file even more (but now loosing something).\r\n\r\nI hope it helps...", "132849": "[quote=Jos Theelen;132845]\r\n\r\nI see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n\r\n\r\n[/quote]\r\n\r\n[quote=Jos Theelen;132845]\r\n\r\nI see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n[/quote]\r\n\r\nMy guess is they are defect categories. \r\n\r\nWhat is not clear to me is whether there is some sort of hierarchical relationship between the codes. For example, \r\n\r\nT1 = Surface Defect\r\n\r\nT16 = Surface Defect : Scratch\r\n\r\nT162 = Surface Defect : Scratch : Deep\r\n\r\nI don't have any evidence that this is the case from the data, but, it should be considered as a possibility.\r\n\r\n", "132855": "There are 93 categorical values, examples are T4, T8912896, T-2147483646. They fit with this regular expression: ^T-?\\d{1,10}$\r\n", "133481": "I wonder what the T would mean. Tech, Test, Type, Transport, Temperature, Technical Error, Teil (german for part), etc. Some numbers seem to be almost binary, like 65536 and 786432.", "132679": "Please find attached the picture showing the distribution of the categorical featues. There are values other than T1 in the categorical.csv. Actually what we need to do is to add a Response column from the numeircal.csv to see its relationship.\r\n\r\nKim", "132668": "What kind of data you got from categorical.csv? I can only see t1 or t, and nearly most of them are missing data. \r\n\r\nAnother thing, what's the interpretation of the value in the cells of numeric.csv? Some values look like 0.030 or -0.197? ", "137210": "Hi r4sn5\r\n\r\nI am not sure but I think numbers are in 32-bit two's complement (see: https://en.wikipedia.org/wiki/Signed_number_representations#Two.27s_complement)\r\n\r\nFor example, in Eight-bit two’s complement -128 is in binary: 1000 0000 (0x80hex)\r\n\r\nProbably, this is not the best way but to remove the most significant bit (2^7) I did it\r\n\r\n 100000000 (2^8) + (negative_number) – (most_significant_bit)\r\n 2^8 + (-128) -2^7 = 256 -128-128 = ~000 0000\r\n\r\nother example, \r\n\r\n -126 is in binary 10000010 \r\n 256 -126 -128 = 2 ( ~000 0010)\r\n\r\nNow, with ‘T-2147483648’ with 32-bit two’s complement:\r\n\r\n -2147483648 is 1000 0000 0000 0000 0000 0000 0000 0000 (-2^31)\r\n\r\nThen: \r\n\r\n (2^32) + (-2147483648) – (2^31) = ~000 0000 0000 0000 0000 0000 0000 0000 (0x80000000hex)\r\n\r\nAnd with other category: \"T-2147482432\"\r\n\r\n > as.hexmode((2^32)-2147482432-(2^31))\r\n \r\n [1] \"4c0\"\r\n ~000 000 0000 0000 0000 1000 1100 0000 (0x000004C0hex)\r\n\r\n\r\nThe question is: Could we consider that each bit has a different meaning?\r\n\r\n\r\n\r\n\r\n", "137181": "Not 100% sure. It looks like he is trying to remove the negative numbers as this cannot be converted to hexa. He is doing that by creating a big number, maximum as.hexmode() can handle, which is 2^32 and adding it to the negative numbers.\r\n[quote=r4sn4;137169]\r\n\r\nHi \r\nF.J.Martinez-de-Pison\r\n\r\nCould you please tell the purpose of doing this - \r\n \r\nnumeric.cat[numeric.cat<0] <- (2^32)+numeric.cat[numeric.cat<0]\r\n\r\nThanks\r\n\r\n\r\n[/quote]\r\n", "134863": "It makes it somewhat likely that they are some kind of error codes. Which doesn't always mean things are wrong. Some machines also give codes to indicate that they are doing something special.\r\n", "134787": " Some interesting facts from http://numbermatics.com/n/786432/\r\n\r\nPrime factorization: (2^18) * 3\r\nBinary: 11000000000000000000 2 <- that 2 is a subscript\r\nHexadecimal: 0xc0000", "134760": "[quote=Jos Theelen;133481]\r\n\r\nSome numbers seem to be almost binary, like 65536 and 786432.\r\n\r\n[/quote]\r\n\r\nYes! Also the minimum number and maximum number after the T are -2,147,483,648 (or -10000000000000000000000000000000 in binary, also the smallest integer that can be represented in a 32bit binary integer) and 268,435,456 (10000000000000000000000000000 in binary). Would that mean something?", "134216": "There are 44 categorical columns which are empty in train and test as well.", "133681": "@Christian Hirsh, I can confirm that 163 for training data. Haven't check the test data though for the respective columns.", "133622": "Yes, there are lots of columns that they have no training data.", "133526": "Can anybody confirm that there are 163 categorical columns for which we don't have any valid training data? *L0_S3_F69* is the first of them.", "132845": "I see more than T1. I also see T3, T4, T5, T16, T145, T65536 and even things likeT786432. And I only looked at the first 3000 records. I still don't know what these categories are, maybe settings of the stations? I assume that stations can do more than one thing, and in that case, there has to be some indication in what state a station is.\r\nHas somebody another idea?\r\n", "132742": "Hello Kim, \r\nThank you for sharing the picture, it seems that it was made by ggplot2 in R? Would you mind to share the R code with me? I still cannot find any features except some \"NA\"... Maybe something was wrong for some reasons, I tried to open the csv with data.table.\r\nThanks again.", "132671": "Thank you so much!\r\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?\r\n\r\n", "132670": "Thank you so much!\r\nA lot of? I can only find some NAs in R or T and T1 in SAS.....what do you mean the features? What do they look like?\r\n\r\n", "132669": "1) Even if you don't build \"smart\" features, there's a lot of features in there in raw format (as is). Keep looking.\r\n\r\n2) None of us have idea (at least, to be sure about it), it's anonymous data.", "137169": "" } }