Cursory EDA suggests - which is confirmed in a few places - that several buildings, especially those from site_id 0 and 1, have a long string of zeroes at the end (which according to the \"partial leak\" info do not repeat later). Some people have reported that trimming those improves the model performance, so I wrote a small R kernel creating such a trimmed version of the dataset:
\n\nhttps://www.kaggle.com/konradb/trim-leading-zeroes
\n\nComments / suggestions / upvotes are most welcome.
\n\nEdit: for some weird reason I am unable to make the notebook public (toggling access has no effect), so here's the code
\n\n```
\n\nlibrary(data.table)\nlibrary(dplyr)\nrequire(lubridate)\nrequire(stringr)\nrequire(Metrics)\nrequire(zoo)
\n\nxtrain <- fread('../input/ashrae-energy-prediction/train.csv')
\n\nxtrain$tokeep <- 0
\n\nbuilding_list <- unique(xtrain$building_id)
\n\nfor (bb in building_list[1:3])\n{\n index0 <- which(xtrain$building_id == bb)\n x0 <- xtrain[index0,]\n x0[, :=(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\n # catch non-negative indices\n idx <- which(x0$rmx > 0)\n # start of a first contiguous block\n cutoff <- which(diff(idx) == 1)\n start_here <- idx[cutoff[1]]\n # mark rows to drop\n x0$tokeep[start_here:nrow(x0)] <- 1\n xtrain$tokeep[index0][start_here:nrow(x0)] <- 1
prop_retained = round(1 - start_here/nrow(x0),2)\n print(paste(bb, ': ', prop_retained, ' kept', sep = ''))\n}
\n\nxtr <- xtrain[xtrain$tokeep == 1,]\nxtr$tokeep <- NULL\nwrite.csv(xtr, 'train_trimmed.csv', row.names = F, quote = F)\n```
", "messages": [ { "id": 670294, "postDate": "2019-11-11T09:34:08.673Z", "content": "Cursory EDA suggests - which is confirmed in a few places - that several buildings, especially those from site_id 0 and 1, have a long string of zeroes at the end (which according to the \"partial leak\" info do not repeat later). Some people have reported that trimming those improves the model performance, so I wrote a small R kernel creating such a trimmed version of the dataset:
\n\nhttps://www.kaggle.com/konradb/trim-leading-zeroes
\n\nComments / suggestions / upvotes are most welcome.
\n\nEdit: for some weird reason I am unable to make the notebook public (toggling access has no effect), so here's the code
\n\n```
\n\nlibrary(data.table)\nlibrary(dplyr)\nrequire(lubridate)\nrequire(stringr)\nrequire(Metrics)\nrequire(zoo)
\n\nxtrain <- fread('../input/ashrae-energy-prediction/train.csv')
\n\nxtrain$tokeep <- 0
\n\nbuilding_list <- unique(xtrain$building_id)
\n\nfor (bb in building_list[1:3])\n{\n index0 <- which(xtrain$building_id == bb)\n x0 <- xtrain[index0,]\n x0[, :=(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\n # catch non-negative indices\n idx <- which(x0$rmx > 0)\n # start of a first contiguous block\n cutoff <- which(diff(idx) == 1)\n start_here <- idx[cutoff[1]]\n # mark rows to drop\n x0$tokeep[start_here:nrow(x0)] <- 1\n xtrain$tokeep[index0][start_here:nrow(x0)] <- 1
prop_retained = round(1 - start_here/nrow(x0),2)\n print(paste(bb, ': ', prop_retained, ' kept', sep = ''))\n}
\n\nxtr <- xtrain[xtrain$tokeep == 1,]\nxtr$tokeep <- NULL\nwrite.csv(xtr, 'train_trimmed.csv', row.names = F, quote = F)\n```
", "rawMarkdown": "Cursory EDA suggests - which is confirmed in a few places - that several buildings, especially those from site_id 0 and 1, have a long string of zeroes at the end (which according to the \"partial leak\" info do not repeat later). Some people have reported that trimming those improves the model performance, so I wrote a small R kernel creating such a trimmed version of the dataset:\n\nhttps://www.kaggle.com/konradb/trim-leading-zeroes\n\nComments / suggestions / upvotes are most welcome.\n\n\nEdit: for some weird reason I am unable to make the notebook public (toggling access has no effect), so here's the code\n\n```\n## load the necessary packages\nlibrary(data.table)\nlibrary(dplyr)\nrequire(lubridate)\nrequire(stringr)\nrequire(Metrics)\nrequire(zoo)\n\n## load data\nxtrain <- fread('../input/ashrae-energy-prediction/train.csv')\n\n# the column we need to know what to drop \nxtrain$tokeep <- 0\n\n## detect the strings of zeroes ----\n\n# list of unique buildings \nbuilding_list <- unique(xtrain$building_id)\n\n# loop over buildings - done in a loop for convenience (I wanted to check results as they came without waiting for a complete job \n# to finish. also, parallelization in data.table on Mac does not work out of the box)\n\nfor (bb in building_list[1:3])\n{\n index0 <- which(xtrain$building_id == bb)\n x0 <- xtrain[index0,]\n x0[, `:=`(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\n # catch non-negative indices\n idx <- which(x0$rmx > 0)\n # start of a first contiguous block\n cutoff <- which(diff(idx) == 1)\n start_here <- idx[cutoff[1]]\n # mark rows to drop\n x0$tokeep[start_here:nrow(x0)] <- 1\n xtrain$tokeep[index0][start_here:nrow(x0)] <- 1\n \n prop_retained = round(1 - start_here/nrow(x0),2)\n print(paste(bb, ': ', prop_retained, ' kept', sep = ''))\n}\n\n## save trimmed version ----\nxtr <- xtrain[xtrain$tokeep == 1,]\nxtr$tokeep <- NULL\nwrite.csv(xtr, 'train_trimmed.csv', row.names = F, quote = F)\n```", "votes": 20 }, { "id": 677584, "postDate": "2019-11-20T10:54:35.473Z", "content": "x0[,:=(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\nwhy aggregate by timestamp? in this way you have as output meter_reading (cause every timestamp is different) and not a rollmax.
For python,\ndata cleaning can be done by slightly modifying @ marshi's code.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#latest-655011
", "rawMarkdown": "For python,\ndata cleaning can be done by slightly modifying @ marshi's code.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#latest-655011" }, { "id": 671611, "postDate": "2019-11-13T00:22:58.677Z", "content": "Just wondering \n+ Why would trimming zeros after decimal may increase the performance of the model?
", "rawMarkdown": "Just wondering \n+ Why would trimming zeros after decimal may increase the performance of the model?", "replies": [ { "id": 672840, "postDate": "2019-11-14T08:15:30.317Z", "content": "Because they don't represent the test set meter_reading distribution. They exist only in training set.
Leading zeroes refers to the zeroes after the decimal? Shouldn't that be trailing zeroes instead? I don't read the R code, so not too sure what Konrad is doing.
\n\nHow do you know the zeroes after decimal don't represent the test set meter_reading distribution?
", "rawMarkdown": "Leading zeroes refers to the zeroes after the decimal? Shouldn't that be trailing zeroes instead? I don't read the R code, so not too sure what Konrad is doing.\n\nHow do you know the zeroes after decimal don't represent the test set meter_reading distribution?" }, { "id": 673250, "postDate": "2019-11-14T18:05:52.477Z", "content": "It was imprecise language on my part: i mean blocks of zero observations at the start of the series.
", "rawMarkdown": "It was imprecise language on my part: i mean blocks of zero observations at the start of the series." }, { "id": 674781, "postDate": "2019-11-17T04:15:28.067Z", "content": "Hi Konrad, thanks for explaining. What's the reason that removing zero observations at the start (for siteid=0 presumably) lead to improved model?
", "rawMarkdown": "Hi Konrad, thanks for explaining. What's the reason that removing zero observations at the start (for siteid=0 presumably) lead to improved model?" } ] }, { "id": 670300, "postDate": "2019-11-11T09:39:30.080Z", "content": "The notebook seems to be private.
", "rawMarkdown": "The notebook seems to be private.", "replies": [ { "id": 670308, "postDate": "2019-11-11T09:52:25.300Z", "content": "Yeah, I noticed - sorry about that. For some reason the switch to public does not seem to work atm, perhaps because kernel is running? I am trying to fix it.
", "rawMarkdown": "Yeah, I noticed - sorry about that. For some reason the switch to public does not seem to work atm, perhaps because kernel is running? I am trying to fix it." }, { "id": 670313, "postDate": "2019-11-11T10:02:20.913Z", "content": "Set sharing to public while editing kernel
", "rawMarkdown": "Set sharing to public while editing kernel" }, { "id": 670319, "postDate": "2019-11-11T10:06:53.473Z", "content": "Ok, worked when done that way - should be fine now. Thanks.
", "rawMarkdown": "Ok, worked when done that way - should be fine now. Thanks." } ] }, { "id": 675413, "postDate": "2019-11-18T03:48:55.337Z", "rawMarkdown": "", "isDeleted": true } ], "comments": [ { "id": 677584, "author_name": "Davide Stenner", "author_url": "", "post_date": "2019-11-20T10:54:35.473000", "content": "x0[,:=(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\nwhy aggregate by timestamp? in this way you have as output meter_reading (cause every timestamp is different) and not a rollmax.
For python,\ndata cleaning can be done by slightly modifying @ marshi's code.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#latest-655011
", "votes": 0, "replies": [] }, { "id": 671611, "author_name": "Vishal Sharma", "author_url": "", "post_date": "2019-11-13T00:22:58.677000", "content": "Just wondering \n+ Why would trimming zeros after decimal may increase the performance of the model?
", "votes": 0, "replies": [ { "id": 672840, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-11-14T08:15:30.317000", "content": "Because they don't represent the test set meter_reading distribution. They exist only in training set.
Leading zeroes refers to the zeroes after the decimal? Shouldn't that be trailing zeroes instead? I don't read the R code, so not too sure what Konrad is doing.
\n\nHow do you know the zeroes after decimal don't represent the test set meter_reading distribution?
", "votes": 0, "replies": [] }, { "id": 673250, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-14T18:05:52.477000", "content": "It was imprecise language on my part: i mean blocks of zero observations at the start of the series.
", "votes": 0, "replies": [] }, { "id": 674781, "author_name": "Lawrence Toh", "author_url": "", "post_date": "2019-11-17T04:15:28.067000", "content": "Hi Konrad, thanks for explaining. What's the reason that removing zero observations at the start (for siteid=0 presumably) lead to improved model?
", "votes": 0, "replies": [] } ] }, { "id": 670300, "author_name": "Fredrik Jonsson", "author_url": "", "post_date": "2019-11-11T09:39:30.080000", "content": "The notebook seems to be private.
", "votes": 0, "replies": [ { "id": 670308, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-11T09:52:25.300000", "content": "Yeah, I noticed - sorry about that. For some reason the switch to public does not seem to work atm, perhaps because kernel is running? I am trying to fix it.
", "votes": 0, "replies": [] }, { "id": 670313, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-11-11T10:02:20.913000", "content": "Set sharing to public while editing kernel
", "votes": 0, "replies": [] }, { "id": 670319, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-11T10:06:53.473000", "content": "Ok, worked when done that way - should be fine now. Thanks.
", "votes": 0, "replies": [] } ] }, { "id": 675413, "author_name": "", "author_url": "", "post_date": "2019-11-18T03:48:55.337000", "content": "", "votes": 0, "replies": [] } ], "raw_markdown_by_id": { "670294": "Cursory EDA suggests - which is confirmed in a few places - that several buildings, especially those from site_id 0 and 1, have a long string of zeroes at the end (which according to the \"partial leak\" info do not repeat later). Some people have reported that trimming those improves the model performance, so I wrote a small R kernel creating such a trimmed version of the dataset:\n\nhttps://www.kaggle.com/konradb/trim-leading-zeroes\n\nComments / suggestions / upvotes are most welcome.\n\n\nEdit: for some weird reason I am unable to make the notebook public (toggling access has no effect), so here's the code\n\n```\n## load the necessary packages\nlibrary(data.table)\nlibrary(dplyr)\nrequire(lubridate)\nrequire(stringr)\nrequire(Metrics)\nrequire(zoo)\n\n## load data\nxtrain <- fread('../input/ashrae-energy-prediction/train.csv')\n\n# the column we need to know what to drop \nxtrain$tokeep <- 0\n\n## detect the strings of zeroes ----\n\n# list of unique buildings \nbuilding_list <- unique(xtrain$building_id)\n\n# loop over buildings - done in a loop for convenience (I wanted to check results as they came without waiting for a complete job \n# to finish. also, parallelization in data.table on Mac does not work out of the box)\n\nfor (bb in building_list[1:3])\n{\n index0 <- which(xtrain$building_id == bb)\n x0 <- xtrain[index0,]\n x0[, `:=`(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]\n # catch non-negative indices\n idx <- which(x0$rmx > 0)\n # start of a first contiguous block\n cutoff <- which(diff(idx) == 1)\n start_here <- idx[cutoff[1]]\n # mark rows to drop\n x0$tokeep[start_here:nrow(x0)] <- 1\n xtrain$tokeep[index0][start_here:nrow(x0)] <- 1\n \n prop_retained = round(1 - start_here/nrow(x0),2)\n print(paste(bb, ': ', prop_retained, ' kept', sep = ''))\n}\n\n## save trimmed version ----\nxtr <- xtrain[xtrain$tokeep == 1,]\nxtr$tokeep <- NULL\nwrite.csv(xtr, 'train_trimmed.csv', row.names = F, quote = F)\n```", "677584": "`x0[, `:=`(rmx = rollmax(meter_reading, k = 50, partial = TRUE, fill = 0, align = 'right')),\n by= list(building_id, timestamp)]`\nwhy aggregate by timestamp? in this way you have as output meter_reading (cause every timestamp is different) and not a rollmax.", "677383": "For python,\ndata cleaning can be done by slightly modifying @ marshi's code.\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/113774#latest-655011", "671611": "Just wondering \n+ Why would trimming zeros after decimal may increase the performance of the model?", "670300": "The notebook seems to be private.", "675413": "" } }