Note: The notebook takes a loooooooong time to load
\n\nHere's my EDA on 25% of the training data: https://pub.beakernotebook.com/publications/5c191804-743a-11e6-9d4c-df93169e4b6e
\n\nI'm still poking away at it, but wanted to share to see if others are getting similar results, have similar interpretations of the data and to get extra eyes to find errors I've made.
\n\nI've stuck to higher level data exploration, as there's too many features to explore individually.
\n\nMy favorite finding was that there isnt any data between timestamps 845-875 in the training data, but a few exist in the test data (Have others also seen this gap in data? Might just be because I'm working on a random subsample of 25%).
\n\nFeel free to share any thoughts or insights.
\n\n-Cole
", "messages": [ { "id": "134414", "postDate": "09/06/2016 14:54:05", "content": "Note: The notebook takes a loooooooong time to load
\n\nHere's my EDA on 25% of the training data: https://pub.beakernotebook.com/publications/5c191804-743a-11e6-9d4c-df93169e4b6e
\n\nI'm still poking away at it, but wanted to share to see if others are getting similar results, have similar interpretations of the data and to get extra eyes to find errors I've made.
\n\nI've stuck to higher level data exploration, as there's too many features to explore individually.
\n\nMy favorite finding was that there isnt any data between timestamps 845-875 in the training data, but a few exist in the test data (Have others also seen this gap in data? Might just be because I'm working on a random subsample of 25%).
\n\nFeel free to share any thoughts or insights.
\n\n-Cole
", "rawMarkdown": "**Note**: The notebook takes a loooooooong time to load\r\n\r\nHere's my EDA on 25% of the training data: https://pub.beakernotebook.com/publications/5c191804-743a-11e6-9d4c-df93169e4b6e\r\n\r\nI'm still poking away at it, but wanted to share to see if others are getting similar results, have similar interpretations of the data and to get extra eyes to find errors I've made. \r\n\r\nI've stuck to higher level data exploration, as there's too many features to explore individually. \r\n\r\nMy favorite finding was that there isnt any data between timestamps 845-875 in the training data, but a few exist in the test data (Have others also seen this gap in data? Might just be because I'm working on a random subsample of 25%).\r\n\r\nFeel free to share any thoughts or insights.\r\n\r\n-Cole", "votes": null }, { "id": "134477", "postDate": "09/06/2016 20:06:19", "content": "Some nice information here! I can confirm the general gap in training data from 845-875. I didn't see it at first because I was just looking at consecutive time measurements, and there are actually 7 parts running through S25 during that time window. But as far as I can tell, that is the sum total of all activity during the window.
\n\nFor the paths, I took a different approach and looked at the stations each part went through. I get a similar count of unique paths with a slightly different shape to the distribution. By this way of counting, the median number of stations for train set is 14 with the most stations being 23.
\n\nAttached is a file of paths and counts aggregated at the station level.
", "rawMarkdown": "Some nice information here! I can confirm the general gap in training data from 845-875. I didn't see it at first because I was just looking at consecutive time measurements, and there are actually 7 parts running through S25 during that time window. But as far as I can tell, that is the sum total of all activity during the window. \r\n\r\nFor the paths, I took a different approach and looked at the stations each part went through. I get a similar count of unique paths with a slightly different shape to the distribution. By this way of counting, the median number of stations for train set is 14 with the most stations being 23. \r\n\r\nAttached is a file of paths and counts aggregated at the station level.", "votes": null }, { "id": "134508", "postDate": "09/07/2016 01:03:54", "content": "@JohnM very nice! Mind sharing how you accomplished your result? I was looking at the data via database the other night and wondering about patterns like that and you went and plucked it out.
", "rawMarkdown": "JohnM very nice! Mind sharing how you accomplished your result? I was looking at the data via database the other night and wondering about patterns like that and you went and plucked it out.", "votes": null }, { "id": "134528", "postDate": "09/07/2016 03:45:54", "content": "Hey Bob - I used the code below to get most of it. Once I had the hash strings with 1s and 0s, I popped it into Excel to get the station numbers based on the positions of the 1s. It's probably only a couple more lines of code to do it in R but after fighting with it for too long I went with what I knew.
", "rawMarkdown": "Hey Bob - I used the code below to get most of it. Once I had the hash strings with 1s and 0s, I popped it into Excel to get the station numbers based on the positions of the 1s. It's probably only a couple more lines of code to do it in R but after fighting with it for too long I went with what I knew.", "votes": null }, { "id": "134608", "postDate": "09/07/2016 14:08:23", "content": "Nice paths workbook. I did the same analysis and then plotted #fails against #passed for all the paths that contained at least one fail.
", "rawMarkdown": "Nice paths workbook. I did the same analysis and then plotted #fails against #passed for all the paths that contained at least one fail.", "votes": null } ], "comments": [ { "id": 134477, "author_name": "jpmiller", "author_url": "", "post_date": "09/06/2016 20:06:19", "content": "Some nice information here! I can confirm the general gap in training data from 845-875. I didn't see it at first because I was just looking at consecutive time measurements, and there are actually 7 parts running through S25 during that time window. But as far as I can tell, that is the sum total of all activity during the window.
\n\nFor the paths, I took a different approach and looked at the stations each part went through. I get a similar count of unique paths with a slightly different shape to the distribution. By this way of counting, the median number of stations for train set is 14 with the most stations being 23.
\n\nAttached is a file of paths and counts aggregated at the station level.
", "votes": null, "replies": [] }, { "id": 134508, "author_name": "rdslater", "author_url": "", "post_date": "09/07/2016 01:03:54", "content": "@JohnM very nice! Mind sharing how you accomplished your result? I was looking at the data via database the other night and wondering about patterns like that and you went and plucked it out.
", "votes": null, "replies": [] }, { "id": 134528, "author_name": "jpmiller", "author_url": "", "post_date": "09/07/2016 03:45:54", "content": "Hey Bob - I used the code below to get most of it. Once I had the hash strings with 1s and 0s, I popped it into Excel to get the station numbers based on the positions of the 1s. It's probably only a couple more lines of code to do it in R but after fighting with it for too long I went with what I knew.
", "votes": null, "replies": [] }, { "id": 134608, "author_name": "knorthover", "author_url": "", "post_date": "09/07/2016 14:08:23", "content": "Nice paths workbook. I did the same analysis and then plotted #fails against #passed for all the paths that contained at least one fail.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "134414": "**Note**: The notebook takes a loooooooong time to load\r\n\r\nHere's my EDA on 25% of the training data: https://pub.beakernotebook.com/publications/5c191804-743a-11e6-9d4c-df93169e4b6e\r\n\r\nI'm still poking away at it, but wanted to share to see if others are getting similar results, have similar interpretations of the data and to get extra eyes to find errors I've made. \r\n\r\nI've stuck to higher level data exploration, as there's too many features to explore individually. \r\n\r\nMy favorite finding was that there isnt any data between timestamps 845-875 in the training data, but a few exist in the test data (Have others also seen this gap in data? Might just be because I'm working on a random subsample of 25%).\r\n\r\nFeel free to share any thoughts or insights.\r\n\r\n-Cole", "134477": "Some nice information here! I can confirm the general gap in training data from 845-875. I didn't see it at first because I was just looking at consecutive time measurements, and there are actually 7 parts running through S25 during that time window. But as far as I can tell, that is the sum total of all activity during the window. \r\n\r\nFor the paths, I took a different approach and looked at the stations each part went through. I get a similar count of unique paths with a slightly different shape to the distribution. By this way of counting, the median number of stations for train set is 14 with the most stations being 23. \r\n\r\nAttached is a file of paths and counts aggregated at the station level.", "134508": "JohnM very nice! Mind sharing how you accomplished your result? I was looking at the data via database the other night and wondering about patterns like that and you went and plucked it out.", "134528": "Hey Bob - I used the code below to get most of it. Once I had the hash strings with 1s and 0s, I popped it into Excel to get the station numbers based on the positions of the 1s. It's probably only a couple more lines of code to do it in R but after fighting with it for too long I went with what I knew.", "134608": "Nice paths workbook. I did the same analysis and then plotted #fails against #passed for all the paths that contained at least one fail." }, "source": "meta" }