{ "id": 23912, "title": "Is there a good way to tidy the data, or should we use sparse matrices?", "url": "/competitions/bosch-production-line-performance/discussion/23912", "author_name": "", "post_date": "2016-09-26T16:19:28.663Z", "votes": 2, "comment_count": 3, "views": 739, "content": "

I'm wondering if this data is close at all to being 'tidy'.

\n\n
\n

Each variable forms a column.

\n \n

Each observation forms a row.

\n \n

Each type of observational unit forms a table.

\n
\n\n

For example, I was thinking maybe the data could be rerranged as:

\n\n

Id, Line, Station, Feature, Feature_value, Result

\n\n

But then I was thinking this would be hard to feed into a ML algorithm. Is there a algo-friendly way to put this into a 'tidy' framework? Or should I go with sparse matrices?

", "messages": [ { "id": "136931", "postDate": "09/26/2016 16:19:28", "content": "

I'm wondering if this data is close at all to being 'tidy'.

\n\n
\n

Each variable forms a column.

\n \n

Each observation forms a row.

\n \n

Each type of observational unit forms a table.

\n
\n\n

For example, I was thinking maybe the data could be rerranged as:

\n\n

Id, Line, Station, Feature, Feature_value, Result

\n\n

But then I was thinking this would be hard to feed into a ML algorithm. Is there a algo-friendly way to put this into a 'tidy' framework? Or should I go with sparse matrices?

", "rawMarkdown": "I'm wondering if this data is close at all to being ['tidy'][1].\r\n\r\n> Each variable forms a column.\r\n> \r\n> Each observation forms a row.\r\n> \r\n> Each type of observational unit forms a table.\r\n\r\nFor example, I was thinking maybe the data could be rerranged as:\r\n\r\n`Id, Line, Station, Feature, Feature_value, Result`\r\n\r\nBut then I was thinking this would be hard to feed into a ML algorithm. Is there a algo-friendly way to put this into a 'tidy' framework? Or should I go with sparse matrices?\r\n\r\n\r\n [1]: https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html", "votes": null }, { "id": "136956", "postDate": "09/26/2016 21:29:13", "content": "

You are describing a type of "tall" format, which is in fact used a lot for storing and sending data with lots of structurally missing values. I can't think of any ML algorithms that can run directly on "tall" data. Maybe you could get a multinomial naive Bayes model to do it, but it wouldn't be off-the-shelf. MNB at least accepts variable length input per instance.

\n\n

Note that the feature number implies a station number, which in turn implies a line number, so the natural (row, column, value) mapping is (id, feature, value). The line and station numbers are possibly useful as grouping variables, though, or as a source of extra information. For instance, you can make features about the count of stations visited per line, or the time spent on a line with the information that we have.

", "rawMarkdown": "You are describing a type of \"tall\" format, which is in fact used a lot for storing and sending data with lots of structurally missing values. I can't think of any ML algorithms that can run directly on \"tall\" data. Maybe you could get a multinomial naive Bayes model to do it, but it wouldn't be off-the-shelf. MNB at least accepts variable length input per instance. \r\n\r\nNote that the feature number implies a station number, which in turn implies a line number, so the natural (row, column, value) mapping is (id, feature, value). The line and station numbers are possibly useful as grouping variables, though, or as a source of extra information. For instance, you can make features about the count of stations visited per line, or the time spent on a line with the information that we have.", "votes": null }, { "id": "136959", "postDate": "09/26/2016 22:17:13", "content": "

I see, thanks! I still need to check and see if any parts are going through multiple stations or lines; if they don't then the id, feature, value, result would work for a tall dataset.

", "rawMarkdown": "I see, thanks! I still need to check and see if any parts are going through multiple stations or lines; if they don't then the `id, feature, value, result` would work for a tall dataset.", "votes": null }, { "id": "136997", "postDate": "09/27/2016 08:33:47", "content": "

Most parts are going through many stations.

", "rawMarkdown": "Most parts are going through many stations.", "votes": null } ], "comments": [ { "id": 136956, "author_name": "davidthaler", "author_url": "", "post_date": "09/26/2016 21:29:13", "content": "

You are describing a type of "tall" format, which is in fact used a lot for storing and sending data with lots of structurally missing values. I can't think of any ML algorithms that can run directly on "tall" data. Maybe you could get a multinomial naive Bayes model to do it, but it wouldn't be off-the-shelf. MNB at least accepts variable length input per instance.

\n\n

Note that the feature number implies a station number, which in turn implies a line number, so the natural (row, column, value) mapping is (id, feature, value). The line and station numbers are possibly useful as grouping variables, though, or as a source of extra information. For instance, you can make features about the count of stations visited per line, or the time spent on a line with the information that we have.

", "votes": null, "replies": [] }, { "id": 136959, "author_name": "wordsforthewise", "author_url": "", "post_date": "09/26/2016 22:17:13", "content": "

I see, thanks! I still need to check and see if any parts are going through multiple stations or lines; if they don't then the id, feature, value, result would work for a tall dataset.

", "votes": null, "replies": [] }, { "id": 136997, "author_name": "cpmpml", "author_url": "", "post_date": "09/27/2016 08:33:47", "content": "

Most parts are going through many stations.

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "136931": "I'm wondering if this data is close at all to being ['tidy'][1].\r\n\r\n> Each variable forms a column.\r\n> \r\n> Each observation forms a row.\r\n> \r\n> Each type of observational unit forms a table.\r\n\r\nFor example, I was thinking maybe the data could be rerranged as:\r\n\r\n`Id, Line, Station, Feature, Feature_value, Result`\r\n\r\nBut then I was thinking this would be hard to feed into a ML algorithm. Is there a algo-friendly way to put this into a 'tidy' framework? Or should I go with sparse matrices?\r\n\r\n\r\n [1]: https://cran.r-project.org/web/packages/tidyr/vignettes/tidy-data.html", "136956": "You are describing a type of \"tall\" format, which is in fact used a lot for storing and sending data with lots of structurally missing values. I can't think of any ML algorithms that can run directly on \"tall\" data. Maybe you could get a multinomial naive Bayes model to do it, but it wouldn't be off-the-shelf. MNB at least accepts variable length input per instance. \r\n\r\nNote that the feature number implies a station number, which in turn implies a line number, so the natural (row, column, value) mapping is (id, feature, value). The line and station numbers are possibly useful as grouping variables, though, or as a source of extra information. For instance, you can make features about the count of stations visited per line, or the time spent on a line with the information that we have.", "136959": "I see, thanks! I still need to check and see if any parts are going through multiple stations or lines; if they don't then the `id, feature, value, result` would work for a tall dataset.", "136997": "Most parts are going through many stations." }, "source": "meta" }