Congrats to Thomas and the Top-100 folks. I would be interested in reading your solutions.
\nMy method was pretty straight forward and based on 1-D Convolutional NNs.
\nFeatures:
\nI used the Accel, Gyro features as it is. Using raw orientation features did not make sense to me, so I used the first order difference of the orientation features (after converting them to Euler angles), so that any vibrations caused by various surfaces are captured.
\nAlso used FFT of Accel/Gyro measurements as features.
CV:
\nAs discussed by multiple people on the forums, GroupKFold makes sense here. k=3 gave me ~0.65-0.7 CV scores which translated to ~0.8 on the Public leaderboard.
Model:
\n1D Convolution based NN. I saw slightly better results with the SeparableConv1D implementation in Keras, but I'm sure the simpler Conv1D layer if tuned properly should perform equally well.
I have shared the kernel with the Starter code for my solution. This kernel does not do any chaining of test samples and by itself should be able to score ~0.87 on the Private Leaderboard. Its not well commented at this time, but I will add comments to it.
\nOn top of it, I used a simple chaining mechanism to stretch the leaderboard score to 0.99 (In my opinion, this does not add much value in practice, and I only did it to stay relevant on the leaderboard. No matter how good your chaining algorithm is, all you have to do in the real world is take some additional steps)
\nHope the models resulting from this contest can be put to practical use by the parties who shared the data.
\nKernel: https://www.kaggle.com/prith189/starter-code-for-3rd-place-solution
\nThanks,
\nPrithvi
Congratulations on your model, the CV scores are already very impressive. I'd be really curious to know how you arrived at such a deep model (which I would not have expected to work in this competition). I tried CNNs and RNNs as well and stopped well short of your model because I couldn't see any improvement. Did you build your model from the ground up, checking whether each additional layer gave an improvement or is this some knowledge you had from previous work? Thanks!
", "votes": 2, "replies": [ { "id": 517350, "author_name": "Prithvi", "author_url": "", "post_date": "2019-04-15T23:24:14.720000", "content": "Hi, I started with a 3 layer network, and built on it gradually as dictated by the training and CV scores. It took me a few iterations to find an architecture that worked reasonably well. I tried other architectures too (InceptionNet, RNNs etc.), but I wasn't able to tune them to perform comparably.
", "votes": 1, "replies": [] } ] }, { "id": 515019, "author_name": "mvaidhyg", "author_url": "", "post_date": "2019-04-12T05:46:48.273000", "content": "I thought it was not a good idea to use first order differences of orientation because the input file had only 5 digits of precision. I did try it though (with random forest) and did not get good results. Also, the derivative of the orientation is directly maps to the angular velocity. So couldn't you have just used the angular velocities? I actually did the differentiation and saw that the computed derivates match very well with those derived from the angular velocity, the derived data is just a bit noisier. See notes about quaternion differentiation here: http://www.euclideanspace.com/physics/kinematics/angularvelocity/QuaternionDifferentiation2.pdf
", "votes": 2, "replies": [ { "id": 515670, "author_name": "Prithvi", "author_url": "", "post_date": "2019-04-12T23:30:41.353000", "content": "Yes, I agree that Integral of Angular velocity matches Euler angle. However, I wasn't convinced that the delta between the two is noise. My thought process started off with maybe there exists some signal in the residual (Angular velocity integrated - Euler angle) which could be beneficial to predicting the surface. I'm still not sure how much value it adds, I have not experimented without the difference features.
", "votes": 2, "replies": [] } ] }, { "id": 547756, "author_name": "Rishu Bagla", "author_url": "", "post_date": "2019-06-08T08:10:13.563000", "content": "Congrats Prithvi
", "votes": 0, "replies": [] }, { "id": 523612, "author_name": "Alex Arlanov", "author_url": "", "post_date": "2019-04-26T16:06:00.363000", "content": "Congratulations. How did you come to the idea to make a feature engineering in this way? Did you work in this field?
", "votes": 0, "replies": [] }, { "id": 520674, "author_name": "Wei Zheng", "author_url": "", "post_date": "2019-04-21T14:57:47.653000", "content": "Hi,\nI am interested in the chaining technique you used. I have never heard of it before. Can you show me a few reference, perhaps some study material?
\nThank you.
i had starting from multi input RNN kernel, than add wavelet transform on the 10 variables and put Con2D 16-32-64 filters (5x5 and 3x3 size filter in parallel) with poolmax and it gave 0.64 on Public LB and 0.65 on private LB. But i 've choosed to select a wrong alternative (ie multi CNN) and just had 0.60 on private LB ;-)
", "votes": 0, "replies": [] }, { "id": 515026, "author_name": "Ilia Zaitsev", "author_url": "", "post_date": "2019-04-12T05:58:36.917000", "content": "Good to know it is still possible to get a good score on this data without exploiting the data leak. I've tried to use Conv1d and LSTM on raw data, without any additional feature engineering, and my results were pretty humble, something around 0.55-0.57 at most. Now I understand that a proper feature engineering is useful even if you go with Deep Learning models.
", "votes": 0, "replies": [ { "id": 517776, "author_name": "Erik Caretta", "author_url": "", "post_date": "2019-04-16T14:19:28.330000", "content": "same here, .58 with a 3-layers LSTM with data augmentation (simply flipping data on the time dimension). Great job Prithvi and thanks for sharing
", "votes": 0, "replies": [] } ] }, { "id": 515005, "author_name": "Eric R", "author_url": "", "post_date": "2019-04-12T05:32:43.617000", "content": "Congrats! Amazing solution. I'm sure you are a good fit for ML position in Bay Area.
", "votes": 0, "replies": [] }, { "id": 514972, "author_name": "Carson Zhang", "author_url": "", "post_date": "2019-04-12T04:38:55.857000", "content": "Congratulations and well done! Thanks for sharing. It would be great if you could share some details on linking the data . Particularly, I think how to use 128 sample points and the so called long runs are the keys. I saw many people, including myself, were super confused on how to stitch samples together and use it as input for the model to learn. I'm really appreciated if you could show some details on that. Thanks!
Hi, I did not use stitched samples as input to my model. I just used the regular 128-pt samples for training. Once predictions from my model were available, I post processed the predictions using a simple linking algorithm (ie. if Sample A and Sample B are linked, then make sure that Surface A = Surface B).
\n\nOn the other hand, if you want to use stitched samples as input to the model, I think you can use it to generate additional training data. For eg, if Sample A and Sample B are linked, then instead of two 128-point input samples, you can generate much more by appending the two to form a 256-point sequence and cutting out multiple 128-point sequences from it. This may lead to a better and generalized model.
", "votes": 0, "replies": [] }, { "id": 515686, "author_name": "Carson Zhang", "author_url": "", "post_date": "2019-04-13T00:38:00.403000", "content": "Thanks a lot for your reply, that's what I thought initially. I guess what I stucked was the part \"used the regular 128-pt samples for training\". To be more specific, the original data we had was having series ids as index. by properly changing the dataset dimension, we then would have 128 indexes. And then you feed the 128-index data into your model, and work on the predicted results to get back to the series_id as index. Am I understanding correctly? Thanks!
", "votes": 0, "replies": [] }, { "id": 516302, "author_name": "Prithvi", "author_url": "", "post_date": "2019-04-14T00:20:31.467000", "content": "\"and work on the predicted results to get back to the series_id as index. \"\nI dont think series_id has any significance. It is just a linearly increasing number. What adds value is identifying which two series_ids are neighbors. There is a public kernel which discusses how we can obtain the neighboring series using orientation data.
", "votes": 0, "replies": [] } ] }, { "id": 514969, "author_name": "mvaidhyg", "author_url": "", "post_date": "2019-04-12T04:30:07.423000", "content": "Thanks a lot for posting this. I learned something new. I am a data science newbie and I hadn't heard about 1-D Convolutional NN. After reading about it, it seems to be the best option for problems like this. Congratulations on your score.
", "votes": 0, "replies": [] }, { "id": 514960, "author_name": "Rajanikant Tenguria", "author_url": "", "post_date": "2019-04-12T04:13:24.860000", "content": "Congratulations Prithvi for top3 finish. Great share :)
", "votes": 0, "replies": [] }, { "id": 514950, "author_name": "Chenghao Luo", "author_url": "", "post_date": "2019-04-12T04:02:38.560000", "content": "well done
", "votes": 0, "replies": [] }, { "id": 514875, "author_name": "swagatron", "author_url": "", "post_date": "2019-04-12T01:59:31.830000", "content": "Cool!
", "votes": 0, "replies": [] } ], "comment_count": 19, "messages": [ { "id": 514847, "postDate": "2019-04-12T01:11:47.767Z", "content": "Congrats to Thomas and the Top-100 folks. I would be interested in reading your solutions.
\nMy method was pretty straight forward and based on 1-D Convolutional NNs.
\nFeatures:
\nI used the Accel, Gyro features as it is. Using raw orientation features did not make sense to me, so I used the first order difference of the orientation features (after converting them to Euler angles), so that any vibrations caused by various surfaces are captured.
\nAlso used FFT of Accel/Gyro measurements as features.
CV:
\nAs discussed by multiple people on the forums, GroupKFold makes sense here. k=3 gave me ~0.65-0.7 CV scores which translated to ~0.8 on the Public leaderboard.
Model:
\n1D Convolution based NN. I saw slightly better results with the SeparableConv1D implementation in Keras, but I'm sure the simpler Conv1D layer if tuned properly should perform equally well.
I have shared the kernel with the Starter code for my solution. This kernel does not do any chaining of test samples and by itself should be able to score ~0.87 on the Private Leaderboard. Its not well commented at this time, but I will add comments to it.
\nOn top of it, I used a simple chaining mechanism to stretch the leaderboard score to 0.99 (In my opinion, this does not add much value in practice, and I only did it to stay relevant on the leaderboard. No matter how good your chaining algorithm is, all you have to do in the real world is take some additional steps)
\nHope the models resulting from this contest can be put to practical use by the parties who shared the data.
\nKernel: https://www.kaggle.com/prith189/starter-code-for-3rd-place-solution
\nThanks,
\nPrithvi
Congratulations on your model, the CV scores are already very impressive. I'd be really curious to know how you arrived at such a deep model (which I would not have expected to work in this competition). I tried CNNs and RNNs as well and stopped well short of your model because I couldn't see any improvement. Did you build your model from the ground up, checking whether each additional layer gave an improvement or is this some knowledge you had from previous work? Thanks!
", "rawMarkdown": "Congratulations on your model, the CV scores are already very impressive. I'd be really curious to know how you arrived at such a deep model (which I would not have expected to work in this competition). I tried CNNs and RNNs as well and stopped well short of your model because I couldn't see any improvement. Did you build your model from the ground up, checking whether each additional layer gave an improvement or is this some knowledge you had from previous work? Thanks!", "votes": 2, "replies": [ { "id": 517350, "postDate": "2019-04-15T23:24:14.720Z", "content": "Hi, I started with a 3 layer network, and built on it gradually as dictated by the training and CV scores. It took me a few iterations to find an architecture that worked reasonably well. I tried other architectures too (InceptionNet, RNNs etc.), but I wasn't able to tune them to perform comparably.
", "rawMarkdown": "Hi, I started with a 3 layer network, and built on it gradually as dictated by the training and CV scores. It took me a few iterations to find an architecture that worked reasonably well. I tried other architectures too (InceptionNet, RNNs etc.), but I wasn't able to tune them to perform comparably.", "votes": 1 } ] }, { "id": 515019, "postDate": "2019-04-12T05:46:48.273Z", "content": "I thought it was not a good idea to use first order differences of orientation because the input file had only 5 digits of precision. I did try it though (with random forest) and did not get good results. Also, the derivative of the orientation is directly maps to the angular velocity. So couldn't you have just used the angular velocities? I actually did the differentiation and saw that the computed derivates match very well with those derived from the angular velocity, the derived data is just a bit noisier. See notes about quaternion differentiation here: http://www.euclideanspace.com/physics/kinematics/angularvelocity/QuaternionDifferentiation2.pdf
", "rawMarkdown": "I thought it was not a good idea to use first order differences of orientation because the input file had only 5 digits of precision. I did try it though (with random forest) and did not get good results. Also, the derivative of the orientation is directly maps to the angular velocity. So couldn't you have just used the angular velocities? I actually did the differentiation and saw that the computed derivates match very well with those derived from the angular velocity, the derived data is just a bit noisier. See notes about quaternion differentiation here: http://www.euclideanspace.com/physics/kinematics/angularvelocity/QuaternionDifferentiation2.pdf", "votes": 2, "replies": [ { "id": 515670, "postDate": "2019-04-12T23:30:41.353Z", "content": "Yes, I agree that Integral of Angular velocity matches Euler angle. However, I wasn't convinced that the delta between the two is noise. My thought process started off with maybe there exists some signal in the residual (Angular velocity integrated - Euler angle) which could be beneficial to predicting the surface. I'm still not sure how much value it adds, I have not experimented without the difference features.
", "rawMarkdown": "Yes, I agree that Integral of Angular velocity matches Euler angle. However, I wasn't convinced that the delta between the two is noise. My thought process started off with maybe there exists some signal in the residual (Angular velocity integrated - Euler angle) which could be beneficial to predicting the surface. I'm still not sure how much value it adds, I have not experimented without the difference features.", "votes": 2 } ] }, { "id": 547756, "postDate": "2019-06-08T08:10:13.563Z", "content": "Congrats Prithvi
", "rawMarkdown": "Congrats Prithvi\n" }, { "id": 523612, "postDate": "2019-04-26T16:06:00.363Z", "content": "Congratulations. How did you come to the idea to make a feature engineering in this way? Did you work in this field?
", "rawMarkdown": "Congratulations. How did you come to the idea to make a feature engineering in this way? Did you work in this field?" }, { "id": 520674, "postDate": "2019-04-21T14:57:47.653Z", "content": "Hi,\nI am interested in the chaining technique you used. I have never heard of it before. Can you show me a few reference, perhaps some study material?
\nThank you.
i had starting from multi input RNN kernel, than add wavelet transform on the 10 variables and put Con2D 16-32-64 filters (5x5 and 3x3 size filter in parallel) with poolmax and it gave 0.64 on Public LB and 0.65 on private LB. But i 've choosed to select a wrong alternative (ie multi CNN) and just had 0.60 on private LB ;-)
", "rawMarkdown": "i had starting from multi input RNN kernel, than add wavelet transform on the 10 variables and put Con2D 16-32-64 filters (5x5 and 3x3 size filter in parallel) with poolmax and it gave 0.64 on Public LB and 0.65 on private LB. But i 've choosed to select a wrong alternative (ie multi CNN) and just had 0.60 on private LB ;-)" }, { "id": 515026, "postDate": "2019-04-12T05:58:36.917Z", "content": "Good to know it is still possible to get a good score on this data without exploiting the data leak. I've tried to use Conv1d and LSTM on raw data, without any additional feature engineering, and my results were pretty humble, something around 0.55-0.57 at most. Now I understand that a proper feature engineering is useful even if you go with Deep Learning models.
", "rawMarkdown": "Good to know it is still possible to get a good score on this data without exploiting the data leak. I've tried to use Conv1d and LSTM on raw data, without any additional feature engineering, and my results were pretty humble, something around 0.55-0.57 at most. Now I understand that a proper feature engineering is useful even if you go with Deep Learning models.", "replies": [ { "id": 517776, "postDate": "2019-04-16T14:19:28.330Z", "content": "same here, .58 with a 3-layers LSTM with data augmentation (simply flipping data on the time dimension). Great job Prithvi and thanks for sharing
", "rawMarkdown": "same here, .58 with a 3-layers LSTM with data augmentation (simply flipping data on the time dimension). Great job Prithvi and thanks for sharing" } ] }, { "id": 515005, "postDate": "2019-04-12T05:32:43.617Z", "content": "Congrats! Amazing solution. I'm sure you are a good fit for ML position in Bay Area.
", "rawMarkdown": "Congrats! Amazing solution. I'm sure you are a good fit for ML position in Bay Area." }, { "id": 514972, "postDate": "2019-04-12T04:38:55.857Z", "content": "Congratulations and well done! Thanks for sharing. It would be great if you could share some details on linking the data . Particularly, I think how to use 128 sample points and the so called long runs are the keys. I saw many people, including myself, were super confused on how to stitch samples together and use it as input for the model to learn. I'm really appreciated if you could show some details on that. Thanks!
Hi, I did not use stitched samples as input to my model. I just used the regular 128-pt samples for training. Once predictions from my model were available, I post processed the predictions using a simple linking algorithm (ie. if Sample A and Sample B are linked, then make sure that Surface A = Surface B).
\n\nOn the other hand, if you want to use stitched samples as input to the model, I think you can use it to generate additional training data. For eg, if Sample A and Sample B are linked, then instead of two 128-point input samples, you can generate much more by appending the two to form a 256-point sequence and cutting out multiple 128-point sequences from it. This may lead to a better and generalized model.
", "rawMarkdown": "Hi, I did not use stitched samples as input to my model. I just used the regular 128-pt samples for training. Once predictions from my model were available, I post processed the predictions using a simple linking algorithm (ie. if Sample A and Sample B are linked, then make sure that Surface A = Surface B). \n\nOn the other hand, if you want to use stitched samples as input to the model, I think you can use it to generate additional training data. For eg, if Sample A and Sample B are linked, then instead of two 128-point input samples, you can generate much more by appending the two to form a 256-point sequence and cutting out multiple 128-point sequences from it. This may lead to a better and generalized model." }, { "id": 515686, "postDate": "2019-04-13T00:38:00.403Z", "content": "Thanks a lot for your reply, that's what I thought initially. I guess what I stucked was the part \"used the regular 128-pt samples for training\". To be more specific, the original data we had was having series ids as index. by properly changing the dataset dimension, we then would have 128 indexes. And then you feed the 128-index data into your model, and work on the predicted results to get back to the series_id as index. Am I understanding correctly? Thanks!
", "rawMarkdown": "Thanks a lot for your reply, that's what I thought initially. I guess what I stucked was the part \"used the regular 128-pt samples for training\". To be more specific, the original data we had was having series ids as index. by properly changing the dataset dimension, we then would have 128 indexes. And then you feed the 128-index data into your model, and work on the predicted results to get back to the series_id as index. Am I understanding correctly? Thanks!" }, { "id": 516302, "postDate": "2019-04-14T00:20:31.467Z", "content": "\"and work on the predicted results to get back to the series_id as index. \"\nI dont think series_id has any significance. It is just a linearly increasing number. What adds value is identifying which two series_ids are neighbors. There is a public kernel which discusses how we can obtain the neighboring series using orientation data.
", "rawMarkdown": "\"and work on the predicted results to get back to the series_id as index. \"\nI dont think series_id has any significance. It is just a linearly increasing number. What adds value is identifying which two series_ids are neighbors. There is a public kernel which discusses how we can obtain the neighboring series using orientation data." } ] }, { "id": 514969, "postDate": "2019-04-12T04:30:07.423Z", "content": "Thanks a lot for posting this. I learned something new. I am a data science newbie and I hadn't heard about 1-D Convolutional NN. After reading about it, it seems to be the best option for problems like this. Congratulations on your score.
", "rawMarkdown": "Thanks a lot for posting this. I learned something new. I am a data science newbie and I hadn't heard about 1-D Convolutional NN. After reading about it, it seems to be the best option for problems like this. Congratulations on your score." }, { "id": 514960, "postDate": "2019-04-12T04:13:24.860Z", "content": "Congratulations Prithvi for top3 finish. Great share :)
", "rawMarkdown": "Congratulations Prithvi for top3 finish. Great share :)" }, { "id": 514950, "postDate": "2019-04-12T04:02:38.560Z", "content": "well done
", "rawMarkdown": "well done" }, { "id": 514875, "postDate": "2019-04-12T01:59:31.830Z", "content": "Cool!
", "rawMarkdown": "Cool!" } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T15:44:46Z", "competition_id": 13242, "competition_slug": "career-con-2019", "team_id": 2899491, "solution_url": "/competitions/career-con-2019/writeups/prith189-3rd-place-solution-0-99-and-code" }