First of all we would like to thank Kaggle and AMP PD for hosting this competition and providing a great dataset to dig into and rich enough to get lost in it for several months. It is always an extra motivation to work on problems that can bring value to the medical field or contribute to scientific research. Additional words of gratitude of course go to @kyakovlev for the amazing work he has done, I think we formed a great team for this competition and the results reflect that.
\nWe publish the full code of the winning notebook here
\nOur final solution is a simple average of two models: LGB and NN. Both models were trained on the same features (+ scaling/binarization for NN):
\nThe winning solution fully ignores the results of the blood tests. We’ve tried hard to find any signal in this crucial piece of the data, but unfortunately we came to the conclusion that none of our approaches or models can benefit from blood test features significant enough to distinguish it from random variations. The final models were trained only on the union of clinical and supplementary datasets.
\nFor the entire duration of the competition LGB was our model to beat and only a NN trained with the competition metric as the loss function was able to achieve competitive performance on CV. At first, we tried running a regression LGB model with different hyperparameters and custom objective functions, but nothing was better than l1 regression, which does not optimize the desired metric SMAPE+1. We also noticed that on CV the performance of every model is always better when the regression outputs are rounded to integers. Then we switched to an alternative approach.
\nOur LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.
\nWe ran an optimization routine to tune LGB hyperparameters to minimize SMAPE+1 on CV using the described post-processing.
\nThe neural network has a simple multi-layer feed-forward architecture with a regression target, using the competition metric SMAPE+1 as the loss function. We fixed the number of epochs and scheduler, and then tuned the learning rate and hidden layer size. The only trick there was to add a leaky relu activation as the last layer to prevent NN from getting stuck at negative predictions. Of course there are alternative ways to solve this issue.
\nWe’ve tried multiple cross-validation schemes due to the small training sample size, all of them were stratified by patient id. Once a sufficient number of folds is used, they all are quite well correlated to each other. Better than to the public leaderboard :) The final scheme we relied on was leave-one-(patient)-out or, in other words, a group k-fold cross validation with a fold for each patient. We used it because it doesn’t depend on random numbers. The cross-validation correlated well enough with the private leaderboard, and the submit we chose turned out to be our best private LB submission.
\nThe most impactful feature was the indication of whether a patient visit happened on the 6th month or not. It correlates strongly with the UPDRS targets (especially 2 and 3) and with frequency of medications being taken. As we can observe only the data correlation, it is impossible to judge what is the core reason for that. During the competition our hypothesis was that the patients that had more severe symptoms during the first examination (UPDRS scores parts 2 and 3) were more likely to get invited for a visit after 6 months and more likely to get medications prescribed. But for model training it was important that the patients that made a visit on the 6th month, have higher UPDRS scores on average. The same is true for an 18th month visit as well, but these 2 features are correlated. I personally wonder if presence / absence of these variables in the models are the reason for the private LB cliff effect around 20th place.
\nAnother curious effect related to it is observed for the forecasts made at visit_month = 0. If you look at the model forecasts for 0, 12 and 24 months ahead, they are consistently lower than the forecasts 6 months ahead. It is very logical from the math point of view - if a patient will show up on the 6th month, they will have higher UPDRS scores on average, and if not - the forecast will be ignored. But such model behaviour is unreasonable from a clinical point of view of course.
\nIt was also important to pay attention to the differences between training and test datasets as e.g. nicely summarized here. That, for instance, explains well why adding a feature indicating the visit on the 30th month could improve the CV, but ruin LB.
\nBlood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities. It worked on public LB after tuning a couple of “magic” coefficients directly on public LB itself. That gave us a boost all the way up to the second place on public LB, but it was clearly an overfit. We chose a “mild” version of that approach as our second final submission. It scored worse on private LB than the other submission, but, interestingly enough, not by as much as one could have expected (60.0 vs 60.3).
\nThanks to everyone who participated in the competition, those who kept many interesting discussions going on the forum and those who suggested improvements! And congrats to all the winners!
", "comments": [ { "id": 2265828, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-19T14:41:03.777000", "content": "It was a huge huge pleasure to team up with @dott1718 (Dmitry) in this competition. I felt as a total junior DS - Dmitry's ideas, approach and dedication were astonishing - I've learned so much and so grateful for all the time we spent together.
\nThis competition was super unique for me with many new puzzles to solve.
\nThank you again and congrats to all the winners and participants - It was remarkable ride.
", "votes": 13, "replies": [ { "id": 2265845, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-19T14:52:08.873000", "content": "A SMAPE1P thing:
\nMAPE / SMAPE / SMAPE1P are very tricky metrics for GBT models:
\nHere is the \"hack\" that is boosting the SMAPE1P score (and works also on real data - confirmed few weeks ago):
\nStep 1: Multiclass Classification
\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'multiclass',\n...\n}\nStep 2: Train model (could be xgboost catboost lgbm)
\nestimator.train(...)\nStep 3: Predict probabilities:
\nprobas = estimator.predict(...)\nStep 4: Make direct SMAPE1P optimization from probabilities
\ndef single_smape1p(preds, tgt):\n x = np.tile(np.arange(preds.shape[1]), (preds.shape[0], 1))\n x = np.abs(x - tgt) / (2 + x + tgt)\n return (x * preds).sum(axis=1)\n\ndef opt_smape1p(preds):\n x = np.hstack([single_smape1p(preds, i).reshape(-1,1) for i in range(preds.shape[1])])\n return x.argmin(axis=1)\n\nfinal_predictions = opt_smape1p(probas)\nIf you are sure that \"SMAPE\" variation is the metric that reflects your business value - do the \"hack\" (at least try).
", "votes": 16, "replies": [ { "id": 2266076, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T17:23:22.423000", "content": "Amazing hack! Great discovery and thanks for sharing!
", "votes": 5, "replies": [] }, { "id": 2266099, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T17:52:21.440000", "content": "Wow amazing it is working. My MLP CV = 54.95. So far using your hack and XGB, I achieved CV = 54.79 with my features. I will finish tuning XGB with your hack and then make a submission to LB. I will also try to ensemble with my MLP. How do you ensemble? Do we convert the XGB preds into integers and then ensemble with MLP, or is there a better way?
We’ve tried all possible ways - weighted rounded ensemble / probabilities blend / voting / … nothing really worked on cv - just a very slight move. We ended up with this
\n sample_test[\"preds_lgb\"] = model_lgb.predict(sample_test)\n sample_test[\"preds_nn\"] = np.round(np.clip(model_nn.predict(sample_test), 0, None))\n\n sample_submission[\"rating\"] = np.round( (sample_test[\"preds_lgb\"] + sample_test[\"preds_nn\"]) / 2)\nMean with rounding.
", "votes": 2, "replies": [] }, { "id": 2266113, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-19T18:05:05.340000", "content": "I’m convincing Dmitry to write a paper about this tiny part of the code - because it’s “gamechanger” for all MAPE lovers)))(no “dancing” around target/straight forward approach / better results) he is still thinking (not a big deal he said - please say that he is wrong)))))
", "votes": 1, "replies": [] }, { "id": 2266149, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T18:48:17.607000", "content": "Wow, i have my XGB CV = 54.64 using your hack. This is +0.3 better than my MLP CV and I'm still tuning. I'm curious if this will boost my LB score this much!
Waiting for your lb probing - all our tests showed that loo cv was very stable and aligned with lb and we had the same boost on loo cv / private / public lb.
", "votes": 2, "replies": [] }, { "id": 2266222, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T20:44:47.883000", "content": "@kyakovlev I made a notebook here with single model XGB and just submitted to the LB. We should have a score in a few hours. Note that this is the first submission, so we have not adjusted the model to boost private LB. The model was tuned with CV. The CV boost +0.4. Let's see what the private LB score is.
Submission finished. Private LB = 60.29. I'm surprised i thought it would do better. Perhaps there is a bug in my code.
\nI also noticed that i can improve my MLP CV score but predicting multi-class and using your SMAPE optimization code. I wonder if that CV improvement would transfer to private LB.
", "votes": 0, "replies": [] }, { "id": 2266384, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-20T02:00:20.640000", "content": "Weird. It improves my MLP CV by +0.3 but degraded the private LB -0.1. So this SMAPE optimization does not help my model. I wonder why.
@cdeotte I’l try to look on it on Sunday and report you.
", "votes": 2, "replies": [] }, { "id": 2267346, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-20T19:45:00.223000", "content": "I just got reg:absoluteerror to work with XGB. It appears that Kaggle's version 1.7.5 has a bug with reg:absoluteerror. I pip installed 1.7.1 and now XGB achieves a similar CV score as MLP (only 0.1 worse) when both use reg:absoluteerror loss. I just created a notebook here and submitted to LB. Let's see what the LB is. Note that version 1 of this notebook uses your multi class trick, so we can compare that too.
Results are in. (Note these CV scores are from my local not on Kaggle CV scheme)
\nXGB with your multiclass hack - CV 54.65 - Private LB 60.29 (notebook version 1)
XGB with reg:absoluteerror - CV 55.05 - Private LB 60.50 (notebook version 4 improved)
MLP with your multiclass hack - CV 54.65 - Private LB 60.23
MLP with Absolute Error loss - CV 54.95 - Private LB 60.17
So it appears that your multiclass hack improves GBT on both CV and LB. But so far multiclass hack has not improved MLP. Perhaps if i spend more time tuning the learning rate and model architecture it would improve CV and LB for MLP
\nAnyway it is a huge success for GBT. Congratulations! Nice job
", "votes": 1, "replies": [] }, { "id": 2267398, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-20T20:50:30.127000", "content": "@cdeotte thank you so much for checking and analysis - your dedication and post competition work amazes me. A was planning to spend Sunday on exactly the same thing that you did with xgb, but now I have different plans - will decompose your mlp model))) - you have fantastic results with it.
", "votes": 2, "replies": [] }, { "id": 2267403, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-20T21:04:29.360000", "content": "Thanks. I will do a few more submissions with XGB. For the above four experiments i use a local (i.e. not on Kaggle) CV scheme which i think is closer to Kaggle's evaluation. However it is slightly different than the CV scheme in the notebook itself. In the notebook i can improve both version 1 and version 4 CV by reducing 'subsample':1.0. So i will try a few lower values and report the CVs and LBs from the notebook itself.
I also plan to do another experiment with your SMAPE optimizer. Regarding MLP, i can predict 11 folds. Then i want to try your trick of finding an optimal integer number which maximizes the SMAPE of those 11 float predictions. And use that as the final pred. Currently i just take the rounded mean.
", "votes": 3, "replies": [] }, { "id": 2268269, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-21T15:49:02.390000", "content": "I finished all my experiments. Using XGB reg:absoluteerror and 'subsample':0.5, i boosted the notebook's CV by +0.2 and improved the Private LB to 60.4. So using multiclass trick is still better (at LB 60.2) but using tuned reg:absoluteerror is close.
Next i took all MLP 11 fold predictions per prediction and found the integer that optimized SMAPE over these 11 fold float numbers. Surprisingly that did not affect the LB. My MLP still had Private LB 60.1.
In the end, i was able to boost my XGB using your tricks but not my MLP. I did not try ensemble. But perhaps now using an ensemble with the XGB and NN could achieve better than my Private LB 60.1
", "votes": 2, "replies": [] }, { "id": 2269551, "author_name": "dott", "author_url": "", "post_date": "2023-05-22T14:32:19.653000", "content": "Thank you for analysis Chris!
\nOn private LB a single NN (SMAPE1p regression) scored around 60.2-60.3 depending on the seed and rounding / not rounding the outputs. The LGB (classification + optimization post-processing) scored 60.122. Both models had approximately the same CV, NN was just a tiny bit better, so LB is also not perfectly correlated in our experiments. But the post-processing proved to be working as an approach.
We've tried also running a classification NN with the same post-processing optimization, but the SMAPE1p loss regression was a bit better on CV. Also we thought it should be better for the blend to use different approaches.
\nWith regards to blending we've tried several ideas, like averaging probabilities before post-processing, taking average/median/max across the predictions of individual models, but nothing really shined on CV. So we chose a simple round + average + round of 2 best models as it gave the best boost on CV (~0.1). It didn't improve public LB, but ended up giving the same boost on private.
\nI personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.
", "votes": 2, "replies": [] }, { "id": 2269572, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-22T14:55:58.637000", "content": "\n\nI personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.
\n
Yes, i was surprised by this too. In every previous competition using more copies of NN ensemble with itself boost LB but not here. In this comp, i tried 1 fold, 5 folds, and 11 folds and they all performed the same both on CV and LB. (On CV i simulated bagging within each fold iteration during KFold)
\nI think it is because the final model is just a linear model as shown by AmbrosM final solution here. I guess using more NN with itself does not improve a straight line any more than one NN.
", "votes": 1, "replies": [] } ] } ] }, { "id": 2266022, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T16:41:05.543000", "content": "Congratulations @dott1718 and @kyakovlev Fantastic job!
\n\n\nThe winning solution fully ignores the results of the blood tests. We’ve tried hard to find any signal in this crucial piece of the data, but unfortunately we came to the conclusion that none of our approaches or models can benefit from blood test features significant enough to distinguish it from random variations. The final models were trained only on the union of clinical and supplementary datasets.
\n
I just had a thought. Now that the competition ended and the signal was fully explained by AmbrosM here, it might be the case that there is signal in the protein peptide information. Perhaps the problem is that once our models detect the 3 groups of patients using visit_month then protein peptide does not provide more information. However if the host could hide the 3 groups of patients (i.e. remove the signal from visit month), I wonder whether the protein peptide information can help locate the 3 groups?
In other words, i do not think that this competition demonstrated that the protein peptide does not contain signal (by the fact that top teams did not use it.). I think this competition demonstrated that the signal from visit_month contains whatever signal may exist in the protein peptide information. Do you think this might be the case? If so, we should let the host and biology community know this.
Now i just read that you did try using blood without visit month. So perhaps you demonstrated that it does not have predictive power.
\n\n", "votes": 1, "replies": [ { "id": 2269564, "author_name": "dott", "author_url": "", "post_date": "2023-05-22T14:44:49.517000", "content": "Blood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities.
\n
Yes, it is exactly why we tried to narrow it down to a single homogenous group, but without much success. At some point I managed to get a CV improvement with some proteins data of a 0.01 magnitude which was much smaller than expected random effect on LB of around 0.1. So a different competition setup with more data might have been successful in identifying proteins/peptides.
\nThe way I managed to \"force\" proteins into the LGB model was to select the top 5-10 ones with the highest absolute correlation with the targets and scale down their gain with feature_contri parameter dramatically.
Why is leave-one-(patient)-out cross-validation also called \"stratified cross validation with a fold for each patient\" in the post?
\nI thought that for stratification we need our folds to be stratified for target, for example )
You are right, thank you for pointing that out! It is indeed group k-fold and not stratified k-fold, I've corrected the text to prevent potential confusion.
", "votes": 1, "replies": [] } ] }, { "id": 2266848, "author_name": "oka_haru", "author_url": "", "post_date": "2023-05-20T12:16:29.457000", "content": "Congratulations!!!
\nThanks for sharing your solution!!
Congratulations🎉
\nI really learned a lot from your source code and explanations.
\nThank you very much.
Congratulations Dmitry and Konstantin! What a brilliant skill to focus on one smaller part of data (visits) and have enough wisdom and experience to ignore the others (CSF). I would say it is one of my main outcomes from this competition.
\nI hope that organizers will not be dissatisfied since that all the data they provided were not used in a winning solutions and the hypothesis about predictive power of proteins and peptides is not confirmed yet. Anyway this competition makes all of us a bit closer to the understanding of the Parkinson's desease.
\nCongratulations to everyone who won in leaderbord and congratulations to all those who have beaten themselves in this competition. It was an addictive trip and interesting experience.
", "votes": 4, "replies": [] }, { "id": 2272120, "author_name": "Jai Chauhan", "author_url": "", "post_date": "2023-05-24T10:06:13.043000", "content": "Congratulations! sir
", "votes": 1, "replies": [] }, { "id": 2268528, "author_name": "Rida Mahmood", "author_url": "", "post_date": "2023-05-21T19:31:50.733000", "content": "Many congratulations @dott1718. Also I highly appreciate you for sharing your winning solution.
\nStay blessed!
Congratulations! Thanks for sharing your approach to a simple but effective solution.
", "votes": 1, "replies": [] }, { "id": 2267910, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-21T10:30:51.380000", "content": "https://www.kaggle.com/code/kyakovlev/amp-blood-randomness/
\nSmall example for \"fake\" correlations in proteins / peptides
", "votes": 1, "replies": [] }, { "id": 2270653, "author_name": "Gaurav Srivastava", "author_url": "", "post_date": "2023-05-23T09:53:36.787000", "content": "Heartiest Congratulations @dott1718. And thanks for sharing. Keep up the good work 😊
", "votes": 2, "replies": [] }, { "id": 2268112, "author_name": "Vincent Schuler", "author_url": "", "post_date": "2023-05-21T13:18:03.707000", "content": "Congratulations @dott1718 and @kyakovlev ! Best team name and best solution 💪
\nI tried multiclass classification as well but couldn't make it work better than a simple regression model. Thanks for sharing your method, it's instructive !
", "votes": 2, "replies": [ { "id": 2269513, "author_name": "dott", "author_url": "", "post_date": "2023-05-22T14:13:22.817000", "content": "Thank you Vincent! I was cheering for you to get to the gold zone on private LB, hopefully you are not too upset about the result.
", "votes": 1, "replies": [ { "id": 2269639, "author_name": "Vincent Schuler", "author_url": "", "post_date": "2023-05-22T15:37:29.450000", "content": "Thanks 😄 Nothing too frustrating, hopefully I'll get it another time !
", "votes": 0, "replies": [] }, { "id": 2269758, "author_name": "dott", "author_url": "", "post_date": "2023-05-22T17:24:39.700000", "content": "I am sure you will!
", "votes": 1, "replies": [] } ] } ] }, { "id": 2267071, "author_name": "Aaron C Bastian", "author_url": "", "post_date": "2023-05-20T15:02:13.793000", "content": "It is unfortunate that no signal was found in the protein data, as that was the entire point of the challenge. Parkinson's remains a clinical diagnosis it seems.
", "votes": 2, "replies": [] }, { "id": 2266900, "author_name": "Ion05", "author_url": "", "post_date": "2023-05-20T13:01:59.950000", "content": "Congratulations!
\nThanks for sharing your solution with us. It has been very helpful for me.
\nThank you!!
very interesting discussion and work, thank you for taking the time to update on this, its interesting that the blood test data was not directly an input
", "votes": 2, "replies": [] }, { "id": 2266752, "author_name": "Jayanth Sai Marreddi", "author_url": "", "post_date": "2023-05-20T10:52:07.053000", "content": "Congratulations! Thanks for sharing your solution gives a lot insights to me to see how does top people in leaderboard approach a project.
", "votes": 2, "replies": [] }, { "id": 2266647, "author_name": "Viji", "author_url": "", "post_date": "2023-05-20T08:39:37.713000", "content": "Congratulations! Thanks for sharing your approach to a simple but effective solution.
", "votes": 2, "replies": [] }, { "id": 2266028, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T16:49:43.167000", "content": "Thanks for sharing your solution. I never achieved good results from GBT. (I think CV was 57 compared with my MLP's 55). And i did not achieve any significant boosts from SMAPE optimization (besides rounding). I also did not find a better loss for my MLP than MeanAbsoluteError loss.
\nNow after reading your solution, i may go back and revisit these ideas and see if i can boost my MLP by ensembling it with GBT and adding some SMAPE optimization.
\nMy two top models were MLP (4th place) and SVR (9th place) but the CV did not boost when i ensembled the two.
", "votes": 2, "replies": [ { "id": 2266044, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-19T17:02:46.523000", "content": "Same with our ensembles - NN with lgbm showed just a tiiiny boost on CV.
", "votes": 1, "replies": [] }, { "id": 2266067, "author_name": "Chris Deotte", "author_url": "", "post_date": "2023-05-19T17:17:00.267000", "content": "I like your trick of training multi-class then optimizing output to SMAPE. That's clever.
\n\n", "votes": 2, "replies": [] } ] }, { "id": 2410427, "author_name": "MichaelDarling", "author_url": "", "post_date": "2023-08-26T23:28:52.627000", "content": "Our LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.
\n
Hi @dott1718, congratulations! Would you be able to send along the testing data provided by the competition? I have a student working with this data for his capstone project and would love to get this data. We contacted the competition organizers but haven't heard back.
", "votes": 0, "replies": [] }, { "id": 2328592, "author_name": "Aleksey SCHUKIN", "author_url": "", "post_date": "2023-07-03T17:45:31.560000", "content": "Congratulations! Thanks for sharing your approach.
", "votes": 0, "replies": [] }, { "id": 2287825, "author_name": "Tomonori Sasaki", "author_url": "", "post_date": "2023-06-05T00:14:32.767000", "content": "Congratulations!
\nThank you for publishing and sharing.
\nThe discussion around the metrics in this competition was difficult and confusing to me. I will learn from the published notes.
Good job,thank you very much. Because I can learn from it .
", "votes": 0, "replies": [] }, { "id": 2276543, "author_name": "Jeong Taewoo", "author_url": "", "post_date": "2023-05-27T00:54:56.890000", "content": "I thought XGBoost almost always had good performance, but I guess it wasn't necessarily. Your writing and notebook helped me improve my skills a lot. Thank you!!
", "votes": 0, "replies": [ { "id": 2292313, "author_name": "Josef Švenda", "author_url": "", "post_date": "2023-06-08T07:59:20.263000", "content": "As I understood the discussion: XGB would be a comparable method if it would have custom objective function aligned with SMAPE metric. Right @cdeotte, @dott1718, @kyakovlev ?
\nAnyway excellent learning opportunity! Thanks guys!
Congratulations !! Well done 🔥
", "votes": 0, "replies": [] }, { "id": 2272174, "author_name": "3SE3_212011556_Aronad Brilliantio Evan Maheswara", "author_url": "", "post_date": "2023-05-24T10:37:07.383000", "content": "Congratulations! Thanks for sharing your approach.
", "votes": 0, "replies": [] }, { "id": 2271694, "author_name": "Nooruddin Hyderabadwalla, CFA", "author_url": "", "post_date": "2023-05-24T05:29:40.277000", "content": "Many congratulations @dott1718 on winning this competition. Reading through your solution is a learning experience for me, very insightful. Thank you so much for sharing.
", "votes": 0, "replies": [] }, { "id": 2267461, "author_name": "Murugesan Narayanaswamy", "author_url": "", "post_date": "2023-05-21T00:35:05.700000", "content": "The objective of this competition is to discern what kind of correlation the protein / peptides (levels) have on disease progression. Hence, I never tried any model (though I joined the competition in the last week) without these data till near last few days when I was shocked to find a simple linear regression model with just visit month column alone as training data could beat all my models and achieve a score of 55!!
\nThe conclusion of the prize winners that the protein/peptides (levels) do not have any correlation is not correct. That visit month alone can accurately predict the disease progression does not imply that the protein/peptides have no correlation.
\nI tried an unusual deep learning model that considers only the peptide information. I created an embedding layer based on the peptide information by coding each amino acid label of the peptide by creating all peptides present during the visit as a list against each visit. The embedding layer weights of the amino acid tokens were trained against the updrs scores. I got near perfect predictions for the training data, though the model did not converge for the validation data. I think deep learning models can learn any kind of functions but might fail in predicting test data which might also imply that there is no correlation. However, on further analysis, I concluded that this could also be because the training data is not sufficient. Each visit and updrs scores are trained against the presence of peptides. The list of peptides in terms of amino acids is huge for each updrs score. Given this, the training data may not be even 10% of what is required for such predictions. Hence, it is quite possible that the presence or absence of peptides have correlation with disease progress but the dataset is not sufficient enough to predict them.
\nThe dataset focusses on the levels of NPX and peptides rather than the number of patients. The training dataset looks sufficient if you go by the levels of peptides and proteins. But if you go by the number of patients with respect to presence or absence of peptide amino acids from the overall universe of possible peptides, we can say for sure that the training dataset is not at all sufficient to predict any relationship between presence or absence of protein/peptides. The dataset size should be measured in terms of number of patients in this case. This is my final conclusion.
\nAnyway, congrats to the winners for their hard work!
", "votes": 0, "replies": [] }, { "id": 2267424, "author_name": "", "author_url": "", "post_date": "2023-05-20T21:55:33.930000", "content": "", "votes": 1, "replies": [ { "id": 2267761, "author_name": "Konstantin Yakovlev", "author_url": "", "post_date": "2023-05-21T08:01:50.460000", "content": "I really like your \"confidence\" in the approach that you've created, but you are wrong.
\nThere is an error in CV or Metric or any other place.
\nIt is easy to check if you really want to dig deeper.
I'm really sorry to say but there is no significant signal in blood samples.
\nOur standard dialog with Dmitry every week:
Some times due to my naiveness and fast prototyping errors I screamed in chat \"I FOUND!!!!!!\" and in a 15 mins Dmitry or even me found and error or leak in pipeline - that was such a joy to see how Dmitry is criticising my \"ideas\" - I'll miss our conversations and discussions.
", "votes": 1, "replies": [ { "id": 2268733, "author_name": "Murugesan Narayanaswamy", "author_url": "", "post_date": "2023-05-22T00:49:33.947000", "content": "As per a research paper I sampled, there are around 13879 peptides which differ from normal healthy people compared to parkinson' diseased patients. Of these peptides, 1466 are observable from the CSF sample taken from brain fluids. Among these 192 peptides are relevant for parkinson's diease. Of these around 20 can be used as markers for Parkinson disease and its progression. (https://www.sciencedirect.com/science/article/pii/S1535947620332217#:~:text=Furthermore%2C%20a%20combination%20of%20two%20peptides%20belonging%20to,validation%20%28r%20%3D%200.339%2C%20p%20%3D%200.032%29%20sets.)
\nHence, that there is signal in peptide information with regard to parkinson disease is scientifically proved and is an important research area in parkinson diesease and hence this competition.
\nIn our training dataset, for each visit of a patient, peptide levels are given. There are around 900 peptides for which measurements were taken during each visit. This actually varies between 600 to 980 across the patients. It is not clear whether for patients with just 600 peptide information, the other 300 peptides are not present in the CSF sample etc.
\nMy very preliminary attempt was to correlate this list of peptides for each patient visit with the UPDRS scores (idea is to consider peptide levels and protein relationships later - subsequent models should incorporate the peptides abundance level and protein relationships that can influence the embedding weights)). To put it more naively and in raw terms, from the perspective of machine learning model, does presence of 900 peptides vs 600 peptides has any impact on the disease condition?
\nSo, I embedded the peptide information using keras embedding layer after tokenizing them based on overall peptide vocabulary. The weights are learnt against the UPDRS scores. This training gave near perfect prediction for training data. But the results against the test data were not good enough to try in the leaderboard(!).
\nMy contention is that the performance in the test data could be poor because of the size of the training dataset size. For example, for each row, there are 900 peptides which are correlated with the scores (If you embed these peptides at the character level which is more correct way, it would be far more). Assuming that the above quoted research is valid, out of these 900 peptides, only around 20 have impact on the disease progression. Now, deep learning model has to study the relationships between the peptides and the scores based on around 300 patients (900 features vs 300 rows). Theoretically, if the model has to learn the relationships, it would definitely require lot more data. This is my conclusion.
\nI am not saying that because my embedding based deep learning model can predict the training data perfectly, that there definitely exists a signal identified by the model. Deep learning models can learn any function and predict perfectly - but it could just be nothing more than overfitting. But in all these cases, only by increasing the dataset size, can we say for sure about existence of any relationships.
\nHere is the code of my naive and very preliminary model (I am not an expert in python - So, I asked chatgpt and created this entire model in just 30 minutes!! :-) :-)):
\ndf=train_peptides
\ngrouped_df = df.groupby('visit_id').agg({'visit_month': 'first', 'patient_id': 'first', 'Peptide': list, 'PeptideAbundance': 'mean'})
\ngrouped_df.reset_index(inplace=True)
\nmerged_df = pd.merge(train_clinical, grouped_df, on=['visit_id', 'patient_id'], how='left')
\nprint(merged_df)
\nmerged_df=merged_df[[\"visit_id\",\"visit_month_x\",\"updrs_1\",\"updrs_2\",\"updrs_3\",\"updrs_4\",\"Peptide\",\"PeptideAbundance\"]]
\nmerged_df.rename(columns={\"visit_month_x\":\"visit_month\"},inplace=True)
df = merged_df.copy(deep=True)
\ndf[\"Peptide\"].replace(\"nan\", np.nan, inplace=True)
\ndf.dropna(inplace=True)
\ndf['Peptide'] = df['Peptide'].astype(str)
\ntokenizer = Tokenizer()
\ntokenizer.fit_on_texts(df['Peptide'])
\nsequences= tokenizer.texts_to_sequences(df['Peptide'])
--- wt=df[\"PeptideAbundance\"] / df[\"PeptideAbundance\"].sum()
\n--- sequences = [x * y if isinstance(x, int) or isinstance(x, float) else [xi * y for xi in x] for x, y in zip(sequences, wt)]
\n--- print(result)
\nmax_length = max([len(seq) for seq in sequences])
\npadded_sequences = pad_sequences(sequences, maxlen=max_length, padding='post')
\nprint(\"padded seq shape\",padded_sequences.shape)
\ndf_vm=df[\"visit_month\"] #*10000
\nsequence_length = max_length # Length of padded sequence
\nvocab_size = len(tokenizer.word_index) + 1 # Vocabulary size
\nembedding_dim = 50 # Dimension of embedding space
\nsequence_input = Input(shape=(sequence_length,), dtype='int32', name='sequence_input')
\ndf_input = Input(shape=(1,), dtype='float32', name='df_input')
\nembedding_layer = Embedding(vocab_size, embedding_dim)(sequence_input)
\nflattened_sequence = tf.keras.layers.Flatten()(embedding_layer)
\nmerged_features = Concatenate(axis=1)([flattened_sequence, df_input])
\ntf.random.set_seed(1123)
\nactivation='relu'
\nx=Dense(64,activation='tanh',kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(merged_features)
\n kernel_regularizer=tf.keras.regularizers.L2(0.01))(merged_features) #9flattened_sequence)
\nx=Dense(32,activation=activation,kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\nx=Dense(10,activation=activation,kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\noutput=Dense(1,activation='relu',kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\ntarget = ['updrs_3', 'updrs_2', 'updrs_1'] #, 'updrs_4']
\nratings=df['updrs_1']
\n--# ratings = df[['updrs_1', 'updrs_2', 'updrs_3']]
\nseq_train, seq_test, dfvm_train,dfvm_test,ratings_train, ratings_test = train_test_split(
\n padded_sequences, df_vm, ratings, test_size=0.2, random_state=42)
\nmodel1= Model(inputs=[sequence_input,df_input], outputs=output)
\nlearning_rate = 0.01
\noptimizer = Adam(learning_rate=learning_rate)
\nmodel1.compile(optimizer=optimizer, loss='mse', metrics=['mae',smape_metric]) #SMAPE()
\nmodel1.fit([seq_train,dfvm_train], ratings_train, epochs=100, batch_size=32, validation_data=([seq_test,dfvm_test], ratings_test))
Thank you. It was helpful.
", "votes": 0, "replies": [] } ], "comment_count": 59, "messages": [ { "id": 2265822, "postDate": "2023-05-19T14:32:14.043Z", "content": "First of all we would like to thank Kaggle and AMP PD for hosting this competition and providing a great dataset to dig into and rich enough to get lost in it for several months. It is always an extra motivation to work on problems that can bring value to the medical field or contribute to scientific research. Additional words of gratitude of course go to @kyakovlev for the amazing work he has done, I think we formed a great team for this competition and the results reflect that.
\nWe publish the full code of the winning notebook here
\nOur final solution is a simple average of two models: LGB and NN. Both models were trained on the same features (+ scaling/binarization for NN):
\nThe winning solution fully ignores the results of the blood tests. We’ve tried hard to find any signal in this crucial piece of the data, but unfortunately we came to the conclusion that none of our approaches or models can benefit from blood test features significant enough to distinguish it from random variations. The final models were trained only on the union of clinical and supplementary datasets.
\nFor the entire duration of the competition LGB was our model to beat and only a NN trained with the competition metric as the loss function was able to achieve competitive performance on CV. At first, we tried running a regression LGB model with different hyperparameters and custom objective functions, but nothing was better than l1 regression, which does not optimize the desired metric SMAPE+1. We also noticed that on CV the performance of every model is always better when the regression outputs are rounded to integers. Then we switched to an alternative approach.
\nOur LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.
\nWe ran an optimization routine to tune LGB hyperparameters to minimize SMAPE+1 on CV using the described post-processing.
\nThe neural network has a simple multi-layer feed-forward architecture with a regression target, using the competition metric SMAPE+1 as the loss function. We fixed the number of epochs and scheduler, and then tuned the learning rate and hidden layer size. The only trick there was to add a leaky relu activation as the last layer to prevent NN from getting stuck at negative predictions. Of course there are alternative ways to solve this issue.
\nWe’ve tried multiple cross-validation schemes due to the small training sample size, all of them were stratified by patient id. Once a sufficient number of folds is used, they all are quite well correlated to each other. Better than to the public leaderboard :) The final scheme we relied on was leave-one-(patient)-out or, in other words, a group k-fold cross validation with a fold for each patient. We used it because it doesn’t depend on random numbers. The cross-validation correlated well enough with the private leaderboard, and the submit we chose turned out to be our best private LB submission.
\nThe most impactful feature was the indication of whether a patient visit happened on the 6th month or not. It correlates strongly with the UPDRS targets (especially 2 and 3) and with frequency of medications being taken. As we can observe only the data correlation, it is impossible to judge what is the core reason for that. During the competition our hypothesis was that the patients that had more severe symptoms during the first examination (UPDRS scores parts 2 and 3) were more likely to get invited for a visit after 6 months and more likely to get medications prescribed. But for model training it was important that the patients that made a visit on the 6th month, have higher UPDRS scores on average. The same is true for an 18th month visit as well, but these 2 features are correlated. I personally wonder if presence / absence of these variables in the models are the reason for the private LB cliff effect around 20th place.
\nAnother curious effect related to it is observed for the forecasts made at visit_month = 0. If you look at the model forecasts for 0, 12 and 24 months ahead, they are consistently lower than the forecasts 6 months ahead. It is very logical from the math point of view - if a patient will show up on the 6th month, they will have higher UPDRS scores on average, and if not - the forecast will be ignored. But such model behaviour is unreasonable from a clinical point of view of course.
\nIt was also important to pay attention to the differences between training and test datasets as e.g. nicely summarized here. That, for instance, explains well why adding a feature indicating the visit on the 30th month could improve the CV, but ruin LB.
\nBlood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities. It worked on public LB after tuning a couple of “magic” coefficients directly on public LB itself. That gave us a boost all the way up to the second place on public LB, but it was clearly an overfit. We chose a “mild” version of that approach as our second final submission. It scored worse on private LB than the other submission, but, interestingly enough, not by as much as one could have expected (60.0 vs 60.3).
\nThanks to everyone who participated in the competition, those who kept many interesting discussions going on the forum and those who suggested improvements! And congrats to all the winners!
", "rawMarkdown": "First of all we would like to thank Kaggle and AMP PD for hosting this competition and providing a great dataset to dig into and rich enough to get lost in it for several months. It is always an extra motivation to work on problems that can bring value to the medical field or contribute to scientific research. Additional words of gratitude of course go to @kyakovlev for the amazing work he has done, I think we formed a great team for this competition and the results reflect that.\n\nWe publish the full code of the winning notebook [here](https://www.kaggle.com/dott1718/amp-model-stack3)\n\n## Quick summary\n\nOur final solution is a simple average of two models: LGB and NN. Both models were trained on the same features (+ scaling/binarization for NN):\n- Visit month\n- Forecast horizon\n- Target prediction month\n- Indicator whether blood was taken during the visit\n- Supplementary dataset indicator\n- Indicators whether a patient visit occurred on 6th, 18th and 48th month\n- Count of number of previous “non-annual” visits (6th or 18th)\n- Index of the target (we pivot the dataset to have a single target column)\n\nThe winning solution fully ignores the results of the blood tests. We’ve tried hard to find any signal in this crucial piece of the data, but unfortunately we came to the conclusion that none of our approaches or models can benefit from blood test features significant enough to distinguish it from random variations. The final models were trained only on the union of clinical and supplementary datasets.\n\n\n## LGB\n\nFor the entire duration of the competition LGB was our model to beat and only a NN trained with the competition metric as the loss function was able to achieve competitive performance on CV. At first, we tried running a regression LGB model with different hyperparameters and custom objective functions, but nothing was better than l1 regression, which does not optimize the desired metric SMAPE+1. We also noticed that on CV the performance of every model is always better when the regression outputs are rounded to integers. Then we switched to an alternative approach.\n\nOur LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.\n\nWe ran an optimization routine to tune LGB hyperparameters to minimize SMAPE+1 on CV using the described post-processing.\n\n\n## NN\n\nThe neural network has a simple multi-layer feed-forward architecture with a regression target, using the competition metric SMAPE+1 as the loss function. We fixed the number of epochs and scheduler, and then tuned the learning rate and hidden layer size. The only trick there was to add a leaky relu activation as the last layer to prevent NN from getting stuck at negative predictions. Of course there are alternative ways to solve this issue.\n\n\n## Cross-Validation\n\nWe’ve tried multiple cross-validation schemes due to the small training sample size, all of them were stratified by patient id. Once a sufficient number of folds is used, they all are quite well correlated to each other. Better than to the public leaderboard :) The final scheme we relied on was leave-one-(patient)-out or, in other words, a group k-fold cross validation with a fold for each patient. We used it because it doesn’t depend on random numbers. The cross-validation correlated well enough with the private leaderboard, and the submit we chose turned out to be our best private LB submission.\n\n\n## What worked\n\nThe most impactful feature was the indication of whether a patient visit happened on the 6th month or not. It correlates strongly with the UPDRS targets (especially 2 and 3) and with frequency of medications being taken. As we can observe only the data correlation, it is impossible to judge what is the core reason for that. During the competition our hypothesis was that the patients that had more severe symptoms during the first examination (UPDRS scores parts 2 and 3) were more likely to get invited for a visit after 6 months and more likely to get medications prescribed. But for model training it was important that the patients that made a visit on the 6th month, have higher UPDRS scores on average. The same is true for an 18th month visit as well, but these 2 features are correlated. I personally wonder if presence / absence of these variables in the models are the reason for the private LB cliff effect around 20th place.\n\nAnother curious effect related to it is observed for the forecasts made at visit_month = 0. If you look at the model forecasts for 0, 12 and 24 months ahead, they are consistently lower than the forecasts 6 months ahead. It is very logical from the math point of view - if a patient will show up on the 6th month, they will have higher UPDRS scores on average, and if not - the forecast will be ignored. But such model behaviour is unreasonable from a clinical point of view of course.\n\nIt was also important to pay attention to the differences between training and test datasets as e.g. nicely summarized [here](https://www.kaggle.com/competitions/amp-parkinsons-disease-progression-prediction/discussion/403045). That, for instance, explains well why adding a feature indicating the visit on the 30th month could improve the CV, but ruin LB.\n\n\n## What didn’t work\n\nBlood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities. It worked on public LB after tuning a couple of “magic” coefficients directly on public LB itself. That gave us a boost all the way up to the second place on public LB, but it was clearly an overfit. We chose a “mild” version of that approach as our second final submission. It scored worse on private LB than the other submission, but, interestingly enough, not by as much as one could have expected (60.0 vs 60.3).\n\n\nThanks to everyone who participated in the competition, those who kept many interesting discussions going on the forum and those who suggested improvements! And congrats to all the winners!", "votes": 99 }, { "id": 2265828, "postDate": "2023-05-19T14:41:03.777Z", "content": "It was a huge huge pleasure to team up with @dott1718 (Dmitry) in this competition. I felt as a total junior DS - Dmitry's ideas, approach and dedication were astonishing - I've learned so much and so grateful for all the time we spent together.
\nThis competition was super unique for me with many new puzzles to solve.
\nThank you again and congrats to all the winners and participants - It was remarkable ride.
", "rawMarkdown": "It was a huge huge pleasure to team up with @dott1718 (Dmitry) in this competition. I felt as a total junior DS - Dmitry's ideas, approach and dedication were astonishing - I've learned so much and so grateful for all the time we spent together.\n\nThis competition was super unique for me with many new puzzles to solve. \n\nThank you again and congrats to all the winners and participants - It was remarkable ride.", "votes": 13, "replies": [ { "id": 2265845, "postDate": "2023-05-19T14:52:08.873Z", "content": "A SMAPE1P thing:
\nMAPE / SMAPE / SMAPE1P are very tricky metrics for GBT models:
\nHere is the \"hack\" that is boosting the SMAPE1P score (and works also on real data - confirmed few weeks ago):
\nStep 1: Multiclass Classification
\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'multiclass',\n...\n}\nStep 2: Train model (could be xgboost catboost lgbm)
\nestimator.train(...)\nStep 3: Predict probabilities:
\nprobas = estimator.predict(...)\nStep 4: Make direct SMAPE1P optimization from probabilities
\ndef single_smape1p(preds, tgt):\n x = np.tile(np.arange(preds.shape[1]), (preds.shape[0], 1))\n x = np.abs(x - tgt) / (2 + x + tgt)\n return (x * preds).sum(axis=1)\n\ndef opt_smape1p(preds):\n x = np.hstack([single_smape1p(preds, i).reshape(-1,1) for i in range(preds.shape[1])])\n return x.argmin(axis=1)\n\nfinal_predictions = opt_smape1p(probas)\nIf you are sure that \"SMAPE\" variation is the metric that reflects your business value - do the \"hack\" (at least try).
", "rawMarkdown": "A SMAPE1P thing:\n\nMAPE / SMAPE / SMAPE1P are very tricky metrics for GBT models:\n- Need custom loss or target transform with l1 loss (l1 is never optimal choice for gbt)\n- Multimodal\n- Zero inflation\n- ....\n\nHere is the \"hack\" that is boosting the SMAPE1P score (and works also on real data - confirmed few weeks ago):\n\nStep 1: Multiclass Classification \n```\nparams = {\n 'boosting_type': 'gbdt',\n 'objective': 'multiclass',\n...\n}\n```\n\nStep 2: Train model (could be xgboost catboost lgbm)\n```\nestimator.train(...)\n```\n\nStep 3: Predict probabilities:\n```\nprobas = estimator.predict(...)\n```\n\nStep 4: Make direct SMAPE1P optimization from probabilities\n```\ndef single_smape1p(preds, tgt):\n x = np.tile(np.arange(preds.shape[1]), (preds.shape[0], 1))\n x = np.abs(x - tgt) / (2 + x + tgt)\n return (x * preds).sum(axis=1)\n\ndef opt_smape1p(preds):\n x = np.hstack([single_smape1p(preds, i).reshape(-1,1) for i in range(preds.shape[1])])\n return x.argmin(axis=1)\n\nfinal_predictions = opt_smape1p(probas)\n\n```\n\nIf you are sure that \"SMAPE\" variation is the metric that reflects your business value - do the \"hack\" (at least try). ", "votes": 16, "replies": [ { "id": 2266076, "postDate": "2023-05-19T17:23:22.423Z", "content": "Amazing hack! Great discovery and thanks for sharing!
", "rawMarkdown": "Amazing hack! Great discovery and thanks for sharing!", "votes": 5 }, { "id": 2266099, "postDate": "2023-05-19T17:52:21.440Z", "content": "Wow amazing it is working. My MLP CV = 54.95. So far using your hack and XGB, I achieved CV = 54.79 with my features. I will finish tuning XGB with your hack and then make a submission to LB. I will also try to ensemble with my MLP. How do you ensemble? Do we convert the XGB preds into integers and then ensemble with MLP, or is there a better way?
We’ve tried all possible ways - weighted rounded ensemble / probabilities blend / voting / … nothing really worked on cv - just a very slight move. We ended up with this
\n sample_test[\"preds_lgb\"] = model_lgb.predict(sample_test)\n sample_test[\"preds_nn\"] = np.round(np.clip(model_nn.predict(sample_test), 0, None))\n\n sample_submission[\"rating\"] = np.round( (sample_test[\"preds_lgb\"] + sample_test[\"preds_nn\"]) / 2)\nMean with rounding.
", "rawMarkdown": "We’ve tried all possible ways - weighted rounded ensemble / probabilities blend / voting / … nothing really worked on cv - just a very slight move. We ended up with this\n\n```\n sample_test[\"preds_lgb\"] = model_lgb.predict(sample_test)\n sample_test[\"preds_nn\"] = np.round(np.clip(model_nn.predict(sample_test), 0, None))\n \n sample_submission[\"rating\"] = np.round( (sample_test[\"preds_lgb\"] + sample_test[\"preds_nn\"]) / 2)\n\n```\nMean with rounding.", "votes": 2 }, { "id": 2266113, "postDate": "2023-05-19T18:05:05.340Z", "content": "I’m convincing Dmitry to write a paper about this tiny part of the code - because it’s “gamechanger” for all MAPE lovers)))(no “dancing” around target/straight forward approach / better results) he is still thinking (not a big deal he said - please say that he is wrong)))))
", "rawMarkdown": "I’m convincing Dmitry to write a paper about this tiny part of the code - because it’s “gamechanger” for all MAPE lovers)))(no “dancing” around target/straight forward approach / better results) he is still thinking (not a big deal he said - please say that he is wrong)))))", "votes": 1 }, { "id": 2266149, "postDate": "2023-05-19T18:48:17.607Z", "content": "Wow, i have my XGB CV = 54.64 using your hack. This is +0.3 better than my MLP CV and I'm still tuning. I'm curious if this will boost my LB score this much!
Waiting for your lb probing - all our tests showed that loo cv was very stable and aligned with lb and we had the same boost on loo cv / private / public lb.
", "rawMarkdown": "Waiting for your lb probing - all our tests showed that loo cv was very stable and aligned with lb and we had the same boost on loo cv / private / public lb.", "votes": 2 }, { "id": 2266222, "postDate": "2023-05-19T20:44:47.883Z", "content": "@kyakovlev I made a notebook here with single model XGB and just submitted to the LB. We should have a score in a few hours. Note that this is the first submission, so we have not adjusted the model to boost private LB. The model was tuned with CV. The CV boost +0.4. Let's see what the private LB score is.
Submission finished. Private LB = 60.29. I'm surprised i thought it would do better. Perhaps there is a bug in my code.
\nI also noticed that i can improve my MLP CV score but predicting multi-class and using your SMAPE optimization code. I wonder if that CV improvement would transfer to private LB.
", "rawMarkdown": "Submission finished. Private LB = 60.29. I'm surprised i thought it would do better. Perhaps there is a bug in my code. \n\nI also noticed that i can improve my MLP CV score but predicting multi-class and using your SMAPE optimization code. I wonder if that CV improvement would transfer to private LB." }, { "id": 2266384, "postDate": "2023-05-20T02:00:20.640Z", "content": "Weird. It improves my MLP CV by +0.3 but degraded the private LB -0.1. So this SMAPE optimization does not help my model. I wonder why.
@cdeotte I’l try to look on it on Sunday and report you.
", "rawMarkdown": "@cdeotte I’l try to look on it on Sunday and report you.", "votes": 2 }, { "id": 2267346, "postDate": "2023-05-20T19:45:00.223Z", "content": "I just got reg:absoluteerror to work with XGB. It appears that Kaggle's version 1.7.5 has a bug with reg:absoluteerror. I pip installed 1.7.1 and now XGB achieves a similar CV score as MLP (only 0.1 worse) when both use reg:absoluteerror loss. I just created a notebook here and submitted to LB. Let's see what the LB is. Note that version 1 of this notebook uses your multi class trick, so we can compare that too.
Results are in. (Note these CV scores are from my local not on Kaggle CV scheme)
\nXGB with your multiclass hack - CV 54.65 - Private LB 60.29 (notebook version 1)
XGB with reg:absoluteerror - CV 55.05 - Private LB 60.50 (notebook version 4 improved)
MLP with your multiclass hack - CV 54.65 - Private LB 60.23
MLP with Absolute Error loss - CV 54.95 - Private LB 60.17
So it appears that your multiclass hack improves GBT on both CV and LB. But so far multiclass hack has not improved MLP. Perhaps if i spend more time tuning the learning rate and model architecture it would improve CV and LB for MLP
\nAnyway it is a huge success for GBT. Congratulations! Nice job
", "rawMarkdown": "Results are in. (Note these CV scores are from my local not on Kaggle CV scheme)\n* XGB with your multiclass hack - CV 54.65 - Private LB 60.29 (notebook version 1)\n* XGB with `reg:absoluteerror` - CV 55.05 - Private LB 60.50 (notebook version 4 improved)\n\n* MLP with your multiclass hack - CV 54.65 - Private LB 60.23\n* MLP with Absolute Error loss - CV 54.95 - Private LB 60.17\n\nSo it appears that your multiclass hack improves GBT on both CV and LB. But so far multiclass hack has not improved MLP. Perhaps if i spend more time tuning the learning rate and model architecture it would improve CV and LB for MLP\n\nAnyway it is a huge success for GBT. Congratulations! Nice job", "votes": 1 }, { "id": 2267398, "postDate": "2023-05-20T20:50:30.127Z", "content": "@cdeotte thank you so much for checking and analysis - your dedication and post competition work amazes me. A was planning to spend Sunday on exactly the same thing that you did with xgb, but now I have different plans - will decompose your mlp model))) - you have fantastic results with it.
", "rawMarkdown": "@cdeotte thank you so much for checking and analysis - your dedication and post competition work amazes me. A was planning to spend Sunday on exactly the same thing that you did with xgb, but now I have different plans - will decompose your mlp model))) - you have fantastic results with it.", "votes": 2 }, { "id": 2267403, "postDate": "2023-05-20T21:04:29.360Z", "content": "Thanks. I will do a few more submissions with XGB. For the above four experiments i use a local (i.e. not on Kaggle) CV scheme which i think is closer to Kaggle's evaluation. However it is slightly different than the CV scheme in the notebook itself. In the notebook i can improve both version 1 and version 4 CV by reducing 'subsample':1.0. So i will try a few lower values and report the CVs and LBs from the notebook itself.
I also plan to do another experiment with your SMAPE optimizer. Regarding MLP, i can predict 11 folds. Then i want to try your trick of finding an optimal integer number which maximizes the SMAPE of those 11 float predictions. And use that as the final pred. Currently i just take the rounded mean.
", "rawMarkdown": "Thanks. I will do a few more submissions with XGB. For the above four experiments i use a local (i.e. not on Kaggle) CV scheme which i think is closer to Kaggle's evaluation. However it is slightly different than the CV scheme in the notebook itself. In the notebook i can improve both version 1 and version 4 CV by reducing `'subsample':1.0`. So i will try a few lower values and report the CVs and LBs from the notebook itself.\n\nI also plan to do another experiment with your SMAPE optimizer. Regarding MLP, i can predict 11 folds. Then i want to try your trick of finding an optimal integer number which maximizes the SMAPE of those 11 float predictions. And use that as the final pred. Currently i just take the rounded mean.", "votes": 3 }, { "id": 2268269, "postDate": "2023-05-21T15:49:02.390Z", "content": "I finished all my experiments. Using XGB reg:absoluteerror and 'subsample':0.5, i boosted the notebook's CV by +0.2 and improved the Private LB to 60.4. So using multiclass trick is still better (at LB 60.2) but using tuned reg:absoluteerror is close.
Next i took all MLP 11 fold predictions per prediction and found the integer that optimized SMAPE over these 11 fold float numbers. Surprisingly that did not affect the LB. My MLP still had Private LB 60.1.
In the end, i was able to boost my XGB using your tricks but not my MLP. I did not try ensemble. But perhaps now using an ensemble with the XGB and NN could achieve better than my Private LB 60.1
", "rawMarkdown": "I finished all my experiments. Using XGB `reg:absoluteerror` and `'subsample':0.5`, i boosted the notebook's CV by `+0.2` and improved the Private LB to `60.4`. So using multiclass trick is still better (at LB 60.2) but using tuned `reg:absoluteerror` is close.\n\nNext i took all MLP 11 fold predictions per prediction and found the integer that optimized SMAPE over these 11 fold float numbers. Surprisingly that did not affect the LB. My MLP still had Private LB `60.1`.\n\nIn the end, i was able to boost my XGB using your tricks but not my MLP. I did not try ensemble. But perhaps now using an ensemble with the XGB and NN could achieve better than my Private LB 60.1", "votes": 2 }, { "id": 2269551, "postDate": "2023-05-22T14:32:19.653Z", "content": "Thank you for analysis Chris!
\nOn private LB a single NN (SMAPE1p regression) scored around 60.2-60.3 depending on the seed and rounding / not rounding the outputs. The LGB (classification + optimization post-processing) scored 60.122. Both models had approximately the same CV, NN was just a tiny bit better, so LB is also not perfectly correlated in our experiments. But the post-processing proved to be working as an approach.
We've tried also running a classification NN with the same post-processing optimization, but the SMAPE1p loss regression was a bit better on CV. Also we thought it should be better for the blend to use different approaches.
\nWith regards to blending we've tried several ideas, like averaging probabilities before post-processing, taking average/median/max across the predictions of individual models, but nothing really shined on CV. So we chose a simple round + average + round of 2 best models as it gave the best boost on CV (~0.1). It didn't improve public LB, but ended up giving the same boost on private.
\nI personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.
", "rawMarkdown": "Thank you for analysis Chris!\nOn private LB a single NN (SMAPE1p regression) scored around 60.2-60.3 depending on the seed and rounding / not rounding the outputs. The LGB (classification + optimization post-processing) scored 60.122. Both models had approximately the same CV, NN was just a tiny bit better, so LB is also not perfectly correlated in our experiments. But the post-processing proved to be working as an approach.\n\nWe've tried also running a classification NN with the same post-processing optimization, but the SMAPE1p loss regression was a bit better on CV. Also we thought it should be better for the blend to use different approaches.\n\nWith regards to blending we've tried several ideas, like averaging probabilities before post-processing, taking average/median/max across the predictions of individual models, but nothing really shined on CV. So we chose a simple round + average + round of 2 best models as it gave the best boost on CV (~0.1). It didn't improve public LB, but ended up giving the same boost on private.\n\nI personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.", "votes": 2 }, { "id": 2269572, "postDate": "2023-05-22T14:55:58.637Z", "content": "\n\nI personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.
\n
Yes, i was surprised by this too. In every previous competition using more copies of NN ensemble with itself boost LB but not here. In this comp, i tried 1 fold, 5 folds, and 11 folds and they all performed the same both on CV and LB. (On CV i simulated bagging within each fold iteration during KFold)
\nI think it is because the final model is just a linear model as shown by AmbrosM final solution here. I guess using more NN with itself does not improve a straight line any more than one NN.
", "rawMarkdown": ">I personally am still puzzled why NN bagging didn't improve CV, I don't think I've seen such behaviour ever before. So instead of e.g. averaging 5-7 NN runs with different seeds, we used a single NN for the final submission, which was a bit shaky (+- 0.1), but hopefully compensated by LGB in the blend.\n\nYes, i was surprised by this too. In every previous competition using more copies of NN ensemble with itself boost LB but not here. In this comp, i tried 1 fold, 5 folds, and 11 folds and they all performed the same both on CV and LB. (On CV i simulated bagging within each fold iteration during KFold)\n\nI think it is because the final model is just a linear model as shown by AmbrosM final solution [here][1]. I guess using more NN with itself does not improve a straight line any more than one NN. \n\n[1]: https://www.kaggle.com/competitions/amp-parkinsons-disease-progression-prediction/discussion/411388", "votes": 1 } ] } ] }, { "id": 2266022, "postDate": "2023-05-19T16:41:05.543Z", "content": "Congratulations @dott1718 and @kyakovlev Fantastic job!
\n\n\nThe winning solution fully ignores the results of the blood tests. We’ve tried hard to find any signal in this crucial piece of the data, but unfortunately we came to the conclusion that none of our approaches or models can benefit from blood test features significant enough to distinguish it from random variations. The final models were trained only on the union of clinical and supplementary datasets.
\n
I just had a thought. Now that the competition ended and the signal was fully explained by AmbrosM here, it might be the case that there is signal in the protein peptide information. Perhaps the problem is that once our models detect the 3 groups of patients using visit_month then protein peptide does not provide more information. However if the host could hide the 3 groups of patients (i.e. remove the signal from visit month), I wonder whether the protein peptide information can help locate the 3 groups?
In other words, i do not think that this competition demonstrated that the protein peptide does not contain signal (by the fact that top teams did not use it.). I think this competition demonstrated that the signal from visit_month contains whatever signal may exist in the protein peptide information. Do you think this might be the case? If so, we should let the host and biology community know this.
Now i just read that you did try using blood without visit month. So perhaps you demonstrated that it does not have predictive power.
\n\n", "rawMarkdown": "Now i just read that you did try using blood without visit month. So perhaps you demonstrated that it does not have predictive power.\n\n>Blood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities.", "votes": 1, "replies": [ { "id": 2269564, "postDate": "2023-05-22T14:44:49.517Z", "content": "Blood test data. We’ve tried many approaches to add proteins and/or peptides data to our models, but none of them improved CV. We narrowed it down to a bag of logistic regressions that forecasted the visit on the 6th month based on the blood test result on the 0th month. We applied soft up/down scaling of model-based predictions for patients that were more/less likely to visit on the 6th month based on the logistic regression probabilities.
\n
Yes, it is exactly why we tried to narrow it down to a single homogenous group, but without much success. At some point I managed to get a CV improvement with some proteins data of a 0.01 magnitude which was much smaller than expected random effect on LB of around 0.1. So a different competition setup with more data might have been successful in identifying proteins/peptides.
\nThe way I managed to \"force\" proteins into the LGB model was to select the top 5-10 ones with the highest absolute correlation with the targets and scale down their gain with feature_contri parameter dramatically.
Why is leave-one-(patient)-out cross-validation also called \"stratified cross validation with a fold for each patient\" in the post?
\nI thought that for stratification we need our folds to be stratified for target, for example )
You are right, thank you for pointing that out! It is indeed group k-fold and not stratified k-fold, I've corrected the text to prevent potential confusion.
", "rawMarkdown": "You are right, thank you for pointing that out! It is indeed group k-fold and not stratified k-fold, I've corrected the text to prevent potential confusion.", "votes": 1 } ] }, { "id": 2266848, "postDate": "2023-05-20T12:16:29.457Z", "content": "Congratulations!!!
\nThanks for sharing your solution!!
Congratulations🎉
\nI really learned a lot from your source code and explanations.
\nThank you very much.
Congratulations Dmitry and Konstantin! What a brilliant skill to focus on one smaller part of data (visits) and have enough wisdom and experience to ignore the others (CSF). I would say it is one of my main outcomes from this competition.
\nI hope that organizers will not be dissatisfied since that all the data they provided were not used in a winning solutions and the hypothesis about predictive power of proteins and peptides is not confirmed yet. Anyway this competition makes all of us a bit closer to the understanding of the Parkinson's desease.
\nCongratulations to everyone who won in leaderbord and congratulations to all those who have beaten themselves in this competition. It was an addictive trip and interesting experience.
", "rawMarkdown": "Congratulations Dmitry and Konstantin! What a brilliant skill to focus on one smaller part of data (visits) and have enough wisdom and experience to ignore the others (CSF). I would say it is one of my main outcomes from this competition.\n\nI hope that organizers will not be dissatisfied since that all the data they provided were not used in a winning solutions and the hypothesis about predictive power of proteins and peptides is not confirmed yet. Anyway this competition makes all of us a bit closer to the understanding of the Parkinson's desease.\n\nCongratulations to everyone who won in leaderbord and congratulations to all those who have beaten themselves in this competition. It was an addictive trip and interesting experience. ", "votes": 4 }, { "id": 2272120, "postDate": "2023-05-24T10:06:13.043Z", "content": "Congratulations! sir
", "rawMarkdown": "Congratulations! sir\n", "votes": 1 }, { "id": 2268528, "postDate": "2023-05-21T19:31:50.733Z", "content": "Many congratulations @dott1718. Also I highly appreciate you for sharing your winning solution.
\nStay blessed!
Congratulations! Thanks for sharing your approach to a simple but effective solution.
", "rawMarkdown": "Congratulations! Thanks for sharing your approach to a simple but effective solution.", "votes": 1 }, { "id": 2267910, "postDate": "2023-05-21T10:30:51.380Z", "content": "https://www.kaggle.com/code/kyakovlev/amp-blood-randomness/
\nSmall example for \"fake\" correlations in proteins / peptides
", "rawMarkdown": "https://www.kaggle.com/code/kyakovlev/amp-blood-randomness/\n\nSmall example for \"fake\" correlations in proteins / peptides", "votes": 1 }, { "id": 2270653, "postDate": "2023-05-23T09:53:36.787Z", "content": "Heartiest Congratulations @dott1718. And thanks for sharing. Keep up the good work 😊
", "rawMarkdown": "Heartiest Congratulations @dott1718. And thanks for sharing. Keep up the good work 😊", "votes": 2 }, { "id": 2268112, "postDate": "2023-05-21T13:18:03.707Z", "content": "Congratulations @dott1718 and @kyakovlev ! Best team name and best solution 💪
\nI tried multiclass classification as well but couldn't make it work better than a simple regression model. Thanks for sharing your method, it's instructive !
", "rawMarkdown": "Congratulations @dott1718 and @kyakovlev ! Best team name and best solution 💪\n\nI tried multiclass classification as well but couldn't make it work better than a simple regression model. Thanks for sharing your method, it's instructive !", "votes": 2, "replies": [ { "id": 2269513, "postDate": "2023-05-22T14:13:22.817Z", "content": "Thank you Vincent! I was cheering for you to get to the gold zone on private LB, hopefully you are not too upset about the result.
", "rawMarkdown": "Thank you Vincent! I was cheering for you to get to the gold zone on private LB, hopefully you are not too upset about the result.", "votes": 1, "replies": [ { "id": 2269639, "postDate": "2023-05-22T15:37:29.450Z", "content": "Thanks 😄 Nothing too frustrating, hopefully I'll get it another time !
", "rawMarkdown": "Thanks 😄 Nothing too frustrating, hopefully I'll get it another time !" }, { "id": 2269758, "postDate": "2023-05-22T17:24:39.700Z", "content": "I am sure you will!
", "rawMarkdown": "I am sure you will!", "votes": 1 } ] } ] }, { "id": 2267071, "postDate": "2023-05-20T15:02:13.793Z", "content": "It is unfortunate that no signal was found in the protein data, as that was the entire point of the challenge. Parkinson's remains a clinical diagnosis it seems.
", "rawMarkdown": "It is unfortunate that no signal was found in the protein data, as that was the entire point of the challenge. Parkinson's remains a clinical diagnosis it seems.", "votes": 2 }, { "id": 2266900, "postDate": "2023-05-20T13:01:59.950Z", "content": "Congratulations!
\nThanks for sharing your solution with us. It has been very helpful for me.
\nThank you!!
very interesting discussion and work, thank you for taking the time to update on this, its interesting that the blood test data was not directly an input
", "rawMarkdown": "very interesting discussion and work, thank you for taking the time to update on this, its interesting that the blood test data was not directly an input", "votes": 2 }, { "id": 2266752, "postDate": "2023-05-20T10:52:07.053Z", "content": "Congratulations! Thanks for sharing your solution gives a lot insights to me to see how does top people in leaderboard approach a project.
", "rawMarkdown": "Congratulations! Thanks for sharing your solution gives a lot insights to me to see how does top people in leaderboard approach a project.\n", "votes": 2 }, { "id": 2266647, "postDate": "2023-05-20T08:39:37.713Z", "content": "Congratulations! Thanks for sharing your approach to a simple but effective solution.
", "rawMarkdown": "Congratulations! Thanks for sharing your approach to a simple but effective solution.", "votes": 2 }, { "id": 2266028, "postDate": "2023-05-19T16:49:43.167Z", "content": "Thanks for sharing your solution. I never achieved good results from GBT. (I think CV was 57 compared with my MLP's 55). And i did not achieve any significant boosts from SMAPE optimization (besides rounding). I also did not find a better loss for my MLP than MeanAbsoluteError loss.
\nNow after reading your solution, i may go back and revisit these ideas and see if i can boost my MLP by ensembling it with GBT and adding some SMAPE optimization.
\nMy two top models were MLP (4th place) and SVR (9th place) but the CV did not boost when i ensembled the two.
", "rawMarkdown": "Thanks for sharing your solution. I never achieved good results from GBT. (I think CV was 57 compared with my MLP's 55). And i did not achieve any significant boosts from SMAPE optimization (besides rounding). I also did not find a better loss for my MLP than MeanAbsoluteError loss.\n\nNow after reading your solution, i may go back and revisit these ideas and see if i can boost my MLP by ensembling it with GBT and adding some SMAPE optimization.\n\nMy two top models were MLP (4th place) and SVR (9th place) but the CV did not boost when i ensembled the two.", "votes": 2, "replies": [ { "id": 2266044, "postDate": "2023-05-19T17:02:46.523Z", "content": "Same with our ensembles - NN with lgbm showed just a tiiiny boost on CV.
", "rawMarkdown": "Same with our ensembles - NN with lgbm showed just a tiiiny boost on CV.", "votes": 1 }, { "id": 2266067, "postDate": "2023-05-19T17:17:00.267Z", "content": "I like your trick of training multi-class then optimizing output to SMAPE. That's clever.
\n\n", "rawMarkdown": "I like your trick of training multi-class then optimizing output to SMAPE. That's clever. \n\n>Our LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.\n\n[1]: https://www.kaggle.com/c/ventilator-pressure-prediction/discussion/278362", "votes": 2 } ] }, { "id": 2410427, "postDate": "2023-08-26T23:28:52.627Z", "content": "Our LGB model is a classification model with 87 target classes (0 to maximum target value) and logloss objective. To produce the forecast we applied the following post-processing: given the predicted distribution of target classes, pick a value that minimizes SMAPE+1. Taking into account the observation that the optimal predictions are always integers, the task boils down to a trivial search among 87 possible values. Such an approach would have worked well for the original SMAPE metric also, because the approach treats cases with multiple local minimums naturally.
\n
Hi @dott1718, congratulations! Would you be able to send along the testing data provided by the competition? I have a student working with this data for his capstone project and would love to get this data. We contacted the competition organizers but haven't heard back.
", "rawMarkdown": "Hi @dott1718, congratulations! Would you be able to send along the testing data provided by the competition? I have a student working with this data for his capstone project and would love to get this data. We contacted the competition organizers but haven't heard back." }, { "id": 2328592, "postDate": "2023-07-03T17:45:31.560Z", "content": "Congratulations! Thanks for sharing your approach.
", "rawMarkdown": "Congratulations! Thanks for sharing your approach.\n\n\n" }, { "id": 2287825, "postDate": "2023-06-05T00:14:32.767Z", "content": "Congratulations!
\nThank you for publishing and sharing.
\nThe discussion around the metrics in this competition was difficult and confusing to me. I will learn from the published notes.
Good job,thank you very much. Because I can learn from it .
", "rawMarkdown": "Good job,thank you very much. Because I can learn from it ." }, { "id": 2276543, "postDate": "2023-05-27T00:54:56.890Z", "content": "I thought XGBoost almost always had good performance, but I guess it wasn't necessarily. Your writing and notebook helped me improve my skills a lot. Thank you!!
", "rawMarkdown": "I thought XGBoost almost always had good performance, but I guess it wasn't necessarily. Your writing and notebook helped me improve my skills a lot. Thank you!!", "replies": [ { "id": 2292313, "postDate": "2023-06-08T07:59:20.263Z", "content": "As I understood the discussion: XGB would be a comparable method if it would have custom objective function aligned with SMAPE metric. Right @cdeotte, @dott1718, @kyakovlev ?
\nAnyway excellent learning opportunity! Thanks guys!
Congratulations !! Well done 🔥
", "rawMarkdown": "Congratulations !! Well done 🔥" }, { "id": 2272174, "postDate": "2023-05-24T10:37:07.383Z", "content": "Congratulations! Thanks for sharing your approach.
", "rawMarkdown": "Congratulations! Thanks for sharing your approach." }, { "id": 2271694, "postDate": "2023-05-24T05:29:40.277Z", "content": "Many congratulations @dott1718 on winning this competition. Reading through your solution is a learning experience for me, very insightful. Thank you so much for sharing.
", "rawMarkdown": "Many congratulations @dott1718 on winning this competition. Reading through your solution is a learning experience for me, very insightful. Thank you so much for sharing." }, { "id": 2267461, "postDate": "2023-05-21T00:35:05.700Z", "content": "The objective of this competition is to discern what kind of correlation the protein / peptides (levels) have on disease progression. Hence, I never tried any model (though I joined the competition in the last week) without these data till near last few days when I was shocked to find a simple linear regression model with just visit month column alone as training data could beat all my models and achieve a score of 55!!
\nThe conclusion of the prize winners that the protein/peptides (levels) do not have any correlation is not correct. That visit month alone can accurately predict the disease progression does not imply that the protein/peptides have no correlation.
\nI tried an unusual deep learning model that considers only the peptide information. I created an embedding layer based on the peptide information by coding each amino acid label of the peptide by creating all peptides present during the visit as a list against each visit. The embedding layer weights of the amino acid tokens were trained against the updrs scores. I got near perfect predictions for the training data, though the model did not converge for the validation data. I think deep learning models can learn any kind of functions but might fail in predicting test data which might also imply that there is no correlation. However, on further analysis, I concluded that this could also be because the training data is not sufficient. Each visit and updrs scores are trained against the presence of peptides. The list of peptides in terms of amino acids is huge for each updrs score. Given this, the training data may not be even 10% of what is required for such predictions. Hence, it is quite possible that the presence or absence of peptides have correlation with disease progress but the dataset is not sufficient enough to predict them.
\nThe dataset focusses on the levels of NPX and peptides rather than the number of patients. The training dataset looks sufficient if you go by the levels of peptides and proteins. But if you go by the number of patients with respect to presence or absence of peptide amino acids from the overall universe of possible peptides, we can say for sure that the training dataset is not at all sufficient to predict any relationship between presence or absence of protein/peptides. The dataset size should be measured in terms of number of patients in this case. This is my final conclusion.
\nAnyway, congrats to the winners for their hard work!
", "rawMarkdown": "The objective of this competition is to discern what kind of correlation the protein / peptides (levels) have on disease progression. Hence, I never tried any model (though I joined the competition in the last week) without these data till near last few days when I was shocked to find a simple linear regression model with just visit month column alone as training data could beat all my models and achieve a score of 55!!\n\nThe conclusion of the prize winners that the protein/peptides (levels) do not have any correlation is not correct. That visit month alone can accurately predict the disease progression does not imply that the protein/peptides have no correlation.\n\nI tried an unusual deep learning model that considers only the peptide information. I created an embedding layer based on the peptide information by coding each amino acid label of the peptide by creating all peptides present during the visit as a list against each visit. The embedding layer weights of the amino acid tokens were trained against the updrs scores. I got near perfect predictions for the training data, though the model did not converge for the validation data. I think deep learning models can learn any kind of functions but might fail in predicting test data which might also imply that there is no correlation. However, on further analysis, I concluded that this could also be because the training data is not sufficient. Each visit and updrs scores are trained against the presence of peptides. The list of peptides in terms of amino acids is huge for each updrs score. Given this, the training data may not be even 10% of what is required for such predictions. Hence, it is quite possible that the presence or absence of peptides have correlation with disease progress but the dataset is not sufficient enough to predict them. \n\nThe dataset focusses on the levels of NPX and peptides rather than the number of patients. The training dataset looks sufficient if you go by the levels of peptides and proteins. But if you go by the number of patients with respect to presence or absence of peptide amino acids from the overall universe of possible peptides, we can say for sure that the training dataset is not at all sufficient to predict any relationship between presence or absence of protein/peptides. The dataset size should be measured in terms of number of patients in this case. This is my final conclusion.\n\nAnyway, congrats to the winners for their hard work!" }, { "id": 2267424, "postDate": "2023-05-20T21:55:33.930Z", "rawMarkdown": "", "votes": 1, "isDeleted": true, "replies": [ { "id": 2267761, "postDate": "2023-05-21T08:01:50.460Z", "content": "I really like your \"confidence\" in the approach that you've created, but you are wrong.
\nThere is an error in CV or Metric or any other place.
\nIt is easy to check if you really want to dig deeper.
I'm really sorry to say but there is no significant signal in blood samples.
\nOur standard dialog with Dmitry every week:
Some times due to my naiveness and fast prototyping errors I screamed in chat \"I FOUND!!!!!!\" and in a 15 mins Dmitry or even me found and error or leak in pipeline - that was such a joy to see how Dmitry is criticising my \"ideas\" - I'll miss our conversations and discussions.
", "rawMarkdown": "I really like your \"confidence\" in the approach that you've created, but you are wrong.\n\nThere is an error in CV or Metric or any other place. \nIt is easy to check if you really want to dig deeper.\n- You are saying that low performance is \"were missing peptides and protein information\" - you can combine any top20 solution with yours and just replace rows with blood tests with predictions from your NN - if there is a signal - you would easily outperfrom our 1st place solution by %%s.\n\nI'm really sorry to say but there is no significant signal in blood samples. \nOur standard dialog with Dmitry every week:\n- It's not possible that there is no signal in blood.\n- What if we try that (any the most weirdest idea here).\n.. few days afer\n- No there is no signal for sure.\n.. few days after - repeat.\n\nSome times due to my naiveness and fast prototyping errors I screamed in chat \"I FOUND!!!!!!\" and in a 15 mins Dmitry or even me found and error or leak in pipeline - that was such a joy to see how Dmitry is criticising my \"ideas\" - I'll miss our conversations and discussions. \n\n ", "votes": 1, "replies": [ { "id": 2268733, "postDate": "2023-05-22T00:49:33.947Z", "content": "As per a research paper I sampled, there are around 13879 peptides which differ from normal healthy people compared to parkinson' diseased patients. Of these peptides, 1466 are observable from the CSF sample taken from brain fluids. Among these 192 peptides are relevant for parkinson's diease. Of these around 20 can be used as markers for Parkinson disease and its progression. (https://www.sciencedirect.com/science/article/pii/S1535947620332217#:~:text=Furthermore%2C%20a%20combination%20of%20two%20peptides%20belonging%20to,validation%20%28r%20%3D%200.339%2C%20p%20%3D%200.032%29%20sets.)
\nHence, that there is signal in peptide information with regard to parkinson disease is scientifically proved and is an important research area in parkinson diesease and hence this competition.
\nIn our training dataset, for each visit of a patient, peptide levels are given. There are around 900 peptides for which measurements were taken during each visit. This actually varies between 600 to 980 across the patients. It is not clear whether for patients with just 600 peptide information, the other 300 peptides are not present in the CSF sample etc.
\nMy very preliminary attempt was to correlate this list of peptides for each patient visit with the UPDRS scores (idea is to consider peptide levels and protein relationships later - subsequent models should incorporate the peptides abundance level and protein relationships that can influence the embedding weights)). To put it more naively and in raw terms, from the perspective of machine learning model, does presence of 900 peptides vs 600 peptides has any impact on the disease condition?
\nSo, I embedded the peptide information using keras embedding layer after tokenizing them based on overall peptide vocabulary. The weights are learnt against the UPDRS scores. This training gave near perfect prediction for training data. But the results against the test data were not good enough to try in the leaderboard(!).
\nMy contention is that the performance in the test data could be poor because of the size of the training dataset size. For example, for each row, there are 900 peptides which are correlated with the scores (If you embed these peptides at the character level which is more correct way, it would be far more). Assuming that the above quoted research is valid, out of these 900 peptides, only around 20 have impact on the disease progression. Now, deep learning model has to study the relationships between the peptides and the scores based on around 300 patients (900 features vs 300 rows). Theoretically, if the model has to learn the relationships, it would definitely require lot more data. This is my conclusion.
\nI am not saying that because my embedding based deep learning model can predict the training data perfectly, that there definitely exists a signal identified by the model. Deep learning models can learn any function and predict perfectly - but it could just be nothing more than overfitting. But in all these cases, only by increasing the dataset size, can we say for sure about existence of any relationships.
\nHere is the code of my naive and very preliminary model (I am not an expert in python - So, I asked chatgpt and created this entire model in just 30 minutes!! :-) :-)):
\ndf=train_peptides
\ngrouped_df = df.groupby('visit_id').agg({'visit_month': 'first', 'patient_id': 'first', 'Peptide': list, 'PeptideAbundance': 'mean'})
\ngrouped_df.reset_index(inplace=True)
\nmerged_df = pd.merge(train_clinical, grouped_df, on=['visit_id', 'patient_id'], how='left')
\nprint(merged_df)
\nmerged_df=merged_df[[\"visit_id\",\"visit_month_x\",\"updrs_1\",\"updrs_2\",\"updrs_3\",\"updrs_4\",\"Peptide\",\"PeptideAbundance\"]]
\nmerged_df.rename(columns={\"visit_month_x\":\"visit_month\"},inplace=True)
df = merged_df.copy(deep=True)
\ndf[\"Peptide\"].replace(\"nan\", np.nan, inplace=True)
\ndf.dropna(inplace=True)
\ndf['Peptide'] = df['Peptide'].astype(str)
\ntokenizer = Tokenizer()
\ntokenizer.fit_on_texts(df['Peptide'])
\nsequences= tokenizer.texts_to_sequences(df['Peptide'])
--- wt=df[\"PeptideAbundance\"] / df[\"PeptideAbundance\"].sum()
\n--- sequences = [x * y if isinstance(x, int) or isinstance(x, float) else [xi * y for xi in x] for x, y in zip(sequences, wt)]
\n--- print(result)
\nmax_length = max([len(seq) for seq in sequences])
\npadded_sequences = pad_sequences(sequences, maxlen=max_length, padding='post')
\nprint(\"padded seq shape\",padded_sequences.shape)
\ndf_vm=df[\"visit_month\"] #*10000
\nsequence_length = max_length # Length of padded sequence
\nvocab_size = len(tokenizer.word_index) + 1 # Vocabulary size
\nembedding_dim = 50 # Dimension of embedding space
\nsequence_input = Input(shape=(sequence_length,), dtype='int32', name='sequence_input')
\ndf_input = Input(shape=(1,), dtype='float32', name='df_input')
\nembedding_layer = Embedding(vocab_size, embedding_dim)(sequence_input)
\nflattened_sequence = tf.keras.layers.Flatten()(embedding_layer)
\nmerged_features = Concatenate(axis=1)([flattened_sequence, df_input])
\ntf.random.set_seed(1123)
\nactivation='relu'
\nx=Dense(64,activation='tanh',kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(merged_features)
\n kernel_regularizer=tf.keras.regularizers.L2(0.01))(merged_features) #9flattened_sequence)
\nx=Dense(32,activation=activation,kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\nx=Dense(10,activation=activation,kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\noutput=Dense(1,activation='relu',kernel_initializer=tf.keras.initializers.HeNormal(seed=1234))(x)
\ntarget = ['updrs_3', 'updrs_2', 'updrs_1'] #, 'updrs_4']
\nratings=df['updrs_1']
\n--# ratings = df[['updrs_1', 'updrs_2', 'updrs_3']]
\nseq_train, seq_test, dfvm_train,dfvm_test,ratings_train, ratings_test = train_test_split(
\n padded_sequences, df_vm, ratings, test_size=0.2, random_state=42)
\nmodel1= Model(inputs=[sequence_input,df_input], outputs=output)
\nlearning_rate = 0.01
\noptimizer = Adam(learning_rate=learning_rate)
\nmodel1.compile(optimizer=optimizer, loss='mse', metrics=['mae',smape_metric]) #SMAPE()
\nmodel1.fit([seq_train,dfvm_train], ratings_train, epochs=100, batch_size=32, validation_data=([seq_test,dfvm_test], ratings_test))
Thank you. It was helpful.
", "rawMarkdown": "Thank you. It was helpful." } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T14:11:17Z", "competition_id": 37794, "competition_slug": "amp-parkinsons-disease-progression-prediction", "team_id": 10075968, "solution_url": "/competitions/amp-parkinsons-disease-progression-prediction/writeups/connecting-dotts-1st-place-solution" }