I started with @taylorsamarel's baseline notebook (i dont know why it is now deleted) but I utilised some of the preprocessing steps it used and made my own version with minimal preprocessing and shared it publicly here.
\nFrom the original work I noticed they were doing extensive preprocessing such as MSC, savgol filter, normalisation. With help of gemini I included other preprocessing steps too and experimented with various combinations of these preprocessing steps and their parameter, on each device's spectrometer data by a simple grid search.
\nThis sequence yielded the best results for all spectrometers -\nOriginal Spectra -> MSC -> Baseline correction -> Smoothing -> Scaling\nR2 scores for other spectrometers
\nI had another interesting find but I did not use that in my final solution. The glucose readings from the test spectrometer was linearly related to the glucose readings from other spectrometers. These experiments are included in this notebook v3 for reference.
\nI finalized my approach with these preprocessing steps.
\nThe final solution was weighted average of my best solution which was an autogluon ensemble here (weight = 0.6) and best public solution here (weight = 0.4)
\nAutogluon models were trained for 1hr/target = 3hrs with preset='best_quality'
\nBut,\nA simple hgb model with the preprocessing steps outperforms the second best solution and I have shared that too here\nThis notebook only takes 45 seconds to train and generate submission, and can be used for efficiency.
\nI tried other methods from the original work like PLS but they did not yield any better results than my setup.
\nI hope this clarifies everything @rosahassfurther
\nP.S. I wrote this in a bit of a rush, so couldn't clean up my code and apologies for any errors.
", "comments": [ { "id": 3306355, "author_name": "Rosa Hassfurther", "author_url": "", "post_date": "2025-10-24T11:37:14.783000", "content": "Thank you for claryfing:) I like your approach!
", "votes": 0, "replies": [ { "id": 3309695, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2025-11-01T07:25:33.167000", "content": "Hi Rosa! Could you please update me on the payment of prize money? I haven't received any further communication via mail.
", "votes": 0, "replies": [ { "id": 3311679, "author_name": "Rosa Hassfurther", "author_url": "", "post_date": "2025-11-05T14:59:34.163000", "content": "Hi, I am trying to reach the contact person from our finance department, to facilitate the payment. \nDue to low personelle in university administration all payment processes are very slow at the moment. I will update you as soon as I got news!
", "votes": 0, "replies": [] }, { "id": 3311702, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2025-11-05T15:49:18.003000", "content": "Okay, Thanks!
", "votes": 0, "replies": [] }, { "id": 3314785, "author_name": "Rosa Hassfurther", "author_url": "", "post_date": "2025-11-10T10:57:32.413000", "content": "Hi, I got contacted by the responsible person from our finance department. To further proceed, could you provide me with your banking details via this secure tool or any other tool you trust:\nhttps://www.read2burn.com/ \nand send the link to my mail: r.hassfurther@tu-berlin.de\nSince this is an extra payment, an additional approval from our department is necessary, which could take a bit. I will make sure, it is done as quick as possible. \nThank you for your patience!
", "votes": 0, "replies": [] }, { "id": 3315546, "author_name": "Paritosh Kumar Tripathi", "author_url": "", "post_date": "2025-11-10T16:52:14.733000", "content": "Hi Rosa,\nThanks for the update. I’ve just shared my banking details with you directly via email and did not use any third-party service (such as Read2Burn or others).
\nPlease confirm receipt and let me know if you need anything else for the additional approval.
", "votes": 0, "replies": [] } ] } ] } ], "comment_count": 6, "messages": [ { "id": 3305729, "postDate": "2025-10-23T10:26:43.827Z", "content": "I started with @taylorsamarel's baseline notebook (i dont know why it is now deleted) but I utilised some of the preprocessing steps it used and made my own version with minimal preprocessing and shared it publicly here.
\nFrom the original work I noticed they were doing extensive preprocessing such as MSC, savgol filter, normalisation. With help of gemini I included other preprocessing steps too and experimented with various combinations of these preprocessing steps and their parameter, on each device's spectrometer data by a simple grid search.
\nThis sequence yielded the best results for all spectrometers -\nOriginal Spectra -> MSC -> Baseline correction -> Smoothing -> Scaling\nR2 scores for other spectrometers
\nI had another interesting find but I did not use that in my final solution. The glucose readings from the test spectrometer was linearly related to the glucose readings from other spectrometers. These experiments are included in this notebook v3 for reference.
\nI finalized my approach with these preprocessing steps.
\nThe final solution was weighted average of my best solution which was an autogluon ensemble here (weight = 0.6) and best public solution here (weight = 0.4)
\nAutogluon models were trained for 1hr/target = 3hrs with preset='best_quality'
\nBut,\nA simple hgb model with the preprocessing steps outperforms the second best solution and I have shared that too here\nThis notebook only takes 45 seconds to train and generate submission, and can be used for efficiency.
\nI tried other methods from the original work like PLS but they did not yield any better results than my setup.
\nI hope this clarifies everything @rosahassfurther
\nP.S. I wrote this in a bit of a rush, so couldn't clean up my code and apologies for any errors.
", "rawMarkdown": "I started with @taylorsamarel's baseline notebook (i dont know why it is now deleted) but I utilised some of the preprocessing steps it used and made my own version with minimal preprocessing and shared it publicly [here](https://www.kaggle.com/code/paritoshtripathi5/public-load-train-val-and-test-data).\n\nFrom the [original work](https://git.tu-berlin.de/bvt-htbd/kiwi/tf3/raman-and-nir/-/tree/main?ref_type=heads) I noticed they were doing extensive preprocessing such as MSC, savgol filter, normalisation. With help of gemini I included other preprocessing steps too and experimented with various combinations of these preprocessing steps and their parameter, on each device's spectrometer data by a simple grid search.\n\nThis sequence yielded the best results for all spectrometers -\nOriginal Spectra -> MSC -> Baseline correction -> Smoothing -> Scaling\nR2 scores for other spectrometers\n- Device0 Score: 0.62\n- Device1 Score: 0.83\n- Device2 Score: 0.92\n- Device3 Score: 0.92\n- Device4 Score: 0.95\n- Device5 Score: 0.96\n- Device6 Score: 0.84\n- Device7 Score: 0.91\n\nI had another interesting find but I did not use that in my final solution. The glucose readings from the test spectrometer was linearly related to the glucose readings from other spectrometers. These experiments are included in [this notebook v3](https://www.kaggle.com/code/paritoshtripathi5/map-idea) for reference.\n\nI finalized my approach with these preprocessing steps.\n\nThe final solution was weighted average of my best solution which was an autogluon ensemble [here](https://www.kaggle.com/code/paritoshtripathi5/fork-of-preprocess-autogluon) (weight = 0.6) and best public solution [here](https://www.kaggle.com/code/nguyenlamphuquy/pca-lr-extraregressiontree) (weight = 0.4)\n\nAutogluon models were trained for 1hr/target = 3hrs with preset='best_quality'\n\nBut,\nA simple hgb model with the preprocessing steps outperforms the second best solution and I have shared that too [here](https://www.kaggle.com/code/paritoshtripathi5/final-solution-notebook/)\nThis notebook only takes 45 seconds to train and generate submission, and can be used for efficiency.\n\nI tried other methods from the original work like PLS but they did not yield any better results than my setup.\n\nI hope this clarifies everything @rosahassfurther \n\nP.S. I wrote this in a bit of a rush, so couldn't clean up my code and apologies for any errors.", "votes": 4 }, { "id": 3306355, "postDate": "2025-10-24T11:37:14.783Z", "content": "Thank you for claryfing:) I like your approach!
", "rawMarkdown": "Thank you for claryfing:) I like your approach!", "replies": [ { "id": 3309695, "postDate": "2025-11-01T07:25:33.167Z", "content": "Hi Rosa! Could you please update me on the payment of prize money? I haven't received any further communication via mail.
", "rawMarkdown": "Hi Rosa! Could you please update me on the payment of prize money? I haven't received any further communication via mail.", "replies": [ { "id": 3311679, "postDate": "2025-11-05T14:59:34.163Z", "content": "Hi, I am trying to reach the contact person from our finance department, to facilitate the payment. \nDue to low personelle in university administration all payment processes are very slow at the moment. I will update you as soon as I got news!
", "rawMarkdown": "Hi, I am trying to reach the contact person from our finance department, to facilitate the payment. \nDue to low personelle in university administration all payment processes are very slow at the moment. I will update you as soon as I got news!" }, { "id": 3311702, "postDate": "2025-11-05T15:49:18.003Z", "content": "Okay, Thanks!
", "rawMarkdown": "Okay, Thanks!" }, { "id": 3314785, "postDate": "2025-11-10T10:57:32.413Z", "content": "Hi, I got contacted by the responsible person from our finance department. To further proceed, could you provide me with your banking details via this secure tool or any other tool you trust:\nhttps://www.read2burn.com/ \nand send the link to my mail: r.hassfurther@tu-berlin.de\nSince this is an extra payment, an additional approval from our department is necessary, which could take a bit. I will make sure, it is done as quick as possible. \nThank you for your patience!
", "rawMarkdown": "Hi, I got contacted by the responsible person from our finance department. To further proceed, could you provide me with your banking details via this secure tool or any other tool you trust:\nhttps://www.read2burn.com/ \nand send the link to my mail: r.hassfurther@tu-berlin.de\nSince this is an extra payment, an additional approval from our department is necessary, which could take a bit. I will make sure, it is done as quick as possible. \nThank you for your patience!" }, { "id": 3315546, "postDate": "2025-11-10T16:52:14.733Z", "content": "Hi Rosa,\nThanks for the update. I’ve just shared my banking details with you directly via email and did not use any third-party service (such as Read2Burn or others).
\nPlease confirm receipt and let me know if you need anything else for the additional approval.
", "rawMarkdown": "Hi Rosa,\nThanks for the update. I’ve just shared my banking details with you directly via email and did not use any third-party service (such as Read2Burn or others).\n\nPlease confirm receipt and let me know if you need anything else for the additional approval." } ] } ] } ], "complete": true, "source": "live", "fetched_at": "2026-10-19T13:37:42Z", "competition_id": 105802, "competition_slug": "dig-4-bio-raman-transfer-learning-challenge", "team_id": 14086972, "solution_url": "/competitions/dig-4-bio-raman-transfer-learning-challenge/writeups/1st-place-solution-preprocessing-is-all-you-need" }