![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2594941%2Faa672ce0b1b7722c24c0b25651e95a1c%2Fdownload.png?generation=1579172598247842&alt=media\")
\nI have seen some kernels in this competition and I think everyone replaces very fast the missing values for example:
\n\n\nfor col in test_df:\n if test_df[col].isna().sum() > 0:\n test_df[col] = test_df[col].fillna(test_df[col].mode()[0])\n
I think the right replacement of the missing values is the key to the best score. On the first view, the columns have very random values, but somehow they are related.
\n\nThe columns: \nnom_4 \narray(['Bassoon', 'Theremin', 'Oboe', 'Piano'], dtype=object)
\n\nand
\n\nord_1\narray(['Contributor', 'Grandmaster', 'Novice', 'Expert', 'Master'], dtype=object)
\n\nare related. ord_1 is the grade of playing these instruments.
\n\nIf we found some 3rd column (maybe in the binary columns), then we can replace in combination with the missing values.
\n\nThe countries are somehow related to ord_2\narray(['Hot', 'Warm', 'Freezing', 'Lava Hot', 'Cold', 'Boiling Hot'], dtype=object)
\n\nIf ord_2 is possibly the wheater, then it could be related to the months.
\n\nThe colors could be related also with ord_2, because the temperatures of the wheater are presented with these colors.
\n\nWhat do you think?
", "messages": [ { "id": "709038", "postDate": "01/03/2020 00:56:34", "content": "I have seen some kernels in this competition and I think everyone replaces very fast the missing values for example:
\n\n\nfor col in test_df:\n if test_df[col].isna().sum() > 0:\n test_df[col] = test_df[col].fillna(test_df[col].mode()[0])\n
I think the right replacement of the missing values is the key to the best score. On the first view, the columns have very random values, but somehow they are related.
\n\nThe columns: \nnom_4 \narray(['Bassoon', 'Theremin', 'Oboe', 'Piano'], dtype=object)
\n\nand
\n\nord_1\narray(['Contributor', 'Grandmaster', 'Novice', 'Expert', 'Master'], dtype=object)
\n\nare related. ord_1 is the grade of playing these instruments.
\n\nIf we found some 3rd column (maybe in the binary columns), then we can replace in combination with the missing values.
\n\nThe countries are somehow related to ord_2\narray(['Hot', 'Warm', 'Freezing', 'Lava Hot', 'Cold', 'Boiling Hot'], dtype=object)
\n\nIf ord_2 is possibly the wheater, then it could be related to the months.
\n\nThe colors could be related also with ord_2, because the temperatures of the wheater are presented with these colors.
\n\nWhat do you think?
", "rawMarkdown": "I have seen some kernels in this competition and I think everyone replaces very fast the missing values for example:\n\n```\nfor col in test_df:\n if test_df[col].isna().sum() > 0:\n test_df[col] = test_df[col].fillna(test_df[col].mode()[0])\n```\n\nI think the right replacement of the missing values is the key to the best score. On the first view, the columns have very random values, but somehow they are related.\n\nThe columns: \nnom_4 \narray(['Bassoon', 'Theremin', 'Oboe', 'Piano'], dtype=object)\n\nand \n\nord_1\narray(['Contributor', 'Grandmaster', 'Novice', 'Expert', 'Master'], dtype=object)\n\nare related. ord_1 is the grade of playing these instruments.\n\nIf we found some 3rd column (maybe in the binary columns), then we can replace in combination with the missing values.\n\n---\n\nThe countries are somehow related to ord_2\narray(['Hot', 'Warm', 'Freezing', 'Lava Hot', 'Cold', 'Boiling Hot'], dtype=object)\n\nIf ord_2 is possibly the wheater, then it could be related to the months. \n\n---\n\nThe colors could be related also with ord_2, because the temperatures of the wheater are presented with these colors.\n\nWhat do you think?", "votes": null }, { "id": "709175", "postDate": "01/03/2020 05:53:15", "content": "Certainly interesting to see how missing data is treated. Do you have any analysis to show nom_4 and ord_1 are related (and other relations you mentioned) or are you just basing it on the value names?
", "rawMarkdown": "Certainly interesting to see how missing data is treated. Do you have any analysis to show nom_4 and ord_1 are related (and other relations you mentioned) or are you just basing it on the value names?", "votes": null }, { "id": "709181", "postDate": "01/03/2020 06:10:58", "content": "This might help: Just encode ordinal variables and binarize nominals. check for correlation. group by most 3-4 correlated feaures. fill with the median according to the groups.
", "rawMarkdown": "This might help: Just encode ordinal variables and binarize nominals. check for correlation. group by most 3-4 correlated feaures. fill with the median according to the groups.", "votes": null }, { "id": "709545", "postDate": "01/03/2020 16:32:35", "content": "Looking at Theil's U and doing some experiments with feature interactions between nom4 and ord1 suggests there might be something to that... but I haven't found anything definitive yet with my current treatment.
", "rawMarkdown": "Looking at Theil's U and doing some experiments with feature interactions between *nom4* and *ord1* suggests there might be something to that... but I haven't found anything definitive yet with my current treatment.", "votes": null }, { "id": "720304", "postDate": "01/16/2020 10:03:22", "content": "Replacing missing values with constant gives better results for me.\nI am using OHE + Target encoders and Logistic regression.
", "rawMarkdown": "Replacing missing values with constant gives better results for me.\nI am using OHE + Target encoders and Logistic regression.", "votes": null }, { "id": "720373", "postDate": "01/16/2020 11:08:22", "content": "About relation between month and weather:\n\n.png?generation=1579172730041840&alt=media)\n
.png?generation=1579172742780935&alt=media)\n
.png?generation=1579172827596653&alt=media)\n
.png?generation=1579172762762348&alt=media)\n
.png?generation=1579172782806899&alt=media)
So, they are not connected. Or maybe I'm doing something wrong.
", "rawMarkdown": "About relation between month and weather:\n\n.png?generation=1579172730041840&alt=media)\n.png?generation=1579172742780935&alt=media)\n.png?generation=1579172827596653&alt=media)\n.png?generation=1579172762762348&alt=media)\n.png?generation=1579172782806899&alt=media)\n\nSo, they are not connected. Or maybe I'm doing something wrong.", "votes": null }, { "id": "720381", "postDate": "01/16/2020 11:23:40", "content": "@vadbeg You need to check which country is above the equator and which below. Because the one has in Dezember winter and the other has summer.
", "rawMarkdown": "vadbeg You need to check which country is above the equator and which below. Because the one has in Dezember winter and the other has summer.", "votes": null }, { "id": "720389", "postDate": "01/16/2020 11:32:42", "content": "'Russia' 'Canada' 'Finland' 'Costa Rica' 'China' 'India'. Each of them is or on equator or above it (but maybe there is some point, I'll check it).
", "rawMarkdown": "'Russia' 'Canada' 'Finland' 'Costa Rica' 'China' 'India'. Each of them is or on equator or above it (but maybe there is some point, I'll check it).", "votes": null }, { "id": "720396", "postDate": "01/16/2020 11:40:06", "content": "@vadbeg you have right! Is there some function to iterate over all features and build groups to check which group of features are correlated to another one feature? For example something like: get feature 1 and feature2 and check if correlates to feature 5. Then get feature1+feature2+feature3 and checks if correlates to feature 5 and so on.\nI've read about the replacement method from many features to one, but first, we need to see which group of features can describe some other feature.\nThe method is Random Simple Imputation with many features
\n\nHere an example from Titanic. The age is replaced from random value of the fare:\n![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F2469731%2Fd6ddc5bc22fd38bc7beaed937a178bbe%2FBildschirmfoto%202020-01-16%20um%2013.45.50.png?generation=1579175207825060&alt=media\")
Thanks for your thoughts... really appreciate !
", "rawMarkdown": "Thanks for your thoughts... really appreciate !", "votes": null } ], "comments": [ { "id": 709175, "author_name": "advaithasabnis", "author_url": "", "post_date": "01/03/2020 05:53:15", "content": "Certainly interesting to see how missing data is treated. Do you have any analysis to show nom_4 and ord_1 are related (and other relations you mentioned) or are you just basing it on the value names?
", "votes": null, "replies": [ { "id": 709545, "author_name": "puremath86", "author_url": "", "post_date": "01/03/2020 16:32:35", "content": "Looking at Theil's U and doing some experiments with feature interactions between nom4 and ord1 suggests there might be something to that... but I haven't found anything definitive yet with my current treatment.
", "votes": null, "replies": [] } ] }, { "id": 709181, "author_name": "cemsarier", "author_url": "", "post_date": "01/03/2020 06:10:58", "content": "This might help: Just encode ordinal variables and binarize nominals. check for correlation. group by most 3-4 correlated feaures. fill with the median according to the groups.
", "votes": null, "replies": [] }, { "id": 720304, "author_name": "pavelvpster", "author_url": "", "post_date": "01/16/2020 10:03:22", "content": "Replacing missing values with constant gives better results for me.\nI am using OHE + Target encoders and Logistic regression.
", "votes": null, "replies": [] }, { "id": 720373, "author_name": "vadbeg", "author_url": "", "post_date": "01/16/2020 11:08:22", "content": "About relation between month and weather:\n\n.png?generation=1579172730041840&alt=media)\n
.png?generation=1579172742780935&alt=media)\n
.png?generation=1579172827596653&alt=media)\n
.png?generation=1579172762762348&alt=media)\n
.png?generation=1579172782806899&alt=media)
So, they are not connected. Or maybe I'm doing something wrong.
", "votes": null, "replies": [ { "id": 720381, "author_name": "nnikolay", "author_url": "", "post_date": "01/16/2020 11:23:40", "content": "@vadbeg You need to check which country is above the equator and which below. Because the one has in Dezember winter and the other has summer.
", "votes": null, "replies": [] }, { "id": 720389, "author_name": "vadbeg", "author_url": "", "post_date": "01/16/2020 11:32:42", "content": "'Russia' 'Canada' 'Finland' 'Costa Rica' 'China' 'India'. Each of them is or on equator or above it (but maybe there is some point, I'll check it).
", "votes": null, "replies": [] }, { "id": 720396, "author_name": "nnikolay", "author_url": "", "post_date": "01/16/2020 11:40:06", "content": "@vadbeg you have right! Is there some function to iterate over all features and build groups to check which group of features are correlated to another one feature? For example something like: get feature 1 and feature2 and check if correlates to feature 5. Then get feature1+feature2+feature3 and checks if correlates to feature 5 and so on.\nI've read about the replacement method from many features to one, but first, we need to see which group of features can describe some other feature.\nThe method is Random Simple Imputation with many features
\n\nHere an example from Titanic. The age is replaced from random value of the fare:\n
Thanks for your thoughts... really appreciate !
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "709038": "I have seen some kernels in this competition and I think everyone replaces very fast the missing values for example:\n\n```\nfor col in test_df:\n if test_df[col].isna().sum() > 0:\n test_df[col] = test_df[col].fillna(test_df[col].mode()[0])\n```\n\nI think the right replacement of the missing values is the key to the best score. On the first view, the columns have very random values, but somehow they are related.\n\nThe columns: \nnom_4 \narray(['Bassoon', 'Theremin', 'Oboe', 'Piano'], dtype=object)\n\nand \n\nord_1\narray(['Contributor', 'Grandmaster', 'Novice', 'Expert', 'Master'], dtype=object)\n\nare related. ord_1 is the grade of playing these instruments.\n\nIf we found some 3rd column (maybe in the binary columns), then we can replace in combination with the missing values.\n\n---\n\nThe countries are somehow related to ord_2\narray(['Hot', 'Warm', 'Freezing', 'Lava Hot', 'Cold', 'Boiling Hot'], dtype=object)\n\nIf ord_2 is possibly the wheater, then it could be related to the months. \n\n---\n\nThe colors could be related also with ord_2, because the temperatures of the wheater are presented with these colors.\n\nWhat do you think?", "709175": "Certainly interesting to see how missing data is treated. Do you have any analysis to show nom_4 and ord_1 are related (and other relations you mentioned) or are you just basing it on the value names?", "709181": "This might help: Just encode ordinal variables and binarize nominals. check for correlation. group by most 3-4 correlated feaures. fill with the median according to the groups.", "709545": "Looking at Theil's U and doing some experiments with feature interactions between *nom4* and *ord1* suggests there might be something to that... but I haven't found anything definitive yet with my current treatment.", "720304": "Replacing missing values with constant gives better results for me.\nI am using OHE + Target encoders and Logistic regression.", "720373": "About relation between month and weather:\n\n.png?generation=1579172730041840&alt=media)\n.png?generation=1579172742780935&alt=media)\n.png?generation=1579172827596653&alt=media)\n.png?generation=1579172762762348&alt=media)\n.png?generation=1579172782806899&alt=media)\n\nSo, they are not connected. Or maybe I'm doing something wrong.", "720381": "vadbeg You need to check which country is above the equator and which below. Because the one has in Dezember winter and the other has summer.", "720389": "'Russia' 'Canada' 'Finland' 'Costa Rica' 'China' 'India'. Each of them is or on equator or above it (but maybe there is some point, I'll check it).", "720396": "vadbeg you have right! Is there some function to iterate over all features and build groups to check which group of features are correlated to another one feature? For example something like: get feature 1 and feature2 and check if correlates to feature 5. Then get feature1+feature2+feature3 and checks if correlates to feature 5 and so on.\nI've read about the replacement method from many features to one, but first, we need to see which group of features can describe some other feature.\nThe method is Random Simple Imputation with many features\n\nHere an example from Titanic. The age is replaced from random value of the fare:\n", "743165": "Thanks for your thoughts... really appreciate !" }, "source": "meta" }