{ "id": 367413, "title": "Idea: cluster the correlation matrix, take simple average over each cluster, only then optimize weights", "url": "/competitions/tabular-playground-series-nov-2022/discussion/367413", "author_name": "Alexander Chervov", "post_date": "2022-11-20T17:15:29.119000", "votes": 14, "comment_count": 0, "views": 0, "content": "

Interesting competition ! Have not yet much time to be involved, but there is some idea which someone may find interesting to try:

\n

Main point: blending uncorrelated solutions is always better - by clear reasons - you might expect that noise also will be uncorrelated and thus blending N solutions would decrease the noise by 1/sqrt(N) factor.
\nIf solutions are correlated - than you might get SYSTEMATIC error and instead of suppressing the random noise you might increase the error - if all solutions are simultaneously making similar error.

\n

The plan:
\n1) calculate the correlation matrix between predictions
\n2) make clustering of predictions based on the correlation matrix.
\n3) For each cluster - take just make simple average for all solutions in cluser - since we may expect these solutions are very similar - something like one solution with small variations. So we get some solution C_i - for each cluster
\n4) Finally - search for the final blend solution by the standard strategy - weighted average Y_blend = w_1C_1+w_2C_2 + .. +w_n*C_n where weights w_i can be found by optimization e.g. scipy.optimize such that score(Y_true , Y_blend )

\n

Details:
\nTo make clustering - one can use different methods - for simple visualization one can start with sns.clustermap - which uses hierarchical clustering, there are many other methods - for example one can make graph and use graph clustering algorithm. Examples of clustering based on the correlation matrix can be found e.g.:

\n

Section \"Correlation heatmaps\":
\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=48
\nSection: \"Visualization of correlated components of correlation graphs\":
\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=54
\nSection \"Blend with scipy.optimize.minimize for all models\"
\nhttps://www.kaggle.com/code/alexandervc/mmscel-blend-analysis#Blend-with-scipy.optimize.minimize--for-all-models

", "messages": [ { "id": 2037454, "postDate": "2022-11-20T17:15:29.120Z", "content": "

Interesting competition ! Have not yet much time to be involved, but there is some idea which someone may find interesting to try:

\n

Main point: blending uncorrelated solutions is always better - by clear reasons - you might expect that noise also will be uncorrelated and thus blending N solutions would decrease the noise by 1/sqrt(N) factor.
\nIf solutions are correlated - than you might get SYSTEMATIC error and instead of suppressing the random noise you might increase the error - if all solutions are simultaneously making similar error.

\n

The plan:
\n1) calculate the correlation matrix between predictions
\n2) make clustering of predictions based on the correlation matrix.
\n3) For each cluster - take just make simple average for all solutions in cluser - since we may expect these solutions are very similar - something like one solution with small variations. So we get some solution C_i - for each cluster
\n4) Finally - search for the final blend solution by the standard strategy - weighted average Y_blend = w_1C_1+w_2C_2 + .. +w_n*C_n where weights w_i can be found by optimization e.g. scipy.optimize such that score(Y_true , Y_blend )

\n

Details:
\nTo make clustering - one can use different methods - for simple visualization one can start with sns.clustermap - which uses hierarchical clustering, there are many other methods - for example one can make graph and use graph clustering algorithm. Examples of clustering based on the correlation matrix can be found e.g.:

\n

Section \"Correlation heatmaps\":
\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=48
\nSection: \"Visualization of correlated components of correlation graphs\":
\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=54
\nSection \"Blend with scipy.optimize.minimize for all models\"
\nhttps://www.kaggle.com/code/alexandervc/mmscel-blend-analysis#Blend-with-scipy.optimize.minimize--for-all-models

", "rawMarkdown": "Interesting competition ! Have not yet much time to be involved, but there is some idea which someone may find interesting to try:\n\n**Main point:** blending uncorrelated solutions is always better - by clear reasons - you might expect that noise also will be uncorrelated and thus blending N solutions would decrease the noise by 1/sqrt(N) factor.\nIf solutions are correlated - than you might get SYSTEMATIC error and instead of suppressing the random noise you might increase the error - if all solutions are simultaneously making similar error.\n\n**The plan:** \n1) calculate the correlation matrix between predictions \n2) make clustering of predictions based on the correlation matrix. \n3) For each cluster - take just make simple average for all solutions in cluser - since we may expect these solutions are very similar - something like one solution with small variations. So we get some solution C_i - for each cluster\n4) Finally - search for the final blend solution by the standard strategy - weighted average Y_blend = w_1*C_1+w_2*C_2 + .. +w_n*C_n where weights w_i can be found by optimization e.g. scipy.optimize such that score(Y_true , Y_blend )\n\n**Details:**\nTo make clustering - one can use different methods - for simple visualization one can start with sns.clustermap - which uses hierarchical clustering, there are many other methods - for example one can make graph and use graph clustering algorithm. Examples of clustering based on the correlation matrix can be found e.g.:\n\nSection \"Correlation heatmaps\":\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=48\nSection: \"Visualization of correlated components of correlation graphs\": \nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=54\nSection \"Blend with scipy.optimize.minimize for all models\"\nhttps://www.kaggle.com/code/alexandervc/mmscel-blend-analysis#Blend-with-scipy.optimize.minimize--for-all-models\n", "votes": 14 } ], "comments": [], "raw_markdown_by_id": { "2037454": "Interesting competition ! Have not yet much time to be involved, but there is some idea which someone may find interesting to try:\n\n**Main point:** blending uncorrelated solutions is always better - by clear reasons - you might expect that noise also will be uncorrelated and thus blending N solutions would decrease the noise by 1/sqrt(N) factor.\nIf solutions are correlated - than you might get SYSTEMATIC error and instead of suppressing the random noise you might increase the error - if all solutions are simultaneously making similar error.\n\n**The plan:** \n1) calculate the correlation matrix between predictions \n2) make clustering of predictions based on the correlation matrix. \n3) For each cluster - take just make simple average for all solutions in cluser - since we may expect these solutions are very similar - something like one solution with small variations. So we get some solution C_i - for each cluster\n4) Finally - search for the final blend solution by the standard strategy - weighted average Y_blend = w_1*C_1+w_2*C_2 + .. +w_n*C_n where weights w_i can be found by optimization e.g. scipy.optimize such that score(Y_true , Y_blend )\n\n**Details:**\nTo make clustering - one can use different methods - for simple visualization one can start with sns.clustermap - which uses hierarchical clustering, there are many other methods - for example one can make graph and use graph clustering algorithm. Examples of clustering based on the correlation matrix can be found e.g.:\n\nSection \"Correlation heatmaps\":\nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=48\nSection: \"Visualization of correlated components of correlation graphs\": \nhttps://www.kaggle.com/code/alexandervc/mmscel-eda-targets-citeseq-02?scriptVersionId=106444735&cellId=54\nSection \"Blend with scipy.optimize.minimize for all models\"\nhttps://www.kaggle.com/code/alexandervc/mmscel-blend-analysis#Blend-with-scipy.optimize.minimize--for-all-models\n" } }