I have been busy with another challenge but noticed that some public kernels use all sorts of algorithms to blend models. Even if you don't have time to use this in this competition, you might want to have a look at lsq_linear from scipy.optimize:\nhttps://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.lsq_linear.html\nIt's basically one line of code and runs in no time.
\n\nIt solves:\nAx = b\nwhere (in this case):\n- A are the stacked predictions of the models you want to blend, \n- b are the validation targets (e.g. the leak, or any validation set), and \n- x will be the optimal weights for the linear combination (If you use the log1p of the target, you are optimizing for the RMSLE metric).
```\nfrom scipy.optimize import lsq_linear\nnmodels = 3
\n\nA = leak_df[[f'pred{i+1}_l1p' for i in range(nmodels)]].values\nb = leak_df.meter_reading_l1p.values\nprint('shapes:', A.shape, b.shape)
\n\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2)\nprint(res)\nbest_weights = res.x\npreds = A.dot(best_weights)\nprint('RMSLE:', np.sqrt(mean_squared_error(preds, leak_df.meter_reading_l1p)))\nprint('Sum of weights:', sum(best_weights))\n```
\n\n\nshapes: (11937501, 3) (11937501,)\nThe unconstrained solution is optimal.\nFinal cost 5.7558e+06, first-order optimality 1.11e+01\n active_mask: array([0., 0., 0.])\n cost: 5755848.0\n fun: array([-0.1326437 , 0.29860926, 0.40199006, ..., 0.19543982,\n 0.12485743, 0.10512161], dtype=float32)\n message: 'The unconstrained solution is optimal.'\n nit: 0\n optimality: 11.09375\n status: 3\n success: True\n x: array([0.52137125, 0.18317722, 0.2858608 ], dtype=float32)\nRMSLE: 0.98201114\nSum of weights: 0.9904092699289322\n
You could also use minimum and maximum bounds for the weights:\n\nlb = np.zeros(preds.shape[1])+0.2 ## lower bound\nub = np.ones(preds.shape[1])/2 ## upper bound\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2, bounds=(lb, ub))\n
I have been busy with another challenge but noticed that some public kernels use all sorts of algorithms to blend models. Even if you don't have time to use this in this competition, you might want to have a look at lsq_linear from scipy.optimize:\nhttps://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.lsq_linear.html\nIt's basically one line of code and runs in no time.
\n\nIt solves:\nAx = b\nwhere (in this case):\n- A are the stacked predictions of the models you want to blend, \n- b are the validation targets (e.g. the leak, or any validation set), and \n- x will be the optimal weights for the linear combination (If you use the log1p of the target, you are optimizing for the RMSLE metric).
```\nfrom scipy.optimize import lsq_linear\nnmodels = 3
\n\nA = leak_df[[f'pred{i+1}_l1p' for i in range(nmodels)]].values\nb = leak_df.meter_reading_l1p.values\nprint('shapes:', A.shape, b.shape)
\n\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2)\nprint(res)\nbest_weights = res.x\npreds = A.dot(best_weights)\nprint('RMSLE:', np.sqrt(mean_squared_error(preds, leak_df.meter_reading_l1p)))\nprint('Sum of weights:', sum(best_weights))\n```
\n\n\nshapes: (11937501, 3) (11937501,)\nThe unconstrained solution is optimal.\nFinal cost 5.7558e+06, first-order optimality 1.11e+01\n active_mask: array([0., 0., 0.])\n cost: 5755848.0\n fun: array([-0.1326437 , 0.29860926, 0.40199006, ..., 0.19543982,\n 0.12485743, 0.10512161], dtype=float32)\n message: 'The unconstrained solution is optimal.'\n nit: 0\n optimality: 11.09375\n status: 3\n success: True\n x: array([0.52137125, 0.18317722, 0.2858608 ], dtype=float32)\nRMSLE: 0.98201114\nSum of weights: 0.9904092699289322\n
You could also use minimum and maximum bounds for the weights:\n\nlb = np.zeros(preds.shape[1])+0.2 ## lower bound\nub = np.ones(preds.shape[1])/2 ## upper bound\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2, bounds=(lb, ub))\n
Nice, this is really cool. I wrote a naive brute-force thingo to search for the best weights to blend some models and it took forever to run haha. This was basically instant and the answer was practically the same.
\n\nThanks for the tip!
", "rawMarkdown": "Nice, this is really cool. I wrote a naive brute-force thingo to search for the best weights to blend some models and it took forever to run haha. This was basically instant and the answer was practically the same.\n\nThanks for the tip!", "votes": null }, { "id": "698978", "postDate": "12/20/2019 00:32:47", "content": "So many thanks! It costed me a lot of time finding the best weights. This one is really cool.
", "rawMarkdown": "So many thanks! It costed me a lot of time finding the best weights. This one is really cool.", "votes": null }, { "id": "699092", "postDate": "12/20/2019 03:33:21", "content": "Thanks. I think the core idea is to do linear regression with leak data, and this method requires indenpend and dependent variables have linear relationships. However, I plotted the scatter pic and found there is not a good linear relationship between my prediction and leak data.
", "rawMarkdown": "Thanks. I think the core idea is to do linear regression with leak data, and this method requires indenpend and dependent variables have linear relationships. However, I plotted the scatter pic and found there is not a good linear relationship between my prediction and leak data.", "votes": null }, { "id": "699233", "postDate": "12/20/2019 08:09:14", "content": "Totally agree @dlutyuanliu \nI just wanted to point out that people were using very inefficient methods to do a simple linear combination!\nPeople have successfully used many non-linear methods for stacking, and I guess so did you? ;)\nYou could even use a gradient boosting tree model to fit the predictions of your many models onto the leaked data, but we need to be careful with overfitting as usual. In that case, linear methods can have some advantage by having an extremely small number of parameters.
", "rawMarkdown": "Totally agree @dlutyuanliu \nI just wanted to point out that people were using very inefficient methods to do a simple linear combination!\nPeople have successfully used many non-linear methods for stacking, and I guess so did you? ;)\nYou could even use a gradient boosting tree model to fit the predictions of your many models onto the leaked data, but we need to be careful with overfitting as usual. In that case, linear methods can have some advantage by having an extremely small number of parameters.", "votes": null }, { "id": "699268", "postDate": "12/20/2019 08:51:10", "content": "Yes, the enumeration is very very slow. I tried to build a nonlinear combinations of my predictions, and applied a heuristic algorithm to find the optimal, but the 'optimal combination' caused overfitting many times 😂. \nI did not have time to build gradient boosting trees but I think it is a potential solution. In that case, I think the train data should be the predictions on validation set, otherwise this stacking will cause overfitting.
", "rawMarkdown": "Yes, the enumeration is very very slow. I tried to build a nonlinear combinations of my predictions, and applied a heuristic algorithm to find the optimal, but the 'optimal combination' caused overfitting many times 😂. \nI did not have time to build gradient boosting trees but I think it is a potential solution. In that case, I think the train data should be the predictions on validation set, otherwise this stacking will cause overfitting.", "votes": null }, { "id": "699566", "postDate": "12/20/2019 15:43:42", "content": "Neat!
", "rawMarkdown": "Neat!", "votes": null } ], "comments": [ { "id": 698889, "author_name": "prschdt", "author_url": "", "post_date": "12/19/2019 20:34:29", "content": "Nice, this is really cool. I wrote a naive brute-force thingo to search for the best weights to blend some models and it took forever to run haha. This was basically instant and the answer was practically the same.
\n\nThanks for the tip!
", "votes": null, "replies": [] }, { "id": 698978, "author_name": "lsyiverson1991", "author_url": "", "post_date": "12/20/2019 00:32:47", "content": "So many thanks! It costed me a lot of time finding the best weights. This one is really cool.
", "votes": null, "replies": [ { "id": 699566, "author_name": "curinga", "author_url": "", "post_date": "12/20/2019 15:43:42", "content": "Neat!
", "votes": null, "replies": [] } ] }, { "id": 699092, "author_name": "dlutyuanliu", "author_url": "", "post_date": "12/20/2019 03:33:21", "content": "Thanks. I think the core idea is to do linear regression with leak data, and this method requires indenpend and dependent variables have linear relationships. However, I plotted the scatter pic and found there is not a good linear relationship between my prediction and leak data.
", "votes": null, "replies": [ { "id": 699233, "author_name": "hmendonca", "author_url": "", "post_date": "12/20/2019 08:09:14", "content": "Totally agree @dlutyuanliu \nI just wanted to point out that people were using very inefficient methods to do a simple linear combination!\nPeople have successfully used many non-linear methods for stacking, and I guess so did you? ;)\nYou could even use a gradient boosting tree model to fit the predictions of your many models onto the leaked data, but we need to be careful with overfitting as usual. In that case, linear methods can have some advantage by having an extremely small number of parameters.
", "votes": null, "replies": [] }, { "id": 699268, "author_name": "dlutyuanliu", "author_url": "", "post_date": "12/20/2019 08:51:10", "content": "Yes, the enumeration is very very slow. I tried to build a nonlinear combinations of my predictions, and applied a heuristic algorithm to find the optimal, but the 'optimal combination' caused overfitting many times 😂. \nI did not have time to build gradient boosting trees but I think it is a potential solution. In that case, I think the train data should be the predictions on validation set, otherwise this stacking will cause overfitting.
", "votes": null, "replies": [] } ] } ], "raw_markdown_by_id": { "698798": "I have been busy with another challenge but noticed that some public kernels use all sorts of algorithms to blend models. Even if you don't have time to use this in this competition, you might want to have a look at **lsq_linear** from scipy.optimize:\nhttps://docs.scipy.org/doc/scipy-0.18.1/reference/generated/scipy.optimize.lsq_linear.html\nIt's basically one line of code and runs in no time.\n\nIt solves:\n`Ax = b`\nwhere (in this case):\n- *A* are the stacked predictions of the models you want to blend, \n- *b* are the validation targets (e.g. the leak, or any validation set), and \n- *x* will be the optimal weights for the linear combination (If you use the log1p of the target, you are optimizing for the RMSLE metric).\n\n```\nfrom scipy.optimize import lsq_linear\nnmodels = 3\n# pred1_l1p, pred2_l1p and pred3_l1p are the predictions for my 3 models\nA = leak_df[[f'pred{i+1}_l1p' for i in range(nmodels)]].values\nb = leak_df.meter_reading_l1p.values\nprint('shapes:', A.shape, b.shape)\n\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2)\nprint(res)\nbest_weights = res.x\npreds = A.dot(best_weights)\nprint('RMSLE:', np.sqrt(mean_squared_error(preds, leak_df.meter_reading_l1p)))\nprint('Sum of weights:', sum(best_weights))\n```\n\n```\nshapes: (11937501, 3) (11937501,)\nThe unconstrained solution is optimal.\nFinal cost 5.7558e+06, first-order optimality 1.11e+01\n active_mask: array([0., 0., 0.])\n cost: 5755848.0\n fun: array([-0.1326437 , 0.29860926, 0.40199006, ..., 0.19543982,\n 0.12485743, 0.10512161], dtype=float32)\n message: 'The unconstrained solution is optimal.'\n nit: 0\n optimality: 11.09375\n status: 3\n success: True\n x: array([0.52137125, 0.18317722, 0.2858608 ], dtype=float32)\nRMSLE: 0.98201114\nSum of weights: 0.9904092699289322\n```\n\nYou could also use minimum and maximum bounds for the weights:\n```\nlb = np.zeros(preds.shape[1])+0.2 ## lower bound\nub = np.ones(preds.shape[1])/2 ## upper bound\nres = lsq_linear(A, b, lsq_solver='exact', method='trf', tol=1e-5, verbose=2, bounds=(lb, ub))\n```", "698889": "Nice, this is really cool. I wrote a naive brute-force thingo to search for the best weights to blend some models and it took forever to run haha. This was basically instant and the answer was practically the same.\n\nThanks for the tip!", "698978": "So many thanks! It costed me a lot of time finding the best weights. This one is really cool.", "699092": "Thanks. I think the core idea is to do linear regression with leak data, and this method requires indenpend and dependent variables have linear relationships. However, I plotted the scatter pic and found there is not a good linear relationship between my prediction and leak data.", "699233": "Totally agree @dlutyuanliu \nI just wanted to point out that people were using very inefficient methods to do a simple linear combination!\nPeople have successfully used many non-linear methods for stacking, and I guess so did you? ;)\nYou could even use a gradient boosting tree model to fit the predictions of your many models onto the leaked data, but we need to be careful with overfitting as usual. In that case, linear methods can have some advantage by having an extremely small number of parameters.", "699268": "Yes, the enumeration is very very slow. I tried to build a nonlinear combinations of my predictions, and applied a heuristic algorithm to find the optimal, but the 'optimal combination' caused overfitting many times 😂. \nI did not have time to build gradient boosting trees but I think it is a potential solution. In that case, I think the train data should be the predictions on validation set, otherwise this stacking will cause overfitting.", "699566": "Neat!" }, "source": "meta" }