Just a quick explanation of why your offer does not make any sense at all.
\n\nThe price of one hour of very good machines (way better than yours) is <4$ per hour. https://aws.amazon.com/ec2/pricing/
\n\nA price for the hour of explanation of more or less complicated R code is > 20$ per hour. On top of that someone has to spend significant amount of time explaining you how to run his code.
\n\nAlso for a person with a code, will not be enough. I ran your code, it finished in 25 minutes, the result is 42. He has to look, tweak, analyze, iterate many times to get some understanding.
", "rawMarkdown": "Just a quick explanation of why your offer does not make any sense at all. \r\n\r\nThe price of one hour of very good machines (way better than yours) is <4$ per hour. https://aws.amazon.com/ec2/pricing/\r\n\r\nA price for the hour of explanation of more or less complicated R code is > 20$ per hour. On top of that someone has to spend significant amount of time explaining you how to run his code.\r\n\r\nAlso for a person with a code, will not be enough. I ran your code, it finished in 25 minutes, the result is 42. He has to look, tweak, analyze, iterate many times to get some understanding.", "votes": null }, { "id": "131945", "postDate": "08/20/2016 20:26:01", "content": "Thank you for your advice SD,\nFirst of all I did not mention that I am an ETL expert and I can help and learn others too and it is my bad.\nsecondly, I have found some usefull features:\none reduces the lrmse to 0.31xx for 8 percent of data\none tune the result about .005 \nThirdly, learning from you is a humble way of asking to team up, I have researched a great deal of mathematics with two separate masters in theoretical physics and Geophysics(seismic data processing)\nso if you or any one else wanted to consider my offer , he/she can send me over his/her proposal
", "rawMarkdown": "Thank you for your advice SD,\r\nFirst of all I did not mention that I am an ETL expert and I can help and learn others too and it is my bad.\r\nsecondly, I have found some usefull features:\r\none reduces the lrmse to 0.31xx for 8 percent of data\r\none tune the result about .005 \r\nThirdly, learning from you is a humble way of asking to team up, I have researched a great deal of mathematics with two separate masters in theoretical physics and Geophysics(seismic data processing)\r\nso if you or any one else wanted to consider my offer , he/she can send me over his/her proposal", "votes": null }, { "id": "132008", "postDate": "08/21/2016 14:49:39", "content": "[quote=AfshinAmiri;131945]
\n\none reduces the lrmse to 0.31xx for 8 percent of data
\n\n[/quote]
\n\nLRMSE is not an appropriate measurement for this dataset as the classes are widely unbalanced. A tuned random number generator can get ~0.03(?) logloss alone just by predicting a constant number.
\n\nActually, anyone with a 4GB computer can work with the train_numeric dataset at least (train_categorical not sure yet, have to look at it). It just requires smart loading and smart computation, along with smart supervised learning on the data.
\n\nSmart loading (load under 4GB the training numeric set), do by batch:
\n\nSmart computation (example on feature selection), do by batch:
\n\nSmart supervised learning (example on xgboost):
\n\nyour result does not show you are a smart guy so please speak with yourself.\nI mentioned that I reduces LRMSE and I did not say that I used LRMSE as cost function so you are kind of biased and I want to know why. I think some guys are just see the text and do not process it in the brain.\nagain, I don't care what any one can do I have the power of computation and I use that. Be sure that when the problem of performance comes I am among the best.\nwhen you have a more powerful tool you can dig deeper (even your own grave) \nsome of your advises like the one "use a truncated SVD" is a change in solution strategy in middle of problem which is not smart at all.
", "rawMarkdown": "your result does not show you are a smart guy so please speak with yourself.\r\nI mentioned that I reduces LRMSE and I did not say that I used LRMSE as cost function so you are kind of biased and I want to know why. I think some guys are just see the text and do not process it in the brain.\r\nagain, I don't care what any one can do I have the power of computation and I use that. Be sure that when the problem of performance comes I am among the best.\r\nwhen you have a more powerful tool you can dig deeper (even your own grave) \r\nsome of your advises like the one \"use a truncated SVD\" is a change in solution strategy in middle of problem which is not smart at all.", "votes": null }, { "id": "132053", "postDate": "08/22/2016 08:13:50", "content": "Dear AfshinAmiri,\nyou talk alot! But please do not just talk the talk, but walk the walk. We have not seen any useful posts from you neither have you done any near useful submission on kaggle. So if you are an expert, show us what you can do and don't talk stupid.
\n\nAlso, calling Prof. Laurae "not a smart guy" seems totally stupid...
\n\nBest,\nSterby
", "rawMarkdown": "Dear AfshinAmiri,\r\nyou talk alot! But please do not just talk the talk, but walk the walk. We have not seen any useful posts from you neither have you done any near useful submission on kaggle. So if you are an expert, show us what you can do and don't talk stupid.\r\n\r\nAlso, calling Prof. Laurae \"not a smart guy\" seems totally stupid...\r\n\r\nBest,\r\nSterby", "votes": null }, { "id": "132070", "postDate": "08/22/2016 10:48:54", "content": "such a dou..
", "rawMarkdown": "such a dou..", "votes": null }, { "id": "132074", "postDate": "08/22/2016 11:00:48", "content": "If you are so illiterate that don't know what the professor position is please see:\nProffesor at Wiki
", "rawMarkdown": "If you are so illiterate that don't know what the professor position is please see:\r\n[Proffesor at Wiki][1]\r\n\r\n\r\n [1]: https://en.wikipedia.org/wiki/Professor", "votes": null }, { "id": "132110", "postDate": "08/22/2016 16:50:21", "content": "When will he get a hint?
", "rawMarkdown": "When will he get a hint?", "votes": null } ], "comments": [ { "id": 131941, "author_name": "salvadordali", "author_url": "", "post_date": "08/20/2016 20:09:39", "content": "Just a quick explanation of why your offer does not make any sense at all.
\n\nThe price of one hour of very good machines (way better than yours) is <4$ per hour. https://aws.amazon.com/ec2/pricing/
\n\nA price for the hour of explanation of more or less complicated R code is > 20$ per hour. On top of that someone has to spend significant amount of time explaining you how to run his code.
\n\nAlso for a person with a code, will not be enough. I ran your code, it finished in 25 minutes, the result is 42. He has to look, tweak, analyze, iterate many times to get some understanding.
", "votes": null, "replies": [] }, { "id": 131945, "author_name": "afshinamiri", "author_url": "", "post_date": "08/20/2016 20:26:01", "content": "Thank you for your advice SD,\nFirst of all I did not mention that I am an ETL expert and I can help and learn others too and it is my bad.\nsecondly, I have found some usefull features:\none reduces the lrmse to 0.31xx for 8 percent of data\none tune the result about .005 \nThirdly, learning from you is a humble way of asking to team up, I have researched a great deal of mathematics with two separate masters in theoretical physics and Geophysics(seismic data processing)\nso if you or any one else wanted to consider my offer , he/she can send me over his/her proposal
", "votes": null, "replies": [] }, { "id": 132008, "author_name": "laurae2", "author_url": "", "post_date": "08/21/2016 14:49:39", "content": "[quote=AfshinAmiri;131945]
\n\none reduces the lrmse to 0.31xx for 8 percent of data
\n\n[/quote]
\n\nLRMSE is not an appropriate measurement for this dataset as the classes are widely unbalanced. A tuned random number generator can get ~0.03(?) logloss alone just by predicting a constant number.
\n\nActually, anyone with a 4GB computer can work with the train_numeric dataset at least (train_categorical not sure yet, have to look at it). It just requires smart loading and smart computation, along with smart supervised learning on the data.
\n\nSmart loading (load under 4GB the training numeric set), do by batch:
\n\nSmart computation (example on feature selection), do by batch:
\n\nSmart supervised learning (example on xgboost):
\n\nyour result does not show you are a smart guy so please speak with yourself.\nI mentioned that I reduces LRMSE and I did not say that I used LRMSE as cost function so you are kind of biased and I want to know why. I think some guys are just see the text and do not process it in the brain.\nagain, I don't care what any one can do I have the power of computation and I use that. Be sure that when the problem of performance comes I am among the best.\nwhen you have a more powerful tool you can dig deeper (even your own grave) \nsome of your advises like the one "use a truncated SVD" is a change in solution strategy in middle of problem which is not smart at all.
", "votes": null, "replies": [] }, { "id": 132053, "author_name": "sterby", "author_url": "", "post_date": "08/22/2016 08:13:50", "content": "Dear AfshinAmiri,\nyou talk alot! But please do not just talk the talk, but walk the walk. We have not seen any useful posts from you neither have you done any near useful submission on kaggle. So if you are an expert, show us what you can do and don't talk stupid.
\n\nAlso, calling Prof. Laurae "not a smart guy" seems totally stupid...
\n\nBest,\nSterby
", "votes": null, "replies": [] }, { "id": 132070, "author_name": "afshinamiri", "author_url": "", "post_date": "08/22/2016 10:48:54", "content": "such a dou..
", "votes": null, "replies": [] }, { "id": 132074, "author_name": "afshinamiri", "author_url": "", "post_date": "08/22/2016 11:00:48", "content": "If you are so illiterate that don't know what the professor position is please see:\nProffesor at Wiki
", "votes": null, "replies": [] }, { "id": 132110, "author_name": "sakvaua", "author_url": "", "post_date": "08/22/2016 16:50:21", "content": "When will he get a hint?
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "131805": "", "131941": "Just a quick explanation of why your offer does not make any sense at all. \r\n\r\nThe price of one hour of very good machines (way better than yours) is <4$ per hour. https://aws.amazon.com/ec2/pricing/\r\n\r\nA price for the hour of explanation of more or less complicated R code is > 20$ per hour. On top of that someone has to spend significant amount of time explaining you how to run his code.\r\n\r\nAlso for a person with a code, will not be enough. I ran your code, it finished in 25 minutes, the result is 42. He has to look, tweak, analyze, iterate many times to get some understanding.", "131945": "Thank you for your advice SD,\r\nFirst of all I did not mention that I am an ETL expert and I can help and learn others too and it is my bad.\r\nsecondly, I have found some usefull features:\r\none reduces the lrmse to 0.31xx for 8 percent of data\r\none tune the result about .005 \r\nThirdly, learning from you is a humble way of asking to team up, I have researched a great deal of mathematics with two separate masters in theoretical physics and Geophysics(seismic data processing)\r\nso if you or any one else wanted to consider my offer , he/she can send me over his/her proposal", "132008": "[quote=AfshinAmiri;131945]\r\n\r\none reduces the lrmse to 0.31xx for 8 percent of data\r\n\r\n[/quote]\r\n\r\nLRMSE is not an appropriate measurement for this dataset as the classes are widely unbalanced. A tuned random number generator can get ~0.03(?) logloss alone just by predicting a constant number.\r\n\r\nActually, anyone with a 4GB computer can work with the train_numeric dataset at least (train_categorical not sure yet, have to look at it). It just requires smart loading and smart computation, along with smart supervised learning on the data.\r\n\r\nSmart loading (load under 4GB the training numeric set), do by batch:\r\n\r\n* Load specific features or rows (the smaller the batch the easier it fits in memory: if you live with 2GB RAM you can still do it but you will end up swapping no matter what you do, even on a 1x1 batch which would be absurd)\r\n* Bind batch as sparse with previous sparse batch\r\n* Remove old dense batch\r\n\r\nSmart computation (example on feature selection), do by batch:\r\n\r\n* Load only specific columns (30 seconds for 50 columns only for instance - the bigger the batch not forcibly better results as high interacting features might impede feature importance scoring visibility, especially if you are using a threshold)\r\n* Run a decision tree that does not consume lot of RAM on loaded columns in single thread\r\n* Get feature importance\r\n* Store in a dictionary the importance\r\n\r\nSmart supervised learning (example on xgboost):\r\n\r\n* Clear up all things you do not need in memory\r\n* Use only 1 core\r\n* Use a minimal amount of features (xgboost 4-fold CV on 17 targeted features can get 0.17 MCC in under 10 minutes).\r\n* If you need more features than you can use in RAM, use a truncated SVD on the sparse matrix of features but you end up with a dense matrix (but it is still better to work with sparse matrices, it is much faster)", "132014": "your result does not show you are a smart guy so please speak with yourself.\r\nI mentioned that I reduces LRMSE and I did not say that I used LRMSE as cost function so you are kind of biased and I want to know why. I think some guys are just see the text and do not process it in the brain.\r\nagain, I don't care what any one can do I have the power of computation and I use that. Be sure that when the problem of performance comes I am among the best.\r\nwhen you have a more powerful tool you can dig deeper (even your own grave) \r\nsome of your advises like the one \"use a truncated SVD\" is a change in solution strategy in middle of problem which is not smart at all.", "132053": "Dear AfshinAmiri,\r\nyou talk alot! But please do not just talk the talk, but walk the walk. We have not seen any useful posts from you neither have you done any near useful submission on kaggle. So if you are an expert, show us what you can do and don't talk stupid.\r\n\r\nAlso, calling Prof. Laurae \"not a smart guy\" seems totally stupid...\r\n\r\nBest,\r\nSterby", "132070": "such a dou..", "132074": "If you are so illiterate that don't know what the professor position is please see:\r\n[Proffesor at Wiki][1]\r\n\r\n\r\n [1]: https://en.wikipedia.org/wiki/Professor", "132110": "When will he get a hint?" }, "source": "meta" }