![\"\"](\"https://www.googleapis.com/download/storage/v1/b/kaggle-user-content/o/inbox%2F219502%2F9a08bfd93c315f4e68ff15420b90cdf9%2FSelection_053.png?generation=1573768313092304&alt=media\")
\nRules clarification from Kaggle:\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345I am wondering which will be my next competition when I saw here that some test labels can be scrapped from web sites.
\n\nThis reminds me of some previous image competitions where web scrapping became the most useful skill to win the competition.
\n\nI welcome any feedback on this question: do you think all publicly available test labels are known and shared, or do you think it is a web scrapping competition now?
", "messages": [ { "id": 670723, "postDate": "2019-11-11T19:27:39.323Z", "content": "I am wondering which will be my next competition when I saw here that some test labels can be scrapped from web sites.
\n\nThis reminds me of some previous image competitions where web scrapping became the most useful skill to win the competition.
\n\nI welcome any feedback on this question: do you think all publicly available test labels are known and shared, or do you think it is a web scrapping competition now?
", "rawMarkdown": "I am wondering which will be my next competition when I saw here that some test labels can be scrapped from web sites.\n\nThis reminds me of some previous image competitions where web scrapping became the most useful skill to win the competition.\n\nI welcome any feedback on this question: do you think all publicly available test labels are known and shared, or do you think it is a web scrapping competition now?", "votes": 45 }, { "id": 670786, "postDate": "2019-11-11T21:01:09.753Z", "content": "Well, it looks like Google Analytics competition now. We need clarification from competition host ASAP.
", "rawMarkdown": "Well, it looks like Google Analytics competition now. We need clarification from competition host ASAP.", "votes": 11, "replies": [ { "id": 670896, "postDate": "2019-11-12T02:31:22.047Z", "content": "Anyway, Google analytics competition has finished as ML competition, not scraping competition.\nI hope this competition finishes as ML competition.
", "rawMarkdown": "Anyway, Google analytics competition has finished as ML competition, not scraping competition.\nI hope this competition finishes as ML competition. ", "votes": 2 }, { "id": 670924, "postDate": "2019-11-12T03:56:47.070Z", "content": "Me too. I have spent two weekends in this competition. So sad.
", "rawMarkdown": "Me too. I have spent two weekends in this competition. So sad.", "votes": 2 }, { "id": 671047, "postDate": "2019-11-12T07:30:26.160Z", "content": "If you mean \"ML competition\" by \"submit all 1e-4 values to get bronze\" then it does. Hope this one will find another way.
", "rawMarkdown": "If you mean \"ML competition\" by \"submit all 1e-4 values to get bronze\" then it does. Hope this one will find another way.", "votes": 6 }, { "id": 674420, "postDate": "2019-11-16T13:30:52.797Z", "rawMarkdown": "", "votes": 1, "isDeleted": true } ] }, { "id": 670737, "postDate": "2019-11-11T19:37:45.797Z", "content": "It seems that more of the test labels are becoming known. site 1 was discovered but not shared so it's only a matter of time all the test labels will be shared and revealed slowly. It is becoming a scraping competition... I find myself looking through university building energy data instead of machine learning.
", "rawMarkdown": "It seems that more of the test labels are becoming known. [site 1](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#670386) was discovered but not shared so it's only a matter of time all the test labels will be shared and revealed slowly. It is becoming a scraping competition... I find myself looking through university building energy data instead of machine learning.", "votes": 6, "replies": [ { "id": 670758, "postDate": "2019-11-11T19:51:02.517Z", "content": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.
", "rawMarkdown": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.", "votes": 6 } ] }, { "id": 671390, "postDate": "2019-11-12T16:21:14.990Z", "content": "I can confirm it is possible to be 2nd on the LB with a simple override of UCF data.\nI don't know if all publicly available test labels are known or shared but I am confident that these would be shared as the community here is focused on learning by sharing. So I hope and think the challenge is still fair even if the leak is disappointing... Moreover there is no proof that private test data is inline with public LB ones.\n@cpmpml I noticed web scraping is not a skill you want to exercise... I showed that I am able to use this skill and, I hope so, some others too. I am here to learn ML. This challenge is still, in my opinion, very... challenging and interesting and I guess you love that! I would enjoy so much to work with you to purpose a ML solution to this challenge. Of course I would do the web scraping (and more) if needed, you would not have to do it! Please contact me to discuss on how we could cooperate if you are interested. Everybody here would win/learn if you participated in this challenge.
", "rawMarkdown": "I can confirm it is possible to be 2nd on the LB with a simple override of UCF data.\nI don't know if all publicly available test labels are known or shared but I am confident that these would be shared as the community here is focused on learning by sharing. So I hope and think the challenge is still fair even if the leak is disappointing... Moreover there is no proof that private test data is inline with public LB ones.\n@cpmpml I noticed web scraping is not a skill you want to exercise... I showed that I am able to use this skill and, I hope so, some others too. I am here to learn ML. This challenge is still, in my opinion, very... challenging and interesting and I guess you love that! I would enjoy so much to work with you to purpose a ML solution to this challenge. Of course I would do the web scraping (and more) if needed, you would not have to do it! Please contact me to discuss on how we could cooperate if you are interested. Everybody here would win/learn if you participated in this challenge.", "votes": 3, "replies": [ { "id": 671968, "postDate": "2019-11-13T11:35:51.633Z", "content": "That's a generous offer but I'll pass. Web scrapping will influence final ranking too much for my taste.
", "rawMarkdown": "That's a generous offer but I'll pass. Web scrapping will influence final ranking too much for my taste.", "votes": 4 }, { "id": 672245, "postDate": "2019-11-13T17:17:51.647Z", "content": "web scraping is not a skill you want to exercise = Strongly agreed.\nI am here to learn ML = Strongly agreed.\nno proof that private test data is inline with public LB ones = Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.
\n\nMy faith do not use these data and some externel datasets, It is unfair. \nYes, someone says life itself is unfair. nope!! Not at all. Strongly disagreed.
", "rawMarkdown": "web scraping is not a skill you want to exercise = Strongly agreed.\nI am here to learn ML = Strongly agreed.\nno proof that private test data is inline with public LB ones = Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.\n\nMy faith do not use these data and some externel datasets, It is unfair. \nYes, someone says life itself is unfair. nope!! Not at all. Strongly disagreed.\n", "votes": 2 }, { "id": 672355, "postDate": "2019-11-13T19:53:13.987Z", "content": "@cpmpml This is not a generous offer ;) My goal is to learn ML and I am sure you would be a fabulous prof/mentor. You would have shared more than I would/could have. I know that doing is the best way to learn and I think that learning by doing beside you would have been even better. I already wanted to purpose to work as a team in Microsoft challenge but I was sure you would have not consider my offer. I tried this time to not regret. My offer is still and will ever be opened! I am disappointed that you have chosen to not be part of this challenge because I would have learned more. I just respect it. In the same time I feel proud you did not just ignore my offer. Maybe you even considered it...\nKaggle is full of disappointments for me... \"Microsoft Malware Prediction\" challenge still does not make sense to me (https://www.kaggle.com/c/microsoft-malware-prediction/discussion/83946#490008) and I still think that I deserved a better reward to my investment. As a lot of the great competitors that gave all... Once again in ASHRAE challenge I am disappointed. I come to learn ML and I have to web scrape... I wonder if Kaggle is the better place to spend (a lot of) my time... But I am sure Kaggle team try to do the best, ASHRAE too, and every participant here seems to try to do the same. I benefit a lot of what is shared here. I benefit a lot of what you (and many others) shared and share. I am grateful to all of you for that... So I am going to keep fighting. I gonna fight with ML tools. I will, as I have done this weekend, just exploit external data shared by other as it is part of the challenge/rules. I do not like it but I have to do it. Narcism!\n@hanjoonchoe \"Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.\" I jumped too to 1st place by replacing UCF data in a submission that scored 1.06 without exploiting the leak. It scored 1.04. I did that to verify (probe) if that data were used in the public LB. I was happy. Narcism! In the same time I was disappointed: this is a huge leak and mean more. The data can probably be used to more than replacement... Is it still a real world problem? A real ML problem? Are we just competing? But it does mean nothing about the data used in the private test. As far as I understand... This is to me the motivation to continue. I try to trust everyone: Kaggle team is going to make the choices to a fair (I think it exists ;)) competition and the competitors gonna play fair and share external data. Not too much narcism!
", "rawMarkdown": "@cpmpml This is not a generous offer ;) My goal is to learn ML and I am sure you would be a fabulous prof/mentor. You would have shared more than I would/could have. I know that doing is the best way to learn and I think that learning by doing beside you would have been even better. I already wanted to purpose to work as a team in Microsoft challenge but I was sure you would have not consider my offer. I tried this time to not regret. My offer is still and will ever be opened! I am disappointed that you have chosen to not be part of this challenge because I would have learned more. I just respect it. In the same time I feel proud you did not just ignore my offer. Maybe you even considered it...\nKaggle is full of disappointments for me... \"Microsoft Malware Prediction\" challenge still does not make sense to me (https://www.kaggle.com/c/microsoft-malware-prediction/discussion/83946#490008) and I still think that I deserved a better reward to my investment. As a lot of the great competitors that gave all... Once again in ASHRAE challenge I am disappointed. I come to learn ML and I have to web scrape... I wonder if Kaggle is the better place to spend (a lot of) my time... But I am sure Kaggle team try to do the best, ASHRAE too, and every participant here seems to try to do the same. I benefit a lot of what is shared here. I benefit a lot of what you (and many others) shared and share. I am grateful to all of you for that... So I am going to keep fighting. I gonna fight with ML tools. I will, as I have done this weekend, just exploit external data shared by other as it is part of the challenge/rules. I do not like it but I have to do it. Narcism!\n@hanjoonchoe \"Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.\" I jumped too to 1st place by replacing UCF data in a submission that scored 1.06 without exploiting the leak. It scored 1.04. I did that to verify (probe) if that data were used in the public LB. I was happy. Narcism! In the same time I was disappointed: this is a huge leak and mean more. The data can probably be used to more than replacement... Is it still a real world problem? A real ML problem? Are we just competing? But it does mean nothing about the data used in the private test. As far as I understand... This is to me the motivation to continue. I try to trust everyone: Kaggle team is going to make the choices to a fair (I think it exists ;)) competition and the competitors gonna play fair and share external data. Not too much narcism!" }, { "id": 672687, "postDate": "2019-11-14T04:19:12.100Z", "content": "@pdnartreb Don't be disappointed by Kaggle. Kaggle competition are hard, many of the best datascientists come and compete.
\n\nIn Malware you should have asked. I see you did well there, being one of the few top 100 public teams to stay in top 100 in private.
", "rawMarkdown": "@pdnartreb Don't be disappointed by Kaggle. Kaggle competition are hard, many of the best datascientists come and compete. \n\nIn Malware you should have asked. I see you did well there, being one of the few top 100 public teams to stay in top 100 in private.\n\n\n\n ", "votes": 1 } ] }, { "id": 674049, "postDate": "2019-11-15T20:54:42.937Z", "content": "\"Is this a webscraping competition?\"
\n\nOh Yes It seems more than illusion !\nThanks once again for your remarks, your warns, and your advices...
", "rawMarkdown": "**\"Is this a webscraping competition?\"**\n\nOh Yes It seems more than illusion !\nThanks once again for your remarks, your warns, and your advices...", "votes": 2 }, { "id": 673349, "postDate": "2019-11-14T21:45:03.797Z", "content": "\nRules clarification from Kaggle:\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345
I am afraid it might end up being one - although to be fair, if you provide super detailed info like measurements for energy and weather but obfuscate the actual locations, you're practically asking for it ...
", "rawMarkdown": "I am afraid it might end up being one - although to be fair, if you provide super detailed info like measurements for energy and weather but obfuscate the actual locations, you're practically asking for it ...", "votes": 1, "replies": [ { "id": 670756, "postDate": "2019-11-11T19:50:40.403Z", "content": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.
", "rawMarkdown": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.", "votes": 1 }, { "id": 670783, "postDate": "2019-11-11T20:52:33Z", "content": "Thanks :-) To be honest, using the scraped public data as part of your training set is not that hard - so I was actually surprised more people have not done it so far.
", "rawMarkdown": "Thanks :-) To be honest, using the scraped public data as part of your training set is not that hard - so I was actually surprised more people have not done it so far. ", "votes": 2 }, { "id": 670820, "postDate": "2019-11-11T22:48:17.837Z", "content": "Maybe not hard, but not a skill I'd like to exercise ;)
", "rawMarkdown": "Maybe not hard, but not a skill I'd like to exercise ;)", "votes": 10 }, { "id": 673337, "postDate": "2019-11-14T21:21:07.690Z", "content": "+1
", "rawMarkdown": "+1 ", "votes": 1 } ] }, { "id": 672160, "postDate": "2019-11-13T15:11:00.297Z", "content": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info here). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.
", "rawMarkdown": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info [here][1]). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.\n\n[1]: https://www.kaggle.com/product-feedback/116079", "votes": -5, "replies": [ { "id": 672168, "postDate": "2019-11-13T15:23:50.887Z", "content": "> scrap data and make high quality Datasets to share with other Kagglers
\n\nIs that legal? I mean we'd need consent of the website owners, no?
", "rawMarkdown": "> scrap data and make high quality Datasets to share with other Kagglers\n\nIs that legal? I mean we'd need consent of the website owners, no?", "votes": 2 }, { "id": 672201, "postDate": "2019-11-13T16:04:22.653Z", "content": "Good question, I don't know the law. But if you read an online news article (or other public information), I think you are allowed to write and/or tell a friend what you read.
", "rawMarkdown": "Good question, I don't know the law. But if you read an online news article (or other public information), I think you are allowed to write and/or tell a friend what you read. ", "votes": 1 }, { "id": 672250, "postDate": "2019-11-13T17:27:48.807Z", "content": "Most of the time those websites have robots.txt in their index. It is a standard used by websites to communicate with web crawlers. The standard informs the web crawler about which areas of the website should not be crawled. It is good practice to obey robots.txt.
Some websites even have sitemap.xml in their index which makes it easier for you to scrape data. If a website doesn't want you to take their data, they already have implemented a bot protection system, so don't worry.
Like anybody cared about the dataset tier...
", "rawMarkdown": "Like anybody cared about the dataset tier...", "votes": 3 }, { "id": 672378, "postDate": "2019-11-13T20:19:54.887Z", "content": "Some apparently do :P
", "rawMarkdown": "Some apparently do :P", "votes": 3 }, { "id": 672679, "postDate": "2019-11-14T04:12:15.683Z", "content": "\n\n\nIs that legal? I mean we'd need consent of the website owners, no?
\n
A very relevant question indeed. From the Terms page (link at the bottom of any Kaggle page):
\n\n\n\n\nYou are responsible for all Content you contribute to the Services, and you represent and warrant you have all rights necessary to do so.
\n
Usually, sites with data have a License page somewhere. That's the best way to see if you can reproduce their data. If there is nothing that says it is open source or public domain then copyright applies, and you must ask permission.
", "rawMarkdown": "> Is that legal? I mean we'd need consent of the website owners, no?\n\nA very relevant question indeed. From the Terms page (link at the bottom of any Kaggle page):\n\n> You are responsible for all Content you contribute to the Services, and you represent and warrant you have all rights necessary to do so.\n\nUsually, sites with data have a License page somewhere. That's the best way to see if you can reproduce their data. If there is nothing that says it is open source or public domain then copyright applies, and you must ask permission.", "votes": 2 }, { "id": 674076, "postDate": "2019-11-15T21:37:44.380Z", "content": "IANAL, and this is only the relevant US law, but databases/datasets have a separate copyright from the data they're scraped from. While downloading the web pages and uploading them to Kaggle would violate copyright, just scraping the data would not, as the database you create is its own copyrightable work. Additionally, publicly available data is perfectly legal to scrape, at least according to the rulings currently on a lawsuit involving LinkedIn.
", "rawMarkdown": "IANAL, and this is only the relevant US law, but databases/datasets have a separate copyright from the data they're scraped from. While downloading the web pages and uploading them to Kaggle **would** violate copyright, just scraping the data would not, as the database you create is its own copyrightable work. Additionally, publicly available data is perfectly legal to scrape, at least according to [the rulings currently](https://www.vice.com/en_us/article/9kek83/linkedin-data-scraping-lawsuit-shot-down) on a lawsuit involving LinkedIn." }, { "id": 674092, "postDate": "2019-11-15T22:01:41.357Z", "content": "The suit you point to isn't about copyright, it is about computer fraud and abuse act. What you missed is that LinkedIn users agreed to make their data publicly available, i.e. make it public domain. Therefore scrapping that data is fine.
\n\nI stand by what I wrote, and I advise anyone having doubt to consult with an attorney at law.
", "rawMarkdown": "The suit you point to isn't about copyright, it is about computer fraud and abuse act. What you missed is that LinkedIn users agreed to make their data publicly available, i.e. make it public domain. Therefore scrapping that data is fine.\n\nI stand by what I wrote, and I advise anyone having doubt to consult with an attorney at law." } ] }, { "id": 676833, "postDate": "2019-11-19T14:34:41.217Z", "content": "I want to, how many of you are only using features provided by the kaggle? I mean I have tried everything with those features but I was unable to get anything. Is it external features that do the magic???
", "rawMarkdown": "I want to, how many of you are only using features provided by the kaggle? I mean I have tried everything with those features but I was unable to get anything. Is it external features that do the magic???" }, { "id": 670746, "postDate": "2019-11-11T19:45:40.373Z", "content": "On the other hand, taking out any possibility of getting close to a medal because who cares about scraping more than others, there is plenty of space to try out some more creative, most likely less performant, approach.
\n\nWin-win. :)
", "rawMarkdown": "On the other hand, taking out any possibility of getting close to a medal because who cares about scraping more than others, there is plenty of space to try out some more creative, most likely less performant, approach. \n\nWin-win. :)", "replies": [ { "id": 670755, "postDate": "2019-11-11T19:50:26.670Z", "content": "There are other competitions where you can exercise ML skills and still get to the top if you're lucky and skilled. Here it seems you need to get into web scrapping, from the other answers I got, which means that I will pass.
", "rawMarkdown": "There are other competitions where you can exercise ML skills and still get to the top if you're lucky and skilled. Here it seems you need to get into web scrapping, from the other answers I got, which means that I will pass.", "votes": 5 }, { "id": 670763, "postDate": "2019-11-11T20:06:09.440Z", "content": "It is really a shame that they didn't notice such a clear flaw in the competition design indeed.
\n\nWhat I meant is that, hopefully, many people are competing only to learn and that aspect is not compromised by the fact that this competition is ruined beyond recovery.
", "rawMarkdown": "It is really a shame that they didn't notice such a clear flaw in the competition design indeed.\n\nWhat I meant is that, hopefully, many people are competing only to learn and that aspect is not compromised by the fact that this competition is ruined beyond recovery.", "votes": 2 }, { "id": 670929, "postDate": "2019-11-12T04:01:51.980Z", "content": "Tensorflow 2.0 competition seems promesing for learning. At my work i use tensorflow for productive modeling which is great for distributed training.
", "rawMarkdown": "Tensorflow 2.0 competition seems promesing for learning. At my work i use tensorflow for productive modeling which is great for distributed training." } ] } ], "comments": [ { "id": 670786, "author_name": "WispZero", "author_url": "", "post_date": "2019-11-11T21:01:09.753000", "content": "Well, it looks like Google Analytics competition now. We need clarification from competition host ASAP.
", "votes": 11, "replies": [ { "id": 670896, "author_name": "Youhan Lee", "author_url": "", "post_date": "2019-11-12T02:31:22.047000", "content": "Anyway, Google analytics competition has finished as ML competition, not scraping competition.\nI hope this competition finishes as ML competition.
", "votes": 2, "replies": [] }, { "id": 670924, "author_name": "Correlation", "author_url": "", "post_date": "2019-11-12T03:56:47.070000", "content": "Me too. I have spent two weekends in this competition. So sad.
", "votes": 2, "replies": [] }, { "id": 671047, "author_name": "WispZero", "author_url": "", "post_date": "2019-11-12T07:30:26.160000", "content": "If you mean \"ML competition\" by \"submit all 1e-4 values to get bronze\" then it does. Hope this one will find another way.
", "votes": 6, "replies": [] }, { "id": 674420, "author_name": "", "author_url": "", "post_date": "2019-11-16T13:30:52.797000", "content": "", "votes": 1, "replies": [] } ] }, { "id": 670737, "author_name": "Tim Yee", "author_url": "", "post_date": "2019-11-11T19:37:45.797000", "content": "It seems that more of the test labels are becoming known. site 1 was discovered but not shared so it's only a matter of time all the test labels will be shared and revealed slowly. It is becoming a scraping competition... I find myself looking through university building energy data instead of machine learning.
", "votes": 6, "replies": [ { "id": 670758, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-11T19:51:02.517000", "content": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.
", "votes": 6, "replies": [] } ] }, { "id": 671390, "author_name": "Bertrand P", "author_url": "", "post_date": "2019-11-12T16:21:14.990000", "content": "I can confirm it is possible to be 2nd on the LB with a simple override of UCF data.\nI don't know if all publicly available test labels are known or shared but I am confident that these would be shared as the community here is focused on learning by sharing. So I hope and think the challenge is still fair even if the leak is disappointing... Moreover there is no proof that private test data is inline with public LB ones.\n@cpmpml I noticed web scraping is not a skill you want to exercise... I showed that I am able to use this skill and, I hope so, some others too. I am here to learn ML. This challenge is still, in my opinion, very... challenging and interesting and I guess you love that! I would enjoy so much to work with you to purpose a ML solution to this challenge. Of course I would do the web scraping (and more) if needed, you would not have to do it! Please contact me to discuss on how we could cooperate if you are interested. Everybody here would win/learn if you participated in this challenge.
", "votes": 3, "replies": [ { "id": 671968, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-13T11:35:51.633000", "content": "That's a generous offer but I'll pass. Web scrapping will influence final ranking too much for my taste.
", "votes": 4, "replies": [] }, { "id": 672245, "author_name": "Hanjoon Choe", "author_url": "", "post_date": "2019-11-13T17:17:51.647000", "content": "web scraping is not a skill you want to exercise = Strongly agreed.\nI am here to learn ML = Strongly agreed.\nno proof that private test data is inline with public LB ones = Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.
\n\nMy faith do not use these data and some externel datasets, It is unfair. \nYes, someone says life itself is unfair. nope!! Not at all. Strongly disagreed.
", "votes": 2, "replies": [] }, { "id": 672355, "author_name": "Bertrand P", "author_url": "", "post_date": "2019-11-13T19:53:13.987000", "content": "@cpmpml This is not a generous offer ;) My goal is to learn ML and I am sure you would be a fabulous prof/mentor. You would have shared more than I would/could have. I know that doing is the best way to learn and I think that learning by doing beside you would have been even better. I already wanted to purpose to work as a team in Microsoft challenge but I was sure you would have not consider my offer. I tried this time to not regret. My offer is still and will ever be opened! I am disappointed that you have chosen to not be part of this challenge because I would have learned more. I just respect it. In the same time I feel proud you did not just ignore my offer. Maybe you even considered it...\nKaggle is full of disappointments for me... \"Microsoft Malware Prediction\" challenge still does not make sense to me (https://www.kaggle.com/c/microsoft-malware-prediction/discussion/83946#490008) and I still think that I deserved a better reward to my investment. As a lot of the great competitors that gave all... Once again in ASHRAE challenge I am disappointed. I come to learn ML and I have to web scrape... I wonder if Kaggle is the better place to spend (a lot of) my time... But I am sure Kaggle team try to do the best, ASHRAE too, and every participant here seems to try to do the same. I benefit a lot of what is shared here. I benefit a lot of what you (and many others) shared and share. I am grateful to all of you for that... So I am going to keep fighting. I gonna fight with ML tools. I will, as I have done this weekend, just exploit external data shared by other as it is part of the challenge/rules. I do not like it but I have to do it. Narcism!\n@hanjoonchoe \"Someone already said that they jumped up to 1st place after they replace meter_recording with the leaked data.\" I jumped too to 1st place by replacing UCF data in a submission that scored 1.06 without exploiting the leak. It scored 1.04. I did that to verify (probe) if that data were used in the public LB. I was happy. Narcism! In the same time I was disappointed: this is a huge leak and mean more. The data can probably be used to more than replacement... Is it still a real world problem? A real ML problem? Are we just competing? But it does mean nothing about the data used in the private test. As far as I understand... This is to me the motivation to continue. I try to trust everyone: Kaggle team is going to make the choices to a fair (I think it exists ;)) competition and the competitors gonna play fair and share external data. Not too much narcism!
", "votes": 0, "replies": [] }, { "id": 672687, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-14T04:19:12.100000", "content": "@pdnartreb Don't be disappointed by Kaggle. Kaggle competition are hard, many of the best datascientists come and compete.
\n\nIn Malware you should have asked. I see you did well there, being one of the few top 100 public teams to stay in top 100 in private.
", "votes": 1, "replies": [] } ] }, { "id": 674049, "author_name": "mezoganet", "author_url": "", "post_date": "2019-11-15T20:54:42.937000", "content": "\"Is this a webscraping competition?\"
\n\nOh Yes It seems more than illusion !\nThanks once again for your remarks, your warns, and your advices...
", "votes": 2, "replies": [] }, { "id": 673349, "author_name": "Vishal Sharma", "author_url": "", "post_date": "2019-11-14T21:45:03.797000", "content": "\nRules clarification from Kaggle:\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345
I am afraid it might end up being one - although to be fair, if you provide super detailed info like measurements for energy and weather but obfuscate the actual locations, you're practically asking for it ...
", "votes": 1, "replies": [ { "id": 670756, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-11T19:50:40.403000", "content": "Thanks, I'll pass then. Good luck to you, you're bound by your good LB rank I'm afraid.
", "votes": 1, "replies": [] }, { "id": 670783, "author_name": "Konrad Banachewicz", "author_url": "", "post_date": "2019-11-11T20:52:33", "content": "Thanks :-) To be honest, using the scraped public data as part of your training set is not that hard - so I was actually surprised more people have not done it so far.
", "votes": 2, "replies": [] }, { "id": 670820, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-11T22:48:17.837000", "content": "Maybe not hard, but not a skill I'd like to exercise ;)
", "votes": 10, "replies": [] }, { "id": 673337, "author_name": "Vishal Sharma", "author_url": "", "post_date": "2019-11-14T21:21:07.690000", "content": "+1
", "votes": 1, "replies": [] } ] }, { "id": 672160, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-13T15:11:00.297000", "content": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info here). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.
", "votes": -5, "replies": [ { "id": 672168, "author_name": "Rishabh Agrahari", "author_url": "", "post_date": "2019-11-13T15:23:50.887000", "content": "> scrap data and make high quality Datasets to share with other Kagglers
\n\nIs that legal? I mean we'd need consent of the website owners, no?
", "votes": 2, "replies": [] }, { "id": 672201, "author_name": "Chris Deotte", "author_url": "", "post_date": "2019-11-13T16:04:22.653000", "content": "Good question, I don't know the law. But if you read an online news article (or other public information), I think you are allowed to write and/or tell a friend what you read.
", "votes": 1, "replies": [] }, { "id": 672250, "author_name": "Gunes Evitan", "author_url": "", "post_date": "2019-11-13T17:27:48.807000", "content": "Most of the time those websites have robots.txt in their index. It is a standard used by websites to communicate with web crawlers. The standard informs the web crawler about which areas of the website should not be crawled. It is good practice to obey robots.txt.
Some websites even have sitemap.xml in their index which makes it easier for you to scrape data. If a website doesn't want you to take their data, they already have implemented a bot protection system, so don't worry.
Like anybody cared about the dataset tier...
", "votes": 3, "replies": [] }, { "id": 672378, "author_name": "Psi", "author_url": "", "post_date": "2019-11-13T20:19:54.887000", "content": "Some apparently do :P
", "votes": 3, "replies": [] }, { "id": 672679, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-14T04:12:15.683000", "content": "\n\n\nIs that legal? I mean we'd need consent of the website owners, no?
\n
A very relevant question indeed. From the Terms page (link at the bottom of any Kaggle page):
\n\n\n\n\nYou are responsible for all Content you contribute to the Services, and you represent and warrant you have all rights necessary to do so.
\n
Usually, sites with data have a License page somewhere. That's the best way to see if you can reproduce their data. If there is nothing that says it is open source or public domain then copyright applies, and you must ask permission.
", "votes": 2, "replies": [] }, { "id": 674076, "author_name": "Rob Rose", "author_url": "", "post_date": "2019-11-15T21:37:44.380000", "content": "IANAL, and this is only the relevant US law, but databases/datasets have a separate copyright from the data they're scraped from. While downloading the web pages and uploading them to Kaggle would violate copyright, just scraping the data would not, as the database you create is its own copyrightable work. Additionally, publicly available data is perfectly legal to scrape, at least according to the rulings currently on a lawsuit involving LinkedIn.
", "votes": 0, "replies": [] }, { "id": 674092, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-15T22:01:41.357000", "content": "The suit you point to isn't about copyright, it is about computer fraud and abuse act. What you missed is that LinkedIn users agreed to make their data publicly available, i.e. make it public domain. Therefore scrapping that data is fine.
\n\nI stand by what I wrote, and I advise anyone having doubt to consult with an attorney at law.
", "votes": 0, "replies": [] } ] }, { "id": 676833, "author_name": "Mukul Sharma", "author_url": "", "post_date": "2019-11-19T14:34:41.217000", "content": "I want to, how many of you are only using features provided by the kaggle? I mean I have tried everything with those features but I was unable to get anything. Is it external features that do the magic???
", "votes": 0, "replies": [] }, { "id": 670746, "author_name": "Luca Basanisi", "author_url": "", "post_date": "2019-11-11T19:45:40.373000", "content": "On the other hand, taking out any possibility of getting close to a medal because who cares about scraping more than others, there is plenty of space to try out some more creative, most likely less performant, approach.
\n\nWin-win. :)
", "votes": 0, "replies": [ { "id": 670755, "author_name": "CPMP", "author_url": "", "post_date": "2019-11-11T19:50:26.670000", "content": "There are other competitions where you can exercise ML skills and still get to the top if you're lucky and skilled. Here it seems you need to get into web scrapping, from the other answers I got, which means that I will pass.
", "votes": 5, "replies": [] }, { "id": 670763, "author_name": "Luca Basanisi", "author_url": "", "post_date": "2019-11-11T20:06:09.440000", "content": "It is really a shame that they didn't notice such a clear flaw in the competition design indeed.
\n\nWhat I meant is that, hopefully, many people are competing only to learn and that aspect is not compromised by the fact that this competition is ruined beyond recovery.
", "votes": 2, "replies": [] }, { "id": 670929, "author_name": "Martin Kovacevic Buvinic", "author_url": "", "post_date": "2019-11-12T04:01:51.980000", "content": "Tensorflow 2.0 competition seems promesing for learning. At my work i use tensorflow for productive modeling which is great for distributed training.
", "votes": 0, "replies": [] } ] } ], "raw_markdown_by_id": { "670723": "I am wondering which will be my next competition when I saw here that some test labels can be scrapped from web sites.\n\nThis reminds me of some previous image competitions where web scrapping became the most useful skill to win the competition.\n\nI welcome any feedback on this question: do you think all publicly available test labels are known and shared, or do you think it is a web scrapping competition now?", "670786": "Well, it looks like Google Analytics competition now. We need clarification from competition host ASAP.", "670737": "It seems that more of the test labels are becoming known. [site 1](https://www.kaggle.com/c/ashrae-energy-prediction/discussion/116773#670386) was discovered but not shared so it's only a matter of time all the test labels will be shared and revealed slowly. It is becoming a scraping competition... I find myself looking through university building energy data instead of machine learning.", "671390": "I can confirm it is possible to be 2nd on the LB with a simple override of UCF data.\nI don't know if all publicly available test labels are known or shared but I am confident that these would be shared as the community here is focused on learning by sharing. So I hope and think the challenge is still fair even if the leak is disappointing... Moreover there is no proof that private test data is inline with public LB ones.\n@cpmpml I noticed web scraping is not a skill you want to exercise... I showed that I am able to use this skill and, I hope so, some others too. I am here to learn ML. This challenge is still, in my opinion, very... challenging and interesting and I guess you love that! I would enjoy so much to work with you to purpose a ML solution to this challenge. Of course I would do the web scraping (and more) if needed, you would not have to do it! Please contact me to discuss on how we could cooperate if you are interested. Everybody here would win/learn if you participated in this challenge.", "674049": "**\"Is this a webscraping competition?\"**\n\nOh Yes It seems more than illusion !\nThanks once again for your remarks, your warns, and your advices...", "673349": "\nRules clarification from Kaggle:\nhttps://www.kaggle.com/c/ashrae-energy-prediction/discussion/117357#latest-673345", "670726": "I am afraid it might end up being one - although to be fair, if you provide super detailed info like measurements for energy and weather but obfuscate the actual locations, you're practically asking for it ...", "672160": "I have not been following this competition so I don't know what is going on, but I would like to remind everyone that Kaggle just added a new Dataset tier to everyone' profile (info [here][1]). Perhaps there is opportunity in this competition to scrap data and make Datasets to share with other Kagglers. You can learn data scraping and start acquiring Kaggle Dataset medals.\n\n[1]: https://www.kaggle.com/product-feedback/116079", "676833": "I want to, how many of you are only using features provided by the kaggle? I mean I have tried everything with those features but I was unable to get anything. Is it external features that do the magic???", "670746": "On the other hand, taking out any possibility of getting close to a medal because who cares about scraping more than others, there is plenty of space to try out some more creative, most likely less performant, approach. \n\nWin-win. :)" } }