Hi,
\nI'm new to data handling entirely, but I was hoping to get a feel for how it's done. I know I have no chance at competing in a test like this, but I was hoping someone could give me some quick pointers on how one goes about solving this kind of thing.
\nI have 2 questions, and since they're so basic, I'm sure it won't violate any rules to answer them. I'm sure they're really dumb actually, but here goes...
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary? I've never used either of them, but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation? If so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
", "messages": [ { "id": "30012", "postDate": "09/02/2013 21:10:20", "content": "Hi,
\nI'm new to data handling entirely, but I was hoping to get a feel for how it's done. I know I have no chance at competing in a test like this, but I was hoping someone could give me some quick pointers on how one goes about solving this kind of thing.
\nI have 2 questions, and since they're so basic, I'm sure it won't violate any rules to answer them. I'm sure they're really dumb actually, but here goes...
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary? I've never used either of them, but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation? If so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
", "rawMarkdown": "", "votes": null }, { "id": "30013", "postDate": "09/02/2013 21:26:00", "content": "Hey Akhil,
\nBasically, you can use whatever techniques (within the rules) that you like. For Facebook Recruiting 1, I had my own libraries that I was building and was already playing around with (on the Heritage Health Prize, but had only started). I'd never analyzed social graph data before. I didn't load the data into databases or use the same libraries (or even languages) that others were using. But at the time I dropped out of the comp, I was doing better than all of those other methods (well, except for the guy at #1). ;)
\nI still got an interview -- and a job. :)
\nIt sounds like you're still on an initial learning curve, so you might not be in exactly the same boat, but you can use whatever you're most familiar with. Remember: many data scientists don't even know how to program in C++ ... :)
\nJohn
", "rawMarkdown": "", "votes": null }, { "id": "30015", "postDate": "09/02/2013 21:33:07", "content": "[quote=Akhil Verghese;30012]
\nHi,
\nI'm new to data handling entirely, but I was hoping to get a feel for how it's done. I know I have no chance at competing in a test like this, but I was hoping someone could give me some quick pointers on how one goes about solving this kind of thing.
\nI have 2 questions, and since they're so basic, I'm sure it won't violate any rules to answer them. I'm sure they're really dumb actually, but here goes...
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary? I've never used either of them, but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation? If so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
\n[/quote]
\n\n
Welcome to the community, Akhil. Kaggle is a great place to start learning.
\n1) Not at all. People are just being helpful by sharing snippets. It is by no means the only way to do it. For instance, I use Python and loading data is as simple as, `pd.read_csv('Train.csv')`. Higher-level languages are definitely worth looking into if you want to play around with data and get a feel for it. If you know MATLAB, you should be good to go. I've heard that it has some good libraries for ML.
\n2) If I understand your question, are you talking about all possible tags? If so, I don't think that it is restricted to what you see in Train.csv. It could be from the Train.csv's Tags data or something else that isn't listed as well. I assume it must be the case since I suspect the top-5 words benchmark counts words from the Body as well.
", "rawMarkdown": "", "votes": null }, { "id": "30017", "postDate": "09/02/2013 21:35:02", "content": "[quote=Akhil Verghese;30012]
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary?
\n[/quote]
\nIt is not necessary. People use whatever tool they are comfortable with. R & Python are the most popular.
\n[quote=Akhil Verghese;30012]
\n... but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation?
\n[/quote]
\nThis kind of data is not too large especially if you are going to be working with C++. Note: R is memory-hog when it comes to certain things. With a language like C++ you could do a lot with much less memory foot print. Try to find something online on how to apply C++ to machine learning.
\n[quote=Akhil Verghese;30012]
\nIf so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n[/quote]
\nNot sure about Matlab's text processing capabilities. The best way, I found, to learn R or SQL (or for that matter any language) is to start with mini tasks and work at it.
\n[quote=Akhil Verghese;30012]
\n\n
2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
\n[/quote]
\nThe challenge is very simple. You are given a set of inputs(the title & body of the questions in the training file) & a set of outputs (the tags/labels). You need to model a mapping which matches the inputs with the outputs as accurately as possible.
\nOnce this mapping has been learnt then apply it to a new set of inputs(i.e., the test dataset) and generate the outputs and submit them to the leaderboard. The organiser knows the tags for the test set- we don't.
", "rawMarkdown": "", "votes": null }, { "id": "30125", "postDate": "09/04/2013 03:14:45", "content": "Thanks so much, guys. I'm clear on what to do now. I'm still no where near having a clue on how to even begin solving it, but at least I know what the question is, so thanks a ton. I started learning natural language processing and machine learning online, and I'm slowly getting the hang of it. Who knows, I may even have a submission by the end of this.
\nGood luck to all of you.
", "rawMarkdown": "", "votes": null }, { "id": "30711", "postDate": "09/11/2013 01:34:00", "content": "@Akhil,
\nIf you are a fan of C++ you may want to check this out::
\n\n", "rawMarkdown": "", "votes": null }, { "id": "30758", "postDate": "09/11/2013 20:35:48", "content": "
Thanks a lot, Damian, this really helps.
", "rawMarkdown": "", "votes": null }, { "id": "31319", "postDate": "09/21/2013 21:57:40", "content": "Hi,
\nI am a data management processional (mainly SQL and SAS) but always fancied statistical side of data.
\nCould be a silly questions, but want to know is this a kind of platform where I can start learning (from online resources) and applying those learning to actual competions posted here to get into the field of data science? If yes, any recommendations as I am thinking of Python?
\nRegards
", "rawMarkdown": "", "votes": null }, { "id": "31335", "postDate": "09/22/2013 09:44:50", "content": "Hi bhavin, Python is great. As mentioned above R is the other language that most people tend to start with. If you go down the Python route then I suggest looking at Pandas package (for loading and looking through data), scikit learn (for machine learning algorithms) and nltk package (which is a specialised package for natural language programming). Python has loads of resources online (as does R). Alternatively you could use SAS if you already know it or even SQL. From my experience it is usually the cleverness of the approach that gets you farthest, not the language you write in.
", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 30013, "author_name": "costella", "author_url": "", "post_date": "09/02/2013 21:26:00", "content": "Hey Akhil,
\nBasically, you can use whatever techniques (within the rules) that you like. For Facebook Recruiting 1, I had my own libraries that I was building and was already playing around with (on the Heritage Health Prize, but had only started). I'd never analyzed social graph data before. I didn't load the data into databases or use the same libraries (or even languages) that others were using. But at the time I dropped out of the comp, I was doing better than all of those other methods (well, except for the guy at #1). ;)
\nI still got an interview -- and a job. :)
\nIt sounds like you're still on an initial learning curve, so you might not be in exactly the same boat, but you can use whatever you're most familiar with. Remember: many data scientists don't even know how to program in C++ ... :)
\nJohn
", "votes": null, "replies": [] }, { "id": 30015, "author_name": "rohits", "author_url": "", "post_date": "09/02/2013 21:33:07", "content": "[quote=Akhil Verghese;30012]
\nHi,
\nI'm new to data handling entirely, but I was hoping to get a feel for how it's done. I know I have no chance at competing in a test like this, but I was hoping someone could give me some quick pointers on how one goes about solving this kind of thing.
\nI have 2 questions, and since they're so basic, I'm sure it won't violate any rules to answer them. I'm sure they're really dumb actually, but here goes...
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary? I've never used either of them, but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation? If so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
\n[/quote]
\n\n
Welcome to the community, Akhil. Kaggle is a great place to start learning.
\n1) Not at all. People are just being helpful by sharing snippets. It is by no means the only way to do it. For instance, I use Python and loading data is as simple as, `pd.read_csv('Train.csv')`. Higher-level languages are definitely worth looking into if you want to play around with data and get a feel for it. If you know MATLAB, you should be good to go. I've heard that it has some good libraries for ML.
\n2) If I understand your question, are you talking about all possible tags? If so, I don't think that it is restricted to what you see in Train.csv. It could be from the Train.csv's Tags data or something else that isn't listed as well. I assume it must be the case since I suspect the top-5 words benchmark counts words from the Body as well.
", "votes": null, "replies": [] }, { "id": 30017, "author_name": "sashikanthdareddy", "author_url": "", "post_date": "09/02/2013 21:35:02", "content": "[quote=Akhil Verghese;30012]
\n1) I've noticed that people seem to be importing this data with postgreSQL or R. Is that necessary?
\n[/quote]
\nIt is not necessary. People use whatever tool they are comfortable with. R & Python are the most popular.
\n[quote=Akhil Verghese;30012]
\n... but would it be possible to import the data with a C++ program and run any manipulation you want on them before spitting out something like the sample output? Or is the data too large for that kind of manipulation?
\n[/quote]
\nThis kind of data is not too large especially if you are going to be working with C++. Note: R is memory-hog when it comes to certain things. With a language like C++ you could do a lot with much less memory foot print. Try to find something online on how to apply C++ to machine learning.
\n[quote=Akhil Verghese;30012]
\nIf so, would MATLAB work? If not, how does one go about learning something like R or postgreSQL?
\n[/quote]
\nNot sure about Matlab's text processing capabilities. The best way, I found, to learn R or SQL (or for that matter any language) is to start with mini tasks and work at it.
\n[quote=Akhil Verghese;30012]
\n\n
2) The solution is evaluated by the tags, right? I assume how you know which tags to use is from the training file? So if I understand the question correctly, we're supposed to generate the tags for the "Test" file based on inferences we can gather on how they are generated in the "Train" file. There is no list of accepted tags, so I assume we should use the ones in the "Train" file?
\nAgain, sorry for the super basic questions, and thanks in advance for taking the time.
\n[/quote]
\nThe challenge is very simple. You are given a set of inputs(the title & body of the questions in the training file) & a set of outputs (the tags/labels). You need to model a mapping which matches the inputs with the outputs as accurately as possible.
\nOnce this mapping has been learnt then apply it to a new set of inputs(i.e., the test dataset) and generate the outputs and submit them to the leaderboard. The organiser knows the tags for the test set- we don't.
", "votes": null, "replies": [] }, { "id": 30125, "author_name": "akhilverghese", "author_url": "", "post_date": "09/04/2013 03:14:45", "content": "Thanks so much, guys. I'm clear on what to do now. I'm still no where near having a clue on how to even begin solving it, but at least I know what the question is, so thanks a ton. I started learning natural language processing and machine learning online, and I'm slowly getting the hang of it. Who knows, I may even have a submission by the end of this.
\nGood luck to all of you.
", "votes": null, "replies": [] }, { "id": 30711, "author_name": "damianmingle", "author_url": "", "post_date": "09/11/2013 01:34:00", "content": "@Akhil,
\nIf you are a fan of C++ you may want to check this out::
\n\n", "votes": null, "replies": [] }, { "id": 30758, "author_name": "akhilverghese", "author_url": "", "post_date": "09/11/2013 20:35:48", "content": "
Thanks a lot, Damian, this really helps.
", "votes": null, "replies": [] }, { "id": 31319, "author_name": "bhavin", "author_url": "", "post_date": "09/21/2013 21:57:40", "content": "Hi,
\nI am a data management processional (mainly SQL and SAS) but always fancied statistical side of data.
\nCould be a silly questions, but want to know is this a kind of platform where I can start learning (from online resources) and applying those learning to actual competions posted here to get into the field of data science? If yes, any recommendations as I am thinking of Python?
\nRegards
", "votes": null, "replies": [] }, { "id": 31335, "author_name": "soates", "author_url": "", "post_date": "09/22/2013 09:44:50", "content": "Hi bhavin, Python is great. As mentioned above R is the other language that most people tend to start with. If you go down the Python route then I suggest looking at Pandas package (for loading and looking through data), scikit learn (for machine learning algorithms) and nltk package (which is a specialised package for natural language programming). Python has loads of resources online (as does R). Alternatively you could use SAS if you already know it or even SQL. From my experience it is usually the cleverness of the approach that gets you farthest, not the language you write in.
", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "30012": "", "30013": "", "30015": "", "30017": "", "30125": "", "30711": "", "30758": "", "31319": "", "31335": "" }, "source": "meta" }