{ "id": 5592, "title": "Restrictions?", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5592", "author_name": "", "post_date": "2013-08-30T20:40:51.973Z", "votes": null, "comment_count": 11, "views": 3662, "content": "

"The task is to predict the tags (a.k.a. keywords, topics, summaries), given only the question text and its title."

\n

 

\n

How literal is this, e.g.: No use of dictionaries? Or is analysis purely based on training data, no legitimate/neutral outside influences?

", "messages": [ { "id": "29799", "postDate": "08/30/2013 20:40:51", "content": "

"The task is to predict the tags (a.k.a. keywords, topics, summaries), given only the question text and its title."

\n

 

\n

How literal is this, e.g.: No use of dictionaries? Or is analysis purely based on training data, no legitimate/neutral outside influences?

", "rawMarkdown": "", "votes": null }, { "id": "29803", "postDate": "08/30/2013 20:47:11", "content": "

Scratch that, another case of RTFM:

\n

 

\n

"No External Data

\n

Participants are not allowed to use external data, including crawling/querying the Stack Exchange sites. You may be asked to provide your code to verify the legitimacy of your approach."

", "rawMarkdown": "", "votes": null }, { "id": "29804", "postDate": "08/30/2013 20:49:12", "content": "

No worries... RTFM is a quality Facebook likes to see in its applicants ;-)

", "rawMarkdown": "", "votes": null }, { "id": "29805", "postDate": "08/30/2013 20:51:08", "content": "

:) (y)

", "rawMarkdown": "", "votes": null }, { "id": "29806", "postDate": "08/30/2013 21:11:08", "content": "

I'd still like a clarification on what "querying the Stack Exchange sites" really is and what information one may use. Example: I am a Stack Exchange user. From my personal experience (a.k.a. querying by hand) I know that different SE sites allow different markup variations, i.e. LaTeX style formulae are only available on very few SE sites. This knowledge might make my choice of tags easier. Is this allowed? Or is this just a "text => algorithm => tags" challenge for really general methods?

\n

As an extension, if your answer to my first question is yes, how much does this generalize to using meta-information from the SE sites? Does the rule only mean one is forbidden from aquiring more questions as training data, or is it also forbidden to aquire meta-data from the SE sites. Examples of this meta-data include:

\n\n

I am asking this question because one of the example submissions is "javascript c# python php java", which assigns the same tags to all questions, based on the prior knowledge that Stack Overflow is one of the most popular (the most popular?) SE sites and that the aforementioned tags are very popular on Stack Overflow, making for a good general guess.

", "rawMarkdown": "", "votes": null }, { "id": "29807", "postDate": "08/30/2013 21:18:26", "content": "

Your question is a bit of a slippery-slope question. We can't forbid you from using some domain knowledge, e.g. "web sites use html and therefore this text may have html tags in it." The no scraping and no external data rules are there to prevent things like:

\n\n

To give a concrete answer, your model should take text and spit out tags. If we pulled new data at the end of the competition, you should be able to rerun your method and achieve similar performance without the need to hand-code new assumptions.

", "rawMarkdown": "", "votes": null }, { "id": "29808", "postDate": "08/30/2013 21:38:03", "content": "

I know this question is kind of mean, that's why it is important to me to clarify this issue. It's obvious that this rule exists to prevent cheating, and it is (or at least should be) obvious that one's model should not access external data sources.

\n

To put another perspective on my question: let's imagine that SE wanted to implement a solution like this on their sites, and provide a tag suggestion feature for new questions, or flag questions where the acutal tags and the computed tags don't line up at all. Let's also say, to comply with this challenge, that all SE sites would use one central tag suggestion service. It is reasonable to assume that this service won't scrape SE for similar questions. But it is also reasonable to assume that the service would have some sort of idea of the tags that are available on each site. This information might not be alwas up-to-date, the service might receive new tag lists from all SE sites only every day, week or month, but it might still help it to do its job.

\n

My previous question was in the light of this example. Should this challenge (and its rules) be seen more like an academic challenge on how to retrieve tags from a text (maybe with a few small, obvious tweaks), or (as this is a recruiting challenge) can one assume (and thus, hard-code) a (static) database of some kind? For example, the ten most popular tags on the ten most popular SE sites.

\n

I am truly sorry that I am trying to strain the rules and your patience, but since it is open for all participants, it could serve to establish a common ground on what's acceptable for this challenge and what's not. This common ground doesn't necessarily need to be laid out in one answer, but might arise through discussion, because it really is a complex thing, and as you said, a slippery slope.

", "rawMarkdown": "", "votes": null }, { "id": "29810", "postDate": "08/30/2013 21:56:53", "content": "

It's a great question, so no need to apologize for asking it.

\n

Our constraints of running a fair competition are certainly different than if Stack were to put this in a production system, and may not be ideal for such a system.  So, while you can build up a library of popular tags from the training set, you can't use Stack's synonym library or other meta data to build this.  The way to think about it is to imagine that this is a brand new site and the provided data is the only Stack data that exists.

\n

Another question that comes up often with NLP problems is whether you can use things like a parts of speech tagger.  We consider these tools more than data, and generally allow them.  In other words, you shouldn't have to reinvent the English language to solve an NLP problem. However, it can be a slippery slope with these things, so it's best to post to the forums if you think what you are using is more data than tool.

", "rawMarkdown": "", "votes": null }, { "id": "29811", "postDate": "08/30/2013 22:08:08", "content": "

Thanks! Your answer is really helpful regarding the questions I'm having. Especially the

\n

The way to think about it is to imagine that this is a brand new site and the provided data is the only Stack data that exists.

\n

is a good rule to go, as it eliminates my questions regarding a list of SE sites and a comprehensive list of all SE tags, as the proper way to generate those lists would be to examine the training data.

", "rawMarkdown": "", "votes": null }, { "id": "29816", "postDate": "08/30/2013 22:58:43", "content": "

Hi William Cukierski!

\n

Following your post about /not reinventing the English language/, I'd like to ask whether the use of external data is allowed provided that the data does not depend on the training data. I'm talking about pure linguistic copora, such as the well-known Brown corpus or other pure semantic lexicon. If not, am I right saying that the problem is more of a Machine Learning problem than of a NLP problem as mentioned earlier?

\n

Cheers!

", "rawMarkdown": "", "votes": null }, { "id": "29821", "postDate": "08/31/2013 01:56:58", "content": "

Similarly to HMICN's question, are we allowed to use lexical databases (ie: Wordnet) for this competition? 

\n

 

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "29822", "postDate": "08/31/2013 03:28:14", "content": "

"Participants are not allowed to use external data" seems pretty clear to me =)

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 29803, "author_name": "justinrovang", "author_url": "", "post_date": "08/30/2013 20:47:11", "content": "

Scratch that, another case of RTFM:

\n

 

\n

"No External Data

\n

Participants are not allowed to use external data, including crawling/querying the Stack Exchange sites. You may be asked to provide your code to verify the legitimacy of your approach."

", "votes": null, "replies": [] }, { "id": 29804, "author_name": "wcukierski", "author_url": "", "post_date": "08/30/2013 20:49:12", "content": "

No worries... RTFM is a quality Facebook likes to see in its applicants ;-)

", "votes": null, "replies": [] }, { "id": 29805, "author_name": "costella", "author_url": "", "post_date": "08/30/2013 20:51:08", "content": "

:) (y)

", "votes": null, "replies": [] }, { "id": 29806, "author_name": "carstenbrachem", "author_url": "", "post_date": "08/30/2013 21:11:08", "content": "

I'd still like a clarification on what "querying the Stack Exchange sites" really is and what information one may use. Example: I am a Stack Exchange user. From my personal experience (a.k.a. querying by hand) I know that different SE sites allow different markup variations, i.e. LaTeX style formulae are only available on very few SE sites. This knowledge might make my choice of tags easier. Is this allowed? Or is this just a "text => algorithm => tags" challenge for really general methods?

\n

As an extension, if your answer to my first question is yes, how much does this generalize to using meta-information from the SE sites? Does the rule only mean one is forbidden from aquiring more questions as training data, or is it also forbidden to aquire meta-data from the SE sites. Examples of this meta-data include:

\n\n

I am asking this question because one of the example submissions is "javascript c# python php java", which assigns the same tags to all questions, based on the prior knowledge that Stack Overflow is one of the most popular (the most popular?) SE sites and that the aforementioned tags are very popular on Stack Overflow, making for a good general guess.

", "votes": null, "replies": [] }, { "id": 29807, "author_name": "wcukierski", "author_url": "", "post_date": "08/30/2013 21:18:26", "content": "

Your question is a bit of a slippery-slope question. We can't forbid you from using some domain knowledge, e.g. "web sites use html and therefore this text may have html tags in it." The no scraping and no external data rules are there to prevent things like:

\n\n

To give a concrete answer, your model should take text and spit out tags. If we pulled new data at the end of the competition, you should be able to rerun your method and achieve similar performance without the need to hand-code new assumptions.

", "votes": null, "replies": [] }, { "id": 29808, "author_name": "carstenbrachem", "author_url": "", "post_date": "08/30/2013 21:38:03", "content": "

I know this question is kind of mean, that's why it is important to me to clarify this issue. It's obvious that this rule exists to prevent cheating, and it is (or at least should be) obvious that one's model should not access external data sources.

\n

To put another perspective on my question: let's imagine that SE wanted to implement a solution like this on their sites, and provide a tag suggestion feature for new questions, or flag questions where the acutal tags and the computed tags don't line up at all. Let's also say, to comply with this challenge, that all SE sites would use one central tag suggestion service. It is reasonable to assume that this service won't scrape SE for similar questions. But it is also reasonable to assume that the service would have some sort of idea of the tags that are available on each site. This information might not be alwas up-to-date, the service might receive new tag lists from all SE sites only every day, week or month, but it might still help it to do its job.

\n

My previous question was in the light of this example. Should this challenge (and its rules) be seen more like an academic challenge on how to retrieve tags from a text (maybe with a few small, obvious tweaks), or (as this is a recruiting challenge) can one assume (and thus, hard-code) a (static) database of some kind? For example, the ten most popular tags on the ten most popular SE sites.

\n

I am truly sorry that I am trying to strain the rules and your patience, but since it is open for all participants, it could serve to establish a common ground on what's acceptable for this challenge and what's not. This common ground doesn't necessarily need to be laid out in one answer, but might arise through discussion, because it really is a complex thing, and as you said, a slippery slope.

", "votes": null, "replies": [] }, { "id": 29810, "author_name": "wcukierski", "author_url": "", "post_date": "08/30/2013 21:56:53", "content": "

It's a great question, so no need to apologize for asking it.

\n

Our constraints of running a fair competition are certainly different than if Stack were to put this in a production system, and may not be ideal for such a system.  So, while you can build up a library of popular tags from the training set, you can't use Stack's synonym library or other meta data to build this.  The way to think about it is to imagine that this is a brand new site and the provided data is the only Stack data that exists.

\n

Another question that comes up often with NLP problems is whether you can use things like a parts of speech tagger.  We consider these tools more than data, and generally allow them.  In other words, you shouldn't have to reinvent the English language to solve an NLP problem. However, it can be a slippery slope with these things, so it's best to post to the forums if you think what you are using is more data than tool.

", "votes": null, "replies": [] }, { "id": 29811, "author_name": "carstenbrachem", "author_url": "", "post_date": "08/30/2013 22:08:08", "content": "

Thanks! Your answer is really helpful regarding the questions I'm having. Especially the

\n

The way to think about it is to imagine that this is a brand new site and the provided data is the only Stack data that exists.

\n

is a good rule to go, as it eliminates my questions regarding a list of SE sites and a comprehensive list of all SE tags, as the proper way to generate those lists would be to examine the training data.

", "votes": null, "replies": [] }, { "id": 29816, "author_name": "david99", "author_url": "", "post_date": "08/30/2013 22:58:43", "content": "

Hi William Cukierski!

\n

Following your post about /not reinventing the English language/, I'd like to ask whether the use of external data is allowed provided that the data does not depend on the training data. I'm talking about pure linguistic copora, such as the well-known Brown corpus or other pure semantic lexicon. If not, am I right saying that the problem is more of a Machine Learning problem than of a NLP problem as mentioned earlier?

\n

Cheers!

", "votes": null, "replies": [] }, { "id": 29821, "author_name": "zachfry", "author_url": "", "post_date": "08/31/2013 01:56:58", "content": "

Similarly to HMICN's question, are we allowed to use lexical databases (ie: Wordnet) for this competition? 

\n

 

\n

 

", "votes": null, "replies": [] }, { "id": 29822, "author_name": "alecradford", "author_url": "", "post_date": "08/31/2013 03:28:14", "content": "

"Participants are not allowed to use external data" seems pretty clear to me =)

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "29799": "", "29803": "", "29804": "", "29805": "", "29806": "", "29807": "", "29808": "", "29810": "", "29811": "", "29816": "", "29821": "", "29822": "" }, "source": "meta" }