{ "id": 5791, "title": "Public data - tag definitions and synonyms", "url": "/competitions/facebook-recruiting-iii-keyword-extraction/discussion/5791", "author_name": "", "post_date": "2013-09-18T01:25:16.277Z", "votes": null, "comment_count": 7, "views": 3108, "content": "

Are we allowed scrape tag definitions from stackoverflow.com/tags?

\n

(Better still, can the admins just dump us a file of tag definitions?)

\n

How otherwise could you automatically disambiguate that (say) cherrypy and scipy are Python-related tags, but robocopy and spyware aren't?

\n

 

\n

Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. 

", "messages": [ { "id": "31067", "postDate": "09/18/2013 01:25:16", "content": "

Are we allowed scrape tag definitions from stackoverflow.com/tags?

\n

(Better still, can the admins just dump us a file of tag definitions?)

\n

How otherwise could you automatically disambiguate that (say) cherrypy and scipy are Python-related tags, but robocopy and spyware aren't?

\n

 

\n

Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. 

", "rawMarkdown": "", "votes": null }, { "id": "31083", "postDate": "09/18/2013 13:40:04", "content": "

[quote=Stephen McInerney;31067]

\n

Are we allowed scrape tag definitions from stackoverflow.com/tags?

\n

(Better still, can the admins just dump us a file of tag definitions?)

\n

How otherwise could you automatically disambiguate that (say) cherrypy and scipy are Python-related tags, but robocopy and spyware aren't?

\n

[/quote]

\n

You are not allowed to use external data.  The reason why we did not share tags and synonyms is on the evaluation page. By giving you a tag list we are potentially leaking information. Also, a keyword extractor should be willing and able suggest keywords it hasn't seen before!  If you want to know that cherrypy and scipy are python tags, you must deduce this from the training data.

\n

 [quote=Stephen McInerney;31067]

\n

Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. 

\n

[/quote]

\n

Yes, you can write a regex, but you can not write a regex that is just a lookup of external data. E.g. you can't dump the list of python libraries and then check if a word is in that list.

", "rawMarkdown": "", "votes": null }, { "id": "31085", "postDate": "09/18/2013 13:48:21", "content": "

Actually in my opinion the most efficient way to find out if X is python related -in general- is exactly to get a dump from stackoverflow and check the co-occurrence for X and python.

", "rawMarkdown": "", "votes": null }, { "id": "31088", "postDate": "09/18/2013 15:22:00", "content": "

My take on this is that you should have a sustainable mechanism that will add proof based on the training data knowledge, but could in the future learn from what else is added. When current knowledge fails (say you are not confident in predicting tags based on your mechanism for a specific post), you could propose some words currently in the title/body that are likely to be tags (so you would exclude words like "I", "which", "have" etc., and then see what might be a good tag.

", "rawMarkdown": "", "votes": null }, { "id": "31089", "postDate": "09/18/2013 15:27:43", "content": "

[quote=William Cukierski;31083][quote=Stephen McInerney;31067]Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. [/quote]

\n

Yes, you can write a regex, but you can not write a regex that is just a lookup of external data. E.g. you can't dump the list of python libraries and then check if a word is in that list.[/quote]

\n

Ok, but what sort of thing is legal to make into a regex? Can you give me a rough idea of where the limit is? Is there any legal way we could know 'memcpy' is a C function whereas 'copy' is just a generic word (to illustrate one pitfall)?

\n

 

\n

[quote=William Cukierski;31083]

\n

By giving you a tag list we are potentially leaking information. Also, a keyword extractor should be willing and able suggest keywords it hasn't seen before!  If you want to know that cherrypy and scipy are python tags, you must deduce this from the training data.

\n

[/quote]

\n

Ok but I'm just pointing out this increases the compute load and reduces accuracy.

\n

We can already build a tag list for those tags occurring in the data - just no definitions - and look at the similarity between actual tag names (as well as other criteria). 

\n

There are T=34945 tags in this training set.

\n

Clustering those is O(T^3) ... O(T^5), depending on how you do it. 

\n

For some clearcut sets of tags like '.*asp.net.*' tags, it is trivial to infer synonymy by using name correspondence. It's also way faster than clustering all of them. Conversely 'windows' tags are unrelated to 'windowless' or 'windowing', but name correspondence would not suggest that.

\n

I bring this up because spending large compute time on something like that conflicts with the optional requirement "run with less than 1GB RAM in under an hour". Especially since you would separately compute tag synonymy, offline, rarely, and they rarely change. And the fewer exemplars you have of a tag, the less likely you can cluster it. But whatever.

\n

 

\n

Just for illustration, all tags containing asp.net and their counts in training-set:

\n

asp.net (29355), asp.net-mvc (9554), asp.net-mvc-3 (5749), asp.net-mvc-2 (1539), asp.net-mvc-4 (1102), asp.net-ajax (523), asp.net-membership (462), asp.net-web-api (460), asp.net-mvc-routing (298), asp.net-webforms (223), asp.net-4.0 (217), asp.net-3.5 (143), asp.net-2.0 (126), jqgrid-asp.net (78), asp.net-mvc-areas (42), asp.net-routing (35), asp.net-profiles (31), asp.net-4.5 (27), asp.net-dynamic-data (25), asp.net-mvc-3-areas (24), asp.net-controls (22), asp.net-webpages (19), asp.net-charts (16), asp.net-mvc-2-validation (16), asp.net-mvc-partialview (15), asp.net-optimization (14), asp.net-mvc-ajax (13), asp.net-authentication (13), asp.net-roles (12), asp.net-1.1 (11), asp.net-mvc-validation (11), asp.net-mvc-controller (11), asp.net-authorization (9), kendo-asp.net-mvc (8), asp.net-web-api-routing (8), asp.net-mvc-views (7), asp.net-webcontrol (6), asp.net-caching (6), asp.net-development-serv (6), asp.net-mvc-futures (5), asp.net-mvc-viewmodel (4), asp.net-spa (3), asp.net-cache (3), asp.net-session (3), asp.net-mail (3), asp.net-mvc-beta1 (3), asp.net-mvc-filters (2), asp.net-mvc-sitemap (2), asp.net-mvc-apiexplorer (2), asp.net-mvc-templates (1), asp.net-mvc-migration (1), asp.net-customcontrol (1), asp.net-compiler (1)

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31090", "postDate": "09/18/2013 15:42:18", "content": "

This is a recruiting competition designed with the intent of not being the most efficient or accurate way to solve the problem.  We don't want it to be easy! We don't want to spoon feed the tags and make this a connect-the-dots game!

\n

I don't want to climb down the slippery slope of specific allowed commands, because it's a slippery slope indeed and ultimately a judgement call.  Nonetheless, I'll expand the example above. You can say things like:

\n

if str.contains('py') then tag 'python'

\n

You cannot say things like:

\n

if str.in('myExternalPythonList.csv') then tag 'python'

\n

 

", "rawMarkdown": "", "votes": null }, { "id": "31092", "postDate": "09/18/2013 15:56:38", "content": "

Agreed. It was only pointing out it makes the problem a little synthetic. But that's inevitable.

", "rawMarkdown": "", "votes": null }, { "id": "31122", "postDate": "09/18/2013 19:25:39", "content": "

It's indeed hard to cluster those "42048?" tags (mistyped tags or synonyms excluded). If you worry about the complexity, then I suggest you to approach the problem from a probabilistic point of view (like parallel monte carlo algorithms).

", "rawMarkdown": "", "votes": null } ], "comments": [ { "id": 31083, "author_name": "wcukierski", "author_url": "", "post_date": "09/18/2013 13:40:04", "content": "

[quote=Stephen McInerney;31067]

\n

Are we allowed scrape tag definitions from stackoverflow.com/tags?

\n

(Better still, can the admins just dump us a file of tag definitions?)

\n

How otherwise could you automatically disambiguate that (say) cherrypy and scipy are Python-related tags, but robocopy and spyware aren't?

\n

[/quote]

\n

You are not allowed to use external data.  The reason why we did not share tags and synonyms is on the evaluation page. By giving you a tag list we are potentially leaking information. Also, a keyword extractor should be willing and able suggest keywords it hasn't seen before!  If you want to know that cherrypy and scipy are python tags, you must deduce this from the training data.

\n

 [quote=Stephen McInerney;31067]

\n

Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. 

\n

[/quote]

\n

Yes, you can write a regex, but you can not write a regex that is just a lookup of external data. E.g. you can't dump the list of python libraries and then check if a word is in that list.

", "votes": null, "replies": [] }, { "id": 31085, "author_name": "abalogh", "author_url": "", "post_date": "09/18/2013 13:48:21", "content": "

Actually in my opinion the most efficient way to find out if X is python related -in general- is exactly to get a dump from stackoverflow and check the co-occurrence for X and python.

", "votes": null, "replies": [] }, { "id": 31088, "author_name": "pvkooten", "author_url": "", "post_date": "09/18/2013 15:22:00", "content": "

My take on this is that you should have a sustainable mechanism that will add proof based on the training data knowledge, but could in the future learn from what else is added. When current knowledge fails (say you are not confident in predicting tags based on your mechanism for a specific post), you could propose some words currently in the title/body that are likely to be tags (so you would exclude words like "I", "which", "have" etc., and then see what might be a good tag.

", "votes": null, "replies": [] }, { "id": 31089, "author_name": "smcinerney", "author_url": "", "post_date": "09/18/2013 15:27:43", "content": "

[quote=William Cukierski;31083][quote=Stephen McInerney;31067]Fail that, is it allowed to write a regex to define which tags are synonymous/related? I don't like the idea of wasting time on clustering things which are self-evident. Or each of constructing our own ontology for SE tags. [/quote]

\n

Yes, you can write a regex, but you can not write a regex that is just a lookup of external data. E.g. you can't dump the list of python libraries and then check if a word is in that list.[/quote]

\n

Ok, but what sort of thing is legal to make into a regex? Can you give me a rough idea of where the limit is? Is there any legal way we could know 'memcpy' is a C function whereas 'copy' is just a generic word (to illustrate one pitfall)?

\n

 

\n

[quote=William Cukierski;31083]

\n

By giving you a tag list we are potentially leaking information. Also, a keyword extractor should be willing and able suggest keywords it hasn't seen before!  If you want to know that cherrypy and scipy are python tags, you must deduce this from the training data.

\n

[/quote]

\n

Ok but I'm just pointing out this increases the compute load and reduces accuracy.

\n

We can already build a tag list for those tags occurring in the data - just no definitions - and look at the similarity between actual tag names (as well as other criteria). 

\n

There are T=34945 tags in this training set.

\n

Clustering those is O(T^3) ... O(T^5), depending on how you do it. 

\n

For some clearcut sets of tags like '.*asp.net.*' tags, it is trivial to infer synonymy by using name correspondence. It's also way faster than clustering all of them. Conversely 'windows' tags are unrelated to 'windowless' or 'windowing', but name correspondence would not suggest that.

\n

I bring this up because spending large compute time on something like that conflicts with the optional requirement "run with less than 1GB RAM in under an hour". Especially since you would separately compute tag synonymy, offline, rarely, and they rarely change. And the fewer exemplars you have of a tag, the less likely you can cluster it. But whatever.

\n

 

\n

Just for illustration, all tags containing asp.net and their counts in training-set:

\n

asp.net (29355), asp.net-mvc (9554), asp.net-mvc-3 (5749), asp.net-mvc-2 (1539), asp.net-mvc-4 (1102), asp.net-ajax (523), asp.net-membership (462), asp.net-web-api (460), asp.net-mvc-routing (298), asp.net-webforms (223), asp.net-4.0 (217), asp.net-3.5 (143), asp.net-2.0 (126), jqgrid-asp.net (78), asp.net-mvc-areas (42), asp.net-routing (35), asp.net-profiles (31), asp.net-4.5 (27), asp.net-dynamic-data (25), asp.net-mvc-3-areas (24), asp.net-controls (22), asp.net-webpages (19), asp.net-charts (16), asp.net-mvc-2-validation (16), asp.net-mvc-partialview (15), asp.net-optimization (14), asp.net-mvc-ajax (13), asp.net-authentication (13), asp.net-roles (12), asp.net-1.1 (11), asp.net-mvc-validation (11), asp.net-mvc-controller (11), asp.net-authorization (9), kendo-asp.net-mvc (8), asp.net-web-api-routing (8), asp.net-mvc-views (7), asp.net-webcontrol (6), asp.net-caching (6), asp.net-development-serv (6), asp.net-mvc-futures (5), asp.net-mvc-viewmodel (4), asp.net-spa (3), asp.net-cache (3), asp.net-session (3), asp.net-mail (3), asp.net-mvc-beta1 (3), asp.net-mvc-filters (2), asp.net-mvc-sitemap (2), asp.net-mvc-apiexplorer (2), asp.net-mvc-templates (1), asp.net-mvc-migration (1), asp.net-customcontrol (1), asp.net-compiler (1)

\n

 

", "votes": null, "replies": [] }, { "id": 31090, "author_name": "wcukierski", "author_url": "", "post_date": "09/18/2013 15:42:18", "content": "

This is a recruiting competition designed with the intent of not being the most efficient or accurate way to solve the problem.  We don't want it to be easy! We don't want to spoon feed the tags and make this a connect-the-dots game!

\n

I don't want to climb down the slippery slope of specific allowed commands, because it's a slippery slope indeed and ultimately a judgement call.  Nonetheless, I'll expand the example above. You can say things like:

\n

if str.contains('py') then tag 'python'

\n

You cannot say things like:

\n

if str.in('myExternalPythonList.csv') then tag 'python'

\n

 

", "votes": null, "replies": [] }, { "id": 31092, "author_name": "smcinerney", "author_url": "", "post_date": "09/18/2013 15:56:38", "content": "

Agreed. It was only pointing out it makes the problem a little synthetic. But that's inevitable.

", "votes": null, "replies": [] }, { "id": 31122, "author_name": "", "author_url": "", "post_date": "09/18/2013 19:25:39", "content": "

It's indeed hard to cluster those "42048?" tags (mistyped tags or synonyms excluded). If you worry about the complexity, then I suggest you to approach the problem from a probabilistic point of view (like parallel monte carlo algorithms).

", "votes": null, "replies": [] } ], "raw_markdown_by_id": { "31067": "", "31083": "", "31085": "", "31088": "", "31089": "", "31090": "", "31092": "", "31122": "" }, "source": "meta" }